Repository: apache/airflow

Issue:
Increase typing coverage for postgres provider
This PR is about increasing typing coverage for the postgres provider. Part of: #9708

I change this file airflow/providers/postgres/hooks/postgres.py

closes: #9943
related: #9708

---
I go throw this guideline [Step 4: Prepare PR](https://github.com/apache/airflow/blob/master/CONTRIBUTING.rst#step-4-prepare-pr)


## [Run the unit tests from the IDE or local virtualenv as you see fit.](https://github.com/apache/airflow/blob/master/TESTING.rst#running-unit-tests)

run the command bellow

``` bash
$ pytest tests/providers/postgres/
```
result

``` bash
============================================================ test session starts =============================================================
platform linux -- Python 3.7.0, pytest-6.0.1, py-1.9.0, pluggy-0.13.1 -- /home/salem/.pyenv/versions/3.7.0/envs/airflow-venv-370/bin/python3.7
cachedir: .pytest_cache
rootdir: /home/salem/WorkSpaces/python/airflow, configfile: pytest.ini
plugins: rerunfailures-9.0, forked-1.3.0, flaky-3.7.0, cov-2.10.1, requests-mock-1.8.0, timeouts-1.2.1, instafail-0.4.2, xdist-2.0.0
setup timeout: 0.0s, execution timeout: 0.0s, teardown timeout: 0.0s
collected 20 items                    

tests/providers/postgres/hooks/test_postgres.py::TestPostgresHookConn::test_get_conn PASSED                                            [  5%]
tests/providers/postgres/hooks/test_postgres.py::TestPostgresHookConn::test_get_conn_cursor PASSED                                     [ 10%]
tests/providers/postgres/hooks/test_postgres.py::TestPostgresHookConn::test_get_conn_from_connection PASSED                            [ 15%]
tests/providers/postgres/hooks/test_postgres.py::TestPostgresHookConn::test_get_conn_from_connection_with_schema PASSED                [ 20%]
tests/providers/postgres/hooks/test_postgres.py::TestPostgresHookConn::test_get_conn_non_default_id PASSED                             [ 25%]
tests/providers/postgres/hooks/test_postgres.py::TestPostgresHookConn::test_get_conn_rds_iam_postgres PASSED                           [ 30%]
tests/providers/postgres/hooks/test_postgres.py::TestPostgresHookConn::test_get_conn_rds_iam_redshift PASSED                           [ 35%]
tests/providers/postgres/hooks/test_postgres.py::TestPostgresHookConn::test_get_conn_with_invalid_cursor PASSED                        [ 40%]
tests/providers/postgres/hooks/test_postgres.py::TestPostgresHook::test_bulk_dump SKIPPED                                              [ 45%]
tests/providers/postgres/hooks/test_postgres.py::TestPostgresHook::test_bulk_load SKIPPED                                              [ 50%]
tests/providers/postgres/hooks/test_postgres.py::TestPostgresHook::test_copy_expert SKIPPED                                            [ 55%]
tests/providers/postgres/hooks/test_postgres.py::TestPostgresHook::test_insert_rows SKIPPED                                            [ 60%]
tests/providers/postgres/hooks/test_postgres.py::TestPostgresHook::test_insert_rows_replace SKIPPED                                    [ 65%]
tests/providers/postgres/hooks/test_postgres.py::TestPostgresHook::test_insert_rows_replace_missing_replace_index_arg SKIPPED          [ 70%]
tests/providers/postgres/hooks/test_postgres.py::TestPostgresHook::test_insert_rows_replace_missing_target_field_arg SKIPPED           [ 75%]
tests/providers/postgres/hooks/test_postgres.py::TestPostgresHook::test_rowcount SKIPPED                                               [ 80%]
tests/providers/postgres/operators/test_postgres.py::TestPostgres::test_overwrite_schema SKIPPED                                       [ 85%]
tests/providers/postgres/operators/test_postgres.py::TestPostgres::test_postgres_operator_test SKIPPED                                 [ 90%]
tests/providers/postgres/operators/test_postgres.py::TestPostgres::test_postgres_operator_test_multi SKIPPED                           [ 95%]
tests/providers/postgres/operators/test_postgres.py::TestPostgres::test_vacuum SKIPPED                                                 [100%]
============================================================== warnings summary ==============================================================
================================================== 8 passed, 12 skipped, 1 warning in 0.91s ==================================================
``` 

## [Run the tests in Breeze.](https://github.com/apache/airflow/blob/master/TESTING.rst#running-tests-for-a-specified-target-using-breeze-from-the-host)

run the command bellow

``` bash
$ sudo ./breeze tests tests/providers/postgres
```

results

``` bash
============================================================== warnings summary ==============================================================
ERROR tests/providers/postgres/hooks/test_postgres.py::TestPostgresHookConn::test_get_conn - Failed: Timeout >10.0s
ERROR tests/providers/postgres/hooks/test_postgres.py::TestPostgresHookConn::test_get_conn_cursor - Failed: Timeout >10.0s
ERROR tests/providers/postgres/hooks/test_postgres.py::TestPostgresHookConn::test_get_conn_from_connection - Failed: Timeout >10.0s
ERROR tests/providers/postgres/hooks/test_postgres.py::TestPostgresHookConn::test_get_conn_from_connection_with_schema - Failed: Timeout >1...
ERROR tests/providers/postgres/hooks/test_postgres.py::TestPostgresHookConn::test_get_conn_non_default_id - Failed: Timeout >10.0s
ERROR tests/providers/postgres/hooks/test_postgres.py::TestPostgresHookConn::test_get_conn_rds_iam_postgres - Failed: Timeout >10.0s
ERROR tests/providers/postgres/hooks/test_postgres.py::TestPostgresHookConn::test_get_conn_rds_iam_redshift - Failed: Timeout >10.0s
ERROR tests/providers/postgres/hooks/test_postgres.py::TestPostgresHookConn::test_get_conn_with_invalid_cursor - Failed: Timeout >10.0s
================================================= 12 skipped, 2 warnings, 8 errors in 15.06s =================================================

Regular tests. NOT Updating Quarantine Issue!

```

## [Running static checks](https://github.com/apache/airflow/blob/master/BREEZE.rst#running-static-checks)

I don't know at this time, which static test should I run for this Issue,  but I used the default one as shown below

``` bash
sudo ./breeze static-check mypy
```

---
**^ Add meaningful description above**

Read the **[Pull Request Guidelines](https://github.com/apache/airflow/blob/master/CONTRIBUTING.rst#pull-request-guidelines)** for more information.
In case of fundamental code change, Airflow Improvement Proposal ([AIP](https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+Improvements+Proposals)) is needed.
In case of a new dependency, check compliance with the [ASF 3rd Party License Policy](https://www.apache.org/legal/resolved.html#category-x).
In case of backwards incompatible changes please leave a note in [UPDATING.md](https://github.com/apache/airflow/blob/master/UPDATING.md).

---
``` bash
    $ pytest tests/providers/postgres/

    $ sudo ./breeze tests tests/providers/postgres

    $ sudo ./breeze static-check pylint -- --all-files

    $ sudo ./breeze static-check pylint -- --files airflow/providers/postgres/

    $ sudo ./breeze static-check mypy

    $ mypy airflow/providers/postgres/hooks/postgres.py


    $ sudo ./breeze static-check pylint --show-diff-on-failure  --files airflow/providers

    $ sudo pre-commit run pylint --show-diff-on-failure --files airflow/providers

    $ sudo ./breeze static-check mypy -- --files airflow/providers/postgres/

```



``` bash
    sudo ./breeze static-check pylint -- --all-files
    sudo ./breeze static-check pylint -- --files airflow/providers/postgres/

    ./breeze static-check mypy -- --all-files
    sudo ./breeze static-check mypy -- --files airflow/providers/postgres/
```

:heart:

``` bash
pre-commit run
pre-commit run --all-files
pre-commit run mypy --all-files
pre-commit run pylint --all-files
```


```
git pull apache master
```

---
# TODO 

## Test fails on this stages

1.  ~~Static checks: except pylint [details](https://github.com/apache/airflow/pull/10864/checks?check_run_id=1116702359)~~
 > ` ./scripts/ci/static_checks/run_static_checks.sh`
2.  ~~Spell check docs [details](https://github.com/apache/airflow/pull/10864/checks?check_run_id=1116702223)~~
> `./scripts/ci/docs/ci_docs.sh --spellcheck-only`
3. CI Build / Core:Pg9.6,Py3.7 [details](https://github.com/apache/airflow/pull/10864/checks?check_run_id=1123482621)
> `./scripts/ci/testing/ci_run_airflow_testing.sh`



Discussion:
Congratulations on your first Pull Request and welcome to the Apache Airflow community! If you have any issues or are unsure about any anything please check our Contribution Guide (https://github.com/apache/airflow/blob/master/CONTRIBUTING.rst)
Here are some useful points:
- Pay attention to the quality of your code (flake8, pylint and type annotations). Our [pre-commits]( https://github.com/apache/airflow/blob/master/STATIC_CODE_CHECKS.rst#prerequisites-for-pre-commit-hooks) will help you with that.
- In case of a new feature add useful documentation (in docstrings or in `docs/` directory). Adding a new operator? Check this short [guide](https://github.com/apache/airflow/blob/master/docs/howto/custom-operator.rst) Consider adding an example DAG that shows how users should use it.
- Consider using [Breeze environment](https://github.com/apache/airflow/blob/master/BREEZE.rst) for testing locally, it’s a heavy docker but it ships with a working Airflow and a lot of integrations.
- Be patient and persistent. It might take some time to get a review or get the final approval from Committers.
- Please follow [ASF Code of Conduct](https://www.apache.org/foundation/policies/conduct) for all communication including (but not limited to) comments on Pull Requests, Mailing list and Slack.
- Be sure to read the [Airflow Coding style]( https://github.com/apache/airflow/blob/master/CONTRIBUTING.rst#coding-style-and-best-practices).
Apache Airflow is a community-driven project and together we are making it better 🚀.
In case of doubts contact the developers at:
Mailing List: dev@airflow.apache.org
Slack: https://apache-airflow-slack.herokuapp.com/

``` bash
MP_DIR=$(mktemp -d); mypy --linecount-report ${MP_DIR} airflow/; cat "${MP_DIR}/linecount.txt" | grep providers | grep -v example_dags | awk '
$4 != 0 {
    split($5, a, ".");
    print 100.00 * ($3/$4), a[3]
}'| awk '
{ sum[$2] += $1; N[$2]++ } 
END { 
    for (key in sum) {
        avg = sum[key] / N[key];
        printf "%f %s\n", avg, key;
    } 
}' | sort -g
```

```
0,000000 dingding
0,000000 opsgenie
0,000000 presto
0,000000 qubole
0,000000 samba
0,000000 zendesk
16,666650 ssh
16,666667 oracle
16,666667 yandex
16,666700 odbc
21,428550 databricks
25,000000 exasol
25,000000 jdbc
25,000000 singularity
25,000000 snowflake
27,380950 slack
27,964227 amazon
33,333333 sftp
46,153850 imap
50,733332 microsoft
66,761375 cncf
67,857150 salesforce
69,117976 google
71,250000 ftp
75,000000 postgres  <<< ==== was 25.00 now 75.00 but should be 100
83,333300 openfaas
87,500000 redis
87,921875 apache
88,333340 mysql
88,888900 papermill
96,875000 mongo
100,000000 celery
100,000000 cloudant
100,000000 datadog
100,000000 discord
100,000000 docker
100,000000 elasticsearch
100,000000 facebook
100,000000 grpc
100,000000 hashicorp
100,000000 http
100,000000 jenkins
100,000000 jira
100,000000 pagerduty
100,000000 segment
100,000000 sendgrid
100,000000 sqlite
100,000000 vertica

```
I run the command bellow according to this note, so  my file `providers/postgres` is reformatted

```  
Make sure your code fulfils all the static code checks we have in our code. The easiest way to make sure of that is to use pre-commit hooks
```

``` bash
$ sudo pre-commit run --all-files
```
The CI errors seem to be similar to #10153 as reported by @yuqian90 
> The CI errors seem to be similar to #10153 as reported by @yuqian90

Seems to be this issue: https://github.com/apache/airflow/pull/10877 @SalAlba can you rebase please?
> > The CI errors seem to be similar to #10153 as reported by @yuqian90
> 
> Seems to be this issue: #10877 @SalAlba can you rebase please?

**For rebase** I made this

```
git checkout sal-dev

git rebase master

git checkout master

git merge sal-dev
```

_Hopefully this is the correct way !_


> > > The CI errors seem to be similar to #10153 as reported by @yuqian90
> > 
> > 
> > Seems to be this issue: #10877 @SalAlba can you rebase please?
> 
> **For rebase** I made this
> 
> ```
> git checkout sal-dev
> 
> git rebase master
> 
> git checkout master
> 
> git merge sal-dev
> ```
> 
> _Hopefully this is the correct way !_

This is nearly perfect. You should first fetch newest changes from apache/master. If you don't have apache remote I suggest this:
```
git remote add apache git@github.com:apache/airflow.git
git fetch --all
# Then on your branch
git rebase apache/master
```
ok git rebase done, time now for skipping ` CI errors ` :fire: 
![Screenshot from 2020-09-14 16-51-50-1](https://user-images.githubusercontent.com/23059577/93101642-d06e8a80-f6aa-11ea-904e-86204c7e5ac6.png)

I'm wondering If I did some bad steps :warning:  , most of codes/commits come from other developers, and I suppose that happen when I do this steps : 

```
git remote add apache https://github.com/apache/airflow.git
git fetch --all
git rebase apache/master
git push
```

then I make some changes in my code especially on this file just `airflow/providers/postgres/hooks/postgres.py
` and then I checkout to my branch because, I wanted to run all actions (the whole tests ) in PR #10864 , so in command looks like this.


```
git checkout sal-dev

git rebase master

git push -u origin sal-dev
```
Is your `master` same as `apache/master`? `git fetch` does not apply remote changes to your local branch. I would suggest to rebase onto `apache/master` and do this always to avoid stale information 
> stale



> Is your `master` same as `apache/master`? `git fetch` does not apply remote changes to your local branch. I would suggest to rebase onto `apache/master` and do this always to avoid stale information


my master yes are the same the only diff in file `airflow/providers/postgres/hooks/postgres.py` 


Eureka guys !! now I start know how the stuffs should be done, and solve open source issues (before I didn't know since I'm new to this type of project)

For first time I should fork and clone the airflow repo, and make the configurations

```
git clone https://......
```

then just onec add ref to remote official airflow repo like bellow

```
git remote add apache https://github.com/apache/airflow.git
```

then fixing issues and make life happier :smile:, when it come to commit I should do this

```
git add .
git commit -m '.....'
```

then do the rebaseing like bellow, with `squash`

```
git fetch --all
git rebase apache/master
```

in case there are conflict fix it, if there are need of pullig also do and then 

```
git push
```

After all of that should come finally the `PR` From MyRepo/master -->> apache/master

So my question now I'm right ? if yes should I close this current PR and open new fresh and clean `PR` for my actual issue (Increase typing coverage for postgres provider) ??
> So my question now I'm right ? if yes should I close this current PR and open new fresh and clean `PR` for my actual issue (Increase typing coverage for postgres provider) ??


Assuming your changes should affect only `airflow/airflow/providers/postgres/hooks/postgres.py` I would the following:
- copy this file to other location outside repository
- do `git fetch --all` and `git reset --hard apache/master` on your branch `sal-dev`
- copy back the file
- create commit
- `git push --force-with-lease`

I think this is the simplest way to clean up this PR. And here are some useful tips:
https://github.com/apache/airflow/blob/master/CONTRIBUTING.rst#id49


Resolving patch:
diff --git a/airflow/providers/postgres/hooks/postgres.py b/airflow/providers/postgres/hooks/postgres.py
--- a/airflow/providers/postgres/hooks/postgres.py
+++ b/airflow/providers/postgres/hooks/postgres.py
@@ -18,12 +18,18 @@
 
 import os
 from contextlib import closing
+from typing import Iterable, Optional, Tuple, Union
 
 import psycopg2
 import psycopg2.extensions
 import psycopg2.extras
+from psycopg2.extensions import connection
+from psycopg2.extras import DictCursor, RealDictCursor, NamedTupleCursor
 
 from airflow.hooks.dbapi_hook import DbApiHook
+from airflow.models.connection import Connection
+
+CursorType = Union[DictCursor, RealDictCursor, NamedTupleCursor]
 
 
 class PostgresHook(DbApiHook):
@@ -53,13 +59,13 @@ class PostgresHook(DbApiHook):
     default_conn_name = 'postgres_default'
     supports_autocommit = True
 
-    def __init__(self, *args, **kwargs):
+    def __init__(self, *args, **kwargs) -> None:
         super().__init__(*args, **kwargs)
-        self.schema = kwargs.pop("schema", None)
-        self.connection = kwargs.pop("connection", None)
-        self.conn = None
+        self.schema: Optional[str] = kwargs.pop("schema", None)
+        self.connection: Optional[Connection] = kwargs.pop("connection", None)
+        self.conn: connection = None
 
-    def _get_cursor(self, raw_cursor):
+    def _get_cursor(self, raw_cursor: str) -> CursorType:
         _cursor = raw_cursor.lower()
         if _cursor == 'dictcursor':
             return psycopg2.extras.DictCursor
@@ -69,8 +75,10 @@ def _get_cursor(self, raw_cursor):
             return psycopg2.extras.NamedTupleCursor
         raise ValueError('Invalid cursor passed {}'.format(_cursor))
 
-    def get_conn(self):
-
+    def get_conn(self) -> connection:
+        """
+        Establishes a connection to a postgres database.
+        """
         conn_id = getattr(self, self.conn_name_attr)
         conn = self.connection or self.get_connection(conn_id)
 
@@ -104,7 +112,7 @@ def get_conn(self):
         self.conn = psycopg2.connect(**conn_args)
         return self.conn
 
-    def copy_expert(self, sql, filename):
+    def copy_expert(self, sql: str, filename: str) -> None:
         """
         Executes SQL using psycopg2 copy_expert method.
         Necessary to execute COPY command without access to a superuser.
@@ -126,13 +134,13 @@ def copy_expert(self, sql, filename):
                     file.truncate(file.tell())
                     conn.commit()
 
-    def bulk_load(self, table, tmp_file):
+    def bulk_load(self, table: str, tmp_file: str) -> None:
         """
         Loads a tab-delimited file into a database table
         """
         self.copy_expert("COPY {table} FROM STDIN".format(table=table), tmp_file)
 
-    def bulk_dump(self, table, tmp_file):
+    def bulk_dump(self, table: str, tmp_file: str) -> None:
         """
         Dumps a database table into a tab-delimited file
         """
@@ -140,7 +148,7 @@ def bulk_dump(self, table, tmp_file):
 
     # pylint: disable=signature-differs
     @staticmethod
-    def _serialize_cell(cell, conn):
+    def _serialize_cell(cell: object, conn: Optional[connection] = None) -> object:
         """
         Postgresql will adapt all arguments to the execute() method internally,
         hence we return cell without any conversion.
@@ -157,7 +165,7 @@ def _serialize_cell(cell, conn):
         """
         return cell
 
-    def get_iam_token(self, conn):
+    def get_iam_token(self, conn: Connection) -> Tuple[str, str, int]:
         """
         Uses AWSHook to retrieve a temporary password to connect to Postgres
         or Redshift. Port is required. If none is provided, default is used for
@@ -191,7 +199,9 @@ def get_iam_token(self, conn):
         return login, token, port
 
     @staticmethod
-    def _generate_insert_sql(table, values, target_fields, replace, **kwargs):
+    def _generate_insert_sql(
+        table: str, values: Tuple[str, ...], target_fields: Iterable[str], replace: bool, **kwargs
+    ) -> str:
         """
         Static helper method that generate the INSERT SQL statement.
         The REPLACE variant is specific to MySQL syntax.


Test patch: