Repository: apache/airflow

Issue:
GenericTransfer and Postgres - ERROR - SET AUTOCOMMIT TO OFF is no longer supported
Trying to implement a generic transfer

``` python
t1 = GenericTransfer(
  task_id = 'copy_small_table',
  sql = "select * from my_schema.my_table",
  destination_table = "my_schema.my_table",
  source_conn_id = "postgres9.1.13",
  destination_conn_id = "postgres9.4.5",
  dag=dag
)
```

I get the following error:

```
--------------------------------------------------------------------------------
New run starting @2015-11-25T11:05:40.673401
--------------------------------------------------------------------------------
[2015-11-25 11:05:40,698] {models.py:951} INFO - Executing <Task(GenericTransfer): copy_my_table_v1> on 2015-11-24 00:00:00
[2015-11-25 11:05:40,711] {base_hook.py:53} INFO - Using connection to: 10.x.x.x
[2015-11-25 11:05:40,711] {generic_transfer.py:53} INFO - Extracting data from my_db
[2015-11-25 11:05:40,711] {generic_transfer.py:54} INFO - Executing: 
select * from my_schema.my_table
[2015-11-25 11:05:40,713] {base_hook.py:53} INFO - Using connection to: 10.x.x.x
[2015-11-25 11:05:40,808] {base_hook.py:53} INFO - Using connection to: 10.x.x.x
[2015-11-25 11:05:45,271] {base_hook.py:53} INFO - Using connection to: 10.x.x.x
[2015-11-25 11:05:45,272] {generic_transfer.py:63} INFO - Inserting rows into 10.x.x.x
[2015-11-25 11:05:45,273] {base_hook.py:53} INFO - Using connection to: 10.x.x.x
[2015-11-25 11:05:45,305] {models.py:1017} ERROR - SET AUTOCOMMIT TO OFF is no longer supported
Traceback (most recent call last):
  File "/usr/local/lib/python2.7/dist-packages/airflow/models.py", line 977, in run
    result = task_copy.execute(context=context)
  File "/usr/local/lib/python2.7/dist-packages/airflow/operators/generic_transfer.py", line 64, in execute
    destination_hook.insert_rows(table=self.destination_table, rows=results)
  File "/usr/local/lib/python2.7/dist-packages/airflow/hooks/dbapi_hook.py", line 136, in insert_rows
    cur.execute('SET autocommit = 0')
NotSupportedError: SET AUTOCOMMIT TO OFF is no longer supported

[2015-11-25 11:05:45,330] {models.py:1053} ERROR - SET AUTOCOMMIT TO OFF is no longer supported
```

Python 2.7
Airflow 1.6.1
psycopg2 2.6 (Also tried 2.6.1)
Postgeres destination 9.4.5

Any idea on what might cause this problem?



Discussion:
We don't run postgres at Airbnb so I can't really test a fix, but the first thing I'd try would be to change that line to `False`.
https://github.com/airbnb/airflow/blob/master/airflow/hooks/postgres_hook.py#L12

As a side note, autocommit in DbApiHook should probably be set in a different way, perhaps a `set_autocommit(cur, conn, autocommit=True)` method that receives an active cursor and connection object and does what it needs to do. Class deriving DbApiHook for specific DBs would have to implement it.

The above fix solved our problems.


Resolving patch:
diff --git a/airflow/hooks/postgres_hook.py b/airflow/hooks/postgres_hook.py
--- a/airflow/hooks/postgres_hook.py
+++ b/airflow/hooks/postgres_hook.py
@@ -11,7 +11,7 @@ class PostgresHook(DbApiHook):
     '''
     conn_name_attr = 'postgres_conn_id'
     default_conn_name = 'postgres_default'
-    supports_autocommit = True
+    supports_autocommit = False
 
     def get_conn(self):
         conn = self.get_connection(self.postgres_conn_id)
@@ -25,4 +25,7 @@ def get_conn(self):
         for arg_name, arg_val in conn.extra_dejson.items():
             if arg_name in ['sslmode', 'sslcert', 'sslkey', 'sslrootcert', 'sslcrl']:
                 conn_args[arg_name] = arg_val
-        return psycopg2.connect(**conn_args)
+        psycopg2_conn = psycopg2.connect(**conn_args)
+        if psycopg2_conn.server_version < 70400:
+            self.supports_autocommit = True
+        return psycopg2_conn


Test patch: