Repository: apache/airflow

Issue:
MySQL hook uses wrong autocommit calls for mysql-connector-python
**Apache Airflow version**: 2.0.1

**Kubernetes version (if you are using kubernetes)** (use `kubectl version`): n/a

**Environment**:

* **Cloud provider or hardware configuration**:  WSL2/Docker running `apache/airflow:2.0.1-python3.7` image
* **OS** (e.g. from /etc/os-release): Host: Ubuntu 20.04 LTS, Docker Image: Debian GNU/Linux 10 (buster)
* **Kernel** (e.g. `uname -a`): 5.4.72-microsoft-standard-WSL2 x86_64
* **Others**:  Docker version 19.03.8, build afacb8b7f0

**What happened**:

Received a `'bool' object is not callable` error when attempting to use the mysql-connector-python client for a task:
```
[2021-03-17 10:20:13,247] {{taskinstance.py:1455}} ERROR - 'bool' object is not callable
Traceback (most recent call last):
  File "/home/airflow/.local/lib/python3.7/site-packages/airflow/models/taskinstance.py", line 1112, in _run_raw_task
    self._prepare_and_execute_task_with_callbacks(context, task)
  File "/home/airflow/.local/lib/python3.7/site-packages/airflow/models/taskinstance.py", line 1285, in _prepare_and_execute_task_with_callbacks
    result = self._execute_task(context, task_copy)
  File "/home/airflow/.local/lib/python3.7/site-packages/airflow/models/taskinstance.py", line 1310, in _execute_task
    result = task_copy.execute(context=context)
  File "/home/airflow/.local/lib/python3.7/site-packages/airflow/providers/mysql/operators/mysql.py", line 74, in execute
    hook.run(self.sql, autocommit=self.autocommit, parameters=self.parameters)
  File "/home/airflow/.local/lib/python3.7/site-packages/airflow/hooks/dbapi.py", line 175, in run
    self.set_autocommit(conn, autocommit)
  File "/home/airflow/.local/lib/python3.7/site-packages/airflow/providers/mysql/hooks/mysql.py", line 55, in set_autocommit
    conn.autocommit(autocommit)
```

**What you expected to happen**:

The task to run without complaints.

**How to reproduce it**:

Create and use a MySQL connection with `{"client": "mysql-connector-python"}` specified in the Extra field.

**Anything else we need to know**:

The MySQL hook seems to be using `conn.get_autocommit()` and `conn.autocommit()` to get/set the autocommit flag for both mysqlclient and mysql-connector-python. These method don't actually exist in mysql-connector-python as it uses autocommit as a property rather than a method.

I was able to work around it by adding an `if not callable(conn.autocommit)` condition to detect when mysql-connector-python is being used, but I'm sure there's probably a more elegant way of detecting which client is being used.

mysql-connector-python documentation:
https://dev.mysql.com/doc/connector-python/en/connector-python-api-mysqlconnection-autocommit.html

Autocommit calls:
https://github.com/apache/airflow/blob/2a2adb3f94cc165014d746102e12f9620f271391/airflow/providers/mysql/hooks/mysql.py#L55
https://github.com/apache/airflow/blob/2a2adb3f94cc165014d746102e12f9620f271391/airflow/providers/mysql/hooks/mysql.py#L66


Discussion:
Thanks for opening your first issue here! Be sure to follow the issue template!

You can know which is used by using `client_name`:
https://github.com/apache/airflow/blob/2a2adb3f94cc165014d746102e12f9620f271391/airflow/providers/mysql/hooks/mysql.py#L140


https://github.com/apache/airflow/blob/2a2adb3f94cc165014d746102e12f9620f271391/airflow/providers/mysql/hooks/mysql.py#L142

https://github.com/apache/airflow/blob/2a2adb3f94cc165014d746102e12f9620f271391/airflow/providers/mysql/hooks/mysql.py#L148

Will you submit a PR to fix the issue?

FYI @feluelle I believe you added the library?
@eladkal I did try that, but `conn` seems to point to the mysql-connector-python instance rather than the Airflow Connection instance in `set_autocommit` and `get_autocommit`

Resolving patch:
diff --git a/airflow/providers/mysql/hooks/mysql.py b/airflow/providers/mysql/hooks/mysql.py
--- a/airflow/providers/mysql/hooks/mysql.py
+++ b/airflow/providers/mysql/hooks/mysql.py
@@ -18,11 +18,17 @@
 
 """This module allows to connect to a MySQL database."""
 import json
-from typing import Dict, Optional, Tuple
+from typing import TYPE_CHECKING, Dict, Optional, Tuple, Union
 
 from airflow.hooks.dbapi import DbApiHook
 from airflow.models import Connection
 
+if TYPE_CHECKING:
+    from mysql.connector.abstracts import MySQLConnectionAbstract
+    from MySQLdb.connections import Connection as MySQLdbConnection
+
+MySQLConnectionTypes = Union['MySQLdbConnection', 'MySQLConnectionAbstract']
+
 
 class MySqlHook(DbApiHook):
     """
@@ -50,20 +56,36 @@ def __init__(self, *args, **kwargs) -> None:
         self.schema = kwargs.pop("schema", None)
         self.connection = kwargs.pop("connection", None)
 
-    def set_autocommit(self, conn: Connection, autocommit: bool) -> None:  # noqa: D403
-        """MySql connection sets autocommit in a different way."""
-        conn.autocommit(autocommit)
+    def set_autocommit(self, conn: MySQLConnectionTypes, autocommit: bool) -> None:
+        """
+        The MySQLdb (mysqlclient) client uses an `autocommit` method rather
+        than an `autocommit` property to set the autocommit setting
+
+        :param conn: connection to set autocommit setting
+        :type MySQLConnectionTypes: connection object.
+        :param autocommit: autocommit setting
+        :type bool: True to enable autocommit, False to disable autocommit
+        :rtype: None
+        """
+        if hasattr(conn.__class__, 'autocommit') and isinstance(conn.__class__.autocommit, property):
+            conn.autocommit = autocommit
+        else:
+            conn.autocommit(autocommit)
 
-    def get_autocommit(self, conn: Connection) -> bool:  # noqa: D403
+    def get_autocommit(self, conn: MySQLConnectionTypes) -> bool:
         """
-        MySql connection gets autocommit in a different way.
+        The MySQLdb (mysqlclient) client uses a `get_autocommit` method
+        rather than an `autocommit` property to get the autocommit setting
 
         :param conn: connection to get autocommit setting from.
-        :type conn: connection object.
+        :type MySQLConnectionTypes: connection object.
         :return: connection autocommit setting
         :rtype: bool
         """
-        return conn.get_autocommit()
+        if hasattr(conn.__class__, 'autocommit') and isinstance(conn.__class__.autocommit, property):
+            return conn.autocommit
+        else:
+            return conn.get_autocommit()
 
     def _get_conn_config_mysql_client(self, conn: Connection) -> Dict:
         conn_config = {
@@ -122,7 +144,7 @@ def _get_conn_config_mysql_connector_python(self, conn: Connection) -> Dict:
 
         return conn_config
 
-    def get_conn(self):
+    def get_conn(self) -> MySQLConnectionTypes:
         """
         Establishes a connection to a mysql database
         by extracting the connection configuration from the Airflow connection.


Test patch:
diff --git a/tests/providers/mysql/hooks/test_mysql.py b/tests/providers/mysql/hooks/test_mysql.py
--- a/tests/providers/mysql/hooks/test_mysql.py
+++ b/tests/providers/mysql/hooks/test_mysql.py
@@ -216,6 +216,21 @@ def test_get_conn_allow_local_infile(self, mock_connect):
         assert kwargs['allow_local_infile'] == 1
 
 
+class MockMySQLConnectorConnection:
+    DEFAULT_AUTOCOMMIT = 'default'
+
+    def __init__(self):
+        self._autocommit = self.DEFAULT_AUTOCOMMIT
+
+    @property
+    def autocommit(self):
+        return self._autocommit
+
+    @autocommit.setter
+    def autocommit(self, autocommit):
+        self._autocommit = autocommit
+
+
 class TestMySqlHook(unittest.TestCase):
     def setUp(self):
         super().setUp()
@@ -233,12 +248,25 @@ def get_conn(self):
 
         self.db_hook = SubMySqlHook()
 
-    def test_set_autocommit(self):
-        autocommit = True
-        self.db_hook.set_autocommit(self.conn, autocommit)
+    @parameterized.expand([(True,), (False,)])
+    def test_set_autocommit_mysql_connector(self, autocommit):
+        conn = MockMySQLConnectorConnection()
+        self.db_hook.set_autocommit(conn, autocommit)
+        assert conn.autocommit is autocommit
+
+    def test_get_autocommit_mysql_connector(self):
+        conn = MockMySQLConnectorConnection()
+        assert self.db_hook.get_autocommit(conn) == MockMySQLConnectorConnection.DEFAULT_AUTOCOMMIT
 
+    def test_set_autocommit_mysqldb(self):
+        autocommit = False
+        self.db_hook.set_autocommit(self.conn, autocommit)
         self.conn.autocommit.assert_called_once_with(autocommit)
 
+    def test_get_autocommit_mysqldb(self):
+        self.db_hook.get_autocommit(self.conn)
+        self.conn.get_autocommit.assert_called_once()
+
     def test_run_without_autocommit(self):
         sql = 'SQL'
         self.conn.get_autocommit.return_value = False