Repository: apache/airflow

Issue:
JdbcOperator should pass handler parameter to JdbcHook.run
### Apache Airflow version

2.2.0 (latest released)

### Operating System

Debian GNU/Linux 11 (bullseye)

### Versions of Apache Airflow Providers

apache-airflow-providers-jdbc==1!2.0.1

### Deployment

Astronomer

### Deployment details

Astro CLI Version: 0.25.4, Git Commit: e95011fbf800fda9fdef1dc1d5149d62bc017aed

### What happened

The execute method calls the [hook run method](https://github.com/apache/airflow/blob/20847fdbf8ecd3be394d24d47ce151c26d018ea1/airflow/providers/jdbc/operators/jdbc.py#L70) without passing any optional `handler` parameter to the DbApiHook parent class of JdbcHook.

### What you expected to happen

Without the handler, the results of the DbApiHook, and the JdbcOperator, [will always be an empty list](https://github.com/apache/airflow/blob/20847fdbf8ecd3be394d24d47ce151c26d018ea1/airflow/hooks/dbapi.py#L206). In the case of the JdbcOperator, the underlying `JayDeBeApi` connection uses a handler of the form `lambda x: x.fetchall()` to return results.

### How to reproduce

1. Using the astro cli
2. Download [postgresql-42.3.0.jar](https://jdbc.postgresql.org/download/postgresql-42.3.0.jar) into the working directory
3. Download [zulu11.52.13-ca-jdk11.0.13-linux_x64.tar.gz](https://cdn.azul.com/zulu/bin/zulu11.52.13-ca-jdk11.0.13-linux_x64.tar.gz) to the working directory
4. Copy the following to the Dockerfile
```
FROM quay.io/astronomer/ap-airflow:2.2.0-buster-onbuild

COPY postgresql-42.3.0.jar /usr/local/airflow/.

USER root
RUN cd /opt && mkdir java
COPY zulu11.52.13-ca-jdk11.0.13-linux_x64.tar.gz /opt/java
RUN cd /opt/java && pwd && ls && tar xfvz ./zulu11.52.13-ca-jdk11.0.13-linux_x64.tar.gz

ENV JAVA_HOME /opt/java/zulu11.52.13-ca-jdk11.0.13-linux_x64
RUN export JAVA_HOME
```
5.  Copy the following into the `airflow_settings.yaml` file
```yml
airflow:
  connections:
    - conn_id: my_jdbc_connection
      conn_type: jdbc
      conn_host: "jdbc:postgresql://postgres:5432/"
      conn_schema:
      conn_login: postgres
      conn_password: postgres
      conn_port: 
      conn_extra: '{"extra__jdbc__drv_clsname":"org.postgresql.Driver", "extra__jdbc__drv_path":"/usr/local/airflow/postgresql-42.3.0.jar"}'
```
6. Copy the following DAG and run it. No data will be passed to XCOM
```python
from datetime import datetime, timedelta

from airflow import DAG
from airflow.operators.dummy import DummyOperator
from airflow.providers.jdbc.operators.jdbc import JdbcOperator

with DAG(
    dag_id='example_jdbc_operator',
    schedule_interval=None,
    start_date=datetime(2021, 1, 1),
    dagrun_timeout=timedelta(minutes=60),
    tags=['example'],
    catchup=False,
) as dag:

    run_this_last = DummyOperator(task_id='run_this_last')

    query_dags_data = JdbcOperator(
        task_id='query_dags_data',
        sql='SELECT dag_id, is_active FROM dag',
        jdbc_conn_id='my_jdbc_connection',
        autocommit=True,
    )

    query_dags_data >> run_this_last 
```
7. Copy the following DAG and run it. All DAGs and their active status will be put onto XCOM
```python
from datetime import datetime, timedelta

from airflow import DAG
from airflow.operators.dummy import DummyOperator
from airflow.providers.jdbc.operators.jdbc import JdbcOperator
from airflow.providers.jdbc.hooks.jdbc import JdbcHook


class JdbcHandlerOperator(JdbcOperator):
    def execute(self, context) -> None:
        self.log.info('Executing: %s', self.sql)
        hook = JdbcHook(jdbc_conn_id=self.jdbc_conn_id)
        return hook.run(
            self.sql,
            self.autocommit,
            parameters=self.parameters,
            # Defined by how JayDeBeApi operates 
            handler=lambda x: x.fetchall()
        )


with DAG(
    dag_id='example_jdbc_operator',
    schedule_interval=None,
    start_date=datetime(2021, 1, 1),
    dagrun_timeout=timedelta(minutes=60),
    tags=['example'],
    catchup=False,
) as dag:

    run_this_last = DummyOperator(task_id='run_this_last')

    query_dags_data = JdbcHandlerOperator(
        task_id='query_dags_data',
        sql='SELECT dag_id, is_active FROM dag',
        jdbc_conn_id='my_jdbc_connection',
        autocommit=True,
    )

    query_dags_data >> run_this_last 
```

### Anything else

_No response_

### Are you willing to submit PR?

- [X] Yes I am willing to submit a PR!

### Code of Conduct

- [X] I agree to follow this project's [Code of Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md)



Discussion:
@cdabella assigned the issue to you, feel free to open PR
@cdabella are you still working on this issue?
Hi @eladkal,

If @cdabella will not take this, I would like to give a try. 
Bit of life got in the way and I have not implemented the fix. Please feel free @nkyuray
Hi @cdabella,
Thanks. 

Hi @eladkal I am going to work on this at the weekend just wanted to let you know.


Hi @nkyuray please have a look #23817

Resolving patch:
diff --git a/airflow/providers/jdbc/operators/jdbc.py b/airflow/providers/jdbc/operators/jdbc.py
--- a/airflow/providers/jdbc/operators/jdbc.py
+++ b/airflow/providers/jdbc/operators/jdbc.py
@@ -16,7 +16,7 @@
 # specific language governing permissions and limitations
 # under the License.
 
-from typing import TYPE_CHECKING, Iterable, Mapping, Optional, Sequence, Union
+from typing import TYPE_CHECKING, Any, Callable, Iterable, Mapping, Optional, Sequence, Union
 
 from airflow.models import BaseOperator
 from airflow.providers.common.sql.hooks.sql import fetch_all_handler
@@ -57,6 +57,7 @@ def __init__(
         jdbc_conn_id: str = 'jdbc_default',
         autocommit: bool = False,
         parameters: Optional[Union[Iterable, Mapping]] = None,
+        handler: Callable[[Any], Any] = fetch_all_handler,
         **kwargs,
     ) -> None:
         super().__init__(**kwargs)
@@ -64,9 +65,13 @@ def __init__(
         self.sql = sql
         self.jdbc_conn_id = jdbc_conn_id
         self.autocommit = autocommit
+        self.handler = handler
         self.hook = None
 
     def execute(self, context: 'Context'):
         self.log.info('Executing: %s', self.sql)
         hook = JdbcHook(jdbc_conn_id=self.jdbc_conn_id)
-        return hook.run(self.sql, self.autocommit, parameters=self.parameters, handler=fetch_all_handler)
+        if self.do_xcom_push:
+            return hook.run(self.sql, self.autocommit, parameters=self.parameters, handler=self.handler)
+        else:
+            return hook.run(self.sql, self.autocommit, parameters=self.parameters)


Test patch:
diff --git a/tests/providers/jdbc/operators/test_jdbc.py b/tests/providers/jdbc/operators/test_jdbc.py
--- a/tests/providers/jdbc/operators/test_jdbc.py
+++ b/tests/providers/jdbc/operators/test_jdbc.py
@@ -28,8 +28,8 @@ def setUp(self):
         self.kwargs = dict(sql='sql', task_id='test_jdbc_operator', dag=None)
 
     @patch('airflow.providers.jdbc.operators.jdbc.JdbcHook')
-    def test_execute(self, mock_jdbc_hook):
-        jdbc_operator = JdbcOperator(**self.kwargs)
+    def test_execute_do_push(self, mock_jdbc_hook):
+        jdbc_operator = JdbcOperator(**self.kwargs, do_xcom_push=True)
         jdbc_operator.execute(context={})
 
         mock_jdbc_hook.assert_called_once_with(jdbc_conn_id=jdbc_operator.jdbc_conn_id)
@@ -39,3 +39,15 @@ def test_execute(self, mock_jdbc_hook):
             parameters=jdbc_operator.parameters,
             handler=fetch_all_handler,
         )
+
+    @patch('airflow.providers.jdbc.operators.jdbc.JdbcHook')
+    def test_execute_dont_push(self, mock_jdbc_hook):
+        jdbc_operator = JdbcOperator(**self.kwargs, do_xcom_push=False)
+        jdbc_operator.execute(context={})
+
+        mock_jdbc_hook.assert_called_once_with(jdbc_conn_id=jdbc_operator.jdbc_conn_id)
+        mock_jdbc_hook.return_value.run.assert_called_once_with(
+            jdbc_operator.sql,
+            jdbc_operator.autocommit,
+            parameters=jdbc_operator.parameters,
+        )