Repository: apache/airflow

Issue:
No more SQL Exception in 2.1.0
**Apache Airflow version**:

2.1.0

**Environment**:

- self hosted docker-compose based stack

**What happened**:

Using JDBCOperator if the sql result in an error we got only 
```
[2021-06-21 11:05:55,377] {local_task_job.py:151} INFO - Task exited with return code 1
```

Before upgrading from 2.0.1 we got error details in logs:
```
jaydebeapi.DatabaseError: java.sql.SQLException:... [MapR][DrillJDBCDriver](500165) Query execution error. Details: VALIDATION ERROR:...
``` 

**What you expected to happen**:

See `SQLException` in logs

**How to reproduce it**:

Perform a generic SQL task with broken SQL.

**Anything else we need to know**:

I think is somehow related to https://github.com/apache/airflow/commit/abcd48731303d9e141bdc94acc2db46d73ccbe12#diff-4fd3febb74d94b2953bf5e9b4a981b617949195f83d96f4a589c3078085959b7R202 


Discussion:
Thanks for opening your first issue here! Be sure to follow the issue template!

I am not sure - it does not look like it. @malthe maybe you could double-check it ?
@potiuk I can't see how it could affect that. It would be nice to see a more complete log of before vs after. That `SQLException` log trace doesn't seem to come from Python directly.
Yep. My though exactly.  @acarmisc  - can you provide some more logs/info ideally detailed extract from before and after where we can see where it came from ? I have a feeling you had different log configuration before, enabling the logs from the driver
Hi! 
My intention was not to point to precise code snippet... I only notice that there are some changes in part of the code involved in this kind of tasks.

Sounds a little strange to me a difference in configurations (logging setup) because the only change is the docker image that is build changing the dag of the base image :

```
1c1
< FROM apache/airflow:2.1.0
---
> FROM apache/airflow:2.0.1
```

The docker-compose file is the same.

Anyway the log is not very useful (actually is the problem :) ):

On 2.0.1
```
[2021-06-19 18:19:54,775] {dbapi.py:180} INFO - Running statement: 
[** OMITTED QUERY FOR PRIVACY **], parameters: None
[2021-06-19 18:19:55,314] {taskinstance.py:1455} ERROR - java.sql.SQLException: [MapR][DrillJDBCDriver](500165) Query execution error. Details: VALIDATION ERROR: From line 86, column 7 to line 86, column 9: Object 'foobar/barfoo' not found within 'db.name'
[Error Id: 9bf75eea-04d4-4791-b31c-3198f1c898ab on mapr-node23:31010].
com.mapr.drill.support.exceptions.com.mapr.drill.support.exceptions.GeneralException: com.mapr.drill.support.exceptions.GeneralException: [MapR][DrillJDBCDriver](500165) Query execution error. Details: VALIDATION ERROR: From line 86, column 7 to line 86, column 9: Object 'foobar/barfoo' not found within 'db.name'
[Error Id: 9bf75eea-04d4-4791-b31c-3198f1c898ab on mapr-node23:31010].
[2021-06-19 18:19:55,972] {local_task_job.py:146} INFO - Task exited with return code 1
```

On 2.1.0
```
[2021-06-21 09:15:46,818] {base_job.py:219} DEBUG - [heartbeat]
[2021-06-21 09:15:48,256] {dbapi.py:204} INFO - Running statement: 
[** OMITTED QUERY **], parameters: None
[2021-06-21 09:15:48,711] {cli_action_loggers.py:84} DEBUG - Calling callbacks: []
[2021-06-21 09:15:48,807] {local_task_job.py:151} INFO - Task exited with return code 1
```

Can you provide some dag code? If there's sensitive things, just change them to use some fake data.  It's still much better than debugging with only logs that no-one but you knows how to generate.
Is not a problem because is very simple... I can reproduce with whatever wrong SQL, like `SELECT foo from bar` where obviously `bar` table does not exist.

The dag perform a JDBCOperator

```
from utils.cdh_airflow_utils import *

from airflow import DAG

from airflow.operators.dummy_operator import DummyOperator
from airflow.providers.jdbc.operators.jdbc import JdbcOperator


"""
 GLOBAL VARS AND FUNCTIONS
"""
args = {...}

dag = DAG(dag_id="my_dag",
          description="...",
          max_active_runs=1,
          concurrency=2,
          start_date=datetime(2020, 9, 6),
          schedule_interval=sched_cron,
          catchup=False,
          tags=args['tags'],
          default_args=args)

connId = CONNECTION_ID_HIVE
scriptPath = SQL_SCRIPT_PATH + SUB_SQL_PATH

start = DummyOperator(task_id="Start", dag=dag)

refresh_TABLE_NAME = JdbcOperator(task_id="refresh_TABLE_NAME",
                                  sql=splitScript(scriptPath + 'query_from_file.sql'),
                                  jdbc_conn_id=connId,
                                  autocommit=True,
                                  dag=dag)

end = DummyOperator(
    task_id='End',
    dag=dag,
)

"""
RELATIONSHIPS
"""

start >> refresh_TABLE_NAME >> end

if __name__ == "__main__":
    dag.cli()
```
@acarmisc might be a change introduced here 258ec5d95e98eac09ecc7658dcd5226c9afe14c6.

An error is no longer logged as an exception and if you look at the changelog of the JDBC driver, then there's a note in there that Java SQL exceptions were added at some point to the exception message – but it wouldn't necessarily be printed after the commit linked above:

<img width="620" alt="image" src="https://user-images.githubusercontent.com/26405/122835864-aeacce80-d2e0-11eb-808b-047146d743e0.png">

thanks @malthe , but I'm not able to understand where I can found the exception message or whatever details about the failure. Usually we rely on logging to understand what goes wrong.
I think @malthe  correctly diagnosed it, I think we should simply update the https://github.com/apache/airflow/commit/258ec5d95e98eac09ecc7658dcd5226c9afe14c6 fix and convert the exception to string before logging. That would probably nicely handle the problem with serialization and give at least somewhat meaningful message for the users.
thank you all! Do you have an eta or an usual timing to share? 

Resolving patch:
diff --git a/airflow/models/taskinstance.py b/airflow/models/taskinstance.py
--- a/airflow/models/taskinstance.py
+++ b/airflow/models/taskinstance.py
@@ -1451,7 +1451,7 @@ def handle_failure(
 
         if error:
             if isinstance(error, Exception):
-                self.log.exception("Task failed with exception")
+                self.log.error("Task failed with exception", exc_info=error)
             else:
                 self.log.error("%s", error)
             # external monitoring process provides pickle file so _run_raw_task


Test patch: