Repository: apache/airflow

Issue:
DruidOperator failing to submit ingestion tasks : Getting 500 error code from Druid
Issue : When trying to submit an ingestion task using DruidOperator, getting 500 error code in response from Druid. And can see no task submitted in Druid console.
In Airflow 1.10.x, everything is working fine. But when upgraded to 2.0.1, it is failing to submit the task. There is absolutely no change in the code/files except the import statements.

Resolution : I compared DruidOperator code for both Airflow 1.10.x & 2.0.1 and found one line causing the issue.
In Airflow 2.0.x, before submitting the indexing job json string is converted to python object. But it should be json string only.
In Airflow 1.10.x there is no conversion happening and hence it is working fine. (Please see below code snippets.)

I have already tried this change in my setup and re-ran the ingestion tasks. It is all working fine.

~~hook.submit_indexing_job(json.loads(self.json_index_file))~~
**hook.submit_indexing_job(self.json_index_file)**


Airflow 1.10.x - airflow/contrib/operators/druid_operator.py
```
def execute(self, context):
    hook = DruidHook(
        druid_ingest_conn_id=self.conn_id,
        max_ingestion_time=self.max_ingestion_time
    )
    self.log.info("Submitting %s", self.index_spec_str)
    hook.submit_indexing_job(self.index_spec_str)
```

Airflow 2.0.1 - airflow/providers/apache/druid/operators/druid.py
```
def execute(self, context: Dict[Any, Any]) -> None:
    hook = DruidHook(druid_ingest_conn_id=self.conn_id, max_ingestion_time=self.max_ingestion_time)
    self.log.info("Submitting %s", self.json_index_file)
    hook.submit_indexing_job(json.loads(self.json_index_file))
```


**Apache Airflow version**: 2.0.x

**Error Logs**:

```
[2021-02-24 06:42:24,287] {{connectionpool.py:452}} DEBUG - http://druid-master:8081 "POST /druid/indexer/v1/task HTTP/1.1" 500 15714
[2021-02-24 06:42:24,287] {{taskinstance.py:570}} DEBUG - Refreshing TaskInstance <TaskInstance: druid_compact_daily 2021-02-23T01:20:00+00:00 [running]> from DB
[2021-02-24 06:42:24,296] {{taskinstance.py:605}} DEBUG - Refreshed TaskInstance <TaskInstance: druid_compact_daily 2021-02-23T01:20:00+00:00 [running]>
[2021-02-24 06:42:24,298] {{taskinstance.py:1455}} ERROR - Did not get 200 when submitting the Druid job to http://druid-master:8081/druid/indexer/v1/task
```



Discussion:
Thanks for opening your first issue here! Be sure to follow the issue template!


Resolving patch:
diff --git a/airflow/providers/apache/druid/operators/druid.py b/airflow/providers/apache/druid/operators/druid.py
--- a/airflow/providers/apache/druid/operators/druid.py
+++ b/airflow/providers/apache/druid/operators/druid.py
@@ -16,7 +16,6 @@
 # specific language governing permissions and limitations
 # under the License.
 
-import json
 from typing import Any, Dict, Optional
 
 from airflow.models import BaseOperator
@@ -55,4 +54,4 @@ def __init__(
     def execute(self, context: Dict[Any, Any]) -> None:
         hook = DruidHook(druid_ingest_conn_id=self.conn_id, max_ingestion_time=self.max_ingestion_time)
         self.log.info("Submitting %s", self.json_index_file)
-        hook.submit_indexing_job(json.loads(self.json_index_file))
+        hook.submit_indexing_job(self.json_index_file)


Test patch:
diff --git a/tests/providers/apache/druid/operators/test_druid.py b/tests/providers/apache/druid/operators/test_druid.py
--- a/tests/providers/apache/druid/operators/test_druid.py
+++ b/tests/providers/apache/druid/operators/test_druid.py
@@ -17,7 +17,9 @@
 # under the License.
 #
 
+import os
 import unittest
+from tempfile import NamedTemporaryFile
 
 from airflow.models import TaskInstance
 from airflow.models.dag import DAG
@@ -31,9 +33,7 @@ class TestDruidOperator(unittest.TestCase):
     def setUp(self):
         args = {'owner': 'airflow', 'start_date': timezone.datetime(2017, 1, 1)}
         self.dag = DAG('test_dag_id', default_args=args)
-
-    def test_render_template(self):
-        json_str = '''
+        self.json_index_str = '''
             {
                 "type": "{{ params.index_type }}",
                 "datasource": "{{ params.datasource }}",
@@ -46,15 +46,7 @@ def test_render_template(self):
                 }
             }
         '''
-        operator = DruidOperator(
-            task_id='spark_submit_job',
-            json_index_file=json_str,
-            params={'index_type': 'index_hadoop', 'datasource': 'datasource_prd'},
-            dag=self.dag,
-        )
-        ti = TaskInstance(operator, DEFAULT_DATE)
-        ti.render_templates()
-        expected = '''
+        self.rendered_index_str = '''
             {
                 "type": "index_hadoop",
                 "datasource": "datasource_prd",
@@ -67,4 +59,33 @@ def test_render_template(self):
                 }
             }
         '''
-        assert expected == getattr(operator, 'json_index_file')
+
+    def test_render_template(self):
+        operator = DruidOperator(
+            task_id='spark_submit_job',
+            json_index_file=self.json_index_str,
+            params={'index_type': 'index_hadoop', 'datasource': 'datasource_prd'},
+            dag=self.dag,
+        )
+        ti = TaskInstance(operator, DEFAULT_DATE)
+        ti.render_templates()
+
+        assert self.rendered_index_str == operator.json_index_file
+
+    def test_render_template_from_file(self):
+        with NamedTemporaryFile("w", suffix='.json') as f:
+            f.write(self.json_index_str)
+            f.flush()
+
+            self.dag.template_searchpath = os.path.dirname(f.name)
+
+            operator = DruidOperator(
+                task_id='spark_submit_job',
+                json_index_file=f.name,
+                params={'index_type': 'index_hadoop', 'datasource': 'datasource_prd'},
+                dag=self.dag,
+            )
+            ti = TaskInstance(operator, DEFAULT_DATE)
+            ti.render_templates()
+
+            assert self.rendered_index_str == operator.json_index_file