Repository: apache/airflow

Issue:
Elasticsearch Backport Provider Incompatible with Airflow 1.10.12
**Apache Airflow version**: 1.10.12


**Kubernetes version (if you are using kubernetes)** (use `kubectl version`): 1.16.9

**Environment**: 

- **Cloud provider or hardware configuration**: AWS
- **OS** (e.g. from /etc/os-release): 
- **Kernel** (e.g. `uname -a`): 
- **Install tools**: Docker image running in k8s Pods
- **Others**: Rancher-provisioned k8s clusters

**What happened**:

After configuring the latest version of the Elasticsearch backport provider as my log handler via `config/airflow_local_settings.py` resulted in an error on the webserver when trying to read logs from Elasticsearch

```
[2020-10-12 21:02:00,487] {app.py:1892} ERROR - Exception on /get_logs_with_metadata [GET]
Traceback (most recent call last):
  File "/usr/local/lib/python3.7/site-packages/flask/app.py", line 2447, in wsgi_app
    response = self.full_dispatch_request()
  File "/usr/local/lib/python3.7/site-packages/flask/app.py", line 1952, in full_dispatch_request
    rv = self.handle_user_exception(e)
  File "/usr/local/lib/python3.7/site-packages/flask/app.py", line 1821, in handle_user_exception
    reraise(exc_type, exc_value, tb)
  File "/usr/local/lib/python3.7/site-packages/flask/_compat.py", line 39, in reraise
    raise value
  File "/usr/local/lib/python3.7/site-packages/flask/app.py", line 1950, in full_dispatch_request
    rv = self.dispatch_request()
  File "/usr/local/lib/python3.7/site-packages/flask/app.py", line 1936, in dispatch_request
    return self.view_functions[rule.endpoint](**req.view_args)
  File "/usr/local/lib/python3.7/site-packages/airflow/www_rbac/decorators.py", line 121, in wrapper
    return f(self, *args, **kwargs)
  File "/usr/local/lib/python3.7/site-packages/flask_appbuilder/security/decorators.py", line 109, in wraps
    return f(self, *args, **kwargs)
  File "/usr/local/lib/python3.7/site-packages/airflow/www_rbac/decorators.py", line 56, in wrapper
    return f(*args, **kwargs)
  File "/usr/local/lib/python3.7/site-packages/airflow/utils/db.py", line 74, in wrapper
    return func(*args, **kwargs)
  File "/usr/local/lib/python3.7/site-packages/airflow/www_rbac/views.py", line 733, in get_logs_with_metadata
    logs, metadata = _get_logs_with_metadata(try_number, metadata)
  File "/usr/local/lib/python3.7/site-packages/airflow/www_rbac/views.py", line 724, in _get_logs_with_metadata
    logs, metadatas = handler.read(ti, try_number, metadata=metadata)
  File "/usr/local/lib/python3.7/site-packages/airflow/utils/log/file_task_handler.py", line 194, in read
    logs[i] += log
TypeError: can only concatenate str (not "list") to str
```

Here is the relevant section of my customized `airflow_local_settings.py` file with the updated Elasticsearch handler from the backport provider:
```
...
    elif ELASTICSEARCH_HOST:
        ELASTICSEARCH_LOG_ID_TEMPLATE: str = conf.get('elasticsearch', 'LOG_ID_TEMPLATE')
        ELASTICSEARCH_END_OF_LOG_MARK: str = conf.get('elasticsearch', 'END_OF_LOG_MARK')
        ELASTICSEARCH_FRONTEND: str = conf.get('elasticsearch', 'frontend')
        ELASTICSEARCH_WRITE_STDOUT: bool = conf.getboolean('elasticsearch', 'WRITE_STDOUT')
        ELASTICSEARCH_JSON_FORMAT: bool = conf.getboolean('elasticsearch', 'JSON_FORMAT')
        ELASTICSEARCH_JSON_FIELDS: str = conf.get('elasticsearch', 'JSON_FIELDS')

        ELASTIC_REMOTE_HANDLERS: Dict[str, Dict[str, Union[str, bool]]] = {
            'task': {
                'class': 'airflow.providers.elasticsearch.log.es_task_handler.ElasticsearchTaskHandler',
                'formatter': 'airflow',
                'base_log_folder': str(os.path.expanduser(BASE_LOG_FOLDER)),
                'log_id_template': ELASTICSEARCH_LOG_ID_TEMPLATE,
                'filename_template': FILENAME_TEMPLATE,
                'end_of_log_mark': ELASTICSEARCH_END_OF_LOG_MARK,
                'host': ELASTICSEARCH_HOST,
                'frontend': ELASTICSEARCH_FRONTEND,
                'write_stdout': ELASTICSEARCH_WRITE_STDOUT,
                'json_format': ELASTICSEARCH_JSON_FORMAT,
                'json_fields': ELASTICSEARCH_JSON_FIELDS
            },
        }

        LOGGING_CONFIG['handlers'].update(ELASTIC_REMOTE_HANDLERS)
...
```

**What you expected to happen**:

Airflow's web UI properly displays the logs from Elasticsearch

**How to reproduce it**:
Configure custom logging via `config/airflow_local_settings.py` to `airflow.providers.elasticsearch.log.es_task_handler.ElasticsearchTaskHandler` and set the `logging_config_class` in `airflow.cfg`

When a task has been run, try to view its logs in the web UI and check the webserver logs to see the error above



Discussion:
This looks like a real problem. Should I assign you to this issue? Do you want to deal with it?
I can take a stab - I think we just need to backport some of the type handling from `airflow/utils/log/file_task_handler.py` on `master` to the 1.10 branch
@marcusianlevine  What do you think to vendorize the `file_task_handler` module to `elasticsearch` package? See: https://github.com/apache/airflow/blob/master/provider_packages/refactor_provider_packages.py#L427-L436
This will allow us to fix the problem without having to make any changes to Airflow core.
Sure, I think that will work, I'll open a PR later today

Resolving patch:
diff --git a/provider_packages/refactor_provider_packages.py b/provider_packages/refactor_provider_packages.py
--- a/provider_packages/refactor_provider_packages.py
+++ b/provider_packages/refactor_provider_packages.py
@@ -432,6 +432,70 @@ def amazon_package_filter(node: LN, capture: Capture, filename: Filename) -> boo
             .rename("airflow.providers.amazon.common.utils.email")
         )
 
+    def refactor_elasticsearch_package(self):
+        """
+        Fixes to "elasticsearch" providers package.
+
+        Copies some of the classes used from core Airflow to "common.utils" package of
+        the provider and renames imports to use them from there.
+
+        We copy file_task_handler.py and change import as in example diff:
+
+        .. code-block:: diff
+
+            --- ./airflow/providers/elasticsearch/log/es_task_handler.py
+            +++ ./airflow/providers/elasticsearch/log/es_task_handler.py
+            @@ -24,7 +24,7 @@
+            from airflow.configuration import conf
+            from airflow.models import TaskInstance
+            from airflow.utils import timezone
+            from airflow.utils.helpers import parse_template_string
+            -from airflow.utils.log.file_task_handler import FileTaskHandler
+            +from airflow.providers.elasticsearch.common.utils.log.file_task_handler import FileTaskHandler
+            from airflow.utils.log.json_formatter import JSONFormatter
+            from airflow.utils.log.logging_mixin import LoggingMixin
+
+        """
+
+        def elasticsearch_package_filter(node: LN, capture: Capture, filename: Filename) -> bool:
+            return filename.startswith("./airflow/providers/elasticsearch/")
+
+        os.makedirs(
+            os.path.join(get_target_providers_package_folder("elasticsearch"), "common", "utils", "log"),
+            exist_ok=True,
+        )
+        copyfile(
+            os.path.join(get_source_airflow_folder(), "airflow", "utils", "__init__.py"),
+            os.path.join(get_target_providers_package_folder("elasticsearch"), "common", "__init__.py"),
+        )
+        copyfile(
+            os.path.join(get_source_airflow_folder(), "airflow", "utils", "__init__.py"),
+            os.path.join(
+                get_target_providers_package_folder("elasticsearch"), "common", "utils", "__init__.py"
+            ),
+        )
+        copyfile(
+            os.path.join(get_source_airflow_folder(), "airflow", "utils", "log", "__init__.py"),
+            os.path.join(
+                get_target_providers_package_folder("elasticsearch"), "common", "utils", "log", "__init__.py"
+            ),
+        )
+        copyfile(
+            os.path.join(get_source_airflow_folder(), "airflow", "utils", "log", "file_task_handler.py"),
+            os.path.join(
+                get_target_providers_package_folder("elasticsearch"),
+                "common",
+                "utils",
+                "log",
+                "file_task_handler.py",
+            ),
+        )
+        (
+            self.qry.select_module("airflow.utils.log.file_task_handler")
+            .filter(callback=elasticsearch_package_filter)
+            .rename("airflow.providers.elasticsearch.common.utils.log.file_task_handler")
+        )
+
     def refactor_google_package(self):
         r"""
         Fixes to "google" providers package.
@@ -651,6 +715,7 @@ def kubernetes_package_filter(node: LN, capture: Capture, filename: Filename) ->
     def do_refactor(self, in_process: bool = False) -> None:  # noqa
         self.rename_deprecated_modules()
         self.refactor_amazon_package()
+        self.refactor_elasticsearch_package()
         self.refactor_google_package()
         self.refactor_odbc_package()
         self.remove_tags()


Test patch: