Repository: apache/airflow

Issue:
Elasticsearch remote log will not fetch task logs from manual dagruns before 2.2 upgrade
### Apache Airflow Provider(s)

elasticsearch

### Versions of Apache Airflow Providers

```
apache-airflow-providers-amazon==1!2.5.0
apache-airflow-providers-cncf-kubernetes==1!2.1.0
apache-airflow-providers-datadog==1!2.0.1
apache-airflow-providers-elasticsearch==1!2.1.0
apache-airflow-providers-ftp==1!2.0.1
apache-airflow-providers-google==1!6.1.0
apache-airflow-providers-http==1!2.0.1
apache-airflow-providers-imap==1!2.0.1
apache-airflow-providers-microsoft-azure==1!3.3.0
apache-airflow-providers-mysql==1!2.1.1
apache-airflow-providers-postgres==1!2.3.0
apache-airflow-providers-redis==1!2.0.1
apache-airflow-providers-slack==1!4.1.0
apache-airflow-providers-sqlite==1!2.0.1
apache-airflow-providers-ssh==1!2.3.0
```

### Apache Airflow version

2.2.2

### Operating System

Debian Bullseye

### Deployment

Astronomer

### Deployment details

_No response_

### What happened

After upgrading to 2.2, task logs from manual dagruns performed before the upgrade could no longer be retrieved, even though they can still be seen in Kibana. Scheduled dagruns' tasks and tasks for dagruns begun after the upgrade are retrieved without issue.

The issue appears to be because these tasks with missing logs all belong to dagruns that do not have the attribute data_interval_start or data_interval_end set.

### What you expected to happen

Task logs continue to be fetched after upgrade.

### How to reproduce

Below is how I verified the log fetching process.

I ran the code snippet in a python interpreter in the scheduler to test log fetching.

```py
from airflow.models import TaskInstance, DagBag, DagRun
from airflow.settings import Session, DAGS_FOLDER
from airflow.configuration import conf
import logging
from dateutil import parser

logger = logging.getLogger('airflow.task')
task_log_reader = conf.get('logging', 'task_log_reader')
handler = next((handler for handler in logger.handlers if handler.name == task_log_reader), None)

dag_id = 'pipeline_nile_reconciliation'
task_id = 'nile_overcount_spend_resolution_task'
execution_date = parser.parse('2022-01-10T11:49:57.197933+00:00')
try_number=1

session = Session()
ti = session.query(TaskInstance).filter(
    TaskInstance.dag_id == dag_id,
    TaskInstance.task_id == task_id,
    TaskInstance.execution_date == execution_date).first()
dagrun = session.query(DagRun).filter(
    DagRun.dag_id == dag_id,
    DagRun.execution_date == execution_date).first()


dagbag = DagBag(DAGS_FOLDER, read_dags_from_db=True)
dag = dagbag.get_dag(dag_id)
ti.task = dag.get_task(ti.task_id)
ti.dagrun = dagrun

handler.read(ti, try_number, {})
```

The following error log indicates errors in the log reading.

```
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/usr/local/lib/python3.9/site-packages/airflow/utils/log/file_task_handler.py", line 239, in read
    log, metadata = self._read(task_instance, try_number_element, metadata)
  File "/usr/local/lib/python3.9/site-packages/airflow/providers/elasticsearch/log/es_task_handler.py", line 168, in _read
    log_id = self._render_log_id(ti, try_number)
  File "/usr/local/lib/python3.9/site-packages/airflow/providers/elasticsearch/log/es_task_handler.py", line 107, in _render_log_id
    data_interval_start = self._clean_date(dag_run.data_interval_start)
  File "/usr/local/lib/python3.9/site-packages/airflow/providers/elasticsearch/log/es_task_handler.py", line 134, in _clean_date
    return value.strftime("%Y_%m_%dT%H_%M_%S_%f")
AttributeError: 'NoneType' object has no attribute 'strftime'
```


### Anything else

_No response_

### Are you willing to submit PR?

- [ ] Yes I am willing to submit a PR!

### Code of Conduct

- [X] I agree to follow this project's [Code of Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md)



Discussion:
All credit to @wolfier for finding this bug.
Seems like the problem in the code is that [this section](https://github.com/apache/airflow/blob/main/airflow/providers/elasticsearch/log/es_task_handler.py#L109-L114) requires resolving the `data_interval_start` and `data_interval_end` even if the format you specify does not use them, and they don't exist on manual dagruns from before 2.2
It should use `dag.get_run_data_interval(dag_run)` instead. How to get the correct DAG object, however, is the problem; `dag_run.get_dag()` *might* work, but if it doesn’t, we need to do something else, perhaps `ti.task.dag` or something else.
Also whoever works on this should also fix the rendering function in `FileTaskHandler` to add those same data-interval-related context variables.

Resolving patch:
diff --git a/airflow/providers/elasticsearch/log/es_task_handler.py b/airflow/providers/elasticsearch/log/es_task_handler.py
--- a/airflow/providers/elasticsearch/log/es_task_handler.py
+++ b/airflow/providers/elasticsearch/log/es_task_handler.py
@@ -103,15 +103,25 @@ def __init__(
         self.handler: Union[logging.FileHandler, logging.StreamHandler]  # type: ignore[assignment]
 
     def _render_log_id(self, ti: TaskInstance, try_number: int) -> str:
-        dag_run = ti.dag_run
+        dag_run = ti.get_dagrun()
+        try:
+            data_interval: Tuple[datetime, datetime] = ti.task.dag.get_run_data_interval(dag_run)
+        except AttributeError:  # ti.task is not always set.
+            data_interval = (dag_run.data_interval_start, dag_run.data_interval_end)
 
         if self.json_format:
-            data_interval_start = self._clean_date(dag_run.data_interval_start)
-            data_interval_end = self._clean_date(dag_run.data_interval_end)
+            data_interval_start = self._clean_date(data_interval[0])
+            data_interval_end = self._clean_date(data_interval[1])
             execution_date = self._clean_date(dag_run.execution_date)
         else:
-            data_interval_start = dag_run.data_interval_start.isoformat()
-            data_interval_end = dag_run.data_interval_end.isoformat()
+            if data_interval[0]:
+                data_interval_start = data_interval[0].isoformat()
+            else:
+                data_interval_start = ""
+            if data_interval[1]:
+                data_interval_end = data_interval[1].isoformat()
+            else:
+                data_interval_end = ""
             execution_date = dag_run.execution_date.isoformat()
 
         return self.log_id_template.format(
@@ -125,14 +135,15 @@ def _render_log_id(self, ti: TaskInstance, try_number: int) -> str:
         )
 
     @staticmethod
-    def _clean_date(value: datetime) -> str:
+    def _clean_date(value: Optional[datetime]) -> str:
         """
         Clean up a date value so that it is safe to query in elasticsearch
         by removing reserved characters.
-        # https://www.elastic.co/guide/en/elasticsearch/reference/current/query-dsl-query-string-query.html#_reserved_characters
 
-        :param execution_date: execution date of the dag run.
+        https://www.elastic.co/guide/en/elasticsearch/reference/current/query-dsl-query-string-query.html#_reserved_characters
         """
+        if value is None:
+            return ""
         return value.strftime("%Y_%m_%dT%H_%M_%S_%f")
 
     def _group_logs_by_host(self, logs):
diff --git a/airflow/utils/log/file_task_handler.py b/airflow/utils/log/file_task_handler.py
--- a/airflow/utils/log/file_task_handler.py
+++ b/airflow/utils/log/file_task_handler.py
@@ -18,8 +18,9 @@
 """File logging handler for tasks."""
 import logging
 import os
+from datetime import datetime
 from pathlib import Path
-from typing import TYPE_CHECKING, Optional
+from typing import TYPE_CHECKING, Optional, Tuple
 
 import httpx
 from itsdangerous import TimedJSONWebSignatureSerializer
@@ -83,9 +84,25 @@ def _render_filename(self, ti: "TaskInstance", try_number: int) -> str:
             context["try_number"] = try_number
             return render_template_to_string(self.filename_jinja_template, context)
         elif self.filename_template:
+            dag_run = ti.get_dagrun()
+            try:
+                data_interval: Tuple[datetime, datetime] = ti.task.dag.get_run_data_interval(dag_run)
+            except AttributeError:  # ti.task is not always set.
+                data_interval = (dag_run.data_interval_start, dag_run.data_interval_end)
+            if data_interval[0]:
+                data_interval_start = data_interval[0].isoformat()
+            else:
+                data_interval_start = ""
+            if data_interval[1]:
+                data_interval_end = data_interval[1].isoformat()
+            else:
+                data_interval_end = ""
             return self.filename_template.format(
                 dag_id=ti.dag_id,
                 task_id=ti.task_id,
+                run_id=ti.run_id,
+                data_interval_start=data_interval_start,
+                data_interval_end=data_interval_end,
                 execution_date=ti.get_dagrun().logical_date.isoformat(),
                 try_number=try_number,
             )


Test patch: