Repository: apache/airflow

Issue:
Webserver shows wrong datetime (timezone) in log
### Apache Airflow version

2.3.0 (latest released)

### What happened

same as #19401 , when I open task`s log in web interface, it shifts this time forward by 8 hours (for Asia/Shanghai), but it's already in Asia/Shanghai.
here is the log in web:
```
*** Reading local file: /opt/airflow/logs/forecast/cal/2022-05-18T09:50:00+00:00/1.log
[2022-05-19, 13:54:52] {taskinstance.py:1037} INFO ...
```

As you seee, UTC time is 2022-05-18T09:50:00，and My timezone is Asia/Shanghai(should shift forward 8 hours)，but it shift forward 16hours!

### What you think should happen instead

_No response_

### How to reproduce

_No response_

### Operating System

Debian GNU/Linux 11 (bullseye)(docker)

### Versions of Apache Airflow Providers

_No response_

### Deployment

Docker-Compose

### Deployment details

1. build my docker image from apache/airflow:2.3.0 to change timezone
```Dockerfile
FROM apache/airflow:2.3.0
# bugfix of log UI in web, here I change ti_log.js file by following on #19401 
COPY ./ti_log.js /home/airflow/.local/lib/python3.7/site-packages/airflow/www/static/js/ti_log.js
USER root
# container timezone changed to CST time
RUN ln -sf /usr/share/zoneinfo/Asia/Shanghai /etc/localtime \
  && rm -rf /etc/timezone \
  && echo Asia/Shanghai >> /etc/timezone \
  && chown airflow /home/airflow/.local/lib/python3.7/site-packages/airflow/www/static/js/ti_log.js
USER airflow
```
2. use my image to run airflow by docker-compose

3. check task logs in web
Although I have changed the file `airflow/www/static/js/ti_log.js`, but it did not work! Then check source from Web, I found another file : `airflow/www/static/dist/tiLog.e915520196109d459cf8.js`, then I replace "+00:00" by  "+08:00" in this file. Finally it works!

```js
# origin tiLog.e915520196109d459cf8.js
replaceAll(c,(e=>`<time datetime="${e}+00:00" data-with-tz="true">${Object(a.f)(`${e}+00:00`)}</time>`))
```

```js
# what I changed
replaceAll(c,(e=>`<time datetime="${e}+08:00" data-with-tz="true">${Object(a.f)(`${e}+08:00`)}</time>`))
```





### Anything else

_No response_

### Are you willing to submit PR?

- [ ] Yes I am willing to submit a PR!

### Code of Conduct

- [X] I agree to follow this project's [Code of Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md)



Discussion:
Thanks for opening your first issue here! Be sure to follow the issue template!

Just to document the history:

This was initially fixed in #19401 and then reverted in #23075

For some users the logs are in the server time and for others the logs are in UTC.
Should we have a separate configuration option so we wont have to "fix" it back and forth anymore?


Also a side note - I think #19401 was a partial fix as it does not take in account that the offset for a timezone can change.
It sounds like a configuration is the only way to satisfy all use cases. There’s no way Airflow can automatically know what timezone the log files use otherwise.
I found workaround. via configuration. here's how: 

1. write `${PYTHONPATH}/airflow_local_settings.py `
```python
import logging
import time
from copy import deepcopy

from airflow.config_templates.airflow_local_settings import DEFAULT_LOGGING_CONFIG
LOGGING_CONFIG = deepcopy(DEFAULT_LOGGING_CONFIG)

# from https://stackoverflow.com/a/60342048
class UTCFormatter(logging.Formatter):
    converter = time.gmtime  # this makes logger's asctime as UTC

LOGGING_CONFIG['formatters']['airflow']['class'] = 'airflow_local_settings.UTCFormatter'
```
2. Edit airflow.cfg

```ini
[logging]
logging_config_class = airflow_local_settings.LOGGING_CONFIG
```

> working as same as https://github.com/apache/airflow/issues/20315

3. then 

![172614806-00307250-aab6-48e4-839f-0a23e0ece84b_png__1865×820_](https://user-images.githubusercontent.com/1422445/172617268-d5c3e3e6-ee60-43db-a3fe-2c2f97359db7.png)

----

I have tried something like `[2022-06-08 10:45:01+09:00]` by appending `%z` to datefmt on logging.Filter. but regex in airflow/www/static/js/ti_log.js couldn't understand it.

 

Resolving patch:
diff --git a/airflow/config_templates/airflow_local_settings.py b/airflow/config_templates/airflow_local_settings.py
--- a/airflow/config_templates/airflow_local_settings.py
+++ b/airflow/config_templates/airflow_local_settings.py
@@ -31,13 +31,16 @@
 # settings.py and cli.py. Please see AIRFLOW-1455.
 LOG_LEVEL: str = conf.get_mandatory_value('logging', 'LOGGING_LEVEL').upper()
 
-
 # Flask appbuilder's info level log is very verbose,
 # so it's set to 'WARN' by default.
 FAB_LOG_LEVEL: str = conf.get_mandatory_value('logging', 'FAB_LOGGING_LEVEL').upper()
 
 LOG_FORMAT: str = conf.get_mandatory_value('logging', 'LOG_FORMAT')
 
+LOG_FORMATTER_CLASS: str = conf.get_mandatory_value(
+    'logging', 'LOG_FORMATTER_CLASS', fallback='airflow.utils.log.timezone_aware.TimezoneAware'
+)
+
 COLORED_LOG_FORMAT: str = conf.get_mandatory_value('logging', 'COLORED_LOG_FORMAT')
 
 COLORED_LOG: bool = conf.getboolean('logging', 'COLORED_CONSOLE_LOG')
@@ -60,10 +63,13 @@
     'version': 1,
     'disable_existing_loggers': False,
     'formatters': {
-        'airflow': {'format': LOG_FORMAT},
+        'airflow': {
+            'format': LOG_FORMAT,
+            'class': LOG_FORMATTER_CLASS,
+        },
         'airflow_coloured': {
             'format': COLORED_LOG_FORMAT if COLORED_LOG else LOG_FORMAT,
-            'class': COLORED_FORMATTER_CLASS if COLORED_LOG else 'logging.Formatter',
+            'class': COLORED_FORMATTER_CLASS if COLORED_LOG else LOG_FORMATTER_CLASS,
         },
     },
     'filters': {
diff --git a/airflow/utils/log/colored_log.py b/airflow/utils/log/colored_log.py
--- a/airflow/utils/log/colored_log.py
+++ b/airflow/utils/log/colored_log.py
@@ -43,6 +43,10 @@ class CustomTTYColoredFormatter(TTYColoredFormatter):
     traceback.
     """
 
+    # copy of airflow.utils.log.timezone_aware.TimezoneAware
+    default_time_format = '%Y-%m-%d %H:%M:%S%z'
+    default_msec_format = '%s %03dms'
+
     def __init__(self, *args, **kwargs):
         kwargs["stream"] = sys.stdout or kwargs.get("stream")
         kwargs["log_colors"] = DEFAULT_COLORS
diff --git a/airflow/utils/log/timezone_aware.py b/airflow/utils/log/timezone_aware.py
new file mode 100644
--- /dev/null
+++ b/airflow/utils/log/timezone_aware.py
@@ -0,0 +1,39 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements.  See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership.  The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License.  You may obtain a copy of the License at
+#
+#   http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied.  See the License for the
+# specific language governing permissions and limitations
+# under the License.
+import logging
+
+
+class TimezoneAware(logging.Formatter):
+    """
+    Override `default_time_format` and `default_msec_format` to specify utc offset.
+
+    utc offset is the matter, without it,  time conversion could be wrong. With this Formatter, `%(asctime)s`
+    will be formatted containing utc offset. (e.g. 2022-06-12 13:00:00+0000 123ms)
+
+    moments.js couldn't parse milliseconds comes after utc offset, so it would be ideal `%(asctime)s`
+    formatted with millisecond comes before utc offset in th first place. (e.g 2022-06-12 13:00:00.123+0000)
+    But python standard lib doesn't support format like that.
+
+    Omitting milliseconds is possible by assigning `default_msec_format` to `None`. But this requires
+    python3.9 or higher, so we can't omit milliseconds until dropping support 3.8 or under.
+
+    Therefore, to use in moments.js, formatted `%(asctime)s` has to be re-formatted by javascript side.
+    """
+
+    default_time_format = '%Y-%m-%d %H:%M:%S%z'
+    default_msec_format = '%s %03dms'


Test patch: