Issue 1:
Apache Airflow version

2.4.2

What happened

We upgraded airflow from 2.4.1 to 2.4.2 and immediately notice that every task log line is duplicated into CloudWatch. Comparing logs from tasks run before upgrade and after upgrade indicates that the issue is not in how the logs are displayed in Airflow, but rather that it now produces two log lines instead of one.

When observing both the CloudWatch log streams and the Airflow UI, we can see duplicate log lines for all most log entries post upgrade, whilst seeing single log lines in tasks before upgrade.

This happens both for tasks ran in a remote EcsRunTaskOperator's as well as in regular PythonOperator's.

What you think should happen instead

A single non-duplicate log line should be produced into CloudWatch.

How to reproduce

From my understanding now, any setup on 2.4.2 that uses CloudWatch remote logging will produce duplicate log lines. (But I have not been able to confirm other setups)

Operating System

Docker: apache/airflow:2.4.2-python3.9 - Running on AWS ECS Fargate

Versions of Apache Airflow Providers

apache-airflow[celery,postgres,apache.hive,jdbc,mysql,ssh,amazon,google,google_auth]==2.4.2
apache-airflow-providers-amazon==6.0.0

Deployment

Other Docker-based deployment

Deployment details

We are running a docker inside Fargate ECS on AWS.

The following environment variables + config in CloudFormation control remote logging:

- Name: AIRFLOW__LOGGING__REMOTE_LOGGING
Value: True
- Name: AIRFLOW__LOGGING__REMOTE_BASE_LOG_FOLDER
Value: !Sub "cloudwatch://${TasksLogGroup.Arn}"

Anything else

We did not change any other configuration during the upgrade, simply bumped the requirements for provider list + docker image from 2.4.1 to 2.4.2.

Are you willing to submit PR?

Yes I am willing to submit a PR!

Code of Conduct

I agree to follow this project's Code of Conduct

Issue 2:
Apache Airflow version:1.10.12

Apache Airflow Provider versions (please include all providers that are relevant to your bug):

No other providers

Kubernetes version (if you are using kubernetes) (use kubectl version):

No using k8s

Environment:

AWS Linux

Cloud provider or hardware configuration:AWS

OS (e.g. from /etc/os-release):

-NAME="Amazon Linux"

VERSION="2"

ID="amzn"

ID_LIKE="centos rhel fedora"

Kernel (e.g. uname -a): 4.14.186-146.268.amzn2.x86_64 #1 SMP Tue Jul 14 18:16:52 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux

Install tools: yum, pip3

Others:

What happened:

I have been running airflow 1.10.12 for several months without any issues. Recently I updated airflow to use SSL cert. After that I reboot the airflow server and then restarted the airflow scheduler and Webserver as following:

airflow Webserver &

airflow scheduler &

The jobs run normally at beginning. However, within one day, all the new jobs will immediately fail. From UI, I saw the task status as failed. However there is no log under the task when I checked the log from UI.

I also checked the /logs/scheduler directory, except a warning of "psycopg2.errors.UniqueViolation: duplicate key value violates unique constraint "variable_key_key", which has existed for long time, I didn't see any other warning or error.

Now this problem happened every day, i.e., if I restart airflow Webserver and scheduler, it works well. But within one day, all the new jobs will immediately fail with no log.

Do you have any insight what can go wrong? Thanks.

What you expected to happen:

How to reproduce it:

Not exactly sure how to reproduce. I just reboot airflow server.

Anything else we need to know:

How often does this problem occur? Once? Every time etc?

I noticed the problem happens within 1 day.

Any relevant logs to include? Put them here in side a detail tag: