Issue 1:
Apache Airflow version

2.6.1

What happened

The AwaitMessageTriggerFunctionSensor is showing some buggy behaviour.

When consuming from a topic, it is correctly applying the apply_function in order to yield a TriggerEvent.

However, it is consuming multiple messages at a time and not yielding a trigger for the correct amount of messages that would be eligble (return a value in the apply_function). The observed behaviour is as follows:

Sensor is deferred and messages start getting consumed

Multiple eligble messages trigger a single TriggerEvent instead of multiple TriggerEvents.

The sensor returns to a deferred state , repeating the cycle.

The event_triggered_function is being called correctly. However, due to the issue in consuming and correctly generating the appropriate TriggerEvents some of them are missed.

What you think should happen instead

Each eligble message should create an individual TriggerEvent to be consumed by the event_triggered_function.

How to reproduce

Use a producer DAG to produce a set amount of messages on your kafka topic

Use a listener DAG to consume this topic, screening for eligble messages (apply_function) and use the event_trigger_function to monitor the amount of events that are being consumed.

Operating System

Kubernetes cluster - Linux

Versions of Apache Airflow Providers

apache-airflow-providers-apache-kafka==1.1.0

Deployment

Official Apache Airflow Helm Chart

Deployment details

helm chart version 1.9.0

Anything else

Every time (independent of topic, message content, apply_function and event_triggered_function)

Are you willing to submit PR?

Yes I am willing to submit a PR!

Code of Conduct

I agree to follow this project's Code of Conduct

Issue 2:
Apache Airflow version

2.3.0 (latest released)

What happened

upgraded from 2.2.5

airflow db upgrade showed a list of messages inline with what is in the changelog:

Database configuration moved to new section (#22284)

The following configurations have been moved from [core] to the new [database] section. However when reading the new option, the old option will be checked to see if it exists. If it does a DeprecationWarning will be issued and the old option will be used instead.

sql_alchemy_conn
sql_engine_encoding
sql_engine_collation_for_ids
sql_alchemy_pool_enabled
sql_alchemy_pool_size
sql_alchemy_max_overflow
sql_alchemy_pool_recycle
sql_alchemy_pool_pre_ping
sql_alchemy_schema
sql_alchemy_connect_args
load_default_connections
max_db_retries

so nice as I am I moved those lines from [core] to a new [database] section and airflow db upgrade did not mention them again so all was well.

After that I started the services

sudo systemctl start airflow-webserver.service

sudo systemctl start airflow-scheduler.service

but the webservice failed with this error:

May 04 08:46:21 postest airflow[108983]: [2022-05-04 08:46:21 +0200] [108983] [ERROR] Exception in worker process

May 04 08:46:21 postest airflow[108983]: [2022-05-04 08:46:21,254] {configuration.py:494} WARNING - section/key [core/sql_alchemy_conn] not found in config

May 04 08:46:21 postest airflow[108982]: [2022-05-04 08:46:21 +0200] [108982] [INFO] Worker exiting (pid: 108982)

May 04 08:46:21 postest airflow[108982]: airflow.exceptions.AirflowConfigException: section/key [core/sql_alchemy_conn] not found in config

for now I duplicated the lines, so they are both in [core] and [database], so no big issue.

What you think should happen instead

service should start and find the settings in the [database] section

How to reproduce

upgrade from 2.2.5, move the lines in question to a new [database] section and start the webservice.

Operating System

Ubuntu 20.04.4 LTS (Focal Fossa)

Versions of Apache Airflow Providers

apache-airflow-providers-ftp==2.1.2

apache-airflow-providers-http==2.1.2

apache-airflow-providers-imap==2.2.3

apache-airflow-providers-postgres==4.1.0

apache-airflow-providers-sqlite==2.1.3

Deployment

Other

Deployment details

No response

Anything else

No response

Are you willing to submit PR?

Yes I am willing to submit a PR!

Code of Conduct

I agree to follow this project's Code of Conduct