Issue 1:
Apache Airflow version

2.2.1 (latest released)

Operating System

debian

Versions of Apache Airflow Providers

apache-airflow==2.2.1

apache-airflow-providers-amazon==2.3.0

apache-airflow-providers-ftp==2.0.1

apache-airflow-providers-google==6.0.0

apache-airflow-providers-http==2.0.1

apache-airflow-providers-imap==2.0.1

apache-airflow-providers-jira==2.0.1

apache-airflow-providers-mysql==2.1.1

apache-airflow-providers-postgres==2.3.0

apache-airflow-providers-redis==2.0.1

apache-airflow-providers-sqlite==2.0.1

apache-airflow-providers-ssh==2.2.0

Deployment

Other Docker-based deployment

Deployment details

Dask executor, custom-built Docker images, postgres 12.7 backend

What happened

Shut off Airflow cluster

Upgraded database from 2.0.2 to 2.2.1

Restarted Airflow cluster

The scheduler failed to start

I checked the logs, and found this:

2021-11-04 21:15:35,566 ERROR - Exception when executing SchedulerJob._run_scheduler_loop
Traceback (most recent call last):
File "/usr/local/lib/python3.9/site-packages/airflow/jobs/scheduler_job.py", line 628, in _execute
self._run_scheduler_loop()
File "/usr/local/lib/python3.9/site-packages/airflow/jobs/scheduler_job.py", line 709, in _run_scheduler_loop
num_queued_tis = self._do_scheduling(session)
File "/usr/local/lib/python3.9/site-packages/airflow/jobs/scheduler_job.py", line 782, in _do_scheduling
self._create_dagruns_for_dags(guard, session)
File "/usr/local/lib/python3.9/site-packages/airflow/utils/retries.py", line 76, in wrapped_function
for attempt in run_with_db_retries(max_retries=retries, logger=logger, **retry_kwargs):
File "/usr/local/lib/python3.9/site-packages/tenacity/__init__.py", line 382, in __iter__
do = self.iter(retry_state=retry_state)
File "/usr/local/lib/python3.9/site-packages/tenacity/__init__.py", line 349, in iter
return fut.result()
File "/usr/local/lib/python3.9/concurrent/futures/_base.py", line 438, in result
return self.__get_result()
File "/usr/local/lib/python3.9/concurrent/futures/_base.py", line 390, in __get_result
raise self._exception
File "/usr/local/lib/python3.9/site-packages/airflow/utils/retries.py", line 85, in wrapped_function
return func(*args, **kwargs)
File "/usr/local/lib/python3.9/site-packages/airflow/jobs/scheduler_job.py", line 847, in _create_dagruns_for_dags
self._create_dag_runs(query.all(), session)
File "/usr/local/lib/python3.9/site-packages/airflow/jobs/scheduler_job.py", line 917, in _create_dag_runs
self._update_dag_next_dagruns(dag, dag_model, active_runs_of_dags[dag.dag_id])
File "/usr/local/lib/python3.9/site-packages/airflow/jobs/scheduler_job.py", line 926, in _update_dag_next_dagruns
if total_active_runs >= dag_model.max_active_runs:
TypeError: '>=' not supported between instances of 'int' and 'NoneType'

I checked the code and saw that dag_model.max_active_runs comes directly from the database and is nullable

I updated all values for dag.max_active_runs to be non-null

Restarted the scheduler

Everything ran fine

What you expected to happen

In this case, I would expect the code to properly handle nullable columns, probably by using the default value provided in the configs.

How to reproduce

This is easily reproducible. Start a fresh instance of Airflow with a database at version 2.2.1 and follow these steps:

Add a DAG to the instance

Manually set dag.max_active_runs to null in the database

Enable the DAG to cause the scheduler to attempt to parse/schedule it

BOOM! The scheduler will crash

Anything else

Newly registered DAGs have this value populated in the database with the default value, so this issue will likely only occur on a database upgrade.

Are you willing to submit PR?

Yes I am willing to submit a PR!

Code of Conduct

I agree to follow this project's Code of Conduct

Issue 2:
Apache Airflow version

2.2.1 (latest released)

Operating System

Linux (ubuntu 20.04)

Versions of Apache Airflow Providers

apache-airflow-providers-ftp==2.0.1
apache-airflow-providers-http==2.0.1
apache-airflow-providers-imap==2.0.1
apache-airflow-providers-microsoft-azure==3.1.0
apache-airflow-providers-microsoft-mssql==2.0.0
apache-airflow-providers-postgres==2.0.0
apache-airflow-providers-salesforce==3.1.0
apache-airflow-providers-sqlite==2.0.1
apache-airflow-providers-ssh==2.1.0

Deployment

Other

Deployment details

Airflow installed on Azure virtual machine (Standard D8s v3 (8 vcpus, 32 GiB memory)), the VM is dedicated for Airflow only

What happened

After I updated Airflow from 2.1.3 to 2.2.1 and run db update, I run airflow scheduler and got an error:

[2021-11-02 11:43:20,846] {scheduler_job.py:644} ERROR - Exception when executing SchedulerJob._run_scheduler_loop
Traceback (most recent call last):
File "/home/DataPipeline/.local/lib/python3.8/site-packages/airflow/jobs/scheduler_job.py", line 628, in _execute
self._run_scheduler_loop()
File "/home/DataPipeline/.local/lib/python3.8/site-packages/airflow/jobs/scheduler_job.py", line 709, in _run_scheduler_loop
num_queued_tis = self._do_scheduling(session)
File "/home/DataPipeline/.local/lib/python3.8/site-packages/airflow/jobs/scheduler_job.py", line 782, in _do_scheduling
self._create_dagruns_for_dags(guard, session)
File "/home/DataPipeline/.local/lib/python3.8/site-packages/airflow/utils/retries.py", line 76, in wrapped_function
for attempt in run_with_db_retries(max_retries=retries, logger=logger, **retry_kwargs):
File "/home/DataPipeline/.local/lib/python3.8/site-packages/tenacity/__init__.py", line 382, in __iter__
do = self.iter(retry_state=retry_state)
File "/home/DataPipeline/.local/lib/python3.8/site-packages/tenacity/__init__.py", line 349, in iter
return fut.result()
File "/usr/lib/python3.8/concurrent/futures/_base.py", line 437, in result
return self.__get_result()
File "/usr/lib/python3.8/concurrent/futures/_base.py", line 389, in __get_result
raise self._exception
File "/home/DataPipeline/.local/lib/python3.8/site-packages/airflow/utils/retries.py", line 85, in wrapped_function
return func(*args, **kwargs)
File "/home/DataPipeline/.local/lib/python3.8/site-packages/airflow/jobs/scheduler_job.py", line 847, in _create_dagruns_for_dags
self._create_dag_runs(query.all(), session)
File "/home/DataPipeline/.local/lib/python3.8/site-packages/airflow/jobs/scheduler_job.py", line 917, in _create_dag_runs
self._update_dag_next_dagruns(dag, dag_model, active_runs_of_dags[dag.dag_id])
File "/home/DataPipeline/.local/lib/python3.8/site-packages/airflow/jobs/scheduler_job.py", line 926, in _update_dag_next_dagruns
if total_active_runs >= dag_model.max_active_runs:
TypeError: '>=' not supported between instances of 'int' and 'NoneType'

The scheduler is not able to run.

What you expected to happen

This behaviour happens right after the update. I have another VM with Airflow (the two machines are similar, the difference in location of the VM).

How to reproduce

after the update to 2.2.1, change the config in accordance with Warnings and run airflow scheduler

Anything else

No response

Are you willing to submit PR?

Yes I am willing to submit a PR!

Code of Conduct

I agree to follow this project's Code of Conduct