Issue 1:
Apache Airflow version:

2.0.1

Kubernetes version (if you are using kubernetes) (use kubectl version):

none

Environment:

Cloud provider or hardware configuration:

OS (e.g. from /etc/os-release):

ubuntu 20.04

Kernel (e.g. uname -a):

Linux rafay-Inspiron-7537 5.8.0-48-generic #54~20.04.1-Ubuntu SMP Sat Mar 20 13:40:25 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux

Install tools:

Others:

What happened:

[2021-04-06 10:58:25,430] {db.py:674} INFO - Creating tables
/usr/local/lib/python3.7/site-packages/sqlalchemy/orm/relationships.py:3463 SAWarning: relationship 'DagRun.serialized_dag' will copy column serialized_dag.dag_id to column dag_run.dag_id, which conflicts with relationship(s): 'TaskInstance.dag_run' (copies task_instance.dag_id to dag_run.dag_id), 'DagRun.task_instances' (copies task_instance.dag_id to dag_run.dag_id). If this is not the intention, consider if these relationships should be linked with back_populates, or if viewonly=True should be applied to one or more if they are read-only. For the less common case that foreign key constraints are partially overlapping, the orm.foreign() annotation can be used to isolate the columns that should be written towards. The 'overlaps' parameter may be used to remove this warning. (Background on this error at: http://sqlalche.me/e/14/qzyx)
/usr/local/lib/python3.7/site-packages/sqlalchemy/orm/relationships.py:3463 SAWarning: relationship 'SerializedDagModel.dag_runs' will copy column serialized_dag.dag_id to column dag_run.dag_id, which conflicts with relationship(s): 'TaskInstance.dag_run' (copies task_instance.dag_id to dag_run.dag_id), 'DagRun.task_instances' (copies task_instance.dag_id to dag_run.dag_id). If this is not the intention, consider if these relationships should be linked with back_populates, or if viewonly=True should be applied to one or more if they are read-only. For the less common case that foreign key constraints are partially overlapping, the orm.foreign() annotation can be used to isolate the columns that should be written towards. The 'overlaps' parameter may be used to remove this warning. (Background on this error at: http://sqlalche.me/e/14/qzyx)
INFO [alembic.runtime.migration] Context impl SQLiteImpl.
INFO [alembic.runtime.migration] Will assume non-transactional DDL.
Traceback (most recent call last):
File "/usr/local/bin/airflow", line 8, in <module>
sys.exit(main())
File "/usr/local/lib/python3.7/site-packages/airflow/__main__.py", line 40, in main
args.func(args)
File "/usr/local/lib/python3.7/site-packages/airflow/cli/cli_parser.py", line 48, in command
return func(*args, **kwargs)
File "/usr/local/lib/python3.7/site-packages/airflow/cli/commands/db_command.py", line 31, in initdb
db.initdb()
File "/usr/local/lib/python3.7/site-packages/airflow/utils/db.py", line 549, in initdb
upgradedb()
File "/usr/local/lib/python3.7/site-packages/airflow/utils/db.py", line 684, in upgradedb
command.upgrade(config, 'heads')
File "/usr/local/lib/python3.7/site-packages/alembic/command.py", line 294, in upgrade
script.run_env()
File "/usr/local/lib/python3.7/site-packages/alembic/script/base.py", line 490, in run_env
util.load_python_file(self.dir, "env.py")
File "/usr/local/lib/python3.7/site-packages/alembic/util/pyfiles.py", line 97, in load_python_file
module = load_module_py(module_id, path)
File "/usr/local/lib/python3.7/site-packages/alembic/util/compat.py", line 182, in load_module_py
spec.loader.exec_module(module)
File "<frozen importlib._bootstrap_external>", line 728, in exec_module
File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
File "/usr/local/lib/python3.7/site-packages/airflow/migrations/env.py", line 108, in <module>
run_migrations_online()
File "/usr/local/lib/python3.7/site-packages/airflow/migrations/env.py", line 102, in run_migrations_online
context.run_migrations()
File "<string>", line 8, in run_migrations
File "/usr/local/lib/python3.7/site-packages/alembic/runtime/environment.py", line 813, in run_migrations
self.get_context().run_migrations(**kw)
File "/usr/local/lib/python3.7/site-packages/alembic/runtime/migration.py", line 548, in run_migrations
for step in self._migrations_fn(heads, self):
File "/usr/local/lib/python3.7/site-packages/alembic/command.py", line 283, in upgrade
return script._upgrade_revs(revision, rev)
File "/usr/local/lib/python3.7/site-packages/alembic/script/base.py", line 365, in _upgrade_revs
revs = list(revs)
File "/usr/local/lib/python3.7/site-packages/alembic/script/revision.py", line 904, in _iterate_revisions
requested_lowers = self.get_revisions(lower)
File "/usr/local/lib/python3.7/site-packages/alembic/script/revision.py", line 455, in get_revisions
return sum([self.get_revisions(id_elem) for id_elem in id_], ())
File "/usr/local/lib/python3.7/site-packages/alembic/script/revision.py", line 455, in <listcomp>
return sum([self.get_revisions(id_elem) for id_elem in id_], ())
File "/usr/local/lib/python3.7/site-packages/alembic/script/revision.py", line 457, in get_revisions
resolved_id, branch_label = self._resolve_revision_number(id_)
File "/usr/local/lib/python3.7/site-packages/alembic/script/revision.py", line 640, in _resolve_revision_number
self._revision_map
File "/usr/local/lib/python3.7/site-packages/alembic/util/langhelpers.py", line 234, in __get__
obj.__dict__[self.__name__] = result = self.fget(obj)
File "/usr/local/lib/python3.7/site-packages/alembic/script/revision.py", line 156, in _revision_map
for revision in self._generator():
File "/usr/local/lib/python3.7/site-packages/alembic/script/base.py", line 115, in _load_revisions
script = Script._from_filename(self, vers, file_)
File "/usr/local/lib/python3.7/site-packages/alembic/script/base.py", line 904, in _from_filename
module = util.load_python_file(dir_, filename)
File "/usr/local/lib/python3.7/site-packages/alembic/util/pyfiles.py", line 97, in load_python_file
module = load_module_py(module_id, path)
File "/usr/local/lib/python3.7/site-packages/alembic/util/compat.py", line 182, in load_module_py
spec.loader.exec_module(module)
File "<frozen importlib._bootstrap_external>", line 728, in exec_module
File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
File "/usr/local/lib/python3.7/site-packages/airflow/migrations/versions/82b7c48c147f_remove_can_read_permission_on_config_.py", line 28, in <module>
from airflow.www.app import create_app
File "/usr/local/lib/python3.7/site-packages/airflow/www/app.py", line 38, in <module>
from airflow.www.extensions.init_views import (
File "/usr/local/lib/python3.7/site-packages/airflow/www/extensions/init_views.py", line 29, in <module>
from airflow.www.views import lazy_add_provider_discovered_options_to_connection_form
File "/usr/local/lib/python3.7/site-packages/airflow/www/views.py", line 96, in <module>
from airflow.www import auth, utils as wwwutils
File "/usr/local/lib/python3.7/site-packages/airflow/www/utils.py", line 27, in <module>
from flask_appbuilder.models.sqla.interface import SQLAInterface
File "/usr/local/lib/python3.7/site-packages/flask_appbuilder/models/sqla/interface.py", line 16, in <module>
from sqlalchemy_utils.types.uuid import UUIDType
File "/usr/local/lib/python3.7/site-packages/sqlalchemy_utils/__init__.py", line 1, in <module>
from .aggregates import aggregated # noqa
File "/usr/local/lib/python3.7/site-packages/sqlalchemy_utils/aggregates.py", line 372, in <module>
from .functions.orm import get_column_key
File "/usr/local/lib/python3.7/site-packages/sqlalchemy_utils/functions/__init__.py", line 1, in <module>
from .database import ( # noqa
File "/usr/local/lib/python3.7/site-packages/sqlalchemy_utils/functions/database.py", line 11, in <module>
from .orm import quote
File "/usr/local/lib/python3.7/site-packages/sqlalchemy_utils/functions/orm.py", line 14, in <module>
from sqlalchemy.orm.query import _ColumnEntity
ImportError: cannot import name '_ColumnEntity' from 'sqlalchemy.orm.query' (/usr/local/lib/python3.7/site-packages/sqlalchemy/orm/query.py)
Traceback (most recent call last):
File "/usr/local/bin/airflow", line 8, in <module>
sys.exit(main())
File "/usr/local/lib/python3.7/site-packages/airflow/__main__.py", line 40, in main
args.func(args)
File "/usr/local/lib/python3.7/site-packages/airflow/cli/cli_parser.py", line 47, in command
func = import_string(import_path)
File "/usr/local/lib/python3.7/site-packages/airflow/utils/module_loading.py", line 32, in import_string
module = import_module(module_path)
File "/usr/local/lib/python3.7/importlib/__init__.py", line 127, in import_module
return _bootstrap._gcd_import(name[level:], package, level)
File "<frozen importlib._bootstrap>", line 1006, in _gcd_import
File "<frozen importlib._bootstrap>", line 983, in _find_and_load
File "<frozen importlib._bootstrap>", line 967, in _find_and_load_unlocked
File "<frozen importlib._bootstrap>", line 677, in _load_unlocked
File "<frozen importlib._bootstrap_external>", line 728, in exec_module
File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
File "/usr/local/lib/python3.7/site-packages/airflow/cli/commands/user_command.py", line 29, in <module>
from airflow.www.app import cached_app
File "/usr/local/lib/python3.7/site-packages/airflow/www/app.py", line 38, in <module>
from airflow.www.extensions.init_views import (
File "/usr/local/lib/python3.7/site-packages/airflow/www/extensions/init_views.py", line 29, in <module>
from airflow.www.views import lazy_add_provider_discovered_options_to_connection_form
File "/usr/local/lib/python3.7/site-packages/airflow/www/views.py", line 96, in <module>
from airflow.www import auth, utils as wwwutils
File "/usr/local/lib/python3.7/site-packages/airflow/www/utils.py", line 27, in <module>
from flask_appbuilder.models.sqla.interface import SQLAInterface
File "/usr/local/lib/python3.7/site-packages/flask_appbuilder/models/sqla/interface.py", line 16, in <module>
from sqlalchemy_utils.types.uuid import UUIDType
File "/usr/local/lib/python3.7/site-packages/sqlalchemy_utils/__init__.py", line 1, in <module>
from .aggregates import aggregated # noqa
File "/usr/local/lib/python3.7/site-packages/sqlalchemy_utils/aggregates.py", line 372, in <module>
from .functions.orm import get_column_key
File "/usr/local/lib/python3.7/site-packages/sqlalchemy_utils/functions/__init__.py", line 1, in <module>
from .database import ( # noqa
File "/usr/local/lib/python3.7/site-packages/sqlalchemy_utils/functions/database.py", line 11, in <module>
from .orm import quote
File "/usr/local/lib/python3.7/site-packages/sqlalchemy_utils/functions/orm.py", line 14, in <module>
from sqlalchemy.orm.query import _ColumnEntity
ImportError: cannot import name '_ColumnEntity' from 'sqlalchemy.orm.query' (/usr/local/lib/python3.7/site-packages/sqlalchemy/orm/query.py)

What you expected to happen:

airflow db init should execute without a hitch

Sqlalchemy package has been updated with some breaking changes. pip is installing sqlalchemy-1.4.5.

https://stackoverflow.com/questions/66644975/importerror-cannot-import-name-columnentity-from-sqlalchemy-orm-query

How to reproduce it:

pip install apache-airflow[slack,statsd]

Anything else we need to know:

Issue 2:
Apache Airflow version: 2.0.1

Kubernetes version (if you are using kubernetes) (use kubectl version): N/A

Environment:

Cloud provider or hardware configuration:

OS (e.g. from /etc/os-release): Mac OS Big Sur

Kernel (e.g. uname -a):

Install tools: pip 20.1.1

Others:

What happened:

Our CI environment broke due to the release of SQLAlchemy 1.4, which is incompatible with the latest version of sqlalchemy-utils. (Related issue)

Partial stacktrace:

File "/Users/samwheating/Desktop/tmp_venv/lib/python3.7/site-packages/airflow/www/utils.py", line 27, in <module>
from flask_appbuilder.models.sqla.interface import SQLAInterface
File "/Users/samwheating/Desktop/tmp_venv/lib/python3.7/site-packages/flask_appbuilder/models/sqla/interface.py", line 16, in <module>
from sqlalchemy_utils.types.uuid import UUIDType
File "/Users/samwheating/Desktop/tmp_venv/lib/python3.7/site-packages/sqlalchemy_utils/__init__.py", line 1, in <module>
from .aggregates import aggregated # noqa
File "/Users/samwheating/Desktop/tmp_venv/lib/python3.7/site-packages/sqlalchemy_utils/aggregates.py", line 372, in <module>
from .functions.orm import get_column_key
File "/Users/samwheating/Desktop/tmp_venv/lib/python3.7/site-packages/sqlalchemy_utils/functions/__init__.py", line 1, in <module>
from .database import ( # noqa
File "/Users/samwheating/Desktop/tmp_venv/lib/python3.7/site-packages/sqlalchemy_utils/functions/database.py", line 11, in <module>
from .orm import quote
File "/Users/samwheating/Desktop/tmp_venv/lib/python3.7/site-packages/sqlalchemy_utils/functions/orm.py", line 14, in <module>
from sqlalchemy.orm.query import _ColumnEntity
ImportError: cannot import name '_ColumnEntity' from 'sqlalchemy.orm.query' (/Users/samwheating/Desktop/tmp_venv/lib/python3.7/site-packages/sqlalchemy/orm/query.py)

I'm not sure what the typical procedure is in the case of breaking changes to dependencies, but seeing as there's an upcoming release I thought it might be worth pinning sqlalchemy to 1.3.x? (Or pin the version of sqlalchemy-utils to a compatible version if one is released before Airflow 2.0.2)

What you expected to happen:

airflow db init to run successfully.

How to reproduce it:

Create a new virtualenv

pip install apache-airflow

airflow db init