Repository: apache/airflow

Issue:
Migrate from 2.1.4 to 2.2.0
### Apache Airflow version

2.2.0

### Operating System

Linux

### Versions of Apache Airflow Providers

default.

### Deployment

Docker-Compose

### Deployment details

Using airflow-2.2.0python3.7

### What happened

Upgrading image from apache/airflow:2.1.4-python3.7
to apache/airflow:2.2.0-python3.7
Cause this inside scheduler, which is not starting:

```
Python version: 3.7.12
Airflow version: 2.2.0
Node: 6dd55b0a5dd7
-------------------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/airflow/.local/lib/python3.7/site-packages/sqlalchemy/engine/base.py", line 1277, in _execute_context
    cursor, statement, parameters, context
  File "/home/airflow/.local/lib/python3.7/site-packages/sqlalchemy/engine/default.py", line 608, in do_execute
    cursor.execute(statement, parameters)
psycopg2.errors.UndefinedColumn: column dag.max_active_tasks does not exist
LINE 1: ..., dag.schedule_interval AS dag_schedule_interval, dag.max_ac...
                                                             ^

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/home/airflow/.local/lib/python3.7/site-packages/flask/app.py", line 2447, in wsgi_app
    response = self.full_dispatch_request()
  File "/home/airflow/.local/lib/python3.7/site-packages/flask/app.py", line 1952, in full_dispatch_request
    rv = self.handle_user_exception(e)
  File "/home/airflow/.local/lib/python3.7/site-packages/flask/app.py", line 1821, in handle_user_exception
    reraise(exc_type, exc_value, tb)
  File "/home/airflow/.local/lib/python3.7/site-packages/flask/_compat.py", line 39, in reraise
    raise value
  File "/home/airflow/.local/lib/python3.7/site-packages/flask/app.py", line 1950, in full_dispatch_request
    rv = self.dispatch_request()
  File "/home/airflow/.local/lib/python3.7/site-packages/flask/app.py", line 1936, in dispatch_request
    return self.view_functions[rule.endpoint](**req.view_args)
  File "/home/airflow/.local/lib/python3.7/site-packages/airflow/www/auth.py", line 51, in decorated
    return func(*args, **kwargs)
  File "/home/airflow/.local/lib/python3.7/site-packages/airflow/www/views.py", line 588, in index
    filter_dag_ids = current_app.appbuilder.sm.get_accessible_dag_ids(g.user)
  File "/home/airflow/.local/lib/python3.7/site-packages/airflow/www/security.py", line 377, in get_accessible_dag_ids
    return {dag.dag_id for dag in accessible_dags}
  File "/home/airflow/.local/lib/python3.7/site-packages/sqlalchemy/orm/query.py", line 3535, in __iter__
    return self._execute_and_instances(context)
  File "/home/airflow/.local/lib/python3.7/site-packages/sqlalchemy/orm/query.py", line 3560, in _execute_and_instances
    result = conn.execute(querycontext.statement, self._params)
  File "/home/airflow/.local/lib/python3.7/site-packages/sqlalchemy/engine/base.py", line 1011, in execute
    return meth(self, multiparams, params)
  File "/home/airflow/.local/lib/python3.7/site-packages/sqlalchemy/sql/elements.py", line 298, in _execute_on_connection
    return connection._execute_clauseelement(self, multiparams, params)
  File "/home/airflow/.local/lib/python3.7/site-packages/sqlalchemy/engine/base.py", line 1130, in _execute_clauseelement
    distilled_params,
  File "/home/airflow/.local/lib/python3.7/site-packages/sqlalchemy/engine/base.py", line 1317, in _execute_context
    e, statement, parameters, cursor, context
  File "/home/airflow/.local/lib/python3.7/site-packages/sqlalchemy/engine/base.py", line 1511, in _handle_dbapi_exception
    sqlalchemy_exception, with_traceback=exc_info[2], from_=e
  File "/home/airflow/.local/lib/python3.7/site-packages/sqlalchemy/util/compat.py", line 182, in raise_
    raise exception
  File "/home/airflow/.local/lib/python3.7/site-packages/sqlalchemy/engine/base.py", line 1277, in _execute_context
    cursor, statement, parameters, context
  File "/home/airflow/.local/lib/python3.7/site-packages/sqlalchemy/engine/default.py", line 608, in do_execute
    cursor.execute(statement, parameters)
sqlalchemy.exc.ProgrammingError: (psycopg2.errors.UndefinedColumn) column dag.max_active_tasks does not exist
LINE 1: ..., dag.schedule_interval AS dag_schedule_interval, dag.max_ac...
                                                             ^

[SQL: SELECT dag.dag_id AS dag_dag_id, dag.root_dag_id AS dag_root_dag_id, dag.is_paused AS dag_is_paused, dag.is_subdag AS dag_is_subdag, dag.is_active AS dag_is_active, dag.last_parsed_time AS dag_last_parsed_time, dag.last_pickled AS dag_last_pickled, dag.last_expired AS dag_last_expired, dag.scheduler_lock AS dag_scheduler_lock, dag.pickle_id AS dag_pickle_id, dag.fileloc AS dag_fileloc, dag.owners AS dag_owners, dag.description AS dag_description, dag.default_view AS dag_default_view, dag.schedule_interval AS dag_schedule_interval, dag.max_active_tasks AS dag_max_active_tasks, dag.max_active_runs AS dag_max_active_runs, dag.has_task_concurrency_limits AS dag_has_task_concurrency_limits, dag.next_dagrun AS dag_next_dagrun, dag.next_dagrun_data_interval_start AS dag_next_dagrun_data_interval_start, dag.next_dagrun_data_interval_end AS dag_next_dagrun_data_interval_end, dag.next_dagrun_create_after AS dag_next_dagrun_create_after 
FROM dag]
(Background on this error at: http://sqlalche.me/e/13/f405)
```

### What you expected to happen

Automatic database migration and properly working scheduler.

### How to reproduce

Ugrade from 2.1.4 to 2.2.0 with some dags history.

### Anything else

_No response_

### Are you willing to submit PR?

- [ ] Yes I am willing to submit a PR!

### Code of Conduct

- [X] I agree to follow this project's [Code of Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md)



Discussion:
Thanks for opening your first issue here! Be sure to follow the issue template!

Did you run `airflow db upgrade`?
Yes, I did. Got this:
```
airflow@6dd55b0a5dd7:/opt/airflow$ airflow db upgrade
/home/airflow/.local/lib/python3.7/site-packages/airflow/configuration.py:357
DeprecationWarning: The dag_concurrency option in [core] has been renamed to max_active_tasks_per_dag - the old setting has been used, but please update your config.
DB: postgresql+psycopg2://airflow:***@postgres/airflow
/home/airflow/.local/lib/python3.7/site-packages/sqlalchemy/dialects/postgresql/base.py:3573 
SAWarning: Predicate of partial index idx_dag_run_queued_dags ignored during reflection
/home/airflow/.local/lib/python3.7/site-packages/sqlalchemy/dialects/postgresql/base.py:3573 
SAWarning: Predicate of partial index idx_dag_run_running_dags ignored during reflection
[2021-10-11 19:21:13,069] {db.py:815} ERROR - Automatic migration is not available
[2021-10-11 19:21:13,069] {db.py:817} ERROR - The task_instance table has 53 rows without a corresponding dag_run row. You must manually correct this problem (possibly by deleting the problem rows).
[2021-10-11 19:21:13,071] {db.py:817} ERROR - The task_fail table has 24 rows without a corresponding dag_run row. You must manually correct this problem (possibly by deleting the problem rows).
```

I have just killed my docker-compose with 2.1.4, changed image version to 2.2.0 and airflow never started again :)
Downgrading to 2.1.4 fixes problem :) 
Did you do the suggested fix ?

```
The task_instance table has 53 rows without a corresponding dag_run row. You must manually correct this problem (possibly by deleting the problem rows).
The task_fail table has 24 rows without a corresponding dag_run row. You must manually correct this problem (possibly by deleting the problem rows).` ?
```

Not yet, but I'm upgrading each new release from version 2.0.2 and this is first time, I'm getting migration problem. So I thought it's bigger problem. Especially I want to upgrade version in my production environment and seams it will be problematic. 
Well, I think you should folllow what the suggestion is. You seem to have some wrong entries in your DB (which migh be result of bugs or some very old versions of airflow or both). 

Closing as invalid unless following the migration suggestion does not fix the problem (likely you will have to do similar fixes in prooduction BTW).
After deleting wrong rows airflow is working.
Thank you for help.
Can be closed.

Hi @sbialkowski-pixel do you have the query to delete those rows in the postgres console ? I'm also facing this issue.
I've deleted this query, sorry.
But it was inspired how creator throw this error. So I found where in airflow code this error is produced:

https://github.com/apache/airflow/blob/bc19ae7932891704dd359df41b4d43a734590cc9/airflow/utils/db.py#L766-L774

And I've recreate this in SQL.
If some references are lost, this kind of error should be repaired automatically IHMO.

> If some references are lost, this kind of error should be repaired automatically IHMO.

Airflow does not delete user-generated data because we don't know if they are important to the user. It would be a disaster if we automatically delete the offending records if the user delibrately added them and did not get a chance to move them somewhere.

For the same reason, please inspect and fully understand any SQL queries suggested in this issue before running them, to make sure you are actually fine with those rows being deleted.

Problem is that there is no clue, which rows need to be deleted. Also there is no straight logic behind "corresponding rows" for me. 

This part of code for me is the only source of some guidelines.

If there is better way to deal with this kind of problem in future, I will appreciate to hear it :) 
> > If some references are lost, this kind of error should be repaired automatically IHMO.
> 
> Airflow does not delete user-generated data because we don't know if they are important to the user. It would be a disaster if we automatically delete the offending records if the user delibrately added them and did not get a chance to move them somewhere.
> 
> For the same reason, please inspect and fully understand any SQL queries suggested in this issue before running them, to make sure you are actually fine with those rows being deleted.

OK. But if failed task has no more reference to dag_run, there is no point to keep it in database...
Or maybe there is something behind this logic, but I dont get it. 
> > If some references are lost, this kind of error should be repaired automatically IHMO.

What's the reason for those rows to appear in the DB ?  How did they get there? Do we know it?

I think there are two possible scenarios:

1) If it is result of normal operation of Airflow ( even some some special circumstances, manually triggered dags using CLI or UI) 

If that's the case, I agree it should be handled better - Airflow cleanig them or adding "fake" run_ids for those during migration is what I would expect as we should handle this as "regular" migration scenario.

2) They are result of user adding manually entries to the database.

If this is the case then the BEST we could do is to spit-out the exact SQL query to run to delete those rows. We should not delete them automaticaly. 

Seems other people have similar issue https://github.com/apache/airflow/discussions/18912 so for me it looks highly unlikely this was manually added. 

I think the number of reports we have in such short time (I saw 3 reports already) indicate that those rows can appear frequently as result of normal operations by Airflow, and many users might hve similar issues soon. 

If this is realy result of "regular" airflow behaviour, for me it calls for a very quick 2.2.1 with improved migration to handle that case (cc: @kaxil @jedcunningham)
> > > If some references are lost, this kind of error should be repaired automatically IHMO.
> 
> What's the reason for those rows to appear in the DB ? How did they get there? Do we know it?
> 
> I think there are two possible scenarios:
> 
> 1. If it is result of normal operation of Airflow ( even some some special circumstances, manually triggered dags using CLI or UI)
> 
> If that's the case, I agree it should be handled better - Airflow cleanig them or adding "fake" run_ids for those during migration is what I would expect as we should handle this as "regular" migration scenario.
> 
> 1. They are result of user adding manually entries to the database.
> 
> If this is the case then the BEST we could do is to spit-out the exact SQL query to run to delete those rows. We should not delete them automaticaly.

Case 1. Yes, I'm triggering dags manually from UI or by RestApi. 
Case 2. That was very first time I had to work with Airflow Database. I have never add or modified it manually.
I'm only reading connection from db.
> Case 2. That was very first time I had to work with Airflow Database. I have never add or modified it manually.
> I'm only reading connection from db.

Thanks for confirming @sbialkowski-pixel! @kaxil @jedcunningham I think we need to seriously consider 2.2.1

Resolving patch:
diff --git a/airflow/settings.py b/airflow/settings.py
--- a/airflow/settings.py
+++ b/airflow/settings.py
@@ -563,3 +563,6 @@ def initialize():
 #
 # DASHBOARD_UIALERTS: List["UIAlert"]
 DASHBOARD_UIALERTS = []
+
+# Prefix used to identify tables holding data moved during migration.
+AIRFLOW_MOVED_TABLE_PREFIX = "_airflow_moved"
diff --git a/airflow/utils/db.py b/airflow/utils/db.py
--- a/airflow/utils/db.py
+++ b/airflow/utils/db.py
@@ -20,7 +20,7 @@
 import time
 from typing import Iterable
 
-from sqlalchemy import Table, exc, func
+from sqlalchemy import Table, exc, func, inspect, or_, text
 
 from airflow import settings
 from airflow.configuration import conf
@@ -51,15 +51,15 @@
 from airflow.models.serialized_dag import SerializedDagModel  # noqa: F401
 
 # TODO: remove create_session once we decide to break backward compatibility
-from airflow.utils.session import (  # noqa: F401 # pylint: disable=unused-import
-    create_global_lock,
-    create_session,
-    provide_session,
-)
+from airflow.utils.session import create_global_lock, create_session, provide_session  # noqa: F401
 
 log = logging.getLogger(__name__)
 
 
+def _format_airflow_moved_table_name(source_table, version):
+    return "__".join([settings.AIRFLOW_MOVED_TABLE_PREFIX, version.replace(".", "_"), source_table])
+
+
 @provide_session
 def merge_conn(conn, session=None):
     """Add new Connection."""
@@ -697,47 +697,77 @@ def check_conn_type_null(session=None) -> Iterable[str]:
         )
 
 
+def _format_dangling_error(source_table, target_table, invalid_count, reason):
+    noun = "row" if invalid_count == 1 else "rows"
+    return (
+        f"The {source_table} table has {invalid_count} {noun} {reason}, which "
+        f"is invalid. We could not move them out of the way because the "
+        f"{target_table} table already exists in your database. Please either "
+        f"drop the {target_table} table, or manually delete the invalid rows "
+        f"from the {source_table} table."
+    )
+
+
+def _move_dangling_run_data_to_new_table(session, source_table, target_table):
+    where_clause = "where dag_id is null or run_id is null or execution_date is null"
+    session.execute(text(f"create table {target_table} as select * from {source_table} {where_clause}"))
+    session.execute(text(f"delete from {source_table} {where_clause}"))
+
+
 def check_run_id_null(session) -> Iterable[str]:
     import sqlalchemy.schema
 
     metadata = sqlalchemy.schema.MetaData(session.bind)
     try:
-        metadata.reflect(only=["dag_run"])
+        metadata.reflect(only=[DagRun.__tablename__])
     except exc.InvalidRequestError:
         # Table doesn't exist -- empty db
         return
 
-    dag_run = metadata.tables["dag_run"]
-
-    for colname in ('run_id', 'dag_id', 'execution_date'):
-
-        col = dag_run.columns.get(colname)
-        if col is None:
-            continue
-
-        if not col.nullable:
-            continue
-
-        num = session.query(dag_run).filter(col.is_(None)).count()
-        if num > 0:
-            yield (
-                f'The {dag_run.name} table has {num} row{"s" if num != 1 else ""} with a NULL value in '
-                f'{col.name!r}. You must manually correct this problem (possibly by deleting the problem '
-                'rows).'
+    # We can't use the model here since it may differ from the db state due to
+    # this function is run prior to migration. Use the reflected table instead.
+    dagrun_table = metadata.tables[DagRun.__tablename__]
+
+    invalid_dagrun_filter = or_(
+        dagrun_table.c.dag_id.is_(None),
+        dagrun_table.c.run_id.is_(None),
+        dagrun_table.c.execution_date.is_(None),
+    )
+    invalid_dagrun_count = session.query(dagrun_table.c.id).filter(invalid_dagrun_filter).count()
+    if invalid_dagrun_count > 0:
+        dagrun_dangling_table_name = _format_airflow_moved_table_name(dagrun_table.name, "2.2")
+        if dagrun_dangling_table_name in inspect(session.get_bind()).get_table_names():
+            yield _format_dangling_error(
+                source_table=dagrun_table.name,
+                target_table=dagrun_dangling_table_name,
+                invalid_count=invalid_dagrun_count,
+                reason="with a NULL dag_id, run_id, or execution_date",
             )
-    session.rollback()
+            return
+        _move_dangling_run_data_to_new_table(session, dagrun_table.name, dagrun_dangling_table_name)
+
+
+def _move_dangling_task_data_to_new_table(session, source_table, target_table):
+    where_clause = f"""
+        where (task_id, dag_id, execution_date) IN (
+            select source.task_id, source.dag_id, source.execution_date
+            from {source_table} as source
+            left join dag_run as dr
+            on (source.dag_id = dr.dag_id and source.execution_date = dr.execution_date)
+            where dr.id is null
+        )
+    """
+    session.execute(text(f"create table {target_table} as select * from {source_table} {where_clause}"))
+    session.execute(text(f"delete from {source_table} {where_clause}"))
 
 
 def check_task_tables_without_matching_dagruns(session) -> Iterable[str]:
-    from itertools import chain
-
     import sqlalchemy.schema
     from sqlalchemy import and_, outerjoin
 
     metadata = sqlalchemy.schema.MetaData(session.bind)
-    models_to_dagrun = [TaskInstance, TaskFail]
-    models_to_ti = []
-    for model in models_to_dagrun + models_to_ti + [DagRun]:
+    models_to_dagrun = [TaskInstance, TaskReschedule]
+    for model in models_to_dagrun + [DagRun]:
         try:
             metadata.reflect(only=[model.__tablename__])
         except exc.InvalidRequestError:
@@ -745,43 +775,57 @@ def check_task_tables_without_matching_dagruns(session) -> Iterable[str]:
             # version
             pass
 
+    # Key table doesn't exist -- likely empty DB.
     if DagRun.__tablename__ not in metadata or TaskInstance.__tablename__ not in metadata:
-        # Key table doesn't exist -- likely empty DB
-        session.rollback()
         return
 
-    for (model, target) in chain(
-        ((m, metadata.tables[DagRun.__tablename__]) for m in models_to_dagrun),
-        ((m, metadata.tables[TaskInstance.__tablename__]) for m in models_to_ti),
-    ):
-        table = metadata.tables.get(model.__tablename__)
-        if table is None:
+    # We can't use the model here since it may differ from the db state due to
+    # this function is run prior to migration. Use the reflected table instead.
+    dagrun_table = metadata.tables[DagRun.__tablename__]
+
+    existing_table_names = set(inspect(session.get_bind()).get_table_names())
+    errored = False
+
+    for model in models_to_dagrun:
+        # We can't use the model here since it may differ from the db state due to
+        # this function is run prior to migration. Use the reflected table instead.
+        source_table = metadata.tables.get(model.__tablename__)
+        if source_table is None:
             continue
-        if 'run_id' in table.columns:
-            # Migration already applied, don't check again
+
+        # Migration already applied, don't check again.
+        if "run_id" in source_table.columns:
             continue
 
-        # We can't use the model here (as that would have the associationproxy, we instead need to use the
-        # _reflected_ table)
-        join_cond = and_(table.c.dag_id == target.c.dag_id, table.c.execution_date == target.c.execution_date)
-        if "task_id" in target.columns:
-            join_cond = and_(join_cond, table.c.task_id == target.c.task_id)
-
-        query = (
-            session.query(table.c.dag_id, table.c.task_id, table.c.execution_date)
-            .select_from(outerjoin(table, target, join_cond))
-            .filter(target.c.dag_id.is_(None))
-        )  # type: ignore
-
-        num = query.count()
-
-        if num > 0:
-            yield (
-                f'The {table.name} table has {num} row{"s" if num != 1 else ""} without a '
-                f'corresponding {target.name} row. You must manually correct this problem '
-                '(possibly by deleting the problem rows).'
+        source_to_dag_run_join_cond = and_(
+            source_table.c.dag_id == dagrun_table.c.dag_id,
+            source_table.c.execution_date == dagrun_table.c.execution_date,
+        )
+        invalid_row_count = (
+            session.query(source_table.c.dag_id, source_table.c.task_id, source_table.c.execution_date)
+            .select_from(outerjoin(source_table, dagrun_table, source_to_dag_run_join_cond))
+            .filter(dagrun_table.c.dag_id.is_(None))
+            .count()
+        )
+        if invalid_row_count <= 0:
+            continue
+
+        dangling_table_name = _format_airflow_moved_table_name(source_table.name, "2.2")
+        if dangling_table_name in existing_table_names:
+            yield _format_dangling_error(
+                source_table=source_table.name,
+                target_table=dangling_table_name,
+                invalid_count=invalid_row_count,
+                reason=f"without a corresponding {dagrun_table.name} row",
             )
-    session.rollback()
+            errored = True
+            continue
+        _move_dangling_task_data_to_new_table(session, source_table.name, dangling_table_name)
+
+    if errored:
+        session.rollback()
+    else:
+        session.commit()
 
 
 @provide_session
diff --git a/airflow/www/views.py b/airflow/www/views.py
--- a/airflow/www/views.py
+++ b/airflow/www/views.py
@@ -82,7 +82,7 @@
 from pendulum.parsing.exceptions import ParserError
 from pygments import highlight, lexers
 from pygments.formatters import HtmlFormatter
-from sqlalchemy import Date, and_, desc, func, union_all
+from sqlalchemy import Date, and_, desc, func, inspect, union_all
 from sqlalchemy.exc import IntegrityError
 from sqlalchemy.orm import joinedload
 from wtforms import SelectField, validators
@@ -692,10 +692,21 @@ def index(self):
             fm for fm in settings.DASHBOARD_UIALERTS if fm.should_show(current_app.appbuilder.sm)
         ]
 
+        def _iter_parsed_moved_data_table_names():
+            for table_name in inspect(session.get_bind()).get_table_names():
+                segments = table_name.split("__", 2)
+                if len(segments) < 3:
+                    continue
+                if segments[0] != settings.AIRFLOW_MOVED_TABLE_PREFIX:
+                    continue
+                # Second segment is a version marker that we don't need to show.
+                yield segments[2], table_name
+
         return self.render_template(
             'airflow/dags.html',
             dags=dags,
             dashboard_alerts=dashboard_alerts,
+            migration_moved_data_alerts=sorted(set(_iter_parsed_moved_data_table_names())),
             current_page=current_page,
             search_query=arg_search_query if arg_search_query else '',
             page_title=page_title,


Test patch:
diff --git a/tests/www/views/test_views_base.py b/tests/www/views/test_views_base.py
--- a/tests/www/views/test_views_base.py
+++ b/tests/www/views/test_views_base.py
@@ -30,7 +30,7 @@
 
 
 def test_index(admin_client):
-    with assert_queries_count(48):
+    with assert_queries_count(49):
         resp = admin_client.get('/', follow_redirects=True)
     check_content_in_response('DAGs', resp)