Repository: apache/airflow

Issue:
Running alembic migration downgrade on database causes error 
### Apache Airflow version

2.2.2 (latest released)

### Operating System

PRETTY_NAME="Debian GNU/Linux 10 (buster)" NAME="Debian GNU/Linux" VERSION_ID="10" VERSION="10 (buster)" VERSION_CODENAME=buster ID=debian HOME_URL="https://www.debian.org/" SUPPORT_URL="https://www.debian.org/support" BUG_REPORT_URL="https://bugs.debian.org/"

### Versions of Apache Airflow Providers

apache-airflow-providers-amazon==2.4.0
apache-airflow-providers-celery==2.1.0
apache-airflow-providers-cncf-kubernetes==2.1.0
apache-airflow-providers-datadog==2.0.1
apache-airflow-providers-docker==2.3.0
apache-airflow-providers-elasticsearch==2.1.0
apache-airflow-providers-ftp==2.0.1
apache-airflow-providers-google==6.1.0
apache-airflow-providers-grpc==2.0.1
apache-airflow-providers-hashicorp==2.1.1
apache-airflow-providers-http==2.0.1
apache-airflow-providers-imap==2.0.1
apache-airflow-providers-microsoft-azure==3.3.0
apache-airflow-providers-mysql==2.1.1
apache-airflow-providers-odbc==2.0.1
apache-airflow-providers-postgres==2.3.0
apache-airflow-providers-redis==2.0.1
apache-airflow-providers-sendgrid==2.0.1
apache-airflow-providers-sftp==2.2.0
apache-airflow-providers-slack==4.1.0
apache-airflow-providers-sqlite==2.0.1
apache-airflow-providers-ssh==2.3.0

### Deployment

Docker-Compose

### Deployment details

_No response_

### What happened

My current database alembic_version is set at e9304a3141f0 

While running the command `alembic -c alembic.ini downgrade a13f7613ad25` I received the following error:

INFO  [alembic.runtime.migration] Running downgrade e9304a3141f0 -> 83f031fd9f1c, make xcom pkey columns non-nullable
Traceback (most recent call last):
  File "/home/airflow/.local/lib/python3.7/site-packages/sqlalchemy/engine/base.py", line 1277, in _execute_context
    cursor, statement, parameters, context
  File "/home/airflow/.local/lib/python3.7/site-packages/sqlalchemy/engine/default.py", line 608, in do_execute
    cursor.execute(statement, parameters)
psycopg2.errors.InvalidTableDefinition: column "key" is in a primary key


The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/home/airflow/.local/bin/alembic", line 8, in <module>
    sys.exit(main())
  File "/home/airflow/.local/lib/python3.7/site-packages/alembic/config.py", line 588, in main
    CommandLine(prog=prog).main(argv=argv)
  File "/home/airflow/.local/lib/python3.7/site-packages/alembic/config.py", line 582, in main
    self.run_cmd(cfg, options)
  File "/home/airflow/.local/lib/python3.7/site-packages/alembic/config.py", line 562, in run_cmd
    **dict((k, getattr(options, k, None)) for k in kwarg)
  File "/home/airflow/.local/lib/python3.7/site-packages/alembic/command.py", line 366, in downgrade
    script.run_env()
  File "/home/airflow/.local/lib/python3.7/site-packages/alembic/script/base.py", line 563, in run_env
    util.load_python_file(self.dir, "env.py")
  File "/home/airflow/.local/lib/python3.7/site-packages/alembic/util/pyfiles.py", line 92, in load_python_file
    module = load_module_py(module_id, path)
  File "/home/airflow/.local/lib/python3.7/site-packages/alembic/util/pyfiles.py", line 108, in load_module_py
    spec.loader.exec_module(module)  # type: ignore
  File "<frozen importlib._bootstrap_external>", line 728, in exec_module
  File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
  File "migrations/env.py", line 107, in <module>
    run_migrations_online()
  File "migrations/env.py", line 101, in run_migrations_online
    context.run_migrations()
  File "<string>", line 8, in run_migrations
  File "/home/airflow/.local/lib/python3.7/site-packages/alembic/runtime/environment.py", line 851, in run_migrations
    self.get_context().run_migrations(**kw)
  File "/home/airflow/.local/lib/python3.7/site-packages/alembic/runtime/migration.py", line 620, in run_migrations
    step.migration_fn(**kw)
  File "/home/airflow/.local/lib/python3.7/site-packages/airflow/migrations/versions/e9304a3141f0_make_xcom_pkey_columns_non_nullable.py", line 76, in downgrade
    bop.alter_column("execution_date", type_=_get_timestamp(conn), nullable=True)
  File "/usr/local/lib/python3.7/contextlib.py", line 119, in __exit__
    next(self.gen)
  File "/home/airflow/.local/lib/python3.7/site-packages/alembic/operations/base.py", line 374, in batch_alter_table
    impl.flush()
  File "/home/airflow/.local/lib/python3.7/site-packages/alembic/operations/batch.py", line 107, in flush
    fn(*arg, **kw)
  File "/home/airflow/.local/lib/python3.7/site-packages/alembic/ddl/postgresql.py", line 185, in alter_column
    **kw
  File "/home/airflow/.local/lib/python3.7/site-packages/alembic/ddl/impl.py", line 240, in alter_column
    existing_comment=existing_comment,
  File "/home/airflow/.local/lib/python3.7/site-packages/alembic/ddl/impl.py", line 197, in _exec
    return conn.execute(construct, multiparams)
  File "/home/airflow/.local/lib/python3.7/site-packages/sqlalchemy/engine/base.py", line 1011, in execute
    return meth(self, multiparams, params)
  File "/home/airflow/.local/lib/python3.7/site-packages/sqlalchemy/sql/ddl.py", line 72, in _execute_on_connection
    return connection._execute_ddl(self, multiparams, params)
  File "/home/airflow/.local/lib/python3.7/site-packages/sqlalchemy/engine/base.py", line 1073, in _execute_ddl
    compiled,
  File "/home/airflow/.local/lib/python3.7/site-packages/sqlalchemy/engine/base.py", line 1317, in _execute_context
    e, statement, parameters, cursor, context
  File "/home/airflow/.local/lib/python3.7/site-packages/sqlalchemy/engine/base.py", line 1511, in _handle_dbapi_exception
    sqlalchemy_exception, with_traceback=exc_info[2], from_=e
  File "/home/airflow/.local/lib/python3.7/site-packages/sqlalchemy/util/compat.py", line 182, in raise_
    raise exception
  File "/home/airflow/.local/lib/python3.7/site-packages/sqlalchemy/engine/base.py", line 1277, in _execute_context
    cursor, statement, parameters, context
  File "/home/airflow/.local/lib/python3.7/site-packages/sqlalchemy/engine/default.py", line 608, in do_execute
    cursor.execute(statement, parameters)
sqlalchemy.exc.ProgrammingError: (psycopg2.errors.InvalidTableDefinition) column "key" is in a primary key

[SQL: ALTER TABLE xcom ALTER COLUMN key DROP NOT NULL]


It seems the DOWN migration step for revision `e9304a3141f0 - Make XCom primary key columns non-nullable` is not backwards compatible as it is attempting to make a Primary Key nullable, which is not possible.

I suspect the revision of `bbf4a7ad0465 - Remove id column from xcom` has something to do with this - id perhaps being the old primary key for the table.

Revision list: https://airflow.apache.org/docs/apache-airflow/stable/migrations-ref.html

### What you expected to happen

I expected the alembic version to downgrade to the appropriate version

### How to reproduce

On airflow image 2.2.2 attempt to run `alembic -c alembic.ini downgrade {any version lower than revision id e9304a3141f0}`



### Anything else

_No response_

### Are you willing to submit PR?

- [ ] Yes I am willing to submit a PR!

### Code of Conduct

- [X] I agree to follow this project's [Code of Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md)



Discussion:
Thanks for opening your first issue here! Be sure to follow the issue template!


Resolving patch:
diff --git a/airflow/migrations/versions/54bebd308c5f_add_trigger_table_and_task_info.py b/airflow/migrations/versions/54bebd308c5f_add_trigger_table_and_task_info.py
--- a/airflow/migrations/versions/54bebd308c5f_add_trigger_table_and_task_info.py
+++ b/airflow/migrations/versions/54bebd308c5f_add_trigger_table_and_task_info.py
@@ -60,8 +60,8 @@ def upgrade():
 def downgrade():
     """Unapply Add trigger table and task info"""
     with op.batch_alter_table('task_instance', schema=None) as batch_op:
+        batch_op.drop_constraint('task_instance_trigger_id_fkey', type_='foreignkey')
         batch_op.drop_index('ti_trigger_id')
-        batch_op.drop_constraint('task_instance_trigger_id_fkey')
         batch_op.drop_column('trigger_id')
         batch_op.drop_column('trigger_timeout')
         batch_op.drop_column('next_method')
diff --git a/airflow/migrations/versions/7b2661a43ba3_taskinstance_keyed_to_dagrun.py b/airflow/migrations/versions/7b2661a43ba3_taskinstance_keyed_to_dagrun.py
--- a/airflow/migrations/versions/7b2661a43ba3_taskinstance_keyed_to_dagrun.py
+++ b/airflow/migrations/versions/7b2661a43ba3_taskinstance_keyed_to_dagrun.py
@@ -329,8 +329,8 @@ def downgrade():
         batch_op.alter_column('execution_date', existing_type=dt_type, existing_nullable=True, nullable=False)
 
         # Can't drop PK index while there is a FK referencing it
-        batch_op.drop_constraint('task_reschedule_ti_fkey')
-        batch_op.drop_constraint('task_reschedule_dr_fkey')
+        batch_op.drop_constraint('task_reschedule_ti_fkey', type_='foreignkey')
+        batch_op.drop_constraint('task_reschedule_dr_fkey', type_='foreignkey')
         batch_op.drop_index('idx_task_reschedule_dag_task_run')
 
     with op.batch_alter_table('task_instance', schema=None) as batch_op:
diff --git a/airflow/migrations/versions/e9304a3141f0_make_xcom_pkey_columns_non_nullable.py b/airflow/migrations/versions/e9304a3141f0_make_xcom_pkey_columns_non_nullable.py
--- a/airflow/migrations/versions/e9304a3141f0_make_xcom_pkey_columns_non_nullable.py
+++ b/airflow/migrations/versions/e9304a3141f0_make_xcom_pkey_columns_non_nullable.py
@@ -50,5 +50,12 @@ def downgrade():
     with op.batch_alter_table('xcom') as bop:
         if conn.dialect.name == 'mssql':
             bop.drop_constraint('pk_xcom', 'primary')
-        bop.alter_column("key", type_=StringID(length=512), nullable=True)
-        bop.alter_column("execution_date", type_=TIMESTAMP, nullable=True)
+
+        # regardless of what the model defined, the `key` and `execution_date`
+        # columns were always non-nullable for sqlite and postgres, so leave them alone
+
+        if conn.dialect.name in ['mysql', 'mssql']:
+            bop.alter_column("key", type_=StringID(length=512), nullable=True)
+        if conn.dialect.name == 'mssql':
+            # execution_date wasn't nullable in the other databases
+            bop.alter_column("execution_date", type_=TIMESTAMP, nullable=True)


Test patch: