Repository: apache/airflow
Pull request: Support Pydantic 2
Fix https://github.com/apache/airflow/issues/32311 eventually. For CI only at the moment.

Problem statement:
Pydantic 2.0.0 support for Airflow
### Body

Currently Pydantic 2.0.0 released on 30th of June 2023 breaks Airflow CI - building providers and running Kubernetes tests.

Therefore we limit Pydantic to < 2.0.0 in setup.cfg for now.

This should be fixed, especially that 2.0.0 brings a number of speed improvements.


### Committer

- [X] I acknowledge that I am a maintainer/committer of the Apache Airflow project.

Hints from issue discussion:
Do we have a way to identify incompatibilities so they can be fixed? Maybe a CI job for example?
It's enough to remove the limit and set `full-tests-needed` for the PR and fix the problems. No need for special job.
Feel free to pick it up if you want :) 
Ah right, will do that
It seems that some of our dependencies have added the `<2` pin as well and removing it from our direct dependencies is no longer sufficient to install Pydantic 2 in CI. Adding `>=2` results in a resolution conflict. Will need to find another way.
> It seems that some of our dependencies have added the `<2` pin as well and removing it from our direct dependencies is no longer sufficient to install Pydantic 2 in CI. Adding `>=2` results in a resolution conflict. Will need to find another way.

Likely waiting for them to support Pydantic 2 :)
Also it is likely some other incompatibilities (aka "teething problems" with Pydantic 2) might simply get fixed - Pydantic 2.0.1 has been released yesterday with ~40 bugs fixed. so waiting is probably a very good strategy in general.

Andthat's a good news actually that other deps limited it for now, because it seems that likely even no-constraints installation will be fine for 2.6.0 - 2.6.2. 
Hey the CI is actually passing with Pydantic 2.0.1. Maybe this is it and there’s nothing to fix at our part.
Nope. Not passing. There are K8S and Docker compose failing (and errors in it suggest Pydantic being the reason).
I might take a look at that later - but I think there are a few more changes to be added.

Commit message: Use base datetime class for Pydantic

Patch to review:
diff --git a/Dockerfile b/Dockerfile
index 35afca6672a77..f89293c5a6f52 100644
--- a/Dockerfile
+++ b/Dockerfile
@@ -385,7 +385,8 @@ function install_airflow_dependencies_from_branch_tip() {
     pip install --root-user-action ignore \
       ${ADDITIONAL_PIP_INSTALL_FLAGS} \
       "https://github.com/${AIRFLOW_REPO}/archive/${AIRFLOW_BRANCH}.tar.gz#egg=apache-airflow[${AIRFLOW_EXTRAS}]" \
-      --constraint "${AIRFLOW_CONSTRAINTS_LOCATION}" || true
+      --constraint "${AIRFLOW_CONSTRAINTS_LOCATION}" \
+      || pip install 'pydantic>=2' || true
     common::install_pip_version
     pip freeze | grep apache-airflow-providers | xargs pip uninstall --yes 2>/dev/null || true
     set +x
diff --git a/Dockerfile.ci b/Dockerfile.ci
index 6df47d6a8a494..f8cb3da0fc55c 100644
--- a/Dockerfile.ci
+++ b/Dockerfile.ci
@@ -345,7 +345,8 @@ function install_airflow_dependencies_from_branch_tip() {
     pip install --root-user-action ignore \
       ${ADDITIONAL_PIP_INSTALL_FLAGS} \
       "https://github.com/${AIRFLOW_REPO}/archive/${AIRFLOW_BRANCH}.tar.gz#egg=apache-airflow[${AIRFLOW_EXTRAS}]" \
-      --constraint "${AIRFLOW_CONSTRAINTS_LOCATION}" || true
+      --constraint "${AIRFLOW_CONSTRAINTS_LOCATION}" \
+      || pip install 'pydantic>=2' || true
     common::install_pip_version
     pip freeze | grep apache-airflow-providers | xargs pip uninstall --yes 2>/dev/null || true
     set +x
diff --git a/airflow/serialization/pydantic/dag_run.py b/airflow/serialization/pydantic/dag_run.py
index a5ae6b4a22b52..1e1528eb35641 100644
--- a/airflow/serialization/pydantic/dag_run.py
+++ b/airflow/serialization/pydantic/dag_run.py
@@ -18,7 +18,6 @@
 from datetime import datetime
 from typing import List, Optional
 
-from pendulum import DateTime
 from pydantic import BaseModel as BaseModelPydantic
 
 from airflow.serialization.pydantic.dataset import DatasetEventPydantic
@@ -30,7 +29,7 @@ class DagRunPydantic(BaseModelPydantic):
     id: int
     dag_id: str
     queued_at: Optional[datetime]
-    execution_date: DateTime
+    execution_date: datetime
     start_date: Optional[datetime]
     end_date: Optional[datetime]
     state: str
diff --git a/scripts/docker/install_airflow_dependencies_from_branch_tip.sh b/scripts/docker/install_airflow_dependencies_from_branch_tip.sh
index 2b58d03950e5b..f2c48c7f5d40f 100644
--- a/scripts/docker/install_airflow_dependencies_from_branch_tip.sh
+++ b/scripts/docker/install_airflow_dependencies_from_branch_tip.sh
@@ -51,7 +51,8 @@ function install_airflow_dependencies_from_branch_tip() {
     pip install --root-user-action ignore \
       ${ADDITIONAL_PIP_INSTALL_FLAGS} \
       "https://github.com/${AIRFLOW_REPO}/archive/${AIRFLOW_BRANCH}.tar.gz#egg=apache-airflow[${AIRFLOW_EXTRAS}]" \
-      --constraint "${AIRFLOW_CONSTRAINTS_LOCATION}" || true
+      --constraint "${AIRFLOW_CONSTRAINTS_LOCATION}" \
+      || pip install 'pydantic>=2' || true
     common::install_pip_version
     pip freeze | grep apache-airflow-providers | xargs pip uninstall --yes 2>/dev/null || true
     set +x
diff --git a/setup.cfg b/setup.cfg
index 91d3b86331de0..d5d91d3d8ff6d 100644
--- a/setup.cfg
+++ b/setup.cfg
@@ -126,9 +126,7 @@ install_requires =
     pendulum>=2.0
     pluggy>=1.0
     psutil>=4.2.0
-    # Limit Pydantic to <2.0.0 because it breaks Kubernetes tests and building providers
-    # This limit should be removed after https://github.com/apache/airflow/issues/32311 is fixed
-    pydantic>=1.10.0,<2.0.0
+    pydantic>=1.10.0
     pygments>=2.0.1
     pyjwt>=2.0.0
     python-daemon>=3.0.0


Code review comments:
diff_hunk: @@ -51,7 +51,8 @@ function install_airflow_dependencies_from_branch_tip() {
     pip install --root-user-action ignore \
       ${ADDITIONAL_PIP_INSTALL_FLAGS} \
       "https://github.com/${AIRFLOW_REPO}/archive/${AIRFLOW_BRANCH}.tar.gz#egg=apache-airflow[${AIRFLOW_EXTRAS}]" \
-      --constraint "${AIRFLOW_CONSTRAINTS_LOCATION}" || true
+      --constraint "${AIRFLOW_CONSTRAINTS_LOCATION}" \
+      || pip install 'pydantic>=2' || true
path: scripts/docker/install_airflow_dependencies_from_branch_tip.sh
text: @author:
This hack (and those in Dockerfile and Dockerfile.ci, those are auto-copied by pre-commit) will be removed before merge.

diff_hunk: @@ -385,7 +385,8 @@ function install_airflow_dependencies_from_branch_tip() {
     pip install --root-user-action ignore \
       ${ADDITIONAL_PIP_INSTALL_FLAGS} \
       "https://github.com/${AIRFLOW_REPO}/archive/${AIRFLOW_BRANCH}.tar.gz#egg=apache-airflow[${AIRFLOW_EXTRAS}]" \
-      --constraint "${AIRFLOW_CONSTRAINTS_LOCATION}" || true
+      --constraint "${AIRFLOW_CONSTRAINTS_LOCATION}" \
+      || pip install 'pydantic>=2' || true
path: Dockerfile
text: @user1:
Looks great. Yeah. Lets' remove it then and see if `pip` can resolve them on their own. If there are problems with "eager upgrade" and pip not finding the right resolution automatically, the right way to handle it will be to add "pydantic>=2" to `EAGER_UPGRADE_ADDITIONAL_REQUIREMENTS` (currently empty so it should be commented that likely we should remove it in the future when the resolution is improved) in both Dockerfile and Dockerfile.ci. This is the way how we "help" `pip` by additionally limiting the `eager upgrade`.

@author:
It would resolve it to 1.18 because (a dependency of) Moto pins it to `pydantic~=1.18` (I tried it locally before settling on this hack)