Issue 1:
Apache Airflow version: 1.10.x, 2.0

Kubernetes version (if you are using kubernetes) (use kubectl version): 1.18.8

Environment: Azure - AKS - ContainerD CRI-O

Cloud provider or hardware configuration: Azure AKS

OS (e.g. from /etc/os-release): Ubuntu 18.04.5 LTS

Kernel (e.g. uname -a): 5.4.0-1031-azure

Install tools: Docker runtime - 19.3.12

Others: ContainerD CRI-O

What happened: Worker pod stuck in 'ContainerCreating' state - no error propagated to Airflow K8s executor for a DAG containing a longer workflow-name and task-name get specified.

What you expected to happen: Worker-pod to success run OR task failure notification received due to K8s error

How to reproduce it: Run any dag/task with longer names - combined number of alphanumeric characters in dag/task name should be more than 214

Anything else we need to know: The error seems to occur due to pod sandbox creation failure. Each K8s pod gets a specific sandbox (directory). If the pod-id generated by Airflow Kubernetes executor is more than 214 characters, K8s also add suffix/prefix to generate a specific sandbox directory (prefix the K8s namespace and suffix a 36 character UUID using '_' as delimiter). This results in the sandbox directory name to cross '255' characters which is the maximum character (bytes) limit in most of the EXT FS based filesystem.

Issue 2:
Apache Airflow version: master

Kubernetes version: v1.19.3

Environment: Ubuntu 18.04 Server

Using task IDs longer than 253 characters seems to cause problems in multiple areas. Considering we can now use nested task groups (which are basically prefixes for the task ID afaik) the final task ID can be pretty long.

pod_generator.py does not use make_safe_label_value and writes the task ID directly as label. For k8s labels the limit is only 63 characters. This causes k8s not being able to schedule the task pod.

airflow/airflow/kubernetes/pod_generator.py

Line 386
in
4ad1acc

'task_id': task_id,

kubernetes_executor.py uses create_pod_id to create a preliminary pod_id (not yet launched to k8s) and later uses make_unique_pod_id to trim the pod_id to 253 characters - though in my test I hit the max. filename length limit of Linux (or rather ext4 filesystem), which is 255 characters. So it seems some code is appending additional characters?

airflow/airflow/executors/kubernetes_executor.py

Line 294
in
4ad1acc

pod_id=create_pod_id(dag_id, task_id),

Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal Scheduled 3m3s default-scheduler Successfully assigned airflow/processairflowtesttgsurvey00000tgincrementaladjustmentsurvey00000f608c63d9btgmsac10survey000001c1a34cf10tgbundleadjuster786931747dtgcolor0521cd0b3f7tbundleadjusterfinalize5b57782bb2-aad4395c77ba400e9ee24c04d2cdbb72 to xxx.lan
Warning FailedCreatePodSandBox 7s (x14 over 3m) kubelet Failed to create pod sandbox: mkdir /var/log/pods/airflow_processairflowtesttgsurvey00000tgincrementaladjustmentsurvey00000f608c63d9btgmsac10survey000001c1a34cf10tgbundleadjuster786931747dtgcolor0521cd0b3f7tbundleadjusterfinalize5b57782bb2-aad4395c77ba400e9ee24c04d2cdbb72_8138b684-e7ce-41bc-8b31-4bbf9774e56a: file name too long

As far as I understood kubernetes_executor.py adopt_launched_task uses the pod labels to "reconstruct" the pod_id - considering the pod was created by additionally applying make_unique_pod_id and the k8s label limit of 63 characters we will never reconstruct the original pod_id?

airflow/airflow/executors/kubernetes_executor.py

Line 628
in
4ad1acc

dag_id = pod.metadata.labels['dag_id']