Repository: apache/airflow

Issue:
The command 'airflow dags test' is deadlocking tasks using AIP-42 features with the PythonVirtualenvOperator
### Apache Airflow version

main (development)

### What happened

When using the command airflow dags test tasks are put in the deadlock state.

### What you think should happen instead

I think the airflow dags test shouldn't deadlock tasks.

### How to reproduce
```
from airflow.models import DAG
from airflow.operators.python import PythonOperator, PythonVirtualenvOperator
from airflow.utils.log.log_reader import TaskLogReader
from airflow.utils.dates import days_ago


from pkgutil import iter_modules
from datetime import datetime, timedelta



the_imports = ["flask", "wheel", "click", "cryptography", "packaging"]

def dynamic_imports(imports: list):
    from random import choices
    return [choices(imports, k=3) for _ in range(7)]

def dynamic_importer(*args):
    import importlib
    for mod in args[0]:
        print(mod)
        importlib.import_module(mod)

with DAG(
    dag_id="sys_site_packages_true",
    schedule_interval=timedelta(days=365),
    start_date=datetime(2001, 1, 1),
    doc_md=docs,
    tags=["core", "extended_tags", "venv_op"],
) as  dag:

    pv0 = PythonVirtualenvOperator.partial(
        task_id="dont_install_packages_already_in_outer_environment",
        python_callable=dynamic_importer,
        python_version=3.8,
        system_site_packages=True,
        requirements=the_imports,
    ).expand(op_args=dynamic_imports(the_imports))

[2022-03-30 17:13:36,447] {taskinstance.py:850} DEBUG - Setting task state for <TaskInstance: sys_site_packages_true.dont_install_packages_already_in_outer_environment backfill__2022-03-30T17:13:35+00:00 [None]> to scheduled
[2022-03-30 17:13:36,455] {backfill_job.py:405} DEBUG - *** Clearing out not_ready list ***
[2022-03-30 17:13:36,458] {taskinstance.py:759} DEBUG - Refreshing TaskInstance <TaskInstance: sys_site_packages_true.dont_install_packages_already_in_outer_environment backfill__2022-03-30T17:13:35+00:00 [scheduled]> from DB
[2022-03-30 17:13:36,461] {backfill_job.py:418} DEBUG - Task instance to run <TaskInstance: sys_site_packages_true.dont_install_packages_already_in_outer_environment backfill__2022-03-30T17:13:35+00:00 [scheduled]> state scheduled
[2022-03-30 17:13:36,461] {taskinstance.py:1061} DEBUG - <TaskInstance: sys_site_packages_true.dont_install_packages_already_in_outer_environment backfill__2022-03-30T17:13:35+00:00 [scheduled]> dependency 'Previous Dagrun State' PASSED: True, The context specified that the state of past DAGs could be ignored.
[2022-03-30 17:13:36,461] {taskinstance.py:1061} DEBUG - <TaskInstance: sys_site_packages_true.dont_install_packages_already_in_outer_environment backfill__2022-03-30T17:13:35+00:00 [scheduled]> dependency 'Trigger Rule' PASSED: True, The task instance did not have any upstream tasks.
[2022-03-30 17:13:36,462] {taskinstance.py:1061} DEBUG - <TaskInstance: sys_site_packages_true.dont_install_packages_already_in_outer_environment backfill__2022-03-30T17:13:35+00:00 [scheduled]> dependency 'Task has been mapped' PASSED: False, The task has yet to be mapped!
[2022-03-30 17:13:36,462] {taskinstance.py:1041} DEBUG - Dependencies not met for <TaskInstance: sys_site_packages_true.dont_install_packages_already_in_outer_environment backfill__2022-03-30T17:13:35+00:00 [scheduled]>, dependency 'Task has been mapped' FAILED: The task has yet to be mapped!
[2022-03-30 17:13:36,462] {taskinstance.py:1061} DEBUG - <TaskInstance: sys_site_packages_true.dont_install_packages_already_in_outer_environment backfill__2022-03-30T17:13:35+00:00 [scheduled]> dependency 'Task Instance Not Running' PASSED: True, Task is not in running state.
[2022-03-30 17:13:36,462] {taskinstance.py:1061} DEBUG - <TaskInstance: sys_site_packages_true.dont_install_packages_already_in_outer_environment backfill__2022-03-30T17:13:35+00:00 [scheduled]> dependency 'Not In Retry Period' PASSED: True, The task instance was not marked for retrying.
[2022-03-30 17:13:36,462] {taskinstance.py:1061} DEBUG - <TaskInstance: sys_site_packages_true.dont_install_packages_already_in_outer_environment backfill__2022-03-30T17:13:35+00:00 [scheduled]> dependency 'Task Instance State' PASSED: True, Task state scheduled was valid.
[2022-03-30 17:13:36,463] {backfill_job.py:536} DEBUG - Adding <TaskInstance: sys_site_packages_true.dont_install_packages_already_in_outer_environment backfill__2022-03-30T17:13:35+00:00 [scheduled]> to not_ready
[2022-03-30 17:13:41,428] {base_job.py:226} DEBUG - [heartbeat]
[2022-03-30 17:13:41,428] {base_executor.py:156} DEBUG - 0 running task instances
[2022-03-30 17:13:41,429] {base_executor.py:157} DEBUG - 0 in queue
[2022-03-30 17:13:41,429] {base_executor.py:158} DEBUG - 32 open slots
[2022-03-30 17:13:41,429] {base_executor.py:167} DEBUG - Calling the <class 'airflow.executors.debug_executor.DebugExecutor'> sync method
[2022-03-30 17:13:41,429] {backfill_job.py:596} WARNING - Deadlock discovered for ti_status.to_run=dict_values([<TaskInstance: sys_site_packages_true.dont_install_packages_already_in_outer_environment backfill__2022-03-30T17:13:35+00:00 [scheduled]>])
[2022-03-30 17:13:41,433] {dagrun.py:628} DEBUG - number of tis tasks for <DagRun sys_site_packages_true @ 2022-03-30T17:13:35+00:00: backfill__2022-03-30T17:13:35+00:00, externally triggered: False>: 1 task(s)
[2022-03-30 17:13:41,433] {dagrun.py:644} DEBUG - number of scheduleable tasks for <DagRun sys_site_packages_true @ 2022-03-30T17:13:35+00:00: backfill__2022-03-30T17:13:35+00:00, externally triggered: False>: 0 task(s)
[2022-03-30 17:13:41,434] {taskinstance.py:1061} DEBUG - <TaskInstance: sys_site_packages_true.dont_install_packages_already_in_outer_environment backfill__2022-03-30T17:13:35+00:00 [scheduled]> dependency 'Not In Retry Period' PASSED: True, The context specified that being in a retry period was permitted.
[2022-03-30 17:13:41,434] {taskinstance.py:1061} DEBUG - <TaskInstance: sys_site_packages_true.dont_install_packages_already_in_outer_environment backfill__2022-03-30T17:13:35+00:00 [scheduled]> dependency 'Previous Dagrun State' PASSED: True, The task did not have depends_on_past set.
[2022-03-30 17:13:41,434] {taskinstance.py:1061} DEBUG - <TaskInstance: sys_site_packages_true.dont_install_packages_already_in_outer_environment backfill__2022-03-30T17:13:35+00:00 [scheduled]> dependency 'Trigger Rule' PASSED: True, The task instance did not have any upstream tasks.
[2022-03-30 17:13:41,434] {taskinstance.py:1061} DEBUG - <TaskInstance: sys_site_packages_true.dont_install_packages_already_in_outer_environment backfill__2022-03-30T17:13:35+00:00 [scheduled]> dependency 'Task has been mapped' PASSED: False, The task has yet to be mapped!
[2022-03-30 17:13:41,434] {taskinstance.py:1041} DEBUG - Dependencies not met for <TaskInstance: sys_site_packages_true.dont_install_packages_already_in_outer_environment backfill__2022-03-30T17:13:35+00:00 [scheduled]>, dependency 'Task has been mapped' FAILED: The task has yet to be mapped!
[2022-03-30 17:13:41,434] {dagrun.py:570} ERROR - Deadlock; marking run <DagRun sys_site_packages_true @ 2022-03-30T17:13:35+00:00: backfill__2022-03-30T17:13:35+00:00, externally triggered: False> failed
[2022-03-30 17:13:41,434] {dagrun.py:594} INFO - DagRun Finished: dag_id=sys_site_packages_true, execution_date=2022-03-30T17:13:35+00:00, run_id=backfill__2022-03-30T17:13:35+00:00, run_start_date=2022-03-30 23:13:36.434073+00:00, run_end_date=2022-03-30 23:13:41.434824+00:00, run_duration=5.000751, state=failed, external_trigger=False, run_type=backfill, data_interval_start=2022-03-30T17:13:35+00:00, data_interval_end=2023-03-30T17:13:35+00:00, dag_hash=None
[2022-03-30 17:13:41,435] {backfill_job.py:362} INFO - [backfill progress] | finished run 1 of 1 | tasks waiting: 0 | succeeded: 0 | running: 0 | failed: 0 | skipped: 0 | deadlocked: 1 | not ready: 1
[2022-03-30 17:13:41,435] {backfill_job.py:376} DEBUG - Finished dag run loop iteration. Remaining tasks dict_values([])
[2022-03-30 17:13:41,438] {taskinstance.py:1061} DEBUG - <TaskInstance: sys_site_packages_true.dont_install_packages_already_in_outer_environment backfill__2022-03-30T17:13:35+00:00 [scheduled]> dependency 'Not In Retry Period' PASSED: True, The task instance was not marked for retrying.
[2022-03-30 17:13:41,438] {taskinstance.py:1061} DEBUG - <TaskInstance: sys_site_packages_true.dont_install_packages_already_in_outer_environment backfill__2022-03-30T17:13:35+00:00 [scheduled]> dependency 'Previous Dagrun State' PASSED: True, The task did not have depends_on_past set.
[2022-03-30 17:13:41,438] {taskinstance.py:1061} DEBUG - <TaskInstance: sys_site_packages_true.dont_install_packages_already_in_outer_environment backfill__2022-03-30T17:13:35+00:00 [scheduled]> dependency 'Trigger Rule' PASSED: True, The task instance did not have any upstream tasks.
[2022-03-30 17:13:41,438] {taskinstance.py:1061} DEBUG - <TaskInstance: sys_site_packages_true.dont_install_packages_already_in_outer_environment backfill__2022-03-30T17:13:35+00:00 [scheduled]> dependency 'Task has been mapped' PASSED: False, The task has yet to be mapped!
[2022-03-30 17:13:41,438] {taskinstance.py:1041} DEBUG - Dependencies not met for <TaskInstance: sys_site_packages_true.dont_install_packages_already_in_outer_environment backfill__2022-03-30T17:13:35+00:00 [scheduled]>, dependency 'Task has been mapped' FAILED: The task has yet to be mapped!
[2022-03-30 17:13:41,440] {taskinstance.py:1061} DEBUG - <TaskInstance: sys_site_packages_true.dont_install_packages_already_in_outer_environment backfill__2022-03-30T17:13:35+00:00 [scheduled]> dependency 'Not In Retry Period' PASSED: True, The task instance was not marked for retrying.
[2022-03-30 17:13:41,440] {taskinstance.py:1061} DEBUG - <TaskInstance: sys_site_packages_true.dont_install_packages_already_in_outer_environment backfill__2022-03-30T17:13:35+00:00 [scheduled]> dependency 'Previous Dagrun State' PASSED: True, The context specified that the state of past DAGs could be ignored.
[2022-03-30 17:13:41,440] {taskinstance.py:1061} DEBUG - <TaskInstance: sys_site_packages_true.dont_install_packages_already_in_outer_environment backfill__2022-03-30T17:13:35+00:00 [scheduled]> dependency 'Trigger Rule' PASSED: True, The task instance did not have any upstream tasks.
[2022-03-30 17:13:41,440] {taskinstance.py:1061} DEBUG - <TaskInstance: sys_site_packages_true.dont_install_packages_already_in_outer_environment backfill__2022-03-30T17:13:35+00:00 [scheduled]> dependency 'Task has been mapped' PASSED: False, The task has yet to be mapped!
[2022-03-30 17:13:41,440] {taskinstance.py:1041} DEBUG - Dependencies not met for <TaskInstance: sys_site_packages_true.dont_install_packages_already_in_outer_environment backfill__2022-03-30T17:13:35+00:00 [scheduled]>, dependency 'Task has been mapped' FAILED: The task has yet to be mapped!
BackfillJob is deadlocked.
These tasks have succeeded:
DAG ID    Task ID    Run ID    Try number
--------  ---------  --------  ------------

These tasks are running:
DAG ID    Task ID    Run ID    Try number
--------  ---------  --------  ------------

These tasks have failed:
DAG ID    Task ID    Run ID    Try number
--------  ---------  --------  ------------

These tasks are skipped:
DAG ID    Task ID    Run ID    Try number
--------  ---------  --------  ------------

These tasks are deadlocked:
DAG ID                  Task ID                                             Run ID                                 Try number
----------------------  --------------------------------------------------  -----------------------------------  ------------
sys_site_packages_true  dont_install_packages_already_in_outer_environment  backfill__2022-03-30T17:13:35+00:00             1
[2022-03-30 17:13:41,446] {cli_action_loggers.py:84} DEBUG - Calling callbacks: []
[2022-03-30 17:13:41,447] {settings.py:383} DEBUG - Disposing DB connection pool (PID 816091)


```
### Operating System

Ubuntu Linux

### Versions of Apache Airflow Providers

_No response_

### Deployment

Docker-Compose

### Deployment details

tested with airflow breeze and a local virtualenv with airflow 2.3.0 installed.

### Anything else

_No response_

### Are you willing to submit PR?

- [ ] Yes I am willing to submit a PR!

### Code of Conduct

- [X] I agree to follow this project's [Code of Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md)



Discussion:
Thanks for opening your first issue here! Be sure to follow the issue template!

Same thing with DockerOperator
```python3
from airflow.decorators import dag
from airflow.operators.docker_operator import DockerOperator
from datetime import datetime, timedelta

@dag(schedule_interval=None, start_date=datetime(1970, 1, 1))
def hello_docker():

    DockerOperator.partial(task_id="say_hi", image="bash:latest").expand(
        command=[
            [
                "-c",
                "echo Hello World",
            ],
            [
                "-c",
                "echo Goodbye World",
            ],
        ]
    )
```
Looks like this is the same problem to the literal not being expanded?
This _almost_ passes now. The Tasks run, but right at the end of the `test` command we have this failure

```
[2022-04-11 10:49:28,894]  2990124 MainProcess {{airflow.models.taskinstance.TaskInstance taskinstance.py:1364}} INFO - Marking task as SUCCESS. dag_id=sys_site_packages_true, task_id=dont_install_packages_already_in_outer_environment, map_index=6, execution_date=20200410T020000, start_date=20220411T094856, end_date=20220411T094928
Traceback (most recent call last):
  File "/home/ash/.virtualenvs/airflow/bin/airflow", line 33, in <module>
    sys.exit(load_entry_point('apache-airflow', 'console_scripts', 'airflow')())
  File "/home/ash/code/airflow/airflow/airflow/__main__.py", line 38, in main
    args.func(args)
  File "/home/ash/code/airflow/airflow/airflow/cli/cli_parser.py", line 51, in command
    return func(*args, **kwargs)
  File "/home/ash/code/airflow/airflow/airflow/utils/session.py", line 71, in wrapper
    return func(*args, session=session, **kwargs)
  File "/home/ash/code/airflow/airflow/airflow/utils/cli.py", line 99, in wrapper
    return f(*args, **kwargs)
  File "/home/ash/code/airflow/airflow/airflow/cli/commands/dag_command.py", line 446, in dag_test
    dag.run(
  File "/home/ash/code/airflow/airflow/airflow/models/dag.py", line 2226, in run
    job.run()
  File "/home/ash/code/airflow/airflow/airflow/jobs/base_job.py", line 244, in run
    self._execute()
  File "/home/ash/code/airflow/airflow/airflow/utils/session.py", line 71, in wrapper
    return func(*args, session=session, **kwargs)
  File "/home/ash/code/airflow/airflow/airflow/jobs/backfill_job.py", line 842, in _execute
    self._execute_dagruns(
  File "/home/ash/code/airflow/airflow/airflow/utils/session.py", line 68, in wrapper
    return func(*args, **kwargs)
  File "/home/ash/code/airflow/airflow/airflow/jobs/backfill_job.py", line 732, in _execute_dagruns
    processed_dag_run_dates = self._process_backfill_task_instances(
  File "/home/ash/code/airflow/airflow/airflow/utils/session.py", line 68, in wrapper
    return func(*args, **kwargs)
  File "/home/ash/code/airflow/airflow/airflow/jobs/backfill_job.py", line 612, in _process_backfill_task_instances
    for node, run_id, mapped_tis in self._manage_executor_state(ti_status.running, session):
  File "/home/ash/code/airflow/airflow/airflow/jobs/backfill_job.py", line 269, in _manage_executor_state
    for node in ti.task.mapped_dependants():
  File "/home/ash/code/airflow/airflow/airflow/models/taskmixin.py", line 317, in mapped_dependants
    raise RuntimeError("Cannot check for mapped_dependants when not attached to a DAG")
RuntimeError: Cannot check for mapped_dependants when not attached to a DAG
```
Ah, I think I know what’s going on. We made changes in the scheduler, but didn’t update BackFillJob to match it.

I should be able to fix it pretty quickly, but what would be an appropriate test for this?

Resolving patch:
diff --git a/airflow/jobs/backfill_job.py b/airflow/jobs/backfill_job.py
--- a/airflow/jobs/backfill_job.py
+++ b/airflow/jobs/backfill_job.py
@@ -102,7 +102,7 @@ class _DagRunTaskStatus:
 
     def __init__(
         self,
-        dag,
+        dag: DAG,
         start_date=None,
         end_date=None,
         mark_success=False,
@@ -238,8 +238,6 @@ def _manage_executor_state(
         :param running: dict of key, task to verify
         :return: An iterable of expanded TaskInstance per MappedTask
         """
-        from airflow.models.mappedoperator import MappedOperator
-
         executor = self.executor
 
         # TODO: query all instead of refresh from db
@@ -266,8 +264,9 @@ def _manage_executor_state(
                 ti.handle_failure_with_callback(error=msg)
                 continue
             if ti.state not in self.STATES_COUNT_AS_RUNNING:
-                for node in ti.task.mapped_dependants():
-                    assert isinstance(node, MappedOperator)
+                # Don't use ti.task; if this task is mapped, that attribute
+                # would hold the unmapped task. We need to original task here.
+                for node in self.dag.get_task(ti.task_id, include_subdags=True).mapped_dependants():
                     yield node, ti.run_id, node.expand_mapped_task(ti.run_id, session=session)
 
     @provide_session
@@ -702,7 +701,7 @@ def tabulate_ti_keys_set(ti_keys: Iterable[TaskInstanceKey]) -> str:
 
         return err
 
-    def _get_dag_with_subdags(self):
+    def _get_dag_with_subdags(self) -> List[DAG]:
         return [self.dag] + self.dag.subdags
 
     @provide_session
diff --git a/airflow/models/taskmixin.py b/airflow/models/taskmixin.py
--- a/airflow/models/taskmixin.py
+++ b/airflow/models/taskmixin.py
@@ -28,6 +28,7 @@
     from logging import Logger
 
     from airflow.models.dag import DAG
+    from airflow.models.mappedoperator import MappedOperator
     from airflow.utils.edgemodifier import EdgeModifier
     from airflow.utils.task_group import TaskGroup
 
@@ -290,7 +291,7 @@ def serialize_for_task_group(self) -> Tuple[DagAttributeTypes, Any]:
         """This is used by SerializedTaskGroup to serialize a task group's content."""
         raise NotImplementedError()
 
-    def mapped_dependants(self) -> Iterator["DAGNode"]:
+    def mapped_dependants(self) -> Iterator["MappedOperator"]:
         """Return any mapped nodes that are direct dependencies of the current task
 
         For now, this walks the entire DAG to find mapped nodes that has this


Test patch:
diff --git a/tests/dags/test_mapped_classic.py b/tests/dags/test_mapped_classic.py
--- a/tests/dags/test_mapped_classic.py
+++ b/tests/dags/test_mapped_classic.py
@@ -32,3 +32,6 @@ def consumer(value):
 
 with DAG(dag_id='test_mapped_classic', start_date=days_ago(2)) as dag:
     PythonOperator.partial(task_id='consumer', python_callable=consumer).expand(op_args=make_arg_lists())
+    PythonOperator.partial(task_id='consumer_literal', python_callable=consumer).expand(
+        op_args=[[1], [2], [3]],
+    )
diff --git a/tests/dags/test_mapped_taskflow.py b/tests/dags/test_mapped_taskflow.py
--- a/tests/dags/test_mapped_taskflow.py
+++ b/tests/dags/test_mapped_taskflow.py
@@ -29,3 +29,4 @@ def consumer(value):
         print(repr(value))
 
     consumer.expand(value=make_list())
+    consumer.expand(value=[1, 2, 3])