Question: Our airflow project has a task that queries from BigQuery and uses Pool to dump in parallel to local JSON files: def dump_in_parallel(table_name): base_query = f"select * from models.{table_name}" all_conf_ids = range(1,10) n_jobs = 4 with Pool(n_jobs) as p: p.map(partial(dump_conf_id, base_query = base_query), all_conf_ids) with open("/tmp/final_output.json", "wb") as f: filenames = [f'/tmp/output_file_{i}.json' for i in all_conf_ids]  This task was working fine for us in airflow v1.10, but is no longer working in v2.1+. Section 2.1 here - https://blog.mbedded.ninja/programming/languages/python/python-multiprocessing/ - mentions "If you try and create a Pool from within a child worker that was already created with a Pool, you will run into the error: daemonic processes are not allowed to have children" Here is the full Airflow error: [2021-08-22 02:11:53,064] {taskinstance.py:1462} ERROR - Task failed with exception Traceback (most recent call last): File "/usr/local/lib/python3.7/site-packages/airflow/models/taskinstance.py", line 1164, in _run_raw_task self._prepare_and_execute_task_with_callbacks(context, task) File "/usr/local/lib/python3.7/site-packages/airflow/models/taskinstance.py", line 1282, in _prepare_and_execute_task_with_callbacks result = self._execute_task(context, task_copy) File "/usr/local/lib/python3.7/site-packages/airflow/models/taskinstance.py", line 1312, in _execute_task result = task_copy.execute(context=context) File "/usr/local/lib/python3.7/site-packages/airflow/operators/python.py", line 150, in execute return_value = self.execute_callable() File "/usr/local/lib/python3.7/site-packages/airflow/operators/python.py", line 161, in execute_callable return self.python_callable(*self.op_args, **self.op_kwargs) File "/usr/local/airflow/plugins/tasks/bigquery.py", line 249, in dump_in_parallel with Pool(n_jobs) as p: File "/usr/local/lib/python3.7/multiprocessing/context.py", line 119, in Pool context=self.get_context()) File "/usr/local/lib/python3.7/multiprocessing/pool.py", line 176, in __init__ self._repopulate_pool() File "/usr/local/lib/python3.7/multiprocessing/pool.py", line 241, in _repopulate_pool w.start() File "/usr/local/lib/python3.7/multiprocessing/process.py", line 110, in start 'daemonic processes are not allowed to have children' AssertionError: daemonic processes are not allowed to have children  If it matters, we run airflow using the LocalExecutor. Any idea why this task that uses Pool would have been working in airflow v1.10 but no longer in airflow 2.1?

Answer: Airflow 2 uses different processing model under the hood to speed up processing, yet to maintain process-based isolation between running tasks. That's why it uses forking and multiprocessing under the hook to run Tasks, but this also means that if you are using multiprocessing, you will hit the limits of Python multiprocessing that does not allow to chain multi-processing. I am not 100% sure if it will work but you might try to set execute_tasks_new_python_interpreter configuration to True. https://airflow.apache.org/docs/apache-airflow/stable/configurations-ref.html#execute-tasks-new-python-interpreter . This setting will cause airflow to start a new Python interpreter when running task instead of forking/using multiprocessing (though I am not 100% sure of the latter). It will work quite a bit slower (up to a few seconds of overhead) though to run your task as the new Python interpreter will have to reinitialize and import all the airflow code before running your task. If that does not work, then you can lunch your multiprocessing job using PythonVirtualenvOperator - that one will launch a new Python interpreter to run your python code and you should be able to use multiprocessing.