Repository: apache/airflow

Issue:
Confusing log for long running tasks: "dependency 'Task Instance Not Running' FAILED: Task is in the running state"
**Apache Airflow version**: 1.10.* / 2.0.* / 2.1.*


**Kubernetes version (if you are using kubernetes)** (use `kubectl version`): Any

**Environment**:

- **Cloud provider or hardware configuration**: Any
- **OS** (e.g. from /etc/os-release): Any
- **Kernel** (e.g. `uname -a`): Any
- **Install tools**: Any
- **Others**: N/A

**What happened**:

This line in the TaskInstance log is very misleading. It seems to happen for tasks that take longer than one hour. When users are waiting for tasks to finish and see this in the log, they often get confused. They may think something is wrong with their task or with Airflow. In fact, this line is harmless. It's simply saying "the TaskInstance is already running so it cannot be run again". 

```
{taskinstance.py:874} INFO - Dependencies not met for <TaskInstance: ... [running]>, dependency 'Task Instance Not Running' FAILED: Task is in the running state
{taskinstance.py:874} INFO - Dependencies not met for <TaskInstance: ... [running]>, dependency 'Task Instance State' FAILED: Task is in the 'running' state which is not a valid state for execution. The task must be cleared in order to be run.
```

**What you expected to happen**:

The confusion is unnecessary. This line should be silenced in the log. Or it should log something clearer.

**How to reproduce it**:

Any task that takes more than an hour to run has this line in the log.


Discussion:
I bumped into this issue and I misunderstood until now 😅 
thanks for pointing out it's okay to wait
After some more investigation, it's very likely we see this log appearing an hour after a long running task started because of the default `visibility_timeout` setting in Celery. This code in default_celery.py sets `visibility_timeout` to 21600 only if the broker_url starts with redis or sqs. In our case we are using redis sentinels so it's still redis although the URL starts with `sentinel`. Therefore the `visibility_timeout` is left at 3600 which is the default according to [celery documentation](https://docs.celeryproject.org/en/v4.4.7/getting-started/brokers/redis.html#visibility-timeout). The weird thing is that after I tried to manually change `visibility_timeout` to a very large integer in airflow.cfg, the same log still showed up exactly an hour after a task started. So it seems changing `visibility_timeout` in this case does not make any difference. Not sure if anyone experienced the same.

@david30907d maybe try changing `visibility_timeout` to a large number in your setup and see if it still happens after an hour. If it stops for you, it means `visibility_timeout` is probably the cause. There may be something wrong in our own setup causing changing `visibility_timeout` not to take effect.

```python
def _broker_supports_visibility_timeout(url):
    return url.startswith("redis://") or url.startswith("sqs://")


log = logging.getLogger(__name__)

broker_url = conf.get('celery', 'BROKER_URL')

broker_transport_options = conf.getsection('celery_broker_transport_options') or {}
if 'visibility_timeout' not in broker_transport_options:
    if _broker_supports_visibility_timeout(broker_url):
        broker_transport_options['visibility_timeout'] = 21600
```
@yuqian90 thanks, very useful information. I'll give `celery_broker_transport_options` a shot!
Moved it to 2.2 now - I guess we have not addressed it yet in 2.1.1
In the case where the visibility timeout is reached, it's confusing that there is not a clear log line that the task has been killed for taking too long to complete.

(If that's indeed what is happening.)

@potiuk is it the case, that the Celery task is killed or is it simply no longer streaming logs into Airflow at that point?
Not sure. Needs investigation
@malthe That's not quite what is happening. My understanding is this:

- The task message on celery is picked up and the TI starts running
- After 1 hour (the celery visiblity timeout) celery makes the message visible again
- A second worker picks up the celery message
- When the second one runs, it gets this message.


So the first attempt is still running, and this message is the _second_ concurrent worker (but for the same try_number)  saying "I can't run this task, it's already running."
@ashb so regardless of whether the first worker is still running or defunct/dead, shouldn't the second worker be able to "take over" somehow?

Otherwise, what's the point in trying?
This is largely airflow and celery "fighting" over behaviour
Hi!
Just ran into the issue, having an "amqp://" starting broker url.
Will it be fixed in 2.3.0?
> @ashb so regardless of whether the first worker is still running or defunct/dead, shouldn't the second worker be able to "take over" somehow?
> 
> Otherwise, what's the point in trying?

From what i've observed, there's no point for the second Airflow worker to try running it. We should just silence the error.

```
{taskinstance.py:874} INFO - Dependencies not met for <TaskInstance: ... [running]>, dependency 'Task Instance Not Running' FAILED: Task is in the running state
{taskinstance.py:874} INFO - Dependencies not met for <TaskInstance: ... [running]>, dependency 'Task Instance State' FAILED: Task is in the 'running' state which is not a valid state for execution. The task must be cleared in order to be run.
```
Hi @yuqian90!
Do I understand correctly that without patching I couldn't configure that `visibility_timeout` with an `amqp`-starting broker 
URL?
@yuqian90 How can the worker tell it is the "second" time it's running?
I have this problem in an AWS MWAA environment (Airflow version 2.2.2). Only it is not harmless, my task is actually getting killed after the visibility_timeout of 12 hours expires. I am not able to split the long-running task into shorter tasks (it involves a CREATE INDEX database statement that takes more than 12 hours).

What I don't understand is, isn't the visibility_timeout setting [related](https://docs.celeryq.dev/en/v4.4.7/glossary.html#term-acknowledged) to celery's [task_acks_late](https://docs.celeryq.dev/en/v4.4.7/userguide/configuration.html#std:setting-task_acks_late) setting? To my understanding, this is False by default so a task is acknowledged just before its execution starts. If that is true, then why is the message picked up by another worker after the visibility_timeout has expired?
>AWS MWAA environment (Airflow version 2.2.2).

+1, Noticed even in AWS MWAA 2.0.2
Having the same problem with Airflow 2.2.5 + celery + redis.

The default visibility_timeout is actually 21600, 6 hours (not None from the airflow.cfg) , which means if the worker cannot pick up the task successfully, the task_instance will be stuck in the queued state for 6 hours and not visible to other workers. I remember when clearing tasks in this situation won't help either, may need to mark task failed first and then clear it.

If I change visibility_timeout to a smaller number, say 300, 5 minutes, then due the default settings of in `airflow.config_templates.default_celery.DEFAULT_CELERY_CONFIG`, `'task_acks_late': True`, the task_instance will be killed by other workers if it keeps running after 5 minutes, complaining about the task is still in the running state. (Or I'm not sure if that is the same case when ack is lost from the worker)

Can we just change the default setting of `task_acks_late` to False? Will it cause any side effects?

Hello,
we are on Version: 2.2.5+composer  and we have the same problem but in our case the task is really stopped and retried by our policy.
So the message is not harmless as it ends the job :

`[2022-11-08, 05:30:41 UTC] {taskinstance.py:1034} INFO - Dependencies not met for <TaskInstance: pipeline_analytics_revenu_billing.analytics_revenu_insert scheduled__2022-11-07T03:10:00+00:00 [running]>, dependency 'Task Instance State' FAILED: Task is in the 'running' state.
[2022-11-08, 05:30:41 UTC] {taskinstance.py:1034} INFO - Dependencies not met for <TaskInstance: pipeline_analytics_revenu_billing.analytics_revenu_insert scheduled__2022-11-07T03:10:00+00:00 [running]>, dependency 'Task Instance Not Running' FAILED: Task is in the running state
[2022-11-08, 05:30:41 UTC] {local_task_job.py:99} INFO - Task is not able to be run
`
just in case someone is on the same boat:

in long-running tasks, this behavior can also cause your logs to look funny because when the 2nd worker picked up the task after the `visibility_timeout` (6 hours), it can only get the the logs starting from the 7th hour, all previous logs obtained by the 1st worker are gone

```
[2022-11-10, 06:30:22 EST] {{taskinstance.py:1155}} INFO - Dependencies not met for <TaskInstance: my_dag.my_task scheduled__2022-11-09T05:30:00+00:00 [running]>, dependency 'Task Instance Not Running' FAILED: Task is in the running state
[2022-11-10, 06:30:22 EST] {{taskinstance.py:1155}} INFO - Dependencies not met for <TaskInstance: my_dag.my_task scheduled__2022-11-09T05:30:00+00:00 [running]>, dependency 'Task Instance State' FAILED: Task is in the 'running' state.
[2022-11-10, 06:30:22 EST] {{local_task_job.py:103}} INFO - Task is not able to be run
[2022-11-10, 06:30:50 EST] {{ecs.py:154}} INFO - [2022-11-10 11:30:36,655] {<LOGS STARTING FROM THE 7th HOUR>}
```
In order to help I noticed a similar behaviour in 2.5.0 (with celery, redis, postgres, default configs).
I also found related issues on Celery ( https://github.com/celery/celery/issues/5935 , https://github.com/celery/celery/issues/6229).

In my case I have a long running task (e.g. 1 day) that logs a reschedule attempts.
It seems it does not have a practical effect on the task or the DAG, the task still runs and logs correctly till the end, no other runs are created.

We are noticing long-running pods being "lost" by Airflow after it logs this message. The last error message is this one, then the task fails. However, the pod is still alive and logging in GKE.
> Hello, we are on Version: 2.2.5+composer and we have the same problem but in our case the task is really stopped and retried by our policy. So the message is not harmless as it ends the job :
> 
> `[2022-11-08, 05:30:41 UTC] {taskinstance.py:1034} INFO - Dependencies not met for <TaskInstance: pipeline_analytics_revenu_billing.analytics_revenu_insert scheduled__2022-11-07T03:10:00+00:00 [running]>, dependency 'Task Instance State' FAILED: Task is in the 'running' state. [2022-11-08, 05:30:41 UTC] {taskinstance.py:1034} INFO - Dependencies not met for <TaskInstance: pipeline_analytics_revenu_billing.analytics_revenu_insert scheduled__2022-11-07T03:10:00+00:00 [running]>, dependency 'Task Instance Not Running' FAILED: Task is in the running state [2022-11-08, 05:30:41 UTC] {local_task_job.py:99} INFO - Task is not able to be run `

Did anyone find a solution to this??
Following the Airflow manual, setting AIRFLOW__CELERY_BROKER_TRANSPORT_OPTIONS__VISIBILITY_TIMEOUT to a high value should fix the problem. I haven't tryed the parameter. Do we have some more report about moving this timeout to higher value?
It's seems that this parameter is not working. We also do not know if this behaviour is due to Airflow or to Celery bugs posted.
This is definetely causing production problems to some Airflow users.
I suppose that the issue title is misleading since it's not a "confusing log" but it forces a behaviour that must be definetely investigated and possibly fixed.
Did any one has any suggestion or solution for this : we are facing this issue from last 1 week. we do have jobs which runs for more than hour approx 6-12 hours. till last week everything is running fine but all of sudden jobs started failing with this issue : maximum job is running for 1 hour and then its getting failed 

{taskinstance.py:999} INFO - Dependencies not met for <TaskInstance: file_data_transfer.taskid_data_transfer 2023-02-19T00:01:00+00:00 [running]>, dependency 'Task Instance Not Running' FAILED: Task is in the running state

{taskinstance.py:999} INFO - Dependencies not met for <TaskInstance: file_data_transfer.taskid_data_transfer 2023-02-19T00:01:00+00:00 [running]>, dependency 'Task Instance State' FAILED: Task is in the 'running' state which is not a valid state for execution. The task must be cleared in order to be run.

{local_task_job.py:99} INFO - Task is not able to be run

We do have provided the maximum visibility timeout to 7 days but still getting failed

Any suggestion pls as its impacting huge
I am also experiencing this in airflow 2.4.3 with a number of DAGs. 

The `Task is not able to be run` error message is interesting, and people seem to think that this message is displayed when the `AIRFLOW__CELERY_BROKER_TRANSPORT_OPTIONS__VISIBILITY_TIMEOUT` is shorter than the DAGs takes to run, but that is not my reading of [the docs for that config](https://airflow.apache.org/docs/apache-airflow/stable/configurations-ref.html#visibility-timeout), it's more of a celery timeout for the worker to acknowledge a task once it has been assigned.

It seems likely that the logs are lost for really long running celery tasks (perhaps due to the visibility_timeout, I am not sure), so people only see the `Task is not able to be run` error, but something else (likely their code) is deadlocking, they just cant see it.

Has anyone tried setting `execution_timeout` on those tasks, I assume that airflow will eventually mark that task as failed.
We had this problem in MWAA environment, task getting killed after visibility timeout expired. But since we moved to EKS using Airflow helm chart, we can see the logs described in the issue but the running task is never killed. It completes successfully even after the visibility timeout has expired.
@yuqian90 is that possible for you to update the references of the issue in order to report updated information about the bug (e.g. version affected, episodes of task killings etc..) so it can be triaged accordingly?
I have an observation : we have started facing this issue exactly from Feb 19th 7.42 AM . And at the same time when i was validating the serving pod information i can see that it was created on Same date and time Feb 19th 7.44 AM . Is something we can find and relate this information for this issue or any suggestion pls

Kubernetes Engine > Services & Ingress page - airflow-sqlproxy-service --serving pod
I am dealing with that issue since March 01, 2023. Airflow version : 2.2.2 and we are using with MWAA. Do you have any solution about it? When I run the tasks I am getting this error and subtask stucked in queued or running state. Here is the logs : 

 INFO - Dependencies not met for <TaskInstancedependency 'Task Instance Not Running' FAILED: Task is in the running state
INFO - Dependencies not met for <TaskInstance: dependency 'Task Instance State' FAILED: Task is in the 'running' state which is not a valid state for execution. The task must be cleared in order to be run.
INFO - Task is not able to be run
As an first assumption : it looks like an environment issue and as a temporary fix : to retires maximum extent. 
For example: if one of the job runs for more than 10+ hours and due to this environment issue job is getting failed every 1 hour . We have added maximum retries so that at 4th retry the job is getting fixed automatically and error message is getting ignored. 
For now retries might be the best option as an temporary fix


> As an first assumption : it looks like an environment issue and as a temporary fix : to retires maximum extent. For example: if one of the job runs for more than 10+ hours and due to this environment issue job is getting failed every 1 hour . We have added maximum retries so that at 4th retry the job is getting fixed automatically and error message is getting ignored. For now retries might be the best option as an temporary fix

I also did the same and opened the ticket to Aws.
Please also help us to know if you get any response for AWS with any solution : as we have also opened a ticket to Google Cloud
I've been facing this problem too, in my case my task doesn't even run. It is a task to run a job in Databricks with DatabricksDeferrableOperator and its upstreams last for approximately 2 hours, but in the log of the tasks that fail, the execution date is less than the end date of the upstreams. So it looks like it ran without the upstreams having successfully completed.

When analyzing the scheduler log I found this:

`[2023-03-08 06:06:06,617] {scheduler_job.py:687} ERROR - Executor reports task instance <TaskInstance: job_orchestration.sourcing.cubo_custos_job_new.cubo_custos_job_new scheduled__2023-03-07T05:00:00+00:00 [queued ]> finished (success) although the task says its queued. (Info: None) Was the task killed externally?`


One more observation : usually the becoz of below error: job used to fail or job will ignore the warning. Now job is getting stuck and its not even getting failed and moving forward. 

Any one got any solution pls help 



{taskinstance.py:999} INFO - Dependencies not met for <TaskInstance: file_data_transfer.taskid_data_transfer 2023-02-19T00:01:00+00:00 [running]>, dependency 'Task Instance Not Running' FAILED: Task is in the running state

{taskinstance.py:999} INFO - Dependencies not met for <TaskInstance: file_data_transfer.taskid_data_transfer 2023-02-19T00:01:00+00:00 [running]>, dependency 'Task Instance State' FAILED: Task is in the 'running' state which is not a valid state for execution. The task must be cleared in order to be run.

{local_task_job.py:99} INFO - Task is not able to be run
Any one has any solution to fix this ??

...
> It seems likely that the logs are lost for really long running celery tasks (perhaps due to the visibility_timeout, I am not sure), so people only see the `Task is not able to be run` error, but something else (likely their code) is deadlocking, they just cant see it.
> 
> Has anyone tried setting `execution_timeout` on those tasks, I assume that airflow will eventually mark that task as failed.

yes I have, and that's not the solution, task keeps being "externally set to failed"
> ...
> 
> > It seems likely that the logs are lost for really long running celery tasks (perhaps due to the visibility_timeout, I am not sure), so people only see the `Task is not able to be run` error, but something else (likely their code) is deadlocking, they just cant see it.
> > Has anyone tried setting `execution_timeout` on those tasks, I assume that airflow will eventually mark that task as failed.
> 
> yes I have, and that's not the solution, task keeps being "externally set to failed"

What is the action you have applied ?? And will that be going to at least complete temporarily ???
I can give you the full logs right now, my task is a PythonOperator that inserts huge CSV files into Postgres. Concretely, I pass a `method` argument to the `to_sql` pandas method, in order to apply an upsert operation.

My task is always externally failed after 6 hours :/
same issue. a task that used to take ~4 hours is now taking ~8 hours since I switched to celery. it seems maybe on the retry it finishes? 
In our context (GCP Airflow 2.2.5 https://github.com/apache/airflow/issues/16163#issuecomment-1306729126) the problem was correlated with Kubernetes pods being evicted due of lacking ressources. After scaling up our kubernetes cluster and updating to Airflow 2.4.3 (+ google-providers=8.8.0 due to https://stackoverflow.com/questions/74730471/gcstobigqueryoperator-not-working-in-composer-2-1-0-airflow-2-3-4/75319966#75319966 ) we have not seen this problem in the last month since upgrading .
As we no longer have pods evictions I cannot conclude if the issue was fixed in Airflow or the workaround was sufficient (preventing Airflow workers to be killed).


 
Hello, are there any new evidence on this bug?
@ashb @potiuk and others watching, if you experience this issue, __it's just that your task is taking more than 6 hours__, in most cases your task will continue running, but you can't see your logs until the task finishes or fails.

As a temporary fix, you can increase the time it takes for this to happen by increasing the value of [`AIRFLOW__CELERY_BROKER_TRANSPORT_OPTIONS__VISIBILITY_TIMEOUT`](https://airflow.apache.org/docs/apache-airflow/stable/configurations-ref.html#visibility-timeout) to something like `86400` (24 hours).

----

I believe I have uncovered the cause of this issue and would appreciate feedback.

The question is, why are we losing the logs after 6 hours?  
I believe this is because of a slightly complex combination of celery configs:

- [`task_acks_late` = `True`](https://github.com/apache/airflow/blob/2.6.1/airflow/config_templates/default_celery.py#L51)
- [`broker_transport_options.visibility_timeout` = `21600`](https://github.com/apache/airflow/blob/2.6.1/airflow/config_templates/default_celery.py#L39)

The `visibility_timeout` is the number of seconds celery will wait for an acknowledgment before it returns a task to the queue (for it to be picked up by another worker). But because `task_acks_late` is `True`, airflow's celery workers will not acknowledge tasks until AFTER the task has finished. Hence, if a task takes more than `visibility_timeout` seconds to finish, celery will tell another worker to start this task.

When this happens, the new instance of this task will realize the task is still running on the old worker (it's not failed, and is even heart-beating the TaskInstance), and will correctly wait (because a task can not be "running" in two places). But now the airflow UI only shows the logs for this new "phantom task", which will always be:

```
{taskinstance.py:874} INFO - Dependencies not met for <TaskInstance: ... [running]>, dependency 'Task Instance Not Running' FAILED: Task is in the running state
{taskinstance.py:874} INFO - Dependencies not met for <TaskInstance: ... [running]>, dependency 'Task Instance State' FAILED: Task is in the 'running' state which is not a valid state for execution. The task must be cleared in order to be run.
```

Effectively, this issue is the result of celery taking matters into its own hands (outside of the control of the airflow scheduler), and telling a new worker to start the task which is still running on the old worker.

----

Setting our [`task_acks_late` to `False`](https://github.com/apache/airflow/blob/2.6.1/airflow/config_templates/default_celery.py#L51) will likely fix this issue. However,  I am not sure what the other implications would be, as I can see that it has been set to `True` for most of Airflow's existence, [it was set to `True` in this commit from 2015](https://github.com/apache/airflow/commit/8bd122f7b3fbfdd5b7741ded8813ca0b8c4eccae#diff-ac6d6f745ae19450e4bfbd1087d865b5784294354c885136b97df437460d5f10R22). 

Celery's purpose of `task_acks_late` is to prevent "zombie" tasks that take too long. But Airflow already has ["heartbeat" checks for zombie tasks](https://airflow.apache.org/docs/apache-airflow/2.6.1/configurations-ref.html#scheduler-zombie-task-threshold), so there is probably no need to have celery re-issuing tasks. 

Setting `task_acks_late` to `False`, and `visibility_timeout` to a much lower number (like `300` sec), will also have the added benefit of detecting celery transport failures 

----

Finally, I want to highlight that Airflow's docs are confusing about what the default `visibility_timeout` is, because [this piece of code will set it to `21600`](https://github.com/apache/airflow/blob/2.6.1/airflow/config_templates/default_celery.py#L39), when [`AIRFLOW__CELERY_BROKER_TRANSPORT_OPTIONS__VISIBILITY_TIMEOUT`](https://airflow.apache.org/docs/apache-airflow/2.6.1/configurations-ref.html#visibility-timeout) is `None`, which is different from the celery default of `3600`
Care for submitting a PR @thesuperzapper   ? That would be a small thing to give back for the years of business you have on top of Airflow 
According to the Celery docs:

> Late ack means the task messages will be acknowledged after the task has been executed, not right before (the default behavior).

It seems that it's a way to ensure _at-least-once_ processing rather than _at-most-once_. I would say that since Airflow does retrying on its own accord, we want the latter which is why it probably should remain disabled.
@potiuk @malthe here is the PR to change `task_acks_late` to `False`:
- https://github.com/apache/airflow/pull/31829

Let's discuss the implications and do some testing in that thread.
@thesuperzapper @potiuk , We are also facing the same issue, however in our case the previous set of logs are getting trimmed and the logs start with 
```
{taskinstance.py:874} INFO - Dependencies not met for <TaskInstance: ... [running]>, dependency 'Task Instance Not Running' FAILED: Task is in the running state
{taskinstance.py:874} INFO - Dependencies not met for <TaskInstance: ... [running]>, dependency 'Task Instance State' FAILED: Task is in the 'running' state which is not a valid state for execution. The task must be cleared in order to be `run.
```
Logs of previously running task append post these lines.

Airflow version: 2.6.0
I am seeing this as well with Airflow v2.5.1 (not using kubernetes) w/Celery Executor. We also utilize s3 remote logging, and observe regular log for the first hour, then `Dependencies not met for <TaskInstance` log is written to s3 and only that log is visible until the task completes (since airflow checks remote log first).

Once the task completes, the "real" log for the original task is appended below the `Dependencies not met for <TaskInstance` log...

Resolving patch:
diff --git a/airflow/providers/celery/executors/default_celery.py b/airflow/providers/celery/executors/default_celery.py
--- a/airflow/providers/celery/executors/default_celery.py
+++ b/airflow/providers/celery/executors/default_celery.py
@@ -72,7 +72,7 @@ def _broker_supports_visibility_timeout(url):
     "accept_content": ["json"],
     "event_serializer": "json",
     "worker_prefetch_multiplier": conf.getint("celery", "worker_prefetch_multiplier", fallback=1),
-    "task_acks_late": True,
+    "task_acks_late": conf.getboolean("celery", "task_acks_late", fallback=True),
     "task_default_queue": conf.get("operators", "DEFAULT_QUEUE"),
     "task_default_exchange": conf.get("operators", "DEFAULT_QUEUE"),
     "task_track_started": conf.getboolean("celery", "task_track_started", fallback=True),


Test patch:
diff --git a/tests/providers/celery/executors/test_celery_executor.py b/tests/providers/celery/executors/test_celery_executor.py
--- a/tests/providers/celery/executors/test_celery_executor.py
+++ b/tests/providers/celery/executors/test_celery_executor.py
@@ -358,3 +358,12 @@ def test_sentinel_kwargs_loaded_from_string():
     assert default_celery.DEFAULT_CELERY_CONFIG["broker_transport_options"]["sentinel_kwargs"] == {
         "service_name": "mymaster"
     }
+
+
+@conf_vars({("celery", "task_acks_late"): "False"})
+def test_celery_task_acks_late_loaded_from_string():
+    import importlib
+
+    # reload celery conf to apply the new config
+    importlib.reload(default_celery)
+    assert default_celery.DEFAULT_CELERY_CONFIG["task_acks_late"] is False