Repository: celery/celery

Issue:
Celery hangs indefinitely (instead of failing) if redis is not started
## Checklist

- [x] I have included the output of ``celery -A proj report`` in the issue.
      (if you are not able to do this, then at least specify the Celery
       version affected).
- [x] I have verified that the issue exists against the `master` branch of Celery.

```
software -> celery:4.1.0 (latentcall) kombu:4.1.0 py:2.7.12
            billiard:3.5.0.3 redis:2.10.6
platform -> system:Linux arch:64bit, ELF imp:CPython
loader   -> celery.loaders.app.AppLoader
settings -> transport:redis results:disabled

broker_url: u'redis://localhost:6379/0'
```

## Steps to reproduce

Stop `redis-server.`
Initialize a celery instance configured with redis.

## Expected behavior

I expected an `OperationalError` or another exception communicating that celery cannot connect to the broker server.

## Actual behavior

It hangs indefinitely.

## Why this is important

In the current situation is not possible to write automated functional and integration tests that check whether the software handles failures when brokers are not reachable.
Celery hangs indefinitely (instead of failing) if redis is not started
## Checklist

- [x] I have included the output of ``celery -A proj report`` in the issue.
      (if you are not able to do this, then at least specify the Celery
       version affected).
- [x] I have verified that the issue exists against the `master` branch of Celery.

```
software -> celery:4.1.0 (latentcall) kombu:4.1.0 py:2.7.12
            billiard:3.5.0.3 redis:2.10.6
platform -> system:Linux arch:64bit, ELF imp:CPython
loader   -> celery.loaders.app.AppLoader
settings -> transport:redis results:disabled

broker_url: u'redis://localhost:6379/0'
```

## Steps to reproduce

Stop `redis-server.`
Initialize a celery instance configured with redis.

## Expected behavior

I expected an `OperationalError` or another exception communicating that celery cannot connect to the broker server.

## Actual behavior

It hangs indefinitely.

## Why this is important

In the current situation is not possible to write automated functional and integration tests that check whether the software handles failures when brokers are not reachable.


Discussion:
Sounds similar to #4328 to me, we've been running into a similar issue, although with RabbitMQ.
I have seen this issue with Redis stopped and when Celery tries to connect to the broker over IPv6 but the firewall is configured to silently drop the connection attempts. It would eventually fall back to IPv4 and get an instant connection refused. Could this be your issue?
@keaneokelley I'm testing this in a development environment with no firewall:

```
iptables -L -v -n
Chain INPUT (policy ACCEPT 799K packets, 2310M bytes)
 pkts bytes target     prot opt in     out     source               destination         
    0     0 ACCEPT     tcp  --  lxcbr0 *       0.0.0.0/0            0.0.0.0/0            tcp dpt:53
    0     0 ACCEPT     udp  --  lxcbr0 *       0.0.0.0/0            0.0.0.0/0            udp dpt:53
    0     0 ACCEPT     tcp  --  lxcbr0 *       0.0.0.0/0            0.0.0.0/0            tcp dpt:67
    0     0 ACCEPT     udp  --  lxcbr0 *       0.0.0.0/0            0.0.0.0/0            udp dpt:67

Chain FORWARD (policy ACCEPT 0 packets, 0 bytes)
 pkts bytes target     prot opt in     out     source               destination         
    0     0 ACCEPT     all  --  *      lxcbr0  0.0.0.0/0            0.0.0.0/0           
    0     0 ACCEPT     all  --  lxcbr0 *       0.0.0.0/0            0.0.0.0/0           

Chain OUTPUT (policy ACCEPT 592K packets, 47M bytes)
 pkts bytes target     prot opt in     out     source               destination
```

`ip6tables` output:

```
ip6tables -L -v -n
Chain INPUT (policy ACCEPT 0 packets, 0 bytes)
 pkts bytes target     prot opt in     out     source               destination         

Chain FORWARD (policy ACCEPT 0 packets, 0 bytes)
 pkts bytes target     prot opt in     out     source               destination         

Chain OUTPUT (policy ACCEPT 0 packets, 0 bytes)
 pkts bytes target     prot opt in     out     source               destination 
```

I have tracked similar issues here on github and it seems to me they have been closed without a real solution.

This issue is easy to test automatically, for example I created a test class which inserts a bogus broker url. The test hangs indefinitely.
@auvipy this is still an issue in v4.3
if you can install celery and kombu from master and try to find the root cause of this hang?
I ran into the same issue, both with Redis and RabbitMQ. I think this is a serious problem, since it could easily lead to a denial-of-service of a webservice.
Did anyone identify the root cause?
Still an issue... Plus fix could easily lead to DOS
You should probably try to set [`broker_connection_max_retries`](https://docs.celeryproject.org/en/latest/userguide/configuration.html#broker-connection-max-retries) to a much lower value.
+1 @thedrow that might be the reason.
I've seen that it's not indefinitely, actually. As @thedrow mentioned, `broker_connection_max_retries` have been set to 100 by default. It means that eventually it will fail in 2+4+6+..+200 secs = 10100 secs (2 hours 48 mins 20 secs, to be exact. Yeah, a long time indeed). When I modify the configuration and either:
1. Set `broker_connection_max_retries` to 1, celery will retry the connection once in 2 secs, if it's still not available then it will throw ConnectionError.
2. Set `broker_connection_retry` to 0, celery will not even try to retry the connection and will throw ConnectionError anyway.

So you might either don't try celery to retry the connection, or maybe set the max retries to maybe 3 tries, depends on your case.
Or maybe I missed something? I don't really sure what do you mean by "hangs". Is it really hangs, like the process is not responding, or it will retry the connection again and again and again?
@georgepsarakis 
I think `broker_connection_max_retry` should be set to lower value by default, maybe 3 retries, for example. 100 is kinda ridiculous especially with increased retry time.
Also the broker keep retrying without saying anything like retry attempt (Attempt 1 of 100..Attempt 2 of 100..etc) or how many attempt for broker to reconnect (Reconnecting to broker in 100 tries..). That made user confused on how long celery worker will reconnect to the broker and assumed that it will retry indefinitely, especially those who does not notice this configuration variable.
That's just my suggestion, feel free to improve or maybe another good suggestion :smile: 
Thank you for your analysis @marfgold1, what do you guys think?
@auvipy @georgepsarakis 
Sounds similar to #4328 to me, we've been running into a similar issue, although with RabbitMQ.
I have seen this issue with Redis stopped and when Celery tries to connect to the broker over IPv6 but the firewall is configured to silently drop the connection attempts. It would eventually fall back to IPv4 and get an instant connection refused. Could this be your issue?
@keaneokelley I'm testing this in a development environment with no firewall:

```
iptables -L -v -n
Chain INPUT (policy ACCEPT 799K packets, 2310M bytes)
 pkts bytes target     prot opt in     out     source               destination         
    0     0 ACCEPT     tcp  --  lxcbr0 *       0.0.0.0/0            0.0.0.0/0            tcp dpt:53
    0     0 ACCEPT     udp  --  lxcbr0 *       0.0.0.0/0            0.0.0.0/0            udp dpt:53
    0     0 ACCEPT     tcp  --  lxcbr0 *       0.0.0.0/0            0.0.0.0/0            tcp dpt:67
    0     0 ACCEPT     udp  --  lxcbr0 *       0.0.0.0/0            0.0.0.0/0            udp dpt:67

Chain FORWARD (policy ACCEPT 0 packets, 0 bytes)
 pkts bytes target     prot opt in     out     source               destination         
    0     0 ACCEPT     all  --  *      lxcbr0  0.0.0.0/0            0.0.0.0/0           
    0     0 ACCEPT     all  --  lxcbr0 *       0.0.0.0/0            0.0.0.0/0           

Chain OUTPUT (policy ACCEPT 592K packets, 47M bytes)
 pkts bytes target     prot opt in     out     source               destination
```

`ip6tables` output:

```
ip6tables -L -v -n
Chain INPUT (policy ACCEPT 0 packets, 0 bytes)
 pkts bytes target     prot opt in     out     source               destination         

Chain FORWARD (policy ACCEPT 0 packets, 0 bytes)
 pkts bytes target     prot opt in     out     source               destination         

Chain OUTPUT (policy ACCEPT 0 packets, 0 bytes)
 pkts bytes target     prot opt in     out     source               destination 
```

I have tracked similar issues here on github and it seems to me they have been closed without a real solution.

This issue is easy to test automatically, for example I created a test class which inserts a bogus broker url. The test hangs indefinitely.
@auvipy this is still an issue in v4.3
if you can install celery and kombu from master and try to find the root cause of this hang?
I ran into the same issue, both with Redis and RabbitMQ. I think this is a serious problem, since it could easily lead to a denial-of-service of a webservice.
Did anyone identify the root cause?
Still an issue... Plus fix could easily lead to DOS
You should probably try to set [`broker_connection_max_retries`](https://docs.celeryproject.org/en/latest/userguide/configuration.html#broker-connection-max-retries) to a much lower value.
+1 @thedrow that might be the reason.
I've seen that it's not indefinitely, actually. As @thedrow mentioned, `broker_connection_max_retries` have been set to 100 by default. It means that eventually it will fail in 2+4+6+..+200 secs = 10100 secs (2 hours 48 mins 20 secs, to be exact. Yeah, a long time indeed). When I modify the configuration and either:
1. Set `broker_connection_max_retries` to 1, celery will retry the connection once in 2 secs, if it's still not available then it will throw ConnectionError.
2. Set `broker_connection_retry` to 0, celery will not even try to retry the connection and will throw ConnectionError anyway.

So you might either don't try celery to retry the connection, or maybe set the max retries to maybe 3 tries, depends on your case.
Or maybe I missed something? I don't really sure what do you mean by "hangs". Is it really hangs, like the process is not responding, or it will retry the connection again and again and again?
@georgepsarakis 
I think `broker_connection_max_retry` should be set to lower value by default, maybe 3 retries, for example. 100 is kinda ridiculous especially with increased retry time.
Also the broker keep retrying without saying anything like retry attempt (Attempt 1 of 100..Attempt 2 of 100..etc) or how many attempt for broker to reconnect (Reconnecting to broker in 100 tries..). That made user confused on how long celery worker will reconnect to the broker and assumed that it will retry indefinitely, especially those who does not notice this configuration variable.
That's just my suggestion, feel free to improve or maybe another good suggestion :smile: 
Thank you for your analysis @marfgold1, what do you guys think?
@auvipy @georgepsarakis 

Resolving patch:
diff --git a/celery/worker/consumer/consumer.py b/celery/worker/consumer/consumer.py
--- a/celery/worker/consumer/consumer.py
+++ b/celery/worker/consumer/consumer.py
@@ -51,7 +51,7 @@
 """
 
 CONNECTION_RETRY_STEP = """\
-Trying again {when}...\
+Trying again {when}... ({retries}/{max_retries})\
 """
 
 CONNECTION_ERROR = """\
@@ -421,8 +421,11 @@ def ensure_connected(self, conn):
         def _error_handler(exc, interval, next_step=CONNECTION_RETRY_STEP):
             if getattr(conn, 'alt', None) and interval == 0:
                 next_step = CONNECTION_FAILOVER
-            error(CONNECTION_ERROR, conn.as_uri(), exc,
-                  next_step.format(when=humanize_seconds(interval, 'in', ' ')))
+            next_step = next_step.format(
+                when=humanize_seconds(interval, 'in', ' '),
+                retries=int(interval / 2),
+                max_retries=self.app.conf.broker_connection_max_retries)
+            error(CONNECTION_ERROR, conn.as_uri(), exc, next_step)
 
         # remember that the connection is lazy, it won't establish
         # until needed.


Test patch:
diff --git a/t/unit/worker/test_consumer.py b/t/unit/worker/test_consumer.py
--- a/t/unit/worker/test_consumer.py
+++ b/t/unit/worker/test_consumer.py
@@ -264,6 +264,22 @@ def test_connect_error_handler(self):
         errback = conn.ensure_connection.call_args[0][0]
         errback(Mock(), 0)
 
+    @patch('celery.worker.consumer.consumer.error')
+    def test_connect_error_handler_progress(self, error):
+        self.app.conf.broker_connection_retry = True
+        self.app.conf.broker_connection_max_retries = 3
+        self.app._connection = _amqp_connection()
+        conn = self.app._connection.return_value
+        c = self.get_consumer()
+        assert c.connect()
+        errback = conn.ensure_connection.call_args[0][0]
+        errback(Mock(), 2)
+        assert error.call_args[0][3] == 'Trying again in 2.00 seconds... (1/3)'
+        errback(Mock(), 4)
+        assert error.call_args[0][3] == 'Trying again in 4.00 seconds... (2/3)'
+        errback(Mock(), 6)
+        assert error.call_args[0][3] == 'Trying again in 6.00 seconds... (3/3)'
+
 
 class test_Heart: