Repository: celery/kombu

Issue:
Prefetch with SQS enters infinite loop
I'm using kombu (4.0.2 or git master) with the SQS transport, starting Celery with `-c 2` and pretty quickly (within seconds of starting Celery, after processing a handful of items), I see the master process at 100% CPU. GDB attach, py-bt always shows 

```
Traceback (most recent call first):
  File "/var/virtualenvs/dbg/lib/python3.5/site-packages/kombu-4.0.2-py3.5.egg/kombu/async/hub.py", line 193, in call_soon
    if not isinstance(callback, Thenable):
  File "/var/virtualenvs/dbg/lib/python3.5/site-packages/kombu-4.0.2-py3.5.egg/kombu/transport/SQS.py", line 284, in _loop1
    self.hub.call_soon(self._schedule_queue, queue)
  File "/var/virtualenvs/dbg/lib/python3.5/site-packages/kombu-4.0.2-py3.5.egg/kombu/transport/SQS.py", line 293, in _schedule_queue
    self._loop1(queue)
  File "/var/virtualenvs/dbg/lib/python3.5/site-packages/vine/promises.py", line 136, in __call__
    retval = fun(*final_args, **final_kwargs)
  File "/var/virtualenvs/dbg/lib/python3.5/site-packages/kombu-4.0.2-py3.5.egg/kombu/async/hub.py", line 282, in create_loop
    item()
....
```
In the SQS channel, I see that such a loop can occur if `qos.can_consume()` always returns `False`, but I don't understand enough of the virtual channel implementation (yet) to see why it would get stuck in that state, when the SQS queue has messages available and celery workers are idling (i.e. waiting on `_recv` somewhere in `billiard`).

Any ideas or debugging tips? Are there some parameters to be tuned to avoid this?

FWIW This doesn't occur with Celery 3.1.25 & kombu 3.0.37, but we'd like to upgrade. 

_edit_ the Celery config is really simple, just pointing at a valid SQS queue and a few routes & queues defined.

_edit 2_ To wit, starting Celery with `-c 1 -P solo`, items are processed normally. I've also noticed that exactly 8 tasks are received each time before the loop, which suggests (to me) that the channel is working, but the workers aren't consuming the messages?


Discussion:
Looks like an known issue in Celery; I didn't find this in my prior searches, but fix described in [celery/issues/3712](https://github.com/celery/celery/issues/3712#issuecomment-272707969).
Reopening for specific issue that prefetching enters infinite loop. Despite the workaround, in our case, with thousands of tasks, we'd prefer our workers don't try to fetch them all at once.
I set up same environment on a single macOS machine and can't seem to reproduce.  Could this be a Linux specific or multi-host specific problem?
We switched to the Redis broker since we already had an instance running (in Elasticache or locally) and haven't had a single issue since.

Resolving patch:
diff --git a/kombu/asynchronous/hub.py b/kombu/asynchronous/hub.py
--- a/kombu/asynchronous/hub.py
+++ b/kombu/asynchronous/hub.py
@@ -3,7 +3,6 @@
 from __future__ import absolute_import, unicode_literals
 
 import errno
-import itertools
 from contextlib import contextmanager
 from time import sleep
 from types import GeneratorType as generator  # noqa
@@ -279,24 +278,15 @@ def create_loop(self,
         consolidate_callback = self.consolidate_callback
         on_tick = self.on_tick
         propagate = self.propagate_errors
-        todo = self._ready
 
         while 1:
+            todo = self._ready
+            self._ready = set()
+
             for tick_callback in on_tick:
                 tick_callback()
 
-            # To avoid infinite loop where one of the callables adds items
-            # to self._ready (via call_soon or otherwise), we take pop only
-            # N items from the ready set.
-            # N represents the current number of items on the set.
-            # That way if a todo adds another one to the ready set,
-            # we will break early and allow execution of readers and writers.
-            current_todos = len(todo)
-            for _ in itertools.repeat(None, current_todos):
-                if not todo:
-                    break
-
-                item = todo.pop()
+            for item in todo:
                 if item:
                     item()
 


Test patch:
diff --git a/t/unit/asynchronous/test_hub.py b/t/unit/asynchronous/test_hub.py
--- a/t/unit/asynchronous/test_hub.py
+++ b/t/unit/asynchronous/test_hub.py
@@ -508,30 +508,27 @@ def test_scheduler_property(self):
         assert list(hub.scheduler), [1, 2 == 3]
 
     def test_loop__tick_callbacks(self):
-        self.hub._ready = Mock(name='_ready')
-        self.hub._ready.__len__ = Mock(name="_ready.__len__")
-        self.hub._ready.__len__.side_effect = RuntimeError()
         ticks = [Mock(name='cb1'), Mock(name='cb2')]
         self.hub.on_tick = list(ticks)
 
-        with pytest.raises(RuntimeError):
-            next(self.hub.loop)
+        next(self.hub.loop)
 
         ticks[0].assert_called_once_with()
         ticks[1].assert_called_once_with()
 
     def test_loop__todo(self):
-        self.hub.fire_timers = Mock(name='fire_timers')
-        self.hub.fire_timers.side_effect = RuntimeError()
-        self.hub.timer = Mock(name='timer')
+        deferred = Mock(name='cb_deferred')
 
-        callbacks = [Mock(name='cb1'), Mock(name='cb2')]
+        def defer():
+            self.hub.call_soon(deferred)
+
+        callbacks = [Mock(name='cb1', wraps=defer), Mock(name='cb2')]
         for cb in callbacks:
             self.hub.call_soon(cb)
         self.hub._ready.add(None)
 
-        with pytest.raises(RuntimeError):
-            next(self.hub.loop)
+        next(self.hub.loop)
 
         callbacks[0].assert_called_once_with()
         callbacks[1].assert_called_once_with()
+        deferred.assert_not_called()