Repository: celery/celery

Issue:
group.skew() removes all tasks if group input is generator
## Checklist

- [X] I have included the output of ``celery -A proj report`` in the issue.
      (if you are not able to do this, then at least specify the Celery
       version affected).

      python 3.6.2
      celery==4.1.0
      amqp==2.2.1
      billiard==3.5.0.3

- [X] I have verified that the issue exists against the `master` branch of Celery.
      As of `be55de6`

## Steps to reproduce
```
@celery.task(bind=True)
def noop(task):
    pass
```

1. `group = celery.group(noop.si() for i in range(10))`
2. `group.skew()`  -> `group(<empty>)`
3. `group.apply_async()` or `group()`  -> `<GroupResult: 66fd517e-123b-4299-8cbe-9183b3f02626 []>`

## Expected behavior
```
<GroupResult: d610c923-e939-4199-9bb7-7bc89daa2ccb [9152bd60-f815-4bbe-a407-a003db34b19d, 7fc2d0ac-72ab-48b1-9083-be0f3b50b00d, d7c0a00a-85c3-4780-88f9-23be8da6fb73, 3737944f-91a4-4886-95c6-4960ef4764a7, a8da96ea-d2c4-4876-8026-e0c24c7d508a, 83ee5316-fb2f-472b-8e94-284144f92438, 5715a751-f28d-4deb-ac15-114854570fde, bc1646bf-cec9-4740-8e74-411d7b66dcca, 337aa1f3-5606-45b2-af68-5c5527289019, f5fffb98-f2f5-4423-a656-8c1d7e93291a]>
```

## Actual behavior

```
<GroupResult: 66fd517e-123b-4299-8cbe-9183b3f02626 []>
```

### Notes
If you `print(group)`(or something that evaluates `self.tasks`) before calling `group.skew()`, it works normally. 


Discussion:
could you plz check the master if it still exists?
Yes.  Still happening as of 2547666a1ea13b27bc13ef296ae43a163ecd4ab3
any workaround? 
I believe this is a bug in `celery.utils.functional.regen`. Compare the outputs of `regen` on 4.1.0 (which has this bug) and 3.1.25 (which doesn't):

On Celery 4.1.0:

```
In [1]: from celery.utils.functional import regen        

In [2]: gen = regen(i for i in range(10))                

In [3]: [i for i in gen]    
Out[3]: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9]                   

In [4]: [i for i in gen]    
Out[4]: []  
```

On Celery 3.1.25:

```
In [1]: from celery.utils.functional import regen        

In [2]: gen = regen(i for i in range(10))                

In [3]: [i for i in gen]                                 
Out[3]: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9]                   

In [4]: [i for i in gen]                                 
Out[4]: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9]                   

In [5]: [i for i in gen]                                 
Out[5]: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9]   
```

Looking at https://github.com/celery/celery/commit/9982773022d3de2d41ca59509220763da527b20e it would appear that `__consumed` doesn't get updated inside `__iter__`.

Doing `print(group)` first makes things work as expected because `__getitem__` is called and populates `__consumed` correctly. Then subsequent calls to `__iter__` work as expected too.

A more proper workaround is to either only pass lists and tuples to `group`, or if you're using something like `Task.chunks`, reassign `tasks` with a list, e.g.:

```
mytask.chunks(range(10), 2).group()
mytask.tasks = list(mytask.tasks)
mytask.skew(step=15)
```

Finally, https://github.com/celery/celery/pull/4459 will (probably) fix this issue too. @auvipy would you mind adding tests for this bug into your PR? Or if you prefer, I can open a separate PR that addresses this issue only.
plz proceed with the tests
Same issue:

    celery==5.0.2

@maybe-sybr would be great if you could also take a look here when possible
I traced this back to get repro on the originally reported versions as well as current master (420e3931a). Worth noting that I made the mistake of thinking it was fixed because I was printing the tasks in the group between steps in the MRTC in this issue's summary to see what was going on. Doing so concretises the generator underlying the `_regen` instance. Here's my new MRTC which needs a backend running for the task submission if the expected behaviour is to be observed:

```python
import celery

app = celery.Celery(name="app")

@app.task(bind=True)
def noop(self):
    pass

def gen():
    #assert False
    yield from (noop.si() for i in range(10))

g = celery.group(gen())
# Uncomment this to see the correct behaviour - but you'll need a backend
# running since that'll mean we acually have tasks to submit.
#print(g.tasks)
gs = g.skew()
r = g()
print(repr(r))
```

https://github.com/celery/celery/issues/4298#issuecomment-377276099 is correct in that this is breakage in `_regen` which has already been fixed in #4459 and #6112, but neither have been merged. I'd like to see the targeted fix pulled out of those PRs and merged to close this issue rather than waiting on the more complex PR to be done. I'll do that when I have some time in the next day or so.

Resolving patch:
diff --git a/celery/utils/functional.py b/celery/utils/functional.py
--- a/celery/utils/functional.py
+++ b/celery/utils/functional.py
@@ -3,7 +3,7 @@
 import sys
 from collections import UserList
 from functools import partial
-from itertools import chain, islice
+from itertools import islice
 
 from kombu.utils.functional import (LRUCache, dictfilter, is_list, lazy,
                                     maybe_evaluate, maybe_list, memoize)
@@ -182,6 +182,7 @@ def __init__(self, it):
         self.__it = it
         self.__index = 0
         self.__consumed = []
+        self.__done = False
 
     def __reduce__(self):
         return list, (self.data,)
@@ -190,7 +191,13 @@ def __length_hint__(self):
         return self.__it.__length_hint__()
 
     def __iter__(self):
-        return chain(self.__consumed, self.__it)
+        for x in self.__consumed:
+            yield x
+        if not self.__done:
+            for x in self.__it:
+                self.__consumed.append(x)
+                yield x
+            self.__done = True
 
     def __getitem__(self, index):
         if index < 0:
@@ -198,14 +205,26 @@ def __getitem__(self, index):
         try:
             return self.__consumed[index]
         except IndexError:
+            it = iter(self)
             try:
                 for _ in range(self.__index, index + 1):
-                    self.__consumed.append(next(self.__it))
+                    next(it)
             except StopIteration:
                 raise IndexError(index)
             else:
                 return self.__consumed[index]
 
+    def __bool__(self):
+        if len(self.__consumed):
+            return True
+
+        try:
+            next(iter(self))
+        except StopIteration:
+            return False
+        else:
+            return True
+
     @property
     def data(self):
         try:


Test patch:
diff --git a/t/unit/utils/test_functional.py b/t/unit/utils/test_functional.py
--- a/t/unit/utils/test_functional.py
+++ b/t/unit/utils/test_functional.py
@@ -1,11 +1,10 @@
 import pytest
-from kombu.utils.functional import lazy
-
 from celery.utils.functional import (DummyContext, first, firstmethod,
                                      fun_accepts_kwargs, fun_takes_argument,
                                      head_from_fun, maybe_list, mlazy,
                                      padlist, regen, seq_concat_item,
                                      seq_concat_seq)
+from kombu.utils.functional import lazy
 
 
 def test_DummyContext():
@@ -94,8 +93,11 @@ def test_list(self):
         fun, args = r.__reduce__()
         assert fun(*args) == l
 
-    def test_gen(self):
-        g = regen(iter(list(range(10))))
+    @pytest.fixture
+    def g(self):
+        return regen(iter(list(range(10))))
+
+    def test_gen(self, g):
         assert g[7] == 7
         assert g[6] == 6
         assert g[5] == 5
@@ -107,17 +109,19 @@ def test_gen(self):
         assert g.data, list(range(10))
         assert g[8] == 8
         assert g[0] == 0
-        g = regen(iter(list(range(10))))
+
+    def test_gen__index_2(self, g):
         assert g[0] == 0
         assert g[1] == 1
         assert g.data == list(range(10))
-        g = regen(iter([1]))
-        assert g[0] == 1
+
+    def test_gen__index_error(self, g):
+        assert g[0] == 0
         with pytest.raises(IndexError):
-            g[1]
-        assert g.data == [1]
+            g[11]
+        assert list(iter(g)) == list(range(10))
 
-        g = regen(iter(list(range(10))))
+    def test_gen__negative_index(self, g):
         assert g[-1] == 9
         assert g[-2] == 8
         assert g[-3] == 7
@@ -128,6 +132,21 @@ def test_gen(self):
 
         assert list(iter(g)) == list(range(10))
 
+    def test_nonzero__does_not_consume_more_than_first_item(self):
+        def build_generator():
+            yield 1
+            self.consumed_second_item = True
+            yield 2
+
+        self.consumed_second_item = False
+        g = regen(build_generator())
+        assert bool(g)
+        assert g[0] == 1
+        assert not self.consumed_second_item
+
+    def test_nonzero__empty_iter(self):
+        assert not regen(iter([]))
+
 
 class test_head_from_fun: