internlm/SWE-Fixer-Train-110K huggingface dataset:
Repository: celery/celery

Issue:
Document headers argument
At the moment there's no much information regarding headers argument for apply_async (and I think also applies to retry).

I found out about this reading the changelog for version 3.1, but maybe it could be specified as an optional property for apply_async in the documentation.

I can provide the PR for this if you want. Let me know if you agree.



Discussion:
Docs improvements are always welcome!

@malinoff Awesome. I'm trying to build the changes I applied but I'm getting the following error:

```
$ make html
mkdir -p .build/html .build/doctrees
sphinx-build -b html -d .build/doctrees   . .build/html
Running Sphinx v1.3.1

Extension error:
Could not import extension celery.contrib.sphinx (exception: cannot import name get_fdmax)
make: *** [html] Error 1
```

I searched a little but wasn't able to find any information regarding that error. I'm using Mac OS X 10.10.3 with virtualenv (All docs requirements installed) and Python 2.7.6. Should I use Python 3? Thanks.

I followed those steps and got that error
On Aug 7, 2015 12:13 AM, "Dmitry Malinovsky" notifications@github.com
wrote:

> https://github.com/celery/celery/blob/master/CONTRIBUTING.rst#building-the-documentation
> 
> —
> Reply to this email directly or view it on GitHub
> https://github.com/celery/celery/issues/2750#issuecomment-128576768.

@juanrossi yeah sorry, you need Celery installed to build docs.

@malinoff worked great with celery installed, thanks. I added the PR.


Resolving patch:
diff --git a/celery/app/task.py b/celery/app/task.py
--- a/celery/app/task.py
+++ b/celery/app/task.py
@@ -437,13 +437,18 @@ def apply_async(self, args=None, kwargs=None, task_id=None, producer=None,
                       if an error occurs while executing the task.
 
         :keyword producer: :class:`kombu.Producer` instance to use.
+
         :keyword add_to_parent: If set to True (default) and the task
             is applied while executing another task, then the result
             will be appended to the parent tasks ``request.children``
             attribute.  Trailing can also be disabled by default using the
             :attr:`trail` attribute
+
         :keyword publisher: Deprecated alias to ``producer``.
 
+        :keyword headers: Message headers to be sent in the
+            task (a :class:`dict`)
+
         :rtype :class:`celery.result.AsyncResult`: if
             :setting:`CELERY_ALWAYS_EAGER` is not set, otherwise
             :class:`celery.result.EagerResult`:


Test patch:
Repository: celery/celery

Issue:
celery multi ignores FQNs and subscribes to incorrect queue.
The following does not work:
`celery multi start celery1@turing.lan celery2@turing.lan -A tserver.app -Q:1 celery,hq1 -Q:2 celery,high,hq1 -c 4 -l info`

Output:

```
celery multi v3.1.23 (Cipater)
> Starting nodes...
    > celery1@turing.lan: OK
    > celery2@turing.lan: OK
```

In this case both workers subscribe only to the `celery` queue.

However, the following works:
`celery multi start 2 -A tserver.app -Q:1 celery,hq1 -Q:2 celery,high,hq1 -c 4 -l info`

In this case, both workers subscribe to correct queues.

Output:

```
celery multi v3.1.23 (Cipater)
> Starting nodes...
    > celery1@turing.lan: OK
    > celery2@turing.lan: OK
```



Discussion:

Resolving patch:
diff --git a/celery/bin/multi.py b/celery/bin/multi.py
--- a/celery/bin/multi.py
+++ b/celery/bin/multi.py
@@ -458,9 +458,14 @@ def _args_for_node(p, name, prefix, suffix, cmd, append, options):
     name, nodename, expand = _get_nodename(
         name, prefix, suffix, options)
 
+    if nodename in p.namespaces:
+        ns = nodename
+    else:
+        ns = name
+
     argv = ([expand(cmd)] +
             [format_opt(opt, expand(value))
-                for opt, value in items(p.optmerge(name, options))] +
+                for opt, value in items(p.optmerge(ns, options))] +
             [p.passthrough])
     if append:
         argv.append(expand(append))


Test patch:
diff --git a/celery/tests/bin/test_multi.py b/celery/tests/bin/test_multi.py
--- a/celery/tests/bin/test_multi.py
+++ b/celery/tests/bin/test_multi.py
@@ -149,6 +149,22 @@ def assert_line_in(name, args):
              ['COMMAND', '-n foo@', '-c 5', '']),
         )
 
+        p4 = NamespacedOptionParser(['foo', '-Q:1', 'test'])
+        names6 = list(multi_args(p4, cmd='COMMAND', suffix='""'))
+        self.assertEqual(
+            names6[0][0:2],
+            ('foo@',
+             ['COMMAND', '-n foo@', '-Q test', '']),
+        )
+
+        p5 = NamespacedOptionParser(['foo@bar', '-Q:1', 'test'])
+        names7 = list(multi_args(p5, cmd='COMMAND', suffix='""'))
+        self.assertEqual(
+            names7[0][0:2],
+            ('foo@bar',
+             ['COMMAND', '-n foo@bar', '-Q test', '']),
+        )
+
 
 class test_MultiTool(AppCase):
Repository: celery/celery

Issue:
test_retry_kwargs_can_be_empty fails on pypy3
From https://travis-ci.org/celery/celery/jobs/151613800:

```
======================================================================
ERROR: test_retry_kwargs_can_be_empty (celery.tests.tasks.test_tasks.test_task_retries)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/travis/build/celery/celery/celery/tests/tasks/test_tasks.py", line 178, in test_retry_kwargs_can_be_empty
    self.retry_task_mockapply.retry(args=[4, 4], kwargs=None)
  File "/home/travis/build/celery/celery/celery/app/task.py", line 611, in retry
    raise_with_context(exc or Retry('Task can be retried', None))
  File "/home/travis/build/celery/celery/celery/utils/serialization.py", line 255, in raise_with_context
    _raise_with_context(exc, exc_info[1])
  File "<string>", line 1, in _raise_with_context
TypeError: exception causes must derive from BaseException, not NoneType
```

https://github.com/celery/celery/blob/5031d6f27862001d3e3bc5a2dacf1185c933f2c9/celery/tests/tasks/test_tasks.py#L169



Discussion:
It looks like the culprit is https://github.com/celery/celery/commit/32b52ca875509b84d786e33ce2d39f62ab7ea050. Since `raise Exception from None` is new in Python 3.3 and PyPy 3 supports Python 3.2, I think the `if PY3` clause needs to be updated to `if PY33`.


Resolving patch:
diff --git a/celery/utils/serialization.py b/celery/utils/serialization.py
--- a/celery/utils/serialization.py
+++ b/celery/utils/serialization.py
@@ -24,7 +24,7 @@
 except ImportError:
     import pickle  # noqa
 
-PY3 = sys.version_info[0] >= 3
+PY33 = sys.version_info >= (3, 3)
 
 __all__ = [
     'UnpickleableExceptionWrapper', 'subclass_exception',
@@ -241,7 +241,9 @@ def jsonify(obj,
         return unknown_type_filter(obj)
 
 
-if PY3:
+# Since PyPy 3 targets Python 3.2, 'raise exc from None' will
+# raise a TypeError so we need to look for Python 3.3 or newer
+if PY33:
     from vine.five import exec_
     _raise_with_context = None  # for flake8
     exec_("""def _raise_with_context(exc, ctx): raise exc from ctx""")


Test patch:
Repository: celery/celery

Issue:
Specifying app/backend in GroupResult.restore is slightly awkward
In Celery 3.x I used `GroupResult.restore('some-id')` and it somehow found my app and figured out the backend (not sure how, but it worked).

In Celery 4.x this does not work (it does not find the backend and crashes). That makes sense to me (I like the idea of explicit apps a lot!). `GroupResult.restore()` does not accept a `app` argument which would be convenient. It does however support a `backend` argument, so I changed my code to use `GroupResult.restore('my-id', backend=app.backend)` and everything is fine.

However, to me this is not that super-intuitive, my suggestion is that we should add an `app` kwarg to `restore()` that can be used like:

```py
GroupResult.restore('someid', app=app)
```

Do you think this makes sense? Then I'd be happy to work on a PR to add the argument!


Discussion:
Yeah, I definitely think that makes sense!  The argument was probably named backend before the concept of apps.
I was not aware, but GroupResult is also available on the app, another way to solve this is to use:
```py
app.GroupResult.restore('someid')
```

Resolving patch:
diff --git a/celery/result.py b/celery/result.py
--- a/celery/result.py
+++ b/celery/result.py
@@ -867,10 +867,13 @@ def children(self):
         return self.results
 
     @classmethod
-    def restore(cls, id, backend=None):
+    def restore(cls, id, backend=None, app=None):
         """Restore previously saved group result."""
+
+        app = app or cls.app
+
         return (
-            backend or (cls.app.backend if cls.app else current_app.backend)
+            backend or (app.backend if app else current_app.backend)
         ).restore_group(id)
 
 


Test patch:
diff --git a/t/unit/tasks/test_result.py b/t/unit/tasks/test_result.py
--- a/t/unit/tasks/test_result.py
+++ b/t/unit/tasks/test_result.py
@@ -19,6 +19,7 @@
     AsyncResult,
     EagerResult,
     ResultSet,
+    GroupResult,
     result_from_tuple,
     assert_will_not_block,
 )
@@ -615,6 +616,13 @@ def test_save_restore(self):
         with pytest.raises(AttributeError):
             self.app.GroupResult.restore(ts.id, backend=object())
 
+    def test_restore_app(self):
+        subs = [MockAsyncResultSuccess(uuid(), app=self.app)]
+        ts = self.app.GroupResult(uuid(), subs)
+        ts.save()
+        restored = GroupResult.restore(ts.id, app=self.app)
+        assert restored.id == ts.id
+
     def test_join_native(self):
         backend = SimpleBackend()
         results = [self.app.AsyncResult(uuid(), backend=backend)
Repository: celery/celery

Issue:
Celery fails to convert tasks w/o kwargs from protocol v1 to v2
InvalidTaskError is being raised when there's no kwargs present in the message.

There's actually a test (https://github.com/celery/celery/blob/master/t/unit/worker/test_strategy.py#L33-L36) ensuring that an`InvalidTaskError` is raised in case there's no kwargs in the task.

This is not what used to happen in v1. Docs also say the following for kwargs:
```
Dictionary of keyword arguments. Will be an empty dictionary if not provided.
```
as you can see in
 http://docs.celeryproject.org/en/latest/internals/protocol.html#id8 

I can open a PR if you guys think this rule should really be removed.

Thanks!



Discussion:
Thanks for reporting this, if it differs from the previous behavior then it's a mistake!

Resolving patch:
diff --git a/celery/worker/strategy.py b/celery/worker/strategy.py
--- a/celery/worker/strategy.py
+++ b/celery/worker/strategy.py
@@ -30,7 +30,7 @@ def proto1_to_proto2(message, body):
         Tuple: of ``(body, headers, already_decoded_status, utc)``
     """
     try:
-        args, kwargs = body['args'], body['kwargs']
+        args, kwargs = body.get('args', ()), body.get('kwargs', {})
         kwargs.items  # pylint: disable=pointless-statement
     except KeyError:
         raise InvalidTaskError('Message does not have args/kwargs')


Test patch:
diff --git a/t/unit/worker/test_strategy.py b/t/unit/worker/test_strategy.py
--- a/t/unit/worker/test_strategy.py
+++ b/t/unit/worker/test_strategy.py
@@ -27,13 +27,13 @@ def setup(self):
 
     def test_message_without_args(self):
         self.body.pop('args')
-        with pytest.raises(InvalidTaskError):
-            proto1_to_proto2(self.message, self.body)
+        body, _, _, _ = proto1_to_proto2(self.message, self.body)
+        assert body[:2] == ((), {'foo': 'baz'})
 
     def test_message_without_kwargs(self):
         self.body.pop('kwargs')
-        with pytest.raises(InvalidTaskError):
-            proto1_to_proto2(self.message, self.body)
+        body, _, _, _ = proto1_to_proto2(self.message, self.body)
+        assert body[:2] == ((1,), {})
 
     def test_message_kwargs_not_mapping(self):
         self.body['kwargs'] = (2,)
Repository: celery/celery

Issue:
mongodb result backend with pickle serialisation broken in 4.0.0
Just upgraded to celery 4.0.0. Mongodb result backend with pickle no longer works with the exception at the end of this message. 

This is because commit https://github.com/celery/celery/commit/639b40f6308267312a1030bb3d6ac5805069510a removes the **essential** Binary() wrapper from the data written to mongo. As a result it writes raw binary data to mongo as a string, which then rejects it.

I'm using settings:
CELERY_ACCEPT_CONTENT = ['pickle']
CELERY_TASK_SERIALIZER = 'pickle'
CELERY_RESULT_SERIALIZER = 'pickle'

Exception seen:
> [2016-11-10 16:01:02,118: ERROR/MainProcess] Pool callback raised exception: InvalidStringData("strings in documents must be valid UTF-8: '\\x80\\x02N.'",)
> Traceback (most recent call last):
>   File "/usr/lib/python2.7/site-packages/billiard/pool.py", line 1748, in safe_apply_callback
>     fun(*args, **kwargs)
>   File "/usr/lib/python2.7/site-packages/celery/worker/request.py", line 366, in on_failure
>     self.id, exc, request=self, store_result=self.store_errors,
>   File "/usr/lib/python2.7/site-packages/celery/backends/base.py", line 163, in mark_as_failure
>     traceback=traceback, request=request)
>   File "/usr/lib/python2.7/site-packages/celery/backends/base.py", line 309, in store_result
>     request=request, **kwargs)
>   File "/usr/lib/python2.7/site-packages/celery/backends/mongodb.py", line 175, in _store_result
>     self.collection.save(meta)
>   File "/usr/lib/python2.7/site-packages/pymongo/collection.py", line 2192, in save
>     check_keys, False, manipulate, write_concern)
>   File "/usr/lib/python2.7/site-packages/pymongo/collection.py", line 715, in _update
>     codec_options=self.__write_response_codec_options).copy()
>   File "/usr/lib/python2.7/site-packages/pymongo/pool.py", line 244, in command
>     self._raise_connection_failure(error)
>   File "/usr/lib/python2.7/site-packages/pymongo/pool.py", line 372, in _raise_connection_failure
>     raise error
> InvalidStringData: strings in documents must be valid UTF-8: '\x80\x02N.'



Discussion:

Resolving patch:
diff --git a/celery/backends/mongodb.py b/celery/backends/mongodb.py
--- a/celery/backends/mongodb.py
+++ b/celery/backends/mongodb.py
@@ -29,6 +29,8 @@ class InvalidDocument(Exception):       # noqa
 
 __all__ = ['MongoBackend']
 
+BINARY_CODECS = frozenset(['pickle', 'msgpack'])
+
 
 class MongoBackend(BaseBackend):
     """MongoDB result backend.
@@ -150,7 +152,12 @@ def encode(self, data):
         if self.serializer == 'bson':
             # mongodb handles serialization
             return data
-        return super(MongoBackend, self).encode(data)
+        payload = super(MongoBackend, self).encode(data)
+
+        # serializer which are in a unsupported format (pickle/binary)
+        if self.serializer in BINARY_CODECS:
+            payload = Binary(payload)
+        return payload
 
     def decode(self, data):
         if self.serializer == 'bson':


Test patch:
Repository: celery/celery

Issue:
celery.utils.time.maybe_make_aware docstring incorrect
The docstring for [celery.utils.time.maybe_make_aware](https://github.com/celery/celery/blob/master/celery/utils/time.py#L321) claims that it doesn't do anything if dt is already timezone-aware, but this is incorrect, as it will try to convert to the localtime or utc instead.  This doesn't look like correct behavior, and doesn't allow timezones to work correctly in the crontab scheduler.


Discussion:

Resolving patch:
diff --git a/celery/utils/time.py b/celery/utils/time.py
--- a/celery/utils/time.py
+++ b/celery/utils/time.py
@@ -322,9 +322,10 @@ def maybe_make_aware(dt, tz=None):
     """Convert dt to aware datetime, do nothing if dt is already aware."""
     if is_naive(dt):
         dt = to_utc(dt)
-    return localize(
-        dt, timezone.utc if tz is None else timezone.tz_or_local(tz),
-    )
+        return localize(
+            dt, timezone.utc if tz is None else timezone.tz_or_local(tz),
+        )
+    return dt
 
 
 @python_2_unicode_compatible


Test patch:
diff --git a/t/unit/utils/test_time.py b/t/unit/utils/test_time.py
--- a/t/unit/utils/test_time.py
+++ b/t/unit/utils/test_time.py
@@ -171,6 +171,13 @@ def test_maybe_make_aware(self):
         assert maybe_make_aware(aware)
         naive = datetime.utcnow()
         assert maybe_make_aware(naive)
+        assert maybe_make_aware(naive).tzinfo is pytz.utc
+
+        tz = pytz.timezone('US/Eastern')
+        eastern = datetime.utcnow().replace(tzinfo=tz)
+        assert maybe_make_aware(eastern).tzinfo is tz
+        utcnow = datetime.utcnow()
+        assert maybe_make_aware(utcnow, 'UTC').tzinfo is pytz.utc
 
 
 class test_localize:
Repository: celery/celery

Issue:
celery.contrib.sphinx fails with Sphinx 1.6.1
When using the `celery.contrib.sphinx` extension with Sphinx 1.6.1 with Celery 4.0.2 the following occurs:

```
Exception occurred:
  File "/home/ubuntu/virtualenvs/venv-system/lib/python2.7/site-packages/celery/contrib/sphinx.py", line 72, in setup
    app.domains['py'].directives['task'] = TaskDirective
AttributeError: 'Sphinx' object has no attribute 'domains'
The full traceback has been saved in /tmp/sphinx-err-oOWabx.log, if you want to report the issue to the developers.
Please also report this if it was a user error, so that a better error message can be provided next time.
A bug report can be filed in the tracker at <https://github.com/sphinx-doc/sphinx/issues>. Thanks!
make: *** [html] Error 1
```

The `domains` property seems to have been removed in sphinx-doc/sphinx#3656 and I think this line needs to be replaced with the [`add_directive` method](http://www.sphinx-doc.org/en/stable/extdev/appapi.html#sphinx.application.Sphinx.add_directive) (or more likely the [`add_directive_to_domain` method](http://www.sphinx-doc.org/en/stable/extdev/appapi.html#sphinx.application.Sphinx.add_directive_to_domain)).


Discussion:

Resolving patch:
diff --git a/celery/contrib/sphinx.py b/celery/contrib/sphinx.py
--- a/celery/contrib/sphinx.py
+++ b/celery/contrib/sphinx.py
@@ -69,5 +69,5 @@ def get_signature_prefix(self, sig):
 def setup(app):
     """Setup Sphinx extension."""
     app.add_autodocumenter(TaskDocumenter)
-    app.domains['py'].directives['task'] = TaskDirective
+    app.add_directive_to_domain('py', 'task', TaskDirective)
     app.add_config_value('celery_task_prefix', '(task)', True)


Test patch:
Repository: celery/celery

Issue:
-u option does not exist
## Steps to reproduce

Start -> celery -A application worker -l info

## Actual behavior

RuntimeWarning: You're running the worker with superuser privileges: this is
absolutely not recommended!

Please specify a different user using the -u option.

User information: uid=0 euid=0 gid=0 egid=0

  uid=uid, euid=euid, gid=gid, egid=egid

## Fixes

When displaying the help menu -> celery -A application worker -l info -h

There is currently no longer a -u option and the warning should be changed to use --uid / --gid options


Discussion:
Nice observation. Would you like to open a pull request for this?

Resolving patch:
diff --git a/celery/platforms.py b/celery/platforms.py
--- a/celery/platforms.py
+++ b/celery/platforms.py
@@ -86,7 +86,7 @@
 You're running the worker with superuser privileges: this is
 absolutely not recommended!
 
-Please specify a different user using the -u option.
+Please specify a different user using the --uid option.
 
 User information: uid={uid} euid={euid} gid={gid} egid={egid}
 """


Test patch:
Repository: celery/celery

Issue:
[BUG] Exception on single task in group.
## Checklist
```
// I have included the output of ``celery -A proj report`` in the issue.
//      (if you are not able to do this, then at least specify the Celery
//       version affected).
//
// Doesn't make sense as it client-side issue.
```
- [x] I have verified that the issue exists against the `master` branch of Celery.

## Steps to reproduce
On client send a group with single task in it
```python3.6
def fta_add_to_pool(applications_id: List[int]):
    return group(
        *[
            app.signature('fta_tasks.add_to_pool', args=[application_id])
            for application_id in applications_id
        ]
    ).apply_async().get()
```

## Expected behavior
I expected group to execute with a single task like there was no group.

## Actual behavior
Exception:
```
  File "***/src/celery/celery/canvas.py", line 976, in apply_async
    app = self.app
  File "***/src/celery/celery/canvas.py", line 1140, in app
    app = self.tasks[0].app
  File "***/src/celery/celery/utils/functional.py", line 213, in __getitem__
    self.__consumed.append(next(self.__it))
TypeError: 'Signature' object is not an iterator```
```

## Comment
It should be easy to fix but I'm not sure about celery typing and etc, so I decided to make the issue.

https://github.com/celery/celery/blob/39d86e7dbe7f3a0a43e973910be880146de96fb7/celery/canvas.py#L948-L958

We just need to check if this single task is Task and not something else.


Discussion:

Resolving patch:
diff --git a/celery/canvas.py b/celery/canvas.py
--- a/celery/canvas.py
+++ b/celery/canvas.py
@@ -950,6 +950,8 @@ def __init__(self, *tasks, **options):
             tasks = tasks[0]
             if isinstance(tasks, group):
                 tasks = tasks.tasks
+            if isinstance(tasks, abstract.CallableSignature):
+                tasks = [tasks.clone()]
             if not isinstance(tasks, _regen):
                 tasks = regen(tasks)
         Signature.__init__(


Test patch:
diff --git a/t/unit/tasks/test_canvas.py b/t/unit/tasks/test_canvas.py
--- a/t/unit/tasks/test_canvas.py
+++ b/t/unit/tasks/test_canvas.py
@@ -547,6 +547,14 @@ def test_iter(self):
         g = group([self.add.s(i, i) for i in range(10)])
         assert list(iter(g)) == list(g.keys())
 
+    def test_single_task(self):
+        g = group([self.add.s(1, 1)])
+        assert isinstance(g, group)
+        assert len(g.tasks) == 1
+        g = group(self.add.s(1, 1))
+        assert isinstance(g, group)
+        assert len(g.tasks) == 1
+
     @staticmethod
     def helper_test_get_delay(result):
         import time