Repository: celery/celery

Issue:
Celery does not consider authMechanism on mongodb backend URLs
<!--
Please fill this template entirely and do not erase parts of it.
We reserve the right to close without a response
bug reports which are incomplete.
-->
# Checklist
<!--
To check an item on the list replace [ ] with [x].
-->

- [x] I have read the relevant section in the
  [contribution guide](http://docs.celeryproject.org/en/latest/contributing.html#other-bugs)
  on reporting bugs.
- [x] I have checked the [issues list](https://github.com/celery/celery/issues?q=is%3Aissue+label%3A%22Issue+Type%3A+Bug+Report%22+-label%3A%22Category%3A+Documentation%22)
  for similar or identical bug reports.
- [x] I have checked the [pull requests list](https://github.com/celery/celery/pulls?q=is%3Apr+label%3A%22PR+Type%3A+Bugfix%22+-label%3A%22Category%3A+Documentation%22)
  for existing proposed fixes.
- [x] I have checked the [commit log](https://github.com/celery/celery/commits/master)
  to find out if the bug was already fixed in the master branch.
- [x] I have included all related issues and possible duplicate issues
  in this issue (If there are none, check this box anyway).

## Mandatory Debugging Information

- [x] I have included the output of ``celery -A proj report`` in the issue.
    (if you are not able to do this, then at least specify the Celery
     version affected).
- [x] I have verified that the issue exists against the `master` branch of Celery.
- [x] I have included the contents of ``pip freeze`` in the issue.
- [x] I have included all the versions of all the external dependencies required
  to reproduce this bug.

## Optional Debugging Information
<!--
Try some of the below if you think they are relevant.
It will help us figure out the scope of the bug and how many users it affects.
-->
- [ ] I have tried reproducing the issue on more than one Python version
  and/or implementation.
- [ ] I have tried reproducing the issue on more than one message broker and/or
  result backend.
- [ ] I have tried reproducing the issue on more than one version of the message
  broker and/or result backend.
- [ ] I have tried reproducing the issue on more than one operating system.
- [ ] I have tried reproducing the issue on more than one workers pool.
- [ ] I have tried reproducing the issue with autoscaling, retries,
  ETA/Countdown & rate limits disabled.
- [ ] I have tried reproducing the issue after downgrading
  and/or upgrading Celery and its dependencies.

## Related Issues and Possible Duplicates
<!--
Please make sure to search and mention any related issues
or possible duplicates to this issue as requested by the checklist above.

This may or may not include issues in other repositories that the Celery project
maintains or other repositories that are dependencies of Celery.

If you don't know how to mention issues, please refer to Github's documentation
on the subject: https://help.github.com/en/articles/autolinked-references-and-urls#issues-and-pull-requests
-->

#### Related Issues

- #4454 

#### Possible Duplicates

- None

## Environment & Settings
<!-- Include the contents of celery --version below -->
**Celery version**: 4.3.0 with fixes in PR #5527 
<!-- Include the output of celery -A proj report below -->
<details>
<summary><b><code>celery report</code> Output:</b></summary>
<p>

```
software -> celery:4.3.0 (rhubarb) kombu:4.6.4 py:3.7.4
            billiard:3.6.1.0 py-amqp:2.5.1
platform -> system:Windows arch:64bit, WindowsPE
            kernel version:10 imp:CPython
loader   -> celery.loaders.default.Loader
settings -> transport:amqp results:disabled
```

</p>
</details>

# Steps to Reproduce
1. Give Celery a Backend URL pointing to a MongoDB like below:
mongodb://admin:admin@192.168.0.1/task?authSource=admin&authMechanism=SCRAM-SHA-256
2. Start Celery worker.
3. Send any task.

## Required Dependencies
<!-- Please fill the required dependencies to reproduce this issue -->
* **Minimal Python Version**: 3.0
* **Minimal Celery Version**: 3.0
* **Minimal Kombu Version**: Unknown
* **Minimal Broker Version**: Unknown
* **Minimal Result Backend Version**: MongoDB 4.0
* **Minimal OS and/or Kernel Version**: N/A
* **Minimal Broker Client Version**: Unknown
* **Minimal Result Backend Client Version**: Unknown

### Python Packages
<!-- Please fill the contents of pip freeze below -->
<details>
<summary><b><code>pip freeze</code> Output:</b></summary>
<p>

```
amqp==2.5.1
asn1crypto==0.24.0
astroid==2.2.5
bcrypt==3.1.7
billiard==3.6.1.0
celery==4.3.0
cffi==1.12.3
Click==7.0
colorama==0.4.1
cryptography==2.7
irectory=backend
Flask==1.1.1
importlib-metadata==0.19
isort==4.3.21
itsdangerous==1.1.0
Jinja2==2.10.1
kombu==4.6.4
lazy-object-proxy==1.4.2
MarkupSafe==1.1.1
mccabe==0.6.1
more-itertools==7.2.0
paramiko==2.6.0
pycparser==2.19
pylint==2.3.1
pymodm==0.4.1
pymongo==3.9.0
PyNaCl==1.3.0
pytz==2019.2
six==1.12.0
typed-ast==1.4.0
vine==1.3.0
Werkzeug==0.15.5
wrapt==1.11.2
zipp==0.6.0
```

</p>
</details>

### Other Dependencies
<!--
Please provide system dependencies, configuration files
and other dependency information if applicable
-->
<details>
<summary><b>MongoDB:</b></summary>
<p>
Version: >= 3.0.0
</p>
</details>

## Minimally Reproducible Test Case
<!--
Please provide a reproducible test case.
Refer to the Reporting Bugs section in our contribution guide.

We prefer submitting test cases in the form of a PR to our integration test suite.
If you can provide one, please mention the PR number below.
If not, please attach the most minimal code example required to reproduce the issue below.
If the test case is too large, please include a link to a gist or a repository below.
-->

<details>
<p>

```python
```

</p>
</details>

# Expected Behavior
<!-- Describe in detail what you expect to happen -->
Support all the authentication options listed at [authentication-options](https://docs.mongodb.com/manual/reference/connection-string/#authentication-options), including _authSource_, _authMechanism_, _authMechanismProperties_ and _gssapiServiceName_.

# Actual Behavior
<!--
Describe in detail what actually happened.
Please include a backtrace and surround it with triple backticks (```).
In addition, include the Celery daemon logs, the broker logs,
the result backend logs and system logs below if they will help us debug
the issue.
-->
An error occurred as below.
```python
Traceback (most recent call last):
  File "D:\GitHub\org\dev\.venv\lib\site-packages\kombu\utils\objects.py", line 42, in __get__
    return obj.__dict__[self.__name__]
KeyError: 'collection'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "D:\GitHub\org\dev\.venv\lib\site-packages\kombu\utils\objects.py", line 42, in __get__
    return obj.__dict__[self.__name__]
KeyError: 'database'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "c:\Users\user\.vscode\extensions\ms-python.python-2019.9.34911\pythonFiles\ptvsd_launcher.py", line 43, in <module>
    main(ptvsdArgs)
  File "c:\Users\user\.vscode\extensions\ms-python.python-2019.9.34911\pythonFiles\lib\python\ptvsd\__main__.py", line 432, in main
    run()
  File "c:\Users\user\.vscode\extensions\ms-python.python-2019.9.34911\pythonFiles\lib\python\ptvsd\__main__.py", line 316, in run_file
    runpy.run_path(target, run_name='__main__')
  File "C:\Program Files\Python37\lib\runpy.py", line 263, in run_path
    pkg_name=pkg_name, script_name=fname)
  File "C:\Program Files\Python37\lib\runpy.py", line 96, in _run_module_code
    mod_name, mod_spec, pkg_name, script_name)
  File "C:\Program Files\Python37\lib\runpy.py", line 85, in _run_code
    exec(code, run_globals)
  File "d:\GitHub\org\dev\backend\tests\workertest.py", line 22, in <module>
    print('Task finished? ', result.ready())
  File "D:\GitHub\org\dev\.venv\lib\site-packages\celery\result.py", line 313, in ready
    return self.state in self.backend.READY_STATES
  File "D:\GitHub\org\dev\.venv\lib\site-packages\celery\result.py", line 473, in state
    return self._get_task_meta()['status']
  File "D:\GitHub\org\dev\.venv\lib\site-packages\celery\result.py", line 412, in _get_task_meta
    return self._maybe_set_cache(self.backend.get_task_meta(self.id))
  File "D:\GitHub\org\dev\.venv\lib\site-packages\celery\backends\base.py", line 386, in get_task_meta
    meta = self._get_task_meta_for(task_id)
  File "D:\GitHub\org\dev\.venv\lib\site-packages\celery\backends\mongodb.py", line 206, in _get_task_meta_for
    obj = self.collection.find_one({'_id': task_id})
  File "D:\GitHub\org\dev\.venv\lib\site-packages\kombu\utils\objects.py", line 44, in __get__
    value = obj.__dict__[self.__name__] = self.__get(obj)
  File "D:\GitHub\org\dev\.venv\lib\site-packages\celery\backends\mongodb.py", line 293, in collection
    collection = self.database[self.taskmeta_collection]
  File "D:\GitHub\org\dev\.venv\lib\site-packages\kombu\utils\objects.py", line 44, in __get__
    value = obj.__dict__[self.__name__] = self.__get(obj)
  File "D:\GitHub\org\dev\.venv\lib\site-packages\celery\backends\mongodb.py", line 288, in database
    return self._get_database()
  File "D:\GitHub\org\dev\.venv\lib\site-packages\celery\backends\mongodb.py", line 271, in _get_database
    conn = self._get_connection()
  File "D:\GitHub\org\dev\.venv\lib\site-packages\celery\backends\mongodb.py", line 161, in _get_connection
    self._connection = MongoClient(**conf)
  File "D:\GitHub\org\dev\.venv\lib\site-packages\pymongo\mongo_client.py", line 668, in __init__
    username, password, dbase, opts)
  File "D:\GitHub\org\dev\.venv\lib\site-packages\pymongo\client_options.py", line 151, in __init__
    username, password, database, options)
  File "D:\GitHub\org\dev\.venv\lib\site-packages\pymongo\client_options.py", line 39, in _parse_credentials
    mechanism, source, username, password, options, database)
  File "D:\GitHub\org\dev\.venv\lib\site-packages\pymongo\auth.py", line 107, in _build_credentials_tuple
    raise ConfigurationError("%s requires a username." % (mech,))
pymongo.errors.ConfigurationError: SCRAM-SHA-256 requires a username.
```


Discussion:
   raise ConfigurationError("%s requires a username." % (mech,))
pymongo.errors.ConfigurationError: SCRAM-SHA-256 requires a username. did you provide the username?
@auvipy, thanks for your reply. The username has been provided in the connection string as 'mongodb://admin:admin@192.168.56.104/task?authSource=admin&authMechanism=SCRAM-SHA-256'. Username must be provided if 'authMechanism' is used while connecting to MongoDB by calling MongoClient at line 161 in 'celery\backends\mongodb.py', but it is missing.
```python
self._connection = MongoClient(**conf)
```
The value of conf is {'maxPoolSize': 10, 'authsource': 'admin', 'authmechanism': 'SCRAM-SHA-256', 'host': ['192.168.56.104:27017']}, there are no 'username' and 'password'. I've found a solution to fix this issue, will create a pull request later.


Resolving patch:
diff --git a/celery/backends/mongodb.py b/celery/backends/mongodb.py
--- a/celery/backends/mongodb.py
+++ b/celery/backends/mongodb.py
@@ -157,6 +157,10 @@ def _get_connection(self):
             # don't change self.options
             conf = dict(self.options)
             conf['host'] = host
+            if self.user:
+                conf['username'] = self.user
+            if self.password:
+                conf['password'] = self.password
 
             self._connection = MongoClient(**conf)
 


Test patch:
diff --git a/t/unit/backends/test_mongodb.py b/t/unit/backends/test_mongodb.py
--- a/t/unit/backends/test_mongodb.py
+++ b/t/unit/backends/test_mongodb.py
@@ -5,6 +5,7 @@
 
 import pytest
 from kombu.exceptions import EncodeError
+from pymongo.errors import ConfigurationError
 
 from case import ANY, MagicMock, Mock, mock, patch, sentinel, skip
 from celery import states, uuid
@@ -220,6 +221,42 @@ def test_get_connection_no_connection_mongodb_uri(self):
             )
             assert sentinel.connection == connection
 
+    def test_get_connection_with_authmechanism(self):
+        with patch('pymongo.MongoClient') as mock_Connection:
+            self.app.conf.mongodb_backend_settings = None
+            uri = ('mongodb://'
+                   'celeryuser:celerypassword@'
+                   'localhost:27017/'
+                   'celerydatabase?authMechanism=SCRAM-SHA-256')
+            mb = MongoBackend(app=self.app, url=uri)
+            mock_Connection.return_value = sentinel.connection
+            connection = mb._get_connection()
+            mock_Connection.assert_called_once_with(
+                host=['localhost:27017'],
+                username='celeryuser',
+                password='celerypassword',
+                authmechanism='SCRAM-SHA-256',
+                **mb._prepare_client_options()
+            )
+            assert sentinel.connection == connection
+
+    def test_get_connection_with_authmechanism_no_username(self):
+        with patch('pymongo.MongoClient') as mock_Connection:
+            self.app.conf.mongodb_backend_settings = None
+            uri = ('mongodb://'
+                   'localhost:27017/'
+                   'celerydatabase?authMechanism=SCRAM-SHA-256')
+            mb = MongoBackend(app=self.app, url=uri)
+            mock_Connection.side_effect = ConfigurationError(
+                'SCRAM-SHA-256 requires a username.')
+            with pytest.raises(ConfigurationError):
+                mb._get_connection()
+            mock_Connection.assert_called_once_with(
+                host=['localhost:27017'],
+                authmechanism='SCRAM-SHA-256',
+                **mb._prepare_client_options()
+            )
+
     @patch('celery.backends.mongodb.MongoBackend._get_connection')
     def test_get_database_no_existing(self, mock_get_connection):
         # Should really check for combinations of these two, to be complete.