Repository: celery/celery

Issue:
[CassandraBackend.process_cleanup] is unnecessary
<!--
Please fill this template entirely and do not erase parts of it.
We reserve the right to close without a response
bug reports which are incomplete.
-->
# Checklist
<!--
To check an item on the list replace [ ] with [x].
-->
- [x] I have verified that the issue exists against the `master` branch of Celery.
- [ ] This has already been asked to the [discussion group](https://groups.google.com/forum/#!forum/celery-users) first.
- [x] I have read the relevant section in the
  [contribution guide](http://docs.celeryproject.org/en/latest/contributing.html#other-bugs)
  on reporting bugs.
- [x] I have checked the [issues list](https://github.com/celery/celery/issues?q=is%3Aissue+label%3A%22Issue+Type%3A+Bug+Report%22+-label%3A%22Category%3A+Documentation%22)
  for similar or identical bug reports.
- [x] I have checked the [pull requests list](https://github.com/celery/celery/pulls?q=is%3Apr+label%3A%22PR+Type%3A+Bugfix%22+-label%3A%22Category%3A+Documentation%22)
  for existing proposed fixes.
- [x] I have checked the [commit log](https://github.com/celery/celery/commits/master)
  to find out if the bug was already fixed in the master branch.
- [x] I have included all related issues and possible duplicate issues
  in this issue (If there are none, check this box anyway).

## Mandatory Debugging Information

- [x] I have included the output of ``celery -A proj report`` in the issue.
    (if you are not able to do this, then at least specify the Celery
     version affected).
- [x] I have verified that the issue exists against the `master` branch of Celery.
- [x] I have included the contents of ``pip freeze`` in the issue.
- [ ] I have included all the versions of all the external dependencies required
  to reproduce this bug.

## Optional Debugging Information
<!--
Try some of the below if you think they are relevant.
It will help us figure out the scope of the bug and how many users it affects.
-->
- [ ] I have tried reproducing the issue on more than one Python version
  and/or implementation.
- [ ] I have tried reproducing the issue on more than one message broker and/or
  result backend.
- [ ] I have tried reproducing the issue on more than one version of the message
  broker and/or result backend.
- [ ] I have tried reproducing the issue on more than one operating system.
- [ ] I have tried reproducing the issue on more than one workers pool.
- [ ] I have tried reproducing the issue with autoscaling, retries,
  ETA/Countdown & rate limits disabled.
- [ ] I have tried reproducing the issue after downgrading
  and/or upgrading Celery and its dependencies.

## Related Issues and Possible Duplicates
<!--
Please make sure to search and mention any related issues
or possible duplicates to this issue as requested by the checklist above.

This may or may not include issues in other repositories that the Celery project
maintains or other repositories that are dependencies of Celery.

If you don't know how to mention issues, please refer to Github's documentation
on the subject: https://help.github.com/en/articles/autolinked-references-and-urls#issues-and-pull-requests
-->

#### Related Issues

- None

#### Possible Duplicates

- None

## Environment & Settings
<!-- Include the contents of celery --version below -->
**Celery version**:
<!-- Include the output of celery -A proj report below -->

<details>
<summary><b><code>celery report</code> Output:</b></summary>
<p>

```
software -> celery:4.4.3 (cliffs) kombu:4.6.9 py:3.7.6
            billiard:3.6.3.0 py-amqp:2.6.0
platform -> system:Linux arch:64bit, ELF
            kernel version:4.15.0-101-generic imp:CPython
loader   -> celery.loaders.app.AppLoader
settings -> transport:amqp results:cassandra

Queue: <class 'kombu.entity.Queue'>
RoundRobinPolicy: <class 'cassandra.policies.RoundRobinPolicy'>
broker_url: 'amqp://guest:********@localhost:5672//'
cassandra_auth_kwargs: {
 'password': '********', 'username': 'cassandra'}
cassandra_auth_provider: 'PlainTextAuthProvider'
cassandra_entry_ttl: 86400
cassandra_keyspace: '********'
cassandra_options: {
    'load_balancing_policy': <cassandra.policies.RoundRobinPolicy object at 0x7f09de9a34d0>}
cassandra_servers: ['127.0.0.1']
cassandra_table: 'celery_jobs_result'
ignore_result: False
result_backend: 'cassandra'
worker_concurrency: 4
```

</p>
</details>

# Steps to Reproduce

## Required Dependencies
<!-- Please fill the required dependencies to reproduce this issue -->
* **Minimal Python Version**: N/A or Unknown
* **Minimal Celery Version**: N/A or Unknown
* **Minimal Kombu Version**: N/A or Unknown
* **Minimal Broker Version**: N/A or Unknown
* **Minimal Result Backend Version**: N/A or Unknown
* **Minimal OS and/or Kernel Version**: N/A or Unknown
* **Minimal Broker Client Version**: N/A or Unknown
* **Minimal Result Backend Client Version**: N/A or Unknown

### Python Packages
<!-- Please fill the contents of pip freeze below -->
<details>
<summary><b><code>pip freeze</code> Output:</b></summary>
<p>

```
amqp==2.6.0
attrs==19.3.0
billiard==3.6.3.0
case==1.5.3
cassandra-driver==3.20.2
future==0.18.2
gevent==20.5.2
greenlet==0.4.15
importlib-metadata==1.6.0
kombu==4.6.9
linecache2==1.0.0
mock==4.0.2
more-itertools==8.3.0
nose==1.3.7
packaging==20.4
pluggy==0.13.1
py==1.8.1
pyparsing==2.4.7
pytest==5.4.2
pytz==2020.1
redis==3.5.2
six==1.15.0
traceback2==1.4.0
unittest2==1.1.0
vine==1.3.0
wcwidth==0.1.9
zipp==3.1.0
zope.event==4.4
zope.interface==5.1.0
```

</p>
</details>

### Other Dependencies
<!--
Please provide system dependencies, configuration files
and other dependency information if applicable
-->
<details>
<p>
N/A
</p>
</details>

## Minimally Reproducible Test Case
<!--
Please provide a reproducible test case.
Refer to the Reporting Bugs section in our contribution guide.

We prefer submitting test cases in the form of a PR to our integration test suite.
If you can provide one, please mention the PR number below.
If not, please attach the most minimal code example required to reproduce the issue below.
If the test case is too large, please include a link to a gist or a repository below.
-->

<details>
<p>

```python
```

</p>
</details>

# Expected Behavior

CassandraBackend work fine

# Actual Behavior

## in prefork pool or solo:

it can work, just not good, i think

## in threads or gevent pool:

get these error frequently

<p>

```
[2020-06-02 21:52:19,592: ERROR/MainProcess] Task add[4bd528a0-c1ba-456c-b74e-7dee054616bb] raised unexpected: AttributeError("'NoneType' object has no attribute 'execute'")
Traceback (most recent call last):
  File "/home/bx/project/celery/celery/app/trace.py", line 480, in trace_task
    uuid, retval, task_request, publish_result,
  File "/home/bx/project/celery/celery/backends/base.py", line 158, in mark_as_done
    self.store_result(task_id, result, state, request=request)
  File "/home/bx/project/celery/celery/backends/base.py", line 443, in store_result
    request=request, **kwargs)
  File "/home/bx/project/celery/celery/backends/cassandra.py", line 201, in _store_result
    self._session.execute(self._write_stmt, (
AttributeError: 'NoneType' object has no attribute 'execute'
```
<p>

and 

<p>

```
[2020-06-02 21:52:29,773: ERROR/MainProcess] Task add[27c980d3-126a-430d-a8e0-92d749ceb735] raised unexpected: OperationTimedOut("errors=errors={'Connection defunct by heartbeat': 'Client request timeout. See Session.execute[_async](timeout)'}, last_host=127.0.0.1:9042, last_host=None")
Traceback (most recent call last):
  File "/home/bx/project/celery/celery/app/trace.py", line 480, in trace_task
    uuid, retval, task_request, publish_result,
  File "/home/bx/project/celery/celery/backends/base.py", line 158, in mark_as_done
    self.store_result(task_id, result, state, request=request)
  File "/home/bx/project/celery/celery/backends/base.py", line 443, in store_result
    request=request, **kwargs)
  File "/home/bx/project/celery/celery/backends/cassandra.py", line 207, in _store_result
    buf_t(self.encode(self.current_task_children(request)))
  File "cassandra/cluster.py", line 2345, in cassandra.cluster.Session.execute
  File "cassandra/cluster.py", line 4304, in cassandra.cluster.ResponseFuture.result
cassandra.OperationTimedOut: errors=errors={'Connection defunct by heartbeat': 'Client request timeout. See Session.execute[_async](timeout)'}, last_host=127.0.0.1:9042, last_host=None
```

</p>

# Fix it

remove  method(CassandraBackend.process_cleanup), it works fine.

because.

 all tasks should share one instance of cassandra.cluster.Cluster and session,   not create one ervery time. that is the correct way to use cassandra-driver



Discussion:
are you up for a PR?
@auvipy I am improving it ,  and up a PR later 
looking forward to it.

Resolving patch:
diff --git a/celery/backends/cassandra.py b/celery/backends/cassandra.py
--- a/celery/backends/cassandra.py
+++ b/celery/backends/cassandra.py
@@ -3,6 +3,7 @@
 from __future__ import absolute_import, unicode_literals
 
 import sys
+import threading
 
 from celery import states
 from celery.exceptions import ImproperlyConfigured
@@ -14,6 +15,7 @@
     import cassandra
     import cassandra.auth
     import cassandra.cluster
+    import cassandra.query
 except ImportError:  # pragma: no cover
     cassandra = None   # noqa
 
@@ -123,17 +125,11 @@ def __init__(self, servers=None, keyspace=None, table=None, entry_ttl=None,
                 raise ImproperlyConfigured(E_NO_SUCH_CASSANDRA_AUTH_PROVIDER)
             self.auth_provider = auth_provider_class(**auth_kwargs)
 
-        self._connection = None
+        self._cluster = None
         self._session = None
         self._write_stmt = None
         self._read_stmt = None
-        self._make_stmt = None
-
-    def process_cleanup(self):
-        if self._connection is not None:
-            self._connection.shutdown()  # also shuts down _session
-        self._connection = None
-        self._session = None
+        self._lock = threading.RLock()
 
     def _get_connection(self, write=False):
         """Prepare the connection for action.
@@ -141,14 +137,17 @@ def _get_connection(self, write=False):
         Arguments:
             write (bool): are we a writer?
         """
-        if self._connection is not None:
+        if self._session is not None:
             return
+        self._lock.acquire()
         try:
-            self._connection = cassandra.cluster.Cluster(
+            if self._session is not None:
+                return
+            self._cluster = cassandra.cluster.Cluster(
                 self.servers, port=self.port,
                 auth_provider=self.auth_provider,
                 **self.cassandra_options)
-            self._session = self._connection.connect(self.keyspace)
+            self._session = self._cluster.connect(self.keyspace)
 
             # We're forced to do concatenation below, as formatting would
             # blow up on superficial %s that'll be processed by Cassandra
@@ -172,25 +171,27 @@ def _get_connection(self, write=False):
                 # Anyway; if you're doing anything critical, you should
                 # have created this table in advance, in which case
                 # this query will be a no-op (AlreadyExists)
-                self._make_stmt = cassandra.query.SimpleStatement(
+                make_stmt = cassandra.query.SimpleStatement(
                     Q_CREATE_RESULT_TABLE.format(table=self.table),
                 )
-                self._make_stmt.consistency_level = self.write_consistency
+                make_stmt.consistency_level = self.write_consistency
 
                 try:
-                    self._session.execute(self._make_stmt)
+                    self._session.execute(make_stmt)
                 except cassandra.AlreadyExists:
                     pass
 
         except cassandra.OperationTimedOut:
             # a heavily loaded or gone Cassandra cluster failed to respond.
             # leave this class in a consistent state
-            if self._connection is not None:
-                self._connection.shutdown()     # also shuts down _session
+            if self._cluster is not None:
+                self._cluster.shutdown()     # also shuts down _session
 
-            self._connection = None
+            self._cluster = None
             self._session = None
             raise   # we did fail after all - reraise
+        finally:
+            self._lock.release()
 
     def _store_result(self, task_id, result, state,
                       traceback=None, request=None, **kwargs):


Test patch:
diff --git a/t/unit/backends/test_cassandra.py b/t/unit/backends/test_cassandra.py
--- a/t/unit/backends/test_cassandra.py
+++ b/t/unit/backends/test_cassandra.py
@@ -10,7 +10,12 @@
 from celery.exceptions import ImproperlyConfigured
 from celery.utils.objects import Bunch
 
-CASSANDRA_MODULES = ['cassandra', 'cassandra.auth', 'cassandra.cluster']
+CASSANDRA_MODULES = [
+    'cassandra',
+    'cassandra.auth',
+    'cassandra.cluster',
+    'cassandra.query',
+]
 
 
 @mock.module(*CASSANDRA_MODULES)
@@ -66,7 +71,6 @@ def test_get_task_meta_for(self, *modules):
         mod.cassandra = Mock()
 
         x = mod.CassandraBackend(app=self.app)
-        x._connection = True
         session = x._session = Mock()
         execute = session.execute = Mock()
         result_set = Mock()
@@ -83,24 +87,24 @@ def test_get_task_meta_for(self, *modules):
         meta = x._get_task_meta_for('task_id')
         assert meta['status'] == states.PENDING
 
+    def test_as_uri(self):
+        # Just ensure as_uri works properly
+        from celery.backends import cassandra as mod
+        mod.cassandra = Mock()
+
+        x = mod.CassandraBackend(app=self.app)
+        x.as_uri()
+        x.as_uri(include_password=False)
+
     def test_store_result(self, *modules):
         from celery.backends import cassandra as mod
         mod.cassandra = Mock()
 
         x = mod.CassandraBackend(app=self.app)
-        x._connection = True
         session = x._session = Mock()
         session.execute = Mock()
         x._store_result('task_id', 'result', states.SUCCESS)
 
-    def test_process_cleanup(self, *modules):
-        from celery.backends import cassandra as mod
-        x = mod.CassandraBackend(app=self.app)
-        x.process_cleanup()
-
-        assert x._connection is None
-        assert x._session is None
-
     def test_timeouting_cluster(self):
         # Tests behavior when Cluster.connect raises
         # cassandra.OperationTimedOut.
@@ -128,40 +132,65 @@ def shutdown(self):
 
         with pytest.raises(OTOExc):
             x._store_result('task_id', 'result', states.SUCCESS)
-        assert x._connection is None
+        assert x._cluster is None
         assert x._session is None
 
-        x.process_cleanup()  # shouldn't raise
-
-    def test_please_free_memory(self):
-        # Ensure that Cluster object IS shut down.
+    def test_create_result_table(self):
+        # Tests behavior when session.execute raises
+        # cassandra.AlreadyExists.
         from celery.backends import cassandra as mod
 
-        class RAMHoggingCluster(object):
+        class OTOExc(Exception):
+            pass
 
-            objects_alive = 0
+        class FaultySession(object):
+            def __init__(self, *args, **kwargs):
+                pass
+
+            def execute(self, *args, **kwargs):
+                raise OTOExc()
+
+        class DummyCluster(object):
 
             def __init__(self, *args, **kwargs):
                 pass
 
             def connect(self, *args, **kwargs):
-                RAMHoggingCluster.objects_alive += 1
-                return Mock()
-
-            def shutdown(self):
-                RAMHoggingCluster.objects_alive -= 1
+                return FaultySession()
 
         mod.cassandra = Mock()
+        mod.cassandra.cluster = Mock()
+        mod.cassandra.cluster.Cluster = DummyCluster
+        mod.cassandra.AlreadyExists = OTOExc
+
+        x = mod.CassandraBackend(app=self.app)
+        x._get_connection(write=True)
+        assert x._session is not None
+
+    def test_init_session(self):
+        # Tests behavior when Cluster.connect works properly
+        from celery.backends import cassandra as mod
+
+        class DummyCluster(object):
+
+            def __init__(self, *args, **kwargs):
+                pass
+
+            def connect(self, *args, **kwargs):
+                return Mock()
 
+        mod.cassandra = Mock()
         mod.cassandra.cluster = Mock()
-        mod.cassandra.cluster.Cluster = RAMHoggingCluster
+        mod.cassandra.cluster.Cluster = DummyCluster
 
-        for x in range(0, 10):
-            x = mod.CassandraBackend(app=self.app)
-            x._store_result('task_id', 'result', states.SUCCESS)
-            x.process_cleanup()
+        x = mod.CassandraBackend(app=self.app)
+        assert x._session is None
+        x._get_connection(write=True)
+        assert x._session is not None
 
-        assert RAMHoggingCluster.objects_alive == 0
+        s = x._session
+        x._get_connection()
+        assert s is x._session
 
     def test_auth_provider(self):
         # Ensure valid auth_provider works properly, and invalid one raises