Repository: celery/celery

Issue:
new_cassandra + Celery leaks memory
Continuing performing stress test I ran into a situation where next jobs done using Celery + new_cassandra backend leak memory.

Running tasks I get bigger and bigger ru_maxrss value

```
[2015-10-31 21:24:46,598: WARNING/Worker-1] resource.struct_rusage(ru_utime=0.010111, ru_stime=0.00629, ru_maxrss=25748, ru_ixrss=0, ru_idrss=0, ru_isrss=0, ru_minflt=1289, ru_majflt=0, ru_nswap=0, ru_inblock=0, ru_oublock=0, ru_msgsnd=0, ru_msgrcv=0, ru_nsignals=0, ru_nvcsw=2, ru_nivcsw=237)
[2015-10-31 21:24:46,601: INFO/MainProcess] Received task: s4jobs.tasks.decommission_device.decommission_device[f2695142-0680-4565-a51c-790c36f685f2]
[2015-10-31 21:25:07,208: INFO/MainProcess] Task s4jobs.tasks.decommission_device.decommission_device[c9cb66b1-d8ce-45c5-ab6b-05c516ec5464] succeeded in 20.61094686s: None
[2015-10-31 21:25:07,211: WARNING/Worker-1] resource.struct_rusage(ru_utime=1.141192, ru_stime=0.249278, ru_maxrss=31512, ru_ixrss=0, ru_idrss=0, ru_isrss=0, ru_minflt=2940, ru_majflt=0, ru_nswap=0, ru_inblock=0, ru_oublock=0, ru_msgsnd=0, ru_msgrcv=0, ru_nsignals=0, ru_nvcsw=19155, ru_nivcsw=1739)
[2015-10-31 21:25:07,221: INFO/MainProcess] Received task: s4jobs.tasks.decommission_device.decommission_device[cadb2d50-eb12-4e4c-aa74-57bfd4bf41d3]
[2015-10-31 21:25:23,323: INFO/MainProcess] Task s4jobs.tasks.decommission_device.decommission_device[5d2ee4a0-b25d-4107-98b7-d55200f18a72] succeeded in 16.112931506s: None
[2015-10-31 21:25:23,325: WARNING/Worker-1] resource.struct_rusage(ru_utime=2.084558, ru_stime=0.413746, ru_maxrss=32288, ru_ixrss=0, ru_idrss=0, ru_isrss=0, ru_minflt=3154, ru_majflt=0, ru_nswap=0, ru_inblock=0, ru_oublock=0, ru_msgsnd=0, ru_msgrcv=0, ru_nsignals=0, ru_nvcsw=34251, ru_nivcsw=2860)
[2015-10-31 21:25:23,338: INFO/MainProcess] Received task: s4jobs.tasks.decommission_device.decommission_device[a2b3b191-3b0a-4bb6-ab77-dbaf2113fe09]
[2015-10-31 21:25:43,668: INFO/MainProcess] Task s4jobs.tasks.decommission_device.decommission_device[d3e5539f-8f12-4bec-b1a8-19a26a6a665e] succeeded in 20.34311718s: None
[2015-10-31 21:25:43,671: WARNING/Worker-1] resource.struct_rusage(ru_utime=3.290886, ru_stime=0.659077, ru_maxrss=33072, ru_ixrss=0, ru_idrss=0, ru_isrss=0, ru_minflt=3343, ru_majflt=0, ru_nswap=0, ru_inblock=0, ru_oublock=0, ru_msgsnd=0, ru_msgrcv=0, ru_nsignals=0, ru_nvcsw=53488, ru_nivcsw=4137)
[2015-10-31 21:25:43,682: INFO/MainProcess] Received task: s4jobs.tasks.decommission_device.decommission_device[99a6ca5f-5734-4eab-b148-b2a077a754b6]
[2015-10-31 21:26:04,140: INFO/MainProcess] Task s4jobs.tasks.decommission_device.decommission_device[283055a5-b466-4435-812b-0d8dbaefb2bf] succeeded in 20.469805369s: None
[2015-10-31 21:26:04,143: WARNING/Worker-1] resource.struct_rusage(ru_utime=4.581904, ru_stime=0.922214, ru_maxrss=33860, ru_ixrss=0, ru_idrss=0, ru_isrss=0, ru_minflt=3537, ru_majflt=0, ru_nswap=0, ru_inblock=0, ru_oublock=0, ru_msgsnd=0, ru_msgrcv=0, ru_nsignals=0, ru_nvcsw=73128, ru_nivcsw=5329)
```



Discussion:
I added monitoring thread count beneath every resource.getrusage call.

It's unsettling at best.

```
[2015-10-31 21:28:57,388: WARNING/Worker-1] resource.struct_rusage(ru_utime=0.008466999999999999, ru_stime=0.00616, ru_maxrss=26192, ru_ixrss=0, ru_idrss=0, ru_isrss=0, ru_minflt=1297, ru_majflt=0, ru_nswap=0, ru_inblock=0, ru_oublock=0, ru_msgsnd=0, ru_msgrcv=0, ru_nsignals=0, ru_nvcsw=2, ru_nivcsw=28)
[2015-10-31 21:28:57,388: WARNING/Worker-1] 1
[2015-10-31 21:28:57,392: INFO/MainProcess] Received task: s4jobs.tasks.decommission_device.decommission_device[197e1f45-c2c9-4bcf-9dc1-014e07a97398]

[2015-10-31 21:29:17,974: INFO/MainProcess] Task s4jobs.tasks.decommission_device.decommission_device[cadb2d50-eb12-4e4c-aa74-57bfd4bf41d3] succeeded in 20.586918129s: None
[2015-10-31 21:29:17,979: INFO/MainProcess] Received task: s4jobs.tasks.decommission_device.decommission_device[560d4dfa-85cc-4901-a097-69f70b3c2463]
[2015-10-31 21:29:17,977: WARNING/Worker-1] resource.struct_rusage(ru_utime=1.162278, ru_stime=0.22641699999999998, ru_maxrss=31880, ru_ixrss=0, ru_idrss=0, ru_isrss=0, ru_minflt=2960, ru_majflt=0, ru_nswap=0, ru_inblock=0, ru_oublock=0, ru_msgsnd=0, ru_msgrcv=0, ru_nsignals=0, ru_nvcsw=18810, ru_nivcsw=1683)
[2015-10-31 21:29:17,980: WARNING/Worker-1] 8
[2015-10-31 21:29:38,340: INFO/MainProcess] Task s4jobs.tasks.decommission_device.decommission_device[a2b3b191-3b0a-4bb6-ab77-dbaf2113fe09] succeeded in 20.363800806s: None
[2015-10-31 21:29:38,343: WARNING/Worker-1] resource.struct_rusage(ru_utime=2.282896, ru_stime=0.463669, ru_maxrss=32540, ru_ixrss=0, ru_idrss=0, ru_isrss=0, ru_minflt=3179, ru_majflt=0, ru_nswap=0, ru_inblock=0, ru_oublock=0, ru_msgsnd=0, ru_msgrcv=0, ru_nsignals=0, ru_nvcsw=37622, ru_nivcsw=2898)
[2015-10-31 21:29:38,344: WARNING/Worker-1] 11
[2015-10-31 21:29:38,356: INFO/MainProcess] Received task: s4jobs.tasks.decommission_device.decommission_device[9b018659-e886-4622-a5b5-36b18f24ce77]
```

It seems that DataStax's Cassandra driver is spawning something behind our backs.

```
[2015-10-31 21:34:35,052: INFO/MainProcess] Task s4jobs.tasks.decommission_device.decommission_device[78aab1fc-f771-4ac5-bff1-8b7600a7760e] succeeded in 20.630556935s: None
[2015-10-31 21:34:35,055: WARNING/Worker-1] Thread 0
[2015-10-31 21:34:35,057: WARNING/Worker-1] <_MainThread(MainThread, started 140697746777856)>
[2015-10-31 21:34:35,058: WARNING/Worker-1] File "/usr/bin/celery", line 11, in <module>
[2015-10-31 21:34:35,059: WARNING/Worker-1] sys.exit(main())
[2015-10-31 21:34:35,059: WARNING/Worker-1] File "/usr/lib/python2.7/site-packages/celery/__main__.py", line 30, in main
[2015-10-31 21:34:35,060: WARNING/Worker-1] main()
[2015-10-31 21:34:35,061: WARNING/Worker-1] File "/usr/lib/python2.7/site-packages/celery/bin/celery.py", line 81, in main
[2015-10-31 21:34:35,061: WARNING/Worker-1] cmd.execute_from_commandline(argv)
[2015-10-31 21:34:35,062: WARNING/Worker-1] File "/usr/lib/python2.7/site-packages/celery/bin/celery.py", line 769, in execute_from_commandline
[2015-10-31 21:34:35,063: WARNING/Worker-1] super(CeleryCommand, self).execute_from_commandline(argv)))
[2015-10-31 21:34:35,064: INFO/MainProcess] Received task: s4jobs.tasks.decommission_device.decommission_device[bda25b52-e221-4797-8d5c-f17e384a82f9]
[2015-10-31 21:34:35,063: WARNING/Worker-1] File "/usr/lib/python2.7/site-packages/celery/bin/base.py", line 311, in execute_from_commandline
[2015-10-31 21:34:35,066: WARNING/Worker-1] return self.handle_argv(self.prog_name, argv[1:])
[2015-10-31 21:34:35,067: WARNING/Worker-1] File "/usr/lib/python2.7/site-packages/celery/bin/celery.py", line 761, in handle_argv
[2015-10-31 21:34:35,068: WARNING/Worker-1] return self.execute(command, argv)
[2015-10-31 21:34:35,069: WARNING/Worker-1] File "/usr/lib/python2.7/site-packages/celery/bin/celery.py", line 693, in execute
[2015-10-31 21:34:35,069: WARNING/Worker-1] ).run_from_argv(self.prog_name, argv[1:], command=argv[0])
[2015-10-31 21:34:35,070: WARNING/Worker-1] File "/usr/lib/python2.7/site-packages/celery/bin/worker.py", line 179, in run_from_argv
[2015-10-31 21:34:35,070: WARNING/Worker-1] return self(*args, **options)
[2015-10-31 21:34:35,071: WARNING/Worker-1] File "/usr/lib/python2.7/site-packages/celery/bin/base.py", line 274, in __call__
[2015-10-31 21:34:35,072: WARNING/Worker-1] ret = self.run(*args, **kwargs)
[2015-10-31 21:34:35,072: WARNING/Worker-1] File "/usr/lib/python2.7/site-packages/celery/bin/worker.py", line 212, in run
[2015-10-31 21:34:35,073: WARNING/Worker-1] state_db=self.node_format(state_db, hostname), **kwargs
[2015-10-31 21:34:35,073: WARNING/Worker-1] File "/usr/lib/python2.7/site-packages/celery/worker/__init__.py", line 206, in start
[2015-10-31 21:34:35,074: WARNING/Worker-1] self.blueprint.start(self)
[2015-10-31 21:34:35,074: WARNING/Worker-1] File "/usr/lib/python2.7/site-packages/celery/bootsteps.py", line 123, in start
[2015-10-31 21:34:35,074: WARNING/Worker-1] step.start(parent)
[2015-10-31 21:34:35,074: WARNING/Worker-1] File "/usr/lib/python2.7/site-packages/celery/bootsteps.py", line 374, in start
[2015-10-31 21:34:35,075: WARNING/Worker-1] return self.obj.start()
[2015-10-31 21:34:35,075: WARNING/Worker-1] File "/usr/lib/python2.7/site-packages/celery/concurrency/base.py", line 131, in start
[2015-10-31 21:34:35,075: WARNING/Worker-1] self.on_start()
[2015-10-31 21:34:35,075: WARNING/Worker-1] File "/usr/lib/python2.7/site-packages/celery/concurrency/prefork.py", line 117, in on_start
[2015-10-31 21:34:35,076: WARNING/Worker-1] **self.options)
[2015-10-31 21:34:35,076: WARNING/Worker-1] File "/usr/lib/python2.7/site-packages/celery/concurrency/asynpool.py", line 400, in __init__
[2015-10-31 21:34:35,077: WARNING/Worker-1] super(AsynPool, self).__init__(processes, *args, **kwargs)
[2015-10-31 21:34:35,077: WARNING/Worker-1] File "/usr/lib64/python2.7/site-packages/billiard/pool.py", line 968, in __init__
[2015-10-31 21:34:35,077: WARNING/Worker-1] self._create_worker_process(i)
[2015-10-31 21:34:35,077: WARNING/Worker-1] File "/usr/lib64/python2.7/site-packages/billiard/pool.py", line 1064, in _create_worker_process
[2015-10-31 21:34:35,078: WARNING/Worker-1] w.start()
[2015-10-31 21:34:35,078: WARNING/Worker-1] File "/usr/lib64/python2.7/site-packages/billiard/process.py", line 137, in start
[2015-10-31 21:34:35,078: WARNING/Worker-1] self._popen = Popen(self)
[2015-10-31 21:34:35,079: WARNING/Worker-1] File "/usr/lib64/python2.7/site-packages/billiard/forking.py", line 105, in __init__
[2015-10-31 21:34:35,079: WARNING/Worker-1] code = process_obj._bootstrap()
[2015-10-31 21:34:35,079: WARNING/Worker-1] File "/usr/lib64/python2.7/site-packages/billiard/process.py", line 292, in _bootstrap
[2015-10-31 21:34:35,080: WARNING/Worker-1] self.run()
[2015-10-31 21:34:35,080: WARNING/Worker-1] File "/usr/lib64/python2.7/site-packages/billiard/pool.py", line 291, in run
[2015-10-31 21:34:35,080: WARNING/Worker-1] sys.exit(self.workloop(pid=pid))
[2015-10-31 21:34:35,080: WARNING/Worker-1] File "/usr/lib64/python2.7/site-packages/billiard/pool.py", line 363, in workloop
[2015-10-31 21:34:35,081: WARNING/Worker-1] result = (True, prepare_result(fun(*args, **kwargs)))
[2015-10-31 21:34:35,081: WARNING/Worker-1] File "/usr/lib/python2.7/site-packages/celery/app/trace.py", line 349, in _fast_trace_task
[2015-10-31 21:34:35,081: WARNING/Worker-1] return _tasks[task].__trace__(uuid, args, kwargs, request)[0]
[2015-10-31 21:34:35,081: WARNING/Worker-1] File "/usr/lib/python2.7/site-packages/celery/app/trace.py", line 240, in trace_task
[2015-10-31 21:34:35,082: WARNING/Worker-1] R = retval = fun(*args, **kwargs)
[2015-10-31 21:34:35,082: WARNING/Worker-1] File "/usr/lib/python2.7/site-packages/celery/app/trace.py", line 438, in __protected_call__
[2015-10-31 21:34:35,082: WARNING/Worker-1] return self.run(*args, **kwargs)
[2015-10-31 21:34:35,082: WARNING/Worker-1] File "/smok4/src/s4jobs/tasks/decommission_device.py", line 33, in decommission_device
[2015-10-31 21:34:35,082: WARNING/Worker-1] traceback.print_stack(sys._current_frames()[th.ident])
[2015-10-31 21:34:35,083: WARNING/Worker-1] ==================================
[2015-10-31 21:34:35,083: WARNING/Worker-1] Thread 1
[2015-10-31 21:34:35,083: WARNING/Worker-1] <Thread(Task Scheduler, started daemon 140697458624256)>
[2015-10-31 21:34:35,085: WARNING/Worker-1] File "/usr/lib64/python2.7/threading.py", line 786, in __bootstrap
[2015-10-31 21:34:35,086: WARNING/Worker-1] self.__bootstrap_inner()
[2015-10-31 21:34:35,086: WARNING/Worker-1] File "/usr/lib64/python2.7/threading.py", line 813, in __bootstrap_inner
[2015-10-31 21:34:35,086: WARNING/Worker-1] self.run()
[2015-10-31 21:34:35,087: WARNING/Worker-1] File "/usr/lib64/python2.7/threading.py", line 766, in run
[2015-10-31 21:34:35,087: WARNING/Worker-1] self.__target(*self.__args, **self.__kwargs)
[2015-10-31 21:34:35,087: WARNING/Worker-1] File "/usr/lib/python2.7/site-packages/cassandra/cluster.py", line 2495, in run
[2015-10-31 21:34:35,087: WARNING/Worker-1] run_at, task = self._queue.get(block=True, timeout=None)
[2015-10-31 21:34:35,088: WARNING/Worker-1] File "/usr/lib64/python2.7/Queue.py", line 168, in get
[2015-10-31 21:34:35,088: WARNING/Worker-1] self.not_empty.wait()
[2015-10-31 21:34:35,089: WARNING/Worker-1] File "/usr/lib64/python2.7/threading.py", line 340, in wait
[2015-10-31 21:34:35,089: WARNING/Worker-1] waiter.acquire()
[2015-10-31 21:34:35,089: WARNING/Worker-1] ==================================
[2015-10-31 21:34:35,090: WARNING/Worker-1] Thread 2
[2015-10-31 21:34:35,090: WARNING/Worker-1] <Thread(cassandra_driver_event_loop, started daemon 140697450231552)>
[2015-10-31 21:34:35,091: WARNING/Worker-1] File "/usr/lib64/python2.7/threading.py", line 786, in __bootstrap
[2015-10-31 21:34:35,091: WARNING/Worker-1] self.__bootstrap_inner()
[2015-10-31 21:34:35,091: WARNING/Worker-1] File "/usr/lib64/python2.7/threading.py", line 813, in __bootstrap_inner
[2015-10-31 21:34:35,091: WARNING/Worker-1] self.run()
[2015-10-31 21:34:35,092: WARNING/Worker-1] File "/usr/lib64/python2.7/threading.py", line 766, in run
[2015-10-31 21:34:35,092: WARNING/Worker-1] self.__target(*self.__args, **self.__kwargs)
[2015-10-31 21:34:35,092: WARNING/Worker-1] File "/usr/lib/python2.7/site-packages/cassandra/io/asyncorereactor.py", line 91, in _run_loop
[2015-10-31 21:34:35,092: WARNING/Worker-1] asyncore.loop(timeout=0.001, use_poll=True, count=1000)
[2015-10-31 21:34:35,093: WARNING/Worker-1] File "/usr/lib64/python2.7/asyncore.py", line 220, in loop
[2015-10-31 21:34:35,093: WARNING/Worker-1] poll_fun(timeout, map)
[2015-10-31 21:34:35,093: WARNING/Worker-1] File "/usr/lib64/python2.7/asyncore.py", line 192, in poll2
[2015-10-31 21:34:35,093: WARNING/Worker-1] r = pollster.poll(timeout)
[2015-10-31 21:34:35,094: WARNING/Worker-1] ==================================
[2015-10-31 21:34:35,094: WARNING/Worker-1] Thread 3
[2015-10-31 21:34:35,094: WARNING/Worker-1] <ConnectionHeartbeat(Connection heartbeat, started daemon 140697441052416)>
[2015-10-31 21:34:35,095: WARNING/Worker-1] File "/usr/lib64/python2.7/threading.py", line 786, in __bootstrap
[2015-10-31 21:34:35,095: WARNING/Worker-1] self.__bootstrap_inner()
[2015-10-31 21:34:35,095: WARNING/Worker-1] File "/usr/lib64/python2.7/threading.py", line 813, in __bootstrap_inner
[2015-10-31 21:34:35,096: WARNING/Worker-1] self.run()
[2015-10-31 21:34:35,096: WARNING/Worker-1] File "/usr/lib/python2.7/site-packages/cassandra/connection.py", line 799, in run
[2015-10-31 21:34:35,097: WARNING/Worker-1] self._shutdown_event.wait(self._interval)
[2015-10-31 21:34:35,097: WARNING/Worker-1] File "/usr/lib64/python2.7/threading.py", line 624, in wait
[2015-10-31 21:34:35,097: WARNING/Worker-1] self.__cond.wait(timeout, balancing)
[2015-10-31 21:34:35,097: WARNING/Worker-1] File "/usr/lib64/python2.7/threading.py", line 362, in wait
[2015-10-31 21:34:35,097: WARNING/Worker-1] _sleep(delay)
[2015-10-31 21:34:35,098: WARNING/Worker-1] ==================================
[2015-10-31 21:34:35,098: WARNING/Worker-1] Thread 4
[2015-10-31 21:34:35,098: WARNING/Worker-1] <Thread(Thread-2, started daemon 140697197856512)>
[2015-10-31 21:34:35,099: WARNING/Worker-1] File "/usr/lib64/python2.7/threading.py", line 786, in __bootstrap
[2015-10-31 21:34:35,099: WARNING/Worker-1] self.__bootstrap_inner()
[2015-10-31 21:34:35,100: WARNING/Worker-1] File "/usr/lib64/python2.7/threading.py", line 813, in __bootstrap_inner
[2015-10-31 21:34:35,100: WARNING/Worker-1] self.run()
[2015-10-31 21:34:35,100: WARNING/Worker-1] File "/usr/lib64/python2.7/threading.py", line 766, in run
[2015-10-31 21:34:35,100: WARNING/Worker-1] self.__target(*self.__args, **self.__kwargs)
[2015-10-31 21:34:35,101: WARNING/Worker-1] File "/usr/lib/python2.7/site-packages/concurrent/futures/thread.py", line 65, in _worker
[2015-10-31 21:34:35,101: WARNING/Worker-1] work_item = work_queue.get(block=True)
[2015-10-31 21:34:35,101: WARNING/Worker-1] File "/usr/lib64/python2.7/Queue.py", line 168, in get
[2015-10-31 21:34:35,101: WARNING/Worker-1] self.not_empty.wait()
[2015-10-31 21:34:35,102: WARNING/Worker-1] File "/usr/lib64/python2.7/threading.py", line 340, in wait
[2015-10-31 21:34:35,102: WARNING/Worker-1] waiter.acquire()
[2015-10-31 21:34:35,102: WARNING/Worker-1] ==================================
[2015-10-31 21:34:35,102: WARNING/Worker-1] Thread 5
[2015-10-31 21:34:35,103: WARNING/Worker-1] <ConnectionHeartbeat(Connection heartbeat, started daemon 140697206249216)>
[2015-10-31 21:34:35,103: WARNING/Worker-1] File "/usr/lib64/python2.7/threading.py", line 786, in __bootstrap
[2015-10-31 21:34:35,103: WARNING/Worker-1] self.__bootstrap_inner()
[2015-10-31 21:34:35,103: WARNING/Worker-1] File "/usr/lib64/python2.7/threading.py", line 813, in __bootstrap_inner
[2015-10-31 21:34:35,104: WARNING/Worker-1] self.run()
[2015-10-31 21:34:35,104: WARNING/Worker-1] File "/usr/lib/python2.7/site-packages/cassandra/connection.py", line 799, in run
[2015-10-31 21:34:35,104: WARNING/Worker-1] self._shutdown_event.wait(self._interval)
[2015-10-31 21:34:35,104: WARNING/Worker-1] File "/usr/lib64/python2.7/threading.py", line 624, in wait
[2015-10-31 21:34:35,105: WARNING/Worker-1] self.__cond.wait(timeout, balancing)
[2015-10-31 21:34:35,105: WARNING/Worker-1] File "/usr/lib64/python2.7/threading.py", line 362, in wait
[2015-10-31 21:34:35,105: WARNING/Worker-1] _sleep(delay)
[2015-10-31 21:34:35,105: WARNING/Worker-1] ==================================
[2015-10-31 21:34:35,105: WARNING/Worker-1] Thread 6
[2015-10-31 21:34:35,106: WARNING/Worker-1] <Thread(Thread-1, started daemon 140697223034624)>
[2015-10-31 21:34:35,106: WARNING/Worker-1] File "/usr/lib64/python2.7/threading.py", line 786, in __bootstrap
[2015-10-31 21:34:35,107: WARNING/Worker-1] self.__bootstrap_inner()
[2015-10-31 21:34:35,107: WARNING/Worker-1] File "/usr/lib64/python2.7/threading.py", line 813, in __bootstrap_inner
[2015-10-31 21:34:35,107: WARNING/Worker-1] self.run()
[2015-10-31 21:34:35,107: WARNING/Worker-1] File "/usr/lib64/python2.7/threading.py", line 766, in run
[2015-10-31 21:34:35,108: WARNING/Worker-1] self.__target(*self.__args, **self.__kwargs)
[2015-10-31 21:34:35,108: WARNING/Worker-1] File "/usr/lib/python2.7/site-packages/concurrent/futures/thread.py", line 65, in _worker
[2015-10-31 21:34:35,108: WARNING/Worker-1] work_item = work_queue.get(block=True)
[2015-10-31 21:34:35,108: WARNING/Worker-1] File "/usr/lib64/python2.7/Queue.py", line 168, in get
[2015-10-31 21:34:35,109: WARNING/Worker-1] self.not_empty.wait()
[2015-10-31 21:34:35,109: WARNING/Worker-1] File "/usr/lib64/python2.7/threading.py", line 340, in wait
[2015-10-31 21:34:35,109: WARNING/Worker-1] waiter.acquire()
[2015-10-31 21:34:35,110: WARNING/Worker-1] ==================================
[2015-10-31 21:34:35,110: WARNING/Worker-1] Thread 7
[2015-10-31 21:34:35,110: WARNING/Worker-1] <Thread(Task Scheduler, started daemon 140697214641920)>
[2015-10-31 21:34:35,110: WARNING/Worker-1] File "/usr/lib64/python2.7/threading.py", line 786, in __bootstrap
[2015-10-31 21:34:35,111: WARNING/Worker-1] self.__bootstrap_inner()
[2015-10-31 21:34:35,111: WARNING/Worker-1] File "/usr/lib64/python2.7/threading.py", line 813, in __bootstrap_inner
[2015-10-31 21:34:35,111: WARNING/Worker-1] self.run()
[2015-10-31 21:34:35,111: WARNING/Worker-1] File "/usr/lib64/python2.7/threading.py", line 766, in run
[2015-10-31 21:34:35,112: WARNING/Worker-1] self.__target(*self.__args, **self.__kwargs)
[2015-10-31 21:34:35,112: WARNING/Worker-1] File "/usr/lib/python2.7/site-packages/cassandra/cluster.py", line 2495, in run
[2015-10-31 21:34:35,112: WARNING/Worker-1] run_at, task = self._queue.get(block=True, timeout=None)
[2015-10-31 21:34:35,112: WARNING/Worker-1] File "/usr/lib64/python2.7/Queue.py", line 168, in get
[2015-10-31 21:34:35,113: WARNING/Worker-1] self.not_empty.wait()
[2015-10-31 21:34:35,113: WARNING/Worker-1] File "/usr/lib64/python2.7/threading.py", line 340, in wait
[2015-10-31 21:34:35,113: WARNING/Worker-1] waiter.acquire()
[2015-10-31 21:34:35,113: WARNING/Worker-1] ==================================
```

OK, this is due to me being smart and not calling shutdown() on Cluster instances, which looks like it might need "shutdowning". Fix and test is pending.


Resolving patch:
diff --git a/celery/backends/cassandra.py b/celery/backends/cassandra.py
--- a/celery/backends/cassandra.py
+++ b/celery/backends/cassandra.py
@@ -125,13 +125,14 @@ def __init__(self, servers=None, keyspace=None, table=None, entry_ttl=None,
         self._session = None
         self._write_stmt = None
         self._read_stmt = None
+        self._make_stmt = None
 
     def process_cleanup(self):
         if self._connection is not None:
-            self._connection = None
-        if self._session is not None:
-            self._session.shutdown()
-            self._session = None
+            self._connection.shutdown() # also shuts down _session
+
+        self._connection = None
+        self._session = None
 
     def _get_connection(self, write=False):
         """Prepare the connection for action
@@ -171,6 +172,7 @@ def _get_connection(self, write=False):
                         Q_CREATE_RESULT_TABLE.format(table=self.table),
                     )
                     self._make_stmt.consistency_level = self.write_consistency
+
                     try:
                         self._session.execute(self._make_stmt)
                     except cassandra.AlreadyExists:
@@ -179,10 +181,11 @@ def _get_connection(self, write=False):
             except cassandra.OperationTimedOut:
                 # a heavily loaded or gone Cassandra cluster failed to respond.
                 # leave this class in a consistent state
-                self._connection = None
-                if self._session is not None:
-                    self._session.shutdown()
+                if self._connection is not None:
+                    self._connection.shutdown()     # also shuts down _session
 
+                self._connection = None
+                self._session = None
                 raise   # we did fail after all - reraise
 
     def _store_result(self, task_id, result, status,


Test patch:
diff --git a/celery/tests/backends/test_cassandra.py b/celery/tests/backends/test_cassandra.py
--- a/celery/tests/backends/test_cassandra.py
+++ b/celery/tests/backends/test_cassandra.py
@@ -41,6 +41,7 @@ def test_init_with_and_without_LOCAL_QUROM(self):
         with mock_module(*CASSANDRA_MODULES):
             from celery.backends import cassandra as mod
             mod.cassandra = Mock()
+
             cons = mod.cassandra.ConsistencyLevel = Object()
             cons.LOCAL_QUORUM = 'foo'
 
@@ -68,6 +69,7 @@ def test_get_task_meta_for(self):
         with mock_module(*CASSANDRA_MODULES):
             from celery.backends import cassandra as mod
             mod.cassandra = Mock()
+
             x = mod.CassandraBackend(app=self.app)
             x._connection = True
             session = x._session = Mock()
@@ -120,6 +122,9 @@ def __init__(self, *args, **kwargs):
                 def connect(self, *args, **kwargs):
                     raise OTOExc()
 
+                def shutdown(self):
+                    pass
+
             mod.cassandra = Mock()
             mod.cassandra.OperationTimedOut = OTOExc
             mod.cassandra.cluster = Mock()
@@ -133,3 +138,37 @@ def connect(self, *args, **kwargs):
             self.assertIsNone(x._session)
 
             x.process_cleanup()  # should not raise
+
+
+    def test_please_free_memory(self):
+        """
+        Ensure that Cluster object IS shut down.
+        """
+        with mock_module(*CASSANDRA_MODULES):
+            from celery.backends import cassandra as mod
+
+            class RAMHoggingCluster(object):
+
+                objects_alive = 0
+
+                def __init__(self, *args, **kwargs):
+                    pass
+
+                def connect(self, *args, **kwargs):
+                    RAMHoggingCluster.objects_alive += 1
+                    return Mock()
+
+                def shutdown(self):
+                    RAMHoggingCluster.objects_alive -= 1
+
+            mod.cassandra = Mock()
+
+            mod.cassandra.cluster = Mock()
+            mod.cassandra.cluster.Cluster = RAMHoggingCluster
+
+            for x in range(0, 10):
+                x = mod.CassandraBackend(app=self.app)
+                x._store_result('task_id', 'result', states.SUCCESS)
+                x.process_cleanup()
+
+            self.assertEquals(RAMHoggingCluster.objects_alive, 0)