Issue 1:
Checklist

I have verified that the issue exists against the master branch of Celery.

This has already been asked to the discussion group first.

I have read the relevant section in the

contribution guide

on reporting bugs.

I have checked the issues list

for similar or identical bug reports.

I have checked the pull requests list

for existing proposed fixes.

I have checked the commit log

to find out if the bug was already fixed in the master branch.

I have included all related issues and possible duplicate issues

in this issue (If there are none, check this box anyway).

Mandatory Debugging Information

I have included the output of celery -A proj report in the issue.

(if you are not able to do this, then at least specify the Celery

version affected).

I have verified that the issue exists against the master branch of Celery.

I have included the contents of pip freeze in the issue.

I have included all the versions of all the external dependencies required

to reproduce this bug.

Optional Debugging Information

I have tried reproducing the issue on more than one Python version

and/or implementation.

I have tried reproducing the issue on more than one message broker and/or

result backend.

I have tried reproducing the issue on more than one version of the message

broker and/or result backend.

I have tried reproducing the issue on more than one operating system.

I have tried reproducing the issue on more than one workers pool.

I have tried reproducing the issue with autoscaling, retries,

ETA/Countdown & rate limits disabled.

I have tried reproducing the issue after downgrading

and/or upgrading Celery and its dependencies.

Related Issues and Possible Duplicates

Related Issues

None

Possible Duplicates

None

Environment & Settings

Celery version: 4.1.0

celery report Output:

Steps to Reproduce

Required Dependencies

Minimal Python Version: 2.7.5

Minimal Celery Version: N/A or Unknown

Minimal Kombu Version: N/A or Unknown

Minimal Broker Version: N/A or Unknown

Minimal Result Backend Version: N/A or Unknown

Minimal OS and/or Kernel Version: N/A or Unknown

Minimal Broker Client Version: N/A or Unknown

Minimal Result Backend Client Version: N/A or Unknown

Python Packages

pip freeze Output:

Other Dependencies

N/A

Minimally Reproducible Test Case

Expected Behavior

Actual Behavior

When I run Celery worker with this command

celery -A sx_workers worker -c1 -l info

it shows me the next error stack trace:

Traceback (most recent call last): File "/Users/amerzildzic/swipestox/sx_distributed_processing/sxdpenv/bin/celery", line 11, in <module> sys.exit(main()) File "/Users/amerzildzic/swipestox/sx_distributed_processing/sxdpenv/lib/python2.7/site-packages/celery/__main__.py", line 14, in main _main() File "/Users/amerzildzic/swipestox/sx_distributed_processing/sxdpenv/lib/python2.7/site-packages/celery/bin/celery.py", line 326, in main cmd.execute_from_commandline(argv) File "/Users/amerzildzic/swipestox/sx_distributed_processing/sxdpenv/lib/python2.7/site-packages/celery/bin/celery.py", line 488, in execute_from_commandline super(CeleryCommand, self).execute_from_commandline(argv))) File "/Users/amerzildzic/swipestox/sx_distributed_processing/sxdpenv/lib/python2.7/site-packages/celery/bin/base.py", line 281, in execute_from_commandline return self.handle_argv(self.prog_name, argv[1:]) File "/Users/amerzildzic/swipestox/sx_distributed_processing/sxdpenv/lib/python2.7/site-packages/celery/bin/celery.py", line 480, in handle_argv return self.execute(command, argv) File "/Users/amerzildzic/swipestox/sx_distributed_processing/sxdpenv/lib/python2.7/site-packages/celery/bin/celery.py", line 412, in execute ).run_from_argv(self.prog_name, argv[1:], command=argv[0]) File "/Users/amerzildzic/swipestox/sx_distributed_processing/sxdpenv/lib/python2.7/site-packages/celery/bin/worker.py", line 219, in run_from_argv *self.parse_options(prog_name, argv, command)) File "/Users/amerzildzic/swipestox/sx_distributed_processing/sxdpenv/lib/python2.7/site-packages/celery/bin/base.py", line 403, in parse_options self.parser = self.create_parser(prog_name, command) File "/Users/amerzildzic/swipestox/sx_distributed_processing/sxdpenv/lib/python2.7/site-packages/celery/bin/base.py", line 419, in create_parser self.add_arguments(parser) File "/Users/amerzildzic/swipestox/sx_distributed_processing/sxdpenv/lib/python2.7/site-packages/celery/bin/worker.py", line 275, in add_arguments default=conf.worker_state_db, File "/Users/amerzildzic/swipestox/sx_distributed_processing/sxdpenv/lib/python2.7/site-packages/celery/utils/collections.py", line 130, in __getattr__ return self[k] File "/Users/amerzildzic/swipestox/sx_distributed_processing/sxdpenv/lib/python2.7/site-packages/celery/utils/collections.py", line 431, in __getitem__ return getitem(k) File "/Users/amerzildzic/swipestox/sx_distributed_processing/sxdpenv/lib/python2.7/site-packages/celery/utils/collections.py", line 280, in __getitem__ return mapping[_key] File "/Users/amerzildzic/swipestox/sx_distributed_processing/sxdpenv/bin/../lib/python2.7/UserDict.py", line 36, in __getitem__ if key in self.data: File "/Users/amerzildzic/swipestox/sx_distributed_processing/sxdpenv/lib/python2.7/site-packages/celery/utils/collections.py", line 130, in __getattr__ return self[k] File "/Users/amerzildzic/swipestox/sx_distributed_processing/sxdpenv/bin/../lib/python2.7/UserDict.py", line 36, in __getitem__ if key in self.data: File "/Users/amerzildzic/swipestox/sx_distributed_processing/sxdpenv/lib/python2.7/site-packages/celery/utils/collections.py", line 130, in __getattr__ return self[k] File "/Users/amerzildzic/swipestox/sx_distributed_processing/sxdpenv/bin/../lib/python2.7/UserDict.py", line 36, in __getitem__ if key in self.data: File "/Users/amerzildzic/swipestox/sx_distributed_processing/sxdpenv/lib/python2.7/site-packages/celery/utils/collections.py", line 130, in __getattr__ return self[k] File "/Users/amerzildzic/swipestox/sx_distributed_processing/sxdpenv/bin/../lib/python2.7/UserDict.py", line 36, in __getitem__ if key in self.data: File "/Users/amerzildzic/swipestox/sx_distributed_processing/sxdpenv/lib/python2.7/site-packages/celery/utils/collections.py", line 130, in __getattr__ return self[k] File "/Users/amerzildzic/swipestox/sx_distributed_processing/sxdpenv/bin/../lib/python2.7/UserDict.py", line 36, in __getitem__ if key in self.data: File "/Users/amerzildzic/swipestox/sx_distributed_processing/sxdpenv/lib/python2.7/site-packages/celery/utils/collections.py", line 130, in __getattr__ return self[k] File "/Users/amerzildzic/swipestox/sx_distributed_processing/sxdpenv/bin/../lib/python2.7/UserDict.py", line 36, in __getitem__ if key in self.data: File "/Users/amerzildzic/swipestox/sx_distributed_processing/sxdpenv/lib/python2.7/site-packages/celery/utils/collections.py", line 130, in __getattr__ return self[k] File "/Users/amerzildzic/swipestox/sx_distributed_processing/sxdpenv/bin/../lib/python2.7/UserDict.py", line 36, in __getitem__ if key in self.data: File "/Users/amerzildzic/swipestox/sx_distributed_processing/sxdpenv/lib/python2.7/site-packages/celery/utils/collections.py", line 130, in __getattr__ return self[k] File "/Users/amerzildzic/swipestox/sx_distributed_processing/sxdpenv/bin/../lib/python2.7/UserDict.py", line 36, in __getitem__ if key in self.data: File "/Users/amerzildzic/swipestox/sx_distributed_processing/sxdpenv/lib/python2.7/site-packages/celery/utils/collections.py", line 130, in __getattr__ return self[k] File "/Users/amerzildzic/swipestox/sx_distributed_processing/sxdpenv/bin/../lib/python2.7/UserDict.py", line 36, in __getitem__ if key in self.data: File "/Users/amerzildzic/swipestox/sx_distributed_processing/sxdpenv/lib/python2.7/site-packages/celery/utils/collections.py", line 130, in __getattr__ return self[k] File "/Users/amerzildzic/swipestox/sx_distributed_processing/sxdpenv/bin/../lib/python2.7/UserDict.py", line 36, in __getitem__ if key in self.data: File "/Users/amerzildzic/swipestox/sx_distributed_processing/sxdpenv/lib/python2.7/site-packages/celery/utils/collections.py", line 130, in __getattr__ return self[k] File "/Users/amerzildzic/swipestox/sx_distributed_processing/sxdpenv/bin/../lib/python2.7/UserDict.py", line 36, in __getitem__ if key in self.data: File "/Users/amerzildzic/swipestox/sx_distributed_processing/sxdpenv/lib/python2.7/site-packages/celery/utils/collections.py", line 130, in __getattr__ return self[k] File "/Users/amerzildzic/swipestox/sx_distributed_processing/sxdpenv/bin/../lib/python2.7/UserDict.py", line 36, in __getitem__ if key in self.data: File "/Users/amerzildzic/swipestox/sx_distributed_processing/sxdpenv/lib/python2.7/site-packages/celery/utils/collections.py", line 130, in __getattr__ return self[k] File "/Users/amerzildzic/swipestox/sx_distributed_processing/sxdpenv/bin/../lib/python2.7/UserDict.py", line 36, in __getitem__ if key in self.data: File "/Users/amerzildzic/swipestox/sx_distributed_processing/sxdpenv/lib/python2.7/site-packages/celery/utils/collections.py", line 130, in __getattr__ return self[k] File "/Users/amerzildzic/swipestox/sx_distributed_processing/sxdpenv/bin/../lib/python2.7/UserDict.py", line 36, in __getitem__ if key in self.data: File "/Users/amerzildzic/swipestox/sx_distributed_processing/sxdpenv/lib/python2.7/site-packages/celery/utils/collections.py", line 130, in __getattr__ return self[k] File "/Users/amerzildzic/swipestox/sx_distributed_processing/sxdpenv/bin/../lib/python2.7/UserDict.py", line 36, in __getitem__ if key in self.data: File "/Users/amerzildzic/swipestox/sx_distributed_processing/sxdpenv/lib/python2.7/site-packages/celery/utils/collections.py", line 130, in __getattr__ return self[k] File "/Users/amerzildzic/swipestox/sx_distributed_processing/sxdpenv/bin/../lib/python2.7/UserDict.py", line 36, in __getitem__ if key in self.data: File "/Users/amerzildzic/swipestox/sx_distributed_processing/sxdpenv/lib/python2.7/site-packages/celery/utils/collections.py", line 130, in __getattr__ return self[k] File "/Users/amerzildzic/swipestox/sx_distributed_processing/sxdpenv/bin/../lib/python2.7/UserDict.py", line 36, in __getitem__ if key in self.data: File "/Users/amerzildzic/swipestox/sx_distributed_processing/sxdpenv/lib/python2.7/site-packages/celery/utils/collections.py", line 130, in __getattr__ return self[k] File "/Users/amerzildzic/swipestox/sx_distributed_processing/sxdpenv/bin/../lib/python2.7/UserDict.py", line 36, in __getitem__ if key in self.data: File "/Users/amerzildzic/swipestox/sx_distributed_processing/sxdpenv/lib/python2.7/site-packages/celery/utils/collections.py", line 130, in __getattr__ return self[k] File "/Users/amerzildzic/swipestox/sx_distributed_processing/sxdpenv/bin/../lib/python2.7/UserDict.py", line 36, in __getitem__ if key in self.data: File "/Users/amerzildzic/swipestox/sx_distributed_processing/sxdpenv/lib/python2.7/site-packages/celery/utils/collections.py", line 130, in __getattr__ return self[k] File "/Users/amerzildzic/swipestox/sx_distributed_processing/sxdpenv/bin/../lib/python2.7/UserDict.py", line 36, in __getitem__ if key in self.data: File "/Users/amerzildzic/swipestox/sx_distributed_processing/sxdpenv/lib/python2.7/site-packages/celery/utils/collections.py", line 130, in __getattr__ return self[k] File "/Users/amerzildzic/swipestox/sx_distributed_processing/sxdpenv/bin/../lib/python2.7/UserDict.py", line 36, in __getitem__ if key in self.data: File "/Users/amerzildzic/swipestox/sx_distributed_processing/sxdpenv/lib/python2.7/site-packages/celery/utils/collections.py", line 130, in __getattr__ return self[k] File "/Users/amerzildzic/swipestox/sx_distributed_processing/sxdpenv/bin/../lib/python2.7/UserDict.py", line 36, in __getitem__ if key in self.data: File "/Users/amerzildzic/swipestox/sx_distributed_processing/sxdpenv/lib/python2.7/site-packages/celery/utils/collections.py", line 130, in __getattr__ return self[k] File "/Users/amerzildzic/swipestox/sx_distributed_processing/sxdpenv/bin/../lib/python2.7/UserDict.py", line 36, in __getitem__ if key in self.data: File "/Users/amerzildzic/swipestox/sx_distributed_processing/sxdpenv/lib/python2.7/site-packages/celery/utils/collections.py", line 130, in __getattr__ return self[k] File "/Users/amerzildzic/swipestox/sx_distributed_processing/sxdpenv/bin/../lib/python2.7/UserDict.py", line 36, in __getitem__ if key in self.data: File "/Users/amerzildzic/swipestox/sx_distributed_processing/sxdpenv/lib/python2.7/site-packages/celery/utils/collections.py", line 130, in __getattr__ return self[k] File "/Users/amerzildzic/swipestox/sx_distributed_processing/sxdpenv/bin/../lib/python2.7/UserDict.py", line 36, in __getitem__ if key in self.data: File "/Users/amerzildzic/swipestox/sx_distributed_processing/sxdpenv/lib/python2.7/site-packages/celery/utils/collections.py", line 130, in __getattr__ return self[k] File "/Users/amerzildzic/swipestox/sx_distributed_processing/sxdpenv/bin/../lib/python2.7/UserDict.py", line 36, in __getitem__ if key in self.data: File "/Users/amerzildzic/swipestox/sx_distributed_processing/sxdpenv/lib/python2.7/site-packages/celery/utils/collections.py", line 130, in __getattr__ return self[k] File "/Users/amerzildzic/swipestox/sx_distributed_processing/sxdpenv/bin/../lib/python2.7/UserDict.py", line 36, in __getitem__ if key in self.data: File "/Users/amerzildzic/swipestox/sx_distributed_processing/sxdpenv/lib/python2.7/site-packages/celery/utils/collections.py", line 130, in __getattr__ return self[k] File "/Users/amerzildzic/swipestox/sx_distributed_processing/sxdpenv/bin/../lib/python2.7/UserDict.py", line 36, in __getitem__ if key in self.data: File "/Users/amerzildzic/swipestox/sx_distributed_processing/sxdpenv/lib/python2.7/site-packages/celery/utils/collections.py", line 130, in __getattr__ return self[k] File "/Users/amerzildzic/swipestox/sx_distributed_processing/sxdpenv/bin/../lib/python2.7/UserDict.py", line 36, in __getitem__ if key in self.data: File "/Users/amerzildzic/swipestox/sx_distributed_processing/sxdpenv/lib/python2.7/site-packages/celery/utils/collections.py", line 130, in __getattr__ return self[k] File "/Users/amerzildzic/swipestox/sx_distributed_processing/sxdpenv/bin/../lib/python2.7/UserDict.py", line 36, in __getitem__ if key in self.data: File "/Users/amerzildzic/swipestox/sx_distributed_processing/sxdpenv/lib/python2.7/site-packages/celery/utils/collections.py", line 130, in __getattr__ return self[k] File "/Users/amerzildzic/swipestox/sx_distributed_processing/sxdpenv/bin/../lib/python2.7/UserDict.py", line 36, in __getitem__ if key in self.data: File "/Users/amerzildzic/swipestox/sx_distributed_processing/sxdpenv/lib/python2.7/site-packages/celery/utils/collections.py", line 130, in __getattr__ return self[k] File "/Users/amerzildzic/swipestox/sx_distributed_processing/sxdpenv/bin/../lib/python2.7/UserDict.py", line 36, in __getitem__ if key in self.data: File "/Users/amerzildzic/swipestox/sx_distributed_processing/sxdpenv/lib/python2.7/site-packages/celery/utils/collections.py", line 130, in __getattr__ return self[k] File "/Users/amerzildzic/swipestox/sx_distributed_processing/sxdpenv/bin/../lib/python2.7/UserDict.py", line 36, in __getitem__ if key in self.data: File "/Users/amerzildzic/swipestox/sx_distributed_processing/sxdpenv/lib/python2.7/site-packages/celery/utils/collections.py", line 130, in __getattr__ return self[k] File "/Users/amerzildzic/swipestox/sx_distributed_processing/sxdpenv/bin/../lib/python2.7/UserDict.py", line 36, in __getitem__ if key in self.data: File "/Users/amerzildzic/swipestox/sx_distributed_processing/sxdpenv/lib/python2.7/site-packages/celery/utils/collections.py", line 130, in __getattr__ return self[k] File "/Users/amerzildzic/swipestox/sx_distributed_processing/sxdpenv/bin/../lib/python2.7/UserDict.py", line 36, in __getitem__ if key in self.data: File "/Users/amerzildzic/swipestox/sx_distributed_processing/sxdpenv/lib/python2.7/site-packages/celery/utils/collections.py", line 130, in __getattr__ return self[k] File "/Users/amerzildzic/swipestox/sx_distributed_processing/sxdpenv/bin/../lib/python2.7/UserDict.py", line 36, in __getitem__ if key in self.data: File "/Users/amerzildzic/swipestox/sx_distributed_processing/sxdpenv/lib/python2.7/site-packages/celery/utils/collections.py", line 130, in __getattr__ return self[k] File "/Users/amerzildzic/swipestox/sx_distributed_processing/sxdpenv/bin/../lib/python2.7/UserDict.py", line 36, in __getitem__ if key in self.data: File "/Users/amerzildzic/swipestox/sx_distributed_processing/sxdpenv/lib/python2.7/site-packages/celery/utils/collections.py", line 130, in __getattr__ return self[k] File "/Users/amerzildzic/swipestox/sx_distributed_processing/sxdpenv/bin/../lib/python2.7/UserDict.py", line 36, in __getitem__ if key in self.data: File "/Users/amerzildzic/swipestox/sx_distributed_processing/sxdpenv/lib/python2.7/site-packages/celery/utils/collections.py", line 130, in __getattr__ return self[k] File "/Users/amerzildzic/swipestox/sx_distributed_processing/sxdpenv/bin/../lib/python2.7/UserDict.py", line 36, in __getitem__ if key in self.data: File "/Users/amerzildzic/swipestox/sx_distributed_processing/sxdpenv/lib/python2.7/site-packages/celery/utils/collections.py", line 130, in __getattr__ return self[k] File "/Users/amerzildzic/swipestox/sx_distributed_processing/sxdpenv/bin/../lib/python2.7/UserDict.py", line 36, in __getitem__ if key in self.data: File "/Users/amerzildzic/swipestox/sx_distributed_processing/sxdpenv/lib/python2.7/site-packages/celery/utils/collections.py", line 130, in __getattr__ return self[k] File "/Users/amerzildzic/swipestox/sx_distributed_processing/sxdpenv/bin/../lib/python2.7/UserDict.py", line 36, in __getitem__ if key in self.data: File "/Users/amerzildzic/swipestox/sx_distributed_processing/sxdpenv/lib/python2.7/site-packages/celery/utils/collections.py", line 130, in __getattr__ return self[k] ... /Users/amerzildzic/swipestox/sx_distributed_processing/sxdpenv/bin/../lib/python2.7/UserDict.py", line 36, in __getitem__ if key in self.data: File "/Users/amerzildzic/swipestox/sx_distributed_processing/sxdpenv/lib/python2.7/site-packages/celery/utils/collections.py", line 130, in __getattr__ return self[k] File "/Users/amerzildzic/swipestox/sx_distributed_processing/sxdpenv/bin/../lib/python2.7/UserDict.py", line 36, in __getitem__ if key in self.data: File "/Users/amerzildzic/swipestox/sx_distributed_processing/sxdpenv/lib/python2.7/site-packages/kombu/utils/objects.py", line 44, in __get__ value = obj.__dict__[self.__name__] = self.__get(obj) File "/Users/amerzildzic/swipestox/sx_distributed_processing/sxdpenv/lib/python2.7/site-packages/celery/app/base.py", line 148, in data return self.callback() File "/Users/amerzildzic/swipestox/sx_distributed_processing/sxdpenv/lib/python2.7/site-packages/celery/app/base.py", line 911, in _finalize_pending_conf conf = self._conf = self._load_config() File "/Users/amerzildzic/swipestox/sx_distributed_processing/sxdpenv/lib/python2.7/site-packages/celery/app/base.py", line 921, in _load_config self.loader.config_from_object(self._config_source) File "/Users/amerzildzic/swipestox/sx_distributed_processing/sxdpenv/lib/python2.7/site-packages/celery/loaders/base.py", line 128, in config_from_object obj = self._smart_import(obj, imp=self.import_from_cwd) File "/Users/amerzildzic/swipestox/sx_distributed_processing/sxdpenv/lib/python2.7/site-packages/celery/loaders/base.py", line 146, in _smart_import return imp(path) File "/Users/amerzildzic/swipestox/sx_distributed_processing/sxdpenv/lib/python2.7/site-packages/celery/loaders/base.py", line 106, in import_from_cwd package=package, File "/Users/amerzildzic/swipestox/sx_distributed_processing/sxdpenv/lib/python2.7/site-packages/celery/utils/imports.py", line 100, in import_from_cwd with cwd_in_path(): File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/contextlib.py", line 84, in helper return GeneratorContextManager(func(*args, **kwds)) RuntimeError: maximum recursion depth exceeded

I dig through the code a bit and as I see there is recutsion between

collections.py : self[k]

and

UserDyct.py: getitem (which should be same as indexing??)

since both are used on the stack trace.

I'm not so familiar with Python nor project, pretty new to this.

Thanks in advance,

Amer

Issue 2:
Checklist

I have checked the issues list

for similar or identical bug reports.

I have checked the pull requests list

for existing proposed fixes.

I have checked the commit log

to find out if the bug was already fixed in the main branch.

I have included all related issues and possible duplicate issues in this issue

(If there are none, check this box anyway).

Related Issues and Possible Duplicates

Related Issues

#4843 - inappropriate usage of eta and countdown can lead to increasing worker RAM usage which may be mistakenly recognized as a memory leak

#4766

Possible Duplicates

None

Description

eta and countdown parameters can be easily overused, especially when using Redis as the broker.

When a task with those parameters is sent to the queue, the worker grabs that task immediately but doesn't actually execute it until eta/countdown conditions pass. Until that, the task is stored in memory. Obviously, one such task won't cause a problem, but in our production environment, we ended up in a situation where hundreds of thousands were accumulated in the worker, causing RAM usage to increase by 30GB over a period of a few days. This behavior is documented very deeply in the documentation, so I guess not many people have reached that page.

The situation can get even worse because of the visibility_timeout setting (1 hour by default), which causes those tasks to be redelivered. If the eta/countdown points to a distant future, tasks will be redelivered every visibility_timeout period.

The documentation suggests increasing the visibility_timeout to match the longest possible ETA in the application:

So you have to increase the visibility timeout to match the time of the longest ETA you’re planning to use

IMO, that's not a good idea as it can have a negative effect on reliability. On worker's failure, dropped tasks won't get redelivered until visibility_timeout passes. If set to some high value, it may impact user experience (e.g. email is sent after x days when its content is no longer relevant, some data important for the user is processed with x days of delay, etc.).

Suggestions

I'd suggest strongly discouraging using eta/countdown parameters. In my opinion, they should be only used with very low values (seconds, minutes). Instead, alternatives could be suggested, e.g. database-backed celery beat.

In Caveats I'd not recommend increasing visibility_timeout as a feasible solution, but if the task needs a longer ETA, I'd point to the alternatives mentioned above.

Optionally, it would be nice to add a warning section with all the implications I've mentioned above.

Lots of credit to the author of that article: https://engineering.instawork.com/celery-eta-tasks-demystified-424b836e4e94 who explained the problem perfectly