Issue 1:
Checklist

I have read the relevant section in the

contribution guide

on reporting bugs.

I have checked the issues list

for similar or identical bug reports.

Mandatory Debugging Information

I have included the output of celery -A proj report in the issue.

Optional Debugging Information

I have tried reproducing the issue on more than one version of the message

broker and/or result backend.

Related Issues and Possible Duplicates

Related Issues

None

Possible Duplicates

None

Environment & Settings

Celery version: 4.2.0 (windowlicker)

celery report Output:

software -> celery:4.2.0 (windowlicker) kombu:4.2.1 py:2.7.5
billiard:3.5.0.3 redis:2.10.6
platform -> system:Linux arch:64bit, ELF imp:CPython
loader -> celery.loaders.app.AppLoader
settings -> transport:redis results:redis://localhost:6543/

CELERY_ACCEPT_CONTENT: ['json']
BROKER_CONNECTION_RETRY: False
CELERY_IGNORE_RESULT: True
CELERY_DEFAULT_QUEUE: 'scheduler'
CELERY_TIMEZONE: 'UTC'
BROKER_URL: u'redis://localhost:6543//'
WORKER_MAX_MEMORY_PER_CHILD: 60000
CELERY_DISABLE_RATE_LIMITS: True
CELERYD_MAX_TASKS_PER_CHILD: 5000
BROKER_POOL_LIMIT: 50
CELERY_TASK_SERIALIZER: 'json'
CELERYD_WORKER_LOST_WAIT: 60.0
CELERY_DEFAULT_EXCHANGE: 'scheduler'
CELERY_QUEUES:
(<unbound Queue default -> <unbound Exchange default(direct)> -> default>,
<unbound Queue scheduler -> <unbound Exchange scheduler(direct)> -> scheduler>,
<unbound Queue volume_monitor -> <unbound Exchange volume_monitor(direct)> -> volume_monitor>)
CELERY_DEFAULT_ROUTING_KEY: u'********'
BROKER_HEARTBEAT: 600
CELERY_RESULT_BACKEND: u'redis://localhost:6543/'
BROKER_CONNECTION_TIMEOUT: 10
CELERY_ROUTES: {
'xxx.xxx.task.internal.engine.rollback_task': { 'queue': 'default',
'routing_key': u'********'},
'xxx.xxx.task.internal.engine.run_task': { 'queue': 'default',
'routing_key': u'********'}}
CELERY_REDIS_MAX_CONNECTIONS: 50
CELERYD_PREFETCH_MULTIPLIER: 0
BROKER_HEARTBEAT_CHECKRATE: 5.0

Steps to Reproduce

Required Dependencies

Minimal Python Version: 2.7.5

Minimal Celery Version: 4.2.0 (windowlicker)

Minimal Kombu Version: 4.2.1

Minimal Broker Version: redis-2.8.21-1.el7.x86_64 and python-redis-2.10.6-1.noarch

Minimal Result Backend Version: redis-2.8.21-1.el7.x86_64 and python-redis-2.10.6-1.noarch

Minimal OS and/or Kernel Version: Linux version 3.10.0-514.44.5.10_136.x86_64

Minimally Reproducible Test Case

This my reproducible steps :

1.kill the process of celery beat ;

2.run cmd 'celery beat --detach -A xxx.xxx.task.internal.engine.app -s /var/run/celerybeat-schedule --max-interval=3 --pidfile=/xxx/xxx/test-beat.pid --logfile=/var/log/xxx/xxx/celery-beat.log --loglevel=DEBUG';

3.check the result of celery beat process and celery-beat.log;

Expected Behavior

Expected Behavior is celery beat process is running.

Actual Behavior

Actual Behavior is celery beat can't start.

celery-beat.log Output:

[2019-03-29 09:33:13,302: DEBUG/MainProcess] Setting default socket timeout to 30
[2019-03-29 09:33:13,303: INFO/MainProcess] beat: Starting...
[2019-03-29 09:33:13,318: CRITICAL/MainProcess] beat raised exception <class 'bsddb.db.DBError'>: DBError(9, 'Bad file descriptor -- BDB0142 ftruncate: 16384: Bad file descriptor')
Traceback (most recent call last):
File "/usr/lib/python2.7/site-packages/celery/apps/beat.py", line 109, in start_scheduler
service.start()
File "/usr/lib/python2.7/site-packages/celery/beat.py", line 579, in start
humanize_seconds(self.scheduler.max_interval))
File "/usr/lib/python2.7/site-packages/kombu/utils/objects.py", line 44, in __get__
value = obj.__dict__[self.__name__] = self.__get(obj)
File "/usr/lib/python2.7/site-packages/celery/beat.py", line 623, in scheduler
return self.get_scheduler()
File "/usr/lib/python2.7/site-packages/celery/beat.py", line 618, in get_scheduler
lazy=lazy,
File "/usr/lib/python2.7/site-packages/celery/beat.py", line 458, in __init__
Scheduler.__init__(self, *args, **kwargs)
File "/usr/lib/python2.7/site-packages/celery/beat.py", line 225, in __init__
self.setup_schedule()
File "/usr/lib/python2.7/site-packages/celery/beat.py", line 508, in setup_schedule
self.sync()
File "/usr/lib/python2.7/site-packages/celery/beat.py", line 544, in sync
self._store.sync()
File "/usr/lib64/python2.7/shelve.py", line 165, in sync
self[key] = entry
File "/usr/lib64/python2.7/shelve.py", line 133, in __setitem__
self.dict[key] = f.getvalue()
File "/usr/lib64/python2.7/bsddb/__init__.py", line 279, in __setitem__
_DeadlockWrap(wrapF) # self.db[key] = value
File "/usr/lib64/python2.7/bsddb/dbutils.py", line 68, in DeadlockWrap
return function(*_args, **_kwargs)
File "/usr/lib64/python2.7/bsddb/__init__.py", line 278, in wrapF
self.db[key] = value
DBError: (9, 'Bad file descriptor -- BDB0142 ftruncate: 16384: Bad file descriptor')
[2019-03-29 09:33:13,319: WARNING/MainProcess] Traceback (most recent call last):
[2019-03-29 09:33:13,320: WARNING/MainProcess] File "/bin/celery", line 9, in <module>
[2019-03-29 09:33:13,320: WARNING/MainProcess] load_entry_point('celery==4.2.0', 'console_scripts', 'celery')()
[2019-03-29 09:33:13,320: WARNING/MainProcess] File "/usr/lib/python2.7/site-packages/celery/__main__.py", line 16, in main
[2019-03-29 09:33:13,320: WARNING/MainProcess] _main()
[2019-03-29 09:33:13,320: WARNING/MainProcess] File "/usr/lib/python2.7/site-packages/celery/bin/celery.py", line 322, in main
[2019-03-29 09:33:13,321: WARNING/MainProcess] cmd.execute_from_commandline(argv)
[2019-03-29 09:33:13,321: WARNING/MainProcess] File "/usr/lib/python2.7/site-packages/celery/bin/celery.py", line 496, in execute_from_commandline
[2019-03-29 09:33:13,321: WARNING/MainProcess] super(CeleryCommand, self).execute_from_commandline(argv)))
[2019-03-29 09:33:13,321: WARNING/MainProcess] File "/usr/lib/python2.7/site-packages/celery/bin/base.py", line 275, in execute_from_commandline
[2019-03-29 09:33:13,322: WARNING/MainProcess] return self.handle_argv(self.prog_name, argv[1:])
[2019-03-29 09:33:13,322: WARNING/MainProcess] File "/usr/lib/python2.7/site-packages/celery/bin/celery.py", line 488, in handle_argv
[2019-03-29 09:33:13,322: WARNING/MainProcess] return self.execute(command, argv)
[2019-03-29 09:33:13,322: WARNING/MainProcess] File "/usr/lib/python2.7/site-packages/celery/bin/celery.py", line 420, in execute
[2019-03-29 09:33:13,323: WARNING/MainProcess] ).run_from_argv(self.prog_name, argv[1:], command=argv[0])
[2019-03-29 09:33:13,323: WARNING/MainProcess] File "/usr/lib/python2.7/site-packages/celery/bin/base.py", line 279, in run_from_argv
[2019-03-29 09:33:13,323: WARNING/MainProcess] sys.argv if argv is None else argv, command)
[2019-03-29 09:33:13,323: WARNING/MainProcess] File "/usr/lib/python2.7/site-packages/celery/bin/base.py", line 363, in handle_argv
[2019-03-29 09:33:13,324: WARNING/MainProcess] return self(*args, **options)
[2019-03-29 09:33:13,324: WARNING/MainProcess] File "/usr/lib/python2.7/site-packages/celery/bin/base.py", line 238, in __call__
[2019-03-29 09:33:13,324: WARNING/MainProcess] ret = self.run(*args, **kwargs)
[2019-03-29 09:33:13,324: WARNING/MainProcess] File "/usr/lib/python2.7/site-packages/celery/bin/beat.py", line 107, in run
[2019-03-29 09:33:13,324: WARNING/MainProcess] return beat().run()
[2019-03-29 09:33:13,325: WARNING/MainProcess] File "/usr/lib/python2.7/site-packages/celery/apps/beat.py", line 81, in run
[2019-03-29 09:33:13,325: WARNING/MainProcess] self.start_scheduler()
[2019-03-29 09:33:13,325: WARNING/MainProcess] File "/usr/lib/python2.7/site-packages/celery/apps/beat.py", line 109, in start_scheduler
[2019-03-29 09:33:13,325: WARNING/MainProcess] service.start()
[2019-03-29 09:33:13,325: WARNING/MainProcess] File "/usr/lib/python2.7/site-packages/celery/beat.py", line 579, in start
[2019-03-29 09:33:13,326: WARNING/MainProcess] humanize_seconds(self.scheduler.max_interval))
[2019-03-29 09:33:13,326: WARNING/MainProcess] File "/usr/lib/python2.7/site-packages/kombu/utils/objects.py", line 44, in __get__
[2019-03-29 09:33:13,326: WARNING/MainProcess] value = obj.__dict__[self.__name__] = self.__get(obj)
[2019-03-29 09:33:13,326: WARNING/MainProcess] File "/usr/lib/python2.7/site-packages/celery/beat.py", line 623, in scheduler
[2019-03-29 09:33:13,327: WARNING/MainProcess] return self.get_scheduler()
[2019-03-29 09:33:13,327: WARNING/MainProcess] File "/usr/lib/python2.7/site-packages/celery/beat.py", line 618, in get_scheduler
[2019-03-29 09:33:13,327: WARNING/MainProcess] lazy=lazy,
[2019-03-29 09:33:13,327: WARNING/MainProcess] File "/usr/lib/python2.7/site-packages/celery/beat.py", line 458, in __init__
[2019-03-29 09:33:13,328: WARNING/MainProcess] Scheduler.__init__(self, *args, **kwargs)
[2019-03-29 09:33:13,328: WARNING/MainProcess] File "/usr/lib/python2.7/site-packages/celery/beat.py", line 225, in __init__
[2019-03-29 09:33:13,328: WARNING/MainProcess] self.setup_schedule()
[2019-03-29 09:33:13,328: WARNING/MainProcess] File "/usr/lib/python2.7/site-packages/celery/beat.py", line 508, in setup_schedule
[2019-03-29 09:33:13,329: WARNING/MainProcess] self.sync()
[2019-03-29 09:33:13,329: WARNING/MainProcess] File "/usr/lib/python2.7/site-packages/celery/beat.py", line 544, in sync
[2019-03-29 09:33:13,329: WARNING/MainProcess] self._store.sync()
[2019-03-29 09:33:13,330: WARNING/MainProcess] File "/usr/lib64/python2.7/shelve.py", line 165, in sync
[2019-03-29 09:33:13,330: WARNING/MainProcess] self[key] = entry
[2019-03-29 09:33:13,330: WARNING/MainProcess] File "/usr/lib64/python2.7/shelve.py", line 133, in __setitem__
[2019-03-29 09:33:13,330: WARNING/MainProcess] self.dict[key] = f.getvalue()
[2019-03-29 09:33:13,330: WARNING/MainProcess] File "/usr/lib64/python2.7/bsddb/__init__.py", line 279, in __setitem__
[2019-03-29 09:33:13,331: WARNING/MainProcess] _DeadlockWrap(wrapF) # self.db[key] = value
[2019-03-29 09:33:13,331: WARNING/MainProcess] File "/usr/lib64/python2.7/bsddb/dbutils.py", line 68, in DeadlockWrap
[2019-03-29 09:33:13,331: WARNING/MainProcess] return function(*_args, **_kwargs)
[2019-03-29 09:33:13,331: WARNING/MainProcess] File "/usr/lib64/python2.7/bsddb/__init__.py", line 278, in wrapF
[2019-03-29 09:33:13,332: WARNING/MainProcess] self.db[key] = value
[2019-03-29 09:33:13,332: WARNING/MainProcess] bsddb.db
[2019-03-29 09:33:13,332: WARNING/MainProcess] .
[2019-03-29 09:33:13,332: WARNING/MainProcess] DBError
[2019-03-29 09:33:13,332: WARNING/MainProcess] :
[2019-03-29 09:33:13,333: WARNING/MainProcess] (9, 'Bad file descriptor -- BDB0142 ftruncate: 16384: Bad file descriptor')
[2019-03-29 09:33:13,368: WARNING/MainProcess] Exception
[2019-03-29 09:33:13,368: WARNING/MainProcess] bsddb.db
[2019-03-29 09:33:13,369: WARNING/MainProcess] .
[2019-03-29 09:33:13,369: WARNING/MainProcess] DBError
[2019-03-29 09:33:13,369: WARNING/MainProcess] :
[2019-03-29 09:33:13,369: WARNING/MainProcess] DBError(29, 'Illegal seek -- BDB3027 /var/run/celerybeat-schedule: unable to flush page: 3')
[2019-03-29 09:33:13,369: WARNING/MainProcess] in

Extra try

But after I commented the following two lines of code in my parameter -A: xxx.xxx.task.internal.engine.py

import celery.concurrency.prefork as celery_prefork
celery_prefork.process_destructor = process_destructor
app = Celery('tasks', backend='redis://localhost:6543', broker='redis://localhost:6543//')
app.conf.update(
CELERY_DEFAULT_QUEUE='xxx',
xxx
)

and try to start celery beat , the celery beat start successfully.

But I want keep two lines of code in -A proj to custom "celery.concurrency.prefork".

I don't know why.

And if I come back to the 4.1.0 branch of Celery, the problem does not appear.

Issue 2:
As a new celery user I cannot discern if celeryd should be ran with with both /etc/default/celeryd config AND Django settings.py, or one or the other? If just Django settings.py, how does the /etc/init.d/celeryd script know that it is there? If both, why are there duplicate settings? Or maybe /etc/default/celeryd isn't needed with django as the project claims it can pull everything from Django settings?

None of the above questions are answered in the documentation. The documentation just sort of lays out options to try and I have to do each one experimentally and see if the result works or not.