Issue 1:
Checklist

I have verified that the issue exists against the master branch of Celery.

This has already been asked to the discussion group first.

I have read the relevant section in the

contribution guide

on reporting bugs.

I have checked the issues list

for similar or identical bug reports.

I have checked the pull requests list

for existing proposed fixes.

I have checked the commit log

to find out if the bug was already fixed in the master branch.

I have included all related issues and possible duplicate issues

in this issue (If there are none, check this box anyway).

Mandatory Debugging Information

I have included the output of celery -A proj report in the issue.

(if you are not able to do this, then at least specify the Celery

version affected).

I have verified that the issue exists against the master branch of Celery.

I have included the contents of pip freeze in the issue.

I have included all the versions of all the external dependencies required

to reproduce this bug.

Optional Debugging Information

I have tried reproducing the issue on more than one Python version

and/or implementation.

I have tried reproducing the issue on more than one message broker and/or

result backend.

I have tried reproducing the issue on more than one version of the message

broker and/or result backend.

I have tried reproducing the issue on more than one operating system.

I have tried reproducing the issue on more than one workers pool.

I have tried reproducing the issue with autoscaling, retries,

ETA/Countdown & rate limits disabled.

I have tried reproducing the issue after downgrading

and/or upgrading Celery and its dependencies.

Related Issues and Possible Duplicates

Related Issues

None

Possible Duplicates

None

Environment & Settings

Celery version: 4.4.7 (cliffs)

celery report Output:

celery report

software -> celery:4.4.7 (cliffs) kombu:4.6.11 py:3.8.12
billiard:3.6.4.0 py-amqp:2.6.1
platform -> system:Linux arch:64bit
kernel version:5.10.47-linuxkit imp:CPython
loader -> celery.loaders.default.Loader
settings -> transport:amqp results:disabled

Steps to Reproduce

Required Dependencies

Minimal Python Version: N/A or Unknown

Minimal Celery Version: N/A or Unknown

Minimal Kombu Version: N/A or Unknown

Minimal Broker Version: N/A or Unknown

Minimal Result Backend Version: N/A or Unknown

Minimal OS and/or Kernel Version: N/A or Unknown

Minimal Broker Client Version: N/A or Unknown

Minimal Result Backend Client Version: N/A or Unknown

Python Packages

pip freeze Output:

amqp==2.6.1
asgiref==3.4.1
attrs==21.2.0
Babel==2.9.1
billiard==3.6.4.0
boto3==1.14.29
botocore==1.17.63
celery==4.4.7
certifi==2021.5.30
chardet==3.0.4
click==8.0.1
Django==3.2
django-cors-headers==3.4.0
django-environ==0.4.5
django-filter==2.3.0
django-phonenumber-field==4.0.0
django-storages==1.9.1
django-tenants==3.3.0
djangorestframework==3.12.4
djangorestframework-camel-case==1.1.2
djangorestframework-simplejwt==4.6.0
docutils==0.15.2
drf-spectacular==0.14.0
gunicorn==20.0.4
h11==0.12.0
idna==2.10
inflection==0.5.1
jmespath==0.10.0
jsonschema==3.2.0
kombu==4.6.11
newrelic==5.22.0.151
phonenumbers==8.12.2
Pillow==8.1.2
psycopg2==2.8.6
PyJWT==2.1.0
pyrsistent==0.18.0
python-dateutil==2.8.2
pytz==2021.1
PyYAML==5.4.1
requests==2.24.0
s3transfer==0.3.7
six==1.16.0
sqlparse==0.4.2
uritemplate==3.0.1
urllib3==1.25.11
uvicorn==0.14.0
vine==1.3.0

Other Dependencies

GDAL 3.2.3
GEOS 3.9.1
Postgres 13.1
Postgis 3.1
Docker/docker-compose

Minimally Reproducible Test Case

When using Postigs with Django, it is required to set a couple of variables in settings.py, so that Django can detect the required dynamic libs. When we run celery, celery calls django.setup(), but that doesn't register the defined paths, which leads to an import error in Django.

# dlibs for gis
if env.get_value("GDAL_LIBRARY_PATH", default=None):
GDAL_LIBRARY_PATH = env("GDAL_LIBRARY_PATH")
if env.get_value("GEOS_LIBRARY_PATH", default=None):
GEOS_LIBRARY_PATH = env("GEOS_LIBRARY_PATH")

Expected Behavior

For settings to be configured properly and celery starts normally.

Actual Behavior

It raises an import error while trying to import the dynamic library, the path of which has been defined in settings.

celery_1 | Traceback (most recent call last):
celery_1 | File "/usr/local/bin/celery", line 8, in <module>
celery_1 | sys.exit(main())
celery_1 | File "/usr/local/lib/python3.8/site-packages/celery/__main__.py", line 16, in main
celery_1 | _main()
celery_1 | File "/usr/local/lib/python3.8/site-packages/celery/bin/celery.py", line 322, in main
celery_1 | cmd.execute_from_commandline(argv)
celery_1 | File "/usr/local/lib/python3.8/site-packages/celery/bin/celery.py", line 499, in execute_from_commandline
celery_1 | super(CeleryCommand, self).execute_from_commandline(argv)))
celery_1 | File "/usr/local/lib/python3.8/site-packages/celery/bin/base.py", line 305, in execute_from_commandline
celery_1 | return self.handle_argv(self.prog_name, argv[1:])
celery_1 | File "/usr/local/lib/python3.8/site-packages/celery/bin/celery.py", line 491, in handle_argv
celery_1 | return self.execute(command, argv)
celery_1 | File "/usr/local/lib/python3.8/site-packages/celery/bin/celery.py", line 415, in execute
celery_1 | return cls(
celery_1 | File "/usr/local/lib/python3.8/site-packages/celery/bin/worker.py", line 223, in run_from_argv
celery_1 | return self(*args, **options)
celery_1 | File "/usr/local/lib/python3.8/site-packages/celery/bin/base.py", line 253, in __call__
celery_1 | ret = self.run(*args, **kwargs)
celery_1 | File "/usr/local/lib/python3.8/site-packages/celery/bin/worker.py", line 253, in run
celery_1 | worker = self.app.Worker(
celery_1 | File "/usr/local/lib/python3.8/site-packages/celery/worker/worker.py", line 99, in __init__
celery_1 | self.app.loader.init_worker()
celery_1 | File "/usr/local/lib/python3.8/site-packages/celery/loaders/base.py", line 114, in init_worker
celery_1 | self.import_default_modules()
celery_1 | File "/usr/local/lib/python3.8/site-packages/celery/loaders/base.py", line 108, in import_default_modules
celery_1 | raise response
celery_1 | File "/usr/local/lib/python3.8/site-packages/celery/utils/dispatch/signal.py", line 288, in send
celery_1 | response = receiver(signal=self, sender=sender, **named)
celery_1 | File "/usr/local/lib/python3.8/site-packages/celery/fixups/django.py", line 84, in on_import_modules
celery_1 | self.worker_fixup.validate_models()
celery_1 | File "/usr/local/lib/python3.8/site-packages/celery/fixups/django.py", line 122, in validate_models
celery_1 | self.django_setup()
celery_1 | File "/usr/local/lib/python3.8/site-packages/celery/fixups/django.py", line 118, in django_setup
celery_1 | django.setup()
celery_1 | File "/usr/local/lib/python3.8/site-packages/django/__init__.py", line 24, in setup
celery_1 | apps.populate(settings.INSTALLED_APPS)
celery_1 | File "/usr/local/lib/python3.8/site-packages/django/apps/registry.py", line 114, in populate
celery_1 | app_config.import_models()
celery_1 | File "/usr/local/lib/python3.8/site-packages/django/apps/config.py", line 301, in import_models
celery_1 | self.models_module = import_module(models_module_name)
celery_1 | File "/usr/local/lib/python3.8/importlib/__init__.py", line 127, in import_module
celery_1 | return _bootstrap._gcd_import(name[level:], package, level)
celery_1 | File "<frozen importlib._bootstrap>", line 1014, in _gcd_import
celery_1 | File "<frozen importlib._bootstrap>", line 991, in _find_and_load
celery_1 | File "<frozen importlib._bootstrap>", line 975, in _find_and_load_unlocked
celery_1 | File "<frozen importlib._bootstrap>", line 671, in _load_unlocked
celery_1 | File "<frozen importlib._bootstrap_external>", line 843, in exec_module
celery_1 | File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
celery_1 | File "/usr/local/lib/python3.8/site-packages/django_tenants/models.py", line 8, in <module>
celery_1 | from .postgresql_backend.base import _check_schema_name
celery_1 | File "/usr/local/lib/python3.8/site-packages/django_tenants/postgresql_backend/base.py", line 19, in <module>
celery_1 | original_backend = import_module(ORIGINAL_BACKEND + '.base')
celery_1 | File "/usr/local/lib/python3.8/importlib/__init__.py", line 127, in import_module
celery_1 | return _bootstrap._gcd_import(name[level:], package, level)
celery_1 | File "/usr/local/lib/python3.8/site-packages/django/contrib/gis/db/backends/postgis/base.py", line 6, in <module>
celery_1 | from .features import DatabaseFeatures
celery_1 | File "/usr/local/lib/python3.8/site-packages/django/contrib/gis/db/backends/postgis/features.py", line 1, in <module>
celery_1 | from django.contrib.gis.db.backends.base.features import BaseSpatialFeatures
celery_1 | File "/usr/local/lib/python3.8/site-packages/django/contrib/gis/db/backends/base/features.py", line 3, in <module>
celery_1 | from django.contrib.gis.db import models
celery_1 | File "/usr/local/lib/python3.8/site-packages/django/contrib/gis/db/models/__init__.py", line 3, in <module>
celery_1 | import django.contrib.gis.db.models.functions # NOQA
celery_1 | File "/usr/local/lib/python3.8/site-packages/django/contrib/gis/db/models/functions.py", line 3, in <module>
celery_1 | from django.contrib.gis.db.models.fields import BaseSpatialField, GeometryField
celery_1 | File "/usr/local/lib/python3.8/site-packages/django/contrib/gis/db/models/fields.py", line 3, in <module>
celery_1 | from django.contrib.gis import forms, gdal
celery_1 | File "/usr/local/lib/python3.8/site-packages/django/contrib/gis/forms/__init__.py", line 3, in <module>
celery_1 | from .fields import ( # NOQA
celery_1 | File "/usr/local/lib/python3.8/site-packages/django/contrib/gis/forms/fields.py", line 2, in <module>
celery_1 | from django.contrib.gis.gdal import GDALException
celery_1 | File "/usr/local/lib/python3.8/site-packages/django/contrib/gis/gdal/__init__.py", line 28, in <module>
celery_1 | from django.contrib.gis.gdal.datasource import DataSource
celery_1 | File "/usr/local/lib/python3.8/site-packages/django/contrib/gis/gdal/datasource.py", line 40, in <module>
celery_1 | from django.contrib.gis.gdal.driver import Driver
celery_1 | File "/usr/local/lib/python3.8/site-packages/django/contrib/gis/gdal/driver.py", line 5, in <module>
celery_1 | from django.contrib.gis.gdal.prototypes import ds as vcapi, raster as rcapi
celery_1 | File "/usr/local/lib/python3.8/site-packages/django/contrib/gis/gdal/prototypes/ds.py", line 9, in <module>
celery_1 | from django.contrib.gis.gdal.libgdal import GDAL_VERSION, lgdal
celery_1 | File "/usr/local/lib/python3.8/site-packages/django/contrib/gis/gdal/libgdal.py", line 43, in <module>
celery_1 | raise ImproperlyConfigured(
celery_1 | django.core.exceptions.ImproperlyConfigured: Could not find the GDAL library (tried "gdal", "GDAL", "gdal3.1.0", "gdal3.0.0", "gdal2.4.0", "gdal2.3.0", "gdal2.2.0", "gdal2.1.0", "gdal2.0.0")
. Is GDAL installed? If it is, try setting GDAL_LIBRARY_PATH in your settings.

Issue 2:
Checklist

I have verified that the issue exists against the master branch of Celery.

This has already been asked to the discussion group first.

I have read the relevant section in the

contribution guide

on reporting bugs.

I have checked the issues list

for similar or identical bug reports.

I have checked the pull requests list

for existing proposed fixes.

I have checked the commit log

to find out if the bug was already fixed in the master branch.

I have included all related issues and possible duplicate issues

in this issue (If there are none, check this box anyway).

Mandatory Debugging Information

I have included the output of celery -A proj report in the issue.

(if you are not able to do this, then at least specify the Celery

version affected).

I have verified that the issue exists against the master branch of Celery.

I have included the contents of pip freeze in the issue.

I have included all the versions of all the external dependencies required

to reproduce this bug.

Optional Debugging Information

I have tried reproducing the issue on more than one Python version

and/or implementation.

I have tried reproducing the issue on more than one message broker and/or

result backend.

I have tried reproducing the issue on more than one version of the message

broker and/or result backend.

I have tried reproducing the issue on more than one operating system.

I have tried reproducing the issue on more than one workers pool.

I have tried reproducing the issue with autoscaling, retries,

ETA/Countdown & rate limits disabled.

I have tried reproducing the issue after downgrading

and/or upgrading Celery and its dependencies.

Related Issues and Possible Duplicates

Related Issues

#5868

Possible Duplicates

None

Environment & Settings

Celery version:

celery report Output:

software -> celery:4.4.1 (cliffs) kombu:4.6.8 py:3.7.4

billiard:3.6.3.0 py-amqp:2.5.2

platform -> system:Linux arch:64bit, ELF

kernel version:3.10.0-957.21.3.el7.x86_64 imp:CPython

loader -> celery.loaders.default.Loader

settings -> transport:amqp results:disabled

Steps to Reproduce

Required Dependencies

Minimal Python Version: N/A or Unknown

Minimal Celery Version: 4.4.1

Minimal Kombu Version: N/A or Unknown

Minimal Broker Version: redis/3.2.12

Minimal Result Backend Version: redis/3.2.12

Minimal OS and/or Kernel Version: N/A or Unknown

Minimal Broker Client Version: N/A or Unknown

Minimal Result Backend Client Version: N/A or Unknown

Python Packages

pip freeze Output:

amqp==2.5.2

billiard==3.6.3.0

celery==4.4.1

importlib-metadata==1.5.0

kombu==4.6.8

pytz==2019.3

redis==3.4.1

vine==1.3.0

zipp==3.1.0

Other Dependencies

N/A

Minimally Reproducible Test Case

app file is task_4_4.py

import time
from celery import Celery
from celery.signals import task_received
from celery.contrib.abortable import AbortableTask
redis_url="redis://localhost/2"
app=Celery(broker=redis_url,backend=redis_url)
@app.task(bind=True,base=AbortableTask,acks_late=False)
def add(self,x,y):
time.sleep(x)
return x+y
@app.task(bind=True)
def mul(self,x,y):
return x*y

start worker

celery worker -c 1 -l debug -A task_4_4

send tasks to the worker

from task_4_4 import add
import time
result1=add.delay(18000,11)
time.sleep(7200)
result2=add.delay(10,12)
result3=add.delay(1,13)

Expected Behavior

worker executes tasks normally

Actual Behavior

celery debug output file:

/root/celery/venv_4/lib/python3.7/site-packages/celery/platforms.py:801: RuntimeWarning: You're running the worker with superuser privileges: this is
absolutely not recommended!

Please specify a different user using the --uid option.

User information: uid=0 euid=0 gid=0 egid=0

uid=uid, euid=euid, gid=gid, egid=egid,

[2020-03-04 23:38:51,487: DEBUG/MainProcess] | Worker: Preparing bootsteps.

[2020-03-04 23:38:51,488: DEBUG/MainProcess] | Worker: Building graph...

[2020-03-04 23:38:51,489: DEBUG/MainProcess] | Worker: New boot order: {Beat, StateDB, Timer, Hub, Pool, Autoscaler, Consumer}

[2020-03-04 23:38:51,498: DEBUG/MainProcess] | Consumer: Preparing bootsteps.

[2020-03-04 23:38:51,498: DEBUG/MainProcess] | Consumer: Building graph...

[2020-03-04 23:38:51,515: DEBUG/MainProcess] | Consumer: New boot order: {Connection, Events, Mingle, Gossip, Tasks, Control, Heart, Agent, event loop}

[2020-03-04 23:38:51,524: DEBUG/MainProcess] | Worker: Starting Hub

[2020-03-04 23:38:51,524: DEBUG/MainProcess] ^-- substep ok

[2020-03-04 23:38:51,524: DEBUG/MainProcess] | Worker: Starting Pool

[2020-03-04 23:38:51,537: DEBUG/MainProcess] ^-- substep ok

[2020-03-04 23:38:51,537: DEBUG/MainProcess] | Worker: Starting Consumer

[2020-03-04 23:38:51,538: DEBUG/MainProcess] | Consumer: Starting Connection

[2020-03-04 23:38:51,554: INFO/MainProcess] Connected to redis://localhost:6379/2

[2020-03-04 23:38:51,554: DEBUG/MainProcess] ^-- substep ok

[2020-03-04 23:38:51,554: DEBUG/MainProcess] | Consumer: Starting Events

[2020-03-04 23:38:51,560: DEBUG/MainProcess] ^-- substep ok

[2020-03-04 23:38:51,560: DEBUG/MainProcess] | Consumer: Starting Mingle

[2020-03-04 23:38:51,560: INFO/MainProcess] mingle: searching for neighbors

[2020-03-04 23:38:52,577: INFO/MainProcess] mingle: all alone

[2020-03-04 23:38:52,577: DEBUG/MainProcess] ^-- substep ok

[2020-03-04 23:38:52,577: DEBUG/MainProcess] | Consumer: Starting Gossip

[2020-03-04 23:38:52,580: DEBUG/MainProcess] ^-- substep ok

[2020-03-04 23:38:52,580: DEBUG/MainProcess] | Consumer: Starting Tasks

[2020-03-04 23:38:52,583: DEBUG/MainProcess] ^-- substep ok

[2020-03-04 23:38:52,583: DEBUG/MainProcess] | Consumer: Starting Control

[2020-03-04 23:38:52,585: DEBUG/MainProcess] ^-- substep ok

[2020-03-04 23:38:52,586: DEBUG/MainProcess] | Consumer: Starting Heart

[2020-03-04 23:38:52,587: DEBUG/MainProcess] ^-- substep ok

[2020-03-04 23:38:52,587: DEBUG/MainProcess] | Consumer: Starting event loop

[2020-03-04 23:38:52,587: DEBUG/MainProcess] | Worker: Hub.register Pool...

[2020-03-04 23:38:52,587: INFO/MainProcess] celery@VM_0_6_centos ready.

[2020-03-04 23:38:52,588: DEBUG/MainProcess] basic.qos: prefetch_count->4

[2020-03-04 23:39:58,427: INFO/MainProcess] Received task: task_4_4.add[e5e6a491-febf-4bf9-8ff1-d31c0f8b45b5]

[2020-03-04 23:39:58,427: DEBUG/MainProcess] TaskPool: Apply <function _fast_trace_task at 0x7fcd0214b7a0> (args:('task_4_4.add', 'e5e6a491-febf-4bf9-8ff1-d31c0f8b45b5', {'lang': 'py', 'task': 'task_4_4.add', 'id': 'e5e6a491-febf-4bf9-8ff1-d31c0f8b45b5', 'shadow': None, 'eta': None, 'expires': None, 'group': None, 'retries': 0, 'timelimit': [None, None], 'root_id': 'e5e6a491-febf-4bf9-8ff1-d31c0f8b45b5', 'parent_id': None, 'argsrepr': '(18000, 11)', 'kwargsrepr': '{}', 'origin': 'gen7195@VM_0_6_centos', 'reply_to': '7f0f5f08-37d3-33c0-8560-6404392956f3', 'correlation_id': 'e5e6a491-febf-4bf9-8ff1-d31c0f8b45b5', 'hostname': 'celery@VM_0_6_centos', 'delivery_info': {'exchange': '', 'routing_key': 'celery', 'priority': 0, 'redelivered': None}, 'args': [18000, 11], 'kwargs': {}}, b'[[18000, 11], {}, {"callbacks": null, "errbacks": null, "chain": null, "chord": null}]', 'application/json', 'utf-8') kwargs:{})

[2020-03-04 23:39:58,429: DEBUG/MainProcess] Task accepted: task_4_4.add[e5e6a491-febf-4bf9-8ff1-d31c0f8b45b5] pid:6979

[2020-03-05 01:39:58,530: INFO/MainProcess] Received task: task_4_4.add[692978f1-df60-430e-acaa-87a224624bfc]

[2020-03-05 01:39:58,532: INFO/MainProcess] Received task: task_4_4.add[9dd6a5d2-1efb-41f2-91f8-7da458b6795e]

[2020-03-05 02:40:22,966: INFO/MainProcess] Received task: task_4_4.add[692978f1-df60-430e-acaa-87a224624bfc]

[2020-03-05 02:40:22,967: INFO/MainProcess] Received task: task_4_4.add[9dd6a5d2-1efb-41f2-91f8-7da458b6795e]

Restarting celery worker (/root/celery/venv_4/bin/celery worker -c 1 -l debug -A task_4_4)

[2020-03-05 02:54:26,797: DEBUG/MainProcess] | Worker: Closing Hub...

[2020-03-05 02:54:26,797: DEBUG/MainProcess] | Worker: Closing Pool...

[2020-03-05 02:54:26,797: DEBUG/MainProcess] | Worker: Closing Consumer...

[2020-03-05 02:54:26,797: DEBUG/MainProcess] | Worker: Stopping Consumer...

[2020-03-05 02:54:26,797: DEBUG/MainProcess] | Consumer: Closing Connection...

[2020-03-05 02:54:26,797: DEBUG/MainProcess] | Consumer: Closing Events...

[2020-03-05 02:54:26,797: DEBUG/MainProcess] | Consumer: Closing Mingle...

[2020-03-05 02:54:26,797: DEBUG/MainProcess] | Consumer: Closing Gossip...

[2020-03-05 02:54:26,797: DEBUG/MainProcess] | Consumer: Closing Tasks...

[2020-03-05 02:54:26,797: DEBUG/MainProcess] | Consumer: Closing Control...

[2020-03-05 02:54:26,797: DEBUG/MainProcess] | Consumer: Closing Heart...

[2020-03-05 02:54:26,797: DEBUG/MainProcess] | Consumer: Closing event loop...

[2020-03-05 02:54:26,797: DEBUG/MainProcess] | Consumer: Stopping event loop...

[2020-03-05 02:54:26,797: DEBUG/MainProcess] | Consumer: Stopping Heart...

[2020-03-05 02:54:26,802: DEBUG/MainProcess] | Consumer: Stopping Control...

[2020-03-05 02:54:26,803: DEBUG/MainProcess] | Consumer: Stopping Tasks...

[2020-03-05 02:54:26,803: DEBUG/MainProcess] Canceling task consumer...

[2020-03-05 02:54:26,803: DEBUG/MainProcess] | Consumer: Stopping Gossip...

[2020-03-05 02:54:26,805: DEBUG/MainProcess] | Consumer: Stopping Mingle...

[2020-03-05 02:54:26,805: DEBUG/MainProcess] | Consumer: Stopping Events...

[2020-03-05 02:54:26,805: DEBUG/MainProcess] | Consumer: Stopping Connection...

[2020-03-05 02:54:26,805: DEBUG/MainProcess] | Worker: Stopping Pool...

[2020-03-05 02:54:27,306: ERROR/MainProcess] Process 'ForkPoolWorker-1' pid:6979 exited with 'exitcode 70'

[2020-03-05 02:54:27,319: ERROR/MainProcess] Task handler raised error: WorkerLostError('Worker exited prematurely: exitcode 70.')

Traceback (most recent call last):

File "/root/celery/venv_4/lib/python3.7/site-packages/celery/worker/worker.py", line 205, in start

self.blueprint.start(self)

File "/root/celery/venv_4/lib/python3.7/site-packages/celery/bootsteps.py", line 119, in start

step.start(parent)

File "/root/celery/venv_4/lib/python3.7/site-packages/celery/bootsteps.py", line 369, in start

return self.obj.start()

File "/root/celery/venv_4/lib/python3.7/site-packages/celery/worker/consumer/consumer.py", line 318, in start

blueprint.start(self)

File "/root/celery/venv_4/lib/python3.7/site-packages/celery/bootsteps.py", line 119, in start

step.start(parent)

File "/root/celery/venv_4/lib/python3.7/site-packages/celery/worker/consumer/consumer.py", line 599, in start

c.loop(*c.loop_args())

File "/root/celery/venv_4/lib/python3.7/site-packages/celery/worker/loops.py", line 74, in asynloop

state.maybe_shutdown()

File "/root/celery/venv_4/lib/python3.7/site-packages/celery/worker/state.py", line 80, in maybe_shutdown

raise WorkerShutdown(should_stop)

celery.exceptions.WorkerShutdown: 0

During handling of the above exception, another exception occurred:

Traceback (most recent call last):

File "/root/celery/venv_4/lib/python3.7/site-packages/billiard/pool.py", line 1267, in mark_as_worker_lost

human_status(exitcode)),

billiard.exceptions.WorkerLostError: Worker exited prematurely: exitcode 70.

[2020-03-05 02:54:27,319: DEBUG/MainProcess] | Worker: Stopping Hub...

[2020-03-05 02:54:27,319: DEBUG/MainProcess] | Consumer: Shutdown Heart...

[2020-03-05 02:54:27,319: DEBUG/MainProcess] | Consumer: Shutdown Control...

[2020-03-05 02:54:27,319: DEBUG/MainProcess] | Consumer: Shutdown Tasks...

[2020-03-05 02:54:27,320: DEBUG/MainProcess] Canceling task consumer...

[2020-03-05 02:54:27,320: DEBUG/MainProcess] Closing consumer channel...

[2020-03-05 02:54:27,320: DEBUG/MainProcess] | Consumer: Shutdown Gossip...

[2020-03-05 02:54:27,320: DEBUG/MainProcess] | Consumer: Shutdown Events...

[2020-03-05 02:54:27,320: DEBUG/MainProcess] | Consumer: Shutdown Connection...

/root/celery/venv_4/lib/python3.7/site-packages/celery/platforms.py:801: RuntimeWarning: You're running the worker with superuser privileges: this is

absolutely not recommended!

Please specify a different user using the --uid option.

User information: uid=0 euid=0 gid=0 egid=0

uid=uid, euid=euid, gid=gid, egid=egid,

[2020-03-05 02:54:27,713: DEBUG/MainProcess] | Worker: Preparing bootsteps.

[2020-03-05 02:54:27,714: DEBUG/MainProcess] | Worker: Building graph...

[2020-03-05 02:54:27,714: DEBUG/MainProcess] | Worker: New boot order: {Beat, Timer, Hub, Pool, Autoscaler, StateDB, Consumer}

[2020-03-05 02:54:27,723: DEBUG/MainProcess] | Consumer: Preparing bootsteps.

[2020-03-05 02:54:27,723: DEBUG/MainProcess] | Consumer: Building graph...

[2020-03-05 02:54:27,740: DEBUG/MainProcess] | Consumer: New boot order: {Connection, Events, Mingle, Tasks, Control, Heart, Agent, Gossip, event loop}

[2020-03-05 02:54:27,749: DEBUG/MainProcess] | Worker: Starting Hub

[2020-03-05 02:54:27,749: DEBUG/MainProcess] ^-- substep ok

[2020-03-05 02:54:27,749: DEBUG/MainProcess] | Worker: Starting Pool

[2020-03-05 02:54:27,761: DEBUG/MainProcess] ^-- substep ok

[2020-03-05 02:54:27,762: DEBUG/MainProcess] | Worker: Starting Consumer

[2020-03-05 02:54:27,762: DEBUG/MainProcess] | Consumer: Starting Connection

[2020-03-05 02:54:27,777: INFO/MainProcess] Connected to redis://localhost:6379/2

[2020-03-05 02:54:27,778: DEBUG/MainProcess] ^-- substep ok

[2020-03-05 02:54:27,778: DEBUG/MainProcess] | Consumer: Starting Events

[2020-03-05 02:54:27,783: DEBUG/MainProcess] ^-- substep ok

[2020-03-05 02:54:27,784: DEBUG/MainProcess] | Consumer: Starting Mingle

[2020-03-05 02:54:27,784: INFO/MainProcess] mingle: searching for neighbors

[2020-03-05 02:54:28,800: INFO/MainProcess] mingle: all alone

[2020-03-05 02:54:28,800: DEBUG/MainProcess] ^-- substep ok

[2020-03-05 02:54:28,800: DEBUG/MainProcess] | Consumer: Starting Tasks

[2020-03-05 02:54:28,803: DEBUG/MainProcess] ^-- substep ok

[2020-03-05 02:54:28,803: DEBUG/MainProcess] | Consumer: Starting Control

[2020-03-05 02:54:28,806: DEBUG/MainProcess] ^-- substep ok

[2020-03-05 02:54:28,806: DEBUG/MainProcess] | Consumer: Starting Heart

[2020-03-05 02:54:28,807: DEBUG/MainProcess] ^-- substep ok

[2020-03-05 02:54:28,807: DEBUG/MainProcess] | Consumer: Starting Gossip

[2020-03-05 02:54:28,809: DEBUG/MainProcess] ^-- substep ok

[2020-03-05 02:54:28,810: DEBUG/MainProcess] | Consumer: Starting event loop

[2020-03-05 02:54:28,810: DEBUG/MainProcess] | Worker: Hub.register Pool...

[2020-03-05 02:54:28,810: INFO/MainProcess] celery@VM_0_6_centos ready.

[2020-03-05 02:54:28,810: DEBUG/MainProcess] basic.qos: prefetch_count->4

[2020-03-05 02:54:29,122: INFO/MainProcess] Received task: task_4_4.add[9dd6a5d2-1efb-41f2-91f8-7da458b6795e]

[2020-03-05 02:54:29,122: DEBUG/MainProcess] TaskPool: Apply <function _fast_trace_task at 0x7fc064cac7a0> (args:('task_4_4.add', '9dd6a5d2-1efb-41f2-91f8-7da458b6795e', {'lang': 'py', 'task': 'task_4_4.add', 'id': '9dd6a5d2-1efb-41f2-91f8-7da458b6795e', 'shadow': None, 'eta': None, 'expires': None, 'group': None, 'retries': 0, 'timelimit': [None, None], 'root_id': '9dd6a5d2-1efb-41f2-91f8-7da458b6795e', 'parent_id': None, 'argsrepr': '(1, 13)', 'kwargsrepr': '{}', 'origin': 'gen7195@VM_0_6_centos', 'redelivered': True, 'reply_to': '7f0f5f08-37d3-33c0-8560-6404392956f3', 'correlation_id': '9dd6a5d2-1efb-41f2-91f8-7da458b6795e', 'hostname': 'celery@VM_0_6_centos', 'delivery_info': {'exchange': '', 'routing_key': 'celery', 'priority': 0, 'redelivered': None}, 'args': [1, 13], 'kwargs': {}}, b'[[1, 13], {}, {"callbacks": null, "errbacks": null, "chain": null, "chord": null}]', 'application/json', 'utf-8') kwargs:{})

[2020-03-05 02:54:29,124: INFO/MainProcess] Received task: task_4_4.add[692978f1-df60-430e-acaa-87a224624bfc]

[2020-03-05 02:54:29,124: DEBUG/MainProcess] Task accepted: task_4_4.add[9dd6a5d2-1efb-41f2-91f8-7da458b6795e] pid:10494

[2020-03-05 02:54:30,128: INFO/ForkPoolWorker-1] Task task_4_4.add[9dd6a5d2-1efb-41f2-91f8-7da458b6795e] succeeded in 1.0049351528286934s: 14

[2020-03-05 02:54:30,128: DEBUG/MainProcess] TaskPool: Apply <function _fast_trace_task at 0x7fc064cac7a0> (args:('task_4_4.add', '692978f1-df60-430e-acaa-87a224624bfc', {'lang': 'py', 'task': 'task_4_4.add', 'id': '692978f1-df60-430e-acaa-87a224624bfc', 'shadow': None, 'eta': None, 'expires': None, 'group': None, 'retries': 0, 'timelimit': [None, None], 'root_id': '692978f1-df60-430e-acaa-87a224624bfc', 'parent_id': None, 'argsrepr': '(10, 12)', 'kwargsrepr': '{}', 'origin': 'gen7195@VM_0_6_centos', 'redelivered': True, 'reply_to': '7f0f5f08-37d3-33c0-8560-6404392956f3', 'correlation_id': '692978f1-df60-430e-acaa-87a224624bfc', 'hostname': 'celery@VM_0_6_centos', 'delivery_info': {'exchange': '', 'routing_key': 'celery', 'priority': 0, 'redelivered': None}, 'args': [10, 12], 'kwargs': {}}, b'[[10, 12], {}, {"callbacks": null, "errbacks": null, "chain": null, "chord": null}]', 'application/json', 'utf-8') kwargs:{})

[2020-03-05 02:54:30,129: DEBUG/MainProcess] Task accepted: task_4_4.add[692978f1-df60-430e-acaa-87a224624bfc] pid:10494

[2020-03-05 02:54:40,139: INFO/ForkPoolWorker-1] Task task_4_4.add[692978f1-df60-430e-acaa-87a224624bfc] succeeded in 10.010058987885714s: 22

As the debug output implies, celery worker shutdown 3 hours after it received the first task.