Issue 1:
Checklist

[ x] I have verified that the issue exists against the master branch of Celery.

This has already been asked to the discussion group first.

[x ] I have read the relevant section in the

contribution guide

on reporting bugs.

[x ] I have checked the issues list

for similar or identical bug reports.

I have checked the pull requests list

for existing proposed fixes.

I have checked the commit log

to find out if the bug was already fixed in the master branch.

I have included all related issues and possible duplicate issues

in this issue (If there are none, check this box anyway).

Mandatory Debugging Information

I have included the output of celery -A proj report in the issue.

(if you are not able to do this, then at least specify the Celery

version affected).

I have verified that the issue exists against the master branch of Celery.

I have included the contents of pip freeze in the issue.

I have included all the versions of all the external dependencies required

to reproduce this bug.

Optional Debugging Information

I have tried reproducing the issue on more than one Python version

and/or implementation.

I have tried reproducing the issue on more than one message broker and/or

result backend.

I have tried reproducing the issue on more than one version of the message

broker and/or result backend.

I have tried reproducing the issue on more than one operating system.

I have tried reproducing the issue on more than one workers pool.

I have tried reproducing the issue with autoscaling, retries,

ETA/Countdown & rate limits disabled.

I have tried reproducing the issue after downgrading

and/or upgrading Celery and its dependencies.

Related Issues and Possible Duplicates

Related Issues

None

Possible Duplicates

None

Environment & Settings

Celery version:

celery report Output:

software -> celery:4.4.2 (cliffs) kombu:4.6.8 py:3.7.5
billiard:3.6.3.0 py-amqp:2.5.2
platform -> system:Darwin arch:64bit
kernel version:18.7.0 imp:CPython
loader -> celery.loaders.default.Loader
settings -> transport:amqp results:disabled

Steps to Reproduce

Required Dependencies

Minimal Python Version: N/A or Unknown

Minimal Celery Version: N/A or Unknown

Minimal Kombu Version: N/A or Unknown

Minimal Broker Version: N/A or Unknown

Minimal Result Backend Version: N/A or Unknown

Minimal OS and/or Kernel Version: N/A or Unknown

Minimal Broker Client Version: N/A or Unknown

Minimal Result Backend Client Version: N/A or Unknown

Python Packages

pip freeze Output:

Other Dependencies

N/A

Minimally Reproducible Test Case

Expected Behavior

Backend: file system

steriliser: pickle

Scenario: Define a group and retrieve the result, for instance, group([add.s(1,2), add.s(3,4)])().get()

This should give the correct result of two adding calculation

Actual Behavior

_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
venv3/lib/python3.7/site-packages/celery/result.py:229: in get
on_message=on_message,
venv3/lib/python3.7/site-packages/celery/backends/base.py:579: in wait_for_pending
no_ack=no_ack,
venv3/lib/python3.7/site-packages/celery/backends/base.py:606: in wait_for
meta = self.get_task_meta(task_id)
venv3/lib/python3.7/site-packages/celery/backends/base.py:457: in get_task_meta
meta = self._get_task_meta_for(task_id)
venv3/lib/python3.7/site-packages/celery/backends/base.py:791: in _get_task_meta_for
return self.decode_result(meta)
venv3/lib/python3.7/site-packages/celery/backends/base.py:322: in decode_result
return self.meta_from_decoded(self.decode(payload))
venv3/lib/python3.7/site-packages/celery/backends/base.py:333: in decode
accept=self.accept)
venv3/lib/python3.7/site-packages/kombu/serialization.py:264: in loads
return decode(data)
/usr/local/Cellar/python/3.7.5/Frameworks/Python.framework/Versions/3.7/lib/python3.7/contextlib.py:130: in __exit__
self.gen.throw(type, value, traceback)
venv3/lib/python3.7/site-packages/kombu/serialization.py:54: in _reraise_errors
reraise(wrapper, wrapper(exc), sys.exc_info()[2])
venv3/lib/python3.7/site-packages/vine/five.py:194: in reraise
raise value.with_traceback(tb)
venv3/lib/python3.7/site-packages/kombu/serialization.py:50: in _reraise_errors
yield
venv3/lib/python3.7/site-packages/kombu/serialization.py:264: in loads
return decode(data)
venv3/lib/python3.7/site-packages/kombu/serialization.py:338: in unpickle
return pickle_loads(str_to_bytes(s))
venv3/lib/python3.7/site-packages/kombu/serialization.py:59: in pickle_loads
return load(BytesIO(s))
venv3/lib/python3.7/site-packages/celery/backends/base.py:69: in unpickle_backend
return cls(*args, app=current_app._get_current_object(), **kwargs)
venv3/lib/python3.7/site-packages/celery/backends/filesystem.py:53: in __init__
path = self._find_path(url)
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _

self = <celery.backends.filesystem.FilesystemBackend object at 0x10b2c7c90>, url = None

def _find_path(self, url):
if not url:
> raise ImproperlyConfigured(E_NO_PATH_SET)
E kombu.exceptions.DecodeError: You need to configure a path for the file-system backend

venv3/lib/python3.7/site-packages/celery/backends/filesystem.py:77: DecodeError

Solution

FilesystemBackend class defined in https://github.com/celery/celery/blob/master/celery/backends/filesystem.py#L34, misses __reduce__ method to pickle the instance properly. The following example would make it work.

def __reduce__(self, args=(), kwargs={}):
kwargs.update(
dict(url=self.url))
return super(FilesystemBackend, self).__reduce__(args, kwargs)

Issue 2:
Checklist

I have verified that the issue exists against the master branch of Celery.

This has already been asked to the discussion group first.

I have read the relevant section in the

contribution guide

on reporting bugs.

I have checked the issues list

for similar or identical bug reports.

I have checked the pull requests list

for existing proposed fixes.

I have checked the commit log

to find out if the bug was already fixed in the master branch.

I have included all related issues and possible duplicate issues

in this issue (If there are none, check this box anyway).

Mandatory Debugging Information

I have included the output of celery -A proj report in the issue.

(if you are not able to do this, then at least specify the Celery

version affected).

I have verified that the issue exists against the master branch of Celery.

I have included the contents of pip freeze in the issue.

I have included all the versions of all the external dependencies required

to reproduce this bug.

Optional Debugging Information

I have tried reproducing the issue on more than one Python version

and/or implementation.

I have tried reproducing the issue on more than one message broker and/or

result backend.

I have tried reproducing the issue on more than one version of the message

broker and/or result backend.

I have tried reproducing the issue on more than one operating system.

I have tried reproducing the issue on more than one workers pool.

I have tried reproducing the issue with autoscaling, retries,

ETA/Countdown & rate limits disabled.

I have tried reproducing the issue after downgrading

and/or upgrading Celery and its dependencies.

Related Issues and Possible Duplicates

Related Issues

#4371

Possible Duplicates

None

Environment & Settings

Celery version: 4.3.0 , 4.4.2

celery report Output:

software -> celery:4.3.0 (rhubarb) kombu:4.6.8 py:3.7.0

billiard:3.6.3.0 py-amqp:2.5.2

platform -> system:Linux arch:64bit

kernel version:4.15.0-91-generic imp:CPython

loader -> celery.loaders.app.AppLoader

settings -> transport:amqp results:disabled

broker_url: 'amqp://guest:********@localhost:5672//'

Steps to Reproduce

Required Dependencies

Minimal Python Version: N/A or Unknown

Minimal Celery Version: N/A or Unknown

Minimal Kombu Version: N/A or Unknown

Minimal Broker Version: N/A or Unknown

Minimal Result Backend Version: N/A or Unknown

Minimal OS and/or Kernel Version: N/A or Unknown

Minimal Broker Client Version: N/A or Unknown

Minimal Result Backend Client Version: N/A or Unknown

Python Packages

pip freeze Output:

aioprocessing==1.0.1

alabaster==0.7.12

alembic==1.2.1

amqp==2.5.2

aniso8601==6.0.0

astroid==2.3.3

astropy==3.2.3

atomicwrites==1.3.0

attrs==19.3.0

auto-sklearn==0.5.2

Babel==2.7.0

backcall==0.1.0

backports.csv==1.0.7

billiard==3.6.3.0

bleach==3.1.0

boto3==1.9.160

botocore==1.12.160

catboost==0.14.2

celery==4.3.0

celery-progress==0.0.3

certifi==2019.11.28

cffi==1.14.0

chardet==3.0.4

Click==7.0

cloudpickle==1.1.1

config==0.4.2

configparser==3.7.4

ConfigSpace==0.4.7

confuse==1.0.0

coreapi==2.3.3

coreschema==0.0.4

coverage==4.5.4

cycler==0.10.0

Cython==0.29.16

databricks-cli==0.9.0

debug==0.3.2

decorator==4.4.1

defusedxml==0.6.0

diff-match-patch==20181111

dill==0.2.9

Django==2.2.2

django-cors-headers==3.0.2

django-dbconn-retry==0.1.5

django-import-export==1.2.0

django-keycloak==0.1.1

django-rest-framework==0.1.0

django-user==0.2.1

djangorestframework==3.9.4

docker==4.0.1

docutils==0.14

drf-yasg==1.16.0

ds-lime==0.1.1.27

ecdsa==0.13.2

eli5==0.10.1

entrypoints==0.3

enum34==1.1.6

et-xmlfile==1.0.1

fastparquet==0.3.2

Flask==1.0.3

future==0.17.1

gitdb2==2.0.5

GitPython==2.1.11

gorilla==0.3.0

GPy==1.9.9

GPyOpt==1.2.5

graphene==2.1.7

graphene-django==2.5.0

graphene-file-upload==1.2.2

graphql-core==2.2.1

graphql-relay==2.0.0

graphviz==0.11

gunicorn==19.9.0

h5py==2.10.0

hpbandster==0.7.4

htmlmin==0.1.12

idna==2.8

imageio==2.5.0

imagesize==1.1.0

importlib-metadata==0.20

inflection==0.3.1

ipdb==0.13.2

ipython==7.13.0

ipython-genutils==0.2.0

isort==4.3.21

itsdangerous==1.1.0

itypes==1.1.0

jdcal==1.4.1

jedi==0.16.0

Jinja2==2.10.1

jmespath==0.9.4

joblib==0.13.2

jsonschema==3.0.1

jupyter-client==5.3.4

jupyter-core==4.6.1

Keras==2.3.1

Keras-Applications==1.0.8

Keras-Preprocessing==1.1.0

kiwisolver==1.1.0

kombu==4.6.8

lazy-import==0.2.2

lazy-object-proxy==1.4.3

liac-arff==2.4.0

lightgbm==2.2.3

lime==0.1.1.34

llvmlite==0.29.0

lockfile==0.12.2

Mako==1.1.0

MarkupSafe==1.1.1

matplotlib==3.1.0

mccabe==0.6.1

memory-profiler==0.55.0

missingno==0.4.2

mistune==0.8.4

mleap==0.8.1

mlflow==1.2.0

mock==3.0.5

more-itertools==8.2.0

multiprocess==0.70.7

nbconvert==5.6.1

nbformat==4.4.0

nest-asyncio==1.2.3

netifaces==0.10.9

networkx==2.3

nose==1.3.7

nose-exclude==0.5.0

numba==0.45.1

numpy==1.16.4

objgraph==3.4.1

odfpy==1.4.0

openpyxl==2.6.2

packaging==19.0

pandas==0.24.2

pandas-profiling==2.3.0

pandocfilters==1.4.2

paramz==0.9.5

parso==0.6.0

patsy==0.5.1

PDPbox==0.2.0

pexpect==4.8.0

phik==0.9.8

pickleshare==0.7.5

Pillow==6.0.0

plotly==3.10.0

pluggy==0.12.0

prometheus-client==0.7.1

promise==2.2.1

prompt-toolkit==3.0.4

protobuf==3.8.0

psutil==5.6.2

psycopg2==2.7.5

ptyprocess==0.6.0

py==1.8.0

py4j==0.10.7

pyarrow==0.14.1

pyasn1==0.4.5

pycparser==2.20

pydot==1.4.1

pyfiglet==0.8.post1

Pygments==2.4.2

PyHive==0.6.1

pylint==2.4.3

Pympler==0.7

pynisher==0.5.0

pyparsing==2.4.0

pyrfr==0.8.0

Pyro4==4.77

pyrsistent==0.15.3

pysasl==0.5.0

pyspark==2.4.3

pytest==5.1.2

pytest-cov==2.7.1

pytest-django==3.5.1

pytest-mock==1.10.4

pytest-pylint==0.14.1

python-dateutil==2.8.0

python-editor==1.0.4

python-jose==3.0.1

python-keycloak-client==0.2.2

pytz==2019.1

PyWavelets==1.0.3

PyYAML==5.1.1

pyzmq==18.1.1

querystring-parser==1.2.3

rabbitmq==0.2.0

requests==2.22.0

retrying==1.3.3

rsa==4.0

ruamel.yaml==0.15.97

Rx==1.6.1

s3transfer==0.2.0

sasl==0.2.1

scikit-image==0.15.0

scikit-learn==0.19.2

scipy==1.3.0

seaborn==0.9.0

see==1.4.1

Send2Trash==1.5.0

serpent==1.28

simplejson==3.16.0

singledispatch==3.4.0.3

six==1.14.0

skater==1.0.4

smac==0.8.0

smmap2==2.0.5

snowballstemmer==1.2.1

sobol-seq==0.1.2

spark-celery==0.1.1

Sphinx==2.1.0

sphinx-rtd-theme==0.4.3

sphinxcontrib-applehelp==1.0.1

sphinxcontrib-devhelp==1.0.1

sphinxcontrib-htmlhelp==1.0.2

sphinxcontrib-jsmath==1.0.1

sphinxcontrib-qthelp==1.0.2

sphinxcontrib-serializinghtml==1.1.3

SQLAlchemy==1.3.8

sqlparse==0.3.0

statsmodels==0.10.2

tablib==0.13.0

tabulate==0.8.3

testpath==0.4.4

thrift==0.11.0

thrift-sasl==0.3.0

tornado==6.0.3

traitlets==4.3.2

typed-ast==1.4.0

typing==3.6.6

uritemplate==3.0.0

urllib3==1.25.3

vine==1.3.0

wcwidth==0.1.7

webencodings==0.5.1

websocket-client==0.56.0

Werkzeug==0.15.4

wrapt==1.11.2

xgboost==0.90

xlrd==1.2.0

xlwt==1.3.0

yellowbrick==0.9.1

zipp==0.6.0

Other Dependencies

N/A

Minimally Reproducible Test Case

1. Write a task that causes an error that does not correspond to BaseExeption.
:

def foo():

process = psutil.Process(os.getpid())
print(f'foo process mem info{process.memory_full_info()}')
dummy = numpy.array(numpy.random.randint(100, 200, 300000000))
print(dummy.nbytes)
idx = 0
while idx < 2:
time.sleep(1)
idx = idx + 1
raise ValueError

Run celery in prefork mode. Set the work volume of each worker pool to 2 or more.

Keep trying foo.delay()

Expected Behavior

Resource release after appropriate handling of exceptions

Actual Behavior

foo.delay()

foo process mem infopfullmem(rss=35368960, vms=384946176, shared=4370432, text=2338816, lib=0, data=300494848, dirty=0, uss=9674752, pss=21053440, swap=0)

foo process mem infopfullmem(rss=2437492736, vms=2785394688, shared=5931008, text=2338816, lib=0, data=2700943360, dirty=0, uss=2410758144, pss=2422296576, swap=0)

foo process mem infopfullmem(rss=4837507072, vms=5185384448, shared=5922816, text=2338816, lib=0, data=5100933120, dirty=0, uss=4810747904, pss=4822307840, swap=0)

Memory builds up continuously