Issue 1:
Checklist

I have verified that the issue exists against the master branch of Celery.

This has already been asked to the discussion group first.

I have read the relevant section in the

contribution guide

on reporting bugs.

I have checked the issues list

for similar or identical bug reports.

I have checked the pull requests list

for existing proposed fixes.

I have checked the commit log

to find out if the bug was already fixed in the master branch.

I have included all related issues and possible duplicate issues

in this issue (If there are none, check this box anyway).

Mandatory Debugging Information

I have included the output of celery -A proj report in the issue.

(if you are not able to do this, then at least specify the Celery

version affected).

I have verified that the issue exists against the master branch of Celery.

I have included the contents of pip freeze in the issue.

I have included all the versions of all the external dependencies required

to reproduce this bug.

Optional Debugging Information

I have tried reproducing the issue on more than one Python version

and/or implementation.

I have tried reproducing the issue on more than one message broker and/or

result backend.

I have tried reproducing the issue on more than one version of the message

broker and/or result backend.

I have tried reproducing the issue on more than one operating system.

I have tried reproducing the issue on more than one workers pool.

I have tried reproducing the issue with autoscaling, retries,

ETA/Countdown & rate limits disabled.

I have tried reproducing the issue after downgrading

and/or upgrading Celery and its dependencies.

Related Issues and Possible Duplicates

Related Issues

#5661

#5660

Possible Duplicates

#5660

Environment & Settings

Celery version:

4.4.7

celery report Output:

types in document index requests is deprecated, use the typeless endpoints instead (/{index}/_doc/{id}, /{index}/_doc, or /{index}/_create/{id}).
warnings.warn(message, category=ElasticsearchDeprecationWarning)
[2020-08-18 09:14:06,421: ERROR/MainProcess] Pool callback raised exception: RequestError(400, 'illegal_argument_exception', {'error': {'root_cause': [{'type': 'illegal_argument_exception', 'reason': 'mapper [result.args] of different type, current_type [long], merged_type [text]'}], 'type': 'illegal_argument_exception', 'reason': 'mapper [result.args] of different type, current_type [long], merged_type [text]'}, 'status': 400})
Traceback (most recent call last):
File "/usr/local/lib/python3.7/site-packages/billiard/pool.py", line 1796, in safe_apply_callback
fun(*args, **kwargs)
File "/usr/local/lib/python3.7/site-packages/celery/worker/request.py", line 528, in on_failure
store_result=self.store_errors,
File "/usr/local/lib/python3.7/site-packages/celery/backends/base.py", line 160, in mark_as_failure
traceback=traceback, request=request)
File "/usr/local/lib/python3.7/site-packages/celery/backends/base.py", line 407, in store_result
request=request, **kwargs)
File "/usr/local/lib/python3.7/site-packages/celery/backends/base.py", line 758, in _store_result
self.set(self.get_key_for_task(task_id), self.encode(meta))
File "/usr/local/lib/python3.7/site-packages/celery/backends/elasticsearch.py", line 109, in set
datetime.utcnow().isoformat()[:-3]
File "/usr/local/lib/python3.7/site-packages/celery/backends/elasticsearch.py", line 126, in _index
**kwargs
File "/usr/local/lib/python3.7/site-packages/elasticsearch/client/utils.py", line 139, in _wrapped
return func(*args, params=params, headers=headers, **kwargs)
File "/usr/local/lib/python3.7/site-packages/elasticsearch/client/__init__.py", line 380, in index
body=body,
File "/usr/local/lib/python3.7/site-packages/elasticsearch/transport.py", line 352, in perform_request
timeout=timeout,
File "/usr/local/lib/python3.7/site-packages/elasticsearch/connection/http_urllib3.py", line 256, in perform_request
self._raise_error(response.status, raw_data)
File "/usr/local/lib/python3.7/site-packages/elasticsearch/connection/base.py", line 288, in _raise_error
status_code, error_message, additional_info
elasticsearch.exceptions.RequestError: RequestError(400, 'illegal_argument_exception', 'mapper [result.args] of different type, current_type [long], merged_type [text]')

Steps to Reproduce

Required Dependencies

Minimal Python Version: 3.7

Minimal Celery Version: 4.4.0

Minimal OS and/or Kernel Version: ubuntu

Python Packages

pip freeze Output:

uvicorn==0.11.3
fastapi>=0.61.0
fastapi_permissions==0.2.6
python_multipart==0.0.5
email_validator==1.0.5
requests==2.23.0
celery==4.4.7

Other Dependencies

N/A

Minimally Reproducible Test Case

Expected Behavior

Expected to Save the result backend as Object to elastic search, So I will be able to use it in the dashboard.

Actual Behavior

The result object is saved as text, and after trying all the solutions found after the search, I still have errors and not able to the result backend to elastic search.

Issue 2:
Checklist

I have checked the issues list

for similar or identical enhancement to an existing feature.

I have checked the pull requests list

for existing proposed enhancements.

I have checked the commit log

to find out if the if the same enhancement was already implemented in the

master branch.

I have included all related issues and possible duplicate issues in this issue

(If there are none, check this box anyway).

Related Issues and Possible Duplicates

Related Issues

#6773

Possible Duplicates

None

Brief Summary

Create a new major version (1.0.0) of the pytest-celery plugin.

Motivation

We want to refactor the Celery integration test suite so that it will be up and running again. We will make the integration tests more readable and flexible and ergonomic to us.

We also want to enable Celery users to test the integration easily and reliably, which is why the changes will eventually be available to all users via the pytest-celery plugin.

Scope

The scope of this project is to migrate the Celery integration suite since it is broken.

In this major version we will only address our use-cases. In the next one, we will begin planning and implementing features which could be important to our userbase as well.

High Level Requirements

Memory and CPU efficiency

The plugin must support GH Actions as this is the CI service we use.

It should support other commonly used CI services (such as Azure Pipelines, Circle CI or other similar CI services for open source).

We will run dependencies in Docker using the testcontainers library.

The containers our plugins runs will be reused from test to test for efficiency instead of tearing them down and re-running them in each test.

When we require multiple instances of the same docker image (e.g. using the same container for the message broker and result backend), a separate virtual host will be created for each service our test requires.

This provides logical separation which is good enough for testing purposes as usually security is not an issue in CI and therefore, we don't need to separate services by using stronger guarantees of isolation such as separate containers or even virtual machines.

Resource leak prevention

The only resource that is opened for each test by default is a queue. Queues will be given an expiration date so that unexpected shutdowns and failures do not cause memory leaks.

Containers are removed when the test suite is torn down. As a consequence, this will make all virtual hosts disappear.

Note that if a test is forced to quit (if you hit ctrl+c twice or if the pytest process is killed for whatever reason) , its virtual hosts will not immediately be removed. However, each time the test suite is completed, it tears down all virtual hosts, so this is manageable.

Run on the entire matrix of optional dependencies in parallel.

A test must be able to receive multiple options for each dependency.

The test will run on the Cartesian product of all optional dependencies.

This should also work in parallel with pytest-xdist.

Meaningful error messages

Error messages must be meaningful and grouped.

When the same test runs in parallel on several configurations, and some configurations fail for the same reason, we will batch those failures together.

All relevant logs will be attached to pytest's report when something goes wrong, including the container logs and the Celery worker(s) logs.

Timeline

ASAP: release version 0.1.0 to pytest-celery plugin.

Celery 5.x (next minor release): deprecation pending for version 0.1.0

Celery 6.0: pytest-celery version 1.0.0 released, version 0.1.0 deprecated

Celery 6.1: pytest-celery version 0.2.0 released which includes the code in celery/contrib/pytest.py

Celery 7.x: delete code for version 0.1.0

Design

Architectural Considerations

None

Proposed Behavior

Proposed UI/UX

The API design will be added soon.

Diagrams

N/A

Alternatives

None