Issue 1:
Checklist

I have verified that the issue exists against the master branch of Celery.

This has already been asked to the discussions forum first.

I have read the relevant section in the

contribution guide

on reporting bugs.

I have checked the issues list

for similar or identical bug reports.

I have checked the pull requests list

for existing proposed fixes.

I have checked the commit log

to find out if the bug was already fixed in the master branch.

I have included all related issues and possible duplicate issues

in this issue (If there are none, check this box anyway).

Mandatory Debugging Information

I have included the output of celery -A proj report in the issue.

(if you are not able to do this, then at least specify the Celery

version affected).

I have verified that the issue exists against the master branch of Celery.

I have included the contents of pip freeze in the issue.

I have included all the versions of all the external dependencies required

to reproduce this bug.

Optional Debugging Information

I have tried reproducing the issue on more than one Python version

and/or implementation.

I have tried reproducing the issue on more than one message broker and/or

result backend.

I have tried reproducing the issue on more than one version of the message

broker and/or result backend.

I have tried reproducing the issue on more than one operating system.

I have tried reproducing the issue on more than one workers pool.

I have tried reproducing the issue with autoscaling, retries,

ETA/Countdown & rate limits disabled.

I have tried reproducing the issue after downgrading

and/or upgrading Celery and its dependencies.

Related Issues and Possible Duplicates

Related Issues

Created in PR #6838

Possible Duplicates

None

Environment & Settings

Celery version: 5.2.0

celery report Output:

TypeError: a bytes-like object is required, not 'str'
File "aspiredu/project/celery.py", line 43, in run_command
sys.stdout.write(line) # Write all output from the command
File "celery/utils/log.py", line 227, in write
data = data.rstrip('\n')

Steps to Reproduce

Required Dependencies

Minimal Python Version: 3.9.6

Minimal Celery Version: 5.2

Minimal Kombu Version: N/A or Unknown

Minimal Broker Version: N/A or Unknown

Minimal Result Backend Version: N/A or Unknown

Minimal OS and/or Kernel Version: N/A or Unknown

Minimal Broker Client Version: N/A or Unknown

Minimal Result Backend Client Version: N/A or Unknown

Python Packages

pip freeze Output:

scout_apm==2.23.3

Other Dependencies

N/A

Minimally Reproducible Test Case

Add the following test to t.unit.app.test_log.test_default_logger

What's happening is that the logger is getting a data instance of type bytes, but can call rstrip with a str parameter which breaks.

@mock.restore_logging()
def test_logging_proxy_bytes(self):
logger = self.setup_logger(loglevel=logging.ERROR, logfile=None,
root=False)

with mock.wrap_logger(logger) as sio:
p = LoggingProxy(logger, loglevel=logging.ERROR)
p.close()
p.write(b'foo')
assert 'foo' not in str(sio.getvalue())
p.closed = False
p.write(b'\n')
assert str(sio.getvalue()) == ''
write_res = p.write(b'foo ')
assert str(sio.getvalue()) == 'foo \n'
assert write_res == 4
p.flush()
p.close()
assert not p.isatty()

with mock.stdouts() as (stdout, stderr):
with in_sighandler():
p.write(b'foo')
assert stderr.getvalue()

Expected Behavior

Calling ProxyLogger(logger).write(b'value') should not break when a bytes instance is passed in.

Actual Behavior

A TypeError: a bytes-like object is required, not 'str' is raised.

I will be creating a PR as a follow-up to this issue.

Issue 2:
Checklist

I have read the relevant section in the

contribution guide

on reporting bugs.

I have checked the issues list

for similar or identical bug reports.

I have checked the pull requests list

for existing proposed fixes.

I have checked the commit log

to find out if the bug was already fixed in the master branch.

I have included all related issues and possible duplicate issues

in this issue (If there are none, check this box anyway).

Mandatory Debugging Information

I have included the output of celery -A proj report in the issue.

(if you are not able to do this, then at least specify the Celery

version affected).

I have verified that the issue exists against the master branch of Celery.

I have included the contents of pip freeze in the issue.

I have included all the versions of all the external dependencies required

to reproduce this bug.

Optional Debugging Information

I have tried reproducing the issue on more than one Python version

and/or implementation.

I have tried reproducing the issue on more than one message broker and/or

result backend.

I have tried reproducing the issue on more than one version of the message

broker and/or result backend.

I have tried reproducing the issue on more than one operating system.

I have tried reproducing the issue on more than one workers pool.

I have tried reproducing the issue with autoscaling, retries,

ETA/Countdown & rate limits disabled.

I have tried reproducing the issue after downgrading

and/or upgrading Celery and its dependencies.

Related Issues and Possible Duplicates

Related Issues

None

Possible Duplicates

None

Environment & Settings

Celery version: 4.3.0

celery report Output:

Steps to Reproduce

Add elasticsearch as backend

Run any command

Required Dependencies

Minimal Python Version: '3.7.1 (default, Nov 6 2018, 18:46:03) \n[Clang 10.0.0 (clang-1000.11.45.5)]'

Minimal Celery Version: 4.3.0

Minimal Kombu Version: N/A or Unknown

Minimal Broker Version: N/A or Unknown

Minimal Result Backend Version: N/A or Unknown

Minimal OS and/or Kernel Version: System Version: macOS 10.14.5 (18F132) Darwin 18.6.0

Minimal Broker Client Version: N/A or Unknown

Minimal Result Backend Client Version: Elasticsearch 7.0.0

Python Packages

pip freeze Output:

Other Dependencies

N/A

Minimally Reproducible Test Case

Expected Behavior

Result is sent as a dictionary. At body dictionary creation, result added as inner dictionary, not as a string. Correct mapping is created. Results in Elasticsearch will be shown correctly and nested type could be used.

Correct mapping example:

{
"celery" : {
"mappings" : {
"backend" : {
"properties" : {
"@timestamp" : {
"type" : "date"
},
"result" : {
"properties" : {
"date_done" : {
"type" : "date"
},
"result" : {
"type" : "long"
},
"status" : {
"type" : "text",
"fields" : {
"keyword" : {
"type" : "keyword",
"ignore_above" : 256
}
}
},
"task_id" : {
"type" : "text",
"fields" : {
"keyword" : {
"type" : "keyword",
"ignore_above" : 256
}
}
}
}
}
}
}
}
}
}

Kibana:

Actual Behavior

Result value is serialized into a JSON string. At body dictionary creation, result added as a string. Elasticsearch can't parse it correctly.

Actual mapping:

{
"celery" : {
"mappings" : {
"backend" : {
"properties" : {
"@timestamp" : {
"type" : "date"
},
"result" : {
"type" : "text",
"fields" : {
"keyword" : {
"type" : "keyword",
"ignore_above" : 256
}
}
}
}
}
}
}
}

Kibana:

My stackoverflow question

Solution

In my fork I've imported json package and added json.loads(value):

# celery/backends/elasticsearch.py
import json

def set(self, key, value):
try:
self._index(
id=key,
body={
'result': json.loads(value),
'@timestamp': '{0}Z'.format(
datetime.utcnow().isoformat()[:-3]
),
},
)
except elasticsearch.exceptions.ConflictError:
# document already exists, update it
data = self.get(key)
data[key] = value
self._index(key, data, refresh=True)

Is there a way to disable JSON serialization of results for the Elasticsearch backend? If no, is this solution good enough for the pull-request?