Issue 1:
Checklist

I have read the relevant section in the

contribution guide

on reporting bugs.

I have checked the issues list

for similar or identical bug reports.

I have checked the pull requests list

for existing proposed fixes.

I have checked the commit log

to find out if the bug was already fixed in the master branch.

I have included all related issues and possible duplicate issues

in this issue (If there are none, check this box anyway).

Mandatory Debugging Information

I have included the output of celery -A proj report in the issue.

(if you are not able to do this, then at least specify the Celery

version affected).

I have verified that the issue exists against the master branch of Celery.

I have included the contents of pip freeze in the issue.

I have included all the versions of all the external dependencies required

to reproduce this bug.

Optional Debugging Information

I have tried reproducing the issue on more than one Python version

and/or implementation.

I have tried reproducing the issue on more than one message broker and/or

result backend.

I have tried reproducing the issue on more than one version of the message

broker and/or result backend.

I have tried reproducing the issue on more than one operating system.

I have tried reproducing the issue on more than one workers pool.

I have tried reproducing the issue with autoscaling, retries,

ETA/Countdown & rate limits disabled.

I have tried reproducing the issue after downgrading

and/or upgrading Celery and its dependencies.

Related Issues and Possible Duplicates

Related Issues

#4651

#3878

Possible Duplicates

None

Environment & Settings

Celery version:

software -> celery:3.1.20 (Cipater) kombu:3.0.37 py:2.7.15rc1
billiard:3.3.0.23 py-amqp:1.4.9
platform -> system:Linux arch:64bit imp:CPython
loader -> celery.loaders.app.AppLoader
settings -> transport:pyamqp results:disabled

Steps to Reproduce

Python Packages

pip freeze Output:

Other Dependencies

pyrfc==1.9.3

Minimally Reproducible Test Case

This is my tasks.py

import pyrfc
import os
import datetime
from celery import Celery
from celery.utils.log import get_task_logger
import logging
import requests
from celery.exceptions import SoftTimeLimitExceeded
from argparse import ArgumentParser

#logger = get_task_logger(__name__)
logging.basicConfig(level=logging.INFO)
logger = logging

app = Celery('tasks', broker='pyamqp://guest@localhost//')

config = # redacted

con = pyrfc.Connection(**config)

function_name = # redacted
function_args = # redacted

@app.task
def extract():
logger.info("PID %s", os.getpid())
logger.info("Connecting")
try:
con.call(function_name, **function_args)
except SoftTimeLimitExceeded:
logger.info("Exceeded RFC time")
logger.info("Disconnected")
return 1

@app.task()
def request():
logger.info("Connecting")
logger.info("PID %s", os.getpid())
try:
r = requests.get('http://127.0.0.1:8000')
logger.info("Disconnected")
return r.text
except SoftTimeLimitExceeded:
logger.info("Exceeded time")

def main():
parser = ArgumentParser()
parser.add_argument('func', choices=['request', 'extract'])
args = parser.parse_args()
if args.func == 'request':
request()
elif args.func == 'extract':
extract()
else:
raise

if __name__ == '__main__':

This is run with celery -A tasks worker --loglevel=info -c 1 --soft-time-limit 30

There are two tasks

Request: Performs a HTTP GET to a dummy server, which contains a delay of 60 seconds.

Extract: Performs a RFC call to a SAP Function, which also contains a delay of 60 seconds.

Expected Behavior

Both tasks should handle the soft time limit equally

Actual Behavior

The extract task doesn't handle the soft time limit exception until the server replies.

Running request.delay() we obtain as expected

[2019-05-29 16:45:17,745: INFO/MainProcess] Received task: tasks.request[47e3fad6-42f2-4896-956d-2e54f7080fb4]
[2019-05-29 16:45:17,746: INFO/Worker-1] Connecting
[2019-05-29 16:45:17,747: INFO/Worker-1] PID 2758
[2019-05-29 16:45:47,750: WARNING/MainProcess] Soft time limit (30.0s) exceeded for tasks.request[47e3fad6-42f2-4896-956d-2e54f7080fb4]
[2019-05-29 16:45:47,751: INFO/Worker-1] Exceeded time
[2019-05-29 16:45:47,752: INFO/MainProcess] Task tasks.request[47e3fad6-42f2-4896-956d-2e54f7080fb4] succeeded in 30.005404981s: None

Note the inmediate logging "Exceeded time" after the Celery warning

When running extract.delay() we obtain

[2019-05-29 16:46:03,361: INFO/MainProcess] Received task: tasks.extract[213f1eae-a872-4bc7-83af-69fd3350bce4]
[2019-05-29 16:46:03,362: INFO/Worker-1] PID 2758
[2019-05-29 16:46:03,362: INFO/Worker-1] Connecting
[2019-05-29 16:46:33,363: WARNING/MainProcess] Soft time limit (30.0s) exceeded for tasks.extract[213f1eae-a872-4bc7-83af-69fd3350bce4]
[2019-05-29 16:47:04,065: INFO/Worker-1] Exceeded RFC time
[2019-05-29 16:47:04,065: INFO/Worker-1] Disconnected
[2019-05-29 16:47:04,066: INFO/MainProcess] Task tasks.extract[213f1eae-a872-4bc7-83af-69fd3350bce4] succeeded in 60.704235596s: 1

Note the "Exceeded RFC time" is logged well after the Celery Warning. Remembering the 60 seconds server delay, we see the task handles the Exception only after the server reply.

I tested this both with my current celery version (3.1.20) and with the newer (4.3.0) with the same results.

Other Tests

Running the tasks outside celery and sending the SIGSUR1 signal via kill -s USR1 <pid>

$ python tasks.py request
INFO:root:Connecting
INFO:root:PID 3645
[1] 3645 user-defined signal 1 python tasks.py request

$ python tasks.py extract
INFO:root:PID 3699
INFO:root:Connecting
[1] 3699 user-defined signal 1 python tasks.py extract

Both functions do receive the signal (without delay) which makes me believe (perhaps incorrectly) the issue is not related to pyrfc compatibility with the signal.

I do understand this may not be a Celery fault, but any hint on where to look at / test is welcomed.

Regards

Issue 2:
Checklist

I have verified that the issue exists against the master branch of Celery.

This has already been asked to the discussions forum first.

I have read the relevant section in the

contribution guide

on reporting bugs.

I have checked the issues list

for similar or identical bug reports.

I have checked the pull requests list

for existing proposed fixes.

I have checked the commit log

to find out if the bug was already fixed in the master branch.

I have included all related issues and possible duplicate issues

in this issue (If there are none, check this box anyway).

Expected Behavior

I can have the following sqs broker configuration

broker_transport_options={
"predefined_queues": {
"MY_QUEUE": {
"url": "MYURL.fifo"
},
},

And no error will raise when calling my tasks.

Actual Behavior

File "/usr/local/lib/python3.9/site-packages/celery/app/task.py", line 425, in delay
return self.apply_async(args, kwargs)
File "/usr/local/lib/python3.9/site-packages/celery/app/task.py", line 575, in apply_async
return app.send_task(
File "/usr/local/lib/python3.9/site-packages/celery/app/base.py", line 788, in send_task
amqp.send_task_message(P, name, message, **options)
File "/usr/local/lib/python3.9/site-packages/celery/app/amqp.py", line 510, in send_task_message
ret = producer.publish(
File "/usr/local/lib/python3.9/site-packages/kombu/messaging.py", line 177, in publish
return _publish(
File "/usr/local/lib/python3.9/site-packages/kombu/connection.py", line 524, in _ensured
return fun(*args, **kwargs)
File "/usr/local/lib/python3.9/site-packages/kombu/messaging.py", line 199, in _publish
return channel.basic_publish(
File "/usr/local/lib/python3.9/site-packages/kombu/transport/virtual/base.py", line 604, in basic_publish
return self._put(routing_key, message, **kwargs)
File "/usr/local/lib/python3.9/site-packages/kombu/transport/SQS.py", line 392, in _put
c.send_message(**kwargs)
File "/usr/local/lib/python3.9/site-packages/botocore/client.py", line 391, in _api_call
return self._make_api_call(operation_name, kwargs)
File "/usr/local/lib/python3.9/site-packages/botocore/client.py", line 719, in _make_api_call
raise error_class(parsed_response, operation_name)
botocore.exceptions.ClientError: An error occurred (MissingParameter) when calling the SendMessage operation: The request must contain the parameter MessageGroupId.

Explanation of the bug:

In the konbu documentation there is the following lines that manage this exception.

if queue.endswith('.fifo'):
if 'MessageGroupId' in message['properties']:
kwargs['MessageGroupId'] = \
message['properties']['MessageGroupId']
else:
kwargs['MessageGroupId'] = 'default'

Since Konbu checks that routing_key ends with .fifo, and not what it has been configured inside celery as MY_URL, means that the property MessageGroupId is never set, and the botocore error is raised.

I can see why this is not a celery bug, or not a bug at all, but I would say this case should be at least documented: having to force celery queues name to follow the SQS paradigm, in my opinion, can mislead people.