Issue 1:
celery issue

Checklist

I have included the output of celery -A proj report in the issue.

(if you are not able to do this, then at least specify the Celery

version affected).

I have included all related issues and possible duplicate issues in this issue.

I have included the contents of pip freeze in the issue.

I have verified that the issue exists against the master branch of Celery.

I have tried reproducing the issue on more than one message broker and/or result backend.

I have tried reproducing the issue on more than one workers pool.

I have tried reproducing the issue with retries, ETA/Countdown & rate limits disabled.

Related Issues and Possible Duplicates

Related Issues

#4867 - this was resolved using master, my issue was not

#5247 - the scenario under which it is reproduced is different and they are using redis for transport and results

Possible Duplicates

None

Environment & Settings

Celery version: 4.3.0rc1 (rhubarb) (e257646)

celery report Output:

software -> celery:4.3.0rc1 (rhubarb) kombu:4.3.0 py:3.7.2
billiard:3.6.0.0 py-amqp:2.4.1
platform -> system:Linux arch:64bit
kernel version:4.19.11-041911-generic imp:CPython
loader -> celery.loaders.app.AppLoader
settings -> transport:pyamqp results:rpc://localhost/

broker_url: 'amqp://guest:********@localhost:5672//'
result_backend: 'rpc://localhost/'

Steps to Reproduce

Required Dependencies

Minimal Python Version: 3.7.2

Minimal Broker Version: RabbitMQ 3.7.12

Minimal Result Backend Version: RabbitMQ 3.7.12

Minimal OS and/or Kernel Version: Ubuntu 18.04, 4.19.11

Minimal Broker Client Version: amqp 2.4.1

Minimal Result Backend Client Version: amqp 2.4.1

Python Packages

pip freeze Output:

amqp==2.4.1
billiard==3.6.0.0
-e git+git@github.com:celery/celery.git@e257646136e6fae73186d7385317f4e20cd36130#egg=celery
kombu==4.3.0
pytz==2018.9
redis==3.2.0
vine==1.2.0

Other Dependencies

N/A

Minimally Reproducible Test Case

Execute

docker run -d --net=host redis:5.0.3
docker run -d --net=host rabbitmq:3.7.12

Save the following as repro.py.

import time

from datetime import datetime
from pathlib import Path

from celery import Celery

# not OK
app = Celery(Path(__file__).stem, backend='rpc://localhost', broker='pyamqp://')
# OK
#app = Celery(Path(__file__).stem, backend='rpc://localhost', broker='redis://')
# OK
#app = Celery(Path(__file__).stem, backend='redis://', broker='pyamqp://')

@app.task
def ping(v):
time.sleep(10)
return v

if __name__ == '__main__':
start = time.time()
tasks = [ping.delay(i) for i in range(100)]
for task in tasks:
result = task.get()
now = time.time()
timestamp = datetime.now().strftime('%Y-%m-%dT%H:%M:%S')
print(f'{timestamp}: {result}: {now - start}')

Execute

celery -A repro worker --detach --hostname repro --logfile celery.log --loglevel DEBUG
python repro.py

Expected Behavior

The client should send all tasks to be executed, then wait on each task in turn. Each task should succeed and output should be printed in the console.

Actual Behavior

180 seconds (3 minutes) after the start of the client, a connection reset is received.

Traceback

...
2019-02-23T12:49:59: 66: 170.30147862434387
2019-02-23T12:49:59: 67: 170.30566883087158
Traceback (most recent call last):
File "repro.py", line 27, in <module>
result = task.get()
File "/tmp/celery-repro/celery/celery/result.py", line 226, in get
on_message=on_message,
File "/tmp/celery-repro/celery/celery/backends/asynchronous.py", line 188, in wait_for_pending
for _ in self._wait_for_pending(result, **kwargs):
File "/tmp/celery-repro/celery/celery/backends/asynchronous.py", line 255, in _wait_for_pending
on_interval=on_interval):
File "/tmp/celery-repro/celery/celery/backends/asynchronous.py", line 56, in drain_events_until
yield self.wait_for(p, wait, timeout=1)
File "/tmp/celery-repro/celery/celery/backends/asynchronous.py", line 65, in wait_for
wait(timeout=timeout)
File "/tmp/celery-repro/celery/celery/backends/rpc.py", line 63, in drain_events
return self._connection.drain_events(timeout=timeout)
File "/tmp/celery-repro/.venv/lib/python3.7/site-packages/kombu/connection.py", line 315, in drain_events
return self.transport.drain_events(self.connection, **kwargs)
File "/tmp/celery-repro/.venv/lib/python3.7/site-packages/kombu/transport/pyamqp.py", line 103, in drain_events
return connection.drain_events(**kwargs)
File "/tmp/celery-repro/.venv/lib/python3.7/site-packages/amqp/connection.py", line 500, in drain_events
while not self.blocking_read(timeout):
File "/tmp/celery-repro/.venv/lib/python3.7/site-packages/amqp/connection.py", line 505, in blocking_read
frame = self.transport.read_frame()
File "/tmp/celery-repro/.venv/lib/python3.7/site-packages/amqp/transport.py", line 256, in read_frame
frame_header = read(7, True)
File "/tmp/celery-repro/.venv/lib/python3.7/site-packages/amqp/transport.py", line 440, in _read
s = recv(n - len(rbuf))
ConnectionResetError: [Errno 104] Connection reset by peer

Daemon logs

[2019-02-23 12:47:04,760: DEBUG/MainProcess] | Worker: Preparing bootsteps.
[2019-02-23 12:47:04,761: DEBUG/MainProcess] | Worker: Building graph...
[2019-02-23 12:47:04,762: DEBUG/MainProcess] | Worker: New boot order: {Beat, StateDB, Timer, Hub, Pool, Autoscaler, Consumer}
[2019-02-23 12:47:04,765: DEBUG/MainProcess] | Consumer: Preparing bootsteps.
[2019-02-23 12:47:04,765: DEBUG/MainProcess] | Consumer: Building graph...
[2019-02-23 12:47:04,768: DEBUG/MainProcess] | Consumer: New boot order: {Connection, Events, Heart, Mingle, Gossip, Agent, Tasks, Control, event loop}
[2019-02-23 12:47:04,773: DEBUG/MainProcess] | Worker: Starting Hub
[2019-02-23 12:47:04,773: DEBUG/MainProcess] ^-- substep ok
[2019-02-23 12:47:04,773: DEBUG/MainProcess] | Worker: Starting Pool
[2019-02-23 12:47:04,811: DEBUG/MainProcess] ^-- substep ok
[2019-02-23 12:47:04,811: DEBUG/MainProcess] | Worker: Starting Consumer
[2019-02-23 12:47:04,811: DEBUG/MainProcess] | Consumer: Starting Connection
[2019-02-23 12:47:04,819: DEBUG/MainProcess] Start from server, version: 0.9, properties: {'capabilities': {'publisher_confirms': True, 'exchange_exchange_bindings': True, 'basic.nack': True, 'consumer_cancel_notify': True, 'connection.blocked': True, 'consumer_priorities': True, 'authentication_failure_close': True, 'per_consumer_qos': True, 'direct_reply_to': True}, 'cluster_name': 'rabbit@repro', 'copyright': 'Copyright (C) 2007-2019 Pivotal Software, Inc.', 'information': 'Licensed under the MPL. See http://www.rabbitmq.com/', 'platform': 'Erlang/OTP 21.2.6', 'product': 'RabbitMQ', 'version': '3.7.12'}, mechanisms: [b'PLAIN', b'AMQPLAIN'], locales: ['en_US']
[2019-02-23 12:47:04,820: INFO/MainProcess] Connected to amqp://guest:**@127.0.0.1:5672//
[2019-02-23 12:47:04,820: DEBUG/MainProcess] ^-- substep ok
[2019-02-23 12:47:04,821: DEBUG/MainProcess] | Consumer: Starting Events
[2019-02-23 12:47:04,826: DEBUG/MainProcess] Start from server, version: 0.9, properties: {'capabilities': {'publisher_confirms': True, 'exchange_exchange_bindings': True, 'basic.nack': True, 'consumer_cancel_notify': True, 'connection.blocked': True, 'consumer_priorities': True, 'authentication_failure_close': True, 'per_consumer_qos': True, 'direct_reply_to': True}, 'cluster_name': 'rabbit@repro', 'copyright': 'Copyright (C) 2007-2019 Pivotal Software, Inc.', 'information': 'Licensed under the MPL. See http://www.rabbitmq.com/', 'platform': 'Erlang/OTP 21.2.6', 'product': 'RabbitMQ', 'version': '3.7.12'}, mechanisms: [b'PLAIN', b'AMQPLAIN'], locales: ['en_US']
[2019-02-23 12:47:04,828: DEBUG/MainProcess] ^-- substep ok
[2019-02-23 12:47:04,828: DEBUG/MainProcess] | Consumer: Starting Heart
[2019-02-23 12:47:04,828: DEBUG/MainProcess] using channel_id: 1
[2019-02-23 12:47:04,829: DEBUG/MainProcess] Channel open
[2019-02-23 12:47:04,830: DEBUG/MainProcess] ^-- substep ok
[2019-02-23 12:47:04,830: DEBUG/MainProcess] | Consumer: Starting Mingle
[2019-02-23 12:47:04,830: INFO/MainProcess] mingle: searching for neighbors
[2019-02-23 12:47:04,831: DEBUG/MainProcess] using channel_id: 1
[2019-02-23 12:47:04,831: DEBUG/MainProcess] Channel open
[2019-02-23 12:47:04,839: DEBUG/MainProcess] Start from server, version: 0.9, properties: {'capabilities': {'publisher_confirms': True, 'exchange_exchange_bindings': True, 'basic.nack': True, 'consumer_cancel_notify': True, 'connection.blocked': True, 'consumer_priorities': True, 'authentication_failure_close': True, 'per_consumer_qos': True, 'direct_reply_to': True}, 'cluster_name': 'rabbit@repro', 'copyright': 'Copyright (C) 2007-2019 Pivotal Software, Inc.', 'information': 'Licensed under the MPL. See http://www.rabbitmq.com/', 'platform': 'Erlang/OTP 21.2.6', 'product': 'RabbitMQ', 'version': '3.7.12'}, mechanisms: [b'PLAIN', b'AMQPLAIN'], locales: ['en_US']
[2019-02-23 12:47:04,840: DEBUG/MainProcess] using channel_id: 1
[2019-02-23 12:47:04,841: DEBUG/MainProcess] Channel open
[2019-02-23 12:47:05,850: INFO/MainProcess] mingle: all alone
[2019-02-23 12:47:05,851: DEBUG/MainProcess] ^-- substep ok
[2019-02-23 12:47:05,851: DEBUG/MainProcess] | Consumer: Starting Gossip
[2019-02-23 12:47:05,851: DEBUG/MainProcess] using channel_id: 2
[2019-02-23 12:47:05,854: DEBUG/MainProcess] Channel open
[2019-02-23 12:47:05,857: DEBUG/MainProcess] ^-- substep ok
[2019-02-23 12:47:05,857: DEBUG/MainProcess] | Consumer: Starting Tasks
[2019-02-23 12:47:05,860: DEBUG/MainProcess] ^-- substep ok
[2019-02-23 12:47:05,860: DEBUG/MainProcess] | Consumer: Starting Control
[2019-02-23 12:47:05,860: DEBUG/MainProcess] using channel_id: 3
[2019-02-23 12:47:05,861: DEBUG/MainProcess] Channel open
[2019-02-23 12:47:05,863: DEBUG/MainProcess] ^-- substep ok
[2019-02-23 12:47:05,863: DEBUG/MainProcess] | Consumer: Starting event loop
[2019-02-23 12:47:05,863: DEBUG/MainProcess] | Worker: Hub.register Pool...
[2019-02-23 12:47:05,864: INFO/MainProcess] celery@repro ready.
[2019-02-23 12:47:05,864: DEBUG/MainProcess] basic.qos: prefetch_count->16
[2019-02-23 12:47:09,698: INFO/MainProcess] Received task: repro.ping[a233da5a-a69f-4d5e-b6e7-c7c29483aec4]
[2019-02-23 12:47:09,702: DEBUG/MainProcess] TaskPool: Apply <function _fast_trace_task at 0x7f92462897b8> (args:('repro.ping', 'a233da5a-a69f-4d5e-b6e7-c7c29483aec4', {'lang': 'py', 'task': 'repro.ping', 'id': 'a233da5a-a69f-4d5e-b6e7-c7c29483aec4', 'shadow': None, 'eta': None, 'expires': None, 'group': None, 'retries': 0, 'timelimit': [None, None], 'root_id': 'a233da5a-a69f-4d5e-b6e7-c7c29483aec4', 'parent_id': None, 'argsrepr': '(0,)', 'kwargsrepr': '{}', 'origin': 'gen24599@repro', 'reply_to': '13b4de4e-3451-330c-837d-91fcc7545757', 'correlation_id': 'a233da5a-a69f-4d5e-b6e7-c7c29483aec4', 'delivery_info': {'exchange': '', 'routing_key': 'celery', 'priority': 0, 'redelivered': False}}, '[[0], {}, {"callbacks": null, "errbacks": null, "chain": null, "chord": null}]', 'application/json', 'utf-8') kwargs:{})
[2019-02-23 12:47:09,703: INFO/MainProcess] Received task: repro.ping[a1569f36-1f95-4d2d-a8d0-94dfa9abe132]
[2019-02-23 12:47:09,703: DEBUG/MainProcess] TaskPool: Apply <function _fast_trace_task at 0x7f92462897b8> (args:('repro.ping', 'a1569f36-1f95-4d2d-a8d0-94dfa9abe132', {'lang': 'py', 'task': 'repro.ping', 'id': 'a1569f36-1f95-4d2d-a8d0-94dfa9abe132', 'shadow': None, 'eta': None, 'expires': None, 'group': None, 'retries': 0, 'timelimit': [None, None], 'root_id': 'a1569f36-1f95-4d2d-a8d0-94dfa9abe132', 'parent_id': None, 'argsrepr': '(1,)', 'kwargsrepr': '{}', 'origin': 'gen24599@repro', 'reply_to': '13b4de4e-3451-330c-837d-91fcc7545757', 'correlation_id': 'a1569f36-1f95-4d2d-a8d0-94dfa9abe132', 'delivery_info': {'exchange': '', 'routing_key': 'celery', 'priority': 0, 'redelivered': False}}, '[[1], {}, {"callbacks": null, "errbacks": null, "chain": null, "chord": null}]', 'application/json', 'utf-8') kwargs:{})
[2019-02-23 12:47:09,703: INFO/MainProcess] Received task: repro.ping[58235f2a-1eb0-42e3-857f-3453aba57d32]
[2019-02-23 12:47:09,703: DEBUG/MainProcess] TaskPool: Apply <function _fast_trace_task at 0x7f92462897b8> (args:('repro.ping', '58235f2a-1eb0-42e3-857f-3453aba57d32', {'lang': 'py', 'task': 'repro.ping', 'id': '58235f2a-1eb0-42e3-857f-3453aba57d32', 'shadow': None, 'eta': None, 'expires': None, 'group': None, 'retries': 0, 'timelimit': [None, None], 'root_id': '58235f2a-1eb0-42e3-857f-3453aba57d32', 'parent_id': None, 'argsrepr': '(2,)', 'kwargsrepr': '{}', 'origin': 'gen24599@repro', 'reply_to': '13b4de4e-3451-330c-837d-91fcc7545757', 'correlation_id': '58235f2a-1eb0-42e3-857f-3453aba57d32', 'delivery_info': {'exchange': '', 'routing_key': 'celery', 'priority': 0, 'redelivered': False}}, '[[2], {}, {"callbacks": null, "errbacks": null, "chain": null, "chord": null}]', 'application/json', 'utf-8') kwargs:{})
[2019-02-23 12:47:09,704: INFO/MainProcess] Received task: repro.ping[c4a63e95-ca68-4d22-9996-96490523b37d]
[2019-02-23 12:47:09,705: DEBUG/MainProcess] TaskPool: Apply <function _fast_trace_task at 0x7f92462897b8> (args:('repro.ping', 'c4a63e95-ca68-4d22-9996-96490523b37d', {'lang': 'py', 'task': 'repro.ping', 'id': 'c4a63e95-ca68-4d22-9996-96490523b37d', 'shadow': None, 'eta': None, 'expires': None, 'group': None, 'retries': 0, 'timelimit': [None, None], 'root_id': 'c4a63e95-ca68-4d22-9996-96490523b37d', 'parent_id': None, 'argsrepr': '(3,)', 'kwargsrepr': '{}', 'origin': 'gen24599@repro', 'reply_to': '13b4de4e-3451-330c-837d-91fcc7545757', 'correlation_id': 'c4a63e95-ca68-4d22-9996-96490523b37d', 'delivery_info': {'exchange': '', 'routing_key': 'celery', 'priority': 0, 'redelivered': False}}, '[[3], {}, {"callbacks": null, "errbacks": null, "chain": null, "chord": null}]', 'application/json', 'utf-8') kwargs:{})
[2019-02-23 12:47:09,705: INFO/MainProcess] Received task: repro.ping[eab3cfe4-2b5e-4de7-bb5f-3d569871c85c]
[2019-02-23 12:47:09,706: INFO/MainProcess] Received task: repro.ping[97a705fc-2208-4551-90b5-3fc1d54d79a0]
[2019-02-23 12:47:09,706: DEBUG/MainProcess] Task accepted: repro.ping[58235f2a-1eb0-42e3-857f-3453aba57d32] pid:24588
[2019-02-23 12:47:09,706: DEBUG/MainProcess] Task accepted: repro.ping[a233da5a-a69f-4d5e-b6e7-c7c29483aec4] pid:24587
[2019-02-23 12:47:09,706: DEBUG/MainProcess] Task accepted: repro.ping[a1569f36-1f95-4d2d-a8d0-94dfa9abe132] pid:24589
[2019-02-23 12:47:09,707: INFO/MainProcess] Received task: repro.ping[a1eb440e-708c-4df2-a843-d678fca5876b]
[2019-02-23 12:47:09,707: INFO/MainProcess] Received task: repro.ping[275b1a5a-6472-45f0-8710-4593e5f80b5a]
[2019-02-23 12:47:09,707: INFO/MainProcess] Received task: repro.ping[6ee7bc0b-1c84-4973-8954-c6b2f17bb9cc]
[2019-02-23 12:47:09,708: INFO/MainProcess] Received task: repro.ping[31c2c135-76a5-4682-b400-3508ab9746ee]
[2019-02-23 12:47:09,708: INFO/MainProcess] Received task: repro.ping[c7a5aba5-29f8-4046-9662-271035823431]
[2019-02-23 12:47:09,708: INFO/MainProcess] Received task: repro.ping[e063884c-b7ba-48f6-8623-38a6d2ab174c]
[2019-02-23 12:47:09,709: DEBUG/MainProcess] Task accepted: repro.ping[c4a63e95-ca68-4d22-9996-96490523b37d] pid:24590
[2019-02-23 12:47:09,709: INFO/MainProcess] Received task: repro.ping[f85a56c1-d26d-4fa2-8f81-1cfe60fff298]
[2019-02-23 12:47:09,710: INFO/MainProcess] Received task: repro.ping[efa85c32-ef70-453b-97e8-57497f1245c2]
[2019-02-23 12:47:09,711: INFO/MainProcess] Received task: repro.ping[b0957ce3-4915-4f86-a728-4d47759bee51]
[2019-02-23 12:47:09,712: INFO/MainProcess] Received task: repro.ping[910ff039-f563-4f00-a631-fe131c9cedb4]
[2019-02-23 12:47:09,714: INFO/MainProcess] Received task: repro.ping[3c059acc-519b-49b2-ba3d-e5d93bbcc2da]
[2019-02-23 12:47:09,715: INFO/MainProcess] Received task: repro.ping[d75703fc-5e87-4152-97c7-72607249ecd8]
[2019-02-23 12:47:09,715: INFO/MainProcess] Received task: repro.ping[b78ee077-4e13-4f38-b7f4-5baef4e88a1b]
[2019-02-23 12:47:09,716: INFO/MainProcess] Received task: repro.ping[82ba77ec-6874-4414-a814-fe432e871005]
[2019-02-23 12:47:19,758: DEBUG/ForkPoolWorker-4] Start from server, version: 0.9, properties: {'capabilities': {'publisher_confirms': True, 'exchange_exchange_bindings': True, 'basic.nack': True, 'consumer_cancel_notify': True, 'connection.blocked': True, 'consumer_priorities': True, 'authentication_failure_close': True, 'per_consumer_qos': True, 'direct_reply_to': True}, 'cluster_name': 'rabbit@repro', 'copyright': 'Copyright (C) 2007-2019 Pivotal Software, Inc.', 'information': 'Licensed under the MPL. See http://www.rabbitmq.com/', 'platform': 'Erlang/OTP 21.2.6', 'product': 'RabbitMQ', 'version': '3.7.12'}, mechanisms: [b'PLAIN', b'AMQPLAIN'], locales: ['en_US']
[2019-02-23 12:47:19,760: DEBUG/ForkPoolWorker-3] Start from server, version: 0.9, properties: {'capabilities': {'publisher_confirms': True, 'exchange_exchange_bindings': True, 'basic.nack': True, 'consumer_cancel_notify': True, 'connection.blocked': True, 'consumer_priorities': True, 'authentication_failure_close': True, 'per_consumer_qos': True, 'direct_reply_to': True}, 'cluster_name': 'rabbit@repro', 'copyright': 'Copyright (C) 2007-2019 Pivotal Software, Inc.', 'information': 'Licensed under the MPL. See http://www.rabbitmq.com/', 'platform': 'Erlang/OTP 21.2.6', 'product': 'RabbitMQ', 'version': '3.7.12'}, mechanisms: [b'PLAIN', b'AMQPLAIN'], locales: ['en_US']
[2019-02-23 12:47:19,762: DEBUG/ForkPoolWorker-4] using channel_id: 1
[2019-02-23 12:47:19,763: DEBUG/ForkPoolWorker-4] Channel open
[2019-02-23 12:47:19,762: DEBUG/ForkPoolWorker-2] Start from server, version: 0.9, properties: {'capabilities': {'publisher_confirms': True, 'exchange_exchange_bindings': True, 'basic.nack': True, 'consumer_cancel_notify': True, 'connection.blocked': True, 'consumer_priorities': True, 'authentication_failure_close': True, 'per_consumer_qos': True, 'direct_reply_to': True}, 'cluster_name': 'rabbit@repro', 'copyright': 'Copyright (C) 2007-2019 Pivotal Software, Inc.', 'information': 'Licensed under the MPL. See http://www.rabbitmq.com/', 'platform': 'Erlang/OTP 21.2.6', 'product': 'RabbitMQ', 'version': '3.7.12'}, mechanisms: [b'PLAIN', b'AMQPLAIN'], locales: ['en_US']
[2019-02-23 12:47:19,764: DEBUG/ForkPoolWorker-3] using channel_id: 1
[2019-02-23 12:47:19,765: INFO/ForkPoolWorker-4] Task repro.ping[c4a63e95-ca68-4d22-9996-96490523b37d] succeeded in 10.05700475804042s: 3
[2019-02-23 12:47:19,765: DEBUG/MainProcess] TaskPool: Apply <function _fast_trace_task at 0x7f92462897b8> (args:('repro.ping', 'eab3cfe4-2b5e-4de7-bb5f-3d569871c85c', {'lang': 'py', 'task': 'repro.ping', 'id': 'eab3cfe4-2b5e-4de7-bb5f-3d569871c85c', 'shadow': None, 'eta': None, 'expires': None, 'group': None, 'retries': 0, 'timelimit': [None, None], 'root_id': 'eab3cfe4-2b5e-4de7-bb5f-3d569871c85c', 'parent_id': None, 'argsrepr': '(4,)', 'kwargsrepr': '{}', 'origin': 'gen24599@repro', 'reply_to': '13b4de4e-3451-330c-837d-91fcc7545757', 'correlation_id': 'eab3cfe4-2b5e-4de7-bb5f-3d569871c85c', 'delivery_info': {'exchange': '', 'routing_key': 'celery', 'priority': 0, 'redelivered': False}}, '[[4], {}, {"callbacks": null, "errbacks": null, "chain": null, "chord": null}]', 'application/json', 'utf-8') kwargs:{})
[2019-02-23 12:47:19,766: DEBUG/ForkPoolWorker-3] Channel open
[2019-02-23 12:47:19,767: INFO/ForkPoolWorker-3] Task repro.ping[a1569f36-1f95-4d2d-a8d0-94dfa9abe132] succeeded in 10.061639431049116s: 1
[2019-02-23 12:47:19,767: DEBUG/MainProcess] Task accepted: repro.ping[eab3cfe4-2b5e-4de7-bb5f-3d569871c85c] pid:24590
[2019-02-23 12:47:19,768: DEBUG/MainProcess] TaskPool: Apply <function _fast_trace_task at 0x7f92462897b8> (args:('repro.ping', '97a705fc-2208-4551-90b5-3fc1d54d79a0', {'lang': 'py', 'task': 'repro.ping', 'id': '97a705fc-2208-4551-90b5-3fc1d54d79a0', 'shadow': None, 'eta': None, 'expires': None, 'group': None, 'retries': 0, 'timelimit': [None, None], 'root_id': '97a705fc-2208-4551-90b5-3fc1d54d79a0', 'parent_id': None, 'argsrepr': '(5,)', 'kwargsrepr': '{}', 'origin': 'gen24599@repro', 'reply_to': '13b4de4e-3451-330c-837d-91fcc7545757', 'correlation_id': '97a705fc-2208-4551-90b5-3fc1d54d79a0', 'delivery_info': {'exchange': '', 'routing_key': 'celery', 'priority': 0, 'redelivered': False}}, '[[5], {}, {"callbacks": null, "errbacks": null, "chain": null, "chord": null}]', 'application/json', 'utf-8') kwargs:{})
[2019-02-23 12:47:19,769: DEBUG/ForkPoolWorker-2] using channel_id: 1
[2019-02-23 12:47:19,769: INFO/MainProcess] Received task: repro.ping[91dbdb86-cbe6-4a57-b669-400211b3d477]
[2019-02-23 12:47:19,770: DEBUG/MainProcess] Task accepted: repro.ping[97a705fc-2208-4551-90b5-3fc1d54d79a0] pid:24589
[2019-02-23 12:47:19,771: DEBUG/ForkPoolWorker-1] Start from server, version: 0.9, properties: {'capabilities': {'publisher_confirms': True, 'exchange_exchange_bindings': True, 'basic.nack': True, 'consumer_cancel_notify': True, 'connection.blocked': True, 'consumer_priorities': True, 'authentication_failure_close': True, 'per_consumer_qos': True, 'direct_reply_to': True}, 'cluster_name': 'rabbit@repro', 'copyright': 'Copyright (C) 2007-2019 Pivotal Software, Inc.', 'information': 'Licensed under the MPL. See http://www.rabbitmq.com/', 'platform': 'Erlang/OTP 21.2.6', 'product': 'RabbitMQ', 'version': '3.7.12'}, mechanisms: [b'PLAIN', b'AMQPLAIN'], locales: ['en_US']
[2019-02-23 12:47:19,771: DEBUG/ForkPoolWorker-2] Channel open
[2019-02-23 12:47:19,771: INFO/MainProcess] Received task: repro.ping[d3767e82-7208-4b02-a8df-467e260f5afd]
[2019-02-23 12:47:19,772: INFO/ForkPoolWorker-2] Task repro.ping[58235f2a-1eb0-42e3-857f-3453aba57d32] succeeded in 10.067062080022879s: 2
[2019-02-23 12:47:19,772: DEBUG/MainProcess] TaskPool: Apply <function _fast_trace_task at 0x7f92462897b8> (args:('repro.ping', 'a1eb440e-708c-4df2-a843-d678fca5876b', {'lang': 'py', 'task': 'repro.ping', 'id': 'a1eb440e-708c-4df2-a843-d678fca5876b', 'shadow': None, 'eta': None, 'expires': None, 'group': None, 'retries': 0, 'timelimit': [None, None], 'root_id': 'a1eb440e-708c-4df2-a843-d678fca5876b', 'parent_id': None, 'argsrepr': '(6,)', 'kwargsrepr': '{}', 'origin': 'gen24599@repro', 'reply_to': '13b4de4e-3451-330c-837d-91fcc7545757', 'correlation_id': 'a1eb440e-708c-4df2-a843-d678fca5876b', 'delivery_info': {'exchange': '', 'routing_key': 'celery', 'priority': 0, 'redelivered': False}}, '[[6], {}, {"callbacks": null, "errbacks": null, "chain": null, "chord": null}]', 'application/json', 'utf-8') kwargs:{})
[2019-02-23 12:47:19,772: DEBUG/ForkPoolWorker-1] using channel_id: 1
[2019-02-23 12:47:19,773: DEBUG/MainProcess] Task accepted: repro.ping[a1eb440e-708c-4df2-a843-d678fca5876b] pid:24588
[2019-02-23 12:47:19,775: DEBUG/ForkPoolWorker-1] Channel open
[2019-02-23 12:47:19,775: INFO/MainProcess] Received task: repro.ping[0208e029-4283-4a1d-b978-a2e12efc892a]
[2019-02-23 12:47:19,775: INFO/ForkPoolWorker-1] Task repro.ping[a233da5a-a69f-4d5e-b6e7-c7c29483aec4] succeeded in 10.070296013960615s: 0
[2019-02-23 12:47:19,776: DEBUG/MainProcess] TaskPool: Apply <function _fast_trace_task at 0x7f92462897b8> (args:('repro.ping', '275b1a5a-6472-45f0-8710-4593e5f80b5a', {'lang': 'py', 'task': 'repro.ping', 'id': '275b1a5a-6472-45f0-8710-4593e5f80b5a', 'shadow': None, 'eta': None, 'expires': None, 'group': None, 'retries': 0, 'timelimit': [None, None], 'root_id': '275b1a5a-6472-45f0-8710-4593e5f80b5a', 'parent_id': None, 'argsrepr': '(7,)', 'kwargsrepr': '{}', 'origin': 'gen24599@repro', 'reply_to': '13b4de4e-3451-330c-837d-91fcc7545757', 'correlation_id': '275b1a5a-6472-45f0-8710-4593e5f80b5a', 'delivery_info': {'exchange': '', 'routing_key': 'celery', 'priority': 0, 'redelivered': False}}, '[[7], {}, {"callbacks": null, "errbacks": null, "chain": null, "chord": null}]', 'application/json', 'utf-8') kwargs:{})
[2019-02-23 12:47:19,776: DEBUG/MainProcess] Task accepted: repro.ping[275b1a5a-6472-45f0-8710-4593e5f80b5a] pid:24587
[2019-02-23 12:47:19,777: INFO/MainProcess] Received task: repro.ping[dd272fa0-226f-4b76-97a7-6b66d2a6f8ec]
[2019-02-23 12:47:25,864: DEBUG/MainProcess] heartbeat_tick : for connection c96dac8c7b174cb3af302fd13ce1db55
[2019-02-23 12:47:25,864: DEBUG/MainProcess] heartbeat_tick : Prev sent/recv: None/None, now - 36/130, monotonic - 827970.404138137, last_heartbeat_sent - 827970.404134595, heartbeat int. - 60 for connection c96dac8c7b174cb3af302fd13ce1db55
[2019-02-23 12:47:29,771: INFO/ForkPoolWorker-4] Task repro.ping[eab3cfe4-2b5e-4de7-bb5f-3d569871c85c] succeeded in 10.002901564934291s: 4
[2019-02-23 12:47:29,771: DEBUG/MainProcess] TaskPool: Apply <function _fast_trace_task at 0x7f92462897b8> (args:('repro.ping', '6ee7bc0b-1c84-4973-8954-c6b2f17bb9cc', {'lang': 'py', 'task': 'repro.ping', 'id': '6ee7bc0b-1c84-4973-8954-c6b2f17bb9cc', 'shadow': None, 'eta': None, 'expires': None, 'group': None, 'retries': 0, 'timelimit': [None, None], 'root_id': '6ee7bc0b-1c84-4973-8954-c6b2f17bb9cc', 'parent_id': None, 'argsrepr': '(8,)', 'kwargsrepr': '{}', 'origin': 'gen24599@repro', 'reply_to': '13b4de4e-3451-330c-837d-91fcc7545757', 'correlation_id': '6ee7bc0b-1c84-4973-8954-c6b2f17bb9cc', 'delivery_info': {'exchange': '', 'routing_key': 'celery', 'priority': 0, 'redelivered': False}}, '[[8], {}, {"callbacks": null, "errbacks": null, "chain": null, "chord": null}]', 'application/json', 'utf-8') kwargs:{})
[2019-02-23 12:47:29,773: DEBUG/MainProcess] Task accepted: repro.ping[6ee7bc0b-1c84-4973-8954-c6b2f17bb9cc] pid:24590
[2019-02-23 12:47:29,776: INFO/MainProcess] Received task: repro.ping[55146a44-57b3-4738-b0d8-af09d99db266]
[2019-02-23 12:47:29,780: INFO/ForkPoolWorker-3] Task repro.ping[97a705fc-2208-4551-90b5-3fc1d54d79a0] succeeded in 10.00881665002089s: 5
[2019-02-23 12:47:29,781: DEBUG/MainProcess] TaskPool: Apply <function _fast_trace_task at 0x7f92462897b8> (args:('repro.ping', '31c2c135-76a5-4682-b400-3508ab9746ee', {'lang': 'py', 'task': 'repro.ping', 'id': '31c2c135-76a5-4682-b400-3508ab9746ee', 'shadow': None, 'eta': None, 'expires': None, 'group': None, 'retries': 0, 'timelimit': [None, None], 'root_id': '31c2c135-76a5-4682-b400-3508ab9746ee', 'parent_id': None, 'argsrepr': '(9,)', 'kwargsrepr': '{}', 'origin': 'gen24599@repro', 'reply_to': '13b4de4e-3451-330c-837d-91fcc7545757', 'correlation_id': '31c2c135-76a5-4682-b400-3508ab9746ee', 'delivery_info': {'exchange': '', 'routing_key': 'celery', 'priority': 0, 'redelivered': False}}, '[[9], {}, {"callbacks": null, "errbacks": null, "chain": null, "chord": null}]', 'application/json', 'utf-8') kwargs:{})
[2019-02-23 12:47:29,783: DEBUG/MainProcess] Task accepted: repro.ping[31c2c135-76a5-4682-b400-3508ab9746ee] pid:24589
[2019-02-23 12:47:29,784: INFO/ForkPoolWorker-2] Task repro.ping[a1eb440e-708c-4df2-a843-d678fca5876b] succeeded in 10.010429346002638s: 6
[2019-02-23 12:47:29,784: INFO/ForkPoolWorker-1] Task repro.ping[275b1a5a-6472-45f0-8710-4593e5f80b5a] succeeded in 10.008141046972014s: 7
[2019-02-23 12:47:29,786: DEBUG/MainProcess] TaskPool: Apply <function _fast_trace_task at 0x7f92462897b8> (args:('repro.ping', 'c7a5aba5-29f8-4046-9662-271035823431', {'lang': 'py', 'task': 'repro.ping', 'id': 'c7a5aba5-29f8-4046-9662-271035823431', 'shadow': None, 'eta': None, 'expires': None, 'group': None, 'retries': 0, 'timelimit': [None, None], 'root_id': 'c7a5aba5-29f8-4046-9662-271035823431', 'parent_id': None, 'argsrepr': '(10,)', 'kwargsrepr': '{}', 'origin': 'gen24599@repro', 'reply_to': '13b4de4e-3451-330c-837d-91fcc7545757', 'correlation_id': 'c7a5aba5-29f8-4046-9662-271035823431', 'delivery_info': {'exchange': '', 'routing_key': 'celery', 'priority': 0, 'redelivered': False}}, '[[10], {}, {"callbacks": null, "errbacks": null, "chain": null, "chord": null}]', 'application/json', 'utf-8') kwargs:{})
[2019-02-23 12:47:29,787: DEBUG/MainProcess] TaskPool: Apply <function _fast_trace_task at 0x7f92462897b8> (args:('repro.ping', 'e063884c-b7ba-48f6-8623-38a6d2ab174c', {'lang': 'py', 'task': 'repro.ping', 'id': 'e063884c-b7ba-48f6-8623-38a6d2ab174c', 'shadow': None, 'eta': None, 'expires': None, 'group': None, 'retries': 0, 'timelimit': [None, None], 'root_id': 'e063884c-b7ba-48f6-8623-38a6d2ab174c', 'parent_id': None, 'argsrepr': '(11,)', 'kwargsrepr': '{}', 'origin': 'gen24599@repro', 'reply_to': '13b4de4e-3451-330c-837d-91fcc7545757', 'correlation_id': 'e063884c-b7ba-48f6-8623-38a6d2ab174c', 'delivery_info': {'exchange': '', 'routing_key': 'celery', 'priority': 0, 'redelivered': False}}, '[[11], {}, {"callbacks": null, "errbacks": null, "chain": null, "chord": null}]', 'application/json', 'utf-8') kwargs:{})
[2019-02-23 12:47:29,790: INFO/MainProcess] Received task: repro.ping[4a9e5f83-6fa2-4c09-a925-e51d3ad9f48f]
[2019-02-23 12:47:29,792: DEBUG/MainProcess] Task accepted: repro.ping[c7a5aba5-29f8-4046-9662-271035823431] pid:24587
[2019-02-23 12:47:29,792: DEBUG/MainProcess] Task accepted: repro.ping[e063884c-b7ba-48f6-8623-38a6d2ab174c] pid:24588
[2019-02-23 12:47:29,795: INFO/MainProcess] Received task: repro.ping[07fcf53c-b692-46ff-bd1a-58f337694f72]
[2019-02-23 12:47:29,797: INFO/MainProcess] Received task: repro.ping[3ffbf0d3-3624-407f-a3fb-ad3dbbab6855]
[2019-02-23 12:47:39,784: INFO/ForkPoolWorker-4] Task repro.ping[6ee7bc0b-1c84-4973-8954-c6b2f17bb9cc] succeeded in 10.011397822992876s: 8
[2019-02-23 12:47:39,785: DEBUG/MainProcess] TaskPool: Apply <function _fast_trace_task at 0x7f92462897b8> (args:('repro.ping', 'f85a56c1-d26d-4fa2-8f81-1cfe60fff298', {'lang': 'py', 'task': 'repro.ping', 'id': 'f85a56c1-d26d-4fa2-8f81-1cfe60fff298', 'shadow': None, 'eta': None, 'expires': None, 'group': None, 'retries': 0, 'timelimit': [None, None], 'root_id': 'f85a56c1-d26d-4fa2-8f81-1cfe60fff298', 'parent_id': None, 'argsrepr': '(12,)', 'kwargsrepr': '{}', 'origin': 'gen24599@repro', 'reply_to': '13b4de4e-3451-330c-837d-91fcc7545757', 'correlation_id': 'f85a56c1-d26d-4fa2-8f81-1cfe60fff298', 'delivery_info': {'exchange': '', 'routing_key': 'celery', 'priority': 0, 'redelivered': False}}, '[[12], {}, {"callbacks": null, "errbacks": null, "chain": null, "chord": null}]', 'application/json', 'utf-8') kwargs:{})
[2019-02-23 12:47:39,787: DEBUG/MainProcess] Task accepted: repro.ping[f85a56c1-d26d-4fa2-8f81-1cfe60fff298] pid:24590
[2019-02-23 12:47:39,790: INFO/MainProcess] Received task: repro.ping[eb59f7a0-16f4-4cd4-9d49-c6f8a08cd46a]
[2019-02-23 12:47:39,792: INFO/ForkPoolWorker-3] Task repro.ping[31c2c135-76a5-4682-b400-3508ab9746ee] succeeded in 10.009430586942472s: 9
[2019-02-23 12:47:39,793: DEBUG/MainProcess] TaskPool: Apply <function _fast_trace_task at 0x7f92462897b8> (args:('repro.ping', 'efa85c32-ef70-453b-97e8-57497f1245c2', {'lang': 'py', 'task': 'repro.ping', 'id': 'efa85c32-ef70-453b-97e8-57497f1245c2', 'shadow': None, 'eta': None, 'expires': None, 'group': None, 'retries': 0, 'timelimit': [None, None], 'root_id': 'efa85c32-ef70-453b-97e8-57497f1245c2', 'parent_id': None, 'argsrepr': '(13,)', 'kwargsrepr': '{}', 'origin': 'gen24599@repro', 'reply_to': '13b4de4e-3451-330c-837d-91fcc7545757', 'correlation_id': 'efa85c32-ef70-453b-97e8-57497f1245c2', 'delivery_info': {'exchange': '', 'routing_key': 'celery', 'priority': 0, 'redelivered': False}}, '[[13], {}, {"callbacks": null, "errbacks": null, "chain": null, "chord": null}]', 'application/json', 'utf-8') kwargs:{})
[2019-02-23 12:47:39,794: DEBUG/MainProcess] Task accepted: repro.ping[efa85c32-ef70-453b-97e8-57497f1245c2] pid:24589
[2019-02-23 12:47:39,797: INFO/MainProcess] Received task: repro.ping[040e0b9e-7bd7-4b25-aa59-660e40531563]
[2019-02-23 12:47:39,800: INFO/ForkPoolWorker-1] Task repro.ping[c7a5aba5-29f8-4046-9662-271035823431] succeeded in 10.009039321914315s: 10
[2019-02-23 12:47:39,800: INFO/ForkPoolWorker-2] Task repro.ping[e063884c-b7ba-48f6-8623-38a6d2ab174c] succeeded in 10.008984215906821s: 11
[2019-02-23 12:47:39,801: DEBUG/MainProcess] TaskPool: Apply <function _fast_trace_task at 0x7f92462897b8> (args:('repro.ping', 'b0957ce3-4915-4f86-a728-4d47759bee51', {'lang': 'py', 'task': 'repro.ping', 'id': 'b0957ce3-4915-4f86-a728-4d47759bee51', 'shadow': None, 'eta': None, 'expires': None, 'group': None, 'retries': 0, 'timelimit': [None, None], 'root_id': 'b0957ce3-4915-4f86-a728-4d47759bee51', 'parent_id': None, 'argsrepr': '(14,)', 'kwargsrepr': '{}', 'origin': 'gen24599@repro', 'reply_to': '13b4de4e-3451-330c-837d-91fcc7545757', 'correlation_id': 'b0957ce3-4915-4f86-a728-4d47759bee51', 'delivery_info': {'exchange': '', 'routing_key': 'celery', 'priority': 0, 'redelivered': False}}, '[[14], {}, {"callbacks": null, "errbacks": null, "chain": null, "chord": null}]', 'application/json', 'utf-8') kwargs:{})
[2019-02-23 12:47:39,802: DEBUG/MainProcess] TaskPool: Apply <function _fast_trace_task at 0x7f92462897b8> (args:('repro.ping', '910ff039-f563-4f00-a631-fe131c9cedb4', {'lang': 'py', 'task': 'repro.ping', 'id': '910ff039-f563-4f00-a631-fe131c9cedb4', 'shadow': None, 'eta': None, 'expires': None, 'group': None, 'retries': 0, 'timelimit': [None, None], 'root_id': '910ff039-f563-4f00-a631-fe131c9cedb4', 'parent_id': None, 'argsrepr': '(15,)', 'kwargsrepr': '{}', 'origin': 'gen24599@repro', 'reply_to': '13b4de4e-3451-330c-837d-91fcc7545757', 'correlation_id': '910ff039-f563-4f00-a631-fe131c9cedb4', 'delivery_info': {'exchange': '', 'routing_key': 'celery', 'priority': 0, 'redelivered': False}}, '[[15], {}, {"callbacks": null, "errbacks": null, "chain": null, "chord": null}]', 'application/json', 'utf-8') kwargs:{})
[2019-02-23 12:47:39,804: DEBUG/MainProcess] Task accepted: repro.ping[910ff039-f563-4f00-a631-fe131c9cedb4] pid:24588
[2019-02-23 12:47:39,804: DEBUG/MainProcess] Task accepted: repro.ping[b0957ce3-4915-4f86-a728-4d47759bee51] pid:24587
[2019-02-23 12:47:39,807: INFO/MainProcess] Received task: repro.ping[c9365f4b-74aa-4ad7-a78b-8527306b5c9b]
[2019-02-23 12:47:39,809: INFO/MainProcess] Received task: repro.ping[f91595cc-22f6-402d-8e5f-0ca907a72b97]
[2019-02-23 12:47:45,866: DEBUG/MainProcess] heartbeat_tick : for connection c96dac8c7b174cb3af302fd13ce1db55
[2019-02-23 12:47:45,866: DEBUG/MainProcess] heartbeat_tick : Prev sent/recv: 36/130, now - 44/185, monotonic - 827990.406034226, last_heartbeat_sent - 827990.406033, heartbeat int. - 60 for connection c96dac8c7b174cb3af302fd13ce1db55
[2019-02-23 12:47:49,795: INFO/ForkPoolWorker-4] Task repro.ping[f85a56c1-d26d-4fa2-8f81-1cfe60fff298] succeeded in 10.008562478935346s: 12
[2019-02-23 12:47:49,796: DEBUG/MainProcess] TaskPool: Apply <function _fast_trace_task at 0x7f92462897b8> (args:('repro.ping', '3c059acc-519b-49b2-ba3d-e5d93bbcc2da', {'lang': 'py', 'task': 'repro.ping', 'id': '3c059acc-519b-49b2-ba3d-e5d93bbcc2da', 'shadow': None, 'eta': None, 'expires': None, 'group': None, 'retries': 0, 'timelimit': [None, None], 'root_id': '3c059acc-519b-49b2-ba3d-e5d93bbcc2da', 'parent_id': None, 'argsrepr': '(16,)', 'kwargsrepr': '{}', 'origin': 'gen24599@repro', 'reply_to': '13b4de4e-3451-330c-837d-91fcc7545757', 'correlation_id': '3c059acc-519b-49b2-ba3d-e5d93bbcc2da', 'delivery_info': {'exchange': '', 'routing_key': 'celery', 'priority': 0, 'redelivered': False}}, '[[16], {}, {"callbacks": null, "errbacks": null, "chain": null, "chord": null}]', 'application/json', 'utf-8') kwargs:{})
[2019-02-23 12:47:49,798: DEBUG/MainProcess] Task accepted: repro.ping[3c059acc-519b-49b2-ba3d-e5d93bbcc2da] pid:24590
[2019-02-23 12:47:49,799: INFO/MainProcess] Received task: repro.ping[af2377f8-3ef9-4750-afee-f70bc9e4a6c8]
[2019-02-23 12:47:49,803: INFO/ForkPoolWorker-3] Task repro.ping[efa85c32-ef70-453b-97e8-57497f1245c2] succeeded in 10.009383243974298s: 13
[2019-02-23 12:47:49,804: DEBUG/MainProcess] TaskPool: Apply <function _fast_trace_task at 0x7f92462897b8> (args:('repro.ping', 'd75703fc-5e87-4152-97c7-72607249ecd8', {'lang': 'py', 'task': 'repro.ping', 'id': 'd75703fc-5e87-4152-97c7-72607249ecd8', 'shadow': None, 'eta': None, 'expires': None, 'group': None, 'retries': 0, 'timelimit': [None, None], 'root_id': 'd75703fc-5e87-4152-97c7-72607249ecd8', 'parent_id': None, 'argsrepr': '(17,)', 'kwargsrepr': '{}', 'origin': 'gen24599@repro', 'reply_to': '13b4de4e-3451-330c-837d-91fcc7545757', 'correlation_id': 'd75703fc-5e87-4152-97c7-72607249ecd8', 'delivery_info': {'exchange': '', 'routing_key': 'celery', 'priority': 0, 'redelivered': False}}, '[[17], {}, {"callbacks": null, "errbacks": null, "chain": null, "chord": null}]', 'application/json', 'utf-8') kwargs:{})
[2019-02-23 12:47:49,805: DEBUG/MainProcess] Task accepted: repro.ping[d75703fc-5e87-4152-97c7-72607249ecd8] pid:24589
[2019-02-23 12:47:49,808: INFO/MainProcess] Received task: repro.ping[c674700d-be4d-4bfa-a890-e26fd937b30a]
[2019-02-23 12:47:49,812: INFO/ForkPoolWorker-2] Task repro.ping[910ff039-f563-4f00-a631-fe131c9cedb4] succeeded in 10.008233243017457s: 15
[2019-02-23 12:47:49,812: INFO/ForkPoolWorker-1] Task repro.ping[b0957ce3-4915-4f86-a728-4d47759bee51] succeeded in 10.008232522988692s: 14
[2019-02-23 12:47:49,812: DEBUG/MainProcess] TaskPool: Apply <function _fast_trace_task at 0x7f92462897b8> (args:('repro.ping', 'b78ee077-4e13-4f38-b7f4-5baef4e88a1b', {'lang': 'py', 'task': 'repro.ping', 'id': 'b78ee077-4e13-4f38-b7f4-5baef4e88a1b', 'shadow': None, 'eta': None, 'expires': None, 'group': None, 'retries': 0, 'timelimit': [None, None], 'root_id': 'b78ee077-4e13-4f38-b7f4-5baef4e88a1b', 'parent_id': None, 'argsrepr': '(18,)', 'kwargsrepr': '{}', 'origin': 'gen24599@repro', 'reply_to': '13b4de4e-3451-330c-837d-91fcc7545757', 'correlation_id': 'b78ee077-4e13-4f38-b7f4-5baef4e88a1b', 'delivery_info': {'exchange': '', 'routing_key': 'celery', 'priority': 0, 'redelivered': False}}, '[[18], {}, {"callbacks": null, "errbacks": null, "chain": null, "chord": null}]', 'application/json', 'utf-8') kwargs:{})
[2019-02-23 12:47:49,813: DEBUG/MainProcess] TaskPool: Apply <function _fast_trace_task at 0x7f92462897b8> (args:('repro.ping', '82ba77ec-6874-4414-a814-fe432e871005', {'lang': 'py', 'task': 'repro.ping', 'id': '82ba77ec-6874-4414-a814-fe432e871005', 'shadow': None, 'eta': None, 'expires': None, 'group': None, 'retries': 0, 'timelimit': [None, None], 'root_id': '82ba77ec-6874-4414-a814-fe432e871005', 'parent_id': None, 'argsrepr': '(19,)', 'kwargsrepr': '{}', 'origin': 'gen24599@repro', 'reply_to': '13b4de4e-3451-330c-837d-91fcc7545757', 'correlation_id': '82ba77ec-6874-4414-a814-fe432e871005', 'delivery_info': {'exchange': '', 'routing_key': 'celery', 'priority': 0, 'redelivered': False}}, '[[19], {}, {"callbacks": null, "errbacks": null, "chain": null, "chord": null}]', 'application/json', 'utf-8') kwargs:{})
[2019-02-23 12:47:49,815: DEBUG/MainProcess] Task accepted: repro.ping[b78ee077-4e13-4f38-b7f4-5baef4e88a1b] pid:24587
[2019-02-23 12:47:49,815: DEBUG/MainProcess] Task accepted: repro.ping[82ba77ec-6874-4414-a814-fe432e871005] pid:24588
[2019-02-23 12:47:49,817: INFO/MainProcess] Received task: repro.ping[5c5fb57e-3408-409c-8c8e-4af1eb8d6e0f]
[2019-02-23 12:47:49,818: INFO/MainProcess] Received task: repro.ping[01dff44f-d3eb-4538-8f07-281e9733531b]
[2019-02-23 12:47:59,809: INFO/ForkPoolWorker-4] Task repro.ping[3c059acc-519b-49b2-ba3d-e5d93bbcc2da] succeeded in 10.011367695988156s: 16
[2019-02-23 12:47:59,810: DEBUG/MainProcess] TaskPool: Apply <function _fast_trace_task at 0x7f92462897b8> (args:('repro.ping', '91dbdb86-cbe6-4a57-b669-400211b3d477', {'lang': 'py', 'task': 'repro.ping', 'id': '91dbdb86-cbe6-4a57-b669-400211b3d477', 'shadow': None, 'eta': None, 'expires': None, 'group': None, 'retries': 0, 'timelimit': [None, None], 'root_id': '91dbdb86-cbe6-4a57-b669-400211b3d477', 'parent_id': None, 'argsrepr': '(20,)', 'kwargsrepr': '{}', 'origin': 'gen24599@repro', 'reply_to': '13b4de4e-3451-330c-837d-91fcc7545757', 'correlation_id': '91dbdb86-cbe6-4a57-b669-400211b3d477', 'delivery_info': {'exchange': '', 'routing_key': 'celery', 'priority': 0, 'redelivered': False}}, '[[20], {}, {"callbacks": null, "errbacks": null, "chain": null, "chord": null}]', 'application/json', 'utf-8') kwargs:{})
[2019-02-23 12:47:59,812: DEBUG/MainProcess] Task accepted: repro.ping[91dbdb86-cbe6-4a57-b669-400211b3d477] pid:24590
[2019-02-23 12:47:59,815: INFO/MainProcess] Received task: repro.ping[9ac70541-1428-4877-a1a7-ca4d9196ea76]
[2019-02-23 12:47:59,816: INFO/ForkPoolWorker-3] Task repro.ping[d75703fc-5e87-4152-97c7-72607249ecd8] succeeded in 10.010803959914483s: 17
[2019-02-23 12:47:59,817: DEBUG/MainProcess] TaskPool: Apply <function _fast_trace_task at 0x7f92462897b8> (args:('repro.ping', 'd3767e82-7208-4b02-a8df-467e260f5afd', {'lang': 'py', 'task': 'repro.ping', 'id': 'd3767e82-7208-4b02-a8df-467e260f5afd', 'shadow': None, 'eta': None, 'expires': None, 'group': None, 'retries': 0, 'timelimit': [None, None], 'root_id': 'd3767e82-7208-4b02-a8df-467e260f5afd', 'parent_id': None, 'argsrepr': '(21,)', 'kwargsrepr': '{}', 'origin': 'gen24599@repro', 'reply_to': '13b4de4e-3451-330c-837d-91fcc7545757', 'correlation_id': 'd3767e82-7208-4b02-a8df-467e260f5afd', 'delivery_info': {'exchange': '', 'routing_key': 'celery', 'priority': 0, 'redelivered': False}}, '[[21], {}, {"callbacks": null, "errbacks": null, "chain": null, "chord": null}]', 'application/json', 'utf-8') kwargs:{})
[2019-02-23 12:47:59,819: DEBUG/MainProcess] Task accepted: repro.ping[d3767e82-7208-4b02-a8df-467e260f5afd] pid:24589
[2019-02-23 12:47:59,823: INFO/MainProcess] Received task: repro.ping[372e0fca-0934-4260-9ac8-5cb5f3c8dfef]
[2019-02-23 12:47:59,824: INFO/ForkPoolWorker-1] Task repro.ping[b78ee077-4e13-4f38-b7f4-5baef4e88a1b] succeeded in 10.009969289996661s: 18
[2019-02-23 12:47:59,827: INFO/ForkPoolWorker-2] Task repro.ping[82ba77ec-6874-4414-a814-fe432e871005] succeeded in 10.012633722042665s: 19
[2019-02-23 12:47:59,827: DEBUG/MainProcess] TaskPool: Apply <function _fast_trace_task at 0x7f92462897b8> (args:('repro.ping', '0208e029-4283-4a1d-b978-a2e12efc892a', {'lang': 'py', 'task': 'repro.ping', 'id': '0208e029-4283-4a1d-b978-a2e12efc892a', 'shadow': None, 'eta': None, 'expires': None, 'group': None, 'retries': 0, 'timelimit': [None, None], 'root_id': '0208e029-4283-4a1d-b978-a2e12efc892a', 'parent_id': None, 'argsrepr': '(22,)', 'kwargsrepr': '{}', 'origin': 'gen24599@repro', 'reply_to': '13b4de4e-3451-330c-837d-91fcc7545757', 'correlation_id': '0208e029-4283-4a1d-b978-a2e12efc892a', 'delivery_info': {'exchange': '', 'routing_key': 'celery', 'priority': 0, 'redelivered': False}}, '[[22], {}, {"callbacks": null, "errbacks": null, "chain": null, "chord": null}]', 'application/json', 'utf-8') kwargs:{})
[2019-02-23 12:47:59,829: DEBUG/MainProcess] TaskPool: Apply <function _fast_trace_task at 0x7f92462897b8> (args:('repro.ping', 'dd272fa0-226f-4b76-97a7-6b66d2a6f8ec', {'lang': 'py', 'task': 'repro.ping', 'id': 'dd272fa0-226f-4b76-97a7-6b66d2a6f8ec', 'shadow': None, 'eta': None, 'expires': None, 'group': None, 'retries': 0, 'timelimit': [None, None], 'root_id': 'dd272fa0-226f-4b76-97a7-6b66d2a6f8ec', 'parent_id': None, 'argsrepr': '(23,)', 'kwargsrepr': '{}', 'origin': 'gen24599@repro', 'reply_to': '13b4de4e-3451-330c-837d-91fcc7545757', 'correlation_id': 'dd272fa0-226f-4b76-97a7-6b66d2a6f8ec', 'delivery_info': {'exchange': '', 'routing_key': 'celery', 'priority': 0, 'redelivered': False}}, '[[23], {}, {"callbacks": null, "errbacks": null, "chain": null, "chord": null}]', 'application/json', 'utf-8') kwargs:{})
[2019-02-23 12:47:59,831: DEBUG/MainProcess] Task accepted: repro.ping[dd272fa0-226f-4b76-97a7-6b66d2a6f8ec] pid:24588
[2019-02-23 12:47:59,832: DEBUG/MainProcess] Task accepted: repro.ping[0208e029-4283-4a1d-b978-a2e12efc892a] pid:24587
[2019-02-23 12:47:59,836: INFO/MainProcess] Received task: repro.ping[23920044-7453-426c-a804-8016859410c4]
[2019-02-23 12:47:59,837: INFO/MainProcess] Received task: repro.ping[0b8c91fe-c9a8-49bc-b857-f04f12692fec]
[2019-02-23 12:48:05,868: DEBUG/MainProcess] heartbeat_tick : for connection c96dac8c7b174cb3af302fd13ce1db55
[2019-02-23 12:48:05,868: DEBUG/MainProcess] heartbeat_tick : Prev sent/recv: 44/185, now - 52/239, monotonic - 828010.408377561, last_heartbeat_sent - 828010.408374154, heartbeat int. - 60 for connection c96dac8c7b174cb3af302fd13ce1db55
[2019-02-23 12:48:09,816: INFO/ForkPoolWorker-4] Task repro.ping[91dbdb86-cbe6-4a57-b669-400211b3d477] succeeded in 10.004684619954787s: 20
[2019-02-23 12:48:09,817: DEBUG/MainProcess] TaskPool: Apply <function _fast_trace_task at 0x7f92462897b8> (args:('repro.ping', '55146a44-57b3-4738-b0d8-af09d99db266', {'lang': 'py', 'task': 'repro.ping', 'id': '55146a44-57b3-4738-b0d8-af09d99db266', 'shadow': None, 'eta': None, 'expires': None, 'group': None, 'retries': 0, 'timelimit': [None, None], 'root_id': '55146a44-57b3-4738-b0d8-af09d99db266', 'parent_id': None, 'argsrepr': '(24,)', 'kwargsrepr': '{}', 'origin': 'gen24599@repro', 'reply_to': '13b4de4e-3451-330c-837d-91fcc7545757', 'correlation_id': '55146a44-57b3-4738-b0d8-af09d99db266', 'delivery_info': {'exchange': '', 'routing_key': 'celery', 'priority': 0, 'redelivered': False}}, '[[24], {}, {"callbacks": null, "errbacks": null, "chain": null, "chord": null}]', 'application/json', 'utf-8') kwargs:{})
[2019-02-23 12:48:09,819: DEBUG/MainProcess] Task accepted: repro.ping[55146a44-57b3-4738-b0d8-af09d99db266] pid:24590
[2019-02-23 12:48:09,822: INFO/MainProcess] Received task: repro.ping[8b919229-36ab-4387-b8ff-446e461e462e]
[2019-02-23 12:48:09,828: INFO/ForkPoolWorker-3] Task repro.ping[d3767e82-7208-4b02-a8df-467e260f5afd] succeeded in 10.0091544049792s: 21
[2019-02-23 12:48:09,829: DEBUG/MainProcess] TaskPool: Apply <function _fast_trace_task at 0x7f92462897b8> (args:('repro.ping', '4a9e5f83-6fa2-4c09-a925-e51d3ad9f48f', {'lang': 'py', 'task': 'repro.ping', 'id': '4a9e5f83-6fa2-4c09-a925-e51d3ad9f48f', 'shadow': None, 'eta': None, 'expires': None, 'group': None, 'retries': 0, 'timelimit': [None, None], 'root_id': '4a9e5f83-6fa2-4c09-a925-e51d3ad9f48f', 'parent_id': None, 'argsrepr': '(25,)', 'kwargsrepr': '{}', 'origin': 'gen24599@repro', 'reply_to': '13b4de4e-3451-330c-837d-91fcc7545757', 'correlation_id': '4a9e5f83-6fa2-4c09-a925-e51d3ad9f48f', 'delivery_info': {'exchange': '', 'routing_key': 'celery', 'priority': 0, 'redelivered': False}}, '[[25], {}, {"callbacks": null, "errbacks": null, "chain": null, "chord": null}]', 'application/json', 'utf-8') kwargs:{})
[2019-02-23 12:48:09,831: DEBUG/MainProcess] Task accepted: repro.ping[4a9e5f83-6fa2-4c09-a925-e51d3ad9f48f] pid:24589
[2019-02-23 12:48:09,834: INFO/MainProcess] Received task: repro.ping[66105ea1-c574-4838-b297-cd1f861ac2fd]
[2019-02-23 12:48:09,840: INFO/ForkPoolWorker-1] Task repro.ping[0208e029-4283-4a1d-b978-a2e12efc892a] succeeded in 10.009812505100854s: 22
[2019-02-23 12:48:09,841: DEBUG/MainProcess] TaskPool: Apply <function _fast_trace_task at 0x7f92462897b8> (args:('repro.ping', '07fcf53c-b692-46ff-bd1a-58f337694f72', {'lang': 'py', 'task': 'repro.ping', 'id': '07fcf53c-b692-46ff-bd1a-58f337694f72', 'shadow': None, 'eta': None, 'expires': None, 'group': None, 'retries': 0, 'timelimit': [None, None], 'root_id': '07fcf53c-b692-46ff-bd1a-58f337694f72', 'parent_id': None, 'argsrepr': '(26,)', 'kwargsrepr': '{}', 'origin': 'gen24599@repro', 'reply_to': '13b4de4e-3451-330c-837d-91fcc7545757', 'correlation_id': '07fcf53c-b692-46ff-bd1a-58f337694f72', 'delivery_info': {'exchange': '', 'routing_key': 'celery', 'priority': 0, 'redelivered': False}}, '[[26], {}, {"callbacks": null, "errbacks": null, "chain": null, "chord": null}]', 'application/json', 'utf-8') kwargs:{})
[2019-02-23 12:48:09,843: INFO/ForkPoolWorker-2] Task repro.ping[dd272fa0-226f-4b76-97a7-6b66d2a6f8ec] succeeded in 10.012293447973207s: 23
[2019-02-23 12:48:09,845: DEBUG/MainProcess] Task accepted: repro.ping[07fcf53c-b692-46ff-bd1a-58f337694f72] pid:24587
[2019-02-23 12:48:09,845: DEBUG/MainProcess] TaskPool: Apply <function _fast_trace_task at 0x7f92462897b8> (args:('repro.ping', '3ffbf0d3-3624-407f-a3fb-ad3dbbab6855', {'lang': 'py', 'task': 'repro.ping', 'id': '3ffbf0d3-3624-407f-a3fb-ad3dbbab6855', 'shadow': None, 'eta': None, 'expires': None, 'group': None, 'retries': 0, 'timelimit': [None, None], 'root_id': '3ffbf0d3-3624-407f-a3fb-ad3dbbab6855', 'parent_id': None, 'argsrepr': '(27,)', 'kwargsrepr': '{}', 'origin': 'gen24599@repro', 'reply_to': '13b4de4e-3451-330c-837d-91fcc7545757', 'correlation_id': '3ffbf0d3-3624-407f-a3fb-ad3dbbab6855', 'delivery_info': {'exchange': '', 'routing_key': 'celery', 'priority': 0, 'redelivered': False}}, '[[27], {}, {"callbacks": null, "errbacks": null, "chain": null, "chord": null}]', 'application/json', 'utf-8') kwargs:{})
[2019-02-23 12:48:09,848: INFO/MainProcess] Received task: repro.ping[322e398a-d9a2-486a-90d4-0012f0f1d804]
[2019-02-23 12:48:09,850: DEBUG/MainProcess] Task accepted: repro.ping[3ffbf0d3-3624-407f-a3fb-ad3dbbab6855] pid:24588
[2019-02-23 12:48:09,852: INFO/MainProcess] Received task: repro.ping[d37541c2-81b1-429c-87b3-842d5bf76242]
[2019-02-23 12:48:19,827: INFO/ForkPoolWorker-4] Task repro.ping[55146a44-57b3-4738-b0d8-af09d99db266] succeeded in 10.008311487035826s: 24
[2019-02-23 12:48:19,828: DEBUG/MainProcess] TaskPool: Apply <function _fast_trace_task at 0x7f92462897b8> (args:('repro.ping', 'eb59f7a0-16f4-4cd4-9d49-c6f8a08cd46a', {'lang': 'py', 'task': 'repro.ping', 'id': 'eb59f7a0-16f4-4cd4-9d49-c6f8a08cd46a', 'shadow': None, 'eta': None, 'expires': None, 'group': None, 'retries': 0, 'timelimit': [None, None], 'root_id': 'eb59f7a0-16f4-4cd4-9d49-c6f8a08cd46a', 'parent_id': None, 'argsrepr': '(28,)', 'kwargsrepr': '{}', 'origin': 'gen24599@repro', 'reply_to': '13b4de4e-3451-330c-837d-91fcc7545757', 'correlation_id': 'eb59f7a0-16f4-4cd4-9d49-c6f8a08cd46a', 'delivery_info': {'exchange': '', 'routing_key': 'celery', 'priority': 0, 'redelivered': False}}, '[[28], {}, {"callbacks": null, "errbacks": null, "chain": null, "chord": null}]', 'application/json', 'utf-8') kwargs:{})
[2019-02-23 12:48:19,828: DEBUG/MainProcess] Task accepted: repro.ping[eb59f7a0-16f4-4cd4-9d49-c6f8a08cd46a] pid:24590
[2019-02-23 12:48:19,830: INFO/MainProcess] Received task: repro.ping[ba330fe7-2223-4581-b803-6ba957c30f56]
[2019-02-23 12:48:19,839: INFO/ForkPoolWorker-3] Task repro.ping[4a9e5f83-6fa2-4c09-a925-e51d3ad9f48f] succeeded in 10.00862043199595s: 25
[2019-02-23 12:48:19,840: DEBUG/MainProcess] TaskPool: Apply <function _fast_trace_task at 0x7f92462897b8> (args:('repro.ping', '040e0b9e-7bd7-4b25-aa59-660e40531563', {'lang': 'py', 'task': 'repro.ping', 'id': '040e0b9e-7bd7-4b25-aa59-660e40531563', 'shadow': None, 'eta': None, 'expires': None, 'group': None, 'retries': 0, 'timelimit': [None, None], 'root_id': '040e0b9e-7bd7-4b25-aa59-660e40531563', 'parent_id': None, 'argsrepr': '(29,)', 'kwargsrepr': '{}', 'origin': 'gen24599@repro', 'reply_to': '13b4de4e-3451-330c-837d-91fcc7545757', 'correlation_id': '040e0b9e-7bd7-4b25-aa59-660e40531563', 'delivery_info': {'exchange': '', 'routing_key': 'celery', 'priority': 0, 'redelivered': False}}, '[[29], {}, {"callbacks": null, "errbacks": null, "chain": null, "chord": null}]', 'application/json', 'utf-8') kwargs:{})
[2019-02-23 12:48:19,841: DEBUG/MainProcess] Task accepted: repro.ping[040e0b9e-7bd7-4b25-aa59-660e40531563] pid:24589
[2019-02-23 12:48:19,843: INFO/MainProcess] Received task: repro.ping[73383b31-c16d-442a-b35e-51dcbd248c64]
[2019-02-23 12:48:19,851: INFO/ForkPoolWorker-1] Task repro.ping[07fcf53c-b692-46ff-bd1a-58f337694f72] succeeded in 10.008613758021966s: 26
[2019-02-23 12:48:19,851: DEBUG/MainProcess] TaskPool: Apply <function _fast_trace_task at 0x7f92462897b8> (args:('repro.ping', 'c9365f4b-74aa-4ad7-a78b-8527306b5c9b', {'lang': 'py', 'task': 'repro.ping', 'id': 'c9365f4b-74aa-4ad7-a78b-8527306b5c9b', 'shadow': None, 'eta': None, 'expires': None, 'group': None, 'retries': 0, 'timelimit': [None, None], 'root_id': 'c9365f4b-74aa-4ad7-a78b-8527306b5c9b', 'parent_id': None, 'argsrepr': '(30,)', 'kwargsrepr': '{}', 'origin': 'gen24599@repro', 'reply_to': '13b4de4e-3451-330c-837d-91fcc7545757', 'correlation_id': 'c9365f4b-74aa-4ad7-a78b-8527306b5c9b', 'delivery_info': {'exchange': '', 'routing_key': 'celery', 'priority': 0, 'redelivered': False}}, '[[30], {}, {"callbacks": null, "errbacks": null, "chain": null, "chord": null}]', 'application/json', 'utf-8') kwargs:{})
[2019-02-23 12:48:19,852: DEBUG/MainProcess] Task accepted: repro.ping[c9365f4b-74aa-4ad7-a78b-8527306b5c9b] pid:24587
[2019-02-23 12:48:19,853: INFO/MainProcess] Received task: repro.ping[f71cd3c6-eeb3-424f-8a8a-a9830fed5a49]
[2019-02-23 12:48:19,856: INFO/ForkPoolWorker-2] Task repro.ping[3ffbf0d3-3624-407f-a3fb-ad3dbbab6855] succeeded in 10.008551157079637s: 27
[2019-02-23 12:48:19,856: DEBUG/MainProcess] TaskPool: Apply <function _fast_trace_task at 0x7f92462897b8> (args:('repro.ping', 'f91595cc-22f6-402d-8e5f-0ca907a72b97', {'lang': 'py', 'task': 'repro.ping', 'id': 'f91595cc-22f6-402d-8e5f-0ca907a72b97', 'shadow': None, 'eta': None, 'expires': None, 'group': None, 'retries': 0, 'timelimit': [None, None], 'root_id': 'f91595cc-22f6-402d-8e5f-0ca907a72b97', 'parent_id': None, 'argsrepr': '(31,)', 'kwargsrepr': '{}', 'origin': 'gen24599@repro', 'reply_to': '13b4de4e-3451-330c-837d-91fcc7545757', 'correlation_id': 'f91595cc-22f6-402d-8e5f-0ca907a72b97', 'delivery_info': {'exchange': '', 'routing_key': 'celery', 'priority': 0, 'redelivered': False}}, '[[31], {}, {"callbacks": null, "errbacks": null, "chain": null, "chord": null}]', 'application/json', 'utf-8') kwargs:{})
[2019-02-23 12:48:19,857: DEBUG/MainProcess] Task accepted: repro.ping[f91595cc-22f6-402d-8e5f-0ca907a72b97] pid:24588
[2019-02-23 12:48:19,857: INFO/MainProcess] Received task: repro.ping[c5e658bb-43d9-4caa-99a6-63c9b5616b06]
[2019-02-23 12:48:25,871: DEBUG/MainProcess] heartbeat_tick : for connection c96dac8c7b174cb3af302fd13ce1db55
[2019-02-23 12:48:25,871: DEBUG/MainProcess] heartbeat_tick : Prev sent/recv: 52/239, now - 60/293, monotonic - 828030.411141046, last_heartbeat_sent - 828030.411137678, heartbeat int. - 60 for connection c96dac8c7b174cb3af302fd13ce1db55
[2019-02-23 12:48:29,840: INFO/ForkPoolWorker-4] Task repro.ping[eb59f7a0-16f4-4cd4-9d49-c6f8a08cd46a] succeeded in 10.011380798998289s: 28
[2019-02-23 12:48:29,841: DEBUG/MainProcess] TaskPool: Apply <function _fast_trace_task at 0x7f92462897b8> (args:('repro.ping', 'af2377f8-3ef9-4750-afee-f70bc9e4a6c8', {'lang': 'py', 'task': 'repro.ping', 'id': 'af2377f8-3ef9-4750-afee-f70bc9e4a6c8', 'shadow': None, 'eta': None, 'expires': None, 'group': None, 'retries': 0, 'timelimit': [None, None], 'root_id': 'af2377f8-3ef9-4750-afee-f70bc9e4a6c8', 'parent_id': None, 'argsrepr': '(32,)', 'kwargsrepr': '{}', 'origin': 'gen24599@repro', 'reply_to': '13b4de4e-3451-330c-837d-91fcc7545757', 'correlation_id': 'af2377f8-3ef9-4750-afee-f70bc9e4a6c8', 'delivery_info': {'exchange': '', 'routing_key': 'celery', 'priority': 0, 'redelivered': False}}, '[[32], {}, {"callbacks": null, "errbacks": null, "chain": null, "chord": null}]', 'application/json', 'utf-8') kwargs:{})
[2019-02-23 12:48:29,842: DEBUG/MainProcess] Task accepted: repro.ping[af2377f8-3ef9-4750-afee-f70bc9e4a6c8] pid:24590
[2019-02-23 12:48:29,845: INFO/MainProcess] Received task: repro.ping[bb2b74b4-1385-4321-8f8a-685dd1198873]
[2019-02-23 12:48:29,852: INFO/ForkPoolWorker-3] Task repro.ping[040e0b9e-7bd7-4b25-aa59-660e40531563] succeeded in 10.01107842195779s: 29
[2019-02-23 12:48:29,853: DEBUG/MainProcess] TaskPool: Apply <function _fast_trace_task at 0x7f92462897b8> (args:('repro.ping', 'c674700d-be4d-4bfa-a890-e26fd937b30a', {'lang': 'py', 'task': 'repro.ping', 'id': 'c674700d-be4d-4bfa-a890-e26fd937b30a', 'shadow': None, 'eta': None, 'expires': None, 'group': None, 'retries': 0, 'timelimit': [None, None], 'root_id': 'c674700d-be4d-4bfa-a890-e26fd937b30a', 'parent_id': None, 'argsrepr': '(33,)', 'kwargsrepr': '{}', 'origin': 'gen24599@repro', 'reply_to': '13b4de4e-3451-330c-837d-91fcc7545757', 'correlation_id': 'c674700d-be4d-4bfa-a890-e26fd937b30a', 'delivery_info': {'exchange': '', 'routing_key': 'celery', 'priority': 0, 'redelivered': False}}, '[[33], {}, {"callbacks": null, "errbacks": null, "chain": null, "chord": null}]', 'application/json', 'utf-8') kwargs:{})
[2019-02-23 12:48:29,855: DEBUG/MainProcess] Task accepted: repro.ping[c674700d-be4d-4bfa-a890-e26fd937b30a] pid:24589
[2019-02-23 12:48:29,857: INFO/MainProcess] Received task: repro.ping[e8d7d0fe-700d-4b55-b3ef-422265f142a1]
[2019-02-23 12:48:29,862: INFO/ForkPoolWorker-1] Task repro.ping[c9365f4b-74aa-4ad7-a78b-8527306b5c9b] succeeded in 10.009829671005718s: 30
[2019-02-23 12:48:29,863: DEBUG/MainProcess] TaskPool: Apply <function _fast_trace_task at 0x7f92462897b8> (args:('repro.ping', '5c5fb57e-3408-409c-8c8e-4af1eb8d6e0f', {'lang': 'py', 'task': 'repro.ping', 'id': '5c5fb57e-3408-409c-8c8e-4af1eb8d6e0f', 'shadow': None, 'eta': None, 'expires': None, 'group': None, 'retries': 0, 'timelimit': [None, None], 'root_id': '5c5fb57e-3408-409c-8c8e-4af1eb8d6e0f', 'parent_id': None, 'argsrepr': '(34,)', 'kwargsrepr': '{}', 'origin': 'gen24599@repro', 'reply_to': '13b4de4e-3451-330c-837d-91fcc7545757', 'correlation_id': '5c5fb57e-3408-409c-8c8e-4af1eb8d6e0f', 'delivery_info': {'exchange': '', 'routing_key': 'celery', 'priority': 0, 'redelivered': False}}, '[[34], {}, {"callbacks": null, "errbacks": null, "chain": null, "chord": null}]', 'application/json', 'utf-8') kwargs:{})
[2019-02-23 12:48:29,864: DEBUG/MainProcess] Task accepted: repro.ping[5c5fb57e-3408-409c-8c8e-4af1eb8d6e0f] pid:24587
[2019-02-23 12:48:29,866: INFO/MainProcess] Received task: repro.ping[1a663699-ca2a-4ae1-a9ab-01094ad98e0d]
[2019-02-23 12:48:29,868: INFO/ForkPoolWorker-2] Task repro.ping[f91595cc-22f6-402d-8e5f-0ca907a72b97] succeeded in 10.01112971897237s: 31
[2019-02-23 12:48:29,869: DEBUG/MainProcess] TaskPool: Apply <function _fast_trace_task at 0x7f92462897b8> (args:('repro.ping', '01dff44f-d3eb-4538-8f07-281e9733531b', {'lang': 'py', 'task': 'repro.ping', 'id': '01dff44f-d3eb-4538-8f07-281e9733531b', 'shadow': None, 'eta': None, 'expires': None, 'group': None, 'retries': 0, 'timelimit': [None, None], 'root_id': '01dff44f-d3eb-4538-8f07-281e9733531b', 'parent_id': None, 'argsrepr': '(35,)', 'kwargsrepr': '{}', 'origin': 'gen24599@repro', 'reply_to': '13b4de4e-3451-330c-837d-91fcc7545757', 'correlation_id': '01dff44f-d3eb-4538-8f07-281e9733531b', 'delivery_info': {'exchange': '', 'routing_key': 'celery', 'priority': 0, 'redelivered': False}}, '[[35], {}, {"callbacks": null, "errbacks": null, "chain": null, "chord": null}]', 'application/json', 'utf-8') kwargs:{})
[2019-02-23 12:48:29,870: DEBUG/MainProcess] Task accepted: repro.ping[01dff44f-d3eb-4538-8f07-281e9733531b] pid:24588
[2019-02-23 12:48:29,873: INFO/MainProcess] Received task: repro.ping[2c529c3a-2a82-45ce-bbf9-6b305a425efa]
[2019-02-23 12:48:39,852: INFO/ForkPoolWorker-4] Task repro.ping[af2377f8-3ef9-4750-afee-f70bc9e4a6c8] succeeded in 10.009791500982828s: 32
[2019-02-23 12:48:39,853: DEBUG/MainProcess] TaskPool: Apply <function _fast_trace_task at 0x7f92462897b8> (args:('repro.ping', '9ac70541-1428-4877-a1a7-ca4d9196ea76', {'lang': 'py', 'task': 'repro.ping', 'id': '9ac70541-1428-4877-a1a7-ca4d9196ea76', 'shadow': None, 'eta': None, 'expires': None, 'group': None, 'retries': 0, 'timelimit': [None, None], 'root_id': '9ac70541-1428-4877-a1a7-ca4d9196ea76', 'parent_id': None, 'argsrepr': '(36,)', 'kwargsrepr': '{}', 'origin': 'gen24599@repro', 'reply_to': '13b4de4e-3451-330c-837d-91fcc7545757', 'correlation_id': '9ac70541-1428-4877-a1a7-ca4d9196ea76', 'delivery_info': {'exchange': '', 'routing_key': 'celery', 'priority': 0, 'redelivered': False}}, '[[36], {}, {"callbacks": null, "errbacks": null, "chain": null, "chord": null}]', 'application/json', 'utf-8') kwargs:{})
[2019-02-23 12:48:39,856: DEBUG/MainProcess] Task accepted: repro.ping[9ac70541-1428-4877-a1a7-ca4d9196ea76] pid:24590
[2019-02-23 12:48:39,859: INFO/MainProcess] Received task: repro.ping[f3835161-e5d0-478a-87fc-b48f4f8ca51a]
[2019-02-23 12:48:39,866: INFO/ForkPoolWorker-3] Task repro.ping[c674700d-be4d-4bfa-a890-e26fd937b30a] succeeded in 10.01175982109271s: 33
[2019-02-23 12:48:39,867: DEBUG/MainProcess] TaskPool: Apply <function _fast_trace_task at 0x7f92462897b8> (args:('repro.ping', '372e0fca-0934-4260-9ac8-5cb5f3c8dfef', {'lang': 'py', 'task': 'repro.ping', 'id': '372e0fca-0934-4260-9ac8-5cb5f3c8dfef', 'shadow': None, 'eta': None, 'expires': None, 'group': None, 'retries': 0, 'timelimit': [None, None], 'root_id': '372e0fca-0934-4260-9ac8-5cb5f3c8dfef', 'parent_id': None, 'argsrepr': '(37,)', 'kwargsrepr': '{}', 'origin': 'gen24599@repro', 'reply_to': '13b4de4e-3451-330c-837d-91fcc7545757', 'correlation_id': '372e0fca-0934-4260-9ac8-5cb5f3c8dfef', 'delivery_info': {'exchange': '', 'routing_key': 'celery', 'priority': 0, 'redelivered': False}}, '[[37], {}, {"callbacks": null, "errbacks": null, "chain": null, "chord": null}]', 'application/json', 'utf-8') kwargs:{})
[2019-02-23 12:48:39,868: DEBUG/MainProcess] Task accepted: repro.ping[372e0fca-0934-4260-9ac8-5cb5f3c8dfef] pid:24589
[2019-02-23 12:48:39,870: INFO/MainProcess] Received task: repro.ping[d3ab0805-a4e6-4e57-a2e3-94e1fa957fa9]
[2019-02-23 12:48:39,871: INFO/ForkPoolWorker-1] Task repro.ping[5c5fb57e-3408-409c-8c8e-4af1eb8d6e0f] succeeded in 10.007380731985904s: 34
[2019-02-23 12:48:39,872: DEBUG/MainProcess] TaskPool: Apply <function _fast_trace_task at 0x7f92462897b8> (args:('repro.ping', '23920044-7453-426c-a804-8016859410c4', {'lang': 'py', 'task': 'repro.ping', 'id': '23920044-7453-426c-a804-8016859410c4', 'shadow': None, 'eta': None, 'expires': None, 'group': None, 'retries': 0, 'timelimit': [None, None], 'root_id': '23920044-7453-426c-a804-8016859410c4', 'parent_id': None, 'argsrepr': '(38,)', 'kwargsrepr': '{}', 'origin': 'gen24599@repro', 'reply_to': '13b4de4e-3451-330c-837d-91fcc7545757', 'correlation_id': '23920044-7453-426c-a804-8016859410c4', 'delivery_info': {'exchange': '', 'routing_key': 'celery', 'priority': 0, 'redelivered': False}}, '[[38], {}, {"callbacks": null, "errbacks": null, "chain": null, "chord": null}]', 'application/json', 'utf-8') kwargs:{})
[2019-02-23 12:48:39,873: DEBUG/MainProcess] Task accepted: repro.ping[23920044-7453-426c-a804-8016859410c4] pid:24587
[2019-02-23 12:48:39,874: INFO/MainProcess] Received task: repro.ping[97d308ed-27d9-4755-88fb-dee6c6ee13f0]
[2019-02-23 12:48:39,879: INFO/ForkPoolWorker-2] Task repro.ping[01dff44f-d3eb-4538-8f07-281e9733531b] succeeded in 10.009314763941802s: 35
[2019-02-23 12:48:39,880: DEBUG/MainProcess] TaskPool: Apply <function _fast_trace_task at 0x7f92462897b8> (args:('repro.ping', '0b8c91fe-c9a8-49bc-b857-f04f12692fec', {'lang': 'py', 'task': 'repro.ping', 'id': '0b8c91fe-c9a8-49bc-b857-f04f12692fec', 'shadow': None, 'eta': None, 'expires': None, 'group': None, 'retries': 0, 'timelimit': [None, None], 'root_id': '0b8c91fe-c9a8-49bc-b857-f04f12692fec', 'parent_id': None, 'argsrepr': '(39,)', 'kwargsrepr': '{}', 'origin': 'gen24599@repro', 'reply_to': '13b4de4e-3451-330c-837d-91fcc7545757', 'correlation_id': '0b8c91fe-c9a8-49bc-b857-f04f12692fec', 'delivery_info': {'exchange': '', 'routing_key': 'celery', 'priority': 0, 'redelivered': False}}, '[[39], {}, {"callbacks": null, "errbacks": null, "chain": null, "chord": null}]', 'application/json', 'utf-8') kwargs:{})
[2019-02-23 12:48:39,881: DEBUG/MainProcess] Task accepted: repro.ping[0b8c91fe-c9a8-49bc-b857-f04f12692fec] pid:24588
[2019-02-23 12:48:39,883: INFO/MainProcess] Received task: repro.ping[7df6a52e-4f4d-4610-94eb-464ef0b4c0e6]
[2019-02-23 12:48:45,873: DEBUG/MainProcess] heartbeat_tick : for connection c96dac8c7b174cb3af302fd13ce1db55
[2019-02-23 12:48:45,873: DEBUG/MainProcess] heartbeat_tick : Prev sent/recv: 60/293, now - 68/347, monotonic - 828050.413444373, last_heartbeat_sent - 828050.413440785, heartbeat int. - 60 for connection c96dac8c7b174cb3af302fd13ce1db55
[2019-02-23 12:48:49,867: INFO/ForkPoolWorker-4] Task repro.ping[9ac70541-1428-4877-a1a7-ca4d9196ea76] succeeded in 10.011515118996613s: 36
[2019-02-23 12:48:49,868: DEBUG/MainProcess] TaskPool: Apply <function _fast_trace_task at 0x7f92462897b8> (args:('repro.ping', '8b919229-36ab-4387-b8ff-446e461e462e', {'lang': 'py', 'task': 'repro.ping', 'id': '8b919229-36ab-4387-b8ff-446e461e462e', 'shadow': None, 'eta': None, 'expires': None, 'group': None, 'retries': 0, 'timelimit': [None, None], 'root_id': '8b919229-36ab-4387-b8ff-446e461e462e', 'parent_id': None, 'argsrepr': '(40,)', 'kwargsrepr': '{}', 'origin': 'gen24599@repro', 'reply_to': '13b4de4e-3451-330c-837d-91fcc7545757', 'correlation_id': '8b919229-36ab-4387-b8ff-446e461e462e', 'delivery_info': {'exchange': '', 'routing_key': 'celery', 'priority': 0, 'redelivered': False}}, '[[40], {}, {"callbacks": null, "errbacks": null, "chain": null, "chord": null}]', 'application/json', 'utf-8') kwargs:{})
[2019-02-23 12:48:49,870: DEBUG/MainProcess] Task accepted: repro.ping[8b919229-36ab-4387-b8ff-446e461e462e] pid:24590
[2019-02-23 12:48:49,872: INFO/MainProcess] Received task: repro.ping[6f6f6073-a6de-4a99-be5d-625ac6c31bd8]
[2019-02-23 12:48:49,878: INFO/ForkPoolWorker-3] Task repro.ping[372e0fca-0934-4260-9ac8-5cb5f3c8dfef] succeeded in 10.009640446980484s: 37
[2019-02-23 12:48:49,879: DEBUG/MainProcess] TaskPool: Apply <function _fast_trace_task at 0x7f92462897b8> (args:('repro.ping', '66105ea1-c574-4838-b297-cd1f861ac2fd', {'lang': 'py', 'task': 'repro.ping', 'id': '66105ea1-c574-4838-b297-cd1f861ac2fd', 'shadow': None, 'eta': None, 'expires': None, 'group': None, 'retries': 0, 'timelimit': [None, None], 'root_id': '66105ea1-c574-4838-b297-cd1f861ac2fd', 'parent_id': None, 'argsrepr': '(41,)', 'kwargsrepr': '{}', 'origin': 'gen24599@repro', 'reply_to': '13b4de4e-3451-330c-837d-91fcc7545757', 'correlation_id': '66105ea1-c574-4838-b297-cd1f861ac2fd', 'delivery_info': {'exchange': '', 'routing_key': 'celery', 'priority': 0, 'redelivered': False}}, '[[41], {}, {"callbacks": null, "errbacks": null, "chain": null, "chord": null}]', 'application/json', 'utf-8') kwargs:{})
[2019-02-23 12:48:49,880: DEBUG/MainProcess] Task accepted: repro.ping[66105ea1-c574-4838-b297-cd1f861ac2fd] pid:24589
[2019-02-23 12:48:49,883: INFO/MainProcess] Received task: repro.ping[19060c06-4928-4f63-88af-f7bdfacdb948]
[2019-02-23 12:48:49,884: INFO/ForkPoolWorker-1] Task repro.ping[23920044-7453-426c-a804-8016859410c4] succeeded in 10.01139860006515s: 38
[2019-02-23 12:48:49,885: DEBUG/MainProcess] TaskPool: Apply <function _fast_trace_task at 0x7f92462897b8> (args:('repro.ping', '322e398a-d9a2-486a-90d4-0012f0f1d804', {'lang': 'py', 'task': 'repro.ping', 'id': '322e398a-d9a2-486a-90d4-0012f0f1d804', 'shadow': None, 'eta': None, 'expires': None, 'group': None, 'retries': 0, 'timelimit': [None, None], 'root_id': '322e398a-d9a2-486a-90d4-0012f0f1d804', 'parent_id': None, 'argsrepr': '(42,)', 'kwargsrepr': '{}', 'origin': 'gen24599@repro', 'reply_to': '13b4de4e-3451-330c-837d-91fcc7545757', 'correlation_id': '322e398a-d9a2-486a-90d4-0012f0f1d804', 'delivery_info': {'exchange': '', 'routing_key': 'celery', 'priority': 0, 'redelivered': False}}, '[[42], {}, {"callbacks": null, "errbacks": null, "chain": null, "chord": null}]', 'application/json', 'utf-8') kwargs:{})
[2019-02-23 12:48:49,886: DEBUG/MainProcess] Task accepted: repro.ping[322e398a-d9a2-486a-90d4-0012f0f1d804] pid:24587
[2019-02-23 12:48:49,888: INFO/ForkPoolWorker-2] Task repro.ping[0b8c91fe-c9a8-49bc-b857-f04f12692fec] succeeded in 10.007518506026827s: 39
[2019-02-23 12:48:49,889: INFO/MainProcess] Received task: repro.ping[6fe4fe91-b320-40f0-ba71-45b013e71837]
[2019-02-23 12:48:49,890: DEBUG/MainProcess] TaskPool: Apply <function _fast_trace_task at 0x7f92462897b8> (args:('repro.ping', 'd37541c2-81b1-429c-87b3-842d5bf76242', {'lang': 'py', 'task': 'repro.ping', 'id': 'd37541c2-81b1-429c-87b3-842d5bf76242', 'shadow': None, 'eta': None, 'expires': None, 'group': None, 'retries': 0, 'timelimit': [None, None], 'root_id': 'd37541c2-81b1-429c-87b3-842d5bf76242', 'parent_id': None, 'argsrepr': '(43,)', 'kwargsrepr': '{}', 'origin': 'gen24599@repro', 'reply_to': '13b4de4e-3451-330c-837d-91fcc7545757', 'correlation_id': 'd37541c2-81b1-429c-87b3-842d5bf76242', 'delivery_info': {'exchange': '', 'routing_key': 'celery', 'priority': 0, 'redelivered': False}}, '[[43], {}, {"callbacks": null, "errbacks": null, "chain": null, "chord": null}]', 'application/json', 'utf-8') kwargs:{})
[2019-02-23 12:48:49,891: DEBUG/MainProcess] Task accepted: repro.ping[d37541c2-81b1-429c-87b3-842d5bf76242] pid:24588
[2019-02-23 12:48:49,894: INFO/MainProcess] Received task: repro.ping[18be5fb9-0d29-405d-a0db-2d38028d1a2a]
[2019-02-23 12:48:59,881: INFO/ForkPoolWorker-4] Task repro.ping[8b919229-36ab-4387-b8ff-446e461e462e] succeeded in 10.011438557994552s: 40
[2019-02-23 12:48:59,882: DEBUG/MainProcess] TaskPool: Apply <function _fast_trace_task at 0x7f92462897b8> (args:('repro.ping', 'ba330fe7-2223-4581-b803-6ba957c30f56', {'lang': 'py', 'task': 'repro.ping', 'id': 'ba330fe7-2223-4581-b803-6ba957c30f56', 'shadow': None, 'eta': None, 'expires': None, 'group': None, 'retries': 0, 'timelimit': [None, None], 'root_id': 'ba330fe7-2223-4581-b803-6ba957c30f56', 'parent_id': None, 'argsrepr': '(44,)', 'kwargsrepr': '{}', 'origin': 'gen24599@repro', 'reply_to': '13b4de4e-3451-330c-837d-91fcc7545757', 'correlation_id': 'ba330fe7-2223-4581-b803-6ba957c30f56', 'delivery_info': {'exchange': '', 'routing_key': 'celery', 'priority': 0, 'redelivered': False}}, '[[44], {}, {"callbacks": null, "errbacks": null, "chain": null, "chord": null}]', 'application/json', 'utf-8') kwargs:{})
[2019-02-23 12:48:59,884: DEBUG/MainProcess] Task accepted: repro.ping[ba330fe7-2223-4581-b803-6ba957c30f56] pid:24590
[2019-02-23 12:48:59,887: INFO/MainProcess] Received task: repro.ping[f793c1c3-f8eb-411b-bf20-50844468c3a4]
[2019-02-23 12:48:59,890: INFO/ForkPoolWorker-3] Task repro.ping[66105ea1-c574-4838-b297-cd1f861ac2fd] succeeded in 10.009387177997269s: 41
[2019-02-23 12:48:59,892: DEBUG/MainProcess] TaskPool: Apply <function _fast_trace_task at 0x7f92462897b8> (args:('repro.ping', '73383b31-c16d-442a-b35e-51dcbd248c64', {'lang': 'py', 'task': 'repro.ping', 'id': '73383b31-c16d-442a-b35e-51dcbd248c64', 'shadow': None, 'eta': None, 'expires': None, 'group': None, 'retries': 0, 'timelimit': [None, None], 'root_id': '73383b31-c16d-442a-b35e-51dcbd248c64', 'parent_id': None, 'argsrepr': '(45,)', 'kwargsrepr': '{}', 'origin': 'gen24599@repro', 'reply_to': '13b4de4e-3451-330c-837d-91fcc7545757', 'correlation_id': '73383b31-c16d-442a-b35e-51dcbd248c64', 'delivery_info': {'exchange': '', 'routing_key': 'celery', 'priority': 0, 'redelivered': False}}, '[[45], {}, {"callbacks": null, "errbacks": null, "chain": null, "chord": null}]', 'application/json', 'utf-8') kwargs:{})
[2019-02-23 12:48:59,894: DEBUG/MainProcess] Task accepted: repro.ping[73383b31-c16d-442a-b35e-51dcbd248c64] pid:24589
[2019-02-23 12:48:59,897: INFO/MainProcess] Received task: repro.ping[ea1ae0a8-800e-4da4-9042-bb32994a75a9]
[2019-02-23 12:48:59,900: INFO/ForkPoolWorker-2] Task repro.ping[d37541c2-81b1-429c-87b3-842d5bf76242] succeeded in 10.009441644069739s: 43
[2019-02-23 12:48:59,901: INFO/ForkPoolWorker-1] Task repro.ping[322e398a-d9a2-486a-90d4-0012f0f1d804] succeeded in 10.015280571999028s: 42
[2019-02-23 12:48:59,904: DEBUG/MainProcess] TaskPool: Apply <function _fast_trace_task at 0x7f92462897b8> (args:('repro.ping', 'f71cd3c6-eeb3-424f-8a8a-a9830fed5a49', {'lang': 'py', 'task': 'repro.ping', 'id': 'f71cd3c6-eeb3-424f-8a8a-a9830fed5a49', 'shadow': None, 'eta': None, 'expires': None, 'group': None, 'retries': 0, 'timelimit': [None, None], 'root_id': 'f71cd3c6-eeb3-424f-8a8a-a9830fed5a49', 'parent_id': None, 'argsrepr': '(46,)', 'kwargsrepr': '{}', 'origin': 'gen24599@repro', 'reply_to': '13b4de4e-3451-330c-837d-91fcc7545757', 'correlation_id': 'f71cd3c6-eeb3-424f-8a8a-a9830fed5a49', 'delivery_info': {'exchange': '', 'routing_key': 'celery', 'priority': 0, 'redelivered': False}}, '[[46], {}, {"callbacks": null, "errbacks": null, "chain": null, "chord": null}]', 'application/json', 'utf-8') kwargs:{})
[2019-02-23 12:48:59,905: DEBUG/MainProcess] TaskPool: Apply <function _fast_trace_task at 0x7f92462897b8> (args:('repro.ping', 'c5e658bb-43d9-4caa-99a6-63c9b5616b06', {'lang': 'py', 'task': 'repro.ping', 'id': 'c5e658bb-43d9-4caa-99a6-63c9b5616b06', 'shadow': None, 'eta': None, 'expires': None, 'group': None, 'retries': 0, 'timelimit': [None, None], 'root_id': 'c5e658bb-43d9-4caa-99a6-63c9b5616b06', 'parent_id': None, 'argsrepr': '(47,)', 'kwargsrepr': '{}', 'origin': 'gen24599@repro', 'reply_to': '13b4de4e-3451-330c-837d-91fcc7545757', 'correlation_id': 'c5e658bb-43d9-4caa-99a6-63c9b5616b06', 'delivery_info': {'exchange': '', 'routing_key': 'celery', 'priority': 0, 'redelivered': False}}, '[[47], {}, {"callbacks": null, "errbacks": null, "chain": null, "chord": null}]', 'application/json', 'utf-8') kwargs:{})
[2019-02-23 12:48:59,907: DEBUG/MainProcess] Task accepted: repro.ping[f71cd3c6-eeb3-424f-8a8a-a9830fed5a49] pid:24587
[2019-02-23 12:48:59,908: DEBUG/MainProcess] Task accepted: repro.ping[c5e658bb-43d9-4caa-99a6-63c9b5616b06] pid:24588
[2019-02-23 12:48:59,911: INFO/MainProcess] Received task: repro.ping[5e3de42f-a8b8-4674-98ee-c3c1242102b5]
[2019-02-23 12:48:59,913: INFO/MainProcess] Received task: repro.ping[57aa3cec-d384-4d90-a96c-a8b02afb0054]
[2019-02-23 12:49:05,876: DEBUG/MainProcess] heartbeat_tick : for connection c96dac8c7b174cb3af302fd13ce1db55
[2019-02-23 12:49:05,876: DEBUG/MainProcess] heartbeat_tick : Prev sent/recv: 68/347, now - 76/401, monotonic - 828070.416255205, last_heartbeat_sent - 828070.416251962, heartbeat int. - 60 for connection c96dac8c7b174cb3af302fd13ce1db55
[2019-02-23 12:49:09,892: INFO/ForkPoolWorker-4] Task repro.ping[ba330fe7-2223-4581-b803-6ba957c30f56] succeeded in 10.008885035058483s: 44
[2019-02-23 12:49:09,894: DEBUG/MainProcess] TaskPool: Apply <function _fast_trace_task at 0x7f92462897b8> (args:('repro.ping', 'bb2b74b4-1385-4321-8f8a-685dd1198873', {'lang': 'py', 'task': 'repro.ping', 'id': 'bb2b74b4-1385-4321-8f8a-685dd1198873', 'shadow': None, 'eta': None, 'expires': None, 'group': None, 'retries': 0, 'timelimit': [None, None], 'root_id': 'bb2b74b4-1385-4321-8f8a-685dd1198873', 'parent_id': None, 'argsrepr': '(48,)', 'kwargsrepr': '{}', 'origin': 'gen24599@repro', 'reply_to': '13b4de4e-3451-330c-837d-91fcc7545757', 'correlation_id': 'bb2b74b4-1385-4321-8f8a-685dd1198873', 'delivery_info': {'exchange': '', 'routing_key': 'celery', 'priority': 0, 'redelivered': False}}, '[[48], {}, {"callbacks": null, "errbacks": null, "chain": null, "chord": null}]', 'application/json', 'utf-8') kwargs:{})
[2019-02-23 12:49:09,895: DEBUG/MainProcess] Task accepted: repro.ping[bb2b74b4-1385-4321-8f8a-685dd1198873] pid:24590
[2019-02-23 12:49:09,899: INFO/MainProcess] Received task: repro.ping[3ddc421f-a045-4c9e-bbcd-9181ee22103f]
[2019-02-23 12:49:09,904: INFO/ForkPoolWorker-3] Task repro.ping[73383b31-c16d-442a-b35e-51dcbd248c64] succeeded in 10.0099140930688s: 45
[2019-02-23 12:49:09,905: DEBUG/MainProcess] TaskPool: Apply <function _fast_trace_task at 0x7f92462897b8> (args:('repro.ping', 'e8d7d0fe-700d-4b55-b3ef-422265f142a1', {'lang': 'py', 'task': 'repro.ping', 'id': 'e8d7d0fe-700d-4b55-b3ef-422265f142a1', 'shadow': None, 'eta': None, 'expires': None, 'group': None, 'retries': 0, 'timelimit': [None, None], 'root_id': 'e8d7d0fe-700d-4b55-b3ef-422265f142a1', 'parent_id': None, 'argsrepr': '(49,)', 'kwargsrepr': '{}', 'origin': 'gen24599@repro', 'reply_to': '13b4de4e-3451-330c-837d-91fcc7545757', 'correlation_id': 'e8d7d0fe-700d-4b55-b3ef-422265f142a1', 'delivery_info': {'exchange': '', 'routing_key': 'celery', 'priority': 0, 'redelivered': False}}, '[[49], {}, {"callbacks": null, "errbacks": null, "chain": null, "chord": null}]', 'application/json', 'utf-8') kwargs:{})
[2019-02-23 12:49:09,907: DEBUG/MainProcess] Task accepted: repro.ping[e8d7d0fe-700d-4b55-b3ef-422265f142a1] pid:24589
[2019-02-23 12:49:09,910: INFO/MainProcess] Received task: repro.ping[b15e95fe-17a1-4f86-a88e-686ec768e4fa]
[2019-02-23 12:49:09,916: INFO/ForkPoolWorker-1] Task repro.ping[f71cd3c6-eeb3-424f-8a8a-a9830fed5a49] succeeded in 10.009642486926168s: 46
[2019-02-23 12:49:09,917: DEBUG/MainProcess] TaskPool: Apply <function _fast_trace_task at 0x7f92462897b8> (args:('repro.ping', '1a663699-ca2a-4ae1-a9ab-01094ad98e0d', {'lang': 'py', 'task': 'repro.ping', 'id': '1a663699-ca2a-4ae1-a9ab-01094ad98e0d', 'shadow': None, 'eta': None, 'expires': None, 'group': None, 'retries': 0, 'timelimit': [None, None], 'root_id': '1a663699-ca2a-4ae1-a9ab-01094ad98e0d', 'parent_id': None, 'argsrepr': '(50,)', 'kwargsrepr': '{}', 'origin': 'gen24599@repro', 'reply_to': '13b4de4e-3451-330c-837d-91fcc7545757', 'correlation_id': '1a663699-ca2a-4ae1-a9ab-01094ad98e0d', 'delivery_info': {'exchange': '', 'routing_key': 'celery', 'priority': 0, 'redelivered': False}}, '[[50], {}, {"callbacks": null, "errbacks": null, "chain": null, "chord": null}]', 'application/json', 'utf-8') kwargs:{})
[2019-02-23 12:49:09,918: INFO/ForkPoolWorker-2] Task repro.ping[c5e658bb-43d9-4caa-99a6-63c9b5616b06] succeeded in 10.011810526018962s: 47
[2019-02-23 12:49:09,919: DEBUG/MainProcess] Task accepted: repro.ping[1a663699-ca2a-4ae1-a9ab-01094ad98e0d] pid:24587
[2019-02-23 12:49:09,920: DEBUG/MainProcess] TaskPool: Apply <function _fast_trace_task at 0x7f92462897b8> (args:('repro.ping', '2c529c3a-2a82-45ce-bbf9-6b305a425efa', {'lang': 'py', 'task': 'repro.ping', 'id': '2c529c3a-2a82-45ce-bbf9-6b305a425efa', 'shadow': None, 'eta': None, 'expires': None, 'group': None, 'retries': 0, 'timelimit': [None, None], 'root_id': '2c529c3a-2a82-45ce-bbf9-6b305a425efa', 'parent_id': None, 'argsrepr': '(51,)', 'kwargsrepr': '{}', 'origin': 'gen24599@repro', 'reply_to': '13b4de4e-3451-330c-837d-91fcc7545757', 'correlation_id': '2c529c3a-2a82-45ce-bbf9-6b305a425efa', 'delivery_info': {'exchange': '', 'routing_key': 'celery', 'priority': 0, 'redelivered': False}}, '[[51], {}, {"callbacks": null, "errbacks": null, "chain": null, "chord": null}]', 'application/json', 'utf-8') kwargs:{})
[2019-02-23 12:49:09,922: DEBUG/MainProcess] Task accepted: repro.ping[2c529c3a-2a82-45ce-bbf9-6b305a425efa] pid:24588
[2019-02-23 12:49:09,924: INFO/MainProcess] Received task: repro.ping[a7795001-72f4-459a-aa67-0f8ff84b8c98]
[2019-02-23 12:49:09,926: INFO/MainProcess] Received task: repro.ping[19b79375-25c4-44f9-a1b8-8eeed0ca2718]
[2019-02-23 12:49:19,903: INFO/ForkPoolWorker-4] Task repro.ping[bb2b74b4-1385-4321-8f8a-685dd1198873] succeeded in 10.007830242044292s: 48
[2019-02-23 12:49:19,904: DEBUG/MainProcess] TaskPool: Apply <function _fast_trace_task at 0x7f92462897b8> (args:('repro.ping', 'f3835161-e5d0-478a-87fc-b48f4f8ca51a', {'lang': 'py', 'task': 'repro.ping', 'id': 'f3835161-e5d0-478a-87fc-b48f4f8ca51a', 'shadow': None, 'eta': None, 'expires': None, 'group': None, 'retries': 0, 'timelimit': [None, None], 'root_id': 'f3835161-e5d0-478a-87fc-b48f4f8ca51a', 'parent_id': None, 'argsrepr': '(52,)', 'kwargsrepr': '{}', 'origin': 'gen24599@repro', 'reply_to': '13b4de4e-3451-330c-837d-91fcc7545757', 'correlation_id': 'f3835161-e5d0-478a-87fc-b48f4f8ca51a', 'delivery_info': {'exchange': '', 'routing_key': 'celery', 'priority': 0, 'redelivered': False}}, '[[52], {}, {"callbacks": null, "errbacks": null, "chain": null, "chord": null}]', 'application/json', 'utf-8') kwargs:{})
[2019-02-23 12:49:19,904: DEBUG/MainProcess] Task accepted: repro.ping[f3835161-e5d0-478a-87fc-b48f4f8ca51a] pid:24590
[2019-02-23 12:49:19,906: INFO/MainProcess] Received task: repro.ping[99f3cc1d-580f-426e-b95a-a36a136fddd1]
[2019-02-23 12:49:19,915: INFO/ForkPoolWorker-3] Task repro.ping[e8d7d0fe-700d-4b55-b3ef-422265f142a1] succeeded in 10.008701610029675s: 49
[2019-02-23 12:49:19,915: DEBUG/MainProcess] TaskPool: Apply <function _fast_trace_task at 0x7f92462897b8> (args:('repro.ping', 'd3ab0805-a4e6-4e57-a2e3-94e1fa957fa9', {'lang': 'py', 'task': 'repro.ping', 'id': 'd3ab0805-a4e6-4e57-a2e3-94e1fa957fa9', 'shadow': None, 'eta': None, 'expires': None, 'group': None, 'retries': 0, 'timelimit': [None, None], 'root_id': 'd3ab0805-a4e6-4e57-a2e3-94e1fa957fa9', 'parent_id': None, 'argsrepr': '(53,)', 'kwargsrepr': '{}', 'origin': 'gen24599@repro', 'reply_to': '13b4de4e-3451-330c-837d-91fcc7545757', 'correlation_id': 'd3ab0805-a4e6-4e57-a2e3-94e1fa957fa9', 'delivery_info': {'exchange': '', 'routing_key': 'celery', 'priority': 0, 'redelivered': False}}, '[[53], {}, {"callbacks": null, "errbacks": null, "chain": null, "chord": null}]', 'application/json', 'utf-8') kwargs:{})
[2019-02-23 12:49:19,916: DEBUG/MainProcess] Task accepted: repro.ping[d3ab0805-a4e6-4e57-a2e3-94e1fa957fa9] pid:24589
[2019-02-23 12:49:19,918: INFO/MainProcess] Received task: repro.ping[fc54e4cd-65c6-4a00-a580-4c09feb1186e]
[2019-02-23 12:49:19,927: INFO/ForkPoolWorker-1] Task repro.ping[1a663699-ca2a-4ae1-a9ab-01094ad98e0d] succeeded in 10.008626613067463s: 50
[2019-02-23 12:49:19,928: DEBUG/MainProcess] TaskPool: Apply <function _fast_trace_task at 0x7f92462897b8> (args:('repro.ping', '97d308ed-27d9-4755-88fb-dee6c6ee13f0', {'lang': 'py', 'task': 'repro.ping', 'id': '97d308ed-27d9-4755-88fb-dee6c6ee13f0', 'shadow': None, 'eta': None, 'expires': None, 'group': None, 'retries': 0, 'timelimit': [None, None], 'root_id': '97d308ed-27d9-4755-88fb-dee6c6ee13f0', 'parent_id': None, 'argsrepr': '(54,)', 'kwargsrepr': '{}', 'origin': 'gen24599@repro', 'reply_to': '13b4de4e-3451-330c-837d-91fcc7545757', 'correlation_id': '97d308ed-27d9-4755-88fb-dee6c6ee13f0', 'delivery_info': {'exchange': '', 'routing_key': 'celery', 'priority': 0, 'redelivered': False}}, '[[54], {}, {"callbacks": null, "errbacks": null, "chain": null, "chord": null}]', 'application/json', 'utf-8') kwargs:{})
[2019-02-23 12:49:19,928: DEBUG/MainProcess] Task accepted: repro.ping[97d308ed-27d9-4755-88fb-dee6c6ee13f0] pid:24587
[2019-02-23 12:49:19,929: INFO/MainProcess] Received task: repro.ping[18d236e6-81b9-4031-b5b5-0166822e03c5]
[2019-02-23 12:49:19,931: INFO/ForkPoolWorker-2] Task repro.ping[2c529c3a-2a82-45ce-bbf9-6b305a425efa] succeeded in 10.009170565987006s: 51
[2019-02-23 12:49:19,931: DEBUG/MainProcess] TaskPool: Apply <function _fast_trace_task at 0x7f92462897b8> (args:('repro.ping', '7df6a52e-4f4d-4610-94eb-464ef0b4c0e6', {'lang': 'py', 'task': 'repro.ping', 'id': '7df6a52e-4f4d-4610-94eb-464ef0b4c0e6', 'shadow': None, 'eta': None, 'expires': None, 'group': None, 'retries': 0, 'timelimit': [None, None], 'root_id': '7df6a52e-4f4d-4610-94eb-464ef0b4c0e6', 'parent_id': None, 'argsrepr': '(55,)', 'kwargsrepr': '{}', 'origin': 'gen24599@repro', 'reply_to': '13b4de4e-3451-330c-837d-91fcc7545757', 'correlation_id': '7df6a52e-4f4d-4610-94eb-464ef0b4c0e6', 'delivery_info': {'exchange': '', 'routing_key': 'celery', 'priority': 0, 'redelivered': False}}, '[[55], {}, {"callbacks": null, "errbacks": null, "chain": null, "chord": null}]', 'application/json', 'utf-8') kwargs:{})
[2019-02-23 12:49:19,932: DEBUG/MainProcess] Task accepted: repro.ping[7df6a52e-4f4d-4610-94eb-464ef0b4c0e6] pid:24588
[2019-02-23 12:49:19,933: INFO/MainProcess] Received task: repro.ping[f7b10df0-5050-4cda-b2a1-6752c6d33002]
[2019-02-23 12:49:25,878: DEBUG/MainProcess] heartbeat_tick : for connection c96dac8c7b174cb3af302fd13ce1db55
[2019-02-23 12:49:25,878: DEBUG/MainProcess] heartbeat_tick : Prev sent/recv: 76/401, now - 84/455, monotonic - 828090.418472588, last_heartbeat_sent - 828090.418469348, heartbeat int. - 60 for connection c96dac8c7b174cb3af302fd13ce1db55
[2019-02-23 12:49:29,915: INFO/ForkPoolWorker-4] Task repro.ping[f3835161-e5d0-478a-87fc-b48f4f8ca51a] succeeded in 10.011374757043086s: 52
[2019-02-23 12:49:29,916: DEBUG/MainProcess] TaskPool: Apply <function _fast_trace_task at 0x7f92462897b8> (args:('repro.ping', '6f6f6073-a6de-4a99-be5d-625ac6c31bd8', {'lang': 'py', 'task': 'repro.ping', 'id': '6f6f6073-a6de-4a99-be5d-625ac6c31bd8', 'shadow': None, 'eta': None, 'expires': None, 'group': None, 'retries': 0, 'timelimit': [None, None], 'root_id': '6f6f6073-a6de-4a99-be5d-625ac6c31bd8', 'parent_id': None, 'argsrepr': '(56,)', 'kwargsrepr': '{}', 'origin': 'gen24599@repro', 'reply_to': '13b4de4e-3451-330c-837d-91fcc7545757', 'correlation_id': '6f6f6073-a6de-4a99-be5d-625ac6c31bd8', 'delivery_info': {'exchange': '', 'routing_key': 'celery', 'priority': 0, 'redelivered': False}}, '[[56], {}, {"callbacks": null, "errbacks": null, "chain": null, "chord": null}]', 'application/json', 'utf-8') kwargs:{})
[2019-02-23 12:49:29,918: DEBUG/MainProcess] Task accepted: repro.ping[6f6f6073-a6de-4a99-be5d-625ac6c31bd8] pid:24590
[2019-02-23 12:49:29,920: INFO/MainProcess] Received task: repro.ping[2b58a237-5dc0-44ad-99a1-70aa12361d02]
[2019-02-23 12:49:29,928: INFO/ForkPoolWorker-3] Task repro.ping[d3ab0805-a4e6-4e57-a2e3-94e1fa957fa9] succeeded in 10.012014006963s: 53
[2019-02-23 12:49:29,929: DEBUG/MainProcess] TaskPool: Apply <function _fast_trace_task at 0x7f92462897b8> (args:('repro.ping', '19060c06-4928-4f63-88af-f7bdfacdb948', {'lang': 'py', 'task': 'repro.ping', 'id': '19060c06-4928-4f63-88af-f7bdfacdb948', 'shadow': None, 'eta': None, 'expires': None, 'group': None, 'retries': 0, 'timelimit': [None, None], 'root_id': '19060c06-4928-4f63-88af-f7bdfacdb948', 'parent_id': None, 'argsrepr': '(57,)', 'kwargsrepr': '{}', 'origin': 'gen24599@repro', 'reply_to': '13b4de4e-3451-330c-837d-91fcc7545757', 'correlation_id': '19060c06-4928-4f63-88af-f7bdfacdb948', 'delivery_info': {'exchange': '', 'routing_key': 'celery', 'priority': 0, 'redelivered': False}}, '[[57], {}, {"callbacks": null, "errbacks": null, "chain": null, "chord": null}]', 'application/json', 'utf-8') kwargs:{})
[2019-02-23 12:49:29,930: DEBUG/MainProcess] Task accepted: repro.ping[19060c06-4928-4f63-88af-f7bdfacdb948] pid:24589
[2019-02-23 12:49:29,933: INFO/MainProcess] Received task: repro.ping[a9e5d083-2314-4d61-b9b3-040231a0f7aa]
[2019-02-23 12:49:29,939: INFO/ForkPoolWorker-1] Task repro.ping[97d308ed-27d9-4755-88fb-dee6c6ee13f0] succeeded in 10.011442068964243s: 54
[2019-02-23 12:49:29,940: DEBUG/MainProcess] TaskPool: Apply <function _fast_trace_task at 0x7f92462897b8> (args:('repro.ping', '6fe4fe91-b320-40f0-ba71-45b013e71837', {'lang': 'py', 'task': 'repro.ping', 'id': '6fe4fe91-b320-40f0-ba71-45b013e71837', 'shadow': None, 'eta': None, 'expires': None, 'group': None, 'retries': 0, 'timelimit': [None, None], 'root_id': '6fe4fe91-b320-40f0-ba71-45b013e71837', 'parent_id': None, 'argsrepr': '(58,)', 'kwargsrepr': '{}', 'origin': 'gen24599@repro', 'reply_to': '13b4de4e-3451-330c-837d-91fcc7545757', 'correlation_id': '6fe4fe91-b320-40f0-ba71-45b013e71837', 'delivery_info': {'exchange': '', 'routing_key': 'celery', 'priority': 0, 'redelivered': False}}, '[[58], {}, {"callbacks": null, "errbacks": null, "chain": null, "chord": null}]', 'application/json', 'utf-8') kwargs:{})
[2019-02-23 12:49:29,942: INFO/ForkPoolWorker-2] Task repro.ping[7df6a52e-4f4d-4610-94eb-464ef0b4c0e6] succeeded in 10.009457940002903s: 55
[2019-02-23 12:49:29,943: DEBUG/MainProcess] TaskPool: Apply <function _fast_trace_task at 0x7f92462897b8> (args:('repro.ping', '18be5fb9-0d29-405d-a0db-2d38028d1a2a', {'lang': 'py', 'task': 'repro.ping', 'id': '18be5fb9-0d29-405d-a0db-2d38028d1a2a', 'shadow': None, 'eta': None, 'expires': None, 'group': None, 'retries': 0, 'timelimit': [None, None], 'root_id': '18be5fb9-0d29-405d-a0db-2d38028d1a2a', 'parent_id': None, 'argsrepr': '(59,)', 'kwargsrepr': '{}', 'origin': 'gen24599@repro', 'reply_to': '13b4de4e-3451-330c-837d-91fcc7545757', 'correlation_id': '18be5fb9-0d29-405d-a0db-2d38028d1a2a', 'delivery_info': {'exchange': '', 'routing_key': 'celery', 'priority': 0, 'redelivered': False}}, '[[59], {}, {"callbacks": null, "errbacks": null, "chain": null, "chord": null}]', 'application/json', 'utf-8') kwargs:{})
[2019-02-23 12:49:29,943: DEBUG/MainProcess] Task accepted: repro.ping[6fe4fe91-b320-40f0-ba71-45b013e71837] pid:24587
[2019-02-23 12:49:29,946: INFO/MainProcess] Received task: repro.ping[55610225-f895-4d65-8c86-30115fe3b289]
[2019-02-23 12:49:29,947: DEBUG/MainProcess] Task accepted: repro.ping[18be5fb9-0d29-405d-a0db-2d38028d1a2a] pid:24588
[2019-02-23 12:49:29,949: INFO/MainProcess] Received task: repro.ping[41f062cc-8d89-4b89-882b-5a16df33c4cf]
[2019-02-23 12:49:39,928: INFO/ForkPoolWorker-4] Task repro.ping[6f6f6073-a6de-4a99-be5d-625ac6c31bd8] succeeded in 10.01031665992923s: 56
[2019-02-23 12:49:39,929: DEBUG/MainProcess] TaskPool: Apply <function _fast_trace_task at 0x7f92462897b8> (args:('repro.ping', 'f793c1c3-f8eb-411b-bf20-50844468c3a4', {'lang': 'py', 'task': 'repro.ping', 'id': 'f793c1c3-f8eb-411b-bf20-50844468c3a4', 'shadow': None, 'eta': None, 'expires': None, 'group': None, 'retries': 0, 'timelimit': [None, None], 'root_id': 'f793c1c3-f8eb-411b-bf20-50844468c3a4', 'parent_id': None, 'argsrepr': '(60,)', 'kwargsrepr': '{}', 'origin': 'gen24599@repro', 'reply_to': '13b4de4e-3451-330c-837d-91fcc7545757', 'correlation_id': 'f793c1c3-f8eb-411b-bf20-50844468c3a4', 'delivery_info': {'exchange': '', 'routing_key': 'celery', 'priority': 0, 'redelivered': False}}, '[[60], {}, {"callbacks": null, "errbacks": null, "chain": null, "chord": null}]', 'application/json', 'utf-8') kwargs:{})
[2019-02-23 12:49:39,930: DEBUG/MainProcess] Task accepted: repro.ping[f793c1c3-f8eb-411b-bf20-50844468c3a4] pid:24590
[2019-02-23 12:49:39,933: INFO/MainProcess] Received task: repro.ping[03218e58-eb5f-4c86-b465-9399bdc3bb9d]
[2019-02-23 12:49:39,941: INFO/ForkPoolWorker-3] Task repro.ping[19060c06-4928-4f63-88af-f7bdfacdb948] succeeded in 10.010959548992105s: 57
[2019-02-23 12:49:39,942: DEBUG/MainProcess] TaskPool: Apply <function _fast_trace_task at 0x7f92462897b8> (args:('repro.ping', 'ea1ae0a8-800e-4da4-9042-bb32994a75a9', {'lang': 'py', 'task': 'repro.ping', 'id': 'ea1ae0a8-800e-4da4-9042-bb32994a75a9', 'shadow': None, 'eta': None, 'expires': None, 'group': None, 'retries': 0, 'timelimit': [None, None], 'root_id': 'ea1ae0a8-800e-4da4-9042-bb32994a75a9', 'parent_id': None, 'argsrepr': '(61,)', 'kwargsrepr': '{}', 'origin': 'gen24599@repro', 'reply_to': '13b4de4e-3451-330c-837d-91fcc7545757', 'correlation_id': 'ea1ae0a8-800e-4da4-9042-bb32994a75a9', 'delivery_info': {'exchange': '', 'routing_key': 'celery', 'priority': 0, 'redelivered': False}}, '[[61], {}, {"callbacks": null, "errbacks": null, "chain": null, "chord": null}]', 'application/json', 'utf-8') kwargs:{})
[2019-02-23 12:49:39,943: DEBUG/MainProcess] Task accepted: repro.ping[ea1ae0a8-800e-4da4-9042-bb32994a75a9] pid:24589
[2019-02-23 12:49:39,946: INFO/MainProcess] Received task: repro.ping[76dee37c-4dd5-4d2a-810d-2f93726706d2]
[2019-02-23 12:49:39,952: INFO/ForkPoolWorker-1] Task repro.ping[6fe4fe91-b320-40f0-ba71-45b013e71837] succeeded in 10.009412415907718s: 58
[2019-02-23 12:49:39,953: DEBUG/MainProcess] TaskPool: Apply <function _fast_trace_task at 0x7f92462897b8> (args:('repro.ping', '5e3de42f-a8b8-4674-98ee-c3c1242102b5', {'lang': 'py', 'task': 'repro.ping', 'id': '5e3de42f-a8b8-4674-98ee-c3c1242102b5', 'shadow': None, 'eta': None, 'expires': None, 'group': None, 'retries': 0, 'timelimit': [None, None], 'root_id': '5e3de42f-a8b8-4674-98ee-c3c1242102b5', 'parent_id': None, 'argsrepr': '(62,)', 'kwargsrepr': '{}', 'origin': 'gen24599@repro', 'reply_to': '13b4de4e-3451-330c-837d-91fcc7545757', 'correlation_id': '5e3de42f-a8b8-4674-98ee-c3c1242102b5', 'delivery_info': {'exchange': '', 'routing_key': 'celery', 'priority': 0, 'redelivered': False}}, '[[62], {}, {"callbacks": null, "errbacks": null, "chain": null, "chord": null}]', 'application/json', 'utf-8') kwargs:{})
[2019-02-23 12:49:39,955: DEBUG/MainProcess] Task accepted: repro.ping[5e3de42f-a8b8-4674-98ee-c3c1242102b5] pid:24587
[2019-02-23 12:49:39,956: INFO/ForkPoolWorker-2] Task repro.ping[18be5fb9-0d29-405d-a0db-2d38028d1a2a] succeeded in 10.010617804946378s: 59
[2019-02-23 12:49:39,957: DEBUG/MainProcess] TaskPool: Apply <function _fast_trace_task at 0x7f92462897b8> (args:('repro.ping', '57aa3cec-d384-4d90-a96c-a8b02afb0054', {'lang': 'py', 'task': 'repro.ping', 'id': '57aa3cec-d384-4d90-a96c-a8b02afb0054', 'shadow': None, 'eta': None, 'expires': None, 'group': None, 'retries': 0, 'timelimit': [None, None], 'root_id': '57aa3cec-d384-4d90-a96c-a8b02afb0054', 'parent_id': None, 'argsrepr': '(63,)', 'kwargsrepr': '{}', 'origin': 'gen24599@repro', 'reply_to': '13b4de4e-3451-330c-837d-91fcc7545757', 'correlation_id': '57aa3cec-d384-4d90-a96c-a8b02afb0054', 'delivery_info': {'exchange': '', 'routing_key': 'celery', 'priority': 0, 'redelivered': False}}, '[[63], {}, {"callbacks": null, "errbacks": null, "chain": null, "chord": null}]', 'application/json', 'utf-8') kwargs:{})
[2019-02-23 12:49:39,959: INFO/MainProcess] Received task: repro.ping[19e75040-a721-44df-9c0f-32a79e51fb98]
[2019-02-23 12:49:39,960: DEBUG/MainProcess] Task accepted: repro.ping[57aa3cec-d384-4d90-a96c-a8b02afb0054] pid:24588
[2019-02-23 12:49:39,963: INFO/MainProcess] Received task: repro.ping[08edbae5-80e1-4f37-bb61-f55c6c6ef618]
[2019-02-23 12:49:45,881: DEBUG/MainProcess] heartbeat_tick : for connection c96dac8c7b174cb3af302fd13ce1db55
[2019-02-23 12:49:45,881: DEBUG/MainProcess] heartbeat_tick : Prev sent/recv: 84/455, now - 92/509, monotonic - 828110.421275947, last_heartbeat_sent - 828110.421272869, heartbeat int. - 60 for connection c96dac8c7b174cb3af302fd13ce1db55
[2019-02-23 12:49:49,942: INFO/ForkPoolWorker-4] Task repro.ping[f793c1c3-f8eb-411b-bf20-50844468c3a4] succeeded in 10.01146399998106s: 60
[2019-02-23 12:49:49,943: DEBUG/MainProcess] TaskPool: Apply <function _fast_trace_task at 0x7f92462897b8> (args:('repro.ping', '3ddc421f-a045-4c9e-bbcd-9181ee22103f', {'lang': 'py', 'task': 'repro.ping', 'id': '3ddc421f-a045-4c9e-bbcd-9181ee22103f', 'shadow': None, 'eta': None, 'expires': None, 'group': None, 'retries': 0, 'timelimit': [None, None], 'root_id': '3ddc421f-a045-4c9e-bbcd-9181ee22103f', 'parent_id': None, 'argsrepr': '(64,)', 'kwargsrepr': '{}', 'origin': 'gen24599@repro', 'reply_to': '13b4de4e-3451-330c-837d-91fcc7545757', 'correlation_id': '3ddc421f-a045-4c9e-bbcd-9181ee22103f', 'delivery_info': {'exchange': '', 'routing_key': 'celery', 'priority': 0, 'redelivered': False}}, '[[64], {}, {"callbacks": null, "errbacks": null, "chain": null, "chord": null}]', 'application/json', 'utf-8') kwargs:{})
[2019-02-23 12:49:49,944: DEBUG/MainProcess] Task accepted: repro.ping[3ddc421f-a045-4c9e-bbcd-9181ee22103f] pid:24590
[2019-02-23 12:49:49,947: INFO/MainProcess] Received task: repro.ping[2328251e-1e29-49d9-8fdc-936a577b64ba]
[2019-02-23 12:49:49,952: INFO/ForkPoolWorker-3] Task repro.ping[ea1ae0a8-800e-4da4-9042-bb32994a75a9] succeeded in 10.008566718897782s: 61
[2019-02-23 12:49:49,953: DEBUG/MainProcess] TaskPool: Apply <function _fast_trace_task at 0x7f92462897b8> (args:('repro.ping', 'b15e95fe-17a1-4f86-a88e-686ec768e4fa', {'lang': 'py', 'task': 'repro.ping', 'id': 'b15e95fe-17a1-4f86-a88e-686ec768e4fa', 'shadow': None, 'eta': None, 'expires': None, 'group': None, 'retries': 0, 'timelimit': [None, None], 'root_id': 'b15e95fe-17a1-4f86-a88e-686ec768e4fa', 'parent_id': None, 'argsrepr': '(65,)', 'kwargsrepr': '{}', 'origin': 'gen24599@repro', 'reply_to': '13b4de4e-3451-330c-837d-91fcc7545757', 'correlation_id': 'b15e95fe-17a1-4f86-a88e-686ec768e4fa', 'delivery_info': {'exchange': '', 'routing_key': 'celery', 'priority': 0, 'redelivered': False}}, '[[65], {}, {"callbacks": null, "errbacks": null, "chain": null, "chord": null}]', 'application/json', 'utf-8') kwargs:{})
[2019-02-23 12:49:49,954: DEBUG/MainProcess] Task accepted: repro.ping[b15e95fe-17a1-4f86-a88e-686ec768e4fa] pid:24589
[2019-02-23 12:49:49,957: INFO/MainProcess] Received task: repro.ping[9d5dc543-cbe5-471f-9cdb-983e6a47ef0b]
[2019-02-23 12:49:49,964: INFO/ForkPoolWorker-1] Task repro.ping[5e3de42f-a8b8-4674-98ee-c3c1242102b5] succeeded in 10.00951830809936s: 62
[2019-02-23 12:49:49,965: DEBUG/MainProcess] TaskPool: Apply <function _fast_trace_task at 0x7f92462897b8> (args:('repro.ping', 'a7795001-72f4-459a-aa67-0f8ff84b8c98', {'lang': 'py', 'task': 'repro.ping', 'id': 'a7795001-72f4-459a-aa67-0f8ff84b8c98', 'shadow': None, 'eta': None, 'expires': None, 'group': None, 'retries': 0, 'timelimit': [None, None], 'root_id': 'a7795001-72f4-459a-aa67-0f8ff84b8c98', 'parent_id': None, 'argsrepr': '(66,)', 'kwargsrepr': '{}', 'origin': 'gen24599@repro', 'reply_to': '13b4de4e-3451-330c-837d-91fcc7545757', 'correlation_id': 'a7795001-72f4-459a-aa67-0f8ff84b8c98', 'delivery_info': {'exchange': '', 'routing_key': 'celery', 'priority': 0, 'redelivered': False}}, '[[66], {}, {"callbacks": null, "errbacks": null, "chain": null, "chord": null}]', 'application/json', 'utf-8') kwargs:{})
[2019-02-23 12:49:49,967: DEBUG/MainProcess] Task accepted: repro.ping[a7795001-72f4-459a-aa67-0f8ff84b8c98] pid:24587
[2019-02-23 12:49:49,968: INFO/ForkPoolWorker-2] Task repro.ping[57aa3cec-d384-4d90-a96c-a8b02afb0054] succeeded in 10.00838961894624s: 63
[2019-02-23 12:49:49,969: DEBUG/MainProcess] TaskPool: Apply <function _fast_trace_task at 0x7f92462897b8> (args:('repro.ping', '19b79375-25c4-44f9-a1b8-8eeed0ca2718', {'lang': 'py', 'task': 'repro.ping', 'id': '19b79375-25c4-44f9-a1b8-8eeed0ca2718', 'shadow': None, 'eta': None, 'expires': None, 'group': None, 'retries': 0, 'timelimit': [None, None], 'root_id': '19b79375-25c4-44f9-a1b8-8eeed0ca2718', 'parent_id': None, 'argsrepr': '(67,)', 'kwargsrepr': '{}', 'origin': 'gen24599@repro', 'reply_to': '13b4de4e-3451-330c-837d-91fcc7545757', 'correlation_id': '19b79375-25c4-44f9-a1b8-8eeed0ca2718', 'delivery_info': {'exchange': '', 'routing_key': 'celery', 'priority': 0, 'redelivered': False}}, '[[67], {}, {"callbacks": null, "errbacks": null, "chain": null, "chord": null}]', 'application/json', 'utf-8') kwargs:{})
[2019-02-23 12:49:49,971: INFO/MainProcess] Received task: repro.ping[8a8bec13-1de9-428a-aaa6-c292b8e9e97c]
[2019-02-23 12:49:49,972: DEBUG/MainProcess] Task accepted: repro.ping[19b79375-25c4-44f9-a1b8-8eeed0ca2718] pid:24588
[2019-02-23 12:49:49,974: INFO/MainProcess] Received task: repro.ping[6ddfa78d-b7a4-4197-a490-fb52e87be667]
[2019-02-23 12:49:59,952: INFO/ForkPoolWorker-4] Task repro.ping[3ddc421f-a045-4c9e-bbcd-9181ee22103f] succeeded in 10.00812103797216s: 64
[2019-02-23 12:49:59,953: DEBUG/MainProcess] TaskPool: Apply <function _fast_trace_task at 0x7f92462897b8> (args:('repro.ping', '99f3cc1d-580f-426e-b95a-a36a136fddd1', {'lang': 'py', 'task': 'repro.ping', 'id': '99f3cc1d-580f-426e-b95a-a36a136fddd1', 'shadow': None, 'eta': None, 'expires': None, 'group': None, 'retries': 0, 'timelimit': [None, None], 'root_id': '99f3cc1d-580f-426e-b95a-a36a136fddd1', 'parent_id': None, 'argsrepr': '(68,)', 'kwargsrepr': '{}', 'origin': 'gen24599@repro', 'reply_to': '13b4de4e-3451-330c-837d-91fcc7545757', 'correlation_id': '99f3cc1d-580f-426e-b95a-a36a136fddd1', 'delivery_info': {'exchange': '', 'routing_key': 'celery', 'priority': 0, 'redelivered': False}}, '[[68], {}, {"callbacks": null, "errbacks": null, "chain": null, "chord": null}]', 'application/json', 'utf-8') kwargs:{})
[2019-02-23 12:49:59,955: DEBUG/MainProcess] Task accepted: repro.ping[99f3cc1d-580f-426e-b95a-a36a136fddd1] pid:24590
[2019-02-23 12:49:59,957: INFO/MainProcess] Received task: repro.ping[20f83297-3bfe-4eb2-8484-2b2f0443dd46]
[2019-02-23 12:49:59,964: INFO/ForkPoolWorker-3] Task repro.ping[b15e95fe-17a1-4f86-a88e-686ec768e4fa] succeeded in 10.009902358986437s: 65
[2019-02-23 12:49:59,965: DEBUG/MainProcess] TaskPool: Apply <function _fast_trace_task at 0x7f92462897b8> (args:('repro.ping', 'fc54e4cd-65c6-4a00-a580-4c09feb1186e', {'lang': 'py', 'task': 'repro.ping', 'id': 'fc54e4cd-65c6-4a00-a580-4c09feb1186e', 'shadow': None, 'eta': None, 'expires': None, 'group': None, 'retries': 0, 'timelimit': [None, None], 'root_id': 'fc54e4cd-65c6-4a00-a580-4c09feb1186e', 'parent_id': None, 'argsrepr': '(69,)', 'kwargsrepr': '{}', 'origin': 'gen24599@repro', 'reply_to': '13b4de4e-3451-330c-837d-91fcc7545757', 'correlation_id': 'fc54e4cd-65c6-4a00-a580-4c09feb1186e', 'delivery_info': {'exchange': '', 'routing_key': 'celery', 'priority': 0, 'redelivered': False}}, '[[69], {}, {"callbacks": null, "errbacks": null, "chain": null, "chord": null}]', 'application/json', 'utf-8') kwargs:{})
[2019-02-23 12:49:59,966: DEBUG/MainProcess] Task accepted: repro.ping[fc54e4cd-65c6-4a00-a580-4c09feb1186e] pid:24589
[2019-02-23 12:49:59,969: INFO/MainProcess] Received task: repro.ping[f73b1e5b-9cec-4fdf-9e1d-ee02dc13716e]
[2019-02-23 12:49:59,976: INFO/ForkPoolWorker-1] Task repro.ping[a7795001-72f4-459a-aa67-0f8ff84b8c98] succeeded in 10.00928835105151s: 66
[2019-02-23 12:49:59,977: DEBUG/MainProcess] TaskPool: Apply <function _fast_trace_task at 0x7f92462897b8> (args:('repro.ping', '18d236e6-81b9-4031-b5b5-0166822e03c5', {'lang': 'py', 'task': 'repro.ping', 'id': '18d236e6-81b9-4031-b5b5-0166822e03c5', 'shadow': None, 'eta': None, 'expires': None, 'group': None, 'retries': 0, 'timelimit': [None, None], 'root_id': '18d236e6-81b9-4031-b5b5-0166822e03c5', 'parent_id': None, 'argsrepr': '(70,)', 'kwargsrepr': '{}', 'origin': 'gen24599@repro', 'reply_to': '13b4de4e-3451-330c-837d-91fcc7545757', 'correlation_id': '18d236e6-81b9-4031-b5b5-0166822e03c5', 'delivery_info': {'exchange': '', 'routing_key': 'celery', 'priority': 0, 'redelivered': False}}, '[[70], {}, {"callbacks": null, "errbacks": null, "chain": null, "chord": null}]', 'application/json', 'utf-8') kwargs:{})
[2019-02-23 12:49:59,978: DEBUG/MainProcess] Task accepted: repro.ping[18d236e6-81b9-4031-b5b5-0166822e03c5] pid:24587
[2019-02-23 12:49:59,980: INFO/ForkPoolWorker-2] Task repro.ping[19b79375-25c4-44f9-a1b8-8eeed0ca2718] succeeded in 10.008277541957796s: 67
[2019-02-23 12:49:59,981: INFO/MainProcess] Received task: repro.ping[cc639408-eda2-4743-ae2d-83a76d66691a]
[2019-02-23 12:49:59,982: DEBUG/MainProcess] TaskPool: Apply <function _fast_trace_task at 0x7f92462897b8> (args:('repro.ping', 'f7b10df0-5050-4cda-b2a1-6752c6d33002', {'lang': 'py', 'task': 'repro.ping', 'id': 'f7b10df0-5050-4cda-b2a1-6752c6d33002', 'shadow': None, 'eta': None, 'expires': None, 'group': None, 'retries': 0, 'timelimit': [None, None], 'root_id': 'f7b10df0-5050-4cda-b2a1-6752c6d33002', 'parent_id': None, 'argsrepr': '(71,)', 'kwargsrepr': '{}', 'origin': 'gen24599@repro', 'reply_to': '13b4de4e-3451-330c-837d-91fcc7545757', 'correlation_id': 'f7b10df0-5050-4cda-b2a1-6752c6d33002', 'delivery_info': {'exchange': '', 'routing_key': 'celery', 'priority': 0, 'redelivered': False}}, '[[71], {}, {"callbacks": null, "errbacks": null, "chain": null, "chord": null}]', 'application/json', 'utf-8') kwargs:{})
[2019-02-23 12:49:59,984: DEBUG/MainProcess] Task accepted: repro.ping[f7b10df0-5050-4cda-b2a1-6752c6d33002] pid:24588
[2019-02-23 12:49:59,986: INFO/MainProcess] Received task: repro.ping[82e68b10-48c8-4343-b1e1-83d65712681a]
[2019-02-23 12:50:05,883: DEBUG/MainProcess] heartbeat_tick : for connection c96dac8c7b174cb3af302fd13ce1db55
[2019-02-23 12:50:05,883: DEBUG/MainProcess] heartbeat_tick : Prev sent/recv: 92/509, now - 100/563, monotonic - 828130.423378809, last_heartbeat_sent - 828130.423375836, heartbeat int. - 60 for connection c96dac8c7b174cb3af302fd13ce1db55
[2019-02-23 12:50:09,966: INFO/ForkPoolWorker-4] Task repro.ping[99f3cc1d-580f-426e-b95a-a36a136fddd1] succeeded in 10.01196253998205s: 68
[2019-02-23 12:50:09,967: DEBUG/MainProcess] TaskPool: Apply <function _fast_trace_task at 0x7f92462897b8> (args:('repro.ping', '2b58a237-5dc0-44ad-99a1-70aa12361d02', {'lang': 'py', 'task': 'repro.ping', 'id': '2b58a237-5dc0-44ad-99a1-70aa12361d02', 'shadow': None, 'eta': None, 'expires': None, 'group': None, 'retries': 0, 'timelimit': [None, None], 'root_id': '2b58a237-5dc0-44ad-99a1-70aa12361d02', 'parent_id': None, 'argsrepr': '(72,)', 'kwargsrepr': '{}', 'origin': 'gen24599@repro', 'reply_to': '13b4de4e-3451-330c-837d-91fcc7545757', 'correlation_id': '2b58a237-5dc0-44ad-99a1-70aa12361d02', 'delivery_info': {'exchange': '', 'routing_key': 'celery', 'priority': 0, 'redelivered': False}}, '[[72], {}, {"callbacks": null, "errbacks": null, "chain": null, "chord": null}]', 'application/json', 'utf-8') kwargs:{})
[2019-02-23 12:50:09,969: DEBUG/MainProcess] Task accepted: repro.ping[2b58a237-5dc0-44ad-99a1-70aa12361d02] pid:24590
[2019-02-23 12:50:09,971: INFO/MainProcess] Received task: repro.ping[0e75e7fd-0925-4a7a-932d-aff67100a893]
[2019-02-23 12:50:09,976: INFO/ForkPoolWorker-3] Task repro.ping[fc54e4cd-65c6-4a00-a580-4c09feb1186e] succeeded in 10.009725094074383s: 69
[2019-02-23 12:50:09,976: DEBUG/MainProcess] TaskPool: Apply <function _fast_trace_task at 0x7f92462897b8> (args:('repro.ping', 'a9e5d083-2314-4d61-b9b3-040231a0f7aa', {'lang': 'py', 'task': 'repro.ping', 'id': 'a9e5d083-2314-4d61-b9b3-040231a0f7aa', 'shadow': None, 'eta': None, 'expires': None, 'group': None, 'retries': 0, 'timelimit': [None, None], 'root_id': 'a9e5d083-2314-4d61-b9b3-040231a0f7aa', 'parent_id': None, 'argsrepr': '(73,)', 'kwargsrepr': '{}', 'origin': 'gen24599@repro', 'reply_to': '13b4de4e-3451-330c-837d-91fcc7545757', 'correlation_id': 'a9e5d083-2314-4d61-b9b3-040231a0f7aa', 'delivery_info': {'exchange': '', 'routing_key': 'celery', 'priority': 0, 'redelivered': False}}, '[[73], {}, {"callbacks": null, "errbacks": null, "chain": null, "chord": null}]', 'application/json', 'utf-8') kwargs:{})
[2019-02-23 12:50:09,978: DEBUG/MainProcess] Task accepted: repro.ping[a9e5d083-2314-4d61-b9b3-040231a0f7aa] pid:24589
[2019-02-23 12:50:09,980: INFO/MainProcess] Received task: repro.ping[6f64bb8f-849d-43cd-8361-64946efd769b]
[2019-02-23 12:50:09,988: INFO/ForkPoolWorker-1] Task repro.ping[18d236e6-81b9-4031-b5b5-0166822e03c5] succeeded in 10.009856261895038s: 70
[2019-02-23 12:50:09,989: DEBUG/MainProcess] TaskPool: Apply <function _fast_trace_task at 0x7f92462897b8> (args:('repro.ping', '55610225-f895-4d65-8c86-30115fe3b289', {'lang': 'py', 'task': 'repro.ping', 'id': '55610225-f895-4d65-8c86-30115fe3b289', 'shadow': None, 'eta': None, 'expires': None, 'group': None, 'retries': 0, 'timelimit': [None, None], 'root_id': '55610225-f895-4d65-8c86-30115fe3b289', 'parent_id': None, 'argsrepr': '(74,)', 'kwargsrepr': '{}', 'origin': 'gen24599@repro', 'reply_to': '13b4de4e-3451-330c-837d-91fcc7545757', 'correlation_id': '55610225-f895-4d65-8c86-30115fe3b289', 'delivery_info': {'exchange': '', 'routing_key': 'celery', 'priority': 0, 'redelivered': False}}, '[[74], {}, {"callbacks": null, "errbacks": null, "chain": null, "chord": null}]', 'application/json', 'utf-8') kwargs:{})
[2019-02-23 12:50:09,992: INFO/ForkPoolWorker-2] Task repro.ping[f7b10df0-5050-4cda-b2a1-6752c6d33002] succeeded in 10.00843263999559s: 71
[2019-02-23 12:50:09,992: DEBUG/MainProcess] Task accepted: repro.ping[55610225-f895-4d65-8c86-30115fe3b289] pid:24587
[2019-02-23 12:50:09,993: DEBUG/MainProcess] TaskPool: Apply <function _fast_trace_task at 0x7f92462897b8> (args:('repro.ping', '41f062cc-8d89-4b89-882b-5a16df33c4cf', {'lang': 'py', 'task': 'repro.ping', 'id': '41f062cc-8d89-4b89-882b-5a16df33c4cf', 'shadow': None, 'eta': None, 'expires': None, 'group': None, 'retries': 0, 'timelimit': [None, None], 'root_id': '41f062cc-8d89-4b89-882b-5a16df33c4cf', 'parent_id': None, 'argsrepr': '(75,)', 'kwargsrepr': '{}', 'origin': 'gen24599@repro', 'reply_to': '13b4de4e-3451-330c-837d-91fcc7545757', 'correlation_id': '41f062cc-8d89-4b89-882b-5a16df33c4cf', 'delivery_info': {'exchange': '', 'routing_key': 'celery', 'priority': 0, 'redelivered': False}}, '[[75], {}, {"callbacks": null, "errbacks": null, "chain": null, "chord": null}]', 'application/json', 'utf-8') kwargs:{})
[2019-02-23 12:50:09,995: INFO/MainProcess] Received task: repro.ping[fe5f5b82-a723-4625-bc47-50a2ce124d55]
[2019-02-23 12:50:09,997: DEBUG/MainProcess] Task accepted: repro.ping[41f062cc-8d89-4b89-882b-5a16df33c4cf] pid:24588
[2019-02-23 12:50:09,999: INFO/MainProcess] Received task: repro.ping[a0c88e06-0bd6-4f20-bfbf-c07848901c1b]
[2019-02-23 12:50:19,972: INFO/ForkPoolWorker-4] Task repro.ping[2b58a237-5dc0-44ad-99a1-70aa12361d02] succeeded in 10.003870679996908s: 72
[2019-02-23 12:50:19,973: DEBUG/MainProcess] TaskPool: Apply <function _fast_trace_task at 0x7f92462897b8> (args:('repro.ping', '03218e58-eb5f-4c86-b465-9399bdc3bb9d', {'lang': 'py', 'task': 'repro.ping', 'id': '03218e58-eb5f-4c86-b465-9399bdc3bb9d', 'shadow': None, 'eta': None, 'expires': None, 'group': None, 'retries': 0, 'timelimit': [None, None], 'root_id': '03218e58-eb5f-4c86-b465-9399bdc3bb9d', 'parent_id': None, 'argsrepr': '(76,)', 'kwargsrepr': '{}', 'origin': 'gen24599@repro', 'reply_to': '13b4de4e-3451-330c-837d-91fcc7545757', 'correlation_id': '03218e58-eb5f-4c86-b465-9399bdc3bb9d', 'delivery_info': {'exchange': '', 'routing_key': 'celery', 'priority': 0, 'redelivered': False}}, '[[76], {}, {"callbacks": null, "errbacks": null, "chain": null, "chord": null}]', 'application/json', 'utf-8') kwargs:{})
[2019-02-23 12:50:19,974: DEBUG/MainProcess] Task accepted: repro.ping[03218e58-eb5f-4c86-b465-9399bdc3bb9d] pid:24590
[2019-02-23 12:50:19,977: INFO/MainProcess] Received task: repro.ping[d895b100-d028-4277-9e24-4c6a03bb03a7]
[2019-02-23 12:50:19,988: INFO/ForkPoolWorker-3] Task repro.ping[a9e5d083-2314-4d61-b9b3-040231a0f7aa] succeeded in 10.010169839020818s: 73
[2019-02-23 12:50:19,989: DEBUG/MainProcess] TaskPool: Apply <function _fast_trace_task at 0x7f92462897b8> (args:('repro.ping', '76dee37c-4dd5-4d2a-810d-2f93726706d2', {'lang': 'py', 'task': 'repro.ping', 'id': '76dee37c-4dd5-4d2a-810d-2f93726706d2', 'shadow': None, 'eta': None, 'expires': None, 'group': None, 'retries': 0, 'timelimit': [None, None], 'root_id': '76dee37c-4dd5-4d2a-810d-2f93726706d2', 'parent_id': None, 'argsrepr': '(77,)', 'kwargsrepr': '{}', 'origin': 'gen24599@repro', 'reply_to': '13b4de4e-3451-330c-837d-91fcc7545757', 'correlation_id': '76dee37c-4dd5-4d2a-810d-2f93726706d2', 'delivery_info': {'exchange': '', 'routing_key': 'celery', 'priority': 0, 'redelivered': False}}, '[[77], {}, {"callbacks": null, "errbacks": null, "chain": null, "chord": null}]', 'application/json', 'utf-8') kwargs:{})
[2019-02-23 12:50:19,990: DEBUG/MainProcess] Task accepted: repro.ping[76dee37c-4dd5-4d2a-810d-2f93726706d2] pid:24589
[2019-02-23 12:50:19,993: INFO/MainProcess] Received task: repro.ping[73f89536-2609-460a-8a40-d5beaf0b1be7]
[2019-02-23 12:50:20,002: INFO/ForkPoolWorker-1] Task repro.ping[55610225-f895-4d65-8c86-30115fe3b289] succeeded in 10.011466949945316s: 74
[2019-02-23 12:50:20,003: DEBUG/MainProcess] TaskPool: Apply <function _fast_trace_task at 0x7f92462897b8> (args:('repro.ping', '19e75040-a721-44df-9c0f-32a79e51fb98', {'lang': 'py', 'task': 'repro.ping', 'id': '19e75040-a721-44df-9c0f-32a79e51fb98', 'shadow': None, 'eta': None, 'expires': None, 'group': None, 'retries': 0, 'timelimit': [None, None], 'root_id': '19e75040-a721-44df-9c0f-32a79e51fb98', 'parent_id': None, 'argsrepr': '(78,)', 'kwargsrepr': '{}', 'origin': 'gen24599@repro', 'reply_to': '13b4de4e-3451-330c-837d-91fcc7545757', 'correlation_id': '19e75040-a721-44df-9c0f-32a79e51fb98', 'delivery_info': {'exchange': '', 'routing_key': 'celery', 'priority': 0, 'redelivered': False}}, '[[78], {}, {"callbacks": null, "errbacks": null, "chain": null, "chord": null}]', 'application/json', 'utf-8') kwargs:{})
[2019-02-23 12:50:20,005: INFO/ForkPoolWorker-2] Task repro.ping[41f062cc-8d89-4b89-882b-5a16df33c4cf] succeeded in 10.008303896989673s: 75
[2019-02-23 12:50:20,005: DEBUG/MainProcess] Task accepted: repro.ping[19e75040-a721-44df-9c0f-32a79e51fb98] pid:24587
[2019-02-23 12:50:20,008: DEBUG/MainProcess] TaskPool: Apply <function _fast_trace_task at 0x7f92462897b8> (args:('repro.ping', '08edbae5-80e1-4f37-bb61-f55c6c6ef618', {'lang': 'py', 'task': 'repro.ping', 'id': '08edbae5-80e1-4f37-bb61-f55c6c6ef618', 'shadow': None, 'eta': None, 'expires': None, 'group': None, 'retries': 0, 'timelimit': [None, None], 'root_id': '08edbae5-80e1-4f37-bb61-f55c6c6ef618', 'parent_id': None, 'argsrepr': '(79,)', 'kwargsrepr': '{}', 'origin': 'gen24599@repro', 'reply_to': '13b4de4e-3451-330c-837d-91fcc7545757', 'correlation_id': '08edbae5-80e1-4f37-bb61-f55c6c6ef618', 'delivery_info': {'exchange': '', 'routing_key': 'celery', 'priority': 0, 'redelivered': False}}, '[[79], {}, {"callbacks": null, "errbacks": null, "chain": null, "chord": null}]', 'application/json', 'utf-8') kwargs:{})
[2019-02-23 12:50:20,010: INFO/MainProcess] Received task: repro.ping[a6927c58-b7c1-4c2e-b7c7-cdfbe7c9ea0b]
[2019-02-23 12:50:20,011: DEBUG/MainProcess] Task accepted: repro.ping[08edbae5-80e1-4f37-bb61-f55c6c6ef618] pid:24588
[2019-02-23 12:50:20,014: INFO/MainProcess] Received task: repro.ping[ea0e4c62-73d1-4d34-b98a-ad7851c1cbc7]
[2019-02-23 12:50:25,885: DEBUG/MainProcess] heartbeat_tick : for connection c96dac8c7b174cb3af302fd13ce1db55
[2019-02-23 12:50:25,885: DEBUG/MainProcess] heartbeat_tick : Prev sent/recv: 100/563, now - 108/617, monotonic - 828150.425361468, last_heartbeat_sent - 828150.425358227, heartbeat int. - 60 for connection c96dac8c7b174cb3af302fd13ce1db55
[2019-02-23 12:50:29,987: INFO/ForkPoolWorker-4] Task repro.ping[03218e58-eb5f-4c86-b465-9399bdc3bb9d] succeeded in 10.012512292014435s: 76
[2019-02-23 12:50:29,988: DEBUG/MainProcess] TaskPool: Apply <function _fast_trace_task at 0x7f92462897b8> (args:('repro.ping', '2328251e-1e29-49d9-8fdc-936a577b64ba', {'lang': 'py', 'task': 'repro.ping', 'id': '2328251e-1e29-49d9-8fdc-936a577b64ba', 'shadow': None, 'eta': None, 'expires': None, 'group': None, 'retries': 0, 'timelimit': [None, None], 'root_id': '2328251e-1e29-49d9-8fdc-936a577b64ba', 'parent_id': None, 'argsrepr': '(80,)', 'kwargsrepr': '{}', 'origin': 'gen24599@repro', 'reply_to': '13b4de4e-3451-330c-837d-91fcc7545757', 'correlation_id': '2328251e-1e29-49d9-8fdc-936a577b64ba', 'delivery_info': {'exchange': '', 'routing_key': 'celery', 'priority': 0, 'redelivered': False}}, '[[80], {}, {"callbacks": null, "errbacks": null, "chain": null, "chord": null}]', 'application/json', 'utf-8') kwargs:{})
[2019-02-23 12:50:29,989: DEBUG/MainProcess] Task accepted: repro.ping[2328251e-1e29-49d9-8fdc-936a577b64ba] pid:24590
[2019-02-23 12:50:29,991: INFO/MainProcess] Received task: repro.ping[c388da65-384b-4fd6-b773-fafb843dca0d]
[2019-02-23 12:50:30,000: INFO/ForkPoolWorker-3] Task repro.ping[76dee37c-4dd5-4d2a-810d-2f93726706d2] succeeded in 10.009812535019591s: 77
[2019-02-23 12:50:30,001: DEBUG/MainProcess] TaskPool: Apply <function _fast_trace_task at 0x7f92462897b8> (args:('repro.ping', '9d5dc543-cbe5-471f-9cdb-983e6a47ef0b', {'lang': 'py', 'task': 'repro.ping', 'id': '9d5dc543-cbe5-471f-9cdb-983e6a47ef0b', 'shadow': None, 'eta': None, 'expires': None, 'group': None, 'retries': 0, 'timelimit': [None, None], 'root_id': '9d5dc543-cbe5-471f-9cdb-983e6a47ef0b', 'parent_id': None, 'argsrepr': '(81,)', 'kwargsrepr': '{}', 'origin': 'gen24599@repro', 'reply_to': '13b4de4e-3451-330c-837d-91fcc7545757', 'correlation_id': '9d5dc543-cbe5-471f-9cdb-983e6a47ef0b', 'delivery_info': {'exchange': '', 'routing_key': 'celery', 'priority': 0, 'redelivered': False}}, '[[81], {}, {"callbacks": null, "errbacks": null, "chain": null, "chord": null}]', 'application/json', 'utf-8') kwargs:{})
[2019-02-23 12:50:30,002: DEBUG/MainProcess] Task accepted: repro.ping[9d5dc543-cbe5-471f-9cdb-983e6a47ef0b] pid:24589
[2019-02-23 12:50:30,004: INFO/MainProcess] Received task: repro.ping[04ce4455-f390-4b6c-adff-2a7771ae9ee5]
[2019-02-23 12:50:30,016: INFO/ForkPoolWorker-1] Task repro.ping[19e75040-a721-44df-9c0f-32a79e51fb98] succeeded in 10.011256138095632s: 78
[2019-02-23 12:50:30,017: DEBUG/MainProcess] TaskPool: Apply <function _fast_trace_task at 0x7f92462897b8> (args:('repro.ping', '8a8bec13-1de9-428a-aaa6-c292b8e9e97c', {'lang': 'py', 'task': 'repro.ping', 'id': '8a8bec13-1de9-428a-aaa6-c292b8e9e97c', 'shadow': None, 'eta': None, 'expires': None, 'group': None, 'retries': 0, 'timelimit': [None, None], 'root_id': '8a8bec13-1de9-428a-aaa6-c292b8e9e97c', 'parent_id': None, 'argsrepr': '(82,)', 'kwargsrepr': '{}', 'origin': 'gen24599@repro', 'reply_to': '13b4de4e-3451-330c-837d-91fcc7545757', 'correlation_id': '8a8bec13-1de9-428a-aaa6-c292b8e9e97c', 'delivery_info': {'exchange': '', 'routing_key': 'celery', 'priority': 0, 'redelivered': False}}, '[[82], {}, {"callbacks": null, "errbacks": null, "chain": null, "chord": null}]', 'application/json', 'utf-8') kwargs:{})
[2019-02-23 12:50:30,018: DEBUG/MainProcess] Task accepted: repro.ping[8a8bec13-1de9-428a-aaa6-c292b8e9e97c] pid:24587
[2019-02-23 12:50:30,021: INFO/MainProcess] Received task: repro.ping[9c9605c3-ab18-45b7-b565-55c5bdb856b7]
[2019-02-23 12:50:30,022: INFO/ForkPoolWorker-2] Task repro.ping[08edbae5-80e1-4f37-bb61-f55c6c6ef618] succeeded in 10.011361771030352s: 79
[2019-02-23 12:50:30,023: DEBUG/MainProcess] TaskPool: Apply <function _fast_trace_task at 0x7f92462897b8> (args:('repro.ping', '6ddfa78d-b7a4-4197-a490-fb52e87be667', {'lang': 'py', 'task': 'repro.ping', 'id': '6ddfa78d-b7a4-4197-a490-fb52e87be667', 'shadow': None, 'eta': None, 'expires': None, 'group': None, 'retries': 0, 'timelimit': [None, None], 'root_id': '6ddfa78d-b7a4-4197-a490-fb52e87be667', 'parent_id': None, 'argsrepr': '(83,)', 'kwargsrepr': '{}', 'origin': 'gen24599@repro', 'reply_to': '13b4de4e-3451-330c-837d-91fcc7545757', 'correlation_id': '6ddfa78d-b7a4-4197-a490-fb52e87be667', 'delivery_info': {'exchange': '', 'routing_key': 'celery', 'priority': 0, 'redelivered': False}}, '[[83], {}, {"callbacks": null, "errbacks": null, "chain": null, "chord": null}]', 'application/json', 'utf-8') kwargs:{})
[2019-02-23 12:50:30,025: DEBUG/MainProcess] Task accepted: repro.ping[6ddfa78d-b7a4-4197-a490-fb52e87be667] pid:24588
[2019-02-23 12:50:30,027: INFO/MainProcess] Received task: repro.ping[441dd75a-5691-4002-a9b6-765ad4485ed1]
[2019-02-23 12:50:39,999: INFO/ForkPoolWorker-4] Task repro.ping[2328251e-1e29-49d9-8fdc-936a577b64ba] succeeded in 10.009627755964175s: 80
[2019-02-23 12:50:40,000: DEBUG/MainProcess] TaskPool: Apply <function _fast_trace_task at 0x7f92462897b8> (args:('repro.ping', '20f83297-3bfe-4eb2-8484-2b2f0443dd46', {'lang': 'py', 'task': 'repro.ping', 'id': '20f83297-3bfe-4eb2-8484-2b2f0443dd46', 'shadow': None, 'eta': None, 'expires': None, 'group': None, 'retries': 0, 'timelimit': [None, None], 'root_id': '20f83297-3bfe-4eb2-8484-2b2f0443dd46', 'parent_id': None, 'argsrepr': '(84,)', 'kwargsrepr': '{}', 'origin': 'gen24599@repro', 'reply_to': '13b4de4e-3451-330c-837d-91fcc7545757', 'correlation_id': '20f83297-3bfe-4eb2-8484-2b2f0443dd46', 'delivery_info': {'exchange': '', 'routing_key': 'celery', 'priority': 0, 'redelivered': False}}, '[[84], {}, {"callbacks": null, "errbacks": null, "chain": null, "chord": null}]', 'application/json', 'utf-8') kwargs:{})
[2019-02-23 12:50:40,001: DEBUG/MainProcess] Task accepted: repro.ping[20f83297-3bfe-4eb2-8484-2b2f0443dd46] pid:24590
[2019-02-23 12:50:40,012: INFO/ForkPoolWorker-3] Task repro.ping[9d5dc543-cbe5-471f-9cdb-983e6a47ef0b] succeeded in 10.010064441012219s: 81
[2019-02-23 12:50:40,013: DEBUG/MainProcess] TaskPool: Apply <function _fast_trace_task at 0x7f92462897b8> (args:('repro.ping', 'f73b1e5b-9cec-4fdf-9e1d-ee02dc13716e', {'lang': 'py', 'task': 'repro.ping', 'id': 'f73b1e5b-9cec-4fdf-9e1d-ee02dc13716e', 'shadow': None, 'eta': None, 'expires': None, 'group': None, 'retries': 0, 'timelimit': [None, None], 'root_id': 'f73b1e5b-9cec-4fdf-9e1d-ee02dc13716e', 'parent_id': None, 'argsrepr': '(85,)', 'kwargsrepr': '{}', 'origin': 'gen24599@repro', 'reply_to': '13b4de4e-3451-330c-837d-91fcc7545757', 'correlation_id': 'f73b1e5b-9cec-4fdf-9e1d-ee02dc13716e', 'delivery_info': {'exchange': '', 'routing_key': 'celery', 'priority': 0, 'redelivered': False}}, '[[85], {}, {"callbacks": null, "errbacks": null, "chain": null, "chord": null}]', 'application/json', 'utf-8') kwargs:{})
[2019-02-23 12:50:40,014: DEBUG/MainProcess] Task accepted: repro.ping[f73b1e5b-9cec-4fdf-9e1d-ee02dc13716e] pid:24589
[2019-02-23 12:50:40,029: INFO/ForkPoolWorker-1] Task repro.ping[8a8bec13-1de9-428a-aaa6-c292b8e9e97c] succeeded in 10.011171153048053s: 82
[2019-02-23 12:50:40,030: DEBUG/MainProcess] TaskPool: Apply <function _fast_trace_task at 0x7f92462897b8> (args:('repro.ping', 'cc639408-eda2-4743-ae2d-83a76d66691a', {'lang': 'py', 'task': 'repro.ping', 'id': 'cc639408-eda2-4743-ae2d-83a76d66691a', 'shadow': None, 'eta': None, 'expires': None, 'group': None, 'retries': 0, 'timelimit': [None, None], 'root_id': 'cc639408-eda2-4743-ae2d-83a76d66691a', 'parent_id': None, 'argsrepr': '(86,)', 'kwargsrepr': '{}', 'origin': 'gen24599@repro', 'reply_to': '13b4de4e-3451-330c-837d-91fcc7545757', 'correlation_id': 'cc639408-eda2-4743-ae2d-83a76d66691a', 'delivery_info': {'exchange': '', 'routing_key': 'celery', 'priority': 0, 'redelivered': False}}, '[[86], {}, {"callbacks": null, "errbacks": null, "chain": null, "chord": null}]', 'application/json', 'utf-8') kwargs:{})
[2019-02-23 12:50:40,032: DEBUG/MainProcess] Task accepted: repro.ping[cc639408-eda2-4743-ae2d-83a76d66691a] pid:24587
[2019-02-23 12:50:40,036: INFO/ForkPoolWorker-2] Task repro.ping[6ddfa78d-b7a4-4197-a490-fb52e87be667] succeeded in 10.011511688004248s: 83
[2019-02-23 12:50:40,037: DEBUG/MainProcess] TaskPool: Apply <function _fast_trace_task at 0x7f92462897b8> (args:('repro.ping', '82e68b10-48c8-4343-b1e1-83d65712681a', {'lang': 'py', 'task': 'repro.ping', 'id': '82e68b10-48c8-4343-b1e1-83d65712681a', 'shadow': None, 'eta': None, 'expires': None, 'group': None, 'retries': 0, 'timelimit': [None, None], 'root_id': '82e68b10-48c8-4343-b1e1-83d65712681a', 'parent_id': None, 'argsrepr': '(87,)', 'kwargsrepr': '{}', 'origin': 'gen24599@repro', 'reply_to': '13b4de4e-3451-330c-837d-91fcc7545757', 'correlation_id': '82e68b10-48c8-4343-b1e1-83d65712681a', 'delivery_info': {'exchange': '', 'routing_key': 'celery', 'priority': 0, 'redelivered': False}}, '[[87], {}, {"callbacks": null, "errbacks": null, "chain": null, "chord": null}]', 'application/json', 'utf-8') kwargs:{})
[2019-02-23 12:50:40,039: DEBUG/MainProcess] Task accepted: repro.ping[82e68b10-48c8-4343-b1e1-83d65712681a] pid:24588
[2019-02-23 12:50:45,887: DEBUG/MainProcess] heartbeat_tick : for connection c96dac8c7b174cb3af302fd13ce1db55
[2019-02-23 12:50:45,887: DEBUG/MainProcess] heartbeat_tick : Prev sent/recv: 108/617, now - 116/659, monotonic - 828170.427266152, last_heartbeat_sent - 828170.427263, heartbeat int. - 60 for connection c96dac8c7b174cb3af302fd13ce1db55
[2019-02-23 12:50:50,012: INFO/ForkPoolWorker-4] Task repro.ping[20f83297-3bfe-4eb2-8484-2b2f0443dd46] succeeded in 10.011206874041818s: 84
[2019-02-23 12:50:50,013: DEBUG/MainProcess] TaskPool: Apply <function _fast_trace_task at 0x7f92462897b8> (args:('repro.ping', '0e75e7fd-0925-4a7a-932d-aff67100a893', {'lang': 'py', 'task': 'repro.ping', 'id': '0e75e7fd-0925-4a7a-932d-aff67100a893', 'shadow': None, 'eta': None, 'expires': None, 'group': None, 'retries': 0, 'timelimit': [None, None], 'root_id': '0e75e7fd-0925-4a7a-932d-aff67100a893', 'parent_id': None, 'argsrepr': '(88,)', 'kwargsrepr': '{}', 'origin': 'gen24599@repro', 'reply_to': '13b4de4e-3451-330c-837d-91fcc7545757', 'correlation_id': '0e75e7fd-0925-4a7a-932d-aff67100a893', 'delivery_info': {'exchange': '', 'routing_key': 'celery', 'priority': 0, 'redelivered': False}}, '[[88], {}, {"callbacks": null, "errbacks": null, "chain": null, "chord": null}]', 'application/json', 'utf-8') kwargs:{})
[2019-02-23 12:50:50,015: DEBUG/MainProcess] Task accepted: repro.ping[0e75e7fd-0925-4a7a-932d-aff67100a893] pid:24590
[2019-02-23 12:50:50,024: INFO/ForkPoolWorker-3] Task repro.ping[f73b1e5b-9cec-4fdf-9e1d-ee02dc13716e] succeeded in 10.010063005029224s: 85
[2019-02-23 12:50:50,025: DEBUG/MainProcess] TaskPool: Apply <function _fast_trace_task at 0x7f92462897b8> (args:('repro.ping', '6f64bb8f-849d-43cd-8361-64946efd769b', {'lang': 'py', 'task': 'repro.ping', 'id': '6f64bb8f-849d-43cd-8361-64946efd769b', 'shadow': None, 'eta': None, 'expires': None, 'group': None, 'retries': 0, 'timelimit': [None, None], 'root_id': '6f64bb8f-849d-43cd-8361-64946efd769b', 'parent_id': None, 'argsrepr': '(89,)', 'kwargsrepr': '{}', 'origin': 'gen24599@repro', 'reply_to': '13b4de4e-3451-330c-837d-91fcc7545757', 'correlation_id': '6f64bb8f-849d-43cd-8361-64946efd769b', 'delivery_info': {'exchange': '', 'routing_key': 'celery', 'priority': 0, 'redelivered': False}}, '[[89], {}, {"callbacks": null, "errbacks": null, "chain": null, "chord": null}]', 'application/json', 'utf-8') kwargs:{})
[2019-02-23 12:50:50,026: DEBUG/MainProcess] Task accepted: repro.ping[6f64bb8f-849d-43cd-8361-64946efd769b] pid:24589
[2019-02-23 12:50:50,040: INFO/ForkPoolWorker-1] Task repro.ping[cc639408-eda2-4743-ae2d-83a76d66691a] succeeded in 10.00802940595895s: 86
[2019-02-23 12:50:50,040: DEBUG/MainProcess] TaskPool: Apply <function _fast_trace_task at 0x7f92462897b8> (args:('repro.ping', 'fe5f5b82-a723-4625-bc47-50a2ce124d55', {'lang': 'py', 'task': 'repro.ping', 'id': 'fe5f5b82-a723-4625-bc47-50a2ce124d55', 'shadow': None, 'eta': None, 'expires': None, 'group': None, 'retries': 0, 'timelimit': [None, None], 'root_id': 'fe5f5b82-a723-4625-bc47-50a2ce124d55', 'parent_id': None, 'argsrepr': '(90,)', 'kwargsrepr': '{}', 'origin': 'gen24599@repro', 'reply_to': '13b4de4e-3451-330c-837d-91fcc7545757', 'correlation_id': 'fe5f5b82-a723-4625-bc47-50a2ce124d55', 'delivery_info': {'exchange': '', 'routing_key': 'celery', 'priority': 0, 'redelivered': False}}, '[[90], {}, {"callbacks": null, "errbacks": null, "chain": null, "chord": null}]', 'application/json', 'utf-8') kwargs:{})
[2019-02-23 12:50:50,041: DEBUG/MainProcess] Task accepted: repro.ping[fe5f5b82-a723-4625-bc47-50a2ce124d55] pid:24587
[2019-02-23 12:50:50,047: INFO/ForkPoolWorker-2] Task repro.ping[82e68b10-48c8-4343-b1e1-83d65712681a] succeeded in 10.008901569060981s: 87
[2019-02-23 12:50:50,048: DEBUG/MainProcess] TaskPool: Apply <function _fast_trace_task at 0x7f92462897b8> (args:('repro.ping', 'a0c88e06-0bd6-4f20-bfbf-c07848901c1b', {'lang': 'py', 'task': 'repro.ping', 'id': 'a0c88e06-0bd6-4f20-bfbf-c07848901c1b', 'shadow': None, 'eta': None, 'expires': None, 'group': None, 'retries': 0, 'timelimit': [None, None], 'root_id': 'a0c88e06-0bd6-4f20-bfbf-c07848901c1b', 'parent_id': None, 'argsrepr': '(91,)', 'kwargsrepr': '{}', 'origin': 'gen24599@repro', 'reply_to': '13b4de4e-3451-330c-837d-91fcc7545757', 'correlation_id': 'a0c88e06-0bd6-4f20-bfbf-c07848901c1b', 'delivery_info': {'exchange': '', 'routing_key': 'celery', 'priority': 0, 'redelivered': False}}, '[[91], {}, {"callbacks": null, "errbacks": null, "chain": null, "chord": null}]', 'application/json', 'utf-8') kwargs:{})
[2019-02-23 12:50:50,049: DEBUG/MainProcess] Task accepted: repro.ping[a0c88e06-0bd6-4f20-bfbf-c07848901c1b] pid:24588
[2019-02-23 12:51:00,025: INFO/ForkPoolWorker-4] Task repro.ping[0e75e7fd-0925-4a7a-932d-aff67100a893] succeeded in 10.010544363060035s: 88
[2019-02-23 12:51:00,025: DEBUG/MainProcess] TaskPool: Apply <function _fast_trace_task at 0x7f92462897b8> (args:('repro.ping', 'd895b100-d028-4277-9e24-4c6a03bb03a7', {'lang': 'py', 'task': 'repro.ping', 'id': 'd895b100-d028-4277-9e24-4c6a03bb03a7', 'shadow': None, 'eta': None, 'expires': None, 'group': None, 'retries': 0, 'timelimit': [None, None], 'root_id': 'd895b100-d028-4277-9e24-4c6a03bb03a7', 'parent_id': None, 'argsrepr': '(92,)', 'kwargsrepr': '{}', 'origin': 'gen24599@repro', 'reply_to': '13b4de4e-3451-330c-837d-91fcc7545757', 'correlation_id': 'd895b100-d028-4277-9e24-4c6a03bb03a7', 'delivery_info': {'exchange': '', 'routing_key': 'celery', 'priority': 0, 'redelivered': False}}, '[[92], {}, {"callbacks": null, "errbacks": null, "chain": null, "chord": null}]', 'application/json', 'utf-8') kwargs:{})
[2019-02-23 12:51:00,026: DEBUG/MainProcess] Task accepted: repro.ping[d895b100-d028-4277-9e24-4c6a03bb03a7] pid:24590
[2019-02-23 12:51:00,035: INFO/ForkPoolWorker-3] Task repro.ping[6f64bb8f-849d-43cd-8361-64946efd769b] succeeded in 10.009111239924096s: 89
[2019-02-23 12:51:00,035: DEBUG/MainProcess] TaskPool: Apply <function _fast_trace_task at 0x7f92462897b8> (args:('repro.ping', '73f89536-2609-460a-8a40-d5beaf0b1be7', {'lang': 'py', 'task': 'repro.ping', 'id': '73f89536-2609-460a-8a40-d5beaf0b1be7', 'shadow': None, 'eta': None, 'expires': None, 'group': None, 'retries': 0, 'timelimit': [None, None], 'root_id': '73f89536-2609-460a-8a40-d5beaf0b1be7', 'parent_id': None, 'argsrepr': '(93,)', 'kwargsrepr': '{}', 'origin': 'gen24599@repro', 'reply_to': '13b4de4e-3451-330c-837d-91fcc7545757', 'correlation_id': '73f89536-2609-460a-8a40-d5beaf0b1be7', 'delivery_info': {'exchange': '', 'routing_key': 'celery', 'priority': 0, 'redelivered': False}}, '[[93], {}, {"callbacks": null, "errbacks": null, "chain": null, "chord": null}]', 'application/json', 'utf-8') kwargs:{})
[2019-02-23 12:51:00,036: DEBUG/MainProcess] Task accepted: repro.ping[73f89536-2609-460a-8a40-d5beaf0b1be7] pid:24589
[2019-02-23 12:51:00,052: INFO/ForkPoolWorker-1] Task repro.ping[fe5f5b82-a723-4625-bc47-50a2ce124d55] succeeded in 10.010521587100811s: 90
[2019-02-23 12:51:00,052: DEBUG/MainProcess] TaskPool: Apply <function _fast_trace_task at 0x7f92462897b8> (args:('repro.ping', 'a6927c58-b7c1-4c2e-b7c7-cdfbe7c9ea0b', {'lang': 'py', 'task': 'repro.ping', 'id': 'a6927c58-b7c1-4c2e-b7c7-cdfbe7c9ea0b', 'shadow': None, 'eta': None, 'expires': None, 'group': None, 'retries': 0, 'timelimit': [None, None], 'root_id': 'a6927c58-b7c1-4c2e-b7c7-cdfbe7c9ea0b', 'parent_id': None, 'argsrepr': '(94,)', 'kwargsrepr': '{}', 'origin': 'gen24599@repro', 'reply_to': '13b4de4e-3451-330c-837d-91fcc7545757', 'correlation_id': 'a6927c58-b7c1-4c2e-b7c7-cdfbe7c9ea0b', 'delivery_info': {'exchange': '', 'routing_key': 'celery', 'priority': 0, 'redelivered': False}}, '[[94], {}, {"callbacks": null, "errbacks": null, "chain": null, "chord": null}]', 'application/json', 'utf-8') kwargs:{})
[2019-02-23 12:51:00,052: DEBUG/MainProcess] Task accepted: repro.ping[a6927c58-b7c1-4c2e-b7c7-cdfbe7c9ea0b] pid:24587
[2019-02-23 12:51:00,059: INFO/ForkPoolWorker-2] Task repro.ping[a0c88e06-0bd6-4f20-bfbf-c07848901c1b] succeeded in 10.009964159922674s: 91
[2019-02-23 12:51:00,059: DEBUG/MainProcess] TaskPool: Apply <function _fast_trace_task at 0x7f92462897b8> (args:('repro.ping', 'ea0e4c62-73d1-4d34-b98a-ad7851c1cbc7', {'lang': 'py', 'task': 'repro.ping', 'id': 'ea0e4c62-73d1-4d34-b98a-ad7851c1cbc7', 'shadow': None, 'eta': None, 'expires': None, 'group': None, 'retries': 0, 'timelimit': [None, None], 'root_id': 'ea0e4c62-73d1-4d34-b98a-ad7851c1cbc7', 'parent_id': None, 'argsrepr': '(95,)', 'kwargsrepr': '{}', 'origin': 'gen24599@repro', 'reply_to': '13b4de4e-3451-330c-837d-91fcc7545757', 'correlation_id': 'ea0e4c62-73d1-4d34-b98a-ad7851c1cbc7', 'delivery_info': {'exchange': '', 'routing_key': 'celery', 'priority': 0, 'redelivered': False}}, '[[95], {}, {"callbacks": null, "errbacks": null, "chain": null, "chord": null}]', 'application/json', 'utf-8') kwargs:{})
[2019-02-23 12:51:00,060: DEBUG/MainProcess] Task accepted: repro.ping[ea0e4c62-73d1-4d34-b98a-ad7851c1cbc7] pid:24588
[2019-02-23 12:51:05,889: DEBUG/MainProcess] heartbeat_tick : for connection c96dac8c7b174cb3af302fd13ce1db55
[2019-02-23 12:51:05,889: DEBUG/MainProcess] heartbeat_tick : Prev sent/recv: 116/659, now - 124/689, monotonic - 828190.429048136, last_heartbeat_sent - 828190.429045052, heartbeat int. - 60 for connection c96dac8c7b174cb3af302fd13ce1db55
[2019-02-23 12:51:10,037: INFO/ForkPoolWorker-4] Task repro.ping[d895b100-d028-4277-9e24-4c6a03bb03a7] succeeded in 10.011421873001382s: 92
[2019-02-23 12:51:10,038: DEBUG/MainProcess] TaskPool: Apply <function _fast_trace_task at 0x7f92462897b8> (args:('repro.ping', 'c388da65-384b-4fd6-b773-fafb843dca0d', {'lang': 'py', 'task': 'repro.ping', 'id': 'c388da65-384b-4fd6-b773-fafb843dca0d', 'shadow': None, 'eta': None, 'expires': None, 'group': None, 'retries': 0, 'timelimit': [None, None], 'root_id': 'c388da65-384b-4fd6-b773-fafb843dca0d', 'parent_id': None, 'argsrepr': '(96,)', 'kwargsrepr': '{}', 'origin': 'gen24599@repro', 'reply_to': '13b4de4e-3451-330c-837d-91fcc7545757', 'correlation_id': 'c388da65-384b-4fd6-b773-fafb843dca0d', 'delivery_info': {'exchange': '', 'routing_key': 'celery', 'priority': 0, 'redelivered': False}}, '[[96], {}, {"callbacks": null, "errbacks": null, "chain": null, "chord": null}]', 'application/json', 'utf-8') kwargs:{})
[2019-02-23 12:51:10,039: DEBUG/MainProcess] Task accepted: repro.ping[c388da65-384b-4fd6-b773-fafb843dca0d] pid:24590
[2019-02-23 12:51:10,044: INFO/ForkPoolWorker-3] Task repro.ping[73f89536-2609-460a-8a40-d5beaf0b1be7] succeeded in 10.008303378010169s: 93
[2019-02-23 12:51:10,045: DEBUG/MainProcess] TaskPool: Apply <function _fast_trace_task at 0x7f92462897b8> (args:('repro.ping', '04ce4455-f390-4b6c-adff-2a7771ae9ee5', {'lang': 'py', 'task': 'repro.ping', 'id': '04ce4455-f390-4b6c-adff-2a7771ae9ee5', 'shadow': None, 'eta': None, 'expires': None, 'group': None, 'retries': 0, 'timelimit': [None, None], 'root_id': '04ce4455-f390-4b6c-adff-2a7771ae9ee5', 'parent_id': None, 'argsrepr': '(97,)', 'kwargsrepr': '{}', 'origin': 'gen24599@repro', 'reply_to': '13b4de4e-3451-330c-837d-91fcc7545757', 'correlation_id': '04ce4455-f390-4b6c-adff-2a7771ae9ee5', 'delivery_info': {'exchange': '', 'routing_key': 'celery', 'priority': 0, 'redelivered': False}}, '[[97], {}, {"callbacks": null, "errbacks": null, "chain": null, "chord": null}]', 'application/json', 'utf-8') kwargs:{})
[2019-02-23 12:51:10,046: DEBUG/MainProcess] Task accepted: repro.ping[04ce4455-f390-4b6c-adff-2a7771ae9ee5] pid:24589
[2019-02-23 12:51:10,064: INFO/ForkPoolWorker-1] Task repro.ping[a6927c58-b7c1-4c2e-b7c7-cdfbe7c9ea0b] succeeded in 10.011372304987162s: 94
[2019-02-23 12:51:10,065: DEBUG/MainProcess] TaskPool: Apply <function _fast_trace_task at 0x7f92462897b8> (args:('repro.ping', '9c9605c3-ab18-45b7-b565-55c5bdb856b7', {'lang': 'py', 'task': 'repro.ping', 'id': '9c9605c3-ab18-45b7-b565-55c5bdb856b7', 'shadow': None, 'eta': None, 'expires': None, 'group': None, 'retries': 0, 'timelimit': [None, None], 'root_id': '9c9605c3-ab18-45b7-b565-55c5bdb856b7', 'parent_id': None, 'argsrepr': '(98,)', 'kwargsrepr': '{}', 'origin': 'gen24599@repro', 'reply_to': '13b4de4e-3451-330c-837d-91fcc7545757', 'correlation_id': '9c9605c3-ab18-45b7-b565-55c5bdb856b7', 'delivery_info': {'exchange': '', 'routing_key': 'celery', 'priority': 0, 'redelivered': False}}, '[[98], {}, {"callbacks": null, "errbacks": null, "chain": null, "chord": null}]', 'application/json', 'utf-8') kwargs:{})
[2019-02-23 12:51:10,066: DEBUG/MainProcess] Task accepted: repro.ping[9c9605c3-ab18-45b7-b565-55c5bdb856b7] pid:24587
[2019-02-23 12:51:10,068: INFO/ForkPoolWorker-2] Task repro.ping[ea0e4c62-73d1-4d34-b98a-ad7851c1cbc7] succeeded in 10.008219892974012s: 95
[2019-02-23 12:51:10,069: DEBUG/MainProcess] TaskPool: Apply <function _fast_trace_task at 0x7f92462897b8> (args:('repro.ping', '441dd75a-5691-4002-a9b6-765ad4485ed1', {'lang': 'py', 'task': 'repro.ping', 'id': '441dd75a-5691-4002-a9b6-765ad4485ed1', 'shadow': None, 'eta': None, 'expires': None, 'group': None, 'retries': 0, 'timelimit': [None, None], 'root_id': '441dd75a-5691-4002-a9b6-765ad4485ed1', 'parent_id': None, 'argsrepr': '(99,)', 'kwargsrepr': '{}', 'origin': 'gen24599@repro', 'reply_to': '13b4de4e-3451-330c-837d-91fcc7545757', 'correlation_id': '441dd75a-5691-4002-a9b6-765ad4485ed1', 'delivery_info': {'exchange': '', 'routing_key': 'celery', 'priority': 0, 'redelivered': False}}, '[[99], {}, {"callbacks": null, "errbacks": null, "chain": null, "chord": null}]', 'application/json', 'utf-8') kwargs:{})
[2019-02-23 12:51:10,070: DEBUG/MainProcess] Task accepted: repro.ping[441dd75a-5691-4002-a9b6-765ad4485ed1] pid:24588
[2019-02-23 12:51:20,050: INFO/ForkPoolWorker-4] Task repro.ping[c388da65-384b-4fd6-b773-fafb843dca0d] succeeded in 10.011102308984846s: 96
[2019-02-23 12:51:20,056: INFO/ForkPoolWorker-3] Task repro.ping[04ce4455-f390-4b6c-adff-2a7771ae9ee5] succeeded in 10.009683967917226s: 97
[2019-02-23 12:51:20,072: INFO/ForkPoolWorker-1] Task repro.ping[9c9605c3-ab18-45b7-b565-55c5bdb856b7] succeeded in 10.00583470694255s: 98
[2019-02-23 12:51:20,072: INFO/ForkPoolWorker-2] Task repro.ping[441dd75a-5691-4002-a9b6-765ad4485ed1] succeeded in 10.002141434000805s: 99
[2019-02-23 12:51:25,891: DEBUG/MainProcess] heartbeat_tick : for connection c96dac8c7b174cb3af302fd13ce1db55
[2019-02-23 12:51:25,891: DEBUG/MainProcess] heartbeat_tick : Prev sent/recv: 124/689, now - 128/719, monotonic - 828210.431072338, last_heartbeat_sent - 828210.431069012, heartbeat int. - 60 for connection c96dac8c7b174cb3af302fd13ce1db55
[2019-02-23 12:51:45,893: DEBUG/MainProcess] heartbeat_tick : for connection c96dac8c7b174cb3af302fd13ce1db55
[2019-02-23 12:51:45,893: DEBUG/MainProcess] heartbeat_tick : Prev sent/recv: 128/719, now - 128/749, monotonic - 828230.433275143, last_heartbeat_sent - 828210.431069012, heartbeat int. - 60 for connection c96dac8c7b174cb3af302fd13ce1db55
[2019-02-23 12:52:05,894: DEBUG/MainProcess] heartbeat_tick : for connection c96dac8c7b174cb3af302fd13ce1db55
[2019-02-23 12:52:05,895: DEBUG/MainProcess] heartbeat_tick : Prev sent/recv: 128/749, now - 128/779, monotonic - 828250.43490132, last_heartbeat_sent - 828210.431069012, heartbeat int. - 60 for connection c96dac8c7b174cb3af302fd13ce1db55

Combinations tested:

broker
backend
issue seen

pyampq://
rpc://localhost
YES

redis://
rpc://localhost
NO

pyamqp://
redis://
NO

Issue 2:
Celery report

software -> celery:4.2.0 (windowlicker) kombu:4.2.1 py:3.6.4
billiard:3.5.0.3 py-amqp:2.3.2
platform -> system:Linux arch:64bit imp:CPython
loader -> celery.loaders.app.AppLoader
settings -> transport:amqp results:rpc:///

broker_url: 'amqp://guest:********@localhost:5672//'

the issue exists on 4.2.0 and master and not on 4.1.1

Steps to reproduce

docker run -p 127.0.0.1:5672:5672 rabbitmq

cat >bug_sample.py <<EOF
import os
import time
import sys

import celery
import logging
logging.basicConfig(level=logging.INFO, stream=sys.stdout)

app = celery.Celery(broker='amqp://', backend='rpc://')

@app.task(name='ping')
def ping():
return 'pong'

def ping_pong():
time_start = time.time()
while True:
try:
logging.info('ping %s', time.time() - time_start)
ping.delay().get()
except Exception:
logging.exception('first exception')
break

logging.info('one more try')
try:
ping.delay().get()
except Exception:
logging.exception('second exception %s', time.time() - time_start)

if __name__ == '__main__':
ping_pong()
EOF

celery -A bug_sample.app worker

python ./bug_sample.py

Expected behavior

The sample loops forever

Actual behavior

After three minutes the OSError (socket closed) exception is raised.

After three minutes after that the same error raised again

INFO:root:ping 180.05654120445251
INFO:root:ping 180.05895519256592
INFO:root:ping 180.06122303009033
INFO:root:ping 180.06365275382996
ERROR:root:first exception
Traceback (most recent call last):
File "./bug_sample.py", line 20, in ping_pong
ping.delay().get()
File "/home/ivanov/files/cwd/celery/celery/result.py", line 224, in get
on_message=on_message,
File "/home/ivanov/files/cwd/celery/celery/backends/async.py", line 188, in wait_for_pending
for _ in self._wait_for_pending(result, **kwargs):
File "/home/ivanov/files/cwd/celery/celery/backends/async.py", line 255, in _wait_for_pending
on_interval=on_interval):
File "/home/ivanov/files/cwd/celery/celery/backends/async.py", line 56, in drain_events_until
yield self.wait_for(p, wait, timeout=1)
File "/home/ivanov/files/cwd/celery/celery/backends/async.py", line 65, in wait_for
wait(timeout=timeout)
File "/home/ivanov/files/cwd/celery/celery/backends/rpc.py", line 63, in drain_events
return self._connection.drain_events(timeout=timeout)
File "/home/ivanov/files/opt/miniconda3/envs/mlservice/lib/python3.6/site-packages/kombu/connection.py", line 301, in drain_events
return self.transport.drain_events(self.connection, **kwargs)
File "/home/ivanov/files/opt/miniconda3/envs/mlservice/lib/python3.6/site-packages/kombu/transport/pyamqp.py", line 103, in drain_events
return connection.drain_events(**kwargs)
File "/home/ivanov/files/opt/miniconda3/envs/mlservice/lib/python3.6/site-packages/amqp/connection.py", line 491, in drain_events
while not self.blocking_read(timeout):
File "/home/ivanov/files/opt/miniconda3/envs/mlservice/lib/python3.6/site-packages/amqp/connection.py", line 496, in blocking_read
frame = self.transport.read_frame()
File "/home/ivanov/files/opt/miniconda3/envs/mlservice/lib/python3.6/site-packages/amqp/transport.py", line 243, in read_frame
frame_header = read(7, True)
File "/home/ivanov/files/opt/miniconda3/envs/mlservice/lib/python3.6/site-packages/amqp/transport.py", line 426, in _read
raise IOError('Socket closed')
OSError: Socket closed
INFO:root:one more try
ERROR:root:second exception 360.07977294921875
Traceback (most recent call last):
File "./bug_sample.py", line 27, in ping_pong
ping.delay().get()
File "/home/ivanov/files/cwd/celery/celery/result.py", line 224, in get
on_message=on_message,
File "/home/ivanov/files/cwd/celery/celery/backends/async.py", line 188, in wait_for_pending
for _ in self._wait_for_pending(result, **kwargs):
File "/home/ivanov/files/cwd/celery/celery/backends/async.py", line 255, in _wait_for_pending
on_interval=on_interval):
File "/home/ivanov/files/cwd/celery/celery/backends/async.py", line 56, in drain_events_until
yield self.wait_for(p, wait, timeout=1)
File "/home/ivanov/files/cwd/celery/celery/backends/async.py", line 65, in wait_for
wait(timeout=timeout)
File "/home/ivanov/files/cwd/celery/celery/backends/rpc.py", line 63, in drain_events
return self._connection.drain_events(timeout=timeout)
File "/home/ivanov/files/opt/miniconda3/envs/mlservice/lib/python3.6/site-packages/kombu/connection.py", line 301, in drain_events
return self.transport.drain_events(self.connection, **kwargs)
File "/home/ivanov/files/opt/miniconda3/envs/mlservice/lib/python3.6/site-packages/kombu/transport/pyamqp.py", line 103, in drain_events
return connection.drain_events(**kwargs)
File "/home/ivanov/files/opt/miniconda3/envs/mlservice/lib/python3.6/site-packages/amqp/connection.py", line 491, in drain_events
while not self.blocking_read(timeout):
File "/home/ivanov/files/opt/miniconda3/envs/mlservice/lib/python3.6/site-packages/amqp/connection.py", line 496, in blocking_read
frame = self.transport.read_frame()
File "/home/ivanov/files/opt/miniconda3/envs/mlservice/lib/python3.6/site-packages/amqp/transport.py", line 243, in read_frame
frame_header = read(7, True)
File "/home/ivanov/files/opt/miniconda3/envs/mlservice/lib/python3.6/site-packages/amqp/transport.py", line 426, in _read
raise IOError('Socket closed')
OSError: Socket closed