Package: celery-prometheus-exporter
celery-prometheus-exporter is a little exporter for Celery related metrics in
order to get picked up by Prometheus. As with other exporters like
mongodb_exporter or node_exporter this has been implemented as a
standalone-service to make reuse easier across different frameworks.So far it provides access to the following metrics:celery_tasksexposes the number of tasks currently known to the queue
grouped bystate(RECEIVED, STARTED, …).celery_tasks_by_nameexposes the number of tasks currently known to the queue
grouped bynameandstate.celery_workersexposes the number of currently probably alive workerscelery_task_latencyexposes a histogram of task latency, i.e. the time until
tasks are picked up by a workercelery_tasks_runtime_secondstracks the number of seconds tasks take
until completed as histogramHow to useThere are multiple ways to install this. The obvious one is usingpip installcelery-prometheus-exporterand then using thecelery-prometheus-exportercommand:$ celery-prometheus-exporter
Starting HTTPD on 0.0.0.0:8888This package only depends on Celery directly, so you will have to install
whatever other dependencies you will need for it to speak with your broker 🙂Celery workers have to be configured to send task-related events:http://docs.celeryproject.org/en/latest/userguide/configuration.html#worker-send-task-events.Runningcelery-prometheus-exporterwith the--enable-eventsargument
will periodically enable events on the workers. This is useful because it
allows running celery workers with events disabled, untilcelery-prometheus-exporteris deployed, at which time events get enabled
on the workers.Alternatively, you can use the bundle Makefile and Dockerfile to generate a
Docker image.By default, the HTTPD will listen at0.0.0.0:8888. If you want the HTTPD
to listen to another port, use the--addroption or the environment variableDEFAULT_ADDR.By default, this will expect the broker to be available throughredis://redis:6379/0, although you can change via environment variableBROKER_URL. If you’re using AMQP or something else other than
Redis, take a look at the Celery documentation and install the additioinal
requirements 😊 Also use the--brokeroption to specify a different broker
URL.If you need to pass additional options to your broker’s transport use the--transport-optionsoption. It tries to read a dict from a JSON object.
E.g. to set your master name when using Redis Sentinel for broker discovery:--transport-options'{"master_name":"mymaster"}'Use--tzto specify the timezone the Celery app is using. Otherwise the
systems local time will be used.By default, buckets for histograms are the same as default ones in the prometheus client:https://github.com/prometheus/client_python#histogram.
It means they are intended to cover typical web/rpc requests from milliseconds to seconds,
so you may want to customize them.
It can be done via environment variableRUNTIME_HISTOGRAM_BUCKETSfor tasks runtime and
via environment variableLATENCY_HISTOGRAM_BUCKETSfor tasks latency.
Buckets should be passed as a list of float values separated by a comma.
E.g.".005, .05, 0.1, 1.0, 2.5".Use--queue-listto specify the list of queues that will have its length
monitored (Automatic Discovery of queues isn’t supported right now, see limitations/
caveats. You can use theQUEUE_LISTenvironment variable as well.If you then look at the exposed metrics, you should see something like this:$ http get http://localhost:8888/metrics | grep celery_
# HELP celery_workers Number of alive workers
# TYPE celery_workers gauge
celery_workers 1.0
# HELP celery_tasks Number of tasks per state
# TYPE celery_tasks gauge
celery_tasks{state="RECEIVED"} 3.0
celery_tasks{state="PENDING"} 0.0
celery_tasks{state="STARTED"} 1.0
celery_tasks{state="RETRY"} 2.0
celery_tasks{state="FAILURE"} 1.0
celery_tasks{state="REVOKED"} 0.0
celery_tasks{state="SUCCESS"} 8.0
# HELP celery_tasks_by_name Number of tasks per state
# TYPE celery_tasks_by_name gauge
celery_tasks_by_name{name="my_app.tasks.calculate_something",state="RECEIVED"} 0.0
celery_tasks_by_name{name="my_app.tasks.calculate_something",state="PENDING"} 0.0
celery_tasks_by_name{name="my_app.tasks.calculate_something",state="STARTED"} 0.0
celery_tasks_by_name{name="my_app.tasks.calculate_something",state="RETRY"} 0.0
celery_tasks_by_name{name="my_app.tasks.calculate_something",state="FAILURE"} 0.0
celery_tasks_by_name{name="my_app.tasks.calculate_something",state="REVOKED"} 0.0
celery_tasks_by_name{name="my_app.tasks.calculate_something",state="SUCCESS"} 1.0
celery_tasks_by_name{name="my_app.tasks.fetch_some_data",state="RECEIVED"} 3.0
celery_tasks_by_name{name="my_app.tasks.fetch_some_data",state="PENDING"} 0.0
celery_tasks_by_name{name="my_app.tasks.fetch_some_data",state="STARTED"} 1.0
celery_tasks_by_name{name="my_app.tasks.fetch_some_data",state="RETRY"} 2.0
celery_tasks_by_name{name="my_app.tasks.fetch_some_data",state="FAILURE"} 1.0
celery_tasks_by_name{name="my_app.tasks.fetch_some_data",state="REVOKED"} 0.0
celery_tasks_by_name{name="my_app.tasks.fetch_some_data",state="SUCCESS"} 7.0
# HELP celery_task_latency Seconds between a task is received and started.
# TYPE celery_task_latency histogram
celery_task_latency_bucket{le="0.005"} 2.0
celery_task_latency_bucket{le="0.01"} 3.0
celery_task_latency_bucket{le="0.025"} 4.0
celery_task_latency_bucket{le="0.05"} 4.0
celery_task_latency_bucket{le="0.075"} 5.0
celery_task_latency_bucket{le="0.1"} 5.0
celery_task_latency_bucket{le="0.25"} 5.0
celery_task_latency_bucket{le="0.5"} 5.0
celery_task_latency_bucket{le="0.75"} 5.0
celery_task_latency_bucket{le="1.0"} 5.0
celery_task_latency_bucket{le="2.5"} 8.0
celery_task_latency_bucket{le="5.0"} 11.0
celery_task_latency_bucket{le="7.5"} 11.0
celery_task_latency_bucket{le="10.0"} 11.0
celery_task_latency_bucket{le="+Inf"} 11.0
celery_task_latency_count 11.0
celery_task_latency_sum 16.478713035583496
celery_queue_length{queue_name="queue1"} 35.0
celery_queue_length{queue_name="queue2"} 0.0LimitationsAmong tons of other features celery-prometheus-exporter doesn’t support stats
for multiple queues. As far as I can tell, only the routing key is exposed
through the events API which might be enough to figure out the final queue,
though.This has only been tested with Redis so far.At this point, you should specify the queues that will be monitored using an
environment variable or an arg (–queue-list).