Package: rq-exporter
Python RQ Prometheus ExporterPrometheus metrics exporter for Python RQ (Redis Queue) job queue library.InstallationInstall the Python package:$# Install the latest version$pipinstallrq-exporter
$# Or install a specific version$pipinstallrq-exporter==2.1.0Or download theDocker image:$# Pull the latest image$dockerpullmdawar/rq-exporter
$# Or pull a specific version$dockerpullmdawar/rq-exporter:v2.1.0The releases are available asDocker image tags.UsagePython package:$# Start the exporter on port 9726$rq-exporter
$# Start the exporter on a specific port and host (Default: 0.0.0.0:9726)$rq-exporter--hostlocalhost--port8080$# By default the exporter will connect to Redis on `localhost` port `6379`$# You can specify a Redis URL$rq-exporter--redis-urlredis://:123456@redis_host:6379/0
$# Or specific Redis options (host, port, db, password)$rq-exporter--redis-host192.168.1.10--redis-port6380--redis-pass123456--redis-db1$# You can also specify a password file path (eg: mounted Docker secret)$rq-exporter--redis-pass-file/run/secrets/redis_passDocker image:$# Run the exporter and publish the port 9726 on the host$dockerrun-it-p9726:9726rq-exporter
$# Use the -d option to run the container in the background (detached)$dockerrun-d-p9726:9726rq-exporter
$# All the command line arguments will be passed to rq-exporter$dockerrun-it-p9726:9726rq-exporter--redis-hostredis--redis-pass123456$# You can also configure the exporter using environment variables$dockerrun-it-p9726:9726-eRQ_REDIS_HOST=redis-eRQ_REDIS_PASS=123456rq-exporterGrafana DashboardAn exampleGrafanadashboardis available with the ID12196for showcasing this exporter's metrics.You can also find theJSON file of the dashboardin this repository.Note:This is just an example dashboard, feel free to use it as a base for your custom dashboardYou need to adjust the color thresholds to suit your needs for the job status percentagesinglestatpanelsSome panels might seem duplicated providing percentages and current values, these are just for showcasing the PromQL queriesExported MetricsRQ metrics:Metric NameTypeLabelsDescriptionrq_workersGaugename,queues,stateRQ workersrq_jobsGaugequeue,statusRQ jobs by queue and statusrq_workers_success_totalCountername,queuesSuccessful job count by workerrq_workers_failed_totalCountername,queuesFailed job count by workerrq_workers_working_time_totalCountername,queuesTotal working time in seconds by workerRequest processing metrics:Metric NameTypeDescriptionrq_request_processing_seconds_countSummaryNumber of times the RQ data were collectedrq_request_processing_seconds_sumSummaryTotal sum of time spent collecting RQ datarq_request_processing_seconds_createdGaugeTime created at (time.time()return value)Example:# HELP rq_request_processing_seconds Time spent collecting RQ data# TYPE rq_request_processing_seconds summaryrq_request_processing_seconds_count1.0
rq_request_processing_seconds_sum0.029244607000009637# TYPE rq_request_processing_seconds_created gaugerq_request_processing_seconds_created1.5878023726039658e+09# HELP rq_workers RQ workers# TYPE rq_workers gaugerq_workers{name="40d33ed9541644d79373765e661b7f38",queues="default",state="idle"}1.0
rq_workers{name="fe9a433575e04685a53e4794b2eaeea9",queues="high,default,low",state="busy"}1.0# HELP rq_jobs RQ jobs by state# TYPE rq_jobs gaugerq_jobs{queue="default",status="queued"}2.0
rq_jobs{queue="default",status="started"}1.0
rq_jobs{queue="default",status="finished"}5.0
rq_jobs{queue="default",status="failed"}1.0
rq_jobs{queue="default",status="deferred"}1.0
rq_jobs{queue="default",status="scheduled"}2.0ConfigurationYou can configure the exporter using command line arguments or environment variables:CLI ArgumentEnv VariableDefault ValueDescription--hostRQ_EXPORTER_HOST0.0.0.0Serve the exporter on this host-p,--portRQ_EXPORTER_PORT9726Serve the exporter on this port--redis-urlRQ_REDIS_URLNoneRedis URL in the formredis://:[password]@[host]:[port]/[db]--redis-hostRQ_REDIS_HOSTlocalhostRedis host name--redis-portRQ_REDIS_PORT6379Redis port number--redis-dbRQ_REDIS_DB0Redis database number--sentinel-hostRQ_SENTINEL_HOSTNoneRedis Sentinel hosts separated by commas e.gsentinel1,sentinel2:26380--sentinel-portRQ_SENTINEL_PORT26379Redis Sentinel port, default port used when not set with the host--sentinel-masterRQ_SENTINEL_MASTERmasterRedis Sentinel master name--redis-passRQ_REDIS_PASSNoneRedis password--redis-pass-fileRQ_REDIS_PASS_FILENoneRedis password file path (e.g. Path of a mounted Docker secret)--worker-classRQ_WORKER_CLASSrq.WorkerRQ worker class--queue-classRQ_QUEUE_CLASSrq.QueueRQ queue class--log-levelRQ_EXPORTER_LOG_LEVELINFOLogging level--log-formatRQ_EXPORTER_LOG_FORMAT[%(asctime)s] [%(name)s] [%(levelname)s]: %(message)sLogging handler format string--log-datefmtRQ_EXPORTER_LOG_DATEFMT%Y-%m-%d %H:%M:%SLogging date/time format stringNotes:When Redis URL is set using--redis-urlorRQ_REDIS_URLthe other Redis options will be ignoredWhen the Redis password is set using--redis-pass-fileorRQ_REDIS_PASS_FILE, then--redis-passandRQ_REDIS_PASSwill be ignoredThe Sentinel port will default to the value of--sentinel-portif not set for each host with--sentinel-hostorRQ_SENTINEL_HOSTServing with GunicornThe WSGI application can be created using therq_exporter.create_app()function:$gunicorn"rq_exporter:create_app()"-b0.0.0.0:9726--log-levelinfoExampleDockerfileto create aDockerimage to serve the application withGunicornFROMmdawar/rq-exporter:latestUSERrootRUNpipinstall--no-cache-dirgunicornUSERexporterENTRYPOINT["gunicorn","rq_exporter:create_app()"]CMD["-b","0.0.0.0:9726","--threads","2","--log-level","info","--keep-alive","3"]Note about concurrency:The exporter is going to work without any problems with multiple workers but you will get different values for these metrics:rq_request_processing_seconds_countrq_request_processing_seconds_sumrq_request_processing_seconds_createdThis is fine if you don't care about these metrics, these are only for measuring the count and time processing the RQ data, so the other RQ metrics are not going to be affected.But you can still use multiple threads with 1 worker process to handle multiple concurrent requests:$gunicorn"rq_exporter:create_app()"-b0.0.0.0:9726--threads2Building the Docker Image$# Build the docker image and tag it rq-exporter:latest$dockerbuild-trq-exporter.
$# Or$makebuild

$# M1 MacOS Build the docker image and tag it rq-exporter:latest$dockerbuildxbuild--platformlinux/amd64-trq-exporter.The image can also be built usingdocker compose:$dockercomposebuildCheck thedocker-compose.ymlfile for usage example.DevelopmentTo start a full development environment withRQworkers,PrometheusandGrafana:$dockercomposeup
$# If you want to start multiple workers use the --compatibility flag$# which will make docker compose read the `deploy` section and start multiple replicas$dockercompose--compatibilityupYou can access the services on these ports on your local machine:RQ exporter:9726Redis:6379RQ Dashboard:9181Prometheus:9090Grafana:3000(Login usingadmin:admin)You can specify the services that you want to start by their name in thedocker-compose.ymlfile:$# Example starting only the `rq_exporter` and `redis` services$dockercomposeuprq_exporterredisTo run more workers and enqueue more jobs you can scale theworkerandenqueueservices:$# Run 5 workers$dockercomposeup-d--scaleworker=5$# Enqueue more jobs$# Scale the enqueue service and the workers$dockercomposeup-d--scaleworker=5--scaleenqueue=2To cleanup after development:$# Use -v to remove volumes$dockercomposedown-vYou can also start anotherrq-exporterinstance that collects stats from a project using customRQWorkerandQueueclasses:$# Using -f to pass multiple docker-compose files$# docker-compose.custom.yml defines services using custom RQ classes$dockercompose-fdocker-compose.yml-fdocker-compose.custom.ymlup
$# To cleanup you need to also pass the same files$dockercompose-fdocker-compose.yml-fdocker-compose.custom.ymldownA newRQ exporterinstance will be exposed on port9727on your local machine.Note: If you don't havedocker composeinstalled follow theinstallationinstructions on the official website.If you want to use the package manually:$# Clone the repository$gitclone<REPO_URL>
$# Change to the project directory$cdrq-exporter
$# Create a new virtualenv$python-mvenv/path/to/env
$# Activate the environment$source/path/to/env/bin/activate
$# Install the requirements$pipinstall-rrequirements.txt
$# Start the exporter on port 9726$python-mrq_exporter
$# You can configure the exporter using command line arguments$python-mrq_exporter--port8080Running the Tests$maketest$# Or$python-munittestContributingFork therepositoryClone the forked repositorygit clone <URL>Create a new feature branchgit checkout -b <BRANCH_NAME>Make changes and add tests if needed and commit your changesgit commit -am "Your commit message"Push the new branch to Githubgit push origin <BRANCH_NAME>Create a pull request