Package: prometheus-client-gc
Prometheus Python ClientThe official Python 2 and 3 client forPrometheus.Three Step DemoOne: Install the client:pip install prometheus_clientTwo: Paste the following into a Python interpreter:fromprometheus_clientimportstart_http_server,Summaryimportrandomimporttime# Create a metric to track time spent and requests made.REQUEST_TIME=Summary('request_processing_seconds','Time spent processing request')# Decorate function with metric.@REQUEST_TIME.time()defprocess_request(t):"""A dummy function that takes some time."""time.sleep(t)if__name__=='__main__':# Start up the server to expose the metrics.start_http_server(8000)# Generate some requests.whileTrue:process_request(random.random())Three: Visithttp://localhost:8000/to view the metrics.From one easy to use decorator you get:request_processing_seconds_count: Number of times this function was called.request_processing_seconds_sum: Total amount of time spent in this function.Prometheus'sratefunction allows calculation of both requests per second,
and latency over time from this data.In addition if you're on Linux theprocessmetrics expose CPU, memory and
other information about the process for free!Installationpip install prometheus_clientThis package can be found onPyPI.InstrumentingFour types of metric are offered: Counter, Gauge, Summary and Histogram.
See the documentation onmetric typesandinstrumentation best practiceson how to use them.CounterCounters go up, and reset when the process restarts.fromprometheus_clientimportCounterc=Counter('my_failures','Description of counter')c.inc()# Increment by 1c.inc(1.6)# Increment by given valueIf there is a suffix of_totalon the metric name, it will be removed. When
exposing the time series for counter, a_totalsuffix will be added. This is
for compatibility between OpenMetrics and the Prometheus text format, as OpenMetrics
requires the_totalsuffix.There are utilities to count exceptions raised:@c.count_exceptions()deff():passwithc.count_exceptions():pass# Count only one type of exceptionwithc.count_exceptions(ValueError):passGaugeGauges can go up and down.fromprometheus_clientimportGaugeg=Gauge('my_inprogress_requests','Description of gauge')g.inc()# Increment by 1g.dec(10)# Decrement by given valueg.set(4.2)# Set to a given valueThere are utilities for common use cases:g.set_to_current_time()# Set to current unixtime# Increment when entered, decrement when exited.@g.track_inprogress()deff():passwithg.track_inprogress():passA Gauge can also take its value from a callback:d=Gauge('data_objects','Number of objects')my_dict={}d.set_function(lambda:len(my_dict))SummarySummaries track the size and number of events.fromprometheus_clientimportSummarys=Summary('request_latency_seconds','Description of summary')s.observe(4.7)# Observe 4.7 (seconds in this case)There are utilities for timing code:@s.time()deff():passwiths.time():passThe Python client doesn't store or expose quantile information at this time.HistogramHistograms track the size and number of events in buckets.
This allows for aggregatable calculation of quantiles.fromprometheus_clientimportHistogramh=Histogram('request_latency_seconds','Description of histogram')h.observe(4.7)# Observe 4.7 (seconds in this case)The default buckets are intended to cover a typical web/rpc request from milliseconds to seconds.
They can be overridden by passingbucketskeyword argument toHistogram.There are utilities for timing code:@h.time()deff():passwithh.time():passInfoInfo tracks key-value information, usually about a whole target.fromprometheus_clientimportInfoi=Info('my_build_version','Description of info')i.info({'version':'1.2.3','buildhost':'foo@bar'})EnumEnum tracks which of a set of states something is currently in.fromprometheus_clientimportEnume=Enum('my_task_state','Description of enum',states=['starting','running','stopped'])e.state('running')LabelsAll metrics can have labels, allowing grouping of related time series.See the best practices onnamingandlabels.Taking a counter as an example:fromprometheus_clientimportCounterc=Counter('my_requests_total','HTTP Failures',['method','endpoint'])c.labels('get','/').inc()c.labels('post','/submit').inc()Labels can also be passed as keyword-arguments:fromprometheus_clientimportCounterc=Counter('my_requests_total','HTTP Failures',['method','endpoint'])c.labels(method='get',endpoint='/').inc()c.labels(method='post',endpoint='/submit').inc()Metrics with labels are not initialized when declared, because the client can't
know what values the label can have. It is recommended to initialize the label
values by calling the.labels()method alone:fromprometheus_clientimportCounterc=Counter('my_requests_total','HTTP Failures',['method','endpoint'])c.labels('get','/')c.labels('post','/submit')Process CollectorThe Python client automatically exports metrics about process CPU usage, RAM,
file descriptors and start time. These all have the prefixprocess, and
are only currently available on Linux.The namespace and pid constructor arguments allows for exporting metrics about
other processes, for example:ProcessCollector(namespace='mydaemon', pid=lambda: open('/var/run/daemon.pid').read())Platform CollectorThe client also automatically exports some metadata about Python. If using Jython,
metadata about the JVM in use is also included. This information is available as
labels on thepython_infometric. The value of the metric is 1, since it is the
labels that carry information.ExportingThere are several options for exporting metrics.HTTPMetrics are usually exposed over HTTP, to be read by the Prometheus server.The easiest way to do this is viastart_http_server, which will start a HTTP
server in a daemon thread on the given port:fromprometheus_clientimportstart_http_serverstart_http_server(8000)Visithttp://localhost:8000/to view the metrics.To add Prometheus exposition to an existing HTTP server, see theMetricsHandlerclass
which provides aBaseHTTPRequestHandler. It also serves as a simple example of how
to write a custom endpoint.TwistedTo use prometheus withtwisted, there isMetricsResourcewhich exposes metrics as a twisted resource.fromprometheus_client.twistedimportMetricsResourcefromtwisted.web.serverimportSitefromtwisted.web.resourceimportResourcefromtwisted.internetimportreactorroot=Resource()root.putChild(b'metrics',MetricsResource())factory=Site(root)reactor.listenTCP(8000,factory)reactor.run()WSGITo use Prometheus withWSGI, there ismake_wsgi_appwhich creates a WSGI application.fromprometheus_clientimportmake_wsgi_appfromwsgiref.simple_serverimportmake_serverapp=make_wsgi_app()httpd=make_server('',8000,app)httpd.serve_forever()Such an application can be useful when integrating Prometheus metrics with WSGI
apps.The methodstart_wsgi_servercan be used to serve the metrics through the
WSGI reference implementation in a new thread.fromprometheus_clientimportstart_wsgi_serverstart_wsgi_server(8000)ASGITo use Prometheus withASGI, there ismake_asgi_appwhich creates an ASGI application.fromprometheus_clientimportmake_asgi_appapp=make_asgi_app()Such an application can be useful when integrating Prometheus metrics with ASGI
apps.FlaskTo use Prometheus withFlaskwe need to serve metrics through a Prometheus WSGI application. This can be achieved usingFlask's application dispatching. Below is a working example.Save the snippet below in amyapp.pyfilefromflaskimportFlaskfromwerkzeug.middleware.dispatcherimportDispatcherMiddlewarefromprometheus_clientimportmake_wsgi_app# Create my appapp=Flask(__name__)# Add prometheus wsgi middleware to route /metrics requestsapp.wsgi_app=DispatcherMiddleware(app.wsgi_app,{'/metrics':make_wsgi_app()})Run the example web application like this# Install uwsgi if you do not have itpipinstalluwsgi
uwsgi--http127.0.0.1:8000--wsgi-filemyapp.py--callableappVisithttp://localhost:8000/metricsto see the metricsNode exporter textfile collectorThetextfile collectorallows machine-level statistics to be exported out via the Node exporter.This is useful for monitoring cronjobs, or for writing cronjobs to expose metrics
about a machine system that the Node exporter does not support or would not make sense
to perform at every scrape (for example, anything involving subprocesses).fromprometheus_clientimportCollectorRegistry,Gauge,write_to_textfileregistry=CollectorRegistry()g=Gauge('raid_status','1 if raid array is okay',registry=registry)g.set(1)write_to_textfile('/configured/textfile/path/raid.prom',registry)A separate registry is used, as the default registry may contain other metrics
such as those from the Process Collector.Exporting to a PushgatewayThePushgatewayallows ephemeral and batch jobs to expose their metrics to Prometheus.fromprometheus_clientimportCollectorRegistry,Gauge,push_to_gatewayregistry=CollectorRegistry()g=Gauge('job_last_success_unixtime','Last time a batch job successfully finished',registry=registry)g.set_to_current_time()push_to_gateway('localhost:9091',job='batchA',registry=registry)A separate registry is used, as the default registry may contain other metrics
such as those from the Process Collector.Pushgateway functions take a grouping key.push_to_gatewayreplaces metrics
with the same grouping key,pushadd_to_gatewayonly replaces metrics with the
same name and grouping key anddelete_from_gatewaydeletes metrics with the
given job and grouping key. See thePushgateway documentationfor more information.instance_ip_grouping_keyreturns a grouping key with the instance label set
to the host's IP address.Handlers for authenticationIf the push gateway you are connecting to is protected with HTTP Basic Auth,
you can use a special handler to set the Authorization header.fromprometheus_clientimportCollectorRegistry,Gauge,push_to_gatewayfromprometheus_client.expositionimportbasic_auth_handlerdefmy_auth_handler(url,method,timeout,headers,data):username='foobar'password='secret123'returnbasic_auth_handler(url,method,timeout,headers,data,username,password)registry=CollectorRegistry()g=Gauge('job_last_success_unixtime','Last time a batch job successfully finished',registry=registry)g.set_to_current_time()push_to_gateway('localhost:9091',job='batchA',registry=registry,handler=my_auth_handler)BridgesIt is also possible to expose metrics to systems other than Prometheus.
This allows you to take advantage of Prometheus instrumentation even
if you are not quite ready to fully transition to Prometheus yet.GraphiteMetrics are pushed over TCP in the Graphite plaintext format.fromprometheus_client.bridge.graphiteimportGraphiteBridgegb=GraphiteBridge(('graphite.your.org',2003))# Push once.gb.push()# Push every 10 seconds in a daemon thread.gb.start(10.0)Custom CollectorsSometimes it is not possible to directly instrument code, as it is not
in your control. This requires you to proxy metrics from other systems.To do so you need to create a custom collector, for example:fromprometheus_client.coreimportGaugeMetricFamily,CounterMetricFamily,REGISTRYclassCustomCollector(object):defcollect(self):yieldGaugeMetricFamily('my_gauge','Help text',value=7)c=CounterMetricFamily('my_counter_total','Help text',labels=['foo'])c.add_metric(['bar'],1.7)c.add_metric(['baz'],3.8)yieldcREGISTRY.register(CustomCollector())SummaryMetricFamilyandHistogramMetricFamilywork similarly.A collector may implement adescribemethod which returns metrics in the same
format ascollect(though you don't have to include the samples). This is
used to predetermine the names of time series aCollectorRegistryexposes and
thus to detect collisions and duplicate registrations.Usually custom collectors do not have to implementdescribe. Ifdescribeis
not implemented and the CollectorRegistry was created withauto_describe=True(which is the case for the default registry) thencollectwill be called at
registration time instead ofdescribe. If this could cause problems, either
implement a properdescribe, or if that's not practical havedescribereturn an empty list.Multiprocess Mode (Gunicorn)Prometheus client libraries presume a threaded model, where metrics are shared
across workers. This doesn't work so well for languages such as Python where
it's common to have processes rather than threads to handle large workloads.To handle this the client library can be put in multiprocess mode.
This comes with a number of limitations:Registries can not be used as normal, all instantiated metrics are exportedCustom collectors do not work (e.g. cpu and memory metrics)Info and Enum metrics do not workThe pushgateway cannot be usedGauges cannot use thepidlabelThere's several steps to getting this working:1. Gunicorn deployment:Theprometheus_multiproc_direnvironment variable must be set to a directory
that the client library can use for metrics. This directory must be wiped
between Gunicorn runs (before startup is recommended).This environment variable should be set from a start-up shell script,
and not directly from Python (otherwise it may not propagate to child processes).2. Metrics collector:The application must initialize a newCollectorRegistry,
and store the multi-process collector inside.fromprometheus_clientimportmultiprocessfromprometheus_clientimportgenerate_latest,CollectorRegistry,CONTENT_TYPE_LATEST# Expose metrics.defapp(environ,start_response):registry=CollectorRegistry()multiprocess.MultiProcessCollector(registry)data=generate_latest(registry)status='200 OK'response_headers=[('Content-type',CONTENT_TYPE_LATEST),('Content-Length',str(len(data)))]start_response(status,response_headers)returniter([data])3. Gunicorn configuration:Thegunicornconfiguration file needs to include the following function:fromprometheus_clientimportmultiprocessdefchild_exit(server,worker):multiprocess.mark_process_dead(worker.pid)4. Metrics tuning (Gauge):WhenGaugemetrics are used, additional tuning needs to be performed.
Gauges have several modes they can run in, which can be selected with themultiprocess_modeparameter.'all': Default. Return a timeseries per process alive or dead.'liveall': Return a timeseries per process that is still alive.'livesum': Return a single timeseries that is the sum of the values of alive processes.'max': Return a single timeseries that is the maximum of the values of all processes, alive or dead.'min': Return a single timeseries that is the minimum of the values of all processes, alive or dead.fromprometheus_clientimportGauge# Example gaugeIN_PROGRESS=Gauge("inprogress_requests","help",multiprocess_mode='livesum')ParserThe Python client supports parsing the Prometheus text format.
This is intended for advanced use cases where you have servers
exposing Prometheus metrics and need to get them into some other
system.fromprometheus_client.parserimporttext_string_to_metric_familiesforfamilyintext_string_to_metric_families(u"my_gauge 1.0\n"):forsampleinfamily.samples:print("Name:{0}Labels:{1}Value:{2}".format(*sample))