Package: pytheus
pytheusplaying with metricsDocumentation:https://pythe.uspytheus is a modern python library for collectingprometheusmetrics built with multiprocessing in mind.Some of the features are:multiple multiprocess support:redis backend ✅Rust powered backend 🧪bring your own ✅support for default labels value ✅partial labels value (built in an incremental way) ✅customizable registry support ✅registry prefix support ✅PhilosophySimply put is to let you work with metrics the way you want.Be extremely flexible, allow for customization from the user for anything they might want to do without having to resort to hacks and most importantly offer the same api for single & multi process scenarios, the switch should be as easy as loading a different backend without having to change anything in the code.What you see is what you get.No differences betweensingleprocess&multiprocess, the only change is loading a different backend and everything will work out of the box.High flexibility with an high degree oflabelscontrol and customization.RequirementsPython 3.8+redis >= 4.0.0 (optional: for multiprocessing)pytheus-backend-rs (optional: for Rust powered multiprocessing 🦀)ArchitectureA small piece of the architecture is described in theARCHITECTURE.md fileInstallpip install pytheusOptionally if you want to use the Redis backend (for multiprocess support) you will need the redis library:pipinstallredis# orpipinstallpytheus[redis]If you want to try the Rust based backend (for multiprocess support):pipinstallpytheus-backend-rsPartial labels support:frompytheus.metricsimportCounter# without labelsmy_metric=Counter('metric_name','desc')my_metric.inc()# example for counter# with labelsmy_metric=Counter('metric_name','desc',required_labels=['req1','req2'])my_metric.labels({'req1':'1','req2':'2'}).inc()# you can pass all the labels at oncepartial_my_metric=my_metric.labels({'req1':'1'})# a cacheable object with one of the required labels already setobservable_my_metric=partial_my_metric.labels({'req2':'2'})# finish setting the remaining values before observingobservable_my_metric.inc()# alternatively to using a `dict` you can use `kwargs` (keyword arguments)my_metric.labels(req1='1',req2='2')Default labels support:frompytheus.metricsimportCounter# with default labelsmy_metric=Counter('metric_name','desc',required_labels=['req1','req2'],default_labels={'req2':'me_set!'})my_metric.labels({'req1':'1'}).inc()# as we have req2 as a default label we only need to set the remaining labels for observingmy_metric.labels({'req1':'1','req2':'2'})# you can still override default labels!Exposing metrics:You can use thegenerate_metricsfunction frompytheus.expositionto generate the metrics and serve them as an endpoint with your favourite web framework.Alternatively you can use themake_wsgi_appfunction that creates a simple wsgi app to serve the metrics.Quickstart / ExampleTheexample.pyfile starts a flask application with three endpoints:/: just returns a phrase while observing the time taken for the request to complete/slow: same as before but will sleep so that values will only end up in higher buckets/metrics: the endpoint used by prometheus to scrape the metricsIt uses two histograms, one without labels, and one with labels required and a default label that makes it observable.
To expose the metrics thegenerate_metrics()function is used.
note: the example file is using the redis backend but you can try without and set up prometheus yourself.Redis versionFor the redis version you can just clone the repository and rundocker-compose upto start both redis and prometheus scraping on localhost:8080.
Then you can start the local server withpython example.py. (flask is required for it to work)Now you can visit the described endpoints and by visitinglocalhost:9090you can query prometheus, for example by looking for all the slow requests buckets:page_visits_latency_seconds_labeled_bucket{speed="slow"}Default versionFor the default single process version you can create your python server like this:importtimefromflaskimportFlaskfrompytheus.metricsimportHistogramfrompytheus.expositionimportgenerate_metricsapp=Flask(__name__)histogram=Histogram('page_visits_latency_seconds','used for testing')# this is the endpoint that prometheus will use to scrape the metrics@app.route('/metrics')defmetrics():returngenerate_metrics()# track time with the context manager@app.route('/')defhome():withhistogram.time():return'hello world!'# you can also track time with the decorator shortcut@app.route('/slow')@histogramdefslow():time.sleep(3)return'hello world! from slow!'app.run(host='0.0.0.0',port=8080)and if you have prometheus installed configure it to scrape on localhost:8080 or you can still use the includeddocker-compose.ymlfile.Metric typesCounterThe Counter is a metric that only increases and can resets to 0. (For example if a service restart, it will start again from zero)frompytheus.metricsimportCountercounter=Counter(name="my_counter",description="My description")# increase by 1counter.inc()# increase by xcounter.inc(7)# it is possible to count exceptionswithcounter.count_exceptions():raiseValueError# increases counter by 1# you can specify which exceptions to watch forwithcounter.count_exceptions((IndexError,ValueError)):raiseValueError.# increases counter by 1# it is possible to use the counter as a decorator as a shortcut to count exceptions@counterdeftest():raiseValueError# increases counter by 1 when called# specifying which exceptions to look for also works with the decorator@counter(exceptions=(IndexError,ValueError))deftest():raiseValueError# increases counter by 1 when calledGaugeThe Gauge can increase and decrease its value. It is also possible to set a specific value.frompytheus.metricsimportGaugegauge=Gauge(name="my_gauge",description="My description")# increase by 1gauge.inc()# increase by xgauge.inc(7)# decrease by 1gauge.dec()# set a specific valuegauge.set(7)# set to current unix timestampgauge.set_to_current_time()# it is possible to track progress so that when entered increases the value by 1, and when exited decreases itwithgauge.track_inprogress():do_something()# you can also time a piece of code, will set the duration in seconds to value when exitedwithgauge.time():do_something()# tracking time can also be done as a decorator@gaugedefdo_something():...# tracking progress is also available via decorator with a flag@gauge(track_inprogress=True)defdo_something():...HistogramA histogram samples observations (usually things like request durations or response sizes) and counts them in configurable buckets. It also provides a sum of all observed values. (taken from prometheus docs)frompytheus.metricsimportHistogramhistogram=Histogram(name="my_histogram",description="My description")# by default it will have the following buckets: (.005, .01, .025, .05, .1, .25, .5, 1, 2.5, 5, 10)# note: the +Inf bucket will be added automatically, this is float('inf') in python# create a histogram specifying bucketshistogram=Histogram(name="my_histogram",description="My description",buckets=(0.2,1,3))# observe a valuehistogram.observe(0.4)# you can also time a piece of code, will set the duration in seconds to value when exitedwithhistogram.time():do_something()# tracking time can also be done as a decorator@histogramdefdo_something():...SummaryA Summary is a metric that captures individual observations and tracks the total size & number of events observed. It can be useful to track latencies for example.frompytheus.metricsimportSummarysummary=Summary(name="my_summary",description="My description")# observe a valuesummary.observe(0.4)# you can also time a piece of code, will set the duration in seconds to value when exitedwithsummary.time():do_something()# tracking time can also be done as a decorator@summarydefdo_something():...Custom CollectorsIt is possible to use a custom collector in cases you can't directly instrument the code.
You will need to inherit fromCustomCollectorand define thecollectmethod.
Also be sure to disable the automatic registering of a newly created metric into the default registry.frompytheus.metricsimportCounter,CustomCollectorfrompytheus.registryimportREGISTRYclassMyCustomCollector(CustomCollector):defcollect():counter=Counter('name','desc',registry=None)# note that we are disabling automatic registeringcounter.inc()yieldcounterREGISTRY.register(MyCustomCollector())Note: if one of the yield metrics have a name already registered with the registry you are trying to register to, the custom collector will be ignored.How to use different backendsThings work out of the box, using the SingleProcessBackend:frompytheus.metricsimportCountercounter=Counter(name="my_metric",description="My description",required_labels=["label_a","label_b"],)print(counter._metric_value_backend.__class__)# <class 'pytheus.backends.base.SingleProcessBackend'>print(counter._metric_value_backend.config)# {}You can define environment configuration to have different defaults, using two environment variables:exportPYTHEUS_BACKEND_CLASS="pytheus.backends.redis.MultiProcessRedisBackend"exportPYTHEUS_BACKEND_CONFIG="./config.json"Now, create the config file,./config.json:{"host":"127.0.0.1","port":6379}Now we can try the same snippet as above:frompytheus.metricsimportCountercounter=Counter(name="my_metric",description="My description",required_labels=["label_a","label_b"],)print(counter._metric_value_backend.__class__)# <class 'pytheus.backends.redis.MultiProcessRedisBackend'>print(counter._metric_value_backend.config)# {"host": "127.0.0.1", "port":  6379}You can also pass the values directly in Python, which would take precedence over the environment
setup we have just described:frompytheus.metricsimportCounterfrompytheus.backendsimportload_backendfrompytheus.backends.redisimportMultiProcessRedisBackendload_backend(backend_class=MultiProcessRedisBackend,backend_config={"host":"127.0.0.1","port":6379})# Notice that if you simply call load_backend(), it would reload config from the environment.# load_backend() is called automatically at package import, that's why we didn't need to call it# directly in the previous examplecounter=Counter(name="my_metric",description="My description",required_labels=["label_a","label_b"],)print(counter._metric_value_backend.__class__)# <class 'pytheus.backends.redis.MultiProcessRedisBackend'>print(counter._metric_value_backend.config)# {'host': '127.0.0.1', 'port': 6379}Create your own BackendCustom BackendYou can create your own backend by implementing a class that fulfills theBackendprotocol.classBackend(Protocol):def__init__(self,config:BackendConfig,metric:"_Metric",histogram_bucket:str|None=None,)->None:...definc(self,value:float)->None:...defdec(self,value:float)->None:...defset(self,value:float)->None:...defget(self)->float:...Initialization hookIt's possible that you want to initialize your custom backend or there are one time steps that you want to happen on import.To achieve that you can use the class method hook called_initializethat accepts aBackendConfigparameter.@classmethoddef_initialize(cls,config:"BackendConfig")->None:# initialization stepsClient patchingAnhighly experimentalfeature is to patch in-place the officialprometheus_clientso that you can test out the multiprocessing mode of this library without committing to it or just to try it out.importprometheus_clientfrompytheus.experimental.compatibilityimportpatch_clientpatch_client(prometheus_client)This has to be done at the start/initialization of your application and you can load a different backend before applying with patch withpatch_clientby usingload_backend.This supports all the metrics typesCounter,Gauge,Summary&Histogram. It supports the defaultREGISTRYandgenerate_latest. It also works with thestart_http_serverfunction.Of course it's extremely limited and you might be using a feature that is not yet supported, but feel free to try! :)(This was used to test an internal library without having to migrate all of it, so since I worked on it might as well share it! )