Package: autometrics
A Python port of the Rustautometrics-rslibraryMetrics are a powerful and cost-efficient tool for understanding the health and performance of your code in production. But it's hard to decide what metrics to track and even harder to write queries to understand the data.Autometrics provides a decorator that makes it trivial to instrument any function with the most useful metrics: request rate, error rate, and latency. It standardizes these metrics and then generates powerful Prometheus queries based on your function details to help you quickly identify and debug issues in production.SeeWhy Autometrics?for more details on the ideas behind autometrics.Features✨@autometricsdecorator instruments any function or class method to track the
most useful metrics💡 Writes Prometheus queries so you can understand the data generated without
knowing PromQL🔗 Create links to live Prometheus charts directly into each function's docstring🔍Identify commitsthat introduced errors or increased latency🚨Define alertsusing SLO best practices directly in your source code📊Grafana dashboardswork out of the box to visualize the performance of instrumented functions & SLOs⚙️Configurablemetric collection library (opentelemetryorprometheus)📍Attach exemplarsto connect metrics with traces⚡ Minimal runtime overheadQuickstartAddautometricsto your project's dependencies:pipinstallautometricsInstrument your functions with the@autometricsdecoratorfromautometricsimportautometrics@autometricsdefmy_function():# ...Configure autometrics by calling theinitfunction:fromautometricsimportinitinit(tracker="prometheus",service_name="my-service")Export the metrics for Prometheus# This example uses FastAPI, but you can use any web frameworkfromfastapiimportFastAPI,Responsefromprometheus_clientimportgenerate_latest# Set up a metrics endpoint for Prometheus to scrape#   `generate_latest` returns metrics data in the Prometheus text format@app.get("/metrics")defmetrics():returnResponse(generate_latest())Run Prometheus locally with theAutometrics CLIorconfigure it manuallyto scrape your metrics endpoint# Replace `8080` with the port that your app runs onamstart:8080(Optional) If you have Grafana, import theAutometrics dashboardsfor an overview and detailed view of all the function metrics you've collectedUsingautometrics-pyYou can import the library in your code and use the decorator for any function:fromautometricsimportautometrics@autometricsdefsayHello:return"hello"To show tooltips over decorated functions in VSCode, with links to Prometheus queries, try installingthe VSCode extension.Note: We cannot support tooltips without a VSCode extension due to behavior of thestatic analyzerused in VSCode.You can also track the number of concurrent calls to a function by using thetrack_concurrencyargument:@autometrics(track_concurrency=True).Note: Concurrency tracking is only supported when you set with the environment variableAUTOMETRICS_TRACKER=prometheus.To access the PromQL queries for your decorated functions, runhelp(yourfunction)orprint(yourfunction.__doc__).For these queries to work, include a.envfile in your project with your prometheus endpointPROMETHEUS_URL=your endpoint. If this is not defined, the default endpoint will behttp://localhost:9090/DashboardsAutometrics providesGrafana dashboardsthat will work for any project instrumented with the library.Alerts / SLOsAutometrics makes it easy to add intelligent alerting to your code, in order to catch increases in the error rate or latency across multiple functions.fromautometricsimportautometricsfromautometrics.objectivesimportObjective,ObjectiveLatency,ObjectivePercentile# Create an objective for a high success rate# Here, we want our API to have a success rate of 99.9%API_SLO_HIGH_SUCCESS=Objective("My API SLO for High Success Rate (99.9%)",success_rate=ObjectivePercentile.P99_9,)@autometrics(objective=API_SLO_HIGH_SUCCESS)defapi_handler():# ...The library uses the concept of Service-Level Objectives (SLOs) to define the acceptable error rate and latency for groups of functions. Alerts will fire depending on the SLOs you set.Not sure what SLOs are?Check out our docsfor an introduction.In order to receive alerts,you need to add a special set of rules to your Prometheus setup. These are configured automatically when you use theAutometrics CLIto run Prometheus.Already running Prometheus yourself?Read about how to load the autometrics alerting rules into Prometheus here.Once the alerting rules are in Prometheus, you're ready to go.To use autometrics SLOs and alerts, create one or multipleObjectives based on the function(s) success rate and/or latency, as shown above.TheObjectivecan be passed as an argument to theautometricsdecorator, which will include the given function in that objective.The example above used a success rate objective. (I.e., we wanted to be alerted when the error rate started to increase.)You can also create an objective for the latency of your functions like so:fromautometricsimportautometricsfromautometrics.objectivesimportObjective,ObjectiveLatency,ObjectivePercentile# Create an objective for low latency#   - Functions with this objective should have a 99th percentile latency of less than 250msAPI_SLO_LOW_LATENCY=Objective("My API SLO for Low Latency (99th percentile < 250ms)",latency=(ObjectiveLatency.Ms250,ObjectivePercentile.P99),)@autometrics(objective=API_SLO_LOW_LATENCY)defapi_handler():# ...ThecallerLabelAutometrics keeps track of instrumented functions that call each other. So, if you have a functionget_usersthat calls another functiondb.query, then the metrics for latter will include a labelcaller="get_users".This allows you to drill down into the metrics for functions that arecalled byyour instrumented functions, provided both of those functions are decorated with@autometrics.In the example above, this means that you could investigate the latency of the database queries thatget_usersmakes, which is rather useful.Settings and ConfigurationAutometrics makes use of a number of environment variables to configure its behavior. All of them are also configurable with keyword arguments to theinitfunction.tracker- Configure the package that autometrics will use to produce metrics. Default isopentelemetry, but you can also useprometheus. Look inpyproject.tomlfor the corresponding versions of packages that will be used.histogram_buckets- Configure the buckets used for latency histograms. Default is[0.005, 0.01, 0.025, 0.05, 0.075, 0.1, 0.25, 0.5, 0.75, 1.0, 2.5, 5.0, 7.5, 10.0].enable_exemplars- Enableexemplar collection. Default isFalse.service_name- Configure theservice name.version,commit,branch,repository_url,repository_provider- Used to configurebuild_info.Below is an example of initializing autometrics with build information, as well as theprometheustracker. (Note that you can also accomplish the same confiugration with environment variables.)fromautometricsimportautometrics,initfromgit_utilsimportget_git_commit,get_git_branchVERSION="0.0.1"init(tracker="prometheus",version=VERSION,commit=get_git_commit(),branch=get_git_branch())Identifying commits that introduced problemsAutometrics makes it easy to identify if a specific version or commit introduced errors or increased latencies.NOTE- As of writing,build_infowill not work correctly when using the default setting ofAUTOMETRICS_TRACKER=opentelemetry. If you wish to usebuild_info, you must use theprometheustracker instead (AUTOMETRICS_TRACKER=prometheus).The issue will be fixed once the following PR is merged and released on the opentelemetry-python project:https://github.com/open-telemetry/opentelemetry-python/pull/3306autometrics-py will track support for build_info using the OpenTelemetry tracker viathis issueThe library uses a separate metric (build_info) to track the version and git metadata of your code - repository url, provider name, commit and branch.It then writes queries that group metrics by these metadata, so you can spot correlations between code changes and potential issues.Configure these labels by setting the following environment variables:LabelRun-Time Environment VariablesDefault valueversionAUTOMETRICS_VERSION""commitAUTOMETRICS_COMMITorCOMMIT_SHA""branchAUTOMETRICS_BRANCHorBRANCH_NAME""repository_urlAUTOMETRICS_REPOSITORY_URL""*repository_providerAUTOMETRICS_REPOSITORY_PROVIDER""** Autometrics will attempt to automagically infer these values from the git config inside your working directory. To disable this behavior, explicitly set the corresponding setting or environment variable to"".This follows the method outlined inExposing the software version to Prometheus.Service nameAll metrics produced by Autometrics have a label calledservice.name(orservice_namewhen exported to Prometheus) attached, in order to identify the logical service they are part of.You may want to override the default service name, for example if you are running multiple instances of the same code base as separate services, and you want to differentiate between the metrics produced by each one.The service name is loaded from the following environment variables, in this order:AUTOMETRICS_SERVICE_NAME(at runtime)OTEL_SERVICE_NAME(at runtime)First part of__package__(at runtime)ExemplarsNOTE- As of writing, exemplars aren't supported by the default tracker (AUTOMETRICS_TRACKER=opentelemetry).
You can track the progress of this feature here:https://github.com/autometrics-dev/autometrics-py/issues/41Exemplars are a way to associate a metric sample to a trace by attachingtrace_idandspan_idto it. You can then use this information to jump from a metric to a trace in your tracing system (for example Jaeger). If you have an OpenTelemetry tracer configured, autometrics will automatically pick up the current span from it.To use exemplars, you need to first switch to a tracker that supports them by settingAUTOMETRICS_TRACKER=prometheusand enable
exemplar collection by settingAUTOMETRICS_EXEMPLARS=true. You also need to enable exemplars in Prometheus by launching Prometheus with the--enable-feature=exemplar-storageflag.Exporting metricsThere are multiple ways to export metrics from your application, depending on your setup. You can see examples of how to do this in theexamples/export_metricsdirectory.If you want to export metrics to Prometheus, you have two options in case of bothopentelemetryandprometheustrackers:Create a route inside your app and respond withgenerate_latest()# This example uses FastAPI, but you can use any web frameworkfromfastapiimportFastAPI,Responsefromprometheus_clientimportgenerate_latest# Set up a metrics endpoint for Prometheus to scrape@app.get("/metrics")defmetrics():returnResponse(generate_latest())Specifyprometheusas the exporter type, and a separate server will be started to expose metrics from your app:exporter={"type":"prometheus","address":"localhost","port":9464}init(tracker="prometheus",service_name="my-service",exporter=exporter)For the OpenTelemetry tracker, you have more options, including a custom metric reader. You can specify the exporter type to beotlp-proto-httporotlp-proto-grpc, and metrics will be exported to a remote OpenTelemetry collector via the specified protocol. You will need to install the respective extra dependency in order for this to work, which you can do when you install autometrics:pipinstallautometrics[exporter-otlp-proto-http]pipinstallautometrics[exporter-otlp-proto-grpc]After installing it you can configure the exporter as follows:exporter={"type":"otlp-proto-grpc","address":"http://localhost:4317","insecure":True}init(tracker="opentelemetry",service_name="my-service",exporter=exporter)To use a custom metric reader you can specify the exporter type to beotel-customand provide a custom metric reader:my_custom_metric_reader=PrometheusMetricReader("")exporter={"type":"otel-custom","reader":my_custom_metric_reader}init(tracker="opentelemetry",service_name="my-service",exporter=exporter)Development of the packageThis package usespoetryas a package manager, with all dependencies separated into three groups:root level dependencies, requireddev, everything that is needed for development or in ciexamples, dependencies of everything inexamples/directoryBy default, poetry will only install required dependencies, if you want to run examples, install using this command:poetryinstall--withexamplesCode in this repository is:formatted usingblack.contains type definitions (which are linted bymypy)tested usingpytestIn order to run these tools locally you have to install them, you can install them using poetry:poetryinstall--withdev--all-extrasAfter that you can run the tools individually# Formatting using blackpoetryrunblack.# Lint using mypypoetryrunmypy.# Run the tests using pytestpoetryrunpytest# Run a single test, and clear the cachepoetryrunpytest--cache-clear-ktest_tracker