Project: Grafana Pyroscope (Observability and Analysis / Observability)
File: about-server-api.md

description: Learn about the Pyrocope server API menuTitle: Server HTTP API title: Pyroscope server HTTP API weight: 20 Pyroscope server exposes an HTTP API for querying profiling data and ingesting profiling data from other sources. Grafana Pyroscope doesn't include an authentication layer. Operators should use an authenticating reverse proxy for security. In multi-tenant mode, Pyroscope requires the X-Scope-OrgID HTTP header set to a string identifying the tenant. This responsibility should be handled by the authenticating reverse proxy. For more information, refer to the . There is one primary endpoint: POST /ingest. It accepts profile data in the request body and metadata as query parameters. The following query parameters are accepted: | Name | Description | Notes | |:-|:-|:-| | `name` | application name | required | | `from` | UNIX time of when the profiling started | required | | `until` | UNIX time of when the profiling stopped | required | | `format` | format of the profiling data | optional (default is `folded`) | | `sampleRate` | sample rate used in Hz | optional (default is `100` Hz) | | `spyName` | name of the spy used | optional | | `units` | name of the profiling data unit | optional (default is `samples` | | `aggregrationType` | type of aggregation to merge profiles | optional (default is `sum`) | `name` specifies application name. For example: ``` my.awesome.app.cpu{env=staging,region=us-west-1} ``` The request body contains profiling data, and the Content-Type header may be used alongside format to determine the data format. Some of the query parameters depend on the format of profiling data. Pyroscope currently supports three major ingestion formats. These formats handle simple ingestion of profiling data, such as `cpu` samples, and typically don't support metadata (e.g., labels) within the format. All necessary metadata is derived from query parameters, and the format is specified by the `format` query parameter. Supported formats: Folded: Also known as `collapsed`, this is the default format. Each line contains a stacktrace followed by the sample count for that stacktrace. For example: ``` foo;bar 100 foo;baz 200 ``` Lines: Similar to `folded`, but it represents each sample as a separate line rather than aggregating samples per stacktrace. For example: ``` foo;bar foo;bar foo;baz foo;bar ``` The `pprof` format is a widely used binary profiling data format, particularly prevalent in the Go ecosystem. When using this format, certain query parameters have specific behaviors: format: This should be set to `pprof`. name: This parameter contains the prefix of the application name. Since a single request might include multiple profile types, the complete application name is formed by concatenating this prefix with the profile type. For instance, if you send CPU profiling data and set `name` to `my-app{}`, it will be displayed in pyroscope as `my-app.cpu{}`. units, aggregationType, and sampleRate: These parameters are ignored. The actual values are determined based on the profile types present in the data (refer to the "Sample Type Configuration" section for more details). Pyroscope server inherently supports standard Go profile types such as `cpu`, `inuseobjects`, `inusespace`, `allocobjects`, and `allocspace`. When dealing with software that generates data in `pprof` format, you may need to supply a custom sample type configuration for Pyroscope to interpret the data correctly. For an example Python script to ingest a `pprof` file with a custom sample type configuration, see

To ingest `pprof` data with custom sample type configuration, modify your requests as follows: Set Content-Type to `multipart/form-data`. Upload the profile data in a form file field named `profile`. Include the sample type configuration in a form file field named `sampletypeconfig`. A sample type configuration is a JSON object formatted like this: ```json { "inuse_space": { "units": "bytes", "aggregation": "average", "display-name": "inusespacebytes", "sampled": false }, "alloc_objects": { "units": "objects", "aggregation": "sum", "display-name": "allocobjectscount", "sampled": true }, "cpu": { "units": "samples", "aggregation": "sum", "display-name": "cpu_samples", "sampled": true }, // pprof supports multiple profiles types in one file, // so there can be multiple of these objects } ``` Explanation of sample type configuration fields: units Supported values: `samples`, `objects`, `bytes` Description: Changes the units displayed in the frontend. `samples` = CPU samples, `objects` = objects in RAM, `bytes` = bytes in RAM. display-name Supported values: Any string. Description: This becomes a suffix of the app name, e.g., `my-app.inusespacebytes`. aggregation Supported values: `sum`, `average`. Description: Alters how data is aggregated on the frontend. Use `sum` for data to be summed over time (e.g., CPU samples, memory allocations), and `average` for data to be averaged over time (e.g., memory in-use objects). sampled Supported values: `true`, `false`. Description: Determines if the sample rate (specified in the pprof file) is considered. Set to `true` for sampled events (e.g., CPU samples), and `false` for memory profiles. This configuration allows for customized visualization and analysis of various profile types within Pyroscope. This is the format, typically used by JVM-based profilers, also supported by our Java integration. When this format is used, some of the query parameters behave slightly different: `format` should be set to `jfr`. `name` contains the prefix of the application name. Since a single request may contain multiple profile types, the final application name is created concatenating this prefix and the profile type. For example, if you send cpu profiling data and set `name` to `my-app{}`, it will appear in pyroscope as `my-app.cpu{}`. `units` is ignored, and the actual units depends on the profile types available in the data. `aggregationType` is ignored, and the actual aggregation type depends on the profile types available in the data. JFR ingestion support uses the profile metadata to determine which profile types are included, which depend on the kind of profiling being done. Currently supported profile types include: `cpu` samples, which includes only profiling data from runnable threads. `itimer` samples, similar to `cpu` profiling. `wall` samples, which includes samples from any threads independently of their state. `allocinnewtlabobjects`, which indicates the number of new TLAB objects created. `allocinnewtlabbytes`, which indicates the size in bytes of new TLAB objects created. `allocoutsidetlab_objects`, which indicates the number of new allocated objects outside any TLAB. `allocinnewtlabbytes`, which indicates the size in bytes of new allocated objects outside any TLAB. In order to ingest JFR data with dynamic labels, you have to make the following changes to your requests: use an HTTP form (`multipart/form-data`) Content-Type. send the JFR data in a form file field called `jfr`. send `LabelsSnapshot` protobuf message in a form file field called

```protobuf message Context { // stringid -> stringid map<int64, int64> labels = 1; } message LabelsSnapshot { // context_id -> Context map<int64, Context> contexts = 1; // string_id -> string map<int64, string> strings = 2; } ``` Where `context_id` is a parameter Here's a sample code that uploads a very simple profile to pyroscope: {{< code >}} ```curl printf "foo;bar 100\n foo;baz 200" | curl \ -X POST \ --data-binary @- \ 'http://localhost:4040/ingest?name=curl-test-app&from=1615709120&until=1615709130' ``` ```python import requests import urllib.parse from datetime import datetime now = round(datetime.now().timestamp()) / 10 * 10 params = {'from': f'{now - 10}', 'name': 'python.example{foo=bar}'} url = f'http://localhost:4040/ingest?{urllib.parse.urlencode(params)}' data = "foo;bar 100\n" \ "foo;baz 200" requests.post(url, data = data) ``` {{< /code >}} Here's a sample code that uploads a JFR profile with labels to pyroscope: {{< code >}} ```curl curl -X POST \ -F jfr=@profile.jfr \ -F labels=@labels.pb \ "http://localhost:4040/ingest?name=curl-test-app&units=samples&aggregationType=sum&sampleRate=100&from=1655834200&until=1655834210&spyName=javaspy&format=jfr" ``` {{< /code >}} There is one primary endpoint for querying profile data: `GET /pyroscope/render`. The search input is provided via query parameters. The output is typically a JSON object containing one or more time series and a flame graph. Here is an overview of the accepted query parameters: | Name | Description | Notes | |:--|:|:--| | `query` | contains the profile type and label selectors | required | | `from` | UNIX time for the start of the search window | required | | `until` | UNIX time for the end of the search window | optional (default is `now`) | | `format` | format of the profiling data | optional (default is `json`) | | `maxNodes` | the maximum number of nodes the resulting flame graph will contain | optional (default is `maxflamegraphnodes_default`) | | `groupBy` | one or more label names to group the time series by (doesn't apply to the flame graph) | optional (default is no grouping) | The `query` parameter is the only required search input. It carries the profile type and any labels we want to use to narrow down the output. The format for this parameter is similar to that of a PromQL query and can be defined as: `<profiletype>{<labelname>="<labelvalue>", <labelname>="<label_value>", ...}` Here is a specific example: `processcpu:cpu:nanoseconds:cpu:nanoseconds{servicename="myapplicationname"}` In a Kubernetes environment, a query could also look like: `processcpu:cpu:nanoseconds:cpu:nanoseconds{namespace="dev", container="myapplication_name"}` {{% admonition type="note" %}} Refer to the for more information and for a list of valid profile types. {{% /admonition %}} The `from` and `until` parameters determine the start and end of the time period for the query. They can be provided in absolute and relative form. Absolute time This table details the options for passing absolute values. | Option | Example | Notes | |:--|:-|:-| | Date | `20231223` | Format: `YYYYMMDD` | | Unix Time seconds | `1577836800` | | | Unix Time milliseconds | `1577836800000` | | | Unix Time microseconds | `1577836800000000` | | | Unix Time nanoseconds | `1577836800000000000` | | Relative time Relative values are always expressed as offsets from `now`. | Option | Example | |:|:| | 3 hours ago | `now-3h` | | 30 minutes ago | `now-30m` | | 2 days ago | `now-2d` | | 1 week ago | `now-7d` or `now-1w` | Note that a single offset has to be provided, values such as `now-3h30m` will not work. Validation The `from` and `until` parameters are subject to validation rules related to `maxquerylookback` and `maxquerylength` server parameters. You can find more details on these parameters in the of the server configuration docs. If `maxquerylookback` is configured and`from` is before `now - maxquerylookback`, `from` will be set to `now - maxquerylookback`. If `maxquerylookback` is configured and `until` is before `now - maxquerylookback` the query will not be

If `maxquerylength` is configured and the query interval is longer than this configuration, the query will no tbe executed. The format can either be: `json`, in which case the response will contain a JSON object `dot`, in which case the response will be text containing a DOT representation of the profile See the section for more information on the response structure. The `maxNodes` parameter truncates the number of elements in the profile response, to allow tools (for example, a frontend) to render large profiles efficiently. This is typically used for profiles that are known to have large stack traces. When no value is provided, the default is taken from the `maxflamegraphnodes_default` configuration parameter. When a value is provided, it is capped to the `maxflamegraphnodes_max` configuration parameter. The `groupBy` parameter impacts the output for the time series portion of the response. When a valid label is provided, the response contains as many series as there are label values for the given label. {{% admonition type="note" %}} Pyroscope supports a single label for the group by functionality. {{% /admonition %}} The output of the `/pyroscope/render` endpoint is a JSON object based on the following : ```go type FlamebearerProfileV1 struct { Flamebearer FlamebearerV1 `json:"flamebearer"` Metadata FlamebearerMetadataV1 `json:"metadata"` Timeline *FlamebearerTimelineV1 `json:"timeline"` Groups map[string]*FlamebearerTimelineV1 `json:"groups"` } ``` The `flamebearer` field contains data in a form suitable for rendering a flame graph. Data within the `flamebearer` is organized in separate arrays containing the profile symbols and the sample values. The `metadata` field contains additional information that is helpful to interpret the `flamebearer` data such as the unit (nanoseconds, bytes), sample rate and more. The `timeline` field represents the time series for the profile. Pyroscope pre-computes the step interval (resolution) of the timeline using the query interval (`from` and `until`). The minimum step interval is 10 seconds. The raw profile sample data is down-sampled to the step interval (resolution) using an aggregation function. Currently only `sum` is supported. A timeline contains a start time, a list of sample values and the step interval: ```json { "timeline": { "startTime": 1577836800, "samples": [ 100, 200, 400 ], "durationDelta": 10 } } ``` The `groups` field is only populated when grouping is requested by the `groupBy` query parameter. When this is the case, the `groups` field has an entry for every label value found for the query. This example groups by a cluster: ```json { "groups": { "eu-west-2": { "startTime": 1577836800, "samples": [ 200, 300, 500 ] }, "us-east-1": { "startTime": 1577836800, "samples": [ 100, 200, 400 ] } } } ``` When the `format` query parameter is `dot`, the endpoint responds with a ) data representing the queried profile. This can be used to create an alternative visualization of the profile. This example queries a local Pyroscope server for a CPU profile from the `pyroscope` service for the last hour. ```curl curl \ 'http://localhost:4040/pyroscope/render?query=processcpu%3Acpu%3Ananoseconds%3Acpu%3Ananoseconds%7Bservicename%3D%22pyroscope%22%7D&from=now-1h' ``` Here is the same query made more readable: ```curl curl --get \ --data-urlencode "query=processcpu:cpu:nanoseconds:cpu:nanoseconds{servicename=\"pyroscope\"}" \ --data-urlencode "from=now-1h" \ http://localhost:4040/pyroscope/render ``` Here is the same example in Python: ```python import requests applicationname = 'myapplication_name' query = f'processcpu:cpu:nanoseconds:cpu:nanoseconds{{servicename="{application_name}"}}' query_from = 'now-1h' url = f'http://localhost:4040/pyroscope/render?query={query}&from={query_from}' requests.get(url) ``` See for a complete example. The `profilecli` tool can also be used to interact with the Pyroscope server API. The tool supports operations such as ingesting profiles, querying for existing profiles, and more. Refer to the page for more information.