Project: Backstage (App Definition and Development / Application Definition & Image Build)
File: prometheus-metrics.md

This is a small tutorial that goes over how to setup your Backstage instance to output metrics in a format that can be pulled in by Prometheus. First we need to add the needed dependencies to the `package.json` in the `\packages\backend`: ```diff // packages/backend/package.json "dependencies": { "express-prom-bundle": "^7.0.0", "prom-client": "^15.0.0", ``` Now we want to run `yarn install` from the root of the project to get those dependencies in place Then we need to add a handler for the metrics by creating a file called `metrics.ts` in the `\packages\backend\src` folder Next we add the following content to the `metrics.ts` file: ```ts // packages/backend/src/metrics.ts import { useHotCleanup } from '@backstage/backend-common'; import { RequestHandler } from 'express'; import promBundle from 'express-prom-bundle'; import prom from 'prom-client'; import * as url from 'url'; const rootRegEx = new RegExp('^/([^/])/.'); const apiRegEx = new RegExp('^/api/([^/])/.'); export function normalizePath(req: any): string { const path = url.parse(req.originalUrl || req.url).pathname || '/'; // Capture /api/ and the plugin name if (apiRegEx.test(path)) { return path.replace(apiRegEx, '/api/$1'); } // Only the first path segment at root level return path.replace(rootRegEx, '/$1'); } / Adds a /metrics endpoint, register default runtime metrics and instrument the router. */ export function metricsHandler(): RequestHandler { // We can only initialize the metrics once and have to clean them up between hot reloads useHotCleanup(module, () => prom.register.clear()); return promBundle({ includeMethod: true, includePath: true, // Using includePath alone is problematic, as it will include path labels with high // cardinality (e.g. path params). Instead we would have to template them. However, this // is difficult, as every backend plugin might use different routes. Instead we only take // the first directory of the path, to have at least an idea how each plugin performs: normalizePath, promClient: { collectDefaultMetrics: {} }, }); } ``` Now we will extend the router configuration with the `metricsHandler`: ```diff +import { metricsHandler } from './metrics'; ... const service = createServiceBuilder(module) .loadConfig(config) .addRouter('', await healthcheck(healthcheckEnv)) .addRouter('', metricsHandler()) .addRouter('/api', apiRouter); ``` You now have everything setup, from the `\packages\backend` folder run `yarn start` this will start up the backend Now in a browser load up `http://localhost:7007/metrics`, if everything went smoothly you should see metrics in your browser something like this: The following sections goes over the included and experimental metrics available once you have completed this tutorial This tutorials uses the and the to make this all work. They both come with some built in metrics: `express-prom-bundle` comes with 2 metrics: `up`: this normally will be just 1 `httprequestdurationseconds`: http latency histogram/summary labeled with `statuscode`, `method` and `path` `prom-client` comes with a collection of metrics around memory, CPU, processes, etc. You can see the supported metrics in the `prom-client's` folder. There are some custom metrics that have been added to Backstage will be output for you, these are currently deemed experimental and may be changed or removed in a future release. Here is a rough list, again subject to changes: `catalogentitiescount`: Total amount of entities in the catalog `catalogregisteredlocations_count`: Total amount of registered locations in the catalog `catalogrelationscount`: Total amount of relations between entities `catalogstitchedentities_count`: Amount of entities stitched `catalogprocessedentities_count`: Amount of entities processed `catalogprocessingduration_seconds`: Time spent executing the full processing flow `catalogprocessorsduration_seconds`: Time spent executing catalog processors `catalogprocessingqueuedelayseconds`: The amount of delay between being scheduled for processing, and the start of actually being processed `scaffoldertaskcount`: Tracks successful task runs. `scaffoldertaskduration`: a histogram which tracks the duration of a task run `scaffolderstepcount`: a count that tracks each step run `scaffolderstepduration`: a histogram which tracks the duration of each step run