Project: Backstage (App Definition and Development / Application Definition & Image Build)
File: discovery.md

id: discovery title: GitLab Discovery sidebar_label: Discovery description: Automatically discovering catalog entities from repositories in GitLab The GitLab integration has a special entity provider for discovering catalog entities from GitLab. The entity provider will crawl the GitLab instance and register entities matching the configured paths. This can be useful as an alternative to static locations or manually adding things to the catalog. This provider can also be configured to ingest GitLab data based on . The events currently accepted are: . As this provider is not one of the default providers, you will first need to install the gitlab catalog plugin: ```bash yarn --cwd packages/backend add @backstage/plugin-catalog-backend-module-gitlab ``` Then add the following to your backend initialization: ```ts title="packages/backend/src/index.ts // optional if you want HTTP endpoints to receive external events // backend.add(import('@backstage/plugin-events-backend/alpha')); // optional if you want to use AWS SQS instead of HTTP endpoints to receive external events // backend.add(import('@backstage/plugin-events-backend-module-aws-sqs/alpha')); // optional - event router for gitlab. See.: https://github.com/backstage/backstage/blob/master/plugins/events-backend-module-gitlab/README.md // backend.add(eventsModuleGitlabEventRouter()); // optional - token validator for the gitlab topic // backend.add(eventsModuleGitlabWebhook()); backend.add(import('@backstage/plugin-catalog-backend-module-gitlab/alpha')); ``` You need to decide how you want to receive events from external sources like Further documentation: Add the segment below to `packages/backend/src/plugins/catalog.ts`: ```ts title="packages/backend/src/plugins/catalog.ts" / highlight-add-next-line / import { GitlabDiscoveryEntityProvider } from '@backstage/plugin-catalog-backend-module-gitlab'; export default async function createPlugin( env: PluginEnvironment, ): Promise<Router> { const builder = await CatalogBuilder.create(env); / highlight-add-start / builder.addEntityProvider( ...GitlabDiscoveryEntityProvider.fromConfig(env.config, { logger: env.logger, // optional: alternatively, use scheduler with schedule defined in app-config.yaml schedule: env.scheduler.createScheduledTaskRunner({ frequency: { minutes: 30 }, timeout: { minutes: 3 }, }), // optional: alternatively, use schedule scheduler: env.scheduler, }), ); / highlight-add-end / // .. } ``` Please follow the installation instructions at Additionally, you need to decide how you want to receive events from external sources like Set up your provider ```ts title="packages/backend/src/plugins/catalog.ts" import { CatalogBuilder } from '@backstage/plugin-catalog-backend'; / highlight-add-next-line / import { GitlabDiscoveryEntityProvider } from '@backstage/plugin-catalog-backend-module-gitlab'; import { ScaffolderEntitiesProcessor } from '@backstage/plugin-scaffolder-backend'; import { Router } from 'express'; import { PluginEnvironment } from '../types'; export default async function createPlugin( env: PluginEnvironment, ): Promise<Router> { const builder = await CatalogBuilder.create(env); builder.addProcessor(new ScaffolderEntitiesProcessor()); / highlight-add-start / const gitlabProvider = GitlabDiscoveryEntityProvider.fromConfig(env.config, { logger: env.logger, // optional: alternatively, use scheduler with schedule defined in app-config.yaml schedule: env.scheduler.createScheduledTaskRunner({ frequency: { minutes: 30 }, timeout: { minutes: 3 }, }), // optional: alternatively, use schedule scheduler: env.scheduler, }); env.eventBroker.subscribe(gitlabProvider); builder.addEntityProvider(gitlabProvider); / highlight-add-end / const { processingEngine, router } = await builder.build(); await processingEngine.start(); return router; } ``` To use the discovery provider, you'll need a GitLab integration with a

Then you can add a provider config per group to the catalog configuration. > NOTE: if you are using the New Backend System, the `schedule` has to be setup in the config, as shown below. ```yaml title="app-config.yaml" catalog: providers: gitlab: yourProviderId: host: gitlab-host # Identifies one of the hosts set up in the integrations branch: main # Optional. Used to discover on a specific branch fallbackBranch: master # Optional. Fallback to be used if there is no default branch configured at the Gitlab repository. It is only used, if `branch` is undefined. Uses `master` as default skipForkedRepos: false # Optional. If the project is a fork, skip repository group: example-group # Optional. Group and subgroup (if needed) to look for repositories. If not present the whole instance will be scanned entityFilename: catalog-info.yaml # Optional. Defaults to `catalog-info.yaml` projectPattern: '[\s\S]' # Optional. Filters found projects based on provided patter. Defaults to `[\s\S]`, which means to not filter anything schedule: # Same options as in TaskScheduleDefinition. Optional for the Legacy Backend System frequency: { minutes: 30 } timeout: { minutes: 3 } ``` As alternative to the entity provider `GitlabDiscoveryEntityProvider` you can still use the `GitLabDiscoveryProcessor`. Note the `gitlab-discovery` type, as this is not a regular `url` processor. The target is composed of three parts: The base URL, `https://gitlab.com` in this case The group path, `group/subgroup` in this case. This is optional: If you omit this path the processor will scan the entire GitLab instance instead. The path within each repository to find the catalog YAML file. This will usually be `/blob/main/catalog-info.yaml`, `/blob/master/catalog-info.yaml` or a similar variation for catalog files stored in the root directory of each repository. If you want to use the repository's default branch use the `*` wildcard, e.g.: `/blob/*/catalog-info.yaml` Finally, you will have to add the processor in the catalog initialization code of your backend. ```ts title="packages/backend/src/plugins/catalog.ts" / highlight-add-next-line / import { GitLabDiscoveryProcessor } from '@backstage/plugin-catalog-backend-module-gitlab'; export default async function createPlugin( env: PluginEnvironment, ): Promise<Router> { const builder = await CatalogBuilder.create(env); / highlight-add-start / builder.addProcessor( GitLabDiscoveryProcessor.fromConfig(env.config, { logger: env.logger }), ); / highlight-add-end / // .. } ``` And add the following to your app-config.yaml ```yaml catalog: locations: type: gitlab-discovery target: https://gitlab.com/group/subgroup/blob/main/catalog-info.yaml ``` If you don't want create location object if file with component definition do not exists in project, you can set the `skipReposWithoutExactFileMatch` option. That can reduce count of request to gitlab with 404 status code. If you don't want to create location object if the project is a fork, you can set the `skipForkedRepos` option.