Package: gitlab-data-export
GitLab Data Exportgitlab-data-exportis a simple utility that exports data from GitLab's REST API into arbitrary data stores.Basicsgitlab-data-exportis designed to be run from directly within GitLab itself usingscheduled CI
pipelines.
Each time the tool is invoked, it searches for a checkpoint persisted from a previous invocation to determine
the subset of data that should be exported from the API; if no checkpoint can be found, a historical backfill
is run overallGitLab resources to seed the data store.Generally, all data that GitLab provides from its API should be available in the exported data.Data can be exported from GitLab groups or projects. By default, exporting data from a group will also recursively
export all data visible from that group's contained projects and all of its subgroups. This may comprise a large
amount of data; recursive behavior can be disabled by usingoptional configurations.QuickstartBy default,gitlab-data-exportwill export data toBigQuery. This requires a properly configured
GCP project as described inGoogle's offical documentation. Once the project
is configured, exporting data is straightforward:Create aGitLab access tokenwithread_apipermissions
to the resources that you wish to exportCreate aservice account keyfilewithBigQuery Data Editorpermissions in your BigQuery-enabled GCP projectCreate a GitLab project to run the CI pipelineYou can also use an existing project - only a single CI job definition is requiredSet therequired environment variablesin the GitLab project's CI/CD settingsTheBigQuery backendalso has required configurations that must be setAdd the following stanza to the GitLab project'sCI
configuration:gitlab_export:image:python:3.11interruptible:trueneeds:[]before_script:-python3.11 -m pip install gitlab_data_export --upgradescript:-python3.11 -m gitlab_data_export.drivercache:key:globalpaths:-gde_checkpoint.jsonrules:-if:$CI_PIPELINE_SOURCE == "merge_request_event"when:manualallow_failure:true-if:$CI_PIPELINE_SOURCE == "schedule"Optionally,run the pipeline
manuallyto ensure that everything
is working properlyImportant: the first export invocation will exportALLhistory visible from the GitLab resource
that you've configured.This could potentially comprise a large amount of data that will incur BigQuery costs.
If you do not want a historical backfill to be performed, see theoptional configurationfor details
on how to disable this behaviorMerge the CI changesThe pipeline should now run according to your chosen frequency and incrementally export data each time the utility
is invoked. Some query examples specific to the BigQuery storage backend can be found in theBigQuery section below.Configurationgitlab-data-exportsupports a number of configuration options. All configuration is managed via environment variables.RequiredGitLab authentication: eitherGITLAB_TOKENorGITLAB_OAUTHmust be setGITLAB_TOKEN, if set, should contain a GitLab access tokenGITLAB_OAUTH, if set, should contain GitLab OAuth credentialsIf both are set,GITLAB_TOKENwill take precedenceGitLab root resource: eitherGITLAB_GROUPorGITLAB_PROJECTmust be setGITLAB_GROUP, if set, should contain a fully-qualified path to a GitLab groupGITLAB_PROJECT, if set, should contain a fully-qualified path to a GitLab projectOptionalGITLAB_URL: override the GitLab instance from which data should be exportedDefaults tohttps://gitlab.comif unsetGITLAB_BACKFILL_TIMESTAMP: if set, overrides the default historical backfill behavior by limiting theupdated_aftertime of GitLab resources. Accepted values are eithernowor a timestamp in%Y-%m-%d %H:%M:%Sformat.nowdisables backfill completely, while a timestamp limits backfillGITLAB_NO_RECURSION: if set (to any value), completely disables group recursionData storage designData is treated as append-only. Once written, records arenevermodified or deleted. In addition to the data
retrieved from the API, every record is enriched with adata_datefield that users can rely upon in analytical
queries, representing the time at which the records were loaded into BigQuery.The "latest" state for each resource can always be found by grouping byidand selecting the greatestupdated_at. Because all written records contain all information returned by the API, this design provides easy
access to the latest state of each record while also providing a basic form of historical "snapshots".For more granular history, users need only rungitlab-data-exportmore frequently. Naturally, more granular
history also implies a greater level of data duplication - choose your export frequency accordingly. For most
use-cases, daily export should be sufficient.Supported data storesEach supported data store may carry its own bespoke requirements and configurations. These details are described
for each implemented store.BigQueryThe BigQuery backend requires the following configurations:Authentication: eitherGOOGLE_APPLICATION_CREDENTIALSorBIGQUERY_SERVICE_ACCOUNT_JSONmust be setGOOGLE_APPLICATION_CREDENTIALS, if set, should containGoogle default
credentialsBIGQUERY_SERVICE_ACCOUNT_JSON, if set, should containa Google service account
keyBIGQUERY_PROJECT: the globally unique project id for the BigQuery-enabled GCP project to use for data storageThe following optional configurations are also supported:BIGQUERY_DATASET: can be set to override the dataset name that will be populated with GitLab dataDefaults togitlab_data_exportif unsetNote that the dataset name willalwayshave the GitLab API version appended to it for future
compatibility purposes