Backport System

[Edit this page](https://github.com/ClickHouse/ClickHouse/tree/master/docs/en/development/backports.md)

This document describes the ClickHouse backport policy and the automated system that implements it.

## Release Model[​](#release-model "Direct link to Release Model")

ClickHouse versions follow the scheme `YY.M.patch.build-type`, where `YY` is the two-digit year, `M` is the release month (no leading zero), `patch` is the patch number within the branch, `build` is a monotonically increasing build number, and `type` is either `stable` or `lts`.

Example: `25.3.8.23-lts` — March 2025 LTS, patch 8, build 23.

There are two release tracks:

- **Stable** releases are published roughly monthly. The three most recent stable releases receive patches, giving approximately three months of active support per release.
- **LTS (Long-Term Support)** releases are published in March and August each year. Two LTS versions are supported simultaneously, each for at least 12 months.

Users running production workloads are encouraged to use either the latest stable or an LTS release, and to upgrade to new patch versions promptly, as patch releases do not introduce breaking changes.

## Backport Policy[​](#backport-policy "Direct link to Backport Policy")

Not all changes are backported. The goal is to keep release branches stable, so the scope of backports is intentionally narrow:

- **Security fixes** — always backported.
- **Critical bug fixes** (exceptions (logical errors), data loss, wrong results, RBAC issues) — automatically selected for backporting under the general backport rules; identified by the `pr-critical-bugfix` label, which causes `pr-must-backport` to be added automatically.
- **Stability and regression fixes** — backported when the risk of the change is low relative to the risk of leaving the bug in place; identified by `pr-must-backport` added manually by maintainers.
- **Minor bug fixes with an available workaround** — generally not backported to avoid destabilising release branches.
- **New features, improvements, performance work** — not backported.

The label `pr-must-backport` is the manual override used by maintainers to mark a PR for backporting. The label `pr-critical-bugfix` causes `pr-must-backport` to be added automatically by the CI hook (see `pr_labels_and_category.py`).

**Conflict escalation.** When automatic backporting cannot resolve merge conflicts, a cherry-pick PR must still be created and assigned to the author, merger, and existing assignees of the original PR so that a human can resolve the conflicts and complete the backport.

## Backport Tool[​](#backport-tool "Direct link to Backport Tool")

The backport policy described above is implemented by the automated tool in `tests/ci/cherry_pick.py`. The tool runs as a GitHub Actions workflow on ClickHouse infrastructure and covers all the requirements: discovering active release branches, selecting PRs that qualify for backporting, performing the two-stage cherry-pick and backport procedure, managing conflicts, enforcing the delay policy, and keeping labels in sync.

The long-term goal is to extract this implementation into a standalone open-source Python tool that other projects can adopt. The target design is:

- **Configurable** — all policy parameters (qualifying labels, delay window, stale PR thresholds, rolling-out behaviour, etc.) expressed as a configuration file so the tool can be adapted to match any project's backport requirements without code changes.
- **Distributable** — packaged as a self-contained Python wheel installable from PyPI, with no dependency on ClickHouse's CI infrastructure.
- **Programmable** — exposing a clean object model for pull requests, labels, and release branches so that users can script custom workflows on top of the core engine.

### Testing[​](#testing "Direct link to Testing")

A planned part of the standalone tool is a dedicated test suite together with a lightweight testing infrastructure. The infrastructure will be able to spin up temporary GitHub repositories (or local equivalents) pre-populated with:

- a configurable set of branches representing release lines,
- pull requests carrying various combinations of backport labels,
- release PRs with the `release` label pointing at the release branches.

This lets tests exercise the full automation loop — label detection, cherry-pick branch creation, conflict handling, backport PR creation, assignee logic, rolling-out skip, and delay policy — against a real but disposable repository, without touching production state. The same infrastructure can be reused to regression-test policy changes before they are deployed.

## Active Release Branches[​](#active-release-branches "Direct link to Active Release Branches")

An active release branch is any branch whose corresponding release PR (carrying the `release` label) is still open on GitHub. The backport automation discovers these dynamically on each run, so no configuration changes are needed when a new release is cut or an old one reaches end-of-life.

A release branch can be in a **rolling-out** state (its release PR carries the `rolling-out` label) during the period when a new release is being deployed. General backports are paused for rolling-out branches to avoid complicating the rollout. Version-specific labels (e.g. `v25.3-must-backport`) override this and force backporting even during a rollout.

## Implementation[​](#implementation "Direct link to Implementation")

### Overview[​](#overview "Direct link to Overview")

The backport automation runs hourly as the `CherryPick` GitHub Actions workflow (`.github/workflows/cherry_pick.yml`), implemented in `tests/ci/cherry_pick.py`. It operates through the GitHub API and local git operations on a self-hosted `style-checker-aarch64` runner.

The process is two-stage for each (original PR, release branch) pair:

1. A **cherry-pick PR** is created to isolate conflict resolution from the actual merge target. If there are no conflicts, it is merged automatically.
2. A **backport PR** is created against the real release branch, with the cherry-picked changes squashed into a single commit.

### Labels[​](#labels "Direct link to Labels")

Labels on the original PR control whether and where backporting happens.

|  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| Label Effect|  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  | | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | | `pr-must-backport` Backport to all active release branches (skipping branches marked `rolling-out`)| `pr-must-backport-force` Backport to all active release branches, ignoring `rolling-out` restrictions| `pr-critical-bugfix` Triggers `pr-must-backport` automatically (via `AUTO_BACKPORT` in `pr_labels_and_category.py`)| `v{VER}-must-backport` (e.g. `v25.3-must-backport`) Backport only to that specific release branch; overrides `rolling-out` skip for that branch| `pr-backports-created` Set by the bot when all required backport PRs have been created; cleared if a cherry-pick PR is reopened|  |  |  |  |  |  |  |  | | --- | --- | --- | --- | --- | --- | --- | --- | | `pr-cherrypick` Applied to cherry-pick PRs created by the bot|  |  |  |  |  |  | | --- | --- | --- | --- | --- | --- | | `pr-backport` Applied to backport PRs created by the bot|  |  |  |  | | --- | --- | --- | --- | | `do not test` Applied to cherry-pick PRs so CI does not run on them|  |  | | --- | --- | | `rolling-out` Set on a **release PR** to indicate its branch is currently being rolled out; general backports skip it | | | | | | | | | | | | | | | | | | | |

### Branch and PR Naming[​](#branch-and-pr-naming "Direct link to Branch and PR Naming")

For each original PR number `N` and release branch `release/X.Y`:

- Cherry-pick branch: `cherrypick/release/X.Y/N`
- Backport branch: `backport/release/X.Y/N`
- Cherry-pick PR title: `Cherry pick #N to release/X.Y: <original title>`
- Backport PR title: `Backport #N to release/X.Y: <original title>`