Backport System

### Branch and PR Naming[​](#branch-and-pr-naming "Direct link to Branch and PR Naming")

For each original PR number `N` and release branch `release/X.Y`:

- Cherry-pick branch: `cherrypick/release/X.Y/N`
- Backport branch: `backport/release/X.Y/N`
- Cherry-pick PR title: `Cherry pick #N to release/X.Y: <original title>`
- Backport PR title: `Backport #N to release/X.Y: <original title>`

### Step-by-Step Process[​](#step-by-step-process "Direct link to Step-by-Step Process")

#### 1. Discover active releases[​](#discover-active-releases "Direct link to 1. Discover active releases")

`BackportPRs.receive_release_prs` queries GitHub for all open PRs with the `release` label. The head refs of these PRs are the release branch names (e.g. `release/25.3`). A compatibility label set is derived from them: `v25.3-must-backport`, etc.

#### 2. Find PRs to backport[​](#find-prs-to-backport "Direct link to 2. Find PRs to backport")

`BackportPRs.receive_prs_for_backport` uses the GitHub search API to find merged PRs that:

- carry at least one backport label (`pr-must-backport`, `pr-must-backport-force`, `pr-critical-bugfix`, or a version-specific label), and
- do **not** already have `pr-backports-created`, and
- were merged after the oldest commit date found on any release branch, and
- were updated within the last 90 days (to keep the search query efficient).

#### 3. Rolling-out branch handling[​](#rolling-out-branch-handling "Direct link to 3. Rolling-out branch handling")

When a release PR carries the `rolling-out` label, general backport labels (`pr-must-backport`, `pr-critical-bugfix`) skip that branch. The bot closes any previously created cherry-pick or backport PRs for that branch with an explanatory comment. A version-specific label (e.g. `v25.3-must-backport`) always overrides this. `pr-must-backport-force` bypasses the `rolling-out` check for all branches.

#### 4. Cherry-pick stage (`ReleaseBranch.create_cherrypick`)[​](#cherry-pick-stage "Direct link to cherry-pick-stage")

For each (original PR, release branch) pair where no cherry-pick PR exists yet:

1. Check out the release branch and create a **backport branch** (`backport/release/X.Y/N`) from it.
2. Perform `git merge -s ours` against the first parent of the merge commit to create a synthetic merge base with no content changes.
3. Force-create a **cherry-pick branch** (`cherrypick/release/X.Y/N`) pointing directly at the merge commit of the original PR.
4. Attempt `git merge --no-commit --no-ff` of the cherry-pick branch into the backport branch:
   - If already up-to-date, the change is already present in the release branch — mark as done and skip.
   - Otherwise (with or without conflicts) reset and push both branches.
5. Create the cherry-pick PR targeting `backport/release/X.Y/N` from `cherrypick/release/X.Y/N`, labelled `pr-cherrypick` and `do not test`.
6. Propagate `pr-bugfix` or `pr-critical-bugfix` from the original PR if applicable.
7. Assignees are **not** set at this point; they are only added when conflicts are detected.

#### 5. Auto-merge of conflict-free cherry-pick PRs[​](#auto-merge-conflict-free-cherry-pick-prs "Direct link to 5. Auto-merge of conflict-free cherry-pick PRs")

If the cherry-pick PR is mergeable (no conflicts), the bot merges it automatically via the GitHub API and proceeds immediately to the backport stage.

#### 6. Backport stage (`ReleaseBranch.create_backport`)[​](#backport-stage "Direct link to backport-stage")

After the cherry-pick PR is merged:

1. Check out and pull the backport branch.
2. Find the merge-base between the release branch and the backport branch.
3. `git reset --soft` to the merge-base, squashing all cherry-picked commits into one.
4. Commit with the backport PR title as the message.
5. Force-push the backport branch and open a backport PR targeting the real release branch.
6. Label the PR `pr-backport` (and `pr-bugfix` / `pr-critical-bugfix` if applicable).
7. Assign the PR to the original PR's author, merger, and existing assignees (excluding robot accounts).

#### 7. Completion[​](#completion "Direct link to 7. Completion")

When all release branches for a given original PR are backported, the bot adds `pr-backports-created` to the original PR.

#### 8. Pre-check[​](#pre-check "Direct link to 8. Pre-check")

Before starting any work on a PR, `ReleaseBranch.pre_check` runs `git merge-base --is-ancestor` to verify the merge commit is not already reachable from the release branch. If it is, the PR is considered already backported and skipped.

### Stale Cherry-pick PR Handling[​](#stale-cherry-pick-pr-handling "Direct link to Stale Cherry-pick PR Handling")

The `CherryPickPRs` class runs at the start of each hourly execution and handles two scenarios:

- **Orphaned cherry-pick PRs**: If a cherry-pick PR's release branch no longer has an open release PR (i.e. the release is closed), the cherry-pick PR is closed automatically.
- **Reopened cherry-pick PRs**: If an original PR already has `pr-backports-created` but a cherry-pick PR for it is still open, the `pr-backports-created` label is removed from the original PR so it can be reprocessed.

For cherry-pick PRs waiting for manual conflict resolution:

- After **3 days** of no updates, the bot posts a ping comment mentioning the assignees.
- After **7 days** of no updates, the bot posts a closing comment and closes the PR.

### Conflict Resolution[​](#conflict-resolution "Direct link to Conflict Resolution")

When a cherry-pick has conflicts, the cherry-pick PR is left open for a human to resolve. The bot assigns it to the original PR's author, merger, and assignees. After the conflicts are resolved and the cherry-pick PR is merged, the bot creates the backport PR on the next hourly run.

To discard a backport entirely, close the cherry-pick PR. The bot will treat it as intentionally skipped.

To recreate a broken cherry-pick PR from scratch:

1. Remove the `pr-cherrypick` label from the cherry-pick PR.
2. Delete the `cherrypick/...` branch.
3. Remove `pr-backports-created` from the original PR if present.

### CI for Backport PRs[​](#ci-for-backport-prs "Direct link to CI for Backport PRs")

Backport PRs target release branches, so they use a dedicated CI workflow (`BackportPR`, defined in `ci/workflows/backport_branches.py`) rather than the standard pull request workflow. This workflow runs a representative subset of CI: ASan/UBSan and TSan builds, release builds, macOS builds, functional tests under ASan, stress tests under TSan, and integration tests. It validates that the backport branch has between 1 and 50 commits and at least one changed file (enforced by `check_backport_branch.py`).

### Authentication[​](#authentication "Direct link to Authentication")

The workflow uses an SSH key (`ROBOT_CLICKHOUSE_SSH_KEY`) for git push operations. GitHub API calls are authenticated via `get_best_robot_token`, which selects the token with the most remaining quota from a pool stored in SSM (`/github-tokens`). `ROBOT_CLICKHOUSE_COMMIT_TOKEN` is used by the checkout step in the Actions workflow, not for API calls. Robot accounts (`robot-clickhouse`, `clickhouse-gh`) are excluded when assigning a responsible person.

### GitHub API Cache[​](#github-api-cache "Direct link to GitHub API Cache")

`GitHubCache` (from `cache_utils.py`) persists the PyGithub object cache to S3, reducing API calls across hourly runs. The cache is downloaded at the start and uploaded at the end of each run.

### Error Handling[​](#error-handling "Direct link to Error Handling")

Errors during individual PR processing are caught and logged but do not stop the run. After all PRs are processed, if any errors occurred, a `BackportException` is raised. In CI, this triggers a notification via `CIBuddy` to the team chat.