Project: kcp (Orchestration & Management / Scheduling & Orchestration)
File: rebasing-kubernetes.md

This describes the process of rebasing kcp onto a new Kubernetes version. For the examples below, we'll be rebasing onto v1.26.3. Create a new branch for the update, such as `1.26-prep`. Update go.mod: You may need to change the go version at the top of the file to match what's in go.mod in the root of the Kubernetes repo. This is not required, but probably a good idea. Update the primary Kubernetes dependencies: ``` go get -u k8s.io/api@v0.26.3 k8s.io/apimachinery@v0.26.3 k8s.io/client-go@v0.26.3 ``` Run `go mod tidy`. Manually review the code that is upstream from `third_party` and make the same/similar edits to anything that changed upstream. For example, we maintain a slightly modified copy of the . In between 1.24 (the previous Kubernetes version upon which kcp was based) and 1.26, the `AddEventHandler` function was modified to return a `ResourceEventHandlerRegistration` and an `error`. To rebase kcp-dev/apimachinery onto Kubernetes v1.26.3, we had to make our copy do the same thing. Run `make lint test`. Fix any issues you encounter. Commit your changes. Push to your fork. Open a PR; get it reviewed and merged. Create a new branch for the update, such as `1.26-prep`. Update go.mod: You may need to change the go version at the top of the file to match what's in go.mod in the root of the Kubernetes repo. This is not required, but probably a good idea. Update the primary Kubernetes dependencies: ``` go get -u k8s.io/apimachinery@v0.26.3 k8s.io/code-generator@v0.26.3 ``` Run `go mod tidy`. Update Kubernetes tooling versions in the `Makefile`: `KUBECLIENTGEN_VER := v0.26.3` `KUBELISTERGEN_VER := v0.26.3` `KUBEINFORMERGEN_VER := v0.26.3` `KUBEAPPLYCONFIGURATIONGEN_VER := v0.26.3` Update generators, if needed: Manually review upstream to check for any changes to these generators in the `kubernetes` repository: staging/src/k8s.io/code-generator/cmd/client-gen staging/src/k8s.io/code-generator/cmd/informer-gen staging/src/k8s.io/code-generator/cmd/lister-gen If there were any substantive changes, make the corresponding edits to our generators. For example, in between 1.24 and 1.26, upstream Kubernetes added shutdown support to shared informer factories. We had to add the same thing to our generated shared informer factories. You'll probably want to commit your changes at this point. Run `make codegen`. You'll probably want to commit these changes as a standalone commit. Run `make lint test build` and fix any issues you encounter. Commit any remaining changes. Push to your fork. Open a PR; get it reviewed and merged. Create a new branch for the update, such as `1.26-prep`. Update go.mod: You may need to change the go version at the top of the file to match what's in go.mod in the root of the Kubernetes repo. This is not required, but probably a good idea. Update the `kcp-dev/apimachinery` dependency: ``` go get -u github.com/kcp-dev/apimachinery@main ``` That should have updated the primary Kubernetes dependencies, but in case it didn't, you can do so manually: ``` go get -u k8s.io/api@v0.26.3 k8s.io/apimachinery@v0.26.3 k8s.io/client-go@v0.26.3 ``` Run `go mod tidy`. Update the kcp code-generator tooling version in the `Makefile`: Look up the latest commit from the main branch in https://github.com/kcp-dev/code-generator. Adjust `CODEGENERATORVER` to match accordingly. Manually review the code that is upstream from `third_party` and make the same/similar edits to anything that changed upstream. Run `make

You'll probably want to commit these changes as a standalone commit. Run `make lint` and fix any issues you encounter. Commit any remaining changes. Push to your fork. Open a PR; get it reviewed and merged. Commits merged into `kcp-dev/kubernetes` follow this commit message format: `UPSTREAM: <UPSTREAM PR ID>` The number identifies a PR in upstream kubernetes (i.e. `https://github.com/kubernetes/kubernetes/pull/<pr id>`) A commit with this message should only be picked into the subsequent rebase branch if the commits of the referenced PR are not included in the upstream branch. To check if a given commit is included in the upstream branch, open the referenced upstream PR and check any of its commits for the release tag (e.g. `v.1.26.3`) targeted by the new rebase branch. For example: <img src="commit-tag.png"> `UPSTREAM: <carry>:` A persistent carry that should probably be picked for the subsequent rebase branch. In general, these commits are used to modify behavior for consistency or compatibility with kcp. `UPSTREAM: <drop>:` A carry that should probably not be picked for the subsequent rebase branch. In general, these commits are used to maintain the codebase in ways that are branch-specific, like the update of generated files or dependencies. If the old branch (generally also the default branch) is `upstream/kcp-feature-logical-cluster-1.24-v3` and the old version is `v1.24.3`, then a tsv file containing the set of carry commits that need to be considered for cherry-picking later can be generated using: ```sh echo 'Comment Sha\tSummary\tCommit link\tPR link\tAction' > ~/Documents/v1.24.3.tsv git log $( git merge-base upstream/kcp-feature-logical-clusters-1.24-v3 v1.24.3 )..upstream/kcp-feature-logical-clusters-1.24-v3 --ancestry-path --reverse --no-merges --pretty='tformat:%h%x09%s%x09https://github.com/kcp-dev/kubernetes/commit/%h?w=1' | grep -E $'\t''UPSTREAM: .'$'\t' | sed -E 's~UPSTREAM: ([0-9]+)(:.)~UPSTREAM: \1\2\thttps://github.com/kubernetes/kubernetes/pull/\1~' >> ~/Documents/v1.24.3.tsv ``` The tsv file can be imported into a google sheets spreadsheet to track the progress of picking commits to the new rebase branch. The spreadsheet can also be a way of communicating with rebase reviewers. For an example, please see the . First and foremost, take notes of what worked/didn't work well. Update this guide based on your experiences! Remember, if you mess up, `git rebase --abort` and `git reflog` are your very good friends! Terminology: "Old" version: the current version of Kubernetes that kcp is using "New" version: the new Kubernetes version on top of which you are rebasing The `upstream` in e.g. `upstream/kcp-feature-logical-cluster-1.24-v3` is the name of the git remote that points to github.com/kcp-dev/kubernetes. Please adjust the commands below if your remote is named differently. Prepare the old branch for rebasing: In this example, the old version is 1.24.3, and the new version is 1.26.3. Create a new temporary branch to clean up the accumulated commits on the old branch: ``` git checkout -b kcp-1.24-clean upstream/kcp-feature-logical-cluster-1.24-v3 ``` Start by doing a soft git reset so your working directory contains all the changes from the branch in an uncommitted state: ``` git reset v1.24.3 # this must be whatever upstream commit was the starting point for the old branch ``` Revert the following types of changes: go.mod/go.sum Generated clients, listers, informers, applyconfigurations vendor/* ``` git checkout go.mod go.sum vendor git clean -n -d vendor/* # to check what files would be removed git clean -f

# actually remove the new untracked files ``` Create a single baseline commit: ``` git add . git commit -m 'UPSTREAM: <carry>: baseline for kcp 1.24' ``` Create a baseline branch that starts from the new version. We'll use this to cherry-pick any commits from upstream that we need to continue carrying (i.e., they have not yet merged into the new version). ``` git checkout -b kcp-1.26-baseline v1.26.3 ``` Review the list of carry commits in the spreadsheet generated above. Look for commit messages of the format `UPSTREAM: 12345: ...`. The number indicates the upstream pull request. You'll need to inspect each pull request to determine when it was merged. Anything that merged before the rebase version can be ignored and omitted from the baseline branch. Anything that merged after the rebase version as well as anything not yet merged must be cherry-picked to the baseline branch. For example, in 1.24, we have `UPSTREAM: 111898: Reflector: support logging Unstructured type`. This was added in 1.27, and therefore needs to be cherry-picked to the `kcp-1.26-baseline` branch. Create a branch where you'll attempt to do the rebase. This will start from the cleaned up old version branch: ``` git checkout -b kcp-1.26-rebase kcp-1.24-clean ``` Rebase onto the baseline branch you created above: ``` git rebase kcp-1.26-baseline ``` You will likely encounter numerous conflicts. This is 100% normal! Go through each file and resolve them as best as you can. Make notes of any particular changes you have questions about, so you don't forget them. Update kcp dependencies, for all kcp repositories that changed. For example: ``` hack/pin-dependency.sh github.com/kcp-dev/logicalcluster/v3 v3.0.4 hack/pin-dependency.sh github.com/kcp-dev/apimachinery/v2 767ac05aebce82530dee46e9dab8c7bb47f1c823 hack/pin-dependency.sh github.com/kcp-dev/client-go 654321d8cac56f9944e8cf801dc15e37b3a582f3 ``` Review changes to the origins of pkg/genericcontrolplane to see if similar changes are needed in the new version: | Destination | Source(s) | ||| | pkg/genericcontrolplane | cmd/kube-apiserver/app, pkg/kubeapiserver/admission | | pkg/genericcontrolplane/apis/apis.go | pkg/controlplane/instance.go | | pkg/genericcontrolplane/options.go | cmd/kube-apiserver/app/options/options.go | | pkg/api/genericcontrolplane/scheme.go | pkg/api/legacyscheme/scheme.go | | pkg/apis/core/install/genericcontrolplane/install.go | pkg/apis/core/install/install.go | | pkg/apis/core/registergenericcontrol_plane.go | pkg/apis/core/register.go | | pkg/apis/core/v1/registergenericcontrol_plane.go | pkg/apis/core/v1/register.go | | staging/src/k8s.io/api/core/v1/registergenericcontrol_plane.go | staging/src/k8s.io/api/core/v1/register.go | | pkg/registry/core/rest/genericcontrolplane/storagecore.go | pkg/registry/core/rest/storagecore.go | | pkg/kubeapiserver/legacystoragefactorybuilder.go | pkg/kubeapiserver/defaultstoragefactorybuilder.go | Update generated clients, informers, listers, etc. because we generate logical cluster aware versions of these for Kubernetes to use: ``` hack/update-codegen.sh ``` Check if any new controllers were added to Kubernetes. If so, determine if they are relevant to the control plane, or if they're specific to workloads (anything related to pods/nodes/etc.). If they're for the control plane and you think they should be enabled in kcp, you have 1 of 2 choices: Modify the controller in Kubernetes to be logical cluster aware Add code to kcp to spawn a new controller instance scoped to a single logical cluster For option 1, follow what we did in pkg/controller/namespace/namespace_controller.go. For option 2, follow what we did in kcp in either the garbage collector or quota controllers. Check if any new admission plugins were added to Kubernetes. Decide if we need them in kcp. If so, make a note of them, and we'll add them to kcp below. Push your commits to your fork of Kubernetes in GitHub. Open a pull request for review against the baseline branch, e.g.

but mark it `WIP` and maybe even open it in draft mode - you don't want to merge anything just yet. At this point, you're ready to try to integrate the updates into kcp proper. There is still likely a good amount of work to do, so don't get discouraged if you encounter dozens or hundreds of compilation issues at this point. That's totally normal! Update go.mod: You'll need to adjust the versions of kcp-dev/apimachinery, client-go, and logicalcluster to match any changes you made in previous steps. Make sure the versions that are in the Kubernetes go.mod match the versions used here. If any new staging repositories were added in between the old and new Kubernetes versions, you'll need to add those in the `replace` directive at the bottom of go.mod. For example, in the 1.26 rebase, we had to add 4: k8s.io/dynamic-resource-allocation k8s.io/kms k8s.io/sample-cli-plugin k8s.io/sample-controller Go ahead and make a commit here, as the next change we'll be making is to point kcp at your local checkout of Kubernetes. Point kcp at your local checkout of Kubernetes: ``` KUBE=../../../go/src/k8s.io/kubernetes gsed -i "s,k8s.io/\(.\) => ./kubernetes/.,k8s.io/\1 => $KUBE/vendor/k8s.io/\1,;s,k8s.io/kubernetes => .,k8s.io/kubernetes => $KUBE," go.mod ``` !!! warning Don't commit your changes to go.mod/go.sum. They point to your local file system. Resolve any conflicts Run `make modules` Run `make codegen` Keep iterating to get all the code to compile Get the `lint` and `test` make targets to pass Get the `e2e-*` make targets to pass. Undo your changes to go.mod and go.sum that point to your local checkout: ``` git checkout -- go.mod go.sum ``` Update go.mod to point to your rebase branch in your Kubernetes fork: ``` GITHUB_USER=<your username here> BRANCH=kcp-1.26-rebase hack/bump-k8s.sh ``` Commit this change with a message such as `UNDO: my Kubernetes`. We won't be using this commit when it's time to merge, so we make it easy to find. Open a pull request in kcp. Wait for CI results. The `deps` job will always fail at this point because it's pointing to your fork of Kubernetes. This is expected, so don't worry. Your job at this point is to get all the other CI jobs to pass. Once CI is passing (except for the `deps` job, as expected), we're ready to merge! Coordinate with another project member - show them the test results, then get them to approve your rebase PR in kcp-dev/kubernetes. Get the PR merged (this is something that currently must be done manually - ask someone with merge rights to do it for you if you need help). Rename/create a new branch in kcp-dev/kubernetes based off what you just merged into kcp-1.26-baseline - this is the actual rebase! The new branch could be named something like `kcp-1.26`. Back in your local checkout of kcp, update to the kcp-dev/kubernetes rebase branch: ``` BRANCH=kcp-1.26 hack/bump-k8s.sh ``` Commit this change. You can either squash this with your previous `UNDO` commit (reword it so it's not an `UNDO` any more), or drop the `UNDO` commit and replace it with this one. Check on CI. Hopefully everything is green. If not, keep iterating on it. Change it to your new rebase branch, e.g. `kcp-1.26`