Repository: kubernetes/test-infra
Pull request #34836: add emulated version upgrade test scripts and config prow job
Author: login: ynfx8
Labels: name: lgtm
description: "Looks good to me", indicates that a PR is ready to be merged.
name: cncf-cla: yes
description: Indicates the PR's author has signed the CNCF CLA.
name: size/L
description: Denotes a PR that changes 100-499 lines, ignoring generated files.
name: approved
description: Indicates a PR has been approved by an approver from all required OWNERS files.
name: sig/testing
description: Categorizes an issue or PR as relevant to SIG Testing.
name: area/jobs
name: area/config
description: Issues or PRs related to code in /config
name: ok-to-test
description: Indicates a non-member PR verified by an org member that is safe to test.
Created: 2025-05-22T01:32:32Z | Merged: 2025-05-22T22:48:34Z

* add test scripts to perform two steps upgrade - binary version upgrade and emulated version upgrade.
* config a n-minus-1 prow job for the new test.
* tests passed locally.


Diff:
diff --git a/config/jobs/kubernetes/sig-testing/compatibility-versions-e2e.yaml b/config/jobs/kubernetes/sig-testing/compatibility-versions-e2e.yaml
index e8d4e3a74b3b..677c2a13ec97 100644
--- a/config/jobs/kubernetes/sig-testing/compatibility-versions-e2e.yaml
+++ b/config/jobs/kubernetes/sig-testing/compatibility-versions-e2e.yaml
@@ -28,7 +28,6 @@ periodics:
   spec:
     containers:
     - image: gcr.io/k8s-staging-test-infra/krte:v20250513-98d205aae3-master
-      imagePullPolicy: Always  # pull latest image for canary testing
       command:
       - wrapper.sh
       - bash
@@ -86,7 +85,6 @@ periodics:
   spec:
     containers:
     - image: gcr.io/k8s-staging-test-infra/krte:v20250513-98d205aae3-master
-      imagePullPolicy: Always  # pull latest image for canary testing
       command:
       - wrapper.sh
       - bash
@@ -165,3 +163,60 @@ periodics:
           # this is mostly for building kubernetes
           memory: 14Gi
           cpu: 7
+- interval: 1h
+  cluster: k8s-infra-prow-build
+  name: ci-kubernetes-e2e-kind-two-steps-upgrade-n-minus-1
+  annotations:
+    testgrid-dashboards: sig-testing-kind
+    testgrid-tab-name: two-steps-upgrade-test-n-minus-1
+    description: Uses kind to run e2e tests from the n-1 kubernetes release against a latest kubernetes master components w/ --emulated-version=n-1 set.
+    # TODO(#34269) update owners in experiment/compatibility-versions
+    testgrid-alert-email: compatibility-version-test@google.com
+    testgrid-num-columns-recent: '6'
+  labels:
+    preset-dind-enabled: "true"
+    preset-kind-volume-mounts: "true"
+  decorate: true
+  decoration_config:
+    timeout: 60m
+  extra_refs:
+  - org: kubernetes
+    repo: kubernetes
+    base_ref: master
+    path_alias: k8s.io/kubernetes
+    workdir: true
+  - org: kubernetes
+    repo: test-infra
+    base_ref: master
+    path_alias: k8s.io/test-infra
+  spec:
+    containers:
+    - image: gcr.io/k8s-staging-test-infra/krte:v20250513-98d205aae3-master
+      command:
+      - wrapper.sh
+      - bash
+      - -c
+      - curl -sSL https://kind.sigs.k8s.io/dl/latest/linux-amd64.tgz | tar xvfz - -C "${PATH%%:*}/" && ./../test-infra/experiment/compatibility-versions/e2e-two-steps-upgrade.sh
+      env:
+      - name: VERSION_DELTA
+        value: "1"
+      - name: SKIP
+        value: Alpha|Disruptive|Slow|Flaky|IPv6|LoadBalancer|PodSecurityPolicy|nfs
+      - name: PARALLEL
+        value: "true"
+      - name: FEATURE_GATES
+        value: '{"AllBeta":true}'
+      - name: RUNTIME_CONFIG
+        value: '{"api/beta":"true", "api/ga":"true"}'
+      # we need privileged mode in order to do docker in docker
+      securityContext:
+        privileged: true
+      resources:
+        limits:
+          memory: 9Gi
+          cpu: 7
+        requests:
+          # these are both a bit below peak usage during build
+          # this is mostly for building kubernetes
+          memory: 9Gi
+          cpu: 7
diff --git a/experiment/compatibility-versions/e2e-two-steps-upgrade.sh b/experiment/compatibility-versions/e2e-two-steps-upgrade.sh
new file mode 100755
index 000000000000..5ccdd257454e
--- /dev/null
+++ b/experiment/compatibility-versions/e2e-two-steps-upgrade.sh
@@ -0,0 +1,135 @@
+#!/usr/bin/env bash
+# Copyright 2024 The Kubernetes Authors.
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+# hack script for running a kind e2e
+# must be run with a kubernetes checkout in $PWD (IE from the checkout)
+# Usage: SKIP="ginkgo skip regex" FOCUS="ginkgo focus regex" kind-e2e.sh
+
+set -o errexit -o nounset -o xtrace
+
+# Settings:
+# SKIP: ginkgo skip regex
+# FOCUS: ginkgo focus regex
+# LABEL_FILTER: ginkgo label query for selecting tests (see "Spec Labels" in https://onsi.github.io/ginkgo/#filtering-specs)
+#
+# The default is to focus on conformance tests. Serial tests get skipped when
+# parallel testing is enabled. Using LABEL_FILTER instead of combining SKIP and
+# FOCUS is recommended (more expressive, easier to read than regexp).
+#
+# GA_ONLY: true  - limit to GA APIs/features as much as possible
+#          false - (default) APIs and features left at defaults
+# FEATURE_GATES:
+#          JSON or YAML encoding of a string/bool map: {"FeatureGateA": true, "FeatureGateB": false}
+#          Enables or disables feature gates in the entire cluster.
+#          Cannot be used when GA_ONLY=true.
+# RUNTIME_CONFIG:
+#          JSON or YAML encoding of a string/string (!) map: {"apia.example.com/v1alpha1": "true", "apib.example.com/v1beta1": "false"}
+#          Enables API groups in the apiserver via --runtime-config.
+#          Cannot be used when GA_ONLY=true.
+
+COMMON_SCRIPT="${COMMON_SCRIPT:-${PWD}/../test-infra/experiment/compatibility-versions/common.sh}"
+source "${COMMON_SCRIPT}"
+
+# setup signal handlers
+# shellcheck disable=SC2317 # this is not unreachable code
+signal_handler() {
+  if [ -n "${GINKGO_PID:-}" ]; then
+    kill -TERM "$GINKGO_PID" || true
+  fi
+  cleanup
+}
+trap signal_handler INT TERM
+
+main() {
+  # create temp dir and setup cleanup
+  TMP_DIR=$(mktemp -d -p /tmp kind-e2e-XXXXXX)
+  echo "Created temporary directory: ${TMP_DIR}"
+
+  # ensure artifacts (results) directory exists when not in CI
+  export ARTIFACTS="${ARTIFACTS:-${PWD}/_artifacts}"
+  mkdir -p "${ARTIFACTS}"
+
+  export VERSION_DELTA=${VERSION_DELTA:-1}
+
+  WORKSPACE_STATUS=$(./hack/print-workspace-status.sh)
+  GIT_VERSION=$(echo "$WORKSPACE_STATUS" | awk '/^gitVersion / {print $2}')
+  # Check if gitVersion contains alpha.0 and increment VERSION_DELTA if needed
+  # If the current version is alpha.0, it means the previous *stable* or developed
+  # branch is actually n-2 relative to the current minor number for compatibility purposes.
+  if [[ "${GIT_VERSION}" == *alpha.0* ]]; then
+    echo "Detected alpha.0 in gitVersion (${GIT_VERSION}), treating as still the previous minor version."
+    VERSION_DELTA=$((VERSION_DELTA + 1))
+    echo "Adjusted VERSION_DELTA: ${VERSION_DELTA}"
+  fi
+
+  MAJOR_VERSION=$(echo "$WORKSPACE_STATUS" | awk '/^STABLE_BUILD_MAJOR_VERSION / {print $2}')
+  MINOR_VERSION=$(echo "$WORKSPACE_STATUS" | awk '/^STABLE_BUILD_MINOR_VERSION / {split($2, minor, "+"); print minor[1]}')
+  export CURRENT_VERSION="${MAJOR_VERSION}.${MINOR_VERSION}"
+  export PREV_VERSION="${MAJOR_VERSION}.$((MINOR_VERSION - VERSION_DELTA))"
+  export EMULATED_VERSION="${PREV_VERSION}"
+
+  # export the KUBECONFIG to a unique path for testing
+  KUBECONFIG="${HOME}/.kube/kind-test-config"
+  export KUBECONFIG
+  echo "exported KUBECONFIG=${KUBECONFIG}"
+
+  # debug kind version
+  kind version
+
+  # in CI attempt to release some memory after building
+  if [ -n "${KUBETEST_IN_DOCKER:-}" ]; then
+    sync || true
+    echo 1 > /proc/sys/vm/drop_caches || true
+  fi
+
+  res=0
+  create_cluster || res=$?
+
+
+  # Perform the upgrade.  Assume kind-upgrade.sh is in the same directory as this script.
+  UPGRADE_SCRIPT="${UPGRADE_SCRIPT:-${PWD}/../test-infra/experiment/compatibility-versions/kind-upgrade.sh}"
+  echo "Upgrading cluster with ${UPGRADE_SCRIPT}"
+
+  upgrade_cluster_components || res=$?
+  # If upgrade fails, we shouldn't run tests but should still clean up.
+  if [[ "$res" -ne 0 ]]; then
+    cleanup
+    exit $res
+  fi
+
+  EMULATED_VERSION_UPGRADE_SCRIPT="${EMULATED_VERSION_UPGRADE_SCRIPT:-${PWD}/../test-infra/experiment/compatibility-versions/emulated-version-upgrade.sh}"
+  echo "Upgrading cluster with ${EMULATED_VERSION_UPGRADE_SCRIPT}"
+  "${EMULATED_VERSION_UPGRADE_SCRIPT}" | tee "${ARTIFACTS}/emulated-upgrade-output.txt"
+
+  # Clone the previous versions Kubernetes release branch
+  # TODO(aaron-prindle) extend the branches to test from n-1 -> n-1..3 as more k8s releases are done that support compatibility versions
+  export PREV_RELEASE_BRANCH="release-${EMULATED_VERSION}"
+  # Define the path within the temp directory for the cloned repo
+  PREV_RELEASE_REPO_PATH="${TMP_DIR}/prev-release-k8s"
+  echo "Cloning branch ${PREV_RELEASE_BRANCH} into ${PREV_RELEASE_REPO_PATH}"
+  git clone --filter=blob:none --single-branch --branch "${PREV_RELEASE_BRANCH}" https://github.com/kubernetes/kubernetes.git "${PREV_RELEASE_REPO_PATH}"
+
+  # enter the cloned prev repo branch (in temp) and run tests
+  pushd "${PREV_RELEASE_REPO_PATH}"
+  build_prev_version_bins || res=$?
+  run_prev_version_tests || res=$?
+  popd
+
+
+  cleanup || res=$?
+  exit $res
+}
+
+main
diff --git a/experiment/compatibility-versions/emulated-version-upgrade.sh b/experiment/compatibility-versions/emulated-version-upgrade.sh
new file mode 100755
index 000000000000..bc5d3933638e
--- /dev/null
+++ b/experiment/compatibility-versions/emulated-version-upgrade.sh
@@ -0,0 +1,95 @@
+#!/usr/bin/env bash
+# Copyright 2025 The Kubernetes Authors.
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+# script adapted from - https://gist.github.com/aojea/2c94034f8e86d08842e5916231eb3fe1
+
+set -e
+set -o pipefail
+
+# Set default values
+CLUSTER_NAME=${CLUSTER_NAME:-kind}
+BUILD_MODE=${BUILD_MODE:-docker}
+UPDATE_KUBE_PROXY=${UPDATE_KUBE_PROXY:-true}
+UPDATE_KUBELET=${UPDATE_KUBE_PROXY:-true}
+# TODO: we can have more granularity here
+UPDATE_CONTROL_PLANE=${UPDATE_CONTROL_PLANE:-true}
+CONTROL_PLANE_COMPONENTS="kube-apiserver kube-controller-manager kube-scheduler"
+
+# Initialize variables
+# Assume go installed
+KUBE_ROOT="."
+source "${KUBE_ROOT}/hack/lib/version.sh"
+kube::version::get_version_vars
+DOCKER_TAG=${KUBE_GIT_VERSION/+/_}
+DOCKER_REGISTRY=${KUBE_DOCKER_REGISTRY:-registry.k8s.io}
+export GOFLAGS="-tags=providerless"
+export KUBE_BUILD_CONFORMANCE=n
+
+# KIND nodes
+NODES=$(kind get nodes --name ${CLUSTER_NAME})
+CONTROL_PLANE_NODES=$(kind get nodes --name ${CLUSTER_NAME} | grep control)
+WORKER_NODES=$(kind get nodes --name ${CLUSTER_NAME} | grep worker)
+
+update_kubelet() {
+ for n in $NODES; do
+   # Backup previous kubelet
+   docker exec $n cp /usr/bin/kubelet /usr/bin/kubelet.bak
+   # Install new kubelet binary
+   docker cp ${KUBELET_BINARY} $n:/usr/bin/kubelet
+   docker exec $n systemctl restart kubelet
+  echo "Updated kubelet on node $n"
+ done
+}
+
+upgrade_emulated_version(){
+  for n in $CONTROL_PLANE_NODES; do
+    for i in $CONTROL_PLANE_COMPONENTS; do
+      docker exec $n sed -e '/\s*--emulated-version=.*/d' -i /etc/kubernetes/manifests/$i.yaml
+      echo "Removed emulated-version flag of component $i static pod on node $n if present"
+      docker exec $n sed -e '/\s*--emulation-forward-compatible=true/d' -i /etc/kubernetes/manifests/$i.yaml
+      echo "Removed emulation-forward-compatible flag of component $i static pod on node $n if present"
+      sleep 10
+    done
+  done
+}
+
+check_emulated_version_removed(){
+  for n in $CONTROL_PLANE_NODES; do
+    for i in $CONTROL_PLANE_COMPONENTS; do
+      if docker exec $n ps aux | grep $i | grep emulated-version; then
+        echo "Error: Component $i still have emulated-version flag"
+        exit 1
+      fi 
+      echo "Confirm component $i does not have --emulated-version flag"
+    done
+  done
+}
+
+# Main
+IMAGES_PATH="${KUBE_ROOT}/_output/release-images/amd64"
+KUBELET_BINARY=$(find ${KUBE_ROOT}/_output/ -type f -name kubelet)
+
+if [[ "$UPDATE_CONTROL_PLANE" == "true" ]]; then
+  upgrade_emulated_version
+fi
+
+if [[ "$UPDATE_KUBELET" == "true" ]]; then
+  update_kubelet
+fi
+
+if [[ "$UPDATE_CONTROL_PLANE" == "true" ]]; then
+  check_emulated_version_removed
+  exit 0
+fi


Reviews:
user: login: BenTheElder
state: COMMENTED
author_association: MEMBER
submitted_at: 2025-05-22T21:43:54Z
user: login: BenTheElder
state: COMMENTED
author_association: MEMBER
submitted_at: 2025-05-22T21:44:28Z
user: login: BenTheElder
state: COMMENTED
author_association: MEMBER
submitted_at: 2025-05-22T21:46:55Z
user: login: BenTheElder
state: COMMENTED
author_association: MEMBER
submitted_at: 2025-05-22T21:48:15Z
user: login: BenTheElder
state: COMMENTED
author_association: MEMBER
submitted_at: 2025-05-22T21:54:00Z
user: login: BenTheElder
body: The imagePullPolicy and bash -x questions are blocking, looking back over the rest.

We will really need to think about rewriting these into something like `kinder` at some point as they grow past small proof of concept, it's hard to review and maintain lots of code with global state and no unit tests (IE typical large shell scripts).
state: COMMENTED
author_association: MEMBER
submitted_at: 2025-05-22T22:00:22Z
user: login: BenTheElder
state: COMMENTED
author_association: MEMBER
submitted_at: 2025-05-22T22:01:24Z
user: login: ynfx8
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-05-22T22:10:30Z
user: login: ynfx8
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-05-22T22:16:00Z
user: login: BenTheElder
state: COMMENTED
author_association: MEMBER
submitted_at: 2025-05-22T22:34:39Z
user: login: ynfx8
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-05-22T22:35:12Z
user: login: BenTheElder
state: COMMENTED
author_association: MEMBER
submitted_at: 2025-05-22T22:35:17Z
user: login: ynfx8
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-05-22T22:37:50Z
user: login: BenTheElder
body: /lgtm
/approve
with follow-ups
state: COMMENTED
author_association: MEMBER
submitted_at: 2025-05-22T22:40:17Z

Review comments:
diff_hunk: @@ -165,3 +165,61 @@ periodics:
           # this is mostly for building kubernetes
           memory: 14Gi
           cpu: 7
+- interval: 1h
+  cluster: k8s-infra-prow-build
+  name: ci-kubernetes-e2e-kind-two-steps-upgrade-n-minus-1
+  annotations:
+    testgrid-dashboards: sig-testing-kind
path: config/jobs/kubernetes/sig-testing/compatibility-versions-e2e.yaml
user: login: BenTheElder
body: Not a blocker for this CL, but please consider it for a future update: cc @siyuanfoundation @aaron-prindle 

I don't think this makes sense for SIG Testing to own at this point, this is primarily testing a SIG API Machinery feature, and not KIND itself.

Let's move these to a SIG API Machinery folder and dashboard (not urgent, but let's do that before it gets a lot of inbound links)
created_at: 2025-05-22T21:43:54Z
author_association: MEMBER
diff_hunk: @@ -165,3 +165,61 @@ periodics:
           # this is mostly for building kubernetes
           memory: 14Gi
           cpu: 7
+- interval: 1h
+  cluster: k8s-infra-prow-build
+  name: ci-kubernetes-e2e-kind-two-steps-upgrade-n-minus-1
+  annotations:
+    testgrid-dashboards: sig-testing-kind
+    testgrid-tab-name: two-steps-upgrade-test-n-minus-1
+    description: Uses kind to run e2e tests from the n-1 kubernetes release against a latest kubernetes master components w/ --emulated-version=n-1 set.
+    # TODO(#34269) update owners in experiment/compatibility-versions
+    testgrid-alert-email: compatibility-version-test@google.com
+    testgrid-num-columns-recent: '6'
+  labels:
+    preset-dind-enabled: "true"
+    preset-kind-volume-mounts: "true"
+  decorate: true
+  decoration_config:
+    timeout: 60m
+  extra_refs:
+  - org: kubernetes
+    repo: kubernetes
+    base_ref: master
+    path_alias: k8s.io/kubernetes
+    workdir: true
+  - org: kubernetes
+    repo: test-infra
+    base_ref: master
+    path_alias: k8s.io/test-infra
+  spec:
+    containers:
+    - image: gcr.io/k8s-staging-test-infra/krte:v20250513-98d205aae3-master
+      imagePullPolicy: Always  # pull latest image for canary testing
path: config/jobs/kubernetes/sig-testing/compatibility-versions-e2e.yaml
user: login: BenTheElder
body: this isn't a canary job and shouldn't set `imagePullPolicy: Always`, it's using a stable image with an immutable tag.
created_at: 2025-05-22T21:44:28Z
author_association: MEMBER
diff_hunk: @@ -165,3 +165,61 @@ periodics:
           # this is mostly for building kubernetes
           memory: 14Gi
           cpu: 7
+- interval: 1h
+  cluster: k8s-infra-prow-build
+  name: ci-kubernetes-e2e-kind-two-steps-upgrade-n-minus-1
+  annotations:
+    testgrid-dashboards: sig-testing-kind
+    testgrid-tab-name: two-steps-upgrade-test-n-minus-1
+    description: Uses kind to run e2e tests from the n-1 kubernetes release against a latest kubernetes master components w/ --emulated-version=n-1 set.
+    # TODO(#34269) update owners in experiment/compatibility-versions
+    testgrid-alert-email: compatibility-version-test@google.com
+    testgrid-num-columns-recent: '6'
+  labels:
+    preset-dind-enabled: "true"
+    preset-kind-volume-mounts: "true"
+  decorate: true
+  decoration_config:
+    timeout: 60m
+  extra_refs:
+  - org: kubernetes
+    repo: kubernetes
+    base_ref: master
+    path_alias: k8s.io/kubernetes
+    workdir: true
+  - org: kubernetes
+    repo: test-infra
+    base_ref: master
+    path_alias: k8s.io/test-infra
+  spec:
+    containers:
+    - image: gcr.io/k8s-staging-test-infra/krte:v20250513-98d205aae3-master
+      imagePullPolicy: Always  # pull latest image for canary testing
+      command:
+      - wrapper.sh
+      - bash
+      - -c
+      - curl -sSL https://kind.sigs.k8s.io/dl/latest/linux-amd64.tgz | tar xvfz - -C "${PATH%%:*}/" && ./../test-infra/experiment/compatibility-versions/e2e-two-steps-upgrade.sh
+      env:
+      - name: VERSION_DELTA
+        value: "1"
+      - name: SKIP
+        value: Alpha|Disruptive|Slow|Flaky|IPv6|LoadBalancer|PodSecurityPolicy|nfs
path: config/jobs/kubernetes/sig-testing/compatibility-versions-e2e.yaml
user: login: BenTheElder
body: TODO: this is no longer the configuration we use in other kind jobs, we set `LABEL_FILTER` to `Feature: isEmpty && !Slow && !Disruptive && !Flaky` ... however I only know that this works for 1.34+ / master. There is one kubectl test that will not work, but I don't see a specific skip for it here either, I think we're getting an implicit `Conformance` focus (we are also implicitly depending on details of the kind e2e scripts which may change ...)

These jobs should either set an explicit FOCUS on migrate skip/focus to an explicit `LABEL_FILTER` (this is a ginkgo --label-filter, see the recent dev@kubernetes.io announcement for details on the labels)
created_at: 2025-05-22T21:46:55Z
author_association: MEMBER
diff_hunk: @@ -165,3 +165,61 @@ periodics:
           # this is mostly for building kubernetes
           memory: 14Gi
           cpu: 7
+- interval: 1h
+  cluster: k8s-infra-prow-build
+  name: ci-kubernetes-e2e-kind-two-steps-upgrade-n-minus-1
+  annotations:
+    testgrid-dashboards: sig-testing-kind
+    testgrid-tab-name: two-steps-upgrade-test-n-minus-1
+    description: Uses kind to run e2e tests from the n-1 kubernetes release against a latest kubernetes master components w/ --emulated-version=n-1 set.
+    # TODO(#34269) update owners in experiment/compatibility-versions
+    testgrid-alert-email: compatibility-version-test@google.com
+    testgrid-num-columns-recent: '6'
+  labels:
+    preset-dind-enabled: "true"
+    preset-kind-volume-mounts: "true"
+  decorate: true
+  decoration_config:
+    timeout: 60m
+  extra_refs:
+  - org: kubernetes
+    repo: kubernetes
+    base_ref: master
+    path_alias: k8s.io/kubernetes
+    workdir: true
+  - org: kubernetes
+    repo: test-infra
+    base_ref: master
+    path_alias: k8s.io/test-infra
+  spec:
+    containers:
+    - image: gcr.io/k8s-staging-test-infra/krte:v20250513-98d205aae3-master
+      imagePullPolicy: Always  # pull latest image for canary testing
+      command:
+      - wrapper.sh
+      - bash
+      - -c
+      - curl -sSL https://kind.sigs.k8s.io/dl/latest/linux-amd64.tgz | tar xvfz - -C "${PATH%%:*}/" && ./../test-infra/experiment/compatibility-versions/e2e-two-steps-upgrade.sh
+      env:
+      - name: VERSION_DELTA
+        value: "1"
+      - name: SKIP
+        value: Alpha|Disruptive|Slow|Flaky|IPv6|LoadBalancer|PodSecurityPolicy|nfs
+      - name: PARALLEL
+        value: "true"
+      - name: FEATURE_GATES
+        value: '{"AllBeta":true}'
+      - name: RUNTIME_CONFIG
+        value: '{"api/beta":"true", "api/ga":"true"}'
+      # we need privileged mode in order to do docker in docker
+      securityContext:
+        privileged: true
+      resources:
+        limits:
+          memory: 9Gi
+          cpu: 7
path: config/jobs/kubernetes/sig-testing/compatibility-versions-e2e.yaml
user: login: BenTheElder
body: If we keep adding more dimensions that need to run continuously on master and later release branches we will need to migrate them to use pre-built images and try to avoid needing all these cores by consuming the CI builds similar to https://github.com/kubernetes/test-infra/issues/33980
created_at: 2025-05-22T21:48:15Z
author_association: MEMBER
diff_hunk: @@ -0,0 +1,135 @@
+#!/usr/bin/env bash
+# Copyright 2024 The Kubernetes Authors.
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+# hack script for running a kind e2e
+# must be run with a kubernetes checkout in $PWD (IE from the checkout)
+# Usage: SKIP="ginkgo skip regex" FOCUS="ginkgo focus regex" kind-e2e.sh
+
+set -o errexit -o nounset -o xtrace
+
+# Settings:
+# SKIP: ginkgo skip regex
+# FOCUS: ginkgo focus regex
+# LABEL_FILTER: ginkgo label query for selecting tests (see "Spec Labels" in https://onsi.github.io/ginkgo/#filtering-specs)
+#
+# The default is to focus on conformance tests. Serial tests get skipped when
+# parallel testing is enabled. Using LABEL_FILTER instead of combining SKIP and
+# FOCUS is recommended (more expressive, easier to read than regexp).
+#
+# GA_ONLY: true  - limit to GA APIs/features as much as possible
+#          false - (default) APIs and features left at defaults
+# FEATURE_GATES:
+#          JSON or YAML encoding of a string/bool map: {"FeatureGateA": true, "FeatureGateB": false}
+#          Enables or disables feature gates in the entire cluster.
+#          Cannot be used when GA_ONLY=true.
+# RUNTIME_CONFIG:
+#          JSON or YAML encoding of a string/string (!) map: {"apia.example.com/v1alpha1": "true", "apib.example.com/v1beta1": "false"}
+#          Enables API groups in the apiserver via --runtime-config.
+#          Cannot be used when GA_ONLY=true.
+
+COMMON_SCRIPT="${COMMON_SCRIPT:-${PWD}/../test-infra/experiment/compatibility-versions/common.sh}"
+source "${COMMON_SCRIPT}"
+
+# setup signal handlers
+# shellcheck disable=SC2317 # this is not unreachable code
+signal_handler() {
+  if [ -n "${GINKGO_PID:-}" ]; then
+    kill -TERM "$GINKGO_PID" || true
+  fi
+  cleanup
+}
+trap signal_handler INT TERM
+
+main() {
+  # create temp dir and setup cleanup
+  TMP_DIR=$(mktemp -d -p /tmp kind-e2e-XXXXXX)
+  echo "Created temporary directory: ${TMP_DIR}"
+
+  # ensure artifacts (results) directory exists when not in CI
+  export ARTIFACTS="${ARTIFACTS:-${PWD}/_artifacts}"
+  mkdir -p "${ARTIFACTS}"
+
+  export VERSION_DELTA=${VERSION_DELTA:-1}
+
+  WORKSPACE_STATUS=$(./hack/print-workspace-status.sh)
+  GIT_VERSION=$(echo "$WORKSPACE_STATUS" | awk '/^gitVersion / {print $2}')
+  # Check if gitVersion contains alpha.0 and increment VERSION_DELTA if needed
+  # If the current version is alpha.0, it means the previous *stable* or developed
+  # branch is actually n-2 relative to the current minor number for compatibility purposes.
+  if [[ "${GIT_VERSION}" == *alpha.0* ]]; then
+    echo "Detected alpha.0 in gitVersion (${GIT_VERSION}), treating as still the previous minor version."
+    VERSION_DELTA=$((VERSION_DELTA + 1))
+    echo "Adjusted VERSION_DELTA: ${VERSION_DELTA}"
+  fi
+
+  MAJOR_VERSION=$(echo "$WORKSPACE_STATUS" | awk '/^STABLE_BUILD_MAJOR_VERSION / {print $2}')
+  MINOR_VERSION=$(echo "$WORKSPACE_STATUS" | awk '/^STABLE_BUILD_MINOR_VERSION / {split($2, minor, "+"); print minor[1]}')
+  export CURRENT_VERSION="${MAJOR_VERSION}.${MINOR_VERSION}"
+  export PREV_VERSION="${MAJOR_VERSION}.$((MINOR_VERSION - VERSION_DELTA))"
+  export EMULATED_VERSION="${PREV_VERSION}"
+
+  # export the KUBECONFIG to a unique path for testing
+  KUBECONFIG="${HOME}/.kube/kind-test-config"
+  export KUBECONFIG
+  echo "exported KUBECONFIG=${KUBECONFIG}"
+
+  # debug kind version
+  kind version
+
+  # in CI attempt to release some memory after building
+  if [ -n "${KUBETEST_IN_DOCKER:-}" ]; then
+    sync || true
+    echo 1 > /proc/sys/vm/drop_caches || true
+  fi
+
+  res=0
+  create_cluster || res=$?
+
+
+  # Perform the upgrade.  Assume kind-upgrade.sh is in the same directory as this script.
+  UPGRADE_SCRIPT="${UPGRADE_SCRIPT:-${PWD}/../test-infra/experiment/compatibility-versions/kind-upgrade.sh}"
+  echo "Upgrading cluster with ${UPGRADE_SCRIPT}"
+
+  upgrade_cluster_components || res=$?
+  # If upgrade fails, we shouldn't run tests but should still clean up.
+  if [[ "$res" -ne 0 ]]; then
+    cleanup
+    exit $res
+  fi
+
+  EMULATED_VERSION_UPGRADE_SCRIPT="${EMULATED_VERSION_UPGRADE_SCRIPT:-${PWD}/../test-infra/experiment/compatibility-versions/emulated-version-upgrade.sh}"
+  echo "Upgrading cluster with ${EMULATED_VERSION_UPGRADE_SCRIPT}"
+  bash -x "${EMULATED_VERSION_UPGRADE_SCRIPT}" | tee "${ARTIFACTS}/emulated-upgrade-output.txt"
path: experiment/compatibility-versions/e2e-two-steps-upgrade.sh
user: login: BenTheElder
body: what is `bash -x`? any reason we don't just exec the script directly and rely on the shebang?

I actually don't see this in the manpage (versus `bash -c 'literal-shell-snippet && true' ...)
created_at: 2025-05-22T21:54:00Z
author_association: MEMBER
diff_hunk: @@ -0,0 +1,95 @@
+#!/usr/bin/env bash
+# Copyright 2025 The Kubernetes Authors.
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+# script adapted from - https://gist.github.com/aojea/2c94034f8e86d08842e5916231eb3fe1
+
+set -e
+set -o pipefail
+
+# Set default values
+CLUSTER_NAME=${CLUSTER_NAME:-kind}
+BUILD_MODE=${BUILD_MODE:-docker}
+UPDATE_KUBE_PROXY=${UPDATE_KUBE_PROXY:-true}
+UPDATE_KUBELET=${UPDATE_KUBE_PROXY:-true}
+# TODO: we can have more granularity here
+UPDATE_CONTROL_PLANE=${UPDATE_CONTROL_PLANE:-true}
+CONTROL_PLANE_COMPONENTS="kube-apiserver kube-controller-manager kube-scheduler"
+
+# Initialize variables
+# Assume go installed
+KUBE_ROOT="."
+source "${KUBE_ROOT}/hack/lib/version.sh"
+kube::version::get_version_vars
+DOCKER_TAG=${KUBE_GIT_VERSION/+/_}
+DOCKER_REGISTRY=${KUBE_DOCKER_REGISTRY:-registry.k8s.io}
+export GOFLAGS="-tags=providerless"
+export KUBE_BUILD_CONFORMANCE=n
+
+# KIND nodes
+NODES=$(kind get nodes --name ${CLUSTER_NAME})
+CONTROL_PLANE_NODES=$(kind get nodes --name ${CLUSTER_NAME} | grep control)
+WORKER_NODES=$(kind get nodes --name ${CLUSTER_NAME} | grep worker)
+
+update_kubelet() {
+ for n in $NODES; do
+   # Backup previous kubelet
+   docker exec $n cp /usr/bin/kubelet /usr/bin/kubelet.bak
+   # Install new kubelet binary
+   docker cp ${KUBELET_BINARY} $n:/usr/bin/kubelet
+   docker exec $n systemctl restart kubelet
+  echo "Updated kubelet on node $n"
+ done
+}
+
+upgrade_emulated_version(){
+  for n in $CONTROL_PLANE_NODES; do
+    for i in $CONTROL_PLANE_COMPONENTS; do
+      docker exec $n sed -e '/\s*--emulated-version=.*/d' -i /etc/kubernetes/manifests/$i.yaml
+      echo "Removed emulated-version flag of component $i static pod on node $n if present"
+      docker exec $n sed -e '/\s*--emulation-forward-compatible=true/d' -i /etc/kubernetes/manifests/$i.yaml
+      echo "Removed emulation-forward-compatible flag of component $i static pod on node $n if present"
+      sleep 10
+    done
+  done
+}
+
+check_emulated_version_removed(){
+  for n in $CONTROL_PLANE_NODES; do
+    for i in $CONTROL_PLANE_COMPONENTS; do
+      if docker exec $n ps aux | grep $i | grep emulated-version; then
+        echo "Error: Component $i still have emulated-version flag"
+        exit 1
+      fi 
+      echo "Confirm component $i does not have --emulated-version flag"
+    done
+  done
+}
+
+# Main
+IMAGES_PATH="${KUBE_ROOT}/_output/release-images/amd64"
path: experiment/compatibility-versions/emulated-version-upgrade.sh
user: login: BenTheElder
body: TODO: arm64 should also work. We can use `$(go env GOARCH)`  most places we're using hard-coded arch instead.
created_at: 2025-05-22T22:01:24Z
author_association: MEMBER
diff_hunk: @@ -0,0 +1,135 @@
+#!/usr/bin/env bash
+# Copyright 2024 The Kubernetes Authors.
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+# hack script for running a kind e2e
+# must be run with a kubernetes checkout in $PWD (IE from the checkout)
+# Usage: SKIP="ginkgo skip regex" FOCUS="ginkgo focus regex" kind-e2e.sh
+
+set -o errexit -o nounset -o xtrace
+
+# Settings:
+# SKIP: ginkgo skip regex
+# FOCUS: ginkgo focus regex
+# LABEL_FILTER: ginkgo label query for selecting tests (see "Spec Labels" in https://onsi.github.io/ginkgo/#filtering-specs)
+#
+# The default is to focus on conformance tests. Serial tests get skipped when
+# parallel testing is enabled. Using LABEL_FILTER instead of combining SKIP and
+# FOCUS is recommended (more expressive, easier to read than regexp).
+#
+# GA_ONLY: true  - limit to GA APIs/features as much as possible
+#          false - (default) APIs and features left at defaults
+# FEATURE_GATES:
+#          JSON or YAML encoding of a string/bool map: {"FeatureGateA": true, "FeatureGateB": false}
+#          Enables or disables feature gates in the entire cluster.
+#          Cannot be used when GA_ONLY=true.
+# RUNTIME_CONFIG:
+#          JSON or YAML encoding of a string/string (!) map: {"apia.example.com/v1alpha1": "true", "apib.example.com/v1beta1": "false"}
+#          Enables API groups in the apiserver via --runtime-config.
+#          Cannot be used when GA_ONLY=true.
+
+COMMON_SCRIPT="${COMMON_SCRIPT:-${PWD}/../test-infra/experiment/compatibility-versions/common.sh}"
+source "${COMMON_SCRIPT}"
+
+# setup signal handlers
+# shellcheck disable=SC2317 # this is not unreachable code
+signal_handler() {
+  if [ -n "${GINKGO_PID:-}" ]; then
+    kill -TERM "$GINKGO_PID" || true
+  fi
+  cleanup
+}
+trap signal_handler INT TERM
+
+main() {
+  # create temp dir and setup cleanup
+  TMP_DIR=$(mktemp -d -p /tmp kind-e2e-XXXXXX)
+  echo "Created temporary directory: ${TMP_DIR}"
+
+  # ensure artifacts (results) directory exists when not in CI
+  export ARTIFACTS="${ARTIFACTS:-${PWD}/_artifacts}"
+  mkdir -p "${ARTIFACTS}"
+
+  export VERSION_DELTA=${VERSION_DELTA:-1}
+
+  WORKSPACE_STATUS=$(./hack/print-workspace-status.sh)
+  GIT_VERSION=$(echo "$WORKSPACE_STATUS" | awk '/^gitVersion / {print $2}')
+  # Check if gitVersion contains alpha.0 and increment VERSION_DELTA if needed
+  # If the current version is alpha.0, it means the previous *stable* or developed
+  # branch is actually n-2 relative to the current minor number for compatibility purposes.
+  if [[ "${GIT_VERSION}" == *alpha.0* ]]; then
+    echo "Detected alpha.0 in gitVersion (${GIT_VERSION}), treating as still the previous minor version."
+    VERSION_DELTA=$((VERSION_DELTA + 1))
+    echo "Adjusted VERSION_DELTA: ${VERSION_DELTA}"
+  fi
+
+  MAJOR_VERSION=$(echo "$WORKSPACE_STATUS" | awk '/^STABLE_BUILD_MAJOR_VERSION / {print $2}')
+  MINOR_VERSION=$(echo "$WORKSPACE_STATUS" | awk '/^STABLE_BUILD_MINOR_VERSION / {split($2, minor, "+"); print minor[1]}')
+  export CURRENT_VERSION="${MAJOR_VERSION}.${MINOR_VERSION}"
+  export PREV_VERSION="${MAJOR_VERSION}.$((MINOR_VERSION - VERSION_DELTA))"
+  export EMULATED_VERSION="${PREV_VERSION}"
+
+  # export the KUBECONFIG to a unique path for testing
+  KUBECONFIG="${HOME}/.kube/kind-test-config"
+  export KUBECONFIG
+  echo "exported KUBECONFIG=${KUBECONFIG}"
+
+  # debug kind version
+  kind version
+
+  # in CI attempt to release some memory after building
+  if [ -n "${KUBETEST_IN_DOCKER:-}" ]; then
+    sync || true
+    echo 1 > /proc/sys/vm/drop_caches || true
+  fi
+
+  res=0
+  create_cluster || res=$?
+
+
+  # Perform the upgrade.  Assume kind-upgrade.sh is in the same directory as this script.
+  UPGRADE_SCRIPT="${UPGRADE_SCRIPT:-${PWD}/../test-infra/experiment/compatibility-versions/kind-upgrade.sh}"
+  echo "Upgrading cluster with ${UPGRADE_SCRIPT}"
+
+  upgrade_cluster_components || res=$?
+  # If upgrade fails, we shouldn't run tests but should still clean up.
+  if [[ "$res" -ne 0 ]]; then
+    cleanup
+    exit $res
+  fi
+
+  EMULATED_VERSION_UPGRADE_SCRIPT="${EMULATED_VERSION_UPGRADE_SCRIPT:-${PWD}/../test-infra/experiment/compatibility-versions/emulated-version-upgrade.sh}"
+  echo "Upgrading cluster with ${EMULATED_VERSION_UPGRADE_SCRIPT}"
+  bash -x "${EMULATED_VERSION_UPGRADE_SCRIPT}" | tee "${ARTIFACTS}/emulated-upgrade-output.txt"
path: experiment/compatibility-versions/e2e-two-steps-upgrade.sh
user: login: ynfx8
body: enable trace mode. This means that before each command is executed, bash will print the command to standard error after expanding all variables and redirections.
created_at: 2025-05-22T22:10:30Z
author_association: CONTRIBUTOR
diff_hunk: @@ -0,0 +1,135 @@
+#!/usr/bin/env bash
+# Copyright 2024 The Kubernetes Authors.
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+# hack script for running a kind e2e
+# must be run with a kubernetes checkout in $PWD (IE from the checkout)
+# Usage: SKIP="ginkgo skip regex" FOCUS="ginkgo focus regex" kind-e2e.sh
+
+set -o errexit -o nounset -o xtrace
+
+# Settings:
+# SKIP: ginkgo skip regex
+# FOCUS: ginkgo focus regex
+# LABEL_FILTER: ginkgo label query for selecting tests (see "Spec Labels" in https://onsi.github.io/ginkgo/#filtering-specs)
+#
+# The default is to focus on conformance tests. Serial tests get skipped when
+# parallel testing is enabled. Using LABEL_FILTER instead of combining SKIP and
+# FOCUS is recommended (more expressive, easier to read than regexp).
+#
+# GA_ONLY: true  - limit to GA APIs/features as much as possible
+#          false - (default) APIs and features left at defaults
+# FEATURE_GATES:
+#          JSON or YAML encoding of a string/bool map: {"FeatureGateA": true, "FeatureGateB": false}
+#          Enables or disables feature gates in the entire cluster.
+#          Cannot be used when GA_ONLY=true.
+# RUNTIME_CONFIG:
+#          JSON or YAML encoding of a string/string (!) map: {"apia.example.com/v1alpha1": "true", "apib.example.com/v1beta1": "false"}
+#          Enables API groups in the apiserver via --runtime-config.
+#          Cannot be used when GA_ONLY=true.
+
+COMMON_SCRIPT="${COMMON_SCRIPT:-${PWD}/../test-infra/experiment/compatibility-versions/common.sh}"
+source "${COMMON_SCRIPT}"
+
+# setup signal handlers
+# shellcheck disable=SC2317 # this is not unreachable code
+signal_handler() {
+  if [ -n "${GINKGO_PID:-}" ]; then
+    kill -TERM "$GINKGO_PID" || true
+  fi
+  cleanup
+}
+trap signal_handler INT TERM
+
+main() {
+  # create temp dir and setup cleanup
+  TMP_DIR=$(mktemp -d -p /tmp kind-e2e-XXXXXX)
+  echo "Created temporary directory: ${TMP_DIR}"
+
+  # ensure artifacts (results) directory exists when not in CI
+  export ARTIFACTS="${ARTIFACTS:-${PWD}/_artifacts}"
+  mkdir -p "${ARTIFACTS}"
+
+  export VERSION_DELTA=${VERSION_DELTA:-1}
+
+  WORKSPACE_STATUS=$(./hack/print-workspace-status.sh)
+  GIT_VERSION=$(echo "$WORKSPACE_STATUS" | awk '/^gitVersion / {print $2}')
+  # Check if gitVersion contains alpha.0 and increment VERSION_DELTA if needed
+  # If the current version is alpha.0, it means the previous *stable* or developed
+  # branch is actually n-2 relative to the current minor number for compatibility purposes.
+  if [[ "${GIT_VERSION}" == *alpha.0* ]]; then
+    echo "Detected alpha.0 in gitVersion (${GIT_VERSION}), treating as still the previous minor version."
+    VERSION_DELTA=$((VERSION_DELTA + 1))
+    echo "Adjusted VERSION_DELTA: ${VERSION_DELTA}"
+  fi
+
+  MAJOR_VERSION=$(echo "$WORKSPACE_STATUS" | awk '/^STABLE_BUILD_MAJOR_VERSION / {print $2}')
+  MINOR_VERSION=$(echo "$WORKSPACE_STATUS" | awk '/^STABLE_BUILD_MINOR_VERSION / {split($2, minor, "+"); print minor[1]}')
+  export CURRENT_VERSION="${MAJOR_VERSION}.${MINOR_VERSION}"
+  export PREV_VERSION="${MAJOR_VERSION}.$((MINOR_VERSION - VERSION_DELTA))"
+  export EMULATED_VERSION="${PREV_VERSION}"
+
+  # export the KUBECONFIG to a unique path for testing
+  KUBECONFIG="${HOME}/.kube/kind-test-config"
+  export KUBECONFIG
+  echo "exported KUBECONFIG=${KUBECONFIG}"
+
+  # debug kind version
+  kind version
+
+  # in CI attempt to release some memory after building
+  if [ -n "${KUBETEST_IN_DOCKER:-}" ]; then
+    sync || true
+    echo 1 > /proc/sys/vm/drop_caches || true
+  fi
+
+  res=0
+  create_cluster || res=$?
+
+
+  # Perform the upgrade.  Assume kind-upgrade.sh is in the same directory as this script.
+  UPGRADE_SCRIPT="${UPGRADE_SCRIPT:-${PWD}/../test-infra/experiment/compatibility-versions/kind-upgrade.sh}"
+  echo "Upgrading cluster with ${UPGRADE_SCRIPT}"
+
+  upgrade_cluster_components || res=$?
+  # If upgrade fails, we shouldn't run tests but should still clean up.
+  if [[ "$res" -ne 0 ]]; then
+    cleanup
+    exit $res
+  fi
+
+  EMULATED_VERSION_UPGRADE_SCRIPT="${EMULATED_VERSION_UPGRADE_SCRIPT:-${PWD}/../test-infra/experiment/compatibility-versions/emulated-version-upgrade.sh}"
+  echo "Upgrading cluster with ${EMULATED_VERSION_UPGRADE_SCRIPT}"
+  bash -x "${EMULATED_VERSION_UPGRADE_SCRIPT}" | tee "${ARTIFACTS}/emulated-upgrade-output.txt"
path: experiment/compatibility-versions/e2e-two-steps-upgrade.sh
user: login: ynfx8
body: I removed -x. it's optional
created_at: 2025-05-22T22:16:00Z
author_association: CONTRIBUTOR
diff_hunk: @@ -0,0 +1,135 @@
+#!/usr/bin/env bash
+# Copyright 2024 The Kubernetes Authors.
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+# hack script for running a kind e2e
+# must be run with a kubernetes checkout in $PWD (IE from the checkout)
+# Usage: SKIP="ginkgo skip regex" FOCUS="ginkgo focus regex" kind-e2e.sh
+
+set -o errexit -o nounset -o xtrace
+
+# Settings:
+# SKIP: ginkgo skip regex
+# FOCUS: ginkgo focus regex
+# LABEL_FILTER: ginkgo label query for selecting tests (see "Spec Labels" in https://onsi.github.io/ginkgo/#filtering-specs)
+#
+# The default is to focus on conformance tests. Serial tests get skipped when
+# parallel testing is enabled. Using LABEL_FILTER instead of combining SKIP and
+# FOCUS is recommended (more expressive, easier to read than regexp).
+#
+# GA_ONLY: true  - limit to GA APIs/features as much as possible
+#          false - (default) APIs and features left at defaults
+# FEATURE_GATES:
+#          JSON or YAML encoding of a string/bool map: {"FeatureGateA": true, "FeatureGateB": false}
+#          Enables or disables feature gates in the entire cluster.
+#          Cannot be used when GA_ONLY=true.
+# RUNTIME_CONFIG:
+#          JSON or YAML encoding of a string/string (!) map: {"apia.example.com/v1alpha1": "true", "apib.example.com/v1beta1": "false"}
+#          Enables API groups in the apiserver via --runtime-config.
+#          Cannot be used when GA_ONLY=true.
+
+COMMON_SCRIPT="${COMMON_SCRIPT:-${PWD}/../test-infra/experiment/compatibility-versions/common.sh}"
+source "${COMMON_SCRIPT}"
+
+# setup signal handlers
+# shellcheck disable=SC2317 # this is not unreachable code
+signal_handler() {
+  if [ -n "${GINKGO_PID:-}" ]; then
+    kill -TERM "$GINKGO_PID" || true
+  fi
+  cleanup
+}
+trap signal_handler INT TERM
+
+main() {
+  # create temp dir and setup cleanup
+  TMP_DIR=$(mktemp -d -p /tmp kind-e2e-XXXXXX)
+  echo "Created temporary directory: ${TMP_DIR}"
+
+  # ensure artifacts (results) directory exists when not in CI
+  export ARTIFACTS="${ARTIFACTS:-${PWD}/_artifacts}"
+  mkdir -p "${ARTIFACTS}"
+
+  export VERSION_DELTA=${VERSION_DELTA:-1}
+
+  WORKSPACE_STATUS=$(./hack/print-workspace-status.sh)
+  GIT_VERSION=$(echo "$WORKSPACE_STATUS" | awk '/^gitVersion / {print $2}')
+  # Check if gitVersion contains alpha.0 and increment VERSION_DELTA if needed
+  # If the current version is alpha.0, it means the previous *stable* or developed
+  # branch is actually n-2 relative to the current minor number for compatibility purposes.
+  if [[ "${GIT_VERSION}" == *alpha.0* ]]; then
+    echo "Detected alpha.0 in gitVersion (${GIT_VERSION}), treating as still the previous minor version."
+    VERSION_DELTA=$((VERSION_DELTA + 1))
+    echo "Adjusted VERSION_DELTA: ${VERSION_DELTA}"
+  fi
+
+  MAJOR_VERSION=$(echo "$WORKSPACE_STATUS" | awk '/^STABLE_BUILD_MAJOR_VERSION / {print $2}')
+  MINOR_VERSION=$(echo "$WORKSPACE_STATUS" | awk '/^STABLE_BUILD_MINOR_VERSION / {split($2, minor, "+"); print minor[1]}')
+  export CURRENT_VERSION="${MAJOR_VERSION}.${MINOR_VERSION}"
+  export PREV_VERSION="${MAJOR_VERSION}.$((MINOR_VERSION - VERSION_DELTA))"
+  export EMULATED_VERSION="${PREV_VERSION}"
+
+  # export the KUBECONFIG to a unique path for testing
+  KUBECONFIG="${HOME}/.kube/kind-test-config"
+  export KUBECONFIG
+  echo "exported KUBECONFIG=${KUBECONFIG}"
+
+  # debug kind version
+  kind version
+
+  # in CI attempt to release some memory after building
+  if [ -n "${KUBETEST_IN_DOCKER:-}" ]; then
+    sync || true
+    echo 1 > /proc/sys/vm/drop_caches || true
+  fi
+
+  res=0
+  create_cluster || res=$?
+
+
+  # Perform the upgrade.  Assume kind-upgrade.sh is in the same directory as this script.
+  UPGRADE_SCRIPT="${UPGRADE_SCRIPT:-${PWD}/../test-infra/experiment/compatibility-versions/kind-upgrade.sh}"
+  echo "Upgrading cluster with ${UPGRADE_SCRIPT}"
+
+  upgrade_cluster_components || res=$?
+  # If upgrade fails, we shouldn't run tests but should still clean up.
+  if [[ "$res" -ne 0 ]]; then
+    cleanup
+    exit $res
+  fi
+
+  EMULATED_VERSION_UPGRADE_SCRIPT="${EMULATED_VERSION_UPGRADE_SCRIPT:-${PWD}/../test-infra/experiment/compatibility-versions/emulated-version-upgrade.sh}"
+  echo "Upgrading cluster with ${EMULATED_VERSION_UPGRADE_SCRIPT}"
+  bash -x "${EMULATED_VERSION_UPGRADE_SCRIPT}" | tee "${ARTIFACTS}/emulated-upgrade-output.txt"
path: experiment/compatibility-versions/e2e-two-steps-upgrade.sh
user: login: BenTheElder
body: Ah, so this is `set -x` but via the CLI? TIL

trace mode can be pretty noisy, I recommend tracing specific values.

I also don't think we should invoke it via `bash`, we can just `"${EMULATED_VERSION_UPGRADE_SCRIPT}" | ...`, and let the script decide which shell to use (via the shebang AKA `#!...`)
created_at: 2025-05-22T22:34:39Z
author_association: MEMBER
diff_hunk: @@ -165,3 +165,61 @@ periodics:
           # this is mostly for building kubernetes
           memory: 14Gi
           cpu: 7
+- interval: 1h
+  cluster: k8s-infra-prow-build
+  name: ci-kubernetes-e2e-kind-two-steps-upgrade-n-minus-1
+  annotations:
+    testgrid-dashboards: sig-testing-kind
+    testgrid-tab-name: two-steps-upgrade-test-n-minus-1
+    description: Uses kind to run e2e tests from the n-1 kubernetes release against a latest kubernetes master components w/ --emulated-version=n-1 set.
+    # TODO(#34269) update owners in experiment/compatibility-versions
+    testgrid-alert-email: compatibility-version-test@google.com
+    testgrid-num-columns-recent: '6'
+  labels:
+    preset-dind-enabled: "true"
+    preset-kind-volume-mounts: "true"
+  decorate: true
+  decoration_config:
+    timeout: 60m
+  extra_refs:
+  - org: kubernetes
+    repo: kubernetes
+    base_ref: master
+    path_alias: k8s.io/kubernetes
+    workdir: true
+  - org: kubernetes
+    repo: test-infra
+    base_ref: master
+    path_alias: k8s.io/test-infra
+  spec:
+    containers:
+    - image: gcr.io/k8s-staging-test-infra/krte:v20250513-98d205aae3-master
+      imagePullPolicy: Always  # pull latest image for canary testing
path: config/jobs/kubernetes/sig-testing/compatibility-versions-e2e.yaml
user: login: ynfx8
body: removed all the imagePullPolicy from the config file
created_at: 2025-05-22T22:35:12Z
author_association: CONTRIBUTOR
diff_hunk: @@ -0,0 +1,135 @@
+#!/usr/bin/env bash
+# Copyright 2024 The Kubernetes Authors.
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+# hack script for running a kind e2e
+# must be run with a kubernetes checkout in $PWD (IE from the checkout)
+# Usage: SKIP="ginkgo skip regex" FOCUS="ginkgo focus regex" kind-e2e.sh
+
+set -o errexit -o nounset -o xtrace
+
+# Settings:
+# SKIP: ginkgo skip regex
+# FOCUS: ginkgo focus regex
+# LABEL_FILTER: ginkgo label query for selecting tests (see "Spec Labels" in https://onsi.github.io/ginkgo/#filtering-specs)
+#
+# The default is to focus on conformance tests. Serial tests get skipped when
+# parallel testing is enabled. Using LABEL_FILTER instead of combining SKIP and
+# FOCUS is recommended (more expressive, easier to read than regexp).
+#
+# GA_ONLY: true  - limit to GA APIs/features as much as possible
+#          false - (default) APIs and features left at defaults
+# FEATURE_GATES:
+#          JSON or YAML encoding of a string/bool map: {"FeatureGateA": true, "FeatureGateB": false}
+#          Enables or disables feature gates in the entire cluster.
+#          Cannot be used when GA_ONLY=true.
+# RUNTIME_CONFIG:
+#          JSON or YAML encoding of a string/string (!) map: {"apia.example.com/v1alpha1": "true", "apib.example.com/v1beta1": "false"}
+#          Enables API groups in the apiserver via --runtime-config.
+#          Cannot be used when GA_ONLY=true.
+
+COMMON_SCRIPT="${COMMON_SCRIPT:-${PWD}/../test-infra/experiment/compatibility-versions/common.sh}"
+source "${COMMON_SCRIPT}"
+
+# setup signal handlers
+# shellcheck disable=SC2317 # this is not unreachable code
+signal_handler() {
+  if [ -n "${GINKGO_PID:-}" ]; then
+    kill -TERM "$GINKGO_PID" || true
+  fi
+  cleanup
+}
+trap signal_handler INT TERM
+
+main() {
+  # create temp dir and setup cleanup
+  TMP_DIR=$(mktemp -d -p /tmp kind-e2e-XXXXXX)
+  echo "Created temporary directory: ${TMP_DIR}"
+
+  # ensure artifacts (results) directory exists when not in CI
+  export ARTIFACTS="${ARTIFACTS:-${PWD}/_artifacts}"
+  mkdir -p "${ARTIFACTS}"
+
+  export VERSION_DELTA=${VERSION_DELTA:-1}
+
+  WORKSPACE_STATUS=$(./hack/print-workspace-status.sh)
+  GIT_VERSION=$(echo "$WORKSPACE_STATUS" | awk '/^gitVersion / {print $2}')
+  # Check if gitVersion contains alpha.0 and increment VERSION_DELTA if needed
+  # If the current version is alpha.0, it means the previous *stable* or developed
+  # branch is actually n-2 relative to the current minor number for compatibility purposes.
+  if [[ "${GIT_VERSION}" == *alpha.0* ]]; then
+    echo "Detected alpha.0 in gitVersion (${GIT_VERSION}), treating as still the previous minor version."
+    VERSION_DELTA=$((VERSION_DELTA + 1))
+    echo "Adjusted VERSION_DELTA: ${VERSION_DELTA}"
+  fi
+
+  MAJOR_VERSION=$(echo "$WORKSPACE_STATUS" | awk '/^STABLE_BUILD_MAJOR_VERSION / {print $2}')
+  MINOR_VERSION=$(echo "$WORKSPACE_STATUS" | awk '/^STABLE_BUILD_MINOR_VERSION / {split($2, minor, "+"); print minor[1]}')
+  export CURRENT_VERSION="${MAJOR_VERSION}.${MINOR_VERSION}"
+  export PREV_VERSION="${MAJOR_VERSION}.$((MINOR_VERSION - VERSION_DELTA))"
+  export EMULATED_VERSION="${PREV_VERSION}"
+
+  # export the KUBECONFIG to a unique path for testing
+  KUBECONFIG="${HOME}/.kube/kind-test-config"
+  export KUBECONFIG
+  echo "exported KUBECONFIG=${KUBECONFIG}"
+
+  # debug kind version
+  kind version
+
+  # in CI attempt to release some memory after building
+  if [ -n "${KUBETEST_IN_DOCKER:-}" ]; then
+    sync || true
+    echo 1 > /proc/sys/vm/drop_caches || true
+  fi
+
+  res=0
+  create_cluster || res=$?
+
+
+  # Perform the upgrade.  Assume kind-upgrade.sh is in the same directory as this script.
+  UPGRADE_SCRIPT="${UPGRADE_SCRIPT:-${PWD}/../test-infra/experiment/compatibility-versions/kind-upgrade.sh}"
+  echo "Upgrading cluster with ${UPGRADE_SCRIPT}"
+
+  upgrade_cluster_components || res=$?
+  # If upgrade fails, we shouldn't run tests but should still clean up.
+  if [[ "$res" -ne 0 ]]; then
+    cleanup
+    exit $res
+  fi
+
+  EMULATED_VERSION_UPGRADE_SCRIPT="${EMULATED_VERSION_UPGRADE_SCRIPT:-${PWD}/../test-infra/experiment/compatibility-versions/emulated-version-upgrade.sh}"
+  echo "Upgrading cluster with ${EMULATED_VERSION_UPGRADE_SCRIPT}"
+  bash "${EMULATED_VERSION_UPGRADE_SCRIPT}" | tee "${ARTIFACTS}/emulated-upgrade-output.txt"
path: experiment/compatibility-versions/e2e-two-steps-upgrade.sh
user: login: BenTheElder
body: NIT: let the script select the shell itself.
```suggestion
"${EMULATED_VERSION_UPGRADE_SCRIPT}" | tee "${ARTIFACTS}/emulated-upgrade-output.txt"
```
created_at: 2025-05-22T22:35:16Z
author_association: MEMBER
diff_hunk: @@ -0,0 +1,135 @@
+#!/usr/bin/env bash
+# Copyright 2024 The Kubernetes Authors.
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+# hack script for running a kind e2e
+# must be run with a kubernetes checkout in $PWD (IE from the checkout)
+# Usage: SKIP="ginkgo skip regex" FOCUS="ginkgo focus regex" kind-e2e.sh
+
+set -o errexit -o nounset -o xtrace
+
+# Settings:
+# SKIP: ginkgo skip regex
+# FOCUS: ginkgo focus regex
+# LABEL_FILTER: ginkgo label query for selecting tests (see "Spec Labels" in https://onsi.github.io/ginkgo/#filtering-specs)
+#
+# The default is to focus on conformance tests. Serial tests get skipped when
+# parallel testing is enabled. Using LABEL_FILTER instead of combining SKIP and
+# FOCUS is recommended (more expressive, easier to read than regexp).
+#
+# GA_ONLY: true  - limit to GA APIs/features as much as possible
+#          false - (default) APIs and features left at defaults
+# FEATURE_GATES:
+#          JSON or YAML encoding of a string/bool map: {"FeatureGateA": true, "FeatureGateB": false}
+#          Enables or disables feature gates in the entire cluster.
+#          Cannot be used when GA_ONLY=true.
+# RUNTIME_CONFIG:
+#          JSON or YAML encoding of a string/string (!) map: {"apia.example.com/v1alpha1": "true", "apib.example.com/v1beta1": "false"}
+#          Enables API groups in the apiserver via --runtime-config.
+#          Cannot be used when GA_ONLY=true.
+
+COMMON_SCRIPT="${COMMON_SCRIPT:-${PWD}/../test-infra/experiment/compatibility-versions/common.sh}"
+source "${COMMON_SCRIPT}"
+
+# setup signal handlers
+# shellcheck disable=SC2317 # this is not unreachable code
+signal_handler() {
+  if [ -n "${GINKGO_PID:-}" ]; then
+    kill -TERM "$GINKGO_PID" || true
+  fi
+  cleanup
+}
+trap signal_handler INT TERM
+
+main() {
+  # create temp dir and setup cleanup
+  TMP_DIR=$(mktemp -d -p /tmp kind-e2e-XXXXXX)
+  echo "Created temporary directory: ${TMP_DIR}"
+
+  # ensure artifacts (results) directory exists when not in CI
+  export ARTIFACTS="${ARTIFACTS:-${PWD}/_artifacts}"
+  mkdir -p "${ARTIFACTS}"
+
+  export VERSION_DELTA=${VERSION_DELTA:-1}
+
+  WORKSPACE_STATUS=$(./hack/print-workspace-status.sh)
+  GIT_VERSION=$(echo "$WORKSPACE_STATUS" | awk '/^gitVersion / {print $2}')
+  # Check if gitVersion contains alpha.0 and increment VERSION_DELTA if needed
+  # If the current version is alpha.0, it means the previous *stable* or developed
+  # branch is actually n-2 relative to the current minor number for compatibility purposes.
+  if [[ "${GIT_VERSION}" == *alpha.0* ]]; then
+    echo "Detected alpha.0 in gitVersion (${GIT_VERSION}), treating as still the previous minor version."
+    VERSION_DELTA=$((VERSION_DELTA + 1))
+    echo "Adjusted VERSION_DELTA: ${VERSION_DELTA}"
+  fi
+
+  MAJOR_VERSION=$(echo "$WORKSPACE_STATUS" | awk '/^STABLE_BUILD_MAJOR_VERSION / {print $2}')
+  MINOR_VERSION=$(echo "$WORKSPACE_STATUS" | awk '/^STABLE_BUILD_MINOR_VERSION / {split($2, minor, "+"); print minor[1]}')
+  export CURRENT_VERSION="${MAJOR_VERSION}.${MINOR_VERSION}"
+  export PREV_VERSION="${MAJOR_VERSION}.$((MINOR_VERSION - VERSION_DELTA))"
+  export EMULATED_VERSION="${PREV_VERSION}"
+
+  # export the KUBECONFIG to a unique path for testing
+  KUBECONFIG="${HOME}/.kube/kind-test-config"
+  export KUBECONFIG
+  echo "exported KUBECONFIG=${KUBECONFIG}"
+
+  # debug kind version
+  kind version
+
+  # in CI attempt to release some memory after building
+  if [ -n "${KUBETEST_IN_DOCKER:-}" ]; then
+    sync || true
+    echo 1 > /proc/sys/vm/drop_caches || true
+  fi
+
+  res=0
+  create_cluster || res=$?
+
+
+  # Perform the upgrade.  Assume kind-upgrade.sh is in the same directory as this script.
+  UPGRADE_SCRIPT="${UPGRADE_SCRIPT:-${PWD}/../test-infra/experiment/compatibility-versions/kind-upgrade.sh}"
+  echo "Upgrading cluster with ${UPGRADE_SCRIPT}"
+
+  upgrade_cluster_components || res=$?
+  # If upgrade fails, we shouldn't run tests but should still clean up.
+  if [[ "$res" -ne 0 ]]; then
+    cleanup
+    exit $res
+  fi
+
+  EMULATED_VERSION_UPGRADE_SCRIPT="${EMULATED_VERSION_UPGRADE_SCRIPT:-${PWD}/../test-infra/experiment/compatibility-versions/emulated-version-upgrade.sh}"
+  echo "Upgrading cluster with ${EMULATED_VERSION_UPGRADE_SCRIPT}"
+  bash "${EMULATED_VERSION_UPGRADE_SCRIPT}" | tee "${ARTIFACTS}/emulated-upgrade-output.txt"
path: experiment/compatibility-versions/e2e-two-steps-upgrade.sh
user: login: ynfx8
body: updated.
created_at: 2025-05-22T22:37:49Z
author_association: CONTRIBUTOR

PR comments:
user: login: k8s-ci-robot
created_at: 2025-05-22T01:32:41Z
body: Hi @ynfx8. Thanks for your PR.

I'm waiting for a [kubernetes](https://github.com/orgs/kubernetes/people) member to verify that this patch is reasonable to test. If it is, they should reply with `/ok-to-test` on its own line. Until that is done, I will not automatically test new commits in this PR, but the usual testing commands by org members will still work. Regular contributors should [join the org](https://git.k8s.io/community/community-membership.md#member) to skip this step.

Once the patch is verified, the new status will be reflected by the `ok-to-test` label.

I understand the commands that are listed [here](https://go.k8s.io/bot-commands?repo=kubernetes%2Ftest-infra).

<details>

Instructions for interacting with me using PR comments are available [here](https://git.k8s.io/community/contributors/guide/pull-requests.md).  If you have questions or suggestions related to my behavior, please file an issue against the [kubernetes-sigs/prow](https://github.com/kubernetes-sigs/prow/issues/new?title=Prow%20issue:) repository.
</details>

author_association: CONTRIBUTOR
user: login: ynfx8
created_at: 2025-05-22T01:34:09Z
body: /test pull-test-infra-unit-test
author_association: CONTRIBUTOR
user: login: k8s-ci-robot
created_at: 2025-05-22T01:34:23Z
body: @ynfx8: Cannot trigger testing until a trusted user reviews the PR and leaves an `/ok-to-test` message.

<details>

In response to [this](https://github.com/kubernetes/test-infra/pull/34836#issuecomment-2899652228):

>/test pull-test-infra-unit-test


Instructions for interacting with me using PR comments are available [here](https://git.k8s.io/community/contributors/guide/pull-requests.md).  If you have questions or suggestions related to my behavior, please file an issue against the [kubernetes-sigs/prow](https://github.com/kubernetes-sigs/prow/issues/new?title=Prow%20issue:) repository.
</details>
author_association: CONTRIBUTOR
user: login: ynfx8
created_at: 2025-05-22T01:35:13Z
body: /assign @BenTheElder 
/cc @siyuanfoundation 
author_association: CONTRIBUTOR
user: login: BenTheElder
created_at: 2025-05-22T17:02:16Z
body: /ok-to-test
author_association: MEMBER
user: login: ynfx8
created_at: 2025-05-22T22:10:58Z
body: > The imagePullPolicy and bash -x questions are blocking, looking back over the rest.
> 
> We will really need to think about rewriting these into something like `kinder` at some point as they grow past small proof of concept, it's hard to review and maintain lots of code with global state and no unit tests (IE typical large shell scripts).

removed all imagePullPolicy from the yaml
author_association: CONTRIBUTOR
user: login: ynfx8
created_at: 2025-05-22T22:16:28Z
body: > The imagePullPolicy and bash -x questions are blocking, looking back over the rest.
> 
> We will really need to think about rewriting these into something like `kinder` at some point as they grow past small proof of concept, it's hard to review and maintain lots of code with global state and no unit tests (IE typical large shell scripts).

removed all the imagePullPolicy in yaml file -x.
author_association: CONTRIBUTOR
user: login: k8s-ci-robot
created_at: 2025-05-22T22:40:25Z
body: [APPROVALNOTIFIER] This PR is **APPROVED**

This pull-request has been approved by: *<a href="https://github.com/kubernetes/test-infra/pull/34836#pullrequestreview-2862721379" title="Approved">BenTheElder</a>*, *<a href="https://github.com/kubernetes/test-infra/pull/34836#" title="Author self-approved">ynfx8</a>*

The full list of commands accepted by this bot can be found [here](https://go.k8s.io/bot-commands?repo=kubernetes%2Ftest-infra).

The pull request process is described [here](https://git.k8s.io/community/contributors/guide/owners.md#the-code-review-process)

<details >
Needs approval from an approver in each of these files:

- ~~[config/OWNERS](https://github.com/kubernetes/test-infra/blob/master/config/OWNERS)~~ [BenTheElder]
- ~~[experiment/compatibility-versions/OWNERS](https://github.com/kubernetes/test-infra/blob/master/experiment/compatibility-versions/OWNERS)~~ [BenTheElder]

Approvers can indicate their approval by writing `/approve` in a comment
Approvers can cancel approval by writing `/approve cancel` in a comment
</details>
<!-- META={"approvers":[]} -->
author_association: CONTRIBUTOR
user: login: k8s-ci-robot
created_at: 2025-05-22T22:48:39Z
body: @ynfx8: Updated the `job-config` configmap in namespace `default` at cluster `test-infra-trusted` using the following files:
 - key `compatibility-versions-e2e.yaml` using file `config/jobs/kubernetes/sig-testing/compatibility-versions-e2e.yaml`

<details>

In response to [this](https://github.com/kubernetes/test-infra/pull/34836):

>* add test scripts to perform two steps upgrade - binary version upgrade and emulated version upgrade.
>* config a n-minus-1 prow job for the new test.
>* tests passed locally.


Instructions for interacting with me using PR comments are available [here](https://git.k8s.io/community/contributors/guide/pull-requests.md).  If you have questions or suggestions related to my behavior, please file an issue against the [kubernetes-sigs/prow](https://github.com/kubernetes-sigs/prow/issues/new?title=Prow%20issue:) repository.
</details>
author_association: CONTRIBUTOR