Repository: kubernetes/minikube
Pull request #15336 (2022-11-09 23:30:01)

fix kubelet localStorageCapacityIsolation option
fixes: #14728
fixes: #15099 (btrfs + k8s 1.25.2 + docker)
fixes: #15050 (btrfs + k8s 1.25.0 + docker)

it might help with #14819 (btrfs + docker, but k8s version is not shared: issue was reported on 20th Aug, so it, in theory, could be k8s [v1.25.0-beta.0](https://github.com/kubernetes/kubernetes/releases/tag/v1.25.0-beta.0), but minikube is v1.26.1 shipped with DefaultKubernetesVersion = "v1.24.3")

i don't think i ever had issues with btrfs + docker + k8s/minikube and also, official [OverlayFS and Docker Performance](https://docs.docker.com/storage/storagedriver/overlayfs-driver/#overlayfs-and-docker-performance) claims that:
> _In certain circumstances_, overlay2 **_may perform better_** than btrfs as well. However, ...

but i know some users reported problems earlier, so, in case of such a combination, this pr should automatically:
- enable `LocalStorageCapacityIsolation` as **_feature flag_** for k8s _before v1.25.0-beta.0_ and
- enable `localStorageCapacityIsolation` as **_config option_** for k8s _v1.25.0-beta.0 and newer_

should they wish, users will also have the option to override this by passing eg, `--extra-config="kubelet.localStorageCapacityIsolation=true"` (which is default) for k8s >= v1.25.0-beta.0 (last example below demonstrates that case)

### before

$ docker exec -ti docker /bin/bash
root@docker:/# journalctl -f
```
...
Nov 09 01:31:37 docker systemd[1]: kubelet.service: Main process exited, code=exited, status=1/FAILURE
Nov 09 01:31:37 docker systemd[1]: kubelet.service: Failed with result 'exit-code'.
Nov 09 01:31:38 docker systemd[1]: kubelet.service: Scheduled restart job, restart counter is at 3802.
Nov 09 01:31:38 docker systemd[1]: Stopped kubelet: The Kubernetes Node Agent.
Nov 09 01:31:38 docker systemd[1]: Started kubelet: The Kubernetes Node Agent.
Nov 09 01:31:38 docker kubelet[76607]: Flag --container-runtime has been deprecated, will be removed in 1.27 as the only valid value is 'remote'
Nov 09 01:31:38 docker kubelet[76607]: Flag --feature-gates has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Nov 09 01:31:38 docker kubelet[76607]: Flag --runtime-request-timeout has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Nov 09 01:31:38 docker kubelet[76607]: E1109 01:31:38.711920   76607 run.go:74] "command failed" err="failed to set feature gates from initial flags-based config: cannot set feature gate LocalStorageCapacityIsolation to false, feature is locked to true"
Nov 09 01:31:38 docker systemd[1]: kubelet.service: Main process exited, code=exited, status=1/FAILURE
Nov 09 01:31:38 docker systemd[1]: kubelet.service: Failed with result 'exit-code'.
...
```

### after

$ time minikube start --driver=docker --kubernetes-version=1.25.3 -p btrfs-docker-k8s1.25.3
```
😄  [btrfs-docker-k8s1.25.3] minikube v1.28.0 on Opensuse-Tumbleweed
✨  Using the docker driver based on user configuration
❗  docker is currently using the btrfs storage driver, consider switching to overlay2 for better performance
📌  Using Docker driver with root privileges
👍  Starting control plane node btrfs-docker-k8s1.25.3 in cluster btrfs-docker-k8s1.25.3
🚜  Pulling base image ...
🔥  Creating docker container (CPUs=2, Memory=15900MB) ...
🐳  Preparing Kubernetes v1.25.3 on Docker 20.10.20 ...
    ▪ kubelet.localStorageCapacityIsolation=false
    ▪ Generating certificates and keys ...
    ▪ Booting up control plane ...
    ▪ Configuring RBAC rules ...
🔎  Verifying Kubernetes components...
    ▪ Using image gcr.io/k8s-minikube/storage-provisioner:v5
🌟  Enabled addons: storage-provisioner, default-storageclass
🏄  Done! kubectl is now configured to use "btrfs-docker-k8s1.25.3" cluster and "default" namespace by default

real    0m31.322s
user    0m2.170s
sys     0m1.091s
```

$ docker exec -ti btrfs-docker-k8s1.25.3 cat /var/lib/kubelet/config.yaml | grep -A1 "kind: KubeletConfiguration"
```
kind: KubeletConfiguration
localStorageCapacityIsolation: false
```

$ docker exec -ti btrfs-docker-k8s1.25.3 cat /etc/systemd/system/kubelet.service.d/10-kubeadm.conf | grep -i "localStorageCapacityIsolation"
```
(null)
```

---

$ time minikube start --driver=docker --kubernetes-version=1.24.7 -p btrfs-docker-k8s1.24.7
```
😄  [btrfs-docker-k8s1.24.7] minikube v1.28.0 on Opensuse-Tumbleweed
✨  Using the docker driver based on user configuration
❗  docker is currently using the btrfs storage driver, consider switching to overlay2 for better performance
📌  Using Docker driver with root privileges
👍  Starting control plane node btrfs-docker-k8s1.24.7 in cluster btrfs-docker-k8s1.24.7
🚜  Pulling base image ...
🔥  Creating docker container (CPUs=2, Memory=15900MB) ...
🐳  Preparing Kubernetes v1.24.7 on Docker 20.10.20 ...
    ▪ Generating certificates and keys ...
    ▪ Booting up control plane ...
    ▪ Configuring RBAC rules ...
🔎  Verifying Kubernetes components...
    ▪ Using image gcr.io/k8s-minikube/storage-provisioner:v5
🌟  Enabled addons: storage-provisioner, default-storageclass
🏄  Done! kubectl is now configured to use "btrfs-docker-k8s1.24.7" cluster and "default" namespace by default

real    0m28.820s
user    0m2.220s
sys     0m1.144s
```

$ docker exec -ti btrfs-docker-k8s1.24.7 cat /var/lib/kubelet/config.yaml | grep -A1 "kind: KubeletConfiguration"
```
kind: KubeletConfiguration
logging:
```

$ docker exec -ti btrfs-docker-k8s1.24.7 cat /etc/systemd/system/kubelet.service.d/10-kubeadm.conf | grep -i "localStorageCapacityIsolation"
```
ExecStart=/var/lib/minikube/binaries/v1.24.7/kubelet --bootstrap-kubeconfig=/etc/kubernetes/bootstrap-kubelet.conf --config=/var/lib/kubelet/config.yaml --container-runtime=remote --container-runtime-endpoint=/var/run/cri-dockerd.sock --feature-gates=LocalStorageCapacityIsolation=false --hostname-override=btrfs-docker-k8s1.24.7 --image-service-endpoint=/var/run/cri-dockerd.sock --kubeconfig=/etc/kubernetes/kubelet.conf --node-ip=192.168.58.2 --runtime-request-timeout=15m
```

---

$ time minikube start --driver=docker --kubernetes-version=1.25.3 -p btrfs-docker-k8s1.25.3-ec --extra-config="kubelet.localStorageCapacityIsolation=true"
```
😄  [btrfs-docker-k8s1.25.3-ec] minikube v1.28.0 on Opensuse-Tumbleweed
✨  Using the docker driver based on user configuration
❗  docker is currently using the btrfs storage driver, consider switching to overlay2 for better performance
📌  Using Docker driver with root privileges
👍  Starting control plane node btrfs-docker-k8s1.25.3-ec in cluster btrfs-docker-k8s1.25.3-ec
🚜  Pulling base image ...
🔥  Creating docker container (CPUs=2, Memory=15900MB) ...
🐳  Preparing Kubernetes v1.25.3 on Docker 20.10.20 ...
    ▪ kubelet.localStorageCapacityIsolation=true
    ▪ Generating certificates and keys ...
    ▪ Booting up control plane ...
    ▪ Configuring RBAC rules ...
🔎  Verifying Kubernetes components...
    ▪ Using image gcr.io/k8s-minikube/storage-provisioner:v5
🌟  Enabled addons: default-storageclass, storage-provisioner
🏄  Done! kubectl is now configured to use "btrfs-docker-k8s1.25.3-ec" cluster and "default" namespace by default

real    0m29.488s
user    0m2.293s
sys     0m1.169s
```

$ docker exec -ti btrfs-docker-k8s1.25.3-ec cat /var/lib/kubelet/config.yaml | grep -A1 "kind: KubeletConfiguration"
```
kind: KubeletConfiguration
localStorageCapacityIsolation: true
```

$ docker exec -ti btrfs-docker-k8s1.25.3-ec cat /etc/systemd/system/kubelet.service.d/10-kubeadm.conf | grep -i "localStorageCapacityIsolation"
```
(null)
```

---

$ minikube profile list
```
|---------------------------|-----------|---------|--------------|------|---------|---------|-------|--------|
|          Profile          | VM Driver | Runtime |      IP      | Port | Version | Status  | Nodes | Active |
|---------------------------|-----------|---------|--------------|------|---------|---------|-------|--------|
| btrfs-docker-k8s1.24.7    | docker    | docker  | 192.168.58.2 | 8443 | v1.24.7 | Running |     1 |        |
| btrfs-docker-k8s1.25.3    | docker    | docker  | 192.168.49.2 | 8443 | v1.25.3 | Running |     1 |        |
| btrfs-docker-k8s1.25.3-ec | docker    | docker  | 192.168.67.2 | 8443 | v1.25.3 | Running |     1 |        |
|---------------------------|-----------|---------|--------------|------|---------|---------|-------|--------|
```


Patch:
diff --git a/.github/workflows/build.yml b/.github/workflows/build.yml
index 36cb78991..73900cd5d 100644
--- a/.github/workflows/build.yml
+++ b/.github/workflows/build.yml
@@ -12,7 +12,7 @@ on:
       - "!deploy/iso/**"
 env:
   GOPROXY: https://proxy.golang.org
-  GO_VERSION: '1.19.3'
+  GO_VERSION: '1.19.2'
 permissions:
   contents: read
 
diff --git a/.github/workflows/docs.yml b/.github/workflows/docs.yml
index 6e00c4d11..65d403379 100644
--- a/.github/workflows/docs.yml
+++ b/.github/workflows/docs.yml
@@ -6,7 +6,7 @@ on:
       - master  
 env:
   GOPROXY: https://proxy.golang.org
-  GO_VERSION: '1.19.3'
+  GO_VERSION: '1.19.2'
 permissions:
   contents: read
 
@@ -26,10 +26,7 @@ jobs:
           c=$(git status --porcelain)
           c="${c//$'\n'/'%0A'}"
           c="${c//$'\r'/'%0D'}"
-          # The following is to support multiline with GITHUB_OUTPUT, see https://docs.github.com/en/actions/using-workflows/workflow-commands-for-github-actions#multiline-strings
-          echo "changes<<EOF" >> $GITHUB_OUTPUT
-          echo "$c" >> $GITHUB_OUTPUT
-          echo "EOF" >> $GITHUB_OUTPUT
+          echo "changes=$c" >> $GITHUB_OUTPUT
       - name: Create PR
         if: ${{ steps.gendocs.outputs.changes != '' }}
         uses: peter-evans/create-pull-request@b4d51739f96fca8047ad065eccef63442d8e99f7
diff --git a/.github/workflows/functional_verified.yml b/.github/workflows/functional_verified.yml
index 64052f0b6..46d967602 100644
--- a/.github/workflows/functional_verified.yml
+++ b/.github/workflows/functional_verified.yml
@@ -22,7 +22,7 @@ on:
       - deleted
 env:
   GOPROXY: https://proxy.golang.org
-  GO_VERSION: '1.19.3'
+  GO_VERSION: '1.19.2'
 
 permissions:
   contents: read
diff --git a/.github/workflows/leaderboard.yml b/.github/workflows/leaderboard.yml
index 4ecc2a223..edde68e2f 100644
--- a/.github/workflows/leaderboard.yml
+++ b/.github/workflows/leaderboard.yml
@@ -6,7 +6,7 @@ on:
       - 'v*-beta.*'
 env:
   GOPROXY: https://proxy.golang.org
-  GO_VERSION: '1.19.3'
+  GO_VERSION: '1.19.2'
 permissions:
   contents: read
 
@@ -25,10 +25,7 @@ jobs:
           c=$(git status --porcelain)
           c="${c//$'\n'/'%0A'}"
           c="${c//$'\r'/'%0D'}"
-          # The following is to support multiline with GITHUB_OUTPUT, see https://docs.github.com/en/actions/using-workflows/workflow-commands-for-github-actions#multiline-strings
-          echo "changes<<EOF" >> $GITHUB_OUTPUT
-          echo "$c" >> $GITHUB_OUTPUT
-          echo "EOF" >> $GITHUB_OUTPUT
+          echo "changes=$c" >> $GITHUB_OUTPUT
         env:
           GITHUB_TOKEN: ${{ secrets.MINIKUBE_BOT_PAT }}
       - name: Create PR
diff --git a/.github/workflows/master.yml b/.github/workflows/master.yml
index c781e102f..45df5da39 100644
--- a/.github/workflows/master.yml
+++ b/.github/workflows/master.yml
@@ -14,7 +14,7 @@ on:
       - "!deploy/iso/**"
 env:
   GOPROXY: https://proxy.golang.org
-  GO_VERSION: '1.19.3'
+  GO_VERSION: '1.19.2'
 permissions:
   contents: read
 
diff --git a/.github/workflows/pr.yml b/.github/workflows/pr.yml
index f1cba5ffb..772bf4196 100644
--- a/.github/workflows/pr.yml
+++ b/.github/workflows/pr.yml
@@ -12,7 +12,7 @@ on:
       - "!deploy/iso/**"
 env:
   GOPROXY: https://proxy.golang.org
-  GO_VERSION: '1.19.3'
+  GO_VERSION: '1.19.2'
 permissions:
   contents: read
 
diff --git a/.github/workflows/sync-minikube.yml b/.github/workflows/sync-minikube.yml
index 3f2041c6e..b2189ffe5 100644
--- a/.github/workflows/sync-minikube.yml
+++ b/.github/workflows/sync-minikube.yml
@@ -6,7 +6,7 @@ on:
     - cron: "0 2,14 * * *"
 env:
   GOPROXY: https://proxy.golang.org
-  GO_VERSION: '1.19.3'
+  GO_VERSION: '1.19.2'
 permissions:
   contents: read
 
diff --git a/.github/workflows/time-to-k8s-public-chart.yml b/.github/workflows/time-to-k8s-public-chart.yml
index dd1375b8d..ce436c3a3 100644
--- a/.github/workflows/time-to-k8s-public-chart.yml
+++ b/.github/workflows/time-to-k8s-public-chart.yml
@@ -6,7 +6,7 @@ on:
     - cron: "0 2,14 * * *"
 env:
   GOPROXY: https://proxy.golang.org
-  GO_VERSION: '1.19.3'
+  GO_VERSION: '1.19.2'
 permissions:
   contents: read
 
diff --git a/.github/workflows/time-to-k8s.yml b/.github/workflows/time-to-k8s.yml
index 5b6fc8db7..5bb512ae0 100644
--- a/.github/workflows/time-to-k8s.yml
+++ b/.github/workflows/time-to-k8s.yml
@@ -5,7 +5,7 @@ on:
     types: [released]
 env:
   GOPROXY: https://proxy.golang.org
-  GO_VERSION: '1.19.3'
+  GO_VERSION: '1.19.2'
 permissions:
   contents: read
 
diff --git a/.github/workflows/translations.yml b/.github/workflows/translations.yml
index 3ee67796b..064b85d09 100644
--- a/.github/workflows/translations.yml
+++ b/.github/workflows/translations.yml
@@ -6,7 +6,7 @@ on:
       - "translations/**"
 env:
   GOPROXY: https://proxy.golang.org
-  GO_VERSION: '1.19.3'
+  GO_VERSION: '1.19.2'
 permissions:
   contents: read
 
diff --git a/.github/workflows/update-golang-version.yml b/.github/workflows/update-golang-version.yml
index f12f68e52..c485a7e9b 100644
--- a/.github/workflows/update-golang-version.yml
+++ b/.github/workflows/update-golang-version.yml
@@ -6,7 +6,7 @@ on:
     - cron: "0 9 * * 1"
 env:
   GOPROXY: https://proxy.golang.org
-  GO_VERSION: '1.19.3'
+  GO_VERSION: '1.19.2'
 permissions:
   contents: read
 
@@ -22,10 +22,7 @@ jobs:
         id: bumpGolang
         run: |
           make update-golang-version
-          # The following is to support multiline with GITHUB_OUTPUT, see https://docs.github.com/en/actions/using-workflows/workflow-commands-for-github-actions#multiline-strings
-          echo "changes<<EOF" >> $GITHUB_OUTPUT
-          echo "$(git status --porcelain)" >> $GITHUB_OUTPUT
-          echo "EOF" >> $GITHUB_OUTPUT
+          echo "changes=$(git status --porcelain)" >> $GITHUB_OUTPUT
       - name: Create PR
         if: ${{ steps.bumpGolang.outputs.changes != '' }}
         uses: peter-evans/create-pull-request@b4d51739f96fca8047ad065eccef63442d8e99f7
diff --git a/.github/workflows/update-golint-version.yml b/.github/workflows/update-golint-version.yml
index ea9b03c41..cd127ad6d 100644
--- a/.github/workflows/update-golint-version.yml
+++ b/.github/workflows/update-golint-version.yml
@@ -6,7 +6,7 @@ on:
     - cron: "0 10 * * 1"
 env:
   GOPROXY: https://proxy.golang.org
-  GO_VERSION: '1.19.3'
+  GO_VERSION: '1.19.2'
 permissions:
   contents: read
 
@@ -22,10 +22,7 @@ jobs:
         id: bumpGolint
         run: |
           make update-golint-version
-          # The following is to support multiline with GITHUB_OUTPUT, see https://docs.github.com/en/actions/using-workflows/workflow-commands-for-github-actions#multiline-strings
-          echo "changes<<EOF" >> $GITHUB_OUTPUT
-          echo "$(git status --porcelain)" >> $GITHUB_OUTPUT
-          echo "EOF" >> $GITHUB_OUTPUT
+          echo "changes=$(git status --porcelain)" >> $GITHUB_OUTPUT
       - name: Create PR
         if: ${{ steps.bumpGolint.outputs.changes != '' }}
         uses: peter-evans/create-pull-request@b4d51739f96fca8047ad065eccef63442d8e99f7
diff --git a/.github/workflows/update-gopogh-version.yml b/.github/workflows/update-gopogh-version.yml
index 1a430823e..56dcb3f2a 100644
--- a/.github/workflows/update-gopogh-version.yml
+++ b/.github/workflows/update-gopogh-version.yml
@@ -6,7 +6,7 @@ on:
     - cron: "0 9 * * 1"
 env:
   GOPROXY: https://proxy.golang.org
-  GO_VERSION: '1.19.3'
+  GO_VERSION: '1.19.2'
 permissions:
   contents: read
 
@@ -22,10 +22,7 @@ jobs:
         id: bumpGopogh
         run: |
           make update-gopogh-version
-          # The following is to support multiline with GITHUB_OUTPUT, see https://docs.github.com/en/actions/using-workflows/workflow-commands-for-github-actions#multiline-strings
-          echo "changes<<EOF" >> $GITHUB_OUTPUT
-          echo "$(git status --porcelain)" >> $GITHUB_OUTPUT
-          echo "EOF" >> $GITHUB_OUTPUT
+          echo "changes=$(git status --porcelain)" >> $GITHUB_OUTPUT
       - name: Create PR
         if: ${{ steps.bumpGopogh.outputs.changes != '' }}
         uses: peter-evans/create-pull-request@b4d51739f96fca8047ad065eccef63442d8e99f7
diff --git a/.github/workflows/update-k8s-versions.yml b/.github/workflows/update-k8s-versions.yml
index eabb9e998..a92794154 100644
--- a/.github/workflows/update-k8s-versions.yml
+++ b/.github/workflows/update-k8s-versions.yml
@@ -6,7 +6,7 @@ on:
     - cron: "0 8 * * 1"
 env:
   GOPROXY: https://proxy.golang.org
-  GO_VERSION: '1.19.3'
+  GO_VERSION: '1.19.2'
 permissions:
   contents: read
 
@@ -27,10 +27,7 @@ jobs:
           c=$(git status --porcelain)
           c="${c//$'\n'/'%0A'}"
           c="${c//$'\r'/'%0D'}"
-          # The following is to support multiline with GITHUB_OUTPUT, see https://docs.github.com/en/actions/using-workflows/workflow-commands-for-github-actions#multiline-strings
-          echo "changes<<EOF" >> $GITHUB_OUTPUT
-          echo "$c" >> $GITHUB_OUTPUT
-          echo "EOF" >> $GITHUB_OUTPUT
+          echo "changes=$c" >> $GITHUB_OUTPUT
       - name: Create PR
         if: ${{ steps.bumpk8s.outputs.changes != '' }}
         uses: peter-evans/create-pull-request@b4d51739f96fca8047ad065eccef63442d8e99f7
diff --git a/.github/workflows/update-kubeadm-constants.yml b/.github/workflows/update-kubeadm-constants.yml
index d5ec2436f..9e597361e 100644
--- a/.github/workflows/update-kubeadm-constants.yml
+++ b/.github/workflows/update-kubeadm-constants.yml
@@ -6,7 +6,7 @@ on:
     - cron: "0 6 * * 1"
 env:
   GOPROXY: https://proxy.golang.org
-  GO_VERSION: '1.19.3'
+  GO_VERSION: '1.19.2'
 permissions:
   contents: read
 
@@ -25,10 +25,7 @@ jobs:
           c=$(git status --porcelain)
           c="${c//$'\n'/'%0A'}"
           c="${c//$'\r'/'%0D'}"
-          # The following is to support multiline with GITHUB_OUTPUT, see https://docs.github.com/en/actions/using-workflows/workflow-commands-for-github-actions#multiline-strings
-          echo "changes<<EOF" >> $GITHUB_OUTPUT
-          echo "$c" >> $GITHUB_OUTPUT
-          echo "EOF" >> $GITHUB_OUTPUT
+          echo "changes=$c" >> $GITHUB_OUTPUT
       - name: Create PR
         if: ${{ steps.bumpKubeadmConsts.outputs.changes != '' }}
         uses: peter-evans/create-pull-request@b4d51739f96fca8047ad065eccef63442d8e99f7
diff --git a/.github/workflows/yearly-leaderboard.yml b/.github/workflows/yearly-leaderboard.yml
index 3c5c91851..5c4b0581c 100644
--- a/.github/workflows/yearly-leaderboard.yml
+++ b/.github/workflows/yearly-leaderboard.yml
@@ -6,7 +6,7 @@ on:
     - cron: "0 0 2 * *"
 env:
   GOPROXY: https://proxy.golang.org
-  GO_VERSION: '1.19.3'
+  GO_VERSION: '1.19.2'
 permissions:
   contents: read
 
@@ -27,10 +27,7 @@ jobs:
         id: yearlyLeaderboard
         run: |
           make update-yearly-leaderboard
-          # The following is to support multiline with GITHUB_OUTPUT, see https://docs.github.com/en/actions/using-workflows/workflow-commands-for-github-actions#multiline-strings
-          echo "changes<<EOF" >> $GITHUB_OUTPUT
-          echo "$(git status --porcelain)" >> $GITHUB_OUTPUT
-          echo "EOF" >> $GITHUB_OUTPUT
+          echo "changes=$(git status --porcelain)" >> $GITHUB_OUTPUT
         env:
           GITHUB_TOKEN: ${{ secrets.MINIKUBE_BOT_PAT }}
       - name: Create PR
diff --git a/Makefile b/Makefile
index 61280a396..30b04ebc8 100644
--- a/Makefile
+++ b/Makefile
@@ -33,7 +33,7 @@ RPM_REVISION ?= 0
 
 # used by hack/jenkins/release_build_and_upload.sh and KVM_BUILD_IMAGE, see also BUILD_IMAGE below
 # update this only by running `make update-golang-version`
-GO_VERSION ?= 1.19.3
+GO_VERSION ?= 1.19.2
 # update this only by running `make update-golang-version`
 GO_K8S_VERSION_PREFIX ?= v1.25.0
 
@@ -44,7 +44,7 @@ KVM_GO_VERSION ?= $(GO_VERSION:.0=)
 INSTALL_SIZE ?= $(shell du out/minikube-windows-amd64.exe | cut -f1)
 BUILDROOT_BRANCH ?= 2021.02.12
 # the go version on the line below is for the ISO and does not need to be updated often
-GOLANG_OPTIONS = GO_VERSION=1.19.3 GO_HASH_FILE=$(PWD)/deploy/iso/minikube-iso/go.hash
+GOLANG_OPTIONS = GO_VERSION=1.19.2 GO_HASH_FILE=$(PWD)/deploy/iso/minikube-iso/go.hash
 BUILDROOT_OPTIONS = BR2_EXTERNAL=../../deploy/iso/minikube-iso $(GOLANG_OPTIONS)
 REGISTRY ?= gcr.io/k8s-minikube
 
diff --git a/cmd/minikube/cmd/start.go b/cmd/minikube/cmd/start.go
index e07fd4d2c..30229af1d 100644
--- a/cmd/minikube/cmd/start.go
+++ b/cmd/minikube/cmd/start.go
@@ -1399,7 +1399,7 @@ func validateChangedMemoryFlags(drvName string) {
 		if err != nil {
 			klog.Warningf("Unable to query memory limits: %+v", err)
 		}
-		req = noLimitMemory(sysLimit, containerLimit, drvName)
+		req = noLimitMemory(sysLimit, containerLimit)
 	} else {
 		req, err = util.CalculateSizeInMB(memString)
 		if err != nil {
@@ -1409,18 +1409,12 @@ func validateChangedMemoryFlags(drvName string) {
 	validateRequestedMemorySize(req, drvName)
 }
 
-func noLimitMemory(sysLimit, containerLimit int, drvName string) int {
+func noLimitMemory(sysLimit int, containerLimit int) int {
 	if containerLimit != 0 {
 		return containerLimit
 	}
 	// Recommend 1GB to handle OS/VM overhead
-	sysOverhead := 1024
-	if drvName == "virtualbox" {
-		// VirtualBox fully allocates all requested memory on start, it doesn't dynamically allocate when needed like other drivers
-		// Because of this allow more system overhead to prevent out of memory issues
-		sysOverhead = 1536
-	}
-	return sysLimit - sysOverhead
+	return sysLimit - 1024
 }
 
 // This function validates if the --registry-mirror
diff --git a/cmd/minikube/cmd/start_flags.go b/cmd/minikube/cmd/start_flags.go
index 74edf48e0..409d74d2b 100644
--- a/cmd/minikube/cmd/start_flags.go
+++ b/cmd/minikube/cmd/start_flags.go
@@ -369,7 +369,7 @@ func getMemorySize(cmd *cobra.Command, drvName string) int {
 		memString := viper.GetString(memory)
 		var err error
 		if memString == constants.MaxResources {
-			mem = noLimitMemory(sysLimit, containerLimit, drvName)
+			mem = noLimitMemory(sysLimit, containerLimit)
 		} else {
 			mem, err = pkgutil.CalculateSizeInMB(memString)
 			if err != nil {
@@ -620,9 +620,18 @@ func generateNewConfigFromFlags(cmd *cobra.Command, k8sVersion string, rtime str
 			}
 			out.Styled(style.Notice, "Using {{.driver_name}} driver with root privileges", out.V{"driver_name": driver.FullName(drvName)})
 		}
+		// for btrfs: if k8s < v1.25.0-beta.0 set kubelet's LocalStorageCapacityIsolation feature gate flag to false,
+		// and if k8s >= v1.25.0-beta.0 (when it went ga and removed as feature gate), set kubelet's localStorageCapacityIsolation option (via kubeadm config) to false.
+		// ref: https://github.com/kubernetes/minikube/issues/14728#issue-1327885840
 		if si.StorageDriver == "btrfs" {
-			klog.Info("auto-setting LocalStorageCapacityIsolation to false because using btrfs storage driver")
-			cc.KubernetesConfig.FeatureGates = addFeatureGate(cc.KubernetesConfig.FeatureGates, "LocalStorageCapacityIsolation=false")
+			if semver.MustParse(strings.TrimPrefix(k8sVersion, version.VersionPrefix)).LT(semver.MustParse("1.25.0-beta.0")) {
+				klog.Info("auto-setting LocalStorageCapacityIsolation to false because using btrfs storage driver")
+				cc.KubernetesConfig.FeatureGates = addFeatureGate(cc.KubernetesConfig.FeatureGates, "LocalStorageCapacityIsolation=false")
+			} else if !cc.KubernetesConfig.ExtraOptions.Exists("kubelet.localStorageCapacityIsolation=false") {
+				if err := cc.KubernetesConfig.ExtraOptions.Set("kubelet.localStorageCapacityIsolation=false"); err != nil {
+					exit.Error(reason.InternalConfigSet, "failed to set extra option", err)
+				}
+			}
 		}
 		if runtime.GOOS == "linux" && si.DockerOS == "Docker Desktop" {
 			out.WarningT("For an improved experience it's recommended to use Docker Engine instead of Docker Desktop.\nDocker Engine installation instructions: https://docs.docker.com/engine/install/#server")
diff --git a/deploy/addons/auto-pause/Dockerfile b/deploy/addons/auto-pause/Dockerfile
index 2b2bd1543..8db63196b 100644
--- a/deploy/addons/auto-pause/Dockerfile
+++ b/deploy/addons/auto-pause/Dockerfile
@@ -1,2 +1,2 @@
-FROM golang:1.19.3
+FROM golang:1.19.2
 ADD auto-pause-hook /auto-pause-hook
diff --git a/deploy/iso/minikube-iso/go.hash b/deploy/iso/minikube-iso/go.hash
index ae24e46cc..ca6cdbbe6 100644
--- a/deploy/iso/minikube-iso/go.hash
+++ b/deploy/iso/minikube-iso/go.hash
@@ -3,5 +3,4 @@ sha256  3a70e5055509f347c0fb831ca07a2bf3b531068f349b14a3c652e9b5b67beb5d  go1.17
 sha256  0012386ddcbb5f3350e407c679923811dbd283fcdc421724931614a842ecbc2d  go1.18.3.src.tar.gz
 sha256  27871baa490f3401414ad793fba49086f6c855b1c584385ed7771e1204c7e179  go1.19.1.src.tar.gz
 sha256  2ce930d70a931de660fdaf271d70192793b1b240272645bf0275779f6704df6b  go1.19.2.src.tar.gz
-sha256  18ac263e39210bcf68d85f4370e97fb1734166995a1f63fb38b4f6e07d90d212  go1.19.3.src.tar.gz
 sha256  2d36597f7117c38b006835ae7f537487207d8ec407aa9d9980794b2030cbc067  LICENSE
diff --git a/deploy/kicbase/Dockerfile b/deploy/kicbase/Dockerfile
index 62227f4cc..7bbae6797 100644
--- a/deploy/kicbase/Dockerfile
+++ b/deploy/kicbase/Dockerfile
@@ -19,7 +19,7 @@
 
 
 # multi-stage docker build so we can build auto-pause for arm64
-FROM golang:1.19.3 as auto-pause
+FROM golang:1.19.2 as auto-pause
 WORKDIR /src
 # auto-pause depends on core minikube code so we need to pass the whole source code as the context
 # copy in the minimal amount of source code possible
diff --git a/hack/jenkins/common.ps1 b/hack/jenkins/common.ps1
index d429dee38..ab73df197 100644
--- a/hack/jenkins/common.ps1
+++ b/hack/jenkins/common.ps1
@@ -30,7 +30,7 @@ function Write-GithubStatus {
 $env:SHORT_COMMIT=$env:COMMIT.substring(0, 7)
 $gcs_bucket="minikube-builds/logs/$env:MINIKUBE_LOCATION/$env:ROOT_JOB_ID"
 $env:MINIKUBE_SUPPRESS_DOCKER_PERFORMANCE="true"
-$GoVersion = "1.19.3"
+$GoVersion = "1.19.2"
 
 # Docker's kubectl breaks things, and comes earlier in the path than the regular kubectl. So download the expected kubectl and replace Docker's version.
 $KubeVersion = (Invoke-WebRequest -Uri 'https://storage.googleapis.com/kubernetes-release/release/stable.txt' -UseBasicParsing).Content
diff --git a/hack/jenkins/installers/check_install_golang.sh b/hack/jenkins/installers/check_install_golang.sh
index 2fbb16a4a..861b30e83 100755
--- a/hack/jenkins/installers/check_install_golang.sh
+++ b/hack/jenkins/installers/check_install_golang.sh
@@ -22,7 +22,7 @@ if (($# < 1)); then
   exit 1
 fi
 
-VERSION_TO_INSTALL=1.19.3
+VERSION_TO_INSTALL=1.19.2
 INSTALL_PATH=${1}
 
 function current_arch() {
diff --git a/pkg/minikube/assets/addons.go b/pkg/minikube/assets/addons.go
index a02ef4c49..d8194d7d2 100644
--- a/pkg/minikube/assets/addons.go
+++ b/pkg/minikube/assets/addons.go
@@ -296,7 +296,7 @@ var Addons = map[string]*Addon{
 			"pod.yaml",
 			"0640"),
 	}, false, "kubevirt", "3rd party (KubeVirt)", "", "https://minikube.sigs.k8s.io/docs/tutorials/kubevirt/", map[string]string{
-		"Kubectl": "bitnami/kubectl:1.24.7@sha256:195f5a7a40cfb06e308701ae850abfa436d23baf9d39c0282298e540c9d07863",
+		"Kubectl": "bitnami/kubectl:1.17@sha256:de642e973d3d0ef60e4d0a1f92286a9fdae245535c5990d4762bbe86fcf95887",
 	}, map[string]string{
 		"Kubectl": "docker.io",
 	}),
diff --git a/pkg/minikube/bootstrapper/bsutil/ktmpl/v1beta3.go b/pkg/minikube/bootstrapper/bsutil/ktmpl/v1beta3.go
index f6df7e499..c2b73c635 100644
--- a/pkg/minikube/bootstrapper/bsutil/ktmpl/v1beta3.go
+++ b/pkg/minikube/bootstrapper/bsutil/ktmpl/v1beta3.go
@@ -78,6 +78,9 @@ authentication:
   x509:
     clientCAFile: {{.ClientCAFile}}
 cgroupDriver: {{.CgroupDriver}}
+{{- range $key, $val := .KubeletConfigOpts}}
+{{$key}}: {{$val}}
+{{- end}}
 clusterDomain: "{{if .DNSDomain}}{{.DNSDomain}}{{else}}cluster.local{{end}}"
 # disable disk resource management by default
 imageGCHighThresholdPercent: 100
diff --git a/pkg/minikube/bootstrapper/bsutil/kubeadm.go b/pkg/minikube/bootstrapper/bsutil/kubeadm.go
index cbe857d08..112a395de 100644
--- a/pkg/minikube/bootstrapper/bsutil/kubeadm.go
+++ b/pkg/minikube/bootstrapper/bsutil/kubeadm.go
@@ -109,6 +109,7 @@ func GenerateKubeadmYAML(cc config.ClusterConfig, n config.Node, r cruntime.Mana
 		ControlPlaneAddress        string
 		KubeProxyOptions           map[string]string
 		ResolvConfSearchRegression bool
+		KubeletConfigOpts          map[string]string
 	}{
 		CertDir:           vmpath.GuestKubernetesCertsDir,
 		ServiceCIDR:       constants.DefaultServiceCIDR,
@@ -133,6 +134,7 @@ func GenerateKubeadmYAML(cc config.ClusterConfig, n config.Node, r cruntime.Mana
 		ControlPlaneAddress:        constants.ControlPlaneAlias,
 		KubeProxyOptions:           createKubeProxyOptions(k8s.ExtraOptions),
 		ResolvConfSearchRegression: HasResolvConfSearchRegression(k8s.KubernetesVersion),
+		KubeletConfigOpts:          kubeletConfigOpts(k8s.ExtraOptions),
 	}
 
 	if k8s.ServiceCIDR != "" {
@@ -215,3 +217,17 @@ func HasResolvConfSearchRegression(k8sVersion string) bool {
 	}
 	return versionSemver.EQ(semver.Version{Major: 1, Minor: 25})
 }
+
+// kubeletConfigOpts extracts only those kubelet extra options allowed by kubeletConfigParams.
+func kubeletConfigOpts(extraOpts config.ExtraOptionSlice) map[string]string {
+	args := map[string]string{}
+	for _, eo := range extraOpts {
+		if eo.Component != Kubelet {
+			continue
+		}
+		if config.ContainsParam(kubeletConfigParams, eo.Key) {
+			args[eo.Key] = eo.Value
+		}
+	}
+	return args
+}
diff --git a/pkg/minikube/bootstrapper/bsutil/kubelet.go b/pkg/minikube/bootstrapper/bsutil/kubelet.go
index dc3a00999..b358d0630 100644
--- a/pkg/minikube/bootstrapper/bsutil/kubelet.go
+++ b/pkg/minikube/bootstrapper/bsutil/kubelet.go
@@ -36,6 +36,11 @@ import (
 	"k8s.io/minikube/pkg/util"
 )
 
+// kubeletConfigParams are the only allowed kubelet parameters for kubeadmin config file and not to be used as kubelet flags
+var kubeletConfigParams = []string{
+	"localStorageCapacityIsolation",
+}
+
 func extraKubeletOpts(mc config.ClusterConfig, nc config.Node, r cruntime.Manager) (map[string]string, error) {
 	k8s := mc.KubernetesConfig
 	version, err := util.ParseKubernetesVersion(k8s.KubernetesVersion)
@@ -95,6 +100,11 @@ func extraKubeletOpts(mc config.ClusterConfig, nc config.Node, r cruntime.Manage
 		extraOpts["feature-gates"] = kubeletFeatureArgs
 	}
 
+	// filter out non-flag extra kubelet config options
+	for _, opt := range kubeletConfigParams {
+		delete(extraOpts, opt)
+	}
+
 	return extraOpts, nil
 }
 


Test patch:
diff --git a/.github/workflows/update-gotestsum-version.yml b/.github/workflows/update-gotestsum-version.yml
index 8ef21b9fc..d50133724 100644
--- a/.github/workflows/update-gotestsum-version.yml
+++ b/.github/workflows/update-gotestsum-version.yml
@@ -6,7 +6,7 @@ on:
     - cron: "0 10 * * 1"
 env:
   GOPROXY: https://proxy.golang.org
-  GO_VERSION: '1.19.3'
+  GO_VERSION: '1.19.2'
 permissions:
   contents: read
 
@@ -22,10 +22,7 @@ jobs:
         id: bumpGotestsum
         run: |
           make update-gotestsum-version
-          # The following is to support multiline with GITHUB_OUTPUT, see https://docs.github.com/en/actions/using-workflows/workflow-commands-for-github-actions#multiline-strings
-          echo "changes<<EOF" >> $GITHUB_OUTPUT
-          echo "$(git status --porcelain)" >> $GITHUB_OUTPUT
-          echo "EOF" >> $GITHUB_OUTPUT
+          echo "changes=$(git status --porcelain)" >> $GITHUB_OUTPUT
       - name: Create PR
         if: ${{ steps.bumpGotestsum.outputs.changes != '' }}
         uses: peter-evans/create-pull-request@b4d51739f96fca8047ad065eccef63442d8e99f7
diff --git a/pkg/minikube/bootstrapper/bsutil/kubeadm_test.go b/pkg/minikube/bootstrapper/bsutil/kubeadm_test.go
index baa9cebe8..8a9548b3b 100644
--- a/pkg/minikube/bootstrapper/bsutil/kubeadm_test.go
+++ b/pkg/minikube/bootstrapper/bsutil/kubeadm_test.go
@@ -307,3 +307,27 @@ func TestEtcdExtraArgs(t *testing.T) {
 		t.Errorf("machines mismatch (-want +got):\n%s", diff)
 	}
 }
+
+func TestKubeletConfig(t *testing.T) {
+	expected := map[string]string{
+		"localStorageCapacityIsolation": "false",
+	}
+	extraOpts := append(getExtraOpts(), []config.ExtraOption{
+		{
+			Component: Kubelet,
+			Key:       "unsupported-config-option",
+			Value:     "any",
+		}, {
+			Component: Kubelet,
+			Key:       "localStorageCapacityIsolation",
+			Value:     "false",
+		}, {
+			Component: Kubelet,
+			Key:       "kubelet.cgroups-per-qos",
+			Value:     "false",
+		}}...)
+	actual := kubeletConfigOpts(extraOpts)
+	if diff := cmp.Diff(expected, actual); diff != "" {
+		t.Errorf("machines mismatch (-want +got):\n%s", diff)
+	}
+}