Change:
a standalone kubelet (not registered to an API server) construction short-circuits here: state is never initialized here: pod syncs then panic on this line: Originally posted by in cc /sig node /kind bug /priority important-soon
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
also, if VPA state cannot be constructed here: logging an error, setting state to nil, and continuing means every use of in the pod sync loop is a potential NPE
/assign
/cc
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
This is still happening in standalone alpha kubelets… the sync got further but is still panicking at a different spot
Panic moved to here:
Can you please share the commit-id on whic this is happening? And any kubelet logs surrounding the crash? With latest master (commit-id ) , I see a different crash (another nil client access from initial glance). But my local cluster code hacks kubeDeps.KubeClient = nil before NewMainKubelet(...) so it is different from your CI env. To unblock CI, I will do a defensive fix (skip and for static pods) but I want to understand why this is even getting to .
the latest run I have was on and failed on that line at
no kubelet logs of interest I see, logs seem pretty normal for a standalone kubelet, just that panic and process exit
the panic I saw wasn't on Windows, it was on Linux, but while spelunking, I saw another possible panic. On Windows, NewPodContainerManager() returns: which implements to return nil,nil: which means currentPodMemoryConfig will be nil here:
that certainly is surprising and suggests that something in computePodResizeAction may not be correct in standalone cases or for static pods
I don't really know if making exceptions for static pods is correct when running against an API server where we're reporting status of used resources. would you expect an updated manifest that changes resources to be handled by VPA?
Yes, that's what I want to get to the bottom of. I suspect that for some reason, computePodResizeAction incorrectly determined that a resize is necessary for a best effort pod or burstable w/o CPU limits pod (which resulted in NP access due to ) is a pretty complex function that needs to correctly account for various states a container may be in (not running, restarting etc) and I need a repro to ensure any fix I make actually works as intended.
When I tried this last time by updating the manifest resources and saving it (just out of curiosity) , I found that pod was restarted (not patched - which triggers the resize workflow). But I don't know if it should be officially supported.
I don't have a repro (attempted testing it without skipping computePodResizeAction / doPodResizeAction for static pods) in local cluster + kubeletonly with either best effort static pod or burstable static pod with cpu requests specified. Do you have the pod yaml that caused this issue in your env?
not offhand, I'd have to spin up a cluster manually and try to binary search the static manifests we use to isolate it, and I have negative time to dig with code freeze reviews :-/
If you mean, when VPA points at a mirror pod can it change spec? No, generally. If you mean, should static pod consumed resources be visible via the API? Yes.
From what we can tell, something in the VPA path () thought a static pod's resources needed to trigger the resize code path. I don't know enough about static pod handling to know if that is a category error and VPA should never expect to resize resources used by static pods. That's what part of the proposed fix in does - excludes static pods from being considered by VPA. For a kubelet running against an API server, would you expect manifest-driven pods to be able to make use of resize functionality? If so, a blanket exclusion of static pods from that code path doesn't seem correct.
I think the implementation can support resizing static pods via patch to its mirror. There is some bug in the pod update -pod worker sync handoff where the resize update is lost. IMHO, we should allow resizing static pods as well via its mirror. (Standalone kubelet is a different story - the current implementation terminates and adds new pod). We just haven't considered this scenario and it needs a bit of work (mainly to develop solid e2e test cases for mirror pod resize and standalone scenario). This is something I'd want to take the time to fix the right way in 1.28, so my preference is to make a defensive fix for v1.27 (aka we don't support static pod resizing for 1.27 alpha) rather than trying to wing it.
In HandlePodUpdates, static pods are processed by handleMirrorPod: which dispatches SyncPodUpdate for the original pod, and not the updated mirror pod. For normal pods, SyncPodUpdate is invoked for the updated pods. This is the reason (bug) I didn't see resize updates come through in SyncPod(). I don't know how long the code has been this way, but I'm hesitant and risk-averse to just swap out with so close to code freeze. This is why I feel it is best to disable static pods for resize at this time and do it right in 1.28
Agreed on not making big changes right before code-freeze. Can you open a tracking issue for that and tag on it? As commented in , it seems like the static pod carve-outs and feature gate checks are leaving gaps in that series of statements relative to 1.26 handling event when the feature is disabled... at the very least, it's really hard to reason about, which is not what we want from feature-gated isolated code I think that PR needs to be made more obviously isolated / no-op when the feature gate is disabled, or we should consider dropping VPA out of 1.27
Tracking issue where the repro steps try updating container image of a static pod (no involvement of in-place resize feature)
About excluding computePodResizeAction and doPodResizeAction for static pods, I did try 'update manifest and save' and found that pod was being removed and with updated spec (with FG enabled). I'll verify again with traditional (non debugger) methods for FG enabled & disabled cases ... we are thin on pre-submit signals for this change.
From sig-node discussion just now, it sounds like updating mirror to effect changes to the pod is discouraged. We probably should be more explicit and block in validation if there are no sound reasons to allow updates to static pods via api. With that discussion, and given current state with , my change to exclude static pods for computePodResizeAction and doPodResizeAction is not only defensive but may also be the right fix. I'll verify standalone kubelet manually for FG enabled/disabled cases later after my meeting with Clayton. /cc
I was able to repro the panic with standalone mode in node e2e: I built on top standalone PR - Here's my branch: I ran the node e2e under standalone mode and enabled all alpha features. Invocation: Build log: Kubelet log:
As next step, I will patch in the locally and verify that it results in the test passing.
I patched in and verified that the panic is fixed. With patched in: build log: kubelet log:

Discussion:
package cm import ( <ins> "fmt" </ins> v1 "k8s.io/api/core/v1" "k8s.io/klog/v2"

} func (m *podContainerManagerStub) GetPodCgroupConfig(_ *v1.Pod, _ v1.ResourceName) (*ResourceConfig, error) { <del> return nil, nil </del> <ins> return nil, fmt.Errorf("not implemented") </ins> } func (m *podContainerManagerStub) SetPodCgroupConfig(_ *v1.Pod, _ v1.ResourceName, _ *ResourceConfig) error { <del> return nil </del> <ins> return fmt.Errorf("not implemented") </ins> } func (cm *containerManagerStub) NewPodContainerManager() PodContainerManager {

return cStatus.ExitCode == 0 } <ins> func isInPlacePodVerticalScalingAllowed(pod *v1.Pod) bool { if !utilfeature.DefaultFeatureGate.Enabled(features.InPlacePodVerticalScaling) { return false } if types.IsStaticPod(pod) { return false } return true } </ins> func (m *kubeGenericRuntimeManager) computePodResizeAction(pod *v1.Pod, containerIdx int, kubeContainerStatus *kubecontainer.Status, changes *podActions) bool { container := pod.Spec.Containers[containerIdx] if container.Resources.Limits == nil || len(pod.Status.ContainerStatuses) == 0 {

return err } if len(podContainerChanges.ContainersToUpdate[v1.ResourceMemory]) > 0 || podContainerChanges.UpdatePodResources { <ins> if podResources.Memory == nil { klog.ErrorS(nil, "podResources.Memory is nil", "pod", pod.Name) result.Fail(fmt.Errorf("podResources.Memory is nil for pod %s", pod.Name)) return } </ins> currentPodMemoryConfig, err := pcm.GetPodCgroupConfig(pod, v1.ResourceMemory) if err != nil { klog.ErrorS(err, "GetPodCgroupConfig for memory failed", "pod", pod.Name)

} } if len(podContainerChanges.ContainersToUpdate[v1.ResourceCPU]) > 0 || podContainerChanges.UpdatePodResources { <ins> if podResources.CPUQuota == nil || podResources.CPUShares == nil { klog.ErrorS(nil, "podResources.CPUQuota or podResources.CPUShares is nil", "pod", pod.Name) result.Fail(fmt.Errorf("podResources.CPUQuota or podResources.CPUShares is nil for pod %s", pod.Name)) return } </ins> currentPodCpuConfig, err := pcm.GetPodCgroupConfig(pod, v1.ResourceCPU) if err != nil { klog.ErrorS(err, "GetPodCgroupConfig for CPU failed", "pod", pod.Name)

return changes } <del> if utilfeature.DefaultFeatureGate.Enabled(features.InPlacePodVerticalScaling) { </del> <ins> if isInPlacePodVerticalScalingAllowed(pod) { </ins> changes.ContainersToUpdate = make(map[v1.ResourceName][]containerToUpdateInfo) latestPodStatus, err := m.GetPodStatus(ctx, podStatus.ID, pod.Name, pod.Namespace) if err == nil {

restart := shouldRestartOnFailure(pod) // Do not restart if only the Resources field has changed with InPlacePodVerticalScaling enabled if _, _, changed := containerChanged(&container, containerStatus); changed && <del> (!utilfeature.DefaultFeatureGate.Enabled(features.InPlacePodVerticalScaling) || </del> <ins> (!isInPlacePodVerticalScalingAllowed(pod) || </ins> kubecontainer.HashContainerWithoutResources(&container) != containerStatus.HashWithoutResources) { message = fmt.Sprintf("Container %s definition changed", container.Name) // Restart regardless of the restart policy because the container

// If the container failed the startup probe, we should kill it. message = fmt.Sprintf("Container %s failed startup probe", container.Name) reason = reasonStartupProbe <del> } else if utilfeature.DefaultFeatureGate.Enabled(features.InPlacePodVerticalScaling) && !m.computePodResizeAction(pod, idx, containerStatus, &changes) { </del> <ins> } else if isInPlacePodVerticalScalingAllowed(pod) && !m.computePodResizeAction(pod, idx, containerStatus, &changes) { </ins> // computePodResizeAction updates 'changes' if resize policy requires restarting this container continue } else {

} // Step 7: For containers in podContainerChanges.ContainersToUpdate[CPU,Memory] list, invoke UpdateContainerResources <del> if utilfeature.DefaultFeatureGate.Enabled(features.InPlacePodVerticalScaling) { </del> <ins> if isInPlacePodVerticalScalingAllowed(pod) { </ins> if len(podContainerChanges.ContainersToUpdate) > 0 || podContainerChanges.UpdatePodResources { m.doPodResizeAction(pod, podStatus, podContainerChanges, result) }