Change:
<!-- This form is for bug reports and feature requests ONLY! If you're looking for help check and the . --Is this a BUG REPORT or FEATURE REQUEST?: What happened: During the update from v1.7.1 to v1.8.0 all docker containers were restarted, because container spec hash changed. What you expected to happen: No restart of the docker containers, because they did not change at all. How to reproduce it (as minimally and precisely as possible): get yourself a k8s cluster in v1.7.x update (at least the kubelet) to v1.8.0 Anything else we need to know?: Environment: Kubernetes version (use ):Cloud provider or hardware configuration**: AWS OS (e.g. from /etc/os-release):Kernel (e.g. ):Install tools: Others:
/area kubelet
I ran into this earlier today. I was able to update the control plane & kube-proxy to 1.8.1 without any problems. Once I upgraded the kubelet though it caused container restarts.
/sig apps /sig api-machinery
the log message one receives is at: The function, that calculates the hashes is at: The hash function seems to not have changed. I suspect changes in the object itself, which is at: One can see: but:
But the new field does not initiated, so the restart seems unreasonable.
the result i tested was that the spew pkg also dumps the new add field(with nil value) to the hasher. is the new field. so, if the container use , then new MountPropagationMode will be included into the hash input, the hash result changed. other new fields are in the same situation.
Do you have any idea about preventing this?
We've hit this as well. Caused a cluster-wide restart of every container.
i also hit this issue as well, when I am upgrading cluster from v1.7.4 to v1.8.5. especially, the kubelet version from v1.7.4 to v1.8.5
Still seems to happen in 1.9.0
I can confirm this. v1.9.0 has introduced a new field attribute for , which is causing the hash value get changed.
It will exist when we use docker. I remember had described it. There are details in this issues below:
thanks for digging out the old issue! Yes, any new field to the container spec would cause the hash to change. This is not a new issue but new fields are to the container spec rarely (compared with pod spec) so it doesn't happen so often. As discussed in , a new way to generate the hash (e.g., using only updatable fields) or other method for kubelet to examine and compare the state of the container against the spec is needed. Calling this (i.e., container restart) out in the 1.9 release note would be nice.
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
Stale issues rot after 30d of inactivity. Mark the issue as fresh with . Rotten issues close after an additional 30d of inactivity. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle rotten /remove-lifecycle stale
/close
Just wondering, why don't we calculate the container hash by since container image is the only mutable field? I know we have to do some refactoring to , but wouldn't it be more reliable?

Discussion:
name = "go_default_test", srcs = [ "cache_test.go", <ins> "container_hash_test.go", </ins> "helpers_test.go", "ref_test.go", "runtime_cache_test.go",

<ins> /* Copyright 2019 The Kubernetes Authors. Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at http://www.apache.org/licenses/LICENSE-2.0 Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License. */ package container import ( "encoding/json" "testing" "k8s.io/api/core/v1" ) var ( sampleContainer = ` { "name": "test_container", "image": "foo/image:v1", "command": [ "/bin/testcmd" ], "args": [ "/bin/sh", "-c", "echo abc" ], "ports": [ { "containerPort": 8001 } ], "env": [ { "name": "ENV_FOO", "value": "bar" }, { "name": "ENV_BAR", "valueFrom": { "secretKeyRef": { "name": "foo", "key": "bar", "optional": true } } } ], "resources": { "limits": { "foo": "1G" }, "requests": { "foo": "500M" } } } ` sampleV115HashValue = uint64(0x311670a) sampleV116HashValue = sampleV115HashValue ) func TestConsistentHashContainer(t *testing.T) { container := &v1.Container{} if err := json.Unmarshal([]byte(sampleContainer), container); err != nil { t.Error(err) } currentHash := HashContainer(container) if currentHash != sampleV116HashValue { t.Errorf("mismatched hash value with v1.16") } if currentHash != sampleV115HashValue { t.Errorf("mismatched hash value with v1.15") } } </ins>

package container import ( <ins> "encoding/json" </ins> "fmt" "hash/fnv" "strings"

// HashContainer returns the hash of the container. It is used to compare // the running container with its desired spec. <ins> // Note: remember to update hashValues in container_hash_test.go as well. </ins> func HashContainer(container *v1.Container) uint64 { hash := fnv.New32a() <del> hashutil.DeepHashObject(hash, *container) </del> <ins> // Omit nil or empty field when calculating hash value // Please see https://github.com/kubernetes/kubernetes/issues/53644 containerJson, _ := json.Marshal(container) hashutil.DeepHashObject(hash, containerJson) </ins> return uint64(hash.Sum32()) }

assert.Equal(t, tt.expectedPortMappings, actual, "[%d]", i) } } <ins> func TestHashContainer(t *testing.T) { testCases := []struct { name          string image         string args          []string containerPort int32 expectedHash  uint64 }{ { name:  "test_container", image: "foo/image:v1", args: []string{ "/bin/sh", "-c", "echo abc", }, containerPort: int32(8001), expectedHash:  uint64(0x3c42280f), }, } for _, tc := range testCases { container := v1.Container{ Name:  tc.name, Image: tc.image, Args:  tc.args, Ports: []v1.ContainerPort{{ContainerPort: tc.containerPort}}, } hashVal := HashContainer(&container) assert.Equal(t, tc.expectedHash, hashVal, "the hash value here should not be changed.") } } </ins>