Change:
I am experiencing some weird issues when enabling HTTP liveness probes for some containers. I configure the following for all HTTP-serving containers (this is part of a template used by a replication controller with a single replica): When this is configured, the following issues start to occur: One pod is already running without problems. When I add another replication controller which creates a new pod (also with the liveness probe set), the first pod (the one that was already running) is restarted. This happens even when the two pods are in different namespaces. But when a pod's HTTP server fails completely (connection to the port is refused), the liveness probe will not trigger and the pod will not be restarted. This behavior occurs consistently when I start with no replication controllers and then add them one by one. When I disable the liveness probes on the affected pods/templates, they are no longer restarted in this manner.
You say you're starting two replication controllers-- can you verify that their label selectors do not intersect?
Yes, each replication controller has a selector over a single label in the form of so the labels do not overlap. Also, this problem occurs even when the apps are in different namespaces and goes away when the liveness probe is removed.
Thanks-- last question-- Do the pods have to be scheduled on the same node for this to occur? Your second point I believe is a problem here: we treat an error during HTTP GET as healthy for some inexplicable reason. I can't really explain your first observation, unless perhaps the pods and containers have the same names, but in different namespaces AND kubelet has a namespace handling bug?
cc
"Unknown" != "Healthy" I think it's an open question about whether we want to treat a GET failure as "Unhealthy" we certainly could... --brendan , Daniel Smith wrote:
Giving this to Saad. I think it's pretty clear that if kubelet can't do an HTTP GET, then the pod isn't healthy-- they're on the same host, HTTP get can't fail for network reasons. Our internal system makes this assumption. (I broke it once!)
tl;dr... When a second replication controller is the hash of the container of the first replication controller changes causing it to be killed and recreated. The hash changes because the LivenessProbe pointer changes. Need to figure out why that happens. Spent time trying to understand why creating two replication controllers, both with http liveness probes, causes the container from the first replication controller to be killed and then recreated when the second replication controller is created. It turns out, as soon as the second replication controller is created, the hash of the container from the first replication controller changes and thus the container is killed: Let's see how the expected and actual hashes are computed and how they end up differing. The actual hash is pulled from the docker container name (dockerContainers.FindPodContainer): The expected hash is created from the object. Specifically, this code (): Expected hash before second replication controller is : Expected hash after second replication controller is : The difference between the container before and after is: So the hash changes because the LivenessProbe pointer in the object changes. Need to figure out why this happens.
Great detective work-- I think it's clear that we shouldn't be including pointers in our hash, so we need to revisit that and explicitly hash only the things that we care about.
Adding second replication controller changes the pointer of LivenessProbe in container config is another mystery. We need to make sure not randomly update other fields too.
Dug in to this some more: confirmed that the value of the original container's LivenessProbe doesn't actually change when the second replication controller is created--only the pointer to the LivenessProbe object changes. It turns out that the instance of the  object being used to compute the expected hash is not guaranteed to be the same instance for the lifespan of the actual docker container. In fact, every time there is an etcd update (e.g. a new pod is created) a new boundpods object is created and although the values of the existing pod/container container objects don't change, because new instances are created the pointers do change (by design). So the fix for this is to ensure that the expected hash is computed only on container values not pointers. One way to ensure this would be to make sure the container object specifies no pointer fields. However, LivenessProbe and Lifecycle, since they are optional, must be able to be set to . Thus, pointer fields can't simply be converted to value fields. Another approach would be to only hash some subset of the values (maybe the name and image) of the container. However, containers should be destroyed/recreated (meaning the hash must change) if the actual values of any of the container fields changes (e.g. the cpu, memory, volume mounts, liveness probe path, etc.), not just the container name/image. Thus, pointer fields can't simply be disregarded for the hash computation. Instead the fix must modify the expected hash to be computed on the complete deep values of the entire container object (including the actual LivenessProbe values, instead of the pointer). The easiest way to do this is to marshal the container object to json and compute the hash from the resulting byte array.

Discussion:
func HashContainer(container *api.Container) uint64 { hash := adler32.New() <del> fmt.Fprintf(hash, "%#v", *container) </del> <ins> util.DeepHashObject(hash, *container) </ins> return uint64(hash.Sum32()) }

"github.com/GoogleCloudPlatform/kubernetes/pkg/api" "github.com/GoogleCloudPlatform/kubernetes/pkg/credentialprovider" <ins> "github.com/GoogleCloudPlatform/kubernetes/pkg/util" </ins> docker "github.com/fsouza/go-dockerclient" )

func verifyPackUnpack(t *testing.T, podNamespace, manifestUUID, podName, containerName string) { container := &api.Container{Name: containerName} hasher := adler32.New() <del> data := fmt.Sprintf("%#v", *container) hasher.Write([]byte(data)) </del> <ins> util.DeepHashObject(hasher, *container) </ins> computedHash := uint64(hasher.Sum32()) podFullName := fmt.Sprintf("%s.%s", podName, podNamespace) name := BuildDockerName(manifestUUID, podFullName, container)

<ins> /* Copyright 2014 Google Inc. All rights reserved. Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at http://www.apache.org/licenses/LICENSE-2.0 Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License. */ package util import ( "hash" "github.com/davecgh/go-spew/spew" ) // DeepHashObject writes specified object to hash using the spew library // which follows pointers and prints actual values of the nested objects // ensuring the hash does not change when a pointer changes. func DeepHashObject(hasher hash.Hash, objectToWrite interface{}) { spew.Fprintf(hasher, "%#v", objectToWrite) } </ins>

<ins> /* Copyright 2014 Google Inc. All rights reserved. Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at http://www.apache.org/licenses/LICENSE-2.0 Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License. */ package util import ( "hash/adler32" "testing" ) type wheel struct { radius uint32 } type unicycle struct { primaryWheel   *wheel licencePlateID string } func TestDeepObjectPointer(t *testing.T) { // Arrange wheel1 := wheel{radius: 17} wheel2 := wheel{radius: 22} wheel3 := wheel{radius: 17} myUni1 := unicycle{licencePlateID: "blah", primaryWheel: &wheel1} myUni2 := unicycle{licencePlateID: "blah", primaryWheel: &wheel2} myUni3 := unicycle{licencePlateID: "blah", primaryWheel: &wheel3} hasher1 := adler32.New() hasher2 := adler32.New() hasher3 := adler32.New() // Act DeepHashObject(hasher1, myUni1) hash1 := hasher1.Sum32() DeepHashObject(hasher2, myUni2) hash2 := hasher2.Sum32() DeepHashObject(hasher3, myUni3) hash3 := hasher3.Sum32() // Assert if hash1 == hash2 { t.Errorf("hash1 (%d) and hash2(%d) must be different because they have different values for wheel size", hash1, hash2) } if hash1 != hash3 { t.Errorf("hash1 (%d) and hash3(%d) must be the same because although they point to different objects, they have the same values for wheel size", hash1, hash3) } } </ins>