Change:
Hit an issue with the pet set states switching from running state to init state. Initially I thought the pods aren't running but later figured out that the containers are actually running but the container state is not consistent with the pod state in kubernetes api.
This sounds like a bug introduced by init containers. Thanks for such detail issue reports and will try to reproduce it. cc/
Just saw a flake test , which might have the same cause as this issue.
I couldn't reproduce the issue at all. Also the flake reported at is never re-occur. I will leave this issue open for a while.
I believe we're hitting this issue too - running a zookeeper petset with init containers, the pod starts up, init containers run, main container runs. Then a short time later the pod status goes back to Init 0/1, but the main container is still running If I can reproduce it reliably what debug output would be helpful?
you are hitting this with 1.4 release? how many kubelet nodes?
If you can reproduce the issue, could you please provide the following information: 1) docker info 2) zookeeper petset podspec 3) kubectl describe pod <pod-id4) related kubelet logs Thanks
This needs to be triaged as a release-blocker or not for 1.5
Not a blocker because we have had no recreations.
Hi, I've hit a similar issue here, running Kubernetes on CoreOS, except it doesn't even create all the pods, just the first one. If there's anything else you need, please let me know. CoreOS Version Kubernetes Cluster Version kubectl get pods state kubectl describe pod Container State kubectl describe service kubectl describe petset kubectl get nodes kubectl describe node docker info PV Spec pv mount state inside container
Hi, Nevermind, I got the zookeeper example working. It was a problem with a miss configuration on kube-controller.
I have this issue also with the new StatefulSets on 1.5.1 This is a pod and the StatefulSet:
This line: Get https://10.10.4.50:6443/api/v1/namespaces/default/secrets/default-token-jd4sr: dial tcp 10.10.4.50:6443: getsockopt: connection refused ​ Usually implies you have an error in your cluster configuration or some sort of transient problem. It is unlikely to be related to StatefulSets themselves, but a more general pod problem.
We have also seen this problem with pods. in 1.3.9. The pod has a init container. It looks like the init container actually completes successfully and later on the container starts and runs correctly ( with the output of the init container). However the pod state is stuck at pending. I will follow up with more information and try to isolate the problem in a way that I can share in this thread.
From what I see, these issues are related to configuration of the StatefulSet and init containers for particular workloads, rather than being issues with the features themselves.
We have run into what appears to be a similar issue. Pod phase is "Pending", but the container is running and ready. Aside from and VerifyControllerAttachedVolume logs, the only ones I see for this pod are
As an update to my previous comment, the pods start up in the expected state, so they will be in the "Running" phase and "ready=true". However, when the pod is restarted (for example when the coreOS cluster kubernetes is running on updates itself), the pods start up in the bad state with the "Pending" phase, but the container is running and ready.
I have encountered this problem too,hope someone can solve.
The initial issue was filed against init container feature. It looks like initial container is finished, all user containers are running properly, but the initContainer itself stuck at waiting state which cause the entire pod stuck at pending phase. But we never reproduce the issue, and e2e test only reported once with the similar failures since the day the issue was reported. The following issues reported by the users are normally caused by misconfigured cluster or bad initContainer itself. I am unassign this from myself, and re-tagged it for sig/apps since I don't think the issue is initContainer feature itself.
what do you mean "when the coreOS cluster kubernetes is running on updates itself"? Do you mean in-place upgrade Kubelet and docker binaries on the node? I noticed all following failure cases are reported on CoreOS. That might be the cause here.
it doesn't look like it is necessarily upgrading the Kubelet or docker binaries, but it looks like it happens whenever the kubernetes node running the service goes down and the service is rescheduled and started on another node.
Given the number of reports and all being on CoreOS, as said, it looks related to CoreOS. Can someone from CoreOS side try to reproduce this? who would be the right people for this?
I've also seen this issue, and we are not running CoreOS. We are running kubernetes 1.5.5
Box has also hit this issue on 1.3.9 and 1.4.7.
Expanding upon my above comment, we've seen this on a variety of resource types. It seems to affect StatefulSets as people have said above, but is also affecting our Deployment resources. It affects resources both with and without PVC. The one unifying pattern is they all have init containers.
Ditto, it's affecting our Deployments as well.
What's the status on this? It seems like a major bug, I'm not sure how it has gone on for so long. When impacted by this bug, scaling, restarting, etc of pods with higher ordinals do not work on StatefulSet. That is, if you have 3 pods for your StatefulSet, and they go into state and you lose mypod-2, kubernetes will never replace it until you manually kill/restart mypod-0 and mypod-1 to remove the state.
/cc
Hi I think I have hit the same issue with init-containers though it is not related to a petset, coreos, or any application. We had our application up and running for several days and suddenly on one machine the pods went back to state from . It seems the simplest way to reproduce this is as follows: a pod with an init-container application come up and pod transition to Running (at this point the init container exists as an docker container) I tried doing this with a pod that had multiple init containers and it seemed like on the last (4/4) container would trigger the condition. Before doing anything: Pod status: Removal state and post: Status post removal: I also started to see these error messages in the kubelet logs: docker version: kubernetes version: In our situtation the removal of the exited containers happened when a host was getting full on disk space an automated process was trying to reclaim space.
sorry to ping but I was able to reproduce this issue pretty reliably without coreos, and with a cluster that is not misconfigured - node has been up for 100+ days, application had been running for several days without issue, I think removing the Exited init-containers might be the issue, and I'm uncertain if that is a known aspect of the design.
Nice catch! I just confirmed that our affected Stateful Sets are in the state you described (init-container Docker containers deleted). After digging further, I realized was experiencing this because our kube nodes have a cron job that deletes old terminated containers that are 2+ days old (the timing lines up perfectly). I can disable / alter our cron job, but it makes me wonder if kubelet's container cleanup routines properly handle this case as well?
Should have clarified in that previous post I did the on two separate containers (Init container 1 and init container 4). I was able to get the Pod to go from -again by running a container with the expected name, i.e.
Here's what I've been using to fix up pods in our system, YMMV: First arg is the pod name (e.g. ) Second arg is the name of the init container in the pod that is supposed to be started, (e.g. ). Seems to have fixed up everything so far, hope that helps someone.
We are having the same problem in one of our clusters. Steps to reproduce: new Deployment with one initcontainer. for it to be in Running state (initcontainer completed successfully and main pod is up and running) init container manually using command. (When we first encountered this problem, the init container was deleted by kubernetes garbage collector). pod will switch to Initializing state and will be in this state indefinitely. Docker container will be running. We are running kubernetes 1.6.3.
This issue happens regularly with almost any deployment I use that create pods with init containers. Further, it may be impacting etcd-operator, at the very least, since it sees the pod as being in an erroneous pending state and waits indefinitely, preventing normal operation. As stated previously, this pending state occurs some time after the pod was created and the init container has previously completed successfully and reported as such. In my experience this bug comes up even without any node or pod restarts, just while running for some period of time. Running k8s v1.6.4 (CoreOS, bootkube)
While a fix is being worked on, the above workarounds have worked for us for 1.5 months. Look into anything that might be deleting/garbagecollecting the containers after the init containers have executed. Disable that if possible, and you'll be unlikely to see this again.
So the bug seems to be that Kubernetes uses stopped container names/UUIDs to determine the status of a Pod with init containers. It should probably just keep the init container's state as Ready or Completed instead, to allow for garbage collection. This is not an intuitive bug to track down and this issue may be overlooked easily in searching for a workaround. Even if this issue is found, the workarounds can be lost in the weeds. With the absence of a pull request to fix the issue, could some documentation be put somewhere to help alleviate some frustration from users that experience this? I'd be happy to write it up, but where do I put it?
Is there an easy workaround like running a sidecar with the same name as the init container? I hate the idea of manually running a docker container with the right name as it won't get auto cleaned up by kube.
I tried adding a sidecar with the same name, but it unfortunately fails validation because the container name collides with the init container name.
I'm surprised this issue hasn't gotten any attention. This consistently causes confusion and pain for our users which are using kubectl. It affects both ReplicaSet and PetSet pods. There doesn't seem to be a realistic work around. Has anyone testing 1.7 to see if the situation has improved? should we see if one of our folk has time to work on it?
Hi, I've got the exact same issue with a Deployment and a initContainer exiting correctly. Impact could be cosmetic for some (pods still running and functioning properly) or functional for the others :) (scripts / automations trusting the status state) Regards,
Any update on this issue? Is it possible to add some kind of filters for kubelet garbage collector to prevent containers matching the filter from being deleted by the collector?
Just did a cluster upgrade to v1.7.2 and this is still happening. I can confirm we have a number of pods with initcontainers created by a daemonset that ran their initcontainer command successfully and are running their container command but kubectl reports them in Init:0/1.
does the priority on this go up given that a number of us are running into issues with this in production?
i've run into this as well with the helm chart. the pods seem to be functioning correctly, but one functional side-effect is the inability to port-forward:
how to solve this issue so that the pod start

Discussion:
Image: container.Image, State: defaultWaitingState, } <del> // Apply some values from the old statuses as the default values. if oldStatus, found := oldStatuses[container.Name]; found { status.RestartCount = oldStatus.RestartCount status.LastTerminationState = oldStatus.LastTerminationState </del> <ins> oldStatus, found := oldStatuses[container.Name] if found { if isInitContainer && oldStatus.State.Terminated != nil { // Do not update status on terminated init containers as // they be removed at any time. status = &oldStatus } else { // Apply some values from the old statuses as the default values. status.RestartCount = oldStatus.RestartCount status.LastTerminationState = oldStatus.LastTerminationState } </ins> } statuses[container.Name] = status }