Issue 1:
Is this a request for help? (If yes, you should use our troubleshooting guide and community support channels, see http://kubernetes.io/docs/troubleshooting/.):

What keywords did you search in Kubernetes issues before filing this one? (If you have found any duplicates, you should instead reply there.): memory

Is this a BUG REPORT or FEATURE REQUEST? (choose one):

BUG REPORT

Kubernetes version (use kubectl version):

Client Version: version.Info{Major:"1", Minor:"3", GitVersion:"v1.3.6+ae4550c", GitCommit:"ae4550cc9c89a593bcda6678df201db1b208133b", GitTreeState:"not a git tree", BuildDate:"2016-08-30T15:45:51Z", GoVersion:"go1.7", Compiler:"gc", Platform:"darwin/amd64"}

Server Version: version.Info{Major:"1", Minor:"3", GitVersion:"v1.3.6", GitCommit:"ae4550cc9c89a593bcda6678df201db1b208133b", GitTreeState:"clean", BuildDate:"2016-08-26T18:06:06Z", GoVersion:"go1.6.2", Compiler:"gc", Platform:"linux/amd64"}

Environment:

Cloud provider or hardware configuration: AWS

OS (e.g. from /etc/os-release):

NAME=CoreOS

ID=coreos

VERSION=1122.1.0

VERSION_ID=1122.1.0

BUILD_ID=2016-08-11-0252

PRETTY_NAME="CoreOS 1122.1.0 (MoreOS)"

ANSI_COLOR="1;32"

HOME_URL="https://coreos.com/"

BUG_REPORT_URL="https://github.com/coreos/bugs/issues"

Kernel (e.g. uname -a):

Linux xxxx 4.7.0-coreos #1 SMP Thu Aug 11 02:44:21 UTC 2016 x86_64 Intel(R) Xeon(R) CPU E5-2676 v3 @ 2.40GHz GenuineIntel GNU/Linux

Install tools:

Others:

What happened:

cAdvisor reporting 0 memory usage for all containers

What you expected to happen:

cAdvisor to report actual memory usage statistics

How to reproduce it (as minimally and precisely as possible):

curl http://localhost:4194/api/v1.1/containers

Observe that all memory blocks in the json look like this:

"memory": { "usage": 0, "cache": 0, "rss": 0, "working_set": 0, "failcnt": 0, "container_data": { "pgfault": 0, "pgmajfault": 0 }, "hierarchical_data": { "pgfault": 0, "pgmajfault": 0 } },

Anything else do we need to know:

This seems to be a new behavior in 1.3.6

Issue 2:
Is this a request for help? (If yes, you should use our troubleshooting guide and community support channels, see http://kubernetes.io/docs/troubleshooting/.):

What keywords did you search in Kubernetes issues before filing this one? (If you have found any duplicates, you should instead reply there.):

Is this a BUG REPORT or FEATURE REQUEST? (choose one):

Kubernetes version (use kubectl version): 1.4.0 with #33967 patch applied.

Environment:

Cloud provider or hardware configuration: GCE + custom kubernetes

OS (e.g. from /etc/os-release): "CoreOS 1068.8.0 (MoreOS)"

Kernel (e.g. uname -a): : 4.6.3-coreos

Install tools:

Others:

What happened:

But pods seem to get lost. I.e.

kubectl get pods
NAME READY STATUS RESTARTS AGE
...other-pods
abcd-3456205130-83b2q 1/1 Running 0 22s

So first time around it comes up ok. Now I try to delete and create this deployment.

kubectl delete -f abcdController.json
deployment "abcd" deleted
kubectl create -f abcdController.json
deployment "abcd" created

After that I can not see my pod anywhere:

kubectl get pods
NAME READY STATUS RESTARTS AGE
..other-pods-- but abcd is not here

I can see the deployment though

kubectl describe deployment abcd
Name: abcd
Namespace: default
CreationTimestamp: Tue, 04 Oct 2016 11:25:54 +0100
Labels: k8s-app=abcd
kubernetes.io/cluster-service=true
version=v1.1.0-beta2
Selector: k8s-app=abcd,version=v1.1.0-beta2
Replicas: 0 updated | 1 total | 0 available | 0 unavailable
StrategyType: RollingUpdate
MinReadySeconds: 0
RollingUpdateStrategy: 1 max unavailable, 1 max surge
OldReplicaSets: <none>
NewReplicaSet: <none>

But there is no corresponding replica set for it.

Here are the logs from scheduler:

listers.go:68] can not retrieve list of objects using index : object has no meta: object does not implement the Object interfaces
Oct 04 10:25:44 h-stg-core-2 docker[3806]: I1004 10:25:44.570168 1 scheduler.go:135] Failed to bind pod: default/abcd-3456205130-15x25
Oct 04 10:25:44 h-stg-core-2 docker[3806]: E1004 10:25:44.570184 1 scheduler.go:137] scheduler cache ForgetPod failed: pod state wasn't assumed but get forgotten. Pod key: default/abcd-3456205130-15x25
Oct 04 10:25:44 h-stg-core-2 docker[3806]: E1004 10:25:44.570191 1 factory.go:530] Error scheduling default abcd-3456205130-15x25: Operation cannot be fulfilled on pods/binding "abcd-3456205130-15x25": pod abcd-3456205130-15x25 is being deleted, cannot be assigned to a host; retrying
Oct 04 10:25:44 h-stg-core-2 docker[3806]: I1004 10:25:44.570212 1 factory.go:607] Updating pod condition for default/abcd-3456205130-15x25 to (PodScheduled==False)
Oct 04 10:25:44 h-stg-core-2 docker[3806]: W1004 10:25:44.570362 1 factory.go:544] Request for pod default/abcd-3456205130-15x25 already in flight, abandoning
Oct 04 10:25:44 h-stg-core-2 docker[3806]: I1004 10:25:44.570380 1 event.go:217] Event(api.ObjectReference{Kind:"Pod", Namespace:"default", Name:"abcd-3456205130-15x25", UID:"e84beb50-8a1c-11e6-85c8-42010a140005", APIVersion:"v1", ResourceVersion:"6090", FieldPath:""}): type: 'Normal' reason: 'FailedScheduling' Binding rejected: Operation cannot be fulfilled on pods/binding "abcd-3456205130-15x25": pod abcd-3456205130-15x25 is being deleted, cannot be assigned to a host
Oct 04 10:25:46 h-stg-core-2 docker[3806]: Pod name is abcd-3456205130-15x25
Oct 04 10:25:46 h-stg-core-2 docker[3806]: Pod node is h-stg-wk2-1
Oct 04 10:25:46 h-stg-core-2 docker[3806]: E1004 10:25:46.645429 1 factory.go:176] scheduler cache UpdatePod failed: no corresponding pod abcd-3456205130-15x25 in pods of node h-stg-wk2-1
Oct 04 10:25:46 h-stg-core-2 docker[3806]: Pod name is abcd-3456205130-15x25
Oct 04 10:25:46 h-stg-core-2 docker[3806]: Pod node is h-stg-wk2-1
Oct 04 10:25:46 h-stg-core-2 docker[3806]: E1004 10:25:46.722604 1 factory.go:176] scheduler cache UpdatePod failed: no corresponding pod abcd-3456205130-15x25 in pods of node h-stg-wk2-1
Oct 04 10:25:46 h-stg-core-2 docker[3806]: Pod name is abcd-3456205130-15x25
Oct 04 10:25:46 h-stg-core-2 docker[3806]: Pod node is h-stg-wk2-0
Oct 04 10:25:46 h-stg-core-2 docker[3806]: (*api.Pod)(0xc421648000)(&TypeMeta{Kind:,APIVersion:,})

Here are thr logs from the worker node to which the new pod was originally assigned:

Oct 04 10:25:44 h-stg-wk2-1 kubelet[3706]: I1004 10:25:44.357190 3706 kubelet.go:1617] SyncLoop (ADD, "api"): "abcd-3456205130-15x25_default(e84beb50-8a1c-11e6-85c8-42010a140005)"
Oct 04 10:25:44 h-stg-wk2-1 kubelet[3706]: I1004 10:25:44.475171 3706 kubelet.go:1627] SyncLoop (REMOVE, "api"): "abcd-3456205130-15x25_default(e84beb50-8a1c-11e6-85c8-42010a140005)"
Oct 04 10:25:44 h-stg-wk2-1 kubelet[3706]: I1004 10:25:44.475232 3706 kubelet.go:1802] Failed to delete pod "abcd-3456205130-15x25_default(e84beb50-8a1c-11e6-85c8-42010a140005)", err: pod not found
Oct 04 10:25:44 h-stg-wk2-1 kubelet[3706]: I1004 10:25:44.484023 3706 kubelet.go:1617] SyncLoop (ADD, "api"): "abcd-3456205130-15x25_default(e84beb50-8a1c-11e6-85c8-42010a140005)"
Oct 04 10:25:44 h-stg-wk2-1 kubelet[3706]: I1004 10:25:44.492239 3706 kubelet.go:1627] SyncLoop (REMOVE, "api"): "abcd-3456205130-15x25_default(e84beb50-8a1c-11e6-85c8-42010a140005)"
Oct 04 10:25:44 h-stg-wk2-1 kubelet[3706]: I1004 10:25:44.492298 3706 kubelet.go:1802] Failed to delete pod "abcd-3456205130-15x25_default(e84beb50-8a1c-11e6-85c8-42010a140005)", err: pod not found
Oct 04 10:25:44 h-stg-wk2-1 kubelet[3706]: I1004 10:25:44.503775 3706 kubelet.go:1617] SyncLoop (ADD, "api"): "abcd-3456205130-15x25_default(e84beb50-8a1c-11e6-85c8-42010a140005)"
Oct 04 10:25:44 h-stg-wk2-1 kubelet[3706]: I1004 10:25:44.516296 3706 kubelet.go:1627] SyncLoop (REMOVE, "api"): "abcd-3456205130-15x25_default(e84beb50-8a1c-11e6-85c8-42010a140005)"
Oct 04 10:25:44 h-stg-wk2-1 kubelet[3706]: I1004 10:25:44.516384 3706 kubelet.go:1802] Failed to delete pod "abcd-3456205130-15x25_default(e84beb50-8a1c-11e6-85c8-42010a140005)", err: pod not found
Oct 04 10:25:44 h-stg-wk2-1 kubelet[3706]: I1004 10:25:44.542153 3706 kubelet.go:1617] SyncLoop (ADD, "api"): "abcd-3456205130-15x25_default(e84beb50-8a1c-11e6-85c8-42010a140005)"
Oct 04 10:25:44 h-stg-wk2-1 kubelet[3706]: I1004 10:25:44.561550 3706 kubelet.go:1627] SyncLoop (REMOVE, "api"): "abcd-3456205130-15x25_default(e84beb50-8a1c-11e6-85c8-42010a140005)"
Oct 04 10:25:44 h-stg-wk2-1 kubelet[3706]: I1004 10:25:44.561617 3706 kubelet.go:1802] Failed to delete pod "abcd-3456205130-15x25_default(e84beb50-8a1c-11e6-85c8-42010a140005)", err: pod not found
Oct 04 10:25:44 h-stg-wk2-1 kubelet[3706]: I1004 10:25:44.567041 3706 kubelet.go:1617] SyncLoop (ADD, "api"): "abcd-3456205130-15x25_default(e84beb50-8a1c-11e6-85c8-42010a140005)"
Oct 04 10:25:44 h-stg-wk2-1 kubelet[3706]: I1004 10:25:44.588436 3706 kubelet.go:1627] SyncLoop (REMOVE, "api"): "abcd-3456205130-15x25_default(e84beb50-8a1c-11e6-85c8-42010a140005)"
Oct 04 10:25:44 h-stg-wk2-1 kubelet[3706]: I1004 10:25:44.588497 3706 kubelet.go:1802] Failed to delete pod "abcd-3456205130-15x25_default(e84beb50-8a1c-11e6-85c8-42010a140005)", err: pod not found
Oct 04 10:25:44 h-stg-wk2-1 kubelet[3706]: I1004 10:25:44.599215 3706 kubelet.go:1617] SyncLoop (ADD, "api"): "abcd-3456205130-15x25_default(e84beb50-8a1c-11e6-85c8-42010a140005)"
Oct 04 10:25:44 h-stg-wk2-1 kubelet[3706]: I1004 10:25:44.657951 3706 docker_manager.go:1906] Need to restart pod infra container for "abcd-3456205130-15x25_default(e84beb50-8a1c-11e6-85c8-42010a140005)" because it is not found
Oct 04 10:25:44 h-stg-wk2-1 kubelet[3706]: E1004 10:25:44.688467 3706 docker_manager.go:746] Logging security options: {key:seccomp value:unconfined msg:}
Oct 04 10:25:44 h-stg-wk2-1 kubelet[3706]: E1004 10:25:44.842306 3706 docker_manager.go:1701] Failed to create symbolic link to the log file of pod "abcd-3456205130-15x25_default(e84beb50-8a1c-11e6-85c8-42010a140005)" container "POD": symlink /var/log/containers/abcd-3456205130-15x25_default_POD-01ee72d6577afa46241df7223f26cc39cd7283c73ee0da2f4b98bbef7a3d6d79.log: no such file or directory
Oct 04 10:25:44 h-stg-wk2-1 kubelet[3706]: I1004 10:25:44.990313 3706 docker_manager.go:2167] Determined pod ip after infra change: "abcd-3456205130-15x25_default(e84beb50-8a1c-11e6-85c8-42010a140005)": "10.120.2.11"
Oct 04 10:25:45 h-stg-wk2-1 kubelet[3706]: I1004 10:25:45.023612 3706 provider.go:119] Refreshing cache for provider: *gcp_credentials.dockerConfigKeyProvider
Oct 04 10:25:45 h-stg-wk2-1 kubelet[3706]: I1004 10:25:45.024235 3706 config.go:156] body of failing http response: &{0x67b3f0 0xc421ada940 0x67b210}
Oct 04 10:25:45 h-stg-wk2-1 kubelet[3706]: E1004 10:25:45.024263 3706 metadata.go:142] while reading 'google-dockercfg' metadata: http status code: 404 while fetching url http://metadata.google.internal./computeMetadata/v1/instance/attributes/google-dockercfg
Oct 04 10:25:45 h-stg-wk2-1 kubelet[3706]: I1004 10:25:45.024290 3706 provider.go:119] Refreshing cache for provider: *gcp_credentials.dockerConfigUrlKeyProvider
Oct 04 10:25:45 h-stg-wk2-1 kubelet[3706]: I1004 10:25:45.025291 3706 config.go:156] body of failing http response: &{0x67b3f0 0xc421ebfc40 0x67b210}
Oct 04 10:25:45 h-stg-wk2-1 kubelet[3706]: E1004 10:25:45.025342 3706 metadata.go:159] while reading 'google-dockercfg-url' metadata: http status code: 404 while fetching url http://metadata.google.internal./computeMetadata/v1/instance/attributes/google-dockercfg-url
Oct 04 10:25:45 h-stg-wk2-1 kubelet[3706]: I1004 10:25:45.688160 3706 kubelet.go:1652] SyncLoop (PLEG): "abcd-3456205130-15x25_default(e84beb50-8a1c-11e6-85c8-42010a140005)", event: &pleg.PodLifecycleEvent{ID:"e84beb50-8a1c-11e6-85c8-42010a140005", Type:"ContainerStarted", Data:"01ee72d6577afa46241df7223f26cc39cd7283c73ee0da2f4b98bbef7a3d6d79"}
Oct 04 10:25:46 h-stg-wk2-1 kubelet[3706]: I1004 10:25:46.575266 3706 kube_docker_client.go:309] Stop pulling image "eu.gcr.io/ourproject/abcd:20160915.125444.4f55766": "Status: Downloaded newer image for eu.gcr.io/ourproject/abcd:20160915.125444.4f55766"
Oct 04 10:25:46 h-stg-wk2-1 kubelet[3706]: E1004 10:25:46.592392 3706 docker_manager.go:746] Logging security options: {key:seccomp value:unconfined msg:}
Oct 04 10:25:46 h-stg-wk2-1 kubelet[3706]: E1004 10:25:46.665024 3706 docker_manager.go:1701] Failed to create symbolic link to the log file of pod "abcd-3456205130-15x25_default(e84beb50-8a1c-11e6-85c8-42010a140005)" container "abcd": symlink /var/log/containers/abcd-3456205130-15x25_default_abcd-5e1101ed8ce36529cc2013333649342d1b4e77c1010b434103cef97225bfd712.log: no such file or directory
Oct 04 10:25:46 h-stg-wk2-1 kubelet[3706]: I1004 10:25:46.693803 3706 kubelet.go:1633] SyncLoop (DELETE, "api"): "abcd-3456205130-15x25_default(e84beb50-8a1c-11e6-85c8-42010a140005)"
Oct 04 10:25:46 h-stg-wk2-1 kubelet[3706]: I1004 10:25:46.701327 3706 kubelet.go:1652] SyncLoop (PLEG): "abcd-3456205130-15x25_default(e84beb50-8a1c-11e6-85c8-42010a140005)", event: &pleg.PodLifecycleEvent{ID:"e84beb50-8a1c-11e6-85c8-42010a140005", Type:"ContainerStarted", Data:"5e1101ed8ce36529cc2013333649342d1b4e77c1010b434103cef97225bfd712"}
Oct 04 10:25:46 h-stg-wk2-1 kubelet[3706]: I1004 10:25:46.765815 3706 kubelet.go:1633] SyncLoop (DELETE, "api"): "abcd-3456205130-15x25_default(e84beb50-8a1c-11e6-85c8-42010a140005)"
Oct 04 10:25:46 h-stg-wk2-1 kubelet[3706]: I1004 10:25:46.829842 3706 kubelet.go:1627] SyncLoop (REMOVE, "api"): "abcd-3456205130-15x25_default(e84beb50-8a1c-11e6-85c8-42010a140005)"
Oct 04 10:25:46 h-stg-wk2-1 kubelet[3706]: I1004 10:25:46.834246 3706 kubelet_pods.go:684] Killing unwanted pod "abcd-3456205130-15x25"
Oct 04 10:25:46 h-stg-wk2-1 kubelet[3706]: I1004 10:25:46.834344 3706 docker_manager.go:1497] Killing container "5e1101ed8ce36529cc2013333649342d1b4e77c1010b434103cef97225bfd712 abcd default/abcd-3456205130-15x25" with 0 second grace period
Oct 04 10:25:47 h-stg-wk2-1 kubelet[3706]: I1004 10:25:47.125227 3706 kube_docker_client.go:309] Stop pulling image "eu.gcr.io/ourproject/abcd:20160915.125444.4f55766": "Status: Image is up to date for eu.gcr.io/ourproject/abcd:20160915.125444.4f55766"
Oct 04 10:25:47 h-stg-wk2-1 kubelet[3706]: E1004 10:25:47.145497 3706 docker_manager.go:746] Logging security options: {key:seccomp value:unconfined msg:}
Oct 04 10:25:47 h-stg-wk2-1 kubelet[3706]: E1004 10:25:47.229814 3706 docker_manager.go:1701] Failed to create symbolic link to the log file of pod "abcd-3456205130-15x25_default(e84beb50-8a1c-11e6-85c8-42010a140005)" container "abcd": symlink /var/log/containers/abcd-3456205130-15x25_default_abcd-890b1ce28b2e4d504143f8328671b1557afd9a2354c4935a18cc1c7fc8c7b20b.log: no such file or directory
Oct 04 10:25:49 h-stg-wk2-1 kubelet[3706]: I1004 10:25:48.873541 3706 docker_manager.go:1536] Container "5e1101ed8ce36529cc2013333649342d1b4e77c1010b434103cef97225bfd712 abcd default/abcd-3456205130-15x25" exited after 2.039169678s
Oct 04 10:25:49 h-stg-wk2-1 kubelet[3706]: I1004 10:25:49.047405 3706 docker_manager.go:1497] Killing container "01ee72d6577afa46241df7223f26cc39cd7283c73ee0da2f4b98bbef7a3d6d79 default/abcd-3456205130-15x25" with 0 second grace period
Oct 04 10:25:49 h-stg-wk2-1 kubelet[3706]: I1004 10:25:49.091565 3706 docker_manager.go:1536] Container "01ee72d6577afa46241df7223f26cc39cd7283c73ee0da2f4b98bbef7a3d6d79 default/abcd-3456205130-15x25" exited after 44.115046ms
Oct 04 10:25:49 h-stg-wk2-1 kubelet[3706]: I1004 10:25:49.704178 3706 kubelet_pods.go:684] Killing unwanted pod "abcd-3456205130-15x25"
Oct 04 10:25:49 h-stg-wk2-1 kubelet[3706]: I1004 10:25:49.706872 3706 docker_manager.go:1497] Killing container "890b1ce28b2e4d504143f8328671b1557afd9a2354c4935a18cc1c7fc8c7b20b /" with 30 second grace period

What you expected to happen:

Expected my pod to appear.

How to reproduce it (as minimally and precisely as possible):

Create a deployment, delete a deployment, create it again. seems to happen pretty regularly.

Anything else do we need to know:

Some of the original discussion is on #33967