Issue 1:
When running make test_e2e_node on systemd, I get a single failure related to a test that verifies the stats endpoint.

------------------------------
• Failure [60.090 seconds]
Kubelet
/home/decarr/go/src/k8s.io/kubernetes/test/e2e_node/kubelet_test.go:267
metrics api
/home/decarr/go/src/k8s.io/kubernetes/test/e2e_node/kubelet_test.go:266
when querying /stats/summary
/home/decarr/go/src/k8s.io/kubernetes/test/e2e_node/kubelet_test.go:258
it should report resource usage through the stats api [It]
/home/decarr/go/src/k8s.io/kubernetes/test/e2e_node/kubelet_test.go:257

Expected
<[]stats.ContainerStats | len:2, cap:4>: [

The root is this line:

https://github.com/kubernetes/kubernetes/blob/master/test/e2e_node/kubelet_test.go#L223

On systemd, I am getting back 2 containers where 1 is expected from my stats endpoint.

For example, if I run a pod with a single nginx container I get the following summary:

$ curl http://127.0.0.1:10255/stats/summary
{
"node": {
"nodeName": "127.0.0.1",
"systemContainers": [
{
"name": "kubelet",
"startTime": "2016-03-07T14:17:36Z",
"cpu": {
"time": "2016-03-07T17:55:11Z",
"usageNanoCores": 221608525,
"usageCoreNanoSeconds": 7174601461254
},
"memory": {
"time": "2016-03-07T17:55:11Z",
"usageBytes": 8132644864,
"workingSetBytes": 5518856192,
"pageFaults": 63072461,
"majorPageFaults": 10940
},
"rootfs": {
"availableBytes": 18328821760,
"capacityBytes": 52710469632
},
"logs": {
"availableBytes": 18328821760,
"capacityBytes": 52710469632
},
"userDefinedMetrics": null
},
{
"name": "runtime",
"startTime": "2016-03-07T17:55:16Z",
"cpu": {
"time": "2016-03-07T17:55:00Z",
"usageNanoCores": 17139366,
"usageCoreNanoSeconds": 213037971066
},
"memory": {
"time": "2016-03-07T17:55:00Z",
"usageBytes": 31969280,
"workingSetBytes": 25382912,
"pageFaults": 9133069,
"majorPageFaults": 14
},
"rootfs": {
"availableBytes": 18328821760,
"capacityBytes": 52710469632
},
"logs": {
"availableBytes": 18328821760,
"capacityBytes": 52710469632
},
"userDefinedMetrics": null
}
],
"startTime": "2016-03-07T14:17:36Z",
"cpu": {
"time": "2016-03-07T17:55:11Z",
"usageNanoCores": 221608525,
"usageCoreNanoSeconds": 7174601461254
},
"memory": {
"time": "2016-03-07T17:55:11Z",
"usageBytes": 8132644864,
"workingSetBytes": 5518856192,
"pageFaults": 63072461,
"majorPageFaults": 10940
},
"fs": {
"availableBytes": 18328821760,
"capacityBytes": 52710469632,
"usedBytes": 31680516096
}
},
"pods": [
{
"podRef": {
"name": "nginx-2040093540-7uo6d",
"namespace": "default",
"uid": "a514f7cf-e48d-11e5-bd2c-28d2444e470d"
},
"startTime": "2016-03-07T17:55:02Z",
"containers": [
{
"name": "nginx",
"startTime": "2016-03-07T17:55:04Z",
"cpu": {
"time": "2016-03-07T17:55:04Z",
"usageCoreNanoSeconds": 21626484
},
"memory": {
"time": "2016-03-07T17:55:04Z",
"usageBytes": 8437760,
"workingSetBytes": 2482176,
"pageFaults": 883,
"majorPageFaults": 48
},
"rootfs": {
"availableBytes": 18328821760,
"capacityBytes": 52710469632
},
"logs": {
"availableBytes": 18328821760,
"capacityBytes": 52710469632
},
"userDefinedMetrics": null
},
{
"name": "nginx",
"startTime": "2016-03-07T17:55:04Z",
"cpu": {
"time": "2016-03-07T17:55:04Z",
"usageCoreNanoSeconds": 0
},
"memory": {
"time": "2016-03-07T17:55:04Z",
"usageBytes": 0,
"workingSetBytes": 0,
"pageFaults": 0,
"majorPageFaults": 0
},
"rootfs": {
"availableBytes": 18328821760,
"capacityBytes": 52710469632
},
"logs": {
"availableBytes": 18328821760,
"capacityBytes": 52710469632
},
"userDefinedMetrics": null
}
],
"network": {
"time": "2016-03-07T17:55:03Z",
"rxBytes": 90,
"rxErrors": 0,
"txBytes": 168,
"txErrors": 0
}
}
]
}

I debugged the issue down to here:

https://github.com/kubernetes/kubernetes/blob/master/pkg/kubelet/server/stats/summary.go#L202

After adding more trace to this code block, I saw that nginx was being found twice for duplicate keys in the for loop.

The first time:

I0307 12:55:16.210107 1856 summary.go:189] STATS buildSummaryPods - key /system.slice/docker-37515a45e527fd66a5ab125f689d0113e2ded24bba6b25453d3d78f66fb1f932.scope
I0307 12:55:16.210115 1856 summary.go:196] STATS buildSummaryPods - ref {nginx-2040093540-7uo6d default a514f7cf-e48d-11e5-bd2c-28d2444e470d}

That makes sense because that is the cgroup for the container as reported by docker.

The second time it appeared in the same for loop was for the key:

I0307 12:55:16.210413 1856 summary.go:189] STATS buildSummaryPods - key /system.slice/var-lib-docker-devicemapper-mnt-37515a45e527fd66a5ab125f689d0113e2ded24bba6b25453d3d78f66fb1f932.mount
I0307 12:55:16.210416 1856 summary.go:196] STATS buildSummaryPods - ref {nginx-2040093540-7uo6d default a514f7cf-e48d-11e5-bd2c-28d2444e470d}
I0307 12:55:16.210420 1856 summary.go:205] STATS buildSummaryPods - podStats &{{nginx-2040093540-7uo6d default a514f7cf-e48d-11e5-bd2c-28d2444e470d} 2016-03-07 17:55:02.669481834 +0000 UTC [{nginx 2016-03-07 17:55:04.121735851 +0000 UTC 0xc820e70030 0xc820826100 0xc8208ab7a0 0xc8208ab760 []}] 0xc8208261c0 []}
I0307 12:55:16.210429 1856 summary.go:209] STATS containerName nginx

It appears that we are treating the following as a different container in the pod summary:

/system.slice/var-lib-docker-devicemapper-mnt-37515a45e527fd66a5ab125f689d0113e2ded24bba6b25453d3d78f66fb1f932.mount

Looking on the file system, I see:

cat /sys/fs/cgroup/cpu/system.slice/var-lib-docker-devicemapper-mnt-37515a45e527fd66a5ab125f689d0113e2ded24bba6b25453d3d78f66fb1f932.mount/

Still investigating this some more, but looks like we are getting tripped up in cadvisor by this...

/cc @vishh @dchen1107 @ncdc @sjenning

Issue 2:
Right now, we have the concept packing api query parameters into a struct, converting that to the versioned (serialized wire) format struct, then packing that into URL query parameters. This needs some slight adjustment for API groups.

Options:

Change nothing; every group needs its own e.g. ListOptions struct in the internal and external versions.

Move common options (e.g. ListOptions) to unversioned.

If we do 2, then we need to modify the code here: https://github.com/kubernetes/kubernetes/blob/master/pkg/client/unversioned/request.go#L414

Options for modifying this code:

RESTClient takes a second "options" codec, which takes a struct, returns JSON, which can be converted to query parameters. (must be a second codec because the codec for the body of the request may be binary.)

RESTClient takes a second "options" codec, which takes a struct and returns URL query parameters directly.

Develop the concept of internal unversioned vs. external unversioned (UGH).

2.1 & 2.2 both have the advantage that RESTClient doesn't need to know about "convertor" at all any more.

2.1 can be done just by reusing our existing codec, which is nice, but 2.2 is probably clearer? 2.3 is confusing. Option 1 is verbose and error prone.

@smarterclayton @krousey @wojtek-t @caesarxuchao please all of you argue about this :)