Issue 1:
As @yujuhong noted it'd be useful to reduce variance of measurements. Additionally we may want to include fluentd pod in the measurements.

Issue 2:
Since 4:25PM the e2e test ResourceUsage should not exceed expected amount has been failing.

Looking at the last batch of merged PRs I can't tell if any of them are responsible, either individually or in combination. Please can the PR authors @eparis @BenTheElder @nikhiljindal @gmarek @feihujiang @yifan-gu have a think about whether their change could have caused:

ResourceUsage should not exceed expected amount.

/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/monitor_resources.go:131
Expected
<map[string]e2e.resourceUsagePerContainer | len:3>: {
"e2e-gce-minion-jt3o": {
"/": {
Name: "",
Timestamp: {sec: 0, nsec: 0, loc: nil},
CPUUsageInCores: 0.32753680271174745,
MemoryUsageInBytes: 5536577126,
MemoryWorkingSetInBytes: 1995254169,
CPUInterval: 0,
},
"/docker-daemon": {
Name: "",
Timestamp: {sec: 0, nsec: 0, loc: nil},
CPUUsageInCores: 0.05330590708577182,
MemoryUsageInBytes: 3765198848,
MemoryWorkingSetInBytes: 674401484,
CPUInterval: 0,
},
"/kubelet": {
Name: "",
Timestamp: {sec: 0, nsec: 0, loc: nil},
CPUUsageInCores: 0.04093180701209144,
MemoryUsageInBytes: 94748672,
MemoryWorkingSetInBytes: 94444748,
CPUInterval: 0,
},
"/kube-proxy": {
Name: "",
Timestamp: {sec: 0, nsec: 0, loc: nil},
CPUUsageInCores: 0.015246457387254289,
MemoryUsageInBytes: 5681152,
MemoryWorkingSetInBytes: 5656576,
CPUInterval: 0,
},
"/system": {
Name: "",
Timestamp: {sec: 0, nsec: 0, loc: nil},
CPUUsageInCores: 0.004761631665055137,
MemoryUsageInBytes: 42254336,
MemoryWorkingSetInBytes: 25874432,
CPUInterval: 0,
},
},
"e2e-gce-minion-yb6a": {
"/system": {
Name: "",
Timestamp: {sec: 0, nsec: 0, loc: nil},
CPUUsageInCores: 0.00338632073811133,
MemoryUsageInBytes: 43447910,
MemoryWorkingSetInBytes: 26343014,
CPUInterval: 0,
},
"/": {
Name: "",
Timestamp: {sec: 0, nsec: 0, loc: nil},
CPUUsageInCores: 0.3377336907593952,
MemoryUsageInBytes: 2699154227,
MemoryWorkingSetInBytes: 936918220,
CPUInterval: 0,
},
"/docker-daemon": {
Name: "",
Timestamp: {sec: 0, nsec: 0, loc: nil},
CPUUsageInCores: 0.06698396036521867,
MemoryUsageInBytes: 1778617548,
MemoryWorkingSetInBytes: 339885260,
CPUInterval: 0,
},
"/kubelet": {
Name: "",
Timestamp: {sec: 0, nsec: 0, loc: nil},
CPUUsageInCores: 0.0644396340620269,
MemoryUsageInBytes: 99394355,
MemoryWorkingSetInBytes: 98837299,
CPUInterval: 0,
},
"/kube-proxy": {
Name: "",
Timestamp: {sec: 0, nsec: 0, loc: nil},
CPUUsageInCores: 0.010085711242831061,
MemoryUsageInBytes: 4357324,
MemoryWorkingSetInBytes: 4330291,
CPUInterval: 0,
},
},
"e2e-gce-minion-itlb": {
"/": {
Name: "",
Timestamp: {sec: 0, nsec: 0, loc: nil},
CPUUsageInCores: 0.45461458647632885,
MemoryUsageInBytes: 4603497676,
MemoryWorkingSetInBytes: 1894986547,
CPUInterval: 0,
},
"/docker-daemon": {
Name: "",
Timestamp: {sec: 0, nsec: 0, loc: nil},
CPUUsageInCores: 0.059021967282394824,
MemoryUsageInBytes: 2742996172,
MemoryWorkingSetInBytes: 488832204,
CPUInterval: 0,
},
"/kubelet": {
Name: "",
Timestamp: {sec: 0, nsec: 0, loc: nil},
CPUUsageInCores: 0.06799305314744905,
MemoryUsageInBytes: 104071168,
MemoryWorkingSetInBytes: 103599308,
CPUInterval: 0,
},
"/kube-proxy": {
Name: "",
Timestamp: {sec: 0, nsec: 0, loc: nil},
CPUUsageInCores: 0.012785393033208461,
MemoryUsageInBytes: 5046272,
MemoryWorkingSetInBytes: 5021696,
CPUInterval: 0,
},
"/system": {
Name: "",
Timestamp: {sec: 0, nsec: 0, loc: nil},
CPUUsageInCores: 0.004112326988701547,
MemoryUsageInBytes: 67950182,
MemoryWorkingSetInBytes: 51807846,
CPUInterval: 0,
},
},
}
to be empty

@ixdy @quinton-hoole @dchen1107