Change:
I have some reports of a Node reporting a a resource capacity for cpu as a negative value. Any idea why cpu could report as a negative value? Is this an expected normal outcome? In its current state, it prevents any pods from being scheduled to that Node. Tips on how to debug further are appreciated. - any ideas?
Can you check machineInfo on Cadvisor or kubelet on the node?
a.k.a:
It is a bug in cAdvisor side. I am assigning it to you :-)
here is contents of /proc/cpuinfo Pertinent pieces of a log:
Machine is AWS instance if that matters....
We build the topology from system/cpu info. I think we can return partial topology information and ignore caches if its not present. This isn't a recent change though. Is it happening on just a single machine? I'll file a cAdvisor issue.
Can you link the referenced issue once created?
Re-assigning to since he's working on google/cadvisor
Could be this was caused by getting a Node on EC2 with old hypervisor, and I am the first victim...
That sounds likely. Most of the cache information is there in /proc/cpuinfo. So looks like the sysfs bits are disabled.
Can you try this cadvisor image on your machine and check /api/v1.3/machine output: sudo docker run  --volume=/:/rootfs:ro  --volume=/var/run:/var/run:rw  --volume=/sys:/sys:ro  --volume=/var/lib/docker/:/var/lib/docker:ro  --publish=8080:8080  --detach=true  --name=cadvisor  jnagal/cadvisor:canary
hmm
ah, selinux :( You'll need to run it with --privileged, I think. who's been dealing with it.
No luck with --privileged either on this box...
This has to do with SELinux, I think we had to disable it for this to work (some of our mounts are being denied). We're working on a better solution... This is not a problem in Kubernetes since we don't run Kubelet/cAdvisor in a Docker container.
ok, unfortunately, i have managed to kill docker on the box in question, and i cannot get the daemon to restart right now so I am not sure when that box will be available to debug against again.
I can provide an environment with a similar issue if required.
The fix is checked in to head. Can you build the binary and try?

Discussion:
}, { "ImportPath": "github.com/google/cadvisor/api", <del> "Comment": "0.13.0-22-ge3aa15a", "Rev": "e3aa15a3bbee5367532ac56aa55be41441c55356" </del> <ins> "Comment": "0.13.0-32-g832c6e9", "Rev": "832c6e94c325ecdf05da884880c503be45594818" </ins> }, { "ImportPath": "github.com/google/cadvisor/collector", <del> "Comment": "0.13.0-22-ge3aa15a", "Rev": "e3aa15a3bbee5367532ac56aa55be41441c55356" </del> <ins> "Comment": "0.13.0-32-g832c6e9", "Rev": "832c6e94c325ecdf05da884880c503be45594818" </ins> }, { "ImportPath": "github.com/google/cadvisor/container", <del> "Comment": "0.13.0-22-ge3aa15a", "Rev": "e3aa15a3bbee5367532ac56aa55be41441c55356" </del> <ins> "Comment": "0.13.0-32-g832c6e9", "Rev": "832c6e94c325ecdf05da884880c503be45594818" </ins> }, { "ImportPath": "github.com/google/cadvisor/events", <del> "Comment": "0.13.0-22-ge3aa15a", "Rev": "e3aa15a3bbee5367532ac56aa55be41441c55356" </del> <ins> "Comment": "0.13.0-32-g832c6e9", "Rev": "832c6e94c325ecdf05da884880c503be45594818" </ins> }, { "ImportPath": "github.com/google/cadvisor/fs", <del> "Comment": "0.13.0-22-ge3aa15a", "Rev": "e3aa15a3bbee5367532ac56aa55be41441c55356" </del> <ins> "Comment": "0.13.0-32-g832c6e9", "Rev": "832c6e94c325ecdf05da884880c503be45594818" </ins> }, { "ImportPath": "github.com/google/cadvisor/healthz", <del> "Comment": "0.13.0-22-ge3aa15a", "Rev": "e3aa15a3bbee5367532ac56aa55be41441c55356" </del> <ins> "Comment": "0.13.0-32-g832c6e9", "Rev": "832c6e94c325ecdf05da884880c503be45594818" </ins> }, { "ImportPath": "github.com/google/cadvisor/http", <del> "Comment": "0.13.0-22-ge3aa15a", "Rev": "e3aa15a3bbee5367532ac56aa55be41441c55356" </del> <ins> "Comment": "0.13.0-32-g832c6e9", "Rev": "832c6e94c325ecdf05da884880c503be45594818" </ins> }, { "ImportPath": "github.com/google/cadvisor/info/v1", <del> "Comment": "0.13.0-22-ge3aa15a", "Rev": "e3aa15a3bbee5367532ac56aa55be41441c55356" </del> <ins> "Comment": "0.13.0-32-g832c6e9", "Rev": "832c6e94c325ecdf05da884880c503be45594818" </ins> }, { "ImportPath": "github.com/google/cadvisor/info/v2", <del> "Comment": "0.13.0-22-ge3aa15a", "Rev": "e3aa15a3bbee5367532ac56aa55be41441c55356" </del> <ins> "Comment": "0.13.0-32-g832c6e9", "Rev": "832c6e94c325ecdf05da884880c503be45594818" </ins> }, { "ImportPath": "github.com/google/cadvisor/manager", <del> "Comment": "0.13.0-22-ge3aa15a", "Rev": "e3aa15a3bbee5367532ac56aa55be41441c55356" </del> <ins> "Comment": "0.13.0-32-g832c6e9", "Rev": "832c6e94c325ecdf05da884880c503be45594818" </ins> }, { "ImportPath": "github.com/google/cadvisor/metrics", <del> "Comment": "0.13.0-22-ge3aa15a", "Rev": "e3aa15a3bbee5367532ac56aa55be41441c55356" </del> <ins> "Comment": "0.13.0-32-g832c6e9", "Rev": "832c6e94c325ecdf05da884880c503be45594818" </ins> }, { "ImportPath": "github.com/google/cadvisor/pages", <del> "Comment": "0.13.0-22-ge3aa15a", "Rev": "e3aa15a3bbee5367532ac56aa55be41441c55356" </del> <ins> "Comment": "0.13.0-32-g832c6e9", "Rev": "832c6e94c325ecdf05da884880c503be45594818" </ins> }, { "ImportPath": "github.com/google/cadvisor/storage", <del> "Comment": "0.13.0-22-ge3aa15a", "Rev": "e3aa15a3bbee5367532ac56aa55be41441c55356" </del> <ins> "Comment": "0.13.0-32-g832c6e9", "Rev": "832c6e94c325ecdf05da884880c503be45594818" </ins> }, { "ImportPath": "github.com/google/cadvisor/summary", <del> "Comment": "0.13.0-22-ge3aa15a", "Rev": "e3aa15a3bbee5367532ac56aa55be41441c55356" </del> <ins> "Comment": "0.13.0-32-g832c6e9", "Rev": "832c6e94c325ecdf05da884880c503be45594818" </ins> }, { "ImportPath": "github.com/google/cadvisor/utils", <del> "Comment": "0.13.0-22-ge3aa15a", "Rev": "e3aa15a3bbee5367532ac56aa55be41441c55356" </del> <ins> "Comment": "0.13.0-32-g832c6e9", "Rev": "832c6e94c325ecdf05da884880c503be45594818" </ins> }, { "ImportPath": "github.com/google/cadvisor/validate", <del> "Comment": "0.13.0-22-ge3aa15a", "Rev": "e3aa15a3bbee5367532ac56aa55be41441c55356" </del> <ins> "Comment": "0.13.0-32-g832c6e9", "Rev": "832c6e94c325ecdf05da884880c503be45594818" </ins> }, { "ImportPath": "github.com/google/cadvisor/version", <del> "Comment": "0.13.0-22-ge3aa15a", "Rev": "e3aa15a3bbee5367532ac56aa55be41441c55356" }, { "ImportPath": "github.com/google/go-github/github", "Rev": "930e6fdb8dc2b11458fdeb55b3cd68e5370a1a28" }, { "ImportPath": "github.com/google/go-querystring/query", "Rev": "547ef5ac979778feb2f760cdb5f4eae1a2207b86" }, </del> <ins> "Comment": "0.13.0-32-g832c6e9", "Rev": "832c6e94c325ecdf05da884880c503be45594818" }, { "ImportPath": "github.com/google/go-github/github", "Rev": "930e6fdb8dc2b11458fdeb55b3cd68e5370a1a28" }, { "ImportPath": "github.com/google/go-querystring/query", "Rev": "547ef5ac979778feb2f760cdb5f4eae1a2207b86" }, </ins> { "ImportPath": "github.com/google/gofuzz", "Rev": "bbcb9da2d746f8bdbd6a936686a0a6067ada0ec5"

now := time.Now() lowestTime := now for _, cgroupPath := range self.cgroupPaths { <del> // The modified time of the cgroup directory is when the container was created. </del> <ins> // The modified time of the cgroup directory changes whenever a subcontainer is created. // eg. /docker will have creation time matching the creation of latest docker container. // Use clone_children as a workaround as it isn't usually modified. It is only likely changed // immediately after creating a container. cgroupPath = path.Join(cgroupPath, "cgroup.clone_children") </ins> fi, err := os.Stat(cgroupPath) if err == nil && fi.ModTime().Before(lowestTime) { lowestTime = fi.ModTime()

func TestGetEventsForTimePeriod(t *testing.T) { myEventHolder, myRequest, fakeEvent, fakeEvent2 := initializeScenario(t) <del> myRequest.StartTime = createOldTime(t).Add(-1 * time.Second * 10) myRequest.EndTime = createOldTime(t).Add(time.Second * 10) </del> <ins> myRequest.StartTime = time.Now().Add(-1 * time.Second * 10) myRequest.EndTime = time.Now().Add(time.Second * 10) </ins> myRequest.EventType[info.EventOom] = true myEventHolder.AddEvent(fakeEvent)

assert.Nil(t, err) checkNumberOfEvents(t, 1, len(receivedEvents)) <del> ensureProperEventReturned(t, fakeEvent, receivedEvents[0]) </del> <ins> ensureProperEventReturned(t, fakeEvent2, receivedEvents[0]) </ins> } func TestGetEventsForNoTypeRequested(t *testing.T) {

for idx, node := range nodes { caches, err := sysinfo.GetCacheInfo(sysFs, node.Cores[0].Threads[0]) if err != nil { <del> return nil, -1, fmt.Errorf("failed to get cache information for node %d: %v", node.Id, err) </del> <ins> glog.Errorf("failed to get cache information for node %d: %v", node.Id, err) continue </ins> } numThreadsPerCore := len(node.Cores[0].Threads) numThreadsPerNode := len(node.Cores) * numThreadsPerCore

"time" ) <ins> type timedStoreDataSlice []timedStoreData func (t timedStoreDataSlice) Less(i, j int) bool { return t[i].timestamp.Before(t[j].timestamp) } func (t timedStoreDataSlice) Len() int { return len(t) } func (t timedStoreDataSlice) Swap(i, j int) { t[i], t[j] = t[j], t[i] } </ins> // A time-based buffer for ContainerStats. // Holds information for a specific time period and/or a max number of items. type TimedStore struct { <del> buffer   []timedStoreData </del> <ins> buffer   timedStoreDataSlice </ins> age      time.Duration maxItems int }

// A maxItems value of -1 means no limit. func NewTimedStore(age time.Duration, maxItems int) *TimedStore { return &TimedStore{ <del> buffer:   make([]timedStoreData, 0), </del> <ins> buffer:   make(timedStoreDataSlice, 0), </ins> age:      age, maxItems: maxItems, }

// Adds an element to the start of the buffer (removing one from the end if necessary). func (self *TimedStore) Add(timestamp time.Time, item interface{}) { <del> // Remove any elements before the eviction time. evictTime := timestamp.Add(-self.age) index := sort.Search(len(self.buffer), func(index int) bool { return self.buffer[index].timestamp.After(evictTime) }) if index < len(self.buffer) { self.buffer = self.buffer[index:] } </del> // Remove any elements if over our max size. if self.maxItems >= 0 && (len(self.buffer)+1) > self.maxItems { startIndex := len(self.buffer) + 1 - self.maxItems self.buffer = self.buffer[startIndex:] } <ins> // Add the new element first and sort. We can then remove an expired element, if required. </ins> copied := item self.buffer = append(self.buffer, timedStoreData{ timestamp: timestamp, data:      copied, }) <ins> sort.Sort(self.buffer) // Remove any elements before eviction time. // TODO(rjnagal): This is assuming that the added entry has timestamp close to now. evictTime := timestamp.Add(-self.age) index := sort.Search(len(self.buffer), func(index int) bool { return self.buffer[index].timestamp.After(evictTime) }) if index < len(self.buffer) { self.buffer = self.buffer[index:] } </ins> } // Returns up to maxResult elements in the specified time period (inclusive).

maxResults = -1 } <del> // NOTE: Since we store the elments in descending timestamp order "start" will // be a higher index than "end". </del> var startIndex int if start.IsZero() { // None specified, start at the beginning.