Change:
<!-- Thanks for filing an issue! Before hitting the button, please answer these questions.--Is this a request for help? (If yes, you should use our troubleshooting guide and community support channels, see ): What keywords did you search in Kubernetes issues before filing this one? (If you have found any duplicates, you should instead reply there.): TPR. GC. OwnerReference. REST mapping. Is this a BUG REPORT or FEATURE REQUEST? (choose one): BUG REPORT. <!-- If this is a BUG REPORT, please: Fill in as much of the template below as you can. If you leave out information, we can't help you as well. If this is a FEATURE REQUEST, please: Describe in detail the feature/behavior/change you'd like to see. In both cases, be ready for followup questions, and please respond in a timely manner. If we can't reproduce a bug or think a feature already exists, we might close your issue. If we're wrong, PLEASE feel free to reopen it and explain why. --Kubernetes version (use ): master. v1.5.1 Environment: Cloud provider or hardware configuration: Repro in "hack/local-up-". But I have seen it in many different env. OS (e.g. from /etc/os-release): Kernel (e.g. ): Install tools: Others: What happened: The controller manager is unresponsive and overloaded with error message: This error just keeps bloating up without any backoff. This seems that REST mapping didn't recognize the kind. What you expected to happen: How to reproduce it (as minimally and precisely as possible): Generic way: Create a TPR. Create a TPR object "A" Create pods that has OwnerReference pointing to "A". Easy way: Use kubectl create -f kubectl create -f example/example-etcd- Anything else do we need to know:
well, the garbage collector still uses a static RESTMapper(), and in this case, TPRs aren't recognized at all.
Not recognizing is fine and expected. But controller manager keeps on printing out error message without backing off. Can you confirm this is unintended? Or we are missing anything here?
Right. I should have mentioned it: the controller manager is unresponsive and overloaded with error message. Updated in title and top comment.
there are two things that need fixing here: garbage collector should use a dynamic restmapper that picks up types the server knows about, not just compiled-in types a. this means, at minimum, discovering types at start-up b. it also means refreshing at some interval cases where an ownerRef references a type that is not found by the garbage collector's restmapper, it should not panic or hotloop. I'm not really sure what the behavior should be, actually should it treat owners it can't look up as if they were deleted, or ignore ownerRefs for unknown kinds?
If a user runs kubernetes and sees messages like "ignore XXX ownerRef due to unknown kind", he can't really do anything with it. It would be great to recognize the kind, even if there is a delay.
ownerref could have bad data, or it could reference a kind that is no longer part of the cluster. need to define what happens in those cases. when it is referencing a dynamic kind (either a kind contributed by an add-on server or by a thirdpartyresource), yes, it should re-discover on some interval to recognize the kind
I have the same issue when launching an etcd cluster with the coreos etcd-operator. If causes the api server to run out of disk space as it logs infinitely. I have to keep restart the controller container to reclaim disk space. From a security point of view this means a tenant of a cluster can bring the whole cluster down.
Why the target fix is in 1.6 not a patch release for 1.5?
I would not anticipate the garbage collector actually enabling support for TPRs in 1.5, but the bug that causes it to become unresponsive when it sees an ownerRef of an unknown type should be fixed
Changed the milestone to 1.5
I agree. I do not expect the gc to work with TPR either. But it should not "kill" controller manager. And it is the issue we should fix in 1.5. We can make GC work for TPR in 1.6.
For me this is currently a DOS attack
Repro instructions on master in GCE:
so you don't even need to have something with an owner ref pointing at the TPR to trigger this?
I would guess an ownerRef pointing to any arbitrary unknown kind would trigger the same behavior
the etcd operator creates etcd pods with: I suspect is correct.
Ah, the operator triggers it. OK, I see.
etcd operator does not rely on the k8s GC features (it has internal GC). But it does write owner ref. when k8s gc works for tpr, we will remove our temp internal GC. but now the owner ref causes controller manager to spam.
shall we reopen? i do not think this is fixed.
Yeah this just happened to me. Does not seem fixed.
what version are you running? This isn't in a release version yet.
"masks" the problem, not fixes it. We should close this issue when the fix lands.
I just installed with kubeadm two days ago. I believe is 1.5.2. I then tried using etcd-operator and it killed my master with out of disk space writing 300MB/s of log from GC
a fix will be in the 1.5.3 release. It was just merged and hasn't had time to make it into a release. Sure, not arguing that this issue should be closed. I do think is probably good enough for the 1.4, 1.5 branches.
I'll do it after the refactor of GC, , lands. makes gc use the rate limited workqueue, so it will prevents the hotloop. [edit] GC treats unresolveable ownerRefs as though they perpetually exist. That is, an object with an unresolvable ownerRef will not be deleted by GC. It is meant to be a safety net. For example, during a cluster upgrades, if the APIVersion of replication controllers bumps from "v1" to "v2", and assuming all the pods' ownerRefs still have "v1", if GC treats those refs as if the owners were deleted, then all the pods will be accidentally deleted.
Does GC only treat the preferred version as a legitimate ownerRef, or all known versions, or was that just an example?
I guess there is a third option as well, treat unresolveable ownerRefs as though they perpetually exist (which is different from ignoring them).
All known versions. What I mean in the example is that assuming v1/replicatoncontroller is not supported by the API server anymore.
Actually this is the current case. GC doesn't delete an object with unresolvable ownerRef. I edited my previous comment.
40Gb per day of logs to this mess for us... So we would love a fix too.
is close to get in. It will fix the hot loops. We still need a dynamic restmapper. I'll start a fix after lands.
FWIW, here's the patch I wrote to get back to a more reasonable behavior: (then, rebuild your controller-manager or hyperkube)
Thanks changed gc to use the ratelimited queue, so the hotloop should have been fixed on the master branch. [edit] That said, we still need to make the restmapper dynamic, i'm working on a fix.
My patch was done on the branch, should anyone prefer to avoid upgrading to alpha.0 :)
BTW, is there a backport to 1.5 planned? is flagged so, should a smaller patch be needed, feel free to use this one of course.
Update: I tried to reuse to dynamically translate TPR, but it didn't seem to work because of I'll find a workaround.
what version are you on? said was landed in v1.5.3, so the problems should be masked. Is that true?
I had to overwrite hyperkube in the docker image
Ahhh it has this commit which says: Adjust global log limit to 1ms So probably not a difference a human can detect.
I see. 1 ms is still too spamming.
At least, it make the log unusable without filtering ;) I have a global feeling that Kubernetes log system could be improved but I still need to have a better of that. The global problem to solve is probably the repetition of log, something that's solved by events (or at the way kubectl shows them); so maybe I should move my practice more to events and, when I don't have the answer in the events, file an issue/PR.
the backoff removed the log flooding, right? if so, I think this can be moved to 1.7
c.f.
Thanks You are right, this is not a 1.6 blocker. makes sure the log message is only printed once per un-resolvable ownerReference.
I think the original issue of "spamming error messages" was solved. I created to track the issue of "GC should support non-core API". Closing this issue.

Discussion:
import ( "fmt" "runtime" <ins> "sync" "time" </ins> "github.com/golang/glog" )

// ErrorHandlers is a list of functions which will be invoked when an unreturnable // error occurs. <del> var ErrorHandlers = []func(error){logError} </del> <ins> // TODO(lavalamp): for testability, this and the below HandleError function // should be packaged up into a testable and reusable object. var ErrorHandlers = []func(error){ logError, (&rudimentaryErrorBackoff{ lastErrorTime: time.Now(), // 1ms was the number folks were able to stomach as a global rate limit. // If you need to log errors more than 1000 times a second you // should probably consider fixing your code instead. :) minPeriod: time.Millisecond, }).OnError, } </ins> // HandlerError is a method to invoke when a non-user facing piece of code cannot // return an error and needs to indicate it has been ignored. Invoking this method

glog.ErrorDepth(2, err) } <ins> type rudimentaryErrorBackoff struct { minPeriod time.Duration // immutable // TODO(lavalamp): use the clock for testability. Need to move that // package for that to be accessible here. lastErrorTimeLock sync.Mutex lastErrorTime     time.Time } // OnError will block if it is called more often than the embedded period time. // This will prevent overly tight hot error loops. func (r *rudimentaryErrorBackoff) OnError(error) { r.lastErrorTimeLock.Lock() defer r.lastErrorTimeLock.Unlock() d := time.Since(r.lastErrorTime) if d < r.minPeriod { time.Sleep(r.minPeriod - d) } r.lastErrorTime = time.Now() } </ins> // GetCaller returns the caller of the function that calls it. func GetCaller() string { var pc [1]uintptr