Code change:
At least twice now, we've created a namespace and seen that the corresponding service account default secret token is never created. (The service account itself is, but not the default secret.) Namespace creation + apiserver logs logstash-ingester: namespace creation triggered service account creation but no secret logstash-broker: namespace creation triggered service account creation and secret creation Resulting service accounts Resulting secrets Controller-manager logs for broken case Controller-manager logs for working case Version Info:
Dupe of , fixed in
Actually, not a dupe. This may be a consequence of other refactors to controller logic - possibly we are missing / dropping events out of the queue. I would expect it to get fixed on the sync interval though.
We are also seeing this happening from time to time. In our case, it wasn't fixed in the next sync interval but it took actually a little bit more of 24 hours.
can we resolve this and remove it from the 1.6 milestone?
fixed in in 1.4
We just hit this exact issue running 1.5.2. Maybe the fix didn't get the root cause or there was a regression since 1.4? This should be reopened. I waited about an hour to see if the sync interval would resolve this but it did not. I fixed by deleting and recreating the namespace. I can share logs, but they are identical to what the OP noted. Version info:
Working with this seems like it's not fixed in v1.5.2 -- what should we do here? We don't have permission to re-open the ticket, but filing the same ticket with the same logs (just fresher dates) seems passive-aggressive. :-) What's your preference?
no problem, just getting back to this. can reopen this issue.
can you recreate with the controller manager running at and include controller manager log lines from ?
Fresh repo, offending namespace "foo96", with "--v=5" since we already have that logging level in place on my dev stack. This took 96 repros of a "create namespace / monitor for token creation / delete namespace" loop. Our script's commands, starting just after 16:36:43: Controller manager log lines from The first reference to the offending namespace (foo96) and the one other entry afterwards: All controller manager log lines citing the offending namespace: For comparison, the previous successfully-created namespace: Status as of 20 minutes later:
Also, here's the apiserver logs + controller manager logs, for more action/reaction context: Namespace: foo96 (problem) Namespace: foo95 (success)
that's helpful, thanks. it looks like while the token controller is processing the creation event for the service account, it does a live lookup against etcd and fails to find the service account. what version of etcd are you running with? are you running HA etcd? do you have quorum read enabled?
We run etcd version 2.3.7 (previously saw this on 2.2.x as well). We run etcd HA, with a 3 node cluster. We do not run the apiserver with the --etcd-quorum-read flag enabled.
(Note: My specific logs above from this morning are on etcd 2.2.5, but yeah we've seen repro on 2.3.7 as well.)
yeah, the non-quorum HA etcd config is the issue. I can look into updating the token controller to tolerate that, but I'm positive there are similar issues in many controllers that assume live lookups are reliable
Ahh excellent, glad to have an explanation! So -- now that we know what to google for, we found -- where you and clayton addressed what appears to be this same problem for service accounts, and specifically noted that you wanted to avoid recommending quorum reads. Why is the answer different here? Is it because the downstream impact of non-quorum reads is too diverse to reliably retry every relevant action? (Totally fine if that's the answer, just trying to make sure we understand the reason for what appears to be an entirely opposite conclusion to the same type of problem.)
I still think controllers should be built to tolerate weak reads, and I'm willing to fix the token controller bug here, but three factors make me think this position is a losing battle: quorum read got a lot cheaper with etcd3 there are strong opinions that etcd quorum read should be on () which means that very few people are writing controllers designed to tolerate weak reads all the CI tests I'm aware of run with a single etcd or with quorum reads on
Perfect, works for us. As for fixing this particular token controller bug, I no longer have skin in that game -- your call entirely. :-)

Related discussion:
text: // This prevents the service account update (below) triggering another token creation, if the referenced token couldn't be found in the store e.secrets.Add(createdToken) <del> liveServiceAccount.Secrets = append(liveServiceAccount.Secrets, v1.ObjectReference{Name: secret.Name}) </del> <ins> // Try to add a reference to the newly created token to the service account addedReference := false err = clientretry.RetryOnConflict(clientretry.DefaultRetry, func() error { // refresh liveServiceAccount on every retry defer func() { liveServiceAccount = nil }() // fetch the live service account if needed, and verify the UID matches and that we still need a token if liveServiceAccount == nil { liveServiceAccount, err = serviceAccounts.Get(serviceAccount.Name, metav1.GetOptions{}) if err != nil { return err } if liveServiceAccount.UID != serviceAccount.UID { // If we don't have the same service account, stop trying to add a reference to the token made for the old service account. return nil } if hasToken, err := e.hasReferencedToken(liveServiceAccount); err != nil { // Don't retry cache lookup errors return nil } else if hasToken { // A service account token already exists, and is referenced, short-circuit return nil } } // Try to add a reference to the token liveServiceAccount.Secrets = append(liveServiceAccount.Secrets, v1.ObjectReference{Name: secret.Name}) if _, err := serviceAccounts.Update(liveServiceAccount); err != nil { return err } </ins> <del> if _, err = serviceAccounts.Update(liveServiceAccount); err != nil { </del> <ins> addedReference = true return nil }) if !addedReference { </ins> // we weren't able to use the token, try to clean it up. glog.V(2).Infof("deleting secret %s/%s because reference couldn't be added (%v)", secret.Namespace, secret.Name, err) deleteOpts := &metav1.DeleteOptions{Preconditions: &metav1.Preconditions{UID: &createdToken.UID}} if deleteErr := e.client.Core().Secrets(createdToken.Namespace).Delete(createdToken.Name, deleteOpts); deleteErr != nil { glog.Error(deleteErr) // if we fail, just log it } <ins> } if err != nil { </ins> if apierrors.IsConflict(err) || apierrors.IsNotFound(err) { // if we got a Conflict error, the service account was updated by someone else, and we'll get an update notification later // if we got a NotFound error, the service account no longer exists, and we don't need to create a token for it