Change:
Env: Kubernetes 1.3.3 on AWS I ran into the same problem described into but now it's happening on every deploy. I would have reopened the issue, but I was not the author so I could not. The bug presents itself because I have a build that does just after a and it gets stuck forever because the rollout is never reported as finished (it does finish tho).
could you provide more detail info? like what's your resource file and how to reproduce it.
yes, of course: We have an internal tool that downloads some manifests from S3 in a temporary folder and then calls . It's harder to narrow it down because that's all there is to it. It caused the issue above for 3/4 builds in a row but in the last build it worked. The three deployment files are fairly standard deployment templates, this is an example of one of them: The is filled by jenkins at build time before the manifest is uploaded on s3. I am not sure about how to narrow this down further, any suggestions?
can somebody triage this?
We just started seeing this as well. kube 1.3.6 on AWS. The cluster was running fine for two weeks, then the problem started on the last CI build yesterday, and shows no signs of fixing itself. Our deploy process is a series of commands called from a bash script during CI.
should fix the hotloop by using a rate limiter in the deployment controller. Seems like it was included in 1.3.7. Also the "already exists" shouldn't be treated as an error but instead we should return the replica set from the cache at this point (since it already exists). I was handling this error in a recent PR of mine, will update this issue back with the #
Are you sure the fix is in 1.3.7? I just hit this problem in a build of 1.3.7, and it kills the master node as it pegs a cpu at 100% in this error loop. edit: it doesn't seem to be on the release-1.3 branch, but is in release-1.4
Hm, git fooled me. The fix is included in 1.4.
From the code, it looks like deployment api generates an adler-32 checksum of the pod template, then uses that to create a new replicaset . If that hash already exists, it barfs with the controller-manager loop. So it looks to me, from a quick read, as simply a hash collision handled poorly. This seems to mirror what I'm seeing, since it only happens on deployments with a history 100 deploys. I'm guessing there isn't enough data change in the pod spec to ensure a decent amount of uniqueness with adler-32 - in my case just the version changes. It should probably use a better hash function (that doesn't duplicate on such a small number of deployments), and/or handle the collision more gracefully.
I don't think you would get a hash collision even with 100 deploys (not sure where Adler starts to fail though). Even the slightest change in the pod template changes the hash. Can you provide the pod template of the existing replica set that blocks the deployment controller and also the latest state of your deployment? I think this issue is simply us not handling AlreadyExists errors when we try to create new replica sets (the replica set cache may get stale).
sure... I admit I may be wrong - that was my just my super quick analysis. :) Although, adler is pretty terrible as a hash function, and I was a little surprised to see it used - which is why I leapt to that conclusion. I need to try reproducing to see if it's actually the case though - which I'll try now. Here's the replicaset that it complains already exists: And the current failing deployment (although, I believe it started failing at generation 204, version 0.612.0).
Hash collisions are definitely a problem. I wrote up a test and can get a collision within a couple hundred versions (only version changes). Here's I'm guessing we can't improve the hash function at this point, to maintain backwards compatibility? We should at least handle the collision somehow - what do you suggest?
Further, in the example I gave above, the hash output only has about 200 unique values, after which all newer versions collide with previous hashed values.
I suppose the immediate workaround for users is to limit deployment revision history to small values ().
thanks for the test! I run it as well and I see the same results. can you have a look? An immediate workaround is to use revisionHistoryLimit, yes. In reality most users won't care about the history of 200 replica sets but it is concerning that Adler breaks that fast.
cc
Adler was only used because we were originally overly concerned about the hash performance. We need to be using a reasonably (but not excessively) fast hash where collision chance is 1 in 1e5, and preferably 1e10 or more.
fnv would be an easy change
Tangentially... having ~200 RS resources hanging around per-deployment seems like a lot and I imagine could result in triggering unknown-unknowns. IMHO the revision history limit should be bounded by default before moving out of beta. I can image things like automated hourly builds creating lots of history.
Generally bounds would be enforced by limit ranges, which already bound number of replica sets On Oct 18, 2016, at 11:31 PM, Phillip Wittrock wrote: Tangentially... having ~200 RS resources hanging around per-deployment seems like a lot and I imagine could result in triggering unknown-unknowns. IMHO the revision history limit should be bounded by default before moving out of beta. I can image things like automated hourly builds creating lots of history. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub , or mute the thread .
Thanks
cc
sent a PR to change the defaults (), but we'd need a new apiversion to do that.
We just ran into this issue with a much smaller history (only 10 RS resources). We're using the revisionHistoryLimit workaround but it seems like the real fix would be using a different hashing algorithm. Any thoughts on when a fix may come out?
hash collision is not the only case where you can stumble upon this issue. Are you sure this was a hash coliision? Did you check that the deployment pod template is different from the rs pod template that you get in the error?
yes, here are two templates that we are seeing hash to the same value: vs In this case there's only a small change for the version bump, and the template is rather short.
Can we even change this in a backwards compatible manner on the cluster safely? On Nov 2, 2016, at 5:52 PM, Lucas Chi wrote: yes, here are two templates that we are seeing hash to the same value: apiVersion: extensions/v1beta1 kind: Deployment metadata: name: exworker spec: replicas: 2 template: metadata: labels: run: exworker language: elixir spec: containers: name: exworker image: teacherspayteachers/exworker:release-2.1.2 env: name: ENVIRONMENT value: "prod" name: COREAPIKEY valueFrom: secretKeyRef: name: core-api-secret-key key: secretkey imagePullSecrets: name: "tptreadonly-docker" vs apiVersion: extensions/v1beta1 kind: Deployment metadata: name: exworker spec: replicas: 2 template: metadata: labels: run: exworker language: elixir spec: containers: name: exworker image: teacherspayteachers/exworker:release-1.3.1 env: name: ENVIRONMENT value: "prod" name: COREAPIKEY valueFrom: secretKeyRef: name: core-api-secret-key key: secretkey imagePullSecrets: name: "tptreadonly-docker" In this case there's only a small change for the version bump, and the template is rather short. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub , or mute the thread .
New hash algo? Unlikely.
So we don't want to blow out an existing cluster immediately on upgrade, so we need a way to stage this so an admin can slowly roll this out under their control. One possible algo: a flag to the controller manager config that decides hash behavior "legacy", "migrate-to-X", "X" the deployment controller calculates hashes, calculate old and new hash there are no existing replicas for a deployment, use the new hash the "old" hash matches and a new hash is available, put the deployment in a queue N at a time from the queue and have them use the new hash time a deployment is successfully moved to a new hash, draw a replacement from the queue all deployments are migrated, admin can change the hash behavior We could potentially get away without config, but I think admins need to opt in to this change in 1.N and be forced to do it in 1.(N+1) , Michail Kargakis wrote:
What about old replica sets? What happens if a new hash collides with an old hash (latest template hashed == old replica set computed with old hashing algo)?
Old replica sets is covered by 4. If you have a hash collision its game over anyway, so not worried about that. , Michail Kargakis wrote:
Bumping to P1 because this makes deployments unpredictable - we need to assess the change required for 1.5 next week and whether we can come up with a backwards compatible, safe, and non-disruptive change.
This is quite a big change for 1.5 and since the problem was always there I would move a fix for this in 1.6.
Tested with the unit test provided by above and it didn't break for 10^3 up to 10^5 (10^6 was timing out the test and I didn't bother to test more).
Results from benchmarking Fnv is a bit slower but much more stable.
so my understanding is that the migration of old replica sets will happen during the normal operation of the cluster, is that correct? We can migrate all replica sets with zero replicas in the background, and migrate running sets once a new rollout happens. We may not even need a separate queue. I am still not sure if we can notify admins somehow when all existing deployments have migrated.
We could expose a metric that is "how many old replication controllers we still have"
For 1.6, we moved Deployments into the apps api group[1] so we can change the defaults (revisionHistoryLimit is now set to 3 by default for newly created deployments) and we also made the cleanup policy run independent of a rollout (so a rollout that gets stuck won't block deletiion of older replica sets)[2]. I've also opened an update to the Deployment proposal for moving away from hashing: Moving the milestone to 1.7 [1] [2]
Changing revision history doesn't seem to alleviate hash collisions, right? Looking at - are we using a poor hash for this?
Actually, it does. If all you have is 3 old replica sets, it's less likely that the controller will break you because of a hash collision. 200 seems to be the limit with adler. Yes. There is that changes us to fnv but there is also that moves us away from hashing. I am going to open an alternative proposal to 384 that covers the transition to fnv and we need to decide with what do we want to proceed.
Opened as an alternative to kubernetes/community
We saw this in a related configuration. Marking --record=true on the deployment [1] and setting the revision history [2] configuration seemed to alleviate the stress. Is there a status on the hash collision bug? [1] [2] Thanks, Brugz
the fix will be included in 1.7 release

Discussion:
} utilruntime.HandleError(err) <ins> glog.V(2).Infof("Dropping deployment %q out of the queue: %v", key, err) </ins> dc.queue.Forget(key) }

createdRS, err := dc.client.Extensions().ReplicaSets(namespace).Create(&newRS) switch { // We may end up hitting this due to a slow cache or a fast resync of the deployment. <del> case errors.IsAlreadyExists(err): return dc.rsLister.ReplicaSets(namespace).Get(newRS.Name) </del> <ins> // TODO: Restore once https://github.com/kubernetes/kubernetes/issues/29735 is fixed // ie. we start using a new hashing algorithm. // case errors.IsAlreadyExists(err): //	return dc.rsLister.ReplicaSets(namespace).Get(newRS.Name) </ins> case err != nil: msg := fmt.Sprintf("Failed to create new replica set %q: %v", newRS.Name, err) if deployment.Spec.ProgressDeadlineSeconds != nil {