Change:
Failing release-blocking suites: [x] https://k8s- [x] https://k8s- [x] https://k8s- all three suites are flaking a lot recently, mind triage? /sig scalability /priority failing-test /kind bug /status approved-for-milestone cc /assign
The correctness job is failing mostly due to timeout (need to adjust our schedule correspondingly) as there have been a bunch of e2es which got to the suite recently (e.g ) For the 100-node flakes, we have (and I believe that's related.. need to check). For the performance job, I believe there's a regression (seems from the last few runs like it's around pod-startup latency). Maybe sth more too. I'll try getting to them sometime during this week (having scarcity of free cycles atm).
is there any update for this issue?
I took a brief look into that. And either some test(s) are extreeeeemely slow or something is hanging somehwere. Par of logs from last run: No test finished within 8h30m
Indeed seems like a regression. I think the regression happened somewhere between runs: 105 (which was still ok) 108 (which had visibly higher startup time) We can try looking into kubemark-5000 to see if it is visible there too.
Kubemark-5000 is pretty stable. 99th percentile on this graph (maybe the regression happened even before, but I think it is somewhere between 105 and 108): !
Regarding the corretness tests - gce-large-correctness is also failing. Maybe some test that is extremely long was at that time?
Thanks a lot for looking Wrt performance job - I too feel strongly there's a regression (though I couldn't get to look properly into them). I was looking into this a while ago. And there were 2 suspicious changes I found: - This a bunch of tests around local storage (the runs after this change started to timeout) StatefulSet with pod anti-affinity should use volumes spread across nodes (this test seems to be running for 3.5 - 5hr) - https://k8s-
cc
/assign Some of the local storage tests will try to use every node in the cluster, thinking that cluster sizes are not that big. I'll add a fix to cap the max number of nodes.
Thanks - that would be great.
Going back to https://k8s- suite I took a closer look into runs up to 105 and runs 108 and after that. The biggest difference wrt to pod startup time seem to appear at the step: [the name of it is misleading - will explain below] Up to 105 run, it generally looked like this: Starting with 108 run, it looks more like: That basically means ~0.85s increase and this roughly what we observe in the end result. Now - what that "watch lag" is. It is basically a time between "Kubelet observed that the pod is running" to "when test observed pod update setting its state to running". There are a couple possibility where we could have regressed: kubelet is slowed in reporting status kubelet is qps-starved (and thus slower in reporting status) apiserver is slower (e.g. cpu-starved) and thus processing requests slower (either write, watch, or both) test is cpu-starved and thus processing incoming events slower Since we don't really observe a difference between "schedule -start" of a pod, that suggests that it most probably isn't apiserver (because processing requests and watch is on that path too), and it's most probably not slow kubelet too (because it starts the pod). So i think the most likely hypothesis is: kubelet is qps-starved (or sth that prevents it from quickly sending status update) test is cpu-starved (or sth like that) Test didn't change at all around that time. So I think it's most probably the first one. That said, I went through PRs merged between 105 and 108 runs and didn't find anything useful so far.
I think the next step is to: look into the slowest pods (there seem to be O(1s) difference between slowest too) and see whether the difference is "before"of "after" the update status request was sent
So I looked into example pods. And I'm already seeing this: So it seems pretty clear that the problem is related to "429"s.
Are those throttled API calls due to a quota on the owner account?
This isn't throttling as I thought initially. These are 429s on apiserver (the reason may be either slower apiserver because of some reason, or more requests coming to apiserver).
Oh, ok. That's not great news.
/milestone clear
/milestone v1.10
/milestone clear
You must be a member of the github team to set the milestone. <detailsIn response to : /milestone clear Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/milestone v1.9
You must be a member of the github team to set the milestone. <detailsIn response to : /milestone v1.9 Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
Seems like PR fixed the timeout issues - thanks for quick response. Our correctness jobs (both 2k and 5k) are back to green now: https://k8s- https://k8s- So about those volume tests was indeed right :)
ACK. In progress ETA: 9/3/2018 Risks: Potential impact on k8s performance
So I digged up a bit into this and from the graph of pod-startup latency for our 5k-node test, I have the feeling that the regression could also be in b/w runs 108 and 109 (see 99%ile): !
I swept quickly through the and the following change seems suspicious to me: "Allow passing request-timeout from NewRequest all the way down" That PR is enabling propagation of the client's timeout as a query parameter to the API call. And I indeed see following difference in calls across those 2 runs (from the apiserver logs): run-108: run-109: My hypothesis is that, because of that timeout of 10s to the PATCH calls, those calls are now lasting for longer time on the server-side (IIUC correctly). This means that they're now in the inflight-queue for longer duration. That, combined with the fact that those PATCH calls occur in huge amounts in such big clusters, is leading to calls unable to get enough bandwidth in inflight-queue and hence are returned with 429s. As a result, the kubelet-side delay in updating the pod-status has increased. This story also fits in well with 's observations above. I'll try to gather more evidence to verify this hypothesis.
So I checked how latencies vary over the test runs, and it indeed seems like there's a rise in the 99th percentile (see the top line) around that time. However it isn't super clear that it happened b/w runs 108 and 109 (though my belief is that's the case): !
[EDIT: My earlier comment wrongly mentioned the count for those 429s (client was npd, not kubelet)] I have more supporting evidence now: in , we had ~479k calls that got a 429: and in , we have ~757k of those:
And... Look at this: in run-108: and in run-109:
I checked the number for few other adjacent runs: for it's for it's for it's -- that PR merged -- for it's for it's for it's While it seems a bit varying, overall it looks like the no. of 429s increased by about 25%.
And for coming from kubelets that recieved 429s, here's how the numbers look like: run-104 = run-105 = run-108 = -- that PR merged -- run-109 = run-110 = run-111 = This is also varying, but it generally seems to increase.
The 99th %ile of call latencies also seem to have generally increased (as I was predicting in ): Btw - Wrt all the above metrics, 108 seems to be a worse-than-normal run and 111 seems to be a better-than-normal run.
I'll try to verify this tomorrow by manually running a large 5k cluster.
thanks for triage
So I ran density test twice against 5k cluster against ~HEAD, and the test surprisingly passed both times with 99%ile pod startup latency as and . However, the 'watch latencies', did show the increase that pointed in In the first run it was: and the second run it was: I'll now try checking what was the case earlier.
- I'm not sure I'm following this Yes - we timeout, but default timeout is bigger than 10s IIRC - so it should only help not make things worse. I think we still don't understand why we observe more 429s (the fact that this is somehow related to 429s I already mentioned in )
And regarding your numbers - I'm not convinced that that the regression was in run 109 - there might have been two regressions - one in somewhere between 105 and 108 and the other in 109.
Hmm... I don't deny the possibilities you mentioned (the above was just my hypothesis). I'm currently doing bisection (right now against 108's commit) to verify.
My feeling that the regression is before run 108 is stronger and stronger. As an example, latencies of api calls are already increased in 108 run. patch node status: 90%: 198ms (105) 447ms (108) 444ms (109) put pod status: 99%: 83ms (105) 657ms (108) 728ms (109)
I guess what I'm trying to say is: the number of 429s is a consequence and we shouldn't spend too much time on that the root cause is either slower api calls or bigger number of those We clearly seem to see slower api calls in 108. The question is whether we also see bigger number of those.
So I think why the requests are visibly slower - there are three main possibilities is significantly more requests (from a first glance it doesn't seem to be the case) something on the path of processing (e.g. additional processing) or the objects themselves are bigger else on master machine (e.g. scheduler) is consuming more cpu and thus starving apiserver more
So me and discussed offline and we both agree now that there's quite likely a regression before 108. Adding some points: Doesn't seem to be the case to me neither My feeling is it's sth more likely in kubelet as opposed to apiserver (as we're not seeing any visible change in patch/put latencies on kubemark-5000) IMO this is not the case as we're having quite some CPU/mem slack on our master. Also perf-dash doesn't suggest any considerable increase in usages of master components. That said, I digged up a bit and "luckily" it seems like we're noticing this increase in watch latencies even for 2k-node clusters: Should make bisecting a bit easier.
Unfortunately, the variation in those watch latencies seems like a one-off (it's otherwise roughly the same). Fortunately, we have latency as a reliable indicator of the regression. I ran 2 rounds of bisection yesterday and narrowed down to (~80 commits). I skimmed through them and have strong suspicion on: - adds a new field to pod-status (though I'm not sure if that'd be filled in our tests, where IIUC preemptions are not happening - but need to check) - updates etcd server version to 3.2.14
I really doubt that the is related here - NominatedNodeName is a string that contains a single node name. Even if it would be filled all the time, the change in object size should be negligible.
- As you were suggesting offline, it indeed seems like we're using a different version (3.2.16) in kubemark () which is the potential reason for not seeing this regression there :) cc
We are using 3.2.16 everywhere now.
Oops.. Sorry for the hindsight - I was looking at a wrong combination of commits.
BTW - that increase in PUT pods/status latency is visible also in load test in large real clusters.
So I digged up a bit more and seems like we started observing greater latencies around that time for write requests in general (which makes me suspect etcd change even more): ! ! ! !
Actually, I vote that at least part of the problem is here: I would be surprised if it would be the whole problem, but it may contribute to it. Will send a PR changing that in a second.
FYI - When I ran against that's before the etcd 3.2.14 change but after the pod-status API change, the put node-status latency seems totally fine (i.e 99%ile = 39ms).
So I verified that it indeed is caused by the etcd bump to 3.2.14. Here's how the put pod-status latency looks like: against that : against ~HEAD (from Mar 5) with that PR reverted (the test is still running, but about to finish soon): 63ms seems quite similar to what . We should revert the version and try to understand: Why we're seeing this increase in etcd 3.2.14? Why we didn't catch this in kubemark? cc
One hypothesis (why we didn't catch that in kubemark - though it's still a guess) is that we may have changed sth wrt to certificates there. When I'm comparing etcd log from kubemark and real cluster, only in the latter I'm seeing the following line: Looking into the PR itself I don't see any changes around that, but I also don't know why we should have see that line only in real clusters... for thoughts
ACK. In progress ETA: 9/3/2018 Risks: Issue root-caused (mostly)
Re peerTLS - it seems to be the case also before (with 3.1.11) so I think that's a red herring
cc
Where do we get those numbers? Is actually measuring latencies of etcd writes? Also, metrics from etcd would help.
This prints, if peer TLS is configured (same in 3.1).
This measures apicalls latency which (at least in case of write calls) includes latency of etcd. We still don't really understand what is happening, but reverting to previous etcd version (3.1) fixes the regression. So clearly the problem is somewhere in etcd.
What Kubemark and Kubernetes versions are you running? We tested Kubemark 1.10 against etcd 3.2 vs 3.3 (500-node workloads), and didn't observe this. How many nodes are needed to reproduce this?
We can't reproduce it with kubemark, even with 5k-node one - see bottom of This appears to be a problem only in real clusters. This is an open question why is it the case.
Since we've rolled back to etcd 3.1. for kubernetes. We also released etcd with the only outstanding critical fix for kubernetes: . Once we've found and fixed the root cause of the performance regression found in this issue we can sketch out a plan to upgrade the etcd server used by kubernetes back up to 3.2.
looks like https://k8s- starts to fail consistently since this morning
From the , the only change is which is enabling quotas in our performance tests by default. The error we're seeing is: - Seems like enabling quotas is affecting our scalability :) Could you look deeper into this - maybe sth serious? Let me file another issue to keep this one on track.
I found something really interesting with the etcd version change, suggesting significant effect of moving from etcd 3.1.11 to 3.2.16. Look at the following graph of etcd memory usage in our 100-node cluster (it has increased ~2x) when the job was moved from k8s release 1.9 to 1.10: ! And next look at how our 100-node job (running against HEAD) sees a drop in mem-usage to half right after my revert from etcd 3.2.16 -3.1.11: ! So etcd 3.2 server version CLEARLY is showing affected performance (with all other variables kept the same) :)
Did we mean 3.1.11?
That's right.. sorry. Edited my comment.
How etcd is configured? We've increased default value from to , in v3.2. So if snapshot count is different, the one with larger snapshot count holds Raft entries for longer, thus requires more resident memory, before discarding old logs.
Aah! That indeed seems like a suspicious change. Wrt flags, I don't think there's any change in those from k8s side. Because, as you can see in my second graph above, the b/w runs and is mainly just my etcd revert change (which doesn't seem to touch any flags). Could you confirm if that's the root-cause of this increased mem usage? If so, we may want to: patch etcd back with original value, or set it to in k8s before switching back to 3.2
Yeah, this change should have been highlighted from etcd side (will improve our change logs and upgrade guides). I am not sure if that would be the root cause. Lower snapshot-count will definitely help alleviate spiky memory usage. If both etcd versions use same snapshot count, but one etcd still shows much higher memory usage, there should be something else.
Update: I verified that the increase in etcd mem usage is indeed caused due to higher --snapshot-count default value. More details here - We should consider setting it to 10,000 when we're bumping to etcd 3.2.16, if we don't want the increased mem usage. cc
Update: With the etcd fix in, the pod startup latency 99% still seems to be close to violating the 5s SLO. There is at least one other regression and I've gathered evidence that it's most likely in b/w runs 111 and 112 of our 5k-node performance job (see the bump b/w those runs in the graph I pasted in ). Currently I'm bisecting the (which has about 50 commits) and the test takes ~4-5 hrs per iteration. The evidence I was referring to above is the following: Watch latencies at 111 were: And overall pod-startup latencies at 111 were: While the same at 112 were: and Meanwhile, if someone's up for the betting game - you can take a look at the commit diff I mentioned above and guess the faulty one :)
ACK. In progress ETA: 13/03/2018 Risks: Can push release date if not debugged before that
toooooooo many commits to place bets :)
That would add more fun I guess ;) Update: So I ran few iterations of the bisection and here's how the relevant metrics looked across commits (ordered chronologically). Note that for the ones I ran manually, I ran them with the earlier regression reverted (i.e 3.2. -3.1.11). From the above, it seems like there may be 2 regressions here (as it's not a straight jump from 2.6s -3.6s) - one b/w " - " and other b/w " - ". Sigh!
expressing as ranges to get compare links: ... ...
Just obtained the results for manual run against and it only makes life harder: as this probably means it might be a flaky regression.
I'm currently running the test against once more to verify the possibility that the regression came in even before it.
So here's the result of running against again: This probably means the regression entered even before run-111 (good news that we have a right end for the bisection now). I'll now try to pursue for a left end. Run-108 (commit ) is a potential candidate, which had the following results when I earlier ran it (with etcd 3.1.11):
My re-run against commit does seem to say it's a good one: I'll take a stab here at bisecting in the range ...
Update: I needed to test the commit thrice in order to gain confidence. It shows quite some variance, but seems like a good commit: I'll continue bisecting further. That said, there seems to have been another spike (of ~1s) in pod-startup latency just couple of days ago. And this one is pushing the 99% to almost 6s: ! My primary suspect from the is the etcd change from 3.1.11 -3.1.12 (). I'd wait for the next run (currently in progress) to confirm it wasn't a one-off - but we really need to understand this. cc
Since I'd be on vacation thu-fri this week, I'm pasting instructions to run density test against 5k-node cluster (so someone with access to the project can continue bisection): IMPORTANT NOTES: Current commit range suspected is ... (let's keep this updated as we bisect) We need to use etcd-3.1.11 instead of 3.2.14 (to avoid including earlier regression). Change the version in the following files to minimally achieve that: cluster/gce/manifests/etcd.manifest cluster/images/etcd/Makefile
cc:
etcd v3.1.12 fixes watch event misses on restore. And this is the only change that we've made from v3.1.11. Does performance test involve anything with etcd restart or multi-node that may trigger snapshot from leader?
From the , doesn't seem like there was any restart. We're using only single-node etcd in our setup (assuming that's what you asked).
I see. Then, v3.1.11 and v3.1.12 should be no different :0 Will take another look if the second run also shows higher latencies.
cc:
Agree with that we should try to get a stronger signal on this one given that the only code change to etcd is isolated to restart/recover code. The only other change is the upgrade of etcd from go1.8.5 to go1.8.7, but I'm doubtful that we'd see a significant performance regression with that.
So continuing with the bisection, seems to be good: I'll update the commit range in accordingly.
Next, commit seems like a good one, though I'd suggest retrying once again to confirm: At this point I'm going to pause the bisection and start my vacation :) I've downed my cluster to make space for the next run.
Thanks Shyam. I am retrying .
For commit I got the following results: So this looks good. Continuing bisection. Current commit range suspected: ...
could you attend our release meeting at 5PM UTC/1PM Eastern/10AM Pacific? It's a Zoom meeting:
I will attend.
Commit looks unclear with the following results: I will run this one more time to get confidence I do not enter the wrong half of the bisection.
OK, so I'm pretty confident now that is bad: Trimming the range down to ... and trying out .
Commit looks good: New suspect commit range is ..., trying out
please note that one of the commits namely got reverted in
Restating comment more clearly, we may be chasing a signal ghost here in the bisection. If is the problem, and it's reverted at already for seemingly unrelated reasons, then does a scalability test against 1.10 branch head show green signal? Can we prioritize re-testing once against 1.10 branch head, rather than continuing bisection?
/ ^^^^
PTAL ASAP
Thanks and Let my try against 1.10 branch head and see what happens.
thanks worst case we go back to what we were bisecting before. right?
10 head has a regression: This is on etcd 3.1.12, not etcd 3.1.11, but if I understand it correctly this should not make much of a difference. Also, looks good: This leaves us with the range ... and there are only 3 commits there. Let me see what I find out. It is also possible that indeed is the culprit here, but then it means that we have another regression that manifests itself on head.
OK, so evidently commit is good: while commit is bad: The bad commit is the merge of the reverted commit mentioned by so we must be observing another regression on head. Let me try rerunning head on etcd 3.1.11 instead of 3.1.12 and see what happens.
ah classic good news bad news :) thanks for keeping this going. any other suggestions?
Head on etcd 3.1.11 is also bad; my next try will be to try directly after the revert (so, at commit ), but with etcd 3.1.11 instead of 3.2.14.
Sounds good Andrzej -- dims
Commit is good, so we have a later regression:
Commit is clearly bad:
So we are basically looking at commits in the range to see what's wrong then? (run bisect between these two)?
Yes. This is a huge range, unfortunately. I am right now investigating .
Thanks what's the current range? (so i can go look at the PR's).
Commit is bad: Commit is also bad: So is commit : And so is : Current range is .... We (me, are starting to suspect that is in fact a flaky pass, so we need a different left end.
/me places bet on as the culprit cc
is, turns out, bad, so (merged as , after ) cannot be the earliest culprit (though it is not impossible that it has introduced yet another regression we will observe once we shake out this one)
Per and I am rerunning , because we suspect this might have been a "flaky good"
I will assume that is actually good based on the following results I have observed:
Currently suspected range: ... Currently testing
Thank you for continuing this work.
per discussion today: fluentd-scaler still has issues: , which have not been fixed by PRs. Is it possible that this regression is caused by fluentd?
one of the PRs relating to fluentd is in the latest range
TBH, I would be really surprised if it would be due to fluentd issues. But I can't exclude this hypothesis for sure. My personal feeling would be some change in Kubelet, but I also looked into PRs in that range and nothing seems really suspicious... Hopefully the range will be 4x smaller tomorrow, which would mean just couple PRs.
OK, so looks good, but I needed three runs to ensure this is not a flake: Next, is bad: Current range is .... Analyzing .
I looked into the range mentioned above - there are only 9 PRs there. - 100% NOT - only changes owners file - potentially - only touching kubectl code, so shouldn't matter for this case - 100% NOT - only touching unrelated e2e tests - potentially (it changes cluster setup) - 100% NOT - only touching unrelated e2e tests - it is changing the test, but not switching the changes on, so unlikely - 100% NOT - only touching AWS code - 100% NOT - only touching owners files So I think we have two candidates here: and I will try to dig deeper into those to understand them.
looks rather bad: Given that this is merge of , one of Wojtek's suspects, I will now run on one commit before that, so . That commit, though, is OWNERS_ALIASES change, and there is nothing left in that range before it, so must be the problem. I will run the test just for safety, anyway.
Discussed offline - we will run tests at head with this commit locally reverted instead.
wow! a one liner causing so much trouble. thanks
One-liners can indeed cause devastation with scalability. Another e.g from past - In general, you might want to see for a good read :)
Update re. test at head: first run with the commit locally reverted passed. This might be a flake, though, so I am rerunning it.
Looking at the commit in wasn't it fixing a bug? It appears to have been fixing a bug placing "scheduled" status into the wrong object. Based on the referenced issue in that PR, it looks like the kubelet could miss reporting a pod was scheduled without that fix?
Who may be able to explain us better what's the effect of that change :)
Yeah - I know it was a bug fix. I just don't fully understand that. It seems to fix the problem of pod reporting as "Scheduled". But we don't see the problem even until time that is reported by kubelet as "StartedAt". The problem is that we see signifcant increase between time reported as "StartedAt" by Kubelet and when pod status update is being reported and watched by the test. So I think the "Scheduled" bit is a red-herring here. My guess (but this is still only just a guess) is that due to this change we are sending more Pod status updates, which in turn is resulting in more 429s or or sth like that. And in the end it takes more time for a Kubelet to report the pod status. But that's something we still need to confirm.
After two runs I am pretty confident that reverting fixes the problem:
This is pretty much the effect I was hypothesizing in (though the cause I guessed was wrong) :) Also, we IIRC did seem to see an increase in # of 429s for put calls (see my ) but that was from an earlier range I think (around etcd change).
My intuition () about issue being on kubelet side quite early-on in the thread was correct after all :)
/sig node the release team could really use review on commit versus revert and the performance issue here
Thanks for explaining this for me. Yeah, that PR is fixing a bug. Previously, kubelet didn't always set . With , kubelet will do that correctly. I'm not sure whether it could introduce more pod status update. If I understand correctly, condition will be set in the first status update and then it will always be there and never changed. I don't understand why it generates more status update. If it really does introduce more status update, it is a problem introduced 2 years ago but covered by a bug, and just fix the bug... Do you have logs for the 2 test run in and Basically a good one and a bad one. The will be very helpful.
and I found that exposed an issue that condition of static pod will keep getting updated. Kubelet generates a new condition for pod which doesn't have it. Static pod doesn't have it, and its status is never updated (expected behavior). Thus kubelet will keep generating new condition for static pod. The issue was introduced in , but covered by a bug. fixed the bug, and exposed the original issue. There are 2 options to quickly fix this: Option 1: Don't let kubelet add condition, kubelet should just preserves condition set by scheduler. Pros: Simple. Cons: Static pod and pod which by-passes scheduler (assign node name directly) won't have condition. Actually without , although kubelet will eventually set this condition for those pods, but it may take a very long time because of a bug. Option 2: Generate a condition for static pod when kubelet sees it initially. Option 2 might introduce less user facing change. But we do want to ask what mean to pods which are not scheduled by scheduler? Do we really need this condition for those pods? /cc Because told me that is used by autoscaling now. /cc
What is the effect of the ? What problem will an end user notice/face (outside of the test harness flakiness)? (from Option )
User won't see condition for a long time.
I have a which implements option 2 in I can change it to option 1 if people think that is a better fix. :)
you better ask folks who know this inside out! (NOT the release team !) ping
IIRC the only static pod running on nodes in our tests is kube-proxy. Could you tell how frequently are these 'continuous updates' made by kubelet? (asking in order to estimate extra qps introduced by the bug)
Yeah, is the only one on node now. I think it depends on pod sync frequency , which is 1 minute. So kubelet generates one extra pod status update every 1 minute.
Thanks. So that means 5000/60 = ~83 qps extra due to those put pod-status calls. Seems to explain the increased 429s noted earlier in the bug.
thank you very much for helping us get this sorted.
np~ helped me a lot!
I still think letting kubelet set the condition is somewhat strange (as I noted in the original PR). Even if kubelet doesn't set this condition, it'd not affect the cluster autoscaler since the autoscaler ignores the pods without the specific condition. Anyway, the fix we eventually came up has very minor footprint, and would maintain the current behavior (i.e., always setting the PodScheduled condition), so we'll go with that.
Also, revived the really old issue for adding tests for steady-state pod update rate
- are you talking about this one: (or do I misunderstand it)? - can you please run 5000-node tests with that PR patched locally (before we merge it) to ensure that this really helps?
I ran the test against 1.10 HEAD + , and the pod-startup latency seems to be fine: Will re-run once more to confirm.
- thanks a lot!
Second run also seems good: Fairly confident now the fix did the trick. Let's get it into 1.10 asap.
Thanks As we talked offline - I think we had one more regression in the last month or so, but that one shouldn't block the release.
Yep. The current fix in that PR is not in the options proposed initially in
re-opening until we have a good performance test result.
any updates on this? This is still blocking 1.10 at the moment.
So the only part of this bug that was blocking the release is the 5k-node performance job. Unfortunately, we lost our run from today due to a different reason (ref: ) That said, we're fairly confident the fix works based on my manual runs (results pasted in ). So IMHO we don't need to block release on it (the next run's going to be on wed).
+1 - I think we can treat this as non-blocker. <updated
Sorry, I edited my post above. I meant that we should treat it as "non-blocker".
Ok, thank you very much. This conclusion represents a tremendous amount of hours you have invested, and I cannot possibly thank you all enough for the work you have done. While we talk in the abstract about "community" and "contributors" you, and the others who have worked this issue represent it in concrete terms. You are the very heart and soul of this project, and I know I speak for everyone involved when I say it is an honor to work alongside such passion, commitment, and professionalism.
[MILESTONENOTIFIER] Milestone Issue: Up-to-date for process <details open<summaryIssue Labels</summary- : Issue will be escalated to these SIGs if needed. : Never automatically move issue out of a release milestone; continually escalate to contributor and SIG through all available channels. : Fixes a bug discovered during the current release. </details<details<summaryHelp</summary<ul<li<a href=""Additional instructions</a</li<li<a href=""Commands for setting labels</a</li</ul</details
This issue was resolved with the relevant fixes in for 1.10. For 1.11, we're tracking the failures under - /close

Discussion:
kl.probeManager.UpdatePodStatus(pod.UID, s) s.Conditions = append(s.Conditions, status.GeneratePodInitializedCondition(spec, s.InitContainerStatuses, s.Phase)) s.Conditions = append(s.Conditions, status.GeneratePodReadyCondition(spec, s.ContainerStatuses, s.Phase)) <del> // s (the PodStatus we are creating) will not have a PodScheduled condition yet, because converStatusToAPIStatus() // does not create one. If the existing PodStatus has a PodScheduled condition, then copy it into s and make sure // it is set to true. If the existing PodStatus does not have a PodScheduled condition, then create one that is set to true. if _, oldPodScheduled := podutil.GetPodCondition(&pod.Status, v1.PodScheduled); oldPodScheduled != nil { s.Conditions = append(s.Conditions, *oldPodScheduled) } podutil.UpdatePodCondition(s, &v1.PodCondition{ </del> <ins> // Status manager will take care of the LastTransitionTimestamp, either preserve // the timestamp from apiserver, or set a new one. When kubelet sees the pod, // `PodScheduled` condition must be true. s.Conditions = append(s.Conditions, v1.PodCondition{ </ins> Type:   v1.PodScheduled, Status: v1.ConditionTrue, })

} // Set ReadyCondition.LastTransitionTime. <del> if _, readyCondition := podutil.GetPodCondition(&status, v1.PodReady); readyCondition != nil { // Need to set LastTransitionTime. lastTransitionTime := metav1.Now() _, oldReadyCondition := podutil.GetPodCondition(&oldStatus, v1.PodReady) if oldReadyCondition != nil && readyCondition.Status == oldReadyCondition.Status { lastTransitionTime = oldReadyCondition.LastTransitionTime } readyCondition.LastTransitionTime = lastTransitionTime } </del> <ins> updateLastTransitionTime(&status, &oldStatus, v1.PodReady) </ins> // Set InitializedCondition.LastTransitionTime. <del> if _, initCondition := podutil.GetPodCondition(&status, v1.PodInitialized); initCondition != nil { // Need to set LastTransitionTime. lastTransitionTime := metav1.Now() _, oldInitCondition := podutil.GetPodCondition(&oldStatus, v1.PodInitialized) if oldInitCondition != nil && initCondition.Status == oldInitCondition.Status { lastTransitionTime = oldInitCondition.LastTransitionTime } initCondition.LastTransitionTime = lastTransitionTime } </del> <ins> updateLastTransitionTime(&status, &oldStatus, v1.PodInitialized) // Set PodScheduledCondition.LastTransitionTime. updateLastTransitionTime(&status, &oldStatus, v1.PodScheduled) </ins> // ensure that the start time does not change across updates. if oldStatus.StartTime != nil && !oldStatus.StartTime.IsZero() {

} } <ins> // updateLastTransitionTime updates the LastTransitionTime of a pod condition. func updateLastTransitionTime(status, oldStatus *v1.PodStatus, conditionType v1.PodConditionType) { _, condition := podutil.GetPodCondition(status, conditionType) if condition == nil { return } // Need to set LastTransitionTime. lastTransitionTime := metav1.Now() _, oldCondition := podutil.GetPodCondition(oldStatus, conditionType) if oldCondition != nil && condition.Status == oldCondition.Status { lastTransitionTime = oldCondition.LastTransitionTime } condition.LastTransitionTime = lastTransitionTime } </ins> // deletePodStatus simply removes the given pod from the status cache. func (m *manager) deletePodStatus(uid types.UID) { m.podStatusesLock.Lock()

verifyActions(t, m, []core.Action{getAction(), updateAction()}) } <ins> func TestUpdateLastTransitionTime(t *testing.T) { old := metav1.Now() for desc, test := range map[string]struct { condition    *v1.PodCondition oldCondition *v1.PodCondition expectUpdate bool }{ "should do nothing if no corresponding condition": { expectUpdate: false, }, "should update last transition time if no old condition": { condition: &v1.PodCondition{ Type:   "test-type", Status: v1.ConditionTrue, }, oldCondition: nil, expectUpdate: true, }, "should update last transition time if condition is changed": { condition: &v1.PodCondition{ Type:   "test-type", Status: v1.ConditionTrue, }, oldCondition: &v1.PodCondition{ Type:               "test-type", Status:             v1.ConditionFalse, LastTransitionTime: old, }, expectUpdate: true, }, "should keep last transition time if condition is not changed": { condition: &v1.PodCondition{ Type:   "test-type", Status: v1.ConditionFalse, }, oldCondition: &v1.PodCondition{ Type:               "test-type", Status:             v1.ConditionFalse, LastTransitionTime: old, }, expectUpdate: false, }, } { t.Logf("TestCase %q", desc) status := &v1.PodStatus{} oldStatus := &v1.PodStatus{} if test.condition != nil { status.Conditions = []v1.PodCondition{*test.condition} } if test.oldCondition != nil { oldStatus.Conditions = []v1.PodCondition{*test.oldCondition} } updateLastTransitionTime(status, oldStatus, "test-type") if test.expectUpdate { assert.True(t, status.Conditions[0].LastTransitionTime.After(old.Time)) } else if test.condition != nil { assert.Equal(t, old, status.Conditions[0].LastTransitionTime) } } } </ins> func getAction() core.GetAction { return core.GetActionImpl{ActionImpl: core.ActionImpl{Verb: "get", Resource: schema.GroupVersionResource{Resource: "pods"}}} }