jiebi/CodeConvo huggingface dataset:
Code change:
// Reboot the node. if err = issueSSHCommand(node, provider, rebootCmd); err != nil { <del> // Just log the error as reboot may cause unclean termination of ssh session, which is expected. </del> Logf("Error while issuing ssh command: %v", err) <ins> result <- false return </ins> } // Wait for some kind of "not ready" status.

Related discussion:
text: Two cases of Reboot e2e are flaky on Jenkins: unclean reboot (echo b sudo tee /proc/sysrq-trigger). They reboot only one node, while reboot of the second doesn't happen and causes test timeout:
CC
Master tracking bug
Re-opening. Current status as follows: each node by dropping all outbound packages for a while and ensure they function afterwards (Failed 8 times in the last 30 runs. Stability: 73 %) each node by dropping all inbound packages for a while and ensure they function afterwards (Failed 5 times in the last 30 runs. Stability: 83 %) each node by ordering unclean reboot and ensure they function upon restart (Failed 3 times in the last 30 runs. Stability: 90 %) each node by triggering kernel panic and ensure they function upon restart (Failed 3 times in the last 30 runs. Stability: 90 %) each node by switching off the network interface and ensure they function upon switch on (Failed 3 times in the last 30 runs. Stability: 90 %) each node by ordering clean reboot and ensure they function upon restart (Failed 3 times in the last 30 runs. Stability: 90 %)
Any progress on this ? cc in case Jerzey is OOTO.
Sorry for replying so late, but I was on holidays. The flakiness was caused by liveness probes of some pods (e.g.: kube-dns).
Also - implemented a fix for it - see
For some reason seems to have fixed most but not all reboot tests. For example: Reboot each node by dropping all outbound packets for a while and ensure they function afterwards: kubernetes-e2e-gce-reboot (Failed 8 times in the last 30 runs. Stability: 73 %) Need to investigate and fix.
perhaps you could take a look in Poland while the U.S. is celebrating July 2nd, 3rd, 4th? :-)
There was a problem with monit restarting kubelet because of blocking localhost communication on firewall. Fixed by .
- since is now merged, can we close this issue? Or is there anything remaining here?
Yes, I think that we can close this now.
Code change:
// Reboot the node. if err = issueSSHCommand(node, provider, rebootCmd); err != nil { <del> // Just log the error as reboot may cause unclean termination of ssh session, which is expected. </del> Logf("Error while issuing ssh command: %v", err) <ins> result <- false return </ins> } // Wait for some kind of "not ready" status.

Related discussion:
text: All reboot tests are pretty stable (1 failure in 180 runs, in total): each node by dropping all inbound packets for a while and ensure they function afterwards (No known failures. Stability 100 %) 4 min 5 sec Passed each node by dropping all outbound packets for a while and ensure they function afterwards (No known failures. Stability 100 %) 4 min 6 sec Passed each node by ordering clean reboot and ensure they function upon restart (Failed 1 times in the last 30 runs. Stability: 96 %) 1 min 41 sec Passed each node by ordering unclean reboot and ensure they function upon restart (No known 1. failures. Stability 100 %) 1 min 25 sec Passed each node by switching off the network interface and ensure they function upon switch on (No known failures. Stability 100 %) 4 min 0 sec Passed each node by triggering kernel panic and ensure they function upon restart (No known failures. Stability 100 %)
Code change:
//  * the cluster comprises at least one Linux node that accepts workloads - it //    can be the master, but any other Linux node is fine too. This is needed for //    the webhook's pod. <ins> //  * in order to run "can read and write file to remote folder" test case, a folder (e.g. "write_test") need to be created //    in that AD domain and it should be shared with that GMSA account. </ins> // All these assumptions are fulfilled by an AKS extension when setting up the AKS // cluster we run daily e2e tests against, but they do make running this test // outside of that very specific context pretty hard.

Related discussion:
text: <!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --There was a regression in containerd that required a fix for gmsa working with UNC paths. See the fix at We should have a test that validates this functionality. See for set up and place to add this test. Kubernetes version (use ): Cloud provider or hardware configuration: OS (e.g: ): Kernel (e.g. ): Install tools: Network plugin and version (if this is a network-related bug): Others: /sig windows /triage accepted /priority important-long-term /help
This request has been marked as needing help from a contributor. Please ensure the request meets the requirements listed . If this request no longer meets these requirements, the label can be removed by commenting with the command. <detailsIn response to : <!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! --#### What happened: There was a regression in containerd that required a fix for gmsa working with UNC paths. See the fix at We should have a test that validates this functionality. Kubernetes version (use ): Cloud provider or hardware configuration: OS (e.g: ): Kernel (e.g. ): Install tools: Network plugin and version (if this is a network-related bug): Others: /sig windows /triage accepted /priority important-long-term /help Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
The label(s) cannot be applied, because the repository doesn't have them. <detailsIn response to : There was a regression in containerd that required a fix for gmsa working with UNC paths. See the fix at We should have a test that validates this functionality. Kubernetes version (use ): Cloud provider or hardware configuration: OS (e.g: ): Kernel (e.g.
Code change:
//  * the cluster comprises at least one Linux node that accepts workloads - it //    can be the master, but any other Linux node is fine too. This is needed for //    the webhook's pod. <ins> //  * in order to run "can read and write file to remote folder" test case, a folder (e.g. "write_test") need to be created //    in that AD domain and it should be shared with that GMSA account. </ins> // All these assumptions are fulfilled by an AKS extension when setting up the AKS // cluster we run daily e2e tests against, but they do make running this test // outside of that very specific context pretty hard.

Related discussion:
text: ): Install tools: Network plugin and version (if this is a network-related bug): Others: /sig windows /triage accepted /priority important-long-term /help Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/priority important-longterm
/assign
if you have questions feel free to reachout on slack in sig-windows
Will do, thanks!
/assign
The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs. This bot triages issues and PRs according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Mark this issue or PR as fresh with Mark this issue or PR as rotten with Close this issue or PR with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at . /lifecycle stale [1]:
/remove-lifecycle stale
The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs. This bot triages issues and PRs according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Mark this issue or PR as fresh with Mark this issue or PR as rotten with Close this issue or PR with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at . /lifecycle stale [1]:
The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs. This bot triages issues and PRs according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Mark this issue or PR as fresh with Close this issue or PR with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at .
Code change:
//  * the cluster comprises at least one Linux node that accepts workloads - it //    can be the master, but any other Linux node is fine too. This is needed for //    the webhook's pod. <ins> //  * in order to run "can read and write file to remote folder" test case, a folder (e.g. "write_test") need to be created //    in that AD domain and it should be shared with that GMSA account. </ins> // All these assumptions are fulfilled by an AKS extension when setting up the AKS // cluster we run daily e2e tests against, but they do make running this test // outside of that very specific context pretty hard.

Related discussion:
text: /lifecycle rotten [1]:
/remove-lifecycle rotten
/remove-lifecycle rotten
/assign
I have thought about this a bit and think it might be easiest to just expose a share on the domain controller. This does push more requirements into the configuration that needs to happen outside of the tests but it will make the tests a lot cleaner IMO.
/assign
GitHub didn't allow me to assign the following users: iXinqi. Note that only , repo collaborators and people who have commented on this issue/PR can be assigned. Additionally, issues/PRs can only have 10 assignees at the same time. For more information please see <detailsIn response to : /assign Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
Code change:
pushd "${KUBE_ROOT}" 2>&1 > /dev/null # Test for diffs if ! _out="$(diff -Naupr --ignore-matching-lines='^s*"GoVersion":' --ignore-matching-line='^s*"GodepVersion":' --ignore-matching-lines='^s*"Comment":' Godeps/Godeps.json ${_kubetmp}/Godeps/Godeps.json)"; then <del> echo "Your Godeps.json is different:" echo "${_out}" echo "Godeps Verify failed." </del> <ins> echo "Your Godeps.json is different:" >&2 echo "${_out}" >&2 echo "Godeps Verify failed." >&2 </ins> echo "${_out}" > godepdiff.patch <del> echo "If you're seeing this locally, run the below command to fix your Godeps.json:" echo "patch -p0 < godepdiff.patch" echo "(The above output can be saved as godepdiff.patch if you're not running this locally)" echo "(The patch file should also be exported as a build artifact if run through CI)" </del> <ins> echo "If you're seeing this locally, run the below command to fix your Godeps.json:" >&2 echo "patch -p0 < godepdiff.patch" >&2 echo "(The above output can be saved as godepdiff.patch if you're not running this locally)" >&2 echo "(The patch file should also be exported as a build artifact if run through CI)" >&2 </ins> KEEP_TMP=true if [[ -f godepdiff.patch && -d "${ARTIFACTS_DIR:-}" ]]; then echo "Copying patch to artifacts.."

Related discussion:
text: <!-- This form is for bug reports and feature requests ONLY! If you're looking for help check and the . --Is this a BUG REPORT or FEATURE REQUEST?: When running on my commits locally I got the following error: Affected PRs:
/sig testing
/cc
There's likely something up with your local environment. These jobs are running correctly in CI, although you need to look at the build log for output
Code change:
[[ -n ${1-} ]] || { kube::log::error_exit "!!! Internal error. No platform set in kube::golang::set_platform_envs" } <del> # make sure we have a clean slate first kube::golang::unset_platform_envs </del> export GOOS=${platform%/*} export GOARCH=${platform##*/}

Related discussion:
text: PR has broken my local build (, ). It unsets GOROOT and go build then looks into which does not exist. PTAL, it's really annoying regression.
same here
ugh, sorry. too many supported ways of building kubernetes. will fix.
Yup, I hit this too.
Pretty much every red hatter will hit this.
thanks for a quick fix
Code change:
[[ -n ${1-} ]] || { kube::log::error_exit "!!! Internal error. No platform set in kube::golang::set_platform_envs" } <del> # make sure we have a clean slate first kube::golang::unset_platform_envs </del> export GOOS=${platform%/*} export GOARCH=${platform##*/}

Related discussion:
text: Similar to this is to test the first contributor experience fails on ubuntu 16.04, docker version 1.12.3:
Got the same error on Linux ubuntu 4.2.0-27-generic ~14.04.1-Ubuntu SMP Fri Jan 22 15:32:26 UTC 2016 x8664 x8664 x86_64 GNU/Linux Docker Client version: 1.6.2 Client API version: 1.18 Go version (client): go1.2.1 Git commit (client): OS/Arch (client): linux/amd64 Server version: 1.6.2 Server API version: 1.18 Go version (server): go1.2.1 Git commit (server): OS/Arch (server): linux/amd64
What git sha of were you building from?
I have cloned the head. I think the reason might be missing package: gccgo-4.8-aarch64-linux-gnu I'm compiling right now and update on result.
were you running or ? runs inside docker, so the gcc error is confusing.
I run: sudo make release in Ubuntu VM running in VMWare workstation if that matters. Are you in kubernetes-users slack channel?
does fail in the same way?
Do you need to run , instead of just ? Can you try running the following:
well, if I don't do sudo docker does not work :) I need to run as root. I'll try the commands as soon as quick-release ends.
ah, gotcha. you can probably fix the docker access issue by following in the docker manual. that's probably not why it's failing, though.
make release still does not work :(
can you please run and then paste the results? ( turns up the debugging verbosity.)
for me: works (using your PR to fixes the gobindata dep) -local Go build- works (docker build) works if you have Docker running. does not work: on OSX did not seem to give more verbosity.
V=4 make release did not help :(
oops, that was not the correct environment variable, sorry. I meant .
here we go:
Aha, I think I figured it out. Sorry for making you rebuild so many times; the issue still isn't super obvious from the logs.
Code change:
[[ -n ${1-} ]] || { kube::log::error_exit "!!! Internal error. No platform set in kube::golang::set_platform_envs" } <del> # make sure we have a clean slate first kube::golang::unset_platform_envs </del> export GOOS=${platform%/*} export GOARCH=${platform##*/}

Related discussion:
text: Basically, the bug only occurs if you don't have enough memory to do the docker builds in parallel; if that happens, you fall into this . On the parallel side of the branch, we set the platform env vars within a subshell, so each of the cross-compilation builds won't affect each other. On the non-parallel side of the branch, we don't use a subshell. Inside , there's no overrides done for . As a result, we end up incorrectly reusing the value for , , which doesn't recognize the flag being applied for the darwin build. The fix should be easy; I'll get that sent out ASAP. we probably want this fix for 1.5, since if you fall back to non-parallel crossbuilds we're doing the wrong thing.
Ack. Will keep an eye out for the fix. Please mark it as cherrypick-candidate
Thank you guys!
I think it is almost there. Got much further in the build (4 GB of RAM). Towards the end there is a failed test that makes the build fail: This is probably a different bug, as your PR got passed the misconfigured compilation.
FWIW, the worked on ubuntu 16.06. It did ran some tests apparently but the build succeeded.
I just tried with and it worked fine
since we have a PR ready and working, can we please switch this to non-release-blocker?
Code change:
v1 "k8s.io/api/core/v1" "k8s.io/apimachinery/pkg/fields" "k8s.io/apimachinery/pkg/labels" <ins> "k8s.io/apimachinery/pkg/types" </ins> "k8s.io/client-go/kubernetes/fake" "k8s.io/kubernetes/pkg/controller/testutil"

Related discussion:
text: What happened: A Pod was scheduled for deletion because a taint w/ effect was observed in the Node the Pod was assigned to, and after the taint is removed from the Node, the Pod scheduled deletion isn't canceled. After scheduling pod deletion, the only cancels pod deletion if all taints are removed from the node, including any taints that the user relies on for their use-cases (example below). What you expected to happen: After the taint that triggered scheduling pod deletion is removed, cancels the scheduled pod deletion. How to reproduce it (as minimally and precisely as possible): A node is tainted w/ , so any workloads already assigned to it which don't tolerate this taint are evicted. A pod is created w/ tolerations: w/ unspecified and w/ . is assigned to . At some point in time after the previous steps took place, is tainted as . gets an update event for and observes two taints. It proceeds to calculate the minimum time (in seconds) that tolerates for the two taints which at this point is 300 seconds, and marks for deletion in ~300 seconds - this happens in-memory, in a timed-queue. At some point in time after the previous steps took place, and before the 300 seconds trigger happen, taint is removed from . gets an update event for and observes only one taint, . It proceeds to calculate the minimum time (in seconds) that tolerates for the this taint which is infinity, and returns, not canceling the previous deletion. It completely ignores the fact that the taint that triggered deletion is no longer observed. Anything else we need to know?: Worked w/ to figure this out after all pods in a node got evicted without apparent reason. Environment: Kubernetes version (use ): 1.17.3 (but looking at the code seems to affect all of 1.17.x and 1.18.x and master. Cloud provider or hardware configuration: N/A OS (e.g: ): N/A Kernel (e.g. ): 5.x Install tools: N/A Network plugin and version (if this is a network-related bug): N/A Others: N/A cc (owners of this code)
There seems to be at least .
Code change:
v1 "k8s.io/api/core/v1" "k8s.io/apimachinery/pkg/fields" "k8s.io/apimachinery/pkg/labels" <ins> "k8s.io/apimachinery/pkg/types" </ins> "k8s.io/client-go/kubernetes/fake" "k8s.io/kubernetes/pkg/controller/testutil"

Related discussion:
text: You can sort of get a feel for what's going on here by looking at these two code snippets: This is what happens in when the node does not have any NoExecute taints: If you have any NoExecute taints on the node processPodOnNode gets called and this routine is really, really eager to evict pods. So : We are currently working around this issue by changing over our NoExecute taints to NoSchedule.
/sig scheduling node
/cc
/remove-sig scheduling scheduler isn't involved in this case.
Which SIG owns the controller-manager or is the node lifecycle manager (and its NoExecuteTaintManager) owned by sig-node alone?
I think node lifecycle is jointly owned by sig-node and sig-cloud-provider
API machinery owns the mechanics of the controller manager (controller loop setup and management), but the individual controllers are SIG-specific
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
Looks like we lost some momentum here and I'd like to avoid this falling into the abyss of bugs that never get fixed. When this bug is triggered the outcome is pretty terrible and once it does happen it is very difficult to dig into and understand what actually happened. Did we ever get this routed to the right owner?
adding sig-node leads for triage/routing. this seems like a clearly reproducible bug with significant correctness issues. /assign
thanks for the detailed report and reproducer. to help accelerate this, do you have a form of your reproducer we could add in the e2e suite (even if its failing initially)?
custom taints are considered disruptive, so it couldn't run in the main suite
It's been a while since I wrote tests for the e2e suite but if you are available over slack in case guidance is needed, I'm willing to put the effort.
Do all suites need to pass for a fix to be merged? I'd assume it is the case, otherwise how would a regression be prevented after the fix lands?
Code change:
v1 "k8s.io/api/core/v1" "k8s.io/apimachinery/pkg/fields" "k8s.io/apimachinery/pkg/labels" <ins> "k8s.io/apimachinery/pkg/types" </ins> "k8s.io/client-go/kubernetes/fake" "k8s.io/kubernetes/pkg/controller/testutil"

Related discussion:
text: i was more inquiring if there was a test that could automate the reproduction, so even just a linked commit may be helpful for someone that wants to pick this up. i am happy to help if you want to reach out over slack later this week.
looks like a unit test could reproduce the issue, opened a WIP fix in
our code is not publicly accessible, I'm sorry. PR linked by seems to cover both the fix and the test that proves it.
/remove-lifecycle frozen
frozen keeps the bot from auto-closing
Ah, sorry then.
Code change:
return fmt.Sprintf("%s.%s", id, secret) } <del> // IsValidBootstrapToken returns whether the given string is valid as a Bootstrap Token and // in other words satisfies the BootstrapTokenRegexp </del> <ins> // IsValidBootstrapToken returns whether the given string is valid as a Bootstrap Token. // Avoid using BootstrapTokenRegexp.MatchString(token) and instead perform constant-time // comparisons on the secret. </ins> func IsValidBootstrapToken(token string) bool { <del> return BootstrapTokenRegexp.MatchString(token) </del> <ins> // Must be exactly two strings separated by "." t := strings.Split(token, ".") if len(t) != 2 { return false } // Validate the ID: t[0] // Using a Regexp for it is safe because the ID is public already if !BootstrapTokenIDRegexp.MatchString(t[0]) { return false } // Validate the secret with constant-time: t[1] secret := t[1] if len(secret) != api.BootstrapTokenSecretBytes { // Must be an exact size return false } for i := range secret { c := int(secret[i]) notDigit := (c < 48 || c > 57)   // Character is not in the 0-9 range notLetter := (c < 97 || c > 122) // Character is not in the a-z range if notDigit && notLetter { return false } } return true </ins> } // IsValidBootstrapTokenID returns whether the given string is valid as a Bootstrap Token ID and

Related discussion:
text: The vendor considers this issue Informational Risk, High Impact, Undetermined Exploitability To view the original finding, begin on page 42 of the Test Environment Kubernetes 1.24.3
/sig security
/sig cluster-lifecycle
/assign
PR that attempts to fix this:
/triage accepted poked #sig-security at slack for review ^