jiebi/CodeConvo huggingface dataset:
Which jobs are failing: Which test(s) are failing: BeforeSuite Overall Test Since when has it been failing: Testgrid link: Reason for failure: Anything else we need to know: /cc /milestone v1.19 /priority critical-urgent /sig testing
Taking an example from 1 of the runs, these failures seem to show up across all failures. These pods failed to reach a "Running" state after 10 minutes Then, while dumping events from the kube-system namespace these couple look interesting That kube proxy image seems to not exist (?), I tried pulling it from another machine and got this The commit that is part of the kube proxy tag, , does coincide with the k/k commit that went in right as this test broke and that commit range only introduces this PR The following CI tests are listed fro that specific commit (when looking at the commit history of k/k) ci-kubernetes-bazel-build ci-kubernetes-bazel-test post-kubernetes-bazel-build post-kubernetes-bazel-test These jobs dont show anything particularly interesting though.
looks like failed in /cc
I think it there may be a bad manifests used for this job. The kube proxy images are being built. If you look for in the logs of other ci jobs, i.e., you will see that an alpha 3 with a k/k commit is being used. For example, The thing is that, I think, there should be a mention of the ARCH in the container image, e.g., amd64 in the above example, where as teh GPU job doesnt have any arch.
Mmm though im not 100 sold on the image name being the problem. Checking out both kube-proxy and kube-proxy-amd64 both exist and both seem to be updated frequently
/assign I had similar issues trying to use the debian-iptables image, let me check if I'm able to solve it here too /cc FYI
this may help:
Seems the job is failing to install the nodes with an error related to the LICENSES folder LICENSES has a recent history of changes: I'm not able to correlate between the changes and job failures, you have better insights in the LICENSES folder, what's the best way to proceed here?
I think changed LICENSES in the root of the built artifacts from a file to a folder /assign
ARAGAGRAGAHHHHH !!!
FTR, I am looking at this, please don't revert my revert revert yet
Failed to repro, but I also get:
seems that this is only affecting the jobs
Yeah, I think I messed up the invocation. It should have run tests.
OK, I got it to run a test but it doesn't fail this way. I have another umask fix, let me back that out.
I can't get it to fail like this. On my run I see 27/27 pods running, but this run seems stuck with: that varies from 24 - 26. My personal kubetest invocation: And my run failed, but with:
Further more, I can't think of ANY way the old non-directory LICENSES file would be present unless there's some cache that is not invalidated when it needs to be and is leaking across tests
-- is this greenhouse/bazel?
I sincerely doubt greenhouse. I don't understand enough about the bazel build in this repo to say one way or the other. Why would it be consistently affecting this and no other job that runs in the same cluster, e.g.
and why started to fail this morning, if the patch was merged a few days ago?
Jordan figured it out. This node image has /home/kubernetes baked in.
2nd time I get to use this today:
It'd be similar to Linux , a command to search the man page files
Why not just make do a search if it doesn't have an exact match?
sounds good.
SGTM
- Can be useful in this issue
I will work on this issue, if nobody is already working on it ( is not working on it).
thanks!
I focused currently on a bit different stuff, sorry. So I will start/finish it later. If anyone want to take over this one, feel free for now.
What happened: What you expected to happen: I am not sure if is it a bug, I think we should catch err when Watch testResource failed in func TestWatchCallNonNamespace. How to reproduce it (as minimally and precisely as possible): Anything else we need to know?: Environment: Kubernetes version (use ): Cloud provider or hardware configuration: OS (e.g. from /etc/os-release): Kernel (e.g. ): Install tools: Others:
Absolutely we should check err gracefully.
/sig cli
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
/fixes it
Getting the following error when creating a service with type when kubernetes has been deployed without tagging the vpc/node. This prevents from using ELBs with the AWS cloud provider when deploying kubernetes in an existing vpc (which cannot be tagged because they already have).
I also experienced this failure when trying to provision Kubernetes on AWS with kube-up and specifying an explicit, existing VPC_ID — which wasn't tagged, but then I didn't know that was required :)
I see we have a pr, can we also update the docs in time for 1.2.0?
sure, what do you want me to update in the docs exactly?
It's 400+mb and we need to pull it on every node. This leads to a lot of image pulling flake (), and also gets very expensive in some countries where bandwidth is sold at a premium. We took a stab at halving the size with: , but then realized that we were using an unsupported base image We tried to optimize to ~300Mb by building a smaller ubuntu:15 base, but that's still huge. Every other mandatory system daemon is built on busybox and so takes ~10Mb.
I can try again now that the agent supports
Newer Docker images using the spec from the official repository are based on AlpineLinux and are around ~38MB:
If you give a prerelease version to , it'll match both and . Based on this logic: We'll end up trying to pull binaries for this prerelease from the CI bucket () rather than the release bucket (). These pre-release versions are usually* pushed to the release bucket as part of the release process (e.g. ), and so should probably prefer these over the CI versions (which may or may not exist, depending on whether a commit was pushed to the release branch between when the tag was created and the build occurred). More explicitly, the current logic is: the release bucket by default. a version matches the CI regex, use the CI bucket. , if a version matches the release regex, we're good - continue to use the default (release bucket). , bail out - invalid version. The correct logic would probably be something like the release bucket by default. a version matches the release version regex, continue to use the release bucket. , if a version doesn't match the release version regex, but it matches the CI regex, use the CI bucket. it matches neither, bail out. Important note: due to our hotfix for , there is a somewhat fake prerelease tag () which doesn't have any bits in the release bucket, and we expect others soon. We should not fix this bug in until the latest pointers in the CI bucket do not point to valid release/prerelease versions.
/sig testing /sig release I guess?
/area release-eng /priority critical-urgent /assign /milestone v1.18
Hello With code freeze approaching for 1.18 (5 March), can you please check if the current issue is still intended for milestone 1.18?
-- I'm pretty sure I can get to this before the cycle is out.
if our current approach for tagging release branch pre-releases doesn't trigger this, this seems like it is not critical for the milestone, is that correct?
-- Good point! /milestone v1.19 /remove-priority critical-urgent /priority important-soon
Given the priority dip on this one, I think it's fine for someone in Releng other than Tim or I to pick this up. /unassign -- This is a good one to work on. /help
I can look that, after the release 1.18, if is that fine then can we have a short discussion about this one? /remove-help /assign
-- SGTM
Kubernetes version (use ): 1.4.0 Environment: Cloud provider or hardware configuration: Azure What happened: Trying to mount an Azure file share by following the directions fails because hyperkube does not include cifs-utils (eg, ). Can this be to the hyperkube image, please?
is this related to GCI image missing nfs related package?
This should fix the reported issue:
Thanks for fixing this so quickly!
No, unrelated
Take great care merging! Example PR run (look at the "succeeded" builds): https://travis-
Somewhat related, (bug) and (fix, as yet unmerged) mean that "guestbook" and "shell" e2e tests are being silently skipped on all platforms since Wed last week (4/15). should merge shortly and resolve the issue.
From the Travis run, So it looks like either the shell script is eating the exit code, or the test is exiting 0 even when tests fail. Similarly for Shippable, it seems.
I think broke this. It appears that will always exit 0, even when tests fail or timeout. We should revert this change ASAP.
Fixed by
<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --In ipvs mode client traffic(UDP) to a pod behind LoadbalancerIP is blackholed after node having the pod is rebooted. The issue seems to be similar to the below iptable issue, since client continuously tries to connect to the loadbalancerIP (even before kubeproxy has applied all rules), and we see that stale conntracks entries remain even after rules are applied. After the stale conntrack entries are cleaned manually the traffic starts flowing : From the below code looks like ipvs currently clears stale conntrack entries for ExternalIPs but not LoadbalancerIP(?): ().String())-,for%20%2C%20extIP%20%3A%3D%20range%20svcInfo.ExternalIPStrings()%20%7B,-staleServices.Insert(extIP UDP traffic between client and pod(behind Loadbalancer IP) should have been established after node reboot. a POD acting as a UDP server and expose it through Loadbalancer service a UDP client to continuously send traffic to the LoadbalancerIP the node having the pod. after the new pod serving UDP is up, the UDP traffic from client would be blackholed(until stale conntrack entries are cleared). Kubernetes version (use ): 1.21.1 Cloud provider or hardware configuration: Manufacturer: Dell Inc. Product Name: PowerEdge R640 Version: Not Specified OS (e.g: ): SUSE Linux Enterprise Server 15 SP2 Kernel (e.g. ): Linux pool16-n108-wk16-n080 5.3.18-24.75.3..0.PTF.-default # 1 SMP Thu Sep 9 23:24:48 UTC 2021 () x8664 x8664 x8664 GNU/Linux Install tools: Network plugin and version (if this is a network-related bug): Calico v3.19.1-12- Others:
/sig network
/assign I'll try to reproduce this using but it may take some days.
yeah, ipvs kube-proxy doesn't have some of the latest "conntrack stale" fixes like , mainly because I'm not very familiar with IPVS and I can't know if all of them are needed or only some of them.
/triage accepted is it a matter of inserting externalIPs in conntrack?
No, when a node is rebootes and udp messages to a loadBalancerIP enters before kube-proxy is started, then invalid (UNREPLIED) conntrack entrys are created. They will black-hole traffic, and since traffic keeps coming they will never timeout. This is not easy to reproduce. I found it best to direct incoming traffic to a node (2), and have one endpoint pod on another node (4). Then reboot the node where traffic enters (and let the pod live) while traffic is sent continously. After reboot, check the conntrack on the rebooted node. In this example 3 out of 10 connections are ok. Those are the fortunate ones that didn't send a packet in the small time span from node start to kube-proxy start. (about 1 pkt/sec is sent on all 10 connections)
/area kube-proxy /area ipvs
Perhaps a bug in
Thanks for catching this! I'm surprised it wasn't noticed earlier. It's an off by 1 (s) error. We pushed to gs://kubernetes-releases vs gs://kubernetes-release. The offending s has been removed. I'll clean up the release itself.
I thought this was right, but I think this is still an issue. I'm trying to release , and it's building wrong. I guess there were actually two problems; you caught one, but not the one I meant. This issue was about: should report itself as , but instead it reports itself as .
I'll look into this, since it's blocking alpha releases.
This seems contrary to , but doesn't have enough context for me to know why it was changed.
This was fixed by a long time ago.
TL;DR prevents us from using structured authz (setting flag) as those flags are mutually exclusive /cc (as the author of ) /cc FYI To use structured authz you need to define the authorization configuration using the flag. However this flag is mutually exclusive with and flags. (refer to & ) Kubeadam adds by default ( & ) which prevents us from using I should be able to define the flag and have structured authz working. Spin up a kind cluster using this kind config (make sure the images are 1.29 for structured authz to be avaliable) (For this bug, the content of doesn't matter.) However, the cluster become unhealthy. docker exec to the kind cluster container, where we can see the inner container of the apiserver has exited. Running we can see this error <img width="1720" alt="image" src=""No response_ <details</details<detailsN/A -I am using kind </details<details</details<details</details<details</details<details</details
There are no sig labels on this issue. Please add an appropriate label by using one of the following commands: - Please see the for a listing of the SIGs, working groups, and committees available. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
FYI, in case someone opens similar issue in Kind. I might be wrong but I think this effectively means that kind users cant use structured authz (unless it supports a mode to spin up a cluster without kubeadam).
This started after was merged. The HPA test started being very flaky since then: https://k8s- The HPA test in containerd e2e become even more flaky since then with the same failure pattern: https://k8s-
I think this is probably /priority critical-urgent Release team hasn't looked close enough at this issue as we've been trying to clear other issues in front of it. If we crack those and this is unrelated root cause, then this one's going to be a release blocker.
It's going to be supremely hard to debug this unless we have logs from metrics-server and/or snapshots of the HPA over time. I'm trying to reproduce on a local cluster, but so far, the test passes consistently. did you manage to reproduce locally?
is there any way to grab logs of pods from a testing cluster?
you probably want to save it as part of the artifacts?
yeah, what's the process for that?
somewhere along
See for an attempt at debugging this.
can we provide the data is looking for? we're seeing this quite often.
ack, more logs would be helpful. Currently trying to get usable logs off the CI with the aforementioned PR, but I keep making minor mistakes in the log collection script :-/. It seems to repro every few runs though, so with any luck I'll see it again soon.
Got a WIP fix up: Need to do one more tweak before merge.
Re-running tests with the above PR, if they succeed I'll merge and get someone to do a build when the people capable of running builds are on next.
status check? I'm worried this is a 1.12 blocker and need to make a call by end of day today regarding delaying the release.
According to our testing, metrics-server 0.3.1 seems to have fixed it - thank you
<!-- Please only use this template for submitting reports about flaky tests or jobs (pass or fail with no underlying change in code) in Kubernetes CI --Which jobs are flaking: Which test(s) are flaking: Testgrid link: Reason for failure: Anything else we need to know: /sig network <!-- Please see the deflaking doc () for more guidance! --
/triage unresolved Comment when the issue is assessed and confirmed. I am a bot run by vllry. ‍
/assign
Reviewing this as I code the CI Signal report No longer flaking on However, the test is flaking on other jobs as can be seen here Jobs affected ci-kubernetes-e2e-gce-cos-k8sbeta-default, ci-kubernetes-e2e-gce-multizone ci-kubernetes-e2e-gci-gce-containerd links to the flakes can be found in Triage link above Tentative Observation : No flaking results on jobs using kind but some flaking on gce jobs?? what would be the next best step to carry out for this? cc
Sorry I've been underwater for a bit. It looks like the test is expecting only one EndpointSlice to be created but sometimes an extra one is created due to informer cache being out of date. Although not ideal, this would still be a valid outcome. I need to update the test to consider this a valid outcome.
any updates on this?
Thanks for the reminder, filed to fix this.
/triage accepted
sending the following json to update a service: "{"apiVersion":"v1beta1","port":6380,"containerPort":6379}" fails with 422 and message: "service "" is invalid: name: required value ''" It's not clear from the message what is exactly the problem. is the "name" field missing? isn't it called "id" at the moment? if yes, another issue () points on the fact that including the id fails the request with another error.
similar issue happens when creating a service: input (based on the guestbook example) : POST fails with :
Internal names are not converted to external during rendering of the status message. Name is the newer api version field name for what used to be called id
but the input json contains "id", so why does it fail in the first place?
I'm referring to the first comment, where update failed because you omitted the ID of the service you were updating
Update requires the full resource to be specified. GET the resource, modify the fields you want to change, and do a PUT.
Using for the field name conversion problem.
As mentioned, PUT should not accept partial updates - PATCH should, and the issue for that is . This is documented in : "Otherwise, PUT expects the whole object to be specified." Where else does it need to be documented?
So I started using namespaces for the first time, and I ran: nothing was showing up...I was really confused, debugging stuff for 30+ minutes.. finally I was like, even tho this is really dumb, maybe they are behind a namespace, so I did: and finally my pods showed up. so yes there is: but this should really be the default, shouldn't it? You will save people a lot of time if you make this fix. tbh, this is a really crappy cli design in this case.
While this might sound like a good idea if you're running a single deployment with a few pods, once you start running multiple dozens of deployments and objects it starts to become less and less useful to have this become the default. I also suspect exposing all objects without specifying namespace is probably a non-goal. Namespaces were created for a reason which is to logically segment resources together. In my opinion, what you are suggesting just nullifies the entire purpose of namespaces.
I think this is more of a usability bug. kubectl get pods gives the list of all pods in default namespace.By default no pods are present in default namespace. So it does give a message saying No Resources found. Improving this message a little bit will be helpful like No Resources found in default namespace
/sig cli
I could implement this if that's alright with you !
sure, how about
We should wait for kubernetes members to comment if this change should be taken up
The title of this issue could use an update, if the priority has been shifted to improving the output of an empty namespace. Maybe it's early to claim the consensus seems to be that --all-namespaces should not be the default, but I personally agree that it shouldn't be. Take note of the shorthand , to make typing it a bit easier, or the option to to something other than to get around typing your namespace altogether.
:) IMHO, is innocent. But we could make the messages more clear. For example, will show also. And is a bit long, but a is too simple to describe our intent, "all namespaces of cluster or all some resource of some namespace", so puzzled. My conclusion: is competent, namespace is ok, making more clear message is enough.
"~" is already implemented as in , in .
<!-- This form is for bug reports and feature requests ONLY! If you're looking for help check and the . --Is this a BUG REPORT or FEATURE REQUEST?: /kind bug What happened: azure file plugin on Windows does not work after node restart, this is due to powershell cmdlet has lost account name/key after reboot, we should remove the invalid link and do the mount again after kubelet restart. What you expected to happen: azure file plugin on Windows should work after node restart How to reproduce it (as minimally and precisely as possible): has created a pod with azure file mount on Windows agent node could get following error: Anything else we need to know?: Environment: Kubernetes version (use ): v1.9, v1.10 Cloud provider or hardware configuration: OS (e.g. from /etc/os-release): Kernel (e.g. ): Install tools: Others: /assign /sig windows /sig azure
Added to v1.10 milestone since it is a critical bug fix. cc
/kind failing-test <!-- If this is a flake, please add: /kind flake --/sig cli
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/assign
What is missing? Based on SIG call it's probably "tests" but I am not sure. Could use some investigation and followup - how do we get it to GA? /sig network /kind cleanup
/triage unresolved Comment when the issue is assessed and confirmed. I am a bot run by vllry. ‍
/assign
This one seems not a good first one, I'll take this to myself and start bothering people in Slack to check what's missing, mostly as I've seen this is also related to cloud providers :)
I think this one originally. According to the KEP in This should have went GA in v.19 and be removed in v1.20. Given it's still marked beta, at the very least we should mark it GA in v1.20 and remove it in v1.21. anything I'm missing here?
yeap. Honest question, changing a Feature Gate to GA process is only changing the related line in and also putting a comment that this goes to GA? If so, probably this one can also be a good first issue and we can open a future issue for the removal in v1.21 once v1.20 ships. I think this will also need to be changed in the in v1.21 right? If is ok with that, I would like then to mark this as a good first issue to turn the feature GA in v1.20 :)
Well, taking a look into how secure is to move to GA: In the KEP there are some considerations about removing the IsMasterNode function, which seems to be already dealt in commit E2E tests needs to be also checked about usage of IsMasterNode(), which I could not find anymore and also seems to be removed from the same commit above. About the same Kep, it's directly related to other Feature Gate: ` So what I think that needs to be done: Promote LegacyNodeRoleBehavior to GA and Default it to false as proposed in the Kep -I'll open another issue just for this According to the LegacyNodeRoleBehavior should be removed in v1.19 but according to the Kep in v1.21, so let's stick with v1.21? ServiceNodeExclusion - Should be moved to GA, already defaulted to true. This issue is going to deal ONLY with this.