jiebi/CodeConvo huggingface dataset:
! ! Which jobs are failing: Which test(s) are failing: Since when has it been failing: failures started between 3-4PM (EST), on 1/11/2019 the failures appear limited to containerd jobs (here's a triage search for note that only containerd jobs appear as failing): /sig node
the relevant commit range on master appears to be ... the relevent commit range in test-infra appears to be
cc
for test-infra: aws-janitor: clean up ELBs - definitely unrelated, only affects some of the AWS testing cleanup Update kubekins tag for eks jobs - definitely unrelated, affects the docker image in which some EKS testing runs only Update ubuntu image used for e2e node test - I think unrelated, appears to only affect the node e2e tests rest are merge commits
thanks for checking. it's also possible the commit ranges I linked to need to be expanded slightly... that was my first attempt at correlating merge timestamps with the triage board failures... it's possible I got timezone correlation wrong a few hours in one direction or the other.
ah, they probably do then. otherwise I'd say something else must have changed out of band (perhaps there's a pointer to a containerd version to use somewhere or something).
flakes or was something fixed offline (gce / gci / gke)? https://k8s- https://k8s- these turned (mostly) green without k/k and k/test-infra changes.
Huh. Disconcerting, but ok. /close
Closing this issue. <detailsIn response to : Huh. Disconcerting, but ok. Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
All test failed because of not being able to pull image from : I guess this is a bug in , and they fixed it now.
nice catch. thanks
should the CSI images be in our regular repo?
<!-- This form is for bug reports and feature requests ONLY! If you're looking for help check and the . If this may be security issue, please disclose it privately via --Is this a BUG REPORT or FEATURE REQUEST?: /kind feature What happened: I have a daemonsets with 250 replicas and maxUnavailable is 20%. When I try to update its image, the ds will scale down 50 old replicas and wait for SatisfiedExpectations, after that it will scale up 50 new replicas. But if a pod delete failed (node is notready, and pod stuck at terminating status), ds controller will never reach the expectation and just stuck here. What you expected to happen: When a pod stuck at terminating, ds controller should start new pods like deployment controller, instead of waiting for the delete batch finish and ready for next poll. How to reproduce it (as minimally and precisely as possible): Anything else we need to know?: Environment: Kubernetes version (use ): Client Version: version.Info{Major:"1", Minor:"8+", GitVersion:"v1.8.1-8+", GitCommit:"", GitTreeState:"clean", BuildDate:"2017-10-25T13:04:04Z", GoVersion:"go1.8.3", Compiler:"gc", Platform:"linux/amd64"} Cloud provider or hardware configuration: OS (e.g. from /etc/os-release): Kernel (e.g. ): Install tools: Others:
/sig apps /cc
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
/remove-lifecycle stale
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
/remove-lifecycle stale
DaemonSet controller waits for old pods to terminate before creating new ones for best-effort at-most-one pod per node. This is a deliberate design decision.
In most cases, DaemonSets won't tolerate two pods scheduled on the same node at the same time; for example, it's common to communicate with DaemonSet pods via . I'm going to close this ticket as this works as designed. Please feel free to reopen the ticket if you think there are still issues to be resolved.
/reopen
Reopened this issue. <detailsIn response to : /reopen Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
why does the DaemonSet controller wait for all pods 50 to be done, before moving to the NEXT 50 ? I completely understand that we want only at most 1 pod per node. But if a pod deletion fails on the node, move on to the next pod as long as the total unavailable pods are less than maxUnavailable. This will allow us to make progress in the face of bad nodes or failed pod deletions due to failing tear down of bad volumes or other reasons.
/assign
thoughts on this one ?
Any progress on this issue? We also encountered this problem. There are thousands of nodes in our prod cluster and nodes often go down, which cause daemonset rolling update stucked, this problem really bother us.
PTAL
/reopen
You can't reopen an issue/PR unless you authored it or you are a collaborator. <detailsIn response to : /reopen Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/reopen
Reopened this issue. <detailsIn response to : /reopen Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/reopen
You can't reopen an issue/PR unless you authored it or you are a collaborator. <detailsIn response to : /reopen Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
<!-- This form is for bug reports and feature requests ONLY! If you're looking for help check and the . If the matter is security related, please disclose it privately via --Is this a BUG REPORT or FEATURE REQUEST?: /kind bug What happened: Default azure file permission is not suitable for most users, dozens of users complain about this, set as is the best choice here, if users don't want , then they could use mountOptions to override and . What you expected to happen: How to reproduce it (as minimally and precisely as possible): Anything else we need to know?: Environment: Kubernetes version (use ): v1.12 Cloud provider or hardware configuration: OS (e.g. from /etc/os-release): Kernel (e.g. ): Install tools: Others: /sig azure /assign
I ran a deployment with replicas 1 and use to set container's subpath dir, which uses NFS as the storage backend, and create about dirs in container's subPath dir,when delete the pod, it spent much more time to end status, and when the replicas increase to 10 and do the rolling update, at the same time dir nums in containers increase to , it spent about 30 minutes to totally remove pod, which is intolerant. The pod is deleted fast. a pod using subPathExpr as the volume mount. dirs in the backend storage corresponds to the container volume mount the pod kubelet log We can find that it spent about 17s to clean the mount dir in kuelet. No response <details</details<details</details<details</details<details</details<details</details<details</details
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
After some debugging, I inspected the pprof with flame graph when delete a pod, it shows that and cost much more time, here is the graph: ! Apparently the Walk func spent much time when doCleanPath, what we care is the top level dir,and this pr has some fast path to avoid much recursive time, but before gol1.16, the filepath.Walk is still less efficient, because when used in kubelet, it reads all the dir in the mount point before call the callback func in Walk even we don't use it after that, so it spends much time. And after go 1.16, filepath.WalkDir is introduced, which is more efficient than Walk, as the comment said in golang WalkDirFunc is called before reading a directory, so it can save much time when there are lots of dirs in the first level of mount points. So it's time wasting and meaningless when we use Walk so far because we traverse the dir but we didn't use it and return SkipDir err, the code is After I replaced it with , the took only about 41ms, which is a huge boost than 17s before.
When master opens up for , we need to add this back in: /sig node
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
note to self / 1.21 and 1.22 release notes team -- let's make sure the PR body on the original PR is copied over to the new PR. will think that the old release note and documentation has been covered in 1.21, but we will (I will update this comment with link) manually override it to not show up in 1.21 release notes.
Deployment is defined in a json file with several environment varibles. Deployment is deployed using "kubectl apply -f ". Pods get most of the environment variables defined, but not all. Specifically, the variables defined in the are named: { "name": "DBHOSTS", "value": "10.0.0.1" }, { "name": "SLACKENDPOINT", "value": "ERRORNOT" }, { "name": "LACKENDPOINT", "value": "ERRORNOT" }, { "name": "SLACK", "value": "ERRORNOT" }, { "name": "CKENDPOINT", "value": "ERRORNOT" }, { "name": "ACKENDPOINT", "value": "ERRORNOT" }, { "name": "KENDPOINT", "value": "ERRORNOT" }, { "name": "CASSANDRALOGGINGLEVEL", "value": "ERROR" }, { "name": "ENABLEGETIP", "value": "" } the environment variables that appear in a pod are: ENABLEGETIP CASSANDRALOGGINGLEVEL KENDPOINT ACKENDPOINT CKENDPOINT DBHOSTS That is, ENV variables named "SLACKENDPOINT", "LACKENDPOINT" and "SLACK" never get passed on to the pods.
It also won't accept "randomshit" as an environment varible. When I add that as an environment variable to an existing deployment and run "kubctl apply -f " nothing happens. This is on version v1.2.4 /
Ehh? Can you post a full but minimized yaml that I can run to reproduce it? , drashkov wrote:
We use JSON files, here is the full one. "SLACK_ENDPOINT" does not appear as an environment variable in the pods. Doing "kubectl apply -f " does not create new pods, and does not pass these variables to the pods. If I change the number of replicas and add a new pod, that pod will be created, but won't have that varible.
I am not able to reproduce on GKE at 1.2.4 when exec'ing into one of the Pods and printing the environment. This is the yaml I used - note the image is different: What are the symptoms you are seeing?
Can you provide yaml using standard public image along with the specific symptoms you are seeing?
did you create the deployment first with out the env vars, then add them via an apply?
Thanks for looking into it I managed to reproduce it with the yaml you posted. Here's what I did: the yaml you provided as '' the "SLACKENDPOINT" env var from that file and saved that as "" (no other changes) This creates the deployment with all env vars as expected This re-creates a new set of pods and they don't have the SLACKENDPOINT var, as expected. This does not re-create the pods, so the env var is not passed along. kubectl does not return errors.
cc cc
Thanks, I was able to reproduce the issue. It looks like the config with the ENV does get stored in the last-applied-configurations annotation: But does not get put into the pods:
After a few tries I found that if you add the env as the last one you'll be able to add it. Definitely a bug.
The generated three way merge patch wasn't correct (the annotation is good, but look at the spec part): "spec":{"template":{"spec":{"containers":[{"env":[{"name":"CASSANDRALOGGING_LEVEL","value":"ERROR"}],"name":"test-api"}]}}}}
It only recognizes the last env as diff
Opened a PR ptal.
Thanks everyone.
When a Pod admission fails due to node-pressure condition then the Pod's looks like this: "Pod The node had condition: [DiskPressure]." The "Pod The node..." reads awkward. A message like: "The node had condition: [DiskPressure].". a job with a pod on a single node cluster, when the node is under disk pressure (then it is tainted with ). The Pod gets stuck in the phase. the node by kubectl command line: Pod is scheduled but admission by Kubelet fails as the taint is re- shortly after its manual removal. Then, the status of the Pod contains the awkward message. No response <details</details<detailsReproducible on kind </details<details</details<details</details<details</details<details</details
/sig node /triage accepted /priority backlog
The first part () is returned here: . I wonder where the is being appended to the beginning. It's my first try to contribute, so I am a little bit lost.
I think this is a candidate place which might be responsible for the prefix (but haven't checked): I would suggest you to try to reproduce the issue locally first then experiment with modifying the code.
/assign
thanks, I've managed to reproduce and used Delve to make sure who's responsible. I opened the changing the message. If it's accepted it will change to .
ref: /sig scheduling /kind cleanup Part of: The unit test in the scheduling queue only assumes the case where the queueing hint is enabled. We should make another unit test to confirm works without QHint. should have been existing since before we introduced QHint. So, just bring the old one back should be fine.
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/priority important-soon
/assign
Kubernetes 1.3.0. During attach disk to VM, call to cleanUpController dereferenced a null pointer. Stack trace:
The seems points to a comment instead of a cleanUpCotroller function in v1.3.0. There might be some modifications later which make the line number inaccurate. Would you please double check which line it really should point to? Thanks.
that is correct, this was a build from v1.3.0 with changes related to this: therefore the line numbers do no line up with v1.3.0. In either case, the panic happened inside
Hi Davide, I saw your PR for this issue and I have another question. Why cleanUpController could be called with an empty newSCSIController? According to the context, an empty newSCSIController should not even be returned by CreateSCSIController rt? Thanks.
it doesn't look like it should happen, but it did and there wasn't any checks for nil pointers in our code or the Govmomi library.
seen in this inside a like loop we execute a function that return a bool and an error. If we want to fail fast we return the error directly and assert on the output of the wait function. If we want to retry we log the error and return false nil /kind flake /kind bug /sig testing /sig network
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/cc since I think you this test but most probable the code is inherited and is a copy paste thing, but besides that, the change is to remove those there
/assign
<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --What happened: there is a disk resize scenario: In the beginning, it’s 3TiB with cachingMode=ReadOnly, and then disk detached, resized to 4TiB, then user find they could not attach that disk back to any agent node, I think that’s a bug we need to fix. What you expected to happen: How to reproduce it (as minimally and precisely as possible): Anything else we need to know?: Environment: Kubernetes version (use ): Cloud provider or hardware configuration: OS (e.g: ): Kernel (e.g. ): Install tools: Network plugin and version (if this is a network-related bug): Others: /kind bug /assign /priority important-soon /sig cloud-provider /area provider/azure
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
Testgrid: https://k8s- https://k8s- https://k8s- https://k8s- Sample failures: This started failing across multiple test jobs last night. The culprit appears to be PR over to you. /kind failing-test /priority critical-urgent /sig storage
/milestone v1.13
/assign
GitHub didn't allow me to assign the following users: mkimuram. Note that only and repo collaborators can be assigned. For more information please see <detailsIn response to : /assign Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
I think should delete the PVC and wait for it to be deleted.
Thank you for your comment. I haven't yet catching up, but as for the PVC deletion code in , which is changed in the PR, PVC deletion is done inside for delete policy case or for retain policy case. Isn't it enough? In addition, could you let me know how I can check the matrix for the failure that we are focusing? I couldn't find the failure for provisioning tests themselves, there.
Is my understanding right that the purpose of is to check whether resource is leaking by checking the diff of resource between the test? If so, is it correct that the sample output above means that one pd is leaking, not the leak of pv or pvc itself? I found that in the test for retain policy is removed in this PR when it is moved to , because it is gce specific logic and looks unnecessary. Do you think that it could be a cause of this issue? If so, should deleting the backend storage be generally required before removing the dynamic provisioned PV with retain policy?
PriorityPidEvictionOrdering [Slow] [Serial] [Disruptive][NodeFeature:Eviction] when we run containers that should cause PIDPressure should eventually evict all of the correct pods /kind failing-test <!-- If this is a flake, please add: /kind flake --/sig node /assign /priority important-soon
fork-bomb-container-container exited with code 2. I tried to create a pod in my env of containerd: The pod can be running and it works as designed. I need to test it in cri-o later.
/cc
/triage accepted
ref:
identified one problem: firewall is not open for icmp in GCI setup. I opened it manually. Still does not work.
Here are the symptoms: node can ping each other one node, it can ping all pods on it, and vice versa. one node or pod, it can ping all cbr0 gateways(10.X.X.1). pod, it cannot ping but able to curl. It cannot ping any other pod, even on the same bridge, but able to curl.
My first thought would be somethings blockign ping. your options are iptables, firewall, icmp is disabled alltogether (sysctl or somewhere in /proc/sys/net/ there should be a setting). Try traceroute and see where it gets to.
I have a working version now. GCI setup icmp ACCEPT rules in INPUT chain of filter table. But not in OUTPUT and FORWARD chain.
In summary, 2 problems: is not opened for icmp is filtering icmp on FORWARD chain. ACCEPT on INPUT chain is not enough Will submit a fix
Great detective work, Is this a ping-specific problem, or are there going to be problems with other types of outgoing traffic from pods on GCI as well?
I do not see any problem for udp/tcp traffic so far. e2e tests already covers lots of them. Unfortunately, no e2e covers ping tests and GCI happens to filter it.
We should add a test in networking suite for ping.
Is this missing a milestone 1.3?
<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --What happened: ! What you expected to happen: How to reproduce it (as minimally and precisely as possible): Anything else we need to know?: Environment: Kubernetes version (use ): v1.20.1 Cloud provider or hardware configuration: OS (e.g: ): 7 Kernel (e.g. ): Linux zsm-test-2 3.10.0-1160.11.1.el7.x8664 SMP Fri Dec 18 16:34:56 UTC 2020 x8664 x8664 x8664 GNU/Linux Install tools: Network plugin and version (if this is a network-related bug): Others:
/sig node
Do you know how to reproduce this problem?
Yes, can reproduce the problem stably, but the steps are a little complicated.
/triage accepted
Since 2 days ago, the test is becoming more and more flaky, and one of the examples is: https://k8s-
There are no sig labels on this issue. Please by:<br(1) mentioning a sig: <bre.g., for API Machinery<br(2) specifying the label manually: <bre.g., for sig/scalability<br<brNote: method (1) will trigger a notification to the team. You can find the team list and label list
please triage this one. Currently I put it as the release blocker, and please feel free to remove that if you think it is not. Thanks!
The test deletes the node, and at the end, waits 5 minutes for the node to become ready, but it timed out. cc
The test logic hasn't changed after I last modified it, so the test deletes the node through gcloud, and the instance should be restarted automatically by GCE because it's part of an instance template. The likely possibility, as pointed out, is that another change made node discovery take longer, so extending the timeout would be enough. The very unlikely possibility is there's a problem discovering brand new instances.