jiebi/CodeConvo huggingface dataset:
Could there be a race with AD controller or external attacher if they try to detach the volume? is another race around this area. Unclear if it's the same yet.
Michelle Au <: No idea. Even if it isn't, it's at least related to the same piece of code. Probably worth fixing both together. Can we get someone who has worked in this area to fix these issues? Today is the first day that I've ever seen this code, so I'm a bit afraid that it'll take me a long time to get it right, and I'm worried about regressions.
I've asked to help investigate
It is not a blocker for beta or freeze. Can be addressed in stabilization. Let's keep the 1.13 tag on it. And based on progress by stabilization we can reassess if it is even a 1.13 blocker (at this point I don't believe it is).
The in the actual state of the world gets reset during unmounting:
That removes the device path looks wrong to me. I've submitted PR which changes that. Let's see how much that breaks...
/assign The conclusion in the CSI working group was that while my analysis and the resulting fix might be correct, further debugging and a second pair of eyes would be good. Let me reassign, because I have done what I could for this issue.
The RT considers this a blocker now, because flakes are happening often enough to interfere with getting clear signal on several jobs. Even if the underlying problem isn't that serious.
/assign I will help triage as well.
Ack Thanks
Instead of which may have negative side effects for non-CSI volumes, let's do this instead: Modify the CSI method in to ignore the passed in (which is populated from devicePath. Instead let's have that method generate the from scratch, e.g. attachID := getAttachmentName(volID, driverName, string(nodeName)) This will remove any CSI dependency on devicePath and not effect other volume plugins.
I am trying to twist my head around the code. IMO it affects also AWS that uses devicePath stored in and it should be reproducible there too. So far I was unsuccessful. In the logs it's clearly visible that UnmountDevice succeeded. So why kubelet continues with MountVolume instead of MountDevice? That's wrong.
A workaround in CSI volume plugin won't really help here, it will work only for drivers without .
I finally reproduced it on AWS with a long sleep in : Steps to reproduce (with the hack above): a pod with a volume the pod. Wait until is called. a new pod with the same volume. - checks that the volume is still attached and the device is still mounted and enqueues an operation to mount the volume. When finishes, starts and 1) uses empty and 2) calls on volume that's not DeviceMounted.
Fix: Any idea how to test it for real? Fake CSI plugin with slow NodeUnstage would be the best, however we don't have one.
I'm going to work on a targeted fix for CSI. We can continue to pursue fixing aws in-tree separately.
PR to fix csi is here:
what are the final set of PRs pending for this? , and ?
Only is needed to fix this particular test failure.
can you please confirm if we are going to fix this using or ?
I discussed with offline, he has some suggestions for that I am investigating. I think we will want both PRs eventually, although only one of them will be needed to fix the CSI test for now. Since it's his off hours, I will continue working on today.
Are we talking 1.13 for as well?
Reopening this since we are seeing consistent failures in latest runs. We also see one more test fail in the job with same error. Failing job https://k8s- Failing Tests [sig-storage] [Serial] CSI Volumes CSI plugin test using CSI driver: gcePD [Serial] should provision storage [sig-storage] [Serial] CSI Volumes CSI plugin test using CSI driver: hostPath [Serial] should provision storage Error This is now 1.13 release blocking.
/priority critical-urgent /kind failing-test
/remove-priority important-soon
I think this is a different issue and could be due to the fact that CSI is intentionally not backwards compatible between 1.12 and 1.13. We may need to enhance the test to figure out what version the cluster is on and launch appropriate versions of the CSI driver.
"intentionally not backwards compatible" -- is there a writeup for the release notes and release announcement for this change?
Yes has the release note
thanks, I'll also call it to the Release Notes lead's attention
what does it mean from the user impact perspective for CSI to not be backward compatible with 1.12? can you please point me to any discussions on this behavior for 1.13?
Kubernetes v1.13 speaks CSI v1.0 and no longer understands CSI v0.x. Any CSI drivers deployed on previous versions of Kubernetes speaking CSI v0.x must removed, updated to be CSI 1.0 compatible, and redeployed on the on Kubernetes v1.13. If only the master is upgraded to 1.13 (and nodes are 1.12 or earlier) older drivers will continue to work. When nodes are upgraded to 1.13, then older drivers will stop working. I will capture this in detail in the "Action Required" section.
Im tracking these upgrade test failures elsewhere: They are not related to this original issue, the flaky test referenced by this issue should have been solved by and for non CSI /close
Closing this issue. <detailsIn response to : Im tracking these upgrade test failures elsewhere: They are not related to this original issue, the flaky test referenced by this issue should have been solved by /close Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
as FYI to call this out clearly in the Release notes.
Runs k8s with master golang: test Failure was observed on Aug 25th, it is passing after that. However it can be reproduced with stress tool. No response No response It is failing on both ppc and x86 architectures. Steps to reproduce: up golang tip version. clone kubernetes master /sig testing
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/assign
it is a sig node test, sig testing charter is not about doing or maintain tests, we need to remove that old mindset of having a QA department, that is not how software development works or should work :) Besides, thanks for spotting it and submittn a patch /sig node /remove-sig testing
Flaky in
merging this PR will fix the issue
It is possible to express a Service where multiple front-side ports target the same back-side port. This is not very useful and may cause trouble for implementations of Services. `
just for the purpose of clarity, are you requesting that duplicate targetPorts should be rejected or the opposite? PS: I am new to Kubernetes so please pardon the dumb question :)
I think that should probably be rejected at validation unless we have a good reason WHY we need that.
Aye aye, thanks for the clarificiation!
/assign
GitHub didn't allow me to assign the following users: xiangpengzhao. Note that only can be assigned. <detailsIn response to : /assign Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
what problems did this cause with services? This change broke backwards compatibility for services that depended on this feature. For example, lets take the load balancer service The load balancer here terminates SSL so the backend is always port 80; however, you want to make sure you support both protocols so having the same targetPort here is a valid use case.
This is an interesting (ab)use of Service. Not really what I had in mind - the LB in a Service is generally assumed to be L4. I'll try to dredge up the issues that this was causing.
FYI, the does non layer4 things as well.
One issue is that this REQUIRES connection tracking when we otherwise might not need it. Consider two ServicePorts: my-service:80 mys-service:443 Let's say we map both services to a backend on port 8080, but we pass-through the VIP, so the pod sees packets as dest = my-service:8080. We MUST conntrack this VIP because we don't know whether to map the src port to 80 or 443. If service backends have unique TargetPorts, then we wouldn't have this issue. Port rewrites are at worst stateless, and at best can engage direct-return mode (no rewrites). It's worth noting that the PR which fixes this, which is proposed for revert, doesn't quite go far enough. It prevents a Service from using a given number twice and from using a given name twice, but not from using a number and a name that resolves to that number.
That is a much trickier (impossible to do statically) proposition. :( Kind of a mess for getting high-perf Service networking.
was this an issue that came up during performance testing or IRL?
Anyone has a workaround to deal with this? I think there is a lot of people using AWS ELBs on this way
if your use-case is Ingress, then you can set . This should avoid generating the invalid Service spec. Obviously, you will lose the redirect behavior.
This is not an abuse of Service, but a quite common and wide-adopted pattern, especially in the AWS world.
you can't re-open an issue/PR unless you authored it or you are assigned to it. <detailsIn response to : /reopen Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
you can't re-open an issue/PR unless you authored it or you are assigned to it. <detailsIn response to : /assign /reopen for further discussion Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/reopen for further discussion.
I suspect we should probably revert the original PR. While I understand it might not have been the intention for Services, if I understand this correctly then it's a breaking change to a v1 API. Can we at least point to any documentation that indicates this was a code bug rather than a supported behavior?
Sentiment on today's sig call was that we should revert the original commit, since it broke lots of users, and then revisit to see if there's another solution.
FYI: the revert PR is
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
Stale issues rot after 30d of inactivity. Mark the issue as fresh with . Rotten issues close after an additional 30d of inactivity. If this issue is safe to close now please do so with .
Reboot tests have been failing since 10/12 Dashboard: https://k8s- Sample Failure: https://k8s- This test failure will block 1.13 beta, so please let's fix it before then. Thanks! /sig cluster-lifecycle /kind failing-test /priority important-soon
Note: this failure is similar to but is not the same failure, according to This is a consistent failure rather than a flake, and all nodes are failing to reboot. It's also been failing consistently since 10/11. /cc
Copying my notes here: Looking at one recent test run: The test failed because kubelet never became "not ready": When looking at one of the kubelet logs, I do see that kubelet had problems making Node updates to the api server: Errors continue until kubelet is restarted 8 minutes later (the test times out at 2 minutes):
So it seems like even though kubelet is unable to update its Node status, the Node is still showing up as Ready
Adding sig-node to this discussion: /sig node
I'm trying to repro on a dev 1.11/1.12 cluster so we can get master logs for the node controller
/cc adding from sig-node
/assign
was the only PR that went in during that time (if we don't count infra or gke changes). This PR touches the node controller and changes the node heartbeat, so it could be the culprit. I was able to grab the logs from the mater was it was still up. Snippet from the kube-controller- showed a lot of error: The PR author is out for a couple of days, so he cannot investigate. I can look at it later today maybe. If anyone from GKE wants to help, I can send you all the logs.
The node controller doesn't seem to be working at all as it wasn't marking the nodes not ready. suggested that the api group may not have been whitelisted in GKE, so I checked and that seemed to be true. We can whitelist it, but for an alpha feature, it should not be properly gated and not affect production. We can revert PR to unblock release, and wait for to re-submit a patched PR. /cc
Thanks for debugging this! Just got time to take a quick look. Does it mean should also be put behind the feature gate?
I think reverting is a good option to unblock stuff. - creating listed may not be necessary, but setting HasSynced is (so if we have to put it after feature gate, we should put both).
Is there any guideline on how to revert my original PR ? I haven't done it before. Just revert in my fork and submit a new PR? I also just put up a fix PR . I am fine with either submitting the fix PR or revert my original PR first.
/reopen /assign I will manually close this after verifying the bots.
Reopening this issue. <detailsIn response to : /reopen /assign I will manually close this after verifying the bots. Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
This is passing in the latest test run. Thanks for the quick turnaround
Nice! Closing this now. /close
Closing this issue. <detailsIn response to : Nice! Closing this now. /close Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --What happened: On running kubetest with external storage driver option with driver name "", encountered error of flag redefinition. The issue happens because: As part of this patch , a flag is defined for number of inline volumes. The driver name is used to construct the flag name. Thus, when running tests on an externally provided driver (in this instance the Persistent Disk CSI driver) with driver name "", a redefinition error happens since the ephemeral test suite code path is executed once for and once for External PD CSI driver. [example run command: kubetest --test --check-version-skew=false --testargs="-- --storage.testdriver=/path/to/gcp-compute-persistent-disk-csi-driver/test/k8s-integration/config/test-" Test config looks something like this: { StorageClass: FromFile: ... SnapshotClass: FromFile: ... DriverInfo: Name: ... } ] What you expected to happen: Test should run and succeed. How to reproduce it (as minimally and precisely as possible): Use an test-, with driver name as "", and run kubetest with focus as "External.Storage" kubetest --test --check-version-skew=false --testargs="-- --storage.testdriver=/path/to/gcp-compute-persistent-disk-csi-driver/test/k8s-integration/config/test-" Anything else we need to know?: Possible fixes options: , (), when using a FromFile or FromExistingClassName option to pass in SnapshotClass information, pickup the provisioner name from the class definition's "driver" field. the flag which has a dependency on driver names. (need to understand the impact of removing the command line flag) Environment: Kubernetes version (use ): latest master Cloud provider or hardware configuration: OS (e.g: ): Kernel (e.g. ): Install tools: Network plugin and version (if this is a network-related bug): Others:
/sig storage
/assign Do you know why this parameter is set as a flag and not a part of the ?
To me, is something that defines what the driver can do. This flag is about what the test does and not related to driver capabilities.
Making it depend on the driver name was probably unnecessary. A single flag for all incarnations of that test probably would have been sufficient and thus better ("keep it simple, stupid!" - the stupid being me here...). It also shouldn't have used the global flag set. In other words:
/reopen I don't really want to get into the habit of defining per test case flags in There are thousands of test cases and it would be unwieldy to discover all the flags available and understand how to use them. What factors would influence how many volumes you want the test to create? Can we just hardcode a value in the test case?
Reopened this issue. <detailsIn response to : /reopen Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
The defaults must be usable so that users do not need to understand them. The flags can be discovered with - not ideal, but at least they show up. That's effectively what the default value for the flag does. This is a way to to run a custom stress test: the default is so low, it just tests that the feature works. By increasing it as high as the underlying CSI driver and storage system supports, one can run this (and potentially other tests) as part of stress testing. Not sure whether it's worth supporting this. It seemed worthwhile at the time, and better than hard-coding just one value.
The bigger issue is: how do users of the test suite actually learn what tests they run and what those tests do? I suspect the answer to that is simply "read the source", in which case they'll also find the knobs that can be used to tweak testing.
Since increasing the parameter depends on the csi driver/storage system, then I would recommending splitting this up into 2 test cases: 1) test basic functionality 2) test stress scenario, and add the parameter to the driver test config similarly to other stress tests:
At least for the storage tests, because we are intending that they be run externally, having all the tunables in one place I think makes it easier to discover.
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close.
If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
Stale issues rot after 30d of inactivity. Mark the issue as fresh with . Rotten issues close after an additional 30d of inactivity. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle rotten
/remove-lifecycle rotten
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
/remove-lifecycle stale /help
This request has been marked as needing help from a contributor. Please ensure the request meets the requirements listed . If this request no longer meets these requirements, the label can be removed by commenting with the command. <detailsIn response to : /remove-lifecycle stale /help Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-contributor-experience at . /lifecycle stale
/remove-lifecycle stale
I saw that the flag was removed in , I think we can close this one if doesn't apply anymore
/close The flag was removed in #diff-20517805986874f69e3254bed93d59996d5a6fd571a70ab8120736ded5aafa24
You can't close an active issue/PR unless you authored it or you are a collaborator. <detailsIn response to : /close The flag was removed in #diff-20517805986874f69e3254bed93d59996d5a6fd571a70ab8120736ded5aafa24 </details
/close
Closing this issue.
<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --What happened: This requires 1.16 alpha feature: Endpoint Slices being enabled. After deleting a , EndpointSliceController kept enqueuing the service after syncing it successfully, until its retry budget exceeded. And the logs are misleading by indicating something wrong happened while the cleanup has been executed successfully: What you expected to happen: The deletion should be triggered only once and there should be no warning and error logs if the cleanup is successfully. How to reproduce it (as minimally and precisely as possible): Endpoint Slices feature a then delete it. kube-controller-manager logs, there would be repeated logs. Anything else we need to know?: Environment: Kubernetes version (use ):Cloud provider or hardware configuration: OS (e.g: ): Kernel (e.g. ): Install tools: Network plugin and version (if this is a network-related bug): Others:
There are no sig labels on this issue. Please add a sig label by either: a sig: e.g., to notify the contributor experience sig, OR the label manually: e.g., to apply the label Note: Method 1 will trigger an email to the group. See the . The in method 1 has to be replaced with one of these: bugs, feature-requests, pr-reviews, test-failures, proposals. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/assign
/assign
Scheduler throughput drops significantly in http://perf- ! After , seems to be the curlspirit.
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/priority critical-urgent /sig scheduling /sig scalability
<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --AppendReason panic when status is nil. Some plugins return nil to mean success. (Or, shouldn't be used on success status? I use field as a reason for why it is success.) not panic or describe this behaviour on the document
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/sig scheduling /assign I think there are some solutions we can take. just add this behaviour to AppendReason document. (most easy solution) Status when status is nil, like below. all plugins not to return nil when success (hard way..) I support the second idea.
I'd expect (and in fact it is) the caller to be responsible to ensure the status is not nil: The 2nd option may work, but semantically should be called by a non-success status.
Hi OK, I see :) I created the small PR to describe that reason field is for non-success status. Please take a look