jiebi/CodeConvo huggingface dataset:
Code change:
// wait until new webhook is called the first time if err := wait.PollImmediate(time.Millisecond*5, wait.ForeverTestTimeout, func() (bool, error) { <del> _, err = client.CoreV1().Pods("default").Patch(reinvocationMarkerFixture.Name, types.JSONPatchType, []byte("[]")) </del> <ins> _, err = client.CoreV1().Pods(markerNs).Patch(marker.Name, types.JSONPatchType, []byte("[]")) </ins> select { case <-upCh: return true, nil

Related discussion:
text: Which jobs are failing: Which test(s) are failing: Since when has it been failing: Testgrid link: Reason for failure: Anything else we need to know: /sig api-machinery /assign
/cc
want to take a look at the audit event issue first? I can dig in after that if issues remain.
yes, I'm looking at the audit event issue now
actually the failure message suggests that the missing audit log is caused by unexpected admission invocation: I think it's unrelated to auditing Note that there are two tests named "no reinvocation of webhook B when in-tree or prior webhook mutations" :( and from the audit log (), it also suggests that the test is hitting a wrong endpoint
Thanks I'll dig in.
Code change:
if err != nil { return false, fmt.Errorf("got error while getting pod events: %s", err) } <del> if len(events.Items) == 0 { return false, nil // no events have occurred yet </del> <ins> for _, event := range events.Items { if strings.Contains(event.Message, msg) { return true, nil } </ins> } <del> return strings.Contains(events.Items[0].Message, msg), nil </del> <ins> return false, nil </ins> } }

Related discussion:
text: <!-- Please only use this template for submitting reports about failing tests in Kubernetes CI jobs --Which jobs are failing: gce-new-master-upgrade-cluster-new-parallel Which test(s) are failing: Varies, but all subpath failures, including: [sig-storage] CSI Volumes [Driver: csi-hostpath-v0] [Testpattern: Dynamic PV (default fs)] subPath should fail if subpath with backstepping is outside the volume [Slow] [sig-storage] CSI Volumes [Driver: csi-hostpath] [Testpattern: Dynamic PV (default fs)] subPath should fail if subpath directory is outside the volume [Slow] [sig-storage] CSI Volumes [Driver: csi-hostpath-v0] [Testpattern: Dynamic PV (default fs)] subPath should fail if subpath file is outside the volume [Slow] [sig-storage] CSI Volumes [Driver: csi-hostpath] [Testpattern: Dynamic PV (default fs)] subPath should fail if subpath file is outside the volume [Slow] ... and pretty much every other subpath test, but never all of them at once. There's also a few other storage tests failing, such as: [sig-storage] Volume expand [Slow] Verify if editing PVC allows resize [sig-storage] Detaching volumes should not work when mount is in progress Since when has it been failing: 11/22 Testgrid link: https://k8s- Reason for failure: These flakes started around the time that merged, but doesn't match up with the exact merge stamp, so it's probably coincidental. The subpath test failures seem to be mostly timeouts: ... so possibly this is just GCE fail. Anything else we need to know: This test job has always been flaky, with around a 40% failure rate. <!-- DO NOT EDIT BELOW THIS LINE --/kind flake /sig storage /priority important-soon cc
can you please help investigate this. It looks to me like the tests are flaking for and . I wonder if the v0 tests are racing with the non-v0 tests for something in this test?
The other tests including for the other CSI driver and non-CSI in-tree plugins are solid green which also makes me think this is a issue with the hostpath tests or driver.
Code change:
if err != nil { return false, fmt.Errorf("got error while getting pod events: %s", err) } <del> if len(events.Items) == 0 { return false, nil // no events have occurred yet </del> <ins> for _, event := range events.Items { if strings.Contains(event.Message, msg) { return true, nil } </ins> } <del> return strings.Contains(events.Items[0].Message, msg), nil </del> <ins> return false, nil </ins> } }

Related discussion:
text: (just adding additional similar flakes here for tracking) Similar flakes in :
adding to the milestone for triage since the flakes look new
These all 4 subpath tests call and it fails in finding a specific failed event by using . As uses to check if the specific error event happens, and , I guess that tests may flake if other failed event happens before the expected one. Also, already wait for the event, we might be able to delete  in . I will create a PR to fix above.
/priority critical-urgent /remove-priority important-soon /milestone v1.13
while we should update the test with your PR, I'm not sure if it's going to fix the failure. Looking at one failing test log, these are the events I see on the failing pod: So it seems to be failing before even reaching the subpath logic.
Timeline of events: Test pod started getting processed by kubelet: It took a minute after the test pod was created for the csi driver to finish registering and mount to succeed, and then subpath event to trigger: So should address the issue. I was initially confused because the test logs didn't show the subpath event for some reason
I opened up to investigate if 1 minute for csi plugin registration is expected
I've a tracker issue for the complete list of storage flakes: This is so that we can track whether we're addressing most of the flakes with the subpath fixes, or if additional fixes are required.
keeping open until flakes are verified cleaned up
/reopen Reopening this issue until we can verify the fix in CI
As of this morning we are seeing CSI "subpath should fail" flakes in the following job Multiple "subpath should fail" flakes in this slow job alone Multiple "subpath should fail" flakes in the 1.13 slow job, similar to master. The following 2 tests are flaky only in this upgrade job can you help us determine if these are cases are flaky tests that you are fixing or is there an underlying CSI bug that needs addressing?
root cause of this is a delay in an external sidecar containers and the tests not waiting long enough to accommodate the delay. There are no changes required in (outside of test) to handle this. These should not be 1.13 blocking. We'll continue to look in to getting them non-flaky today.
Code change:
if err != nil { return false, fmt.Errorf("got error while getting pod events: %s", err) } <del> if len(events.Items) == 0 { return false, nil // no events have occurred yet </del> <ins> for _, event := range events.Items { if strings.Contains(event.Message, msg) { return true, nil } </ins> } <del> return strings.Contains(events.Items[0].Message, msg), nil </del> <ins> return false, nil </ins> } }

Related discussion:
text: In the meantime, we can mark these tests as flaky to reduce the noise, if you'd like. looks like the same issue () is still being hit. See : And logs from (): We need to make the tests handle this condition more gracefully. The suggestion here is one possible solution.
Would be good to capture this as a known issue in release notes.
to followup on capturing this in Release notes
per , can you create a targeted fix increasing the timeout? Looks like marking these tests as flaky will be difficult for 1.13, but if that doesn't fix it we will need to do that.
Short term fix to bump the test timeout:
As per discussion in release burndown meeting, this is not a 1.13 blocker. please move this issue to 1.14 if there are no pending issues you are concerned about
Ok looks like the test still flaked. Looking at , the mount succeeded after a minute and proceeded to subpath processing, which (correctly) failed. However, for some resaon, the test is not seeing the subpath event and still failed after 4 minutes. On kubelet, subpath processing failed after a minute and generated the event: But the test doesn't see it for some reason, and still times out after 4 more minutes:
I saw the same on CSI volume mount succeeded at: Kubelet correctly failed the pod because of subpath But test still failed: Because the test failed to find the pod event. This looks like a test issue. Not a 1.13 blocker.
can you PTAL at this
Is there a way to check if events are overloaded and not able to handle new events? The MountVolume retries while the driver is not registered is very frequent and could have overloaded the event server.
Checking for the existence of an event seems like the wrong way to test this. Would checking pod status alone be insufficient? Ultimately that is what we really care about.
Checking for the specific event guarantees that we are failing with the correct error. But perhaps events are not reliable enough to depend on. Checking if the Pod never came up running would probably reduce the flakes but could potentially mask false negatives. cc for any thoughts.
Code change:
if err != nil { return false, fmt.Errorf("got error while getting pod events: %s", err) } <del> if len(events.Items) == 0 { return false, nil // no events have occurred yet </del> <ins> for _, event := range events.Items { if strings.Contains(event.Message, msg) { return true, nil } </ins> } <del> return strings.Contains(events.Items[0].Message, msg), nil </del> <ins> return false, nil </ins> } }

Related discussion:
text: Tests are still flaking and actually it seems like in general storage flakes in slow are flaking more since 11-21 or so: https://k8s- My theory is that the driver registration delay is causing a flood of events, which causes tests that check for events to fail. I think this is revealing multiple issues in both the tests and product, and we should work on fixing them all, although any one of the fixes should mitigate the test flakes: Driver registrar can sometimes take a minute to come up: Volume manager operation executor doesn't seem to be throttling retries or backing off: Tests should not rely on events: Tests should wait for CSI driver to be registered on all nodes before creating test pods:
and have been merged and will address the biggest issues. has been backported to 1.13 already, backport is still pending.
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
Stale issues rot after 30d of inactivity. Mark the issue as fresh with . Rotten issues close after an additional 30d of inactivity. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle rotten
Rotten issues close after 30d of inactivity. Reopen the issue with . Mark the issue as fresh with . Send feedback to sig-testing, kubernetes/test-infra and/or . /close
Closing this issue. <detailsIn response to : Rotten issues close after 30d of inactivity. Reopen the issue with . Mark the issue as fresh with . /close Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
Code change:
import ( "bytes" <ins> "errors" </ins> "fmt" "math/rand" "net"

Related discussion:
text: introduced somethng that really should be e2e tested.
Would be nice to get this in in the 2 weeks we have after 1.2 branch cut, if no more flakes crop up (but I'm not too optimistic). Bumping to next-release.
/sig network I'm assuming sig-network owns kube-proxy
I thought I noted a change related to this recently, but I may be mistaken.
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. Prevent issues from auto-closing with an comment. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
Stale issues rot after 30d of inactivity. Mark the issue as fresh with . Rotten issues close after an additional 30d of inactivity. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle rotten /remove-lifecycle stale
/lifecycle frozen /remove-lifecycle stale
/remove-lifecycle rotten
Going to try to take this on.
thanks!
I've had some trouble constructing a test for this. should I determine the timeout? There doesn't appear to be an existing mechanism for waiting for service creation. seem to time out rather than be refused - I'm confused by this, as it seems to be the opposite of the intended behavior.
can look at the endpoints + wait for longer than kube-proxy sync time. refused means packets got to their destination but there was nothing listening there. If traffic is blocked, then you will just get a timeout.
I created a service with no matching pods to test manually outside of the e2e suite. # curl test-api curl: (7) Failed to connect to athena-api port 80: Connection timed out # curl --connect-timeout 8 test-api curl: (28) Connection timed out after 8000 milliseconds Looking back at original example in I see that I'm getting the "before" behaviour. An iptables REJECT should send a denial, not a timeout. Am I looking at the wrong thing here? To me this looks like a regression, but I don't think that's likely.
Code change:
import ( "bytes" <ins> "errors" </ins> "fmt" "math/rand" "net"

Related discussion:
text: I have a cluster from HEAD as of 2 weeks ago: So indeed it seems to not work. If I hit that same service IP from the VM itself I immediately get curl: (7) Failed to connect to 10.0.199.54 port 80: Connection refused. If I insert an iptables jump to KUBE-SERVICES at the front of FORWARD, it correctly rejects. This seems in accordance with the iptables flowchart. I'll do some spelunking to see if there's a reason NOT to reject in this case. Vallery Lancey <:
Can you check IPVS mode?
This is why we need tests. Sigh.
Proposed fix is - still needs your e2e :)
Thanks Tim, if I have time I’ll get a PR up tonight.
Sure will check IPVS mode.
Base on my test, IPVS has no such issue.
Code change:
nodeName k8stypes.NodeName, isAttached bool, asw cache.ActualStateOfWorld, <ins> timeout time.Duration, </ins> ) { <del> result := false volumes := asw.GetVolumesToReportAttached(logger) for _, volume := range volumes[nodeName] { if volume.Name == volumeName { result = true </del> <ins> var result bool var lastErr error err := wait.PollUntilContextTimeout(context.TODO(), 50*time.Millisecond, timeout, false, func(context.Context) (done bool, err error) { volumes := asw.GetVolumesToReportAttached(logger) for _, volume := range volumes[nodeName] { if volume.Name == volumeName { result = true } </ins> } <del> } if result == isAttached { t.Logf("Volume <%v> is reported as attached to node <%v>: %v", volumeName, nodeName, result) return </del> <ins> if result == isAttached { t.Logf("Volume <%v> is reported as attached to node <%v>: %v", volumeName, nodeName, result) return true, nil } lastErr = fmt.Errorf("Check volume <%v> is reported as attached to node <%v>, got %v, expected %v", volumeName, nodeName, result, isAttached) return false, nil }) if err != nil { t.Fatalf("last error: %q, wait timeout: %q", lastErr, err.Error()) </ins> } <del> t.Fatalf("Check volume <%v> is reported as attached to node <%v>, got %v, expected %v", volumeName, nodeName, result, isAttached) </del> }

Related discussion:
text: TestRunOneVolumeDetachFailNodeWithReadWriteOnce 4 times since 03-19 23:00 PDT, intermittently ! https://k8s- Still investigating whether this is a tolerable flake or not with more history than testgrid view, or otherwise would appreciate input from anyone with expertise in this test, especially as regards to whether this should be a blocker for tomorrows beta cut. maybe: /sig storage /sig testing
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
is already open for this, looks like it is attempting to fix it
thank you!!! closing this in favor of that one, which I also to our CI signal project board for visibility. For completeness: since it's a longer standing issue, well identified as a flake and not a failure and also actively in progress, NOT considering it a blocker for the go/nogo signal for the beta tomorrow. /close
Closing this issue. <detailsIn response to : thank you!!! closing this in favor of that one, which I also to our CI signal project board for visibility. For completeness: since it's a longer standing issue, well identified as a flake and not a failure and also actively in progress, NOT considering it a blocker for the go/nogo signal for the beta tomorrow. Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
Code change:
nodeName k8stypes.NodeName, isAttached bool, asw cache.ActualStateOfWorld, <ins> timeout time.Duration, </ins> ) { <del> result := false volumes := asw.GetVolumesToReportAttached(logger) for _, volume := range volumes[nodeName] { if volume.Name == volumeName { result = true </del> <ins> var result bool var lastErr error err := wait.PollUntilContextTimeout(context.TODO(), 50*time.Millisecond, timeout, false, func(context.Context) (done bool, err error) { volumes := asw.GetVolumesToReportAttached(logger) for _, volume := range volumes[nodeName] { if volume.Name == volumeName { result = true } </ins> } <del> } if result == isAttached { t.Logf("Volume <%v> is reported as attached to node <%v>: %v", volumeName, nodeName, result) return </del> <ins> if result == isAttached { t.Logf("Volume <%v> is reported as attached to node <%v>: %v", volumeName, nodeName, result) return true, nil } lastErr = fmt.Errorf("Check volume <%v> is reported as attached to node <%v>, got %v, expected %v", volumeName, nodeName, result, isAttached) return false, nil }) if err != nil { t.Fatalf("last error: %q, wait timeout: %q", lastErr, err.Error()) </ins> } <del> t.Fatalf("Check volume <%v> is reported as attached to node <%v>, got %v, expected %v", volumeName, nodeName, result, isAttached) </del> }

Related discussion:
text: But I may have no time to dig into the cause. Let's see.
cc as the test author.
The problem is that this test has a lot of dependency on the sleeps that are hardcoded with a very narrow scope, but our CI seems to be able to stress it much more. As soon as you slightly change one of the sleep duration it starts to flake locally The test can not make an assumption that timing is perfect, even more in the sub second range can someone in sig-storage review this test and make it less dependant on those tight sleeps?
According to most failures, it failed at Line 660. Let me run some stress locally. with 250ms sleep with 100ms sleep
I agree this test case is too brittle due to its specific timing dependencies. To fix this, I suggest reworking the test case to: 1) Add retry with timeout to wait for final conditions 2) Remove the intermediate timing-based state transition checks. Instead, we can keep track of state transitions in the Fakes and validate those stats. wdyt?
Friendly ping from 1.27 CI Signal that this test is still a noisy flake on the master-blocking dashboard as CI Signal has flagged it in and
Code change:
"github.com/google/go-cmp/cmp" "k8s.io/apimachinery/pkg/util/json" "k8s.io/apimachinery/pkg/util/sets" <ins> "k8s.io/apimachinery/pkg/util/wait" </ins> "k8s.io/apimachinery/pkg/util/waitgroup" auditinternal "k8s.io/apiserver/pkg/apis/audit" "k8s.io/apiserver/pkg/audit"

Related discussion:
text: <!-- Please only use this template for submitting reports about flaky tests or jobs (pass or fail with no underlying change in code) in Kubernetes CI --Which jobs are flaking: bazel-test-master Which test(s) are flaking: gotest: Testgrid link: Reason for failure: Anything else we need to know: links to appreciated [Will update shortly] links to specific failures in spyglass appreciated <!-- Please see the deflaking doc () for more guidance! --/sig api-machinery
The label(s) cannot be applied, because the repository doesn't have them <detailsIn response to : Which jobs are flaking: bazel-test-master Which test(s) are flaking: gotest: Testgrid link: Reason for failure: links to appreciatedlinks to specific failures in spyglass appreciated /sig api-machinary Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/sig api-machinery
/assign /triage accepted
Flaked twice ~2 weeks ago but now looking good: Since it's been relatively quiet, I'm going to go ahead and close this for now. /close
Closing this issue. <detailsIn response to : Flaked twice ~2 weeks ago but now looking good: Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/reopen /assign I meet this test failure in And I can reproduce it with local stress testing. The reason is the same with
Reopened this issue. <detailsIn response to : /reopen /assign Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
Code change:
// ExcludeMasterFromStandardLB excludes master nodes from standard load balancer. // If not set, it will be default to true. ExcludeMasterFromStandardLB *bool `json:"excludeMasterFromStandardLB" yaml:"excludeMasterFromStandardLB"` <ins> // DisableOutboundSNAT disables the outbound SNAT for public load balancer rules. // It should only be set when loadBalancerSku is standard. If not set, it will be default to false. DisableOutboundSNAT *bool `json:"disableOutboundSNAT" yaml:"disableOutboundSNAT"` </ins> // Maximum allowed LoadBalancer Rule Count is the limit enforced by Azure Load balancer MaximumLoadBalancerRuleCount int `json:"maximumLoadBalancerRuleCount" yaml:"maximumLoadBalancerRuleCount"`

Related discussion:
text: <!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --What happened: is the recommended way to configure the outbound networks together with Azure standard loadbalancer (SLB). However, for load balancer rules are disabled by default. That means all frontend IPs would be used as the SNAT IPs for virtual machines, according to . This would introduce troubles for various security scenarios, as there's no way to get a stable list of egress IPs. Azure cloud provider should support this scenario, and disable the outbound SNAT when outbound rules are configured by provisioning tools. What you expected to happen: How to reproduce it (as minimally and precisely as possible): Anything else we need to know?: Environment: Kubernetes version (use ): Cloud provider or hardware configuration: OS (e.g: ): Kernel (e.g. ): Install tools: Others: /kind bug /sig azure /priority critical-urgent