jiebi/CodeConvo huggingface dataset:
Code change:
// newAgnhostPod returns a pod that uses the agnhost image. The image's binary supports various subcommands // that behave the same, no matter the underlying OS. func newAgnhostPod(name string, args ...string) *v1.Pod { <ins> zero := int64(0) </ins> return &v1.Pod{ ObjectMeta: metav1.ObjectMeta{ Name: name, }, Spec: v1.PodSpec{ <ins> TerminationGracePeriodSeconds: &zero, </ins> Containers: []v1.Container{ { Name:  "agnhost",

Related discussion:
text: I don't have full understanding of what exactly is happening, but I already see a couple things that may be causing issues: when creating the first pod, you wait for corresponding endpoint for that being created; but that doesn't mean that kube-proxy was actually programmed correctly at that time (in general it isn't); given iptables programming is rate-limitted, it may easily happen that actually programming happens 10s+ later That explains some delays, but it doesn't explain e.g. this run: where iptables were programmed at but the test didn't respond until If someone more familiar with networking stuff could look into it, it would be great. - can you help triaging ^^ for deletions it's even worse, because you don't even wait for endpoints to be there (i.e. Endpoints can still contain the old pod when the test start pinging to look for the second one) So I claim that timeouts are visibly to low in this test, but I suspect it's not everything. I will send out the first PR that tries to improve it in couple minutes.
thanks wojtek, have been slammed but Ill try to collect more data on it this wknd as well.
It seems that my PR didn't really help much. We should dig deeper, but I would need someone with more networking expertise to look a bit into
Actually - it seems only failures from one group are on non . So maybe that's actually related. The non-kind related ones look like:
OK - for this one, I actually know what is happening - I will send out a PR with a fix for this too.
for the KIND jobs already increased the resource requests on some of the jobs, there are issues on KIND jobs with kube-proxy not able to take the iptables lock and not being able to create the iptables rules for the services until the next resync
is to fix the flakes I mentioned above; all other flakes seem to be kind-oriented to me (which I'm not super familiar with).
We did not adjust the CPU on all jobs yet, pending how to manage that on prow ... I do see just a few other flakes at the bottom of the triage dashboard that look like timeouts instead.
Actually - almost all failures are effectively timeout. The top one that is is actually also a timeout. Didn't look into kind failures, but this may easily be related to iptables issues.
Code change:
// newAgnhostPod returns a pod that uses the agnhost image. The image's binary supports various subcommands // that behave the same, no matter the underlying OS. func newAgnhostPod(name string, args ...string) *v1.Pod { <ins> zero := int64(0) </ins> return &v1.Pod{ ObjectMeta: metav1.ObjectMeta{ Name: name, }, Spec: v1.PodSpec{ <ins> TerminationGracePeriodSeconds: &zero, </ins> Containers: []v1.Container{ { Name:  "agnhost",

Related discussion:
text: After my second deflaking PR, we seem to be in the stage where: tests are failing on dozens of different things (probably iptables-issues related) tests are failing only on NodePort tests on "Failed to connecto to backend 1" which it would be great if someone else could look into (kube-proxy seemed to be programmed correctly based on logs)
If this issue has been triaged, please comment . If you aren't able to handle this issue, consider unassigning yourself and/or adding the label. I am a bot run by vllry. ‍
This has failed again - Linking to specific part of log where timeout occurred .... we see this
So reviewing the above logs note how in STEP: checking client pod connected to the backend 1 on Node IP 10.138.0.5 28 attempts are made all in the 25th second of the 2 minute at 11am Should this Step back off and wait a few seconds in between tries? So looking at this code we see the following
From reading ... the intention expressed in the test code looks good to me. It is not clear to me why there is a mismatch between the 28 tries in the POD Client Logs and the expressed intention running the logContainsFn(podBackend1) every 5 seconds still digging ....
we are trying to catch one failure in this PR that has additional debugging, it dumps the conntrack table of the nodes on a failure
for those following the issue, we've extra debugging to the test so it dumps the whole conntrack table once it finish. We should look that there is NO stale entries, connections from the pod client to the deleted endpoint) and there is conntrack entries to the new endpoint from the pod client.
- we have runs with the conntrack dump now, e.g. this one: would you be able to take a look into it?
not yet :/ , but I think tomorrow can have some time
cc
For anyone reproducing this issue btw, you need a multinode k8s cluster (it isnt doable on kind i guess bc less likeliehood of network failures when running local) -
Curious about two potential hypotheses to fix this, have we looked into either of these? 1) can we confirm that is turned on/off? 2) is it possible that scaling the number of coredns replicas could help ?
Code change:
// newAgnhostPod returns a pod that uses the agnhost image. The image's binary supports various subcommands // that behave the same, no matter the underlying OS. func newAgnhostPod(name string, args ...string) *v1.Pod { <ins> zero := int64(0) </ins> return &v1.Pod{ ObjectMeta: metav1.ObjectMeta{ Name: name, }, Spec: v1.PodSpec{ <ins> TerminationGracePeriodSeconds: &zero, </ins> Containers: []v1.Container{ { Name:  "agnhost",

Related discussion:
text: 3) Would simulating this in kind by overloading coredns be a possibility as a way to make a more reproducible bug model?
regarding 1) kubeadm does not ship nodeLocalDNSCache I don't think GCE kube-up does by default either
ok, so now im curious about (2) - i.e. what if we just bumped coredns replica count?
interesting, :thinking: why do you think is a DNS thing? the netcat command that polls the services use the IP addresses not the hostname. I think that it has to be related to some iptables/conntrack race or similar,at least in KIND jobs we know that we have an iptables problem that affects network tests, that has to be fixed before this because it affect this test You can instrument the test locally with this and dump the state of iptables and conntrack on failure if you are able to reproduce it
i figured DNS bc i believe its known that conntrack and DNS at scale can get very messy, right ?
one other curious thing about this test is that we only create 2 pods in the recycling event for the svc. that likely means that any real error is hidden by chance..... If we created a 3rd pod, then we might potentially double the failure rate thus making the flakyness go away and expose the real underlying issue .
There's an invocation of dumpConntrack(cs) here [sig-network] Conntrack should be able to preserve UDP traffic when server pod cycles for a ClusterIP service
is updating the logging to run in an infinite loop ..
This is improving in KIND with the new changes we rolled out yesterday, let's see how it evolves this week
- which changes ? Are you saying this is no longer reproducible on Kind?
there was a lot of noise in the CI signal, especially in KIND jobs due to the CNI portmap bug we finally solved yesterday now we should have a better signal and we can go to the bottom because probably the failure will be "legit" and not a consequence, so this comment should apply
Hi ! I ran this a few times on kind and it passed, but I see , so I'm going to try the loop approach you suggested. Here's what I'm using to test:
quick note, <- potential motivation to possibly have runnable on some subset of tests in upstream eventually, i.e.
Code change:
// newAgnhostPod returns a pod that uses the agnhost image. The image's binary supports various subcommands // that behave the same, no matter the underlying OS. func newAgnhostPod(name string, args ...string) *v1.Pod { <ins> zero := int64(0) </ins> return &v1.Pod{ ObjectMeta: metav1.ObjectMeta{ Name: name, }, Spec: v1.PodSpec{ <ins> TerminationGracePeriodSeconds: &zero, </ins> Containers: []v1.Container{ { Name:  "agnhost",

Related discussion:
text: so we can do things like to reproduce this sort of thing. matt did you get around to the 'try-till-it-fails' scenario?
I haven't been able to get it to fail on kind yet, -- 100 cycles and 0 failures on a three-node kind cluster! I'm letting the loop run for a few hours, will check back in when I have that data. Is it worth running this test on a non-kind environment?
I ran for several hours on my mac, through 1144 iterations no error yet. My conclusion: this test isn't flaky on KinD. Possible next step: run a large number of iterations of the same test on a GKE cluster.
Flakes on kind are also happening:
not many since the portmap fix on the 14th though ( / ) <img width="1111" alt="Screen Shot 2020-07-23 at 11 45 46 PM" src="https://user-"
Agree - it's much better than it was before.
I was analyzing this failure, the problem is that kind jobs does not have verbosity enough in kube-proxy We can see that the packet was never replied, this is the NAT entry from client -nodePort in the node with the client This is the same flow, but in the node with the server and the pod-server-1 seems to be healthy we need more logs :eyeglasses:
We have an open ticket to bump those in CI, it's just been lost in the noise. We should still do that. Antonio Ojea <:
If you are busy just assign it to me, I'll need this verbosity soon :)
It's in the kind repo, shouldn't be too hard to find, we have ~60 open atm. Antonio Ojea <:
If this issue has been triaged, please comment . If you aren't able to handle this issue, consider unassigning yourself and/or adding the label. I am a bot run by vllry. ‍
I made another attempt to reproduce these failures locally on KinD, but still no success. Here's what I did: create a KinD cluster run two instances of sonobuoy at the same time This took ~4 hours to run; all the kube tests passed and sonobuoy reported all successes.
Code change:
// newAgnhostPod returns a pod that uses the agnhost image. The image's binary supports various subcommands // that behave the same, no matter the underlying OS. func newAgnhostPod(name string, args ...string) *v1.Pod { <ins> zero := int64(0) </ins> return &v1.Pod{ ObjectMeta: metav1.ObjectMeta{ Name: name, }, Spec: v1.PodSpec{ <ins> TerminationGracePeriodSeconds: &zero, </ins> Containers: []v1.Container{ { Name:  "agnhost",

Related discussion:
text: Would love to help out in reproing this, I'm at a loss for good next step so any suggestions welcome!
that's the problem with these flakes, they are not easy to repro, and with this low rate of failures it may be caused for combination of running tests, environmental problems (cpu, memory, ...) , ... I always start for checking the current failures and try to find patterns
Let's take a look to one occurrence The tests fails because can't connect to the first pod If we check the code of the tests, we can see that it always uses the port as source port, we can use that to identify the nodePort assigned interesting thing here, we are source nating the pod ip, and we don't see any other conntrack entry The test verifies it has endpoints however, kube-proxy is using endpointlsices since this version but it has no entries so the iptables rules to forward the traffic are not there Seems we have to adapt this function in the e2e framework to use endpointslices cc: EDIT seems that this function is being used by all the network tests that can cause flakes if the endpoint is updated before the endpointslice (we switched recently kube-proxy to use the later)
Great catch. Ideally this function should validate if both Endpoint and appropriate EndpointSlice objects(s) exist.
Actually - let me take it and send a PR for it.
is out for review
Interesting failures here It fails to connect to the first pod
Hmm - I took a quick look at kube-proxy logs and these look fine to me....
I clicked too fast, sorry ... I was looking at the wrong service and I though I've found something
If this issue has been triaged, please comment . If you aren't able to handle this issue, consider unassigning yourself and/or adding the label. I am a bot run by vllry.
Code change:
// newAgnhostPod returns a pod that uses the agnhost image. The image's binary supports various subcommands // that behave the same, no matter the underlying OS. func newAgnhostPod(name string, args ...string) *v1.Pod { <ins> zero := int64(0) </ins> return &v1.Pod{ ObjectMeta: metav1.ObjectMeta{ Name: name, }, Spec: v1.PodSpec{ <ins> TerminationGracePeriodSeconds: &zero, </ins> Containers: []v1.Container{ { Name:  "agnhost",

Related discussion:
text: ‍
just saw it as well
can you please add links to the job with the failures, I just merged new code to improve the logging 2 days ago
ok, here there is a new occurrence and with the new debug logs is easier to follow has unreplied conntrack entries: looking at the other side The iptables rules are there but no conntrack entries related to 10.244.2.32
In I revert PR with debug info as I think that it likely caused significant memory growth in scalability tests, but we have one run that has finished without being killed by OOMKiller (I increased memory limit for that jobs as a mitigation). Maybe logs from this run will be useful for debugging:
If this issue has been triaged, please comment . If you aren't able to handle this issue, consider unassigning yourself and/or adding the label. I am a bot run by vllry. ‍
Two more flakes today:
It seems that the packet never gets DNATed in the receiving host, so it does not reach the pod in the node. There is the same test that is using ClusterIP instead of NodePort that is more reliable. This is being a tough one
/triage accepted
Yesterday's flake - I don't think I'm seeing the log messages from the latest log PR you made ? Or are the prow logs not logging at a high-enough level?
Logs are there, please bear with me: The error on the test is: click on the job failure and you'll see a larger output where you can see the server pod name and get the node where the pod is running: and the client-pod name and its node: and the flow we are looking for: with the node-name we can get the kube-proxy logs there, clicking in artifacts: and you can see the "new" conntrack log entries : and in the client: wow, there is a kernel bug there , can it be related? :thinking:
wohoo - some progress :) I'm not an expert here, but I think it can be? Adding here - they may be able to confirm/reject the hypothesis.
This kernel panic may be a coincidence, maybe this is the reason
Adding some recent failures:
Spotted another flake today:
Another flake in the 5k job:
Code change:
// NewDelayingQueueWithCustomClock constructs a new named workqueue // with ability to inject real or fake clock for testing purposes func NewDelayingQueueWithCustomClock(clock clock.Clock, name string) DelayingInterface { <ins> return newDelayingQueue(clock, NewNamed(name), name) } func newDelayingQueue(clock clock.Clock, q Interface, name string) *delayingType { </ins> ret := &delayingType{ <del> Interface:       NewNamed(name), </del> <ins> Interface:       q, </ins> clock:           clock, heartbeat:       clock.NewTicker(maxWait), stopCh:          make(chan struct{}),

Related discussion:
text: <!-- Please only use this template for submitting enhancement requests --What would you like to be : Current constructors: Could we export a new constructor like to inject custom queue. Why is this needed: In my project, i want to create a with a custom queue based on workqueue.Interface but have no way to do that.
/sig api-machinery
thanks for the issue! We've reviewed it in the bug triage, and the consensus was that to ask you to send a PR with this and discuss it there. Would you be able to do it? thanks!
Yes, i will do that.
Code change:
t.Errorf("Service must be namespace scoped") } oldService := makeValidService() <del> newService := makeValidService() </del> <ins> oldService.Spec.Type = api.ServiceTypeLoadBalancer </ins> oldService.ResourceVersion = "4" <del> newService.ResourceVersion = "4" </del> <ins> oldService.Spec.SessionAffinity = "None" newService := oldService.DeepCopy() </ins> newService.Spec.SessionAffinity = "ClientIP" newService.Status = api.ServiceStatus{ LoadBalancer: api.LoadBalancerStatus{

Related discussion:
text: xref I think this may be one of the few cases where we can actually change the behavior in a "breaking" way. User creates service with , is empty Controller or user updates the status subresource, sets (not wiped or validated, should be) User updates in any way, is wiped It's clear that the is not meant to be set when is not "LoadBalancer". The question is: 1) Fail validation? 2) Silently discard the value? 3) Warn and then discard the value? 4) Just warn and leave it. Because we allowed it in the past, but it is so volatile, I lean towards 3. and - the people most familair with this type. Edit: PrepareForUpdate() runs before warnings. So: (1) is easy (2) is easy (4) is slightly less easy than (2) (3) is slightly less easy than (4)
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
OMG change it to clusterIP as expected Status is wiped update status as ClusterIP ... and if I edit the service back to loadbalancer it keeps the status??
/kind bug
/retitle Service can be set when !=
We talked about this one at the SIG Network meeting today. suggested we use the feature gate for deprecation option to phase the current (problematic) functionality out.
/triage accept
The label(s) cannot be applied, because the repository doesn't have them. <detailsIn response to : /triage accept Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
Code change:
// +listType=map // +listMapKey=name // +optional <del> Variables []Variable `json:"variables" patchStrategy:"merge" patchMergeKey:"name" protobuf:"bytes,7,rep,name=variables"` </del> <ins> Variables []Variable `json:"variables,omitempty" patchStrategy:"merge" patchMergeKey:"name" protobuf:"bytes,7,rep,name=variables"` </ins> } type MatchCondition v1.MatchCondition

Related discussion:
text: spec.variables should be marked omitempty. I'm trying to create a validating admission policy that uses in v1.27 as follows: I'm setting : to "Strict" in CreateOptions so that the server will reject requests with unknown fields but it complains that the don't exist in v1.27 and it does make sense because this field was introduced in v1.28: I'm using a 1.28 client.
/sig api-machinery
/assign
/assign /triage accepted
Thanks for raising this! /cc
Code change:
}, }, { <del> name: "StorageCapacity changed", </del> <ins> name: "TokenRequests invalidated", </ins> modify: func(new *storage.CSIDriver) { <del> new.Spec.StorageCapacity = &notStorageCapacity </del> <ins> new.Spec.TokenRequests = []storage.TokenRequest{{Audience: gcp}, {Audience: gcp}} </ins> }, }, { <del> name: "TokenRequests invalidated", </del> <ins> name: "invalid nil StorageCapacity", </ins> modify: func(new *storage.CSIDriver) { <del> new.Spec.TokenRequests = []storage.TokenRequest{{Audience: gcp}, {Audience: gcp}} </del> <ins> new.Spec.StorageCapacity = nil </ins> }, }, }

Related discussion:
text: Up to 1.21, that field is immutable. That means in practice, a CSIDriver object has to be deleted and then re-created with a different value instead. Mutation via delete/created is unnecessarily complex and as the only consumer of that field (the kube-scheduler) can handle mutation just fine (it'll always ask the informer cache for the current value), it's better to allow mutation.
/sig storage /assign
This is independent of allowing other fields to become mutable, but that might also make sense.
/triage accepted
Code change:
return iface.Name, nil } <del> func setUpAllInterfaces() error { interfaces, err := net.Interfaces() if err != nil { return err } for _, netIf := range interfaces { setUpInterface(netIf.Name) // ignore errors } return nil } </del> func setUpInterface(ifName string) error { glog.V(3).Infof("Enabling hairpin on interface %s", ifName) ifPath := path.Join(sysfsNetPath, ifName)

Related discussion:
text: From another issue, said: DockerManager.SyncPod() calls the network Plugin and then sets hairpin mode unilaterally. This is wrong. We should either pass the bool down to here or else we should get rid of the default no-op plugin and use the non-existence of a plugin as a flag to control this. This should be a relatively easy one for people who want to contribute but don't know where to start.
I volunteer myself to do this. Actually I have done something about kubernetes network plugin so guess I can do this. I will start to do this when I am on the office tomorrow.
Awesome. I'd advise starting with a sketch so we can all make sure it is a good approach, before jumping in with both feet. , TonyAdo wrote:
ok, I will start with a sketch as soon as possible.
I thought about this for a while: if we pass the bool down to here, of course we can check this flag, if it is set, we should set veth hairpin mode, but problem is that if user set this flag, but still use a custom network plugin just like you mentioned the ovs plugin, I am afraid we will still enable hairpin mode in all interfaces on this host, since we cannot find a peer interface. Is that correct?
Yeah, I think the better fix is to use the existence of a network plugin as an indicator. It sort of makes --configure-cbr0 redundant , TonyAdo wrote:
ok, I will try to get rid of the default no-op plugin and use the non-existence of a plugin as a flag to control this.
I decided to use the existence of a network plugin as an indicator, also I will get ride of the no-op network plugin, I am not going to add new flag to kubelet structure, if the user don't specify network plugin name, the networkPlugin within kubelet will be which just means non-existence of a plugin, and we have to check the existence in the docker manager to avoid nil pointer exception also for setting, I think if a custom network plugin exists, kubelet should not do this. How about this? Should I open a PR?
fyi, saw a hairpin pr today that might be related. ymmv.
what do you think about this?
What you described sounds right to me.