jiebi/CodeConvo huggingface dataset:
Code change:
} return nil } <ins> // ClearEntriesForPortNAT uses the conntrack tool to delete the contrack entries // for connections specified by the {dest IP, port} pair. // Known issue: // https://github.com/kubernetes/kubernetes/issues/59368 func ClearEntriesForPortNAT(execer exec.Interface, dest string, port int, protocol v1.Protocol) error { if port <= 0 { return fmt.Errorf("Wrong port number. The port number must be greater then zero") } parameters := parametersWithFamily(utilnet.IsIPv6String(dest), "-D", "-p", protoStr(protocol), "--dport", strconv.Itoa(port), "--dst-nat", dest) err := Exec(execer, parameters...) if err != nil && !strings.Contains(err.Error(), NoConnectionToDelete) { return fmt.Errorf("error deleting conntrack entries for UDP port: %d, error: %v", port, err) } return nil } </ins>

Related discussion:
text: <!-- This form is for bug reports and feature requests ONLY! If you're looking for help check and the . --Is this a BUG REPORT or FEATURE REQUEST?: BUG REPORT What happened: When deleting/recreating pods backing a UDP service, traffic from external senders stops flowing. What you expected to happen: Traffic would continue to flow to new pods. How to reproduce it (as minimally and precisely as possible): Based on reading of similar issues and commit messages it seems like this would be most noticable on high-volume UDP applications, statsd and syslog are good examples. a deployment and service for a UDP app. In my test case, this would be the container. an external node, create a small test script utilizing a statsd library such as pystatsd. The most important part is that the sender uses the same source-port for every packet sent. Ex: the sender. Install tcpdump into the pod and watch for incoming packets, example from the test case: the pod tcpdump in the newly created pod. Observe that no new matching packets are flowing. existing conntrack rules on all nodes in the cluster. You should see traffic start flowing to the pod again. example command for clearing conntrack rules on the nodes: Anything else we need to know?: I have tested this only on GKE-managed clusters that utilize google forwarding-rules for external LBs, but I believe the issue may be present on any platform where iptables mode for kube-proxy is used. Tested and verified on 1.8.6-gke.0 and 1.7.11-gke.1. Also tested with both Google Container-optimized OS and the Ubuntu-based node images. I have only test this on GKE It appears this was a known issue and should have been cleared up by: From my reading of , it seems as if the func should exec() to clear up UDP entries on service endpoint change, but the only place I see that func called is from syncProxyRules(). That func looks complex and possibly like it is only run occasionally and not on service/endpoint changes. Environment: Kubernetes version (use ): (Also tested on 1.7.11-gke.1) Cloud provider or hardware configuration: GKE-managed clusters, google cloud. OS (e.g.
Code change:
} return nil } <ins> // ClearEntriesForPortNAT uses the conntrack tool to delete the contrack entries // for connections specified by the {dest IP, port} pair. // Known issue: // https://github.com/kubernetes/kubernetes/issues/59368 func ClearEntriesForPortNAT(execer exec.Interface, dest string, port int, protocol v1.Protocol) error { if port <= 0 { return fmt.Errorf("Wrong port number. The port number must be greater then zero") } parameters := parametersWithFamily(utilnet.IsIPv6String(dest), "-D", "-p", protoStr(protocol), "--dport", strconv.Itoa(port), "--dst-nat", dest) err := Exec(execer, parameters...) if err != nil && !strings.Contains(err.Error(), NoConnectionToDelete) { return fmt.Errorf("error deleting conntrack entries for UDP port: %d, error: %v", port, err) } return nil } </ins>

Related discussion:
text: from /etc/os-release): tried both: and Kernel (e.g. ): and Install tools: Others:
/sig network
We also ran into this in kubernetes v1.7.6 using OpenShift 3.7:
~I believe this is the same as ~If so, it is fixed in 1.10: These are about HostPort, not NodePort, sorry for the noise.
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
Is there any other known workaround for this besides updating conntrack rules on the kube nodes? I tried deleting and recreating the service, but this didn't update the rules. When I restart the stream on the client side data flows correctly again (perhaps it's perceived as a new connection), but I have to restart the stream for each sender.
/remove-lifecycle stale
FWIW, this is not fixed in 1.10 - that fix was for HostPort, and this is still a bug for NodePort.
I am seeing this issue in 1.11. I am using LoadBalancer type.
has been working on a fix for this, FYI.
This issue is extremely impactful for telephony protocols that rely on static source ip and source port. Any update on a possible workaround? I was thinking a potential workaround might be to reduce the UDP timeout to something really short so the stale conntrack entries go away faster. Or maybe there is a way to flush conntrack on the event of pod deletion? Edit: Removed stupid idea of a workaround to disable conntrack for UDP. Can't disable conntrack for UDP because it's used for DNAT from node to container.
The workaround is to not use NodePorts. Other kinds of services are already correct.
not sure what you mean. I am seeing the issue with a LoadBalancer type so it does not affect NodePorts only. As a workaround I have created a small python script that watches the K8s API for pod deletions and clears the related conntrack entries when that happens. I can share it when I get to work on Monday if you want to try it out.
Code change:
} return nil } <ins> // ClearEntriesForPortNAT uses the conntrack tool to delete the contrack entries // for connections specified by the {dest IP, port} pair. // Known issue: // https://github.com/kubernetes/kubernetes/issues/59368 func ClearEntriesForPortNAT(execer exec.Interface, dest string, port int, protocol v1.Protocol) error { if port <= 0 { return fmt.Errorf("Wrong port number. The port number must be greater then zero") } parameters := parametersWithFamily(utilnet.IsIPv6String(dest), "-D", "-p", protoStr(protocol), "--dport", strconv.Itoa(port), "--dst-nat", dest) err := Exec(execer, parameters...) if err != nil && !strings.Contains(err.Error(), NoConnectionToDelete) { return fmt.Errorf("error deleting conntrack entries for UDP port: %d, error: %v", port, err) } return nil } </ins>

Related discussion:
text: I've my script . Note that it's not tested much so don't use it in a production environment. Also it cleans up all conntrack entries for a deleted pod, not only the UDP related ones.
We are also seeing this issue with type loadbalancer. We are on 1.11. Our use-case is the following: multiple devices are sending to an aggreagator machine. This machine forwards the UDP packages to a loadBalancer in gcp. When debugging the service, I noticed the following: The traffic is assigned to one pod, and the packages are not loadbalanced (client and server are the same cfr use-case) When redeploying a new cluster, Conntrack is cleared on termination signal of the old pod. (client receives connection refused) red/black and highlander deployment strategies have the same effect. How would I expect this to be working then? New cluster is . Conntrack is cleared Data is routed to NEW pods old pods are terminated Best Regards, Tim
I am seeing this exact same issue on our cluster which has k8s v1.14.5 . We have endpoint nginx pods backing a UDP service with nodeport. Suddenly one day we see traffic not hitting into nginx pod but reaching the node hosting the nginx pod. We could simulate the problem by restarting the nginx pods. I am assuming with this issue marked as closed; and having made its way into v1.14; we should not be seeing this. Would also like to point out; the one thing, in addition we have in the service is ExternalTrafficPolicy set to Local. Can anyone please guide/help on why we are seeing this in v.1.14.5 of our cluster.
Seeing this with k8s v1.16.5
Code change:
importpath = "k8s.io/kube-aggregator/pkg/apiserver", deps = [ "//vendor/github.com/golang/glog:go_default_library", <del> "//vendor/k8s.io/api/core/v1:go_default_library", </del> "//vendor/k8s.io/apimachinery/pkg/api/errors:go_default_library", "//vendor/k8s.io/apimachinery/pkg/apimachinery/announced:go_default_library", "//vendor/k8s.io/apimachinery/pkg/apimachinery/registered:go_default_library",

Related discussion:
text: <!-- This form is for bug reports and feature requests ONLY! If you're looking for help check and the . --Is this a BUG REPORT or FEATURE REQUEST?: /kind bug What happened: Currently APIServiceRegistrationController watch service change and find the related APIServices related if any. Then it AddAPIService/RemoveAPIService for APIAggregator. I dig into the kube-aggregator and find it no need totally, because update APIServices status when related service or endpoint change. So if does not watch services, it will not miss any event for APIServices. What you expected to happen: How to reproduce it (as minimally and precisely as possible): Anything else we need to know?: Environment: Kubernetes version (use ): Cloud provider or hardware configuration: OS (e.g. from /etc/os-release): Kernel (e.g. ): Install tools: Others:
/cc
/kind enhancement /sig api-machinery
/assign
that does not follow. AvailableConditionController modifies the APIService object. APIServiceRegistrationController must still watch the APIService objects to observe those changes.
oh, I misread the description. You're saying APIServiceRegistrationController doesn't need to watch v1.Service objects
Code change:
if len(psp.Spec.AllowedFlexVolumes) > 0 { w.Write(LEVEL_1, "Allowed FlexVolume Types:t%sn", flexVolumesToString(psp.Spec.AllowedFlexVolumes)) } <ins> if len(psp.Spec.AllowedUnsafeSysctls) > 0 { w.Write(LEVEL_1, "Allowed Unsafe Sysctls:t%sn", sysctlsToString(psp.Spec.AllowedUnsafeSysctls)) } if len(psp.Spec.ForbiddenSysctls) > 0 { w.Write(LEVEL_1, "Forbidden Sysctls:t%sn", sysctlsToString(psp.Spec.ForbiddenSysctls)) } </ins> w.Write(LEVEL_1, "Allow Host Network:t%tn", psp.Spec.HostNetwork) w.Write(LEVEL_1, "Allow Host Ports:t%sn", hostPortRangeToString(psp.Spec.HostPorts)) w.Write(LEVEL_1, "Allow Host PID:t%tn", psp.Spec.HostPID)

Related discussion:
text: /kind bug What happened: does not show the description for the new field in What you expected to happen: should describe the new field. How to reproduce it (as minimally and precisely as possible): Environment: Kubernetes version (use ): master
/sig auth
GitHub didn't allow me to assign the following users: fanzhangio. Note that only and repo collaborators can be assigned. <detailsIn response to : /assign Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
If nobody has started yet, I wanna to pick it up.
was working on it AFAIR.
Indeed. if you don't mind, I'd take over.
Code change:
// applyPlatformSpecificDockerConfig applies platform-specific configurations to a dockertypes.ContainerCreateConfig struct. // The containerCleanupInfo struct it returns will be passed as is to performPlatformSpecificContainerCleanup <del> // after either: //   * the container creation has failed //   * the container has been successfully started //   * the container has been removed // whichever happens first. </del> <ins> // after either the container creation has failed or the container has been removed. </ins> func (ds *dockerService) applyPlatformSpecificDockerConfig(*runtimeapi.CreateContainerRequest, *dockertypes.ContainerCreateConfig) (*containerCleanupInfo, error) { return nil, nil } // performPlatformSpecificContainerCleanup is responsible for doing any platform-specific cleanup <del> // after either: //   * the container creation has failed //   * the container has been successfully started //   * the container has been removed // whichever happens first. // Any errors it returns are simply logged, but do not prevent the container from being started or // removed. </del> <ins> // after either the container creation has failed or the container has been removed. </ins> func (ds *dockerService) performPlatformSpecificContainerCleanup(cleanupInfo *containerCleanupInfo) (errors []error) { return }

Related discussion:
text: What would you like to be : Periodically remove cleanupInfos for old containers from dockershim's map. One way of doing that would be to just forget about s older than say an hour, as they should always be short-lived. Why is this needed: introduced a new in-memory map for the dockershim, that could potentially (in pathological cases) cause memory leaks - for containers that use GMSA cred specs, get created successfully, but then never get started nor removed. This might be okay for the alpha release of the GMSA feature, but should be addressed in the beta. See
/assign wk8
/sig windows
/milestone v1.15
got moved out. let's move this issue with it as well /milestone v1.16
Hello! I'm bug triage lead for the 1.16 release cycle and considering this issue is tagged for 1.16, but not updated for a long time, I'd like to check its status. The code freeze is starting on August 29th (about 1.5 weeks from now), which means that there should be a PR ready (and merged) until then. Do we still target this issue to be fixed for 1.16?
: yup we'd love to have this in 1.16, still waiting for kind reviewers on :) ...
Code change:
utilerrors "k8s.io/apimachinery/pkg/util/errors" serviceapi "k8s.io/kubernetes/pkg/api/v1/service" <ins> "github.com/Azure/azure-sdk-for-go/arm/compute" </ins> "github.com/Azure/azure-sdk-for-go/arm/network" "github.com/Azure/go-autorest/autorest/to" "github.com/golang/glog"

Related discussion:
text: <!-- This form is for bug reports and feature requests ONLY! If you're looking for help check and the . --Is this a BUG REPORT or FEATURE REQUEST?: /kind feature What happened: In certain large cluster scenarios we would like to anticipate kubelet behavior hitting request limits, and backoff in response. E.g., What you expected to happen: This is a request for add'l backoff coverage, so nothing surprising. How to reproduce it (as minimally and precisely as possible): Build a cluster with as many nodes as your subscription will support on azure until kubelet behavior contributes to backend API request limits. Environment: Kubernetes version (use ): and Cloud provider or hardware configuration**: azure OS (e.g. from /etc/os-release): Ubuntu 16.04.2 LTS (Xenial Xerus) Kernel (e.g. ): Linux k8s-pool7--7 4.4.0-83-generic -Ubuntu SMP Mon Jun 26 17:54:43 UTC 2017 x8664 x8664 x86_64 GNU/Linux Install tools: acs-engine Others:
There are no sig labels on this issue. Please by: a sig: e.g., to notify the contributor experience sig, OR the label manually: e.g., to apply the label Note: Method 1 will trigger an email to the group. You can find the group list and label list . The in the method 1 has to be replaced with one of these: bugs, feature-requests, pr-reviews, test-failures, proposals
/sig azure
Code change:
fs.IntVar(&s.HardPodAffinitySymmetricWeight, "hard-pod-affinity-symmetric-weight", api.DefaultHardPodAffinitySymmetricWeight, "RequiredDuringScheduling affinity is not symmetric, but there is an implicit PreferredDuringScheduling affinity rule corresponding "+ "to every RequiredDuringScheduling affinity rule. --hard-pod-affinity-symmetric-weight represents the weight of implicit PreferredDuringScheduling affinity rule.") <ins> fs.MarkDeprecated("hard-pod-affinity-symmetric-weight", "This option was moved to the policy configuration file") </ins> fs.StringVar(&s.FailureDomains, "failure-domains", api.DefaultFailureDomains, "Indicate the "all topologies" set for an empty topologyKey when it's used for PreferredDuringScheduling pod anti-affinity.") fs.MarkDeprecated("failure-domains", "Doesn't have any effect. Will be removed in future version.") leaderelection.BindFlags(&s.LeaderElection, fs)

Related discussion:
text: ( noticed this) HardPodAffinitySymmetricWeight should be in the , not in KubeSchedulerConfiguration If/when fixing this, we need to be careful to do it in a backward-compatilble way.
Do you mean ? It seems scheduler policy config's args can not passed to FitPredicates; it need to update related interface. For the : mark to be deprecated and set scheduler policy config to HardPodAffinitySymmetricWeight if policy config is empty.
Are you working on this? if no, I can take it.
, nop; please go ahead.
/ / / , When reviewing the PR, there's one question on : the default value of is 1, which means the weight (1) will always add to priorities if term matched (), does it our expectation? And the user has to set it to 0 to disable it. IMO, the default value of should be 0.
The value 1 (instead of 0) was chosen so that if you say A must run with B, there is an automatic preference that B should run with A. This seems like reasonable behavior.
Thanks very much; reasonable case to me. But it seems we can not disable it: if set to 0, the default funcs will set it to default (1); similar result to un-set. Anyway, no user complain this now :).
Code change:
// Run begins watching and syncing daemon sets. func (dsc *DaemonSetsController) Run(workers int, stopCh <-chan struct{}) { defer util.HandleCrash() <ins> glog.Infof("Starting Daemon Sets controller manager") controller.SyncAllPodsWithStore(dsc.kubeClient, dsc.podStore.Store) </ins> go dsc.dsController.Run(stopCh) go dsc.podController.Run(stopCh) go dsc.nodeController.Run(stopCh)

Related discussion:
text: The test is in flaky suite and no one seems to care. We need an owner for this test. cc Last failure log:
do you think you can take a look at this?
This is happening because of a bad assumption. The assumption is: if the reflector's store is consistent with etcd, the store given back to clients of the framework.Controller is too. I'd really like a general fix applicable to all controllers for this problem, but fixing it just for the RCs is pretty easy. In more detail: This function reports when the reflector has listed from apiserver. But the store given to the reflector is the fifo of the framework.Controller. The store given back to the caller is still not consistent, and will not be till the framework.Controller puts all the objects from the fifo into the client state store (). The rc specific fix: Before starting the rc manager, seed the expectations store with an expectation like {rc:foo, adds: foo.Status.Replicas, deletes:0}. This will wait for status.Replicas number of ADD events till a timeout. The general fix: Somehow expose this through framework.Controller.HasSynced (clients don't care about the fifo anyway, shouldn't it just be an implementation detail?) ideas? cc'ing people who have written controllers that might be interested I don't think this (and generally anything flaky for 1 release) is a p0, I can fix it if you don't mind dealing with the flakes till I've cleared things off my plate. I'd be more than happy to review a fix.
Well, it's a P0 because test is flaky. E.g last run:
- did you have chance to look into it?
The "DaemonRestart" tests are still labeled at . I will send a PR to promote them.
are you still planning to send aPR to promote the DaemonRestart tests out of flaky? I think this is the right thing to do.
Ack; sorry, I lost this in the muck.
Code change:
}) It("should write files of various sizes, verify size, validate content", func() { <del> fileSizes := []int64{1 * framework.MiB, 100 * framework.MiB} </del> <ins> fileSizes := []int64{fileSizeSmall, fileSizeMedium} </ins> fsGroup := int64(1234) podSec := v1.PodSecurityContext{ FSGroup: &fsGroup,

Related discussion:
text: The "Volume plugin streaming [Slow] NFS should write files of various sizes, verify size, validate content" e2e test is failing on our 2k-node gce clusters (https://k8s-) with the following error: cc
Could you take a look into it or reassign as apt? This is part of the correctness suite and large cluster tests are release-blocking.
/assign
is this run using the kubemark environment? I'll also take a look. It could be that the commands this test needs to execute may not work with kubemark. /assign
cc
No, this is not kubemark. This is on a real 2000-node cluster.
Also all other volume-related e2e tests are passing, except this one.
Looking at the log I see that the nfs test fails trying to verify the test file's content. The test exec's into the nfs client pod and executes: The return code from above is 127. The test then deletes the nfs client pod calling which works. The nfs-server pod is deleted in the AfterEach. For some reason the reported error is 137 but the returned err 127. The same command is exec'd for all of the tested storage plugins so not sure why it's failing just for nfs. Error 127 might be "command not found".
The other strange thing is that it's only failing in this suite and not the others.
When I look at the kubelet log, I see the exec command coming through: I also see these suspicious kubelet soon afterwards: So I think the grep command is causing the node to run out of memory and get evicted. I think we can fix this either by: Making the grep command more memory efficient. I'm guessing it's reading the whole file into memory or, Make this test Serial so that it doesn't run in parallel with other pods, since it consumes a lot of memory.
An unrelated issue, is that the node allocatable feature didn't actually evict the pod. Instead this pod got a system OOM kill. /cc
related to the memcg notifications, not node allocatable. Thanks!
any theories why we see this only for the nfs-io tests, and not the glusterfs-io test for instance?
the gluster test didn't run on this in this suite.
Code change:
}) It("should write files of various sizes, verify size, validate content", func() { <del> fileSizes := []int64{1 * framework.MiB, 100 * framework.MiB} </del> <ins> fileSizes := []int64{fileSizeSmall, fileSizeMedium} </ins> fsGroup := int64(1234) podSec := v1.PodSecurityContext{ FSGroup: &fsGroup,

Related discussion:
text: I'm suspecting that the reason why we didn't see a problem with this test on other suites is that the load could be higher here. I've asked to see if we can reduce the memory usage of the grep call.
pointed out that there are no newlines in the generated file, so when grep is called it could buffer the entire file in memory. Adding a newline at the end of the dd input file should do it.
/priority critical-urgent
[MILESTONENOTIFIER] Milestone Labels Complete Issue label settings: sig/scalability sig/storage: Issue will be escalated to these SIGs if needed. priority/critical-urgent: Never automatically move out of a release milestone; continually escalate to contributor and SIG through all available channels. kind/bug: Fixes a bug discovered during the current release. <detailsAdditional instructions available <a href=""here</a</details
It seems like even if the generated file is broken up by newlines, grep could still use on the order of hundreds of MB of memory for a 100MiB file (even with the -m1 and -F options). We could check the file line by line, but it would take a long time. A better solution would be to check the MD5 file hash. It's fast and uses little memory. The expected hashes will be hard-coded into the test, and if different file sizes are in the future, a hash needs to be computed and as well.
Thanks a lot for fixing this!
Ref