jiebi/CodeConvo huggingface dataset:
Change:
What happened: Network services with heavy load will cause "connection reset" from time to time. Especially those with big payloads. When packets with sequence number out-of-window arrived k8s node, conntrack marked them as INVALID. kube-proxy will ignore them, without rewriting DNAT. The packet goes back the the original pod, who doesn't recognize the packet because of the wrong source ip, end up RSTing the connection. What you expected to happen: connection not reset How to reproduce it (as minimally and precisely as possible): wrote this just to reproduce the issue: Basically run it in a cluster, and after a while you will be able to see "Connection reset by peer" on both server and client pods. Anything else we need to know?: Related issue: Environment: Kubernetes version (use ): 1.11.7 Cloud provider or hardware configuration: gke (should be general to all provider) OS (e.g: ): PRETTYNAME="Debian GNU/Linux 9 (stretch)" NAME="Debian GNU/Linux" VERSIONID="9" VERSION="9 (stretch)" ID=debian HOMEURL="" SUPPORTURL="" BUGREPORTURL="" Kernel (e.g. ): Linux gke-cluster-1-11-default-pool--6nzv 4.14.89+ SMP Wed Jan 9 13:35:00 PST 2019 x86_64 GNU/Linux
/sig network
I created a small app to reproduce this issue:
I've been hitting this exact problem for packets between a Pod and the outside world. So the DROP rule has to go into the INPUT chain. Would that be a job for a CNI driver or the kube-proxy?
was put to iptables mode but not to , right? Isn't ipvs mode affected by this issue, too?
Can anyone point me out how netfilter ignore this STATE invalid packet? I checked the code in netfilter for function iptablenatdo_chain, but didn't find the code that handles the STATE=invalid packet. Thanks for help.
Invalid is connection tracking state. My understanding is: If for some reason conntrack doesn't think it's the packet is "tracked" it could be grouped into "invalid" state. The problem is once it happens, .
ipvs should be fine. do you see similar issue in ipvs?
I also encountered this problem. Have you solved it
is this a similar issue for ipvs?
The error is a very general networking issue, which could be caused by many different reasons. This specific cause can be reproduced by the It looks like you got an answer already there :)

Discussion:
"k8s.io/klog" <del> "k8s.io/api/core/v1" </del> <ins> v1 "k8s.io/api/core/v1" </ins> "k8s.io/apimachinery/pkg/types" utilversion "k8s.io/apimachinery/pkg/util/version" "k8s.io/apimachinery/pkg/util/wait"

} } <ins> // Drop the packets in INVALID state, which would potentially cause // unexpected connection reset. // https://github.com/kubernetes/kubernetes/issues/74839 writeLine(proxier.filterRules, "-A", string(kubeForwardChain), "-m", "conntrack", "--ctstate", "INVALID", "-j", "DROP", ) </ins> // If the masqueradeMark has been added then we want to forward that same // traffic, this allows NodePort traffic to be forwarded even if the default // FORWARD policy is not accept.
Change:
What happened: I realized csi local driver with the function of IsLikelyNotMountPoint in kubernetes pkg/util/mount library. The function as follows: We notice that If i use the symbolic link filepath, it will return false which cause the error to umount after this function. What you expected to happen: return true How to reproduce it (as minimally and precisely as possible): A directory link A to someone device the function with symbolic link Anything else we need to know?: Environment: Kubernetes version (use ): 1.15 OS (e.g: ): centos Kernel (e.g. ): Linux 3.10.0-862.3.2.el7.x8664 SMP Mon May 21 23:36:36 UTC 2018 x8664 x8664 x8664 GNU/Linux Install tools: yum
/sig storage

Discussion:
// is a mountpoint. // It should return ErrNotExist when the directory does not exist. // IsLikelyNotMountPoint does NOT properly detect all mountpoint types <del> // most notably linux bind mounts. </del> <ins> // most notably linux bind mounts and symbolic link. </ins> IsLikelyNotMountPoint(file string) (bool, error) // DeviceOpened determines if the device is in use elsewhere // on the system, i.e. still mounted.

// IsLikelyNotMountPoint determines if a directory is not a mountpoint. // It is fast but not necessarily ALWAYS correct. If the path is in fact // a bind mount from one part of a mount to another it will not be detected. <ins> // It also can not distinguish between mountpoints and symbolic links. </ins> // mkdir /tmp/a /tmp/b; mount --bind /tmp/a /tmp/b; IsLikelyNotMountPoint("/tmp/b") // will return true. When in fact /tmp/b is a mount point. If this situation // if of interest to you, don't use this function...
Change:
<!-- Please only use this template for submitting enhancement requests --What would you like to be : Enable it in the default storageclass. Why is this needed: Since resizing behavior has to be triggered by user action, I don't see a reason why we shouldn't allow it by default. /assign
There are no sig labels on this issue. Please add a sig label by either: a sig: e.g., to notify the contributor experience sig, OR the label manually: e.g., to apply the label Note: Method 1 will trigger an email to the group. See the . The in method 1 has to be replaced with one of these: bugs, feature-requests, pr-reviews, test-failures, proposals. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
/remove-lifecycle stale

Discussion:
provisioner: kubernetes.io/gce-pd parameters: type: pd-standard <ins> allowVolumeExpansion: true </ins>
Change:
<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --What happened: Providing an empty to the dynamic client's method results in a panic. What you expected to happen: An error should be returned on empty . How to reproduce it (as minimally and precisely as possible): Call with Anything else we need to know?: The expects a nonempty to be provided, but does not validate this. If an empty name is provided, this method will panic because it expects an , but the response will actually be an . Environment: Kubernetes version (use ): Cloud provider or hardware configuration: OS (e.g: ): Kernel (e.g. ): Install tools: Others: client-go 10.0.0
/sig api-machinery

Discussion:
package dynamic import ( <ins> "fmt" </ins> "io" "k8s.io/apimachinery/pkg/api/meta"

return nil, err } name = accessor.GetName() <ins> if len(name) == 0 { return nil, fmt.Errorf("name is required") } </ins> } result := c.client.client.

if err != nil { return nil, err } <ins> name := accessor.GetName() if len(name) == 0 { return nil, fmt.Errorf("name is required") } </ins> outBytes, err := runtime.Encode(unstructured.UnstructuredJSONScheme, obj) if err != nil { return nil, err

result := c.client.client. Put(). <del> AbsPath(append(c.makeURLSegments(accessor.GetName()), subresources...)...). </del> <ins> AbsPath(append(c.makeURLSegments(name), subresources...)...). </ins> Body(outBytes). SpecificallyVersionedParams(&opts, dynamicParameterCodec, versionV1). Do()

if err != nil { return nil, err } <ins> name := accessor.GetName() if len(name) == 0 { return nil, fmt.Errorf("name is required") } </ins> outBytes, err := runtime.Encode(unstructured.UnstructuredJSONScheme, obj) if err != nil {

result := c.client.client. Put(). <del> AbsPath(append(c.makeURLSegments(accessor.GetName()), "status")...). </del> <ins> AbsPath(append(c.makeURLSegments(name), "status")...). </ins> Body(outBytes). SpecificallyVersionedParams(&opts, dynamicParameterCodec, versionV1). Do()

} func (c *dynamicResourceClient) Delete(name string, opts *metav1.DeleteOptions, subresources ...string) error { <ins> if len(name) == 0 { return fmt.Errorf("name is required") } </ins> if opts == nil { opts = &metav1.DeleteOptions{} }

} func (c *dynamicResourceClient) Get(name string, opts metav1.GetOptions, subresources ...string) (*unstructured.Unstructured, error) { <ins> if len(name) == 0 { return nil, fmt.Errorf("name is required") } </ins> result := c.client.client.Get().AbsPath(append(c.makeURLSegments(name), subresources...)...).SpecificallyVersionedParams(&opts, dynamicParameterCodec, versionV1).Do() if err := result.Error(); err != nil { return nil, err

} func (c *dynamicResourceClient) Patch(name string, pt types.PatchType, data []byte, opts metav1.PatchOptions, subresources ...string) (*unstructured.Unstructured, error) { <ins> if len(name) == 0 { return nil, fmt.Errorf("name is required") } </ins> result := c.client.client. Patch(pt). AbsPath(append(c.makeURLSegments(name), subresources...)...).
Change:
Looks like CSI volume limit test is flaking. /sig storage /assign
This flake isn't happening in normal clusters, appears to be alpha only issue.

Discussion:
const ( csiNodeLimitUpdateTimeout  = 5 * time.Minute <del> csiPodUnschedulableTimeout = 2 * time.Minute </del> <ins> csiPodUnschedulableTimeout = 5 * time.Minute </ins> ) var _ = utils.SIGDescribe("CSI mock volume", func() {

// define volume limit to be 2 for this test var err error <del> init(testParameters{nodeSelectorKey: "node-attach-limit-csi", attachLimit: 2}) </del> <ins> nodeSelectorKey := fmt.Sprintf("attach-limit-csi-%s", f.Namespace.Name) init(testParameters{nodeSelectorKey: nodeSelectorKey, attachLimit: 2}) </ins> defer cleanup() nodeName := m.config.ClientNodeName attachKey := v1.ResourceName(volumeutil.GetCSIAttachLimitKey(m.provisioner))
Change:
Which jobs are failing: pull-kubernetes-conformance-image-test job Reason for failure: This job creates a Pod and Pod goes from being pending into "Pod status is: Failed". Please see for example the following PR to see that the job is stuck in pending state.
/assign /sig testing

Discussion:
if [[ "$STATUS" == "Succeeded" ]]; then echo "$timestamp Done." break <ins> elif [[ "$STATUS" == "Failed" ]]; then echo "$timestamp Failed." break </ins> else sleep 5 fi
Change:
Which jobs are failing: Which test(s) are failing: Since when has it been failing: 2019-03-07, in master, in 1.14 Testgrid link: /sig testing /priority critical-urgent /milestone v1.14 cc
/milestone v1.14
/remove-sig testing /sig api-machinery Related: , ,
cc It looks like this started failing right after this merge: (commit )
fix in
/reopen For CI Signal to verify that things look green once we got more job runs
Reopened this issue. <detailsIn response to : /reopen For CI Signal to verify that things look green once we got more job runs Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
All green :tada: /close
Closing this issue. <detailsIn response to : All green :tada: /close Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details

Discussion:
ret=1 fi <del> if ! _out="$(diff -Naupr -x "BUILD" -x "AUTHORS*" -x "CONTRIBUTORS*" vendor "${_kubetmp}/vendor")"; then </del> <ins> if ! _out="$(diff -Naupr -x "BUILD" -x "zz_generated.openapi.go" -x "AUTHORS*" -x "CONTRIBUTORS*" vendor "${_kubetmp}/vendor")"; then </ins> echo "Your vendored results are different:" >&2 echo "${_out}" >&2 echo "Godeps Verify failed." >&2
Change:
<!-- Please only use this template for submitting reports about failing tests in Kubernetes CI jobs --Which jobs are failing: Which test(s) are failing: Since when has it been failing: It has been consistently failing since 3/6, flaking a lot before that. Failure coincides with . Testgrid link: Reason for failure: cc /kind flake /kind failing-test /priority important-soon /milestone v1.14 /sig cluster-lifecycle
sig gcp may be of help here /sig gcp
/assign for routing from SCL side. These are google owned tests.
Looking into this now. Any other googlers also doing so feel free to ping me :-)
Looks like this bug is known with a fix in flight: cc
/assign
Thanks ! Will keep an eye on this for 5 consistent passes, will reopen if there's a problem.
/reopen Most gke jobs are able to create clusters now, however the serial jobs are still seeing failures
Reopened this issue. <detailsIn response to : /reopen Most gke jobs are able to create clusters now, however the serial jobs are still seeing failures Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
Nevermind, it's because the serial job with the fix is still running. I've verified that the cluster creation succeeds in the latest job. /close
Closing this issue. <detailsIn response to : Nevermind, it's because the serial job with the fix is still running. I've verified that the cluster creation succeeds in the latest job. /close Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/area platform/gke <img width="1106" alt="Screen Shot 2019-03-16 at 12 00 52 PM" src="https://user-"Think we're done here

Discussion:
if utilfeature.DefaultFeatureGate.Enabled(features.CSIDriverRegistry) { csiClient := host.GetKubeClient() if csiClient == nil { <del> return errors.New("unable to get Kubernetes client") </del> <ins> klog.Warning(log("kubeclient not set, assuming standalone kubelet")) } else { // Start informer for CSIDrivers. factory := csiapiinformer.NewSharedInformerFactory(csiClient, csiResyncPeriod) p.csiDriverInformer = factory.Storage().V1beta1().CSIDrivers() p.csiDriverLister = p.csiDriverInformer.Lister() go factory.Start(wait.NeverStop) </ins> } <del> // Start informer for CSIDrivers. factory := csiapiinformer.NewSharedInformerFactory(csiClient, csiResyncPeriod) p.csiDriverInformer = factory.Storage().V1beta1().CSIDrivers() p.csiDriverLister = p.csiDriverInformer.Lister() go factory.Start(wait.NeverStop) </del> } // Initializing the label management channels
Change:
<!-- Please only use this template for submitting reports about failing tests in Kubernetes CI jobs --Which jobs are failing: /sig storage /kind flake
/assign The fake watch is not set up correctly.

Discussion:
fakeClient = fakeclient.NewSimpleClientset() } fakeWatcher := watch.NewRaceFreeFake() <del> fakeClient.Fake.AddWatchReactor("*", core.DefaultWatchReactor(fakeWatcher, nil)) </del> <ins> fakeClient.Fake.PrependWatchReactor("volumeattachments", core.DefaultWatchReactor(fakeWatcher, nil)) </ins> host := volumetest.NewFakeVolumeHostWithCSINodeName( tmpDir, fakeClient,
Change:
<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --What happened: pod could not be terminated when pod volume path is corrupted, and it looks like it mainly affect flexvolume, not sure What you expected to happen: How to reproduce it (as minimally and precisely as possible): a pod with smb mount (we use flexvol in this case, driver is ) smb server password the pod could not be terminated forever Anything else we need to know?: Environment: Kubernetes version (use ): master Cloud provider or hardware configuration: OS (e.g: ): Kernel (e.g. ): Install tools: Others: /sig storage /assign cc
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
Workaround on agent node, run “mount | grep cifs” to get all cifs mounts check every cifs mount, failed (in above example), run: And then the pod will be in terminated soon.

Discussion:
} func (f *flexVolumeUnmounter) TearDownAt(dir string) error { pathExists, pathErr := mount.PathExists(dir) <del> if !pathExists { klog.Warningf("Warning: Unmount skipped because path does not exist: %v", dir) return nil } if pathErr != nil && !mount.IsCorruptedMnt(pathErr) { return fmt.Errorf("Error checking path: %v", pathErr) </del> <ins> if pathErr != nil { // only log warning here since plugins should anyways have to deal with errors klog.Warningf("Error checking path: %v", pathErr) } else { if !pathExists { klog.Warningf("Warning: Unmount skipped because path does not exist: %v", dir) return nil } </ins> } call := f.plugin.NewDriverCall(unmountCmd)
Change:
On Kubernetes v1.13.4 nodes with the vSphere Cloud Provider I see a lot of log spam (15k messages in one hour, with many duplicate messages logged just microseconds apart) like this one: This MAC address belongs to the interface which is (afaik) created by kube-proxy running in IPVS mode. I see the same message for the MAC address belonging to the interface created by Flannel. I see a few issues here: vSphere Cloud Provider tries to detect IP addresses for kube-proxy and CNI network interfaces; the MAC address does not have a known VMware prefix this gets logged as a warning instead of a debug message; process is apparently retried thousands of times per hour and in quick succession.
/sig vmware
/sig network These SIGs are my best guesses for this issue. Please comment if I am incorrect about one. I am an (alpha) bot run by ‍
/remove-sig network
We are experiencing the same even though we fixed the ip address using flag on the kubelet.
Hello, when updating from v1.12.5 to v1.12.6, the following message appears: This MAC corresponds to the Docker NIC: This error does not appear in version v1.12.5
Hi, Same issue here. Upgrading our cluster to v1.12.6 we have this message. In previous version (v.1.12.5) everything was fine
This is the relevant PR that was backported to v1.11.7, v1.12.6 and v1.13.4:
/assign
/area cloudprovider
Opened a PR to fix this:
I've created cherry pick PRs to fix this in v1.11, v1.12 and v1.13 as well: v1.11: v1.12: v1.13:

Discussion:
) klog.V(4).Infof("Detected local IP address as %q", ipnet.IP.String()) } else { <del> klog.Warningf("Failed to patch IP as MAC address %q does not belong to a VMware platform", vmMACAddr) </del> <ins> klog.V(4).Infof("Failed to patch IP for interface %q as MAC address %q does not belong to a VMware platform", i.Name, vmMACAddr) </ins> } } }
Change:
<!-- Please only use this template for submitting reports about failing tests in Kubernetes CI jobs --/sig testing /sig windows /area conformance Which jobs are failing: N/A Which test(s) are failing: [sig-storage] HostPath should support r/w [NodeConformance] [sig-storage] HostPath should support subPath [NodeConformance] Since when has it been failing: Before 2019-03-07. Since then, the Windows nodes' kubelet service has been passed the , which allows the test to pass. Testgrid link: (older than 2019-03-07) Reason for failure: The tests are trying to spawn pods with a HostPath volume ( folder has 777 file permissions by default, meaning it should pass without privileged access. The tests have turned green in the linked testgrid because the Windows Kubelet flag ` has been set to true, but it shouldn't have been necessary in the first place. Anything else we need to know:
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale

Discussion:
source := &v1.HostPathVolumeSource{ Path: "/tmp", } <del> pod := testPodWithHostVol(volumePath, source, true) </del> <ins> // we can't spawn privileged containers on Windows, nor do we need to. privileged := !framework.NodeOSDistroIs("windows") pod := testPodWithHostVol(volumePath, source, privileged) </ins> pod.Spec.Containers[0].Args = []string{ fmt.Sprintf("--new_file_0644=%v", filePath),

source := &v1.HostPathVolumeSource{ Path: "/tmp", } <del> pod := testPodWithHostVol(volumePath, source, true) </del> <ins> // we can't spawn privileged containers on Windows, nor do we need to. privileged := !framework.NodeOSDistroIs("windows") pod := testPodWithHostVol(volumePath, source, privileged) </ins> // Write the file in the subPath from container 0 container := &pod.Spec.Containers[0]
Change:
What happened: How to reproduce it (as minimally and precisely as possible): Anything else we need to know?: Maybe this error is introduced in by one commit (I haven't yet found it). And we need init DefaultProvider when we setup Scheduler in . /sig scheduling /kind bug /assign /cc
This is an interesting issue! I also see it on my machine, but as you pointed out, there is no recent change that could possibly cause this. In fact, we were in code freeze and no PR got merged into the schedule code base in the past ~couple of weeks. I wonder if a change in our build tools are made recently that introduced this.

Discussion:
"//pkg/api/legacyscheme:go_default_library", "//pkg/cloudprovider/providers/gce:go_default_library", "//pkg/scheduler:go_default_library", <ins> "//pkg/scheduler/algorithmprovider/defaults:go_default_library", </ins> "//pkg/scheduler/api:go_default_library", "//pkg/scheduler/factory:go_default_library", "//staging/src/k8s.io/api/core/v1:go_default_library",

"k8s.io/klog" "k8s.io/kubernetes/pkg/api/legacyscheme" "k8s.io/kubernetes/pkg/scheduler" <ins> // import DefaultProvider _ "k8s.io/kubernetes/pkg/scheduler/algorithmprovider/defaults" </ins> schedulerapi "k8s.io/kubernetes/pkg/scheduler/api" "k8s.io/kubernetes/pkg/scheduler/factory" "k8s.io/kubernetes/test/integration/framework"