jiebi/CodeConvo huggingface dataset:
Code change:
git add "${VERSION_FILE}" git commit -m "Kubernetes version ${NEW_VERSION}-dev" <ins> echo "+++ Constructing backmerge branches" function return_to_kansas { git checkout -f "${current_branch}" } trap return_to_kansas EXIT backmerge="v${VERSION_MAJOR}.${VERSION_MINOR}.${VERSION_PATCH}-merge-to-master" backmergetmp="${backmerge}-tmp-$(date +%s)" # Now we create a temporary branch to revert the doc commit, then # create the backmerge branch for the convenience of the user. git checkout -b "${backmergetmp}" git revert "${dochash}" --no-edit git checkout -b "${backmerge}" "${fetch_remote}/master" git merge -s recursive -X ours "${backmergetmp}" -m "${VERSION_MAJOR}.${VERSION_MINOR}.${VERSION_PATCH} merge to master" git checkout "${current_branch}" git branch -D "${backmergetmp}" </ins> echo "" echo "Success you must now:" echo ""

Related discussion:
text: doesn't work with our existing release flow. I thought it did, but, from cc closing out .
Actually, I just realized my last suggestion in isn't the worst. The pull requests to master are essentially only there to get the tags backmerged, so we can construct a release branch like so: and then a release branch backmerge branch like so: and it'll still work.
Code change:
partition = strconv.Itoa(int(pdSource.Partition)) } <del> pv := &v1.PersistentVolume{ </del> <ins> var am v1.PersistentVolumeAccessMode if pdSource.ReadOnly { am = v1.ReadOnlyMany } else { am = v1.ReadWriteOnce } return &v1.PersistentVolume{ </ins> ObjectMeta: metav1.ObjectMeta{ // A.K.A InnerVolumeSpecName required to match for Unmount Name: volume.Name,

Related discussion:
text: When running CSI migration tests focussing on multi-attach scearnios with GCE PD inline volumes that are read-only, the following error pops up in the logs and the second attachment takes 6 mins: For read-only inline volumes, the above warning/error around is unexpected with CSI migration enabled and needs to be root-caused/resolved. What happened: CSI migration tests for read-only inline GCE PD volumes occasionally fails with the above messages in the logs when executing tests like What you expected to happen: The tests and multi-attach scenarios to execute smoothly with no warnings in the logs. How to reproduce it (as minimally and precisely as possible): Execute with focus set to and run a few times. Anything else we need to know?: This has been observed with CSI migration enabled and the tests exercising read-only inline volume scenarios. Environment: Kubernetes version (use ): v1.16 Cloud provider or hardware configuration: GCE OS (e.g: ): Kernel (e.g. ): Install tools: kube- Network plugin and version (if this is a network-related bug): Others:
/cc
/sig storage
Looks like hard coding access modes to ReadWriteOnce during translation for GCE PD at is causing isMultiAttachForbidden to return leading down to We need to set the access mode to ReadOnlyMany for in-tree inline volumes like in GCE PD (as well as others like EBS that support multi-attach) depending on instead of always setting it to ReadWriteOnce.
I can pick up this fix /assign
Code change:
} func (g *gcpAuthProvider) WrapTransport(rt http.RoundTripper) http.RoundTripper { <del> return &oauth2.Transport{ Source: g.tokenSource, Base:   rt, } </del> <ins> return &conditionalTransport{&oauth2.Transport{Source: g.tokenSource, Base: rt}} </ins> } func (g *gcpAuthProvider) Login() error { return nil }

Related discussion:
text: What keywords did you search in Kubernetes issues before filing this one? (If you have found any duplicates, you should instead reply there.): kubectl token BUG REPORT Kubernetes version (use ): kubectl version Client Version: version.Info{Major:"1", Minor:"6", GitVersion:"v1.6.0", GitCommit:"", GitTreeState:"clean", BuildDate:"2017-03-28T16:36:33Z", GoVersion:"go1.7.5", Compiler:"gc", Platform:"darwin/amd64"} Server Version: version.Info{Major:"1", Minor:"6", GitVersion:"v1.6.1", GitCommit:"", GitTreeState:"clean", BuildDate:"2017-04-03T20:33:27Z", GoVersion:"go1.7.5", Compiler:"gc", Platform:"linux/amd64"} Environment: Cloud provider or hardware configuration: GKE OS (e.g. from /etc/os-release): macOS Sierra 10.12.4 Kernel (e.g. ): uname -a Darwin jacobsimpson- 16.5.0 Darwin Kernel Version 16.5.0: Fri Mar 3 16:52:33 PST 2017; root:xnu-3789.51.2~3/RELEASEX8664 x8664 Install tools: Others: What happened: I set up a second user, then I use to authenticate as the second user. The parameter is ignored and the authentication token from the file is used instead, so I'm authenticated as the user configured in the config file, not as the user set on the command line parameters. I further verified this by commenting out the section in the config file with the authentication token and then the parameter starts to work as expected. What you expected to happen: I expect the command line parameters, when present, to override the config file settings. How to reproduce it (as minimally and precisely as possible): At this point, the error message should be indicating kubectl tried to authenticate with the account. Something like this: Now edit , search for the section for your cluster and comment out the whole section for the test cluster. Now it should be possible to run these two commands. The one using the user_token will fail with the error message indicating the user name.
Code change:
} func (g *gcpAuthProvider) WrapTransport(rt http.RoundTripper) http.RoundTripper { <del> return &oauth2.Transport{ Source: g.tokenSource, Base:   rt, } </del> <ins> return &conditionalTransport{&oauth2.Transport{Source: g.tokenSource, Base: rt}} </ins> } func (g *gcpAuthProvider) Login() error { return nil }

Related discussion:
text: Anything else we need to know:
The token arg on the command line overrides the bearer token field in the config file, but I'm not sure the precedence of the token arg and the auth provider in the config file. cc
All auth providers currently override the "Authorization" header even if it's set. Should be an easy-ish check to add. Long term we probably want to have the auth provider stop wrapping the transport and just act like a token source , that way some central code would handle the HTTP headers and we wouldn't have to enforce this on a per plugin basis.
So the tokens from command line should be used, right? if so I can send a pr to fix this.
the client auth plugins shouldn't override the "Authorization" headers. Yeah feel free to send a PR.
Is this a regression or why 1.7?
bug is pretty self contained in the GCP client auth plugin and the fix is short Not strictly a regression, but something that popped up over this cycle as a bug impacting users. Is that reason enough to get it in during the 1.7 freeze?
Code change:
if resizeError != nil { klog.Errorf("MountVolume.NodeExpandVolume failed with %v", resizeError) <ins> // Resize failed. To make sure NodeExpand is re-tried again on the next attempt // *before* SetUp(), mark the mounted device as uncertain. markDeviceUncertainErr := actualStateOfWorld.MarkDeviceAsUncertain( volumeToMount.VolumeName, devicePath, deviceMountPath) if markDeviceUncertainErr != nil { // just log, return the resizeError error instead klog.Infof(volumeToMount.GenerateMsgDetailed( "MountVolume.MountDevice failed to mark volume as uncertain", markDeviceUncertainErr.Error())) } </ins> return volumeToMount.GenerateError("MountVolume.MountDevice failed while expanding volume", resizeError) } }

Related discussion:
text: What happened: A custom CSI driver implementing returns a failure response, indicating that it did not complete its intended steps to prepare the volume for the mounting pod. Let's say that these steps involve a filesystem at the target directory, data from a service, and that data to a set of files in the mounted filesystem. If the first filesystem mounting step succeeds, but either of the subsequent two steps fail, the volume is still not ready for use by the pod. However, if this CSI driver returns a failure response, and the CSI mounter comes back and, before invoking the RPC again, it finds a filesystem already mounted at the target directory, , and ceases further calls to to allow the CSI driver to try again. What you expected to happen: If a CSI driver returns a failure response to a RPC, Kubernetes should honor that declaration of failure and invoke the RPC again, disallowing the mounting pod from starting until this RPC succeeds. Even though it's possible to partially work around the current behavior by having the driver mount the filesystem as late as possible, and then attempting to unmount it if fails to populate the volume with its intended data, doing so makes writing drivers more complicated and forces them to do extra work. How to reproduce it (as minimally and precisely as possible): In a CSI driver (even one that serves just ephemeral, inline volumes), implement to mount a filesystem at the target path, then return a failure response. Confirm that Kubernetes does not invoke again, because there's now a filesystem mounted at the target path. Anything else we need to know?: This topic came up for discussion in the "Kubernetes" Slack team. There, helped investigate this complaint, and to discuss changing this behavior. Environment: Kubernetes version (use ):Cloud provider or hardware configuration: AWS EC2 OS (e.g: ):Kernel (e.g. ):Install tools: kubeadm Others: csi-node-driver-registrar container image tag: v1.0.2
/sig storage
While the plugin should try to clean up as much as possible, I agree that kubelet should not be assuming that if there's a mount point, the publish is successful. NodePublish is supposed to be idempotent so it should theoretically be safe to call it when it's already mounted, although there is the possibility that plugins may have written against the K8s behavior.
Code change:
if resizeError != nil { klog.Errorf("MountVolume.NodeExpandVolume failed with %v", resizeError) <ins> // Resize failed. To make sure NodeExpand is re-tried again on the next attempt // *before* SetUp(), mark the mounted device as uncertain. markDeviceUncertainErr := actualStateOfWorld.MarkDeviceAsUncertain( volumeToMount.VolumeName, devicePath, deviceMountPath) if markDeviceUncertainErr != nil { // just log, return the resizeError error instead klog.Infof(volumeToMount.GenerateMsgDetailed( "MountVolume.MountDevice failed to mark volume as uncertain", markDeviceUncertainErr.Error())) } </ins> return volumeToMount.GenerateError("MountVolume.MountDevice failed while expanding volume", resizeError) } }

Related discussion:
text: For the mount point checking here before calling NodePublishVolume, I am wondering it is just for performance optimization, or it has other purposes. Have concern about some drivers might depend on this check if they are not implemented with idenpotency.
Those drivers would be operating in violation of the specification. I'd prefer that we don't penalize already conforming and yet-to-be written drivers in order to accommodate these existing drivers—if they even exist.
More fallout in last night's testing: I introduced a bug into my driver that caused it to panic during ("Writing that data to a set of files in the mounted filesystem"). My handler function is set up to unmount a successfully mounted volume via if it doesn't complete steps 2 and 3 above, in order to deal with this issue's principal complaint. My driver container would crash due to this panic. The kubelet would call on my driver twice, it would crash twice, but the pod still started running. I'm not yet sure whether or not the filesystem got unmounted successfully via the on the first call, but somehow the pod started despite my handler function never completing successfully. I'm going to try to get some more debugging time on this later today to see if I can figure out what the kubelet was doing there.
Studying my code more, I see that my function only unmounted the mounted directory when the containing handler function was returning an error. When unwinding due to a panic, the named return value of type I was inspecting was still nil. Revising the criteria in that function covers both failure reasons (returning an and unwinding due to a panic). Regardless, this further underscores the problem: If a driver makes it as far as mounting a filesystem at the target directory, but fails afterward—including the process crashing and never responding positively to the RPC—the mounting pod will start. That's likely to then cause the pod to fail, for lack of the files it expected to find mounted.
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
/remove-lifecycle stale
Issues go stale after 90d of inactivity.
Code change:
if resizeError != nil { klog.Errorf("MountVolume.NodeExpandVolume failed with %v", resizeError) <ins> // Resize failed. To make sure NodeExpand is re-tried again on the next attempt // *before* SetUp(), mark the mounted device as uncertain. markDeviceUncertainErr := actualStateOfWorld.MarkDeviceAsUncertain( volumeToMount.VolumeName, devicePath, deviceMountPath) if markDeviceUncertainErr != nil { // just log, return the resizeError error instead klog.Infof(volumeToMount.GenerateMsgDetailed( "MountVolume.MountDevice failed to mark volume as uncertain", markDeviceUncertainErr.Error())) } </ins> return volumeToMount.GenerateError("MountVolume.MountDevice failed while expanding volume", resizeError) } }

Related discussion:
text: Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
/remove-lifecycle stale
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
/remove-lifecycle stale
Code change:
Ref:         ref("k8s.io/apimachinery/pkg/apis/meta/v1.Duration"), }, }, <ins> "tcpBeLiberal": { SchemaProps: spec.SchemaProps{ Description: "tcpBeLiberal, if true, kube-proxy will configure conntrack to run in liberal mode for TCP connections and packets with out-of-window sequence numbers won't be marked INVALID.", Default:     false, Type:        []string{"boolean"}, Format:      "", }, }, </ins> "udpTimeout": { SchemaProps: spec.SchemaProps{ Description: "udpTimeout is how long an idle UDP conntrack entry in UNREPLIED state will remain in the conntrack table (e.g. '30s'). Must be greater than 0 to set.",

Related discussion:
text: What happened: When kube-proxy is configured with a cluster-cidr range, it drops packets with an INVALID conntrack state for all sources on the KUBE-FORWARD chain. This is meant to prevent spurious retransmits in long-lived TCP connections to a service IP, but it prevents clusters from having asymmetrical routing on private subnets with services outside of a cluster. What you expected to happen: Packets on the KUBE-FORWARD chain with an INVALID conntrack state should be dropped for only sources inside of the cluster-cidr if set. How to reproduce it (as minimally and precisely as possible): N/A Anything else we need to know?: Change to drop INVALID conntrack packets was in Backing out that commit and compiling is my current workaround. Environment: Kubernetes version (use ): 1.17.2 Cloud provider or hardware configuration: OS (e.g: ):Kernel (e.g. ):
/sig network
/triage unresolved Comment when the issue is assessed and confirmed. I am a bot run by vllry. ‍
I am reading but I don't fully grok what caused the problem in the first place, or why we think "only drop if it was in-cluster" is a correct fix? Edit, OK I get the original problem, but still don't understand why packets from outside the pod range would not exhibit , thereby requiring yet another revisit of this problem. Is this a conntrack bug? Should it be applying the DNAT?
/triage unresolved Comment when the issue is assessed and confirmed. I am a bot run by vllry. ‍
I can try to give a bit more context to our setup, but the knowledge I have around this all dates back to Feb, unfortuately, so I may have forgotten details. We were replacing our mesos cluster with a kubernetes cluster in Feb and using a custom ingress solution (HAProxy w/an outside process configuring it for pod ips / mesos ip-ports) for both. The ingress nodes lived in the private subnet, but outside of the kubernetes cluster cidr range. When we upgraded from kubernetes 1.14 to kubernetes 1.15 (the first version with PR ) our ingress could no longer talk to any pod ips.
Code change:
Ref:         ref("k8s.io/apimachinery/pkg/apis/meta/v1.Duration"), }, }, <ins> "tcpBeLiberal": { SchemaProps: spec.SchemaProps{ Description: "tcpBeLiberal, if true, kube-proxy will configure conntrack to run in liberal mode for TCP connections and packets with out-of-window sequence numbers won't be marked INVALID.", Default:     false, Type:        []string{"boolean"}, Format:      "", }, }, </ins> "udpTimeout": { SchemaProps: spec.SchemaProps{ Description: "udpTimeout is how long an idle UDP conntrack entry in UNREPLIED state will remain in the conntrack table (e.g. '30s'). Must be greater than 0 to set.",

Related discussion:
text: I discovered that the KUBE-FORWARD chain had a rule to drop invalid packets, which was in Compiling kubernetes 1.15 without this commit fixed our problems, and we have been carrying that patch forward ever since. I would like to get rid of our forked kubernetes, hence this ticket. When I was writing this ticket up, I also found also mentions the same issue with the iptables dropping INVALID conntrack state packets near the end, regarding asymmetrical routing. Edit, OK I get the original problem, but still don't understand why packets from outside the pod range would not exhibit , thereby requiring yet another revisit of this problem. I think you are correct, I can imagine packets from outside of a cluster could exhibit the same behavior as packets inside the cluster and cause spurious resets. However, I am wondering if kube-proxy should be managing packets from outside of the cluster on a node... That seems outside of the scope of kube-proxy.
If this issue has been triaged, please comment . If you aren't able to handle this issue, consider unassigning yourself and/or adding the label. I am a bot run by vllry. ‍
I took a stab at implementing what I think is the correct behavior for kube-proxy in .
/assign
I think that another option is to set
I don't understand how that will help me here. The packets with a ctstate INVALID are not generated due to conntrack saturation (), but due to asymmetric routing in our network. See for an example of a similar setup.
are the packets on your asymmetric routing setup marked as invalid if you set ? it should only mark the ones that are out of window per the description
/triage accepted
I'd love to pin this down, but I don't have the means to set up a repro. Is there a way I can set up a way to experiment?
We were bit by this too after revisiting an old service that had been unused since the heady days of pre-, and would also like to see the proposed patch merged (or a configuration option to prevent insertion of the INVALID state drop rule - either suits us). I unfortunately don't have any useful input as to setting up for reproduction. In our case we have a pretty odd anycast set up with a small pool of (public) addresses managed by calico.
Code change:
Ref:         ref("k8s.io/apimachinery/pkg/apis/meta/v1.Duration"), }, }, <ins> "tcpBeLiberal": { SchemaProps: spec.SchemaProps{ Description: "tcpBeLiberal, if true, kube-proxy will configure conntrack to run in liberal mode for TCP connections and packets with out-of-window sequence numbers won't be marked INVALID.", Default:     false, Type:        []string{"boolean"}, Format:      "", }, }, </ins> "udpTimeout": { SchemaProps: spec.SchemaProps{ Description: "udpTimeout is how long an idle UDP conntrack entry in UNREPLIED state will remain in the conntrack table (e.g. '30s'). Must be greater than 0 to set.",

Related discussion:
text: However I do seem to have a dirty workaround for small clusters. By setting notrack (ctstate UNTRACKED?) in the raw table for packets in our specific ranges that would otherwise match ctstate INVALID, the packets aren't dropped. Which may be of help to someone else who finds this issue.
We are seeing something very similar on large file downloads and our issue exactly fits this one's subject. These RESET (corrected after are happening intermittently and I am seeing similar behavior mentioned in the blog mentioned below. I am unsure if someone has found a workaround for this issue or not, so yeah my +1 to this issue being looked up by the community.
We're using as a solution for the connection reset we were seeing and might fix this issue.
Let me try to avoid that this thread derails and see if I can get current status: -proxy install rules to drop INVALID cstate connections, this solves the ~REJECT~ RESET problems are some scenarios, with asymmetric routing per example, that this rules may drop unrelated traffic. The request is to remove the rule that drops those INVALID states or to try to restrict it, and that can cause that people start to have the ~REJECT~ RESET problems and drops again. If we have a reproducer we can be more precise in the final solution, but removing one rule without any strong evidence in kube-proxy is very risky As a workaround for the asymetric routing problem, I suggested to set the nfconntracktcpbeliberal on the kernel: I think that will solve the problems, it will be good if someone can test it and provide feedback
No sadly, its my bad, it is not REJECT, it is RESET that's happening and conntrack has nothing to do with it, mostly (I hope). Kubeproxy is doing what is intended but specifically when there are external agents in play, the packet doesn't sends an ACK back to the response of SYN-ACK. Our case, we hit A to initiate a download, A calls B to get the action completed to generate download data for the user finally via their browser. A and B both are on k8s, we saw A starts and a complete cycle goes for sometime until there multiple parallel downloads requested where few gets completed and few gets a RESET.