jiebi/CodeConvo huggingface dataset:
Code change:
# This seems to matter to some tools export GO15VENDOREXPERIMENT=1 <ins> } kube::golang::setup_gomaxprocs() { </ins> # GOMAXPROCS by default does not reflect the number of cpu(s) available # when running in a container, please see https://github.com/golang/go/issues/33803 <del> if ! command -v ncpu >/dev/null 2>&1; then # shellcheck disable=SC2164 pushd "${KUBE_ROOT}/hack/tools" >/dev/null GO111MODULE=on go install ./ncpu # shellcheck disable=SC2164 popd >/dev/null </del> <ins> if [[ -z "${GOMAXPROCS:-}" ]]; then if ! command -v ncpu >/dev/null 2>&1; then # shellcheck disable=SC2164 pushd "${KUBE_ROOT}/hack/tools" >/dev/null GO111MODULE=on go install ./ncpu || echo "Will not automatically set GOMAXPROCS" # shellcheck disable=SC2164 popd >/dev/null fi if command -v ncpu >/dev/null 2>&1; then GOMAXPROCS=$(ncpu) export GOMAXPROCS kube::log::status "Set GOMAXPROCS automatically to ${GOMAXPROCS}" fi </ins> fi <del> GOMAXPROCS=${GOMAXPROCS:-$(ncpu)} export GOMAXPROCS kube::log::status "Setting GOMAXPROCS: ${GOMAXPROCS}" </del> } # This will take binaries from $GOPATH/bin and copy them to the appropriate

Related discussion:
text: fails in build environments without network connection due to missing in-tree vendored dependency . Build to succeed with without needing to download any go dependencies. In a network-less build environment, run . No response Building off of the v1.28.0-rc.0 tag N/A <details</detailsN/A N/A N/A
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/sig release
This is a hack/tools dependency here by , he may have more context
hack/tools aren't vendored. if you want to ensure those are available offline, will do that
My hack for building with OBS (), see
Is this actually a bug?
in I agree we probably don't want to calls in the build path... in code generation or verify tools, that's more ok, but the build target of a particular SHA should be able to execute hermetically
opened to scope down when we do this to test targets, and avoid the install if the env already explicitly set GOMAXPROCS
fix merged to master and head of release-1.28 branch
Code change:
export SECONDARY_RANGE_NAME="pods-default" export STORAGE_BACKEND="etcd3" export STORAGE_MEDIA_TYPE="application/vnd.kubernetes.protobuf" <del> export ETCD_IMAGE=3.5.0-rc.0-0 export ETCD_VERSION=3.5.0-rc.0 </del> <ins> export ETCD_IMAGE=3.5.0-0 export ETCD_VERSION=3.5.0 </ins> # Upgrade master with updated kube envs "${KUBE_ROOT}/cluster/gce/upgrade.sh" -M -l

Related discussion:
text: /sig scalability This issue is tracking process of upgrading etcd to 3.5.0 planned for 1.22 K8s release. Etcd has recently released Beta versions and is starting the process of correctness and performance testing. To reduce change of finding performance regression late, we will start qualifying Beta Etcd releases on GKE master branch and will upgrade to 3.5.0 as soon as it is available. Steps: [x] Do preliminary scale testing of etcd v3.5.0-beta.3 release [x] Release etcd v3.5.0-beta.3 image (, ) [x] Release etcd v3.5.0-rc.0 image () [x] Upgrade etcd server to v3.5.0-rc.0 () [x] Wait for 3.5.0 release and upgrade etcd client () [x] Release etcd v3.5.0 image (, ) [x] Upgrade etcd server to v3.5.0 /milestone v1.22
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/area etcd
Does this meant that the 3.5.0 should be promoted like
Yes, created
/area release-eng release-team /sig release For tracking: cc:
The label(s) cannot be applied, because the repository doesn't have them. <detailsIn response to : /area release-eng release-team /sig release cc: Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
I do think this can be pointed in the Major Themes for the 1.22 release. I would like your opinions on this. If you do think it has a place in it, I will add it to the .
How can I be a contributor to this issue? what are the technical checklist that needs to be achieved to close out on the issue?
Code change:
export SECONDARY_RANGE_NAME="pods-default" export STORAGE_BACKEND="etcd3" export STORAGE_MEDIA_TYPE="application/vnd.kubernetes.protobuf" <del> export ETCD_IMAGE=3.5.0-rc.0-0 export ETCD_VERSION=3.5.0-rc.0 </del> <ins> export ETCD_IMAGE=3.5.0-0 export ETCD_VERSION=3.5.0 </ins> # Upgrade master with updated kube envs "${KUBE_ROOT}/cluster/gce/upgrade.sh" -M -l

Related discussion:
text: PR to update Etcd is ready, but we found a blocker for merging . There is one test failing I will take a look into it on Monday, still more eyes on the problem could speed up resolution.
We'll also need a new kube-cross image w/ etcd v3.5.0. I'm working on that here:
We just need to upgrade the server to use new image like in
ok, looks like is busy. Let me take a look /assign
Thanks.
Code change:
tags = append(tags, tag) } <del> tagRequest := &ec2.CreateTagsInput{} tagRequest.Resources = []*string{&groupID} tagRequest.Tags = tags if _, err := s.createTags(tagRequest); err != nil { // Not clear how to recover fully from this; we're OK because we don't match on tags, but that is a little odd return "", fmt.Errorf("error tagging security group: %v", err) </del> <ins> if len(tags) > 0 { tagRequest := &ec2.CreateTagsInput{} tagRequest.Resources = []*string{&groupID} tagRequest.Tags = tags if _, err := s.createTags(tagRequest); err != nil { // Not clear how to recover fully from this; we're OK because we don't match on tags, but that is a little odd return "", fmt.Errorf("error tagging security group: %v", err) } </ins> } return groupID, nil }

Related discussion:
text: Getting the following error when creating a service with type when kubernetes has been deployed without tagging the vpc/node. This prevents from using ELBs with the AWS cloud provider when deploying kubernetes in an existing vpc (which cannot be tagged because they already have).
I also experienced this failure when trying to provision Kubernetes on AWS with kube-up and specifying an explicit, existing VPC_ID — which wasn't tagged, but then I didn't know that was required :)
I see we have a pr, can we also update the docs in time for 1.2.0?
sure, what do you want me to update in the docs exactly?
Code change:
name = "go_default_test", srcs = [ "aws_ebs_test.go", <ins> "azure_disk_test.go", </ins> "gce_pd_test.go", ], embed = [":go_default_library"],

Related discussion:
text: <!-- Please only use this template for submitting enhancement requests --What would you like to be : Azure is not listed as "migratable" in could refer to Why is this needed:
/sig azure /assign
Code change:
"source":                   source, }.AsSelector().String() eventListOptions = metav1.ListOptions{FieldSelector: selector} <del> ginkgo.By("Create the test log file") framework.ExpectNoError(err) </del> ginkgo.By("Create config map for the node problem detector") _, err = c.CoreV1().ConfigMaps(ns).Create(context.TODO(), &v1.ConfigMap{ ObjectMeta: metav1.ObjectMeta{Name: configName}, Data:       map[string]string{path.Base(configFile): config}, }, metav1.CreateOptions{}) framework.ExpectNoError(err) <ins> ginkgo.By("Create the service account for node problem detector") _, err = f.ClientSet.CoreV1().ServiceAccounts(f.Namespace.Name).Create(context.TODO(), &v1.ServiceAccount{ ObjectMeta: metav1.ObjectMeta{ Name:      name, Namespace: ns, }, }, metav1.CreateOptions{}) framework.ExpectNoError(err) ginkgo.By("Bind the cluster role system:node-problem-detector with the service account for node problem detector") _, err = f.ClientSet.RbacV1().ClusterRoleBindings().Create(context.TODO(), &rbacv1.ClusterRoleBinding{ ObjectMeta: metav1.ObjectMeta{ Name: name, }, RoleRef: rbacv1.RoleRef{ APIGroup: rbacv1.GroupName, Kind:     "ClusterRole", Name:     "system:node-problem-detector", }, Subjects: []rbacv1.Subject{ { Kind:      rbacv1.ServiceAccountKind, Name:      name, Namespace: ns, }, }, }, metav1.CreateOptions{}) framework.ExpectNoError(err) </ins> ginkgo.By("Create the node problem detector") hostPathType := new(v1.HostPathType) <del> *hostPathType = v1.HostPathType(string(v1.HostPathFileOrCreate)) </del> <ins> *hostPathType = v1.HostPathFileOrCreate </ins> pod := f.PodClient().CreateSync(&v1.Pod{ ObjectMeta: metav1.ObjectMeta{ Name: name, }, Spec: v1.PodSpec{ <del> HostNetwork:     true, SecurityContext: &v1.PodSecurityContext{}, </del> <ins> HostNetwork:        true, SecurityContext:    &v1.PodSecurityContext{}, ServiceAccountName: name, </ins> Volumes: []v1.Volume{ { Name: configVolume,

Related discussion:
text: <detailsIn response to : Thanks for your report, I believe the reason is admission controller has been disabled in node e2e tests: We need to figure out a new approach now. Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
filed . I hope we could completely address this issue this time.
/milestone v1.20
Code change:
if err != nil { return nil, err } <ins> // Avoid collector collects it as a timestamped metric // See PR #95210 and #97006 for more details. machineInfo.Timestamp = time.Time{} </ins> klet.setCachedMachineInfo(machineInfo) imageBackOff := flowcontrol.NewBackOff(backOffPeriod, MaxContainerBackOff)

Related discussion:
text: <!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --What happened: prometheus cadvisor machine metrics from kubelet are missing What you expected to happen: How to reproduce it (as minimally and precisely as possible): install k8s 1.19, compare behavior to k8s <1.19 Anything else we need to know?: Looks like missing call to in , machine metrics collection was split out in cadvisor 0.37 Environment: Kubernetes version (use ):
/sig node (hope that's the correct sig :)
Suggest from as an alternative (declared as "stable").
Related PR Their output format changed too:
Multiple queries is invalide now. Is there any workaroud for this? Please help. Thanks.
if you run see for an alternative metric.
+1 on the issue. This issue has caused the node metric collection for Azure Monitor for Containers to be broken which also impacts alerting.
Exposed metrics are a kind of an API and should not be removed without warning.
any news, when this will be fixed? Thanks in advance!
It is not possible to have a metric available without exposing it. To provide stability information on metrics, similar to APIs, you might want to read up on the . There currently are no metrics in Kubernetes itself that are considered stable, but SIG Instrumentation hopes to be selecting some for the 1.21 release. This particular issue seems like an unintended bug as opposed to an intentional deprecation. Suspect this issue got lost in SIG Node's backlog. I'll put it on the agenda for the next SIG Instrumentation meeting. /sig instrumentation
Backports in and
Code change:
conntrack: maxPerCore: 32768 min: 131072 <ins> tcpBeLiberal: false </ins> tcpCloseWaitTimeout: 1h0m0s tcpEstablishedTimeout: 24h0m0s udpStreamTimeout: 0s

Related discussion:
text: What happened: When kube-proxy is configured with a cluster-cidr range, it drops packets with an INVALID conntrack state for all sources on the KUBE-FORWARD chain. This is meant to prevent spurious retransmits in long-lived TCP connections to a service IP, but it prevents clusters from having asymmetrical routing on private subnets with services outside of a cluster. What you expected to happen: Packets on the KUBE-FORWARD chain with an INVALID conntrack state should be dropped for only sources inside of the cluster-cidr if set. How to reproduce it (as minimally and precisely as possible): N/A Anything else we need to know?: Change to drop INVALID conntrack packets was in Backing out that commit and compiling is my current workaround. Environment: Kubernetes version (use ): 1.17.2 Cloud provider or hardware configuration: OS (e.g: ):Kernel (e.g. ):
/sig network
/triage unresolved Comment when the issue is assessed and confirmed. I am a bot run by vllry. ‍
I am reading but I don't fully grok what caused the problem in the first place, or why we think "only drop if it was in-cluster" is a correct fix? Edit, OK I get the original problem, but still don't understand why packets from outside the pod range would not exhibit , thereby requiring yet another revisit of this problem. Is this a conntrack bug? Should it be applying the DNAT?
/triage unresolved Comment when the issue is assessed and confirmed. I am a bot run by vllry. ‍
I can try to give a bit more context to our setup, but the knowledge I have around this all dates back to Feb, unfortuately, so I may have forgotten details. We were replacing our mesos cluster with a kubernetes cluster in Feb and using a custom ingress solution (HAProxy w/an outside process configuring it for pod ips / mesos ip-ports) for both. The ingress nodes lived in the private subnet, but outside of the kubernetes cluster cidr range. When we upgraded from kubernetes 1.14 to kubernetes 1.15 (the first version with PR ) our ingress could no longer talk to any pod ips.
Code change:
conntrack: maxPerCore: 32768 min: 131072 <ins> tcpBeLiberal: false </ins> tcpCloseWaitTimeout: 1h0m0s tcpEstablishedTimeout: 24h0m0s udpStreamTimeout: 0s

Related discussion:
text: I discovered that the KUBE-FORWARD chain had a rule to drop invalid packets, which was in Compiling kubernetes 1.15 without this commit fixed our problems, and we have been carrying that patch forward ever since. I would like to get rid of our forked kubernetes, hence this ticket. When I was writing this ticket up, I also found also mentions the same issue with the iptables dropping INVALID conntrack state packets near the end, regarding asymmetrical routing. Edit, OK I get the original problem, but still don't understand why packets from outside the pod range would not exhibit , thereby requiring yet another revisit of this problem. I think you are correct, I can imagine packets from outside of a cluster could exhibit the same behavior as packets inside the cluster and cause spurious resets. However, I am wondering if kube-proxy should be managing packets from outside of the cluster on a node... That seems outside of the scope of kube-proxy.
If this issue has been triaged, please comment . If you aren't able to handle this issue, consider unassigning yourself and/or adding the label. I am a bot run by vllry. ‍
I took a stab at implementing what I think is the correct behavior for kube-proxy in .
/assign
I think that another option is to set
I don't understand how that will help me here. The packets with a ctstate INVALID are not generated due to conntrack saturation (), but due to asymmetric routing in our network. See for an example of a similar setup.
are the packets on your asymmetric routing setup marked as invalid if you set ? it should only mark the ones that are out of window per the description
/triage accepted
I'd love to pin this down, but I don't have the means to set up a repro. Is there a way I can set up a way to experiment?
We were bit by this too after revisiting an old service that had been unused since the heady days of pre-, and would also like to see the proposed patch merged (or a configuration option to prevent insertion of the INVALID state drop rule - either suits us). I unfortunately don't have any useful input as to setting up for reproduction. In our case we have a pretty odd anycast set up with a small pool of (public) addresses managed by calico.
Code change:
conntrack: maxPerCore: 32768 min: 131072 <ins> tcpBeLiberal: false </ins> tcpCloseWaitTimeout: 1h0m0s tcpEstablishedTimeout: 24h0m0s udpStreamTimeout: 0s

Related discussion:
text: However I do seem to have a dirty workaround for small clusters. By setting notrack (ctstate UNTRACKED?) in the raw table for packets in our specific ranges that would otherwise match ctstate INVALID, the packets aren't dropped. Which may be of help to someone else who finds this issue.
We are seeing something very similar on large file downloads and our issue exactly fits this one's subject. These RESET (corrected after are happening intermittently and I am seeing similar behavior mentioned in the blog mentioned below. I am unsure if someone has found a workaround for this issue or not, so yeah my +1 to this issue being looked up by the community.
We're using as a solution for the connection reset we were seeing and might fix this issue.
Let me try to avoid that this thread derails and see if I can get current status: -proxy install rules to drop INVALID cstate connections, this solves the ~REJECT~ RESET problems are some scenarios, with asymmetric routing per example, that this rules may drop unrelated traffic. The request is to remove the rule that drops those INVALID states or to try to restrict it, and that can cause that people start to have the ~REJECT~ RESET problems and drops again. If we have a reproducer we can be more precise in the final solution, but removing one rule without any strong evidence in kube-proxy is very risky As a workaround for the asymetric routing problem, I suggested to set the nfconntracktcpbeliberal on the kernel: I think that will solve the problems, it will be good if someone can test it and provide feedback
No sadly, its my bad, it is not REJECT, it is RESET that's happening and conntrack has nothing to do with it, mostly (I hope). Kubeproxy is doing what is intended but specifically when there are external agents in play, the packet doesn't sends an ACK back to the response of SYN-ACK. Our case, we hit A to initiate a download, A calls B to get the action completed to generate download data for the user finally via their browser. A and B both are on k8s, we saw A starts and a complete cycle goes for sometime until there multiple parallel downloads requested where few gets completed and few gets a RESET.
Code change:
conntrack: maxPerCore: 32768 min: 131072 <ins> tcpBeLiberal: false </ins> tcpCloseWaitTimeout: 1h0m0s tcpEstablishedTimeout: 24h0m0s udpStreamTimeout: 0s

Related discussion:
text: On primary debugging, it appears that RESET was getting invoked by the pod but pod is just sending a RST-ACK not RST itself. The missing packet is never found in the default route, unless we see that there is a chance where ACK is missed and we found some lines in tcpdump with missing sequence for ACKs. Thats why i doubt my issue is related here. One more weird coincidence is few downloads have spurious transmission.
Thankyou, I will take a look, but mostly I would like to wait before using it, debug more to understand the actual cause and then act on it.
sorry , I wanted to say RESET, thanks for catching it. If you think is different and you can provide information and how to reproduce it, I suggest you to open a new issue
I will sure do, when it is different. For now I would like to watch and submit my findings if it is related. PS: sysctl -w net.netfilter.nfconntracktcpbeliberal=1 is already implemented and we are testing if it does something helpful. so far it doesn't looks like it. thanks for your help.
/area kube-proxy
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-contributor-experience at . /lifecycle stale
Stale issues rot after 30d of inactivity. Mark the issue as fresh with . Rotten issues close after an additional 30d of inactivity. If this issue is safe to close now please do so with . Send feedback to sig-contributor-experience at . /lifecycle rotten
I have the feeling we are also affected by this with a standard calico installation: every node announces a /122 IPv6 network to the upstream router the upstream router uses ECMP to route the packet back packets that are sent from* the pod and returned via asymmetric path from outside seem to be dropped
and if you set ?
Same behaviour with set on all nodes, the outgoing pod traffic does not return. And I instantly see packets going back into the drop rule:
:thinking:
The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs.
Code change:
conntrack: maxPerCore: 32768 min: 131072 <ins> tcpBeLiberal: false </ins> tcpCloseWaitTimeout: 1h0m0s tcpEstablishedTimeout: 24h0m0s udpStreamTimeout: 0s

Related discussion:
text: This bot triages issues and PRs according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Reopen this issue or PR with Mark this issue or PR as fresh with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at . /close [1]:
Closing this issue. <detailsIn response to : The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs. After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closedReopen this issue or PR with Mark this issue or PR as fresh with Offer to help out with [Issue Triage][1] Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
Hi, are you still working on this? Same problem happened in our clusters, just because of asymmetrical routing.
/reopen
Reopened this issue. <detailsIn response to : /reopen </details
Also the command suggested by is helping us, but I still think this should be a bug of kube-proxy.
The problem here is: what happens if the solutions are exclusive or we regress? The addition of this rule fixed a legit bug that was documented and there is also a nice blog explaining the details Removing this rule without a repro and understanding what is the root cause of this issue is not really the best way to proceed. I think that people affected should try to find the root cause and explain the problem or just provide a way to reproduce it, so we can fix it
The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs. This bot triages issues and PRs according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Reopen this issue or PR with Mark this issue or PR as fresh with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at .