jiebi/CodeConvo huggingface dataset:
Change:
{utiliptables.TableFilter, KubeForwardChain}, } <ins> var iptablesCleanupChains = []struct { table utiliptables.Table chain utiliptables.Chain }{ {utiliptables.TableNAT, kubeServicesChain}, {utiliptables.TableNAT, kubePostroutingChain}, {utiliptables.TableNAT, KubeFireWallChain}, {utiliptables.TableNAT, KubeNodePortChain}, {utiliptables.TableNAT, KubeLoadBalancerChain}, {utiliptables.TableFilter, KubeForwardChain}, } </ins> // ipsetInfo is all ipset we needed in ipvs proxier var ipsetInfo = []struct { name    string

Discussion:
<!-- This form is for bug reports and feature requests ONLY! If you're looking for help check and the . If the matter is security related, please disclose it privately via --Is this a BUG REPORT or FEATURE REQUEST?: /sig network What happened: Kube-proxy () deployed as daemonset with ipvs mode produce during startup following error: With verbosity -v5 the logs looks like this: Looks like proxier is trying to perform cleanup part: but failed. Such error is generated only once during startup. What you expected to happen: Error should not exist :) Environment: Kubernetes version (use ): 1.11.1 Cloud provider or hardware configuration: none OS (e.g. from /etc/os-release): Ubuntu 16.04 LTS Kernel (e.g. ): Install tools: custom script based on docker Others:
I've build kube-proxy with additional information: and the logs looks like this:
same issue
Looks like also fixed this one.
I'm on 1.12.0 and the issue is still there I suppose is included in 1.12?
Updated my kube to 1.12.3 and I got same
I'm still seeing this on v1.12.4 as well
same here on v1.13.2, throws this error when kube-proxy initializes. Running on CoreOS in ipvs mode:
Hello, I also encountered this problem in Kubernetes 1.13.3. The operating system is CentOS 7.6. The kernel version is: Linux linux-node4 3.10.0-957.el7.x86_64
Why is this closed? It looks like an unresolved issue. I have the same errors with 1.12.1.
same as me
same as me,1.12.3
same problem OS: Ubuntu 18 kernel: 4.15.0-46-generic Kubernetes: 1.13.4
same problem OS: CentOS 7.6 kernel: 3.10.0-957.el7.x86_64 Kubernetes: 1.13.4
I met this issue on Centos 7.6 with NetworkManager installed, it seems related to the network.
Same problem OS: CentOS 7.4 Kubernetes: 1.12.5 NetworkManager not installed.

Some of the nodes does not have this issue while others do
I am getting in 1.14.0 as well in some worker nodes.
same problem,how to solve it
same problem 1.13.2 ,how fix it?
i update kube-proxy 1.14.2, the error message is disappear
same problem OS: Debian 8 Kubernetes: v1.15.0 iptables v1.4.21 kube-proxy mode ipvs I write some test code like kube-proxy --cleanup and found line 7 is , and run , the error is The problem is we should flush KUBE-LOAD-BALANCER and KUBE-NODE-PORT before delete KUBE-MARK-MASQ , I can try fix it.
Change:
csiDriverRegistrarImage     string = "quay.io/k8scsi/driver-registrar:v0.2.0" ) <del> func externalAttacherServiceAccount( </del> <ins> func csiServiceAccount( </ins> client clientset.Interface, config framework.VolumeTestConfig, teardown bool,

Discussion:
Current CSI E2E test has some issues which prevent it to run successfully the a local cluster. /sig storage /kind bug /assign
Change:
func deletePDWithRetry(diskName string) { var err error <del> for start := time.Now(); time.Since(start) < 180*time.Second; time.Sleep(5 * time.Second) { </del> <ins> for start := time.Now(); time.Since(start) < gcePDRetryTimeout; time.Sleep(gcePDRetryPollTime) { </ins> if err = deletePD(diskName); err != nil { framework.Logf("Couldn't delete PD %q. Sleeping 5 seconds (%v)", diskName, err) continue

Discussion:
For: "Pod Disks should schedule a pod w/ a RW PD, remove it, then schedule it on another host" "Pod Disks should schedule a pod w/ a readonly PD on two hosts, then remove both." (this is not happening on v1.3 master with v1.1 nodes, nor with v1.2 tests against a v1.3 cluster) View logs and artifacts at
Mind taking a look? You should be able to repro by turning up a v1.3 cluster and running v1.1 tests against it.
for your queue
I found the cause of the test failure. The following are the main steps in the test host0Pod, and wait for it running a file to the host0Pod, and then delete it host1Pod, and waiting for it running the file from host1Pod, and then delete it disk with deletePDWithRetry() function, which has 3 minutes timeout defer() function, clean up test environment, and also run deletePDWithRetry() again just in case The problem arises from step 5. In the above test, deletePDWithRetry tried to delete PD every 5 second, but failed in 3 minutes. After it exits retry loop, it failed for the expectNoError(err, "Error deleting PD") in this function. Although it deleted the PD successfully right after this during the clean up phase (step 6). So the main issue for this test failure is the timeout is a little bit short. To fix it issue, we can either increase the timeout, or not fail when calling deletePDWithRetry for the first time. We can fail it if it cannot delete it after calling deletePDWIthRetry second time in step 6. The following is the retry loop and error code in the deletePDWIthRetry(). In current 1.3 version, we only call deletePDWithRetry once in defer() function during clean up. I think to be safe we should increase timeout to 5 minutes, and set it in the constance instead of coding it directly inside of the function.
So two options to fix this: port test fixes from main branch to the 1.1 branch to make the test more robust against delays. for to be merged, which will decrease the time it takes for a volume to be detached and therefore how long it takes to be deleted.
Change:
assert.Equal(t, testLogs[4]+compressSuffix, logs[2].Name()) assert.Equal(t, testLogs[2]+"."+timestamp, logs[3].Name()) assert.Equal(t, testLogs[3], logs[4].Name()) <del> c.queue.ShutDown() </del> } func TestClean(t *testing.T) {

Discussion:
TestRotateLogs from kubelet/logs/containerlogmanagertest Probably since the merge of change No response_ /sig node
/triage accepted /cc
/cc
/priority important-longterm Seems to impact on this arch only?
cc
/sig windows
ack.. Checking
what is the easiest way for me to test this one locally ? using based vm ?
poke and on slack please
unfortunately, I couldn't reproduce this flake locally. Will update here once I am able to.
I am now able to see the flake, when I run the test within a Pod with a resource limit as our CI job does. Steps to reproduce: clone kubernetes test -c -race - TestRotateLogs root stress - TestRotateLogs Please find the attached trace file
This is with or on a native x86_64 as well ?
I could reproduce this even on x8664. Also, the failures do happen even when run on the VM itself and NOT ONLY within the pod as I mentioned above. The flakiness % is higher when run from a pod rather than a VM.
I was able to reproduce this on regular machines as well with while running tests. Let me check and see what needs to be done to fix this.
This seem to take care of the issue. Do you think the poll interval of is too much ? cc
bash stress - TestRotateLogs -test.v 5s: 0 runs so far, 0 failures 10s: 48 runs so far, 0 failures 15s: 96 runs so far, 0 failures 20s: 146 runs so far, 0 failures 25s: 196 runs so far, 0 failures 30s: 244 runs so far, 0 failures 35s: 292 runs so far, 0 failures 40s: 339 runs so far, 0 failures 45s: 387 runs so far, 0 failures
Spoke too early.

bash 1m30s: 812 runs so far, 4 failures (0.49%) /var/folders/l3/tf0xj3gj67bfmk13h462668w0000gn/T/go-stress-20240223T144024- === RUN TestRotateLogs Error Trace: Error: "[- test-log-1 - test-log-2.- - test-log-3 - test-log-3.- - test-log-3.- - test-log-4]" should have 5 item(s), but has 6 Test: TestRotateLogs Error Trace: Error: Not equal: expected: "test-log-3.-" actual : "test-log-3" Diff: --- Expected +++ Actual -1 +1 -test-log-3.- +test-log-3 Test: TestRotateLogs Error Trace: Error: Not equal: expected: "test-log-3.-" actual : "test-log-3.-" Diff: --- Expected +++ Actual -1 +1 -test-log-3.- +test-log-3.- Test: TestRotateLogs Error Trace: Error: Not equal: expected: "test-log-4" actual : "test-log-3.-" Diff: --- Expected … 1m35s: 851 runs so far, 5 failures (0.59%)
/assign
/triage accepted
Had to add a few additional logs to identify the issue, but has a fix now. PTAL when possible.
Change:
"fmt" "time" <del> "k8s.io/api/core/v1" </del> <ins> v1 "k8s.io/api/core/v1" </ins> apierrors "k8s.io/apimachinery/pkg/api/errors" metav1 "k8s.io/apimachinery/pkg/apis/meta/v1" utilruntime "k8s.io/apimachinery/pkg/util/runtime"

Discussion:
What happened: kube-controller-manager with enabled option --controllers=,tokencleaner is not removing newly created tokens. It works only after kube-controller-manager restart or after editing secret with token. What you expected to happen: It should remove all expired tokens from kube-system namespace, without restarting or touching secrets. How to reproduce it (as minimally and precisely as possible): enable tokencleaner in kube-controller-manager start controller manager create bootstrap token, you can use script wait 5 minutes token is still there restart kube-controller-manager OR edit newly created token token should be gone after a while Anything else we need to know?: It could be related with Environment: Kubernetes version (use ): Server Version: version.Info{Major:"1", Minor:"13", GitVersion:"v1.13.0", GitCommit:"", GitTreeState:"clean", BuildDate:"2018-12-03T20:56:12Z", GoVersion:"go1.11.2", Compiler:"gc", Platform:"linux/amd64"}
Do you see the following log ? thanks
no errors at all
/sig auth
Change:
// implementation of the error interface func (f *FitError) Error() string { <del> output := fmt.Sprintf("failed to find fit for pod: %v", f.Pod) </del> <ins> output := fmt.Sprintf("failed to find fit for pod, ") </ins> for node, predicateList := range f.FailedPredicates { output = output + fmt.Sprintf("Node %s: %s", node, strings.Join(predicateList.List(), ",")) }

Discussion:
If a pod is not scheduled because it cannot find a fit, you run into really ugly event output like the following: This makes the output of horrific to work with. A better event would just read: The pod is known from the involved object already so no need to encode it all.
Change:
InformerFactory:  informers.NewSharedInformerFactoryWithOptions(testClientset, time.Duration(1)), InformersStarted: make(chan struct{}), } <del> _, _, err := startResourceQuotaController(ctx) </del> <ins> for funcName, controllerInit := range controllerInitFuncMap { _, _, err := controllerInit(ctx) if test.expectedErr != (err != nil) { t.Errorf("%v test failed for use case: %v", funcName, name) } } _, _, err := startModifiedNamespaceController( ctx, testClientset, testClientBuilder.ConfigOrDie("namespace-controller")) </ins> if test.expectedErr != (err != nil) { <del> t.Errorf("test failed for use case: %v", name) </del> <ins> t.Errorf("Namespace Controller test failed for use case: %v", name) </ins> } } }

Discussion:
In we observed a crashlooping controller cause a cluster cold boot failure, as controller-manager wasn't able to stay up long enough to run the deployment that would have fixed the problem. We should audit everything that runs in kube-controller-manager and ensure that they all fail gracefully enough to prevent crashloops, or make them individually restart in that case, instead of the process. Longer term, we should consider classifying the controllers by run level or bootstrap order (), and splitting them into separate binaries or changing the start-up order in kcm to make sure this sort of thing doesn't happen in the future.
Same issue with metrics-server, using minikube. Also reproduced on our test cluster. Steps to reproduce: Set up localkube Install metrics-server Stop minikube: sudo minikube stop Start minikube: sudo -E minikube start --vm-driver=none Expected: Kubernetes is running Actual: Kubernetes is dead Logs: Workaround: To fix the system, we must delete the metrics-server: Now kubernetes will start normally. After that we can reinstall the metrics server, and it will be ok.
Related:
This will be a hard problem to fix permanently. In all likelihood, we should stop allowing Fatalf() in kubernetes/kubernetes, but as you can see we do this to ourselves right here...
/kind bug
I also think this bug would benefit from an e2e test where we misconfigure things in a variety of ways and verify that KCM still comes up and stays up: [ ] misconfigured apiservice [ ] broken service attached to that apiservice [ ] RBAC rules that disallow things. maybe everything! [ ] webhooks that disallow everything [ ] yr example here
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
/remove-lifecycle stale
Change:
s.mux.Handle("/logs/", http.StripPrefix("/logs/", http.FileServer(http.Dir("/var/log/")))) s.mux.HandleFunc(s.prefix+"/", s.ServeREST) healthz.InstallHandler(s.mux) <del> s.mux.HandleFunc("/index.html", s.handleIndex) </del> <ins> s.mux.HandleFunc("/", s.handleIndex) </ins> // Handle both operations and operations/* with the same handler s.mux.HandleFunc(s.operationPrefix(), s.handleOperationRequest)

Discussion:
sig-release master-blocking build-masterci-kubernetes-build.Overall 2022-06-01 20:49 CEST, commit No response /sig release cc
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
I'm checking that
/assign
this pr should fix
it looks like it's still failing: https://k8s-
/reopen reopening as this is causing all our CI to run against stale artifacts at: v1.25.0-alpha.0.734+
Reopened this issue. <detailsIn response to : /reopen v1.25.0-alpha.0.734+ </details
my previous fix only fixed one side of the things, missing now job are green ! closing for now /close
Closing this issue. <detailsIn response to : my previous fix only fixed one side of the things, missing now job are green /close Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
thank you !
Change:
} // only send events newer than resourceVersion if event.ResourceVersion > resourceVersion { <del> c.sendWatchCacheEvent(event) </del> <ins> c.sendWatchCacheEvent(&event) </ins> } } }

Discussion:
Our scalability tests started failing on Thu last week: https://k8s- [250 run passed, 251 failed and since then everything failed] I started looking into it and it seems that something happened to how we are handling watches (i.e. I'm seeing a huge number of watches failing with "err too old" errors (which I didn't see before).
As an example in run 258 of kubemark-scale, I'm seeing this log in the test (after 5 minutes of running test); This is extremely strange and unexpected. I looked into apiserver logs, and there is expected number of POST and PUT operations on pods (and no DELETE and PATCH operations until then). Until then, there were alltogethere ~60.000 operations on pods. However, in 5000-node cluster, the size of watchcache of pods is supposed to be 250.000, which means that at that moment we should have all the history from the beginning in watch cache. And still we are observing those errors. Something must have changed around that recently...
As an example, logs from apiserver from a given node: This means something is really wrong: the first watch started at 45, then after it finished, we tried starting watch from rv= and it failed with "rv too old" the list, returned rv= from which we started the next watch. If the watch cache size for pods would be 250.000 (50*cluster size), there there is no way that with current rv=, the rv= would be outside history window...
I had a hypothesis that it might be the problem of breaking watch to etcd. But in that case there should be a warning "Terminating all watchers" log in apiserver, and there is no such log. It also seems that the size of cache is being set correctly...
So I tried running large kubemark locally and it seems that the problem is that the size of cache is too small. I few more logs, and here is what I'm seeing: So it seems that we are correctly initializing cachesizes, but then watch cache capacity is 100 (which explains why we are seeing all those "rv too old" errors. I will try to debug why the capacity is so small.
OK - I found it. contains the fix.
Change:
} // TODO: move this into the kubelet itself <del> func MonitorCAdvisor(k *Kubelet) { </del> <ins> func MonitorCAdvisor(k *Kubelet, cp uint) { </ins> defer util.HandleCrash() // TODO: Monitor this connection, reconnect if needed? glog.V(1).Infof("Trying to create cadvisor client.") <del> cadvisorClient, err := cadvisor.NewClient("http://127.0.0.1:4194") </del> <ins> cadvisorClient, err := cadvisor.NewClient("http://127.0.0.1:" + strconv.Itoa(int(cp))) </ins> if err != nil { glog.Errorf("Error on creating cadvisor client: %v", err) return

Discussion:
Currently, kubelet uses a for communicating with cAdvisor. , were having an issue because we didn't realize making port available was required for kubernetes.
I prefer not do this since we plan to make 4194 publically reserved port for cAdvisor. Also cAdvisor is the default deamon running on every node of a kubernetes cluster. The issue you encountered sounds like a bad document issue we had for cAdvisor. We should fix the document instead of allow the port being configurable.
can you expound on 'bad document' issue?
Change:
t.Fatalf("%s: actual warning message did not match expected warning message.n Expecting:n%vn  Got:n%v", test.description, e, a) } } <ins> if len(test.expectOutputToContain) > 0 { out := outBuf.String() if !strings.Contains(out, test.expectOutputToContain) { t.Fatalf("%s: expected output to contain: %snGot:n%s", test.description, test.expectOutputToContain, out) } } </ins> }) } }

Discussion:
When draining a node, using kubectl drain [NODE], the output resembles this: Expected output should indicate that the node was drained, for example: kubectl drain [NODE] No response <details</details<detailsCan reproduce locally using and printing the output </details<details</details<details</details<details</details<details</details
/sig cli
/triage accepted
Change:
"os" "runtime" "sync" <del> "syscall" </del> "time" "unsafe" "github.com/golang/glog" cadvisorapi "github.com/google/cadvisor/info/v1" <ins> "golang.org/x/sys/windows" </ins> "k8s.io/apimachinery/pkg/util/wait" ) <ins> // MemoryStatusEx is the same as Windows structure MEMORYSTATUSEX // https://msdn.microsoft.com/en-us/library/windows/desktop/aa366770(v=vs.85).aspx type MemoryStatusEx struct { Length               uint32 MemoryLoad           uint32 TotalPhys            uint64 AvailPhys            uint64 TotalPageFile        uint64 AvailPageFile        uint64 TotalVirtual         uint64 AvailVirtual         uint64 AvailExtendedVirtual uint64 } </ins> var ( <del> modkernel32                            = syscall.NewLazyDLL("kernel32.dll") procGetPhysicallyInstalledSystemMemory = modkernel32.NewProc("GetPhysicallyInstalledSystemMemory") </del> <ins> modkernel32              = windows.NewLazySystemDLL("kernel32.dll") procGlobalMemoryStatusEx = modkernel32.NewProc("GlobalMemoryStatusEx") </ins> ) // NewPerfCounterClient creates a client using perf counters

Discussion:
<!-- This form is for bug reports and feature requests ONLY! If you're looking for help check and the . --Is this a BUG REPORT or FEATURE REQUEST?: /kind bug What happened: Windows uses GetPhysicallyInstalledSystemMemory to get physical memory installed on Windows machine and reports it as part of node metrics to heapster, etc. On VMWare Fusion, GetPhysicallyInstalledSystemMemory fails to read data from SMBIOS firmware tables and returns error code 0x0d. That causes on Windows to bail out. It is also reported on Virtualbox. What you expected to happen: Windows should be able read to physical memory on the system and shouldn't exit. How to reproduce it (as minimally and precisely as possible): Run on VMWare Fusion or Virtualbox . Anything else we need to know?: Environment: Kubernetes version (use ): latest Cloud provider or hardware configuration: OS (e.g. from /etc/os-release): Windows Kernel (e.g. ): Install tools: Others: VMWare Fusion or Virtualbox
/sig windows /sig node
this will likely get fixed after 1.9 is released
Sure, I am fixing it. Unfortunately it missed the 1.9 train.
Hi, any idea on when this might get fixed? Thanks!
It is fixed in Got test failure in pull-kubernetes-e2e-kops-aws yesterday but not before. I will look into it shortly.
thanks for the swift reply! I need to learn some new (for me) github skills to get that PR.
i just approved the PR again. we should be good to go. can you please check
Change:
replicaSets := reaper.Extensions().ReplicaSets(namespace) rsReaper, _ := ReaperFor(extensions.Kind("ReplicaSet"), reaper) <del> deployment, err := deployments.Get(name) if err != nil { return err } // set deployment's history and scale to 0 // TODO replace with patch when available: https://github.com/kubernetes/kubernetes/issues/20527 zero := 0 deployment.Spec.RevisionHistoryLimit = &zero deployment.Spec.Replicas = 0 // TODO: un-pausing should not be necessary, remove when this is fixed: // https://github.com/kubernetes/kubernetes/issues/20966 // Instead deployment should be Paused at this point and not at next TODO. deployment.Spec.Paused = false deployment, err = deployments.Update(deployment) </del> <ins> deployment, err := reaper.updateDeploymentWithRetries(namespace, name, func(d *extensions.Deployment) { // set deployment's history and scale to 0 // TODO replace with patch when available: https://github.com/kubernetes/kubernetes/issues/20527 zero := 0 d.Spec.RevisionHistoryLimit = &zero d.Spec.Replicas = 0 // TODO: un-pausing should not be necessary, remove when this is fixed: // https://github.com/kubernetes/kubernetes/issues/20966 // Instead deployment should be Paused at this point and not at next TODO. d.Spec.Paused = false }) </ins> if err != nil { return err }

Discussion:
It looks that we lack retry in the test. cc
As can be seen by number of references, this seems to be happening often. Bumping to P0.
is waiting to be merged.

http://pr- cc
This can be reproduced by running cc
Update: It didn't always generate errors; when this happened, the error came from
The conflict happens because when we stop the deployment, we first update it: cc
Change:
"github.com/spf13/cobra" "k8s.io/kubernetes/pkg/api" "k8s.io/kubernetes/pkg/api/meta" <ins> "k8s.io/kubernetes/pkg/api/unversioned" batchv1 "k8s.io/kubernetes/pkg/apis/batch/v1" "k8s.io/kubernetes/pkg/apis/extensions/v1beta1" </ins> client "k8s.io/kubernetes/pkg/client/unversioned" "k8s.io/kubernetes/pkg/kubectl" cmdutil "k8s.io/kubernetes/pkg/kubectl/cmd/util"

Discussion:
Tested with: Most of the simple basic starting guides direct users to create a cluster with . If I run kubectl 1.2 against a 1.1 cluster, I'll get the following behavior now I happen to know I can get the old behavior by doing but I don't expect an average user to know that or be able to find it easily. I'm mainly concerned that we will be breaking recommended flow for GKE users if they happen to use a new kubectl before their cluster's master is upgraded. We can just update our docs, but breaking a "porcelain" command seems like poor form. Other options include defaulting to the generator for a few patch releases, and/or making do some kind of version check so it can fallback to the generator if deployments/jobs are not available. thoughts?
We should update those docs and document how to achieve stable behavior: .
This would ideally use API discovery, though, to check whether the API exists.
How hard would it be to check whether Deployment existed? We had the same issue in 1.1, btw, with Job.
I'd accept a hack, also, which checked the k8s version if there wasn't enough discovery info in 1.1.
We have the same issue with Job in 1.2, also.
Fix is in