jiebi/CodeConvo huggingface dataset:
Code change:
for _, tt := range tests { t.Run(tt.name, func(t *testing.T) { <ins> featuregatetesting.SetFeatureGateDuringTest(t, utilfeature.DefaultFeatureGate, features.SchedulerQueueingHints, tt.qHintEnabled) </ins> nodeInfo := framework.NewNodeInfo(tt.existingPods...) nodeInfo.SetNode(tt.nodeFn()) preCheckFn := preCheckForNode(nodeInfo) <del> var got []bool </del> <ins> got := make([]bool, 0, len(tt.pods)) </ins> for _, pod := range tt.pods { <del> got = append(got, preCheckFn(pod)) </del> <ins> got = append(got, preCheckFn == nil || preCheckFn(pod)) </ins> } if diff := cmp.Diff(tt.want, got); diff != "" {

Related discussion:
text: If we can move all such logic to each plugin, that'd be perfect.
^ is more like a new feature than just a solution to this problem. I created another issue and describe my proposal in detail. Let's have a discussion there.
It can be resolved by moving all preCheck into each plugin's QueueingHint. Let me retitle this issue so that the current status is easy to understand. /retitle scheduler: move all preCheck to QueueingHint
This issue has not been updated in over 1 year, and should be re-triaged. You can: Confirm that this issue is still relevant with (org members only) Close this issue with For more details on the triage process, see /remove-triage accepted
/triage accepted
Code change:
} } <ins> func TestProxyRedirectsforRootPath(t *testing.T) { tests := []struct { name               string method             string requestPath        string expectedHeader     http.Header expectedStatusCode int redirect           bool }{ { name:               "root path, simple get", method:             "GET", requestPath:        "", redirect:           true, expectedStatusCode: 301, expectedHeader: http.Header{ "Location": []string{"/"}, }, }, { name:               "root path, simple put", method:             "PUT", requestPath:        "", redirect:           false, expectedStatusCode: 200, }, { name:               "root path, simple head", method:             "HEAD", requestPath:        "", redirect:           true, expectedStatusCode: 301, expectedHeader: http.Header{ "Location": []string{"/"}, }, }, { name:               "root path, simple delete with params", method:             "DELETE", requestPath:        "", redirect:           false, expectedStatusCode: 200, }, } for _, test := range tests { func() { w := httptest.NewRecorder() req, err := http.NewRequest(test.method, test.requestPath, nil) if err != nil { t.Fatal(err) } redirect := proxyRedirectsforRootPath(test.requestPath, w, req) if got, want := redirect, test.redirect; got != want { t.Errorf("Expected redirect state %v; got %v", want, got) } res := w.Result() if got, want := res.StatusCode, test.expectedStatusCode; got != want { t.Errorf("Expected status code %d; got %d", want, got) } if res.StatusCode == 301 && !reflect.DeepEqual(res.Header, test.expectedHeader) { t.Errorf("Expected location header to be %v, got %v", test.expectedHeader, res.Header) } }() } } </ins> // exampleCert was generated from crypto/tls/generate_cert.go with the following command: //    go run generate_cert.go  --rsa-bits 1024 --host example.com --ca --start-date "Jan 1 00:00:00 1970" --duration=1000000h var exampleCert = []byte(`-----BEGIN CERTIFICATE-----

Related discussion:
text: This bot triages issues and PRs according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Mark this issue or PR as fresh with Mark this issue or PR as rotten with Close this issue or PR with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at . /lifecycle stale [1]:
The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs. This bot triages issues and PRs according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Mark this issue or PR as fresh with Close this issue or PR with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at . /lifecycle rotten [1]:
/remove-lifecycle stale
/remove-lifecycle stale
Code change:
] + select({ "@io_bazel_rules_go//go/platform:android": [ "//pkg/util/procfs:go_default_library", <ins> "//staging/src/k8s.io/api/rbac/v1:go_default_library", </ins> "//staging/src/k8s.io/apimachinery/pkg/fields:go_default_library", "//staging/src/k8s.io/apimachinery/pkg/labels:go_default_library", "//staging/src/k8s.io/apimachinery/pkg/types:go_default_library",

Related discussion:
text: Which jobs are failing: Which test(s) are failing: Since when has it been failing: Last successful run (23 Oct 2020, 10:07 PDT): First unsuccessful run (23 Oct 2020, 12:08 PDT): Testgrid link: Reason for failure: Anything else we need to know: presubmit job in repository also started to fail in the same fashion. Attempted to debug it myself, found out the following stuff: last successful run was accessing the apiserver via HTTP: all unsuccessful runs attempted to access the apiserver via HTTPS but to no avail: perused the history of to find out that the PR is almost certainly the reason of the failures (or rather uncovered some problem with the NPD or its tests): first, it coincides with the start of the failures of , second, we changed the default hostname of the apiserver in node tests in that PR. Judging by the code and the changes in the aforementioned PR I think that a quick mitigation would be to add a flag to the of these jobs. /assign /sig node /cc
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
Opened for a mitigation that would make those jobs stop failing.
This won't work, unfortunately. Copying response:
I have filed to address this issue, but it seems I could not run the test in advance to verify my patch, did you know how to manually trigger the presubmit job?
Unfortunately I don't :/. My PR was also a wild guess. could you take a look at the proposed fix? Thanks once again for your help - really appreciated :).
the submission of hasn't fixed the problems, unfortunately :/. Sample failing run: We still have some peculiar problem related to health checking: and now NPD panics, too:
Thanks for your report, I believe the reason is admission controller has been disabled in node e2e tests: We need to figure out a new approach now. /reopen
Reopened this issue.
Code change:
] + select({ "@io_bazel_rules_go//go/platform:android": [ "//pkg/util/procfs:go_default_library", <ins> "//staging/src/k8s.io/api/rbac/v1:go_default_library", </ins> "//staging/src/k8s.io/apimachinery/pkg/fields:go_default_library", "//staging/src/k8s.io/apimachinery/pkg/labels:go_default_library", "//staging/src/k8s.io/apimachinery/pkg/types:go_default_library",

Related discussion:
text: <detailsIn response to : Thanks for your report, I believe the reason is admission controller has been disabled in node e2e tests: We need to figure out a new approach now. Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
filed . I hope we could completely address this issue this time.
/milestone v1.20
Code change:
if nvcSize > 0 && nvcSize+vcSize >= numCandidates { cancel() } <del> } else { statusesLock.Lock() nodeStatuses[nodeInfoCopy.Node().Name] = status statusesLock.Unlock() </del> <ins> return } if status.IsSuccess() && len(pods) == 0 { status = framework.AsStatus(fmt.Errorf("expected at least one victim pod on node %q", nodeInfoCopy.Node().Name)) </ins> } <ins> statusesLock.Lock() nodeStatuses[nodeInfoCopy.Node().Name] = status statusesLock.Unlock() </ins> } fh.Parallelizer().Until(parallelCtx, len(potentialNodes), checkNode) return append(nonViolatingCandidates.get(), violatingCandidates.get()...), nodeStatuses

Related discussion:
text: <!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --Kubernetes 1.19 (confirmed with 1.19.7 and 1.19.10) kube-scheduler crashes and restarts with the following errors in . panic: runtime error: index out of range [0] with length 0 The problem code is line 389. When is nil or is empty, the error happens. If we skip when it's nil or empty, the error is gone. panic: runtime error: invalid memory address or nil poster dereference The problem is does not exist and returns nil sometimes. Simply skipping it won't solve the problem. The scheduler will reach either line 456 or 341. The scheduler works without failures. When there are preemptions in a cluster. Kubernetes version (use ): v1.19.10 or v1.19.7 Cloud provider or hardware configuration: bare metal OS (e.g: ): centos 7 Kernel (e.g. ): Linux 5.4.77-7.el7pie SMP Sat Nov 21 01:16:27 UTC 2020 x8664 x8664 x8664 GNU/Linux Install tools: Network plugin and version (if this is a network-related bug): Others:
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/sig scheduling
/cc
/assign
Are you running the vanilla scheduler? or a customized one that leverages the utilities in
It's a custom scheduler with some internal out-of-tree plugins (just like ), but it doesn't have any custom postFilter plugins. Which utility functions are you referring to? Thanks.
are those custom plugins filter plugins? do they maintain state?
Code change:
if nvcSize > 0 && nvcSize+vcSize >= numCandidates { cancel() } <del> } else { statusesLock.Lock() nodeStatuses[nodeInfoCopy.Node().Name] = status statusesLock.Unlock() </del> <ins> return } if status.IsSuccess() && len(pods) == 0 { status = framework.AsStatus(fmt.Errorf("expected at least one victim pod on node %q", nodeInfoCopy.Node().Name)) </ins> } <ins> statusesLock.Lock() nodeStatuses[nodeInfoCopy.Node().Name] = status statusesLock.Unlock() </ins> } fh.Parallelizer().Until(parallelCtx, len(potentialNodes), checkNode) return append(nonViolatingCandidates.get(), violatingCandidates.get()...), nodeStatuses

Related discussion:
text: If yes, then one hypothesis is that those custom filters filter the node in the filter phase, but not when executed in the preemption phase, and so this will result in a candidate node with no victims?
Yes, it includes filter plugins and uses cycle state.
you need to make sure that those filter plugins play nicely when executed again in the preemption phase in the same cycle: i.e., produce the same result.
Thanks What do you mean by performing filter in the preemption phase? would you mind elaborating it a little?
What additional logic is needed to handle the preemption case?
in the preemption phase we run the filters again to check if removing lower priority pods make the node schedulable: we later add them one by one to reduce the set of victim pods to the absolute minimum: My hypothesis is that somehow your filter returns success when adding all the pods back, and so you end up with zero victims and a candidate node. This shouldn't happen because if the pod fits the node without removing any victims, then we shouldn't be running preemption in the first place. So the theory is that the custom filter is returning false when run in the filter phase, but returns true when executed in the preemption phase perhaps because it makes some assumptions about cyclestate that makes it behave this way (e.g. that a the filter will be executed once per node in a scheduling cycle).
Disabled the filter plugin, but still see the same issues.
are you able to change the scheduler code and run the test again? I can send a patch tomorrow to add some debugging messages to help us root cause the issue.
yes, thanks a lot! really appreciate it!
You are right. We still use a deprecated Predicate extender, which causes the problem. After disabling it, it's working fine. It used to work with 1.18 though. We are retiring it. Thank you so much!!!
If the test cases can help debug scheduler extender Webhook (Predicate) too, I'd like to try it. Thanks again!
How can we improve the preemption logic to prevent bugs and issues in custom Filters or Predicates crashing the scheduler?
for starter, we should check the list len, the scheduler should not panic in all cases.
Code change:
if nvcSize > 0 && nvcSize+vcSize >= numCandidates { cancel() } <del> } else { statusesLock.Lock() nodeStatuses[nodeInfoCopy.Node().Name] = status statusesLock.Unlock() </del> <ins> return } if status.IsSuccess() && len(pods) == 0 { status = framework.AsStatus(fmt.Errorf("expected at least one victim pod on node %q", nodeInfoCopy.Node().Name)) </ins> } <ins> statusesLock.Lock() nodeStatuses[nodeInfoCopy.Node().Name] = status statusesLock.Unlock() </ins> } fh.Parallelizer().Until(parallelCtx, len(potentialNodes), checkNode) return append(nonViolatingCandidates.get(), violatingCandidates.get()...), nodeStatuses

Related discussion:
text: Other than that, I think we need to clearly document what the preemption logic does (that it executes the prefilter extensions points and the filter plugins multiple times) and the expectations from filter plugins (that they may get executed more than once in the same cycle)
Is it a predicate extender or preemption extender? If it's a predicate extender, I don't quite think we found the root cause - b/c predicate extenders are not invoked during preemption (). It makes more sense if it's a preemption extender - if the preemption extender mutates the to make it invalid. If it's the case, could you verify it by adding debug info before and after L157 to check the validness of ?
That's always true :)
filed a PR to mitigate it.
It's a predicate extender. I'll debug it more.
Added some debug info. after and . Here's the result. returned 4 candidates, but one of candidates () has zero pod. The candidate happened to be the best candidate and hence caused the "" error. It should have nothing to do with since our extender doesn't support Preemption. Disabling it may change pod placements/candidates selection and bypassed the error. It's like a coincidence. To remove the panic, a simple fix is to check if is empty. If it is, do not add the node to candidates. Alternatively, a change to may work too. Here is the log with debug info.
This looks more promising. Back to digging into the root cause, it seems returns an empty but with . So that implies firstly we have a non-nil , and then in , the is separated into two list: violatingVictims and nonViolatingVictims. Finally the two victims list are tried one by one to run . It looks like returns {fits=true, nil error} for every victim, which is quite abnormal. I think it's problematic that every pod is retrievable and in which we can debug into.
Yes, should not return fit=true with empty pods in the first place, but it's hard to locate the problematic code in this function.
The following code in looks questionable to me. If is empty, should the function return nil, nil? Also, if is true for all extenders, the loop won't do anything and candidates will remain the same.
Code change:
if nvcSize > 0 && nvcSize+vcSize >= numCandidates { cancel() } <del> } else { statusesLock.Lock() nodeStatuses[nodeInfoCopy.Node().Name] = status statusesLock.Unlock() </del> <ins> return } if status.IsSuccess() && len(pods) == 0 { status = framework.AsStatus(fmt.Errorf("expected at least one victim pod on node %q", nodeInfoCopy.Node().Name)) </ins> } <ins> statusesLock.Lock() nodeStatuses[nodeInfoCopy.Node().Name] = status statusesLock.Unlock() </ins> } fh.Parallelizer().Until(parallelCtx, len(potentialNodes), checkNode) return append(nonViolatingCandidates.get(), violatingCandidates.get()...), nodeStatuses

Related discussion:
text: Why do we bother creating a new candidates later? Can it just return candidates?
It can prevent from returning candidates with empty pods, but what about ? It's possible to add nodes without victim pods to candidates if returned by contain such nodes. How about adding the following check to CallExtender?
Updated the PR based on the finding and discussions.
Yes, we can use a flag to mark if all extenders don't support preemption. If yes, simply return the candidates.
Technically this can exist due to faulty extender implementation. If we really want to guard it, instead of logging the error and continue, I'm more inclined to return the error immediately as this is a fatal error - the victimsMap cannot be used either for latter extender or further preemptor nominating.
Is my understanding correct? In , if victimsMap is empty, should it return instead of ?
The only case victimsMap can be empty is empty candidates, and so return either doesn't quite matter, right?
You are right.
As long as one extender returns an invalid victimsMap (with empty pods), s returns an error immediately? What about ? Should it return an error immediately if it returnswith empty pods?
I think so. Because the result it returned is faulty, and we don't want to continue based on it The same. We may don't want to continue the scheduling cycle based on faulty result.
Code change:
// Do this multiple times to show that it rate limit rejected requests don't block. for i := 0; i < 2; i++ { <del> expectHTTP(server.URL, errors.StatusTooManyRequests, t) </del> <ins> if err := expectHTTP(server.URL, errors.StatusTooManyRequests); err != nil { t.Error(err) } </ins> } // Validate that non-accounted URLs still work <del> expectHTTP(server.URL+"/dontwait/watch", http.StatusOK, t) </del> <ins> if err := expectHTTP(server.URL+"/dontwait/watch", http.StatusOK); err != nil { t.Error(err) } </ins> // Let all hanging requests finish block.Done() // Show that we recover from being blocked up. // Too avoid flakyness we need to wait until at least one of the requests really finishes. <del> oneAccountedFinished.Wait() expectHTTP(server.URL, http.StatusOK, t) </del> <ins> responses.Wait() if err := expectHTTP(server.URL, http.StatusOK); err != nil { t.Error(err) } </ins> } func TestReadOnly(t *testing.T) {

Related discussion:
text: Have only seen this once, personally.
This flaked in Jenkins as well. [job/kubernetes-test-go/5351]
can you take a look at this? I think has a lot already :)
Code change:
}, func(client kubeclientset.Interface, obj pkgruntime.Object) error { configmap := obj.(*apiv1.ConfigMap) <del> err := client.Core().ConfigMaps(configmap.Namespace).Delete(configmap.Name, &metav1.DeleteOptions{}) </del> <ins> orphanDependents := false err := client.Core().ConfigMaps(configmap.Namespace).Delete(configmap.Name, &metav1.DeleteOptions{OrphanDependents: &orphanDependents}) </ins> return err })

Related discussion:
text: Problem: When users sends a cascading deletion request to federation i.e they set orphanDependents=false, then federation should also set orphanDependents=false, when it sends the request to underlying clusters.
Fixed by
Code change:
serviceAccountName: fluentd-es containers: - name: fluentd-es <del> image: gcr.io/google-containers/fluentd-elasticsearch:v2.0.0 </del> <ins> image: gcr.io/google-containers/fluentd-elasticsearch:v2.0.1 </ins> env: - name: FLUENTD_ARGS value: --no-supervisor -q

Related discussion:
text: <!-- This form is for bug reports and feature requests ONLY! If you're looking for help check and the . --Is this a BUG REPORT or FEATURE REQUEST?: What happened: [root EFK]# kubectl logs -f fluentd-es-jrsf7 -n kube-system /var/lib/gems/2.1.0/gems/ffi- block in ffilib': Could not open library '': cannot open shared object file: No such file or directory. (LoadError) Could not open library '': cannot open shared object file: No such file or directory. Could not open library 'libsystemd-': cannot open shared object file: No such file or directory. Could not open library 'libsystemd-': libsystemd- cannot open shared object file: No such file or directory from /var/lib/gems/2.1.0/gems/ffi-map' from /var/lib/gems/2.1.0/gems/ffi- ffilib' from /var/lib/gems/2.1.0/gems/systemd-journal-<module:Native' from /var/lib/gems/2.1.0/gems/systemd-journal- &lt;module:Id128' from /var/lib/gems/2.1.0/gems/systemd-journal-<module:Systemd' from /var/lib/gems/2.1.0/gems/systemd-journal- &lt;top (required)' fromrequire' from require' from /var/lib/gems/2.1.0/gems/systemd-journal-<top (required)' from require' fromrequire' from /var/lib/gems/2.1.0/gems/systemd-journal- &lt;top (required)' fromrequire' from rescue in require' fromrequire' from /var/lib/gems/2.1.0/gems/fluent-plugin-systemd- &lt;top (required)' fromrequire' from require' from /var/lib/gems/2.1.0/gems/fluentd-block in tryloadplugin' from /var/lib/gems/2.1.0/gems/fluentd- each' from /var/lib/gems/2.1.0/gems/fluentd-tryloadplugin' from /var/lib/gems/2.1.
Code change:
serviceAccountName: fluentd-es containers: - name: fluentd-es <del> image: gcr.io/google-containers/fluentd-elasticsearch:v2.0.0 </del> <ins> image: gcr.io/google-containers/fluentd-elasticsearch:v2.0.1 </ins> env: - name: FLUENTD_ARGS value: --no-supervisor -q

Related discussion:
text: 0/gems/fluentd- newimpl' from /var/lib/gems/2.1.0/gems/fluentd-newinput' from /var/lib/gems/2.1.0/gems/fluentd- addsource' from /var/lib/gems/2.1.0/gems/fluentd-block in configure' from /var/lib/gems/2.1.0/gems/fluentd- each' from /var/lib/gems/2.1.0/gems/fluentd-configure' from /var/lib/gems/2.1.0/gems/fluentd- configure' from /var/lib/gems/2.1.0/gems/fluentd-runconfigure' from /var/lib/gems/2.1.0/gems/fluentd- runconfigure' from /var/lib/gems/2.1.0/gems/fluentd-block in start' from /var/lib/gems/2.1.0/gems/fluentd- call' from /var/lib/gems/2.1.0/gems/fluentd-mainprocess' from /var/lib/gems/2.1.0/gems/fluentd- start' from /var/lib/gems/2.1.0/gems/fluentd-<top (required)' from require' fromrequire' from /var/lib/gems/2.1.0/gems/fluentd-0.12.39/bin/fluentd:5:in &lt;top (required)' from /usr/local/bin/fluentd:23:inload' from /usr/local/bin/fluentd:23:in &lt;main' What you expected to happen: fluentd-es run success How to reproduce it (as minimally and precisely as possible): kubectl create -f fluentd-es- the image is google-containers/fluentd-elasticsearch:v2.0.0 Anything else we need to know?