jiebi/CodeConvo huggingface dataset:
Change:
accessibleTopology map[string]string, err error) { klog.V(4).Info(log("calling NodeGetInfo rpc")) <del> if c.nodeV1ClientCreator != nil { return c.nodeGetInfoV1(ctx) } else if c.nodeV0ClientCreator != nil { return c.nodeGetInfoV0(ctx) } err = fmt.Errorf("failed to call NodeGetInfo. Both nodeV1ClientCreator and nodeV0ClientCreator are nil") </del> <ins> // TODO retries should happen at a lower layer (issue #73371) backoff := wait.Backoff{Duration: initialDuration, Factor: factor, Steps: steps} err = wait.ExponentialBackoff(backoff, func() (bool, error) { var getNodeInfoError error if c.nodeV1ClientCreator != nil { nodeID, maxVolumePerNode, accessibleTopology, getNodeInfoError = c.nodeGetInfoV1(ctx) } else if c.nodeV0ClientCreator != nil { nodeID, maxVolumePerNode, accessibleTopology, getNodeInfoError = c.nodeGetInfoV0(ctx) } if nodeID != "" { return true, nil } // kubelet plugin registration service not implemented is a terminal error, no need to retry if strings.Contains(getNodeInfoError.Error(), "no handler registered for plugin type") { return false, getNodeInfoError } // Continue with exponential backoff return false, nil }) </ins> return nodeID, maxVolumePerNode, accessibleTopology, err }

Discussion:
<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks!--What happened: If kubelet plugin registration fails during the registration process, the kubelet never attempts to register the socket again (assuming the error was transient), unless the socket for the driver is deleted & recreated to trigger a new registration. What you expected to happen: Maybe retry with exponential backoff when kubelet plugin registration fails instead of just logging and waiting for another filesystem event like it does today. Of course, if a socket returns a terminal error (does not implement the kubelet plugin registration service, for example), we should not retry. Follow up on /milestone v1.14 /sig storage How to reproduce it (as minimally and precisely as possible): Anything else we need to know?: Environment: Kubernetes version (use ): Cloud provider or hardware configuration: OS (e.g. from /etc/os-release): Kernel (e.g. ): Install tools: Others: <!-- DO NOT EDIT BELOW THIS LINE --/kind bug
I'd like to work on this issue
/assign
Re opening. was a temporary work around for this. is the permanent fix.
Hello! We are starting Bug Freeze for 1.15. Is this issue still planned on being incorporated into 1.15?
the dependent PR is not yet merged. can we get this in for 1.15?
said that is the "permanent fix." and that has merged. So let's close this out /close
Closing this issue. <detailsIn response to : said that is the "permanent fix." and that has merged. So let's close this out Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
Change:
Image: framework.GetPauseImageName(f.Client), }, }, <ins> NodeSelector: map[string]string{k: v}, // only launch on our two nodes </ins> }, }) framework.ExpectNoError(err)

Discussion:
https://k8s- Failed: [] SchedulerPredicates [Serial] validates that InterPodAntiAffinity is respected if matching 2 {Kubernetes e2e suite}
https://k8s- Failed: [] SchedulerPredicates [Serial] validates that InterPodAntiAffinity is respected if matching 2 {Kubernetes e2e suite}
https://k8s- Failed: [] SchedulerPredicates [Serial] validates that InterPodAntiAffinity is respected if matching 2 {Kubernetes e2e suite}
https://k8s- Failed: [] SchedulerPredicates [Serial] validates that InterPodAntiAffinity is respected if matching 2 {Kubernetes e2e suite}
https://k8s- Failed: [] SchedulerPredicates [Serial] validates that InterPodAntiAffinity is respected if matching 2 {Kubernetes e2e suite}
https://k8s- Failed: [] SchedulerPredicates [Serial] validates that InterPodAntiAffinity is respected if matching 2 {Kubernetes e2e suite}
https://k8s- Failed: [] SchedulerPredicates [Serial] validates that InterPodAntiAffinity is respected if matching 2 {Kubernetes e2e suite}
https://k8s- Failed: [] SchedulerPredicates [Serial] validates that InterPodAntiAffinity is respected if matching 2 {Kubernetes e2e suite}
https://k8s- Failed: [] SchedulerPredicates [Serial] validates that InterPodAntiAffinity is respected if matching 2 {Kubernetes e2e suite}
https://k8s- Failed: [] SchedulerPredicates [Serial] validates that InterPodAntiAffinity is respected if matching 2 {Kubernetes e2e suite}
https://k8s- Failed: [] SchedulerPredicates [Serial] validates that InterPodAntiAffinity is respected if matching 2 {Kubernetes e2e suite}
Looking into this...
trying to track this flake down. It looks like the pod is launched despite AntiAffinity in . In my local setup I saw the same, but noticed that the new admission controller was not activate.

I have activate it and now get this:
https://k8s- Failed: [] SchedulerPredicates [Serial] validates that InterPodAntiAffinity is respected if matching 2 {Kubernetes e2e suite}
https://k8s- Failed: [] SchedulerPredicates [Serial] validates that InterPodAntiAffinity is respected if matching 2 {Kubernetes e2e suite}
Should be fixed by
Change:
}) /* <del> Release: v1.16 </del> <ins> Release: v1.16, v1.21 </ins> Testname: StatefulSet resource Replica scaling Description: Create a StatefulSet resource. Newly created StatefulSet resource MUST have a scale of one.

Discussion:
[ ] two weeks soak start date : testgrid-link [ ] two weeks soak end date : [ ] test promotion pr : kubernetes/kubernetes#? According to this APIsnoop query, there are still some remaining StatefulsetScale endpoints that are untested. with this query, you can filter untested endpoints by their category and eligibility for conformance. e.g below shows a query to find all conformance eligible untested, stable, apps endpoints sql-mode select distinct endpoint, right(useragent,73) AS useragent from testing.auditevent -- where useragent ilike '%subresource%' where endpoint ilike '%AppsV1NamespacedStatefulSetScale%' and releasedate::BIGINT round(((EXTRACT(EPOCH FROM NOW()))::numeric)1000,0) - and useragent like 'e2e%' order by endpoint limit 30; example endpoint | useragent -----------------------------------------|--------------------------------------------------------------------------- patchAppsV1NamespacedStatefulSetScale | [StatefulSetBasic] should have a working scale subresource [Conformance] readAppsV1NamespacedStatefulSetScale | [StatefulSetBasic] should have a working scale subresource [Conformance] replaceAppsV1NamespacedStatefulSetScale | [StatefulSetBasic] should have a working scale subresource [Conformance] (3 rows) If a test with these calls gets merged, test coverage will go up by 1 point* This test is also created with the goal of conformance promotion. /sig testing /sig architecture /area conformance
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
Change:
firstManifestURL := ServeCachedManifestFile(testPodSpecFile) secondManifestURL := ServeCachedManifestFile(testManifestFile) <del> apiServerURL, configFilePath := startComponents(firstManifestURL, secondManifestURL, apiVersion) </del> <ins> apiServerURL, _ := startComponents(firstManifestURL, secondManifestURL, apiVersion) </ins> // Ok. we're good to go. glog.Infof("API Server started on %s", apiServerURL)

Discussion:
Probably not the same as ? To for triage
Shouldn't this fixed by ?
this has nothing to do with . kubelet is trying to create the mirror pod over and over again because it still hasn't observed the mirror pod from watch. It is probably the same as # .
I thought they might be different because this one has a bunch of "failed creating ... already exists" messages in it, but doesn't.
It is essentially the same problem -- the watch was delayed so kubelet kept seeing the old snapshot of the pods where the mirror didn't exist. That's why it's flaky and only happens on shippable (which I suspect has poorer performance) :( This becomes worse every time a new test is . I think we need to limit the number of concurrent tests in
limits the number of concurrent tests on travis/shippable to 4. Let's see if this helps reduce the timeout flakes.
Another example:
Here is yet another example: Retitled the issue since limiting the concurrent tests doesn't work. It could be that the server is heavily loaded and the test timed out, or there was a delay/drop in kubelet's pod watch stream.
Now example:
If, for whatever reason, a watch update for pod creation is dropped, kubelet would need to wait until re-listing to notice that. This would certainly cause the test to time out. would it be useful to shorten the re-listing interval for tests like this?
I am not quite sure how this works--let's chat about it tomorrow?
From , we can see that kubelet was lightly loaded as the update channel was mostly empty while waiting. This corroborates the theory that watch might be at fault.
cc: quinton-hoole
The WIP reflector benchmark () may shed some light on this issue once it's integrated with the continuous e2e performance suite. I will keep this issue open for now to gather more data (since this is not reproducible locally at all), and to see if there are other potential causes.
The PR above disables the static pod test temporarily so that we can track down the root cause without spamming people with the failures. Note that other tests in the same file () may still trigger the flaky failure, though less frequently.
Can this be closed yet?
No, integration test is still flaky. See for example.

Many tests in rely on checking the pod status with a preset timeout. I disabled one test which failed most often, but other tests still fail from time to time.
The issue has been inactive for months, and there have not been any new reports. Closing.
Change:
<del> check process kube-apiserver with pidfile /var/run/kube-apiserver.pid group kube-apiserver start program = "/etc/init.d/kube-apiserver start" stop program = "/etc/init.d/kube-apiserver stop" if failed host 127.0.0.1 port 8080 protocol http request "/index.html" then restart </del>

Discussion:
cluster/saltbase/salt/monit/kube-apiserver appears not to be referenced by and so can be deleted, I think
/cc
created PR thanks!
Change:
package util import ( <ins> "net" </ins> "regexp" )

Discussion:
When spinning up VMs w/ kube, I attempted to use but this fails. ironically, 127.0.0.1 succeeds (internally you can then access a kube service via , for example, but you cannot do this if you try to bind to 0.0.0.0. binding works to 127.0.0.1, but is a horrible idea (since this IP wont be reachable by the outside world), so in that case kube should print a clear message on why this is the case and fail. binding to 0.0.0.0 fails right out of the bat, so in this case, also, kube should fail immediately. To be clear, im suggesting that these should fail at the service submission layer. Possibly a module for validating logic of submitted services/pods , which extends syntactical validation is in order if one doesn't already exist?
Please also see
Since this is very similar to I'm gonna take this.
Change:
if err != nil { return ControllerManagerMetrics{}, err } <del> output, err := g.getMetricsFromPod(g.client, metricsProxyPod, metav1.NamespaceSystem, kubeControllerManagerPort) if err != nil { return ControllerManagerMetrics{}, err } </del> return parseControllerManagerMetrics(output) }

Discussion:
! /kind failing-test <!-- If this is a flake, please add: /kind flake --/sig instrumentation /assign
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
The flake happened in before.
This is fixed. !
/reopen This is still failing. Sample run:
Reopened this issue. <detailsIn response to : /reopen Sample run: Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/priority critical-urgent
The error message in makes me think the apiserver is not available during that test run: . Can someone link to the frequency graphs to see if the problem is widespread?
got it:
I don't think I was right above. I think the "the server is currently unable to handle the request" is an error coming from the proxy. The proxy was around the time this issue was filed (), so it seems likely to be related to it. can you take a look?
This is a known issue, and is working on it . Now that is being blocked by the debugging handler problem, could we merge to mitigate this problem first? cc
I would prefer to reach a decision about enabling the debug handlers, but if that's not possible, then merging PR as mitigation might make sense.
Think I hit this also.
It appears to still be failing based on
Why rebase? Any new test run will be based on merge with current k/k master and thus contain
<img width="1111" alt="Screen Shot 2021-05-28 at 13 04 52" src="https://user-" looks like it has become more stable now.
/close
Closing this issue. <detailsIn response to : /close Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
Change:
e2enode "k8s.io/kubernetes/test/e2e/framework/node" e2epod "k8s.io/kubernetes/test/e2e/framework/pod" "k8s.io/kubernetes/test/e2e/manifest" <ins> e2ereporters "k8s.io/kubernetes/test/e2e/reporters" </ins> testutils "k8s.io/kubernetes/test/utils" utilnet "k8s.io/utils/net"

Discussion:
What would you like to be : I would like a method of seeing more clear data while tests are running, e.g.: How many tests will be run, have been run, have passed, have failed, etc. Name of the tests passed/failed/skipped Why is this needed: While a run is going we only have the output from stdout to investigate. There is a LOT of noise in those logs which is great for debugging after the fact, but difficult to consume most of the time. you can't tell how many tests have been run or get a sense of how much time is left you cant easily get a list of all tests that have failed so far My proposal Adding a new, custom ginkgo reporter that will post the data to a given URL (passed via a flag) if the URL isn't set, the reporter is a noop and doesn't waste time/resources Use cases: A user runs the tests dumps stdout to a file for debugging later, meanwhile they run a local server that will just echo the status updates it gets. At any point they look at the output they can get good data about what would be run, how far along it is, how successful it has been so far. Using the conformance test image and Sonobuoy, the Sonobuoy worker can act as the server, taking those status updates and reporting them to the aggregator. This would allow Sonobuoy to present that information to users on demand or stream it depending on the context. The work of adding this is pretty minimal and is clearly useful. In the case of Sonobuoy; this has been the most requested feature for a long time showing that people in the community really do want to have a better high-level picture of the progress of these tests. I think emitting the data to an endpoint like this is a flexible implementation that can help any consumer of the tests (not just Sonobuoy).
/sig testing /area test /area test-framework
One notification URL should work fine. Can you give me details on the data model for the status data to be published?
That is one, slightly TBD area. The reporter in the test framework has a bunch of data and I'm not sure what we exactly Sonobuoy may want from any given plugin (not just the e2e plugin).

I don't want this implementation to be overly Sonobuoy specific or overly e2e specific. Will update when something more clear is pinned down.
Brought this up in the testing commons meeting and it was agreed that this approach seemed reasonable and that the customReporters seemed like the only way to generate this data. In addition, it was requested that we find a way to also emit this (or some variation) into stdout/logs so that it is easier for a human to navigate those verbose logs.
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
/remove-lifecycle stale
Change:
INSTANCE_PREFIX="${KUBE_AWS_INSTANCE_PREFIX:-kubernetes}" CLUSTER_ID=${INSTANCE_PREFIX} AWS_SSH_KEY=${AWS_SSH_KEY:-$HOME/.ssh/kube_aws_rsa} <ins> CONFIG_CONTEXT="${KUBE_CONFIG_CONTEXT:-aws_${INSTANCE_PREFIX}}" </ins> IAM_PROFILE_MASTER="kubernetes-master" IAM_PROFILE_NODE="kubernetes-minion"

Discussion:
I've run to create a cluster, which resulted in a cluster and user being created in kube-config. Now I want to run again with different parameters, to create a second cluster in a different AWS account. I would like to use a different name for the cluster and user for this cluster. From what I can see from the , there seems to be no way to completely override the name used. I understand that I can use a separate kube-config file, but I don't want to do that. I want to use the same kube-config file, with multiple clusters and users, all handled by .
Manually overwriting file to set CONTEXT to a different value works to achieve the effect that I want. This value should be allowed to be set externally so that we don't have to resort to manually editing the file for each cluster.
Fixed in
Change:
return volumetypes.GeneratedOperations{ OperationFunc:     mountVolumeFunc, EventRecorderFunc: eventRecorderFunc, <del> CompleteFunc:      util.OperationCompleteHook(util.GetFullQualifiedPluginNameForVolume(volumePlugin.GetPluginName(), volumeToMount.VolumeSpec), "volume_mount"), }, nil </del> <ins> CompleteFunc:      util.OperationCompleteHook(util.GetFullQualifiedPluginNameForVolume(volumePluginName, volumeToMount.VolumeSpec), "volume_mount"), } </ins> } func (og *operationGenerator) resizeFileSystem(volumeToMount VolumeToMount, devicePath, deviceMountPath, pluginName string) (simpleErr, detailedErr error) {

Discussion:
<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks!--What happened: In this test, volume mount failed because the driver-registrar took a minute to come up. However, it seems like volume mount operation is retrying without any sort of backoff. This message appears in 400+ times in less than a minute. What you expected to happen: Operation executor should not be retrying operations so frequently. <!-- DO NOT EDIT BELOW THIS LINE --/kind bug
I'll take a look at this. /assign
Problem is that reconciler depends on operation executor to do backoff. In the normal case operation executor returns a method that does the actual work. Any errors by that method result in exponential backoff. But in this case, the error is coming from the code creating the mount method which is not handled by the back off logic, and is called again and again by the reconciler resulting in log and event spam. This didn't use to be a problem because initialization errors were uncommon (plugins were built in), but with CSI the driver may not exist and these initialization errors are expected. I'll look in to cleaning this up.
Is this something that will affect users in 1.13 and needs CPing?
This will result in kubelet log spam and possibly kubernetes event spam. It happens when someone creates a pod referencing a CSI volume before the CSI driver for that volume is installed. Which happens frequently in E2E tests because they install/remove driver with every test. But should be rare in real world usage since most users are likely to install a driver first before using it.
If release team is worried about this, we could create a quick PR to suppress logging/event generation of this these errors. Otherwise we plan to address this more holistically for 1.14 and 1.13.1.
Change:
} return TestContext.GCEUpgradeScript } <ins> func waitForSSHTunnels() { Logf("Waiting for SSH tunnels to establish") RunKubectl("run", "ssh-tunnel-test", "--image=gcr.io/google_containers/busybox:1.24", "--restart=Never", "--command", "--", "echo", "Hello") defer RunKubectl("delete", "pod", "ssh-tunnel-test") // allow up to a minute for new ssh tunnels to establish wait.PollImmediate(5*time.Second, time.Minute, func() (bool, error) { _, err := RunKubectl("logs", "ssh-tunnel-test") return err == nil, nil }) } </ins>

Discussion:
Upgrade fails with: Error from: https://k8s- Related failures: https://k8s- https://k8s-
/cc
might require how do the GKE tests interact with the defaults you set up?
Looks like are pretty isolated to upgrade tests.
Also we didn't start seeing this until the ABAC default change merged so I think change might help
it will definitely help with the GCE jobs mentioned in the description of
I haven't looked at the logs, so sorry if this is just noise, but the symptom (couldn't get logs b/c pod doesn't exist) looks very similar to what happened when the kubectl taint e2e tests were running in parallel instead of serial.
Judging by the comment and digging into the history , this used to be wrapped in a retry loop. It was removed in PR
it looks like this is limited to gke jobs in the upgrade tests. since getting pod logs depends on master-node communications, it would be affected by the SSH tunnels issue ()
Not blocking under guidance
Change:
run(ctx, filterInitializers(controllerInitializers, leaderMigrator.FilterFunc, leadermigration.ControllerMigrated)) }, OnStoppedLeading: func() { <del> klog.Fatalf("migration leaderelection lost") </del> <ins> klog.ErrorS(nil, "migration leaderelection lost") klog.FlushAndExit(klog.ExitFlushTimeout, 1) </ins> }, }) }

Discussion:
kube-controller-manager yields whole stack trace when it loses leaderelection. kube-controller-manager to only log and to exit (without logging 17K lines of stack trace that is not useful for anyone). kube-controller-manager yields around 17K lines of stack trace (omitted in the above example). No response <detailsv1.20.12 </details<details</details<details</details<details</details<details</details<details</details
/sig api-machinery
/assign /triage accepted
has Fatalf always triggered a stack dump of all goroutines?
/cc
Seems like does a dump of all goroutines Could we and then maybe? /cc in case you have thoughts about this
If I read the right, it only printed the current goroutine to stderr. Dumping everything seems to have been later. make sense to me.
Dumping of all goroutines was in
I have similar doubts. I want to fix it. /assign
kube-scheduler seems to have a similar issue (but a shorter stack trace)
When replacing (useful because it doesn't do structured logging, not just because of the stack trace problem), remember to use + the new . Updated migration instructions are pending in There's also - reverting the klog change might be a faster way to address this.