jiebi/CodeConvo huggingface dataset:
Code change:
t.Error(err) } <del> expectPath(t, swagger, "/apis/apiservicegroup/v1") </del> <ins> expectPath(t, swagger, "/apis/apiservicegroup/v1/path1") </ins> expectPath(t, swagger, "/apis/apiregistration.k8s.io/v1") t.Log("Update APIService OpenAPI")

Related discussion:
text: In our staging cluster I tried bringing back the in the custom metrics server configuration and restarting all the masters, and the problem did not come back, everything seems OK now. So it maybe isn't that easy to reproduce
After updating the configmap, restarting the prometheus-adapter pod to pickup the changes and restarting the apiserver are both required to induce the issue. I have been able to deterministically reproduce this following those steps, please follow up here if you are still having issues.
That's good to hear. I'll need some more help with reproducing it. I've been able to verify that restarting the prometheus adapter pod picks up the configmap changes but restarting the apiserver hasn't caused any crashes for me. I've tested this both on the latest master branch and 1.28.
While I'm yet to fully understand these changes, the 2 PRs below (both of which went into 1.28) seem likely candidates for a regression here: - introduced lazy loading in openAPI v2 handler to load/compute the aggregator spec when requested by client - switched the spec merging function from to any clue if one of the above changes could be causing mischief?
Also from example request at the top, the impact seems to be to the v2 API: Nathan - when you get a chance, could you also confirm if v1 API doesn't run into this?
OpenAPI only supports v2 and v3.
hi, any clue about this issue?
It's possible but I haven't been able to reproduce it. Can I ask if this is happening on cloud providers outside AWS?
Here are the apiservices and relevant pod objects with a working repro from these steps If it's useful, I can provide yamls for these objects in addition to the settings applied in the prometheus-adapter configmap. Can you confirm that the pods backing your prometheus-adapter and metrics-server are running?
I meet this problem
We the problem in GCP as well.
ACK in Alibaba Cloud also meets this problem
We were able to fix the v2 availability by and re-installing the kube-prometheus helm chart. Unfortunately, after restarting the k8s api server, v2 availability is broken again.
yes, I'm running on kind. I'll do some more digging, assuming it's related to those PRs linked by but still haven't pinpointed anything.
Code change:
t.Error(err) } <del> expectPath(t, swagger, "/apis/apiservicegroup/v1") </del> <ins> expectPath(t, swagger, "/apis/apiservicegroup/v1/path1") </ins> expectPath(t, swagger, "/apis/apiregistration.k8s.io/v1") t.Log("Update APIService OpenAPI")

Related discussion:
text: I tried to reproduce on my local setup with based on steps + , but I could still get a successful response when I executed . Then I reverted one of the step mentioned in to and waited until the new pod came alive and then did a apiserver restart and I was able to reproduce the same thing on as well i.e; I get the 503 response when I execute following command after the fact. After this point, even if i change it back to and restart the apiserver , the error is persistent. Let me know if this works for you to reproduce ? PS: Also, i want to call out that there is no issue with , its .
Thanks, alright I found a way to reproduce it consistently without restarting the apiserver. Still investigating the root cause. Thanks for the detailed reproduction and
Quick update: This issue was a combination of bugs both in k/k and prometheus adapter. I opened to mitigate on k/k side which should fix the issue. The issue was difficult to reproduce because it requires a very specific ordering of OpenAPI merging to trigger, which is why I suspect many people were able to reproduce after some restarts to kube-apiserver. Prometheus and metrics server serve differing OpenAPIs for the endpoint. Prometheus also doesn't seem to be registering discovery's OpenAPI properly (missing OpenAPI for but having the OpenAPI for whereas it is all present in metrics server. This inconsistency in how prometheus and metrics server handle the OpenAPI surfaced the merging issue.
a little wired question: why do we need to restart kube-apiserver to reproduce this problem?
Tried to reproduce the issue after cherry picking the fix and didn't run into below error. Fix is working as expected.
It can look like restarting prometheus-adapter "fixes" the issue? I am also curious about suggestion of removing the section from the config map works as a temporary workaround until released upstream (we use and cannot delete the apiregistration)? As I understand the fix for this issue is currently in
Code change:
expectedList: []HostPriority{{"machine1", 0}, {"machine2", 0}}, test:         "no resources requested", pods: []api.Pod{ <del> {CurrentState: machine1State, Labels: labels2}, {CurrentState: machine1State, Labels: labels1}, {CurrentState: machine2State, Labels: labels1}, {CurrentState: machine2State, Labels: labels1}, </del> <ins> {DesiredState: machine1State, Labels: labels2}, {DesiredState: machine1State, Labels: labels1}, {DesiredState: machine2State, Labels: labels1}, {DesiredState: machine2State, Labels: labels1}, </ins> }, }, {

Related discussion:
text: When creating replication controller (e.g. frontend php-redis from guestbook example), scheduler gets stuck and never recovers. This is a race, I have to run several times to see the error (A script to delete controller and pods, then create new controller, leave enough time in between for k8s to react). I've created PR for an attempted fix.
I saw this too while following the guest book and creating the php/frontend step. In my case, I had 2 minions while the replication controller configuration had configured 3 replicas, so 2 pods got scheduled -each on another minion, the third pod had empty Host (saw that when doing 'list pods'), and the 3 of them were stuck in "waiting" status. I deleted the controller and the pods, fixed the configuration in json file to be 2 replicas and recreated the controller and everything worked. Does your fix take care of this use case? (too many replicas, too few minions/one of the pods is not scheduled)
This is different from your use case. If you have more replicas than minions, then the scheduler will get stuck for sure (with the error msg probably being 'failed to find a fit for pod'). AFAICT, it's not support yet.
If you remove the HostPorts, it should work for you. thanks for report.
, thanks for your replies. I understand why it isn't working, but I'd expect one of the 2 options to happen instead of the current situation: the creation of replication controller if there are less minions than replicas, AND there is a hostPort property. In other words - validation of the content during creation. least schedule all replicas that can get a minion (in my case 2 out of 3) and leave only one at "waiting". Is there an issue open about the use case I describe?
should be true today. is less possible than it seems like-- minions can come and go, and become over or under loaded in the time between creation and when the pod shows up at the scheduler. So even if we check up front, we still have to check again later, and sometimes the first check will pass but not the second.
Code change:
import ( "context" "fmt" <del> "k8s.io/kubernetes/pkg/apis/core/v1/helper/qos" "k8s.io/kubernetes/pkg/kubelet/apis/config" e2eskipper "k8s.io/kubernetes/test/e2e/framework/skipper" "k8s.io/kubernetes/test/e2e/nodefeature" imageutils "k8s.io/kubernetes/test/utils/image" </del> "math/big" "path/filepath" "strconv" "strings" "time" <ins> "k8s.io/kubernetes/pkg/apis/core/v1/helper/qos" "k8s.io/kubernetes/pkg/kubelet/apis/config" e2eskipper "k8s.io/kubernetes/test/e2e/framework/skipper" "k8s.io/kubernetes/test/e2e/nodefeature" imageutils "k8s.io/kubernetes/test/utils/image" </ins> "github.com/onsi/ginkgo/v2" "github.com/onsi/gomega" v1 "k8s.io/api/core/v1"

Related discussion:
text: master-informing: ci-crio-cgroupv1-node-e2e-conformanceE2eNode Suite.[It] [sig-node] Swap [LinuxOnly] [NodeFeature:NodeSwap] [NodeConformance] with configuration QOS Best-effort E2eNode Suite.[It] [sig-node] Swap [LinuxOnly] [NodeFeature:NodeSwap] [NodeConformance] with configuration QOS Burstable E2eNode Suite.[It] [sig-node] Swap [LinuxOnly] [NodeFeature:NodeSwap] [NodeConformance] with configuration QOS Burstable with memory request equals to limit E2eNode Suite.[It] [sig-node] Swap [LinuxOnly] [NodeFeature:NodeSwap] [NodeConformance] with configuration QOS Guaranteed After was merged No response /sig node
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/assign
/cc
Code change:
} return c.Name, nil } <ins> // reattachMessage returns a message to print after attach has completed, or // the empty string if no message should be printed. func (o *AttachOptions) reattachMessage(containerName string, rawTTY bool) string { if o.Quiet || !o.Stdin || !rawTTY || o.Pod.Spec.RestartPolicy != corev1.RestartPolicyAlways { return "" } if _, path := podcmd.FindContainerByName(o.Pod, containerName); strings.HasPrefix(path, "spec.ephemeralContainers") { return fmt.Sprintf("Session ended, the ephemeral container will not be restarted but may be reattached using '%s %s -c %s -i -t' if it is still running", o.CommandName, o.Pod.Name, containerName) } return fmt.Sprintf("Session ended, resume using '%s %s -c %s -i -t' command when the pod is running", o.CommandName, o.Pod.Name, containerName) } </ins>

Related discussion:
text: When attaching to an ephemeral container, prints a message suggesting that the container can be reattached after it exists. This is not true because an ephemeral container is never restarted. This cosmetic issue should be resolved for
/assign /sig node /triage accepted /priority important-longterm
Code change:
This is enforced throughout this document. <del> <img src="http://kubernetes.io/img/warning.png" alt="WARNING" width="25" height="25"> Current pod descriptors use an `emptyDir` for storing data in each data node container. This is meant to be for the sake of simplicity and [should be adapted according to your storage needs](../../../docs/design/persistent-storage.md). </del> <ins> <img src="http://kubernetes.io/kubernetes/img/warning.png" alt="WARNING" width="25" height="25"> Current pod descriptors use an `emptyDir` for storing data in each data node container. This is meant to be for the sake of simplicity and [should be adapted according to your storage needs](../../../docs/design/persistent-storage.md). </ins> ## Docker image

Related discussion:
text: This includes but effects multiple docs.
This issue was also raised on the repository, see:
What's needed to fix this? Are the broken headers being generated automatically somewhere?
- The warning section is pre-pended to the md docs from the following script The fixes this issue, updating this script and all the docs... However, there is an open question as to whether "old" docs should be removed from this repo, which has prevented this PR from being merged.
Code change:
} func getNodeDevices() (*podresourcesapi.ListPodResourcesResponse, error) { <del> endpoint := util.LocalEndpoint(defaultPodResourcesPath, podresources.Socket) </del> <ins> endpoint, err := util.LocalEndpoint(defaultPodResourcesPath, podresources.Socket) if err != nil { return nil, fmt.Errorf("Error getting local endpoint: %v", err) } </ins> client, conn, err := podresources.GetClient(endpoint, defaultPodResourcesTimeout, defaultPodResourcesMaxSize) if err != nil { return nil, fmt.Errorf("Error getting grpc client: %v", err)

Related discussion:
text: What happened: I replaced the kubelet on one of my 1.14 nodes with a build from master, and now I'm getting this error starting up. Test passes since 5/31/2019 commit have been failing on testgrid. If you look at the logs, the Windows nodes are not coming up. What you expected to happen: No errors starting up kubelet How to reproduce it (as minimally and precisely as possible): Build master, then try to run the kubelet on Windows Anything else we need to know?: /sig windows Environment: Kubernetes version (use ): Cloud provider or hardware configuration: Azure OS (e.g: ): Windows Server 2019 Kernel (e.g. ): 10.0..529 Install tools: aks-engine
cc /priority critical-urgent
/milestone v1.15 Release team - it looks like a recent PR broke windows nodes. We will need a fix here for 1.15
Ok, so the problem here is a little more complicated and we're actually seeing a bug that is actually in the code for quite some time, just that we never used that particular code path. But first: TL;DR: Quick fix is to start the kubelet on Windows with: --feature-gates="KubeletPodResources=false" Now, as to why this is happening: PR [1] was merged 3 days ago and it graduates KubeletPodResources feature as Beta. That means kubelet will enable it by default.With the introduction of the above feature, this branch of code is entered when starting kubelet [2]. We can see a call to the function: defined here [3].While innocent enough, gets the necessary npipe by calling: . is defined here [4] for Windows . The problem is actually that url.URL can't handle Windows style paths [5] and will replace "" with "%5C" as it's to be expected. The paths are transformed along the way from the default "/var/lib/kubelet" to "varlibkubelet" by defined here [6]. More specifically in the use of to create a path. It's inputs are unix style but returns the path Windows style as it's to be expected.The solution is to replace the usage of url.URL in with some alternative package or just implement it.
Code change:
} func getNodeDevices() (*podresourcesapi.ListPodResourcesResponse, error) { <del> endpoint := util.LocalEndpoint(defaultPodResourcesPath, podresources.Socket) </del> <ins> endpoint, err := util.LocalEndpoint(defaultPodResourcesPath, podresources.Socket) if err != nil { return nil, fmt.Errorf("Error getting local endpoint: %v", err) } </ins> client, conn, err := podresources.GetClient(endpoint, defaultPodResourcesTimeout, defaultPodResourcesMaxSize) if err != nil { return nil, fmt.Errorf("Error getting grpc client: %v", err)

Related discussion:
text: [1] [2] [3] [4] [5] [6]
Thanks for the analysis! I was a bit concerned around the fact that this started showing up around my commit: but the root cause here is indeed, as you said, the graduation of feature to Beta and the way the invalid escape sequences are being introduced.
GCE tests failing too /cc
I'm trying a quick to see if anything else is broken. There are other functions in that same file doing that substitution already so it may make sense to use that as a workaround until a better solution is available.
Still working on it - looks like the path to create the directories is also busted. With still getting
Code change:
cancel() // mutate request Headers // Authorization filter does it for example <del> for j := 0; j < 10000; j++ { req.Header.Set("Test", "post") </del> <ins> for { select { case <-testDone: return default: req.Header.Set("Test", "post") } </ins> } }), func(r *http.Request, requestInfo *request.RequestInfo) bool {

Related discussion:
text: Similar to and easily reproducible. The handler occasionally completes and sends a response before timing out. /kind failing-test /kind flake /sig api-machinery
/assign
/assign Thank you very much for fixing these tests.
/triage accepted
Code change:
sleepingPod := getSleepingPod(f.Namespace.Name) sleepingPod = runPodAndWaitUntilScheduled(f, sleepingPod) <del> gomega.Expect(isPodCgroupV2(f, sleepingPod)).To(gomega.BeTrueBecause("node uses cgroup v1")) </del> <ins> if !isPodCgroupV2(f, sleepingPod) { e2eskipper.Skipf("swap tests require cgroup v2") } </ins> nodeName = sleepingPod.Spec.NodeName gomega.Expect(nodeName).ToNot(gomega.BeEmpty(), "node name is empty")

Related discussion:
text: master-informing: ci-crio-cgroupv1-node-e2e-conformanceE2eNode Suite.[It] [sig-node] Swap [LinuxOnly] [NodeFeature:NodeSwap] [NodeConformance] with configuration QOS Best-effort E2eNode Suite.[It] [sig-node] Swap [LinuxOnly] [NodeFeature:NodeSwap] [NodeConformance] with configuration QOS Burstable E2eNode Suite.[It] [sig-node] Swap [LinuxOnly] [NodeFeature:NodeSwap] [NodeConformance] with configuration QOS Burstable with memory request equals to limit E2eNode Suite.[It] [sig-node] Swap [LinuxOnly] [NodeFeature:NodeSwap] [NodeConformance] with configuration QOS Guaranteed After was merged No response /sig node
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/assign
/cc
Code change:
// Important to wait here to ensure we prime the cache with the initial list // of documents in order to exercise 304 Not Modified <del> require.True(t, waitForEmptyQueue(testCtx.Done(), aggregatedManager)) </del> <ins> require.True(t, waitForQueueComplete(testCtx.Done(), aggregatedManager)) </ins> <del> // Now add all groups. We excluded one group before so that AllServicesSynced // could include it in this round. Now, if AllServicesSynced ever returns </del> <ins> // Now add all groups. We excluded one group before so that waitForQueueIsComplete // could include it in this round. Now, if waitForQueueIsComplete ever returns </ins> // true, it must have synced all the pre-existing groups before, which would // return 304 Not Modified for _, s := range apiServices {

Related discussion:
text: pull-kubernetes-unit sometimes fails with data race. TestDirty I think the test was 3 months ago with the flaking state. There are two issues. First, updating and reading a bool-type variable is not thread-safe. and there is also another problem checking the state before completely syncing it. No response /sig api-machinery
/assign
/triage accepted
Code change:
if e.stdout != nil { waitCount++ <del> headers.Set(api.StreamType, api.StreamTypeStdout) remoteStdout, err := conn.CreateStream(headers) if err != nil { return err } defer remoteStdout.Reset() </del> go cp(api.StreamTypeStdout, e.stdout, remoteStdout) } if e.stderr != nil && !e.tty { waitCount++ <del> headers.Set(api.StreamType, api.StreamTypeStderr) remoteStderr, err := conn.CreateStream(headers) if err != nil { return err } defer remoteStderr.Reset() </del> go cp(api.StreamTypeStderr, e.stderr, remoteStderr) }

Related discussion:
text: We create the stdin stream and immediately start copying local stdin to the container. We do the same for stdout and stderr (create stream, start copying immediately). In the kubelet, we wait to receive all the streams before we start the exec and copy between the streams and the container. The spdy connection frame handling code has 5 frame workers, each with a queue of up to 50 frames. In the event that we try to push more than 50 frames to a single queue before all streams are ready, the 51st attempt to push a frame blocks until there is space available in the queue. But there won't be any space until the server starts reading from the streams, which won't happen until all the streams have been received, so we're stuck at this point. The fix is to create all the streams first, and only start copying after all the streams have been created and the server is ready to copy between the streams and the container. Originally reported here:
FYI
This seems pretty high priority.
It's an easy fix. has already tested the patch for OpenShift and it works. Are we willing to pull it in before is resolved, as that will take longer to put in place?
Here's the patch, for reference:
5 * 50 = 250 frames before you get stuck? Or did I misunderstand?
Oh, I guess all the frames go to the same worker for some reason?
anyway, I'm fine with taking this and cherry picking it. Order doesn't matter; someone has to rebase either way, right?
frames are assigned to each queue based on stream id, so if you sent 50 frames to the stdin stream, that queue fills up and the loop that processes incoming frames is blocked until something starts reading from the stream. Working on a PR now.
Gotcha, thanks. , Andy Goldstein wrote:
an update: I spent some time trying to create an e2e that would fail more often than not without this fix, but I haven't had any luck so far (I have it stream a copy of the kubectl binary from the local system to the pod). The issue that originally uncovered the problem was from running exec on a Windows system, where it reproduces consistently, but I haven't seen it fail much on Linux. I will keep trying...
Code change:
// services that need to be synced queue           *workqueue.Type knownClusterSet sets.String <ins> // endpoint worker map contains all the clusters registered with an indication that worker exist // key clusterName endpointWorkerMap map[string]bool // channel for worker to signal that it is going out of existence endpointWorkerDoneChan chan string // service worker map contains all the clusters registered with an indication that worker exist // key clusterName serviceWorkerMap map[string]bool // channel for worker to signal that it is going out of existence serviceWorkerDoneChan chan string </ins> } // New returns a new service controller to keep DNS provider service resources

Related discussion:
text: <!-- Thanks for filing an issue! Before hitting the button, please answer these questions.--Is this a request for help? (If yes, you should use our troubleshooting guide and community support channels, see ): No Is this a BUG REPORT or FEATURE REQUEST? (choose one): BUG REPORT <!-- If this is a BUG REPORT, please: Fill in as much of the template below as you can. If you leave out information, we can't help you as well. If this is a FEATURE REQUEST, please: Describe in detail the feature/behavior/change you'd like to see. In both cases, be ready for followup questions, and please respond in a timely manner. If we can't reproduce a bug or think a feature already exists, we might close your issue. If we're wrong, PLEASE feel free to reopen it and explain why. --Kubernetes version (use ): Environment: Cloud provider or hardware configuration: GCP OS (e.g. from /etc/os-release):Kernel (e.g. ):Install tools: Others: What happened: Memory usage of federation-control-manager keeps on increasing to very large size overtime. What you expected to happen: Memory usage is stable How to reproduce it (as minimally and precisely as possible): Register one or more cluster to federation control plane and watch the top output for the federation-control-manager Also can check the pprof, here is the one such output after federation-control-manager is run for an hour. We can see there are around goroutines created in an hour which are leaked and would consume memory. Anything else do we need to know: I do have a fix for the issue and would be happy to submit it, if this issue is accepted.
plz check the issue
feel free to just submit a pull request, we don't really have a concept of an issue being "accepted" :)
This looks like a legitimate bug. could you please clarify what version of the federated service controller you are running? The image path and version of the container, or a git hash of the source should be sufficient.
i am taking the latest code from master branch.