jiebi/CodeConvo huggingface dataset:
Change:
} for vmssName := range vmssNamesMap { <del> vmss, err := ss.GetScaleSetWithRetry(service, ss.ResourceGroup, vmssName) </del> <ins> vmss, err := ss.getVMSS(vmssName, cacheReadTypeDefault) </ins> // When vmss is being deleted, CreateOrUpdate API would report "the vmss is being deleted" error. // Since it is being deleted, we shouldn't send more CreateOrUpdate requests for it.

Discussion:
<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --What happened: Recently we upgraded our cluster from 1.14.4 to 1.14.7. Then we started to observe tons of following errors: I guess the cause may be related to this line which was introduced recently: As we have hundreds of services with LoadBalancer type, and we also enable cluster-autoscaler, the nodes list change very often. For each service, there will be one call to GetScaleSet. The quota will be exhausted quickly. When fails, new nodes created by cluster-autoscaler will lose connectivity to load balancer. What you expected to happen: Kubernetes should not hit VMSS throttling threshold. How to reproduce it (as minimally and precisely as possible): a cluster with aks-engine. Use VMSS. 200 services with LoadBalancer type. up VMSS to 50 nodes down VMSS to 60 nodes scaling up and down VMSS Anything else we need to know?: Environment: Kubernetes version (use ): 1.14.7 Cloud provider or hardware configuration: Azure, VMSS OS (e.g: ): Ubuntu Kernel (e.g. ): Install tools: Network plugin and version (if this is a network-related bug): Others:
/sig cloud-provider /area provider/azure
I create a PR to mitigate the issue.
do we have a rough num about how this PR would mitigate this issue? looks like we need to cherry-pick this PR to old releases. it's better if we have a rough estimation.
The revised version of code calls instead of azure API. The cache refreshes itself every 10 min. I believe this will solve the throttling issue.
I believe we're being affected by this. Apart from getting the - when creating our to use a pre-configured Azure Public IP, the ingress appears to be getting a different IP each time
Thank you all!
The PR is merged. Does your issue remain?

pls cherry-pick that fix to old release, thanks
Cherry picked to release 1.14-1.16
I’ll retry today and let you know Wysłane z iPhone'a
Issue returned in K8S 1.15.4 Warning UpdateLoadBalancerFailed service/nginx-ingress-controller Error updating load balancer with new hosts map[<nodes names]: [EnsureHostInPool(ingress/nginx-ingress-controller): backendPoolID(/subscriptions/<subscription/resourceGroups/<rg/providers/Microsoft.Network/loadBalancers/<rg/backendAddressPools/<rg) - failed to ensure host in pool: "azure - cloud provider rate limited(read) for operation:VMSSGet", EnsureHostInPool(ingress/nginx-ingress-controller):
Change:
"k8s.io/apimachinery/pkg/api/resource" metav1 "k8s.io/apimachinery/pkg/apis/meta/v1" <ins> "k8s.io/apimachinery/pkg/util/sets" </ins> "k8s.io/apimachinery/pkg/util/uuid" "k8s.io/kubernetes/pkg/api/v1" "k8s.io/kubernetes/test/e2e/framework"

Discussion:
The test is failing in . https://k8s- https://k8s- https://k8s- The reason is that the test expects that all pause containers () should have . However, in many tests, we are starting test containers with . Those pause containers have . Ideally, containers started in previous test should be cleaned up, so in serial suite it should not affect later test. However, it is not always true when CRI is enabled. For example, in the oom message, we could see: There are 2 problems here: Why are containers not properly cleaned up when CRI enabled? I can't even find which test leaked pause container from the log, because there are many test using pause image, and some of them start a lot of containers, such as density test, restart test. The oom score adj test should not rely on process name . However, I can't find an alternative for now, because we don't report pause container status in our api, we don't even know the container id to inspect with docker or CRI. /cc
This test is still flaking in cri-serial. Some examples: https://k8s- https://k8s- Need to check the logs to see if the symptoms are the same.
It could be that with CRI, it takes kubelet longer to clean up. Or, even worse, that kubelet is not cleaning up properly. We need more information on this on failure. Dumping pods on the node Dumping containers () on the node. Accessing docker directly is not ideal, but it's okay because the test itself only applies to docker. Alternatively, we can get the current running containers from kubelet's /runningPods endpoint. The downside is that this may lag behind the actual state a little bit (seconds). We should also make the test suite run more frequently. It runs once a day, which makes the debugging round-trip overhead too high.
I can reproduce this process leak by running the "Network should recover from ip leaks" on my own node, without CRI enabled. The image used in the test was , and the docker version 1.11.2. The test creates 100 pods, then restarts the docker service by 6 times, with 20s interval in between.

After the test ends, there could be multiple processes running left on the node. Those processes were reparented to pid 1, and showed no containers running. /cc
On the side note, the test should not be blindly checking all processes. It should, instead, take a snapshot of the system before the test and only check the new processes created by the tests.
I filed a new issue for the docker bug since it's not unique to CRI and this test is catching it by accident: Leaving this issue to fix the test by not blindly checking all processes:
The docker bug is not a regression, and is tracked in . The problem of the test is not necessarily to be fixed in v1.6, at least it's not release blocker. :)
Change:
return res } <ins> func newEphemeralStorageResourceList(ephemeral, cpu, memory string) v1.ResourceList { res := v1.ResourceList{} if ephemeral != "" { res[v1.ResourceEphemeralStorage] = resource.MustParse(ephemeral) } if cpu != "" { res[v1.ResourceCPU] = resource.MustParse(cpu) } if memory != "" { res[v1.ResourceMemory] = resource.MustParse("1Mi") } return res } </ins> func newResourceRequirements(requests, limits v1.ResourceList) v1.ResourceRequirements { res := v1.ResourceRequirements{} res.Requests = requests

Discussion:
It is possible to have multiple signals that point to the same type of resource, e.g., both SignalNodeFsAvailable and SignalAllocatableNodeFsAvailable refer to the same resource NodeFs. Change the map from map[v1.ResourceName]evictionapi.Signal to map[v1.ResourceName][]evictionapi.Signal
There are no sig labels on this issue. Please by: a sig: e.g., to notify the contributor experience sig, OR the label manually: e.g., to apply the label Note: Method 1 will trigger an email to the group. You can find the group list and label list . The in the method 1 has to be replaced with one of these: bugs, feature-requests, pr-reviews, test-failures, proposals
/kind cleanup /priority important/soon taking a guess at labels to satisfy the milestone munger
[MILESTONENOTIFIER] Milestone Labels Complete Issue label settings: sig/node sig/storage: Issue will be escalated to these SIGs if needed. priority/important-soon: Escalate to the issue owners and SIG owner; move out of milestone after several unsuccessful escalation attempts. kind/cleanup: Adding tests, refactoring, fixing old bugs. <detailsAdditional instructions available <a href=""here</aThe commands available for adding these labels are documented <a href=""here</a</details
Looks like the PR for this is just awaiting merge
Change:
&Foo{}, &FooList{}, ) <ins> metav1.AddToGroupVersion(scheme, SchemeGroupVersion) </ins> return nil }

Discussion:
Is this a BUG REPORT or FEATURE REQUEST?: Bug report /kind bug What happened: I'm following the directions to run the sample controller on When running it as directed, it gives these errors: E1214 10:34:25. ] Failed to list v1alpha1.Foo: v1.ListOptions is not suitable for converting to "" E1214 10:34:26. ] Failed to list v1alpha1.Foo: v1.ListOptions is not suitable for converting to "" What you expected to happen: Not to generate these errors. How to reproduce it (as minimally and precisely as possible): 1) Have a working kube config. 2) checkout kubernetes/sample-controller 3) run it: Anything else we need to know?: Environment: Kubernetes version (use ): Client Version: version.Info{Major:"1", Minor:"7", GitVersion:"v1.7.6", GitCommit:"", GitTreeState:"clean", BuildDate:"2017-09-14T06:55:55Z", GoVersion:"go1.8.3", Compiler:"gc", Platform:"linux/amd64"} Server Version: version.Info{Major:"1", Minor:"9+", GitVersion:"v1.9.0-alpha.0.845+-dirty", GitCommit:"", GitTreeState:"dirty", BuildDate:"2017-10-02T22:37:04Z", GoVersion:"go1.8.3", Compiler:"gc", Platform:"linux/amd64"} Cloud provider or hardware configuration: GKE OS (e.g. from /etc/os-release): my workstation is 14.04.5 LTS, Trusty Tahr Kernel (e.g. ): my workstation kernel is 4.4.0-96-generic Install tools: Others: go version 1.9
cc - Hi, it looks like you checked this in with . What all do I need to do to get this to work?
/sig api-machinery
Hey - thanks for this report. I'm not sure how, but we were missing an extra line in I've put in to resolve this (and it's working for me now :tada:)
Change:
m.incoming <- Event{action, obj} } <ins> // Action distributes the given event among all watchers, or drops it on the floor // if too many incoming actions are queued up.  Returns true if the action was sent, // false if dropped. func (m *Broadcaster) ActionOrDrop(action EventType, obj runtime.Object) bool { select { case m.incoming <- Event{action, obj}: return true default: return false } } </ins> // Shutdown disconnects all watchers (but any queued events will still be distributed). // You must not call Action or Watch* after calling Shutdown. This call blocks // until all events have been distributed through the outbound channels. Note

Discussion:
The standard event recorder, when recording an event, calls the underlying Broadcaster in a goroutine, so that even recording is non-blocking. Things like leader election emit events over the course of their lifecycle via this broadcaster. Consider the following scenario: GR1: spawn leader election ----GR2 GR2: Leader election: emit event "Lease Acquired" ----spawn GR3 () GR1: Shutdown leader election ----blocks until GR2 is done GR1: Shutdown broadcaster --() GR3: ----send on closed channel () Practically, we've observed this behavior popping up occaisionally in tests in controller-runtime:
/sig api-machinery /kind bug
<details<summaryrelevant race detector output/stack traces</summary</details
were you planning on sending a fix, or need our help to send one to fix it?
I can probably try to figure out a fix if you want, but the whole model for how event broadcasters currently work is inverse to how Go suggests you manage channels (only writers should ever close channels, not readers). I'll need to do a bit of playing around with things.
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
/reopen
Reopened this issue. <detailsIn response to : /reopen Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
cc
I think this is fixed by . Please reopen if it's not the case.
/triage accepted
Would be possible to cherry-pick this fix in 1.20?
Change:
} func LoadAppArmorProfiles(f *framework.Framework) { <del> _, err := createAppArmorProfileCM(f) framework.ExpectNoError(err) _, err = createAppArmorProfileLoader(f) framework.ExpectNoError(err) </del> <ins> createAppArmorProfileCM(f) createAppArmorProfileLoader(f) </ins> } // CreateAppArmorTestPod creates a pod that tests apparmor profile enforcement. The pod exits with

Discussion:
The "AppArmor load AppArmor profiles should enforce an AppArmor profile" e2e test in failing on our 5k-node gce clusters - https://k8s- (run no. 8) due to timeout: Seems like the test is trying to load the profiles across all the nodes using a daemonset and then running a test pod to check that. It's passing on our 2k-node clusters (https://k8s-) but failing for 5k - suggesting that it's likely due to size of the cluster. I think it's the ds creation taking longer on large cluster. cc
Seems like you're the most relevant here. Could one of you take a look into it? And does increasing the timeout for the test-pod based on no. of nodes sound reasonable?
Assigning it you since is OOO. Feel free to reassign as apt.
This is part of the correctness suite and large cluster tests are release-blocking.
FYI
Reassigning to who is back on Tuesday. He wrote the AppArmor stuff and should be able to debug this quickly. /assign
Ref
Could you manage to take a look into this?
Sorry, I meant to reply to this! I have a fix, just need to confirm it works. I think you're right about the DaemonSet timing out. There's no reason this needs to load the profiles on every node, so the fix just runs a single loader, and uses pod affinity to co-schedule the test pod.
Smooth!
[MILESTONENOTIFIER] Milestone Labels Complete Issue label settings: sig/node sig/scalability: Issue will be escalated to these SIGs if needed. priority/important-soon: Escalate to the issue owners and SIG owner; move out of milestone after several unsuccessful escalation attempts. kind/bug: Fixes a bug discovered during the current release. <detailsAdditional instructions available <a href=""here</aThe commands available for adding these labels are documented <a href=""here</a</details
Change:
go_library( name = "go_default_library", srcs = [ <ins> "conversion.go", </ins> "deepcopy.go", "doc.go", "generated.pb.go",

Discussion:
Making an API request for table output and specifying the parameter causes a server error in parameter conversion, and an inaccurate error message in response: server log: E0804 09:41:14. ] apiserver received an error that is not an metav1.Status: converting (url.Values).[]string to (v1alpha1.TableOptions).v1alpha1.IncludeObjectPolicy: couldn't copy '[]string' into 'v1alpha1.IncludeObjectPolicy'; didn't understand types
Ugh, typed strings.
Change:
pkg_tar( name = "kubernetes-cni-data", <del> package_dir = "/opt/cni", </del> <ins> package_dir = "/opt/cni/bin", </ins> deps = ["@kubernetes_cni//file"], )

Discussion:
Is this a BUG REPORT or FEATURE REQUEST?: What happened: Kubeadm E2E test is currently broken due to the commit I report the same issue on my local cluster. If I revert PR , everything works fine. We believe the current weave-net plugin doesn't work with the CNI version change. Right now all kubeadm related development is blocked on this. Could we consider reverting this change from head until we figure out what's wrong in weave-net/CNI? More context in
/assign
Please reopen this in kubernetes/kubeadm. There is an issue as e2e tests & PR job are failing, but locally the CNI bit seem to work for me, so I think it has something with building the debs to do. Not technically. We just don't have any coverage right now. Not IMO. We just need to forward-fix the issue. Closing this; you can reopen in kubernetes/kubeadm
kubeadm e2e test still red. Do you have any progress or hint in terms how to fix it forward? Just to clarify, If I install k8s and kubeadm locally using deb, I can repro the issue on my test box. I could believe it's sth wrong with debs. In the meantime, CNI and plug-in like weave-net change is a big deal. We clearly didn't test it thoroughly for this PR. Reverting the change seems to be the clean solution.
FYI kubernetes/release bumps new CNI releases for deb/rpm packages.
Thanks.
Is this the fix needed to fix these tests?
I actually think that will take us in the other direction. During the kubeadm e2e tests, we use the deb packages built by bazel. During our official release process, we use the debs built by anago (in the release repo). Bumping CNI in our bazel build broken our e2e tests. Bumping CNI in the release repo scripts will similarly break our official release builds as well. If I understand the problem correctly, the only way to fix both of them is for the CNI providers like weave-net to be updated to handle the changes of upgrading CNI to 0.6.0.
I think this issue worth some more discussion.

Re-open the case as it might not be affecting kubeadm only
sorry, i cannot reopen the issue as it's closed by But let's continue to discuss the option here.
Yes, I think so
This will be fixed by ; this was due to the CNI binaries being extracted in the wrong place. It is kind of a kubeadm packaging issue. (Reopening just to make the PR close this)
nice find!
Change:
name                      string rejectForwardingRedirects bool serverStatusCode          int <ins> redirect                  string </ins> expectStatusCode          int expectBody                []byte }{

Discussion:
Upgraded from Kubernetes v1.25.0 to v1.25.1. Standard metrics-server deployment now causes the following message to appear in the kube-apiserver logs on a recurring basis: Adding a quick hack debug Infof print shows: It appears that the apiserver is sending an If-None-Match cache control header when refreshing the openapi spec, and the metrics-server service is correctly responding with a 304 Not Modified. No errors logged; proper refresh of OpenAPI specs Deploy metrics-server, or any other service that uses apiserver aggregation and honors caching headers such as , to a Kubernetes 1.25.1 cluster. made bad assumptions; not all HTTP 3XX responses are redirects. It should probably only reject the response if it is 300-399 AND the header is set. <details</detailsnone <details</detailsk3s containerd flannel
Taking a stab at some tags here... /cc /cc /cc /area apiserver /sig api-machinery
thanks for the report, looks like this should be /priority critical-urgent /kind regression cc since this requires a respin of a final 1.22 patch release :-/
Fix open in , will get cherry-picks open to start greening up CI
/triage accepted
I need to make a redirect to the external URL but it's not allowed :(
Change:
err := f.FederationClientset.Services(nsName).Delete(service.Name, &metav1.DeleteOptions{}) framework.ExpectNoError(err, "Error deleting service %q in namespace %q", service.Name, nsName) }() <ins> By(fmt.Sprintf("Wait for service shards to be created in all clusters for service "%s/%s"", nsName, service.Name)) </ins> waitForServiceShardsOrFail(nsName, service, clusters) <ins> framework.Logf("Successfully created and synced service "%s/%s" to all clusters", nsName, service.Name) By(fmt.Sprintf("Update federated service "%s/%s"", nsName, service.Name)) service = updateServiceOrFail(f.FederationClientset, nsName, FederatedServiceName) waitForServiceShardsOrFail(nsName, service, clusters) framework.Logf("Successfully updated and synced service "%s/%s" to clusters", nsName, service.Name) </ins> }) It("should be deleted from underlying clusters when OrphanDependents is false", func() {

Discussion:
As mentioned in when the service is deleted from the underlying cluster, the controller should recreate it, but it does not. Below's a transcript. is talking to the federation-apiserver, and is talking to the normal cluster it's pointed at.
Thanks for catching this cc
cc
This is definitely a bug, but I think that we can kick it out into v1.4. Until fixed (i.e. while in beta), we can tell customers not to delete the underlying services in their clusters.
it should not be, I have tested this before. How long did you wait when you check the service existence in last step? Can you send me the logs? The service deletion will cause federation service update, and then the federation service watcher will catch the update even and call another round of service updates to all clusters. When you delete the service, if the service got ingress ip already? I am thinking if the deletion did not trigger federation service update? So so no update event generated..
I think that the best way forward will be to encode this in a reproducible e2e test, otherwise there are too many variables to debug effectively. It's great to know that you've tested this case, and it works in at least some scenarios.
one possible solution come into my mind, currently we set the service full sync time to 10 minutes , if we change it back to 30s, then at least during the full sync, the deleted service will be re-created. I will send a PR anyway, you can judge if it should be put to v1.3
I think we need to find and fix the underlying problem, not cover it up with a shorter sync period. If we have the correct watch on each service, we will be notified as soon as they are deleted. Why is the watch notification not firing correctly?
Keeping this open until we've verified this fix in e2e tests.
I believe this to be fixed, but we still lack an e2e test to verify conclusively. would you like to take this on along with the other e2e tests that you are busy adding?
ping
plan on doing this for 1.4? or bump to 1.5?
Yes, 1.5 is fine.
switching this to 1.6 as it's too late for 1.5. ok?

Moving to 1.7 as late to happen in 1.6. Feel free to switch back if this is incorrect.
Change:
} // wait for deployment to create some rs ginkgo.By("Wait for the Deployment to create new ReplicaSet") <ins> var replicaset appsv1.ReplicaSet </ins> err = wait.PollImmediate(500*time.Millisecond, 1*time.Minute, func() (bool, error) { rsList, err := rsClient.List(context.TODO(), metav1.ListOptions{}) if err != nil { return false, fmt.Errorf("failed to list rs: %v", err) } <del> return len(rsList.Items) > 0, nil </del> <ins> if len(rsList.Items) > 0 { replicaset = rsList.Items[0] return true, nil } return false, nil </ins> }) if err != nil { framework.Failf("Failed to wait for the Deployment to create some ReplicaSet: %v", err) } <ins> desiredGeneration := replicaset.Generation if err := wait.PollImmediate(100*time.Millisecond, 60*time.Second, func() (bool, error) { newRS, err := clientSet.AppsV1().ReplicaSets(replicaset.Namespace).Get(context.TODO(), replicaset.Name, metav1.GetOptions{}) if err != nil { return false, err } return newRS.Status.ObservedGeneration >= desiredGeneration && newRS.Status.Replicas == *replicaset.Spec.Replicas, nil }); err != nil { framework.Failf("failed to verify .Status.Replicas is equal to .Spec.Replicas for replicaset %q: %v", replicaset.Name, err) } </ins> ginkgo.By("delete the deployment") deleteOptions := getOrphanOptions() deleteOptions.Preconditions = metav1.NewUIDPreconditions(string(createdDeployment.UID))

Discussion:
Hi, first I have to mention I didn't saw this in Kubernetes CI, we're running the conformance tests on our internal platform and from time to time the test fails. Which test(s) are flaking: Reason for failure: The test works like this: create deployment wait until replicaset is created delete deployment (with orphan) count if we have 0 deployments, 1 replicaset and 2 pods In our environment it fails because from time to time we don't already have 2 pods. Imho this happens because the test doesn't wait until all Pods are created before deleting the Deployment Proposed
/sig apps These SIGs are my best guesses for this issue. Please comment if I am incorrect about one. I am a bot run by vllry. ‍
/sig api-machinery (as per the test location)
/cc /sig apps
/milestone v1.19 Since I milestoned