jiebi/CodeConvo huggingface dataset:
Code change:
"k8s.io/client-go/pkg/version" "k8s.io/client-go/tools/cache" "k8s.io/client-go/tools/record" <del> "k8s.io/cloud-provider" </del> <ins> cloudprovider "k8s.io/cloud-provider" </ins> nodehelpers "k8s.io/cloud-provider/node/helpers" servicehelpers "k8s.io/cloud-provider/service/helpers" cloudvolume "k8s.io/cloud-provider/volume"

Related discussion:
text: E2e tests occasionally flake on AWS. It happens randomly to all tests that create a pod with a volume, write something to the volume, delete the pod and create a new one that reads from the volume. For example: /D controller detaches a volume from a node (= the old pod died) after that (under 1s) it tries to attach the same volume to the same node (=the new pod is scheduled and coincidentally it uses the same node). In most cases this works, however, few times per day it happens that the second attach (in Kubernetes terms, see below) does not succeed and the test times out. AWS cloud provider in Kubernetes heavily uses DescribeInstances and DescribeVolumes to check that a volume has been attached or detached: In step 1. above, Kubernetes issues AWS API call and polls until it reports that the volume has been detached. In step 2. above, Kubernetes issues AWS API call to check what devices are used on a node and then it tries to find a free device for the volume that should be attached. In this bug, in step 2 reports the volume as detaching, even that reported the volume as detached in step 1. There was no call in between! Kubernetes does not expect that this could ever happen and does not try attach the volume (it's not reported as detached, so why bother, right?). This is the code that finds out that the volume is not detached ("detaching" != "detached"): And few lines below we skip : We (OpenShift) run lot of Kubernetes e2e tests on AWS. We run them in a single AWS project and I guess we hit some timing issues / races on AWS side. We hit also AWS API quota pretty hard, but it does not seem to be related to this bug, I mention it only to illustrate the amount of things that run in parallel and that AWS may not like that load. Environment: Kubernetes v1.13.4 /kind bug /sig aws /sig storage
Flow from AWS API perspective: So far so good. = volume vol- is back detaching: All timing is related to a response received, processed and logged by Kubernetes, which may be different to time corresponding request was sent. Everything in this report happens in sequence, there is no pararellism for a single volume.
Code change:
"k8s.io/client-go/pkg/version" "k8s.io/client-go/tools/cache" "k8s.io/client-go/tools/record" <del> "k8s.io/cloud-provider" </del> <ins> cloudprovider "k8s.io/cloud-provider" </ins> nodehelpers "k8s.io/cloud-provider/node/helpers" servicehelpers "k8s.io/cloud-provider/service/helpers" cloudvolume "k8s.io/cloud-provider/volume"

Related discussion:
text: (Other volumes may be and often are attached / detached in parallel to this flow).
this looks like a bug in AWS itself, seems to return old data randomly. Can you please check AWS side of things? And we probably need fix / workaround on Kubernetes side. How can we check that a volume has been 100% detached if returns inconsistent data? We can check if a volume is in and not report is as attached here: This will cause Kubernetes to re-use the same device name and call . How bulletproof is it going to be? And most importantly, what other parts of Kubernetes depend on being consistent?
I confirmed internally. This is expected behavior for and since ec2 API follows a where resource mutation might be not reflected immediately. This could explain why returns a volume in detaching state even after the previous call returns without the volume (detached) I think this won't work in the case when the volume is actually stuck in detaching state, since attaching a stuck volume could simply make it worse. Given that, how about, in call, we return error when notice the volume is still being returned from and we rely on the AD controller retry to check the state next time, and hopefully the state converges.
If can return stale data, depending on it in (or anywhere else) probably won't help, because we may again get new or stale data "randomly". We call in today to get fresh list of used/free devices. Depending on timing, two cases happen: Sometimes it returns that the volume is detaching and I can see in logs. Handling as you suggest would help here. Sometimes it returns that the volume is detached. We find a new device name and call AWS and start polling for finished attach. This then returns that the volume is detaching from the node with the old device name. So we're actually attaching a volume that's being detached and you wrote that's a bad thing. In my logs, I can see that Kubernetes recovers from this error: Volume plugin AttachVolume() fails with A/D controller re-tries AttachVolume() calls AttachDisk() that calls to find a free device. returns that the volume is actually being attached to the node (I can see in logs) AttachDisk() polls until the volume is attached. -it works, but to me it feels like it's just an accident. Checking can't help in this case.
Code change:
"k8s.io/client-go/pkg/version" "k8s.io/client-go/tools/cache" "k8s.io/client-go/tools/record" <del> "k8s.io/cloud-provider" </del> <ins> cloudprovider "k8s.io/cloud-provider" </ins> nodehelpers "k8s.io/cloud-provider/node/helpers" servicehelpers "k8s.io/cloud-provider/service/helpers" cloudvolume "k8s.io/cloud-provider/volume"

Related discussion:
text: So, is there another way how to check that a volume is really detached? Or should we promote the error flow described above to official fix and add bunch of comments around so we won't remove / tweak the flow in future? What bad can happen when we call AWS on a volume that's detaching? AWS at least once reported that the volume is detached, so the volume probably is not be stuck.
I think it is only bad if the volume was previously stuck. But feels acceptable for me if it is being detached and the previous call in at least confirmed once. Confirming internally. But I suspect there isn't any. As long as the volume is not stuck (unmounted properly), we should be able to attach a detaching volume especially when it is at least confirmed being detached once from call in . Even if the first time it doesn't work, retry should help recover the situation. How do you think?
According pointed, AWS recommends two actions to manage the eventual consistency model (k8s already does 1, but AFAIK not 2, except when there's an exponential backoff among calls: 1) Run the appropriate call before modifying a resource 2) Add wait time between subsequent commands, even if returns a green response (this does infer that might return stale data) Currently, k8s assumes a state based on a single green response, e.g., is called repeatedly with an exponential backoff until it returns a given volume state. What if it required at least two positive responses instead? What are the chances that would return stale data twice? I'd assume that two calls returning a detached volume (with exponential backoff in between, which enforces point 2 above) would be reliable enough. The downside is that this would increase the API usage, which is already high.
do you know if the native AWS calls may be more stale-data-proof?
Looking at implementation, it tries with constant backoff of 15 seconds (which I consider pretty large) and it returns when the first returns that the volume is in use, so basically the same as what Kubernetes does.
From what I saw, sometimes it happens that reports a volume as detached, subsequent reports that the volume is not attached, succeeds and only after that shows that the volume is (from the old device name) instead of . So Kubernetes got at least two confirmations that the volume is detached.
Code change:
"k8s.io/client-go/pkg/version" "k8s.io/client-go/tools/cache" "k8s.io/client-go/tools/record" <del> "k8s.io/cloud-provider" </del> <ins> cloudprovider "k8s.io/cloud-provider" </ins> nodehelpers "k8s.io/cloud-provider/node/helpers" servicehelpers "k8s.io/cloud-provider/service/helpers" cloudvolume "k8s.io/cloud-provider/volume"

Related discussion:
text: What would be the correct delay to wait for second (third, ...) confirmation that the volume is really detached? IMO it would hurt our performance a lot.
The Wait function simply calls DescribeXX under the hood and uses some hard coded timeout. So it's similar to what kubernetes does like said.
Code change:
// deleteSecret deletes the secret with the given name from the host // cluster. func deleteSecret(clientset internalclientset.Interface, name, namespace string) error { <del> return clientset.Core().Secrets(namespace).Delete(name, &metav1.DeleteOptions{}) </del> <ins> orphanDependents := false return clientset.Core().Secrets(namespace).Delete(name, &metav1.DeleteOptions{OrphanDependents: &orphanDependents}) </ins> } // isNotFound checks if the given error is a NotFound status error.

Related discussion:
text: Problem: When users sends a cascading deletion request to federation i.e they set orphanDependents=false, then federation should also set orphanDependents=false, when it sends the request to underlying clusters.
Fixed by
Code change:
"//pkg/scheduler/internal/queue:go_default_library", "//pkg/scheduler/util:go_default_library", "//staging/src/k8s.io/api/core/v1:go_default_library", <ins> "//staging/src/k8s.io/apimachinery/pkg/api/resource:go_default_library", </ins> "//staging/src/k8s.io/apimachinery/pkg/apis/meta/v1:go_default_library", "//staging/src/k8s.io/apimachinery/pkg/runtime:go_default_library", "//staging/src/k8s.io/apimachinery/pkg/util/sets:go_default_library",

Related discussion:
text: <!-- Please only use this template for submitting enhancement requests --What would you like to be : Today, the scheduler re-processes unschedulable nodes . Node updates are sent by Kubelet frequently (every 10 seconds) and often times they don't change any of the node properties that could potentially make them feasible. We should be smarter about these updates and move pods to the schedulable queue only when certain properties of nodes change that could make them feasible. These properties include node allocatable, node conditions, taints, labels, .... Why is this needed: Improves performance of the scheduler and avoids doing unnecessary attempts to schedule pods. This is particularly helpful when a cluster has a lot of unschedulable pods. <!-- DO NOT EDIT BELOW THIS LINE --/kind feature This is not a feature really. It is performance optimization, but in the absence of such a label, I need to use "feature" here. /sig scheduling
I can help to fix it. /assign mlmhl
The main reason for this problem is / update their status too frequently and most of these updates are heartbeats without any meaningful change. Recently sig-node introduces a new to alleviate this problem. With this mechanism, kubelet will split heartbeat and node status updates into two independent operations, and reduce the frequency of status updates to once per 1 minute(or more). In the long run this mechanism can indeed alleviate our problems, but at present it does not help us much as this is still an alpha feature and the node status updates frequency is still a bit high(per 1 minute right now).
Maybe we can try comparing old and new nodes after stripping their heartbeat info, like this:
I think that had sent earlier is more efficient. However, that PR does not filter out . In Wei's code, the heartbeat time is removed correctly. I think we should fix this issue and resubmit that PR.
could you resubmit a PR to: disregard in conditions comparison: and (optionally) disregard timestamp in taints comparison:
Thanks for all your advice, a new PR is submitted: This PR does not include your second suggestion(disregard timestamp in taints comparison) as I'm not sure if can be changed when node updating.
Code change:
"//pkg/scheduler/internal/queue:go_default_library", "//pkg/scheduler/util:go_default_library", "//staging/src/k8s.io/api/core/v1:go_default_library", <ins> "//staging/src/k8s.io/apimachinery/pkg/api/resource:go_default_library", </ins> "//staging/src/k8s.io/apimachinery/pkg/apis/meta/v1:go_default_library", "//staging/src/k8s.io/apimachinery/pkg/runtime:go_default_library", "//staging/src/k8s.io/apimachinery/pkg/util/sets:go_default_library",

Related discussion:
text: If it can be changed, I will add this filter later, otherwise I think we can keep the current implementation to make the compare operation simpler and faster. What's more, just as I said in the PR's description, I'm not sure how much side effects this change has on scheduler's resource consumption, as it always copy node's conditions before comparison.
I am thinking whether we could move this logic(whether node became more schedulable) to kubelet, kubelet reports it through heartbeats.
Code change:
"time" "k8s.io/kubernetes/pkg/api" <ins> "k8s.io/kubernetes/pkg/api/errors" </ins> "k8s.io/kubernetes/pkg/api/meta" "k8s.io/kubernetes/pkg/api/testapi" apitesting "k8s.io/kubernetes/pkg/api/testing"

Related discussion:
text: When attempting to fetch a resource version that is too old, the behavior of watch (for both HTTP and web sockets) changed when watch cache was enabled. The watch cache returns a 410 - without the watch cache, we return 200 and then write an "error" entry to the watch stream. This broke a client that dependent on the watch behavior, and moreover is inconsistent with other errors we set in the registry. Also, for watch over web sockets, most javascript clients will be unable to get at a 410 error from the connection, because browsers don't get access to the data. We should restore the watch cache behavior to be consistent with the previous behavior as this is a regression in API semantics. Resource without watch cache, watching too old: Resource with watch cache, watching too old:
I think this is candidate for back port to 1.2 to restore the previous behavior.
since this broke your javascript client, FYI because of watch cache
+1 for backport
hmm - we enabled watch cache in 1.1 - so this should also be broken in 1.1
It's possible that it was always nondeterministic and just depends on how fast the call to the storage layer executes. I think all of our in-tree clients will handle it either way.
I'm fairly certain this never used to behave this way, because OpenShift hasn't enabled the watch catch and we have never had a report of this until we enabled the watch cache for Kube resources just recently. , Daniel Smith wrote:
I see what you mean about ordering - etcd remote call would almost never be fast enough to beat the watch event. In practice, I suspect no one ever saw this without the watch error. Either way, for web sockets if we don't fix this clients can't get this error (the browser eats the 410 during the websocket negotiation and client code can't get access). So it's at least broken for web sockets. , Clayton Coleman wrote:
Well, it should be easy to just not wait at all, maybe? I'm OK with changing this behavior.
Although, are web clients XSS hardened? I'm not sure that we're optimizing for them currently.
If you mean CSRF, it isn't an issue with bearer token auth.
Code change:
"time" "k8s.io/kubernetes/pkg/api" <ins> "k8s.io/kubernetes/pkg/api/errors" </ins> "k8s.io/kubernetes/pkg/api/meta" "k8s.io/kubernetes/pkg/api/testapi" apitesting "k8s.io/kubernetes/pkg/api/testing"

Related discussion:
text: Basic auth should already have a big warning around it saying "not safe with browsers"
Maybe I'm missing something, but it seems deterministic to me. This is the uncached watch impl: once the resourceVersion parses, there is no path that returns a direct error, so all errors would have to be returned as an event via the ResultChan(). I don't see us reading from that until after we've already written a 200 and flushed:
Having read the docs, I was very surprised that "type": "ERROR" notices are sent as 200 (even over plain http), with the semantic code e.g. 410 only represented under inside. I see there was good motivation here to stick to 200, but it should be called out in the documentation. doesn't show at all how watch returns errors, and it even says "clients must handle the case by recognizing the status code 410 Gone" which really makes one expect an actual HTTP 410.
I agree this should be called out in the docs. Can you file a bug there and/or send a PR? I don't think this ~4 year old issue is the best place to track that. (Good detective work finding this, though!)
Code change:
// BusyBoxImage is the image URI of BusyBox. BusyBoxImage = imageutils.GetE2EImage(imageutils.BusyBox) <ins> // AgnHostImage is the image URI of AgnHost AgnHostImage = imageutils.GetE2EImage(imageutils.Agnhost) </ins> // For parsing Kubectl version for version-skewed testing. gitVersionRegexp = regexp.MustCompile("GitVersion:"(v.+?)"")

Related discussion:
text: <!-- Please only use this template for submitting reports about failing tests in Kubernetes CI jobs --Which jobs are failing: N/A Which test(s) are failing: Since when has it been failing: Ever Reason for failure: Vm on Azure with no public IP assigned aren't able to perform a command to the internet. Anything else we need to know: The test (see ) tries to test the internet connection for a pod by pinging 8.8.8.8. But azure does not provide ICMP support for VM without a public IP therefore this test cannot succeed.(see ) A fix would be to change the test method to a tcp request instead of a ping request (the test already performs a http get to in its BeforeSuite (see ))
/sig network /sig testing /sig azure
you may refer to (Pods don't have connectivity to external networks )
cc WDYT about the proposal to change this from an ICMP call? We probably need input from sig-architecture as well since this as well /sig architecture
/triage unresolved
An alternative is to issue a TCP/UDP request to 8.8.8.8 on port 53.
ICMP is dropped by Azure for security reasons. This is intended. Can use curl as a replacement. I will fix the corresponding test.
SGTM
Can you please review the related PR? I'll in the PR again. Thanks!
Code change:
"//test/e2e/framework:go_default_library", "//vendor/github.com/onsi/ginkgo:go_default_library", "//vendor/github.com/onsi/gomega:go_default_library", <ins> "//vendor/k8s.io/utils/exec:go_default_library", </ins> ], )

Related discussion:
text: It would better to have e2e for iSCSI persistent volume. /sig storage
/assign
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
Stale issues rot after 30d of inactivity. Mark the issue as fresh with . Rotten issues close after an additional 30d of inactivity. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle rotten /remove-lifecycle stale
Rotten issues close after 30d of inactivity. Reopen the issue with . Mark the issue as fresh with . Send feedback to sig-testing, kubernetes/test-infra and/or . /close
Code change:
glog.V(4).Infof("UpdateLoadBalancer(%v, %v, %v)", clusterName, loadBalancerName, hosts) ports := service.Spec.Ports <del> if len(ports) > 1 { return fmt.Errorf("multiple ports are not yet supported in openstack load balancers") } else if len(ports) == 0 { </del> <ins> if len(ports) == 0 { </ins> return fmt.Errorf("no ports provided to openstack load balancer") }

Related discussion:
text: LBaaS implementation (v1 or v2) will not support services that require to load balancer more than one port.
Fixed for LBaaS v2 by I suspect LBaaS v1 can't support multiple ports (not sure if reusing the VIP across multiple loadbalancers is a universally supported thing). I also believe LBaaS v1 is also mostly gone in favour of v2 in deployments so trying to address this in v1 is extremely low priority unless someone steps forward to ask for it. So: I think we can close this bug now, but I lack the powers to do that directly.
thank you for the update.
Code change:
apiService := &v1.APIService{ Spec: v1.APIServiceSpec{ <ins> Group:   "apiservicegroup", Version: "v1", </ins> Service: &v1.ServiceReference{Name: "dummy"}, }, }

Related discussion:
text: When installing the metrics-server to implement the API and the prometheus-adapter to implement the API, it is possible to make the OpenAPI handler fail to initialize with this error log in apiserver: This will result in kubectl discovery calls to /openapi/v2 to fail with a 503: The OpenAPI handler should not fail to initialize if both and are serving the same path for different API groups/versions. I can provide exact installation steps if needed, however, a high-level repro is: install metrics-server to serve install prometheus-adapter to serve ensure resourceRules is populated in the prometheus-adapter config so that resource metrics are also served by restarting apiserver will result in the OpenAPI failure and 503 responses to /openapi/v2 path No response <details</details<detailsAWS </details<details</details<details</details<details</details<details</details
Please let me know if I should reopen the issue in
/sig api-machinery
/sig instrumentation [metrics apis]
Would you mind taking a look when have time? Thank you^^ /triage accepted
I have a question. should be serving under . Why does an apiservice serving another gvk register paths under ? cc we've made the assumption that an APIService (such as metrics-server or prometheus-adapter) serves only gvks under its own registered group version. I'll do some more investigation but just wondering if prometheus doesn't follow that assumption?
the prometheus-adapter can also be configured to serve and replace the metrics-server by enabling these metrics, However, if the prometheus-adapter fronts and metrics-server fronts , it does seem like the correct behavior would be for prometheus-adapter to serve until an operator updates the APIService to reference prometheus-adapter rather than metrics-server. We're seeing this issue occur for upgrades between 1.27 and 1.28 so it does look this path duplication did not cause the OpenAPI handler to fail on 1.27.
I just upgraded from 1.27 to 1.28 and running into this problem. Some kubectl apply operations fail, and ArgoCD cannot load its custom resources. Any suggested workaround? Would removing the from the prometheus configuration resolve it? Or perhaps disable metrics-server?
Simply looking at the code diff between 1.27 and 1.28 I can't pinpoint the cause.