jiebi/CodeConvo huggingface dataset:
Code change:
opName := op.Name return wait.Poll(operationPollInterval, operationPollTimeoutDuration, func() (bool, error) { start := time.Now() <del> gce.operationPollRateLimiter.Accept() </del> duration := time.Now().Sub(start) if duration > 5*time.Second { glog.Infof("pollOperation: waited %v for %v", duration, opName)

Related discussion:
text: On a large cluster, the routecontroller basically makes O(n) requests all at once, and most of them error out with "Rate Limit Exceeded". We then wait for operations to complete on the ones that did stick, and damn the rest, they'll be caught next reconcile (which is a while). In fact, in some cases it appears we're spamming hard enough to get rate limited by the upper-level API DOS protections and seeing something akin to , e.g.: We can do better than this!
c.f. , which is the other leg of this problem
I took a look into code and it seems that we are doing full reconciliation every 10s. If we need to create 2000 routes at the beginning it seems to frequently. I think that we need to: first compute how many routes we have to create based on that try to spread them over some time to reduce possibility of causing 'rate limit exceeded' I will try to put some small PR together as a starting point.
I'm reopening this one, since this doesn't seem to be solved. So basically something strange is happening here. With PR we have 10/s limit on API calls send to GCE. And we are still rate-limitted. However, according to documentation we are allowed for 20/s: So something either doesn't work or is weird.
What is super interesting from the logs of recent run: It starts to creating routes: After 2 minutes from that, we are hitting "Rate limit Exceeded" for the first time: And only after 9 minutes we create the first route ever. The second one is created after 15 minutes, and from that point we create a number of them So it seems that nothing useful is happening for the first 2 minutes, and then between 3m30 and 9m30
OK - so I think that what is happening is that all controllers are sharing GCE interface (and thuse they have common throttling). So if one controller is generating a lot of API calls, then other controllers may be throttled. has a hypothesis that it may be caused by nodecontroller.
One thing my PR didn't address (because it was late): Now we have some obnoxious differences in logging when you go to make a request, because the versus could actually be absorbing a fair amount of ratelimiter.
Code change:
opName := op.Name return wait.Poll(operationPollInterval, operationPollTimeoutDuration, func() (bool, error) { start := time.Now() <del> gce.operationPollRateLimiter.Accept() </del> duration := time.Now().Sub(start) if duration > 5*time.Second { glog.Infof("pollOperation: waited %v for %v", duration, opName)

Related discussion:
text: It could before because of the operation rate limiter and operation polling anyways, but now could be worse.
- yes I'm aware of that. And probably this is exactly the case here.
I got fooled by that - I guess we should fix it somehow.
One approach to fixing it is a similar approach to the operation polling code, but in the : warn if we ratelimited for more than N seconds (and dump the request or something? not clear how to clarify what we were sending).
I found something else - Kubelet is also building GCE interface: We have 2000 kubelets, so if all of them send multiple requests to GCE, we may hit limits.
Any idea in what circumstances Kubelet will contact cloud provider?
Hmm - it seems it contacts GCE exactly once at the beginning. So it doesn't explain much.
I think it's only here: I've rarely seen GCE ratelimit readonly calls, in practice, and kubelets come up pretty well staggered.
OK - so with logs that I , it is pretty clear that we are very heavily throttled on GCE api calls. In particular, when I was running 1000-node cluster, I see lines where we were throttle for 1m30. I'm pretty sure it's significantly higher in 2000-node cluster. I'm going to investigate a bit more where all those reuqests come from.
Actually - it seems that we have a pretty big problem here. Basically, Kubelet seems to send a lot of requests to GCE. So to clarify: every time Kubelet is updating NodeStatus (so every 10s by default) it is sending an API call to GCE to get it's not addresses: that means, that if we have 2000 nodes, kubelets generate 200 QPS to GCE (which is significantly more than we can afford) We should think how to solve this issue, but I think the options are: increasing QPS quota or calling GCE only once per X statusUpdates.
The second issue is that we are throttled at the controller level too, and this one I'm tryin to understand right now.
Code change:
opName := op.Name return wait.Poll(operationPollInterval, operationPollTimeoutDuration, func() (bool, error) { start := time.Now() <del> gce.operationPollRateLimiter.Accept() </del> duration := time.Now().Sub(start) if duration > 5*time.Second { glog.Infof("pollOperation: waited %v for %v", duration, opName)

Related discussion:
text: - that said, any rate-limitting on our side, will not solve the kubelet-related problem
Hmm - actually looking into implementation it seems that NodeAddresses are only contacting metadata server, so I'm not longer that convinced...
Also, I think that important thing is that a single call to GCE API can translate to multiple api calls to GCE. In particular - CreateRoute translates to: get instance insert route a bunch of getOperation (until this is finished)
OK - so I think that what is happening here is that since CreateRoute translates to: get instance insert route a bunch of getOperations That means that if we call say 2000 CreateRoute() at the same time, all of them will try to issue: getInstance at the beginning. So we will quickly accumulate 2000 GET requests in the queue. Once they are processed, we generate the "POST route" then for any processed. So It's kind of expected what is happening here. So getting back to the Zach`s PR - I think that throttling POST and GET and OPERATION calls separate is kind of good idea in general.
But I still don't really understand why do we get "RateLimitExceeded". Is that because of Kubelets?
I think that one problem we have at the RouteController level is that if CreateRoute fails, we simply don't react on it. And we will wait at least 5 minutes before even retrying it. We should add some retries at the route controller level too. I will send out some short PR for it.
But I still don't understand why do we get those "RateLimitExceeded" error
OK - so as an update. In GCE, we have a separate rate-limit for number of in-flight CreateRoute call per project. is supposed to address this issue.
OK - so this one is actually fixed. I mean that it is still long, but currently it's purely blocked on GCE.
As a comment - we have an internal bug for it already.
Code change:
} if err := proxyHandler.ServeConn(conn); err != nil && !shoulderror { <ins> // If the connection request is closed before the channel is closed // the test will fail with a ServeConn error. Since the test only return // early if expects shouldError=true, the channel is closed at the end of // the test, just before all the deferred connections Close() are executed. </ins> if isClosed() { return }

Related discussion:
text: TestRoundTripSocks5AndNewConnection Disabled by No response No response No response /sig
There are no sig labels on this issue. Please add an appropriate label by using one of the following commands: - Please see the for a listing of the SIGs, working groups, and committees available. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
there is a race shutting down the socks5 server
Code change:
clusterAddonLabelKey   = "k8s-app" kubeAPIServerLabelName = "kube-apiserver" clusterComponentKey    = "component" <ins> svcReadyTimeout = 1 * time.Minute </ins> ) var (

Related discussion:
text: [x] two weeks soak end date : 18 March 2021 According to this APIsnoop query, there are still some remaining RESOURCENAME endpoints which are untested. with this query you can filter untested endpoints by their category and eligiblity for conformance. e.g below shows a query to find all conformance eligible untested,stable,core endpoints Note: Community feedback for the e2e test has lead to a number of improvements including extending the test coverage to all outstanding endpoints listed above. As each endpoint is tested a &rsquo;watch&rsquo; confirms that the result is valid before testing the next endpoint. readCoreV1NamespacedServiceStatus (get /status) patchCoreV1NamespacedServiceStatus (patch /status) replaceCoreV1NamespacedServiceStatus (update /status) patchCoreV1NamespacedService Create a Service with a static label Patch the Service with a new label and updated data Get the Service to ensure it&rsquo;s patched Upate the Service with a new label and updated data Get the Service to ensure it&rsquo;s updated Delete Namespaced Service via a Collection with a LabelSelector This query by apisnoop shows that all outstanding endpoints as list at the start of this document have been hit by the e2e test. Note that the results do include other service endpoints that have been addressed in other conformance tests. If a test with these calls gets merged, test coverage will go up by 4 points This test is also created with the goal of conformance promotion. /sig testing /sig architecture /area conformance
Can you please review the proposed test and advise if we could go ahead with a PR
/assign
It looks like it's patching metadata and spec but not status?
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
/remove-lifecycle stale
I've been retesting this mock test and apisnoop does log as been hit. From my testing below it looks like the patchStatus is working as expected or have I missed something?
Code change:
clusterAddonLabelKey   = "k8s-app" kubeAPIServerLabelName = "kube-apiserver" clusterComponentKey    = "component" <ins> svcReadyTimeout = 1 * time.Minute </ins> ) var (

Related discussion:
text: Using the following code before the patch request Gives this response Then after running the patch ServiceStatus Then running The response shows the patched changes.
During the Conformance Meeting on 12th of January 2021, reviewed approach with Clayton, and we will start on the test PR this week!
/reopen
Reopened this issue. <detailsIn response to : /reopen </details
/triage accepted
Code change:
_, result.MissingAggregationRuleSelectors = aggregationRuleCovers(existing.GetAggregationRule(), expected.GetAggregationRule()) switch { <ins> case expected.GetAggregationRule() == nil && existing.GetAggregationRule() != nil: // we didn't expect this to be an aggregated role at all, remove the existing aggregation result.Role.SetAggregationRule(nil) result.Operation = ReconcileUpdate </ins> case !removeExtraPermissions && len(result.MissingAggregationRuleSelectors) > 0: // add missing rules in the union case aggregationRule := result.Role.GetAggregationRule()

Related discussion:
text: /kind bug What happened: I deployed a fresh custom HA kubernetes v1.10.2 cluster with 3 nodes (all controller and worker currently, more workers planned). I then created a namespace called "stummi" and a rolebinding with "user=stummi clusterrole=admin" in this namespace. User stummi (successfully authenticated by TLS client certificate) was not able to list pods or anything else in his namespace: While trying to figure things out, I downgraded to v1.8.11 (a version without cluster role aggregation, that worked) and also tried v1.11 alpha2 (which did not work). I never purged my etcd and I sadly do not have logs from before the different versions. Shortly after posting on slack and before writing this ticket, I dumped the cluster roles in question: Note the rules. What you expected to happen: I expected user stummi to be able to do most things in his namespace. How to reproduce it (as minimally and precisely as possible): Anything else we need to know?: I posted this problem on slack and helped me and wanted this issue opened. /sig auth The cluster is deployed with ansible and I can give the playbooks when required. Environment: Kubernetes version (use ):Cloud provider or hardware configuration: 3 virtual machines (2 cores, 8GB memory) OS (e.g. from /etc/os-release): Debian Stretch Kernel (e.g. ): Install tools: custom ansible playbooks Others: Please let me know if you need more information. I'm happy to provide everything you need to fix this :)
/assign
Code change:
{utiliptables.TableNAT, KubeNodePortChain}, {utiliptables.TableNAT, KubeLoadBalancerChain}, {utiliptables.TableNAT, KubeMarkMasqChain}, <del> {utiliptables.TableNAT, KubeMarkDropChain}, </del> {utiliptables.TableFilter, KubeForwardChain}, } <ins> var iptablesEnsureChains = []struct { table utiliptables.Table chain utiliptables.Chain }{ {utiliptables.TableNAT, KubeMarkDropChain}, } </ins> var iptablesCleanupChains = []struct { table utiliptables.Table chain utiliptables.Chain

Related discussion:
text: <!-- Please only use this template for submitting reports about continuously failing tests or jobs in Kubernetes CI --Which jobs are failing: ci-kubernetes-e2e-gci-gce-ipvs Which test(s) are failing: [sig-network] Services should only allow access from service loadbalancer source ranges [Slow] Testgrid link: https://k8s- Reason for failure: Anything else we need to know: The test is failing consistently, however, there is also test that is failing a lot but not always, may it be related?
/sig-network /area ipvs /cc /help wanted
/sig network
/triage unresolved Comment when the issue is assessed and confirmed. I am a bot run by vllry. ‍
/assign
/remove-triage unresolved
Hi, , turnout this is because mark drop rule in be deleted, I raied a pr to fix this issue, PTAL. : )
/close as duplicate of
Closing this issue. <detailsIn response to : /close as duplicate of Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/reopen this was affecting ipvs only before , let's keep it open until we verify is not failing anymore
Reopened this issue. <detailsIn response to : /reopen this was affecting ipvs only before , let's keep it open until we verify is not failing anymore Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/assign Ok I tested this out with kind in ipvs-mode with metallb. In iptables mode I had zero failures but ipvs does not pass at all during the timeout window. This is likely due to
Code change:
continue } // Write the log line into the stream. <del> if err := writer.write(msg); err != nil { </del> <ins> if err := writer.write(msg, isNewLine); err != nil { </ins> if err == errMaximumWrite { klog.V(2).InfoS("Finished parsing log file, hit bytes limit", "path", path, "limit", opts.bytes) return nil

Related discussion:
text: <!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --What happened: Docker has a hardcoded 16K limit on log line length, which splits lines longer than that into multiple messages in the log file. in , however, doesn't account for this when is set to and prints the timestamp in front of every entry in the log file instead of every logical line. This leads to random timestamps appearing in the middle of log lines, potentially breaking the applications that parse these logs. What you expected to happen: inserts timestamps once per logical line, so original log data is preserved. Alternatively, allow API to request the logs in a sane format like JSON, which would allow merging the lines back and combining them with the timestamps more intelligently on the application side. How to reproduce it (as minimally and precisely as possible): Create : apiVersion: v1 kind: Pod metadata: name: log-lines spec: containers: name: main image: busybox command: ['sh', '-c', 'while sleep 1; do echo -n ; for i in $(seq 1 ); do echo -n A; done; echo ; done'] Start the pod with Fetch the logs with , or using the API () Observe random timestamps inserted in the middle of the log lines. Environment: Kubernetes version (use ):
Reiterating the mentions to trigger a notification: <detailsIn response to : Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
I will work on it. /sig cli /remove-sig node
The logs are being corrupted by the API endpoint in kubelet, so shouldn't it be in scope of SIG node?
The log is not corrupted by k8s. The same log could be got using "docker container logs -t CONTAINERID" I think you should post your requirement to Docker.
Code change:
continue } // Write the log line into the stream. <del> if err := writer.write(msg); err != nil { </del> <ins> if err := writer.write(msg, isNewLine); err != nil { </ins> if err == errMaximumWrite { klog.V(2).InfoS("Finished parsing log file, hit bytes limit", "path", path, "limit", opts.bytes) return nil

Related discussion:
text: Please refer following code Line 50 ~ 52 if config.Timestamps { logLine = append([]byte(msg.Timestamp.Format(jsonmessage.RFC3339NanoFixed)+" "), logLine...) }
kubelet doesn't rely on Docker when reading logs, it reads the files itself (see the code linked in the issue). I don't think Docker would change the forced line splitting (see for example), so this should be fixed in Kubernetes.
Your are right and kubelet read log file from host such as /var/log/pods/pods/NAMESPACEPOD-NAMExxx/....(JSON format) I will post a PR for it. /remove-sig cli /sig node
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
/remove-lifecycle stale
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
/remove-lifecycle stale
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
/remove-lifecycle stale
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
/remove-lifecycle stale
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with .
Code change:
continue } // Write the log line into the stream. <del> if err := writer.write(msg); err != nil { </del> <ins> if err := writer.write(msg, isNewLine); err != nil { </ins> if err == errMaximumWrite { klog.V(2).InfoS("Finished parsing log file, hit bytes limit", "path", path, "limit", opts.bytes) return nil

Related discussion:
text: Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
/remove-lifecycle stale
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
/remove-lifecycle stale
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-contributor-experience at . /lifecycle stale
/remove-lifecycle stale
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-contributor-experience at . /lifecycle stale
Stale issues rot after 30d of inactivity. Mark the issue as fresh with . Rotten issues close after an additional 30d of inactivity. If this issue is safe to close now please do so with . Send feedback to sig-contributor-experience at . /lifecycle rotten
I saw the attempt fixing this but it was scary (echo this comment: ). I suggested a way it may be improved. Also the proposed fix didn't introduce any e2e tests to validate that runtime behavior is not changed and runtimes keep the no-newline-on-splitted-log logic. are you or anybody on this thread are still interested to work on this? Please re-open if so. Meanwhile closing this ahead of bot doing it because the inactivity. /close
Closing this issue. <detailsIn response to : I saw the attempt fixing this but it was scary (echo this comment: ). I suggested a way it may be improved. Also the proposed fix didn't introduce any e2e tests to validate that runtime behavior is not changed and runtimes keep the no-newline-on-splitted-log logic. Instructions for interacting with me using PR comments are available .