jiebi/CodeConvo huggingface dataset:
Code change:
package gce import ( <ins> "net/http" "net/http/httptest" </ins> "reflect" "testing" compute "google.golang.org/api/compute/v1" <ins> "k8s.io/kubernetes/pkg/util/flowcontrol" </ins> "k8s.io/kubernetes/pkg/util/rand" <ins> utiltesting "k8s.io/kubernetes/pkg/util/testing" </ins> ) func TestGetRegion(t *testing.T) {

Related discussion:
text: It could before because of the operation rate limiter and operation polling anyways, but now could be worse.
- yes I'm aware of that. And probably this is exactly the case here.
I got fooled by that - I guess we should fix it somehow.
One approach to fixing it is a similar approach to the operation polling code, but in the : warn if we ratelimited for more than N seconds (and dump the request or something? not clear how to clarify what we were sending).
I found something else - Kubelet is also building GCE interface: We have 2000 kubelets, so if all of them send multiple requests to GCE, we may hit limits.
Any idea in what circumstances Kubelet will contact cloud provider?
Hmm - it seems it contacts GCE exactly once at the beginning. So it doesn't explain much.
I think it's only here: I've rarely seen GCE ratelimit readonly calls, in practice, and kubelets come up pretty well staggered.
OK - so with logs that I , it is pretty clear that we are very heavily throttled on GCE api calls. In particular, when I was running 1000-node cluster, I see lines where we were throttle for 1m30. I'm pretty sure it's significantly higher in 2000-node cluster. I'm going to investigate a bit more where all those reuqests come from.
Actually - it seems that we have a pretty big problem here. Basically, Kubelet seems to send a lot of requests to GCE. So to clarify: every time Kubelet is updating NodeStatus (so every 10s by default) it is sending an API call to GCE to get it's not addresses: that means, that if we have 2000 nodes, kubelets generate 200 QPS to GCE (which is significantly more than we can afford) We should think how to solve this issue, but I think the options are: increasing QPS quota or calling GCE only once per X statusUpdates.
The second issue is that we are throttled at the controller level too, and this one I'm tryin to understand right now.
Code change:
package gce import ( <ins> "net/http" "net/http/httptest" </ins> "reflect" "testing" compute "google.golang.org/api/compute/v1" <ins> "k8s.io/kubernetes/pkg/util/flowcontrol" </ins> "k8s.io/kubernetes/pkg/util/rand" <ins> utiltesting "k8s.io/kubernetes/pkg/util/testing" </ins> ) func TestGetRegion(t *testing.T) {

Related discussion:
text: - that said, any rate-limitting on our side, will not solve the kubelet-related problem
Hmm - actually looking into implementation it seems that NodeAddresses are only contacting metadata server, so I'm not longer that convinced...
Also, I think that important thing is that a single call to GCE API can translate to multiple api calls to GCE. In particular - CreateRoute translates to: get instance insert route a bunch of getOperation (until this is finished)
OK - so I think that what is happening here is that since CreateRoute translates to: get instance insert route a bunch of getOperations That means that if we call say 2000 CreateRoute() at the same time, all of them will try to issue: getInstance at the beginning. So we will quickly accumulate 2000 GET requests in the queue. Once they are processed, we generate the "POST route" then for any processed. So It's kind of expected what is happening here. So getting back to the Zach`s PR - I think that throttling POST and GET and OPERATION calls separate is kind of good idea in general.
But I still don't really understand why do we get "RateLimitExceeded". Is that because of Kubelets?
I think that one problem we have at the RouteController level is that if CreateRoute fails, we simply don't react on it. And we will wait at least 5 minutes before even retrying it. We should add some retries at the route controller level too. I will send out some short PR for it.
But I still don't understand why do we get those "RateLimitExceeded" error
OK - so as an update. In GCE, we have a separate rate-limit for number of in-flight CreateRoute call per project. is supposed to address this issue.
OK - so this one is actually fixed. I mean that it is still long, but currently it's purely blocked on GCE.
As a comment - we have an internal bug for it already.
Code change:
if err != nil { assert.NoErrorf(t, err, "unable to create temp file") } stdoutBuf := &bytes.Buffer{} stderrBuf := &bytes.Buffer{} containerID := "fake-container-id"

Related discussion:
text: Runs k8s with master golang: test Failure was observed on Aug 25th, it is passing after that. However it can be reproduced with stress tool. No response No response It is failing on both ppc and x86 architectures. Steps to reproduce: up golang tip version. clone kubernetes master /sig testing
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/assign
it is a sig node test, sig testing charter is not about doing or maintain tests, we need to remove that old mindset of having a QA department, that is not how software development works or should work :) Besides, thanks for spotting it and submittn a patch /sig node /remove-sig testing
Flaky in
merging this PR will fix the issue
Code change:
} // writeLogs writes logs into stdout, stderr. <del> func (w *logWriter) write(msg *logMessage) error { </del> <ins> func (w *logWriter) write(msg *logMessage, addPrefix bool) error { </ins> if msg.timestamp.Before(w.opts.since) { // Skip the line because it's older than since return nil } line := msg.log <del> if w.opts.timestamp { </del> <ins> if w.opts.timestamp && addPrefix { </ins> prefix := append([]byte(msg.timestamp.Format(timeFormatOut)), delimiter[0]) line = append(prefix, line...) }

Related discussion:
text: <!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --What happened: Docker has a hardcoded 16K limit on log line length, which splits lines longer than that into multiple messages in the log file. in , however, doesn't account for this when is set to and prints the timestamp in front of every entry in the log file instead of every logical line. This leads to random timestamps appearing in the middle of log lines, potentially breaking the applications that parse these logs. What you expected to happen: inserts timestamps once per logical line, so original log data is preserved. Alternatively, allow API to request the logs in a sane format like JSON, which would allow merging the lines back and combining them with the timestamps more intelligently on the application side. How to reproduce it (as minimally and precisely as possible): Create : apiVersion: v1 kind: Pod metadata: name: log-lines spec: containers: name: main image: busybox command: ['sh', '-c', 'while sleep 1; do echo -n ; for i in $(seq 1 ); do echo -n A; done; echo ; done'] Start the pod with Fetch the logs with , or using the API () Observe random timestamps inserted in the middle of the log lines. Environment: Kubernetes version (use ):
Reiterating the mentions to trigger a notification: <detailsIn response to : Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
I will work on it. /sig cli /remove-sig node
The logs are being corrupted by the API endpoint in kubelet, so shouldn't it be in scope of SIG node?
The log is not corrupted by k8s. The same log could be got using "docker container logs -t CONTAINERID" I think you should post your requirement to Docker.
Code change:
} // writeLogs writes logs into stdout, stderr. <del> func (w *logWriter) write(msg *logMessage) error { </del> <ins> func (w *logWriter) write(msg *logMessage, addPrefix bool) error { </ins> if msg.timestamp.Before(w.opts.since) { // Skip the line because it's older than since return nil } line := msg.log <del> if w.opts.timestamp { </del> <ins> if w.opts.timestamp && addPrefix { </ins> prefix := append([]byte(msg.timestamp.Format(timeFormatOut)), delimiter[0]) line = append(prefix, line...) }

Related discussion:
text: Please refer following code Line 50 ~ 52 if config.Timestamps { logLine = append([]byte(msg.Timestamp.Format(jsonmessage.RFC3339NanoFixed)+" "), logLine...) }
kubelet doesn't rely on Docker when reading logs, it reads the files itself (see the code linked in the issue). I don't think Docker would change the forced line splitting (see for example), so this should be fixed in Kubernetes.
Your are right and kubelet read log file from host such as /var/log/pods/pods/NAMESPACEPOD-NAMExxx/....(JSON format) I will post a PR for it. /remove-sig cli /sig node
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
/remove-lifecycle stale
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
/remove-lifecycle stale
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
/remove-lifecycle stale
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
/remove-lifecycle stale
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with .
Code change:
} // writeLogs writes logs into stdout, stderr. <del> func (w *logWriter) write(msg *logMessage) error { </del> <ins> func (w *logWriter) write(msg *logMessage, addPrefix bool) error { </ins> if msg.timestamp.Before(w.opts.since) { // Skip the line because it's older than since return nil } line := msg.log <del> if w.opts.timestamp { </del> <ins> if w.opts.timestamp && addPrefix { </ins> prefix := append([]byte(msg.timestamp.Format(timeFormatOut)), delimiter[0]) line = append(prefix, line...) }

Related discussion:
text: Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
/remove-lifecycle stale
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
/remove-lifecycle stale
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-contributor-experience at . /lifecycle stale
/remove-lifecycle stale
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-contributor-experience at . /lifecycle stale
Stale issues rot after 30d of inactivity. Mark the issue as fresh with . Rotten issues close after an additional 30d of inactivity. If this issue is safe to close now please do so with . Send feedback to sig-contributor-experience at . /lifecycle rotten
I saw the attempt fixing this but it was scary (echo this comment: ). I suggested a way it may be improved. Also the proposed fix didn't introduce any e2e tests to validate that runtime behavior is not changed and runtimes keep the no-newline-on-splitted-log logic. are you or anybody on this thread are still interested to work on this? Please re-open if so. Meanwhile closing this ahead of bot doing it because the inactivity. /close
Closing this issue. <detailsIn response to : I saw the attempt fixing this but it was scary (echo this comment: ). I suggested a way it may be improved. Also the proposed fix didn't introduce any e2e tests to validate that runtime behavior is not changed and runtimes keep the no-newline-on-splitted-log logic. Instructions for interacting with me using PR comments are available .
Code change:
} // writeLogs writes logs into stdout, stderr. <del> func (w *logWriter) write(msg *logMessage) error { </del> <ins> func (w *logWriter) write(msg *logMessage, addPrefix bool) error { </ins> if msg.timestamp.Before(w.opts.since) { // Skip the line because it's older than since return nil } line := msg.log <del> if w.opts.timestamp { </del> <ins> if w.opts.timestamp && addPrefix { </ins> prefix := append([]byte(msg.timestamp.Format(timeFormatOut)), delimiter[0]) line = append(prefix, line...) }

Related discussion:
text: If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/reopen
You can't reopen an issue/PR unless you authored it or you are a collaborator. <detailsIn response to : /reopen Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/reopen
Reopened this issue. <detailsIn response to : /reopen Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs. This bot triages issues and PRs according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Reopen this issue or PR with Mark this issue or PR as fresh with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at . /close [1]:
Closing this issue. <detailsIn response to : The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs. After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closedReopen this issue or PR with Mark this issue or PR as fresh with Offer to help out with [Issue Triage][1] Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/remove-lifecycle rotten /reopen will fix it.
Reopened this issue. <detailsIn response to : /remove-lifecycle rotten /reopen Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/triage accepted
The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs.
Code change:
} // writeLogs writes logs into stdout, stderr. <del> func (w *logWriter) write(msg *logMessage) error { </del> <ins> func (w *logWriter) write(msg *logMessage, addPrefix bool) error { </ins> if msg.timestamp.Before(w.opts.since) { // Skip the line because it's older than since return nil } line := msg.log <del> if w.opts.timestamp { </del> <ins> if w.opts.timestamp && addPrefix { </ins> prefix := append([]byte(msg.timestamp.Format(timeFormatOut)), delimiter[0]) line = append(prefix, line...) }

Related discussion:
text: This bot triages issues and PRs according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Mark this issue or PR as fresh with Mark this issue or PR as rotten with Close this issue or PR with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at . /lifecycle stale [1]:
The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs. This bot triages issues and PRs according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Mark this issue or PR as fresh with Close this issue or PR with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at . /lifecycle rotten [1]:
The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs. This bot triages issues and PRs according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Reopen this issue or PR with Mark this issue or PR as fresh with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at . /close [1]:
Closing this issue. <detailsIn response to : The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs. After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closedReopen this issue or PR with Mark this issue or PR as fresh with Offer to help out with [Issue Triage][1] Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/reopen
Reopened this issue. <detailsIn response to : /reopen Instructions for interacting with me using PR comments are available .
Code change:
} // writeLogs writes logs into stdout, stderr. <del> func (w *logWriter) write(msg *logMessage) error { </del> <ins> func (w *logWriter) write(msg *logMessage, addPrefix bool) error { </ins> if msg.timestamp.Before(w.opts.since) { // Skip the line because it's older than since return nil } line := msg.log <del> if w.opts.timestamp { </del> <ins> if w.opts.timestamp && addPrefix { </ins> prefix := append([]byte(msg.timestamp.Format(timeFormatOut)), delimiter[0]) line = append(prefix, line...) }

Related discussion:
text: If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
I received a bug report about this issue. The root cause is that the buffer reads up to a new line. If the line is greater than 4096 bytes and '--timestamps' is turrned on the kubelet will write the timestamp and the partial log line. We will need to refactor the ReadLogs function to allow for a partial line read.
Code change:
// in node status. By calling this function (GetVolumesToReportAttached), node status should be updated, and the volume // will not need to be updated until new changes are applied (detach is triggered again) verifyVolumeAttachedToNode(t, generatedVolumeName, nodeName1, cache.AttachStateAttached, asw) <del> verifyVolumeReportedAsAttachedToNode(t, logger, generatedVolumeName, nodeName1, true, asw) </del> <ins> verifyVolumeReportedAsAttachedToNode(t, logger, generatedVolumeName, nodeName1, true, asw, volumeAttachedCheckTimeout) </ins> }

Related discussion:
text: TestRunOneVolumeDetachFailNodeWithReadWriteOnce 4 times since 03-19 23:00 PDT, intermittently ! https://k8s- Still investigating whether this is a tolerable flake or not with more history than testgrid view, or otherwise would appreciate input from anyone with expertise in this test, especially as regards to whether this should be a blocker for tomorrows beta cut. maybe: /sig storage /sig testing
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
is already open for this, looks like it is attempting to fix it
thank you!!! closing this in favor of that one, which I also to our CI signal project board for visibility. For completeness: since it's a longer standing issue, well identified as a flake and not a failure and also actively in progress, NOT considering it a blocker for the go/nogo signal for the beta tomorrow. /close
Closing this issue. <detailsIn response to : thank you!!! closing this in favor of that one, which I also to our CI signal project board for visibility. For completeness: since it's a longer standing issue, well identified as a flake and not a failure and also actively in progress, NOT considering it a blocker for the go/nogo signal for the beta tomorrow. Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details