jiebi/CodeConvo huggingface dataset:
Code change:
q99 := hist.Quantile(0.95) avg := hist.Average() <del> // clear the metrics so that next test always starts with empty prometheus // metrics (since the metrics are shared among all tests run inside the same binary) hist.Clear() </del> msFactor := float64(time.Second) / float64(time.Millisecond) // Copy labels and add "Metric" label for this metric.

Related discussion:
text: In the new scheduler performance testing framework, (bool) is provided to benchmark performance of pods created in an "operator": (L11 below) However, the initialized Pods created in L5~L7 seem to be also benchmarked, although it doesn't specify . (please see the reproduce steps below). I think it's b/c once scheduler starts, the associated Prometheus artifacts are also up, and it's a single in-memory datastore. So no matter we start to collect the metric in which step, the prometheus data is already up-to-date. In the above workload config, the final data have benchmarked 1500 pods instead of 1000 pods. I guess it won't work either if running in the pod initialization step or not. Unless we manage a breakpoint at the end of each step, and calculate delta histogram for the step that specifies . Pods created in step without are not benchmarked. Apply the following diff: And then run: You will get an output sth. like: /sig scheduling
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
Could you please help confirm if it's an issue?
cc
Unfortunately, does not apply for metrics gathered from . As you mentioned. The default gatherer collects all metrics produced by the scheduler. It's the scheduler itself who knows pod specs and would have to decide which pods will get ignored when updating metrics. The question is if computing metrics for a subset of pods processed by the scheduler is a valid use case.
This is probably what we want. Is something like that possible ?
I think being able to absolutely ignore certain pods for metrics is an overkill.
The use case is to ignore the pods created in steps that don't specify , so as to collect the Pod metrics create at particular step(s) (). Otherwise, the semantics of option is quite misleading. Nope, that's not the intention. We don't need to target certain pods, like using label selector or whatever. Just as I mentioned, measure pods created at particular steps.
Code change:
q99 := hist.Quantile(0.95) avg := hist.Average() <del> // clear the metrics so that next test always starts with empty prometheus // metrics (since the metrics are shared among all tests run inside the same binary) hist.Clear() </del> msFactor := float64(time.Second) / float64(time.Millisecond) // Copy labels and add "Metric" label for this metric.

Related discussion:
text: I would do something like the following: if a step has collectMetrics:false, we wait for them to finish scheduling, and then we take a snapshot of the histogram. Then for the next step that has collectMetrics:true, we take the diff with the latest snapshot to report the results. This might not be perfect, as we don't control if the metrics have finished syncing when we take the snapshot. But I think that's fine. Would doing this be possible? Now, if we have steps that run in parallel, one with collectMetrics:false and another with collectMetrics:true, then this doesn't work. Note that I don't know the details of how the gatherer works.
That adheres to what I have in my mind - we can only manage this on client side. I think it's fine. As we are already fetching the "snapshot" when pods get scheduled when equals true. True. An alternative is to design the case to be like: the step with to employ Pods that are all unschedulable, and then specify a custom metric "schedulere2eschedulingdurationseconds[result="unschedulable"]" to target a particular histogram instead of all histograms (default option) inside a histogramVec. Note this can only be possible when and some subsequent changes get in place.
We might just call Histogram.Clear() method which will clear what has been collected so far at the end of phase: Assuming individual phases have no pods in common and each phase can clear metrics without affecting other phases. Though, as said, it will work only for serial case.
WDYT
I'm afraid doesn't work. That was my initial thought as well, but no luck. I guess it's b/c, theoretically, we cannot reset a metric on the server side - Promoetheus' model doesn't support this. But let me know if I miss anything.
The performance tests assume all the metrics all gathered and accessed in the same binary. As long as that's the case, works.
Code change:
q99 := hist.Quantile(0.95) avg := hist.Average() <del> // clear the metrics so that next test always starts with empty prometheus // metrics (since the metrics are shared among all tests run inside the same binary) hist.Clear() </del> msFactor := float64(time.Second) / float64(time.Millisecond) // Copy labels and add "Metric" label for this metric.

Related discussion:
text: We already call in the code:
That was my understanding as well, but no luck :( Here is a diff to call in case when "concreteOp.CollectMetrics" is false: <details<summaryDiff:</summary<p</p</detailsAnd below is the output - it's still 1500 (500 init pods plus 1000 measured pods) instead of 1000 being benchmarked: <details<summaryOutput:</summary<p</p</details
Also, if you run the tests and together, the result proves doesn't work as expected:
But I do find a way to fix this: .
Code change:
// we didn't validate against any provider, reject the pod and give the errors for each attempt klog.V(4).Infof("unable to validate pod %s (generate: %s) in namespace %s against any pod security policy: %v", pod.Name, pod.GenerateName, a.GetNamespace(), validationErrs) <del> return admission.NewForbidden(a, fmt.Errorf("unable to validate against any pod security policy: %v", validationErrs)) </del> <ins> return admission.NewForbidden(a, fmt.Errorf("PodSecurityPolicy: unable to validate pod: %v", validationErrs)) </ins> } func shouldIgnore(a admission.Attributes) (bool, error) {

Related discussion:
text: What happened: e2e tests detect whether PSP is enabled based on presence of PSP API objects. An addon that proactively includes a PSP in a manifest can trigger that detection and enable PSP-specific e2e tests that should not be run. What you expected to happen: In addition to detecting presence of PSP objects, consider dry-run creating a pod and verifying the annotation indicating the selected PSP policy is , indicating the admission plugin is active. /sig auth /assign
Code change:
return c.callCustom(sv, dv, fv, scope) } <ins> return c.defaultConvert(sv, dv, scope) } // defaultConvert recursively copies sv into dv. no conversion function is called // for the current stack frame (but conversion functions may be called for nested objects) func (c *Converter) defaultConvert(sv, dv reflect.Value, scope *scope) error { dt, st := dv.Type(), sv.Type() </ins> if !scope.flags.IsSet(AllowDifferentFieldTypeNames) && c.NameFunc(dt) != c.NameFunc(st) { return scope.error("type names don't match (%v, %v)", c.NameFunc(st), c.NameFunc(dt)) }

Related discussion:
text: Files storing latest versions for minors are not updated since infix was to the version string. Example Similar issues can be found in 1.25 and 1.24 builds (last 1.23 build infix so it has not happened there yet): I expect the files to be overwritten with newer version of the build. Manually run build job for one of the minors. No response Not related. Not related. No response No response No response No response
I believe this may be also responsible for breaking kubemark tests: since they fail due to version mismatch
/sig release
The label(s) cannot be applied, because the repository doesn't have them. <detailsIn response to : /priority high Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
Thanks for raising the issue! /cc who is working on before
I assume this will be fixed automatically with the next patches, because then we would compare with . Anyway, I think we should fix that intermediately by manually updating the version markers to match their correct version. I did that now for the bucket
We had to update the versions markers again, because those were pointing to two commits earlier and didn't include the Go updates. The latest version markers are: v1.24.12-35+ v1.25.8-37+ v1.26.3-41+
latest-1.27 has been affected by this as well. I updated it manually to v1.27.0-9+. See this thread for more details:
/triage accepted
This should be fixed now, closing it. Feel free to reopen if we missed anything.
This issue is still present, see this Slack thread for more details: /reopen
Reopened this issue. <detailsIn response to : This issue is still present, see this Slack thread for more details: /reopen Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
We have a similar situation with latest-1.28 which is currently pointing to: . This is one commit before the release commit: ! It's mostly like that the release step didn't update version markers, we should look into this.
Code change:
return c.callCustom(sv, dv, fv, scope) } <ins> return c.defaultConvert(sv, dv, scope) } // defaultConvert recursively copies sv into dv. no conversion function is called // for the current stack frame (but conversion functions may be called for nested objects) func (c *Converter) defaultConvert(sv, dv reflect.Value, scope *scope) error { dt, st := dv.Type(), sv.Type() </ins> if !scope.flags.IsSet(AllowDifferentFieldTypeNames) && c.NameFunc(dt) != c.NameFunc(st) { return scope.error("type names don't match (%v, %v)", c.NameFunc(st), c.NameFunc(dt)) }

Related discussion:
text: As of now, I'm going to manually update this version marker to unblock 1.28.1 patch release. I'll also take a look into this. /assign
The PR that fixed this issue got reverted so this is still relevant: /reopen
Reopened this issue. <detailsIn response to : The PR that fixed this issue got reverted so this is still relevant: /reopen Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
We're running into this issue again with stable-1.29 marker. It's currently pointing to which is It should instead point to I'm going to manually change it following the same procedure as the last time. Version marker updated to
This issue is labeled with but has not been updated in over 90 days, and should be re-triaged. Important-soon issues must be staffed and worked on either currently, or very soon, ideally in time for the next release. You can: Confirm that this issue is still relevant with (org members only) Deprioritize it with or Close this issue with For more details on the triage process, see /remove-triage accepted
/triage accepted
Same issue for 1.30, where I had to update the version marker manually again.
Code change:
return fmt.Errorf("failed to watch %s, err: %v", path, err) } case mode&os.ModeSocket != 0: <ins> w.wg.Add(1) </ins> go func() { <ins> defer w.wg.Done() </ins> w.fsWatcher.Events <- fsnotify.Event{ Name: path, Op:   fsnotify.Create,

Related discussion:
text: <!-- Please only use this template for submitting reports about failing tests in Kubernetes CI jobs --Which jobs are failing: https://k8s- Which test(s) are failing: should unmount if pod is force deleted while kubelet is down [Disruptive][Slow] Since when has it been failing: 11/07/18 15:02 Testgrid link: https://k8s- Reason for failure: The test case stops and starts kubelet, and it seems like kubelet is unable to start again Anything else we need to know: <!-- DO NOT EDIT BELOW THIS LINE --/kind failing-test
cc
Thanks for opening this Looking at , looks like could be the cause? /assign can you please look into this as priority. This is failing critical release blocking tests. /cc /sig cli /priority critical-urgent /milestone v1.13
It looks like kubelet keeps restarting. The error right before restarting is:
/assign
This bug could have been exposed by the newly tests in
Hmm, so I see two errors: Which is a Fatal error and would prevent Kubelet from starting anyways. In which case this error is handled correctly by the pluginwatcher, but I'm not sure preventing Kubelet from starting is the desired behavior. Should we log this error and skip it? There also seem to be a cleanup issue here on that file is this expected? This one is clearly a bug and not really handled correctly. It's due to the fact that we encounter an error at start, cleanup but don't wait for the gorountines to finish before cleaning up. I can send in a fix pretty easily for the second issue, basically add a waitgroup in the . For the first error, what do we think is the expected behavior?
This was mentioned in as well. /cc
Is this related to the issue where plugin watcher is recursively scanning for sockets?
Also, it should be noted that the path it's erroring on is not a socket type.
That's a bit different, as it is intended to scan recursively. Also note that the error here is produced by . What should be our behavior in such a case? Ignore that error?
I'm not sure if we want to be recursively scanning. The volume directories are mount points, which means there could be lots of files/directories existing in those volumes.
Code change:
return fmt.Errorf("failed to watch %s, err: %v", path, err) } case mode&os.ModeSocket != 0: <ins> w.wg.Add(1) </ins> go func() { <ins> defer w.wg.Done() </ins> w.fsWatcher.Events <- fsnotify.Event{ Name: path, Op:   fsnotify.Create,

Related discussion:
text: maybe we can put a limit on depth of plugin directory for on the same PR:
Ideally the directory that the plugin sockets are dropped in should be different from the directory where mounts happens. Currently the plugin socket directory is: ) The mounts directory is (so far so good). Problem is the "global mount" directory is also :( So options are: recursively scanning like suggested, and say plugins will can only be directly under This will break existing plugins, for example, the GCP plugin is at (see ). recursively scanning but change the plugin socket directory to ? This will also break existing plugins, like the GCP one pointed out above. the plugin dir the same, and leave recursion in, but put a limit on depth of plugin directory as suggested by This is ugly but may be the cleanest solution.
This conversation is completely different from the bug surfaced by this issue. Let's move that to the corresponding issue: I have a fix ready to go out for the panic, we should figure out what behavior we expect Kubelet to have when encountering this error from filepath.Walk: Should we ignore it?
Ignoring the error (with a warning maybe) instead of failing all of kubelet seems reasonable to me. If the plugin did not setup their registration socket correctly, then the end result should be that the plugin fails to register, but all other plugins should continue to work correctly.
the latest runs of this job no longer timesout - Instead I am seeing a new CSI related failure Is this expected. Do we want to track that failure in this bug now or do we have a separate issue?
the latest test logs indicate some test-infra issue:
the same shows up as a timeout on that particular test. Lets wait for the next run. but somehow the timeout seems to have gone. Are we still planning to merge and associated with this issue?
should be merged as it fixes a bug. can you help with approval? Thanks!
are you missing the right PR# in the comment above?
this is the correct PR -
/reopen Keeping it open until we can see the job pass a couple of rounds
Reopened this issue. <detailsIn response to : /reopen Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository.
Code change:
return fmt.Errorf("failed to watch %s, err: %v", path, err) } case mode&os.ModeSocket != 0: <ins> w.wg.Add(1) </ins> go func() { <ins> defer w.wg.Done() </ins> w.fsWatcher.Events <- fsnotify.Event{ Name: path, Op:   fsnotify.Create,

Related discussion:
text: </details
I see just one CSI test timing out in serial job Is that something the fixes here attempted to resolve? if not I can open a new issue if it persists and close this one out. Thanks.
The test mentioned in my comment above was a flake.. closing this issue now. will open a new issue for the flake
I'm getting the following related error on a busy Node: In the above case it's a full filesystem, but I imagine the same error would pop up with a RO filesystem as well. Looking at the patch, I'd expect kubelet would still be failing to start when it tries to watch it. (above was from a v1.12.3 cluster) , what do you reckon?
plugin watcher fixes for 1.12 are expected to be released with 1.12.5:
, I understand that I'm missing that patch. The patch code, however, would not make a difference in this case, if I read it correctly. It would not prevent the recursive dive into mounted volumes, as the errors happening at the deeper levels of that dive.
The patch blacklists the plugin watcher from searching the directory
OK, I was looking at the wrong patch after all (). Thanks for the clarifications.
Code change:
Name:      "total_preemption_attempts", Help:      "Total preemption attempts in the cluster till now", }) <ins> equivalenceCacheLookups = prometheus.NewCounterVec( prometheus.CounterOpts{ Subsystem: SchedulerSubsystem, Name:      "equiv_cache_lookups_total", Help:      "Total number of equivalence cache lookups, by whether or not a cache entry was found", }, []string{"result"}) EquivalenceCacheHits   = equivalenceCacheLookups.With(prometheus.Labels{"result": "hit"}) EquivalenceCacheMisses = equivalenceCacheLookups.With(prometheus.Labels{"result": "miss"}) EquivalenceCacheWrites = prometheus.NewCounterVec( prometheus.CounterOpts{ Subsystem: SchedulerSubsystem, Name:      "equiv_cache_writes", Help:      "Total number of equivalence cache writes, by result", }, []string{"result"}) </ins> metricsList = []prometheus.Collector{ SchedulingLatency, E2eSchedulingLatency,

Related discussion:
text: /kind feature The equivalence cache feature is planned to move to beta in 1.11, and it would be useful to have some metrics attached to it. The equivalence cache should improve scheduling throughput, so we want to be able to correlate e.g. throughput with eCache hit rate. /sig scheduling /sig instrumentation
/assign misterikkit
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
/remove-lifecycle stale
Code change:
} func getCPUStat(f *framework.Framework, host string) (usage, uptime float64) { <del> cpuCmd := "cat /sys/fs/cgroup/cpu/system.slice/node-problem-detector.service/cpuacct.usage && cat /proc/uptime | awk '{print $1}'" </del> <ins> var cpuCmd string if isHostRunningCgroupV2(f, host) { cpuCmd = " cat /sys/fs/cgroup/cpu.stat | grep 'usage_usec' | sed 's/[^0-9]*//g' && cat /proc/uptime | awk '{print $1}'" } else { cpuCmd = "cat /sys/fs/cgroup/cpu/system.slice/node-problem-detector.service/cpuacct.usage && cat /proc/uptime | awk '{print $1}'" } </ins> result, err := e2essh.SSH(cpuCmd, host, framework.TestContext.Provider) framework.ExpectNoError(err) framework.ExpectEqual(result.Code, 0)

Related discussion:
text: N/A [sig-node] NodeProblemDetector: N/A https://prow- Not cgroupv2 compatible when access cpu and memory data
/sig node
Hi, can i work on this ?
Sure. Thanks
/assign
/triage accepted
/priority important-soon
/cc
are you still working on this? Were you able to setup environment to repro the test failure? Let us know if you need any help. Thanks!
Sorry for the delay, I followed the wrong guide and ended up getting stuck. But now I have managed to set it up. Please give me few more days to work on it.
Thanks Feel free to reach out here or on slack if you need any help, thanks for working on this!
/retitle Make nodeproblemdetector test cgroupv2 compatible
Failing job:
Testgrid:
Code change:
// we expect following values for: Major -> digit, Minor -> numeric followed by an optional '+',  GitCommit -> alphanumeric requiredItems := []string{"Client Version: ", "Server Version: "} for _, item := range requiredItems { <del> if matched, _ := regexp.MatchString(item+`vd.d+.[dw-.+]+`, versionString); !matched { </del> <ins> // prior to 1.28 we printed long version information oldMatched, _ := regexp.MatchString(item+`version.Info{Major:"d", Minor:"d++?", GitVersion:"vd.d+.[dw-.+]+", GitCommit:"[0-9a-f]+"`, versionString) // 1.28+ prints short information newMatched, _ := regexp.MatchString(item+`vd.d+.[dw-.+]+`, versionString) // due to backwards compatibility we need to match both until 1.30 most likely if !oldMatched && !newMatched { </ins> framework.Failf("Item %s value is not valid in %sn", item, versionString) } }

Related discussion:
text: master-informing: capz-windows-master ! 07-10 17:46 IST No response /sig cli cc
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/kind failing-test
Looking at the failure and tracing the recent code changes, the test was recently modified for the regex pattern match as part of this PR which comes with deprecation of the golang-based short version information in favor of making the --short the new default. But the output of kubectl version is still reflecting the older long format which is causing this test to fail.
/assign