jiebi/CodeConvo huggingface dataset:
Code change:
<!-- Please only use this template for submitting enhancement requests --What would you like to be : We should allow collecting both default scheduler metrics (Prometheus) and custom metrics easily. At a high level, these collectors should produce a json formatted result which will then be consumed by perf-dash to present in the dashboard. Prometheus metrics We should provide a metrics collector that collects and summarizes default scheduler metrics defined in . To begin with, we can manually pick a list of interesting metrics. Longer term, we could allow configuring the metric list in test definition. already had a PoC . As a reference, is how clusterloader collects scheduler latency metrics. Custom metrics The most important metric is scheduling throughput. is how clusterloader calculates the throughput. We can start with just calculating the average throughput without the percentiles. We may want other custom metrics that are either common to all tests or specific to certain tests. We should allow these custom collectors to be easily and ideally configurable. Why is this needed: Give more insights into scheduler performance. Part of /sig scheduling /assign
/reopen
Reopened this issue. <detailsIn response to : /reopen Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details

Related discussion:
text: // start benchmark b.ResetTimer() <ins> // Start measuring throughput stopCh := make(chan struct{}) throughputCollector := newThroughputCollector(podInformer) go throughputCollector.run(stopCh) // Scheduling the main workload </ins> config = testutils.NewTestPodCreatorConfig() config.AddStrategy(testNamespace, test.PodsToSchedule.Num, testPodStrategy) podCreator = testutils.NewTestPodCreator(clientset, config) podCreator.CreatePods() <-completedCh <ins> close(stopCh) </ins> // Note: without this line we're taking the overhead of defer() into account. b.StopTimer() <ins> setNameLabel := func(dataItem *DataItem) DataItem { if dataItem.Labels == nil { dataItem.Labels = map[string]string{} } dataItem.Labels["Name"] = b.Name() return *dataItem } dataItems := []DataItem{ setNameLabel(throughputCollector.collect()), } for _, metric := range defaultMetrics { dataItem := newPrometheusCollector(metric).collect() if dataItem == nil { continue } dataItems = append(dataItems, setNameLabel(dataItem)) } return dataItems </ins> } func getPodStrategy(pc podCase) testutils.TestPodCreateStrategy {
Code change:
<!-- Please only use this template for submitting enhancement requests --What would you like to be : We should allow collecting both default scheduler metrics (Prometheus) and custom metrics easily. At a high level, these collectors should produce a json formatted result which will then be consumed by perf-dash to present in the dashboard. Prometheus metrics We should provide a metrics collector that collects and summarizes default scheduler metrics defined in . To begin with, we can manually pick a list of interesting metrics. Longer term, we could allow configuring the metric list in test definition. already had a PoC . As a reference, is how clusterloader collects scheduler latency metrics. Custom metrics The most important metric is scheduling throughput. is how clusterloader calculates the throughput. We can start with just calculating the average throughput without the percentiles. We may want other custom metrics that are either common to all tests or specific to certain tests. We should allow these custom collectors to be easily and ideally configurable. Why is this needed: Give more insights into scheduler performance. Part of /sig scheduling /assign
/reopen
Reopened this issue. <detailsIn response to : /reopen Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details

Related discussion:
text: package benchmark import ( <ins> "encoding/json" "flag" "fmt" "io/ioutil" "math" "path" "sort" "time" dto "github.com/prometheus/client_model/go" </ins> v1 "k8s.io/api/core/v1" "k8s.io/apimachinery/pkg/labels" "k8s.io/apimachinery/pkg/runtime/schema" coreinformers "k8s.io/client-go/informers/core/v1" clientset "k8s.io/client-go/kubernetes" restclient "k8s.io/client-go/rest" <ins> "k8s.io/component-base/metrics/legacyregistry" "k8s.io/klog" </ins> "k8s.io/kubernetes/test/integration/util" ) <ins> const ( dateFormat                = "2006-01-02T15:04:05Z" throughputSampleFrequency = time.Second ) var dataItemsDir = flag.String("data-items-dir", "", "destination directory for storing generated data items for perf dashboard") </ins> // mustSetupScheduler starts the following components: // - k8s api server (a.k.a. master) // - scheduler
Code change:
This issue proposes the writing of a new test for the API endpoint with the intention of promoting it to Conformance. This issue contains a very masic mock test in order to start the conversation about what a good e2e test would look like for this endpoint. The code below is verified to be hitting the intended endpoint, show as per the queries in APIsnoop's live view of the cluster's audit logs. According to this query, there is only one API endpoint which is for Namespace that isn't being covered. We selected patchCoreV1Namespace to write a test for. Dependencies: The mock test: If a test with these calls is build and merged, conformance will go up by 1 endpoint.
/sig testing /sig architecture /area conformance

Related discussion:
text: <ins> # See the OWNERS docs at https://go.k8s.io/owners approvers: - lavalamp - smarterclayton - deads2k - sttts - liggitt - caesarxuchao reviewers: - thockin - lavalamp - smarterclayton - wojtek-t - deads2k - derekwaynecarr - caesarxuchao - cheftako - mikedanese - liggitt - gmarek - sttts - ncdc - logicalhan - tallclair labels: - sig/api-machinery </ins>  No newline at end of file
Code change:
This issue proposes the writing of a new test for the API endpoint with the intention of promoting it to Conformance. This issue contains a very masic mock test in order to start the conversation about what a good e2e test would look like for this endpoint. The code below is verified to be hitting the intended endpoint, show as per the queries in APIsnoop's live view of the cluster's audit logs. According to this query, there is only one API endpoint which is for Namespace that isn't being covered. We selected patchCoreV1Namespace to write a test for. Dependencies: The mock test: If a test with these calls is build and merged, conformance will go up by 1 endpoint.
/sig testing /sig architecture /area conformance

Related discussion:
text: package apimachinery import ( <ins> "encoding/json" </ins> "fmt" "strings" "sync"
Code change:
This issue proposes the writing of a new test for the API endpoint with the intention of promoting it to Conformance. This issue contains a very masic mock test in order to start the conversation about what a good e2e test would look like for this endpoint. The code below is verified to be hitting the intended endpoint, show as per the queries in APIsnoop's live view of the cluster's audit logs. According to this query, there is only one API endpoint which is for Namespace that isn't being covered. We selected patchCoreV1Namespace to write a test for. Dependencies: The mock test: If a test with these calls is build and merged, conformance will go up by 1 endpoint.
/sig testing /sig architecture /area conformance

Related discussion:
text: apierrors "k8s.io/apimachinery/pkg/api/errors" metav1 "k8s.io/apimachinery/pkg/apis/meta/v1" "k8s.io/apimachinery/pkg/util/intstr" <ins> "k8s.io/apimachinery/pkg/util/uuid" </ins> "k8s.io/apimachinery/pkg/util/wait" "k8s.io/kubernetes/test/e2e/framework" e2epod "k8s.io/kubernetes/test/e2e/framework/pod" imageutils "k8s.io/kubernetes/test/utils/image" "github.com/onsi/ginkgo" <ins> "k8s.io/apimachinery/pkg/types" </ins> ) func extinguish(f *framework.Framework, totalNS int, maxAllowedAfterDel int, maxSeconds int) {
Code change:
This issue proposes the writing of a new test for the API endpoint with the intention of promoting it to Conformance. This issue contains a very masic mock test in order to start the conversation about what a good e2e test would look like for this endpoint. The code below is verified to be hitting the intended endpoint, show as per the queries in APIsnoop's live view of the cluster's audit logs. According to this query, there is only one API endpoint which is for Namespace that isn't being covered. We selected patchCoreV1Namespace to write a test for. Dependencies: The mock test: If a test with these calls is build and merged, conformance will go up by 1 endpoint.
/sig testing /sig architecture /area conformance

Related discussion:
text: ginkgo.It("should always delete fast (ALL of 100 namespaces in 150 seconds) [Feature:ComprehensiveNamespaceDraining]", func() { extinguish(f, 100, 0, 150) }) <ins> ginkgo.It("should patch a Namespace", func() { ginkgo.By("creating a Namespace") namespaceName := "nspatchtest-" + string(uuid.NewUUID()) ns, err := f.CreateNamespace(namespaceName, nil) framework.ExpectNoError(err, "failed creating Namespace") namespaceName = ns.ObjectMeta.Name ginkgo.By("patching the Namespace") nspatch, err := json.Marshal(map[string]interface{}{ "metadata": map[string]interface{}{ "labels": map[string]string{"testLabel": "testValue"}, }, }) framework.ExpectNoError(err, "failed to marshal JSON patch data") _, err = f.ClientSet.CoreV1().Namespaces().Patch(namespaceName, types.StrategicMergePatchType, []byte(nspatch)) framework.ExpectNoError(err, "failed to patch Namespace") ginkgo.By("get the Namespace and ensuring it has the label") namespace, err := f.ClientSet.CoreV1().Namespaces().Get(namespaceName, metav1.GetOptions{}) framework.ExpectNoError(err, "failed to get Namespace") framework.ExpectEqual(namespace.ObjectMeta.Labels["testLabel"], "testValue", "namespace not patched") }) </ins> })
Code change:
<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --What happened: With client-go v0.17.0, I observed many "Throttling request" when DiscoveryClient is used to get API group resources from API server. This didn't happen with client-go kubernetes-1.16.0. It looks like originally this kind of discovery requests are not throttled because no rate limiter provided and QPS == 0. But with rate limiter is always set if nil rate limiter is provided. is particularly suspicious because it is always true because DefaultQPS is nonzero. What you expected to happen: Discovery requests should not be throttled as before. How to reproduce it (as minimally and precisely as possible): Create DiscoveryClient and then call with it. Anything else we need to know?: Environment: Kubernetes version (use ): Cloud provider or hardware configuration: OS (e.g: ): Kernel (e.g. ): Install tools: Network plugin and version (if this is a network-related bug): Others:
/sig api-machinery
The NewTokenBucketRateLimiter was via Always negotiate a decoder using ClientNegotiator /cc
To be clear, that was fixing a bug where the documented defaults for rate limiting were not taking effect. I don't think we should completely disable throttling until more of the server-side priority work is completed, but discovery is expected to be bursty, so I could see defaulting to a different amount in the discovery client constructor, similar to
Yeah, everyone must be rate limited. I think we could give discovery a big burst value.
opened

Related discussion:
text: ## Known Issues - volumeDevices mapping ignored when container is privileged - The `Should recreate evicted statefulset` conformance [test]( https://github.com/kubernetes/kubernetes/blob/master/test/e2e/apps/statefulset.go) fails because `Pod ss-0 expected to be re-created at least once`. This was caused by the `Predicate PodFitsHostPorts failed` scheduling error. The root cause was a host port conflict for port `21017`. This port was in-use as an ephemeral port by another application running on the node. This will be looked at for the 1.18 release. <ins> - client-go discovery clients constructed using `NewDiscoveryClientForConfig` or `NewDiscoveryClientForConfigOrDie` default to rate limits that cause normal discovery request patterns to take several seconds. This is fixed in https://issue.k8s.io/86168 and will be resolved in v1.17.1. As a workaround, the `Burst` value can be adjusted higher in the rest.Config passed into `NewDiscoveryClientForConfig` or `NewDiscoveryClientForConfigOrDie`. </ins> ## Urgent Upgrade Notes ### (No, really, you MUST read this before you upgrade)
Code change:
<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --What happened: With client-go v0.17.0, I observed many "Throttling request" when DiscoveryClient is used to get API group resources from API server. This didn't happen with client-go kubernetes-1.16.0. It looks like originally this kind of discovery requests are not throttled because no rate limiter provided and QPS == 0. But with rate limiter is always set if nil rate limiter is provided. is particularly suspicious because it is always true because DefaultQPS is nonzero. What you expected to happen: Discovery requests should not be throttled as before. How to reproduce it (as minimally and precisely as possible): Create DiscoveryClient and then call with it. Anything else we need to know?: Environment: Kubernetes version (use ): Cloud provider or hardware configuration: OS (e.g: ): Kernel (e.g. ): Install tools: Network plugin and version (if this is a network-related bug): Others:
/sig api-machinery
The NewTokenBucketRateLimiter was via Always negotiate a decoder using ClientNegotiator /cc
To be clear, that was fixing a bug where the documented defaults for rate limiting were not taking effect. I don't think we should completely disable throttling until more of the server-side priority work is completed, but discovery is expected to be bursty, so I could see defaulting to a different amount in the discovery client constructor, similar to
Yeah, everyone must be rate limited. I think we could give discovery a big burst value.
opened

Related discussion:
text: if config.Timeout == 0 { config.Timeout = defaultTimeout } <ins> if config.Burst == 0 && config.QPS < 100 { // discovery is expected to be bursty, increase the default burst // to accommodate looking up resource info for many API groups. // matches burst set by ConfigFlags#ToDiscoveryClient(). // see https://issue.k8s.io/86149 config.Burst = 100 } </ins> codec := runtime.NoopEncoder{Decoder: scheme.Codecs.UniversalDecoder()} config.NegotiatedSerializer = serializer.NegotiatedSerializerWrapper(runtime.SerializerInfo{Serializer: codec}) if len(config.UserAgent) == 0 {
Code change:
<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --What happened: With client-go v0.17.0, I observed many "Throttling request" when DiscoveryClient is used to get API group resources from API server. This didn't happen with client-go kubernetes-1.16.0. It looks like originally this kind of discovery requests are not throttled because no rate limiter provided and QPS == 0. But with rate limiter is always set if nil rate limiter is provided. is particularly suspicious because it is always true because DefaultQPS is nonzero. What you expected to happen: Discovery requests should not be throttled as before. How to reproduce it (as minimally and precisely as possible): Create DiscoveryClient and then call with it. Anything else we need to know?: Environment: Kubernetes version (use ): Cloud provider or hardware configuration: OS (e.g: ): Kernel (e.g. ): Install tools: Network plugin and version (if this is a network-related bug): Others:
/sig api-machinery
The NewTokenBucketRateLimiter was via Always negotiate a decoder using ClientNegotiator /cc
To be clear, that was fixing a bug where the documented defaults for rate limiting were not taking effect. I don't think we should completely disable throttling until more of the server-side priority work is completed, but discovery is expected to be bursty, so I could see defaulting to a different amount in the discovery client constructor, similar to
Yeah, everyone must be rate limited. I think we could give discovery a big burst value.
opened

Related discussion:
text: "net/http/httptest" "reflect" "testing" <ins> "time" </ins> "github.com/gogo/protobuf/proto" "github.com/googleapis/gnostic/OpenAPIv2"
Code change:
<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --What happened: With client-go v0.17.0, I observed many "Throttling request" when DiscoveryClient is used to get API group resources from API server. This didn't happen with client-go kubernetes-1.16.0. It looks like originally this kind of discovery requests are not throttled because no rate limiter provided and QPS == 0. But with rate limiter is always set if nil rate limiter is provided. is particularly suspicious because it is always true because DefaultQPS is nonzero. What you expected to happen: Discovery requests should not be throttled as before. How to reproduce it (as minimally and precisely as possible): Create DiscoveryClient and then call with it. Anything else we need to know?: Environment: Kubernetes version (use ): Cloud provider or hardware configuration: OS (e.g: ): Kernel (e.g. ): Install tools: Network plugin and version (if this is a network-related bug): Others:
/sig api-machinery
The NewTokenBucketRateLimiter was via Always negotiate a decoder using ClientNegotiator /cc
To be clear, that was fixing a bug where the documented defaults for rate limiting were not taking effect. I don't think we should completely disable throttling until more of the server-side priority work is completed, but discovery is expected to be bursty, so I could see defaulting to a different amount in the discovery client constructor, similar to
Yeah, everyone must be rate limited. I think we could give discovery a big burst value.
opened

Related discussion:
text: list = &beta case "/apis/extensions/v1beta2": list = &beta2 <ins> case "/apis/extensions/v1beta3": list = &extensionsbeta3 case "/apis/extensions/v1beta4": list = &extensionsbeta4 case "/apis/extensions/v1beta5": list = &extensionsbeta5 case "/apis/extensions/v1beta6": list = &extensionsbeta6 case "/apis/extensions/v1beta7": list = &extensionsbeta7 case "/apis/extensions/v1beta8": list = &extensionsbeta8 case "/apis/extensions/v1beta9": list = &extensionsbeta9 case "/apis/extensions/v1beta10": list = &extensionsbeta10 case "/apis/apps/v1beta1": list = &appsbeta1 case "/apis/apps/v1beta2": list = &appsbeta2 case "/apis/apps/v1beta3": list = &appsbeta3 case "/apis/apps/v1beta4": list = &appsbeta4 case "/apis/apps/v1beta5": list = &appsbeta5 case "/apis/apps/v1beta6": list = &appsbeta6 case "/apis/apps/v1beta7": list = &appsbeta7 case "/apis/apps/v1beta8": list = &appsbeta8 case "/apis/apps/v1beta9": list = &appsbeta9 case "/apis/apps/v1beta10": list = &appsbeta10 </ins> case "/api": list = &metav1.APIVersions{ Versions: []string{
Code change:
<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --What happened: With client-go v0.17.0, I observed many "Throttling request" when DiscoveryClient is used to get API group resources from API server. This didn't happen with client-go kubernetes-1.16.0. It looks like originally this kind of discovery requests are not throttled because no rate limiter provided and QPS == 0. But with rate limiter is always set if nil rate limiter is provided. is particularly suspicious because it is always true because DefaultQPS is nonzero. What you expected to happen: Discovery requests should not be throttled as before. How to reproduce it (as minimally and precisely as possible): Create DiscoveryClient and then call with it. Anything else we need to know?: Environment: Kubernetes version (use ): Cloud provider or hardware configuration: OS (e.g: ): Kernel (e.g. ): Install tools: Network plugin and version (if this is a network-related bug): Others:
/sig api-machinery
The NewTokenBucketRateLimiter was via Always negotiate a decoder using ClientNegotiator /cc
To be clear, that was fixing a bug where the documented defaults for rate limiting were not taking effect. I don't think we should completely disable throttling until more of the server-side priority work is completed, but discovery is expected to be bursty, so I could see defaulting to a different amount in the discovery client constructor, similar to
Yeah, everyone must be rate limited. I think we could give discovery a big burst value.
opened

Related discussion:
text: list = &metav1.APIGroupList{ Groups: []metav1.APIGroup{ { <ins> Name: "apps", Versions: []metav1.GroupVersionForDiscovery{ {GroupVersion: "apps/v1beta1", Version: "v1beta1"}, {GroupVersion: "apps/v1beta2", Version: "v1beta2"}, {GroupVersion: "apps/v1beta3", Version: "v1beta3"}, {GroupVersion: "apps/v1beta4", Version: "v1beta4"}, {GroupVersion: "apps/v1beta5", Version: "v1beta5"}, {GroupVersion: "apps/v1beta6", Version: "v1beta6"}, {GroupVersion: "apps/v1beta7", Version: "v1beta7"}, {GroupVersion: "apps/v1beta8", Version: "v1beta8"}, {GroupVersion: "apps/v1beta9", Version: "v1beta9"}, {GroupVersion: "apps/v1beta10", Version: "v1beta10"}, }, }, { </ins> Name: "extensions", Versions: []metav1.GroupVersionForDiscovery{ {GroupVersion: "extensions/v1beta1", Version: "v1beta1"}, {GroupVersion: "extensions/v1beta2", Version: "v1beta2"}, <ins> {GroupVersion: "extensions/v1beta3", Version: "v1beta3"}, {GroupVersion: "extensions/v1beta4", Version: "v1beta4"}, {GroupVersion: "extensions/v1beta5", Version: "v1beta5"}, {GroupVersion: "extensions/v1beta6", Version: "v1beta6"}, {GroupVersion: "extensions/v1beta7", Version: "v1beta7"}, {GroupVersion: "extensions/v1beta8", Version: "v1beta8"}, {GroupVersion: "extensions/v1beta9", Version: "v1beta9"}, {GroupVersion: "extensions/v1beta10", Version: "v1beta10"}, </ins> }, }, },