jiebi/CodeConvo huggingface dataset:
Change:
return } <del> ch <- metrics.NewLazyConstMetric(containerStartTimeDesc, metrics.GaugeValue, float64(s.StartTime.UnixNano())/float64(time.Second), s.Name, pod.PodRef.Name, pod.PodRef.Namespace) </del> <ins> ch <- metrics.NewLazyMetricWithTimestamp(s.StartTime.Time, metrics.NewLazyConstMetric(containerStartTimeDesc, metrics.GaugeValue, float64(s.StartTime.UnixNano())/float64(time.Second), s.Name, pod.PodRef.Name, pod.PodRef.Namespace)) </ins> } func (rc *resourceMetricsCollector) collectContainerCPUMetrics(ch chan<- metrics.Metric, pod summary.PodStats, s summary.ContainerStats) {

Discussion:
<!-- Please only use this template for submitting reports about continuously failing tests or jobs in Kubernetes CI -- (containerd) (crio) [sig-node] ResourceMetricsAPI [NodeFeature:ResourceMetrics] when querying /resource/metrics should report resource usage through the resource metrics api Ever since this commit was merged (containerd) (crio) We need to see what the commit is breaking this particular NodeFeatures test on both crio and containerd.
/sig node
/cc
/cc
/triage accepted
ah, let's use the RT's issue /close
Closing this issue. <detailsIn response to : ah, let's use the RT's issue Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
Change:
func TestWatchSucceedsWithoutArgs(t *testing.T) { _, s := framework.RunAMaster(t) <del> defer s.Close() </del> <ins> // TODO: Uncomment when fix #19254 // defer s.Close() </ins> resp, err := http.Get(s.URL + "/api/v1/namespaces?watch=1") if err != nil {

Discussion:
http://pr- is another failure.
Also seen in http://pr-
Again. Not assigned to anyone, so I am sending to for routing
Again: Is there anyone working on this? I will spend some time looking into this.
I think that the name of this issue is misleading - the issue seems to be unrelated to etcd_watcher Basically, if you look into this part of log: So it seems that, the WaitGroup that is problematic is in httptest.Server. So my feeling is that we are simply talking to a server that is already being close and this is ~roughly duplicate of We should just find where exactly the problem is and try to work-around it.
I think this is exactly what we had before. I will send a PR with the fix today.
Change:
} } <del> const networkContainerName = "k8snet" </del> <ins> const networkContainerName = "net" </ins> func (kl *Kubelet) networkContainerExists(manifest *api.ContainerManifest) (string, bool, error) { pods, err := kl.ListContainers()

Discussion:
release-master-blocking Conformance-GCE-master-kubetest2Kubernetes e2e suite.[It] [sig-network] Services should be able to change the type from ExternalName to ClusterIP [Conformance] ci-kubernetes-gce-conformance-latest-kubetest2.Overall kubetest2.Test 07-11 15:51 EEST No response /sig cloud-provider cc
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
I remember some timeout has changed with ginkgo2 and something moved from 24hours to 1 hours, or something like that, is it possible that this is the problem here, in this job?
The Grinko timeout change could also be the root problem with this job:
hmm, we have to revisit the flags
hmm, this seems correct
that was a good observations, seems that job is failing because gathering the logs takes too much time !
there is no timeout set in the kubetest2 command here (not sure if this would be the place to set a timeout)
it seems kubetest has to be migrated to ginkgov2 it should use this as basis I'm on vacation this week, can somebody take over and open the issue and patch kubetest2?
looks like this file needs to get updated: add line 72:
oh, you already commented this :D
I will have a look later
Related: umbrella issue:
last testgrid run passed :)
Change:
apiextensions.CRDRemoveFinalizer(crd, apiextensions.CustomResourceCleanupFinalizer) crd, err = c.crdClient.CustomResourceDefinitions().UpdateStatus(crd) <ins> if apierrors.IsNotFound(err) || apierrors.IsConflict(err) { // deleted or changed in the meantime, we'll get called again return nil } </ins> if err != nil { return err }

Discussion:
k8s version: v1.9.3 When I create customresourcedefinition, There is error like Operation cannot be fulfilled on "": the object has been modified; please apply your changes to the latest version and try again . apiVersion: kind: CustomResourceDefinition metadata: name: annotations: "": crd-install labels: app: istio-pilot spec: group: names: kind: VirtualService listKind: VirtualServiceList plural: virtualservices singular: virtualservice categories: istio-io networking-istio-io scope: Namespaced version: v1alpha3 So what is the reason of it?
a few questions: where are you seeing the error, in the apiserver log or in the output of the kubectl command? what kubectl command are you using to create the object? are you running HA apiservers?
The log is shown by apiserver. I run k8s in HA apiservers. The command I used is "kubectl apply -f "+file.
API servers attempt to update CRD status to indicate whether the names were available. In an HA apiserver setup, the API servers race to do so, and all but the first lose the race with a resource conflict error. That is not unexpected. Opened to avoid logging normal API responses in that scenario.
Thanks ! But I want to know whether the CRD has been set successfully in HA apiservers ? I use the command " kubectl get crd -o yaml" and get the following results. ! I see its status is true. But when i create a virtualservices, it returns error. Error from server (Timeout): error when creating "cms-api-": Timeout: request did not complete within allowed duration ! So i want to know how to solve this error.
/assign
Excuse me. Do you have some greate ideas to solve this issue??
There is a PR for this sent by
I see your request is timed-out not failed. In HA, I expect the request to fail in the first seconds of the CRD creation, but after that it should go through. I need more logs during this timeout or a consistent way to reproduce it.
Change:
} configFileName := filepath.Join(dir, "config.yaml") <del> if err := os.WriteFile(configFileName, []byte(` </del> <ins> if err := atomicWriteFile(configFileName, []byte(` </ins> apiVersion: apiserver.config.k8s.io/v1alpha1 kind: AuthorizationConfiguration authorizers:

Discussion:
ci-kubernetes-integration-master --- FAIL: TestMultiWebhookAuthzConfig (9.23s) (03-20) I cannot search for previous flake. updated recently /sig auth /cc
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
Updated the description, the actual cause of the flake is an unexpected change in the file reload timestamp metric
interestingly, doesn't show any failures of this test for this reason (there are a few failures on a PR that broke a bunch of integration tests for a different reason)
Fix open in , switched the file write to an atomic write
No flake after 03-29. Thanks.
Change:
} } <del> func (fopg *fakeOperationGenerator) GenerateMountVolumeFunc(waitForAttachTimeout time.Duration, volumeToMount VolumeToMount, actualStateOfWorldMounterUpdater ActualStateOfWorldMounterUpdater, isRemount bool) (volumetypes.GeneratedOperations, error) { </del> <ins> func (fopg *fakeOperationGenerator) GenerateMountVolumeFunc(waitForAttachTimeout time.Duration, volumeToMount VolumeToMount, actualStateOfWorldMounterUpdater ActualStateOfWorldMounterUpdater, isRemount bool) volumetypes.GeneratedOperations { </ins> opFunc := func() (error, error) { startOperationAndBlock(fopg.ch, fopg.quit) return nil, nil } return volumetypes.GeneratedOperations{ OperationFunc: opFunc, <del> }, nil </del> <ins> } </ins> } func (fopg *fakeOperationGenerator) GenerateUnmountVolumeFunc(volumeToUnmount MountedVolume, actualStateOfWorld ActualStateOfWorldMounterUpdater, podsDir string) (volumetypes.GeneratedOperations, error) { opFunc := func() (error, error) {

Discussion:
<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks!--What happened: In this test, volume mount failed because the driver-registrar took a minute to come up. However, it seems like volume mount operation is retrying without any sort of backoff. This message appears in 400+ times in less than a minute. What you expected to happen: Operation executor should not be retrying operations so frequently. <!-- DO NOT EDIT BELOW THIS LINE --/kind bug
I'll take a look at this. /assign
Problem is that reconciler depends on operation executor to do backoff. In the normal case operation executor returns a method that does the actual work. Any errors by that method result in exponential backoff. But in this case, the error is coming from the code creating the mount method which is not handled by the back off logic, and is called again and again by the reconciler resulting in log and event spam. This didn't use to be a problem because initialization errors were uncommon (plugins were built in), but with CSI the driver may not exist and these initialization errors are expected. I'll look in to cleaning this up.
Is this something that will affect users in 1.13 and needs CPing?
This will result in kubelet log spam and possibly kubernetes event spam. It happens when someone creates a pod referencing a CSI volume before the CSI driver for that volume is installed. Which happens frequently in E2E tests because they install/remove driver with every test. But should be rare in real world usage since most users are likely to install a driver first before using it.
If release team is worried about this, we could create a quick PR to suppress logging/event generation of this these errors. Otherwise we plan to address this more holistically for 1.14 and 1.13.1.
Change:
options.Limit = 0 options.Continue = "" options.ResourceVersion = requestedResourceVersion <ins> options.ResourceVersionMatch = requestedResourceVersionMatch </ins> result, err := p.PageFn(ctx, options) return result, paginatedResult, err }

Discussion:
We recently started using the list pager from client-go as part of Calico, and have seen reports of the following error showing up: No error, just beautiful paged list calls. Create 1,000 resources of a particular type, such as a NetworkPolicy. (Need 500 to trigger paging) Do a of all those resources with set. Originally filed here (but maybe that repo is not monitored): Reference issue in Calico: projectcalico/calico (comment) Reference Calico list pager code: We think this is the required fix: <detailsclient-go version (but bug seems to be in master too). </details<details</details<details</details<details</details<details</details<details</details
/sig api-machinery
/triage accepted
Change:
package glusterfs import ( <del> "math/rand" </del> "os" <ins> "path" </ins> "github.com/golang/glog" "k8s.io/kubernetes/pkg/api"

Discussion:
<!-- Please only use this template for submitting reports about failing tests in Kubernetes CI jobs --Which jobs are failing: pull-kubernetes-e2e-aks-engine-azure Which test(s) are failing: Since when has it been failing: Testgrid link: No Testgrid, but job history: Reason for failure: Anything else we need to know: ref:
/assign cc: /milestone v1.16 /priority important-soon
/sig azure
cc
Related with
Update: has been merged, but a new release with this change is not available yet. Pending on aks-engine's next release.
/sig cloud-provider
v1.16.0-alpha.1 is available in
Will submit a PR after testing this in our own prow first.
Change:
if err == nil { t.Errorf("%s: Expected failure", protoStr) } <ins> if string(buffer.Bytes()) != stderrOutput { t.Errorf("%s: Expected output '%s', got '%v'", protoStr, stderrOutput, string(buffer.Bytes())) } </ins> } func TestSaveIntoIPv4(t *testing.T) {

Discussion:
On some systems, iptables-save outputs the following warning to stderr: Note that the error message includes a newline. Because kube-proxy captures both stdout and stderr when calling , the newline might cause rules to be garbled. For example, the output might contain the following line: The parser treats this as a chain line, and panics, as predicted in
/sig network
FYI
/triage unresolved Comment when the issue is assessed and confirmed. I am a bot run by vllry. ‍
/assign
Just FYI here's another example of garbled output:
Anyway, fix is in Please review.
/assign
/remove-triage unresolved
Change:
- list - watch - apiGroups: <ins> - discovery.k8s.io resources: - endpointslices verbs: - get - list - watch - apiGroups: </ins> - apps resources: - controllerrevisions

Discussion:
Users are not able to view by default as part of the or roles like they are with Users with or roles should be able to view like they are with Have a user with permissions on a namespace. Try to run the command I asked about this in slack and was told that it was an oversight that it wasn't to the default roles. Kubernetes version (use ): 1.21 Cloud provider or hardware configuration: n/a OS (e.g: ): n/a Kernel (e.g. ): n/a Install tools: n/a Network plugin and version (if this is a network-related bug): n/a Others: n/a
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/sig security
/sig auth
Change:
// calculatePodResourceRequest returns the total non-zero requests. If Overhead is defined for the pod and the // PodOverhead feature is enabled, the Overhead is added to the result. <ins> // podResourceRequest = max(sum(podSpec.Containers), podSpec.InitContainers) + overHead </ins> func calculatePodResourceRequest(pod *v1.Pod, resource v1.ResourceName) int64 { var podRequest int64 for i := range pod.Spec.Containers {

Discussion:
The scheduler does not properly calculate pod requests for a resource as - it ignores init containers completely in . This would allow a workload to get "free" scheduling above the limit of containers (mitigated by quota, which calculates this properly) and so the scheduler would make non-optimal fits. Both kubelet and scheduler enforce the , so you can't actually schedule, but you sometimes get steered to the wrong node if you have more init container resource usage than containers. in should use and the similar helpers instead of writing its own method to calculate the request for resources. /sig scheduling
/priority important-soon
Let me follow this issue. /assign
fyi
After I checked the implementation of function, I think we just need to use it to instead of the function in the scheduler.
/priority critical-urgent
Is the pod phase not considered? When the pod phase is running, the requested resource is actually considered as . This change seems to reduce the resource utilization of the node.
Actually, I don't think we need to consider the pod.Phase. The resource request & limit are mainly used in the scheduling process. The scenario you said is not belong to the responsibility of kube-scheduler
Correct. Setting appropriate requests/limits that maintain good node usage is users responsibility. Considering the max(initContainers, sum(containers)) is the documented behavior.
I understand. Thank you for your comments!
Change:
KUBE_DELETE_NETWORK=${KUBE_DELETE_NETWORK:-true} MASTER_OS_DISTRIBUTION=${KUBE_MASTER_OS_DISTRIBUTION:-${KUBE_OS_DISTRIBUTION:-gci}} <del> NODE_OS_DISTRIBUTION=${KUBE_NODE_OS_DISTRIBUTION:-${KUBE_OS_DISTRIBUTION:-debian}} </del> <ins> NODE_OS_DISTRIBUTION=${KUBE_NODE_OS_DISTRIBUTION:-${KUBE_OS_DISTRIBUTION:-gci}} </ins> if [[ "${MASTER_OS_DISTRIBUTION}" == "coreos" ]]; then MASTER_OS_DISTRIBUTION="container-linux" fi

Discussion:
The debian-based CVM image is going to be deprecated in September. We should: Change the default OS image for GCE cluster setup script. This means that all the GCE CI builds will use the new default OS image. Add deprecation warning to the cluster setup script if CVM is specified. Maintain the node e2e test coverage for CVM for 1.8. /cc
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. Prevent issues from auto-closing with an comment. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
Stale issues rot after 30d of inactivity. Mark the issue as fresh with . Rotten issues close after an additional 30d of inactivity. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle rotten /remove-lifecycle stale
The CVM tests were turned off a while ago.
Change:
// The tests assume that Get/GetList/Watch calls shouldn't fail. // However, 429 error can now be returned if watchcache is under initialization. // To avoid rewriting all tests, we wait for watcache to initialize. <del> if err := cacher.ready.wait(context.Background()); err != nil { </del> <ins> if err := cacher.Wait(context.Background()); err != nil { </ins> return nil, storage.APIObjectVersioner{}, err } }

Discussion:
pull-kubernetes-unit TestStoreListResourceVersion in seems like July 7th 2024: ! Don't know, here is the log: No response /sig
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/sig api-machinery
See also
/sig etcd
/assign /cc
Bisected this to cc
I'm not sure if this is the proper way to fix it, but setting a limit in the ListOptions deflakes the test. The flakiness comes from the fact that the cache is sometimes not initialized in time which causes it to return a "too many requests". Perhaps in tests we could wait till the cache is ready in order to proceed?
Right, setting a limit when a resource version is set would delegate the list to etcd when cache is not initialized based on the KEP description. It is due to this type of request is not that costly to etcd. Looks like it is exactly how the wait cache ready is in See below So I think we should implement this same thing in when initializing the .
Agree - waiting for cacher to be initialized when creating it here is a better fix for it:
I thought about this as well, but waiting on is not possible outside of the cacher module, since it is a private field. I'm thinking to either add an option to cacherstorage.Config and make wait or add a new public method to . Do you guys have a better idea? Thanks.
Exposing Wait(context) method sounds fine
I have pushed the changes we discussed. Thank you all for your guidance.
Change:
pp.timeoutPermit = false pp.waitAndRejectPermit = false pp.waitAndAllowPermit = false <ins> pp.allowPermit = false </ins> } // NewPermitPlugin is the factory for permit plugin.

Discussion:
<!-- Please only use this template for submitting enhancement requests --What would you like to be : When preempting a pod that is a that permit plugins are waiting for , we need send a message(actually send a ) to the , otherwise permit plugins might wait until time out, and do not know the pod has been preempted. Why is this needed: /sig scheduling /kind cleanup /assign
cc
Interesting interaction between the permit and preempt, I will take a look at the PR you sent, thanks!
Change:
original[k] = patchV continue } <ins> // If the data type is a pointer, resolve the element. if t.Kind() == reflect.Ptr { t = t.Elem() } </ins> // If they're both maps or lists, recurse into the value. // First find the fieldPatchStrategy and fieldPatchMergeKey. fieldType, fieldPatchStrategy, fieldPatchMergeKey, err := forkedjson.LookupPatchMetadata(t, k)

Discussion:
PATCH http://IP:8080/api/v1beta3/namespaces/dev--ea6f-49cb-b5bb-/replicationcontrollers/redis-master {"spec":{"replicas":0}} answer: { "kind": "Status", "apiVersion": "v1beta3", "metadata": {}, "status": "Failure", "message": "the server responded with the status code 415 but did not return more information", "details": {}, "code": 415 } Before v0.15.0 it had worked well.
That error is returned by go-restful because it did not find any valid route with the same content type as specified in the request: Can you try setting the content-type in your request as is described here: cc
well, I've the same issue which is not fixed by setting the value, at least with the "strategic merge patch". Here's the request: Here's the response body: And here's the log trace of the request in the apiserver pod:
Come from a typo, purposed PR here:
Btw, I'm still having strange issues with sub-objects. Note that it appears only is the sub-object is already set. Here's the request: Here's the API log:
Looks like a bug in strategic merge -- it doesn't deal with pointers correctly:
cc
is it possible to have a milestone for this so I can tell if I should merger ? Thanks.
Change:
"time" autoscalingv2 "k8s.io/api/autoscaling/v2beta2" <del> "k8s.io/api/core/v1" </del> <ins> v1 "k8s.io/api/core/v1" </ins> "k8s.io/apimachinery/pkg/api/meta/testrestmapper" "k8s.io/apimachinery/pkg/api/resource" metav1 "k8s.io/apimachinery/pkg/apis/meta/v1"

Discussion:
<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --What happened: When the HPA scales up beyond the capacity of the cluster, many unscheduled pods are created. When load is removed, the usage ratio drops and the HPA tries to scale down. The unscheduled pods do not produce metrics, so the HPA fills in the 100% usage and re-evaluates the usage ratio. Because of the many fake 100% metrics, the new usage ratio is above the target value and the direction of scale is reversed. In this case, the HPA decides not to make any change to the current scale. What you expected to happen: I would expect the HPA to not get stuck when load is removed. How to reproduce it (as minimally and precisely as possible): and put it under load in a cluster without cluster autoscaling. the Deployment to increase the CPU request beyond the capability of the cluster (e.g. 500m). This should produce many Pending pods. load. The Pending pods will keep the HPA scaled indefinitely. Anything else we need to know?: Environment: Kubernetes version (use ): Client Version: version.Info{Major:"1", Minor:"14", GitVersion:"v1.14.3", GitCommit:"", GitTreeState:"clean", BuildDate:"2019-06-06T01:44:30Z", GoVersion:"go1.12.5", Compiler:"gc", Platform:"linux/amd64"} Server Version: version.Info{Major:"1", Minor:"12+", GitVersion:"v1.12.9-gke.3", GitCommit:"", GitTreeState:"clean", BuildDate:"2019-06-06T21:30:53Z", GoVersion:"go1.10.8b4", Compiler:"gc", Platform:"linux/amd64"} Cloud provider or hardware configuration: OS (e.g: ): Kernel (e.g. ): Install tools: Network plugin and version (if this is a network-related bug): Others:
/sig autoscaling
This might Previous I suspected that this was the same issues as (HPA positive feedback loop).

However it turns out to be different, that Pending pods cause HPA to get stuck.
Original reason for using 100% utilization for missing metrics when scaling down: