jiebi/CodeConvo huggingface dataset:
Change:
return nil } <ins> // Wait blocks until the cacher is Ready or Stopped, it returns an error if Stopped. func (c *Cacher) Wait(ctx context.Context) error { return c.ready.wait(ctx) } </ins> // errWatcher implements watch.Interface to return a single error type errWatcher struct { result chan watch.Event

Discussion:
pull-kubernetes-unit TestStoreListResourceVersion in seems like July 7th 2024: ! Don't know, here is the log: No response /sig
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/sig api-machinery
See also
/sig etcd
/assign /cc
Bisected this to cc
I'm not sure if this is the proper way to fix it, but setting a limit in the ListOptions deflakes the test. The flakiness comes from the fact that the cache is sometimes not initialized in time which causes it to return a "too many requests". Perhaps in tests we could wait till the cache is ready in order to proceed?
Right, setting a limit when a resource version is set would delegate the list to etcd when cache is not initialized based on the KEP description. It is due to this type of request is not that costly to etcd. Looks like it is exactly how the wait cache ready is in See below So I think we should implement this same thing in when initializing the .
Agree - waiting for cacher to be initialized when creating it here is a better fix for it:
I thought about this as well, but waiting on is not possible outside of the cacher module, since it is a private field. I'm thinking to either add an option to cacherstorage.Config and make wait or add a new public method to . Do you guys have a better idea? Thanks.
Exposing Wait(context) method sounds fine
I have pushed the changes we discussed. Thank you all for your guidance.
Change:
// Put one job and one pod into the store testJob := newJob(2, 2) manager.jobStore.Store.Add(testJob) <del> received := make(chan string) </del> <ins> received := make(chan struct{}) </ins> // The pod update sent through the fakeWatcher should figure out the managing job and // send it into the syncHandler. manager.syncHandler = func(key string) error {

Discussion:
Ran a test executable with more than 20,000 times and no failures. Any hints as to how to get it to show up? I'm running 100 jobs at a time, so my system is stressed already.
I to am unable to reproduce this. I get similar errors if I reduce the timeout at the top of the file. I spent a bit looking at how to rewrite the tests without the timeout, but nothing jumped out at me. Looking at it though, it's really just a 500ms timeout for the test to complete. I'm just going to remove them. Go has a default timeout of 10 minutes. If it's going to pass, it will pass then. If not, Go will give you a nice traceback where it deadlocked.
Change:
existingContainers, _ := kl.ListContainers() log.Printf("Existing:n%#v Desired: %#v", existingContainers, desired) for _, container := range existingContainers { <del> // This is slightly hacky, but we ignore containers that lack '--' in their name // to allow users to manually spin up their own containers if they want. if !strings.Contains(container, "--") { </del> <ins> // Skip containers that we didn't create to allow users to manually // spin up their own containers if they want. if !strings.HasPrefix(container, "/"+containerNamePrefix+"--") { </ins> continue } if !desired[container] {

Discussion:
release-master-blocking Conformance-GCE-master-kubetest2Kubernetes e2e suite.[It] [sig-network] Services should be able to change the type from ExternalName to ClusterIP [Conformance] ci-kubernetes-gce-conformance-latest-kubetest2.Overall kubetest2.Test 07-11 15:51 EEST No response /sig cloud-provider cc
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
I remember some timeout has changed with ginkgo2 and something moved from 24hours to 1 hours, or something like that, is it possible that this is the problem here, in this job?
The Grinko timeout change could also be the root problem with this job:
hmm, we have to revisit the flags
hmm, this seems correct
that was a good observations, seems that job is failing because gathering the logs takes too much time !
there is no timeout set in the kubetest2 command here (not sure if this would be the place to set a timeout)
it seems kubetest has to be migrated to ginkgov2 it should use this as basis I'm on vacation this week, can somebody take over and open the issue and patch kubetest2?
looks like this file needs to get updated: add line 72:
oh, you already commented this :D
I will have a look later
Related: umbrella issue:
last testgrid run passed :)
Change:
import ( "fmt" <ins> "strings" </ins> "testing" "github.com/Azure/azure-sdk-for-go/services/compute/mgmt/2019-12-01/compute"

Discussion:
<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --What happened: Azure cloud provider gets the availability zone from IMDS directly and the zone is usually in lower cases in most regions. But there're chances this is not true in some regions, hence customer's workloads depend on availability zone may not work. What you expected to happen: The availability zone is always in lower cases. How to reproduce it (as minimally and precisely as possible): Anything else we need to know?: Environment: Kubernetes version (use ): Cloud provider or hardware configuration: OS (e.g: ): Kernel (e.g. ): Install tools: Network plugin and version (if this is a network-related bug): Others: /kind bug /sig cloud-provider /area provider/azure
/assign
Fixed by .
Change:
ret=1 fi <del> if ! _out="$(diff -Naupr -x "BUILD" -x "zz_generated.openapi.go" -x "AUTHORS*" -x "CONTRIBUTORS*" vendor "${_kubetmp}/vendor")"; then </del> <ins> if ! _out="$(diff -Naupr -x "BUILD" -x "AUTHORS*" -x "CONTRIBUTORS*" vendor "${_kubetmp}/vendor")"; then </ins> echo "Your vendored results are different:" >&2 echo "${_out}" >&2 echo "Godeps Verify failed." >&2

Discussion:
<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --What happened: I'm trying to vendor , which due to (recently ?) code in depends on The problem now is that the content of is incomplete: apiextensions- calls generatedopenapi.GetOpenAPIDefinitions, but apiextensions-apiserver/pkg/generated/openapi/ doesn't contain the generated code. What you expected to happen: I expect to compile when fetching it with "go get" or vendoring it with tools like "dep". How to reproduce it (as minimally and precisely as possible): Environment: Kubernetes version (use ): master
/assign
/sig release
/assign
sent a fix in
Change:
"watch":    true, } <ins> // Constant for the retry-after interval on rate limiting. // TODO: maybe make this dynamic? or user-adjustable? const RetryAfter = "1" </ins> // IsReadOnlyReq() is true for any (or at least many) request which has no observable // side effects on state of apiserver (though there may be internal side effects like // caching and logging).

Discussion:
Related with issue , the API server should have a cap on the number of outstanding operations/requests it's handling, and return an error otherwise. A simple semaphore is an idea for this.
+1 !
The good news/bad news on this is that there is a rate limiter in the API server (), but the defaults for it seem too high: 10 qps average, 200 burst (). We could start by lowering the burst level to something smaller, like 30. It may be interesting to also differentiate between GET and PUT/POST requests.
I think that we need limits on the number of outstanding requests, not the number of QPS.
-An max outstanding would be useful. But it may force us to have separate buckets for different types of requests, i.e. a small number of long running requests causing API server to reject short requests. -Heard that you might be looking into adding support for flow control (or something like it?). We may need it soon to help avoid bugs like runaway node status updates (see issue ).
I had not - the things we were looking at were the bastion changes to apiserver which will break logs and proxying into their own rest resources so they can be granularly controlled. Being able to timebox and classify requests differently would be good. ----- Original Message -----
- Ah, np, I probably misheard.
What are our plans for handling this client side? Do we want clients to handle "you're being throttled" error themselves, or do we plan to handle it in the library? I'm asking because currently kubelet does not have a 'short' retry loop, hence it tries to report NodeStatus every X (2) seconds, whether it succeed or not. Neither we have a sidechannel for heartbeat messages. This would mean, that if API server will be under heavy load, and with not-negligible probability NodeStatus reporting will be throttled, we may incorrectly mark a Node as unreachable. Currently it's not a huge problem, because X << Y, where Y is the time we wait before we decide a Node is unreachable. But because of the effort somewhere close to it'll probably get worse.
The client library should have options for retry on client.Config which include the ability to disable the behavior. Automatic retry is probably an "opt-out" behavior.

----- Original Message -----
says there are people working on rate limiting client connections to API server.
I have in the past adjusted the rate limiter to support the creation of large clusters. I also the logic. If you adjust the rate limiter settings, please make sure it is still possible to create large clusters.
I'm closing this as fixed, given the original subject. If we want to expand client-side support for retry, please open a dedicated issue. (Also note that QPS based throttling was to the client recently too)
Change:
url.Host += ":80" } } <del> headers, err := headersForConfig(config) </del> <ins> headers, err := headersForConfig(config, url) </ins> if err != nil { return nil, fmt.Errorf("failed to load http headers: %v", err) }

Discussion:
<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks!--What happened: When I did e2e conformance test with log level 5, it completely passed! When I did e2e conformance test with log level 8, it just failed. The only thing I did is just leveling up from log level 5 to 8. But, I've got this error message. What you expected to happen: test would be completely passed and just get more verbose log Environment: local cluster <!-- DO NOT EDIT BELOW THIS LINE --/kind bug
/sig testing
Nice report. I confirmed this issue happens on my local environment also like:
Change:
flag.StringVar(&TestContext.UpgradeImage, "upgrade-image", "", "Image to upgrade to (e.g. 'container_vm' or 'gci') if doing an upgrade test.") flag.StringVar(&TestContext.GCEUpgradeScript, "gce-upgrade-script", "", "Script to use to upgrade a GCE cluster.") flag.StringVar(&TestContext.FederationUpgradeTarget, "federation-upgrade-target", "ci/latest", "Version to upgrade to (e.g. 'release/stable', 'release/latest', 'ci/latest', '0.19.1', '0.19.1-669-gabac8c8') if doing an federation upgrade test.") <del> flag.StringVar(&TestContext.PrometheusPushGateway, "prom-push-gateway", "", "The URL to prometheus gateway, so that metrics can be pushed during e2es and scraped by prometheus. Typically something like 127.0.0.1:9091.") </del> flag.BoolVar(&TestContext.CleanStart, "clean-start", false, "If true, purge all namespaces except default and system before running tests. This serves to Cleanup test namespaces from failed/interrupted e2e runs in a long-lived cluster.") flag.BoolVar(&TestContext.GarbageCollectorEnabled, "garbage-collector-enabled", true, "Set to true if the garbage collector is enabled in the kube-apiserver and kube-controller-manager, then some tests will rely on the garbage collector to delete dependent resources.") }

Discussion:
TLDR: removed a feature but left documentation and the flag intact. Kubernetes version (use ): Client Version: version.Info{Major:"1", Minor:"6", GitVersion:"v1.6.0", GitCommit:"", GitTreeState:"clean", BuildDate:"2017-03-28T16:36:33Z", GoVersion:"go1.7.5", Compiler:"gc", Platform:"linux/amd64"} Server Version: version.Info{Major:"1", Minor:"5", GitVersion:"v1.5.2", GitCommit:"", GitTreeState:"clean", BuildDate:"2017-01-12T04:52:34Z", GoVersion:"go1.7.4", Compiler:"gc", Platform:"linux/amd64"} Environment: Cloud provider or hardware configuration: irrelevant OS (e.g. from /etc/os-release): irrelevant Kernel (e.g. ): irrelevant Install tools: kops Others: What happened: I am trying to set up periodic conformance tests on our cluster to make sure it is working as expected. Prometheus will be used for monitoring. After ing kubernetes I executed a command along the lines of: GINKGOPARALLEL=y KUBERNETESPROVIDER=local KUBEMASTER=local --host="" --test.v=true --"[(SlowSerial)]" --"[Conformance]" --prom-push-gateway=prometheus-push- The push gateway does never receive the metrics, because the feature was removed. What you expected to happen: The tests send the metrics to the push gateway. How to reproduce it (as minimally and precisely as possible): Search for the "PrometheusPushGateway" flag in this repo. It is used nowhere. Anything else we need to know: The feature was removed in but the flags were left in place and also in the documentation. It is frustrating to see that this flag is just a black hole. A solution to this would be to either reinstate the feature or remove the flag from docs and flags.
Blerg. pointed this out back in the PR that remove the functionality, let's follow through and remove the flags. Easy PR if someone wants it
Change:
ARTIFACTS=${WORKSPACE}/_artifacts mkdir -p ${ARTIFACTS} <ins> if [[ -f "${KUBEKINS_SERVICE_ACCOUNT_FILE:-}" ]]; then echo 'Activating service account...'  # No harm in doing this multiple times. gcloud auth activate-service-account --key-file="${KUBEKINS_SERVICE_ACCOUNT_FILE}" # https://developers.google.com/identity/protocols/application-default-credentials export GOOGLE_APPLICATION_CREDENTIALS="${KUBEKINS_SERVICE_ACCOUNT_FILE}" unset KUBEKINS_SERVICE_ACCOUNT_FILE elif [[ -n "${KUBEKINS_SERVICE_ACCOUNT_FILE:-}" ]]; then echo "ERROR: cannot access service account file at: ${KUBEKINS_SERVICE_ACCOUNT_FILE}" fi </ins> go run test/e2e_node/runner/run_e2e.go  --logtostderr --vmodule=*=4 --ssh-env="gce"  --zone="$GCE_ZONE" --project="$GCE_PROJECT" --hosts="$GCE_HOSTS"  --images="$GCE_IMAGES" --image-project="$GCE_IMAGE_PROJECT" 

Discussion:
I want to isolate these in their own project:
Right now just the CI jobs, but after we also need to do this for the PR jobs.
Change:
var _ = SIGDescribe("Hostname of Pod [Feature:SetHostnameAsFQDN][NodeFeature:SetHostnameAsFQDN]", func() { f := framework.NewDefaultFramework("hostfqdn") <del> dnsDomain := "cluster.local" if cdn := framework.TestContext.ClusterDNSDomain; cdn != "" { dnsDomain = cdn } </del> /* Release: v1.19 Testname: Create Pod without fully qualified domain name (FQDN)

Discussion:
<!-- Please only use this template for submitting reports about continuously failing tests or jobs in Kubernetes CI --Which jobs are failing: / Which test(s) are failing: - Since when has it been failing: Since was merged Testgrid link: Reason for failure: Example: Looks like the at L122 and L144 doesn't actually get printed? Anything else we need to know: /sig node /cc /priority important-soon
Hi sorry about that, I will work on it asap. Are you aware of any difference between the CI that executes NodeFeature and NodeAlphaFeature? FYI this other PR merged at the same time
/assign javidiaz
Somehow, the /etc/hosts of the pod does not have subdomain # Kubernetes-managed hosts file. 127.0.0.1 localhost ::1 localhost ip6-localhost ip6-loopback fe00::0 ip6-localnet fe00::0 ip6-mcastprefix fe00::1 ip6-allnodes fe00::2 ip6-allrouters 172.17.0.2 hostfqdn-.t.hostfqdn- hostfqdn- I think the issue is that framework.TestContext.ClusterDNSDomain is empty String.
Hi any idea of why cluster-subdomain is empty String in e2e node tests? These tests are failing because I made the wrong assumption that cluster-subdomain should be by default. So I set as the expected subdomain when framework.TestContext.ClusterDNSDomain is empty.
/triage accepted Thank you for looking into this, and for opening the issue :pray: Given i think you found the issue. node e2e tests have theirwn entrypoint so i guess we forgot to completely account for all the settings the e2e framework sets and to use those to configure the kubelet. I think in general we need to refactor the node test suite to prevent this sort of issue. Small note: full diff for the timeframe around which these tests began to fail (betwen 11:53 EST and 12:54 EST on 11-05).
Change:
// Build ipvs kernel routes for each node ip address nodeIPs := make([]net.IP, 0) <del> addresses, err := utilproxy.GetNodeAddresses(proxier.nodePortAddresses, proxier.networkInterfacer) if err != nil { glog.Errorf("Failed to get node ip address matching nodeport cidr") continue } </del> for address := range addresses { if !utilproxy.IsZeroCIDR(address) { nodeIPs = append(nodeIPs, net.ParseIP(address))

Discussion:
Is this a BUG REPORT or FEATURE REQUEST?: /kind bug What happened: kube-proxy was still binding to ::{port} for a node port service after it was started with the --nodeport-addresses argument What you expected to happen: kube-proxy to only bind to interfaces in the CIDR ranges specified by --nodeport-addresses. How to reproduce it (as minimally and precisely as possible): Start kube-proxy with --nodeport-addresses "127.0.0.1,::1" Create a service of type NodePort Observe kube-proxy bind to all listening interfaces Anything else we need to know?: In the proposal for the --nodeport-addresses feature, it specifies being able to have other services answering or listening on a given port. Since kube-proxy binds to all interfaces, this is still not possible so it appears one of the initial intents of the proposal was net met with the existing solution. [] Doesn't set an interface. Given that nodeport-addresses is specified as CIDR ranges extra work must be done to determine which IP's on a host exist that could be bound to. Alternatively, a flag such as --nodeport-addresses-bind could allow an operator to set an explicit value. To test functionality I was able to hardcode an address here and observe the desired behavior could be achieved. Environment: Kubernetes version:Cloud provider or hardware configuration: AWS OS (e.g. from /etc/os-release): Kernel (e.g. ): Install tools: Custom fabricated Others: As expected, this behaves the same way regardless of iptables or ipvs being used as the forwarding mechanism.
/sig network
/assign
Nice catch! I will fix it ASAP~
Please check
Change:
ctx, cancel := context.WithCancel(context.Background()) defer cancel() err = PatchPodStatus(ctx, client, &tc.pod, &tc.statusToUpdate) <del> if err != nil { </del> <ins> if err != nil && tc.validateErr == nil { // shouldn't be error </ins> t.Fatal(err) } <ins> if tc.validateErr != nil { if !tc.validateErr(err) { t.Fatalf("Returned unexpected error: %v", err) } return } </ins> retrievedPod, err := client.CoreV1().Pods(tc.pod.Namespace).Get(ctx, tc.pod.Name, metav1.GetOptions{}) if err != nil {

Discussion:
If there is a connection failure when updating the Pod status, we don't retry. Furthermore, due to , we won't have another chance until much later. The problem is that if we don't mark the Pod as Unschedulable, other controllers (such as cluster-autoscaler) wouldn't react to these pods. We should have stronger guarantees to mark a Pod as Unschedulable. You need an unschedulable Pod (for example, a Pod with a non matching node affinity) and a flaky connection to apiserver. No response master This is particularly problematic in 1.24, due to . Any <details</details<details</details<details</details<details</details
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/sig scheduling
That is bad, retrying in place on transient errors should probably be ok.
I'll address it. /assign
Thanks!
Change:
const nodeWithImpairedVolumes = "NodeWithImpairedVolumes" const ( <ins> // These constants help to identify if a node is a master or a minion labelKeyNodeRole    = "kubernetes.io/role" nodeMasterRole      = "master" nodeMinionRole      = "node" labelKeyNodeMaster  = "node-role.kubernetes.io/master" labelKeyNodeCompute = "node-role.kubernetes.io/compute" labelKeyNodeMinion  = "node-role.kubernetes.io/node" ) const ( </ins> // volumeAttachmentConsecutiveErrorLimit is the number of consecutive errors we will ignore when waiting for a volume to attach/detach volumeAttachmentStatusConsecutiveErrorLimit = 10 // most attach/detach operations on AWS finish within 1-4 seconds

Discussion:
<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --What happened: We have pretty large clusters in aws and in production, we found an inefficiency in provisioning dynamic EBS volumes, where aws-ebs volume provisioner tries to look for available zone from aws directly. With scale, this can trigger massive from ec2, and affect cluster stability. This is also related to What you expected to happen: To simply get zones with available instances, we can get information from API server for efficiency. There might be a delay between node gets shutting down and such change is reflected in api server, but in case node is not ready or gets terminated, there will be higher level reconciliation triggered so we are safe. How to reproduce it (as minimally and precisely as possible): We only expect this to be hit when the cluster is huge (or aws account is huge) so not easy to reproduce. /assign /sig aws /sig cloud-provider
/assign