Change:
Verify Master - Master Blocking verify.openapi-spec Inconsistently since 2 November. First failure (shown in testgrid) API Server is not becoming ready during Relevant logs /sig-testing
cc for cidr controller
and may be related.
/sig auth network
/remove-sig auth Is almost certainly related
/assign it seems the servicecidr controller takes time to start and it blocks the apiserver readiness
/reopen Test run failed after related PR merger:
Reopened this issue. <detailsIn response to : /reopen Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
yeah /assign
/triage accepted
this is sig network can you pelase update the title?
/retitle [Flaking Test][sig-network] ci-kubernetes-verify-master verify-openapi-spec
This still flake in master blocking test grid.
use 60s timeout and here uses 30s timeout by default. It flakes. 10 lines of log cannot show other details. My local test with 15s timeout failed several times. 30s is enough for my local tests. Or we just enlarge the timeout for CI.
having both scripts the same timeout sounds reasonable I'm still worried why it keeps timing out, but for that we should first define a test that measure the apiserver startup and then decide what value is correct and enforce it.
/priority important-soon /milestone v1.30 backport to v1.29 may be needed.
Hello! Release Signal shadow here. This issue has not been updated for a long time, so I'd like to check what's the status. The code freeze is starting (less than a month from now), and while there is still plenty of time, we want to ensure that each PR has a chance to be merged on time. As this issue is tagged for 1.30, is it still planned for this release?
This flakes in Do we have some action items for v1.30?
I need to fix it, is a shame on me, sorry
Hello! Release Signal shadow here. This issue has not been updated for a long time, so we'd like to check the status. The code freeze is starting less than a week, and we want to ensure that each PR has a chance to be merged on time. As this issue is tagged for 1.30, is it still planned for this release? Please let us know the status if possible.
yeah, I like to get it fixed, there is some race condition that makes some of the boostrap to deadlock , I still didn't find why
didn't forget but didn't have time to figure out the root cause, will try to get back to this end of the week
I got one failure locally of an apiserver that took much time The apiservers need the servicecidr controller to start to be healthy, and the controllers need to sync the informer, but checking one of the requests from the informer I can see this error logging error output: "k8sx00nfnx02v1x12x06Statusx12x96x01nx06nx00x12x00x1ax00x12aFailurex1a2Timeout: Too large resource version: 2, current: 1"aTimeoutCnx00x12x00x1ax00"7nx17ResourceVersionTooLargex12x1aToo large resource versionx1ax00(x012x000xf8x03x1ax00"x00" and this keep going for a while logging error output: "k8sx00nfnx02v1x12x06Statusx12x98x01nx06nx00x12x00x1ax00x12aFailurex1a4Timeout: Too large resource version: 203, current: 1"aTimeoutCnx00x12x00x1ax00"7nx17ResourceVersionTooLargex12x1aToo large resource versionx1ax00(x012x000xf8x03x1ax00"x00" do you know why the resource version is growing?
looks like the server returns that error in a few places, not sure which one is getting exercised: update-openapi-spec starts with AllAlpha=true in order to enable all features / APIs ... are there any alpha features around watch / resourceVersion that could cause this?
Didn't read carefully the issue, so just quickly sharing some thoughts: 1) streaming lists might potentially be related, but it needs to be explicitly requested by the client. Reflector is the only place that can currently do that, but it requires explicitly set env-var: Is it possible that this value is set? if so, then streaming lists will result is calling etcd for current RV and will wait for watchache to be synced with it - if progress-notify-interval is large and manual progress notify is disabled, we could face things like that ^^ 2) that said, if Alpha features are enabled, then manual-progress notify should also be set: 3) is it possible that we're facing the bug that Marek found 2 days before release that effectively was the main reason we didn't try to promote consistnet read to beta? ^^
Based on the log from I would guess this is the 3002ms implies the 3 second timeout of waiting for revision Empty implies consistent list from cache implies non-recursive list Planned to fix it, but PR was blocked on
thanks a ton for digging into that to deflake this test, I'd set this in the update-openapi- file (since it doesn't impact the generated openapi) with a comment to remove once is resolved

Discussion:
c.eventBroadcaster = broadcaster c.eventRecorder = recorder <del> c.readyCh = make(chan struct{}) </del> return c }

serviceCIDRLister   networkingv1alpha1listers.ServiceCIDRLister serviceCIDRsSynced  cache.InformerSynced <del> readyCh chan struct{} // channel to block until the default ServiceCIDR exists </del> interval time.Duration }

return } <del> go wait.Until(c.sync, c.interval, stopCh) </del> <ins> // derive a context from the stopCh so we can cancel the poll loop ctx := wait.ContextForChannel(stopCh) // wait until first successfully sync // this blocks apiserver startup so poll with a short interval err := wait.PollUntilContextCancel(ctx, 100*time.Millisecond, true, func(ctx context.Context) (bool, error) { syncErr := c.sync() return syncErr == nil, nil }) if err != nil { klog.Infof("error initializing the default ServiceCIDR: %v", err) </ins> <del> select { case <-stopCh: case <-c.readyCh: </del> } <ins> // run the sync loop in the background with the defined interval go wait.Until(func() { err := c.sync() if err != nil { klog.Infof("error trying to sync the default ServiceCIDR: %v", err) } }, c.interval, stopCh) </ins> } <del> func (c *Controller) sync() { </del> <ins> func (c *Controller) sync() error { </ins> // check if the default ServiceCIDR already exist serviceCIDR, err := c.serviceCIDRLister.Get(DefaultServiceCIDRName) // if exists if err == nil { <del> c.setReady() </del> c.syncStatus(serviceCIDR) <del> return </del> <ins> return nil </ins> } // unknown error if !apierrors.IsNotFound(err) { <del> klog.Infof("error trying to obtain the default ServiceCIDR: %v", err) return </del> <ins> return err </ins> } // default ServiceCIDR does not exist

} serviceCIDR, err = c.client.NetworkingV1alpha1().ServiceCIDRs().Create(context.Background(), serviceCIDR, metav1.CreateOptions{}) if err != nil && !apierrors.IsAlreadyExists(err) { <del> klog.Infof("error creating default ServiceCIDR: %v", err) </del> c.eventRecorder.Eventf(serviceCIDR, v1.EventTypeWarning, "KubernetesDefaultServiceCIDRError", "The default ServiceCIDR can not be created") <del> return </del> <ins> return err </ins> } <del> c.setReady() </del> c.syncStatus(serviceCIDR) <del> } func (c *Controller) setReady() { select { case <-c.readyCh: default: close(c.readyCh) } </del> <ins> return nil </ins> } func (c *Controller) syncStatus(serviceCIDR *networkingapiv1alpha1.ServiceCIDR) {

eventRecorder:      record.NewFakeRecorder(100), serviceCIDRLister:  serviceCIDRInformer.Lister(), serviceCIDRsSynced: func() bool { return true }, <del> readyCh:            make(chan struct{}), </del> } return client, c