Repository: kubernetes-sigs/kwok
Pull request #899 (2023-12-29 09:05:28)

Fix the logic of the node leases
<!--  Thanks for sending a pull request!  Here are some tips for you:

1. If this is your first time, please read our contributor guidelines: https://git.k8s.io/community/contributors/guide/first-contribution.md#your-first-contribution and developer guide https://git.k8s.io/community/contributors/devel/development.md#development-guide
2. Please label this pull request according to what type of issue you are addressing, especially if this is a release targeted pull request. For reference on required PR/issue labels, read here:
https://git.k8s.io/community/contributors/devel/sig-release/release.md#issuepr-kind-label
3. Ensure you have added or ran the appropriate tests for your PR: https://git.k8s.io/community/contributors/devel/sig-testing/testing.md
4. If you want *faster* PR reviews, read how: https://git.k8s.io/community/contributors/guide/pull-requests.md#best-practices-for-faster-reviews
5. If the PR is unfinished, see how to mark it: https://git.k8s.io/community/contributors/guide/pull-requests.md#marking-unfinished-pull-requests
-->

#### What type of PR is this?

<!--
Add one of the following kinds:
/kind bug
/kind cleanup
/kind documentation
/kind feature

Optionally add one or more of the following kinds if applicable:
/kind api-change
/kind deprecation
/kind failing-test
/kind flake
/kind regression
-->

/kind bug

#### What this PR does / why we need it:

#### Which issue(s) this PR fixes:
<!--
*Automatically closes linked issue when PR is merged.
Usage: `Fixes #<issue number>`, or `Fixes (paste link of issue)`.
_If PR is about `failing-tests or flakes`, please post the related issues/tests in a comment and do not use `Fixes`_*
-->
xref #886


#### Special notes for your reviewer:

#### Does this PR introduce a user-facing change?
<!--
If no, just write "NONE" in the release-note block below.
If yes, a release note is required:
Enter your extended release note in the block below. If the PR requires additional action from users switching to the new release, include the string "action required".

For more information on release notes see: https://git.k8s.io/community/contributors/guide/release-notes.md
-->
```release-note
Fix the logic of the node leases
```

#### Additional documentation e.g., KEPs (Kubernetes Enhancement Proposals), usage docs, etc.:

<!--
This section can be blank if this pull request does not require a release note.

When adding links which point to resources within git repositories, like
KEPs or supporting documentation, please reference a specific commit and avoid
linking directly to the master branch. This ensures that links reference a
specific point in time, rather than a document that may change over time.

See here for guidance on getting permanent links to files: https://help.github.com/en/articles/getting-permanent-links-to-files

Please use the following format for linking documentation:
- [KEP]: <link>
- [Usage]: <link>
- [Other doc]: <link>
-->
```docs

```


k8s-ci-robot: [APPROVALNOTIFIER] This PR is **APPROVED**

This pull-request has been approved by: *<a href="https://github.com/kubernetes-sigs/kwok/pull/899#" title="Author self-approved">wzshiming</a>*

The full list of commands accepted by this bot can be found [here](https://go.k8s.io/bot-commands?repo=kubernetes-sigs%2Fkwok).

The pull request process is described [here](https://git.k8s.io/community/contributors/guide/owners.md#the-code-review-process)

<details >
Needs approval from an approver in each of these files:

- ~~[OWNERS](https://github.com/kubernetes-sigs/kwok/blob/main/OWNERS)~~ [wzshiming]

Approvers can indicate their approval by writing `/approve` in a comment
Approvers can cancel approval by writing `/approve cancel` in a comment
</details>
<!-- META={"approvers":[]} -->
netlify[bot]: ### <span aria-hidden="true">👷</span> Deploy Preview for *k8s-kwok* processing.


|  Name | Link |
|:-:|------------------------|
|<span aria-hidden="true">🔨</span> Latest commit | b93a5572839d8213d2aedbba8dd602e2e3d92b45 |
|<span aria-hidden="true">🔍</span> Latest deploy log | https://app.netlify.com/sites/k8s-kwok/deploys/658e8bdb462ae70007f528a5 |
k8s-ci-robot: @wzshiming: The following test **failed**, say `/retest` to rerun all failed tests or `/retest-required` to rerun all mandatory failed tests:

Test name | Commit | Details | Required | Rerun command
--- | --- | --- | --- | ---
pull-kwok-unit-test-main | 924255a7fa641c5119a90fff77ea4d77969c3047 | [link](https://prow.k8s.io/view/gs/kubernetes-jenkins/pr-logs/pull/kubernetes-sigs_kwok/899/pull-kwok-unit-test-main/1740661095091146752) | true | `/test pull-kwok-unit-test-main`

[Full PR test history](https://prow.k8s.io/pr-history?org=kubernetes-sigs&repo=kwok&pr=899). [Your PR dashboard](https://prow.k8s.io/pr?query=is%3Apr%20state%3Aopen%20author%3Awzshiming). Please help us cut down on flakes by [linking to](https://git.k8s.io/community/contributors/devel/sig-testing/flaky-tests.md#filing-issues-for-flaky-tests) an [open issue](https://github.com/kubernetes-sigs/kwok/issues?q=is:issue+is:open) when you hit one in your PR.

<details>

Instructions for interacting with me using PR comments are available [here](https://git.k8s.io/community/contributors/guide/pull-requests.md).  If you have questions or suggestions related to my behavior, please file an issue against the [kubernetes/test-infra](https://github.com/kubernetes/test-infra/issues/new?title=Prow%20issue:) repository. I understand the commands that are listed [here](https://go.k8s.io/bot-commands).
</details>
<!-- test report -->
k8s-ci-robot: @wzshiming: The following test **failed**, say `/retest` to rerun all failed tests or `/retest-required` to rerun all mandatory failed tests:

Test name | Commit | Details | Required | Rerun command
--- | --- | --- | --- | ---
pull-kwok-unit-test-main | 924255a7fa641c5119a90fff77ea4d77969c3047 | [link](https://prow.k8s.io/view/gs/kubernetes-jenkins/pr-logs/pull/kubernetes-sigs_kwok/899/pull-kwok-unit-test-main/1740662800419655680) | true | `/test pull-kwok-unit-test-main`

[Full PR test history](https://prow.k8s.io/pr-history?org=kubernetes-sigs&repo=kwok&pr=899). [Your PR dashboard](https://prow.k8s.io/pr?query=is%3Apr%20state%3Aopen%20author%3Awzshiming). Please help us cut down on flakes by [linking to](https://git.k8s.io/community/contributors/devel/sig-testing/flaky-tests.md#filing-issues-for-flaky-tests) an [open issue](https://github.com/kubernetes-sigs/kwok/issues?q=is:issue+is:open) when you hit one in your PR.

<details>

Instructions for interacting with me using PR comments are available [here](https://git.k8s.io/community/contributors/guide/pull-requests.md).  If you have questions or suggestions related to my behavior, please file an issue against the [kubernetes/test-infra](https://github.com/kubernetes/test-infra/issues/new?title=Prow%20issue:) repository. I understand the commands that are listed [here](https://go.k8s.io/bot-commands).
</details>
<!-- test report -->
k8s-ci-robot: @wzshiming: The following test **failed**, say `/retest` to rerun all failed tests or `/retest-required` to rerun all mandatory failed tests:

Test name | Commit | Details | Required | Rerun command
--- | --- | --- | --- | ---
pull-kwok-e2e-test-main | 3c9fefee928a93261046b1397b3a1d82a522d418 | [link](https://prow.k8s.io/view/gs/kubernetes-jenkins/pr-logs/pull/kubernetes-sigs_kwok/899/pull-kwok-e2e-test-main/1740684935401836544) | true | `/test pull-kwok-e2e-test-main`

[Full PR test history](https://prow.k8s.io/pr-history?org=kubernetes-sigs&repo=kwok&pr=899). [Your PR dashboard](https://prow.k8s.io/pr?query=is%3Apr%20state%3Aopen%20author%3Awzshiming). Please help us cut down on flakes by [linking to](https://git.k8s.io/community/contributors/devel/sig-testing/flaky-tests.md#filing-issues-for-flaky-tests) an [open issue](https://github.com/kubernetes-sigs/kwok/issues?q=is:issue+is:open) when you hit one in your PR.

<details>

Instructions for interacting with me using PR comments are available [here](https://git.k8s.io/community/contributors/guide/pull-requests.md).  If you have questions or suggestions related to my behavior, please file an issue against the [kubernetes/test-infra](https://github.com/kubernetes/test-infra/issues/new?title=Prow%20issue:) repository. I understand the commands that are listed [here](https://go.k8s.io/bot-commands).
</details>
<!-- test report -->
k8s-ci-robot: @wzshiming: The following tests **failed**, say `/retest` to rerun all failed tests or `/retest-required` to rerun all mandatory failed tests:

Test name | Commit | Details | Required | Rerun command
--- | --- | --- | --- | ---
pull-kwok-e2e-test-main | 3c9fefee928a93261046b1397b3a1d82a522d418 | [link](https://prow.k8s.io/view/gs/kubernetes-jenkins/pr-logs/pull/kubernetes-sigs_kwok/899/pull-kwok-e2e-test-main/1740684935401836544) | true | `/test pull-kwok-e2e-test-main`
pull-kwok-verify-main | f804c542cf8e406bc6844a1611fa830a345ec92b | [link](https://prow.k8s.io/view/gs/kubernetes-jenkins/pr-logs/pull/kubernetes-sigs_kwok/899/pull-kwok-verify-main/1740794367930535936) | true | `/test pull-kwok-verify-main`

[Full PR test history](https://prow.k8s.io/pr-history?org=kubernetes-sigs&repo=kwok&pr=899). [Your PR dashboard](https://prow.k8s.io/pr?query=is%3Apr%20state%3Aopen%20author%3Awzshiming). Please help us cut down on flakes by [linking to](https://git.k8s.io/community/contributors/devel/sig-testing/flaky-tests.md#filing-issues-for-flaky-tests) an [open issue](https://github.com/kubernetes-sigs/kwok/issues?q=is:issue+is:open) when you hit one in your PR.

<details>

Instructions for interacting with me using PR comments are available [here](https://git.k8s.io/community/contributors/guide/pull-requests.md).  If you have questions or suggestions related to my behavior, please file an issue against the [kubernetes/test-infra](https://github.com/kubernetes/test-infra/issues/new?title=Prow%20issue:) repository. I understand the commands that are listed [here](https://go.k8s.io/bot-commands).
</details>
<!-- test report -->
k8s-ci-robot: @wzshiming: The following test **failed**, say `/retest` to rerun all failed tests or `/retest-required` to rerun all mandatory failed tests:

Test name | Commit | Details | Required | Rerun command
--- | --- | --- | --- | ---
pull-kwok-e2e-test-main | 545cc88ab5708888cdc60369f3966f0615e1010b | [link](https://prow.k8s.io/view/gs/kubernetes-jenkins/pr-logs/pull/kubernetes-sigs_kwok/899/pull-kwok-e2e-test-main/1741077680150810624) | true | `/test pull-kwok-e2e-test-main`

[Full PR test history](https://prow.k8s.io/pr-history?org=kubernetes-sigs&repo=kwok&pr=899). [Your PR dashboard](https://prow.k8s.io/pr?query=is%3Apr%20state%3Aopen%20author%3Awzshiming). Please help us cut down on flakes by [linking to](https://git.k8s.io/community/contributors/devel/sig-testing/flaky-tests.md#filing-issues-for-flaky-tests) an [open issue](https://github.com/kubernetes-sigs/kwok/issues?q=is:issue+is:open) when you hit one in your PR.

<details>

Instructions for interacting with me using PR comments are available [here](https://git.k8s.io/community/contributors/guide/pull-requests.md).  If you have questions or suggestions related to my behavior, please file an issue against the [kubernetes/test-infra](https://github.com/kubernetes/test-infra/issues/new?title=Prow%20issue:) repository. I understand the commands that are listed [here](https://go.k8s.io/bot-commands).
</details>
<!-- test report -->
k8s-ci-robot: @wzshiming: The following test **failed**, say `/retest` to rerun all failed tests or `/retest-required` to rerun all mandatory failed tests:

Test name | Commit | Details | Required | Rerun command
--- | --- | --- | --- | ---
pull-kwok-e2e-test-main | c8d59282642400a7865ac6f980c1ddf124b01f21 | [link](https://prow.k8s.io/view/gs/kubernetes-jenkins/pr-logs/pull/kubernetes-sigs_kwok/899/pull-kwok-e2e-test-main/1741087670286684160) | true | `/test pull-kwok-e2e-test-main`

[Full PR test history](https://prow.k8s.io/pr-history?org=kubernetes-sigs&repo=kwok&pr=899). [Your PR dashboard](https://prow.k8s.io/pr?query=is%3Apr%20state%3Aopen%20author%3Awzshiming). Please help us cut down on flakes by [linking to](https://git.k8s.io/community/contributors/devel/sig-testing/flaky-tests.md#filing-issues-for-flaky-tests) an [open issue](https://github.com/kubernetes-sigs/kwok/issues?q=is:issue+is:open) when you hit one in your PR.

<details>

Instructions for interacting with me using PR comments are available [here](https://git.k8s.io/community/contributors/guide/pull-requests.md).  If you have questions or suggestions related to my behavior, please file an issue against the [kubernetes/test-infra](https://github.com/kubernetes/test-infra/issues/new?title=Prow%20issue:) repository. I understand the commands that are listed [here](https://go.k8s.io/bot-commands).
</details>
<!-- test report -->
k8s-ci-robot: @wzshiming: The following test **failed**, say `/retest` to rerun all failed tests or `/retest-required` to rerun all mandatory failed tests:

Test name | Commit | Details | Required | Rerun command
--- | --- | --- | --- | ---
pull-kwok-unit-test-main | c4962b45b8b1c26ccbafe1866d4a93bc1016eed2 | [link](https://prow.k8s.io/view/gs/kubernetes-jenkins/pr-logs/pull/kubernetes-sigs_kwok/899/pull-kwok-unit-test-main/1742000910927663104) | true | `/test pull-kwok-unit-test-main`

[Full PR test history](https://prow.k8s.io/pr-history?org=kubernetes-sigs&repo=kwok&pr=899). [Your PR dashboard](https://prow.k8s.io/pr?query=is%3Apr%20state%3Aopen%20author%3Awzshiming). Please help us cut down on flakes by [linking to](https://git.k8s.io/community/contributors/devel/sig-testing/flaky-tests.md#filing-issues-for-flaky-tests) an [open issue](https://github.com/kubernetes-sigs/kwok/issues?q=is:issue+is:open) when you hit one in your PR.

<details>

Instructions for interacting with me using PR comments are available [here](https://git.k8s.io/community/contributors/guide/pull-requests.md).  If you have questions or suggestions related to my behavior, please file an issue against the [kubernetes/test-infra](https://github.com/kubernetes/test-infra/issues/new?title=Prow%20issue:) repository. I understand the commands that are listed [here](https://go.k8s.io/bot-commands).
</details>
<!-- test report -->
k8s-ci-robot: @wzshiming: The following tests **failed**, say `/retest` to rerun all failed tests or `/retest-required` to rerun all mandatory failed tests:

Test name | Commit | Details | Required | Rerun command
--- | --- | --- | --- | ---
pull-kwok-unit-test-main | c4962b45b8b1c26ccbafe1866d4a93bc1016eed2 | [link](https://prow.k8s.io/view/gs/kubernetes-jenkins/pr-logs/pull/kubernetes-sigs_kwok/899/pull-kwok-unit-test-main/1742000910927663104) | true | `/test pull-kwok-unit-test-main`
pull-kwok-verify-main | c4962b45b8b1c26ccbafe1866d4a93bc1016eed2 | [link](https://prow.k8s.io/view/gs/kubernetes-jenkins/pr-logs/pull/kubernetes-sigs_kwok/899/pull-kwok-verify-main/1742000910818611200) | true | `/test pull-kwok-verify-main`

[Full PR test history](https://prow.k8s.io/pr-history?org=kubernetes-sigs&repo=kwok&pr=899). [Your PR dashboard](https://prow.k8s.io/pr?query=is%3Apr%20state%3Aopen%20author%3Awzshiming). Please help us cut down on flakes by [linking to](https://git.k8s.io/community/contributors/devel/sig-testing/flaky-tests.md#filing-issues-for-flaky-tests) an [open issue](https://github.com/kubernetes-sigs/kwok/issues?q=is:issue+is:open) when you hit one in your PR.

<details>

Instructions for interacting with me using PR comments are available [here](https://git.k8s.io/community/contributors/guide/pull-requests.md).  If you have questions or suggestions related to my behavior, please file an issue against the [kubernetes/test-infra](https://github.com/kubernetes/test-infra/issues/new?title=Prow%20issue:) repository. I understand the commands that are listed [here](https://go.k8s.io/bot-commands).
</details>
<!-- test report -->
k8s-ci-robot: @wzshiming: The following test **failed**, say `/retest` to rerun all failed tests or `/retest-required` to rerun all mandatory failed tests:

Test name | Commit | Details | Required | Rerun command
--- | --- | --- | --- | ---
pull-kwok-e2e-test-main | 1bb5079422ada27257bed1579b46c28f627a36d1 | [link](https://prow.k8s.io/view/gs/kubernetes-jenkins/pr-logs/pull/kubernetes-sigs_kwok/899/pull-kwok-e2e-test-main/1742019175741657088) | true | `/test pull-kwok-e2e-test-main`

[Full PR test history](https://prow.k8s.io/pr-history?org=kubernetes-sigs&repo=kwok&pr=899). [Your PR dashboard](https://prow.k8s.io/pr?query=is%3Apr%20state%3Aopen%20author%3Awzshiming). Please help us cut down on flakes by [linking to](https://git.k8s.io/community/contributors/devel/sig-testing/flaky-tests.md#filing-issues-for-flaky-tests) an [open issue](https://github.com/kubernetes-sigs/kwok/issues?q=is:issue+is:open) when you hit one in your PR.

<details>

Instructions for interacting with me using PR comments are available [here](https://git.k8s.io/community/contributors/guide/pull-requests.md).  If you have questions or suggestions related to my behavior, please file an issue against the [kubernetes/test-infra](https://github.com/kubernetes/test-infra/issues/new?title=Prow%20issue:) repository. I understand the commands that are listed [here](https://go.k8s.io/bot-commands).
</details>
<!-- test report -->

Patch:
diff --git a/pkg/kwok/controllers/controller.go b/pkg/kwok/controllers/controller.go
index 76b58d77..7f983893 100644
--- a/pkg/kwok/controllers/controller.go
+++ b/pkg/kwok/controllers/controller.go
@@ -288,7 +288,6 @@ func (c *Controller) initNodeLeaseController(ctx context.Context) error {
 	c.nodeLeases, err = NewNodeLeaseController(NodeLeaseControllerConfig{
 		Clock:                c.conf.Clock,
 		TypedClient:          c.conf.TypedClient,
-		NodeCacheGetter:      c.nodeCacheGetter,
 		LeaseDurationSeconds: c.conf.NodeLeaseDurationSeconds,
 		LeaseParallelism:     c.conf.NodeLeaseParallelism,
 		RenewInterval:        renewInterval,
diff --git a/pkg/kwok/controllers/node_lease_controller.go b/pkg/kwok/controllers/node_lease_controller.go
index 008f5092..3084a46d 100644
--- a/pkg/kwok/controllers/node_lease_controller.go
+++ b/pkg/kwok/controllers/node_lease_controller.go
@@ -39,7 +39,6 @@ import (
 // NodeLeaseController is responsible for creating and renewing a lease object
 type NodeLeaseController struct {
 	typedClient          clientset.Interface
-	nodeCacheGetter      informer.Getter[*corev1.Node]
 	leaseDurationSeconds uint
 	leaseParallelism     uint
 	renewInterval        time.Duration
@@ -63,7 +62,6 @@ type NodeLeaseControllerConfig struct {
 	Clock                clock.Clock
 	HolderIdentity       string
 	TypedClient          clientset.Interface
-	NodeCacheGetter      informer.Getter[*corev1.Node]
 	LeaseDurationSeconds uint
 	LeaseParallelism     uint
 	RenewInterval        time.Duration
@@ -85,7 +83,6 @@ func NewNodeLeaseController(conf NodeLeaseControllerConfig) (*NodeLeaseControlle
 	c := &NodeLeaseController{
 		clock:                conf.Clock,
 		typedClient:          conf.TypedClient,
-		nodeCacheGetter:      conf.NodeCacheGetter,
 		leaseDurationSeconds: conf.LeaseDurationSeconds,
 		leaseParallelism:     conf.LeaseParallelism,
 		renewInterval:        conf.RenewInterval,
@@ -134,35 +131,39 @@ loop:
 }
 
 func (c *NodeLeaseController) syncWorker(ctx context.Context) {
+	logger := log.FromContext(ctx)
 	for ctx.Err() == nil {
 		nodeName := c.delayQueue.GetOrWait()
-		if c.nodeCacheGetter != nil {
-			_, ok := c.nodeCacheGetter.Get(nodeName)
-			if !ok {
-				continue
-			}
-		}
 
 		now := c.clock.Now()
-		c.sync(ctx, nodeName)
-		nextTime := c.nextTryTime(nodeName, now)
-		dur := nextTime.Sub(now)
-		if dur > 0 {
-			_ = c.delayQueue.AddAfter(nodeName, dur)
+		hold := c.sync(ctx, nodeName)
+
+		lease, _ := c.latestLease.Load(nodeName)
+
+		expireTime, ok := expireTime(lease)
+		if !ok {
+			if hold {
+				// If the lease is not found, we should retry immediately.
+				c.delayQueue.Add(nodeName)
+			} else {
+				logger.Warn("Lease not held",
+					"node", nodeName,
+				)
+			}
+			continue
 		}
+
+		renewIntervalDuration := c.interval()
+		expireDuration := expireTime.Sub(now)
+
+		nextTry := nextTryDuration(renewIntervalDuration, expireDuration, hold)
+
+		_ = c.delayQueue.AddAfter(nodeName, nextTry)
 	}
 }
 
-func (c *NodeLeaseController) nextTryTime(name string, now time.Time) time.Time {
-	next := now.Add(wait.Jitter(c.renewInterval, c.renewIntervalJitter))
-	lease, ok := c.latestLease.Load(name)
-	if !ok || lease == nil ||
-		lease.Spec.HolderIdentity == nil ||
-		lease.Spec.LeaseDurationSeconds == nil ||
-		lease.Spec.RenewTime == nil {
-		return next
-	}
-	return nextTryTime(lease, c.holderIdentity, next)
+func (c *NodeLeaseController) interval() time.Duration {
+	return wait.Jitter(c.renewInterval, c.renewIntervalJitter)
 }
 
 // TryHold tries to hold a lease for the NodeLeaseController
@@ -187,7 +188,7 @@ func (c *NodeLeaseController) Held(name string) bool {
 }
 
 // sync syncs a lease for a node
-func (c *NodeLeaseController) sync(ctx context.Context, nodeName string) {
+func (c *NodeLeaseController) sync(ctx context.Context, nodeName string) bool {
 	logger := log.FromContext(ctx)
 	logger = logger.With("node", nodeName)
 
@@ -195,13 +196,13 @@ func (c *NodeLeaseController) sync(ctx context.Context, nodeName string) {
 	if ok && latestLease != nil {
 		if !tryAcquireOrRenew(latestLease, c.holderIdentity, c.clock.Now()) {
 			logger.Debug("Lease already acquired by another holder")
-			return
+			return false
 		}
 		logger.Info("Syncing lease")
 		lease, transitions, err := c.renewLease(ctx, latestLease)
 		if err != nil {
 			logger.Error("failed to update lease using latest lease", err)
-			return
+			return false
 		}
 		c.latestLease.Store(nodeName, lease)
 		if transitions {
@@ -226,7 +227,7 @@ func (c *NodeLeaseController) sync(ctx context.Context, nodeName string) {
 				_, err = c.syncLease(ctx, nodeName)
 				if err != nil {
 					logger.Error("failed to sync lease", err)
-					return
+					return false
 				}
 				if c.onNodeManagedFunc != nil {
 					if c.Held(nodeName) {
@@ -235,12 +236,12 @@ func (c *NodeLeaseController) sync(ctx context.Context, nodeName string) {
 						logger.Warn("Lease not held")
 					}
 				}
-				return
+				return true
 			}
 
 			if !apierrors.IsNotFound(err) || !c.latestLease.IsEmpty() {
 				logger.Error("failed to create lease", err)
-				return
+				return false
 			}
 
 			// kube-apiserver will not have finished initializing the resources when the cluster has just been created.
@@ -249,7 +250,7 @@ func (c *NodeLeaseController) sync(ctx context.Context, nodeName string) {
 			latestLease, err = c.ensureLease(ctx, nodeName)
 			if err != nil {
 				logger.Error("failed to create lease secondly", err)
-				return
+				return false
 			}
 		}
 
@@ -262,6 +263,7 @@ func (c *NodeLeaseController) sync(ctx context.Context, nodeName string) {
 			}
 		}
 	}
+	return true
 }
 
 func (c *NodeLeaseController) syncLease(ctx context.Context, leaseName string) (*coordinationv1.Lease, error) {
@@ -357,14 +359,38 @@ func tryAcquireOrRenew(lease *coordinationv1.Lease, holderIdentity string, now t
 	return expireTime.Before(now)
 }
 
-// nextTryTime returns the next time to try to acquire or renew the lease
-func nextTryTime(lease *coordinationv1.Lease, holderIdentity string, next time.Time) time.Time {
+// expireTime returns the expire time of a lease
+func expireTime(lease *coordinationv1.Lease) (time.Time, bool) {
+	if lease == nil ||
+		lease.Spec.HolderIdentity == nil ||
+		lease.Spec.LeaseDurationSeconds == nil ||
+		lease.Spec.RenewTime == nil {
+		return time.Time{}, false
+	}
+
 	expireTime := lease.Spec.RenewTime.Add(time.Duration(*lease.Spec.LeaseDurationSeconds) * time.Second)
-	if *lease.Spec.HolderIdentity == holderIdentity {
-		if next.Before(expireTime) {
-			return next
+	return expireTime, true
+}
+
+// nextTryDuration returns the next to try to acquire or renew the lease
+func nextTryDuration(renewInterval, expire time.Duration, hold bool) time.Duration {
+	switch {
+	case expire <= 0:
+		// If the lease is expired, we should retry immediately.
+		return 0
+	case !hold:
+		// If the lease is not held, we should retry at the expire time.
+		return expire
+	case hold && renewInterval >= expire-time.Second:
+		// If the lease is held and the renew interval is greater than the expire time,
+		// we should retry earlier than the expire time.
+		expire -= time.Second
+		if expire < 0 {
+			expire = 0
 		}
+		return expire
+	default:
+		// Otherwise, we should retry at the renew interval.
+		return renewInterval
 	}
-
-	return expireTime
 }


Test patch:
diff --git a/pkg/kwok/controllers/node_lease_controller_test.go b/pkg/kwok/controllers/node_lease_controller_test.go
index 3d597d04..4d800381 100644
--- a/pkg/kwok/controllers/node_lease_controller_test.go
+++ b/pkg/kwok/controllers/node_lease_controller_test.go
@@ -218,68 +218,97 @@ func Test_tryAcquireOrRenew(t *testing.T) {
 	}
 }
 
-func Test_nextTryTime(t *testing.T) {
-	now := time.Now()
-	type args struct {
-		lease          *coordinationv1.Lease
-		holderIdentity string
-		next           time.Time
-	}
+func TestNextTryDuration(t *testing.T) {
 	tests := []struct {
-		name string
-		args args
-		want time.Time
+		name           string
+		renewInterval  time.Duration
+		expire         time.Duration
+		hold           bool
+		expectedResult time.Duration
 	}{
 		{
-			name: "holder self and not expired",
-			args: args{
-				lease: &coordinationv1.Lease{
-					Spec: coordinationv1.LeaseSpec{
-						HolderIdentity:       format.Ptr("test"),
-						LeaseDurationSeconds: format.Ptr(int32(40)),
-						RenewTime:            format.Ptr(metav1.NewMicroTime(now)),
-					},
-				},
-				holderIdentity: "test",
-				next:           now.Add(39 * time.Second),
-			},
-			want: now.Add(39 * time.Second),
+			name:           "Lease expired",
+			renewInterval:  10 * time.Second,
+			expire:         -1 * time.Second,
+			hold:           true,
+			expectedResult: 0,
 		},
 		{
-			name: "holder self and expired",
-			args: args{
-				lease: &coordinationv1.Lease{
-					Spec: coordinationv1.LeaseSpec{
-						HolderIdentity:       format.Ptr("test"),
-						LeaseDurationSeconds: format.Ptr(int32(40)),
-						RenewTime:            format.Ptr(metav1.NewMicroTime(now)),
-					},
-				},
-				holderIdentity: "test",
-				next:           now.Add(41 * time.Second),
-			},
-			want: now.Add(40 * time.Second),
+			name:           "Lease not held",
+			renewInterval:  10 * time.Second,
+			expire:         5 * time.Second,
+			hold:           false,
+			expectedResult: 5 * time.Second,
 		},
 		{
-			name: "holder not self and not expired",
-			args: args{
-				lease: &coordinationv1.Lease{
-					Spec: coordinationv1.LeaseSpec{
-						HolderIdentity:       format.Ptr("test"),
-						LeaseDurationSeconds: format.Ptr(int32(40)),
-						RenewTime:            format.Ptr(metav1.NewMicroTime(now)),
-					},
+			name:           "Lease held, renew interval greater than expire time",
+			renewInterval:  10 * time.Second,
+			expire:         5 * time.Second,
+			hold:           true,
+			expectedResult: 4 * time.Second,
+		},
+		{
+			name:           "Lease held, renew interval less than expire time",
+			renewInterval:  5 * time.Second,
+			expire:         10 * time.Second,
+			hold:           true,
+			expectedResult: 5 * time.Second,
+		},
+		{
+			name:           "Lease held, renew interval equals expire time",
+			renewInterval:  5 * time.Second,
+			expire:         5 * time.Second,
+			hold:           true,
+			expectedResult: 4 * time.Second,
+		},
+	}
+
+	for _, tt := range tests {
+		t.Run(tt.name, func(t *testing.T) {
+			result := nextTryDuration(tt.renewInterval, tt.expire, tt.hold)
+			if result != tt.expectedResult {
+				t.Errorf("nextTryDuration() = %v, want %v", result, tt.expectedResult)
+			}
+		})
+	}
+}
+
+func TestExpireTime(t *testing.T) {
+	now := time.Now()
+	tests := []struct {
+		name     string
+		lease    *coordinationv1.Lease
+		wantTime time.Time
+		wantBool bool
+	}{
+		{
+			name:     "Lease with missing fields",
+			lease:    &coordinationv1.Lease{},
+			wantTime: time.Time{},
+			wantBool: false,
+		},
+		{
+			name: "Lease with all fields",
+			lease: &coordinationv1.Lease{
+				Spec: coordinationv1.LeaseSpec{
+					HolderIdentity:       format.Ptr("test"),
+					LeaseDurationSeconds: format.Ptr(int32(10)),
+					RenewTime:            format.Ptr(metav1.NewMicroTime(now)),
 				},
-				holderIdentity: "test-new",
-				next:           now.Add(39 * time.Second),
 			},
-			want: now.Add(40 * time.Second),
+			wantTime: now.Add(10 * time.Second),
+			wantBool: true,
 		},
 	}
+
 	for _, tt := range tests {
 		t.Run(tt.name, func(t *testing.T) {
-			if got := nextTryTime(tt.args.lease, tt.args.holderIdentity, tt.args.next); !reflect.DeepEqual(got, tt.want) {
-				t.Errorf("nextTryTime() = %v, want %v", got, tt.want)
+			gotTime, gotBool := expireTime(tt.lease)
+			if !reflect.DeepEqual(gotTime, tt.wantTime) {
+				t.Errorf("expireTime() gotTime = %v, want %v", gotTime, tt.wantTime)
+			}
+			if gotBool != tt.wantBool {
+				t.Errorf("expireTime() gotBool = %v, want %v", gotBool, tt.wantBool)
 			}
 		})
 	}