Repository: etcd-io/etcd

Issue:
Flaky integration test case `TestV3AuthWithLeaseRevokeWithRootJWT`
### Which github workflows are flaking?

[test (linux-amd64-integration-1-cpu)](https://github.com/etcd-io/etcd/actions/runs/5976797722/job/16215421678?pr=16472#logs)

### Which tests are flaking?

TestV3AuthWithLeaseRevokeWithRootJWT

### Github Action link

https://github.com/etcd-io/etcd/actions/runs/5976797722/job/16215421678?pr=16472

### Reason for failure (if possible)

```
v3_auth_test.go:190: lease 61048a2d0509a20b should have been revoked, got cluster_id:3559229121123123792 member_id:564804756584587524 revision:2 raft_term:2 
```

### Anything else we need to know?

_No response_

Hints from issue comments:
@YaoC: I think the reason for the failure is that the JWT has expired when server sending `LeaseRevoke` request.

https://github.com/etcd-io/etcd/blob/27633923e45bca9e6699c9d68df5790840370728/server/etcdserver/server.go#L875-L886

The log shows:
```txt
logger.go:130: 2023-08-25T14:06:59.695Z	DEBUG	client	retrying of unary invoker	{"target": "etcd-endpoints://0xc000302780/localhost:m0", "method": "/etcdserverpb.Lease/LeaseGrant", "attempt": 0}
logger.go:130: 2023-08-25T14:06:59.702Z	DEBUG	client	retrying of unary invoker	{"target": "etcd-endpoints://0xc000302780/localhost:m0", "method": "/etcdserverpb.KV/Put", "attempt": 0}
logger.go:130: 2023-08-25T14:07:02.013Z	WARN	m0	failed to parse a JWT token	{"member": "m0", "error": "Token is expired"}
logger.go:130: 2023-08-25T14:07:02.013Z	WARN	m0	invalid auth token	{"member": "m0", "token": "***"}
logger.go:130: 2023-08-25T14:07:02.013Z	WARN	m0	failed to revoke lease	{"member": "m0", "lease-id": "61048a2d0509a20b", "error": "auth: invalid auth token"}
```
Since the lease ttl is 2, It would expired at **2023-08-25T14:07:01.695Z**. And I believe the server called `s.authStore.WithRoot` before **2023-08-25T14:07:02.000Z**, which will result in the expiration of JWT being **2023-08-25T14:07:02.000Z**.  JWT ttl is 1s and the precision of its expiration is in seconds.

https://github.com/etcd-io/etcd/blob/27633923e45bca9e6699c9d68df5790840370728/server/auth/jwt.go#L101-L106

I have run this test on my devbox and usually it only takes less than 10ms to send `LeaseRevoke` request. Perhaps the CPU throttling caused this method to take longer, and it just happened to jump to the next second when checking for expiration. Maybe we can change JWT ttl to 2s, so that it won't expire immediately.@fuweid: @YaoC would you please to send patch to fix this? Thanks!@YaoC: > @YaoC would you please to send patch to fix this? Thanks!

Sure, I'll fix it.@tjungblu: same happened on the 3.5 branch PR here: 
https://github.com/etcd-io/etcd/actions/runs/8895381443/job/24425596792?pr=17914

```
2024-04-30T13:18:16.0246105Z     logger.go:130: 2024-04-30T13:14:37.006Z	WARN	m0	failed to parse a JWT token	{"member": "m0", "token": "***", "error": "Token is expired"}
2024-04-30T13:18:16.0255248Z     logger.go:130: 2024-04-30T13:14:37.006Z	WARN	m0	invalid auth token	{"member": "m0", "token": "***"}
2024-04-30T13:18:16.0258064Z     logger.go:130: 2024-04-30T13:14:37.006Z	WARN	m0	failed to revoke lease	{"member": "m0", "lease-id": "4ebb8f2f24887a0b", "error": "auth: invalid auth token"}
2024-04-30T13:18:16.0266679Z     logger.go:130: 2024-04-30T13:14:37.613Z	WARN	m0	failed to parse a JWT token	{"member": "m0", "token": "***", "error": "Token is expired"}
2024-04-30T13:18:16.0274791Z     logger.go:130: 2024-04-30T13:14:37.613Z	WARN	m0	invalid auth token	{"member": "m0", "token": "***"}
```

@YaoC do you want to backport your PR from #16523?@stale: This issue has been automatically marked as stale because it has not had recent activity. It will be closed after 21 days if no further activity occurs. Thank you for your contributions.

Solution patch:
diff --git a/Dockerfile-release.amd64 b/Dockerfile-release.amd64
index bcae3ea2b..6ac8c877e 100644
--- a/Dockerfile-release.amd64
+++ b/Dockerfile-release.amd64
@@ -1,4 +1,4 @@
-FROM --platform=linux/amd64 gcr.io/distroless/static-debian11@sha256:046b92c933032a8ca99a66f4c79a68ac029d9a4ababd1a806a82140b3b899fd3
+FROM --platform=linux/amd64 gcr.io/distroless/static-debian11@sha256:6d31326376a7834b106f281b04f67b5d015c31732f594930f2ea81365f99d60c
 
 ADD etcd /usr/local/bin/
 ADD etcdctl /usr/local/bin/
diff --git a/Dockerfile-release.arm64 b/Dockerfile-release.arm64
index 4eef98282..f8048498e 100644
--- a/Dockerfile-release.arm64
+++ b/Dockerfile-release.arm64
@@ -1,4 +1,4 @@
-FROM --platform=linux/arm64 gcr.io/distroless/static-debian11@sha256:046b92c933032a8ca99a66f4c79a68ac029d9a4ababd1a806a82140b3b899fd3
+FROM --platform=linux/arm64 gcr.io/distroless/static-debian11@sha256:6d31326376a7834b106f281b04f67b5d015c31732f594930f2ea81365f99d60c
 
 ADD etcd /usr/local/bin/
 ADD etcdctl /usr/local/bin/
diff --git a/Dockerfile-release.ppc64le b/Dockerfile-release.ppc64le
index fee36a38d..6c0f53fb6 100644
--- a/Dockerfile-release.ppc64le
+++ b/Dockerfile-release.ppc64le
@@ -1,4 +1,4 @@
-FROM --platform=linux/ppc64le gcr.io/distroless/static-debian11@sha256:046b92c933032a8ca99a66f4c79a68ac029d9a4ababd1a806a82140b3b899fd3
+FROM --platform=linux/ppc64le gcr.io/distroless/static-debian11@sha256:6d31326376a7834b106f281b04f67b5d015c31732f594930f2ea81365f99d60c
 
 ADD etcd /usr/local/bin/
 ADD etcdctl /usr/local/bin/
diff --git a/Dockerfile-release.s390x b/Dockerfile-release.s390x
index 9ccdffa29..0c8336c99 100644
--- a/Dockerfile-release.s390x
+++ b/Dockerfile-release.s390x
@@ -1,4 +1,4 @@
-FROM --platform=linux/s390x gcr.io/distroless/static-debian11@sha256:046b92c933032a8ca99a66f4c79a68ac029d9a4ababd1a806a82140b3b899fd3
+FROM --platform=linux/s390x gcr.io/distroless/static-debian11@sha256:6d31326376a7834b106f281b04f67b5d015c31732f594930f2ea81365f99d60c
 
 ADD etcd /usr/local/bin/
 ADD etcdctl /usr/local/bin/
diff --git a/etcdctl/go.mod b/etcdctl/go.mod
index 8a3620207..8db919673 100644
--- a/etcdctl/go.mod
+++ b/etcdctl/go.mod
@@ -47,7 +47,7 @@ require (
 	github.com/russross/blackfriday/v2 v2.0.1 // indirect
 	github.com/shurcooL/sanitized_anchor_name v1.0.0 // indirect
 	github.com/xiang90/probing v0.0.0-20190116061207-43a291ad63a2 // indirect
-	go.etcd.io/bbolt v1.3.9 // indirect
+	go.etcd.io/bbolt v1.3.10 // indirect
 	go.etcd.io/etcd/raft/v3 v3.5.13 // indirect
 	go.etcd.io/etcd/server/v3 v3.5.13 // indirect
 	go.opentelemetry.io/contrib/instrumentation/google.golang.org/grpc/otelgrpc v0.46.0 // indirect
diff --git a/etcdctl/go.sum b/etcdctl/go.sum
index 5c78acab0..309808810 100644
--- a/etcdctl/go.sum
+++ b/etcdctl/go.sum
@@ -274,8 +274,8 @@ github.com/xiang90/probing v0.0.0-20190116061207-43a291ad63a2/go.mod h1:UETIi67q
 github.com/yuin/goldmark v1.1.27/go.mod h1:3hX8gzYuyVAZsxl0MRgGTJEmQBFcNTphYh9decYSb74=
 github.com/yuin/goldmark v1.2.1/go.mod h1:3hX8gzYuyVAZsxl0MRgGTJEmQBFcNTphYh9decYSb74=
 go.etcd.io/bbolt v1.3.2/go.mod h1:IbVyRI1SCnLcuJnV2u8VeU0CEYM7e686BmAb1XKL+uU=
-go.etcd.io/bbolt v1.3.9 h1:8x7aARPEXiXbHmtUwAIv7eV2fQFHrLLavdiJ3uzJXoI=
-go.etcd.io/bbolt v1.3.9/go.mod h1:zaO32+Ti0PK1ivdPtgMESzuzL2VPoIG1PCQNvOdo/dE=
+go.etcd.io/bbolt v1.3.10 h1:+BqfJTcCzTItrop8mq/lbzL8wSGtj94UO/3U31shqG0=
+go.etcd.io/bbolt v1.3.10/go.mod h1:bK3UQLPJZly7IlNmV7uVHJDxfe5aK9Ll93e/74Y9oEQ=
 go.opencensus.io v0.21.0/go.mod h1:mSImk1erAIZhrmZN+AvHh14ztQfjbGwt4TtuofqLduU=
 go.opencensus.io v0.22.0/go.mod h1:+kGneAE2xo2IficOXnaByMWTGM9T73dGwxeWcUqIpI8=
 go.opentelemetry.io/contrib/instrumentation/google.golang.org/grpc/otelgrpc v0.46.0 h1:PzIubN4/sjByhDRHLviCjJuweBXWFZWhghjg7cS28+M=
diff --git a/etcdutl/go.mod b/etcdutl/go.mod
index 6b0318e77..e1c68a3ac 100644
--- a/etcdutl/go.mod
+++ b/etcdutl/go.mod
@@ -24,7 +24,7 @@ require (
 	github.com/dustin/go-humanize v1.0.0
 	github.com/olekukonko/tablewriter v0.0.5
 	github.com/spf13/cobra v1.1.3
-	go.etcd.io/bbolt v1.3.9
+	go.etcd.io/bbolt v1.3.10
 	go.etcd.io/etcd/api/v3 v3.5.13
 	go.etcd.io/etcd/client/pkg/v3 v3.5.13
 	go.etcd.io/etcd/client/v3 v3.5.13
diff --git a/etcdutl/go.sum b/etcdutl/go.sum
index 6047de820..b33297c47 100644
--- a/etcdutl/go.sum
+++ b/etcdutl/go.sum
@@ -264,8 +264,8 @@ github.com/xiang90/probing v0.0.0-20190116061207-43a291ad63a2/go.mod h1:UETIi67q
 github.com/yuin/goldmark v1.1.27/go.mod h1:3hX8gzYuyVAZsxl0MRgGTJEmQBFcNTphYh9decYSb74=
 github.com/yuin/goldmark v1.2.1/go.mod h1:3hX8gzYuyVAZsxl0MRgGTJEmQBFcNTphYh9decYSb74=
 go.etcd.io/bbolt v1.3.2/go.mod h1:IbVyRI1SCnLcuJnV2u8VeU0CEYM7e686BmAb1XKL+uU=
-go.etcd.io/bbolt v1.3.9 h1:8x7aARPEXiXbHmtUwAIv7eV2fQFHrLLavdiJ3uzJXoI=
-go.etcd.io/bbolt v1.3.9/go.mod h1:zaO32+Ti0PK1ivdPtgMESzuzL2VPoIG1PCQNvOdo/dE=
+go.etcd.io/bbolt v1.3.10 h1:+BqfJTcCzTItrop8mq/lbzL8wSGtj94UO/3U31shqG0=
+go.etcd.io/bbolt v1.3.10/go.mod h1:bK3UQLPJZly7IlNmV7uVHJDxfe5aK9Ll93e/74Y9oEQ=
 go.opencensus.io v0.21.0/go.mod h1:mSImk1erAIZhrmZN+AvHh14ztQfjbGwt4TtuofqLduU=
 go.opencensus.io v0.22.0/go.mod h1:+kGneAE2xo2IficOXnaByMWTGM9T73dGwxeWcUqIpI8=
 go.opentelemetry.io/contrib/instrumentation/google.golang.org/grpc/otelgrpc v0.46.0 h1:PzIubN4/sjByhDRHLviCjJuweBXWFZWhghjg7cS28+M=
diff --git a/go.mod b/go.mod
index 6f675db95..8f3d4f990 100644
--- a/go.mod
+++ b/go.mod
@@ -19,7 +19,7 @@ require (
 	github.com/bgentry/speakeasy v0.1.0
 	github.com/dustin/go-humanize v1.0.0
 	github.com/spf13/cobra v1.1.3
-	go.etcd.io/bbolt v1.3.9
+	go.etcd.io/bbolt v1.3.10
 	go.etcd.io/etcd/api/v3 v3.5.13
 	go.etcd.io/etcd/client/pkg/v3 v3.5.13
 	go.etcd.io/etcd/client/v2 v2.305.13
diff --git a/go.sum b/go.sum
index f5c4efc4f..9dec29397 100644
--- a/go.sum
+++ b/go.sum
@@ -308,8 +308,8 @@ github.com/xiang90/probing v0.0.0-20190116061207-43a291ad63a2/go.mod h1:UETIi67q
 github.com/yuin/goldmark v1.1.27/go.mod h1:3hX8gzYuyVAZsxl0MRgGTJEmQBFcNTphYh9decYSb74=
 github.com/yuin/goldmark v1.2.1/go.mod h1:3hX8gzYuyVAZsxl0MRgGTJEmQBFcNTphYh9decYSb74=
 go.etcd.io/bbolt v1.3.2/go.mod h1:IbVyRI1SCnLcuJnV2u8VeU0CEYM7e686BmAb1XKL+uU=
-go.etcd.io/bbolt v1.3.9 h1:8x7aARPEXiXbHmtUwAIv7eV2fQFHrLLavdiJ3uzJXoI=
-go.etcd.io/bbolt v1.3.9/go.mod h1:zaO32+Ti0PK1ivdPtgMESzuzL2VPoIG1PCQNvOdo/dE=
+go.etcd.io/bbolt v1.3.10 h1:+BqfJTcCzTItrop8mq/lbzL8wSGtj94UO/3U31shqG0=
+go.etcd.io/bbolt v1.3.10/go.mod h1:bK3UQLPJZly7IlNmV7uVHJDxfe5aK9Ll93e/74Y9oEQ=
 go.etcd.io/gofail v0.1.0 h1:XItAMIhOojXFQMgrxjnd2EIIHun/d5qL0Pf7FzVTkFg=
 go.etcd.io/gofail v0.1.0/go.mod h1:VZBCXYGZhHAinaBiiqYvuDynvahNsAyLFwB3kEHKz1M=
 go.opencensus.io v0.21.0/go.mod h1:mSImk1erAIZhrmZN+AvHh14ztQfjbGwt4TtuofqLduU=
diff --git a/server/config/config.go b/server/config/config.go
index 0dac25c41..4f1f21af7 100644
--- a/server/config/config.go
+++ b/server/config/config.go
@@ -187,6 +187,9 @@ type ServerConfig struct {
 	// a shared buffer in its readonly check operations.
 	ExperimentalTxnModeWriteWithSharedBuffer bool `json:"experimental-txn-mode-write-with-shared-buffer"`
 
+	// ExperimentalStopGRPCServiceOnDefrag enables etcd gRPC service to stop serving client requests on defragmentation.
+	ExperimentalStopGRPCServiceOnDefrag bool `json:"experimental-stop-grpc-service-on-defrag"`
+
 	// ExperimentalBootstrapDefragThresholdMegabytes is the minimum number of megabytes needed to be freed for etcd server to
 	// consider running defrag during bootstrap. Needs to be set to non-zero value to take effect.
 	ExperimentalBootstrapDefragThresholdMegabytes uint `json:"experimental-bootstrap-defrag-threshold-megabytes"`
diff --git a/server/embed/config.go b/server/embed/config.go
index 084796812..f5bd3db52 100644
--- a/server/embed/config.go
+++ b/server/embed/config.go
@@ -414,6 +414,9 @@ type Config struct {
 	// ExperimentalTxnModeWriteWithSharedBuffer enables write transaction to use a shared buffer in its readonly check operations.
 	ExperimentalTxnModeWriteWithSharedBuffer bool `json:"experimental-txn-mode-write-with-shared-buffer"`
 
+	// ExperimentalStopGRPCServiceOnDefrag enables etcd gRPC service to stop serving client requests on defragmentation.
+	ExperimentalStopGRPCServiceOnDefrag bool `json:"experimental-stop-grpc-service-on-defrag"`
+
 	// V2Deprecation describes phase of API & Storage V2 support
 	V2Deprecation config.V2DeprecationEnum `json:"v2-deprecation"`
 }
@@ -515,6 +518,7 @@ func NewConfig() *Config {
 		ExperimentalDowngradeCheckTime:           DefaultDowngradeCheckTime,
 		ExperimentalMemoryMlock:                  false,
 		ExperimentalTxnModeWriteWithSharedBuffer: true,
+		ExperimentalStopGRPCServiceOnDefrag:      false,
 
 		ExperimentalCompactHashCheckEnabled: false,
 		ExperimentalCompactHashCheckTime:    time.Minute,
diff --git a/server/embed/etcd.go b/server/embed/etcd.go
index 09a9dfb03..78d9bef34 100644
--- a/server/embed/etcd.go
+++ b/server/embed/etcd.go
@@ -222,6 +222,7 @@ func StartEtcd(inCfg *Config) (e *Etcd, err error) {
 		WarningApplyDuration:                     cfg.ExperimentalWarningApplyDuration,
 		ExperimentalMemoryMlock:                  cfg.ExperimentalMemoryMlock,
 		ExperimentalTxnModeWriteWithSharedBuffer: cfg.ExperimentalTxnModeWriteWithSharedBuffer,
+		ExperimentalStopGRPCServiceOnDefrag:      cfg.ExperimentalStopGRPCServiceOnDefrag,
 		ExperimentalBootstrapDefragThresholdMegabytes: cfg.ExperimentalBootstrapDefragThresholdMegabytes,
 		V2Deprecation: cfg.V2DeprecationEffective(),
 	}
diff --git a/server/etcdmain/config.go b/server/etcdmain/config.go
index 11364e5a8..a91c3997f 100644
--- a/server/etcdmain/config.go
+++ b/server/etcdmain/config.go
@@ -301,6 +301,7 @@ func newConfig() *config {
 	fs.DurationVar(&cfg.ec.ExperimentalWarningApplyDuration, "experimental-warning-apply-duration", cfg.ec.ExperimentalWarningApplyDuration, "Time duration after which a warning is generated if request takes more time.")
 	fs.BoolVar(&cfg.ec.ExperimentalMemoryMlock, "experimental-memory-mlock", cfg.ec.ExperimentalMemoryMlock, "Enable to enforce etcd pages (in particular bbolt) to stay in RAM.")
 	fs.BoolVar(&cfg.ec.ExperimentalTxnModeWriteWithSharedBuffer, "experimental-txn-mode-write-with-shared-buffer", true, "Enable the write transaction to use a shared buffer in its readonly check operations.")
+	fs.BoolVar(&cfg.ec.ExperimentalStopGRPCServiceOnDefrag, "experimental-stop-grpc-service-on-defrag", cfg.ec.ExperimentalStopGRPCServiceOnDefrag, "Enable etcd gRPC service to stop serving client requests on defragmentation.")
 	fs.UintVar(&cfg.ec.ExperimentalBootstrapDefragThresholdMegabytes, "experimental-bootstrap-defrag-threshold-megabytes", 0, "Enable the defrag during etcd server bootstrap on condition that it will free at least the provided threshold of disk space. Needs to be set to non-zero value to take effect.")
 	fs.Uint64Var(&cfg.ec.SnapshotCatchUpEntries, "experimental-snapshot-catchup-entries", cfg.ec.SnapshotCatchUpEntries, "(WARNING: Use this flag with caution!) Number of entries for a slow follower to catch up after compacting the raft storage entries.")
 
diff --git a/server/etcdmain/help.go b/server/etcdmain/help.go
index 510022661..8d44691f0 100644
--- a/server/etcdmain/help.go
+++ b/server/etcdmain/help.go
@@ -282,6 +282,8 @@ Experimental feature:
     Enable the write transaction to use a shared buffer in its readonly check operations.
   --experimental-bootstrap-defrag-threshold-megabytes
     Enable the defrag during etcd server bootstrap on condition that it will free at least the provided threshold of disk space. Needs to be set to non-zero value to take effect.
+  --experimental-stop-grpc-service-on-defrag
+    Enable etcd gRPC service to stop serving client requests on defragmentation.
 
 Unsafe feature:
   --force-new-cluster 'false'
diff --git a/server/etcdserver/api/v3client/v3client.go b/server/etcdserver/api/v3client/v3client.go
index 8342dc434..de0da81af 100644
--- a/server/etcdserver/api/v3client/v3client.go
+++ b/server/etcdserver/api/v3client/v3client.go
@@ -39,7 +39,7 @@ func New(s *etcdserver.EtcdServer) *clientv3.Client {
 	wc := adapter.WatchServerToWatchClient(v3rpc.NewWatchServer(s))
 	c.Watcher = &watchWrapper{clientv3.NewWatchFromWatchClient(wc, c)}
 
-	mc := adapter.MaintenanceServerToMaintenanceClient(v3rpc.NewMaintenanceServer(s))
+	mc := adapter.MaintenanceServerToMaintenanceClient(v3rpc.NewMaintenanceServer(s, nil))
 	c.Maintenance = clientv3.NewMaintenanceFromMaintenanceClient(mc, c)
 
 	clc := adapter.ClusterServerToClusterClient(v3rpc.NewClusterServer(s))
diff --git a/server/etcdserver/api/v3rpc/grpc.go b/server/etcdserver/api/v3rpc/grpc.go
index 349ebea40..6947903a3 100644
--- a/server/etcdserver/api/v3rpc/grpc.go
+++ b/server/etcdserver/api/v3rpc/grpc.go
@@ -76,13 +76,10 @@ func Server(s *etcdserver.EtcdServer, tls *tls.Config, interceptor grpc.UnarySer
 	pb.RegisterLeaseServer(grpcServer, NewQuotaLeaseServer(s))
 	pb.RegisterClusterServer(grpcServer, NewClusterServer(s))
 	pb.RegisterAuthServer(grpcServer, NewAuthServer(s))
-	pb.RegisterMaintenanceServer(grpcServer, NewMaintenanceServer(s))
 
-	// server should register all the services manually
-	// use empty service name for all etcd services' health status,
-	// see https://github.com/grpc/grpc/blob/master/doc/health-checking.md for more
 	hsrv := health.NewServer()
-	hsrv.SetServingStatus("", healthpb.HealthCheckResponse_SERVING)
+	healthNotifier := newHealthNotifier(hsrv, s)
+	pb.RegisterMaintenanceServer(grpcServer, NewMaintenanceServer(s, healthNotifier))
 	healthpb.RegisterHealthServer(grpcServer, hsrv)
 
 	// set zero values for metrics registered for this grpc server
diff --git a/server/etcdserver/api/v3rpc/health.go b/server/etcdserver/api/v3rpc/health.go
new file mode 100644
index 000000000..f58396d1c
--- /dev/null
+++ b/server/etcdserver/api/v3rpc/health.go
@@ -0,0 +1,77 @@
+// Copyright 2023 The etcd Authors
+//
+// Licensed under the Apache License, Version 2.0 (the "License");
+// you may not use this file except in compliance with the License.
+// You may obtain a copy of the License at
+//
+//     http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+
+package v3rpc
+
+import (
+	"go.etcd.io/etcd/server/v3/etcdserver"
+	"go.uber.org/zap"
+	"google.golang.org/grpc/health"
+	healthpb "google.golang.org/grpc/health/grpc_health_v1"
+)
+
+const (
+	allGRPCServices = ""
+)
+
+type notifier interface {
+	defragStarted()
+	defragFinished()
+}
+
+func newHealthNotifier(hs *health.Server, s *etcdserver.EtcdServer) notifier {
+	if hs == nil {
+		panic("unexpected nil gRPC health server")
+	}
+	hc := &healthNotifier{hs: hs, lg: s.Logger(), stopGRPCServiceOnDefrag: s.Cfg.ExperimentalStopGRPCServiceOnDefrag}
+	// set grpc health server as serving status blindly since
+	// the grpc server will serve iff s.ReadyNotify() is closed.
+	hc.startServe()
+	return hc
+}
+
+type healthNotifier struct {
+	hs *health.Server
+	lg *zap.Logger
+
+	stopGRPCServiceOnDefrag bool
+}
+
+func (hc *healthNotifier) defragStarted() {
+	if !hc.stopGRPCServiceOnDefrag {
+		return
+	}
+	hc.stopServe("defrag is active")
+}
+
+func (hc *healthNotifier) defragFinished() { hc.startServe() }
+
+func (hc *healthNotifier) startServe() {
+	hc.lg.Info(
+		"grpc service status changed",
+		zap.String("service", allGRPCServices),
+		zap.String("status", healthpb.HealthCheckResponse_SERVING.String()),
+	)
+	hc.hs.SetServingStatus(allGRPCServices, healthpb.HealthCheckResponse_SERVING)
+}
+
+func (hc *healthNotifier) stopServe(reason string) {
+	hc.lg.Warn(
+		"grpc service status changed",
+		zap.String("service", allGRPCServices),
+		zap.String("status", healthpb.HealthCheckResponse_NOT_SERVING.String()),
+		zap.String("reason", reason),
+	)
+	hc.hs.SetServingStatus(allGRPCServices, healthpb.HealthCheckResponse_NOT_SERVING)
+}
diff --git a/server/etcdserver/api/v3rpc/maintenance.go b/server/etcdserver/api/v3rpc/maintenance.go
index 42f8b0da2..7db17e1a0 100644
--- a/server/etcdserver/api/v3rpc/maintenance.go
+++ b/server/etcdserver/api/v3rpc/maintenance.go
@@ -76,10 +76,12 @@ type maintenanceServer struct {
 	hdr    header
 	cs     ClusterStatusGetter
 	d      Downgrader
+
+	healthNotifier notifier
 }
 
-func NewMaintenanceServer(s *etcdserver.EtcdServer) pb.MaintenanceServer {
-	srv := &maintenanceServer{lg: s.Cfg.Logger, rg: s, hasher: s.KV().HashStorage(), kg: s, bg: s, a: s, lt: s, hdr: newHeader(s), cs: s, d: s}
+func NewMaintenanceServer(s *etcdserver.EtcdServer, healthNotifier notifier) pb.MaintenanceServer {
+	srv := &maintenanceServer{lg: s.Cfg.Logger, rg: s, hasher: s.KV().HashStorage(), kg: s, bg: s, a: s, lt: s, hdr: newHeader(s), cs: s, d: s, healthNotifier: healthNotifier}
 	if srv.lg == nil {
 		srv.lg = zap.NewNop()
 	}
@@ -88,6 +90,8 @@ func NewMaintenanceServer(s *etcdserver.EtcdServer) pb.MaintenanceServer {
 
 func (ms *maintenanceServer) Defragment(ctx context.Context, sr *pb.DefragmentRequest) (*pb.DefragmentResponse, error) {
 	ms.lg.Info("starting defragment")
+	ms.healthNotifier.defragStarted()
+	defer ms.healthNotifier.defragFinished()
 	err := ms.bg.Backend().Defrag()
 	if err != nil {
 		ms.lg.Warn("failed to defragment", zap.Error(err))
diff --git a/server/go.mod b/server/go.mod
index f05e0d50c..4b2e16eb5 100644
--- a/server/go.mod
+++ b/server/go.mod
@@ -23,7 +23,7 @@ require (
 	github.com/stretchr/testify v1.8.4
 	github.com/tmc/grpc-websocket-proxy v0.0.0-20201229170055-e5319fda7802
 	github.com/xiang90/probing v0.0.0-20190116061207-43a291ad63a2
-	go.etcd.io/bbolt v1.3.9
+	go.etcd.io/bbolt v1.3.10
 	go.etcd.io/etcd/api/v3 v3.5.13
 	go.etcd.io/etcd/client/pkg/v3 v3.5.13
 	go.etcd.io/etcd/client/v2 v2.305.13
diff --git a/server/go.sum b/server/go.sum
index 56a6118b8..7164b26e2 100644
--- a/server/go.sum
+++ b/server/go.sum
@@ -298,8 +298,8 @@ github.com/xiang90/probing v0.0.0-20190116061207-43a291ad63a2/go.mod h1:UETIi67q
 github.com/yuin/goldmark v1.1.27/go.mod h1:3hX8gzYuyVAZsxl0MRgGTJEmQBFcNTphYh9decYSb74=
 github.com/yuin/goldmark v1.2.1/go.mod h1:3hX8gzYuyVAZsxl0MRgGTJEmQBFcNTphYh9decYSb74=
 go.etcd.io/bbolt v1.3.2/go.mod h1:IbVyRI1SCnLcuJnV2u8VeU0CEYM7e686BmAb1XKL+uU=
-go.etcd.io/bbolt v1.3.9 h1:8x7aARPEXiXbHmtUwAIv7eV2fQFHrLLavdiJ3uzJXoI=
-go.etcd.io/bbolt v1.3.9/go.mod h1:zaO32+Ti0PK1ivdPtgMESzuzL2VPoIG1PCQNvOdo/dE=
+go.etcd.io/bbolt v1.3.10 h1:+BqfJTcCzTItrop8mq/lbzL8wSGtj94UO/3U31shqG0=
+go.etcd.io/bbolt v1.3.10/go.mod h1:bK3UQLPJZly7IlNmV7uVHJDxfe5aK9Ll93e/74Y9oEQ=
 go.opencensus.io v0.21.0/go.mod h1:mSImk1erAIZhrmZN+AvHh14ztQfjbGwt4TtuofqLduU=
 go.opencensus.io v0.22.0/go.mod h1:+kGneAE2xo2IficOXnaByMWTGM9T73dGwxeWcUqIpI8=
 go.opentelemetry.io/contrib/instrumentation/google.golang.org/grpc/otelgrpc v0.46.0 h1:PzIubN4/sjByhDRHLviCjJuweBXWFZWhghjg7cS28+M=



Test patch:
diff --git a/tests/e2e/cluster_downgrade_test.go b/tests/e2e/cluster_downgrade_test.go
new file mode 100644
index 000000000..8729bbda5
--- /dev/null
+++ b/tests/e2e/cluster_downgrade_test.go
@@ -0,0 +1,299 @@
+// Copyright 2024 The etcd Authors
+//
+// Licensed under the Apache License, Version 2.0 (the "License");
+// you may not use this file except in compliance with the License.
+// You may obtain a copy of the License at
+//
+//     http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+
+package e2e
+
+import (
+	"encoding/json"
+	"fmt"
+	"strings"
+	"testing"
+	"time"
+
+	"github.com/coreos/go-semver/semver"
+	"github.com/stretchr/testify/assert"
+	"github.com/stretchr/testify/require"
+	"go.uber.org/zap"
+
+	"go.etcd.io/etcd/api/v3/version"
+	"go.etcd.io/etcd/client/pkg/v3/fileutil"
+	clientv3 "go.etcd.io/etcd/client/v3"
+	"go.etcd.io/etcd/server/v3/datadir"
+	"go.etcd.io/etcd/server/v3/etcdserver"
+	"go.etcd.io/etcd/server/v3/etcdserver/api/snap"
+	"go.etcd.io/etcd/tests/v3/framework/e2e"
+	"go.etcd.io/etcd/tests/v3/testutils"
+)
+
+func TestDowngradeUpgradeClusterOf1(t *testing.T) {
+	testDowngradeUpgrade(t, 1, false)
+}
+
+func TestDowngradeUpgradeClusterOf3(t *testing.T) {
+	testDowngradeUpgrade(t, 3, false)
+}
+
+func TestDowngradeUpgradeClusterOf1WithSnapshot(t *testing.T) {
+	testDowngradeUpgrade(t, 1, true)
+}
+
+func TestDowngradeUpgradeClusterOf3WithSnapshot(t *testing.T) {
+	testDowngradeUpgrade(t, 3, true)
+}
+
+func testDowngradeUpgrade(t *testing.T, clusterSize int, triggerSnapshot bool) {
+	currentEtcdBinary := e2e.BinPath
+	lastReleaseBinary := e2e.BinPathLastRelease
+	if !fileutil.Exist(lastReleaseBinary) {
+		t.Skipf("%q does not exist", lastReleaseBinary)
+	}
+
+	currentVersion, err := e2e.GetVersionFromBinary(currentEtcdBinary)
+	require.NoError(t, err)
+	// wipe any pre-release suffix like -alpha.0 we see commonly in builds
+	currentVersion.PreRelease = ""
+
+	lastVersion, err := e2e.GetVersionFromBinary(lastReleaseBinary)
+	require.NoError(t, err)
+
+	require.Equalf(t, lastVersion.Minor, currentVersion.Minor-1, "unexpected minor version difference")
+	currentVersionStr := currentVersion.String()
+	lastVersionStr := lastVersion.String()
+
+	currentClusterVersion := semver.New(currentVersionStr)
+	currentClusterVersion.Patch = 0
+	currentClusterVersionStr := currentClusterVersion.String()
+
+	lastClusterVersion := semver.New(lastVersionStr)
+	lastClusterVersion.Patch = 0
+	lastClusterVersionStr := lastClusterVersion.String()
+
+	e2e.BeforeTest(t)
+
+	t.Logf("Create cluster with version %s", currentVersionStr)
+	snapshotCount := 10
+	epc := newCluster(t, clusterSize, snapshotCount)
+	for i := 0; i < len(epc.Procs); i++ {
+		validateVersion(t, epc.Cfg, epc.Procs[i], version.Versions{
+			Cluster: currentClusterVersionStr,
+			Server:  version.Version,
+		})
+	}
+	cc := epc.Etcdctl()
+	t.Logf("Cluster created")
+	if len(epc.Procs) > 1 {
+		t.Log("Waiting health interval to required to make membership changes")
+		time.Sleep(etcdserver.HealthInterval)
+	}
+
+	t.Log("Adding member to test membership, but a learner avoid breaking quorum")
+	resp, err := cc.MemberAddAsLearner("fake1", []string{"http://127.0.0.1:1001"})
+	require.NoError(t, err)
+	if triggerSnapshot {
+		t.Logf("Generating snapshot")
+		generateSnapshot(t, snapshotCount, cc)
+		verifySnapshot(t, epc)
+	}
+	t.Log("Removing learner to test membership")
+	_, err = cc.MemberRemove(resp.Member.ID)
+	require.NoError(t, err)
+	beforeMembers, beforeKV := getMembersAndKeys(t, cc)
+
+	t.Logf("Starting downgrade process to %q", lastVersionStr)
+	for i := 0; i < len(epc.Procs); i++ {
+		t.Logf("Downgrading member %d by running %s binary", i, lastReleaseBinary)
+		stopEtcd(t, epc.Procs[i])
+		startEtcd(t, epc.Procs[i], lastReleaseBinary)
+	}
+
+	t.Log("All members downgraded, validating downgrade")
+	for i := 0; i < len(epc.Procs); i++ {
+		validateVersion(t, epc.Cfg, epc.Procs[i], version.Versions{
+			Cluster: lastClusterVersionStr,
+			Server:  lastVersionStr,
+		})
+	}
+
+	t.Log("Downgrade complete")
+	afterMembers, afterKV := getMembersAndKeys(t, cc)
+	assert.Equal(t, beforeKV.Kvs, afterKV.Kvs)
+	assert.Equal(t, beforeMembers.Members, afterMembers.Members)
+
+	if len(epc.Procs) > 1 {
+		t.Log("Waiting health interval to required to make membership changes")
+		time.Sleep(etcdserver.HealthInterval)
+	}
+	t.Log("Adding learner to test membership, but avoid breaking quorum")
+	resp, err = cc.MemberAddAsLearner("fake2", []string{"http://127.0.0.1:1002"})
+	require.NoError(t, err)
+	if triggerSnapshot {
+		t.Logf("Generating snapshot")
+		generateSnapshot(t, snapshotCount, cc)
+		verifySnapshot(t, epc)
+	}
+	t.Log("Removing learner to test membership")
+	_, err = cc.MemberRemove(resp.Member.ID)
+	require.NoError(t, err)
+	beforeMembers, beforeKV = getMembersAndKeys(t, cc)
+
+	t.Logf("Starting upgrade process to %q", currentVersionStr)
+	for i := 0; i < len(epc.Procs); i++ {
+		t.Logf("Upgrading member %d", i)
+		stopEtcd(t, epc.Procs[i])
+		startEtcd(t, epc.Procs[i], currentEtcdBinary)
+		// NOTE: The leader has monitor to the cluster version, which will
+		// update cluster version. We don't need to check the transient
+		// version just in case that it might be flaky.
+	}
+
+	t.Log("All members upgraded, validating upgrade")
+	for i := 0; i < len(epc.Procs); i++ {
+		validateVersion(t, epc.Cfg, epc.Procs[i], version.Versions{
+			Cluster: currentClusterVersionStr,
+			Server:  version.Version,
+		})
+	}
+	t.Log("Upgrade complete")
+
+	afterMembers, afterKV = getMembersAndKeys(t, cc)
+	assert.Equal(t, beforeKV.Kvs, afterKV.Kvs)
+	assert.Equal(t, beforeMembers.Members, afterMembers.Members)
+}
+
+func newCluster(t *testing.T, clusterSize int, snapshotCount int) *e2e.EtcdProcessCluster {
+	copiedCfg := e2e.NewConfigNoTLS()
+	copiedCfg.ClusterSize = clusterSize
+	copiedCfg.SnapshotCount = snapshotCount
+	copiedCfg.KeepDataDir = true
+	copiedCfg.BasePeerScheme = "unix" // to avoid port conflict
+
+	epc, err := e2e.NewEtcdProcessCluster(t, copiedCfg)
+	if err != nil {
+		t.Fatalf("could not start etcd process cluster (%v)", err)
+	}
+	t.Cleanup(func() {
+		if errC := epc.Close(); errC != nil {
+			t.Fatalf("error closing etcd processes (%v)", errC)
+		}
+	})
+	return epc
+}
+
+func startEtcd(t *testing.T, ep e2e.EtcdProcess, execPath string) {
+	ep.Config().ExecPath = execPath
+	if execPath == e2e.BinPathLastRelease {
+		ep.Config().Args = addOrRemoveArg(ep.Config().Args, "--next-cluster-version-compatible", true)
+	} else {
+		ep.Config().Args = addOrRemoveArg(ep.Config().Args, "--next-cluster-version-compatible", false)
+	}
+	err := ep.Restart()
+	if err != nil {
+		t.Fatalf("could not start etcd process cluster (%v)", err)
+	}
+}
+
+func addOrRemoveArg(args []string, arg string, isAdd bool) []string {
+	newArgs := []string{}
+	for _, s := range args {
+		if s != arg {
+			newArgs = append(newArgs, s)
+		}
+	}
+	if isAdd {
+		newArgs = append(newArgs, arg)
+	}
+	return newArgs
+}
+
+func stopEtcd(t *testing.T, ep e2e.EtcdProcess) {
+	if err := ep.Stop(); err != nil {
+		t.Fatal(err)
+	}
+}
+
+func validateVersion(t *testing.T, cfg *e2e.EtcdProcessClusterConfig, member e2e.EtcdProcess, expect version.Versions) {
+	testutils.ExecuteWithTimeout(t, 30*time.Second, func() {
+		for {
+			result, err := getMemberVersionByCurl(cfg, member)
+			if err != nil {
+				cfg.Logger.Warn("failed to get member version and retrying", zap.Error(err), zap.String("member", member.Config().Name))
+				time.Sleep(time.Second)
+				continue
+			}
+			cfg.Logger.Info("Comparing versions", zap.String("member", member.Config().Name), zap.Any("got", result), zap.Any("want", expect))
+			if err := compareMemberVersion(expect, result); err != nil {
+				cfg.Logger.Warn("Versions didn't match retrying", zap.Error(err), zap.String("member", member.Config().Name))
+				time.Sleep(time.Second)
+				continue
+			}
+			cfg.Logger.Info("Versions match", zap.String("member", member.Config().Name))
+			break
+		}
+	})
+}
+
+func compareMemberVersion(expect version.Versions, target version.Versions) error {
+	if expect.Server != "" && expect.Server != target.Server {
+		return fmt.Errorf("expect etcdserver version %v, but got %v", expect.Server, target.Server)
+	}
+
+	if expect.Cluster != "" && expect.Cluster != target.Cluster {
+		return fmt.Errorf("expect etcdcluster version %v, but got %v", expect.Cluster, target.Cluster)
+	}
+
+	return nil
+}
+
+func getMemberVersionByCurl(cfg *e2e.EtcdProcessClusterConfig, member e2e.EtcdProcess) (version.Versions, error) {
+	args := e2e.CURLPrefixArgsCluster(cfg, member, "GET", e2e.CURLReq{Endpoint: "/version"})
+	lines, err := e2e.RunUtilCompletion(args, nil)
+	if err != nil {
+		return version.Versions{}, err
+	}
+
+	data := strings.Join(lines, "\n")
+	result := version.Versions{}
+	if err := json.Unmarshal([]byte(data), &result); err != nil {
+		return version.Versions{}, fmt.Errorf("failed to unmarshal (%v): %w", data, err)
+	}
+	return result, nil
+}
+
+func generateSnapshot(t *testing.T, snapshotCount int, cc *e2e.Etcdctl) {
+	t.Logf("Adding keys")
+	for i := 0; i < snapshotCount*3; i++ {
+		err := cc.Put(fmt.Sprintf("%d", i), "1")
+		assert.NoError(t, err)
+	}
+}
+
+func verifySnapshot(t *testing.T, epc *e2e.EtcdProcessCluster) {
+	for i := range epc.Procs {
+		t.Logf("Verifying snapshot for member %d", i)
+		ss := snap.New(epc.Cfg.Logger, datadir.ToSnapDir(epc.Procs[i].Config().DataDirPath))
+		_, err := ss.Load()
+		assert.NoError(t, err)
+	}
+	t.Logf("All members have a valid snapshot")
+}
+
+func getMembersAndKeys(t *testing.T, cc *e2e.Etcdctl) (*clientv3.MemberListResponse, *clientv3.GetResponse) {
+	kvs, err := cc.GetWithPrefix("")
+	assert.NoError(t, err)
+
+	members, err := cc.MemberList()
+	assert.NoError(t, err)
+
+	return members, kvs
+}
diff --git a/tests/e2e/ctl_v3_grpc_test.go b/tests/e2e/ctl_v3_grpc_test.go
index 7cdf27d1a..9c7bb068f 100644
--- a/tests/e2e/ctl_v3_grpc_test.go
+++ b/tests/e2e/ctl_v3_grpc_test.go
@@ -26,8 +26,8 @@ import (
 
 	"github.com/stretchr/testify/assert"
 
-	"go.etcd.io/etcd/client/pkg/v3/testutil"
 	"go.etcd.io/etcd/tests/v3/framework/e2e"
+	"go.etcd.io/etcd/tests/v3/testutils"
 )
 
 func TestAuthority(t *testing.T) {
@@ -140,7 +140,7 @@ func TestAuthority(t *testing.T) {
 					}
 				}
 
-				executeWithTimeout(t, 5*time.Second, func() {
+				testutils.ExecuteWithTimeout(t, 5*time.Second, func() {
 					assertAuthority(t, tc.expectAuthorityPattern, epc)
 				})
 			})
@@ -179,20 +179,6 @@ func assertAuthority(t *testing.T, expectAuthorityPattern string, clus *e2e.Etcd
 	}
 }
 
-func executeWithTimeout(t *testing.T, timeout time.Duration, f func()) {
-	donec := make(chan struct{})
-	go func() {
-		defer close(donec)
-		f()
-	}()
-
-	select {
-	case <-time.After(timeout):
-		testutil.FatalStack(t, fmt.Sprintf("test timed out after %v", timeout))
-	case <-donec:
-	}
-}
-
 type etcdctlV3 struct {
 	cfg       *e2e.EtcdProcessClusterConfig
 	endpoints []string
diff --git a/tests/e2e/failover_test.go b/tests/e2e/failover_test.go
new file mode 100644
index 000000000..505cef01e
--- /dev/null
+++ b/tests/e2e/failover_test.go
@@ -0,0 +1,159 @@
+// Copyright 2023 The etcd Authors
+//
+// Licensed under the Apache License, Version 2.0 (the "License");
+// you may not use this file except in compliance with the License.
+// You may obtain a copy of the License at
+//
+//     http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+
+//go:build !cluster_proxy
+
+package e2e
+
+import (
+	"context"
+	"testing"
+	"time"
+
+	"github.com/stretchr/testify/require"
+	"golang.org/x/sync/errgroup"
+	"google.golang.org/grpc"
+	_ "google.golang.org/grpc/health"
+
+	clientv3 "go.etcd.io/etcd/client/v3"
+	"go.etcd.io/etcd/tests/v3/framework/e2e"
+)
+
+const (
+	// in sync with how kubernetes uses etcd
+	// https://github.com/kubernetes/kubernetes/blob/release-1.28/staging/src/k8s.io/apiserver/pkg/storage/storagebackend/factory/etcd3.go#L59-L71
+	keepaliveTime    = 30 * time.Second
+	keepaliveTimeout = 10 * time.Second
+	dialTimeout      = 20 * time.Second
+
+	clientRuntime  = 10 * time.Second
+	requestTimeout = 100 * time.Millisecond
+)
+
+func TestFailoverOnDefrag(t *testing.T) {
+	tcs := []struct {
+		name string
+
+		experimentalStopGRPCServiceOnDefragEnabled bool
+		gRPCDialOptions                            []grpc.DialOption
+
+		// common assertion
+		expectedMinQPS float64
+		// happy case assertion
+		expectedMaxFailureRate float64
+		// negative case assertion
+		expectedMinFailureRate float64
+	}{
+		{
+			name: "defrag failover happy case",
+			experimentalStopGRPCServiceOnDefragEnabled: true,
+			gRPCDialOptions: []grpc.DialOption{
+				grpc.WithDisableServiceConfig(),
+				grpc.WithDefaultServiceConfig(`{"loadBalancingPolicy": "round_robin", "healthCheckConfig": {"serviceName": ""}}`),
+			},
+			expectedMinQPS:         20,
+			expectedMaxFailureRate: 0.01,
+		},
+		{
+			name: "defrag blocks one-third of requests with stopGRPCServiceOnDefrag set to false",
+			experimentalStopGRPCServiceOnDefragEnabled: false,
+			gRPCDialOptions: []grpc.DialOption{
+				grpc.WithDisableServiceConfig(),
+				grpc.WithDefaultServiceConfig(`{"loadBalancingPolicy": "round_robin", "healthCheckConfig": {"serviceName": ""}}`),
+			},
+			expectedMinQPS:         20,
+			expectedMinFailureRate: 0.25,
+		},
+		{
+			name: "defrag blocks one-third of requests with stopGRPCServiceOnDefrag set to true and client health check disabled",
+			experimentalStopGRPCServiceOnDefragEnabled: true,
+			expectedMinQPS:         20,
+			expectedMinFailureRate: 0.25,
+		},
+	}
+
+	for _, tc := range tcs {
+		t.Run(tc.name, func(t *testing.T) {
+			e2e.BeforeTest(t)
+			cfg := e2e.EtcdProcessClusterConfig{
+				ClusterSize:                         3,
+				GoFailEnabled:                       true,
+				ExperimentalStopGRPCServiceOnDefrag: tc.experimentalStopGRPCServiceOnDefragEnabled,
+			}
+			clus, err := e2e.NewEtcdProcessCluster(t, &cfg)
+			require.NoError(t, err)
+			t.Cleanup(func() { clus.Stop() })
+
+			endpoints := clus.EndpointsGRPC()
+
+			requestVolume, successfulRequestCount := 0, 0
+			start := time.Now()
+			g := new(errgroup.Group)
+			g.Go(func() (lastErr error) {
+				clusterClient, cerr := clientv3.New(clientv3.Config{
+					DialTimeout:          dialTimeout,
+					DialKeepAliveTime:    keepaliveTime,
+					DialKeepAliveTimeout: keepaliveTimeout,
+					Endpoints:            endpoints,
+					DialOptions:          tc.gRPCDialOptions,
+				})
+				if cerr != nil {
+					return cerr
+				}
+				defer clusterClient.Close()
+
+				timeout := time.After(clientRuntime)
+				for {
+					select {
+					case <-timeout:
+						return lastErr
+					default:
+					}
+					getContext, cancel := context.WithTimeout(context.Background(), requestTimeout)
+					_, err := clusterClient.Get(getContext, "health")
+					cancel()
+					requestVolume++
+					if err != nil {
+						lastErr = err
+						continue
+					}
+					successfulRequestCount++
+				}
+			})
+
+			triggerDefrag(t, clus.Procs[0])
+
+			err = g.Wait()
+			if err != nil {
+				t.Logf("etcd client failed to fail over, error (%v)", err)
+			}
+			qps := float64(requestVolume) / float64(time.Since(start)) * float64(time.Second)
+			failureRate := 1 - float64(successfulRequestCount)/float64(requestVolume)
+			t.Logf("request failure rate is %.2f%%, qps is %.2f requests/second", failureRate*100, qps)
+
+			require.GreaterOrEqual(t, qps, tc.expectedMinQPS)
+			if tc.expectedMaxFailureRate != 0.0 {
+				require.LessOrEqual(t, failureRate, tc.expectedMaxFailureRate)
+			}
+			if tc.expectedMinFailureRate != 0.0 {
+				require.GreaterOrEqual(t, failureRate, tc.expectedMinFailureRate)
+			}
+		})
+	}
+}
+
+func triggerDefrag(t *testing.T, member e2e.EtcdProcess) {
+	require.NoError(t, member.Failpoints().SetupHTTP(context.Background(), "defragBeforeCopy", `sleep("10s")`))
+	require.NoError(t, member.Etcdctl(e2e.ClientNonTLS, false, false).Defragment(time.Minute))
+}
diff --git a/tests/e2e/v3_curl_test.go b/tests/e2e/v3_curl_test.go
index 581d56d89..032f6377c 100644
--- a/tests/e2e/v3_curl_test.go
+++ b/tests/e2e/v3_curl_test.go
@@ -18,6 +18,7 @@ import (
 	"encoding/base64"
 	"encoding/json"
 	"fmt"
+	"math/rand"
 	"path"
 	"strconv"
 	"testing"
@@ -244,7 +245,7 @@ func testV3CurlAuth(cx ctlCtx) {
 			lineFunc   = func(txt string) bool { return true }
 		)
 
-		cmdArgs = e2e.CURLPrefixArgsCluster(cx.epc, "POST", e2e.CURLReq{Endpoint: path.Join(p, "/auth/authenticate"), Value: string(authreq)})
+		cmdArgs = e2e.CURLPrefixArgsCluster(cx.epc.Cfg, cx.epc.Procs[rand.Intn(cx.epc.Cfg.ClusterSize)], "POST", e2e.CURLReq{Endpoint: path.Join(p, "/auth/authenticate"), Value: string(authreq)})
 		proc, err := e2e.SpawnCmd(cmdArgs, cx.envMap)
 		testutil.AssertNil(cx.t, err)
 		defer proc.Close()
@@ -283,7 +284,7 @@ func testV3CurlCampaign(cx ctlCtx) {
 	if err != nil {
 		cx.t.Fatal(err)
 	}
-	cargs := e2e.CURLPrefixArgsCluster(cx.epc, "POST", e2e.CURLReq{
+	cargs := e2e.CURLPrefixArgsCluster(cx.epc.Cfg, cx.epc.Procs[rand.Intn(cx.epc.Cfg.ClusterSize)], "POST", e2e.CURLReq{
 		Endpoint: path.Join(cx.apiPrefix, "/election/campaign"),
 		Value:    string(cdata),
 	})
diff --git a/tests/framework/e2e/cluster.go b/tests/framework/e2e/cluster.go
index 4a52fac24..250900bb3 100644
--- a/tests/framework/e2e/cluster.go
+++ b/tests/framework/e2e/cluster.go
@@ -191,6 +191,8 @@ type EtcdProcessClusterConfig struct {
 	CompactHashCheckTime       time.Duration
 	WatchProcessNotifyInterval time.Duration
 	CompactionBatchLimit       int
+
+	ExperimentalStopGRPCServiceOnDefrag bool
 }
 
 // NewEtcdProcessCluster launches a new cluster from etcd processes, returning
@@ -329,6 +331,9 @@ func (cfg *EtcdProcessClusterConfig) EtcdServerProcessConfig(tb testing.TB, i in
 	if cfg.InitialCorruptCheck {
 		args = append(args, "--experimental-initial-corrupt-check")
 	}
+	if cfg.ExperimentalStopGRPCServiceOnDefrag {
+		args = append(args, "--experimental-stop-grpc-service-on-defrag")
+	}
 	var murl string
 	if cfg.MetricsURLScheme != "" {
 		murl = (&url.URL{
diff --git a/tests/framework/e2e/curl.go b/tests/framework/e2e/curl.go
index 9df19620a..6ad5a01a6 100644
--- a/tests/framework/e2e/curl.go
+++ b/tests/framework/e2e/curl.go
@@ -43,13 +43,12 @@ type CURLReq struct {
 
 // CURLPrefixArgsCluster builds the beginning of a curl command for a given key
 // addressed to a random URL in the given cluster.
-func CURLPrefixArgsCluster(clus *EtcdProcessCluster, method string, req CURLReq) []string {
-	member := clus.Procs[rand.Intn(clus.Cfg.ClusterSize)]
+func CURLPrefixArgsCluster(cfg *EtcdProcessClusterConfig, member EtcdProcess, method string, req CURLReq) []string {
 	clientURL := member.Config().Acurl
 	if req.MetricsURLScheme != "" {
 		clientURL = member.EndpointsMetrics()[0]
 	}
-	return CURLPrefixArgs(clientURL, clus.Cfg.ClientTLS, !clus.Cfg.NoCN, method, req)
+	return CURLPrefixArgs(clientURL, cfg.ClientTLS, !cfg.NoCN, method, req)
 }
 
 // CURLPrefixArgs builds the beginning of a curl command for a given key
@@ -111,13 +110,13 @@ func CURLPrefixArgs(clientURL string, connType ClientConnType, CN bool, method s
 }
 
 func CURLPost(clus *EtcdProcessCluster, req CURLReq) error {
-	return SpawnWithExpect(CURLPrefixArgsCluster(clus, "POST", req), req.Expected)
+	return SpawnWithExpect(CURLPrefixArgsCluster(clus.Cfg, clus.Procs[rand.Intn(clus.Cfg.ClusterSize)], "POST", req), req.Expected)
 }
 
 func CURLPut(clus *EtcdProcessCluster, req CURLReq) error {
-	return SpawnWithExpect(CURLPrefixArgsCluster(clus, "PUT", req), req.Expected)
+	return SpawnWithExpect(CURLPrefixArgsCluster(clus.Cfg, clus.Procs[rand.Intn(clus.Cfg.ClusterSize)], "PUT", req), req.Expected)
 }
 
 func CURLGet(clus *EtcdProcessCluster, req CURLReq) error {
-	return SpawnWithExpect(CURLPrefixArgsCluster(clus, "GET", req), req.Expected)
+	return SpawnWithExpect(CURLPrefixArgsCluster(clus.Cfg, clus.Procs[rand.Intn(clus.Cfg.ClusterSize)], "GET", req), req.Expected)
 }
diff --git a/tests/framework/e2e/etcd_process.go b/tests/framework/e2e/etcd_process.go
index c20844286..bb464e4db 100644
--- a/tests/framework/e2e/etcd_process.go
+++ b/tests/framework/e2e/etcd_process.go
@@ -27,6 +27,7 @@ import (
 	"testing"
 	"time"
 
+	"github.com/coreos/go-semver/semver"
 	"go.etcd.io/etcd/client/pkg/v3/fileutil"
 	"go.etcd.io/etcd/pkg/v3/expect"
 	"go.etcd.io/etcd/pkg/v3/proxy"
@@ -405,3 +406,27 @@ func parseFailpointsBody(body io.Reader) (map[string]string, error) {
 	}
 	return failpoints, nil
 }
+
+func GetVersionFromBinary(binaryPath string) (*semver.Version, error) {
+	lines, err := RunUtilCompletion([]string{binaryPath, "--version"}, nil)
+	if err != nil {
+		return nil, fmt.Errorf("could not find binary version from %s, err: %w", binaryPath, err)
+	}
+
+	for _, line := range lines {
+		if strings.HasPrefix(line, "etcd Version:") {
+			versionString := strings.TrimSpace(strings.SplitAfter(line, ":")[1])
+			version, err := semver.NewVersion(versionString)
+			if err != nil {
+				return nil, err
+			}
+			return &semver.Version{
+				Major: version.Major,
+				Minor: version.Minor,
+				Patch: version.Patch,
+			}, nil
+		}
+	}
+
+	return nil, fmt.Errorf("could not find version in binary output of %s, lines outputted were %v", binaryPath, lines)
+}
diff --git a/tests/framework/e2e/etcdctl.go b/tests/framework/e2e/etcdctl.go
index 7e7361b17..9bf18811f 100644
--- a/tests/framework/e2e/etcdctl.go
+++ b/tests/framework/e2e/etcdctl.go
@@ -18,6 +18,7 @@ import (
 	"encoding/json"
 	"fmt"
 	"strings"
+	"time"
 
 	clientv3 "go.etcd.io/etcd/client/v3"
 	"go.etcd.io/etcd/tests/v3/integration"
@@ -61,6 +62,12 @@ func (ctl *Etcdctl) Get(key string) (*clientv3.GetResponse, error) {
 	return &resp, err
 }
 
+func (ctl *Etcdctl) GetWithPrefix(key string) (*clientv3.GetResponse, error) {
+	var resp clientv3.GetResponse
+	err := ctl.spawnJsonCmd(&resp, "get", key, "--prefix")
+	return &resp, err
+}
+
 func (ctl *Etcdctl) Put(key, value string) error {
 	if ctl.v2 {
 		panic("Unsupported method for v2")
@@ -149,6 +156,15 @@ func (ctl *Etcdctl) MemberAdd(name string, peerURLs []string) (*clientv3.MemberA
 	return &resp, err
 }
 
+func (ctl *Etcdctl) MemberAddAsLearner(name string, peerURLs []string) (*clientv3.MemberAddResponse, error) {
+	if ctl.v2 {
+		panic("Unsupported method for v2")
+	}
+	var resp clientv3.MemberAddResponse
+	err := ctl.spawnJsonCmd(&resp, "member", "add", name, "--learner", "--peer-urls", strings.Join(peerURLs, ","))
+	return &resp, err
+}
+
 func (ctl *Etcdctl) MemberRemove(id uint64) (*clientv3.MemberRemoveResponse, error) {
 	if ctl.v2 {
 		panic("Unsupported method for v2")
@@ -166,6 +182,15 @@ func (ctl *Etcdctl) Compact(rev int64) (*clientv3.CompactResponse, error) {
 	return nil, SpawnWithExpectWithEnv(args, ctl.env(), fmt.Sprintf("compacted revision %v", rev))
 }
 
+func (ctl *Etcdctl) Defragment(timeout time.Duration) error {
+	args := append(ctl.cmdArgs(), "defrag")
+	if timeout != 0 {
+		args = append(args, fmt.Sprintf("--command-timeout=%s", timeout))
+	}
+
+	return SpawnWithExpectWithEnv(args, ctl.env(), "Finished defragmenting etcd member")
+}
+
 func (ctl *Etcdctl) Status() ([]*clientv3.StatusResponse, error) {
 	var epStatus []*struct {
 		Endpoint string
diff --git a/tests/go.mod b/tests/go.mod
index 19f09c2c6..bd3ad9d94 100644
--- a/tests/go.mod
+++ b/tests/go.mod
@@ -15,6 +15,7 @@ replace (
 )
 
 require (
+	github.com/coreos/go-semver v0.3.0
 	github.com/dustin/go-humanize v1.0.0
 	github.com/etcd-io/gofail v0.0.0-20190801230047-ad7f989257ca
 	github.com/gogo/protobuf v1.3.2
@@ -54,7 +55,6 @@ require (
 	github.com/beorn7/perks v1.0.1 // indirect
 	github.com/cenkalti/backoff/v4 v4.2.1 // indirect
 	github.com/cespare/xxhash/v2 v2.2.0 // indirect
-	github.com/coreos/go-semver v0.3.0 // indirect
 	github.com/coreos/go-systemd/v22 v22.3.2 // indirect
 	github.com/creack/pty v1.1.11 // indirect
 	github.com/davecgh/go-spew v1.1.1 // indirect
@@ -77,7 +77,7 @@ require (
 	github.com/sirupsen/logrus v1.9.3 // indirect
 	github.com/tmc/grpc-websocket-proxy v0.0.0-20201229170055-e5319fda7802 // indirect
 	github.com/xiang90/probing v0.0.0-20190116061207-43a291ad63a2 // indirect
-	go.etcd.io/bbolt v1.3.9 // indirect
+	go.etcd.io/bbolt v1.3.10 // indirect
 	go.opentelemetry.io/otel/exporters/otlp/otlptrace v1.20.0 // indirect
 	go.opentelemetry.io/otel/exporters/otlp/otlptrace/otlptracegrpc v1.20.0 // indirect
 	go.opentelemetry.io/otel/metric v1.20.0 // indirect
diff --git a/tests/go.sum b/tests/go.sum
index 54936cde0..3f76c93c7 100644
--- a/tests/go.sum
+++ b/tests/go.sum
@@ -302,8 +302,8 @@ github.com/xiang90/probing v0.0.0-20190116061207-43a291ad63a2/go.mod h1:UETIi67q
 github.com/yuin/goldmark v1.1.27/go.mod h1:3hX8gzYuyVAZsxl0MRgGTJEmQBFcNTphYh9decYSb74=
 github.com/yuin/goldmark v1.2.1/go.mod h1:3hX8gzYuyVAZsxl0MRgGTJEmQBFcNTphYh9decYSb74=
 go.etcd.io/bbolt v1.3.2/go.mod h1:IbVyRI1SCnLcuJnV2u8VeU0CEYM7e686BmAb1XKL+uU=
-go.etcd.io/bbolt v1.3.9 h1:8x7aARPEXiXbHmtUwAIv7eV2fQFHrLLavdiJ3uzJXoI=
-go.etcd.io/bbolt v1.3.9/go.mod h1:zaO32+Ti0PK1ivdPtgMESzuzL2VPoIG1PCQNvOdo/dE=
+go.etcd.io/bbolt v1.3.10 h1:+BqfJTcCzTItrop8mq/lbzL8wSGtj94UO/3U31shqG0=
+go.etcd.io/bbolt v1.3.10/go.mod h1:bK3UQLPJZly7IlNmV7uVHJDxfe5aK9Ll93e/74Y9oEQ=
 go.etcd.io/gofail v0.1.0 h1:XItAMIhOojXFQMgrxjnd2EIIHun/d5qL0Pf7FzVTkFg=
 go.etcd.io/gofail v0.1.0/go.mod h1:VZBCXYGZhHAinaBiiqYvuDynvahNsAyLFwB3kEHKz1M=
 go.opencensus.io v0.21.0/go.mod h1:mSImk1erAIZhrmZN+AvHh14ztQfjbGwt4TtuofqLduU=
diff --git a/tests/testutils/execute.go b/tests/testutils/execute.go
new file mode 100644
index 000000000..a80bf7a0b
--- /dev/null
+++ b/tests/testutils/execute.go
@@ -0,0 +1,37 @@
+// Copyright 2024 The etcd Authors
+//
+// Licensed under the Apache License, Version 2.0 (the "License");
+// you may not use this file except in compliance with the License.
+// You may obtain a copy of the License at
+//
+//     http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+
+package testutils
+
+import (
+	"fmt"
+	"testing"
+	"time"
+
+	"go.etcd.io/etcd/client/pkg/v3/testutil"
+)
+
+func ExecuteWithTimeout(t *testing.T, timeout time.Duration, f func()) {
+	donec := make(chan struct{})
+	go func() {
+		defer close(donec)
+		f()
+	}()
+
+	select {
+	case <-time.After(timeout):
+		testutil.FatalStack(t, fmt.Sprintf("test timed out after %v", timeout))
+	case <-donec:
+	}
+}