Repository: etcd-io/etcd

Issue:
Using a direct method call with v3client that doesn't go through gRPC to call the Endpoints() method results in a null pointer, causing a panic. 
### Bug report criteria

- [X] This bug report is not security related, security issues should be disclosed privately via security@etcd.io.
- [X] This is not a support request or question, support requests or questions should be raised in the etcd [discussion forums](https://github.com/etcd-io/etcd/discussions).
- [X] You have read the etcd [bug reporting guidelines](https://github.com/etcd-io/etcd/blob/main/Documentation/contributor-guide/reporting_bugs.md).
- [X] Existing open issues along with etcd [frequently asked questions](https://etcd.io/docs/latest/faq) have been checked and this is not a duplicate.

### What happened?

Using a direct method call with v3client that doesn't go through gRPC to call the Endpoints() method results in a null pointer, causing a panic. The reason is that a variable named `mu` of type `*sync.RWMutex` is not initialized.

## Background

In an effort to minimize Kubernetes (k8s), the apiserver code has been intrusively modified to directly use a v3client for invoking embedded etcd. However, calling the Endpoints() method of etcd's client using this approach leads to a null pointer dereference, causing a panic. The issue stems from an uninitialized variable named `epMu` of type `*sync.RWMutex`.

### Issue Description

The panic occurs as follows:

```
panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x10 pc=0xa41eac]

goroutine 1 [running]:
sync.(*RWMutex).RLock(...)
        C:/Program Files/Go1.19/go/src/sync/rwmutex.go:69
go.etcd.io/etcd/client/v3.(*Client).Endpoints(0xc0000fe380)
        /go/pkg/mod/go.etcd.io/etcd/client/v3@v3.5.10/client.go:164 +0x4c
main.main()
        D:/01work/code/go/test/main.go:32 +0x1cc    // Endpoints lists the registered endpoints for the client.
```



The null pointer occurs in `client/v3/client.go`, where `c.mu` is not initialized:

```go
// Endpoints lists the registered endpoints for the client.
func (c *Client) Endpoints() []string {
	// copy the slice; protect original endpoints from being changed
	c.mu.RLock()
	defer c.mu.RUnlock()
	eps := make([]string, len(c.cfg.Endpoints))
	copy(eps, c.cfg.Endpoints)
	return eps
}
```



Looking at the initialization code in `server/etcdserver/api/v3client/v3client.go`:

```go
// New creates a clientv3 client that wraps an in-process EtcdServer. Instead
// of making gRPC calls through sockets, the client makes direct function calls
// to the etcd server through its api/v3rpc function interfaces.
func New(s *etcdserver.EtcdServer) *clientv3.Client {
	c := clientv3.NewCtxClient(context.Background(), clientv3.WithZapLogger(s.Logger()))

	kvc := adapter.KvServerToKvClient(v3rpc.NewQuotaKVServer(s))
	c.KV = clientv3.NewKVFromKVClient(kvc, c)

	lc := adapter.LeaseServerToLeaseClient(v3rpc.NewQuotaLeaseServer(s))
	c.Lease = clientv3.NewLeaseFromLeaseClient(lc, c, time.Second)

	wc := adapter.WatchServerToWatchClient(v3rpc.NewWatchServer(s))
	c.Watcher = &watchWrapper{clientv3.NewWatchFromWatchClient(wc, c)}

	mc := adapter.MaintenanceServerToMaintenanceClient(v3rpc.NewMaintenanceServer(s))
	c.Maintenance = clientv3.NewMaintenanceFromMaintenanceClient(mc, c)

	clc := adapter.ClusterServerToClusterClient(v3rpc.NewClusterServer(s))
	c.Cluster = clientv3.NewClusterFromClusterClient(clc, c)

	a := adapter.AuthServerToAuthClient(v3rpc.NewAuthServer(s))
	c.Auth = clientv3.NewAuthFromAuthClient(a, c)

	return c
}

```



And in `client/v3/client.go`, `NewCtxClient` only initializes `lgMu`, not `mu`:

```go
// NewCtxClient creates a client with a context but no underlying grpc
// connection. This is useful for embedded cases that override the
// service interface implementations and do not need connection management.
func NewCtxClient(ctx context.Context, opts ...Option) *Client {
	cctx, cancel := context.WithCancel(ctx)
	c := &Client{ctx: cctx, cancel: cancel, lgMu: new(sync.RWMutex)}
	for _, opt := range opts {
		opt(c)
	}
	if c.lg == nil {
		c.lg = zap.NewNop()
	}
	return c
}
```



### Problem

The main issue is that when using a direct method call approach with a v3client to access an embedded etcd, the necessary mutex (`mu`) in the etcd client is not being initialized. This oversight leads to a runtime panic when attempting to lock this mutex for reading endpoint information.

### Resolution

To resolve this issue, it's necessary to ensure that all required fields, including `mu`, are properly initialized in the etcd client when it's created. This might involve modifying the `New` function in `v3client.go` or the `NewCtxClient` function in `client.go` to include initialization of the `mu` field.

### What did you expect to happen?

Running normally, no null pointer.

### How can we reproduce it (as minimally and precisely as possible)?

demo

```go
package main

import (
	"go.etcd.io/etcd/server/v3/etcdserver/api/v3client"
	"log"
	"time"

	"go.etcd.io/etcd/server/v3/embed"
)

func main() {
	cfg := embed.NewConfig()
	cfg.Dir = "default.etcd"
	cfg.MaxSnapFiles = 10
	cfg.MaxSnapFiles = 10

	e, err := embed.StartEtcd(cfg)
	if err != nil {
		log.Fatal(err)
	}
	select {
	case <-e.Server.ReadyNotify():
		log.Println("Embedded etcd is ready!")
		cli := v3client.New(e.Server)
		endpoints := cli.Endpoints()
		log.Printf("endpoint is %v", endpoints)

	case <-time.After(60 * time.Second):
		e.Server.Stop()
		e.Close()
		log.Fatal("Embedded etcd took too long to start!")
	}

	<-e.Server.StopNotify()
	log.Println("Embedded etcd is stopped")

}

```



### Anything else we need to know?

Hope to solve version 3.5.4.and master

### Etcd version (please run commands below)

test 3.5.4, 3.5.10 still exist

### Etcd configuration (command line flags or environment variables)

see demo

### Etcd debug information (please run commands below, feel free to obfuscate the IP address or FQDN in the output)

no need

### Relevant log output

```Shell
log


{"level":"warn","ts":"2023-11-23T11:33:41.920945+0800","caller":"embed/config.go:676","msg":"Running http and grpc server on single port. This is not recommended for production."}
{"level":"info","ts":"2023-11-23T11:33:41.921003+0800","caller":"embed/etcd.go:127","msg":"configuring peer listeners","listen-peer-urls":["http://localhost:2380"]}
{"level":"info","ts":"2023-11-23T11:33:41.921678+0800","caller":"embed/etcd.go:135","msg":"configuring client listeners","listen-client-urls":["http://localhost:2379"]}
{"level":"info","ts":"2023-11-23T11:33:41.921826+0800","caller":"embed/etcd.go:309","msg":"starting an etcd server","etcd-version":"3.5.10","git-sha":"Not provided (use ./build instead of go build)","go-version":"go1.19.4","go-os":"linux","go-arch":"amd64","max-cpu-set":16,"max-cpu-available":16,"member-initialized":true,"name":"default","data-dir":"default.etcd","wal-dir":"","wal-dir-dedicated":"","member-dir":"default.etcd/member","force-new-cluster":false,"heartbeat-interval":"100ms","election-timeout":"1s","initial-election-tick-advance":true,"snapshot-count":100000,"max-wals":5,"max-snapshots":10,"snapshot-catchup-entries":5000,"initial-advertise-peer-urls":["http://localhost:2380"],"listen-peer-urls":["http://localhost:2380"],"advertise-client-urls":["http://localhost:2379"],"listen-client-urls":["http://localhost:2379"],"listen-metrics-urls":[],"cors":["*"],"host-whitelist":["*"],"initial-cluster":"","initial-cluster-state":"new","initial-cluster-token":"","quota-backend-bytes":2147483648,"max-request-bytes":1572864,"max-concurrent-streams":4294967295,"pre-vote":true,"initial-corrupt-check":false,"corrupt-check-time-interval":"0s","compact-check-time-enabled":false,"compact-check-time-interval":"1m0s","auto-compaction-mode":"","auto-compaction-retention":"0s","auto-compaction-interval":"0s","discovery-url":"","discovery-proxy":"","downgrade-check-interval":"5s"}
{"level":"info","ts":"2023-11-23T11:33:41.92286+0800","caller":"etcdserver/backend.go:81","msg":"opened backend db","path":"default.etcd/member/snap/db","took":"860.481µs"}
{"level":"info","ts":"2023-11-23T11:33:41.923029+0800","caller":"etcdserver/server.go:530","msg":"No snapshot found. Recovering WAL from scratch!"}
{"level":"info","ts":"2023-11-23T11:33:41.924212+0800","caller":"etcdserver/raft.go:530","msg":"restarting local member","cluster-id":"cdf818194e3a8c32","local-member-id":"8e9e05c52164694d","commit-index":31}
{"level":"info","ts":"2023-11-23T11:33:41.924461+0800","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"8e9e05c52164694d switched to configuration voters=()"}
{"level":"info","ts":"2023-11-23T11:33:41.924495+0800","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"8e9e05c52164694d became follower at term 11"}
{"level":"info","ts":"2023-11-23T11:33:41.924507+0800","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"newRaft 8e9e05c52164694d [peers: [], term: 11, commit: 31, applied: 0, lastindex: 31, lastterm: 11]"}
{"level":"warn","ts":"2023-11-23T11:33:41.925618+0800","caller":"auth/store.go:1241","msg":"simple token is not cryptographically signed"}
{"level":"info","ts":"2023-11-23T11:33:41.926804+0800","caller":"mvcc/kvstore.go:407","msg":"kvstore restored","current-rev":10}
{"level":"info","ts":"2023-11-23T11:33:41.928037+0800","caller":"etcdserver/quota.go:94","msg":"enabled backend quota with default value","quota-name":"v3-applier","quota-size-bytes":2147483648,"quota-size":"2.1 GB"}
{"level":"info","ts":"2023-11-23T11:33:41.929179+0800","caller":"etcdserver/server.go:854","msg":"starting etcd server","local-member-id":"8e9e05c52164694d","local-server-version":"3.5.10","cluster-version":"to_be_decided"}
{"level":"info","ts":"2023-11-23T11:33:41.92923+0800","caller":"fileutil/purge.go:44","msg":"started to purge file","dir":"default.etcd/member/snap","suffix":"snap.db","max":10,"interval":"30s"}
{"level":"info","ts":"2023-11-23T11:33:41.929267+0800","caller":"fileutil/purge.go:44","msg":"started to purge file","dir":"default.etcd/member/snap","suffix":"snap","max":10,"interval":"30s"}
{"level":"info","ts":"2023-11-23T11:33:41.929468+0800","caller":"fileutil/purge.go:44","msg":"started to purge file","dir":"default.etcd/member/wal","suffix":"wal","max":5,"interval":"30s"}
{"level":"info","ts":"2023-11-23T11:33:41.929236+0800","caller":"etcdserver/server.go:754","msg":"starting initial election tick advance","election-ticks":10}
{"level":"info","ts":"2023-11-23T11:33:41.92995+0800","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"8e9e05c52164694d switched to configuration voters=(10276657743932975437)"}
{"level":"info","ts":"2023-11-23T11:33:41.930018+0800","caller":"membership/cluster.go:421","msg":"added member","cluster-id":"cdf818194e3a8c32","local-member-id":"8e9e05c52164694d","added-peer-id":"8e9e05c52164694d","added-peer-peer-urls":["http://localhost:2380"]}
{"level":"info","ts":"2023-11-23T11:33:41.930098+0800","caller":"membership/cluster.go:584","msg":"set initial cluster version","cluster-id":"cdf818194e3a8c32","local-member-id":"8e9e05c52164694d","cluster-version":"3.5"}
{"level":"info","ts":"2023-11-23T11:33:41.93012+0800","caller":"api/capability.go:75","msg":"enabled capabilities for version","cluster-version":"3.5"}
{"level":"info","ts":"2023-11-23T11:33:41.93128+0800","caller":"embed/etcd.go:278","msg":"now serving peer/client/metrics","local-member-id":"8e9e05c52164694d","initial-advertise-peer-urls":["http://localhost:2380"],"listen-peer-urls":["http://localhost:2380"],"advertise-client-urls":["http://localhost:2379"],"listen-client-urls":["http://localhost:2379"],"listen-metrics-urls":[]}
{"level":"info","ts":"2023-11-23T11:33:41.931291+0800","caller":"embed/etcd.go:597","msg":"serving peer traffic","address":"127.0.0.1:2380"}
{"level":"info","ts":"2023-11-23T11:33:41.93132+0800","caller":"embed/etcd.go:569","msg":"cmux::serve","address":"127.0.0.1:2380"}
{"level":"info","ts":"2023-11-23T11:33:43.825288+0800","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"8e9e05c52164694d is starting a new election at term 11"}
{"level":"info","ts":"2023-11-23T11:33:43.825331+0800","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"8e9e05c52164694d became pre-candidate at term 11"}
{"level":"info","ts":"2023-11-23T11:33:43.825356+0800","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"8e9e05c52164694d received MsgPreVoteResp from 8e9e05c52164694d at term 11"}
{"level":"info","ts":"2023-11-23T11:33:43.825371+0800","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"8e9e05c52164694d became candidate at term 12"}
{"level":"info","ts":"2023-11-23T11:33:43.825378+0800","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"8e9e05c52164694d received MsgVoteResp from 8e9e05c52164694d at term 12"}
{"level":"info","ts":"2023-11-23T11:33:43.825388+0800","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"8e9e05c52164694d became leader at term 12"}
{"level":"info","ts":"2023-11-23T11:33:43.825396+0800","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"raft.node: 8e9e05c52164694d elected leader 8e9e05c52164694d at term 12"}
{"level":"info","ts":"2023-11-23T11:33:43.826476+0800","caller":"etcdserver/server.go:2062","msg":"published local member to cluster through raft","local-member-id":"8e9e05c52164694d","local-member-attributes":"{Name:default ClientURLs:[http://localhost:2379]}","request-path":"/0/members/8e9e05c52164694d/attributes","cluster-id":"cdf818194e3a8c32","publish-timeout":"7s"}
2023/11/23 11:33:43 Embedded etcd is ready!
panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x10 pc=0xa41eac]

goroutine 1 [running]:
sync.(*RWMutex).RLock(...)
        C:/Program Files/Go1.19/go/src/sync/rwmutex.go:69
go.etcd.io/etcd/client/v3.(*Client).Endpoints(0xc00066e540)
        go/pkg/mod/go.etcd.io/etcd/client/v3@v3.5.10/client.go:164 +0x4c
main.main()
        D:/01work/code/go/test/main.go:25 +0x18a

```
```


Discussion:
@serathius: Issue is in `server/etcdserver/api/v3client` not `client/v3`@jmhbnz: Discussed during sig etcd triage meeting, assigned to @moficodes to investigate. Feel free to ask any questions here if you get stuck Mofi 🙏🏻 @moficodes: @jmhbnz Thanks. Please assign this to me. @113xiaoji: Please share any updates here. Thank you. @moficodes @moficodes: @jmhbnz 

The easiest fix seems to be adding a `epMu: new(sync.RWMutex)` in `etcd/client/v3/client.go` Line 94.

I tested with this and It starts without the np exception. 

@serathius mentioned the issue is in `etcd/server/etcdserver/api/v3client`. I am not sure how that would work.

If the change in `etcd/client/v3/client.go` makes sense I can have a PR ready.

@serathius: @moficodes, looks you are right. Please disregard my comment.@113xiaoji: @moficodes Please merge the code into version 3.5.4. Thank you.@jmhbnz: > @moficodes Please merge the code into version 3.5.4. Thank you.

Hi @113xiaoji - We would not be backporting this change to the `3.5.4` release, once a release has published we do not alter it.  This change will be implemented into `main` and then backported to `release-3.5` branch to be included in the next release which is currently `3.5.11`.@jmhbnz: Re-open for backport.@stale: This issue has been automatically marked as stale because it has not had recent activity. It will be closed after 21 days if no further activity occurs. Thank you for your contributions.

Solution patch:
diff --git a/.github/workflows/tests.yaml b/.github/workflows/tests.yaml
index 117bba5c6..e59c2865c 100644
--- a/.github/workflows/tests.yaml
+++ b/.github/workflows/tests.yaml
@@ -38,13 +38,16 @@ jobs:
             GOARCH=amd64 PASSES='fmt bom dep' ./test
             ;;
           linux-amd64-integration-1-cpu)
-            GOARCH=amd64 CPU=1 RACE='false' make test-integration
+            make install-gofail
+            GOARCH=amd64 CPU=1 RACE='false' FAILPOINTS='true' make test-integration
             ;;
           linux-amd64-integration-2-cpu)
-            GOARCH=amd64 CPU=2 RACE='false' make test-integration
+            make install-gofail
+            GOARCH=amd64 CPU=2 RACE='false' FAILPOINTS='true' make test-integration
             ;;
           linux-amd64-integration-4-cpu)
-            GOARCH=amd64 CPU=4 RACE='false' make test-integration
+            make install-gofail
+            GOARCH=amd64 CPU=4 RACE='false' FAILPOINTS='true' make test-integration
             ;;
           linux-amd64-functional)
             ./build && GOARCH=amd64 PASSES='functional' ./test
diff --git a/Makefile b/Makefile
index 0af7d1e2d..5e06d3308 100644
--- a/Makefile
+++ b/Makefile
@@ -538,3 +538,12 @@ GOFAIL_VERSION = $(shell cd tools/mod && go list -m -f {{.Version}} go.etcd.io/g
 .PHONY: install-gofail
 install-gofail:
 	go install go.etcd.io/gofail@${GOFAIL_VERSION}
+
+
+.PHONY: gofail-enable
+gofail-enable: install-gofail
+	PASSES="toggle_failpoints" FAILPOINTS=true ./test
+
+.PHONY: gofail-disable
+gofail-disable: install-gofail
+	PASSES="toggle_failpoints" ./test
diff --git a/build b/build
index 3fa8cbeae..7cc3d2f95 100755
--- a/build
+++ b/build
@@ -21,7 +21,7 @@ GOFAIL_VERSION=$(cd tools/mod && go list -m -f "{{.Version}}" go.etcd.io/gofail)
 toggle_failpoints() {
 	mode="$1"
 	if command -v gofail >/dev/null 2>&1; then
-		gofail "$mode" etcdserver/ mvcc/ mvcc/backend/ wal/
+		gofail "$mode" etcdserver/ lease/leasehttp/ mvcc/ mvcc/backend/ wal/
 		# shellcheck disable=SC2086
 		if [[ "$mode" == "enable" ]]; then
 			go get go.etcd.io/gofail@${GOFAIL_VERSION}
diff --git a/clientv3/client.go b/clientv3/client.go
index e51ed6735..dc5dbbb99 100644
--- a/clientv3/client.go
+++ b/clientv3/client.go
@@ -89,7 +89,7 @@ func New(cfg Config) (*Client, error) {
 // service interface implementations and do not need connection management.
 func NewCtxClient(ctx context.Context) *Client {
 	cctx, cancel := context.WithCancel(ctx)
-	return &Client{ctx: cctx, cancel: cancel, lgMu: new(sync.RWMutex), lg: zap.NewNop()}
+	return &Client{ctx: cctx, cancel: cancel, lgMu: new(sync.RWMutex), epMu: new(sync.RWMutex), lg: zap.NewNop()}
 }
 
 // NewFromURL creates a new etcdv3 client from a URL.
diff --git a/etcdserver/v3_server.go b/etcdserver/v3_server.go
index a1040aca6..698a524a8 100644
--- a/etcdserver/v3_server.go
+++ b/etcdserver/v3_server.go
@@ -363,6 +363,9 @@ func (s *EtcdServer) leaseTimeToLive(ctx context.Context, r *pb.LeaseTimeToLiveR
 		if err := s.waitAppliedIndex(); err != nil {
 			return nil, err
 		}
+
+		// gofail: var beforeLookupWhenLeaseTimeToLive struct{}
+
 		// primary; timetolive directly from leader
 		le := s.lessor.Lookup(lease.LeaseID(r.ID))
 		if le == nil {
@@ -378,6 +381,15 @@ func (s *EtcdServer) leaseTimeToLive(ctx context.Context, r *pb.LeaseTimeToLiveR
 			}
 			resp.Keys = kbs
 		}
+
+		// The leasor could be demoted if leader changed during lookup.
+		// We should return error to force retry instead of returning
+		// incorrect remaining TTL.
+		if le.Demoted() {
+			// NOTE: lease.ErrNotPrimary is not retryable error for
+			// client. Instead, uses ErrLeaderChanged.
+			return nil, ErrLeaderChanged
+		}
 		return resp, nil
 	}
 
diff --git a/go.mod b/go.mod
index d3e067177..46b1e87b0 100644
--- a/go.mod
+++ b/go.mod
@@ -33,6 +33,7 @@ require (
 	github.com/urfave/cli v1.20.0
 	github.com/xiang90/probing v0.0.0-20190116061207-43a291ad63a2
 	go.etcd.io/bbolt v1.3.9
+	go.etcd.io/gofail v0.1.0
 	go.uber.org/zap v1.10.0
 	golang.org/x/crypto v0.21.0
 	golang.org/x/net v0.23.0
diff --git a/go.sum b/go.sum
index 1cb942002..26ae45f87 100644
--- a/go.sum
+++ b/go.sum
@@ -177,6 +177,8 @@ github.com/yuin/goldmark v1.1.27/go.mod h1:3hX8gzYuyVAZsxl0MRgGTJEmQBFcNTphYh9de
 github.com/yuin/goldmark v1.2.1/go.mod h1:3hX8gzYuyVAZsxl0MRgGTJEmQBFcNTphYh9decYSb74=
 go.etcd.io/bbolt v1.3.9 h1:8x7aARPEXiXbHmtUwAIv7eV2fQFHrLLavdiJ3uzJXoI=
 go.etcd.io/bbolt v1.3.9/go.mod h1:zaO32+Ti0PK1ivdPtgMESzuzL2VPoIG1PCQNvOdo/dE=
+go.etcd.io/gofail v0.1.0 h1:XItAMIhOojXFQMgrxjnd2EIIHun/d5qL0Pf7FzVTkFg=
+go.etcd.io/gofail v0.1.0/go.mod h1:VZBCXYGZhHAinaBiiqYvuDynvahNsAyLFwB3kEHKz1M=
 go.uber.org/atomic v1.3.2 h1:2Oa65PReHzfn29GpvgsYwloV9AVFHPDk8tYxt2c2tr4=
 go.uber.org/atomic v1.3.2/go.mod h1:gD2HeocX3+yG+ygLZcrzQJaqmWj9AIm7n08wl/qW/PE=
 go.uber.org/multierr v1.1.0 h1:HoEmRHQPVSqub6w2z2d2EOVs2fjyFRGyofhKuyDq0QI=
diff --git a/lease/leasehttp/http.go b/lease/leasehttp/http.go
index e2f5b1ce1..d59868cde 100644
--- a/lease/leasehttp/http.go
+++ b/lease/leasehttp/http.go
@@ -103,6 +103,9 @@ func (h *leaseHandler) ServeHTTP(w http.ResponseWriter, r *http.Request) {
 			http.Error(w, ErrLeaseHTTPTimeout.Error(), http.StatusRequestTimeout)
 			return
 		}
+
+		// gofail: var beforeLookupWhenForwardLeaseTimeToLive struct{}
+
 		l := h.l.Lookup(lease.LeaseID(lreq.LeaseTimeToLiveRequest.ID))
 		if l == nil {
 			http.Error(w, lease.ErrLeaseNotFound.Error(), http.StatusNotFound)
@@ -126,6 +129,14 @@ func (h *leaseHandler) ServeHTTP(w http.ResponseWriter, r *http.Request) {
 			resp.LeaseTimeToLiveResponse.Keys = kbs
 		}
 
+		// The leasor could be demoted if leader changed during lookup.
+		// We should return error to force retry instead of returning
+		// incorrect remaining TTL.
+		if l.Demoted() {
+			http.Error(w, lease.ErrNotPrimary.Error(), http.StatusInternalServerError)
+			return
+		}
+
 		v, err = resp.Marshal()
 		if err != nil {
 			http.Error(w, err.Error(), http.StatusInternalServerError)
diff --git a/lease/lessor.go b/lease/lessor.go
index 02718b94e..2a53be322 100644
--- a/lease/lessor.go
+++ b/lease/lessor.go
@@ -901,6 +901,13 @@ func (l *Lease) forever() {
 	l.expiry = forever
 }
 
+// Demoted returns true if the lease's expiry has been reset to forever.
+func (l *Lease) Demoted() bool {
+	l.expiryMu.Lock()
+	defer l.expiryMu.Unlock()
+	return l.expiry == forever
+}
+
 // Keys returns all the keys attached to the lease.
 func (l *Lease) Keys() []string {
 	l.mu.RLock()
diff --git a/test b/test
index 6a916687c..4cb508696 100755
--- a/test
+++ b/test
@@ -690,6 +690,10 @@ function build_cov_pass {
 	go test -tags cov -c -covermode=set -coverpkg="$PKGS_COMMA" -o "${out}/etcdctl_test" "${REPO_PATH}/etcdctl"
 }
 
+function toggle_failpoints_pass {
+  toggle_failpoints_default
+}
+
 # fail fast on static tests
 function build_pass {
 	echo "Checking build..."



Test patch:
diff --git a/integration/v3_lease_test.go b/integration/v3_lease_test.go
index 7f9742ccd..bd6f01edd 100644
--- a/integration/v3_lease_test.go
+++ b/integration/v3_lease_test.go
@@ -21,10 +21,14 @@ import (
 	"testing"
 	"time"
 
+	"github.com/stretchr/testify/assert"
+	"github.com/stretchr/testify/require"
+	"go.etcd.io/etcd/clientv3"
 	"go.etcd.io/etcd/etcdserver/api/v3rpc/rpctypes"
 	pb "go.etcd.io/etcd/etcdserver/etcdserverpb"
 	"go.etcd.io/etcd/mvcc/mvccpb"
 	"go.etcd.io/etcd/pkg/testutil"
+	gofail "go.etcd.io/gofail/runtime"
 
 	"google.golang.org/grpc/codes"
 	"google.golang.org/grpc/metadata"
@@ -1021,6 +1025,76 @@ func TestV3LeaseRecoverKeyWithMutipleLease(t *testing.T) {
 	}
 }
 
+func TestV3LeaseTimeToLiveWithLeaderChanged(t *testing.T) {
+	t.Run("normal", func(subT *testing.T) {
+		testV3LeaseTimeToLiveWithLeaderChanged(subT, "beforeLookupWhenLeaseTimeToLive")
+	})
+
+	t.Run("forward", func(subT *testing.T) {
+		testV3LeaseTimeToLiveWithLeaderChanged(subT, "beforeLookupWhenForwardLeaseTimeToLive")
+	})
+}
+
+func testV3LeaseTimeToLiveWithLeaderChanged(t *testing.T, fpName string) {
+	if len(gofail.List()) == 0 {
+		t.Skip("please run 'make gofail-enable' before running the test")
+	}
+
+	clus := NewClusterV3(t, &ClusterConfig{Size: 3})
+	defer clus.Terminate(t)
+
+	ctx, cancel := context.WithTimeout(context.Background(), 30*time.Second)
+	defer cancel()
+
+	oldLeadIdx := clus.WaitLeader(t)
+	followerIdx := (oldLeadIdx + 1) % 3
+
+	followerMemberID := clus.Members[followerIdx].ID()
+
+	oldLeadC := clus.Client(oldLeadIdx)
+
+	leaseResp, err := oldLeadC.Grant(ctx, 100)
+	require.NoError(t, err)
+
+	require.NoError(t, gofail.Enable(fpName, `sleep("3s")`))
+	t.Cleanup(func() {
+		terr := gofail.Disable(fpName)
+		if terr != nil && terr != gofail.ErrDisabled {
+			t.Fatalf("failed to disable %s: %v", fpName, terr)
+		}
+	})
+
+	readyCh := make(chan struct{})
+	errCh := make(chan error, 1)
+
+	var targetC *clientv3.Client
+	switch fpName {
+	case "beforeLookupWhenLeaseTimeToLive":
+		targetC = oldLeadC
+	case "beforeLookupWhenForwardLeaseTimeToLive":
+		targetC = clus.Client((oldLeadIdx + 2) % 3)
+	default:
+		t.Fatalf("unsupported %s failpoint", fpName)
+	}
+
+	go func() {
+		<-readyCh
+		time.Sleep(1 * time.Second)
+
+		_, merr := oldLeadC.MoveLeader(ctx, uint64(followerMemberID))
+		assert.NoError(t, gofail.Disable(fpName))
+		errCh <- merr
+	}()
+
+	close(readyCh)
+
+	ttlResp, err := targetC.TimeToLive(ctx, leaseResp.ID)
+	require.NoError(t, err)
+	require.GreaterOrEqual(t, int64(100), ttlResp.TTL)
+
+	require.NoError(t, <-errCh)
+}
+
 // acquireLeaseAndKey creates a new lease and creates an attached key.
 func acquireLeaseAndKey(clus *ClusterV3, key string) (int64, error) {
 	// create lease