Repository: hashicorp/raft (language: go)

Issue:
DATA RACE between runCandidate() and appendEntries()
Got this race with latest v1.3.9 release:
```
==================
WARNING: DATA RACE
Write at 0x00c0000b6f08 by goroutine 257:
  github.com/hashicorp/raft.(*Raft).runCandidate.func1()
      /Users/ivan/dev/go/pkg/mod/github.com/hashicorp/raft@v1.3.9/raft.go:293 +0x38
  runtime.deferreturn()
      /usr/local/go/src/runtime/panic.go:436 +0x34
  github.com/hashicorp/raft.(*Raft).run()
      /Users/ivan/dev/go/pkg/mod/github.com/hashicorp/raft@v1.3.9/raft.go:146 +0x84
  github.com/hashicorp/raft.(*Raft).run-fm()
      <autogenerated>:1 +0x38
  github.com/hashicorp/raft.(*raftState).goFunc.func1()
      /Users/ivan/dev/go/pkg/mod/github.com/hashicorp/raft@v1.3.9/state.go:146 +0x74

Previous read at 0x00c0000b6f08 by goroutine 381:
  github.com/hashicorp/raft.(*Raft).appendEntries()
      /Users/ivan/dev/go/pkg/mod/github.com/hashicorp/raft@v1.3.9/raft.go:1384 +0x270
  github.com/hashicorp/raft.(*Raft).processHeartbeat()
      /Users/ivan/dev/go/pkg/mod/github.com/hashicorp/raft@v1.3.9/raft.go:1353 +0x10c
  github.com/hashicorp/raft.(*Raft).processHeartbeat-fm()
      <autogenerated>:1 +0x7c
  github.com/hashicorp/raft.(*NetworkTransport).handleCommand()
      /Users/ivan/dev/go/pkg/mod/github.com/hashicorp/raft@v1.3.9/net_transport.go:640 +0x9cc
  github.com/hashicorp/raft.(*NetworkTransport).handleConn()
      /Users/ivan/dev/go/pkg/mod/github.com/hashicorp/raft@v1.3.9/net_transport.go:542 +0x438
  github.com/hashicorp/raft.(*NetworkTransport).listen.func1()
      /Users/ivan/dev/go/pkg/mod/github.com/hashicorp/raft@v1.3.9/net_transport.go:520 +0x68
```

Looks like the runCandidate() routine on exit set it: https://github.com/hashicorp/raft/blob/c3134d73a1508e66f9cfb5918b2b71849b18438f/raft.go#L293
and it is being read here: https://github.com/hashicorp/raft/blob/c3134d73a1508e66f9cfb5918b2b71849b18438f/raft.go#L1384




Discussion / hints:
@kozlovic How did you start with `-race` ?
What k/v storage are you using?

Why am I asking: https://github.com/hashicorp/raft/issues/521
@kmlebedev I got this race when running the test suite (with -race) of the NATS Streaming Server project (https://github.com/nats-io/nats-streaming-server). Specifically the clustering tests: `go test -race -v -run=TestClustering ./server -count=1`
> @kmlebedev I got this race when running the test suite (with -race) of the NATS Streaming Server project (https://github.com/nats-io/nats-streaming-server). Specifically the clustering tests: `go test -race -v -run=TestClustering ./server -count=1`

Thanks, I think I found a way to migrate to bbolt
I got the same race in v1.4.0
```
==================
WARNING: DATA RACE
Read at 0x00c0011aa148 by goroutine 6863:
  github.com/hashicorp/raft.(*Raft).appendEntries()
      /go/pkg/mod/github.com/hashicorp/raft@v1.4.0/raft.go:1395 +0x3b8
  github.com/hashicorp/raft.(*Raft).processHeartbeat()
      /go/pkg/mod/github.com/hashicorp/raft@v1.4.0/raft.go:1364 +0x1a4
  github.com/hashicorp/raft.(*Raft).processHeartbeat-fm()
      <autogenerated>:1 +0xa4
  github.com/hashicorp/raft.(*NetworkTransport).handleCommand()
      /go/pkg/mod/github.com/hashicorp/raft@v1.4.0/net_transport.go:639 +0xcda
  github.com/hashicorp/raft.(*NetworkTransport).handleConn()
      /go/pkg/mod/github.com/hashicorp/raft@v1.4.0/net_transport.go:541 +0x5a4
  github.com/hashicorp/raft.(*NetworkTransport).listen.func1()
      /go/pkg/mod/github.com/hashicorp/raft@v1.4.0/net_transport.go:519 +0x71

Previous write at 0x00c0011aa148 by goroutine 435:
  github.com/hashicorp/raft.(*Raft).timeoutNow()
      /go/pkg/mod/github.com/hashicorp/raft@v1.4.0/raft.go:1980 +0x7e
  github.com/hashicorp/raft.(*Raft).processRPC()
      /go/pkg/mod/github.com/hashicorp/raft@v1.4.0/raft.go:1340 +0x17d
  github.com/hashicorp/raft.(*Raft).runFollower()
      /go/pkg/mod/github.com/hashicorp/raft@v1.4.0/raft.go:170 +0xbe4
  github.com/hashicorp/raft.(*Raft).run()
      /go/pkg/mod/github.com/hashicorp/raft@v1.4.0/raft.go:147 +0x97
  github.com/hashicorp/raft.(*Raft).run-fm()
      <autogenerated>:1 +0x39
  github.com/hashicorp/raft.(*raftState).goFunc.func1()
      /go/pkg/mod/github.com/hashicorp/raft@v1.4.0/state.go:149 +0x77

Goroutine 6863 (running) created at:
  github.com/hashicorp/raft.(*NetworkTransport).listen()
      /go/pkg/mod/github.com/hashicorp/raft@v1.4.0/net_transport.go:519 +0x60e
  github.com/hashicorp/raft.NewNetworkTransportWithConfig.func1()
      /go/pkg/mod/github.com/hashicorp/raft@v1.4.0/net_transport.go:179 +0x39

Goroutine 435 (running) created at:
  github.com/hashicorp/raft.(*raftState).goFunc()
      /go/pkg/mod/github.com/hashicorp/raft@v1.4.0/state.go:147 +0xee
  github.com/hashicorp/raft.NewRaft()
      /go/pkg/mod/github.com/hashicorp/raft@v1.4.0/api.go:604 +0x188b
  github.com/arzahs/lib/pkg/consensus.(*RAFT).reconfigureRaft()
      /root/project/pkg/consensus/raft.go:374 +0x12de
  github.com/arzahs/lib/pkg/consensus.NewRAFT()
      /root/project/pkg/consensus/raft.go:408 +0x373
  github.com/arzahs/lib/pkg/consensus.(*Consensus).Activate()
      /root/project/pkg/consensus/consensus.go:544 +0x184
==================
```

Looks like it's related to simultaneous read/write in `r.candidateFromLeadershipTransfer = true` 

Fix patch:
diff --git a/api.go b/api.go
index 25f4dd43..a3c1826f 100644
--- a/api.go
+++ b/api.go
@@ -134,7 +134,7 @@ type Raft struct {
 	// candidate because the leader tries to transfer leadership. This flag is
 	// used in RequestVoteRequest to express that a leadership transfer is going
 	// on.
-	candidateFromLeadershipTransfer bool
+	candidateFromLeadershipTransfer atomic.Bool
 
 	// Stores our local server ID, used to avoid sending RPCs to ourself
 	localID ServerID
diff --git a/raft.go b/raft.go
index 98fe7044..4fcb95f7 100644
--- a/raft.go
+++ b/raft.go
@@ -293,7 +293,7 @@ func (r *Raft) runCandidate() {
 	// which will make other servers vote even though they have a leader already.
 	// It is important to reset that flag, because this priviledge could be abused
 	// otherwise.
-	defer func() { r.candidateFromLeadershipTransfer = false }()
+	defer func() { r.candidateFromLeadershipTransfer.Store(false) }()
 
 	electionTimeout := r.config().ElectionTimeout
 	electionTimer := randomTimeout(electionTimeout)
@@ -1390,7 +1390,7 @@ func (r *Raft) appendEntries(rpc RPC, a *AppendEntriesRequest) {
 
 	// Increase the term if we see a newer one, also transition to follower
 	// if we ever get an appendEntries call
-	if a.Term > r.getCurrentTerm() || (r.getState() != Follower && !r.candidateFromLeadershipTransfer) {
+	if a.Term > r.getCurrentTerm() || (r.getState() != Follower && !r.candidateFromLeadershipTransfer.Load()) {
 		// Ensure transition to follower
 		r.setState(Follower)
 		r.setCurrentTerm(a.Term)
@@ -1839,7 +1839,7 @@ func (r *Raft) electSelf() <-chan *voteResult {
 		Candidate:          r.trans.EncodePeer(r.localID, r.localAddr),
 		LastLogIndex:       lastIdx,
 		LastLogTerm:        lastTerm,
-		LeadershipTransfer: r.candidateFromLeadershipTransfer,
+		LeadershipTransfer: r.candidateFromLeadershipTransfer.Load(),
 	}
 
 	// Construct a function to ask for a vote
@@ -1972,7 +1972,7 @@ func (r *Raft) initiateLeadershipTransfer(id *ServerID, address *ServerAddress)
 func (r *Raft) timeoutNow(rpc RPC, req *TimeoutNowRequest) {
 	r.setLeader("", "")
 	r.setState(Candidate)
-	r.candidateFromLeadershipTransfer = true
+	r.candidateFromLeadershipTransfer.Store(true)
 	rpc.Respond(&TimeoutNowResponse{}, nil)
 }
 


Test patch: