Repository: sofastack/sofa-jraft (Java)

Issue:
Seems that readIndex() may return inconsistent value (e.g. 0) after node restart, is this a known issue?
Within node restart `lastCommittedIndex` initialized as 0
```
public class BallotBox implements Lifecycle<BallotBoxOptions>, Describer {
    ...
    private long lastCommittedIndex = 0;
    ...
}
```
Taking into the consideration that local log application triggered by `NodeImpl#init ` is asynchronous, it's possible to handle readIndex request before local log re-play is finished and thus see inconsistent readIndex value. In other words, seems that there's a race between readIndex evaluation and lastCommittedIndex restoration on node restart.


Discussion: I think it's impossible, the `handleReadIndexRequest` will check the state of the node before reading the `lastCommittedIndex`, and if its state is not the leader or the current leader doesn't commit an entry before, the `readIndex` will fail.

https://github.com/sofastack/sofa-jraft/blob/2ea82d6d0605af737af59c853a3c00e643b0384e/jraft-core/src/main/java/com/alipay/sofa/jraft/core/NodeImpl.java#L1511

https://github.com/sofastack/sofa-jraft/blob/2ea82d6d0605af737af59c853a3c00e643b0384e/jraft-core/src/main/java/com/alipay/sofa/jraft/core/NodeImpl.java#L1566

And when a node becomes a leader, it will update the `lastCommittedIndex` before releasing the write lock:

https://github.com/sofastack/sofa-jraft/blob/2ea82d6d0605af737af59c853a3c00e643b0384e/jraft-core/src/main/java/com/alipay/sofa/jraft/core/NodeImpl.java#L1264

https://github.com/sofastack/sofa-jraft/blob/2ea82d6d0605af737af59c853a3c00e643b0384e/jraft-core/src/main/java/com/alipay/sofa/jraft/core/NodeImpl.java#L502




Hmm, seems that `if (this.logManager.getTerm(lastCommittedIndex) != this.currTerm) ` will be skipped in case of `quorum <= 1` and this is the case when the issue is reproduced.
> Hmm, seems that `if (this.logManager.getTerm(lastCommittedIndex) != this.currTerm) ` will be skipped in case of `quorum <= 1` and this is the case when the issue is reproduced.

If so, that would be possible, good catch! But I really doubt if is this a common case(just one node) for the production usage of jraft.

We can fix it, but maybe it is not urgent. 
> We can fix it

Well, that'll be nice))

BTW, seems that it's not the difficult to fix it. In case of single noded cluster (and only single noded cluster). We may consider `pendingIndex` on restart as committed one, just because in case of single node no-one will discard log entries. 
So basically, it should be possible to do following to solve the issue. _In case of single noded clusters only._
```
    public boolean resetPendingIndex(final long newPendingIndex) {
            ...
            this.lastCommittedIndex = newPendingIndex - 1;
            ...
        }
```
What do you think?
> > We can fix it
> 
> Well, that'll be nice))
> 
> BTW, seems that it's not the difficult to fix it. In case of single noded cluster (and only single noded cluster). We may consider `pendingIndex` on restart as committed one, just because in case of single node no-one will discard log entries. So basically, it should be possible to do following to solve the issue. _In case of single noded clusters only._
> 
> ```
>     public boolean resetPendingIndex(final long newPendingIndex) {
>             ...
>             this.lastCommittedIndex = newPendingIndex - 1;
>             ...
>         }
> ```
> 
> What do you think?

It seems like the fix will work as expected. Would you be able to create a pull request for this change?


Sure



Fix (patch):
diff --git a/jraft-core/src/main/java/com/alipay/sofa/jraft/core/BallotBox.java b/jraft-core/src/main/java/com/alipay/sofa/jraft/core/BallotBox.java
index d24223beb..34ac827ec 100644
--- a/jraft-core/src/main/java/com/alipay/sofa/jraft/core/BallotBox.java
+++ b/jraft-core/src/main/java/com/alipay/sofa/jraft/core/BallotBox.java
@@ -88,6 +88,7 @@ public boolean init(final BallotBoxOptions opts) {
         this.opts = opts;
         this.waiter = opts.getWaiter();
         this.closureQueue = opts.getClosureQueue();
+        this.lastCommittedIndex = opts.getLastCommittedIndex();
         return true;
     }
 
@@ -165,6 +166,7 @@ public void clearPendingTasks() {
      * committed until a log at the new term becomes committed, so
      * |newPendingIndex| should be |last_log_index| + 1.
      * @param newPendingIndex pending index of new leader
+     *
      * @return returns true if reset success
      */
     public boolean resetPendingIndex(final long newPendingIndex) {
@@ -180,6 +182,7 @@ public boolean resetPendingIndex(final long newPendingIndex) {
                     this.opts.getNodeId(), newPendingIndex, this.lastCommittedIndex);
                 return false;
             }
+
             this.pendingIndex = newPendingIndex;
             this.closureQueue.resetFirstIndex(newPendingIndex);
             return true;
diff --git a/jraft-core/src/main/java/com/alipay/sofa/jraft/core/NodeImpl.java b/jraft-core/src/main/java/com/alipay/sofa/jraft/core/NodeImpl.java
index ae1c0df4a..25b8c4774 100644
--- a/jraft-core/src/main/java/com/alipay/sofa/jraft/core/NodeImpl.java
+++ b/jraft-core/src/main/java/com/alipay/sofa/jraft/core/NodeImpl.java
@@ -1013,15 +1013,6 @@ protected int adjustTimeout(final int timeoutMs) {
             LOG.error("Node {} initFSMCaller failed.", getNodeId());
             return false;
         }
-        this.ballotBox = new BallotBox();
-        final BallotBoxOptions ballotBoxOpts = new BallotBoxOptions();
-        ballotBoxOpts.setWaiter(this.fsmCaller);
-        ballotBoxOpts.setClosureQueue(this.closureQueue);
-        ballotBoxOpts.setNodeId(getNodeId());
-        if (!this.ballotBox.init(ballotBoxOpts)) {
-            LOG.error("Node {} init ballotBox failed.", getNodeId());
-            return false;
-        }
 
         if (!initSnapshotStorage()) {
             LOG.error("Node {} initSnapshotStorage failed.", getNodeId());
@@ -1044,6 +1035,12 @@ protected int adjustTimeout(final int timeoutMs) {
             this.targetPriority = getMaxPriorityOfNodes(this.conf.getConf().getPeers());
         }
 
+        // It must be initialized after initializing conf and log storage.
+        if (!initBallotBox()) {
+            LOG.error("Node {} init ballotBox failed.", getNodeId());
+            return false;
+        }
+
         if (!this.conf.isEmpty()) {
             Requires.requireTrue(this.conf.isValid(), "Invalid conf: %s", this.conf);
         } else {
@@ -1120,6 +1117,29 @@ protected int adjustTimeout(final int timeoutMs) {
         return true;
     }
 
+    private boolean initBallotBox() {
+        this.ballotBox = new BallotBox();
+        final BallotBoxOptions ballotBoxOpts = new BallotBoxOptions();
+        ballotBoxOpts.setWaiter(this.fsmCaller);
+        ballotBoxOpts.setClosureQueue(this.closureQueue);
+        ballotBoxOpts.setNodeId(getNodeId());
+        // Try to initialize the last committed index in BallotBox to be the last snapshot index.
+        long lastCommittedIndex = 0;
+        if (this.snapshotExecutor != null) {
+            lastCommittedIndex = this.snapshotExecutor.getLastSnapshotIndex();
+        }
+        if (this.getQuorum() == 1) {
+            // It is safe to initiate lastCommittedIndex as last log one because in case of single peer no one will discard
+            // log records on leader election.
+            // Fix https://github.com/sofastack/sofa-jraft/issues/1049
+            lastCommittedIndex = Math.max(lastCommittedIndex, this.logManager.getLastLogIndex());
+        }
+
+        ballotBoxOpts.setLastCommittedIndex(lastCommittedIndex);
+        LOG.info("Node {} init ballot box's lastCommittedIndex={}.", getNodeId(), lastCommittedIndex);
+        return this.ballotBox.init(ballotBoxOpts);
+    }
+
     @OnlyForTest
     void tryElectSelf() {
         this.writeLock.lock();
@@ -1127,6 +1147,11 @@ void tryElectSelf() {
         electSelf();
     }
 
+    @OnlyForTest
+    BallotBox getBallotBox() {
+        return this.ballotBox;
+    }
+
     // should be in writeLock
     private void electSelf() {
         long oldTerm;
@@ -1360,6 +1385,18 @@ public void run(final Status status) {
     }
 
     private void executeApplyingTasks(final List<LogEntryAndClosure> tasks) {
+	if (!this.logManager.hasAvailableCapacityToAppendEntries(1)) {
+		// It's overload, fail-fast
+		final List<Closure> dones = tasks.stream().map(ele -> ele.done).filter(Objects::nonNull)
+				.collect(Collectors.toList());
+		ThreadPoolsFactory.runInThread(this.groupId, () -> {
+			for (final Closure done : dones) {
+				done.run(new Status(RaftError.EBUSY, "Node %s log manager is busy.", this.getNodeId()));
+			}
+		});
+		return;
+	}
+
         this.writeLock.lock();
         try {
             final int size = tasks.size();
diff --git a/jraft-core/src/main/java/com/alipay/sofa/jraft/option/BallotBoxOptions.java b/jraft-core/src/main/java/com/alipay/sofa/jraft/option/BallotBoxOptions.java
index 10e7e5c07..309f349d7 100644
--- a/jraft-core/src/main/java/com/alipay/sofa/jraft/option/BallotBoxOptions.java
+++ b/jraft-core/src/main/java/com/alipay/sofa/jraft/option/BallotBoxOptions.java
@@ -32,11 +32,20 @@ public class BallotBoxOptions {
     private FSMCaller    waiter;
     private ClosureQueue closureQueue;
     private NodeId       nodeId;
+    private long         lastCommittedIndex;
 
     public NodeId getNodeId() {
         return nodeId;
     }
 
+    public long getLastCommittedIndex() {
+        return lastCommittedIndex;
+    }
+
+    public void setLastCommittedIndex(long lastCommittedIndex) {
+        this.lastCommittedIndex = lastCommittedIndex;
+    }
+
     public void setNodeId(NodeId nodeId) {
         this.nodeId = nodeId;
     }
diff --git a/jraft-core/src/main/java/com/alipay/sofa/jraft/storage/SnapshotExecutor.java b/jraft-core/src/main/java/com/alipay/sofa/jraft/storage/SnapshotExecutor.java
index 2e5369163..938456c42 100644
--- a/jraft-core/src/main/java/com/alipay/sofa/jraft/storage/SnapshotExecutor.java
+++ b/jraft-core/src/main/java/com/alipay/sofa/jraft/storage/SnapshotExecutor.java
@@ -48,6 +48,12 @@ public interface SnapshotExecutor extends Lifecycle<SnapshotExecutorOptions>, De
      */
     void doSnapshot(final Closure done);
 
+    /**
+     * Returns the last snapshot index.
+     * @return
+     */
+    long getLastSnapshotIndex();
+
     /**
      * Start to snapshot StateMachine immediately with the latest log applied to state machine.
      * You MUST call this method in {@link StateMachine} callback methods to trigger a snapshot synchronously, otherwise throws {@link IllegalStateException}.
diff --git a/jraft-core/src/main/java/com/alipay/sofa/jraft/storage/snapshot/SnapshotExecutorImpl.java b/jraft-core/src/main/java/com/alipay/sofa/jraft/storage/snapshot/SnapshotExecutorImpl.java
index af5218d28..6b5e0d09a 100644
--- a/jraft-core/src/main/java/com/alipay/sofa/jraft/storage/snapshot/SnapshotExecutorImpl.java
+++ b/jraft-core/src/main/java/com/alipay/sofa/jraft/storage/snapshot/SnapshotExecutorImpl.java
@@ -112,10 +112,6 @@ public long getLastSnapshotTerm() {
         return this.lastSnapshotTerm;
     }
 
-    /**
-     * Only for test
-     */
-    @OnlyForTest
     public long getLastSnapshotIndex() {
         return this.lastSnapshotIndex;
     }
diff --git a/jraft-extension/java-log-storage-impl/src/main/java/com/alipay/sofa/jraft/storage/HybridLogStorage.java b/jraft-extension/java-log-storage-impl/src/main/java/com/alipay/sofa/jraft/storage/HybridLogStorage.java
index 4f9d7c176..2be22f4ed 100644
--- a/jraft-extension/java-log-storage-impl/src/main/java/com/alipay/sofa/jraft/storage/HybridLogStorage.java
+++ b/jraft-extension/java-log-storage-impl/src/main/java/com/alipay/sofa/jraft/storage/HybridLogStorage.java
@@ -16,10 +16,12 @@
  */
 package com.alipay.sofa.jraft.storage;
 
+import java.io.File;
 import java.io.IOException;
 import java.nio.file.Paths;
 import java.util.List;
 
+import org.apache.commons.io.FileUtils;
 import org.slf4j.Logger;
 import org.slf4j.LoggerFactory;
 
@@ -48,6 +50,11 @@ public class HybridLogStorage implements LogStorage {
     private long                                thresholdIndex;
 
     public HybridLogStorage(final String path, final StoreOptions storeOptions, final LogStorage oldStorage) {
+        try {
+            FileUtils.forceMkdir(new File(path));
+        } catch (IOException e) {
+            LOG.error("Failed to create directory at path: {}", path, e);
+        }
         final String newLogStoragePath = Paths.get(path, storeOptions.getStoragePath()).toString();
         this.newLogStorage = new LogitLogStorage(newLogStoragePath, storeOptions);
         this.oldLogStorage = oldStorage;