Project: Xline (Orchestration & Management / Coordination & Service Discovery)
File: curp.pdf

module curp
extends FiniteSets ,Naturals ,Sequences
Constants:
commands :set of records like [key7→“key” ,value7→“value” ].
replicas :set of replicas .
constants commands ,replicas
assume IsFiniteSet (replicas )
Variables :
leader :records the leader of each epoch .
epoch :current epoch (the number of leader changes ).
proposedCmds :the set of proposed commands .
proposeRequests :the set of propose could be received by each replica .
proposeResponses :the set of responses in each epoch to each proposed
command .
specPools :the speculative pool of each replica .
uncommittedCmds :the sequence of back −end protocol uncommitted commands .
committedCmds :the sequence of back −end protocol committed commands .
commitMsgs :the set of commit messages could be received by each replica .
specExecPrevCmd :the index of the last same −key command in the committed
sequence at the time the leader responds to the proposal .
variables leader ,epoch ,proposedCmds ,proposeRequests ,
proposeResponses ,specPools ,uncommittedCmds ,
committedCmds ,commitMsgs ,specExecPrevCmd
The epoch space .
epoches∆=Nat
Special noLeader value ,for future epoches .
noLeader∆=choose r:r/∈replicas
In N = 2∗f+ 1replicas :
quorum :a set of replicas that contains at least f + 1replicas .
superQuorum :a set of replicas that contains at least f + (f+ 1) /2 + 1
replicas .
recoverQuorum :a set of replicas that contains at least (f+ 1) /2 + 1
replicas .
quorums∆=
let f∆=Cardinality (replicas )÷2
size∆=f+ 1
in{q∈subset replicas :Cardinality (q)≥size}
superQuorums∆=
letf∆=Cardinality (replicas )÷2
size∆=f+ (f+ 1)÷2 + 1
1in{q∈subset replicas :Cardinality (q)≥size}
recoverQuorums∆=
letf∆=Cardinality (replicas )÷2
size∆= (f+ 1)÷2 + 1
in{q∈subset replicas :Cardinality (q)≥size}
Helper function for converting a set to a set containing all sequences
containing the elements of the set exactly once and no other elements .
SetToSeqs (set)∆=
letlen∆= 1 . .Cardinality (set)in
{f∈[len→set] :∀i,j∈len:i̸=j⇒f[i]̸=f[j]}
Helper function for getting the index of the last element in a sequence
satisfying the predicate .
GetIdxInSeq (seq,Pred())∆=
letI∆={i∈1. .Len(seq) :Pred(seq[i])}in
ifI̸={}then choose i∈I:∀j∈I:j≤ielse 0
Propose a command .
This is done by the client sending a proposeRequest to all replicas .
Propose (cmd)∆=
∧proposedCmds′=proposedCmds ∪ {cmd}
∧proposeRequests′=
[r∈replicas 7→proposeRequests [r]∪ {cmd}]
∧unchanged ⟨leader ,epoch ,specPools ,proposeResponses ,
uncommittedCmds ,committedCmds ,commitMsgs ,
specExecPrevCmd ⟩
How the leader process a proposeRequest .
ProcessProposeLeader (r,cmd)∆=
letspecPoolHasConflict∆=
∃specCmd ∈specPools [r] :specCmd .key=cmd .key
uncommittedCmdsHasConflict∆=
GetIdxInSeq (uncommittedCmds ,lambda e:e.key=cmd .key)̸= 0
in
∧proposeRequests′=
[proposeRequests except ![r] = @\{cmd}]
∧specPools′=
[specPools except ![r] =
if¬specPoolHasConflict then @∪ {cmd}else @]
∧uncommittedCmds′=Append (uncommittedCmds ,cmd)
∧proposeResponses′=
[proposeResponses except ![cmd][epoch ] =
if¬specPoolHasConflict ∧ ¬uncommittedCmdsHasConflict
2then @∪ {r}
else @]
∧specExecPrevCmd′=
[specExecPrevCmd except ![cmd] =
if¬specPoolHasConflict ∧ ¬uncommittedCmdsHasConflict
then GetIdxInSeq (committedCmds ,lambda e:e.key=cmd .key)
else @]
∧unchanged ⟨leader ,epoch ,proposedCmds ,committedCmds ,
commitMsgs ⟩
How a non −leader replica process a proposeRequest .
ProcessProposeNonLeader (r,cmd)∆=
letspecPoolHasConflict∆=
∃specCmd ∈specPools [r] :specCmd .key=cmd .key
in
∧proposeRequests′=
[proposeRequests except ![r] = @\{cmd}]
∧specPools′=
[specPools except ![r] =
if¬specPoolHasConflict then @∪ {cmd}else @]
∧proposeResponses′=
[proposeResponses except ![cmd][epoch ] =
if¬specPoolHasConflict then @∪ {r}else @]
∧unchanged ⟨leader ,epoch ,proposedCmds ,uncommittedCmds ,
committedCmds ,commitMsgs ,specExecPrevCmd ⟩
Syncing a command using the back −end protocol (Raft).The implementation details
are omitted .
A replica may not be able to receive the commit message at the exact time the
the leader sends it .
Commit∆=
∧committedCmds′=Append (committedCmds ,Head (uncommittedCmds ))
∧commitMsgs′=
[r∈replicas 7→commitMsgs [r]∪ {Head (uncommittedCmds )}]
∧uncommittedCmds′=Tail(uncommittedCmds )
∧unchanged ⟨leader ,epoch ,specPools ,proposedCmds ,proposeRequests ,
proposeResponses ,specExecPrevCmd ⟩
How a replica process a commit message .
ProcessCommitMsg (r,cmd)∆=
∧commitMsgs′=
[commitMsgs except ![r] = @\{cmd}]
∧specPools′= [specPools except ![r] = @\{cmd}]
∧unchanged ⟨leader ,epoch ,proposedCmds ,proposeRequests ,
proposeResponses ,uncommittedCmds ,committedCmds ,
3specExecPrevCmd ⟩
Leader Change Action
The new leader should gather at least a quorum of replicas specPool to recover the commands.
Commands existed in the specPool of aRecoverQuorum of replicas need to be recovered.
LeaderChange (l)∆=
∧leader′= [e∈epoches 7→ife=epoch + 1then lelse leader [e]]
∧epoch′=epoch + 1
∧ ∃q∈quorums :
letspecCmds∆=union {specPools [r] :r∈q}
newSpecPool∆=
{cmd∈specCmds :
{r∈q:cmd∈specPools [r]} ∈recoverQuorums }
in
∧specPools′= [specPools except ![l] =newSpecPool ]
∧uncommittedCmds′∈SetToSeqs (newSpecPool )
∧unchanged ⟨proposedCmds ,proposeRequests ,proposeResponses ,
committedCmds ,commitMsgs ,specExecPrevCmd ⟩
The initial state of the system.
Init∆=
∃r∈replicas :
letinitEpoch∆= 1initLeader∆=rin
∧leader = [e∈epoches 7→
ife=initEpoch then initLeader else noLeader ]
∧epoch =initEpoch
∧proposedCmds ={}
∧proposeRequests = [replica ∈replicas 7→ {} ]
∧proposeResponses =
[cmd∈commands 7→[e∈epoches 7→ {} ]]
∧specPools = [replica ∈replicas 7→ {} ]
∧uncommittedCmds =⟨⟩
∧committedCmds =⟨⟩
∧commitMsgs = [replica ∈replicas 7→ {} ]
∧specExecPrevCmd = [cmd∈commands 7→0]
Next∆=
∨ ∃cmd∈(commands \proposedCmds ) :Propose (cmd)
∨ ∃r∈replicas :∃cmd∈proposeRequests [r] :
ifleader [epoch ] =r
then ProcessProposeLeader (r,cmd)
else ProcessProposeNonLeader (r,cmd)
∨uncommittedCmds ̸=⟨⟩ ∧Commit
∨ ∃r∈replicas :∃cmd∈commitMsgs [r] :ProcessCommitMsg (r,cmd)
∨ ∃l∈replicas :LeaderChange (l)
4Spec∆=Init∧2[Next]⟨leader ,epoch ,specPools ,proposedCmds ,
proposeRequests ,proposeResponses ,
uncommittedCmds ,committedCmds ,commitMsgs ,
specExecPrevCmd ⟩
Type Invariants
TypeOK∆=
∧leader ∈[epoches →(replicas ∪ {noLeader })]
∧epoch ∈epoches
∧proposedCmds ⊆commands
∧proposeRequests ∈[replicas →subset commands ]
∧proposeResponses ∈[commands →[epoches →subset replicas ]]
∧specPools ∈[replicas →subset commands ]
∧uncommittedCmds ∈union {SetToSeqs (s) :s∈subset commands }
∧committedCmds ∈union {SetToSeqs (s) :s∈subset commands }
∧commitMsgs ∈[replicas →subset commands ]
∧specExecPrevCmd ∈[commands →0. .Cardinality (commands )]
Stability Property
This is the key property of CURP :
1. If a command is committed by CURP , command will eventually be synced by the back-end
protocol.
2. If a command is committed by CURP , when the command is synced be the back-end protocol,
there will never be a command with the same key between the command and the recorded
previous same-key command in the synced sequence.
Stability∆=
∀cmd∈commands :∀e∈epoches :
(∧leader [e]∈proposeResponses [cmd][e]
∧proposeResponses [cmd][e]∈superQuorums )⇒
letidx∆=GetIdxInSeq (committedCmds ,lambda t:t=cmd)
prevExecCmds∆=SubSeq (committedCmds ,1,idx)
in
∧idx̸= 0
∧GetIdxInSeq (prevExecCmds ,lambda t:t.key=cmd .key) =
specExecPrevCmd [cmd]
theorem Spec⇒2TypeOK ∧3Stability
5