Project: Nomad (Orchestration & Management / Scheduling & Orchestration)
File: architecture-state-store.md

Nomad server state is an in-memory state store backed by raft. All writes to state are serialized into message pack and written as raft logs. The raft logs are replicated from the leader to the followers. Once each follower has persisted the log entry and applied the entry to its in-memory state ("FSM"), the leader considers the write committed. This architecture has a few implications: The `fsm.Apply` functions must be deterministic over their inputs for a given state. You can never generate random IDs or assign wall-clock timestamps in the state store. These values must be provided as parameters from the RPC handler. ```go func (s *StateStore) UpsertObject(...) { obj.CreateTime = time.Now() } func (s *StateStore) UpsertObject(..., timestamp time.Time) { obj.CreateTime = timestamp } ``` Every object you read from the state store must be copied before it can be mutated, because mutating the object modifies it outside the raft workflow. The result can be servers having inconsistent state, transactions breaking, or even server panics. ```go job, err := state.JobByID(ws, namespace, id) job.Status = structs.JobStatusRunning job, err := state.JobByID(ws, namespace, id) updateJob := job.Copy() updateJob.Status = structs.JobStatusRunning ``` Adding new objects to the state store should be done as part of adding new RPC endpoints. See the . ```mermaid flowchart TD %% entities ext(("API\nclient")) any("Any node (client or server)") follower(Follower) rpcLeader("RPC handler (on leader)") writes("writes go thru raft raftApply(MessageType, entry) in nomad/rpc.go structs.MessageType in nomad/structs/structs.go go generate ./... for nomad/msgtypes.go") click writes href "https://github.com/hashicorp/nomad/tree/main/nomad" _blank reads("reads go directly to state store Typical state_store.go funcs to implement: state.GetMyThingByID state.GetMyThingByPrefix state.ListMyThing state.UpsertMyThing state.DeleteMyThing") click writes href "https://github.com/hashicorp/nomad/tree/main/nomad/state" _blank raft("hashicorp/raft") bolt("boltdb") fsm("Application-specific Finite State Machine (FSM) (aka State Store)") click writes href "https://github.com/hashicorp/nomad/tree/main/nomad/fsm.go" _blank memdb("hashicorp/go-memdb") %% style classes classDef leader fill:#d5f6ea,stroke-width:4px,stroke:#1d9467 classDef other fill:#d5f6ea,stroke:#1d9467 class any,follower other; class rpcLeader,raft,bolt,fsm,memdb leader; %% flows ext -- HTTP request --> any any -- "RPC request to connected server (follower or leader)" --> follower follower -- "(1) srv.Forward (to leader)" --> rpcLeader raft -- "(3) replicate to a quorum of followers wait on their fsm.Apply" --> follower rpcLeader --> reads reads --> memdb rpcLeader --> writes writes -- "(2)" --> raft raft -- "(4) write log to disk" --> bolt raft -- "(5) fsm.Apply nomad/fsm.go" --> fsm fsm -- "(6) txn.Insert" --> memdb bolt <-- "Snapshot Persist: nomad/fsm.go Snapshot Restore: nomad/fsm.go" --> memdb %% notes note1("Typical structs to implement for RPC handlers: structs.MyThing .Diff() .Copy() .Merge() structs.MyThingUpsertRequest structs.MyThingUpsertResponse structs.MyThingGetRequest structs.MyThingGetResponse structs.MyThingListRequest structs.MyThingListResponse structs.MyThingDeleteRequest structs.MyThingDeleteResponse Don't forget to register your new RPC in nomad/server.go!") note1 -.- rpcLeader ```