FOROS
Failover ROS Framework
Wonguk Jeong ( wonguk.jeong@42dot.ai )github.com/42dot/foros

Overview

github.com/42dot/foros

Motivation
Reliability and Safety are essential for autonomous vehicles and robots.

github.com/42dot/foros

How?
Mitigating Single Points of Failure(SPOFs)
FailedControl

Control

Planner

Planner

Perception

Perception

SensorSensor
System Health Monitor

System Health Monitor

FailedSPoF

github.com/42dot/foros

So, How?
Enhancing high availability of safety - critical modules using redundancy ( = clustering)Control

Control

Planner

Planner

Perception

PerceptionSensor

SensorActiveSystem Health Monitor

System Health Monitor

FailedStandby

System Health Monitor

System Health Monitor

ActiveStandby

System Health Monitor

System Health Monitor

Standbygithub.com/42dot/foros

FOROS

github.com/42dot/foros

Failover ROS Framework
An open source ROS2 framework that can be used to provide
critical nodes using a RAFT consensus algorithm

redundancy for safety -with minimal effort.

Fail Operational

github.com/42dot/foros

Constraint
This framework can tolerate failures equal to thecluster size minus quorum

Cluster Size(N)

Quorum(Q = N / 2 + 1)

Number of fault tolerant nodes(N - Q)

110220321

43

1532.

This framework tolerates fail - stop failure but NOT Byzantine failure•Fail- stop failure : the component stops operating.

•Byzantine failure: there is imperfect information on whether a component has failed
github.com/42dot/foros

Core Features● Leader election: determination of active nodes by election● Log replication: consensus - based data storage. Mainly used for state replication● Inspector : a tool for monitoring the status of clusters.
Leader election

Leader election

A1A2A3

A1A2

A3

ActiveStandbyStandby

FailedActiveStandby

StateStateStateStateStateState

Log replication = State replication

State restoration
github.com/42dot/foros

Leader Election
All nodes have one of the following states:'Follower', 'Candidate', or 'Leader'Election timeout, starts election

Follower

Candidate

Receives votes frommajority

Leader

Discovers new leader<State Machine>
github.com/42dot/foros

Leader Election
All nodes start in ‘ Follower’ state

A1

A2

A3

<Follower><Follower><Follower>

github.com/42dot/foros

Leader Election
If a ‘Follower’ does not receive a ‘Leader’ heartbeat for a certain period of time, it ischanged to ‘Candidate’ and an election is held
1. Election timeout, starts election

A1

A2

A3

<Candidate><Follower><Follower>

2. Vote for me

3. Request a vote

github.com/42dot/foros

Leader Election
When a ‘Candidate’ receives a majority of the votes, it becomes the‘Leader’ .

2. Received votes from majority

A1

A2

A3

<Leader><Follower><Follower>

1. Get a vote
github.com/42dot/foros

Leader Election
The 'Leader' periodically sends heartbeats to prevent elections for new leaders.

A1A2

A3

<Leader><Follower><Follower>

1. Send heartbeat

github.com/42dot/foros

Leader Election
The complex leader election process is all handled within the FOROS framework.
Developers only need to consider ‘ Standby ’ and ‘Active ’ states.
Transitioned to ‘Leader’ state

Filter the messages below :- Published topics

StandbyActive- Received service requests

Transitioned to ‘Follower’ state

github.com/42dot/foros

Leader Election : How to Use
Simple! Use ClusterNode c la s s ins te a d of rclcpp ::Node .
auto node = akit::failover::foros::ClusterNode::make_shared("Test_cluster",// Cluster Name

std::initializer_list<uint32_t>{0, 1, 2}// Node IDs in the given cluster0,);// Node ID

Re g is te r s ta te tra ns ition c a llb a c ks us ing register_on_activated, register_on_standby

node->register_on_activated([&]() { RCLCPP_INFO(logger, "activated"); });node->register_on_standby([&]() { RCLCPP_INFO(logger, "standby"); });github.com/42dot/foros

Log Replication
When the ‘Leader’ requests to store data, it requests data synchronization from othernodes and succeeds when more than half of the nodes are
Request to store

synchronized

Synchronization

StoredA1<Leader>

A2<Follower>

A3<Failed>github.com/42dot/foros

Log Replication : How to Use
Use commit_command

to re q ue s t to s tore d a ta

node->commit_command(akit::failover::foros::Command::make_shared(std::initializer_list<uint8_t>{1}),[&](akit::failover::foros::CommandCommitResponseSharedFuture
response_future) {auto response = response_future.get();if (response->result() == true) {RCLCPP_INFO(logger, "commit completed");} else {}RCLCPP_ERROR(logger, "commit failed");});github.com/42dot/foros

Log Replication : How to Use
Use get_commands_size , get_command to g e t s tore d d a ta .
int len = node->get_commands_size();
auto command = get_command(len - 1);

Us e register_on_committed, register_on_reverted

to re g is te r c ommit/re ve rt c a llb a c k.

node->register_on_committed([&](int64_t id, akit::failover::foros::Command::SharedPtr command) {RCLCPP_INFO(logger, "command commited : %ld, %d", id, command->data()[0]);});node->register_on_reverted([&](int64_t id) {RCLCPP_INFO(logger, "command reverted until : %ld", id);});

github.com/42dot/foros

Inspector
Visualize active cluster information and node information in the cluster with TUISummary : Cluster Information
Name

Cluster Name

Size

Cluster Size

Term

Election No.

ActiveActive Node IDs

LeaderLeader ID

Details : Node Information
Node ID

Node ID

StateState

Term

Election No.

Voted ForID

Data SizeStored Data Size

Size

Cluster Size

github.com/42dot/foros

Links-FOROS Github : https://github.com/42dot/foros-

FOROS Wiki : https://github.com/42dot/foros/wiki-

RAFT : https://raft.github.io/-

RAFT Paper : https://raft.github.io/raft.pdf-42dot : https://42dot.ai/

github.com/42dot/foros

Q&A

github.com/42dot/foros