Adaptive Fault Tolerance on ROS:
A Component-Based Approach

Jean-Charles Fabre, Michael Lauer, Matthieu Amy

LAAS-CNRS, Ave du Colonel Roche, F-31400 Toulouse, France-1-1-

Definitions
Dependability: Ability to provide services that can defensibly betrusted within a time-period.

Fault Tolerance (FT) : Design and implementation of mechanisms tocontrol errors (residual, random, systematic…) by detecting them andensuring transitions to a safe state

Resiliency: The persistence of dependability when facing changes-2-Definitions
Dependability: Ability to provide services that can defensibly betrusted within a time-period.

Fault Tolerance (FT) : Design and implementation of mechanisms tocontrol errors (residual, random, systematic…) by detecting them andensuring transitions to a safe state

Resiliency: The persistence of dependability when facing changes

Adaptive Fault Tolerance (AFT): Design and implementation of Fault
Tolerant Mechanisms (FTM) to ensure the dependability of thesystem at runtime when facing changes-3-Problem statement and key concepts
Once the system is deployed, it faceschanges.
System designers cannot predict everything.
Persistence of dependability requires theadaptation of safety mechanism-4-Problem statement and key concepts
Once the system is deployed, it faceschanges.
System designers cannot predict everything.
Persistence of dependability requires theadaptation of safety mechanism

Key concepts for Adaptive Fault Tolerance- Separation of concerns- Design for adaptation- Remote fine-grained updates-5-Overall process
FTM as a Lego system-6-Overall process
FTM as a Lego system-7-

Overall process
FTM as a Lego system

Remote update• Component graph• Suspend execution• Modification of the graph• Re-activate-8-Overall process
FTM as a Lego system
Change Safety analysis / FMECA
 Impact on safety mechanism
 Agile update of FTM
 Remote update

Remote update• Component graph• Suspend execution• Modification of the graph• Re-activate-9-Assumptions and FTM Characteristics

PBR=Primary-Backup Replication
LFR=Leader-Follower Replication
TR=Time Redundancy-10-Assumptions and FTM Characteristics

LFR

FTA, R

PBR

LFR⊕TRA, RFTPBR⊕TRPBR=Primary-Backup Replication
LFR=Leader-Follower Replication
TR=Time Redundancy-11-

Assumptions and FTM Characteristics
TRANSITIONS

Trigger: high rate of HWtransient faults observed

LFR

FTA, RPBR

LFR⊕TRA, RFTPBR⊕TRTrigger: Non deterministic
SW application version
PBR=Primary-Backup Replication
LFR=Leader-Follower Replication
TR=Time Redundancy

Trigger: bandwidth drop
below a given threshold-12-Componentization of FTM

Component-based
implementation

Transitions
between FTMs

Design for adaptationof FTMsrequestChange model

fault tolerant application
processingservice

replyClientServer-13-Componentization of FTM

Component-based
implementation

Transitions
between FTMs

Design for adaptationof FTMsrequestbeforeproceedafter

Change model

reply
Clientapplication
service

Server-14-

Componentization of FTM
replyLog
protocolsyncBefore

Component-based
implementation

Transitions
between FTMs

Design for adaptationof FTMs

proceedsyncAfter
FTM

requestbeforeproceedafter

Change model

reply
Clientapplication
service

Server-15-

Design for FTM adaptation on ROS
Generic computation graph for FTM(Boxes represent nodes)

clt2srv

ServerClient• Topics(0)• Nodes(2)– Client– ServerServices: clt2srv (client to server)-16-

Design for FTM adaptation on ROS
Generic computation graph for FTM(Boxes represent nodes)
pro2bfr
pxy2proclt2srv

ClientProxy
pro2pxyP
rotocol

Beforebfr2prdprd2srv

ProceedServer

prd2aft

After

Service

aft2proTopic

FTM• Topics(6)– pxy2pro– pxy2bfr, bfr2prd,prd2aft– aft2pro– pro2pxy• Nodes(5+2)– Client– Server– Proxy– Protocol– Before, Proceed, After

Services: clt2pxy (client to proxy) and prd2srv (proceed to server)-17-

Implementing PBR on ROS
CLIENTpxy2proclt2pxy

ClientProxy

Recovery

pro2pxyProtocol

MASTER

PRIMARY

pro2bfr

Beforebfr2prdprd2srv_M

ProceedServer_M

prd2aftgetstate

After
aft2proServiceCD_M

recovery

Topic

SLAVE
BACK-UP

pro2bfr

aft2aftP
rotocol

Beforebfr2prdProceed

prd2srv_S

Server_S

prd2aft
setstate

After
aft2proCD_S
cd2rec-18-

Implementing PBR on ROS
MASTER

PRIMARY

CLIENTpro2bfr
pxy2proclt2pxy

ClientProxy

Recovery

pro2pxyP
rotocol

Beforebfr2prdprd2srv_M

ProceedServer_M

prd2aftgetstate

After
aft2proServiceCD_M

Topic

SLAVE

recovery

BACK-UP

pro2bfr

aft2aft

ProtocolBeforebfr2prdProceed

prd2srv_S

Server_S

prd2aft
setstate

After
aft2proCD_S
cd2rec-19-

Implementing PBR on ROS
CLIENT

clt2pxy

ClientProxy

RecoveryServiceTopic
SLAVE

recovery

BACK-UP

pro2bfr

pxy2propro2pxy

aft2aftP
rotocol

Beforebfr2prdProceed

prd2srv_S

Server_S

prd2aft
setstate

Afteraft2pro

CD_S
cd2rec-20-

Implementing TR on ROS

TRCLIENT

pro2bfr
pxy2proclt2pxy

ClientMASTERProxy
pro2pxyP
rotocol

getstate_M
setstate_M

Beforebfr2prdprd2srv_M

aft2bfr

ProceedServer_M

ServiceTopic

prd2aft

After
aft2pro-21-

Combining FTM on ROS
Generic composition graph for FTM
pro2bfr
pxy2proclt2srv

ClientProx
ypro2pxyProtocol

Beforebfr2prdprd2srv

ProceedServer

prd2aft

After

Service

aft2pro
Topic

FTM1

• Protocol node is a software rack of nodes– Before– Proceed  activation of services or protocols– After• Protocol node can substitute for proceed node
– It can be view as a frontend of the server…

-22-Combining FTM on ROS
Generic composition graph for FTM
pro2bfr
pxy2proclt2srv

ClientProx
ypro2pxyP
roto
col

Beforebfr2prdprd2srv

ProceedServer

prd2aft

AfterService

aft2pro
Topic

FTM1

• Protocol node is a software rack of nodes– Before– Proceed  activation of services or protocols– After• Protocol node can substitute for proceed node
– It can be view as a frontend of the server…

-23-

Combining FTM on ROS
Generic composition graph for FTM
pro2bfr
pxy2proclt2srv

ClientProxy
pro2pxyP
rotocol

pro2bfr

Beforebfr2prdprd2aft

After

P
roto
col

aft2pro

FTM1

Beforebfr2prdProceed

prd2srv

Server

prd2aft

After

Service

aft2proFTM2Topic• Protocol node is a software rack of nodes– Before– Proceed  activation of services or protocols– After• Protocol node can substitute for proceed node– It can be view as a frontend of the server…

-24-Combining PBR+TR on ROS
CLIENTpro2bfr
pxy2proclt2pxy

ClientProxy

Recovery

pro2pxy

PRIMARY
BeforeProtocol aft2pro

bfr2prd_M

prd2aft_M

ProtocolMASTERTR

pro2bfr

getstate_M
setstate_M

Beforebfr2prdaft2bfr

prd2srv_M

ProceedServer_M

prd2aft

After
aft2proAfter
getstate_M

CD_M

aft2aftrecoveryTR

BACK-UP
pro2bfr

BeforeProtocolSLAVE

pro2bfrbfr2prd_S

prd2aft_S
aft2proProtocol

getstate_S
setstate_S

Beforebfr2prdaft2bfr

Proceed

prd2srv_S

Server_M

prd2aft

After
aft2proAfterrestorestate_S/cd2rec

CD_S-25-Case Study• Initialization– Initialisation time around 0,5s
– Time due to the initialization of communications by the ROS Master• Execution– Around 5ms for the PBR and 2ms for the TR– Requests every 7cm for a car driving at 50km.h-1• Recovery– Recovery → Reactivation of 2 Topics– Recovery time around 1ms• Adaptation & Composition– Adaptation → Initialization of new nodes– Same order as Initialization time (≈ 0,3s)

Ubuntu Trusty 14.04
I5 Dual Core 2,5GHz
8Go DDR3 RAM-26-ROS Master : A single point of failure• The ROS Master is requisite for:
– The control over the system– The control over communication– The control over the graph– The control over the Nodes• If the ROS Master crash:– Loss of the software architecture– Nodes have to be reloaded– The state of the system is reinitialized– Critical loss in case of embedded
systems

Solutions to assure the reliability of the ROS Master:‒ Launching it on a distinct and reliable machine‒ Check-pointing its state and restoring it-27-

DMTCP: Check Pointing the ROS Master• DMTCP, how does it work:– Works with Linux kernel 2.6.9 and later– Transparent (no recompilation…)– Virtualization of Process ID• Check pointing with DMTCP:– Process is launch along the coordinator– A checkpoint image is created for each process– A restart script is created by coordinator→ DMTCP should be able to checkpoint the ROS Master→ The lost of the ROS Master should no longer be a problem-28-

Lessons learnt• Adaptive fault tolerance– Separation of Concern– Design for Adaptation
SoC+D4A  FTM isolation and componentization• Installation or adaptation of an FTM online– Node can be started and stopped– Mapping at initialization• Node Management– APIs are not provided by ROS for Node Management– User signals and System calls fulfill the missing requirements• Implementing dynamic binding– Natural dynamic binding is also not provided by ROS– Topics and Services are remapped at the initialization-29-

Summary of dynamic adaptation
SoC

ROS nodes, component mapping to nodes

D4AComponentized FT design patterns
Protocol-Before-Proceed-After

Nodes
MngmntUnix system calls and ROS commands

DynamicBinding

ROS services, ports, topics
Additional logic to create ports and topics

MasterCKPTCheck point of the ROS Master
ROS Master is no longer a Single Point of Failure-30-Conclusion• Now…
– Adaptive Fault Tolerance for Resilient Computing is possible on ROS– Design and validation of FTMs is always carried out offline– If application can be terminated and re-launched : adaptation OK– Dynamic adaptation :• Extended API for dynamic binding• Consistency of reconfiguration?

• Proceeding...
– Experiments on ADAS with Renault SAS– Evolution of AUTOSAR into Adaptive AUTOSAR– Experimentation on ROS Master with DMTCP-31-