DMTCP: Fixing the Single Point ofFailure of the ROS Master
Tw i n k l e J a i n

Gene Cooperman

j a i n . t @ h u s k y. n e u . e d u

gene@ccs.neu.edu

College of Computer and Information ScienceN o r t h e a s t e r n U n i v e r s i t y, B o s t o n , U S A

September 21, 2017

DMTCP: Fixing The Single Point Of Failure Of The ROS Master | Gene Cooperman and Twinkle Jain1

Roadmap• Core Problem: ROS master failure• Possible Solutions• Effective solution: Transparent checkpointing (DMTCP)• DMTCP’s workflow• Barrier: DMTCP didn’t support Pseudoterminals• Pseudoterminal plugin• Extending the DMTCP• Live demo
• DMTCP’s growing number of use cases• DMTCP, ROS2, and the future!
• Summary
September 21, 2017

DMTCP: Fixing The Single Point Of Failure Of The ROS Master | Gene Cooperman and Twinkle Jain

2

Core Problem: ROS master failure• ROS master contributes in:• Provides name and parameter services to the rest of the nodes• Control over system, graphs, nodes• Helps a node to discover other nodes

Ros Master• Why is the ROS master’s failure a problem?

• No way to recover the system after crash• Nodes need to reload• System reinitialization• Whole system will compromise• The amount of loss can be huge in a robotics application

Node 1Communication

Node 2• See talk by our collaborator, Matthieu Amy, at ROSCon-2016

September 21, 2017

DMTCP: Fixing The Single Point Of Failure Of The ROS Master | Gene Cooperman and Twinkle Jain

3

Possible Solutions• Switch to a more reliable hardware• Expensive solution• Chances of failure decreases but crash can happen• Reduce the dependency on the ROS master• Currently unavailable in ROS1
• Can be applied in the future versions of ROS• Checkpoint and Restart (presented here)• Save the state periodically• Resume from the last checkpointed state

September 21, 2017

DMTCP: Fixing The Single Point Of Failure Of The ROS Master | Gene Cooperman and Twinkle Jain

4

Effective solution:
Transparent Checkpointing (DMTCP)• DMTCP: Distributed Multi-Threaded CheckPointing(http://dmtcp.sourceforge.net/)• Active project for 13 years• Over 10,000 downloads of source, unkown # downloads of binary package• Open source package for transparent checkpointing• Checkpointing – saving restorable application state like a snapshot• Transparent – application remains unmodified• User space – kernel remains unmodified• Easily extensible – Plugin-based architecture• Both manual and periodic checkpointing options are available• NEWS: Being extended to support NVIDIA GPUs
September 21, 2017

DMTCP: Fixing The Single Point Of Failure Of The ROS Master | Gene Cooperman and Twinkle Jain

5

DMTCP’s workflow: Checkpointing
Start the application with DMTCP
Gain control over threads

DMTCPCOORDINATOR

Normal execution
Initiate checkpoint
CKPT MSGCKPT MSG

Suspend user threads

CKPT THREAD
USER THREAD A
USER THREAD B

SOCKET
CONNECTION

SIGUSR2

SIGUSR2

Save application state
CKPT THREAD
USER THREAD C

Write checkpoint image to a file

USER PROCESS 1

USER PROCESS 2

Node 1

Node 2September 21, 2017

Resume user threads

DMTCP: Fixing The Single Point Of Failure Of The ROS Master | Gene Cooperman and Twinkle Jain

6

DMTCP’s workflow: Restart
Restart the application with DMTCP
DMTCPCOORDINATORStart a new process on each node
Exec into MTCP restart
CKPT MSGCKPT MSG

Overwrite memory with ckpt image

CKPT THREAD
USER THREAD A
USER THREAD B

SOCKET
CONNECTION

SIGUSR2

SIGUSR2Restore application state
CKPT THREAD
USER THREAD C

Normal execution

USER PROCESS 1

USER PROCESS 2

Node 1

Node 2September 21, 2017

Resume user threads

Ready for the checkpoint
again

DMTCP: Fixing The Single Point Of Failure Of The ROS Master | Gene Cooperman and Twinkle Jain

7

Barrier: DMTCP didn’t support Pseudoterminals!
• When DMTCP tried to checkpoint a ROS application, it didn’t work!
• Pseudoterminals• Pair of two virtual devices connected witha bidirectional IPC channel• Emulates a device terminal for terminal oriented
programs (e.g., vi editor)• What is the problem?

Kernel space

User space• A typical ROS application uses ptys
• DMTCP doesn’t handle pty’s checkpointing correctly• The state of a pty-based application was not same
Fig. General approach for a pty-basedapplication
at the time of checkpoint, resume after checkpoint and at restart

September 21, 2017

DMTCP: Fixing The Single Point Of Failure Of The ROS Master | Gene Cooperman and Twinkle Jain

8

Pseudoterminal plugin• Wrapper functions for the pty related functions• Packet or non-packet mode
• Raw or cooked mode
• Terminal settings• Drain the kernel buffer• Whether process running in the foreground or background• Characteristics of bidirectional IPC channel between master and slave
devices• Order of refilling the kernel buffer

September 21, 2017

DMTCP: Fixing The Single Point Of Failure Of The ROS Master | Gene Cooperman and Twinkle Jain

9

Extending DMTCP: Checkpoint & Restart

September 21, 2017

DMTCP: Fixing The Single Point Of Failure Of The ROS Master | Gene Cooperman and Twinkle Jain

10

Live Demo: checkpoint a ROS application• ROS application details:• 3 processes running in 3 different terminal windows• roscore• Listener node
• Talker node

• DMTCP demo details:• DMTCP coordinator running in the fourth terminal window• Manual checkpointing and restart• Let’s start!

September 21, 2017

DMTCP: Fixing The Single Point Of Failure Of The ROS Master | Gene Cooperman and Twinkle Jain

11

DMTCP’s growing number of use cases• Fault tolerance• Process migration• Skip past long startup times• Debugging• Ultimate bug report• Speculative execution• Easily Scalable – over 32,000 nodes checkpointed at ICPADS’16(Over 50 refereed publications by external teams, describing their use of DMTCP)

September 21, 2017

DMTCP: Fixing The Single Point Of Failure Of The ROS Master | Gene Cooperman and Twinkle Jain

12

DMTCP, ROS2, and the future!
• An RMW(ROS MiddleWare) based on a ROS master and DMTCP• DMTCP is transparent to the end user because it is built into the RMW• Combining DMTCP with a small log-and-replay implementation allows rollback
and replay during recovery for full reliability.

• This allows an RMW developer to provide a small, reliable RMW, while
delegating to DMTCP developers the burden of guaranteeing fault tolerance.
• DMTCP is free and open-source. It uses GNU LGPL (non-contagious): no license
fees and can be combined with proprietary corporate software.

September 21, 2017

DMTCP: Fixing The Single Point Of Failure Of The ROS Master | Gene Cooperman and Twinkle Jain13

Summary• DMTCP now solves the single point of failure problem of the ROSmaster.
• Can now checkpoint-restart a typical ROS application.
• Plugin-based architecture helps in extending it easily.

• Can be bundle with larger application package.
• In the future, we hope to work along with upcoming ROS2framework.

September 21, 2017

DMTCP: Fixing The Single Point Of Failure Of The ROS Master | Gene Cooperman and Twinkle Jain

14

Thank you
This work is also the result of a joint collaboration with Jean-Charles Fabre, Michael Lauer, and Matthieu
Amy of the dependable computing and fault tolerance team At LAAS, Toulouse, France.

September 21, 2017

DMTCP: Fixing The Single Point Of Failure Of The ROS Master | Gene Cooperman and Twinkle Jain

15

Appendix: Using DMTCP with ROS
Requires developer version of DMTCP and Linux kernel version above 3.8.0
Execute on <COORD_HOST>:
dmtcp_coordinator --port 7779
Execute once for each ROS master/node (ckpt interval of 5 seconds):
dmtcp_launch --join --interval 5 --port 7779 \<path_to_the_ROS_executable>
Execute once for each ROS master/node:
dmtcp_restart --join --port 7779 --host <COORD_HOST> \<path_to_each_checkpoint_file>

September 21, 2017

DMTCP: Fixing The Single Point Of Failure Of The ROS Master | Gene Cooperman and Twinkle Jain

16

Appendix: Using DMTCP with ROS (cont.)• Can change 7779 (the default port) of the DMTCP coordinator to another TCP port, as desired.
• On each ROS node, the checkpoint files will be of the form ckpt_*.dmtcp. Note that ROS sometimes
creates additional Python processes, resulting in checkpoint image files for Python. Be sure to include
the related Python checkpoint files when restarting the roscore/master.
• Note that if we checkpointed in three different Linux terminals (e.g., master, talker, listener), then wemust invoke dmtcp_restart in the three different terminals, with the appropriate ckpt_*.dmtp files forthe corresponding ROS node.
• If one prefers manual checkpointing (no ‘--interval’ flag), then one can execute ‘c’ (checkpoint) in aterminal window running the DMTCP coordinator, optionally followed by ‘k’ (kill).
• The DMTCP coordinator is stateless. It’s fine to kill any old coordinator and start a new one onrestart.

September 21, 2017DMTCP: Fixing The Single Point Of Failure Of The ROS Master | Gene Cooperman and Twinkle Jain

17

References:1.

Ansel, Jason, Kapil Arya, and Gene Cooperman. “DMTCP: Transparent checkpointing forcluster computations and the desktop.” Parallel & Distributed Processing, 2009. IPDPS 2009.
IEEE International Symposium on. IEEE, 20092.

Jean-Charles Fabre, Michael Lauer, Matthieu Amy. “Adaptive Fault Tolerance on ROS: A
Component-Based Approach” ROSCon 2016.

3.

Kerrisk, Michael. The Linux programming interface. No Starch Press, 2010.

September 21, 2017

DMTCP: Fixing The Single Point Of Failure Of The ROS Master | Gene Cooperman and Twinkle Jain18