FlexBE -The Flexible Behavior Engine:
Collaborative Autonomy for ROS 2
David C. Conner, PhD
Associate Professor
robotics@cnu.edu
October 2023

1

FlexBE Overview• History and background• Key features and design• Related ROS packages• Ongoing research and development

2

October 2023
© 2023 David C. Conner

Team ViGIR’s Atlas at the2015 DARPA Robotics Challenge Finals

3

October 2023
© 2023 David C. Conner

DARPA Robotics Challenge (DRC)• Human-robot teams• Supervised autonomy– Operators can inject information– Operators can preempt behaviors• Constrained communications4October 2023
© 2023 David C. Conner

Hierarchical Finite State Machines

Composebehaviors
withinbehaviors“Hierarchical”
FSM states cancontain otherFSM

2015 DRC
Open Door Task
High-levelBehavior
HFSM acts as ascript forsupervisors/robot

5

October 2023
© 2023 David C. Conner

FlexBE : The Flexible Behavior Engine• Originally conceived as an extension of ROS 1 SMACH– http://wiki.ros.org/smach– Hierarchical Finite State Machines (HFSM)– Python-based state implementations• Initially developed by Philipp Schillinger@ TU Darmstadt (Germany)• ROS 1 open-source release in Fall 2015
– http://wiki.ros.org/flexbe– http://github.com/FlexBE

http://philserver.bplaced.net/fbe/applications.php
6

October 2023
© 2023 David C. ConnerKey Design Concepts•

Support for high-level behavior control––––

Hierarchical Finite State Machines
Natural interaction with system capabilities
Concurrent state execution
Adjustable/sliding autonomy levels•

Support for unsupervised fully autonomous mode

– Runtime modifiable behaviors•

Intuitive GUI– State machine (behavior) editor– Interactive operator/supervisor runtime interface•

Enable “Collaborative Autonomy”

https://onlinelibrary.wiley.com/doi/full/10.1002/rob.21671
7October 2023
© 2023 David C. Conner

Continuing Development @ CNU• Christopher Newport University
Dept. of Physics, Computer Science and Engineering– FlexBE used in “Introduction to Robotics”• Recent releases
In collaboration with Philipp Schillinger– Final ROS 1 Noetic release May 2023
– ROS 2 Conversion

https://cnu.edu
4500 students in SE VA• Initial source released summer of ’22• Humble and Iron binaries summer ‘23
8

October 2023
© 2023 David C. ConnerKey Features and Design
Key Features and Design

9October 2023
© 2023 David C. ConnerFlexBE is• Python-based– Easy state implementation development• NOT for high-rate control• Desired update rate in tens of Hz• NOT for real-time control• NOT for verifiable safety critical systems

FlexBE caneasilyinteractwith suchsystems

The purpose of FlexBE is high-level behavioral control systems.
10

October 2023
© 2023 David C. Conner

FlexBE GUI : Behavior Dashboard
FlexBE
allowsmodifiable“userdata”
to be passed
from state tostate

FlexBE
allowsparameters
to beconfigured
at runtime
http://wiki.ros.org/flexbe/Tutorials and https://github.com/FlexBE/flexbe_turtlesim_demo

11

October 2023
© 2023 David C. Conner

FlexBE GUI : Statemachine Editor

List of available Python
state implementations
that interact withsystem capabilities
http://wiki.ros.org/flexbe/Tutorials and https://github.com/FlexBE/flexbe_turtlesim_demo

12

October 2023
© 2023 David C. Conner

FlexBE Behavior Example

Walk to Template
Behavior
A state machine realizes a desired “behavior” by invoking system capabilities;
behavior state machines can be composed.
13

October 2023
© 2023 David C. Conner

FlexBE GUI : Runtime Control
Potentialoutcomes

Autonomylevel

Current
active state
Operator can selectoutcome to preempt
state and force atransition
http://wiki.ros.org/flexbe/Tutorials and https://github.com/FlexBE/flexbe_turtlesim_demo

14

October 2023
© 2023 David C. ConnerKey Design Concepts• Sliding autonomy levels– Low requires operator to confirm some transitions• i.e., it blocks exit transition– Full allows fully autonomous transitions• Lockable states and edit on the fly• Enable operator forced transitions
15

October 2023
© 2023 David C. Conner

FlexBE Communications
Operator Control Station (OCS)

Onboard Robot Software

States sharepub/sub/action
interfaces via“proxies”
DRC constrained communications
between robot and operator team© 2023 David Conner (david.conner@cnu.edu)

FlexBE States and ROS 2 Actions• States interact with system capabilities• Commonly implement an action client interface– Send goal on_enter– Monitor feedback and result in execute– Return outcome and on_exit transition on action result
For example, see code for topic-, service-, and action-based state implementations at
https://github.com/FlexBE/flexbe_turtlesim_demo/tree/ros2-devel/flexbe_turtlesim_demo_flexbe_states/flexbe_turtlesim_demo_flexbe_states

17

October 2023
© 2023 David C. ConnerQuick Start Demonstration

https://github.com/FlexBE/flexbe_turtlesim_demo
Supported under Naval Engineering Education Consortium (NEEC) grant N00174-23-1-0018

18
October 2023
Examplequick start demonstration at https://github.com/flexbe/flexbe_turtlesim_demo© 2023 David C. Conner

Significant Upgrades @ CNU• ROS 2 conversion– Initial source released summer of ’22– Refinements and cleanup in summer ‘23– Humble and Iron binaries summer ’23– Enhancements to concurrent states in ros2-pre-release branch• Planned release to Iron coming soon
Summer ’23 work supported under Naval Engineering Education Consortium (NEEC) grant N00174-23-1-0018

19

October 2023
© 2023 David C. ConnerRelated ROS Packages
Related ROS Packages

20

October 2023
© 2023 David C. Conner

Flexible Navigation Package• Collaborative navigation– Allow approval of plans or replan– Separate global and local planners

Allow operator toconfirm planned pathprior to execution inlow autonomy

ROS 1versionshown• FlexBE state implementations
interface to Nav2 capabilities– State implementations– Special Nav2 compatible nodes

https://github.com/FlexBE/flexible_navigation
https://ieeexplore.ieee.org/document/7925266
https://ieeexplore.ieee.org/document/9764047
https://github.com/FlexBE/flex_nav_turtlebot3_demo
21

October 2023
© 2023 David C. Conner

Flexible Behavior Trees• Behavior trees are popular alternative to HFSM
• In search of the “Mythical HFSMBTH” – HFSM-BT Hybrid– from a 2017 Game Developers conference talk by Bobby Anguelov
https://www.youtube.com/watch?v=Qq_xX1JCreI&t=1159s– Combine each method’s strengths• BT: Reactive decisions, high-speed• HFSM: cyclical/repetitive behaviors , collaborative autonomy• Flexible Behavior Trees : The “Mythical HFSMBTH” with FlexBE• The paper: https://arxiv.org/abs/2203.05389• The code: https://github.com/FlexBE/flexible_behavior_trees• The demo: https://github.com/FlexBE/flex_bt_turtlebot3_demo
22

October 2023
© 2023 David C. Conner

Flexible Manipulation• FlexBE interface to MoveIt!
https://ieeexplore.ieee.org/document/8478933
https://github.com/CNURobotics/flexible_manipulation• Currently only ROS 1 (Kinetic) and Python 2• Planning for ROS 2 conversion late 2024
– MoveIt! 2 stabilizing– Stable ROS 2 physics-based simulations of robot arms
23

October 2023
© 2023 David C. Conner

Ongoing Research and Development

Ongoing Research and Development

24

October 2023
© 2023 David C. Conner

System Improvements• Testing, demonstrations, and tutorials for packages• FlexBE WebUI– Improved graphics– Simplified Python comms integration with UI– Expect alpha pre-release January 2024
– Planned UI Advancements

Adjustablearcs/labels• Improved operator control over concurrent states• HFSM Synthesis and debugging tools
Supported under Naval Engineering Education Consortium (NEEC) grant N00174-23-1-0018

25

October 2023
© 2023 David C. Conner

HFSM Synthesis in FlexBE• Designing HFSM (or BT) is hard
– Requires significant testing and validation• Goal: “Correct-by-construction” synthesis tools– Specialized research in formal methods community– Less accessible to general robotics community

26

October 2023
© 2023 David C. ConnerPrior Work with ROS 1 FlexBEand Reactive GR1 Synthesis
Formal Task
Specification

Workspace

ReactiveSynthesis

Discrete Strategy(Automaton)
S2

S1

Maniatopoulos et al., "Reactive highlevel behavior synthesis for an Atlas
humanoid robot," ICRA 2016,
https://ieeexplore.ieee.org/
document/7487613

SystemEnvironment

Grounding toSystem Capabilities

Hayhurst and Conner, "Towards
Capability-Based Synthesis of Executable
Robot Behaviors," SoutheastCon 2018,
St. Petersburg, FL, USA, 2018, pp. 1-8,
https://ieeexplore.ieee.org/
document/8479047(e.g. FlexBE state implementations)27

October 2023
© 2023 David C. ConnerGR1 Synthesis Example w/ Slugs
https://link.springer.com/chapter/10.1007/978-3-319-41540-6_18

Assumptions → Guarantees
GR1 fragment of LTL
complexity 끫뢄 2끫뢶
vs.
끫뢶
Full LTL w/ 끫뢄 22time 

“If in next step the goat and wolf are on
the left bank, then on the next step
the farmer better be on left bank”
X(!goat & !wolf) -> X!farmer28

October 2023
© 2023 David C. Conner

Confirm transition in
low autonomy
Switch to
Full Autonomy
Operator preempt

Synthesized
Realization
Manually
Parametersofcreatedstatemachines
SLUGS
definedstatesynthesized
bymachine
placedMealyautomatamachine
infromstatemachine
SLUGS
fromoutputs
synthesis
GR1containerspecs
Run time execution monitor

29October 2023
© 2023 David C. Conner

Ongoing Synthesis Work in FlexBE• Converting 2018 system into ROS 2 version• Develop several tutorials and demonstrations– Make synthesis more accessible to general community
The paper: coming Spring ’24
The code: coming Dec ‘23
The demo: coming Dec ‘23• Refactor and redesign to simplify usage• Integrate automatic discovery of system capabilities
New work supported under Naval Engineering Education Consortium (NEEC) grant N00174-23-1-0018

30

October 2023
© 2023 David C. Conner

Conclusion• FlexBE is now available for ROS 2• Quick start demo at
https://github.com/FlexBE/flexbe_turtlesim_demo• Available extension packages• Development is active and ongoing• Active research integrating HFSM synthesis
For more information
robotics@cnu.edu

31

October 2023
© 2023 David C. Conner

32

October 2023
© 2023 David C. Conner

FlexBE State Implementations• Each state corresponds to a Python implementation of theEventState classhttps://github.com/FlexBE/flexbe_behavior_engine/blob/ros2-devel/flexbe_core/flexbe_core/core/event_state.py– on_start – invoked when behavior initialized– on_enter – invoked when state becomes active– execute– invoked at (approximately) specified rate
– on_exit– invoked when state returns outcome– on_stop– invoked when behavior is shutdown– on_pause/resume – invoked when state is locked/unlocked
33

October 2023
© 2023 David C. Conner

FlexBE in Education• Currently use in CPSC 472/572“Introduction to Robotics”@CNU• FlexBE in low-autonomy acts as “script”– “Get goal”, “Plan path”, “Execute path follower”– Allows users to better see interaction of components• Teach HFSM-based behavior control– Students can write Python-based state implementations– Use FlexBE to control high level system behaviors– Reinforce use of object-oriented paradigm
34

October 2023
© 2023 David C. ConnerWGCF Specs (Slugs format)[ENV_INIT][INPUT]# our farmer prefers 4 letter words not cabbages
goat# Everyone on the left bank!goat!wolf(wolf & move_wolf & farmer) -> (!wolf' & !farmer')wolf!corn(!wolf & move_wolf & !farmer) -> (wolf' & farmer')corn!farmer(corn & move_corn & farmer) -> (!corn' & !farmer')farmer[OUTPUT][SYS_INIT](!corn & move_corn & !farmer) -> (corn' & farmer')move_goat# The game solver should figure out that one thing needs# to be moved initially(move_empty & farmer) -> !farmer'

move_wolf
move_corn
move_empty(move_empty & !farmer) -> farmer'[ENV_TRANS]# What occurs in environment# Not moving leaves the environment alone# transitions due to move

!move_goat -> ((goat & goat') | (!goat & !goat'))(!goat & move_goat & !farmer) -> (goat' & farmer')!move_wolf -> ((wolf & wolf') | (!wolf & !wolf'))(goat & move_goat & farmer) -> (!goat' & !farmer')!move_corn -> ((corn & corn') | (!corn & !corn'))35

October 2023
© 2023 David C. ConnerWGCF Spec (continued)[SYS_TRANS]# Allowable commands from our controller# What we need our controller to enforce[ENV_LIVENESS]# Farmer must stay when goat and corn are together# Nothing(goat' & wolf') -> farmer'# Can only move one thing at a time

[SYS_LIVENESS](!goat' & !wolf') -> !farmer'# Let's get across infinitely often!(move_goat & move_wolf)(goat' & corn') -> farmer'!(move_goat & move_corn)(!goat' & !corn') -> !farmer'!(move_goat & move_empty)# Cannot move unless boat on same side!(move_corn & move_wolf)(goat & !farmer) -> !move_goat

goat & wolf & corn!(move_corn & move_empty)(!goat & farmer) -> !move_goat(wolf & !farmer) -> !move_wolf!(move_wolf & move_empty)(!wolf & farmer) -> !move_wolf(corn & !farmer) -> !move_corn(!corn & farmer) -> !move_corn

36

October 2023
© 2023 David C. Conner