On multi-ﬂeet operations
with the Open Robotics
Middleware Framework

19 Oct 2022, ROSCon JP 2022, Kyoto, Japan

Open Robotics Stewarded Platforms• World’s most widely-used• Open-source robot

open-source SDK forrobotics

simulation software• ROS is to robotics asAndroid is to mobile
devicesros.org• Gazebo is to robotics
what AutoCAD is toarchitecturegazebosim.org• Open-source multi-fleet
management andinteroperability software• Open-RMF is to robotics
as SAP is to operations

open-rmf.org

We’re asking
more of robots(a lot more)

The industry is growing dramatically~15%

>1K

88%

53%

Global robot
density CAGR

companies planning
robotic adoption

global robot supplierswith diverse solutions

companies listing costof adoption as a top 5
challenge to adoption
Source IFR
Source IFR
Source Zippia

Many
Amazing
RobotsCleaning

The interoperability
dilemma

Logistics

Security

Delivery

Guten
Tag!

but they needtocommunicate
with eachotherand resources
have alreadybeen spent todevelop anddeploy disparate
systems
こんにちは

We can’tX
all getalongTraffic
Management
Task
Dispatcher

Traffic
EditorInfrastructure
Integration

FreeFleet

RMF
SimulationRMF Web
Ecosystem ofSolutions

What is Open-RMF?

Open-RMF Creates Market Opportunities

Facility
Owner/
Operator
Reduces TCO
and frees up
resources forfurther growth
Photo by maitree rimthong

RobotVendor

InfraProvider

Solution
Provider

Reduces
integration
costs,
complexity
and time

Reduces thetotal numberof integrations

Reduces
integration
costs,
complexity
and time

Photo by Lenin Estrada

Photo by Steve Johnson

Photo by Pixabay

Robot Providers

Collaborators

Infrastructure

SI’s andSolution Providers

Construction

Healthcare

Airports

Application Fields
Seaports

Manufacturing &Logistics

Hospitality

SpaceAgricultureTRAFFIC EDITOR

By PNW Production

Start using RMF in three easy steps
1. Find or create a building map
2. Create a simulation model
3. Make a robot traffic map
Step 1 may require human interaction.
Traffic Editor helps you with steps 2 and 3.

Building floorplan

Annotated in traﬃc_editor

Physics-based Gazebo
simulation world

Create ﬂoor polygons● First we need to deﬁne the ﬂoor polygon● It can be precise to just look nice, or just abounding-box so robots don't fall down
● Each ﬂoor polygon has the same material● Multiple polygons can be used to model
different ﬂooring types

Create walls● Walls can be traced from the ﬂoor plans● Typically not all walls are needed or wanted● Materials (textures) can be deﬁned for eachsegment

Conﬁgure door parameters● Doors are surprisingly complicated!
● Use the property editor to conﬁgure themtypehingeddouble_hinged

motion_degrees

90

135

motion_axis

start

end

motion_direction

1-1

right_left_ratio

12

sliding170

0.5

double_sliding

Conﬁgure elevators● Elevators (lifts) are a critical partof any multi-level robot site
● They are incredibly expensive (!)● They often set the throughput
limit for the entire system● Careful modeling is important

Drawing robot traffic lanes● Traffic lanes can be drawn on the map● This helps simplify the planning problem andmake the robot motion patterns easier tounderstand for people nearby● Traffic lanes can be extended, branched,moved, etc., in the GUI● Next question: how do we align this with therobot coordinate system?

Aligning real-robot maps●●●Fully-automatic alignment is hard
○ As-built often differs from floorplan, especially wall partitions○ Cabinets, pillars, chairs, etc., can occlude walls
Humans can easily spot these issues and ignore them○ This is difficult to program for computers"Semi-Automatic" alignment workflow:○ Roughly align the robot map using manual transform parameters○ Click "features" that you can see in both robot map and floorplan■ corners of load-bearing walls■ pillars○ Click "constraints" between corresponding features○ Select Edit->"Optimize Layer Transforms" (Ctrl+T)■ Numeric solver tries to find a transformationthat minimizes the constraint lengths

18

Aligning maps with numerical optimization

19

Aligning maps

20

Building Map Tools

building_map_tools

Annotated map in traﬃc_editor

http://github.com/open-rmf/rmf_traffic_editor

Physics-based simulation world with
3d assets, robots, doors, plugins

21

https://github.com/open-rmf/rmf_site

The RMF Site Editor

Try the

https

webG

://opL ve

en-rm rsion inyour wf.git
Next generation of RMF site editing
hub.io/rmf eb browser!
_site• Full 3D previews of the simulated facility, updated live as you edit it/•••

Runs natively on Linux, Windows, Mac, or web browsers
Edit the placement of furniture, walls, doors, lifts, traffic lanes, human
crowd behavior, cameras, and lights
Easy to extend with custom plugins for application-specific features

Upcoming Features••

Preview camera feeds
Save and link "read-only" portions, to reuse site information acrossdifferent scenarios and allow manual additions to upstream data

Beyond•A single application that can run on any computer (including Windows)
to simulate a full deployment of RMF. All RMF nodes would run inside
the site editor, running the same code as the real deployments. Users
can add their own custom nodes as plugins.

CAPABILITIESOverview●

Traffic Negotiation○ Full Control○ Traffic Light○ Read-Only●

Shared Infrastructure○ Doors○ Elevators (Lifts)○ Workcells●Task Management○ Customize○ Dispatch○ ExecuteTraffic Negotiation
Full Control: Robot/Fleet API accepts arbitrary waypoint commands
Traffic Light: Robot/Fleet API accepts pause/resume commands
Read-Only: Robot/Fleet API only provides current location and destination information
Full Control

Traffic Light

Read-Only

Can be tracked and displayed on a dashboard
Can be avoided as an obstacle
Can yield to higher priority agents
Can stop at intersections to prevent simple conflicts
Can reroute to avoid or resolve complex conflicts
Can reroute for optimal traffic flow
Future work
Native Integration: Negotiation takes place on board the robot and takes advantage of the robot's perceptionTraffic Negotiation

The robots will negotiate whether to wait
for each other or reroute depending on
which is computed to require less time.

Traffic Negotiation
When rerouting is not possible, therobots will wait on each other for as longas necessary.

Head-to-head conflicts are anticipatedwithin the long stretches, and one robot
in the head-to-head conflict will stay outof the long lane while the other robot isapproaching.

Shared Infrastructure
●
●
●

Doors
Elevators (Lifts)
Workcells

The rmf_fleet_adapter library synchronizes robot traffic withthe use of shared infrastructureShared Infrastructure
Robots can negotiate their trafficthrough a shared door.

Shared Infrastructure
Robots can also negotiate traffic aroundothers that are using doors.

Shared Infrastructure
Robots can negotiate the use of elevators.
RMF integration allows any RMF-compliant
robot to use any RMF-compliant elevator.
If certain elevators should be reserved forcertain robots, that can be expressed in thenavigation graphs that are used by thefleets.

Shared Infrastructure

Workcells, such as robot arms, havestandardized interfaces that allow thefleet adapter to coordinate pickups,
drop-offs, docking procedures, or otherinteractions between robots.

Task Management●●●Customize
Dispatch
ExecuteWhat is a "Task"?
Task Description
serializable data structure thatcan be interpreted into...

Predictive Model

Runtimeinputs: (initial state prediction, robot description)
output: predicted state after task completion

generates a sequence of task "phases"

Provided to a multi-agent task planner tosearch for a "minimum-cost" assignmentof tasks to robots

Task Phase⚠ Current Scope ⚠
The current implementation assumes each task is assigned toone mobile robot and that individual tasks do not depend on each other.
Future versions of RMF will support
multi-agent tasks and constraints between tasks.

monitors state of robot and infrastructure to issuecommands (e.g. navigate to location, open door,
summon elevator) to fulfill an objective of the task
Human operators or external systems canrequest that a phase is skipped or repeated.
This is helpful if a phase did not go as intended.

Task Descriptions

More detailed instructions:
https://osrf.github.io/ros2multirobotbook/task_new.html

Simple, premade

Custom, composed{{"category": "compose","description": {"detail": "Drop off medication and then greet the patient","phases": [{"activity": {"category":"pickup","description": {"place":"L2_pharmacy","items": [{"sku":"48052", "quantity": 2}]}}},{"activity": {"category":"dropoff","description": {"place":"L3_ward10_bed4","items": [{"sku":"48052", "quantity": 2}]}},"on_cancel": [{"category":"dropoff","description": {"place":"L2_pharmacy"}}]},{"activity": {"category":"greet","description": {"place":"L3_ward10_bed4","language":"Hokkien"}}}]}"category": "delivery","description": {"pickup": {"place": "L2_pharmacy","payload": [{"sku": "48052", "quantity": 2},{"sku": "37981", "quantity": 1}]},"dropoff": {"place": "L3_ward32_bed4"}}}

Common tasks can be given simple premade
description schemas with a minimal set ofparameters to fill in
Each category is associated with its own
description schema that can be interpreted bytask planners and executors.
}

Allocation of tasks
A* Solver●

Priority Assignment○ Add a Priority field to task request○ During node expansion, check if newnode assignments are valid■ Valid = high priority tasks are
assigned prior to low priority ones
○ If invalid, f(n) = g(n) + h(n) * penalty●

Finishing Task
○ Automatically include a task that the robot has
to perform at the end of its assignments○ Park, ChargeBattery, etc●

Fleet adapters automatically replan task
assignments when a task is cancelled●

Fleet adapters use rmf_battery to check they should
automatically return an idle robot to its charger

Execution
Interrupt / Resume / Cancel●●●Tasks can be interrupted at any time
○ Not all actions can be interrupted○ Non-interruptible actions will finish, and then the task will switch to
an interrupted state
When a robot's task is interrupted, the robot can be given directcommands without conflicting with any commands RMF○ Useful when a robot needs to be teleoperated
Tasks can be canceled at any time
○ When a task is canceled, it will perform a "cancelation sequence"that depends on how far the task progressed

THANK YOU!!!

Questions?