Simulation of Highly Dynamic Omnidirectional Robots in Isaac Sim
Marvin Wiedemann - Fraunhofer Institute of Material Flow and Logistics© Fraunhofer IML

From Highly Dynamic Omnidirectional Real-World Robots…

2© Fraunhofer IML© Fraunhofer IML

Public information

… to Highly Dynamic Omnidirectional Simulation Models!

3© Fraunhofer IML

Public information

AgendaActuators & Dynamics

01

Simulation© Fraunhofer IMLSensors

02

03

5

How to model the robot in simulation?

How to replicate the sensors of the robot?

Communication & Interfaces
How to replicate the robots’ interfaces?

Public informationAgendaActuators & Dynamics

01

Simulation© Fraunhofer IMLSensors02

03

6

How to model the robot in simulation?

How to replicate the sensors of the robot?

Communication & Interfaces
How to replicate the robots’ interfaces?

Public information

The Demonstrator: O³dyn - a Next-Generation Logistics Robot

Video last GTC?
New images of Odyn?

O³ = Omnidirectional, Outdoor, Open-Source
dyn = Dynamics
Seite 7

© Fraunhofer IML© Fraunhofer IML

Public information

Actuators & Dynamics

Get The Robot Into Simulation
Import from URDF

Import from CAD data8© Fraunhofer IML

Public information

The Resulting Simulation Model

Public information© Fraunhofer IML

Attention: The Wheel Modeling

Public information© Fraunhofer IML

Actuators & Dynamics

Smoothness is Key: The Roller Colliders
Auto-Generated Meshes
Convex Hull11

© Fraunhofer IML

Spheric Colliders

Convex Decomposition

6 spheres

Public information

11 spheres

Smoothness is Key: The Roller Colliders

Convex Hull

11 spheres
Public information© Fraunhofer IML

Actuators & Dynamics

Entering the ROS Ecosystem

Isaac Sim13rviz© Fraunhofer IML

Public information

Actuators & Dynamics

The Robot Base Controller
OmniGraph based Controller

Custom Controller•

Controller can be built in Isaac Sim•

Implementation via ROS or Python API•

Visual Scripting Language OmniGraph•

Controller node allows to implement robot
specific dynamics & behaviors (e.g., delay)
Isaac SimRobot
/cmd_vel
controller/odom/joint_states…

14© Fraunhofer IML

Public information/joint_commandActuators & Dynamics

Sim2Real Comparison – Do They Behave Similarly?
RealityGiventrajectories

Motion Capturerosbag/cmd_vel/poseMsg
Timing

Sim Ground Truth

Simulation
15© Fraunhofer IMLrosbag

Public information

Sim2RealGapSim2Real Comparison – Do They Behave Similarly?

Reality

Seite 16

© Fraunhofer IML© Fraunhofer IML

Public information

Simulation

AgendaActuators & Dynamics

01

Simulation© Fraunhofer IMLSensors

02

03

17

How to model the robot in simulation?

How to replicate the sensors of the robot?

Communication & Interfaces
How to replicate the robots’ interfaces?

Public information

AgendaActuators & Dynamics

01

Simulation© Fraunhofer IMLSensors

02

03

18

How to model the robot in simulation?

How to replicate the sensors of the robot?

Communication & Interfaces
How to replicate the robots’ interfaces?

Public information

Sensors

Available Sensor TypesCamera

LiDAROthers

19•••

RGB camera
Depth camera
Replicator for synthetic data generation••

PhysX-based LiDAR
RTX-based LiDAR••••

PhysX-based range sensors (generic, ultrasonic)
Force and contact sensors
IMUProximity© Fraunhofer IML

Public information

Structure only.
Slide design and content
missing.

Example:Multi-Layer LiDAR
Sensors
LiDAR – Choosing the Best Scanner

20© Fraunhofer IML© Fraunhofer IML

Public information

LiDAR
Sensors– Choosing the Best ScannerReality21© Fraunhofer IML© Fraunhofer IML

Public information

Simulation

AgendaActuators & Dynamics

01

Simulation© Fraunhofer IMLSensors

02

03

22

How to model the robot in simulation?

How to replicate the sensors of the robot?

Communication & Interfaces
How to replicate the robots’ interfaces?

Public information

AgendaActuators & Dynamics

01

Simulation© Fraunhofer IMLSensors

02

03

23

How to model the robot in simulation?

How to replicate the sensors of the robot?

Communication & Interfaces
How to replicate the robots’ interfaces?

Public information

Communication & Interfaces

TopicsBase Controller /cmd_vel, /odom

24© Fraunhofer IML

Joints/joint_state, /joint_command

TF/tf, /tf_static

Sensors/laser, /point_cloudCustom Msgs

/battery_soc, …

Public information

Communication & Interfaces

TopicsBase Controller /cmd_vel, /odom

25© Fraunhofer IML

Joints/joint_state, /joint_command

TF/tf, /tf_static

Sensors/laser, /point_cloudCustom Msgs

/battery_soc, …

Public information

Communication & Interfaces

TopicsBase Controller /cmd_vel, /odom

26© Fraunhofer IML

Joints/joint_state, /joint_command

TF/tf, /tf_static

Sensors/laser, /point_cloudCustom Msgs

/battery_soc, …

Public information

Communication & Interfaces

TopicsBase Controller /cmd_vel, /odom

27© Fraunhofer IML

Joints/joint_state, /joint_command

TF/tf, /tf_static

Sensors/laser, /point_cloudCustom Msgs

/battery_soc, …

Public information

Communication & Interfaces

Actions and Services

Real O³dyn
Actuators
PLCSensors

ROBOT BASE NODE
Action Server/load/control_lift

Simulation Model…

Joints28© Fraunhofer IML

Public information

Conclusion

Simulation of highly dynamic robots is challenging – but possible!
Take care of the collider modeling and use real-world data.

Isaac Sim offers a lot for robotics and has its strength within theGPU usage mainly for sensor simulation and rendering.

ROS is the bridge between simulation tools and the reality.

29© Fraunhofer IML

Public information

Check Out the O³dyn Simulation Model at Home!

30

O³dyn on your Nucleus

Open source @ Open Logistics Foundation

omniverse://localhost/NVIDIA/Assets/Isaac/2023.1.0/Isaac/Robots/O3dyn

https://git.openlogisticsfoundation.org/silicon-economy/simulation-model/o3dynsimmodel/© Fraunhofer IML

Public information

Contact
Marvin Wiedemann
AI and Autonomous Systems
marvin.wiedemann@iml.fraunhofer.deFraunhofer Institute for Material Flow and Logistics
Dortmund, Germany
https://www.iml.fraunhofer.de/en/

Thanks to:
Renato Gasoto2, Ossama Ahmed1, Anna Vasileva1, Elisabeth Drache1,Steffen Daniel1, Mrunal Hatwar1, Jan Finke1, and Sören Kerner1
1 Fraunhofer IML
2 Nvidia

Funded by