An Integrated Distributed
Simulation Environment
weaving by Hakoniwa and mROS 2
Hideki Takase (The Univ. of Tokyo)
Shintaro Hosoai (The Univ. of Tokyo)
Tatsuya Fukuta (INTEC Inc.)
Mitsutaka Takada (Nagoya Univ.)
Shin Kuboaki (Change Vision, Inc.)
Takashi Mori (ESM, Inc. & Hakoniwa Lab.)We are TOPPERS/Hakoniwa WG!!
• Hideki Takase• Tatsuya Fukuta• Shin Kuboaki• @takasehideki• @fudekunjp• @kuboaki• The Univ. of Tokyo• INTEC Inc.

• Change Vision, Inc.

• EMB/IoT PF, ROS• Cloud, Robotics• MBD/MDD, App• Shintaro Hosoai• Mitsutaka Takada• Takashi Mori

• @s-hosoai• @mitsu48• @kanetugu2020• The Univ. of Tokyo• Nagoya Univ.

• ESM & Hakoniwa Lab• IDE, Modeling• Kernel, Automotive• Tech Leader!!

• About TOPPERS Project (Toyohashi OPen Platform for Embedded Real-time Systems)• NPO in Japan to promote embedded systems technology and its industry by developing andreleasing high-quality open-source software, especially in real-time kernels and components• Hakoniwa WG: working group to establish simulation technology for the IoT fields2

Agenda1. What is Hakoniwa?
2. Our Past Achievements, especially for ROS Robot
3. Latest Update: hakoniwa-mros2sim
4. Fusion of Virtual and Real weaving by Hakoniwa and mROS 2
5. Wrapping Up

3

Agenda1. What is Hakoniwa?
• What’s the issue• Concept: HUB to Everything for Simulation!
• Key Functions: Hakoniwa Core, Conductor and Protocol Data Unit
2. Our Past Achievements, especially for ROS Robot
3. Latest Update: hakoniwa-mros2sim
4. Fusion of Virtual and Real weaving by Hakoniwa and mROS 25. Wrapping Up

4Issue and Background• IoT system spans a variety of technical domains,

so its development requires the collaboration ofexperts from different fields.
• Integration test and validation are so challenging
due to these technologies’ diversity.
• When a problem unfortunately happens,investigating the cause and its path
becomes intricate.
• Costs for conducting
empirical experiments
also increase significantly.

5

Let’s bring them into a BOX!!
• Arranging various technologies according to everyone's preferences• Trying out various combinations as often we want• Observing them according to everyone's preferences into the same box• Bringing together things (e.g., software) from various tech fields to perform

end-to-end integration, and verification on the desk!
-> Developing IoT/robot systems in a virtual environment Hakoniwa
Robot Control
Engineers

Embedded Systems
Engineers

NetworkEngineers

Cloud
Engineers

Web
Engineers

BTY, why ”Hakoniwa”?
- Hako(箱) = box- Niwa(庭) = garden
a miniature garden,
one of the traditional JP
cultures (like “Bonsai”)

6Concept

Hakoniwa APIs
(for prog. Languages)

GazeboHakoniwa Library(for Physics Engines)

Hakoniwa
Core(C/C++)

Hakoniwa Daemon(for Remote Access)

Hakoniwa Commands(for Test Automation)• HUB to Everything for Simulation!
• not just a simulator,
but also a framework
to construct simulators• Hakoniwa Assets:
the components ofthe simulation target

PyBullet7

Hakoniwa Core
• Functions for the seamless integration of various assets
Hakoniwa
Asset

Microcontroller
Simulator

Visualization Tool

Scheduling
AssetManagement

Hakoniwa
CoreSynchronization& Communication

Physics
Simulator

TimeManagement

Automated TestingTool8

Hakoniwa Conductor• Mediate simulation between assets on Hakoniwa• gRPC based communication between the server and clients
Clients-side

Server-side• registration of Hakoniwa assets• start/stop/reset of each simulation

Hakoniwa
assets
箱庭アセット
箱庭アセットServer for
Hakoniwa Conductor

マシン２• connect to the server
マシン２
as the Hakoniwa assets
Client for
Hakoniwa
Conductor

Hakoniwa
assets
箱庭アセット
箱庭アセットHakoniwa API

Shared Memory
Hakoniwa Core functions

9

Protocol Data Unit (PDU)• Common data format to exchange in Hakoniwa• Function design is separated to asset-dependent and -independent• An example between Unity and ROS (topics)• Note: Unity and ROS are positioned as one of the "assets" in Hakoniwa

10Distributed Simulation with Hakoniwa
Computer1

Computer2

Server-side Hakoniwa Conductor
Client-side Hakoniwa Conductor

Computer3

Hakoniwa Proxy for PDU
Computer4

Hakoniwa Asset

11

Agenda1. What is Hakoniwa?
2. Our Past Achievements, especially for ROS Robot• 4 Prototypes to Ensure the Hakoniwa’s Concept• hakoniwa-ros2sim: ROS controlled Robot Simulation• Actual Use Cases
3. Latest Update: hakoniwa-mros2sim
4. Fusion of Virtual and Real weaving by Hakoniwa and mROS 2
5. Wrapping Up

12

Hakoniwa Prototypes Previously Achieved

Microcomputer-controlled
robot simulation
Collaborative simulation ofmultiple robots
Integration withreinforcement learning
ROS controlled
Robot Simulation

13

ROS controlled Robot Simulation• Docker for the platform• pre-built container image
for building & executing ros2app• able to use on multiple environments(Windows/WSL2 as the default)• Unity for physics and visualization• TurtleBot3 as the reference• ROS-TCP-Endpoint for
assets communication

14

Actual Use Cases• Athrill (micro-controller simulator) for ET-robocon https://www.etrobo.jp/• Group exercises (PBL) in some Japanese universities• Automotive software education for embedded engineers
https://github.com/ncesnagoya/autosar_os_training

15

Agenda1. What is Hakoniwa?
2. Our Past Achievements, especially for ROS Robot
3. Latest Update: hakoniwa-mros2sim• Introduction of mROS 2• [FYI] ROSCon JP 2023 Workshop• hakoniwa-mros2sim: Simulator for mROS 2 app by Hakoniwa Tech.
4. Fusion of Virtual and Real weaving by Hakoniwa and mROS 2
5. Wrapping Up

16

What is mROS 2?

mROS 2
on EMB board
better perf. & mem. usage
partially compatible with rclcpp
only for Topic comm.,
and many unsupported features
such as QoS, Service,,,https://vimeo.com/showcase/9954564/video/767140724

17

Currently Supported

watchlater!!

18

News!!

https://roscon.jp/
2023/#workshop

contents is public!!
(only in JP:D19

hakoniwa-mros2sim :D• Target: mros2 app & robot

Pi:Co Classic3 (MicroMouse)
https://rt-net.jp/products/pico-classic3/https://github.com/mROSbase/rcjp2023_mros2

20

Agenda1. What is Hakoniwa?
2. Our Past Achievements, especially for ROS Robot
3. Latest Update: hakoniwa-mros2sim
4. Fusion of Virtual and Real weaving by Hakoniwa and mROS 2• Hakoniwa’s Future Vision for Robotic Service• Expected System in the Future and its Architecture Design• Integration of mROS 2 into Hakoniwa Communication• Demonstration!!
5. Wrapping Up21

Our Future Vision for Robotic Service
What to do?

How to Integrate?

A new trend!!

AI Analysis
SpaceGenerative AI AgentSpaceGamingSpace

Challenges faced in
the embedded
systems domain

Cloud Service

Robot ServiceNetworks

Virtual Service

Challenges faced in
the cloud domain

How to Integrate?

Coverage of ROS

Coverage of ROS
Robot
Robot
Robot
Robot
Robot
Controller
Robot
Controller
Controller

Micro
Micro
Controller
MicroController
Controller

22

Fusion of Virtual and Real• Integration of Virtual and Real in Simulation
Generative AI AgentSpaceAI Analysis
SpaceVirtualGaming
SpaceReal
Robot

23

Expected System in the Future
Virtual Signal

Virtual World

Virtual Robot(Digital Twin)

AI Agent

Virtual People(Digital Avatar)

AR Agent
Real Robot

Real WorldAR Agent
People24Architecture Design: Overview
CentralServer(ROS)(MQTT)
CloudNetwork

Wireless Network
Virtual World

mROS

Infrastructure

Real Robot

Virtual Signal

SensorActuator

Virtual Service

Real WorldAR
DeviceVirtual People

Virtual Robot

mROS

AI Agent
Virtual/Real Map

Cloud Server

25Architecture Design: Detail
AR
Application

AR COM PROTOCOL
mros2

AR Device

Embedded Device(Real Robot)

ROS

mros2-posix

Hakoniwa Asset

Hakoniwa Asset

Hakoniwa Asset

AR Service(Python)

AI Agent(Python)

VirtualAgent
Agent
VirtualVirtualAgent
VirtualAgentControllers
Controllers
Controllers
Controllers(Python)(Python)(Python)(Python)Hakoniwa AssetAPIs(Python)APIs(C/C++)
mros2
Embedded Device(Infrastructure)

Shared Memory(PDU)
Hakoniwa CoreAPIs(Python)APIs(C#)

Hakoniwa Asset

Hakoniwa Asset
Field

Virtual/Real Map(Python)
VirtualSignalMac (Apple Silicon)

Virtual
Robot

AI
Agent

VirtualPeople26

but Why mROS 2?
• Integration of mROS 2 communication into Hakoniwa!
• lightweight but direct communication on RTPS
• mros2-posix can run on general-purpose OSes (even on macOS)-> support mros2 communication as one of Hakoniwa PDUs!!
Host ComputerEmbeddedDevice
Hakoniwa
Conductor(Server)

ROS 2 app

ROS 2 ProxyUnity

箱庭依存部

PDU shared memory

mros2posix

RTPS comm.
mros2RTOS

Hakoniwa CoreMac/Ubuntu

Embedded Device

27

Demonstration!!
Hakoniwa/Unity

ROS node

AR application

actual robot

Hakoniwa Core28

Agenda1. What is Hakoniwa?
2. Our Past Achievements, especially for ROS Robot
3. Latest Update: hakoniwa-mros2sim
4. Fusion of Virtual and Real weaving by Hakoniwa and mROS 2
5. Wrapping Up

29

Wrapping Up!
• Introduction of Hakoniwa (箱庭)• HUB to Everything for Simulation!
not just a simulator, but also a framework to construct simulators• hakoniwa-ros2sim: our past achievement for ROS robot• hakoniwa-mros2sim: new simulator for mros2 robot• Fusion of Virtual and Real weaving by Hakoniwa and mROS 2• Integration of mROS 2 into Hakoniwa Communication• Expected System in the Future and its Architecture Design• What’s Next??
• Open up the future by increasing the adoption in actual development scenes• and, enrich documentation, especially in English ;(30