Distributed robotics simulator
with Unreal Engine
Larry Ng(Rapyuta Robotics)
Yu Okamoto (Rapyuta Robotics)

https://rclue.readthedocs.io
https://rapyutasimulationplugins.readthedocs.io

Select investors
170 employees,
35 nationalities
66 patents
13 granted
USD 110M+
raisedRapyuta Robotics - confidential information2

Why use Unreal Engine(UE)●Many tools have already been developed by the GameIndustry○ Photorealistic rendering■ data generation for machine learning■ demo/marketing○ Game assets are available. Epic marketplace.
○ Blueprint Editor○ Can get talent from Game Industry○ Physics engine

FF IIV Remake
https://www.jp.square-enix.com/ffvii_remake/about/index.html

Fortnitehttps://www.epicgames.com/fortnite/en-US/homerclUE is a ROS 2 client libra ry for UE4
●
●
●
●
●

ROS 2 Nodes are implemented as UE Actors
Allows for Publishers/Subscribers etc to be defined
inside UE4
Supports: Ubuntu, ROS 2 Foxy, UE 4
ROS components can be accessed from Blueprint
no bridge node

Unreal Networking: Replication and RPC
Replication:
Copy State fromServer to Clients
every tick

RPC:
Call a function inserver from theclient (and ViceVersa)

Client
1. Player Controller
receives PlayerInput
2. Player Controller
moves local Actor
Server
1. Server Side PlayerController applies
new state to Serverside Actor
Other clients
1. Server Actor State is
replicated to allother Clients

Our Use of Unreal Networking
Replication:
Duplicate
Position ofServer’s Robots
to all otherClients

RPC:
Move Server’s
copy of the
RobotClient
1. Robot ROS Nodereceives ROS Topic
2. Robot moves basedon cmd_vel
3. Pose and Velocity
sent to NetworkController
Server
1. Server Side NetworkController moves
relevant robot toPoseOther clients
1. Server’s Robot’s
Pose are replicatedto other clientsDistributed Simulator architecture
Goal: Simulating large number of robots● Separate Viewer from server○ GPU is only in the Viewer client,
reduces costs significantly● Separate Robot software from server○ ROS 2 Node only exist in client○ Sensor simulation at client● No delay between Robot app andsimulator ROS 2 Node
○ However, delays can occur whensyncing a large number of clientsto the server

Standalone vs distributed simulation

Standalone vs Distributed Simulation- 10 robots● Standalone Instance specs
TypeCPU(Cores)

RAM (GB)GPU(Yes/No)

Instances

Cost (¥/month)Standalone

856

Yes

1~$550● Distributed Simulation Instance specs (5 robots per Robot Client)
TypeCPU(Cores)

RAM (GB)GPU(Yes/No)

Instances

Cost (¥/month)Server

24No

1~$35

Robot Client

2*2

4*2

No

1*2

~$35*2Viewer Client

24Yes

1~$200Totals816

Yes

4~$305

Demolink: https://roscondemo-kqdbp.ep-r.io/
backup video: https://drive.google.com/file/d/18XCkvZkeP_FZLiUbffVMac_EE5yiEcyu/view?usp=sharing

For Further Information
Check out our documentation!

https://rclue.readthedocs.io

Or!
Check us out at our boothhttps://rapyutasimulationplugins.readthedocs.io

Thank you!
We’re Hiring!

hiring-tyo_sig@rapyuta-robotics.com

For Further Information
Check out our documentation at:https://rclue.readthedocs.io
https://rapyutasimulationplugins.readthedocs.io

Or!
Check us out at our boothStandalone vs Distributed Simulation- 10 robots● Standalone Instance specs
TypeCPU(Cores)

RAM (GB)GPU(Yes/No)

Instances

Cost (¥/month)Standalone856

Yes1~¥83,000● Distributed Simulation Instance specs (5 robots per Robot Client)
TypeCPU(Cores)

RAM (GB)GPU(Yes/No)

Instances

Cost (¥/month)Server24

No1~¥5,000

Robot Client

0.4*100.8*10No1*2~¥5,000*2

Viewer Client

24

Yes1

~¥30,000

Unreal Network: Replication and RPCs
x: pose
V: velocity
t: timeServer
Robot 1xs(ts + dts) = xc1(tc1) + u(tc1)* (dts + ts - tc1)
vs(ts + dts) = u(tc1)xc1(tc1), u(tc1), tc1
RPCxs(ts)
Robot Client 1
Robot 1 (Controlled in this client)xc1(tc1 + dtc1) = xc1(tc1) + u(tc1)*dtc1
vc1(tc1 + dtc1) = u(tc1)

cmd_vel inputu(tc1)

1.
2.
3.
4.
5.

Robot Client 2
Robot 1xc2(tc2) = xs(ts)Robot Client1 receives velocity input u.
Locally update state
Robot state and timestamp are sent to Server from Robot Client1
State update in Server
Server state and timestap are sent to Robot Client2

Replication

Why use Unreal Engine(UE)●●●●Photorealistic rendering○ data generation for machine learning○ demo/marketingGame assets are available. Epic marketplace.
Physics engine: PhysX*Editor：○ GUI→ easy/intuitive to edit environment.
○ Blueprint → Visual scripting. Creating logic from GUI FF IIV Remake
https://www.jp.square-enix.com/ffvii_remake/about/index.html○ Behaviour Tree → Human/non robot simulation*UE5 uses Chaos physics engine*List of Unreal Engine games
Behaviour tree editor

https://docs.unrealengine.com/4.27/enUS/InteractiveExperiences/ArtificialIntelligence/Behavio
rTrees/BehaviorTreesOverview/More About
Pick Assist
RobotsAnother Thing
About Pick Assist
RobotsA Bit More About
Pick Assist Robots

Also About PickAssist Robots

One More Thing..

And FinallyRapyuta Robotics is a spin-off from ETH Zürich,
which has produced many famous researchers including Einstein.

ラピュタロボティクス株式会社は、アインシュタインをはじめ、数々の著名研究者を輩出しているチューリッヒ工科大学（ETH Zürich）からスピンオフした大学発ベンチャー企業です。

Index

●
●
●
●ABCD

List

● A
● B
● C
● D

“A great testimonial about
Rapyuta Robotics’ cool andamazing pick assist robots.

”

By So m e o n eDividerDividerDividerThank you!