2015 IEEE International Conference on Advanced Intelligent Mechatronics (AIM)
July 7-11, 2015. Busan, Korea

Immersive VR Interface for Informationally Structured Environment
Yoonseok Pyo1 , Tokuo, Tsuji2 , Yuuka Hashiguchi1 , and Ryo Kurazume2

Abstract— This paper presents a new immersive VR interface
bridging cyber and physical worlds for Cyber Physical System(CPS). Informationaly structured environment (ISE) has been
proposed in service robotics so far. In ISE, environmental
information such as positions of objects, furniture, humans,
and robots is gathered by embedded sensor networks andstored in a database structurally. A service robot is able toutilize these information anytime and anywhere by connecting
to the network. In this paper, we introduce the ISE architecture
named ROS-TMS and a new cyber-physical interface for ROSTMS. The proposed system consists of an immersive wearable
display, a stereo camera, an optical tracking system, and an
environmental simulator, and is able to present forecasted
images with high reality based on the structured informationin ISE.
Index Terms— Cyber Physical System, Service robot, Immersive display, Environmental simulator

I. I NTRODUCTION
CPS (Cyber Physical System) is a system to develop an
efficient and affluent society by solving various issues inreal world utilizing a great amount of computer resources incyber world. After the NSF has identified the CPS as a key
research area in 2006, this concept has been attracting muchattention and a number researches have been presented sofar. In recent years, the concept of IoT (Internet of Things),
which connects all the objects and machines through theinternet and controls them efficiently, has been becoming
very popular. Besides, the CPS is to forecast an ideal world
based on a wealth of sensory information obtained throughIoT and huge computer power in cyber world, and plan andexecute optimum strategies for realizing the ideal physicalworld.
On the other hand, the concept of an informationallystructured environment (ISE) or an intelligent space (IS)has been proposed so far in the robotics research field. In
this concept, rich information about a daily life environment
where a service robot performs a variety of service tasks
to humans is acquired using embedded sensors and sensory
networks, and stored in a database structurally. All therequired information to perform proper service tasks by aservice robot, such as position of objects, furniture, human,or robots, are provided on-demand from a database anytime
and anywhere by connecting to the network.
1 Yoonseok Pyo and Yuuka Hashiguchi are with Faculty ofInformation Science and Electrical Engineering, Kyushu University,744, Motooka, Nishi-ku, Fukuoka, 8190395, JAPAN {pyo,hashiguchi}@irvs.ait.kyushu-u.ac.jp

2 Ryo Kurazume and Tokuo Tsuji are with the Department ofInformation Science and Electrical Engineering, Kyushu University,744, Motooka, Nishi-ku, Fukuoka, 8190395, JAPAN {kurazume,tsuji}@ait.kyushu-u.ac.jp
978-1-4673-9106-1/15/$31.00 ©2015 IEEE

A number of studies for the ISE has been presented so far,including the “Smart room” (Media Lab at MIT), “Intelligentroom” (AI Lab at MIT), “Robotic room” (The university ofTokyo), “Intelligent space” (The university of Tokyo), etc.
and various new researches have been reported to this day
actively [1–9]. The concept of the ISE is very closely related
to the CPS mentioned above in terms of the purposes andmeans.
We also have been proposing a framework for the ISE
named Town Management System (TMS) so far [10]. In thissystem, a variety of information regarding objects, furniture,humans, and robots which is acquired by distributed cameras,laser range finders, and RFID tag readers is stored to a
database and provided according to the request from a service
robot. We have been developing API libraries of the TMS
and releasing them.
In 2012, we started to develop a new version of TMS
named ROS-TMS, which adopts Robot Operating System(ROS) [11] as a middleware. Owing to the ROS, we canintegrate a variety of sensors and robots adaptively to avariety of environments. We have been developing more than
95 modules for the ROS-TMS so far such as sensing system
or motion planning for a service robot [12].
In this paper, we propose a new ISE platform for an indoor
environment based on the ROS-TMS technology. This platform consists of several humanoids-type and wheeled-typeservice robots, a wheeled chair robot, a robot refrigerator,
furniture such as desk, chair, bed, bookshelf, cabinet, etc.,
and various embedded sensors. The position and translationof objects, furniture, humans, and robots are detected andtracked by laser range finders, an optical tracker, RGBD cameras, RFID readers in intelligent cabinets and robot
refrigerators, etc. and stored to the database in ROS-TMS.
In addition, we also propose a new immersive VR interface
for the developed ISE platform which bridges cyber world
and physical world intuitively. The events which will occur
in real world are predicted and simulated though a computer
simulation according to the sensory information acquired by
distributed sensory networks. The simulation results are displayed immersively by a wearable VR interface. Therefore,
the person wearing this system has an experience for thesimulated events in cyber world with a high realistic manner
before it will happen actually in real world. In other words,
this system is a potent interface of the CPS which bridges
cyber and physical worlds.
The remainders of the paper are organized as follows. In
Section 2, some related works for an immersive VR interface
in the robotics field will be introduced. In Section 3, we
will introduce a ROS-TMS and ISE platform for an indoor

1766

User

environment. A new immersive VR interface for the ISE will
be proposed in Section 4.

Environmental
information
Service request

II. R ELATED WORK

Robot

III. I NFORMATIONALLY STRUCTURED ENVIRONMENTPLATFORM

Task command
Informationally structured environment

Fig. 1. Concept of ROS-TMS. ROS-TMS connects various sensors, robots,and a database based on Robot Operating System (ROS) in order to provide
proper service tasks for users.
Service Target

User

TabletTMS_UR : User Request
TMS_TS : Task Scheduler

ServerBoard PC
Sensor & Robot

TMS_SA
State Analyzer

TMS_RP
TMS_SS
Sensor System Robot Planning

TMS_SD
Sensor Driver

TMS_RC
Robot Controller

SensorRobot

TMS_DB : Data Base

An immersive VR interface has been mainly utilized in
the research on tele-robotics. Several applications using an
immersive VR interface have been presented, for instance,
an intuitive operation of a surgical robot such as da Vinci
Surgical System (Intuitive Surgical, Inc.) [13], surgical simulator [14], motion control of a space robot [15] and a rescue
robot [16].
An immersive VR interface can be categorized into twogroups. Surround screen projection system such as CAVE
[17] utilizes multiple screens and projectors, and display
whole directional images to walls, a floor, and a roof independently of a user’s line of sight. On the other hand, awearable device such as Google Glass or Epson Moverio
overlays a VR image onto a real image directly usingtransparent glasses, and have been attracting much attentions
since they have a high affinity for AR (Argumented reality),
MR (Mixed Reality), or First Person Vision. In recent years,
new wearable devices such as Oculus Rift have been produced as a wearable immersive VR interface. These devices
consist of a small display and an IMU (Inertial Measurement
Unit), and synthesize a virtual image corresponding to thehead direction. Thus, these wearable devices are simple andlow-cost comparing to the system with large screens andprojectors, and thus very popular in recent years.
Immersive Virtual Robotics Environment (IVRE) [18] at
Johns Hopkins University utilizes Oculus Rift and ROS, anddemonstrates a collaborative task with a robot and a humanin a virtual world. However this system is mainly for a safe
path/motion planning through virtual single human-robot
interaction, and human-environment interaction including
multiple objects, robots, and humans proposed in this paper
has not been considered.

DB
ServerFile
ServerFig. 2.
ROS-TMS architecture. ROS-TMS consists of more than 95
modules including sensor processing, robot planning, user interface, anda database. See the wiki page for more information [12].

We have developed a number of nodes for ROS-TMS andreleased them [12]. These nodes can be categorized into thefollowing modules.
TMS DB Environmental databases using MySQLTMS UR User interfaces using wearable devices or mobile devices
TMS TS Task schedulers based on ROS SMACHTMS SA State analyzer
TMS SS Sensor fusion module
TMS RP Motion planning module for service robots
TMS SD Sensor drivers
TMS RC Robot controllers
B. Indoor ISE platform for daily life environment

A. ROS-TMS
We have been developing an informationally structured
environment architecture named ROS-TMS [12]. The ROSTMS is a mechanism which connects various sensors distributed in an environment such as laser range finders,cameras, RFID tag readers, or proximity sensors, various
robots, and a database based on Robot Operating System(ROS) [11]. The ROS-TMS is able to acquire, store, andanalyze environmental information, and plan and control
robot motion adaptively. ROS is an open source middleware
which absorbs differences of various sensors and robots,and provides uniform interface to users. In ROS-TMS, eachcontroller of a robot and a sensor is implemented as anode (a process for a particular task) in ROS and thus thecommunication between nodes and addition and removal ofa new node are quite simple and easy. Figures 1 and 2 show
a concept of ROS-TMS and its architecture.

Figure 3 shows the developed indoor ISE platform simulating a daily life environment and the environmental simulator(Choreonoid [19]). Choreonoid is an integrated robotics
GUI environment with a dynamics engine, which allows
users to add their own functions as plugins on the basisof various basic robotics functions. We utilize Choreonoid
and its motion planner plugin named GraspPlugin [20] tosimulate the indoor ISE platform. Using the GraspPlugin,various robot motions can be designed with a simple mannerconsidering their physical properties and mutual interference.
Collision detection function is also provided by GraspPlugin.
The indoor ISE platform consists of several service robots(SmartPAL IV and V (Yaskawa Electric), KXP (Katana,
Pioneer 3 AT), Kobuki (Yujin Robotics), a wheeled chairrobot, and a robot refrigerator), furniture (desk, chair, bed,
bookshelf, cabinet, etc.), and embedded sensors. The position
and translation of objects, furniture, humans, and robots

1767are detected and tracked by laser range finders (UTM30LX-EW, Hokuyo Automatic), an optical tracker (Bonita,
Vicon), RGB-D cameras (Kinect, Xtion), RFID readers in
intelligent cabinets and robot refrigerators, etc. and stored tothe database, TMS DB.
Based on the environmental data stored in TMS DB, thephysical simulation can be conducted using Choreonoid in
the virtual environment which reflects the real situation in
the indoor ISE platform exactly. Therefore, the developedindoor ISE platform and the Choreonoid simulator can beconsidered as a physical and cyber systems in the CPS,
respectively.
Tracker (Vicon)

loadcell
RFID antenna
Fig. 4. Object detection using Intelligent Cabinet [21]. The names and thepositions of objects are detected by a RFID tag reader and load cells.

robot refrigerator
SmartPal IVIntelligent cabinet
SmartPal V

Fig. 5. Detected objects in the Intelligent Cabinet. Real and cyber worlds
are synchronized in real time.

Laser range finder
KXP robot

and carry task controlled and processed by the task scheduler”SMACH” [23] in TMS TS.

Wheeled chair robot(a) Informationally structured room

IV. I MMERSIVE VR INTERFACE BRIDGING CYBER AND
PHYSICAL WORLDS FOR CPS
A. System configuration(b) Simulated room (Choreonoid)
Fig. 3. Information structured room (physical system) in an real environment and simulated room (cyber system) in Choreonoid simulator.

Figures 4, 5, and 6 show the examples of the object detection function in TMS SS. Objects placed in the intelligent
cabinet or on a table are detected as follows [21]. In theintelligent cabinet, a RFID tag reader is installed under thebase plate and the object is identified by reading a RFIDtag
attached on the bottom of the object. The position of theobject is measured by four load cells which support the baseplate as shown in Fig. 4,. The name and the position of thedetected object are registered in the TMS DB and displayedin Choreonoid (Fig.5) in real time. On the other hand, theobject placed on the table is scanned by the Xtion mounted
on a humanoid robot (SmartPal V, Fig.7) and labeled by the
database using CSHOT [22].
When a user request is detected by TMS UR, Choreonoid
refers TMS DB and simulates a safety motion by a service
robot which will not collide with the environment using the
GraspPlugin. Robot motions designed above are transmitted
to the motion control nodes in TMS RP and TMS TS, andservice tasks such as a fetch and carry task by a service
robot and a robot refrigerator, or a transportation task using
a wheeled chair robot are executed. Figure 8 shows the fetch

This section introduces an immersive VR interface bridging cyber and physical worlds to realize the concept of CPS.
This system consists of an immersive VR display (Oculus
Rift DK2, Oculus VR, Fig. 9), a stereo camera (Ovrvision,
Wizapply, Fig. 9), an optical tracking system (Bonita, Vicon,
Fig. 10), and an environmental simulator (Choreonoid, Fig.
11).
The immersive VR display (Oculus Rift DK2) is a wearable display which enables a binocular stereo by displaying
left and right images individually. A three-axis attitude sensor

Color and range images by Xtion

Detected objects
CSHOT in database

Fig. 6. Object detection on a table using a RGB-D sensor (Xtion) andCSHOT descripter.

1768

XtionVelodyne
HDL-32eFig. 7. Xtion mounted on the humanoid robot (SmartPal V). A laser
scanner (Velodyne HDL-32E) is also attached.

is also installed in it. The head orientation measured by thethree-axis attitude sensor is sent to the PC. On the other
hand, the position of the immersive VR display in the room
is tracked by optical markers and the optical tracking system(Bonita) with the accuracy of less than few millimeters(Fig.9). Based on the position and orientation information of
the immersive VR display, the VR images of the room fromthe same view point are synthesized and displayed on theimmersive VR display. Figure 12 shows the graphic pipeline
using Oculus Rift DK2, Bonita Vicon, and Choreonoid.
The scene is firstly modelled using environmental information stored in the TMS DB, then the OpenGL-based graphics
engine in Choreonoid synthesizes distorted stereo imagesby calling a distortion function in libOVR (LibOVR 0.4.4,
Oculus VR). Finally, stereo images are displayed on the left
and right displays in the Oculus Rift individually.
The stereo camera (Ovrvision) is mounted on the front
chassis of the Oculus Rift and captures images from the same
viewpoints of the person wearing the Oculus Rift. Captured
stereo images are displayed on the left and right imagesof the Oculus Rift individually, and thus we can see thesurrounding real images without detaching the Oculus Rift.
Figure 14 shows some synthesized stereo VR images for
Oculus Rift and real images captured by the stereo camera
when the person wearing the system walks through the room
as shown in Fig. 13. In these examples, there is no serious
time-delay and both images are almost synchronized.
We can see that proper VR images which are quite similar
to real images are created and displayed at any positionin the room according to the change of head position andorientation even if the person wearing this system walks
anywhere in the room.
B. Time shift effect
The most important characteristic of the proposed systemis the so-called time-shift effect. We can show the highly
realistic VR images forecasted by the Choreonoid simulator.
The simulation is executed based on the current real statusof objects, furniture, humans, and robots sensed and storedby the ROS-TMS.

Fig. 8. Fetch and carry task by service robot. Robot motions are planed
and executed by ROS-TMS. Left column shows the current state in a wholeprocedure executed by SMACH state machine. Dotted lines indicate thedesigned trajectory of the service robot.

Figure 15 shows the examples of the time shift effect by
the proposed system. The left column of Figure 15 shows VR
images from the eye position of the person who ordered afetch and carry task to the service robot. The robot performs
the service task as shown in Fig. 8. The designed trajectoryof the service robot can be seen as dotted lines depicted
on a floor in Fig. 15. Moving objects can be reflected on
the simulator in real time if these objects are detected byembedded sensors such as LRFs and cameras. Note that theseimages are synthesized before the service task is actually
executed by the service robot. With this system, we can see
the forecasted synthesized images beforehand and confirm
whether the planned service action is safe and proper or not.
with our own eyes. If these actions are not appropriate, we
can stop and rearrange the service action to be safe. Real
images taken by the stereo camera when the service robot
executes the planned service task actually are shown in theright column of Fig. 15.
V. C ONCLUSION
This paper introduced the informationally structured environment architecture named ROS-TMS and the indoor
platform for Cyber Physical System, CPS. Then we proposed

1769

Bonita, Vicon

Oculus Rift DK2Position

Orientation

Immersive display
Stereo camera

Choreonoid

Gyro sensor

TMS_DB

Optical marker

Modeling
Renderer (OpenGL)

libOVR 0.4.4

VR Display

Fig. 9.
Immersive VR display device consisting of Oculus Rift DK2(immersive stereo display), Ovrvision (stereo camera), and Vicon Bonita(optical position tracker)

Fig. 12. Graphic pipeline of the proposed immersive VR interface. The
position of the VR display is tracked by an optical tracker (Bonita, Vicon)
and the pose is sensed by a three-axis attitude sensor in Oculus Rift DK2.
The VR stereo images of the real world from the current position of theuser are synthesized and displayed on the immersive VR display in real
time.

4
5

Fig. 10.

3
2
1

Optical position tracking system (Bonita, Vicon)

76

the new immersive VR interface using Oculus Rift, Bonita
Vicon, and Choreonoid simulator which bridge cyber andphysical worlds to realize the concept of CPS. As a typicalexample of the CPS, we demonstrated the time shift effectby the proposed system. The proposed system is one of theexamples of the CPS implementation, which forecasts an
ideal world, designs the optimum strategy, and presents thepredicted results with high reality using the immersive VR
display.
The current system is, however, not practical for natural
human-system interactions due to its size. The proposedsystem can be applied with smart glasses such as Moverio(Epson) or Google glass by overlaying simulated images onreal scene.
This research is supported in part by a Grant-in-Aid for
Scientific Research (A) (26249029) and the Japan Scienceand Technology Agency (JST) through its ”Center of Innovation Science and Technology based Radical Innovation and

Fig. 11.

Fig. 13.

Walk-through path in a room

Entrepreneurship Program (COI Program).”Synthesized VR stereo images by Choreonoid

1770

R EFERENCES[1] Hiroshi Noguchi, Taketoshi Mori, and Tomomasa Sato. Automatic
generation and connection of program components based on rdf
sensor description in network middleware. In IEEE/RSJ International
Conference on Intelligent Robots and Systems, pages 2008–2014,
2006.
[2] Yuka Kato, Toru Izui, Yosuke Tsuchiya, Masahiko Narita, Miwa
Ueki, Yoshihiko Murakawa, and Keijyu Okabayashi. Rsi-cloud for
integrating robot services with internet services. In IECON 2011 37th Annual Conference on IEEE Industrial Electronics Society, pages
2158–2163, 2011.
[3] Horst-Michael Gross, Christof Schroter, Steffen Muller, Michael
Volkhardt, Erik Einhorn, Andreas Bley, Tim Langner, Christian Martin,and Matthias Merten. I’ll keep an eye on you: Home robot companion
for elderly people with cognitive impairment. In IEEE International
Conference on Systems, Man, and Cybernetics, pages 2481–2488,
2011.
[4] Moritz Tenorth, Alexander Perzylo, Reinhard Lafrenz, and Michael
Beetz. The roboearth language: Representing and exchanging knowledge about actions, objects, and environments. In IEEE International
Conference on on Robotics and Automation, pages 1284–1289, 2012.
[5] Young-Guk Ha, Joo-Chan Sohn, and Young-Jo Cho. Service-oriented
integration of networked robots with ubiquitous sensors and devices
using the semantic web services technology. In 2005 IEEE/RSJ
International Conference on Intelligent Robots and Systems (IROS),
pages 3947–3952, Aug 2005.

1

Shelf
Shelf

Robot

Robot
Robot

2

Planned trajectory

Robot

Robot
Robot

3
Object4

Object(a) Virtual task planned previously(b) Actual task

Fig. 15. VR and actual images for fetch and carry task. Real task is
executed after the task is confirmed to be safe and executable previously
using the Choreonoid simulator.

5

6

7

Fig. 14. Walk-through images of a room displayed on the stereo screens ofOculus Rift. VR images from the current view points (right column) which
are quite similar to real images (left column) are appropriately synthesized.

[6] Md. Jayedur Rashid, Mathias Broxvall, and Alessandro Saffiotti. A
middleware to integrate robots, simple devices and everyday objects
into an ambient ecology. Pervasive and Mobile Computing, 8(4):522–
541, 2012.
[7] Miki Sato, Koji Kamei, Shuichi Nishio, and Norihiro Hagita. The
ubiquitous network robot platform: Common platform for continuous
daily robotic services. In 2011 IEEE/SICE International Symposium
on System Integration (SII), pages 318–323, Dec 2011.
[8] Abdelghani Chibani, Yacine Amirat, Samer Mohammed, Eric Matson,
Norihiro Hagita, and Marcos Barreto. Ubiquitous robotics: Recent
challenges and future trends. Robotics and Autonomous Systems,
61(11):1162–1172, 2013.
[9] Abdelghani Chibani, Antonis Bikakis, Theodore Patkos, Yacine Amirat, Sofiane Bouznad, Naouel Ayari, and Lyazid Sabri. Using cognitive
ubiquitous robots for assisting dependent people in smart spaces. InIntelligent Assistive Robots, pages 297–316. Springer, 2015.
[10] Ryo Kurazume, Yumi Iwashita, Koji Murakami, and Tsutomu
Hasegawa. Introduction to the robot town project and 3-d co-operative
geometrical modeling using multiple robots. In 15th International
Symposium on Robotics Research (ISRR 2011), 2011.
[11] Morgan Quigley, Ken Conley, Brian Gerkey, Josh Faust, Tully Foote,

Jeremy Leibs, Rob Wheeler, and Andrew Y Ng. Ros: an open-source
robot operating system. In International Conference on Robotics andAutomation, ser. Open-Source Software workshop, 2009.
[12] https://github.com/irvs/ros_tms/wiki.
[13] John F. Boggess. Robotic surgery in gynecologic oncology: evolutionof a new surgical paradigm. Journal of Robotic Surgery, 1:31–37, 3
2007.
[14] Morten Bro-Nielsen, Joseph L Tasto, Richard Cunningham, and Gregory L Merril. Preop endoscopic simulator: a pc-based immersive
training system for bronchoscopy. Stud Health Technol Inform, 62:76–
82, 1999.
[15] http://www.engadget.com/2013/12/23/
nasa-jpl-control-robotic-arm-kinect-2/.
[16] Henrique Martins, Ian Oakley, and Rodrigo Ventura. Design andevaluation of a head-mounted display for immersive 3d teleoperationof field robots. Robotica, FirstView:1–20, 1 2015.
[17] Carolina Cruz-Neira, Daniel J. Sandin, and Thomas A. DeFanti.
Surround-screen projection-based virtual reality: The design and implementation of the cave. In Proceedings of the 20th Annual Conference on Computer Graphics and Interactive Techniques, SIGGRAPH’93, pages 135–142, New York, NY, USA, 1993. ACM.
[18] http://cirl.lcsr.jhu.edu/research/human-machine-collaborative-systems/ivre/.
[19] Shin’ichiro Nakaoka. Choreonoid: Extensible virtual robot environment built on an integrated gui framework. In Proc. of the 2012
IEEE/SICE International Symposium on System Integration (SII2012),
pages 79–85, 2012.
[20] http://choreonoid.org/GraspPlugin/i/?q=en.
[21] Oscar Martinez Mozos, Tokuo Tsuji, Hyunuk Chae, Shunya Kuwahata,
Tsutomu Hasegawa, Ken’ichi Morooka, and Ryo Kurazum. The
intelligent room for elderly care. In 1The fifth International Workshop on the Interplay between Natural and Artificial Computation(IWINAC2013), 2013.
[22] Federico Tombari, Samuele Salti, and Luigi Di Stefano. A combined
texture-shape descriptor for enhanced 3d feature matching. In Image
Processing (ICIP), 2011 18th IEEE International Conference on, pages
809–812, Sept 2011.
[23] Jonathan Bohren, Radu Bogdan Rusu, E. Gil Jones, Eitan MarderEppstein, Caroline Pantofaru, Melonee Wise, Lorenz Mosenlechner,
Wim Meeussen, and Stefan Holzer. Towards autonomous robotic
butlers: Lessons learned with the pr2. In International Conferenceon Robotics and Automation, 2011.

1771