2012 IEEE/SICE International Symposium on System Integration (SII)
Kyushu University, Fukuoka, Japan
December 16-18, 2012

Virtual Telesar - Designing and implementation of a modular based
immersive virtual telexistence platform
MHD YAMEN SARAIJI, CHARITH LASANTHA FERNANDO, MASAHIRO FURUKAWA,
KOUTA MINAMIZAWA and SUSUMU TACHIAbstract— In this paper, we focus on designing acustomizable modular based virtual platform for modeling,simulating and testing telexistence applications, where thephysical parameters are preserved in the virtual environment;
for motor control of physical characteristics of robot as well as
sensory feedback of vision, auditory, haptic. We proposed“Virtual Telesar” which allows telexistence engineers to model a
prototype system before manufacturing it and to experience
how the final model will look-like, perform manipulations etc…
in real world without building it. The platform consists of threefeatures: first, the user can define a robot using predefinedmodular components. Second, the user can customize and tune
up parameters. Third, the user can have immersive experiencesof operating the robot with visual, auditory and hapticsensation. In this paper, we describe a design concept of Virtual
Telesar platform and report modeling result based on a physicalrobot and result of immersive experience with it.

Fig. 1 First person view of Virtual Telesar

I. INTRODUCTION

Telexistence allows a person to have a real-time sensationof being somewhere else than his or her current locationand to interact with the remote environment which might
be real, virtual, or a combination of both ‎[1]. In reality,
telexistence‎uses‎the‎operators‟‎natural‎movements‎to‎control‎a robot, and in return operator feels the natural sensation as ifhe was in the remote environment. There has been number ofworks related to telexistence in real environments including
TELESAR ‎[1] involved in creating an exoskeleton typemaster-slave Telexistence cockpit, a 6 DOF telexistence headsystem TELESAR III ‎[3] and an avatar system TELESAR
V ‎[4]. Also telexistence in combination of real and virtualenvironment exists including a master-slave system called
TELESAR II ‎[5] which enables a remote participant to not
only interact with the robot, but also see the operator througha projected image based on retro reflective projectiontechnology ‎[6]. Also ability to view the operators projected
image in a multi angle view was introduced in TELESAR
IV ‎[7].
Most of the humanoid robots including telexistence slave
robots had been tested and experienced only after physicallybuilding it. In case of unexpected user experience occurred, orM.Y. Saraiji, C.L.Fernando, M. Furukawa,K. Minamizawa and S. Tachi
is with Graduate School of Media Design, Keio University, Japan. Address:
4-4-1, Hiyoshi, Kohoku-ku, Yokohama, JAPAN. 223-0061; Telephone:+81-45-564-2449; email: {yamen , charith, m.furukawa, kouta,tachi}@tachilab.org

978-1-4673-1497-8/12/$31.00 ©2012 IEEE595

a new modification was required, the researcher or theengineer will have to wait until the physical fabrication isdone. Although some of the previous robots had its own CG
simulator environments, they lack the physical properties like
editing and tuning features and the immersive experience ofthe robot as if it exists.
Other‎robots‎like‎Rollin‟‎Justin,‎a‎humanoid‎robot‎‎[8] had
its simulation environment which is used for prototyping andtesting. The robot, however, aims for performing automatedcomplex tasks without providing user feedback orexperiencing the robot.
So rather than rebuilding or modifying the real robot whenrequired to test new features, its more efficient to use low
cost, faster methods which can mimic the real process. Suchtools were commonly known as simulators, but the graphics
simulation alone is not enough to reproduce realistic andimmersive user experience because it lacks of feedback
systems including first person vision, auditory and haptic
sensation when touching remote objects within the simulatorenvironment.
Thus we propose “Virtual‎Telesar”‎a virtual environment
for building, testing and fine-tuning telexistence robot
systems in a virtual environment and to experience theimmersive experience as if the physical robot exists. In thispaper we describe about the modular based design,
customizability, how to fine-tune the components and anexample‎ implementation‎ of‎ “TELESAR‎ V”‎ telexistence‎system using the proposed method (Fig.1). Our proposed
method focuses on the following points:
1.

User can define a robot entity using basic building
blocks including, robot model, input sensors, output
actuators,‎etc…

2.

User can tune component specific parameters

3.

The method can integrate with physical sensory inputand motor output represented as a set of components
with virtual environment.
4.

manipulating‎ it‟s parameters (Field of view, Frame
Resolution, Stereoscopic distance between cameras,..). By
manipulating these parameters in real time will provide better
understanding of the desired robot.

The method enables the user to provide immersive
experience similar to physical robots, via stereovision,audio and haptic feedback.

Audio - by attaching stereo audio listener on the robot, andspecifying related parameters like its direction, audio gain...

II. PREVIOUS WORKS

Haptics – specifying the shape of the sensor, its placement,
and sensor transfer function. By testing it and experiencing it,then re-tuning and engineering it to get the requiredperformance.

There has been many related works on modular based
design related to robotics. Farritor and Dubowsky ‎[2] had
proposed a modular design approach for field robots systems
using prefabricated modules to give low-cost system forspecific tasks. This approach would help in simplifying
building process into discrete set of options.
Related work regarding component based robot design isWebots ‎[10], a robot simulation environment that uses
components to describe the model and the modules.
Additionally, Microsoft Robotics Studio focuses providing
mobile and automated robots ‎[11]. Gazebo ‎[12] is provided as
open source. ABB Robot studio [13] specified for industrial
purpose also allows us to tune performance and confirm theresult in real-time. These examples support for modular
design, performance tuning in real-time.
However, it is difficult to use these tools for telexistence
applications because it lacks the immersive experience.
Furthermore the representation of most of these parametertuning is dedicated only for the simulator parameters. Thesetuning might be useful for CG experts, but not for telexistence
researches as they are more focusing on the robot dynamics
and overall immersive experience.

When defining new component, the designer (for example
the component manufacturer) takes into consideration
specifying component‟s attributes and parameters which aregoing to be exposed in the simulation and design environment.
Also other considerations like component communicationinterface should take place within the design step. Suchconsiderations are essential because it will define thebehavior of the component when communicating in thechannel. For example, sensors like tactile sensors take into
consideration communication frequency which will affect thebandwidth of the channel, and thus it will affect the size ofdata communication.
The‎ ability‎ to‎ modify‎ components‟‎ parameters‎ in‎ the‎simulation environment will give the user the ability toexperience alternative configurations of it quickly and zero
cost in terms of time and money. Such configurations can bebody length or size, or his vision like his field of view, thus hecan experience how such configurations will be reflected on
the operator.

III. IMPLEMENTATION OF VIRTUAL TELESAR
A. Design Concept
In our proposed environment, the user should be able todefine his robot, scene and environment in any CAD
software. Then create basic primitives or import existing
meshes for the intended robot and provide textures for thosedefined meshes. Then physical parameters like weight,
collision model mesh, friction and elasticity parameters,
adding joints can be done inside the CAD software using acustom build component toolbox for the Virtual Telesar.
Component toolbox should include the following:
Physical – defining its physical parameters consists ofspecifying the primitive or a custom mesh model (collisionlayer) and physical material parameters (specifying friction,elasticity) and other related parameters such as velocity
parameters,‎acceleration‎parameters‎etc….

Fig. 2 Communication between Telesar and operator

Kinematics – defining Degrees of freedom for the robot isdone via using joints or motors which links the segments of
the robot. When defining the joints, the designer will have
similar representation with the real robot definition, likedefining the rotation axis or the motor shaft axis, and defining
joints hardware limits, also the ability to define joints‟‎
stiffness and damping parameters (P gain, Dgain).
Vision – the ability to define visionary parameters isessential, for example adding a stereoscopic camera and596

Regarding the communication with simulator, thesimulator as listed previously plays the rule of the controlledavatar, so the communication with it should be abstract andisolated from the simulation environment by defining
data-providing layer. Fig.2 shows an overview of thecommunication between operator and remote avatar which
can be either real or virtual one.
B. Implementation
For developing the simulator we used a custom gameengine with Nvidia PhysX simulation. Above this engine we
implemented the required libraries that provide the requiredcomponents for the simulation. The simulator accepts an xml
file describing the scene and for each object in the scene, adescription components file is provided. Those files can beeither manually created or automatically generated from thedesign tool, which is in our case: Autodesk Maya. We
designed the required tools (Fig.3) in Maya which allows thedesigner to add, modify and export the components.

Fig. 5 Design flowchart

Fig. 3 Virtual Telesar Components Toolbox for Autodesk Maya

After modeling the physical representation and visualrepresentation, control units should be attached to the joints inorder to handle the input and feed it into the connectedjoint(s), if more than one joint where attached to the same
control unit, they would become coupled together, coupled
joints represents hardware coupling of set of rotational joints,
these joints had a coupling ratio, which means the same input
value will be weighted and fed into each coupled joint.
Even though the robot side does not contain sensing thedepth in a scene, visual enhancement, and to create better
visualization of the surrounding environment, the platform
automatically generates real-time ambient occlusion for thescene and High Dynamic Range rendering. Also cameras
support Depth of Field.
Fig.4 shows an example of the panels (Constraint –or themotor-, Stereo Camera, Physical Node) and the generated
xml file. The general design method can be represented in theFig.5. It shows how the simulator accepts the generated
model, and the ability to tune it based on the model.
C. Virtual Telesar implementation for TELESAR V
We have modeled a virtual representation of TELESARV,
which allows a person to experience telexistence in a virtualenvironment. Fig.6 shows Master/Slave operation, and a
description of TELESAR V system. The physical robot
supports stereo cameras, and force, temperature and tactile
sensors on 3 fingers per hand.
A real-time simulation has been carried out with theexisting TELESAR V system. The results was observed ageneral simulator tool in 3rd person view as well as animmersive 1st person experience wearing the original HMD
used for TELESAR V system.
In this experiment a two WXGA (1280x720) stereoscopic
screens were rendered in real-time providing the user to see
the virtual environment in first person 3D view. We
experience a real time video and clear vision also was able torender at 30 FPS.

Fig. 4 Components' panels (top)the generated simulator's xml file (bottom)

Designing process starts by using definition components,
these components will serve for local configurations of themodel. Defining the model begin by the visual representation,
which includes object mesh and visual material. For thephysical‎model,‎the‎designer‎can‎specify‎objects‟‎properties,‎and an estimated representation for the underlying collisionmodels.

597

ACKNOWLEDGEMENT
This work was supported by JSPS Grant-in-Aid for
Scientific Research A (23240017).
REFERENCES[1]S. Tachi, Telexistence, World Scientific Publishing Company, ISBN
978-981-283-633-5, (2009).
[2] S.Tachi, H. Arai, and T. Maeda, Tele-existence master-slave system forremote manipulation, in Proceedings. IROS ’90. IEEE International
Workshop on, jul 1990, pp. 343-348 vol.1’.
[3] S. Tachi, K. Watanabe, I. Kawabuchi, N. Kawakami & T. Maeda,
TORSO: Development of a Telexistence Visual System Using a 6-DOF
Robot Head, Advanced Robotics 22 (2008), pp. 1053 – 1073.
[4] C.L. Fernando, M. Furukawa, T. Kurogi, S. Kamuro, K. Sato, K.
Minamizawa,‎ S.‎ Tachi.‎ ,‎ “Design‎ of‎ TELESAR‎ V‎ for Transferring
Bodily Consciousness‎ in‎ Telexistence” IROS 2012, IEEE/RSJ
International Conference on Intelligent Robots and Systems.
[5] S. Tachi, T. Riichiro, K. Naoka, K. Hiroyuki, S. Dairoku, Developmentof Master-Slave‎ Arm‎ for‎ „Telesar‎ 2‟,‎ in International Symposium inRobotics, 2005, pp. 1-4.
[6] S.Tachi, Augmented Telexistence, Mixed Reality, pp.251-260,
Published by Springer-Verlag, 1999.
[7] S. Tachi, K. Watanabe, K. Takeshita, K. Minamizawa, T. Yoshida, andK.‎Sato,‎“Mutual‎telexistence‎surrogate‎system:‎Telesar4‎– telexistencein real environments using autostereosopic immersive display”,‎ inIntelligent Robots and Systems (IROS), 2011 IEEE/RSJ International
Conference on, sept. 2011, pp. 157-162.
[8] C. Borst, T. Wimbock, F. Schmidt, M. Fuchs, B. Brunner, F. Zacharias,¨P. R. Giordano, R. Konietschke, W. Sepp, S. Fuchs, C. Rink, A.
AlbuSchaffer,‎and‎G.‎Hirzinger,‎“Rollin‟‎Justin‎- Mobile Platform with
Variable‎Base,”‎in Proc. of the 2009 IEEE International Conference on
Robotics and Automation, May 2009, pp. 1597–1598.
[9] S. Farritor , S.Dubowsky., On Modular Design of Field RoboticSystems, (2001) Autonomous Robots, vol.10, pp. 57-65.
[10] Webots, http://www.cyberbotics.com. Commercial Mobile Robot
Simulation Software.
[11] Microsoft Robotics Studio, http://www.microsoft.com/robotics/,
Commercial Robot Simulation Software.
[12] Gazebo Robot Simulator, http://gazebosim.org/, Open Source
Simulation Software.
[13] ABB Robot Studio, http://www.abb.com/.

Fig. 6 Features of TELESAR V Master-Slave System

Furthermore the collision model stability was sufficient tohave a very smooth movement of the virtual Telesar. Also we
conducted manipulation tasks in CG environments. Fig.7shows both: third view camera for the robot while it washolding two cups and the representative physical environment
for the robot (physical visualization has been done using
PhysX Remote Debugger).

Fig. 7 Telesar in action, physical simulation for cup holding

IV. CONCLUSION
We proposed a customizable modular based virtual
platform for modeling, simulating and testing telexistence
applications, where the physical parameters of vision,
auditory and haptic are preserved in virtual environment. The
effectiveness of the system is tested by implementing a virtual
telexistence platform based on existing TELESAR V system.
Virtual Telesar platform will help telexistence engineers tomodel a prototype system before manufacturing it and theability to experience how the final model will look-like,
perform‎manipulations‎etc…‎in‎real‎world‎without‎building.
In the future, we aim to provide more support for developers
and researches by giving the features of generating manifest
data or blueprints and send to fabrication houses.

598