11F-04
This article is a technical report without peer review, and its polished and/or extended version may be published elsewhere

第 21 回日本バーチャルリアリティ学会大会論文集（2016 年 9 月）

Study on Telexistence LXXXV
Layered Presence: Expanding Visual Presence using
Simultaneously Operated Telexistence Avatars
1)

1)

MHD Yamen SARAIJI ，Charith Lasantha FERNANDO ，Kouta MINAMIZAWA

1)，Susumu TACHI 2)1) 慶應義塾大学大学院メディアデザイン研究科（〒223-8521 神奈川県横浜市港北区日吉 4-1-1, yamen@kmd.keio.ac.jp)
2) 東京大学 高齢社会総合研究機構 （〒113-8656 東京都文京区本郷 7-3-1）
Abstract: This paper contains an overview for a novel technique to expand visual perception in Telexistencesystems from one location to multiple locations. Layered Perception (LP) uses eye gaze perceptual awarenessblending to combine visual feedback from several Telexistence robots into a single location, and each robot
represents a layer of presence. By estimating the visual saliency of these layers and combining them into asingle visual space based on user's eye gaze motion, we can expand user's visual awareness to these multiple
locations simultaneously. Here we describe the design of the proposed system as well as several applications
using it.
Keywords: Teleimmersion, Telexistence, Augmented Reality.

1. Introduction
Several off-the-shelf services provide tele-conferencing support
for multiple users (such as Skype, Google Hangouts, etc…)
simultaneously through Picture-in-picture (PiP) grid of all users.
However, in these services the user is always engaged with one
participant during the session. Also, to switch between locations,the user either has to manually select which person to talk to, or
the system automatically frames the active participant using
Voice Activity Detection (VAD) engine. This type of manual or
automatic switching reduces the engagement of the user due tothe non-intuitive mechanism of switching between the remotelocations.
Telexistence base systems [1] provide the user full or partial
representation of his body while maintaining the visual andauditory mapping with user’s body, achieving an intuitive
Figure 1 Layered Presence system overview.

interaction in the remote site. However, the user body is
restricted to a single location at a single time.

The proposed LP system addresses the previous points by using

In order to achieve the experience of being presented at
multiple locations simultaneously and to be capable to have fullawareness of these locations, the following requirements should
be addressed:•Robots are represented as layers of awareness, and these layers
gaze is tracked and used to identify the target layer to behighlighted among the other layers, in which the layer use is

Natural mechanism of blending the visuals from themultiple sources.

•and provides real-time visual and auditory feedback to the user.
are blended and presented to the user’s displays. User’s eye

Real-time simultaneous representation of body, visuals,and auditory feedback in multiple locations.

•Telexistence robots that are synchronized with user’s motion,

looking at becomes focused while the other layers aredefocused using an artificial depth of field effect. Figure 1

Intuitive mechanism for switching the visual perception
between the locations, while maintaining the awareness ofshows the proposed system in action in which the user
perceives two different locations simultaneously.

the other locations.

© 2016 日本バーチャルリアリティ学会- 11F-04 -11F-04

2. Related Work
Our presented system draws from the following area ofresearch: Vision Augmentation, Eye Gaze Applications andMulti-Space Image Fusing.
2.1 Vision Augmentation
Several approaches were proposed to address blending two
different locations simultaneously at one physical or virtual
location. Lindlbauer et al. [2] have proposed to use a physicalsee-through LCD display to mix the environment behind thescreen and the contents of the screen achieving seamless
blending between both contents. Fan et al. [3] proposed video
based image blending using see-through HMD that provides theuser the awareness of both locations behind and front of him.
Both approaches focus on expanding visual perception locally.
2.2 Eye Gaze Applications
In this work, eye gaze is an important input to assist the system
to bring to focus the layer of interest. Eye gaze applications had
been an interesting area of research in various fields[4][5].
Previous researches showed the effectiveness of using the eyegaze input for selection applications compared with pointing

Figure 2 LP system flow and interaction between user
and remote telexistence robots.

input using mouse [6]. The proposed system adopts eye gaze
input modality to achieve natural and selective navigation
within the different remote locations.

(HerkuleX DRS-0201/DRS-0101). The joints are driven by

2.3 Multi-Space Image Fusing
In physical environments, when multiple objects arranged at
different depth distance from the perspective point, a wellknown phenomenon occurs in our visual perception: Depth offield, or image blurriness for objects out of focal plane.
Previous works used this phenomenon to visualize data [7], andwere also used in a multi-user applications [8] to selectively
focus on the person of interest . In the proposed method, we use
artificial depth of field to combine the layers of presence, inwhich the layers focused will be brought to the foreground andthe user can perceive it clearly, and layers residing in user’s
peripheral vision are blurred out according to their priority
according to user’s gaze.

micro controller (Arduino ProMicro) that is connected overserial port to an embedded PC (Intel NUC D54250WYK1). For
the cameras used, a low latency capture cameras were selectedfor this design (See3CAM CU130) that outputs sufficient
frame-rate for the used head mounted display (format YUYV
640x480@60 FPS), and equipped with a 90° wide field of viewlens. The image stream captured by the cameras from eachrobot is compressed using H264 format (using library
GStreamer v1.6.1) with bit rate 3000 bps, and streamed to theuser side over UDP connection. The measured end-to-end
latency of the image stream using 802.11 wireless LAN isaround 100-130ms for dual stream image streams. This low
latency performance is required in order to reduce VR sickness
for the users while using HMD.

3. System DesignThe developed system is divided into a Master-Slaves
Telexistence systems. The master side is the operating side
where the user is located, and it contains a set of tracking toolsthat are used to capture user's head movement and eye gaze.
Robots located at the remote sides are the same design and areconnected with the user over LAN network. Figure 2 shows anoverview of the system.

3.2 User Side
User side (or master side) operates the remote robots motion
using head rotation. Two types of setup were used for the user:•Exocentric viewpoint type, in which an external display isused to project the layers.

•Egocentric viewpoint type that uses HMD to immerse theuser with the layers.

For exocentric type that uses an external monitor to fuse the3.1 Robot SideIn this system, a custom three degrees of freedom Telexistence
robot head was designed. HD stereo cameras and binauralmicrophones are used to enable stereo visual and binaural
auditory communication to the user from robot side. In this© 2016 日本バーチャルリアリティ学会

design, we used for the pan, tilt, and roll joints servo modellayers, we used eye gaze axis (X and Y) to drive two angles ofthe robot (Pan and Tilt respectively). And for Egocentric mode,
we used a commercial HMD (Oculus DK2) that embeds
gyroscope sensor and provides three axis angles. For eye gaze
tracking, we used an off-the-shelf eye gaze sensor (Tobii eyex)- 11F-04 -11F-04

that provides X-Y eye gaze coordinates in the screen space. Eyegaze is used as an input to LP system to determine which spot
the user is looking at, and based on this input, the system
controls the focus of the layers to the corresponding layer user
is looking at. To determine the candidate layer that should be infocus, visual saliency maps were generated for each layer.
User side software was developed under Unity3D environment,

Figure 3 Saliency maps generation.

a custom video and audio streaming plugin was developed thatports image and audio data to Unity. The plugin uses GStreamer
library to handle media streaming and decoding. Most of image
processing parts were handled using EmguCV library(OpenCV .NET wrapper library) under Unity3D.
The system tested on desktop PC setup with the following
specifications: processing unity is Intel Core i7@3.40GHz,
memory 16 GB, and graphics processing unit model NVidia
GeForce GTX980. All the reported results regarding theperformance, frame-rate, and latency were done using theprevious setup. In this setup, the system runs at an average of
60FPS regardless of the number of layers used due to themulti-threaded design of the system.

temporal process is applied to the calculated maps, a window of
500ms of previously calculated frames is used to calculate theweighted sum of the final saliency map. Using this procedure,
the saliency maps maintained higher consistency in bothtracking and representation of participant’s area in video frames.
Figure 3 (B) shows the final saliency map for a single layer.
3.4 Layer Fusing
Fusing the layers, or mixing them, is considered the final stepof this method to deliver the layers to the user view area. One ofthe main considerations in LP is the user should maintain clear
visuals and auditory feedback from the location he is engaged
at, while being aware of the other locations simultaneously.

3.3 Saliency Map GenerationSaliency maps in this method are responsible to represent thepresence of remote participants as a weight map generated foreach captured frame while taking to consideration the temporalfactor of the frames. The process of generating the saliency
maps is done by a combination of two image analysis andfeatures extraction methods. First the layers are processed forhuman presence, the procedure is done by applying Haar
cascades classifier on each captured frame and the results arerectangles set representing the detected faces regions. These
rectangles are expanded proportionally to their size to coveruser body size (manually tuned, Width factor: 200%, Heightfactor: 400%). In practice however, using facial detection only
fails to provide continuous tracking of presented people forseveral reasons such as partial occlusion of the face, lighting
conditions, and resolution of the captured images. Also relying
on facial detection only limits the visual saliency maps tocapture the information of moving objects in scene. We
addressed this limitation by adding a second layer of tracking
using optical flow detection in the layers. Lucas-Kanademethod was used to track scene features points for changing,Firstly, the layers are fused together based on the weight of eachlayer which is assigned based on user's eye gaze. Each layer's
weight is calculated by sampling saliency map corresponding toeach layer using eye gaze coordinates.
In preliminary experiments of layer fusing, we used a basicalpha blending to all layers based on the calculated weight ofeach layer, however we found that its difficult for the users toclearly distinguish the visuals of the layers due to visualoverlapping between all locations. To address this issue, weconsidered using a similar phenomenon seen in image reflection
over window glass. Basically this phenomenon of reflection andtransparency of window glass allows to see two different
locations simultaneously as well as the ability to focus atdifferent depths that would result blurriness of objects in thebackground of both locations (depth of field). Using thisphenomenon, a pseudo model was defined that defines a focal
value for each layer, that is basically driven from the calculated
layer weight, is used to control the amount of shallowness oflayers out of focus. Figure 4 shows the final results of fusing
two layers using the proposed method.

when local changes occur in the layers, motion vectors arerecorded for later registration in the corresponding saliency mapof the layer. Figure 3 (A) shows the detected motion vectors ofa remote person.
Next, the saliency maps are filled with weight of 1 for the pixels
corresponding to the registered feature points and facial regions
for each layer. To avoid the presence of hard edges along thedetected regions, a Gaussian blur filter is applied to the saliency
maps. To assist feature tracking consistency over time, aFigure 4 Two different locations fused together
with an artificial depth of field .

© 2016 日本バーチャルリアリティ学会- 11F-04 -11F-04

Figure 5 Saliency maps generation.

4. Applications

applications. By using the concept of layering, its possible

This method can be expanded to more than just Telexistence

to achieve an intuitive and seamless sense of presence inrelated applications. By using the concept of layered perception,

multiple physical or virtual locations simultaneously.

its possible to generalize the layers to be media or eveninteractive applications, and apply the same procedure in
combining them into a single space. Also, we define two modesof viewpoint (Egocentric & Exocentric), in which are based on
user's point of view and how the user perceives the remote

Acknowledgement
This research is supported by the JST-ACCEL Embodied
Media Project.

environment or the layers.
4.1

References

Egocentric Applications

The user in this type of application is immersed with the task heis doing, for example when the user is being engaged in avirtual reality game while wearing HMD. By using the concept[1] Tachi, Susumu. "Telexistence." Virtual Realities. Springer
International Publishing, 2015. 229-259.

[2] Lindlbauer, David, et al. "A collaborative see-throughdisplayof LP, the game it self is also considered as a layer that can befused with other layers with eye gaze enabled. By using thismethod, the user is capable to be simultaneously engaged withprivacy." ACM

field of view for augmented awareness." Proceedings ofthe 5th Augmented Human International Conference.

Figure 5 (A) shows an example of LP with virtual game. Also
local location as a layer, and thus it can be seamlessly fused

on-demand[3] Fan, Kevin, et al. "SpiderVision: extending the human

the activity he is doing as well as with remote discussion.
by using a video see-through HMD, its possible to consider thesupporting

SIGGRAPH 2014 Emerging Technologies. ACM, 2014.

ACM, 2014.

[4] Hutchinson, Thomas E., et al. "Human-computerinteraction using eye-gaze input." IEEE Transactions onwith a remote location as shown in Figure 5 (B).

systems, man, and cybernetics 19.6 (1989): 1527-1534.

4.2 Exocentric Applications
In this category of applications, the user perceives the layers[5] Morimoto, Carlos H., and Marcio RM Mimica. "Eye gaze
trackingfrom an external display with an external eye tracker mounted
into the display. The contents of the display (application, visualmedia, etc.) can thus be considered as a layer of presence, while
remote robots are blended with contents of the display. Figure 5locations.

for
VisioninteractiveandImage

Understanding 98.1 (2005): 4-24.

[6] Sibert, Linda E., and Robert JK Jacob. "Evaluation of eyegaze interaction."Proceedings of the SIGCHI conference(C) shows an example of interacting with a software while
being engaged in a simultaneous meeting in two different

techniquesapplications." Computeron Human Factors in Computing Systems. ACM, 2000.

[7] Kosara, Robert. Semantic Depth of Field-Using Blur for
Focus+ Context Visualization. Diss. Kosara, 2004.

[8] Yao, Lining, et al. "FocalSpace: multimodal activity

5. Conclusion
This paper presented Layered Presence, a novel system for
operatingandblending

locations

simultaneously

multiple

Telexistence

usinggazeeye

based

perceptualawareness. The proposed system treats each robot as a layerof presence which can be blended together with otherperceptual layers based on user’s eye gaze motion. This
paper lists two distinguished types of applications using theproposed method: Egocentric and Exocentric types of© 2016 日本バーチャルリアリティ学会- 11F-04 -tracking, synthetic blur and adaptive presentation forvideo conferencing." Proceedings of the 1st symposiumon Spatial user interaction. ACM, 2013.