1G2-05
This article is a technical report without peer review, and its polished and/or extended version may be published elsewhere.

第 22 回日本バーチャ ルリ ア リ ティ 学会大会論文集（ 2017 年 9 月）

Study on Telexistence LXXXVIII
Foveated Streaming: Optimizing video streaming for Telexistence systems
using eye-gaze based foveation”
MHD Yamen Saraiji1) ， Kouta Minamizawa1) ， Susumu Tachi2)
1) 慶應義塾大学大学院メ ディ ア デザイ ン 研究科(〒 223-8521 神奈川県横浜市港北区日吉 4-1-1, {yamen,kouta}@kmd.keio.ac.jp)
2) 東京大学 高齢社会総合研究機構 (〒 113-8656 東京都文京区本郷 7-3-1, tachi@tachilab.org)
Abstract:

Current Telexistence systems use full resolution image stream that is sent over thenetwork to the user side. However, due to performance and network limitations, the resolution issignificantly limited and thus can not provide sufficient details for the human eyes. This paper
addresses the topic of increasing spatial visual acuity in telexistence related applications by taking
the advantage of human eyes retina properties. Instead of streaming full resolution images over thenetwork, hierarchical, multi-resolution image regions are sent to the user based on his eyes gazeposition. Using this method, it’s possible to increase the spatial visual information perceived by
the user, while significantly reducing network bandwidth and increasing processing performance ofthe sender/receiver sides. Here we present the overall system including the challenges in maintaining per-frame eye gaze synchronization over the network, and quantitative and qualitative studies
verifying the effectiveness of foveated streaming are described.
Keywords：

1.

Telexistenc, Network Transmission, Perception

Introduction
Recent advances in virtual reality display systems be-gan to offer end users immersive and high fidelity experiences in a rather accelerating manner. Current Head
Mounted Displays (HMD) such as Oculus, Vive, andHigh Resolution

Samsung GearVR offer wide field of view (FoV) displays

Medium Resolution(100◦ ∼ 120◦ ) with a pixel density of 100 ∼ 175 per deLow Resolution

gree at refresh rate ranging from 60 ∼ 90 frame per second (FPS). In future, expected HMDs are to offer muchhigher resolution and refresh rate than now in order toenhance the visual experience and immersion towards human eye. However, such visual experiences would require
high processing bandwidth to deliver such information
stream to the HMD, and performance would decrease exponentially based on the target pixel density and linearlybased on the target frame rate.
On one hand, virtual reality applications that run locally would only require a system capable to operate
at such high bandwidth, which is mostly addressed byhardware manufacturers. On the other, Telexistence andTelepresence systems would suffer more at addressing
this problem due to network bandwidth limitations. Even
with the current standard compression codecs such as
MJPEG, H264/MPEG-4, and VP8 will have limited bi-© 2017 日本バーチャルリアリティ学会

Figure.1: Foveated Streaming overview: multiresolution regions based on eye gaze position.
trate to accommodate for network streaming. Also, in
this type of systems, encoding/decoding are required before streaming and after receiving the image stream, which
performance is also dependent on the pixel resolution of
the images. Extending the problem for omnidirectional
type cameras which sends the entire 360◦ view, the resultant pixel resolution of the equirectangular image (thestitched omnidirectional 2D image) would increase 5 to
8 times more than the limited field of view pinhole cameras image type (at 90 − 110◦ ). As a consequence, theperformance of encoding and streaming would increase
the latency and decrease the overall experience for Teleoperation.

- 1G2-05 -1G2-05

Robot Sideand Telepresence systems, a perceptual driven approach

Camera Image(N)Eyegaze(w’ x h’)Image Combiner

H264 Encoder(N x w’ x h’)(RTP Stream)...

Region N(X,Y)

User Side

Region 2...

at the fovea in the retina, which is only about 2 of the(w’ x h’)

Foveated Images
Generatorwas used. Human eyes have the highest visual acuity◦Region 1(W x H)(w’ x h’)Fovea Size

User Side

Embed toRTP StreamTo address the previous issues related to Telexistence(Degrees)

visual field, and this visual acuity decreases the further
it is from the center of the fovea towards the periphFigure.2: Streaming pipeline.

eral vision. By taking the advantages of this property, amulti-resolution image stream is generated from the original high-resolution images of the Teleoperated system.

received from the user side. The pipeline generates aThis multi-resolution stream maintains the high visual

number of regions (N) which can be set in using system

quality at the center of the eye gaze, while lower spa-parameters. Region size in degrees is also determinedtial resolution images are used for the peripheral area asby the Fovea Size parameter, in the conducted experi-

shown in Figure 1. Using this method, the image stream

ments, a value of 15 deg produced a good balance be-would result in a higher compression ratio based on thetween streaming compression and perceptual awareness.

selected fovea size, and the encoding/decoding perfor-

The generated regions are combined into a single image

mance would increase according to that too. This paper

frame as in Figure 3 (Streamed Regions). Then the im-is based on the previous research area of foveated render-ages are compressed using the H264 encoder and con-ing, and it contributes as follows:

verted to RTP stream to be sent to the user side. One• Providing real-time network streaming synchronization for eye-gaze using RTP stream.

important thing is to synchronize the eye gaze information in order to reproduce the same arrangements whencombining the images on the user side. To do that, eye• Quantitative and qualitative studies showing thegaze information used for foveation is embedded into theeffect of foveated streaming for performance andfirst RTP packet of the image frame before being sent tohuman perception.

the user. The main advantage of embedding this information into the RTP stream is to ensure exact synchroniza-2.

Related Worktion of the data and the image stream when rebuilding

The idea of using eye fovea as a driver to optimize
the image spatial resolution has developed an importantthe original image at the user side.
3.2

Unpacking & Presentationbody of research in the area of multimedia and computer

User side receives the packed regions as an image stream,graphics optimization. In image compression, Wang et al.

each frame contains foveation levels as was described in[1] proposed image coding system by taking into consid-

the previous section. When decoding the stream, first theeration the nonlinear decrease of spatial resolution in theeye gaze position is extracted from the first RTP packet

human eye, and removing the high-frequency informa-of the packed frame and is used to arrange the regions

tion from the peripheral area, thus improving the overall

and positioning them exactly as they were sent. Aftercompression performance. Itti [2] used a different ap-unpacking the images, the regions are presented starting

proach to address image compression by using saliency

from the last region. Region number N is rendered into

information rather than just the eye gaze position. And

an offscreen image covering the entire field of view, next

Ryoo et al. [5] proposed a video streaming service based

Region number N-1 is rendered at the original scale, andon eye gaze foveation to enhance network bandwidth. In

so on until Region 1 is rendered on top of them as shown

computer graphics, Guenter et al. [3] used foveated ren-in Figure 3 (Foveated Image). To mitigate the effect ofdering in order to enhance rendering performance by 5-change in resolution between two consecutive regions, a

6 times than direct rendering. Patney et al. [4] usedmask is applied to each region which helps to fade the

foveated rendering in virtual reality applications to re-edges and to reproduce similar effect of eye retina de-duce the rendering cost, the work showed that foveatedcrease of visual acuity, while maintaining high resolution

rendering did not introduce a significant difference in vi-at the center of the region as shown in Figure 3 (Region

sual quality compared to direct rendering.

Masking).

3.

System Design & Implementation

4.

3.1

Encoding & Streaming

To evaluate foveated streaming method, two types ofFoveated streaming pipeline is shown in Figure 2. Thisevaluation were conducted:

pipeline runs on the robot side, and the eye-gaze data is© 2017 日本バーチャルリアリティ学会

Evaluation- 1G2-05 -1G2-05

Foveated Image

Region MaskingR1. 15◦
R2. 30◦R3. 45◦

Streamed Regions

R1. 15◦R2. 30◦R3. 45◦Background. 90◦

Figure.3: Foveation streaming decomposition (1)Foveated Image: final reproduced image at theuser side, (2) Region Masking: masks used for
combining the regions, and (3) Streamed Regions:streamed frame containing regions of foveation.
• Performance Evaluation: to evaluate the overall
performance of the system compared with full resolution image streaming, the bandwidth usage, andFigure.4:

CPU consumption are tested.

Performance evaluation of foveatedstreaming compared with full resolution stream• Qualitative Evaluation: a user study confirminging.

the effectiveness of foveated streaming in maintaining the perceptual consistency of visual informa-4.2tion.

Qualitative Evaluation

A user study was conducted to evaluate the visualSame system and hardware setup were used in both

perceptual effectiveness when using foveated streaming

evaluations. USB3.0 camera module (Model no. See3CAM

compared to full resolution streaming. The hypothesis

CU130 by e-consystems) were used to capture image stream,is when using foveated streaming, there is no significant

and connected to Intel NUC for video encoding and stream-difference in reading speed compared when using full res-ing over an ethernet LAN to the user side. For media en-olution streaming.

coding and streaming, GStreamer library was used. The

4.2.1

video stream is compressed using H.264 video codec pro-Study Setup

The study setup consisted of a three-axis telexistence

vided by the library.

robot head (providing pitch, yaw, roll motion) equipped

4.1

with 90◦ cameras for video streaming. The cameras werePerformance Evaluation

The goal of performance evaluation is to measure theset to capture 1920x1080 pixels at 30 FPS when stream-

effectiveness of using foveated streaming network band-ing.

width, and CPU performance at the encoder side com-

was 15◦ with 3 levels of foveation resulting a stream

pared with full resolution streaming. In this evaluation,of 1528x320 pixels per image. An A3 paper containing

three different streaming modes were used: 640x480 pix-

one page of text (font used bold Arial at a font size ofels at 60 FPS, 1280x720 pixels at 60 FPS, and 1920x1080

24points, and 1.5 line spacing) placed in front of the robot

pixels at 30 FPS. To maintain image quality among theat a distance of 30cm. Page was fixed that the first line

images in both cases, adaptive bitrate quantizer set atof text is placed at the eye level of the robot. The to-20% was used for the video codec. Foveated streaming

tal number of pages is 5, with an average of 225 words

parameters used were: 15◦ Fovea Size, and 3 levels ofper page. The text used was a short English essay (Thefoveation. Figure 4 shows the quantitative results thePlot by Luke Thompson). The user is connected to theproposed system. The results show the performance ef-

robot over the network and uses an HMD to control thefectiveness of foveated streaming compared to full res-head motion of the robot and to perceive the visual infor-

olution streaming for both the bandwidth (compression

mation from it. The HMD is equipped with an eye gazerate varies from 5 to 8 depending on the streaming res-tracker (Pupil-lab for Oculus DK2) to measure eye move-olution) and CPU performance (about half CPU usage

ment when foveated streaming mode is enabled. In thisfor foveated streaming).

setup, the user will be required to use both his head mo-© 2017 日本バーチャルリアリティ学会- 1G2-05 -For foveated streaming settings, fovea size used

1G2-05

tion and eye motion to read the text from the beginning!"#$%&'()*""$(+,-*#.%/%,&

to the end of the page along yaw and pitch axises.

<=>?@A?B2 ,AC?@DEFG

4.2.2

Procedure<HII24?J=IHAE=F!"+!"*,-./01,23-425/41

Prior to the study, participants are asked to fill a questionnaire containing basic information such as the age,eyes visual acuity for the left and right eye, and frequencyof using HMDs in general. The participants are only in-!")!"(!"'!"&!"%!"$!"#

formed of the task of reading and essay at one page a!
#$%

&'()

36478.836072 09:;-4

time and were asked to inform the experimenter once
they finished reading the active page. After wearing theHMD, the participants are asked to calibrate their eye

Figure.5: Subjective evaluation of reading speed

gaze using 9-points eye gaze calibration procedure, thenfor foveated streaming and full resolution stream-a waiting gray screen is shown before starting reading.

ing.

Once they inform their readiness to start reading, theuser gets connected to the robot, and they start reading

5.

Conclusion

the text from top to bottom, left to right. After readingThis paper shows a foveated streaming system for Telex-of the page is finished, the waiting screen is shown again

istence applications to reduce the network bandwidth andand the procedure continues for the next page until theprocessing requirements for encoding and decoding video

5 pages are over. The streaming mode for the pages

images. Performance evaluation shows a significant de-is switched between full streaming/foveated streamingcrease in bandwidth size, 5-8 times less than full resolu-

when changing the pages, and the first page’s mode istion streaming, and twice the performance increase. A

randomly selected between both modes. Reading timeuser study showed no significant difference when using

is measured per page and stored along with the stream-

foveated streaming compared to full resolution stream-ing mode for performance evaluation. After the study ising.

over, qualitative questions are asked about the reading
experience and whether anything has been noticed while
reading and switching the pages.
4.2.3

Acknowledgment
This work is supported by JST ACCEL Embodied Media
Project Grant Number JPMJAC1404.

Results and Discussion

References

In this evaluation, 7 participants joined the study (6males and 1 female with an average age of 24 ± 3) from

[1] Wang, Zhou, and Alan C. Bovik. ”Embedded
foveation image coding.” IEEE Transactions on im-different ethnicities. Overall feedback did not include
any significant difference in the reading experience of thepages. Two participants reported seeing blurry wordsage processing 10, no. 10 (2001): 1397-1410.
[2] Itti, Laurent. ”Automatic foveation for video compression using a neurobiological model of visual at-around the edges of gaze point (participant 1 and 4),
this is due to the eye gaze tracker which could have been
moved during the experiments (HMD was moved fromthe calibration position), the performance results show

tention.” IEEE Transactions on Image Processing 13,no. 10 (2004): 1304-1318.
[3] Guenter, Brian, Mark Finch, Steven Drucker, DesneyTan, and John Snyder. ”Foveated 3D graphics.” ACM

the impact of the calibration mismatch to their reading
speed in foveated streaming. For performance results ofthe study, Figure 5 shows the average time per word forTransactions on Graphics (TOG) 31, no. 6 (2012):
164.
[4] Patney, Anjul, Joohwan Kim, Marco Salvi, Antonboth full resolution and foveated streaming per partic-

Kaplanyan, Chris Wyman, Nir Benty, Aaron Lefohn,ipant. Overall reading speed can of both modes showsand David Luebke. ”Perceptually-based foveated vir-no significant difference between both modes. Reading
speed varies per page, and overall the speed slightly in-tual reality.” In ACM SIGGRAPH 2016 Emerging

creases when proceeding in the study, this can be due toTechnologies, p. 17. ACM, 2016.
[5] Ryoo, Jihoon, Kiwon Yun, Dimitris Samaras, Samirthe adaptation time of reading over a Telexistence robot.

R. Das, and Gregory Zelinsky. ”Design and evalua-From these results, it can be shown that foveated stream-tion of a foveated video streaming service for com-ing provided similar results to the full streaming undermodity client devices.” In Proceedings of the 7th In-

the condition of consistent eye gaze tracking.

ternational Conference on Multimedia Systems, p. 6.
ACM, 2016.

© 2017 日本バーチャルリアリティ学会- 1G2-05 -