ReactSpace: Spatial-Aware User Interactions
for Collocated Social Live Streaming Experiences
Bektur Ryskeldiev, Michael Cohen

Jens Herder

Yoichi Ochiai

Spatial Media Group
University of Aizu
Aizu-Wakamatsu, Fukushima 965-8580
Japan
Email: {d8171101, mcohen}@u-aizu.ac.jp

Hochschule Düsseldorf:
University of Applied Sciences
40476 Düsseldorf
Germany
Email: herder@hs-duesseldorf.de

Digital Nature Group
University of Tsukuba
Tsukuba, Ibaraki 305-0821
Japan
Email: wizard@slis.tsukuba.ac.jp

Abstract—Live streaming is becoming an increasingly popular
medium for social interaction on mobile devices. However, inmost cases live streaming experience is still confined to a singleviewpoint, monoscopic video stream, which, in active scenarios
such as outdoor or public event broadcasting, makes it hard oreven impossible for viewers to understand a remote situation.
This study investigates how live stream viewing can be improved
through introduction of user spatial data into video stream
composition and interaction, by developing a system that allows
watching multiple spatially oriented video streams in virtual
spaces simultaneously, and interaction with them through voice
and spatially distributed reaction buttons. According to results ofa preliminary study, users show a statistically significant interestin using spatial live stream viewing and interaction features.
Believing that this solution presents a novel approach to social
live video streaming, the rest of the article discusses the results
and possible extensions of developed system.

I. I NTRODUCTION
Live video streaming is becoming an increasingly popular
medium for social interactions online. Through such applications like Twitch, YouTube, Periscope, Snapchat, Instagram,and Facebook Live, users can quickly share their viewpointwith multiple remote users in real time. Such video streams
can cover a wide range of activities: from video games to live
coverage of different events by streamers located onsite. The
latter form of live streaming has become especially popular on
mobile devices due to their social availability and relative easeof use: most of the time mobile streaming applications do not
require prior setup or special equipment to start a streaming
session.
At the same time, however, the simplicity of mobile live
media sharing is also one of its major drawbacks. Firstly,
the applications listed above only support viewing of a one
livestream at a time, and for instance in case if users would
want to watch the same event from different viewpoints
simultaneously, they would have to manually close one video
stream and open another. Secondly, the applications do nottake full advantage of mobile device spatial data. For example,
such applications as Periscope and Facebook Live use mobile rotational tracking for navigation around spherical video
streams, but the rest of the interactions, such as text messages
and “Like” buttons, remain two-dimensional, without any
attachment to specific locations around the spherical scene.

Therefore, this study aims to answer the following question:“How can spatial data be introduced within the context ofmonoscopic live video streams, and how can it benefit theusers?”
To address this question, this study proposes a systemthat creates an interactive live streaming experience in mixed
reality (MR) for multiple users while exploiting mobile device
rotational sensing. Named “ReactSpace,” a live streaming
system uses a spherical image (captured by streaming user or
downloaded from elsewhere) to support a shared virtual spacefor multiple collocated live video streams, in which users canplace spatially-oriented reaction buttons.
The main scientific contributions of this work are:
• Proposition of a new taxonomy for classification of social
live streaming systems• Introduction of a novel video stream composition andinteraction method in mobile mixed reality that takes
advantage of user spatial data
• Investigation of effects of such approach on user engagement in social live streaming sessions, and comparison
with a similar commercially-available application
To test the validity of presented approach, a preliminary user
study was conducted, which shown a statistically significant
interest (p <0.05) among users in presented modes of interaction. The rest of this paper discusses similar state-of-the-art
solutions, an implementation, the user study and its results,
and possible future extensions of the developed system.
II. BACKGROUND
A. The classification of live streaming systems
Before discussing similar video streaming applications withspatialized MR features, it is necessary to establish taxonomy
upon which MR systems are classified. The term “mixed
reality” was discussed in a work by Milgram et al. [1] which
proposed a reality-virtuality (RV) continuum that placed all
mixed reality (including augmented reality and augmented
virtuality) systems in between. Augmented reality comprises
digital content overlaid on top of the real world (e.g., avideo of a real location with digital objects in it), whereas
augmented virtuality comprises real-world content placed oninformation available) on the right.
Extent of Presence Metaphor (EPM) – represents the levelof immersion. For example a simple monitor would be on
the left side of the spectrum (low-immersion), whereas
immersive VR systems would be on the right (capable ofsurrogate travel and telepresence).
• Resolution Fidelity (RF) – the resolution of the display
used by a mixed reality system, from low resolution on
the left side, to the highest possible on the right. As theinitial continuum was introduced over twenty years ago,
we have updated it to reflect changes in available display
technology (in bold font).
Based on this extended mixed reality system taxonomy,
this article proposes a similar approach to classification ofsocial media streaming systems (Fig. 2). Similarly to theextended mixed reality taxonomy, it builds on three continua:
Extent of Social Interaction Space (ESIS), Extent of User
Communication (EUC), and Reproduction Fidelity (RF).
1) Extent of Social Interaction Space (ESIS): The first
proposed continuum measures the extent of the social interaction space available to livestream viewers. For example,
such livestreaming applications as Periscope allow viewing
only one stream at a time and thus minimize the social
interaction space to only a single stream. Stream-aggregating
services like multitwitch [2] or kbmod [3] allow combining
multiple streams visually, but the interaction is still restricted
to each individual stream. Similarly, Google Hangouts allows
streaming live group chats to YouTube [4], combining multiple
streams into a single video stream, which allows interactingwith all video streams at the same time, but not with each ofthem independently. Finally, there is a new emerging approach
to live stream interaction presented in Rivulet by Hamilton etal. [5], which aggregates multiple streams and allows users
to interact both within a single stream and across multiple
streams simultaneously.
2) Extent of User Communication (EUC): The second continuum classifies different means of communication between
viewers and streamers. It is based on McLuhan’s “hot” and“cool” media concept [6] and was used for classification ofuser interactions in [5] and [7]. The hot and cool concept
proposes a spectrum where different media are classified basedon amount of mental workload. Cool media requires users toapply a considerable mental effort to understand it, unlike
hot media, which is more understandable and thus easily“digestible.” In McLuhan’s example, books would be classified
as cool media, while movies would be on the hot side of thespectrum.
Similarly, this concept is employed for categorization ofavailable modes of user interaction observed in similar studies:• Reaction buttons, which are also sometimes referred toas “hearts” or “like buttons,” are implemented in social
live streaming services, including Periscope, Instagram,and YouTube. Reaction buttons are one of the “coolest”
forms of media, because they do not provide a complete
representation of a person’s reaction. Although in mostservices such buttons are represented by 2D icons, in•Fig. 1: Milgram-Kishino RV continuum and extended taxonomy

Fig. 2: Proposed live streaming system taxonomy

top of virtual. Such distinction, however, did not provide themeans for qualitative comparison of mixed reality systems, as
it was impossible to say how an augmented reality application
might be better than augmented virtuality one, or vice versa.
Therefore, the RV continuum was further extended by theauthors into three spectra (Fig. 1):• Extent of World Knowledge (EWK) – represents how
much a system knows about the world around it, withworld unmodelled (no or little information available) on
the left side, and world completely modelled (all or mostthe recent project by Facebook [8] the reaction buttons
can be placed around virtual 3D space and thus could
theoretically provide more context for the participants.
• Text messages and annotations / Physical controls. Similarly to reaction buttons, almost all the services mentioned above provide support for text messaging. Some
researchers have even extended the messaging capabilities
to include specific commands. For example, Yonezawa
and Tokuda [9] integrated a set of commands that allows
users to change camera orientation and light conditionsin the live video stream, and Nassani et al. tested both
2D and 3D text annotations for social applications [10].
• Monophonic audio and monoscopic photo or video isperhaps the most common form of hot media streaming,and is available on all live streaming services.
• Spatial audio and visually spatialized, or spherical,
photographs and video is supported on services like
Periscope, YouTube, and Facebook Live in the form of
360° video streaming, providing a spherical overview
of a scene, and thus becoming a hotter form of media
in comparison with regular monoscopic video streaming.
Another example is LiveSphere by Nagai et al. [11] whichused a set of head-mounted cameras to present a “firstperson” spherical live video feed. Similarly, Kasahara etal. in JackIn [12] streamed from a single head-mounted
monoscopic camera, but through simultaneous localization and mapping (SLAM) they were able to create aconstantly updating spherical panorama from different
video frames and spatially arrange the live video stream
inside of it.
• Avatars is an emerging form of communication in live
streamed media. For instance, in Facebook Spaces [8]users are represented as virtual avatars that can navigate
around a scene and interact with everybody through othercooler forms of media. Most likely future forms of live
streamed media will also include different forms of haptic
interaction that would fall in this category.
3) Reproduction Fidelity: As in the extended mixed reality
taxonomy, this continuum describes the level of fidelity oflive streamed media. Similarly to the updated extended mixed
reality taxonomy, the scale reflects recent changes in live
streamed media displays. The continuum starts at standard
definition color video, which is supported by systems like
Periscope [13] or Rivulet [5], and moves towards highdefinition, stereoscopic (HMD-based), and spherical video.
Finally, this spectrum includes volumetric video, showcasedin Facebook Surround 360 [14] and by Zhou in Visibit [15].
B. Applying livestreaming system taxonomy
The relevant studies were arranged based on the developed
taxonomy (Figs. 3 and 4). It is notable that relatively few systems support multi-stream interactions and spatialized media
streaming. Therefore, based on such shortcomings, ReactSpace
was designed to support the most advanced approach to social
interaction between live media streams, as well as employ theFig. 3: Livestreaming systems arranged by RF and ESIS
continua. By “most social networks” the authors assume suchwebsites and applications as Facebook, Periscope, YouTube,
Instagram, and Snapchat

Fig. 4: Livestreaming systems arranged by EUC spectrum“hottest” forms of interaction: spatialized reaction buttons andvideo streaming.
III. I MPLEMENTATION
The application uses Unity game engine running on Android
and iOS devices, and achieves multistream interaction bycreating a single virtual space where all streamers and viewers
can communicate with each other. The space is represented bya photosphere (taken by one of the streamers or downloaded
from elsewhere) of a real space in which all streamers aresupposedly collocated (Fig. 5). The spherical background canbe uploaded by a streamer at any time throughout a session,and it provides a spatial background for both viewers andstreamers. In such case, in comparison with ReactSpace, live
media streaming applications such as Periscope can provide
a single spherical video stream, whereas ReactSpace provides
a static spherical background with multiple different media
streams embedded into it.
Spatialized live video streams are represented by video
billboards (rectangles that always face the viewer), spatially

TABLE I: ReactSpace user evaluation questionnaire
Q1
Q2
Q3
Q4

(a)

Q5-R
Q5-P
Q6-R
Q6-P

I enjoyed being able to view multiple streams
in a single virtual space
I was able to understand what was happening
in each stream
I enjoyed interacting with multiple live streams
at once
I felt like I was able to influence the live streams
using the voice feature
I felt like I was able to influence the live streams
using hearts in ReactSpaceI felt like I was able to influence the live streams
using hearts in Periscope
Using ReactSpace was fun
Using Periscope was fun(b)Fig. 7: User evaluation results (error bars represent 95%
confidence intervals)(c)Fig. 5: Example of a live session featuring two streamers
and a viewer: (a) one streamer’s perspective, (b) viewer’s
perspective, (c) scene overview

tracing in front of the camera (user’s current viewpoint in
virtual space), which converts touchscreen coordinates into 3D
coordinates within the virtual space.
The network communication between viewers and streamers is handled through the Web Real-Time Communication
protocol (WebRTC) [16], which is implemented through the
WebRTC Videochat [17] plugin for Unity. WebRTC plugin
handles the transfer of both media (audio and video) andtext (reaction button coordinates, rotational tracking) data (asdescribed by Fig. 6).
IV. USER EVALUATION
A. Experiment design

Fig. 6: ReactSpace network dataflow

arranged using rotational tracking data from the Google Cardboard SDK (which allows the application to run both inhandheld and HMD modes). Each billboard’s rotation aroundthe sphere shows the streamer’s current viewpoint and is
updated in real time.
The reaction button functionality is implemented by ray

To test the effect of spatialized interactions in social
livestreaming applications on user engagement, a subjective
user evaluation was conducted. In this case by user engagement the study considers the subjective opinion of users
regarding whether they could influence a remote situation
using interaction modalities in ReactSpace, and whether theyfelt if ReactSpace was more interesting and interactive than
Periscope.
Each trial consisted of two treatments. In one treatment
users were asked to watch a mobile spherical live video stream
located in the experiment location over Periscope. In the othertreatment the users were asked to watch multiple streams
from the same location using ReactSpace. In both cases
the participants were encouraged to use available means ofinteraction in each streaming service, including text messages

and heart buttons in Periscope and spatial reaction buttons andaudio in ReactSpace, and navigate around the scene using theirphone in both applications. The order of the treatments was
randomized before each session.
At the end of each trial, the users were asked to fillout a questionnaire (Table I) which subjectively compared
their experiences of using ReactSpace and Periscope. The
questionnaire was based on a study by Hamilton et al. [5],and included eight questions. In each questions participants
graded their answer based on Likert items (1 - Disagree ..
5 - Agree). After completing the questionnaire, participants
were also encouraged to provide any freeform comments andfeedback.
B. Analysis and results
The participants were recruited through social media networks, and comprise 9 men and 1 woman, aged between 20
and 35 years old, 10 participants total. Due to the limited
pool of participants, a sensitivity analysis was conducted, withsample size of 10, α= 0.05, and power = 0.8, showing minimal
effect size that this experiment could possibly detect dz = 0.9,
which according to [18] can be considered large.
Due to ordinal nature of collected data, the results wereanalyzed through one-sample (Q1-Q4) and paired (Q5-Q6)
Wilcoxon Signed Rank test. Statistically significant (p <0.05)
differences were noticed over an average score of 3 in Q1-Q4,
which indicates that the users might interested in spatial forms
interaction (both visual and aural) presented in the application,
but the questions that compare ReactSpace with Periscope(Q5-Q6) have not shown any improvement. Perhaps this result
could be explained by the fact that users were confused byinterface differences between Periscope application and ReactSpace. In the future studies, a more appropriate option could
be using an application based on the ReactSpace project, butwith functionality similar to Periscope (e.g., a non-spatializedversion of ReactSpace).
V. C ONCLUSION AND FUTURE WORK
ReactSpace represents an example of how spatialized interactions can be integrated into social livestreaming applications. Preliminary results indicate that users do seem to likespatialized interactions in media streaming, but a larger study
is necessary to confirm whether they would prefer it over aregular video streaming application.
Furthermore, aside from the positive feedback, the test participants requested such features as haptics (e.g., introducing avibration feedback when a reaction button is placed in virtual
space), automatic stream rearrangement in cases when multiple
streams coincide in one location within a spatial background,and support for spatialized audio (e.g., spatializing streamer
voices depending on their rotational orientation).
In the next iterations of ReactSpace system it would beinteresting to introduce support for photospherical video,
and, depending on the availability of more complex tracking
solutions, volumetric video streaming, which can possibly
make the livestream viewing experience more interactive incomparison with its currently implemented static photospherical background. We are also hoping to explore possible
applications of developed system for remote assistance andentertainment applications that use telepresence (such as [19],[20]) as the main form of interaction.
ACKNOWLEDGMENT
This work was supported by JST CREST Grant Number
JPMJCR1781, Japan.
R EFERENCES[1] P. Milgram, H. Takemura, A. Utsumi, and F. Kishino, “Augmented
reality: A class of displays on the reality-virtuality continuum,” inPhotonics for industrial applications.
Int. Society for Optics andPhotonics, 1995, pp. 282–292.
[2] MultiTwitch.tv, http://multitwitch.tv/, (Date last accessed 20-April2018).
[3] kbmod, https://kbmod.com/, (Date last accessed 20-April-2018).
[4] Google, “Hangouts on air with youtube live,” https://goo.gl/rBdC6x,(Date last accessed 20-April-2018).
[5] W. A. Hamilton, J. Tang, G. Venolia, K. Inkpen, J. Zillner, and D. Huang,“Rivulet: Exploring participation in live events through multi-stream
experiences,” in Proc. of the ACM Int. Conf. on Interactive Experiences
for TV and Online Video. ACM, 2016, pp. 31–42.
[6] M. McLuhan, Understanding media: The extensions of man. MIT press,
1994.
[7] W. A. Hamilton, O. Garretson, and A. Kerne, “Streaming on twitch:
Fostering participatory communities of play within live mixed media,”in Proc. of the 32Nd Annual ACM Conf. on Human Factors in
Computing Systems. New York: ACM, 2014, pp. 1315–1324. [Online].
Available: http://doi.acm.org/10.1145/2556288.2557048[8] Facebook, “Facebook spaces,” https://goo.gl/c8hNSb, (Date last accessed 20-April-2018).
[9] T. Yonezawa and H. Tokuda, Enhancing communication and dramatic
impact of online live performance with cooperative audience control,
2012, pp. 103–112.
[10] A. Nassani, H. Kim, G. Lee, M. Billinghurst, T. Langlotz, and R. W.
Lindeman, “Augmented reality annotation for social video sharing,” in
SIGGRAPH ASIA 2016 Mobile Graphics and Interactive Applications.
ACM, 2016, p. 9.
[11] S. Nagai, S. Kasahara, and J. Rekimoto, “Livesphere: Sharing thesurrounding visual environment for immersive experience in remotecollaboration,” in Proc. Ninth Int. Conf. on Tangible, Embedded, andEmbodied Interaction. ACM, 2015, pp. 113–116.
[12] S. Kasahara and J. Rekimoto, “Jackin: integrating first-person view with
out-of-body vision generation for human-human augmentation,” in Proc.
of the 5th Augmented Human Int. Conf. ACM, 2014, p. 46.
[13] Periscope, “What is periscope producer?” https://goo.gl/g6q1uQ, (Datelast accessed 20-April-2018).
[14] Facebook, “Introducing facebook surround 360: An open, high-quality
3d-360 video capture system,” https://goo.gl/8BHRko, (Date last accessed 20-April-2018).
[15] D. C. C. Zhou, “Making 12k 360 vr streaming a reality: Why and how
we did it,” https://goo.gl/RX355T, (Date last accessed 20-April-2018).
[16] A. Bergkvist, D. C. Burnett, C. Jennings, and A. Narayanan, “Webrtc
1.0: Real-time communication between browsers,” Working draft, W3C,
vol. 91, 2012.
[17] C. Kutza, “Webrtc video chat,” https://goo.gl/aGNThV, (Date last accessed 20-April-2018).
[18] K. Yatani, “Effect sizes and power analysis in hci,” in Modern Statistical
Methods for HCI. Springer, 2016, pp. 87–110.
[19] S. Hashizume, I. Suzuki, K. Takazawa, R. Sasaki, andY. Ochiai, “Telewheelchair: The remote controllable electric
wheelchair system combined human and machine intelligence,”in Proc. of the 9th Augmented Human Int. Conf., ser. AH’18. New York: ACM, 2018, pp. 7:1–7:9. [Online]. Available:
http://doi.acm.org/10.1145/3174910.3174914[20] B. Ryskeldiev, M. Cohen, and J. Herder, “Streamspace: Pervasive mixed
reality telepresence for remote collaboration on mobile devices,” Journalof Information Processing, vol. 26, pp. 177–185, 2018.