arXiv:submit/3134899 [cs.HC] 17 Apr 2020

SportsXR - Immersive Analytics inSportsTica Lin
Harvard University
Cambridge, MA 02138, USA
mlin@g.harvard.edu

Yalong Yang
Harvard University
Cambridge, MA 02138, USA
yalongyang@g.harvard.edu

Johanna Beyer
Harvard University
Cambridge, MA 02138, USAjbeyer@g.harvard.edu

Hanspeter Pfister
Harvard University
Cambridge, MA 02138, USA
pfister@g.harvard.edu

Abstract
We present our initial investigation of key challenges andpotentials of immersive analytics (IA) in sports, which wecall SportsXR. Sports are usually highly dynamic and collaborative by nature, which makes real-time decision making ubiquitous. However, there is limited support for athletes
and coaches to make informed and clear-sighted decisions
in real-time. SportsXR aims to support situational awareness for better and more agile decision making in sports.
In this paper, we identify key challenges in SportsXR, including data collection, in-game decision making, situatedsport-specific visualization design, and collaborating withdomain experts. We then present potential user scenariosin training, coaching, and fan experiences. This position paper aims to inform and inspire future SportsXR research.

Author Keywords
Immersive analytics; sport analytics; visual analytics; information visualization.
Permission to make digital or hard copies of part or all of this work for personal orclassroom use is granted without fee provided that copies are not made or distributedfor profit or commercial advantage and that copies bear this notice and the full citationon the first page. Copyrights for third-party components of this work must be honored.
For all other uses, contact the owner/author(s).
Copyright held by the owner/author(s).
CHI’20,, April 25–30, 2020, Honolulu, HI, USA
ACM 978-1-4503-6819-3/20/04.
https://doi.org/10.1145/3334480.XXXXXXXCCS Concepts•Human-centered computing → Mixed / augmented reality; Virtual reality; Visualization; Human computerinteraction (HCI);

Introduction
Immersive Analytics (IA) builds upon novel immersive technologies and extends data visualization and analytics ca-

pabilities beyond traditional desktop workspaces. For example, IA can facilitate the exploration of heterogeneous
sports datasets by making use of large display spaces,
providing multi-sensory interfaces, promoting expert collaboration, and enabling situated analytics. Situating thevisualization right in the context of the data and events eliminates the gap between people, data, and tools [14]. Potential applications of immersive analytics cover a wide rangeof domains, such as life and health sciences, construction
site management, supply chain, and factory planning [8].
Immersive analytics in sports has received relatively low attention so far. However, due to its highly collaborative andstrategic nature, sports have a huge potential for immersive
analytics.

Figure 1: A simulated basketball
fan experience with player statistics
shown in AR.

Figure 2: Interactive play sketchingwith synthesized defenses by Seidl
et al. [12].

Sports generate a huge amount of heterogeneous data,such as real-time positions of players, physiological measurements of players, tactical trajectories, and scouting insights. Current sports data visualization research mostly focuses on visual analytics systems on traditional 2D screens [9],which are usually used off-line. However, in practice, there
is a huge demand for in-game decision making. Thus, amore situated and user-centered approach to visualize andanalyze sports data is required to bring real-time analytics to decision-makers (e.g., players, coaches, and teamexecutives). Applying IA to sports can engage a broader
audience with embodied analytic capabilities and enabledata-driven decision making in various use cases. For example, a real-time overlay of a player’s box score and an
intuitive data analysis interface on a heads-up display could
bring fan engagement to the next level (see Fig. 1).
In this paper, we present the challenges and potentials of
applying IA to sports to eliminate the gap between sports
data, people, and tools from the perspectives of IA researchers.
We, furthermore, outline a vision of SportsXR in three spe-cific user scenarios that empower athletes, coaches, andfans.

Related Research Fields
SportsXR builds upon a variety of research fields. At its
heart, SportsXR integrates sports analytics with situatedanalytics. Situated analytics is a subset of the larger field of
immersive analytics, which focuses on applying embodied
display and interaction techniques (e.g., augmented andvirtual reality) for data visualization and analytics. Situatedanalytics, in addition, emphasizes the use of augmented
reality to link abstract visual representations to objects inthe physical world.
Sports Analytics
Sports analytics has become popular with the successof the MLB Moneyball story in 2002. Oakland Athletics
General Manager Billy Beane, with a limited budget, usedsabermetrics to draft undervalued players and subsequently
won games against rich teams like the New York Yankees [7].
The recent explosion of computational power, motion capture technology, and statistical methods make advanced
sports analytics feasible in many scenarios. This includes
game outcome prediction, measurement and evaluation ofplayer performance, analysis of rules and adjudication, andwithin-game strategy.
However, these fully automatic methods do not integrate theknowledge of decision-makers (i.e., coaches, scouts, andmanagers) in an effective way [1]. Data visualization andvisual analytics have been integrated into the sports analytics workflow to enable human-in-the-loop analytics. For
example, Seidl et al. [12] use NBA game tracking data toperform data-driven ghosting defense prediction. They alsooffer a tablet-based interface for coaches to perform realtime strategy planning (see Fig. 2). Wu et al. [17] create aholistic visualization system for table tennis match data andempower domain experts to find unnoticed patterns (seeFig. 3). Perin et al. [9] have compiled a comprehensive survey of these systems. However, there is still a clear demand
for more natural and transparent interface design to intuitively interact with sports data to perform meaningful data
analytics and prediction.
Figure 3: Interactive visualizationof table tennis data by Wu etal. [17]. © 2018 IEEE, reprinted
with permission.

Figure 4: An in-situ collaborative
scenario demonstrated by Yang etal. [18]. © 2019 IEEE, reprinted
with permission.

Immersive and Situated Analytics
Immersive analytics leverage new display and interaction
techniques, such as AR/VR headsets, for visual analytics [8]. Compared to traditional displays, AR/VR headsets
can render very large displays at low cost with a high degree of portability compared to wall displays and CAVEs.
Therefore, AR/VR headsets can create immersion throughsituated and collaborative visualization. This opens up opportunities for direct interaction with visuals that are radically different from desktop computing paradigms. Situatedanalytics is a form of in-situ interactive visual analysis. In
situated analytics, visual representation of information isimmediately linked to physical objects to facilitate sensemaking [15] (e.g., see Fig. 4).
VR for sports training has been preliminarily explored, e.g.,
for basketball and baseball: Tsai et al. [16] use a VR simulation to train basketball players’ decision-making skills, andZou et al. [19] visualize simulated pitch and bat swing data
in VR to improve a baseball player’s batting eye. Applying
AR in sports is largely unexplored and can bring unique
opportunities for the integration of digital information andphysical movements.
Industrial Sports Analytic Platforms
There are several innovative commercial products applying
AR/VR for sports training, coaching, and fan experiences.
STRIVR [13] uses VR as an immersive learning platform
to simulate real game scenarios to train decision making

and mental preparation for professional football players,
skiers, and basketball players. Rezzil [10] provides similar
VR training with coaching and analytic features for professional soccer players. Second Spectrum [11] experimentson player tracking and AR features for visual information
augmentation on broadcasting and video re-play. CourtVision [4] provides information and stats overlays in AR for
enhancing live sports viewing experiences for NBA basketball fans within minutes of the live game. However, those
applications only cover some sub-areas of SportsXR, leaving many other areas still being unexplored.

Challenges
We have identified four main challenges in SportsXR research: 1) sports data collection; 2) in-game decision making; 3) situated sports-specific visualization design; and 4)
collaboration with domain experts.
Sports Data Collection and Extraction
Sports data are inherently complex. Typical datasets contain a combination of heterogeneous, multi-dimensional,
and unstructured data. Examples include box score data,
tracking data, scouting reports, game video clips, players’
mentality, and many other qualitative metrics [9]. Standards
for data collection need to be enforced to ensure that thecollected data is unbiased and compatible with the subsequent analytic workflow. Meeting these standards requires considerable efforts for real-time data recording, preprocessing, data cleaning, and formatting. Furthermore, inthe context of situated and in-game analytics, data often
needs to be extracted from video feeds and live sensors.
This, in turn, requires state-of-the-art computer vision techniques to track game objects and players. For example,
basketball analytics collects dynamic movement data of all
players and detects shot types and defensive vs. offensive
tactics to evaluate the performance of each player, whilebaseball analytics focuses more on pitch type and base rate
to evaluate player value.
The first challenge, therefore, lies in identifying, collecting,and extracting various data required for different sports andcontexts.
In-game Decision Making
Research on optimal statistical models and visualization
tools for game prediction and evaluation in various sports
has seen a huge growth in the last decade. However, current sports analytics only occur asynchronously in the handsof data scientists, leading to inefficient communication andinaccurate decision making on the athlete side. By bringing
in-situ analytic power to domain experts such as coaches
and athletes, immersive analytics will play an important rolein eliminating the gap between data-driven strategies andexecutions.
The second challenge, therefore, lies in making analytics
models not only understandable to coaches and athletes
but to also enable fast in-game and in-situ analytics.
Situated Sport-specific Visualization Design
One distinguishing feature of SportsXR is that analytics
need to be performed in-situ and need to be updated dynamically to maximize its impact. However, coaches andathletes primarily have to focus on real-world movements
and events. Thus, the visualizations in SportsXR need to besituated, highly dynamic, concise, and context-dependent
to play an auxiliary role. For example, a basketball coachusing SportsXR to evaluate offensive options needs different analytics based on a player’s location and a defender’smovement. A spatial shot percentage map of the ball handler might be helpful when the defense is wide open, butwhen the player is being contested, the shot percentagesof other shooters on the team are more important. Deciding

when and how to change the presented data to optimallysupport data-driven decision making will be a key design
consideration in SportsXR.
The third challenge, therefore, lies in defining the contextdependent factors based on real-world events to decide
how to present and interact with dynamic situated data.
Collaboration with Domain Experts
User-centered design has been proven to be an effective
way of engaging end-users [6]. Batch et al. [2] worked
closely with economic data analysts to integrate immersive visualization into their actual workflow. In contrast, domain experts in sports rarely perform data analytics themselves but make in-game decisions based on experiencerather than based on data. To understand the decisionmaking workflow of coaches and athletes, researchers firstneed to gain in-depth domain knowledge. Based on thisdomain knowledge, they can then identify and extract analytic components from the heuristic insights in order to improve the accuracy of their decision making. Furthermore,
researchers need to educate their sports collaborators on
analytics and visualization methods, as well as introduce
them to novel AR interfaces for situated analytics.
In our current research project, we collaborate closely withboth Harvard Men’s and Women’s Basketball teams togather first-hand expert knowledge. Building personal sport
expertise and working with data analysts on an interdisciplinary sports analytics team can lead to more effective
collaborations with coaches and players.
The fourth challenge, therefore, lies in establishing an effective collaboration between immersive analytics researchersand sports experts.

Figure 5: Basketball shooting training in SportsXR.

Potential User Scenarios
We have identified the following SportsXR scenarios: athlete training, coaching, and fan experience.
Training
Technological innovations in sports training are not just crucial to improve training effectiveness with instant performance feedback and overall body condition monitoring, butalso to reduce injuries and to excel former records. Most
data collected during training are interpreted and communicated to athletes by their coaches. However, embodied
data analytics can empower athletes with the ability to selfevaluate and modify their techniques in real-time. Furthermore, immersive visualizations can enhance the collaboration between coaches and athletes by embedding thevisuals spatially into the real world (see Fig. 5). A pioneer
example of this was demonstrated by Eliud Kipchoge breaking the 2-hour marathon barrier with the help of a projectedpacing visualization in real-time [3].

Coaching
So far, sports analytics has had its highest impact on coaching decision making. Data scientists in the backroom apply
advanced machine learning and computer vision to convert
video and statistical data into strategic insights. On top of
the advanced statistical models that track detailed winning
factors in playersâĂŹ performance, the close collaboration between analysts and coaches has been the key factor
for the successful team rebuild of the NBA Philadelphia
76ers [5]. However, for many teams and individual sports
like tennis or track-and-field, employing a professional data
analytics team is unfeasible. The spatial immersion and insitu decision making of situated analytics allow SportsXR toembed analytics into the coaching workflow for a seamlessintegration of analytic and coaching insights.
Fan Experience
Immersive technologies have gradually become a big partof social experiences, such as AR filters on Snapchat or
Instagram and VR game watching of the FIFA World Cup.
To bring fan experience to the next level, personal storytelling through data visualization and content creation plays
an important role in deepening a sports fan’s engagement.
SportsXR can add digital information overlays in real-time,and provide a more engaging interactive experience suchas video annotation, dynamic data lookup, performancecomparison, and customized view manipulation. The design
challenges for immersive visualization and interaction, in
this case, are largely unsolved research questions.

Conclusions
In this paper, we discuss the trends and challenges IA research communities are facing in the new research field ofSportsXR. We have outlined potential research areas andencourage the exploration of SportsXR applications in training, coaching, and fan experience. Building upon the con-

tinuous research efforts in immersive analytics, SportsXR
presents unique challenges for research: First, sports datacollection is challenging due to complex data types andcontexts. Second, analytics models need to be suitable for
fast, situated decision making by athletes and coaches.
Third, SportsXR development requires the design of novel,
sports-specific, dynamic, and situated data visualizationand interaction methods, and fourth, SportsXR relies on
a close collaboration between researchers and sports domain experts. We hope that SportsXR will eliminate gaps
between analytic and athletic insights, propel innovation in
sports, and make sports analytics available to larger audiences.

Acknowledgments
We wish to thank Coach Kathy Delaney-Smith, Mike Roux,
Mark Kaliris, and Lindsay Werner at Harvard Women’s Basketball, and Mike Sotsky and Casey Brinn at Harvard Men’s
Basketball for their time and expertise. This research is
supported in part by King Abdullah University of Scienceand Technology (KAUST) and the KAUST Office of Sponsored Research (OSR) award OSR-2015-CCF-2533-01.

REFERENCES[1] Benjamin C Alamar. 2013. Sports Analytics: A Guide
for Coaches, Managers, and Other Decision Makers.
Columbia University Press.
[2] Andrea Batch, Andrew Cunningham, Maxime Cordeil,
Niklas Elmqvist, Tim Dwyer, Bruce H. Thomas, andKim Marriott. 2019. There Is No Spoon: Evaluating
Performance, Space Use, and Presence with Expert
Domain Users in Immersive Analytics. IEEE
Transactions on Visualization and Computer Graphics
26, 1 (2019), 536–546. DOI:
http://dx.doi.org/10.1109/TVCG.2019.2934803[3] Kim Bellware. 2019. Lasers, rabbits and new Nikes:
How the 2-hour marathon barrier was broken. (2019).
https://www.washingtonpost.com/sports/2019/10/
15/lasers-rabbits-new-kicks-how-hour-marathonbarrier-was-broken/[4] CourtVision. 2020. (2020).
https://www.clipperscourtvision.com/[5] Marcus Hayes. 2018. Analytics-driven Sixers ride thenumbers to NBA playoffs. (2018).
https://www.inquirer.com/philly/sports/sixers/
sixers-76ers-philadelphia-analytics-processnumbers-nba-playoffs-miami-heat-brett-brownbryan-colangelo-20180418.html[6] Heidi Lam, Melanie Tory, and Tamara Munzner. 2018.
Bridging from Goals to Tasks with Design Study
Analysis Reports. IEEE Transactions on Visualization
and Computer Graphics 24, 1 (Jan. 2018), 435–445.
DOI:http://dx.doi.org/10.1109/TVCG.2017.2744319[7] Michael Lewis. 2004. Moneyball: The Art of Winning
an Unfair Game. W. W. Norton & Company.
[8] Kim Marriott, Falk Schreiber, Tim Dwyer, Karsten
Klein, Nathalie Henry Riche, Takayuki Itoh, Wolfgang
Stuerzlinger, and Bruce H. Thomas (Eds.). 2018.
Immersive Analytics. Lecture Notes in Computer
Science, Vol. 11190. Springer International Publishing,
Cham. DOI:
http://dx.doi.org/10.1007/978-3-030-01388-2[9] C. Perin, R. Vuillemot, C. D. Stolper, J. T. Stasko, J.
Wood, and S. Carpendale. 2018. State of the Art ofSports Data Visualization. Computer Graphics Forum37, 3 (June 2018), 663–686. DOI:
http://dx.doi.org/10.1111/cgf.13447[10] Rezzil. 2020. (2020). https://rezzil.com/

[11] Second Spectrum. 2020. (2020).
https://www.secondspectrum.com/[12] Thomas Seidl, Aditya Cherukumudi, Andrew Hartnett,
Peter Carr, and Patrick Lucey. 2018. Bhostgusters:
Realtime interactive play sketching with synthesizednba defenses. In Proceeding of the 12th MIT Sloan
Sports Analytics Conference, Boston, MA. Boston:MIT.
[13] STRIVR. 2020. (2020). https://www.strivr.com/[14] T Chandler, M Cordeil, T Czauderna, T Dwyer, J
Glowacki, C Goncu, M Klapperstueck, K Klein, K
Marriott, F Schreiber, and E Wilson. 2015. Immersive
Analytics (2015 Big Data Visual Analytics (BDVA)). 1
8. DOI:
http://dx.doi.org/10.1109/bdva.2015.7314296[15] Bruce H. Thomas, Gregory F. Welch, Pierre
Dragicevic, Niklas Elmqvist, Pourang Irani, Yvonne
Jansen, Dieter Schmalstieg, AurÃl’lien Tabard, Neven
A. M. ElSayed, Ross T. Smith, and Wesley Willett.
2018. Situated Analytics. In Immersive Analytics, Kim
Marriott, Falk Schreiber, Tim Dwyer, Karsten Klein,
Nathalie Henry Riche, Takayuki Itoh, Wolfgang
Stuerzlinger, and Bruce H. Thomas (Eds.). Vol. 11190.
Springer International Publishing, Cham, 185–220.

DOI:
http://dx.doi.org/10.1007/978-3-030-01388-2_7[16] Wan-Lun Tsai, Li-wen Su, Tsai-Yen Ko, Cheng-Ta
Yang, and Min-Chun Hu. 2019. Improve theDecision-making Skill of Basketball Players by an
Action-aware VR Training System. In 2019 IEEE
Conference on Virtual Reality and 3D User Interfaces(VR). IEEE, Osaka, Japan, 1193–1194. DOI:
http://dx.doi.org/10.1109/VR.2019.8798309[17] Yingcai Wu, Ji Lan, Xinhuan Shu, Chenyang Ji, Kejian
Zhao, Jiachen Wang, and Hui Zhang. 2017. iTTVis:
Interactive Visualization of Table Tennis Data. IEEE
Transactions on Visualization and Computer Graphics
24, 1 (2017), 709–718. DOI:
http://dx.doi.org/10.1109/tvcg.2017.2744218[18] Yalong Yang, Tim Dwyer, Bernhard Jenny, Kim
Marriott, Maxime Cordeil, and Haohui Chen. 2019.
Origin-Destination Flow Maps in Immersive
Environments. IEEE Transactions on Visualization andComputer Graphics 25, 1 (2019), 693–703. DOI:
http://dx.doi.org/10.1109/tvcg.2018.2865192[19] Liyuan Zou, Takatoshi Higuchi, Haruo Noma,
Lopez-Gulliver Roberto, and Tadao Isaka. 2019.
Evaluation of a Virtual Reality-based Baseball Batting
Training System Using Instantaneous Bat Swing
Information. In 2019 IEEE Conference on Virtual
Reality and 3D User Interfaces (VR). IEEE, Osaka,
Japan, 1289–1290. DOI:
http://dx.doi.org/10.1109/VR.2019.8798041