The Quest for: Embedded Visualization forAugmenting Basketball Game Viewing Experiences
Tica Lin, Zhutian Chen, Yalong Yang, Daniele Chiappalupi, Johanna Beyer, Hanspeter PfisterFig. 1: Our Omnioculars prototype supports a personalized and interactive game viewing experience with embedded visualizations
for in-game analysis. We created a simulated basketball game environment to support our design probe of embedded visualizations.
Abstract—Sports game data is becoming increasingly complex, often consisting of multivariate data such as player performance stats,historical team records, and athletes’ positional tracking information. While numerous visual analytics systems have been developed forsports analysts to derive insights, few tools target fans to improve their understanding and engagement of sports data during live games.
By presenting extra data in the actual game views, embedded visualization has the potential to enhance fans’ game-viewing experience.
However, little is known about how to design such kinds of visualizations embedded into live games. In this work, we present auser-centered design study of developing interactive embedded visualizations for basketball fans to improve their live game-watchingexperiences. We first conducted a formative study to characterize basketball fans’ in-game analysis behaviors and tasks. Basedon our findings, we propose a design framework to inform the design of embedded visualizations based on specific data-seeking
contexts. Following the design framework, we present five novel embedded visualization designs targeting five representative contexts
identified by the fans, including shooting, offense, defense, player evaluation, and team comparison. We then developed Omnioculars,
an interactive basketball game-viewing prototype that features the proposed embedded visualizations for fans’ in-game data analysis.
We evaluated Omnioculars in a simulated basketball game with basketball fans. The study results suggest that our design supports
personalized in-game data analysis and enhances game understanding and engagement.
Index Terms—Sports Analytics, Embedded Visualization, Data Visualization

1

I NTRODUCTION

Sports broadcasting and streaming have seen exponential growth inrecent years. Basketball, for example, is one of the most popular teamsports worldwide. The NBA league alone averages about 1.4 million
viewers per game on ESPN [3]. With the advent of novel sensing andimage processing techniques, an increasing amount of live data can
now be collected in each NBA game, including scores, play-by-play
data, and even player trajectories. This data is released online in realtime and is widely used by sports analysts and coaches to measure
the performance of teams, analyze the strengths and weaknesses ofplayers, and make in-game decisions. Overall, in-game sports data
has gradually become indispensable for professional basketball teams
to develop winning strategies [24, 31, 50]. Several interactive sports• Tica Lin, Zhutian Chen, Johanna Beyer, and Hanspeter Pfister are with John
A. Paulson School of Engineering and Applied Sciences, Harvard University,
Cambridge, MA. E-mail: {mlin, ztchen, jbeyer, pfister}@g.harvard.edu• Yalong Yang is with the Department of Computer Science, Virginia Tech,Blacksburg, VA. E-mail: yalongyang@vt.edu• Daniele Chiappalupi is with the Department of Computer Science, ETH
Zürich, Switzerland. E-mail: daniele.chiappalupi@inf.ethz.ch. Work wasdone while Daniele Chiappalupi was an intern at Harvard University.

analysis systems have been proposed to help experts consume andanalyze in-game data [28, 40, 55]. However, these systems are not yet
readily available for sports fans. Subsequently, the information needsof a general audience during a game are largely unmet.
During a game, sports spectators and fans seek different data thanexpert analysts to facilitate their own game understanding and engagement. After decades of development, scoreboards or data panels shown
on TV are still the dominant method for spectators to obtain data during
live games. However, presenting data in this way often does not fulfill
the individual needs of the audience. Consequently, perhaps the mostcommon solution for spectators is to look up extra data on mobile
devices or separate screens. For example, basketball fans usually lookup live box scores and play-by-play data during a game on the official
NBA website [7], ESPN [3], or on the respective phone apps. Such
practices, however, distract from the game and limit the amount of
in-game data analysis that would increase game understanding and engagement for fans and viewers. Hence, spectators would greatly benefit
from an effective way of analyzing in-game data and information that
does not distract from the actual game.
Embedded visualization provides a promising opportunity to satisfy
the in-game information needs of spectators, as it directly visualizes
data within its physical context [53]. Thereby, embedded visualizations
eliminate the need for context switching and reduce distractions whenlooking up data. Recently, some commercial [10] and research [15] sys-

tems have utilized embedded visualizations to augment sports videos(so-called augmented sports videos), providing extra information together with the video in a seamless and engaging manner. However,
all these systems are designed for experts to create augmented sports
videos post-game and do not support custom visualizations for generalaudiences during a live game. In the live game, visualizations need tobe simple and allow viewers to consume and analyze data on the fly,which can differ significantly from visualizations in post-game scenarios. As such, it is still an open question on how to best use embedded
visualizations to satisfy the in-game data needs of a sports audience.
In this study, we aim to fill this gap by exploring the design spaceof embedded visualizations for live sports in-game data analysis. We
adopted a user-centered design approach throughout the study to answer three questions: First, we observe user needs and their workflow
through a formative user study to answer “What data do fans desire
in a live basketball game?”. Based on survey responses and in-depth
user interviews with active basketball fans, we identified user needs in
obtaining more advanced game data and control over their live gameviewing experience. Next, we tackle the question of “What are thedesign considerations for embedded visualizations in live games, basedon user needs?”, by proposing a design framework for embedded
sports data visualization based on scenario, data, task, and embedded
vis. We further identify five representative game contexts, i.e., shooting,offense, defense, player performance, and team comparison, and develop embedded visualizations for them. To answer our final questionof “How well do embedded visualizations help users enhance gameunderstanding and engagement?”, we developed Omnioculars, anembedded visualization prototype for live TV basketball game viewing
to support our design exploration. We took inspiration from Harry
Potter’s Omnioculars for Quiddich games [8], which gives fans theability to see everything during a game. We created a simulated gameenvironment in Unity3D [52] based on NBA player tracking data to embed live visualizations on top of the simulated game. We conducted acontrolled user study with 16 active basketball fans using Omnioculars.
Users were able to generate distinct game insights with each embedded
visualization and to utilize Omnioculars to personalize their in-game
analysis in our simulated basketball game videos.
In summary, our contribution consists of the first design study onembedded visualization for in-game data analysis of basketball fans.
Second, we propose a design framework to support embedded visualization designs based on game contexts. Third, we built a simulated gamewatching environment and implemented Omnioculars, a novel design
prototype for showing five embedded visualizations in live basketballgame videos based on our design framework. Finally, we evaluate themerits and limitations of embedded visualization for in-game analysis
with basketball fans in a user study.
2

R ELATED W ORK

Visual Analytics for Sports. Advances in sensors and computer vision
techniques have led to the collection of more fine-grained, high-quality
sports data. Perin et al. [38] categorized sports data in visualizations
into box score data, tracking data, and metadata. Given the highly
competitive nature of sports, numerous visual analytics systems havebeen developed to allow domain experts to analyze complex sports datafor discovering winning strategies. For example, SoccerStories [37]employs a series of soccer-related visualizations, such as in-court trajectories and heatmaps, to analyze spatio-temporal data collected in soccer
games. BKViz [28], an interactive data exploration tool for basketballgames, focuses on the analysis of play-by-play data. Fu and Stasko [22]developed interactive visualization systems specifically targeting NBA
journalists. Some sports analytics approaches also leverage machinelearning models to predict and extract patterns from the data [31,34,50].
Recently, with the proliferation of low-cost immersive devices such as
Virtual and Augmented Reality (VR/AR), immersive sports analytics
systems have been gaining traction. Compared to traditional desktop
systems, immersive sports analytics systems provide unique benefits,
including increased spatial understanding, rich embodied interaction,
peripheral awareness, and a large information space [27, 30]. ShuttleSpace [54], a VR-based visual analytics system for badminton data,

enables users to analyze shuttle trajectories from a player’s first-person
view. Following their work, TIVEE [17] allows experts to explore alarge scale of badminton trajectories from both first and third-person
views in VR. While these systems have demonstrated their effectiveness in analyzing sports data, most of them are designed for post-game
analysis by domain experts, leading to a high entry barrier and steep
learning curve. On the other hand, compared to post-game analysis systems, in-game systems usually can be used by both domain experts andfans to complete lightweight analysis. However, only a few approaches
focus on in-game analysis. GameViews [55], for example, supports
data-driven sports storytelling using in-game data for sportswriters andfans. Yet, GameViews mainly uses simple visualizations separated fromthe videos, forcing users to switch contexts when watching a game.
We draw on this line of research to develop an in-game analysis tool
for basketball fans, which features embedded visualizations to enable
lightweight analytics while retaining an engaging game experience.
Embedded Visualization in Sports Videos. Sports data, generally
speaking, is inherently associated with its physical environment (e.g., abasketball court). Visualizing sports data within this physical environment can significantly facilitate the understanding and analysis of thedata. Thus, numerous works have explored visualizing sports data instatic court diagrams. For example, CourtVision [23] uses point-based
visualizations to present the shot distribution and density for NBA data.
Sacha et al. [39] introduced a trajectory-based visualization systemthat allows users to interactively explore the moving trajectories ofsoccer players. To improve the analysis of soccer data, Stein et al. [47]developed a region-based visualization that shows the interaction spaceof each soccer player. In recent years, with the rapid developmentof computer vision and image processing techniques, visualization researchers have started to directly embed their visualizations into sports
videos. For instance, Stein et al. [48] developed a system to assist
soccer game analysis, which automatically extracts and visualizes datafrom and in soccer videos. Stein et al. [46] then extended their work bysemi-automatically selecting the presented data based on the game’s semantic context. However, these works mainly target experts for analytic
purposes and do not fulfill general spectators’ needs. More recently,
Chen et al. [15] explored the design space of augmented sports videos
and presented VisCommentator, a rapid prototyping tool for authoring augmented racket-based sports videos. In summary, current workin augmented sports videos focuses on domain experts’ analytic andauthoring needs. Embedded visualizations designed for sports fans
are still under-explored. In this work, we aim to understand fans’ ingame analysis needs and propose a structured design framework forembedded visualization in basketball videos.
Personalized Game Viewing Systems. When watching a sports game,spectators often look up extra information to satisfy their curiosity,
deepen their understanding of the game, or generate game insights (asfound in our formative study in Sec 3). These information needs varyfrom person to person, leading to a great demand for personalized gameviewing systems. The most common way fans access additional data isusing mobile devices to search the internet when watching the game.
However, this approach inevitably incurs context-switching costs andsignificantly impedes an engaging watching experience. To overcome
this issue, HCI and Computer Graphics researchers have studied interactive game viewing systems that enable spectators to look up data
while still focusing on the game. For instance, Gamebot [56] uses
a chatbot interface on a mobile app for fans to request and consume
game data visualization. ARSpectator [57] uses mobile AR devices tooverlay basic game data onto the scene for on-site spectators. Despite
the focus on interaction techniques to access game data, the existing
work did not evaluate the presentation of game data to facilitate gameunderstanding and engagement. On the other hand, E-Sports, suchas Defense of the Ancients2 [2] (Dota2) and League of Legends [4](LoL), are ahead of most traditional sports in providing personalized
game viewing experiences due to their interactive, online nature. In
Dota2, for example, the audience can interact with the viewing system
to inspect the gaming data (e.g., gold over time) of individual players or
the whole team. Nevertheless, most of these systems only present thelater discussion and affinity diagramming with another author, agreed
on a single coding scheme. Then the lead author finished the coding on
the remaining records collected from the interviews.
3.2

Findings and Discussions

We present findings on fans’ motivation, workflow, and pain points ofperforming in-game data analysis.
Fig. 2: We adopted a user-centered design process throughout the study.
We elicited user needs through a survey and interviews, and iterated
our designs based on user feedback.
data in side-by-side settings, in which the visualizations are displayedin panels separated from the game view. We envision that embedded
visualizations have the potential to seamlessly blend the extra information with the game view, providing a natural, intuitive, and engaging
watching experience. Our work takes a first step towards designingand evaluating personalized game viewing systems with embedded
visualizations for traditional sports.
3

F ORMATIVE S TUDY WITH BASKETBALL FANS

Section 3, 4, and 5 present our user-centered design process towards
designing interactive basketball game-viewing system with embedded
visualizations (Fig. 2). In this section, we detail a mixed-method
formative study using an online survey and interviews with active
basketball fans to better understand the general workflow, pain points,
and best practices of fans to acquire desired data during a live game.
3.1

Procedures

We first used an online survey to collect initial feedback on basketballfans’ data needs while watching live games. We analyzed survey
responses to understand fans’ in-game data analysis behaviors and to
inform the questions of the follow-up interviews (Fig. 2).
Online Survey. The game-watching experiences survey (AppendixA) asked about fandom levels (novice, casual, engaged, and die-hardfans), game watching frequency, and in-game data analysis records.
We asked respondents to watch a live basketball game and record threemoments when they wanted to look up additional data. We advertised
the survey using university mailing lists and only collected responses
from casual fans and above. In total, we gathered 14 responses (R1-14;
M = 10, F = 4; Age: 18 - 45) and 40 records of their data-seeking
moments when watching a live game. All respondents are NBA fans (4casual, 6 engaged, and 4 die-hard fans).
In-depth Interview. We then interviewed six fans from the survey
respondents (I1-6; 4 die-hard and 2 engaged fans) who expressedinterest in the survey. Among them, five have been NBA basketballfans for over ten years. We conducted the interviews remotely using
Zoom in two parts (Appendix B). We first asked respondents about
their overall game viewing experiences and data analysis behaviors inrecent games they watched, including occasions they looked at data
and whether the data was helpful and engaging. We then asked them towatch three video clips selected from a 9-minute post-game highlight
video of an NBA regular-season game1 and think aloud about whatgame insights and data they observed and wanted to see. Each clip
consisted of two to four plays for about 30 seconds and collectively
covered a diverse range of gameplays, including offense strategy, aplayer’s highlight (a layup with a foul), defense performance (a block),or clutch time. The clips also included video effects to augment certain
gameplay, such as replays from different angles, showing a player’s
box score on the screen, etc. The interviewees contributed 63 in-gamedata analysis records on the three clips. No further interviews wereconducted as saturation was met in the analysis.
Analysis. We performed a reflexive thematic analysis [11] on thesurvey and interview data. Three authors began coding separately on
the 40 survey records to create sets of plausible codes, and through1 Golden State Warriors vs. Cleveland Cavaliers, 2015

3.2.1

Practices of Watching Live Basketball Games

Overall, most of the respondents watched NBA games at least once
per week (4 watched 3 games per week, 9 watched 1-3 games per
week, 1 watched less). 12 of them followed a team or a player in
the NBA league. All six interviewees watched 1-3 NBA games per
week, mainly to follow their favorite players. As for the game watching
media, the interviewees explained that while watching games on-site
is engaging and exciting for all fans, they mostly watch live streams
for convenience and economic reasons. When asked to compare theirgame-viewing experience between live streams and highlight videos,interviewees expressed that they preferred to watch live streams overhighlights. One key reason is that highlight videos do not present theflow of the game. Although highlight videos usually provide additional
visual effects, such as play breakdown or embedded data overlays, fanscannot observe how the game developed to the current moment andfeel “it shows no rhythm of the game” (I4). In summary, compared
to on-site watching and highlight videos, live streaming is the favorite
game watching medium among fans we surveyed.
3.2.2

Seeking Data in Various Contexts During Live Games

Seeking data while watching a live game was a common behavior
among fans we surveyed. While watching a game, all 14 respondents in the survey expressed interest in looking up game data; all six
interviewed fans confirmed that they always look up game stats to complement the viewing experience. Based on our analysis of the records,
a context in which a fan seeks data while watching a live game can becharacterized by a,, and. Specifically, in different scenarios (When), fans seek different data (What) to accomplish
different tasks (Why):
Fans seek data in three typical(When). First, fans often
look for data at specific game times, such as before the game, during
halftime, or in the last quarter. These specific game times are defined
based on the rules of the sport, like basketball, and thus usually havespecial meanings. Second, fans seek data when observing a game
event. These events include seeing a game stat, a game action, or theappearance of a new player, “D. Rose made an and-1 play in the 2nd
quarter” (R8). Third, when multiple repetitive game events happen,
fans sometimes want to seek data to evaluate a player’s or team’s
performance related to this specific type of game event, “Thunder ismissing a lot of 3 pointers” (R5).
Fans seek two types of(What). Based on the survey and interviews, the data sought by the fans can be divided into two types,
inspired by Perin et al. [38]. First, game data are directly collected fromgames, including box score, “foul number of Embiid” (R4), tracking
data, “the usage of this particular play” (R9), and video data. The
game data sought by the fans can range from a single game, to multiple
games, or an entire career. Second, metadata includes data beyond
games, such as ranking, player background, and news.
Fans seek data to finish three(Why), ultimately generating insights to deepen their understanding of the game. Inspired by Brehmer
and Munzner [12, 32], we summarized three typical tasks that the fansperform with data during a live game. First, fans seek data to identify
the performance of players or teams. For example, a respondent replied,“I was wondering whether he was getting a triple-double” (R4). In addition to looking at the data of a specific player, fans are also interested in
the data of the whole team, such as the moving trajectories of a team to“understand the offensive and defensive strategies” (I3). Second, fansoften need data to compare the in-game performance of a player orteam to others’ in-game performance or to their historical performance.
For example, a respondent replied, “I want to know how many assists

our team makes compared to other teams” (R5). Five interviewees also
provided that they wanted to compare a player’s in-game performanceto their average performance to evaluate the consistency. Third, fansusually seek data to summarize the performance of players or teams.
Compared to the other two tasks, as pointed out by Brehmer and Munzner [12, 32], the summarize task often involves a deeper analysis of
the data, such as overviewing the game and observing patterns, deriving the cause of the performance, and speculating possible outcomes.
Representative responses include “to have a quick understanding ofboth teams so I know what to expect in the match” (R10) (overview &possible outcome) and “to understand what causes the score difference
between two teams” (R3) (cause of the performance).
3.2.3

Pain Points of Seeking Data in Watching Live Streams

While seeking data was common among basketball fans, the data acquisition process in practice remains hindered by current technologies.
According to the survey, the main method of acquiring data is throughwebsites or mobile apps (30 out of 40). In the interviews, the fansfurther detailed that they access websites (ESPN [3], NBA [7], or Reddit [9]) on the phone or on a separate screen to regularly track box scoredata or team standing. Such practices reflect several issues in the livegame-watching experience:
Data provided by live streams lack diversity. The interviewees agreed
that most information available on live streams, such as game stats andtimers, was useful. However, they also confirmed that the information
provided by the live streams is not diverse and sufficient enough, e.g.,“usually they are not sufficient, only showing a matrix of all player names
and scores (I2)”. Some information that the interviewees would love
to have is usually absent in live streams, such as seeing all box scoresand lineups information, or comparing shooting performance by zones.
This is the main reason that fans search websites to obtain extra data
while watching live games.
Data provided by live streams lack controllability. Interestingly,
although live streams can provide some basic box score data (likegame stats and timers), the interviewees still will check this data on
the website regularly, “Right now, on a separate screen, I can look at
the full box scores to get an updated impression of the players” (I4).
This practice reflects that when watching live games, fans need not
only sufficient data but also the ability to control when and what data
to show. This finding also links to our characterization that users seek
different data in various contexts (Sec. 3.2.2).
Data provided by websites are separated from the game videos.
Unsurprisingly, the interviewees complained that looking up data on
the phone or a separate screen reduces the engagement of the gamewatching experience as it introduces numerous context switches between the game videos and websites. For example, “in a live game
there’s a lot going on and likely I missed some events [when looking updata]” (I1). Thus, interviewees would like a more intuitive method topresent the data directly within the game videos.
3.3

Summary

Our formative study found that live streams were the favorite watching
medium for basketball fans we interviewed. While watching live games,
fans seek different types of data in different scenarios for different tasks.
However, current live streams fall short in fulfilling fans’ diverse dataneeds, forcing fans to look up data on separate screens (phones ormonitors), which reduces the engagement in watching games. To tackle
these issues, we propose to embed visualizations into live game videos
to present data in-situ and support in-game data analysis.
4

D ESIGN S PACE E XPLORATION

To answer our second research question of how to design embedded
visualizations for fans’ in-game data analysis, we first developed adesign framework based on the formative study findings (Fig. 3). With
this framework, we then identified 20 contexts from the records offans’ in-game data seeking moments. By walking through these designcontexts using design mockups with the six interviewees, we prioritized
five contexts to be implemented in our prototype.

Fig. 3: Design framework for context-driven embedded visualizations.
4.1

Design Framework

Based on the data-seeking contexts we summarized in Sec. 3.2.2, we
propose a context-driven design framework for embedded visualizations for in-game data analysis (Fig. 3). The framework scaffolds therelationship between four design aspects – “In a certain, theuser performs a
on
using
”:
describes the background of fans’ data analysis behav•
iors. Three design elements can be used to describe a scenario thatdrives a fan’s data need:target (team or player), event (anygame event that is happening), and time (a specific moment ora duration). For example, “When Joel Embiid fouled” describes
a player (target) with a foul (event) at the current time (time);“When we have been making strides with offense for a streak oftime in the game” describes a team (target) make shots (event)over a certain period (time).
that is of interest to fans can be collected in-game or beyond•
games. For embedded visualizations, we focus on in-game dataand exclude metadata. Thus, the data types to be considered
include box scores (game summary scores), tracking data(like player positions), and video. Different data can be further
combined to specify advanced stats, such as “the lineup’s plusminus” or “the shot percentage of #30 when there is no defenderin the front.”• A
is performed on game data by fans to generate insights
about players or teams, including identify (on a player or team),compare (between players or teams), and summarize (amongplayers or teams).
can present in-game data directly in videos to•support fans performing in-game data analysis naturally. Prior research [15] systematically studied the design space of augmented
sports videos and summarized the embedded visualizations usedin augmented videos to present sports data. We follow their design space with a focus on embedded visualizations for basketball,and identified five relevant graphical marks and basic visual elements:
point,
line, area,label, andside panel.
These basic visuals can be composited to construct embedded
visualizations in basketball game viewing.
This design framework helps designers identify fans’ in-game data
analysis contexts by specifying the,, and. Sucha context captures the fan’s specific requirements for in-game data
analysis at a given moment. Once a context is identified, the designer
can then focus on the design of embedded visualizations by composing
the five basic visuals, thereby ensuring the embedded visualizations aredeveloped based on and can satisfy the fan’s analysis needs.
4.2

Design Context WalkthroughFollowing our design framework, we identified 20 design contexts from
103 records of in-game data seeking moments in the formative study
by grouping similar records and summarizing representative contexts.
We created 20 design mockups to demonstrate the contexts (AppendixC) and walked through them with the six fans in follow-up interviews.
Based on their ratings of importance, we prioritized five contexts thatdrove the design of five embedded visualizations in our prototype(Fig. 4).
C1 Shooting – A player shoots (), and the fan wants to
identify and compare () the player’s box score ().

Table 1: The required data for the game simulator.

Fig. 4: We developed five embedded visualization designs based on ourdesign framework and the five prioritized contexts in Sec. 4.2.
Fans found this context useful and suggested avoiding information
overload by only showing the most important stat based on player
and location.
C2 Offense – The offense team runs a play (), and the fanwants to identify and compare () the players’ movement(). Fans commented that this context is very useful to follow
the ball and understand the play better, especially for tracking
off-ball players and identifying open shot opportunities.
C3 Defense – The defense team defends well/poorly (), andthe fan wants to identify () the defense scheme ().
Fans commented that this context helps identify the effective
defenders and track how defensive focus is changing.
C4 Player Performance – A player has made/missed shots(), and the fan wants to compare and summarize () the player’s shot performance by zones (). Fans foundthis context helpful for evaluating a player’s game performancein more detail with direct reference to their average performance.
C5 Team Comparison – At the clutch time (), the fan wants
to compare and summarize () team stats (). Fans
commented instant access to this data is very convenient during
the game break.

Type

Description

Tracking Data
Event Data

The players and the ball’s position over time.
The game events, each event is described by
its time, player, action, and outcome.
The position of each shot event.
Team and player statistics of the game.
Player’s detailed shot percentage by shot locations grouped into zones.
The NBA league’s average shot percentage byshot locations grouped into zones.

Shot Locations
Box Score
Player Zoned
Shot %
League Average
Zoned Shot %Table 1 shows the required data in our current implementation. Overall,as suggested in the user study (Sec. 7.4), the simulated game wasperceived as comparable to an actual game in terms of acquiring gameinsights by the users.
5.2

Embedded Visualization Designs

We iterated over each of our visualization designs with two fans weinterviewed. Both were die-hard fans with substantial basketball knowledge and game viewing experiences. Therefore, we considered them tobe domain experts and asked for feedback on our visualization designs.
5.2.1

Shot Label5
Building upon our design space exploration in Sec. 4, we present Omnioculars, an interactive embedded visualization prototype for live gamedata analysis for fans. Omnioculars consists of three main components:a game simulator, embedded visualizations, and interactions.
5.1

Game Simulator

Ideally, Omnioculars should be built on top of live stream TVs as athird-party game viewing system to enhance the real-world watchingexperience. However, we decided to develop Omnioculars on top of agame simulator instead of real-world broadcasting videos, considering
two aspects: On the one hand, implementing Omnioculars based on
publicly available broadcasting videos is extremely difficult. First,
the data that can be extracted from broadcasting videos is limited
due to the reduced frame rate and low resolution. Second, rendering
embedded visualizations into broadcasting videos is challenging as thecamera parameters of these videos are usually not made public. Third,
broadcasting videos often switch between cameras and insert replays,
reducing the controllability of Omnioculars. On the other hand, ourgoal is not to develop Omnioculars as a fully functional system used inreal-world scenarios but as a design probe [25] to explore the notion ofusing embedded visualizations to improve game viewing experiences.
To gain feedback on our design without being bogged down by the
aforementioned engineering challenges, we use a game simulator as thesystem’s basis, thereby obtaining richer data, better rendering results,
and controllability. Please note that the aforementioned challenges canbe tackled once the source video streams and sensing data from the live
game collected by the TV companies and NBA league are available.
The game simulator is implemented using Unity3D [52], with a 3D
basketball arena and 10 player models (Fig. 1). To simulate a basketballgame, we adopt a data-driven strategy. Specifically, the simulator takes
a set of spatio-temporal data as the input and outputs a virtual basketballgame by moving the players and triggering events based on the data.

Fig. 5: Shot Label shows (a) static outcome and (b) dynamic percentage.
In C1 Shooting, we identified that fans want to see the shot performance of the shooter before and after shooting to predict and evaluate
the shot outcome, and compare the shooter’s shot percentages to theiraverage or others’ performance. We propose athat consistsof two components: 1) a static label with the shooter’s shot outcome at
the shot location (Fig. 5a) and 2) a dynamic label showing the player’s
shot percentage based at their current location (Fig. 5b).
The static label is shown at the shot location after a player shoots
and is visible for 5 seconds and presents the shot outcome, the shooter’s
game, and the average shot percentage with bar charts. This allows fansto get a quick comparison of the shooter’s own in-game performance
without interfering with the continuing game.
The dynamic label shows the shooter’s average shot percentage
based on the location above their head and moves with the player. This
supports comparing the player’s shot performance among their teamand instantly evaluating the player’s shot opportunity based on location.
The dynamic shot percentage is obtained from one of the seven shot
chart zones (see Fig. 8). To support direct visual comparison, the label
is color-coded to compare the player’s shot percentage to the league’s
average, colored in red (hot) if above and in blue (cold) if below. The
dynamic label supports fans to compare the shooter’s shot percentage
to other players, such as evaluating if the shooter is shooting from their
hot spot, and the quality of shot selection.
Alternatively, our original design had a label pop up right after ashot and move with the shooter for a few seconds, showing the scoreand shot percentage of the shooter. However, fans’ feedback suggested
that dynamic labels with dense information are difficult to read andshowing a player’s shot outcome as the player moves into the next playdoes not help with game analysis. Thus, we abandoned this design.

5.2.2 Offense Trajectory
In C2 Offense, fans were eager touse visualization to track the playset and spot open shot opportunities from the off-ball players. Our
design (Fig. 6)
shows the offense players’ trails andhighlights the open space around an
offense player with a circle underneath the player (area). The radiusof the open space is drawn basedon the player’s distance to the closest defender and changes dynamiFig. 6: Offense Trajectory
cally to reflect the interaction between the offense and defense team. We also color the ball handler’s
open space in red to guide the viewer’s attention. Fans can identify
the play run by the offense team through the trails, and easily tell the
speed and rotation of the players. The open space circles reveal how
the defense team reacts to the play and help fans immediately spot open
shot opportunities (larger circles).
Before finalizing the design, we iterated on the design by encoding
other useful tracking data, such as speed measurement, player role,and distance to other players. User feedback suggested that encoding
speed and player roles does not provide much value as a player’s speed
changes constantly and does not represent the actual moves of theplayer. The differentiation of player roles also became less prominent
in modern basketball games. However, encoding the distance to thedefense player was deemed very useful for fans to track the executionof the play. We thus kept this encoding.
5.2.3

Defense Form

Fig. 7: Defense Form shows (a) ball defender and (b) defensive focus.
In C3 Defense, fans wanted to identify the defense scheme and key defenders, and track the change of defenders when a switch happens. The
highlights the key defenders by tracking the distance
between the defenders and the ball handler (Fig. 7a). When a defenderis within 6 ft of the ball, that player is marked as the ball defender (darkblue) and connected to the ball handler with a line. The defender ismarked as a helper (blue) if within 12 ft of the ball. When multiple
defenders are identified as key defenders on the strong side (the ballhandler’s side of the court), the enclosed area is colored to highlightthe region of defensive focus (Fig. 7b). Fans can evaluate the defense
scheme from these visual marks without being overwhelmed.
An alternative design was to highlight the position of all defensive
players (point) and color the enclosed region (area) to achieve the goalof tracking the defense scheme and transformation. However, thisdesign did not specify the key defenders, including the ball defenderand helpers on the strong side. Furthermore, the large enclosed area
can cause visual clutter, and does not represent the actual defense focus,
as the further away the defense players are from each other, the lessof a threat they are. Instead of highlighting all defensive players, ourfinal design focuses on drawing user attention to the effective defenders
and showing the defensive focus of the enclosed area between key
defenders.
5.2.4 Shot Chart
In C4 Player Performance, fans valued the benefit of using
embedded visualization to compare the shooter’s shot performance in game to their own seasonal average in more detail.

We designed
to have twocomponents, a heat map and a player
panel (Fig. 8). The heat map shows
the player’s seasonal shot percentage
by zones (detailed in Sec. 5.2.6). The
areas are color-coded by the comparison to the league shot percentage
average, from dark blue (≤ −10%),
blue (≤ −5%), yellow (±5%), orange(≥ 5%), and red (≥ 10%). We alsovisualize shot attempts and locations
Fig. 8: Shot Chartwith 3D basketball icons on the court
to represent made (colored ball) and missed shots (black ball). The heatmap design allows fans to evaluate the shooter’s detailed game performance directly within the game contexts, such as observing whether
the shooter made shots in their hot zone or comparing the shot number
in each zone. The player panel is shown on the courtside to show
player box score stats, providing a complete picture of the player’s
game performance besides shooting.
5.2.5 Team Panel
Based on C5 Team Comparison,
is designed to provide
an overview of team stats and compare team performance. As shownin Fig. 9, the panel is shown at the
courtside to allow instant look-ups
without losing sight of the game in
the front. Each stat label uses thecolor of the team with the better performance, such as blue for 3 Pointers(Warriors have a higher shot percentage). The bars for each shooting statFig. 9: Team panel(Field Goals, 3 Pointers, and Free
Throws) are colored based on the shot percentage compared to theleague’s average using the same colors as Shot Chart. For example,
Cavaliers have a lower 3PT shot percentage than the league’s average
and are coded in dark blue. Showing a team stats comparison on a panel
is common in current TV streaming services during breaks, but it usually requires cutting away from the court view. Instead, our team panel
design is always-on, without interfering with the game and thereby
providing an effective direct comparison through colors.
5.2.6

Other Design Considerations

Color Usage. Our color encoding of shot percentages uses blue to
red and is based on the convention of basketball analysis, where red
indicates a hot hand (good performance), and blue indicates cold (badperformance). Similarly, Offense Trajectory uses a color scheme ofreddish tones to indicate offense strengths, while Defense Form uses
blue tones to show defense efforts.
Shot Percentage Threshold. We use the NBA’s official shot charts [6]and color palette to compare player and league shot percentages.
Defense Distance Threshold. We define the effective defender as
being within 6 ft of the offense player based on the definition of an open
shot [29]. The threshold decreases to 3 ft if the offensive player has
passed the defender, as a threat from the back is low. We double thethreshold (12 ft) for helpers as an approximation to indicate the strong
side range as there is no accurate definition for defensive helpers.
Shot Chart Zones. We segmented the shot chart into seven zones,
including rim (within 8 ft), mid-range (left/right-wing), corner three(left/right), and 3-point range (left/right). We aggregated the 19 zones
in the NBA shot chart [5] to strike a balance betwen visual complexity
and usefulness.
5.3 Interactions
While the five embedded visualizations can be displayed based on
events, fans still desire control over when and what data should be
displayed. During the user study, we applied the Wizard-of-Oz (WoZ)

method [19, 20] to allow fans to control the visualizations with voicecommands, while the experimenter manually turned each visualization
on and off. The WoZ method is commonly used to evaluate complex,
intelligent systems while reducing implementation effort [13, 14, 36].
Most importantly, the current speech recognition systems cannot meetthe speed and accuracy requirements for fast-paced use cases, likesports. We envision with the advance in speech recognition and natural
language processing, a robust system will be available in the near
future. By using a WoZ method, we can ensure the consistency ofuser experiences without being hindered by current speech recognition
systems. Nevertheless, while interaction techniques are not the focus of
our study, we identified insights regarding users’ interaction preferences
as implications for future work in Sec. 7.3.
6 U SER S TUDY
We conducted a controlled user study with active basketball fans toevaluate the usefulness and user engagement of Omnioculars.
6.1 Experiment Set-Up & Participants
To simulate the game contexts, we chose a famous game between
Golden State Warriors and Cleveland Cavaliers on Dec 25th, 2015. This
Christmas game featured the best teams from the previous season’s
final, who faced each other in the championship again, and was selected
as one of the best games by the NBA in the 2015-16 season. All therequired data, like event data, shot locations, and player zoned shot
percentage, is publicly available [1, 5, 21, 26], except for the tracking
data. We could only obtain 2D tracking data of the ball and players [43].
To preserve the authenticity of the game viewing experience in our
prototype, we manually annotated the vertical positions and polishedthe player model animations to reflect the game events as closely as
possible.
For our study, we selected three video clips from the simulated game.
Each clip consists of about 30 seconds with two to three plays, focusing
on shooting, offense and defense performance, and the clutch timesituation. We used two video clips in the first part of the user study andcounter-balanced them for use as either training or study clip among
participants. We used the third clip, which contains the clutch timesituation, for the second part of personalized game analysis.
We recruited 16 participants through the university mailing list after
an initial screening of their fandom levels (P1-16; M = 13, F = 3; Age:
18 - 45). We targeted active fans who reported watching games at least
once a week during the game season and were interested in looking up
data during the game. Four identified as casual fans, seven as engagedfans, and five as die-hard fans. We conducted the user study over Zoom,
which took one hour per participant to complete. We compensated eachparticipant with a $20 gift card.
6.2 Design & Procedures
The study had two parts. The first part focused on assessing theunderstandability and usefulness of the five embedded visualizations (
,
,
,
, and
). After being introduced to the study and filling out aconsent form, the participants went through five rounds, one for eachvisualization condition. At the beginning of each round, we introduced
each visualization design and led participants through a training task tothink aloud their game insights from analyzing the embedded visualization in a game clip. After they were familiar with the visualization
technique, they completed one study task of analyzing the game withthe embedded visualization on another game clip. The participants
were asked to rate and comment on the useful contexts supported byeach visualization. The second part focused on the usefulness and engagement of the Omnioculars system as a whole. We asked participants
to freely explore different visualization combinations using voice commands. Participants first completed a training task to analyze a gameclip with interactive visualizations using voice commands. They thencompleted one study task of thinking aloud their game insights fromusing interactive visualizations in another game clip. We asked them torate the usefulness and engagement of the overall Omnioculars systemand to comment on their strategy of using different visualizations toFig. 10: User study results. In Part 1, (a) participants rated all fiveembedded visualizations to be easy to understand, helpful, fun to use,and novel ways to present game data (Mdn ≥ 5). In Part 2, (b) participants confirmed the usefulness and engagement of Omnioculars, werelikely to use it in future games (Mdn= 7), and perceived the simulated
game as comparable to an actual game (Mdn= 6). (c) Participants useddifferent visualizations to perform game analysis.
generate game insights. At the end of the study, participants filled out apost-study questionnaire.
6.3

Measures

Part 1. For each embedded visualization condition, we collected thesubjective ratings on a 7-point Likert scale from low to high, including
understandability, usefulness, engagement, and novelty. We also collected qualitative feedback, including the game insights generated by
the participants while watching the game clip with each visualization,and the oral feedback about the contexts in which they found eachvisualization helpful and interesting to use.
Part 2. We collected subjective ratings on the usefulness and engagement of Omnioculars, where we derived our questions from priorwork [35], including “It was helpful”, “It was fun”, “I felt in control”, “I felt encouraged”, and “I am likely” to use Omnioculars to
consume the desired game data in-game. We collected qualitative feedback of participants’ strategy of using different visualizations and thestrengths, weaknesses, and suggestions for Omnioculars in a post-study
questionnaire.
Qualitative Analysis. Participants’ responses were analyzed by two
authors using affinity mapping. In Part 1, we derived primary use cases
for each visualization by grouping similar user insights. In Part 2, wecategorized user strategy in personalizing the visualizations and thesubsequent insights. The grouping results are shown in Appendix D.
7

R ESULTS & D ISCUSSION

We present the user study results on how well embedded visualizations
help users enhance game understanding and engagement, and discuss
the design implications for future interactive game viewing.
7.1 What Game Insights Do Fans Generate with Each Embedded Visualization?
In Part 1 of our study, participants provided their game insights while
using each embedded visualization, and commented on helpful use
cases. We discuss user subjective ratings and findings below. A detailed
categorization of user comments can be found in Appendix D.
Five embedded visualizations were perceived to be useful and novel.
In Fig. 10a, participants rated each visualization from 1 (low) to 7
(high) on Q1) easy to understand, Q2) helpful, Q3) fun to use, and
Q4) novel ways to present game data. The majority rated all fiveembedded visualizations positively (> 4) in usefulness and engagement,
reporting that they were easy to understand (≥88%), helpful (≥75%),
and fun (≥69%). For novelty, Shot Label was rated novel by 94% ofthe participants, Offense, Defense, and Shot Chart by 75%, and Team
Panel by 56%.
Participants used theto evaluate individual player performance and player decisions. Participants used color on the dynamic label to evaluate players’ shot performance, “Sean Livingston isvery low from the 3-point line (P2)”. Furthermore, some participants
contemplated the player’s decision-making, “When I see Lebron drives,the color changed drastically, so I get that why he did not make thethree-point shots and tried to take the drive. (P1)”. Overall, participants found the Shot Label most helpful to focus on player and ballmovement, and after shooting.
Participants used theto spot the shot opportunity
and team strategy. Participants used the circle underneath the players
to evaluate the shot opportunity, “They passed to people who had more
space, or a larger circle, at the time (P7)”, and the trails to read the teamstrategy, “Like this run, [trails] help me understand the team better,
not just the player individually (P1)”. Participants found the Offense
Trajectory most useful when they focused on team offense strategy
and following offense during the transition, “ When defense changes
quickly, seeing opportunity arose when somebody was breaking away
to the basket or slipped away from the defenders is helpful (P3)”.
Participants used theto track defense changes andstrategy. Participants used the color of the circle underneath the players
to identify defenders. They also observed the defense strategy, “Theshaded area between the defenders shows how the defense blocked thepaint area to get the rebound (P6)”. Participants found the Defense
Form most useful when they wanted to track defense changes during aswitch or transition and to identify the defense strategy.
Participants used theto examine the overall shot performance and shot decisions. Participants used the heat map and virtual
balls on the court to evaluate the player’s shot performance. They also
derived insights on players’ shot decisions and on how players compare between each other, “It’s interesting to see the chart differences
between shooters and role players. (P13)”. Participants found the Shot
Chart most useful to understand shot selection, “When there were lotsof shots from a particular area, to understand if they were high percentages shot or not (P11)”, and to evaluate players’ shot performance
immediately, “to understand someone’s performance and contribution,
especially with players I don’t know (P14)”.
Participants used theto compare team performance.
Participants used the team panel to check on game stats. They also
derived game insights from the color encoding, “Interesting to see bothteams had lower shot percentages compared to the league average(P10)”. Participants found the Team Panel most useful to follow theteam’s progress throughout the game and to analyze areas of focus,“When the game score is close, I can see the key factors to evaluate andpredict the outcome (P8)”.
Overall, each embedded visualization provides helpful data on
different game aspects for the participants. The obtained insights andidentified use cases aligned with our design contexts. While individual
needs and knowledge levels vary, all participants were able to generate
different game insights from each visualization.
7.2 How Do Fans Personalize Omnioculars to Enhance
Game Understanding and Engagement?
In Part 2, participants interacted with Omnioculars to use different
visualizations to generate game insights. We were interested to see
how they combined or altered visualizations during the game, how theirindividual preferences impacted their strategies, and how the selected
embedded visualizations helped users generate game insights.
Participants used various combinations of the different visualizations. Fig. 10c shows the usage patterns of the five visualizations
for each participant (the amount of time a certain visualization wasdisplayed). On the individual level, each participant chose different visualization combinations and focused on different aspects of the game.
Six participants used all five visualizations, and all users used at least
three. Each visualization was used frequently (> 33%) by several users

Fig. 11: (a) and (b) show personalized visualizations configured bydifferent users during the user study.
each, i.e., Shot Label (by 13 users), Offense (9), Defense (9), Shot
Chart (4), and Team Panel (7).
Participants developed different strategies to engage with the visualizations based on individual preferences and their game focus. We
observed two primary user strategies. The first strategy was “configure
a fixed set”, which was used by eight participants. These users first
identified their preferred combination of visualizations and used this
fixed set throughout the game clip. They hoped to access personalizedvisualizations to facilitate game analysis without having to actively
control them. Among them, we identified two styles of rationales,
analytic-driven and experience-driven. Analytic-driven users decided
the visualization set based on their game analysis needs. For example,
P6 used Team Panel, Defense, Shot Label all together (Fig. 11a), andexplained that “these help me focus on why they are playing this way,
see if they switch or not, and track the shot percentage”. On the other
hand, P3 used Team Panel at first, and used Shot Label and Shot Chart
together for the rest of the game clip. P3 explained that “I wanted theteam panel first to get an idea of how things were going thus far. Shot
Label and Shot Chart are very useful for seeing current instant threatsby the ball handler. I also appreciated seeing whether a player was
hot compared to their average”. Experience-driven users chose thevisualizations based on engagement. For example, P2 wanted to startwith everything on because “it’s already such a fast-paced sport that
I liked just having them on.” P15 used Shot Label, Defense, and Shot
Chart together because “they are very fun to use and watch” (Fig. 1).
The second strategy was “alter based on context” adopted by eightusers. Participants selected visualizations based on their needs, which
varied based on the current game context. For example, P5 explainedthat “during the clutch time, I’ll look at team panel. When the game is
not as intense, I will look at defense and offense”. Some participants
selected visualizations based on the team they support, “If I’m watchinga game of a team I am rooting for, I focus on the visualization on that
(offense/defense vis when the team is attacking/defending)”. Other fansshowed more flexibility, “I first use Offense and Defense to understandthe team drills (Fig. 11b). When I am familiar with them, I’ll use Shot
Label to evaluate if the shot selection is reasonable (P9)”.
Embedded visualizations help participants derive insights. When
asked about the game insights they obtained with Omnioculars, participants focused on distinct aspects, such as identify player movementand decision making, compare player’s in-game stats, or summarize
game performance. These goals align with the data and tasks we identified in Sec. 3.2.2. For example, P2 used the Offense Trajectory to
focus on player decisions, “when he just passed, I think it’s cool to
see that there’s not really open shots as there’s no big yellow circle
anywhere”. P3 evaluated player’s shooting with the Shot Label, stating“it’s so interesting to see the shot percentage climb up as they drive tothe basket”. Additionally, P1 used the Team Panel to derive a gamesummary, “We are doing really well in the assist and block, wow that’s
unexpected. The turnover is a problem, but if you have more assists,
you’re supposed to have more turnovers. I think that’s acceptable.”.
7.3

Implications for Future Interactive Game Viewing

Flexible embedded visualization design is important to enable acustomized experience. A key finding is that individual needs for dataand desired interaction levels vary. Based on the user’s preferences andbasketball experience, some people felt a visualization was distracting
and overwhelming, while others felt the same visualization helped themsee the data more easily. For example, the trails in the Offense Trajectory visualization were perceived useful to evaluate offense strategy
by some die-hard fans, “The trails are great to see the players’ strategy when the players are closer, like pick-and-roll. It makes it easierto see the tactics (P9)”. However, other fans felt they only focusedon the circles to identify the shot opportunity and did not utilize thetrails, “I just feel like it’s too much information to follow. (P2)”. A
recommended solution is to decouple the individual components ofa visualization, such as separating the trail and circle in the Offense
Trajectory visualization, or the heat map and player panel in the Shot
Chart visualization. As participants focus on a range of goals based on
their preferences, allowing them to select and mix from independentvisual components can best fulfill their individual needs.
Embedded sports visualizations are still underexplored. We iterated
our designs with targeted users, and we believe they are the most suitable in the scope of this project. However, we also believe there is aneed for more refined embedded visualizations targeting specific tasks
and contexts based on our proposed design framework. Additionally,
designing embedded sports visualizations not only requires consideration for objective dimensions such as perceptual effectiveness, but also
for subjective dimensions, such as engagement, which plays a morevital role compared to traditional visualization design.
Expressive interactions allow users to go beyond a passive viewing
experience. We used voice commands as the interaction mechanism
for our study, as we believe, with overwhelming choices, it can bestarticulate the user’s intention [41,44,45], especially in fast-paced sports
watching. Participants had different opinions about the provided interactions: a few participants preferred to have a certain level of default
behaviors, “I wish Shot Chart would turn on automatically for a few
seconds after shots (P2)” and “I feel like it would be a hassle to beturning it on and off and on and off (P3)”, while others enjoyed having
interaction, “I liked the voice activated part of it, very useful to follow
the game and still visualize what I wanted (P11)”. To this end, wesuggest an interaction approach that would allow users to configure
default settings based on their game focus on desired data. Future work
can explore interaction design space in the spectrum from manual toautomatic and train personalized AI models to support semi-automaticinteractions. In terms of functionality, we only allowed simple interactions to toggle visualizations on and off, mainly considering the
fast-paced nature of sports. However, more complicated interactions
could be required (check the average stats of three players). Future
research can explore other interactions to support in-game data analysis.
A systematic evaluation of human perception and attention could
shape embedded visualization design. Participants highlighted theadvantages of our embedded visualization design for game viewing,
including providing immediate feedback, being easily accessible while
watching the game video, and alleviating mental load. For example,
the Shot Label visualization provides “immediate visual feedback after
the pass. I’ve seen the color flip from red to blue to immediately knowthat (P3)”. With Team Panel sitting next to the court, “I can look at
it, for as long as I need to and then switch over to looking at what’s
happening now (P2)”. The visualization can help fans focus on theball, “I could still be watching the ball, but out of the corner of myeye could see whether or not they had options (P12)”. As the firststep towards understanding the use of embedded visualization in sports,
our work aims to provide proof-of-concept results for the design space
and its effectiveness. As a result, we did not systematically evaluate
these potential advantages in our study. We believe, as future work,
obtaining empirical knowledge of these effects can largely facilitate thedevelopment of embedded sports visualization.
Future broadcasting with interactive embedded visualizations. In
current live streams of games, camera angles may change and cut off
the scene. Without access to camera configurations and the complete
video footage, embedding visualizations into videos accurately canbe difficult. However, broadcasters already collect all the required

information, such as tracking data and the full video footage. Therefore,
we believe our design framework and visualizations can be adopted
and extended to augment actual live games by the broadcasters, or byresearchers once the data are made available. Similar to our envisioning
direction, a few personalizations are already offered by the NBA, suchas customized viewing angles with the NBA League Pass [33] andaugmented visual overlays on the live game view by CourtVision [18].
Beyond-the-screen game viewing. Many people prefer watchinggames in person to experience the live atmosphere. Unlike watchinga game through digital devices with visual effects, it is challenging
to overlay extra information on a physical court. Fortunately, withrecent advances in augmented reality (AR), it is now possible to embed
digital graphics into the physical world. We believe that using AR toembed visualizations into an in-person game will enhance the gameviewing experience, and help spectators make sense of the real-timegame data. We also believe our results can be largely generalized to an
AR in-person game viewing scenario, as we designed and implemented
our system in a 3D environment (Unity3D). However, using our designsin AR still differs from a flat screen and we need to consider: the effectof stereoscopy vision and depth perception; the limited field of view
(FoV) in current AR devices; and the ability to freely change viewing
direction so that physical and digital objects can be out-of-view. In the
future, we plan to adapt our current system to AR (by adding out-ofview notifications and by simplifying visualization designs based on
the FoV) and to evaluate its effectiveness.
7.4 Simulating Basketball Game in Virtual Environment
Our study shows that simulation can be a promising solution to easethe development and evaluation of visualizations embedded in physicalcontexts. 94% of all users considered game insights derived from thesimulated game comparable to an actual game (Fig. 10b). Compared to
traditional visualizations presented in digital environments, embedded
and situated visualizations presented in physical contexts are difficult todevelop and evaluate due to the limited accessibility and controllabilityof the physical contexts. Moreover, unlike web-based visualizations,embedded and situated visualizations are inherently challenging to
distribute, reproduce, and benchmark, hindering the research progressin this emerging direction. By compromising a certain degree of fidelity,
simulation enables a highly accessible and controllable physical context
that can be shared, reproduced, and compared universally, significantly
lowering the barrier to designing embedded or situated visualizations.
In recent years, thanks to the proliferation of low-cost VR devices, many
researchers [16, 17, 42, 49, 51, 54] have leveraged VR to simulate realworld environments in user-centered design studies. We envision andadvocate a standard simulation protocol for developing visualizations
in physical contexts, which may eventually facilitate the research ofembedded and situated visualizations.
8 C ONCLUSION & F UTURE W ORKThis study explored the design space of embedded visualization forbasketball in-game data analysis. Through a user-centered design
process, we presented a formative study of basketball fans’ in-gamedata needs and proposed a context-driven design framework of four
design elements — Scenario, Data, Task, and Embedded Vis. We
designed five embedded visualizations for five primary game contexts
and developed Omnioculars, a prototype for personalized game viewing
with embedded visualizations. Our evaluation results suggest that each
embedded visualization provides helpful data on different game aspects,
and fans developed different strategies to engage with visualizations
based on their individual preferences and game focus with Omnioculars.
For future work, we identified the necessity to allow customizablevisual complexity and interaction with embedded sports visualizations
based on individual preferences and knowledge levels. We also hope tobuild upon the current design framework to explore using embedded
visualizations for immersive and situated in-game data analysis.
ACKNOWLEDGMENTS
This work is supported by NSF grants III-2107328 and IIS-1901030. We thankHans H., Joan C., Harvey H., Ahmed S., Nhan H., and Max M. for their time.

R EFERENCES[1] Basketball-reference. https://www.basketball-reference.com/.
[2] Dota 2. https://www.dota2.com/.
[3] Espn - nba. https://www.espn.com/nba/.
[4] League of legends. https://www.leagueoflegends.com/.
[5] Nba shot charts. http://nbashotcharts.com/.
[6] Nba stats. https://www.nba.com/stats/.
[7] The offical site of nba. https://www.nba.com/.
[8] Omnioculars — harry potter wiki. https://harrypotter.fandom.
com/wiki/Omnioculars/.
[9] Reddit - nba. https://www.reddit.com/r/nba/.
[10] Second spectrum. https://www.secondspectrum.com/.
[11] V. Braun and V. Clarke. Reflecting on reflexive thematic analysis. Qualitative research in sport, exercise and health, 11(4):589–597, 2019.
[12] M. Brehmer and T. Munzner. A Multi-level Typology of Abstract Visualization Tasks. IEEE TVCG, 19(12):2376–2385, 2013.
[13] J. T. Browne. Wizard of oz prototyping for machine learning experiences.
In Extended Abstracts of the 2019 CHI Conference on Human Factors in
Computing Systems, pages 1–6, 2019.
[14] P. Budzianowski, T.-H. Wen, B.-H. Tseng, I. Casanueva, S. Ultes, O. Ramadan, and M. Gašić. Multiwoz–a large-scale multi-domain wizardof-oz dataset for task-oriented dialogue modelling. arXiv preprint
arXiv:1810.00278, 2018.
[15] Z. Chen, S. Ye, X. Chu, H. Xia, H. Zhang, H. Qu, and Y. Wu. Augmenting
Sports Videos with Viscommentator. IEEE TVCG, 28(1):824–834, 2021.
[16] Y. Cheng, Y. Yan, X. Yi, Y. Shi, and D. Lindlbauer. SemanticAdapt:
Optimization-based Adaptation of Mixed Reality Layouts Leveraging
Virtual-Physical Semantic Connections. In Proc. of UIST, pages 282–297.
ACM, 2021.
[17] X. Chu, X. Xie, S. Ye, H. Lu, H. Xiao, Z. Yuan, Z. Chen, H. Zhang, andY. Wu. TIVEE: Visual Exploration and Explanation of Badminton Tactics
in Immersive Visualizations. IEEE TVCG, 28(1):118–128, Jan. 2022.
[18] CourtVision.
Clippers courtvision.
https://www.
clipperscourtvision.com/.
[19] N. Dahlbäck, A. Jönsson, and L. Ahrenberg. Wizard of oz studies—why
and how. Knowledge-based systems, 6(4):258–266, 1993.
[20] S. Dow, B. MacIntyre, J. Lee, C. Oezbek, J. D. Bolter, and M. Gandy. Wizard of oz support throughout an iterative design process. IEEE Pervasive
Computing, 4(4):18–26, 2005.
[21] ESPN. Cavaliers vs. warriors - play-by-play - december 25, 2015. https://www.espn.com/nba/playbyplay/_/gameId/400828325.
[22] Y. Fu and J. Stasko. Supporting Data-Driven Basketball Journalism
through Interactive Visualization. In Proc. of CHI, 2022.
[23] K. Goldsberry. Courtvision: New visual and spatial analytics for the nba,
in ‘2012 mit sloan sports analytics conference’. In MIT Sloan Sports
Analytics Conference, 2012.
[24] M. Hayes. Analytics-driven sixers ride the numbers to nba playoffs, 2018.
[25] H. B. Hutchinson, W. E. Mackay, B. Westerlund, B. B. Bederson, A. Druin,
C. Plaisant, M. Beaudouin-Lafon, S. Conversy, H. Evans, H. Hansen,
N. Roussel, and B. Eiderbäck. Technology probes: inspiring design for
and with families. In Proc. of CHI, pages 17–24. ACM, 2003.
[26] Kaggle. Nba play-by-play data 2015-2021. https://www.kaggle.com/
schmadam97/nba-playbyplay-data-20182019.
[27] T. Lin, Y. Yang, J. Beyer, and H. Pfister. SportsXR – Immersive Analytics
in Sports. In 4th Workshop on Immersive Analytics: Envisioning Future
Productivity for Immersive Analytics at ACM CHI, 2020.
[28] A. G. Losada, R. Theron, and A. Benito. BKViz: A Basketball Visual
Analysis Tool. IEEE CG&A, 36(6):58–68, Nov. 2016.
[29] P. Lucey, A. Bialkowski, P. Carr, Y. Yue, and I. Matthews. How to get an
open shot: Analyzing team movement in basketball using tracking data.
In Proceedings of the 8th annual MIT SLOAN sports analytics conference.
Citeseer, 2014.
[30] K. Marriott, F. Schreiber, T. Dwyer, K. Klein, N. H. Riche, T. Itoh,
W. Stuerzlinger, and B. H. Thomas. Immersive analytics, volume 11190.
Springer, 2018.
[31] A. McIntyre, J. Brooks, J. Guttag, and J. Wiens. Recognizing and analyzing ball screen defense in the nba. In Proceedings of the MIT Sloan Sports
Analytics Conference, Boston, MA, USA, pages 11–12, 2016.
[32] T. Munzner. Visualization analysis and design. CRC press, 2014.
[33] NBA. Nba league pass. https://support.watch.nba.com/hc/
en-us/articles/115000585974-NBA-League-Pass.
[34] A. Nistala. Using deep learning to understand patterns of player movement

in basketball. PhD thesis, Massachusetts Institute of Technology, 2018.
[35] H. L. O’Brien and E. G. Toms. The development and evaluation of asurvey to measure user engagement. Journal of the American Society for
Information Science and Technology, 61(1):50–69, 2010.
[36] A. R. Palmeiro, S. van der Kint, L. Vissers, H. Farah, J. C. de Winter,and M. Hagenzieker. Interaction between pedestrians and automatedvehicles: A wizard of oz experiment. Transportation research part F:
traffic psychology and behaviour, 58:1005–1020, 2018.
[37] C. Perin, R. Vuillemot, and J.-D. Fekete. SoccerStories: A Kick-off for
Visual Soccer Analysis. IEEE TVCG, 19(12):2506–2515, Dec. 2013.
[38] C. Perin, R. Vuillemot, C. D. Stolper, J. T. Stasko, J. Wood, and S. Carpendale. State of the Art of Sports Data Visualization. Computer Graphics
Forum, 37(3):663–686, June 2018.
[39] D. Sacha, F. Al-Masoudi, M. Stein, T. Schreck, D. A. Keim, G. Andrienko,
and H. Janetzko. Dynamic Visual Abstraction of Soccer Movement. In
Computer Graphics Forum, volume 36, pages 305–315. Wiley Online
Library, 2017.
[40] T. Seidl, A. Cherukumudi, A. Hartnett, P. Carr, and P. Lucey. Bhostgusters:
Realtime Interactive Play Sketching with Synthesized NBA Defenses.
page 13.
[41] L. Shen, E. Shen, Y. Luo, X. Yang, X. Hu, X. Zhang, Z. Tai, and J. Wang.
Towards natural language interfaces for data visualization: A survey. IEEE
transactions on visualization and computer graphics, 2022.
[42] S. Shimizu and K. Surni. Sports Training System for Visualizing Bird’sEye View from First-Person View. In Proc. of VR, pages 1156–1158.
IEEE, 2019.
[43] SportVU.
Papers with code - nba sportvu dataset.
https://
paperswithcode.com/dataset/nba-sportvu.
[44] A. Srinivasan, N. Nyapathy, B. Lee, S. M. Drucker, and J. Stasko. Collecting and Characterizing Natural Language Utterances for Specifying Data
Visualizations. In Proc of CHI, pages 1–10, 2021.
[45] A. Srinivasan and J. Stasko. How to ask what to say?: Strategies for evaluating natural language interfaces for data visualization. IEEE Computer
Graphics and Applications, 40(4):96–103, 2020.
[46] M. Stein, T. Breitkreutz, J. Häussler, D. Seebacher, C. Niederberger,
T. Schreck, M. Grossniklaus, D. A. Keim, and H. Janetzko. Revealing
the Invisible: Visual Analytics and Explanatory Storytelling for Advanced
Team Sport Analysis. In Proc. of BDVA, pages 1–9. IEEE, 2018.
[47] M. Stein, H. Janetzko, T. Breitkreutz, D. Seebacher, T. Schreck, M. Grossniklaus, I. D. Couzin, and D. A. Keim. Director’s Cut: Analysis andAnnotation of Soccer Matches. IEEE CG&A, 36(5):50–60, 2016.
[48] M. Stein, H. Janetzko, A. Lamprecht, T. Breitkreutz, P. Zimmermann,
B. Goldlucke, T. Schreck, G. Andrienko, M. Grossniklaus, and D. A.
Keim. Bring It to the Pitch: Combining Video and Movement Data to
Enhance Team Sport Analysis. IEEE TVCG, 24(1):13–22, Jan. 2018.
[49] Y. Tanaka, T. Shiokawa, and M. Shiokawa. Scope of Manipulability
Sharing: A Case Study for Sports Training. In Proc. of VR, pages 701–702.
IEEE, 2018.
[50] C. Tian, V. De Silva, M. Caine, and S. Swanson. Use of machine learning
to automate the identification of basketball strategies using whole teamplayer tracking data. Applied Sciences, 10(1):24, 2019.
[51] W.-L. Tsai, T.-Y. Pan, and M.-C. Hu. Feasibility Study on Virtual Reality
Based Basketball Tactic Training. IEEE TVCG, 2020.
[52] Unity Technologies. https://unity.com/, 2022.
[53] W. Willett, Y. Jansen, and P. Dragicevic. Embedded data representations.
IEEE transactions on visualization and computer graphics, 23(1):461–470,
2016.
[54] S. Ye, Z. Chen, X. Chu, Y. Wang, S. Fu, L. Shen, K. Zhou, and Y. Wu.
Shuttlespace: Exploring and analyzing movement trajectory in immersive
visualization. IEEE TVCG, 27(2):860–869, 2020.
[55] Q. Zhi, S. Lin, P. Talkad Sukumar, and R. Metoyer. GameViews: Understanding and Supporting Data-driven Sports Storytelling. In Proceedingsof the 2019 CHI Conference on Human Factors in Computing Systems CHI ’19, pages 1–13, Glasgow, Scotland Uk, 2019. ACM Press.
[56] Q. Zhi and R. Metoyer. GameBot: A Visualization-augmented Chatbot
for Sports Game. In Extended Abstracts of the 2020 CHI Conference on
Human Factors in Computing Systems, pages 1–7, Honolulu HI USA, Apr.
2020. ACM.
[57] S. Zollmann, T. Langlotz, M. Loos, W. H. Lo, and L. Baker. ARSpectator:
Exploring Augmented Reality for Sport Events. In SIGGRAPH Asia 2019Technical Briefs on - SA ’19, pages 75–78, Brisbane, QLD, Australia,
2019. ACM Press.