Towards an Understanding of Situated AR Visualization for
Basketball Free-Throw Training
Tica Lin

Rishi Singh

Harvard University,
Cambridge, MA, USA
mlin@g.harvard.edu

Yalong Yang

Harvard University,
Harvard University,
Cambridge, MA, USA
Cambridge, MA, USArishibalsingh@g.harvard.edu yalongyang@g.harvard.edu

Carolina Nobre
Harvard University,
Cambridge, MA, USA
cnobre@g.harvard.edu

Johanna Beyer

Maurice A. Smith

Hanspeter Pfster

Harvard University,
Cambridge, MA, USAjbeyer@g.harvard.edu

Harvard University,
Cambridge, MA, USA
mas@g.harvard.edu

Harvard University,
Cambridge, MA, USA
pfster@g.harvard.edu

Figure 1: Visualization design for real-time visual feedback during basketball free-throw shot practice. (a) A co-located 2D
visualization on a desktop display (yellow arrow). (b) First-person view of a situated 3D visualization on an AR HMD.

ABSTRACT
We present an observational study to compare co-located and situated real-time visualizations in basketball free-throw training. Our
goal is to understand the advantages and concerns of applying
immersive visualization to real-world skill-based sports training
and to provide insights for designing AR sports training systems.
We design both a situated 3D visualization on a head-mounted
display and a 2D visualization on a co-located display to provide
immediate visual feedback on a player’s shot performance. Using a
within-subject study design with experienced basketball shooters,
we characterize user goals, report on qualitative training experiences, and compare the quantitative training results. Our results
show that real-time visual feedback helps athletes refne subsequentshots. Shooters in our study achieve greater angle consistency with
Permission to make digital or hard copies of all or part of this work for personal orclassroom use is granted without fee provided that copies are not made or distributedfor proft or commercial advantage and that copies bear this notice and the full citationon the frst page. Copyrights for components of this work owned by others than ACM
must be honored. Abstracting with credit is permitted. To copy otherwise, or republish,
to post on servers or to redistribute to lists, requires prior specifc permission and/or afee. Request permissions from permissions@acm.org.
CHI ’21, May 8–13, 2021, Yokohama, Japan© 2021 Association for Computing Machinery.
ACM ISBN 978-1-4503-8096-6/21/05. . . $15.00
https://doi.org/10.1145/3411764.3445649

our visual feedback. Furthermore, AR visualization promotes anincreased focus on body form in athletes. Finally, we present suggestions for the design of future sports AR studies.

CCS CONCEPTS• Human-centered computing → Mixed / augmented reality;
Visualization application domains.

KEYWORDS
Immersive Analytics, Situated Analytics, SportsXR, Augmented
Reality, Data Visualization
ACM Reference Format:
Tica Lin, Rishi Singh, Yalong Yang, Carolina Nobre, Johanna Beyer, Maurice
A. Smith, and Hanspeter Pfster. 2021. Towards an Understanding of Situated
AR Visualization for Basketball Free-Throw Training. In CHI Conference on
Human Factors in Computing Systems (CHI ’21), May 8–13, 2021, Yokohama,Japan. ACM, New York, NY, USA, 13 pages. https://doi.org/10.1145/3411764.
3445649

1

INTRODUCTION

Learning and perfecting a new skill takes practice. For example,
playing sports or a musical instrument requires developing andCHI ’21, May 8–13, 2021, Yokohama, Japanrefning specifc motor control skills to increase accuracy, consistency, speed, and timing. Athletes, as well as musicians, need tolearn new movement patterns, integrate those new patterns withpreviously learned movements, and learn proper sequencing of
those patterns. Motor skill development is a crucial part of sports
practice, next to other factors such as team strategy, endurance,or scene understanding. The quality of practice greatly infuences
the pace of improvement. Therefore, efcient practice and training
typically require personalized coaching and evaluation. Complex
motor skill learning, in particular, requires frequent and immediate
feedback [64].
In recent years, sports analytics [3] has gained popularity insports management, coaching, and training, by embedding datadriven decision making into the training process. Motion capturesystems can now collect player-specifc data, which can be used invisual analytic systems [48] for detailed performance analysis [65].
However, most of these systems do not provide immediate feedback
during practice, which is crucial for rapid skill improvement [24,64]. To augment sports skill training with immediate feedback,co-located and situated displays are both pragmatic approaches.
HomeCourt [44] and Noah [49] are two commercial applications
that track a basketball player’s movement to give shot feedback.
However, they do not give immediate visual feedback to athletes
but focus on auditory feedback or rely on the presence of coaches.
Additionally, their efectiveness has not been formally evaluated.
With the emergence of head-mounted displays (HMD), augmented reality (AR) can now provide immediate and situated visualfeedback in real-world training settings, potentially further shortening the feedback loop and speeding up motor skill learning. HMDs
support the embedding of virtual data into the real world, haverelatively low hardware costs, and support hands-free user input,
all crucial for sports training. While there are some user studies
evaluating training with situated AR visualization in some sports,
like dancing [12, 34], climbing [30], hand ball [28], and table tennis [4], to the best of our knowledge and based on recent surveys
of AR applications [20, 22, 32, 55], no previous work has evaluatedsituated visualizations for motor skill training in basketball withreal athletes.
In this paper, we examine how athletes use situated AR visualization in realistic sports training and what aspects of augmented
information are useful. We aim to characterize AR in sports training to answer: “What are the unique aspects of using AR for sports
training?”, “What are the advantages and concerns of applying AR toactual sports training?”, and “What are the most important aspects
to take into account when designing sports AR studies?”.
This work presents the frst study to compare co-located versus
situated real-time visualizations in basketball free-throw training.
We collaborated with two National Collegiate Athletic Association(NCAA) basketball teams to design a novel situated visualization
system for basketball free-throw shot training. We discovered that
players seek real-time feedback to achieve a better and more consistent shot arc in their shooting practice. Therefore, we implemented
a co-located 2D and a situated AR visualization with immediate
visual feedback on an athlete’s shooting arc. For the 2D condition,
we designed a 2D visualization on a standard monitor display that is
positioned next to the basketball player during free-throw practice.

Lin et al.

For the AR condition, we designed a situated 3D visualization using
an HMD that is worn during practice (Fig. 1).
We present the results of a quantitative and qualitative user
study with nine experienced basketball players to characterize how
the two visualization conditions afect performance and user goals.
Based on the results of our experiments, we can ofer evidence indirect support of several guidelines for immersive sports visualizations, including that: (1) participants appreciated real-time visualfeedback when gauging their performance and actively used it torefne their subsequent shots, (2) each visualization modality naturally supports diferent user goals, (3) performance in shot angle
consistency improved throughout the study in both, the 2D and
AR conditions, (4) the AR condition was initially less familiar tousers and required more fne-grained adjustment to each individual,and (5) the extra spatial information provided by the AR condition
provided a more realistic feel and resulted in a more holistic senseof participant performance and body position.

2

RELATED WORKAR technology can render interactive 2D and 3D graphics in any
space or surface around the user, which provides possibilities toimprove the existing workfows in many felds. Two survey papers reviewed AR technologies and applications of the last two
decades [22, 32]. Particularly, Kim et al. [32] identifed maintenance,
simulation, and training in industrial, military, and medical felds
as the most popular AR application topics. The overarching assumption behind these applications is that real-time situated visual
guidance or feedback can potentially augment the user’s capabilities and subsequently improve the user’s skills [59]. Sports sharemany characteristics with these popular applications. However,using AR in sports has been less explored [35]. In this paper, we
focus on basketball free-throw training. There is limited researchon this specifc topic, so we review papers on providing immediate
visual support in general sports training with an emphasis on using
AR in basketball free-throw training, when possible.
Broadly speaking, there are three main approaches for providing
immediate visual support in sports training: situated AR, co-locateddisplays, and VR simulations. VR simulations for sports have been
extensively studied [2, 21, 39, 43]. Although some research demonstrates preliminary positive results of using VR for sports training,many challenges remain for achieving ideal fdelity in VR simulations. Specifcally, Covaci et al. [17] investigated basketball freethrow training in a projector-based VR environment and found thatparticipants underestimated distance in VR. In our study, we focus
on situated AR and co-located display approaches, where players
can see accurate visual information in their physical environment
during shot practice.
Situated AR approaches. Basketball and rock climbing are thesports that have received the most attention in the AR literature so
far [55]. For basketball, approaches include mobile AR games [26,52], audio and visual feedback for pickup games [14, 50], wearable
displays on a player’s uniform to display game-related statistics [46],and AR games with a virtual ball [8]. While these systems are
relevant to basketball, none of them focuses on motor skill training.
Kahrs et al. [29] developed an early AR prototype for providing
visual guidance in basketball shooting. Their system visualizes a 3D

Towards an Understanding of Situated AR Visualization for Basketball Free-Throw Training

trajectory to the center of the rim. However, while it provides visual
guidance before a shot, the system does not track and visualize theplayer’s actual shots. The player cannot compare the diference
between the guided trajectories and their real shots, thus making
it difcult to identify what they need to change to improve. Kahrs
et al. also did not formally evaluate their system. Furthermore, thehardware of AR HMDs has improved dramatically since that early
prototype, now ofering higher resolution, more stable tracking,and a larger Field of View (FoV). Compared to their system, oursystem enables real-time basketball tracking using pre-installed
cameras. With real-time basketball tracking, we can render the
trajectory of the real shot. Thus, the user can visually compare
the diference between the guided trajectory and the real shot. We
also systematically evaluated the efectiveness of our system witha state-of-the-art AR HMD (Microsoft HoloLens).
Co-located display approaches. Placing a display next or onto
a sports feld is an easy way to provide immediate visual feedback
to athletes. A few user studies have demonstrated positive resultsin diferent sports. Crowell et al. [18] found that plotting real-time
tibial acceleration data on a 2D screen placed in front of a treadmill
can reduce peak tibial acceleration associated with stress fractures.
Kaplan et al. [31] display pedal pressure overlaid on videos for
cycling, and their participants were generally positive about theirsystem. Chan et al.[13] and Anderson et al. [6] used a similar ideaof presenting the skeleton of a tracked user in front of the user, for
dancing and general motion training, respectively.
Overall, co-located display approaches are well-studied and arelikely to be benefcial for specifc sports. Compared to situatedAR approaches, co-located displays may introduce extra context
switching costs, i.e., players need extra cognitive eforts to map
the visual feedback from the 2D display to the physical space. We
developed a co-located display system with the same basketball
tracking setup as our situated AR system. We placed a 2D monitor
close to the player to show visual guidance and feedback and toallow players to access real-time visual information on the screen
during training. We compared this co-located display setup with
our situated AR system in a controlled user study.
Applications for basketball free-throw training. The Noah
Shooting System [49] and HomeCourt [44] are the two most wellknown applications for basketball free-throw training, but neither
has been quantitatively evaluated. The Noah Shooting System uses
pre-installed cameras and sensors, and HomeCourt uses the camera
on a mobile device to track the ball. Both systems provide real-time
audio feedback about the previous shot, most commonly the captured shooting angle. The Noah Shooting System does not haveany displays on the court, so the player cannot access any visualfeedback during training. In Homecourt, the mobile device needs
to be placed at a distance to capture the full shooting trajectorywith its built-in camera. Therefore, players cannot see the mobile
display showing shot data during training. In practice, both systems
provide either real-time audio feedback or need a second person(e.g., a coach) to access the visual information on a screen.
However, getting audio feedback on a player’s numerical shot
angle does not necessarily give the player intuitive clues on how
to best improve their shot. One of our main hypotheses in thiswork is that real-time visual guidance and feedback may be able to

CHI ’21, May 8–13, 2021, Yokohama, Japan

convey more nuanced details than audio feedback alone and may
give players more intuitive clues for improvement.
Our situated AR and co-located display systems can provide realtime visual guidance and feedback to the player during training. In
this paper, we investigate the potential benefts of such real-timevisual guidance and feedback for basketball free-throw training.
Whether those potential benefts are transferable (i.e., improvements persist after removing the visual guidance and feedback)is beyond the scope of this paper, and studying transferability is
discussed as a future research direction in Sec 7.

3 BASKETBALL FREE-THROW TRAINING
3.1 Background
Unlike regular shots in a basketball game, the free-throw shot isa pure test of skill unfettered by defending players and changing
conditions. Free-throw shots are awarded when a player is fouled
during a shot attempt and allow the player to shoot at the hoop froma distance of 15 feet (i.e., the free-throw line) while all other players
stand aside. Despite its sizable contribution to team ofense, scoring,
winning games [19, 33, 51], and essentially being "free" points, thefree-throw shot in basketball is a difcult skill to improve [9, 38]. For
the last 50 years, the league-average free-throw shot percentages in
the NBA and the NCAA have remained nearly static at a mediocre
75% [7] and 70% [56]. Due to its controlled nature and the lack ofimprovement over several decades, the free-throw shot is a valuable
testing ground for strategies to improve motor skill learning.
A likely contributor to the lack of precise tools and methods
for improving a player’s free-throw shooting is an insufcient understanding of what factors a skilled shooter optimizes to become
more consistent and accurate. Studies found that the movementof a skilled shooter is less variable from shot-to-shot compared tothose less skilled [1, 40, 41, 45]. Tran and Silverberg focused on how
the ball’s release parameters afect the probability of success [60].
However, their approach does not account for how humans’ natural movement inconsistency may be afected when shooting in aparticular way. More recently, Nakano et al. [42] found that moderately skilled shooters grossly minimize the ball release speed during
shooting to reduce motor noise. However, they do not take into
account individual diferences in the shooter’s release positions andheights. Generally, the scientifc understanding of optimal shooting
is not complete enough to provide evidence-based recommendations on shooting form outside of some small basic factors.
Assuming there is an optimal shooting motion for a given player,
improvement in free-throw shooting would require that players
achieve the optimal shooting motion on average, but also that they
become more consistent around that average motion. Variability inshooting form is a shot-by-shot phenomenon. Therefore, it would
be extremely helpful for players to get feedback on each shot’s
deviation in shooting form. There is evidence that even reducing
the frequency of feedback from every trial to every other trial cansignifcantly slow down learning and reduce retention [23, 24, 64].

3.2

Goal and Task Analysis

Based on the above literature review on basketball free-throw shooting, there currently is an incomplete scientifc understanding ofwhat factors constitute an optimal shooting form. However, it has

CHI ’21, May 8–13, 2021, Yokohama, Japan

been shown that real-time feedback on one’s shooting form is benefcial for motor learning [54]. We conducted semi-structured userinterviews with several coaches and players on Harvard Women’sand Men’s Basketball teams to identify the important factors in theirfree-throw shot training. When asked “What factors do you lookat to evaluate your shooting performance during practice?”, coaches
focus on watching players’ “shooting form” while players focus
on their “shot arc.” In practice, most players practice free-throwshooting alone without coaching feedback.
The skilled shooters we interviewed, on average, make 80% of
their free-throws and all aim for shooting at a “perfect arc” or“better arc”. However, only one of them knew the precise angle oftheir target arc. She practices shooting free-throws at a specifc
entry angle (44-46 degrees) with the Noah training system [49].
She considers the real-time audio feedback of angle number andafter-the-fact shot visual analysis from the Noah system useful totrain her shot arc. The other players reported to only subjectively
evaluate their shot performance by visually following the shot arc
to the rim (e.g., identifying that a shot was fat and short). Thenthey make small adjustments to their shooting forms to change balldirection, spin, release height, and speed according to “their feeling.”
All players were highly interested in accessing immediate visualfeedback of their shot arc during shot training.
We extracted some main recurring themes and goals for shottraining from our user interviews: G1 - improving free-throw
performance by using quantitative feedback immediately during practice, G2 - increasing shot consistency, and G3 - improving one’s shot arc. With only auditory or no real-time feedback at all, there is currently a lack of precise goal specifcation andactionable outcome evaluation. For example, there is no intuitive
way for a player to specify what an ideal arc should look like andsubsequently evaluating how much higher the current shot wascompared to the previous shot. Therefore, it is difcult for a player
to make an accurate adjustment. Furthermore, immediate feedback
from coaches is not available most of the time. Thus, the goal of oursystem is to ofer intuitive visual feedback for players in real-timewithout requiring the presence of coaching staf.
Based on our interviews, we have identifed the following tasks
that our system needs to support in free-shot training:
T1: Analysis of the target shooting arc before a shot. Currently, players can only visualize the shooting arc they aim for(i.e., target arc) in their minds before they take a shot. However,seeing the actual target arc spatially situated in the player’s environment immediately before taking a shot would give athletes
highly accurate visual guidance.
Challenge - Spatial visualization of shot arcs: Any shooting arc
in our system needs to be shown as a visual arc, situated in theplayer’s actual physical space to give depth cues and support threedimensional reasoning. Furthermore, the target arc visualizationneeds to be intuitive and unobtrusive to not distract the player
during their shot.
T2: Analysis of one’s shooting arc during and after each individual throw during practice. The most crucial task for basketball players during shot practice is to get detailed spatial information about their shots, directly during their training session.
Challenge - Real-time visualization and immediate visual feedback
directly on the court: Visual feedback should be available during

Lin et al.

practice and after every shot. Specifcally, players should not haveto pause their practice to analyze their last shot, but have direct
hands-free access to visual feedback. Furthermore, the time between the end of the shooter’s motion (ball release) and when thevisual feedback is available should be minimized. Delays in sensory
feedback have been shown to impact motor learning ability [11, 27].
T3: Comparing one’s actual shooting arc to the target shooting arc. To extract actionable insights from shot feedback, players
must be able to compare their latest shot to their set target arc. This
allows players to make informed decisions about how to adjust
their next shot.
Challenge - Visual comparison with minimal user input: Currently,
there is no way for players to get quantitative feedback on how
their shot compares to their set target shot. Comparing both shotarcs should be immediate, easy to understand, and not require any
user input.
T4: Adjusting one’s personal target arc to get consistently
closer to an ideal arc. The ideal shooting arc varies from player
to player, depending on height, physical ftness, and personal preferences. It is crucial for players to be able to set their personal
target arc, and to be able to adjust it over weeks or months as they
improve their form.
Challenge - Adjustable target shot angle: User input for situated andco-located visualizations, especially in sports settings, should be
minimal and not require complicated input devices.

4

SYSTEM DESIGNWe implemented two systems for basketball free-throw training (situated AR and co-located 2D display). Both systems share the same
computer vision tracking component, but difer in their display
mechanisms and visualizations.
Shot detection and tracking. Our system centers on the visualization of a player’s actual shot arc compared to a target shot arc.
To defne the target shot arc, users only need to specify the ball’s
launch angle and release point, since the arc endpoint is alreadygiven by the hoop. To capture a player’s shot arc, we implemented
a shot tracking system that detects the ball’s launch angle, release
point, initial velocity, and endpoint at the rim (Fig. 2).
Our shot tracking system is a real-time markerless 3D motion
capture system consisting of four machine vision cameras (FLIR
Imaging Systems) running at 80 fps. Two synchronized shooter
cameras capture the initial ball motion upon ball release. Separately, two synchronized hoop cameras capture the ball’s motion as

Figure 2: Our 3D motion capture system consists of two
shooter and two hoop cameras, which track the frst 50 ms
upon ball release and the last 250 ms toward the end of theshot, respectively, to calculate launch angle and shot arc.

Towards an Understanding of Situated AR Visualization for Basketball Free-Throw Training

CHI ’21, May 8–13, 2021, Yokohama, Japan

Figure 3: (a) and (b) show the visualization design and interaction loop of the 2D and AR visualizations. Users interactively
specify their target shot arc, shoot the ball, and compare their shot arc to the target arc.
it approaches the hoop and until the shot is over. To maximize thecontrast between the background and the basketball, we use whitewalls and foors in the background of each camera’s view. Simultaneously captured images from the synchronized cameras are sent
frame-by-frame to a desktop machine. We threshold incoming images of the ball by brightness to detect pixels belonging to the ball.
Next, we calculate the ball’s position (in pixels) as the area centroidof the detected pixels within each camera’s view. We then adjust
this position to compensate for lens distortion and triangulate theball’s position in 3D world coordinates using calibrated intrinsic
and extrinsic camera parameters. We repeat these steps for eachconsecutive pair of captured images to get the precise ball position
for every time step.
The release point is the frst available ball position upon releasefrom the shooter’s hand. We use tracking information of the 50 ms
after ball release to calculate the ball’s initial velocity and launchangle. The shot endpoint is extrapolated from the ball’s tracked
positions via the hoop cameras as it approaches the rim. We use
approximately 250 ms of tracking data (20 measurements) before
the ball touches or reaches the rim to extrapolate the ball’s expectedlanding position, or shot endpoint. For our extrapolation, we assume
that the ball reaches the 10 feet height of the rim without touching
the rim, net, or backboard. Prior to our experiments, we performed
an intrinsic calibration for each camera to compensate for distortiondue to lens curvature and sensor misalignment. We also performed
an extrinsic calibration between pairs of cameras to detect theirrelative position and orientation [10, 61, 66]. All image acquisition
and tracking were implemented in Matlab [37].
Our intrinsic calibration error is approximately 0.007 pixels,
which corresponds to approximately 14 micrometers (µm) of error
in the undistorted ball position in a single camera (2D). The only
ground truth available for each shot is that, according to physics,
the ball’s motion must be smooth from frame-to-frame. Thus, tomeasure the accuracy in tracking the ball’s position in 3D, we quantify the high-frequency (frame-to-frame) noise in the ball’s tracked
position by ftting a 2nd order polynomial (constant acceleration)
to the fve position measurements over the 50 ms window after
release for each shot. The standard deviation of the residuals fromeach ft represents the uncertainty in measuring the ball’s position.
From this, we calculate an average 3D tracking precision of ∼70 µm.
After tracking, we immediately transmit our shot tracking data toboth the 2D and AR displays to create the arc visualizations within
100 ms.

Based on the tracking data, we can compute the shot arc visualization by using the projectile motion formula [63], wherex = v 0t cos(θ ) and y = v 0t sin(θ ) − 12 дt 2 . We obtain the full trajectory of x and y positions by considering the efect of gravitational
acceleration (д), ball launch angle θ , and the initial velocity v 0 .
Other factors, such as air resistance and ball spin, are omitted basedon the assumption that they are negligible compared to gravity [15].
Co-located display with 2D visualization. Fig. 3a shows thedesign and interaction methods of our 2D condition. We display aside-view of the shot arc on a typical 27" monitor (Fig. 1a-2), which is
directly connected to the desktop machine performing the tracking.
We encode the spatial trajectory, ball launch angle, and release point
into an arc visualization (T1). However, the co-located display does
not aford the situatedness in the actual physical space. We designed
our 2D system as the baseline condition in our comparative study,
keeping the visualization and user input simple and easy to learn.
We set up the co-located display parallel to the shooter facing
the hoop. This allows us to display the shot arc aligned to the realshooting direction, and minimizes the shooter’s physical movement
to access the visualization (T2). We display both the target andactual shot arcs immediately after our tracking system has detected
the shot outcome (T3). We encode the shot outcome as the color of
the shot arc (i.e., green for a make and blue for a miss). Furthermore,
we show the running score on the top-right of the screen. Users
can use the arrow keys on a keyboard to adjust the angle of thetarget shot arc at any time (T4).
AR HMD with situated 3D visualization. Fig. 3b shows the design and interaction methods of our AR condition. We display
situated 3D shot arcs projected directly into the player’s physicalspace, leading from the shooter towards the hoop (T2, Fig. 1b-2).
We use a HoloLens (1st gen) and transmit our shot tracking data
through WiFi. The hologram anchors the hoop’s position and theposition of the free-throw line and reconstructs the 3D shot arc
based on the shooter’s coordinates relative to the hoop center. The
main visual encoding is similar to the 2D condition and constitutes
a shot arc, launch angle, and release point information. Additionally, we encode lateral spatial information based on the shot arc’s
three-dimensional embodied placement (T1). We display both thetarget and actual shot arcs and show the current score immediately after each shot is made (T3). We use the same color scheme
as in the 2D condition, with careful selection of hue and opacity,
to not distract the shooter. In the AR condition, we do not show
the actual shot’s launch angle in textual form because situated AR
intrinsically supports shape understanding [57]. Furthermore, weCHI ’21, May 8–13, 2021, Yokohama, Japan

Lin et al.

Figure 4: Experimental procedure. Each condition was measured in two consecutive sessions, with each session on a separate
day. The starting condition was counterbalanced between 2D and AR. The study took four hours in total for each participant.
want to avoid potential information overload in AR [47]. Instead,users make a direct comparison of the shape of their target andactual shot arcs and adjust their angle and release point to refne
their target arcs through gestural interaction (T4). Our hologram
program runs at 60 fps to produce a stable visualization and minimize latency [58]. Hologram stability in the HoloLens has been
analyzed previously [25, 62] and Hologram drift under movement
and sudden acceleration was determined to be 6.18mm on average.
Our 2D and AR systems are shown in Fig. 1 and in the supplemental video. Note that the AR shooter in the video looks at the shotarc instead of the hoop while shooting for demonstration purposes.

5

EXPERIMENT OVERVIEW

Our study goal is to compare user training free-throw shooting withreal-time co-located 2D and situated AR visual feedback. We performed a controlled user study to collect qualitative user feedback
and quantitative shot performance data with each condition.

5.1

Experimental Conditions

2D Condition. In the co-located 2D visualization condition (Fig. 3a),participants used the keyboard to adjust their target shot angle andfne-tune their horizontal shooting distance. They then could use
the visualization on the co-located screen to compare their target
and actual shot arc with both the color-coded visual arc and thelaunch angle values on the screen.
AR Condition. In the situated AR visualization condition (Fig. 3b),
participants wore a HoloLens. They used gestures to adjust theirtarget shot angle and ball release point. They then could look at
the situated visualization in AR and compare the 3D arc shape ofthe target shot and actual shot after shooting.

5.2

Experiment Set-Up

The study took place in an indoor laboratory setting with a designated area for free-throw shooting of approximate 6 × 20 feet (120f t 2 ) in size, separated by a safety net from a working and rest area.
The hoop was installed at the standard 10 ft height, 15 ft from thefree-throw line. Participants were instructed to stand behind thefree-throw line to shoot, and an instructor stood under the hoop to
rebound for participants.
The monitor used in the 2D condition is a standard 27-inch
monitor with a 1920 × 1080 resolution, and was placed on a 30-inch
high desk with the keyboard, positioned to the participant’s right,
parallel to the shooting direction. Participants had to turn theirheads about 75◦ to look at the screen. The Hololens used in the AR
condition has a feld of view of 30◦ × 17.5◦ , a 60 Hz refresh rate, andweighs 579д. The instructor facilitated the initial spatial anchoringof the hoop location in the HoloLens.

5.3

Participants

We targeted experienced basketball players eager to improve theirfree-throw shooting. We recruited 10 participants from Harvard
school basketball clubs and university mailing lists.
One participant was unable to complete the study due to the
COVID-19 pandemic and has been excluded from the results. Amongnine participants, three identifed as female and six as male. They
ranged in age from 18-35 years. Seven participants reported having
prior experience with virtual reality without discomfort, but none
had experience with AR HMDs.
Five participants reported playing basketball for more than 10
years, three for 5-10 years, and one for 3-5 years. Seven participants
reported playing at the intermediate or competitive level, and fveof them actively played on school club teams or intramural teams.
In terms of current training behaviors, seven participants playedbasketball more than once a week. Seven participants shot 100-300
shots during typical shooting practice, and the rest shot less than
100 shots. Seven out of nine participants reported knowledge about
their shot percentages, but none knew their shot angles or target
shot angle as they had no way of capturing them. Four participants
reported that they kept their shooting records through oral counting
or on paper.

5.4

Design and Procedures

We followed a within-subject experimental design to account forindividual diferences in free-throw skills. We tested each condition in two consecutive sessions, with 300 shots per session and
600 shots per condition. To compare performance improvement,
we measure users’ baseline performance through 100 shots prior
to each condition. The number of free-throws in each session isbased on our user interviews and a pre-pilot test with experienced
shooters. The experienced shooters in our study have regularly
practiced free-throws, with 100-300 shots in a typical practice, andtens of thousands of free-throws in their career. Therefore, we anticipate that without additional feedback from a coach or our tools
their shot percentage will not improve. Subsequently, to minimize
the physical burden on participants, we decided on measuring thebaseline performance of participants in just 100 shots.
We counterbalanced the starting order of both conditions between participants. The two sessions of the same condition werescheduled on separate days, one or two days apart, to avoid physicalTowards an Understanding of Situated AR Visualization for Basketball Free-Throw Training

fatigue. Session 2 and 3 were scheduled a week apart to mitigate
learning efects between the two conditions.
Here, we demonstrate our study procedure, starting with the
2D condition (Fig. 4). In Session 1, we frst introduced the purposeof the study and asked participants to sign a consent form. We
instructed participants to use the tool to specify their desired angle
and evaluate their shot arc within the visualization to improve freethrow shooting performance. We emphasized that our goal was toevaluate the usefulness of our visualization design instead of theirperformance. Next, participants flled out a pre-study survey on
their background and basketball experience. After doing sufcientwarm-ups, they shot 100 baseline shots in their regular form. We
then introduced them to the 2D visualization tool. Participants weretold to use the tool to adjust the target angle and compare the shotarc as frequently as they wanted during the training. The instructor
was present to answer any questions. Next, participants continued
to shoot 300 shots in the 2D condition. We encouraged them to takebreaks between blocks (every 50 shots) and provided water. At theend of Session 1, we asked them for feedback about their training
experience. In Session 2, they made another 300 shots with the 2D
tool and flled out a post-study survey for the 2D condition.
Session 3 was conducted about one week after Session 2. Due tothe unfamiliarity of the HoloLens in our participants, they started
with a gesture tutorial for the HoloLens. Next, participants shot 100
baseline shots in the AR condition, wearing a HoloLens without
any visual feedback. We then introduced them to the AR visualization and let users get comfortable with it, which took roughly
ten minutes. We told them to use the visualization as frequently as
they wanted. Next, participants shot 300 shots with the AR tool andgave session feedback in the end. In Session 4, they shot another300 shots with the AR tool and flled out a post-study survey for
the AR condition. They were asked to complete a follow-up survey
two weeks after the study was complete. Overall, each session tookroughly one hour to complete, and the whole study took aboutfour hours per participant. We compensated participants with a$40 Amazon gift card.

5.5

Performance Metrics

We collected shot data from our shot tracking system and user inputs in the 2D and AR systems. In Sec. 3.2, we identifed improving
shot arc and consistency with quantitative feedback to improve shot
performance as a main user goal. Therefore, we measure participants’ shot angle, angle consistency, and shot percentage to capture
training performance. Below, we defne the performance metrics
we used to conduct the quantitative analysis.
Initial baseline: To evaluate how user performance is impacted
by the HMD, we compare the angle average of 100 regular baseline shots and 100 baseline shots while wearing the HMD (baseline+HMD).
Shot angle consistency: To compare how user shot angle performance has changed with training, we take the shot angle averageand standard deviation of 100 regular baseline shots, and the last
100 shots in the 2D and AR conditions.
Shot accuracy: To evaluate shot accuracy, we compare the shot
percentage between 100 baseline shots, 100 baseline shots with
HMD, and the last 100 shots in the 2D and AR conditions.

CHI ’21, May 8–13, 2021, Yokohama, Japan

5.6

Qualitative Feedback Collection

We collected subjective responses in session feedback, post-study
surveys for both conditions, and a follow-up survey. To extract
insights from the qualitative feedback, we performed qualitativecoding among three coders. We categorized insights into a set ofcodes derived and agreed upon from an initial open coding by all
three coders and applied the code set to the rest. The complete
survey questions with our coding results are provided in the supplemental material. The following categories were analyzed.
Visualization usefulness: To characterize participants’ training
experiences, we analyze the open-ended user responses from session feedback and post-study surveys. These include their strategies
to use the visualization, the insights derived from the visual feedback, and useful and missing aspects of the visualization to support
their training.
User goals: To quantify the helpfulness of each visualization, we
asked participants to rank their top goals from a list of goals (derived from our interviews). We also asked them to rate each toolaccordingly in a follow-up survey.
Usability: To evaluate the usability of both visualization tools, weanalyzed the user ratings of 5-point Likert-scale questions frompost-study surveys.

6

RESULTS AND DISCUSSION

This section presents the fndings of our quantitative and qualitative
analysis. We frst focus on the observational aspects of our study
and discuss the general user experience before analyzing training
performance. Finally, we highlight unique aspects of AR and discuss
tool usability.

6.1

Real-time visual feedback is useful torefne subsequent shots

None of the participants had experienced free-throw shot training
with real-time visual feedback before. To best capture individualand collective insights, we categorize user responses from the poststudy surveys with representative quotations from participants
and present the qualitative coding results. We found that while theperceived usefulness of 2D and AR visualization varies by individual
preference, real-time visual feedback helps users evaluate theirperformance and acquire self-generated insights.
AR visualization provides useful guidance during shooting and helps generate insights on shooting form. Using our
AR condition, participants analyzed their previous shot arc, release
point, and consistency, and focused on body form such as followthrough during shooting. “[I used] the 3D visualization to evaluate
the shooting arc of the previous shot and using it as a reminder tofollow through.” (P1, AR). ”It was helpful to actually look and be able
to visualize your shot angle and the ball going through the hoop.” (P2,
AR). As a consequence, participants shifted their attention towards
their shooting form and refned subsequent shots. “It forced me tofeel the shot more than look at the rim, it forced me to arc higher.”(P6, AR). “I like that it was able to analyze and help me fne-tune
my release point.” (P8, AR). Participants also reported subjective
improvement with the visual AR feedback. “[I] found it can help toenhance the muscle memory under a better shooting form and further
improve the shot accuracy.” (P5, AR). “I think AR visualization was

CHI ’21, May 8–13, 2021, Yokohama, Japanuseful in guiding my shooting form, especially when I was more fatigued." (P9, AR). Collectively, 6 out of 9 participants commented on
how the AR visualization led to improved form, either by allowing
them to make informed decisions when shooting or by giving themspecifc insights on their form. Six participants commented on thevisual representation of the arc shape as one of the most usefulaspects. Four participants also listed extra shot details such as direction, alignment, and exact release point as useful components.
Three participants said they used the visualization to make explicit
comparisons of aspects of their previous throws.
2D visualization allows direct comparisons with priorthrows on angle performance. In 2D, users focused on analyzing
shot angles after their shot. “I think it is most helpful to analyze theshot angle after a missed shot to see what might have gone wrong.”(P2, 2D). ”I look at the arc after my shot. I think my optimal angle
is 54.5. It helps me with consistency.” (P3, 2D). Participants derived
insights on their performance to refne their shots. ”It helped melearn a lot about my faws - I seem to consistently make the same
mistake of shooting at a smaller angle.” (P3, 2D). “I really liked seeing
the instant feedback on my previous shot. It actually helped me honein on what was my ideal shot angle.” (P6, AR). Participants also
noted diferences between the two conditions. ”The 2D visualization
itself was not as helpful as the 3D one, but it was nice to pin down
exactly what my target angle was.” (P8, 2D). “I felt like 2D gave better
immediate feedback on performance since I could easily compare
my arc vs. my desired arc.” (P6, 2D). Overall, participants foundthe numeric display of the angle number useful (8 out of 9). Fourparticipants reported using the angle information to make afterthe-fact comparisons with prior throws. Three participants alsocommented on the ease of use of the visualization.

6.2

Each visualization modality naturally
supports diferent user goals

In the follow-up survey, we asked participants to rank their top
goals when using the AR and 2D systems among “Improve shootingform”, “Match target angle” and “Improve shot accuracy”. We foundthat the majority of AR users (6) indicated their main goal was toimprove shooting form, while most 2D users (5) reported their main
goal was to improve shot accuracy (Fig. 5). The remaining AR users(3) reported their main goal was to improve shot accuracy, and theremaining 2D users spread evenly between improving form (2) andmatching their target angle (2).
We asked participants to rate how helpful each tool was in reaching their goals from 1 (not helpful at all) to 5 (very helpful). Within
participants who reported that improving form was their top goal,
4 of 6 participants rated the AR system as 4 or 5 for helpfulness
in achieving their goals. These participants reported that it aided
their consistency and muscle memory. The remaining 2 of 6 participants gave ratings of 2, citing the headset’s discomfort as a major
detractor. For the 2D system, both participants aiming to improvetheir form gave ratings of 3.
Within participants who aimed to improve shot accuracy as
their top goal, 3 out of 3 gave the AR system a rating of 3 or 4 on
the helpfulness scale, citing improvements in their release point
or arc height. For the 2D system, 4 out of 5 participants gave thesystem a rating of 4 or 5 on the helpfulness scale. Participants

Lin et al.

Figure 5: Main user goals of the 2D and AR visualizations.

particularly appreciated the ability to make a numeric comparison
between their target and actual shooting angles. At the same time,
participants reported that the 2D system did not help them improvetheir overall form and did not allow them to fne-tune other aspectsof the feedback, such as the release point.
Taken together, we see that our AR and 2D systems difer inwhich goals they promote within users, and in how helpful they
are in supporting those goals. Particularly, the AR system appears to promote a focus on improving shooting form, forwhich it was ranked more helpful than the 2D system. The 2D
system, however, appears to promote a focus on improving
shot accuracy, for which it was ranked more helpful than the AR
system.

6.3

Performance in shot angle consistency
improved throughout the study

We analyzed shot performance of our participants before and after training, including shot angle and consistency. Our hypothesis
was that training with real-time visual feedback would improveparticipants’ shot arc consistency, which can further lead to shot
percentage improvement. We used the Wilcoxon signed-rank test
to check for signifcance in shot angle comparison, angle consistency, and shot percentage improvement. In Fig. 6, we show thechanges in shot angle consistency (∆σ ) and angle (∆deg) before andafter training. We show the shot angle distribution of 100 regularbaseline shots compared to the end of training for the 2D (top row)and AR (bottom row) condition. Due to tracking errors, a portion ofangle data is unavailable for P2, P6, and P7. We interpret the r efectsize using Cohen’s classifcation, which is 0.1, 0.3, and 0.5 and above
for small, moderate, and large efects, respectively [16]. Among theavailable records, 6 out of 7 participants in 2D increased their angle
consistency (∆σ =-0.27, r efect size= 0.83, p=0.03). 7 out of 8 participants in AR increased their angle consistency (∆σ =-0.19, r efectsize=0.83, p=0.02). The results show signifcant improvementof angle consistency in both AR and 2D after training withvisual feedback, which complies with our hypothesis.
Fig. 7 shows a comparison of shot percentages of all participants
between their baseline shots and end of training in 2D and AR. To
understand the impact of shooting with a headset, in Fig. 7a, we
compare the baseline performance change between regular baseline (i.e., shooting without any visual feedback) and baseline+HMD(i.e., shooting without any visual feedback but while wearing aheadset). 7 out of 9 participants had a drop in shot percentage due
to wearing a headset. Fig. 7b and c show the performance change
from start of training to end of training, for the 2D and AR conditions, respectively. 6 out of 9 in 2D and 7 out of 9 in AR increased
their shot percentage. Fig. 7d shows performance diferences at

Towards an Understanding of Situated AR Visualization for Basketball Free-Throw Training

CHI ’21, May 8–13, 2021, Yokohama, Japan

Figure 6: Players’ shot angle distribution plots, comparing their baseline performance to their end of training performance inthe 2D (top row) and AR (bottom row) condition, respectively. Each panel shows the diference in standard deviation (∆σ ) andangle (∆deg). Comparing standard deviation (∆σ ), most participants (7/8 in AR and 6/7 in 2D) improved on angle consistency(p<0.05). Comparing shot angle (∆deg), most 2D users (5/7) changed their angle after training (∆deg> 1◦ ) while most AR users(5/8) shot at the same angle (∆deg≤ 1◦ ).

Figure 7: Players’ shot percentage comparison before and after training. For initial performance (a), 7/9 participants had lower
performance wearing an HMD. Between end of training and baseline performance, most reached a higher shot percentage, 6/9
in 2D (b), and 7/9 in AR (c). Between 2D and AR end performance (d), more perform better in 2D (5) than in AR (3).
the end of training between 2D and AR. Despite initially being
distracted by the headset, 3 out of 9 participants had a higher shot
percentage in AR, while 5 performed better in 2D. However, there
is no signifcant improvement among all participants’ shot
percentage in both conditions. This is likely due to the shortterm exploratory design of our experiment, as training sports skills
requires repetitive practice over a longer time. Another observationis the negative impact on initial shot performance from wearing an
HMD. We discuss the impact of wearing an HMD in more detail in
Sec. 6.4.

6.4

Immersive visualization requires a higher
level of adjustment

The shot data and user feedback indicate that wearing an HMD
requires varying degrees of adjustment. The immersive visualization also requires some adaptation. However, the impact on shot
performance and user perception was mitigated over time.

Here, we investigate the impact of wearing an HMD on the initialshot angle and shot performance. Fig. 8 shows that 6 out of 9 participants had a noticeable change in their shot angle (> 1◦ , p<0.001)
in baseline+HMD, 3 of which (P3, P7, P11) had a massive change(> 3◦ ). Notably, they also had the sharpest decline in shot percentage compared to their baseline performance (−10%, −25%, −19%,
respectively), shown in Fig. 7. This shows that the impact of theAR headset varies by individuals. Nevertheless, over the 2-day
training period, most AR participants improved on shot angle
consistency (7 out of 8) and outperformed their initial shot
percentage (7 out of 9).
From our qualitative coding of the post-study surveys, 6 out of 9participants cited the discomfort and intrusiveness of the headset
as a negative aspect. They also said it took some time to adapt
to the headset. “In the frst two rounds, the headset was distracting,especially when it was slipping of. It took a bit of time to get used toit.” (P1, AR). “At the beginning, I didn’t feel too comfortable with this.
[The] device is on my shooting path, so I have to change my posture

CHI ’21, May 8–13, 2021, Yokohama, Japan

Lin et al.

Figure 8: Shot angle comparison between 100 baseline shots (Baseline) and while wearing an HMD without visual feedback(Baseline+HMD). 6 out of 9 participants had an angle change (∆deg> 1◦ ). P3, P7, and P11 had a larger than 3◦ angle change,
with the most initial shot percentage decrease in Fig. 7.
to accommodate the device.” (P11, AR). Users had to get used to thesituated AR visualization due to its unfamiliarity but appreciatedthe benefts of an AR visualization. “The visualization took a bit to
get used to but was interesting to analyze previous shots or any that
did not feel right on release.” (P8, AR). “Nice not to have to wear theheadset [in 2D], but the visualization in AR is easier to use.” ( P2, 2D).
As the physical presence and associated discomfort of the HoloLens
was the biggest detractor from the AR condition, a more lightweight headset should be used for future sports AR design. Afterinitial adaptation, most participants showed improvementwith AR visual feedback during the study. Conducting a longerterm study is likely to lead to more measurable improvement inshooting performance.

6.5

Immersive visualization resulted in a more
holistic sense of participant performance

As shown in Fig. 5, user goals difer in the AR and 2D conditions.
The AR condition promotes a focus on body form compared to theemphasis on shot accuracy in the 2D condition. Our quantitative
angle data and qualitative feedback both support this observation.
In Fig. 6, we investigate how the shot angle of participantschanged (∆deg) from their baseline shots to the end of their training. We found that most participants in AR (5 out of 8) stayed at
a constant angle (|∆deg| ≤ 1◦ ), while in 2D, most participants (5out of 7) ended up adjusting their shot angle (|∆deg| > 1◦ ). This
refects the focus on the overall form instead of a specifc
angle in the AR condition.
From the user feedback on the visualization’s usefulness, AR
users commented that situated visualization feels more realistic andprovides useful additional information, such as the ball’s release
point. “The visualization felt more realistic with the 3D tool.” (P1, AR).
Four participants cited extra shot details such as shot alignment
and exact release point as useful. ”I can see the release point in the
AR. It also visualized my shooting path, which kept reminding me[of] my form.” (P11, AR). On the other hand, some participants
feel the AR visualization cause overthinking. “Sometimes I thinkit might cause me to overthink my shot, however. Still, it defnitely
makes me more aware of the way in which I am releasing the balland at what angle I am releasing it.” (P2, AR). Participants foundthe 2D visualization to be limited for evaluating their shots more
holistically and were able to extract more shot details, such as
direction, alignment, and release point from the AR visualization.
“I think it is useful to evaluate the shot angle, but it seems not very

useful for the body form.” (P9, 2D). “The visualization itself was not as
helpful as the 3D one. This doesn’t allow for corrections in the release
point like the 3D visualization.” (P8, 2D).
Taken together with the results from Sec. 6.1, situated AR visualization encourages training with a focus on body form andprovides more holistic feedback on a player’s performance.
It gives useful guidance during shooting and supports specifc insight generation. Users also indicated that the highly situated nature of an AR display makes it increasingly important toonly display relevant information to avoid sensory overload. To accomplish this, interface designers should tailor the visual feedback
to the user’s specifc goals.

6.6Lighter headsets will likely increase thereal-world applicability of AR training

We want to understand participants’ acceptance and preferenceon our shot arc visualization approaches, as they are considerednovel for shot training. In Fig 9, we asked for user ratings on bothtool usability and real-world applicability. All nine participants rate
questions Q1 to Q4 positively, reporting the ease of understanding
the visualization and learning to use the tool for both the AR and
2D conditions. The high average ratings for all questions combinedin both AR (µ=4.8) and 2D (µ=4.9) refect the high usability of thevisualization and tools. In Fig 9b, we ask how likely participants
are to user our tools in real training (Q5). Of all 9 participants,
67% rate AR positively (µ=3.4) and 56% rate 2D positively (µ=3.3),
while 33% in AR and 22% in 2D rate it negatively. In Q6, we report participants’ willingness to use the tools after adding missing
features (less intrusive HMD in AR and extra shot details in 2D
as described in Sec. 6.5). 78% in AR (µ=4.1) and 67% in 2D (µ=3.6)
report positively on this. It is worth noting that 22% and 33% ofparticipants in Q5 and Q6, respectively, rate AR with 5s, while nobody rated the 2D condition with 5s. From Q5 to Q6, the responses
in AR also noticeably shift from 33% negative responses to no
negative responses. The results show that the AR condition received more contrasting ratings than the 2D condition, andbecame more favorable than 2D if the missing features, suchas a lighter headset weight, were to be improved.

7

IMPLICATIONS FOR SPORTS XR

We provide a set of design considerations for AR sports skill training tools and study design. These considerations include device

Towards an Understanding of Situated AR Visualization for Basketball Free-Throw Training

CHI ’21, May 8–13, 2021, Yokohama, Japan

have focused on the benefts of real-time visual feedback but havenot investigated whether the gained skills are transferable. Evaluating the post training performance without any feedback fromthe tool is necessary to support long term skill learning [5]. Future studies should also evaluate motor-skill improvements
over a larger time span and adopt adaptive guidance in thesystem design.

8

Figure 9: Tool usability ratings. The percentage number on
the left represents negative responses (1 and 2) in beige, andthe percentage on the right represents positive responses (4and 5) in teal. Neutral responses (3) are not included.
recommendations, study goal settings, and player-related guidelines.
The largest hurdle in the AR condition was the headset itself.
The headset was heavy and bulky, required frequent re-positioning
for some participants, and provided a limited FoV. The limitations
imposed by the headset can be greatly mitigated by using the increasingly lightweight AR devices that are entering the market,
such as Nreal AR glasses [36] that weighs only 88д. When designing future studies and analyzing participants’ performance
results, we need to consider the impact of wearing an AR headset
on participants’ performance. We believe a lightweight device
can mitigate the negative impact of wearing a headset onparticipants’ performance.
One of our major takeaways is the importance of giving athletes
clear instructions on a system’s training goal. Fig. 5 shows thatparticipants in our study had a wide range of goals they targeted.
Some goals (e.g., shot angle vs. shot percentage) might be competing goals, where, in the short run, changing one might negatively
impact the other. This is common in sports skill training, where
athletes often see an initial decrease in overall performance whenchanging a specifc aspect of their technique. While the open-ended
nature of the current study allowed us to better understand how
participants naturally interact with the diferent interfaces, a future study that controls for participant goals will likely beable to draw more conclusive insights in regards to actual
improvements that are facilitated by diferent interfaces.
Finally, the targeted user group’s skill level plays a large rolein the design of the training system. Player skill levels and priorexperiences impact the types of feedback needed. The basketballcoaches in our interviews confrmed that they often tailor their
feedback based on a player’s skill level. A practical consequenceof this is that a single training system design will likely not leadto improvement for all players, and experienced players will likely
not exhibit an improvement over the relatively short time spanaforded in a typical user study. This is refected in the diverse user
response on visualization usefulness in Sec. 6.1 and performance
metrics observed in Sec. 6.3. Future studies would beneft fromtailoring the types and amount of visual feedback to the athlete’s skill level. Furthermore, within the scope of our study, we

CONCLUSIONS AND FUTURE WORKThis paper aims to provide insights for applying co-located 2D
and situated AR visualizations to realistic sports skill training ofbasketball free-throw shooting with immediate visual feedback.
Results suggest that, frst, real-time visual feedback supports users
to refne subsequent shots, and each visualization modality supports
distinct user goals. Second, user shot angle consistency improved
with both 2D and AR visualizations. Third, despite the initial impact
of wearing an AR headset, immersive visualization facilitates a more
holistic sense of performance and body form in athletes. Overall,players favored AR in real-world training if the headset discomfort
were to be improved.
As the frst comprehensive study on applying situated AR visualization to basketball free-throw training, our insights set a starting
point for future research in immersive sports skill training. For
future work, identifying specifc training goals based on sports typeand skill levels, and designing tailored visual feedback with consideration of the guidance hypothesis for motor learning [53], areconsiderable HCI research challenges. We also envision integrating
body tracking measures to the visual feedback. Finally, Sports AR
can provide personalized quantitative visual feedback at low cost,and opens up exciting new opportunities for remote collaboration
between coaches and athletes.

ACKNOWLEDGMENTS
We wish to thank coaches and players on Harvard Women’s andMen’s Basketball teams for their time and expertise. This research is
supported in part by the National Science Foundation (NSF) under
NSF Award Number IIS-1901030, and the Harvard Physical Sciencesand Engineering Accelerator Award.

REFERENCES[1] Salvatore M. Aglioti, Paola Cesari, Michela Romani, and Cosimo Urgesi. 2008.
Action anticipation and motor resonance in elite basketball players. Nature
Neuroscience 11, 9 (2008), 1109–1116.
[2] Anna Akbaş, Wojciech Marszałek, Anna Kamieniarz, Jacek Polechoński, Kajetan J.
Słomka, and Grzegorz Juras. 2019. Application of Virtual Reality in Competitive
Athletes – A Review. Journal of Human Kinetics 69 (Oct. 2019), 5–16. https://doi.org/10.2478/hukin-2019-0023[3] Benjamin Alamar and Dean Oliver. 2013. Sports Analytics: A Guide for Coaches,
Managers, and Other Decision Makers. Columbia University Press. https://doi.
org/10.7312/columbia/9780231162920.001.0001[4] David Altimira, Florian "Floyd" Mueller, Jenny Clarke, Gun Lee, Mark Billinghurst,
and Christoph Bartneck. 2016. Digitally Augmenting Sports: An Opportunity
for Exploring and Understanding Novel Balancing Techniques. In Proceedings ofthe 2016 CHI Conference on Human Factors in Computing Systems. ACM, San Jose
California USA, 1681–1691. https://doi.org/10.1145/2858036.2858277[5] Fraser Anderson and Walter F Bischof. 2013. Learning and performance withgesture guides. In Proceedings of the SIGCHI Conference on Human Factors in
Computing Systems. 1109–1118.
[6] Fraser Anderson, Tovi Grossman, Justin Matejka, and George Fitzmaurice. 2013.
YouMove: enhancing movement training with an augmented reality mirror. In
Proceedings of the 26th annual ACM symposium on User interface software and technology (UIST ’13). Association for Computing Machinery, St. Andrews, Scotland,

CHI ’21, May 8–13, 2021, Yokohama, Japan

United Kingdom, 311–320. https://doi.org/10.1145/2501988.2502045[7] Basketball Reference. 2020. NBA League Averages - Per Game. https://www.
basketball-reference.com/leagues/NBA_stats_per_game.html Accessed: 2020-1231.
[8] Patrick Baudisch, Henning Pohl, Stefanie Reinicke, Emilia Wittmers, Patrick
Lühne, Marius Knaust, Sven Köhler, Patrick Schmidt, and Christian Holz. 2013.
Imaginary reality gaming: ball games without a ball. In Proceedings of the 26th
annual ACM symposium on User interface software and technology - UIST ’13. ACM
Press, St. Andrews, Scotland, United Kingdom, 405–410. https://doi.org/10.1145/
2501988.2502012[9] John Branch. March 3, 2009. For free throws, 50 years of practice is no help. New
York Times (March 3, 2009).
[10] Duane C. Brown. 1966. Decentering Distortion of Lenses. PhotogrammetricEngineering 32, 3 (1966), 444–462.
[11] Samuel N Brudner, Nikhit Kethidi, Damaris Graeupner, Richard B Ivry, andJordan A Taylor. 2016. Delayed feedback during sensorimotor learning selectively
disrupts adaptation but not strategy use. J. Neurophysiol. 115, 3 (March 2016),
1499–1511.
[12] Jacky CP Chan, Howard Leung, Jef KT Tang, and Taku Komura. 2010. A virtual
reality dance training system using motion capture technology. IEEE transactions
on learning technologies 4, 2 (2010), 187–195.
[13] J C P Chan, H Leung, J K T Tang, and T Komura. 2011. A Virtual Reality
Dance Training System Using Motion Capture Technology. IEEE Transactions on
Learning Technologies 4, 2 (April 2011), 187–195. https://doi.org/10.1109/TLT.
2010.27[14] Alan Chatham and Florian’Floyd’ Mueller. 2013. Adding an interactive display toa public basketball hoop can motivate players and foster community. In Proceedings of the 2013 ACM international joint conference on Pervasive and ubiquitous
computing. 667–676.
[15] Hua-Tsung Chen, Ming-Chun Tien, Yi-Wen Chen, Wen-Jiin Tsai, and Suh-Yin
Lee. 2009. Physics-based ball tracking and 3D trajectory reconstruction withapplications to shooting location estimation in basketball video. Journal of Visual
Communication and Image Representation 20, 3 (2009), 204–216.
[16] Jacob Cohen. 2013. Statistical power analysis for the behavioral sciences. Academicpress.
[17] Alexandra Covaci, Anne-Helene Olivier, and Franck Multon. 2015. Visual Perspective and Feedback Guidance for VR Free-Throw Training. IEEE Computer Graphics
and Applications 35, 5 (Sept. 2015), 55–65. https://doi.org/10.1109/MCG.2015.95[18] Harrison Philip Crowell, Clare E. Milner, Joseph Hamill, and Irene S. Davis. 2010.
Reducing impact loading during running with the use of real-time visual feedback.
The Journal of Orthopaedic and Sports Physical Therapy 40, 4 (April 2010), 206–213.
https://doi.org/10.2519/jospt.2010.3166[19] Gabor Csataljay, Peter O’Donoghue, Mike Hughes, and Henriette Dancs. 2009.
Performance indicators that distinguish winning and losing teams in basketball.
International Journal of Performance Analysis in Sport 9, 1 (2009), 60–66.
[20] Arindam Dey, Mark Billinghurst, Robert W. Lindeman, and J. Edward Swan. 2018.
A Systematic Review of 10 Years of Augmented Reality Usability Studies: 2005 to
2014. Frontiers in Robotics and AI 5 (April 2018), 1–28. https://doi.org/10.3389/
frobt.2018.00037[21] Charles Faure, Annabelle Limballe, Benoit Bideau, and Richard Kulpa. 2020.
Virtual reality to assess and train team ball sports performance: A scoping review.
Journal of Sports Sciences 38, 2 (Jan. 2020), 192–205. https://doi.org/10.1080/
02640414.2019.1689807[22] Feng Zhou, Henry Been-Lirn Duh, and Mark Billinghurst. 2008. Trends in
augmented reality tracking, interaction and display: A review of ten years of
ISMAR. In 2008 7th IEEE/ACM International Symposium on Mixed and Augmented
Reality. IEEE, Cambridge, UK, 193–202. https://doi.org/10.1109/ISMAR.2008.
4637362[23] Shinya Fujii, Tea Lulic, and Joyce L Chen. 2016. More feedback is better than less:Learning a novel upper limb joint coordination pattern with augmented auditory
feedback. Frontiers in Neuroscience 10 (2016), 251.
[24] Mark A. Guadagnoli and Robert M. Kohl. 2001. Knowledge of Results for Motor
Learning: Relationship Between Error Estimation and Knowledge of ResultsFrequency. Journal of Motor Behavior 33, 2 (2001), 217–224.
[25] Anne-Laure Guinet, Guillaume Bouyer, Samir Otmane, and Eric Desailly. 2019.
Reliability of the head tracking measured by Microsoft Hololens during different walking conditions. Computer Methods in Biomechanics and Biomedical
Engineering 22, sup1 (2019), S169–S171.
[26] Andreas Hebbel-Seeger. 2015. Physical Skills and Digital Gaming: The Relationship between Basketball and an Augmented Reality Adaption. In Gamifcation in
Education and Business, Torsten Reiners and Lincoln C. Wood (Eds.). Springer
International Publishing, Cham, 291–313. https://doi.org/10.1007/978-3-31910208-5_15[27] Takuya Honda, Masaya Hirashima, and Daichi Nozaki. 2012. Adaptation to visualfeedback delay infuences visuomotor learning. PLoS One 7, 5 (May 2012), e37900.
[28] Mads Mueller Jensen, Majken K. Rasmussen, Florian "Floyd" Mueller, and Kaj
Grønbæk. 2015. Keepin’ it Real: Challenges when Designing Sports-Training
Games. In Proceedings of the 33rd Annual ACM Conference on Human Factors in

Lin et al.

Computing Systems. ACM, Seoul Republic of Korea, 2003–2012. https://doi.org/
10.1145/2702123.2702243[29] Lüder A. Kahrs, Jörg Raczkowsky, Jürgen Manner, Andreas Fischer, and Heinz
Wörn. 2006. Supporting Free Throw Situations of Basketball Players with Augmented Reality. International Journal of Computer Science in Sport 5 (2006),
72–75.
[30] Raine Kajastila, Leo Holsti, and Perttu Hämäläinen. 2016. The augmented climbing wall: High-exertion proximity interaction on a wall-sized interactive surface.
In Proceedings of the 2016 CHI conference on human factors in computing systems.
758–769.
[31] Oral Kaplan, Goshiro Yamamoto, Yasuhide Yoshitake, Takafumi Taketomi,
Christian Sandor, and Hirokazu Kato. 2016. In-situ visualization of pedaling forces on cycling training videos. In 2016 IEEE International Conference
on Systems, Man, and Cybernetics (SMC). IEEE, Budapest, Hungary, 994–999.
https://doi.org/10.1109/SMC.2016.7844371[32] Kangsoo Kim, Mark Billinghurst, Gerd Bruder, Henry Been-Lirn Duh, andGregory F. Welch. 2018. Revisiting Trends in Augmented Reality Research:
A Review of the 2nd Decade of ISMAR (2008–2017). IEEE Transactions on
Visualization and Computer Graphics 24, 11 (Nov. 2018), 2947–2962. https://doi.org/10.1109/TVCG.2018.2868591[33] Bill Kozar, Ross E. Vaughn, Russel H. Whitfeld, and Bobby Dye. 1994. Importanceof free-throws at various stages of basketball games. Perceptual and Motor Skills
1578, 1 (1994), 243–248.
[34] Matthew Kyan, Guoyu Sun, Haiyan Li, Ling Zhong, Paisarn Muneesawang, Nan
Dong, Bruce Elder, and Ling Guan. 2015. An approach to ballet dance training
through ms kinect and visualization in a cave virtual reality environment. ACM
Transactions on Intelligent Systems and Technology (TIST) 6, 2 (2015), 1–37.
[35] Tica Lin, Yalong Yang, Johanna Beyer, and Hanspeter Pfster. 2020. SportsXR – Immersive Analytics in Sports. In 4th Workshop on Immersive Analytics: Envisioning
Future Productivity for Immersive Analytics at ACM CHI. arXiv:2004.08010 [cs.HC][36] Nreal Ltd. 2020. Nreal Light AR Glasses. https://www.nreal.ai/ Accessed: 202009-01.
[37] MATLAB. 2016. version 2016a. The MathWorks Inc., Natick, Massachusetts.
[38] Ian McMahon. November 22, 2017. Hacking the free throw: the science behindthe most practiced shot in sports. The Guardian (November 22, 2017).
[39] Helen C. Miles, Serban R. Pop, Simon J. Watt, Gavin P. Lawrence, and Nigel W.
John. 2012. A review of virtual environments for training in ball sports. Computers& Graphics 36, 6 (Oct. 2012), 714–726. https://doi.org/10.1016/j.cag.2012.04.007[40] David R. Mullineaux and Timothy L. Uhl. 2010. Coordination-variability andkinematics of misses versus swishes of basketball free throws. Journal of Sports
Sciences 28, 9 (2010), 1017–1024.
[41] David R. Mullineaux and Timothy L. Uhl. 2010. Kinematics of the free throw inbasketball. American Journal of Physics 49, 6 (2010), 1017–1024.
[42] Nobuyasu Nakano, Yuki Inaba, Senshi Fukashiro, and Shinsuke Yoshioka. 2020.
Basketball players minimize the efect of motor noise by using near-minimum
release speed in free-throw shooting. Human Movement Science 70 (2020), 102583.
[43] David L. Neumann, Robyn L. Moftt, Patrick R. Thomas, Kylie Loveday, David P.
Watling, Chantal L. Lombard, Simona Antonova, and Michael A. Tremeer. 2018. A
systematic review of the application of interactive virtual reality to sport. Virtual
Reality 22, 3 (Sept. 2018), 183–198. https://doi.org/10.1007/s10055-017-0320-5[44] NEX Team Inc. 2020. HomeCourt. https://www.homecourt.ai/ Accessed: 202009-01.
[45] Hiroki Okubo and Mont Hubbard. 2015. Kinematics of arm joint motions inbasketball shooting. Procedia Engineering 112 (2015), 443–448.
[46] Mitchell Page and Andrew Vande Moere. 2007. Evaluating a Wearable Display
Jersey for Augmenting Team Sports Awareness. In Pervasive Computing, Anthony
LaMarca, Marc Langheinrich, and Khai N. Truong (Eds.). Vol. 4480. Springer
Berlin Heidelberg, Berlin, Heidelberg, 91–108. https://doi.org/10.1007/978-3540-72037-9_6 Series Title: Lecture Notes in Computer Science.
[47] Rui Miguel Pascoal and Sérgio Luís Guerreiro. 2017. Information Overload inAugmented Reality: The Outdoor Sports Environments. In Information andCommunication Overload in the Digital Age. IGI Global, 271–301.
[48] C. Perin, R. Vuillemot, C. D. Stolper, J. T. Stasko, J. Wood, and S. Carpendale. 2018.
State of the Art of Sports Data Visualization. Computer Graphics Forum 37, 3(June 2018), 663–686. https://doi.org/10.1111/cgf.13447[49] Pillar Vision Inc. 2020. Noah Shooting System. https://www.noahbasketball.com/
Accessed: 2020-09-01.
[50] Timothy P. Ryan and Jonathan Duckworth. 2017. 2K-Reality: An Acoustic Sports
Entertainment Augmentation for Pickup Basketball Play Spaces. In Proceedingsof the 12th International Audio Mostly Conference on Augmented and Participatory
Sound and Music Experiences. ACM, London United Kingdom, 1–8. https://doi.
org/10.1145/3123514.3123529[51] Jaime Sampaio and Manuel Janeira. 2003. Importance of free-throw performancein game outcome during the fnal series of basketball play-ofs. International
Journal of Applied Sports Sciences 15, 2 (2003), 9–16.
[52] Markus Santoso. 2018. Markerless augmented reality technology for real-spacebasketball simulation. In 2018 IEEE International Conference on Consumer Electronics (ICCE). IEEE, Las Vegas, NV, 1–3. https://doi.org/10.1109/ICCE.2018.8326335

Towards an Understanding of Situated AR Visualization for Basketball Free-Throw Training[53] Richard A Schmidt. 1991. Frequent augmented feedback can degrade learning:
Evidence and interpretations. In Tutorials in motor neuroscience. Springer, 59–75.
[54] Leandro de Carvalho da Silva, Carla Ferro Pereira-Monfredini, and Luis Augusto
Teixeira. 2017. Improved children’s motor learning of the basketball free shooting
pattern by associating subjective error estimation and extrinsic feedback. Journalof sports sciences 35, 18 (2017), 1825–1830.
[55] Pooya Soltani and Antoine H.P. Morice. 2020. Augmented reality tools for
sports education and training. Computers & Education 155 (Oct. 2020), 103923.
https://doi.org/10.1016/j.compedu.2020.103923[56] Sports Reference. 2020. NCAA Seasons - Game Averages. https://www.sportsreference.com/cbb/seasons/game-averages.html Accessed: 2020-12-31.
[57] Mark St. John, Michael B Cowen, Harvey S Smallman, and Heather M Oonk. 2001.
The Use of 2D and 3D Displays for Shape-Understanding versus Relative-Position
Tasks. Hum. Factors 43, 1 (March 2001), 79–98.
Hologram stability - Mixed Reality.
https:[58] Thetuvix. [n.d.].
//docs.microsoft.com/en-us/windows/mixed-reality/hologram-stability#:~:text=HoloLenshashardwarethatadjusts,location,andhologramsfeelstable.
[59] Bruce H. Thomas, Gregory F. Welch, Pierre Dragicevic, Niklas Elmqvist, Pourang
Irani, Yvonne Jansen, Dieter Schmalstieg, Aurélien Tabard, Neven A. M. ElSayed, Ross T. Smith, and Wesley Willett. 2018. Situated Analytics. In Immersive Analytics, Kim Marriott, Falk Schreiber, Tim Dwyer, Karsten Klein,
Nathalie Henry Riche, Takayuki Itoh, Wolfgang Stuerzlinger, and Bruce H.

CHI ’21, May 8–13, 2021, Yokohama, Japan

Thomas (Eds.). Vol. 11190. Springer International Publishing, Cham, 185–220.
https://doi.org/10.1007/978-3-030-01388-2_7[60] Chau M. Tran and Larry M. Silverberg. 2008. Optimal release conditions for thefree throw in men’s basketball. Journal of Sports Sciences 26, 11 (2008), 1147–1155.
[61] Roger Y. Tsai. 1987. A Versatile Camera Calibration Technique for High-Accuracy
3D Machine Vision Metrology Using Of-the-Shelf TV Cameras and Lenses. IEEE
Journal of Robotics and Automation RA-3, 4 (1987), 323–344.
[62] Reid Vassallo, Adam Rankin, Elvis CS Chen, and Terry M Peters. 2017. Hologram stability evaluation for Microsoft HoloLens. In Medical Imaging 2017: Image
Perception, Observer Performance, and Technology Assessment, Vol. 10136. International Society for Optics and Photonics, 1013614.
[63] James S. Walker. 2017. Physics. Pearson.
[64] Gabriele Wulf, Charles H. Shea, and Sabine Matschiner. 1997. Frequent Feedback
Enhances Complex Motor Skill Learning. Journal of Motor Behavior 30, 2 (1997),
180–192.
[65] Shuainan Ye, Zhutian Chen, Xiangtong Chu, Yifan Wang, Siwei Fu, Lejun Shen,
Kun Zhou, and Yingcai Wu. 2020. Shuttlespace: Exploring and analyzing movement trajectory in immersive visualization. IEEE Transactions on Visualization
and Computer Graphics (2020).
[66] Zhengyou Zhang. 2000. A Flexible New Technique for Camera Calibration. IEEE
Transactions on Pattern Analysis and Machine Intelligence 22, 11 (2000), 1330–1334.