Supplementary Materials
Zona Kostic*

Catherine Dumas†

1 E LICITATION S TUDY (ES)In the ES, we prompted participants to suggest the gesture for 14
different referents (animated sequences illustrating the result ofinteraction with visual representations). Figure 1 shows a small
sample of participants and their suggestions, and Figure 2 shows aclose–up of an individual participant. Our research team used thesevideos to calculate agreement rates between suggested gestures forthe same referent. We asked participants to keep their hands in theview of the camera for better classification into gesture types andthemes.

Sarah Pratt‡Johanna Beyer§

Table 1 lists questions presented to participants in our Elicitation
Study. We ordered tasks to start with simple operations and visualizations (such as hovering and selecting with scatterplot and bar
chart) before prompting for gestures for more complex operations
such as animated transition. Since the participants were not datavisualization experts, some of the referents introduced later in the
study were completely new to them (for example, changes over timeor transitioning to/from donut chart and barchart).
Table 1: The table shows the ES prompt numbers (Q) that correspondto the numbers of referents in Figure 3, and the columns in Figure 4that follows.
QuestionsFigure 1: A group of participants proposes gestures for referents.

Figure 2: A close–up of an individual participant in the Elicitation
Study.
* e–mail: zonakostic@seas.harvard.edu† e–mail: catherine.dumas2@simmons.edu‡ e–mail: sarah.pratt@simmons.edu§ e–mail: jbeyer@seas.harvard.edu

Q1. What gesture would you use if you were interested inhovering and selecting in this scatterplot?
Q1.1 What do you think of using the “pinch” gesturefor the selection?
Q2. What gesture would you use if you were interested inhovering and selecting in this bar chart?
Q2.1 What do you think of using the “pinch” gesture forselection?
Q3. What is the single gesture you would use to mark objectsof interest in this scatterplot?
Q4. What is the gesture you would use to separate this bubble
chart into clusters?
Q5.1 What gesture would you use to explore data changes
over time in this scatterplot?
Q5.2 What gesture would you use to filter data by Region
in this scatterplot?
Q6.1 What gesture would you use to explore data changes
over time in this bar chart?
Q6.2 What gesture would you use to filter data by Genderin this bar chart?
Q5.3 and 6.3 Would it be possible to perform both actions (changes over time and filtering) at the same time using gestures?
Q7. What gesture would you use to zoom intoa particular cluster?
Q8. What gesture would you use to perform zoom & dragoperation with this scatterplot?
Q9. What gesture would you use to perform zoom & dragoperation with this bubble chart?
Q8.1 and 9.1 Would you rather use a single gesture to perform
zoom & drag operation or multiple gestures in sequence?
Q10. What gesture would you use if you were interested in
exploring data outside of the screen area (different timeframe)?
Q11. What gesture would you use if you were interested inrotating the pie chart?
Q12. What gesture would you use if you were interested inrotating the donut chart?
Q13. What gesture would you use to sort this vertical bar
chart in ascending/descending order?
Q14. What gesture would you use to sort this horizontal
bar chart in ascending/descending order?
Q15. What gesture would you use if you were to performthe transition, from bar chart to donut chart?

Figure 3: Elicitation Study referents. Numbers behind each referent correspond to the prompt number (Q) in Table 1
Figure 3 shows the 14 referents of the Elicitation Study. Numbers in parenthesis correspond to the prompt number (Q) in Table 1. We designed referents to explore the following gesturecategories: (1, 2) hovering and selecting; (3) marking items of interest (lasso); (4) reconfiguring data; (5.1, 6.1, 10) exploring changes over time; (5.2, 6.2) filtering; (7, 8, 9) zoom&pan; (11)rotation; (12, 13) sorting; and (14) animated transition. Desired effects were repeated and shown across different visual representations (e.g., selecting a data point on a scatter plot (Q.1) anda bar–chart (Q.2)) to explore if elicited gestures were influenced by the visualization type itself.

1.1 ES Results
Figure 4 shows participants’ suggestions for prompts (Table 1) and referents (Figure 3) presented in our Elicitation Study. The figure shows individual suggestions for each gesture effect,with the following properties: gesture types (in color) associated with each gesture effect; gesture themes grouped for every gesture effect (“grey” shades for the same theme, “n/a” thatmeans no agreement, “no suggestion” means that a participant could not suggest a suitable gesture for the effect presented); handedness such as “unimanual,” “consecutive unimanual”(e.g., two different gestures performed one after the other with one hand), “unimanual+bimanual” (e.g., unimanual gesture followed by bimanual gesture), “bi-manual synchronous,” and“bimanual asynchronous.” The bottom row illustrates gestures for a set of similar concepts demonstrated across various tasks for both “horizontal” and “vertical” themes.

Figure 4: The figure shows individual suggestions for each gesture effect, with gesture types (color) and gesture themes grouped for every gesture effect.

1.2

Post–ES Survey

This section focuses on survey questions following the ES. Table 2
displays the post–ES survey questions, while Table 3 reflects participant feedback or relevant comments regarding whether or not theywould recommend the mid–air gesture system to others. Figure 5
shows the answers to the post–ES survey questions answers to thefollowing questions Q1–Q7.
Table 2: Post–ES survey questions.
Questions
Q1. Of the following, which do you use most often?
- PC (Windows)- Mac (iOS)- Other
Q2. Of the following, which do you use most often- Desktop- Laptop- OtherTable 3: Comments provided to the Q8 post–ES survey question.
Please use this space to reflect or offer feedback?
I could see this type of tool being interesting/useful for data
analysts or policy professionals
Additionally, I think it could make for dynamic and fun teachingtools for educators at both the K–12 and college levels!
Good luck with the study! I’m interested in the results
I think the mid–air gesture concept is too new to us. I think once
systems are designed that facilitate that kind of interaction,
we’d get used to it
It sounds like it might be useful at some point and I don’t doubtthat I might use it some day, but it isn’t on my radar right now.
This study was a good opportunity to think about the types of
mid–air gestures to use.
I didn’t have enough experience with the system to say one
way or the other.
Great learning experience.
Yes, to try to provide better explanations.

Q3. Of the following, which do you use most often whenusing a computer (laptop or desktop)?
- Mouse- Trackpad- Other
Q4. Do you use a tablet (e.g., iPad, Galaxy, Fire, et al.) for
personal and/or professional use?
- Yes
- No
Q5. What make of smartphone do you use?
Two distinct answers provided:
iPhone and Samsung Galaxy.
Q6. Based on your own experience(s), can you imagine
using mid–air gestures in your life and/or work?
- Yes
- No
- Other
Q6.1. If “yes” to the question above, use this space to elaborate
Limited answers provided:
I can imagine using them to interact with digital resourcesof various sorts, particularly since most of my work is doneon the computer.
I’d use it for controlling home systems, especially homemanagement systems, as well as cooking devices,
entertainment, and communication.
Visualizing inside a property
Q7. Based on your own experience(s), would you recommend
mid–air gesture manipulations systems to others?
- Yes
- No
Q8. Please use this space to reflect or offer feedback on yourelicitation study experience or any other relevant comments.
Answers provided in Table 3

Figure 5: Post–ES survey answers to the following questions: Q1,
Q2, Q3, Q4, Q6 and Q7.

2 D ESIGN S PACE
Table 4 shows the design space dimensions for designing gesture
vocabulary. The highlighted rows illustrate the need for disambiguation as well as how gesture types and themes can be accounted for
an appropriate redesign. Methods describe how tasks are carried out
and establish a link between the operations of elicited and designed
gestures.
Table 4: Leveraging gesture types and themes to design gesture
vocabulary.
GESTUREDESIGNEDGESTURETYPEGESTURETHEME

INTERACTIONMETHOD

HOVER

Pointing

Index up

Exploredatapoints

SELECT

Pantomimic

Pinch, closing
index and thumbMark somethingof interest

RECONFIG.

Sem.
Strokes

Bimanualstrokes

Reorder data

ZOOM&DRAG

Pantomimic

Grab&dragNavigate

SWIPE

Sem.
Strokes
Manipulation

Horizontal
theme

Filter features

SLIDE

Sem.
Dynamic

Index slide

Filter time

Z–PAN

Manipulation

Index
hand move

Filter categories

ROTATE

Sem.
Dynamic

Turning the nob

Change view

Figure 7 shows the tutorial that introduced participants to gesture vocabulary through the use of a simple web–based application.
Initially, we asked participants to comprehend the position of theirhands in space or to seek out the “sweet spot.” We instructed participants to place their hands over the Leap sensor so they could see their“virtual hands” as they moved them on the screen. The remaining tutorial sections focused on gesture vocabulary. Participants followed
instructions on how to perform gestures, and moderators offeredsuggestions (without making gestures) if they became confused.

3 U SER S TUDY (SS)All user study participants (20) received temporary Leap Motion
Controller devices, which were mailed to their provided addresses.
In addition, prior to their scheduled user study session, we provided
participants with links to instructions developed by our researchteam for setting up their Leap Motion device. We used Zoom torecord the user study sessions and asked participants to think aloud.
We asked participants to show their hands for each gesture in front of
their cameras for our research team to review later. A demonstrationof the participant in the SS is presented in Figure 6.

Figure 7: Gesture explanations in the tutorial.
We emailed consent forms to all user study participants, which
they read, signed, and sent back prior to their session. Participantsreceived $25 Amazon gift cards for taking part in the study. Figure 8
show one of the SS participants performing gestures on Gapminder
visualization, and Table 5 gives more details about the post–study
survey questions.

Figure 6: Participant in the User Study.

Figure 8: A single participant is performing gestures in our User
Study using Gapminder visualization.

Figure 9: Efficacy rates of gesture recall. The figure provides details on the possibility of parallel inputs and individual participants‘ rates.
3.1

SS Results

Figure 9 shows the recall success rates, which are indicated by acheck mark if the participant correctly remembered the gesture on
the first try and an X if they did not. The majority of gestures,with the exception of the Z–PAN gesture, have high recall rates.
It’s interesting to see how individual recall rates vary (for example,
participant number 7, highlighted in red, compared to others). Participants who struggled to recall the Z–PAN gesture also struggled
to recall other gestures (due to the sensor‘s input range); other participants failed to recall the Z–PAN gesture when asked a second time(Q9), owing to the exhaustion of the Z–PAN gesture in previous
tasks (Q5). Figure 9 also depicts the follow–up questions for bimanual manipulation. Post–User Study (Post–SS) survey questions andanswers are presented with Table 5, Figure 10, Table 6, and Table 7

Table 5: Post–SS survey questions, Q1.–Q4. The following questions are based on participants’ experience with interacting with thenovel data visualization using mid–air gestures in the user study.
Questions

Q1. I found interacting with the data visualization
using mid–air gestures unnecessarily complex in completingthe tasks.
Q2. I found using mid–air gestures very easy to use incompleting the tasks.
Q3. I liked using mid–air gestures to navigate the datavisualization.
Q4. Most people would learn to use this method ofinteraction very quickly.
Figure 10: Post–SS survey answers to questions, Q1–Q4.

Table 6: Post–SS survey questions and answers, Q5., Q6., Q7. The
following questions center on the participants’ interactions with thenovel interaction modality as well as their backgrounds.

Table 7: Post–SS survey questions and answers, Q8.–Q12. The
following questions center on the participants’ interactions with thenovel interaction modality as well as their backgrounds.

Questions and Answers

Questions and Answers

Q5. What do you like about using mid–air gestures tointeract with the data visualizations?
Because it is new technology, it is more fun than what I’m
used to.
I like it because it seems like you can do a lot morevisualizations of data very quickly than by using typicalmouse and keyboard technologies.
You can view the data in a variety of ways in just seconds,
without having to go back to the raw data or configure thesettings.
More on touch with your body and less in your own mind.
I liked the ease of hand motions and the potential uses
for this technology.
It’s a very organic method of navigating data and feels
pretty intuitive(although I think if I was learning to use it for workor school, I would keep a cheat sheet until I remembered
the gestures better!)
The gestures felt very intuitive and matched how you would
move data if it were in front of you on paper.

Q8. Narrate your experience as you would explain the interface
to your friend. Would you recommend it to them for exploringdata visualizations?
Once some of the kinks are worked out, I would definitely
recommend it.
You can work through the data in a much more organic way,
the way you would a physical object.
The interface requires your thoughts to communicate withyour hands in a detailed “conversation”
I used hand gestures to explore data like you often see in
Marvel movies/high tech film. I would recommend it to them.
This interface allows you to look at data visualizations andalter them simply by using your hands and gesturing with them
There are a few different hand gestures that allow you tomanipulate data visualizations so you can change the way
you view the data on your screen
I would recommend it to them because it makes viewing
huge sets of data much easier and it is not very difficult
to learn the gestures.
I felt like I understood the gestures well, and what theywould do, after practicing them each two times.
I would recommend this to someone for exploring datavisualizations. It was very intuitive and I feel thatmanipulating the data in this semi–physical way may help
them to see new patterns in the data.

Q6. What do you dislike about using mid–air gestures tointeract with the data visualizations?
Sometimes the device would lose track of my hands and it
was frustrating to get it to detect them again.
Sometimes Leap Motion would think I was gesturingsomething else and so it would do a different commandthan I wanted it to.
the location needed has to be a certain height forinteraction to work
It’s censor dependent, so if that isn’t functioning then it
highly impacts the entire thing.
My computer wasn’t quite up to the task, and so the gesturesoften didn’t get read properly, making it a bit harderfor me to learn them.
It is difficult to move your hand away without further
manipulating the data.
Q7. How would you improve this system?
I would have the gestures drawn out or
photographed and explained so users can reference themwhile they are interacting with the device.
Something similar to the tutorial but on a document so users
always have it in front of them while using the application.
Tutorial should include the “vocabulary” of signs too
prepare for their application.
Have a “refresh” or reset for the censor so you canreconfigure it if it is malfunctioning.
Such as holding hand up flat to re–calibrate.
Have a “refresh” or reset for the censor so you canreconfigure it if it is malfunctioning.
Such as holding hand up flat to re–calibrate.
I would have the gestures drawn out or photographedand explained so users can reference them while they
are interacting with the device. Something similar to thetutorial but on a document so users always have it
in front of them while using the application.

Q9. Can you think of another way in which you would use
gesture technology?
One way is to use the computer just for basic tools,
such as Internet browsing.
You can use your hands instead the mouse to commandspecific things, such as closing an application window.
It also seems like a great technology to use when editing
photos or videos.
For example, if you’re editing a video, you can use theswiping timeline gesture to move forward or back in thevideo timeframe.
left and right hand gestures could “increase”
the vocabulary.
For planning/architectural/blue print.
You can use your hands instead the mouse to commandspecific things, such as closing an application window.
It also seems like a great technology to use when
editing photos or videos.
you can use the swiping timeline gesture to move
forward or back in the video feed.
Q10. What is your gender?

Q11. What is your age?

Q12. Prior to participating in this study, have you had any
experience with gesture technology? (all answers “no”)

4 E XPERT S TUDY (XS)In an expert study, we evaluated the ergonomics of designed gestures.
We asked expert participants to examine their hand postures andmovements and provide feedback on reducing hand fatigue. Table 8shows the video recordings presented to experts, and Table 9 shows
the questions asked for each of the eight gestures in the vocabulary.
Table 8: XS video recordings presented to experts. Table 9 (Q4.–
Q11.) lists questions asked for each gesture presented below.
Questions
HOVER gesture (index finger up; pointing)

SELECT (pinch) gesturenote that each gesture could be performed with or without the elbow
resting on the table, using the dominant hand (left or right).
Table 9: XS questions we asked for every gesture in the vocabulary.
Questions
Q1. Are you an expert in HCI?
(all answered ‘yes‘)Q2. Do you have experience with hand gesture–based interfaces?
(eight answered ‘yes‘)Q3. Do you have experience with information visualization?
(all answered ‘no‘)
Please, watch the video showing the hand gesture and its effect.
(applies to each gesture presented with Table 8)
Please use the following scales to assign scores for the gesture.
Q4. User posture (the position in which someone holds theirhand). Does the hand position relative to your body give you
any discomfort?
Please rate the level of difficulty. Link it to physical feelings.

SLIDE gesture (two fingers up; move left–right)
Q5. Any additional comments about the posture?

SWIPE gestureQ6. Muscle use (strain or fatigue). Does the position causemuscles to tense up in an uncomfortable manner?

ZOOM&DRAG gesture (“grab” and “move”)
Q7. Any additional comments about muscle use?
Q8. Wrist movements and/or hand position

Z–PAN gesture

Q9. Any additional comments about wrist movement
or hand position?
ROTATE gestureQ10. Are there any other factors that contribute to or detractfrom a user’s well–being? (yes or no, if yes, what are they)

RECONFIGURE gesture (regroup or cluster data)

The questions presented with Table 9 helped us understand theergonomics of gestures in the vocabulary. To help explain the postureof each gesture, we present the video with additional images. Please

Q11. Please rate “other factors” ONLY if any listed with theprevious question. Otherwise, proceed to the next
section.

4.1 XS Results
We discuss the tradeoffs between the memorability and ergonomics
challenges of designed gestures in the manuscript. HOVER and
SELECT gestures, in particular, have the highest memorability rates,
but face minor ergonomics challenges. Specifically, the ergonomic
challenges are related to two attributes: hand posture and wrist movements. Figure 11 depicts the rates of posture and wrist movement
for the HOVER and SELECT gestures. This pattern (high recall andlow ergonomics score, and vice versa) is also observed in gesturesthat are not significantly influenced by legacy bias.

In contrast to HOVER and SELECT, Z–PAN had the lowest recall
score and the highest rates for ergonomics characteristics. Figure 8
shows the Wrist movement ratings for Z–PAN gesture. Figure 12
shows gesture ratings for Z–PAN gesture in the XS study.

Figure 12: Wrist movement rating for the Z–PAN gesture.

Figure 11: XS study ratings for users’ hand posture and wrist movements for the HOVER and SELECT gestures.