<figure><img src="image_1.png" /></figure>

HAL
open science

# Sci-Fi Gestures Catalog

Lucas S. Figueiredo, Mariana Pinheiro, Edvar Vilar Neto, Thiago Chaves, Veronica Teichrieb

<figure><img src="image_2.png" /></figure>

## ► To cite this version:

Lucas S. Figueiredo, Mariana Pinheiro, Edvar Vilar Neto, Thiago Chaves, Veronica Teichrieb. Sci-Fi Gestures Catalog. 15th Human-Computer Interaction (INTERACT), Sep 2015, Bamberg, Germany. pp.395-411, 10.1007/978-3-319-22668-2_30. [https://doi.org/10.1007/978-3-319-22668-2_30](https://doi.org/10.1007/978-3-319-22668-2_30). [hal-01599861](https://HAL.inria.fr/hal-01599861)

HAL Id: hal-01599861

https://inria.hal.science/hal-01599861

Submitted on 2 Oct 2017

HAL is a multi-disciplinary open access archive for the deposit and dissemination of scientific research documents, whether they are published or not. The documents may come from teaching and research institutions in France or abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est destinée au dépôt et à la diffusion de documents scientifiques de niveau recherche, publiés ou non, émanant des établissements d'enseignement et de recherche français ou étrangers, des laboratoires publics ou privés.

<figure><img src="image_3.png" /></figure>

Distributed under a Creative Commons Attribution 4.0 International License
# Sci-Fi Gestures Catalog
## Understanding the future of gestural interaction

Lucas S. Figueiredo, Mariana Pinheiro, Edvar Vilar Neto, Thiago Chaves, and
Veronica Teichrieb

Federal University of Pernambuco, Recife PE 50740-560, Brazil
{lsf, mgmp, excvn, tmc2, vt}@cin.ufpe.br

cin.ufpe.br/voxarlabs

**Abstract.** In Science Fiction (Sci-Fi) movies, filmmakers try to anticipate trends and new forms of interaction. Metaphors are created allowing their characters to interact with futuristic devices and environments. These devices and metaphors should be target of research considering they have proven to be useful before. Moreover, the impact of the new interfaces on the audience may indicate their expectations regarding future gesture interactions. Thus, the first goal of this work is to collect and expose a compilation of gestural interactions in Sci-Fi movies, providing a catalog to researchers as resource to future discussions. The second goal is to classify the collected data according to a series of criteria. The catalog is also open to new content contribution, and fellow researchers are invited to provide additional entries of hand gesture scenes from any Sci-Fi title as well as suggestions about new classification criteria and amendments on the already provided content.

**Keywords.** Sci-Fi Movies, Hand Gestures, Gesture Interaction, User Experience.

## 1. Introduction

Movies are a kind of entertainment that has a high impact in the formation of the general public mindset. In particular, Science Fiction (Sci-Fi) movies try to understand and anticipate trends of the future mainly related to new technologies [1]. Some producers and directors often use emerging interaction paradigms, such as gesture-based interaction, to create a plausible vision of the future.

Given that moviemakers have resources and freedom to create characters, stories and scenarios, without common limitations of the real world, Sci-Fi movies can present a particular vision of the future that we are not familiarized with. This can help to make new technologies and interaction models widely known to the general public, contributing to popularize their adoption and to highlight aspects that can be improved by the industry and academy.

On the other hand the Sci-Fi media has potential to reveal or emphasize research trends regarding particular interaction paradigms, devices and interfaces for specific tasks and application domains. Once particular visions of filmmakers are well accept-
ed by the audience, the research on the same topic is boosted by additional motivation, the upcoming technology starts to be part of public's imaginary and an inner curiosity grows with questions like “how would this work in real life?”. Figure 1 shows the influence cycle between the filmmaking industry and the interaction designing community. At first, there is the local influence cycle, which makes explicit that technologists and filmmakers end up influencing themselves. The cross influence cycle is also possible, when one side influences or inspires the other through their products (technologies or films). In addition, as suggested in [1], the flow of influence between technologists and filmmakers can potentially create a collaboration environment in which both sides can create together new interactions and present them in movies.

<figure><img src="image_4.png" /><figcaption>Fig. 1. Filmmaking industry and interaction designing community influence flow</figcaption></figure>

In cases that producers and directors are aided by interaction designers the movie can be seen as a powerful tool for the designer to explore a new concept, envisioning with high visual fidelity how the interface should work. In this case it should be taken into account that there is a certain amount of influence of the entertainment industry over the designer vision. A set of additional concerns like the visual appeal of the scene, the plot development and how the character will look like while using the new interface which may not intersect with goals like realizing the targeted task efficiently or providing a good experience for the user.

Thus, the movies vision of Future User Interfaces (FUIs) often says more about us and the characters than they do about the future [2]. In others words, Sci-Fi movies
end up adapting technologies to the characters characteristics. This point of view can be explored on future user analysis, as for example examining reactions of potential users while watching FUIs being used by the characters and understanding their expectations regarding the same FUIs in the real world. Design problems can be anticipated by gathering feedback from those potential users about the applicability of a particular FUI, understanding if they fit in the specific application domain or, for example, if the input device (e.g. gloves) is accepted on the aimed context.

Despite the industry influence on the FUI design, the scene content is valuable for a range of research purposes. Even if we do not know how the audience responds to a specific FUI, the design process can take it as a source of inspiration. This gives pre-made concepts to designers and developers to create solutions that can be both made with current technologies and be a start point to explore new concepts. Prototyping processes like the Wizard-of-Oz design can benefit from visuals to aid the setup definition including the particular task of designing FUIs which demands knowledge about similar interface concepts that have been designed before. With speculative design in mind, visualizing new concepts of interaction can emerge provocative questions, dialoguing with different ideas of future.

On the other hand, the state of art of hand tracking algorithms and devices is showing promising results [3, 4]. The industry is also presenting accessible devices which provide real time tracking results such as the Microsoft Kinect [5], Leap Motion [6] and Myo [7]. These devices represent a turning point on the design of gestural interfaces due to their low price and the minimum required hardware and setup of the scene broadening the range of gesture interfaces applications and demanding research on the gestural interaction design.

Considering these topics, this paper focuses on hand gestures interaction compiled from Sci-Fi movies. The main goal of this work is to provide an open catalog of these interactions on Sci-Fi scenes, empowering researchers with a tool to gather inspiration for the task of designing new interfaces as well as to perform analysis over the target content. In order to accomplish this we collected and categorized scene parts of hand gestures being performed on different titles. Moreover, our work examines specific aspects related to the interactions and, as for example, the used input and output devices, the performed gesture and the result task executed by the system, among others. Each gesture is tagged considering each one of the chosen criteria. This allowed us to verify supposed lessons from the cinema to designers and researchers as well as identify opportunities to maturing this area. Having this in mind, a web application is presented in order to make public the data from this work for academics as well as the general public. The application is a collaborative system, allowing visitors to contribute by increasing the data.

Supported by the current emerging devices and algorithms for hand tracking as mentioned before, in the first version of the catalog the scope is narrowed to arm and hand gestures. Moreover, currently the input data is gathered exclusively from movies and TV series in order to build a first set of video snippets. However the application may support content from other media such as video game scenes as well. Since the catalog targets the visual analysis of FUIs, it is required a small video for each entry, and so other art forms, such as comic books and novels, are out of scope.

As technical contributions there is the concept and implementation of the catalog as a tool for video analysis. The web application code is available and may be reused
for similar video analysis of different data; the application is based on a Google
Spreadsheet content being adaptable to any type of categories as long as there is a
video for each row entry. Secondly, video snippets containing hand and arm interac-
tions were collected and categorized along 215 scene parts from 24 different titles.
Figure 2 shows some of the scenes in which the users interact with the systems using
gestures. In Ender's Game (Fig. 2a) and Sleep Dealer (Fig. 2c) the protagonists use
their hands to remotely control a spaceship and an aircraft, respectively. The woman
from Black Mirror (Fig. 2b) is playing a virtual violin, while Tony Stark from Iron
Man (Fig. 2d) controls a hologram that represents the city and then he can zoom in
and out any place of interest, or delete things from the model with just a flick. At last,
in order to support additional analysis, all the scene parts were categorized according
to some established criteria as for example the application domain, if there was an
identified interaction pattern in the scene or which was the used input device.

<figure><img src="image_5.png" /><figcaption>Fig. 2. Examples of movies that compose the used dataset in this study</figcaption></figure>

The main scientific contributions are related to the performed analysis and the used
methodology. We found that the major part of the scenes do not fit on previous classi-
fications proposed by the related literature. In some cases it is even difficult to relate
the performed gesture to an accomplished task. This opens space for further investiga-
tions in order to understand the industry demands for interactions which are not yet
established. Moreover, as an incremental contribution, we validate and extend the
number of interaction patterns identified on the analyzed FUIs. As methodological
contribution we present an end-to-end methodology to gather and display the material
in order to perform video analysis on a target subject, including tools and resources.
The result enables researchers to directly relate the video material with its classifica-
tion as well as contribute to increase the gathered data. At last, we considered particu-
lar scenes containing the use of Telekinesis by superheroes which are not directly
used for HCI purposes. However, in several cases, these gestures and their corre-
sponding actions do fit in previous established patterns signalizing that there is a rela-
tion, mainly regarding manipulation tasks on gestural interfaces.
This paper is organized as follows: the next section presents works that also make use of information from Sci-Fi movies to perform analysis on various fields of knowledge. Section 3 discusses the methodology of the work. Section 4 presents the web application developed and section 5 the results obtained. Section 6 presents a discussion about the results. Section 7 concludes the study and section 8 makes a survey of possible future work that the outcome of the study can develop.

## 2. Related Work

Although this work is focused on gesture interactions in Sci-Fi movies, it represents only a subset in the space of human-computer interactions found in these films. Looking at different interactions from Sci-Fi movies, researchers have been dedicated to understand this new set of communications between human and machine. In one of his publications in 2012, "The Past 100 Years of the Future" [8], Aaron Marcus selected scenes from around 20 Sci-Fi movies along the last 100 years that contain communication between human-machine and described them according to the used interaction. His study is organized chronologically around the dates at which these movies were launched to the public. In some cases, he comments on the budget that the moviemakers had available to produce the scenes.

Schmitz et al. show a similar analysis [1], surveying ways of HCI from Sci-Fi movies, categorizing them according to their application domains and relating them to current technologies and prototypes under research. The authors indicate that movies can be a two-way road to HCI, i.e., they can anticipate trends and inspire future concepts of interaction and also collaborate with researchers and visionaries to the conception of scenarios using emerging concepts. The work suggests an influence flow between moviemakers and scientists regarding the use of HCI in movies, where producers and researchers can develop new ideas about future interactions (Figure 1). Thus, technology can inspire movies in the same way that movies interpretation can give feedback about new concepts of devices and interactions. The authors discuss the inspiration that Sci-Fi movies pose to future technologies.

Shedroff and Noesel [9] contribute analyzing around 30 Sci-Fi movies and presenting lessons and insights about the interactions shown on them to HCI. As mentioned, Sci-Fi movies can take advantage to create their own vision because they do not need to limit themselves according to the current technologies. Particularly regarding the gesture-based interactions, the authors point to the existence of seven patterns that can be considered established in HCI. The identified patterns are represented as physical analogies from the real world, i.e., possess a sense of direct manipulation. In addition, this work shows that more complex gestures (that need to get support from GUI) can be difficult to be memorized and advice the use of others input channels to work around possible abstractions, such as speech interactions.

In [10], Christina York focuses on good practices for observation techniques aiming the creation of better interactions. For this purpose her work makes use of Sci-Fi media (e.g., scenes from the StarTrek movie). The analysis includes gestural interactions, as well as touch and voice interfaces.

Asian contributions in Sci-Fi movies are also reviewed by Aaron Marcus [11, 12]. According to the author, studying Chinese, Indian and Japanese productions shows
more creations based on different cultures than properly a copy of western approach-
es. The survey collected in this work initiates a look at the differences and similarities
among different cultures, contributing to extend the perspectives about the future of
user experience design. Moreover it suggests that the metaphors, mental models, in-
teractions and visual appeal on some titles can reveal cross-cultural influences.

Beside papers that study HCI in Sci-Fi films on a generic way, there are other stud-
ies on specific areas of knowledge seeking the correlation between the illustrated
interactions and real life. For example, the work of Lorencik et al. [13] investigates
the mutual influence of science fiction films in the field of Artificial Intelligence and
Robotics. The scenes of six Sci-Fi movies were analyzed in relation to the systems
that controlled the robots and the actions that the robots could perform. The work
concludes that one of the main contributions of Sci-Fi movies to the fields of Robotics
and AI is that they provide sources of inspiration about what can be done and also
increase people interest in the area. The analysis performed by the authors focuses on
the distinction of whether or not there is a real technology or a study on it.

Another example is the work of Boutillette et al. [14] that addressed the influence
of Sci-Fi films in the development of biomedical instrumentation. In the work, more
than 50 Sci-Fi movies were analyzed and, for each one, the scenes of instrumentation
were extracted and analyzed in terms of knowledge and existing technology at that
time. As a result, the films were divided into four categories in chronological order,
and the analysis was made related to how the instruments were used, presented and
whether it could be used at the time of the survey or in the future. At the end, a non-
linear relationship was found between the development of instrumentation and the
ones shown in the Sci-Fi movies.

Although there are many studies regarding the relationship between movies and
HCI, it is not possible yet to say which aspects can be incorporated or not in the de-
sign process. Even if producers and interaction researchers are working together, it is
hard to understand which are the lessons to learn about this partnership. There are still
opportunities to study the features of gesture-based interaction in this knowledge
field.

3. Methodology

The aim of this work is to understand how the motion picture industry has been deal-
ing with the hand gesture interaction paradigm as an insight of what can be built by
tackling the previous problem of collecting and tagging the target data. The process
can be divided into selecting the movies, searching for the gestural interactions within
each of them, and then classifying it. Each of these steps is better explained in the
following subsections.

3.1. Data collection

**Movies selection.** The catalog concept is that it can grow as time passes by being open to anyone who wants to contribute adding new scenes, categories and more. This way, the first set of selected movies aims to represent an initial set that is large enough to provide an analysis basis as well as a test set for the proposed tool. Since it
is impossible for a small group to analyze all existing Sci-Fi movies, 25 volunteers were recruited by email. These volunteers were colleagues and they were asked to inform any movie they remembered that had any kind of gestural interaction using hands and/or arms. Together with the authors, there were a total of ten collaborators who indicated over 200 movies. These indications were checked to be Sci-Fi at IMDb site, which is a popularly known movie database. It was also checked whether they really had the kind of interaction this research was looking for. The remaining titles were restricted by their release date, i.e., the movies from 1995 until the date of this research were prioritized in order to reduce the scope of the work. This choice was made because newer movies are often inspired by the new emerging technologies, such as gesture-based interaction, giving them new insights to go even further. Additionally, there are already studies regarding the older ones (in the future, they will also be added in order to have a more complete database). Finally, the resulting subset contained a total of 24 movies.

**Gesture search procedure.** After the set of movies were defined, two of the authors were assigned to watch all the selected movies, each being responsible for half set, in order to identify the exact time frame the gestures occur during each movie. In order to analyze them all in a plausible time, they were watched up to 8x of the normal speed. For each scene with hand gesture found, the starting and ending time were annotated and filled in the Google Spreadsheet. With this in hand, a script was written (running the FFmpeg library [15]) in order to capture the scenes noted before. This allowed the scenes to be analyzed separately.

**Screening criteria.** As mentioned before, each scene of interest was cut out of the movie for further analysis. Although all kind of hand interaction was collected, i.e., not only human-computer interactions, this allowed us to collect more kinds of metaphors used in gestures. The intention is to preferably allow false-positives rather than favoring the occurrence of false-negatives while segmenting the movies. Later on the scene parts were reviewed and the false-positives were filtered. Moreover, some non HCI interactions were selected, e.g., fighting scenes and Telekinesis interactions in super-heroes movies, as these scenes may serve as inspiration and be useful while considering the audience reaction to it.

**Analysis criteria.** For this analysis, some aspects related to human-computer interaction were chosen. The first aspect to be analyzed was the relation between the action and executed task, i.e., the gesture performed in the scene and the consequent action realized by the system. In a first moment, the person who is analyzing only looks for the type of feedback passed to the user; the types of input and output devices used; and which kind of user the gesture was applied for. During the step of collecting parts containing gestural interaction, other data were becoming common between scenes, so it was decided to increase the amount of items for analysis. After a brief research about their validity, Telekinesis and Established Gestures (Pattern) were added as new items. The remaining categories are:
* Pattern: denotes if the gesture performed by the user and the corresponding action fit in a previous identified pattern (for example, “push to move” or “swipe to dismiss”).

* Feedback: denotes if the system provides any type of feedback and which type it is (e.g., “visual”, “audible” or “haptic”).

* Input Device: if the system presents clearly any required input device in order to track user actions (for example, “haptic gloves”).

* Output Device: denotes the device used for feedback (examples: “hologram”, “HMD”, “CAVE”, etc.).

* Domain: denotes the main application domain (examples include “medical”, “military” and “entertainment”).

The classification of each spotted gesture was done alongside cropping the videos. The researcher responsible for each scene filled a spreadsheet with the characteristics of the interaction according to the criteria discussed above. At completion, they reviewed together all the spreadsheet and discussed about the best classification whenever there was something they both didn't agree with.

## 4. Web Interface

The gathered data was stored in its completion in Google Spreadsheets. By using tabletop.js javascript library it is possible to access the spreadsheet data and use it as database resource to the interface. It contains the selectable filters and the selected video snippets on its top part and the categories table at the bottom showing only the rows relative to the selected entries. The web application is available at http://goo.gl/XSX5fn and its source code is stored on a GitHub repository and can be found at http://goo.gl/IpcAfl.

We understand that to create an expressive compilation of Sci-Fi scene containing gesture interactions is an ambitious goal and our initial data-set represents a small sample of the target goal. Taking it into consideration we introduced the “Add a Scene” button on the top screen of the interface (Figure 3), through which collaborators can send new entries of movies scenes containing human-computer gestural interactions. Each new entry is revised by the authors to check, among other details, if they fit in the Sci-Fi field, and then ported to the online data set.
<figure><img src="image_6.png" /><figcaption>Fig. 3. Web application implemented interface</figcaption></figure>

Moreover, the web application can be useful for different data sets since it is almost entirely based on the data set stored in a Google Spreadsheet. By changing the source data set (by altering the targeted spreadsheet link) the web application adapts itself showing the new content in a similar way. The new columns will be categorized as filters and the videos from each row entry will be gathered and presented in the interface. This way, the interface is replicable and can serve to other focuses as long as they relate to video analysis in some way.
# 5. Results

The following subsections will receive the name and title of each category concerned and will explain the results obtained for each one of them.

## 5.1. Movies Segmentation and Classification

After selecting 24 Sci-Fi movies, a total of 219 different excerpts containing hand gestures were extracted from them. Each segmented scene was classified in categories according to aspects related to human-computer interaction that will be addressed in the next subsections.

In a first moment, the person who is analyzing only looks for the type of feedback passed to the user; which kind of user the gesture was applied for; and the types of input and output devices used. During the step of collecting parts containing gestural interaction, other data were becoming common between scenes, so it was decided to increase the amount of items for analysis. After a brief research about their validity, Telekinesis and Established Gestures were added as new items.

**Patterns.** Some pattern classifications regarding gestures are already defined in literature, for example the Shedroff et al. [9]. They conducted a major study of Sci-Fi movies in the HCI field regarding many aspects. The classification highlights a basic gesture vocabulary commonly used in these types of movies. For this category (Patterns) a benchmarking using this work was conducted and the classification of gestures established by them was applied because they standardize hand gestures used in Sci-Fi movies, which fits in this work. The classification divides the gestures into seven types relating action and result:

* Wave to activate

* Push to move

* Turn to rotate

* Swipe to dismiss

* Point or touch to select

* Extend the hand to shoot

* Pinch and spread to scale

Another pattern classification, is suggested by the work of Wobbrock et al. [16]. But this work is related only to touch surfaces, therefore, initially, it was decided not to include their definitions into the classification. However, during the analysis of the scenes, it was noticed that one of Wobbrock's pattern definition fits with a gesture found in two scenes from two different movies, thus, it was further included in this work. This pattern, named "Knock-Knock to Turn On", denotes gestures with any part of the user's hand or arm touching twice the target system surface.

During the categorization process 37.83% of the analyzed scenes contained some gesture that could be embedded into this classification, namely the vast majority of gestures found were too complex or not established in the classification of Shedroff et al. Regarding the scenes that fit in one of the categories there were a predominance in
the use of gestures "Push To Move" and "Swipe to Dismiss", occurring on 11.41% and 10.04% of the scenes, respectively, as can be seen in the Table 1.

Another observation found is that some gestures, categorized or not, were used for completely different tasks in the same or different movies. When they occurred in the same movie, it was often a superhero film, mainly because the filmmakers are able to use the freedom that superheroes provide of moving things using the power of mind, or Telekinesis (to be discussed in section 6.3). There is a part of the uncategorized gestures related to specific activities that humans are used to perform daily and these gestures were, mostly, exactly as it would be done in the real world turning difficult the task of create a really perfect pattern for them, since all human actions have at least one performance standard that all the people do in the same way. The other part is composed of complex gestures using intense arm and hand movements without a particular pattern. Because of this, is considered a good result that more than 30% of the gestures found match with one of the suggested patterns.

<table><caption>**Table 1.** Frequency of each one of the interaction patterns found in the scenes of this survey.</caption><thead><tr><td>Identified Interaction Pattern</td><td>Percentage of Occurrences</td></tr></thead><tbody><tr><td>Push to move</td><td>11.41%</td></tr><tr><td>Swipe to dismiss</td><td>10.04%</td></tr><tr><td>Point or touch to select</td><td>5.47%</td></tr><tr><td>Pinch and spread to scale</td><td>4.56%</td></tr><tr><td>Turn to rotate</td><td>4.10%</td></tr><tr><td>Knock-knock to turn on</td><td>0.9%</td></tr><tr><td>Wave to activate</td><td>0.9%</td></tr><tr><td>Extend the hand to shoot</td><td>0.45%</td></tr><tr><td>None</td><td>62.17%</td></tr></tbody></table>

**Input and Output Devices.** By considering the input and output devices, this work allows a notion of future technologies. The fact that one particular device appears in movies facilitates the audience acceptance of similar devices and understanding of the interactions, reducing the impact of its reception if at some point it becomes a real product. Table 2 shows the values of the occurrences for input devices.

About the input devices found, it can be seen in Table 2 that the most common case was the absence of identification of any kind of input device (70.81% of the scenes). It reveals the idea that the interaction will be so natural that we will not need any physical device to interact with. In the cases a device was needed, the most used were gloves, optical wireless and haptic (21%), and it shows that filmmakers suppose natural interactions should be used even if a device is needed. Summarizing, both approaches try to predict that future interfaces will be as natural as possible, not needing any device or complex interaction, using only natural gestures that users are used to perform.

The output devices have a wider variety, as shown in Table 3, but coincide with what, today, are considered high-tech display environments, such as HMDs and CAVE environments. There are still devices that are not so high-tech, as for example TVs. As can be seen, the output device often proposed by filmmakers, but not yet
implemented, is the volumetric projection or, as it is commonly known, the hologram projection. Found in 31.96% of the scenes, this device is a strong proof of the influence of films in the research in HCI as several studies on the subject can be found in the literature, for example, [17], [18] and [19]. It is also a supposition of what the filmmakers guess the society expects from new output devices, a new and exciting way to experience virtual contents. The hologram has been studied as a substitute for physical prototypes and as an alternative to 3D displays. However, when the topic is interaction, researchers have different opinions and, until now, the main works converge that it is not possible to interact with holograms directly. In resume, the hologram has been generally used to insinuate a high-tech vision of the future and the interactions have been performed in a straightforward manner, as if a physical object was being manipulated by the user.

<table><caption>**Table 2.** Proportion of different input devices found in the scenes.</caption><thead><tr><td>Input Devices</td><td>Percentage of Occurrences</td></tr></thead><tbody><tr><td>Optical Wireless Gloves</td><td>17.35%</td></tr><tr><td>Cables Plugged Into The Body</td><td>5.47%</td></tr><tr><td>Haptic Glove</td><td>3.65%</td></tr><tr><td>Motion Sensors</td><td>0.91%</td></tr><tr><td>Pen</td><td>0.91%</td></tr><tr><td>Biological Recognition Control</td><td>0.45%</td></tr><tr><td>Mug</td><td>0.45%</td></tr><tr><td>None</td><td>70.81%</td></tr></tbody></table>

<table><caption>**Table 3.** Proportion of different output devices found in the scenes.</caption><thead><tr><td>Output Devices</td><td>Percentage of Occurrences</td></tr></thead><tbody><tr><td>Hologram</td><td>31.96%</td></tr><tr><td>Transparent Screen</td><td>19.63%</td></tr><tr><td>Monitor</td><td>13.24%</td></tr><tr><td>CAVE</td><td>4.56%</td></tr><tr><td>HMD</td><td>4.56%</td></tr><tr><td>Contact Lens</td><td>4.1%</td></tr><tr><td>Digital Table</td><td>0.91%</td></tr><tr><td>Projection</td><td>0.91%</td></tr><tr><td>Analog Pointer</td><td>0.45%</td></tr><tr><td>Robot</td><td>0.45%</td></tr><tr><td>Shapeshifting Display</td><td>0.45%</td></tr><tr><td>Water Tap</td><td>0.45%</td></tr><tr><td>None</td><td>18.33%</td></tr></tbody></table>

**Application Domain.** Within the analyzed data set, the predominant application domains were military, public security, corporative and domotic (Table 4). In addition to these domains, it was also perceived the targeting of HCI applications in the areas of IT, entertainment and medical applications. This guidance allows the estimative of
the main areas that filmmakers are directing their attention regarding FUIs. Besides, it, points to areas that are receiving more attention from the industry innovation departments. The appearance of FUIs in movies acts like a preliminary test of acceptance by potential users. Having people getting used to see the FUIs in the movies makes it easier to begin their insertion in some application domains.

The application domains found in the movies of the database are of high importance for society. As it can be seen in Table 4, the first 4 most covered domains may affect the majority of people. In the public security area, for example, our second most covered domain, there is a growing number of studies that point to the use of the human body for safety increase, e.g., the use of fingertips [20] or eye-tracking [21] to access or activate some systems. It is known that in the home automation area there are many researches about how to make it more effective and easy to use robots or machines so that they only need a few commands to perform a task. Some examples can be seen in [22] who have made an extensive study of the domotic area using gesture-based interaction since the state of art of gestural interfaces for intelligent domotic until socio-technical aspects of gestural interaction. These interactions were commonly seen in Sci-Fi movies, including the ones produced in 1995, what shows us that movies have been influencing research fields with their visionary interfaces and interactions.

<table><caption>**Table 4.** Percentage of gestural interactions for each one of the found application domains.</caption><thead><tr><td>Application Domain</td><td>Percentage of Occurrences</td></tr></thead><tbody><tr><td>Public Security</td><td>17.35%</td></tr><tr><td>Military</td><td>16.43%</td></tr><tr><td>Corporative</td><td>10.95%</td></tr><tr><td>Domotic</td><td>10.5%</td></tr><tr><td>Information Technology</td><td>10.04%</td></tr><tr><td>Entertainment</td><td>9.58%</td></tr><tr><td>Research</td><td>8.21%</td></tr><tr><td>Construction</td><td>1.37%</td></tr><tr><td>Medical</td><td>1.37%</td></tr><tr><td>Robotic</td><td>0.9%</td></tr><tr><td>Aerospace</td><td>0.45%</td></tr><tr><td>Musical</td><td>0.45%</td></tr><tr><td>Not Defined</td><td>12.4%</td></tr></tbody></table>

## 6. Discussion

After creating a movie scenes collection composed of more than two hundred scenes from 24 Sci-Fi movies, these scenes were categorized according to five criteria: established gestures, feedback, input and output type of device and application domain. In the following subsections we discussed about the results found in each of these criteria.
## 6.1. Focused Analysis

As shown in Table 1, the set of gestures considered established were confirmed in our analysis. In total, 81 gestures were categorized into this set, being 11.41% of them classified as “Push to Move” followed by “Swipe to Dismiss” (10.04%). All the others gestures were found in our review. “Wave to Activate”, “Knock-Knock to Turn On” and “Extend the Hand to Shoot” were the least found. On the other hand, we found at least an indication of another pattern in gesture-based interaction. The interaction here named “Knock-Knock to Turn On” is present in two scenes from two movies. This action/task relation may already be found in the real world, for instance in the LG G2 phone, in which the user can touch the screen twice to turn the screen on [23]. This gesture is also described as a tabletop interaction in the Wobbrock’s work [16], which reinforces that it can be considered an established interaction pattern. In spite of that, the majority of gestures gathered in the present paper seem to be way more complex and abstract than those classified into the pattern category. It is not possible to say that there are no precedent occurrences of these gestures either in the academy or in the industry, though. That said, a more in-depth analysis regarding gestures classification must be done to find out whether the filmmaker’s vision is plausible or aligned with current researches at least. Moreover, even if these gestures are unprecedented in the literature, a public survey can be done to get feedbacks from the general public in order to figure out their applicability and their potential to be studied and carried out.

Looking at the environment and purpose of the gestures - the Application Domain - most of them were based on Military and Public Security applications. Corporative environment, Domotic (including home automation and similar), Information Technology, Entertainment and Scientific Research were often shown in this review. Medical, Construction and Robotics come next. Musical applications were also listed. The frequency in which each application domain appears is related to the movie's theme, but the fact they appeared possibly means they have a high potential to be explored within this interaction paradigm.

Regarding the types of input shown in the listed gestures, although Optical Marker Gloves, Body Plugs and Haptic Gloves appear in many scenes, they belong to, respectively, Minority Report, Sleep Dealer and Johnny Mnemonic. A lot of other devices were found, including Motion Sensor, Pen, Staff and a Mug. This variety of devices may occur due to the Sci-Fi nature of presenting their own vision of the future. However, most of the interactions were performed without any kind of support device to promote input. In this case, this absence of input devices may be a trend to seek more natural interaction interfaces. A special attention should be given regarding this type of interaction, since there are a lot of Sci-Fi movies that involve ETs and their hypothetical high advanced technologies, there is also a new whole group of gestures and interactions they use to control spaceships, weapons, among others, that should be observed. For example, Marcus, A. [8] already discusses this topic and compares the way a three fingered alien, acted by Sharlto Copley in District 9, controls a spaceship to the way Tom Cruise, in Minority Report, interacts to the Precog scrubber interface. The author affirms that in District 9 the alien creature controls the spaceship interacting with its interface in an elegant and fast-paced way that is much more beautiful and fluid than the interaction done by Tom Cruise's character.
Hologram has been explored a lot as an output device being found in 31.96% of the scenes and 11/24 of the movies. It is followed by Transparent Screen (19.63% of the scenes) and Monitor (13.24%). Other output devices include Water Tap, HMD, Shapeshifting Display, Robot, Projection, Digital Table, Contact Lens, Analog Pointer and CAVE. Both Hologram and Transparent Screen are technologies currently under research, i.e., we do not find them in the market yet. One reason they appear often in this review is because moviemakers are connected to researchers and developers in order to create a plausible scenario of the near future.

## 6.2. Use of Telekinesis and Superhero Cases

In addition to Sci-Fi, we also watched two superhero and one alien movies. These movies have scenes with a type of Telekinesis interaction supported by hand-based gestures. We defined Telekinesis as "the power to move objects from a distance without physical contact". That means some characters have the ability to move matter using their hands as a kind of pointer. We collected these scenes because we believe these interactions may show us gestures beyond the usual paradigm. If we are able to understand how to make this type of interaction feasible, we can appropriate the movie's vision to start new ideas and researches.

In many cases the hand gestures from these movies have no relation to computer interfaces. In these cases they do not have any input or output devices, since the most gestures are performed in fight scenes and use Telekinesis approach. Nevertheless, some gestures use established pattern of gesture-based interface, for instance, "Extend the Hand to Shoot" and "Push to Move". However, we have the opportunity in these movies to look forward to new forms of interaction that we are not familiarized with. Thus, these scenes may provide us a rich source of inspiration to create innovative solutions.

## 6.3. Methodological Challenges

The large amount of Sci-Fi movies demands a lot of time to extract the gestural interaction scenes. In this survey 24 movies were analyzed, that means watching them, extracting the exact timestamp of the desired scenes and fit the interaction into the defined criteria to add the appropriate tags. So, a challenge is to continue expanding this database of scenes, i.e., to include movies that were published after this research and also those ones that were not included because of the restriction in the scope of the research.

## 7. Conclusion

This paper sought to collect hand gestures from Sci-Fi movies aiming to classify and catalog them in order to make a database to future works. From 24 movies, 219 scenes were captured and classified according to five criteria that take into account movies approaches and interfaces present in each interaction. Moreover, with this
result in hand, a web application was developed in order to make all the results avail-
able to anyone. These results give pre-made concepts to designers and developers and
they are a source of inspiration to create solutions that can be both made with current
technologies and be a starting point to explore new concepts. Besides, the visualiza-
tion of new concepts of interaction can emerge provocative questions, dialoguing with
different ideas of future.

Furthermore, we were able to notice that some previous results from the literature
were confirmed in our study. We also noticed the gesture-based paradigm is maturing
since we could observe previous indications of other emerging patterns as the gesture
patterns suggested by Shedroff and Noessel in [9] and the “knock-knock” pattern
perceived during the development of this study.

Simultaneously, we have found some shortcomings in our analysis. Some criteria
were flawed and we decided to take them out of this review. More work is needed to
correct them. Analyze others data, such as gesture classification, was in our previous
goals and it will be added in future studies. In addition, more research must be con-
ducted, analyzing possible correlations between criteria and extending the previous
results.

8. Future Work

As future works, further analysis about the data collected in this survey should be
made. It is also intended to combine columns of the scene database, and observe
trends and similarities with what is happening in the real world. For example, it could
be done a cross-cultural study, i.e., to watch and observe patterns of similarity or dif-
ferences between gestures across different cultures, namely western/eastern or to no-
tice patterns from each continent.

Besides, the database should be incremented with more movies and more catego-
ries of analysis. In this initial work, there were only seven categories but it is already
in research the addition of more. For example, some simpler categories detailing the
year of launch of the title as well as its production region (in order to perform cross-
cultural analysis). Other more elaborated criteria are also target for future work such
as the feasibility of the gestures or whether the gestures already exist in the real
world. Also, not only the current database should be expanded but it may also include
other sources of material aside of movie scenes, for example, video game scenes,
cartoons, among others.

At last, the portal will demand additional solid crowdsourcing mechanics to en-
courage collaborators to add, modify and remove content in a safe and coherent way.
The use of pre-analysis tools for dynamic chart generations, and the display of selec-
tive table content are also in the scope for future improvements. Finally, a video com-
pilation module is desired to create a single video containing all the filtered scene
parts enabling a more direct analysis of the target content.
References

1. Schmitz, M., Endres, C., Butz, A.: A survey of human-computer interaction design in science fiction movies. In: 2nd International Conference on Intelligent Technologies for Interactive enterTAINment, vol. 7, pp. 1–10, Cancun (2007)

2. Leap Motion: how do fictional uis influence todays motion controls?, http://bit.ly/1sV4NOC

3. Oikonomidis, I., Kyriazis, N., Argyros, A.: A Efficient model-based 3d tracking of hand articulations using kinect. In: BMVC, vol. 1, pp. 3, Dundee (2011)

4. Stenger, B., Mendonça, P. R., Cipolla, R.: Model-based 3D tracking of an articulated hand. In: Computer Vision and Pattern Recognition, vol. 2, pp. 310-315, IEEE Press, Kauai (2001)

5. Microsoft kinect for windows, http://www.microsoft.com/en-us/kinectforwindows

6. Leap Motion mac and pc motion controller for games, design, and more, https://www.leapmotion.com

7. Thalmic Labs Inc. myo - gesture control armband, https://www.thalmic.com/en/myo

8. Marcus, A.: The Past 100 Years of the Future: Human-Computer Interaction in Science-Fiction Movies and Television, http://www.amanda.com/wp-content/uploads/2012/10/AM+A.SciFI+HCI.eBook_.LM10Oct12.pdf

9. Shedroff, N., Noessel, C.: Make it So: Interaction Design Lessons from Science Fiction. Rosenfeld Media (2012)

10. York, C.: To Boldly Go Where No Observer Has Gone Before: Sci-Fi Movies for UX Practice. In: User Experience Magazine, vol. 13, pp. 2 (2013)

11. Marcus, A.: User-experience and science-fiction in chinese, indian, and japanese films. In: Design, User Experience, and Usability. Health, Learning, Playing, Cultural, and Cross-Cultural User Experience. Lecture Notes in Computer Science, vol. 8013, pp. 72-78. Springer, Heidelberg (2013)

12. Marcus, A.: On the Edge: Beyond UX in Western Science Fiction. In: User Experience Magazine, vol. 11, pp. 30 (2013).

13. Lorencik, D., Tarhanicova, M., Sincak, P.: Influence of sci-fi films on artificial intelligence and vice-versa. In: IEEE 11th International Symposium on Applied Machine Intelligence and Informatics (SAMI), pp. 27–31. Herl’any (2013)

14. Boutillette, M., Coveney, C., Kun, S., Menides, L.: The influence of science fiction films on the development of biomedical instrumentation. In: Bioengineering Conference, pp. 143–144, Hartford (1999)

15. Bellard, F., Niedermayer, M., et al. Ffmpeg, http://ffmpeg.org

16. Wobbrock, J. O., Morris, M. R., Wilson, A. D.: User-defined gestures for surface computing. In: SIGCHI Conference on Human Factors in Computing Systems, pp. 1083-1092. ACM, Boston (2009)

17. Opiyo, E., Horváth, I., Rusák, Z.: Investigation of the scope of support provided by holographic displays in conceptual design. In: Product Engineering, pp. 353-365. Springer, Netherlands. (2008)

18. Lee, H.: 3D Holographic Technology and Its Educational Potential. In: TechTrends, vol. 57, pp. 34-39, Springer, US (2013)

19. Bimber, O.: Combining optical holograms with interactive computer graphics. In: ACM SIGGRAPH 2005 Courses, pp. 4. ACM, Los Angeles (2005)
20. Jing, P., Yepeng, G.: Human-computer Interaction using Pointing Gesture based on an Adaptive Virtual Touch Screen. In: International Journal of Signal Processing, Image Pro-cessing, vol. 6, no. 4, pp. 81–92 (2013)

21. Cantoni, V., Galdi, C., Nappi, M., Porta, M., Riccio, D.: GANT: Gaze analysis technique for human identification. In: Pattern Recognition, vol. 48, pp. 1027–1038 (2014)

22. de Carvalho Correia, A. C., de Miranda, L. C., Hornung, H.: Gesture-based interaction in domotic environments: State of the art and HCI framework inspired by the diversity. In: Human-Computer Interaction, pp. 300-317. Springer, Heidelberg (2013)

23. User Guide LG G2. LG, http://goo.gl/Mei0d5