# Instruction Pictograms for Interactive Entertainment

Kei Kobayashi, Junichi Hoshino

► To cite this version:

Kei Kobayashi, Junichi Hoshino. Instruction Pictograms for Interactive Entertainment. 20th International Conference on Entertainment Computing (ICEC), Nov 2021, Coimbra, Portugal. pp.209-222, 10.1007/978-3-030-89394-1_16. hal-04144361

HAL Id: hal-04144361

https://inria.hal.science/hal-04144361

Submitted on 28 Jun 2023

HAL is a multi-disciplinary open access
archive for the deposit and dissemination of sci-
entific research documents, whether they are pub-
lished or not. The documents may come from
teaching and research institutions in France or
abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est
destinée au dépôt et à la diffusion de documents
scientifiques de niveau recherche, publiés ou non,
émanant des établissements d'enseignement et de
recherche français ou étrangers, des laboratoires
publics ou privés.

<figure><img src="image_1.png" /></figure>
<figure><img src="image_2.png" /></figure>

ifip

International Federation for Information Processing

This document is the original author manuscript of a paper submitted to an IFIP
conference proceedings or other IFIP publication by Springer Nature. As such, there
may be some differences in the official published version of the paper. Such
differences, if any, are usually due to reformatting during preparation for publication or
minor corrections made by the author(s) during final proofreading of the publication
manuscript.
# Instruction Pictograms for Interactive Entertainment

Kei Kobayashi¹ and Junichi Hoshino¹

¹ University of Tsukuba, Ibaraki, Japan
kobayashi.kei@entcomp.esys.tsukuba.ac.jp, jhoshino@esys.tsukuba.ac.jp

**Abstract.** With advancements in detection devices and cognitive technology, experience-based displays (exhibits) reacting to user behaviors are gaining popularity. These systems use sensors to recognize users' upper arm and forearm movements to switch images or the environment in images; however, communicating exactly what user action is required for the display to respond is difficult. To have viewers (users) behave in the manner intended by experience-based displays incorporated in media art and museum exhibits, this study proposes methods for instructing behaviors through pictograms. This study produced still (frozen) and moving pictograms after elaborating on forms and compositions that specify desired behaviors and compared the correct answer rates and average time spent by users in their twenties to seventies to achieve the desired behaviors. For the assessment experiment targeting users in their twenties, the correct answer rates varied among different still images, whereas all moving images successfully navigated >88.5% of users to correctly behave. Pictograms were successful in making middle-aged and older users understand the desired behaviors; however, they were frequently required more time to absorb the instructions. This study introduces use cases incorporating these results into a real-world experience-based system.

**Keywords:** Pictogram, Interaction, Museum, Augmented Reality.

## 1 Introduction

With advancements in detection devices and cognitive technologies, additional media arts and museum exhibits have been introducing experience-based displays, incorporating behavioral recognition using sensors and images [1][2][3]. In most systems, sensors recognize the movements of users' hands, upper arms, and forearms and switch images or accordingly change the environment in the images. In an exhibition environment, however, it is difficult to exactly inform users what movement is required.

Audio or text instructions using natural languages as a means to instruct users what to do would require the preparation of instructions in multiple languages to consider users from different language backgrounds. Moreover, depending on the type of behaviors, a word may mean several different movements, necessitating explanations in detail to accurately communicate behaviors intended by the system designer. Such
descriptions, however, would take a long time to understand and require a large dis-
play area to accommodate lengthy texts, which may compromise the expressions of
the exhibit. Furthermore, the use of live action video or animation as a means to in-
struct behaviors would require the production of such materials along with the exhibit
content's world views and atmospheres.

A method extensively used in communicating required behaviors in public spaces, which users in different age groups or with different nationalities can understand, is a pictogram. Pictograms refer to visual signals designed to provide information or call attention using simple graphics and color combinations. They are extensively used in use globally as evacuation guidance, as traffic signs, in restrooms, or at Olympic games, and their application in various fields has been examined in previous studies [4][5][6]. However, no report on the use case or effectiveness of pictograms for experience-based exhibits requiring gesture inputs has been published to date.

This study proposes methods to communicate behaviors through pictograms to en-
able users to input correct behaviors in experience-based exhibits such as media arts
and exhibits in museums. First, it classifies typical experiential behaviors used in
media artworks, art museums, and museums, into 1) an input method based on the
status of hands; 2) an input method using upper arms and forearms; 3) an input meth-
od by a way of pressing buttons and scrolling; and 4) an input method using external
objects.

Next, the study creates two types of behavior pictograms for each category: a mov-
ing pictogram and a still pictogram; it compares the correct answer rates and time
spent to correctly completing behaviors among users in their twenties up to their sev-
enties. Through the comparison, it determines whether behaviors intended for experi-
ence-based displays are correctly communicated to users and identifies differences
between moving and still images, in addition to areas of considerations required for
particular age groups.

2 Previous Studies

## 2.1 Communication Tool

In addition to their roles as a sign to inform users about functions in facilities, or in-
struct them or warn them of possible risks, pictograms are used as a communication
tool. Isotype [7], devised by Neurath in the 1920s, uses simplified shapes to explain
statistical data to the working class. Isotype has significantly impacted signages and
infographics, which followed the development of isotype.

Moreover, pictograms are used in computers. In an Internet-based trial, Pere et al. [8] developed an instant message service based on pictograms to bridge the social and digital gap of people with cognitive impairment. In an exploratory evaluation, the study had people with cognitive impairment use the service and observed improvement in the users' communication ability. Mori et al. [9] examined social network services featuring pictograms used by and among children with different nationalities to identify communication patterns. Pictograms were frequently used as syntax or
pictorially or as a storytelling tool, whereas their uses may vary depending on users'
personalities and different cultural backgrounds.

## 2.2 Moving Pictograms

Regarding the addition of movements to pictograms, actograms [10] developed by
Japan BoSign Association aim to raise awareness of desirable behaviors at the time of
disasters using “moving” pictogram signs. The designers propose dividing time after
the occurrence of an earthquake into four phases and provide information specifically
required for each phase. Based on the assumption that highly abstract word classes
such as verbs and adjectives would be unexplainable by still images, Ohno et al. [11]
generated moving pictograms for certain verbs. They compared still images and mov-
ing pictograms based on user questionnaires to analyze comprehensibility and pro-
pose designing guidelines.

Compared to previous studies, this study focuses on designs and display methods
that do not distract users' attention away from the contents of the main display (art-
works).

## 2.3 Behavior Communication Methods

To communicate the intended behaviors, this study used the method proposed by
Ogawa et al. [12] to navigate users to use hands and fingers to input behaviors in in-
terfaces in a public space as icons and texts on a display. Generally, when users are
aware of reactions triggered by their movements, they become fully engaged in the
interaction, whereas they are hesitant to read text-based instructions. Okuma et al.
[13] used text instructions displayed on handheld PCs for experience-based displays
as part of the guide system of a science museum. However, the research results
demonstrated that users' attention was distracted from exhibits to the screen of the
PCs. In communicating how to experience exhibits, measures with minimal interfer-
ence to exhibit would be ideal.

# 3 Designing Instruction Pictograms

## 3.1 Types of Experiential Behaviors Required for Exhibits

As outlined in Section 1, previous works and research are characterized using arm-
and hand-based input methods based on the assumption that they are typically used
for media artworks and exhibits in art museums and museums [1][2][3]. In these sys-
tems, the challenge is the difficulty in communicating desirable movements to users
in an exhibit environment.

A combination of input methods using arms and hands may include the following:

(1) Input method based on the status of hands;

(2) Input method using upper arms and forearms;

(3) Input method using buttons and scrolling; and
(4) Input method using external objects.

"(1) Input method based on the status of hands" assumes interfaces that recognize the statuses of hands such as open palm or clenched fists. Among "(2) upper arm and forearm behaviors," behaviors having certain meanings can include the following:

(a) Behaviors to change the environment: fan, blow, and throw.

(b) Behaviors to respond to an environmental change: shiver, hold up an umbrella, and block the sun.

This study focused on (a) where content images change in response to users' behaviors in experience-based displays. Upper arms and forearms are used not only in systems recognizing behaviors through RGB-D cameras but also for systems using head-mounted displays, where users control attached controllers to experience contents [14]. In the "input methods using upper arms and forearms," individual differences become an issue because certain users make big movements, whereas others make minimal movements to the same instruction. Therefore, precise instructions, including detailed movements, must be communicated.

Because "(3) Input method using buttons and scrolling" has been traditionally used in interfaces, it may be incorporated into the actual space as concept users are familiar with. For example, they are used in systems where users operate images by holding a hand above an icon or above a seek bar and moving the hand sideways [15].

"(4) Input method using external objects" refers to the relocation or modification of an object as a behavioral input. For example, a work featuring the process of wrapping an object with a wrapping cloth [16] and a work featuring the process of holding an object and placing it on a designated place [15] use this method. Actual use cases of the method are introduced in Section 5 in more detail.

## 3.2 Requirements of Instruction Pictograms

The purpose of this study is to explain how to experience exhibits through pictograms to permanently enable users to use experience-based displays correctly. Therefore, a pictogram must have the following features:

* Make users understand intended experiential behaviors concisely;

* Be easily comprehensible by most of the users;

* Minimize its interference with the exhibited works' concepts or expressions;

* Be easy to produce.

First, experiential behaviors required for each exhibit must be communicated within a very short time period, typically several seconds. Because pictograms will be installed in public spaces, they should ideally be easily comprehensible by users belonging to various age groups and nationalities. In media artworks, minimizing interference to expressions intended by the artist is important. Furthermore, because there are myriad different behaviors, sample pictograms not introduced in this study may be required for an actual display. To this end, this study intends to enable the easy production of new pictograms by proposing a production method.
3.3 Design Requirements and Production Methods

To achieve the aforementioned requirements, designing instruction pictograms may
require to consider the following aspects.

* Forms exclusively communicating certain behaviors

Pictograms introduced in this study use simple, flat figures exclusively focusing on communicating behaviors and removing other elements to keep users' attention on the exhibits and to make it easy to produce them as well.

Alternative design ideas may include a 3D depiction of human figures in the picto-
grams. Compared to a clear, dichromatic design (Fig. 1, Left), however, the shade
would make certain areas of the figures dimmer (Fig. 1, Center), reducing visibility
because of the brightness of the figure approaching that of the base brightness. Fur-
thermore, a design incorporating elements such as body shape, sex, and hairstyle (Fig.
1, Right) necessitates a selection process based on the exhibit contents, reducing pro-
duction convenience and universality. Therefore, pictograms in this study have no
attributes other than behaviors.

<figure><img src="image_3.png" /><figcaption>Fig. 1. Pictogram Design</figcaption></figure>

* Composition concisely communicating behaviors

Behaviors used in exhibits may apply to situations other than the ones introduced in
this study. Therefore, composition types are limited in the study to ensure the easy
production of new pictograms. To indicate the statuses of upper arms, forearms, and
hands, the human figures in pictograms either face sideways or front.

For “Input method using buttons and scrolling” and “Input method using external objects,” however, compositions featuring the human figure in the center obstruct the visibility of icons, seek bar, image to be swiped, and object, as well as compromises clarity. Therefore, pictograms must keep the target icon and seek bar in the center of the composition.

* Production methods

For each composition, this study prepares basic components—the head, trunk,
upper arms, forearms, and hands—to enable the production of pictograms simply by
combining the components. If and when adjustment is necessary for a behavior, the
study modifies these components. The use of 10 × 10 square grid lines as the base for
pictograms limits object shape options, reduces the time spent on determining shapes,
and provides consistency to the design (Fig. 2).

Following the PIC Symbol design concept [17], this study exclusively uses black
and white to retain visibility, minimize colors, and maximize the brightness
difference. Based on the background of pictograms, the colors of the base and figure
may be reversed. Certain colors are assigned specific roles in chromatic symbols used
<figure><img src="image_4.png" /><figcaption>Fig. 2. Grid Lines</figcaption></figure>

in safety signs, such as red for prohibition, blue for instruction, and yellow for warning [18]. To avoid providing unnecessary information or suggestions to users, pictograms in this study avoid the use of chromatic colors. If and when it is desirable to use chromatic colors because of the features of the contents, it is recommendable to use non-safety colors that are in sharp contrast to white.

Moving pictograms in this study use animation comprising three frames—the initial, middle, and final images. To complete a cycle of behavior when moving images are shown in a loop, both the 1st and 2nd frames are displayed for 0.25 s, whereas the 3rd frame is displayed for 0.5 s. Based on the moving pictograms, this study generates still pictograms by adding arrows to demonstrate the trajectories of behaviors and the relocation of objects to the frames.

## 3.4 Production Case of Pictograms

Figs. 3 and 4 show the results of nine movements incorporated into pictograms based on discussions in Sections 3.1–3.3. Section 5.1 introduces use cases of some of these pictograms. Because each exhibit requires different behaviors, pictograms for behaviors not included in this section are separately fabricated as per the design requirements and production method introduced in Section 3.3.

### (1) Input method based on the status of hands

Assuming an interface that recognizes statuses of hands to prompt reactions in an exhibited image, this study fabricated two pictograms instructing users to “Face front and spread hands” and “Face front and clench fists.” These types of behaviors can be applied to pictograms describing other hand statuses by replacing the statuses of hands, as shown below on the left. Use cases may include the assignment of different hand statuses with different operations such as “Face front and spread hands” to “re-play movie” and “Face front and clench fists” to “pause movie.”

### (2) Input method using upper arms and forearms

For behaviors involving upper arms and forearms, this study selected behaviors in which upper arms and forearms are positioned in different places. Some interfaces using such behaviors only target the final status of a single behavior, whereas others recognize the entire flow of movements as a single behavior. For interfaces targeting the final phase of a single behavior, this study prepared a pictogram describing “Blow.” For the ones recognizing the entire flow of movements as a single behavior, this study produced two pictograms depicting “Fan” and “Throw.” Use cases may include the assignment of a reaction relevant to each behavior, such as “Blow,” causing dandelion puffs to float and then blossom in the image, “Fan,” causing a wind to
<figure><img src="image_5.png" /><figcaption>Fig. 3. Moving Pictograms</figcaption></figure>

<caption>Fig. 4. Still Pictograms</caption>

blow in the landscape in the image, and “Throw,” triggering a ball to be flying in the image.

### (3) Input method using buttons and scrolling

Because input methods using buttons and scrolling, this study fabricated pictograms telling users to “Hold a hand above something,” “Manipulate a seek bar,” and “Swipe.” For example, “Hold a hand above something” can function as various switching operations such as pressing, clicking, or tapping buttons or icons to switch a projected image to a different image. “Manipulate a seek bar” realizes the manipulation of a seek bar, an interface used in video websites in an actual space where users move their hands to the right and left to manipulate videos. “Swipe” is primarily incorporated into smartphones and tablets, allowing users to relocate images to intended directions. Traditionally, swiping action is caused by fingers placed on a display, but in an actual space, arms are probably used to manipulate projected videos.

### (4) Input method using external objects

As an input method using external objects, this study fabricated a pictogram in-structing users to “Place an object.” A use case may include an object in an exhibit space, whose relocation changes the display.
# 4 Evaluation

## 4.1 Designing Systems for Evaluation

This section describes the structure of evaluation systems to confirm the effect of pictograms in communicating behaviors. The systems use RGB-D cameras to detect an approaching user and display a pictogram with a sound effect above the content images to ensure that the user notices the pictogram while avoiding interference with the experience provided by the exhibit. Moving pictograms are replayed in a loop. Once the user successfully executes the intended behavior, the pictogram disappears and the result of the behavior is shown in the exhibit image. The pictogram disappears when the user steps away from the exhibit without achieving the intended behavior.

To recognize “Face front and spread hands” and “Face front and clench fists,” this study used a recognition library included in Kinect. This study created original movements to recognize “Blow,” “Fan,” “Throw,” and “Swipe” and registered them in Visual Gesture Builder. For “Hold a hand above something” and “Manipulate a seek bar,” this study designed the display to switch images when the hands of a user’s skeletal frame detected by RGB-D cameras enter the space coordinates above the icon. For “Place an object,” this study used an OpenCV cascade classifier to switch the displayed image when the user’s hands are above the icon showing a location to place an object while RGB-D cameras recognize the object.

For icons, to manipulate seek bar, and to place an object, pictograms instructing “Hold a hand above something,” “Manipulate a seek bar,” and “Place an object” are projected on a tabletop with a mobile projector (Fig. 5, Left), whereas other pictograms are projected on the wall with short-focus projectors (Fig. 5, Right). Wall pictograms are projected on the same location in the center. Because the exhibit contents may provide users with clues to understand behaviors meant to be communicated through pictograms, the systems in this study did not display the exhibit images. Instead, the systems showed “OK” on the screen when RGB-D cameras recognize the user’s correct behaviors (Fig. 5, Bottom Right).

<figure><img src="image_6.png" /><figcaption>Fig. 5. Experimental Environment</figcaption></figure>

Correct movements were defined as follows, and behaviors were judged correct when they were visually identified as being correctly executed:

* For “Clench fists,” hands are clenched into fists while facing front;
* For "Blow," hands are clenched into fists and placed in front of the head;

* For "Fan," arms are extended forward with hands swinging vertically down from a position above the head to the position below the head;

* For "Throw," hands are moved from behind the head to the front of the head;

* For "Hold a hand above something," a projected icon is covered by a hand;

* For "Manipulate a seek bar," a hand is placed above the seek bar and waved sideways;

* For "Swipe," arms are extended and spread open from an inwardly bent position;

* For "Place an object," the entire bottom of the object is placed in the icon, indicating the place to place the object.

## 4.2 Comparison of Still Pictograms and Moving Pictograms

To confirm the comprehensibility of still and moving pictograms, this study conducted experiments from September 3rd to 19th, 2019, August 4th to 15th, 2020, and September 19th to 25th, 2020, in which 52 undergraduate students, postgraduate students, and researchers (43 men and 9 women) were randomly shown eight types of both still and moving pictograms, respectively. The participants were then divided into two groups of 26 people to perform behaviors described in the pictograms, and the correct answer rate was measured. Those who correctly responded to the pictograms were then timed to determine the average time it took to complete the behaviors.

The correct answer condition was when the RGB-D camera could recognize a user's action and switch the displayed image or when the user fulfilled the requirements for each action. Incorrect answers were given when the RGB-D camera could not recognize the user's action, even after 5 s had passed after the pictogram was displayed or when the user did not satisfy the requirements for the action. However, "Place an object" was set to 8 s because it involves the process of picking up an object from an object storage area adjacent to the projection surface. Moreover, because the basic actions of "Face front and spread hands" and "Face front and clench fists" are the same, only "Face front and clench fists" was used for evaluation, considering the burden on the user. Fig. 6 shows the respective correct answer rates for still and moving pictograms, respectively, and average time and the dispersion of time spent to complete behaviors.

Note that >88.5% (rounded to the tenth place) of participants (i.e., 23 of 26 participants) correctly understood moving images for all behaviors, among which three behaviors were understood by all participants (100%). Behaviors with the largest discrepancies in the correct answer rates between the still and moving images were "Blow," "Throw," "Hold a hand above something," "Manipulate a seek bar," and "Place an object." An evaluation of ratio differences with a significance level of 5% revealed significant differences in "Blow," "Throw," and "Hold a hand above something."

In response to the still pictograms indicating "Blow," certain participants just blew without moving hands while certain participants extended arms sideways to "Throw." Tabletop operations were assumed for "Hold a hand above something," "Manipulate a seek bar," and "Place an object," but certain participants reacting to still images erro-
<figure><img src="image_7.png" /><figcaption>**Fig. 6.** Comparison of Still and Moving Images (Top Row: Correct Answer Rate / Bottom Row: Average Time Spent on Achieving Desired Behaviors)</figcaption></figure>

neously extended arms in the air the same way as the person pictogram projected on the wall. Moving pictograms achieved higher correct answer rates in these behaviors probably because users could easily assume the behaviors to take place on the tabletop from the direction of the hand's movement shown in the pictogram. Furthermore, only pictograms were displayed in this experiment, whereas the image of interfaces would be shown on the tabletop in actual use cases (see Figs. 8 and 9). Therefore, in actual usage, erroneous movements, such as holding a hand in the air, would possibly be reduced.

The bottom row of Fig. 6 (time spent on achieving the correct behaviors) indicates that, compared to still images, moving pictograms require a shorter time to achieve the correct behaviors. In this evaluation experiment, different people viewed and acted on pictograms in both still and moving images; therefore, we conducted an unpaired t-test (Welch's t-test). A two-tailed t-test was conducted, with a significance level of 5%, and the p-values for "Blow," "Hold a hand above something," "Swipe," and "Place an object" were each .05, indicating a significant difference. There is a wide gap between the moving and still images in terms of time spent to correctly execute "Place an object," probably because the moving pictograms depict the act of moving and placing the object within a designated circle more clearly, reducing the time spent on wondering what to do or misplacing the object.
4.3 Evaluation of Users in Their Forties Up to Their Seventies

This study had 14 users in their fifties up to their seventies (one user in his forties, seven users in their fifties, four users in their sixties, and two users in their seventies) watch the moving pictograms and behave accordingly to determine whether the pictograms can effectively communicate the intended behaviors to older users. In this evaluation, pictograms projected on the wall in actual use were shown on a large TV screen.

In Fig. 7, the numbers shown in gray and black are the number of participants who correctly responded to the pictograms within 5 and 10 s, respectively. Compared to the evaluation made in Section 4.2, the participants tended to take a longer time in responding to pictograms with some reacting correctly only after 5 s. Moreover, regarding tabletop interfaces, users shown in red in “Hold a hand above something” and “Manipulate a seek bar” categories are the ones who responded with correct behaviors but not in the correct place, which were above icons and seek bars. Regarding the “Place an object” pictogram, certain users were observed to be obsessed with the projected image and failed to notice the object. Elaborations such as providing additional explanations to navigate users to the tabletop or circling the object in red may be required to facilitate user understanding.

<figure><img src="image_8.png" /><figcaption>Fig. 7. Evaluation by Users in Their Forties up to Their Seventies</figcaption></figure>

5 Practical Use Cases of Instruction Pictograms

5.1 Use Case of Pictograms in Section 3

This section introduces a use case of certain pictograms produced in this study, namely, ones prompting users to “Hold a hand above something,” “Place an object,” “Face front and spread hands,” “Face front and clench fists,” and “Manipulate a seek bar,” for an artwork introducing the culture of shrine worship [15]. In a work explaining a deity (Fig. 8), the movement to “Hold a hand above something” is used to switch texts projected on a scroll painting, whereas the movement to “Place an object” is used to display information about a deity by placing an object of the divine messenger corresponding to the deity on a designated place.
Furthermore, in the animation explaining how to ritually cleanse hands (Fig. 9), the movement to “Face front and spread hands” prompts replay, “Face front and clench fists” prompts pause, and “Manipulate a seek bar” enables the manipulation of the starting point of replay in the movie.

<figure><img src="image_9.png" /><figcaption>**Fig. 8.** Use Examples Explaining about an Informative Work on Enshrined Deities</figcaption></figure>

<figure><img src="image_10.png" /><figcaption>**Fig. 9.** Use Examples Explaining Ritual Hand Cleansing Manner</figcaption></figure>

## 5.2 Application Case

In an artwork designed to wrap objects with a wrapping cloth [16], this study fabricated pictograms to instruct how to use a wrapping cloth by applying design requirements and production methods described in Section 3.3 (Fig. 10). Because the instruction requires an input method using external objects, the pictograms featured the wrapping cloth in the center. Because the use of wrapping cloth involves a workflow comprising multiple steps, a pictogram is fabricated for each action, and they were arranged in order from left to right. To prevent the pictograms from interfering with the user's act of wrapping with a wrapping cloth, the systems placed them on top (when seen by the user) of the projected image. The systems were on display in an event space of BiVi Tsukuba (a commercial complex) from August 8, 2019, to August 10, 2019, and were visited by 23 users. This study observed behaviors of 14 users aged under ten up to their seventies to determine whether they can correctly use the wrapping cloth through the pictogram guidance. Note that 12 of 14 users were observed to correctly wrap an object with the help of pictograms, achieving a correct answer rate of 85.7% (rounded to the tenth place).

In the MR system featuring building blocks [19], users can freely pile blocks, while the display shows a concrete image suitable for the shape of the arranged blocks. Because the creator desires users to freely imagine through their experience, the system inserted pictograms on the introductory screen showing the title and explanation and depicted the construction of a house shape as an example (Fig. 11).

In the furoshiki and building block systems, where pictograms were practically introduced, users could correctly act as the author intended. Currently, the pictogram is displayed until the user performs the correct action; however, if the correct action cannot be recognized for a few seconds, a pictogram informing the user that the action
<figure><img src="image_11.png" /><figcaption>**Fig. 10.** Experience-based System Using Wrapping Cloth</figcaption></figure>

<figure><img src="image_12.png" /><figcaption>**Fig. 11.** Experience-based System Using Woodblocks</figcaption></figure>

has not been performed or a pictogram encouraging the correct action could be displayed.

# 6 Conclusion

This study classified behaviors required for experience-based displays incorporating interfaces using gesture input, which can be communicated through pictograms, and generated pictograms after clarifying design requirements. Consequently, behaviors were successfully communicated both through still and moving images, with moving pictograms achieving higher correct answer rates and >80% of all users successfully comprehending all behaviors. Users in their forties up to their seventies successfully understood communicated behaviors, but they occasionally chose the incorrect place to make movements when it must be done on a tabletop interface, indicating the requirement for more detail on how to navigate users to make the required movements on a tabletop.

Future research may require to verify the effect of pictograms on communicating behaviors in different languages and cultural settings, as well as their impacts and effects when incorporated into museum exhibits.

# References

1. Eyal Krupka, et al.: Toward Realistic Hands Gesture Interface: Keeping it Simple for Developers and Machines. ACM CHI 2017, pp. 1887–1898. (2017).

2. MANOMOTION, http://www.manomotion.com, last accessed 2021/7/21.

3. Huidong Bai, et al.: Free-hand Gesture Interfaces for an Augmented Exhibition Podium. OzCHI 2015, pp. 182-186. (2015).

4. Leila E Mansoor, Ros Dowse.: Design and evaluation of a new pharmaceutical pictogram sequence to convey medicine usage. Ergonomics 16(2), pp. 29–41. (2004).
5. Sojung Kim.: Trends in Olympic Pictograph Design: A comparative study using Olympic Games' Sports Symbols, PARSONS JOURNAL FOR INFORMATION MAPPING, vol.4, no.4, pp. 1-12. (2012).

6. Margaret I. Fitch, et al.: The Fatigue Pictogram: Psychometric evaluation of a new clinical tool. CONJ • RCSIO, vol.21, no. 4, pp. 205-210. (2011).

7. Nader Vossoughian.: Otto Neurath: The Language of the Global Polis, Nai Uitgevers Pub, pp. 1-174. (2011).

8. Pere Tuset, et al.: Messenger Visual: A pictogram-based IM service to improve communications among disabled people. NordiCHI 2010, pp. 797–800. (2010).

9. Yumiko Mori, et al.: Patterns in Pictogram Communication. IWIC 2009, pp. 277–280. (2009).

10. BoSign, http://actgram.org, last accessed 2021/7/21.

11. Ohno Shintaro, et al.: Proposal of Design Indexes in Pictograms Using Dynamic Expression. Bulletin of Japanese Society for the Science of Design, vol.60, no.1, pp. 95-102. (2013).

12. Masaki Ogawa, et al.: An Instruction Method for Gestural Interaction with Interactive Public Media. IPSJ Journal, vol.56, no.1, pp. 316-328. (2015).

13. Takashi Okuma, et al.: Effects of Virtual Viewpoint Control of 3-D Maps and How-to-Experience Instructions on a Mobile Science-Museum Guide. The Virtual Reality Society of Japan, vol.14, no.2, p. 213-222. (2009).

14. Oculus Quest, https://www.oculus.com/quest/?locale=ja_JP, last accessed 2021/7/21.

15. Kei Kobayashi, Junichi Hoshino.: Augmented Reality Media for Experiencing Worship Culture in Japanese Shrines. IFIP International Conference on Entertainment Computing (ICEC 2020), pp. 270-276. (2020).

16. Kei Kobayashi, et al.: FUROSHIKI: Augmented Reality Media That Conveys Japanese Traditional Culture. ACM SIGGRAPH VRCAI 2019, pp. 1–5. (2019).

17. Subhas C. Maharaj.: Pictogram ideogram communication: what's it all about?. George Reed Foundation for the Handicapped. (1980).

18. INTERNATIONAL STANDARD ISO 3864-1, https://irp-cdn.multiscreensite.com/e4e8d52c/files/uploaded/ISO%203864-1_2002.pdf, last accessed 2021/7/21.

19. Kazuma Nagata, et al.: Point cloud processing in mixed reality content using building blocks. IPSJ Interaction 2020, pp. 626-628, http://www.interaction-ipsj.org/proceedings/2020/data/pdf/2B-37.pdf, last accessed 2021/7/21.