# The Alteration of Gustatory Sense by Virtual Chromatic Transition of Food Items

Yuto Sugita, Keiichi Zempo, Koichi Mizutani, Naoto Wakatsuki

► To cite this version:

Yuto Sugita, Keiichi Zempo, Koichi Mizutani, Naoto Wakatsuki. The Alteration of Gustatory Sense by Virtual Chromatic Transition of Food Items. 16th International Conference on Entertainment Computing (ICEC), Sep 2017, Tsukuba City, Japan. pp.211-217, 10.1007/978-3-319-66715-7_23 . hal-01771254

HAL Id: hal-01771254

https://inria.hal.science/hal-01771254

Submitted on 19 Apr 2018

HAL is a multi-disciplinary open access
archive for the deposit and dissemination of sci-
entific research documents, whether they are pub-
lished or not. The documents may come from
teaching and research institutions in France or
abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est
destinée au dépôt et à la diffusion de documents
scientifiques de niveau recherche, publiés ou non,
émanant des établissements d'enseignement et de
recherche français ou étrangers, des laboratoires
publics ou privés.

<figure><img src="image_1.png" /></figure>

Distributed under a Creative Commons Attribution 4.0 International License
# The Alteration of Gustatory Sense by Virtual Chromatic Transition of Food Items

Yuto Sugita<fnref n="1" />, Keiichi Zempo<fnref n="2" />,<fnref n="3" />(✉), Koichi Mizutani<fnref n="2" />, and Naoto Wakatsuki<fnref n="2" />

**Abstract.** When we visually recognize food items, their color influences the impression of their taste, because the gustatory sense is affected by other senses such as smell and eyesight. In this study, in order to control the gustatory sense of human beings, we examined the influence on gustatory sense by giving a virtual chromatic transition of food through Head-Mounted Display. The experiment was conducted under six conditions that changed three factors: "before or after eating", "viewing through HMD or naked eyes", "with virtual chromatic transition or without". As a result, we confirmed that the difference in the amount of visual information influenced the impression of food items before eating. It is expected to realize more applicative control of the gustatory sense by analyzing this difference.

**Keywords:** Gustatory Sense • Virtual reality • Head-Mounted Display

• Visual information

## 1 Introduction

The taste information obtained from the taste cells present in the tongue accounts for the majority of the human gustatory sense [1]. However, it is known that the gustatory sense is affected not only by taste information but also by other sensory information such as visual information [2, 3]. Recently, studies have been increasingly being made to change dining experience by externally changing sensory information [4]. In particular, the influence of olfactory and visual information on taste sensation is large, therefore a lot of studies have been done by using this [5–8]. These effects are recognized also in psychological experiments [9]. For example, Meta Cookie gives a change to the feeling of human gustatory sense by changing both visual information and olfactory information [8]. It's also said that auditory information when chewing food, tactile information such as the texture of food and chewy feeling influences the feeling of taste [10–12]. Previous research suggests that changes in sensory information from the outside can also change taste information. However, quantitative evaluation of how taste

<footnote n="1">¹ Graduate School of Systems and Information Engineering, University of Tsukuba, Tsukuba, Ibaraki 305-8573, Japan</footnote>

<footnote n="2">² Faculty of Engineering, Information and Systems, University of Tsukuba, Tsukuba, Ibaraki 305-8573, Japan</footnote>

<footnote n="3">³ Artificial Intelligence Research Center, AIST</footnote>

<footnote>zempo@iit.tsukuba.ac.jp</footnote>
information is affected by changes given to sensory information has not yet been made. If a quantitative evaluation is possible, more applicative control of the gustatory sense becomes possible. Among the sensory information affecting the taste like this, the visual information can change most easily the feeling of the taste significant. Unlike other sensory information, the visual information can be separated three primary colors of light, therefore it is easy to present any visual information. Therefore, by changing the color of the food through the visual sensor and visually recognizing the food through the display, it is possible to change the visual information of the food. In addition, as the penetration rate of smartphones is increasing in recent years, there are many scenes of possessing visual sensors such as cameras. There are also many Head-Mounted Display (HMD) that can provide VR environment easily by wearing smartphones, therefore it is possible to impart real world to a chromatic transition of food.

Also, in recent years, content production related to dining experience has been actively performed [13, 14]. Dining experiences that are familiar to humans should be always new and enjoyable by adding such entertainment characteristics. It is expected that the control of the gustatory sense will greatly contribute to such content creation in the future.

In this research, we aim to verify the influence of a chromatic transition of food through a HMD on the impression of taste for food. For this purpose, we constructed a system that gives a chromatic transition to the color of food through HMD. In this research, we also examine the difference between visually recognizing food with the naked eye and the recognition it through HMD. Therefore, to control the impression of human taste by optimal color change, it is necessary to consider the difference between the case of visually recognizing with the naked eye and the case of seeing through the display.

## 2 VR System of a Chromatic Transition

In this research, smartphone and HMD were used as a color change presentation device. iPhone 7 Plus which iOS 10 introduced by Apple Co., Ltd. was introduced, and VOX + 3DVR goggle provided by VOX was used for HMD. We can really find the image which we acquired from the camera of the smartphone like the image actually in front of the eyes by attaching HMD which set the smartphone. By giving a color change to the image acquired from this camera and outputting it to the display, it is also possible to give a chromatic transition to the user. These devices are shown in **Fig. 1**.

And, we installed the application created in Xcode on iPhone 7 Plus and used it to detect target food and change color to that food. The source code of the application is written in Objective-C and introduced the OpenCV framework for image processing such as a color change to target food. A flow chart showing the flow of the created application is shown in **Fig. 1**, and detailed specifications are shown below.
<figure><img src="image_2.png" /><figcaption>Fig. 1. Experiment environment and a virtual chromatic transition system</figcaption></figure>

1. Acquire an image from a camera mounted on a smartphone

2. Convert all the pixels in the image obtained from the camera from the original RGB color space to the HSV color space

3. Of the parameters of the HSV color space, make specified color change for pixels with *V* (Value, Brightness) above a certain value

4. Output the pixel on which the color change was made on the original image to the display

The food whose color has been changed by the above operation is outputted to the display and made to be visible to the human through the HMD.

# 3 Experiment of Verification

## 3.1 The Experimental Environment and Target Food Item

This experiment was conducted for 11 males and females in their twenties who are healthy with both visual and taste. The experimental environment is shown in **Fig. 1**. In this experiment, the application was carried out in a room that turned off the light, unifying the background to black so that the application can easily detect white food. A target food was placed on the inside of which processing was made by attaching a black felt fabric around an acrylic box which is a cube having one side of 300 mm and the front side does not exist, and this subject was visually recognized by the subject.

In this experiment, the self-made fizzing candy was used as the target food item. All of the ingredients were made equal in 0.1 g unit so that each of the fizzing candy had the same taste. Those colored by using coloring agents were prepared for each of them. The colors used this time are five types of red, blue,
<table><caption>**Table 1.** Situation of experiment</caption><thead><tr><th>Experiment No.</th><th>Before or After</th><th>Viewing Method</th><th>Chromatic Transition</th></tr></thead><tbody><tr><td>#1</td><td>Before</td><td>Naked eye</td><td>None</td></tr><tr><td>#2</td><td>After</td><td>Naked eye</td><td>None</td></tr><tr><td>#3</td><td>Before</td><td>HMD equipped</td><td>None</td></tr><tr><td>#4</td><td>After</td><td>HMD equipped</td><td>None</td></tr><tr><td>#5</td><td>Before</td><td>HMD equipped</td><td>Transitioned</td></tr><tr><td>#6</td><td>After</td><td>HMD equipped</td><td>Transitioned</td></tr></tbody></table>

yellow, green and purple shown in **Fig. 1**. Experiments were carried out using a total of six kinds of color fizzing candies to which white of the reference color was added.

## 3.2 Method of Experimentation

In this experiment, we asked each subject to answer the questionnaire about the impression of the taste of the target food. In addition, the experiment was conducted under six situations in which three factors, “before or after eating,” “viewing through HMD or naked eyes,” “with Virtual chromatic transition or without” were changed. A table summarizing these is shown in **Table 1**, a diagram showing these conditions is shown in **Fig. 2**. To this “six colors of fizzing candy” will be added.

The questionnaire prepared two kinds of contents different “before or after eating”. The contents of the questionnaire before eating are two items, “What kind of taste do you think it is?,” and “What kind of food do you imagine?”. Next, the contents of the questionnaire after eating are three items, “What kind of taste did you feel?”, “What kind of food did you imagine?”, “Were there any differences of impressions before eating?”. The first questionnaire item evaluated what kind of taste it felt before and after eating about five colors excluding white, in 5 grades each of the five flavors. At this time, white was defined as the evaluation value 3 in all the five flavors before and after eating, and it was used as a standard.

## 3.3 Result

From the experimental results, evaluate how far away the impression was from the white fizzing candy received as a reference against the colored. At this time, we define the taste space as 5-dimensional space and evaluate the evaluation value of each basic taste answered by the subject to Euclidean distance when white is taken as the origin and evaluate.

First, in order to compare “before or after eating” and “viewing through HMD or naked eyes”, we perform Analysis of variance using the result of experiment 1, 2, 3 and 4, as factors. Next, in order to compare “before or after
<figure><img src="image_3.png" /><figcaption>Fig. 2. Situation of experiment</figcaption></figure>

<figure><img src="image_4.png" /><figcaption>Fig. 3. Result of analysis of variance</figcaption></figure>

eating" and "giving or not virtual chromatic transition", we perform an analysis of variance using the result of experiment 3, 4, 5 and 6, as factors. Graph showing these results is shown in **Fig. 3**.

The abscissa of **Fig. 3** shows the six situations of experiment shown in **Table 1**, and the ordinate shows the average value of the Euclidean distance of the result calculated for each situation of the experiment.

Firstly, as a result of analyzing the experiment #1, #2, #3 and #4, the interaction of “before or after eating” and “viewing through the HMD or naked eyes” was superior at 5% level or less. It was also revealed that the impression before eating was different depending on the presence or absence of the HMD by the simple main effect test. It seems to be due to the difference in the amount of visual information obtained when comparing the state in which the food is viewed through the HMD and the state in which the food is visually recognized with the naked eye. The information acquired from the camera has a difference
in information amount as compared with information obtained from the naked
eyes due to various factors such as image quality and viewing angle. It seems
that the difference in this amount of information affected the impression of the
food before eating.

Next, as a result of analyzing the experiment #3, #4, #5 and #6, the in-
teraction of “before or after eating” and “with virtual chromatic transition or
without” was superior at 1% level or less. Moreover, it was revealed that the
impression before eating is caused by the presence or absence of color change by
examination of simple main effect. This seems to be related to the amount of vi-
sual information as well. The amount of visual information decreases if it passes
through a display like a camera. The application used in this experiment was to
add a chromatic transition from the top of the image acquired from the camera.
In other words, when a chromatic transition is added by the application used
in this experiment, it can be said that it is easier to obtain visual information
because clear colors can be seen compared with the case where no chromatic
transition. It is seemed that the difference in the amount of visual information
has influenced the impression of the food before eating.

4 Conclusion

In this study, we examined the difference in the feeling of taste especially when visually recognizing the food with the naked eye and through the HMD. For this purpose, we installed an application on the HMD and made an application to change the color of the image acquired from the camera mounted on the smartphone. The experiment was conducted under six situations in which three situations, “before or after eating”, “viewing through HMD or naked eyes”, “with virtual chromatic transition or without” were changed. In each case, differences in the taste impression due to these factors were verified by subjective evaluation about the taste impression of the target food. The result of analysis revealed that “viewing through HMD or naked eyes,” and “with virtual chromatic transition or without” respectively influence the impression before eating food. Therefore, the experiments suggested that the impression of food tastes were changed due to the difference in the amount of visual information between the naked eye and the screen. In other words, when giving a chromatic transition to a food through a screen such as HMD, it is possible that considering the difference in impression of the case of visual recognition with the naked eye leads to more advanced control of taste.

Moreover, verification experiments have revealed that there is almost no in-
fluence on the feeling of the taste of eating food merely by changing the visual
information. Changes in olfaction, tactile sensation, and auditory information
are thought to be useful for effecting this. From now on, we will investigate how
they affect taste sensation by changing these, and also realize a taste control
system by constructing a multimodal interface including these.
References

1. Yutaka,M.:Mechanisms for taste signaling in taste buds. In:The Japanese Associa-
tion for the Study of Taste and Smell, 15(1), pp.5–10 (2008)

2. Tomomi,K.:Nippon jin no mikaku [Japanese taste], Tamagawa sensyo, Tokyo (1980)

3. Keiji,U.,Masao,O.:Mikaku,Kyukaku [Taste,Smell], Asakura syoten, Tokyo (2008)

4. Marcelo C.: DinnerWare: Why Playing With Food Should Be Encouraged. In:CHI Extended Abstracts, pp.3505–3506 (2009)

5. Alley,RL., Alley,TR.:The influence of physical state and color on perceived sweet-
ness. In:J. Psychology Interdisciplinary Applied 132, pp.561–568 (1997)

6. Bayarri,S., Calvo,C., Costell,E., Duran,L.:Influence of color on perception of sweet-
ness and fruit flavor of fruit drinks. In:Food Science and Technology International
7, pp.399–404 (2001)

7. Stillman J.A.:Color influences flavor identification in fruit-flavored beverages.
In:Journal of Food Science 58, pp.810–812 (1993)

8. Takuji,N., Tomohiro,T., Takashi,K., Michitaka,H.:Meta Cookie:Pseudo – Gustatory Display Based on Cross – Modal Integration. In:Transactions of the Virtual Reality Society of Japan 15, pp.579–588 (2010)

9. Johnson,J.:The psychophysical relationship between color and flavor. Ph.D. the-
sis.University of Massachusetts at Amherst (1982)

10. Daichi,I.,Takafumi,K.:Change of Mouthfeel by Means of Cross-modal Effect Us-
ing Mastication Sound and Visual Information of Food. In:IEICE technical report
I.M.Q. 114, pp.83–86 (2014)

11. Tetsuro,M.,Hiroaki,Y.,Hiroo,I.:Research on Sensory Integration in a Taste of Food.
In:Transactions of the Virtual Reality Society of Japan 9, pp.259–264 (2004)

12. Naoya Koizumi, Hidekazu Tanaka, Yuji Uema, and Masahiko Inami. 2011. Chew-
ing jockey: augmented food texture by using sound based on the cross-modal effect.
In:Proceedings of the 8th International Conference on Advances in Computer En-
tertainment Technology (ACE ’11), Article 21, pp.1–4 (2011)

13. Verchan Reality, http://www.awajikoku.com/

14. PROJECT NOURISHED, http://www.projectnourished.com/