# Augmented Happiness: Simple Color Changes Influence Users' Conceptual Choices

Diana Löffler, Wolfgang Paier, Takashi Toriizuka, Mio Ikeda, Jörn Hurtienne

► To cite this version:

Diana Löffler, Wolfgang Paier, Takashi Toriizuka, Mio Ikeda, Jörn Hurtienne. Augmented Happiness: Simple Color Changes Influence Users' Conceptual Choices. 15th Human-Computer Interaction (INTERACT), Sep 2015, Bamberg, Germany. pp.248-255, 10.1007/978-3-319-22723-8_20. hal-01610770

HAL Id: hal-01610770

https://inria.hal.science/hal-01610770

Submitted on 5 Oct 2017

HAL is a multi-disciplinary open access
archive for the deposit and dissemination of sci-
entific research documents, whether they are pub-
lished or not. The documents may come from
teaching and research institutions in France or
abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est
destinée au dépôt et à la diffusion de documents
scientifiques de niveau recherche, publiés ou non,
émanant des établissements d'enseignement et de
recherche français ou étrangers, des laboratoires
publics ou privés.

<figure><img src="image_1.png" /></figure>

Distributed under a Creative Commons Attribution 4.0 International License
Augmented Happiness: Simple Color Changes
Influence Users' Conceptual Choices

Diana Löffler¹, Wolfgang Paier², Takashi Toriizuka³, Mio Ikeda³, and Jörn Hurtienne¹

¹ University of Würzburg, Chair of Psychological Ergonomics, Würzburg, Germany
{diana.loeffler,joern.hurtienne}@uni-wuerzburg.de

² Fraunhofer HHI, Computer Vision and Graphics, Berlin, Germany
wolfgang.paier@hhi.fraunhofer.de

³ Nihon University, College of Industrial Technology, Chiba, Japan
toriiduka.takashi@nihon-u.ac.jp, mio.ikeda1220@gmail.com

**Abstract.** Color is a powerful visual property and is used to make interferences about the world. However, no theoretical framework is available that explains precisely where color associations come from and how they affect psychologi- cal functioning, making it difficult to predict how color affects human-computer interaction. This paper aims at closing this gap by suggesting an Embodied Cognition view on color, which assumes that the aggregate of our perceptual color experiences is part of the mental representation of tactile object attributes and thus systematically influences our abstract thinking via the process of metaphorical mapping. An empirical study is presented in which hues and satu- ration of objects were manipulated via Augmented Reality. Participants matched objects to abstract concepts, e.g., happiness. 83% of the participants’ choices were correctly predicted, suggesting that color information is consid- ered during the processing of abstract information. Eight color-to-abstract- concept mappings are recommended and possible areas of application are dis- cussed.

**Keywords:** Augmented Reality, Color, Embodied Cognition, Conceptual Me-
taphor.

# 1 Introduction

Color is a ubiquitous perceptual experience that influences perception, cognition and behavior. However, color research mainly focuses on establishing relations between color and psychological functioning and often lacks a rigorous theoretical understand- ing of why these relations occur [7]. This might be a main reason why findings on the impact of color on psychological functioning are largely inconsistent, unreliable and inconclusive [e.g., 22; 13]. In the field of human-computer interaction (HCI), design recommendations on color are usually restricted to issues of readability, aesthetics or color preferences [3]. This paper aims to advance our theoretical understanding of how and why color affects psychological functioning to leverage a systematic applica-
tion of color in the design of human-computer interaction. To achieve this, the framework of embodied cognition, which holds the promise of providing a unifying perspective for psychological research in general [8], is briefly reviewed and then applied to the field of color.

In recent years the predominant views of the human mind, such as the computa-
tional theories of information processing, have been increasingly challenged by find-
ings that cognition is embodied (cf. [6]). Embodied cognition grants the body a cen-
tral role in cognitive processing. Physical experiences we have, moving through the
material world, scaffold the development of our conceptual knowledge. Many of such
embodied experiences are manifested in language [15]. Expressions such as “the
heavy burden of guilt” or “that warmed my spirits” reflect how the mental representa-
tions of guilt and happiness are metaphorically conceptualized. These conceptual
metaphors involve a mapping from a physical source domain (weight, temperature) to
another, more abstract, target domain (guilt, happiness) [15]. Being equipped with
conceptual metaphors in our embodied minds, even seemingly unrelated visual or
haptic properties of objects or technical devices influence our perception, feeling, and
behavior (cf. [12]), in a way predicted by the framework of embodied cognition.

## 2 An Embodied Cognition View on Color

Color is an important perceptual feature and in natural environments often diagnos-
tic for other perceptual properties [4], like edibility and taste [19], weight [21; 1] or
temperature [10]. Thus, it is likely that color affects psychological functioning in a
similar way than related perceptual properties via the process of metaphorical map-
ping [16; 20].

Many studies already investigated associations between color and other perceptual attributes like weight, size and temperature. For example, according to [5], tempera- ture attributes are solely related to hue (red is warm, blue is cold). Increasing weight is related to decreasing brightness and an increase in saturation [1], while when the size of a stimulus is increased, it is perceived as brighter and more saturated [23].

In recent years, a few studies have been conducted to empirically test these asso-
ciations between colors and perceptual properties. For example, in an HCI relevant
application, Ban and colleagues [2] proposed an Augmented Reality (AR) system to
manipulate weight perception through an increase in object brightness. They demon-
strated that this facilitated the lifting of medium-weight objects. However, as with
most of the research on relations between color and psychological functioning [7], no
theoretical background is provided on how and why this association is shaped, mak-
ing it difficult to transfer the results into other situations and contexts. An embodied
cognition view on color may close this theoretical gap, as it explains that the associa-
tion between weight and brightness arises through the repeated experience that heav-
ier objects appear to be darker than light objects (e.g., when a shirt gets wet it will be
heavier and darker, [21]). Thus, the perception of brightness and weight mutually
influence each other. In addition, this association between brightness and weight
might be extended to more abstract concepts via metaphorical mapping [17; 20]. For
example, weight is metaphorically linked to psychological significance, or, in other
words, weight is conceptualized as an embodiment of importance [14]. Therefore, it
can be assumed that the perception of the color property brightness will bias informa-
tion processing in contexts where psychological significance plays a role. This predic-
tion can now be put to direct empirical test.

In this paper, we aim to experimentally test the embodied cognition view on color
and investigate whether perceiving certain color characteristics will influence the
cognitive processing of more abstract information (like happiness, emotionality, or
power), which is metaphorically linked to the tactile properties weight, size and tem-
perature.

3 Method

**3.1 Participants**

Fourteen volunteers were recruited at the campus of Nihon University, Japan, (seven male, seven female) to participate in this study. The participants' ages ranged from 21 to 24 years (M = 22.7, SD = 0.8). All participants had Japanese as their mother tongue and had no reported defective color vision. They were all naïve as to the subject under investigation.

**3.2 Procedure**

The experimental procedure was adopted from a study by Hurtienne et al. [12]. Each participant was asked to complete a short demographic questionnaire. The participants were then told that they would wear AR glasses and, in each trial, would be presented with two colored objects (spheres) and an adjective describing an abstract property (e.g. difficulty). AR technology was utilized because it allows for a convenient manipulation of object colors in the viewers' sight.

The participants were instructed to touch the objects with their dominant hand that they felt corresponded most closely to the adjective (Figure 1). Participants were told to decide quickly and instinctively in favor of one object. Each session consisted of 30 trials. To avoid sequence effects, the adjectives and colored objects were presented in random order. The whole experiment lasted about 15 minutes.

**3.3 Apparatus and material**

Each of the two wooden spheres was 9 cm in diameter and weighed 250 g. They
were colored in blue and green (controlled for saturation and brightness) in order to
facilitate technical object recognition against the background and against the partici-
pants' hands and arms. These spheres were the target objects for the color changes,
which did perfectly obscure the spheres. The spheres' position was counterbalanced
across participants. The participants were seated in an experimental room; the table
and wall in front of them were covered in gray cloth. The spheres were placed on the table within the reach of the participants, with 15 cm distance between them.

<figure><img src="image_2.png" /><figcaption>**Fig. 1.** A participant wearing AR-glasses that change the object colors in the participants' sight.</figcaption></figure>

The AR system used in the study was composed of a video see-through realizing Head Mounted Display VUZIX Wrap1200AR (resolution 852 x 480, 35 degrees diagonal field of view) and a notebook (Intel Core i7 4x2.4 GHz 8.00 GB RAM, Windows 8.1 64 bit). The color manipulation software was based on OpenCV, which provides convenient interfaces for processing and capturing images from the stereo-webcam integrated in the AR glasses. Each pixel was classified in background, green sphere or blue sphere, using a histogram-based color-classifier. The classifier was trained beforehand using six images with proper fore/background masks to fill the color-histograms for each pixel-class. The masks were generated using a semi-automatic approach for foreground extraction [18]. To guarantee the realtime-capability of the system, a static look-up table, containing the color-transform (RGBin to RGBout) for each camera was calculated after the color classifiers were trained.

Three HCI-relevant tactile properties with well-documented metaphorical extensions were selected for the experiment: weight (heavy-light), size (big-small) and temperature (warm-cold) [12]. A pre-study was conducted to determine the relationship between these tactile properties and color characteristics (hue, saturation and brightness). In an online questionnaire, 80 Japanese subjects rated how well 35 different colors (varying in hue, saturation and brightness) matched the properties weight, size and temperature. The results showed that weight and size were mainly associated with the color properties of saturation and brightness, and the temperature is almost solely associated with hue (red vs. blue). These results confirmed those of previous research [5, 23]. The tactile properties of weight, size and temperature were therefore operationalized through their associated color properties in the following way (numbers depict HSB values): heavy, big vs. light, small: violet (270/100/100) vs. (270/60/100), green (140/100/100) vs. (140/60/100), red (360/100/100) vs. (360/60/100), yellow (60/100/100) vs. (60/60/100), orange (40/100/100) vs. (40/60/100); cold: blue (200/100/100), cyan (180/100/100); warm: red (360/100/100),
orange (20/100/100). Note that, although both, saturation and brightness, influence the perception of weight, size and strength, the brightness of the objects in the experiment was not altered. This would have led to a loss in texture and surface shading, thereby negatively affecting the three-dimensional impression of the objects. Hues in the weight and size condition were randomly chosen to create some variation and should, according to theory, have no effect, because hue is not diagnostic for these tactile properties.

For each condition, well-documented metaphorically linked abstract concepts were taken from literature [12; 17] (Table 1). Two lists of adjectives were prepared. One word from each adjective pair was placed on list A (e.g., important), and its opposite was placed on list B (e.g., unimportant). Both groups were balanced with regard to the number of positive and negative words they contained. Half of the participants were presented with adjectives from list A, and the other half with adjectives from list B.

<table><caption>**Table 1.** Tactile properties, conceptual metaphors, stimulus adjectives and results of color-to-abstract-concept mappings; all mappings with str < 0.60 have been grayed.</caption><thead><tr><th>Tactile Property</th><th>Conceptual metaphor with linguistic examples</th><th>Stimulus adjectives</th><th>Related color property</th><th>%</th><th>str</th></tr></thead><tbody><tr><td rowspan="6">Weight is..</td><td>Difficulty: heavy reading; light duties</td><td>difficult, easy</td><td>Saturation</td><td>93</td><td>0.86</td></tr><tr><td>Importance: heavy matters of state; light, idle chatter</td><td>important, unimportant</td><td>Saturation</td><td>91</td><td>0.82</td></tr><tr><td>Guilt: the heavy burden of guilt; easing conscience</td><td>guilty, innocent</td><td>Saturation</td><td>61</td><td>0.21</td></tr><tr><td>Sadness: heavy news; light-hearted girls</td><td>sad, happy</td><td>Saturation</td><td>61</td><td>0.21</td></tr><tr><td>Significance: he is a big man in industry; only a small crime</td><td>significant, insignificant</td><td>Saturation</td><td>96</td><td>0.93</td></tr><tr><td>Quantity: I only have a small amount of money</td><td>much, less</td><td>Saturation</td><td>95</td><td>0.90</td></tr><tr><td rowspan="2">Size is..</td><td>Education: he expanded his horizons</td><td>educated, uneducated</td><td>Saturation</td><td>75</td><td>0.50</td></tr><tr><td>Activity: a hot debate; frozen accounts</td><td>active, inactive</td><td>Hue</td><td>93</td><td>0.86</td></tr><tr><td rowspan="4">Temperature is..</td><td>Emotionality: a warm smile; cold logic</td><td>emotional, unemotional</td><td>Hue</td><td>93</td><td>0.86</td></tr><tr><td>Difficulty: a smoldering conflict; the debate cooled down</td><td>problematic, unproblematic</td><td>Hue</td><td>86</td><td>0.71</td></tr><tr><td>Happiness: that warmed my spirits; a deep cold sadness</td><td>happy, sad</td><td>Hue</td><td>82</td><td>0.64</td></tr><tr><td>Intimacy: giving the cold shoulder; warm feelings</td><td>intimate, distant</td><td>Hue</td><td>75</td><td>0.50</td></tr></tbody></table>
The hypothesis for this experiment was that since specific color characteristics are diagnostic for certain tactile properties, viewing these color characteristics will trigger choices consistent with metaphorically related abstract concepts. If the color characteristics are not associated with the abstract concepts, participants will choose one of the two objects by chance.

# 4 Results and Discussion

The results are depicted in Table 1. The columns show the percentage of answers that were consistent with the metaphorically related concept (averaged over all color samples) and Cohen's kappa values are used as an index for associative strength (str) that takes into account the agreement occurring by chance (50%). According to [16]'s categories of agreement, negative str values indicate “poor” agreement, $0 < \text{str} \le 0.19$ indicate “slight” agreement, $0.20 \le \text{str} \le 0.39$ “fair” agreement, $0.40 \le \text{str} \le 0.59$ “moderate” agreement, $0.60 \le \text{str} \le 0.79$ “substantial” agreement, and $\text{str} \ge 0.80$ “almost perfect” agreement. Because of the limited and non-representative sample size we only interpreted strength values of above 0.60 as supportive of the color-to-abstract-concept mappings (8 of 12). However, across all 12 color-to-abstract-concept mappings, the mean value of str was 0.67 (SD = 0.26), suggesting that at least 83% of the participants' choices were consistent with the conceptual metaphor. This value differs significantly from a chance str of zero, $t(11) = 8.94, p < .001$, Cohen's d = 3.64, and indicates non-randomness in the choices of the participants.

Overall, the results are interpreted in favor of an embodied cognition view on color: 8 out of 12 evaluated abstract concepts showed on average “almost perfect” agreement, while the remaining four concepts showed on average “fair” agreement [16]. These results were predicted by associations between color attributes and tactile properties: because heavier objects tend to be darker and less saturated [21, 1], darker and less saturated hues are assumed to bias our thinking about metaphorically related abstract concepts, like psychological significance, because weight is an embodiment of psychological significance. The link between brightness, saturation and size can be explained by the bodily constraint of a nonuniform distribution of photoreceptors across the retina, resulting in different color vision in the periphery compared to vision in the fovea [24]. The association between physical temperature and hue can be explained by the internalization of correlations between stimuli that are present in the environment (like fire, sun, water) and our bodies (blood circulation) [9]. As predicted, multiple McNemar tests (Bonferroni corrected) revealed no difference between the different hues tested across the weight and size conditions.

# 5 Conclusion and Future Work

With the help of the AR system it could be demonstrated that simple color changes in the users' sight resulted in a bias in decision-making in favor of metaphorically related abstract concepts. This is explained by an embodied cognition view of cognition on colors: because abstract concepts are grounded in observables in the world, we
automatically consider perceived physical properties, such as color and tactile proper-
ties, during reasoning about these abstract concepts. The value of this embodied cog-
nition view on colors lies in its ability to explain and predict why and which effects of
color on psychological functioning occur, taking the guesswork out of color theory
and design.

Moreover, this research has shown that not only hue but also saturation plays a
great role in conveying abstract content [7]. Three abstract concepts were identified
that can be influenced by a change in saturation: difficulty, importance, significance
and quantity. Three concepts were influenced by hue: emotionality, activity and hap-
piness. The concept of difficulty was influenced by both saturation and hue. We ex-
pect that the effects would have been even stronger had the brightness parameter been
manipulated as well. Although selected well above chance level, four color-to-
abstract mappings were weaker than expected and require further investigation.

By extending this approach to applied contexts, the validity and effectiveness of
predictions derived from an embodied cognition view on colors is an area for future
research. Changes in saturation, for example, could be used to indicate the most sig-
nificant, or most-often used objects or functions (significance is saturation); they
could be used to express priority among different items, such as the most valued menu
option in a restaurant, or the best selling products in an online shop (importance is
saturation). Reddish vs. bluish hues could be used to communicate or visualize subtle
mood changes in social networks, online games or in real life situations (happiness is
redness). Changing font or background color could influence the perceived difficulty
of a computerized cognitive training (difficulty is saturation) – and this should also
work for people with cognitive impairment, since embodied cognition effects are
assumed to be rather independent of higher cognitive processing [11].

Although the results of this pilot study are promising, the sample size of the study
reported here is very small (n=14). Larger scale empirical studies have yet to be con-
ducted in order to validate the results obtained here, also with regard to different cul-
tures. Since color and also, to some extent, conceptual metaphors are subject to cul-
tural influences, studies on the applicability of the results within and across countries
and cultures are required.

**Acknowledgements.** We thank the Japan Society for the Promotion of Science for funding this research (grant ID PE14746).

References

1. Alexander, K. R., & Shansky, M. S. (1976). Influence of hue, value, and chroma on the perceived heaviness of colors. *Perception & Psychophysics, 19*(1), 72-74.

2. Ban, Y., Narumi, T., Fujii, T., Sakurai, S., Imura, J., Tanikawa, T., and Hirose, M. Augmented Endurance: Controlling Fatigue while Handling Objects by Affecting Weight Perception using Augmented Reality. In Proc. CHI'13, ACM Press (2013), 69-77.

3. Bonnardel, N., Piolat, A., & Le Bigot, L. (2011). The impact of colour on Website appeal and users' cognitive processes. *Displays, 32*(2), 69-80.
4. Cant, J. S., Large, M.E., McCall, L., and Goodale, M.A. (2008). Independent processing of form, color and texture in object perception. *Perception* 37, 57-78.

5. Cheng, K. M. (2002). *Quantitative evaluation of colour emotions* (Doctoral dissertation, The Hong Kong Polytechnic University).

6. Clark, A. Being There: Putting Brain, Body and World Together Again. Bradford Books, MIT Press, Cambridge, MA, USA, 2007.

7. Elliot, A. J., & Maier, M. A. (2007). Color and psychological functioning. *Current Directions in Psychological Science*, 16(5), 250-254.

8. Glenberg, A. M. (2010). Embodiment as a unifying perspective for psychology. *Wiley Interdisciplinary Reviews: Cognitive Science*, 1(4), 586-596.

9. Ho, H. N., Iwai, D., Yoshikawa, Y., Watanabe, J., & Nishida, S. Y. (2014). Combining colour and temperature: A blue object is more likely to be judged as warm than a red object. *Scientific reports*, 4.

10. Ho, H. N., Van Doorn, G. H., Kawabe, T., Watanabe, J., & Spence, C. (2014). Colour-temperature correspondences: When reactions to thermal stimuli are influenced by colour. *PloS one*, 9(3), e91854.

11. Hurtienne, J., Horn, A.-M., Langdon, P. M., and Clarkson, P. J. Facets of prior experience and the effectiveness of inclusive design. Universal Access in the Information Society 12, 3 (2013).

12. Hurtienne, J., Stöbel, C. and Weber, K. Sad is Heavy and Happy is Light – Population Stereotypes of Tangible Object Attributes. In Proc. TEI’09, ACM Press (2009), 61–68.

13. Jalil, N. A., Yunus, R. M., & Said, N. S. (2012). Environmental colour impact upon human behaviour: A review. *Procedia-Social and Behavioral Sciences*, 35, 54-62.

14. Jostmann, N. B., Lakens, D., & Schubert, T. W. (2009). Weight as an embodiment of importance. *Psychological science*, 20(9), 1169-1174.

15. Lakoff, G., and Johnson, M. *Metaphors We Live By*. University of Chicago Press, Chicago, 1980.

16. Landis, J.R., and Koch, G.G. The Measurement of Observer Agreement for Categorical Data. *Biometrics* 33 (1977), 159–174.

17. Löffler, D. (2014). Population stereotypes of color attributes for tangible interaction design. In *Proceedings of the 8th International Conference on Tangible, Embedded and Embodied Interaction* (pp. 285-288). ACM.

18. Rother, C., Kolmogorov, V., and Blake, A. "GrabCut": interactive foreground extraction using iterated graph cuts. In *SIGGRAPH '04*, ACM Press (2004), 309-314.

19. Shankar MU1, Levitan CA, Spence C. Grape expectations: the role of cognitive influences in color-flavor interactions. *Conscious Cogn.* 2010 Mar;19(1):380-90.

20. Valdez, Patricia; Mehrabian, Albert. Effects of color on emotions. Journal of Experimental Psychology: General, Vol 123(4), Dec 1994, 394-409.

21. Walker, P., Francis, B. J., & Walker, L. (2010). The brightness-weight illusion: darker objects look heavier but feel lighter. *Experimental psychology*, 57(6), 462.

22. Whitfield, T. W., & Whiltshire, T. J. (1990). Color psychology: A critical review. *Genetic, social, and general psychology monographs*.

23. Xiao, K., Li, C. J., Luo, M. R., & Taylor, C. (2004, January). Colour Appearance for Dis-similar Sizes. In *Conference on Colour in Graphics, Imaging, and Vision* (Vol. 2004, No. 1, pp. 12-16). Society for Imaging Science and Technology.

24. Xiao, K., Luo, M. R., Li, C., Cui, G., & Park, D. (2011). Investigation of colour size effect for colour appearance assessment. *Color Research & Application*, 36(3), 201-209.