# Cultural Computing - How to Investigate a Form of Unconscious User Experiences in Mixed Realities

Matthias Rauterberg, Jun Hu, Geert Langereis

► To cite this version:

Matthias Rauterberg, Jun Hu, Geert Langereis. Cultural Computing - How to Investigate a Form of Unconscious User Experiences in Mixed Realities. Second IFIP TC 14 Entertainment Computing Symposium (ECS) / Held as Part of World Computer Congress (WCC), Sep 2010, Brisbane, Australia. pp.190-197, 10.1007/978-3-642-15214-6_19. hal-01056343

HAL Id: hal-01056343

https://inria.hal.science/hal-01056343

Submitted on 18 Aug 2014

HAL is a multi-disciplinary open access
archive for the deposit and dissemination of sci-
entific research documents, whether they are pub-
lished or not. The documents may come from
teaching and research institutions in France or
abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est
destinée au dépôt et à la diffusion de documents
scientifiques de niveau recherche, publiés ou non,
émanant des établissements d'enseignement et de
recherche français ou étrangers, des laboratoires
publics ou privés.

<figure><img src="image_1.png" /></figure>
# Cultural Computing – How to Investigate a Form of Unconscious User Experiences in Mixed Realities

Matthias Rauterberg, Jun Hu, Geert Langereis

Industrial Design, Eindhoven University of Technology, The Netherlands
g.w.m.rauterberg@tue.nl

**Abstract:** This paper presents a new direction of research in user experiences and cognitive science. The problem addressed is drawing on results from different disciplines: psychology, brain and cognitive sciences, physics, and interaction design. As main objective we plan the empirical validation of the claim: Cultural computing as enabling technology for social transformations. Cultural computing is based on a form of cultural translation that uses scientific methods to capture and represent essential aspects of a particular culture. Cultural computing will enable particular cognitive and emotional responses from users as reflections on and of their inner, subliminal consciousness. Cultural computing is not only integrating cultural aspects into the interaction but also allowing the user to experience an interaction that is closely related to the core aspects of his/her own culture. As such it is important to understand one's cultural determinants and how to render them during the interaction. We will address individually and collectively the cultural determinants of the Western culture. Based on the given narrative 'Alice Adventures in Wonderland' we have already built the first demo version and plan to build an optimized version of a mixed reality installation to provide and investigate cultural user experiences. To address the research questions we propose three main research lines: (1) designing a mixed reality environment to provide certain unconscious user experiences; (2) inducing and measuring the changes in the individual's unconscious knowledge structure; and (3) empirical validation of a fundamentally new synchronization mechanism for sharing individual changes collectively. We hypothesize that there eventually exists an interconnecting knowledge field at the foundation of reality that conserves and conveys information collectively. This knowledge field looks like a possible candidate for a required 'supra-natural' memory for cultural knowledge to tap into for social transformations.

**Keywords:** cultural computing, mixed reality, social transformation, user experience, unconsciousness

## Introduction

This paper presents a new direction of research in user experiences and cognitive science [1]. The problem addressed is drawing on results from different disciplines: psychology, brain and cognitive sciences, physics, and interaction design [2]. The main objective is the empirical validation of the claim: Cultural computing as enabling technology for social
transformations [3]. Cultural computing is based on a form of cultural translation that uses scientific methods to capture and represent essential aspects of a culture. Cultural computing will enable particular cognitive and emotional responses from users as reflections on and of their inner, subliminal consciousness [4]. Cultural computing is not only integrating cultural aspects into the interaction but also allowing the user to experience an interaction that is closely related to the core aspects of his/her own culture [5]. Hence it is important to understand one's cultural determinants and how to render them during the interaction [6].

For the Japanese culture the existing ZENetic computer [7] is an innovative tabletop application to address cultural computing in the East (see also [8]). ZENetic computer “brings a kind of ‘awakening’ within users’ ‘information selves,’ and strives to stimulate their unconscious imagination. As the system ‘classifies’ the user’s personality based on the composition of her sansui landscape design, it generates a story for the user, drawing her into an alternate world through the display” [9]. The following three main research lines can be foreseen to address our main research questions: (1) How to design a mixed reality (MR) environment to provide certain unconscious user experiences for Westerners [10-12]; (2) how to induce and measure the changes in the individual’s unconscious knowledge structure due to these unusual experiences triggered by cultural computing [13]; and (3) which synchronization mechanism exist to share these individual changes collectively? [14-15]

Designing a mixed reality environment for subconscious user experiences

Assuming the intended individual unconscious effects might be small [16-17] we have to
achieve a maximum in user immersion/presence and therefore a high impact on the user's
unconscious knowledge structure. The amount of user's immersion is influenced by several
factors: the amount of sensory involvement [18], the richness of the narrative [19], amount of
engagement through action [20], and perceived realism of the situation engaged [21]. Hence
a MR installation is the best choice for an experimental test bed. MR installations are close to
a realistic environment, provides sufficient experimental control and a maximum of user
engagement through physical actions. But which narrative as design concept for such MR is
most appropriate? The narrative should touch on deep cultural dimensions, should be widely
accepted, and should provide unusual user experiences. After rigorous selection the narrative
of ‘Alice adventures in wonderland’ [22] was chosen. Several scenes of this narrative are
already partially realized in the ALICE installation at TU/e [23]. This MR installation is 10
meter wide, 10 meter deep and 7 meter high. Inside this cube are following stages foreseen:
.stage-1) in the park (investigating the relationship of boredom/curiosity and behavior [24]),
.stage-2) down the rabbit hole (investigating the cognitive mechanisms for perceived
.movement), (3) 5-side virtual reality cave with ‘eat me’ and ‘drink me’ (investigating the
cognitive mechanisms for self-perceived body size and body image [18, 25]), (4) pool of
tears (investigating the effects of ‘water’ on emotions), (5) encounter with caterpillar
(investigating the self-concept and ego [26]), and (6) discussion with Cheshire cat
(investigating the relation between logic and emotion [27]). So far the achieved state of our
MR installation was sufficient for demonstrations and small experiments, but a full
experience over all stages for experiments still needs developments in technology and
measurement procedures.

Measuring user's effects on his/her unconscious knowledge

Recent investigation in cognitive psychology of consciousness and unconsciousness are promising. Most of the brain's energy consumption is not used for processing responses to external stimuli; but what is this brain energy then for? [28-29]. One possible answer is for subconscious information processing to forcast the future. Dijksterhuis, Bos, Nordgren, and van Baaren [29-30] could show the advantages of the unconscious information processing, and therefore they open a door to a new view on the relationship towards unconsciousness [31-33]. The possible effects of extreme user experiences on their cognitive structure can be addressed as follows: (1) through psychological measures (e.g. Implicit Association Test, [34]), (2) through unobtrusive observation via cameras of user's shown behaviour (incl. facial expressions [35]), (3) through physiological sensors on the human body or in the neighbourhood [36-37], and (4) measuring emotionally influenced quantum effects through dedicated hardware (i.e. Random Event Generator - REG [38-39]). The Global Consciousness Project [40-42] builds on astonishing experiments conducted over the past years, demonstrating that human consciousness interacts with REGs, so that they produce non-random patterns. For measuring unconscious processes, it is ultimately important not to affect the person under test by the infringement of observing hardware. Therefore, we would like to implement the system with option (4) rather than option (3).

The Implicit Association Test (IAT) focuses only on the implicit (unconscious) part of
cognition [34, 43]. The IAT measures differential (unconscious) association of two target
concepts with an attribute. The two concepts appear in a binary choice task (e.g., *self* vs.
*others* names), and the attribute in a second task (e.g., *pleasant* vs. *unpleasant* words for an
evaluation attribute). When instructions oblige highly associated categories (e.g., *self* +
*pleasant*) to share a response key, performance is faster than when less associated categories
(e.g., *others* + *pleasant*) share a key. This response time performance difference implicitly
measures differential association of the two concepts with the attribute. We used already the
IAT version that especially measures self-concept [26]. The IAT seems to be a useful
measure for the effects of unconscious experiences.

One of the research questions is to which extent humans can unconsciously process
semantically rich information [33, 44]. Although the state of art is controversial, we assume
that in rich settings (see R1) unconscious processing takes place. The challeng is how to
measure these cognitive processes in an unobtrusive manner, e.g. compared to fMRI scans
[17, 45]; portable Electroencephalography (EEG) and Random Event Generator (REG)
might be an option.
Three different REGs are used<fnref n="1" />. The REG electronics produces ‘white noise’ based on completely unpredictable quantum fluctuations. PEAR-REG is based on Johnson noise (extremely low-level fluctuations in electron flow in a resistor due to thermal influences). It has a built-in logic transformation (XOR) of every other bit from 1 to 0 or vice versa for eliminating any bias of the mean output of the device. The MICRO-REG utilizes on the Field-Effect Transistor for the ‘white noise’. The ORION-REG uses two diodes which independently produce a random bitstream each. Nelson et al. [39] found empirical evidence for a replicable effect on data from a global network of REGs that is correlated with designated periods of intense collective human activity or engagement (e.g. September 11, 2001), but not with any other physical sources of influence.

Investigation of synchronization mechanism for disseminating changes

If we acknowledge that people’s individual norms may deviate from an established collective
norm, than we should identify a way to address these relationships [46]. Culture can only
exist on the collective level, beyond the individual. The cultural determinates are mainly
unconscious [12]: Culture is always ambient to individuals as carrier of cultural knowledge
and collective as a synchronisation mechanism for the behaviour of these individuals. The
following metaphor helps to describe this double nature of culture: ‘the drop is in the ocean,
but the ocean is in the drop as well’. Jung [47] has seen this with remarkable clarity and
provides a framework to describe the knowledge structure of the collective unconsciousness.
But what might this ‘collective unconscious’ be? A possible answer can be drawn from
quantum physics.

Quantum physics describes the properties of matter and energy at near atomic scale [17]: (1) Quantum coherence (individual particles yield identity to a collective, unifying wave function). (2) Non-local quantum entanglement (spatially separated particle states are nonetheless connected or related [48-50]). (3) Quantum superposition (particles exist in two or more states or locations simultaneously). (4) Quantum state reduction or ‘collapse of the wave function’ (super positioned particles reduce or collapse to specific choices) [51]. Of particular interest is the non-local quantum entanglement that shows there might be an interconnecting layer in nature, beyond the forces we are familiar with [50]. Information might be stored in some yet unknown way, and we might tap into this knowledge field, or exchange information with other minds, if the necessary conditions of ‘sympathetic resonance’ exist (see also the ‘morphic resonance’ concept of Sheldrake [52], and the Global Conscious Project [39]). Now the remaining question is how can a cognitive system interact with such kind of knowledge field? [53].

The Penrose–Hameroff ‘OrchOR’ model portrays consciousness as quantum computation
in microtubules which collapse or reduce by an objective factor related to quantum gravity.
Penrose [54-55] proposed a quantum gravity mechanism that following periods of

<footnote n="1">¹ See at http://www.psyleron.com/reg1.aspx</footnote>
preconscious quantum computation tubulin super-positions self-collapse at an objective threshold. During classical phases protein connections associated with microtubules provide input. The basic ideas are [56, p. 160-161]: (1) Conformational states of individual tubulins within neuronal microtubules are determined by quantum mechanical London forces within the tubulin interiors which can induce conformational quantum superposition; (2) in superposition tubulins 'communicate' with entangled tubulins in the same microtubule, and in other microtubules in the same neuron; (3) quantum states in microtubules in any given neuron may extend to microtubules in neighboring neurons, and through macroscopic regions of brain via tunneling through gap junctions; (4) quantum states of tubulin, resp. microtubules are isolated/protected from environmental decoherence by biological mechanisms which include phases of actin gelation, ordered water, coherent pumping and topological quantum error correction; (5) microtubule quantum computations, resp. super-positions are tuned or 'orchestrated' (Orch) by microtubule-associated proteins during a classical, liquid phase which with a quantum, solid state phase; (6) following periods of pre-conscious quantum computation (e.g. on the order of tens to hundreds of milliseconds) tubulin superpositions reduce or collapse by Penrose quantum gravity 'objective reduction' (OR). The classical output states which result from the OR process are chosen non-algorithmically ('non-computably') and then govern neurophysiologic events by binding of microtubule-associated proteins, regulating synapses and membrane functions etc.; (7) the reduction or 'self-collapse' in the orchestrated objective reduction 'OrchOR' model is suggested to be a 'conscious moment', (see Penrose's quantum gravity mechanism which ties the process to fundamental space-time geometry). According OrchOR model [57], quantum computation occurs in microtubules inside neurons. In the OrchOR model microtubules exist transiently in quantum superposition of two or more conformational states. Each OrchOR quantum computation determines classical output states of tubulin to establish starting conditions for the next conscious event [57-59].

We hypothesize that there eventually exists an interconnecting knowledge field at the foundation of our quantum reality that conserves and conveys information collectively. This knowledge field looks like a possible candidate for a required 'supra-natural' memory for cultural knowledge to tap into for social transformations [60].

## References

1. Nakatsu, R. and M. Rauterberg, *Entertainment computing: Inaugural Editorial*. Entertainment Computing, 2009. **1**(1): p. 1-7.

2. Nijholt, A., D.P.-O. Bos, and B. Reuderink, *Turning shortcomings into challenges: Brain-computer interfaces for games*. Entertainment Computing, 2009. **1**(2): p. 85-94.

3. Tosa, N., S. Matsuoka, and H. Thomas, *Inter-culture computing: ZENetic computer*, in *ACM SIGGRAPH 2004 Emerging technologies*, H. Elliott-Famularo, Editor. 2004, ACM: Los Angeles. p. 11-11.
4. Tosa, N., *Expression of emotion, unconsciousness with art and technology*, in *Affective minds*, G. Hatano, N. Okada, and H. Tanabe, Editors. 2000, Elsevier: Amsterdam New York. p. 183-205.

5. Hu, J. and C. Bartneck, *Culture matters: A study on presence in an interactive movie*. Cyberpsychology and Behavior, 2008. **11**(5): p. 529-536.

6. Tosa, N., *Unconscious flow*. Leonardo, 2000. **33**(5): p. 442-442.

7. Tosa, N. and S. Matsuoka, *Recreating our selves: ZENetic computer*, in *International Conference on Information Visualisation - IV'03*. 2003, IEEE Computer Society: Los Alamitos. p. 614-618.

8. Cheok, A.D., et al., *Confucius computer: Transforming the future through ancient philosophy*, in *ACM SIGGRAPH 2008 new tech demos*. 2008, ACM: Los Angeles, California.

9. Tosa, N., *Cultural Computing: ZENetic Computer*, in *Intelligent Agent 6.1.1*. 2006: Kyoto.

10. Nisbett, R.E., et al., *Culture and systems of thought: Holistic versus analytic cognition*. Psychological Review, 2001. **108**(2): p. 291-310.

11. Nisbett, R.E. and T. Masuda, *Culture and point of view*. Proceedings National Academy of Sciences, 2003. **100**(19): p. 11163-11170.

12. Rauterberg, M., *Ambient culture: A possible future for entertainment computing*, in *Interactive TV: a shared experience - Adjunct Proceedings of EuroITV-2007*, A. Lugmayr and P. Golebiowski, Editors. 2007, Tampere International Center for Signal Processing, Tampere, Finland. p. 37-39.

13. Rauterberg, M., *How to assess the user's experience in cultural computing*, in *Usability Professionals*, T. Bosenick, et al., Editors. 2006, Fraunhofer Informationszentrum Raum und Bau: Gelsenkirchen. p. 12-17.

14. Rauterberg, M., *Usability in the future – explicit and implicit effects in cultural computing*, in *Mensch und Computer im Strukturwandel*, A.M. Heinecke and H. Paul, Editors. 2006, Oldenbourg: München. p. 29-36.

15. Peng, K., R.E. Nisbett, and N.Y.C. Wong, *Validity problems comparing values across cultures and possible solutions*. Psychological Methods, 1997. **2**(4): p. 329-344.

16. Merikle, P.M. and M. Daneman, *Memory for unconsciously perceived events: Evidence from anesthetized patients*. Consciousness and Cognition, 1996. **5**: p. 525-541.

17. Noort, M.v.d., K. Hugdahl, and P. Bosch, *Human machine interaction: The special role for human unconscious emotional information processing*, in *Proceedings of First International Conference on Affective Computing & Intelligent Interaction - ACII 2005*, J. Tao, T. Tan, and R.W. Picard, Editors. 2005, Springer: Berlin Heidelberg New York. p. 598-605.

18. Drascic, D. and P. Milgram, *Perceptual issues in augmented reality*, in *Proceedings of SPIE - Stereoscopic Displays and Virtual Reality Systems III*. 1996, The International Society for Optical Engineering: Bellingham, Washington. p. 123-134.

19. Qin, H., P.-L. Rau, and G. Salvendy, *Player immersion in the computer game narrative*, in *Proceedings of International Conference on Entertainment Computing - ICEC*, L. Ma,
R. Nakatsu, and M. Rauterberg, Editors. 2007, Springer: Berlin Heidelberg New York. p. 458-461.

20. Milgram, P. and A.F. Kishino, *Taxonomy of mixed reality visual displays*. IEICE Transactions on Information and Systems, 1994. E77-D(12): p. 1321-1329.

21. Busselle, R., A. Ryabovolova, and B. Wilson, *Ruining a good story: Cultivation, perceived realism and narrative*. Communications, 2004. 29(3): p. 365-378.

22. Carroll, L., *Alice's adventures in wonderland*. 1865, London: Macmillan.

23. ALICE. *Project ALICE: Not just another interactive installation*. 2010; Available from: http://www.alice.id.tue.nl/.

24. Aart van, J., et al., *Designing for experience: Arousing boredom to evoke predefined user behaviour*, in *Proceedings of the 6th Design & Emotion Conference*, P. Desmet, et al., Editors. 2008: Hong Kong. p. 1607-1611.

25. Schilder, P., *The image and appearance of the human body*. 1950, New York: International University Press.

26. Kooijmans, T. and M. Rauterberg, *Cultural computing and the self concept: Towards unconscious metamorphosis*, in *Entertainment Computing - ICEC 2007*, L. Ma, R. Nakatsu, and M. Rauterberg, Editors. 2007, Springer: Berlin Heidelberg New York. p. 171-181.

27. Sobol-Shikler, T. and P. Robinson, *Classification of complex information: Inference of co-occuring affective states from their expressions in speech*. IEEE Transactions on Pattern Analysis and Machine Intelligence, in press.

28. Raichle, M.E., *The brain's dark energy*. Science, 2006. 314 (24 November 2006): p. 1249-1250.

29. Raichle, M.E., *A brief history of human brain mapping*. Trends in Neurosciences, 2008. 32 (2): p. 118-126.

30. Dijksterhuis, A., et al., *On making the right choice: the deliberation-without-attention effect*. Science, 2006. 311: p. 1005-1007.

31. Nakatsu, R., M. Rauterberg, and B. Salem, *Forms and theories of communication: From multimedia to Kansei mediation*. Multimedia Systems, 2006. 11(3): p. 304-312.

32. Dijksterhuis, A. and L. Nordgren, *A theory of unconscious thought*. Perspectives on Psychology, 2006. 1(2): p. 95-109.

33. Dehaene, S., et al., *Imaging unconscious semantic priming*. Nature, 1998. 395(8 October 1998): p. 597-600.

34. Greenwald, A.G. and S.D. Farnham, *Using the Implicit Association Test to measure self-esteem and self-concept*. Journal of Personality and Social Psychology, 2000. 79(6): p. 1022-1038.

35. Barakova, E. and T. Laurens, *Expressing and interpreting emotional movements in social games with robots*. Personal and Ubiquitous Computing, 2010. online (16 January 2010).

36. Taelman, J., et al., *Contactless EMG sensors for continuous monitoring of muscle activity to prevent musculoskeletal disorders*, in *Proceedings of the first Annual Symposium of the IEEE-EMBS Benelux Chapter*. 2006, IEEE Benelux chapter on Engineering in Medicine and Biology Society: Brussels. p. 223-226.
37. Ouwerkerk, M., F. Pasveer, and G. Langereis, *Unobtrusive sensing of psychophysiological parameters*, in *Probing experience: From assessment of user emotions and behaviour to development of products*, J. Westerink, et al., Editors. 2008, Springer: Dordrecht. p. 163-193.

38. REG. *Random event generator design*. 2010; Available from: http://noosphere.princeton.edu/reg.html.

39. Nelson, R.D., et al., *Correlations of continuous random data with major world events*. Foundations of Physics Letters, 2002. **15**(6): p. 537-550.

40. Nelson, R., *The Global Consciousness Project - Part 2*. The Golden Thread, 2002. (August 2002): p. 6-10.

41. Nelson, R., *The Global Consciousness Project - Part 3*. The Golden Thread, 2002. (November 2002): p. 30-32.

42. Nelson, R., *The Global Consciousness Project - Part 1*. The Golden Thread, 2002. (May 2002): p. 8-12.

43. Greenwald, A.G., D.E. McGhee, and J.K.L. Schwartz, *Measuring individual differences in implicit cognition: The Implicit Association Test*. Journal of Personality and Social Psychology, 1998. **74**(6): p. 1464-1480.

44. Naccache, L. and S. Dehaene, *Unconscious semantic priming extends to novel unseen stimuli*. Cognition, 2001. **80**: p. 215-229.

45. Noort, M.v.d., P. Bosch, and K. Hugdahl, *Understanding the unconscious brain: Evidence for a non-linear information processing, in Proceedings of the International Conference on Cognitive Systems*. 2005. p. 2254-2258.

46. DiMaggio, P., *Culture and cognition*. Annual Review of Sociology, 1997. **23**: p. 263-287.

47. Jung, C.G., *Die Archetypen und das kollektive Unbewußte*, in *Gesammelte Werke von C.G. Jung*. 1934, Walter.

48. Einstein, A., B. Podolsky, and N. Rosen, *Can quantum-mechanical description of physical reality be considered complete?* Physical Review, 1935. **47**(10): p. 777-780.

49. Aspect, A., P. Grangier, and G. Roger, *Experimental realization of Einstein-Podolsky-Rosen-Bohm Gedankenexperiment: a new violation of Bell's inequalities*. Physical Review Letters, 1982. **49**(2): p. 91-94.

50. Aspect, A., *Quantum mechanics: to be or not to be local*. Nature, 2007. **446**: p. 866-867.

51. Woolf, N.J. and S.R. Hameroff, *A quantum approach to visual consciousness*. Trends in Cognitive Sciences, 2001. **5**(11): p. 472-478.

52. Sheldrake, R., *New science of life: The hypothesis of morphic resonance*. 1981: Blond & Briggs.

53. Noort, M.v.d., *The unconscious brain - the relative time and information theory of emotions*. 2003, Oegstgeest: Citadel.

54. Penrose, R., *The emperor's new mind*. 1989, Oxford: Oxford University Press.

55. Penrose, R., *Shadows of the mind: A search for the missing science of consciousness*. 1994, Oxford: Oxford University Press.
56. Hameroff, S., et al., *Conduction pathways in microtubules, biological quantum computation, and consciousness*. BioSystems, 2002. 64: p. 149-168.

57. Hameroff, S.R., *Quantum computation in brain microtubules? The Penrose-Hameroff 'OrchOR' model of consciousness*. Philosophical Transactions Royal Society London A, 1998. 356 p. 1869-1896.

58. Penrose, R. and S.R. Hameroff, *What gaps? Reply to Grush and Churchland*. Journal of Consciousness Studies, 1995. 2: p. 98-112.

59. Hameroff, S.R. and R. Penrose, *Conscious events as orchestrated space-time selections*. NeuroQuantology, 2003. 1: p. 10-35.

60. Rauterberg, M., *Entertainment computing, social transformation and the quantum field, in Intelligent Technologies for Interactive Entertainment - INTETAIN 2009*, A. Nijholt, D. Reidsma, and H. Hondorp, Editors. 2009, Springer: Berlin Heidelberg. p. 1-8.