# Brain-Computer Interface Games: Towards a Framework

Hayrettin Gürkök, Anton Nijholt, Mannes Poel

► To cite this version:

Hayrettin Gürkök, Anton Nijholt, Mannes Poel. Brain-Computer Interface Games: Towards a Framework. 11th International Conference on Entertainment Computing (ICEC), Sep 2012, Bremen, Germany. pp.373-380, 10.1007/978-3-642-33542-6_33. hal-01556145

HAL Id: hal-01556145

https://inria.hal.science/hal-01556145

Submitted on 4 Jul 2017

HAL is a multi-disciplinary open access
archive for the deposit and dissemination of sci-
entific research documents, whether they are pub-
lished or not. The documents may come from
teaching and research institutions in France or
abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est
destinée au dépôt et à la diffusion de documents
scientifiques de niveau recherche, publiés ou non,
émanant des établissements d'enseignement et de
recherche français ou étrangers, des laboratoires
publics ou privés.

<figure><img src="image_1.png" /></figure>
# Brain-Computer Interface Games:
## Towards a Framework

Hayrettin Gürkök, Anton Nijholt, and Mannes Poel

Human Media Interaction, University of Twente, Enschede, The Netherlands

`{h.gurkok,a.nijholt,m-poel}@utwente.nl`

**Abstract.** The brain-computer interface (BCI) community started to consider games as potential applications while the games community started to consider BCI as a game controller. However, there is a discrepancy between the BCI games developed by the two communities. In this paper, we propose a preliminary BCI games framework that we constructed with respect to the research conducted in both the BCI and the games communities. Developers can situate their BCI games within this framework and benefit from the guidelines we provide and also extend the framework further.

**Keywords:** Brain-computer interface, games, flow, presence, challenge, fantasy, sociality, concentration

# 1 Introduction

A brain-computer interface (BCI) is an input modality that can infer certain actions, intentions and psychological (e.g. cognitive, emotional) states by analysing the brain activity it captures. Besides its classical purpose of redressing the communication and mobility of disabled people, BCI has been proposed as a candidate modality for a range of recreational HCI applications to be used by the general population [7]. Among these, BCI games [17] attract the interest of researchers and developers from both BCI and games communities. However, we see a discrepancy between the BCI games developed by the two communities.

Many of the BCI games developed by the BCI community aim at testing some psychological hypotheses or evaluating the performance of signal analysis and classification techniques. Thus, less attention is paid to game characteristics than to technical aspects. These games do not usually have any narrative or rich feedback or visuals. User (i.e. player) experience evaluations are almost never carried out. This leads to BCI games that are reliable but often not enjoyable. On the contrary, BCI games from the games community are developed with respect to game design principles. However, the neurophysiology and signal analysis techniques they rely on are largely unknown. Because, these games mostly make use of the commercial BCI headsets which have their private technical details. This leads to BCI games that are potentially entertaining but unsatisfactory in terms of feeling of control.
In this paper, we will try to transfer some knowledge from the games and the BCI communities into a shared preliminary framework to make them aware of each other's research. From the games community, we will show some game playing motivations which can be satisfied by the features of BCI. From the BCI community, we will take the current interaction paradigms used in general and show the ways they can be used in games. This way, we hope to contribute to bridging the gap between the two communities and promoting the development of entertaining and reliable BCI games.

# 2 A Framework of BCI Games

In the framework that we propose, we represent a BCI game using two descriptors. The first descriptor specifies the player motivation(s) the BCI game satisfies. In other words, it answers the question: ‘Why is the game played?’. The second descriptor specifies the interaction paradigm(s) the BCI game is built upon. So, it answers the question: ‘How is the game played?’. Next, we will elaborate on these descriptors and provide some guidelines for each. For the former descriptor, we will mainly benefit from research in the games community while for the latter in the BCI community. Finally, we will discuss the relation between the two descriptors.

## 2.1 Motivations Satisfied by the BCI Game

People play games which potentially fulfill their psychological needs. We see a correspondence between some psychological needs [20] and some game playing motivations [19], such as competence and challenge or relatedness and socialisation. In this section, we will provide some example game playing motivations (or needs) in which BCI can make a difference and discuss the ways to make the difference.

**Challenge** Challenge is one of the elements of flow, which is the optimal experience for any activity and described as “*so gratifying that people are willing to do it for its own sake, with little concern for what they will get out of it, even when it is difficult, or dangerous*” [4]. Sweetser and Wyeth [22] proposed a model describing which elements a game should have in order to provide flow. Their model suggests that a game should offer challenges matching player skills and both must exceed a certain threshold.

BCI is an imperfect recognition technology and this is partly because there are no guidelines nor ground truth (e.g. an internal feedback mechanism such as a human sense) for generating particular mental activity. But, people can manipulate and learn to improve their voluntary mental actions as well as involuntary reactions as they keep interacting with a BCI that provides accurate feedback [24]. So, BCI game players need to show continuous effort and even repeat their actions until they are understood by BCI. Not only the successful end result leads to a positive affect but also the purposeful repetition brings fun [1].
**Fantasy** Games let players do things that they cannot do -at least safely or without being criticised- in real life, such as flying or smashing cars. However, in a virtual world, it is not trivial to provide the very same sensation resulting from doing something in the real world. Such a sensation is known as presence and defined as "*the perception in which even though part or all of an individual's current experience is generated by and/or filtered through human-made technology, part or all of the individual's perception fails to accurately acknowledge the role of the technology in the experience*" [9]. Riva [18] claims that rather than our perception, it is our chain of actions that create the presence. He explains that a user "*is more present in a perceptually poor virtual environment ... where he/she can act in many different ways than in a real-like virtual environment where he/she cannot do anything.*" In achieving this, it is essential to have a one-to-one correspondence between player actions (as well as reactions) in the real world and those in the game world, as provided by the Kinect.

We take the claim of Riva further and propose that ‘to act’ is not our ultimate goal. We actually aim ‘to be’ in the virtual world and to act is one way of satisfying our aim. So, we are more present in a virtual world in which we can represent ourselves more. We would like to be represented with our emotions and thoughts. In cases such as these, BCI can provide a translation between the psychological state of the player in the real world and the dynamics of the game world, just as Kinect provides correspondence between real-world and game-world actions. So, the additional inner state information can strengthen the feeling of presence.

**Sociality** Some people enjoy playing computer games with other people. They play not necessarily for the challenge but just to be with others. They enjoy spending time with friends, seeing their reactions and expressions, and gloating or feeling proud upon winning [22]. Any multi-player version of a BCI game can provide such an interactive environment. Players may cooperate or compete using BCI or they can share their experiences, such as difficulties or enjoyment with control, while playing the game. These are, of course, not specific to BCI games. But, there are other ways in which BCI can provide sociality and which cannot be replicated easily or at all by other controllers.

Many social actions are related to expressing and perceiving emotions. Previous studies have shown that communication of heartbeat, which is a reflection of emotional activity, can improve the co-presence [3] and intimacy [10] of players. Heartbeat is certainly not the only nor the best indicator of emotion. BCI can recognise certain psychological states and let us share them. Since involuntary brain activity, such as emotional response, is not easily controllable, BCI can provide objective information about our emotional state. For this reason, BCI can also be used in game situations where players would like to hide their psychological states from each other. For example, in a bluffing game, players can restrict their bodily movements and to some extent even their physiological activity but not their brain activity. So, BCI can be used for emotion-awareness or, more generally, psychological awareness in two opposite game logics.
## 2.2 Interaction Paradigms Used by the BCI Game

BCI applications rely on brain signals originating from player actions and reactions to events. We call these actions and reactions interaction paradigms and collect them under three categories: mental state regulation, movement imagery and evoked response generation.

**Mental State Regulation** Mental state games are usually played via two activities: relaxing or concentrating. Most of the mental state games let players move objects [8] but there are other uses such as changing the game avatar [17].

Relaxing is a preferable activity in a game as it leads to a positive affective state that players would like to reach while playing games [12]. Therefore, even if the game environment is not an affective one, people may play such games for the end effect of being relaxed. Moreover, they might easily refer their acquaintances and even children to play such games. Concentration is also a preferable game activity due to its absorbing effect. According to the flow [4, 22] and immersion theories [2], concentration is the key to successful games. Therefore, games requiring concentration or paying attention, which is one of the activities leading to concentration, ought to provide a positive play experience.

Mental state regulation games should either be slow paced or in these games BCI should be used as an auxiliary controller along with a primary controller which is faster than BCI. Because, the speed with which we can change our state of relaxedness or concentration is much slower than the speed with which we can press buttons or use any other modality. Mental state games usually allow only binary control. For example, in a relaxation game, players can either be relaxed or not relaxed so they can communicate a maximum of two discrete commands. It is possible to fit a continuous scale between these two states but validating such a scale is non-trivial. Therefore, mental state regulation is less suitable for games that require large numbers of distinct commands.

**Movement Imagery** Movement imagery games require no physical movement but imagery of limb movements, mostly the hands, fingers or feet. Players imagine movements to navigate [11] or to make selections [23]. To provide intuitive interaction, the mapping of imaginary movements to game commands should be coherent. For example, grasping an object at the left or right hand side can be matched to left or right hand imagery while walking can be matched to foot imagery. Such intuitive mappings can create the illusion that the game is recognising player's actions, even before they move.

Movement imagery can be recognised quickly, without needing to average the signal [23]. Therefore, movement imagery games are suitable for fast interaction. On the contrary, the number of commands in these games is limited to the number of distinguishable imaginary actions players can perform. Using other modalities in combination with BCI can increase the number of commands. However, the movements made to control other modalities, such as pushing a button or speaking, might contaminate the movement imagery signal. Especially when the signal is not averaged, the signal-to-noise ratio (SNR) is very low.
**Evoked Response Generation** This class of games is dominated by steady-state visually evoked potential (SSVEP) games, accompanied rarely by P300 games (e.g. [6]). SSVEP is a brain response to flickering light or images. When we observe visual stimulus, say an image, that is constantly re-appearing at a frequency of *f* then the amplitude of the signals measured from our visual cortex are enhanced at frequency *f* and its harmonics (2*f*, 3*f*, and so on).

One way of using SSVEP is to map the strength (amplitude) of SSVEP that is evoked by single stimulus to game actions. For example, a weak SSVEP can steer a virtual plane to the left while a strong one to the right [15]. Research has shown that sustained attention can strengthen SSVEP [5]. Sustained attention is an activity which can lead to a state of concentration [14]. This makes SSVEP suitable for concentration games. Another, and the more popular, approach is to use multiple stimuli each of which is associated with a command. With this approach, BCI is usually used to select a direction, for example to steer a racing car [13]. With this approach, a greater number of commands can be issued.

Evoked response generation is less suitable for fast games due to the signal averaging process, which requires signals to accumulate for some time. But they are suitable for multimodality thanks to their high SNR. The number of distinct commands in evoked potential games depends only on the number of stimuli. So, as long as the stimulation space is large enough to accommodate, (finitely) many stimuli can be presented to the player. But, a computer screen is a limited space so the number of stimuli that can be placed on the screen is also limited.

## 2.3 The Complete Picture

In this section, we will discuss the relation between the two descriptors of our framework. Specifically, we will discuss which interaction paradigms can satisfy which player motivations and in which ways? Where possible, we will mention existing or hypothetical BCI games supporting our discussion.

<figure><img src="image_2.png" /><figcaption>**Fig. 1.** Diagram showing the set of player motivations and example BCI games. The italic, underlined and bold fonts denote mental state regulation, movement imagery and evoked response generation games respectively.</figcaption></figure>

As any game can do, a BCI game can satisfy more than one set of player motivations (the first descriptor) at the same time. It might not satisfy any player
motivations, for example if it is just an experimental game. This is illustrated by the set diagrams in Figure 1. A BCI game can make use of multiple interaction paradigms (the second descriptor). The font stylings of the games in Figure 1 indicate the different interaction paradigms and their combinations. For example in the game Bacteria Hunt [16] the players chase fleeing bacteria by controlling an amoeba. It is a mental state regulation game because when the players are relaxed, the bacteria flee more slowly. It is also an evoked response game because when the amoeba catches a bacterium the players concentrate on a flickering circle to eat the bacterium. While there is no restriction on using any interaction paradigm to satisfy any player motivation, there might be preferred and non-preferred matchings.

A BCI game can offer a positively affective challenge if the game can hide BCI recognition errors under the game tasks or story. The game should provide the players with the illusion that when they encounter an error, this is not simply a technological fault of BCI but rather an expected situation given the player tasks and game environment. This way, they should be motivated to repeat the tasks until they learn how to generate the desired brain activity. In the mental state regulation game alpha World of Warcraft (aWoW) [17], the players play a druid that transforms from a bear to a humanoid when the players are relaxed and back again when they are stressed. The level of relaxedness is determined according to the alpha band power. But the players always think of the high level tasks of getting relaxed or stressed to transform. There might be times when players cannot transform desirably in the game. Especially for players who are familiar with the game World of Warcraft or similar games, this is not an unexpected situation. These players would consider that it should take some effort and time for a druid to transform. So, they would keep trying to find the right strategy to transform. The same principle may be applied to movement imaginary games. For example, in a game (let us call it Lift Stone) the players might lift a stone by imagining arm movements. Players familiar with the phenomenon of telekinesis would find this mapping (imagining arm movement and lifting) an intuitive one and consider that it should take some effort to lift something from far. For evoked potential games, due to the explicit stimuli, offering meaningful challenges is more difficult though not impossible. For example in a game (let us call it Kill Monster) players might try to kill a monster by concentrating on its heart which flickers like an SSVEP stimulus. People know that, though it does not flicker, the heart beats and it is the organ of vitality. So, they may accept that killing a monster should require some effort. In contrast, in the Bacteria Hunt game, the flickering circle is not related to bacteria so provides no motivation to concentrate on the stimulus.

To provide fantasy, a BCI game should incorporate additional, inner state information from the players to improve their sense of presence in the virtual world. In the game World of Warcraft, players play the role of a druid. So, the idea is that the players should put themselves in the place of their avatar. The original game translates players actions (e.g. pressing the key W) into game actions (e.g. moving the avatar forward) but it cannot go beyond that. In aWoW,
players are represented not only by their actions but also by their psychologi-
cal states. This way, they can feel more present in the game world. Movement
imaginary games cannot convey psychological state information but they can
represent covert player intentions. For example in the hypothetical Lift Stone
game we mentioned in the previous paragraph, the players might feel as if the
game is understanding their intention without having them move.

In a BCI game, sociality might emerge from the explicit social behaviours
(e.g. vocalisations, gestures) of co-players while they are competing against or
cooperating with each other. In this sense, any interaction paradigm is suitable
for use. For example in the game Brainball [8], in which co-players regulate their
mental states to roll a ball on the table toward each other, players might, for
instance, talk to tease each other or generate facial expressions to manifest the
difficulty they are experiencing. Sociality might also be formed through commu-
nication of implicit player states. For this sort of sociality, mental state regulation
is the natural paradigm of choice. For example, in the game Brainwave Drawing
[21], co-players try to synchronise their mental states in terms of their brain
signals in different frequency bands.

3 Conclusion

In this paper, we proposed a preliminary framework for BCI games formed partly
by research in the BCI community and partly by research in the games com-
munity. We represented a BCI game with two descriptor. One of these specifies
which player motivation(s) the BCI game can satisfy. We mentioned three exam-
ple player motivations which are challenge, fantasy and sociality. We showed the
ways BCI games can satisfy each of these motivations and provided some guide-
lines for them. The other descriptor specifies which interaction paradigm(s) the
game is built upon. We described three types of interaction paradigms which are
mental state regulation, movement imagery and evoked response generation. We
relied on literature on BCI and games research while constructing our categories.
But there may be many more player motivations that a BCI game can fulfill or
interaction paradigms it can make use of. So, obviously, developers should not
feel restricted by our categorisations while developing their BCI games. On the
contrary, they should investigate the alternative categories.

**Acknowledgments** The authors gratefully acknowledge the support of the BrainGain Smart Mix Programme of the Netherlands Ministry of Economic Affairs and the Netherlands Ministry of Education, Culture and Science. They would also like to thank L.E. Packwood for her help in improving the language of the article.

References

1. Blythe, M., Hassenzahl, M.: The semantics of fun: Differentiating enjoyable experiences. In: Funology. Kluwer Academic Publishers (2003) 91–100
2. Brown, E., Cairns, P.: A grounded investigation of game immersion. In: CHI '04 Extended Abstracts. ACM (2004) 1297–1300

3. Chanel, G., Pelli, S., Ravaja, N., Kuikkaniemi, K.: Social interaction using mobile devices and biofeedback (2010) Presented at BioS-Play 2010.

4. Csíkszentmihályi, M.: Flow. Harper Perennial (1990)

5. di Russo, F., Spinelli, D.: Effects of sustained, voluntary attention on amplitude and latency of steady-state visual evoked potential. Clinical Neurophysiology 113(11) (2002) 1771–1777

6. Finke, A., Lenhardt, A., Ritter, H.: The MindGame: A P300-based brain-computer interface game. Neural Networks 22(9) (2009) 1329–1333

7. Gürkök, H., Nijholt, A.: Brain-computer interfaces for multimodal interaction. International Journal of Human-Computer Interaction 28(5) (2012) 292–307

8. Hjelm, S.I.: Research+design: the making of Brainball. interactions 10(1) (2003) 26–34

9. International Society for Presence Research: The Concept of Presence: Explication Statement, http://ispr.info/about-presence-2/about-presence/ (2000)

10. Janssen, J.H., Bailenson, J.N., IJsselsteijn, W.A., Westerink, J.H.D.M.: Intimate heartbeats. IEEE Transactions on Affective Computing 1(2) (2010) 72–80

11. Krepki, R., Blankertz, B., Curio, G., Müller, K.R.: The Berlin Brain-Computer Interface (BBCI). Multimedia Tools and Applications 33(1) (2007) 73–90

12. Lazzaro, N.: Why we play games: Four keys to more emotion without story, http://www.xeodesign.com/xeodesign_whyweplaygames.pdf (March 2004)

13. Martinez, P., Bakardjian, H., Cichocki, A.: Fully online multicommand brain-computer interface with visual neurofeedback using SSVEP paradigm. Computational Intelligence and Neuroscience 2007 (2007) 94561

14. Mateer, C.A., Sohlberg, M.M., eds.: Cognitive Rehabilitation: An Integrative Neuropsychological Approach. 2nd edn. The Guilford Press (2001)

15. Middendorf, M., McMillan, G., Calhoun, G., Jones, K.: Brain-computer interfaces based on the steady-state visual-evoked response. IEEE Transactions on Rehabilitation Engineering 8(2) (2000) 211–214

16. Mühl, C., Gürkök, H., Plass-Oude Bos, D., Thurlings, M., Scherffig, L., Duvinage, M., Elbakyan, A., Kang, S., Poel, M., Heylen, D.: Bacteria hunt. Journal on Multimodal User Interfaces 4(1) (2010) 11–25

17. Plass-Oude Bos, D., Reuderink, B., van de Laar, B., Gürkök, H., Mühl, C., Poel, M., Nijholt, A., Heylen, D.: Brain-computer interfacing and games. In: Brain-Computer Interfaces. Springer-Verlag (2010) 149–178

18. Riva, G.: Is presence a technology issue? Virtual Reality 13(3) (2009) 159–169

19. Rouse, R.: Game Design: Theory & Practice. 2nd edn. Wordware (2005)

20. Sheldon, K.M., Elliot, A.J., Kim, Y., Kasser, T.: What is satisfying about satisfying events? Journal of Personality and Social Psychology 80(2) (2001) 325–339

21. Sobell, N., Trivich, M.: Brainwave drawing game. In: Delicate Balance: Techniques and Consequences, 1989. IEEE (1989) 360–362

22. Sweetser, P., Wyeth, P.: Gameflow. Computers in Entertainment 3(3) (2005) 1–24

23. Tangermann, M., Krauledat, M., Grzeska, K., Sagebaum, M., Blankertz, B., Vidaurde, C., Müller, K.R.: Playing Pinball with non-invasive BCI. In: Advances in Neural Information Processing Systems 21. The MIT Press (2009) 1641–1648

24. Wolpaw, J.R., Birbaumer, N., McFarland, D.J., Pfurtscheller, G., Vaughan, T.M.: Brain-computer interfaces for communication and control. Clinical Neurophysiology 113(6) (2002) 767–791