# Enriching Evaluation in Video Games

José L. González Sánchez, Rosa M. Gil Iranzo, Francisco L. Gutiérrez Vela

► To cite this version:

José L. González Sánchez, Rosa M. Gil Iranzo, Francisco L. Gutiérrez Vela. Enriching Evaluation in Video Games. 13th International Conference on Human-Computer Interaction (INTERACT), Sep 2011, Lisbon, Portugal. pp.519-522, 10.1007/978-3-642-23768-3_72. hal-01596995

HAL Id: hal-01596995

https://inria.hal.science/hal-01596995

Submitted on 28 Sep 2017

HAL is a multi-disciplinary open access
archive for the deposit and dissemination of sci-
entific research documents, whether they are pub-
lished or not. The documents may come from
teaching and research institutions in France or
abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est
destinée au dépôt et à la diffusion de documents
scientifiques de niveau recherche, publiés ou non,
émanant des établissements d'enseignement et de
recherche français ou étrangers, des laboratoires
publics ou privés.

<figure><img src="image_1.png" /></figure>

Distributed under a Creative Commons Attribution 4.0 International License
# Enriching Evaluation in Video Games

J. L. González Sánchez¹, R. M. Gil Iranzo², F. L. Gutiérrez Vela³

¹,³GEDES – Universidad de Granada (Spain), ²GRIHO – Universitat de Lleida (Spain)

¹joseluisgs@ugr.es, ²rgil@griho.net and ³fgutierr@ugr.es

**Abstract.** One of the greatest challenges to the evaluation of UX in video games is to ascertain if the experience is appropriate for the game. Thus, it is necessary to know how to measure Playability in order to analyze, optimize and adapt it to the player's preferences. However, it is also important to remember that the final satisfaction of the user depends on his or her emotional response, social and cultural influences and hedonic properties of the interaction process with a video game. In this paper we present a way to perform a UX evaluation based on Playability by adding hedonic factors. The aim is to easily and cost-effectively analyze the UX in an agile video game development process.

**Keywords:** Playability, UX Evaluation, Emotions, Cross-Cultural Factors.

## 1 Introduction: Playability and UX in Videogames

Player eXperience (PX or User Experience in Video Games) depends not only on the product itself, but also on the user and the situation in which he or she uses the product [1]. Due to the nature and design of videogames, user experience is enriched by recreational, cultural, and other subjective factors that make analysis and evaluation difficult using traditional methods commonly used in interactive systems. Playability is a term used in the design and analysis of video is a live topic in the scientific community; it has been studied from different points of view and with different objectives. A comprehensive source of the most important references about playability and user experience in video games research works can be found in [2,3]. In this work we use the Playability Model to characterize the PX in Video Games, which attempts to unify the playability characteristics mentioned in the previous references [4,5]. This model proposes the following attributes for defining the degree of PX: Satisfaction, Learnability, Effectiveness, Immersion, Motivation, Emotion and Socialization. Playability is defined as: "a set of properties that describe the Player Experience using a specific game system whose main objective is to provide enjoyment and entertainment, by being credible and satisfying, when the player plays alone or in company", Also, Playability represents the degree to which specified users can achieve specified goals with effectiveness, efficiency and, especially, satisfaction and fun in a playable context of use.

The importance of the hedonic UX factor in video games justifies the need to enrich video game evaluation methods and game metrics with hedonic properties in order to analyze the degree of experience according to a social and cultural background. In this paper we use the concept of Playability [4] as a measure of PX with the objective of performing an analysis based on "rich" evaluation, guided by facets of Playability. The aim is to enhance the overall game experience by
completing information about the PX and determining which video game elements have more influence on the final experience through the emotional response, playability factors and social and cultural background. To illustrate these objectives we present a practical example of how to evaluate the PX by obtaining pragmatic and hedonic information about it and determining which game elements have more influence on the overall experience. To achieve this, we present an evaluation methodology which allows us to easily and cost-effectively perform a quantitative /qualitative and pragmatic/hedonic Playability analysis to apply it in video game playable prototypes in order to improve the overall experience during an agile video game development process.

## 2 Practical Example of Rich Evaluation in Video Games

To illustrate how the UX in video games can be evaluated we performed a videogame analysis taking into consideration the following objectives: analyze quantitatively/qualitatively the degree of PX in an easy and cost-effective manner; test the effect of different video game elements on the overall PX and complete the functional assessment and objectives of PX with hedonic evaluations (emotional response or social/cultural influence). The evaluation of playability was carried out in a laboratory, in order to observe how people actually play a game. The evaluation was divided into four stages. *Pre-Test*: Questionnaires and a short test to obtain information about player profiles. These were completed with emotional information and multicultural background influences. *Test*: We collected information in real time about player experience while users played a video game. We used observation techniques to measure facial and corporal expressions or biometric constants. *Post-Test*: We gave players different questionnaires and interviews to complete with information, especially subjective information related to hedonic properties (hedonic properties are a crucial factor in motivation, emotion and satisfaction attributes). *Facets of Playability* guided these questionnaires. *Reports*: We obtained a number of reports about the PX with information about which playability attributes had more influence, or which type of elements were more valued by the players. We also obtained special emotional information and cultural information using Emocards (Emontional cards/emoticons). The game chosen as an example for evaluation was "Castlevania: Lords of Shadow" by MercurySteam and Konami. To obtain the maximum information about UX we used different player profiles: 'expert' (a person who is a good player, knows the game platform perfectly and is comfortable with difficult game challenges) and 'casual' (a person who plays infrequently and looks for quick entertainment). The experiment involved the participation of 35 student volunteers from different degrees courses students at University of Lleida, Spain.

From the Pre-Test, we were able to collect information about the profile of participants using multimedia questionnaires (images, pictures, music, videos, etc.). The most significant results about *Pre-Test*: The majority of participants were male (75%) between 19 and 22 years old (90%). They were considered to be casual players (play approximately 5 hours per week, and have experience of only one game on console or a mobile phone). They had knowledge of different gaming platforms,
including a mobile and a desktop platform (90%). The preferences for different game genres were: adventure (≈60%) and fighting (≈30%). 87% preferred to play in company. To identify emotional state, we reproduced a fragment of game OST (Original Sound Tracks). For 83% the music inspired “action” and made the user feel “energetic”. In analyzing the adequacy of the game hero and the context of the video game, 77% of students said that hero transmitted the feeling “honest”, 43% reported that the most valuable characteristic was the eye expression and 15% said the face. These factors represent hedonic properties that produce better immersion in the video game and socio cultural background influences.

During the *Test*, users played the videogame. Their facial and body expression was recorded and their heart rate monitored [2,3,6]. The aim was to detect visceral and emotional reactions occurring during the interaction process. With the arrival of challenges at the climax of the level, stress increased, provoking surprise and agitation in the player (lifting of the eyebrows, ≈94%). The stress caused by the challenges increased the player's concentration. Players pressed the pad quicker and more violently (≈89%) than at the beginning of the game. At the climax of the level, two types of strategies were detected: a defense strategy, which was adopted by 100% of female users and 25% of men, who were intimidated by the game enemies and natural catastrophes. However, 75% of male users preferred a direct and violent confrontation. In both cases the degree of immersion was high, a factor, which was indicated by the increase in heart, beats per minute (≈18bpm at the climax of the level). Finally, stress became satisfaction, which was expressed as a slight smile or slight gasp (≈82%). It is interesting to note that the strategy depends on cultural factors such as gender and other factors, which may be related to the survival instinct of the user when faced with real challenges.

At the *Post-Test* we used informal interviews and questionnaires to obtain information about the player experience. The questions and heuristics are designed to extract information about the pragmatic and hedonic PX dimensions. This result was obtained thanks to the relation to the Playability Model, Facets and design of questionnaires [7]. One of the clearest results was that players were happiest with the game. This was reflected in the high values scored for the interactive facet, and ease of use of game controller (see learnability attribute). The positive result for ‘casual’ players meant a negative experience for ‘expert’ players, due to the excessive ease of play. Using the Emocards, we are able to analyze factors that are difficult to assess objectively using a number on a scale of values. However, at the same time, players are able to easily identify the emotional impact they have felt. Post-Test results reaffirm the positive feedback from users with regards to the setting and context as well as sound effects and music in general (high level of satisfaction, mainly characterized by a state of pleasure, which varies between excitation and neutral state). On a global level, the UX results were very positive, and the game satisfied players. From the results, we consider that enriching the PX analysis with multicultural and emotional factors (hedonic properties) helps us to establish information about player preferences and likes. The techniques demonstrated in this work can assist in the improvement of a video game as a product by enhancing the final experience, where the cultural or emotional impact can influence how a game should be designed or modified according to different player profiles or the market in a cost-effective manner.
# 3 CONCLUSION

This work reflects the importance of analyzing the experience that players have with video games in a pragmatic and hedonic way. Understanding and evaluating the user experience in video games is important for developing more efficient and successful products in terms of entertainment where cultural influence and emotional impact are crucial properties. To acquire more information about the player experience, we propose the enrichment of tests with hedonic factors (techniques of emotional design and cultural analysis) to increase the efficiency of questionnaires and heuristics. An example of this includes the use of Emocards, tracking the player response in real time, and measuring biometric constants. The measurement and analysis process is considered to be easy and cost effective to perform, thus making it appropriate for agile development based on playable video game prototypes. Through a practical example, we have demonstrated the importance of this kind of evaluation for the development of a product that can incite better experiences or specific experiences depending on the target market and it is advisable to use in every interactive system. As a future work, user mental model will be incorporated in the evaluation process.

# Acknowledgments

This research is financed by: the Spanish International Commission for Science and Technology (CICYT); the DESACO Project (TIN2008-06596-C02) and OMediaDis (TIN2008-06228). Thanks to MercurySteam for the support in this work.

# Referencias

1. Law, E. et al. Understanding, Scoping and Defining User Experience: A Survey Approach. Proc. Human Factors in Computing Systems (CHI’09). pp.719-728. (2009)

2. Isbister, K., Schaffer, N. (Ed) Game Usability: Advancing the Player Experience. Morgan Kaufmann (2008)

3. Bernhaupt, R (Ed). Evaluating User Experience in Games: Concepts and Methods. Springer. (2010)

4. González Sánchez, J. L.; Padilla Zea, N.; Gutiérrez, F. L. From Usability to Playability: Introduction to Player-Centred Video Game Development Process. Proc. Human-Computer Interaction International (HCII’09). pp 65–74. (2009).

5. González Sánchez, J.L.; Padilla Zea, N.; Gutiérrez, F.L. Playability: How to Identify the Player Experience in a Video Game. Proc. 12th IFIP TC13 Conference on Human-Computer Interaction (INTERACT’09). pp. 356-359 (2009)

6. De Lera, E. & Garreta-Domingo, M. Ten Emotion Heuristics: Guidelines for Assessing the User’s Affective Dimension Easily and Cost-Effectively. Proc. 21st BCS HCI Group Confereces (HCI’07). pp 163-166. (2007)

7. González Sánchez, J. L.; Jugabilidad y Videojuegos. Análisis y Diseño de la Experiencia del Jugador en Sistemas Interactivos de Ocio Electrónico. Editorial Académica Española, Lambert Academic Publishing GmbH & Co KG. (2011).