# Adding Images to Psychometric Questionnaires to Improve User Engagement

Mohammed Khwaja, Sarah Shepherd

<figure><img src="image_1.png" /></figure>

## To cite this version:

Mohammed Khwaja, Sarah Shepherd. Adding Images to Psychometric Questionnaires to Improve User Engagement. 17th IFIP Conference on Human-Computer Interaction (INTERACT), Sep 2019, Paphos, Cyprus. pp.621-625, 10.1007/978-3-030-29390-1_49. hal-02878653

HAL Id: hal-02878653

https://inria.hal.science/hal-02878653v1

Submitted on 23 Jun 2020

HAL is a multi-disciplinary open access
archive for the deposit and dissemination of sci-
entific research documents, whether they are pub-
lished or not. The documents may come from
teaching and research institutions in France or
abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est
destinée au dépôt et à la diffusion de documents
scientifiques de niveau recherche, publiés ou non,
émanant des établissements d'enseignement et de
recherche français ou étrangers, des laboratoires
publics ou privés.

<figure><img src="image_2.png" /></figure>

Distributed under a Creative Commons CC BY 4.0 - Attribution - International License
# Adding Images to Psychometric Questionnaires to Improve User Engagement

Mohammed Khwaja<fnref n="1" />,<fnref n="2" /> and Sarah Shepherd<fnref n="1" />

**Abstract.** This paper presents an experiment to obtain personality traits using a psychometric questionnaire that is augmented with visual representations that support answer options. From the results, it is suggested that the use of visual media improves participants’ perception of ‘fun’ or pleasure, while answering the questionnaire; thereby creating the potential to improve user engagement in commercial applications.

**Keywords:** Personality Detection · Psychometric Questionnaires · User Engagement

## 1 Introduction

Most behaviour change applications in today’s market are one-size-fits-all, and evidence suggests that personalisation is needed in order to bring about effective, engaging and long-lasting change [8]. Traditionally obtained using text questionnaires, personality traits have been used for personalising behaviour change applications [6]. However, completing questionnaires requires focused attention, as numerous cognitive processes are involved when answering questions about one’s behaviours [3]. Past research has indicated that focused attention is an important component of user engagement [2], and when users have fun, there is increase in cognitive absorption of technology use [1]. Measuring and improving user engagement is important to the Human Computer Interaction (HCI) community, as is evidenced by past literature [4, 9] that focused on this.

We present a study with a modified psychometric questionnaire using images and GIFs (including pop culture references) to obtain the Big 5 personality traits. Inspired by the growth of communication through emojis, GIFs, images and videos, over text, we test if the addition of a visual layer on top of psychometric questionnaires is as accurate in capturing personality as a gold-standard questionnaire; and makes the questionnaire more fun - potentially improving the engagement of commercial application that use these. Work has been done in the past to obtain personality scores by creating image based quizzes<fnref n="3" />, and a patent [10] has been filed on this system. Our work does not create a new personality quiz for detecting personality traits, but rather augments visual imagery to a gold-standard personality questionnaire to improve its perception with users.

<footnote n="1">¹ Telefonica Alpha, Barcelona, Spain
name.surname@telefonica.com</footnote>

<footnote n="2">² Imperial College, London, UK</footnote>

<footnote n="3">³ http://you.visualdna.com/quiz/whoami?c=us#/quiz</footnote>
## 2 Study - Methodology

Our study required participants to complete two questionnaires: 1. the baseline questionnaire - the gold-standard that provides 'ground truth' data and 2. the visual questionnaire - our modified questionnaire. We do this to compare the scores and perception of our modified questionnaire with the gold-standard. The questionnaires were spaced one week apart so that participants don't rely on memory to provide answers. Half of the participants were asked to complete the baseline questionnaire first, while the other half were asked to complete the visual questionnaire first.

Before the first questionnaire, we provided an onboarding survey to collect demographics, including gender and age range. At the end of both questionnaires, we provided an exit survey with two questions: 1. 'Ease of Completion' to understand how easy the questionnaire was to complete and 2. 'Pleasure' to understand how fun the questionnaire was. We use a 5 point Likert scale for these. The questionnaire was anonymous and no personal data including names and email IDs were collected from participants. Users were given unique codes to answer both questionnaires, enabling us to match responses from the same user.

There are multiple questionnaires to obtain the Big 5 personality traits, including the mini-IPIP developed by Donnellan et al. [5]. In this work, the authors developed a 20 item questionnaire with a likert scale to obtain the Big 5 personality traits, namely Extroversion, Agreeableness, Conscientiousness, Neuroticism and Openness. The questionnaire can be broken down into 4 brackets, with each bracket containing 1 question related to each of the 5 personalities. The questionnaire is well regarded as the gold-standard in the Psychological community, for its length and accuracy. We use this questionnaire as the baseline, over which we build the visual questionnaire.

In the visual questionnaire, we used questions from the baseline in the same order with the same connotation, but modified it to suit images and gifs that we selected. The selection of images or.gifs for answer options was based on mutual agreement between the researchers conducting the study, with subsequent validation from a subject expert. We provided three options for each question in a Likert scale, ranging from positive/agree/strong to negative/disagree/week. This was done to reduce the cognitive effort it takes for the user to select between strongly agree and agree (and the opposite), and to overcome the difficulty of obtaining images that clearly differentiate two consecutive options. While it would have been ideal to use a 5 point Likert scale, and most questionnaires and surveys use this, evidence has indicated that 3 point Likert scales are just as accurate to obtain user opinions and can be used for surveys that don't need a breadth of statistical tests [7].

Two questions from the visual questionnaire are presented in Fig. 1. These are similar to the questions presented in the baseline questionnaire, albeit in a different form. To maintain consistency in the scores obtained from the baseline and visual questionnaires, the options for the baseline questionnaire were also presented in a 3 point Likert scale. The scores for each trait are calculated in
<figure><img src="image_3.png" /><figcaption>**Fig. 1.** Question to obtain a component of (a) Extroversion, (b) Conscientiousness</figcaption></figure>

the manner defined in [5] - positive questions were scored in the same order, and negative questions were reverse scored. Each trait has four questions that contribute to its score.

The study was performed by 26 participants, employed at a research institute in Barcelona, Spain. The call for participation was sent out to over 60 individuals using an internal mailing list, and participants registered to volunteer. The study was performed in late 2018. The number of females to males was 48% to 52%, and participants belonged to the age ranges of 18-24 (9%), 25-34 (56%), 35-44 (26%) and 45-54 (9%). Participants were citizens of countries including Spain, United Kingdom, United States, Germany and others, and included a mix of native and non-native English speakers.

## 3 Results

First we obtained the correlations ($r$ values) by regressing the trait scores obtained from both questionnaires. This is presented in Table 1. High correlation scores with significant $p$-values indicate that the visual questionnaire is able to obtain the personality scores with a high similarity as the text based questionnaire. We also observed that no difference was observed between the two groups of participants, i.e., correlations obtained from participants who took the baseline questionnaire first were similar to those who took the visual questionnaire first.

Next we obtained the correlation between the answer from each question in a bracket and the personality score obtained from the baseline, and the highest correlations are shown in Table 1. We note that the correlations are not comparable to those obtained from using all questions. Subsequently, we selected the 2 questions from the visual questionnaire with the highest correlations to obtain a score with 2 questions and correlated this with the baseline scores. It
<table><caption>**Table 1.** Correlations (*r*- values) obtained between the visual questionnaire and the baseline questionnaire</caption><thead><tr><th></th><th>Extr.</th><th>Agree.</th><th>Cons.</th><th>Neur.</th><th>Open.</th></tr></thead><tbody><tr><td>All questions</td><td>0.88***</td><td>0.74**</td><td>0.77**</td><td>0.73***</td><td>0.88***</td></tr><tr><td>One Question</td><td>0.75**</td><td>0.72**</td><td>0.68**</td><td>0.78***</td><td>0.82***</td></tr><tr><td>Two Questions</td><td>0.84***</td><td>0.73**</td><td>0.77***</td><td>0.71**</td><td>0.84***</td></tr></tbody></table>

is observed that the correlations obtained are similar to those obtained using all 4 questions, indicating that using 2 questions for each trait, or 10 questions overall can provide similar scores versus using 20.

<figure><img src="image_4.png" /><figcaption>**Fig. 2.** Bar charts representing the rating distribution of (a) Ease of Completion, (b) Pleasure</figcaption></figure>

The results obtained from the exit questionnaire are represented using bar graphs in Fig 2. Results from the 'Ease of Completion' question indicate that both the baseline and visual questionnaires are easy to complete, with slight preference for the baseline as they maintain the same text answer options for all questions. However the 'Pleasure' question indicated that the visual questionnaire was perceived as significantly more fun than the baseline.

## 4 Conclusion and Future Work

This study indicates that the use of visual media to obtain personality traits from psychometric questionnaires is both accurate and fun to complete - providing the potential to improvement user engagement. Feedback from one of the participants: "I loved this survey so much!! it was really engaging. I think all surveys should look like this." further highlights its merit.

In the future, we plan to conduct a large scale study and explore if similar results hold with a larger population sample. We will incorporate this method in a commercial application for personalised interventions, explore improvements in user engagement and obtain further user feedback on this medium of answering questionnaires. This method also has the potential to be applied to other psychometric questionnaires, improving the application of psychology to HCI.

<footnote>** *p* < 0.01, *** *p* < 0.001</footnote>
Acknowledgements

We would like to thank Dr Aleksandar Matic for his guidance. This work has
been supported by the EU's Horizon 2020 research and innovation programme,
under the Marie Sklodowska-Curie grant agreement no. 722561.

References

1. Agarwal, R., Karahanna, E.: Time flies when you’re having fun: Cognitive absorption and beliefs about information technology usage. MIS quarterly pp. 665–694 (2000)

2. Attfield, S., Kazai, G., Lalmas, M., Piwowarski, B.: Towards a science of user engagement (position paper). In: WSDM workshop on user modelling for Web applications. pp. 9–12 (2011)

3. Blair, E., Burton, S.: Cognitive processes used by survey respondents to answer behavioral frequency questions. Journal of consumer research 14(2), 280–288 (1987)

4. Doherty, K., Doherty, G.: Engagement in HCI: Conception, Theory and Measurement. ACM Computing Surveys (CSUR) 51(5), 99 (2018)

5. Donnellan, M.B., Oswald, F.L., Baird, B.M., Lucas, R.E.: The mini-IPIP scales: tiny-yet-effective measures of the Big Five factors of personality. Psychological assessment 18(2), 192 (2006)

6. Halko, S., Kientz, J.A.: Personality and persuasive technology: an exploratory study on health-promoting mobile applications. In: International conference on persuasive technology. pp. 150–161. Springer (2010)

7. Jacoby, J., Matell, M.S.: Three-point likert scales are good enough (1971)

8. Noar, S.M., Benac, C.N., Harris, M.S.: Does tailoring matter? meta-analytic review of tailored print health behavior change interventions. Psychological bulletin 133(4), 673 (2007)

9. Peters, C., Castellano, G., de Freitas, S.: An exploration of user engagement in hci. In: Proceedings of the International Workshop on Affective-Aware Virtual Agents and Social Robots. p. 9. ACM (2009)

10. Willcock, A.: System and method of segmenting and tagging entities based on profile matching using a multi-media survey (Feb 11 2014), uS Patent 8,650,141