# Multi-level Engagement in Augmented Reality Children's Picture Books

Nicholas Vanderschantz, Annika Hinze, Aysha Al-Hashami

► To cite this version:

Nicholas Vanderschantz, Annika Hinze, Aysha Al-Hashami. Multi-level Engagement in Augmented Reality Children's Picture Books. 17th IFIP Conference on Human-Computer Interaction (INTER-ACT), Sep 2019, Paphos, Cyprus. pp.558-562, 10.1007/978-3-030-29390-1_37. hal-02877657

HAL Id: hal-02877657

https://inria.hal.science/hal-02877657v1

Submitted on 22 Jun 2020

HAL is a multi-disciplinary open access
archive for the deposit and dissemination of sci-
entific research documents, whether they are pub-
lished or not. The documents may come from
teaching and research institutions in France or
abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est
destinée au dépôt et à la diffusion de documents
scientifiques de niveau recherche, publiés ou non,
émanant des établissements d'enseignement et de
recherche français ou étrangers, des laboratoires
publics ou privés.

<figure><img src="image_1.png" /></figure>

Distributed under a Creative Commons CC BY 4.0 - Attribution - International License
Multi-level Engagement in Augmented Reality
Children's Picture Books

Nicholas Vanderschantz¹, Annika Hinze¹, and Aysha AL-Hashami²

¹,² Computer Science Department, University of Waikato, Hamilton, New Zealand
¹{vtwoz,hinze}@waikato.ac.nz ²assha1@students.waikato.ac.nz

**Abstract.** We demonstrate our AR enhanced picture book that provides multiple levels of interaction and engagement. Holding the camera at a range of heights facilitates reader exploration of layered book features and content. These multi-level enhancements extend the traditional learning possibilities of books while providing increased opportunities for both shared and individual reading.

**Keywords:** Augmented Reality, children's books, reading, shared reading.

# 1 Introduction

The merging of augmented reality (AR) and picture book material has been explored for close to two decades, starting with Billinghurst et al.'s Magic Book [1]. Augmented reality adds virtual elements to a real scene, typically viewed through a headset or a phone's camera. In this way, AR interfaces enhance real-world experiences of books [2]. Inspiring examples of extending the experience of books through AR are the haunted book enhancing poetry by Scherrer [7], the AR colouring book [3], and the Vivid Encyclopedia, an AR book of insects [5]. A number of AR-enhanced picture books are available, few of which focus on providing extended engagement over time as a child grows in understanding. This demonstration showcases our picture book AR app that provides multiple levels for children at different learning stages.

Traditional book enhancements in the form of questions, answers, and content sequencing can prompt children to interact and engage with a book at a deeper level [8]. While ‘touch and feel’ books promote learning for very young readers, ‘Search and Find’ or ‘Question and Answer’ books are often considered useful to teach children how to solve problems, make connections and follow cues because they require critical inspection of a page [12]. Pop-up books are often considered to make the learning experience interactive, playful, and memorable. By promoting a hands-on approach to learning – both figuratively and literally – interactive books allow the depiction of a written concept in visual form. This concept can be translated into using AR enhancements for children books to encourage exploration and learning.

While the use of AR in formal learning environments (see e.g., [2][3][9]) and digital book use and design (see e.g., [6][8][10]) has been extensively explored our application focusses on playful learning during shared reading at home. Yuen et al. [11] observe that most likely AR-enhanced books will provide a stepping stone for the public to
engage in the cross-over between digital and physical worlds. We agree with their observation that AR-enhanced books have the potential to provide many paths for learners. In our application, we provide these interaction and learning paths within the same page of a book, addressing young readers at different levels of comprehension through the layering of AR features.

## 2 Demonstration Application

Our AR-enhanced book prototype is based on the children's book 'Hannah's Favourite Place' by Fiona Mason, consisting of the printed book source and an AR companion app. The mobile app provides AR animations, audio, and additional textual elements to enhance the content of the printed book. Elements of the original book were removed from the printed page to allow these to be augmented within the AR app. The app delivers the animations based on recognition of visual marker location anchors which are extracted from the book's original illustrations. No further changes were made to the printed book source to allow for the augmentation provided by the app.

**Scenario.** We propose a scenario where we first meet Maia as a 4-year-old pre-reader sat with her dad John. In front of them is the book Hannah's Favourite Place which is open to a middle spread. At first, Maia holds dad's smart-phone in landscape mode with her two small hands holding the camera quite close to the book. She moves the phone around and explores the interactions that occur when individual characters in the book are viewed filling the camera's screen. John then helps Maia lift the phone a little further from the page. The two share the task of holding the phone and here the camera views more than one character or page feature at a time. Finally, Maia hands John the phone and he holds the camera farther from the page so that they can both see the entire spread through the phone.

A few years later we again meet Maia who as she grows older and more confident with books and reading and as her fine motor skills develop we find that she is able to read and interact with Hannah's Favourite Place on her own. Yet there are features of this book that have held her interest for longer than some other books in her collection. Interactive, animated, textual, and audio features of the book have engaged her at different learning phases in her life as if to say the book has grown with her. The features that gripped her during the first read with John are less amusing, yet the opportunities to practice spelling and pronunciation, as well as to read along with the story carry interest for Maia today.

**Implementation.** Exploration of multiple levels of interaction within a single page is achieved in our app through interactive features that become visible on the phone at three different camera heights, as shown on the left side of Figure 1.

At Level 1, the camera is held close to the page and playful interactions of individual image elements are provided. One of these is an animation and audio of a panda sleeping in the tree (see top of Figure 1).
<figure><img src="image_2.png" /><figcaption>**Fig. 1.** Interacting with the book to encounter layered AR animations</figcaption></figure>

At Level 2, the camera is held slightly higher from the page and simple educational interactions with two or more objects at a time result. For example, in Figure 1, middle we illustrate the identification of the two chimps; here they are pointed to by large playful arrows and the spelling of the animals' name is animated.

Level 3 requires the child or parent to hold the camera to view almost the entire page; the app provides both playful and educational interactions. In the example shown in Figure 1, bottom, the story is narrated and animated bees fly across the page. These bees are objects that do not occur in the illustrated storybook, nor are they a part of the written text. However, the inclusion of such non-text illustration is common in children's books and enhances opportunities for discussion, emersion, and realistic building of a scene or environment.

Further details of design thinking for our AR-enhancement is discussed in [9]. An interaction of the app with the book spread (engaging with level 2) is shown in Figure 2.

<figure><img src="image_3.png" /><figcaption>**Fig. 2.** User interacting with the enhanced book via AR app (Level 2 animation)</figcaption></figure>
## 3 Summary

This paper discussed our mobile phone-based augmented reality picture book prototype that provides AR animations, audio, and additional textual elements to enhance the content of the printed book. We believe that enhanced picture books, be they with augmentation, physical interaction, or cognitive interaction will increase the learning possibilities and extend the useful lifespan of that book along with the child. The significant feature of our work is the investigation of the affordances of the 3-dimensional space between the camera and the book surface. At the time of our investigation, we are not aware of other children's books that presently exploit this multi-level enhancement in their implementation. Our future work will extend this prototype based on insights from parents about the role of books and technology in their homes.

## References

1. Billinghurst, M., Duenser, A.: Augmented reality in the classroom. Computer (45)7, 56-63 (2012).

2. Billinghurst, M., Kato, H., Poupyrev, I.: The magicbook-moving seamlessly between reality and virtuality. IEEE Computer Graphics and applications (21)3, 6-8 (2001).

3. Clark, A., Dunser, A.: An interactive augmented reality coloring book. In 2012 IEEE Symposium on 3D User Interfaces (3DUI), pp. 7-10. IEEE, (2012).

4. da Silva, M., Marcelo J., Teixeira, XN., Cavalcante, PS., Teichrieb, V.: Perspectives on how to evaluate augmented reality technology tools for education: a systematic review. Journal of the Brazilian Computer Society (25)1, 3 (2019).

5. Fumihisa, S., Yusuke, Y., Koki, F., Toshio, S., Kenji, K., Asako, K., Hideyuki, T.: Vivid Encyclopedia: MR Pictorial Book of Insects. In Proceedings of Virtual Reality Society of Japan Annual Conference. (2004).

6. Kucirkova, N.: An integrative framework for studying, designing and conceptualising interactivity in children's digital books. British Educational Research Journal (43)6, 1168-1185 (2017).

7. Scherrer, C., Pilet, J., Fua, P., Lepetit, V.: The haunted book. In Proceedings of the 7th IEEE/ACM International Symposium on Mixed and Augmented Reality (ISMAR '08). pp.163-164. IEEE, (2008).

8. Timpany, C., Vanderschantz, N.: A Categorisation Structure for Interactive Children's Books. International Journal of the Book (9)4, 97-110 (2012).

9. Vanderschantz, N., Hinze, A., Al-Hashami, A.: Multiple Level Enhancement of Children's Picture Books with Augmented Reality. In Proceedings of International Conference on Asian Digital Libraries, pp. 256-260. Springer, (2018).

10. Vanderschantz, N., Timpany, C.: Analysing Interaction in Children's Digital Books. International Journal of the Book (9)4, 31-47 (2012).

11. Yuen, SCY., Yaoyuneyong, G., Johnson, E.: Augmented reality: An overview and five directions for AR in education. Journal of Educational Technology Development and Exchange (JETDE) (4)1, 11 (2011).

12. Zeece, PD.: Using current literature selections to nurture the development of kindness in young children. Early Childhood Education Journal (36) 5, 447-452 (2009).