Liu, C.-C. et al. (Eds.) (2014). Proceedings of the 22nd International Conference on Computers inEducation. Japan: Asia-Pacific Society for Computers in Education

Authoring Augmented Reality
as Situated Multimedia
Marc Ericson C. SANTOSa*, Jayzon F. TYb, Arno in Wolde LUEBKEa,
Ma. Mercedes T. RODRIGOb, Takafumi TAKETOMIa, Goshiro YAMAMOTOa,
Christian Sandora & Hirokazu KATOaaInteractive Media Design Laboratory, Nara Institute of Science and Technology, Japanb
Ateneo Laboratory for the Learning Sciences, Ateneo de Manila University, Philippines*chavez-s@is.naist.jp
Abstract: Augmented reality (AR) is an enabling technology for presenting information inrelation to real objects or real environments. AR is situated multimedia or information that ispositioned in authentic physical contexts. In this paper, we discuss how we address issues increating AR content for educational settings. From the learning theory perspective, we explainthat AR is a logical extension of multimedia learning theory. From the development
perspective, we demonstrate how AR content can be created through our in situ authoring tooland our platform for handheld AR.
Keywords: augmented reality, authoring tool, multimedia learning, situated learning

1. Introduction
Augmented reality (AR) is an emerging technology for educational settings. Our review (Santos et al.
2014) provides a useful summary of AR prototypes applied to learning. Currently, researchers havevarying definitions for AR, depending on which of its features are emphasized. We define augmented
reality learning experiences based on Azuma’s (1997) definition of AR – “3-D virtual objects areintegrated into a 3-D real environment in real time”. This is a conceptual definition that is independent
from enabling technologies. The “objects” mentioned are more understood as inserting computer
graphics and aligning it to a video feed of the real environment. However, Azuma et al. (2001)
explained that AR can potentially apply to all senses.
Although there are many AR prototypes for learning, researchers do not usually use the mostimportant feature of AR – showing an explicit relationship between the virtual learning content and real
objects found in the natural environment. There are many definitions for AR. As a technology foreducation, we propose to define AR to be situated multimedia. In other words, AR is multimedia (text,sound, images, animations, etc.) that is displayed in relation to the real environment. Thereby, the realenvironment becomes the “authentic context” of learning that is characteristic of situated learning(Herrington & Oliver, 1995). From this formalization, we then discuss our implementation of authoring
tools that teachers can use for creating such situated multimedia.

2. Extending Multimedia Learning Theory to Augmented Reality
Previous research works have described AR to offer contextual learning (Specht et al., 2011)
and ubiquitous learning (Dede, 2011). Indeed, AR has several benefits because it applies situatedcognition (Wu et al., 2013). In this paper, we take a step back and explain that AR is essentiallymultimedia. As such, multimedia learning theory applies. In this theory, multimedia refers to pictures
and words. It assumes dual-channels, limited capacity, and active processing (Mayer, 2009). First,
humans have two separate channels for perceiving visual and auditory information. Second, humans
have a limited capacity of information that they can attend to. Lastly, learning only takes place if
learners actively make sense of incoming information using their prior knowledge. AR visualization
can reduce cognitive load, thereby, allowing students to allot more effort to actively processing

information. In a previous study (Fujimoto et al., 2012), we have shown that using AR visualization
enables better memorization of abstract symbols. This result is consistent with the prediction of spatial
and time contiguity principles of multimedia learning.
Multimedia learning is learning with pictures and words (both written and spoken). This theory
applies to AR if we make the following logical substitutions: The real objects or the real environment isthe picture. The virtual texts, symbols and pre-recorded sounds are the words. A new picture is created
when real objects, real environment and the original picture are combined.

3. Authoring Augmented Reality Content
One of the difficulties for adapting situated multimedia is the lack of authoring tools to makeeducational content. Currently, several authoring tools exist. However, they do not address the needs ofteachers for a simple but flexible tool for everyday use. Software libraries like the ARToolkit (Kato &Billinghurst, 1999), Vuforia1 and PointCloud SDK2 are authoring tools for programmers.
MacIntyre et al. (2004) and Hengel et al. (2009) developed desktop-based systems for
non-programmers so that they can author AR content. On the other hand, Langlotz et al. (2012) createda system for handheld-based authoring. Although handhelds have less processing power than desktops,
it has the key advantage of mobile authoring at any place in any time. Currently, Langlotz et al. (2012)draws basic 3D shapes and other features such as copying, deleting, etc.

3.1 Simple In Situ Authoring for Teacher’s Use
For the purposes of teachers, it is enough for them to download pictures from the internet, and thenplace it on a real environment. As such, we implemented an authoring tool that enables teachers todownload a picture, and perform affine transformations. Our prototype uses the ARToolkit running on
iPad 2 (dual-core A5, 512MB DDR2 RAM, 32GB, 601 grams, 9.7 in display, 1024-by-768 at 132 ppi).
Figure 1 shows the interface for our simple authoring tool for teachers and a sample use case. In
this example, teachers can download any image from the internet. This image is converted into a texture
on the screen. Using gestures like swipe, pinch, and so on, the teachers can modify the appearance of theimage. In this example, it is desirable to scale and position the lungs correctly on the body.

Figure 1. Simple Authoring Tool (left); Sample Use of the Authoring Tool (right)

3.2 Handheld AR Platform for Situated Multimedia
Currently, authoring tools focus on rendering fast and beautiful graphics on a real world scene. Thereare few authoring tools that emphasize on the use of sounds and text. For educational settings, teachers
use a combination of image, sound and text to facilitate learning. As such, we developed a handheld AR
platform for presenting images, sound and text onto real objects.
We implemented the whole platform on iOS7 running on iPad 2. Figure 2 shows the package
diagram and a sample application for vocabulary learning. The main part of the platform is theController, which has access to learning contents. It receives the marker ID and camera view matrix
12

https://developer.vuforia.com/
http://developer.pointcloud.io/sdk

from the Tracker and uses these information to specify the behavior of the on-screen display. TheTracker was built using ARToolkit, and the Renderer was built on OpenGL ES 2.04. In the sample use
case, students can learn the word “pindutin” (Filipino for “to push”) by animating a hand which pushes
a real button. This is an example of situated multimedia because it shows explicitly the relationship of
the animation with real objects such as a coffee maker found in the learner’s natural environment.

Figure 2. Package diagram of handheld AR platform (left); Sample Interface using the platform (right)

4. Conclusion
We defined augmented reality as situated multimedia. For the first time, we point out that AR could bebetter understood and designed by applying multimedia learning theory. Aside from graphics, soundand text is also required for creating AR learning materials. As such, we implemented two prototypes
for authoring situated multimedia content namely, in situ authoring and our handheld AR platform.
Currently, we are conducting evaluations of the interface with teachers. Educators can benefit from thisresearch because it will enable them to design educational content in authentic contexts.

References
Azuma, R. T. (1997). A survey of augmented reality. Presence, 6(4), 355-385.
Azuma, R., Baillot, Y., Behringer, R., Feiner, S., Julier, S., & MacIntyre, B. (2001, Nov). Recent advances inaugmented reality. Computer Graphics and Applications, IEEE, 21(6), 34-47.
Dede, C. (2011). Emerging technologies, ubiquitous learning, and educational transformation. In Towards
Ubiquitous Learning (pp. 1-8). Springer.
Herrington, J., & Oliver, R. (1995). Critical Characteristics of Situated Learning: Implications for the Instructional
Design of Multimedia. In J. Pearce, & A. Ellis, Learning with technology (pp. 235-262). Victoria:
University of Melbourne.
Fujimoto, Y., Yamamoto, G., Kato, H., & Miyazaki, J. (2012). Relation between location of information
displayed by augmented reality and user's memorization. Proceedings of the 3rd Augmented Human
International Conference (pp. 7:1--7:8). New York, NY, USA: ACM.
Kato, H., & Billinghurst, M. (1999). Marker tracking and hmd calibration for a video-based augmented reality
conferencing system. International Workshop on Augmented Reality, 1999.(IWAR'99)
Langlotz, T., Mooslechner, S., Zollmann, S., Degendorfer, C., Reitmayr, G., & Schmalstieg, D. (2012). Sketchingup the world: in situ authoring for mobile augmented reality. Personal and ubiquitous computing, 16(6),623-630.
MacIntyre, B., Gandy, M., Dow, S., & Bolter, J. D. (2004). DART: a toolkit for rapid design exploration ofaugmented reality experiences. User interface software and technology, (pp. 197-206).
Mayer, R. E. (2009). Multimedia learning. Cambridge University Press.
Santos, M., Chen, A., Taketomi, T., Yamamoto, G., Miyazaki, J., & Kato, H. (2014, March). Augmented Reality
Learning Experiences: Survey of Prototype Design and Evaluation. Learning Technologies, IEEE
Transactions on, 7(1), 38-56.
Specht, M., Ternier, S., & Greller, W. (2011). Mobile augmented reality for learning: A case study. Journal of theResearch Center for Educational Technology, 7(1), 117-127.
van den Hengel, A., Hill, R., Ward, B., & Dick, A. (2009). In situ image-based modeling. Mixed and AugmentedReality, 2009. ISMAR 2009. 8th IEEE International Symposium on, (pp. 107-110).
Wu, H. K., Lee, S. W., Chang, H. Y., & Liang, J. C. (2013). Current status, opportunities and challenges ofaugmented reality in education. Computers & Education, 62, 41-49.