# Dual Camera Magic Lens for Handheld AR Sketching

Klen Čopič Pucihar, Jens Grubert, Matjaž Kljun

► To cite this version:

Klen Čopič Pucihar, Jens Grubert, Matjaž Kljun. Dual Camera Magic Lens for Handheld AR Sketching. 15th Human-Computer Interaction (INTERACT), Sep 2015, Bamberg, Germany. pp.523-527, 10.1007/978-3-319-22723-8_53. hal-01610773

HAL Id: hal-01610773

https://inria.hal.science/hal-01610773

Submitted on 5 Oct 2017

HAL is a multi-disciplinary open access
archive for the deposit and dissemination of sci-
entific research documents, whether they are pub-
lished or not. The documents may come from
teaching and research institutions in France or
abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est
destinée au dépôt et à la diffusion de documents
scientifiques de niveau recherche, publiés ou non,
émanant des établissements d'enseignement et de
recherche français ou étrangers, des laboratoires
publics ou privés.

<figure><img src="image_1.png" /></figure>

Distributed under a Creative Commons Attribution 4.0 International License
# Dual Camera Magic Lens for Handheld AR Sketching

Klen Čopič Pucihar, Jens Grubert, Matjaž Kljun

Computer Science Department, University of Primoraska, Koper, SLO
Embedded Interactive Systems Laboratory (EISLab), University of Passau, Passau, DE
{klen.copic,matjaz.kljun}@famnit.upr.si, jg@jensgrubert.de

**Abstract.** One challenge of supporting in-situ sketching tasks with Magic Lenses on handheld Augmented Reality systems is to provide accurate and robust pose tracking without disrupting the sketching experience. Typical tracking approaches rely on the back-facing camera both for tracking and providing the view of the physical scene. This typically requires a fiducial to be in the scene which can disrupt the sketching experience on a blank sheet of paper. We address this challenge by proposing a Dual Camera Magic Lens approach. Specifically, we use the front facing camera for tracking while the back camera concurrently provides the view of the scene. Preliminary evaluation on a virtual tracing task with an off-the-shelf handheld device suggests that the Dual Camera Magic Lens approach has the potential to be both faster and lead to a higher perceived satisfaction compared to Magic Lens and Static Peephole interfaces.

**Keywords:** Magic-lens, dual-camera, sketching, trace-drawing, virtual-tracing.

## 1 Introduction

Sketching is an important ancient human skill stimulating creative, visual and spatial thinking. Computer systems that support sketching have been studied since the early days of computer science [5]. Through the development of mobile computing devices, such as smartphones, Magic Lens (ML) became a popular interface to support user sketching (e.g. [2,3,4,6,7]). ML acts as filter augmenting the scene with additional digital content, for example, by adding 3D models based on a recognized paper sketch [2] or by allowing the creation of new 3D content [4,7]. In this paper we explore how novice users can be supported in creating physical sketches through *virtual tracing*, i.e. creating a physical sketch on paper given a virtual image on the handheld device.
<figure><img src="image_2.png" /><figcaption>(a) Static Peephole SP</figcaption></figure>

<figure><img src="image_3.png" /><figcaption>(b) Magic Lens (ML)</figcaption></figure>

<figure><img src="image_4.png" /><figcaption>(c) Dual Camera ML</figcaption></figure>

<figure><img src="image_5.png" /><figcaption>**Fig. 1.** (a), (b), (c) Mobile device sketching aid tools on the study; (d) D-ML transformations;</figcaption></figure>

The core challenge for this (and other sketching) tasks, which involve a physical pen and paper, is to provide accurate and robust pose tracking without disrupting the sketching process. So far, AR sketching systems focused on authoring digital objects and relied on external tracking systems (e.g. [4]) or on marker based tracking (e.g. [6]). Here we focus on utilizing AR as a crafting tool for curtain of real objects (e.g. pen drawings). While marker based tracking would allow in-situ sketching on physical paper in otherwise unprepared environments (Fig 1b.), it limits the sketching experience in a fundamental way: the marker has to be in the camera view taking away valuable space for sketching. While approaches, such as contour tracking [1,2] circumvent the use of a marker, they are prone to failure as they cannot be occluded during interaction.

An alternative is to eliminate the need for pose tracking by placing the device and drawing surface at fixed position, such as in the case of a virtual mirror<fnref n="1" /> or camera sketcher<fnref n="2" />. In both situations the user has to manually position the graphical content into the real world using traditional handheld interfaces such as static peephole (SP). Hence, if the drawing format does not fit into camera's field-of-view (FOV), the user is required to manually realign the graphic every time the device is moved.

In this paper we address the challenge of pose tracking while mitigating the effects on in-situ sketching experiences. Specifically, we evaluate how utilizing both front and back facing cameras concurrently could improve

<footnote n="1">¹ https://www.playosmo.com/en/</footnote>

<footnote n="2">² https://play.google.com/store/apps/details?id=com.aku.drawissimo</footnote>
the utility of ML as a sketching aid tool. In order to do so, we: (i) design and build a Dual Camera Magic Lens (D-ML) system utilizing the front camera for pose tracking and the back camera for scene capture and rendering; (ii) evaluate the proposed solution on a commercially available handheld device by conducting a preliminary user study with 6 participants performing virtual tracing task. We compare 3 interaction methods: Static Peephole (SP), Magic Lens (ML) with fiducial marker and Dual Camera Magic Lens (D-ML).

## 2 Dual Camera Magic Lens

In contrast to using the back facing camera as in standard handheld AR applications we propose to utilize the front facing camera for pose tracking while providing the view of the scene through the back facing camera. We do this in order to mitigate the effects on in-situ sketching experiences.

In order to enable front camera pose tracking, a marker is placed above the drawing surface in parallel orientation (in our case ca. 60cm, see Figure 1c.). As the tracking and rendering camera are not the same, a set of additional transformations (Figure 1d.) needs to be added to the tracked pose result (on Figure 2a. denoted as $RT_{FC->TM}$). However, as the two cameras on the phone are fixed and the top marker is rigidly attached above the drawing area, the only two transformations that change are $RT_{FC->TM}$ and $RT_{BC->DS}$. Hence, as long as the front camera tracking is successful and the two constant transformations are known, $RT_{BC->DS}$ can be calculated (Figure 1d.).

## 3 User Study

The preliminary user study asked participants to perform a virtual tracing task on A3 paper. Participants were instructed to sit at a table and draw a cartoon character as quickly and as accurately as possible. To estimate participants' perceived satisfaction we are utilizing the “overall reactions” part of the Questionnaire for User Interaction Satisfaction (QUIS). In addition, we asked participants to rank interaction modes and
to justify their choice. As objective measure we recorded task completion time.

We used a within-subjects design. Each participant drew three different contours with each interaction method: SP, ML, D-ML (see Fig. 1.). In SP manual alignment is required each time the phone is moved. In ML the drawing is possible whilst holding the phone in hand, whereas in SP it is mandatory to place the phone on the stand. In case of the D-ML, the stand was included because contrary to the tracker used in ML implementation, the tracker used in D-ML did not provide sufficiently robust and accurate orientation tracking results. By placing the phone on stand, we locked two degrees of freedom (Rx and Ry) improving tracking quality. We recognize this as a limitation. However, this decision was mandatory as poor tracking quality is bound to undermine performance of the proposed interaction paradigm. Additionally, as it should be possible to improve tracking performance of future D-ML systems, this does not undermine the proposed interaction concept per say, but rather affects the direct comparability of captured results. Yet, within the context of this study which is of exploratory nature and predominantly based on qualitative data, we consider our study design as appropriate.

<figure><img src="image_6.png" /><figcaption>**Fig. 2.** (a) Drawing quality ranking results (smaller is better); (b) "Overall reactions" part of QUIS scores [1-9]. (c) Preference ranking results (smaller is better); (d) Task time in minutes;</figcaption></figure>
# 4 Results

The study was completed by six participants. All were male, aged between 24 and 45 years. Due to the small number of participants null-hypothesis significance testing would result in poor statistical power. Hence, we present solely descriptive statistics. By overlaying drawn contours with template contours, two researchers independently and subjectively compared the quality of all three drawings for each participant and ranked drawings from best to worst. As shown in Figure 2.e, the comparison did not highlight any obvious deviations in obtained rankings. The results in Figure 2 also suggest that D-ML has the potential to (i) be the fastest mode; (ii) have the highest QUIS score across all properties; and (iii) have best rank. Again those results could not be reliably tested for statistical difference. Five participants that ranked D-ML as the best method justified their ranking choice by highlighting the advantage of automatic alignment and the fact that marker was not in their way.

# 5 Discussion and Conclusion

The results show that the utility of handheld AR as an in-situ sketching tool has the potential to be improved utilizing the D-ML approach. The designed and implemented system demonstrates such a solution is feasible on commercially available mobile devices. The preliminary study indicated that whilst achieving comparable quality of drawing, compared to SP and ML, D-ML is: potentially faster, users perceived higher satisfaction, and is preferred by participants. We believe the main reason for such a result is the fact the camera tracking did not interfere with user sketching. As underlined by participants themselves, the main benefit is automatic alignment of virtual image with the real world. Even though, until the user moves the marker, automatic alignment is also present in case of ML, in D-ML the user did not have to put up with the marker and avoid occluding the marker whilst trace drawing onto the paper. Although one could argue that the stand ambiguity increased the divide between the ML and D-ML, none of participants highlighted the stand as the factor influencing their ranking choice, suggesting the importance of the stand might be limited. However, in future work we will explore the effects of stand vs. handheld mode in more detail. Additionally, due to the
small sample size our results are of preliminary nature and hence should
be verified with a larger number of users. Finally, future research should
look at less intrusive ways of placing a marker above the user.

References

1. N. Hagbi, O. Bergig, J. El-Sana, and M. Billinghurst, "Shape Recognition and Pose Estimation for Mobile Augmented Reality," In IEEE TVCG, pp. 65-71, Oct. 2009.

2. N. Hagbi, R. Grasset, O. Bergig, M. Billinghurst, & J. El-Sana, "In-Place Sketching for Content Authoring in Augmented Reality Games," CIE, 12(3), 3, (2015).

3. K. Čopič Pucihar, P. Coulton & J. Alexander, "The use of surrounding visual context in handheld AR: device vs. user perspective rendering," CHI, 197–206, (2014).

4. H. Seichter, "Sketchand+," In CAADRIA 2003, pp. 209-219, 2003.

5. I. E. Sutherland, "Sketchpad: A Man-Machine Graphical Communication System," Technical Report No. 296, Lincoln Laboratory, Massachusetts Institute of Technology, 1963.

6. M. Xin, E. Sharlin, & M. C. Sousa, "Napkin sketch: handheld mixed reality 3D sketching," In Proc. VRST '08, ACM, pp. 223-226, 2008.

7. B. Yee, Y. Ning, H. Lipson, "Augmented Reality In-Situ 3D Sketching of Physical Objects," In Proc. IUI'09, 2009.