Motion parallax representation for indirect augmented reality
Fumio Okura∗

Yuya Nishizaki

Tomokazu Sato†

Norihiko Kawai†

Naokazu Yokoya†

Osaka UniversityNAISTNAISTNAISTNAISTA BSTRACT
Indirect augmented reality (IAR) presents pre-generated augmented
images for achieving high-quality geometric and photometric registration between pre-captured images and virtual objects. Meanwhile, IAR causes spatial inconsistency between the real world andpresented images when users move from the location where the realscene was captured. This paper describes a novel way to address thespatial inconsistency; namely, enabling viewpoint change in IAR.
The key idea of this study is to employ an image-based rendering
technique using pre-captured multi-view omnidirectional images toprovide free-viewpoint navigation. For a pilot study, we have developed an IAR system representing a motion parallax effect using
an optical-flow-based camera motion estimation.
Index Terms: H.5.1 [Information Interfaces and Presentation]:
Multimedia Information Systems—Artificial, augmented, and virtual realities
1 I NTRODUCTION
In augmented reality (AR) applications using mobile devices, geometric registration that accurately aligns virtual objects onto realscenes is one of the most essential requirements. As a unique approach to address the registration problems, indirect AR (IAR) has
been proposed [8]. This approach captures an omnidirectional image of a target scene and superimposes virtual objects on the imagein advance. The online process crops the pre-generated augmented
image according to the device orientation obtained by internal sensors and presents the cropped image to users. Unlike traditional AR
that utilizes real scenes captured in real-time, IAR never produces
jitters between pre-captured scenes and virtual objects.
However, IAR causes inconsistency between the real world andpresented images, which can be categorized into spatial and temporal ones [5]. One major issue of IAR is spatial inconsistency;
i.e., traditional IAR cannot present scenes from viewpoints otherthan where omnidirectional images have been captured. This study
proposes an IAR approach presenting scenes from viewpoints different from those for input images according to user’s device poses,
which allows users to move around, while preserving the advantage
of jitter-free AR. To achieve this, we employ image-based rendering using omnidirectional images captured at multiple locations.
For the first step of the study, we have developed a prototype
system representing relatively small viewpoint change, i.e., motion parallax. We propose an optical-flow-based estimation of relative motion of the device camera, which is robust to texture-lessscenes compared with visual simultaneous localization and mapping (vSLAM)-based camera pose estimation approaches.
2 IAR WITH M OTION PARALLAX R EPRESENTATION
The proposed approach generates IAR scenes with viewpointchanges using image-based rendering. Our prototype system does
not target large viewpoint change but handles small motion of theviewpoint such as motion parallax by optical-flow-based camera
motion estimation. Similarly to previous IAR systems [5, 8], ourapproach consists of offline and online processes.
∗ e-mail: okura@am.sanken.osaka-u.ac.jp† e-mail: {tomoka-s, norihi-k, yokoya}@is.naist.jp

Figure 1: Examples of input omnidirectional images and a reconstructed 3D model.
3D surface(−1 ,Virtual viewpoint( − 1 th frame)−1−1 )( ,[ | ])Virtual viewpoint( th frame)

Figure 2: Relative camera motion estimation.

2.1 Offline process: 3D reconstruction
In offline processes, omnidirectional images of a target scene arecaptured at various locations in an area where users are supposed
to move in a specific application. The omnidirectional images areutilized to reconstruct camera poses and a 3D model of the real environment using structure from motion and multi-view stereo. Weemployed VisualSFM [9] and CMPMVS [3] for the reconstruction.
A reconstructed 3D model and camera pose information employedfor our experiment are shown in Figure 1.
2.2 Online process 1: Camera motion estimation
An augmented image from the device viewpoint is displayed on
the mobile device based on real-time camera pose estimation andimage-based rendering.
VSLAM can be utilized for acquiring device viewpoint, wherestate-of-the-art methods mark high localization accuracy. However,the camera position estimated by vSLAM does occasionally notwell correspond to the short-term relative motion of the camera because of the lack of robustness to texture-less scenes or behaviors
during the cancellation of accumulative errors. Users recognize thedifference of relative motion as the spatial (i.e. geometric) inconsistency between IAR scenes and the real world. For displaying motion parallax, it is important to estimate camera position well corresponding to the relative motion of the device rather than achieving
high localization accuracy.
We reconstruct relative camera motion between two consecutive
frames (see Figure 2). We directly employ device orientation R
acquired from an inertial orientation sensor. The relative changeof the camera position t is estimated using optical flows acquired
from the images captured by the device camera. In a similar situation, Ventura et al. [7] estimated poses of upright panoramas by
reducing two degrees-of-freedom (DoF); meanwhile we estimate 3DoF relative camera translation under the constraint of given 3-DoF
orientation. In our case, the position can be estimated using minimal two pairs of flows; therefore the system performs robustly inenvironment with less textures.

Position A

Virtual object

Position B

Position C

Input images
TraditionalIAR(w/o motionparallax)

Position A(original positon)
Position B
Position C

ProposedIAR(w/ motionparallax)

Figure 3: A top view of the experimental environment.

Figure 4: Results with/without motion parallax.

Let the corresponding points between the (i − 1)-th and i-thframes be (ui−1 , vi−1 ) and (ui , vi ) assuming the intrinsic camera parameter is the identity matrix. The scene depths of these points aredenoted as di−1 and di , respectively. We assume that di−1 , the scene
depth in the (i − 1)-th frame, is known based on the reconstructed
3D model and the camera pose in the (i − 1)-th frame. The cameratranslation t as well as the depth value di in the current frame are
estimated based on the following equation.
 di ui  di−1 ui−1 ff
 di vi 
d v
R t 
 f =
 i−1f i−1  .
(1)
0 1  ddii−111
In our implementation, t is estimated through a RANSAC-based
outlier rejection process inputting optical flows computed by LucasKanade method [4].
Notable accumulative errors in the estimated camera pose canappear by only employing a sequence of relative motion. To mitigate the accumulative errors, here we add ad-hoc implementations
by assuming that users do not go far from the original position in a
specific application in which virtual objects are seen from a limited
area. Instead of directly employing t estimated using optical flows,
the position of the viewpoint is modified to gradually get closer tothe original position.
2.3 Online process 2: Image-based rendering with
virtual object superimposition
View-dependent texture mapping (VDTM) [2], an image-based rendering technique utilizing multi-view images and a 3D model, generates a scene from the estimated position of the device camera.
Onto the image generated by VDTM, virtual objects are then rendered through a traditional graphics pipeline. The relative pose ofvirtual objects and the reconstructed 3D models are fixed; therefore
geometric misalignment between the virtual objects and the precaptured scenes does not occur principally. Note that, to generate
more photorealistic IAR scenes, static virtual objects can be superimposed offline onto pre-captured omnidirectional images [6].

Experimental results. Figure 4 shows IAR scenes generated by theproposed and a traditional IAR [8], where the virtual viewpoint oftraditional IAR was fixed on Position A. The traditional IAR does
not represent the viewpoint change. The IAR scenes generated by
the proposed system include a motion parallax effect, while the position of the virtual viewpoints mostly corresponded to the actual
viewpoint (compare with input images). Our VDTM implementation performed at approximately 40 fps. Note that PTAMM [1], avSLAM algorithm, failed to perform the localization in this environment due to the lack of informative feature points.

3 P RELIMINARY E XPERIMENT
We conducted a preliminary experiment to confirm whether the motion parallax is plausibly represented by the proposed system.
Experimental conditions. We captured 30 images using an omnidirectional camera, Ladybug3 (Point Grey Research, Inc.), in anarea where users are supposed to move. A virtual object of an ancient tower was superimposed onto the VDTM images. The online process was performed on a mobile device, Surface Pro 3 (Microsoft Corporation; Core i7 1.7 GHz, 8GB RAM), and the resolution of IAR scenes was 640×480 pixels. We moved the device
from the original position (Position A) to Position C as illustrated
in Figure 3. In the initial state, the device was mounted on a jig
placed at the original position and orientation.

4 C ONCLUSIONS AND F UTURE W ORK
Traditional IAR systems [5, 8] do not represent user’s viewpointchange; therefore large spatial inconsistency can appear between
IAR scenes and the real world when users change the viewpoint.
The proposed system represents the viewpoint change in IAR
scenes using image-based rendering, where a virtual viewpoint isacquired based on a relative motion estimated using an inertial sensor and optical flows. The experiment showed that the proposed
approach presented IAR scenes including the motion parallax according to the user’s behavior. We plan to evaluate the performanceof the proposed approach through a subjective experiment, and actualize IAR systems enabling viewpoint change in wider areas.
ACKNOWLEDGEMENTS
This work was supported in part by MIC/SCOPE #162107002.
R EFERENCES[1] R. Castle, G. Klein, and D. W. Murray. Video-rate localization in multiple maps for wearable augmented reality. In Proc. 12th IEEE Int’l
Symp. on Wearable Computers (ISWC’08), pages 15–22, 2008.
[2] P. E. Debevec, C. J. Taylor, and J. Malik. Modeling and renderingarchitecture from photographs: A hybrid geometry- and image-basedapproach. In Proc. ACM SIGGRAPH’96, pages 11–20, 1996.
[3] M. Jancosek and T. Pajdla. Multi-view reconstruction preserving
weakly-supported surfaces. In Proc. 2011 IEEE Conf. on Computer
Vision and Pattern Recognition (CVPR’11), pages 3121–3128, 2011.
[4] B. D. Lucas and T. Kanade. An iterative image registration techniquewith an application to stereo vision. In Proc. Int’l Joint Conf. on Artificial Intelligence (IJCAI’81), pages 674–679, 1981.
[5] F. Okura, T. Akaguma, T. Sato, and N. Yokoya. Addressing temporal inconsistency in indirect augmented reality. Multimedia Tools andApplications, Online first, DOI: 10.1007/s11042-015-3222-0, 2016.
[6] F. Okura, M. Kanbara, and N. Yokoya. Mixed-reality world exploration using image-based rendering. ACM Journal on Computing andCultural Heritage, 8(2):9, 2015.
[7] J. Ventura and T. Höllerer. Structure and motion in urban environments
using upright panoramas. Virtual Reality, 17(2):147–156, 2013.
[8] J. Wither, Y.-T. Tsai, and R. Azuma. Indirect augmented reality. Computers & Graphics, 35(4):810–822, 2011.
[9] C. Wu. Towards linear-time incremental structure from motion. InProc. 2013 Int’l Conf. on 3D Vision (3DV’13), pages 127–134, 2013.