IPSJ Transactions on Computer Vision and Applications Vol.6 68–72 (July 2014)[DOI: 10.2197/ipsjtcva.6.68]

Express Paper

Background Estimation fora Single Omnidirectional Image Sequence
Captured with a Moving Camera
Norihiko Kawai1,a)Naoya Inoue1

Tomokazu Sato1,b) Fumio Okura1,c)
Naokazu Yokoya1,e)Yuta Nakashima1,d)Received: March 14, 2014, Accepted: April 24, 2014, Released: July 25, 2014

Abstract: This paper proposes a background estimation method from a single omnidirectional image sequence forremoving undesired regions such as moving objects, specular regions, and uncaptured regions caused by the camera’s blind spot without manual specification. The proposed method aligns multiple frames using a reconstructed 3Dmodel of the environment and generates background images by minimizing an energy function for selecting a frame
for each pixel. In the energy function, we introduce patch similarity and camera positions to remove undesired regions
more correctly and generate high-resolution images. In experiments, we demonstrate the eﬀectiveness of the proposed
method by comparing the result given by the proposed method with those from conventional approaches.
Keywords: background estimation, object removal, omnidirectional image

1. Introduction
Omnidirectional image sequences captured with a moving
camera have a wide variety of applications, such as telepresence and vehicle navigation systems. For example, Google Street
View is one of the most well-known telepresence systems thatare already available to the public. A vision-based vehicle navigation system [11] is another interesting example of such applications, which uses preliminary captured omnidirectional image
sequences to estimate and track the pose of the vehicle using image matching between the sequences and an image from the invehicle camera.
Usually, omnidirectional image sequences contain undesired
regions that cause serious problems for the applications. GoogleStreet View often suﬀers from the privacy issue since its images
come with the appearances of pedestrians. The vehicle navigation
system can fail in pose estimation and tracking when the preliminarily captured omnidirectional image sequences contain a lotof moving objects (e.g., cars and pedestrians), specular surfaces,
and the camera’s blind spot. Methods for removing such undesired regions by replacing them with an estimated background as
shown in Fig. 1 are strongly required.
To remove such undesired regions and replace them withthe actual background, a straightforward approach is (1) finding undesired regions and (2) estimating their background by
1a)
b)
c)
d)
e)

Graduate School of Information Science, Nara Institute of Science andTechnology, Ikoma, Nara 630–0192, Japan
norihi-k@is.naist.jp
tomoka-s@is.naist.jp
fumio-o@is.naist.jp
n-yuta@is.naist.jp
yokoya@is.naist.jp

c 2014 Information Processing Society of JapanFig. 1 Example of removal of undesired regions by the proposed method(39th frame).

frame alignment, which makes pixelwise correspondences between background pixels in diﬀerent frames and pixels in undesired regions in a target frame.
Flores et al. [4] proposed a method to remove pedestrians from
omnidirectional images, in which regions of pedestrians are specified with pedestrian detection algorithm [8]. Our previous paper [7] proposed a method to fill in an uncaptured region caused

68

IPSJ Transactions on Computer Vision and Applications Vol.6 68–72 (July 2014)

by a blind spot of an omnidirectional camera, in which the region
is preliminarily specified with camera calibration. These methods
use the assumption that the undesired region is planar for framealignment. However, the planner assumption limits applicablescenes.
To relax the geometric constraint, some methods have been
proposed that take diﬀerent approaches to align frames in sequences of ordinary perspective images. Matsushita et al. [9] estimate motion in the undesired regions using the optical flows
around them. Bhat et al. [1] estimate a dense depth map of thetarget scene, and Granados et al. [5] use multiple homographies.
These methods fill in the manually specified regions using different frames aligned to the target one. However, the methods
that require manually specifying undesired regions may not beapplied to image sequences with a large number of frames.
Cohen [3] proposed a method to remove moving objects without specifying the target objects, assuming that the alignment
among frames is given. This method treats object removal as alabeling problem; a frame index is regarded as a label and an appropriate label that is likely to be the background is assigned toeach pixel using energy minimization. The result is generated byusing pixel values from diﬀerent frames. Specifically, the energyfunction used in this method is defined based on the variance overthe intensities of a single pixel in multiple aligned frames. The
similar energy functions are also used in Refs. [1], [5], and they
are based on the pixel diﬀerences. However, using only the pixelwise diﬀerences of intensities is sometimes insuﬃcient to remove
moving objects and generate good quality of background texture.
For omnidirectional images, Uchiyama et al. [12] remove moving objects from an omnidirectional street image without manual specification of them. This method requires multiple image
sequences taken along almost the same path because it replaces
moving objects in a target frame with background from an image captured at a position very close to that of the target frame.
Therefore, this method requires much time to be applied to large
environments.
In this paper, we propose a method for removing undesired regions in a single omnidirectional image sequence. Assuming thatthe background of the undesired regions in some frames is visible in the other frames, we replace the undesired regions with
their background without explicitly specifying them in the image
labeling framework. In order to relax the assumption of scene geometry, we adopt 3D reconstruction-based frame alignment using
the structure-from-motion and multiple-view stereo techniques.
In the proposed method, we use patch similarity in our energyfunction of the image labeling problem, rather than pixelwise differences, which empirically demonstrates better background assignment. In addition, our energy function leverages a camera
position of each frame to obtain the background with higher resolution.

sequence as shown in Fig. 2. The reasons why we employ 3D reconstruction for frame alignment are that (1) an omnidirectional
image can more easily and accurately reconstruct the geometryof the whole scene than an ordinary image because an omnidirectional camera can capture almost the entire field of view, and that(2) the 3D geometry of background, not moving objects, can beobtained by using photo consistency among frames in MVS [6].
We then generate a depth map for each frame from the reconstructed 3D geometry. Usually the reconstructed 3D geometry
does not cover an entire frame because objects or the sky thatare suﬃciently far from the camera cannot be reconstructed. For
such regions in the frame, we deem the depth values to be infinity. Next, multiple frames whose camera positions are within acertain distance threshold d from that of a target frame are extracted. The extracted frames are warped to the viewpoint of thetarget frame using the depth map as shown in Fig. 3. The black
region on the ground surface in Fig. 3 (b) corresponds to that in
the bottom of Fig. 3 (a), which is the omnidirectional camera’s
blind spot. Among the warped images, the same position’s pixels
are the corresponding pixels.

Fig. 2

Reconstructed 3D geometry of target scene.

2. Background Estimation
2.1 Frame Alignment Based on 3D Reconstruction
In the proposed method, we first estimate a camera pose of eachframe and 3D geometry of the scene by applying Structure-fromMotion [13] and Multi-View Stereo (MVS) [6] to a single imagec 2014 Information Processing Society of Japan
Fig. 3 Example of image warping. The 42nd frame is warped to the viewpoint of the 39th frame (Fig. 1 (a)).

69

IPSJ Transactions on Computer Vision and Applications Vol.6 68–72 (July 2014)

2.2 Image Composition Based on Frame Selection
Using warped omnidirectional images of the extracted frames,
an appropriate frame, which is likely to be background, for eachpixel is selected by minimizing an energy function with graphcut [2]. Given frame t as the target frame, energy function E isdefined as:
E=E1 ( f p ) + λ
E2 ( f p , fq ),(1)p∈A(p,q)∈B

where f p and fq are labels for pixel p and q, respectively, representing frame indexes from which the pixel colors are copied. Ais a set of all pixels in the omnidirectional image, and B is a set ofpairs of adjacent pixels. This pixel adjacency considers that ourimages are omnidirectional panoramas. λ is a weight to control
the contribution of the second term.
E1 is defined as a linear combination of two terms:
E1 ( f p ) = L( f p ) + αD( f p ),(2)where α is a weight to balance the two terms.
The first term L( f p ) is based on distance between camera positions of the target frame and frame f p , and defined as:L( f p ) = 1 −k,||x f p − xt ||22 + kFig. 4(3)Table 1 Parameters in experiments.

where x f p and xt are camera positions of frame f p and the target frame, respectively. The parameter k controls the influenceof the distance between camera positions. Among the warped
omnidirectional images, the resolution of texture is inversely proportional to the square of distance from each object in the background to the camera. Therefore, a high resolution backgroundimage can be obtained if we select frames that are captured fromthe cameras as close to that of the target frame as possible. To thebest of our knowledge, this is the first work that incorporates thespatial relationship among cameras into the energy function.
D( f p ) is a dissimilarity measure of patches centered at pixel p
between frame f p and other warped frames, defined as:
g∈G f p S S D( f p , g)
D( f p ) =,(4)NwhereS S D( f p , g) =||I f p (p ) − Ig (p )||22 .

(5)

p

I f p (p) and Ig (p) are the colors of pixel p in frames f p and g in
the RGB color space. The summation is calculated over pixel p
in the patch of M × M pixels centered at p. N is a normalization
constant so that the value of D( f p ) can be in [0, 1]. G f p is a set ofthe frames that given by
G f p = {g|S S D( f p , g) < median S S D( f p , i)},i∈Ft(6)where Ft is a set of the extracted frames for target frame t. Function median gives the median value. Unlike pixelwise diﬀerencesused in Refs. [1], [3], [5], the patch similarity enables us to selectframes capturing the background more correctly.
Energy E2 is a smoothness term and defined as:

c 2014 Information Processing Society of Japan

Example of generated background image with and without Poisson
blending (45th frame).

d10 [m]λ0.025

M11 [pixel]

α60k10E2 ( f p , fq ) = μ( f p , fq )(||I f p (p)−I fq (p)||+||I f p (q)−I fq (q)||), (7)where μ( f p , fq ) gives 0 when f p = fq , otherwise 1. This term prevents from frequently changing frames between adjacent pixels.
After frame selection, we obtain an omnidirectional background image by copying pixel values of the selected frames withcolor adjustment by Poisson blending [10] as shown in Fig. 4.

3. Experiments
We experimentally demonstrate the eﬀectiveness of the proposed method using an image sequence with 61 frames, comparing the obtained background images by the proposed methodwith those by conventional approaches, which use pixelwise differences of intensities or do not consider the camera positionbased term. For the experiments, we used a PC with Windows7,
Core i7-990X 3.47 GHz CPU, and 12 GB of memory. The image
sequence was captured with Point Grey’s Ladybug3 while moving almost straight. The distance of cameras between each pair ofadjacent frames was about 1 meter, and each frame was resized to1080 × 540 pixels. Parameters used in the experiments are summarized in Table 1. We generated the background images using
the following methods:
Method A Proposed method
Method B Method using pixelwise diﬀerences rather thanpatch similarity (Eq. (5))
Method C Method without the camera position-based term(Eq. (3))
Method D Median of intensities among extracted frames
Figure 5 shows the estimated background images when the70

IPSJ Transactions on Computer Vision and Applications Vol.6 68–72 (July 2014)

Fig. 5Experiments for the 41st frame. The left images are composite images and the right images are
selected frames in (c), (d), (e), and (f).

c 2014 Information Processing Society of Japan
71

IPSJ Transactions on Computer Vision and Applications Vol.6 68–72 (July 2014)

target frame is the 41st frame. Using the distance threshold, 19frames (from 32nd to 50th frame) were extracted. Figure 5 (a)shows the original 41st frame. Figure 5 (c) to (f) show the generated images and selected frames. In the following, we discuss theexperimental results.
As shown in Fig. 5 (c), the proposed method successfully removed undesired regions, i.e., moving objects, specular surfaces,
and the blind spot, and the generated background regions gave
the detailed texture. On the other hand, the method using pixelwise diﬀerences shown in Fig. 5 (d) failed in removing somemoving objects and cast shadows. From the comparison between
Fig. 5 (c) and (d), we confirmed that the patch similarity was
more eﬀective to remove moving objects than pixelwise diﬀerences. The method that does not consider camera position-basedterm gave blurring texture as shown in Fig. 5 (e). Compared tothis method, the proposed method generated clearer edges of thetiles on the ground surface. Also as seen in the right image ofFig. 5 (e), a larger number of distant frames were selected thanthe proposed method. This results in decreasing the resolutionof the generated background image. These results indicated thatconsidering the distances among camera positions was crucial forgenerating background images with high resolution. In the result
given by median of pixel values among the extracted frames asshown in Fig. 5 (f), although undesired regions were removed, theresolution of the generated image is the lowest. In the proposed
method, it took 157 seconds for the frame selection by graph cut,and 194 seconds for Poisson blending.

[6]
[7]
[8]
[9]
[10]
[11]

[12]

[13]

Jancosek, M. and Pajdla, T.: Multi-view Reconstruction Preserving
Weakly-supported Surfaces, Proc. IEEE Conf. Computer Vision andPattern Recognition, pp.3121–3128 (2011).
Kawai, N., Machikita, K., Sato, T. and Yokoya, N.: Video Completion
for Generating Omnidirectional Video without Invisible Areas, IPSJ
Trans. Computer Vision and Applications, Vol.2, pp.200–213 (2010).
Leibe, B., Leonardis, A. and Schiele, B.: Robust Object Detection
with Interleaved Categorization and Segmentation, Int. Journal ofComputer Vision, Vol.77, No.1–3, pp.259–289 (2007).
Matsushita, Y., Ge, W., Tang, X. and Shum, H.-Y.: Full-Frame Video
Stabilization with Motion Inpainting, IEEE Trans. Pattern Analysis
and Machine Intelligence, Vol.28, No.7, pp.1150–1163 (2006).
Pérez, P., Gangnet, M. and Blake, A.: Poisson Image Editing, Proc.
ACM SIGGRAPH, pp.313–318 (2003).
Uchiyama, H., Deguchi, D., Takahashi, T., Ide, I. and Murase, H.:
Ego-localization Using Streetscape Image Sequences from In-vehicle
Cmeras, Proc. IEEE Intelligent Vehicles Symposium, pp.185–190(2009).
Uchiyama, H., Deguchi, D., Takahashi, T., Ide, I. and Murase, H.: Removal of Moving Objects from a Street-view Image by Fusing Multiple Image Sequences, Proc. Int. Conf. Pattern Recognition, pp.3456–3459 (2010).
Wu, C.: VisualSFM: A Visual Structure from Motion System (2011),
available from http://ccwu.me/vsfm/.

(Communicated by Mitsuru Ambai)

4. Conclusion
In this paper, we have proposed a background estimation
method for a single omnidirectional image sequence withoutmanually specifying undesired regions. In the proposed method,
we used a reconstructed 3D geometry for frame alignment, andwe newly introduced patch similarity and camera positions. The
experimental results successfully demonstrated that the patch
similarity- and camera position-based terms in our energy function were beneficial to correct background frame selection andmaintain the resolution of background images. In future work,
we need to address the misalignment of multiple frames because
the reconstructed 3D model is not always accurate.
Acknowledgments This work was partially supported byJSPS KAKENHI Nos. 23240024 and 25540086.
References
[1]

[2]

[3]
[4]
[5]

Bhat, P., Zitnick, C.L., Snavely, N., Agarwala, A., Agrawala, N.,
Cohen, M., Curless, B. and Kang, S.B.: Using Photographs to Enhance Videos of a Static Scene, Proc. Eurographics Symp. Rendering,pp.327–338 (2007).
Boykov, Y. and Kolmogorv, V.: An Experimental Comparison ofMin-Cut/Max-Flow Algorithms for Energy Minimization in Vision,
IEEE Trans. Pattern Analysis and Machine Intelligence, Vol.26, No.9,pp.1124–1137 (2004).
Cohen, S.: Background Estimation as a Labeling Problem, Proc. Int.
Conf. Computer Vision, pp.1034–1041 (2005).
Flores, A. and Belongie, S.: Removing Pedestrians from Google Street
View Images, Proc. Int. Workshop on Mobile Vision, pp.53–58 (2010).
Granados, M., Kim, K.I., Tompkin, J., Kautz, J. and Theobalt, C.:
Background Inpainting for Videos with Dynamic Objects and a Freemoving Camera, Proc. Europian Conf. Computer Vision, pp.682–695(2012).

c 2014 Information Processing Society of Japan
72