IPSJ Transactions on Computer Vision and Applications

Vol. 2

200–213 (Nov. 2010)

Research Paper

Video Completion for Generating Omnidirectional Video
without Invisible Areas
Norihiko Kawai,†1,†2 Kotaro Machikita,†1
Tomokazu Sato†1 and Naokazu Yokoya†1
Omnidirectional multi-camera systems cannot capture entire ﬁelds of view because of their inability to view areas directly below them. Such invisible areasin omnidirectional video decrease the resulting realistic sensation experienced
when using a telepresence system. In this study, we generate omnidirectional
video without invisible areas using an image completion technique. The proposed method compensates for the change in appearance of textures caused
by camera motion and searches for appropriate exemplars considering threedimensional geometric information. In our experiments, the eﬀectiveness of our
proposed method has been demonstrated by successfully ﬁlling in missing regions in real video sequences captured using an omnidirectional multi-camera
system.

1. Introduction
Telepresence systems that enable us to experience remote sites are expected
to be used in various ﬁelds such as virtual sightseeing, education, and digitalarchiving. In these ﬁelds, omnidirectional video captured with an omnidirectional multi-camera system (OMS) composed of radially arranged cameras isused 5),6) ; however, as shown in Fig. 1, an ordinary OMS cannot capture entire
ﬁelds of view because of the absence of a camera unit to capture areas directly
below the OMS. Such invisible areas decreases the realistic sensation createdby a telepresence system. In this study, we aim at generating omnidirectional
video without invisible areas for oﬄine telepresence applications by completingthe missing regions directly below the OMS to achieve telepresence with highly
realistic sensation.
†1 Graduate School of Information Science, Nara Institute of Science and Technology†2 Research Fellow of the Japan Society for the Promotion of Science200

Fig. 1 Unwrapped spherical panoramic image (left) and a standard perspective image generated from the panoramic image (right). The black regions on these images are causedby the system’s inability to capture the area directly below it.

Numerous video completion methods have been proposed for video captured
with a monocular camera. While some have successfully completed missing regions, most of the methods assume that the appearance of objects rarely changes
between frames. It is diﬃcult for such methods to successfully complete missing
regions in omnidirectional video in which the appearance of textures may change
drastically.
In this paper, we propose a new method to complete missing regions in omnidirectional video considering the change in appearance of textures; the proposed
method is based on the following three assumptions: (1) all cameras in the given
OMS are fully calibrated; (2) the ground directly below the given OMS is roughly
planar; and (3) textures in the lower regions of a frame are captured from diﬀerent
viewpoints.
Our procedure is summarized as follows. First, we compensate for the change
in appearance of textures caused by camera motion by projecting omnidirectional
images onto the plane ﬁtted to feature points around the ground; these featurepoints are acquired using a structure-from-motion (SFM) technique. Second, using the geometric relationship between the plane and camera motion, we identify
data regions in which appropriate example textures for missing regions may exist.
Third, we complete the missing regions by minimizing an energy function basedon pattern similarity between the missing and data regions. Our contributions
are the suggestion of a novel pipeline of the above processes for generating high-c 2010 Information Processing Society of Japan

201

Video Completion for Generating Omnidirectional Video without Invisible Areas

quality omnidirectional video without invisible areas and the validation using
real video sequences, which have yet to be realized in the literature.
This paper is organized as follows. Related work is reviewed in Section 2. The
proposed method for generating omnidirectional video without invisible areas is
described in Section 3. Experiments using two video sequences and our results
are summarized in Section 4. Section 5 concludes our work and provides direction
for the future.
2. Related WorkFor video completion, image completion methods for a still image 1),2),4),8),9) canbe applied to each frame in a video; however, textures may discontinuously change
between successive frames because these methods use only spatial information.
Therefore, in video completion, we typically use not only spatial information, butalso temporal information.
Video completion methods using spatial and temporal information can be classiﬁed into two categories: one that does not use the motion information of a scene
and one that does. Methods that do not use motion information complete missing
regions using spatial-temporal image volumes as exemplars 3),16) . These methods
deﬁne the similarity of local volumes between the missing region and the rest of
the video and copy the similar local volumes or optimize a similarity-based objective function. While these methods can generate continuous textures throughsuccessive frames, an occluded scene is not always used as an exemplar, even if itappears in diﬀerent frames because of the movement of the camera. As a result,
the textures that do not actually exist behind the occlusion may appear in the
missing regions. In addition, these methods do not consider the change in theappearance of textures caused by the camera motion. In summary, using thesemethods, it is diﬃcult to successfully complete missing regions in omnidirectional
video in which the appearance of textures changes between frames.
Methods that use motion information for video completion are based on thecorrespondence of pixels between a target and other frames. The pixel correspondences are determined based on the estimated motion of objects or that ofa camera. These methods can be classiﬁed into two types: one that uses the
motion of objects on an image plane 7),10)–12),18) and one that uses the orientation

IPSJ Transactions on Computer Vision and Applications

Vol. 2

200–213 (Nov. 2010)of a camera 15),17) .
Using the motion of objects on an image plane, Litvin, et al. 10) and Matsushita,
et al. 11) have proposed methods that estimate the motion of textures in missing
regions. Speciﬁcally, they use optical ﬂows across the whole image and copy the
pixel values to missing regions from diﬀerent frames based on the estimated motion; however, it is diﬃcult for these methods to determine appropriate opticalﬂows in omnidirectional video because the appearance of textures changes between successive frames and the motion of pixels in large missing regions cannot
be accurately estimated by such two-dimensional interpolation. To treat relatively complicated motion in missing regions, other methods that use the motion
model of objects have been proposed 7),12),18) . In these methods, the motion of
objects is modeled and the moving objects are reproduced in missing regions
based on the motion model; however, the motion model is designed for cyclic
motion and these methods cannot handle complex three-dimensional motion.
Using the motion of a camera, Shen, et al. 15) and Yamashita, et al. 17) haveproposed methods that use a ﬁxed-viewpoint pan-tilt camera. Missing regions
are successfully completed using the same direction of lighting captured in neighboring frames; however, these methods cannot be applied to video captured witha freely moving camera since the ﬁxed viewpoint is the indispensable condition
to properly correspond pixels across diﬀerent frames.
To complete the missing region directly below an OMS in omnidirectional video,
it is essential to consider the appearance changes of textures, which has yet to beconsidered in the literature. Furthermore, two-dimensional optical ﬂows cannot
accurately estimate corresponding pixels in omnidirectional vision because themissing region is relatively large and the motion of corresponding pixels is quite
complex because of a characteristic of optical distortion in omnidirectional video.
To address these problems, the proposed method utilizes three-dimensional information (including the motion of a camera and the structure around the missing
region directly below the OMS) simultaneously estimated by a SFM technique foromnidirectional video. From the three-dimensional information, the appropriate
appearance of textures can be generated. In addition, the relatively accurate
correspondence of pixels between quite diﬀerent frames can be determined byconsidering the projection model of the OMS.

c 2010 Information Processing Society of Japan

202

Video Completion for Generating Omnidirectional Video without Invisible Areas

3. Generation of Omnidirectional Video without Invisible Areas
3.1 Proposed Pipeline
Figure 2 shows the pipeline of the proposed method. Following the ﬁgure,the position and posture of an OMS and the three-dimensional positions of feature points are estimated using SFM for omnidirectional video 14) (A). Next, aplane for each frame is ﬁtted to feature points around the ground (B). An image
sequence projected on each of the ﬁtted planes is generated from the omnidirectional video for compensating for the change in appearance of textures (C). A
missing region in the projected image of each frame is completed frame-by-frameby minimizing an energy function based on the similarity of textures between the
missing and data regions (D). In this process, ﬁrst, data regions in which appropriate exemplars for missing regions may exist are determined on the projected
image (D-i). The energy function is minimized by repeating two processes: aparallel search for similar textures (D-ii) and a parallel update of all pixel val-

Fig. 2 Pipeline of the proposed video completion method for omnidirectional video.

IPSJ Transactions on Computer Vision and Applications

Vol. 2

200–213 (Nov. 2010)

ues (D-iii). Finally, omnidirectional video with no invisible areas is generated
by re-projecting the completed images onto the spherical panoramic video (E).
Processes (B), (C) and (D) are described in detail in the following subsections.
3.2 Generation of Images Projected on Planes
In ordinary omnidirectional video, it is diﬃcult to complete missing regions
using originally captured textures, because the appearance of textures changes
because of camera motion. In this study, we generate an image sequence in which
the appearance of textures is compensated by projecting omnidirectional video
to a plane of each frame. The method for generating an image projected on aplane in the f -th frame is described below, covering processes (B) and (C).
Corresponding to process (B) in Fig. 2, a plane representing the ground is
ﬁtted to feature points, where such points are selected from the points obtained
through the SFM technique 14) in process (A). Here, points that satisfy thefollowing conditions are selected: (i) the point exists in the spherical area withradius l whose center is a projection center of a representative camera unit inan OMS; and (ii) the height z of the point in the world coordinate system is(p < z < p + m) (p and m are constants), as shown in Fig. 3. Once points are
selected, the expression of the plane in the world coordinate system is calculatedby the least-squares method.
Next, as shown in Fig. 4, an image is generated by projecting omnidirectional
video to the estimated plane (process (C) in Fig. 2). Here, a missing region should
be included in the generated image and the scale and orientation of textures

Fig. 3 Selection of feature points around a missing region.

c 2010 Information Processing Society of Japan

203

Video Completion for Generating Omnidirectional Video without Invisible Areas

Fig. 4 Generation of an image projected on a plane.

should be invariant through all frames for image completion described below in
Section 3.3. The method to generate the projected image of the f -th frame isdescribed in the following paragraphs.
First, the center of the projected image is determined. For a missing region to
be the center of the projected image, a point on the plane and just under the
projection center of a representative camera is set as the center. Concretely, thetransformation matrix Mf from the representative camera coordinate system of
the f -th frame to the world coordinate system is expressed as follows:⎞⎛m f 1 mf 2 mf 3 mf 4
⎜ mf 5 mf 6 mf 7 mf 8 ⎟⎟(1)
Mf = ⎜⎝ mf 9 mf 10 mf 11 mf 12 ⎠ .
0001The central coordinate of the projected image (xf , yf , zf ) can be expressed usingparameter t as follows:
⎞
⎛
⎞
⎛ ⎞ ⎛
mf 4
mf 3
xf
⎝ y f ⎠ = ⎝ mf 8 ⎠ + t ⎝ mf 7 ⎠ .
(2)
zf
mf 12
mf 11
From the expression of the plane and that of the straight line, Eq. (2), the inter-

IPSJ Transactions on Computer Vision and Applications

Vol. 2

200–213 (Nov. 2010)section point can be calculated as the central coordinate (xf , yf , zf ).
Next, basis vectors of the projected image are determined. To prevent therotation of textures in the projected image between diﬀerent frames, basis vectors(uf , vf ) in the world coordinate system are set so as to satisfy the following
equation:(3)
uf · y = 0,
where y is one of the basis vectors of the world coordinate system.
Using the calculated center position (xf , yf , zf ) and basis vectors (uf , vf ),the relationship between coordinate (uf , vf ) in the image coordinate system andthree-dimensional position (xf uv , yf uv , zf uv ) in the world coordinate system isexpressed as follows:
⎛
⎞ ⎛ ⎞
xf uv
xf
⎝ yf uv ⎠ = ⎝ yf ⎠ + ruf uf + rvf vf ,
(4)
zf uv
zf
where the size of a pixel is r × r and the center in the image is origin in the image
coordinate system. Note that the scale change of textures in the projected image
are also prevented by ﬁxing the size of a pixel through all frames.
3.3 Video Completion through Energy Minimization
In process (D) of Fig. 2, we apply an image completion method for a still image
to video frames, proceeding in a frame-by-frame manner. In this study, ourprevious image completion method for a still image 8) is employed because it cangenerate geometrically and optically natural textures by minimizing an energyfunction based on the pattern similarity that considers brightness changes oftextures. Although the original image completion method for a still image deﬁnes
data regions based on only the given image, we identify data regions pixel by
pixel from multiple frames using geometric information to preserve the temporalcontinuity.
In the following subsections, we describe the deﬁnition of the energy function,a method for determining data regions (process (D-i) in Fig. 2), and a method
for minimizing the energy function (processes (D-ii) and (D-iii) in Fig. 2).
3.3.1 Definition of Energy Function Based on Pattern Similarity
As shown in Fig. 5, a missing region in the projected image of the target (f -th)frame is completed using an energy function based on the similarity of texturesc 2010 Information Processing Society of Japan

204

Video Completion for Generating Omnidirectional Video without Invisible Areas

SSD(xf i , x̂f i ) ={I(xf i + q) − αxf i x̂f i I(x̂f i + q)}2 ,

(6)

q∈W

where I(x) represents the pixel value of pixel x, q is a shift vector in window W ,and αxf i x̂f i is the intensity modiﬁcation coeﬃcient. Note that textures arounda missing region may change because of the reﬂection of light on the groundand the shade of the camera and operator. The intensity modiﬁcation coeﬃcientadjusts the brightness of textures in data regions to that of the missing region.
In our work, αxf i x̂f i is deﬁned as the ratio of average pixel values around pixels
xf i and x̂f i as follows:
αxf i x̂f i =2q∈W I(xf i + q)
2q∈W I(x̂f i + q).

(7)

Fig. 5 Missing and data regions in projected images for completion process.

between region Ωf , including missing region Ωf in the f -th frame, and data region
Φki in the reference (k-th) frame (k = f ). Ωf is the expanded area of missing
region Ωf in which there is a central pixel, xf i , of a square window W overlapping
region Ωf . Each data region Φki corresponding to each pixel xf i in region Ωfis individually determined using the method described in Section 3.3.2. Energyfunction E is deﬁned as the weighted sum of each sum of squared diﬀerences(SSD) between textures around pixel xf i in region Ωf and x̂f i in data region
Φki ,

E=
wxf i SSD(xf i , x̂f i ),(5)
xf i ∈Ωfwhere wxf i is the weight of pixel xf i . This weight is set as 1 if xf i is inside regionΩf ∩ Ωf because pixel values in this region are ﬁxed; otherwise, wxf i = g −d(where d is the distance from the boundary of Ωf and g is a constant) because
pixel values around the boundary have higher conﬁdence than those in the centerof the missing region.
SSD(xf i , x̂f i ), which represents the similarity of textures around pixel xf i andx̂f i , is deﬁned as follows:

IPSJ Transactions on Computer Vision and Applications

Vol. 2

200–213 (Nov. 2010)

3.3.2 Determination of Data Region
In process (D-i) in Fig. 2, data regions in which appropriate exemplars for a
missing region exist are determined using the geometrical pixel-by-pixel relationship between the position and posture of an OMS, estimated in process (A), andthe planes, generated in process (B).
In the subsections that follow, we describe how to determine a data region (i.e.,
both a frame and a region) corresponding to each pixel xf i in missing region Ωfin the target f -th frame. Methods to determine data regions for an initial frame
and for subsequent frames are described.
Determination of Data Regions for Initial Frame
Data regions for the initial frame are determined through pixel-by-pixel consideration of the texture resolutions and the diﬀerence of frame numbers between
the initial and reference frames.
First, the three-dimensional coordinate of pixel xf i is re-projected on the image
plane of a camera unit in the k-th frame. As shown in Fig. 6, the pixel coordinate
pk (xf i ) on the k-th projected image is determined by computing the intersection
point of the k-th plane with the projecting line of pixel xf i .
Next, the best frame for exemplars is determined considering the position
pk (xf i ) in the projected image and the diﬀerence of frame numbers between
the target and reference frames. In the projected images, the texture resolution

c 2010 Information Processing Society of Japan

205

Video Completion for Generating Omnidirectional Video without Invisible Areas

Fig. 6 Projection to the other frame.

Fig. 7 Utilization of the result of the previous frame.

decreases as a pixel becomes farther from the center of the image, because textures of objects far from the camera are small in input images of an OMS. Thus,
textures near the center of the image should be used as exemplars for completion.
Because temporally close frames often have similar brightness values, a temporally close frame should be used. Therefore, the best frame s(xf i ) is selectedfrom candidate frames K = (k1 , . . . , kn ) by applying the following equation:

ing to xf i is calculated based on the estimated geometry.
( 2 ) Frame s(xf i ) corresponding to xf i is calculated by Eq. (8).
( 3 ) If s(xf i ) does not coincide with s(x(f −1)j ), a data region is determinedusing the same approach as with the initial frame.
( 4 ) Otherwise, the square area whose center is pixel t̃(x(f −1)j ) that was selected
for completion in the previous frame is set as data region Φs(xf i )i (seeFig. 7).
It should be noted that the size of the data region in case (4) is set smaller thanthe other case, not only to preserve the temporal continuity of textures, but also
decrease the computational cost.
3.3.3 Energy Minimization
We minimize energy function E from Eq. (5) by using a greedy algorithm.
In our deﬁnition of E, the energy for each pixel can be treated independently
if pattern pairs (xf i , x̂f i ) are ﬁxed and the change of coeﬃcient αxf i x̂f i in theiterative process of energy minimization is much smaller than the change of pixel
values in the missing region. Thus, we repeat the following two processes until theenergy converges on a minimum: a parallel search for the most similar pattern,keeping pixel values ﬁxed (D-ii); and a parallel update of all pixel values, keeping
pattern pairs ﬁxed (D-iii). Each approach is described below.
(D-ii) Parallel Search for Similar Texture Patterns
In process (D-ii) of Fig. 2, data region Φki (k = s(xf i )) is searched for positions(xf i ) = argmin( pk (xf i ) − xcenter  +λ|k − f |),

(8)

k∈K

where candidate frames K are picked such that the texture pattern of the exemplar whose center is pk (xf i ) does not include the missing region. xcenter is thecentral pixel of the k-th projected image and λ is the weight for the diﬀerence offrame numbers. The ﬁxed square area whose center is pixel ps(xf i ) (xf i ) is set asdata region Φs(xf i )i .
Determination of Data Regions for Subsequent Frames
In addition to the initial frame, data regions for subsequent frames are determined considering not only the resolution of textures and the diﬀerence of frame
numbers but also the positions of previously selected exemplars to preserve thetemporal continuity of generated textures.
In this research, the availability of the previously selected exemplars is ﬁrst
checked and a data regions is then determined by following steps.
( 1 ) Pixel p(f −1) (xf i ) (= x(f −1)j ) in the previous ((f −1)-th) frame correspond-

IPSJ Transactions on Computer Vision and Applications

Vol. 2

200–213 (Nov. 2010)

c 2010 Information Processing Society of Japan

206

Video Completion for Generating Omnidirectional Video without Invisible Areast(xf i ) of the most similar pattern corresponding to pixel xf i , keeping pixel values
I(xf i ) ﬁxed. t(xf i ) is determined pixel by pixel in parallel as follows:t(xf i ) = x̂f i = argmin(SSD(xf i , x)).

(9)

x∈Φki

(D-iii) Parallel Update of Pixel Values
In process (D-iii) of Fig. 2, all pixel values I(xf i ) are updated in parallel so
as to minimize the energy, keeping the similar pattern pairs ﬁxed. To calculate
pixel values I(xf i ), we ﬁrst resolve energy E into element energy E(xf i ) for eachpixel xf i in missing region Ωf . Element energy E(xf i ) can be expressed in termsof the pixel values of xf i and f (xf i + q) − q, and coeﬃcient α as follows:w(xf i +q) {I(xf i ) − α(xf i +q)t(xf i +q) I(t(xf i + q) − q)}2 . (10)E(xf i ) =computational cost and avoids local minima.
4. Experiments and Results
In this section, we demonstrate the eﬀectiveness of the proposed method bycompleting missing regions in two video sequences. For our experiments, weused a PC (CPU: Xeon 3.0 GHz, Memory: 8 GB), and as shown in Fig. 8, theLadybug 13) OMS. Ladybug has six radially located camera units, their positions
and postures ﬁxed. Each camera unit can acquire 768×1,024 images. Parametersin our experiments were empirically determined and summarized in Table 1.
We determined a missing region in each image through the combination of
two regions: (1) the region unprojected from the six input images directly belowq∈W

The relationship between energy E and element energy E(xf i ) for each pixel isthen written as follows:E=
E(xf i ) + C,
(11)
xf i ∈Ω

where C, the energy of pixels in region Ωf ∩ Ωf , is treated as a constant, because
pixel values in the region and all the pattern pairs are ﬁxed in process (D-iii).
Therefore, by minimizing element energy E(xf i ), total energy E can be minimized. If we assume that the change of αxf i t(xf i ) is much smaller than that ofpixel value I(xf i ), by diﬀerentiating E(xf i ) with respect to I(xf i ), each pixel
value I(xf i ) in missing region Ωf can be calculated in parallel as follows:I(xf i ) =q∈W w(xf i +q) α(xf i +q)t(xf i +q) I(t(xf i + q) − q)q∈W w(xf i +q)

.

(12)

In addition to the energy minimization algorithm described above, a coarseto-ﬁne approach is also employed to eﬃciently avoid local minima. An image
pyramid is generated and energy minimization processes (D-ii) and (D-iii) arerepeated from higher-level to lower-level layers. Good initial values are seededin the lower layers by projecting results from the higher layer. In addition, thecorrespondences of pixels are inherited and new data regions are set so that thecorresponding pixels are the center of the new data regions. This both decreases

IPSJ Transactions on Computer Vision and Applications

Vol. 2

200–213 (Nov. 2010)

Fig. 8 Omnidirectional multi-camera system “Ladybug”.

Table 1 Experimental parameters.
For generation of panoramic and projected images
Resolution of panoramic image
2,048 × 1,024 (pixel)
Resolution of projected image
1,200 × 1,200 (pixel)
Range for selecting feature points−1,000 (mm) < z < height of camera
l = 6,000 (mm)Pixel size
10 (mm) × 10 (mm)Weight λ2.0
For energy minimization
Scale Level (Coarse-to-ﬁne approach)
1/4
1/2
1
g in weight w
1.1
Window size
51 × 51 (pixel)
Data region (case (a))
11 × 11 (pixel)
5 × 5 (pixel)
5 × 5 (pixel)
3 × 3 (pixel)
3 × 3 (pixel)
3 × 3 (pixel)
Data region (case (b))

c 2010 Information Processing Society of Japan

207

Video Completion for Generating Omnidirectional Video without Invisible Areas

Fig. 9 First frames of input video sequences obtained by six camera units; top images are forscene (1) and bottom images are for scene (2).
1st frame

101st frame

201st frame

301st frameFig. 10 Projected Images; top images are for scene (1) and bottom images are for scene (2).

Ladybug; and (2) the regions projected from six input images that were manuallymasked, because they contain such unneeded regions as the Ladybug equipment
and the shadow of the OMS operator. The following subsections describe ourexperiments for two video sequences in detail.
4.1 Acquisition of Input Information
In our experiments, two video sequences were captured from Ladybug while theperson on whom it was mounted was walking. Each video consists of 301 framesof omnidirectional video, totaling 1,806 images across the six cameras. Figure 9shows the ﬁrst frames of two video sequences captured by Ladybug. In scene (1),the top six images of Fig. 9, the ground is almost planar through all frames. In
scene (2), the bottom six images of Fig. 9, the ground is slightly rugged becauseof various stones, and there are steps near missing regions at the beginning andend of the scene.
In the following, experiments of completion for images projected on planes anda prototype telepresence system using the omnidirectional video without invisible
areas are presented.
4.2 Completion for Projected Images
We generated 1,200×1,200 projected images using the method described abovein Section 3.2. Of the 301 frames, four selected images of scenes (1) and (2) are

IPSJ Transactions on Computer Vision and Applications

Vol. 2

200–213 (Nov. 2010)

shown in Fig. 10. The round black regions shown are unprojected regions below
the Ladybug. As shown in the ﬁgure, textures of tiles in scene (1) are uniform
regardless of the position of pixels; furthermore, textures of the same objects do
not rotate in each frame. As a result, we conﬁrm that appropriate exemplars
were generated in the projected images. Similarly, textures of stones in scene (2)are also the same and do not rotate from frame to frame; however, the brightnessof stones are diﬀerent between the left and right sides of the missing regions.
Furthermore, the brightness of the stones changes dramatically from frame toframe. Given that strong sunlight comes from the right, we observe that thebrightness of objects has changed because of the reﬂection of the sunlight on thestones.
Next, a missing region in each projected image was completed. Figures 11
and 12 show a variety of image results for the projected images of the 11th frameof scene (1) and the 71st frame of scene (2). Figures 11 (a) and 12 (a) show thetarget frames in which the missing region (shown in red) exists. Figures 11 (b)and 12 (b) show the central pixels of the data regions in the reference frames
corresponding to pixel location (600, 600) in the target frames. Figures 11 (c)and 12 (c) show the results of copying values of central pixels in data regions

c 2010 Information Processing Society of Japan

208

Video Completion for Generating Omnidirectional Video without Invisible Areas

(a)

(b)

(a)

(b)

(c)

(d)

(c)

(d)

Fig. 11 Comparison of results for the 11th frame: Image (a) is the target image with a missing
region; image (b) shows the center of the data region in the 67th frame corresponding
to pixel (600, 600) of the target image; image (c) is the resultant image produced bycopying values of central pixels in data regions; and image (d) is the resultant image
produced by the proposed method.

IPSJ Transactions on Computer Vision and Applications

Vol. 2

200–213 (Nov. 2010)

Fig. 12 Comparison of results for the 71st frame: Image (a) is the target image with a missing
region; image (b) shows the center of the data region in the 15th frame corresponding
to pixel (600, 600) of the target image; image (c) is the resultant image producedby copying values of central pixels in data regions; image (d) is the resultant image
produced by the proposed method.

c 2010 Information Processing Society of Japan209

Video Completion for Generating Omnidirectional Video without Invisible Areas

12th frame

13th frame

14th frame

15th frame

72nd frame

73rd frame

74th frame

75th frameFig. 13 Completion results for successive frames of scene (1).

Fig. 14 Completion results for successive frames of scene (2).

without the energy minimization process described above in Section 3.3. In
Fig. 11 (c), the geometrical and optical disconnection of textures in the boundaryof the missing region appears. We conclude that planes ﬁtted to feature points
contained some errors due to the ground not being absolutely planar. As forFig. 12 (c), we also conﬁrm the geometrical disconnection of textures in the middleof the missing region. In this scene, textures in the left side of the missing region
are copied from earlier frames and those in the right side are copied from later
frames. Therefore, the large disconnection appears. Conversely, Figs. 11 (d) and
12 (d) show resultant images in which textures are continuously connected on
the boundary and middle of the missing region. As a result, natural textures are
generated in the missing region. From these results, we conﬁrm that the energy
minimization process using pattern similarity considering brightness change of
textures is eﬀective for generating natural textures even if the ground is not
absolutely planar.
Figures 13 and 14 show projected images of successive frames (the 12th to the15th frames of scene (1) and the 72nd to the 75th frames of scene (2)). From theseﬁgures, the missing region in each frame of each scene was successfully completed.
In addition, textures in the missing region change smoothly between successiveframes and plausible videos are generated; however, as shown in Fig. 15, for scene(2), unnatural texture changes appeared from the 298th to the 301st frames. The
completed textures are gradually distorted through these frames. We conclude
that correspondences of pixels were inaccurate because the angle of the plane ofeach frame diﬀered from the ground truth due to steps near the missing region.
To overcome these problems in the future, we should use a more adaptive planeﬁtting approach that considers the range for feature points. In addition, to obtain
good results for more complex scenes, the detailed three-dimensional surface
should be ﬁtted to feature points.
We discuss the processing time. For scene (1), it took 722 seconds to complete
the initial frame (i.e., determine data regions and minimize the energy function)
and an average of 513 seconds to complete each subsequent frame. Similarly forscene (2), it took 776 seconds to complete the initial frame and an average of 523
seconds to complete the each subsequent frame. We observe a decrease in computational cost from the initial frame to subsequent frames, because completion
results for a previous frame are reused.
4.3 Omnidirectional Telepresence without Invisible Areas
The eﬀectiveness of the proposed method is fully realized by creating a pro-IPSJ Transactions on Computer Vision and Applications

Vol. 2

200–213 (Nov. 2010)

c 2010 Information Processing Society of Japan

210

Video Completion for Generating Omnidirectional Video without Invisible Areas

298th frame

299th frame

300th frame

301st frameFig. 15 Results showing problems with our proposed method.

totype telepresence system using omnidirectional video in which missing regions
are ﬁlled in with the completed images shown above in the previous section.
Top images of Figs. 16 and 17 show omnidirectional panoramic images witha missing region; bottom images of these same ﬁgures show the omnidirectional
panoramic images without invisible areas, as generated by projecting completed
images (see Figs. 11 (d) and 12 (d)) onto the panoramic images. Note that thesepanoramic images are 2,048 × 1,024 images. Figures 18 and 19 show example
user views in the telepresence system using the above panoramic images. Comparing images with and without the missing regions, we conﬁrm that realistic
sensation is drastically increased by the proposed method; however, we also conﬁrm that textures in the missing regions are slightly blurry. In the completionof the proposed method, the texture resolution for exemplars becomes small because the textures are captured from diﬀerent viewpoints away from the camera
position of the target frame. To increase the resolution of generated textures, weconsider that a super-resolution method may be useful.
5. Conclusion
In this paper, we proposed a novel video completion pipeline for omnidirec-

IPSJ Transactions on Computer Vision and Applications

Vol. 2

200–213 (Nov. 2010)

Fig. 16 Panoramic image of the 11th frame of scene (1) before and after video completion.

tional video. This pipeline compensated for the change in texture appearance
and determined appropriate searching regions using three-dimensional geometric
information for successful video completion.
In experiments, missing regions in images projected on planes were successfullycompleted and the eﬀectiveness of an energy minimization based on pattern similarity was demonstrated by comparing results of the proposed method with andc 2010 Information Processing Society of Japan

211

Video Completion for Generating Omnidirectional Video without Invisible Areas

Fig. 17 Panoramic image of the 71st frame of scene (2) before and after video completion.

without an energy minimization process. In addition, we conﬁrmed that texturesin missing regions change smoothly between successive frames. Furthermore, omnidirectional telepresence without missing regions was achieved; we improved therealistic sensation in such an oﬄine telepresence system by successfully ﬁlling in
missing regions.
In the proposed method, we assume the ground around a missing region is

IPSJ Transactions on Computer Vision and Applications

Vol. 2

200–213 (Nov. 2010)

Fig. 18 Examples of looking around scene (1) using omnidirectional video with and without
missing regions. Note that these are the 11th frames (top), 101st frames (middle) and201st frames (bottom).

c 2010 Information Processing Society of Japan

212

Video Completion for Generating Omnidirectional Video without Invisible Areasroughly planar. Although such an assumption is applicable to many scenes, acompletion method that can be applied to a wider variety of shapes of missing
regions, including stairs and rugged ground, should be developed. Furthermore,a method for increasing the resolution of textures in missing regions should be
explored to reduce the blurriness of resultant images.
Acknowledgments This research was partially supported by the Ministry of
Education, Culture, Sports, Science and Technology, Grant-in-Aid for ScientiﬁcResearch (A), 19200016, and JSPS Fellows, 21-8045.
References

Fig. 19 Examples of looking around scene (2) using omnidirectional video with and without
missing regions. Note that these are the 71st frame (top), 101st frames (middle) and201st frames (bottom).

IPSJ Transactions on Computer Vision and Applications

Vol. 2

200–213 (Nov. 2010)

1) Bertalmio, M., Sapiro, G., Caselles, V. and Ballester, C.: Image Inpainting, ACM
Trans. on Graphics, pp.417–424 (2000).
2) Bertalmio, M., Bertozzi, A. and Sapiro, G.: Navier-Stokes, Fluid Dynamics, andImage and Video Inpainting, Proc. IEEE Conf. on Computer Vision and Pattern
Recognition, Vol.1, pp.355–362 (2001).
3) Cheung, V., Frey, B. and Jojic, N.: Video epitomes, Int. Journal of Computer
Vision, Vol.2, No.2, pp.141–152 (2008).
4) Criminisi, A., Pérez, P. and Toyama, K.: Region Filling and Object Removal by
Exemplar-Based Image Inpainting, IEEE Trans. on Image Processing, Vol.13, No.9,
pp.1200–1212 (2004).
5) Hori, M., Kanbara, M. and Yokoya, N.: Novel Stereoscopic View Generation by
Image-Based Rendering Coordinated with Depth Information, Proc. Scandinavian
Conf. on Image Analysis, pp.193–202 (2007).
6) Ikeda, S., Sato, T. and Yokoya, N.: Immersive Telepresence System with a Locomotion Interface Using High-resolution Omnidirectional Videos, Proc. IAPR Conf.
on Machine Vision Applications, pp.602–605 (2005).
7) Jia, J., Tai, Y., Wu, T. and Tang, C.: Video Repairing under Variable Illumination
Using Cyclic Motions, IEEE Trans. on Pattern Analysis and Machine Intelligence,
Vol.28, No.5, pp.832–839 (2006).
8) Kawai, N., Sato, T. and Yokoya, N.: Image Inpainting Considering Brightness
Change and Spatial Locality of Textures and Its Evaluation, Proc. Pacific-Rim
Symp. on Image and Video Technology, pp.271–282 (2009).
9) Komodakis, N. and Tziritas, G.: Image Completion Using Global Optimization,
Proc. IEEE Conf. on Computer Vision and Pattern Recognition, pp.442–452 (2006).
10) Litvin, A., Konrad, J. and Karl, W.: Probabilistic video stabilization using Kalmanfiltering and mosaicking, Proc. SPIE Electronic Imaging, Vol.5022, pp.663–674(2003).
11) Matsushita, Y., Ofek, E., Ge, W., Tang, X. and Shum, H.: Full-Frame Video Stabilization with Motion Inpainting, IEEE Trans. on Pattern Analysis and Machinec 2010 Information Processing Society of Japan

213

Video Completion for Generating Omnidirectional Video without Invisible Areas

Intelligence, Vol.28, No.7, pp.1150–1163 (2006).
12) Patwardhan, K., Sapiro, G. and Bertalmio, M.: Video Inpainting Under Constrained Camera Motion, IEEE Trans. on Image Processing, Vol.16, pp.545–553(2007).
13) Point Grey Research Inc.: Ladybug.
http://www.ptgrey.com/products/spherical.asp
14) Sato, T., Ikeda, S. and Yokoya, N.: Extrinsic Camera Parameter Recovery from
Multiple Image Sequences Captured by an Omni-directional Multi-camera System,
Proc. European Conf. on Computer Vision, Vol.2, pp.326–340 (2004).
15) Shen, Y., Lu, F., Cao, X. and Foroosh, H.: Video Completion for Perspective
Camera Under Constrained Motion, Proc. IEEE Int. Conf. on Pattern Recognition,
pp.63–66 (2006).
16) Wexler, Y., Shechtman, E. and Irani, M.: Space-Time Completion of Video, IEEE
Trans. on Pattern Analysis and Machine Intelligence, Vol.29, No.3, pp.463–476(2007).
17) Yamashita, A., Fukuchi, I., Kaneko, T. and Miura, T.: Removal of Adherent Noises
from Image Sequences by Spatio-temporal Image Processing, Proc. IEEE Int. Conf.
on Robotics and Automation, pp.2386–2391 (2008).
18) Zhang, Y., Xiao, J. and Shah, M.: Motion Layer Based Object Removal in Videos,Proc. IEEE Workshops on Application of Computer Vision, Vol.1, pp.516–521(2005).

(Received January 29, 2010)(Accepted August 5, 2010)(Released November 15, 2010)(Communicated by Hajime Nagahara)

Kotaro Machikita received his B.E. degree in electrical andelectronic engineering from Kobe University in 2006. He received
his M.E. degree in information science from Nara Institute of Science and Technology in 2009. He has been working at SANYOElectric Co., Ltd. since 2009.

Tomokazu Sato received his B.E. degree in computer and system science from Osaka Prefecture University in 1999. He received
his M.E. and Ph.D. degrees in information science from Nara Institute of Science and Technology in 2001 and 2003, respectively.
He has been an assistant professor at Nara Institute of Scienceand Technology since 2003.

Naokazu Yokoya received his B.E., M.E., and Ph.D. degrees
in information and computer science from Osaka University in
1974, 1976, and 1979, respectively. He joined ElectrotechnicalLaboratory (ETL) in 1979. He was a visiting professor at McGill
University in 1986–1987 and has been a professor at Nara Instituteof Science and Technology since 1992.

Norihiko Kawai received his B.E. degree in informatics andmathematical science from Kyoto University in 2005. He received
his M.E. and Ph.D. degrees in information science from Nara Institute of Science and Technology in 2007 and 2010, respectively. He
is currently a research fellow of the Japan Society for the Promotion of Science. He is also a postdoctoral fellow at the Universityof California at Berkeley.

IPSJ Transactions on Computer Vision and Applications

Vol. 2

200–213 (Nov. 2010)

c 2010 Information Processing Society of Japan