Removal of Moving Objects and Inconsistencies
in Color Tone for an Omnidirectional
Image Database
Maiya Hori, Hideyuki Takahashi, Masayuki Kanbara and Naokazu Yokoya
Nara Institute of Science and Technology (NAIST),
8916-5 Takayama, Ikoma, Nara, Japan{maiya-h, kanbara, yokoya}@is.naist.jp

Abstract. This paper proposes a method for removing image inconsistencies which occur by an existence of moving objects or a change ofillumination condition when an omnidirectional image database is generated. The database is used for archiving an outdoor scene in wide areas or
generating novel view images with an image-based rendering approach.
In related work, it is diﬃcult to remove moving objects in an outdoor
environment where illumination condition drastically changes, and to
remove inconsistencies of color tone of images which included moving
objects. The proposed method iterates the two processes which are theestimation of candidate region of moving objects and the achievementof color consistency to split regions. The color consistency is achievedby estimating linear color transformation parameters which change ahistogram of an input image to that of the standard image.

1

Introduction

In a panoramic image view system such as Google Street View, a user can see
images from a street using omnidirectional images. Some studies [1, 2] which
use omnidirectional images can also generate a novel view with an image-based
rendering (IBR) approach in an outdoor environment. These studies use an image database which consists of many images captured with an omnidirectional
camera. When the image database is generated from many images which arecaptured at diﬀerent position and time, these images have inconsistencies which
occur by an existence of moving objects or a change of illumination condition.
As a method for removing moving objects in images, a technique of compensation using images which is captured at diﬀerent time is used usually. A
color tone diﬀers only in the complemented regions when a simple compensation approach is applied using an image whose color tone is diﬀerent from theoriginal image. Shadow in an image is also treated as a region of moving object.
A technique of removing shadows corresponding to change of illumination [3] isproposed by estimating a light source condition. However, it is diﬃcult to detectan object whose color is similar to the color of background.
On the other hand, as one of the methods for removing an inconsistency incolor tone of images, a technique which handles an image whose color tone isdiﬀerent locally [4] is proposed. In this technique, color consistency in images isachieved by splitting the input image to small regions. However, the method isdiﬃcult to be applied when moving objects exist in an image, because a static
environment is assumed in this method. The work [5] which removes moving
objects after correcting a color tone detects moving objects with a slight changeof illumination condition. This study cannot be applied to the case that an
illumination condition changes drastically such as an outdoor scene.
If the conventional approaches are applied to the images captured in an
outdoor environment, there are many problems such as the existence of moving objects and the change of illumination conditions. Furthermore, in orderto capture in an outdoor environment eﬃciently, when omnidirectional camera is used, moving objects are easy to be captured and change of illumination
condition is large. This paper proposes a method for removing inconsistencies
among omnidirectional images captured at diﬀerent positions and times. This
research assumes an outdoor environment is a target of an omnidirectional image database. We use omnidirectional images which are captured several times
along similar paths with a car-mounted omnidirectional camera. We assume thatthese images add position and posture information, and are captured densely. To
remove inconsistencies of omnidirectional images, the proposed method iterates
the two processes which are removal of moving objects and achievement of color
consistency of images. The iteration of two processes can narrow down a regionof moving objects and omnidirectional images with consistency of color tone are
generated.

2

Removal of moving object and inconsistencies in color
tone for omnidirectional image database

2.1

Outline of the proposed method

This section describes a method for generating an omnidirectional image databasewith consistency of images. Fig. 1 shows the ﬂow diagram of the proposed
method. The proposed method consists of three principal processes.
First, omnidirectional image sequences with camera positions and postures
are acquired in an outdoor environment. As the pre-processing, we remove theregions which cannot be corrected by linear color transformation in phase (A). In
the iteration processing (phase (B)-(D)), two processes which are estimation of
linear color transformation parameters in phase (B) and estimation of candidate
region of moving object in phase (C) are iterated by splitting regions in phase(D). Finally, as the post-processing, candidate regions of moving objects arecompensated in phase (E). Details of each phase are given below.
2.2Pre-processing: removal of exceptive regions for iteration
process

When inconsistencies in color tone are removed, we assume that color tone ofimages can be changed by linear color transformation except in a region whereOmnidirectional images

Pre-processing:(A) Removal of exceptive regions
for iteration process

Iteration processing:(B) Estimation of linear color transformation parameters(C) Estimation of candidate region of moving objects

Yes

Candidate region of moving objects is below certain size or
Iteration times is over certain times

No(D) Split of object region

Post-processing:(E) Compensation of moving objects

Omnidirectional image databasewhich consists of many consistent images

Fig. 1. Flow diagram of proposed method

moving object is observed. It cannot assume that color of the sky region is transformed to the color of standard image by linear transformation since intensityof the pixel in the region is often saturated or background image cannot be deﬁned due to cloud. In this study, the sky region in the omnidirectional images is
detected and removed in advance. These can be realized by using the previous
methods [6, 7].
2.3

Iteration processing: estimation of candidate region of moving
object and linear color transformation parameters

This section explains the method to realize a consistency of omnidirectional
images with the following iteration processes.
Estimation of linear color transformation parameters for color correction A color tone of input omnidirectional images is corrected with a standard
image. The standard image which is suitable for an IBR approach is selectedmanually. Since input images have a few disparities when they are captured
with a moving vehicle, the color transformation parameters cannot be estimatedfor every pixel. In this research, to reduce the inﬂuence of the disparities, histograms of the standard image and the input image are used for estimating colortransformation parameters.

We assume that it is possible to correct the color tone of an image witha linear color transformation if a diﬀerent appearance depends on changes inilluminate conditions. An equation which changes intensity in image is shown inEq.(1).
(1)
I  (x, y) = pa I(x, y) + pb ,where linear color transformation parameters are pa , pb , I(x, y) is intensity at(x, y) of input image and I  (x, y) is intensity at (x, y) after correcting color. The
color transformation parameters are estimated in such way that the similarity
value of histogram between the input image and the standard image becomes
the highest. In this method, Bhattacharyya coeﬃcient(2) is used as a similarityof histogramsγ. Bhattacharyya coeﬃcient has the advantage of robustness foroutlier by using inner product.
hA(i) hB(i) ,
(2)
γ=
i

where hA(i) shows a frequency of intensity i in image A, and hB(i) shows thatin image B. Histograms are generated in each spectrum.
Color correction based on robust estimation Color of moving objects can
not be corrected with the linear color transformation. Then, after removing moving objects, to correct a color tone is desired. The region of the moving objects
is diﬃcult to extract from one image or some images which have a diﬀerent color
tone. In this research, we try to estimate color transformation parameters by
eliminating the moving objects with a LMeds [8] approach as a robust estimation. In order to estimate color transformation parameters based on the LMedS
method, the candidate region of moving objects needs to be less than half of an
object region. If regions of moving objects are less than half of an object region,
the color transformation parameters can be estimated by iterating a random
sampling.
The color transformation parameters in every region which is extracted bya random sampling are estimated with an evaluation function based on a histogram’s similarity. If there are no moving objects in the region, the same colortransformation parameters should be estimated in each region. Then, color transformation parameters which are estimated in each region are applied to otherregions. If there are no moving objects in the applied regions, the similarity valueof histograms between the color corrected image and the standard image becomes
higher. On the contrary, if there is the region which includes moving objects, thesimilarity value of histograms becomes lower. If an area where moving objects
do not exist is more than half of the region, similarity value of histograms is calculated without the eﬀect of moving objects by extracting a median of similarity
values. Finally, color transformation parameters pa , pb when a similarity value isthe highest in all regions are applied for the input image which is pre-processedin section 2.2. As a result, consistency of color tone is possible by removing themoving object. However, if an occupied rate of moving objects is more than halfof the region, iteration process which is explained below is needed.

Input image
Candidate regionsof moving object
Split

End

Split

End

End

End

End

Fig. 2. Split of object region

Estimation of candidate region of moving object Regions of moving objects are estimated by calculating a diﬀerence of intensity between the corrected
image and the standard image. Since the omnidirectional images are captured
with a motion omnidirectional camera, a few disparities are existed between input images. Therefore, when the diﬀerence of an intensity value is calculated,a template matching approach is performed for every region and a candidate
region of moving objects is estimated in consideration of the disparities between
omnidirectional images. Here, an image which is masked to the region of moving
objects is generated and is used for the region split in the following paragraph.
Split of object region In each region for processing, when an occupied rateof moving objects is more than a ﬁxed rate, the object region is split and colortransformation parameters are estimated. This is based on an idea that areas
with diﬀerent transformation parameters are existing in one region for processing. Appearance of re-splitting the input image is shown in Fig.2. The colortransformation parameters of a major object in the region can be estimatedwith this approach.
By iterating these processes, color consistency of images except in a region
where the moving object is observed is achieved. Estimation of color transformation parameters which are robust to the inﬂuence of disparity is possible bymaintaining a certain size of the split region.
2.4

Post-processing: compensation of moving objects

Even if the calculated transformation parameters are applied for the input image, the moving objects which exist in the image can not be removed only byiterating the color consistency processing. In our work, the candidate object regions are compensated using other corrected omnidirectional images which arecaptured at near positions. Here, there is an assumption that a background of
the moving object exists in the corrected input images. When the regions ofmoving objects are compensated, to consider the disparities between input images, the corresponded region with the area of moving objects are searched fromthe input images.

3

Experiments

3.1

Experimental environment

In the experiment, we used omnidirectional images which are captured with a
car-mounted omnidirectional camera as input images and removed inconsistencies among them. We used an omnidirectional multi-camera system (Ladybug2,
Point Grey Research) for capturing in an outdoor environment. 5 omnidirectional images which were captured at near positions were used for input images.
Since each image is captured at diﬀerent time, color tones diﬀer respectively. In
this research, we assume that camera positions and postures can be acquiredwith some sensors [1] or by a vision-based approach [9]. Fig.3 shows examples
of input images. There are some moving objects in each image and it turns outthat color tones are diﬀerent due to a change of illumination condition. Each
image has disparities due to a diﬀerence of captured positions.
3.2

Experimental results

The standard image which has no moving object and are suitable for an inputof an IBR approach are selected as shown in Fig.3(a). The regions which are
not necessary for color consistency, like sky region and equipment of capturingsystem were removed in advance from the input images.
Fig.4 shows the intermediate results of color consistency using LMeds method
to image A. The result of color consistency without splitting regions is shownin Fig.4(a). Here, since a set of transformation parameters were estimated towhole image and were applied to the input image, it turns out that some regions
has a diﬀerent color tone between the input image and the standard image. The
diﬀerence of intensity value between the color corrected image and the standard
image was computed as shown in Fig.4(b), In this ﬁgure, regions which havelarge diﬀerence of intensity are masked red. This result shows that it is diﬃcult
to correct color tone only with a set of transformation parameters. Input image
should be split into small regions and color transformation parameters should be
estimated in each region. Next, the region which is estimated to be a moving object is split and the color transformation parameters based on robust estimation
are estimated recursively. The result of color consistency with iteration processes
is shown in Fig.4(c). The comparison of the results as shown in Fig.4(d) showsthat iteration processes are eﬀective for removing inconsistencies among omnidirectional images. Fig.5 shows the results of compensation of moving objects
using the corrected images. In each image, it turn out that moving objects are
removed by the compensation.

(a) standard color image(b) image A(c) image B(d) image C

Fig. 3. Examples of input images which are captured at diﬀerent times in nearby
positions.

(a) result of color consistency without splitting regions(b) estimation of candidate of moving object (Red areas are estimatedregions of moving object.)(c) result of color consistency with iteration processes(d) estimation of candidate of moving
object after iteration processes (Redareas are estimated regions of moving
object.)Fig. 4. Result of color consistency with iteration processes to image A in Fig.3

To verify the validity of the proposed method, we conducted a quantitativeevaluation. Similarity of histogram between the standard image and the color
corrected image was used for evaluation. Bhattacharyya coeﬃcient was used asa similarity of histograms. Table.1 shows the similarity value of histogram withthe standard image. It turned out that the similarity of the histogram with thestandard image was improved for every result.
Finally, as an application, novel view images were generated with the IBR
approach[1] using the generated omnidirectional image database. Fig.6 shows
the novel view images which are generated by using omnidirectional images.
The result as shown in Fig.6(a) has inconsistencies which occur by a changeof illumination condition. Fig.6(b) shows good result by using the proposed
omnidirectional image database.

4

Conclusion

In this paper, we have proposed the method for removing inconsistencies which
occur by an existence of moving objects or a change of illumination condition
when an omnidirectional image database is generated. Our approach has realized consistency of images with the iteration processes which are the estimationof candidate region of moving objects and the achievement of color consistency
to split regions. Consistency of color tone is realized by estimating linear colortransformation parameters which change histogram of the input image to thatof the standard image. In experiments, we have conﬁrmed that the proposed
method can remove inconsistencies among omnidirectional images for an image
database. As a future work, we have to make a large-scale omnidirectional image
database.
Acknowledgement
This research was partially supported by the Ministry of Education, Culture,
Sports, Science and Technology, Grant-in-Aid for Scientiﬁc Research (A), 19200016.

Table 1. Similarity value of histogram with color standard image (1 is the highest
similarity value.)

Image A
Image B
Image CImage D
Image E

input image
proposed method
input image
proposed method
input image
proposed method
input image
proposed method
input image
proposed method

R

G

B

0.84290.99450.95850.99430.97170.99550.9036
0.99440.92030.99130.8996
0.99250.97030.99210.97400.99350.92650.99090.9346
0.98170.8877
0.9916
0.96690.99180.9697
0.99340.9251
0.99200.9306
0.9856

(a) standard color image(b) image A(c) image B(d) image C
Fig. 5. Result for removing moving object and inconsistencies of color tone to input
images in Fig.3

(a) Generation result fromoriginal omnidirectional images.

(b) Generation result fromproposed omnidirectional
image database.

Fig. 6. Novel view images which are generated from omnidirectional images with the
IBR approach.

References
1. Hori, M., Kanbara, M., Yokoya, N.: Novel stereoscopic view generation by imagebased rendering coordinated with depth information. Scandinavian Conference on
Image Analysis (2007) 193–202
2. Sato, R., Ono, S., Kawasaki, H., Ikeuchi, K.: Real-time image-based renderingsystem for virtual city based on image compression technique and eigen texture
method. International Conference on Pattern Recognition (2008)
3. Finlayson, G., Hordley, S., Drew, M.: Removing shadows from images. Proc. ofEuropean Conference on Computer Vision IV (2002) 823–836
4. Moroney, N.: Local color correction using non-linear masking. Color Science andEngineering Systems, Technologies, Applications (2000) 108–111
5. Tsuchida, M., Kawanishi, T., Murase, H., Takagi, S.: Sequential monte-carlo estimation of background image for background subtraction under changing illumination.
Int. Conf. on Visualization, Imaging, and Image Processing (2003) 421–425
6. Rother, C., Kolmogorov, V., Blake, A.: ”GrabCut” - interactive foreground extraction using iterated graph cuts. ACM Transactions on Graphics 23 (2004) 309–314
7. Zafarifar, B., With, P.: Blue sky detection for picture quality enhancement. Advanced Concepts for Intelligent Vision Systems (2006) 522–532
8. Massart, D., Kaufman, L., Rousseeuw, P., Leroy, A.: Least median of squares: arobust method for outlier and model error detection in regression and calibration.
Analytica Chimica Acta 187 (1986) 171–179
9. Sato, T., Ikeda, S., Yokoya, N.: Extrinsic camera parameter recovery from multiple
image sequences captured by an omni-directional multi-camera system. Proc. ofEuropean Conference on Computer Vision 2 (2004) 326–340