Video-based raindrop detection for improved image registration
Martin Roser
Department of Measurement and Control
Karlsruhe Institute of Technology

Andreas Geiger
Department of Measurement and Control
Karlsruhe Institute of Technology

martin.roser@kit.edu

geiger@kit.edu

Abstract
In this paper we present a novel approach to improved
image registration in rainy weather situations. To this end,
we perform monocular raindrop detection in single images
based on a photometric raindrop model. Our method is capable of detecting raindrops precisely, even in front of complex backgrounds. The effectiveness is demonstrated by asignificant increase in image registration accuracy which
also allows for successful image restoration. Experiments
on video sequences taken from within a moving vehicle
prove the applicability to real-world scenarios.

1. Introduction
Many state-of-the-art computer vision algorithms are designed to work in well-posed visibility conditions. Though
this assumption may hold for most indoor and simulation
scenarios, environmental noise can badly influence theirperformance in outdoor settings, e.g. in surveillance applications or in driver assistance systems where a camera ismounted behind the windshield of a moving vehicle.
However, proper operation in the presence of rain is asecurity-relevant prerequisite to many applications, particularly on board mobile vehicles. For example image registration accuracy declines in the presence of raindrops on
the windshield due to mismatched features. In this paper
we propose a novel approach to video-based raindrop detection with an application to monocular image registration improvement. Detecting raindrops using a photometricraindrop model also allows for restoring occluded regions
by fusing intensity information from nearby image frames
as illustrated in Fig. 1.
Traditional image mosaicing approaches [2] fail in registering images from in-vehicle cameras correctly, since theyeither assume rotations around the yaw axis merely or insignificant scene depth. Our image sequence, however, isrecorded at very small frame rates (10 Hz) from within avehicle moving at 15 − 25 ms which changes scene depth
significantly. Furthermore, feature localization noise is high(a) Raindrop on windshield(b) 3D raindrop refraction model(c) Raindrop detections(d) Restored image

Figure 1. Raindrop model and image restoration. (a) shows atypical raindrop’s appearance on the windshield. Modeling therefraction of light rays (b) allows for precisely detecting raindrops(c). After registering consecutive frames, occluded areas can berestored using the intensity from neighboring image frames (d).

due to the tilted orientation of the camera with respect to theroad and the presence of raindrops on the windshield. This
makes direct estimation of the fundamental matrix between
consecutive frames difficult [10].
On the other hand, visual SLAM methods [3] are hard to
employ due to road homogeneities, pattern recurrences (e.g.
lane markings or guardrails) and a wide baseline between
frames. Since we are using only monocular sequences,stereo information [15] is not available to us.
For registration we therefore extended the approach proposed in [8], which uses prior knowledge of the vehicle dynamics and camera setup in combination with homography
constraints. However, this only works reliably, if enoughfeature correspondences on the road plane can be matchedbetween consecutive frames. Due to mismatches and occluded image regions this is not the case in the presence ofraindrops in front of the camera. In this paper we show that
automatic raindrop detection helps alleviating this problem.
However, raindrop detection is a challenging problem

due to several reasons: Raindrops on windshields exhibit
a large variety in shape and size. Their short distance to thecamera results in out-of-focus blur, hereby decreasing distinctiveness of raindrop features. Moreover, transparency
and light refraction makes the raindrops appearance highly
dependent on the image background. Recently, several approaches have been proposed to overcome these problems:
Garg and Nayar [5, 6, 7] proposed an accurate photometric model for stationary spherical raindrops and determined the effect of camera parameters on image disturbance. However, a static observer or high camera exposure
times are assumed making the approach not applicable inscenarios where the camera is moving.
Kurihata et al. [12] used a machine learning approachwith raindrop templates, so called eigendrops, to detectraindrops on windshields. Results within the sky area werepromising, whereas the proposed method produced a large
number of false positives within the non-sky regions of theimage where raindrop appearance modeling becomes morechallenging.
Halimeh and Roser [9] developed the basics of a photometric raindrop model to predict the appearance of a raindrop on the windshield. However, they only give experimental validation in a well-defined laboratory setting. They
presently disregard the fact that raindrops appear blurredsince they are outside the scene focus of the camera.
Yamashita et al. [16, 19, 18, 17] proposed several approaches for detecting and removing waterdrops from images. They detect noisy regions using difference images[19] and replace contaminated image parts with patterns
from a second camera, assuming a distant scene. Yamashita
et al. further developed a spatio-temporal approach for detecting adherent noise by image sequence analysis [18, 17],
supposing a camera with constant and known yaw rate.
In this paper, we present an integrated concept for raindrop detection (Section 2) that contains the generation of artificial raindrop patterns at regions of interest (ROIs), ROI
initialization, and raindrop verification by intensity-based
correlation. Then we omit disturbing image regions to improve registration accuracy. Thereto we use prior knowledge of the vehicle dynamics and camera setup in combination with homography constraints (Section 3). Finally,
we make use of the image registration results and the accurately detected raindrop positions for image enhancement
by restoring occluded image areas with intensity information from neighboring image frames. In section 4 we compare the different raindrop detection algorithms in terms ofPrecision-Recall curves and demonstrate improved image
registration performance. Finally, we conclude this paperwith an outlook on future work (Section 5).

Potential raindrops
Comparison
Render artificial raindrop

Figure 2. Flowchart of raindrop detection with RIGSEC. Artificial raindrop patterns are compared to potential raindrops using
intensity-based correlation.

C

Environment
point

B

nwaterAD

nairnglass

E
nair

P

ROI

^

P

CameraImage plane

Figure 3. Refraction model. Light ray tracing allows for accurate reconstruction of raindrops on transparent surfaces from background scene information.

2. Raindrop Detection
Fig. 2 shows an overview over the proposed raindrop
detection approach. First, we extract ROIs from the image via interest point detection. In order to check if a region contains a raindrop, we use a photometric raindrop
model which renders artificial raindrop patterns for each
ROI. These patterns show the raindrop appearance at thislocation and scale in the image (See Fig. 6). In a verification step, we compare the extracted region with the artificial
raindrop pattern.

2.1. Photometric raindrop model
In this section, we give a brief overview of the raindrop model we used, called Raindrop Intelligent Geometric Scanner and Environment Constructor (RIGSEC) andpresent all necessary extensions. Our raindrop model is inspired by the model of [9], but we enhance the model by
realistic out-of-focus blur and validate the model in realworld scenarios. Furthermore we compare several strategies
for finding suitable regions of interest.
Given a potential raindrop, our algorithm models the geometric shape of that droplet on the car windshield. It utilizes its photometric properties and renders an artificial raindrop pattern from points in the environment.
Drop modeling can be performed with varying
parametrizations of raising complexity like spherical, ellipsoidal or even more complex models. We use a spheresection, since for most raindrops, it models the shape sufficiently well (see Fig. 1(a)).
Assuming a potential raindrop with radius r at position
P = (x, y) on the image plane, we determine the part ofthe scene that will be observed through this raindrop. As
depicted in Fig. 3, a light ray emanating from a point in
the environment will be refracted by the raindrop and thewindshield and finally reaches point P on the image plane.

The exactly same point can also be seen directly at point P̂ .
By tracing the light rays passing through the raindrop, weare able to virtually render the appearance of any raindrop
at any location. Snell’s law describes the transition of lightrays from medium 1 to medium 2. The angle of refractionµ2 is a function of the inclination angle µ1 and the refractive
indices n1 and n2 . Hence, for all transitions A − E, theangle of refraction µ2 can be calculated as
n1 sin µ1.
(1)µ2 = arcsin
n2Thus, an accurate geometric relationship between P and P̂
can be derived. Fig. 6(a) shows the sampling of points inthe world for rendering an artificial raindrop as determinedby our refraction model.
However, in order to compare the intensity values of the
mapped environment to the potential raindrop, a predictionof the raindrop photometry is equally essential as the geometric ray tracing discussed above. According to [11], Fresnel’s reflectivity coefficients for partly polarized sunlight in
the atmosphere going from medium 1 to medium 2 can be
expressed as

1 2
2
r12k + r12
(2)
R12 =
⊥
2
r12⊥ =

n1 cos µ1 − n2 cos µ2
n1 cos µ1 + n2 cos µ2(3)

r12k =

n1 cos µ2 − n2 cos µ1
n1 cos µ2 + n2 cos µ1(4)where µ1 , µ2 , n1 , and n2 are the angles and refractive indices of media 1 and 2 respectively. Since the pixel intensity
IP̂ is known from the camera image, the estimated raindrop
intensity at point P can be determined asIP̂IP = Q.
(1− Ri )ji(5)Here Ri denotes Fresnel’s reflectivity coefficients at points
A − E and j = ±1 stands for the direction of the intensity prediction. The geometric relations and the photometric properties allows an artificial raindrop pattern generation
as depicted in Fig. 6(c).
The sharp optical imaging of an object onto a camera
sensor by means of a thin lens can be described by
1 1
1
= +
f
b g

(6)

where f is the focal length of the camera, g is the object distance and b is the distance between lens and image
plane. Since video sensors for environment perception are

A

B
image
plane

Figure 4. Optical path for out-of-focus imaging. Objects that areout-of-focus are imaged blurred. If the camera is focusing point A,
point B is imaged onto a disc with diameter ǫ in the image plane.

d1
d2

Camera(a) Distance of raindrops tocamera(b) Out-of-focus blur map

Figure 5. Computing out-of-focus blur. The distance of raindrops
to the camera is dependent from their position on the windshield(a). This leads to an out-of-focus blur map where each image pixel
is mapped to its corresponding unsharp diameter ǫ (b).

adjusted for sharp optical imaging of distant objects, close
scene points like raindrops on the windshield violate this assumption. They are not imaged sharply, but rather projected
onto a disc with diameter ǫ in the image plane (see Fig. 4).
We model this effect by applying an out-of-focus blur using a disc kernel. According to Fig. 4, the lens equation forout-of-focus objects is given by
1
1
1
=
+
f
b + ∆b g − ∆g(7)where ∆g is known as depth of field. Using the geometric
relationǫ∆b=Db + ∆b(8)

with lens diameter D, equation 6 and 7 we get an expression
for the disc kernel diameter ǫ
ǫ=

∆gf 2
O(g − ∆g)(g − f )(9)fis the camera aperture size. Since the diswhere O = D
tance and relative orientation of the windshield with respect
to the camera is known, ǫ can easily be computed for eachpoint on the windshield as shown in Fig. 5. The blurred
raindrop pattern is illustrated in Fig. 6(d). Comparison tothe original raindrop (Fig. 6(b)) shows good visual agreement.
Rendering raindrops at multiple scales and all image locations is computationally expensive. On the other hand,(b) original raindrop(c) reconstruction(a) Geometric ray tracing(d) blurred patternFigure 6. Artificial raindrop generation. (a) all pixels of a raindrop at a specified location and scale (red circle) are projected tothe environment by our raindrop model (green dots). In (b) theoriginal raindrop is shown, while (c) depicts our reconstruction
using all points from (a) and photometric constraints. The resultof applying out-of-focus blur to our reconstruction is shown in (d).

present a novel method for detecting blurred image regions
by adaptive bandpass filtering. The intuition behind this isthat raindrops can be located at specific spatial frequencies
which are upper bounded by their out-of-focus blur.
We create a bandpass image B for each image I by computing the Difference of Gaussians (DoG) for each pixel location according toB(x, y) = (I ⊗ gσ−∆σ )(x, y) − (I ⊗ gσ+∆σ )(x, y)
where ⊗ denotes the convolution operator and g is a 2DGaussian. Here σ is taken from the out-of-focus blur map in
Fig. 5(b) and ∆σ has experimentally be determined as 5px.
ROIs are extracted by thresholding the bandpass image andfinding segments using a connected component algorithmwith a 4-neighborhood. Finally, an eigenvalue decomposition of the covariance matrix of all pixel positions belonging
to a segment is performed. Convexity ratio (segment area toarea of convex hull), dominant orientation (first eigenvector), and aspect ratio (first eigenvalue to second one) are extracted to distinguish raindrops from background features.
In our experiments typical raindrops exhibit a convexity ratio of > 0.9, possess an orientation difference of < 15◦ tothe horizontal and have an aspect ratio between 1 and 3.

2.3. Raindrop verification

Figure 7. Fast RIGSEC algorithm with 4 × 3 artifical raindrops.

raindrop appearance is highly dependent from its backgound scenario, which lets simple filter masks or eigendrops as discussed in [12] only perform insufficiently. For
this reason, we developed a fast RIGSEC implementationthat synthesizes a limited number of equally spaced raindrop temlates and correlates them in a surrounding region.
Fig. 7 shows artificial raindrop patterns and the correlation
areas for a 4 × 3 grid.

2.2. Initialization
In order to further narrow the search space we first find
ROIs using two different interest point detectors describedin this section. In this paper we choose the SURF featuredetector [1] as a baseline, which approximates the second
order image derivatives with box filters. By looking at thescale of the SURF filter response we also get an estimate forthe raindrop size in addition to its location.
Gradient based feature detectors like SURF are not performing well for detecting raindrops, because blurred regions only respond weakly. On the other hand, decreasing
the detection threshold leads to many false detections thatmust be rejected by the raindrop model. For this reason, we

Verification of a raindrop candidate is performed bycomparing the observed image region with the artificial created raindrop pattern in a small surrounding. We maximize
the correlation coefficient of their intensity values CCintensity
as well as the correlation coefficient of their first derivative(gradient map) CCgradient . In order to take different raindrop
sizes into account, we slightly vary the scale of the artificial
raindrop by a factor of 1 − 1.5.

3. Registration and restoration
In this section we show how raindrop detections can beused to improve image registration accuracy. Furthermore,
we fuse intensity information from multiple views into one
single frame to restore image areas occluded by raindrops.

3.1. Geometric scene description
Approximating the road surface by planes enables us todescribe perspective mappings via homographies [10].
Fig. 8 illustrates the geometric scene description. We
keep the road-to-camera transformation matrix TRC andthe calibration matrix K fixed in all frames and estimate theparameters of the road-to-road mappings TRR (Θi ). The
6D-vector Θi = (rx (i), ry (i), rz (i), tx (i), ty (i), tz (i)) represents the rotational and translational parameters between
frame i and frame i + 1. The extrinsic and intrinsic parameters are determining the transformation from the road

ity P (Zi |Θi ) is assumed to be normally distributed
1
P (Zi |Θ) ∝ exp(− dTi Σ−1 di )2

Figure 8. Geometric scene description. Transformations used for
projecting between different coordinate systems.

with covariance matrix Σ = diag(σd2 , ..., σd2 )T and di containing the euclidean reprojection errors between the features connecting frame i and frame i + 1.
Further assuming independence in the translational androtational parameters (rx , ry , rz , tx , ty , tz ) over the whole
sequence gives
P (Θ) = P (rx )P (ry )P (rz )P (tx )P (ty )P (tz ).

to the camera coordinate system TRC and the calibration
matrix K. They are estimated using standard calibration
techniques [20] and kept constant over time.
However, only small parts of the scene (e.g. the road)can be represented well by a single plane. We tackle thisproblem by approximating the world as a box using fivehomographies, namely the ground plane, the left wall, theright wall, the sky and the background plane. Please notethat depth is not represented accurately due to our simplified
assumption. However, we model the distance to the left
and right wall such that objects of interest (e.g. closeby
trees and guardrails) are represented sufficiently well. We
achieve satisfactory results in image restoration since only
nearby image frames are considered.

3.2. Registration
We initially warp the road plane from frame i+1 to framei using a prior estimate for Θi (by means of a first orderMarkov model) and bilinear interpolation. This projection
ensures feature similarity in appearance between frames.
We then compute Harris corners in frame i + 1 that do
not intersect with detected raindrops and search for correspondences in frame i using normalized cross-correlation.
Because of low saliency in our test sequences we compute center-surround feature responses for the maxima incorrelation space which are weighted by the template variance. Thereof we only keep the 50% best matches. Further
outliers are removed by applying the Direct Linear Transform (DLT) algorithm [10] in combination with RANSAC[4] which is especially important in situations where otherdynamic objects (e.g. vehicles) are present in the scene.
N −1
Refining the final parameter set Θ = {Θi }i=1
is doneby searching for the MAP solution of
P (Θ|Z1 , ..., ZN ) ∝ P (Z1 , ..., ZN |Θ)P (Θ)(10)

with respect to Θ, where N denotes the total numberof frames. We assume independence of non-consecutive
frames and therefore the likelihood P (Z1 , ..., ZN |Θ) facQN
torizes to i=1 P (Zi |Θi ) where the observation probabil-(11)

Here the vector rx ∈ RN contains the pitch angle rate for all
frames of the sequence, for example. Ideally one could use
a uniform prior for the parameters. However, since in our
setting the camera is highly tilted with respect to the registration surface, small registration errors have a large impact on the parameters. Thus we encourage smoothness by
putting Gaussian process priors [14] on the function spaceof the parameters
f (i) ∼ GP(µf (i), σf (i, i′ ))(12)with f (i) ∈ {rx (i), ry (i), rz (i), tx (i), ty (i), tz (i)} where idenotes the frame number and rx (i) is the i’th entry of rx .
For all f we fix the mean function to an initial estimate of
the parameter µf (i) ≡ f0 and model the covariance function σf (i, i′ ) using the squared exponential kernel.
Integrating the prior (11) and the likelihood into(10) and taking the logarithm yields the log-posterior
log P (Θ|Z1 , ..., ZN ) which we maximize using standard
gradient descent techniques [13].

3.3. Multi-band blending
After estimating the parameters Θ, reconstruction of image areas covered by raindrops is possible using intensity
information from nearby frames. Since simple merging ofimages leads to unsatisfactory results due to differences in
gain, vignetting, object shadows and registration errors we
use multiresolution splines [2] which are outlined in the following section.
For each frame one base image and one base mask
are generated. The base image contains all visible pixels
warped into the target coordinate system using bilinear interpolation and the homography model from section 3.1,
whereas the base mask contains the warped binary raindrop
detection images. Combining base images is done by generating difference masks and applying multi-band blending[2]. We compensate the gain in each contiguous area independently. This is done by comparing pixels around thearea of interest and adjusting the mean intensity value.
Bandpass images Bki are generated by differencing thepreviously smoothed base images. Here the index i denotesthat the image and its mask has been projected from frame
i into the current frame. Please note that only neighboring frames (we typically used a neighborhood size of 6) areconsidered. For k ≥ 1 we have
Wik

(k−1)

= Wi

Iki

=

Bki

=

⊗ gσ̃(k)(k−1)
Ii⊗ gσ̃(k)(k−1)
Ii− Iki .

where the standard deviation of the Gaussian kernel gσ̃(k) is√
set to σ̃(k) = 2k + 1σ. Combining overlapping regionsPkki Wi (x, y)Ii (x, y)PIΣk (x, y) =ki Wi (x, y)Pkkki Wi (x, y)Bi (x, y)PBΣ
(x, y) =ki Wi (x, y)
finally yields the mosaic

Imosaic = IΣKσ +

K
X

k
BΣ

k=1

with K the total number of bands.

4. Results
In this section we evaluate our approach using two
types of experiments: Raindrop detection rates in termsof Precision-Recall and image registration errors with andwithout raindrop detection. The results are generated by applying our algorithm to an outdoor sequence of 302 frames
recorded in a rural environment. We used monochrome images with a resolution of 1024 × 768. The camera system
was mounted at a distance of ≈ 14cm from the windshieldwith a focal length of 6mm.

4.1. Raindrop detection experiments
First, we perform experiments on raindrop detection forthe different methods described in section 2. To this end we
asked several persons for labeling elliptical regions whichappear to contain raindrops within our 302 test frames. Detection results for the different methods are illustrated infigure 9. We present results in terms of Precision-Recall
for varying detection thresholds. Precision and recall aredefined asTPprecision =TP + FPTP
recall =TP + FN(13)

where TP = true positives is the number of correctly discovered raindrops, FP = false positives is the number ofdetections which are no raindrops and FN = false negatives
is the number of raindrops which are not detected. A detection is considered to be correct if its center is within the true

raindrop. Other closeby detections are removed to avoid
multiple true positives for one drop. The ground truth andtwo exemplary detection results for the image sequence arepresented in Fig. 10(a). We further position each frame in
Precision-Recall space (Fig. 10(b)) and take the mean value
for each configuration to generate the Precision-Recall plots
shown in Fig. 10(c).
Fig. 10(c) compares the performance of the proposed algorithms SURF, BLUR, fastRIGSEC and a combination of
BLUR and RIGSEC for varying parameters. SURF is tested
with varying filter response thresholds. A recall close to
1 is achieved when keeping the threshold low. However,up to 2000 detected interest points lead to very low precision. Higher thresholds increase precision slowly but at
the expense of decreasing recall. In contrast, BLUR showsa non-monotonic behaviour for varying parameters. The
curve reaches its best value at a threshold of 0.05. Lower
thresholds lead to an insufficient segmentation of raindrops
in the binary image and hence show low precision and recall. Higher thresholds omit raindrops that are less distinctive. Recall rates up to 0.67 are achievable with thismodel. The fastRIGSEC algorithm uses a 4 × 3 grid and wevary the threshold for the intensity correlation coefficient
CCintensity between 0.1 and 0.95. The best compromise ofrecall and precision with regards to subsequent image registration was found at values of 0.85 − 0.9. As can be seen,
fastRIGSEC dominates SURF and BLUR in performance,
verifying the validity of the raindrop model. As depicted
in Fig. 10(c), we further combined BLUR with RIGSEC
and varied the threshold for CCintensity , and CCgradient respectively. The presented results reveal that BLUR+RIGSEC
clearly outperforms all other proposed methods. However,in terms of recall, fastRIGSEC can exceed BLUR+RIGSEC(at the cost of low precision). This is caused by the fact that
RIGSEC only validates interest points. Hence, the recallof BLUR+RIGSEC is bounded by the recall of the BLUR
model.
Please also note that neither temporal nor stereo information are used in our system. All tested algorithms canbe applied to single monocular grayscale images in contrast
to methods using multiple cameras [19] or image sequence
analysis [18, 17] for raindrop detection. We believe that further improvements can be made by temporally integrating
the respective methods.

4.2. Image registration experiments
In this experiment we compare different raindrop detection methods and their impact on image registration. For
this purpose we remove all feature detections which are hidden by drops detected with one of the methods describedin section 2 and perform image registration as described in
section 3.2. We further gather ground truth information visually by maximizing equation 10 using hand selected fea-

(a) SURF(b) BLUR(c) fastRIGSEC(d) BLUR+RIGSEC

Figure 9. Qualitative raindrop detection results. This figure shows our raindrop detection results for the different methods (columns)
and two test images (rows). True positives are marked in green, false positives in red.
1.0

Raindrop ground truth
SURF
BLUR

20

SURF
BLUR

0.8

10

Precision

TP

30

0

FP

600
400

0.6
0.4
0.2

200
0

0.0
0

100

200

300

0.0

0.2

(a) Timeline
No raindrop detection
Raindrop ground truth
BLUR
fastRIGSEC
BLUR+RIGSEC

SURF
BLUR
fastRIGSEC
BLUR+RIGSEC, CCintensity
BLUR+RIGSEC, CCgradient

0.6

0.8

1.0

No raindrop detection
Raindrop ground truth
BLUR
fastRIGSEC
BLUR+RIGSEC

Translational error (m)

0.8

0.6

0.4

0.2

Rotational error (rad)

1.0

1.0

Precision

0.4(b) Recall

0.8

0.6

0.4

0.2

0.0

0.0
0.0

0.2

0.4

0.6

(c) Recall

0.8

1.0

0.03

0.02

0.01

0.00
0

4

8

(d) Road area
occluded by raindrops (%)

12

0

4

8

12

(e) Road area
occluded by raindrops (%)

Figure 10. Quantitative raindrop detection and image registration results. (a) shows the raindrop ground truth with TP and FP rate of
two detectors over time. (b) shows the results of (a) transformed to Precision-Recall space to permit a better comparison. (c) illustrates
the performance of raindrop detection algorithms. A combination of BLUR+RIGSEC leads to a precision up to 0.8 at good recall ratesof ∼ 0.67. (d) and (e) show the translational and rotational registration error over the percentage of occluded road area for the proposed
methods. Image registration accuracy can be significantly improved by considering the raindrop detection results.

tures and prior knowledge about the camera setup and thevehicle speed. Results are presented in terms of the meantranslational and rotational error between frames.
Fig. 10(d+e) show error histograms over the percentageof occluded road area for the respective methods. Both error measures increase with the number of raindrops whenno raindrop detection is performed. This is due to the factthat raindrops are matched with high reliability in consecutive frames. Please note that location and appearance ofraindrops are not reflected by this figure, leading to nonmonotonic curve characteristics. However, using our raindrop detection approach lowers the mean registration er-

ror for all test scenarios. We observed that combining the
BLUR detector with RIGSEC performed best, which is confirmed by the results in section 4.1. The ground truth acts
as an approximate lower bound on the attainable registration
error. It is nearly reached using the combined method whichshows the effectiveness of our raindrop detection model onreal data. We used the successful image registration provided by this method to reconstruct occluded image areas.
An example is shown in Fig. 1(d). Please note that some artifacts have been left mainly in the image center. This stems
from the fact that regions close to the vanishing point of theimage are unobservable since no intensity information fromneighboring frames can be used for reconstruction.

5. Conclusion and Future WorkIn this paper we have presented a novel approach that
improves image registration accuracy in rainy weather conditions by considering raindrop detections. We compareddifferent detectors and combined the complementary BLUR
detector with the presented raindrop model. This combination revealed superior performance compared to the SURF
baseline. We further showed that restoring occluded areas ispossible by considering information from neighboring image frames and leads to improved visibility.
Since the sphere section assumption of our model does
not cover all raindrop shapes, we plan to extend the raindrop
model flexibility. Making use of temporal dependencies inraindrop location and appearance will further improve raindrop detection rates and hence registration results. We believe that this could pave the path for applying many visionalgorithms even in rainy weather conditions.

6. Acknowledgment
This work is partly supported by Honda R&D Europe. The authors would also like to thank the Karlsruhe
School of Optics and Photonics (KSOP) and the Collaborative Research Center (SFB/Tr28) ”Cognitive Automobiles”
granted by the Deutsche Forschungsgemeinschaft.

References[1] H. Bay, T. Tuytelaars, V. Gool, and L. Surf: Speeded up
robust features. In 9th European Conference on Computer
Vision, Graz Austria, May 2006.
[2] P. J. Burt and E. H. Adelson. A multiresolution spline withapplication to image mosaics. ACM Transactions on Graphics, 2:217–236, 1983.
[3] A. J. Davison, I. D. Reid, N. D. Molton, and O. Stasse.
Monoslam: Real-time single camera slam. Pattern Analysis and Machine Intelligence, IEEE Transactions on,
29(6):1052–1067, 2007.
[4] M. A. Fischler and R. C. Bolles. Random sample consensus: a paradigm for model fitting with applications to image

analysis and automated cartography. Readings in computer
vision: issues, problems, principles, and paradigms, pages
726–740, 1987.
[5] K. Garg and S. Nayar. Detection and removal of rain fromvideos. In IEEE Conference on Computer Vision and Pattern
Recognition (CVPR), volume I, pages 528–535, Jun 2004.
[6] K. Garg and S. Nayar. When does a camera see rain? In
IEEE International Conference on Computer Vision (ICCV),
volume 2, pages 1067–1074, Oct 2005.
[7] K. Garg and S. Nayar. Vision and rain. Internatl. Journal ofComputer Vision, 75(1):3–27, 2007.
[8] A. Geiger. Monocular road masaicing for urban environments. In Intelligent Vehicle Symposium, 2009 IEEE, Xi’an,
China, 2009.
[9] J. Halimeh and M. Roser. Raindrop detection on car windshields using geometric-photometric environment construction and intensity-based correlation. In Intelligent Vehicle
Symposium, 2009 IEEE, Xi’an, China, 2009.
[10] R. I. Hartley and A. Zisserman. Multiple View Geometryin Computer Vision. Cambridge University Press, second
edition, 2004.
[11] C. Klingshirn. Semiconductor Optics. Springer, 3. ed. edition, 2007.
[12] H. Kurihata, T. Takahashi, I. Ide, Y. Mekade, H. M. Y.
Tamatsu, and T. Miyahara. Rainy weather recognition fromin-vehicle camera images for driver assistance. In IEEE Intelligent Vehicles Symposium, pages 205–210, 2005.
[13] M. F. Moeller. A scaled conjugate gradient algorithm for fast
supervised learning. Neural Networks, 6:525–533, 1993.
[14] C. E. Rasmussen and C. K. I. Williams. Gaussian Processes
for Machine Learning (Adaptive Computation and Machine
Learning). The MIT Press, 2005.
[15] D. Scharstein and R. Szeliski. A taxonomy and evaluation ofdense two-frame stereo correspondence algorithms. International Journal of Computer Vision, 47:7–42, 2002.
[16] Y. Tanaka, A. Yamashita, T. Kaneko, and K. T. Miura. Removal of adherent waterdrops from images acquired witha stereo camera system. IEICE - Trans. Inf. Syst., E89D(7):2021–2027, 2006.
[17] A. Yamashita, I. Fukuchi, T. Kaneko, and K. Miura. Removalof adherent noises from image sequences by spatio-temporal
image processing. Robotics and Automation, 2008. ICRA
2008. IEEE International Conference on, pages 2386–2391,
May 2008.
[18] A. Yamashita, T. Harada, T. Kaneko, and K. Miura. Removalof adherent noises from images of dynamic scenes by using apan-tilt camera. Intelligent Robots and Systems, 2004. (IROS
2004). Proceedings. 2004 IEEE/RSJ International Conference on, 1:437–442 vol.1, Sept.-2 Oct. 2004.
[19] A. Yamashita, M. Kuramoto, T. Kaneko, and K. Miura. A
virtual wiper - restoration of deteriorated images by using
multiple cameras. Intelligent Robots and Systems, 2003.
(IROS 2003). Proceedings. 2003 IEEE/RSJ International
Conference on, 4:3126–3131 vol.3, Oct. 2003.
[20] Z. Zhang. Flexible camera calibration by viewing a planefrom unknown orientations. IEEE International Conference
on Computer Vision (ICCV), 1:666–673, 1999.