IEEE ROBOTICS AND AUTOMATION LETTERS. PREPRINT VERSION. ACCEPTED JANUARY, 2020

1

Self-Supervised Linear Motion Deblurring
Peidong Liu1 , Joel Janai2 , Marc Pollefeys1,3 , Torsten Sattler4 and Andreas Geiger2

Abstract—Motion blurry images challenge many computer
vision algorithms, e.g., feature detection, motion estimation, orobject recognition. Deep convolutional neural networks are stateof-the-art for image deblurring. However, obtaining training datawith corresponding sharp and blurry image pairs can be difficult.
In this paper, we present a differentiable reblur model for selfsupervised motion deblurring, which enables the network to learn
from real-world blurry image sequences without relying on sharp
images for supervision. Our key insight is that motion cues
obtained from consecutive images yield sufficient information
to inform the deblurring task. We therefore formulate deblurring as an inverse rendering problem, taking into account thephysical image formation process: we first predict two deblurred
images from which we estimate the corresponding optical flow.
Using these predictions, we re-render the blurred images andminimize the difference with respect to the original blurry
inputs. We use both synthetic and real dataset for experimental
evaluations. Our experiments demonstrate that self-supervised
single image deblurring is really feasible and leads to visuallycompelling results. Both the code and datasets are available athttps://github.com/ethliup/SelfDeblur.
Index Terms—Computer Vision for Automation, Deep Learning in Robotics and Automation

I. I NTRODUCTION
OTION blur is one of the most common factors degrading image quality. It often arises when the image
content changes quickly (e.g., due to fast camera motion)or when the environment is illuminated poorly, hence necessitating longer exposure times. Combining both situations,e.g., a self-driving car driving at dusk, further aggravatesthe problem. As many computer vision algorithms such as
semantic segmentation, object detection, or visual odometry
rely on visual input, blurry images challenge the performanceof these algorithms. It is well known that many algorithms(e.g., depth prediction, feature detection, motion estimation,or object recognition) suffer from motion blur [17], [25],[26], [33]. The motion deblurring problem has thus received
considerable attention in the past [7], [17], [21], [28], [32].
Existing techniques to solve this problem can be classified
into two categories: the first type of approaches formulate theM

Manuscript received: September, 10, 2019; Revised December, 06, 2019;
Accepted January, 20, 2020.
This paper was recommended for publication by Editor Cesar Cadena
Lerma upon evaluation of the Associate Editor and Reviewers’ comments.
1 Peidong Liu and Marc Pollefeys are with Computer Vision and Geometry Group, Department of Computer Science, ETH Zürich, Switzerland
peidong.liu@inf.ethz.ch
2 Joel Janai and Andreas Geiger are with Autonomous Vision Group, Max
Planck Institute for Intelligent Systems and Univeristy of Tübingen, Tübingen,Germany
3 Marc Pollefeys is with Microsoft Mixed Reality and Artificial Intelligence
Lab, Zürich, Switzerland
4 Torsten Sattler is with Computer Vision and Medical Image Analysis
Group, Chalmers University of Technology, Sweden
Digital Object Identifier (DOI): see top of this page.

problem as an optimization problem [2], [4], [16], [18], [28],[37] where the latent sharp image and/or the blur kernel are
optimized using gradient descent. One of the advantages ofthis kind of methods is that they do not require any groundtruth sharp images. However, the resulting solvers usually
have a large computational complexity, which limits theirapplicability in time-constrained settings, such as real-timerobotic visual perception. Handcrafted priors on either theimage or the blur kernel further limit their performances.
The second type of approaches phrase the task as a learning problem. Building upon the recent advances of deep
convolutional neural networks, state-of-the-art results havebeen obtained for both single image deblurring [21], [32]and video deblurring [29], outperforming optimization-based
techniques in terms of both quality and efficiency. However,learning-based methods typically require full supervision in
the form of corresponding pairs of blurred and sharp images.
Unfortunately, obtaining such pairs is not always easy due
to two main reasons. One is that not every camera has thecapability to capture images at enough high frame rate (1000or more frames per second), such that we can use the recorded
frames to synthesize the training data. Another reason is that
it would also be difficult to obtain good quality images in real
scenarios where the motion blur really occurs (e.g., at night).
High frame rate limits the exposure time and would thus makethe captured image extremely dark or even invisible.
Inspired by recent progress in self-supervised depth [5],[41], flow [12], [20] and representation learning [3], [24], we
propose a novel approach for self-supervised image deblurring which only relies on real-world blurry image sequences
for training. Self-supervised learning improves the network’s
generalization performance, by enabling the network to adapt
to scenarios where ground truth sharp images are not available.
Our network contains a deblurring network and an optical flow
estimation network. However, instead of using ground truth
sharp images [9], [17], [21], [32], we pose the task as aninverse rendering problem and take advantage of the physicalimage formation process for supervision. More specifically,
given two consecutive blurry frames of a video sequence, wefirst predict the corresponding deblurred images using a deepneural network. A second deep neural network takes both
deblurred images as input and computes the corresponding
optical flow. Using this prediction, and assuming a locallylinear blur kernel, our model re-renders the blurred images
and compares the results to the original blurry inputs usinga photometric loss function. Moreover, we constrain the optical flow network using a photo-consistency loss function.
Our entire model can be trained end-to-end from pairs ofconsecutive blurry images captured with a consumer video
camera. At test time, our network takes a single blurred image

2

IEEE ROBOTICS AND AUTOMATION LETTERS. PREPRINT VERSION. ACCEPTED JANUARY, 2020

Fig. 1: Self-supervised motion deblurring. First: Sharp ground truth image. Second: Blurry input image. Third: Deblurring
results of our self-supervised method. Fourth: Deblurring results of the supervised method from Tao et al. [32].

and deblurs it in real time on a single GTX 1080Ti graphic
card using the learned parameters. As illustrated in Fig. 1,
our approach is competitive with respect to a state-of-the-art
supervised method [32] despite being fully self-supervised.
Our second contribution is a novel synthetic dataset anda real dataset. The synthetic dataset has 3606 blurry-sharp
image pairs recorded with a professional high-speed camera
mounted on a ground vehicle. The real dataset has 2302 blurry
images and is recorded with a normal camera. We use bothdatasets to evaluate our algorithm against several baselines
both quantitatively and qualitatively.

to constrain the solution space [2], [4], [16], [16], [28], [37],[37].
While optimization based approaches often offers better
generalization performance, they are usually computational
expensive, which prevents them from time constrained applications. We leverage the commonly used image formation
model by optimization based approaches to construct a loss
term for training our network in a self-supervised fashion.
Since we optimize the parameters of our network once, thecomputationally complex optimization problem does not occur
at test time, where we use the standard efficient feed-forward
inference.

II. R ELATED W ORK

Deep Learning based methods use convolutional neuralnetworks (CNNs) to recover the latent sharp image. Xu et al.
[38] propose a CNN with two sub-networks, a two-hiddenlayer deconvolution network and a two-hidden-layer outlier
rejection network. The network is trained end-to-end withknown ground truth sharp images. An even deeper networkwith 15 layers was proposed in [8] for text image deblurring.
[21] further increased the number of layers to 40 in a multiscale manner, resulting in a network with 120 layers forthree scales. To further improve the network performance, anadversarial loss [6] was used in [17]. [32], [40] use recurrent
neural networks for single image deblurring.
More recently, network architectures for multi-frame inputs,
which are able to exploit temporal information, have been
proposed [15], [29], [36]. Su et al. [29] uses a networkwith skip connections for video deblurring and Wieschollek
et al. [36] exploit temporal information using a recurrent
architecture. A spatio-temporal recurrent architecture with asmall computational footprint was proposed in [15].
Deep learning-based approaches usually outperform
optimization-based methods in terms of both efficiency andimage quality. However, nearly all of the recent deep learning
based methods are trained in a fully supervised manner with
only few notable exceptions. Madam et al. [19] adapted
CycleGAN [43] to the single image deblurring task. Using
unpaired sharp images for training, they obtained good
performance in the specific domain of images (e.g., text,
faces). In contrast to our approach, they require (unpaired)
sharp images for training and perform poorly on blurry images

Motion deblurring methods can be categorized into twogroups: those that assume spatially uniform blur [2], [4], [16],[18], [27], [28], [37] and those considering spatially varying
blur [7], [31], [35]. Uniform deblurring methods assume thatthe blur kernel is identical for each pixel of the input image.
Spatially varying deblurring methods assume that the blurkernels for each pixel may change with respect to its spatial
location. Motion deblurring methods can also be classified into
non-blind deblurring [16], [27], [31] and blind deblurring [2],[4], [7], [18], [28], [35], [37] methods. Non-blind deblurring
methods assume a known blur kernel to recover the latent
sharp image. In contrast, blind deblurring methods need tosimultaneously recover both the latent image and the blurkernel. In this paper, we solve the challenging blind single
image deblurring problem with spatially varying blur.
Optimization-based methods rely on the blur formation
model to recover the latent sharp image by minimizing anenergy function [2], [4], [16], [18], [28], [37], [42], e.g.,using Gaussian [2], [16], [37] or Poisson [27], [31] likelihood
functions in the context of maximum-a-posteriori (MAP) estimation. Depending on the number of input blurry images,
additional terms can be formulated by warping the otherimage(s) to a reference image using either dense flow or by
combining relative camera poses with dense depth maps [14],[22]. Due to the nonlinear and ill-posed (in the case of a single
image) nature of the problem, prior information on either the
motion blur kernel or the latent sharp image must be used

LIU et al.: SELF-SUPERVISED LINEAR MOTION DEBLURRING“in the wild” as demonstrated by our experiments. Chen et al.
[1] propose to include a self-consistency loss for supervision.
However, they report that their self-supervised model leads
to degenerated solutions and hence optimize a hybrid loss
function which heavily relies on supervision in the form of
sharp images. Furthermore, in contrast to our approach, their
method requires triplets of blurry images for training anduses a memory and computational expensive look-up table
for representing the blur kernel in a differentiable fashion. We
avoid this problem using forward warping and differentiablemesh rendering.
III. M ETHOD
Fig. 1 shows the overall architecture of our model. It
comprises four main parts, i.e., the DeblurNet, the FlowNet,
the reblur block, and image warping. The DeblurNet is used
for single image motion deblurring. It accepts a single blurry
image as input and outputs the corresponding sharp image. The
two deblurred images are then fed into the FlowNet to estimate
a bi-directional dense optical flow field, which will be used tocompute spatially varying blur kernels for each blurry image.
Given the estimated blur kernels, we reblur the latent sharp
image to form a self-consistency loss to supervise the trainingof our network. The FlowNet is trained by maximizing crossview photometric consistency, which is estimated by image
warping. While our approach uses two images for training,
the DeblurNet only uses a single input image. After training,
our method can thus be used for single image deblurring. The
whole network is trained end-to-end without using any groundtruth data in the form of sharp images or optical flow. We will
now present all components of our model (i.e., the deblurring,
optical flow, reblurring and image warping components as wellas the loss functions) in detail.
A. Deblurring and Optical Flow
For the deblurring and optical flow modules, we takeadvantage of existing neural network architectures which have
performed well in the past for the respective supervised
learning tasks [9], [17], [21], [30], [32]. In particular, we
adopt the single image deblurring network from Tao et al. [32]and the dense optical flow estimation network PWC-Net fromSun et al. [30]. We make the following modifications for the
deblurring network for our particular problem: 1) We replace
the deconvolution layer with bilinear upsampling followed by
a 3x3 convolution to avoid upsampling artifacts. 2) We add
one more Encoder-Decoder block to increase the capacityof the network. 3) We train the network at a single scale
without using the LSTM layer to improve both the training
and test efficiency. The resulting network is more efficientthan the original network while keeping similar deblurring
performance.
B. Reblurring
The reblurring module encapsulates the physical image formation process, which blurs a sharp image based on the opticalflow. Digital cameras operate by collecting photons during the3time of exposure and converting those into measurable charge.
This process can be formalized by considering the blurredcolor image B ∈ RW ×H×3 as the result of integrating virtual
sharp images It ∈ RW ×H×3 :NZ τIt (x)dt ≈B(x) =0X
1
Ii (x)
2N + 1(1)i=−NHere, τ is the exposure time, x ∈ R2 represents the pixel
location, B(x) denotes the motion blurred image at pixel x,
and It (x) is the virtual sharp image at pixel x and time t. The
continuous integration can be approximated by using a finite
sample size of 2N + 1 virtual sharp frames Ii . We denote thecentral reference frame, which is the latent sharp image to be
estimated, as I0 .
As the exposure time τ is typically small (<200 ms), we
may assume that during the time of exposure the image content
is primarily affected by image motion and not by other changes
like object appearance or illumination. We thus model thevirtual sharp frames Ii as the result of the sharp central
reference frame I0 warped by optical flow ui→0 :Ii (x) = I0 (x + ui→0 )(2)Here, ui→0 ∈ R2 denotes the optical flow from virtual image
Ii to reference image I0 at pixel x. Thus, we can reformulate(1) asNX
1
I0 (x + ui→0 )(3)
B(x) ≈
2N + 1i=−Nand estimate I0 as well as the optical flow fields Ui→0 insteadof all virtual frames Ii for solving the deblurring problem.
However, the problem is still severely underconstrained as
we would need to estimate one optical flow per frame i ∈{−N, . . . , N }.
We therefore further simplify the model by assuming linear
motion during the time of exposure. This is a reasonableassumption in many scenarios where the exposure time iscomparably small and rapid motion changes during this time
are prevented by the mass and inertia of physical objects (e.g.,when the camera is mounted to a vehicle).
Let u ∈ R2 denote the optical flow from frame I0 to frame
I1 at pixel x. Assuming linear motion and equidistant time
steps, we obtain the optical flow from frame 0 to frame i as
u0→i = i · u. Note that in this model the direction of the
optical flow is reversed compared to (3). We must therefore
apply forward warping to obtain the virtual sharp images Ii .
This yields
NX
1
B(x) ≈(W0→i ◦ I0 )(x)2N + 1(4)i=−Nwhere the operator W0→i warps the reference frame I0 into thevirtual frame Ii based on the interpolated flow u0→i . We next
describe our implementation of the forward warping operator
W0→i . Note that this operator needs to be differentiable asboth the reference frame I0 and the optical flow U are outputsof neural networks.
We first construct a regular triangular lattice from the pixel
grid by connecting vertices from adjacent pixels as shown in4IEEE ROBOTICS AND AUTOMATION LETTERS. PREPRINT VERSION. ACCEPTED JANUARY, 2020

Deblur (Sec 3.1)

Testing
Training

Reblur (Sec 3.2)

Flow (Sec 3.1)

Warp (Sec 3.3)

Flow (Sec 3.1)

Warp (Sec 3.3)

Deblur (Sec 3.1)

Reblur (Sec 3.2)

Fig. 2: Architecture of the proposed network. Given two consecutive blurry images, Ba and Bb , as input, our network
computes the corresponding deblurred images, Ia and Ib , as well as the bidirectional optical flow, Ua and Ub . To selfsupervise the training of the network, we construct a self-consistency photometric loss Lself and a forward-backward photometricconsistency losses, Lfw/bw .
Fig. 3 for an example image of size 3×3 pixels. We then warp
each vertex of this lattice according to the optical flow u0→i .
The intensities of Ii (i.e., of the blue pixels) are obtained bylinear interpolation a . Consider the red pixel x in Fig. 3 as an
example. Let further x0 , x1 and x2 denote the positions of thevertices belonging to the triangle which covers the red pixel.
Then, Ii is obtained as
Ii (x) = ω0 I0 (x0 ) + ω1 I0 (x1 ) + ω2 I0 (x2 )Optical Flow(5)

where ω0 , ω1 and ω2 denote the barycentric coordinates ofthe point in the triangle. The synthesized motion blurred
image can then be computed as the average of all the warped
frames as described in (1). In the case of occlusions, i.e.,
when multiple triangles overlap a single pixel, we consider
the triangle with the largest motion to be in front. This is
a commonly used heuristics [13] which often holds true inpractice (in particular when image motion is dominated bycamera motion). Note that due to the linear interpolation, thewarping function W0→i is piecewise smooth. As illustrated inFig. 1, our model is symmetric, thus we reblur both the first
and the second frame and compare the reblurred result to theoriginal blurry images using a photoconsistency loss. We will
use B0 to denote the reblurred image and B to denote theblurred input image in the following.
C. Image Warping
We found that a photoconsistency loss on the reblurred
images alone is insufficient to constrain the optical flow. We
thus add an additional self-supervised photometric loss on the
optical flow as proposed in prior work [12], [20], [34] anddetailed in Section III-D. The input to this loss function isa Note that bilinear interpolation cannot be applied since the warped gridmight not be rectangular due to the non-uniform optical flow, as shown in
Fig. 3.

Fig. 3: Differentiable forward warping. We construct aregular triangular lattice from the pixel grid of the reference
image I0 (left). We then warp each vertex of this lattice
according to the optical flow u0→i . The intensities of Ii (i.e.,of the blue pixels) are obtained by linear interpolation.
the deblurred image and the deblurred image from the otherframe warped based on the estimated optical flow. To warp
the images into each other, we exploit backward warping as
the optical flow in both directions is known. Let Ia ≡ Ia0 andIb ≡ Ib0 denote the first and the second deblurred image, andlet ua→b and ub→a denote the optical flow between themb .
The warped deblurred images are obtained as
I0a (x) = Ib (x + ua→b )(6)I0b (x) = Ia (x + ub→a )(7)using bilinear interpolation [11]. Note that no triangular meshneeds to be constructed during backward warping.
D. Loss Functions
Our network comprises two types of losses: a selfconsistency loss Lself and a forward-backward consistency lossb Note that the flow u
a→b / ub→a and u from the previous section are
related by a known constant that depends on the frame rate, exposure timeand N .

LIU et al.: SELF-SUPERVISED LINEAR MOTION DEBLURRING

5

Lfw/bw . The self-consistency loss
Lself = kB0a − Ba k1 + kB0b − Bb k1

IV. E XPERIMENTAL E VALUATION(8)

penalizes differences between the synthesized motion blurred
images B0a , B0b and the original blurred inputs Ba , Bb using a`1 loss function. Similarly, the forward-backward consistency
loss
Lfw/bw = kI0a − Ia k1 + kI0b − Ib k1(9)penalizes differences between the warped deblurred images I0a ,I0b and the estimated deblurred images Ia , Ib . The final loss isa weighted combination
L = Lself + λLfw/bw ,(10)

where λ is a hyper-parameter to balance both losses.
E. Occlusion HandlingAs occlusions affect the training of our network, especiallyat image boundaries, we detect occluded image regions andmask the loss functions (8) and (9) accordingly. We follow themethod used in [34] to detect occluded image regions. Morespecifically, we compute the non-occluded regions in Ia byfollowing the optical flow Ub→a from Ib to Ia . We consider
all pixels of Ia which can be reached from Ib via Ub→a as
non-occluded. Similarly, we can also compute a mask for each
virtual frame by following the optical flow from the central
image to the virtual frame u0→i . Since the synthesized blurry
image is computed as the average of these virtual frames, we
compute the final mask for Lself as the product of all masks
for the virtual frames.
F. Differences with the method proposed by [1]The overall structure of our method is similar to the workfrom [1]. However, we are different in the following two key
aspects: 1) In order to achieve state-of-the-art performance,[1] uses a supervised loss (section 3.4 from [1]). [1] is thus
actually a supervised method. 2) The core component ofboth methods, i.e., the reblurring module, is different. In fact,
blurring a sharp image using convolutions as done in [1] is
physically incorrect (section 3.3 from [1]) and only holds for
spatially uniform blur. We will make the differences to [1]more clear.
For simplicity, let us assume we have a one dimensional
sharp image I with N pixels. We further assume the blur kernel
corresponds to pixel Ii as {−2, 2} in the form of bidirectional
1D flow. Using the definition from [1], the convolution
based
P2
model results in a blurred image of Ii as Bi = 15 j=−2 Ii+j .
A blur kernel {−2, 2} of Ii means that Ii will contributeto Bi−2 , Bi−1 , Bi , Bi+1 , Bi+2 physically, in contrast to thatIi−2 , Ii−1 , Ii , Ii+1 , Ii+2 will contribute to Bi as what the
convolution based model in [1] does. Our model eliminates
this problem by forward warping the sharp image I by afraction of the blur kernels at each sampled timestamp. The
blurred image is computed by averaging all these forward
warped sharp images to simulate the real motion blurring
image formation process. Experimental results shown later
demonstrate that the algorithm relying on convolution basedmodel exhibits ringing artifacts on egde boundaries, which
degrade the deblurred images.

Datasets: The dataset from [21] is commonly used to benchmark single image motion deblurring algorithms. It is collectedfrom a hand-held camera. Stronger blur was artificially createdby shaking the camera during the recordings. It results invery non-linear camera motions, which violates our motion
assumption. Therefore, we collected a new large dataset using
a professional Fastec TS5c high speed camera mounted on
a car. The dataset consists of 196 sequences in total, which
are collected at 1200 fps with VGA resolution in diverse
environments. The motion blurred images are generated byaveraging several consecutive frames (i.e., 1∼50 frames) tosimulate the real physical image formation process. To reduce
the redundancy per image sequence, we limit the maximum
number of blurry-sharp image pairs to 20 per sequence, which
results in a total of 3606 pairs. We split the dataset into
157 training sequences and 39 test sequences, which resultsin 2820 image pairs for training and 786 image pairs forevaluation.
We also collect a real motion blurry dataset with 2302
images. The camera is mounted on a tram and captures images
at around 50 FPS with a resolution of 752×480 pixels. The
dataset is collected at late afternoon and night, when the
motion blur would really occur. We split 2062 images fortraining and 240 images for test.
Implementation details: We implemented our network in
PyTorch [23]. We empirically set the hyper-parameter λ tobe 2.0. To better initialize the network, we pretrain both the
DeblurNet and PWC-Net on the blurry images. In particular,
we pretrain the DeblurNet for 30 epochs to learn the identity
mapping from blurry image to blurry image. We pretrain the
PWC-Net for 200 epochs with the blurry sequences in a selfsupervised manner. The learning rate used for both networks
is 10−4 . The whole network is then trained jointly for another500 epochs, with a learning rate of 10−4 for the first 260
epochs and then decayed by half every 40 epochs.
Baselines and experimental settings: We compare the single
image deblurring results of our network quantitatively andqualitatively with a state-of-the-art optimization-based method[39], supervised methods [17], [21], [32] as well as thedomain specific self-supervised method from [19]. We train
all networks with their recommended hyperparameter settingson our synthetic dataset. For the optimization-based methodfrom [39], we increase the blur kernel size to 10 pixels toaccount for the large motions present in our dataset.
Evaluation metrics: We use the Peak Signal to Noise Ratio(PSNR) and the Structural Similarity Index (SSIM) measurescommonly used in the community [21], [28], [32] to evaluate
the quality of the deblurring results. Larger PSNR/SSIM
values indicate better image quality. The efficiency of themethods is evaluated by their total time consumption, but
excluding the image loading and saving time.
Ablation studies on the modified DeblurNet architecture:
As discussed in Sec.III-A, we did several improvements toc https://www.fastecimaging.com/fastec-high-speed-cameras-ts-series/

6

IEEE ROBOTICS AND AUTOMATION LETTERS. PREPRINT VERSION. ACCEPTED JANUARY, 2020

Network

PSNR↑

SSIM↑

Time ↓SRN-Deblur [32]34.64 dB

0.93

0.13 sOurs (supervised)

35.04 dB

0.94

0.05 sTABLE I: Single image deblurring comparison on thesynthetic dataset. We compare our modified network withthe original network from Tao et al. [32] by training both ina supervised way.

the original network from [32]. To evaluate the efficacy of the
new network, we train both networks in a supervised manneron our synthetic dataset. Table I presents the comparisons
when evaluated under the same settings. It demonstrates our
DeblurNet is more efficient than the original network while
has slightly better deblurring performance.
Ablation studies on the self-supervision loss for the flow
network: To better understand the proposed algorithm, we
perform an ablation study on the necessity to train the flow
network in a self-supervised manner. We train our proposed
network with and without the self-supervision loss for theflow network. The officially provided pretrained model on
FlyingChair dataset [10] is used if the self-supervision loss
is disabled. Experimental results demonstrate that the flow
network pretrained on the FlyingChair dataset [10] can generalize to our dataset, but with limited performance. The
resulting deblur network gives a PSNR metric as 31.23dB anda SSIM metric as 0.89 on our synthetic dataset, in contract
to 32.24dB/0.91 if the network is trained in a fully selfsupervised manner. It proves it is beneficial to train the flow
network with a self-supervision loss.
Ablation studies on our proposed reblur model: As
discussed in Sec.III-F, our ablation study supports our claim
about the difference between the reblurring modules. For faircomparisons, we trained both the network with convolution
based reblur model and the network with our physically correct
reblur model under the same settings in an unsupervised
fashion. The network with convolution based image formation
model yields a PSNR metric of 27.22dB and a SSIM metricof 0.8 on our synthetic dataset, while ours yields PSNR and
SSIM metrics as 32.24dB and 0.91 respectively.
The necessity to do self-supervised motion deblurring: In
real scenarios, motion blur usually occurs in bad illumination
conditions. In these cases, it impedes the acquisition with low
shutter times to obtain sharp images for supervised learning.
One way to address this problem is to train a network withdatasets collected under good illumination conditions andtransfer the model to scenarios, where motion blur would
occur. However, the generalization ability is still questionable
due to the large difference between the image textures for bothscenarios. We thus evaluate the generalization performancequantitatively and qualitatively with both our synthetic dataset
and real dataset respectively. Note that it is not easy to obtainground truth sharp images in real scenarios. We apply the
pretrained networks on our test data directly, to evaluate thegeneralization performance. Table II and Fig. 5 present the

Method

PSNR↑

SSIM↑

Time ↓

Opt.-based

Xu et al. [39]26.04 dB

0.78

377.8 sSupervised-retrained

DeepDeblur [21]DeblurGAN [17]SRN-Deblur [32]33.55 dB
33.23 dB
34.64 dB

0.92
0.91
0.93

3.45 s
0.06 s0.13 sSupervised-pretrained

DeepDeblur [21]DeblurGAN [17]SRN-Deblur [32]29.91 dB
28.70 dB
30.71 dB

0.87
0.88
0.88

3.45 s
0.06 s0.13 sselfsupervised

Madam et al. [19]Ours

21.69 dB
32.24 dB

0.75
0.91

0.25 s0.05 sTABLE II: Single image deblurring on synthetic dataset.
Supervised-retrained denotes we retrained the networks with
our training data. Supervised-pretrained denotes we use theofficial pretrained models to evaluate on our test data directly.

experimental results. It demonstrates that all the baseline networks have limited generalization ability and perform worse
than our method with a large margin. It proves that selfsupervision is beneficial for the network to adapt to scenarios,
where the ground truth data is difficult to obtain.
Quantitative and qualitative evaluations on synthetic
dataset: Table II and Fig. 4 show quantitative and qualitative comparisons on the synthetic dataset. For the qualitative
results, we only compare against the best supervised method[32]. As can be seen in Table II, our method outperforms
both [39] and [19] significantly in terms of PSNR and SSIM.
For optimization-based single image deblurring algorithms(e.g., [39]), they usually assume the motion blur is causedby either camera rotation or in-plane translation. However, thisassumption is violated in our setting for a self-driving scenario.
Thus, [39] leads to poor performance on our dataset. [19] isdesigned for simple domain-specific blurry images, such as
text and facial images. Therefore, it struggles on our datasetthat exhibits complex real-world challenges which are harder
to learn. In comparison to supervised methods, our method
demonstrates competitive results in our quantitative and qualitative evaluation. As expected, there is still a gap between our
method and the supervised methods if the ground truth sharp
images are available. However, our method outperforms themwith a large margin if they are pretrained on other datasets.
It demonstrates that self-supervision enables the network to
generalize better to real scenarios, where the ground truth
data is usually difficult to obtain. It also demonstrates that
our method is amongst the fastest methods and can run in realtime on a single GTX1080Ti Graphic card.
Qualitative evaluations on real dataset: Since we do
not have ground truth sharp images in our real dataset,
we cannot refine the baseline networks on it. We thus use
the official pretrained networks for the experiments. Fig. 5
demonstrates that our method can successfully deblur theblurry images, while the pretrained network from Tao et al.
[32] results in images with artifacts. More experimental
results can be found from our supplementary material athttps://github.com/ethliup/SelfDeblur.

LIU et al.: SELF-SUPERVISED LINEAR MOTION DEBLURRING

7

Fig. 4: Qualitative comparisons on synthetic dataset. First: Ground truth sharp image. Second: Input blurry image. Third:
Deblurring results of the supervised method from Tao et al. [32]; The network is retrained on our dataset. Fourth: Deblurring
results of the proposed self-supervised learning method.

Fig. 5: Qualitative evaluations on real dataset. First Blurry image. Second Deblurred image by the official pretrained
network from Tao et al. [32]. Third Deblurred image by our method. The images are post processed for better visualization.
Best viewed in digital version.

8IEEE ROBOTICS AND AUTOMATION LETTERS. PREPRINT VERSION. ACCEPTED JANUARY, 2020

V. C ONCLUSION AND F UTURE W ORK
In this paper, we have presented a self-supervised learningalgorithm for image deblurring. Instead of using ground truth
sharp images, we leverage the geometric constraints between
two consecutive blurry images to supervise training of our
network. Both the latent sharp image and motion blur kernel
are estimated by a deblur network and an optical flow estimation nework, respectively. Experimental results show that the
proposed algorithm outperforms the previously self-supervised
method and can produce competitive results compared tosupervised methods. It further demonstrates that our method
can be trained with real motion blurry data and generalizes
well to real unseen data.
R EFERENCES[1] H. Chen, J. Gu, O. Gallo, M. Liu, and J. Kautz. Reblur2deblur: Deblurring videos via self-supervised learning. In International Conference on
Computational Photography (ICCP), 2018. 3, 5
[2] S. Cho and S. Lee. Fast motion deblurring. ACM Transactions on
Graphics (SIGGRAPH ASIA 2009), 28(5), 2009. 1, 2
[3] C. Doersch, A. Gupta, and A. A. Efros. Unsupervised visual representation learning by context prediction. In Proc. of the IEEE International
Conf. on Computer Vision (ICCV), 2015. 1[4] R. Fergus, B. Singh, A. Hertzmann, S. T. Roweis, and W. T. Freeman.
Removing camera shake from a single photograph. In ACM Trans. on
Graphics, 2006. 1, 2
[5] C. Godard, O. Mac Aodha, and G. J. Brostow. Unsupervised monoculardepth estimation with left-right consistency. In Proc. IEEE Conf. on
Computer Vision and Pattern Recognition (CVPR), 2017. 1[6] I.-J. Goodfellow, J. PougetAbadie, M. Mirza, B. Xu, D. Warde-Farley,
S. Ozair, A. Courville, and Y. Bengio. Generative adversarial nets. In
Advances in Neural Information Processing Systems (NIPS), 2014. 2[7] A. Gupta, N. Joshi, C. L. Zitnick, M. Cohen, and B. Curless. Single image deblurring using motion density functions. In European Conference
on Computer Vision (ECCV), 2010. 1, 2
[8] M. Hradis, J. Kotera, P. Zemcik, and F. Sroubek. Convolutional neuralnetworks for direct text deblurring. In Proc. of the British Machine
Vision Conf. (BMVC), 2015. 2[9] E. Ilg, N. Mayer, T. Saikia, M. Keuper, A. Dosovitskiy, and T. Brox.
FlowNet 2.0: Evolution of optical flow estimation with deep networks. In
Proc. IEEE Conf. on Computer Vision and Pattern Recognition (CVPR),
2017. 1, 3
[10] E. Ilg, T. Saikia, M. Keuper, and T. Brox. Occlusions, motion and depth
boundaries with a generic network for disparity, optical flow or scene
flow estimation. In Proc. of the European Conf. on Computer Vision(ECCV), 2018. 6[11] M. Jaderberg, K. Simonyan, A. Zisserman, and K. Kavukcuoglu. Spatialtransformer networks. In Advances in Neural Information Processing
Systems (NIPS), 2015. 4[12] J. Janai, F. Güney, A. Ranjan, M. Black, and A. Geiger. Unsupervised
learning of multi-frame optical flow with occlusions. In Proc. of theEuropean Conf. on Computer Vision (ECCV), 2018. 1, 4
[13] H. Kato, Y. Ushiku, and T. Harada. Neural 3d mesh renderer. In Proc.
IEEE Conf. on Computer Vision and Pattern Recognition (CVPR), 2018.
4[14] T. H. Kim and K. M. Lee. Generalized video deblurring for dynamic
scenes. In Proc. IEEE Conf. on Computer Vision and Pattern Recognition (CVPR), 2015. 2[15] T. H. Kim, K. M. Lee, B. Scholkopt, and M. Hirsch. Online video
deblurring via dynamic temporal blending network. In Proc. of theIEEE International Conf. on Computer Vision (ICCV), 2017. 2[16] D. Krishnan and R. Fergus. Fast image deconvolution using hyperlaplacian priors. In Neural Information Processing Systems Conference(NIPS), 2009. 1, 2
[17] O. Kupyn, V. Budzan, M. Mykhailych, D. Mishkin, and J. Matas.
Deblurgan: Blind motion deblurring using conditional adversarial networks. In Proc. IEEE Conf. on Computer Vision and Pattern Recognition(CVPR), July 2018. 1, 2, 3, 5, 6
[18] A. Levin, Y. Weiss, F. Durand, and W. T. Freeman. Understanding
and evaluating blind deconvolution algorithm. In Proc. IEEE Conf. on
Computer Vision and Pattern Recognition (CVPR), 2009. 1, 2
[19] N. T. Madam, S. Kumar, and R. A.N. Unsupervised class-specificdeblurring. In Proc. of the European Conf. on Computer Vision (ECCV),
2018. 2, 5, 6

[20] S. Meister, J. Hur, and S. Roth. UnFlow: Unsupervised learning of
optical flow with a bidirectional census loss. In Proc. of the Conf. on
Artificial Intelligence (AAAI), 2018. 1, 4[21] S. Nah, T. H. Kim, and K. M. Lee. Deep multi-scale convolutional
neural network for dynamic scene deblurring. In Proc. IEEE Conf. on
Computer Vision and Pattern Recognition (CVPR), July 2017. 1, 2, 3,
5, 6[22] H. Park and K. M. Lee. Joint estimation of camera pose, depth,
deblurring and super-resolution from a blurred image sequence. In Proc.
of the IEEE International Conf. on Computer Vision (ICCV), 2017. 2[23] A. Paszke, S. Gross, S. Chintala, G. Chanan, E. Yang, Z. DeVito, Z. Lin,A. Desmaison, L. Antiga, and A. Lerer. Automatic differentiation in
PyTorch. 2017. 5[24] D. Pathak, P. Krähenbühl, J. Donahue, T. Darrell, and A. A. Efros.
Context encoders: Feature learning by inpainting. In Proc. IEEE Conf.
on Computer Vision and Pattern Recognition (CVPR), 2016. 1[25] A. Pretto, E. Menegatti, M. Bennewitz, W. Burgard, and E. Pagello.
A visual odometry framework robust to motion blur. In Proc. IEEE
International Conf. on Robotics and Automation (ICRA), 2009. 1[26] J. Qiu, X. Wang, S. J. Maybank, and D. Tao. World from blur. In Proc.
IEEE Conf. on Computer Vision and Pattern Recognition (CVPR), 2019.
1[27] W. Richardson. Bayesian-based iterative method of image restoration.
Journal of the optical society of America, 62(1), 1972. 2[28] Q. Shan, J. Jia, and A. A. High-quality motion deblurring from a single
image. ACM Transactions on Graphics, 27(3), 2008. 1, 2, 5[29] S. Su, M. Delbracio, and J. Wang. Deep video deblurring for handheld cameras. In Proc. IEEE Conf. on Computer Vision and Pattern
Recognition (CVPR), 2017. 1, 2[30] D. Sun, X. Yang, M.-Y. Liu, and J. Kautz. PWC-Net: CNNs for opticalflow using pyramid, warping, and cost volume. In Proc. IEEE Conf. on
Computer Vision and Pattern Recognition (CVPR), 2018. 3[31] Y.-W. Tai, P. Tan, and M. S. Brown. Richardson-lucy deblurringfor scenes under a projective motion path. IEEE Transactions on
Pattern Analysis and Machine Intelligence (PAMI), 33(8):1603–1618,
Aug. 2011. 2[32] X. Tao, H. Gao, Y. Wang, X. Shen, J. Wang, and J. Jia. Scale-recurrent
network for deep image deblurring. In Proc. IEEE Conf. on Computer
Vision and Pattern Recognition (CVPR), July 2018. 1, 2, 3, 5, 6, 7
[33] S. Tourani, M. Sudhanshu, A. Nagariya, V. Chari, and M. Krishna.
Rolling shutter and motion blur removal for depth cameras. In Proc.
IEEE International Conf. on Robotics and Automation (ICRA), 2016. 1[34] Y. Wang, Y. Yang, Z. Yang, L. Zhao, P. Wang, and W. Xu. Occlusion
aware unsupervised learning of optical flow. In Proc. IEEE Conf. on
Computer Vision and Pattern Recognition (CVPR), 2018. 4, 5[35] O. Whyte, J. Sivic, A. Zisserman, and J. Ponce. Non-uniform deblurringfor shaken images. In Conference on Computer Vision and Pattern
Recognition (CVPR), 2010. 2[36] P. Wieschollek, M. Hirsch, and B. Scholkopt. Learning blind motion
deblurring. In Proc. of the IEEE International Conf. on Computer Vision(ICCV), 2017. 2[37] L. Xu and J. Jia. Two-phase kernel estimation for robust motion
deblurring. In European Conference on Computer Vision (ECCV), 2010.
1, 2[38] L. Xu, J.-S. Ren, C. Liu, and J. Jia. Deep convolutional neural network
for image deconvolution. In Advances in Neural Information Processing
Systems (NIPS), 2014. 2[39] L. Xu, S. Zheng, and J. Jia. Unnatural l0 sparse representation for
natural image deblurring. In Proc. IEEE Conf. on Computer Vision and
Pattern Recognition (CVPR), 2013. 5, 6[40] J. Zhang, J. Pan, J. Ren, and Y. Song. Dynamic scene deblurring using
spatially variant recurrent neural networks. In Proc. IEEE Conf. on
Computer Vision and Pattern Recognition (CVPR), 2018. 2[41] Y. Zhang, S. Khamis, C. Rhemann, J. P. C. Valentin, A. Kowdle,
V. Tankovich, M. Schoenberg, S. Izadi, T. A. Funkhouser, and S. R.
Fanello. Activestereonet: End-to-end self-supervised learning for active
stereo systems. In Proc. of the European Conf. on Computer Vision(ECCV), 2018. 1[42] S. Zheng, L. Xu, and J. Jia. Forward motion deblurring. In Proc. of theIEEE International Conf. on Computer Vision (ICCV), 2013. 2[43] J.-Y. Zhu, T. Park, P. Isola, and A. A. Efros. Unpaired image-to-imagetranslation using cycle-consistent adversarial networks. In Proc. of theIEEE International Conf. on Computer Vision (ICCV), 2017. 2

LIU et al.: SELF-SUPERVISED LINEAR MOTION DEBLURRING

1Supplemental Materials: Self-Supervised Linear Motion Deblurring
I. I NTRODUCTION
In this supplementary material, we present details on the relationship between ua→b /ub→a (i.e., the optical flow between
the latent sharp images) and u (i.e., the optical flow between the central virtual frame and the first virtual frame), describedin Section III.B and III.C of the main paper, the architecture of the deblurring network, as well as additional qualitative
experimental results on single image deblurring.

II. R ELATIONSHIP BETWEEN ua→b /ub→a AND u
In this section, we present the relationship between ua→b /ub→a and u. ua→b and ub→a are the bidirectional dense opticalflows between the latent sharp images Ia and Ib , respectively. We assume the motion between Ia and Ib to be linear. Without
loss of generality, we assume u is used to synthesize the first blurry image Ba . Thus, we obtain u as the flow from the central
virtual frame I0 to the first virtual image I1 by linearly scaling ua→b according tou≈τaua→b ,
2N∆t(1)

where τa is the exposure time of Ba , 2N + 1 is the number of sampled virtual sharp frames to synthesize Ba , and ∆t is thetime interval between Ia and Ib . Similarly, if u is used to synthesize the second blurry image Bb , we getu≈τbub→a ,2N∆t(2)where τb is the exposure time of Bb .

III. A RCHITECTURE OF THE D EBLURRING N ETWORK

Fig. 1: Architecture of the deblurring network. Given the input blurry image Ba , the deblurring network outputs the deblurred
latent sharp image Ia . Best viewed in enlarged digital version.
We adapt the single image deblurring network from Tao et al. [3] for our approach. We make the following modifications
for our particular problem: 1) We replace the deconvolution layer with bilinear upsampling followed by a 3x3 convolution
to avoid upsampling artifacts. 2) We train the network at a single scale without using the LSTM layer to improve both thetraining and test efficiency. 3) We add one more Encoder-Decoder block to increase the capacity of the network. The details
are shown in Fig. 1.

2IEEE ROBOTICS AND AUTOMATION LETTERS. PREPRINT VERSION. ACCEPTED JANUARY, 2020

IV. A DDITIONAL Q UALITATIVE E XPERIMENTAL R ESULTSIn Fig. 2 to Fig. 8, we present additional qualitative experimental results on single image deblurring on the synthetic dataset.
The results demonstrate that our method can generate visually compelling sharp images that are competitive to three state-ofthe-art supervised methods [2]–[4]. For fair comparisons, we retrain all the networks on our Fastec dataset. It also significantly
outperforms the state-of-the-art optimization-based method from Xu et al. [1] and the self-supervised method from [5]. The
optimization based method from Xu et al. [1] fails to deblur blurry images from this dataset. To make the problem tractable,
they assume that the motion blur is caused by either camera rotation or in-plane translation. However, those assumptions are
violated in our Fastec dataset, where the motion blur is also caused by the 3D scene geometry.
To further demonstrate the temporal consistency of our method, we also present the experimental results for image sequencesfrom Fig. 9 to Fig. 14. The experimental results demonstrate that our network can deblur an image sequence temporally
consistent, on both the synthetic and real datasets.
R EFERENCES[1] L. Xu, S. Zheng and J. Jia, Unnatural L0 sparse representation for natural image deblurring, In Proc. IEEE Conf. on Computer Vision and Pattern
Recognition (CVPR), 2013. 2[2] S. Nah, T. Kim and K. Lee, Deep Multi-Scale Convolutional Neural Network for Dynamic Scene Deblurring, In Proc. IEEE Conf. on Computer Vision
and Pattern Recognition (CVPR), 2017. 2[3] X. Tao, H. Gao, Y. Wang, X. Shen, J. Wang and J. Jia, Scale-recurrent network for deep image deblurring, In Proc. IEEE Conf. on Computer Vision
and Pattern Recognition (CVPR), 2018. 1, 2
[4] O. Kupyn, V. Budzan, M. Mykhailych, D. Mishkin and J. Matas, DeblurGAN: Blind motion deblurring using conditional adversarial networks, In Proc.
IEEE Conf. on Computer Vision and Pattern Recognition (CVPR), 2018. 2[5] N. Madam, S. Kumar and A.N. Rajagopalan, Unsupervised Class-Specific Deblurring, In Proc. of the European Conf. on Computer Vision (ECCV),
2018.

2

LIU et al.: SELF-SUPERVISED LINEAR MOTION DEBLURRING

Fig. 2: Qualitative comparisons on the Fastec dataset. All the baseline networks are retrained on our Fastec dataset.

34IEEE ROBOTICS AND AUTOMATION LETTERS. PREPRINT VERSION. ACCEPTED JANUARY, 2020

Fig. 3: Qualitative comparisons on the Fastec dataset. All the baseline networks are retrained on our Fastec dataset.

LIU et al.: SELF-SUPERVISED LINEAR MOTION DEBLURRING

Fig. 4: Qualitative comparisons on the Fastec dataset. All the baseline networks are retrained on our Fastec dataset.

56IEEE ROBOTICS AND AUTOMATION LETTERS. PREPRINT VERSION. ACCEPTED JANUARY, 2020

Fig. 5: Qualitative comparisons on the Fastec dataset. All the baseline networks are retrained on our Fastec dataset.

LIU et al.: SELF-SUPERVISED LINEAR MOTION DEBLURRING

Fig. 6: Qualitative comparisons on the Fastec dataset. All the baseline networks are retrained on our Fastec dataset.

78IEEE ROBOTICS AND AUTOMATION LETTERS. PREPRINT VERSION. ACCEPTED JANUARY, 2020

Fig. 7: Qualitative comparisons on the Fastec dataset. All the baseline networks are retrained on our Fastec dataset.

LIU et al.: SELF-SUPERVISED LINEAR MOTION DEBLURRING

Fig. 8: Qualitative comparisons on the Fastec dataset. All the baseline networks are retrained on our Fastec dataset.

910IEEE ROBOTICS AND AUTOMATION LETTERS. PREPRINT VERSION. ACCEPTED JANUARY, 2020

Fig. 9: Temporal consistency on the Fastec dataset (frame 1-5). Left: Ground truth. Middle: Blurry image. Right: Deblurred
image by our network.

LIU et al.: SELF-SUPERVISED LINEAR MOTION DEBLURRING

11

Fig. 10: Temporal consistency on the Fastec dataset (frame 6-10). Left: Ground truth. Middle: Blurry image. Right:
Deblurred image by our network.

12IEEE ROBOTICS AND AUTOMATION LETTERS. PREPRINT VERSION. ACCEPTED JANUARY, 2020

Fig. 11: Temporal consistency on the Fastec dataset (frame 11-15). Left: Ground truth. Middle: Blurry image. Right:
Deblurred image by our network.

LIU et al.: SELF-SUPERVISED LINEAR MOTION DEBLURRING

13

Fig. 12: Temporal consistency on the Fastec dataset (frame 16-19). Left: Ground truth. Middle: Blurry image. Right:
Deblurred image by our network.

14

IEEE ROBOTICS AND AUTOMATION LETTERS. PREPRINT VERSION. ACCEPTED JANUARY, 2020

Fig. 13: Temporal consistency on the real dataset. Odd rows: Blurry image. Even rows: Deblurred image by our network.

LIU et al.: SELF-SUPERVISED LINEAR MOTION DEBLURRING

15

Fig. 14: Temporal consistency on the real dataset. Odd rows: Blurry image. Even rows: Deblurred image by our network.