This article has been accepted for inclusion in a future issue of this journal. Content is final as presented, with the exception of pagination.
IEEE TRANSACTIONS ON INTELLIGENT TRANSPORTATION SYSTEMS

1

Self-Supervised Optical Flow Estimation
by Projective Bootstrap
Stefano Alletto , Davide Abati , Simone Calderara , Member, IEEE, Rita Cucchiara, and Luca Rigazio
Abstract— Dense optical flow estimation is complex and time
consuming, with state-of-the-art methods relying either on large
synthetic data sets or on pipelines requiring up to a few minutes
per frame pair. In this paper, we address the problem of opticalflow estimation in the automotive scenario in a self-supervisedmanner. We argue that optical flow can be cast as a geometrical
warping between two successive video frames and devise a deeparchitecture to estimate such transformation in two stages. First,
a dense pixel-level flow is computed with a projective bootstrap on
rigid surfaces. We show how such global transformation can beapproximated with a homography and extend spatial transformer
layers so that they can be employed to compute the flow field
implied by such transformation. Subsequently, we refine theprediction by feeding a second, deeper network that accounts
for moving objects. A final reconstruction loss compares thewarping of frame X t with the subsequent frame X t +1 and guides
both estimates. The model has the speed advantages of end-toend deep architectures while achieving competitive performances,both outperforming recent unsupervised methods and showing
good generalization capabilities on new automotive data sets.
Index Terms— Computer vision, image motion analysis, unsupervised learning.

I. I NTRODUCTION
N THE last few years, we assisted to a growing interest
from the computer vision and machine learning community
towards autonomous and assisted driving applications. In thiscontext, optical flow estimation represents one of the mostactive research fields but, despite the efforts, it is still an open
problem.
This is due to two main factors. In the first place,the car ego-motion heavily affects the optical flow field.
Indeed, urban scenes are mainly composed of static, still
objects, whose moving patterns within the image plane arestrongly correlated to the camera movement. Nevertheless,several surrounding objects can move independently (suchas cars and pedestrians), and their movements are perhaps
even more important. For an optical flow model to be reliable, both types of motion need to be correctly estimated.

I

Manuscript received March 24, 2018; revised August 24, 2018; accepted
September 30, 2018. The Associate Editor for this paper was S. C. Wong.
(Corresponding authors: Stefano Alletto; Luca Rigazio.)
S. Alletto was with the Enzo Ferrari Department of Engineering, University of Modena and Reggio Emilia, 41125 Modena, Italy. He is
now with Panasonic Beta, Mountain View, CA 94043 USA (e-mail:stefano.alletto@us.panasonic.com).
D. Abati, S. Calderara, and R. Cucchiara are with the Enzo Ferrari
Department of Engineering, University of Modena and Reggio Emilia, 41125
Modena, Italy.
L. Rigazio was with the Panasonic Silicon Valley Laboratory, Cupertino,
CA 95014 USA. He is now with Panasonic Beta, Mountain View, CA 94043
USA (e-mail: luca.rigazio@us.panasonic.com).
Color versions of one or more of the figures in this paper are available
online at http://ieeexplore.ieee.org.
Digital Object Identifier 10.1109/TITS.2018.2873980

Fig. 1.
Examples of motion flows obtained through our projectivebootstrapped network. Rows from top to bottom: input example, projective
bootstrap, final flow estimate.

Furthermore, existing machine learning models are trainedwith precise pixel-level optical flow ground-truth maps, thatare extremely hard to collect. This represents a significant
issue in general, and is even more critical in the automotive field. For this reason, automotive datasets typicallylack optical flow ground-truth information [1], [4], [8]. The
only suitable real-world labeled dataset is the popular Kitti
Flow benchmark [11], that is collected by means of expensive, dedicated hardware. Indeed, ground-truth flow maps arecomputed by means of 2D-3D matching of point clouds
acquired by a LIDAR sensor. Alternatively, computer graphics
approaches [10] or videogames [24] are employed to acquirelarger synthetic datasets at the expense of photorealism. On theother hand, acquiring a video from the perspective of a car istrivial. Despite the availability of unlabeled driving sequences
is high, recent unsupervised approaches to optical flow estimation either struggle in achieving competitive results [35] or
perform far from real-time [21].
In this paper, we design an end-to-end network that copeswith both issues. Our main contribution is to bootstrap the optical flow estimation by a projective geometric transformation
between consecutive image frames. We argue that the global
ego-motion of a car can be coarsely approximated by the
motion field implied by such transformation, and motivate ourstrategy within the plane2parallax framework [16]. We show
that, when dealing with static scenes and in presence of camera
motion, a specific implicit plane exists, that allows to embedboth planar and parallax components of the flow field in thesame projective matrix. This finding allows us to delegate theestimation of most flow vectors to an extremely shallow, fast
deep network, and then refine its prediction accounting for
depth structure and moving objects (Fig. 1). Moreover, we only
rely on self-supervision to train our model. Building on

1524-9050 © 2018 IEEE. Personal use is permitted, but republication/redistribution requires IEEE permission.
See http://www.ieee.org/publications_standards/publications/rights/index.html for more information.

This article has been accepted for inclusion in a future issue of this journal. Content is final as presented, with the exception of pagination.
2

IEEE TRANSACTIONS ON INTELLIGENT TRANSPORTATION SYSTEMS

the recently proposed Spatial Transformer Layer (STL) [17],we embrace an optimization framework in which, instead of
directly minimizing flow vector errors, we warp images in
estimates of future frames and penalize reconstruction failures.
We show that, despite being very different, such framework
effectively reduces optical flow average point error during
training.
To summarize, here we propose a method which is specifically designed for the challenging automotive scenario and isparticularly suitable for tasks where optical flow ground-truth
cannot be employed. Our self-supervised architecture results insignificant differences compared to traditional supervised deeplearning approaches; on the other hand we improve on recent
unsupervised and self-supervised methods by introducing ouridea of projective bootstrap, a computationally inexpensive
step that we experimentally show to significantly boost theperformance.
The paper is structured as follows: The next section discusses recent works on the topics of optical flow estimation.
Sec. III describes the intuition behind the projective bootstrap,
and the proposed end-to-end model for optical flow estimation.
Sec. IV measures performances and generalization capabilitiesof our model in different settings. Eventually, conclusions andfurther development strategies are discussed in Sec. V.
II. R ELATED W ORK
Here, we identify three main approaches to optical flow
estimation: hand-crafted methods based on heuristics, composed methods and end-to-end strategies, with every category
having its own strengths and drawbacks in terms of speed,
performance and complexity.
A. Hand-Crafted Methods
Since its early days, optical flow estimation has beenaddressed from an image processing perspective, mainly
adopting strategies that rely on the brightness constancy
principle [13]. Models that solely rely on pixel brightness
and motion smoothness suffer of a high sensitivity to outliers, that often occur due to changes in illumination condition and large displacements. To this end, Brox et al. [5]propose a variational approach that deals with the shortcomings of previous methods by jointly accounting for brightness constancy and brightness gradient constancy, as longas introducing piecewise smoothness. On a different note,
Chen et al. [7] approach large displacement optical flow
estimation by first computing a nearest-neighbor field foreach pixel, which is shown to coarsely approximate the flowtransformation. Subsequently, the prediction is refined byestimating and successively transforming dominant motion
patterns. More recently, EpicFlow [23] and DeepFlow [30]have been proposed, achieving excellent performance. The first
method is based on the idea of coarse to fine energy-based
flow refinement that interpolates on a set of matches in anedge-preserving fashion. Similarly, DeepFlow relies on a deep
matching algorithm to obtain point-wise correspondences andon an energy minimization framework that enforces smoothness among the computed flow field. These methods have theadvantage of not relying on specific, dataset-dependent training and thus are better at generalizing; they suffer nonethelessfrom high computational complexities that result in highexecution times.
B. End-to-End Methods
More recently, deep learning based models outperformed
traditional approaches on public benchmarks. In their seminal
work, FlowNet, Fischer et al. [9] introduce one of the firstend-to-end deep architectures for dense optical flow. Usingan conv-deconv network, they address the lack of large
datasets by creating a synthetic image collection featuring
random chairs flying over random landscapes. Authors show
the surprising generalization capability of the proposed model
when inference is performed over real-world sequences froma different domain. Building on their insights, a plethora ofmethods relying on deep neural networks to approach the
problem have been proposed [2], [3], [15], [26], [28]. Someof the drawbacks of this initial method have been addressed in
FlowNet 2.0, a much deeper network achieving performancethat have marked the state of the art. While these methods
obtain good performance in an end-to-end fashion, they rely onlarge synthetic datasets to obtain a ground truth to be used during their training. To deal with this issue, several works address
flow estimation from an unsupervised or self-supervised perspective. Long et al. [21] reformulate the problem as image
interpolation and matching. Training a network to learn theinterpolation between two frames – i.e. to obtain image X tfrom X t −1 and X t +1 – they show how performing per-pixel
backpropagation at test time results into sensitivity maps from
where pixel level matches can be obtained. However, thismethod suffers of high computational cost: despite relying on
modern GPU architectures, authors show how obtaining theflow for an image pair requires n/s backward passes, where
n is the total number of image pixels and s is an arbitrary
stride controlling the sparseness of the resulting map. In [35]another unsupervised model is illustrated, employing a similararchitecture but embedding into its loss function brightness
constancy and motion smoothness constraints, first introducedin [13]. Surprisingly, this method achieves state of the art
performance on the Kitti dataset when compared to other
unsupervised approaches, showing the benefits of leveraging
the representation power of deep learning and traditional
image approaches. More recently, Liu et al. [20] obtain theflow field between adjacent frames by learning to interpolate
adjacent frames through a differentiable volume samplinglayer, that learns whether to sample from the previous or next
frame in order to better deal with occluding and disappearing
pixels. While traditionally supervised methods achieve better
performance, all the aforementioned methods are united bytheir being end-to-end and hence requiring a single forward
pass through a neural network (with the exception of [21]),hence being very fast.
C. Composed Strategies
Finally, methods that rely on a composition ofdifferent components hold the current state of the art

This article has been accepted for inclusion in a future issue of this journal. Content is final as presented, with the exception of pagination.
ALLETTO et al.: SELF-SUPERVISED OPTICAL FLOW ESTIMATION BY PROJECTIVE BOOTSTRAP

on different datasets [2], [14], [29], [31]–[34]. For example,
Yamaguchi et al. [33] propose a method that performs
superpixel segmentation, boundary and occlusion labeling,stereo estimation and computes the flow by using thisinformation jointly. Conversely, Bai et al. [2] propose
to exploit semantic information to compute the flow ofbackground and foreground objects separately. They exploit
a convolutional architecture to obtain a segmentation of thescene and rely on deep matching to obtain the flow fieldof individual objects and the background. Wulff et al. [31]propose instead to rely on a lower level segmentation input,
namely a rigidity estimation map. By initializing their flow
computation using a standard method and strong geometricalpriors and estimating a binary segmentation of rigid versus
non-rigid objects using a convolutional neural network, they
achieve good performance thanks to their ability to adapt theflow computation to different objects. Intuitively, this classof models achieves state-of-the-art performances, but suffersfrom a high complexity resulting in high execution times.
In this picture, we collocate our proposal in the end-to-end
category, aiming at achieving both competitive performanceand the execution times of a forward only neural network,
which is suitable for an automotive scenario. Furthermore,
the proposed method is fully self-supervised thus not requiringextensive ground truth flow information which is an especiallyrare commodity in the automotive setting.

Fig. 2.

3

The plane-parallax setting in presence of a camera translation T .

p its projection in the image plane of C and C  respectively.
By taking an auxiliary plane , the homography matrix Ĥwarps the point p from C  image plane to C in pw . The
motion field projection onto C image plane can be computedin homogeneous coordinate for p as (see [16] for details):tz(1)p − p = ( p  − pw ) − γ  ( pw − e)  dπplanarparallaxIII. M ODEL
Estimating a dense flow between two adjacent frames X t ,X t +1 is an inherently difficult problem. Indeed, for each pixelin X t , two translation components u, v need to be estimated,
resulting in a transformation with 2 × h × w parameters(where h, w represent the height and width of the two frames
respectively). In what follows, we advocate for the existenceof recurrent camera movements in the automotive scenario,
according to which the motion field of a static scene can beroughly estimated through a projective transformation. This
reduces the dimensionality of the output space dramatically,
since the parameters characterizing this projection drop tonine.
A. Projective Bootstrap
It is well studied that the motion field in the case of rigid
structures can be decomposed in two contributions, namelyplane and parallax contributions [16], [19], [25], [27]. This is
the case of automotive settings where the camera is roof
mounted on a moving vehicle: Indeed, a global motion affects
the majority of scene points, and deviations are mainly due tomoving objects in the scene. The plane-parallax decomposition
holds in presence of roto-translational transformations between
two cameras. Specifically, we consider the case of a pure translation between two cameras C and C  , described by the vector
T = {tx , t y , tz }. The geometrical setup is depicted in Fig. 2.
In this particular setting, the projection of the motion field into
the image plane of C moves along a pencil of lines originatingfrom the epipole that is the focus of expansion of the motion
direction. Given a 3D point P = (X, Y, Z ), we define p andwhere dπ is the distance of C  from plane , γ = ZL

is theprojective structure of P w.r.t. plane  and e is the epipole
location in the camera C image plane. By recalling pw = Ĥ p
the first term of equation (1) is the planar component of
the motion field since it depends only on the choice of thereference plane . The term γ dtz ( pw − e) instead, accountsπ
for the parallax effect that is not negligible for points that do
not lie on  and it is directly modulated by L and suppressedby Z distances. When using the four points method [12]for computing the matrix Ĥ point correspondences must be
observable on both image planes. This constraint is also
required by the approach in [31] for optical flow initialization.
Nevertheless, a degenerate configuration for which a projective transformation can simultaneously model both theparallax and planar component can be found by considering
the plane  that pass through camera C and is parallel to its
image plane. Moving the plane toward C makes the two termstz
L
Z and dπ close to 1 independently from the point P. Eq. (1)then simplifies top − p = ( p − pw ) − 2( pw − e)and, since pw(2)= Ĥ p , can be written as
p = 2 Ĥ p − e.

(3)

It is important to notice that the aforementioned plane cannot
be computed from image points correspondences because all
Pw points are projected to the C image plane at infinite. In thecase of pure translation along the Z axis, the epipole locationis the image center (i.e. e = [0, 0, 1]T ) and can be neglected.
Oppositely, in the more general setting the position of theThis article has been accepted for inclusion in a future issue of this journal. Content is final as presented, with the exception of pagination.
4

IEEE TRANSACTIONS ON INTELLIGENT TRANSPORTATION SYSTEMS

Fig. 3. Architecture of the proposed model. Frames X t and X t+1 are fed into a first ego-motion network that estimates a projective transformation. ST L H
H . The bootstrapped flow is then fed to a second network along with input frames, and its
then warps X t accordingly, resulting in the estimated frame X̃ t+1
F . Both reconstructions are guided by a Charbonnier penalty L
refinement F is employed to obtain a second reconstruction, X̃ t+1
ch to approximate X t+1 .
During inference, the output of F-net, F , is the predicted optical flow.

epipole needs to be estimated, and then embedded in the thirdcolumn of Ĥ .1
Eventually we can estimate the motion field by a 3×3 matrixof parameters H asp − p = p − H p.

(4)

B. Architecture and Optimization
In order to leverage the projective bootstrap described inSec. III-A, our architecture estimates the motion field in two
sequential steps. First, a shallow network provides a projective
transformation embedding a motion field approximating theone of a static scene in presence of camera motion (thatwe refer to as homographic flow). Then, a second deeper
network is responsible for its refinement, and for the recovery
of depth and local fields belonging to moving objects. Before
introducing these two components, we briefly illustrate how
a STL can be employed to train an optical flow estimator by
self-supervision.
Given a geometric transformation Tφ estimated from twoframes X t , X t +1 by a generic parametric function (e.g. a neuralnetwork), Tφ (X t ) can be computed by means of a STL [17].
Being fully differentiable, this layer also allows to propagate
the gradient of a given reconstruction loss to the transformation
estimator. In our setting, Tφ embeds optical flow vectors whileX̃ t +1 = Tφ (X t ) results in the estimate of frame X t +1 under
the transformation Tφ . During training, our model is tasked to
optimize a Charbonnier reconstruction penalty, expressed as
(5)Lch = ( X̃ t +1 − X t +1 )2 + ,
where  is a small regularization constant (fixed to 0.1 inour experiments). The Charbonnier penalty is a differentiableversion of the l1 norm and guides the model towards betterreconstructions, i.e. better optical flow estimates (see Fig. 5).
Moreover, this loss penalizes more errors within textured
1 those terms are multiplied by 1 when p  is expressed in projective
coordinates p = (i, j, 1) thus adding a bias independent from p locationin the image

regions or object boundaries, while being more permissive inhomogeneous portions of the reconstructed image.
Notably, this framework requires no supervision other than
the frame X t +1 itself, enabling the training of our model
without ground-truth fields.
We design an end-to-end network, Fig. 3, composed of two
main modules that, as previously introduced, first bootstraps
the flow on rigid structures of the scene (ego-motion network)and subsequently refines the computation accounting for moving objects and errors in the estimation (refinement network).
1) Ego-Motion Network: The bootstrap is demanded toa shallow architecture, from now on named H-net, providing the nine coefficients of the projective transformation H(Sec. III-A) that better describes the ego-motion of the car.
Fig. 4 (a) reports the details of H-net architecture. The
network itself is small sized, featuring 6 convolutional layers
and 3 fully connected layers resulting in 600k parameters
overall. H-net is fed with two subsequent frames stacked on
the channel dimension and outputs the 9 parameters of thetransformation, which are input to the spatial transformer layer
ST L H , that warps Xt into X̃ tH+1 . The projective transformation
affects the whole image and embeds a global, coarse flow fieldthat can be recovered from ST L H . More precisely, given a
uniform sampling grid G holding column-wise pixel homogeneous coordinates, ST L H applies the input transformation
to obtain the warped grid G̃ = H · G. Since G̃ holds thewarped coordinates of each pixel, the flow field can seamlessly
be recovered as FH = G̃ − G. As the operations on G in
ST L H are differentiable, it is then possible to propagate error
gradients through the layer towards H-net.
2) Refinement Network: The flow field produced by H-net
cannot cope with details and moving objects. Therefore we
refine it employing a second, deeper network (F-net) inspiredby the model in [9]. Fig 4 (b) reports the details of F-net
hourglass architecture. The encoder consists of five layer
blocks, each of which is composed by three 3×3 convolutional
layers with leaky ReLU activations. All convolutions in a
block share unary strides except for the last one, which
has stride 2. The decoder mirrors the encoder structure with

This article has been accepted for inclusion in a future issue of this journal. Content is final as presented, with the exception of pagination.
ALLETTO et al.: SELF-SUPERVISED OPTICAL FLOW ESTIMATION BY PROJECTIVE BOOTSTRAP

5

Fig. 4. (a) H-net. All convolutional layers have stride s = 2 and all layers feature leaky ReLU activations, except for the top linearly activated fully connected
module. (b) F-net. Layers in the same block share the number of output features. All activations are leaky ReLU. Convolutional layers feature a 3x3 kernel(both in H-net and F-net), whereas transposed convolutions feature 4x4 kernel. Best viewed on screen and zoomed in.

TABLE IR ESULTS O BTAINED BY THE D IFFERENT C OMPONENTS
OF O UR N ETWORK ON K ITTI T RAINING S ET

Fig. 5. Average point error and reconstruction loss during training on Virtual
Kitti. Note that the first 50 batches are omitted in favor of a more readable
plot scale.

transposed-convolution blocks, and a top 2-channel convolution layer produces the final estimate in the range [−1, 1] using
tanh as the non-linearity. F-net is fed with a channel-wiseconcatenation of X t , X t +1 and FH , and outputs a dense
flow map F ∈ R2×w×h . Hence, the spatial transformer layer
ST L F warps X t into X̃ tF+1 and the reconstruction error w.r.t.
the frame X t +1 is minimized by means of the Charbonnierloss (Eq. (5)). This guides F-net towards focusing on moving
objects and fine details neglected by the projective bootstrap.
IV. E XPERIMENTAL R ESULTS
A. Datasets and Training
Due to the self-supervised nature of our method, we do not
require ground truth flow information to train our network.
For this reason, we are able to employ large-scale automotive datasets such as Kitti raw [11] and DR(eye)VE [1].
In particular, the Kitti raw dataset includes 44,000 framesacquired in the city of Karlsruhe, while DR(eye)VE features
555,000 frames including steep changes in image conditions
due to transitions between day and night, sun, rain and asignificant variance in scenarios such as highway, downtown
or countryside. Notice how the size of the aforementioned
datasets is suitable for training a neural network; on theother hand, the biggest real-world automotive dataset including
ground truth flow information nowadays is Kitti Flow [11],that combines less that 800 annotated pairs split between
training and testing in its two versions. Recently, a synthetic
automotive dataset inspired by Kitti has been released [10].

The Virtual Kitti dataset features more than 21,000 frames
fully annotated with optical flow, semantic segmentation, depth
and object bounding boxes. Due to its recent release, a stateof the art on the Virtual Kitti dataset is not yet established,
nonetheless we include it in our evaluation to show thegeneralization capabilities of our method when challenged
with different automotive scenarios.
1) Training Details: To train our network, we build a setof image pairs sampled from the Kitti raw dataset; when
not specifically mentioned otherwise, no fine-tuning has been
performed on the individual datasets. We train the network
using the Adam optimizer [18] with default parameters except
for the learning rate (10−4 in our setting) and β1 that was set
to 0.5. Mini-batch size was set to 16, and training was stopped
after 250 epochs, each of which was composed of 1000 minibatches. The loss function of Eq. 5 is employed during thetraining by weighting the reconstruction errors of H-net andF-net as follows:
Lch = Lch ( X̃ tH+1 , X t +1 ) × α + Lch ( X̃ tF+1 , X t +1 ) × β(6)where α and β are set to 0.5 and 1 respectively. Concerning
the baseline proposed in Tab. II, the F-net has been trainedusing the same setup just described for our full architecture,but substituting the H-net with a traditional computation of
the homography matrix based on SIFT+RANSAC.
In order to show the effectiveness of the proposed loss
function in the context of optical flow estimation, we reportin Fig. 5 the APE and reconstruction loss during training. For
the purpose of this particular experiment, the Virtual Kitti
dataset has been employed since it provides both enoughdata to train the network and dense ground-truth flow fields
to compute the APE. Notably, the two metrics follow thesame trend and minimizing the reconstruction error of Eq. 6
effectively decreases the flow error. All the reported times forour method are computed using a NVIDIA GTX 1080 Ti GPU
for forward propagation.

This article has been accepted for inclusion in a future issue of this journal. Content is final as presented, with the exception of pagination.
6

IEEE TRANSACTIONS ON INTELLIGENT TRANSPORTATION SYSTEMS

Fig. 6.

Some estimated optical flow fields on the Kitti 2012 dataset.
TABLE IIP ERFORMANCE C OMPARISON ON THE K ITTI F LOW 2012 D ATASET. N OTE T HAT N OT A LL THE M ETHODS R EPORT R ESULTS ON THE P UBLIC
L EADERBOARDS (T ESTING S ET ). T HE TABLE I S D IVIDED IN T HREE S ECTIONS : H AND -C RAFTED AND S UPERVISED M ETHODS ,U NSUPERVISED /S ELF -S UPERVISED , O UR P ROPOSAL . (ft) I NDICATES THE M ODEL F INE -T UNED ON K ITTI F LOW

B. Evaluation: Kitti 2012
Following standard optical flow evaluation benchmarks,
to evaluate the performance of our method we adopt thefollowing metrics: Accuracy@5, meaning the ratio of motion
vectors with end point error lower than 5 pixels, and APE
which is the average point error of all motion vectors. Notethat [21] and [35] only provide Accuracy@5 results instead of
Accuracy@3 and do not have entries on the public benchmark,hence the use of Accuracy@5 as accuracy measure.
Tab. I reports the performance of the different componentsof our architecture: First, the H-net and F-net were trained
independently optimizing Eq. 5, then the whole network wastrained end-to-end by minimizing Eq. 6 (Joint entry in thetable). It can be noticed how the performance of F-net ispoor compared to both H-net and joint training: this is not
surprising, since F-net shares similarities with the FlowNetS
architecture [9], which requires large amounts of ground truth
information to achieve good performance. On the other hand,due to the predominant component of homographic flow in
automotive scenes, H-net in its simplicity can achieve performance comparable to unsupervised methods such as [21]and [36] and even outperform the original FlowNetS model(Tab. II) in as little as 12 ms for a forward propagation. Finally,
the end-to-end training of our full architecture clearly shows
an improvement over both of its two components, validating
the idea of applying a geometrical bootstrap to optical flow

estimation. It is also worth noticing that the computational
impact of the H-net is negligible when comparing the forward
prop times of F-net and joint architecture. Some qualitative
results illustrating the behavior of both network modules arereported in Fig. 6.
We then evaluate our proposal against three recent
self-supervised deep-learning based approaches [20], [21],[35]. We also report the results of recent methods tackling optical flow estimation using both hand-crafted andlearned features [9], [23], [30]. The performance of a baseline where the homography matrix between the two frames
is computed using traditional computer vision techniques(SIFT+RANSAC) instead of using the H-net is also evaluated, aiming at showing the actual contribution of the H-net.
Finally, to show the effectiveness of the projective bootstrap
strategy we test the flow performance of H-net and F-net
separately, and then their performance when jointly trained.
Tab. II reports the results of this evaluation. In particular,
our proposal performs favorably against recent self-supervised
approaches. It shows competitive performance and while notreaching the average point error results of DeepFlow [30]or EpicFlow [23], it is three orders of magnitude faster,requiring 0.05 sec/pair compared to the 17 sec/pair and 15
sec/pair of DeepFlow and EpicFlow respectively, which isan important factor for an automotive method. Furthermore,
we compare our results to more recent deep-learning based
methods who can achieve comparable running-times to ours,

This article has been accepted for inclusion in a future issue of this journal. Content is final as presented, with the exception of pagination.
ALLETTO et al.: SELF-SUPERVISED OPTICAL FLOW ESTIMATION BY PROJECTIVE BOOTSTRAP

7

TABLE IIITABLE IVP ERFORMANCE C OMPARISON ON THE V IRTUAL K ITTI D ATASETP ERFORMANCE C OMPARISON ON THE DR( EYE )VE D ATASET (H IGHERI S B ETTER ). T HE E VALUATION I S P ERFORMED ON Downtown
S EQUENCES OF THE D ATASET

namely Flownet 2.0 [15] and SpyNet [22]. Tab. II shows that
our method outperforms both of these recent approaches when
no fine-tuning is performed. On the other hand, both Flownet
2.0 and SpyNet considerably improve their performance whenfine-tuning on Kitti training set. This result supports our claimthat our method, thanks to its self-supervised nature, should
be used when ground-truth flow maps are unavailable.
Concerning the baseline performance, it can be seen how themethod, while achieving acceptable results, performs worse
than bootstrapping with the transformation learned by the
H-net. Indeed, while the transformation learned by the H-net
is inspired by this process, the experiment shows that learning
the transformation instead of relying on a purely geometricalhomography estimate result in an approximation that is better
suited to bootstrap the subsequent flow refinement. A second
advantage of our method over this baseline is that the entire
process is end to end and only requires a forward propagation,
thus eliminating the burdensome extraction and matching of
SIFT keypoints, which results in our approach being more than
5 time faster than the baseline.

our fine-tuning does not require optical flow ground truth, andis hence always possible). We obtain the best APE resultscompared to the other methods while still being 2.5× fasterthan the fastest competitor. Indeed, the self-supervised natureof our proposal does not require optical flow ground-truth
information, making fine-tuning suitable for every dataset.
Fig. 7 displays a qualitative evaluation of the results on the
Virtual Kitti dataset.

C. Evaluation: Virtual Kitti
To further evaluate our model, we assess its performanceon the Virtual Kitti dataset. While still featuring the typical
automotive perspective, Virtual Kitti presents some noteworthy
differences from the other datasets, the most significant being
the presence of the typical artifacts of computer renderedscenes. Nonetheless, it is currently the biggest dataset providing dense ground truth optical flow for automotive, obtainedusing the computer graphics warping of the scene and ishence guaranteed to be exact, as opposed to LIDAR based
ground truth measures. Due to the dataset being very recent,no results are publicly available and we evaluate DeepFlow,
EpicFlow, FlowNet 2.0, SpyNet and MRFlow [31] relying on
the provided source codes and pretrained models. In all theexperiments, the methods have been used with their default
parameters, and no fine-tuning has been performed on the
Virtual Kitti ground-truth fields. For [31], we used the publicly
available code with the rigidity maps computed using semantic
segmentation ground truth and default flow initializer. Tab. III
provides the results of this evaluation. It can be noticed how
the performance of all the methods, except for SpyNet, falls
into a similar range. Nonetheless, EpicFlow and our proposal
are shown to outperform the other competitors. The failureof SpyNet is this scenario is likely tied to the synthetic
nature of the dataset, requiring either fine-tuning or a thoroughhyper-parameter search. Finally, the last line of Tab. III reports
the performance of our model after a fine-tuning step (note that

D. Evaluation: DR(Eye)VE
To evaluate the proposed approach on the DR(eye)VE
dataset, which lacks ground truth flow information, we rely on
our initial assumption that the optical flow can be computed as
the transformation warping two consecutive frames. It is hence
possible to approximately estimate the optical flow quality by
estimating the reconstruction itself. That is, given a frame X t
and the optical flow transformation Tφ , we use the spatialtransformer layer to get the reconstructed estimate X̃ tF+1 andmeasure its peak signal-to-noise ratio (PSNR) according to
the protocol in [20]. Due to PSNR sensibility to small changes
resulting, for example, from small flow intensity errors shifting
objects by a few pixels but keeping the overall scene intact,
we also evaluate the quality of X̃ tF+1 using the Structural
Similarity Index Measure (SSIM), which not only considers
individual pixel intensities but also accounts for the structuralsimilarity of small patches. Tab. IV reports the results ofsuch evaluation where no fine-tuning has been performed. To
analyze the impact of different environmental conditions on
optical flow estimation, Tab. IV also reports results divided bysequence type. The slightly higher PSNR across all methods
during the Night sequences is due to the lower intensity of
the images, effectively resulting in a lower error. Beside that,the experiment shows that environmental conditions do nothave a significant impact on optical flow estimation. SSIM,
while confirming the same overall trend, shows a smaller
gap between methods. As mentioned earlier, this is due to
SSIM being a more stable metric where the coherence of
pixel neighborhoods are also taken into account. Note thatusing the spatial transformer layer as warper ensures thatthe reconstruction pipeline is the same for every method anddifferences in PSNR are only due to the flow.
E. Sintel: a Non-Homographic Flow Case Study
To assess the generalization capability of our model tonon-automotive contexts, we design a case study where
our network is tested on a dataset featuring strong
non-homographic components in its motion patterns: the MPI
Sintel flow dataset [6]. It presents highly dynamic scenes

This article has been accepted for inclusion in a future issue of this journal. Content is final as presented, with the exception of pagination.
8

Fig. 7.

IEEE TRANSACTIONS ON INTELLIGENT TRANSPORTATION SYSTEMS

Examples of flow fields estimated by the proposed model on the Virtual Kitti dataset.
TABLE VP ER -S EQUENCE APE ON S INTEL T RAINING S ET. N OTE T HAT All R EPORTS THE P ER -F RAME APE, AND D IFFERS
F ROM THE M EAN OF O THER C OLUMNS D UE TO D IFFERENT S EQUENCE L ENGTHS

Fig. 8. Examples our predicted flow in two different sequences. In each triplet, from left to right: blend between frame t and t + 1, ground truth, prediction.
First triplet: bamboo, ape 3.93; second triplet: cave, image pair ape: 19.75.

often dominated by large non-rigid objects in the foregroundwith very articulated motion, e.g. the protagonist fighting anassailant or a dragon. Tab. V reports the performance of our
method on the Sintel training set2 (Final pass) where nofine-tuning has been performed (i.e. with the model trainedon the Kitti raw sequences). To provide a better picture of
the results, the table reports the APE for each sequence.
Notably, the proposed model is able to generalize from theautomotive scenario to several Sintel sequences. Nonetheless, some sequences (such as ambush, cave, market) exhibit
high average point errors (APE≥10); this is not surprising,as these sequences feature the largest displacements and astrong component of non-homographic flow. Fig. 8 depicts
two examples extracted from two different sequences: The first
sequence depicts a scene where the motion is dominated by
the camera motion and the network correctly identifies theflow between the input frames. In the other sequence it canbe noticed how the input pair contains severe motion blurand two large non-rigid foreground objects, preventing thecorrect projective bootstrap of the flow and hence resulting in a
failure.
V. C ONCLUSIONSIn this paper, we showed a self-supervised approach to
optical flow estimation that jointly accounts for the geometriccues in a scene, and for the pixel-level motion patterns ofdifferent objects. We address the complexity of self-supervised
training by first estimating the global motion of the carusing a lightweight network that bootstraps a more complex,
2 Note that the submission system does not provide per-sequence information, hence the use of the training sequences

pixel-level transformation. Our experimental evaluation shows
how the proposed approach outperforms recent self-supervised
methods while maintaining the advantages in terms of simplicity and speed of an end-to-end forward only neural network.
In fact, we strongly believe that, especially in motion flow estimation, obtaining ground truth information can be prohibitiveand self-supervision will in turn become a key component of
any computer vision pipeline relying on flow information.
R EFERENCES[1] S. Alletto, A. Palazzi, F. Solera, S. Calderara, and R. Cucchiara,“DR(eye)VE: A dataset for attention-based tasks with applications toautonomous and assisted driving,” in Proc. IEEE Int. Conf. Comput.
Vis. Pattern Recognit. Workshops (CVPRW), Jun./Jul. 2016, pp. 54–60.
[2] M. Bai, W. Luo, K. Kundu, and R. Urtasun, “Exploiting semantic
information and deep matching for optical flow,” in Proc. Eur. Conf.
Comput. Vis. Cham, Switzerland: Springer, 2016, pp. 154–170.
[3] C. Bailer, K. Varanasi, and D. Stricker. (Jul. 2016). “CNN-based patch
matching for optical flow with thresholded hinge embedding loss.”[Online]. Available: https://arxiv.org/abs/1607.08064[4] G. J. Brostow, J. Fauqueur, and R. Cipolla, “Semantic object classes invideo: A high-definition ground truth database,” Pattern Recognit. Lett.,
vol. 30, no. 2, pp. 88–97, 2008.
[5] T. Brox, A. Bruhn, N. Papenberg, and J. Weickert, “High accuracy
optical flow estimation based on a theory for warping,” in Proc. Eur.
Conf. Comput. Vis. Berlin, Germany: Springer, 2004, pp. 25–36.
[6] D. J. Butler et al., “A naturalistic open source movie for opticalflow evaluation,” in Proc. Eur. Conf. Comput. Vis. (ECCV), in Lecture
Notes in Computer Science, A. Fitzgibbon, Eds. Berlin, Germany:
Springer-Verlag, Oct. 2012, pp. 611–625.
[7] Z. Chen, H. Jin, Z. Lin, S. Cohen, and Y. Wu, “Large displacement
optical flow from nearest neighbor fields,” in Proc. IEEE Conf. Comput.
Vis. Pattern Recognit., Jun. 2013, pp. 2443–2450.
[8] M. Cordts et al., “The cityscapes dataset for semantic urban
scene understanding,” in Proc. IEEE Conf. Comput. Vis. Pattern
Recognit. (CVPR), Jun. 2016, pp. 3213–3223.
[9] P. Fischer et al. (Apr. 2015). “FlowNet: Learning optical flow withconvolutional networks.” [Online]. Available: https://arxiv.org/abs/1504.
06852

This article has been accepted for inclusion in a future issue of this journal. Content is final as presented, with the exception of pagination.
ALLETTO et al.: SELF-SUPERVISED OPTICAL FLOW ESTIMATION BY PROJECTIVE BOOTSTRAP[10] A. Gaidon, Q. Wang, Y. Cabon, and E. Vig, “Virtual worlds as proxy for
multi-object tracking analysis,” in Proc. CVPR, 2016, pp. 4340–4349.
[11] A. Geiger, P. Lenz, and R. Urtasun, “Are we ready for autonomous
driving? The KITTI vision benchmark suite,” in Proc. Conf. Comput.
Vis. Pattern Recognit. (CVPR), 2012, pp. 3354–3361.
[12] R. Hartley and A. Zisserman, Multiple View Geometry in Computer
Vision. Cambridge, U.K.: Cambridge Univ. Press, 2003.
[13] B. K. P. Horn and B. G. Schunck, “Determining optical flow,” Artif.
Intell., vol. 17, nos. 1–3, pp. 185–203, Aug. 1981.
[14] J. Hur and S. Roth, “MirrorFlow: Exploiting symmetries in joint opticalflow and occlusion estimation,” in Proc. Int. Conf. Comput. Vis. (ICCV),
2017, pp. 312–321.
[15] E. Ilg, N. Mayer, T. Saikia, M. Keuper, A. Dosovitskiy, and T. Brox.
(Dec. 2016). “FlowNet 2.0: Evolution of optical flow estimation withdeep networks.” [Online]. Available: https://arxiv.org/abs/1612.01925[16] M. Irani and P. Anandan, “Parallax geometry of pairs of points for 3D
scene analysis,” in Computer Vision—ECCV. Berlin, Germany: Springer,
1996, pp. 17–30.
[17] M. Jaderberg, K. Simonyan, A. Zisserman, and K. Kavukcuoglu, “Spatial transformer networks,” in Proc. Adv. Neural Inf. Process. Syst., 2015,
pp. 2017–2025.
[18] D. Kingma and J. Ba. (Dec. 2014). “Adam: A method for stochastic
optimization.” [Online]. Available: https://arxiv.org/abs/1412.6980[19] R. Kumar, P. Anandan, and K. Hanna, “Direct recovery of shape frommultiple views: A parallax based approach,” in Proc. 12th IAPR Int.
Conf. Pattern Recognit., Conf., Comput. Vis. Image Process., vol. 1.
Oct. 1994, pp. 685–688.
[20] Z. Liu, R. A. Yeh, X. Tang, Y. Liu, and A. Agarwala, “Video
frame synthesis using deep voxel flow,” in Proc. Int. Conf. Comput.
Vis. (ICCV), 2017, pp 4463–4471.
[21] G. Long, L. Kneip, J. M. Alvarez, H. Li, X. Zhang, and Q. Yu, “Learning
image matching by simply watching video,” in Proc. Eur. Conf. Comput.
Vis. Cham, Switzerland: Springer, 2016, pp. 434–450.
[22] A. Ranjan and M. J. Black, “Optical flow estimation using a spatial pyramid network,” in Proc. IEEE Conf. Comput. Vis. Pattern
Recognit. (CVPR), vol. 2. Jul. 2017, pp. 4161–4170.
[23] J. Revaud, P. Weinzaepfel, Z. Harchaoui, and C. Schmid, “EpicFlow:
Edge-preserving interpolation of correspondences for optical flow,”in Proc. IEEE Conf. Comput. Vis. Pattern Recognit., Jun. 2015,
pp. 1164–1172.
[24] S. R. Richter, Z. Hayder, and V. Koltun, “Playing for benchmarks,” inProc. Int. Conf. Comput. Vis. (ICCV), 2017, pp. 2213–2222.
[25] H. S. Sawhney, “3D geometry from planar parallax,” in Proc. CVPR,
vol. 94. 1994, pp. 929–934.
[26] T. Schuster, L. Wolf, and D. Gadot. (Nov. 2016). “Optical flow
requires multiple strategies (but only one network).” [Online]. Available:
https://arxiv.org/abs/1611.05607[27] A. Shashua and N. Navab, “Relative affine structure: Theory andapplication to 3D reconstruction from perspective views,” in Proc.
CVPR, vol. 94. 1994, pp. 483–489.
[28] D. Tran, L. Bourdev, R. Fergus, L. Torresani, and M. Paluri, “Deepend2end voxel2voxel prediction,” in Proc. IEEE Conf. Comput. Vis.
Pattern Recognit. Workshops, Jun. 2016, pp. 17–24.
[29] A. S. Wannenwetsch, M. Keuper, and S. Roth, “ProbFlow: Joint
optical flow and uncertainty estimation,” in Proc. Int. Conf. Comput.
Vis. (ICCV), vol. 3. Oct. 2017, pp. 1173–1182.
[30] P. Weinzaepfel, J. Revaud, Z. Harchaoui, and C. Schmid, “DeepFlow:
Large displacement optical flow with deep matching,” in Proc. IEEE
Int. Conf. Comput. Vis. (ICCV), Sydney, NSW, Australia, Dec. 2013,
pp. 1385–1392.
[31] J. Wulff, L. Sevilla-Lara, and M. J. Black, “Optical flow in mostly rigid
scenes,” in Proc. IEEE Conf. Comput. Vis. Pattern Recognit. (CVPR),
Jul. 2017, pp. 4671–4680.
[32] J. Xu, R. Ranftl, and V. Koltun. (Apr. 2017). “Accurate optical flow via
direct cost volume processing.” [Online]. Available: https://arxiv.org/abs/
1704.07325[33] K. Yamaguchi, D. McAllester, and R. Urtasun, “Efficient joint segmentation, occlusion labeling, stereo and flow estimation,” in Proc. Eur. Conf.
Comput. Vis. Cham, Switzerland: Springer, 2014, pp. 756–771.
[34] Y. Yang and S. Soatto, “S2F: Slow-to-fast interpolator flow,” in Proc.
IEEE Conf. Comput. Vis. Pattern Recognit., Jul. 2017, pp. 2087–2096.
[35] J. J. Yu, A. W. Harley, and K. G. Derpanis. (Aug. 2016). “Back tobasics: Unsupervised learning of optical flow via brightness constancy
and motion smoothness.” [Online]. Available: https://arxiv.org/abs/1608.
05842

9

Stefano Alletto received the master’s degree incomputer engineering and the Ph.D. degree from the
University of Modena and Reggio Emilia in 2014
and 2018, respectively, where he is currently a
Research Contractor. His main research interests
include computer vision and machine learning
applied to first person camera views, automotive andenhanced cultural experiences.

Davide Abati received the master’s degree in computer engineering from the University of Modena
and Reggio Emilia in 2015. He is currently pursuing the Ph.D. degree with the AImageLab Group,
Modena, with a focus on computer vision and deeplearning for image and video understanding.

Simone Calderara (S’04–M’05) received the master’s degree in computer engineering and the Ph.D.
degree from the University of Modena and Reggio
Emilia, in 2005 and 2009 respectively and, where
he is currently an Assistant Professor with the
AImagelab Group. His current research interests
include computer vision and machine learning
applied to human behavior analysis, visual tracking
in crowded scenarios, and time series analysis forforensic applications.

Rita Cucchiara received the master’s degree inelectronic engineering and the Ph.D. degree in computer engineering from the University of Bologna,
Italy, in 1989 and 1992, respectively. Since 2005,
she has been a Full Professor at the University of
Modena and Reggio Emilia, Italy, where she is also
the Head of the ImageLab Group and is also the
Director of the SOFTECH-ICT Research Center. She
is currently the President of the Italian Associationof Pattern Recognition, affiliated with IAPR. She
has authored over 300 papers on pattern recognition
computer vision and multimedia, and in particular in human analysis, HBU,
and egocentric-vision. The research carried out spans different application
fields, such as video-surveillance, automotive and multimedia big data annotation. She is currently an Associate Editor for the IEEE T RANSACTIONSON M ULTIMEDIA and serves on the Governing Board of IAPR and on theAdvisory Board of the CVF.
Luca Rigazio received the B.S. degree in electricalengineering from the Politecnico di Torino and the
M.S. degree in computer engineering from Institut
Eurecom, with a focus on signal processing andmachine learning. He started his career at the Panasonic Speech Technology Laboratory, Santa Barbara,
CA, USA, where he was involved in low complexity
speech recognition models and real-time decoders.
He then moved to the Panasonic Silicon Valley
Laboratory, the company’s premiere research center,
Bay Area, CA, USA, where he started the deeplearning groups and the robotics groups, with a focus on computer visionand end-to-end robotics control, with a direct application to the company’s
autonomous drive solutions. He has over 40 patents and over 35 publicationsin the areas of machine learning, deep learning, signal processing, andcontrol for domains, such as computer vision, speech processing, digital signal
processing, and robots control.