arXiv:1801.05746v1 [] 17 Jan 2018

TernausNet: U-Net with VGG11 Encoder
Pre-Trained on ImageNet for Image Segmentation
Vladimir Iglovikov

Alexey Shvets

Lyft Inc.
San Francisco, CA 94107, USA
Email: iglovikov@gmail.com

Massachusetts Institute of Technology
Cambridge, MA 02142, USA
Email: shvets@mit.edu

Abstract—Pixel-wise image segmentation is demanding task in
computer vision. Classical U-Net architectures composed of encoders and decoders are very popular for segmentation of medical
images, satellite images etc. Typically, neural network initialized
with weights from a network pre-trained on a large data set
like ImageNet shows better performance than those trained from
scratch on a small dataset. In some practical applications, particularly in medicine and traffic safety, the accuracy of the models
is of utmost importance. In this paper, we demonstrate how the
U-Net type architecture can be improved by the use of the pretrained encoder. Our code and corresponding pre-trained weights
are publicly available at https://github.com/ternaus/TernausNet.
We compare three weight initialization schemes: LeCun uniform,
the encoder with weights from VGG11 and full network trained
on the Carvana dataset. This network architecture was a part
of the winning solution (1st out of 735) in the Kaggle: Carvana
Image Masking Challenge.
Keywords—Computer Vision, Image Segmentation, Image
Recognition, Deep learning, Medical Image Processing, Satellite
Imagery.

I.

I NTRODUCTION

Recent progress in computer hardware with the democratization to perform intensive calculations has enabled researchers to work with models, that have millions of free
parameters. Convolutional neural networks (CNN) have already demonstrated their success in image classification, object
detection, scene understanding etc. For almost any computer
vision problems, CNN-based approaches outperform other
techniques and in many cases even human experts in the corresponding field. Now almost all computer vision application
try to involve deep learning techniques to improve traditional
approaches. They influence our everyday lives and the potential
uses of these technologies look truly impressive.
Reliable image segmentation is one of the important tasks
in computer vision. This problem is especially important for
medical imaging that can potentially improve our diagnostic
abilities and in scene understanding to make safe self-driving
vehicles. Dense image segmentation essentially involves dividing images into meaningful regions, which can be viewed
as a pixel level classification task. The most straightforward
(and slow) approach to such problem is manual segmentation
of the images. However, this is a time-consuming process that
is prone to mistakes and inconsistencies that are unavoidable
when human data curators are involved. Automating the treatment provides a systematic way of segmenting an image on
the fly as soon as the image is acquired. This process requires

providing necessary accuracy to be useful in the production
environment.
In the last years, different methods have been proposed
to tackle the problem of creating CNN’s that can produce
a segmentation map for an entire input image in a single
forward pass. One of the most successful state-of-the-art deep
learning method is based on the Fully Convolutional Networks
(FCN) [2]. The main idea of this approach is to use CNN as
a powerful feature extractor by replacing the fully connected
layers by convolution one to output spatial feature maps instead
of classification scores. Those maps are further upsampled to
produce dense pixel-wise output. This method allows training
CNN in the end to end manner for segmentation with input
images of arbitrary sizes. Moreover, this approach achieved an
improvement in segmentation accuracy over common methods
on standard datasets like PASCAL VOC [3]. This method
has been further improved and now known as U-Net neural
network [4]. The U-Net architecture uses skip connections to
combine low-level feature maps with higher-level ones, which
enables precise pixel-level localization. A large number of
feature channels in upsampling part allows propagating context
information to higher resolution layers. This type of network
architecture proven themselves in binary image segmentation
competitions such as satellite image analysis [5] and medical
image analysis [6], [7] and other [9].
In this paper, we show how the performance of U-Net
can be easily improved by using pre-trained weights. As an
example, we show the application of such approach to Aerial
Image Labeling Dataset [8], that contains aerospace images of
several cities with high resolution. Each pixel of the images
is labeled as belonging to either ”building” or ”not-building”
classes. Another example of the successful application of such
an architecture and initialization scheme is Kaggle Carvana
image segmentation competition [9], where one of the authors
used it as a part of the winning (1st out 735 teams) solution.
II.

N ETWORK A RCHITECTURE

In general, a U-Net architecture consists of a contracting
path to capture context and of a symmetrically expanding
path that enables precise localization (see for example Fig.
1). The contracting path follows the typical architecture of
a convolutional network with alternating convolution and
pooling operations and progressively downsamples feature
maps, increasing the number of feature maps per layer at the
same time. Every step in the expansive path consists of an
upsampling of the feature map followed by a convolution.

copy and concatanate

copy and concatanate
copy and concatanate
copy and concatanate

sigmoid output
256

512

512

256

512

512

256

512

512

512

512

512

512

512

128
256

256

256

3x3 Conv2d+ReLU (pre-trained)
3x3 Conv2d+ReLU

64
128
128

64
128

128

256

input image

copy and concatanate

2x2 MaxPool

32
64
1

3
64

3x3 ConvTranspose2d(stride=2)+ReLU

Fig. 1. Encoder-decoder neural network architecture also known as U-Net where VGG11 neural network without fully connected layers as its encoder. Each
blue rectangular block represents a multi-channel features map passing through a series of transformations. The height of the rod shows a relative map size (in
pixels), while their widths are proportional to the number of channels (the number is explicitly subscribed to the corresponding rod). The number of channels
increases stage by stage on the left part while decrease stage by stage on the right decoding part. The arrows on top show transfer of information from each
encoding layer and concatenating it to a corresponding decoding layer.

Hence, the expansive branch increases the resolution of the
output. In order to localize, upsampled features, the expansive
path combines them with high-resolution features from the
contracting path via skip-connections [4]. The output of the
model is a pixel-by-pixel mask that shows the class of each
pixel. This architecture proved itself very useful for segmentation problems with limited amounts of data, e.g. see [5].
U-Net is capable of learning from a relatively small training
set. In most cases, data sets for image segmentation consist of
at most thousands of images, since manual preparation of the
masks is a very costly procedure. Typically U-Net is trained
from scratch starting with randomly initialized weights. It is
well known that training network without over-fitting the data
set should be relatively large, millions of images. Networks
that are trained on the Imagenet [10] data set are widely used
as a source of the initialization for network weights in other
tasks. In this way, the learning procedure can be done for nonpre-trained several layers of the network (sometimes only for
the last layer) to take into account features of the date set.
As an encoder in our U-Net network, we used relatively
simple CNN of the VGG family [11] that consists of 11
sequential layers and known as VGG11 see Fig. 2. VGG11
contains seven convolutional layers, each followed by a ReLU

activation function, and five max polling operations, each
reducing feature map by 2. All convolutional layers have 3 × 3
kernels and the number of channels is given in Fig. 2. The
first convolutional layer produces 64 channels and then, as the
network deepens, the number of channels doubles after each
max pooling operation until it reaches 512. On the following
layers, the number of channels does not change.
To construct an encoder, we remove the fully connected
layers and replace them with a single convolutional layer of
512 channels that serves as a bottleneck central part of the
network, separating encoder from the decoder. To construct the
decoder we use transposed convolutions layers that doubles the
size of a feature map while reducing the number of channels
by half. And the output of a transposed convolution is then
concatenated with an output of the corresponding part of the
decoder. The resultant feature map is treated by convolution
operation to keep the number of channels the same as in
a symmetric encoder term. This upsampling procedure is
repeated 5 times to pair up with 5 max poolings, as shown
in Fig. 1. Technically fully connected layers can take an input
of any size, but because we have 5 max-pooling layers, each
downsampling an image two times, only images with a side
divisible by 32 (25 ) can be used as an input to the current

3x3 conv, 64
2x2 MaxPool

3x3 conv, 128
2x2 MaxPool

3x3 conv, 256
3x3 conv, 256
2x2 MaxPool

3x3 conv, 512
3x3 conv, 512
Fig. 3. Jaccard index as a function of a training epoch for three U-Net
models with different weight initialization. The blue line shows a model with
randomly initialized weights, orange line shows a model, where the encoder
was initialized with VGG11 network pre-trained on ImageNet. Green line
shows a model, where the entire network was pre-trained on Carvana data set.

2x2 MaxPool

3x3 conv, 512
3x3 conv, 512
2x2 MaxPool

Every image is consists of pixels. To adapt the last expression
for discrete objects, we can write it in the following way

n 
1X
yi ŷi
(2)
J=
n i=1 yi + ŷi − yi ŷi

fcc, 4096
fcc, 4096
fcc, num. classes
Fig. 2. VGG11 network architecture. In this picture each convolutional layer
is followed by ReLU activation function. The number in each box represents
the number of channels in the corresponding feature map.

network implementation.
III.

Since, we can consider image segmentation task as a pixel
classification problem, we also use the common loss function
for binary classification tasks - binary cross entropy that is
defined as:

R ESULTS

n

We applied our model to Inria Aerial Image Labeling
Dataset [8]. This dataset consists of 180 aerial images of urban
settlements in Europe and United States, and is labeled as a
building and not building classes. Every image in the data set
is RGB and has 5000×5000 pixels resolution where each pixel
corresponds to a 30 × 30 cm2 of Earth surface. We used 30
images (5 from every 6 cities in the train set) for validation,
as suggested in [12] (valid. IoU ' 0.647) and [13] (best valid.
IoU ' 0.73) and trained the network on the remaining 150
images for 100 epochs. Random crops of 768 × 768 were used
for training and central crops 1440×1440 for validation. Adam
with learning rate 0.001 as an optimization algorithm [14].
We choose Jaccard index (Intersection Over Union) as
evaluation metric. It can be interpreted as similarity measure
between a finite number of sets. Intersection over union for
similarity measure between two sets A and B, can be defined
as following:
J(A, B) =

where yi is a binary value (label) of the corresponding pixel
i and ŷi is predicted probability for the pixel.

|A ∩ B|
|A ∩ B|
=
|A ∪ B|
|A| + |B| − |A ∩ B|

where normalization condition takes place:
0 ≤ J(A, B) ≤ 1

(1)

H=−

1X
(yi log ŷi + (1 − yi ) log(1 − ŷi ))
n i=1

(3)

Join these expressions, we can generalized the loss function,
namely,
L = H − log J
(4)
Therefore, minimizing this loss function, we simultaneously maximize probabilities for right pixels to be predicted
and maximize the intersection, J between masks and corresponding predictions. For more details, see [5].
At the output of a given neural network, we obtain an
image where each pixel corresponds to a probability to detect
interested area. The size of the output image is coincides with
the input image. In order to have only binary pixel values,
we choose a threshold 0.3. This number can be found using
validation data set and it is pretty universal for our generalized
loss function and many different image data sets. For different
loss function this number is different and should be found
independently. All pixel values below the specified threshold,
we set to be zero while all values above the threshold, we set
to be 1. Then, multiplying by 255 every pixel in an output
image, we can get a black and white predicted mask

A) ground true

C) pre-trained: VGG11

B) no pre-trained

D) pre-trained: Carvana

Fig. 4. Binary masks with green pixels indicate class membership (buildings). Image A) shows an original image with the superimposed ground true mask;
Images B) to D) show predictions, initialized with different schemas and trained for 100 epochs. Network in image B) had randomly initialized weights. The
model in image C) used randomly initialized decored weights and encoder weights initialized with VGG11, pre-trained on ImageNet. The model in image D)
used weights, pre-trained on Carvana data set.

In our experiment, we test 3 U-Nets with the same architecture as shown in Fig. 1 differing only in the way of
weights initialization. For the basic model we use network
with weights initialized by LeCun uniform initializer. In this
initializer samples drawpfrom a uniform distribution within
[−L, L], where L =
1/fin and fin is the number of
input units in the weight tensor. This method is implement
in pytorch [15] as a default method of weight initialization
in convolutional layers. Next, we utilize the same architecture
with VGG11 encoder pre-trained on ImageNet while all layers
in decoder are initialized by the LeCun uniform initializer.
Then, as a final example, we use network with weights pretrained on Carvana dataset [9] (both encoder and decoder).
Therefore, after 100 epochs, we obtain the following results
for validation subset:
1) LeCun uniform initializer: IoU = 0.593
2) The Encoder is pre-trained on ImageNet: IoU = 0.686
3) Fully pre-trained U-Net on Carvana: IoU = 0.687
Validation learning curves in Fig. 3 show benefits of our approach. First of all, pre-trained models converge much faster to
its steady value in comparison to the non-pre-trained network.
Moreover, the steady-state value seems higher for the pretrained models. Ground truth, as well as three masks, predicted

by these three models, are superimposed on an original image
in Fig. 4. One can easily notice the difference in the prediction
quality after 100 epochs. Moreover, validation learning curves
in Our results for the Inria Aerial Image Labeling Dataset can
be easily further improved using different hyper-parameters
optimization techniques or standard computer vision methods
applying them during pre- and post-processing.
IV.

C ONCLUSION

In this paper, we show how the performance of U-Net
can be improved using technique knows as fine-tuning to
initialize weights for an encoder of the network. This kind
of neural network is widely used for image segmentation
tasks and shows state of the art results in many binary image
segmentation, competitions. Fine-tuning is already widely used
for image classification tasks, but to our knowledge is not with
U-Net type family architectures. For the problems of image
segmentation, the fine-tuning should be considered even more
natural because it is problematic to collect a large volume of
training dataset (in particular for medical images) and qualitatively label it. Furthermore, pre-trained networks substantially
reduce training time that also helps to prevent over-fitting. Our
approach can be further improved considering more advanced
pre-trained encoders such as VGG16 [11] or any pre-trained

network from ResNet family [16]. With this improved encoders
the decoders can be kept as simple as we use. Our code is
available as an open source project under MIT license and can
be found at https://github.com/ternaus/TernausNet.
ACKNOWLEDGMENT
The authors would like to thank Open Data Science community [17] for many valuable discussions and educational
help in the growing field of machine/deep learning. The authors
also express their sincere gratitude to Alexander Buslaev who
originally suggested to use a pre-trained VGG network as an
encoder in a U-Net network.
R EFERENCES
[1]
[2]

[3]

[4]
[5]

[6]

[7]
[8]
[9]
[10]

[11]
[12]

[13]

[14]
[15]
[16]
[17]

Y. LeCun, Y. Bengio and G. Hinton, Deep learning, Nature 521, 436444,
2015.
J. Long, E. Shelhamer and T. Darrell. Fully Convolutional Networks
for Semantic Segmentation. Proceedings of the IEEE Conference on
Computer Vision and Pattern Recognition. 2015.
M. Everingham, et al. The Pascal Visual Object Classes Challenge: A
Retrospective. International Journal of Computer Vision 111, 98-136,
2015.
O. Ronneberger, P. Fischer and T. Brox, U-Net: Convolutional Networks
for Biomedical Image Segmentation, arXiv:1505.04597, 2015.
V. Iglovikov, S. Mushinskiy and V. Osin, Satellite Imagery Feature Detection using Deep Convolutional Neural Network: A Kaggle Competition,
arXiv:1706.06169, 2017.
V. Iglovikov, A. Rakhlin, A. Kalinin and A. Shvets, Pediatric
Bone Age Assessment Using Deep Convolutional Neural Networks,
arXiv:1712.05053, 2017.
T. Ching et al., Opportunities And Obstacles For Deep Learning In
Biology And Medicine, www.biorxiv.org:142760, 2017.
project.inria.fr/aerialimagelabeling/
Kaggle: Carvana Image Masking Challenge.
O. Russakovsky, J. Deng, H. Su, J. Krause, S. Satheesh, S. Ma, Z.
Huang, A. Karpathy, A. Khosla, M. Bernstein, A. Berg and Li Fei-Fei,
ImageNet Large Scale Visual Recognition Challenge, arXiv:1409.0575,
2014.
K. Simonyan and A. Zisserman, Very Deep Convolutional Networks for
Large-Scale Image Recognition, arXiv:1409.1556, 2014.
B. Bischke, P. Helber, J. Folz, D. Borth and A. Dengel, Multi-Task
Learning for Segmentation of Building Footprints with Deep Neural
Networks, arXiv:1709.05932, 2017.
E. Maggiori, Y. Tarabalka, G. Charpiat and P. Alliez, Can Semantic
Labeling Methods Generalize to Any City? The Inria Aerial Image
Labeling Benchmark, hal.inria.fr/hal-01468452, IGARSS, 2017.
D. Kingma and J. Ba, Adam: A Method for Stochastic Optimization,
arXiv:1412.6980, 2014.
pytorch.org
K. He, X. Zhang, Sh. Ren and J. Sun, Deep Residual Learning for
Image Recognition, arXiv:1512.03385, 2015.
ods.ai

