EUROGRAPHICS 2016 / J. Jorge and M. Lin(Guest Editors)

Volume 35 (2016), Number 2

DeepProp: Extracting Deep Featuresfrom a Single Image for Edit Propagation
Yuki Endo1

Satoshi Iizuka2

1 University of Tsukuba

Yoshihiro Kanamori1

Jun Mitani1

2 Waseda University / JST CREST

Abstract
Edit propagation is a technique that can propagate various image edits (e.g., colorization and recoloring) performed via user
strokes to the entire image based on similarity of image features. In most previous work, users must manually determine theimportance of each image feature (e.g., color, coordinates, and textures) in accordance with their needs and target images.
We focus on representation learning that automatically learns feature representations only from user strokes in a single image
instead of tuning existing features manually. To this end, this paper proposes an edit propagation method using a deep neuralnetwork (DNN). Our DNN, which consists of several layers such as convolutional layers and a feature combiner, extracts strokeadapted visual features and spatial features, and then adjusts the importance of them. We also develop a learning algorithm for
our DNN that does not suffer from the vanishing gradient problem, and hence avoids falling into undesirable locally optimal
solutions. We demonstrate that edit propagation with deep features, without manual feature tuning, can achieve better results
than previous work.
Categories and Subject Descriptors (according to ACM CCS): I.4.0 [Image Processing And Computer Vision]: General—

1. Introduction
Edit propagation enables us to easily edit images based on simple user input. Various applications of edit propagation exist, suchas grayscale image colorization, color image recoloring, segmentation and tone adjustment. Many efforts have been made to attack the edit propagation problem [LLW04, LWCO∗ 07, LAA08,
XLJ∗ 09,XWT∗ 09,LJH10,CZZT12,XYJ13,CZL∗ 14,YY15]. Specifying sparse image edits, users can propagate the image edits to theentire image according to the propagation principle based on pixel
similarity (e.g., proximities of positions, colors or textures).

Input image and user strokes

submitted to EUROGRAPHICS 2016.

[XYJ13]= 10)

Our DNN

We can regard edit propagation as a sort of multi-class classification problem. From image feature vector xi at pixel i, a typical edit propagation pipeline estimates multi-class probability vector yi ∈ [0, 1]n that represents how likely it is that a pixel i belongs to each of n types of strokes (class labels). A model for estimating yi is constructed from user-specified strokes and xi on the
strokes. A number of techniques have been proposed for edit propagation using various models, such as optimization-based methods [LLW04, AP08], gentle boost classifier [LAA08], function interpolation with radial basis functions (RBFs) [LJH10], manifold
learning [CZZT12], and a probabilistic model [XYJ13]. Most ofthese previous studies use a combination of multiple features withdifferent meanings as input. For example, they use concatenatedfeatures x = [σc cT , σt tT , σs sT ]T where c denotes pixel color, t tex-

[LJH10][LJH10]= 10)[XYJ13]= 5)= 5)

Figure 1: Image colorization using edit propagation. While existing methods require manual parameter tuning for each image feature, our DNN-based method automatically extracts stroke-adaptedfeatures.
ture feature, and s spatial coordinate whereas σc , σt , and σs denoteparameters that determine the importance of each feature.
In most previous work, users must heuristically select the image
features that they use and adjust parameters for the features in accordance with their needs and target images. As shown in Figure 1,
the editing results can be drastically changed depending on the parameters. For example, edits on strokes with a given intensity prop-

2Endo, Iizuka, Kanamori and Mitani / DeepProp: Extracting Deep Features from a Single Image for Edit Propagation

User feedback

Update
DNN
model

&

Estimation

Post-processing

Learning
User strokes andfeatures on

Input image I on entire region

Probability maps

ResultFigure 2: System overview. The system first learns a DNN model from an input image and user strokes. Next, stroke probabilities on all pixels
are estimated using the DNN model, and probability maps are obtained. Finally, the probability maps are refined by post-processing. Everytime the user updates strokes, the system updates the DNN model efficiently using previously learned parameters.
agate to pixels that have similar intensity when σc is large. Furthermore, using too many visual features might limit the range/diversityof input images on which the propagation succeeds. For example,
the Gabor feature, which is typically used as a texture feature, has
multiple parameters such as a kernel size and rotation angle. Using
various patterns of the parameters as well as other image featurescauses an overfitting problem of the estimated model to training
data on strokes due to the increased dimension of input vectors.
Consequently, unneeded visual features degenerate estimated results [LAA08]. On the other hand, manually selecting appropriate
visual features from many candidates is labor intensive.
Considering the above, we conducted a study to address the following research question:
Can we automatically extract effective features for edit
propagation without selecting image features manually?
To this end, we adopt deep learning [Ben09], which is a techniquefor learning deep neural network (DNN) models. Recently, thistechnique dramatically improved the accuracy of image recognitionand speech recognition [KSH12,DLH∗ 13]. While conventional approaches manually engineer features for classification, DNNs canbe used for representation learning, which automatically extracts
effective high-level features for a task from low-level features (e.g.,
image pixels and audio spectrum).
In this paper, we propose DeepProp for extracting Deep features from a single image for edit Propagation. Our method uses
low-level visual patches and spatial pixel coordinates as input of a
DNN that automatically extracts features adapted to user-specified
strokes from a single image. In contrast to most previous work, we
do not need to adjust the importance of the input features. Then,
we use the DNN as a classifier that estimates user stroke probabilities, which represent how likely it is that each pixel belongs toeach stroke, from extracted features on the entire image. Figure 1
demonstrates that our method can, without tuning parameters forimage features, generate a better result than previous work.
For edit propagation with deep features, this paper makes thefollowing contributions:• Edit propagation system using deep learning: We propose asystem design of a framework that consists of the following
three main modules (Figure 2): learning, estimation, and postprocessing. First, a DNN model is learned using user strokes

and pixels on the strokes as input. Then, using the learned DNN
model, user stroke probabilities are estimated on the entire image(on pixels or, for more efficiency, superpixels), and probability
maps are obtained. Finally, probability maps or application results (e.g., colorization) obtained from the probability maps arerefined by post-processing. Every time the user updates strokes,
the system re-learns the DNN model efficiently using previously
learned parameters.
• DNN architecture for edit propagation: We propose a DNN architecture that extracts stroke-adapted visual features and spatial
features using convolutional and fully-connected network structures. The importance of the extracted two features is also automatically determined using a feature combiner layer, and stroke
probability vectors are computed from the combined features using a soft-max layer.
• Learning algorithm for our DNN: A number of local minima
can be found during optimization of parameters of DNN models,
especially when DNNs have complicated structures with deep
layers and many neurons. Optimizing the entire network of our
DNN tends to fall into such undesirable local minima due to theerror function’s vanishing gradients. This is because our DNN
contains two types of the networks with different depths. We develop a learning algorithm that efficiently propagates the gradients to these networks and finds more desirable solutions, andcan generate better results than a naïve learning algorithm.

2. Related Work2.1. Edit propagation
The first studies related to edit propagation are colorizations using optimization [LLW04] and chrominance blending [YS06].
In addition to these approaches based on pixel colors, colorizations using texture features in manga [QWH06] and natural images [LWCO∗ 07] have been also proposed. Furthermore, we canregard tone adjustment [LFUS06], material editing [PL07], intrinsic images [BPD09] and editing of bidirectional texture functions(BTFs) [XWT∗ 09] as applications of edit propagation because they
all propagate user edits based on similarities between on-stroke pixels and the rest of the image. Such stroke-based approaches are also
used in matting [LRAL07, RRW∗ 09], edge-preserving smoothing [XLXJ11,CLKL14], and temporally consistent propagation forvideo editing [CZL∗ 14].
submitted to EUROGRAPHICS 2016.

Endo, Iizuka, Kanamori and Mitani / DeepProp: Extracting Deep Features from a Single Image for Edit Propagation

3

Visual feature extractor (VFE)
FeatureCombiner (FC)Label estimator (LE)–dim.
visual feature(c-channel patch)-dim.
featureConv + ReLU

-dim.
feature-dim.
featureMax-pooling

Fully-conn.

Conv + ReLU

Max-pooling+
256-dim. feature ReLU+

2-dim. spatial feature(x and y coordinates)
Input image (c channels)

Fully-conn.
+ ReLUsoft-max
User stroke
probability

256-dim. feature
Fully-conn.
256-dim. feature

Spatial feature extractor (SFE)

Figure 3: Our deep neural network architecture.
Other than the various applications, advances have been also
made in the fundamental algorithms for edit propagation. Li etal. proposed an approach using a gentle boost classifier by formulating edit propagation as a pixel classification problem [LAA08].
AppProp [AP08] yields better propagation by optimizing color differences for optimizing color differences not only between nearby
pixels, but also between non-neighboring ones. Computational
efficiency has been also improved by using a kd-tree [XLJ∗ 09],
continuously approximating feature space using radial basis functions (RBFs) [LJH10], manifold learning [MCY∗ 13], efficientstroke sampling [BHW11], and sparse pixel sampling [YY15].
Most of the previous approaches share a common issue, namelythat halo artifacts occur across object boundaries [LWA∗ 12]. Chenet al. [LWA∗ 12] achieved graceful color blending along object
boundaries based on locally linear embedding (LLE), which was
further improved in terms of efficiency using dictionary learning [CZL∗ 14]. Xu et al. proposed a stochastic modeling of thesimilarities between on-stroke and off-stroke pixels, based on iterative feature discrimination and sparse sampling of on-stroke pixels [XYJ13]. Also, there are approaches using specific distancemetrics such as geodesic distance [CSRP10] and diffusion distance [FFL10]. Whereas these previous approaches use various image features, effective features vary depending on input images
and user-specified strokes. Therefore, if multiple image features are
considered simultaneously, we have to tune the importance that enhances or suppresses each feature for each image. Even worse, themore image features are used, the more over-fitting occurs.

ergy functions, which hinders deep layers from sufficiently learning effective features, especially in deeper neural networks. Recent progress including the unsupervised pretraining with deep belief network (DBN) [BLP∗ 07] and the use of rectified linear unit
(ReLU) [GBB11] as well as performance improvement of computers in the last few years have enabled efficient learning using DNN.
With regards to the types of neural networks used for deeplearning, fully-connected neural network, convolutional neural network (CNN) and recurrent neural network (RNN) are commonlychosen [Ben09]. Fully-connected neural networks such as multilayer perceptron (MLP) are typical examples in DNN and havebeen applied to classification and regression problems in several research fields (e.g., speech recognition [KSH12, DLH∗ 13],
activity recognition [LGQ15], and exemplar-based photo adjustment [YZW∗ 15]). On the other hand, CNNs are often used if the inputs are images. Unlike general fully-connected networks, a CNN
is a network consisting of convolutional layers with multiple filters and pooling layers, which enables CNN to achieve invariant
features (e.g., shift invariance) under various deformations duringfeature extraction. CNNs have been applied to many image processing tasks, such as image classification [XXY∗ 15], contour detection [SWW∗ 15], motion deblurring [SCXP15], saliency detection [LY15], and depth estimation [LSL15]. We adopt a combination of a CNN and a fully-connected network for edit propagation
for the first time, and demonstrate its effectiveness through several
applications.

2.2. Deep learning

3. DNN Architecture

Recently, deep learning has gained significant attention in computer science. Deep learning is a generic term for techniques using deep neural network (DNN), and has achieved outstanding results in various fields. An advantage of DNN is that it can beused for representation learning, where low-level features (e.g.,
pixel values) are directly used as input and automatically converted to higher-level features through intermediate layers, without conventional manual design of discriminative features. A commonly occurring problem is that of vanishing gradients of the en-

The basic idea for designing a DNN is to extract high-level features
for edit propagation from low-level visual and spatial features. In
edit propagation, we can assume that the lowest-level features arecolor (or grayscale) values and coordinates. For visual features, it is
known that effective high-level features can be extracted using convolutional layers, as explained in Section 2. Instead of using single
pixels, we use color (or grayscale) patches as low-level visual features for convolution. This is because a patch enables implicitly
handling pixel gradients and capturing more various patterns thansubmitted to EUROGRAPHICS 2016.

4

Endo, Iizuka, Kanamori and Mitani / DeepProp: Extracting Deep Features from a Single Image for Edit Propagation

additionally apply convolution fconv2 with a 3 × 3 × 128 kernel andthe activation function f ReLU . Finally, a 2 × 2 max-pooling withstride 2 is used again and a 256-dimensional feature vector fv isobtained. θv denotes filter parameter (i.e., filter kernels k and bias
terms b) and is learned using user strokes as training data, as explained in Section 4.

Table 1: Definitions of main symbols.
Symbols
I
xi
gi
Ω
XΩ
GΩ
yi
Z
U
θv , θs , θc , θlGv , Gs , Gc , Gl

Descriptions
input image.
image feature vector at pixel i.
binary vector for user strokes at pixel i.
user stroke region.
set of input features xi in stroke region Ω.
set of binary vectors gi in stroke region Ω.
probability vector at pixel i.
probability map or colorized result.
result obtained by post-processing.
parameters of DNN model.
layer functions of DNN model.

3.2. Spatial feature extractor (SFE)The SFE extracts an abstracted feature fs ∈ R256 from a spatial
pixel coordinate s:a single pixel. Additionally, we use pixel coordinates as low-levelspatial features. To extract the corresponding high-level spatial features, we apply a non-linear transformation, since a convolutional
layer cannot be applied in this case. Both of the extracted high-levelfeatures are then combined and used for edit propagation. We empirically experimented with several structures of the DNN, varying
in, e.g., the sizes of convolutional kernels and the dimensions ofhigh-level features.
As illustrated in Figure 3, our DNN is a feedforward neural network that has four structures: visual feature extractor (VFE), spatial
feature extractor (SFE), feature combiner (FC), and label estimator (LE). The main notations used in this paper are summarized inTable 1. The DNN estimates probability vector yi of user strokes
from feature vector xi = [pTi , sTi ]T at pixel i in input image I. Here,pi ∈ R9×9×c is a 9 × 9 patch feature centered at pixel i with c color
channels (e.g., three-channel RGB or one-channel grayscale), andsi ∈ R2 is a pixel coordinate. Each element of input feature vectors is normalized from 0 to 1. In contrast to most previous work,
determining importance of each image feature is not required. In
the following sections, we describe the role of each structure of ourDNN.
3.1. Visual feature extractor (VFE)The VFE extracts a high-level visual feature, such as complicatedtexture patterns or simple colors adapted to user strokes, from a
low-level visual patch feature p. Specifically, the VFE computes
high-level visual features fv ∈ R256 from p using function Gv withmodel parameter θv . Gv consists of two convolutional functions
fconv1 and fconv2 , a max-pooling function fmp , and an activation
function of the rectifier linear unit (ReLU) :fv = Gv (p; θv ),(1)

fs = Gs (s; θs ),(3)

Gs (s; θs ) = fReLU (Ws s + bs ),(4)

where θs = {Ws , bs } is a model parameter of the SFE, and Ws ∈R256×2 and bs ∈ R256 are a weight matrix and bias term. The dimension of the output feature vectors is set to 256 so that they havethe same dimension as the visual features. In the same way, the feature combiner (explained in the next section) can handle both thevisual features and spatial features fairly.

3.3. Feature combiner (FC)The function Gc of the FC converts fv and fs extracted by the VFE
and the SFE into a single feature vector fc ∈ R256 :fc = Gc (fv , fs ; θc ),(6)

Gcv (fv ; θcv ) = Wcv fv + bcv ,(7)

Gcs (fs ; θcs ) = Wcs fs + bcs ,(8)

where θc = {θcv , θcs } is a model parameter of the FC (where θcv ={Wcv ∈ R256×256 , bcv ∈ R256 } and θcs = {Wcs ∈ R256×256 , bcs ∈R256 }). Network structures similar to the FC have also been introduced for combining multimodal features, such as image and audio [NKK∗ 11, WJW∗ 14]. Such a network structure can determine
the importance of two features by capturing correlations across thetwo modalities. In this paper, we utilize this structure for combining
the visual features and spatial features.

3.4. Label estimator (LE)The function Gl of the LE estimates user stroke probability vectors
y from feature vectors fc extracted by the FC:

Gv (p; θv ) =fmp ( fReLU ( fconv2 ( fmp ( fReLU ( fconv1 (p; θconv1 ))); θconv2))),(2)where fconv1 applies 128 types of 3 × 3 × c filter kernels k with biases b to input patch feature p. That is, f conv1 obtains 128 filteredmaps (k ∗ p + b) by convolving input patch features p with different
filter kernels k and biases b. Then, the ReLU activation function
fReLU (v) = [max(0, v1 ), max(0, v2 ), ..., max(0, vm )]T is applied tothe convolved patch (where the subscripts of v denote element indices of the vector). Next, we use a 2 × 2 max-pooling function f mpwith stride 2, which yields position invariance over the patches. We(5)

Gc (fv , fs ; θc ) = fReLU (Gcv (fv ; θcv ) + Gcs (fs ; θcs )),y = Gl (fc ; θl ),(9)

Gl (fc ; θl ) = fso f tmax (Wl fc + bl ),(10)

exp(v )

exp(v )

exp(v )

i

i

i

1
n
, n 2 , ..., ∑n exp(v]T is the softwhere fso f tmax (v) = [ ∑n exp(vi ) ∑ exp(vi )i)max function and θl = {Wl ∈ Rn×256 , bl ∈ Rn } is a model parameter of the LE. The LE determines, for each off-stroke pixel,
a fractional weight for each stroke, based on the off-stroke pixel’s
similarity to pixels on the stroke. This is done using the soft-max
layer, which acts as a standard regression function for probabilistic
multi-class classification.
submitted to EUROGRAPHICS 2016.

Endo, Iizuka, Kanamori and Mitani / DeepProp: Extracting Deep Features from a Single Image for Edit Propagation(Visual feature pre-training)and1st backpropagation based on

4. Learning DNN from User Strokes
This section explains how to learn the model parameters(θv , θs , θc , θl ) of our DNN using training data: user strokes GΩ andinput features x ∈ XΩ on stroke region Ω.
A straightforward way for learning the model parameters is tominimize the error function E between probability vectors estimated by the DNN and binary vectors gi ∈ GΩ obtained from user
strokes:(θ̂v , θ̂s , θ̂c , θ̂l ) = argmin E(θv , θs , θc , θl ),θv ,θs ,θc ,θl

VFE

FC

LE

Use leaned parametersfor 2nd initialization
Minimize
loss

VFE
FC

Each binary vector
obtained from user strokes(12)
2nd backpropagation based onwhere L indicates the cross-entropy loss function defined as
L(y, g) = − ∑nk=1 {gk ln yk + (1 − gk ) ln (1 − yk )}.
Unfortunately, this naïve approach does not work in practice. Because of the structural complexity of our DNN, we need to consider
the fact that many local minima can be found in an optimization
process such as the one described above. In general, model parameters of feedforward neural networks are learned using the backpropagation of error with a gradient-based optimization algorithm,
such as the stochastic gradient descent (SGD) [KB14]. However, in
our case, when the model parameters of the entire network are optimized simultaneously using the backpropagation, the algorithm
falls into local minima before the gradient error propagates sufficiently into the deep layers of the VFE, due to the vanishing gradient problem. That is, the learning speed of the SFE is much fasterthan that of the VFE because the SFE is shallow whereas the VFE
is much deeper.
To address this issue, we develop a learning algorithm for optimizing the model parameters efficiently in two steps. Figure 4 illustrates this learning strategy. In the first step, we omit the SFE fromthe DNN and optimize the deep network consisting of the VFE, FC,
and LE. Specifically, we pre-train visual features by minimizing a
new error function Ev :(θ̂v , θ̂cv , θ̂l ) = argmin Ev (θv , θcv , θl ),θv ,θcv ,θl(13)

Ev (θv , θcv , θl ) =∑ Lv (Gl (Gcv (Gv (pi ; θv); θcv); θl ), gi ),

.
.
.

LE

SFE

i∈Ω

(14)

i∈Ω

where Lv is the cross-entropy loss of the above network. We op∂Lv ∂Lv
v
timize the parameters based on gradients ∂L∂θv , ∂θcv , ∂θl using the
backpropagation. We initialize the model parameters with random
numbers following a normal distribution with zero mean and standard deviation equal to the inverse of the dimension of each model
parameter. For optimization, we use the mini-batch Adam algorithm [KB14] because of its fast convergence. To determine convergence, we check if the error in all training data is smaller thanε|Ω| or the difference between the current error and previous error
is larger than γ|Ω| (where ε and γ are coefficients, and |Ω| is thenumber of pixels in region Ω). Note that the latter condition is used
to abort optimization if it does not converge completely. In thesecond step, we optimize the entire network with the SFE by initializing the DNN with the model parameters θ̂v , θ̂cv , and θ̂l learned
submitted to EUROGRAPHICS 2016.

Minimize
loss

(11)

E(θv , θs , θc , θl ) =∑ L(Gl (Gc (Gv (pi ; θv), Gs (si ; θs ); θc); θl ), gi ),5

andFigure 4: Our strategy of DNN learning. We first pre-train visualfeatures on the network consisting of the VFE, FC, and LE using
backpropagation, and then learn the entire network together with
the SFE.
in the first step. The model parameters θs and θcs not learned in thefirst step are randomly initialized and optimized using the minibatch Adam algorithm in the same way as above. For optimization,mini-batch size is empirically set to 10, and both ε and γ are set to
0.01.
As can be seen in Equations (12) and (14), the computational
complexity of the optimization w.r.t. the number of training data
|Ω| is O(|Ω|). To accelerate the learning, we subsample pixels Ωby 10% of Ω for optimization. Even with subsampling the learned
result is similar to the result with all data due to inherent redundancy, as demonstrated in Section 6.1. Algorithm 1 summarizes
our DNN learning algorithm.
Algorithm 1 Learning DNN from user strokes
Inputs: stroke region Ω, image feature vectors xi = {pi , si } ∈XΩ , and stroke binary vectors gi ∈ GΩ
Outputs: θv , θs , θc , θl
1: Ω ← R ANDOM S AMPLING(Ω);
2: θv , θs , θc , θl ← N ORMAL D ISTRIBUTION();
3: preEv ← 0;
4: while true dov ∂Lv ∂Lv
5:
θv , θcv , θl ← MINI BATCH A DAM(XΩ , GΩ , ∂L∂θv , ∂θcv , ∂θl );// Eq. (14)6:
if Ev < ε|Ω | or Ev − preEv > γ|Ω | then7:
break;
8:
end ifpreEv ← Ev ;9: end while
10: preE ← 0;
11: while true do∂L ∂L ∂L ∂L
12:
θv , θs , θc , θl ← MINI BATCH A DAM(XΩ , GΩ , ∂θ
, , , );v ∂θs ∂θc ∂θl// Eq. (12)13:
if E < ε|Ω | or E − preE > γ|Ω | then14:
break;
15:
end if
preE ← E;
16: end while6

Endo, Iizuka, Kanamori and Mitani / DeepProp: Extracting Deep Features from a Single Image for Edit Propagation

4.1. Efficient DNN update per user edit

5.2. Post-processing

Every time the user adds or removes a stroke in order to obtain adesired result, the system has to re-learn the model parameters fromthe update strokes. Naïvely re-learning from scratch is costly, andthus accelerating the re-learning is crucial for interactive editing.

We refine the estimated result by post-processing, which has mainly
two roles, i.e., smoothing and interpolation. First, we can improve
the result by smoothing it across superpixels. Second, interpolation can alleviate noise and halo artifacts along object boundaries [CZZT12, XYJ13]. Similarly to [XYJ13], we obtain a final
editing result (e.g., colorized or segmented images) U from the result Z estimated with the DNN model, by solving the following
optimization problem:

To update the model parameters efficiently, we reuse the model
parameters learned with the previous user strokes as the initialization of the learning algorithm (line 4 in Algorithm 1). To achievethis, we need to adapt the LE structure accordingly. As shown in
Figure 5, if the total number of different types of user strokes (e.g.,
strokes that edit color for colorization) is reduced, correspondingrows of the model parameters (i.e., the weight matrix Wl and bias
term bl ) are deleted. If a new type of user stroke is introduced, we
add new rows and initialize them with random numbers following
the normal distribution.

Label estimator (LE)Removeold stroke
..
.

=
..
.

Add
new stroke

=

..
.

=

Weight matrix

Figure 5: Efficient model parameter update. If user strokes areadded or removed, model parameters of the LE are updated efficiently using previous parameters.

5. Edit Propagation Using DNN
5.1. Estimating probability maps
We now propagate information from the user strokes to all pixels in
the input image using the learned DNN model. Given a patch feature and coordinate feature of each pixel, our feedforward neuralnetwork outputs a stroke probability vector using the learned parameters. However, processing all pixels one-by-one takes a large
amount of time since the feedforward computation for one data
point includes many applications of convolution filtering.
We adopt superpixel-wise propagation to reduce the computational time. For each superpixel, a center pixel is calculated, andthen a user stroke probability is computed using the DNN modelfrom features calculated only at the center pixel. Finally, the samestroke probability is assigned to all the pixels in the corresponding
superpixel. For generating superpixels, we use the simple linear iterative clustering (SLIC) [ASS∗ 12] because this algorithm can generate regularly-shaped superpixels that can reduce the gap between
shapes of superpixels and patches.

min ∑ βi ||ui − zi ||2 + λ ∑ui ∈U iβi = {ei = 0},∑ φi j ||ui − u j ||2 ,(15)

i j∈N(i)(16)

where zi ∈ Z denotes the quantity being post-processed, e.g., thestroke probability vector yi (in case the probability map itself is thetarget of the post-processing) or the result computed from yi in a
specific application (e.g., color in a colorization application). ei isbinary edge at pixel i in input image I, and  is the morphologicalerosion operator. λ determines the degree of smoothing, and N(i)is a set of neighbor pixels of pixel i. We can use several types ofweighting coefficients φi j that represent relations between neighborpixels, such as the affinity function [LLW04] based on similarity of
pixel values based on similarity of pixel values (when considering
pixel gradient information) and matting Laplacian [LRAL07] formatting. In our experiment, λ is set to 5 for all images. To solve
this linear system, we use the Gauss-Seidel method because of its
simplicity of implementation. Finally, Algorithm 2 summarizes ouredit propagation using the DNN model.
Algorithm 2 Edit propagation using DNN
Inputs: input image I, image feature vectors xi = {pi , si },learned DNN parameters θv , θs , θc , and θl
Output: final editing result U
1: Ωs ← E XTRACT S UPERPIXEL(I);
2: for each superpixel S ∈ Ωs do
3:k ← E XTRACT C ENTER P IXEL (S);4:yk ← Gl (Gc (Gv (pk ; θv ), Gs (sk ; θs ); θc ); θl );5:
Z ← A SSIGN R ESULT T O S UPERPIXEL(S, yk);
6: end for
7: U ← P OSTPROCESS U SING O PTIMIZATION(I, Z);
6. Experiments
We implemented our prototype system with C++ and the
OpenCV2.4.9 library except for deep learning. For implementing
deep learning, we used Python and the chainer library. The system was run on a PC equipped with a 2.80 GHz CPU and 8 GB ofmemory. The image sizes we used in this paper, the number of user
strokes (total pixels), and computational times of our method aresummarized in Table 2. As can be seen in the table, the results of
Figure 1 and the lower center of Figure 10 took longer than mostof the others. This is because the convolutional layers require substantial learning to extract effective visual features such as texturesfrom little available information (i.e., 1-channel grayscale values).
We also tested with the high-resolution image in Figure 6 in our
experiments. Although it took the longest, the time needed for thatimage is relatively short given that the number of pixels is tens oftimes larger than the other images. This is because only pixels on
submitted to EUROGRAPHICS 2016.

Endo, Iizuka, Kanamori and Mitani / DeepProp: Extracting Deep Features from a Single Image for Edit Propagation

Input image and user strokes

Without visual feature pre-training

7

With visual feature pre-training

Figure 6: Comparisons of recolorization results without and with visual feature pre-training.

Table 2: Total amount of inputs and computational times (seconds)of our method on CPU.
ImagesImage sizes# of user strokes(# of total pixels)

Times

Fig.1
Fig.2
Fig.6 top
Fig.6 bottom
Fig.10 top
Fig.10 upper center
Fig.10 lower center
Fig.10 bottom
Fig.12 top
Fig.12 center
Fig.12 bottom

620 × 413
400 × 314
392 × 70
1800 × 1200
480 × 360
640 × 480
500 × 375
640 × 427
400 × 285
400 × 267
400 × 280

20 (7701)
10 (1818)
3 (1882)
19 (17964)
4 (2457)
5 (3208)13 (20342)
9 (3285)
4 (5353)
8 (9666)
3 (6940)

97.4
23.0
18.3
432.4
22.4
36.5
271.6
40.0
36.1
49.9
39.3

Input image anduser strokes

add stroke

remove strokes

w/o update

w/ update

7.1 sec.

7.1 sec.

37.0 sec.

19.8 sec.

5.9 sec.

2.5 sec.

strokes are used for learning and superpixels are used for estimation.
6.1. Evaluation of proposed method
Before comparing our method to previous work, we evaluate ourmethod itself.
Learning algorithm. We first evaluate effectiveness of the proposed learning algorithm for our DNN. Figure 6 shows comparisons of image recoloring results with and without the visual feature
pre-training. Without this, the visual features (texture patterns orcolor) were not sufficiently learned from the image, and the image
edits propagated too strongly, only relying on the spatial features.
This means that the optimization process fell into the local minimum before the gradient of the error function sufficiently propagated to the network of the VFE. Specifically, in the upper-center
image in the figure, the specified colors (pink and blue) are leaking across the different textures because texture features are notextracted sufficiently. Also, in the red box of the lower-center image, the color specified with the green stroke did not propagatedaccording to the texture patterns of the center of the sunflower, due
to the same reason. By contrast, we can see that using our learningalgorithm with visual feature pre-training alleviates these problems
in the right images.
submitted to EUROGRAPHICS 2016.

Figure 7: Comparison of colorization results without (w/o) andwith (w/) updating parameters. The caption under each resultshows computational time of learning.

Updating algorithm. We evaluate the effectiveness of the algorithm for updating the model parameters in Figure 7, where wecompare the editing results and computational times of learning
without and with update. When we do not use the updating algorithm, model parameters are learned from scratch every time
strokes are added or removed. As can be seen in the figure, there isalmost no difference between the two results, whereas the computational times with update become about half of the others.
Training data size. Figure 8 shows the editing results and computational times of learning with different amounts of training data.
As shown in the figure, the results almost converged when we usedequal to or more than 10% of the training data on the strokes, andcomputational times were linearly reduced according to the amount8

Endo, Iizuka, Kanamori and Mitani / DeepProp: Extracting Deep Features from a Single Image for Edit Propagation

1%, 12.3 sec.

10%, 78.3 sec.

Input image & user strokes

100%, 26.5 sec.

10%, 12.0 sec.

2%, 6.2 sec.

1%, 4.8 sec.

0.1%, 4.1 sec.

5%, 55.5 sec.

20%, 228.8 sec.

Figure 8: Results with training data of different ratio. The input
image and user strokes are the same as Figure 1. The caption undereach image shows the percentage of samples of training data (left)
and computational time of learning (right).
of the training data. Although it is generally known that deep learning requires a large amount of training data, in fact, a few hundredsof instances per one category are sufficient for learning models insome image recognition tasks [WYHY15]. Given this fact and theexperimental results, we can confirm that our method works wellwith relatively little training data, and only a few types of user
strokes.
Superpixel-based estimation. Figure 9 shows the editing results
and computational times of estimating with different numbers ofsuperpixels. In this example, the results are visually indistinguishable even when the number of superpixels decreases to 1% of thetotal image pixels. However, the computational times are significantly reduced according to the number of superpixels. Only when
the number of superpixels is 0.1% of the total image pixels, the result starts deteriorating. This is because the superpixels become toolarge to fit the object shape. On the other hand, computational times
were almost unchanged when less than 1% were used: of the totalreported time, 4 seconds are due to the post-processing, which canbe accelerated by using more efficient solvers.
6.2. Comparison with previous methods
To verify the effectiveness of our feature extraction method, we
compare our method with the handcrafted feature-based methods.
6.2.1. Compared features
We prepared the following image features used in previous work.
• color feature: three (or one)-dimensional RGB (or grayscale)
vectors c.
• spatial feature: two-dimensional pixel coordinate vectors s.
• patch feature: 243 (or 81)-dimensional vectors p of 9 × 9 RGB(or grayscale) patches.
• texture feature: 40-dimensional vectors t obtained by applying
the Gabor filter [Dau85, CZL∗ 14].

Figure 9: Results with different number of superpixels. The caption
under each image shows the percentage of the number of superpixels to that of the original image pixels (left) and computational timeof estimation and post-processing (right).

• dense SIFT feature: 128-dimensional vectors d that are scaleinvariant feature transform (SIFT) features [Low99] on eachpixel.
For the texture features, we selected the parameters of the Gabor filter, i.e., kernel size, wavelength of the sinusoidal factor,
spatial aspect ratio, phase offset, standard deviation of the gaussian envelope, and kernel orientation as 30 × 30, π, 1, π/2,
{1, 2, 3, 4, 5}, and {0, π/8, 2π/8, 3π/8, 4π/8, 5π/8, 6π/8, 7π/8},
respectively, and consequently 40-dimensional vectors wereobtained. We concatenated these features and used x =[σc cT , σs sT , σ p pT , σt tT , σd dT ]T in the comparisons with previous
methods. The experiments were conducted by adjusting the importance of each feature σc , σs , σ p , σt , and σd . As mentioned above,
the proposed method uses x = [pT , sT ]T as input feature and we donot manually adjust the importance of them.
6.2.2. Results
The state-of-the-art for edit propagation is the sparse control model(SCM) [XYJ13] except for methods that focus on acceleration andmemory efficiency. Given that comparisons with other edit propagation methods have been conducted in the SCM paper [XYJ13],
and the fact that this paper focused mainly on feature extraction,
we only compare our method to SCM and instant propagation(IP) [LJH10], which is closely related to SCM, using our own implementations.
Colorization and recoloring. Figure 10 shows comparisons ofimage recoloring and colorization between our method (DNN) andthe previous methods with different parameters. The color of eachpixel is computed as the probability-weighted average of the abchannels of each stroke in Lab color space using the probability
maps obtained by our DNN. The colorized results are then postprocessed. The parameters are shown at the bottom of the figure
and correspond to results in each column. First, the result of our
DNN obviously outperformed the leftmost results, which were obtained via the previous methods but using the same features as ours,
i.e., patch and spatial features. Second, the use of all features in theprevious methods tends to deteriorate the estimated results due tosubmitted to EUROGRAPHICS 2016.

[LJH10][LJH10][XYJ13][XYJ13][XYJ13][LJH10]9[LJH10][XYJ13]

Input image & user strokes
Input image & user strokes
Our DNN
Our DNN Input image & user strokes

Our DNN

Our DNN

Input image & user strokes

Endo, Iizuka, Kanamori and Mitani / DeepProp: Extracting Deep Features from a Single Image for Edit Propagation

= 1,
= 0,

= 0)

= 1,
= 1,

= 1)

= 0,

= 0,
= 0,

= 0)

= 0,

= 0)

= 0,
= 1,

= 1)

Figure 10: Comparisons of color image recoloring and grayscale image colorization. For the existing methods, the feature parameters usedin each column are shown in the bottom.
the over-fitting. Third, it is difficult to handle some images even if
individual features are selectively used. In contrast to these results,
our method can generate better results than the previous methods
overall. This means that, from low-level patch and coordinate features, our method extracted stroke-adapted high-level features thatare effective for edit propagation, without manual feature selection.
Foreground segmentation. We compared results of foregroundsubmitted to EUROGRAPHICS 2016.

segmentation using 50 images randomly selected from MSRA 1k
dataset [AHES09]. We quantitatively evaluate the methods via a
precision-recall (PR) curve, which is often used for saliency detection [WLRY15]. While precision indicates the ratio of groundtruth pixels among pixels estimated as foreground, recall indicatesthe ratio of pixels estimated as foreground among all the groundtruth pixels. To extract foreground regions, we segmented probability maps using a threshold. PR curves were plotted with precision10

Endo, Iizuka, Kanamori and Mitani / DeepProp: Extracting Deep Features from a Single Image for Edit Propagation

Input image anduser strokes

Ground truth[LJH10][XYJ13]

Our DNN

Figure 12: Comparisons of several results of foreground segmentation selected from MSRA 1k dataset [AHES09]. Each segmented result isvisualized by binarizing a probability map using a threshold (50%).

Table 3: Macro average precision, recall, and F1 ± standard deviation for binarization of probability maps using a threshold (50%)
on the 50 images. Results marked with ’*’ show statistically significant differences between our method and the others as measured
by paired t-test.
Method[LJH10][XYJ13]
Our DNN(a)

Precision
0.905 ± 0.113
0.900±0.131
0.945 ± 0.060*(b)

Figure 11: (a) Micro average PR curve that shows comparisons offoreground segmentation using 50 images randomly selected fromMSRA 1k dataset [AHES09]. (b) Magnified PR curve.
and recall obtained using different thresholds. We specified foreground and background manually using strokes on each image andused the same strokes for all of the methods with the fixed featureparameters that were used in several of the results of [LJH10], i.e.,σc = 1/0.2 = 5 and σs = 1. As summarized in the PR curve in
Figure 11 and Table 3, our method overall outperforms the existing
methods. Figure 12 also shows some results that are difficult for
the existing methods to segment with only color information, thatis, when the foreground and background colors are very similar.
Although we tried to increase the importance of the texture features in the existing methods, over-fitting problems occurred andresults degenerated in some cases. While the results of the existing
methods might be improved if the feature parameters are tuned intensively, our method can generate relatively good results without
this process.
As shown in Figures 1 and 10, the two existing methods propa-

RecallF10.900 ± 0.168
0.885 ± 0.145
0.892 ± 0.115
0.887 ± 0.099
0.940 ± 0.065
0.940 ± 0.048**∗ : p < 0.05, ∗∗ : p < 0.01

gate image edits based on different points of view. That is, IP estimates stroke probabilities smoothly, whereas SCM estimates them
discriminatively. As explained in [XYJ13], SCM is based on iterative feature discrimination and associates each pixel with only apart of the control samples. For example, SCM can clearly propagate different image edits to neighbor objects that have similarcolor by discriminating the spatial features, even if the importanceof these features is somewhat small. However, it is difficult to takeadvantage of this method unless appropriate features are selected
and the importance of them is appropriately determined. Our focus
is to address this problem, which was demonstrated in this section.
It is an interesting avenue for future work to integrate deep features
into the existing edit propagation methods.
6.3. User study
We also conducted a user study to validate whether users can generate plausible results using our system in as few attempts as possible. We recruited eight users, six novice and two experienced users
in image processing. The four images used in this study were Figures 1, 2, and 10 top and upper center. For each image, we showedsubmitted to EUROGRAPHICS 2016.

Endo, Iizuka, Kanamori and Mitani / DeepProp: Extracting Deep Features from a Single Image for Edit Propagation

Evaluation score

5***: p < 0.05, **: p < 0.01, ***: p < 0.001[XYJ13]
***

***

4
3
2
1
0

*

# of trial and error

Our DNN
***

11

*

***

3
2
1
0

Task 1 Task 2 Task 3 Task 4

Task 1 Task 2 Task 3 Task 4(a) Quality(b) Number of trial and error

Figure 13: Results of user study. Error bars show the standard deviation, and results marked with ’*’ show statistically significant
differences as measured by paired t-test.
sample results to the users and asked them to conduct the following tasks using our system and SCM [XYJ13], allowing them up to
three trials until they are satisfied:• Task 1 (Fig. 2): recolor groups of flower petals at different locations with the same colors.
• Task 2 (Fig. 10 upper center): recolor each type of cookies.
• Task 3 (Fig. 10 top): recolor only the peels of the pear.
• Task 4 (Fig. 1): colorize each object such as leaves, woods, anda cat.
Before conducting these tasks, we briefly explained how to use thesystems and the image features to the users, and they practiced forabout five minutes using other images such as Figure 6 top. Because the users might get used to editing the same image from thesecond time, we divided the users into two groups and shuffled theorders of images and methods for each group. For SCM, we limited the appropriate ranges of the feature weights from zero to fivein order to avoid eccentric feature weights. The number of trials
were recorded, and the resulting images were evaluated by another11 evaluators using a subjective score ranging from one to five, assummarized in Figure 13. Thanks to the tuning-free characteristicof our system, the number of trials for the editing was typicallyaround two, which was less than that of SCM. Additionally, thequality of resultant images using our system consistently outperforms that of the existing method.
7. Discussion and Future Work
Other possible approaches. In the context of the proposed
method, it is possible to consider two other approaches: (1) datadriven pre-training and (2) network expansion.
As for (1), in several applications for computer vision, DNNs are
pre-trained using datasets containing a large number of general images such as ImageNet [DDS∗ 09] before training on data for specific tasks [LY15, LSL15]. This improves the generalization abilityof models and accuracy on the tasks. In contrast, our method does
not use such data-driven approaches. Nevertheless, it successfully
generates several image editing results. It is not clear that datadriven pre-training would improve the task handled in this paper
because learned features strongly depend on the variously defineduser strokes. Additionally, applying such pre-trained models to ourtask is not straightforward because existing pre-trained models aredesigned for image-level classification, but our task is pixel-levelclassification. For example, Network in Network (NIN) [LCY13]
submitted to EUROGRAPHICS 2016.

Figure 14: Results with NIN model instead of our VFE. The same
input image and user strokes are used for each image in Figures 1,
2, 10 top and upper center.

and AlexNet [KSH12] require a relatively-large image as input, that
is, an image of 227 × 227 pixels in order to classify the image as"scenery", "objects", or "animals", etc. On the other hand, our taskneeds to classify individual pixels in a single image, and thus we
used relatively-small 9 × 9 image patches as input. If we were to
disregard this essential difference and still integrate a pre-trainedmodel into our framework, a naïve way would be to simply replace
our VFE with a pre-trained model. We conducted such an experiment with NIN, which is a lightweight pre-trained model using ImageNet and whose performance is sometimes slightly better than
AlexNet in image classification tasks. Specifically, we used NIN
without its classification layer, which outputs 1000-dimensionalfeature vectors from input images, and fixed its model parameters
during a learning session in order to leverage pre-trained features.
Because NIN (and also AlexNet) requires images of 227 × 227 pixels, we magnified 9 × 9 input patches to 227 × 227 patches and fed
these large patches to NIN. As clearly shown in Figure 14, the NIN
model cannot appropriately capture visual features. Even worse,
this approach took a significantly longer time (five minutes on average) despite the fact that the NIN model is lightweight one, as thisnetwork has a too deep structure for our task. Because such datadriven approaches are out of the scope of our research question, we
consider such problems as future work.
As for (2), it is interesting to use more wide-scale networks, andwe tried to use DNNs with deeper layers and larger kernels. However, there were no significant changes in results while the computational time increased when we used images with up to 2M pixels
in our experiments. If we use larger images, a network that automatically changes its scale according to image sizes could possibly
improve editing results.
GPU acceleration. Most of the modules in our system can be
accelerated using parallel computation on GPU. To this end, theGPU-based SLIC algorithm for generating superpixels was proposed [RR11], linear solvers for the post-processing was also accelerated [JCVV09], and learning and estimation algorithms for12Endo, Iizuka, Kanamori and Mitani / DeepProp: Extracting Deep Features from a Single Image for Edit Propagation

DNNs were parallelized [LNC∗ 11]. The chainer library we usedmakes GPU implementation easy, and thus we tried to use GPU acceleration for deep learning with our method. The algorithm on PC
with NVIDIA GeForce GTX 760 was about five times faster than
CPU implementation. For high resolution images, our system may
provide instant feedback with a low resolution result as a preview
during editing sessions. We would like to accelerate our system for
more interactive editing in the future.
8. Conclusion
We have proposed DeepProp, which achieves various image edits from only simple user strokes using deep leaning. As for our
research question, we conclude that, without manual feature selection, effective features for edit propagation can be automaticallyextracted by (i) deep learning from sparse user inputs in a single image and (ii) efficiently learning the DNN in order to avoid
falling into undesirable local solutions due to the vanishing gradient problem. Our edit propagation system using deep features has
generated better results than previous work in several applications
such as grayscale image colorization, image recoloring, and foreground segmentation. In the future, we plan to try the data-driven
approaches for learning DNNs, design more flexible DNN structures to handle various images, and accelerate our system for moreinteractive editing.
Acknowledgements
We would like to thank Dr. Olga Diamanti for proofreading ourpaper. We also thank anonymous reviewers for their insightful suggestions. Yoshihiro Kanamori is funded by JSPS Postdoctoral Fellowships for Research Abroad.
References[AHES09] A CHANTA R., H EMAMI S., E STRADA F., S USSTRUNK S.:
Frequency-tuned Salient Region Detection. In CVPR 2009 (2009),
pp. 1597 – 1604. 9, 10[AP08] A N X., P ELLACINI F.: AppProp: All-pairs appearance-space edit
propagation. In ACM SIGGRAPH ’08 (2008), pp. 40:1–40:9. 1, 3
[ASS∗ 12] A CHANTA R., S HAJI A., S MITH K., L UCCHI A., F UA P.,
S USSTRUNK S.: SLIC superpixels compared to state-of-the-art superpixel methods. IEEE Trans. Pattern Anal. Mach. Intell. 34, 11 (Nov.
2012), 2274–2282. 6[Ben09] B ENGIO Y.: Learning deep architectures for AI. Found. Trends
Mach. Learn. 2, 1 (Jan. 2009), 1–127. 2, 3
[BHW11] B IE X., H UANG H., WANG W.: Real time edit propagation
by efficient sampling. Comput. Graph. Forum 30, 7 (2011), 2041–2048.
3[BLP∗ 07] B ENGIO Y., L AMBLIN P., P OPOVICI D., L AROCHELLE H.,
M ONTREAL U. D., Q UEBEC M.: Greedy layer-wise training of deep
networks. In In NIPS (2007), MIT Press. 3[BPD09] B OUSSEAU A., PARIS S., D URAND F.: User-assisted intrinsic
images. In ACM SIGGRAPH Asia ’09 (2009), pp. 130:1–130:10. 2[CZL∗ 14] C HEN X., Z OU D., L I J., C AO X., Z HAO Q., Z HANG H.:
Sparse dictionary learning for edit propagation of high-resolution images. In CVPR 2014 (2014), pp. 2854–2861. 1, 2, 3, 8
[CZZT12] C HEN X., Z OU D., Z HAO Q., TAN P.: Manifold preserving
edit propagation. ACM Trans. Graph. 31, 6 (2012), 132:1–132:7. 1, 6
[Dau85] D AUGMAN J. G.: Uncertainty relation for resolution in space,spatial frequency, and orientation optimized by two-dimensional visual
cortical filters. J. Opt. Soc. Am. A 2, 7 (1985), 1160–1169. 8[DDS∗ 09] D ENG J., D ONG W., S OCHER R., L I L.-J., L I K., F EI -F EI
L.: ImageNet: A Large-Scale Hierarchical Image Database. In CVPR
’09 (2009). 11[DLH∗ 13] D ENG L., L I J., H UANG J., YAO K., Y U D., S EIDE F.,S ELTZER M. L., Z WEIG G., H E X., W ILLIAMS J., G ONG Y., A CERO
A.: Recent advances in deep learning for speech research at microsoft.
In IEEE ICASSP ’13 (2013), pp. 8604–8608. 2, 3
[FFL10] FARBMAN Z., FATTAL R., L ISCHINSKI D.: Diffusion maps
for edge-aware image editing. ACM Trans. Graph. 29, 6 (Dec. 2010),
145:1–145:10. 3[GBB11] G LOROT X., B ORDES A., B ENGIO Y.: Deep sparse rectifier
neural networks. In AISTATS ’11 (2011), Gordon G. J., Dunson D. B.,(Eds.), vol. 15, pp. 315–323. 3[JCVV09] J OST T., C ONTASSOT-V IVIER S., V IALLE S.: An efficient
multi-algorithms sparse linear solver for GPUs. In ParCo2009 (Lyon,
France, Sept. 2009). 11[KB14] K INGMA D. P., BA J.: Adam: A method for stochastic optimization. CoRR abs/1412.6980 (2014). 5[KSH12] K RIZHEVSKY A., S UTSKEVER I., H INTON G. E.: ImageNetclassification with deep convolutional neural networks. In Advances in
Neural Information Processing Systems 25. 2012, pp. 1097–1105. 2, 3,
11[LAA08] L I Y., A DELSON E., A GARWALA A.: Scribbleboost: Addingclassification to edge-aware interpolation of local image and video adjustments. In EGSR ’08 (2008), pp. 1255–1264. 1, 2, 3
[LCY13] L IN M., C HEN Q., YAN S.: Network in network. CoRR
abs/1312.4400 (2013). 11[LFUS06] L ISCHINSKI D., FARBMAN Z., U YTTENDAELE M.,
S ZELISKI R.: Interactive local adjustment of tonal values. ACM Trans.
Graph. 25, 3 (July 2006), 646–653. 2[LGQ15] L ANE N. D., G EORGIEV P., Q ENDRO L.: DeepEar: Robust
smartphone audio sensing in unconstrained acoustic environments using
deep learning. In UbiComp ’15 (2015), pp. 283–294. 3[LJH10] L I Y., J U T., H U S.-M.: Instant propagation of sparse edits on
images and videos. Comput. Graph. Forum 29, 7 (2010), 2049–2054. 1,
3, 8, 10
[LLW04] L EVIN A., L ISCHINSKI D., W EISS Y.: Colorization using optimization. ACM Trans. Graph. 23, 3 (Aug. 2004), 689–694. 1, 2, 6
[LNC∗ 11] L E Q. V., N GIAM J., C OATES A., L AHIRI A., P ROCHNOW
B., N G A. Y.: On optimization methods for deep learning. In ICML
(2011), pp. 265–272. 12[Low99] L OWE D. G.: Object recognition from local scale-invariant features. In ICCV ’99 (1999), pp. 1150–1157. 8[LRAL07] L EVIN A., R AV-A CHA A., L ISCHINSKI D.: Spectral matting.
In CVPR (2007), IEEE Computer Society. 2, 6
[LSL15] L IU F., S HEN C., L IN G.: Deep convolutional neural fields for
depth estimation from a single image. In CVPR 2015 (June 2015). 3, 11[CLKL14] C HO H., L EE H., K ANG H., L EE S.: Bilateral texture filtering. ACM Trans. Graph. 33, 4 (July 2014), 128:1–128:8. 2[LWA∗ 12] L ANG M., WANG O., AYDIN T., S MOLIC A., G ROSS M.:
Practical temporal consistency for image-based graphics applications.
ACM Trans. Graph. 31, 4 (July 2012), 34:1–34:8. 3[CSRP10] C RIMINISI A., S HARP T., ROTHER C., P’ EREZ P.: Geodesic
image and video editing. ACM Trans. Graph. 29, 5 (2010), 134:1–
134:15. 3[LWCO∗ 07] L UAN Q., W EN F., C OHEN -O R D., L IANG L., X U Y.-Q.,
S HUM H.-Y.: Natural image colorization. In EGSR ’07 (2007), pp. 309–
320. 1, 2
submitted to EUROGRAPHICS 2016.

Endo, Iizuka, Kanamori and Mitani / DeepProp: Extracting Deep Features from a Single Image for Edit Propagation[LY15] L I G., Y U Y.: Visual saliency based on multiscale deep features.
In CVPR 2015 (June 2015). 3, 11[MCY∗ 13] M USIALSKI P., C UI M., Y E J., R AZDAN A., W ONKA P.: A
framework for interactive image color editing. Vis. Comput. 29, 11 (Nov.
2013), 1173–1186. 3[NKK∗ 11] N GIAM J., K HOSLA A., K IM M., N AM J., L EE H., N G
A. Y.: Multimodal deep learning. In ICML (2011), pp. 689–696. 4[PL07] P ELLACINI F., L AWRENCE J.: AppWand: Editing measured materials using appearance-driven optimization. ACM Trans. Graph. 26, 3(July 2007). 2[QWH06] Q U Y., W ONG T.-T., H ENG P.-A.: Manga colorization. ACM
Trans. Graph. 25, 3 (July 2006), 1214–1220. 2[RR11] R EN C. Y., R EID I.: gSLIC: a real-time implementation of SLIC
superpixel segmentation. Tech. rep., University of Oxford, Departmentof Engineering Science, 2011. 11[RRW∗ 09] R HEMANN C., ROTHER C., WANG J., G ELAUTZ M.,
K OHLI P., ROTT P.: A perceptually motivated online benchmark forimage matting. In CVPR ’09 (2009), pp. 1826–1833. 2[SCXP15] S UN J., C AO W., X U Z., P ONCE J.: Learning a convolutional neural network for non-uniform motion blur removal. In CVPR
’15 (2015). 3[SWW∗ 15] S HEN W., WANG X., WANG Y., BAI X., Z HANG Z.: Deepcontour: A deep convolutional feature learned by positive-sharing loss
for contour detection. In CVPR ’15 (June 2015). 3[WJW∗ 14] W U Z., J IANG Y.-G., WANG J., P U J., X UE X.: Exploring
inter-feature and inter-class relationships with deep neural networks forvideo classification. In ACM MM ’14 (2014), pp. 167–176. 4[WLRY15] WANG L., L U H., RUAN X., YANG M.-H.: Deep networks
for saliency detection via local estimation and global search. In CVPR
’15 (June 2015). 9[WYHY15] W U J., Y U Y., H UANG C., Y U K.: Deep multiple instancelearning for image classification and auto-annotation. In CVPR 2015
(2015). 8[XLJ∗ 09] X U K., L I Y., J U T., H U S.-M., L IU T.-Q.: Efficient affinitybased edit propagation using k-d tree. In ACM SIGGRAPH Asia ’09(New York, NY, USA, 2009), pp. 118:1–118:6. 1, 3
[XLXJ11] X U L., L U C., X U Y., J IA J.: Image smoothing via l0 gradient
minimization. ACM Trans. Graph. 30, 6 (Dec. 2011), 174:1–174:12. 2[XWT∗ 09] X U K., WANG J., T ONG X., H U S.-M., G UO B.: Edit propagation on bidirectional texture functions. Computer Graphics Forum28, 7 (2009), 1871–1877. 1, 2
[XXY∗ 15] X IAO T., X U Y., YANG K., Z HANG J., P ENG Y., Z HANG Z.:
The application of two-level attention models in deep convolutional neural network for fine-grained image classification. In CVPR ’15 (2015).
3[XYJ13] X U L., YAN Q., J IA J.: A sparse control model for image andvideo editing. ACM Trans. Graph. 32, 6 (Nov. 2013), 197:1–197:10. 1,
3, 6, 8, 10, 11[YS06] YATZIV L., S APIRO G.: Fast image and video colorization using
chrominance blending. IEEE Transactions on Image Processing 15, 5(2006), 1120–1129. 2[YY15] YATAGAWA T., YAMAGUCHI Y.: Sparse pixel sampling for appearance edit propagation. The Visual Computer 31, 6-8 (2015), 1101–
1111. 1, 3
[YZW∗ 15] YAN Z., Z HANG H., WANG B., PARIS S., Y U Y.: Automatic
photo adjustment using deep neural networks. ACM Transactions on
Graphics (2015). 3

submitted to EUROGRAPHICS 2016.

13