Laplacian Denoising Autoencoder
Authors: Jianbo Jiao
Linchao Bao
Yunchao Wei
Shengfeng He
Honghui Shi
Rynson Lau
Thomas Huang
Venue:  (2020)
Keywords: unsupervised
representation learning
Laplacian

Abstract: While deep neural networks have been shown to perform remarkably well in many machine learning tasks, labeling a large amount of supervised data is usually very costly to scale. Therefore, learning robust representations with unlabeled data is critical in relieving human effort and vital for many downstream applications. Recent advances in unsupervised and self-supervised learning approaches for visual data benefit greatly from domain knowledge. Here we are interested in a more generic unsupervised learning framework that can be easily generalized to other domains. In this paper, we propose to learn data representations with a novel type of denoising autoencoder, where the input noisy data is generated by corrupting the clean data in gradient domain. This can be naturally generalized to span multiple scales with a Laplacian pyramid representation of the input data. In this way, the agent has to learn more robust representations that can exploit the underlying data structures across multiple scales. Experiments on several visual benchmarks demonstrate that better representations can be learned with the proposed approach, compared to its counterpart with single-scale corruption. Besides, we also demonstrate that the learned representations perform well when transferring to other vision tasks.

Under review as a conference paper at ICLR 2020
LAPLACIAN DENOISING AUTOENCODER
Anonymous authors
Paper under double-blind review
ABSTRACT
While deep neural networks have been shown to perform remarkably well in many machine learning tasks, labeling a large amount of supervised data is usually very costly to scale. Therefore, learning robust representations with unlabeled data is critical in relieving human effort and vital for many downstream applications. Re- cent advances in unsupervised and self-supervised learning approaches for visual data have benefited greatly from domain knowledge. Here we are interested in a more generic unsupervised learning framework that can be easily generalized to other domains. In this paper, we propose to learn data representations with a novel type of denoising autoencoder, where the noisy input data is generated by corrupting clean data in the gradient domain. This can be naturally generalized to span multiple scales with a Laplacian pyramid representation of the input data. this way, the agent learns more robust representations that exploits the underlying data structures across multiple scales. Experiments on several visual benchmarks demonstrate that better representations can be learned with the proposed approach, compared to its counterpart with single-scale corruption. Furthermore, we also demonstrate that the learned representations perform well when transferring other vision tasks.
INTRODUCTION
In recent years, deep learning has made significant improvements on machine learning tasks. How- ever, the success of deep-based approaches relies greatly on using a large amount of human labeled data for supervision, which is usually very costly and infeasible to scale on new data. Actually, humans are exceptional experts at learning abstract knowledge from unsupervised data, i.e., with- out knowing the specific labels of the data. Thus, how imitate such human cognitive ability and effectively learn robust representations from massive sums of unlabeled data in an unsupervised manner have attracted the interests of machine learning researchers.
Representation learning is a popular framework for unsupervised learning that aims to learn trans- ferable representations from unlabeled data (Bengio et al., 2013). Although great progress has been achieved for visual data by some recent advances (Zhang al., 2016; 2017; Pathak et al., 2016; Noroozi & Favaro, 2016; Doersch et al., 2015; Noroozi et al., 2017: Pathak et al., 2017: Gidaris mostly designed to boost the performance of high-level recognition
al., 2018), the approaches tasks like classification and detection. We argue that good representations should benefit multiple kinds of tasks, including both high-level recognition tasks and low-level pixel-wise prediction tasks. We, in this paper, present a novel unsupervised representation learning approach that is applicable more generic type of data and tasks. The only assumption about the input data form that the learned representations should incorporate the underlying data structures along some certain dimen- sions. For example, one would expect the representations for visual data to incorporate underlying image structures along the spatial dimension, while the representations for speech data might need to be exploited along the temporal dimension.
Specifically, we propose to decouple the representations into different semantic levels in the Lapla- cian domain. A novel type denoising autoencoder (DAE) (Vincent et al., 2010) is proposed to distill both high- and low-level representations accordingly. Different from the conventional DAE, where the noisy input is generated from the clean data by adding noises in the original space, we propose to generate noisy input by corrupting the clean data in the gradient domain. By perturb- ing the clean data in such manner, the corruptions are diffused into larger scales and made more difficult to remove (Fig. (a)). More importantly, the gradient domain corruption can be naturally

---

Under review as a conference paper at ICLR 2020
Raw
DAE
Spatial
LPS-4 LPS-6 LPS-8
LapDAE
(a)
(b)
Figure I: (a) Top: illustration of our Laplacian pyramid based corruption construction strategy com- pared traditional spatial corruption, where "LPS" indicates the Laplacian pyramid scale; Bottom: learned kernels when corruption is added in spatial domain (left) and gradient domain (right). (b) Vi- sualization of the discriminative capability on the MNIST test dataset, where samples are projected to the 2D domain by using the t-SNE technique (Maaten & Hinton, 2008).
extended to span multiple scales with a Laplacian pyramid representation of the data (Burt & Adel- son, 1983). To this end, the DAE is enforced to learn more robust representations that can exploit the underlying data structures across multiple scales. In addition, the proposed learning approach can easily be incorporated into other representation learning frameworks, and boosts their performance accordingly.
Our motivation is inspired by the human knowledge learning by visual perception. Instead trying to remember every single detail, human vision focuses more the general concept of the ob- jeet/scene, which favors a combined perception of both local and non-local information (Bubić al., 2010). An example of the proposed gradient-domain corruption illustrated in Fig. (a). can be observed that compared to directly adding noise spatially, editing on different scales of the Laplacian pyramid leads to non-local random corruptions. We also show the learned kernels by the corruption in the spatial domain and that in the gradient domain at the bottom of Fig. 1 (a). be observed that more edge-sensitive and color-sensitive kernels and non-local responses learned by the gradient domain corruption (right) in comparison to spatial corruption (left) which preferring local responses. We argue that order for agent to be able recover the corruptions from different scales non-locally, it requires an understanding of the context in the scene.
In Fig. 1 (b), we illustrate the discriminative capability of our model on the MNIST (LeCun al., 1998) testing set. The visualization is achieved by projecting the high-dimensional data or feature to a two-dimension space, using the t-SNE (Maaten & Hinton, 2008) technique. Compared to the raw data distribution, the embedding space of the conventional denoising autoencoder shows better clustering ability while with some background noise. When compared to the embedding of the pro- posed Laplacian denoising autoencoder (LapDAE), are well discriminated from each other and with much less noise. For example, the digit 5 and 3 are better discriminated compared to those from the data and from DAE.
observe that different categories (digits)
We demonstrate the effectiveness of the proposed unsupervised learning approach in two folds: 1) by evaluating the clustering and discriminative capability on classic benchmarks; 2) by training large-scale data and transferring the learned visual features to a variety of vision tasks including multi-label classification, object detection, and semantic segmentation. The main contributions of our work are summarized as follows:
We propose a new unsupervised representation learning framework by enforcing the model to learn more context and discriminative information in the Laplacian domain.
The proposed framework is trained only based the data itself and neither the data domain assumptions nor pseudo labels are necessary.
Our framework is superior to the conventional DAE and achieves state-of-the of-thh-rrt perfor- mance several benchmarks.
2

---

Under review as a conference paper at ICLR 2020
noise
revonstruci
Clean image
Gaussian pyramid
Laplacian pyramid
Corrupted image
Figure 2: The illustration of the corruption with a Laplacian pyramid. A Gaussian pyramid is first constructed from the clean image, from which Laplacian pyramid built. After randomly selecting level (slice) from the Laplacian pyramid and adding random corruptions (e.g., noise), the final corrupted image is obtained by reconstruction from the modified Laplacian pyramid.
RELATED WORKS
The conventional autoencoder (AE) (Hinton & Salakhutdinov, 2006) is based on the idea of learning a mapping from high-dimension low-dimension so that the encoded repre- sentation can be used to reconstruct the original raw input. Bengio et al. (2007) propose to learn the abstract representation by stacking single-layer autoencoders. Poultney et al. (2007) impose sparsity prior for the latent encoded space. Furthermore, the denoising autoencoder (DAE) (Vincent al., 2010) proposed to achieve abstraction that is robust noise and proven to be able to learn better representations. The variational autoencoder (VAE) (Kingma & Welling, 2013) aims to learn a parametric latent variable model by encouraging the latent space to satisfy a distribution. refer to (Bengio et al., 2013) for broader view of autoencoder-brit based approaches.
Autoencoders:
Representation learning: As fundamental problem, representation learning has been studied for years. A comprehensive review could be referred to (Bengio et al., 2013). Early classical algorithms mainly focus on reconstructing the raw data by learning compressed features (Hinton & Salakhut- dinov, 2006; Vincent et al., 2010). Some other methods use probabilistic models like Boltzmann machines (Hinton & Sejnowski, 1986; Salakhutdinov Larochelle, 2010) and GANs (Goodfellow et al., 2014; Donahue al., 2016). Recently, some approaches address the problem by defining some pretext tasks (termed "self- sueff-sueee learning") which have shown promising performance, includ- ing predicting relative spatial location/orderin of image patches (Doersch et al., 2015; Noroozi & Favaro, 2016; Noroozi et al., 2017), motion in video (Misra et al., 2016; Pathak et al., 2017; Wang & Gupta, 2015), colorization (Zhang al., 2016; 2017), predicting rotations (Gidaris et al., 2018) and transformations (Zhang et al., 2019), to name few. Such pretext-task-bet categorized into a different group compared the AE/DAE- ones. Compared to these repre- sentation learning methods, the proposed approach does not make any assumptions to the data thus more generic.
methods can be
LAPLACIAN DENOISING AUTOENCODER
3.1 BACKGROUND
The original Laplacian pyramid was proposed for image editing (Burt & Adelson, 1983) and can easily be generalized to other types of data where a low-pass filter is applicable. Given input data x, its Gaussian pyramid is composed of a set progressively lower resolution versions of the data, denoted as {af) where / is pyramid level. In the pyramid, the bottom level is the data itself, = {af} is constructed by subtracting the neighboring levels in the Gaussian pyramid, 2/ 2 (apsapple(=__], ) Note that the top level of the Laplacian pyramid is the residual and the same as that in the Gaussian pyramid, æ - where N is the top level of the pyramid. The construction process is shown in Fig. 2. Given a Laplacian pyramid, the original data can be reconstructed by recursively applying x = af + upsample(z (= until 2 is reached. Gradient domain editing on x can be achieved by editing its Laplacian pyramid and then reconstruct the resulting & from the modified Laplacian pyramid.
and 2 downsample(xf)) The Laplacian pyramid
3

---

Under review as a conference paper at ICLR 2020
3.2 LAPDAE METHODOLOGY
Following the denoising autoencoder framework (Vincent al., 2010), attempt distill the essential representations by training a convolutional network (ConvNet) to restore the clean data E X' from the corrupted data ğ E X. In contrast to a standard DAE, we generate the corrupted data from with the aid of a Laplacian pyramid. Specifically, we construct Laplacian pyramid from the clean data and randomly corrupt level of the pyramid, such that 7 is reconstructed from the corrupted pyramid. Fig. 2 illustrates the process of the corruption with an example of image data. Since the corruption applied to higher levels of the pyramid affects larger spatial scales of the image (see Fig. 1), the randomly corrupted levels will enforce the network to learn features that can represent underlying structures across multiple scales. As known in the literature (Zeiler & Fergus, 2014), the ConvNet is inherently in favor of both local and non-local features at different levels of layers. Hence, with only local disturbances, it is difficult capture the non-local semantic concepts. This has also been verified to some extent in the self-supervised learning methods that attempt to leverage patch-based context information (Doersch et al., 2015; Noroozi & Favaro, 2016; Noroozi et al., 2017), and similarly to the non-local scheme (Wang et al., 2018) on the network design side. By adding corruptions across multiple scales, the objective is capture both local and non-local information during the representation learning phase. Additionally, in order to incorporate diverse types of corruptions and to force the network to "learn harder", it is also possible apply multiple types of corruptions to the pyramid during learning.
Lap(æ;; Ti) (note that we use ãi instead of x7 for concise- ness to denote a corrupted level / in the Lapla- cian pyramid). The corrupted input space is achieved as L {Lap(x; Fy)e, E C}, where is the corruption type set. Then the corrupted space is mapped to hidden space y = Fo(C.) through an encoder with parameter 0. Differ- ing from DAE that uses sparse code den representation, each sample in our hidden space has its resolution, from which we recover the reconstruction space Z = Go ()) with a decoder with parameter 0'. During the optimization in mini-batch, each time a training sample x is presented, one or multiple versions of corruptions are constructed according to C. Therefore each time the optimization based on sub-mini-bant batch, for which sub-batch re- construction objective is defined as:
Denote the corrupted data as
Algorithm 1: LapDAE Optimization Initialize corruption set C and parameters 0, H' while not converged do for x E X do
Compose the Gaussian pyramid (=f}
Construct the Laplacian pyramid
{a/} from {af*
for c E C do
hid-
Randomly select pyramid level I
T7 + apply corruption e on level (
end
Reconstruct the corrupted data
ā Lap(<; T.) in image domain
y (2) 2 Ggr
end
min
ΣΣ || x Ze 11 .EX'EC
end
Return 0,0 0' for the LapDAE
Troc >EE
F.z x|
2e 112
where 2e E ZZ is the reconstructed data by the network. The learning process of the proposed LapDAE is summarized in Algorithm 1.
The proposed Laplacian denoising autoencoder performs data reconstruction in the Laplacian pyra- mid space across multiple scales. Despite simple and making no assumptions about the data and requiring no specially designed domain-specific loss functions, the proposed framework able to learn representations competitively with existing unsupervised (as well as some self-supervised) ap- proaches. This will be shown the following evaluation section. Similar to some recent work which has explored withholding parts of the data (e.g., AE to remove noise; inpainting and context-encoder to drop data in spatial domain; colorization to drop data along channel direction), our LapDAE model can be considered as removing context-aware noise along the scale direction in Laplacian domain.
Being purely unsupervised model, this is a generic framework that can be applied to other domains in addition to visual data. The proposed framework opens a new potential direction for representa- tion learning in another transferred domain (e.g., gradient domain), which we believe to be beneficial
4

---

Under review as a conference paper at ICLR 2020
{
2 3 4
6
7
8
9
Original images
7
g
9
Reconstruction conventional DAE from noise space
Loss (log)
DAE
9
-6-
LapDAE
Reconstruction by conventional
Lap space
2
4
5
7
8
9
-8-
Reconstruction by the proposed LapDAE from noise on input space
2
4
S
6
7
6
9
100
200
600
800
1000
Reconstruction by the proposed LapDAE from noise
Lap space
lterations
Figure 3: Left: An illustration of the reconstruction performance on the MNIST dataset. The original raw input images randomly selected from the test set and are shown in the first row, while the second and last rows show the reconstructed results from conventional DAE and our LapDAE, respectively. DAE applied on Lap noise space and LapDAE for reference in the third and fourth rows. Right: Illustration on model convergence. The horizontal axis shows the training iterations while vertical axis the training loss (in log scale).
spatial noise space are also shown
to the community, in which current work focuses mainly on knowledge mining in the original (spa- tial) domain.
3.3 LAPDAE ARCHITECTURE
In this section, we describe the detailed architecture of the proposed LapDAE. Specifically, we utilize a convolutional neural network (CNN) to implement the LapDAE and showcase on visual data. The corruption in the Laplacian domain is modeled as a Laplacian layer. Specifically, we randomly add Gaussian noise (with a = 25) to a randomly chosen level / the Laplacian pyramid. The encoder consists of several simple convolutional (conv) layers, while the decoder is of a mirrored structure to the encoder and consists of up-conv (also termed deconv in some literature) layers. The model is trained with supervision from the reconstruction objective defined in Equation (1).
EXPERIMENTS
4.1 EXPERIMENTAL SETUP
The setup of the proposed framework is described in Sec. 3.3, For the basic LapDAE architecture, four 3 x 3 cone layers are used to construct the encoder, while the decoder consists of three similar up-conv layers. For the experiments performed on large-scale datasets (Sec. 4.4, 4.5), we use the AlexNet (Krizhevsky et al., 2012) structure for the encoder and similarly the decoder consists of three up-conv layers. For simplicity in this study, only one corruption type, random noise, set in C. The Laplacian pyramid is constructed with eight levels. The whole model is trained end-to-end by the Adam optimizer (Kingma & Ba, 2014), with the learning rate set to 10 The learning rate decreases at factor of 10 for every 20 epochs. A batch size of 128 is used throughout the experiments.
4.2 EVALUATION ON MNIST
The MNIST' dataset consists of 70,000 images of handwritten digits with size 28 x 28, in which 60,000 are used for training and the rest 10,000 for testing. Randomly selected example images from the MNIST are shown in Fig. 3 (left). In this experiment, the input images are fed into the model fixed size of 28 28 with only horizontal flipping data augmentation during training. As the objective for our model is set as the reconstruction error (as in Equation 1), we first illustrate the qualitative performance on the image reconstruction, shown Fig. (left). As we can see from the
httpp://yann, lecun. com/ /ewp//xddmmmmmm / st./
5

---

Under review as a conference paper at ICLR 2020
Aig
Fed
die
Aeg
Frop
Akpp
Original raw imagass
Reconstirction conversional DAE
Reconstraction by the proposed LagIAR
Query
LapDAE
(R) Comparison reconstruction
(b) Comparison on image retrieval
Figure 4:
Evaluation using the CIFAR-I0 dataset. (a) The reconstruction result comparison, in which examples from each category are randomly selected for visualization.
(b) The image retrieval results by nearest neighbor. Given the query on the left, the top-5 (from left to right) retrieved results of DAE (middle) and LapDAE (right) are presented, in which red the ones indicate wrong category.
reconstruction results, the conventional DAE generally reconstructs the digits but they are unclear and include some noise. In contrast, the reconstruction of our LapDAE model is evidently much clearer and includes more details, e-g., -, the numbers 0 and 4. To better understand the reconstruction capability, we apply the conventional DAE to images corrupted with Laplacian noise, with compar- ison to applying our LapDAE images where the noise is added on the input space. The results again suggest that the proposed LapDAE performs better on reconstructing context information, i.e., digits here. We also compare the convergence property of the proposed LapDAF and conventional DAE, as shown in Fig. 3 (right). It can be observed that with the proposed LapDAE, the model converges faster and results in reaching a much more optimum level.
We perform experiment on image clustering for both DAE and the proposed LapDAE models. The result shown Fig. (b). From the results, we can see that the proposed LapDAE has a far better discriminative capability compared to the conventional DAE.
4.3 EVALUATION ON CIFAR
In comparison to the MNIST dataset, the CIFAR-10 (Krizhevsky, 2009) dataset is composed of RGB natural images with a size of 32 x 32, covering different categories of natural objects. The training set consists of 50,000 images while the testing set is 10,000. Each category includes 6,000 images. In this experiment, we first visualize the reconstructed images and compare to those that by using the
reconstructed by the conventional DAE, Fig. 4 (a). From the results proposed LapDAE, the representative context is well reconstructed, e.g., the face of the cat and the deer in the forest. We also explore the quality of the learned representation, by performing retrieval task. The retrieval is based on the similarity of the embedding space, by using the nearest neighbor scheme. Given input query image, the feature the bottleneck (latent space) of the model is extracted for the retrieval in the whole testing set. For this experiment, we compare with results from the conventional DAE. Fig. 4 (b) shows example results. From the results in Fig. 4 (b) we can observe that our LapDAE model learns much better representation compared to the DAE. The conventional DAE tends to retrieve based on the appearance of the images, while our LapDAF focuses more on the context/semantic information, e.g., the airplane in the first row. We attribute this to the multiple scale corruptions in the Laplacian domain. Overall, these results together with the above evaluation on the MNIST dataset can be considered as proof of concept that the proposed LapDAE is capable of capturing both low-level and high-level context information.
image
4.4 EVALUATION ON IMAGENET
In this section, we aim to investigate the representation learning capability of the proposed LapDAE on large-scale data. To achieve this goal, we perform evaluations on the ImageNet (Deng et al., 2009) dataset. Specifically, we use the training set without labels from ImageNet (Deng et al., 2009) to train our LapDAE model. The training set includes 1.2 million images covering 1,000 categories. Each image is first resized to 256 x 256 and randomly cropped to 227. Horizontal flipping is also applied for data augmentation.

---

Under review as a conference paper at ICLR 2020
Supervised vervviied
LapDAE
DAE
Figure 5: The learned convolutional filters (kernels) from the first layer of AlexNet (convl) trained on the ImageNet dataset. Left: result with fully-supervision from the labeled data; Middle and Right: the filters learned in an unsupervised manner by DAE and our LapDAE respectively.
Table 1: Top-1 accuracy on ImageNet and Places classification with linear classifier. The results are reported based on the ImageNet/ Places validation set and all the listed methods except ImageNet / Pllaces labels, Domain Adapt are pre-trained on ImageNet without ground truth labels.
Method
Conv1
Conv2
Conv3
Conv4
Conv5
ImageNet/ / Places labels
19.3 / 22.1
36.3 / 35.1
44.2/ / 40.2
48.3 / 43.3
50.5 / 44.6
Random Gaussian
11.6/ 15.7
17.1 20.3
16.9 / 19.8 16.3/ 19.1
14.1 / 17.5
Random rescaled (Krähenbühl et al., 2015)
17.5 / 21.4
23.0 /2.2
24.5 27.1
23.2/26.1
20.6 / 24.0
Context (Doersch et al., 2015)
16.2/ 19.7 23.3/26.
30.2/3199
31.7//2.7
/9.6/30.9
Context Encoder (Pathak et al., 2016)
14.1 18.2
20.7/23.2
21.0//34 19.8/2199
15.5 18.4
Colorization (Zhang et al., 2016)
12.5/16.0
24.5/25.7
30.4/29.6
31.5/30.3
30.3/
Jigsaw (Noroozi & Favaro, 2016)
18.2/23.0
28.8/319
34.0/35.0
33.9/34.
27.1/29.3
BiGAN (Donahue et al., 2016)
17.7/22.0
24.5/28.7
31.0/31.8
29.9/31.3
28.0/
Split-Brain (Zhang et al., 2017)
17.7/21.3
29.3/30.7
35.4/34.0
35.2/34.1
32.8/32.5
Counting (Noroozi et al., 2017)
18.0/23.3
30.6/33.9
34.3/36.3
32.5/34.7
25.7
RotNet (Gidaris et al., 2018)
18.8/215
31.7/31.0
38.7/35.1
38.2/34.6
36.5/33.
Domain Adapt (Ren & Lec, 2018)
16.5/
27.0/
30.5/
30.1/
26.5/
Instance (Wu et al., 2018)
26.5 16.8/1888
31.8/319
34.1/34.5
35.6/ 33.6
AND (Huang et al., 2019)
15.6/
27.0/
35.9/
39.7/
37.9/
AET-project (Zhang et al., 2019)
19.2
32.8
0.88/22
32.9
30.6/37.1
39.7 36.2
37.7 34.7
DAE
12.5 / 15.9
18.5/22.6
21.8/24.2
20.4/22..
14.8 18.1
DAE + Trans
17.6/
31.8/317
39.2/35.7
37.4/34.1
34.5 32.8
(Ours) LapDAE
18.4/2100
27.4/0.9
29.9/3166
22.7 26.1 /7.0/29.2
(Ours) LapDAE + Trans
19.3 22.2 33.2 33.8
43.2/38.. 41.1 / 37.3
39.6 36.1
Convl learned filter visualization: In Fig. 5, we show the comparison for the learned filters from the first layer (i.e., convl) of AlexNet between our approach and the fully-supervised ones. In the supervised version (the left panel), both color blobs and edge filters are learned. We can see that although not as sharp as those filters learned by the supervised setup for some blobs, our approach (the right panel) learns quite good filters including edges along different directions, edges with different frequencies, color contrast along different directions, etc., similar to the supervised ones. Comparing with conventional DAE (the middle panel), the learned representations much better.
Controlled classification: Here we quantitatively evaluate our learned representations on the Im- ageNet classification task (Deng al., 2009). Following the experimental settings in (Zhang al., 2016), we freeze the pre-trained weights of our model and train a linear classifier on the top of each conv layer, to perform the 1000-category classification task. In order have approximately the same dimensions across different layers, the feature maps of each layer is interpolated to have around 9000 elements. Table shows the evaluation results. Several state-of-the-ar representation learning methods (Zhang et al., 2016; 2017; Pathak et al., 2016; Doersch et al., 2015; Noroozi & Favaro, 2016; Donahue et al., 2016; Noroozi al., 2017: Gidaris et al., 2018; Ren & Lee, 2018; Wu et al., 2018; Zhang et al., 2019 Huang et al., 2019) are included for comparison. Since the proposed approach can be easily integrated into other representation learning frameworks, we also present the performance of our LapDAE combined with the task of predicting transformations (LapDAE+ Trans). Specifically, we base on the AET framework (Zhang et al., 2019) while reasoning the transformation between the original image and transformed one corrupted by LapDAF.
self-supervised

---

Under review as a conference paper at ICLR 2020
Table Comparison with state-of-the-ar representation learning methods on PASCAL VOC vision tasks of classification, detection on 2007, and semantic segmentation on 2012. For classification we also compare a setup that fixes the layers before conv5 and only train the fc6-8. The learned weights from unlabeled ImageNet are transferred for the new tasks except the ImageNei labels.
Classification
Detection Segmentation (9mAP)
(%mAP)
(%mloU)
Method
fc6-8
all
all
all
ImageNet labels
78.9
79.9
56.8
48.0
Random Gaussian
53.3
43.4
19.8
Random rescaled (Krähenbühl et al., 2015)
39.2
56.6
45.6
32.6
Context (Doersch et al., 2015)
55.3
45.7
Context Encoder (Pathak et al., 2016)
34.6
56.5
44.5
29.7
Colorization (Zhang et al., 2016)
61.5
65.5
46.9
35,6
Counting (Noroozi al., 2017)
67.7
51.4
36.6
GAN (Goodfellow et al., 2014)
40.5
56.4
BiGAN (Donahue al., 2016)
52.3
60.1
46.9
34.9
RotNet (Gidaris et al., 2018)
70.9
73.0
54.4
39.1
ArT-project (Zhang et al., 2019)
70.5
73.1
54.2
39.3
DAE
37.0
54.6
43.4
29.1
DAE + Trans
66.7
70.1
51.0
36.8
(Ours) LapDAE
50.6
59.0
45.6
38.3
(Ours) LapDAE Trans
71.4
74.2
55.2
41.1
From the results, we observe that the proposed method with the Laplacian pyramid largely improves the performance compared to its counterpart without the Laplacian pyramid, especially for the lower convolutional layers. This is consistent with the above visualization and analysis of the conv layer, where the filter kernels have more representative power the proposed LapDAE. When incorporat- ing with the transformation prediction task, we can see that the performance is further boosted by large margin. Even when compared to the AET-project method, approach performs much better, which again validates the effectiveness of the proposed LapDAE.
4.5 TRANSFER LEARNING ON PASCAL VOC AND PLACES
Next we perform a transfer learning evaluation on the PASCAL VOC dataset (Everingham et al., 2010), for the tasks of classification, detection on VOC 2007, and semantic segmentation on VOC 2012. The learned weights of model trained on ImageNet are transferred for the evaluation. We then fine-tune the model on the PASCAL VOC trainval set and test on the test set. Note that we do not apply any "magic" techniques such as weights rescaling (Krähenbühl et al., 2015). For the classification task, evaluation, while for the semantic segmentation tasks we use the publicly available frameworks FCN (Long et al., 2015) and follow the same setups for other state-of-the-an methods e-g-, (Doersch et al., 2015; Zhang et al., 2016). The results Table show that the learned visual features by the proposed LapDAE exhibit good performances when transferred other vision datasets tasks, and performs favorably against the other state-of-the-art methods. For the classification task of PASCAL VOC, the proposed LapDAE outperforms all other generic unsupervised learning methods like DAE and GAN. More impressively, its segmentation performance on PASCAL VOC surpasses most of the representation learning methods including those self superviedd learning approaches with specifi- cally designed pretext tasks. Comparison between the proposed framework and its counterpart DAE suggests that the improvement partly due the Laplacian pyramid. When incorporating the transformation prediction task (LapDAE+ Trans), approach on PASCAL VOC transfer learning further improved and achieves state-of-theaat performance.
standard AlexNet
the same network architecture in the ImageNet
In addition to PASCAL VOC, we further evaluate the representation learning capability by transfer learning on the Places dataset (Zhou et al., 2014). We use the same experimental settings as the ImageNet experiment and the result is shown in Table 1. Differing from the ImageNet experiments, the classifier trained on top of different layers 205-way logistic regression layer and which is then trained with the Places labels. From the result can that, the proposed LapDAE performs
8

---

Under review as a conference paper at ICLR 2020
better than its counterparts and outperforms other methods when incorporating with transformation prediction.
5 CONCLUSION
In this paper we introduced a novel type of denoising autoencoder for unsupervised representation learning. In contrast conventional DAE, the corrupted data input the proposed DAE is produced with the aid of a Laplacian pyramid. By adding corruptions to randomly chosen levels in Laplacian pyramid, the resulting data corruptions spans multiple scales across the original data space. From this, the network is forced to learn to represent underlying data structures across multiple scales. The proposed learning framework ensures that the agent learns better representations when compared to conventional DAE and other self-supervised learning methods. While in this paper we showcase the effectiveness of the proposed method for learning transferable representations for vision tasks, would be interesting to see how it performs with other types of data. It is also interesting to explore the possibility of extending the proposed idea to other representations like wavelet, SIFT, and other deep features. The core idea of performing both local and non-local learning that is consistent with the hierarchical nature of ConyNets, is the Laplacian pyramid space, which we believe a promising direction for future research.
REFERENCES
Yoshua Bengio, Pascal Lamblin, Dan Popovici, and Hugo Larochelle. Greedy layer-wise training of deep networks. In NeurIPS, 2007.
Yoshua Bengio, Aaron Courville, and Pascal Vincent. Representation learning: A review and new perspectives. IEEE TPAMI, 3798) 1828, 2013.
Andreja Bubić, D Yves von Cramon, and Ricarda I Schubotz. Prediction, cogmition and the brain. Frontiers Human Neuroscience, 4(25), 2010.
Peter Burt and Edward Adelson. The laplacian pyramid as a compact image code. IEEE Transac- fions communications, 31(4):532-540, 1983.
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. In CVPR, 2009.
Carl Doersch, Abhinav Gupta, and Alexei A Efros. Unsupervised visual representation learning by context prediction. In ICCV, 2015.
Jeff Donahue, Philipp Krähenbühl, and Trevor Darrell. Adversarial feature learning, arXiv preprint arXiv:1605.09782. 2016.
Mark Everingham, Luc Van Gool, Christopher KI Williams, John Winn, and Andrew Zisserman. The pascal visual object classes (voc) challenge. International journal of computer vision, 88(2): 303-338, 2010.
Spyros Gidaris, Praveer Singh, and Nikos Komodakis. Unsupervised representation learning by predicting image rotations. arXiv preprint arXiv: 1803.07788. 2018
Ian Goodfellow, Jean Pouget-Abadie, Mchdi Mirza, Bing Xu, David Warde-Farley. Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial nets. In NeurIPS, 2014.
Geoffrey E Hinton and Ruslan R Salakhutdinov. Reducing the dimensionality of data with neural networks. science, 31 313(578)):504-500 2006.
Geoffrey E Hinton and Terrence Sejnowski. Learning and releaming in boltzmann machines.
Parallel distributed processing: Explorations in the microstructure of cognition, 1(282-317:2, 1986.
Jiabo Huang, Qi Dong, Shaogang Gong, and Xiatian Zhu. Unsupervised deep learning by neigh- bourhood discovery. In ICML., 2019.

---

Under review as a conference paper at ICLR 2020
Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980, 2014.
Diederik P Kingma and Max Welling. arXiv:1312.6114, 2013.
Auto-encoding
variational bayes.
arXiv preprint
Philipp Krähenbühl, Carl Doersch, Jeff Donahue, and Trevor Darrell. Data-dependent initializations of convolutional neural networks. arXiv preprint arXiv:/511.06830 2015.
Alex Krizhevsky. Learning multiple layers of features from tiny images. Technical report, Citeseer, 2009.
Alex Krizhevsky, lla Sutskever, and Geoffrey E. Hinton. Imagenet classification with deep convo- lutional neural networks. In NeurlPS, 2012.
Yann LeCun, Léon Botton, Yoshua Bengio, Patrick Haffner, et al. Gradient-bssed learning applied to document recognition. Proceedings of the IEFE, 86(1 86(11):2278-2324 1998.
Jonathan Long, Evan Shelhamer, and Trevor Darrell. segmentation. In CVPR, 2015.
Fully convolutional networks for semantic
Laurens van der Maaten and Geoffrey Hinton. Visualizing data using t-sne. Journal of machine learning research, 9Nov))2579-2605, 2008.
Ishan Misra, C Lawrence Zitnick, and Martial Hebert. Shuffle and learn: unsupervised learning using temporal order verification. In ECCV, 2016.
Mehdi Noroozi and Paolo Favaro. Unsupervised learning of visual representations by solving jigsaw puzzles. ECCV, 2016.
Mehdi Noroozi, Hamed Pirsiavash, and Paolo Favaro. Representation learning by learning to count. arXiv preprint arXiv: /t1.//088..7555 2017.
Deepak Pathak, Philipp Krahenbuhl, Jeff Donahue, Trevor Darrell, and Alexci A Efros. Context encoders: Feature learning by inpainting. In CVPR, 2016.
Deepak Pathak, Ross Girshick, Piotr Dollár, Trevor Darrell, and Bharath Hariharan. Learning fea- tures by watching objects move. In CVPR, 2017.
Christopher Poultney, Sumit Chopra, Yann L Cun, et al. Efficient learning of sparse representations with an energy-based model. In NeurlPS, 2007.
Zhongzheng Ren and Yong Jae Lee. Cross-domain self-supervised multi-tase task feature learning using synthetic imagery. CVPR, 2018
Ruslan Salakhutdinov and Hugo Larochelle. Efficient learning of deep boltzmann machines. In Proceedings of the Thirteenth International Conference on Artificial Intelligence and Statistics, 2010.
Pascal Vincent, Hugo Larochelle, Isabelle Lajoie, Yoshua Bengio, and Picrre-Antoine Manzagol. Stacked denoising autoencoders: Learning useful representations in a deep network with local denoising criterion. Journal of Machine Learning Research, 11(Dec):3371-33 34088 2010.
Xiaolong Wang and Abhinav Gupta. Unsupervised learning of visual representations using videos.
arXiv preprint arXiv: 2015.
Xiaolong Wang, Ross Girshick, Abhinav Gupta, and Kaiming He. Non-local neural networks. In CVPR, 2018.
Zhirong Wu, Yuanjun Xiong, Stella X Yu, and Dahua Lin. Unsupervised feature learning via non- parametric instance discrimination. CVPR, 2018.
Matthew Zeiler and Rob Fergus. Visualizing and understanding convolutional networks. In ECCV, 2014.
10

---

Under review as a conference paper at ICLR 2020
Liheng Zhang, Guo-Jun Qi, Liqiang Wang, and Jiebo Luo. Act vs. aed: Unsupervised represetttton learning by auto-encoding transformations rather than data. CVPR, 2019.
Richard Zhang, Phillip Isola, and Alexei A Efros. Colorful image colorization. In ECCV, 2016.
Richard Zhang, Phillip Isola, and Alexei A Efros. Split-brain autoencoders: Unsupervised learning by cross-channel prediction. CVPR, 2017.
Bolei Zhou, Agata Lapedriza, Jianxiong Xiao, Antonio Torralba, and Aude Oliva. Learning deep features for scene recognition using places database. In NeurlPS, 2014.

---

Under review as a conference paper at ICLR 2020
APPENDIX
Here we present more qualitative results to show the performance the proposed method, addi- tion those shown in Fig. and Fig. in the main paper. Specifically, we first show the performance on MNIST dataset in Fig. 6. Additional qualitative reconstruction performance on the CIFAR-IU 10 dataset is presented in Fig. 7. We also show more results on the image retrieval task that is presented in Fig. 4 (b). The additional results on image retrieval is shown in Fig. 8. The additional qualitative results again validate the effectiveness of the proposed approach on sulf-supervised representation learning.
12

---

Under review as a conference paper at ICLR 2020
2
3
4
5
9
8
7
6
5
3
2
6
2
3
4
5
6
8
9
8
7
6
5
1
3
7
/
O
2
3
5
4
9:
g
7
Le
5
4
3
2
Original raw images
2
3
4
5
8
9
7
6
5
4
O
6
1
2
3
4
5
6
7
9
3
7
6
5
1
3
7
/
O
0
2
3
5
6
7
7
f
7
7
U
5
4
3
7
1
Reconstruction by conventional DAE
0
1
2
3
4
5
8
9
8
7
6
5
4
3
2
O
6
2
3
4
5
6
7
8
O
8
7
6
5
1
3
Z
O
0
2
3
7
5
6
-
7
9
f
8
7
Le
5
4
3
2
Reconstruction by the proposed LapDAE
Figure 6: Additional results illustration of the reconstruction performance using the MNIST
dataset. The original raw input images are randomly selected from the test set and are shown the first part, while the second and third parts show the reconstructed results from the conventional DAE and the proposed LapDAE, respectively.
13

---

Under review as a conference paper at ICLR 2020
sirplane
automobile
bird
cat
deer
dog
fing
horse
ship
truck
Sertian
Original images
Reconstruction conventional DAE
Reconstruction by the proposed LapDAE
Figure 7: Additional reconstruction results comparison using the CIFAR-10 dataset, in which ex- amples from each category in the test set are randomly selected for visualization. The original raw input images are shown in the first part, while the second and third parts show the reconstructed results from the conventional DAE and the proposed LapDAE, respectively.
14

---

Under review as a conference paper at ICLR 2020
cat
ship
automobile
frog
airplane
deer
bird
automobile
bird
deer
dog
horse
ship
truck
Query
DAE
LapDAE
Figure 8: Additional image retrieval results using the CIFAR-10 dataset by nearest neighbor. Given the query on the left, the top-5 (from left to right) retrieved results of DAE (middle) and LapDAE (right) presented, in which red marked ones indicate wrong category.
15


Reviews:
[{"review_id": "rkxhzEjk5B", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper proposes a denoising auto-encoder where the input image is corrupted by adding noises to its Laplacian pyramid representation. Then a DAE is trained to predict the original data and learn a good representation of the data. By corrupting the Laplacian representation, which is multi-scale, the corruption of the image is not local and thus more robust representations are learned.\n\nI personally like this idea. However, it seems a simple extension of the classical DAE. \n\n1. Is it possible to generalize this idea to other representations of the images such as wavelets or sift, or the representations learned by other neural networks? It seems that you can add corruptions to any image representations as long as you can reconstruct the image from the representation. Here, you can reconstruct from  Laplacian pyramid representation.\n\n2. As an empirical work, the experiments in this work is rather small-scale, using only CIFAR10 and MNIST. That seems far from sufficient. ", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I carefully checked the derivations and theory."}, "extra_text": {}}, {"review_id": "S1g7TmTQFr", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Denoising auto-encoder (DAE)  is a representation learning framework proposes in [1], which uses noisy input to reconstruct the clean “repaired” input. In the conventional DAE, the noise is directly added to the input space. This paper introduces a novel type of noise, which corrupts the Laplacian pyramid representation, and uses it to train a DAE. The effect of the proposed perturbation is to allow larger scale and semantically meaningful corruption, which may help to learn more transferable representation. Several experiments are conducted showing the effectiveness of the method\n1) On MNIST, LapDAE provides better reconstruction images\n2) On Cifar-10, LapDAE provides better image retrieval results\n3) On Imagenet, combining with the transformation technique in [2], LapDAE achieve state-of-the-art result\n4) On Pascal VOC,  combining with the transformation technique, LapDAE achieve state-of-the-art result on transfer learning\nOverall, I find the idea natural and simple (simplicity of an approach is a good quality to me). At the same time, I also find the contribution a bit limited. The following are further comments and questions.\n\n* From the experiments, standard DAE are harder to train comparing to the proposed LapDAE. In my opinion, this suggests that the local noise are harder to remove comparing to the Laplacian noise. It would be interesting to perform the following experiment to further understand the difference between DAE and LapDAE: train a DAE and LapDAE then \na) Apply LapDAE to reconstruct an image where the random noise is added on the input space (as the standard DAE setting).\nb) Apply DAE to reconstruct an image where the noise is added to the Laplacian pyramid representation.\n\n* How does the corrupted set C been selected? Moreover, how does the selected layer in the Laplacian pyramid effects the performance? For example, is there a difference in terms of performance of the model when training on LPS4 versus LPS8?\n\n* The transformation technique seems to be very helpful for the performance, is it possible to combine it with other benchmark methods like RotNet or Counting?\n\n[1]  Vincent et al. 2010, Stacked denoising autoencoders: Learning useful representations in a deep network with a local denoising criterion \n[2] Zhang et al. 2019, AET vs. AED: Unsupervised Representation Learning by Auto-Encoding Transformations rather than Data", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_checking_correctness_of_derivations_and_theory": "N/A", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly."}, "extra_text": {}}, {"review_id": "HJgRL-32KH", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper provides a novel approach to learning useful representations with deep learning models. They formulate an entirely unsupervised framework based off autoencoders to accomplish this task. Their data differs from previous works by upsampling a noise-corrupted downsampling of the original inputs. The autoencoder is then trained to reconstruct the original image from this new data. Their experiments demonstrate that using this Laplacian pyramid scheme to generate noisy data leads to an autoencoder that learns better representations compared to a standard DAE. \nOverall, the work in this paper has the potential to be a contribution to ICLR but lacks experimental completeness and clarity. Moreover, the main contribution is a better denoising autoencoder, but in the grand scheme of representation learning, it is unclear how broad of a contribution this is. I would be willing to change my score, upon addressing the following details:\n•\tAs mentioned above, the results as presented provide a better DAE. The paper would be much stronger if it also provided comparisons to more recent models in representation learning closer to state of the art. For instance, they choose BiGAN as a model for comparison, but these were developed over three years ago and are now outdated in favor of better GAN models.\n•\tThe paper would be further strengthened with additional experiments of their representations being qualitatively better than previous models. Their example of image retrieval via nearest neighbor is quite limited when compared to the wealth of tasks GAN models can accomplish.\n•\tIn general, when presenting qualitative results (Figures 3,4, and 5), additional examples should be put into supplementary materials so as to demonstrate the paper did not cherry pick.\n•\tThe presentation of the quantitative results is peculiar. The paper chooses to combine their Laplacian DAE with the AET framework. As such, all the results tables should include numbers for AET alone and the conventional DAE with AET.\n•\tLastly, the paper needs a revision for ease of readability, as there are a significant number of grammatical errors that make it hard to read at times. \nWhile the transfer learning experiments seem incomplete to me, that is not my area of expertise and I cannot judge how convincing that setup is as well as other reviewers. \n", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I carefully checked the derivations and theory."}, "extra_text": {}}]

Author rebuttal:

Meta-review: The main idea proposed by the work is interesting. The reviewers had several concerns about applicability and the extent of the empirical work. The authors responded to all the comments, added more experiments, and as reviewer 2 noted, the method is interesting because of its ability to handle local noise. Despite the author's helpful responses, the ratings were not increased, and it is still hard to assess the exact extent of how the proposed approach improves over state of the art.   Because some concerns remained, and due to a large number of stronger papers, this paper was not accepted at this time.
Decision: Reject