Title: Deep unsupervised learning through spatial contrasting
Authors: Elad Hoffer
Itay Hubara
Nir Ailon
Venue: ICLR 2017
Keywords: Unsupervised Learning
Deep learning
Computer vision

Abstract: Convolutional networks have marked their place over the last few years as the
best performing model for various visual tasks. They are, however, most suited
for supervised learning from large amounts of labeled data. Previous attempts
have been made to use unlabeled data to improve model performance by applying
unsupervised techniques. These attempts require different architectures and training methods.
In this work we present a novel approach for unsupervised training
of Convolutional networks that is based on contrasting between spatial regions
within images.  This criterion can be employed within conventional neural net-
works and trained using standard techniques such as SGD and back-propagation,
thus complementing supervised methods.


Review discussion:
Did the authors experiment at all with the patch size? Seems like this could be a valuable addition to the understanding of the model (including the two extremes!)
Thank you for your question. Patch size was indeed found to be important, where datasets dependent on more global features (such as MNIST, where small patches might be devoid of any information) needed a bigger patch, while other datasets used a smaller one. We thank you for the suggestion and will include information regarding patch sizes in the next version of this paper.
What conceptual (or empirical) advantages does the proposed loss have over Doersch et al. ICCV 2015 (cited) and
"Learning visual groups from co-occurrences in space and time" Isola et al. ICLR 2016 workshop (not cited)?
It seems that  Doersch et al. make use of a stronger supervision (exact spatial relations) also in a
self-supervised manner.
Thank you for your question, these two works indeed share conceptual similarity to ours (and missing citing will be amended in next version). We do however note our use of Euclidean distance comparison as a novel unsupervised learning criterion, which allow us to use samples from a batch without explicit supervision of exact spatial location.
In Table 2, the results of exemplar convnets (Dosovitskiy 2015) is for unsupervised learning (the network is not finetuned with labeled samples, and only a classifier is trained upon the features), but the result of the proposed methods are is not (the whole network is finetuned). Ideally the two methods should be compared with the same setting.
In general, the whole networks are finetuned using labeled examples in this paper, so it is not unsupervised learning. Can you show unsupervised learning results, where a classifier is trained using the pretrained neural network features?
Which layers did you choose for the features? How sensitive is the performance to this hyperparameter?
The paper does not describe the network architectures used for each dataset. Can you elaborate this?
Thank you for your question, you are correct that exemplar convnets setting didn’t include any fine tuning and we will note this and include our results without finetuning in next version. We do however, note that it included no additional supervision beyond what use by Dosovitskiy, since both methods used the same set of labeled samples.
The architecture and layers were mistakenly omitted from this version (they appear in https://arxiv.org/abs/1610.00243) and will also be included in the next version.
The choice for the location of the layer used spatial contrasting was found to correspond to the depths commonly used for networks trained on the discussed datasets. The performance is sensitive in a similar fashion to the choice of a network depth.
This paper presents a novel way to do unsupervised pretraining in a deep convolutional network setting (though likely applicable to fully-connected nets as well). The method is that of ‘spatial constrasting’, i.e. of building triplets from patches of input images and learning a presentation that assigns a high score for patches coming from the same image and a low score for patches from diferent images. The method is simple enough that I am surprised that no-one has tried this before (at least according to the previous work in the submission). Here are some comments:
The usage of P(f_i^1 | f_i^2) in Section 4.1 is a bit odd. May be worth defining mathematically what kind of probability the authors are talking about, or just taking that part out (“probability” can be replaced with another word).
I would like to know more about how the method is using the “batch statistics” (end of Section 4.2) by sampling from it, unless the authors simply mean that the just sample from all the possible triples in their batch.
Shouldn’t the number of patches sampled in Algorithm 1 be a hyper-parameter rather than just be 1? Have the authors tried any other value?
I think there are some missing details in the paper, like the patch size or whether the authors have played with it at all (I think this is an important hyper-parameter).
The STL results are quite impressive, but CIFAR-10 maybe not so much. For CIFAR I’d expect that one can try to pre-train on, say, Imagenet + CIFAR to build a better representation. Have the authors considered this?
All in all, this is an interesting piece of work with some obvious applications, and it seems relatively straightforward to implemenent and try. I think I would’ve liked more understanding of what the spatial contrasting actually learns, more empirical studies on the effects of various parameter choices (e.g., patch size) and more attempts at beating the state of the art (e.g. CIFAR).
Thank you for the review, we appreciate your remarks and incorporated them into latest revision.
We will clarify our notation in section 4.1, thank you for indicating this is not well defined.
The batch statistics we were referring to is the fact that each image in a batch serves as contrasting noise to other samples. This can be viewed as a simple sampling from the space of “patch features”. We will try to make that more clear in our revision.
Missing implementation details that you mentioned will be added to the paper as well as code to reproduce our results.
We did not try to learn from Imagenet data, as this will not allow us to compare with previous works on CIFAR, we thank the reviewer for this suggestion.
We will try to shed some more light into what spatial contrasting learns and the effect of parameter choices.
The proposed self supervised loss is formulated using a Siamese architecture and encourages patches from the same image to lie closer in feature space than a contrasting patch taken from a different, random image. The loss is very similar in spirit to that of Doersch et al. ICCV 2015 and Isola et al. ICLR 2016 workshop. It seems that the proposed loss is actually a simplified version of Doersch et al. ICCV 2015 in that it does not make use of the spatial offset, a freely available self supervised signal in natural images. Intuitively, it seems that the self-supervised problem posed by this method is strictly simpler, and therefore less powerful, than that of the aforementioned work. I would like to see more discussion on the comparison of these two approaches.
Nevertheless the proposed method seems to be effective in achieving good empirical results using this simple loss.
Though more implementation details should be provided, such as the effect of patch size, overlap between sampled patches, and any other important measures taken to avoid trivial solutions.
Thank you for the review, we appreciate your remarks and incorporated them into latest revision.
We would like to emphasize some difference between our work and that of Doersch 2015.
Although both works used patches from an image to learn unsupervised learning, the method is quite different.
Whereas Doersch used a classification criterion over the spatial location of each patch within a single image, our work is concerned comparing patches from several images to each other using a distance ratio loss. We claim that this encourage discriminability between images (which we feel to be important aspect of feature learning), and is not an explicit goal in the work of Doersch.
Thus, we feel that both approaches are viable methods for unsupervised learning using samples from spatial data. We will include this distinction in our revision as well as the implementation details that you indicated missing.
This paper proposes an unsupervised training objective based on patch contrasting for visual representation learning using deep neural networks.
In particular, the feature representations of the patches from the same image are encouraged to be closer than the those from different images. The distance ratios of positive training pairs are optimized. The proposed method are empirically shown to be effective as an initialization method for supervised training.
Strengths:
- The training objective is reasonable.
In particular, high-level features show translation invariance.
- The proposed methods are effective for initializing neural networks for supervised training on several datasets.
Weaknesses:
- The methods are technically similar to the “exemplar network” (Dosovitskiy 2015). Cropping patches from a single image can be taken as a type of data augmentation, which is comparable to the data augmentation of positive sample (the exemplar) in (Dosovitskiy 2015).
- The paper is experimentally misleading.
The results reported in this paper are based on fine-tuning the whole network with supervision.
However, in Table 2, the results of exemplar convnets (Dosovitskiy 2015) is from unsupervised feature learning (the network is not finetuned with labeled samples, and only a classifier is trained upon the features).
Therefore, the comparison is not fair. I suspect that exemplar convnets (Dosovitskiy 2015) would achieve similar improvements from fine-tuning;
so, without such comparisons (head-to-head comparison with and without fine-tuning based on the same architecture except for the loss), the experimental results are not fully convincing.
Regarding the comparison to “What-where” autoencoder (Zhao et al, 2015), it will be interesting to compare against it in large-scale settings, as shown by Zhang et al, ICML 2016 (Augmenting Supervised Neural Networks with Unsupervised Objectives for Large-Scale Image Classification). Training an AlexNet is not very time-consuming with latest (e.g., TITAN-X level) GPUs.
The proposed method seems useful only for natural images where different patches from the same image can be similar to each other.
Thank you for the review, we appreciate your remarks and incorporated them into latest revision.
As you pointed out, this work shares motivation with that of Dosovitskiy, of learning unsupervised representations of images using the spatial invariance properties. We do how note that this work suggests a novel method of achieving that goal, by explicitly learning features that correspond to the spatial invariance assumptions. We claim that this method may hold several advantages over that of Dosovitsky, such as removing the need for surrogate classes of augmented images, and being able to apply it in parallel to standard classification objective. We thus feel that both are viable methods for unsupervised learning on image data.
We understand your objection to the comparison without fine-tuning the exemplar network model, although we note that both models used exactly the same data in both cases. We will include a clarifying remark over this issue.
We thank you for your suggestion for large-scale comparison of this work, and will try to incorporate it into later revision.

Meta-review:
role: PC
data: comment: The paper proposes a formulation for unsupervised learning of ConvNets based on the distance between patches sampled from the same and different images. The novelty of the method is rather limited as it's similar to [Doersch et al. 2015] and [Dosovitsky et al. 2015]. The evaluation is only performed on the small datasets, which limits the potential impact of the contribution.

Decision: Reject