PatchFormer: A neural architecture for self-supervised representation learning on images
Authors: Aravind Srinivas
Pieter Abbeel
Venue:  (2020)
Keywords: Unsupervised Learning
Representation Learning
Transformers

Abstract: Learning rich representations from predictive learning without labels has been a longstanding challenge in the field of machine learning. Generative pre-training has so far not been as successful as contrastive methods in modeling representations of raw images. In this paper, we propose a neural architecture for self-supervised representation learning on raw images called the PatchFormer which learns to model spatial dependencies across patches in a raw image. Our method learns to model the conditional probability distribution of missing patches given the context of surrounding patches. We evaluate the utility of the learned representations by fine-tuning the pre-trained model on low data-regime classification tasks. Specifically, we benchmark our model on semi-supervised ImageNet classification which has become a popular benchmark recently for semi-supervised and self-supervised learning methods. Our model is able to achieve 30.3% and 65.5% top-1 accuracies when trained only using 1% and 10% of the labels on ImageNet showing the promise for generative pre-training methods.

Under review as a conference paper at ICLR 2020
PATCHFORMER:
A NEURAL ARCHITECTURE FOR SELF-SUPERVISED REPRESENTATION LEARNING ON IMAGES
Anonymous authors Paper under double-blind review
ABSTRACT
Learning rich representations from predictive learning without labels has been a longstanding challenge in the field of machine learning. Generative pre-training has so far not been as successful as contrastive methods in modeling represen- tations of raw images. In this paper, we propose a neural architecture for self- supervised representation learning on raw images called the PatchFormer which learns to model spatial dependencies across patches in a raw image. Our method learns to model the conditional probability distribution of missing patches given the context of surrounding patches. We evaluate the utility of the learned repre- sentations by fine-tuning the pre-trained model low data-regime classification tasks. Specifically, we benchmark our model on semi-supervised ImageNet classi- fication which has become popular benchmark recently for semi-supervised and self-supervised learning methods. Our model is able to achieve 30.3% and 65.5% top-1 1 accuracies when trained only using 1% and 10% of the labels ImageNet showing the promise for generative pre-traie pre-training methods.
INTRODUCTION
Deep neural networks are capable of learning rich abstract representations from raw high dimen- sional data in an end-to-end fashion (LeCun et al. 2015). A big weakness of these neural networks the reliance on abundant labeled datasets. Self supervide and unsupervised representation learn- ing approaches have been proposed to address this problem (Bengio al., 2007). It is still an open problem the field to figure out how take advantage of large unlabeled datasets, use them for learning rich representations and improving the data-efficiency of supervised learning systems.
classic example of successful unsupervised learning of rich representations is word2vec (Mikolov et al., (013)) where the authors showed that distributed vector representations of words could be learned by contrastively predicting the neighboring words given surrounding words. The shift from word embeddings to sequence embeddings in recent times began when (Dai & Lel 2015) showed that pre-trained sequence to sequence autoencoders on text corpora could be useful for a number of downstream tasks such text classification and sentiment analysis. Followed by this, it was shown in (Peters et al., 2018)) that language modeling is useful in providing deep contextual sentence em- beddings that could be fine-tuned
number of natural language understanding tasks. (Howard & Ruder 2018) is another example of such success. In more recent times, the transformer (Vaswani al. 2017) has emerged as powerful architecture to model complex dependencies across a long sequence using global self-attention. OpenAl Generative Pre-Train Training (GPT) (Radford et al. 2018) showed that training large Transformer models on BooksCorpus could lead rich and useful rep- resentations that could be fine-tuned on variety of downstream tasks covering language under- standing, commonsense reasoning and question-answering pre-training was achieved by BERT (Devlin et al. 2018)) where the assumption for using causal lan- guage modeling was pointed out as unnecessary and it was shown that training deep transformers in si-directional fashion to perform the objective of masked language modeling and next sentence prediction could lead to rich and useful representations covering a wide span of natural language understanding downstream tasks.
The biggest success in unsupervised

---

Under review as a conference paper at ICLR 2020
Therefore, it is useful to address the following question: How do we translate the successes of masked language modeling and deep transformers to images? Unlike language which is a layer of abstraction to be able understand the world and communicate thoughts, images are raw sensory observations. It is therefore much harder to model the relationship across pixels both spatially and temporally simply because the dimensionality is much higher.
Let's first look at the question of whether generative pre-training is well suited for images or not. There is belief that generative approaches are more suited to abstract inputs such as language wordpieces but not for less abstract entities like pixels or audio waveform bits (van den Oord et al.) 2018 Hjelm al. 2018 Bachman al. 2019] Trinh al. 2019). While it may as well turn out to be true, it is useful to investigate how far we could push generative approaches for pre- pre-training even on domains they are not well suited for, such as images.
A successful example of such an approach is the adversarial method BiGAN (Donahue et al. 2016 Donahue & Simonyan, 2019). While BiGAN (and BigBiGAN) are meant for learning useful high- level representations of raw images, they still retain the generative modeling aspect of unsupervised learning by learning to jointly model an encoder and a generator using the generative adversarial
loss.
On the other hand, there has been incredible progress in recent years in generative modeling of raw pixels and audio waveforms using maximum likelihood. Beginning with (Oord al. 2016b), we have seen successes in generating diverse images by modeling the conditional distribution of pix- els given context of neighboring pixels. WaveNet (Oord et al., 2016a) is example of successful deployment of such techniques for modeling the distribution of raw audio waveforms when condi- tioned on text. (Kalchbrenner al. 2017) adopt similar technique for generating future frames of a video conditioned on the past. More recently, (Child et al. (219) have pushed on using strided self-attention achieve high-quality unconditional samples of ImageNet building upon successes of (Parmar et al. 2018) and (Menick & Kalchbrenner 2018).
Therefore, it is very reasonable to ask ourselves the following question: If generative models can work on such high dimensional data, is it necessarily the case that they would be ill-suited from a representation learning perspective? If no, how do we leverage these successes for representation learning? Further, how do we take inspiration from the big representation learning successes in natural language processing (Devlin et al.) 2018) and the generative modeling successes for images and audio and design representation learning approach for images?
As far as representation learning on images goes, the state-of-the-ar systems at the moment are contrastive methods. Specifically, Contrastive Predictive Coding (CPC) (van den Oord et al.) 2018) which learns to contrastively predict the future given the past by sampling negatives across and be- tween sequences has been shown to be universally powerful representation learning approach for multiple modalities (audio, images, text, control). (Hénaff' et al. 2019) and (Bachman et al. 2019) achieve impressive linear classifier probe metrics for their representations that were trained con- trastively to maximize mutual information across views and space. (Hénaff et al. 2019) also show that these representations could be used for downstream tasks such semi-supervised image clas- sification in the low-data regime going on to record impressive results in the 1% and 10% ImageNet classification.
While such impressive results have been shown using the contrastive methods, methods of such quality for generative approaches are ye to be shown on images. Secondly, CPC and related methods adopt convolutional architectures for learning the representations. We believe it is worth the research effort to investigate architectures that incorporate self-attention so that we could translate language domain's 's success to other domains. Stand-Alone Self-Attention (Ramachandran et al., 2019) has shown that self-attentive architectures could be designed to match convolutional architectures on image classification and object detection. Such result is promising in the sense that now know that self-attentive architectures are not a limiting factor for downstream classification performance.
In this paper, we attempt to inspire from few key engineering deicisons that have benefitted the various successful approaches discussed above to motivate our design of a generative pre-training method for images.
1. Predicting subscales and low-bit depth for pixels: (Menick & Kalchbrenner 2018) showed that modeling pixels by sequentially modeling the subscales and low-bit depth
2

---

Under review as a conference paper at ICLR 2020
versions of the raw image is extremely useful. (Oord et al. 2016a) also attempted to ini- tially model 8-bit audio rather than 16-bit. Therefore, makes sense to model the only the most significant few bits while attempting to decode pixels for representation learning. Higher order bits are more relevant for texture and finer-details and may not be crucial for representation learning performance.
2. Use of self-attention for aggregating global context: Self-Attention (Vaswani et al, 2017 is an extremely powerful approach for aggregating global contextual representations across large sequences. The adoption of self-attention for images began with (Wang et al, 2018 who used non-local layers for activity recognition. (Zhang al.. 2018 and (Brock 2018) exploit non-local layers for high-fidelity image generation. (Razavi et al., 2019) has also shown that self-atiention can be used to good effect for modeling distribu-
et al.
tion of latents for likelihood-based image generation while (Parmar et al. 2018 Menick &
Kalchbrenner 2018 Child al. 2019) are examples for self-attentive density models.
3. Learning spatial dependencies across patches: CPC learns to spatially predict neighbor- ing patches given context of surrounding patches. Image Transformers (Parmar al. 2018) adopts self-attention that takes into account local as well as global dependencies behaving like patch-based generative model. (Menick & Kalchbrenner 2018 explot modeling spa- tial PixeICNNs over subscales for global image dependencies. (Trinh et al. 2019) attempt modify CPC for image representation learning by using the patch-based data extraction and modeling dependencies in BERT-like fashion using self-attention.
Our key contributions are follows:
1. We propose a new architecture, PatchFormer, for modeling bi-directional dependencies across patches. Our architecture learning decode missing patches in image by extract- ing represenstations of the given patches, using intentionn-poling to aggregate the context, and decode the low-bit grayscale sub-sampled versions of the missing patches. Specifically, we decode only the 2-bit grayscale version of the missing patch.
2. We show that our model could be pre-trained the unsupervised objective of decoding missing patches and fine-tuned on downstream low-data regime classification tasks.
3. We achieve somewhat competitive downstream ImageNet classification results with CPC (Hénaff et al., (219)) and are surprisingly even better than the other contrastive approach for semi-supervised downstream classification, Selfie (Trinh al. 2019) in spite of adopting generative approach.
ARCHITECTURE
2.1 PRE-TNAINING
Our patch-extraction setup is described in Figure Our input images are 224x224x3. We extract 16x 16 patches from this global image with stride of 16x16. This results in grid of 14x14 patches. Among the 296 patches, we spatially mask 60% of the patches (118) and use the remaining 78 patches. Our masks are designed with a 7x7 grid first (with the same masking ratio) and then up- sampled to 14x with nearest-neighhor upsampling. This ensure that our masks are contiguous and blocky to make the prediction task harder.
We then use a Transformer spatially associate the convolutional features extracted from the non- masked patches. Our self-attention mechanism flattens the grid and adds learned position position embeddings to each. To regularize the model, we use factorized position embeddings (similar to TTrnn et al., 2019)) for the X and Y coordinates. We use 10 layers of seaseed-slf--attentio with 1024 dimensional embeddings and 16 attention heads. Our pointwise MLP uses the GeLU non- linearity and widening layer of 4x similar to BERT.
We then use the extracted attention-pooles context vectors to decode the missing patches. We use regular residual upsampling decoder decode 8x8 versions of the missing patches originally of shape 16x The ground truth for the missing patches comes from the 2-bit geay-scale centre crop of these patches. We also additionally add soft-label noise cross entropy loss (0.01).
3

---

Under review as a conference paper at ICLR 2020
Our model is trained with Adam Deigh Optimizer using learning rate of 2e-4 and weight- decay coefficient of 0.001 with a batch size 128 on a v3-32 cloud TPU over 300,000 update steps.
Convolutional Encoding, Attention Pooling
Spatial Upsampling, Channel Downsampling
Create split image
Figure 1: PatchFormer Architecture: We extract grid of non-overlapping patches from the raw image. We then pick random crops of unmasked patches for spatial jittering. This is done to ensure the model does cheat similar to (van den Oord et al. 2018). We then apply random data-augmentations to these patches such as Inception Noise (random brightness, hue, saturation and contrast) as well as arbitrarily flip the patches 20% of the times. We also randomly convert the patches to their low-bit depth versions with the depth sampled randomly from 2 to 8 and apply random grayscale blending with 20% probability. The pixels are then scaled to [-1, 1] after applying these augmentations. A convolutional neural network (ResNet-41 a customized version of the first 3 stacks of ResNet-50) is applied to extract the embeddings for these patches. A deep self-attentive transformer then globally associates the patches across space (with added factorized position encod- ings) before decoding the missing patches. We decode subsampled and low-bit grayscale version of the missing patches make the prediction task simpler and more suited for representation learning.
2.2 FINE-TUNING
We remove the decoder and extract the attention sitentioo--ooled embedding of patches for classification (a residual layer operates on top of the attention pooled embedding). To ensure there is no mismatch between training and fine-tuning times (because of masking while pre-training), we randomly mask out 10% of the patches even while fine-tuning. While this does hurt the model in terms of good performance, also helps by regularizing the model. We fine-tune the model using the same opti- mizer as used for pre-training with batch sizes in {16, 32, 64, 128, 256} and learning rates in {3e-5, 2e-4, 4e-4, 1e-3). We also use soft-labels while fine- to--inunng to add some regularization to the model. Additionally, model also employs dropout noise at the end to prevent overfitting the low-data regime classification. The residual block we use on top of the iitentiion-oolingg the same as what for convolutional encoder described in Figure]
4

---

Under review as a conference paper at ICLR 2020
3 RESULTS
We pre-train using the 1.27 million unlabeled ImageNet training dataset. We then fine-tune for image classification 1%, 10% and 20% of the dataset and report the accuracies (top-1 and top-5) on the validation set of 50000 images for each of these subsets.
Table
Top-1 Accuracies
Model
1 1 ImageNet
10% ImageNet
20% ImageNet
CPC++ Unknown Unkown 30.3
Unknown 61.9 65.5
Unknown
Selfie Ours
67.1
70.2
Table Top-5 Accuracies
Model
1% ImageNet
10% ImageNet
20% ImageNet
CPC++
64.03
84.88
88.5 (based on plot)
Selfie
Unknown 52.0
Unknown 82.3
Unknown
Ours
87.6
CONCLUSION
We have proposed new architecture for generative pre-training on images called the PatchFormer. We highlighted the key tricks to making model learn useful representations for downstream classification tasks in spite of decoding pixels. We have shown that we are competitive with state-of- the-art contrastive pre-training methods such CPC on the low data-regime ImageNet classification benchmark.
REFERENCES
Philip Bachman, Devon Hjelm, and William Buchwalter. Learning representations by maximizing mutual information across views. arXiv preprint arXiv: 1906:009000009 2019.
Yoshua Bengio, Pascal Lamblin, Dan Popovici, and Hugo Larochelle. Greedy layer-wise training of deep networks. In Advances in neural information processing systems, pp. 153-160, 2007.
Andrew Brock, Jeff Donahue, and Karen Simonyan. Large scale gan training for high fidelity natural image synthesis. arXiv preprint arXiv:1809. 1109, 2018.
Rewon Child, Scott Gray, Alec Radford, and llya Sutskever. Generating long sequences with sparse transformers. arXiv preprint 19X4::9904.10000 2019.
Andrew M Dai and Quoc Lc. Semi-supervised sequence learning. In Advances in neural infor- mation processing systems, pp. 3079-3087, 2015.
Jacob Devlin, Ming- Wei Chang, Kenton Lee, and Kristina Toutanova. BERT: pre-training of deep bidirectional transformers for language understanding. CoRR, abs/1810.048055 2018.
Jeff Donahue and Karen Simonyan. Large scale adversarial representation learning. arXiv preprint arXiv:1907.025.4 2019.
Jeff Donahue, Philipp Krähenbühl, and Trevor Darrell. Adversarial feature learning. arXiv preprint 10X5::1605.00772. 2016.
Olivier J Hénaff, Ali Razavi, Carl Doersch, SM Eslami, and Aaron van den Oord. Dfta-efficient image recognition with contrastive predictive coding. arXiv preprint arXiv: /905.0927, 2019.
5

---

Under review as a conference paper at ICLR 2020
R Devon Hjelm, Alex Fedorov, Samuel Trischler, and Yoshua Bengio. Learning deep representations by mutual information estimation and maximization. arXiv preprint arXiv: 1808006670. 2018.
Karan Grewal, Phil Bachman, Adam
Jeremy Howard and Sebastian Ruder. Universal language model fine-tuning for classification. arXiv preprini arXiv: 10001100000000 10000.0666, 2018.
Nal Kalchbrenner, Aäron van den Oord, Karen Simonyan, Ivo Danihelka, Oriol Vinyals, Alex Graves, and Koray Kavukcuoglu. Video pixel networks. In Proceedings of the 34th Interna- tional Conference on Machine Learning- Volume 70, pp. 1771-1779. JMLR. org, 2017.
Yann LeCun, Yoshua Bengio, and Geoffrey Hinton. Deep learning. nature, 521((553) 2015.
Jacob Menick and Nal Kalchbrenner. Generating high fidelity images with subscale pixel networks and multidimensional upscaling. arXiv preprint arXiv:1812.01608, 2018.
Tomas Mikolov, Ilya Sutskever, Kai Chen, Greg S Corrado, and Jeff Dean. Distributed representa- tions of words and phrases and their compositionalitty In C. Burges, L. Bottou, M. Welling. Z. Ghahramani, and K. Q Weinberger (eds.), Advances in Neural Information Processing Systems 26, pp. 3111-3119, Curran Associates, Inc., 2013.
Aaron van den Oord, Sander Dieleman, Heiga Zen, Karen Simonyan, Oriol Vinyals, Alex Graves, Nal Kalehbrenner, Andrew Senior, and Koray Kavukcuoglu. Wavenet: A generative model for raw audio. arXiv preprint arXiv: 1609.034999 201
Aaron van den Oord, Nal Kalchbrenner, and Koray Kavukcuoglu. Pixel recurrent neural networks. arXiv preprint 111.::66611006752 2016b.
Niki Parmar, Ashish Vaswani, Jakob Uszkoreit, Lukasz Kaiser, Noam Shazeer, Alexander Ku, and Dustin Tran. Image transformer. arXiv preprint arXiv: 1802.00000 2018.
Matthew E Peters, Mark Neumann, Mohit lyyer, Matt Gardner, Christopher Clark, Kenton Lee, and Luke Zettlemoyer. Deep contextualized word representations. arXiv preprint arXiv:1802.00 1802.05365, 2018.
Alec Radford, Karthik Narasimhan, Tim Salimans, and Ilya Sutskever. Improving language un- derstanding by generative pre-training. URL mapp:://3--sswesttts amazonaws. com/openai- understanding paper. pdf. 2018.
Prajit Ramachandran, Niki Parmar, Ashish Vaswani, Irwan Bello, Anselm Levskaya, and Jonathon Shlens. Stand-alone self-attention in vision models. arXiv preprint arXiv: 19060.9900005999, 2019.
Ali Razavi, Aaron den Oord, and Oriol Vinyals. Generating diverse high-fidelity images with vq-vae-2. arXiv preprint aeee::906604444 2019.
Trieu H Trinh, Minh-Thann Thang Luong, and Quoc V Le. Selfie: Self-supervised pretraining for image embedding. arXiv preprint 1906::/9900099440 2019.
Aaron van den Oord, Yazhe Li, and Oriol Vinyals. Representation learning with contrastive predic- tive coding, arXiv preprint arXiv: 2018.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, łukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Advances in neural information processing systems, pp. 999986008. 2017.
Xiaolong Wang, Ross Girshick, Abhinav Gupta, and Kaiming He. Non-local neural networks. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 7794- 7803, 2018
Han Zhang, Goodfellow, Dimitris Metaxas, and Augustus Odena. Self-attention generative adversarial networks. arXiv preprint arXiv:1805.083118 2018.

---

Under review as a conference paper at ICLR 2020
Image
5x5 Conv, 2x2 Stride
LayerNorm
GeLU
Conv, 1x1 Stride
1x1 Conv
ResBlock
LayerNorm
GeLU
1x1 Conv, 2x2 Stride
3x3 Conv
ResBlock
LayerNorm
GeLU
1x1 Conv, 2x2 Stride
Add
1x1 Conv
ResBlock
ResBlock
AveragePool2D
Conv-Encoder
Figure 2: Convolutional Encoder used for patch embeddings


Reviews:
[{"review_id": "ryxGPZfzqS", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 1, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Contributions:\nThe paper aims to develop generative pre-training method for learning representations of images. Although representation learning for images has been widely investigated, the present work distinguishes itself by a combination of the following: \na) building on the use of transformers as a series of layers after initial convolutional layers; \nb) using self attention for aggregating context; \nc) learning spatial dependencies across patches;\nd) training on the task of predicting two bit gray scale version of randomly masked patches in an image\n\nResults: \nLimited experiments aim to compare against the CPC (Hefnaf et al 2019) and Selfie (Trinh et al 2019) algorithms both of which are contrastive unlike the generative approach adopted in the paper. After pre-training on unlabeled imageNet datasets the proposed approach is competitive with these algorithms with roughly similar results. \n\nEvaluation/Suggestions:\nOverall the paper combines ideas from several previous works in ways that are not sufficiently novel in the opinion of this reviewer and the experiments are very limited to the imageNet dataset with 1%, 10% and 20% of labels provided to downstream classification modeling, and evaluated on top-1 and top-5 accuracies. The paper could improve on its experimental evaluation bycomparing on multiple datasets, showing error bars when averaging across multiple samplings (eg for getting the 1% label set from the entire imageNet dataset) and also comparing with other approaches even when they dont directly aim to learn representation from unlabeled data (eg Image Transformers by Parmar et al). In addition the description is very high level and does not provide enough details for experimental reproducibility. For example the reviewer had to actually guess at some of specifics of the overall end-to-end architecture since it was not fully described precisely eg in a diagram. It would be relatively easy (but important) to provide suffcient detail for reproducibility", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "N/A"}, "extra_text": {}}, {"review_id": "S1gbpSTe9B", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 1, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper attempts unsupervised representation learning, via a patch prediction task on ImageNet. The paper is sparse on details, but the method appears to be: (1) split the image into non-overlapping visible and masked patches, (2) from features extracted from the visible patches, predict the masked patches. Rather than predict RGB, they choose to predict 2-bit grayscale images. Also, rather than use the full patches, they use random crops of the input ones, and a center crop of the output ones.\n\nThe paper seems to be an early draft of something bigger, submitted with the hope of getting some feedback. The method description is mostly composed of tiny details, such as the number and sizes of the patches; I recommend rewriting this to focus on the big idea first, and pack the details into another sub section like \"Implementation Details\". The paper barely includes any evaluation. Also, the method does not appear to be very novel: I recommend the authors look at and compare against \"Unsupervised Visual Representation Learning by Context Prediction\" (ICCV 2015), which is conceptually very similar.\n\nThe evaluation right now is not good. \"Unknown\" is not a valid point of comparison. I understand the code for CPC++ might not be released yet, but the authors could at least implement their best approximation of it, and also find older works (which CPC compared against in their paper), to fill out the results and make a convincing argument.\n\nIn Table 2, the proposed model performs worse than CPC++, yet its values are bolded anyway. Please only put the best-performing result in bold. ", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}, {"review_id": "B1xXZpgaFB", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 1, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The motivation of this paper is to use the idea of Transformer-based NLP models in image data, which is appreciated. However, this seems to be a far unfinished paper. The introduction part is well written. But, the method is not well described.  It is very unclear how exactly the model is built. Moreover, the network structure in Figure 2 is not explained.  The experimental part is very brief, and unconvincing. Much more investigations and comparisons are needed.\n\nMinors:\ndeicisons? \nmodel the only the most significant few bits -> model only the most significant few bits\nposition position embedding -> position embedding\n", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I carefully checked the derivations and theory."}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper presents a generative approach to learn an image representation along a self-supervised scheme.  

The reviews state that the paper is premature for publication at ICLR 2020 for the following reasons:
* the paper is unfinished (Rev#3); in particular the description of the approach is hardly reproducible (Rev#1);
* the evaluation is limited to ImageNet and needs be strenghtened (all reviewers)
* the novelty needs be better explained (Rev#1).
It might be interesting to discuss the approach w.r.t. "Unsupervised Learning of Visual Representations by Solving Jigsaw Puzzles", Noroozi and Favaro.

I recommend the authors to rewrite and better structure the paper (claim, state of the art, high level overview of the approach, experimental setting, discussion of the results, discussion about the novelty and limitations of the approach). 


Decision: Reject