Venue: ICLR 2020 Conference

Reviews:

final_score: rating: 3: Weak Reject
review_content: The paper proposes to use Contrastive Predictive Coding (CPC), an unsupervised learning approach, to learn representations for further image classification. The authors show that using CPC for representation learning allows to achieve better results than other self-supervised methods. Moreover, CPC is shown to be useful for semi-supervised learning (on par with SOTA method), and transfer learning. All results are very impressive and is in-line with current trends of using a linear classifier on top of a deep feature extractor (e.g., Nalisnick et al., "Hybrid Models with Deep and Invertible Features"). The paper is rather well written and the results are convincing. However, The whole idea of the paper is based on the original paper:
* Oord, Aaron van den, Yazhe Li, and Oriol Vinyals. "Representation learning with contrastive predictive coding." arXiv preprint arXiv:1807.03748 (2018).
Technically speaking, the paper is outstanding, but it lacks novelty in terms of new ideas. I highly appreciate new results and new architectures, but it is not enough for a full conference paper.

Remarks
- In Section 2.1, the problem statement for Contrastive Predictive Coding (CPC) is unclear. For instance, the authors explain CPC by mentioning about masked convolutional layers that is unnecessary at this point. I understand that from engineering perspective it is crucial information, but it does not help to understand CPC. As a result, without knowing the original paper on CPC, Section 2.1 is hard to follow.

- The paper can be treated as an uptaded version of the original CPC paper. I really appreciate all new results and implementation of the idea. The paper is well written and it is technically correct. However, I do not find much novelty compared to the original paper. This would be a perfect workshop contribution, but I am afraid that it is not enough for a full paper.

==== AFTER REBUTTAL ====
I would like to thank the reviewers for their rebuttal. It was not my intention to dismiss your effort in providing new technical results. Please forgive me if you read it in this way. My point is that the paper presents exactly the same idea as the original paper of CPC, but with new, very interesting results. However, I doubt if this is enough for a full conference paper. This point is debatable and I would be happy to further discuss it with other reviewers and the AC. At this point, I keep my original score, but of I am open for a discussion.
review_title: Official Blind Review #4

final_score: rating: 3: Weak Reject
review_content: Title: DATA-EFFICIENT IMAGE RECOGNITION
[Summary]
-This paper introduces Contrastive Predictive Coding (CPC) image recognition in the data-efficient regime. Concretely, the authors improve CPC in terms of its architecture and training strategy. The extensive experiments show that CPC enables data-efficient image classification and surpassed other unsupervised approaches. 

[Pros]
- Although the CPC was proposed and evaluated in vision task in [1], a new implementation of CPC with dramatically-improved ability is presented in this paper.
- The CPC is utilized to enhance spatially predictable representations which benefits a lot data-efficient image recognition.

[Cons]
- In Sec. 4.1, four axes are identified to upgrade CPC v1 to CPC v2. But they are not well motivated. More discussions about why this four axes are investigated in image recognition.

-The core idea is motivated by a critical hypothesis that good representations should make spatio-temporal variability in natural signals more predictable. However, this hypothesis is not well verified. The concept of amount of ‘predictability’ in page 7 is not clear. It would be great if you provide more evidence that the improvement in low-data classification results from the increased ‘predictability’.

- The comparison in Sec. 4.3 seems unfair. The pretrain model trained with different methods should be the same. For example, the Faster RCNN trained on CPC v2 uses ResNet-101 as backbone but Local Aggregation method uses ResNet-50.

[Summary]
- This work extends CPP to data-efficient image recognition by simply adjusting network architectures and training strategies, which makes it less interesting. Besides, the major hypothesis is not well validated. 
- The experimental results are convincing and encouraging. Some minor flaws such as unfair comparison should be fixed.
- I want to see how the four axes in Sec. 4.1 are related to core motivation (more predictable) since they are major adjustments from CPP v1 to CPP v2. If the author provides a profound explanation of the problem, I would consider changing the rating. 

review_title: Official Blind Review #2

final_score: rating: 6: Weak Accept
review_content: The authors augment contrastive predictive coding (CPC), a recent representation learning technique organized around making local representations maximally useful for predicting other nearby representations, and evaluates their augmented architecture in several image classification problems. Although the modifications to CPC aren't particularly original, the authors show first that these yield a significant improvement in linear classification accuracy. They then use this improved model to obtain impressive performance in classification within semi-supervised and transfer learning settings, giving strong support for the use of such methods within image processing applications.

Pros:
Owing to its generality (CPC assumes only a weak spatial prior in the input data), and cheap computational cost relative to earlier generative approaches, CPC is already a promising unsupervised representation learning technique. The paper gives more evidence of this usefulness for image data, yielding leading performance on several different image classification benchmarks.

The authors also make the observation that linear separability, the standard benchmark for evaluating unsupervised representations, correlates poorly with efficient prediction in the presence of limited labeled data. This observation should be of interest in the broader community, and points to the need for more diverse metrics for unsupervised representations.

Cons:
The improvements given in the paper are quite useful within their stated domain (image data), but aren't directly applicable to other types of input data. Although the authors make a point of emphasizing the relevance of CPC for other problem domains, they don't currently provide any suggestions for how this current work could be generalized to handle these other cases. In this sense, I think it is a bit deceptive to refer to their model as "CPC v2", as the majority of their changes have no bearing on the intrinsic CPC algorithm itself.

I am sure that some of the methods used here could lead to improvements in the use of CPC for other types of data, but the authors currently don't provide any insight on this issue. In line with that, I think their work would be improved by some commentary on this, in particular by any concrete suggestions they have about how similar augmentations to CPC could be carried out in text, audio, and/or video data.

Verdict:
Owing to the reasons given above, I recommend acceptance.

Minor suggestions:
Please use a different color scheme for your figures that is still meaningful if the paper is printed in greyscale.
review_title: Official Blind Review #1

final_score: rating: 3: Weak Reject
review_content: This paper improves Contrastive Predictive Coding method and reaches a good performance in several downstream tasks. However, the novelty and technical contributions are limited.

Strengths: 
+ The experimental results seem good. The reimplemented CPC v2 performs much better than the original version. And the performance of down-stream tasks is comparable or better than the state-of-the-art methods.
+ The paper is well written. The paper structure is clear and figures are well illustrated.
+ Figure 3 shows clearly the performance improvements of a series of incremental modifications to the original CPC methods.

Weaknesses:
- The novelty and technical contributions are limited. This paper only proposes some minor improvements based on the original CPC method and use a deeper network to get better performance. The proposed method lacks of important insights for the research community.
- The capacity of network architecture is crucial for self-supervised learning. But in Table 1,2,3, the network architecture of the proposed method is deeper than that in the comparison methods, which is unfair for the comparison methods. Meanwhile, the network architectures of many compared methods are not listed in the tables, which may be misleading. For example, Unsupervised Data Augmentation (Xie et al., 2019) in table 2 and Instance Discrimination (Wu et al., 2018) in table 3 use ResNet50, which is much more shallow than ResNet-161 in this paper.
- In section 2.1, the paper doesn't describe clearly what's the input of masked convolutional network $g_{\phi}$ and how to calculate $c_{i, j}$.


review_title: Official Blind Review #3

Meta-review: The paper tackles the key question of achieving high prediction performances with few labels. The proposed approach builds upon Contrastive Predictive Coding (van den Oord et al. 2018). The contribution lies in i) refining CPC along several axes including model capacity, directional predictions, patch-based augmentation; ii) showing that the refined representation learned by the called CPC.v2 supports an efficient classification in a few-label regime, and can be transferred to another dataset; iii) showing that the auxiliary losses involved in the CPC are not necessarily predictive of the eventual performance of the network.

This paper generated a hot discussion. Reviewers were not convinced that the paper contributions are sufficiently innovative to deserve being published at ICLR. Authors argued that novelty does not have to lie in equations, and that the new ideas and evidence presented are worth. 

The area chair thinks that the paper raises profound questions (e.g., what auxiliary losses are most conducive to learning a good representation; how to divide the computational efforts among the preliminary phase of representation learning and the later phase of classifier learning), but given the number of options and details involved, these results may support several interpretations besides the authors'. 

The authors might also want to leave the claim about the generality of the CPC++ principles (e.g., regarding audio) for further work - or to bring additional evidence backing up this claim. 

In conclusion, this paper contains brilliant ideas and I hope to see them published with a strengthened analysis of its components. 
Decision: Reject