Venue: ICLR 2021 Conference

Reviews:

final_score: rating: 5: Marginally below acceptance threshold
review_content: The paper proposes to find a good representation of the underlying data generating function (data distribution) via contrastive learning. In contrast to existing works on applying contrastive learning to learn sample-wise representation, this idea is novel.

Pros:
The insight that "two sets of examples of the same function should have similar latent representations, while the representations of different functions should be easily distinguishable" is interesting. This can also relate to learning from a sets of tasks in meta-learning, where most tasks are related but some tasks are out of the distribution.

Cons:

On page 3, the illustration on training is like meta-learning. Is it? If so, please rewrite it using meta-learning notations. If not, please describe the difference with meta-learning. 

Also, sentences under equation 2 use the terms "projected representations" and "encoded representations", what do they refer to? Can you use notations? 

The paper provides a number of experiments on diverse data sets. These are valuable. However, as most of these tasks (especially Sec 4.2 and 4.3) are formulated into few-shot problems by authors, can you provide mathematical problem formulation at the beginning? It is hard to get why it is few-shot and what is the target. 

Overall, I find some clarifications are required. The current version is not clear enough to convey its importance. 


review_title: A few concerns about the paper.

final_score: rating: 5: Marginally below acceptance threshold
review_content: The authors propose a novel training approach that uses a self-supervised signal to learn whether two sets of data are generated from the same function.  The goal is to align representations of different samples generated from the same function for use in few shot learning.

Background section should be expanded to provide a better understanding of the contrastive learning space.  In general section 2 needs to be improved.  The problem is not clear in this section as the main problem of few-shot learning is not clearly set up as an intuitive result of the approach of aligning representations of different samples that stem from the same function.

The writing in Section 3 needs to be improved.  Particularly the "Encoder Training" section as it is a little hard to follow.  There are K functions but the observations from the K functions are split into J sets of size N/J.  Is this just having N total observations that stem from K functions where positive pairs are pairs of observations from the same function and negative pairs are from different functions?  Or is this a method to sample a set of partial observations that are used in Equation 2 to generate the representation?  I am assuming the latter but this is not clear.

In Equation 4, when J > 2 then doesn't this become a multi-label problem where there are multiple pairs of observations from the same function?  I am confused on this equation in how it will prevent different positive pairs from competing with eachother.

I am not sure what the main contribution here is.  The paper describes a method for using contrastive learning to align representations of observations from the same generator.  The learning approach is not novel but the applications are interesting and the idea of aligning representations of observations from the same generator is good.

recommendation and reasoning

This paper is slightly below the acceptance threshold.  The general approach is clear and the evaluations show good performance but the writing needs improvement and the novelty is lacking.

I have read the author responses but my initial review remains unchanged.  The paper needs to be greatly improved in terms of clarity and the responses from the authors help but I think it is better off being resubmitted to another conference with heavy revisions. 
review_title: This paper is slightly below the acceptance threshold.  The general approach is clear and the evaluations show good performance but the writing needs improvement and the novelty is lacking.

final_score: rating: 5: Marginally below acceptance threshold
review_content: The paper works on the problem of few-shot learning. They propose to use contrastive learning to obtain good representations for different mapping functions. Experiments on 1-D, 2-D image and scene representation show certain advantages.

[Strengths]
- The idea of using contrastive learning to obtain good representations for different mapping functions is interesting and novel.
- The writing is in general clear and there are also a lot of experimental results.

[Weakness]
- There is a strong assumption that the mapping function during the few-shot task is one of the functions during training. Is this practical?
- The contrast learning only encourages those function representations to be different and it might not be able to learn the underlying function characteristics. Why can it help during down-stream few-shot tasks? In other words, I could use one-hot vector to represent each function. I could give the function index to the down-stream task. Is it helpful?
- Although contrast learning is hot, why must use contrast learning to learn function representation? You know the function labels, so you could also use triplet loss during the representation learning.
- Although the authors consider quite a few downstream tasks, all the comparisons are quite preliminary. It would be more convincing to show that in at least one task your approach can outperform SOTA or your function representation can improve SOTA.
- Figure 3(a) shows a better result with 50 shots, but with 200 shots. Why? 
- There is no explanation for Fig. 2(b). It seems you use N samples to identify the function and the rest N-M samples to train the decoder. Is it correct? What's a good N here? What is the "S" operation?
- Eq. (4) does not look correct.
- Is Section 3.1 complete or not ? The organization is a bit wired.

---
Update:
- I think the idea of this paper is very interesting. The authors' responses also address most of my concerns. The remaining issues are: 1) the writing needs to be improved to make the paper easy to read; 2) I am still not so convinced with the experiments. It would be more convincing if the authors can demonstrate this on few-shot image classification to improve SOTA, for which the joint (x,y) distribution might be too big at the image level. On the other hand, I am not so familiar with this topic. If such experiments are quite normal in this area, then I am fine to increase the rating.

review_title: An interesting idea of using contrast learning to distinguish different mapping functions, but there are still some major issues.

final_score: rating: 5: Marginally below acceptance threshold
review_content: As far as I can tell, this paper proposes to use contrastive learning to solve few-shot learning -- not just few-shot learning, but meta "task-wise prediction" as well (predict some properties of the task itself). The method learns representation of each task, by forcing random instantiations of the same tasks to have similar representations. The method tests on sine-function fitting + param prediction, image (MNIST) inpainting + classification, and (robot) scene understanding + RL (both are meta "task-wise prediction").

Pro:
- Method seems reasonable and improves performance over compared methods (NP and CNP).
- Highlights an interesting point of view (which I think is relatively novel -- it's glossed over in the NP paper) that treats location-value prediction for pixels in different images as different tasks to test meta-learning algorithms. This change of view encourages the analogy and idea of using e.g. self-supervised learning or inpainting methods for meta-learning.

Con:
- The goal and motivation of this paper is lost to me. The abstract and intro does not inform readers of the goal of the paper and how it can be used in applications (such as ImageNet meta-learning). It seems to be all of (1) few-shot learning, (2) extension of existing works NP & CNP, and (3) learning representations for down-stream tasks. But it seems to be suboptimal for all of these goals.
  (1) If this is few-shot learning, then it would make sense to compare to other few-shot methods, e.g. ones using meta-learning.
  (2) If this is an extension of NP and CNP, then this method has no grounding probabilistically. It cannot provide estimates such as P(x|y), for example, which is the point of these methods. It is not very interesting that non-generative methods can outperform NP/CNP which are generative models.
  (3) If this is learning representations for down-stream tasks, the the down-stream-focus of the experiment setup violates the "few-shot" part since only the original formulation (e.g. image inpainting) is few-shot. For down-stream tasks (e.g. image classification) this is rather a "sparse input" problem. Despite the method being about learning representations, it does not compare to regular methods that treat images as images, not a set of (location,value) tuples. For example, a regular CNN may do better than the results reported in Figure 5.
  (4) If this is a method that can do both few-shot and learning representations, it should be still compared to regular few-shot and representation learning methods, especially because there are so many papers that do few-shot and representation learning, independently or combined.
- The motivation is quite similar to self-supervised learning, when considering down-stream applications where we treat each "task" as samples again.
- The statistical significance is confusing. For example, Table 1 has 0.0087±0.0007 and 0.0078±0.0004 being significantly different with 3 runs, which does not sound plausible.

Other minor issues:
- The method assumes each task is different. If there are many duplicated tasks (e.g. very similar images) then making each task representation different from all others would be suboptimal.
- In eq. (5), it seems the input for downstream tasks include the pixels/view-images, so it will be unclear if the knowledge is from X or $h_\Phi(D^T)$.

Summary:
Although viewing location-value prediction for pixels in different images as different tasks for meta-learning is really interesting, the paper fails to motivate its goal, and by extension, designing convincing experimental comparisons. I will be reserving judgement and lean towards rejection.

Update post-rebuttal:
The rebuttal clarified the motivation, but has yet to address the flaws that was associated with the choice of motivation and positioning among related work. Unfortunately that means I will not be changing my score.
review_title: I am completely lost. (update)

Meta-review: While the authors provided extensive responses to the reviewers and most of the reviewers did a good job of accounting for the author responses the final ratings for this paper was unanimously 5s -- all marginally below acceptance. The paper's positioning, writing were identified as key issues that remained to be addressed. The AC recommends rejection.
Decision: Reject