Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper proposes POEM, a generalization of prototypical networks designed to handle few-shot learning when each support or query example is only a partial view/observation of the underlying concept. The derivation follows from modeling the joint distribution of support representations as the product of conditionally independent Gaussians. Prediction for a query example is done via Bayes's rule and marginalizing over the representation. For their experiments, the authors adapt a subset of Meta-Dataset so that each sample exhibits substantial partial observability. POEM is shown to improve upon baselines that do not model the partial observability. The authors also compare POEM and prototypical networks on a map reconstruction task based on the MiniGrid environment, finding that POEM enables better reconstructions faster.
strength_and_weaknesses: Strengths

The work studies partial observability, a realistic property of high-dimensional embodied observations, in the context of few-shot learning. The proposed method is reasonable and generalizes a standard method from prior work. Steps are taken to ensure fair empirical comparisons. The authors also contribute a new benchmark based on adapting an existing dataset, and experiment with an embodied toy mapping task.

Weaknesses

The baseline algorithms used in the PO-Meta-Dataset are rather weak, and it would be interesting to see to what extent more expressive models such as latent embedding optimization [A] are able to meta-learn strategies to handle the partial observability in comparison to the more explicit modeling done in POEM. Some relevant references also seem to be missing; it would also be good to discuss POEM in relation to amortized Bayesian prototype meta-learning [B], which also considers a probabilistic model of semi-parametric few-shot learning for classification. The authors also don't discuss whether the RoamingX datasets proposed in [C] could be suitable for evaluation, given the close relationship between embodiment and partial observability (as evidenced by the MiniGrid experiment).
clarity,_quality,_novelty_and_reproducibility: Clarity

It would be good to provide the concrete instantiation of POEM in algorithm box form, i.e. provide the missing link between the mathematical statement of the objective (Eqs. 8 and 9) and the reduced form afforded by simplifying assumptions, e.g. Gaussianity of each expert. 

I'm also not sure that emphasizing the "contrastive" aspect of the work (e.g. in the title) is particularly important. You could say that few-shot learning has always been contrastive in the sense that different samples of the same class are "augmentations" of one another. The method doesn't seem to have particularly strong connections to standard contrastive learning objectives.

Quality

There are several minor weaknesses as detailed above. Otherwise, the technical contribution seems correct and the empirical evaluation sound.

Novelty

There is some technical novelty in the proposed method, and some empirical novelty in the new dataset.

Reproducibility

Code is promised upon publication. As mentioned above, it would be good to have a concise description of the objective as pseudocode to aid in long-term reproducibility.
summary_of_the_review: Overall, the work is interesting and of good quality, but is currently mired by a few individually minor weaknesses as detailed above. As such, I currently recommend borderline rejection.




References

[A] Rusu et al., Meta-Learning with Latent Embedding Optimization, ICLR 2019.

[B] Sun et al., Amortized Bayesian Prototype Meta-learning: A New Probabilistic Meta-learning Approach to Few-shot Image Classification, AISTATS 2021.

[C] Ren et al., Wandering Within a World: Online Contextualized Few-Shot Learning, ICLR 2021.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: The paper proposes a contrastive training approach for few-shot learning. The main benefit of the proposed approach is that it can perform few-shot learning in partially-observable settings.

The paper describes a formal framework for learning in partially-observable settings based on the product of experts model. Specifically, a potential function is associated with the representation learned for each view.  The factors over views are combined using a product of mixtures. To make the learning objective tractable the potential functions are assumed to be Gaussians though other types of distributions may be explored in future.

A new dataset is developed that augments an existing dataset for few-shot learning (META-DATASET). The idea is to incorporate different views with partial observability in the dataset. The proposed approach is compared on all tasks available in the META-DATASET and compared with state-of-the-art for few-shot classification. Results show that under partial observability the proposed approach outperforms existing methods while being close to state-of-the-art on datasets that o not require reasoning over partially observable views. Further, the same approach is applied to learning the representation of a grid environment based on partial observations and it is shown that the proposed approach performs well in this case.

strength_and_weaknesses: Strengths
Seems to be an intuitive idea with a nice formalism of connecting uncertainty in views to the training function
The new dataset seems to be a good contribution for others working in this area
Shows empirical performance that is quite good in different settings

Weakness
Is there any bias in the creation of PO Meta-datasets. Specifically, since the other state-of-the-art models were not designed to solve the task in PO-Meta (if I understood it correctly), then are the results fair towards these models?

For the agent environment, it seems like this type of representation learning is bit more forced as compared to using other techniques such as POMDPs. Is the Prototypical network the state-of-the-art for this type of learning? Is there a stronger, more realistic case for this representational learning approach in partially observable environments
clarity,_quality,_novelty_and_reproducibility: The writing is clear an seems well-motivated. The contributions do use existing methods (product of mixtures) but frame them in a novel manner to perform learning from multiple views. The creation of a novel dataset and a new task based on an existing dataset could also be significant in future work.
summary_of_the_review: Overall, the paper seems to have a clear motivation backed by a nice formulation. However, since the task and the dataset is new, the significance of the comparison studies is one possible issue.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: +++++After Rebuttal+++++

I would like to thank the author(s) for the rebuttal and revisions made. I believe the my main concern, which is the clarity of the presentation, has been greatly improved. In response to the improvements made, I have adjusted my score. Note that I am still not fully convinced that $P(X^*|X_m)$ is comparable for different classes. 

++++++++++++++++++++

This paper develops a Bayesian formalism for few-shot learning that captures representations with different levels of uncertainty in different components. In particular, this work is motivated  by the current gap in handling views that only contains partial observations, which violates the assumption made by most existing few-shot solutions. The proposed method, Partial Observation Experts Modelling (POEM), efficiently integrates different but overlapping information from different pieces to enable better global decision making. The proposed method is extensively benchmarked on the large-scale few-shot benchmark datasets, reporting encouraging results in partial observation settings. The author also demonstrated a synthetic experiment where POEM is used to facilitate an agent exploring the environment using only partial observations. 
strength_and_weaknesses: ### Strength
* This work is very-well motivated. Given the scarcity of support examples, it is desirable to quantify the relative similarity between support and query examples with the notation of uncertainty for robust inference and better generalization. 
* In the experiment results, the proposed PEOM showed significant performance gains over the baseline solutions under the same benchmark parameters and model architectures in most cases. 

### Weakness
The main weakness with this submission is the lack of clarity. I am unable to establish a mental model of how everything works together after a few attempts with the paper. The mathematical notations are not intuitively mapped to intuitive concepts/examples in few-shot learning. The local expert $\phi(z|x)$ is not defined when it appears in the text, and I can not find a final training objective in the main text. In Fig 1 & 2, it seems to suggest that each view is a partial observation of the object, potentially augmented with some transformations. And it seems to suggest each sample in the support represent one particular class. So are the views for each X_m extracted from patches of all supporting images of class-m? Also $V^m = dim(X^m)$ seems to suggest each view is one dimension of the vector X. Additionally, is the query a view or a collection of views? These are all not clear from the text. If the size of support views differ for each class m, then is P(x^*|X^m) still comparable? p(z) seems to be a mixture of Gaussian on page 4, how is that tractable (are you just summing over the finite support set)? Then on page 5 the author(s) claimed that p(z) is approximated with a Gaussian, which again does not give any detail how the approximation is implemented. Eq 18 and Eq 19 (from the Appendix) are introduced on page 5 without any context, I guess that’s a typo in referencing equations. While the author(s) criticized prior work for using diagonal covariance (sec 2.2), base on the notations it seems that the this work also relies on diagonal covariance. Finally, there is a lot of marginalization going on, and for real image data the probability might vary considerably in scale, causing concerns for numerical overflow issues. 

clarity,_quality,_novelty_and_reproducibility: Quality. Fair
The abstract and introduction are very well-written. The author(s) clearly introduces the problem and opportunities for filling the gap in few-shot learning where the partial view and representation uncertainty are inadequately accounted for. The quality starts to degrade moving to method sections, where the problem setup is not clearly defined and the notations are confusing, there is also not enough detail on the final learning objective and how it is derived. The experiment results show solid gains over the competing baselines, but that needs to be interpreted with salt and gain: (1) these are all classical baselines and do not include more recent SOTA solutions; (2) the experiment setup differs from standard settings and it seems that no proper adjustment have been applied to the baseline solutions to accommodate the change. 

Clarity. Poor
With the current presentation, this work is not reproducible because of poor clarity. Detailed comments are given in the Weakness section above. 

Originality. Fair
Patched-based query (partial observation) and augmented views have been explored by prior works [1-3].

[1] Rethinking Generalization in Few-Shot Classification 
[2] Learning Task-aware Local Representations for Few-shot Learning
[3] Few-Shot Learning with Part Discovery and Augmentation from Unlabeled Images

summary_of_the_review: The current presentation lacks clarity and the claimed novelty failed to set it apart from prior arts, the experimental evidence also needs to be strengthened. Consequently, I am recommending reject at this moment. I will be happy to reconsider my evaluation if the author(s) can present strong supporting arguments or significantly improve their work during the rebuttal phase.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper aims to tackle the partially observable few-shot learning problem where useful features are only contained in some of the views of a data instance. It develops a product of experts model to map different views to the representations with different uncertainty, which can capture the importance of different features. The proposed method is demonstrated to outperform previous few-shot learning methods on the newly created Partially Observable MetaDataset and MiniGrid environment.
strength_and_weaknesses: Strengths:
1. This paper identifies a novel problem that is also very important to the community, partial observation is very common in the reinforcement learning problem.
2. The proposed method is well-formulated and the authors also provide an intuitive interpretation of the training objective.
3. The experimental shows the advantage of the proposed POEM method to some extent and the ablation study further verifies the source of improvement comes from the feature uncertainty.
4. The newly created PO-Meta-Dataset could be useful for future studies on this problem.

Weaknesses:
1. The baselines compared with POEM are all too old, the authors have mentioned many recent strong meta-learning methods in the related work part, I think the authors should also compare POEM with some of them. DeepEMD [1] is also a strong baseline.
2. I do not see any point to mention contrastive learning in this paper, although there are some weak connections, like the objective in embedding-based few-shot learning and multi-views used by SimCLR, POEM is in general not strongly connected with the contrastive representation learning methods and I feel the title is kind of confusing.
3. Although partial observation is an interesting problem, it is not commonly seen in the way PO-Meta-Dataset developed. The two experiments are more like toy problems where the feature uncertainty is manually inserted, I do not think creating the views through data augmentations used by SimCLR is the correct way to generate a realistic partial observation problem.

   The most common partial observation is in reinforcement learning, known as POMDP, but the authors do not explore this problem enough in this paper. Another way I can think of is to make use of the annotation in the CUB Bird dataset, create a support set with only some parts of a bird observed, and the query set with other parts appearing. It could be more close to the real-world scenario where objects are not always observed from the same angle, so the parts observed in the support set may be different from the parts observed in the query set for the object from the same class.

[1] DeepEMD: Few-Shot Image Classification with Differentiable Earth Mover's Distance and Structured Classifiers. Zhang et al., CVPR 2020.

clarity,_quality,_novelty_and_reproducibility: This part is generally well-written and technically sound. But I feel the 'contrastive learning' mentioned in this paper is confusing, which seems not closely related to the method and the problem studied in this paper. The problem it studies is significant and the method is novel. Details about the model, dataset, and environment are provided, which enhances its reproducibility, but I think the authors could give details about the whole algorithm in the appendix.
summary_of_the_review: This paper studies an interesting problem as partial observation in few-shot learning. It derives a novel method to tackle this problem and gives an intuitive interpretation. The method is proven to outperform previous baselines on two problems. But generally, the baselines chosen in the experiment are too old and the problems designed are not realistic enough, which makes the significance of the proposed method in doubt. I think this paper is on the borderline and authors should try to add additional experiments on more realistic problems/datasets to further demonstrate the effectiveness of POEM.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: This paper addresses few-shot learning in partially-observable settings, where the goal is to meta-learn fruitful features that may be present in only some of the views. It bases its development on the product of experts to map different views to the representations with different uncertainty, capturing the contributions of different features. All of reviewers agree that the partially-observable setting seems to be novel in few-shot learning environment and the method developed here is sound. Authors did a good job in the rebuttal phase, resolving most of concerns raised by reviewers by adding baseline methods in empirical comparisons. While there is a disagreement on the issue whether the term ‘contrastive’ fits the method developed here, all reviewers agree to champion this paper.

justification_for_why_not_higher_score: Accept (poster) is the best place for this paper. 
justification_for_why_not_lower_score: Authors did a good job in responding to reviewers' concerns, leading that all reviewers suggested 'marginal accept'. In fact, this paper received the best score in my batch.
Decision: Accept: poster