Venue: ICLR 2021 Conference

Reviews:

final_score: rating: 7: Good paper, accept
review_content: 
***
***
Update after author response: 
The authors have addressed my comments and my recommendation remains unchanged.
***
***

##################################################

Summary:
The paper proposes a novel Bayesian method for few-shot classification. The proposed classifier makes use of the commonly-used Polya-gamma augmentation, but with likelihood replaced by a one-vs-each softmax approximation. The one-vs-each softmax approximation allows efficient computation in the posterior inference. The authors demonstrate better accuracy and uncertainty quantification in benchmark datasets. 

##################################################

Pros:
1. The paper is nicely implemented and the proposed method is clearly motivated from existing methods and show promising performance.


##################################################

Cons:
1. While the robustness to input noise is evaluated, it is unclear why the proposed method can handle the situation where training and testing data contain different noise levels or generated from different mechanism.

2. It may be better to highlight more the challenges specifically in few-shot classification problems and motivate the OVE approximation in this context. 

3. More justification of the the one-vs-each likelihood in a Bayesian setup is needed. It is unclear why it should be preferred except for computational reasons.

review_title: Review of "BAYESIAN FEW-SHOT CLASSIFICATION WITH ONE-VS-EACH POLYA-GAMMA AUGMENTED GAUSSIAN PROCESSES"

final_score: rating: 7: Good paper, accept
review_content: 1. Summary and contributions
This paper aims to improve the accuracy and uncertainty quantification in FSC using GP classifier. They use Polya-Gamma augmentation for tractable inference and introduce one-vs-each (OVE) approximation instead of softmax to apply PG’s property to the multi-class scenario. 

2. Strengths
- Although using GP classifier with PG augmentation is not a novel idea, this is the first work tried in FSC. Also, OVE for handling multiple classes in PG augmentation is an interesting contribution.
- The method is clearly stated. It looks much more efficient than BNN-based algorithms.
- The authors demonstrated a range of experiments, including uncertainty quantification, noise robustness, and out-of-episode detection. Their OVE PG GP with cosine kernel consistently performs well in every experiment compared to other prevalent methods. 

3. Weaknesses
- It is quite a novel idea to use PG augmentation with OVE approximation, but I think the authors are not well motivated about why their PG augmentation is necessary in FSC. If it were to quantify uncertainty, there are already GP-based algorithms as GPNet or LSM GP. Which property of PG augmentation makes it advantageous to other algorithms?
- In the experiments section, while the authors made an effort to implement extensive demonstrations, the discussion lacks. Specifically, while their OVE PG GP is consistently better than other GP-based classifiers, metric learning models, or Bayesian NN, the reason why their method outperforms the counterparts is still ambiguous. The authors should put more effort into explaining why OVE PG GP better classifies and captures uncertainty.
- Table 1 & Table 2. For OVE PG GP, it seems there is no clear winner between ML and PL objectives. For example, in CUB experiment, ML wins PL in 1-shot and PL wins ML in 5-shot. Some results have large deviations between them (e.g., nearly 9% difference in Omniglot->EMNIST 1-shot). Can you give a comment on the results of ML or PL: which is better to choose, and why they make inconsistent results in different experimental settings? 
- sec 6. ‘The OVE likelihood is better suited to classification ~’: Is there a theoretical or intuitive ground on this claim? I understand that OVE likelihood is introduced due to PG’s incompatibility to multiple classification. However, is there a specific reason why OVE is better than LSM with respect to classification ability?
- In Appendix I, the authors compared several likelihood functions. It seems that OVE is the most similar to Gaussian likelihood. Then, why is OVE likelihood free of the ‘ill-suited nature of applying Gaussian likelihoods to the fundamentally discrete task of classification (sec 6)’? 

4. Correctness
- sec 4.2. eq. (8) n→N, Y_(.c)→Y_(.c') Otherwise, entries of Af is not equal to f_i^(y_i )-f_i^c.

5. Additional questions or feedback
- sec 4.3. ‘We consider a zero-mean GP ~’: Can you explain why we should consider independent GPs for each class? Is it a common principle in GP classifier?
- Table 1. In mini-ImageNet 1-shot experiment, ABML result is extremely low and even lower than the original paper (37.65 in Table 1 vs. 45.0 in Ravi & Beatson, 2019.). Is there a significant change in the experimental setting?

6. Recommendation
This paper combines a novel idea of PG augmentation and OVE approximation into FSC, but still requires clear placement among the existing methods and more discussion on the reasoning of the results. I expect the authors to answer my concerns stated above during the rebuttal period. Thus, I vote this paper for rejecting (weak reject).

review_title: First Round Review by AnonReviewer1

final_score: rating: 8: Top 50% of accepted papers, clear accept
review_content: Summary
--------
The authors considers the problem in Few-shot classification and addresses the need for uncertainty management (calibrated output uncertainty, robustness to input noise, and out-of-episode detection) while maintaining high accuracy. To this end the authors propose a novel approach based on Gaussian process classification with Polya-gamma augmentation, one-vs-each Softmax posterior approximation and with a novel cosine *similarity* kernel (in composition with deep kernels). The latent variables from the Poly-gamma augmentation and latent GP function are Gibbs sampled and GP hyperparameters (including the parameters of the NN making up the deep kernel) are optimized using gradient decent based on the samples. The approach is validated in comprehensive comparative empirical experiments involving multiple datasets, and is demonstrated to be top performing in both accuracy and uncertainty management. 


Strong points
-------------
1. Well written paper, addressing an important problem in FSC with a well motivated and promising novel approach, filled with technical and methodology detail for completeness.
2. The approach combine high accuracy with calibrated output probabilities.
3. The performance is consistently strong at both robustness to input noise and out-of-episode detection.
4. The experiments are extensive w.r.t. the competitive approaches and have wide coverage given the different data sets used.

Weak points
-------------
Nothing obvious to me.

Reason for score
----------------
A well written paper, with several well motivated and empirically validated contributions, on an important topic. The paper seem to be technically correct and well placed in the literature. Especially the contributions on uncertainty quantification (both benchmarks and the proposed method) i believe are valuable and important for the FSC field, as well as for ICLR at large.

Minor comments
--------------
It would be better if Figure 2 can be made larger. Maybe by sharing the legend with Figure 3 and shortening one or two sentences slightly to make room?
review_title: Exellent paper on Bayesian modeling with Gaussian processes for important uncertainty management in Few-shot classificaiton without compromizing on accuracy.

final_score: rating: 6: Marginally above acceptance threshold
review_content: This work studies the GP-based few-shot classification problem with one-vs-each softmax approximation and polya-gamma augmentation. It points out the existing problems of the current schemes: (1) the non-conjugacy of GP classification, which is solved by the augmentation of Polya-gamma random variables; (2) the incompatibility of Polya-gamma augmentation with softmax link function, which is addressed by the one-vs-each softmax approximation. The theoretical analysis is solid with only some minor typos. Also, a lot of experimental comparisons are conducted between the proposed model with alternatives w.r.t classification accuracy, uncertainty quantification, noise robustness and out-of-episode detection. 

I recommend acceptance of the paper for the reasons below. Although the GP classification with Polya-gamma random variables is not new, the one-vs-each softmax approximation is a new idea as far as I know to reconcile the softmax link function with Polya-gamma augmentation. Another similar work is “Multi-class gaussian process classification made conjugate: Efficient inference via data augmentation, UAI 2019, providing a logistic-softmax likelihood to achieve the same effect, which is cited as LSM-GP by the work. Also, a large number of experiments are conducted and baselines are compared to validate the conclusion.

Weakness: A major concern is the work does not provide enough theoretical analysis about why the OVE likelihood is better than LSM. From the conjugacy perspective, both likelihood functions achieve the conjugate form. Besides, if my understanding is right, the LSM is an accurate “softmax” likelihood while the OVE is an approximation (lower-bound of the real softmax), why does the approximated likelihood (OVE) performs better than the accurate one (LSM) in all experiments? Is there any intuition behind this phenomenon? Although the author provided some evidence in experiments and sec.6 that OVE is better than LSM, some deep theoretical analysis is needed. Also, in almost all experiments, the author stated the proposed OVE “is one of the top performing methods”. I am not interested in such statement but the intuition analysis why it is better or worse. The work should not focus on SOTA-chasing but the delicate analytical solution, even if the performance is not the best. 

Some minor concerns: In the comparison of likelihood in Fig.5, it is farfetched to stated the OVE is close to softmax, as it is obviously closer to the Gaussian likelihood which is overconfident as stated in the paper. 
In sec4.2, it is unclear how to reduce the complexity from C^3N^3 to CN^3. It would be better to add a few words to describe it. 
In sec4.3, how to marginalize over latent functions in both objectives? I did not find any details in the submission and appendix.
In sec5.3, if my understanding is right, the robustness to input noise comes from the GP smooth effect, so again why does the OVE-GP model outperform other GP-based models? Any intuition behind this?


Typo: sec4.1 \vec(f)\simGP(m,k)   f should be a function not vector
Eq(7) equal -> proportional or add normalization. The right hand side is not a pdf. 
review_title: review for #2193

Meta-review: The paper presents a Bayesian approach for classification able  to  adapt  to  novel  classes  given  only  a  few  labeled  examples. The models combines a one-vs-each approximation of the likelihood combined with a Gaussian process. This allows to resort to a data-augmentation scheme based on Polya-gamma random variables. 
The paper is clearly written and combines existing techniques in a convincing manner; the experiments demonstrate better accuracy and uncertainty quantification on benchmark datasets. 

I recommend acceptance.
Decision: Accept (Poster)