Venue: ICLR 2022 Conference

Reviews:

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: The authors explore the uncertainty calibration problem in meta-learning few-shot classification, which is a relatively new topic that was seldom discussed before. 
To solve the challenges that the few samples in few-shot learning are usually insufficient for providing reliable estimation to the covariance matrices, the authors introduce meta-learning covariance matrices parametrized by a set encoder (implemented as set-transformer). And the authors further introduce scaled energy to parameterize a logit-normal softmax distribution for improving the uncertainty calibration of the softmax classification layer.
main_review: Strengths:
The overall presentation of this paper is both clear and straightforward.

The figures and equations precisely facilitate the reading. 

No noticeable typos were spotted. 

The topic that the authors selected is important and largely overlooked by the research of meta-learning and few-shot learning. The effectiveness of the proposed method is reflected by the quantitative comparisons. 

Weakness:
My major concern to this paper mainly arises from the use of an additional meta-learned set encoder to predict the task-specific covariance. 
It has been shown in previous work that some meta-learners have natural advantages in quantifying uncertainty. 

For example, relying on kernel methods, Gaussian processes based meta-learners [1,2] have demonstrated strong uncertainty calibration without leveraging any auxiliary components. 

Additionally, I might be wrong, but I'm curious that if this class-level covariance can be obtained through aggregating sample-wise information so that an additional set encoder can be removed. 
Please refer to Eq.6 to Eq. 9 of [3] for an example. Although the motivations are different, I believe there is some shared wisdom. 

[1] Deep kernel transfer in gaussian processes for few-shot learning, NeurIPS 2020

[2] Bayesian Few-Shot Classiﬁcation with One-vs-Each Pólya-Gamma Augmented Gaussian Processes, ICLR 2021

[3] Amortized Bayesian Prototype Meta-learning: A New Probabilistic Meta-learning Approach to Few-shot Image Classification, AISTAT 2021
summary_of_the_review: Overall this paper attacks a very important problem of meta learning and few-shot learning. And considering the quantitative results, the proposed method does achieve better uncertainty calibration. 
I expect further discussion on justifying the necessity of adopting an additional set encoder to achieve this capability.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This submission focus on improving the performance of uncertainty quantification and OOD detection using bi-Lipschitz regularized neural networks. There are two involved challenges here: (i) we are required to estimate the class-conditional covariance matrix of the latent features. This estimation is challenging under low sample size. (ii) generative classifiers using softmax activation function suffer from shift invariance, and thus may become over-confident for OOD samples.

To resolve these issues, the paper focus on two innovations:

(i) a meta-learning model to estimate the class-conditional covariance matrices,
(ii) a modification of the inference procedure with the energy function to detect OOD samples.

The paper concludes with numerical results showing the performance of the proposed solution approach.
main_review: This submission is empirical: the paper is well-motivated, and the paper adapts existing, popular techniques to engineer a solution to the standing problem (using quadratic discriminant analysis, modeling the covariance matrix as a sum of a diagonal matrix and a low-rank psd matrix, etc.).

On the downside, there are several theoretical questions that are left unanswered:
1. Why is the Set Transformer a good choice? On which ground can we justify that the Set Transformer can generate sensible values of Lambda and Phi to construct the covariance matrices?
2. There are a lot of methods from statistics to estimate covariance matrices from low sample size dataset (linear/nonlinear shrinkage, sparse, etc.). Why can't they be used in this application? Is it possible to show that the Set Transformer is better than these traditional approach (for any possible criteria)?
3. In page 8, why should the eigenvalue distribution be diversified? It is well known that the eigenvalue can be sensitive to small perturbations of the off-diagonal entries of the matrix. Thus, what we observed (diversity of the eigenvalues) can be just an artifact of the numerical errors from the Set transformer network. Diverse eigenvalues also imply that the condition number is large, which leads to higher numerical errors to the log-determinant term. Thus, I am not fully convinced that a diverse set of eigenvalues is of any advantage.


Minor:
- What is f(E) and min(E) after equation (10)?
- The abstract and intro focus heavily on bi-Lipschitz regularized NN, however, the exposition in section 3 (except for Section 3.4 which is 10 lines long) relies minimally on this architecture. This emphasis may be misleading in the first read.
summary_of_the_review: In my opinion, the paper is a good engineering attempt to solve a specific problem. It lacks, unfortunately, concrete scientific contributions.


======== Post-rebuttal ========
The authors have provided further empirical results to justify the use of the ST to estimate the covariance matrix. I have raised my evaluation from 3 to 5.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: In this paper, the authors proposed methodologies to estimate the covariance matrix of embedded features under the meta-learning setting. The authors first showed that the empirical covariance matrix is inappropriate in this setting and that due to the shift invariant property of softmax function, ProtoNet could produce overly confident predictions. 

The authors instead propose to learn the covariance matrix through a Set Transformer structure, where the covariance matrix can be decomposed into the sum of a diagonal component and a low rank component. The authors used scaled energy and bi-Lipschitz regularization to make the methodology more performant. 

Numerical studies are performed to evaluate the calibration performance as well as the eigenvalue distribution of several methods.
main_review: The paper is well written and easy to follow. Some specific comments / suggestions:
1. It would be great if the authors could expand the discussion of bi-Lipschitz regularization in the related work section. Some discussion and literature review of set-input functions, e.g., Set Transformer, Deep Sets will also be helpful.
2. What are the practical ways to ensure that Lambda_c matrix is non-negative, without making too many entries of Lambda_c to be zero?
3. Is there any benchmark on the computational requirement of the proposed methods, compared to competing methods? The covariance inversion and log-determinant will add computational cost to the methods, and it would be good to add some analysis to show the tradeoff between computational cost (i.e., number of iterations) versus performance.
4. In the numerical studies, it seems that the proposal performed comparatively better in the low-way low-shot setting. Is this the correct observation? If so, is there any intuition behind this?
summary_of_the_review: The paper is well-written with clear motivations and easy to follow presentations of the methodology. The proposal is relatively novel, and could be a meaningful contribution to the meta-learning community.
review_title: null

Meta-review: Reviewers all found the work well-motivated in addressing uncertainty, a topic that has not seen much focus in meta-learning and few-shot learning. They describe the challenges well: small sample sizes and OOD shift. They then propose a solution they find works well empirically to overcome these challenges based on a set encoder and an energy function respectively.

The proposal is largely one of engineering components that have been found to work well in the literature. I'm sympathetic to this style of research (particularly in today's neural network research), although the reviewers raise a primary concern about whether the choices leading to the proposal are justified. In particular, two Reviewers argue that there are no clear ablations compared to alternative simpler approaches, and so the approach of selecting a Set Transformer is rather arbitrary. My perspective is that theory provides one sufficient but not necessary angle to do this, and I do find the authors' replies to the two reviewers convincing. In particular, they add a baseline to estimate covariances suggested by Reviewer Zz5v and they describe how the current baselines do in fact use the shrinkage suggestion by Reviewer QrCN.

I recommend the authors use the reviewers' feedback to enhance their submission's clarity and overall quality.
Decision: Accept (Poster)