Venue: ICLR 2022 Conference

Reviews:

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: The submission examines the impact of Maximum-Likelihood Estimation (MLE)'s small-sample bias in the few-shot classification setting. The authors point out that while asymptotically consistent, MLE is biased in the finite-data regime, and propose to apply Firth bias reduction to few-shot classifiers to counter this bias. In this setting, the technique simplifies to a KL divergence regularizer between the uniform distribution over class labels and the learner's predictions.

Experiments on mini-ImageNet, CIFAR-FS, and tiered-ImageNet using ResNet and WideResNet network architectures show modest but consistent improvements across all settings. Firth bias reduction is shown to outperform other regularization approaches such as L2 regularization and label smoothing alternatives (confidence penalty and unigram label smoothing).

main_review: I enjoyed reading the submission. It's clear, well-written, and it explores a simple idea with rigor. The proposed idea is theoretically grounded and the experimental results are convincing.

Strengths:
- Clarity, writing quality.
- Novelty: as far as I know, the issue of MLE's bias in the finite-data regime has not been discussed in the few-shot classification literature.
- Rigor: the proposed approach is compared against sensible baselines (L2 regularization, label smoothing), controls for randomized factors, computes and compares confidence intervals, and considers multiple benchmarks and backbone architectures.

Weaknesses:
- The submission goes a bit quickly over the small-sample bias of MLE and Firth bias reduction for MLE. Without going into complete proofs, sketching the main arguments in both cases would make the submission accessible to a wider audience. As it stands, I feel like the submission is fully clear only to readers with prior knowledge of these notions.

Additional questions/comments:
- Can the authors elaborate on why the number of novel classes subsampled for Figure 1 is 16, specifically?
- In Figure 1, the improvements obtained from using Firth bias reduction are magnified by the backbone capacity. Is there an explanation for this?
summary_of_the_review: The submission is clear, well-written, and it explores a simple idea with rigor. The proposed idea is theoretically grounded and the experimental results are convincing. The one thing in the way of a higher score is that the submission could be made more accessible to readers with less experience with the theory of small-sample bias in MLE and Firth's bias reduction for MLE.

---

**Post-rebuttal update**: The authors' response addresses my concerns, and I now feel comfortable making a clear acceptance recommendation.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: This work introduces a method to reduce the bias of MLE for FSL problems. Authors developed the bias reduction formulation based on the Firth bias reduction method and extended it for cosine classifier as well. A through set of experiments demonstrate that the proposed method provides statistically significant improvement over the baselines. 
main_review: Q1- According to the intro, which I quote here ``We achieve this by deriving a simplified yet effective Firth formulation that penalizes the KL-divergence between the predicted distribution and the uniform distribution of classes'', the proposed solution assumes that the distribution of query samples is uniform. One wonders how restrictive this assumption might be, can authors comment on this? I understand that in FSL, current protocols have that property but when it comes to learning, many algorithms do not make such an assumption.
I have to add that I understand that authors performed experiments in section 4.3 about imbalanced datasets and discussed replacing the uniform prior with non-uniform one, but from a theoretical point of view, is it possible to provide more insights?

Q2- Based on Fig.1, it seems that the proposed method works better if number of shots is large, and specifically for very deep models. Would be very useful if authors can provide some insights about this.

Q3- I can see some sort of inconsistency in the experiments. It would be good to see why authors didn't use the same datasets in Fig.3 for experiments in Fig.1. Also, the choice of 16class in Fig.1 is not well-justified. How does the method work if less classes were considered?

Q4- Out of curiosity, did authors evaluate their method in conjunction with other FSL techniques, say other embedding techniques?

A suggestion,  it is very hard for me to fully read the improvements in Fig.3, why not simply providing a table so the reader can better understand the improvements?


summary_of_the_review: I believe the proposed method is novel and interesting.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: The paper proposes to reduce the bias inherent to the estimation of the classifier used in Few Shot Classification settings. To this purpose Firth bias reduction is proposed to be used. Theoretical computation of this bias reduction is carried out within the context of Few Shot Classification resulting to a practical implementation. It turns out that it could be associated to a regularization term encouraging uniform class assignment probabilities. This results is also generalized to the context of using a cosine classifier on the features.

Extensive experiments are carried out to show the benefits of this bias reduction on SOTA approaches and with various backbone architectures.
main_review: # Main review

This paper is clearly written with extensive experiments to support the proposed approach.

The strength of this paper is related to the effective computation of the Firth bias reduction term resulting to a simple pragmatical implementation. This pragmatical implementation turns out to be a theoretical justification of the Label smoothing proposed in Szegedy et al.

In the mathematical notations the logistic regression weights are defined so that $\beta_0=0$. This allows to have a non-redundant notation for logistic regression weights and to easily carry out computation of the loss associated to Firth bias reduction. However it came with some drawbacks:
- when using L2 regularization, the regularization could be non optimal. What about the impact here on the performance of L2 regularization? (from table A2, we can also observe that the set of weights for L2 regularization is quite high - usually when using L2 regularization we observe the use of weights of order 10-4 ... 10-2)
- when extending the computation of the Firth Loss to cosine classifiers, logistic regression weights are normalized... this is not compatible with the setting $\beta_0=0$. The associated demonstration needs to be revisited. (although intuitively if we omit this constraint on notations, we should be able to redo the same reasoning)

Extensive experiments clearly show the benefits of using the proposed bias reduction. To complement those results, it would be also interesting to have a final table with resuming the newly obtained SOTA results. 

Also it would be interesting to have an idea on the effective value of $\lambda$ to be used. Table A.2 shows a large set of values there. Could the author share what are the typical values observed on some configuration tested? (e.g. for figure 3?). Also what is the sensitivity of $\lambda$ on the performance?

# Minor remarks
- equation 1: $j$ is used in the summation in the denominator... it should be changed to e.g. $j'$
- just after equation (10), it should be mentioned in the text what is $\lambda$ used for.
- annex B: in the definition of the C term, $\lambda$ should appear

summary_of_the_review: Overall I am in favour of accepting this paper. The strength of this paper is on the theoretical justification of the proposed regularization term introduced associated to the general Firth bias reduction. Also extensive results are carried out to evaluate fairly the benefits of the proposed approach on SOTA solution.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper tackles the few-shot learning problem, which is a hot topic in the representation learning field. Specifically, the authors aim to train accurate classifiers using a small number of samples – Maximum Likelihood Estimators are biased for them. As a result, the authors propose to utilize Firth’s Penalized Maximum Likelihood Estimator to modify the ordinary MLEs, and prevent the bias when training classifiers for few-shot instances. The authors have validated the effectiveness of Firth bias reduction from multiple aspects, and the experimental results show the performance improvement is consistent across various circumstances, e.g., with different backbones, data distributions, and classifiers.
main_review: This paper has multiple advantages.
-This is the first paper addressing the effectiveness of Firth bias reduction in few-shot learning.
-Vast empirical study validates the proposed method consistently improves the performance of the baseline method, and no performance penalty is introduced.
-Apart from typical logistic regression models, the firth bias reduction can also be applied to cosine classifiers.
-Comprehensive discussion about other regularization techniques is provided.

I also have several questions and concerns about this paper.
-The relative improvements are validated through empirical investigations. I expect to see the theoretical guarantee for the improvement of firth bias reduction in few-shot learning.
-The idea of this paper partially comes from [1], which weakens the novelty of this manuscript. I would suggest the authors discuss more the extensions of [1] in current machine learning tasks, and discuss the difference with them.
-All the experimental metrics are reported by relative improvement, making it hard to be compared for other upcoming works. I suggest the authors report the absolute accuracy on benchmark datasets in the supplementary material.

[1] D. Firth. Bias reduction of maximum likelihood estimates. Biometrika, 80(1):27–38, 1993

summary_of_the_review: To conclude, this paper is clearly elaborated with sufficient experimental results and promising improvement. I suggest a ‘weak accept.’
review_title: null

Meta-review: This work starts from the observation that maximum likelihood estimation, while consistent, has a bias on a finite sample which is likely to hurt for small sample sizes. From this, they apply Firth bias reduction to the few-shot learning setting and demonstrate its empirical benefits, notably relatively to L2 regularization or label smoothing alternatives.

After some discussion with the authors, all reviewers are supportive of this work being accepted. Two are also suggesting this work be featured as a spotlight.

The proposed method is simple, well motivated, and appears to be effective. Therefore, I'm happy to recommend this work be accepted and receive a spotlight presentation.
Decision: Accept (Spotlight)