Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: This paper proposes a novel few-shot classification algorithm based on distribution calibration. The proposed work is mainly based on Free-Lunch (Yang et al., 2021), where a feature extractor is first trained using base classes and the features of few-shot classes are augmented by the additional features drawn from Gaussian distributions. Here, the mean and covariance of the Gaussians are constructed by adequately adapting the statistics computed from the base classes, and Yang et al., 2021 proposed a simple heuristic approach for that. This paper suggests using a more theoretically grounded approach for this procedure, where the transfer of statistics from base classes is designed to be more sensitive to the importance of individual samples included in base classes or novel classes. Specifically, the paper introduces a hierarchical optimal transport algorithm where the relevance between the base and novel classes is estimated via solving hierarchical optimal transport problems and using the estimated transition matrices for constructing parameters for Gaussians to generate novel features. 
strengths_and_weaknesses: Strength
- The paper is well-written and easy to follow.
- The hierarchical optimal transport approach seems reasonable and sound.
- The experimental results are extensive and convincing. The paper also provides an ablation study and helpful visualizations.

Weakness
- The algorithm can be considered somewhat incremental because most of the procedure is based on (Yang et al., 2021).
questions: - How does the running time of the proposed algorithm (especially the time for solving the hierarchical OT problem) scale with the number of data ($N$ or $K$)? Does hierarchical OT incur a serious computational bottleneck?
- Can you notice a notable difference in the learned transport plan when there is a significant shift between base and novel classes? For instance, in the case of the cross-domain setting?
- Regarding the hierarchical optimal transport, I could find a relevant paper (Yurochkin et al., 2019) where the goal is to measure the distance between documents via hierarchical optimal transport; to better capture the semantic difference between documents, the paper proposes to first solve the lower-level topic space and use the learned topic transport for solving the higher-level document space optimal transport. I guess it is worth citing this paper and discussing its relevance. 
limitations: As I mentioned before, most of the algorithm is based on Free-Lunch (Yang et al., 2021) and the only different part is the construction of Gaussian parameters through advanced weighted sums where the weights are computed from hierarchical optimal transport. Still, I think the contribution is quite valuable, and the algorithm demonstrated excellent performance.
review_title: null

final_score: rating: 5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.
review_content: summary: This paper presents a calibration-based few-shot learning method to address the problem of the biased model toward novel samples. The main tool to that end is optimal transport (OT), and thereby this work attempts to overcome the limitations of its preliminary work [14] by replacing Euclidean distance with a proposed distance based on OT that can measure the distance between two distributions in a way of obtaining matching cost of one distribution to another. In order to get such a matching cost matrix, the paper suggests to learn those costs between base classes to novel samples by a low-level OT optimization problem, where the goal is to capture the importance of a sample with respect to its class. The experimental results show that the proposed method performs better than the existing few-shot learning methods including Free-Lunch [14]. 
strengths_and_weaknesses: (Strengths)
1. [14] is a seminal work with clear novelty, and this work adequately finds room for improvement and indeed improves the performance of [14] by designing a more statistically sophisticated distance metric. 
2. Experiments are well designed and conducted, and the proposed H-OT method seems to work well in most of the settings. 
3. In particular, H-OT is highly effective in cross-domain settings, compared to Free-Lunch.

(Weaknesses)
1. There is no experiment to support the limitations of Free-Lunch as explicitly claimed by the authors. Even though it is observed that the overall performance has been improved, it is not quite clear whether such a performance gain indeed comes from addressing the limitations of Free-Lunch. 
2. The presentation can be further improved as the current notations are too complicated and some consecutive sentences are not smoothly linked to each other. 
3. In the overall performance comparison (Table 1), the performance gain of H-OT is somewhat marginal particularly in 1shot settings. 

questions: 1. It would be better if any specific example is given to illustrate the problem of Euclidean distance is truly addressed by the H-OT distance.
2. The main strength of this work seems to lie in the effectiveness of cross-domain settings, and hence this part can be more emphasized throughout the paper.
3. In Sections 3.2-3.3, notations are too complicated and descriptions are not very effective. This part can hopefully be revised in a much simpler manner.
limitations: Yes.
review_title: null

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: This work studies few-shot learning and proposes a optimal-transport-based weighting scheme to leverage examples from base classes for updating the classifiers of the learning of novel classes. Built on the framework of previous work [14], this work develops a optimal-transport-based weighting scheme instead of euclidean distance, and the resulting weights per base class implicitly considers similarity of individual examples within that base class to examples in the novel classes. Evaluation shows improved performance over [14] and competitive performance on standard few-shot image classification benchmarks across multiple backbones. Extensive ablation studies are conducted to verify superiority to [14].
strengths_and_weaknesses: Strength

1. Extensive experiments across multiple backbones, cross-domain results and ablation studies. Qualitative results on optimal transport weights helps understanding of the approach.
2. Code provided for reproducing experiment results.

Weakness

1. The improvement over [14] in performance is relatively small with respect to how "principled" (L.59, L.191) the approach is. It would be nice to get an understanding on 
- How much room of improvement is available from distribution calibration, to put the improvements that has been extracted by the proposed approach into perspective. For example, if there's oracle base class weighting, how good the performance can be on the query set? 
- Are we moving in the right direction? For example, do the class weights (e.g. Appendix Figure 4) match the oracle weights? From [14] to this paper, are weights extracted looking more like oracle weights? 

2. Would be nice to demonstrate how the per-example weights Mjn catches differences among examples in the same class. For example, baby is similar to sofa. Does Mjn tell apart sofa images with a baby and sofa images without a baby? How do the transport probabilities generalize in cross-domain settings?

======

The authors' response has adequately addressed my concerns. I have updated the score.
questions: See weakness.
limitations: Limitations are sufficiently addressed.
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: This paper is on the topic of distribution calibration methods for few-shot learning, which seek to align the novel class feature distribution to that of the base classes. Methodological and experimental comparisons are primarily with a prior distribution calibration method (Free Lunch [14]) used the average features of the two closest base classes in Euclidean space. The authors instead propose using optimal transport to compare novel classes with all base classes. Additionally, the cost function to compare a base class with a novel sample is also learned as an optimal transport problem. Experiments are done on 5-way 1-shot and 5-way 5-shot miniImageNet, TieredImageNet, CUB, and CIFAR-FS, in comparison with Free Lunch and other popular few-shot methods. Overall, results are impressive, establishing a new state-of-the-art.
strengths_and_weaknesses: Strengths:
1. The proposed usage of optimal transport to align base and novel distributions is very sensible, as are the improvements over the previous distribution alignment method Free Lunch. It’s almost surprising that such an approach hasn’t been attempted before.
2. Background concepts (e.g. few-shot learning, optimal transport) are introduced well for unfamiliar readers, though some of the writing could use some improvement (see below), and the authors should take care to avoid text that is too similar to prior works (see below). 
3. The empirical results are impressive. The main results are on 5-way 1-shot and 5-way 5-shot miniImageNet, TieredImageNet, CUB, and CIFAR-FS, which are the common FSL benchmarks in recent works. While all comparisons aren’t on the same backbone, the backbones used are listed in Table 1. Regardless, I believe these are the highest accuracies I’ve seen on these few-shot benchmarks, and the improvements over the most relevant baseline are consistent. Additional experiments in cross-domain settings and ablation studies lend further support to the efficacy of the method.

Weaknesses:
1. There isn’t a discussion on computational cost of this method. Optimal transport has a reputation for being expensive, so such a discussion would be very helpful, particularly given the two-level OT used here. How long does this method take to compute the transport plan? 
2. Similar to the above point, the computational cost of the optimal transport plan scales with the number of classes involved. While this may be manageable for the small size of common FSL benchmarks, I suspect that datasets with larger numbers of classes may prove more expensive.
3. This work seems heavily based on the “Free Lunch” paper [14], not just in methodology, but also the paper itself. Some similarities are perhaps natural, as this work takes a similar strategy and builds off of [14]. However, Section 2.2 is mostly a paraphrase of [14]’s Section 3.1, there are strong similarities in certain lines of the Related Works, and several of the tables have the same formatting. There are also some odd variable substitutions from those of [14] that seem unnecessary (e.g. topw instead of the more common top-k, \beta for Tukey’s Ladder of Powers transformation instead of the more common \lambda).
4. Writing: While generally understandable, the writing isn’t always idiomatically or grammatically correct. I strongly recommend another thorough round of edits. A non-exhaustive list of examples appear under “Miscellaneous” below.

Miscellaneous:
- Line 21: [1] is a strange choice as the only cited reference for image classification.
- Line 30: “introduce to calibrate”
- Line 38: “Although with reasonably good performance”
- Line 51 (and elsewhere): “labelled” vs “labeled” <= Both spellings are correct, but generally one should stick to a single version.
- Line 75: “Owing a rich theory” <= “Due to its rich theory”? Though the “rich theory” isn’t the reason why people use OT for comparing probability distributions.
- Line 87: “We divide whole dataset” <= missing “the”
- Line 89/98: B is being used to denote both the number of base classes and the set of base classes.
- Line 99: “to generate from” => “to be generated from”
- Line 167: While I see why using only the mean may be inaccurate, I fail to see why it’d be “severely biased”.
- Line 175: “each distributions”
- Line 176: “should subject to”
- Line 214: incorrect usage of “Besides”
- Algorithm 1: “t” should be in math mode. “test the query set on classifier \theta” => “test classifier \theta on the query set”
- Line 238, 269: missing Oxford comma.
- Datasets shouldn’t be preceded with the word “the” unless followed by the word “dataset”
- Table 1: Why aren’t the best results on CUB bolded?
- Line 269: “CIFAR-FAS”
- Line 341: Starting a sentence with “And”
- [2] and [5] in the references are the same paper.

questions: 1. Line 263: Why must the features be nonnegative? 
2. How does the number of base classes affect this method? I presume that a higher number of classes scales the computational cost of this OT-based approach significantly, in a way that previous approaches (e.g. ProtoNet) do not.
3. Free Lunch [14] found that increasing the number of top-k base classes decreased performance, while the proposed method is somewhat akin to k=B and yet sees improvement. Is the difference due to the low-level OT method?

limitations: Discussion of limitations absent. What is the added computation cost of the method? When does OT tend to fail? How limiting is the Gaussian assumption of features?
review_title: null

Meta-review: metareview: This paper builds off of the distribution calibration approach for few-shot learning known as “Free Lunch,” replacing the Euclidean metric with Hierarchical-Optimal Transport. The result is a more principled and empirically effective approach. The main issues among the reviewers were concerned with the limitations of Free Lunch and whether this new approach could overcome them, and what sort of additional computational cost would be incurred by using optimal transport. While more expensive, it is felt that overall the approach is sufficiently novel and effective.

confidence: Certain
Decision: Accept