Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: The goal of the paper is to improve the robustness of classifier for task-specific generalisation in the meta-testing stage.  The work introduces Vicinal Risk Minimization in which a Gaussian distribution is fitted on the unlabeled data samples and the probabilities of the labeled samples are obtained using lazy random walk. The work proved the expected vicinal loss functions for cross-entropy loss and SVM. Experiments on miniImageNet, CUB, CIFAR-FS for few-shot learning demonstrate the effectiveness of the proposed approach.
strengths_and_weaknesses: Strengths and Weaknesses:

* The paper is well-motivated and easy to follow. The overall idea of constructing vicinal distribution of the training samples using unlabeled training data is novel

* The experimental set-up is sound and demonstrates the usefulness of the approach. The results show the improvement in the generalization performance of the models with the proposed method. The experiments are extensive and outline the set-up clearly.  

* Thorough ablations in Table 3 clearly show the advantage of using the VRM principle over the CE-Mixup principle. 



There are no major weaknesses in the paper, however, certain claims in the manuscript can be supported with citations. There are a few typos and can be corrected in the revision.



 - Minor line 36: “the significant deviation of the empirical distribution..”  Add citation here?

 - Line 81: benchmars-> benchmarks

 - Line 99: It [37] -> In

 - Lime 110: . Besides, (author?)

 - Line 126: tge -> the

 - Line 133: expect loss -> expected loss

 - Line 177 it replace->replaces
questions: * "Our approach also considers Gaussian vicinities but estimates the mean by the unlabeled data, which is generally not on the training samples". How is this a better formulation?

* What is the dimensionality of the Gaussians being considered here? How realistic is the approach as the dataset scales both in terms of number of samples and size of each image.

limitations: The limitations have been adequately addressed.
review_title: null

final_score: rating: 4: Borderline reject: Technically solid paper where reasons to reject, e.g., limited evaluation, outweigh reasons to accept, e.g., good evaluation. Please use sparingly.
review_content: summary: This submission points out that, in few-shot learning, a limited number of support data can hardly represent the true distribution, which results in poor generalization performance of the empirical risk minimization based methods. To tackle this problem, the authors propose to minimize the vicinal risk and estimate parameters of vicinal distribution of each sample by a random walk algorithm with additional unlabeled data. They demonstrate that their algorithm outperforms baseline methods on three datasets.
strengths_and_weaknesses: Strengths:
1. This paper points out an interesting problem for few-shot image classification.
2. The experimental evaluation is adequate, and the results convincingly support the main claims.

Weaknesses:
1. The paper is not very clear, and many aspects make the reviewer's understanding of the entire paper very confusing. Moreover, many typos should be carefully checked before submission. 
2. The main concern is that the core idea is not innovative, similar to the incremental superposition of multiple algorithms. 
3. Some experimental settings are unfair, making it impossible to measure the effectiveness of the method in terms of performance gains.
4. See questions for more details.
questions: Methods:
1. The pre-training phase is the meta-training phase? if not, the authors should detail the learning process of ADV in the meta-training and meta-testing phases.
2. The main problem is the difference between ADV-CE and ADV-SVM methods. They seem to be two separate branches, and ADV-CE significantly outperforms ADV-SVM. Therefore why provides AVD-SVM?
3. If i=j, the D_{ij}^{UU} = 0 in equation (11)?

Experiments:
1. How many query samples are used as unlabeled data to estimate the distribution parameters in experiments?  In general, the commonly used sample size is 15, and the more samples, the better the performance.
2. In lines 238-239, the authors propose that the support set can deviate from the true distribution. If there is a visualization to support this claim? 
3. In the meta-testing phase, the meta-learning methods (e.g., MAML) are optimized by using 5-10 steps in the fine-tuning phase and the more steps have good performance [1]. However, ADV has 100 steps in the fine-tuning phases. if ADV also uses the same steps (5-10), how does it perform?   
4. Only four transductive algorithms (TPN, FEAT, TEAM, SIB) are tested with the proposed method, it would be more convincing if more recent state-of-the-art algorithms can be considered.
5. The performance gain of the proposed method seems to diminish quickly from 1-shot to 5-shot settings. Does this mean in the case of 5 (or more)-shot setting, considering unlabeled test data with the proposed method could even adversely affect the performance?

Typos:
1. imporving --- imporve in the title
2. author? in the 110 line 
3. 4.0.1 in the 222 line
4. ishas in the 254 line

[1] How to Train Your MAML to Excel in Few-Shot Classification
limitations: Yes, they discuss the limitations in Section 5.
review_title: null

final_score: rating: 5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.
review_content: summary: In this paper, the distribution of support data is estimated using transition probability between all samples within a meta-test task (including labeled support data and unlabeled query data). A logistic regression or an SVM model is learned via vicinity risk minimizing based on the estimated distribution of support data. The expected vicinity loss is derived in the paper. The experiments indicate that the proposed method outperforms some few-shot learning methods.
strengths_and_weaknesses: The expected vicinity loss is derived.

Estimate the distribution of support data using weighed unlabeled data, where the weight is based on the transition probability in lazy random walk.


Weaknesses:

Some typos in the manuscript. Some technical details are missing. See the next section for details.

The proposed method is a transductive few-shot learning method as it uses unlabeled data in meta-test tasks. Therefore, it is important to compare to transductive few-shot learning methods with the same backbone. However, the comparison to transductive methods in Table 1 is not fair. For example, the reported results of some methods (e.g. FEAT) are based on shallow backbones (4-layer CNN). It is better to reproduce those methods with the same backbone (WRN), especially since the code is available online for those methods.

In addition, some recent transductive few-shot methods [1, 2, 3, 4, 5] achieve stronger performance using the same backbone (WRN). The proposed method should be compared with those methods to show the effectiveness of adaptive vicinity risk minimization. Since the authors claim that the proposed method outperforms SOTA few-shot learning methods, it is necessary to compare the true SOTA methods in a fair manner.

[1] Boudiaf, Malik, et al. "Information maximization for few-shot learning." Advances in Neural Information Processing Systems 33 (2020): 2445-2457.

[2] Ziko, Imtiaz, et al. "Laplacian regularized few-shot learning." International Conference on Machine Learning. PMLR, 2020.

[3] Cui, Wentao, and Yuhong Guo. "Parameterless transductive feature re-representation for few-shot learning." International Conference on Machine Learning. PMLR, 2021.

[4] Lee, Dong Hoon, and Sae-Young Chung. "Unsupervised Embedding Adaptation via Early-Stage Feature Reconstruction for Few-Shot Classification." International Conference on Machine Learning. PMLR, 2021.

[5] Qi, Guodong, et al. "Transductive Few-Shot Classification on the Oblique Manifold." Proceedings of the IEEE/CVF International Conference on Computer Vision. 2021.
questions: Typos in line 110. 

What is "tge" in line 126? 

According to line 210 and 211, it is not clear how to compute $u_i$ and $\Sigma_i$. The definition of $V_i$ is not given in line 211. Is it a typo? Is it $v_i$ or $V_i$? It is better to give equations about how to compute $u_i$ and $\Sigma_i$.

What is the value of $N_u$?
limitations: The author discussed limitations in the paper.
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: The paper proposes to use vicinal loss function in few shot classification to increase the generalization capability.  The vicinal loss is associated with  some prior distributions over the features, which is adaptively obtained via random walk algorithm with the assistance of a set of unlabeled examples.  
strengths_and_weaknesses: * Originality: The task of few shot learning and the techniques of vicinal loss and random walk algorithms are not new. To the best of the reviewer’s knowledge, the use of these techniques in few shot learning is novel. However, there is a lack of comparison with other methods（eg [15]）use additional unlabeled data. 
* Quality: All claims are supported by the experimental results and the work is complete. The reviewer do not agree with the statement that the proposed method can beat SOTA. 
* Clarity: The idea of the paper is presented clearly. However, there are many typos, especially in the equations.
	* typos (an incomplete list): 
		* L110: citation is not right
		* L126: should be “the” not “tge”
		* L130: model is $F(x;\theta)$ not $f_{\theta}(x)$
		* L140: model parameter is $\theta$ not $w$
		* L146: the density function is a function of $\tilde z$ on the LHS but does not depend on $\tilde z$ on the RHS
		* L71: the expectation is with respect to z not $\bar z_i$
	* technical error: 
		* L133-L135: Although the form of $dP_{\delta}(x,y)$ is given in the original paper, this is technically not right. The decomposition is saying that x and y are independent. The vicinal loss from a Bayesian point of view can be seen as adding a prior on the predictor x
* Significance: The paper proposes to use a loss function that is different from the conventional empirical loss under few shot learning setting. The motivation for this approach may be insightful for developing  new methods in few shot learning. The results, while not earthshaking, may be of practical use.
questions: * The reviewer believe the statement in L133-L135 is technically not correct, can the authors address this concern?
* The reviewer believe the vicinal loss is the same as using prior distribution on the features, can the authors clarify?
* The details of the ablation study is not very clearly presented in the paper.  For example, it is not clear to the reviewer what the delta vicinal distribution is and how it is different from the distribution used in ADV-CE.
* Unlabeled data.
    * The paper does not provide any information on the size of unlabeled data used in the experiment.
    * Does the proposed method give higher classification accuracy with more unlabeled data? This should be shown in the ablation study.  
limitations: Some limitations of the work has been briefly mentioned by the authors. The limitations and questions of the paper the reviewer had is given in Questions section.
review_title: null

Meta-review: metareview: In this paper, authors study a few-shot learning setting where the training distribution deviates from true distribution. To achieve a more accurate approximation of the true distribution, authors propose assuming Gaussian-like vicinal distribution around each training data point which results in a vicinal loss. Extensive empirical results in the paper show that the proposed method improves over the baseline methods. While vicinal loss has been used in other settings, this problem formulation and use of vicinal loss is novel and the empirical advantage is significant. Therefore, I am recommending acceptance. Given reviewers' concerns about typos and presentation issues, I encourage the authors to make a few more passes over the paper and improve the writing and presentation.
confidence: Certain
Decision: Accept