Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: The paper explores the importance of the decision boundary in few-shot learning in language models with GPT. Then, the authors propose an improvement over the existing few shot methods by adaptively learning this decision boundary. The decision boundary is learned by modeling the output distributions of GPT few-shot model as GMM and fitting to the class labels. The results shown are better than the baselines up to 13%. The ablation study is done by measuring the effect of template choices, class imbalance and class permutations. The robustness is measured by running with several random initializations.
strength_and_weaknesses: Strengths:

* The experimental results and applicability are well supported by using multiple datasets and their ablation study. Substantial improvements are obtained in terms of the accuracies. 

* The idea of focusing on the decision boundary is well motivated and demonstrated. 

Weaknesses:

* The label assignment part and multiple initializations and runs in the proposed algorithm can be computationally heavy as the authors mentioned. However, this might not be concerning since it is performed at the output space after the training.

*  Why the authors chose 4 and 8 shot in their experiments? It looks limiting and not reasoned well. Is it a standard procedure for these datasets?

* Although the authors mentioned and used averaging and careful initializations, the stability of the method is still a bit concerning to me. Too much unstable/sensitive methods are combined and dataset and task itself also poses another sensitivity issue by nature and since they are scarce. However, the results suggest otherwise unless it is 0 or 1 shot.

* It would be good to explore the boundaries of the method in terms of the original dataset/labels and the desired dataset/task, and a notion of closeness between these tasks. 

clarity,_quality,_novelty_and_reproducibility: The paper is written clearly and with a good writing quality. The experimental section is also extensive and well written. 

In terms of the originality, it is on the incremental side since 1-the decision boundary problem in few shot is known in the literature, 2-all the methods are borrowed from the literature. However, the authors did a good job with the presentation of the decision boundary problem and the way the methods are used and context can be considered as novel. 

The authors provide the code, I skimmed through the lines but I did not run their code. I would say that it was reproducible from my impression, but I cannot comment in a certain way.
summary_of_the_review: The authors propose a method where they learn decision boundary for an in context GPT few shot learning method. The motivation, story and the experiments parts are the strongest side of the paper. The ideas and results are plenty and well supported, which surpass the disadvantage coming from the computational complexity.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: The paper proposes prototypical calibration for zero-shot and few-shot classification tasks when evaluating large language models. The approach first uses Gaussian mixture distribution to estimate the prototypical clusters for all categories of the classification task and then assigns each cluster to the corresponding label by solving a weighted bipartite matching problem. Each example’s prediction is calibrated by the likelihood of the prototypical clusters. They demonstrate significant improvements over contextual calibration (Zhao et al., 2021) on a wide range of tasks.

strength_and_weaknesses: Strength
- Very strong and consistent empirical results from calibration. The performance is exceptionally strong for relatively smaller-sized models with fewer in-context learning examples. 


Weaknesses
- There exist related works, though they do not necessarily claim to develop calibration approaches, they share the same spirit as this method to modify the output distributions for zero-shot or few-shot approaches [1][2]. Discussing these previous works and comparing results with them would help better position this paper. 
- Related to the previous point, I think the related work section does not present the best relevant paper, especially the first section. I suggest the authors conduct a more thorough literature review, along with the direction of calibrating language models. 
- I would also love to see if the calibration method works for masked language models given that MLMs have much stronger zero-shot performance on these evaluation datasets. 

[1] Surface Form Competition: Why the Highest Probability Answer Isn’t Always Right 
[2] Noisy Channel Language Model Prompting for Few-Shot Text Classification

clarity,_quality,_novelty_and_reproducibility: The paper is written very clearly and easy to follow. The idea is novel and I believe it's easy to reproduce the results. 
summary_of_the_review: The paper has a novel idea for calibrating language models and shows strong empirical improvements over baselines. One concern is that the paper does not discuss a few critical related works or show empirical comparisons to these works.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: This paper presents a highly empirically effective approach for in-context learning using so-called _prototype calibration_. The simple, yet dramatically effective approach is well motivated by the authors and demonstrated to be empirically effective under a wide variety of settings, testing sensitivity to number of shots, prompt template, class imbalance and more. 
strength_and_weaknesses: Overall I believe this to be a very strong empirical investigation into in-context learning. The proposed approach is clearly motivated, simple and seems to be highly effective. 

Strengths:
* Dramatic performance benefits from the proposed approach. 10+ pt improvement in most cases on average, 5+ in a few others (Table 1). ]
* Detailed empirical analysis that includes: effectiveness across templates, demonstration perturbation, class imbalance, estimate set construction, estimate set size, 
* The results and analysis are clearly presented as are the methodological approach.
* Provides basis for future work on using using prototypes/clusters in the midst of in-context learning

Weaknesses: 
* It might be interesting to consider how the fit of the GMM distribution effects the downstream task performance. Also since the number of points in the estimate set is not so many, it could be interesting to consider a wider array of clustering inference methods including more exhaustive/expensive methods. 
clarity,_quality,_novelty_and_reproducibility: The presentation is clear and quality of experiments are apparent to me.
summary_of_the_review: This is a strong paper which presents a simple and effective approach for in context learning. The approach uses a GMM cluster-based approach to make predictions. The effectiveness of the approach is demonstrated through extensive experiments that evaluate the performance along many dimensions of number of shots, prompts, estimate set, etc.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This work presents the analysis that decision boundary is of critical importance to the performance of few-shot demonstrations and the traditional decision boundary leads to the fragility of prompting LMs. The authors propose prototypical calibration to adaptively learn a more robust decision boundary for few-shot learning of language models.
strength_and_weaknesses: Strength: The analysis that decision boundary is of critical importance to the performance and Instability of few-shot demonstrations is interesting and suggestive. The authors propose prototypical calibration to adaptively learn a decision boundary for
few-shot classification of language models. They present provide rich experiments to show the effectiveness of their proposed PROCA on a wide range of text classification tasks.


Weaknesses: The paper is somewhat vague for me and I am still confused about some notions or symbols. For example, μn and Σn should be placed with vector and matrix symbols. I am confused the relationship between μn and μn,l. Does the {μn,l} is the l-th element of μn or they are different variables? Since tuple k is a cluster-label assignment, does this mean that k represents a matrix of N * N?  The authors transforme the assignment score to a weighted bipartite matching problem between N clusters and N labels. It would be better if the authors can provide more ditails about the  bipartite matching problem. In my opinion, the key problem in this work is learn cluster-label assignment, which however omits useful details and also overall algorithm. Since we are dealing a N way few shot problem, whether the few labeled samples in N way few shot task are utilized for the learning of assignment score. Besides, the authors utilize a small-scale unlabeled in-domain dataset, named as estimate set (Desti), to estimate the parameters. Does this mean the estimate set is sampled from "N way" without labeling?  In that cases, is the PROCA still fair for a typical N way few shot setting? Maybe the authors can clarify the estimate set  from this view.



clarity,_quality,_novelty_and_reproducibility: For me, this submission omitts some useful information, making it difficult to understand the novelty of this work.
summary_of_the_review: Although calibrating prototypes for few-shot learning sounds reasonable, I have difficulty in understanding the specific approach of this submission, especially 3.2 and 3.3.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: This work proposes an alternative to naive greedy decoding during generation in few-shot learning. Namely, instead of simply selecting the token with highest probability, they first adopt a Gaussian mixture model to estimate the distribution for different categories under clusters, then perform weighted bipartite matching to match clusters to corresponding labels so as to calibrate the decision boundaries.

The reviewers found the work generally well written, and the empirical results from calibration are strong and quite convincing.

All reviewers leaned toward accept with the exception of one reviewer who gave a score with low confidence and whose primary criticism was lack of clarity. Unfortunately, they also did not respond during the discussions. I agree with the majority consensus.
justification_for_why_not_higher_score: + Novelty is somewhat incremental
+ Related works are not explained much or compared against in experiments
+ Method can be potentially unstable under several settings
justification_for_why_not_lower_score: The reviewers found the work generally well written, and the empirical results from calibration are strong and quite convincing.
Decision: Accept: poster