Unleashing the Multilingual Encoder Potential: Boosting Zero-Shot Performance via Probability Calibration
Authors: Ercong Nie
Helmut Schmid
Hinrich Schuetze
Venue: EMNLP 2023 Findings (2023)
Keywords: prompt learning
multilingual encoders
calibration
TL;DR: We introduce several calibration methods to prompt learning of multilingual encoders and improve the zero-/few-shot performance.

Abstract: Pretrained multilingual encoder models can directly perform zero-shot multilingual tasks or linguistic probing by reformulating the input examples into cloze-style prompts.
 This is accomplished by predicting the probabilities of the label words at the masked token position, without requiring any updates to the model parameters.
 However, the performance of this method is limited by the model's bias toward predicting label words which frequently occurred during the pretraining. 
 These words typically receive high probabilities. 
 To address this issue, we combine the models with calibration techniques which modify the probabilities of label words predicted by the models.
We first validate the effectiveness of a proposed simple calibration method together with other existing techniques on monolingual encoders in both zero- and few-shot scenarios.
We subsequently employ these calibration techniques on multilingual encoders, resulting in substantial performance improvements across a wide range of tasks.

Unleashing the Multilingual Encoder Potential:
Boosting Zero-Shot Performance via Probability Calibration
Ercong Nie!.
Helmut Schmid
Hinrich Schütze!. 1.2
Center for Information and Language Processing (CIS), LMU Munich, Germany 2 Munich Center for Machine Learning (MCML), Germany
nie@cis 1mu. de
Abstract
For example, the sentiment analysis of assigning the product review "Worked as stated!" to class POS can be reformulated as: "Worked as stated! All in all, it was [MASK]." The model is re- quested to fill in the word "good" at the mask token position, which is mapped to the POS label.
Pretrained multilingual encoder models can di- rectly perform zero-shot multilingual tasks or linguistic probing by reformulating the input examples into cloze-style prompts. This is ac- complished by predicting the probabilities of the label words at the masked token position, without requiring any updates to the model pa- rameters. However, the performance of this method is limited by the model's bias toward predicting label words which frequently oc- curred during the pretraining. These words typically receive high probabilities. To address this issue, we combine the models with calibra- tion techniques which modify the probabilities of label words predicted by the models. We first validate the effectiveness of a proposed simple calibration method together with other existing techniques on monolingual encoders in both zero- and few-shot scenarios. We sub- sequently employ these calibration techniques on multilingual encoders, resulting substan- tial performance improvements across wide range of tasks'.
However, earlier studies indicate that the output of masked token prediction is biased towards cer- tain label words in the candidate token list (Weiss- weiler et al., 2022; Nie et al., 2023). This bias not only affects the predicted class probabilities (Holtz- man et al., 2021; Ahuja et al., 2022), but also deteri- orates the model's overall performance (Zhao et al., 2021; Lu et al., 2022). According to Weissweiler et al. (2022) and Zhao et al. (2021), label words with higher frequency in the pretraining corpus tend to be predicted with higher probabilities, Besides, the prompt context can also influence the degree of bias present in the masked token prediction.
000
075
1 Introduction
Prompt-based learning (Brown et al., 2020; Liu al., 2021) has emerged as an important research area. Recent research demonstrates that multilin- gual encoder models are capable of accomplishing cross-lingual learning (Zhao and Schütze, 2021: Huang et al., 2022) and linguistic prob- ing (Shapiro et al., 2021; Hartmann et al., 2021) by using cloze-style prompts. These prompts consist of input sample, task-specific context and mask token. The encoder model applies Masked Language Modeling (MLM) (Devlin et al., 2019) to generate predictions for the mask token using a selection of prescribed candidate tokens from the vocabulary. These predictions can be subsequently utilized for label classification or probing purposes.
050
0.0
2.00
0.55
2.2
0.5
0.90 555
335
cero-shottrross-innuull
Figure 1: Example of the model predictions bias. The graph shows the accuracy on the amazon polarity test data (equally distributed) as function of the classifica- tion threshold. æ-axis refers to the threshold probability of good to classify examples with the class POS. The best results are obtained by classifying examples as if the probability of good exceeds 0.96.
Figure 1 illustrates the impact of the above men- tioned biases on the model predictions. It shows the results binary sentiment analysis task with the BERTBase (Devlin et al., 2019) model. The prompt template and label words used for this example can
'The code and data for this work are publicly available: https:// thtbbb.ttttuttt wtw/ congg11 2aanng///aall /ctp/baaabaaooon

---

Method
Description
Probability Calculation
Source
CC
Contextual Calibration
ğ(y x t) Wp(y| Jr, t) b
Zhao et al. (2021)
@(y] x t) = log p(yla,)) ply
PMIpc
Domuin Conditional Pointwise Mutual Information
Holtzman et al. (2021)
CBM
Calibration By Marginalizatton
@(y) x2,) 1) =
Yang et al. (2023)
[X] Exeex p(rZ)
Penalty
Probability Penalty
@(y) x. () p(ylo,
Our proposed method
Table I:
Overview of Calibration Methods, y refers to the label words. the test dataset, æ is an input sample,
and t is the prompt template.
be found in Table 6. By shifting the threshold for predicting POS from 0.5 to approx. 0.95, the perfor- mance can be improved by more than 25%. Given only mask token as input, the model predicts 0.92 and 0.08 as probabilities for the label words good and bad, respectively. To tackle the bias in the dis- tribution of label words, our proposed solution in this work is combine pretrained encoder models with calibrution methods.
Calibration By Marginalization (CBM) Yang et al. (2023) are inspired by PMIpc- Unlike PMInc- CBM approximates p(y|ar f) in a more precise manner by computing its marginalized probabil- ity, as the third equation in Table 1 shows. For each prediction, the sum probability Ex |a',, are calculated by taking all test inputs into account.
2.2 Our Method: Probability Penalty
In this paper, we contribute by (1) proposing a simple yet effective calibration method that in- volves adding trainable penalties to the probabili- ties of the label words, (2) demonstrating its effec- tiveness in achieving performance enhancements comparable to other existing calibration techniques, (3) refining the calibration parameters with only few training examples for further improvement, and (4) boosting the zero-shot performance of multilin- gual encoders by introducing calibration methods,
Motivated by the observation in Figure 1 that simple shift in the model's output distribution can substantially alleviate the label bias, we propose a penalty-based calibration approach as the equa- tion in the last row of Table shows. The core idea is to introduce penalty term that is added to each individual label word probability. We initial- ize the corresponding parameter vector with the negative prior probabilities of the label words. We estimate these prior probabilities using the output distribution of MLM applied to a mask token as input.
2 Calibration Methods
Experimental Setup
2.1 Existing Calibration Methods
Dataset We first validate the effectiveness of the different calibration methods on several monolin- gual tasks. We study sentiment analysis using two datasets: binary Amazon Polarity (McAuley and Leskovec, 2013) and the English subset of 5- label Multilingual Amazon Reviews (Keung al., 2020), topic categorization using two datasets: the Ag News and Yahoo Answers Topics (Zhang et al., 2015), sentence pair classification using two datasets: English subsets of MNLI (Con- neau et al., 2018) and PAWS-X (Yang et al., 2019), and 5 datasets from the GLUE bench- mark (Wang et al., 2019): CoLA (Warstadt et al., 2019), MRPC (Dolan and Brockett, 2005), QQP, RTE (Dagan al., 2005), and WNLI (Levesque et al., 2012). For the evaluation of multilingual encoders, we use Multilingual Amazon Reviews, XNLI and PAWS-X. Besides, following Nie et al.
Contextual Calibration (CC) Zhao al. (2021) apply an affine transformation (Platt et al., 1999) to the original probabilities, as the first equation in Table 1 shows. The parameters of the affine trans- formation are obtained from the output probability distribution of the content-free input, e.g., the mask token, denoted Pof. W diag(P )-1 is the in- verse diagonal matrix of Pof and b is all-zero vector.
Domain Conditional Pointwise Mutual Informa- tion (PMIdc) Holtzman al. (2021) adjust the conditional class probability p(ylar, I) by dividing it with the prior probability p(ylt) of that class. We estimate p(ylt) for given template / using ML. with prompt created by instantiating the prompt template with an empty input.

---

Balanced datasets (Acc.)
Imhalanced datasets (PI Score)
AmmzzonsS
XNLI
CoLa
MRPC
QUI
RTE
WNLI
Arg-
BERT
calib,
54.6
41.3
41.2
39.5
39.2
CC
74.6
61.7
27.4
41
36.2
31.6
51.1
46.1
26.9
39.5
43.1
45.6
PM
62.1
70.8
29.9
37.9
32.1
33.8
51.3
44.3
49.5
38.2
30.4
43.7
CBM
73.6
71.3
33.6
42.9
45.2
49.3
49.9
30.6
52.6
50,9
42.3
51.1
Penalty
67.9
61.7
26.3
42.6
39.4
31.6
51.1
46.1
26.9
39.5
43.1
43.3
RoBERTauus
calib.
76.2
66.1
24.3
44.0
32.4
31.2
39.6
45.3
26.9
37.1
31.6
41.3
PMlloc
62.3
79.4
34.2
45.6
25.3
43.3
43.33
+89.4
27.
37.0
30.4
43.4
CBM
78.4
76.5
34.1
44.4
48..2
43.3
51.0
Pemalty
75.6
79.5
30.1
41.4
26.9
33.7
23.6
46.6
39.8
35.9
32.1
42.3
Table 2: Results of zero-shot calibration methods on monolingual tasks. Amazon-P refers to Amazon Polarity (binary classification). Amazon-S refers Amazon Star (5-way classification).
Taaee
AG News
Amazon-P
Pawsx
XNLI
Avg
nli based ZR
54.9
82.3
48.2
34.8
55.1
calibration
Penalty
CC
Penalty
CC
Penalty
CC
Penalty
CC
Penalty
CC
zero-shot
67.9
74.6
61.7
61.7
45.4
45.4
42.6
41.4
54.4
55.8
65.63.
75.71.0
67.87.6
71.0s.6
51.10.9
51.4p.9
42.0
41.21.9
56..6..
59.82.4
67.2a.
75.9..
71.94.4
72.23.2
51.01.1
50.71.0
42.70.s
42.50.9
58. 58.22.3
60.3
few-shot
4
67.93.9
76.60.7
73.4a.s
70.32.9
51.613
50.91.3
42.8o.s
42.8p.a
58.92.4
60.21.a
69.11.5
76.9o.1
75.22.3
71.81.2
51.611
49.9o.6
42.9u2
42.70.2
59.71.3
60.3as
69.61.7
76.9u.
76.01.0
71.41.2
51.4r.1
49.71.0
42.8o.s
42.6p.2
60.01.o
60.2p.6
RoBERTan Tause au
News 67.9
Amazon-P
Pawsx
XNLI
Avg
nli-based
45.3
58.1
calibration
Penalty
CC
Penalty
CC
Penalty
CC
Penalty
CC
Penalty
CC
zero-shot
0
75.6
74.1
79.5
79.5
45.4
45.4
41.4
39.8
60.5
59.7
75.62.a
77.21.5
77.4s.0
81.34.9
48.41.
48.41.4
45.90.9
14.81.5
61.8s.a
62.92.a
2
73.92.a
77.31.2
81.64.3
80.82.4
49.01.a
48.3o.0
46.30.7
45.8o.7
62.72.4
63.1
few-shot
4
74.51.9
77.61.0
82.24.4
79.61.s
49.3o.s
48.5p.9
47.2a.2
46.00.3
63.31.a
62.9.o .o
8
76.61.1
78. 1o.5
85.2
79.7
49.604
48.
47.
46.0r.
64.6o.7
63.0p.9
16
78.30.5
78.4na
85.1
79.71.6
49.4o.
48.10.4
47.00.2
46.00.9
65.0aus
63.1os.
Table 3: Results of few-shot calibration methods on monolingual tasks. nli-based ZR refers the NLI-based zero-shot classification baseline (Yin al., 2019).
(2023), we expand the AG News dataset 25 lan- guages using machine translation to conduct a wide range of cross-lingual analyses.
Results and Analysis
4.1 Results on Monolingual Encoders
4.1.1 Zero-shot calibration
We first validate the effectiveness of the various cal- ibration methods on monolingual encoders. Table 2 shows the results of zero-shot calibration, where we directly calculate the calibrated probabilities without using additional training samples. We re- port accuracies for evenly distributed datasets and Fl scores for imbalanced datasets. Compared to the uncalibrated baseline systems, we obtain im- provements across most of the tasks, except for the CC method combined with the RoBERTa model. In this specific case, the average performance wors- ens compared to the no calibration baseline due outlier performance observed in several tasks, such as Yahoo and CoLA.
Setup In our monolingual experiments, we use
the pretrained models bert-base-cased (De- vlin et al., 2019) and roberta-base (Liu et al., 2019). In the multilingual experi- ments, we use their multilingual counter- parts xlm-roberta-base (Conneau et al., 2020). We use PyTorch (Paszke et al., 2019) and the HuggingFace framework (Wolf et al., 2020). We repeat each experiment times with different random seeds and report the mean and variance. Details of the experimental setting can be found in Appendix A.
and

---

line proposed by Yin et al. (2019). Details of the baseline setting and the few-shot training process are described in Appendices A.3 and B.
70
76
Figure 2 shows the few-shot calibration results of the RoBERTa model on the AG News and NLI tasks. Prior research (Zhao and Schütze, 2021) showed that few-shot learning can be unstable due to the randomness. However, as Figure 2 shows, the variation in performance diminishes obviously as the number of shots increases. Our experimental results indicate that few-shot calibration not only enhances the performance but also increases the steadiness.
Accuracy Nouss
penalty calioration
nii-ased ZR
70
69
16
#Shot
(a) AG News
penalty
4.2 Results on Multilingual Encoders
CC calibration based
Table 4 shows our experimental results on multilin- gual datasets, indicating that calibration methods are also effective for multilingual encoders.
Accuraty on (1)
45-
Our experiments cover large range of lan- guages considering both language availability, i.e., or how much language data exists the pretrain- ing corpus, and language diversity, i.e., to which language family a language belongs. Specifically, for Amazon-S, XNLI and PAWS-X, we use the original test sets, mainly containing high-resource languages. In the multilingual AG News task, we include many low-resource and unseen languages by generating parallel multilingual test sets using machine translation techniques. Recent research by Hu ct al. (2020) and Liu et al. (2022) shows that automatically translated test sets are useful for measuring cross-lingual performance. Hence, we adopt their methodology and expand the language coverage of the AG News dataset to 25. The list of languages can be found in Appendix C.
40
(b) NLI
Figure 2: Performance and variation of few-shot cali- bration on the RoBERTa model.
4.1.2 Adding few-shot samples further boosts the performance
As the formulas in Table 1 show, PM and CBM directly modify the probabilities without introduc- ing additional parameters, while CC and Penalty use specific calibration parameters, which are train- able. In zero-shot calibration, these parameters are initialized by prior probabilities without being up- dated. We will now make use of the trainability of parameters in CC and Penalty to investigate if ap- plying few-shot training to calibration parameters further improves the performance.
The results on multilingual BERT and XLM-R show that all four calibration methods improve the multilingual performance averaged across all tasks. For both models, CBM always emerges the top-performing approach. Different from other approaches predicting the label with input by another, CBM is the only method which leverages the test (without labels) adjust the calibration parameters. This could account for the substantial advantage of CBM over the others in terms of the performance.
Table 3 shows the results of few-shot calibra- tion. We observe that training the calibration pa- rameters just few samples further enhances the performance of the calibrated systems. Com- pared zero-shot calibration, few-shot calibration achieves better performance most cases. We also compare calibration methods in few-shot scenarios with the NLI-based zero-shot classification base-
4.3 Multilingual Analysis
Now we analyze how different language proper- ties correlate with the performance of multilingual BERT on the AG News task.

---

AG News
Amazon-S
XNLI
PAWS-X
Avg.
mBERT lase
no calib.
32.8
20.5
33.6
33,9
30.2
PMIDC
48.8
22.5
33.6
44.4
37.3
CBM
53.8
25.1
34.9
49.2
(max)
53.9
23.9
35.1
44.8
+ Penalty (max)
54.6
23.8
35.3
47.1
40.2
XLM-Raase
+ no calib,
45.4
21.9
35.0
31,7
33.5
+ PMIDC
59.8
23.0
33.6
37.8
38,6
+ CBM
63.3
28,9
37.8
46,3
44.1
+ CC (max)
59.6
23.7
35.3
43,7
40.6
Penalty (max)
57.5
23.6
35.8
43.4
40.1
Table 4: Results of calibration methods multilingual datasets. We report the best results for CC and Penalty in different few-shot settings.
4.3.1 Language Accessibility
0.42
We first group the evaluation languages into low- resource languages, unseen languages, and lan- guages with unseen scripts to determine the in- fluence of language accessibility. Low-resource languages are languages which are contained in the pretraining corpus, but only account for small amount of it. Unseen languages do not occur in the pretraining, thus the multilingual encoder has never seen them. The hardest case involves languages with unseen scripts, where the model has not even encountered the characters of the language. How- ever, our test set contains no languages with com- pletely unseen scripts because machine translation frequently generates code-switched data. Figure 3 (a) shows that low-resource languages perform gen- erally better than the other two types of unseen lan- guages, indicating that the multilingual encoder's access languages in the pretraining is crucial for the performance enhancement via calibration.
0.33
0.32
0.22
E-00
0.22
6.10
0.05
ureen
no sofo:
10 scrpe
(a) Language Accessibility
Poralty Calbrotion
CBN Colibration
0.20
0.33
9.25
0.32
0.23
0.35
0.22
0.30
0.35
(b) Language Diversity
Figure 3: Performance Improvement multilingual BERT with calibration methods.
4.3.2 Language Diversity
Conclusion
We further group the languages according to their phylogenetic relationships. i.e., from which lan- guage family they are. We analyze the language families containing at least 3 languages. The box plots in Figure 3 (b) reveal that the impact of cali- brating multilingual encoders varies across differ- ent language groups. Specificallyy observe that Indo-European and Dravidian languages tend to benefit more from calibration than Austronesian and Niger-Congo languages.
In conclusion, our work focuses boosting the zero-shot learning performance of multilingual en- coders in language understanding tasks through probability calibration techniques. We address the bias issue in the mask token prediction bel words by introducing various calibration tech- niques that modify the probabilities of these words. We first test the efficacy of different calibration methods in monolingual encoders. We also prove that with minimal number of training exam- ples. the calibrated probabilities yield further en- hancements compared the zero-shot calibration method. Our experiments on multilingual encoders demonstrate that all calibration methods bring a performance improvement across various tasks.
This discrepancy suggests that the effectiveness of calibration techniques can be influenced by the language accessibility of multilingual encoders and the linguistic characteristics of language families.

---

Limitations
Guzmán, Edouard Grave, Myle Ott, Luke Zettle- moyer, and Veselin Stoyanov. 2020. Unsupervised cross-lingual representation learning at scale. In Pro- ceedings of the 58th Annual Meeting of the Asso- ciation for Computational Linguistics, pages 8440- 8451, Online. Association for Computational Lin- guistics.
We propose a simple yet effective calibration method to enhance the zero-shot performance for monolingual and multilingual encoders. While our work shows the effectiveness of calibration for en- hancing the prediction with multilingual tasks, it is important note that our research is primarily focused on classification tasks with multilingual encoders. As a result, our findings and proposed methods may not directly translate to generation tasks, such as question answering (QA), which in- volve the use of generative multilingual models. Future investigationns should explore the application of our calibration methods on generation tasks and evaluate their effectiveness in enhancing the per- formance of generative multilingual models. This extension could provide valuable insights into the potential benefits and limitations of our approaches across a broader range of NLP tasks.
Alexis Conneau, Ruty Rinott, Guillaume Lample, Adina Williams, Samuel Bowman. Holger Schwenk, and Veselin Stoyanov. 2018. XNLI: Evaluating cross- lingual sentence representations. the 2018 Conference on Empirical Methods in Nat- ural Language Processing, pages 2475-2485 Brus- sels, Belgium. Association for Computational Lin- guistics.
Proceedings of
Ido Dagan, Oren Glickman, and Bernrrdo Magnini. 2005. The pascal recognising textual entailment chal- lenge. In Machine learning challenges workshop. pages 177-190. Springer.
Jacob Devlin, Ming-War Wei Chang. Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language under- standing. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguisties: Human Language Tech- nologies, Volume (Long and Short Papers), pages 4171-4186, Minneapolis, Minnesota. Association for Computational Linguistics.
Ethics Statement
This research was conducted in accordance with the ACM Code of Ethics. All the datasets that we use are publicly available. We report only aggregated results in the main paper. We do not share any Personally Identifiable Data in this paper.
Bill Dolan and Chris Brockett. 2005. Automati- cally constructing a corpus of sentential paraphrases. In Third International Workshop on Paraphrasing (IWP2005).
Acknowledgements
We extend our sincere gratitude to the anonymous reviewers for their invaluable contributions and con- structive feedback that have greatly enriched the quality and scope of this paper. This work was supported by Munich Center for Machine Learning (MCML) and China Scholarship Council (CSC).
Mareike Hartmann, Miryam de Lhoneux, Daniel Her- shcovich, Yova Kementchedjhhieaa, Lukas Nielsen, Chen Qiu, and Anders Søgaard. 2021. A multilingual benchmark for probing regation-awarenee with min- imal pairs. In Proceedings of the 25th Conference on Computational Natural Language Learning, pages 244-257, Online. Association for Computational Lin- guistics.
References
Ari Holtzman, Peter West, Vered Shwartz, Yejin Choi, and Luke Zettlemoyer. 2021. Surface form com- petition: Why the highest probability answer isn't always right. In Proceedings of the 2021 Conference Empirical Methods in Natural Language Process- ing, pages 7038-700 7051, Online and Punta Cana, Do- minican Republic. Association for Computational Linguisties.
Kabir Ahuja, Sunayana Sitaram, Sandipan Dandapat, and Monojit Choudhury. 2022. On the calibration of massively multilingual language models. In Proceed- ings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 4310-4323 Abu Dhabi, United Arab Emirates. Association for Computational Linguistics.
Junjie Hu, Sebastian Ruder, Aditya Siddhant, Gra- ham Neubig, Orhan Firat, and Melvin Johnson. 2020. Xtreme: A massively multilingual multi-task benchmark for evaluating cross-lingual generalisa- tion. International Conference on Machine Learn- ing. pages 4421. PMLR.
Tom Brown Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. 2020. Language models are few-shot learners. Advances in neural information processing systems, 33:1877-1901.
Lianzhe Huang, Shuming Ma, Dongdong Zhang, Furu Wei, and Houfeng Wang. 2022. Zero-shot cross- lingual transfer of prompt-based tuning with a unified
Alexis Conneau, Kartikay Khandelwal, Naman Goyal, Vishrav Chaudhary, Guillaume Wenzek, Francisco

---

multilingual prompt. In Proceedings of the 2022 Con- ference on Empirical Methods in Natural Language Processing, pages 1488-1 1888-11497, 1497, Abu Dhabi, United Arab Emirates. Association for Computational Lin-
John Platt al. 1999. Probabilistic outputs for support vector machines and comparisons to regularized like- lihood methods. Advances in large margin classifiers, 10(3):61-74.
guistics.
Naomi Shapiro, Amandalynne Paullada, and Shane Secinert-Threlkee 2021. A multilabel approach to morphosyntactic probing. In Findings of the Associ- ation for Computational Linguistics: EMNLP 2021, pages 4486-4524 Punta Cana, Dominican Republic. Association for Computational Linguisties.
Phillip Keung, Yichao Lu, György Szarvas, and Noah A. Smith. 2020. The multilingual Amazon reviews cor- pus. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 4563-4568, Online, Association for Computational Linguistics.
Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R. Bowman, 2019. GLUE: A multi-task benchmark and analysis plat- form for natural language understanding. In Pro- ceedings of the Seventh International Conference on Learning Representations.
Hector Levesque, Ernest Davis, and Leora Morgenstern. 2012. The winograd schema challenge. In Thir- leenth international conference on the principles of knowledge representation and reasoning.
Pengfei Liu, Weizhe Yuan, Jinlan Fu, Zhengbao Jiang.
Hiroaki Hayashi, and Graham Neubig. 2021. Pre-
Alex Warstadt, Amanpreet Singh, and Samuel R. Bow- man. 2019. Neural network acceptability judgments. Transactions of the Association for Computational Linguistics, 7:625-641.
train. prompt, and predict: A systematic survey of prompting methods in natural language processing. arXiv preprint arXiv:2107. 13586.
Yinhan Liu, Myle Out, Naman Goyal, Jingfei Du, Man- dar Joshi, Danqi Chen, Omer Levy, Mike Lewis. Luke Zettlemoyer, and Veselin Stoyanov. 2019. Roberta: A robustly optimized bert pretraining ap- proach. arXiv preprint arXiv: 1907.1 1669..
Leonie Weissweiler. Valentin Hofmann, Abdullatif Kök- sal, and Hinrich Schütze. 2022. The better your syn- tax, the better your semantics? probing pretrained language models for the English comparative cor- relative. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 10859-10882 Abu Dhabi, United Arab Emi- rates. Association for Computational Linguistics.
Yongkang Liu, Shi Feng, Daling Wang, and Yifei Zhang. 2022. MulZDG: Multilingual code-switching frame- work for zero-shot dialogue generation. In Proceed- ings of the 29th International Conference on Com- putational Linguistics, pages 1199659,660 Gyeongju, Republic of Korea. International Committee on Com- putational Linguistics.
Thomas Wolf, Lysandre Debut, Victor Sanh. Julien Chaumond, Clement Delangue, Anthony Moi, Pier- ric Cistac, Tim Rault, Rémi Louf, Morgan Funtowicz, et al. 2020. Transformers: language processing. In Proceedings of the 2020 con- ference on empirical methods in natural language processing: system demonstrations, pages 38-45.
ffate-of-the-ar natural
Yao Lu, Max Bartolo, Alastair Moore, Sebastian Riedel, and Pontus Stenetorp. 2022. Fantastically ordered prompts and where to find them: Overcoming few- shot prompt order sensitivity. In Proceedings of the 60th Annual Meeting of the Association for Compu- tational Linguistics (Volume 1: Long Papers), pages 8086-8098, Dublin, Ireland. Association for Compu- tational I. Linguistics.
Sohee Yang. Jonghyeon Kim, Joel Jang, Seonghyeon Ye. Hyunji Lee, and Minjoon Seo. 2023. Im- proving probability- prompt selection through unified evaluation and analysis. arXiv preprint arXiv:2305.143 14877.
Julian McAuley and Jure Leskovec. 2013. Hidden fac- tors and hidden topics: understanding rating dimen- sions with review text. In Proceedings of the 7th ACM conference on Recommender systems, pages 165-172.
Yinfei Yang. Yuan Zhang, Chris Tar, and Jason Baldridge. 2019. PAWSSX::A A cross- loos-lliguual adversar- ial dataset for paraphrase identification. In Proceed- ings of the 2019 Conference on Empirical Methods in Natural Languuge Processing and the 9th Inter- national Joint Conference on Natural Language Pro- cessing (EMNLP-IJCNV pages 3687-3692, Hong Kong, China. Association for Computational Linguis- tics.
Ercong Nic, Sheng Liang, Helmut Schmid, and Hinrich Schütze. 2023. Cross-lingual retrieval augmented prompt for low-resource languages. In Findings of the Association for Computational Linguistics: ACI. 2022. Toronto, Canada. Association for Computa- tional Linguistics.
Wenpeng Yin, Jamaal Hay, and Dan Roth. 2019. Bench- marking zero-shot text classification: Datasets, eval- uation and entailment approach. In Proceedings of the 2019 Conference on Empirical Methods Natu- ral Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 3914-3923, Hong Kong. China. Association for Computational Linguistics.
Adam Paszke. Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, et al. 2019. Pytorch: An imperative style, perforreffee deep learning library. Advances in neural information processing systems, 32.

---

Xiang Zhang, Junbo Zhao, and Yann LeCun. 2015. Character-level convolutional networks for text classi- fication. Advances in neural information processing systems, 28.
MNLI dataset, then they reformulate common clas- sification tasks to an NLI task format. The input sample is regarded as the premise, while the label serves as the hypothesis. The zero-shot classifica- tion is performed by directly comparing the proba- bilities of predicting entai for all imput-label pairs. For this baseline, we finetune BERT model and a RoBERTa model on the MNLI task.
Mengjie Zhao and Hinrich Schütze. 2021. Discrete and soft prompting for multilingual models. In Proceed- ings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 8547-8555, Online and Punta Cana, Dominican Republic. Asso- ciation for Computational Linguistics.
B Few-Shot Training of Calibration Parameters
Zihao Zhao, Eric Wallace, Shi Feng, Dan Klein, and Sameer Singh. 2021. Calibrate before use: Improv- ing few-shot performance of language models. In Proceedings of the 38th International Conference on Machine Learning, volume 139 of Proceedings of Machine Learning Research, pages 12697-12706. PMLR.
Algorithm I presents the process of few-shot train- ing of penalty calibration used in our few-shot in- vestigation.
Algorithm Few-Shot Training of Penalty
A Experimental Details
Calibration
Input: set of few-shot training samples D, initial calibration parameter vector Po: number of epochs E, learning rate
This section provides a comprehensive overview of our experimental setup, including hyperparameters, prompt templates that we use in our experiments, and the baselines.
Output: Trained parameters
A.1 Hyperparameters
Initialize p <- Po:
for epoch in 1, 2, ... E do
To ensure experimental reproducibility, we present the hyperparameter settings used in our study in Table 5.
foreach (x, y) in D do
+ get probs(x);
1 + 1 - p calibration;
Hyperparameter
Value
ÿ + argmax, (1(u))
Evaliaation hatch size
if y * ÿ then
Learning
Randem seeds
(42, 512, 1213. 1234)
P p p(u] + m7;
sequence longth
128
Fow-shot numbers
4, 16:
P ply] plul r7;
GPU type
NVIDIA Gefonce GTX
end
Number of GPU
end end
Table 5: Overview of hyperparameters.
A.2 Prompt Engineering
C Detailed Results
We select a set of prompt templates for the tasks through our preliminary experiments. Table shows the prompt templates and the label words used in our experiment.
Detailed results of the experiments in the main text can be found in this section. Table 8 shows the complete results of mBERT on the multilingual AG News dataset across all 25 languages. Table 7 provides an overview of languages covered by the multilingual AG News dataset.
A.3 Baseline
To establish a baseline, we initially conduct experi- ments without employing any calibration methods. Subsequently, introduce four calibration meth- ods individually and evaluate their impact on the performance. Besides, we compare our calibration methods with an NI.I-ased zero-shot classification bascline proposed by Yin et al. (2019), where they first finetune a pretrained language model on the

---

Task
Prompt template
Label words
News: [X]
World. 'Sports" Bussness
Amazon-P
[X] All in all, it was mask.
"bud.. "good"
Amazon-P
[x].
mask.
"trrible" 'aad' 'ok'' 'good', 'great"
XNLI
[X]? nack, [Y]
"Yes, 'Mayhe", 'No'
Yahoo
mask Question: DX [V]
"Society" "Science. 'Heath"' 'Education", ....
PAWS-X
[X] mask[ YJ
"Wrong" 'Right"
Cal
[X] It linguistially nask.
"wrong: 'right"
MRPC
[X]?
"Wrong", 'Right"
QQF
Question 1: [X] Question 2: [Y] Question I and Question are nask
ddffteent" "samm"
RTF.
[x]? nask,
'Wrongg. 'Right"
WNLI
[X]? nask, [Y]
WWrong. RRight
Table 6: Overview of prompt templates.
Code
Languages
Language Accessibility
Language Family
af
Afrikaans
Low-resource
Indo-European
co
Corsican
Unseen languages
Indo-Europenn
co
Esperanto
Unseen languages
Artificial
haw
Hawaïan
Unseen languages
Austroncsian
hmn
Hmong
Unseen languages
Sino-Tibetan
ht
Haitian Creole
Low-resource
ano--Euoopenn
ig
Igho
Unseen languages
Niger-Congo
jw
Javanese
Low-resource
Austronesian
km
Khmer
Unseen script
Austronesian
mi
Maori
Low-resource
Austronesian
mn
Mongolian
Low-resource
mongolian
mt
Maltese
Unseen languages
Afro-Asiatic
my
Burmese
Low-resource
Sino- Tibetab
ny
Chichewa
Unscen languages
Niger-Congo
or
Odia
Unseen script
Indo-Europenn
sm
Samoan
Unseen languages
Austronesian
Shona
Unseen languages
Dravadian
Sesotho
Unseen languages
Dravadian
sw
Swahili
Low-resource
Dravadian
ta
Tagalog
Low-resource
Austroncsian
tc
Telugu
Low-resource
Dravadian
tl
Tamil
Low-resource
Dravadian
ug
Uighur
Unseen languages
Turkic
ur
Urdu
Low-resource
Indo-European
Uzbek
Low-resource
Turkic
Zulu
Unseen languages
Niger-Congo
Table 7: Overview of languages covered by the multilingual AG News dataset.
af
haw
hmn
ig
jw
km
my
No calib.
40.4
32.6
47.3
31.9
27.1
30.9
35.7
30.2
38.0
33.3
29.0
32.0
29.9
33.8
Penalty
64.3
44.2
69.6
72.3
40.1
49.6
55.2
48.8
62.6
51.2
46.3
62.2
57.6
64.7
CBM
64.7
58.3
69.1
62.4
42.0
50.8
49.6
63.9
47.8
49.5
53.0
57.2
54.1
CC
65.6
59.7
67.8
68.0
43.4
49.7
65.2
52.4
66.4
41.4
51.2
55.4
57.4
51.7
PMIpe
60.2
35.3
60.0
61.7
35.9
33.5
33.5
49.2
61.5
42.2
49.6
61.1
47.6
ny
sm
Sw
te
ug
2u
avg.
No calib.
29.8
25.4
30.3
32.2
30.4
33.4
28.8
32.5
42.6
25.5
33.2
33.9
34.5
32.8
Penalty
51.4
45.2
43.5
52.4
44.8
72.9
65.6
59.9
61.7
27.0
52.6
59.1
50.3
$4.6
CBM
52.4
46.1
53.4
48.8
59.9
60.0
29.5
56.8
53.7
53.8
CC
51.2
28.7
47.5
52.5
64.1
56.5
52.4
62.6
27.9
53.1
49.6
$3.7
PMIpe
50.2
28.6
43.9
50.9
44.6
61.6
43.6
66.1
29.3
55.0
56.4
51.3
Table 8: Results of mBERT on the multilingual AG News dataset across all languages.


Reviews:
[{"review_id": "4xMr00T0te", "reviewer": "Reviewer_3ZFj", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 5, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely appreciate your thoughtful review and valuable feedback on our paper. Your acknowledgement of our work serves as a source of encouragement to us. Subsequently, we would like to address a few points that were raised in your review, particularly regarding some aspects that might have led to certain misconceptions.\n\n**Response to your primary concern**    \nYou expressed your concern that “the results, analysis and conclusions of the paper do not line up with the claimed contributions” and advise us to “slightly reframe the paper to be more exploratory/case-study like”. \n\nYour insightful concern helped us realise that there might exist some unclarity and ambiguity in our writing. This could potentially lead to deviations from our intended meaning and create misunderstandings among our readers. We would like to take this opportunity to explain our thinking. In the conclusion part, by writing “Notably, with a minimal number of training examples, the calibrated probabilities yield significant enhancements”, we are trying to highlight the effectiveness of few-shot training of calibration, for both calibration methods, i.e. our proposed $Penalty$ and the existing $CC$. As Table 7 shows, most of the few-shot cases outperforms the zero-shot case across all experimented tasks, in many cases even with a substantial margin. \n\nWe did not intend to amplify the effect of our proposed $Penalty$ calibration in our work. Our contributions (as outlined in Sec. 1) can be categorised into two aspects: \n1. proving the effectiveness of the $Penalty$ calibration method whose performance is comparable to the existing methods\n2. proving the general effectiveness across different calibration methods in multilingual encoders and in few-shot settings: few-shot calibration performs better than zero-shot in principle; calibration methods take effect on multilingual encoders as well. \n\nWe appreciate the valuable concern you pointed out. To mitigate any potential misunderstandings, we would reorganise our language, ensuring that our contributions are conveyed with greater clarity and precision. Besides, as you suggested, reframing the paper to add more exploratory case analysis would contribute to enhancing its clarity. While we initially considered conducting a more in-depth case analysis, we regrettably had to forgo this due to the constraints in scope. However, if our paper is accepted, we would like to make use of the additional page for the camera-ready version to incoporate more case studies, aligning with your advice. \n\n**Response to Missing references**  \nThank you for pointing this out. We neglected the paper you referred to regarding the relevant work on autoregressive models. We will add this reference in our next version.\n\n**Response to Typos**  \nThanks for your careful reading! We will promptly fix these typos.\n\nThank you once again for your thorough review, and we hope our explanations could help address your concerns.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "I would like to take this opportunity to congratulate the authors on a nice piece of work.\n- The work tackles an important, under-explored problem in the are of calibrating MLMs which has implications on model reuse and the use of task specific fintetuning.\n- The method proposed is simple and intuitive and does better than no calibration in most cases and sometimes outperforms SOTA calibration methods. (e.g. Amazon-P for the unilingual case and AGNews and XNLI for the multilingual case)\n- The experiments are thorough and surpass my expectations for a short paper.", "reasons_to_reject": "I believe this paper is worth accepting but only if the primary concern discussed below is addressed.\n- Primary concern: The results, analysis and conclusions of the paper do not line up with the claimed contributions. In particular, paragraph 73-82 claims that \"its [the calibration method] effectiveness in achieving performance enhancements comparable to other existing calibration techniques\" which is amplified in the conclusion where a claim is made that \"the calibrated probabilities yield significant enhancements\". I don't believe this to be entirely true as, in several cases, the \"Penalty\" method performs much worse than current methods. This point is made not to devalue the author's contributions nor to use a \"Not SOTA so Reject\" argument, but rather because several, if not the majority, of results seem to dispute the authors' claim. I would consider slightly reframing the paper to be more exploratory/case-study-like. ", "questions_to_authors": "", "paper_topic_and_main_contributions": "This paper considers the setting in which a the input to a task is reformulated as a masked language modeling task e.g., in the sentiment classification of a movie review converting the input and label (\"This was a great movie\", Positive) to a masked language modeling task \"This was a great movie. The movie was [MASK].\" The authors discuss that, although this task conversion is flexible, it often suffers from poorly calibrated logits being used to decide the replacement of the [MASK]. The authors then offer a calibration method which they test on both unilingua multilingual language modeling encoders and which essentially \"shifts\" the thresholding based on the uncalibrated logit score. The authors show that this simple calibration method allows unilingual and multilingual MLMs to perform better than uncalibrated ones.", "excitement": 4, "reproducibility": 5, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Typos_Grammar_Style_and_Presentation_Improvements": "- Line 001: Pretraiend\n- Line 181: \"after of\""}, "extra_text": {"Missing_References": "Although you discuss them in your limitations section, I believe calibration work in autoregressive LMs should be at least discussed briefly. One reference which I was surprised was not included is the following:\n1. Min, Sewon, et al. \"Noisy channel language model prompting for few-shot text classification.\" arXiv preprint arXiv:2108.04106 (2021)."}}, {"review_id": "zsnPHLzALe", "reviewer": "Reviewer_LfMb", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for your time and effort you have invested in providing feedback on our work. In order to avoid misleading, we would like to take the opportunity to emphasise the contributions of our work again:\n\nOur contributions can be summarised from two aspects: \n1. We prove the efficacy of our proposed $Penalty$ calibration method:   \na. We proposed a simple but effective calibration method   \nb. Our method can achieve comparable performance to the existing calibration techniques. \n2. The next aspect of our work’s contributions treats different calibration methods as a whole:   \nc. few-shot calibration performs better than zero-shot in principle.   \nd. calibration methods take effect on multilingual encoders as well. \n\nRegarding the \"use of calibration techniques\" which you mentioned in the \"Reasons to reject\", as mentioned in the Limitation section, as a short paper, we focus on validating the effectiveness of the calibration techniques for multilingual encoders and in few-shot scenarios. We will continue to explore the calibration methods for autoregressive models and multilingual LLMs in our future work.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "(1) This paper shows the effectiveness of calibration techniques in the zero-shot scenario; (2) The experiments are sufficient and the results are comprehensive and convincing.", "reasons_to_reject": "The novelty is limited. The use of calibration techniques is straight.", "questions_to_authors": "", "paper_topic_and_main_contributions": "This paper proposes to combine the models with various calibration techniques which modify the probabilities of label words predicted by the models. The experimental results on several benchmarks verify the effectiveness of the proposed method.", "excitement": 3, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "6NqG17sccN", "reviewer": "Reviewer_Pst8", "summary": "", "questions": "A) In figure 2, should we also compare as a baseline the other zero-shot calibration methods? For example, CBM  zero-shot in AgNews does 78.4 which is better than the baseline and also CC and \"penalty\" few-shot. \n\nB) In the figure 1 case what is the class distribution probability?", "limitations": "", "rating": 3, "confidence": 4, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We extend our gratitude for your careful evaluation of our paper. Your acknowledgement of contributions we have presented is sincerely appreciated. We are glad to address your insightful comments and suggestions, particularly with regard to certain aspects where there might be some misunderstanding. Before elaborating on concrete points, I would like to first reiterate the contributions of our work:  \nOur contributions encompass two distinct aspects.  \n1. Our work establishes the effectiveness of the proposed $Penalty$ calibration method:  \na. We introduce a straightforward yet potent calibration method.  \nb. Our method demonstrates performance on par with existing calibration techniques.  \n2. Our contributions extend to a comprehensive examination of various calibration methods:  \n     c.  Few-shot calibration basically outperforms zero-shot calibration.  \n     d. Calibration methods work on multilingual encoders as well.  \n\n**Response to Reasons to Reject**  \nAs pointed out in your **first** piece of reason, our proposed calibration method failed to outperform other calibration methods in many cases. However, as claimed in the Introduction section (line 073-078), the major contribution with regard to the proposed $penalty$ method is that we introduce a simple yet effective calibration method that can achieve **comparable** results to other existing techniques. We did not intend to invent a SOTA calibration method, but to offer more effective options that can be applied to few-shot and multilingual scenarios (the second contribution as mentioned above).\n\nAs response to the **second** point stating that the setup of this study in both monolingual and multilingual aspects is not in line with the title and abstract framed on multilingual encoders, we would like to sort out our overall thoughts of this paper: The motivation of this paper is to boost the zero-shot learning performance of multilingual encoder models through probability calibration techniques, including both existing calibration techniques and one proposed calibration approach. For better using calibration techniques, we also proposed a few-shot calibration method: training calibration parameters based on very few training samples. Before implementing these techniques into multilingual encoders, we initially assessed their efficacy through experiments in monolingual settings. After that, we extended their application to multilingual encoders, effectively showcasing their utility in multilingual encoders.\n\nWe would like to modify our writing to present our contributions and thinking in a more clear, more organised and more easily understandable way.\n\n**Response to Questions**  \n**Q(a)**: As explained in line 171-178, the reason why we used CC and Penalty calibration methods for few-shot training but excluded $PMI_{DC}$ and $CBM$ is that $PMI_{DC}$ and $CBM$ directly calculate the calibrated probabilities using probabilistic formulas without using additional trainale parameters. Different from $PMI_{DC}$ and $CBM$, $CC$ and $Penalty$ contain trainable parameters. As Section 2 shows, the parameters for $CC$ and $Penalty$ can be initialised in the zero-shot scenario. Only calibration methods with trainable parameters fit the few-shot training methods.\nIt was regrettable that due to the 4-page range limitation, we did not give a very explicit explanation to this point in the paper. If we have the chance to have our paper accepted, we would like to use the additional one page for the camera-ready version to add more detailed elaborations in the main text.  \n\n**Q(b)**: Thanks for your question on the class distribution probability in Figure 1. It is 0.5:0.5. We presented our example on a balanced test set. It is our negligence not to mark it. We will add this information in the future version. Thanks for the reminder!\n\n**Response to Typos**:\nThanks for your thorough review! We will promptly rectify these typo errors. \nThanks for your advice on the colouring match of Figure 2. We will recolor it to make the lines more differentiated! \n\nThank you once more for your comprehensive review. We sincerely wish that we delve into your concerns in a detailed manner.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": " - The paper introduces a new calibration method that often improves against the base model without any modifications.\n\n- The experimental setup is solid with multiple datasets and other methods for comparison.\n\n - The paper is clear and well written,", "reasons_to_reject": "- While the proposed approach shows improvement over the bare model, the results are not good when compared with other methods from the literature which are consistently outperforming it.\n\n- The paper is framed (e.g. both title and abstract) on multilingual encoders. However in the setup both the monolingual and multilingual aspects are equally studied.", "questions_to_authors": "A) In figure 2, should we also compare as a baseline the other zero-shot calibration methods? For example, CBM  zero-shot in AgNews does 78.4 which is better than the baseline and also CC and \"penalty\" few-shot. \n\nB) In the figure 1 case what is the class distribution probability?", "paper_topic_and_main_contributions": "The paper focuses on enhancing zero-shot performance of pretrained encoders studying classification tasks. The method involves modifying the word probabilities of the model with some calibration method. The main novelty is the introduction of a new calibration method \"penalty\", which adds a penalty to each word. The approach is then tested on multiple datasets for both the monolingual and multilingual case.", "excitement": 3, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Typos_Grammar_Style_and_Presentation_Improvements": "- The Colors of figure 2 can be differentiated a bit more, in particular AgNews\n\n- The first word of the abstract is misspelled"}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper propose a calibration method to mitigate the bias towards high confidence predictions. The experimental results of this paper show empirical evidence of substantial performance gains, even with just a few training samples. Overall this incremental, simple, yet effective method is a perfect fit for a shot paper. The main concern that the authors should address are the ones expressed by rev. 3ZFj on mismatch of the empirical results with some claims. It would also help to include the monolingual contributions into the title, as the multilingual focused title is misleading (rev. Pst8).
Decision: Accept-Findings