On Calibration of Multilingual Question Answering LLMs
Authors: Yahan Yang
Soham Dan
Dan Roth
Insup Lee
Venue: Accepted by TMLR ()

Abstract: Multilingual pre-trained Large Language Models (LLMs) are incredibly effective at Question Answering (QA), a core task in Natural Language Understanding, achieving high accuracies on several multilingual benchmarks. However, little is known about how well their confidences are calibrated. In this paper, we comprehensively benchmark the calibration of several multilingual LLMs (MLLMs) on a variety of QA tasks. We perform extensive experiments, spanning encoder-only, encoder-decoder, and decoder-only QA models (size varying from 110M to 7B parameters) and diverse languages, including both high- and low-resource ones. We study different dimensions of calibration in in-distribution, out-of-distribution, and cross-lingual transfer settings, and investigate strategies to improve it, including post-hoc methods and regularized fine-tuning. For decoder-only LLMs such as LlaMa2, we additionally find that in-context learning improves confidence calibration on multilingual data.
We also conduct several ablation experiments to study the effect of language distances, language corpus size, and model size on calibration, and how multilingual models compare with their monolingual counterparts for diverse tasks and languages. Our experiments suggest that the multilingual QA models are poorly calibrated for languages other than English and incorporating a small set of cheaply translated multilingual samples during fine-tuning/calibration effectively enhances the calibration performance.

Published in Transactions on Machine Learning Research (12/2025)
On Calibration of Multilingual Question Answering LLMs
Yahan Yang' University of Pennsylania
yangy996@scassuueeeatt www.sssee...seee..cc.c...ee.
Soham Dan Microsoft
sokamdann@iccrrooff..
Dan Roth University of Pennsyluania
dany Bseas, wpprenneemmmmm edu
Insup Lee University of Pennsyloania
lee aseassss upenn. edu
Reviewed on OpenReview: https: //::://ooppp openreview. net/ forum? id= 4d4ggh22P
Abstract
Multilingual pre-trained Large Language Models (LLMs) are incredibly effective Question Answering (QA), task in Natural Language Understanding, achieving high accuracies on several multilingual benchmarks. However, little is known about how well their con- fidences are calibrated. In this paper, we comprehensively benchmark the calibration of several multilingual LLMs (MLLMs) on a variety of QA tasks. We perform extensive exper- iments, spanning encoder-only, encoder-decode from 110M to 7B parameters) and diverse languages, including both high- and low-resource ones. We study different dimensions of calibration in-distribution, out-of-distribution, and cross-lingual transfer settings, and investigate strategies to improve it, including post-hoc methods and regularized fine-tuning. For decoder-only LLMs such as LlaMa2, we addition- ally find that in-context lcarning improves confidence calibration on multilingual data. We also conduct several ablation experiments to study the effect of language distances, language corpus size, and model size on calibration, and how multilingual models compare with their monolingual counterparts for diverse tasks and languages. Our experiments suggest that the multilingual QA models are poorly calibrated for languages other than English and incorpo- rating small set of cheaply translated multilingual samples during fine-tuning/ent caauuinnnaaalbbattionn effectively enhances the calibration performance.
and decoder-only QA models (size varying
1 Introduction
Pre-trained Large Language Models (LLMs) like BERT, RoBERTA, T5, BART (Devlin et al. 2019 Liu
et al. 2019 Wolf et al. 2020 Raffel et al. 2020 Lewis et al. 2019a), and their multilingual counterparts like mT5, BBART, XLM (Nue et al. 2021 Liu et al. 2020 Conneau et al. 2019), have greatly advancod Natural Language Understanding. The multilingual LLMs (MLLMs) are able to transfer knowledge acquired from English other languages, leading to impressive cross- lingual performances on various downstream tasks the zero-shot and few-shot settings, i.e., for languages not seen during fine-tuning (Wu & Dredze 2020 Xue
et al. 2021). Question Answering (QA) is common task for understanding how well machines understand human language, and more importantly, it comes in variely of formats such multiple choice, span selection, or free-form answer generation. Several NLP/ ILP/multimodal tasks can be cast as QA, highlighting the universality of the QA format. Multilingual QA as task becoming incrcasingly important with the globally widespread deployment and democralization of AT systems Lossnovv el al. 2021
The first two authors contributed equally this paper.
Work done prior joining Microsoft.

---

Published in Transactions on Machine Learning Research (12/2025)
18.142
12.285
11.255
7.896
2.0
1.0
1.0
1.0
0.8
0.8
0.8
0.8
0.6
0.6
accuracy
0.6
0.4
0.4
0.2
0.0
1.0
0.0
0.4
0.0
0.0
0.4 confidence
confidence
confidence
(a) XLM
(b) With TS
(c) With more English
(d) With Bengali
Figure 1: In this plot show that pre-trained multilingual models, fine-tuned on English QA, are not well calibrated in languages other than English, specifically low-rcsource ones like Bengali. (a) shows the reliability diagram for XLM Conneau et al. (2019) fine-tuned on the English TyDiQA training set, evaluated Bengali TyDiQA test set. The large deviation from the diagonal Y=X line indicates it is not well calibrated. (b), (c) and (d) show that Temperature Scaling (TS), fine-tuning with more English data, and fine-tuning on Bengali TyDiQA training data respectively, all improve calibration compared to (a), as indicated by the better alignment with the diagonal and the lower ECE score. This indicates that despite high zero-shot cross-lingual accuracy, zero-shot cross-lingual calibration not good for LLMs, unless dedicated calibration strategies, like TS, are used to improve them.
the
Despite the amazing progress of LLMs across several benchmarks, they unfortunately suffer from sometimes generating incorrect answers, with very high confidence. This can have severe consequences for safety-critical applications such as healthcare, autonomous driving, or finances where mistakes can be very costly. With the increasing application of LLMs to such lasks, is crucial to understand whether the predictions are reliable, and when the models are unsure of the answer. Confidence calibration one such reliability metric that measures whether the model's prediction probability estimates of the answer being correct. Confidence calibration has been studied in Computer Vision Guo et al. (0017)
aligned with the actual probability
Minderer et al. (2021) and Natural Language Processing Desai & Durrett (202)); Dan & Roth 2022
However, most of the prior works are limited to classification settings, which is inapplicable to the generality of the QA task. Recently, Zhang et al. (2021); Jiang et al. 2021 have shown that state-of -ae---o-tthhheee the-art English QA
models are surprisingly poorly calibrated. However, there remains gap understanding of the calibration properties of multilingual QA models. this paper, we address this gap by comprchensive study on the Calibration of Mullilingual Question Answering Large Language Models. The main research questions we investigate in this paper are:
1) How well are MLLMs calibrated in the cross-lingual transfer scenario?
2) How can we improve MLLMs' confidence calibration on multilingual QA datasels?
The contributions of our work are as follows
We provide the first comprehensive benchmarking of confidence calibration of multilingual QA models (architectures including extractive models: mBERT, XLM-R and generatiue models: mT5, mBART, and LLaMa2) over both low- and high-resource languages, in-distribution and out-of-distribution settings.
We observe that the calibration performance on English is not transferable to other languages, across various datasets and architectures. Distance between the target languages and English, and the distribution of different languages at the pre-training stage, are all highly correlated with calibration performance, across the various model lypes.
An investigation of various calibration strategies including post-hoc methods and regularization methods, aimed at enhancing cross- pross-lingual calibration. Temperature scaling (optimized a cross-lingual validation dataset) shows the most significant improvement even if the target language is absent in the validation data.
We consider the In- mn-Context Learning (ICL) scenario for LLMs such as LlaMa-2 and show that ICL improves
both accuracy and calibration on multilingual QA tasks.
We perform a suite of ablation experiments to study the role of example diversity, language diversity,
'Our code available http:::/ggttb...cmm///n///////// http.///waggg/////mtiiiaaaa.aaa
2

---

Published in Transactions on Machine Learning Research (12/2025)
and model size on calibration, and to compare the calibration of MLLMs with that of their monolingual counterparts.
2
Related Work
2.1 Calibration of Large Language Models
Prior studies (Desai& & Durrett 2020
Dan & Roth 2021 Xu & Zhang 2023 He al. 2021 have investigated
the calibration of pre-trained LLMs on downstream tasks, primarily with a focus monolingual English models and tasks, and primarily in the classification setting. Kuleshov & Liang (2015): Jagannatha &
Yu 20200] study calibration for structured prediction, but not in the context of LLMs, Recently, Ahuja 2022 studics calibration of multilingual pre-trained LLMs, specifically mBERT and XLM (Devlin
ct al.
et al. 2019 Conneau et al.] 2019) on various downstream classification tasks including natural language
inference and commonsense reasoning. They show that multilingual models are not well-calibrated in the
classification setting, especially for low-resource languages. [iang al. (2022) also explores cross-lingual calibration performance for mBERT and XLM, comparing various post-hoc calibration methods both structured and unstructured prediction tasks. In this work, we extend this line of work to calibration of MLLMs for QA, in both classification and generative settings, and to cross-lingual and distribution shift settings.
2.2 Calibration of models on Question Answering tasks
Recently, there has been growing interest in studying calibration of English QA models (Kamath et al. 2020;
Zhang et al.) 2021 Jiang ct al.) 2021 Si et al. 2022). Kamath et al. (2020) trains an extra calibrator of confidence scores to improve the calibration performance and examines the calibration performance on an out-of-domain (OOD) setting. They utilize the scores from the calibrator and uses it as a reranker to select the answers. Zhang et al. (2021) extends this work by adding the features the context and back-translated context. Jiang et al. (2021) analyzes the calibration performance of generative language models, and find that the generative models on QA are not well-calibrated. Our work in contrast. investigates the calibration of pre-trained multitingual LLMs (both extractive, with encoder-only architecture, and generative, with encoder-decoder decoder-only architecture) on QA, and various techniques to improve calibration such as temperature scaling Guo et al. (2017), label smoothing Szegedy et al. (0166) and cross-lingus data augmentation.
3 Background
3.1 Question Answering
Context: wahrend die Cscar-Gowinnert Marloe
Context: l....whhrend die Oscar-Gewinnerin Marlee Matlin die Amerikanische Gebardensprache (ASL)
Matlin Amenkanische cebtrder.sprachee (ASL)
Ubersetzte.) }
Obersetzte.)
Question: (Weiche Schauspielerin übersetzte Spiel in die Amerikanische Gebardensprachaz)
Question: (Welche Schauspiclerin übersetzte beim Spiel die Amerikanishee Geoärdensprache ?)
Encoder-only Model [e.g., MBERT,
Encoder-Decoder, Decoder-only Model 2LaMa2)
Answer start index: Answer and Index: 15
Answer: Marice Matlin
Answer Extraction
Answer Generation
Figure 2: Differences in the output format between extractive and generative QA models.
In this work, we focus on two broad types of models for QA: extractive (or discriminative) and generative (as shown in Figure 2). For models based encoder-only language models like mBERT and XTM-R (Conneau)
3

---

Published in Transactions on Machine Learning Research (12/2025)
et al. 2019 Devlin et al. 2019 Liu et al. 2019). the prediction of the answer span within the given context is framed as classification task and achieved using two linear layers. These linear layers are placed on top of the hidden representations and are responsible for predicting the start and end indices of the answer span Y, respoctively. The logit score of the answer Zans is defined as the sum of the logits of the start and end positions, Estart and Zend [Si et al. p2022). Unlike slandard classification problems which contain a fixed number of classes and compute the logits score from the model over those classes, here we sample K-sized candidate set I (Y) k of answers for question (Jiang et al., 2021], by selecting answers with top K logits scores Zans- The confidence of each candidate answer is the score after softmax is applied to its logit Zans
In the case of generative models like mT5 and mBART Xue et al. [2021] Liu et al. (2020); Touvron et al. 2023) we frame question-answeringg task as
toteeeeee----eueeneee generation problem Khashabi et al. tokens, the probability of the answer is defined as P"(Y|X)= = -I P(ul|,,e<d product of the individual token probabilities P(ys|X, vye)) that are generated by the decoder. Beam Search is used to find the candidate Pra((VIx)
2020 If y/s denotes the tt generated token and Yes denotes all the previously generated
answers and the normalized probability is defined as P(PIX) = "xver P'YY'X)> and used as the confidence of the prediction. The final answer of the model is the with the highest normalized probability, all answers in the candidate set.
3.2 Metrics to measure the Confidence Calibration of a model
A model is considered well-calibrated if the confidence estimates of its prediction are well-aligned with the actual probability of the answer being correct. Given an input X for which the gold output is Y and the model output is Y, with confidence P. a perfectly calibrated model satisfies the following condition:
P( = Y P( X) p) p. Vp [0.1].
The above probability cannot be computed using finitely many samples since P a continuous random variable and, in practice, it is approximated by bucketing the prodictions.
Expected calibration error (ECE) A popular metric to measure confidence calibration is called the Expected Calibration Error (ECE) Guo et al. (2017). The predictions are bucketed into M disjoint equally sized interval bins based on the confidence values, and the weighted average of the difference between cach bucket's s accuracy and confidence calculated:
M
B
Σ m=1
acce conf(B..) I.
where Bm is the th bucket containing samples whose prediction confidence lics in (">-1 1 1. acc(Bm) and conf(Bm) is the average accuracy and prediction confidence of examples in the bucket Bm respectively. We set the number of bins as 10 in all our experiments.
3.3 Techniques to improve the Confidence Calibration of a model
The calibration propertics of model can be evaluated dircctly out-of-box basod on the probabilities it assigns the predicted answers. Further, one can adopt strategies to calibrate model, which can be broadly categorized into:
Post-hoc calibration methods that do not require any additional training, for example, Temperature Scaling
TS) Guo et al. (2017).
Specialized fine-tuning, such as Label Smoothing and Few Shot Learning, which regularizes training, or leverages augmented data respectively.
Temperature Scaling (TS) a technique to improve the calibration of a machine learning model Guo et al. 2017 When applying the softmax function to output the probability distribution of the logits, TS utilizes a single parameter т to scale the logits: softmar(zz) Σ. exp(2s/7)
ezp(ss/T) We perform TS in two ways
specialized to the QA model type:
4

---

Published in Transactions on Machine Learning Research (12/2025)
Extractive Models: compute the TS factors on the start logits and the end logits separately (Tstart and Fend), obtained by optimizing the Negative Log-Likelinn Log-Likelihood (NT..) loss on the validation set, extending the standard classification setting in Guo et al. (2017). The softmax score of the answer is computed as follows:
exp( Setent
Zend
sof LoftmaaaZaani
Σi
eap(
Eetart Tstart
Zend
Tend
Generative Models: For each of the K candidate answers, we have an associated normalized probability P(Y X). We use the log probabilities these as logits z = logP(ř X), in the softmax function Jiang et al. (2021). We then similarly optimize for a temperature T, with respect to the NLL loss, on the validation set.
Because minimizing cross-entropy during training encourages the model to make the correct-class logit arbitrarily large, it often produces overconfident predictions. Temperature scaling learns an optimal scalar T that rescales these logits so that the prodicted probabilities more closcly match empirical accuracy, thereby improving calibration.
Few-shot Learning (FS) Incorporating a small number of examples from the target language during fine-tuning, mixed in with the English examples improves the calibration performance in classification tasks (Ahuja et al. 2022). Pre-trained language models are predominantly trained English, which leads them to make overconfident predictions on non- Lon-English inputs, especially those from distant languages (Jiang et al. 2022). Incorporating a small set of cheaply translated multilingual examples during fine-tuning provides the model with direct exposure to these famiiiar languages, helping it better estimate its uncertainty and reducing the confidence bias when handling non-English text. We investigate whether this strategy helps calibrate multilingual QA models by randomly selecting 1000 samples from 5 different languages to fine-tune LLMs, in addition to the almost 100 times larger English data.
4 Benchmarking Calibration of Multilingual QA Models
In this section, we perform a comprehensive set of experiments spanning various LLMs and multi-lingual QA datasots. We focus on the cross-lingual zero-shot sctting where the models are fine-tuned only on the English QA training data and tested on other languages not seen during fine-tuning, and also show results under challenging distribution shift scenarios.
Pre-trained MLLMs: In our experiments, we investigate the calibration performance of five different models: mBERT Devlin et al. 2019). XLM-R (Conneau et al. 2019), mT5 (Xue et al. 2021), mBART Liu et al. 2020 L.aMa22 (Touvron et al. 2023 and Aya-expanse DDang et al. 2024h. mBERT and XLM-R are encoder-only extractive architectures, while mBART and mT5 are eecoder- generative architectures. LLaMa2 ana Aya are decoder-only LLMs with much greater ICL capabilities compared M and we use Llama-2-7b-ht and aya-expanse-8b. We fine-tune all these models the SQuAD 1.1 training data (Rajpurkar et al. 2016
Datasets
XQuAD (Artetxe et al. 2019) is a popular benchmark to evaluate the cross-lingual ability of multilingual models on QA. The datasct is dorived from the development set of SQuAD 1.1. It contains 1190 parallel pairs for 12 languages including English.
MLQA (MultiLingual Question Answering) has a similar extractive-answer format as XQuAD and contains
QA pairs in seven languages. (Lewis al. 2019b)
TyDiQA SyyDiQA-Gold?" (Clark et al. 2020) also shares the extractive-answer format and contains 9
topologically diverse languages such as Korean and Telugy" 2
2The size of the candidate set, K, is set 20 for all MLLMs and for LLaMa2, K is set 3 Additional details of the datascts and experiments are presented in Appendix A.1
We the secondary task (Gold-P) in TyDiQA, which aims at prodicting the specific span within context passage that serves as the answer the question. The TyDiQA data collected directly in each language without the use of translation (unlike MT.Q and XQuAD).
5

---

Published in Transactions on Machine Learning Research (12/2025)
Metrics
FM: Exact Match Rate. The prediction should be the same the gold answer.
ECE: Expected Calibration Error. In this paper, ECE denotes the Expected Calibration Error averaged over all the languages in the corresponding dataset.
FCE((en): (CE(enn denoles the Expected Calibration Error on the English-data only.
4.1 Results for In-Distribution Tasks
Figure and Table show the calibration performance of the five different MLLMs for the XQuAD dataset in the cross-lingual zero-shot setting. We nolice that the relative increase in answer error for languages other than English is smaller compared to the relative increase in ECE across all models. For example, as shown in Table D the average prediction error for all the non-English languages is 56% while for English answer error is 83%, i.c. a 69.7 % increase. On the other hand, the average ECE for non- non-English languages 18 % vs 7.32 % for English, an increase of 145%. LLaMa2 has a comparable calibration performance as the extractive models but the exact match rate is lower, especially for languages that less represented in LlaMa2's pre-training data, such Hindi and Greek
MBERTT
40
mTS MBART
LlaMa2-78
Aya-BB
ECE
10
ru
Figure 3: Calibration performance of different models on the XQuAD datasot. Note that ECE lower the better. mBART gets higher variance on TH and EL because has not the two languages at the pre-training stage.
Table 1: Average performance across six different models on the XQuAD dataset (12 languages). EM is higher the better, ECE lower the better.
EN
AR
DE
EL
ES
HI
EM
ECE
EM
ECE
EM
ECE
EM
ECE
EM
ECE
EM
ECE 18.66
70.24
7.84
42.37
18.78
55.47
13.0
39.38
19.27
55.20
13.13
43.63
RO
RU
TH
TR
V1
ZH
EM
ECE
EM
ECE
EM 40.52
BCE
EM 44.00
ECE 17.70
EM 48.82
ECE 18.26
EM 56.47
ECE 12.59
56.70
13.06
45.42
18.01
18.54
4.2 Results for Out-of-Distribution (OOD) Tasks
Additionally, we evaluate the performance of MLLMs on the TyDiQA dataset, which collected from different resource and different setup than SQuAD, resulting in distribution shift for the model fine-tuned SQuAD. In Table 2 show the performance of the MLLMs on the TyDiQA test set, and observe the following:
Fine-tuning on SQuAD helps more than fine-tuning on TyDiQA (en), even though the task is OOD. This can be attributed to the much larger amount of fine-tuning data SQuAD compared to TyDiQA-en which
The accuracy of the models XQuAD Tahle Appendix. The language distribution of pre-training LLaMa2 is in Appendix A.6

---

Published in Transactions on Machine Learning Research (12/2025)
makes up for the task distribution shift.
Fine-tuning all TyDiQA languages helps much more than just TyDiQA-en SQuAD.
Table 2: Comparison of the calibration performance of various multilingual LLMs. The models are fine-tuned on SQuAD, TyDiQA-en, and TyDiQA-all, respectively and are evaluated on the test set of TyDiQA. Note that the results are averaged over all the 9 languages.
XLM
EM(()
ECE(L)
ECE(en)(()
mT5
EM(1)
ECE(()
ECE((e))())
SQUAD
50.04
14.78
14.11
SQUAD
50.31
24.37
18.47
TyDiQA-en
35.74
16.40
7.33
TyDiQA-en
10.13
26.96
19.16
-yDiQA-all
66.09
12.83
11.37
TyDiQA-all
67.11
18.03
19.22
In our experiments, for both in-domain and out-of-domain tasks, we find that extractive and generative models (scaled from 100M to 8B parameters) are consistently more miscalibrated on non-English inputs, with substantially higher calibration error than on English. This persistent cross-lingual calibration gap highlights the need to improve prediction reliability in multilingual settings.
5 Techniques to Improve the Calibration of MLLMs
In this section, we explore the role of various strategies to improve calibration including post-hoc methods, regularization methods, and ICL. We aim to address the following questions:
Do existing calibration techniques work in our cross-lingual transfer settings?
Can data-augmented fine-tuning on translated pross-lingual data improve calibration?
What are the comparative impacts of having more monolingual data versus having more diverse, cross-
lingual data?
Can using in-context examples improve confidence calibration?
5.1 Post-hoc Methods: Temperature Scaling
In Tables 12 demonstrate the benefits of using temperature scaling (TS) and few-shot learning (FS) on calibration for extractive models and generative models. TS does not affect accuracy by design, but it pro- vides significant. benefits in calibration in most cases. The optimized T > 1 pushes overconfident predictions into confidence ranges that more accurately reflect their true correctness, thereby reducing calibration error. Our experiments explore the impact different validation sets: 1) the SQuAD validation dataset (10.6k English sentences); 2) Merged MLQA validation dataset (with 7 languages, 3k sentences). We also observe that optimizing the temperature on a relatively small multilingual validation dataset more powerful than larger Euglish-only validation dataset. Notably, even though some of the languages (e.g. SW, KO) do not occur in the merged validation dataset of XQuAD and TyDiQA, the temperature computed the merged dataset still effectively improves the calibration performance in those languages
Table 3: Calibration performance after applying temperature scaling (TS) and few-shot learning (FS) on mBERT and mT5, evaluated on the XQuAD test set.
mBERT
EM(1)
ECE(L)
ECF((n)
m55
EM(1)
ECE(L)
F.C((en)(L)
46.77
16.36
6.78
50.95
23.06
10.88
TS (SQuAD)
16.77
6.3
6.66
TS (SQuAD)
50.95
14.05
4.84
TS (Merge)
16.77
5.86
7.87
TS (Merge)
50.95
10.19
2.77
FS
48.38
5.87
5.27
FS
54.10
21.22
11.04
Labeel Smoothing experiments and the observed benefits on calibration are in Appendix A.3

---

Published in Transactions on Machine Learning Research (12/2025)
5.2 Data Augmentation via Translation
We now investigate the effects of augmenting the training data by incorporating translated data. In our experiments, we sampled 9929 training examples in English and obtained their translations in five different languages (AR, DE, ES, HI, VI We have four dataset configurations: En denotes a subset of the English data; En-Large demotes the full English data. with available translations; En-tr denotes the En subset along with its translations in other languages; Mixed denotes each subset is from different language. We fine-tune the MsLMs this mixed dataset and evaluate their calibration performances. We show the detailed results in Table [ We see that data augmentation via translation improves calibration performance (even for languages not included in the translation) by almost 75% and we observe the following helpfulness ranking; Mixed > En-tr > En-large
En.
Table 4: Calibration after adding translated data during fine-tuning mBERT and mT5, evaluated the XQuAD lest set. The size of En, En-tr is 9929, and En-large, Mixed is 59574.
m BERT
EM(1)
ECE(1)
ECF((n)(L))
mT5
EM(()
ECC(())
RCF((en(()
En
43.51
17.75
9.93
En
49.88
24.05
18.36
En-tr
16.69
5.58
10.91
En-tr
53.63
22.37
14.66
En-large
16.18
12.18
3.73
En-large
50.99
23.27
11.91
Mixed
49.39
5.56
10.47
Mixed
54.93
20.97
13.48
5.3 In-context Learning
In-context learning (ICL) (Brown et al. 2020) has been used as an efficient way to direct LLMs to quickly adapt to a new task. It appends the task demonstrations with the input prompts which enables the LLMs learn the task from the examples effectively. While ICL is known boost accuracy, here, we demonstrato it also improves calibration on multilingual QA" We experimented with two ICL variants: 1) RANDOM: we randomly select two samples of the target language from the training dataset 2) ADAPTIVE: choose the two samples of the target language from the training data most similar to the test input, measured by the simi- larity between the contextual embeddings. Specifically, we use the contextual embedding from the encoder LLM for the test input and all the training examples and find the examples with the highest cosine similarity.
60
D shot RANDOM ADAFTIVE
16
shot
RANDON
ADAPTIVE
12
Exact Match
40
ECE
30
20
10
0
40
SW
k0
SW
Figure 4: Comparison of accuracy (Left) and calibration (Right) of LLaMa2 across six languages of TyDiQA under 1) zero-shot setting, 2) RANDOM: random 2-shot ICL 3) ADAPTIVE: most similar 2-shot ICL. Lower ECE, and higher EM indicate better performance.
7The translated sentences obtained from the MLQA-translated dataset. We diagrammatically illustrate the different configurations in Figure 10 Appendix A.11 9ICL is particularly useful in calibrating massive L.Lss such as IlaMa where TS hecomes prohibitively expensive due temperature optimization the validation data.

---

Published in Transactions on Machine Learning Research (12/2025)
80
PANDOM ADAFTIVE
20.0
PANDUM
shot
70
17.5
Exact Match
60
15.0
50
12.5
ECE
40
10.0
30
7.5
20
5.0
10
2.5
0
0.0
en
10
2
SW
en
1d
ko
ru
SW
Figure 5: Comparison of accuracy (Left) and calibration (Right) of Aya across six languages of TyDiQA
under 1) zero-shot setling, 2) RANDOM: random 2-shot ICL 3) ADAPTIVE: most similar 2-shot ICL. Lower ECE, and higher EM indicate better performance.
We present and compare the calibration performance of LLaMa2-7B (fine-tuned on SQuAD) and Aya- expanse-8B across six different languages with different ICL choices on TyDiQA in Figure 4 and Figure Lu We observe that the exact match rate is low for zero-shot learning but ICL with samples can significantly improve the performance for Llama-2. For Aya, English has higher ECE than the other languages. Never- theless, in-context learning still remains an efficient way to improve confidence calibration, consistent with our observations Llama-2. We additionally demonstrate that selecting the most similar 2 samples for the in-context learning improves both accuracy and calibration performance, and decreases ECE by more than 10% compared to RANDOM ICL. For a low-rcsource language like Swahili, ADAPTIVE ICL generates much beller-calibrated output compared to zero-shot and RANDOM for LlaMa-2. Figure i shows some qualitative examples where ICL helps in model calibration for Korean and Swahili.
Prediction
Confidence
w/o ICL
1894
Contect: of 2/ 2. 2 (a
21 13 7- 41 2/4,
E:0) 3E 2 2/
ICL
1894
0.72
Context: Jean Bodel Bokassa (" 22 Fe21 133 Novembe 1096) aikuwa
w/o
Septemba
X
Esadaye Kaiseri we Jamburi
Afrika
Milti Afrila Kati
supinduliwa Septemmb
X
Question: Jejean Bedel Bokassa alicalwa masak upi?
Figure 6: When appending the Korean and Swahili examples in the prompts, the model is more confident about the correct prediction and less confidence about the wrong prediction.
Given our experimental results, we confirm that existing calibration techniques are effective in cross-lingual settings. In particular, optimizing temperature validation set yields stronger multilingual confidence calibration than temperature scaling English only and other calibration techniques such as label smoothing and few-shot learning. We also observe that data augmenting with semantically diverse cross-lingual data is more helpful than simply adding more monolingual data for improving multilingual confidence calibration. The observations apply for both extractive and gencrative models. Finally, for more powerful docoder-only language models such as LLaMA-2-7B and Aya- 8B, in-context learning improves both accuracy and confidence calibration across languages, with larger gains for lower-resource languages.
relatively small but semantically diverse multilingual
10 results of the random 2-shot. ICL experiment are averaged over runs.

---

Published in Transactions on Machine Learning Research (12/2025)
6 Additional Ablation Experiments and Discussion
In this section, perform ablations to answer the following research questions:
the relationship between linguistic and non-linguistic features, and the transferability of calibration per-
formance from English to other languages.
the effect of model size on calibration, and accuracy, for a particular model family TT
6.1 Investigating the effects of language distance and corpus size on calibration
One important aspect of the multilingual model analysis is the factors that affect the model's cross-lingual ability (Jones et al. 2021 Jiang et al., 2022 Abuja et al. (distances between languages) and non linguistt factors (pre-training data size) for calibration performance. Lingustic Features Following previous work (Ahuja et al. 2022 Jones et al., 2021 J. we load the syntactic and genetic distances that are pre-computed by the URIEL project (Littell et al. 2017], as the distance measurements between Euglish and other target languages. The syntactic distance measures the structural differences between various languages. The genetic distance quantifies the genetic relation between languages according to the Glottolog tree. We investigate whether the closeness of the target language to the source language, English, implies betler calibration performance. To measure this compute Pearson's 's correlation coefficient between the language distance and ECE of the standard models. Table shows that the calibra- tion performance is highly correlated with the syntactic distances between English and the corresponding languages.
2022). Here we focus on analyzing both linguistic
Ron-usiguttic Features We also explore the impact of non-linguistic features from training. In this sec- tion, we compute the correlation between training data size and calibration performance and Table 5 indicates that the size of different languages in the pre-training influences the cross-lingual calibration of QA models.
Table 5: Pearson's correlation coefficient between segguistii/noon-lnnniiss characteristics and calibration error on the XQuAD dataset for various models (Left: Extractive QA models, Right: Generative QA models). Higher absolute values indicate better correlation. Syn and Gen denote the syntactic and genetic distance between English and other languages respectively, and Size indicates the proportion of the pre- training data in each language.
Model
Syn
Gen
Size
Model
Syn
Gen
Size
mBERT 0.74
0.68
-0.72
mT5
0.77
0.69
-0.68
XLM-R
0.56
0.54
-0.40
mBART
0.76
0.72
-0.31
LLaMa2
0.60
0.53
-0.57
6.2 Investigating the effects of model size on calibration
Lastly, we investigate the effect of model size on confidence calibration. While Guo et al. (2017) demonstrated that calibration degrades with increasing model size (on ResNet), we show that this not the case for pre- trained MLLMs. We demonst.ate the effect of model size on extractive and generative models for XQuAD and TyDiQA (Tables 6 The table shows that the accuracy increases with model size, expected. Further, note that confidence calibration also improves with increasing model size, which also supports prior findings on the calibration of monolingual encoder LLMs in Dan & Roth (021)).
In this section, we have examined how both linguistic and non-linguistic factors shape calibration perfor- mance. On the linguistic side, we focus on the syntactic and genetic distance between English and the target language; on the non-lingiistic side, we consider pre-training size. We find that all these factors affect the model's calibration performance. In addition, we observe consistent scaling pattern: model size increases, accuracy improves and calibration error decreases.
We additionally investigate how the calibration of multilingual models corresponds to their monolingual counterparts in A Appendix A.13.2 Additional madel details are in Appendix A.2
10

---

Published in Transactions on Machine Learning Research (12/2025)
Table 6: Calibration performance of XLM-R (Left) and m'T5 (Right) across different sizes, evaluated on the XQuAD test set and TyDiQA test sets.
XLM-R
XQUAD
TyDiQA
m'T5
XQuAD
TyDiQA
EM(1)
ECE())
EM(1) RCE(1)
EM(1)
ECE(L)
EM((1)
ECF((1)
Small
38.44
27.44
32.05
30.29
Base
54.93
12.38 10.69
50.04 61.02
14.78
Base
50.95
23.06
50.31
24.37
Large
60.50
13.42
Large
56.02
21.60
56.91
20.27
7 Conclusion
In this work, we performed comprehensive study of the confidence calibration of MLLMs, focusing on the QA task. We studied diverse scenarios, covering extractive and generative LLMs, several QA datasets, different calibration strategies, different inference settings (TCL, and fine-tuning), and generalization under distribution shifts. We summarize the key insights from our paper: (1) Multilingual models need to be calibrated, especially in zero-shot settings, before deployment in real-world applications. 2) Temperature scaling on a Laaguu-- validation dataset is a very effective calibration strategy. 3) Adding cheap translo-translated data at the fine-tuning stage helps improve calibration even on languages unseen during fine-tuning. 4) ICL benefits not only the accuracy of powerful LLMs, but also their confidence calibration multilingual tasks. We believe our work will be fundamental in spurring progress towards developing reliable multilingual systems and advancing NLP forlow-resources languages.
Acknowledgment
We thank the anonymous reviewers for their constructive feedback and suggestions. This research sponsored by the Army Research Office and was accomplished under Grant Number 20---0000-0000. The views expressed are those of the authors and do not reflect the official policy or position of the Army Research Office or the U.S. Government.
References
Kabir Ahuja, Sunayana Sitaram, Sandipan Dandapat, and Monojit Choudhury. On the calibration of mas- sively multilingual language models. arXie preprint arXio:2210. 12265, 2022.
Mikel Artetxe, Sebastian Ruder, and Dani Yogatama. On the cross-lingual transfcrability of monolingual representations. ions. CoRR, abs/1910.11856, 2019.
Tom Brown, Benjamin Mann, Nick Ryder, Melanic Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. Advances in neural information processing systems, 33:1877 1901, 2020.
Jonathan H. Clark, Funsol Choi, Michael Collins, Dan Garrette, Tom Kwiatkowski, Vitaly Nikolaev, and Jennimaria Palomaki. Tydi qa: A benchmark for information-sseking question answering in typologically diverse languages. Transactions of the Association for Compulational Linguislics, 2020.
Alexis Conneau, Kartikay Khandelwal, Naman Goyal, Vishrav Chaudhary, Guillaume Wenzek, Francisco Guzmán, Edouard Grave, Myle Ott, Luke Zettlemoyer, and Veselin Stoyanov. Unsupervised cross-lingus representation learning at scale. CoRR, abs/ 1ts//9911022116 2019.
Soham Dan and Dan Roth. On the effects of transformer size on in-and out-of-domain calibration. Findings of the Association for Computational Linguistics: EMNLP 2021, pp. 2096 2101, 2021.
John Dang, Shivalika Singh, Daniel D'souza, Arash Ahmadian, Alejandro Salamanca, Madeline Smith, Aidan Peppin, Sungjin Hong, Manoj Govindassamy, Terrence Zhao, Sandra Kublik, Mcor Amer, Viraat
Aryabumi, Jon Ander Campos, Yi-Chern Tan, Tom Koemi, Florian Strub, Nathan Grinsztajn, Yannis Flet-Berliac, Acyr Locatelli, Hangyu Lin, Dwarak Talupuru, Bharat Venkitesh, David Cairuz, Bowen
11

---

Published in Transactions on Machine Learning Research (12/2025)
Yang, Tim Chung, Win Ko, Sylvie Shang Shi, Amir Shukayev, Sammie Bae, Aleksandra Piktus, Roman Castagné, Felipe Cruz-Salinas, Eddie Kim, Lucas Crawhall-Stein, Adrien Morisot, Sudip Roy, Phil Blunsom, Ivan Zhang, Aidan Gomez, Nick Frosst, Marzieh Fadaee, Beyza Ermis, Ahmet Üstün, and Sara Hooker. Aya expanse: Combining rescarch breakthroughs for a new multilingual fronticr, 2024. URL https://arrs wttp://w..gggbb////.eaae 04261
Shrey Desai and Greg Durrett. Calibration of pre-trained transformers. In Proceedings of the 2020 Conference on Empirical Methods in Natural Longuage Processing (EMNLP), pp. 2020.
Tim Dettmers, Artidoro Pagnoni, Ari Holtzman, and Luke Zettlemoyer. Qlora: Efficient finetuning of quantized llms, arxiv preprint arx::22305. 14314, 2023.
Jacob Devlin, Ming-Wor Wei Chang, Kenton Lee, and Kristina Toutanova. BERT: Pre-training of deep bidi- rectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Compulational Linguisties: Human Language Technologies, Vol- umn (Long and Short Papers), pp. 4171 4186, Minneapolis, Minnesota, June 2019. Association for Computational Linguistics. doi: 10. 10.18653 11/119--N19/ URL htttps://aclatthoo/ hww.:aanttoooooooooo////. 1423
Rotem Dror, Gili Baumer,, Segev Shlomov, and Roi Reichart. The hitchhiker's guide to testing statistical significance in natural language processing. In Iryna Gurevych and Yusuke Miyao (eds.), Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 1383-1392, Melbourne, Australia, July 2018. Association for Computational Linguistics. doi: 10.18653/ v1/P18-1128. URL
http://ccaachoooooooooo/////111 org/P18-1128
Chuan Guo, Geoff Pleiss, Yu Sun, and Kilian Q Weinberger. On calibration of modern neural networks. In International conference on machine learning, pp. 1321-1330. PMTR, 2017.
Tianxing He, Bryan McCann, Caiming Xiong, and Ehsan Hosseini-Asl. Joint energy-based model training for better calibrated natural language understanding models. arXiv preprint arXi::2101.06829, 2021.
Abhyuday Jagannatha and Hong Yu. Calibrating structured output predictors for natural language pro- cessing. In Proceedings of the 58th Anreual Meeling of the Associalion for Compulational Linguistics, pp. 2078 2092, 2020.
Zhengbao Jiang, Jun Araki, Haibo Ding, and Graham Neubig. How can we know when language models know? on the calibration of language models for question answering. Transactions of the Association for Computational Linguistics, 9:962 977, 2021.
Zhengping Jiang, Anqi Liu, and Benjamin Van Durme. Calibrating zero-shot cross-lingual (un))structured predictions. In Yoav Goldberg, Zornitsa. Kozareva, and Yue Zhang (eds.), Proceedings of the 2022 Confer- ence on Empirical Methods in Natural Language Processing, pp. 2648 2674, Abu Dhabi, United Arab Emi- rates, December 2022. Association for Computational Linguistios. doi: 10.18653 hwwp:///...nnpppppmeeepe 170.. URL https:://acaatthollltl//
org/2rg222 emnlp-main. 170
Alex Jones, William Yang Wang, and Kyle Mahowald. A massively multilingual analysis of cross-linguality in shared embedding space. ar preprint arXiv:2109.000 203.6344,,, 2021.
Amita. Kamath, Robin Jia, and Percy Liang. Selective question answering under domain shift. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pp. 5684-5696, Online, July 2020. Association for Computational Linguistics. doi: 33855//112020.ac--mm www.athwwwwooooo//./22. acl miel-main.55. 503
URL https:
Daniel Khashabi, Sewon Min, Tushar Khot, Ashish Sabharwal, Oyvind Tafjord, Peter Clark, and Hannaneh Hajishirzi. Unifiedqa: Crossing format boundaries with a single qa system. In Findings of the Association for Computational Linguislics: EMNLP 2020, pp. 1896 1907, 2020.
Volodymyy Kuleshov and Percy S Liang. Calibratod structured prediction. Advances in Neural Information Processing Systems, 28, 2015.
12

---

Published in Transactions on Machine Learning Research (12/2025)
Mike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, Abdelrahman Mohamed, Omer Levy, Ves Stoyanov, and Luke Zettlemoyer. Bart: Denoising sequence-to-sequenee pre-training for natural language generation, translation, and comprehension. arXiv preprint arXiv:1910.13461, 2019a.
Patrick Lewis, Barlas Oguz, Ruty Rinott, Sebastian Riedel, and Holger Schwenk. MIqa: Evaluating cross- lingual extractive question answering. arXiv preprint arXiv:1910.072 art. arXiv: 1910.07475, 2019b.
Patrick Littell, David R. Mortensen, Ke Lin, Katherine Kairis, Carlisle Turner, and Lori Levin. URIEL and lang2vec: Representing languages as typological, goographical, and phylogenetic vectors. In Mirella Lap- ata, Phil Blunsom, and Alexander Koller (eds.), Proceedings of the 15th Conference of the European Chap- ter of the Association for Computational Linguistics: Volume 2, Short Papers, pp. 8-14, Valencia, Spain, April 2017. Association for Computational Linguisties. URL
Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. Roberta: A robustly optimized bert pretraining approach. arXiv preprint or Xiu: 1907 1 1692, 2019.
Yinhan Liu, Jiatao Gu, Naman Goyal, Xian Li, Sergey Edunov, Marjan Ghazvininejad, Mike Lewis, and Luke Zettlemoyer. Multilingual denoising pre-training for neural machine translation. Transactions of the Association for Computational Linguistics, 8:726-742, 2020. doi: 10.1162/tacl 00343. URL https:
//aclamtboloogyorr22202....aa.la wttp://00....1-111211
Ekaterina Loginova, Stalin Varanasi, and Günter Neumann. Towards end-to-end multilingual question
answering. Information Systems Frontáers, 23:227 241, 2021.
Matthias Minderer, Josip Djolonga, Rob Romijnders, Frances Hubis, Xiaohua Zhai, Neil Houlsby, Dustin Tran, and Mario Lucic. Revisiting the calibration of modern neural networks. Advances in Neural Infor- mation Processing Systems, 34:15682-15694, 2021.
Rafacl Muller, Simon Kornblith, and Geoffrey E Hinton. When does label smoothing help? Advances in neural information processing systems, 32, 2019.
Colin Raffel, Noam Shazcer, Adam Roberts, Katherine Lec, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. Exploring the limits of transfer learning with a unified text-to- text transformer. Journal of Machine Learning Research, (1(10)):1-6) 67, 2020.
prgg:papers//v2/2000744
html
Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang. SQuAD: 100,000+ Questions for Machine Comprehension of Text. arXiv e-prints, art. arXiv:1606.05254 2016.
Nils Reimers and Iryna Gurevych. Sentence-bert: Sentence embeddings using siamese bert-networks. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing. Association for Computational Linguistics, 2019. URL. hhttp://axxiv/ ietaxiv..org. org/ a9s/ 10084
Ali Salaya, Moutasem Abdullatif, and Deniz Yuret. KUISAIL, at SemFval-2020 task 12: BERT-CNN for offensive speech identification in social media. In Proceedings of the Fourteenth Workshop on Semantic
Evaluation, pp. 2054-2059, Barcclona (online), December 2020. International Committee for Computa-
tional Linguistics. URL lattps://mrtt http://wwwwwwwww..//pp.ptp aclweb. wwrg/anthology/222200 2atthologg/22220..mee11111111111 semeval-1.275
Chenglei Si, Chen Zhao, Sewon Min, and Jordan Boyd-Graber. Re-examining calibration: The case of question answering. In Findings of the Association for Computational Lingwistics: EMNLP 2022, pp. 2814-2829, Abu Dhabi, United Arab Emirates, December 2022. Association for Computational Linguistics.
URL attpss//aclanthan www.lo/ww.....222...iiii----eeee
Robert Smith. Llama squad. httpp:///gttaucccm////ccaaaaaaaa
git, 2023.
Christian Szogedy, Vincent Vanhoucke, Sergey Toffe, Jon Shlens, and Zbigniew Wojna. Rothinking the inception architecture for computer vision. In Proceedings of the IEEE conference on computer vision and pollern recognition, pp. 2818-2826. 2826, 2016.
13

---

Published in Transactions on Machine Learning Research (12/2025)
Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaci, Nikolay Bash- lykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. Llama 2: Open foundation and fine-tuned chat models. arXic preprint orXiu:2307.09288, 2023.
Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Remi Louf, Morgan Funtowicz, Joc Davison, Sam Shleifer, Patrick von Platen, Clara Ma, Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gugger, Mariama Drame, Quentin Lhoest, and Alexander Rush. Transformers: State-of-the-art natural language processing. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations, pp. 38 45, Online, October 2020. Association for Computational Linguistics. doi: 10.18653/v1/2000 emnlp-demos.6. URL hattps:/aallantholoyy org/ org/2020. 2020. canlp-demos..
Shijie Wu and Mark Dredze. Are all languages created equal in multilingual BERT? In Proceedings of the 5th Workshop on Representation Learning for NLP, pp. 120-130, Online, July 2020. Association for Computational Linguistics. doi: (0.18653/1112202.xepbbiil 1.16. URL https://aclanthoolgyy org/
2020.rep14mlpp11116
Haotian Xu and Yingying Zhang. Triple-hybrid energy-based model makes better calibrated natural language understanding models. In Proceedings of the 17th Conference of the European Chapler of the Association for Computational Linguistics, pp. 274 285, Dubrovnik, Croatia, May 2023. Association for Computational Linguistics. URL atttp:://aclanthology org/ 2023. eacl-main. 21
Linting Xue, Noah Constant, Adam Roberts, Mihir Kale, Rami AI-Rfou, Aditya Siddhant, Aditya Barua, and Colin Raffel. mT5:: A massively multilingual pre-trained text-to-text transformer. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Lènguistics: Human Language Technologies, pp. 483 498, Online, June 2021. Association for Computational Linguistics. doi: 10.18653 /t///211112122 maacl- URL
Shujian Zhang, Chengyue Gong, and Eunsol Choi. Knowing more about questions can help: Improving calibration in question answering. Xixiv preprint 2XXx:210600149. 2021.
14

---

Published in Transactions on Machine Learning Research (12/2025)
A Appendix
A.1 Details of datasets
A.2 Details of architectures
A.3 Details experiments
A.4 4 Details of LLaMa2 experiments
A.5 Additional calibration metrics
A.6 Distribution of languages for pre-training the models
A.7 Additional Calibration Results
A.8 Additional results on MI.QA
A.9 Additional results on TyDiQA
A.10 Additional result on XQuAD
A.11 Additional results for data augmentation
A.12 Additional In-context Learning Results
A.13 Additional Analysis
A.1 Dataset Details
Here we provide more statistics the datascts.
Table 7: The number of training and test samples in SQuAD, XQUAD, MLQA, TyDiQA-GoldP.
Train
Test
SQuAD
87.6K
10.6K
XQuAD
N/A
14.28K
MLQA
N/A
46.14K
TyDiQA-GoldP
49.88K
5.08K
Table 8: List of languages in XQUAD, MLQA, and TyDiQA-GoldP
XQuAD
MLQA
TyDiQA-GoldP
Arabic
Arabic
Arabic
German
German
Bengali
Greek
English
Finnish
English
Spanish
English
Spanish
Hindi
Swahili
Hindi
Vietnamese
Korean
Romanian
Chinese
Indonesian
Russian
Russian
Thai
Telugu
Turkish
Vietnamese
Chinese
15

---

Published in Transactions on Machine Learning Research (12/2025)
A.2
Architecture Details
In section 6.2
investigate the relationship between calibration performance and model size. Here, we provide more details of XLM-R and m115 in a range of different sizes. XLM-R-base R-base contains 12 layers with 768 hidden states and 8 heads while the large version contains 24 layers, 1027 hidden states, and 16 heads (Conneau et al.) 2019; Wolf et al. 2020). The small variant mT5 consists layers for both the encoder and decoder components, where the base mT5 model has 12 layers, and the large mT5 model has 24 layers.
Table The number of parameters for XLM-R and m
No. of parameters
XLM-R
m15
Small
N/A
300M
Base
125M
580M
Large
355M
1.2B
A.3
Experiment Details
The monolingual models in other languages (section 6.2);
Hemman-BRRT. http:::/wwww.deepstttaaa///////e
Arabic-BERT:
Chinese-BERil http::///anggiiggeeeecc////////a arabic
All the experiments are run on two 48G NVIDIA RTX A6000 GPUs. During the fine-tuning process of the multilingual language models, learning rate of 3e-5 is chosen from hyperparameter selection range of [le-5, 2e-5, 3e-5]. The batch size is set to 32. The extractive QA models are fine-tuned for 3 epochs and the generative QA models are fine-tuned for epochs with AdamW optimizer. LLaMa2 is fine-tuned on SQuAD dataset for epoch with QLoRA (Dettmers et al. 2023 mechanism (Smith 2023) and Aya is finc-tuned for 1000 steps. It takes about 2 hours to train extractive models, about hours to train generalive models, and 24 hours to train LLaMa2. We follow the implernenaations from Huggingface (Wolf lt al. (020) load the dataset and fine-tune the pre-trained models. The license of LLaMa2 can be found in m5T5 and multilingual BERT are using apache-2.0 license, mBART
https: //ai. wwta. com/llama/license and XLM-R are using MIT license. TyDiQA is using apache-2.0 2.0 license, MLQA and XQuAD are using by-sa-3.0. The models and datasets are consistent with their use in research.
A.4 LLaMa2 experiments configurations
We provide additional details of in-context learning experiments in this section. The prompts used for querying LLaMa2 is
Extract the minimal span word from the following context that best answers the question.
### Question:
(question)
### Context:
(context)
### Answer:
The encodcr-only language model used for extracting contextual embodding is sentenesstraasfomeesssshh xlm-r-multilingual Reimers & Gurevych (0199).
A.5 Additional calibration metrics
Reliability Diagrams is visual depiction of confidence calibration. These diagrams plot the average accuracy of each bin B,, (Y-axis), acc( Bm)) as function of the bin confidences (B, (X-axis, sorted
16

---

Published in Transactions on Machine Learning Research (12/2025)
in increasing order). If the model is perfectly calibrated, then the diagram should plot the identity function (Y = X line). The greater the deviation from the diagonal, the more miscalibrated the model is. We show an example of reliability diagram in Figure !
A.6 Language Distribution for different models
In this section, we show the language code mapping and the proportion training data that comes from a specific language for various pro-trained multilingual LLMs.
Table 10: Language code mapping.
LANG CODE
EN
English
AR
Arabic
DE
German
EL
Greek
ES
Spanish
HI
Hindi
RO
Romainian
RU
Russian
TH
Thai
TR
Turkish
VI
Vietnamese
ZH
Chinese
KO
Korean
FI
Finnish
SW
Swahili
ID
Indonesian
BN
Bengali
TE
Telugu
Table 11: The distribution of different languages in the pre-training data for LLaMa2, m mBRART, mBERT, XLM-R, shown in percentages. Note that only display the languages tested in the XQuAD and TyDiQa dataset.
LANG
LLaMa2
T5
mBART
mBRT
XLM-R
EN
89.70
5.67
21.67
22.54
12.56
AR
$0.005
1.66
2.04
0.68
1.17
DE
0.17
3.05
4.86
7.10
2.78
EL
<0.005
1.54
0.0
0.33
1.96
ES
0.13
3.09
3.89
3.53
2.22
HI
<0.005
1.21
1.47
0.19
0.84
RO
0.03
1.58
4.48
2.80
2.56
RU
0.13
1.21
20.30
2.80
11.61
TH
<0.005
1.14
0.0
0.34
2.99
TR
<0.005
1.80
1.52
1.03
0.87
VI
0.08
1.87
10.02
5.73
ZH
0.13
1.67
3.42
1.80
1.69
KO
0.06
1.14
3.96
0.66
2.26
FI
0.03
1.35
3.96
1.58
2.27
SW
<0.005
0.5
0.0
0.07
0.07
ID
0.03
1.80
0.0
0.80
0.80
BN
50.005
0.91
0.0
0.16
0.35
TE
<0.005
0.52
0.0
0.36
0.20
17

---

Published in Transactions on Machine Learning Research (12/2025)
A.7 Additional Calibration Result
A.7.1 Label Smoothing (LS)
Szegody et al. 2016; Muller et al. 2019) is regularization technique that constructs new target vector (112) from the one-hot larget vector (h4), where h = -4 a/C for a C class classification problem. The range of the label smoothing weight a from 0 to For question answering via encoder-based models, C corresponds to the length of the context, since we want seleet one of C' start positions and one of C end positions for the answer. Accordingly, the a mass for the correct index distributed to all other indices. Note that have two hyper-parameters on and 02 corresponding to the start and end locations, which can be selected independently via optimizing the validation set. For all the Label Smoothing experiments, we set the hyperparameter = 0.1.
We show additional calibration performance after confidence calibration for mBERT, XLM-R and mBART on XQUAD.
Table 12: Calibration performance after applying temperature scaling (TS), label smoothing (TSS), and few- shot learning (FS) mBERT and m'T5, evaluated the XQuAD) test set.
mBERT
EM(t)
ECE(L)
(CE(en))
XLM-R
EM(1)
ECE(J)
(CE(en)
46.77
16.36
6.78
54.93
12.38
6.84
TS (SQuAD)
46.77
6.3
6.66
TS (SQuAD)
54.93
4.16
4.25
TS (Merge)
46.77
5.86
7.87
TS (Merge)
54.93
4.29
5.86
LS
47.80
14.65
1.88
LS 55.08
9.88
4.23
FS 48.38
5.87
5.27
FS
53.99
4.39
5.16
Table 13: Calibration performance after applying temperature scaling (TS), label smoothing (LS) and few- shot learning (FS) on QA models: XLM-R and mBART, evaluated on the XQuAD test set.
mBART
EM(1)
ECE(L)
ECE(en)(L)
39.18
22.64
9.19
TS (SQuAD)
39.18
15.05
4.26
(Merge)
39.18
10.23
2.97
FS
54.14
17.49
9.7
A.8 Additional Results MLQA
Table 14 shows the accuracy and calibration performance (ECE) for mBERT, XLM-R, mT5, mBART, LLaMa2 on MLQA.
Calibration performance of four different models on the MLQA dataset (7 languages). exact match rate. (L) denotes higher (lower) is better respectively.
Table 14:
EM denotes
EN
AR
DE
ES
HI
ZH
EM
ECE
EM
ECE
EM ECE
EM
ECE
EM
ECE
EM
ECE
EM
ECE
(t)
(L)
(1) (4) (t)
(L)
(1)
(4)
(t)
(4)
(1)
(4)
(1)
(4)
BBERT
58.28
12.95
24.76 36.08
32.42
33.62
33.0
36.21
28.99
30.83
29.16
33.50
36.31
25.79
XLM-R
57.06
12.79
28.82 30.29
31.84
30.31
36.20
31.64
42.21
18.24
36.59
27.79
40.28
20.58
m55
64.94
15.24
30.46
37.73
44.28
30.17
43.35
30.66
35.60
32.09
36.89
33.77
37.03
27.89
mBART
60.02
13.82
19.01
33.56
36.64
32.02
29.02
39.47
24.30
31.25
31.44
33.34
30.11
28.42
LLaMa2
57.72
1.96
18.11
24.73
35.09
16.54
33.79
17.7
19.24
22.27
28.37
17.98
34.3
18.11
18

---

Published in Transactions on Machine Learning Research (12/2025)
A.9 Additional Results TyDiQA
We present accuracy (Figure 7 and ECE (Figure 8 for XLM-R, mT5 and mBART across 9 languages on
TyDiQA. We also show additional results of hut-oo-Distributioo comparison in Table 15
60
XLM
mT55
55
mBART
Accuracy
50
45
40
35
en
ko
bn
Language
Figure 7: This figure compares the accuracy (EM) of mBERT, XLM-R, mT5, mBART across nine languages of TyDiQA.
Table 15: Comparison of the calibration performance of mBERT and mBART. The models are evaluated on the test set of TyDiQA. EM: exact match rate.
EM(1)
ECE())
ECE((n))()
mBERT-SQUAD
44.68
17.30
11.46
mBERT-Ty TyDiA-e
44.99
11.12
5.95
mBERT- IBEETT-TYDIQA--..
68.74
10.98
8.02
mBART -QuAD
37.41
27.04
16.56
mBART-TTDiQQ--u
41.52
35,55
25.18
TBBAIAI-T
64.87
14.33
12.35
A.10
Additional Result on XQuAD
We show the detailed performance of five different models on XQuAD in Table 16
We also show the individual ECE before/after calibration for mBERT, XLM-R, mT5 and mBART on XQuAD in Figure
A.11 Additional Results for Data Augmentation
Here show diagram for visualizing the setting of our data augmentation experiment in Figure 10 and provide additional results for XLM and mBART in Table 17
Furthermore, it is worth noting that even for languages such RU (Russian), Ro (Romanian) and Tr (Turkish), that were not. included in the fine-tuning stage, we observed significant improvement in their performance as shown in Figure[I]] This suggests that the benefits of the fine-tuning process extend beyond the explicitly included languages and have positive impact on the overall performance across various lan- guages in the evaluation set. Theoretically studying how translated sentences affect fine-tuning performance, and the relation with language similarities interesting avenue for future study.
19

---

Published in Transactions on Machine Learning Research (12/2025)
no
20
squad merge
20
squad
merge
15
15
ECE
ECE
10
10
5
5
0
en
ru
1d
f7
SW
bn
en
ru
1d
SW
ar
ko
bn
te
40
no
35
squad
merge
30
25
ECE
20
15
10
5
0
2
ar
ko
bn
Figure 8: This figure compares the calibration performance (ECE) of XLM-R, mBERT, mBART across nine languages of TyDiQA under 1) no calibration 2) temperature scaling on English validation dataset 3) temperature scaling on a 7-language validation dataset.
Calibration performance of four different models on the XQuAD dataset (12 languages). EM denotes exact match rate. 1 (4) denotes higher (lower) is better respectively. "Note that mBART has not seen EL even during pre-training so cannot generate answers correctly in that language.
Table 16:
EN
AR
DE
EL
ES
HII
EM(1)
ECE(L)EM(1)
ECE(L)EM(1)
ECE())EM(1) ECE(LJEM(() ECE(L)EM(()
BCE(L)
mBERT
66.92
5.21
41.76
17.49
54.99
11.09
42.44
15.77
55.71
9.37
39.86
19.46
XLM-R
66.58
5.68
47.7
15.49
57.2
9.99
52.58
11.76
54.29
12.24
50.78
12.25
mT55-
72.21
10.9
45.88
26.01
56.13
19.84
37.87
28.91
56.81
19.36
46.64
26.65
mBART
67.59
9.07
23.98
29.91
46.13
21.53
7.23*
33.84*
43.61
20.58
30.06
29.56
LLaMa2
64.31
5.72
25.97
17.06
44.76
23.84
18.15
44.96
8.47
23.81
18.62
RO
RU
TH
EM(1)
(CE(())M(()
ECE())EM(1)
ECE(J)EM(() (CE())EM(()
ECE())EM(()
ECE(L)
mBERT
58.74
9.91
50.78
15.96
28.21
23.2
35.1
16.45
47.48
15.22
46.64
16.72
XLM
61.4
9.29
54.59
12.51
55.57
6.75
50.5
12.73
51.82
14.37
55.55
10.05
mT5
59.19
19.11
39.55
26.43
49.92
23.62
49.08
24.79
47.76
27.84
57.93
18.92
mBART
47.79
21.57
32.3
27.04
22.86
31.89
40.08
26.32
36.69
30.52
47.9
20.73
LLaMa2
36.41
11.91
33.61
16.68
22.49
18.13
23.89
15.62
37.06
12.99
57.23
3.73
20

---

Published in Transactions on Machine Learning Research (12/2025)
16
squad merçe
20
merge
14
12
15
8
10
6
4
2
2
ar
tr
no
no
SQU-dd
Squad
25
merge ge
merge
20
20
ECE
ECE
15
15
10
10
5
0
Figure 9: The figure presents results of the calibration performance, measured by ECE of various models mBERT, XLM-R, mT5, and mBART across 13 languages in the XQuAD dataset. 1) no calibration 2) temperature scaling on English validation dataset 3) temperature scaling on 7-language validation dataset. ECE is lower the better.
En
En-Large
Mixed
En-tr
Figure 10: This figure describes the setting of Sec. 5.2 Different colors denote different. examples and the different shapes denote different languages, eg: squares are English and circles are German. Thus each example (color) has a corresponding translation in the other languages (shapes). En denotes a subset of the English data and Large denotes the full English data with available translations. En-tr denotes the En subset along with its translations other languages. Mixed denotes each subset from different language. Note: Each colored shape has the same number of examples and thus En-Large, En-tr and Mixed have the same size.
21

---

Published in Transactions on Machine Learning Research (12/2025)
Table 17: Calibration performance after adding translated pairs at the fine-tuning stage for XLM-R and mBART, evaluated on the XQuAD test set. In this case, the size of En is 9929, the size of the En-tr, and En-large, Mixed 59574.
EM(1)
ECE(L)
ECE(en)
En
46.0
13.27
7.56
XLM-R
En-tr
49.65
5.24
11.34
En-large
53.66
11.82
5.82
Mixed
52.93
7.04
7.48
En 48.41
32.97
22.62
mBART
En-tr 49.96
32.42
24.02
En-large
45.18
35.79
20.83
Mixed
52.60
29.85
20.80
20
1833
20.0
En-tr Mixed
En-large
17.00
17.5
En-large
16.03
En-tr
15.03
15
Mixed
15.0
13.35
3.08
1.96
12.5
ECE
10
9.90
ECH
10.0
0.13
7.16
7.5
6.99
5
2.9
3.6g
4.40
5.0
3.95
3.90
3.02
2.79
2.24
2.5
0
Ru
Ro Languages
Tr
0.0
Ru
Ro
Tr
Languages
Figure 11: This figure shows the improvement of calibration brought by data augmentation via translation for different languages on mBERT (Left) and XLM-R (Right). En denotes subset of the English data and En-Large denotes the full English data, with available translations. En-tr denotes the En subset along with its translations in other languages. Mixed denotes each subset from different language.
A.12 Additional In-context Learning Results
Here we report additional results of confidence calibration with in-context learning. The model we used in this experiment is chat-optimized decoder-only model, LlaMa2-7B-chat. Note that we report Validation Answer (VA), which checks whether the gold answer is contained in the generated answer, instead of Exact Match (EM).
Table 18: Calibration performance of LlaMa2-chal with/without ADAPTIVE ICL.
EN
ID
FI
VA
ECE
VA
ECE
VA
ECE
0
41.36
10.82
26.19
14.16
18.29
23.18
ADAPTIVE
56.82
16.04
71.33
8.02
51.28
14.14
KO
SW
RU
VA
ECE
VA
ECE
VA
ECE
0
38.41
14.53
7.01
45.57
27.09
12.41
ADAPTIVE
52.54
8.32
44.69
27.71
43.72
23.97
22

---

Published in Transactions on Machine Learning Research (12/2025)
A.13 Additional Results for Analysis
A.13.1 Linguistic distances versus calibration performance
We performed additional statistical lesting to determine which factor significantly impacts calibration per- formance. More specifically, we applied the Wilcoxon signed-rank test as suggested in Dror al. (2018) and compared the Pearson correlation coefficients for the three factors across different models and multiple Our results indicate that the syntactic distance has more impact on the multilingual calibration performance compared to the genetic distance and pre-training size (with p-value= 0.0008 and p-value=0.000) when alpha = 0.05). We also compute the correlation between other types of linguistic features in Table 19 and the calibration performance. More details about the features can be found in Littell et al. (2017)
Pearson Correlation Coefficient between other linguistic characteristics and calibration error on XQuAD dataset for various models. The absolute value higher, the better. Here, Gen denotes the genetic distances between English and other languages, Geo indicates the geographical distance, Ive is the inventory distance, Pho denotes the phonological distance.
Table 19:
Model
Gen
Geo
Ive
Pho
mBRT
0.602
0.503
0.736
0.494
XLM-R
0.541
0.334
0.680
0.528
m5
0.687
0.213
0.629
0.353
mBART
0.716
0.301
0.751
0.495
XQuAD is parallel dataset in 12 languages and another interesting question for the multilingual language model is: if a model is confident in predicting the answer of English question-context pair, will it be confident in predicting the answer of a parallel question-context pair in other languages? Here, we compute the Pearson's 's correlation coefficient between the confidence for predicting answers of source language and those of target languages and plot the comparison Figure 12 We observe that languages more similar English exhibit a stronger correlation the model's confidence levels for identical question-context pairs across languages. More specifically, when the model shows higher confidence in an English input, tends to also display similar confidence in inputs from closer languages like Spanish and German. The correlations of the encoder-only models are much higher than encoder-decoder models.
Correlation Coefficient
1.0
MBERT
XLM
0.8
mT5
MBARTT
0.6
0.1
0.2
hl
Language
Figure 12: This figure presents the distance between English and other languages versus the correlation of the confidence between source language across four different models. Here Correlation Coefficient denotes Pearson's correlation coefficients. The higher the better. the y-axis labels, languages are ordered by their syntactic distance from English.
A.13.2 Monolingual vs Multilingual Models
We also want to compare the multilingual models with their monolingual counterparts on extractive English QA, focusing the SQuAD dataset. In Table 20 we compare the accuracy and FCE of English LLMss
23

---

Published in Transactions on Machine Learning Research (12/2025)
and multilingual LLMs across both the extractive and generative architectures the SQuAD test set. Our results confirm that multilingual models are generally worse than monolingual English LLMs when tested on English QA for either architecture type, which has also been noted in Wu & Dredze (2020). However, we notice a different trend for calibration. Both accuracy and confidence calibration are comparable when evaluating the multilingual model and English monolingual model on the same English task (both generative and extractive).
Table 20: Comparison of the calibration performance between English LMs and multilingual LMs. The models are evaluated on the test set of SQuAD.
Exact Match (1)
ECE (1)
Exact Match (t)
ECE (1)
BERT
76.61
4.09
T5
80.67
5.42
mBERT
77.02
6.45
m'
78.90
5.32
RoBEITTa
82.26
6.46
BART
84.11
29.24
XLM-R
77.21
8.46
mBART
77.14
8.05
Then we want to compare the multilingual models with their monolingual counterparts for languages other than English. We select BERT-German and BERT-Arabic (Safaya al. 2020), and BERT-Chincso 13 and fine-tune them on the translated training dataset of SQuAD 1.1. The results are shown in Table 21 and Table ??. We observe that the monolingual BERT models always achieve better calibration than their multilingual counterparts, although the multilingual models sometimes more accurate than the corresponding monolingual models.
Table 21:
Comparison of the calibration performance between German (DE)/Arabic (AR) (ARi/Cinese (ZH) BERT and multilingual BERT (mBERT). The models are evaluated on the XQuAD test set for each language.
EM(1)
ECE(L)
EM(f)
ECE (4)
EM(1)
ECE (4)
BERT-DE
52.10
4.71
BERT-AR
48.66
11.30
BERT-ZH
37.22
6.88
mBERT
54.37
12.55
mBERT
40.92
18.50
mBERT
46.81
17.78
Table 22:
Comparison of the calibration performance between German (DE)/Arabic (AR)/Chinese (ZH)
BERT and multilingual BERT (mBERT). The models are evaluated on the MLQA test set for each language.
EM(1)
ECE(L)
EM(1))
ECE (1)
EM(1)
ECE (4)
BERT- DE
28.13
17.96
BERT-AR AR
24.95
12.80
BERT-Z ZH
30.75
6.22
mBERT
32.42
33.62
mBERT
28.82
30.29
mBERT
36.31
25.79
A.13.3 Size of Multilingual Models
Table 23: Calibration performance of m15 and XLM-R across different sizes on the MLQA test sets.
mT5
XLM-R
EM(1)
BCE(L)
EM(t)
ECE(L)
Small
28.88
33.80
N/A
N/A
Base
41.79
29.65
39.00
24.95
Large
45.73
27.40
43.94
26.86
13TThe details of the monalingual models of other languages in Appendix A.3
24


Reviews:
[{"review_id": "GeeO6Fzjwp", "reviewer": "Reviewer_hivZ", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper presents an empirical study of confidence calibration in multilingual QA models across extractive (mBERT, XLM-R), encoder–decoder (mT5, mBART), and decoder-only (LLaMA-2-7B) architectures. Models are fine-tuned on English (SQuAD 1.1) and evaluated zero-shot on XQuAD/MLQA as well as under distribution shift on TyDiQA. The authors measure calibration with Expected Calibration Error (ECE) and exact match (EM). They then test the effect of interventions on confidence calibration: temperature scaling (TS), label smoothing (LS), few-shot cross-lingual fine-tuning via machine-translated data, and in-context learning (ICL) for LLMs. The authors find that (a) calibration on English does not transfer well to other languages; (b) TS tuned on a small multilingual validation set consistently outperforms tuning on a larger English-only set; (c) adding machine-translated samples during fine-tuning improves cross-lingual calibration; and (d) ICL improves both EM and ECE for LLaMA-2 on TyDiQA.\n\n**Strengths:**\n- The paper provides a large empirical study across multiple model types, languages (both high and low-resource), and datasets.\n- The paper provides practical and actionable findings, such as temperature scaling on multilingual validation data being more effective than English-only validation, and the benefits of incorporating translated data during fine-tuning.\n\n**Weaknesses:**\n- The paper studies only a single decoder-only LLMs. It would be informative to replicate ICL results at larger scales and compare to other multilingual-capable LLMs, such as Aya-23.\n- All models are fine-tuned exclusively on English SQuAD before evaluation. While this setup cleanly isolates cross-lingual transfer, it may also introduce artefacts: improvements from translation-based augmentation could partly reflect recovery from an artificial bottleneck introduced during fine-tuning.", "claims_and_evidence_yn": "Yes", "claims_explanation": "The main claims are supported by sufficient empirical evidence. For example, the finding that calibration on English does not transfer to other languages is demonstrated in Table 1. The benefits of temperature scaling tuned on small multilingual validation sets over larger English-only ones is shown in Table 3. The benefits of adding translated samples is demonstrated in Table 4 and Table 5. ICL results for LLaMA-2 on TyDiQA show improved EM and ECE in Figure 4, though they are limited to a single model.", "audience_yn": "Yes", "audience_explanation": "The findings are relevant for researchers and practitioners working on multilingual NLP, calibration, and QA systems. The paper offers practical insights, such as the effectiveness of multilingual validation for temperature scaling, benefits of small-scale translated data, and ICL improvements.", "requested_changes": "**Critical for recommendation:** \n- Extend the ICL experiments beyond LLaMA-2-7B to other multilingual models such as Aya-23.\n\n**Strengthen the work:**\n- Analyze how calibration results depend on candidate set size K and decoding strategy (greedy, beam, sampling), especially for generative models.\n- Add statistical significance tests to support claims of improvement, particularly where ECE differences are small.", "broader_impact_concerns": "NA", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "sPzyBnC161", "reviewer": "Reviewer_VPdc", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper proposes a benchmark for calibration multilingual question answering. They want to show that the current LLM is not well-calibrated for other languages other than English.", "claims_and_evidence_yn": "Yes", "claims_explanation": "The main claim of this paper is  \"multilingual QA models are poorly calibrated for languages other than English\" and \" incorporating a small set of cheaply translated multilingual samples during fine-tuning/calibration effectively enhances the calibration performance\".\n\n\nFor the first claim:\n\n- The claim is evidenced by testing the confidence/accuracy over encoder-only language models like mBERT and XLM-R, and decoder only  models like Llama2.\n\n- The claim is evidenced by testing over in-distribution and Out-of-Distribution (OOD) Tasks\n\nFor the second claim:\n\n-The claim is evidenced by testing the data augmentation strategy, basically translate from english to other languages. \n\n- The authors also test other techniques, e.g., in-context learning, Temperature Scaling.", "audience_yn": "Yes", "audience_explanation": "Some individual interest in model calibration might be of interest to this paper.", "requested_changes": "I am not a expert of model calibration, and I am not a fan of this research topic as well. I find that this research topic does not make much sense to me honestly, and I can't see its research value. Please try to convince me by providing evidence to answer my following question:\n\n\n1. (major) Is making the model calibrate between confidence and accuracy really a different problem than only improving the accuracy of the model on specific task? Let say the model has high accuracy on one task, is that possible that it is ill-calibrated, i.e., its accuracy does not match its confidence? I want to raise the question because from your conclusion, if we finetune on Chinese data, then its calibration will improve. Isn't this improve of calibration is just because the model's accuracy on Chinese (which you finetune on) is improved? If that's the case, it seems that the reason leading to better calibration of these three strategies, i.e., Data Augmentation, in context learning. etc is obvious, because they all improve the accuracy of the corresponding language tasks (e.g., Chinese).  \n\n2. What is the point of calibrating a model? How does it generally improve the service performance of the model? \n\nThe paper is just fine with a lot of evaluation results. But I could not see any novelty and something that interest me in general. However, based on TMLR criterion, we should not reject a paper based on novelty but should focused on empirical evidence. Therefore, I will not stop your publication if you can answer my sort of \"mean\" questions.", "broader_impact_concerns": "NA", "additional_comments": "NA", "extra_scores": {}, "extra_text": {}}, {"review_id": "P2LRRAedkm", "reviewer": "Reviewer_UJ5S", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper performs a comprehensive empirical evaluation of the confidence calibration on Multilingual Large Language Models.  The paper found that the MLLMs need to be calibrated specially in zero-shot, temperature scaling is effective specially on the mixed-language validation setup, adding data in the fine-tuning step helps to improve the calibration, and that in-context learning improves the accuracy and confidence calibration on multilingual tasks.", "claims_and_evidence_yn": "Yes", "claims_explanation": "The paper presents a comprehensive set of results that support the claims of the results.  The descriptions and interpretations of the results could be better.", "audience_yn": "Yes", "audience_explanation": "Yes, the LLM community will be interested in this empirical findings.", "requested_changes": "Overall, the paper presents a comprehensive set of results.  Each section links the results and presents the setup.  However, the final conclusions and claims could be better presented and expanded upon.  Perhaps, highlighting them or reiterating the main findings at the end of each (sub)section could help the reader to better understand the paper and get back to particular results more easily.", "broader_impact_concerns": "I do not have any concerns.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Decision: Accept (poster)