MlingConf: A Comprehensive Study of Multilingual Confidence Estimation on Large Language Models
Authors: Boyang XUE, Hongru WANG, Rui Wang, Sheng Wang, Zezhong WANG, Yiming Du, Kam-Fai Wong


Reviewer paper summaries: The paper focuses the problem of confidence estimation in multilingual settings, introduces a multilingual QA dataset. It uses confidence estimation methods including likelihood-based, true probability, and self-verbalized confidence in multiple languages and attempts to enhance LLM performance through self-refinement. Experiments found that verbalized numerical confidence estimation has the best performance.
Summary of strengths: 1. The paper introduces Mling-TriviaQA, which covers ten languages and has been rigorously checked by linguistic experts.

2. The paper provides comprehensive set o experiments of different confidence estimation methods and their effectiveness across various languages.

3. The paper is the first to empirically investigate multilingual confidence estimation and showed that self-refinement with confidence estimation improves performance.
Summary of weaknesses: 1. The selection of LLMs in the experiments is limited. Smaller and open-source LLMs should be included in the evaluation. GPT-3.5 turbo date version should also be reported for better reproducibility.

2. How exactly is the self-refinement done? How are the confidence estimation incorporated? The methods section only describes confidence estimation and dataset curation, but lacks description of the self-refinement process completely.

3. Although a justification for language selection was provided in the appendix, I do not find this convincing. It is understandable that the language selection is based on the availability of linguistic experts, but the paper should avoid claims such as it covers a diverse range of language families, especially when the justification is based on “Indo-European languages” and “non-Indo-European languages”.

4. Analysis of results seems very limited. The analysis only compares some basic observations — performance varies across languages and verbalized numerical confidence is the best. But why? For example, the behavior in different languages are different, but what exactly is different? what are some examples? Are the models less confidence in general in some languages but not others? I find it interesting that performance of many languages (even the low resource ones) are higher than English, but unfortunately the authors did not provide explanation or insights into more fine-grained analysis of the results. 

Overall, the paper is clearly written but more insights and discussion are needed to have impact in the community. The authors ran many experiments and described the results, but the conclusion is not surprising.
Comments, suggestions and typos: Line 83: "exhibited noteworthy proficiencies" —> “exhibited noteworthy proficiency".

Line 115: "hindering the global proliferation…” - consider rephrasing for clarity.

Line 158: period should be after right parenthesis.

Space before conclusion section header
Overall assessment scores: 2.5