Error Analysis of  Multilingual Language Models in Machine Translation for Low-resource Languages, in the case of Amharic to English Bi-directional Machine Translation
Authors: Hizkiel Mitiku Alemayehu, Hamada M Zahera, Axel-Cyrille Ngonga Ngomo


Reviewer paper summaries: This paper evaluates the performance of state-of-the-art multilingual large language models (mLLMs) for translating between Amharic and English. The study uses models such as NLLB-200 (NLLB3.3B, NLLB1.3B, NLLB600M) and M2M (M2M1.2B, M2M418) and applies both automatic and human evaluation metrics, including BLEU, METEOR, chrF, TER, and Multidimensional Quality Metrics (MQM). The results indicate that the NLLB3.3B model outperformed other mLLMs in all evaluation techniques. Common error types identified include mistranslation, omission, untranslated segments, and addition errors, with mistranslation being the most frequent.
Summary of strengths: The paper presents a thorough and novel analysis of multilingual large language models' performance in translating between a low-resource language (Amharic) and English. By evaluating multiple state-of-the-art models using a combination of automatic and human evaluation metrics, the study provides a comprehensive assessment of these models' strengths and weaknesses. The use of the MQM framework for human evaluation adds depth to the analysis, highlighting specific error types that are prevalent in the translations. The paper's focus on low-resource languages addresses an important gap in the current research, making it a valuable contribution to the field of machine translation. Additionally, the annotated dataset will be openly released, providing a useful resource for further research and development in this area.
Summary of weaknesses: Despite its contributions, the study has several limitations. The findings are primarily based on the specific set of models and datasets chosen, which may limit the generalizability of the results to other low-resource languages or different types of machine translation tasks. The relatively small sample size and the focus on Amharic-English translation may also affect the robustness and applicability of the conclusions. Moreover, the paper could benefit from a more detailed discussion of practical applications and strategies for improving the performance of mLLMs in real-world scenarios. The complexity of the evaluation setup, particularly the human evaluation component, may pose reproducibility challenges. Providing more detailed guidelines and sharing the code for replication would enhance the study's accessibility and usability by other researchers. Addressing these weaknesses could improve the paper's overall impact and make it a stronger candidate for publication.
Comments, suggestions and typos: N/A
Overall assessment scores: 3.5