# LLMs Will Be the Future of NLP for Endangered Languages

Mika Hämäläinen

<figure><img src="image_1.png" /></figure>

## To cite this version:

Mika Hämäläinen. LLMs Will Be the Future of NLP for Endangered Languages. Lightning Proceedings of the 9th International Workshop on Computational Linguistics for Uralic Languages, Nov 2024, Helsinki, Finland. pp.14-18. hal-04800902

HAL Id: hal-04800902

https://hal.science/hal-04800902v1

Submitted on 24 Nov 2024

HAL is a multi-disciplinary open access archive for the deposit and dissemination of scientific research documents, whether they are published or not. The documents may come from teaching and research institutions in France or abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est destinée au dépôt et à la diffusion de documents scientifiques de niveau recherche, publiés ou non, émanant des établissements d'enseignement et de recherche français ou étrangers, des laboratoires publics ou privés.

<figure><img src="image_2.png" /></figure>
# LLMs Will Be the Future of NLP for Endangered Languages

Mika Hämäläinen
Metropolia University of Applied Sciences, Finland

## Introduction

The pace at which large language models (LLMs) have been developing in the past few years has been extremely fast. LLMs are getting more and more capable of doing more and more complex tasks. Currently, even the largest commercial LLMs like ChatGPT<fnref n="1" /> struggle with endangered languages (see Choudhury, 2023), which is something that has made some people in the endangered language NLP community believe that LLMs cannot or should not be used in the context of endangered languages. This, I feel, is quite short-sighted, and I firmly believe otherwise.

In the near future, LLMs will be the only meaningful way of doing NLP for endangered languages. Rule-based NLP solutions can only take us so far, and there is a good reason why they have been long forgotten about in the context of larger languages. The reality around us is complex – capturing it would require an almost infinite amount of rules. Not to mention that rules need updating on a continuous basis.

<figure><img src="image_3.png" /></figure>

<footnote n="1">¹ https://chatgpt.com/</footnote>
# Overcoming the issue of data

What has really been the biggest challenge for machine learning (ML) for endangered languages in the past has been the issue with data. There is never enough of it and the little there is, tends to have a lot of variety (see Hämäläinen, 2021). Both are big issues for ML techniques, but this time it will be different.

LLMs showcase emergent capabilities (see Anil et al., 2023) – their intelligence is not limited to supervised training with the exact solutions – rather they have generalization capabilities far beyond anything we have seen with any previous technologies (see Wang et al., 2024). What this means is that the hardest part has been achieved, we have artificial intelligence (AI) that is fluent in several languages and has a vast knowledge of the real world.

# The inevitable future of LLMs

Because intelligence has been reached already, it does not make any sense to reach it over and over again – each time from scratch by training a new LLM to reach the current state of the art but just for a different set of languages. It is an inevitable future direction for LLMs to be able to learn directly from user input by adjusting their own weights accordingly. This will be done without the need for an extensive dataset to learn from. Soon we will be able to merely tell an LLM about new things, and it will pick them up as soon as it hears them.
Surely, we have methods like LoRA (Hu et al., 2021) that make it possible to fine-tune LLMs with more data, but this won't quite solve the problem for endangered languages. We can also augment the knowledge of an LLM by retrieval augmented generation (RAG) (see Gao et al., 2023), which is useful for making LLMs' replies better informed, but it is not enough to teach an LLM a new skill like a completely new language. However, we seem to be able to "teach" an LLM to speak an endangered language if we tell it how to do so (see Gin et al., 2024; Zhang et al., 2024). Thus far, this method does not have any long-lasting impact on the LLM because everything we tell it is just in the context window of tokens fed to the model.

I do not believe that we will head to this direction with LLMs because of endangered languages. We will simply have these kinds of LLMs in the future because of the business value. LLMs bring a lot of money currently to many companies because they are what AI means to most people. One of the challenges client companies face currently is the adaptability of an LLM for their needs. Many companies aren't simply ready to think about their data in such a way that they could train a LoRA model or have a functional RAG model. Having a possibility to teach an LLM like one would teach a new hire is a more natural way of embracing AI than getting company data in order.

In the very near future, we will be able to teach an LLM an endangered language like we would teach it to a student. This is an inevitable course of AI development, and it will happen whether the endangered language NLP research community researches the use of LLMs or not. As a community, we should have our eyes in the future, not in the past. Nobody can hide behind the idea of a lack of data for endangered languages anymore to justify the need for rule-based methods. LLMs will be compatible with endangered languages sooner or later, and when that time comes, we had better be ready.
References

Anil, R., Dai, A. M., Firat, O., Johnson, M., Lepikhin, D., Passos, A., ... & Wu, Y. (2023). Palm 2 technical report. arXiv preprint arXiv:2305.10403.

Choudhury, M. (2023). Generative AI has a language problem. Nature Human Behaviour, 7(11), 1802-1803.

Gao, Y., Xiong, Y., Gao, X., Jia, K., Pan, J., Bi, Y., ... & Wang, H. (2023). Retrieval-augmented generation for large language models: A survey. arXiv preprint arXiv:2312.10997.

Ginn, M., Hulden, M., & Palmer, A. (2024). Can we teach language models to gloss endangered languages?. arXiv preprint arXiv:2406.18895.

Hu, E. J., Shen, Y., Wallis, P., Allen-Zhu, Z., Li, Y., Wang, S., ... & Chen, W. (2021). Lora: Low-rank adaptation of large language models. arXiv preprint arXiv:2106.09685.

Hämäläinen, M. (2021). Endangered Languages are not Low-Resourced!. In Multilingual Facilitation. University of Helsinki.

Wang, X., Antoniades, A., Elazar, Y., Amayuelas, A., Albalak, A., Zhang, K., & Wang, W. Y. (2024). Generalization vs Memorization: Tracing Language Models' Capabilities Back to Pretraining Data. arXiv preprint arXiv:2407.14985.

Zhang, K., Choi, Y., Song, Z., He, T., Wang, W. Y., & Li, L. (2024). Hire a linguist!: learning endangered languages in LLMs with in-context linguistic descriptions. In Findings of
the Association for Computational Linguistics ACL 2024 (pp.
15654-15669).