Package: tetun-lid
Tetun LIDTetun Language Identification (Tetun LID) model is a machine learning model that automatically identifies the language of a given text. It was specifically designed to recognize four languages commonly spoken in Timor-Leste: Tetun, Portuguese, English, and Indonesian.InstallationWith pip:pip install tetun-lidDependenciesThe Tetun LID model depends on the following packages:joblibscikit-learnInstall the dependencies packages with pip:pip install joblib
pip install scikit-learnUsageTo use the Tetun LID model, from thetetunlidpackage, import thelidmodule, and then call the respective functions. The examples of its usage are as follows:To predict the language of an input text, use thepredict_language()function.fromtetunlidimportlidinput_text="Sé mak toba iha ne'ebá?"output=lid.predict_language(input_text)print(output)This will be the output:TetunTo print the details of the Probability of being predicted to Tetun use thepredict_detail()function.fromtetunlidimportlidinput_list_of_str=["Sé mak toba iha ne'ebá?"]output_detail=lid.predict_detail(input_list_of_str)print('\n'.join(output_detail))This will be the outpu:Input text: "Sé mak toba iha ne'ebá?"
Probability:
        English: 0.0010
        Indonesian: 0.0014
        Portuguese: 0.0082
        Tetun: 0.9967
Thus, the input text is "Tetun" with a confidence level of 99.67%.Note: The output ofpredict_detail()is a list of strings and therefore to print the its result in the console, useforloop orjoin()as in the previous example.We can feed a mixed corpus containing multiple languages into the LID model as the input list. Observe the following example:fromtetunlidimportlidmultiple_langs=["Ha'u ema baibain","I am not available","Apa kabar kawan?","Estou a estudar"]output=[(ml,lid.predict_language(ml))formlinmultiple_langs]print(output)This will be the outpu:[("Ha'u ema baibain", 'Tetun'), ('I am not available', 'English'), ('Apa kabar kawan?', 'Indonesian'), ('Estou a estudar', 'Portuguese')]You can use print the output in the console as follows:fromtetunlidimportlidinput_texts=["Ha'u ema baibain","I am not available","Apa kabar kawan?","Estou a estudar"]forinput_textininput_texts:lang=lid.predict_language(input_text)print(f"{input_text}({lang})")This will be the outpu:Ha'u ema baibain (Tetun)
I am not available (English)
Apa kabar kawan? (Indonesian)
Estou a estudar (Portuguese)To print the details of each input, use the same function as previously explained. Here is the example:fromtetunlidimportlidinput_texts=["Ha'u ema baibain","I am not available","Apa kabar kawan?","Estou a estudar"]output_multiple_detail=lid.predict_detail(input_texts)print('\n'.join(output_multiple_detail))This will be the outpu:Input text: "Ha'u ema baibain"
Probability:
        English: 0.0032
        Indonesian: 0.0032
        Portuguese: 0.0028
        Tetun: 0.9907
Thus, the input text is "Tetun" with a confidence level of 99.07%.

Input text: "I am not available"
Probability:
        English: 0.9999
        Indonesian: 0.00001
        Portuguese: 0.00001
        Tetun: 0.00001
Thus, the input text is "English" with a confidence level of 99.99%.

Input text: "Apa kabar kawan?"
Probability:
        English: 0.0011
        Indonesian: 0.9961
        Portuguese: 0.0015
        Tetun: 0.0184
Thus, the input text is "Indonesian" with a confidence level of 99.61%.

Input text: "Estou a estudar"
Probability:
        English: 0.0003
        Indonesian: 0.002
        Portuguese: 0.9810
        Tetun: 0.0184
Thus, the input text is "Portuguese" with a confidence level of 98.10%.We can filter only Tetun text from a mixed corpus containing multiple languages using thepredict_language()function.fromtetunlidimportlidinput_texts=["Ha'u ema baibain","I am not available","Apa kabar kawan?","Estou a estudar"]output=[textfortextininput_textsiflid.predict_language(text)=='Tetun']print(output)This will be the output:["Ha'u ema baibain"]We can also use Tetun LID to predict a text from a file containing various languages or texts extracted from the web. Here is an example:frompathlibimportPathfromtetunlidimportlidfile_path=Path("myfile/example.txt")try:withfile_path.open('r',encoding='utf-8')asf:contents=[line.strip()forlineinf]exceptFileNotFoundError:print(f"File not found at:{file_path}")output=[(content,lid.predict_language(content))forcontentincontents]print(output)Additional notesAll the dependencies need to be installed accordingly before using the model.If you encountered anAttributeError: 'list' object has no attribute 'predict_proba', you might have some issues while installing the package. Please send me an email, and I will guide you on how to handle the error.Please make sure that you use the latest version of Tetun LID by running this command in your console:pip install --upgrade tetun-lid.To get the source code, visit theGitHub repositoryfor this project.CitationIf you use this repository or any of its contents for your research, academic work, or publication, we kindly request that you cite it as follows:@misc{jesus-nunes-2024,
  author       = {Gabriel de Jesus and Sérgio Nunes},
  title        = {Data Collection Pipeline for Low-Resource Languages: A Case Study on Constructing a Tetun Text Corpus},
  year         = {2024},
  note         = {Accepted at LREC-COOLING, 2024},
}AcknowledgementThis work is financed by National Funds through the Portuguese funding agency, FCT - Fundação para a Ciência e a Tecnologia under the PhD scholarship grant number SFRH/BD/151437/2021.LicenseContact InformationIf you have any questions or feedback, please feel free to contact mestregabrieldejesus[at]gmail.com