Package: tetun-tokenizer
Tetun TokenizerTetun tokenizer is a Python package used to tokenize an input text into tokens.  It offers several tokenization techniques:TetunStandardTokenizer: This tokenizer segments the input text into individual tokens based on word boundaries, punctuation, and special characters.TetunSentenceTokenizer: It splits sentences using ending delimiters such as periods (.), question marks (?), and exclamation marks (!). Titles represented by periods, such as Dr. and Ph.D., are preserved.TetunBlankLineTokenizer: This tokenizer segments the input text based on the presence of blank lines.TetunSimpleTokenizer: It extracts only strings and numbers from the input text while discarding punctuations and special characters.TetunWordTokenizer: This tokenizer extracts word units from the input text, excluding numbers, punctuation, and special characters.InstallationYou can install the Tetun tokenizer via pip:pip install tetun-tokenizerUsageTo utilize the Tetun tokenizer, simply import the desired tokenizer feature/class from thetokenizermodule within the Tetun tokenizer package. Here are some examples demonstrating its usage:UtilizingTetunStandardTokenizerto tokenize the input text.fromtetuntokenizer.tokenizerimportTetunStandardTokenizer# Instantiate TetunStandardTokenizertetun_tokenizer=TetunStandardTokenizer()# Input texttext="Ha'u mak ita-nia maluk di'ak. Ha'u iha $0.25 atu fó ba ita."# Tokenize the textoutput=tetun_tokenizer.tokenize(text)# Print the outputprint(output)Expected output:["Ha'u", 'mak', 'ita-nia', 'maluk', "di'ak", '.', "Ha'u", 'iha', '$', '0.25', 'atu', 'fó', 'ba', 'ita', '.']UsingTetunSentenceTokenizerto tokenize the input text.fromtetuntokenizer.tokenizerimportTetunSentenceTokenizertetun_tokenizer=TetunSentenceTokenizer()text="Ha'u ema-ida ne'ebé baibain de'it. Tebes ga? Ita-nia maluk Dr. ka Ph.D sira hosi U.S.A mós dehan!"output=tetun_tokenizer.tokenize(text)print(output)Expected output:["Ha'u ema-ida ne'ebé baibain de'it.", 'Tebes ga?', 'Ita-nia maluk Dr. ka Ph.D sira hosi U.S.A mós dehan!']UtilizingTetunBlankLineTokenizerto tokenize the input text.fromtetuntokenizer.tokenizerimportTetunBlankLineTokenizertetun_tokenizer=TetunBlankLineTokenizer()text="""Ha'u mak ita-nia maluk di'ak.Ha'u iha $0.25 atu fó ba ita."""output=tetun_tokenizer.tokenize(text)print(output)Expected output:["\n            Ha'u mak ita-nia maluk di'ak.\n            Ha'u iha $0.25 atu fó ba ita.\n            "]UsingTetunSimpleTokenizerto tokenize a given text.fromtetuntokenizer.tokenizerimportTetunSimpleTokenizertetun_tokenizer=TetunSimpleTokenizer()text="Ha'u mak ita-nia maluk di'ak. Ha'u iha $0.25 atu fó ba ita."output=tetun_tokenizer.tokenize(text)print(output)Expected output:["Ha'u", 'mak', 'ita-nia', 'maluk', "di'ak", "Ha'u", 'iha', '0.25', 'atu', 'fó', 'ba', 'ita']UsingTetunWordTokenizerto tokenize the input text.fromtetuntokenizer.tokenizerimportTetunWordTokenizertetun_tokenizer=TetunWordTokenizer()text="Ha'u mak ita-nia maluk di'ak. Ha'u iha $0.25 atu fó ba ita."output=tetun_tokenizer.tokenize(text)print(output)Expected output:["Ha'u", 'mak', 'ita-nia', 'maluk', "di'ak", "Ha'u", 'iha', 'atu', 'fó', 'ba', 'ita']You can also use the tokenizer to tokenize a text from a file. Below is an example:# Assume that we use Path instead of a string for the file pathfrompathlibimportPathfromtetuntokenizer.tokenizerimportTetunSimpleTokenizerfile_path=Path("myfile/example.txt")try:withfile_path.open('r',encoding='utf-8')asf:contents=[line.strip()forlineinf]exceptFileNotFoundError:print(f"File not found at:{file_path}")# You can also lowercase the contents before tokenizing them.lowercase_contents=contents.lower()tetun_tokenizer=TetunSimpleTokenizer()output='\n'.join(tetun_tokenizer.tokenize(str(lowercase_contents)))print(output)This is the example of the output:ha'u
orgullu
dezenvolve
ha'u-nia
lian
tetun 
...CitationIf you use this repository or any of its contents for your research, academic work, or publication, we kindly request that you cite it as follows:@misc{jesus-nunes-2024,
  author       = {Gabriel de Jesus and Sérgio Nunes},
  title        = {Data Collection Pipeline for Low-Resource Languages: A Case Study on Constructing a Tetun Text Corpus},
  year         = {2024},
  note         = {Accepted at LREC-COOLING, 2024},
}AcknowledgementThis work is financed by National Funds through the Portuguese funding agency, FCT - Fundação para a Ciência e a Tecnologia under the PhD scholarship grant number SFRH/BD/151437/2021.LicenseAdditional InformationFor the source code, visit theGitHub repositoryfor this project.