Package: botok
Botok – Python Tibetan TokenizerDescription•Install•Example•Commented Example•Docs•Owners•Acknowledgements•Maintainance•LicenseDescriptionBotok tokenizes Tibetan text into words with optional attributes such as lemma, POS, clean form.InstallRequires to have Python3 installed.pip3 install botokExamplefrom botok import WordTokenizer
from botok.config import Config
from pathlib import Path

def get_tokens(wt, text):
    tokens = wt.tokenize(text, split_affixes=False)
    return tokens

if __name__ == "__main__":
    config = Config(dialect_name="general", base_path= Path.home())
    wt = WordTokenizer(config=config)
    text = "བཀྲ་ཤིས་བདེ་ལེགས་ཞུས་རྒྱུ་ཡིན་ སེམས་པ་སྐྱིད་པོ་འདུག།"
    tokens = get_tokens(wt, text)
    for token in tokens:
        print(token)https://user-images.githubusercontent.com/24893704/148767959-31cc0a69-4c83-4841-8a1d-028d376e4677.mp4Commented Example>>>frombotokimportText>>># input is a multi-line input string>>>in_str="""ལེ གས། བཀྲ་ཤིས་མཐའི་ ༆ ཤི་བཀྲ་ཤིས་  tr... བདེ་་ལེ གས། བཀྲ་ཤིས་བདེ་ལེགས་༡༢༣ཀཀ།... མཐའི་རྒྱ་མཚོར་གནས་པའི་ཉས་ཆུ་འཐུང་།། །།མཁའ།"""### STEP1: instanciating Text>>># A. on a string>>>t=Text(in_str)>>># B. on a file...# note all following operations can be applied to files in this way.>>>frompathlibimportPath>>>in_file=Path.cwd()/'test.txt'>>># file content:>>>in_file.read_text()'བཀྲ་ཤིས་བདེ་ལེགས།།\n'>>>t=Text(in_file)>>>t.tokenize_chunks_plaintext>>># checking an output file has been written:...# BOM is added by default so that notepad in Windows doesn't scramble the line breaks>>>out_file=Path.cwd()/'test_pybo.txt'>>>out_file.read_text()'\ufeffབཀྲ་ ཤིས་ བདེ་ ལེགས །།'### STEP2: properties will perform actions on the input string:### note: original spaces are replaced by underscores.>>># OUTPUT1: chunks are meaningful groups of chars from the input string....# see how punctuations, numerals, non-bo and syllables are all neatly grouped.>>>t.tokenize_chunks_plaintext'ལེ_གས །_ བཀྲ་ ཤིས་ མཐའི་ _༆_ ཤི་ བཀྲ་ ཤིས་__ tr_\nབདེ་་ ལེ_གས །_ བཀྲ་ ཤིས་ བདེ་ ལེགས་ ༡༢༣ ཀཀ །_\nམཐའི་ རྒྱ་ མཚོར་ གནས་ པའི་ ཉས་ ཆུ་ འཐུང་ །།_།། མཁའ །'>>># OUTPUT2: could as well be acheived by in_str.split(' ')>>>t.tokenize_on_spaces'ལེ གས། བཀྲ་ཤིས་མཐའི་ ༆ ཤི་བཀྲ་ཤིས་ tr བདེ་་ལེ གས། བཀྲ་ཤིས་བདེ་ལེགས་༡༢༣ཀཀ། མཐའི་རྒྱ་མཚོར་གནས་པའི་ཉས་ཆུ་འཐུང་།། །།མཁའ།'>>># OUTPUT3: segments in words....# see how བདེ་་ལེ_གས was still recognized as a single word, even with the space and the double tsek....# the affixed particles are separated from the hosting word: མཐ འི་ རྒྱ་མཚོ ར་ གནས་པ འི་ ཉ ས་>>>t.tokenize_words_raw_textLoadingTrie...(2s.)'ལེ_གས །_ བཀྲ་ཤིས་ མཐ འི་ _༆_ ཤི་ བཀྲ་ཤིས་_ tr_ བདེ་་ལེ_གས །_ བཀྲ་ཤིས་ བདེ་ལེགས་ ༡༢༣ ཀཀ །_ མཐ འི་ རྒྱ་མཚོ ར་ གནས་པ འི་ ཉ ས་ ཆུ་ འཐུང་ །།_།། མཁའ །'>>>t.tokenize_words_raw_lines'ལེ_གས །_ བཀྲ་ཤིས་ མཐ འི་ _༆_ ཤི་ བཀྲ་ཤིས་__ tr_\nབདེ་་ལེ_གས །_ བཀྲ་ཤིས་ བདེ་ལེགས་ ༡༢༣ ཀཀ །_\nམཐ འི་ རྒྱ་མཚོ ར་ གནས་པ འི་ ཉ ས་ ཆུ་ འཐུང་ །།_།། མཁའ །'>>># OUTPUT4: segments in words, then calculates the number of occurences of each word found...# by default, it counts in_str's substrings in the output, which is why we have བདེ་་ལེ གས	1, བདེ་ལེགས་	1...# this behaviour can easily be modified to take into account the words that pybo recognized instead (see advanced usage)>>>print(t.list_word_types)འི་3།2བཀྲ་ཤིས་2མཐ2ལེགས1༆1ཤི་1བཀྲ་ཤིས་1tr\n1བདེ་་ལེགས1བདེ་ལེགས་1༡༢༣1ཀཀ1།\n1རྒྱ་མཚོ1ར་1གནས་པ1ཉ1ས་1ཆུ་1འཐུང་1།།།།1མཁའ1།1Custom dialect pack:In order to use custom dialect pack:You need to prepare your dialect pack in same folder structure likegeneral dialect packThen you need to instaintiate a config object where you will pass dialect name and pathYou can instaintiate your tokenizer object using that config objectYour tokenizer will be using your custom dialect pack and it will be using trie pickled file in future to build the custom trie.DocsNo documentations.Owners@drupchen@eroux@ngawangtrinley@10zinten@kaldan007Acknowledgementsbotokis an open source library for Tibetan NLP.We are always open to cooperation in introducing new features, tool integrations and testing solutions.Many thanks to the companies and organizations who have supported botok's development, especially:Khyentse Foundationfor contributing USD22,000 to kickstart the projectTheBarom/Esukhia canon projectfor sponsoring training data curationBDRCfor contributing 2 staff for 6 months for data curationMaintainanceBuild the source dist:rm -rf dist/
python3 setup.py clean sdistand upload on twine (version >=1.11.0) with:twine upload dist/*LicenseThe Python code is Copyright (C) 2019 Esukhia, provided underApache 2.contributors:DrupchenÉlie RouxNgawang TrinleyMikko KotilaThubten RinzinTenzinJoyce Mackzenzie for reworking the logo