Package: thai-segmenter
This package provides utilities for Thai sentence segmentation, word tokenization and POS tagging.
Because of how sentence segmentation is performed, prior tokenization and POS tagging is required and therefore also provided with this package.Besides functions for doing sentence segmentation, tokenization, tokenization with POS tagging for single sentence strings,
there are also functions for working with large amounts of data in a streaming fashion.
They are also accessible with a commandline scriptthai-segmenterthat accepts file or standard in/output.
Options allow working with meta-headers or tabulator separated data files.The main functionality for sentence segmentation was extracted, reformatted and slightly rewritten from another project,Question Generation Thai.LongLexTois used as state-of-the-art word/lexeme tokenizer. An implementation was packaged in the above project but there are also (original?) versionsgithubandhomepage. To better use it for bulk processing in Python, it has been rewritten from Java to pure Python.For POS tagging a Viterbi-Model with the annotated Orchid-Corpus is used,paper.Free software: MIT licenseInstallationpip install thai-segmenterDocumentationTo use the project:sentence="""foo bar 1234"""# [A] Sentence Segmentationfromthai_segmenter.tasksimportsentence_segment# or even easier:fromthai_segmenterimportsentence_segmentsentences=sentence_segment(sentence)forsentenceinsentences:print(str(sentence))# [B] Lexeme Tokenizationfromthai_segmenterimporttokenizetokens=tokenize(sentence)fortokenintokens:print(token,end=" ",flush=True)# [C] POS Taggingfromthai_segmenterimporttokenize_and_postagsentence_info=tokenize_and_postag(sentence)fortoken,posinsentence_info.pos:print("{}|{}".format(token,pos),end=" ",flush=True)See more possibilities intasks.pyorcli.py.Streaming larger sequences can be achieved like this:# Streamingsentences=["sent1\n","sent2\n","sent3\n"]# or any iterable (like File)fromthai_segmenterimportline_sentence_segmentersentences_segmented=line_sentence_segmenter(sentences)Commandline toolThis project also provides a nifty commandline toolthai-segmenterthat does most of the work for you:usage:thai-segmenter[-h]{clean,sentseg,tokenize,tokpos}...ThaiSegmentationutilities.optionalarguments:-h,--helpshowthishelpmessageandexitTasks:{clean,sentseg,tokenize,tokpos}cleanCleaninputfromnon-thaiandblanklines.sentsegSentencesegmentizeinputlines.tokenizeTokenizeinputlines.tokposTokenizeandPOS-taginputlines.You can run sentence segmentation like this:thai-segmenter sentseg -i input.txt -o output.txtor even pipe data:cat input.txt | thai-segmenter sentseg > output.txtUse-h/--helpto get more information about possible control flow options.You can run it somewhat interactively with:thai-segmenter tokpos --statsand standard input and output are used. Lines terminated withEnterare immediatly processed and printed. Stop work with key combinationCtrl+Dand the--statsparameter will helpfully output some statistics.WebAppThe project also provides a demo WebApp (usingFlaskandgevent) that can be installed with:pip install -e .[webapp]and then simply run (in the foreground):thai-segmenter-webappConsider running it in ascreensession.# create the screen detached and then attachscreen-dmSthai-senseg-webappscreen-rthai-senseg-webapp# in the screen:thai-segmenter-webapp# and detach with keys [Ctrl]+[D]Please note that it only is a demo webapp to test and visualize how the sentence segmentor works.DevelopmentTo install the package for development:git clone https://github.com/Querela/thai-segmenter.git
cd thai-segmenter/
pip install -e .[dev]After changing the source, run auto code formatting with:isort <file>.py
black <file>.pyAnd check it afterwards with:flake8 <file>.pyThesetup.pyalso contains theflake8subcommand as well as an extendedcleancommand.TestsTo run the all tests run:toxYou can also optionally runpytestalone:pytestOr with:python setup.py testNote, to combine the coverage data from all the tox environments run:Windowsset PYTEST_ADDOPTS=--cov-append
toxOtherPYTEST_ADDOPTS=--cov-append toxChangelog0.4.2 (2023-08-23)Fix signature oftasks.tokenize_and_postagfunctionUpdatetox.inito include newer python version, as well as older parameters and flagsReformat und Lint0.4.1 (2019-04-08)Fix tokenization / tokenization + POS tagging: return words instead of subwordsAdd--escape-specialand--subwordsparameter to CLI script for tokenization.
Allows tokenization to further tokenize unknown words (e. g. names)
as well as escape special characters with angle bracket entities.0.4.0 (2019-04-08)Add demo webapp with sentence segmentation.
(NOTE: Running both the webapp and (batch) sentence segmentation at the same time from the same installation is not recommeded. It can have unexpected side-effects.)Some reformat ofREADME.rst0.3.3 (2019-04-07)Fix duplicate names (class/method forsentence_segment), rename class tosentence_segmenter(.py).0.3.2 (2019-04-07)Addtwineto extras dependencies.Publish module onPyPI. (Onlysdist,bdist_wheelcan’t be built currently.)Fix some TravisCI warnings.0.3.1 (2019-04-07)Add tasks to__init__.pyfor easier access.0.3.0 (2019-04-06)Refactor tasks intotasks.pyto enable better import in case of embedding thai-segmenter into other projects.Have it almost release ready. :-)Add some more parameters to functions (optional header detection function)Flesh outREADME.rstwith examples and descriptions.Add Changelog items.0.2.1 / 0.2.2 (2019-04-05)Many changes,bumpversionneeds to run where.bumpversion.cfgis located else it silently fails …Strip Typehints and add support for Python3.5 again.Add CLI tasks for cleaning, sentseg, tokenize, pos-tagging.Add various params, e. g. for selecting columns, skipping headers.Fix many bugs for TravisCI (isort, flake8)Use iterators / streaming approach for file input/output.0.2.0 (2019-04-05)Remove support of Python 2.7 and lower equal to Python 3.5 because of Typehints.Added CLI skeleton.Add really goodsetup.py. (withblack,flake8)0.1.0 (2019-04-05)First release version as package.