Package: punctuator-lvl9-inga
READMEPunctuation PredictionA python package that punctuates Icelandic text. The input data is unpunctuated text and punctuated text is returned. The user can choose between two punctuation models, a bidirectional RNN (Punctuator 2) in Tensorflow 2, and a pretrained ELECTRA Transformer, fine-tuned for punctuation prediction,based on a Hugging Face NER recipe. The pretrained ELECTRA model was trained by Jón Friðrik Daðason on data from the Icelandic Gigaword corpus. Both punctuation models are trained/fine-tuned using Gigaword corpus data.Table of ContentsInstallationRunningExamplePython moduleThe punctuate functionLicenseAuthors/CreditAcknowledgementsInstallationTo install, first create a conda environment:conda create --name {venv}Then activate it:conda activate {venv}Install the requirement(s):conda install tensorflow==2.1.0pip install tensorflow==2.1.0could also work, if you run into problems.The transformer models are created with PyTorch, which is needed when reading the models:conda install pytorchTo use the ELECTRA model one needs access to Hugging Face functions, installed with:pip install transformersThen finally, run:pip install punctuator-islRunningThe program can be run either from a command line or from inside a python script.To run it on a command line:$ punctuate input.txt output.txtThe default model is thebiRNNmodel, you can also specify another model, e.g.:$ punctuate input.txt output.txt --electraThe input usesstdinand the outputstdout. Both files are encoded in UTF-8.Empty lines in the input are treated as sentence boundaries.Which of the two models to be used can be specified on the command line. The default isbiRNN.ModelDescriptionbiRNNThe Punctuator 2 model in Tensorflow.ELECTRAThe ELECTRA Transformer (HuggingFace)For a short help message of how to use the package, typepunctuate -horpunctuate --help.The input text should be like directly from automatic speech recognition, without capitalizations or punctuations.The first time the program is run the punctuation models are downloaded intopunctuation_models, in the user
home directory for Linux and Mac users, and in APPDATA for Windows users. The user can put a new model path inpath_config.jsoninside the punctuator directory is another location is desired.ExampleIn this case, the default model is used. An input string is specified and the punctuate function returns a punctuated string, words that appear after an end-of-sentence punctuation mark are capitalized.$ echo "næsti fundur er fyrirhugaður í næstu viku að sögn kristínar jónsdóttur hópstjóra náttúruvárvöktunar hjá veðurstofu íslands verður áfram fylgst grannt með jarðhræringum á svæðinu" | punctuate
$ Næsti fundur er fyrirhugaður í næstu viku. Að sögn kristínar jónsdóttur, hópstjóra náttúruvöktunar hjá veðurstofu íslands, verður áfram fylgst grannt með jarðhræringum á svæðinu.Python moduleThe punctuate functionfrom punctuator-is import punctuate

# A string to be punctuated
s = "næsti fundur er fyrirhugaður í næstu viku að sögn kristínar jónsdóttur hópstjóra náttúruvárvöktunar hjá veðurstofu íslands verður áfram fylgst grannt með jarðhræringum á svæðinu"

punctuated = punctuate(s, model='biRNN')

print(punctuated)The program should output:Næsti fundur er fyrirhugaður í næstu viku. Að sögn kristínar jónsdóttur, hópstjóra náttúruvöktunar hjá veðurstofu íslands, verður áfram fylgst grannt með jarðhræringum á svæðinu.LicenseThis code is licensed under the MIT license.Authors/CreditReykjavik UniversityMain authors: Helga Svala Sigurðardóttir -helgas@ru.is, Inga Rún Helgadóttir -ingarun@ru.isAcknowledgementsThis project was funded by the Language Technology Programme for Icelandic 2019-2023. The programme, which is managed and coordinated byAlmannarómur, is funded by the Icelandic Ministry of Education, Science and Culture.