Package: sacremoses
SacremosesLicenseMIT License.Installpip install -U sacremosesNOTE: Sacremoses only supports Python 3 now (sacremoses>=0.0.41). If you're using Python 2, the last possible version issacremoses==0.0.40.Usage (Python)Tokenizer and Detokenizer>>>fromsacremosesimportMosesTokenizer,MosesDetokenizer>>>mt=MosesTokenizer(lang='en')>>>text='This, is a sentence with weird\xbbsymbols\u2026appearing everywhere\xbf'>>>expected_tokenized='This , is a sentence with weird\xbbsymbols\u2026appearing everywhere\xbf'>>>tokenized_text=mt.tokenize(text,return_str=True)>>>tokenized_text==expected_tokenizedTrue>>>mt,md=MosesTokenizer(lang='en'),MosesDetokenizer(lang='en')>>>sent="This ain't funny. It's actually hillarious, yet double Ls. | [] < > [ ] & You're gonna shake it off? Don't?">>>expected_tokens=['This','ain','&apos;t','funny','.','It','&apos;s','actually','hillarious',',','yet','double','Ls','.','&#124;','&#91;','&#93;','&lt;','&gt;','&#91;','&#93;','&amp;','You','&apos;re','gonna','shake','it','off','?','Don','&apos;t','?']>>>expected_detokens="This ain't funny. It's actually hillarious, yet double Ls. | [] < > [] & You're gonna shake it off? Don't?">>>mt.tokenize(sent)==expected_tokensTrue>>>md.detokenize(tokens)==expected_detokensTrueTruecaser>>>fromsacremosesimportMosesTruecaser,MosesTokenizer# Train a new truecaser from a 'big.txt' file.>>>mtr=MosesTruecaser()>>>mtok=MosesTokenizer(lang='en')# Save the truecase model to 'big.truecasemodel' using `save_to`>>tokenized_docs=[mtok.tokenize(line)forlineinopen('big.txt')]>>>mtr.train(tokenized_docs,save_to='big.truecasemodel')# Save the truecase model to 'big.truecasemodel' after training# (just in case you forgot to use `save_to`)>>>mtr=MosesTruecaser()>>>mtr.train('big.txt')>>>mtr.save_model('big.truecasemodel')# Truecase a string after training a model.>>>mtr=MosesTruecaser()>>>mtr.train('big.txt')>>>mtr.truecase("THE ADVENTURES OF SHERLOCK HOLMES")['the','adventures','of','Sherlock','Holmes']# Loads a model and truecase a string using trained model.>>>mtr=MosesTruecaser('big.truecasemodel')>>>mtr.truecase("THE ADVENTURES OF SHERLOCK HOLMES")['the','adventures','of','Sherlock','Holmes']>>>mtr.truecase("THE ADVENTURES OF SHERLOCK HOLMES",use_known=True)['the','ADVENTURES','OF','SHERLOCK','HOLMES']>>>mtr.truecase("THE ADVENTURES OF SHERLOCK HOLMES",return_str=True)'the adventures of Sherlock Holmes'Normalizer>>>fromsacremosesimportMosesPunctNormalizer>>>mpn=MosesPunctNormalizer()>>>mpn.normalize('THIS EBOOK IS OTHERWISE PROVIDED TO YOU "AS-IS."')'THIS EBOOK IS OTHERWISE PROVIDED TO YOU "AS-IS."'Usage (CLI)Since version0.0.42, the pipeline feature for CLI is introduced, thus there
are global options that should be set first before calling the commands:languageprocessesencodingquiet$pipinstall-Usacremoses>=0.1

$sacremoses--help
Usage:sacremoses[OPTIONS]COMMAND1[ARGS]...[COMMAND2[ARGS]...]...

Options:-l,--languageTEXTUselanguagespecificruleswhentokenizing-j,--processesINTEGERNo.ofprocesses.-e,--encodingTEXTSpecifyencodingoffile.-q,--quietDisableprogressbar.--versionShowtheversionandexit.-h,--helpShowthismessageandexit.

Commands:detokenizedetruecasenormalizetokenizetrain-truecasetruecasePipelineExample to chain the following commands:normalizewith-coption to remove control characters.tokenizewith-aoption for aggressive dash split rules.truecasewith-aoption to indicate that model is for ASRifbig.truemodelexists, load the model with-moption,otherwise train a model and save it with-moption tobig.truemodelfile.save the output to console to thebig.txt.norm.tok.truefile.catbig.txt|sacremoses-len-j4\normalize-ctokenize-atruecase-a-mbig.truemodel\>big.txt.norm.tok.trueTokenizer$sacremosestokenize--help
Usage:sacremosestokenize[OPTIONS]Options:-a,--aggressive-dash-splitsTriggersdashsplitrules.-x,--xml-escapeEscapespecialcharactersforXML.-p,--protected-patternsTEXTSpecifyfilewithpatterstobeprotectedintokenisation.-c,--custom-nb-prefixesTEXTSpecifyacustomnon-breakingprefixesfile,addprefixestothedefaultonesfromthespecifiedlanguage.-h,--helpShowthismessageandexit.$sacremoses-len-j4tokenize<big.txt>big.txt.tok100%|██████████████████████████████████|128457/128457[00:05<00:00,24363.39it/s$wgethttps://raw.githubusercontent.com/moses-smt/mosesdecoder/master/scripts/tokenizer/basic-protected-patterns$sacremoses-len-j4tokenize-pbasic-protected-patterns<big.txt>big.txt.tok100%|██████████████████████████████████|128457/128457[00:05<00:00,22183.94it/sDetokenizer$sacremosesdetokenize--help
Usage:sacremosesdetokenize[OPTIONS]Options:-x,--xml-unescapeUnescapespecialcharactersforXML.-h,--helpShowthismessageandexit.$sacremoses-len-j4detokenize<big.txt.tok>big.txt.tok.detok100%|██████████████████████████████████|128457/128457[00:16<00:00,7931.26it/s]Truecase$sacremosestruecase--help
Usage:sacremosestruecase[OPTIONS]Options:-m,--modelfileTEXTFilenametosave/loadthemodelfile.[required]-a,--is-asrAflagtoindicatethatmodelisforASR.-p,--possibly-use-first-tokenUsethefirsttokenaspartoftruecasetraining.-h,--helpShowthismessageandexit.

$sacremoses-j4truecase-mbig.model<big.txt.tok>big.txt.tok.true100%|██████████████████████████████████|128457/128457[00:09<00:00,14257.27it/s]Detruecase$sacremosesdetruecase--help
Usage:sacremosesdetruecase[OPTIONS]Options:-j,--processesINTEGERNo.ofprocesses.-a,--is-headlineWhetherthefileareheadlines.-e,--encodingTEXTSpecifyencodingoffile.-h,--helpShowthismessageandexit.

$sacremoses-j4detruecase<big.txt.tok.true>big.txt.tok.true.detrue100%|█████████████████████████████████|128457/128457[00:04<00:00,26945.16it/s]Normalize$sacremosesnormalize--help
Usage:sacremosesnormalize[OPTIONS]Options:-q,--normalize-quote-commasNormalizequotationsandcommas.-d,--normalize-numbersNormalizenumber.-p,--replace-unicode-punctsReplaceunicodepunctuationsBEFOREnormalization.-c,--remove-control-charsRemovecontrolcharactersAFTERnormalization.-h,--helpShowthismessageandexit.

$sacremoses-j4normalize<big.txt>big.txt.norm100%|██████████████████████████████████|128457/128457[00:09<00:00,13096.23it/s]