Package: augtxt
augtxt – Text AugmentationYet another text augmentation python package.Table of ContentsUsage`augtxt.augmenters- Pipelines <#pipelines>`__`sentaugm- Sentence
Augmentation <#sentence-augmentations>`__`wordtypo- Word Typos <#word-typos>`__`senttypo- Word typos for a
sentence <#word-typos-for-a-sentence>`__`augtxt.typo- Typographical
Errors <#typographical-errors-tippfehler>`__`augtxt.punct- Interpunctation
Errors <#interpunctation-errors-zeichensetzungsfehler>`__`augtxt.order- Word Order
Errors <#word-order-errors-wortstellungsfehler>`__`augtxt.wordsubs- Word substitutions <#word-substitutions>`__AppendixInstallationCommandsSupportContributingUsageimportaugtxtimportnumpyasnpPipelinesSentence AugmentationsCheck thedemo notebookfor an
usage example.Word typosThe functionaugtxt.augmenters.wordtypoapplies randomly different
augmentations to one word. The result is a simulated distribution of
possible word augmentations, e.g. how are possible typological errors
distributed for a specific original word. The procedure doesnot
guaranteethat the original word will be augmented.Check thedemo notebookfor
an usage example.Word typos for a sentenceThe functionaugtxt.augmenters.senttypoapplies randomly different
augmentations to a) at least one word in a sentence, or b) not more than
a certain percentage of words in a sentence. The procedureguaranteesthat the sentence is augmented.The functions also allows to exclude specific strings from augmentation
(e.g.exclude=("[MASK]","[UNK]")). However, these stringscannotinclude the special characters.,;:!?(incl. whitespace).Check thedemo notebookfor an usage example.Typographical Errors (Tippfehler)Theaugtxt.typomodule is about augmenting characters to mimic human
errors while using a keyboard device.Swap two consecutive characters (Vertauscher)A user mix two consecutive characters up.Swap 1st and 2nd characters:augtxt.typo.swap_consecutive("Kinder",loc=0)(Result:iKnder)Swap 1st and 2nd characters, and enforce letter cases:augtxt.typo.swap_consecutive("Kinder",loc=0, keep_case=True)(Result:Iknder)Swap randomi-th andi+1-th characters that are more likely
at the end of the word:np.random.seed(seed=123);augtxt.typo.swap_consecutive("Kinder",loc='end')Add double letter (Einfüger)User presses a key twice accidentalyMake 5th letter a double letter:
`augtxt.typo.pressed_twice("Eltern",loc=4)(Result:Elterrn)Drop character (Auslasser)User presses the key not enough (Lisbach, 2011, p.72), the key is
broken, finger motion fails.Drop the 3rd letter:augtxt.typo.drop_char("Straße",loc=2)(Result:Staße)Drop character followed by double letter (Vertipper)Letter is left out, but the following letter is typed twice. It’s a
combination ofaugtxt.typo.pressed_twiceandaugtxt.typo.drop_char.fromaugtxt.typoimportdrop_n_next_twiceaugm=drop_n_next_twice("Tante",loc=2)# TattePressed SHIFT, ALT, or SHIFT+ALTUsuallySHFITis used to type a capital letter, andALTorALT+SHIFTfor less common characters. A typo might occur because
these special keys are nor are not pressed in combination with a normal
key. The functionaugtxt.typo.pressed_shiftaltsuch errors randomly.fromaugtxt.typoimportpressed_shiftaltaugm=pressed_shiftalt("Onkel",loc=2)# OnKel, On˚el, OnelThekeymapcan differ depending on the language and the keyboard
layout.fromaugtxt.typoimportpressed_shiftaltimportaugtxt.keyboard_layoutsaskblaugm=pressed_shiftalt("Onkel",loc=2,keymap=kbl.macbook_us)# OnKel, On˚el, OnelFurther, transition probabilities in case of a typo can be specifiedfromaugtxt.typoimportpressed_shiftaltimportaugtxt.keyboard_layoutsaskblkeyboard_transprob={"keys":[.0,.75,.2,.05],"shift":[.9,0,.05,.05],"alt":[.9,.05,.0,.05],"shift+alt":[.3,.35,.35,.0]}augm=pressed_shiftalt("Onkel",loc=2,keymap=kbl.macbook_us,trans=keyboard_transprob)ReferencesLisbach, B., 2011. Linguistisches Identity Matching. Vieweg+Teubner,
Wiesbaden.https://doi.org/10.1007/978-3-8348-9791-6Interpunctation Errors (Zeichensetzungsfehler)Remove PUNCT and COMMA tokensThe PUNCT (.?!;:) and COMMA (,) tokens carrysyntaticinformation. An use caseimportaugtxt.puncttext=("Die Lehrerin [MASK] einen Roman. ""Die Schülerin [MASK] ein Aufsatz, der sehr [MASK] war.")augmented=augtxt.punct.remove_syntaxinfo(text)# 'Die Lehrerin [MASK] einen Roman Die Schülerin [MASK] ein Aufsatz der sehr [MASK] war'Merge two consequitive wordsThe functionaugtxt.punct.merge_wordsremoves randomly whitespace or
hyphens between words, and transform the second word to lower case.importaugtxt.puncttext="Die Bindestrich-Wörter sind da."np.random.seed(seed=23)augmented=augtxt.punct.merge_words(text,num_aug=1)assertaugmented=='Die Bindestrich-Wörter sindda.'np.random.seed(seed=1)augmented=augtxt.punct.merge_words(text,num_aug=1)assertaugmented=='Die Bindestrichwörter sind da.'Word Order Errors (Wortstellungsfehler)Theaugtxt.ordersimulate errors on word token level.Swap wordsnp.random.seed(seed=42)text="Tausche die Wörter, lasse sie weg, oder [MASK] was."print(augtxt.order.swap_consecutive(text,exclude=["[MASK]"],num_aug=1))# die Tausche Wörter, lasse sie weg, oder [MASK] was.Write twicenp.random.seed(seed=42)text="Tausche die Wörter, lasse sie weg, oder [MASK] was."print(augtxt.order.write_twice(text,exclude=["[MASK]"],num_aug=1))# Tausche die die Wörter, lasse sie weg, oder [MASK] was.Drop wordnp.random.seed(seed=42)text="Tausche die Wörter, lasse sie weg, oder [MASK] was."print(augtxt.order.drop_word(text,exclude=["[MASK]"],num_aug=1))# Tausche Wörter, lasse sie weg, oder [MASK] was.Drop word followed by a double wordnp.random.seed(seed=42)text="Tausche die Wörter, lasse sie weg, oder [MASK] was."print(augtxt.order.drop_n_next_twice(text,exclude=["[MASK]"],num_aug=1))# die die Wörter, lasse sie weg, oder [MASK] was.[STRIKEOUT:Word substitutions] (Deprecated)Deprecation Notice:augtxt.wordsubswill be deleted in 0.6.0 and
replaced. Especially synonym replacement is not trivial in German
language. Please checkhttps://github.com/ulf1/flexionfor further
information.Theaugtxt.wordsubsmodule is about replacing specific strings,
e.g. words, morphemes, named entities, abbreviations, etc.Using pseudo-synonym dictionaries to augment tokenized sequencesIt is recommend to filtervocabfurther. For example, PoS tag the
sequences and only augment VERB and NOUN tokens.importitertoolsimportaugtxt.wordsubsimportnumpyasnporiginal_seqs=[["Das","ist","ein","Satz","."],["Dies","ist","ein","anderer","Satz","."]]vocab=set([s.lower()forsinitertools.chain(*original_seqs)iflen(s)>1])synonyms={'anderer':['verschiedener','einiger','vieler','diverser','sonstiger','etlicher','einzelner','bestimmter','ähnlicher'],'satz':['sätze','anfangssatz','schlussatz','eingangssatz','einleitungssatzes','einleitungsssatz','einleitungssatz','behauptungssatz','beispielsatz','schlusssatz','anfangssatzes','einzelsatz','#einleitungssatz','minimalsatz','inhaltssatz','aufforderungssatz','ausgangssatz'],'.':[',','🎅'],'das':['welches','solches'],'ein':['weiteres'],'dies':['was','umstand','dass']}np.random.seed(42)augmented_seqs=augtxt.wordsubs.synonym_replacement(original_seqs,synonyms,num_aug=10,keep_case=True)# check results for 1st sentenceforsinaugmented_seqs[0]:print(s)AppendixInstallationTheaugtxtgit repois available
asPyPi packagepipinstallaugtxt>=0.5.0pipinstallgit+ssh://git@github.com/ulf1/augtxt.gitCommandsInstall a virtual environmentpython3 -m venv .venv
source .venv/bin/activate
pip install --upgrade pip
pip install -r requirements.txt
pip install -r requirements-dev.txt
pip install -r requirements-demo.txt(If your git repo is stored in a folder with whitespaces, then don’t use
the subfolder.venv. Use an absolute path without whitespaces.)Python commandsCheck syntax:flake8--ignore=F401--exclude=$(grep-v'^#'.gitignore | xargs | sed-e's//,/g')Run Unit Tests:pytestPublishpandocREADME.md--frommarkdown--torst-s-oREADME.rstpythonsetup.pysdisttwineupload-rpypidist/*Clean upfind . -type f -name "*.pyc" | xargs rm
find . -type d -name "__pycache__" | xargs rm -r
rm -r .pytest_cache
rm -r .venvSupportPleaseopen an issuefor
support.ContributingPlease contribute usingGithub
Flow. Create a branch,
add commits, andopen a pull
request.