Package: artext
Artext: Artificial Text GenerationProbabilistic Noising of Natural LanguageArtext is a work on injecting noise into text without affecting the core meaning for a human reader.
This kind of data can be useful for many NLP tasks, particulary in making models robust to noisy/erroneous input.Note: Noising will generally increase the vocabulary size of the data sets, as it introduces word inflections and orthographic variations that may not have existed before. Therefore, it should be used with caution, especially for closed-vocabulary neural network models such as machine translation. In such scenarios, consider using subword based vocabulary (BPEfor instance).This is a work in progress, and the result of our experiments we will published soon.
Meanwhile, if you useartextin your research please cite this repository.Setupartext's developed and tested withPython 3.6and can be installed in two ways:Usingpip:pip install artextFrom source code:git clone https://github.com/fgaim/artext
cd artext
pip install -r requirements.txt
python setup.py installGet required resources:python-mspacydownload'en_core_web_sm'python-mnltk.downloader'punkt'python-mnltk.downloader'wordnet'UsageUse from command-lineGenerate sentence (sent) or document (doc) level noise samples for a text file as follows:python -m artext -src source.txt -out output.txt -l sent -er 0.5 -n 10[or] From source code usinginject.pyas follows:python inject.py -src source.txt -out output.txt -l sent -er 0.5 -n 10Use-hto see all options.Use as a libraryfromartextimportArtextartxt=Artext()artxt.samples=5artxt.error_rate=0.25sent='This is a sample sentence to be noised.'noises=artxt.noise_sentence(sent)print(noises)Examplespython example.py -er 0.5 -n 10Sentence Level ExamplesInput (clean sentence from Lang-8):So , I think if we have to go somewhere on foot , we must put on our hat .Human (error example from Lang-8):So , I think if we have to go somewhere on foot , we must putonour hat .Output (artext):So , I think if we have togogoing somewhere onfootfeet , we must put on our hat.?So,Ithinkthinking if we have to go somewhere on foot , we must put on ! ourhathats .So , I think ifwehave we to go somewhere onfootfeet , we must put on our hat.;So,I think if we have to go somewhere on foot , wemustput must on ourhathats .So , I think if we have to go somewhere onfootfeet , we mustputon put our hat .So,; I think if wehavetake to go somewhere on foot,we must put on ourhathats .So , I think if we have to gosomewheresomeplace  on foot , we mustputputting on ourhathats .So , I think if we have to go somewhere on foot , we must put on ourhat .chapeau ;So,I think ifwehave we togosomewhere go on foot , we must put on our hat .So , Ithinkretrieve if wehavehaving togogoing somewhere onfoot ,substructure we mustputputting on our hat .Document Level ExamplesInput (clean sentence from Lang-8):This morning I found out that one of my favourite bands released a new album .
I already forgot about Rise Against and it is a great surprise for me, because I haven't listened to them for 2 years .
I hope this band did n't become worse, like many others big ones did , and I 'll enjoy listening to it .
Well , I just have to get it and check it out .Human (error example from Lang-8):This morning I found out that one of my favouritebandsband releasedahis new album . I already forgot about Rise Againstandan it is a great surprise for me , because Ihavedid n'tlistenedreturn to them for 2 years . I hope this band did n't become worse,yet like many others big onesdid ,and I 'll enjoy listeningtoit . Well ,I just havethere remains to get it and check it out .Output (artext):This morning I found out that one of myfavouritefavored bands released a new album . I already forgot aboutRise Againstgrow Agianst and itisare a great surprise for me , because I have n'tlistenedlisten to them for 2 years.Ihopehoping thisband didbands serve n't become worse , like many others big ones did , and I 'll enjoylisteningto listening it . Well , I justhavedeliver to get it and check it out .This morning I found out that one of my favouritebands releasedband a released new album . I alreadyforgotforget about RiseAgainstAigniast and it is a great surprise for me ,becauseI beceause have n't listened to them for 2yearsgeezerhood . Ihopehoping thisband didbands n'tbecome worse ,did becoming wore like manyothersother big onesdid ,didding ; and I 'll enjoy listening to it. WelleWll , I just have to get it and check it out .This morning I found outthatone that of my favourite bands released a newalbumalbums . I alreadyforgotforgotting about RiseAgainstAainst and itisbe a greatsurprisesurprisal for me , because Ihavehaving n'tlistenedlistneed tothemtem for 2 years . I hope this banddiddo n't becomeworse ,like many others big onesdiddidding , and I 'll enjoy listening to it . Well , I just have to get it andcheckchecking it out .This morning I found out that one of my favourite bands released a new album . I already forgotaboutabuot RiseAgainstAgaiinst and it is a greatsurprisesrrpuise for me , because I have n'tlistenedlistening to them for 2yearsyear . I hope this band did n't become worse , like manyothersbig other ones did , and I 'llenjoy listeningenjoying litening to it . Well , Ijustscarce have togetgetting it andcheck itchecking out it .Thismorningmornings Ifoundground outthathTat one of myfavourite bandsfavorite band released a new album . I alreadyforgotforget aboutRise Againstarise Agsinat and it is a greatsurprisesurprisal for me ,becauseIhavebecause n'tlistenedhave listen to them for 2yearsyear . I hope this band did n't becomeworsetough , like manyothersother big ones did , and I 'llenjoylistening enjoy to it.? Well , Ijusthardly have to get it and check it out .This morning Ifoundfnuod outthathtat one of my favourite bandsreleasedreleasing a newalbum . I already forgotaboutabut RiseAgainstAigainst and it is a greatsurprisesurprises for me ,becausebecuasae I have n't listened to them for 2yearsyear . I hope this band did n'tbecomebecoming worse , like manyothersother bigonesone did,and I 'llenjoylistening enjoying to it . Well , I justhave tohaving get to it and check it out.!This morning I found out that one ofmyfavourite my bandsreleasedrelease a new album . Ialready forgotalraedyy forgotting about RiseAgainstAagaianst and itisare a greatsurprisesurprises for me,. because I have n'tlistenedlisten to them for 2 years . I hope thisbanddid band n't become worse , likemanyothers big ones did , and I 'll enjoy listening to it . Well , I just have to get it and check it out .This morning Ifoundincur out that one of myfavouritefavored bandsreleasedreleaseed a newalbumalbums . I already forgot about RiseAgainstigAanst and it is agreatgrat surprisefor me , because Ihavehaving n'tlistenedlisten to them for 2 years.I hope this band did n't becomeworse , like many others bigones didone do , and I 'llenjoyenjoying listening to it . Well,: I justhavehaving togetgetting it and check it out .This morning Ifoundfounding outthathTat one of my favourite bandsreleasedreleasing anewnewfangled album . I alreadyforgotblock about RiseAgainstAganst and it is a great surprise for me ,becausebecuasee I have n't listened to them for 2 years . I hopethistthis band did n'tbecomebecoming worse,: like many others big ones did , and I 'll enjoy listening to it . Well , I justhavehaving to get it and check it out . .This morningIfound I out that one of my favouritebands releasedband releasing a new album . I already forgot aboutRiseRising Against and itisagreatis Greeat surprise for me , because I have n't listened to them for 2 years . Ihopedesire this band did n't become worse , like many others big onesdiddidding , and I 'llenjoyenjoying listening to it . Well,? I just have to get it and check it out .