Package: autoclean
AutocleanAutoclean is a set of tools to clean textual data with minimal human intervention:autoclean.filteringuses unsupervised learning to filter out of domain sequencesautoclean.segmentationuses unsupervised learning to segment joined up sequencesFilteringThe filtering package contains both an api and a cli.An example use through the cli to filter out out-of-domain/anomalous data points:Download an English book from Project Gutenberg(venv)$wgethttps://www.gutenberg.org/cache/epub/29440/pg29440.txt
--2022-02-1500:14:04--https://www.gutenberg.org/cache/epub/29440/pg29440.txt
Resolvingwww.gutenberg.org(www.gutenberg.org)...152.19.134.47,2610:28:3090:3000:0:bad:cafe:47
Connectingtowww.gutenberg.org(www.gutenberg.org)|152.19.134.47|:443...connected.
HTTPrequestsent,awaitingresponse...200OK
Length:890028(869K)[text/plain]Savingto:‘pg29440.txt’

pg29440.txt100%[==================================================================================================================>]869.17K730KB/sin1.2s2022-02-1500:14:06(730KB/s)-‘pg29440.txt’saved[890028/890028]Run the filtering giving a high cost-threshold, so the output is just all the lines but ordered in decreasing order of
cost:(venv)$timepython-mautoclean.filtering.clifilterpg29440.txtfiltered.pg20440.txt10000[2022-02-1500:26:38][INFO][autoclean._estimate:178]>estimating[6]-gramfrom[pg29440.txt]...[2022-02-1500:26:40][INFO][autoclean._estimate:182]>binarising[6]-gramfrom[/tmp/pg29440.txt.2022-02-15T00:26:38.2265355794898408859.6.arpa]...[2022-02-1500:26:40][INFO][autoclean._estimate:178]>estimating[1]-gramfrom[pg29440.txt]...[2022-02-1500:26:40][INFO][autoclean._estimate:182]>binarising[1]-gramfrom[/tmp/pg29440.txt.2022-02-15T00:26:40.2265355794898408859.1.arpa]...[2022-02-1500:26:40][INFO][autoclean.filtering.filter_out:30]>Readingfrom[pg29440.txt]andwritingto[filtered.pg20440.txt]...[2022-02-1500:26:40][INFO][autoclean.filtering.filter_out:43]>Done.

real0m3.201s
user0m1.882s
sys0m2.803sCheck the top of the output file for standard English sentences:(venv)$head-n25filtered.pg20440.txtinsome
ofthecountry?
ofthewhole:--
accordingtothepoet,
Election--Responsibility--InfluenceoftheChurchinsuchaintheoriginal.
characteroftheschoolmaster,andtherightanddutyofthe
thereoftheinfluencewhichthepersonalcharacterofthe
theirChurchistheChurchofthenation,andthatitisthey
true,andinthejustandthetrueonly.
streamofnothalfthevolumeofthatinwhichthemoneyofthe
savageismoftheother.
classwithwhichtheschoolmasterandtheclasswithwhichthe
betweenthefeelingsofthepeopleandtheanticipationsofsomeof
thelightoftheflames.'in any age of the Presbyterian Church, in one of the parishAccording to the poet,circumstances of Scotland in his days and of Scotland in the presentthe State is the money of the people, and that the people have a rightis not according to the nature of things that the case of the tenanttheir part of the scheme.'whichallrecognitionofthereligiouselementonthepartofthe
simplythoseofthesubjectandtheratepayer.
characterofourpoorHighlanders,andoftheinfluenceofthebothie
themselves,andnottheministersoftheEstablishment,whoareonCheck the bottom of the output file for non-standard English text:(venv)$tail-n25filtered.pg20440.txt
residence.'periodicals.consistency.predecessor.differently?FINE-BODYISM.ELIGIBLE.'{8}Glencalvie:--
photographer.
resuscitated.
PERIODICALISM.
double-quote}relinquished.
circumstances.
Protestantism.
significantly,
impossibility.
responsibility.
redistribution.
gbnewby@pglaf.org
kingdom.'--SISMONDI.
self-recommendation.
http://www.gutenberg.org
http://gutenberg.org/license).
http://www.gutenberg.org/2/9/4/4/29440/The cli can also be used to evaluate the filtering algorithm with an in-domain and an out-of-domain text:Download French book from Project Gutenberg(venv)$wgethttps://www.gutenberg.org/cache/epub/46916/pg46916.txt
--2022-02-1500:41:00--https://www.gutenberg.org/cache/epub/46916/pg46916.txt
Resolvingwww.gutenberg.org(www.gutenberg.org)...152.19.134.47,2610:28:3090:3000:0:bad:cafe:47
Connectingtowww.gutenberg.org(www.gutenberg.org)|152.19.134.47|:443...connected.
HTTPrequestsent,awaitingresponse...200OK
Length:120716(118K)[text/plain]Savingto:‘pg46916.txt’

pg46916.txt100%[==================================================================================================================>]117.89K347KB/sin0.3s2022-02-1500:41:01(347KB/s)-‘pg46916.txt’saved[120716/120716]Run evaluation on the Downloaded English and French books, using the former as in-domain and the latter as out-of-domain(venv)$timepython-mautoclean.filtering.clievalpg29440.txtpg46916.txt[2022-02-1500:47:29][INFO][autoclean.filtering.evaluate:62]>Readingfrom[pg29440.txt]andto[pg46916.txt]...[2022-02-1500:47:29][INFO][autoclean._estimate:178]>estimating[6]-gramfrom[/tmp/pg29440.txt.pg46916.txt.2022-02-15T00:47:29.1382531328938857964.txt]...[2022-02-1500:47:31][INFO][autoclean._estimate:182]>binarising[6]-gramfrom[/tmp/pg29440.txt.pg46916.txt.2022-02-15T00:47:29.1382531328938857964.txt.2022-02-15T00:47:29.6537006284435656167.6.arpa]...[2022-02-1500:47:31][INFO][autoclean._estimate:178]>estimating[1]-gramfrom[/tmp/pg29440.txt.pg46916.txt.2022-02-15T00:47:29.1382531328938857964.txt]...[2022-02-1500:47:31][INFO][autoclean._estimate:182]>binarising[1]-gramfrom[/tmp/pg29440.txt.pg46916.txt.2022-02-15T00:47:29.1382531328938857964.txt.2022-02-15T00:47:31.6537006284435656167.1.arpa]...[2022-02-1500:47:31][INFO][autoclean.filtering.evaluate:74]>Calculating10%-cutoffrankingsmetrics...

@100%precision[94.6832%]recall[94.6832%]fallout[38.4824%]@90%precision[96.7801%]recall[87.1050%]fallout[20.9756%]@80%precision[97.4352%]recall[77.9467%]fallout[14.8509%]@70%precision[97.6359%]recall[68.3466%]fallout[11.9783%]@60%precision[97.8532%]recall[58.7090%]fallout[9.3225%]@50%precision[98.1878%]recall[49.0939%]fallout[6.5583%]@40%precision[98.4650%]recall[39.3889%]fallout[4.4444%]@30%precision[98.4024%]recall[29.5192%]fallout[3.4688%]@20%precision[98.5399%]recall[19.7095%]fallout[2.1138%]@10%precision[98.4270%]recall[9.8397%]fallout[1.1382%][2022-02-1500:47:32][INFO][autoclean.filtering.evaluate:104]>Done.

real0m3.400s
user0m1.967s
sys0m2.622sSegmentationSegmentation is based on the recursive dynamic-programming algorithm presented by Peter Norvig inchapter 14 of "Beautiful Data", by Seagaran and Hammerbacher.It features an api and cli. An example usage through the cli:Download file to use as corpus (complete works of shakespeare)(venv)$wgethttp://norvig.com/ngrams/shakespeare.txt
--2022-02-1512:47:22--http://norvig.com/ngrams/shakespeare.txt
Resolvingnorvig.com(norvig.com)...158.106.138.13
Connectingtonorvig.com(norvig.com)|158.106.138.13|:80...connected.
HTTPrequestsent,awaitingresponse...200OK
Length:4538523(4.3M)[text/plain]Savingto:‘shakespeare.txt’

shakespeare.txt100%[==================================================================================================================>]4.33M3.29MB/sin1.3s2022-02-1512:47:24(3.29MB/s)-‘shakespeare.txt’saved[4538523/4538523]Create the file to be segmented by deleting spaces between words:(venv)$sed's/\s//g'shakespeare.txt>joined.shakespeare.txtRun the segmentation with a sixgram language model:(venv)$timepython-mautoclean.segmentation.cliseg--lm=5-cshakespeare.txtjoined.shakespeare.txtseg.shakespeare.txt[2022-02-1512:48:35][INFO][autoclean.segmentation.segment:29]>Readingincorpusfrom[shakespeare.txt]...[2022-02-1512:48:35][INFO][autoclean.segmentation.segment:42]>Estimating[6]-ngramlanguagemodelfrom[shakespeare.txt]...[2022-02-1512:48:35][INFO][autoclean._estimate:178]>estimating[6]-gramfrom[/tmp/corpus.2022-02-15T12:48:35.6.txt]...[2022-02-1512:48:44][INFO][autoclean._estimate:182]>binarising[6]-gramfrom[/tmp/corpus.2022-02-15T12:48:35.6.txt.2022-02-15T12:48:35.3998131699779813468.6.arpa]...[2022-02-1512:48:45][INFO][autoclean.segmentation.segment:45]>Segmentingandsavingto[seg.shakespeare.txt]...[2022-02-1512:48:45][INFO][autoclean.segmentation.segment:49]>Readinginfiletosegmentfrom[joined.shakespeare.txt]...[2022-02-1512:52:56][INFO][autoclean.segmentation.segment:59]>Done.

real4m32.020s
user4m30.212s
sys0m2.493sEvaluate the segmentation using the original text as a gold standard:(venv)$timepython-mautoclean.segmentation.clievalshakespeare.txtseg.shakespeare.txt[2022-02-1512:55:32][INFO][autoclean.segmentation.evaluate:70]>Readingsegmentedtextfrom[shakespeare.txt]...[2022-02-1512:55:32][INFO][autoclean.segmentation.evaluate:71]>Readinggoldstandardtextfrom[seg.shakespeare.txt]...[2022-02-1512:55:32][INFO][autoclean.segmentation.evaluate:84]>Accuracy:[99.80%],[128,844]correctoutof[129,107]sequences[2022-02-1512:55:32][INFO][autoclean.segmentation.evaluate:87]>Done.

real0m0.624s
user0m0.732s
sys0m0.723sInstallationAs Autoclean is on Pypi and has direct dependencies like Pytorch and KenLM, and Pypi doesn't support them,
they need to be installed separately and previous to installing Autoclean:$gitclonegit@github.com:JoseLlarena/autoclean.git
$python3virtualenv-p=3.8venv
$sourcevenv/bin/activate(venv)$pip3install-fhttps://download.pytorch.org/whl/cu113/torch_stable.html-r./autoclean/requirements.txtThen install Autoclean with pip:(venv)$pip3installautocleanThe code requires python 3.8+ on and Nvidia GPU. The KenLM/SRILM binaries have been compiled against Linux x86_64 and may
not work on other architectures.TestingThe unit tests are written withpytest. Run with:$pip3installpytest

$pytestChangelogCheck theChangelogfor fixes and enhancements of
each version.LicenseCopyright Jose Llarena 2022.Distributed under the terms of theMITlicense,
autoclean is free and open source software.