Package: textnoisr
textnoisr: Adding random noise to a datasettextnoisris a python package that allows toadd random noise to a text dataset,
and tocontrol very accuratelythe quality of the result.Here is an example if your dataset consists on the first few lines ofthe Zen of python:Raw textThe Zen of Python, by Tim Peters

Beautiful is better than ugly.
Explicit is better than implicit.
Simple is better than complex.
Complex is better than complicated.
Flat is better than nested.
...Noisy textTheO Zen of Python, by Tim Pfter

BzeautiUful is ebtter than ugly.
Eqxplicin is better than imlicit.
Simple is beateUr than comdplex.
Complex is better than comwlicated.
Flat is bejAter than neseed.
...Four types of "actions" are implemented:inserta random character, e.g.        STEAM  →  STREAM,deletea random character, e.g.STEAM  →  TEAM,substitutea random character, e.g.    STEAM→  STEAL.swaptwoconsecutivecharacters, e.g.  STEAM→  STEMAThe general philosophy of the package is that onlyone single parameteris needed to control the noise level.
This "noise level" is applied character-wise,
and correspondsroughlyto the probability for a character to be impacted.More precisely, this noise level is calibrated so that
theCharacter Error Rateof a noised dataset converges to this value as the amount of text increases.Why a whole package for such a simple task?In the case of inserting, deleting and substituting characters at random with a probability $p$,
the Character Error Rate is only the average number of those operations,
so it will converge to the input value $p$ due to theLaw of Large Numbers.However, the case of swapping consecutive characters isnot trivial at allfor two reasons:First, swapping two characters is not an "atomic operation" with respect to the Character Error Rate metric.Second, we do not want to swap repeatedly the same character over and over again
if the probability to apply the swap action is high:STEAM  →TSEAMTSEAM  →  TESAMTESAM  →  TEASMTEASM→  TEAMSThis would be equivalent toSTEAM  →
TEAMS, so this cannot be considered "swapping consecutive characters".
To avoid this behavior, we must avoid swapping a character if it has just been swapped.
This breaks the independency between one character and the following one,
and makes the Law of Large Numbers not applicable.We use Markov Chains to model the swapping of characters.
This allows us to compute and correct the corresponding bias in order to make itstraightforward
for the user to get the desired Character Error Rate, as if the Law of Large Number could beapplied!All the details of this unbiasingare here.
The goal of this package is for the user to be confident on the result
without worrying about the implementation details.The documentation follows this plan:You may want to followa quick tutorialto learn the basics of the package,TheResultspage illustrates howno calibration is neededin order to add noise to a corpus with a target Character Error Rate.TheHow this works sectionexplains the mechanisms, and some design choices of this package.
We have been extra careful to explain how some statistical bias have been avoided,
for the package to be both user-friendly and correct.
Adedicated pagedeeps dive in the case of theswapaction.TheAPI Referencedetails all the technical descriptions needed.