Package: markov-word-generator
markov-word-generator`A small Python library to generate random credible/plausible words based on a  list of words by estimating the probability of the next character from the frequency of the previous N ones.
This usesMarkov chainInstallationpipinstallmarkov-word-generatorPrincipleIn order to generate random words that sounds like real words, we need to analyze character distribution among a corpus in a given language.
We can start by analyzing the character apparition frequency based on the previous character.Here are heatmaps showing the distribution of each character (column) given the previous one (row).$= End of word^= Start of a wordIn English:And in French:Estimating the probability of a character to appear given the probability of the given previous one works fine but is still hazardous. We can increase the likelihood of the world sounds true by looking at the next N previous characters.The generator will parse an input text file containing one word per line (dictionary), count each character occurrence based on the occurrence of the N previous ones  create a mapping table for each character-combination and its associated frequency in the corpus.UsageParsing the English dictionary to create a pseudo-word that sounds English by generating characters one by one.
In this example, it works by analyzing the probability of each character to appear based on the last 4 ones.frommarkov_word_generatorimportMarkovWordGenerator,WordType# Generate a random word in English by predicting the probability of each new character based on its last 4 last charactersgenerator=MarkovWordGenerator(markov_length=4,language='en',word_type=WordType.WORD,)print(generator.generate_word())output:rebutaneouslyParametersMarkovWordGenerator():markov_length:int. Number of previous characters the generator will take into account to compute probability of apparition of each the next character.language:str. Language to use to generate the word. Must be part of the supported languages.word_type:str. Type of word to generate. Must be part of the supported word types.dictionary_filename:str. Corpus the generator will parse to analyze character apparition frequency. Must be used only iflanguageandword_typeare not set.ignore_accents:Optional boolean. If set toTrue, Accents will not be considered while parsingdictionary_filename. Default toFalsegenerate_word()seed:Optional str. If seed is set, it will generate a word starting with this seedfrommarkov_word_generatorimportMarkovWordGenerator,WordType,AllowedLanguages# Generate a random German name by predicting the probability of each new character based on its last 3 last charactersgenerator=MarkovWordGenerator(markov_length=3,language=AllowedLanguages.DE,word_type=WordType.NAME,)print(generator.generate_word())ludgertenSupported languages and word_typesimportmarkov_word_generator# List supported languagesprint(markov_word_generator.get_supported_languages())# ['EN', 'FR', 'DE', 'FI', 'IT', 'PT', 'SE']# List supported word_typeprint(markov_word_generator.get_supported_word_types())# ['WORD', 'NAME']More languages and word types (plants, movie names, cities...) can be added in the future.Impact of the markov_length parameterThe higher the number of characters N we take into account, the more credible the word will be. We may end up with already existing words (seeImpact of the markov_length parameterbelow).Lowering N will lead to words that sound less real. Some words will also either very short (1-2 chars) or very long (>20chars)frommarkov_word_generatorimportMarkovWordGenerator,WordType,AllowedLanguagesgenerator=MarkovWordGenerator(markov_length=N,# N=1,2,3,4 or 5 in following exampleslanguage=AllowedLanguages.EN,word_type=WordType.WORD)foriinrange(0,10):print(generator.generate_word())Length 1output:eroun
unteticakreatintes
sucle
erarums
eablatirlac
e
ghils
rllig
beseleforuat
deLength 2output:malle
dallintathilight
boaddly
nobtiousle
ing
alaymplaings
rusle
sprevircirdbages
bant
ritablegruphicallsLength 3output:blungalinther
super
solder
degreetricked
mittlessly
out
hearf
fracertory
gyny
lociousLength 4output:authering
negligented
manoeistical
bleat
lover
confusions
dest
hand
display
entwinkleLength 5output:significative
contention
grandmaidens
aidesdecamped
paralleled
contradicate
thereby
numskull
crises
battlegroBenchmarksEmpirically generating 5000 random words for each of the tests and checking the percentage of them which do exist as actual valid words.
10 tests have been running. From N=1 to N=5 in both English and French languages
Results are the followingN\LanguageENFR*14.61%6.15%28.89%10.60%314.80%10.04%433.08%33.88%562.84%65.68%Empirical measurements of the percentage of output words from the generator that are real words (exists in the dictionary) based on the number of characters N we take into account in the markov_chainover 5000 samples
*accents have been ignored in benchmarks in FrenchFrom N=5, there are more than 50% of chances to generate an existing word.More examplesRandom generated wordsENFRESDEITSEduplicableschouchonnéesinflamandandoregenfreundenscommissarimedmännensfeathenismfumigentsdiacontenderárechtsbeleuchtesinsortitimetallösningensconvolutionalistsaponisassieztransnacionaránunerschiebenimmalintenzialestationskligtjinglehandpareraientabundeounstimmepronometroarbetslöftenasstarinesstoniciensencuestionóüberredeteacconcilianiutredningsviljandetrellishchalleabombearánzwischereafferrofiliatributionsverktygssubsidiariestpotamesbanderolasteisplädiertemdispiacereteslappningarnasdiscourselfrudoyersconstruéiswolkentrisecchereraitidsnäringsmelanchoristreluisionnésdesagüensekompetentenzeichnenriappaviaspagatelligcleavagerysacagneusedesvergonzaremosdümmstsgancializzandoyngstakternasRandom generated namesENFRESDEITSEcharlenaariansandrogermannseveronicabrittansorrellclementinuliseogundeevarissakristinaustineethéophieteofilomenawertholdflorenafridhardineaugustineherinahanneliatiziatorsteinshantaljeanninetteamilohelmarleonardinandagittakristianflavierleandratatjafortunatalekerstinalessicaisidoniedolorenciosieghardtsimondosigfridareanaclothairedionaneliageltrudenziothorstenleanoreenfabrielanunciatrudbattiagunilsroslynbastiennecaliseleonhardlorentinajerkelGiven other types of dictionaries, generator can create random words in some specific topics: Random jobs, random plants, random animals, random cities...