Package: wagoner
wagoneris an anagram for rawogen, a random words generator.wagonergenerates random words based on a given text. A random word is
generated incrementally, and the given text is used to know which character
should follow.Usagewagonergenerates random words from tables extracted from texts. You need
first to extract such a table from a text. To extract a table, usewagoner.table:python -m wagoner.table TEXT --output text.tableThis command will extract the table corresponding toTEXTinto the filetext.table.Then, given the extracted table,wagoner.wordcan extract random words:python -m wagoner.word text.tableThis command will generate ten words of ten characters, based on the
information intext.table. To generate another number of words, use the--countoption and to change their length, the--lengthoption.Random words can also be generated from trees. While the tables only tell
the generation which character should follow, trees also ensure that the
generated word will start like a regular word (from the text it comes from) and
will end like a regular (maybe different word). Trees are built from a
table or from a text:python -m wagoner.tree CONTENT --output text.treeThis command will extract a tree fromCONTENTand save it into the filetext.tree.Warning: trees can be very large and expensive to build; to control their
complexity, you can use the--prefixand--lengthoptions. See below
for more information.How does it workThe main idea behind wagoner is to use a text to conduct the generation of
random words. The words in the text show what usually follows a given prefix,
and this information is used to choose the next character when building a word
incrementally.For example, given the text composed of the two wordsambiguousandgamblings, and the prefixig, the first word tells us that the next
character can beu(because inambiguous, the sub-wordigis
followed byu), and the second word tells us that the next character can
beaors(because ingamblings, the prefixgis followed byaors). The next character is chosen randomly among the possible next
characters and the word is incrementally generated following the same rule.This next character is chosen randomly among the possible next characters, but
weights are used to more probably use frequent characters, and longer
prefixes. In other words, when choosing the next character for the prefixig, more weight is given touthan toaandsbecauseufollowsiginambiguouswhileaandsonly followgingamblings. Furthermore, if in the given text, a character follows more
frequently a given prefix than another, the first one will have a higher
chance to be chosen. This last characteristics of the generation can be turned
down by using the--flattenoption ofwagoner.word. If this option is
active, two characters following the same prefix will have the same chance to
be chosen; nevertheless, a character following a shorter prefix will have less
chance to be chosen than a character following a longer prefix.This information of the frequence of successing characters in the words of
the given text is stored in tables. Note that the--flattenoption is
also present withwagoner.table; this forceswagoner.tableto
extract a table in which the number of occurrences of a character following a
given prefix is forgotten, giving the same result as the--flattenoption
ofwagoner.wordat generation.Finally, when incrementally generating a random word, the full prefix is used
to extract next character. In usual languages such as English, knowing the
full prefix is not necessary to generate a pronounceable word; instead, three
or four characters are usually sufficient to choose a next character that will
sound correctly. Restricting the generation of words based on a bounded prefix
instead of the full word can be achieved by using the--prefixoption ofwagoner.word. Similarly, the same option can be used onwagoner.tableto directly generate (smaller) tables that will not propose next characters
for longer prefixes.Trees store more information than tables, but are also tailored to more
strict generation. Trees store the information about which character should
follow a given prefix, but only for words of a given length. They also store
information to generate words starting as words of the text and ending as them.
Because of this addition information, trees can be really complex, huge and
long to generate; they can potentially store each word of a given length
generable from the table. To tackle this complexity, trees can be limited to
looking at only a suffix of the current prefix by using the--prefixoption; furthermore, by building trees for shorter words (using the--lengthoption), these trees are smaller.ExampleLet’s take the list of English words available onhttp://www-01.sil.org/linguistics/wordlists/english/. This list contains more
than 100000 English words and should be useful to generate random words that
sound English.First thing to do before generating random words is to extract the table:python -m wagoner.table wordsEn.txt --output wordsEn.tableThe table is extracted and saved in thewordsEn.tablefile. Then, we can
generate random words:python -m wagoner.word wordsEn.tablethat outputs, for example:joggeriest
lvinistica
cleiadicat
zenerousne
ulencering
mmanencien
zeratenesi
keynoteric
encientnes
crappinesaYou can see that some words are really like real words:crappinesareally
looks likecrappiness, a word ofwordsEn.txt. To avoid generating such
too real words, we can ask wagoner to only consider finite prefixes; this will
avoid to be trapped in the increasing proability of ressembling the wordcrappinessas the word is generated:python -m wagoner.word wordsEn.table --prefix=2that outputs, for example:keyelittat
retimcenve
quedectrot
fodcalitur
xcedission
queffliqui
eshedlerad
ficklapett
quatersous
sulationurIn this case, the words are still pronounceable, but are not words of thewordsEn.txtfile.If you want to generate words that start and end the same way words of the
text start and end, you can generate a tree from the table:python -m wagoner.tree wordsEn.table --output=wordsEn.tree --prefix=3The tree can then be used to generate words:python -m wagoner.word wordsEn.treethat outputs, for example:sinationso
disjudging
titualimal
avespolybd
prophology
japackersc
nonneappra
overedefra
oxideodore
wordshedroLibrary usagewagonercan also be used as a library.wagoner.table.Tablerepresent
tables and a table can be extracted from a text file with:table = Table.from_words(wagoner.utils.extract_words(text_file))Table.from_wordsaccept two optional arguments:prefix: if greater than 0, the length of the prefixes to take into
account when generating the word (default is 0).flatten: ifTrue, the table is flattened and two successing
characters for the same prefix will have the same weight (default isFalse).From such a table, a random word can be extracted:word = table.random_word(word_length)whereword_lengthis the length of the desired word.random_wordaccepts several optional arguments:prefixandflatten, as above.start: ifTrue, the generated word starts as a word of the text
the table is extracted from (default isFalse).end: ifTrue, the generated word ends as a word of the text
the table is extracted from (default isFalse).
Warning: this option should not be used because it is very time consuming.Furthermore,wagoner.tree.Treerepresent trees and a tree can be
extracted from a table with:tree = Tree.from_table(table, word_length)wheretableis a table like the one above andword_lengthis the
length of the words the tree will produce. LikeTable.from_words,Tree.from_tablesupports two optional arguments:prefixandflatten. In this case, theprefixargument is very important
because, if set to 0 (the default value), the tree can be really huge, even
impossible to stay in memory, and will need a lot of time to be built.From such a te, a random word can be extracted:word = tree.random_word()Unlike the tables, trees contain the complete information to extract words
(their length, whether flattening the weights) andrandom_wordtakes no
optional argument (technically, it accepts any arguments, but will ignore
them).Changelog1.1: support for Python 2.7.