Package: pyDAWG
ContentsIntroductionLicenseAuthorInstallationAPIModuleUnicode and bytesDAWGclassPropertyBasic mathodsIteratorMinimal perfect hashingExampleOtherIntroductionPyDAWGis a python module implementsDAWGgraph structure,
which allow to store set of strings and check existence of a string in
linear time (in terms of a string length).DAWG is constructed byincremental algorithmdescribed inIncremental
algorithm for sorted data,Jan Daciuk,Stoyan Mihov,Bruce Watson,
andRichard Watson, Computational Linguistics, 26(1), March 2000.
Prof. Jan Daciuk offers also some useful documentation, presentations and
even sample code onhis site.The algorithm asserts that input words aresortedinlexicographic order; default Pythonsort()orders strings correctly.Alsominimal perfect hashing(MPH) is supported, i.e. there is a function
that maps words to unique number; this function is bidirectional, its possible
to find number for given word or get word from number.There are two versions of module:C extension, compatible only with Python3;pure python module, compatible with Python 2 and 3.Python module implements subset of C extension API.LicenseLibrary is licensed under very liberal three-clauses BSD license.
Some portions has been released into public domain.Full text of license is available in LICENSE file.AuthorWojciech Muła,wojciech_mula@poczta.onet.plInstallationCompile time settings (can be change in setup.py):DAWG_UNICODE— if defined, DAWG accepts and returns
unicode strings, else bytes are supportedDAWG_PERFECT_HASHING— when defined, minimal perfect
hashing is enabled (methods word2index and index2word are
available)Just run:$ python setup.py installIf compilation succed, module is ready to use.APIModuleModulepydawgprovides classDAWGand following members:EMPTY,ACTIVE,CLOSED— symbolic constants forstatemember ofDAWGobjectperfect_hashing– seeMinimal perfect hashingunicode– seeUnicode and bytesUnicode and bytesType of strings accepted and returned byDAWGmethods can be
eitherunicodeorbytes, depending on compile time
settings (preprocessor definitionDAWG_UNICODE). Value of
module memberunicodeinforms about chosen type.DAWGclassDAWGclass ispicklable, and also provide independent
way of marshaling with methodsbinload()andbindump().Propertystate[read-only integer]Following values are possible:pydawg.EMPTY— no words in a set;pydawg.ACTIVE— there is at least one word in a set,
and adding new words is possible (seeadd_word&add_word_unchecked);pydawg.CLOSED— there is at least one word in a set,
but adding new words is not allowed (seeclose/freeze).Basic mathodsadd_word(word) => boolAdd word, returns True if word didn’t exists in a set.
Procedure checks ifwordis greater then previously
added word (in lexicography order).add_word_unchecked(word) => boolDoes the same thing asadd_wordbut do not checkwordorder. Method should be used if one is sure, that input data
satisfy algorithm requirements, i.e. words order is valid.exists(word) => boolorword in ...Check if word is in set.match(word) => boolCheck if word or any of its prefix is in a set.longest_prefix(word) => intReturns length of the longest prefix of word that exists in a set.len()protocolReturns number of distinct words.words() => listReturns list of all words.find_all([word,[wildchar,[how]]])=> iteratorReturns iterator that match words depending onwordargument.find_all()does the same job asiter()find_all(prefix)Yields words that share a prefixfind_all(pattern, wildchar, [how])Yields words that match apatternwith givenwildchar(wildchar
matches any char). Parameterhowcontrols which words are matched:MATCH_EXACT_LENGTHwords with the same length as a patternMATCH_AT_LEAST_PREFIXwords of length not less then patternMATCH_AT_MOST_PREFIXwords of length no greater then patternclear()Erase all words from set.close()orfreeze()Don’t allow to add any new words,statevalue becomepydawg.CLOSED. Also free memory occupied by a hash table
used to perform incremental algorithm (see alsoget_hash_stats()).Can be reverted only byclear().IteratorClass supportsiterprotocol, i.e.iter(DAWGobject)returns
iterator, a lazy version ofwords()method.Minimal perfect hashingMinimalperfect hashing(MPH) allows to find unique number
representing any word from DAWG, and also find word with given number.
Numbers are in always in range 1 …len(DAWG).Finally, this feature makes possible to perform fast lookups as
in a regular dictionary.Algorithm used for MPH is described inApplications of Finite Automata
Representing Large Vocabularies,Claudio LucchesiandTomasz
Kowaltowski, Software Practice and Experience, 23(1), pp. 15–30, Jan.
1993.MPH feature is enabled during compilation time if preprocessor
definitionDAWG_PERFECT_HASHINGexists. Module memberperfect_hashingreflects this setting.WarningWords numbering is done for the whole DAWG. If new words
are added withadd_wordoradd_word_unchecked,
then current numbering is lost and when methodword2indexorindex2wordis called, then DAWG is renumbered.Because of that frequent mixing these two groups of method
will degrade performance.word2index(word) => indexReturns index of word, or None if word is not present in a DAWG.index2word(index) => wordReturns words associated with index, or None if index isn’t valid.ExampleD = pydawg.DAWG()

# fill DAWG with keys
for key in sorted(dict):
        D.add_word_unchecked(key)

# prepare values array
V = [None] * len(D)

for key, value in dict.items():
        index = D.word2index(key)
        assert index is not None

        V[index - 1] = value


# lookups are possible now
for word in user_input:
        index = D.word2index(word)
        if index is not None:
                print(word, "=>", V[index - 1])Otherdump() => (set of nodes, set of edges)Returns sets describing DAWG, elements are tuples.Node tuple:unique id of node (number)end of word markerEdge tuple:source node idedge label — letterdestination node idDistribution contains programdump2dot.pythat shows how to
convert output of this function tographvizDOT language.bindump() => bytesReturns binary DAWG data.binload(bytes)Restore DAWG from binary data. Example:import pydawg

A = pydawg.DAWG()
with open('dump', 'wb') as f:
        f.write(A.bindump())

B = pydawg.DAWG()
with open('dump', 'rb') as f:
        B.binload(f.read())get_stats() => dictReturns dictionary containing some statistics about
underlaying data structure:words_count— number of distinct words (same aslen(dawg))longest_word— length of the longest wordnodes_count— number of nodesedges_count— number of edgessizeof_node— size of single node (in bytes)sizeof_edge— size of single node (in bytes)graph_size— size of whole graph (in bytes); it’s aboutnodes_count * sizeof_node + edges_count * sizeof_edgeget_hash_stats() => dictReturns some statistics about hash table used by DAWG.table_size— number of table’s elementselement_size— size of single table itemitems_count— number of items saved in a tableitem_size— size of single itemApprox memory occupied by hash table istable_size * element_size + items_count * item_size.