Package: lexicons-builder
Thelexicons_builderpackage aims to provide a basic API to create lexicons related to specific words.Key principle: Given the input words, it will look for synonyms or neighboors in the dictionnaries or in the NLP model. For each of the new retreiven terms, it will look again for its neighboors or synonyms and so on..The general method is implemented on 3 different supports:Synonyms dictionnaries (See list of the dictionnaries : ref:here <list_dictionnaries.rst>)NLP language modelsWordNet(orWOLF)Output can be text file, xlsx file, turtle file or a Graph object. See <Quickstart> section for examples.Full documentation available onreadthedocsNoteFeel free to raise an issue onGitHubif something isn’t working for you.InstallationWith pipIt is recommanded to use a virtual environment.$python-mvenvenv$sourceenv/bin/activate$pipinstalllexicons-builderFrom sourceTo install the module from source:$pipinstallgit+git://github.com/GuillaumeLNB/lexicons_builderDownload NLP models (optionnal)Here’s a non exhaustive list of websites where you can download NLP models manually.
The models should be in word2vec or fasttext format.LinkLanguage(s)https://fauconnier.github.io/#dataFrenchhttps://wikipedia2vec.github.io/wikipedia2vec/pretrained/Multilingualhttp://vectors.nlpl.eu/repository/Multilingualhttps://github.com/alexandres/lexvec#pre-trained-vectorsMultilingualhttps://fasttext.cc/docs/en/english-vectors.htmlEnglish / Multilingualhttps://github.com/mmihaltz/word2vec-GoogleNews-vectorsEnglishDownload wordnet>>>importnltk>>>nltk.download()Download WOLF (French WordNet) (optionnal)$# download WOLF (French wordnet if needed)$wgethttps://gforge.inria.fr/frs/download.php/file/33496/wolf-1.0b4.xml.bz2$# (and extract it)$bzip2-dwolf-1.0b4.xml.bz2QuickStartCommand line interface (CLI)To get words from input words through CLI, run$python-mlexicons_builder<words>\--lang<LANG>\--out-file<OUTFILE>\--format<FORMAT>\--depth<DEPTH>\--nlp-model<NLP_MODEL_PATHS>\--web\--wordnet\--wolf-path<WOLF_PATH>\--strictWith:<words>The word(s) we want to get synonyms from<LANG>The word language (eg:fr,en,nl, …)<DEPTH>The depth we want to dig in the models, websites, …<OUTFILE>The file where the results will be stored<FORMAT>The wanted output format (txt with indentation, ttl or xlsx)At least ONE of the following options is needed:--nlp-model<NLP_MODEL_PATHS>The path to the nlp model(s)--webSearch online for synonyms--wordnetSearch on WordNet using nltk--wolf-path<WOLF_PATH>The path to WOLF (French wordnet)Optional--strictremove non relevant wordsEg:if we want to look for related terms linked to ‘eat’ and ‘drink’ on wordnet at a depth of 2, excecute:$python-mlexicons_buildereatdrink\--langen\--out-filetest_en.txt\--formattxt\--depth1\--wordnet$Notetheindentationislinkedtothedepthawhichthewordwasfound$headtest_en.txtdrinkeatabsorbadeaerophagiaalcoholalcoholic_beveragealcoholic_drinkbanquetbar_hopbelt_downbeveragebi...PythonTo get related terms interactively through Python, run>>>fromlexicons_builderimportbuild_lexicon>>># search for related terms of 'book' and 'read' in English at depth 1 online>>>output=build_lexicon(["book","read"],'en',1,web=True)...>>># we then get a graph object>>># output as a list>>>output.to_list()['PS','accept','accommodate','according to the rules','account book','accountability','accountancy','accountant','accounting','accounts','accuse','acquire','act','adjudge','admit','adopt','afl','agree','aim',"al-qur'an",'album','allege','allocate','allow','analyse','analyze','annuaire','anthology','appear in reading','apply','appropriate','arrange','arrange for','arrest','articulate','ascertain'...>>># output as rdf/turtle>>>print(output)@prefixns1:<http://taxref.mnhn.fr/lod/property/>.@prefixns2:<urn:default:baseUri:#> .@prefixns3:<http://www.w3.org/2004/02/skos/core#> .@prefixxsd:<http://www.w3.org/2001/XMLSchema#> .ns2:PSns1:isSynonymOfns2:root_word_uri;ns3:prefLabel"PS";ns2:comesFrom<synonyms.com>;ns2:depth1.ns2:acceptns1:isSynonymOfns2:root_word_uri;ns3:prefLabel"accept";ns2:comesFrom<synonyms.com>;ns2:depth1....>>># output to an indented file>>>output.to_text_file("filename.txt")>>>withopen("filename.txt")asf:...print(f.read(1000))...readbookPSacceptaccommodateaccordingtotherulesaccountbookaccountability...>>># output to xslx file>>>output.to_xlsx_file("results.xlsx")>>># full search with 2 nlp models, wordnet and on the web>>># download and extract google word2vec model>>># from https://github.com/mmihaltz/word2vec-GoogleNews-vectors>>>>>># download and extract FastText models>>># from https://fasttext.cc/docs/en/english-vectors.html>>>>>>nlp_models=["GoogleNews-vectors-negative300.bin","wiki-news-300d-1M.vec"]>>>output=build_lexicon(["book","letter"],"en",1,web=True,wordnet=True,nlp_model_paths=nlp_models)>>># can take a while>>>len(output.to_list())614>>># deleting non relevant words>>>output.pop_non_relevant_words()>>>len(output.to_list())57NoteIf the depth parameter is too high (higher than 3), the words found could seem unrelated to the root words. It can take also a long time to compute too.NoteThe word senses are taken equally, which means that you might get terms you would think are not related to the input word.
Eg: looking for the word ‘test’ might give you words linked to Sea urchins, as a ‘test’ is also a type of shellhttps://en.wikipedia.org/wiki/Test_(biology).