Package: deepsearch-glm
Graph Language ModelsGetting StartedFinding entities and relations via NLP on text and documentsTo get easily started, simply install thedeepsearch-glmpackage from PyPi. This can be
done using the traditionalpip install deepsearch-glmor via poetrypoetry add deepsearch-glm.Below, you can find the code-snippet to process pieces of text,fromdeepsearch_glm.utils.load_pretrained_modelsimportload_pretrained_nlp_modelsfromdeepsearch_glm.nlp_utilsimportinit_nlp_model,print_on_shellload_pretrained_nlp_models(force=False,verbose=False)mdl=init_nlp_model()# from Wikipedia (https://en.wikipedia.org/wiki/France)text="""France (French: [fʁɑ̃s] Listen), officially the French Republic(French: République française [ʁepyblik fʁɑ̃sɛz]),[14] is a countrylocated primarily in Western Europe. It also includes overseas regionsand territories in the Americas and the Atlantic, Pacific and IndianOceans,[XII] giving it one of the largest discontiguous exclusiveeconomic zones in the world."""res=mdl.apply_on_text(text)print_on_shell(text,res)The last command will print the pandas dataframes on the shell and provides the
following output,text:#France (French: [fʁɑ̃s] Listen), officially the French Republic(French:Républiquefrançaise[ʁepyblikfʁɑ̃sɛz]),[14]isacountry
locatedprimarilyinWesternEurope.Italsoincludesoverseasregions
andterritoriesintheAmericasandtheAtlantic,PacificandIndian
Oceans,givingitoneofthelargestdiscontiguousexclusiveeconomic
zonesintheworld.

properties:typelabelconfidence0languageen0.897559

instances:typesubtypesubj_pathchar_ichar_joriginal
-------------------------------------------------------------------------------------------------------------------------------
sentence#                   1       180  France (French: [fʁɑ̃s] Listen), officially the French Republic(French:Républiquefrançaise[ʁepyblikfʁɑ̃sɛz]),[14]isacountrylocatedprimarilyinWesternEurope.
termsingle-term#                   1         8  #Franceexpressionwtoken-concatenation#                   1         8  #Franceparenthesisroundbrackets#                   9        36  (French: [fʁɑ̃s] Listen)expressionwtoken-concatenation#                  18        28  [fʁɑ̃s]termsingle-term#                  29        35  Listentermsingle-term#                  53        68  French Republicparenthesisroundbrackets#                  69       125  (French: République française [ʁepyblik fʁɑ̃sɛz])termsingle-term#                  78       100  République françaisetermsingle-term#                 112       124  fʁɑ̃sɛz]parenthesisreference#                 126       130  [14]numvalival#                 127       129  14termsingle-term#                 136       143  countrytermsingle-term#                 165       179  Western Europesentence#                 181       373  It also includes overseas regions and territories in the Americas andtheAtlantic,PacificandIndianOceans,givingitoneofthelargestdiscontiguousexclusiveeconomiczonesintheworld.
termsingle-term#                 198       214  overseas regionstermenum-term-mark-3#                 207       230  regions and territoriestermsingle-term#                 219       230  territoriestermsingle-term#                 238       246  Americastermenum-term-mark-4#                 255       290  Atlantic, Pacific and Indian Oceanstermsingle-term#                 255       263  Atlantictermsingle-term#                 265       272  Pacifictermsingle-term#                 277       290  Indian Oceanstermsingle-term#                 313       359  largest discontiguous exclusive economic zonestermsingle-term#                 367       372  worldThe NLP can also be applied on entire documents which were converted using
Deep Search. A simple example is shown below,fromdeepsearch_glm.utils.load_pretrained_modelsimportload_pretrained_nlp_modelsfromdeepsearch_glm.nlp_utilsimportinit_nlp_model,print_on_shellload_pretrained_nlp_models(force=False,verbose=False)mdl=init_nlp_model()withopen("<path-to-json-file-of-converted-pdf-doc>","r")asfr:doc=json.load(fr)enriched_doc=mdl.apply_on_doc(doc)Creating Graphs from NLP entities and relations in document collectionsTo create graphs, you need two ingredients, namely,a collection of text or documentsa set of NLP models that provide entities and relationsBelow is a code snippet to create the graph using these basic ingredients,odir="<ouput-dir-to-save-graph>"json_files=["json-file of converted PDF document"]model_names="<list of NLP models:langauge;term;verb;abbreviation>"glm=create_glm_from_docs(odir,json_files,model_names)Querying GraphsTBDInstall for developmentPython installationTo use the python interface, first make sure all dependencies are installed. We usepoetryfor that. To install all the dependent python packages and get the python bindings, simply execute,poetryinstallCXX compilationTo compile from scratch, simply run the following command in thedeepsearch-glmroot folder to
create thebuilddirectory,cmake-B./build;Next, compile the code from scratch,cmake--build./build-jRun using the Python InterfaceNLP and GLM examplesTo run the examples, simply do execute the scripts aspoetry run python <script> <input>. Examples are,apply NLP on document(s)poetryrunpython./deepsearch_glm/nlp_apply_on_docs.py--pdf'./data/documents/articles/2305.*.pdf'--models'language;term'analyse NLP on document(s)poetryrunpython./deepsearch_glm/nlp_apply_on_docs.py--json'./data/documents/articles/2305.*.nlp.json'create GLM from document(s)poetryrunpython./deepsearch_glm/glm_create_from_docs.py--pdf./data/documents/reports/2022-ibm-annual-report.pdfDeep Search utilitiesQuery and download document(s)poetryrunpython./deepsearch_glm/utils/ds_query.py--indexpatent-uspto--query"\"global warming potential\" AND \"etching\""Converting PDF document(s) into JSONpoetryrunpython./deepsearch_glm/utils/ds_convert.py--pdf'./data/documents/articles/2305.*.pdf'"Run using CXX executablesIf you like to be bare-bones, you can also use the executables for NLP and GLM's directly. In general, we
follow a simple scheme of the form./nlp.exe-m<mode>-c<JSON-configfile>
./glm.exe-m<mode>-c<JSON-configfile>In both cases, the modes can be queried directly via the-hor--help./nlp.exe-h
./glm.exe-hand the configuration files can be generated,./nlp.exe-mcreate-configs
./glm.exe-mcreate-configsNatural Language Processing (NLP)After you have generated the configuration files (see above), you cantrain simple NLP models./nlp.exe-mtrain-cnlp_train_config.jsonleverage pre-trained models./nlp.exe-mpredict-cnlp_predict.example.jsonGraph Language Models (GLM)create a GLM./glm.exe-mcreate-cglm_config_create.jsonexplore interactively the GLM./glm.exe-mexplore-cglm_config_explore.jsonTestingTo run the tests, simply execute (after installation),poetryrunpytest./tests-vvv-s