Package: hades-nlp
HADES: Homologous Automated Document Exploration and SummarizationA powerful tool for comparing similarly structured documentsOverviewHADESis aPythonpackage for comparing similarly structured documents. HADES is designed to streamline the work of professionals dealing with large volumes of documents, such as policy documents, legal acts, and scientific papers. The tool employs a multi-step pipeline that begins with processing PDF documents using topic modeling, summarization, and analysis of the most important words for each topic. The process concludes with an interactive web app with visualizations that facilitate the comparison of the documents. HADES has the potential to significantly improve the productivity of professionals dealing with high volumes of documents, reducing the time and effort required to complete tasks related to comparative document analysis.InstallationLatest released version of theHADESpackage is available onPython Package Index (PyPI):Install spacyen-core-web-smoren-core-web-lgmodel for English language according to theinstructionsInstallHADESpackage using pip:pipinstall-Uhades-nlpThe source code and development version is currently hosted onGitHub.UsageTheHADESpackage is designed to be used in a Python environment. The package can be imported as follows:fromhades.data_loadingimportload_processed_datafromhades.topic_modelingimportModelOptimizer,save_data_for_app,set_openai_keyfrommy_documents_dataimportPARAGRAPHS,COMMON_WORDS,STOPWORDSTheload_processed_datafunction loads the documents to be processed. TheModelOptimizerclass is used to optimize the topic modeling process. Thesave_data_for_appfunction saves the data for the interactive web app. Theset_openai_keyfunction sets the OpenAI API key.my_documents_datacontains the informations about the documents to be processed. ThePARAGRAPHSvariable is a list of strings that represent the paragraphs of the documents. TheCOMMON_WORDSvariable is a list of strings that represent the most common words in the documents. TheSTOPWORDSvariable is a list of strings that represent the most common words in the documents that should be excluded from the analysis.First, the documents are loaded and processed:set_openai_key("my openai key")data_path="my/data/path"processed_df=load_processed_data(data_path=data_path,stop_words=STOPWORDS,id_column='country',flattened_by_col='my_column',)After the documents are loaded, the topic modeling process is optimized for each paragraph:model_optimizers=[]forparagraphinPARAGRAPHS:filter_dict={'paragraph':paragraph}model_optimizer=ModelOptimizer(processed_df,'country','section',filter_dict,"lda",COMMON_WORDS[paragraph],(3,6),alpha=100)model_optimizer.name_topics_automatically_gpt3()model_optimizers.append(model_optimizer)For each paragraph, theModelOptimizerclass is used to optimize the topic modeling process. Thename_topics_automatically_gpt3function automatically names the topics using the OpenAI GPT-3 API. User can also use thename_topics_manuallyfunction to manually name the topics.Finally, the data is saved for the interactive web app:save_data_for_app(model_optimizers,path='path/to/results',do_summaries=True)Thesave_data_for_appfunction saves the data for the interactive web app. Thedo_summariesparameter is set toTrueto generate summaries for each topic.When the data is saved, the interactive web app can be launched:hadesrun-app--configpath/to/results/config.json