Package: pdf2embeddings
NLP tool for scraping text from a corpus of PDF files, embedding the sentences in the text and finding semantically similar sentences to a given search queryThe code in this repository performs 3 main tasks.Scraping the text from a corpus of PDF files. The text is then cleaned, split into sentences, and saved into a pd.DataFrame, .csv or .parquet file containing 3 columns. One column contains the text of all the PDFs in the corpus (one sentence per row), the second column contains the title of the PDF where each sentence is taken from, and the third column contains the number of the page where each sentence is located within that PDF. This enables easy lookup.Embedding all the scraped sentences in the corpus of PDFs using three different NLP models: Word2Vec (with the option to include Tf-Idf weights), ELMo and BERT. For each model, sentence-level embeddings are generated.Corpus querying. This is in the form of a search tool, where the user can input a search query (one to a few words), and the tool will output the most similar sentences in the PDF corpus to the user query. This is done by comparing the embedding of the user query against all the embeddings of each sentence in the scraped corpus of PDFs. This effectively acts as a search engine. It is important that the model used to embed the user's search query matches the NLP model used to embed the PDF corpus. The default similarity metric is cosine similarity, although this can be changed by the user.Project Organization├── Makefile           <- Makefile with commands like `make data` or `make train`
├── README.md          <- The top-level README for developers using this project.
├── .envrc             <- The file containing the set up for environment variables (required if using the runner 
│                         scripts). `$PWD` should correspond to the directory where you clone this repository.
├── .gitignore         <- The files (including data) which are not uploaded to GitHub. Edit as required.
├── data
│   ├── external       <- Data from third party sources.
│   ├── interim        <- Intermediate data that has been transformed.
│   ├── processed      <- The final, canonical data sets for modeling. This is where you cleaned datasets will be saved.
│   └── raw            <- The original, immutable data dump.
│       └── pdfs       <- Where your PDF files to be scraped are stored. The other data folders are there if you need
│                         to do other cleaning tasks, but are not used in this project as provided.
│
├── models             <- Trained and serialized models. This is where your NLP models will be saved. No models have 
│                         been uploaded to GitHub.
│
├── requirements.txt   <- The requirements file for reproducing the analysis environment.
│
├── setup.py           <- Makes project pip installable (pip install -e .) so pdf2embeddings can be imported
│
├── MANIFEST.in        <- Tells setup.py which package data to include and exclude.
│
├── config             <- This folders stores configuration files (for example suggested filenames for saving 
│   │                     specific objects) that are read in by some of the runner scripts. Edit as required.
│   ├── filenames.json            
│   └── words_to_replace.json
│
├── scripts            <- Executable scripts are saved here. They should be run in the order listed below.
│   ├── data_processing_runner.py   
│   ├── embeddings_runner.py
│   ├── user_search_runner.py
│   └── tmp            <- The folder where the loggers are saved (for example, debug.log, info.log, warning.log)
│                         Logs have not been uploaded to GitHub.
│
├── pdf2embeddings     <- Source code for use in this project. See description below for how to use the files.
│   ├── __init__.py    <- Makes pdf2embeddings a Python module
│   ├── arrange_text.py
│   ├── embedder.py
│   ├── json_creator.py
│   ├── logging.yaml
│   ├── process_user_queries.py
│   └── scraper.py
│
├── tests              <- Unit tests for all functions and methods defined in all modules within the pdf2embeddings folder, to                
│   │                     be run using pytest. It also includes an end-to-end test. These should not be modified by
│   │                     the user.  
│   ├── conftest.py
│   ├── end_to_end_test.py
│   ├── test_arrange_text.py
│   ├── test_embedder.py
│   ├── test_process_user_queries.py
│   ├── test_scraper.py
│   └── fixtures       <- This folder contains all the pytest fixtures required to run the tests. These should not
│       │                 be modified by the user.                
│       ├── dummy_embeddings.npy
│       ├── dummy_sentences.csv
│       ├── dummy_sentences.parquet
│       ├── dummy_sentences.txt
│       ├── expected_bert_embeddings.npy
│       ├── expected_elmo_embeddings.npy
│       ├──  expected_tfidf_scores.json
│       ├── expected_w2v_embeddings_tfidf_false.npy
│       ├── expected_w2v_embeddings_tfidf_true.npy
│       ├── full_df_with_embeddings.parquet.gzip
│       ├── test_pdf_1.pdf
│       ├── test_pdf_2.pdf
│       ├── tfidf_vectorizer.pickle
│       ├── word2vec.pickle
│       ├── word2vec_tfidf.pickle
│       └── words_to_replace.json
│
└── test_environment.pyClone this repository from GitHub, or install this project by running$ pip install pdf2embeddingson the terminal command (orpip3, as appropriate).If you have cloned it from GitHub, you can run a test to ensure your environment is properly set-up. This project has
not been tested on versions of Python older than 3.6, and some versions of thenumpylibrary older than 1.17 are also known
to cause issues. Please run the following line in your terminal$ PYTHONHASHSEED=123 python3 -m pytestThere should be 32 tests. If they all pass, you're good to start using this package. If some of the tests fail, please
check your environment. This project has only been tested with the environment as described in therequirements.txtfile. Note that the environment variablePYTHONHASHSEEDmust be set to"123"while running the tests, to ensure deterministic reproducibility of the Word2Vec
models. Two tests will fail if this is not set up correctly.Each module has been fully documented.Before you start, please configure your environment variables according to your own directory path. Please refer to
the.envrcfile, where$PWDcorresponds to the directory path where this repository has been cloned.In order to scrape the text from a corpus of PDF files, you will need to save your PDFs in the folder (~/data/raw/pdfs).Alternatively to scraping files from local storage, this package also supports cloud storage on AWS S3 buckets only
(other cloud storage solutions are not natively supported).You can make use of thedata_processing_runner.pyscript to scrape the PDFs, clean the text, split all the text into
sentences,  and save this into a .csv file. The script imports the two modulesscraperandarrange_text.importosimportyamlimportlogging.configfrompdf2embeddings.scraperimportDocumentScraperfrompdf2embeddings.arrange_textimportCorpusGeneratorif__name__=="__main__":DATA_DIR=os.getenv('DATA_DIR')CONFIG_DIR=os.getenv('CONFIG_DIR')LOGGING_CONFIG=os.getenv('LOGGING_CONFIG')withopen(LOGGING_CONFIG,'r')asf:config=yaml.safe_load(f)logging.config.dictConfig(config)pdfs_folder=os.path.join(DATA_DIR,'raw','pdfs')json_path=os.path.join(CONFIG_DIR,'words_to_replace.json')scraper=DocumentScraper(pdfs_folder,json_path)df_by_page=scraper.document_corpus_to_pandas_df()generator=CorpusGenerator(df_by_page)df_by_sentence=generator.df_by_page_to_df_by_sentence()df_by_page.to_csv(os.path.join(DATA_DIR,'processed','corpus_by_page.csv'))# optional, for referencedf_by_sentence.to_csv(os.path.join(DATA_DIR,'processed','corpus_by_sentence.csv'),index=False)The filewords_to_replace.jsonin theconfigfolder is used for ad-hoc text cleaning.If using an AWS S3 bucket for storing PDFs, please setfrom_s3_bucket=True, so thescraperwill bescraper=DocumentScraper(pdfs_folder,json_path,from_s3_bucket=True)If you choose this option, thepdfs_foldercan either start with"s3://"or omit this prefix. Please ensure you already have the
correct credentials for accessing the S3 bucket you are pointing to, and that the path is correct.When runningscraper.document_corpus_to_pandas_df(), the json is deserialised into a python dictionary, and the corpus text will be
cleaned by replacing each key in this dictionary with its value. In order to modify and customize the content of this
json file, run the scriptpdf2embeddings/json_creator.pyand adapt it as necessary.Once you have created a filecorpus_by_sentence.csv, you can embed the sentences in this file using your model of choice out of
Word2Vec (with the option to include Tf-Idf weights), ELMo and BERT. For each model, sentence-level embeddings are
generated. Where the original model would generate word-level embeddings, sentence-level embeddings have been created by
averaging all the word embeddings of the respective sentence. Theembeddings_runner.pyscript is an example of how you
could run all 4 NLP models and save them separately. It imports theembeddermodule.importosimportjsonimportyamlimportlogging.configimportpandasaspdfrompdf2embeddings.embedderimportEmbeddermodels_to_be_run=['Word2Vec_tfidf_weighted',# comment out as needed'Word2Vec','BERT','ELMo',]if__name__=='__main__':DATA_DIR=os.getenv('DATA_DIR')MODELS_DIR=os.getenv('MODELS_DIR')CONFIG_DIR=os.getenv('CONFIG_DIR')LOGGING_CONFIG=os.getenv('LOGGING_CONFIG')withopen(LOGGING_CONFIG,'r')asf:config=yaml.safe_load(f)logging.config.dictConfig(config)withopen(os.path.join(CONFIG_DIR,'filenames.json'),'r')asf:file_names=json.load(f)corpus_filename="corpus_by_sentence.csv"corpus_by_sentence=pd.read_csv(os.path.join(DATA_DIR,"processed",corpus_filename))list_of_sentences=corpus_by_sentence['sentence'].values.tolist()print("Instantiating Embedder class.")embedder=Embedder(list_of_sentences)formodelinmodels_to_be_run:print(f"Calculating{model}embeddings.")ifmodel=='Word2Vec_tfidf_weighted':sentence_embeddings,model_obj,tfidf_vectorizer=embedder.compute_word2vec_embeddings(tfidf_weights=True)embedder.save_model(tfidf_vectorizer,MODELS_DIR,file_names[model]['vectorizer_filename'])# the line above is specific to Word2Vec with TfIdf vectorizer and cannot be generalized to other modelselifmodel=='Word2Vec':sentence_embeddings,model_obj,_=embedder.compute_word2vec_embeddings(tfidf_weights=False)elifmodel=='BERT':bert_model='bert-base-nli-stsb-mean-tokens'# This line is specific to BERTsentence_embeddings,model_obj=embedder.compute_bert_embeddings(bert_model)elifmodel=='ELMo':sentence_embeddings,model_obj=embedder.compute_elmo_embeddings()else:raiseKeyError(f'The model{model}is not recognized as input.')print(f"{model}embeddings calculated. Saving model.")embedder.save_embeddings(sentence_embeddings,MODELS_DIR,file_names[model]['embeddings_filename'])embedder.save_model(model_obj,MODELS_DIR,file_names[model]['model_filename'])print(f"{model}model saved. Saving .parquet file.")df=embedder.add_embeddings_to_corpus_df(os.path.join(DATA_DIR,"processed",corpus_filename),sentence_embeddings,file_names[model]['column_name'])embedder.df_to_parquet(df,os.path.join(DATA_DIR,"processed",file_names[model]['parquet_filename']))print(f"Parquet file saved. All steps done for the{model}model.")Each model has been saved as a.picklefile in themodelsfolder, each model's embeddings as a.npyfile in themodelsfolder, and each pd.DataFrame as a.parquetfile in thedata/processedfolder. Each.parquetfile
contains the same data as thecorpus_by_sentence.csvfile previously saved, with an added column, representing the sentence embeddings for the
chosen model. A separate.parquethas been saved for each model, although the user may modify the script above to save
all models' embeddings in the same.parquetfile. The file names of the.pickle,.npyand.parquetfiles are
stored in thefilenames.jsonin theconfigfolder. In order to modify and customize these names, run the scriptpdf2embeddings/json_creator.pyand adapt it as necessary.Finally, in order to search through your corpus of PDF files given auser search query(which can be a single word or a
few words), run theuser_search_runner.pyscript in thescriptsfolder, which imports theprocess_user_queriesmodule:importosimportyamlimportjsonimportlogging.configfrompdf2embeddings.process_user_queriesimportquery_embeddingsif__name__=='__main__':user_search_input='cell phone'model_name='BERT'# change as appropriateDATA_DIR=os.getenv("DATA_DIR")CONFIG_DIR=os.getenv('CONFIG_DIR')MODELS_DIR=os.getenv("MODELS_DIR")LOGGING_CONFIG=os.getenv("LOGGING_CONFIG")withopen(LOGGING_CONFIG,'r')asf:config=yaml.safe_load(f)logging.config.dictConfig(config)withopen(os.path.join(CONFIG_DIR,'filenames.json'),'r')asf:file_names=json.load(f)tfidf_vectorizer=os.path.join(MODELS_DIR,"tfidf_vectorizer.pickle")model=os.path.join(MODELS_DIR,file_names[model_name]["model_filename"])# this is optional for ELMo and BERT.trained_df_path=os.path.join(DATA_DIR,'processed',file_names[model_name]["parquet_filename"])user_input_embedding,trained_df=query_embeddings(user_search_input,trained_df_path,file_names[model_name]["column_name"],model_name,model,distance_metric='cosine',tfidf_vectorizer=tfidf_vectorizer)# tfidf_vectorizer is not used (and optional) when model is not 'Word2Vec_TfIdf_weighted'ifuser_input_embedding.sizeandnottrained_df.empty:# they must not be emptyprint(trained_df.sort_values('metric_distance',ascending=True)[['sentence','metric_distance']].reset_index(drop=True).head(10))At this point, theuser_input_embeddingis the embedding of the user search query, andtrained_dfis the
pd.DataFrame containing a column with the metric distance between the user embedding and each individual sentence
embedding in the corpus (default metric: cosine similarity). If you want to visualise the most similar sentences to the
user search query, you can simply sort the pd.DataFrame by itsmetric_distancecolumn.print(trained_df.sort_values('metric_distance',ascending=True)[['sentence','metric_distance']].reset_index(drop=True)Project description adapted from thecookiecutter data science project template. #cookiecutterdatascience