How to Extract Keywords from PDFs and arrange in order of their weights using Python | by Prasad Patil | Towards Data Science

Keyword extraction is nothing but the task of identification of terms that best describe the subject of a document.
“Key phrases”, “key terms”, “key segments” or just “keywords” are the different nomenclatures often used for defining the terms that represent the most relevant information contained in the document. Although they may sound distinct,but they all serve the same purpose: characterization of the topic discussed in a document.
Extracting keywords is one of the most important tasks while working with text data in the domain of Text Mining, Information Retrieval and Natural Language Processing.
Readers benefit from keywords because they can judge more quickly whether the given text is worth reading or not.
Website makers benefit from keywords because they can group similar content by their topics.
Algorithm programmers benefit from keywords because they reduce the dimensionality of text to the most important features.
These are just few examples among many.
Let’s get down to the nitty-gritty of the topic.Up next follows a tutorial on how you can parse through a PDF file and convert it in to a list of keywords.We will learn and cement our understanding by taking a hands-on problem, so code along!
(I have used Python 2.7.15 version for this tutorial.)
You will need below mentioned libraries installed on your machine for the task.In case you don’t have it,I have inserted codes for each dependency in code block below, which you can type it on command prompt for windows or on terminal for mac operating system.
PyPDF2 (To convert simple, text-based PDF files into text readable by Python)
pip install PyPDF2
textract (To convert non-trivial, scanned PDF files into text readable by Python)
pip install textract
re (To find keywords)
pip install regex
Note : I have attempted three approaches for this task.Above libraries would be suffice for approach 1.However I have just touched upon two other approaches which I found online.Treat them as alternatives. Down below is the jupyter notebook with all three approaches.Take a look!
All necessary remarks are denoted with ‘#’.
Approach 1 unboxed
Step 1: Import all libraries.
Step 2: Convert PDF file to txt format and read data.
Step 3: Use “.findall()” function of regular expressions to extract keywords.
Step 4: Save list of extracted keywords in a DataFrame.
Step 5 : Apply concept of TF-IDF for calculating weights of each keyword.
Step 6 : Save results in a DataFrame and use “.sort_values()” to arrange keywords in order.
import pandas as pd
import numpy as np
import PyPDF2
import textract
import re

converted PDF file to txt format for better pre-processing
filename ='JavaBasics-notes.pdf' 

pdfFileObj = open(filename,'rb')               #open allows you to read the file
pdfReader = PyPDF2.PdfFileReader(pdfFileObj)   #The pdfReader variable is a readable object that will be parsed
num_pages = pdfReader.numPages                 #discerning the number of pages will allow us to parse through all the pages


count = 0
text = ""
                                                            
while count < num_pages:                       #The while loop will read each page
    pageObj = pdfReader.getPage(count)
    count +=1
    text += pageObj.extractText()
    
#Below if statement exists to check if the above library returned #words. It's done because PyPDF2 cannot read scanned files.

if text != "":
    text = text
    
#If the above returns as False, we run the OCR library textract to #convert scanned/image based PDF files into text

else:
    text = textract.process('http://bit.ly/epo_keyword_extraction_document', method='tesseract', language='eng')

    # Now we have a text variable which contains all the text derived from our PDF file.

text = text.encode('ascii','ignore').lower() #Lowercasing each word

keywords = re.findall(r'[a-zA-Z]\w+',text)
len(keywords)                               #Total keywords in document

3410
df = pd.DataFrame(list(set(keywords)),columns=['keywords'])  #Dataframe with unique keywords to avoid repetition in rows

In information retrieval, tf–idf or TFIDF, short for term frequency–inverse document frequency, is a numerical statistic that is intended to reflect how important a word is to a document in a collection or corpus.It is often used as a weighting factor in searches of information retrieval, text mining, and user modeling. 
TF: Term Frequency, which measures how frequently a term occurs in a document. Since every document is different in length, it is possible that a term would appear much more times in long documents than shorter ones. Thus, the term frequency is often divided by the document length (aka. the total number of terms in the document) as a way of normalization: 
TF(t) = (Number of times term t appears in a document) / (Total number of terms in the document).
IDF: Inverse Document Frequency, which measures how important a term is. While computing TF, all terms are considered equally important. However it is known that certain terms, such as "is", "of", and "that", may appear a lot of times but have little importance. Thus we need to weigh down the frequent terms while scale up the rare ones, by computing the following: 
IDF(t) = log_e(Total number of documents / Number of documents with term t in it).
def weightage(word,text,number_of_documents=1):
    word_list = re.findall(word,text)
    number_of_times_word_appeared =len(word_list)
    tf = number_of_times_word_appeared/float(len(text))
    idf = np.log((number_of_documents)/float(number_of_times_word_appeared))
    tf_idf = tf*idf
    return number_of_times_word_appeared,tf,idf ,tf_idf    

df['number_of_times_word_appeared'] = df['keywords'].apply(lambda x: weightage(x,text)[0])
df['tf'] = df['keywords'].apply(lambda x: weightage(x,text)[1])
df['idf'] = df['keywords'].apply(lambda x: weightage(x,text)[2])
df['tf_idf'] = df['keywords'].apply(lambda x: weightage(x,text)[3])

df = df.sort_values('tf_idf',ascending=True)
df.to_csv('Keywords.csv')
df.head(25)

from gensim.summarization import keywords
import warnings
warnings.filterwarnings("ignore")

RequestsDependencyWarning: urllib3 (1.22) or chardet (2.3.0) doesn't match a supported version! [__init__.py:80]
UserWarning: detected Windows; aliasing chunkize to chunkize_serial [utils.py:1197]

values = keywords(text=text,split='\n',scores=True)

data = pd.DataFrame(values,columns=['keyword','score'])
data = data.sort_values('score',ascending=False)
data.head(10)

from rake_nltk import Rake

r = Rake()
r.extract_keywords_from_text(text)

phrases = r.get_ranked_phrases_with_scores()

table = pd.DataFrame(phrases,columns=['score','Phrase'])
table = table.sort_values('score',ascending=False)
# table.head(10)

For more details find GitHub repo HERE !
References :
www.wikipedia.org
www.wikipedia.org
2. Medium post for PDF to Text Conversion
3. keyword extraction tutorial
4. Regular expressions
I hope you find this tutorial fruitful and worth reading. Also,I am sure there must be tons of other approaches with which you can perform the said task.Do share them in comment section if you have came across any.
Code for the Masked Word Cloud :
Find GitHub repo HERE !
# modules for generating the word cloud from os import path, getcwd from PIL import Image import numpy as np import matplotlib.pyplot as plt from wordcloud import WordCloud%matplotlib inline d = getcwd()  text = open('nlp.txt','r').read()#Image link = 'https://produto.mercadolivre.com.br/MLB-693994282-adesivo-decorativo-de-parede-batman-rosto-e-simbolo-grande-_JM'  mask = np.array(Image.open(path.join(d, "batman.jpg"))) wc = WordCloud(background_color="black",max_words=3000,mask=mask,\               max_font_size=30,min_font_size=0.00000001,\               random_state=42,)    wc.generate(text) plt.figure(figsize=[100,80]) plt.imshow(wc, interpolation="bilinear") plt.axis("off") plt.savefig('bat_wordcloud.jpg',bbox_inches='tight',pad_inches=0.3);
Read Other Articles I Published -
How to create a PDF report from Excel using PythonFinding the favorite team in 2018 FIFA World Cup through scraping TweetsWhat is Exploratory Data Analysis?
How to create a PDF report from Excel using Python
Finding the favorite team in 2018 FIFA World Cup through scraping Tweets
What is Exploratory Data Analysis?