Manipulate PDF Files, Extract Information from Text Files | Towards Data Science

Undoubtedly, modern technology has made our life easy. Even we can not think of a day without the usages of modern techniques and technologies. Natural language processing is one of the most used technologies in the recent time. Basically, natural language processing plays a significant role in our daily communication. If we come to know about the insight into the technology and can implement the technologies with our own it will be a great pleasure for us.
At the very beginning, we should know about the various techniques of playing with text files. If you are not familiar with it, I would recommend you to read my previous article.
towardsdatascience.com
Then we need to know about extracting text information from the text files like pdf or any other formats. But in this article we will discuss about exploring the pdf documents with PyPDF2 library. Another most important tools for extracting information from a text file is regular expression. With regular expression, we can easily get our desired information like phone number, address, email and many more. Specially, regular expression is the combination of some symbols and alphabets for having our desired information from a hundreds of thousands of text data.
[N.B. — For your convenience, I have linked the full source code of the article at the end of the conclusion.]
The article has written to intend to cover the prerequisite of NLP with python. The following topics will be cover in the following article.
Part-I
Overview of PyPDF2
Reading PDF files
Adding text to the PDF
Grab all text from PDF
Part-II
Overview of Python re library
Searching basic pattern
Complex pattern overview
Pattern search using quantifiers
Group with re library
or operator
Wildcard character
Starts and ends with character
Scattering the character form a string
Removing the punctuations from a string
Brackets for grouping
Parentheses for multiple option
Practice problem
Let’s start.....................
The following part is all about the PyPDF2 library of python for working with PDF files.
Overview of PyPDF2 library
Overview of PyPDF2 library
Often you will have to deal with PDF files. There are many libraries in Python for working with PDFs, each with their pros and cons, the most common one being PyPDF2. You can install it with (note the case-sensitivity, you need to make sure your capitalization matches):
pip install PyPDF2
Keep in mind that not every PDF file can be read with this library. PDFs that are too blurry, have a special encoding, encrypted, or maybe just created with a particular program that doesn’t work well with PyPDF2 won’t be able to be read. If you find yourself in this situation, try using the libraries linked above, but keep in mind, these may also not work. The reason for this is because of the many different parameters for a PDF and how non-standard the settings can be, text could be shown as an image instead of a utf-8 encoding. There are many parameters to consider in this aspect.
As far as PyPDF2 is concerned, it can only read the text from a PDF document, it won’t be able to grab images or other media files from a PDF.
2. Reading PDF files
First of all need to import the library PyPDF2 as follows
# note the capitalizationimport PyPDF2
Now, we open a pdf, then create a reader object for it. Notice how we use the binary method of reading , ‘rb’, instead of just ‘r’.
# Notice we read it as a binary with 'rb'f = open('US_Declaration.pdf','rb')
Read more why ‘rb’ is used instead of ‘r’
Here, the file ‘US_Declaration.pdf’ is located in the same directory of the jupyter notebook file location.
You may download the US_Declaration.pdf file from the here.
It’s time to read the text form the page. The following piece of codes will help us to read form the pages of pdf.
#pdfFileReader() reads the text  form the pdfpdf_reader = PyPDF2.PdfFileReader(f) #the following lines of code will output the number of pages of the pdfpdf_reader.numPages#getPage()reads the text of a specific page. Here the parameter 0 indicates the first page of the pdfpage_one = pdf_reader.getPage(0)page_one_text = page_one.extractText()#Finally the extractText() extracts the the texts in a text format of page 1. 
If you run the above code and want to see what page_one_text variable holds, you will find the following output.
3. Adding text to a pdf
We can not write to PDFs using Python because of the differences between the single string type of Python, and the variety of fonts, placements, and other parameters that a PDF could have.
What we can do is copy pages and append pages to the end.
We are going to provide an example for adding text to a new pdf file. It’s easy.
f = open('US_Declaration.pdf','rb')pdf_reader = PyPDF2.PdfFileReader(f)first_page = pdf_reader.getPage(0)#pdfFileWriter() enables to create a pagepdf_writer = PyPDF2.PdfFileWriter()#addPage() adds the content of first_page to a new pagepdf_writer.addPage(first_page)#The open() function with "wb" mode creates a document named "Some_New_Doc.pdf" to the directorypdf_output = open("Some_New_Doc.pdf","wb")#finally we will get a pdf file with the contents of the first page of the previous pdf by write() functionpdf_writer.write(pdf_output)
Done! Now we have copied a page and added it to another new document!
4. Grab all text from a pdf file
From the above codes, we come to know that with PyPDF2, we can only read the text from a specific page at a time. But how can we get all the text of a pdf at a time? Is there any solution? Yeah! obviously, a simple loop can solve the issue. Let’s see in practical.
f = open('US_Declaration.pdf','rb')# List of every page's text.# The index will correspond to the page number.pdf_text = [0]  # zero is a placehoder to make page 1 = index 1pdf_reader = PyPDF2.PdfFileReader(f)for p in range(pdf_reader.numPages):        page = pdf_reader.getPage(p)        pdf_text.append(page.extractText())
The peace of code will help to hold all the text into a list object. Now, we can easily print out the page content by inserting the index to the list like print(pdf_text[1]) . It will print the text of page 2.
[N.B. — It is always a good practice to close a document by using f.close() command.]
Overview of regular expression
Overview of regular expression
Regular Expressions (sometimes called regex for short) allow a user to search for strings using almost any sort of rule they can come up with. For example, finding all capital letters in a string, or finding a phone number in a document.
Regular expressions are notorious for their seemingly strange syntax. This strange syntax is a by product of their flexibility. Regular expressions have to be able to filter out any string pattern you can imagine, which is why they have a complex string pattern format.
Regular expressions are handled using Python’s built-in re library. See the docs for more information.
Let’s begin by explaining how to search for basic patterns in a string!
2. Searching Basic Pattern
Let’s imagine that we have the following string:
text = "The agent's phone number is 408-555-1234. Call soon!"
We’ll start off by trying to find out if the string “phone” is inside the text string. Now we could quickly do this with:
'phone' in text
It will return the True as it is found in the statement.
But let’s show the format for regular expressions, because later on we will be searching for patterns that won’t have such a simple solution. Look at a simple example with the python regular expression library re .
Firstly,
import re
Now,
pattern = 'phone're.search(pattern,text)
Output
<_sre.SRE_Match object; span=(12, 17), match='phone'>
It indicates that the phone is matched with the string of the variable text and it exists 12 to 17 index of the string. Another example can be as follows.
pattern = "NOT IN TEXT"re.search(pattern,text)
It won’t return anything as no match found. Now we’ve seen that re.search() will take the pattern, scan the text, and then returns a Match object. If no pattern is found, a None is returned .
Let’s take a closer look at this Match object. The span match.span() has start and end index. If we run the code match.start() , it will output 12. On the other hand, match.end() output the end index of matching string which is 17.
But what if the pattern occurs more than once?
text = "my phone is a new phone"match = re.search("phone",text)
It will return the output <re.Match object; span=(3, 8), match=’phone’> . Notice only first match is found though there are 2 matches of the string phone.
If we wanted a list of all matches, we can use .findall() method:
matches = re.findall("phone",text)
Output
['phone', 'phone']
If you wanted the actual text that matched, you can use the .group() method.
match.group()
Output
'phone'
3. Overview of complex pattern
So far we’ve learned how to search for a basic string. What about more complex examples? Such as trying to find a telephone number in a large string of text? Or an email address?
We could just use search method if we know the exact phone or email, but what if we don’t know it? We may know the general format, and we can use that along with regular expressions to search the document for strings that match a particular pattern.
This is where the syntax may appear strange at first, but take your time with this; often it’s just a matter of looking up the pattern code.
Let’s begin!
4. Identifiers for Characters in Patterns
Characters such as a digit or a single string have different codes that represent them. You can use these to build up a pattern string. Notice how these make heavy use of the backwards slash \ . Because of this when defining a pattern string for regular expression we use the format:
r'mypattern'
placing the r in front of the string allows python to understand that the \ in the pattern string are not meant to be escape slashes.
Below you can find a table of all the possible identifiers:
You may have a look at the table while working because it is not possible to memorize the symbols.
Look at the following code
text = "My telephone number is 408-555-1234"phone = re.search(r'\d\d\d-\d\d\d-\d\d\d\d',text)phone.group() #there are three groups separated with (-) symbol
Output
'408-555-1234'
Notice the repetition of \d. That is a bit of an annoyance, especially if we are looking for very long strings of numbers. And the quantifiers has made the task easy.
5. Pattern search using quantifiers
Let’s explore the possible quantifiers.
Now, we know the special character designations, we can use them along with quantifiers to define how many we expect. We needn’t use the identifiers repeatedly.
Let’s rewrite our pattern using these quantifiers:
text = "My telephone number is 408-555-1234"phone = re.search(r'\d{3}-\d{3}-\d{4}',text)phone.group()
The output of the code is as same as the previous one. But now it seems easy and can be used easily for complex and big pattern.
6. Group with re library
What if we wanted to do two tasks, find phone numbers, but also be able to quickly extract their area code (the first three digits) of the above example. We can use groups for any general task that involves grouping together regular expressions (so that we can later break them down).
Using the phone number example, we can separate groups of regular expressions using parentheses:
# The entire resultresults.group()
Output
'408-555-1234'
Then
# Can also call by group position.# remember groups were separated by parentheses ()# Something to note is that group ordering starts at 1. Passing in 0 returns everythingresults.group(1)
The piece of code returns the first group (3 digits of the the telephone number).
7. or operator
Use the pipe operator to have an or statment. For example
re.search(r"man|woman","This man was here.")
Output
<_sre.SRE_Match object; span=(5, 8), match='man'>
Again..
re.search(r"man|woman","This woman was here.")
Output
<_sre.SRE_Match object; span=(5, 10), match='woman'>
8. Wild card character
Use a “wildcard” as a placement that will match any character placed there. You can use a simple period . for this. For example:
re.findall(r".at","The cat in the hat sat here.")
Output
['cat', 'hat', 'sat']
Suppose, we want to match the first 3 letters.
re.findall(r"...at","The bat went splat")
It generates the output like [‘e bat’, ‘splat’].Notice how we only matched the first 3 letters, that is because we need a . for each wildcard letter. Or use the quantifiers described above to set its own rules.
If we want words that end with “at”, what should we do?
# One or more non-whitespace that ends with 'at're.findall(r'\S+at',"The bat went splat")
The above code has done the job and return output [‘bat’, ‘splat’] .
9. Starting and Ending Character
We can use the ^ to find starts with character, and the $ to ends with character:
# Ends with a numberre.findall(r'\d$','This ends with a number 2')# Starts with a numberre.findall(r'^\d','1 is the loneliest number.')
The above code returns [‘2’] ,[‘1’] as end and start character of the string.
Note that this is for the entire string, not for individual words!
10. Scattering the characters from string
To exclude characters, we can use the ^symbol in conjunction with a set of brackets [] .Anything inside the brackets is excluded. For example:
phrase = "there are 3 numbers 34 inside 5 this sentence."re.findall(r'[^\d]',phrase)
Output
['t', 'h', 'e', 'r', 'e'....]
11. Removing punctuation marks
test_phrase = 'This is a string! But it has punctuation. How can we remove it?'re.findall('[^!.? ]+',test_phrase)
It will return only the words.
['This', 'is', 'a', 'string', 'But', 'it', 'has', 'punctuation', 'How', 'can', 'we', 'remove', 'it']
We may join the word serially as follows.
clean = ' '.join(re.findall('[^!.? ]+',test_phrase))clean
It will return the string without punctuation marks.
'This is a string But it has punctuation How can we remove it'
12. Brackets for Grouping
As showed above, we can use brackets to group together options, for example if we wanted to find hyphenated words:
text = 'Only find the hypen-words in this sentence. But you do not know how long-ish they are're.findall(r'[\w]+-[\w]+',text)
Output
['hypen-words', 'long-ish']
13. Parentheses for Multiple Options
If we have multiple options for matching, we can use parentheses to list out these options. For Example:
# Find words that start with cat and end with one of these options: 'fish','nap', or 'claw'text = 'Hello, would you like some catfish?'re.search(r'cat(fish|nap|claw)',text)
Output
<_sre.SRE_Match object; span=(27, 34), match='catfish'>
14. Practice problem
It’s bonus task. I have upload a jupyter notebook file to the github. You may download the files form here and fill up the cells according to the instruction. There are six tasks. Try to do with your own. I think it’s easy and interesting. For any problem or confusion just comment below.
We have learned a lot, I think. Though the topics and syntaxes are easy, these are so important for Natural Language Processing (NLP) and other data science tasks. I have started these article as a series of NLP. From the next article we will learn the core basics of NLP. Stay connected with me.
Complete Jupyter notebook of part one is given below
github.com
Find the full regular expression source code is given below
github.com
If you enjoy the article, follow me on medium for more.
Connect me on LinkedIn for collaboration.