Text Extraction using Regular Expression (Python) | by KahEm Chu | Towards Data Science

“Regular Expression (RegEx) is one of the unsung successes in standardization in computer science,” [1].
In the example of my previous article, the regular expression is used to clean up the noise and perform tokenization to the text. Well, what we can do with RegEx in Text Analytics is far more than that. In this article, I am sharing how to use RegEx to extract the sentences which contain any keyword in a defined list from the text data or corpus. For example, you may want to extract the reviews made on features of a particular product, or you may which to extract all emails discussing urgent or critical subjects.
For me, the text documents I used in my text analytics project is huge, which take a long time if I going to train the model on my device. Also, I am not interested in every sentence in the datasets, hence text extraction is acting as a part of the data cleaning process for my project. Extracting only the sentences containing keywords defined not only able to reduce the vocab size, but also make the model more precise according to my need.
However, this method might not be suitable for every text analytics project. For example, if you are studying the overall sentiment of people toward a particular subject, object or event, extracting sentences might affect the accuracy of the overall sentiment. On another way round, this method is suitable if you are sure on what are the key matter you wish to study on. For example, you run an eCommerce that selling household products, you are curious about people feelings toward your new product, you may directly extract all the reviews containing the name or model of the new product.
As an example, the eCommerce store introduces a bar stool with a different height from the past product and would like to know how their customers react. The following highlighted sentence will be the output of text extraction with keywords ‘bar stool’ and ‘height’
Well, this introduction is longer than I expected. Let’s waste no more time and get started!
## read sentences and extract only line which contain the keywordsimport pandas as pdimport re# open filekeyword = open('keyword.txt', 'r', encoding = 'utf-8').readlines()texts = open('sent_token.txt', 'r', encoding = 'utf-8').readlines()# define function to read file and remove next line symboldef read_file(file):    texts = []    for word in file:        text = word.rstrip('\n')        texts.append(text)    return texts# save to variable        key = read_file(keyword)corpus = read_file(texts)
In the script above, the inputs are sentence tokens and the list of keywords stored in a text file. You may tokenize your dataset from documents into paragraphs or sentences, and then extract the paragraphs or sentences which contain the keywords. Sentence tokenization can be done easily with sent_tokenize from nltk.tokenize as below.
from nltk.tokenize import sent_tokenizetext = open('Input/data.txt', 'r', encoding = 'utf-8')text_file = open("Output/sent_token.txt", "w", encoding='utf-8')### This part to remove end line breakstring_without_line_breaks = ""for line in text:    stripped_line = line.rstrip() + " "    string_without_line_breaks += stripped_linesent_token = sent_tokenize(string_without_line_breaks)for word in sent_token:    text_file.write(word)    text_file.write("\n")text_file.close()
As the text data I used is extracted from a PDF file, there are a lot of line breaks, hence I will remove the line breaks before sentence tokenization.
# open file to write line which contain keywordsfile = open('Output/keyline.txt', 'w', encoding = 'utf-8') def write_file(file, keyword, corpus):        keyline = []          for line in corpus:        line = line.lower()        for key in keyword:            result = re.search(r"(^|[^a-z])" + key + r"([^a-z]|$)", line)            if result != None:                keypair = [key, line]                keyline.append(keypair)                 file.write(line + " ")                                                break                             else:                pass                                                          return(keyline)output = write_file(file,key,corpus)
The function above is the function I used to extract all the sentences which contain the keywords. A break is added to prevent copy the same line with multiple keywords to lower file size. The key script of doing so is just one line of code.
result = re.search(r”(^|[^a-z])” + key + r”([^a-z]|$)”, line)
The ”(^|[^a-z])” and ”([^a-z]|$)” surrounding the key is to make sure the word is identical with the keyword. For example, without these two part of the script, when we searching for a keyword like “act”, the result return might be the word “act” with prefix or suffix, such as “react” or “actor”.
# create DataFrame using data df = pd.DataFrame(output, columns =['Key', 'Line']) 
After extract the lines, you may create DataFrame from the keywords and the respective line extracted for further analysis. Note that in my example, I do not extract the same line again if it contains multiple keywords. If you need to do so, you may remove the break command from the script above.
That’s all on how to extract the line of sentences with keywords. Simple right?
Last but not least, sharing a script I have been using lately, os.listdir.startswith() and os.listdir.endswith() which help you to get all the file you need efficiently. This is sort of a similar concept with Regular Expression, where we define a particular pattern, and then we search for it.
# collect txt file with name start with 'data' import ospath = 'C:/Users/Dataset txt'folder = os.fsencode(path)filenames_list = []for file in os.listdir(folder):    filename = os.fsdecode(file)    if filename.startswith( ('data') ) and filename.endswith( ('.txt') ):         filenames1.append(filename)filenames_list.sort() 
If you are interested in text processing with NLTK and SpaCy: Text Processing in Python.
If you are interested in explore the gigantic text with Word Cloud: Text Exploration with Word Cloud.
Subscribe on YouTube
[1] D. Jurafsky and J. H. Martin, “Speech and Language Processing,” 3 December 2020. [Online]. Available: https://web.stanford.edu/~jurafsky/slp3/.
Congrats and thanks for reading to the end. Hope you enjoy this article. ☺️