Pattern-based recognition did help in NLP | by Edward Ma | Towards Data Science

Nowadays, everyone talks about machine learning and deep learning. Definitely, we enjoy the result from deep learning but it is extremely hungry for data. So how can we build the initialise data set? We can allocate a group of resource on labelling data but it is very expensive. Alternatively, we always use regular expression (for text) in the initialise stage to understand data and building handcraft feature for validation and kick-off the project.
Regular expression is a fundamental skill in software engineering. It exists in front-end development (e.g. Javascript), backend development (e.g. Java). How about Data Science? It is incredibly useful in NLP area and allowing us to retrieve huge amount of valuable information from unstructured text.
In this article, I will focus on how we can leverage regular expression in data exploration, preprocessing and feature engineering stage rather than the regular expression implementation. After reading this article, you will understand:
What is regular expression?
How can we leverage regular expression in data science life cycle?
Take Away
The concept was initialized by Stephen in the 1950s. The objective of regular expression (aka regex or regexp) is identifying expected pattern by given text which can include character, symbol and number. You may visit here for detail history.
Given an sentence, we want to extract a date. How can we achieve this?
sentence = "Jul 29 is the 210th day of the year"pattern = r'((Jan|Feb|Mar|Apr|May|Jun|Jul|Aug|Sep|Oct|Nov|Dec) [0-9]{2})'results = re.findall(pattern, sentence)print('Result:', results)
Output
Result: [('Jul 29', 'Jul')]
It supports other patterns including “any digit”, “any word character”. For detail, you can check out python official page or searching in google or stackoverflow.
Data Exploration
When you kick-off a project, the first step is exploring what you have. For numeric feature, we can do some basic statistical calculation such as min, max , average. For categorical feature, we can number of possible value and distribution. How about text? Besides calculating unique tokens (e.g. word) and the highest frequency word, you want to explore more feature from unstructured text.
Understand Data: Given a news, you want to know whether news includes event’s date and showing the distribution whether or not including date. It will be very good if the data source come with it. However, it will not happen most of the time.
Validate Hypothesis: In previous project, we believe that weight of product is a valuable signal. However, we do not know whether unstructured text include this information. So we use regular expression to validate it. Luckily, there is well-defined format such as mg, milligram, g, kg etc. We can build a new feature easily by using regular expression.
Preprocessing
We cannot get clean data or the time (unlike Kaggle competition). So we have to clean raw data. Of course, people believe that deep learning can learn everything and reducing the weight of those meaningless. From my experience, better quality data leads better result.
Getting data via web crawler system. It can be HTML format and we parse raw data by leveraging “Beautiful Soap”. We still need to use regular expression to clean unwanted data
Getting unexpected data via OCR. Assuming original input is English only but OCR recognize it incorrect and introducing non English character. It can be filter out by easily.
Feature Engineering
After confirming patterns (e.g. date, weight etc), we can use regular expression to dig it out from unstructured data. Extracting correct “date” can be very challenge task. Here is some of the example
Different format: 01/01/2018, 01 Jan 2018, January 1, 2018 etc.
Different interpretation: How do you interpret 08/01/2018? If you are in Hong Kong, it should be eighth day of January but it is the first day of August in USA.
In general, there are 3 possible scenarios:
Well-defined Format: Date, email and phone number are well-defined format. Just google it and you can find reliable regular expression.
Custom Regular Expression: Most of the time, we need to build custom regular expression. In system log analysis project, one of the target is extracting domain key from log (given that domain key is persisted into log file). Domain key can be system generated 15 digit numbers or combing both digit and character.
No format: For example, you want to get product name such as iMac, Goolge Home etc. There is no standard format so we may not able to extract it by regular expression.
Again, regular expression is an essential and fundamental skill if you work in NLP area. It is an advantage if you can master it but at least you have to implement some simple regular expression. Regular expression is a pre-defined rule base extraction method. It:
may request lots of time to implement a common pattern. As mentioned, regular expression is a rule base system, we need to build custom regular expression piece by piece.
cannot support unseen pattern. It is a rule base and implying that it is unable to handle new pattern.
cannot guarantee cater all scenario. Even email, we cannot guarantee 100% success.
I am Data Scientist in Bay Area. Focusing on state-of-the-art in Data Science, Artificial Intelligence , especially in NLP and platform related.