Mastering Regular Expressions made incredibly easy - It's all about the patterns
By: Günter Röhrich
Tags: Regex, Python, Computer Science, Data Science, Analytics

The number of use cases for which Regular Expressions (Regex) can be applied on is immense. No matter whether you are a data scientist collection data, white collar worker automating business processes or student who aims at extracting data from academic journals on a larger scale, Regex is likely to become your best friend.

Think of the world wide web is an incredibly comprehensive source of (structured, semi or rather unstructured) data, very often interesting facts and numbers are kept on a web page and you may find yourself in a situation where you would like to access this data. In most cases the web page, software provider, etc. is not offering a simple API to access or download data (for free), this is why you might want to find a tailored solution to obtain data yourself.

It’s all about the pattern © me

This is how we will address Regex in this quick read

I often found readings covering Regex with short unrelated examples and explanations which were quite hard to follow. I will try to approach this topic differently:

Provide a very short and handy Regex Cheat Sheet Have a quick look at groups and how they perfectly enhance our patterns Definition of a Problem rather than the description of a given patterns, e.g. “As a user, I want to find all data until the dot “.” is reached

In order to give you an impression of a couple simple regex examples, we will browse through a colourful mixture of use cases — and no matter where we heading to, Regex will allow us to master all our requests.

Why are we doing this? Data analytics and further reaching data science disciplines heavily rely on accurate and clean data, and because of this, cleaning and organising as well as collecting data roughly contributes to 80% of the data science profession! A proper understanding of regex will make your day way easier.

Covering most of the data science daily business — Forbes

You will find an exhaustive supply of cheat sheets online — I prefer to just have the most common characters ready when needed. If you feel proficient with those, you should not have a hard time feeling comfortable writing patterns:

Thanks to cheatography

We will focus on two examples:

The history of tea (tea_text) A football results table

Before we start, let’s first have a look at our cheat sheet below, followed by a short introduction to groups.

Groups ( )

Groups are essential when it comes to work with (fractions of the) pattern results. I often avoided to use groups, however this made it very hard to extract relevant data only. Groups are defined through using ‘(‘ and ‘)’ in the pattern. Groups allow to explicitly target a group of characters, even if the group is embedded in a another pattern.

Programming languages then allow to query for these specific groups and address them directly. To outline the general purpose of groups, follow this example and make sure you got the idea behind:

“This is the day you will always remember as the day you almost caught Captain Jack Sparrow”

“This is the day you will always remember as the day you have not caught Captain Jack Sparrow”

“This is the day you will always remember as the day you have caught Captain Jack Sparrow”

“This is the day you will always remember as the day you certainly have caught Captain Jack Sparrow”

It seems Jack Sparrow, apologies, Captain Jack Sparrow gets more or less caught on a regular basis, but we want to know which exact words are used. So we define the relevant words in our pattern through a group.

Through our pattern ‘you\s(\w*\s*?\w*)\scaught’ we are able to extract values that lie in between our determined pattern.

Depending on the programming language you use (I will use Python), the syntax to implement Regex varies, however the idea is always the same. If your pattern finds a match, you are able to extract the entire match but also the single group(s) only. Convenient!

Now, being equipped with group knowledge, let’s go through the examples.

First, let’s start with the tea text.

Chinese small-leaf-type tea was introduced into India in 1836 by the British in an attempt to break the Chinese monopoly on tea.[57] In 1841, Archibald Campbell brought seeds of Chinese tea from the Kumaun region and experimented with planting tea in Darjeeling. The Alubari tea garden was opened in 1856 and Darjeeling tea began to be produced.[58] In 1848, Robert Fortune was sent by the East India Company on a mission to China to bring the tea plant back to Great Britain. He began his journey in high secrecy as his mission occurred in the lull between the Anglo-Chinese First Opium War (1839–1842) and Second Opium War (1856–1860).[59] .....

[57] Tea was originally consumed only by anglicized Indians; however, it became widely popular in India in the 1950s because of a successful advertising campaign by the India Tea Board.[57]

Assume we would like to extract the annotations from the tea excerpt. For this purpose we use the group logic, to only extract the values, not the brackets — however we consider brackets to be necessary to identify the annotation rather than receiving any sort of single or double digit.

pattern = “\[(\d{1,2})\]”

annotations = re.findall(pattern,tea_text)

[‘57’, ‘58’, ‘59’, ‘57’, ‘57’]

If we ignored the usage of groups, we would obtain the annotations as they occur in the text — surrounded by brackets.

pattern = “\[\d{1,2}\]”

annotations = re.findall(pattern,tea_text)

['[57]', '[58]', '[59]', '[57]', '[57]']

It may be required to extract all time spans, hence we do not want to extract e.g. 1836 but we would like to obtain 1839–1842.

pattern = “\(\d*.\d*\)” # . could also be explicitly '-'

years = re.findall(pattern,tea_text)

[’(1839–1842)’, '(1856–1860)’]

More specifically, we might want to extract the years for the time spans we have already been able to locate. For this purpose we will greatly rely on the group idea previously outlined.

pattern = “\((\d*).(\d*)\)” # year range

first_years = re.search(pattern,tea_text)

first_years.group(0) # '(1839–1842)'

first_years.group(1) # '1839'

first_years.group(2) # '1842'

first_years.groups() # ('1839', '1842') ## Iterating over all year spans is the easy part:

single_years = []

for match in re.finditer(pattern,tea_info):

if match:

for i in range(len(match.groups())):

single_years.append(match.group(i+1)) single_years # ['1839', '1842', '1856', '1860']

Replacing values

Regex is not only a good way to find patterns, but also to replace them. Depending on the language you use, regex can be used to substitute patterns through your desired string. This is particularly useful, when you would like to avoid replacing to many values that may affect other words, e.g. ‘is’ to ‘was’ replacement may result in words like Thwas…

pattern = "(First.*\(\d{4}.\d{4}\))\s*and.(Second.*\(\d{4}.\d{4}\))" # using \s and . to show that results are the same here # Provides the following elements:

[('First Opium War (1839–1842)', 'Second Opium War (1856–1860)')] pattern = "First\sOpium\sWar"

replace_word = "1st Opium War"

re.sub(pattern,replace_word, tea_text) # inplace modification ! #[..] between the Anglo-Chinese 1st Opium War (1839–1842) and Second Opium War (1856–1860).

Counting values

Very often the number of occurrences is an interesting fact. The problem that may arise is, that values may be a mix of upper and lower case combinations, this makes it very hard to bring those values together.

pattern = “[Cc]hinese.(?=[tT]ea)” # positive lookahead

occ = len(re.findall(pattern,tea_text))

# normally we would use re.IGNORECASE to achieve this result

print(‘There are {} occurrences of "Chinese Tea"’.format(occ)) # There are 3 occurrences of "Chinese Tea"

In the second example, the score of the some last football games is captured as a string. The intention is to be able to extract teams and the goals they scored, half and full time. A potential use case could be, that a final table shall be calculated.

Given to following game results:

game_day = """

Juventus F.C. - Napoli 4:1 (2:0)

A.C. Milan - Internazionale F.C. 2:2 (1:0)

A.C. Fiorentina - Torino 1:0 (0:0)

Lazio - Atalanta 0:3 (0:2)

Lazio - Juventus F.C. 0:4 (0:2)

"""

There are many ways to extract team names and scores, punctuation blanks and special characters however make it more complex to find a one-suits-all solution.

Think of these two patterns — or come up with a better one!

# A more generic approach

pattern = "

([A-Za-z][\w\.\s]*)-

([a-zA-Z\.\s]*)

(\d):(\d)\s*

\((\d):(\d)" # A very tailored approach

pattern = "(\w*\s?\w\.\w\.|\w\.\w\.\s*?\w*|\w*) # note the OR |

\s*-\s*

(\w*\s?\w\.\w\.|\w\.\w\.\s*?\w*|\w*)\s*

(\d):(\d)\s*\

((\d):(\d)"

The above statements may seem a complex at the first sight, but, it really isn’t. I used two ways to pretty much achieve the same result, however one pattern intends to reflect the exact pattern as indicated in our example data, the other follows a far more generic approach.

Before focusing on the letters brackets and numbers, let’s have a first look at the overall structure of the pattern.

As indicated through ‘(‘ and ‘)’ there are several groups that allow us to extract and address data from every single group. This is particularly handy, if we only wanted to use the name and score of one team, or only the half time score, etc. We further see ‘-’ and ‘:’ which give us a clear idea of the regex structure in terms of the data given.

Pattern 1 (group 1 and 2):

#generic:

[a-zA-Z0–9\.\s]* #or more precise:

(\w*\s?\w\.\w\.|\w\.\w\.\s*?\w*|\w*)\s*

The brackets provide us with the letters and characters we allow, these can be letters and dots. In the other case we refer to any word character (\w), explicit dots (\.) and blanks(\s).

Then first approach [a-zA-Z0–9\.\s]* is great to read, however has one caveat, trailing blanks will be included in our results — so make sure your programme can handle these. The second approach is much more specific and perfectly covers our test data points — the caveat here, make sure there are no teams that may not be covered by the pattern.

Note that we separate the patterns in the second approach through OR (|) which is necessary due to the fact, that clubs do not have F.C.’s or A.C.’s as part of their name. Although “uglier”, I prefer the more specific version.

The next group is simply the same pattern again, as we are looking for football teams again.

Pattern 2 - the score:

(\d):(\d)\s*\ # note: groups!

Simply queries for decimals (numbers), so for the end score. We have not considered limitations to the number of occurrences, but in order to avoid issues, we could also use a quantifier:

(\d{,2}):(\d{,2})\s*\

# we don’t expect any team to score higher than 99 goals a game. Pattern 3:

\((\d):(\d) # Just repeats the second pattern, however excludes

# the first bracket ‘\(‘.

In order to provide an idea of what could be done with the extracted values, I created a brief “table calculator” below. This calculator uses the entered teams and scores and assigns their points.

If we used this regex on the all given matches, we could eventually calculate the entire league table — just an idea:

There is a lot more to discover, I hope these few lines could spark your interest into regex.

See you next time, stay safe!

#Regex #Python #Data Science #6040 #CS6040