The Basics of Regular Expressions | by SAURABH RANA | Towards Data Science

What are Regular Expressions?
Regular Expression is a special text string command which is used to match specific string sequences from huge chunks of data which if done manually by a person can take a lot of time. You can use regular expressions to match various patterns of string like:-
To extract all the email addresses
To extract only Gmail email addresses
To extract all the names starting with a specific alphabet and ending with a specific alphabet
To extract all the names where the first alphabet is capital
To extract all the numbers which have decimals points
To extract numbers in a specific range
And the list is endless, the above-mentioned cases are the first six scenarios that crossed my mind while writing this article but honestly you can use regular expressions to extract any kind of string pattern from data no matter how big it is within seconds and that is the beauty of regular expressions. These days the importance of regular expression has increased because many companies are using natural language processing techniques where regular expressions are used very frequently.
The best thing about regular expression is that it is supported by most of the popular programming languages therefore once you understand the syntax and concept of regular expressions, you can use it with various languages.
So let’s get started with regular expressions, I am using regular expressions with python but you guys can use any language of your choice but it should support regular expressions.
First is to match only the first time occurrence of a particular string pattern, second is to match all the occurrences of a particular string pattern.
Now whether you want to match only first occurrence or all the occurrence depends on your requirement. In my experience wherever I have used Regular Expressions most of the problems require all the occurrences to be matched therefore in all the examples in this blog I have intentionally written Regular expressions to match with all the occurrences.
Let’s get started with the most basic regex expression that is literal characters where you are basically searching a particular alphabet or a word from the data, it is similar to what you get when you search something on a web page or pdf using Ctrl+F. Let’s try one example:-
Since I am a big marvel fan so I will be using marvel as a reference in my example
Input = “Avatar did a business of 2.788 billion dollars($) where as Endgame has grossed 2.79 billions dollars($) till date”Regular Expression => “a”It will match with every single alphabet — “a”“Avatar did a business of 2.788 billion dollars($) where as Endgame has grossed 2.79 billions dollars($) till date”Output => a a a a a a a a a a
In output “a” is repeated 10 times because there are exactly ten occurrences of the alphabet “a”Now let’s try a combination of alphabets for our next example:-
Regular Expression => “as”Avatar did a business of 2.788 billion dollars($) where as Endgame has grossed 2.79 billion dollars($) till dateOutput => as as
Note: The sequence of alphabets is very important, in the above example only those stings will be matched where alphabet “a” is followed by the alphabet “s”.
“\w” stands for word character which basically is shorthand for [a-zA-Z0–9_] which means it matches with all capital alphabet, small alphabets and all the digits. Let’s try this regular expression in our marvel text to analyse which characters doesn’t match with “\w”:-
Text=”Avatar did a business of 2.788 billion dollars($) where as Endgame has grossed 2.79 billions dollars($) till date”Regular Expression => “\w”Output => “A v a t a r d i d a b u s i n e s s o f 2 7 8 8 b i l l i o n d o l l a r s w h e r e a s E n d g a m e h a s g r o s s e d 2 7 9 b i l l i o n s d o l l a r s t i l l d a t e”
As you can see the expression “\w” has matched with all the characters in the sentence except for [“$”,”(“,”)”,”.”]
Even though ”\w” is matching with all the characters in [a-zA-Z0–9_] but it matches with them one at a time because of which the output will display one alphabet after another, therefore I have used for loop in python to display them all together because of which there is space between every character that is matched.
Now let’s move onto the second example where we are searching for two consecutive characters
Regular Expression => “\w\w”Text=”Avatar did a business of 2.788 billion dollars($) where as Endgame has grossed 2.79 billions dollars($) till date”Note:It will match with two consecutive alphabet one at a timeOutput =>Av at ar di bu si ne ss of 78 bi ll io do ll
I have not copied the whole output but a part of it just to give you an idea
Note: Words which have an even number of characters are printed fully in groups of two but words where there are odd number of characters the last alphabet is missing in the output(Hint: When you divide an odd number by 2 the output is 1) because the last alphabet could not create a pair with another alphabet.
Similarly, we can search for three consecutive characters one at a time
Regular Expression => “\w\w\w”Text=”Avatar did a business of 2.788 billion dollar where as Endgame has grossed 2.79 billions dollars till date”Output => Ava tar did bus ine 788 bil lio dol lar
Instead of repeating “\w”, we can directly mention the number of consecutive alphabets we are searching for using curly({}) brackets about which we will discuss later
This shorthand character where “W” is capital will match every character except these [a-zA-Z0–9_]. So only those characters which were not matching in our previous example where we were using shorthand character class “\w” (w is small) will match in this case.
Text=”Avatar did a business of 2.788 billion dollar where as Endgame has grossed 2.79 billions dollars till date”Regular Expression => “\W”Output =>. ( $ ) . ( $ )
This shorthand character matches all numbers ([0–9]) in a source string.
Regular Expression => “\d”It will match with every single alphabet one at a time,unlike “\w” which matches with alphabets as well as digits,”\d” matches only with digitsText=”Avatar did a business of 2.788 billion dollars($) where as Endgame has grossed 2.79 billions dollars($) till date”Output =>2 7 8 8 2 7 9
Now let’s search for two consecutive numbers in source string.
Regular Expression =>” \d\d”It will match with two consecutive digits one at a timeAvatar did a business of 2.788 billion dollars($) \n where as Endgame has grossed 2.79 billions dollars($) till dateOutput =>78 79
Similar to “\W” (W is capital) this regular expression is also the complete opposite of its counterpart “\d” i.e. it will match only those characters which shorthand character “d” doesn’t matches.
.     -> Matches any single character except a line break[ ]   -> Matches any character contained between the square brackets[^ ]  -> Matches any character that is not contained between the square brackets*     -> Matches 0 or more repetitions of the preceding symbol+     -> Matches 1 or more repetitions of the preceding symbol?     -> Makes the preceding symbol optional.{n,m} -> Matches at least "n" but not more than "m" repetitions of the preceding symbol|     -> Matches either the characters before or the characters after the symbol\     -> Escapes the next character. This allows you to match     reserved characters { } . * + ? ^ $ \ |^     -> Matches the beginning of the input$     -> Matches the end of the input
Following meta characters +, * or ? are used to specify how many times a subpattern can occur. These meta characters act differently in different situations.
Meta character “+” is used to match with one or more than one occurrences of the preceding symbol.
For example:-
“\w+” Matches with strings which have one or more than one [a-zA-Z0–9_] character
“\d+” Matches with numbers which have one or more than one digits
a+ Matches with one or more than one occurrences of the alphabet “a”.We can use any alphabet in place of “a”.
Now let’s try out these examples on raw text
Regular Expression=”\w+”Text = “Avatar did a business of 2.788 billion dollars($) where as Endgame has grossed 2.79 billions dollars($) till date”Output =>Avatar did a business of 2 788 billion dollars where as Endgame has grossed 2 79 billions dollars till date
Now let’s try the “+” operator with “\d”
Regular Expression =”\d+”Text=”Avatar did a business of 2.788 billion dollar where as Endgame has grossed 2.79 billions dollars till date”Output => 2 788 2 79
Note: In the output “2” and “788” are not matched together because these two numbers are separated by “.”
Now, what if want the number“2.788” to be matched, in that case, we can use the following regular expression:-
Regular Expression => "\d+.\d+"Text=”Avatar did a business of 2.788 billion dollar where as Endgame has grossed 2.79 billions dollars till date”Output=> 2.788 2.79Regular Expression Breakdown=>Match one or more digit(\d+) + Match a full stop(.) + Match one or more digit(\d+)
Note: In complex regular expressions like the one mentioned above where we search for strings with multiple conditions, the conditions should match consecutively in the same order as mentioned in regular Expression, for instance in the example mentioned above we are searching for numbers which have one or more than one digits, followed by a full stop, followed by one or more than one digits. If these three conditions are not matched consecutively then there won’t be any match.
Exercise:Question 1) Find all gmail email id’s from given below data.Text =“abc@yahoo.com saurabh@outlook.com abc@gmail.com saurabh88@gmail.com saurabh11@outlook.com xyz@gmail.com”Solution:Regular Expression=”\w+@gmail.com”Output =>”abc@gmail.com saurabh88@gmail.com xyz@gmail.com”Question 2) Find all words in the text which are ending with “ing”Text=”dancing trek swimming sleeping jump running studying”Solution:Regular Expression=”\w+ing”Output => dancing swimming sleeping running studying
Unlike “+” operator which is used to match with one or more repetitions of preceding symbol,”*” is used to match with 0 or more repetitions of the preceding symbol.
For example:-
Suppose we are asked to find out of all the user id’s mentioned below only those which have string “saurabh” in it then, in that case, we will have to use “*”
Text =”saurabh13 kartik95 shreyashsingh ranasaurabh15 kumararindam yash_paryani saurabh_rana_22"Regular Expression = “\w*saurabh\w*”Output => saurabh13 ranasaurabh15 saurabh_rana_22
A string of characters enclosed in square brackets ([]) matches any one character in that string.Let’s try to understand it with the help of an example:-
Text=”ring king ping ding running”Regular Expression => “[rk]ing”Regular Expression Breakdown => Match with alphabet “r” or “k” + Match with “ing”Output => ring king
Note: As I mentioned before that in regular expressions the conditions should match consecutively in the same order as mentioned in Regular Expression. In the above example “running” is not matched even though it starts with “r” and ends with “ing” because these conditions are not matched consecutively since there are other alphabets between “r” and “ing” in running.
The special character “^” is used inside square brackets when we want to match all characters except for those mentioned in square brackets
Text=”ring king ping ding running”Regular Expression =>“‘[^rk]ing”Output => ping ding ning
We can use a hyphen(“-”)between two characters set to specify the characters range
Text=”ring king ping ding running”Regular Expression => “[a-z]ing”Output => ring king ping ding ning
The meta character “.” matches any single character except for return or newline characters
Input=”1_3@a!b+c de”Regular Expression => “.”Output =>1 _ 3  @ a !  b + c   d e
Here the regular expression is matching with every type of character including the whitespace between alphabet “d” and “e”
We use curly braces ({}) when we are very specific about the number of occurrences an operator or sub-expression must match in the source string. In the below-mentioned example, we are trying to derive the same output as we did in earlier examples where we wanted to match 3 consecutive digits for which we used regular expression “\d\d\d”, now instead of repeating the expression we can mention the specific number of occurrences that we want inside the curly brackets.
Text=”123 4643 6642 5 6545 8974 45 17 8659 1 34 88 654 564 786 4536 4 76 89 684568 "regular expression = “\d{3}”Output=>123 464 664 654 897 865 654 564 786 453
“?” in regex is used for making the previous group/character optional.
Let me explain the importance of this expression using an example=>
Suppose we are asked to extract all the numbers from the text given below, now in the given text there are three types of numbers:-
some numbers are whole numbers(without fraction)
some numbers which are greater than 1 and have decimal values
and some are less than “1” which will have decimal values
Now we want to extract all three types of numbers, let’s try it out
text=”2.55 300 153.54 .45 1234.1234 435 5.2435 .345"Regular expression = “\d*\.?\d*”Output = 2.55 300 153.54 .45 1234.1234 435 5.2435 .345Regular expression breakdown = Match 0 or more digits(\d*) + Match a decimal point on not(.?) + Match 0 or more digits(\d*)
The caret ^ and dollar $ characters are called “anchors” where the caret ^ matches at the beginning of the text, and the dollar $ at the end. They do not match any character at all. Instead, they match a position before, after, or between characters
Suppose in a data set we want to extract mobile numbers but there are many errors in the data set because of which there might be cases where the mobile number is not valid(Valid mobile number consists of 10 digits). In that case, anchors are very useful. Let’s try to understand this using various cases:-
Case-1Text=”9851245245s"Regular Expression=”^\d\d{8}\d$”Regular expression breakdown = First character of string should be a digit + match 8 digits + Last character of the string should be a digitsOutput= Null
Output is null because the number is not ending with a number which makes it an invalid mobile number.
Case-2Text=”98512f45245”Regular Expression=”^\d\d{8}\d$”Output= Null
Output is again null because there is an alphabet in between the numbers
Case-3Text=”985125245”Regular Expression=”^\d\d{8}\d$”Output= Null
This time there is no alphabet string in the source string but still, we are getting null output because the length of our source string is not equal to 10.
Case-4Text=”9851425245”Regular Expression=”^\d\d{8}\d$”Output= 9851425245
In this case, we are getting output because the source string full fills all the conditions of a valid mobile number as we mentioned in our regular expression.
So these are the most fundamental and important concepts of Regular expressions which I have tried to explain using some interesting examples, some of them were made up but most of them were actual problems that I came across while data cleaning so in future if you are stuck on a problem then just go through the examples once again and you might find the exact answer in one of the examples.
Apart from the basic regular expressions, you might come across regular expressions like this “ /^[a-z0–9_-]{6,18}$/”.When people see such long regular expression they just ignore them as if they have seen a Russian word in an English sentence. The trick to understanding the meaning of such regular expressions is to break them down and solve them separately.
If you are still not confident enough to understand such regular expression then please mention in the comment section, I may upload a second article on Regular Expression to explain how to solve such regular expression.