All you need to know about Regular Expressions | by Ria Kulshrestha | Towards Data Science

Regular expressions are a generalized way to match patterns with sequences of characters. They define a search pattern, mainly for use in pattern matching with strings, or string matching, i.e. “find and replace” like operations.
Let’s say you have to search a corpus and return all the email addresses mentioned in it. A pretty straightforward task for a human as every email address can be generalized as a string of alphanumeric characters with some permissible special characters like . (dot), _(underscore), -(hyphen), followed by a ‘@{domainName}.com’.
But how do you pass such a template to a computer? The fact that email addresses can be of variable length, don’t have a fixed location for numeric/special characters (could be in the beginning, the end, or anywhere in between), doesn’t make the task any easier for a computer.
This is where regular expressions come to our rescue! By using them we can easily create a generalized template that a not only a computer will understand but will satisfy all of our restrictions.
The term regular expression comes from mathematics and computer science theory, where it reflects a trait of mathematical expressions called regularity. The text patterns used by the earliest grep tools were regular expressions in the mathematical sense. Though the name has stuck, modern-day Perl-style regular expressions (which we still use with other programming languages) are not regular expressions at all in the mathematical sense.
They are often called REs, or regexes, or regex patterns.
#1: Most characters match with themselves. So if you have to write a regular expression for the word “wubbaLubbaDubDub”, the regex for it would be ‘wubbaLubbaDubDub’. Keep in mind that REs are case-sensitive.
But there are some characters called meta-characters which don’t match to themselves rather signal that some out-of-the-ordinary thing should be matched, or they affect other portions of the RE by repeating them or changing their meaning. These meta-characters make regular expressions the powerful tool they are.
#2: Repeaters *,+,{}. A lot of time you’ll face a scenario where you don't know how many times a character will be repeated or if it will be repeated at all. Repeaters allow us to tackle such cases.
The asterisk (*): It implies that the preceding character is present 0 or more times. There is no upper bound on the number of occurrences of the preceding character.
In ‘a*b’, ‘a’ precedes the *, which means ‘a’ can occur 0/0+ times. So it will match to ‘b’, ‘ab’, ‘aab’, ‘aaab’, ‘aa..(infinite a’s)..b’.
The plus(+): It is the same as * but implies at least 1 occurrence of the previous character.
‘a+b’ will match to ‘ab’, ‘aab’, ‘aaab’, ‘aaa...aab’ but not just ‘b’.
The curly braces {}: These are used to define a range of occurrences of the previous character.
‘a{3},b’: will match ‘aaab’ only. ‘a{min,}b’: restricting minimum occurrences of ‘a’, no upper limit restrictions. ‘a{3,}b’ will match ‘aaab’, ‘aaaab’, ‘aaa...aab’.‘a{min, max}b’: ‘a’ should occur at least min times and at most max times. ‘a{3,5}b’ will match ‘aaab’, ‘aaaab’, ‘aaaaab’.
#3 The wildcard(.): Knowing the order of characters in a string is not always feasible. A wildcard comes in really handy in these situations.It implies that any character can take its place in the sequence.
‘.’ will match all strings with just one character.‘.*’ will match all possible strings of any length.‘a.b’ will match ‘aab’, ‘abb’, ‘acb’, ‘adb’, .... ‘a b’ [a(space)b], ‘a/b’ and so on. Basically, any sequence of length 3 starting with ‘a’ and ending with ‘b’.
#4 The Optional character(?): Sometimes multiple variations of a word are possible. Like ‘color’ (American English) and ‘colour’ (British English) are different spellings of the same word and both are correct, the only difference being an extra ‘u’ in the British version. The ‘?’ implies that the previous character may or may not be present in the final string. In a way, ‘?’ implies a 0 or 1 occurrence.
‘colou?r’ will match both ‘color’ and ‘colour’.‘docx?’ will match both ‘doc’ and ‘docx’.
#5 The Caret(^): It implies the first character of a string.
‘^a.*’ will match all strings starting with ‘a’.‘^a{2}.*’ will match all strings starting with ‘aa’.
#6 The dollar($): It implies the last character of a string.
‘.*b$’ will match all strings ending with a ‘b’.
#7 Character classes([]): More often than not, for a particular position in a string, there will be more than one possible character. To accommodate all the possible characters we use character classes. They specify a set of characters that you wish to match. Characters can be listed individually, or a range of characters can be indicated by giving two characters and separating them by a ‘-’.
‘[abc]’: will match ‘a’, ‘b’, or ‘c’. ‘[^abc]’:Negation will match any character except ‘a’, ‘b’, and ‘c’. Note -- Not to be confused with caret where ^ implies begins with. If inside a character class ^ implies negation.Character range: ‘[a-zA-Z]’ will match any character from a-z or A-Z. 
\d matches any decimal digit; this is equivalent to the class [0–9].\D Matches any non-digit character; this is equivalent to the class [^0–9]. \s Matches any whitespace character; this is equivalent to the class[\t\n\r\f\v].\S Matches any non-whitespace character; this is equivalent to the class[^\t\n\r\f\v].\w Matches any alphanumeric character; this is equivalent to the class [a-zA-Z0–9].\W Matches any non-alphanumeric character; this is equivalent to the class [^a-zA-Z0–9].
#9 Grouping character: A set of different symbols of a regular expression can be grouped together to act as a single unit and behave as a block, for this, you need to wrap the regular expression in the parenthesis( ).
‘(ab)+’ will match ‘ab’, ‘abab’,’abab...’.‘^(th).*” will match all string starting with ‘th’.
#10 Vertical Bar ( | ): Matches any one element separated by the (|).
th(e|is|at) will match words - the, this and that.
#11 The Escape character (\): What if you want to match a meta-character with itself i.e match ‘*’ with a ‘*’ and not use it is a wild card?You can do so by adding a backslash( \ ) before that character. This will allow special characters to be used without invoking their special meaning.
\d+[\+-x\*]\d+ will match patterns like "2+2" and "3*9" in          "(2+2)*3*9".
In order to write a regex for any email address, we need to keep in mind the following constraints:
It can only start with an alphabet. ^([a-zA-Z])
It is of variable length and can contain any alphanumeric character along with ‘.’, ‘_’, ‘-’. ([a-zA-Z0–9_\-\.]*)
It should have a ‘@’ followed by a domain name. @([a-zA-Z0–9_\-\.]+)
It should end with a .(dot) something, usually 2/3 character long. \.([a-zA-Z]){2,3}$
Our final regular expression for an email address :^([a-zA-Z])([a-zA-Z0–9_\-\.]*)@([a-zA-Z0–9_\-\.]+)\.([a-zA-Z]){2,3}$
Regexes are useful in a wide variety of text processing tasks, and more generally string processing, where the data need not be textual. Common applications include data validation, data scraping (especially web scraping), data wrangling, simple parsing, the production of syntax highlighting systems, and many other tasks.
While regexes would be useful on Internet search engines, processing them across the entire database could consume excessive computer resources depending on the complexity and design of the regex. Although in many cases system administrators can run regex-based queries internally, most search engines do not offer regex support to the public.
Part 2 of this tutorial is up now, where we use REs in python through their re module. It also discusses some complex problems one faces while writing Regular Expression i.e. “The backlash plague”. See you there! 👋
Wikipedia: Regular Expression
Geeksforgeeks
Python Docs
Regular Expressions Cookbook by Steven Levithan, Jan Goyvaerts
Learning Git in Under 8 minutes!
I‘m glad you made it till the end of this article. 🎉I hope your reading experience was as enriching as the one I had writing this. 💖
Do check out my other articles here.
If you want to reach out to me, my medium of choice would be Twitter.