Regular Expressions — An excellent tool for text analysis or NLP
By: Niwratti Kasture
Tags: Regex, Machine Learning Ai, Regular Expressions, Naturallanguageprocessing, Text Analytics

A fascinating programming tool available within most of the programming languages — Regular expressions also called regex. It is a very powerful programming tool that is used for a variety of purposes such as feature extraction from text, string replacement and other string manipulations. A regular expression is a set of characters, or a pattern, which is used to find sub strings in a given string. for ex. extracting all hashtags from a tweet, getting email id or phone numbers etc..from a large unstructured text content.

In short, if there’s a pattern in any string, you can easily extract, substitute and do variety of other string manipulation operations using regular expressions. Regular expressions are a language in itself since they have their own compilers and almost all popular programming languages support working with regexes.

It won’t be an exaggeration to mention that without having understanding of regular expressions, it is not possible to really build a NLP based system such as chatbots, conversational UI etc.. Regex has many important elements and features that help to build a useful, fit to purpose expressions solution for string extraction or manipulation.

Let’s look at each of these in next few sections.

Common Regex Functions

re.search() — detects whether the given regular expression pattern is present in the given input string. It returns a Regexobject if the pattern is found in the string, else it returns a None object.

re.search(patterns, text)

Within search functions, different flags can be used to perform specific operations. For example

‘re.I’ — to ignore the case of the text

re.M — enables to search in multiple lines

re.search(pattern, string, flags=re.I | re.M)

2. match.start() and match.end() — Returns the index of the starting and ending position of the match found respectively.

3. re.match() — The match function will only match if the pattern is present at the very start of the string.

re.match(patterns, text)

4. re.sub() — It is used to substitute a substring with another substring. for e. replacing ‘color’ with ‘colour’.

re.sub(Pattern, Substitute, Input text)

5. finditer() or the findall() — The result of the findall() function is a list of all the matches and the finditer() function is used in a ‘for’ loop to iterate through each separate match one by one.

6. re.compile() — This function stores the regular expression pattern in the cache memory for faster searches. You need to pass the regex pattern to re.compile() function.

# without re.compile() function

result = re.search(“a+”, “abc”) # using the re.compile() function

pattern = re.compile(“a+”)

result = pattern.search(“abc”)

Quantifiers — Allows to mention and have control over how many times specific character(s) pattern should occur in the given text . for ex. In some given data which have the variations of word ‘Program’ in it. for ex. ‘Program’, ‘Programmed’, ‘Programmer’. if the requirement is to find only those words where one or more occurrence of ‘e’appearing in the word ‘Program’. You could use any of the following quantifiers to indicate the presence of specific character

‘?’ — matches the preceding character zero or one time. It is generally used to mark the optional presence of a character.

‘*’ — quantifier is used to mark the presence of the preceding character zero or more times.

‘+’ — quantifier matches the preceding character one or more times. That means the preceding character has to be present at least once for the pattern to match the string.

{m,n} quantifier — There are four variants of this quantifier

{m, n}: Matches the preceding character ‘m’ times to ’n’ times.

{m, }: Matches the preceding character ‘m’ times to infinite times, i.e. there is no upper limit to the occurrence of the preceding character.

{, n}: Matches the preceding character from zero to ’n’ times, i.e. the upper limit is fixed regarding the occurrence of the preceding character.

{n}: Matches if the preceding character occurs exactly ’n’ number of times.

Each of the earlier mentioned quantifiers can be written in the form of {m,n} quantifier —

‘?’ is equivalent to zero or once, or {0, 1}

‘*’ is equivalent to zero or more times, or {0, }

‘+’ is equivalent to one or more times, or {1, }

Whitespace — A white-space comprises of a single space, multiple spaces, tab space or a newline character (also known as a vertical space). These whitespaces will match the corresponding spaces in the string. For example, the pattern ‘ +’, i.e. a space followed by a plus sign will match one or more spaces. Similarly, you could use spaces with other characters inside the pattern. The pattern, ‘Sachin Tendulkar’ will allow you to look for the name ‘Sachin Tendulkar’ in any given string.

Parentheses — Using parentheses around some characters will help to repeat a group of characters rather than just looking for repetitions of the preceding character. This concept is called grouping in regular expression. For ex. the pattern ‘(xyz){1, 3}’ will match the following strings:

xyz

xyzxyz

xyzxyzxyz

Use of parentheses is also helpful when you need to extract sub-patterns out of a larger pattern. for ex. if you have textual data with dates in it and you want to extract only the year from the dates. You can use a regular expression pattern with grouping to match dates and then you can extract the any of the elements such as the day, month or the year. Grouping is achieved using the parenthesis operators. for ex. 10/03/1985”. To extract the date from this string you can use the pattern — “\d{1,2}\/\d{1,2}\/\d{4}”. Now to extract the year, you can put parentheses around the year part of the pattern. The pattern is: “^\d{1,2}/\d{1,2}/(\d{4})$”.

Pipe operator — It’s denoted by ‘|’. The pipe operator is used as an OR operator. You need to use it inside the parentheses. For example, the pattern ‘(play|walk)ed’ will match both the strings — ‘played’ and ‘walked’. The pipe operator tells that the place inside the parentheses can be either of the strings or characters.

Special Characters — The characters that are mentioned as quantifiers above such as ‘?’, ‘*’, ‘+’, ‘(‘, ‘)’, ‘{‘, etc. can also appear in the input text. In such cases to extract these specific characters you’ll need to use escape sequences. The escape sequence, denoted by a backslash ‘\’, is used to escape the special meaning of the special characters. To match a question mark, you need to use ‘\?’ (this is called escaping the character) or to match a ‘+’ sign, you need to provide ‘\+’ in the regular expression. The ‘\’ itself is a special character, and to match the ‘\’ character ,you need to use the pattern ‘\\’ to escape the backslash.

Anchors — These are special characters to perform string operations at the beginning and end of a text input.

Carat character‘^’ — specifies the start of the string. The character followed by the ‘^’ in the pattern should be the first character of the string in order for a string to match the pattern.

Dollar character ‘$’ — specifies the end of the string. The character that precedes the ‘$’ in the pattern should be the last character in the string in order for the string to match the pattern.

Both the anchors can be specified in a single regular expression itself. For example, the regular expression pattern ‘^ 01+2$’ will match any string that starts with zeroes and ends with 2 with occurrence of 1s between them. It will match ‘012’, ‘0112’, ‘01111111112’ .

Wildcard Character — the ‘.’ (dot) character is also called the wildcard character. It acts as a placeholder and can match any character. For example, if a string that starts with four characters, followed by three 0s and two 1s, followed by any two characters. The valid strings can be cghj00011op, kghj00011pk, etc. The pattern that satisfies this kind of condition would be ‘.{4}0{3}1{2}.{2}’. You can also use ‘….00011..’ where the dot acts as a placeholder which means anything can sit on the place of the dot.

Character Sets — Character sets provide lot more flexibility than just typing a wildcard or the literal characters. These are group of characters specified inside square brackets. Character sets can be specified with or without a quantifier. When no quantifier succeeds the character set, it matches only one character and the match is successful only if the character in the string is one of the characters present inside the character set. For example, the pattern ‘[a-z]ed’ will match strings such as ‘ted’, ‘bed’, ‘red’ and so on because the first character of each string — ‘t’, ‘b’ and ‘r’ — is present inside the range of the character set.On the other hand, when we use a character set with a quantifier, such as in this case — ‘[a-z]+ed’, it will match any word that ends with ‘ed’ such as ‘watched’, ‘baked’, ‘jammed’, ‘educated’ and so on. Essentially a quantifier loses its special meaning and treated as a normal character when it’s present inside the character set.

The ‘^’ has two use cases. Outside a character set it works as an anchor but when used inside a character set, it acts as a complement operator, i.e. it specifies that it will match any character other than the ones mentioned inside the character set. for ex. The pattern [0–9] matches any single digit number. On the other hand, the pattern ‘[^ 0-9]’ matches any single digit character that is not a digit.

Meta Sequences — These are basically shorthand way to write commonly used character sets in regular expressions. These are the commonly used meta-sequences.

You can use meta-sequences in two ways — either use them without the square brackets. For example, the pattern ‘\w+’ will match any alphanumeric character.Or you can write them inside the square brackets. For example, the pattern ‘[\w]+’ is same as ‘\w+’.

In summary regular expressions can be used as a handy tool with many practical applications and it is fundamental to implement NLP. Along with the applications in NLP, regular expressions can be used for various other applications such as checking if input given by user in a form is meeting the minimum criteria or not etc..and similarly many other possibilities.