Introduction to Regular Expressions (Regex) in R | Towards Data Science

We live in a data-centric age. Data has been described as the new oil. But just like oil, data isn’t always useful in its raw form. One form of data that is particularly hard to use in its raw form is unstructured data.
A lot of data is unstructured data. Unstructured data doesn’t fit nicely into a format for analysis, like an Excel spreadsheet or a data frame. Text data is a common type of unstructured data and this makes it difficult to work with. Enter regular expressions, or regex for short. They may look a little intimidating at first, but once you get started, using them will be a picnic!
More comfortable with python? Try my tutorial for using regex with python instead:
towardsdatascience.com
We’ll use the stringr library. The stringr library is built off a C library, so all of its functions are very fast.
To install and load the stringr library in R, use the following commands:
See how easy that is? To make things even easier, most function names in the stringr package start with str. Let’s take a look at a couple of the functions we have available to us in this module:
str_extract_all(string, pattern): This function returns a list with a vector containing all instances of pattern in stringstr_replace_all(string, pattern, replacement): This function returns string with instances of pattern in string replaced with replacement
str_extract_all(string, pattern): This function returns a list with a vector containing all instances of pattern in string
str_replace_all(string, pattern, replacement): This function returns string with instances of pattern in string replaced with replacement
You may have already used these functions. They have pretty straightforward applications without adding regex. Think back to the times before social distancing and imagine a nice picnic in the park, like the image above. Here’s an example string with what everyone is bringing to the picnic. We can use it to demonstrate the basic usage of the regex functions:
basicString <- "Drew has 3 watermelons, Alex has 4 hamburgers, Karina has 12 tamales, and Anna has 6 soft pretzels"
If I want to pull every instance of one person’s name from this string, I would simply pass the name and basic_string to str_extract_all():
The result will be a list with all occurrences of the pattern. Using this example, basicExtractAll will have the following list with 1 vector as output:
[[1]][1] "Drew"
Now let’s imagine that Alex left his 4 hamburgers unattended at the picnic and they were stolen by Shawn. str_replace_all can replace any instances of Alex with Shawn:
The resulting string will show that Shawn now has 4 hamburgers. What a lucky guy 🍔.
"Drew has 3 watermelons, Shawn has 4 hamburgers, Karina has 12 tamales, and Anna has 6 soft pretzels"
The examples so far are pretty basic. There is a time and place for them, but what if we want to know how many total food items there are at the picnic? Who are all the people with items? What if we need this data in a data frame for further analysis? This is where you will start to see the benefits of regex.
There are several concepts that drive regex:
Character setsMeta charactersQuantifiersCapture Groups
Character sets
Meta characters
Quantifiers
Capture Groups
This is not an exhaustive list, but is plenty to help us hit the ground running.
Character sets represent options inside of brackets, with regex matching only one of the options. There are multiple things we can do with character sets:
Match a group of characters: We can find all of the vowels in our string by putting every vowel in brackets, for example,[aeiou]
[[1]] [1] "e" "a" "a" "e" "e" "o" "e" "a" "a" "u" "e" "a" "i" "a"[15] "a" "a" "a" "e" "a" "a" "a" "o" "e" "e"
Match a range of characters: We can find any capital letter from “A” to “F,” by using a hyphen, [A-F]. Character sets are case sensitive, so [A-F] is not the same as [a-f]
[[1]][1] "D" "A" "K" "A"
Match a range of numbers: We can find numbers between a range by adding numbers to our character set, [0-9] to find any number. Notice that the numbers are extracted as strings, not converted to numbers
[[1]][1] "3" "4" "1" "2" "6"
Character sets can contain everything from this section simultaneously, so something like [A-Ct-z7-9] is still valid. It will match every character from capital “A” to capital “C,” lowercase “t” to lowercase “z,” and 7 through 9.
So far we can’t answer any of the questions posed earlier with just bracket groups. Let’s add some more weapons to our regex arsenal.
Meta characters represent a type of character. They will typically begin with a backslash \. Since the backslash \ is a special character in R, it needs to be escaped each time it is used with another backslash. In other words, R requires 2 backslashes when using meta characters. Each meta character will match to a single character. Here are some of the most important ones in action:
\\s: This meta character represents spaces. This will match to each space, tab, and newline character. You may also specify \\t and \\n for tab and newline characters respectively. Side note: our example string does not have any tabs, but be cautious when looking for them. Many integrated development environments, or IDE’s, have a setting that will replace all tabs with spaces while you are typing. In the example string, \\s returns a list of a vector of 17 spaces, the exact number of spaces in our example string!
[[1]] [1] " " " " " " " " " " " " " " " " " " " " " " " " " " " "[15] " " " " " "
\\w: This meta character represents alphanumeric characters. This includes all the letters a-z, capital and lowercase, and the numbers 0–9. This would be the equivalent of the bracket group [A-Za-z0-9], just much quicker to write. Take caution in remembering that the \\w meta character on its own only captures a single character, not entire words or numbers. You’ll see that in the example. Don’t worry, we’ll handle that in the next section.
[[1]] [1] "D" "r" "e" "w" "h" "a" "s" "3" "w" "a" "t" "e" "r" "m"[15] "e" "l" "o" "n" "s" "A" "l" "e" "x" "h" "a" "s" "4" "h"[29] "a" "m" "b" "u" "r" "g" "e" "r" "s" "K" "a" "r" "i" "n"[43] "a" "h" "a" "s" "1" "2" "t" "a" "m" "a" "l" "e" "s" "a"[57] "n" "d" "A" "n" "n" "a" "h" "a" "s" "6" "s" "o" "f" "t"[71] "p" "r" "e" "t" "z" "e" "l" "s"
\\d: This meta character represents numeric digits. Using our picnic example, see how it only finds the digits in the string. You’ll notice that like bracket groups, it picks up 5 numbers instead of the 4 we expect. This is because it is looking for each individual digit, not groups of digits. We’ll see how to fix that with quantifiers next.
[[1]][1] "3" "4" "1" "2" "6"
As we saw in the previous section, a single meta character can have somewhat limited functionality. When it comes to words or numbers, we usually want to find more than 1 character at a time. This is where quantifiers come in. They allow you to quantify how many of a character you are expecting. They always come after the character they are quantifying and come in a few flavors:
+ quantifies 1 or more matches. Let’s look at a new example to develop some intuition about what each quantifier will return: quantExample
When we use the + quantifier on quantExample, it will return 4 matches. This is a good point to mention that regex looks for non-overlapping matches. In this case, it looks at each B and the character that follows it. Since we used the + quantifier, it continues to match until it reaches the end of a group of B’s.
[[1]][1] "B"    "BB"   "BBB"  "BBBB"
{} quantifies a specific number or range of matches. When written like {2} it will match exactly 2 of the preceding character. We’ll see some interesting results. It picked up 4 matches. This is because it is looking for each non-overlapping group of 2 B’s. There is a match in the 1st group, only 1 non-overlapping match in the 2nd group, and 2 non-overlapping matches in the 4th.
[[1]][1] "BB" "BB" "BB" "BB"
When written like {2,4}, it will match any number of B’s from 2 to 4 occurrences. Note that putting a space in your regex will NOT work. It will return an empty list.
[[1]][1] "BB"   "BBB"  "BBBB"
We can also write this quantifier and omit the upper bound like {2,}. This will match 2 or more instances. For quantExample, it will return the exact same result as {2,4}.
* quantifies zero or more matches. This can be helpful when we are looking for something that may or may not be in our string.
The * quantifier returns some strange matches when used by itself, so we can omit an example with quantExample. We will see in a following example how it can be applied when someone at our picnic is bringing a food item with a multiple word name. Without it, we wouldn’t correctly capture that Anna is bringing soft pretzels!
Let’s combine what we know so far about character sets, meta characters, and quantifiers to answer some questions about our picnic string. We want to know all of the words that are in the string and also the numbers in the string.
For words, we can use a character set with all upper and lower case letters, adding a + quantifier to it. This will find any length of alpha characters grouped together. Said another way, it finds all of the words. Regex is starting to look much more helpful.
[[1]] [1] "Drew"        "has"         "watermelons" "Alex"        [5] "has"         "hamburgers"  "Karina"      "has"         [9] "tamales"     "and"         "Anna"        "has"        [13] "soft"        "pretzels"
To find the quantity of each food item, we can use the \\d meta character and the quantifier {1,2}. This will find the groups of digits that are 1 or 2 characters long. This is a much more useful output as we have the same number of quantities as we have food items and people!
[[1]][1] "3"  "4"  "12" "6"
To find the quantity and name of each food item, we can combine quantifiers with meta characters. We know that each number has a food item directly after it, so we can just add on to the previous example. We know there is a space and a word (\\s\\w+) that could be followed by another word like how “soft pretzel” appears. To specify the second word might not be there, we can use the * quantifier with the second word. Just like that we have a list containing the quantity and name of every good at our picnic.
[[1]][1] "3 watermelons"   "4 hamburgers"    "12 tamales"     [4] "6 soft pretzels"
Capture groups allow you to look for entire phrases and only return parts of them. With our example, I want each person’s name, what they are bringing, and how much of it they are bringing. Up until this point we have been using str_extract_all. It has a clean output that is easy to read for our examples, but it doesn’t actually work with capture groups. Helpfully, stringr provides str_match_all which does work with capture groups. It does however output the results in a list containing a matrix as opposed to a list containing a vector.
[[1]]     [,1][1,] "Drew has 3 watermelons"[2,] "Alex has 4 hamburgers"[3,] "Karina has 12 tamales"[4,] "Anna has 6 soft pretzels"
The regex we used in captureGroup1 is looking for a name, which starts with a capital letter and has any amount of lowercase letters after it ([A-Z][a-z]+). Then after a space it matches the pattern space, word, space \\s\\w+\\s. Next we are looking for a 1 to 2 digit number followed by a space and a word (\\d{1,2}\\s\\w+). You can see in the output each row of the matrix is a character string with the details for each person.
Now this is a big step up from where we started, but we don’t really care about the word “has”, and we want to be able to make a data frame out of the quantities. Let’s add in capture groups. By using capture groups, we can return a matrix where each column contains a specific piece of information. We’ll create capture groups containing each name, quantity, and item. Capture groups are simply sections of the regex that you wrap in parenthesis.
[[1]]     [,1]                       [,2]     [,3]  [,4][1,] "Drew has 3 watermelons"   "Drew"   "3"   "watermelons"[2,] "Alex has 4 hamburgers"    "Alex"   "4"   "hamburgers"[3,] "Karina has 12 tamales"    "Karina" "12"  "tamales"[4,] "Anna has 6 soft pretzels" "Anna"   "6"   "soft pretzels"
The first column in the matrix has the entire regex, ignoring the capture groups. The remaining columns of the matrix each correspond to the capture groups we defined for name, quantity, and item.
When doing data analysis, one of the most useful R data structures is a data frame. No doubt you already knew this if you clicked on this article. Data frames enable things like calculating column statistics and plotting data. Since we have a matrix with all of the information we want, turning it into a data frame isn’t too hard. We will use the data.frame function on everything except for the first column of the matrix. data.frame gives default columns names, so we will change those to match up to what is in each column.
A quick note on the notation: the first set of brackets after captureGroup2 ([[1]]) accesses the first element of the list, our matrix. The second set of brackets ( [,-1]) selects all rows and every column except the first one.
|   | Name   | Quantity | Item          || - | ------ | -------- | ------------- || 1 | Drew   | 3        | watermelons   || 2 | Alex   | 4        | hamburgers    || 3 | Karina | 12       | tamales       || 4 | Anna   | 6        | soft pretzels |
We only covered a small subset of how regex can help handle unstructured text data. This is a good foundation to get started, but before long you will need to know concepts like how to find everything BUT a character (negation) or find something immediately before or after something else (lookarounds). You can learn all about those in my followup post here:
medium.com
Here are some more resources to help you learn more about these other concepts in regex:
The official stringr page on the tidyverse site: The folks over at RStudio have compiled resources to help learn packages like stringr. They even included a stringr cheat sheet that you can print out and reference.
R for Data Science: Written by Hadley Wickham, author of the stringr package, this book is a good reference for anything in R. There is even a chapter that covers more advanced regex in R. It is available online for free here, or you can purchase a hardcopy here. Disclaimer: I receive a commission of your purchase through this link.
Datacamp Courses: An online learning community dedicated to data science, machine learning, and data visualization. Check out their course “String Manipulation with stringr in R.” The first chapter of every course on the site is free! Disclaimer: You will receive a discount on your subscription and I receive a commission if you sign up for a monthly Datacamp subscription using this link. Support my writing while you learn!