Using Stringr and Regex to Extract Features in R | Towards Data Science

Large data sets are often awash with data that is difficult to decipher. One may think of textual data, names, unique identifiers, and other sorts of codes. Frequently, people analyzing these datasets are quick to discard these variables. However, sometimes there might be valuable information in this type of data, which might help you with your analysis.
Gladly, R offers the amazing package “stringr,” which is perfect for these purposes. This quick tutorial will show you how you can extract minute but still insightful data from these kinds of variables. In this case, we will be extracting this kind of data from the Titanic dataset.
If you’d like to follow the tutorial, load the Titanic data set using the below commands. The Titanic data set features all kinds of information on the Titanic passengers to practice predicting their survival. This tutorial aims to look at some of the variables (i.e., name and ticket) that most people discard immediately when beginning their analysis. We, however, want to work with them to see if we can extract some useful information. If you want to figure out how to use Stringr and regex, skip this part.
### install packageinstall.packages("titanic")### load packagelibrary(titanic)### load datasetdf = titanic_train #load dataset
Before we begin, we need to install and load the Stringr package.
### install packageinstall.packages("stringr")### load packagelibrary(stringr)
Now, we’re all set, and we can begin.
As a first exercise, I’d like to extract the titles from the names. This is because I believe, e.g., the nobility is more likely to survive than mere mortals. Also, I hypothesize that young unmarried women (“Miss”) are more likely to survive. Possibly, there might be a lot of information in the title.
The names in the titanic dataset (df$Name) are all saved like this example:
Futrelle, Mrs. Jacques Heath (Lily May Peel)
To avoid any mistakes, we are first going to run a command that converts all these strings to lowercase and saves them as new variables:
df$lcName = str_to_lower( df$Name )
The title — Mrs in the example above— is preceded by a whitespace and is preceded by a dot. We can use regex (Regular Expression) as a language to communicate this pattern to Stringr and ask it to look for the title in the following way:
(?<=\\s)[[:alpha:]]+(?=\\.)
This pattern consists of three parts:
(?<=\\s) tells Stringr that the piece of text we are looking for is preceded (?<=) by a whitespace (\\s).[[:alpha:]]+ tells Stringr that the piece of text we are looking for consists of one or more (+) letters ([[:alpha:]]).(?=\\.) tells Stringr that the piece of text we are looking for is proceeded (?=) by a dot (\\.).
(?<=\\s) tells Stringr that the piece of text we are looking for is preceded (?<=) by a whitespace (\\s).
[[:alpha:]]+ tells Stringr that the piece of text we are looking for consists of one or more (+) letters ([[:alpha:]]).
(?=\\.) tells Stringr that the piece of text we are looking for is proceeded (?=) by a dot (\\.).
You can see that regex uses all kinds of symbols to communicate patterns. The Stringr Cheat Sheet is a helpful guide for when you want to develop your own patterns. This website provides an easy way of testing regex patterns.
We extract the title and save it as a new variable by asking Stringr to look for this pattern in the lowercase “Name” strings.
df$title = str_extract( df$lcName, "(?<=\\s)[[:alpha:]]+(?=\\.)" )
Now let’s plot our new variable “title” using Ggplot2 and the commands below.
### install package (ggplot2 is a must have)install.packages("ggplot2")### load packagelibrary(ggplot2)### plot titlesggplot(data.frame(table(df$title)),aes(x=reorder(Var1, -Freq),y=Freq))+  geom_col()+  labs(title="Count of Title", x="Title",y="Count")
You can see, Stringr nicely extracted all the titles from the names, and we now have a new categorical variable we can use as input for our analysis.
As a second exercise, I’d like to extract the family names from the names string. I believe that family membership could be a predictor of survival, i.e., if one family member survives, maybe the others are also more likely to survive, controlling for other factors such as gender, etc.
In the example “name” string above, you can see that the person’s family name, e.g., Futrelle, is the first word. So we can tell Stringr to take the first word (str_extract) that matches the pattern ([[:alpha:]]+) and to take the first word and that this is the family.
However, there are family names in the dataset, such as “Vander Planke” or “O’Dwyer.” If we were to take the first word in this case, we would only get “vander” or “o,” not the whole family name. Therefore, we need to adapt the pattern to include any kinds of characters (.) that are 1 character or more in length (+) and are followed by a comma(?=\\,).
df$family=str_extract(df$lcName,".+(?=,)")
This time, because we have many family names, let’s plot the 10 most frequent family names.
t=data.frame(table(df$family))t=t[order(-t$Freq),]ggplot(head(t,10),aes(x=reorder(Var1,-Freq),y=Freq))+  geom_col()+  labs(title="Count of Family Name", x="Title",y="Family Name")
Again, we now have the novel variable “family name,” which we can use for further analysis.
In most analyses of the Titanic data set, people were quick to discard the ticket information. However, some letters in the ticket codes could potentially mean something. They could, e.g., signal that the tickets were last minute purchases and the passengers were not very prepared for the voyage.
However, these acronyms and codes are written inconsistently (e.g., “SOC” and “S.o.C.”), which is why this data needs some preparation. Let’s clean it.
### set ticket codes to lowercasedf$lcTicket=str_to_lower(df$Ticket)### remove punctuation from ticket codesdf$lcTicket=str_replace_all(df$lcTicket,"[:punct:]","")
Now that we have cleaned the data a bit let’s extract these acronyms. Usually, the ticket code now looks something like this:
sotono 2 3101287
In my mind, “sotono 2” is an acronym that could mean something, and “3101287” is some ticket number (which I’ll talk about more later). If we think about the pattern of the acronym, it’s characters (.) followed by a whitespace (\\s), which itself is followed by a number ([[:digit]]) that is three digits in length or longer ({3,}). Sometimes, the entire ticket code is some acronym (e.g., one passenger features “line” as a ticket), which means that we can give Stringer the alternative option (“|” is the “or” operator) that the entire code is an acronym of interest ([[:alpha:]]+). We tell Stringr to look for this pattern and save it in a new variable, “ticketinfo,” using the following command. We then remove whitespaces and replace NA values.
### identify acronymdf$Ticketinfo = str_extract( df$lcTicket, ".+(?=\\s(?=[[:digit:]]{3,}))|[[:alpha:]]+" )### remove whitespaces from ticketinfodf$Ticketinfo = str_replace_all( df$Ticketinfo, "\\s", "" )### remove NA valuesdf$Ticketinfo = ifelse( is.na(df$Ticketinfo), "none", df$Ticketinfo )
Let’s again plot the 10 most frequent ticketinfo values.
t=data.frame(table(df$Ticketinfo[df$Ticketinfo!="none"]))t=t[order(-t$Freq),]ggplot(head(t,10),aes(x=reorder(Var1,-Freq),y=Freq))+  geom_col()+  labs(title="Count of Ticketinfo Acronym", x="Ticketinfo Acronym",y="Count")
You can see that this adds quite some information for a lot of passengers. Almost 60 passengers have the value “PC.” After inspecting the data, I believe it could mean “private cabin” as many (but not all) 1st class passengers have this value printed on their ticket.
The interesting thing about the Titanic’s fateful maiden voyage was that the ship was only around half full. The ship could have easily transported many more passengers across the Atlantic. This made me think that some passengers might have gotten “free upgrades” to higher classes. A superficial inspection of the ticket number shows that the first digit seems to be correlated to the passenger class (Pclass), but there are exceptions. I hypothesize that the first digit could carry some information about class changes, which would have far-reaching implications for the passenger’s survival probability. Let’s extract the first digit of the number after the acronym and save it as a presumed passenger class (PresumedPclass).
Let’s first extract the ticket number. Again, the ticket number consists of digits ([[:digit:]]) and is 3 or more characters in length ({3,}). Let’s also store it as a character (as.character), not as a number, to still use Stringr on it.
df$Ticketno = as.character( str_extract( df$lcTicket, "[[:digit:]]{3,}" ))
Then let’s take the first digit ([[:digit:]]) of this ticket number and store it as the presumed passenger class.
df$PresumedPclass = str_extract( as.character(df$Ticketno) ,"[[:digit:]]{1}")
Now, let’s again plot this new variable, but let’s plot it in the context of the provided passenger class to see how it correlates.
ggplot(df, aes(x=Pclass, fill=PresumedPclass))+  geom_bar()+  labs(title="Count of Passenger Class", x="Passenger Class",y="Count")+  scale_fill_discrete(name="Presumed Passenger Class")
As we can see in the graph below, these two values mostly correlate. But sometimes they do not. This might be exciting information in those cases in which these two classes did not match.
You would like to find out more? Follow me on Medium to see my other stories!