Pandas vs Tidyverse on Textual Data | by Soner Yıldırım | Towards Data Science

Textual data does not usually come in a nice and clean format so it requires a lot of preprocessing and manipulation. A substantial amount or raw data is textual so a data analysis library should be able to handle strings very well.
In this article, we will compare two popular libraries in terms of working on strings. The first one is pandas, a data analysis and manipulation library for Python. The other is the tidyverse which is a collection of R packages designed for data science.
We will be using the cities dataframe for the examples in Pandas. I have also created a tibble that contains the same data using the readr package in R.
Pandas:
The string functions and methods can be used using the str accessor. The split function splits a string at the given character.
The city column includes both city and state names. We can create a new column for the state part.
import pandas as pdcities.city.str.split(",")0    [Houston, TX]1     [Dallas, TX]2    [Seattle, WA]3    [Atlanta, GA]
Each value in the returned series is a list of city and state. We can assign a new column to the state part by extracting it using str indexing.
cities['state'] = cities.city.str.split(",").str[1]
Tidyverse:
Tidyverse is a collection of packages. We will be using the following packages of tidyverse to work on the cities tibble (same as dataframe in pandas)
readr: Create a tibble from a csv file
dplyr: Data manipulation on tibble
stringr: Functions on strings
cities <- mutate(cities, state = str_split_fixed(city, ",", n=2)[,2])
The mutate function of dplyr package can be used to create new columns based on the existing ones.
The str_split_fixed function of stringr package splits the city column at comma. Since there are two words in each values, the n parameter is 2. Finally, we select the second part after splitting.
Converting strings to upper or lower case is a common operation to have consistent values.
Let’s convert the “ctg1” column to upper case letters.
Pandas:
The upper function is used as follows.
cities['ctg1'] = cities['ctg1'].str.upper()
Tidyverse:
The str_to_upper function of stringr package is used as follows.
cities <- mutate(cities, ctg1 = str_to_upper(ctg1))
Just like we split strings, we may also need to combine them. For instance, we can create a new category column by combining the “ctg1” and “ctg2” columns.
Pandas:
The cat function of str accessor is used. We specify the character to be used as separator along with the column names.
cities['new_ctg'] = cities['ctg1'].str.cat(cities['ctg2'], sep="-")
Tidyverse:
The str_c function of the stringr package is used.
cities <- mutate(cities, new_ctg = str_c(ctg1, ctg2, sep="-"))
We can inspect string to check if they contains a specific character or a set of characters. Let’s create a new column indicating if the “new_ctg” column contains the letter “f”.
Pandas:
The function to use is pretty self-explanatory.
cities["is_f"] = cities.new_ctg.str.contains("f")
Tidyverse:
The str_detect function of the stringr package accomplishes what the contains function of pandas does.
cities <- mutate(cities, is_f = str_detect(new_ctg, "f"))
Strings can be considered as a collection of characters. They do not have to be words or meaningful textual data. For instance, “foo123” is a valid string.
Indexing or slicing on strings is a highly practical operations in data manipulation. We can extract any part of strings by providing the indices. Both pandas and tidyverse provide functions to extract part of strings based on an index of characters.
Let’s create a new column by extracting the last two characters of the “ctg2” column.
Pandas:
We can pass the index from the beginning starting from 0 or from the end starting from -1. Since we need the last two, it is more convenient to use the one from the end.
cities["sub_ctg2"] = cities.ctg2.str[-2:]
The start and end of the desired part of string is specified by passing the related indices to the str accessor. The starting index is the second from the end (-2). The ending index is left blank to indicate the end of string.
Tidyverse
The same operation can be done using the str_sub function as follows:
cities <- mutate(cities, sub_ctg2 = str_sub(ctg2, -2, -1))
In some cases, we may need the length of a string in terms of the number of characters it contains. How many times a particular character exists in a string can also be a useful piece of information.
Pandas:
The len function returns the length of a string and the count function returns the number of occurrences of a character in a string.
# Length of citiescities.city.str.len()0    101     92    103    10# How many "a" exists in city namescities.city.str.count("a")0    01    22    13    2
Tidyverse:
The name of the functions are quite similar to those in pandas. We just need to add “str” at the beginning.
# Number of occurrencesstr_count("Houston", "o")[1] 2# Length of a stringstr_length("Houston")[1] 7
We have covered some basic operations that are commonly used to manipulate strings. Both pandas and tidyverse are highly efficient at performing such tasks.
There are many more operations that can be done to manipulate or transform strings. I think what we covered in this article is enough to warm you up. In a later article, I plan to demonstrate more complex operations on strings using both pandas and tidyverse.
Thank you for reading. Please let me know if you have any feedback.