5 Must-Know Pandas Operations on Strings | by Soner Yıldırım | Towards Data Science

We have to represent every bit of data in numerical values to be processed and analyzed by machine learning and deep learning models. However, strings do not usually come in a nice and clean format and require preprocessing to convert to numerical values. Pandas offers many versatile functions to modify and process string data efficiently.
In this post, we will discover how Pandas can manipulate strings. I grouped string functions and methods under 5 categories:
SplittingStrippingReplacingFilteringCombining
Splitting
Stripping
Replacing
Filtering
Combining
Let’s first create a sample dataframe to work on for examples.
import numpy as npimport pandas as pdsample = {'col_a':['Houston,TX', 'Dallas,TX', 'Chicago,IL', 'Phoenix,AZ',      'San Diego,CA'],'col_b':['$64K-$72K', '$62K-$70K', '$69K-$76K', '$62K-$72K', '$71K-$78K' ],'col_c':['A','B','A','a','c'],'col_d':['  1x', ' 1y', '2x  ', '1x', '1y  ']}df_sample = pd.DataFrame(sample)df_sample
Sometimes strings carry more than one piece of information and we may need to use them separately. For instance, “col_a” contains both city and state. The split function of pandas is a highly flexible function to split strings.
df_sample['col_a'].str.split(',')0      [Houston, TX] 1       [Dallas, TX] 2      [Chicago, IL] 3      [Phoenix, AZ] 4    [San Diego, CA] Name: col_a, dtype: object
Now each element is converted to a list based on the character used for splitting. We can easily export individual elements from those lists. Let’s create a “state” column.
df_sample['state'] = df_sample['col_a'].str.split(',').str[1]df_sample
Warning: Subscript ([1]) must be applied with str keyword. Otherwise, we will get the list in the specified row.
df_sample['col_a'].str.split(',')[1]['Dallas', 'TX']
The splitting can be done on any character or letter.
The split function returns a dataframe if expand parameter is set as True.
df_sample['col_a'].str.split('a', expand=True)
By default, splitting is done from the left. To do splitting on the right, use rsplit.
Consider the series below:
Let’s apply split function and limit the number of splits with n parameter:
categories.str.split('-', expand=True, n=2)
Only 2 splits on the left are performed. If we do the same operation with rsplit:
categories.str.rsplit('-', expand=True, n=2)
Same operation is done but on the right.
Stripping is like trimming tree branches. We can remove spaces or any other characters at the beginning or end of a string.
For instance, the strings in “col_b” has $ character at the beginning which can be removed with lstrip:
df_sample['col_b'].str.lstrip('$')0    64K-$72K 1    62K-$70K 2    69K-$76K 3    62K-$72K 4    71K-$78K Name: col_b, dtype: object
Similary, rstrip is used to trim off characters from the end.
Strings may have spaces at the beginning or end. Consider “col_d” in our dataframe.
Those leading and trailing spaces can be removed with strip:
df_sample['col_d'] = df_sample['col_d'].str.strip()
Pandas replace function is used to replace values in rows or columns. Similarly, replace as a string operation is used to replace characters in a string.
Let’s replace “x” letters in “col_d” with “z”.
df_sample['col_d'] = df_sample['col_d'].str.replace('x', 'z')
We can filter strings based on the first and last characters. The functions to use are startswith() and endswith().
Here is our original dataframe:
Here is a filtered version that only includes rows in which “col_a” ends with the letter “x”.
df_sample[df_sample['col_a'].str.endswith('X')]
Or, rows in which “col_b” starts with “$6”:
df_sample[df_sample['col_b'].str.startswith('$6')]
We can also filter strings by extracting certain characters. For instace, we can get the first 2 character of strings in a column or series by str[:2].
“col_b” represents a value range but numerical values are hidden in a string. Let’s extract them with string subscripts:
lower  = df_sample['col_b'].str[1:3]
upper  = df_sample['col_b'].str[-3:-1]
Cat function can be used to concatenate strings.
We need pass an argument to put between concatenated strings using sep parameter. By default, cat ignores missing values but we can also specify how to handle them using na_rep parameter.
Let’s create a new column by concatenating “col_c” and “col_d” with “-” separator.
df_sample['new']=df_sample['col_c'].str.cat(df_sample['col_d'], sep='-')df_sample
Before pandas 1.0, only “object” datatype was used to store strings which cause some drawbacks because non-string data can also be stored using “object” datatype. Pandas 1.0 introduces a new datatype specific to string data which is StringDtype. As of now, we can still use object or StringDtype to store strings but in the future, we may be required to only use StringDtype.
One important thing to note here is that object datatype is still the default datatype for strings. To use StringDtype, we need to explicitly state it.
We can pass “string” or pd.StringDtype() argument to dtype parameter to string datatype.
Thank you for reading. Please let me know if you have any feedback.