How to Efficiently Remove Punctuations from a String | by Eunjoo Byeon | Towards Data Science

Recently I found myself spending many hours trying to make sense of messy text data, and decided to review some of the preprocessing involved. There are many different ways to achieve a simple cleaning step. Today, I will review a couple of different methods to remove punctuations from a string and compare their performances.
The string translate method is a convenient way to change multiple characters to different values at once. Translate requires a table that will work as a dictionary to map the strings. The maketrans does that job for you.
The maketrans syntax works like str.maketrans('abcd', '0123', 'xyz'). It will create a table that tells translate to change all a with 0, b with 1, c with 2, etc., and remove x, y, and z.
Full syntax to remove punctuations and digits using translate is as below.
# importing a string of punctuation and digits to removeimport stringexclist = string.punctuation + string.digits# remove punctuations and digits from oldtexttable_ = str.maketrans('', '', exclist)newtext = oldtext.translate(table_)
This approach will entirely remove any character that is in string.punctuation and string.digits. That includes !”#$%&\’()*+,-./:;<=>?@[\\]^_`{|}~’ and all numbers.
But sometimes, we might want to add a space in place of these special characters instead of getting rid of them entirely. We can do so by telling a table to change special characters to space instead of excluding them.
table_ = str.maketrans(exclist, ' '*len(exclist))
Additionally, we can simply split and join to make sure this operation does not result in multiple spaces between words.
newtext = ' '.join(oldtext.translate(table_).split())
We can also just use join instead of translate, taking the same exclusion list from the string package we made above.
# using exclist from abovenewtext = ''.join(x for x in oldtext if x not in exclist)
We can forego the exclusion list and just use the string method to call only the alphabets.
newtext = ''.join(x for x in oldtext if x.isalpha())
This approach will only keep the alphabet. As a result, it will also eliminate space between words.
Instead of the list comprehension, we can do the same thing using the filter. This is slightly more efficient than using a list comprehension but outputs a new text in the same manner.
newtext = ''.join(filter(str.isalpha, oldtext))
Another way to remove punctuations (or any select characters) is to iterate through each special character and remove them one at a time. We can do this by using the replace method.
# using exclist from abovefor s in exclist:     text = text.replace(s, '')
There are many ways to accomplish a similar thing using regex depending on the exact goal. One way to do it is to replace characters that are not alphabets with space.
import renewtext = re.sub(r'[^A-Za-z]+', ' ', oldtext)
[^A-Za-z]+ selects any character that matches the rule inside of the square bracket ([]), that does not (^) have at least one (+) letter in upper case alphabets (A-Z) or lower case alphabets (a-z). Then the regex sub replaces these characters in the old text with space.
Another method is to select all the non-words by using a metacharacter \W. This metacharacter does not include underscore (-) and numbers.
newtext = re.sub(r'\W+', ' ', oldtext)
We reviewed a handful of methods, but which one is the best? I used the timeit module to measure how long each of the methods takes to process approximately 1kb string data 10000 times.
The test shows that using translate takes much less time compared to other methods! On the other hand, using the join with the list comprehension seems to be the least efficient way to clean select characters. Translate is the most versatile and fast option out of all reviewed today.
If you have any other method, please leave a comment and I will add the test result to the post!