Python Speed Test: 5 Methods To Remove The ‘$’ From Your Data in Python | by Chaim Gluck | Towards Data Science

In a previous post about a regression project on Iowa liquor sales, I mentioned that it was my first time working with data large enough to worry about writing code to optimize speed. In this post, I’ll walk through a relatively simple example of that process.
Here’s a look at the data we’re working with. Notice the ‘state_bottle_retail’ column. Every entry starts with a dollar sign, and to make the values numeric, I’ll need to remove those dollar signs.
There are a bunch of different ways to accomplish this in Python. But due to the size of this data set, optimization becomes important. I’ll demonstrate some of the ways, and report how much time they took. One note: I’ll be doing these tests on a small subset of about 10% of the entire data set. So even though the speeds are all very fast, with the slowest at just over 130 milliseconds, when the scale gets larger, it will matter more. This is also intended as a representation of the importance and practice of optimization. This is a relatively simplistic example, but in certain situation, practices like these can save hours or even days.
For these tests, I’ll be using the %timeit cell magic in Jupyter Notebooks. This is a convenient tool which runs multiple loops of the operation and reports it’s best performance time. To do it, you simply type %timeit at the beginning of the row with your operation, run the cell, and see the results.
First, I’ll show you a picture of all the tests I ran, and then we’ll walk through them one by one.
First, I used the str.replace('$','') method on the entire column. This is the most straightforward method, as it simply replaces the ‘$’ with a blank space for each item in the column. This was the slowest option, as you can see, but it still relatively quick like I mentioned above. That means it would only take about a second to do this on the full data set with over 2 million rows. That’s fast. But this article is about getting faster.
The next method uses the pandas ‘apply’ method, which is optimized to perform operations over a pandas column. The ‘apply’ method requires a function to run on each value in the column, so I wrote a lambda function to do the same function. In this post, I talk more about using the ‘apply’ method with lambda functions. In this example, it looks like this:
%timeit df.state_bottle_retail.apply(lambda x: x.replace('$',''))
The .apply method worked just like it’s supposed to, and sped up the operation to 117 ms. Lookin’ good.
For the next step, I changed the .replace method to the .strip method. It does one less operation. Instead of replacing the ‘$’ with a blank space, it just takes out the ‘$.’ That looks like this:
%timeit df.state_bottle_retail.apply(lambda x: x.strip('$'))
That sped it up to just under 100 ms for the whole column. Getting better!
Next up was a list comprehension. List comprehensions are a very efficient method of iterating over a lot of objects in Python. So I tried the same .strip method with a list comprehension instead of the .apply method. That looks like this:
%timeit [x.strip('$') for x in df.state_bottle_retail]
The list comprehension bumped us up to 72.3 ms. Nice!
Lastly, I tried another way. Instead of using a function to pull out the ‘$’, I used Python built in [] slicing. It’s often used to slice and select the values you need from a list, but it can slice strings as well. So [1:] slices each string from the second value until the end. Since Python is zero-indexed, which means it starts counting at 0, the number 1 is the second value. The ‘:’ tells it to slice until the end of the string. Here’s the final list comprehension using the string slicing method:
%timeit [x[1:] for x in df.state_bottle_retail]
That clocks in at a blazing 31.4 ms, which is not only the fastest time, but also the largest increase in speed for any of these tests. This method is far and away the fastest, but there’s one caveat to using it. If any of the values don’t have a ‘$’ in front, this will actually take off the first number in that string. So you have to be careful when using this method. My personal choice would be to use the fourth method, the list comprehension with the .strip method. Even though it isn’t the fastest, it’s less risky. Depending on the size of your data and your confidence in it’s integrity, you’ll have to make the decision.
One small note: to make any of these changes actually work, you’d have to assign the changed values back to the column you are changing in your DataFrame. That would look like this:
df.state_bottle_retail = [x.strip('$') for x in df.state_bottle_retail]
Optimizing your code’s speed is a fun and interesting process. It’s not always necessary to do, but it’s a good idea to get used to thinking in that way, especially if you want to work with big data or deploy code to customers. Have fun!
Update: ‘nzdatascientist’ commented with a different method below. I gave it a try on the same data, and it’s lightning quick. It outperforms the other methods by far without the danger of removing other values if the entry doesn’t have a ‘$’. Here it is:
np.fromstring(df.state_bottle_retail.values.astype('|S7').tobytes().replace(b'$',b''), dtype='|S6')
That clocks in at a blazing 14.3 ms, more than double as quick as the risky string slicing method, and almost 10 times as fast as the slowest demonstrated method. It looks like numpy’s .fromstring method is optimized for this type of process. Also, converting to bytes and replacing those quickens the process as well. Thanks ‘nzdatascientist’!