All the Pandas shift() you should know for data analysis | by B. Chen | Towards Data Science

Suppose you encountered a situation where you need to push all rows in a DataFrame or require to use the previous row in a DataFrame. Maybe you want to calculate the difference in consecutive rows, Pandas shift() would be an ideal way to achieve these objectives.
In this article, we’ll be going through some examples of manipulating data using Pandas shift() function. We will focus on practical problems and should help you get started with data analysis.
Shifting values with periodsShifting time-series data with freqA practical example: calculating the difference in consecutive rowsA practical example: calculating the 7 days difference for time-series data
Shifting values with periods
Shifting time-series data with freq
A practical example: calculating the difference in consecutive rows
A practical example: calculating the 7 days difference for time-series data
Please check out Notebook for the source code.
Pandas shift() shift index by the desired number of periods. The simplest call should have an argument periods (It defaults to 1) and it represents the number of shifts for the desired axis. And by default, it is shifting values vertically along the axis 0 . NaN will be filled for missing values introduced as a result of the shifting.
Let’s see how this works with the help of an example.
df = pd.DataFrame({    "A": [1, 2, 3, 4, 5],    "B": [10, 20, 30, 40, 50]})
To shift index by 1 row vertically
df.shift(periods=1)# You can do below insteaddf.shift(1)
To replace NaN , you can use the argument fill_value , for example, replace NaN with 0
df.shift(periods=1, fill_value=0)
In addition, you can pass a negative number to periods and it will shift values in the oppositive direction.
To shift values horizontally, you can set axis=1
df.shift(periods=1, axis=1)
Pandas shift() function has an argument called freq which allows you to do the frequency-based shifting. The Pandas shift() function is beneficial when dealing with time-series data.
In order to use the argument freq, you need to make sure the index of DataFrame is date or datetime, otherwise, it will raise a NotImplementedError.
Let’s see how this works with the help of an example.
df = pd.DataFrame({        "A": [1, 2, 3, 4, 5],        "B": [10, 20, 30, 40, 50]    },      index=pd.date_range("2020-01-01", freq='D', periods=5))
To shift the index by 10 days.
df.shift(freq='10D')# The equivalentdf.shift(periods=10, freq='D')
You will get the following result if run with periods only. Certainly, that isn’t what we want.
Suppose you need to use the previous row value to calculate the sales change, Pandas shift() would be a way to achieve this task.
df = pd.DataFrame({    "date": pd.date_range("2020-01-01", freq='D', periods=5),    "sales": [22, 30, 32, 25, 42]})
To calculate the sales change in consecutive rows.
df['diff'] = df['sales'] - df.shift(1)['sales']
This function is beneficial when dealing with time-series data. Let’s take a look at another practical example.
Now, suppose you have been asked to calculate the 7 days sales change as follows
value_1 = Day_8 - Day_1value_2 = Day_9 - Day_2value_3 = Day_10 - Day_3...value_n = Day_N - Day_N-7
Pandas shift() with the argument freq would be an ideal way to achieve this task. Let’s use read_csv() with the argument parse_dates and index_col to load data into a DataFrame.
df = pd.read_csv(    'data/sales.csv',     parse_dates=['date'],     index_col=['date'],)
Notice that the record for “2020–01–08” is missing in the DataFrame. By running df.shift(freq='7D') , you should get a result with a record for “2020–01–08”.
Then, to calculate the 7 days sales change:
the_7_days_diff = df['sales'] - df.shift(freq='7D')['sales']
Notice that
There is a record for “2020–01–08” in the result
The value of “2020–01–08” is NaN because df doesn’t have this value
Values for the date from “2020–01–01” to “2020–01–07” are NaN. This is because df.shift(freq='7D') doesn’t have these values.
The last 6 records are NaN because df doesn’t have these values
Pandas shift() function can be very useful when you need to push all rows in a DataFrame or require to use the previous row in a DataFrame. The function is beneficial when dealing with time-series data, as shown in the above examples.
I hope this article will help you to save time in analyzing data. I recommend you to check out the documentation for the shift() API and to know about other things you can do.
Thanks for reading. Please check out the notebook for the source code and stay tuned if you are interested in the practical aspect of machine learning.
Using Pandas method chaining to improve code readability
How to do a Custom Sort on Pandas DataFrame
When to use Pandas transform() function
Pandas concat() tricks you should know
Difference between apply() and transform() in Pandas
All the Pandas merge() you should know
Working with datetime in Pandas DataFrame
Pandas read_csv() tricks you should know
4 tricks you should know to parse date columns with Pandas read_csv()
More tutorials can be found on my Github