Patsy: Build Powerful Features with Arbitrary Python Code
By: Khuyen Tran
Tags: Feature Engineering, Editors Pick, Data Science, Data Visualization, Machine Learning

Wouldn’t it be nice if you could quickly create features using arbitrary Python code? That is when Patsy comes in handy.

What is Patsy?

Patsy is a Python library that allows data transformation using arbitrary Python code.

With Patsy, you could use human-readable syntax such as life_expectancy ~ income_group + year + region (life expectancy depends on income group, year, and region).

To install Patsy, type:

pip install patsy

Getting Started

To try out the library, we will use the Life Expectancy at Birth for different countries dataset from Kaggle. To make it easier for us to observe the outputs, we will take only a subset of the original data.

Our task is to predict the life expectancy of a region in a particular year.

Since income group, region, and year may play a big part in determining the life expectancy, we will use Patsy to transform these 3 features into a matrix.

In the code above,

outcome is a DesignMatrix object that represents life_expectancy , which is the variable we want to predict

>>> outcome DesignMatrix with shape (140, 1)

life_expectancy

60.73805

59.74837

68.94266

46.45793

54.71871

[135 rows omitted]

Terms:

'life_expectancy' (column 0)

(to view full data, use np.asarray(this_obj))

predictors is a DesignMatrix object that represents the combination of income_group , region , and year

>>> predictors DesignMatrix with shape (140, 11)

Columns:

['Intercept',

'income_group[T.Low income]',

'income_group[T.Lower middle income]',

'income_group[T.Upper middle income]',

'region[T.Europe & Central Asia]',

'region[T.Latin America & Caribbean]',

'region[T.Middle East & North Africa]',

'region[T.North America]',

'region[T.South Asia]',

'region[T.Sub-Saharan Africa]',

'year']

Terms:

'Intercept' (column 0)

'income_group' (columns 1:4)

'region' (columns 4:10)

'year' (column 10)

(to view full data, use np.asarray(this_obj))

Cool! It seems like we got a feature matrix of shape (140, 11) . 140 indicates that there are 140 examples, and 11 indicates that there are 11 features. These 11 features are:

Intercept (column 0) — an array of 1s

income_group (column 1 to 4)

(column 1 to 4) region (column 4 to 10)

(column 4 to 10) year (column 10)

Why are there extra columns for the income_group and regions features? It is because Patsy encodes these categorical features using One-Hot encoding so they can be interpreted by a machine learning model.

That means, a categorical column:

income_group

1. low_income

2. upper_middle_income

3. lower_middle_income

4. low_income

5. upper_middle_income

… will be transformed into a matrix of 0s and 1s:

low_income lower_middle_income upper_middle_income

1. 1 0 0

2. 0 0 1

3. 0 1 0

4. 1 0 0

5. 0 0 1

Okay, I see how easy it is to create features with Patsy. But how can I use these features?

The features created by Patsy could be easily used by popular machine learning libraries like scikit-learn. In other words, you can treat the DesignMatrix object like a NumPy array.

This means you can split the DesignMatrix object into train set and test set using train_test_split :

… and feed the train set and test set to a machine learning model.

We will use Yellowbrick’s prediction error plot to compare between the actual targets and the predicted values generated by our model.

To install Yellowbrick, type:

pip install yellowbrick

Image by Author

In the plot above,

y is the axis that presents real values

ŷ is the axis that presents predicted values

The black dotted line is the fitted line created by the current model

The gray dotted line is where the fitted line would be if the predicted values exactly matched the real values

The linear model follows the observed targets pretty well. The R² score is 0.911. Let’s create a function so that we can observe the fitted line given the equation.

What happens if we predict the target values using only the income_group column?

Image by Author

The R² score is pretty low. We can see if the output is better by combining income_group column and region column:

Image by Author

The output doesn’t look very good either. Let’s try to combine income_group and year :

Image by Author

Aha! Combining the income_group and year features yield a very good result. This could be because income group and year strongly affect the life expectancy of a country in a year.

Built-in Functions

Sometimes, we might want to apply extra processing to a feature such as scaling or standardizing. Luckily, Patsy provides built-in functions that make it easy for us to do that.

To demonstrate these functions, let’s switch to the Bikeshare dataset and take only 2k samples from the data.

Note that the columns season , year , month , hour , holiday , workingday , and weather are categorical variables. However, since their data type is an integer, Patsy considers them as numerical variables.

DesignMatrix with shape (2000, 4)

Intercept year hour temp

1 0 4 0.58

1 1 0 0.46

1 0 0 0.44

1 1 7 0.64

1 1 3 0.44

[1995 rows omitted]

Terms:

'Intercept' (column 0)

'year' (column 1)

'hour' (column 2)

'temp' (column 3)

(to view full data, use np.asarray(this_obj))

Specify Categorical Variables

To solve the problem above, we can specify some columns as being categorical by using C(column) :

DesignMatrix with shape (2000, 26)

Columns:

['Intercept',

'C(year)[T.1]',

'C(hour)[T.1]',

'C(hour)[T.2]',

....

'C(hour)[T.22]',

'C(hour)[T.23]',

'temp']

Terms:

'Intercept' (column 0)

'C(year)' (column 1)

'C(hour)' (columns 2:25)

'temp' (column 25)

(to view full data, use np.asarray(this_obj))

The features are better represented now.

Standardize

We can also standardize a particular feature (subtract the mean and divide by the standard deviation) by using the standardize method:

This function will transform the initial distribution of humidity :

Image by Author

… to this distribution:

Image by Author

You can find other built-in methods Patsy provides here.

Custom Functions

Besides the built-in functions, you can also write your own function, such as multiplying a column by two:

The first 5 rows of the temp feature:

temp

0.58

0.46

0.44

0.64

0.44

The first 5 rows of the multiply_by_two(temp) feature:

multiply_by_two(temp)

1.16

0.92

0.88

1.28

0.88

Operators

Besides the + operator, you can use other operators to combine features in other ways.

- Operator

This operator is used to remove terms on the right. For example, using month + temp — 1 will remove the intercept:

DesignMatrix with shape (2000, 2)

month temp

9 0.58

10 0.46

5 0.44

6 0.64

1 0.44

[1995 rows omitted]

Terms:

'month' (column 0)

'temp' (column 1)

(to view full data, use np.asarray(this_obj))

: Operator

This operator computes the interaction between each term on the left and each term on the right. For example, using riders ~ month:temp will divide the values in the month column by the values in the temp column.

DesignMatrix with shape (2000, 2)

Intercept month:temp

1 5.22

1 4.60

1 2.20

1 3.84

1 0.44

[1995 rows omitted]

Terms:

'Intercept' (column 0)

'month:temp' (column 1)

If there are multiple terms on each side, each term on the left will be divided by each term on the right. For example,

(C(year) + C(hour) + C(workingday)) : temp

is expanded to:

C(year):temp + C(hour):temp + C(workingday):temp

Let’s plot the equation above and see how the output looks like:

Image by Author

We got an R² of 0.709. This is a pretty good score!

* Operator

a:b includes the interaction between a and b but doesn’t include the features a and b themselves. If you want to include both the variables and the interaction between variables, use a * b .

For example,

month * temp

is shorthand for:

month + temp + month:temp

** Operator

This operator computes the * of a set of terms with itself n times.

For example,

(temp + humidity + windspeed)**3

is expanded to:

(temp + humidity + windspeed) * (temp + humidity + windspeed) * (temp + humidity + windspeed)

Let’s plot the equation riders ~ (C(year) + C(hour) + C(workingday) + temp) ** 3 and see how it looks like:

Image by Author

Wow! We got an R² of 0.812, which is a pretty good score. Does it mean having more features results in a better output? Let’s try to switch 3 with 6 and see what happens.

Image by Author

Not really. Having more features can result in overfitting, so adding more features isn’t necessarily better.

You can find all operators and their uses here.

Conclusion

Congratulations! You have just learned how to use Patsy to build features with arbitrary Python code. I hope this tool will make it easier for you experiment with different combinations of features and understand their impacts on the outputs.

Feel free to play and fork the source code here: