Going From Stata to Pandas
By: Adam Ross Nelson
Tags: Data Visualization, Programming, Statistics, Dataviz, Data Science

Going From Stata to Pandas

A Single Guide In Three Parts, Overview

This guide is for users of Stata that want to begin learning Pandas. It has been written using examples and workflows that Stata users will know well. Also, this article references data many Stata users will recognize.

In Part 1.1 of this guide, I review a variety of reasons Stata users might like to begin exploring the option to work with Python and Pandas.

In the next section, Part 1.2, I demonstrate cross-tabulations and summary statistics that will help you begin the data exploration process.

Lastly, in Part 1.3, I illustrate a variety of simple visualizations that will help you continue the data exploration process.

Part 1.1: So, you installed Python and Pandas but you’re not sure the next steps.

Both Stata and Pandas provide complete (or near-complete) platforms for data manipulation, visualization, statistics, etc. Stata purports to be …

A complete, integrated software package that provides all your data science needs — data manipulation, visualization, statistics, and reproducible reporting. (stata.com)

Pandas purports to be…

An open source, BSD-licensed library providing high-performance, easy-to-use data structures and data analysis tools for the Python programming language. (pandas.pydata.org)

Reasons Stata users might look to learn Pandas

Stata & Pandas Marketing Images.

A collaborator is using Pandas. You’re a Stata user but the research group you’re collaborating with isn’t. They use Pandas. To be in the same sandbox you decide to learn Pandas.

You’ve decided you want to expand your skillset. Ready for a new challenge? Maybe you’ve read several books from Stata’s bookstore. You’ve created and published user-written programs (known to Stata users as packages). Maybe you’ve even published one or more programs to the user community’s preferred repository at Boston College Statistical Software Components (SSC) archive. Clearly then, you’re ready to expand your skillset.

You’ve lost access to Stata. Or you anticipate losing access to Stata. Stata is proprietary and its licensing fees are costly. If you have lost or anticipate losing access to Stata and you’re not interested in purchasing a license for yourself (despite the good value), Pandas is a good option.

Other reasons. It doesn’t really matter why a Stata user might look to learn Pandas. This guide is an agnostic how-to that acknowledges both languages have their strengths and weaknesses. In my work, I often find that one picks up where the other leaves off.

This guide assumes you have installed Python and Pandas

To avoid reinventing the wheel, this guide does not discuss Python or Pandas installation beyond offering a recommendation that Stata users consider installing the intended-for-science Anaconda distribution.

The examples and explanations given in this guide have been tested using the Anaconda distribution. Specifically, examples given here were created in Jupyter Notebooks which is software included in the Anaconda distribution of Python.

Using Stata’s well-known example automobile dataset. I have previously discussed this dataset as one that is useful for testing or training purposes. Here is what loading this dataset in a Jupyter Notebook will typically look:

Jupyter Notebook running a Python 3 Kernel. Pandas illustration using Stata’s example automobile dataset.

Next in Part 1.2, I’ll demonstrate cross-tabulations and summary statistics that will help you begin the data exploration process. In the last section, Part 1.3, I will illustrate simple visualizations that will let you continue the data exploration process.

Part 1.2: After installing Python and Pandas here are your next steps.

Loading The Auto Dataset

Loading the automobile dataset in Stata is easy: sysuse auto . In Python the process can be done in two lines. But to make the code more readable, I break it into three lines:

import pandas as pd auto = 'http://www.stata-press.com/data/r15/auto2.dta'

df = pd.read_stata(auto)

That’s it! Easy. If you’re a new Pandas user, and if you’re following along congratulations! You’ve loaded your first dataset into memory.

One of the first things you’ll notice is that working in Python, there is less ‘output.’ Output is information the language provides to help validate that the command did for you what you intended for it to do. For more on this, see my related article on Merging Data: The Pandas Missing Output.

In Python, this feedback is much less extensive. To verify you’ve properly loaded the automobile dataset. Try the code df.head() which is roughly equivalent to Stata’s list in 1/5 . Here is what you should see:

Jupyter Notebook running a Python Kernel. Pandas illustration using Stata’s automobile dataset.

Cross-Tabulation

A cross-tabulation is one of the simplest analytical techniques available. This technique involves tabulating occurrences among two or more categorical variables.

A limitation of Stata is that cross-tabulations involving more than two or three categorical variables is limited. In Pandas three or more categoricals is no problem.

A two-way tabulation with the automobile data set:

pd.crosstab(df['rep78'], df['foreign'])

Jupyter Notebook running a Python Kernel. A cross-tabulation of two variables (repair record, and manufacturing origins). Pandas illustration using Stata’s automobile dataset.

Looking closely, this tabulation leaves out missing data. To get Stata to include missing data in its cross-tabulations the mi option following the tab command will provide a more complete view of the data. Putting it together in Stata it would be tab foreign rep78, mi .

In Pandas two lines are needed to display a more complete view of the data:

# Specify a new category among the repair records.

df['rep78'].cat.add_categories('No Record', inplace=True) # Generate a tabulation using the fillna() method.

pd.crosstab(df['foreign'], df['rep78'].fillna('No Record'))

Jupyter Notebook running a Python Kernel. A cross-tabulation of two variables (repair record, and manufacturing origins, including missing data). Pandas illustration using Stata’s automobile dataset.

Summary Statistics

Summary statistics in Stata are not complicated. One of the quickest ways to get a look at continuous variables is to type sum . With the automobile data set this gives you:

Summary statistics from the Stata automobile dataset.

In Pandas the command that most closely mimics this output is df.describe().transpose() which will give:

Jupyter Notebook running a Python Kernel. Summary statistics with the variable names across rows and statistics down the columns. Pandas illustration using Stata’s automobile dataset.

Most Pandas users would use df.describe() which displays variable names across the top of the output and the statistics across each row. As a Stata user, I add the transpose() method which rotates the output to more closely match the convention that I’m used to with Stata.

Bonus, in Pandas the default output also gives the 25th, 50th, and 75th percentile. Of course, the 50th percentile is the ever-useful-to-know median. These additional statistics are not offered in Stata’s default output.

If you’re following along and feeling adventurous, test your intuition by executing the following:

df[['price','mpg','foreign']].groupby('foreign').describe()

See if you can correctly anticipate what the output will be. Also, work on adjusting the code to customize it so that you can explore other variables beyond price (price) or mileage (mpg).

Up next Part 1.3, illustrates a variety of simple visualizations that will let you continue the data exploration process.

Part 1.3: After cross-tabulating and summarizing your data proceed with simple visualizations.

Histograms and Scatter Plots Get Upgraded

One of my favorite parts about conducting data exploration in Pandas instead of Stata is the combination of scatterplots and histograms — in the same plot.

A histogram illustrates the distribution of one variable at a time. A scatterplot illustrates the relationship of two or more variables.

Pandas provides ready access to the jointplot which combines the best of both scatterplots and histograms into one. Again working with Stata’s automobile dataset:

import pandas as pd

import seaborn as sns

%matplotlib inline auto = 'http://www.stata-press.com/data/r15/auto2.dta'

df = pd.read_stata(auto) sns.jointplot(x='weight',y='mpg',data=df, kind='scatter')

Jupyter Notebook running a Python Kernel. Jointplot illustrating the relationship between mileage (mpg) and automobile weight in pounds (weight). Pandas illustration using Stata’s automobile dataset.

This visualization explores the relationship between mileage (in miles per gallon on the verticle y axis) and vehicle weight (in pounds on the x axis). The visualization indicates, consistent with intuition and common understanding, that as vehicle weight increases, the vehicle mileage decreases.

This jointplot shows the relationship between these two variables (weight and mpg) in addition to visualizing the distribution of these to variables. As an added bonus this visualization indicates the Pearson correlation coefficient for these two variables along with its p-value.

The Pairplot

Also, handy for comparing many variables across a dataset is the pairplot. The pairplot provides a matrix of scatterplots, similar to a typical correlation matrix. The following will compare mileage (mpg), weight (weight), length (length), and price (price):

sns.pairplot(df[['mpg','weight','length','price']])

Jupyter Notebook running a Python Kernel. Pairplot illustrating the relationship between mileage (mpg), weight (weight), length (length), and price (price). Pandas illustration using Stata’s automobile dataset.

Bonus to add a categorical comparison, here we’ll compare foreign manufacturing with domestic, we can adjust the code to include a couple of additional arguments:

sns.pairplot(df[['mpg','weight','length','price','foreign']],

kind='scatter', plot_kws={'alpha': 0.5},

hue='foreign')

Jupyter Notebook running a Python Kernel. Pairplot illustrating the relationship between mileage (mpg), weight (weight), length (length), and price (price). In this visualization, we also compare foreign and domestic manufacturing origins. Pandas illustration using Stata’s automobile dataset.

After adding the additional arguments which generate a visualization that lets you compare domestically manufactured vehicles with foreign manufacture new patterns begin to go emerge which might inform further analysis.

Conclusion

In three parts, this article is designed to help users of Stata think about making the transition to Python and Pandas. If the goal isn’t to transition, this guide will help Stata users begin learning a new tool that can complement your on-going work in Stata. Both Stata and Pandas have their strengths and weaknesses.

At the most basic level, once you reach an intermediate or beyond level of proficiency with Stata it is worth adding to your skillset by also learning Python and Pandas. This article demonstrates techniques that will help you begin the data exploration process.

Part 1.1 reviewed a variety of reasons Stata users might like to begin exploring the option to work with Python and Pandas. Parts 1.2 & 1.3 walkthrough rudimentary examples of data exploration, analysis, and visualization using Stata’s popular automobile dataset.

Thanks For Reading

Thanks for reading. Send me your thoughts and ideas. You can write just to say hey. And if you really need to tell me how I got it wrong I look forward to chatting soon. Twitter: @adamrossnelson LinkedIn: Adam Ross Nelson.