Ultimate Pandas Guide — Inspecting Data Like a Pro | by Skyler Dale | Towards Data Science

Whether you’re working on a simple analysis or a complex machine learning model, there’s a lot of value in being able to answer quick, exploratory questions about the nature of your data.
Fortunately, Pandas makes this easy. In this post, I’ll walk through several DataFrame attributes and methods that make data inspection painless and productive.
Use Case: Taking a quick look at our DataFrame.
By default, the head method returns the first 5 rows of our data:
#returns the first five rowsdf.head()
But we can also specify a different number of rows to return:
#returns the first six rows df.head(6)
The tail method works the same way, but returns the rows starting from the bottom of the DataFrame instead of the top:
df.tail(2)
Use Case: Get the column names from our DataFrame.
df.columns
I find this useful when I need to remember the exact spelling of a column name or when I’m discovering which data points are available.
We can also convert the index to a list if this object is more convenient:
list(df.columns)
Use Case: Determine the # of rows and columns in our DataFrame.
This method returns a tuple whose first value is the number of rows and whose second value is the number of columns:
df.shape
To get only the number of rows, we simply index the tuple to pull out the first element:
df.shape[0]
I like to use this attribute to find the percentage of my data that falls within a certain condition.
For example, we can filter a DataFrame to get the percentage of values that are less than 70 in one column and less than 1000 in another column:
df[ (df.ints < 70) & (df.floats < 1000) ].shape[0] / df.shape[0]
Use case: Get the distinct values in a DataFrame column (ie. series object).
df[‘strings’].unique()
Value_counts is similar to unique, but instead of returning an array of unique values, it returns a series with the frequency for each value:
df[‘strings’].value_counts()
Use Case: Get summary information about our index, columns, and memory usage.
df.info(verbose = True)
We pass “True” to the “verbose” parameter to ensure we’ll get the full column summary when we are working with a larger dataset.
Use Case: Get summary statistics about the columns in our data.
By default, the describe method only includes numeric columns:
df.describe()
We can customize our output in a couple of ways.
First, we can get different percentile values by passing a list of values between 0 and 1 to the “percentiles” parameter:
df.describe(percentiles = [.1, .99])
We can also use the “include” and “exclude” parameters to specify which types of columns we want in our output.
If we pass “all” to the “include” parameter, we will get summary of the non-numeric columns, in addition to the numeric ones.
For the non-numeric columns, we’ll get the count, unique count, top (most common value) and freq (frequency of most common value):
df.describe(include = 'all')
Finally, we can pass a list of data types to the include or exclude parameter if we want to further customize columns in our output.
By default, pandas caps the number of rows and columns we get in our output at 60 and 20, respectively.
If we try to return a DataFrame with 70 rows we’ll get the following:
This can be pretty annoying when we want to see all of our data.
We can fix this by passing either “max_rows” or “max_columns” to the pandas “set_option” method and specifying a value:
pd.set_option(‘max_rows’, 70)
I find this trick very useful when I want the flexibility to see more than a small chunk of my data.
Pandas makes data inspection fairly straightforward with a number of user-friendly methods and attributes. You can also write your own inspection and exploration methods to extend the functionality. This is one of the advantages of using a scripting language like Python for data analysis.