Getting more value from the Pandas count() | by B. Chen | Towards Data Science

The Pandas library is one of the most preferred tools for data manipulation and analysis. Data Scientists often spend most of their time exploring and preprocessing the data. When it comes to data profiling and understanding the dataset, Pandas count() is one of the most commonly used methods for knowing the number of non-NA.
The method is quick, fast, easy to understand, however, most of the time, we end up using count() with the default arguments. In this article, you’ll learn how to get more value from it. This article is structured as follows:
Counting non-NA cells for each column and rowCounting non-NA cells on a MultiIndex DataFrameCounting numeric only with numeric_onlyApplying count() to the groupby() resultCombine the count back into the original DataFrame
Counting non-NA cells for each column and row
Counting non-NA cells on a MultiIndex DataFrame
Counting numeric only with numeric_only
Applying count() to the groupby() result
Combine the count back into the original DataFrame
Please check out Notebook for source code.
Visit Github Repo for other tutorials
Pandas count() is used to count the number of non-NA cells across the given axis. The values None, NaN, NaT, and optionally numpy.inf are considered NA.
The method is counting non-NA for each column by default, for instance
df = pd.DataFrame({    "Person": ["John", "Tom", "Lewis", "John", "Myla"],    "Age": [24., np.nan, 21., 33, 26],    "Single": [False, True, True, None, np.datetime64('NaT')],    "Department": ["Product", "IT", "IT", "IT", "Product"]})>>> df.count()Person        5Age           4Single        3Department    5dtype: int64
If we would like to count non-NA for each row, we can set the axis argument to 1 or 'columns':
>>> df.count(axis = 1)0    41    32    43    34    3dtype: int64>>> df.count(axis = 'columns')0    41    32    43    34    3dtype: int64
A MultiIndex DataFrame allows multiple columns acting as a row identifier and multiple rows acting as a header identifier. For example, let’s load the Titanic dataset with the Sex and Survived columns as the index.
df = pd.read_csv(    'titanic_train.csv',     index_col=['Sex', 'Survived'])df.head()
We can set the argument level to count the non-NA along with a particular index level, for instance
df.count(level='Sex')
df.count(level='Survived')
If you want to learn more about MultiIndex, please check out:
towardsdatascience.com
There is an argument numeric_only in Pandas count() to configure the count on numeric only. The values float, int, and boolean are considered numeric.
The argument defaults to False, and we can set it to True to count on numeric only:
>>> df.count(numeric_only=True)PassengerId    891Pclass         891Age            714SibSp          891Parch          891Fare           891dtype: int64
Pandas groupby() allows us to split data into separate groups to perform computations for better analysis.
One of the common use cases is to group by a certain column and then get the count of another column. For example, let’s group by “Department” column and get the count of “Single” values.
df = pd.DataFrame({    "Person": ["John", "Tom", "Lewis", "John", "Myla"],    "Age": [24., np.nan, 21., 33, 26],    "Single": [False, True, True, None, np.datetime64('NaT')],    "Department": ["Product", "IT", "IT", "IT", "Product"]})>>> df.groupby('Department')['Single'].count()DepartmentIT         2Product    1Name: Single, dtype: int64
Alternatively, we can also use the aggregation agg('count') method.
>>> df.groupby('Department')['Single'].agg('count')DepartmentIT         2Product    1Name: Single, dtype: int64
We have been performing count() to a specific column of groupby() result. Turns out we don’t actually have to specify a column like Single. Without a column, it will apply count() across all columns, for instance:
>>> df.groupby('Department').count()>>> df.groupby('Department').agg('count')
If you want to learn more about groupby(), please check out:
towardsdatascience.com
In some cases, you may want to write the count back to the original DataFrame, for instance, we would like to append a column department_total_single as follows:
The tricky part in this task is that the df.groupby()['Single'].cout() returns a Series with only 2 rows (as shown below) and it won’t match the number of rows in the original DataFrame.
>>> df.groupby('Department')['Single'].count()DepartmentIT         2Product    1Name: Single, dtype: int64
One solution is to convert the above result into a DataFrame and use merge() method to combine the result.
>>> temp_df = df.groupby('Department')['Single'].count().rename('department_total_count').to_frame()>>> temp_df.reset_index()>>> df_new = pd.merge(df, temp_df, on='Department', how='left')
This certainly does our work. But it is a multistep process and requires extra code to get the data in the form we require.
We can solve this effectively using the transform() function. A single line of code can solve the apply and merge.
>>> df.groupby('Department')['Single'].transform('count')0    11    22    23    24    1Name: Single, dtype: int64
We can see that the result of transform() retains the same number of rows as the original dataset. So we can simply assign the result to a new column in the original DataFrame:
>>> df['department_total_single'] = df.groupby('Department')['Single'].transform('count')
If you want to learn more about merge() and transform(), please check out:
towardsdatascience.com
towardsdatascience.com
In this article, we have explored the different use cases of Pandas counts(). It’s very handy and one of the top favorite methods during Exploratory Data Analysis and Data Preprocessing.
I hope this article will help you to save time in learning Pandas. I recommend you to check out the documentation for the counts() API and to know about other things you can do.
Thanks for reading. Please check out the notebook for the source code and stay tuned if you are interested in the practical aspect of machine learning.
9 Pandas value_counts() tricks to improve your data analysis
All Pandas json_normalize() you should know for flattening JSON
Using Pandas method chaining to improve code readability
How to do a Custom Sort on Pandas DataFrame
All the Pandas shift() you should know for data analysis
When to use Pandas transform() function
Pandas concat() tricks you should know
Difference between apply() and transform() in Pandas
All the Pandas merge() you should know
Working with datetime in Pandas DataFrame
Pandas read_csv() tricks you should know
4 tricks you should know to parse date columns with Pandas read_csv()
More tutorials can be found on Github