Repository: ericmjl/pyjanitor
Pull request #807 (2021-02-27 11:57:51)

[ENH] groupby_agg refactor

# PR Description

Please describe the changes proposed in the pull request:

- Add `dropna` parameter, to include null values in groupby column. (thanks to minimum Pandas version 1.1)
- Add examples to docstrings


**This PR improves groupby_agg.**

<!-- As you go down the PR template, please feel free to delete sections that are irrelevant. -->

# PR Checklist

<!-- This checklist exists for newcomers who are not yet familiar with our requirements. If you are experienced with
the project, please feel free to delete this section. -->

Please ensure that you have done the following:

1. [x] PR in from a fork off your branch. Do not PR from `<your_username>`:`dev`, but rather from `<your_username>`:`<feature-branch_name>`.
<!-- Doing this helps us keep the commit history much cleaner than it would otherwise be. -->
2. [x] If you're not on the contributors list, add yourself to `AUTHORS.rst`.
<!-- We'd like to acknowledge your contributions! -->
3. [x] Add a line to `CHANGELOG.md` under the latest version header (i.e. the one that is "on deck") describing the contribution.
    - Do use some discretion here; if there are multiple PRs that are related, keep them in a single line.

## Quick Check

To do a very quick check that everything is correct, follow these steps below:

- [x] Run the command `make check` from pyjanitor's top-level directory. This will automatically run:
    - black formatting
    - flake8 checking
    - running the test suite
    - docs build

Once done, please check off the check-box above.

If `make check` does not work for you, you can execute the commands listed in the Makefile individually.

## Code Changes

<!-- If you have not made code changes, please feel free to delete this section. -->

If you are adding code changes, please ensure the following:

- [x] Ensure that you have added tests.
- [x] Run all tests (`$ pytest .`) locally on your machine.
    - [x] Check to ensure that test coverage covers the lines of code that you have added.
    - [x] Ensure that all tests pass.

## Documentation Changes

<!-- If you have not made documentation changes, please feel free to delete this section. -->

If you are adding documentation changes, please ensure the following:

- [x] Build the docs locally.
- [x] View the docs to check that it renders correctly.

# Relevant Reviewers

<!-- Finally, please tag relevant maintainers to review. -->

Please tag maintainers to review.

- @ericmjl


codecov[bot]: # [Codecov](https://codecov.io/gh/ericmjl/pyjanitor/pull/807?src=pr&el=h1) Report
> Merging [#807](https://codecov.io/gh/ericmjl/pyjanitor/pull/807?src=pr&el=desc) (2f8b97c) into [dev](https://codecov.io/gh/ericmjl/pyjanitor/commit/c20e4973fa77c2ece189a1721e3e52079771253f?el=desc) (c20e497) will **decrease** coverage by `0.58%`.
> The diff coverage is `99.07%`.


```diff
@@            Coverage Diff             @@
##              dev     #807      +/-   ##
==========================================
- Coverage   96.08%   95.49%   -0.59%     
==========================================
  Files          17       19       +2     
  Lines        1046     1952     +906     
==========================================
+ Hits         1005     1864     +859     
- Misses         41       88      +47     
```

samukweku: @ericmjl ; once `complete_refactor` is ok, this one should be resolved; one of the fails is dependent on the `complete_refactor` PR
ericmjl: Let's merge!

Patch:
diff --git a/CHANGELOG.md b/CHANGELOG.md
index 3dbf1cb..469ca60 100644
--- a/CHANGELOG.md
+++ b/CHANGELOG.md
@@ -1,6 +1,11 @@
 # Changelog
 
 ## [Unreleased]
+
+- [ENH] Add `dropna` parameter to groupby_agg. @samukweku
+- [ENH] `complete` adds a `by` parameter to expose explicit missing values per group, via groupby. @samukweku
+
+- [ENH] Add `dropna` parameter to groupby_agg. @samukweku
 - [ENH] `complete` adds a `by` parameter to expose explicit missing values per group, via groupby. @samukweku
 
 ## [v0.20.13] - 2021-02-25
diff --git a/janitor/functions.py b/janitor/functions.py
index 303c613..e41da21 100644
--- a/janitor/functions.py
+++ b/janitor/functions.py
@@ -3668,8 +3668,10 @@ def groupby_agg(
     new_column_name: str,
     agg_column_name: str,
     agg: Union[Callable, str],
+    dropna: bool = True,
 ) -> pd.DataFrame:
-    """Shortcut for assigning a groupby-transform to a new column.
+    """
+    Shortcut for assigning a groupby-transform to a new column.
 
     This method does not mutate the original DataFrame.
 
@@ -3688,9 +3690,75 @@ def groupby_agg(
         df = pd.DataFrame(...).groupby_agg(by='group',
                                            agg='mean',
                                            agg_column_name="col1"
-                                           new_column_name='col1_mean_by_group')
+                                           new_column_name='col1_mean_by_group',
+                                           dropna = True/False)
+
+    Examples::
 
-    Example Link : https://pyjanitor.readthedocs.io/notebooks/groupby_agg.html
+        import pandas as pd
+        import janitor as jn
+
+            group  var1
+        0      1     1
+        1      1     1
+        2      1     1
+        3      1     1
+        4      1     2
+        5      2     1
+        6      2     2
+        7      2     2
+        8      2     2
+        9      2     3
+
+    Let's get the count per `group` and `var1`::
+
+        df.groupby_agg(
+            by = ['group', 'var1'],
+            agg = 'size',
+            agg_column_name = 'var1',
+            new_column_name = 'count'
+        )
+
+            group  var1  size
+        0      1     1     4
+        1      1     1     4
+        2      1     1     4
+        3      1     1     4
+        4      1     2     1
+        5      2     1     1
+        6      2     2     3
+        7      2     2     3
+        8      2     2     3
+        9      2     3     1
+
+    If the data has null values,
+    you can include the null values by passing `False` to `dropna`;
+    this feature was introduced in Pandas 1.1::
+
+            name   type  num  nulls
+        0  black  chair    4    1.0
+        1  black  chair    5    1.0
+        2  black   sofa   12    NaN
+        3    red   sofa    4    NaN
+        4    red  plate    3    3.0
+
+    Let's get the count, including the null values,
+    grouping on `nulls` column::
+
+        df.groupby_agg(
+            by="nulls",
+            new_column_name="num_count",
+            agg_column_name="num",
+            agg="size",
+            dropna=False,
+        )
+
+            name   type  num  nulls  num_count
+        0  black  chair    4    1.0          2
+        1  black  chair    5    1.0          2
+        2  black   sofa   12    NaN          2
+        3    red   sofa    4    NaN          2
+        4    red  plate    3    3.0          1
 
     :param df: A pandas DataFrame.
     :param by: Column(s) to groupby on, either a `str` or
@@ -3698,31 +3766,15 @@ def groupby_agg(
     :param new_column_name: Name of the aggregation output column.
     :param agg_column_name: Name of the column to aggregate over.
     :param agg: How to aggregate.
+    :param dropna: Whether or not to include null values,
+        if present in the `by` column(s). Default is True.
     :returns: A pandas DataFrame.
     """
     df = df.copy()
-    # convert to list
-    # needed when creating a mapping through the iteration
-    if isinstance(by, str):
-        by = [by]
-    # this is a temporary measure, till the minimum Pandas version is 1.1,
-    # which supports null values in the group by
-    # If any of the grouping columns has null values, we temporarily
-    # replace the values with some outrageous value, that should not exist
-    # in the column. Also, the hasnans property is significantly faster than
-    # .isnull().any()
-    if any(df[col].hasnans for col in by):
-
-        mapping = {
-            column: ".*^%s1ho1go1logoban?*&-|/\\gos1he()#_" for column in by
-        }
 
-        df[new_column_name] = (
-            df.fillna(mapping).groupby(by)[agg_column_name].transform(agg)
-        )
-
-    else:
-        df[new_column_name] = df.groupby(by)[agg_column_name].transform(agg)
+    df[new_column_name] = df.groupby(by, dropna=dropna)[
+        agg_column_name
+    ].transform(agg)
     return df
 
 


Test patch:
diff --git a/tests/functions/test_groupby_agg.py b/tests/functions/test_groupby_agg.py
index e3c74c9..123403b 100644
--- a/tests/functions/test_groupby_agg.py
+++ b/tests/functions/test_groupby_agg.py
@@ -113,15 +113,16 @@ def test_groupby_agg_null_columns(df):
     the right aggregated value.
     """
     expected = df.copy()
-    expected["num_count"] = (
-        expected.fillna("rar").groupby(["nulls"])["type"].transform("size")
-    )
+    expected["num_count"] = expected.groupby(["nulls"], dropna=False)[
+        "type"
+    ].transform("size")
     assert_frame_equal(
         df.groupby_agg(
             by="nulls",
             new_column_name="num_count",
             agg_column_name="num",
             agg="size",
+            dropna=False,
         ),
         expected,
     )