Repository: pyjanitor-devs/pyjanitor
Pull request #1289 (2023-09-18 08:21:57)

[BUG] fillna downcast fix for `complete` function
# PR Description

Please describe the changes proposed in the pull request:

- remove `downcast=infer` from fillna - pandas no longer supports that option
- remove support for completing on the index - remove complexity
- fix bug in groupby logic - inspired by a [Stack Overflow](https://stackoverflow.com/q/77123843/7175713) question
- callables in dictionary evaluates on the entire dataframe, not a Series

**This PR improves the `complete` function.**

<!-- As you go down the PR template, please feel free to delete sections that are irrelevant. -->

# PR Checklist

<!-- This checklist exists for newcomers who are not yet familiar with our requirements. If you are experienced with
the project, please feel free to delete this section. -->

Please ensure that you have done the following:

1. [x] PR in from a fork off your branch. Do not PR from `<your_username>`:`dev`, but rather from `<your_username>`:`<feature-branch_name>`.
<!-- Doing this helps us keep the commit history much cleaner than it would otherwise be. -->
2. [x] If you're not on the contributors list, add yourself to `AUTHORS.md`.
<!-- We'd like to acknowledge your contributions! -->
3. [x] Add a line to `CHANGELOG.md` under the latest version header (i.e. the one that is "on deck") describing the contribution.
    - Do use some discretion here; if there are multiple PRs that are related, keep them in a single line.

# Automatic checks

There will be automatic checks run on the PR. These include:

- Building a preview of the docs on Netlify
- Automatically linting the code
- Making sure the code is documented
- Making sure that all tests are passed
- Making sure that code coverage doesn't go down.

# Relevant Reviewers

<!-- Finally, please tag relevant maintainers to review. -->

Please tag maintainers to review.

- @ericmjl


ericmjl: <!-- NETLIFY DEPLOY COMMENT GENERATED BY ACTIONS_NETLIFY -->
🚀 Deployed on https://deploy-preview-1289--pyjanitor.netlify.app
codecov[bot]: ## [Codecov](https://app.codecov.io/gh/pyjanitor-devs/pyjanitor/pull/1289?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=pyjanitor-devs) Report
> Merging [#1289](https://app.codecov.io/gh/pyjanitor-devs/pyjanitor/pull/1289?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=pyjanitor-devs) (3ce5fa4) into [dev](https://app.codecov.io/gh/pyjanitor-devs/pyjanitor/commit/8454ce9267320030fde12b2adf5c404678c2e1d4?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=pyjanitor-devs) (8454ce9) will **decrease** coverage by `14.11%`.
> Report is 2 commits behind head on dev.
> The diff coverage is `100.00%`.

```diff
@@             Coverage Diff             @@
##              dev    #1289       +/-   ##
===========================================
- Coverage   97.07%   82.96%   -14.11%     
===========================================
  Files          78       78               
  Lines        4172     4110       -62     
===========================================
- Hits         4050     3410      -640     
- Misses        122      700      +578     
```

ericmjl: @samukweku this branch appears to have a conflict that I'm not confident in resolving. Would you be kind enough to help here please?
ericmjl: # GitBot Summary of Changes

The pull request includes changes to the `complete` function in the `janitor/functions/complete.py` file and its corresponding tests in `tests/functions/test_complete.py`. 

In `complete.py`, the logic for groupby in the `complete` function has been fixed. The support for index has been deprecated and the deprecation warning for fillna in `complete` has been fixed. The `select` function now supports variable arguments. 

In `test_complete.py`, a new test case has been added to raise a ValueError if the passed label is not found. The test case for multiindex names not found has been updated. 

In `CHANGELOG.md`, the changes have been documented under the [Unreleased] section. 

The changes ensure that the `complete` function works as expected and improves its overall functionality.

cc: @samukweku, please check for correctness!

ericmjl: Thank you, @samukweku!
ericmjl: Let's merge, @samukweku! It's been long enough :smile:.

Patch:
diff --git a/CHANGELOG.md b/CHANGELOG.md
index 6ec1b62..44e388c 100644
--- a/CHANGELOG.md
+++ b/CHANGELOG.md
@@ -1,7 +1,10 @@
 # Changelog
 
 ## [Unreleased]
+
+-   [BUG] Fix logic for groupby in complete. Index support deprecated. Fix deprecation warning for fillna in `complete` PR #1289 @samukweku
 -   [ENH] `select` function now supports variable arguments - PR #1288 @samukweku
+
 ## [v0.26.0] - 2023-09-18
 
 -   [ENH] `clean_names` can now be applied to column values. Issue #995 @samukweku
diff --git a/janitor/functions/complete.py b/janitor/functions/complete.py
index 9d1666a..05adfd4 100644
--- a/janitor/functions/complete.py
+++ b/janitor/functions/complete.py
@@ -1,6 +1,5 @@
 from typing import Optional, Union, List, Tuple, Dict, Any
 from pandas.core.common import apply_if_callable
-import numpy as np
 import pandas_flavor as pf
 import pandas as pd
 import functools
@@ -33,14 +32,6 @@ def complete(
     the user is required to ensure that the values are unique 1-D arrays.
     The keys in a dictionary must be present in the dataframe.
 
-    `complete` can also be executed on the names of the index -
-    the index should have names
-    (`df.index.names` or `df.index.name` should not have any None).
-    Groupby is not applicable if `completing` on the index.
-    When `completing` on the index, only the passed levels are returned.
-
-
-
     Examples:
         >>> import pandas as pd
         >>> import janitor
@@ -119,14 +110,14 @@ def complete(
         ...     sort=True
         ... )
            group  item_id item_name  value1  value2
-        0      1        1         a       1       4
-        1      1        2         a       0      99
-        2      1        2         b       3       6
-        3      1        3         b       0      99
-        4      2        1         a       0      99
-        5      2        2         a       0       5
-        6      2        2         b       0      99
-        7      2        3         b       4       7
+        0      1        1         a     1.0     4.0
+        1      1        2         a     0.0    99.0
+        2      1        2         b     3.0     6.0
+        3      1        3         b     0.0    99.0
+        4      2        1         a     0.0    99.0
+        5      2        2         a     0.0     5.0
+        6      2        2         b     0.0    99.0
+        7      2        3         b     4.0     7.0
 
         Limit the fill to only implicit missing values
         by setting explicit to `False`:
@@ -147,47 +138,9 @@ def complete(
         6      2        2         b     0.0    99.0
         7      2        3         b     4.0     7.0
 
-        Expose missing values via the index:
-        >>> index = pd.date_range('1/1/2000', periods=4, freq='T')
-        >>> series = pd.Series([0.0, None, 2.0, 3.0], index=index)
-        >>> DT = pd.DataFrame({'s': series})
-        >>> DT.index.name = 'dates'
-        >>> DT
-                               s
-        dates
-        2000-01-01 00:00:00  0.0
-        2000-01-01 00:01:00  NaN
-        2000-01-01 00:02:00  2.0
-        2000-01-01 00:03:00  3.0
-        >>> dates = {'dates':lambda f: pd.date_range(f.min(), f.max(), freq='30S')}
-        >>> DT.complete(dates)
-                               s
-        dates
-        2000-01-01 00:00:00  0.0
-        2000-01-01 00:00:30  NaN
-        2000-01-01 00:01:00  NaN
-        2000-01-01 00:01:30  NaN
-        2000-01-01 00:02:00  2.0
-        2000-01-01 00:02:30  NaN
-        2000-01-01 00:03:00  3.0
-
-        The above can be solved in a simpler way with `pd.DataFrame.asfreq`:
-
-        >>> DT.asfreq(freq='30S')
-                               s
-        dates
-        2000-01-01 00:00:00  0.0
-        2000-01-01 00:00:30  NaN
-        2000-01-01 00:01:00  NaN
-        2000-01-01 00:01:30  NaN
-        2000-01-01 00:02:00  2.0
-        2000-01-01 00:02:30  NaN
-        2000-01-01 00:03:00  3.0
-
     Args:
         df: A pandas DataFrame.
-        *columns: This refers to the columns to be
-            completed. It can also refer to the names of the index.
+        *columns: This refers to the columns to be completed.
             It could be column labels (string type),
             a list/tuple of column labels, or a dictionary that pairs
             column labels with new values.
@@ -233,7 +186,6 @@ def _computations_complete(
     (
         columns,
         column_checker,
-        index,
         sort,
         by,
         fill_value,
@@ -260,30 +212,23 @@ def _computations_complete(
             df=df,
             columns=columns,
             all_scalars=all_scalars,
-            index=index,
             sort=sort,
         )
     else:
         column_checker = by + column_checker
-        uniques = df.groupby(by, group_keys=True)
-        # apply is basically a for loop
-        # for scenarios where Pandas does not have
-        # a vectorized option
+        uniques = df.groupby(by, sort=False)
         uniques = {
             key: _generic_complete(
                 df=value,
                 columns=columns,
                 all_scalars=all_scalars,
-                index=index,
                 sort=sort,
             )
             for key, value in uniques
         }
-        uniques = pd.concat(uniques, names=column_checker, copy=False)
-        if not index:
-            uniques = uniques.droplevel(-1, axis=0)
+        uniques = pd.concat(uniques, names=by, copy=False)
+        uniques = uniques.droplevel(-1, axis=0)
     columns = df.columns
-    index_labels = df.index.names
     indicator = False
     if fill_value is not None and not explicit:
         # to get a name that does not exist in the columns
@@ -299,86 +244,63 @@ def _computations_complete(
     )
 
     if fill_value is not None:
-        if is_scalar(fill_value):
-            # faster when fillna operates on a Series basis
-            fill_value = {
-                col: fill_value for col in columns if out[col].hasnans
-            }
-        if explicit:
-            out = out.fillna(fill_value, downcast="infer")
-        else:
-            # keep only columns that are not part of column_checker
-            # IOW, we are excluding columns that were not used
-            # to generate the combinations
-            fill_value = {
-                col: value
-                for col, value in fill_value.items()
-                if col not in column_checker
-            }
-            if fill_value:
-                # when explicit is False
-                # use the indicator parameter to identify rows
-                # for `left_only`, and fill the relevant columns in fill_value
-                # with the associated value.
-                boolean_filter = out.loc[:, indicator] == "left_only"
-                out = out.drop(columns=indicator)
-                # iteration used here,
-                # instead of assign (which is also a for loop),
-                # to cater for scenarios where the column_name is not a string
-                # assign only works with keys that are strings
-                # Also, the output will be floats (for numeric types),
-                # even if all the columns could be integers
-                # user can always convert to int if required
-                for column_name, value in fill_value.items():
-                    # for categorical dtypes, set the categories first
-                    if isinstance(out[column_name].dtype, pd.CategoricalDtype):
-                        out[column_name] = out[column_name].cat.add_categories(
-                            [value]
-                        )
-                    out.loc[boolean_filter, column_name] = value
-
-    if index and (index_labels != out.index.names):
-        labels = [label for label in index_labels if label in out.index.names]
-        return out.reorder_levels(order=labels, axis="index")
+        null_columns = [col for col in columns if out[col].hasnans]
+        if null_columns:
+            if is_scalar(fill_value):
+                # faster when fillna operates on a Series basis
+                fill_value = {col: fill_value for col in null_columns}
+            if explicit:
+                out = out.fillna(fill_value)
+            else:
+                # keep only columns that are not part of column_checker
+                # IOW, we are excluding columns that were not used
+                # to generate the combinations
+                fill_value = {
+                    col: value
+                    for col, value in fill_value.items()
+                    if col not in column_checker
+                }
+                if fill_value:
+                    # when explicit is False
+                    # use the indicator parameter to identify rows
+                    # for `left_only`, and fill the relevant columns
+                    # in fill_value with the associated value.
+                    boolean_filter = out.loc[:, indicator] == "left_only"
+                    out = out.drop(columns=indicator)
+                    # iteration used here,
+                    # instead of assign (which is also a for loop),
+                    # to cater for scenarios where the column_name
+                    # is not a string
+                    # assign only works with keys that are strings
+                    # Also, the output wil be floats (for numeric types),
+                    # even if all the columns could be integers
+                    # user can always convert to int if required
+                    for column_name, value in fill_value.items():
+                        # for categorical dtypes, set the categories first
+                        if isinstance(
+                            out[column_name].dtype, pd.CategoricalDtype
+                        ):
+                            out[column_name] = out[
+                                column_name
+                            ].cat.add_categories([value])
+                        out.loc[boolean_filter, column_name] = value
     if not out.columns.equals(columns):
-        return out.reindex(columns=columns)
+        return out.loc(axis=1)[columns]
     return out
 
 
 def _generic_complete(
-    df: pd.DataFrame, columns: list, all_scalars: bool, index: bool, sort: bool
+    df: pd.DataFrame, columns: list, all_scalars: bool, sort: bool
 ):
     """Generate cartesian product for `_computations_complete`.
 
     Returns a DataFrame, with no duplicates.
     """
     if all_scalars:
-        if index:
-            uniques = {
-                col: pd.factorize(df.index.get_level_values(col), sort=sort)[
-                    -1
-                ]
-                for col in columns
-            }
-        else:
-            uniques = {
-                col: pd.factorize(df[col], sort=sort)[-1] for col in columns
-            }
+        uniques = {
+            col: pd.factorize(df[col], sort=sort)[-1] for col in columns
+        }
         uniques = _computations_expand_grid(uniques)
-        if index:
-            # it is assured that scalars cannot be single
-            # hence a MultiIndex
-            uniques = {key[0]: value for key, value in uniques.items()}
-            data = list(uniques.values())
-            uniques = pd.MultiIndex.from_arrays(data, names=uniques)
-            if df.columns.nlevels == 1:
-                columns = pd.Index([], name=df.columns.name)
-            else:
-                length = len(df.columns.names)
-                columns = pd.MultiIndex.from_arrays(
-                    [[]] * length, names=df.columns.names
-                )
-            return pd.DataFrame([], index=uniques, columns=columns, copy=False)
         uniques = pd.DataFrame(uniques, copy=False)
         uniques.columns = columns
         return uniques
@@ -387,42 +309,22 @@ def _generic_complete(
     for ind, column in enumerate(columns):
         if isinstance(column, dict):
             len_columns = len(columns)
-            column = _complete_column(column, df=df, index=index, sort=sort)
+            column = _complete_column(column, df=df, sort=sort)
             # iteration here avoids any potential index collision
             column = {
                 ind + len_columns + key: value for key, value in column.items()
             }
             uniques.update(column)
         else:
-            uniques[ind] = _complete_column(
-                column, df=df, index=index, sort=sort
-            )
+            uniques[ind] = _complete_column(column, df=df, sort=sort)
     uniques = _computations_expand_grid(uniques)
-    if index:
-        uniques = {key[-1]: value for key, value in uniques.items()}
-        if len(uniques) > 1:
-            data = list(uniques.values())
-            uniques = pd.MultiIndex.from_arrays(data, names=uniques)
-        else:
-            key = next(iter(uniques))
-            data = uniques[key]
-            uniques = pd.Index(data, name=key)
-        if df.columns.nlevels == 1:
-            columns = pd.Index([], name=df.columns.name)
-        else:
-            length = len(df.columns.names)
-            columns = pd.MultiIndex.from_arrays(
-                [[]] * length, names=df.columns.names
-            )
-        return pd.DataFrame([], index=uniques, columns=columns, copy=False)
-
     uniques = pd.DataFrame(uniques, copy=False)
     uniques.columns = uniques.columns.droplevel(0)
     return uniques
 
 
 @functools.singledispatch
-def _complete_column(column, df, index, sort):
+def _complete_column(column, df, sort):
     """
     Args:
         column: scalar/list/dict
@@ -434,15 +336,12 @@ def _complete_column(column, df, index, sort):
         or a dictionary of unique Pandas Series.
     """
 
-    if index:
-        _, arr = pd.factorize(df.index.get_level_values(column), sort=sort)
-    else:
-        _, arr = pd.factorize(df.loc(axis=1)[column], sort=sort)
+    _, arr = pd.factorize(df.loc(axis=1)[column], sort=sort)
     return pd.Series(arr, name=column)
 
 
 @_complete_column.register(list)  # noqa: F811
-def _sub_complete_column(column, df, index, sort):  # noqa: F811
+def _sub_complete_column(column, df, sort):  # noqa: F811
     """
     Args:
         column: list
@@ -452,41 +351,26 @@ def _sub_complete_column(column, df, index, sort):  # noqa: F811
     Returns:
         Pandas DataFrame
     """
+    outcome = df.loc(axis=1)[column]
 
-    if index:
-        outcome = df.index
-        exclude = [label for label in outcome.names if label not in column]
-        if exclude:
-            outcome = outcome.droplevel(level=exclude)
-        # ideally, there shouldn't be nulls in the index
-        exclude = [outcome.get_level_values(label).isna() for label in column]
-        exclude = np.column_stack(exclude).all(axis=1)
-        if exclude.any():
-            outcome = outcome[~exclude]
-        _, outcome = pd.factorize(outcome, sort=sort)
-        outcome.names = column
+    exclude = outcome.isna().all(axis=1)
 
-    else:
-        outcome = df.loc(axis=1)[column]
-
-        exclude = outcome.isna().all(axis=1)
+    if exclude.any(axis=None):
+        outcome = outcome.loc[~exclude]
 
-        if exclude.any(axis=None):
-            outcome = outcome.loc[~exclude]
+    exclude = outcome.duplicated()
 
-        exclude = outcome.duplicated()
+    if exclude.any():
+        outcome = outcome.loc[~exclude]
 
-        if exclude.any():
-            outcome = outcome.loc[~exclude]
-
-        if sort:
-            outcome = outcome.sort_values(by=column)
+    if sort:
+        outcome = outcome.sort_values(by=column)
 
     return outcome
 
 
 @_complete_column.register(dict)  # noqa: F811
-def _sub_complete_column(column, df, index, sort):  # noqa: F811
+def _sub_complete_column(column, df, sort):  # noqa: F811
     """
     Args:
         column: dictionary
@@ -499,10 +383,7 @@ def _sub_complete_column(column, df, index, sort):  # noqa: F811
 
     collection = {}
     for ind, (key, value) in enumerate(column.items()):
-        if index:
-            arr = apply_if_callable(value, df.index.get_level_values(key))
-        else:
-            arr = apply_if_callable(value, df[key])
+        arr = apply_if_callable(value, df)
         if not is_list_like(arr):
             raise ValueError(f"value for {key} should be a 1-D array.")
         if not hasattr(arr, "shape"):
@@ -552,30 +433,14 @@ def _data_checks_complete(
     Check is conducted to ensure that column names are not repeated.
     Also checks that the names in `columns` actually exist in `df`.
 
-    Returns `df`, `columns`, `column_checker`, `index`, `by`, `fill_value`,
+    Returns `df`, `columns`, `column_checker`, `by`, `fill_value`,
     and `explicit` if all checks pass.
     """
 
-    df_columns = df.columns
-    index_labels = (
-        df.index.names
-        if isinstance(df.index, pd.MultiIndex)
-        else [df.index.name]
-    )
     if by:
         if not isinstance(by, list):
             by = [by]
-        for label in by:
-            if label in df_columns:
-                continue
-            elif label in index_labels:
-                index = True
-                continue
-            else:
-                raise ValueError(
-                    f"{label} in by is neither in the dataframe's columns, "
-                    "nor is it a label in the dataframe's index names."
-                )
+        check_column(df, column_names=by, present=True)
 
     columns = [
         [*grouping] if isinstance(grouping, tuple) else grouping
@@ -592,38 +457,20 @@ def _data_checks_complete(
             column_checker.extend(grouping)
 
     # columns should not be duplicated across groups
+    # nor should it exist in `by`
     column_checker_no_duplicates = set()
     for column in column_checker:
         if column is None:
             raise ValueError("label in the columns argument cannot be None.")
         if column in column_checker_no_duplicates:
             raise ValueError(f"{column} should be in only one group.")
-        column_checker_no_duplicates.add(column)  # noqa: PD005
-
-    # columns should either be all in columns
-    # or all in index.names/index.name
-    # ideally there shouldn't be None in either index names or columns
-    index = False
-    for label in column_checker:
-        if by and (label in by):
-            raise ValueError(f"{label} already exists in by.")
-        if label in df_columns:
-            continue
-        elif label in index_labels:
-            index = True
-            continue
-        else:
+        if by and (column in by):
             raise ValueError(
-                f"{label} is neither in the dataframe's columns, "
-                "nor is it a label in the dataframe's index names."
+                f"{column} already exists as a label in the `by` argument."
             )
+        column_checker_no_duplicates.add(column)  # noqa: PD005
 
-    if index:
-        for label in column_checker:
-            if label not in index_labels:
-                raise ValueError(
-                    f"{label} not found in the dataframe's index names."
-                )
+    check_column(df, column_names=column_checker, present=True)
 
     check("explicit", explicit, [bool])
 
@@ -644,4 +491,4 @@ def _data_checks_complete(
                     f"The value for {column_name} should be a scalar."
                 )
 
-    return columns, column_checker, index, sort, by, fill_value, explicit
+    return columns, column_checker, sort, by, fill_value, explicit


Test patch:
diff --git a/tests/functions/test_complete.py b/tests/functions/test_complete.py
index 14e9c51..717308b 100644
--- a/tests/functions/test_complete.py
+++ b/tests/functions/test_complete.py
@@ -41,6 +41,17 @@ def MI():
     return dfmi
 
 
+def test_multiindex_names_not_found(MI):
+    """
+    Raise ValueError if the passed label is not found
+    """
+    MI.index.names = list("ABCD")
+    with pytest.raises(
+        ValueError, match="group not present in dataframe columns!"
+    ):
+        MI.complete("group")
+
+
 @pytest.fixture
 def fill_df():
     """pytest fixture"""
@@ -96,17 +107,6 @@ def test_dict_not_list_like(fill_df):
         fill_df.complete("group", {"item_id": "cities"})
 
 
-def test_multiindex_names_not_found(MI):
-    """
-    Raise ValueError if the passed label is not found
-    """
-    MI.index.names = list("ABCD")
-    with pytest.raises(
-        ValueError, match="group is neither in the dataframe's columns.+"
-    ):
-        MI.complete("group")
-
-
 def test_dict_not_1D(fill_df):
     """
     Raise ValueError if `*columns`
@@ -231,14 +231,16 @@ def test_all_strings_no_nulls(df):
 
 @given(df=categoricaldf_strategy())
 @settings(deadline=None)
-def test_dict(df):
+def test_dict_callable(df):
     """
     Test `complete` output when *columns
     is a dictionary.
     """
     cols = ["names", "numbers"]
     df = df.assign(names=[*ascii_lowercase[: len(df)]])
-    new_numbers = {"numbers": lambda df: range(df.min(), df.max() + 1)}
+    new_numbers = {
+        "numbers": lambda df: range(df.numbers.min(), df.numbers.max() + 1)
+    }
     cols = ["numbers", "names"]
     result = df.complete(new_numbers, "names", sort=True)
     columns = df.columns
@@ -428,13 +430,15 @@ def test_complete_multiple_groupings():
         ("project_id", "question_count"),
         fill_value={"tag_count": 0},
         sort=True,
-    )
+    ).astype({"tag_count": int})
     assert_frame_equal(result, output3)
 
 
 def test_fill_value_scalar(taxonomy_df):
     """Test output if the fill_value is a scalar."""
-    result = taxonomy_df.complete("Year", "Taxon", fill_value=0, sort=False)
+    result = taxonomy_df.complete(
+        "Year", "Taxon", fill_value=0, sort=False
+    ).astype({"Abundance": int})
     expected = (
         taxonomy_df.encode_categorical(Taxon="appearance")
         .set_index(["Year", "Taxon"])
@@ -455,7 +459,7 @@ def test_dict_tuple_callable(taxonomy_df):
     """
 
     result = taxonomy_df.complete(
-        {"Year": lambda x: range(x.min(), x.max() + 1)},
+        {"Year": lambda x: range(x.Year.min(), x.Year.max() + 1)},
         ("Taxon", "Abundance"),
         sort=True,
     )
@@ -515,7 +519,9 @@ def test_complete_groupby():
     )
 
     result = df.complete(
-        {"year": lambda x: range(x.min(), x.max() + 1)}, by="state", sort=True
+        {"year": lambda x: range(x.year.min(), x.year.max() + 1)},
+        by="state",
+        sort=True,
     )
 
     expected = (
@@ -661,7 +667,7 @@ def test_explicit_(fill_df):
     and the columns used for the combination
     are reused in the fill_value.
     """
-    trimmed = fill_df.select_columns("value*", invert=True)
+    trimmed = fill_df.select("value*", axis="columns", invert=True)
     result = trimmed.complete(
         "group",
         ("item_id", "item_name"),
@@ -684,99 +690,93 @@ def test_explicit_(fill_df):
     assert_frame_equal(result, expected)
 
 
-@pytest.fixture
-def dt():
-    """Fixture for testing on index"""
-    # copied from https://stackoverflow.com/q/63271274/7175713
-    dt = pd.DataFrame(
-        zip(
-            ["x"] * 4 + ["y"] * 4,
-            range(8),
-            list(
-                pd.period_range(
-                    "2020-08-02T00:00:00", "2020-08-02T03:00:00", freq="H"
-                )
-            )
-            * 2,
-        ),
-        columns=["a", "b", "d"],
-    ).set_index(["a", "d"])
-    dt = dt.drop(
-        [
-            ("x", pd.Period("2020-08-02 01:00", "H")),
-            ("y", pd.Period("2020-08-02 01:00", "H")),
-        ]
-    )
-    return dt
-
-
-def test_index_only(dt):
+def test_nulls(fill_df):
     """
-    Test output if completing on index
+    Test output if nulls are present
     """
-    # https://stackoverflow.com/a/63273339/7175713
-    expected = dt.reindex(
-        pd.MultiIndex.from_product(
-            [
-                dt.index.get_level_values("a").unique(),
-                pd.period_range(
-                    dt.index.get_level_values("d").min(),
-                    dt.index.get_level_values("d").max(),
-                    freq="H",
-                ),
-            ],
-            names=dt.index.names,
-        )
+    actual = fill_df.complete(["value1"], "value2", sort=True)
+    ind = [fill_df.value1.dropna().unique(), fill_df.value2.unique()]
+    ind = pd.MultiIndex.from_product(ind, names=["value1", "value2"])
+    ind = pd.DataFrame([], index=ind)
+    expected = fill_df.merge(
+        ind, on=["value1", "value2"], how="outer", sort=True
     )
-
-    period = dict(d=lambda f: pd.period_range(f.min(), f.max()))
-    actual = dt.complete("a", period)
     assert_frame_equal(actual, expected)
 
 
-def test_index_groupby(dt):
-    """Test grouping, with index as the grouper"""
-    expected = dt.reindex(
-        pd.MultiIndex.from_product(
-            [
-                dt.index.get_level_values("a").unique(),
-                pd.period_range(
-                    dt.index.get_level_values("d").min(),
-                    dt.index.get_level_values("d").max(),
-                    freq="H",
-                ),
-            ],
-            names=dt.index.names,
+def test_groupby_tuple():
+    """Test output for groupby on a tuple of columns."""
+    # https://stackoverflow.com/q/77123843/7175713
+    data_dict = {
+        "Grid Cell": [1, 1, 1, 1, 1, 1, 1, 2, 2, 2, 2, 2, 2, 2],
+        "Site": [
+            "A",
+            "A",
+            "A",
+            "A",
+            "B",
+            "B",
+            "B",
+            "C",
+            "C",
+            "C",
+            "D",
+            "D",
+            "D",
+            "D",
+        ],
+        "Date": [
+            "1999-01-01",
+            "1999-02-01",
+            "1999-03-01",
+            "1999-04-01",
+            "1999-01-01",
+            "1999-02-01",
+            "1999-03-01",
+            "2000-01-01",
+            "2000-02-01",
+            "2000-03-01",
+            "2000-01-01",
+            "2000-02-01",
+            "2000-03-01",
+            "2000-04-01",
+        ],
+        "Value": [
+            -2.45,
+            -3.72,
+            1.34,
+            4.56,
+            0.23,
+            3.26,
+            6.76,
+            -7.45,
+            -6.43,
+            -2.18,
+            -10.72,
+            -8.97,
+            -5.32,
+            -1.73,
+        ],
+    }
+    df = pd.DataFrame.from_dict(data_dict)
+    expected = df.complete("Date", "Site", by="Grid Cell").sort_values(
+        ["Grid Cell", "Site", "Date"], ignore_index=True
+    )
+
+    # https://stackoverflow.com/a/77123963/7175713
+    def reindex(g):
+        idx = pd.MultiIndex.from_product(
+            [g["Grid Cell"].unique(), g["Site"].unique(), g["Date"].unique()],
+            names=["Grid Cell", "Site", "Date"],
+        )
+        return g.set_index(["Grid Cell", "Site", "Date"]).reindex(
+            idx, fill_value=np.nan
         )
-    )
-
-    period = dict(d=lambda f: pd.period_range(f.min(), f.max()))
-    actual = dt.reset_index("d").complete(period, by="a")
-    assert_frame_equal(actual, expected.reset_index("d"))
-
-
-def test_label_in_by(dt):
-    """Raise if label already in by"""
-    with pytest.raises(ValueError, match="a already exists in by."):
-        dt.reset_index("d").complete("a", by="a")
-
-
-def test_index_not_found(dt):
-    """Raise if completing on index and label not found"""
-    with pytest.raises(
-        ValueError, match="b not found in the dataframe's index names."
-    ):
-        dt.complete("b", "d")
-
 
-def test_MI(MI):
-    """
-    Test output on multiindex index
-    """
-    MI.index.names = list("ABCD")
-    expected = MI.iloc[[0, -1]].unstack("D").stack("D", dropna=False)
-    actual = MI.iloc[[0, -1]].complete(("A", "B", "C"), {"D": ["D0", "D1"]})
-    assert_frame_equal(actual, expected)
+    actual = (
+        df.groupby("Grid Cell", group_keys=False).apply(reindex).reset_index()
+    )
+    assert_frame_equal(expected, actual)
 
 
 def test_MI_1(MI):
@@ -792,77 +792,3 @@ def test_MI_1(MI):
     ).rename_axis(columns=[None, None])
     actual = MI.iloc[:2].complete({("a", "bar"): range(1, 5)})
     assert_frame_equal(actual, expected)
-
-
-def test_MI_2(MI):
-    """
-    Test output on multiindex columns
-    """
-    MI.index.names = list("ABCD")
-    frame = MI.iloc[[0, -1]].droplevel(["A", "C"], axis=0)
-    actual = frame.complete("D", "B").sort_index(level="B")
-    expected = frame.unstack("D").stack("D", dropna=False)
-    assert_frame_equal(actual, expected)
-
-
-def test_nulls(fill_df):
-    """
-    Test output if nulls are present
-    """
-    actual = fill_df.complete(["value1"], "value2", sort=True)
-    ind = [fill_df.value1.dropna().unique(), fill_df.value2.unique()]
-    ind = pd.MultiIndex.from_product(ind, names=["value1", "value2"])
-    ind = pd.DataFrame([], index=ind)
-    expected = fill_df.merge(
-        ind, on=["value1", "value2"], how="outer", sort=True
-    )
-    assert_frame_equal(actual, expected)
-
-
-def test_nulls_index(fill_df):
-    """
-    Test output if nulls are present
-    """
-    actual = (
-        fill_df.set_index(["value1", "value2"])
-        .complete(["value1"], "value2", sort=True)
-        .reset_index()
-    )
-    ind = [fill_df.value1.dropna().unique(), fill_df.value2.unique()]
-    ind = pd.MultiIndex.from_product(ind, names=["value1", "value2"])
-    ind = pd.DataFrame([], index=ind)
-    expected = ind.merge(
-        fill_df, on=["value1", "value2"], how="outer", sort=True
-    )
-    assert_frame_equal(actual, expected)
-
-
-def test_nulls_index_1(fill_df):
-    """
-    Test output if nulls are present
-    """
-    actual = (
-        fill_df.set_index(["value1", "value2"])
-        .complete("value1", "value2", sort=True)
-        .reset_index()
-    )
-    ind = [fill_df.value1.dropna().unique(), fill_df.value2.unique()]
-    ind = pd.MultiIndex.from_product(ind, names=["value1", "value2"])
-    ind = pd.DataFrame([], index=ind)
-    expected = ind.merge(
-        fill_df, on=["value1", "value2"], how="outer", sort=True
-    )
-    assert_frame_equal(actual, expected)
-
-
-def test_single_index_dict():
-    """Test output on a single index with a dictionary"""
-    # https://pandas.pydata.org/docs/reference/api/pandas.DataFrame.asfreq.html
-    index = pd.date_range("1/1/2000", periods=4, freq="T")
-    series = pd.Series([0.0, None, 2.0, 3.0], index=index)
-    df = pd.DataFrame({"s": series})
-    df.index.name = "dates"
-    dates = {"dates": lambda f: pd.date_range(f.min(), f.max(), freq="30S")}
-    assert_frame_equal(
-        df.complete(dates), df.asfreq(freq="30S"), check_freq=False
-    )