Repository: pyjanitor-devs/pyjanitor
Pull request #1052 (2022-03-23 09:39:55)

[ENH] pivot_wider names_glue
# PR Description

Please describe the changes proposed in the pull request:

- names_glue is now a string, and allows for easier interpolation, similar to tidyr's pivot_wider names_glue
- levels_order is redundant and excised
- edited docs
- more heavy name modification can be handled by the user via sort_index/swaplevel/reorder_levels/collapse_levels

**This PR improves pivot_wider.**

<!-- As you go down the PR template, please feel free to delete sections that are irrelevant. -->

# PR Checklist

<!-- This checklist exists for newcomers who are not yet familiar with our requirements. If you are experienced with
the project, please feel free to delete this section. -->

Please ensure that you have done the following:

1. [ ] PR in from a fork off your branch. Do not PR from `<your_username>`:`dev`, but rather from `<your_username>`:`<feature-branch_name>`.
<!-- Doing this helps us keep the commit history much cleaner than it would otherwise be. -->
2. [ ] If you're not on the contributors list, add yourself to `AUTHORS.rst`.
<!-- We'd like to acknowledge your contributions! -->
3. [ ] Add a line to `CHANGELOG.md` under the latest version header (i.e. the one that is "on deck") describing the contribution.
    - Do use some discretion here; if there are multiple PRs that are related, keep them in a single line.

# Automatic checks

There will be automatic checks run on the PR. These include:

- Building a preview of the docs on Netlify
- Automatically linting the code
- Making sure the code is documented
- Making sure that all tests are passed
- Making sure that code coverage doesn't go down.

# Relevant Reviewers

<!-- Finally, please tag relevant maintainers to review. -->

Please tag maintainers to review.

- @ericmjl


ericmjl: <!-- NETLIFY DEPLOY COMMENT GENERATED BY ACTIONS_NETLIFY -->
🚀 Deployed on https://deploy-preview-1052--pyjanitor.netlify.app
codecov[bot]: # [Codecov](https://codecov.io/gh/pyjanitor-devs/pyjanitor/pull/1052?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=pyjanitor-devs) Report
> Merging [#1052](https://codecov.io/gh/pyjanitor-devs/pyjanitor/pull/1052?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=pyjanitor-devs) (6015114) into [dev](https://codecov.io/gh/pyjanitor-devs/pyjanitor/commit/a6925cbeb4a0732eb6ed89db2580cbad1bf3ed44?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=pyjanitor-devs) (a6925cb) will **decrease** coverage by `0.02%`.
> The diff coverage is `100.00%`.

> :exclamation: Current head 6015114 differs from pull request most recent head 9d00c64. Consider uploading reports for the commit 9d00c64 to get more accurate results


```diff
@@            Coverage Diff             @@
##              dev    #1052      +/-   ##
==========================================
- Coverage   97.72%   97.70%   -0.03%     
==========================================
  Files          77       77              
  Lines        2863     2874      +11     
==========================================
+ Hits         2798     2808      +10     
- Misses         65       66       +1     
```


Patch:
diff --git a/CHANGELOG.md b/CHANGELOG.md
index 095a593..52ca80e 100644
--- a/CHANGELOG.md
+++ b/CHANGELOG.md
@@ -21,7 +21,8 @@
 -   [ENH] Performance improvement for `conditional_join`. @samukweku
 -   [ENH] Multiple `.value` is now supported in `pivot_longer`. Multiple values_to is also supported, when names_pattern is a list or tuple. #1034, #1048 @samukweku
 -   [ENH] Add `xlsx_cells` for reading a spreadsheet as a table of individual cells. #929 @samukweku.
--   [EHN] Let `filter_string` suit parameters of `Series.str.contains` Issue #1003 and #1047. @Zeroto521
+-   [ENH] Let `filter_string` suit parameters of `Series.str.contains` Issue #1003 and #1047. @Zeroto521
+-   [ENH] `names_glue` in `pivot_wider` now takes a string form, using str.format_map under the hood. `levels_order` is also deprecated. @samukweku
 
 ## [v0.22.0] - 2021-11-21
 
diff --git a/janitor/functions/pivot.py b/janitor/functions/pivot.py
index ee9d9fd..a6a789c 100644
--- a/janitor/functions/pivot.py
+++ b/janitor/functions/pivot.py
@@ -1,5 +1,6 @@
 import re
-from typing import Callable, Optional, Pattern, Union
+import warnings
+from typing import Optional, Pattern, Union
 
 import numpy as np
 import pandas as pd
@@ -1029,10 +1030,9 @@ def pivot_wider(
     index: Optional[Union[list, str]] = None,
     names_from: Optional[Union[list, str]] = None,
     values_from: Optional[Union[list, str]] = None,
-    levels_order: Optional[list] = None,
     flatten_levels: Optional[bool] = True,
-    names_sep="_",
-    names_glue: Callable = None,
+    names_sep: str = "_",
+    names_glue: str = None,
 ) -> pd.DataFrame:
     """
     Reshapes data from *long* to *wide* form.
@@ -1083,17 +1083,6 @@ def pivot_wider(
         0  5.5   20   25   30   37
         1  6.1   22   18   19   29
 
-    Change the order of the column labels:
-
-        >>> df.pivot_wider(
-        ...     index = "dep",
-        ...     names_from = "step",
-        ...     levels_order = ["step", None],
-        ... )
-           dep  1_a  2_a  1_b  2_b
-        0  5.5   20   25   30   37
-        1  6.1   22   18   19   29
-
     Modify columns with `names_sep`:
 
         >>> df.pivot_wider(
@@ -1110,14 +1099,12 @@ def pivot_wider(
         >>> df.pivot_wider(
         ...     index = "dep",
         ...     names_from = "step",
-        ...     names_sep = None,
-        ...     names_glue = lambda col: f"{col[0]}_step{col[1]}",
+        ...     names_glue = "{_value}_step{step}",
         ... )
            dep  a_step1  a_step2  b_step1  b_step2
         0  5.5       20       25       30       37
         1  6.1       22       18       19       29
 
-
     :param df: A pandas DataFrame.
     :param index: Name(s) of columns to use as identifier variables.
         It should be either a single column name, or a list of column names.
@@ -1129,20 +1116,20 @@ def pivot_wider(
         the new DataFrame's values.
         If `values_from` is not specified,  all remaining columns
         will be used.
-    :param levels_order: Applicable if there are multiple `names_from`
-        and/or `values_from`. Reorders the levels. Accepts a list of strings.
-        If there are multiple `values_from`, pass `None` to represent that
-        level.
     :param flatten_levels: Default is `True`. If `False`, the DataFrame stays
         as a MultiIndex.
     :param names_sep: If `names_from` or `values_from` contain multiple
         variables, this will be used to join the values into a single string
         to use as a column name. Default is `_`.
         Applicable only if `flatten_levels` is `True`.
-    :param names_glue: A callable to control the output
-        of the flattened columns.
+    :param names_glue: A string to control the output of the flattened columns.
+        It offers more flexibility in creating custom column names,
+        and uses python's `str.format_map` under the hood.
+        Simply create the string template,
+        using the column labels in `names_from`,
+        and special `_value` as a placeholder
+        if there are multiple `values_from`.
         Applicable only if `flatten_levels` is `True`.
-        Function should be acceptable to pandas’ `map` function.
     :returns: A pandas DataFrame that has been unpivoted from long to wide
         form.
     """
@@ -1154,7 +1141,6 @@ def pivot_wider(
         index,
         names_from,
         values_from,
-        levels_order,
         flatten_levels,
         names_sep,
         names_glue,
@@ -1166,16 +1152,14 @@ def _computations_pivot_wider(
     index: Optional[Union[list, str]] = None,
     names_from: Optional[Union[list, str]] = None,
     values_from: Optional[Union[list, str]] = None,
-    levels_order: Optional[list] = None,
     flatten_levels: Optional[bool] = True,
-    names_sep="_",
-    names_glue: Callable = None,
+    names_sep: str = "_",
+    names_glue: str = None,
 ) -> pd.DataFrame:
     """
     This is the main workhorse of the `pivot_wider` function.
 
-    It is a wrapper around `pd.pivot`. For a MultiIndex, the
-    order of the levels can be changed with `levels_order`.
+    It is a wrapper around `pd.pivot`.
     The output for multiple `names_from` and/or `values_from`
     can be controlled with `names_glue` and/or `names_sep`.
 
@@ -1187,7 +1171,6 @@ def _computations_pivot_wider(
         index,
         names_from,
         values_from,
-        levels_order,
         flatten_levels,
         names_sep,
         names_glue,
@@ -1196,55 +1179,87 @@ def _computations_pivot_wider(
         index,
         names_from,
         values_from,
-        levels_order,
         flatten_levels,
         names_sep,
         names_glue,
     )
-    # check dtype of `names_from` is string
-    names_from_all_strings = (
-        df.filter(names_from).agg(is_string_dtype).all().item()
-    )
+    if flatten_levels:
+        # check dtype of `names_from` is string
+        names_from_all_strings = (
+            df.filter(names_from).agg(is_string_dtype).all().item()
+        )
 
-    # check dtype of columns
-    column_dtype = is_string_dtype(df.columns)
+        # check dtype of columns
+        column_dtype = is_string_dtype(df.columns)
 
     df = df.pivot(  # noqa: PD010
         index=index, columns=names_from, values=values_from
     )
 
-    if levels_order and (isinstance(df.columns, pd.MultiIndex)):
-        df = df.reorder_levels(order=levels_order, axis="columns")
-
     # an empty df is likely because
     # there is no `values_from`
     if any((df.empty, not flatten_levels)):
         return df
 
-    # ensure all entries in names_from are strings
-    if (not names_from_all_strings) or (not column_dtype):
-        if isinstance(df.columns, pd.MultiIndex):
-            new_columns = [tuple(map(str, ent)) for ent in df]
-            df.columns = pd.MultiIndex.from_tuples(new_columns)
+    if isinstance(df.columns, pd.MultiIndex):
+        if (not names_from_all_strings) or (not column_dtype):
+            new_columns = [tuple(map(str, entry)) for entry in df]
         else:
-            df.columns = df.columns.astype(str)
-
-    if (names_sep is not None) and (isinstance(df.columns, pd.MultiIndex)):
-        df.columns = df.columns.map(names_sep.join)
+            new_columns = [entry for entry in df]
+        if names_glue is not None:
+            if ("_value" in names_from) and (None in df.columns.names):
+                warnings.warn(
+                    "For names_glue, _value is used as a placeholder "
+                    "for the values_from section. "
+                    "However, there is a '_value' in names_from; "
+                    "this might result in incorrect output. "
+                    "If possible, kindly change the column label "
+                    "from `_value` to another name, "
+                    "to avoid erroneous results."
+                )
+            try:
+                # there'll only be one None
+                names_from = [
+                    "_value" if ent is None else ent
+                    for ent in df.columns.names
+                ]
+                new_columns = [
+                    names_glue.format_map(dict(zip(names_from, entry)))
+                    for entry in new_columns
+                ]
+            except KeyError as error:
+                raise KeyError(
+                    f"{error} is not a column label in names_from."
+                ) from error
+        else:
+            if names_sep is None:
+                names_sep = "_"
+            new_columns = [names_sep.join(entry) for entry in new_columns]
 
-    if names_glue:
-        df.columns = df.columns.map(names_glue)
+        df.columns = new_columns
+    else:
+        if (not names_from_all_strings) or (not column_dtype):
+            df.columns = df.columns.astype(str)
+        if names_glue is not None:
+            try:
+                df.columns = [
+                    names_glue.format_map({names_from[0]: entry})
+                    for entry in df
+                ]
+            except KeyError as error:
+                raise KeyError(
+                    f"{error} is not a column label in names_from."
+                ) from error
 
     # if columns are of category type
     # this returns columns to object dtype
     # also, resetting index with category columns is not possible
     df.columns = [*df.columns]
-
     if index:
         df = df.reset_index()
 
     if df.columns.names:
-        df = df.rename_axis(columns=None)
+        df.columns.names = [None]
 
     return df
 
@@ -1254,7 +1269,6 @@ def _data_checks_pivot_wider(
     index,
     names_from,
     values_from,
-    levels_order,
     flatten_levels,
     names_sep,
     names_glue,
@@ -1289,23 +1303,19 @@ def _data_checks_pivot_wider(
         if len(values_from) == 1:
             values_from = values_from[0]
 
-    if levels_order is not None:
-        check("levesl_order", levels_order, [list])
-
     check("flatten_levels", flatten_levels, [bool])
 
     if names_sep is not None:
         check("names_sep", names_sep, [str])
 
     if names_glue is not None:
-        check("names_glue", names_glue, [callable])
+        check("names_glue", names_glue, [str])
 
     return (
         df,
         index,
         names_from,
         values_from,
-        levels_order,
         flatten_levels,
         names_sep,
         names_glue,


Test patch:
diff --git a/tests/functions/test_pivot_wider.py b/tests/functions/test_pivot_wider.py
index 6d70785..765733c 100644
--- a/tests/functions/test_pivot_wider.py
+++ b/tests/functions/test_pivot_wider.py
@@ -6,6 +6,7 @@ from pandas.testing import assert_frame_equal
 
 @pytest.fixture
 def df_checks_output():
+    """pytest fixture"""
     return pd.DataFrame(
         {
             "geoid": [1, 1, 13, 13],
@@ -44,7 +45,7 @@ def test_type_names_from(df_checks_output):
         df_checks_output.pivot_wider(index="geoid", names_from=("variable",))
 
 
-def test_names_from_None(df_checks_output):
+def test_names_from_none(df_checks_output):
     """Raise ValueError if no value is provided for `names_from`."""
     with pytest.raises(ValueError):
         df_checks_output.pivot_wider(index="geoid", names_from=None)
@@ -76,16 +77,6 @@ def test_presence_names_from2(df_checks_output):
         df_checks_output.pivot_wider(index="geoid", names_from=["estimat"])
 
 
-def test_names_sort_wrong_type(df_checks_output):
-    """Raise TypeError if the wrong type is provided for `names_sort`."""
-    with pytest.raises(TypeError):
-        df_checks_output.pivot_wider(
-            index="name",
-            names_from=["estimate", "variable"],
-            names_sort=2,
-        )
-
-
 def test_flatten_levels_wrong_type(df_checks_output):
     """Raise TypeError if the wrong type is provided for `flatten_levels`."""
     with pytest.raises(TypeError):
@@ -96,40 +87,43 @@ def test_flatten_levels_wrong_type(df_checks_output):
         )
 
 
-@pytest.mark.xfail(reason="parameter is deprecated.")
-def test_names_from_position_wrong_type(df_checks_output):
-    """
-    Raise TypeError if the wrong type
-    is provided for `names_from_position`.
-    """
-    with pytest.raises(TypeError):
+def test_names_glue_wrong_label(df_checks_output):
+    """Raise KeyError if the wrong column label is provided in `names_glue`."""
+    with pytest.raises(
+        KeyError, match="'variabl' is not a column label in names_from."
+    ):
         df_checks_output.pivot_wider(
-            index="name",
-            names_from=["estimate", "variable"],
-            names_from_position=2,
+            index=["geoid", "name"],
+            names_from="variable",
+            values_from=["estimate", "error"],
+            names_glue="{variabl}_{_value}",
         )
 
 
-@pytest.mark.xfail(reason="parameter is deprecated.")
-def test_names_from_position_wrong_value(df_checks_output):
+def test_names_glue_wrong_label1(df_checks_output):
     """
-    Raise ValueError if `names_from_position`
-    is not "first" or "last".
+    Raise KeyError if the wrong column label is provided in `names_glue`,
+    And the columns is a single Index.
     """
-    with pytest.raises(ValueError):
+    with pytest.raises(
+        KeyError, match="'variabl' is not a column label in names_from."
+    ):
         df_checks_output.pivot_wider(
-            index="name",
-            names_from=["estimate", "variable"],
-            names_from_position="1st",
+            ["geoid", "name"],
+            "variable",
+            "estimate",
+            names_glue="{variabl}_estimate",
         )
 
 
-@pytest.mark.xfail(reason="parameter is deprecated.")
-def test_name_prefix_wrong_type(df_checks_output):
-    """Raise TypeError if the wrong type is provided for `names_prefix`."""
-    with pytest.raises(TypeError):
-        df_checks_output.pivot_wider(
-            index="name", names_from=["estimate", "variable"], names_prefix=1
+def test_names_glue_wrong_label2(df_checks_output):
+    """Raise Warning if _value is in `names_glue`."""
+    with pytest.warns(UserWarning):
+        df_checks_output.rename(columns={"variable": "_value"}).pivot_wider(
+            index=["geoid", "name"],
+            names_from="_value",
+            values_from=["estimate", "error"],
+            names_glue="{_value}_{_value}",
         )
 
 
@@ -149,32 +143,6 @@ def test_name_glue_wrong_type(df_checks_output):
         )
 
 
-def test_levels_order_wrong_type(df_checks_output):
-    """Raise TypeError if the wrong type is provided for `levels_order`."""
-    with pytest.raises(TypeError):
-        df_checks_output.pivot_wider(
-            index="name", names_from=["estimate", "variable"], levels_order=1
-        )
-
-
-@pytest.mark.xfail(reason="parameter is deprecated.")
-def test_fill_value_wrong_type(df_checks_output):
-    """Raise TypeError if the wrong type is provided for `fill_value`."""
-    with pytest.raises(TypeError):
-        df_checks_output.pivot_wider(
-            index="name", names_from=["estimate", "variable"], fill_value={2}
-        )
-
-
-@pytest.mark.xfail(reason="parameter is deprecated.")
-def test_aggfunc_wrong_type(df_checks_output):
-    """Raise TypeError if the wrong type is provided for `aggfunc`."""
-    with pytest.raises(TypeError):
-        df_checks_output.pivot_wider(
-            index="name", names_from=["estimate", "variable"], aggfunc={2}
-        )
-
-
 def test_non_unique_index_names_from_combination():
     """
     Raise ValueError for non-unique combination of
@@ -213,7 +181,9 @@ def test_pivot_long_wide_long():
         ]
     )
 
-    result = df_in.pivot_wider(index=["a", "b"], names_from="name")
+    result = df_in.pivot_wider(
+        index=["a", "b"], names_from="name", names_sep=None
+    )
 
     result = result.pivot_longer(
         index=["a", "b"],
@@ -274,7 +244,6 @@ def test_flatten_levels_false():
     assert_frame_equal(
         result,
         expected_output,
-        # check_dtype=False,
     )
 
 
@@ -369,30 +338,17 @@ def test_names_glue():
         index="family",
         names_from="n",
         values_from="name",
-        names_glue=lambda col: f"name{col}",
+        names_glue="name{n}",
     )
     assert_frame_equal(result, df_out)
 
 
-def test_change_level_order():
+def test_change_level_order(df_checks_output):
     """
     Test output with `levels_order`,
     while maintaining order from `names_from`.
     """
-    df_in = pd.DataFrame(
-        {
-            "geoid": [1, 1, 13, 13],
-            "name": ["Alabama", "Alabama", "Georgia", "Georgia"],
-            "variable": [
-                "pop_renter",
-                "median_rent",
-                "pop_renter",
-                "median_rent",
-            ],
-            "estimate": [1434765, 747, 3592422, 927],
-            "error": [16736, 3, 33385, 3],
-        }
-    )
+
     df_out = pd.DataFrame(
         {
             "geoid": [1, 13],
@@ -404,11 +360,34 @@ def test_change_level_order():
         }
     )
 
-    result = df_in.encode_categorical(variable="appearance").pivot_wider(
+    result = df_checks_output.encode_categorical(
+        variable="appearance"
+    ).pivot_wider(
         index=["geoid", "name"],
         names_from="variable",
         values_from=["estimate", "error"],
-        levels_order=["variable", None],
+        names_glue="{variable}_{_value}",
+    )
+    assert_frame_equal(result, df_out)
+
+
+def test_names_glue_single_column(df_checks_output):
+    """
+    Test names_glue for single column.
+    """
+
+    df_out = (
+        df_checks_output.pivot(["geoid", "name"], "variable", "estimate")
+        .add_suffix("_estimate")
+        .rename_axis(columns=None)
+        .reset_index()
+    )
+
+    result = df_checks_output.pivot_wider(
+        ["geoid", "name"],
+        "variable",
+        "estimate",
+        names_glue="{variable}_estimate",
     )
     assert_frame_equal(result, df_out)