Repository: pyjanitor-devs/pyjanitor

Issue:
Add an `also` method akin to Kotlins `also` 
# Brief Description
I would like to propose a new function `also` that takes a function with a single dataframe for input, runs the function on the chained dataframe, and returns the input dataframe.  

I would love to take a whack at implementing this if you're interested incorporating it into the library.

As I've been trying to adopt chaining as much as possible, I've routinely bumped into scenarios where I need to break the chain for one reason or another. This API would eliminate most reasons for breaking the chain

This function is inspired by Kotlin's `also` function.  If we were to mimic Kotlin's `also`, the signature would look something like

`also: (f: T -> None) -> T` 

It would behave very similar to `pipe` or `then`, but can be used on functions that do not return anything. This can be very useful for any sort of side-effect operations within a long chain, such as periodic logging or debugging.

# Example API

<!-- One of the selling points of pyjanitor is the API. Hence, we guard the API very carefully, and want to
make sure that it is accessible and understandable to many people. Please provide a few examples of what the API
of the new function you're proposing will look like. We have provided an example that you should modify. -->

Below are some examples of how this API would simplify code I've written in the past. 

### print the df while processing.
```python
# instead of defining an extra function
def print_and(df):
    print(df)
    return df

df.groupby(["column1"]).agg(sum).then(print_and).then(my_transform)

# or breaking the chain up
df_tmp = df.groupby(["column1"]).agg(sum)
print(df_tmp)
df_tmp.then(my_transform)

# you could just inline the print statement
df.groupby(["column1"]).agg(sum).also(print).then(my_transform)
```

### Insert a breakpoint while debugging, but keep going when you're done
Similar to above, to add a breakpoint for debugging, you would either have to create a new `pipe` compatible function or break the chain into smaller steps.

Instead, you could chain a breakpoint like so

```python
df.groupby(["column1"]).agg(sum).also(lambda x: breakpoint()).then(my_transform)
```

### Save a midpoint of a processing chain and then keep going

Another possibility would be running IO in the middle of the chain.  Often, a midpoint in the chain is just as important as the end result.  `also` would allow me to save the results and keep going without breaking a chain.

```
df.groupby(["column1"]).agg(sum).also(lambda x: x.to_csv("filename.csv)).then(my_transform)
```

### pipe-dream: Save a midpoint as a variable

I haven't figured out how to get this to work given Python's scoping rules, maybe the new `:=` operator will make this possible. 

This could be really helpful in scenarios where you're running groupbys or other aggregations and want to join those back into an ungrouped frame.

```
df.groupby(["column1"]).agg(sum).also(lambda x: df_mid = x).then(my_transform).assign(original_var = df_mid)
```



Discussion:
Couldn't those lambda functions also be placed in the "then" methods?
The `then` method returns the output of the lambdas and none of these lambdas have any output. 
@sauln Hi. Just to make your feature proposal clearer, could you share some concrete examples, with an input dataframes as expected outputs 
@sauln I think I get your drift! :smile:

Let me see, the implementation of `.also()` probably looks like this:

```python
def also(df, func, *args, **kwargs):
    func(df.copy(), *args, **kwargs)
    return df
```

Is that correct? The key distinction here is as you mentioned - that no modifications to the state of the dataframe is intended to be made, and the dataframe is passed through in the form that it was passed into `also()`, is that right? (Unless, of course, the user modifies the dataframe in the `func` -- in which case, all bets are off, and user beware.)

Here's an example, one that a user might want to use as an ad-hoc logging tool (lightweight, no need to worry about logging frameworks):

```python
df = (
    pd.DataFrame(...)
    .query(...)
    .also(lambda df: print(f"DataFrame shape is: {df.shape}))  # print shape in-line
    .transform_column(...)
    .also(lambda df: print(f"Column col_name has these values: {set(df["col_name"].values)}")
    .more_stuff_happens_here(...)
)
```

I'm game for this inside `pyjanitor` - it'll enable the dataframe checkpointing that you mention without breaking flow (assuming breaking flow is bad for reading the program, or troublesome). (In fact, adding in `.also()` opens up the door to providing a `.checkpoint_disk(file_format, *to_format_args, **to_format_kwargs)` later on, that saves a checkpoint in the desired file format!

Btw, on this point:

> This could be really helpful in scenarios where you're running groupbys or other aggregations and want to join those back into an ungrouped frame.

There is a function in the library called `groupby_agg`. Wasn't my idea, but I think it was genius and I ended up using it pretty often! That said, you've definitely thought of the more general case; on this one, I'd personally want to see more individual examples of the general case of midpoint assignment. I'm sure it's useful, though I'm probably just having some trouble with my own imagination haha.
@sauln Nice idea! What happens if the `also` function mutates the data, like performing an “inplace” operation? Does the next function in the chain receive the mutated data or the original un-mutated data (which would require an explicit copy, I believe)?

Edit: @lmcinnes was thinking the same thing at the same time. 🤣
In an ideal world I would think that ``also`` would run ``func`` on a copy of the dataframe (yes, this duplicates the dataframe in memory, but it is probably short lived). This saves the "all bets are off" problem of the ``func`` modifying the dataframe. I could easily imagine a user having a function that does some computations and logs results without thinking that those computations occur in-place. The results would be a nasty surprise.
Good point, gents! I have updated the example to explicitly just copy the DataFrame.
Very good points.  I could imagine a user would like it both ways though.  Could there be operations that can only happen in place? Originally, I imagined the only contract `also` would abide by is `id(df_in) == id(df_out)`.  The unintended side effect you point out @lmcinnes definitely has me spooked though.  Maybe there should be flag like `safe=True` or `inplace=False` that determines if a copy is made or not.

> I'd personally want to see more individual examples of the general case of midpoint assignment.

I don't want to derail the `also` progress for this as unfortunately, I can't figure out how to get this to work. I believe Pythons scoping rules prevent it, or at least make it very difficult without invoking some global variable shenanigans.  An example in a codebase I'm working on right requires the function return both a middle df and a final df.  

Something like
```
df_1 = df.pipe(f1).pipe(f2)
df_2 = df_1.pipe(f3).pipe(f4).pipe(f5, mid=df_1)
return df_1, df_2
```

could potentially replaced with 
```
df_2 = df.pipe(f1).pipe(f2).checkpoint("df_1").pipe(f3).pipe(f4).pipe(f5, mid=df_1)
return df_1, df_2
```

We can leave this pipe dream for a later conversation though.





I feel like making a copy is the right thing to do -- it avoids any problems about in place operations, and there won't be any surprises except in complicated cases where a deepcopy would be required. I suspect that may be rare enough to not worry about.

Resolving patch:
diff --git a/janitor/functions.py b/janitor/functions.py
--- a/janitor/functions.py
+++ b/janitor/functions.py
@@ -2963,6 +2963,43 @@ def then(df: pd.DataFrame, func: Callable) -> pd.DataFrame:
     return df
 
 
+@pf.register_dataframe_method
+def also(df: pd.DataFrame, func: Callable, *args, **kwargs) -> pd.DataFrame:
+    """Add an arbitrary function with no return value to run in the
+    ``pyjanitor`` method chain. This returns the input dataframe instead,
+    not the output of `func`.
+
+    This method does not mutate the original DataFrame.
+
+    Example usage:
+
+    .. code-block:: python
+
+        df = (
+            pd.DataFrame(...)
+            .query(...)
+            .also(lambda df: print(f"DataFrame shape is: {df.shape}"))
+            .transform_column(...)
+            .also(lambda df: df.to_csv("midpoint.csv"))
+            .also(
+                lambda df: print(
+                    f"Column col_name has these values: {set(df['col_name'].unique())}"
+                )
+            )
+            .group_add(...)
+        )
+
+    :param df: A pandas dataframe.
+    :param func: A function you would like to run in the method chain.
+        It should take one DataFrame object as a parameter and have no return.
+        If there is a return, it will be ignored.
+    :param args, kwargs: Optional arguments and keyword arguments for `func`.
+    :returns: The input pandas DataFrame.
+    """  # noqa: E501
+    func(df.copy(), *args, **kwargs)
+    return df
+
+
 @pf.register_dataframe_method
 @deprecated_alias(column="column_name")
 def dropnotnull(df: pd.DataFrame, column_name: Hashable) -> pd.DataFrame:


Test patch:
diff --git a/tests/functions/test_also.py b/tests/functions/test_also.py
new file mode 100644
--- /dev/null
+++ b/tests/functions/test_also.py
@@ -0,0 +1,83 @@
+"""Unit tests for `.also()`."""
+from unittest.mock import Mock
+import pytest
+
+
+def remove_first_two_letters_from_col_names(df):
+    """Helper function to mutate dataframe by changing column names."""
+    col_names = df.columns
+    col_names = [name[2:] for name in col_names]
+    df.columns = col_names
+    return df
+
+
+def remove_rows_3_and_4(df):
+    """Helper function to mutate dataframe by removing rows."""
+    df = df.drop(3, axis=0)
+    df = df.drop(4, axis=0)
+    return df
+
+
+def drop_inplace(df):
+    """
+    Helper function to mutate dataframe by dropping a column.
+
+    We usually would not use `inplace=True` in a block,
+    but the intent here is to test that
+    the in-place modification of a dataframe
+    doesn't get passed through in the `.also()` function.
+    Hence, we tell Flake8 to skip checking `PD002` on that line.
+    """
+    df.drop(columns=[df.columns[0]], inplace=True)  # noqa: PD002
+
+
+@pytest.mark.functions
+def test_also_column_manipulation_no_change(dataframe):
+    """Test that changed dataframe inside `.also()` doesn't get returned."""
+    cols = tuple(dataframe.columns)
+    df = dataframe.also(remove_first_two_letters_from_col_names)
+    assert dataframe is df
+    assert cols == tuple(df.columns)
+
+
+@pytest.mark.functions
+def test_also_remove_rows_no_change(dataframe):
+    """Test that changed dataframe inside `.also()` doesn't get returned."""
+    df = dataframe.also(remove_rows_3_and_4)
+    rows = tuple(df.index)
+    assert rows == (0, 1, 2, 3, 4, 5, 6, 7, 8)
+
+
+@pytest.mark.functions
+def test_also_runs_function(dataframe):
+    """Test that `.also()` executes the function."""
+    method = Mock(return_value=None)
+    df = dataframe.also(method)
+    assert id(df) == id(dataframe)
+    assert method.call_count == 1
+
+
+@pytest.mark.functions
+def test_also_args(dataframe):
+    """Test that the args are passed through to the function."""
+    method = Mock(return_value=None)
+    _ = dataframe.also(method, 5)
+
+    assert method.call_args[0][1] == 5
+
+
+@pytest.mark.functions
+def test_also_kwargs(dataframe):
+    """Test that the kwargs are passed through to the function."""
+    method = Mock(return_value=None)
+    _ = dataframe.also(method, n=5)
+
+    assert method.call_args[1] == {"n": 5}
+
+
+@pytest.mark.functions
+def test_also_drop_inplace(dataframe):
+    """Test that in-place modification of dataframe does not pass through."""
+    cols = tuple(dataframe.columns)
+    df = dataframe.also(drop_inplace)
+    assert tuple(df.columns) == cols