Repository: pyjanitor-devs/pyjanitor

Issue:
[ENH] Modifying and Documenting Coalesce
Coalesce currently has new_column_name as a default parameter, which is not currently in the code example. This behavior can be changed by making new_column_name default to None, and have the coalesced column inherit the name of columns[0]. 

The actual behavior of this function will be clarified, along with type hinting to be consistent with the most of the rest of "functions.py".
[DOC] bin_numeric is missing from the docs "General Functions" page
As per title.
[DOC] Produce notebook showing how to normalize and standardize columns using pyjanitor
I like to add a normalize and standardize function to pyjanitor. I guess we can do this with transform function, but I like the readability of the code over a lambda function to make it work. Like normalize_columns() or standardize_columns(), since we already have expand_columns() (aka. dummy codes) and max_min_scale(). This will put preprocessing into the hands of more beginners, even though I have nothing against sklearn. I will say, doing a .fit() call on preprocessing bothers me, and I dont know why? What are your thoughts?


Discussion:

I will work on this one.
@jcvall I debated a bit with myself when thinking about whether to put min_max_scale and expand_columns inside. I think it's cool to put in `normalize`/`standardize`, but I think that it (and now that I'm thinking about it, `min_max_scale` as well) should come with a warning in the docs, that state that any transformations that involve a summary statistic of a dataset (e.g. mean, median, max, min, etc.) will result in testing set "contamination", or "leakage of information from training set into testing set". For this reason, sklearn's preprocessing is instead more principled.

There is one setting where I think having those functions could be useful though - in Bayesian modelling, when we are already sure of the model structure, then fitting the model necessarily means using the entire dataset, hence no train/test splitting happens. Here, having these functions could be useful as well.

Even in light of these statistical considerations, I would encourage you to submit the PR! We can probably do it with the right caveats written into the docstring, so that other end-users are aware of how to not abuse the convenience functionality. What do you think?
I actually didn't think about leakage in machine learning.  I guess what you are saying is that when you perform the preprocessing  you can split your data; however, the deviations from the mean for normalized data will be in both the training and test set.  So the mean is inferred in both data sets.  If I am thinking correctly, then that is a problem...Let me know if I am off base here.  I guess in Bayesian and more classical statistics (linear regression) it is rather helpful function for the coefficients, so I see your point.  Okay.  I will go ahead  knowing that this should be used with caution.  
Yep, you're on the right track there!
Sorry, you don't have to answer this right away, but would the sklearn-pandas package be helpful here?  

Example from GitHub page:
`Outputting a dataframe

By default the output of the dataframe mapper is a numpy array. This is so because most sklearn estimators expect a numpy array as input. If however we want the output of the mapper to be a dataframe, we can do so using the parameter df_out when creating the mapper:

```
mapper_df = DataFrameMapper([
('pet', sklearn.preprocessing.LabelBinarizer()),
['children'], sklearn.preprocessing.StandardScaler())
], df_out=True)
np.round(mapper_df.fit_transform(data.copy()), 2)
   pet_cat  pet_dog  pet_fish  children
0        1        0         0      0.21
1        0        1         0      1.88
2        0        1         0     -0.63
3        0        0         1     -0.63
4        1        0         0     -1.46
5        0        1         0     -0.63
6        1        0         0      1.04
7        0        0         1      0.21`
```
I gave it some thought last night and this morning.

It could be good! We already use scikit-learn underneath the hood, I think. 

One thing I did was to profile how fast using sklearn was compared to a custom implementation of one-hot encoding, and I found my custom implementation was faster. That said, for convenience, I ended up just using the get_dummies function provided by pandas. Maybe measuring function speed could be helpful here in deciding which implementation to use?
I am sort of now leaning toward doing the documentation of both methods.  One with the normalize_columns and standardize_columns with Bayesian or classical statistics and another doing machine learning.  However, I am going to study how the mlxtend package handles this and see what I can learn. 
I like to close this one as I have really learn to use your transform_columns function for this.  I also love to use the transform_column to edit text stings with a lambda s: s.str.replace().  It works great.  What are your thoughts? Maybe we show examples of this in the documentation?  I can maybe help provide you a couple examples if you don’t have some ready?

<sub>Sent with <a href="http://githawk.com">GitHawk</a></sub>
@jcvall examples are amazing! I would love to have them added to pyjanitor's documentation. One of the "curses" of being a package developer is that we get too familiar with the package to write good examples. Would you like to get a few started?
Yes!

[pyjanitor_normalize.pdf](https://github.com/ericmjl/pyjanitor/files/3129933/pyjanitor_normalize.pdf)

not sure what form to send it in.  Here is pdf....
@jcvall I like what you did! I think you can do the following to make it a contribution to pyjanitor's documentation:

1. Perhaps annotate which functions you used are `pyjanitor` functions?
2. Add some prose that puts in plain English what you're trying to do?
3. Submit a PR with the notebook! (The best place for this is probably under `docs/` for now.) Once it's PR-d in, I'll find a way to include it as part of the hosted documentation.

As usual, I'm happy to guide you along :smile:, especially if there are `git` things that are blocking you.
I think I can produce this as requested from @jcvall 's PDF if needed!
Thank you, @catherinedevlin, this would be a welcome contribution! I will mark this as being worked on.

Resolving patch:
diff --git a/janitor/functions.py b/janitor/functions.py
--- a/janitor/functions.py
+++ b/janitor/functions.py
@@ -331,7 +331,8 @@ def get_features_targets(
         import janitor
         df = pd.DataFrame(...)
         target_cols = ['output1', 'output2']
-        X, y = df.get_features_targets(target_column_names=target_cols)  # noqa: E501
+        X, y = df.get_features_targets(target_column_names=target_cols)
+        # noqa: E501
 
     :param df: The pandas DataFrame object.
     :param str/iterable target_column_names: Either a column name or an
@@ -377,7 +378,8 @@ def rename_column(
 
         import pandas as pd
         import janitor
-        df = pd.DataFrame(...).rename_column("old_column_name", "new_column_name")  # noqa: E501
+        df = pd.DataFrame(...).rename_column("old_column_name",
+        "new_column_name")  # noqa: E501
 
     This is just syntactic sugar/a convenience function for renaming one column
     at a time. If you are convinced that there are multiple columns in need of
@@ -456,7 +458,7 @@ def reorder_columns(
 def coalesce(
     df: pd.DataFrame,
     column_names: Union[str, Iterable[str], Any],
-    new_column_name,
+    new_column_name: str = None,
 ) -> pd.DataFrame:
     """
 
@@ -466,7 +468,7 @@ def coalesce(
 
     .. code-block:: python
 
-        df = coalesce(df, column_names=['col1', 'col2'])
+        df = coalesce(df, columns=['col1', 'col2'], 'col3'
 
     Method chaining example:
 
@@ -476,8 +478,9 @@ def coalesce(
         import janitor
         df = pd.DataFrame(...).coalesce(['col1', 'col2'])
 
-    The result of this function is that we take the first non-null value across
-    rows.
+    The first example will create a new column called 'col3' with values from
+    'col2' inserted where values from 'col1' are NaN, then delete the original
+    columns. The second example will keep the name 'col1' in the new column.
 
     This is more syntactic diabetes! For R users, this should look familiar to
     `dplyr`'s `coalesce` function; for Python users, the interface
@@ -495,6 +498,8 @@ def _coalesce(series1, series2):
         return series1.combine_first(series2)
 
     df = df.drop(columns=column_names)
+    if not new_column_name:
+        new_column_name = column_names[0]
     df[new_column_name] = reduce(_coalesce, series)  # noqa: F821
     return df
 
@@ -819,7 +824,8 @@ def filter_string(
     .. code-block:: python
 
         df = (pd.DataFrame(...)
-              .filter_string('column', search_string='pattern', complement=False)  # noqa: E501
+              .filter_string('column', search_string='pattern', c
+                                omplement=False)  # noqa: E501
               ...)  # chain on more data preprocessing.
 
     This stands in contrast to the in-place syntax that is usually used:


Test patch:
diff --git a/tests/functions/test_coalesce.py b/tests/functions/test_coalesce.py
--- a/tests/functions/test_coalesce.py
+++ b/tests/functions/test_coalesce.py
@@ -4,9 +4,21 @@
 
 
 @pytest.mark.functions
-def test_coalesce():
+def test_coalesce_with_title():
     df = pd.DataFrame(
         {"a": [1, np.nan, 3], "b": [2, 3, 1], "c": [2, np.nan, 9]}
-    ).coalesce(["a", "b", "c"], "a")
+    ).coalesce(["a", "b", "c"], "d")
+    assert "a" not in df.columns
+    assert df.shape == (3, 1)
+    assert pd.isnull(df).sum().sum() == 0
+
+
+@pytest.mark.functions
+def test_coalesce_without_title():
+    df = pd.DataFrame(
+        {"a": [1, np.nan, 3], "b": [2, 3, 1], "c": [2, np.nan, 9]}
+    ).coalesce(["a", "b", "c"])
+    assert "a" in df.columns
+    assert "b" not in df.columns
     assert df.shape == (3, 1)
     assert pd.isnull(df).sum().sum() == 0