Repository: ericmjl/pyjanitor (python)

Issue:
[ENH] Proposing a "jitter" function
# Brief Description

Had to use this at work recently, where I had to jitter the data that we had to slightly anonymize it.

All that we did here was to add Gaussian noise scaled by a fraction of the magnitude of each  value.

# Example API

This could operate at two levels: at the pandas Series and DataFrame levels. I implemented the "Series" version at work, but not the "DataFrame" one.

As a Series method, we would do something like this:

```python
from typing import Optional, Tuple

def jitter(s: pd.Series, scale_magnitude: np.number, clip: Optional[Tuple[float, float]] = None):
    """
    Jitter a pandas Series by applying Gaussian noise.
    
    By default, the jitter function takes the numeric value of each series element as the mean,
    and uses a multiplier of the numeric value (scale_magnitude) as the standard deviation.
    """
    loc = 0
    scale = s * scale_magnitude
    noise = np.random.normal(loc=0, scale=scale)
    result = s + noise
    if clip:
        result = np.clip(result, *clip)
        
    return result
```

I think it might be possible to do the analogous thing for DataFrames.

The API would look like the following:

```python
# series  API
s = df['column_name']
s.jitter()

# dataframe API
df = pd.DataFrame(...).jitter("column_name", scale_magnitude=0.1, clip=(0, None))
```

What do others think?


Pull request:
[ENH] Issue 622: Jitter Method
# PR Description

Please describe the changes proposed in the pull request:

- Added jitter() method to functions.py.
- Added related tests in test_jitter.py.
- Updated docs in general_functions.rst.
- Updated CHANGELOG.rst.
- Linting and Black formatting updated some other xarray modules as well (not sure why that is, but they are included here).

**This PR resolves #622.**

# PR Checklist

Please ensure that you have done the following:

1. [x] PR in from a fork off your branch. Do not PR from `<your_username>`:`dev`, but rather from `<your_username>`:`<feature-branch_name>`.
2. [x] If you're not on the contributors list, add yourself to `AUTHORS.rst`.
3. [x] Add a line to `CHANGELOG.rst` under the latest version header (i.e. the one that is "on deck") describing the contribution.
    - Do use some discretion here; if there are multiple PRs that are related, keep them in a single line.

## Quick Check

To do a very quick check that everything is correct, follow these steps below:

- [x] Run the command `make check` from pyjanitor's top-level directory. This will automatically run:
    - black formatting
    - pycodestyle checking
    - running the test suite
    - docs build

**Everything ran, and tests passed for what I added; but, some xarray tests failed that I did not touch.**

## Code Changes

If you are adding code changes, please ensure the following:

- [x] Ensure that you have added tests.
- [x] Run all tests (`$ pytest .`) locally on your machine.
    - [x] Check to ensure that test coverage covers the lines of code that you have added.
    - [x] Ensure that all tests pass.

## Documentation Changes

If you are adding documentation changes, please ensure the following:

- [x] Build the docs locally.
- [x] View the docs to check that it renders correctly.

# Relevant Reviewers

Please tag maintainers to review.

- @ericmjl
- @anzelpwj 


Patch:
diff --git a/CHANGELOG.rst b/CHANGELOG.rst
index b44b717..9d64564 100644
--- a/CHANGELOG.rst
+++ b/CHANGELOG.rst
@@ -6,6 +6,7 @@ new version (on deck)
 - [DOC] Update API policy for clarity. @ericmjl
 - [ENH] Enforce string conversion when cleaning names. @ericmjl
 - [ENH] Change ``find_replace`` implementation to use keyword arguments to specify columns to perform find and replace on. @ericmjl
+- [ENH] Add ``jitter()`` dataframe function by @rahosbach
 
 v0.19.0
 =======
diff --git a/docs/reference/general_functions.rst b/docs/reference/general_functions.rst
index b1f7006..706c03a 100644
--- a/docs/reference/general_functions.rst
+++ b/docs/reference/general_functions.rst
@@ -44,6 +44,7 @@ Modify values
     round_to_fraction
     update_where
     to_datetime
+    jitter
 
 Filtering
 ~~~~~~~~~
diff --git a/docs/reference/janitor.functions/janitor.jitter.md b/docs/reference/janitor.functions/janitor.jitter.md
new file mode 100644
index 0000000..dc4f14a
--- /dev/null
+++ b/docs/reference/janitor.functions/janitor.jitter.md
@@ -0,0 +1,6 @@
+janitor.jitter
+==============
+
+.. currentmodule:: janitor
+
+.. autofunction:: jitter
\ No newline at end of file
diff --git a/janitor/functions.py b/janitor/functions.py
index 6b78be1..7434549 100644
--- a/janitor/functions.py
+++ b/janitor/functions.py
@@ -13,12 +13,12 @@ from typing import (
     Hashable,
     Iterable,
     List,
+    Optional,
     Set,
     Tuple,
     Union,
 )
 
-
 import numpy as np
 import pandas as pd
 import pandas_flavor as pf
@@ -3707,3 +3707,111 @@ def toset(series: pd.Series) -> Set:
     """
 
     return set(series.tolist())
+
+
+@pf.register_dataframe_method
+def jitter(
+    df: pd.DataFrame,
+    column_name: Hashable,
+    dest_column_name: str,
+    scale: np.number,
+    clip: Optional[Iterable[np.number]] = None,
+    random_state: Optional[np.number] = None,
+) -> pd.DataFrame:
+    """
+    Adds Gaussian noise (jitter) to the values of a column.
+
+    Functional usage example:
+
+    .. code-block:: python
+
+        import pandas as pd
+        import janitor as jn
+
+        df = pd.DataFrame(...)
+
+        df = jn.functions.jitter(
+            df=df,
+            column_name='values',
+            dest_column_name='values_jitter',
+            scale=1.0,
+            clip=None,
+            random_state=None,
+        )
+
+    Method chaining usage example:
+
+    .. code-block:: python
+
+        import pandas as pd
+        import janitor.functions
+
+        df = pd.DataFrame(...)
+
+        df = df.jitter(
+            column_name='values',
+            dest_column_name='values_jitter',
+            scale=1.0,
+            clip=None,
+            random_state=None,
+        )
+
+    A new column will be created containing the values of the original column
+    with Gaussian noise added.
+    For each value in the column, a Gaussian distribution is created
+    having a location (mean) equal to the value
+    and a scale (standard deviation) equal to `scale`.
+    A random value is then sampled from this distribution,
+    which is the jittered value.
+    If a tuple is supplied for `clip`,
+    then any values of the new column less than `clip[0]`
+    will be set to `clip[0]`,
+    and any values greater than `clip[1]` will be set to `clip[1]`.
+    Additionally, if a numeric value is supplied for `random_state`,
+    this value will be used to set the random seed used for sampling.
+    NaN values are ignored in this method.
+
+    This method mutates the original DataFrame.
+
+    :param df: A pandas dataframe.
+    :param column_name: Name of the column containing
+        values to add Gaussian jitter to.
+    :param dest_column_name: The name of the new column containing the
+        jittered values that will be created.
+    :param scale: A positive value multiplied by the original
+        column value to determine the scale (standard deviation) of the
+        Gaussian distribution to sample from. (A value of zero results in
+        no jittering.)
+    :param clip: An iterable of two values (minimum and maximum) to clip
+        the jittered values to, default to None.
+    :param random_state: A interger or 1-d array value used to set the random
+        seed, default to None.
+
+    :returns: A pandas DataFrame with a new column containing Gaussian-
+        jittered values from another column.
+    """
+
+    # Check types
+    check("scale", scale, [int, float])
+
+    # Check that `column_name` is a numeric column
+    if not np.issubdtype(df[column_name].dtype, np.number):
+        raise TypeError(f"{column_name} must be a numeric column.")
+
+    if scale <= 0:
+        raise ValueError("`scale` must be a numeric value greater than 0.")
+    values = df[column_name]
+    if random_state is not None:
+        np.random.seed(random_state)
+    result = np.random.normal(loc=values, scale=scale)
+    if clip:
+        # Ensure `clip` has length 2
+        if len(clip) != 2:
+            raise ValueError("`clip` must be an iterable of length 2.")
+        # Ensure the values in `clip` are ordered as min, max
+        if clip[1] < clip[0]:
+            raise ValueError("`clip[0]` must be less than `clip[1]`.")
+        result = np.clip(result, *clip)
+    df[dest_column_name] = result
+
+    return df
diff --git a/janitor/xarray/functions.py b/janitor/xarray/functions.py
index 7e55732..80ae6b5 100644
--- a/janitor/xarray/functions.py
+++ b/janitor/xarray/functions.py
@@ -7,12 +7,13 @@ functionality.
 from typing import Union
 
 import numpy as np
-import xarray as xr
 from pandas_flavor import (
     register_xarray_dataarray_method,
     register_xarray_dataset_method,
 )
 
+import xarray as xr
+
 
 @register_xarray_dataarray_method
 def clone_using(


Test patch:
diff --git a/tests/functions/test_jitter.py b/tests/functions/test_jitter.py
new file mode 100644
index 0000000..d1eaee5
--- /dev/null
+++ b/tests/functions/test_jitter.py
@@ -0,0 +1,119 @@
+import pandas as pd
+import pytest
+
+from janitor.functions import jitter
+
+
+@pytest.mark.functions
+def test_datatypes_check(dataframe):
+    # `scale` should be a numeric value > 0
+    with pytest.raises(TypeError):
+        assert dataframe.jitter(
+            column_name="a", dest_column_name="a_jitter", scale="x"
+        )
+
+    # `random_state` should be an integer or 1-d array
+    # (see documentation for np.random.seed)
+    with pytest.raises(TypeError):
+        assert dataframe.jitter(
+            column_name="a",
+            dest_column_name="a_jitter",
+            scale=1,
+            random_state="x",
+        )
+
+    # `clip` should only contain numeric values
+    with pytest.raises(TypeError):
+        assert dataframe.jitter(
+            column_name="a",
+            dest_column_name="a_jitter",
+            scale=1,
+            clip=["x", 2],
+        )
+
+    # The column to jitter should be numeric
+    with pytest.raises(TypeError):
+        assert dataframe.jitter(
+            column_name="cities", dest_column_name="cities_jitter", scale=1
+        )
+
+    # `scale` should be greater than 0
+    with pytest.raises(ValueError):
+        assert dataframe.jitter(
+            column_name="a", dest_column_name="a_jitter", scale=-5
+        )
+
+    # `clip` should be a size-2 tuple of numeric values
+    with pytest.raises(ValueError):
+        assert dataframe.jitter(
+            column_name="a",
+            dest_column_name="a_jitter",
+            scale=1,
+            clip=[-10, 10, 5],
+        )
+
+    # `clip[0]` should be less than `clip[1]`
+    with pytest.raises(ValueError):
+        assert dataframe.jitter(
+            column_name="a", dest_column_name="a_jitter", scale=1, clip=[10, 5]
+        )
+
+
+@pytest.mark.functions
+def test_jitter(dataframe):
+    # Functional test to ensure jitter runs without error
+    dataframe.jitter(column_name="a", dest_column_name="a_jitter", scale=1.0)
+
+
+@pytest.mark.functions
+def test_jitter_with_nans(missingdata_df):
+    # Functional test to ensure jitter runs without error if NaNs are present
+    missingdata_df.jitter(
+        column_name="a", dest_column_name="a_jitter", scale=1.0
+    )
+
+
+@pytest.mark.functions
+def test_jitter_random_state(dataframe):
+    # Functional test to ensure jitter runs when setting random seed
+    dataframe.jitter(
+        column_name="a",
+        dest_column_name="a_jitter",
+        scale=1.0,
+        random_state=77,
+    )
+
+
+@pytest.mark.functions
+def test_jitter_clip(dataframe):
+    # Ensure clip works as intended
+    df = dataframe.jitter(
+        column_name="a",
+        dest_column_name="a_jitter",
+        scale=1.0,
+        clip=[1.5, 2.5],
+    )
+    assert (min(df["a_jitter"]) >= 1.5) & (max(df["a_jitter"]) <= 2.5)
+
+
+@pytest.mark.functions
+def test_jitter_results():
+    """Ensure the mean of the jittered values is approximately
+    equal to the mean of the original values, and that the
+    standard deviation of the jittered value is approximately
+    equal to the `scale` parameter."""
+    error_tolerance = 0.05  # 5%
+    scale = 2.0
+
+    df = pd.DataFrame({"original": [1] * 1000})
+    results = df.jitter(
+        column_name="original", dest_column_name="jittered", scale=scale
+    )
+    assert (
+        abs(
+            (results["jittered"].mean() - results["original"].mean())
+            / results["original"].mean()
+        )
+        <= error_tolerance
+    )
+    assert abs((results["jittered"].std() - scale) / scale) <= error_tolerance
diff --git a/tests/xarray/conftest.py b/tests/xarray/conftest.py
index 1b67307..81b3f16 100644
--- a/tests/xarray/conftest.py
+++ b/tests/xarray/conftest.py
@@ -1,5 +1,6 @@
 import numpy as np
 import pytest
+
 import xarray as xr
 
 
diff --git a/tests/xarray/test_clone_using.py b/tests/xarray/test_clone_using.py
index 25fe37e..8341ffa 100644
--- a/tests/xarray/test_clone_using.py
+++ b/tests/xarray/test_clone_using.py
@@ -1,8 +1,8 @@
 import numpy as np
 import pytest
-import xarray as xr
 
 import janitor  # noqa: F401
+import xarray as xr
 
 
 @pytest.mark.xarray
diff --git a/tests/xarray/test_convert_datetime_to_number.py b/tests/xarray/test_convert_datetime_to_number.py
index 4bfe924..f877bd1 100644
--- a/tests/xarray/test_convert_datetime_to_number.py
+++ b/tests/xarray/test_convert_datetime_to_number.py
@@ -1,5 +1,6 @@
 import numpy as np
 import pytest
+
 import xarray as xr