Repository: pyjanitor-devs/pyjanitor

Issue:
[ENH] Proposing a "jitter" function
# Brief Description

Had to use this at work recently, where I had to jitter the data that we had to slightly anonymize it.

All that we did here was to add Gaussian noise scaled by a fraction of the magnitude of each  value.

# Example API

This could operate at two levels: at the pandas Series and DataFrame levels. I implemented the "Series" version at work, but not the "DataFrame" one.

As a Series method, we would do something like this:

```python
from typing import Optional, Tuple

def jitter(s: pd.Series, scale_magnitude: np.number, clip: Optional[Tuple[float, float]] = None):
    """
    Jitter a pandas Series by applying Gaussian noise.
    
    By default, the jitter function takes the numeric value of each series element as the mean,
    and uses a multiplier of the numeric value (scale_magnitude) as the standard deviation.
    """
    loc = 0
    scale = s * scale_magnitude
    noise = np.random.normal(loc=0, scale=scale)
    result = s + noise
    if clip:
        result = np.clip(result, *clip)
        
    return result
```

I think it might be possible to do the analogous thing for DataFrames.

The API would look like the following:

```python
# series  API
s = df['column_name']
s.jitter()

# dataframe API
df = pd.DataFrame(...).jitter("column_name", scale_magnitude=0.1, clip=(0, None))
```

What do others think?



Discussion:
I think it'd be great! Jitter is a very helpful tool for plotting.

I would suggest being able to set the random seed, if only for testing purposes.
It's been a while since I've contributed a PR :)  If no one else wants it, I'd be happy to put something together (hopefully in the next week or so).

This should be added to the functions module, I presume?
Woohoo, that’d be great, @rahosbach! 

Yes, functions sounds like the right place to put it. I can’t think of another place where this might fit.
One other thought - if you have any data with 0 value, it won't get jittered, and if your data spans a order on magnitude or more, you'll see noticably differing levels of jitter. Maybe include the option for a constant about of jitter as well?
That's an interesting point, @anzelpwj.  

I've never had to implement a jitter function before, but one option might be to add an optional argument called `zero_scale` (or similar) where the user could input a floating point value that would correspond to `scale` parameter in `np.random.normal()` when the `s` value is 0.  That way, noise could still be randomly generated for those values.

Thoughts?  Bad idea?
Maybe you have two parameters, `scale` and `scale_const`, and your jitter magnitude is `scale*x + scale_const`
But `scale` is used to set the standard deviation of the Gaussian distribution we're sampling the noise from (at least in @ericmjl's example).  So are you saying that the Gaussian standard deviation should be set to `scale*x + scale_const`?  In that case, `scale_const` acts in the same way as my proposed `zero_scale`.

To simplify things, what if we changed our approach to have the mean of the Gaussian distribution be the actual series value (`s`), and then the user simply specifies the `scale` parameter that will set the standard deviation of the Gaussian distribution?  Then, the actual values we sample are the jittered values, and we would avoid both the zero-magnitude issue and the issue of differing levels of jitter for series with a wide range of values.

Example:
```python
def jitter(s: pd.Series, scale_magnitude: np.number, clip: Optional[Tuple[float, float]] = None):
    result = np.random.normal(loc=s, scale=scale_magnitude)
    if clip:
        result = np.clip(result, *clip)
        
    return result
@anzelpwj @rahosbach I _think_ I'm correct on the following point, but would love to be corrected if I'm wrong: `constant + N(0, scale) == N(constant, scale)` - it's where the reparameterization trick in stats comes from, I think. 

Pragmatically, your implementation is probably less confusing than mine, @rahosbach, so let's go with your implementation. 

@anzelpwj if we have data that span multiple orders of magnitude, is it a better idea to jitter the log-transform? I don't know why but I have this hunch that this might be a "good" way to jitter, but if I'm wrong, let me know.
Scaling the jitter by x value should basically jitter the log transform. But there's still the problem of pesky 0 values (or even negative values? Why shouldn't they get some of the fun?).

I see the code working like this

```
from typing import Optional, Tuple

def jitter(s: pd.Series, scale_magnitude: np.number, scale_constant: np.number = 0, clip: Optional[Tuple[float, float]] = None, random_state: np.number = None):
    """
    Jitter a pandas Series by applying Gaussian noise.
    
    By default, the jitter function takes the numeric value of each series element as the mean,
    and uses a multiplier of the numeric value (scale_magnitude) as the standard deviation.
    """
    if random_state is not None:
        np.random.seed(random_state)
    scale = s * scale_magnitude + scale_constant
    result = np.random.normal(loc=s, scale=scale)
    if clip:
        result = np.clip(result, *clip)
        
    return result
```
I guess I'm still wondering why `scale` needs to be tied to the actual series value, `s`?  In that case, you would see more jitter for larger values, and vice-versa, which doesn't sound necessary.

This is why I proposed letting the user set a constant `scale_magnitude` value that would be used, by itself, to define the standard deviation of the distribution we sample from.  In effect, this sets a constant jitter level that does not depend on the actual series value.  Then, if we set the distribution's mean to be the actual series value, then the value we sample will necessarily be a jittered value that we can use as the result.  _And_, this would even jitter 0 values.

Sorry to keep this thread going and going.  I just want to ensure I completely understand the implementation choices we're making and why we're making them.
Mostly that's what I saw in Eric's original implementation `scale = s * scale_magnitude` -- if we want to just do a constant scale that could work as well, though maybe if we want to have constant jitter for a log-plot, then scaling based on series value would be a good idea.
Good point.  But, if the idea is to jitter for a log-plot, then couldn't the series values we use be log-transformed first?  Then, we would be adding jitter to (previously) log-transformed values and still adding a defined amount of random jitter.

Essentially, we would leave any log-transformation (as well as whatever `scale_magnitude` is appropriate) up to the user to handle prior to running this method.  I think that should still work.
@rahosbach in the spirit of explicit is better than implicit, I think you've nailed what the function ought to be like. Let's go ahead with what you have, what say you?
@ericmjl I'm happy to move ahead with the PR now.  @anzelpwj I'll be sure to add you as a reviewer on it so that you can see what I've done and provide any more suggestions.

Resolving patch:
diff --git a/janitor/functions.py b/janitor/functions.py
--- a/janitor/functions.py
+++ b/janitor/functions.py
@@ -13,12 +13,12 @@
     Hashable,
     Iterable,
     List,
+    Optional,
     Set,
     Tuple,
     Union,
 )
 
-
 import numpy as np
 import pandas as pd
 import pandas_flavor as pf
@@ -3707,3 +3707,111 @@ def toset(series: pd.Series) -> Set:
     """
 
     return set(series.tolist())
+
+
+@pf.register_dataframe_method
+def jitter(
+    df: pd.DataFrame,
+    column_name: Hashable,
+    dest_column_name: str,
+    scale: np.number,
+    clip: Optional[Iterable[np.number]] = None,
+    random_state: Optional[np.number] = None,
+) -> pd.DataFrame:
+    """
+    Adds Gaussian noise (jitter) to the values of a column.
+
+    Functional usage example:
+
+    .. code-block:: python
+
+        import pandas as pd
+        import janitor as jn
+
+        df = pd.DataFrame(...)
+
+        df = jn.functions.jitter(
+            df=df,
+            column_name='values',
+            dest_column_name='values_jitter',
+            scale=1.0,
+            clip=None,
+            random_state=None,
+        )
+
+    Method chaining usage example:
+
+    .. code-block:: python
+
+        import pandas as pd
+        import janitor.functions
+
+        df = pd.DataFrame(...)
+
+        df = df.jitter(
+            column_name='values',
+            dest_column_name='values_jitter',
+            scale=1.0,
+            clip=None,
+            random_state=None,
+        )
+
+    A new column will be created containing the values of the original column
+    with Gaussian noise added.
+    For each value in the column, a Gaussian distribution is created
+    having a location (mean) equal to the value
+    and a scale (standard deviation) equal to `scale`.
+    A random value is then sampled from this distribution,
+    which is the jittered value.
+    If a tuple is supplied for `clip`,
+    then any values of the new column less than `clip[0]`
+    will be set to `clip[0]`,
+    and any values greater than `clip[1]` will be set to `clip[1]`.
+    Additionally, if a numeric value is supplied for `random_state`,
+    this value will be used to set the random seed used for sampling.
+    NaN values are ignored in this method.
+
+    This method mutates the original DataFrame.
+
+    :param df: A pandas dataframe.
+    :param column_name: Name of the column containing
+        values to add Gaussian jitter to.
+    :param dest_column_name: The name of the new column containing the
+        jittered values that will be created.
+    :param scale: A positive value multiplied by the original
+        column value to determine the scale (standard deviation) of the
+        Gaussian distribution to sample from. (A value of zero results in
+        no jittering.)
+    :param clip: An iterable of two values (minimum and maximum) to clip
+        the jittered values to, default to None.
+    :param random_state: A interger or 1-d array value used to set the random
+        seed, default to None.
+
+    :returns: A pandas DataFrame with a new column containing Gaussian-
+        jittered values from another column.
+    """
+
+    # Check types
+    check("scale", scale, [int, float])
+
+    # Check that `column_name` is a numeric column
+    if not np.issubdtype(df[column_name].dtype, np.number):
+        raise TypeError(f"{column_name} must be a numeric column.")
+
+    if scale <= 0:
+        raise ValueError("`scale` must be a numeric value greater than 0.")
+    values = df[column_name]
+    if random_state is not None:
+        np.random.seed(random_state)
+    result = np.random.normal(loc=values, scale=scale)
+    if clip:
+        # Ensure `clip` has length 2
+        if len(clip) != 2:
+            raise ValueError("`clip` must be an iterable of length 2.")
+        # Ensure the values in `clip` are ordered as min, max
+        if clip[1] < clip[0]:
+            raise ValueError("`clip[0]` must be less than `clip[1]`.")
+        result = np.clip(result, *clip)
+    df[dest_column_name] = result
+
+    return df
diff --git a/janitor/xarray/functions.py b/janitor/xarray/functions.py
--- a/janitor/xarray/functions.py
+++ b/janitor/xarray/functions.py
@@ -7,12 +7,13 @@
 from typing import Union
 
 import numpy as np
-import xarray as xr
 from pandas_flavor import (
     register_xarray_dataarray_method,
     register_xarray_dataset_method,
 )
 
+import xarray as xr
+
 
 @register_xarray_dataarray_method
 def clone_using(


Test patch:
diff --git a/tests/functions/test_jitter.py b/tests/functions/test_jitter.py
new file mode 100644
--- /dev/null
+++ b/tests/functions/test_jitter.py
@@ -0,0 +1,119 @@
+import pandas as pd
+import pytest
+
+from janitor.functions import jitter
+
+
+@pytest.mark.functions
+def test_datatypes_check(dataframe):
+    # `scale` should be a numeric value > 0
+    with pytest.raises(TypeError):
+        assert dataframe.jitter(
+            column_name="a", dest_column_name="a_jitter", scale="x"
+        )
+
+    # `random_state` should be an integer or 1-d array
+    # (see documentation for np.random.seed)
+    with pytest.raises(TypeError):
+        assert dataframe.jitter(
+            column_name="a",
+            dest_column_name="a_jitter",
+            scale=1,
+            random_state="x",
+        )
+
+    # `clip` should only contain numeric values
+    with pytest.raises(TypeError):
+        assert dataframe.jitter(
+            column_name="a",
+            dest_column_name="a_jitter",
+            scale=1,
+            clip=["x", 2],
+        )
+
+    # The column to jitter should be numeric
+    with pytest.raises(TypeError):
+        assert dataframe.jitter(
+            column_name="cities", dest_column_name="cities_jitter", scale=1
+        )
+
+    # `scale` should be greater than 0
+    with pytest.raises(ValueError):
+        assert dataframe.jitter(
+            column_name="a", dest_column_name="a_jitter", scale=-5
+        )
+
+    # `clip` should be a size-2 tuple of numeric values
+    with pytest.raises(ValueError):
+        assert dataframe.jitter(
+            column_name="a",
+            dest_column_name="a_jitter",
+            scale=1,
+            clip=[-10, 10, 5],
+        )
+
+    # `clip[0]` should be less than `clip[1]`
+    with pytest.raises(ValueError):
+        assert dataframe.jitter(
+            column_name="a", dest_column_name="a_jitter", scale=1, clip=[10, 5]
+        )
+
+
+@pytest.mark.functions
+def test_jitter(dataframe):
+    # Functional test to ensure jitter runs without error
+    dataframe.jitter(column_name="a", dest_column_name="a_jitter", scale=1.0)
+
+
+@pytest.mark.functions
+def test_jitter_with_nans(missingdata_df):
+    # Functional test to ensure jitter runs without error if NaNs are present
+    missingdata_df.jitter(
+        column_name="a", dest_column_name="a_jitter", scale=1.0
+    )
+
+
+@pytest.mark.functions
+def test_jitter_random_state(dataframe):
+    # Functional test to ensure jitter runs when setting random seed
+    dataframe.jitter(
+        column_name="a",
+        dest_column_name="a_jitter",
+        scale=1.0,
+        random_state=77,
+    )
+
+
+@pytest.mark.functions
+def test_jitter_clip(dataframe):
+    # Ensure clip works as intended
+    df = dataframe.jitter(
+        column_name="a",
+        dest_column_name="a_jitter",
+        scale=1.0,
+        clip=[1.5, 2.5],
+    )
+    assert (min(df["a_jitter"]) >= 1.5) & (max(df["a_jitter"]) <= 2.5)
+
+
+@pytest.mark.functions
+def test_jitter_results():
+    """Ensure the mean of the jittered values is approximately
+    equal to the mean of the original values, and that the
+    standard deviation of the jittered value is approximately
+    equal to the `scale` parameter."""
+    error_tolerance = 0.05  # 5%
+    scale = 2.0
+
+    df = pd.DataFrame({"original": [1] * 1000})
+    results = df.jitter(
+        column_name="original", dest_column_name="jittered", scale=scale
+    )
+    assert (
+        abs(
+            (results["jittered"].mean() - results["original"].mean())
+            / results["original"].mean()
+        )
+        <= error_tolerance
+    )
+    assert abs((results["jittered"].std() - scale) / scale) <= error_tolerance
diff --git a/tests/xarray/conftest.py b/tests/xarray/conftest.py
--- a/tests/xarray/conftest.py
+++ b/tests/xarray/conftest.py
@@ -1,5 +1,6 @@
 import numpy as np
 import pytest
+
 import xarray as xr
 
 
diff --git a/tests/xarray/test_clone_using.py b/tests/xarray/test_clone_using.py
--- a/tests/xarray/test_clone_using.py
+++ b/tests/xarray/test_clone_using.py
@@ -1,8 +1,8 @@
 import numpy as np
 import pytest
-import xarray as xr
 
 import janitor  # noqa: F401
+import xarray as xr
 
 
 @pytest.mark.xarray
diff --git a/tests/xarray/test_convert_datetime_to_number.py b/tests/xarray/test_convert_datetime_to_number.py
--- a/tests/xarray/test_convert_datetime_to_number.py
+++ b/tests/xarray/test_convert_datetime_to_number.py
@@ -1,5 +1,6 @@
 import numpy as np
 import pytest
+
 import xarray as xr