Repository: mitre/menelaus (language: python)

Issue:
Design synthetic drift utilities
Section 3 of Souza et. al. 2020 gives a good summary of potential approaches.

The "floor" could be including examples of using these methods to inject drift.

The "ceiling" would be developing independent utilities to make some of the work easier. Even if not, they may be worth making note of, in case the code-base reaches a point where replicating the examples "by hand" is annoying, e.g. pipeline-like objects.


Discussion / hints:
Comments from gitlab issue: 

1. paper here, ["Challenges in Benchmarking Stream Learning
Algorithms with Real-world Data"](https://arxiv.org/pdf/2005.00113.pdf).
- Page 20 has a list. Having a utility with a couple dials to moderate these approaches in reasonable ways, e.g. "proportion of labels flip-flopped" might be useful.



2. Convert ARFF format to something we can use? https://sites.google.com/view/uspdsrepository
  * Forest covertype seems like a candidate for temporospatial data we could use as a test.
* General approaches for ensembling models of different ages?
  * Conceptually, building an incremental learner on top of a "batch learner," so that the same tools and approach can be used, at least temporarily, in a context which is now recognized as streaming.
* Need to read through section 4 more thoroughly.

3. can consider adding a random walk / brownian noise as described in this [paper](http://csis.p%5D(url)ace.edu/~ctappert/srd2020/b12.pdf). it is intended for time series data but we should be able to modify it for streaming / batch


Adding a note to myself about using `toolz` or the `@curry` operator to make any function I develop, able to be passed along as a pipeline. (Just an idea):

```python
while not finished:
    data = pipe(*list_of_fns, data)

list_of_fns = [join_class_function, swap_class_function]

def join_class_function():
    ...
```

Fix patch:
diff --git a/LICENSE.txt b/LICENSE.txt
index 48597a1b..c390c0c1 100644
--- a/LICENSE.txt
+++ b/LICENSE.txt
@@ -1,3 +1,4 @@
+
                                  Apache License
                            Version 2.0, January 2004
                         http://www.apache.org/licenses/
diff --git a/docs/source/refs.bib b/docs/source/refs.bib
index 9927a01b..f93f2478 100644
--- a/docs/source/refs.bib
+++ b/docs/source/refs.bib
@@ -1,4 +1,24 @@
-  @inproceedings{macielelectionreference,
+@inproceedings{fields2019mitigating,
+  title={Mitigating drift in time series data with noise augmentation},
+  author={Fields, Tonya and Hsieh, George and Chenou, Jules},
+  booktitle={2019 International Conference on Computational Science and Computational Intelligence (CSCI)},
+  pages={227--230},
+  year={2019},
+  organization={IEEE}
+}
+
+@article{souza2020challenges,
+  title={Challenges in benchmarking stream learning algorithms with real-world data},
+  author={Souza, Vinicius and dos Reis, Denis M and Maletzke, Andre G and Batista, Gustavo EAPA},
+  journal={Data Mining and Knowledge Discovery},
+  volume={34},
+  number={6},
+  pages={1805--1858},
+  year={2020},
+  publisher={Springer}
+}
+
+@inproceedings{macielelectionreference,
   author = {Maciel, Bruno Iran Ferreira and Santos, Silas and Barros, Roberto},
   year = {2015},
   month = {11},
@@ -7,7 +27,7 @@ @inproceedings{macielelectionreference
   doi = {10.1109/ICTAI.2015.151},
   booktitle={2015 IEEE 27th International Conference on Tools with Artificial Intelligence (ICTAI)},
   organization={IEEE}
-  }
+}
 
 @article{duelectionreference,
   author = {Du, Lei and Song, Qinbao and Zhu, Lei and Zhu, Xiaoyan},
diff --git a/menelaus/injection/__init__.py b/menelaus/injection/__init__.py
new file mode 100644
index 00000000..70f81995
--- /dev/null
+++ b/menelaus/injection/__init__.py
@@ -0,0 +1,8 @@
+"""This module contains functions with which to artificially inject drift into a dataset."""
+
+from menelaus.injection.feature_manipulation import (
+    feature_swap,
+    feature_hide_and_sample,
+)
+from menelaus.injection.class_manipulation import class_swap, class_join
+from menelaus.injection.noise import brownian_noise, random_walk
diff --git a/menelaus/injection/class_manipulation.py b/menelaus/injection/class_manipulation.py
new file mode 100644
index 00000000..6e382c47
--- /dev/null
+++ b/menelaus/injection/class_manipulation.py
@@ -0,0 +1,96 @@
+import numpy as np
+import pandas as pd
+
+
+def class_swap(data, target_col, class_1, class_2, from_index, to_index):
+    """
+    Swaps two classes in a target column of a given dataset with each other.
+    Accepts ``pandas.DataFrame`` with column names or ``numpy.ndarray``
+    with column indices.
+
+    Ref. :cite:t:`souza2020challenges`
+
+    Args:
+        data (np.array): data to inject with drift
+        target_col (int or str): column index/label of targets column
+        class_1 (int): value of first label in class swap
+        class_2 (int): value of second label in class swap
+        from_index: row index at which to start class swap
+        to_index: row index at which to end (non-inclusive) class swap
+
+    Returns:
+        np.array or pd.DataFrame: copy of data, with two classes swapped
+            in given target column, over given indices
+    """
+    if not isinstance(data, pd.DataFrame) and not isinstance(data, np.ndarray):
+        raise ValueError(f"Data of type {type(data)} not supported")
+    elif isinstance(data, pd.DataFrame):
+        ret = data.copy()
+        class_1_idx = ret.loc[
+            (ret.index < to_index)
+            & (ret.index >= from_index)
+            & (ret[target_col] == class_1)
+        ].index
+        class_2_idx = ret.loc[
+            (ret.index < to_index)
+            & (ret.index >= from_index)
+            & (ret[target_col] == class_2)
+        ].index
+        ret.loc[class_1_idx, target_col] = class_2
+        ret.loc[class_2_idx, target_col] = class_1
+    else:
+        ret = np.copy(data)
+        # np.where returns tuples pertaining to dimensions, only need the row
+        class_1_idx = np.where(ret[:, target_col] == class_1)[0]
+        class_1_idx = class_1_idx[
+            (class_1_idx < to_index) & (class_1_idx >= from_index)
+        ]
+        class_2_idx = np.where(ret[:, target_col] == class_2)[0]
+        class_2_idx = class_2_idx[
+            (class_2_idx < to_index) & (class_2_idx >= from_index)
+        ]
+        ret[class_1_idx, target_col] = class_2
+        ret[class_2_idx, target_col] = class_1
+    return ret
+
+
+def class_join(data, target_col, class_1, class_2, new_class, from_index, to_index):
+    """
+    Joins two [TODO or more?] classes in a unique class. Accepts
+    ``pandas.DataFrame`` with column names or ``numpy.ndarray``
+    with column indices.
+
+    Ref. :cite:t:`souza2020challenges`
+
+    Args:
+        data (np.array): data to inject with drift
+        target_col (int or str): column index/label of targets column
+        class_1 (int): value of first label in class join
+        class_2 (int): value of second label in class join,
+        new_class (int): new label value to assign to old classes
+        from_index: row index at which to start class join
+        to_index: row index at which to end (non-inclusive) class join
+
+    Returns:
+        np.array or pd.DataFrame: copy of data, with two classes joined
+            in given target column, over given indices, into new class
+    """
+    if not isinstance(data, pd.DataFrame) and not isinstance(data, np.ndarray):
+        raise ValueError(f"Data of type {type(data)} not supported")
+    elif isinstance(data, pd.DataFrame):
+        ret = data.copy()
+        class_idx = ret.loc[
+            (ret.index < to_index)
+            & (ret.index >= from_index)
+            & ((ret[target_col] == class_1) | (ret[target_col] == class_2))
+        ].index
+        ret.loc[class_idx, target_col] = new_class
+    else:
+        ret = np.copy(data)
+        # np.where returns tuples pertaining to dimensions, only need the row
+        class_idx = np.where(
+            (ret[:, target_col] == class_1) | (ret[:, target_col] == class_2)
+        )[0]
+        class_idx = class_idx[(class_idx < to_index) & (class_idx >= from_index)]
+        ret[class_idx, target_col] = new_class
+    return ret
diff --git a/menelaus/injection/feature_manipulation.py b/menelaus/injection/feature_manipulation.py
new file mode 100644
index 00000000..97e61d60
--- /dev/null
+++ b/menelaus/injection/feature_manipulation.py
@@ -0,0 +1,68 @@
+import numpy as np
+import pandas as pd
+
+
+def feature_swap(data, col_1, col_2, from_index, to_index):
+    """
+    Swaps two features/columns of a given dataset with each other.
+    Accepts ``pandas.DataFrame`` with column names or ``numpy.ndarray``
+    with column indices.
+
+    Ref. :cite:t:`souza2020challenges`
+
+    Args:
+        data (np.ndarray or pd.DataFrame): data to inject with drift
+        col_1 (int or str): column index/label of first column
+        col_2 (int or str): column index/label of second column
+        from_index (int): row index at which to start column swap
+        to_index (int): row index at which to end (non-inclusive) column swap
+
+    Returns:
+        np.ndarray or pd.DataFrame: copy of data, with two columns swapped
+            over given indices
+    """
+    if not isinstance(data, pd.DataFrame) and not isinstance(data, np.ndarray):
+        raise ValueError(f"Data of type {type(data)} not supported")
+    elif isinstance(data, pd.DataFrame):
+        ret = data.copy()
+        ret_col_1 = ret.loc[from_index:to_index, col_1].copy()
+        ret.loc[from_index:to_index, col_1] = ret.loc[from_index:to_index, col_2]
+        ret.loc[from_index:to_index, col_2] = ret_col_1
+    else:
+        ret = np.copy(data)
+        ret[from_index:to_index, [col_1, col_2]] = ret[
+            from_index:to_index, [col_2, col_1]
+        ]
+    return ret
+
+
+def feature_hide_and_sample(data, col, sample_size, random_state=0):
+    """
+    Hides a feature, then treats it as a shared concept by which to group the data.
+    Afterwards samples are uniformly drawn from each group. Accepts
+    ``pandas.DataFrame`` with column names or ``numpy.ndarray`` with column indices.
+
+    Ref. :cite:t:`souza2020challenges`
+
+    Args:
+        data (np.ndarray or pd.DataFrame): data to inject with drift
+        col (int or str): index/label of column to hide and re-sample (note this
+            should be a categorical feature that can be treated as a concept)
+        sample_size (int): data points to be drawn from each group in new concept
+        random_state (int): optional random seed. Default 0
+
+    Returns:
+        np.ndarray or pd.DataFrame: copy of data, grouped by indicated column,
+            with each group sampled and column removed
+    """
+    if not isinstance(data, pd.DataFrame) and not isinstance(data, np.ndarray):
+        raise ValueError(f"Data of type {type(data)} not supported")
+    elif isinstance(data, pd.DataFrame):
+        df = data.copy()
+    else:
+        df = np.copy(data)
+        df = pd.DataFrame(df)
+    ret = df.groupby(col).sample(n=sample_size, random_state=random_state)
+    ret = ret.drop(columns=[col]).reset_index(drop=True)
+    ret = ret if isinstance(data, pd.DataFrame) else ret.to_numpy()
+    return ret
diff --git a/menelaus/injection/noise.py b/menelaus/injection/noise.py
new file mode 100644
index 00000000..be512461
--- /dev/null
+++ b/menelaus/injection/noise.py
@@ -0,0 +1,55 @@
+import numpy as np
+import pandas as pd
+
+
+def brownian_noise(data, feature, x0, from_index, to_index):
+    """
+    Adds Brownian noise to a column of a given dataset. Accepts
+    ``pandas.DataFrame`` with column names or ``numpy.ndarray``
+    with column indices. Calls ``random_walk`` for full computation.
+
+    Ref. :cite:t:`fields2019mitigating`
+
+    Args:
+        data (np.ndarray or pd.DataFrame): data to inject with drift
+        feature (int or str): index/label of column to add noise to
+        x0 (int): starting point for random walk noise
+        from_index (int): row index at which to start injection
+        to_index (int): row index at which to end (non-inclusive) injection
+
+    Returns:
+        np.ndarray or pd.DataFrame: copy of data, with two columns swapped
+            over given indices
+    """
+    steps = to_index - from_index
+    if not isinstance(data, pd.DataFrame) and not isinstance(data, np.ndarray):
+        raise ValueError(f"Data of type {type(data)} not supported")
+    elif isinstance(data, pd.DataFrame):
+        ret = data.copy()
+        # unlike other operations, .loc[0:r] is inclusive, so -1
+        ret_col = ret.loc[from_index : to_index - 1, feature].copy()
+        ret_col = ret_col + random_walk(steps, x0)
+        ret.loc[from_index:to_index, feature] = ret_col
+    else:
+        ret = np.copy(data)
+        ret[from_index:to_index, feature] = ret[
+            from_index:to_index, feature
+        ] + random_walk(steps, x0)
+    return ret
+
+
+def random_walk(steps, x0):
+    """
+    Performs Weiner process given a number of steps and initial starting point.
+
+    Ref. :cite:t:`fields2019mitigating`
+
+    Args:
+        steps (int): number of steps, corresponds to size of output vector
+        x0 (int): initial starting value for output vector values
+    """
+    w = np.ones(steps) * x0
+    for i in range(1, steps):
+        yi = np.random.choice([1, -1])
+        w[i] = w[i - 1] + (yi / np.sqrt(steps))
+    return w
diff --git a/setup.cfg b/setup.cfg
index 4df610a4..296960b8 100644
--- a/setup.cfg
+++ b/setup.cfg
@@ -8,7 +8,7 @@ license = Apache 2.0
 description = This library implements algorithms for detecting data drift and concept drift for ML and statistics applications.
 long_description = file: readme_short.rst
 long_description_content_type = text/x-rst
-author = Leigh Nicholl, Thomas Schill, India Lindsay, Anmol Srivastava, Kodie P McNamara, Austin Downing
+author = Leigh Nicholl, Thomas Schill, India Lindsay, Anmol Srivastava, Kodie P McNamara, Shashank Jarmale
 author_email = tschill@mitre.org
 classifiers=
    Intended Audience :: Developers
@@ -27,7 +27,7 @@ install_requires =
    numpy >= 1.22.0 # This is due to the call to np.quantile in data_drift.kdq_tree, which uses an updated arg name. The code is otherwise compatible with python >= 3.5.
    pandas
    scipy
-   sklearn
+   scikit-learn
 
 [options.extras_require]
 test =


Test patch:
diff --git a/tests/menelaus/injection/__init__.py b/tests/menelaus/injection/__init__.py
new file mode 100644
index 00000000..e69de29b
diff --git a/tests/menelaus/injection/test_class_manipulation.py b/tests/menelaus/injection/test_class_manipulation.py
new file mode 100644
index 00000000..f8bb58be
--- /dev/null
+++ b/tests/menelaus/injection/test_class_manipulation.py
@@ -0,0 +1,52 @@
+import pytest
+import pandas as pd
+import numpy as np
+
+from menelaus.injection import class_join, class_swap
+
+
+def test_join_1():
+    ''' Check correct join in pd.DataFrame '''
+    data = pd.DataFrame({'a': [0,0,1,1], 'b': [5,5,5,5]})
+    expected_data = pd.DataFrame({'a': [0,2,2,1], 'b':[5,5,5,5]})
+    data = class_join(data, 'a', 0, 1, 2, 1, 3)
+    assert data.equals(expected_data)
+
+def test_join_2():
+    ''' Check correct join in numpy.ndarray '''
+    data = np.array([[0,5], [0,5], [1,5], [1,5]])
+    expected_data = np.array([[0,5], [2,5], [2,5], [1,5]])
+    data = class_join(data, 0, 0, 1, 2, 1, 3)
+    assert np.array_equal(data, expected_data)
+
+def test_join_3():
+    ''' Check ValueError if data neither pandas.DataFrame nor np.ndarray '''
+    data = [[0,0], [1,1], [0,2], [1,3]]
+    with pytest.raises(ValueError):
+        class_join(
+            data, target_col=0, class_1=0, class_2=1,
+            new_class=2, from_index=1, to_index=3
+        )
+
+def test_swap_1():
+    ''' Check correct swap in pd.DataFrame '''
+    data = pd.DataFrame({'a': [0,0,1,1], 'b': [5,5,6,6]})
+    expected_data = pd.DataFrame({'a': [1,1,0,0], 'b':[5,5,6,6]})
+    data = class_swap(data, 'a', 0, 1, 0, 4)
+    assert data.equals(expected_data)
+
+def test_swap_2():
+    ''' Check correct swap in numpy.ndarray'''
+    data = np.array([[0,5], [0,5], [1,6], [1,6]])
+    expected_data = np.array([[1,5], [1,5], [0,6], [0,6]])
+    data = class_swap(data, 0, 0, 1, 0, 4)
+    assert np.array_equal(data, expected_data)
+
+def test_swap_3():
+    ''' Check ValueError if data neither pandas.DataFrame nor np.ndarray '''
+    data = [[0,0], [1,1], [0,2], [1,3]]
+    with pytest.raises(ValueError):
+        class_swap(
+            data, target_col=0, class_1=0, class_2=1, 
+            from_index=1, to_index=3
+        )
\ No newline at end of file
diff --git a/tests/menelaus/injection/test_feature_manipulation.py b/tests/menelaus/injection/test_feature_manipulation.py
new file mode 100644
index 00000000..f90a762d
--- /dev/null
+++ b/tests/menelaus/injection/test_feature_manipulation.py
@@ -0,0 +1,49 @@
+import pytest
+import pandas as pd
+import numpy as np
+
+from menelaus.injection import feature_swap, feature_hide_and_sample
+
+
+def test_feature_swap_1():
+    ''' Check correct swap in pandas.DataFrame '''
+    data = pd.DataFrame({'a':[0,1,2,3,4], 'b':[0,8,9,3,4]})
+    expected_data = pd.DataFrame({'a':[0,8,9,3,4], 'b':[0,1,2,3,4]})
+    data = feature_swap(data, 'a', 'b', 1, 3)
+    assert data.equals(expected_data)
+
+def test_feature_swap_2():
+    ''' Check correct swap in numpy.ndarray '''
+    data = np.array([[0,0], [1,8], [2,9], [3,3], [4,4]])
+    expected_data = np.array([[0,0], [8,1], [9,2], [3,3], [4,4]])
+    data = feature_swap(data, 0, 1, 1, 3)
+    assert np.array_equal(data, expected_data)
+
+def test_feature_swap_3():
+    ''' Check function fails when data is neither numpy nor pandas '''
+    data = [1,2,3,4]
+    with pytest.raises(ValueError):
+        feature_swap(data, 'col1', 'col2', 0, 2)
+
+def test_feature_hide_1():
+    ''' Check correct operation in pandas.DataFrame '''
+    data = pd.DataFrame({'a': [0, 0, 0, 1, 1, 1], 'b': [0, 1, 2, 3, 4, 5]})
+    expected_data = pd.DataFrame({'b': [2,1,5,3]}) 
+    # expect result, since random state is default 0 in function call
+    data = feature_hide_and_sample(data, 'a', 2)
+    assert data.equals(expected_data)
+
+def test_feature_hide_2():
+    ''' Check correct operation in numpy.ndarray '''
+    data = np.array([[0,0], [0,1], [0,2], [1,3], [1,4], [1,5]])
+    expected_data = np.array([2,1,5,3]) 
+    # expect result, since random state is default 0 in function call
+    data = feature_hide_and_sample(data, 0, 2)
+    data = data.reshape(-1)
+    assert np.array_equal(data, expected_data)
+
+def test_feature_hide_3():
+    ''' Check function fails when data is neither numpy nor pandas '''
+    data = [1,2,3,4]
+    with pytest.raises(ValueError):
+        feature_hide_and_sample(data, 'col', 2)
\ No newline at end of file
diff --git a/tests/menelaus/injection/test_noise.py b/tests/menelaus/injection/test_noise.py
new file mode 100644
index 00000000..8bca2a44
--- /dev/null
+++ b/tests/menelaus/injection/test_noise.py
@@ -0,0 +1,36 @@
+""" Module to test noise injection functions """
+
+import pytest
+import pandas as pd
+import numpy as np
+
+from menelaus.injection import brownian_noise, random_walk
+
+
+def test_brownian_noise_1():
+    ''' Check general behavior for pandas DataFrame'''
+    old_values = [0,1,2,3,4,5,6]
+    data = pd.DataFrame({'a': old_values})
+    data = brownian_noise(data, 'a', 0, 1, 5)
+    assert not data.equals(pd.Series(old_values))
+
+def test_brownian_noise_2():
+    ''' Check general behavior for numpy array '''
+    old_values = np.array([[0],[1],[2],[3],[4],[5],[6]])
+    data = np.copy(old_values)
+    data = brownian_noise(data, 0, 0, 1, 5)
+    assert not np.array_equal(data, old_values)
+
+def test_bronwian_noise_3():
+    ''' Check ValueError if data neither pandas DataFrame nor numpy array '''
+    data = [[1,2], [0,1]]
+    with pytest.raises(ValueError):
+        brownian_noise(data, 0, 0, 0, 2)
+
+def test_random_walk():
+    ''' Check random walk generally behaves within bounds '''
+    vec = random_walk(10, 0)
+    for i in range(1,10):
+        assert vec[i] <= vec[i-1] + 1/np.sqrt(10)
+        assert vec[i] >= vec[i-1] - 1/np.sqrt(10)
+