Repository: mitre/menelaus (language: python)

Issue:
Revisit generic setup for detectors
*tms: Updating this to reflect the current code. There are a couple of bits that could be made more generic. This ought to be split up into several issues, when we choose to tackle it.

1.  If `update` could always take the same input, then it'd be easier to swap out detector objects, instead of having to check "is this an ADWIN object?" or similar. Immediate use cases: comparing performance of two detectors; ensembling detectors.
      - e.g. `ADWIN.update` in the typical use case takes "whether the most recent classification was correct." `kdq_tree.update` takes, instead, an array of the new sample(s) containing each feature.
      - For further work with semi-supervised detectors, where there may or may not be a label with a new sample, we'll eventually have an algorithm that _requires_ more generic input to be acceptable.
      - The batch algorithms present an additional wrinkle: their default behaviors differ, in that e.g. HDDDM currently puts all of the new test data into the reference window (by default), when drift is not detected. Given that the drift decision is _correct_, this would mean an increasingly accurate empirical distribution for the reference data. kdq_tree, as currently implemented, maintains an unchanging reference window, because repartitioning an ever-larger sample at each step would be expensive. Doing so is _possible_, now, using `set_reference` and maintaining the data outside the object, though.
2. Validation of input. We probably want decisions on the prior point before troubling with this.
      - For streaming detectors, `update` should check whether more than one sample (/row) has been passed. `KdqTree.set_reference` currently does something *like* this to stop the user from inappropriately calling it when `KdqTree` is in streaming mode instead of batch.
      - We also should add checks that confirm the dimension (and/or column names) of the passed input matches prior updates.
      - We could likely set up `DriftDetector.validate` to be called by `DriftDetector.update`, so that it only needs implementation once. This assumes that each detector is calling `super().update` at the very beginning of their implementation, which should be correct in most cases.
4. Having a `set_reference` method for streaming detectors means we could potentially have faster code for processing many samples at once, especially for those which have an explicit wait period before doing the real calculations. This seems like a much lower value use case.


Discussion / hints:
I think the smallest change to enable all the behavior we might want is to change these two methods:
```
set_reference(X, y_true, y_pred)
update(X, y_true, y_pred)
```
- `y_true, y_pred` lets semi-supervised detectors accrue labels as they go (if they aren't supposed to consult some "Oracle function." In that case, it's not clear how it would be handled operationally anyway).
- `y_true, y_pred` lets any performance-based detectors monitor whichever error statistic is necessary (... for a binary classifier)
- `X` can be used to pass several columns for data drift detectors, or a single column for e.g. CUSUM, P-H, or ADWIN used off-label.

We can require validation in the abstract base class, so that each detector implements some logic constraining initial/subsequent input, even if that logic is "do nothing."

---

Considering the case of an ensemble detector, which passes whatever input onto the detectors, introduces the wrinkle of having a common interface for batch vs. streaming:
- Different treatments of batch detectors. E.g. if `set_reference` is called manually, it'd be most natural for that to be called for each detector in the ensemble.
   - If different behavior here is desired, access to the member detectors is sufficient, albeit inconvenient. I guess you could always monkeypatch it.
- Several streaming detectors build up a "batch" of incoming data against which to compare the reference window. This isn't universal, nor is the "burn-in" period for some detectors that can't alarm for a certain period after being reset.
   - I don't think adding functions to handle this across detectors would get us much except making the code harder to read.
   - The lazy implementation for this could involve a check against detector type, e.g.
```
for detector in ensemble:
   if detector.input_type == 'stream':
      for X, y_true, y_pred in input.iterrows(): #or whatever
         detector.update(X, y_true, y_pred)
   elif detector.input_type == 'batch':  
         detector.update(X, y_true, y_pred)
```
   - This would avoid adding any new code to try to bring these in sync. I don't *think* we're going to run into somebody trying to use streaming detectors to catch "when" from what would essentially be batch data, but I guess we'll see when we get there. I guess something like described in Dasu, with multiple parallel reference windows of a specific length vs. incoming data might call for something like a streaming-batch mix, but that specific context seems more easily accomplished by several parallel detectors with different init params.
In addition to updating the jupyter notebook examples, the example in the README will also need to be updated.

Fix patch:
diff --git a/README.md b/README.md
index 1a6da52a..943cde28 100644
--- a/README.md
+++ b/README.md
@@ -119,21 +119,32 @@ follows:
 ```python
 import pandas as pd
 from menelaus.concept_drift import ADWIN
+from menelaus.data_drift import KdqTreeStreaming
+
 df = pd.read_csv('example.csv')
+
+# use a detector that searches for concept drift
 detector = ADWIN()
 for i, row in df.iterrows():
-   detector.update(row['y_predicted'], row['y_true'])
+   detector.update(row['y_true'], row['y_predicted'], X=None)
+   if detector.drift_state is not None:
+      print("Drift has occurred!")
+
+# use a detector that searches for data drift
+detector = KdqTreeStreaming()
+feature_cols = ['a', 'b', 'c']
+for i in range(len(df)):
+   detector.update(X=df.loc[[i], feature_cols], y_true=None, y_pred=None)
    if detector.drift_state is not None:
       print("Drift has occurred!")
 ```
 
-For this example, because ADWIN is a concept drift detector, it requires
-both a predicted value (`y_predicted`) and a true value (`y_true`), at
-each update step. Note that this requirement is not true for the
-detectors in other modules. More detailed examples, including code for
-visualizating drift locations, may be found in the ``examples`` directory, as
-stand-alone python scripts. The examples along with output can also be viewed on
-the RTD website.
+As a concept drift detector, ADWIN requires both a true value (`y_true`) and a
+predicted value (`y_predicted`) at each update step. The data drift detector
+KdqTreeStreaming only requires the feature values at each step (`X`). More
+detailed examples, including code for visualizating drift locations, may be
+found in the ``examples`` directory, as stand-alone python scripts. The examples
+along with output can also be viewed on the RTD website.
 
 # Contributing
 Install the library using the `[dev]` option, as above.
diff --git a/docs/source/examples/change_detection/change_detection_examples.ipynb b/docs/source/examples/change_detection/change_detection_examples.ipynb
index 7d753db3..a8f532b0 100644
--- a/docs/source/examples/change_detection/change_detection_examples.ipynb
+++ b/docs/source/examples/change_detection/change_detection_examples.ipynb
@@ -77,7 +77,7 @@
     "# iterate through data; feed each sample to the detector, in turn\n",
     "for i in range(len(df)):\n",
     "    obs = df[\"var2\"][i]\n",
-    "    ph.update(next_obs=obs, obs_id=i)\n",
+    "    ph.update(X=obs)\n",
     "    status.loc[i] = [i, obs, ph.drift_state]"
    ]
   },
diff --git a/docs/source/examples/concept_drift/concept_drift_examples.ipynb b/docs/source/examples/concept_drift/concept_drift_examples.ipynb
index ecd9a927..379848f8 100644
--- a/docs/source/examples/concept_drift/concept_drift_examples.ipynb
+++ b/docs/source/examples/concept_drift/concept_drift_examples.ipynb
@@ -1,1159 +1,1158 @@
 {
- "cells": [
-  {
-   "cell_type": "markdown",
-   "metadata": {},
-   "source": [
-    "# Concept Drift Detector Examples"
-   ]
-  },
-  {
-   "cell_type": "markdown",
-   "metadata": {},
-   "source": [
-    "The examples in this notebook show how to set up, run, and produce output from detectors in the \n",
-    "concept_drift module. The parameters aren't necessarily tuned for best \n",
-    "performance for the input data, just notional.\n",
-    "\n",
-    "Circle is a synthetic data source, where drift occurs in both var1, var2, and the \n",
-    "conditional distributions P(y|var1) and P(y|var2). The drift occurs from index \n",
-    "1000 to 1250, and affects 66% of the sample.\n",
-    "\n",
-    "Rainfall is a real data source that concept drift has been injected into. This\n",
-    "set contains approximately 18,000 samples, and the data has been standardized.\n",
-    "Drift starts from index 12,000 and continues through the rest of the dataset.\n",
-    "In this example, we take the first 10,000 samples of the dataset for training\n",
-    "an initial classifier, and then use the remaining samples for testing.\n",
-    "\n",
-    "These detectors are generally to be applied to the true class and predicted class \n",
-    "from a particular model. ADWIN is an exception in that it could also be used to \n",
-    "monitor an arbitrary real-valued feature. So, each of the summary plots displays \n",
-    "the running accuracy of the classifier alongside the drift detector's output.\n",
-    "\n",
-    "They also track the indices of portions of the incoming data stream which are \n",
-    "more similar to each other -- i.e., data that seems to be part of the same \n",
-    "concept, which could be used to retrain a model.\n",
-    "\n",
-    "NOTE: The LinearFourRates example has a relatively long runtime, roughly 5 minutes."
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": null,
-   "metadata": {},
-   "outputs": [],
-   "source": [
-    "## Imports ##\n",
-    "\n",
-    "import pandas as pd\n",
-    "import numpy as np\n",
-    "import matplotlib.pyplot as plt\n",
-    "from sklearn.naive_bayes import GaussianNB\n",
-    "from sklearn.linear_model import SGDClassifier\n",
-    "from sklearn import svm\n",
-    "from sklearn.base import clone\n",
-    "from menelaus.concept_drift import LinearFourRates, ADWIN, DDM, EDDM, STEPD, MD3\n",
-    "from menelaus.datasets import fetch_circle_data\n",
-    "from menelaus.datasets import fetch_rainfall_data"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": null,
-   "metadata": {},
-   "outputs": [],
-   "source": [
-    "## Import Data ##\n",
-    "\n",
-    "# read in Circle dataset\n",
-    "df = fetch_circle_data()\n",
-    "drift_start, drift_end = 1000, 1250\n",
-    "training_size = 500\n",
-    "\n",
-    "# read in Rainfall dataset\n",
-    "rainfall_df = fetch_rainfall_data()\n",
-    "rainfall_drift_start, rainfall_drift_end = 12000, 18158\n",
-    "rainfall_training_size = 10000\n",
-    "rainfall_columns = [\"temperature\", \"dew_point\", \"sea_level_pressure\", \"visibility\", \"average_wind_speed\", \"max_sustained_wind_speed\", \"minimum_temperature\", \"maximum_temperature\", \"rain\"]\n",
-    "rainfall_features = [\"temperature\", \"dew_point\", \"sea_level_pressure\", \"visibility\", \"average_wind_speed\", \"max_sustained_wind_speed\", \"minimum_temperature\", \"maximum_temperature\"]\n",
-    "rainfall_df[rainfall_features] = rainfall_df[rainfall_features].astype(float)"
-   ]
-  },
-  {
-   "cell_type": "markdown",
-   "metadata": {},
-   "source": [
-    "## Linear Four Rates (LFR)"
-   ]
-  },
-  {
-   "cell_type": "markdown",
-   "metadata": {},
-   "source": [
-    "Linear Four Rates monitors the four cells of the confusion matrix (TPR, FPR, TNR, FNR) and alarms when one of these becomes different enough from earlier performance."
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": null,
-   "metadata": {},
-   "outputs": [],
-   "source": [
-    "## Setup ##\n",
-    "\n",
-    "# Set up classifier: train on first training_size rows\n",
-    "X_train = df.loc[0:training_size, [\"var1\", \"var2\"]]\n",
-    "y_train = df.loc[0:training_size, \"y\"]\n",
-    "clf = GaussianNB()\n",
-    "clf.fit(X_train, y_train)\n",
-    "\n",
-    "# Set up LFR detector to detect at significance of .001. 5000 Monte Carlo\n",
-    "# simulations will be run every 10 samples to detect drift.\n",
-    "lfr = LinearFourRates(\n",
-    "    time_decay_factor=0.6,\n",
-    "    warning_level=0.01,\n",
-    "    detect_level=0.001,\n",
-    "    num_mc=5000,\n",
-    "    burn_in=10,\n",
-    "    subsample=10,\n",
-    ")\n",
-    "\n",
-    "# Set up DF to store results.\n",
-    "status = pd.DataFrame(columns=[\"index\", \"y\", \"y_pred\", \"drift_detected\", \"accuracy\"])\n",
-    "correct = 0"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": null,
-   "metadata": {},
-   "outputs": [],
-   "source": [
-    "## Run LFR ##\n",
-    "\n",
-    "np.random.seed(123)  # set seed for this example\n",
-    "\n",
-    "# Run LFR and retrain.\n",
-    "\n",
-    "rec_list = []\n",
-    "n = 1\n",
-    "for i in range(training_size, len(df)):\n",
-    "    X_test = df.loc[[i], [\"var1\", \"var2\"]]\n",
-    "    y_pred = int(clf.predict(X_test))\n",
-    "    y_true = int(df.loc[[i], \"y\"])\n",
-    "\n",
-    "    # increment accuracy\n",
-    "    if y_pred == y_true:\n",
-    "        correct += 1\n",
-    "    accuracy = correct / n\n",
-    "\n",
-    "    lfr.update(y_pred, y_true)\n",
-    "    status.loc[i] = [i, y_true, y_pred, lfr.drift_state, accuracy]\n",
-    "\n",
-    "    # If drift is detected, examine the retraining recommendations and retrain.\n",
-    "    if lfr.drift_state == \"drift\":\n",
-    "\n",
-    "        retrain_start = lfr.retraining_recs[0] + training_size\n",
-    "        retrain_end = lfr.retraining_recs[1] + training_size\n",
-    "        if (\n",
-    "            retrain_start == retrain_end\n",
-    "        ):  # minimum retraining window in case of sudden drift\n",
-    "            retrain_start = max(0, retrain_start - 300)\n",
-    "        rec_list.append([retrain_start, retrain_end])\n",
-    "\n",
-    "        # If retraining is not desired, omit the next four lines.\n",
-    "        X_train = df.loc[retrain_start:retrain_end, [\"var1\", \"var2\"]]\n",
-    "        y_train = df.loc[retrain_start:retrain_end, \"y\"]\n",
-    "        clf = GaussianNB()\n",
-    "        clf.fit(X_train, y_train)\n",
-    "\n",
-    "    n += 1"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": null,
-   "metadata": {},
-   "outputs": [],
-   "source": [
-    "## Plotting ##\n",
-    "\n",
-    "plt.figure(figsize=(20, 5))\n",
-    "plt.scatter(\"index\", \"accuracy\", data=status, label=\"Accuracy\")\n",
-    "plt.grid(False, axis=\"x\")\n",
-    "plt.xticks(fontsize=16)\n",
-    "plt.yticks(fontsize=16)\n",
-    "plt.title(\"LFR Results: Accuracy\", fontsize=22)\n",
-    "plt.ylabel(\"Value\", fontsize=18)\n",
-    "plt.xlabel(\"Index\", fontsize=18)\n",
-    "ylims = [-0.05, 1.1]\n",
-    "plt.ylim(ylims)\n",
-    "\n",
-    "plt.axvspan(1000, 1250, alpha=0.5, label=\"Drift Induction Window\")\n",
-    "\n",
-    "# Draw red lines that indicate where drift was detected\n",
-    "plt.vlines(\n",
-    "    x=status.loc[status[\"drift_detected\"] == \"drift\"][\"index\"],\n",
-    "    ymin=ylims[0],\n",
-    "    ymax=ylims[1],\n",
-    "    label=\"Drift Detected\",\n",
-    "    color=\"red\",\n",
-    ")\n",
-    "\n",
-    "# Draw orange lines that indicate where warnings of drift were provided\n",
-    "plt.vlines(\n",
-    "    x=status.loc[status[\"drift_detected\"] == \"warning\"][\"index\"],\n",
-    "    ymin=ylims[0],\n",
-    "    ymax=ylims[1],\n",
-    "    label=\"Warning\",\n",
-    "    color=\"orange\",\n",
-    "    alpha=0.3,\n",
-    ")\n",
-    "\n",
-    "# Create a list of lines that indicate the retraining windows.\n",
-    "# Space them evenly, vertically.\n",
-    "rec_list = pd.DataFrame(rec_list)\n",
-    "rec_list[\"y_val\"] = np.linspace(\n",
-    "    start=0.05 * (ylims[1] - ylims[0]) + ylims[0],\n",
-    "    stop=0.2 * ylims[1],\n",
-    "    num=len(rec_list),\n",
-    ")\n",
-    "\n",
-    "# Draw green lines that indicate where retraining occurred\n",
-    "plt.hlines(\n",
-    "    y=rec_list[\"y_val\"],\n",
-    "    xmin=rec_list[0],\n",
-    "    xmax=rec_list[1],\n",
-    "    color=\"green\",\n",
-    "    label=\"Retraining Windows\",\n",
-    ")\n",
-    "\n",
-    "plt.legend()"
-   ]
-  },
-  {
-   "cell_type": "markdown",
-   "metadata": {},
-   "source": [
-    "\n",
-    "One of the four rates immediately passes outside its threshold when drift is\n",
-    "induced. The same occurs shortly after leaving the drift region. The\n",
-    "recommended retraining data includes most of the drift induction window and\n",
-    "the data after regime change.\n",
-    "\n",
-    "The classifier's accuracy decreases again later, which causes the detector to\n",
-    "enter a \"warning\" state. Note that the retraining recommendations *begin* with\n",
-    "the index corresponding to the warning state, and end where drift is detected.\n"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": null,
-   "metadata": {},
-   "outputs": [],
-   "source": [
-    "plt.show()"
-   ]
-  },
-  {
-   "cell_type": "markdown",
-   "metadata": {},
-   "source": [
-    "## ADaptive WINdowing (ADWIN)"
-   ]
-  },
-  {
-   "cell_type": "markdown",
-   "metadata": {},
-   "source": [
-    "ADWIN can be used to monitor the average of a given real-valued feature. In this case, we use it to monitor the accuracy of a classifier. ADWIN maintains a window of the data stream, which grows to the right as new elements are received. When the mean of the feature in one of the subwindows is different enough, ADWIN drops older elements in its window until this ceases to be the case."
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": null,
-   "metadata": {},
-   "outputs": [],
-   "source": [
-    "## Setup ##\n",
-    "\n",
-    "# Set up classifier: train on first training_size rows\n",
-    "X_train = df.loc[0:training_size, [\"var1\", \"var2\"]]\n",
-    "y_train = df.loc[0:training_size, \"y\"]\n",
-    "\n",
-    "np.random.seed(123)\n",
-    "clf = GaussianNB()\n",
-    "clf.fit(X_train, y_train)\n",
-    "\n",
-    "adwin = ADWIN()\n",
-    "\n",
-    "# Set up DF to record results.\n",
-    "status = pd.DataFrame(\n",
-    "    columns=[\"index\", \"results\", \"accuracy\", \"adwin mean\", \"drift_detected\"]\n",
-    ")\n",
-    "correct = 0\n",
-    "rec_list = []"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": null,
-   "metadata": {},
-   "outputs": [],
-   "source": [
-    "# run ADWIN\n",
-    "n = 1\n",
-    "for i in range(training_size, len(df)):\n",
-    "\n",
-    "    X_test = df.loc[[i], [\"var1\", \"var2\"]]\n",
-    "    y_pred = int(clf.predict(X_test))\n",
-    "    y_true = int(df.loc[[i], \"y\"])\n",
-    "\n",
-    "    # increment accuracy\n",
-    "    if y_pred == y_true:\n",
-    "        correct += 1\n",
-    "    accuracy = correct / n\n",
-    "\n",
-    "    adwin.update(int(y_true == y_pred))\n",
-    "    status.loc[i] = [\n",
-    "        i,\n",
-    "        int(y_true == y_pred),\n",
-    "        accuracy,\n",
-    "        adwin.mean(),\n",
-    "        adwin.drift_state,\n",
-    "    ]\n",
-    "\n",
-    "    # If drift is detected, examine the window and retrain.\n",
-    "    if adwin.drift_state == \"drift\":\n",
-    "        retrain_start = adwin.retraining_recs[0] + training_size\n",
-    "        retrain_end = adwin.retraining_recs[1] + training_size\n",
-    "        rec_list.append([retrain_start, retrain_end])\n",
-    "\n",
-    "        # The retraining recommendations produced here correspond to the samples\n",
-    "        # which belong to ADWIN's new, smaller window, after drift is detected.\n",
-    "        # If retraining is not desired, omit the next four lines.\n",
-    "        X_train = df.loc[retrain_start:retrain_end, [\"var1\", \"var2\"]]\n",
-    "        y_train = df.loc[retrain_start:retrain_end, \"y\"]\n",
-    "        clf = GaussianNB()\n",
-    "        clf.fit(X_train, y_train)\n",
-    "\n",
-    "    n += 1"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": null,
-   "metadata": {},
-   "outputs": [],
-   "source": [
-    "## Plotting ##\n",
-    "\n",
-    "plt.figure(figsize=(20, 6))\n",
-    "plt.scatter(\"index\", \"accuracy\", data=status, label=\"Accuracy\")\n",
-    "plt.grid(False, axis=\"x\")\n",
-    "plt.xticks(fontsize=16)\n",
-    "plt.yticks(fontsize=16)\n",
-    "plt.title(\"ADWIN Results: Accuracy\", fontsize=22)\n",
-    "plt.ylabel(\"Value\", fontsize=18)\n",
-    "plt.xlabel(\"Index\", fontsize=18)\n",
-    "ylims = [0, 1.1]\n",
-    "plt.ylim(ylims)\n",
-    "\n",
-    "plt.axvspan(1000, 1250, alpha=0.5, label=\"Drift Induction Window\")\n",
-    "\n",
-    "# Draw red lines that indicate where drift was detected\n",
-    "plt.vlines(\n",
-    "    x=status.loc[status[\"drift_detected\"] == \"drift\"][\"index\"],\n",
-    "    ymin=ylims[0],\n",
-    "    ymax=ylims[1],\n",
-    "    label=\"Drift Detected\",\n",
-    "    color=\"red\",\n",
-    ")\n",
-    "\n",
-    "# Create a list of lines that indicate the retraining windows.\n",
-    "# Space them evenly, vertically.\n",
-    "rec_list = pd.DataFrame(rec_list)\n",
-    "rec_list[\"y_val\"] = np.linspace(\n",
-    "    start=0.05 * (ylims[1] - ylims[0]) + ylims[0],\n",
-    "    stop=0.2 * ylims[1],\n",
-    "    num=len(rec_list),\n",
-    ")\n",
-    "\n",
-    "# Draw green lines that indicate where retraining occurred\n",
-    "plt.hlines(\n",
-    "    y=rec_list[\"y_val\"],\n",
-    "    xmin=rec_list[0],\n",
-    "    xmax=rec_list[1],\n",
-    "    color=\"green\",\n",
-    "    label=\"Retraining Windows\",\n",
-    ")\n",
-    "\n",
-    "plt.legend(loc='lower right')"
-   ]
-  },
-  {
-   "cell_type": "markdown",
-   "metadata": {},
-   "source": [
-    "\n",
-    "After drift is induced, the accuracy decreases enough for ADWIN to shrink its\n",
-    "window and alarm;  subsequent windows also include data from the old regime,\n",
-    "so drift continues to be detected until the window shrinks enough to be\n",
-    "comprised mostly by the new regime.\n"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": null,
-   "metadata": {},
-   "outputs": [],
-   "source": [
-    "plt.show()\n",
-    "# plt.savefig(\"example_ADWIN.png\")"
-   ]
-  },
-  {
-   "cell_type": "markdown",
-   "metadata": {},
-   "source": [
-    "## Drift Detection Method (DDM)"
-   ]
-  },
-  {
-   "cell_type": "markdown",
-   "metadata": {},
-   "source": [
-    "DDM can enter either a \"drift\" or \"warning\" state, depending on how close a classifier's error rate has approached to those respective thresholds, defined by the warning_scale and drift_scale parameters."
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": null,
-   "metadata": {},
-   "outputs": [],
-   "source": [
-    "## Setup ##\n",
-    "\n",
-    "np.random.seed(123)\n",
-    "# setup classifier: train on first training_size rows\n",
-    "X_train = df.loc[0:training_size, [\"var1\", \"var2\"]]\n",
-    "y_train = df.loc[0:training_size, \"y\"]\n",
-    "clf = GaussianNB()\n",
-    "clf.fit(X_train, y_train)"
-   ]
-  },
-  {
-   "cell_type": "markdown",
-   "metadata": {},
-   "source": [
-    "\n",
-    "These parameter values are chosen somewhat arbitrarily.\n",
-    "At least 100 samples must be seen before DDM tests for drift (the n_threshold\n",
-    "parameter); the other two define the warning and drift regions. The minimum\n",
-    "error rate (and its standard deviation) are found during a stable regime; the\n",
-    "warning_scale and drift_scale roughly correspond to how many standard standard\n",
-    "deviations away the current estimate must be in order for the detector to\n",
-    "alarm.\n"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": null,
-   "metadata": {},
-   "outputs": [],
-   "source": [
-    "ddm = DDM(n_threshold=100, warning_scale=7, drift_scale=10)\n",
-    "\n",
-    "# setup DF to store results\n",
-    "status = pd.DataFrame(columns=[\"index\", \"y\", \"y_pred\", \"drift_detected\", \"accuracy\"])\n",
-    "correct = 0\n",
-    "rec_list = []"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": null,
-   "metadata": {},
-   "outputs": [],
-   "source": [
-    "# run DDM and retrain\n",
-    "n = 1\n",
-    "for i in range(training_size, len(df)):\n",
-    "\n",
-    "    X_test = df.loc[[i], [\"var1\", \"var2\"]]\n",
-    "    y_pred = int(clf.predict(X_test))\n",
-    "    y_true = int(df.loc[[i], \"y\"])\n",
-    "\n",
-    "    # increment accuracy\n",
-    "    if y_pred == y_true:\n",
-    "        correct += 1\n",
-    "    accuracy = correct / n\n",
-    "\n",
-    "    ddm.update(y_pred, y_true)\n",
-    "    status.loc[i] = [i, y_true, y_pred, ddm.drift_state, accuracy]\n",
-    "\n",
-    "    # If drift is detected, examine the window and retrain.\n",
-    "    if ddm.drift_state == \"drift\":\n",
-    "        retrain_start = ddm.retraining_recs[0] + training_size\n",
-    "        retrain_end = ddm.retraining_recs[1] + training_size\n",
-    "        if (\n",
-    "            retrain_start == retrain_end\n",
-    "        ):  # minimum retraining window in case of sudden drift\n",
-    "            retrain_start = max(0, retrain_start - 300)\n",
-    "        rec_list.append([retrain_start, retrain_end])\n",
-    "\n",
-    "        # If retraining is not desired, omit the next four lines.\n",
-    "        X_train = df.loc[retrain_start:retrain_end, [\"var1\", \"var2\"]]\n",
-    "        y_train = df.loc[retrain_start:retrain_end, \"y\"]\n",
-    "        clf = GaussianNB()\n",
-    "        clf.fit(X_train, y_train)\n",
-    "\n",
-    "    n += 1"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": null,
-   "metadata": {},
-   "outputs": [],
-   "source": [
-    "## Plotting ##\n",
-    "\n",
-    "plt.figure(figsize=(20, 6))\n",
-    "plt.scatter(\"index\", \"accuracy\", data=status, label=\"Accuracy\")\n",
-    "plt.grid(False, axis=\"x\")\n",
-    "plt.xticks(fontsize=16)\n",
-    "plt.yticks(fontsize=16)\n",
-    "plt.title(\"DDM Results: Accuracy\", fontsize=22)\n",
-    "plt.ylabel(\"Value\", fontsize=18)\n",
-    "plt.xlabel(\"Index\", fontsize=18)\n",
-    "ylims = [-0.05, 1.1]\n",
-    "plt.ylim(ylims)\n",
-    "\n",
-    "plt.axvspan(1000, 1250, alpha=0.5, label=\"Drift Induction Window\")\n",
-    "\n",
-    "# Draw red lines that indicate where drift was detected\n",
-    "plt.vlines(\n",
-    "    x=status.loc[status[\"drift_detected\"] == \"drift\"][\"index\"],\n",
-    "    ymin=ylims[0],\n",
-    "    ymax=ylims[1],\n",
-    "    label=\"Drift Detected\",\n",
-    "    color=\"red\",\n",
-    "    linewidth=3,\n",
-    ")\n",
-    "\n",
-    "# Draw orange lines that indicate where warnings of drift were provided\n",
-    "plt.vlines(\n",
-    "    x=status.loc[status[\"drift_detected\"] == \"warning\"][\"index\"],\n",
-    "    ymin=ylims[0],\n",
-    "    ymax=ylims[1],\n",
-    "    label=\"Warning\",\n",
-    "    color=\"orange\",\n",
-    "    alpha=0.3,\n",
-    ")\n",
-    "\n",
-    "# Create a list of lines that indicate the retraining windows.\n",
-    "# Space them evenly, vertically.\n",
-    "rec_list = pd.DataFrame(rec_list)\n",
-    "rec_list[\"y_val\"] = np.linspace(\n",
-    "    start=0.05 * (ylims[1] - ylims[0]) + ylims[0],\n",
-    "    stop=0.2 * ylims[1],\n",
-    "    num=len(rec_list),\n",
-    ")\n",
-    "\n",
-    "# Draw green lines that indicate where retraining occurred\n",
-    "plt.hlines(\n",
-    "    y=rec_list[\"y_val\"],\n",
-    "    xmin=rec_list[0],\n",
-    "    xmax=rec_list[1],\n",
-    "    color=\"green\",\n",
-    "    label=\"Retraining Windows\",\n",
-    ")\n",
-    "\n",
-    "plt.legend()"
-   ]
-  },
-  {
-   "cell_type": "markdown",
-   "metadata": {},
-   "source": [
-    "\n",
-    "DDM initially alarms during the drift induction window, triggering retraining.\n",
-    "The subsequent dip in accuracy is large enough to put the detector in the\n",
-    "\"warning\" state, but not large enough for \"drift\" to be identified.\n",
-    "\n"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": null,
-   "metadata": {},
-   "outputs": [],
-   "source": [
-    "plt.show()\n",
-    "# plt.savefig(\"example_DDM.png\")"
-   ]
-  },
-  {
-   "cell_type": "markdown",
-   "metadata": {},
-   "source": [
-    "## Early Drift Detection Method (EDDM)"
-   ]
-  },
-  {
-   "cell_type": "markdown",
-   "metadata": {},
-   "source": [
-    "EDDM monitors the distance between two errors of a classifier - i.e., the number of samples between errors - rather than monitoring the error rate itself. Similar to DDM, it uses separate thresholds for \"warning\" and \"drift.\""
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": null,
-   "metadata": {},
-   "outputs": [],
-   "source": [
-    "## Setup ##\n",
-    "\n",
-    "np.random.seed(123)\n",
-    "# setup classifier: train on first 500 rows\n",
-    "X_train = df.loc[0:training_size, [\"var1\", \"var2\"]]\n",
-    "y_train = df.loc[0:training_size, \"y\"]\n",
-    "clf = GaussianNB()\n",
-    "clf.fit(X_train, y_train)"
-   ]
-  },
-  {
-   "cell_type": "markdown",
-   "metadata": {},
-   "source": [
-    "- n_threshold specifies the number of new samples which must be observed before\n",
-    "tests for drift are run.\n",
-    "\n",
-    "- The warning_thresh and drift_thresh values roughly correspond to the ratio of\n",
-    "the 95th percentile for the current distance distribution vs. the 95th percentile\n",
-    "for the \"best\" distance distribution observed so far. So, lower values correspond to less conservative monitoring - the current\n",
-    "distance between errors is allowed to be a smaller fraction of the \"best\"\n",
-    "distance.\n"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": null,
-   "metadata": {},
-   "outputs": [],
-   "source": [
-    "eddm = EDDM(n_threshold=30, warning_thresh=0.7, drift_thresh=0.5)\n",
-    "\n",
-    "# setup DF to store results\n",
-    "status = pd.DataFrame(columns=[\"index\", \"y\", \"y_pred\", \"drift_detected\", \"accuracy\"])\n",
-    "correct = 0\n",
-    "rec_list = []"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": null,
-   "metadata": {},
-   "outputs": [],
-   "source": [
-    "# run EDDM and retrain\n",
-    "n = 1\n",
-    "for i in range(training_size, len(df)):\n",
-    "\n",
-    "    X_test = df.loc[[i], [\"var1\", \"var2\"]]\n",
-    "    y_pred = int(clf.predict(X_test))\n",
-    "    y_true = int(df.loc[[i], \"y\"])\n",
-    "\n",
-    "    # increment accuracy\n",
-    "    if y_pred == y_true:\n",
-    "        correct += 1\n",
-    "    accuracy = correct / n\n",
-    "\n",
-    "    eddm.update(y_pred, y_true)\n",
-    "    status.loc[i] = [i, y_true, y_pred, eddm.drift_state, accuracy]\n",
-    "\n",
-    "    # If drift is detected, examine the window and retrain.\n",
-    "    if eddm.drift_state == \"drift\":\n",
-    "        retrain_start = eddm.retraining_recs[0] + training_size\n",
-    "        retrain_end = eddm.retraining_recs[1] + training_size\n",
-    "        if (\n",
-    "            retrain_start == retrain_end\n",
-    "        ):  # minimum retraining window in case of sudden drift\n",
-    "            retrain_start = max(0, retrain_start - 300)\n",
-    "        rec_list.append([retrain_start, retrain_end])\n",
-    "\n",
-    "        # If retraining is not desired, omit the next four lines.\n",
-    "        X_train = df.loc[retrain_start:retrain_end, [\"var1\", \"var2\"]]\n",
-    "        y_train = df.loc[retrain_start:retrain_end, \"y\"]\n",
-    "        clf = GaussianNB()\n",
-    "        clf.fit(X_train, y_train)\n",
-    "\n",
-    "    n += 1"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": null,
-   "metadata": {},
-   "outputs": [],
-   "source": [
-    "## Plotting ##\n",
-    "\n",
-    "plt.figure(figsize=(20, 6))\n",
-    "plt.scatter(\"index\", \"accuracy\", data=status, label=\"Accuracy\")\n",
-    "plt.grid(False, axis=\"x\")\n",
-    "plt.xticks(fontsize=16)\n",
-    "plt.yticks(fontsize=16)\n",
-    "plt.title(\"EDDM Results: Accuracy\", fontsize=22)\n",
-    "plt.ylabel(\"Value\", fontsize=18)\n",
-    "plt.xlabel(\"Index\", fontsize=18)\n",
-    "ylims = [-0.05, 1.1]\n",
-    "plt.ylim(ylims)\n",
-    "\n",
-    "plt.axvspan(1000, 1250, alpha=0.5, label=\"Drift Induction Window\")\n",
-    "\n",
-    "# Draw orange lines that indicate where warnings of drift were provided\n",
-    "plt.vlines(\n",
-    "    x=status.loc[status[\"drift_detected\"] == \"warning\"][\"index\"],\n",
-    "    ymin=ylims[0],\n",
-    "    ymax=ylims[1],\n",
-    "    label=\"Warning\",\n",
-    "    color=\"orange\",\n",
-    "    alpha=0.3,\n",
-    ")\n",
-    "\n",
-    "# Draw red lines that indicate where drift was detected\n",
-    "plt.vlines(\n",
-    "    x=status.loc[status[\"drift_detected\"] == \"drift\"][\"index\"],\n",
-    "    ymin=ylims[0],\n",
-    "    ymax=ylims[1],\n",
-    "    label=\"Drift Detected\",\n",
-    "    color=\"red\",\n",
-    ")\n",
-    "\n",
-    "# Create a list of lines that indicate the retraining windows.\n",
-    "# Space them evenly, vertically.\n",
-    "rec_list = pd.DataFrame(rec_list)\n",
-    "rec_list[\"y_val\"] = np.linspace(\n",
-    "    start=0.05 * (ylims[1] - ylims[0]) + ylims[0],\n",
-    "    stop=0.2 * ylims[1],\n",
-    "    num=len(rec_list),\n",
-    ")\n",
-    "\n",
-    "# Draw green lines that indicate where retraining occurred\n",
-    "plt.hlines(\n",
-    "    y=rec_list[\"y_val\"],\n",
-    "    xmin=rec_list[0],\n",
-    "    xmax=rec_list[1],\n",
-    "    color=\"green\",\n",
-    "    label=\"Retraining Windows\",\n",
-    "    alpha=0.3,\n",
-    ")\n",
-    "\n",
-    "plt.legend()"
-   ]
-  },
-  {
-   "cell_type": "markdown",
-   "metadata": {},
-   "source": [
-    "EDDM enters a drift state shortly after the drift induction window, triggering\n",
-    "retraining. The later increase in the error rate causes the detector to enter\n",
-    "the warning state, but is not large enough to be identified as drift with this\n",
-    "threshold setting.\n"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": null,
-   "metadata": {},
-   "outputs": [],
-   "source": [
-    "plt.show()\n",
-    "# plt.savefig(\"example_EDDM.png\")"
-   ]
-  },
-  {
-   "cell_type": "markdown",
-   "metadata": {},
-   "source": [
-    "## Statistical Test of Equal Proportions to Detect Concept Drift (STEPD)"
-   ]
-  },
-  {
-   "cell_type": "markdown",
-   "metadata": {},
-   "source": [
-    "STEPD is a detector specifically intended for online classifiers, where each new sample is used to update the parameters of the classifier. STEPD monitors the accuracy in two windows, \"recent\" and \"past,\" and compares those in order to detect drift in classifier accuracy."
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": null,
-   "metadata": {},
-   "outputs": [],
-   "source": [
-    "## Setup ##\n",
-    "\n",
-    "np.random.seed(123)\n",
-    "df_ex = df\n",
-    "train_ix = [0, training_size]\n",
-    "\n",
-    "\n",
-    "# For the purposes of this example, our online classifier is an SGDClassifier\n",
-    "# with a constant learning rate, since this is available in the pre-existing\n",
-    "# package dependency sklearn.\n",
-    "clf = SGDClassifier(learning_rate=\"constant\", shuffle=False, eta0=0.2)\n",
-    "clf.fit(\n",
-    "    df_ex.loc[train_ix[0] : train_ix[1], [\"var1\", \"var2\"]].values,\n",
-    "    df_ex.loc[train_ix[0] : train_ix[1], \"y\"].values,\n",
-    ")\n",
-    "\n",
-    "\n",
-    "stepd = STEPD(window_size=100)\n",
-    "\n",
-    "# setup DF to store results\n",
-    "status = pd.DataFrame(columns=[\"index\", \"y\", \"y_pred\", \"drift_detected\", \"accuracy\"])\n",
-    "correct = 0\n",
-    "rec_list = []"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": null,
-   "metadata": {},
-   "outputs": [],
-   "source": [
-    "# run STEPD and retrain\n",
-    "n = 1\n",
-    "for i, row in df_ex.iloc[training_size:].iterrows():\n",
-    "    y_pred = clf.predict(np.array(row[[\"var1\", \"var2\"]]).reshape(1, -1))\n",
-    "    y_true = row[\"y\"]\n",
-    "\n",
-    "    if y_pred == y_true:\n",
-    "        correct += 1\n",
-    "    accuracy = correct / n\n",
-    "\n",
-    "    stepd.update(y_pred, y_true)\n",
-    "    status.loc[i] = [i, y_true, y_pred, stepd.drift_state, accuracy]\n",
-    "    # train_ix[1] = train_ix[1] + 1\n",
-    "\n",
-    "    if stepd.drift_state == \"drift\":\n",
-    "        rec_list.append(stepd.retraining_recs)\n",
-    "        # retrain the classifier using STEPD's recommendations\n",
-    "        train_ix = stepd.retraining_recs\n",
-    "        train_ix[0] = train_ix[0] + training_size  # adjust for starting index\n",
-    "        train_ix[1] = train_ix[1] + training_size\n",
-    "        clf.fit(\n",
-    "            df_ex.loc[train_ix[0] : train_ix[1], [\"var1\", \"var2\"]].values,\n",
-    "            df_ex.loc[train_ix[0] : train_ix[1], \"y\"].values,\n",
-    "        )\n",
-    "\n",
-    "    else:\n",
-    "        # update the classifier with the newest sample\n",
-    "        clf.partial_fit(row[[\"var1\", \"var2\"]].values.reshape(1, -1), [row[\"y\"]])\n",
-    "\n",
-    "    n += 1"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": null,
-   "metadata": {},
-   "outputs": [],
-   "source": [
-    "## Plotting ##\n",
-    "\n",
-    "plt.figure(figsize=(20, 6))\n",
-    "plt.scatter(\"index\", \"accuracy\", data=status, label=\"Accuracy\")\n",
-    "plt.grid(False, axis=\"x\")\n",
-    "plt.xticks(fontsize=16)\n",
-    "plt.yticks(fontsize=16)\n",
-    "plt.title(\"STEPD Results: Accuracy\", fontsize=22)\n",
-    "plt.ylabel(\"Value\", fontsize=18)\n",
-    "plt.xlabel(\"Index\", fontsize=18)\n",
-    "ylims = [-0.05, 1.1]\n",
-    "plt.ylim(ylims)\n",
-    "\n",
-    "plt.axvspan(1000, 1250, alpha=0.5, label=\"Drift Induction Window\")\n",
-    "\n",
-    "# Draw orange lines that indicate where warnings of drift were provided\n",
-    "plt.vlines(\n",
-    "    x=status.loc[status[\"drift_detected\"] == \"warning\"][\"index\"],\n",
-    "    ymin=ylims[0],\n",
-    "    ymax=ylims[1],\n",
-    "    label=\"Warning\",\n",
-    "    color=\"orange\",\n",
-    "    alpha=0.3,\n",
-    ")\n",
-    "\n",
-    "# Draw red lines that indicate where drift was detected\n",
-    "plt.vlines(\n",
-    "    x=status.loc[status[\"drift_detected\"] == \"drift\"][\"index\"],\n",
-    "    ymin=ylims[0],\n",
-    "    ymax=ylims[1],\n",
-    "    label=\"Drift Detected\",\n",
-    "    color=\"red\",\n",
-    "    linewidth=3,\n",
-    ")\n",
-    "\n",
-    "# Create a list of lines that indicate the retraining windows.\n",
-    "# Space them evenly, vertically.\n",
-    "rec_list = pd.DataFrame(rec_list)\n",
-    "rec_list[\"y_val\"] = np.linspace(\n",
-    "    start=0.05 * (ylims[1] - ylims[0]) + ylims[0],\n",
-    "    stop=0.2 * ylims[1],\n",
-    "    num=len(rec_list),\n",
-    ")\n",
-    "\n",
-    "# Draw green lines that indicate where retraining occurred\n",
-    "plt.hlines(\n",
-    "    y=rec_list[\"y_val\"],\n",
-    "    xmin=rec_list[0],\n",
-    "    xmax=rec_list[1],\n",
-    "    color=\"green\",\n",
-    "    label=\"Retraining Windows\",\n",
-    ")\n",
-    "\n",
-    "plt.legend(loc='upper left')"
-   ]
-  },
-  {
-   "cell_type": "markdown",
-   "metadata": {},
-   "source": [
-    "STEPD identifies drift quite early in the drift induction window, triggering\n",
-    "retraining on a relatively small amount of data; after this, the online\n",
-    "classifier updates sufficiently that its accuracy is roughly flat over the\n",
-    "remaining data, albeit with a big enough change to trigger more warnings.\n"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": null,
-   "metadata": {},
-   "outputs": [],
-   "source": [
-    "plt.show()\n",
-    "# plt.savefig(\"example_STEPD.png\")"
-   ]
-  },
-  {
-   "cell_type": "markdown",
-   "metadata": {},
-   "source": [
-    "## Margin Density Drift Detection (MD3) Method"
-   ]
-  },
-  {
-   "cell_type": "markdown",
-   "metadata": {},
-   "source": [
-    "MD3 is a detector intended for semi-supervised and online classifier contexts. A cumulative margin density statistic (unsupervised) is tracked, representing the number of samples that fall into the uncertainty region, or margin, of the classifier. When margin density increases or decreases beyond a certain threshold, a drift warning is issued. When a warning is issued, a set of labeled samples is requested. If prediction accuracy by the model on these samples is lower than on the initial labeled reference dataset, drift is confirmed. If not, drift is ruled out."
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": null,
-   "metadata": {},
-   "outputs": [],
-   "source": [
-    "## Setup ##\n",
-    "\n",
-    "# Set up classifier: train on first training_size rows\n",
-    "training_data = rainfall_df.loc[0:rainfall_training_size, rainfall_columns]\n",
-    "X_train = rainfall_df.loc[0:rainfall_training_size, rainfall_features]\n",
-    "y_train = rainfall_df.loc[0:rainfall_training_size, \"rain\"]\n",
-    "\n",
-    "np.random.seed(123)\n",
-    "clf = svm.SVC(kernel='linear')\n",
-    "clf.fit(X_train, y_train.values.ravel())\n",
-    "retrain_clf = clone(clf)\n",
-    "retrain_clf.fit(X_train, y_train.values.ravel())\n",
-    "oracle_labels = 1000\n",
-    "\n",
-    "# Initialize detector\n",
-    "md3 = MD3(clf=clf, sensitivity=1.5, oracle_data_length_required=oracle_labels)\n",
-    "md3.set_reference(training_data, \"rain\")\n",
-    "\n",
-    "# Set up DF to record results.\n",
-    "status = pd.DataFrame(\n",
-    "    columns=[\"index\", \"y\", \"margin_density\", \"original_accuracy\", \"retrain_accuracy\", \"drift_detected\"]\n",
-    ")\n",
-    "correct_orig, correct_retrain = 0, 0\n",
-    "n = 1\n",
-    "rec_list = []\n",
-    "oracle_list = []"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": null,
-   "metadata": {},
-   "outputs": [],
-   "source": [
-    "# run MD3 and track results for both original model and retrained model\n",
-    "for i in range(rainfall_training_size, len(rainfall_df)):\n",
-    "\n",
-    "    X_test = rainfall_df.loc[[i], rainfall_features]\n",
-    "    y_pred_orig = int(clf.predict(X_test))\n",
-    "    y_pred_retrain = int(retrain_clf.predict(X_test))\n",
-    "    y_true = int(rainfall_df.loc[[i], \"rain\"])\n",
-    "    \n",
-    "    # increment accuracy\n",
-    "    if y_pred_orig == y_true:\n",
-    "        correct_orig += 1\n",
-    "    if y_pred_retrain == y_true:\n",
-    "        correct_retrain += 1\n",
-    "    accuracy_orig = correct_orig / n\n",
-    "    accuracy_retrain = correct_retrain / n\n",
-    "\n",
-    "    # call give_oracle_label if detector is currently waiting for oracle data\n",
-    "    if md3.waiting_for_oracle == True:\n",
-    "        oracle_label = rainfall_df.loc[[i], rainfall_columns]\n",
-    "        md3.give_oracle_label(oracle_label)\n",
-    "        if md3.waiting_for_oracle == False:\n",
-    "            retrain_clf.fit(md3.reference_batch_features, md3.reference_batch_target.values.ravel())\n",
-    "        status.loc[i] = [\n",
-    "            i,\n",
-    "            y_true,\n",
-    "            None,\n",
-    "            accuracy_orig,\n",
-    "            accuracy_retrain,\n",
-    "            md3.drift_state,\n",
-    "        ]\n",
-    "\n",
-    "    # call update otherwise\n",
-    "    else:\n",
-    "        md3.update(X_test)\n",
-    "        status.loc[i] = [\n",
-    "            i,\n",
-    "            y_true,\n",
-    "            md3.curr_margin_density,\n",
-    "            accuracy_orig,\n",
-    "            accuracy_retrain,\n",
-    "            md3.drift_state,\n",
-    "        ]\n",
-    "    \n",
-    "    # If there was a drift warning, track the window of the labeled\n",
-    "    # oracle data used\n",
-    "    if md3.drift_state == \"warning\":\n",
-    "        oracle_start = i + 1\n",
-    "        oracle_end = i + md3.oracle_data_length_required\n",
-    "        \n",
-    "        oracle_list.append([oracle_start, oracle_end])\n",
-    "\n",
-    "    n += 1"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": null,
-   "metadata": {},
-   "outputs": [],
-   "source": [
-    "## Plotting ##\n",
-    "\n",
-    "plt.figure(figsize=(20, 6))\n",
-    "plt.scatter(\"index\", \"margin_density\", data=status, label=\"Margin Density\")\n",
-    "plt.scatter(\"index\", \"original_accuracy\", data=status, label=\"Original Accuracy\", color=\"red\")\n",
-    "plt.scatter(\"index\", \"retrain_accuracy\", data=status, label=\"Retrain Accuracy\", color=\"green\")\n",
-    "plt.grid(False, axis=\"x\")\n",
-    "plt.xticks(fontsize=16)\n",
-    "plt.yticks(fontsize=16)\n",
-    "plt.title(\"MD3 Results: Margin Density and Accuracy\", fontsize=22)\n",
-    "plt.ylabel(\"Value\", fontsize=18)\n",
-    "plt.xlabel(\"Index\", fontsize=18)\n",
-    "ylims = [-0.05, 1.1]\n",
-    "plt.ylim(ylims)\n",
-    "\n",
-    "plt.axvspan(rainfall_drift_start, rainfall_drift_end, alpha=0.5, label=\"Drift Induction Window\")\n",
-    "\n",
-    "# Draw red lines that indicate where drift was detected\n",
-    "plt.vlines(\n",
-    "    x=status.loc[status[\"drift_detected\"] == \"drift\"][\"index\"],\n",
-    "    ymin=ylims[0],\n",
-    "    ymax=ylims[1],\n",
-    "    label=\"Drift Detected\",\n",
-    "    color=\"red\",\n",
-    ")\n",
-    "\n",
-    "# Draw orange lines that indicate where warnings of drift were provided\n",
-    "plt.vlines(\n",
-    "    x=status.loc[status[\"drift_detected\"] == \"warning\"][\"index\"],\n",
-    "    ymin=ylims[0],\n",
-    "    ymax=ylims[1],\n",
-    "    label=\"Warning\",\n",
-    "    color=\"orange\",\n",
-    "    alpha=0.3,\n",
-    ")\n",
-    "\n",
-    "# Create a list of lines that indicate the retraining windows.\n",
-    "# Space them evenly, vertically.\n",
-    "oracle_list = pd.DataFrame(oracle_list)\n",
-    "oracle_list[\"y_val\"] = np.linspace(\n",
-    "    start=0.05 * (ylims[1] - ylims[0]) + ylims[0],\n",
-    "    stop=0.2 * ylims[1],\n",
-    "    num=len(oracle_list),\n",
-    ")\n",
-    "\n",
-    "# Draw green lines that indicate where retraining occurred\n",
-    "plt.hlines(\n",
-    "    y=oracle_list[\"y_val\"],\n",
-    "    xmin=oracle_list[0],\n",
-    "    xmax=oracle_list[1],\n",
-    "    color=\"green\",\n",
-    "    label=\"Labeled Oracle Data\",\n",
-    ")\n",
-    "\n",
-    "plt.legend()"
-   ]
-  },
-  {
-   "cell_type": "markdown",
-   "metadata": {},
-   "source": [
-    "After drift is induced, the margin density decreases enough for MD3 to emit a warning. From there, the predictive accuracy of the classifier is tested, and this has already decreased sufficiently for the detector to alarm. Then, a new reference batch is set and the detector continues tracking the margin density statistic until the next warning. It can be seen from the plot that retraining at drift results in better accuracy moving forward compared with a model that is kept static."
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": null,
-   "metadata": {},
-   "outputs": [],
-   "source": [
-    "plt.show()\n",
-    "# plt.savefig(\"example_MD3.png\")"
-   ]
-  }
- ],
- "metadata": {
-  "kernelspec": {
-   "display_name": "Python 3.10.2 ('molten_env')",
-   "language": "python",
-   "name": "python3"
-  },
-  "language_info": {
-   "codemirror_mode": {
-    "name": "ipython",
-    "version": 3
-   },
-   "file_extension": ".py",
-   "mimetype": "text/x-python",
-   "name": "python",
-   "nbconvert_exporter": "python",
-   "pygments_lexer": "ipython3",
-   "version": "3.10.2"
-  },
-  "orig_nbformat": 4,
-  "vscode": {
-   "interpreter": {
-    "hash": "cebda30e5482c3c83cb9c7c7d8ef1c1d1f67dcf93284b9e88f69b68f560a7bf6"
-   }
-  }
- },
- "nbformat": 4,
- "nbformat_minor": 2
+    "cells": [
+        {
+            "cell_type": "markdown",
+            "metadata": {},
+            "source": [
+                "# Concept Drift Detector Examples"
+            ]
+        },
+        {
+            "cell_type": "markdown",
+            "metadata": {},
+            "source": [
+                "The examples in this notebook show how to set up, run, and produce output from detectors in the \n",
+                "concept_drift module. The parameters aren't necessarily tuned for best \n",
+                "performance for the input data, just notional.\n",
+                "\n",
+                "Circle is a synthetic data source, where drift occurs in both var1, var2, and the \n",
+                "conditional distributions P(y|var1) and P(y|var2). The drift occurs from index \n",
+                "1000 to 1250, and affects 66% of the sample.\n",
+                "\n",
+                "Rainfall is a real data source that concept drift has been injected into. This\n",
+                "set contains approximately 18,000 samples, and the data has been standardized.\n",
+                "Drift starts from index 12,000 and continues through the rest of the dataset.\n",
+                "In this example, we take the first 10,000 samples of the dataset for training\n",
+                "an initial classifier, and then use the remaining samples for testing.\n",
+                "\n",
+                "These detectors are generally to be applied to the true class and predicted class \n",
+                "from a particular model. So, each of the summary plots displays the running\n",
+                "accuracy of the classifier alongside the drift detector's output.\n",
+                "\n",
+                "They also track the indices of portions of the incoming data stream which are \n",
+                "more similar to each other -- i.e., data that seems to be part of the same \n",
+                "concept, which could be used to retrain a model.\n",
+                "\n",
+                "NOTE: The LinearFourRates example has a relatively long runtime, roughly 5 minutes."
+            ]
+        },
+        {
+            "cell_type": "code",
+            "execution_count": null,
+            "metadata": {},
+            "outputs": [],
+            "source": [
+                "## Imports ##\n",
+                "\n",
+                "import pandas as pd\n",
+                "import numpy as np\n",
+                "import matplotlib.pyplot as plt\n",
+                "from sklearn.naive_bayes import GaussianNB\n",
+                "from sklearn.linear_model import SGDClassifier\n",
+                "from sklearn import svm\n",
+                "from sklearn.base import clone\n",
+                "from menelaus.concept_drift import LinearFourRates, ADWIN, DDM, EDDM, STEPD, MD3\n",
+                "from menelaus.datasets import fetch_circle_data\n",
+                "from menelaus.datasets import fetch_rainfall_data"
+            ]
+        },
+        {
+            "cell_type": "code",
+            "execution_count": null,
+            "metadata": {},
+            "outputs": [],
+            "source": [
+                "## Import Data ##\n",
+                "\n",
+                "# read in Circle dataset\n",
+                "df = fetch_circle_data()\n",
+                "drift_start, drift_end = 1000, 1250\n",
+                "training_size = 500\n",
+                "\n",
+                "# read in Rainfall dataset\n",
+                "rainfall_df = fetch_rainfall_data()\n",
+                "rainfall_drift_start, rainfall_drift_end = 12000, 18158\n",
+                "rainfall_training_size = 10000\n",
+                "rainfall_columns = [\"temperature\", \"dew_point\", \"sea_level_pressure\", \"visibility\", \"average_wind_speed\", \"max_sustained_wind_speed\", \"minimum_temperature\", \"maximum_temperature\", \"rain\"]\n",
+                "rainfall_features = [\"temperature\", \"dew_point\", \"sea_level_pressure\", \"visibility\", \"average_wind_speed\", \"max_sustained_wind_speed\", \"minimum_temperature\", \"maximum_temperature\"]\n",
+                "rainfall_df[rainfall_features] = rainfall_df[rainfall_features].astype(float)"
+            ]
+        },
+        {
+            "cell_type": "markdown",
+            "metadata": {},
+            "source": [
+                "## Linear Four Rates (LFR)"
+            ]
+        },
+        {
+            "cell_type": "markdown",
+            "metadata": {},
+            "source": [
+                "Linear Four Rates monitors the four cells of the confusion matrix (TPR, FPR, TNR, FNR) and alarms when one of these becomes different enough from earlier performance."
+            ]
+        },
+        {
+            "cell_type": "code",
+            "execution_count": null,
+            "metadata": {},
+            "outputs": [],
+            "source": [
+                "## Setup ##\n",
+                "\n",
+                "# Set up classifier: train on first training_size rows\n",
+                "X_train = df.loc[0:training_size, [\"var1\", \"var2\"]]\n",
+                "y_train = df.loc[0:training_size, \"y\"]\n",
+                "clf = GaussianNB()\n",
+                "clf.fit(X_train, y_train)\n",
+                "\n",
+                "# Set up LFR detector to detect at significance of .001. 5000 Monte Carlo\n",
+                "# simulations will be run every 10 samples to detect drift.\n",
+                "lfr = LinearFourRates(\n",
+                "    time_decay_factor=0.6,\n",
+                "    warning_level=0.01,\n",
+                "    detect_level=0.001,\n",
+                "    num_mc=5000,\n",
+                "    burn_in=10,\n",
+                "    subsample=10,\n",
+                ")\n",
+                "\n",
+                "# Set up DF to store results.\n",
+                "status = pd.DataFrame(columns=[\"index\", \"y\", \"y_pred\", \"drift_detected\", \"accuracy\"])\n",
+                "correct = 0"
+            ]
+        },
+        {
+            "cell_type": "code",
+            "execution_count": null,
+            "metadata": {},
+            "outputs": [],
+            "source": [
+                "## Run LFR ##\n",
+                "\n",
+                "np.random.seed(123)  # set seed for this example\n",
+                "\n",
+                "# Run LFR and retrain.\n",
+                "\n",
+                "rec_list = []\n",
+                "n = 1\n",
+                "for i in range(training_size, len(df)):\n",
+                "    X_test = df.loc[[i], [\"var1\", \"var2\"]]\n",
+                "    y_pred = int(clf.predict(X_test))\n",
+                "    y_true = int(df.loc[[i], \"y\"])\n",
+                "\n",
+                "    # increment accuracy\n",
+                "    if y_pred == y_true:\n",
+                "        correct += 1\n",
+                "    accuracy = correct / n\n",
+                "\n",
+                "    lfr.update(y_true, y_pred)\n",
+                "    status.loc[i] = [i, y_true, y_pred, lfr.drift_state, accuracy]\n",
+                "\n",
+                "    # If drift is detected, examine the retraining recommendations and retrain.\n",
+                "    if lfr.drift_state == \"drift\":\n",
+                "\n",
+                "        retrain_start = lfr.retraining_recs[0] + training_size\n",
+                "        retrain_end = lfr.retraining_recs[1] + training_size\n",
+                "        if (\n",
+                "            retrain_start == retrain_end\n",
+                "        ):  # minimum retraining window in case of sudden drift\n",
+                "            retrain_start = max(0, retrain_start - 300)\n",
+                "        rec_list.append([retrain_start, retrain_end])\n",
+                "\n",
+                "        # If retraining is not desired, omit the next four lines.\n",
+                "        X_train = df.loc[retrain_start:retrain_end, [\"var1\", \"var2\"]]\n",
+                "        y_train = df.loc[retrain_start:retrain_end, \"y\"]\n",
+                "        clf = GaussianNB()\n",
+                "        clf.fit(X_train, y_train)\n",
+                "\n",
+                "    n += 1"
+            ]
+        },
+        {
+            "cell_type": "code",
+            "execution_count": null,
+            "metadata": {},
+            "outputs": [],
+            "source": [
+                "## Plotting ##\n",
+                "\n",
+                "plt.figure(figsize=(20, 5))\n",
+                "plt.scatter(\"index\", \"accuracy\", data=status, label=\"Accuracy\")\n",
+                "plt.grid(False, axis=\"x\")\n",
+                "plt.xticks(fontsize=16)\n",
+                "plt.yticks(fontsize=16)\n",
+                "plt.title(\"LFR Results: Accuracy\", fontsize=22)\n",
+                "plt.ylabel(\"Value\", fontsize=18)\n",
+                "plt.xlabel(\"Index\", fontsize=18)\n",
+                "ylims = [-0.05, 1.1]\n",
+                "plt.ylim(ylims)\n",
+                "\n",
+                "plt.axvspan(1000, 1250, alpha=0.5, label=\"Drift Induction Window\")\n",
+                "\n",
+                "# Draw red lines that indicate where drift was detected\n",
+                "plt.vlines(\n",
+                "    x=status.loc[status[\"drift_detected\"] == \"drift\"][\"index\"],\n",
+                "    ymin=ylims[0],\n",
+                "    ymax=ylims[1],\n",
+                "    label=\"Drift Detected\",\n",
+                "    color=\"red\",\n",
+                ")\n",
+                "\n",
+                "# Draw orange lines that indicate where warnings of drift were provided\n",
+                "plt.vlines(\n",
+                "    x=status.loc[status[\"drift_detected\"] == \"warning\"][\"index\"],\n",
+                "    ymin=ylims[0],\n",
+                "    ymax=ylims[1],\n",
+                "    label=\"Warning\",\n",
+                "    color=\"orange\",\n",
+                "    alpha=0.3,\n",
+                ")\n",
+                "\n",
+                "# Create a list of lines that indicate the retraining windows.\n",
+                "# Space them evenly, vertically.\n",
+                "rec_list = pd.DataFrame(rec_list)\n",
+                "rec_list[\"y_val\"] = np.linspace(\n",
+                "    start=0.05 * (ylims[1] - ylims[0]) + ylims[0],\n",
+                "    stop=0.2 * ylims[1],\n",
+                "    num=len(rec_list),\n",
+                ")\n",
+                "\n",
+                "# Draw green lines that indicate where retraining occurred\n",
+                "plt.hlines(\n",
+                "    y=rec_list[\"y_val\"],\n",
+                "    xmin=rec_list[0],\n",
+                "    xmax=rec_list[1],\n",
+                "    color=\"green\",\n",
+                "    label=\"Retraining Windows\",\n",
+                ")\n",
+                "\n",
+                "plt.legend()"
+            ]
+        },
+        {
+            "cell_type": "markdown",
+            "metadata": {},
+            "source": [
+                "\n",
+                "One of the four rates immediately passes outside its threshold when drift is\n",
+                "induced. The same occurs shortly after leaving the drift region. The\n",
+                "recommended retraining data includes most of the drift induction window and\n",
+                "the data after regime change.\n",
+                "\n",
+                "The classifier's accuracy decreases again later, which causes the detector to\n",
+                "enter a \"warning\" state. Note that the retraining recommendations *begin* with\n",
+                "the index corresponding to the warning state, and end where drift is detected.\n"
+            ]
+        },
+        {
+            "cell_type": "code",
+            "execution_count": null,
+            "metadata": {},
+            "outputs": [],
+            "source": [
+                "plt.show()"
+            ]
+        },
+        {
+            "cell_type": "markdown",
+            "metadata": {},
+            "source": [
+                "## ADaptive WINdowing (ADWIN)"
+            ]
+        },
+        {
+            "cell_type": "markdown",
+            "metadata": {},
+            "source": [
+                "ADWIN can be used to monitor the accuracy of a classifier. ADWIN maintains a window of the data stream, which grows to the right as new elements are received. When the mean of the feature in one of the subwindows is different enough, ADWIN drops older elements in its window until this ceases to be the case."
+            ]
+        },
+        {
+            "cell_type": "code",
+            "execution_count": null,
+            "metadata": {},
+            "outputs": [],
+            "source": [
+                "## Setup ##\n",
+                "\n",
+                "# Set up classifier: train on first training_size rows\n",
+                "X_train = df.loc[0:training_size, [\"var1\", \"var2\"]]\n",
+                "y_train = df.loc[0:training_size, \"y\"]\n",
+                "\n",
+                "np.random.seed(123)\n",
+                "clf = GaussianNB()\n",
+                "clf.fit(X_train, y_train)\n",
+                "\n",
+                "adwin = ADWIN()\n",
+                "\n",
+                "# Set up DF to record results.\n",
+                "status = pd.DataFrame(\n",
+                "    columns=[\"index\", \"results\", \"accuracy\", \"adwin mean\", \"drift_detected\"]\n",
+                ")\n",
+                "correct = 0\n",
+                "rec_list = []"
+            ]
+        },
+        {
+            "cell_type": "code",
+            "execution_count": null,
+            "metadata": {},
+            "outputs": [],
+            "source": [
+                "# run ADWIN\n",
+                "n = 1\n",
+                "for i in range(training_size, len(df)):\n",
+                "\n",
+                "    X_test = df.loc[[i], [\"var1\", \"var2\"]]\n",
+                "    y_pred = int(clf.predict(X_test))\n",
+                "    y_true = int(df.loc[[i], \"y\"])\n",
+                "\n",
+                "    # increment accuracy\n",
+                "    if y_pred == y_true:\n",
+                "        correct += 1\n",
+                "    accuracy = correct / n\n",
+                "\n",
+                "    adwin.update(y_true, y_pred)\n",
+                "    status.loc[i] = [\n",
+                "        i,\n",
+                "        int(y_true == y_pred),\n",
+                "        accuracy,\n",
+                "        adwin.mean(),\n",
+                "        adwin.drift_state,\n",
+                "    ]\n",
+                "\n",
+                "    # If drift is detected, examine the window and retrain.\n",
+                "    if adwin.drift_state == \"drift\":\n",
+                "        retrain_start = adwin.retraining_recs[0] + training_size\n",
+                "        retrain_end = adwin.retraining_recs[1] + training_size\n",
+                "        rec_list.append([retrain_start, retrain_end])\n",
+                "\n",
+                "        # The retraining recommendations produced here correspond to the samples\n",
+                "        # which belong to ADWIN's new, smaller window, after drift is detected.\n",
+                "        # If retraining is not desired, omit the next four lines.\n",
+                "        X_train = df.loc[retrain_start:retrain_end, [\"var1\", \"var2\"]]\n",
+                "        y_train = df.loc[retrain_start:retrain_end, \"y\"]\n",
+                "        clf = GaussianNB()\n",
+                "        clf.fit(X_train, y_train)\n",
+                "\n",
+                "    n += 1"
+            ]
+        },
+        {
+            "cell_type": "code",
+            "execution_count": null,
+            "metadata": {},
+            "outputs": [],
+            "source": [
+                "## Plotting ##\n",
+                "\n",
+                "plt.figure(figsize=(20, 6))\n",
+                "plt.scatter(\"index\", \"accuracy\", data=status, label=\"Accuracy\")\n",
+                "plt.grid(False, axis=\"x\")\n",
+                "plt.xticks(fontsize=16)\n",
+                "plt.yticks(fontsize=16)\n",
+                "plt.title(\"ADWIN Results: Accuracy\", fontsize=22)\n",
+                "plt.ylabel(\"Value\", fontsize=18)\n",
+                "plt.xlabel(\"Index\", fontsize=18)\n",
+                "ylims = [0, 1.1]\n",
+                "plt.ylim(ylims)\n",
+                "\n",
+                "plt.axvspan(1000, 1250, alpha=0.5, label=\"Drift Induction Window\")\n",
+                "\n",
+                "# Draw red lines that indicate where drift was detected\n",
+                "plt.vlines(\n",
+                "    x=status.loc[status[\"drift_detected\"] == \"drift\"][\"index\"],\n",
+                "    ymin=ylims[0],\n",
+                "    ymax=ylims[1],\n",
+                "    label=\"Drift Detected\",\n",
+                "    color=\"red\",\n",
+                ")\n",
+                "\n",
+                "# Create a list of lines that indicate the retraining windows.\n",
+                "# Space them evenly, vertically.\n",
+                "rec_list = pd.DataFrame(rec_list)\n",
+                "rec_list[\"y_val\"] = np.linspace(\n",
+                "    start=0.05 * (ylims[1] - ylims[0]) + ylims[0],\n",
+                "    stop=0.2 * ylims[1],\n",
+                "    num=len(rec_list),\n",
+                ")\n",
+                "\n",
+                "# Draw green lines that indicate where retraining occurred\n",
+                "plt.hlines(\n",
+                "    y=rec_list[\"y_val\"],\n",
+                "    xmin=rec_list[0],\n",
+                "    xmax=rec_list[1],\n",
+                "    color=\"green\",\n",
+                "    label=\"Retraining Windows\",\n",
+                ")\n",
+                "\n",
+                "plt.legend(loc='lower right')"
+            ]
+        },
+        {
+            "cell_type": "markdown",
+            "metadata": {},
+            "source": [
+                "\n",
+                "After drift is induced, the accuracy decreases enough for ADWIN to shrink its\n",
+                "window and alarm;  subsequent windows also include data from the old regime,\n",
+                "so drift continues to be detected until the window shrinks enough to be\n",
+                "comprised mostly by the new regime.\n"
+            ]
+        },
+        {
+            "cell_type": "code",
+            "execution_count": null,
+            "metadata": {},
+            "outputs": [],
+            "source": [
+                "plt.show()\n",
+                "# plt.savefig(\"example_ADWIN.png\")"
+            ]
+        },
+        {
+            "cell_type": "markdown",
+            "metadata": {},
+            "source": [
+                "## Drift Detection Method (DDM)"
+            ]
+        },
+        {
+            "cell_type": "markdown",
+            "metadata": {},
+            "source": [
+                "DDM can enter either a \"drift\" or \"warning\" state, depending on how close a classifier's error rate has approached to those respective thresholds, defined by the warning_scale and drift_scale parameters."
+            ]
+        },
+        {
+            "cell_type": "code",
+            "execution_count": null,
+            "metadata": {},
+            "outputs": [],
+            "source": [
+                "## Setup ##\n",
+                "\n",
+                "np.random.seed(123)\n",
+                "# setup classifier: train on first training_size rows\n",
+                "X_train = df.loc[0:training_size, [\"var1\", \"var2\"]]\n",
+                "y_train = df.loc[0:training_size, \"y\"]\n",
+                "clf = GaussianNB()\n",
+                "clf.fit(X_train, y_train)"
+            ]
+        },
+        {
+            "cell_type": "markdown",
+            "metadata": {},
+            "source": [
+                "\n",
+                "These parameter values are chosen somewhat arbitrarily.\n",
+                "At least 100 samples must be seen before DDM tests for drift (the n_threshold\n",
+                "parameter); the other two define the warning and drift regions. The minimum\n",
+                "error rate (and its standard deviation) are found during a stable regime; the\n",
+                "warning_scale and drift_scale roughly correspond to how many standard standard\n",
+                "deviations away the current estimate must be in order for the detector to\n",
+                "alarm.\n"
+            ]
+        },
+        {
+            "cell_type": "code",
+            "execution_count": null,
+            "metadata": {},
+            "outputs": [],
+            "source": [
+                "ddm = DDM(n_threshold=100, warning_scale=7, drift_scale=10)\n",
+                "\n",
+                "# setup DF to store results\n",
+                "status = pd.DataFrame(columns=[\"index\", \"y\", \"y_pred\", \"drift_detected\", \"accuracy\"])\n",
+                "correct = 0\n",
+                "rec_list = []"
+            ]
+        },
+        {
+            "cell_type": "code",
+            "execution_count": null,
+            "metadata": {},
+            "outputs": [],
+            "source": [
+                "# run DDM and retrain\n",
+                "n = 1\n",
+                "for i in range(training_size, len(df)):\n",
+                "\n",
+                "    X_test = df.loc[[i], [\"var1\", \"var2\"]]\n",
+                "    y_pred = int(clf.predict(X_test))\n",
+                "    y_true = int(df.loc[[i], \"y\"])\n",
+                "\n",
+                "    # increment accuracy\n",
+                "    if y_pred == y_true:\n",
+                "        correct += 1\n",
+                "    accuracy = correct / n\n",
+                "\n",
+                "    ddm.update(y_true, y_pred)\n",
+                "    status.loc[i] = [i, y_true, y_pred, ddm.drift_state, accuracy]\n",
+                "\n",
+                "    # If drift is detected, examine the window and retrain.\n",
+                "    if ddm.drift_state == \"drift\":\n",
+                "        retrain_start = ddm.retraining_recs[0] + training_size\n",
+                "        retrain_end = ddm.retraining_recs[1] + training_size\n",
+                "        if (\n",
+                "            retrain_start == retrain_end\n",
+                "        ):  # minimum retraining window in case of sudden drift\n",
+                "            retrain_start = max(0, retrain_start - 300)\n",
+                "        rec_list.append([retrain_start, retrain_end])\n",
+                "\n",
+                "        # If retraining is not desired, omit the next four lines.\n",
+                "        X_train = df.loc[retrain_start:retrain_end, [\"var1\", \"var2\"]]\n",
+                "        y_train = df.loc[retrain_start:retrain_end, \"y\"]\n",
+                "        clf = GaussianNB()\n",
+                "        clf.fit(X_train, y_train)\n",
+                "\n",
+                "    n += 1"
+            ]
+        },
+        {
+            "cell_type": "code",
+            "execution_count": null,
+            "metadata": {},
+            "outputs": [],
+            "source": [
+                "## Plotting ##\n",
+                "\n",
+                "plt.figure(figsize=(20, 6))\n",
+                "plt.scatter(\"index\", \"accuracy\", data=status, label=\"Accuracy\")\n",
+                "plt.grid(False, axis=\"x\")\n",
+                "plt.xticks(fontsize=16)\n",
+                "plt.yticks(fontsize=16)\n",
+                "plt.title(\"DDM Results: Accuracy\", fontsize=22)\n",
+                "plt.ylabel(\"Value\", fontsize=18)\n",
+                "plt.xlabel(\"Index\", fontsize=18)\n",
+                "ylims = [-0.05, 1.1]\n",
+                "plt.ylim(ylims)\n",
+                "\n",
+                "plt.axvspan(1000, 1250, alpha=0.5, label=\"Drift Induction Window\")\n",
+                "\n",
+                "# Draw red lines that indicate where drift was detected\n",
+                "plt.vlines(\n",
+                "    x=status.loc[status[\"drift_detected\"] == \"drift\"][\"index\"],\n",
+                "    ymin=ylims[0],\n",
+                "    ymax=ylims[1],\n",
+                "    label=\"Drift Detected\",\n",
+                "    color=\"red\",\n",
+                "    linewidth=3,\n",
+                ")\n",
+                "\n",
+                "# Draw orange lines that indicate where warnings of drift were provided\n",
+                "plt.vlines(\n",
+                "    x=status.loc[status[\"drift_detected\"] == \"warning\"][\"index\"],\n",
+                "    ymin=ylims[0],\n",
+                "    ymax=ylims[1],\n",
+                "    label=\"Warning\",\n",
+                "    color=\"orange\",\n",
+                "    alpha=0.3,\n",
+                ")\n",
+                "\n",
+                "# Create a list of lines that indicate the retraining windows.\n",
+                "# Space them evenly, vertically.\n",
+                "rec_list = pd.DataFrame(rec_list)\n",
+                "rec_list[\"y_val\"] = np.linspace(\n",
+                "    start=0.05 * (ylims[1] - ylims[0]) + ylims[0],\n",
+                "    stop=0.2 * ylims[1],\n",
+                "    num=len(rec_list),\n",
+                ")\n",
+                "\n",
+                "# Draw green lines that indicate where retraining occurred\n",
+                "plt.hlines(\n",
+                "    y=rec_list[\"y_val\"],\n",
+                "    xmin=rec_list[0],\n",
+                "    xmax=rec_list[1],\n",
+                "    color=\"green\",\n",
+                "    label=\"Retraining Windows\",\n",
+                ")\n",
+                "\n",
+                "plt.legend()"
+            ]
+        },
+        {
+            "cell_type": "markdown",
+            "metadata": {},
+            "source": [
+                "\n",
+                "DDM initially alarms during the drift induction window, triggering retraining.\n",
+                "The subsequent dip in accuracy is large enough to put the detector in the\n",
+                "\"warning\" state, but not large enough for \"drift\" to be identified.\n",
+                "\n"
+            ]
+        },
+        {
+            "cell_type": "code",
+            "execution_count": null,
+            "metadata": {},
+            "outputs": [],
+            "source": [
+                "plt.show()\n",
+                "# plt.savefig(\"example_DDM.png\")"
+            ]
+        },
+        {
+            "cell_type": "markdown",
+            "metadata": {},
+            "source": [
+                "## Early Drift Detection Method (EDDM)"
+            ]
+        },
+        {
+            "cell_type": "markdown",
+            "metadata": {},
+            "source": [
+                "EDDM monitors the distance between two errors of a classifier - i.e., the number of samples between errors - rather than monitoring the error rate itself. Similar to DDM, it uses separate thresholds for \"warning\" and \"drift.\""
+            ]
+        },
+        {
+            "cell_type": "code",
+            "execution_count": null,
+            "metadata": {},
+            "outputs": [],
+            "source": [
+                "## Setup ##\n",
+                "\n",
+                "np.random.seed(123)\n",
+                "# setup classifier: train on first 500 rows\n",
+                "X_train = df.loc[0:training_size, [\"var1\", \"var2\"]]\n",
+                "y_train = df.loc[0:training_size, \"y\"]\n",
+                "clf = GaussianNB()\n",
+                "clf.fit(X_train, y_train)"
+            ]
+        },
+        {
+            "cell_type": "markdown",
+            "metadata": {},
+            "source": [
+                "- n_threshold specifies the number of new samples which must be observed before\n",
+                "tests for drift are run.\n",
+                "\n",
+                "- The warning_thresh and drift_thresh values roughly correspond to the ratio of\n",
+                "the 95th percentile for the current distance distribution vs. the 95th percentile\n",
+                "for the \"best\" distance distribution observed so far. So, lower values correspond to less conservative monitoring - the current\n",
+                "distance between errors is allowed to be a smaller fraction of the \"best\"\n",
+                "distance.\n"
+            ]
+        },
+        {
+            "cell_type": "code",
+            "execution_count": null,
+            "metadata": {},
+            "outputs": [],
+            "source": [
+                "eddm = EDDM(n_threshold=30, warning_thresh=0.7, drift_thresh=0.5)\n",
+                "\n",
+                "# setup DF to store results\n",
+                "status = pd.DataFrame(columns=[\"index\", \"y\", \"y_pred\", \"drift_detected\", \"accuracy\"])\n",
+                "correct = 0\n",
+                "rec_list = []"
+            ]
+        },
+        {
+            "cell_type": "code",
+            "execution_count": null,
+            "metadata": {},
+            "outputs": [],
+            "source": [
+                "# run EDDM and retrain\n",
+                "n = 1\n",
+                "for i in range(training_size, len(df)):\n",
+                "\n",
+                "    X_test = df.loc[[i], [\"var1\", \"var2\"]]\n",
+                "    y_pred = int(clf.predict(X_test))\n",
+                "    y_true = int(df.loc[[i], \"y\"])\n",
+                "\n",
+                "    # increment accuracy\n",
+                "    if y_pred == y_true:\n",
+                "        correct += 1\n",
+                "    accuracy = correct / n\n",
+                "\n",
+                "    eddm.update(y_true, y_pred)\n",
+                "    status.loc[i] = [i, y_true, y_pred, eddm.drift_state, accuracy]\n",
+                "\n",
+                "    # If drift is detected, examine the window and retrain.\n",
+                "    if eddm.drift_state == \"drift\":\n",
+                "        retrain_start = eddm.retraining_recs[0] + training_size\n",
+                "        retrain_end = eddm.retraining_recs[1] + training_size\n",
+                "        if (\n",
+                "            retrain_start == retrain_end\n",
+                "        ):  # minimum retraining window in case of sudden drift\n",
+                "            retrain_start = max(0, retrain_start - 300)\n",
+                "        rec_list.append([retrain_start, retrain_end])\n",
+                "\n",
+                "        # If retraining is not desired, omit the next four lines.\n",
+                "        X_train = df.loc[retrain_start:retrain_end, [\"var1\", \"var2\"]]\n",
+                "        y_train = df.loc[retrain_start:retrain_end, \"y\"]\n",
+                "        clf = GaussianNB()\n",
+                "        clf.fit(X_train, y_train)\n",
+                "\n",
+                "    n += 1"
+            ]
+        },
+        {
+            "cell_type": "code",
+            "execution_count": null,
+            "metadata": {},
+            "outputs": [],
+            "source": [
+                "## Plotting ##\n",
+                "\n",
+                "plt.figure(figsize=(20, 6))\n",
+                "plt.scatter(\"index\", \"accuracy\", data=status, label=\"Accuracy\")\n",
+                "plt.grid(False, axis=\"x\")\n",
+                "plt.xticks(fontsize=16)\n",
+                "plt.yticks(fontsize=16)\n",
+                "plt.title(\"EDDM Results: Accuracy\", fontsize=22)\n",
+                "plt.ylabel(\"Value\", fontsize=18)\n",
+                "plt.xlabel(\"Index\", fontsize=18)\n",
+                "ylims = [-0.05, 1.1]\n",
+                "plt.ylim(ylims)\n",
+                "\n",
+                "plt.axvspan(1000, 1250, alpha=0.5, label=\"Drift Induction Window\")\n",
+                "\n",
+                "# Draw orange lines that indicate where warnings of drift were provided\n",
+                "plt.vlines(\n",
+                "    x=status.loc[status[\"drift_detected\"] == \"warning\"][\"index\"],\n",
+                "    ymin=ylims[0],\n",
+                "    ymax=ylims[1],\n",
+                "    label=\"Warning\",\n",
+                "    color=\"orange\",\n",
+                "    alpha=0.3,\n",
+                ")\n",
+                "\n",
+                "# Draw red lines that indicate where drift was detected\n",
+                "plt.vlines(\n",
+                "    x=status.loc[status[\"drift_detected\"] == \"drift\"][\"index\"],\n",
+                "    ymin=ylims[0],\n",
+                "    ymax=ylims[1],\n",
+                "    label=\"Drift Detected\",\n",
+                "    color=\"red\",\n",
+                ")\n",
+                "\n",
+                "# Create a list of lines that indicate the retraining windows.\n",
+                "# Space them evenly, vertically.\n",
+                "rec_list = pd.DataFrame(rec_list)\n",
+                "rec_list[\"y_val\"] = np.linspace(\n",
+                "    start=0.05 * (ylims[1] - ylims[0]) + ylims[0],\n",
+                "    stop=0.2 * ylims[1],\n",
+                "    num=len(rec_list),\n",
+                ")\n",
+                "\n",
+                "# Draw green lines that indicate where retraining occurred\n",
+                "plt.hlines(\n",
+                "    y=rec_list[\"y_val\"],\n",
+                "    xmin=rec_list[0],\n",
+                "    xmax=rec_list[1],\n",
+                "    color=\"green\",\n",
+                "    label=\"Retraining Windows\",\n",
+                "    alpha=0.3,\n",
+                ")\n",
+                "\n",
+                "plt.legend()"
+            ]
+        },
+        {
+            "cell_type": "markdown",
+            "metadata": {},
+            "source": [
+                "EDDM enters a drift state shortly after the drift induction window, triggering\n",
+                "retraining. The later increase in the error rate causes the detector to enter\n",
+                "the warning state, but is not large enough to be identified as drift with this\n",
+                "threshold setting.\n"
+            ]
+        },
+        {
+            "cell_type": "code",
+            "execution_count": null,
+            "metadata": {},
+            "outputs": [],
+            "source": [
+                "plt.show()\n",
+                "# plt.savefig(\"example_EDDM.png\")"
+            ]
+        },
+        {
+            "cell_type": "markdown",
+            "metadata": {},
+            "source": [
+                "## Statistical Test of Equal Proportions to Detect Concept Drift (STEPD)"
+            ]
+        },
+        {
+            "cell_type": "markdown",
+            "metadata": {},
+            "source": [
+                "STEPD is a detector specifically intended for online classifiers, where each new sample is used to update the parameters of the classifier. STEPD monitors the accuracy in two windows, \"recent\" and \"past,\" and compares those in order to detect drift in classifier accuracy."
+            ]
+        },
+        {
+            "cell_type": "code",
+            "execution_count": null,
+            "metadata": {},
+            "outputs": [],
+            "source": [
+                "## Setup ##\n",
+                "\n",
+                "np.random.seed(123)\n",
+                "df_ex = df\n",
+                "train_ix = [0, training_size]\n",
+                "\n",
+                "\n",
+                "# For the purposes of this example, our online classifier is an SGDClassifier\n",
+                "# with a constant learning rate, since this is available in the pre-existing\n",
+                "# package dependency sklearn.\n",
+                "clf = SGDClassifier(learning_rate=\"constant\", shuffle=False, eta0=0.2)\n",
+                "clf.fit(\n",
+                "    df_ex.loc[train_ix[0] : train_ix[1], [\"var1\", \"var2\"]].values,\n",
+                "    df_ex.loc[train_ix[0] : train_ix[1], \"y\"].values,\n",
+                ")\n",
+                "\n",
+                "\n",
+                "stepd = STEPD(window_size=100)\n",
+                "\n",
+                "# setup DF to store results\n",
+                "status = pd.DataFrame(columns=[\"index\", \"y\", \"y_pred\", \"drift_detected\", \"accuracy\"])\n",
+                "correct = 0\n",
+                "rec_list = []"
+            ]
+        },
+        {
+            "cell_type": "code",
+            "execution_count": null,
+            "metadata": {},
+            "outputs": [],
+            "source": [
+                "# run STEPD and retrain\n",
+                "n = 1\n",
+                "for i, row in df_ex.iloc[training_size:].iterrows():\n",
+                "    y_pred = clf.predict(np.array(row[[\"var1\", \"var2\"]]).reshape(1, -1))\n",
+                "    y_true = row[\"y\"]\n",
+                "\n",
+                "    if y_pred == y_true:\n",
+                "        correct += 1\n",
+                "    accuracy = correct / n\n",
+                "\n",
+                "    stepd.update(y_true, y_pred)\n",
+                "    status.loc[i] = [i, y_true, y_pred, stepd.drift_state, accuracy]\n",
+                "    # train_ix[1] = train_ix[1] + 1\n",
+                "\n",
+                "    if stepd.drift_state == \"drift\":\n",
+                "        rec_list.append(stepd.retraining_recs)\n",
+                "        # retrain the classifier using STEPD's recommendations\n",
+                "        train_ix = stepd.retraining_recs\n",
+                "        train_ix[0] = train_ix[0] + training_size  # adjust for starting index\n",
+                "        train_ix[1] = train_ix[1] + training_size\n",
+                "        clf.fit(\n",
+                "            df_ex.loc[train_ix[0] : train_ix[1], [\"var1\", \"var2\"]].values,\n",
+                "            df_ex.loc[train_ix[0] : train_ix[1], \"y\"].values,\n",
+                "        )\n",
+                "\n",
+                "    else:\n",
+                "        # update the classifier with the newest sample\n",
+                "        clf.partial_fit(row[[\"var1\", \"var2\"]].values.reshape(1, -1), [row[\"y\"]])\n",
+                "\n",
+                "    n += 1"
+            ]
+        },
+        {
+            "cell_type": "code",
+            "execution_count": null,
+            "metadata": {},
+            "outputs": [],
+            "source": [
+                "## Plotting ##\n",
+                "\n",
+                "plt.figure(figsize=(20, 6))\n",
+                "plt.scatter(\"index\", \"accuracy\", data=status, label=\"Accuracy\")\n",
+                "plt.grid(False, axis=\"x\")\n",
+                "plt.xticks(fontsize=16)\n",
+                "plt.yticks(fontsize=16)\n",
+                "plt.title(\"STEPD Results: Accuracy\", fontsize=22)\n",
+                "plt.ylabel(\"Value\", fontsize=18)\n",
+                "plt.xlabel(\"Index\", fontsize=18)\n",
+                "ylims = [-0.05, 1.1]\n",
+                "plt.ylim(ylims)\n",
+                "\n",
+                "plt.axvspan(1000, 1250, alpha=0.5, label=\"Drift Induction Window\")\n",
+                "\n",
+                "# Draw orange lines that indicate where warnings of drift were provided\n",
+                "plt.vlines(\n",
+                "    x=status.loc[status[\"drift_detected\"] == \"warning\"][\"index\"],\n",
+                "    ymin=ylims[0],\n",
+                "    ymax=ylims[1],\n",
+                "    label=\"Warning\",\n",
+                "    color=\"orange\",\n",
+                "    alpha=0.3,\n",
+                ")\n",
+                "\n",
+                "# Draw red lines that indicate where drift was detected\n",
+                "plt.vlines(\n",
+                "    x=status.loc[status[\"drift_detected\"] == \"drift\"][\"index\"],\n",
+                "    ymin=ylims[0],\n",
+                "    ymax=ylims[1],\n",
+                "    label=\"Drift Detected\",\n",
+                "    color=\"red\",\n",
+                "    linewidth=3,\n",
+                ")\n",
+                "\n",
+                "# Create a list of lines that indicate the retraining windows.\n",
+                "# Space them evenly, vertically.\n",
+                "rec_list = pd.DataFrame(rec_list)\n",
+                "rec_list[\"y_val\"] = np.linspace(\n",
+                "    start=0.05 * (ylims[1] - ylims[0]) + ylims[0],\n",
+                "    stop=0.2 * ylims[1],\n",
+                "    num=len(rec_list),\n",
+                ")\n",
+                "\n",
+                "# Draw green lines that indicate where retraining occurred\n",
+                "plt.hlines(\n",
+                "    y=rec_list[\"y_val\"],\n",
+                "    xmin=rec_list[0],\n",
+                "    xmax=rec_list[1],\n",
+                "    color=\"green\",\n",
+                "    label=\"Retraining Windows\",\n",
+                ")\n",
+                "\n",
+                "plt.legend(loc='upper left')"
+            ]
+        },
+        {
+            "cell_type": "markdown",
+            "metadata": {},
+            "source": [
+                "STEPD identifies drift quite early in the drift induction window, triggering\n",
+                "retraining on a relatively small amount of data; after this, the online\n",
+                "classifier updates sufficiently that its accuracy is roughly flat over the\n",
+                "remaining data, albeit with a big enough change to trigger more warnings.\n"
+            ]
+        },
+        {
+            "cell_type": "code",
+            "execution_count": null,
+            "metadata": {},
+            "outputs": [],
+            "source": [
+                "plt.show()\n",
+                "# plt.savefig(\"example_STEPD.png\")"
+            ]
+        },
+        {
+            "cell_type": "markdown",
+            "metadata": {},
+            "source": [
+                "## Margin Density Drift Detection (MD3) Method"
+            ]
+        },
+        {
+            "cell_type": "markdown",
+            "metadata": {},
+            "source": [
+                "MD3 is a detector intended for semi-supervised and online classifier contexts. A cumulative margin density statistic (unsupervised) is tracked, representing the number of samples that fall into the uncertainty region, or margin, of the classifier. When margin density increases or decreases beyond a certain threshold, a drift warning is issued. When a warning is issued, a set of labeled samples is requested. If prediction accuracy by the model on these samples is lower than on the initial labeled reference dataset, drift is confirmed. If not, drift is ruled out."
+            ]
+        },
+        {
+            "cell_type": "code",
+            "execution_count": null,
+            "metadata": {},
+            "outputs": [],
+            "source": [
+                "## Setup ##\n",
+                "\n",
+                "# Set up classifier: train on first training_size rows\n",
+                "training_data = rainfall_df.loc[0:rainfall_training_size, rainfall_columns]\n",
+                "X_train = rainfall_df.loc[0:rainfall_training_size, rainfall_features]\n",
+                "y_train = rainfall_df.loc[0:rainfall_training_size, \"rain\"]\n",
+                "\n",
+                "np.random.seed(123)\n",
+                "clf = svm.SVC(kernel='linear')\n",
+                "clf.fit(X_train, y_train.values.ravel())\n",
+                "retrain_clf = clone(clf)\n",
+                "retrain_clf.fit(X_train, y_train.values.ravel())\n",
+                "oracle_labels = 1000\n",
+                "\n",
+                "# Initialize detector\n",
+                "md3 = MD3(clf=clf, sensitivity=1.5, oracle_data_length_required=oracle_labels)\n",
+                "md3.set_reference(X=training_data, target_name=\"rain\")\n",
+                "\n",
+                "# Set up DF to record results.\n",
+                "status = pd.DataFrame(\n",
+                "    columns=[\"index\", \"y\", \"margin_density\", \"original_accuracy\", \"retrain_accuracy\", \"drift_detected\"]\n",
+                ")\n",
+                "correct_orig, correct_retrain = 0, 0\n",
+                "n = 1\n",
+                "rec_list = []\n",
+                "oracle_list = []"
+            ]
+        },
+        {
+            "cell_type": "code",
+            "execution_count": null,
+            "metadata": {},
+            "outputs": [],
+            "source": [
+                "# run MD3 and track results for both original model and retrained model\n",
+                "for i in range(rainfall_training_size, len(rainfall_df)):\n",
+                "\n",
+                "    X_test = rainfall_df.loc[[i], rainfall_features]\n",
+                "    y_pred_orig = int(clf.predict(X_test))\n",
+                "    y_pred_retrain = int(retrain_clf.predict(X_test))\n",
+                "    y_true = int(rainfall_df.loc[[i], \"rain\"])\n",
+                "    \n",
+                "    # increment accuracy\n",
+                "    if y_pred_orig == y_true:\n",
+                "        correct_orig += 1\n",
+                "    if y_pred_retrain == y_true:\n",
+                "        correct_retrain += 1\n",
+                "    accuracy_orig = correct_orig / n\n",
+                "    accuracy_retrain = correct_retrain / n\n",
+                "\n",
+                "    # call give_oracle_label if detector is currently waiting for oracle data\n",
+                "    if md3.waiting_for_oracle == True:\n",
+                "        oracle_label = rainfall_df.loc[[i], rainfall_columns]\n",
+                "        md3.give_oracle_label(oracle_label)\n",
+                "        if md3.waiting_for_oracle == False:\n",
+                "            retrain_clf.fit(md3.reference_batch_features, md3.reference_batch_target.values.ravel())\n",
+                "        status.loc[i] = [\n",
+                "            i,\n",
+                "            y_true,\n",
+                "            None,\n",
+                "            accuracy_orig,\n",
+                "            accuracy_retrain,\n",
+                "            md3.drift_state,\n",
+                "        ]\n",
+                "\n",
+                "    # call update otherwise\n",
+                "    else:\n",
+                "        md3.update(X_test)\n",
+                "        status.loc[i] = [\n",
+                "            i,\n",
+                "            y_true,\n",
+                "            md3.curr_margin_density,\n",
+                "            accuracy_orig,\n",
+                "            accuracy_retrain,\n",
+                "            md3.drift_state,\n",
+                "        ]\n",
+                "    \n",
+                "    # If there was a drift warning, track the window of the labeled\n",
+                "    # oracle data used\n",
+                "    if md3.drift_state == \"warning\":\n",
+                "        oracle_start = i + 1\n",
+                "        oracle_end = i + md3.oracle_data_length_required\n",
+                "        \n",
+                "        oracle_list.append([oracle_start, oracle_end])\n",
+                "\n",
+                "    n += 1"
+            ]
+        },
+        {
+            "cell_type": "code",
+            "execution_count": null,
+            "metadata": {},
+            "outputs": [],
+            "source": [
+                "## Plotting ##\n",
+                "\n",
+                "plt.figure(figsize=(20, 6))\n",
+                "plt.scatter(\"index\", \"margin_density\", data=status, label=\"Margin Density\")\n",
+                "plt.scatter(\"index\", \"original_accuracy\", data=status, label=\"Original Accuracy\", color=\"red\")\n",
+                "plt.scatter(\"index\", \"retrain_accuracy\", data=status, label=\"Retrain Accuracy\", color=\"green\")\n",
+                "plt.grid(False, axis=\"x\")\n",
+                "plt.xticks(fontsize=16)\n",
+                "plt.yticks(fontsize=16)\n",
+                "plt.title(\"MD3 Results: Margin Density and Accuracy\", fontsize=22)\n",
+                "plt.ylabel(\"Value\", fontsize=18)\n",
+                "plt.xlabel(\"Index\", fontsize=18)\n",
+                "ylims = [-0.05, 1.1]\n",
+                "plt.ylim(ylims)\n",
+                "\n",
+                "plt.axvspan(rainfall_drift_start, rainfall_drift_end, alpha=0.5, label=\"Drift Induction Window\")\n",
+                "\n",
+                "# Draw red lines that indicate where drift was detected\n",
+                "plt.vlines(\n",
+                "    x=status.loc[status[\"drift_detected\"] == \"drift\"][\"index\"],\n",
+                "    ymin=ylims[0],\n",
+                "    ymax=ylims[1],\n",
+                "    label=\"Drift Detected\",\n",
+                "    color=\"red\",\n",
+                ")\n",
+                "\n",
+                "# Draw orange lines that indicate where warnings of drift were provided\n",
+                "plt.vlines(\n",
+                "    x=status.loc[status[\"drift_detected\"] == \"warning\"][\"index\"],\n",
+                "    ymin=ylims[0],\n",
+                "    ymax=ylims[1],\n",
+                "    label=\"Warning\",\n",
+                "    color=\"orange\",\n",
+                "    alpha=0.3,\n",
+                ")\n",
+                "\n",
+                "# Create a list of lines that indicate the retraining windows.\n",
+                "# Space them evenly, vertically.\n",
+                "oracle_list = pd.DataFrame(oracle_list)\n",
+                "oracle_list[\"y_val\"] = np.linspace(\n",
+                "    start=0.05 * (ylims[1] - ylims[0]) + ylims[0],\n",
+                "    stop=0.2 * ylims[1],\n",
+                "    num=len(oracle_list),\n",
+                ")\n",
+                "\n",
+                "# Draw green lines that indicate where retraining occurred\n",
+                "plt.hlines(\n",
+                "    y=oracle_list[\"y_val\"],\n",
+                "    xmin=oracle_list[0],\n",
+                "    xmax=oracle_list[1],\n",
+                "    color=\"green\",\n",
+                "    label=\"Labeled Oracle Data\",\n",
+                ")\n",
+                "\n",
+                "plt.legend()"
+            ]
+        },
+        {
+            "cell_type": "markdown",
+            "metadata": {},
+            "source": [
+                "After drift is induced, the margin density decreases enough for MD3 to emit a warning. From there, the predictive accuracy of the classifier is tested, and this has already decreased sufficiently for the detector to alarm. Then, a new reference batch is set and the detector continues tracking the margin density statistic until the next warning. It can be seen from the plot that retraining at drift results in better accuracy moving forward compared with a model that is kept static."
+            ]
+        },
+        {
+            "cell_type": "code",
+            "execution_count": null,
+            "metadata": {},
+            "outputs": [],
+            "source": [
+                "plt.show()\n",
+                "# plt.savefig(\"example_MD3.png\")"
+            ]
+        }
+    ],
+    "metadata": {
+        "kernelspec": {
+            "display_name": "Python 3.10.2 ('molten_env')",
+            "language": "python",
+            "name": "python3"
+        },
+        "language_info": {
+            "codemirror_mode": {
+                "name": "ipython",
+                "version": 3
+            },
+            "file_extension": ".py",
+            "mimetype": "text/x-python",
+            "name": "python",
+            "nbconvert_exporter": "python",
+            "pygments_lexer": "ipython3",
+            "version": "3.10.2"
+        },
+        "orig_nbformat": 4,
+        "vscode": {
+            "interpreter": {
+                "hash": "cebda30e5482c3c83cb9c7c7d8ef1c1d1f67dcf93284b9e88f69b68f560a7bf6"
+            }
+        }
+    },
+    "nbformat": 4,
+    "nbformat_minor": 2
 }
diff --git a/examples/change_detection_examples.py b/examples/change_detection_examples.py
index cbb7b4cd..90aa8af3 100644
--- a/examples/change_detection_examples.py
+++ b/examples/change_detection_examples.py
@@ -37,7 +37,7 @@
 drift_start, drift_end = 1000, 1250
 
 
-# ## Page-Hinkley (PH) Test Example
+# ## Page-Hinkley (PH) Test
 
 # In[ ]:
 
@@ -54,7 +54,7 @@
 # iterate through data; feed each sample to the detector, in turn
 for i in range(len(df)):
     obs = df["var2"][i]
-    ph.update(next_obs=obs, obs_id=i)
+    ph.update(X=obs)
     status.loc[i] = [i, obs, ph.drift_state]
 
 
@@ -102,7 +102,7 @@
 # plt.savefig("example_Page-Hinkley_detections.png")
 
 
-# ## Cumulative Sum (CUSUM) Test Example
+# ## Cumulative Sum (CUSUM) Test
 
 # - This monitors a moving average of var2, starting from an initial estimate of mean
 # and standard deviation.
@@ -177,6 +177,7 @@
 # In[ ]:
 
 
+
 plt.show()
 # plt.savefig("example_CUSUM_detections.png")
 
diff --git a/examples/concept_drift_examples.py b/examples/concept_drift_examples.py
index 934c675e..efcc927c 100644
--- a/examples/concept_drift_examples.py
+++ b/examples/concept_drift_examples.py
@@ -18,9 +18,8 @@
 # an initial classifier, and then use the remaining samples for testing.
 # 
 # These detectors are generally to be applied to the true class and predicted class 
-# from a particular model. ADWIN is an exception in that it could also be used to 
-# monitor an arbitrary real-valued feature. So, each of the summary plots displays 
-# the running accuracy of the classifier alongside the drift detector's output.
+# from a particular model. So, each of the summary plots displays the running
+# accuracy of the classifier alongside the drift detector's output.
 # 
 # They also track the indices of portions of the incoming data stream which are 
 # more similar to each other -- i.e., data that seems to be part of the same 
@@ -28,7 +27,7 @@
 # 
 # NOTE: The LinearFourRates example has a relatively long runtime, roughly 5 minutes.
 
-# In[1]:
+# In[ ]:
 
 
 ## Imports ##
@@ -45,7 +44,7 @@
 from menelaus.datasets import fetch_rainfall_data
 
 
-# In[2]:
+# In[ ]:
 
 
 ## Import Data ##
@@ -64,7 +63,7 @@
 rainfall_df[rainfall_features] = rainfall_df[rainfall_features].astype(float)
 
 
-# ## Linear Four Rates (LFR) Example
+# ## Linear Four Rates (LFR)
 
 # Linear Four Rates monitors the four cells of the confusion matrix (TPR, FPR, TNR, FNR) and alarms when one of these becomes different enough from earlier performance.
 
@@ -116,7 +115,7 @@
         correct += 1
     accuracy = correct / n
 
-    lfr.update(y_pred, y_true)
+    lfr.update(y_true, y_pred)
     status.loc[i] = [i, y_true, y_pred, lfr.drift_state, accuracy]
 
     # If drift is detected, examine the retraining recommendations and retrain.
@@ -214,9 +213,9 @@
 plt.show()
 
 
-# ## ADaptive WINdowing (ADWIN) Example
+# ## ADaptive WINdowing (ADWIN)
 
-# ADWIN can be used to monitor the average of a given real-valued feature. In this case, we use it to monitor the accuracy of a classifier. ADWIN maintains a window of the data stream, which grows to the right as new elements are received. When the mean of the feature in one of the subwindows is different enough, ADWIN drops older elements in its window until this ceases to be the case.
+# ADWIN can be used to monitor the accuracy of a classifier. ADWIN maintains a window of the data stream, which grows to the right as new elements are received. When the mean of the feature in one of the subwindows is different enough, ADWIN drops older elements in its window until this ceases to be the case.
 
 # In[ ]:
 
@@ -257,7 +256,7 @@
         correct += 1
     accuracy = correct / n
 
-    adwin.update(int(y_true == y_pred))
+    adwin.update(y_true, y_pred)
     status.loc[i] = [
         i,
         int(y_true == y_pred),
@@ -345,7 +344,7 @@
 # plt.savefig("example_ADWIN.png")
 
 
-# ## Drift Detection Method (DDM) Example
+# ## Drift Detection Method (DDM)
 
 # DDM can enter either a "drift" or "warning" state, depending on how close a classifier's error rate has approached to those respective thresholds, defined by the warning_scale and drift_scale parameters.
 
@@ -399,7 +398,7 @@
         correct += 1
     accuracy = correct / n
 
-    ddm.update(y_pred, y_true)
+    ddm.update(y_true, y_pred)
     status.loc[i] = [i, y_true, y_pred, ddm.drift_state, accuracy]
 
     # If drift is detected, examine the window and retrain.
@@ -494,7 +493,7 @@
 # plt.savefig("example_DDM.png")
 
 
-# ## Early Drift Detection Method (EDDM) Example
+# ## Early Drift Detection Method (EDDM)
 
 # EDDM monitors the distance between two errors of a classifier - i.e., the number of samples between errors - rather than monitoring the error rate itself. Similar to DDM, it uses separate thresholds for "warning" and "drift."
 
@@ -548,7 +547,7 @@
         correct += 1
     accuracy = correct / n
 
-    eddm.update(y_pred, y_true)
+    eddm.update(y_true, y_pred)
     status.loc[i] = [i, y_true, y_pred, eddm.drift_state, accuracy]
 
     # If drift is detected, examine the window and retrain.
@@ -642,7 +641,7 @@
 # plt.savefig("example_EDDM.png")
 
 
-# ## Statistical Test of Equal Proportions to Detect Concept Drift (STEPD) Example
+# ## Statistical Test of Equal Proportions to Detect Concept Drift (STEPD)
 
 # STEPD is a detector specifically intended for online classifiers, where each new sample is used to update the parameters of the classifier. STEPD monitors the accuracy in two windows, "recent" and "past," and compares those in order to detect drift in classifier accuracy.
 
@@ -687,7 +686,7 @@
         correct += 1
     accuracy = correct / n
 
-    stepd.update(y_pred, y_true)
+    stepd.update(y_true, y_pred)
     status.loc[i] = [i, y_true, y_pred, stepd.drift_state, accuracy]
     # train_ix[1] = train_ix[1] + 1
 
@@ -781,11 +780,11 @@
 # plt.savefig("example_STEPD.png")
 
 
-# ## Margin Density Drift Detection (MD3) Method Example
+# ## Margin Density Drift Detection (MD3) Method
 
 # MD3 is a detector intended for semi-supervised and online classifier contexts. A cumulative margin density statistic (unsupervised) is tracked, representing the number of samples that fall into the uncertainty region, or margin, of the classifier. When margin density increases or decreases beyond a certain threshold, a drift warning is issued. When a warning is issued, a set of labeled samples is requested. If prediction accuracy by the model on these samples is lower than on the initial labeled reference dataset, drift is confirmed. If not, drift is ruled out.
 
-# In[3]:
+# In[ ]:
 
 
 ## Setup ##
@@ -804,7 +803,7 @@
 
 # Initialize detector
 md3 = MD3(clf=clf, sensitivity=1.5, oracle_data_length_required=oracle_labels)
-md3.set_reference(training_data, "rain")
+md3.set_reference(X=training_data, target_name="rain")
 
 # Set up DF to record results.
 status = pd.DataFrame(
@@ -816,7 +815,7 @@
 oracle_list = []
 
 
-# In[4]:
+# In[ ]:
 
 
 # run MD3 and track results for both original model and retrained model
@@ -873,7 +872,7 @@
     n += 1
 
 
-# In[5]:
+# In[ ]:
 
 
 ## Plotting ##
@@ -935,7 +934,7 @@
 
 # After drift is induced, the margin density decreases enough for MD3 to emit a warning. From there, the predictive accuracy of the classifier is tested, and this has already decreased sufficiently for the detector to alarm. Then, a new reference batch is set and the detector continues tracking the margin density statistic until the next warning. It can be seen from the plot that retraining at drift results in better accuracy moving forward compared with a model that is kept static.
 
-# In[6]:
+# In[ ]:
 
 
 plt.show()
diff --git a/examples/data_drift_examples.py b/examples/data_drift_examples.py
index d53ef11d..07353e67 100644
--- a/examples/data_drift_examples.py
+++ b/examples/data_drift_examples.py
@@ -49,7 +49,7 @@
 circle_data.shape
 
 
-# ## Confidence Distribution Batch Detection (CDBD) Example
+# ## Confidence Distribution Batch Detection (CDBD)
 
 # This section details how to setup, run, and produce plots for CDBD. This script
 # monitors the feature "confidence", simulated confidence scores output by a
@@ -154,7 +154,7 @@ def distance_metric(reference_histogram, test_histogram):
 cdbd.update(pd.DataFrame(example_data[example_data.year == 2008].loc[:, "confidence"]))
 
 
-# ## Hellinger Distance Drift Detection Method (HDDDM) Example
+# ## Hellinger Distance Drift Detection Method (HDDDM)
 
 # This section details how to setup, run, and produce plots for HDDDM, using both
 # numeric and categorical data. Drift occurs in 2009, 2012, 2015, 2018, and 2021.
@@ -310,7 +310,7 @@ def distance_metric(reference_histogram, test_histogram):
 hdddm.update(example_data[example_data.year == 2008].iloc[:, 1:-2])
 
 
-# ## PCA-Based Change Detection (PCA-CD) Example
+# ## PCA-Based Change Detection (PCA-CD)
 
 # PCA-CD is a drift detector that transforms the passed data into its principal components, then watches the transformed data for signs of drift by monitoring the KL-divergence via the Page-Hinkley algorithm.
 
@@ -395,7 +395,7 @@ def distance_metric(reference_histogram, test_histogram):
 # plt.savefig("example_PCA_CD.png")
 
 
-# ## KDQ-Tree Detection Method Example
+# ## KDQ-Tree Detection Method (Streaming Setting)
 
 # KdqTree monitors incoming features by constructing a tree which partitions the feature-space, and then monitoring a divergence statistic that is defined over that partition. It watches data within a sliding window of a particular size. When that window is full, it builds the reference tree. As the window moves forward, point-by-point, the data in that new window is compared against the reference tree to detect drift.
 
@@ -496,7 +496,7 @@ def distance_metric(reference_histogram, test_histogram):
     # fig.write_html(f"example_streaming_kdqtree_treemap_{title}.html")
 
 
-# ## KDQ-Tree Detection Method Example (Batch Setting)
+# ## KDQ-Tree Detection Method (Batch Setting)
 
 # This example shows up how to set up, run, and produce output from the kdq-Tree
 # detector, specifically in the batch data setting. The parameters aren't 
diff --git a/examples/partitioner_plots_example.py b/examples/partitioner_plots_example.py
index 026fd2dc..8226f743 100644
--- a/examples/partitioner_plots_example.py
+++ b/examples/partitioner_plots_example.py
@@ -24,7 +24,7 @@
 from menelaus.partitioners.KDQTreePartitioner import KDQTreePartitioner, KDQTreeNode
 
 
-# ## Basic Plot Example
+# ## Basic Plot
 
 # In[ ]:
 
@@ -80,7 +80,7 @@
 # fig.write_html(f"example_partitioner_lots_basic_plot_depth.html")
 
 
-# ## Modifications Example
+# ## Modifications
 
 # In[ ]:
 
@@ -175,7 +175,7 @@
 # fig.write_html(f"example_partitioner_lots_modifications4_outline.html")
 
 
-# ## Alternatives Example
+# ## Alternatives
 
 # In[ ]:
 
diff --git a/src/menelaus/change_detection/cusum.py b/src/menelaus/change_detection/cusum.py
index 4b4d73d5..7bbb5923 100644
--- a/src/menelaus/change_detection/cusum.py
+++ b/src/menelaus/change_detection/cusum.py
@@ -75,11 +75,13 @@ def reset(self):
         self._upper_bound = [0]
         self._lower_bound = [0]
 
-    def update(self, next_obs):
+    def update(self, X, y_true=None, y_pred=None):
         """Update the detector with a new sample.
 
         Args:
-          next_obs: The value of the new sample.
+            X (numpy.ndarray): The value of the new sample.
+            y_true (numpy.ndarray): True label of new sample - not used in CUSUM.
+            y_pred (numpy.ndarray): Predicted label of new sample - not used in CUSUM.
         """
         # if the last run resulted in drift, reset everything and use last 30
         # obs to estimate stats
@@ -88,8 +90,8 @@ def update(self, next_obs):
             self.sd_hat = np.std(self._stream[-self.burn_in :])
             self.reset()
 
-        super().update()
-        self._stream.append(next_obs)
+        super().update(X, y_true, y_pred)
+        self._stream.append(X)
 
         # cannot compute s_h/s_l so set to 0
         if (self.target is None) & (self.updates_since_reset < self.burn_in):
diff --git a/src/menelaus/change_detection/page_hinkley.py b/src/menelaus/change_detection/page_hinkley.py
index 90cb87b7..e92df494 100644
--- a/src/menelaus/change_detection/page_hinkley.py
+++ b/src/menelaus/change_detection/page_hinkley.py
@@ -70,7 +70,6 @@ def __init__(self, delta=0.01, threshold=20, burn_in=30, direction="positive"):
 
         # currently, if these need to be made available, they are through the
         # to_dataframe method
-        self._ids = []
         self._change_scores = []
         self._page_hinkley_values = []
         self._page_hinkley_differences = []
@@ -80,19 +79,21 @@ def __init__(self, delta=0.01, threshold=20, burn_in=30, direction="positive"):
         self._mins = []
         self._means = []
 
-    def update(self, next_obs, *args, obs_id=None, **kwargs):
+    def update(self, X, y_true=None, y_pred=None):
         """Update the detector with a new sample.
 
         Args:
-          next_obs: The value of the new sample.
-          obs_id: Index of new sample to store in dataframe. Defaults to ``None``.
+          X (numpy.ndarray): The value of the new sample.
+          y_true (numpy.ndarray): True label of new sample - not used in PageHinkley
+          y_pred (numpy.ndarray): Predicted label of new sample - not used in PageHinkley
+          obs_id (numpy.ndarray): Index of new sample to store in dataframe. Defaults to ``None``.
         """
         if self.drift_state == "drift":
             self.reset()
-        super().update()
+        super().update(X, y_true, y_pred)
 
-        self._mean = self._mean + (next_obs - self._mean) / self.updates_since_reset
-        self._sum = self._sum + next_obs - self._mean - self.delta
+        self._mean = self._mean + (X - self._mean) / self.updates_since_reset
+        self._sum = self._sum + X - self._mean - self.delta
         theta = self.threshold * self._mean
 
         if self._sum < self._min:
@@ -111,8 +112,7 @@ def update(self, next_obs, *args, obs_id=None, **kwargs):
         if drift_check and self.updates_since_reset > self.burn_in:
             self.drift_state = "drift"
 
-        self._ids.append(obs_id)
-        self._change_scores.append(next_obs)
+        self._change_scores.append(X)
         self._page_hinkley_values.append(self._sum)
         self._page_hinkley_differences.append(ph_difference)
         self._drift_detected.append(drift_check)
@@ -132,7 +132,6 @@ def reset(self):
         self._sum = 0
         self._mean = 0
 
-        self._ids = []
         self._change_scores = []
         self._page_hinkley_values = []
         self._page_hinkley_differences = []
@@ -147,7 +146,6 @@ def to_dataframe(self):
         """Returns a dataframe storing current statistics"""
         return pd.DataFrame(
             {
-                "ids": self._ids,
                 "change_scores": self._change_scores,
                 "page_hinkley_values": self._page_hinkley_values,
                 "page_hinkley_differences": self._page_hinkley_differences,
diff --git a/src/menelaus/concept_drift/adwin.py b/src/menelaus/concept_drift/adwin.py
index 028c8468..5e2fe160 100644
--- a/src/menelaus/concept_drift/adwin.py
+++ b/src/menelaus/concept_drift/adwin.py
@@ -104,17 +104,26 @@ def __init__(
         self._window_size = 0
         self._retraining_recs = [None, None]
 
-    def update(self, new_value):
+    def update(self, y_true=None, y_pred=None, X=None):
         """Update the detector with a new sample.
 
         Args:
-          new_value: next sample in the stream of data being fed to ADWIN
+          y_true: actual class of next sample
+          y_pred: predicted class of next sample
+          X: next sample in the stream of data - not used in ADWIN
         """
+
+        # Technically, ADWIN could monitor the running mean of some feature
+        # aside from accuracy, but that leads to potentially indeterminate input
+        # (user passes the full y_true, y_pred, X triplet), so -- maybe if
+        # there's a big enough need.
+        new_value = int(y_true == y_pred)
+
         if self.drift_state is not None:
             # note that the other attributes should *not* be initialized after drift
             self.reset()
 
-        super().update()
+        super().update(X, y_true, y_pred)
         # add new sample to the head of the window
         self._window_size += 1
         self._add_sample(new_value)
diff --git a/src/menelaus/concept_drift/ddm.py b/src/menelaus/concept_drift/ddm.py
index af570395..68c9b785 100644
--- a/src/menelaus/concept_drift/ddm.py
+++ b/src/menelaus/concept_drift/ddm.py
@@ -69,17 +69,24 @@ def reset(self):
         self._error_std_min = float("inf")
         self._initialize_retraining_recs()
 
-    def update(self, y_pred, y_true):
+    # XXX - Order of y_true, y_pred, X differs from abstractmethod signature
+    #       for update(). This is done for convenience, so users can call e.g.
+    #       DDM.update(1,1) without misinterpretation, but exposes them to a
+    #       potential issue where LFR.update(X, y, y) would assign arguments
+    #       incorrectly.
+    def update(self, y_true, y_pred, X=None):
         """Update the detector with a new sample.
 
         Args:
-          y_pred: predicted class
           y_true: actual class
+          y_pred: predicted class
+          X: new sample - not used in DDM
         """
+
         if self.drift_state == "drift":
             self.reset()
 
-        super().update()
+        super().update(X, y_true, y_pred)
         classifier_result = int(y_pred != y_true)
 
         # with each sample, update estimate of error and its std, along with minimums
diff --git a/src/menelaus/concept_drift/eddm.py b/src/menelaus/concept_drift/eddm.py
index 37c62a16..187c43ec 100644
--- a/src/menelaus/concept_drift/eddm.py
+++ b/src/menelaus/concept_drift/eddm.py
@@ -73,7 +73,12 @@ def reset(self):
         self._test_statistic = None
         self._initialize_retraining_recs()
 
-    def update(self, y_pred, y_true):
+    # XXX - Order of y_true, y_pred, X differs from abstractmethod signature
+    #       for update(). This is done for convenience, so users can call e.g.
+    #       EDDM.update(1,1) without misinterpretation, but exposes them to a
+    #       potential issue where LFR.update(X, y, y) would assign arguments
+    #       incorrectly.
+    def update(self, y_true, y_pred, X=None):
         """Update the detector with a new sample.
 
         Args:
@@ -83,7 +88,7 @@ def update(self, y_pred, y_true):
         if self.drift_state == "drift":
             self.reset()
 
-        super().update()
+        super().update(X, y_true, y_pred)
         classifier_result = int(y_pred == y_true)
 
         # found a new error, so update
diff --git a/src/menelaus/concept_drift/lfr.py b/src/menelaus/concept_drift/lfr.py
index 476d9dbd..68afbf63 100644
--- a/src/menelaus/concept_drift/lfr.py
+++ b/src/menelaus/concept_drift/lfr.py
@@ -121,7 +121,12 @@ def reset(self):
         }
         self._initialize_retraining_recs()
 
-    def update(self, y_pred, y_true, *args, round_val=4, **kwargs):
+    # XXX - Order of y_true, y_pred, X differs from abstractmethod signature
+    #       for update(). This is done for convenience, so users can call e.g.
+    #       LFR.update(1,1) without misinterpretation, but exposes them to a
+    #       potential issue where LFR.update(X, y, y) would assign arguments
+    #       incorrectly.
+    def update(self, y_true, y_pred, X=None, round_val=4):
         """Update detector with a new observation:
 
         #. Updates confusion matrix (``self._confusion``) with new predictions
@@ -135,17 +140,19 @@ def update(self, y_pred, y_true, *args, round_val=4, **kwargs):
            ``"warning"`` or ``"drift"``
 
         Args:
-            y_pred: predicted class
             y_true: actual class
+            y_pred: predicted class
+            X: new sample - not used in LFR
             round_val: number of decimal points the estimate rate is rounded to
                 when stored in bounds dictionary. The greater the ``round_val``, the
                 more precise the bounds dictionary will be, and the longer the
                 runtime. (Default value = 4)
         """
+
         if self.drift_state == "drift":
             self.reset()
 
-        super().update()
+        super().update(X, y_true, y_pred)
         y_p = 1 * y_pred
         y_t = 1 * y_true
 
diff --git a/src/menelaus/concept_drift/md3.py b/src/menelaus/concept_drift/md3.py
index 64621350..f4f3d8d6 100644
--- a/src/menelaus/concept_drift/md3.py
+++ b/src/menelaus/concept_drift/md3.py
@@ -110,28 +110,24 @@ def __init__(
         self.oracle_data = None
         self.waiting_for_oracle = False
 
-    def set_reference(self, reference_batch, target_name):
+    def set_reference(self, X, y_true=None, y_pred=None, target_name=None):
         """
         Initialize detector with a reference batch. Reference batch must be
         manually set and updated by user using this method. Reference batch
         is not automatically updated after a drift is detected.
 
         Args:
-            reference_batch (DataFrame): initial baseline dataset
+            reference_batch (pandas.DataFrame): initial baseline dataset
+            y_true (numpy.array): true labels of dataset - not used in MD3
+            y_pred (numpy.array): predicted labels of dataset - not used in MD3
             target_name (string): name of the column in the reference_batch
                 dataframe which is the target variable
         """
 
-        self.reference_batch_features = reference_batch.loc[
-            :, reference_batch.columns != target_name
-        ]
-        self.reference_batch_target = reference_batch.loc[
-            :, reference_batch.columns == target_name
-        ]
+        self.reference_batch_features = X.loc[:, X.columns != target_name]
+        self.reference_batch_target = X.loc[:, X.columns == target_name]
 
-        self.reference_distribution = self.calculate_distribution_statistics(
-            reference_batch
-        )
+        self.reference_distribution = self.calculate_distribution_statistics(X)
 
         if self.oracle_data_length_required is None:
             self.oracle_data_length_required = self.reference_distribution["len"]
@@ -216,12 +212,14 @@ def calculate_distribution_statistics(self, data):
             "acc_std": acc_std,
         }
 
-    def update(self, new_sample):
+    def update(self, X, y_true=None, y_pred=None):
         """
         Update the detector with a new sample.
 
         Args:
-            new_sample (DataFrame): feature values/sample data for the new incoming sample
+            X (DataFrame): feature values/sample data for the new incoming sample
+            y_true (numpy.array): true label of new sample - not used in MD3
+            y_pred (numpy.array): predicted label of new sample - not used in MD3
         """
 
         if self.waiting_for_oracle == True:
@@ -230,7 +228,7 @@ def update(self, new_sample):
                 labeled sample to confirm or rule out drift."""
             )
 
-        if len(new_sample) != 1:
+        if len(X) != 1:
             raise ValueError(
                 """This method is only available for data inputs in the form of 
                 a Pandas DataFrame with exactly 1 record."""
@@ -239,9 +237,9 @@ def update(self, new_sample):
         if self.drift_state == "drift":
             self.reset()
 
-        super().update()
+        super().update(X, y_true, y_pred)
 
-        sample_np_array = new_sample.to_numpy()[0]
+        sample_np_array = X.to_numpy()[0]
         margin_inclusion_signal = self.margin_calculation_function(
             self, sample_np_array, self.classifier
         )
@@ -318,7 +316,7 @@ def give_oracle_label(self, labeled_sample):
                 self.drift_state = "drift"
 
             # update reference distribution
-            self.set_reference(self.oracle_data, target_column[0])
+            self.set_reference(self.oracle_data, target_name=target_column[0])
             self.oracle_data = None
             self.waiting_for_oracle = False
 
diff --git a/src/menelaus/concept_drift/stepd.py b/src/menelaus/concept_drift/stepd.py
index b1728ed0..0f680e1a 100644
--- a/src/menelaus/concept_drift/stepd.py
+++ b/src/menelaus/concept_drift/stepd.py
@@ -75,17 +75,18 @@ def reset(self):
         self._test_p = None
         self._initialize_retraining_recs()
 
-    def update(self, y_pred, y_true):
+    def update(self, y_true, y_pred, X=None):
         """Update the detector with a new sample.
 
         Args:
-          y_pred: predicted class
-          y_true: actual class
+          X (numpy.ndarray): new sample - not used in STEPD
+          y_pred (numpy.ndarray): predicted class
+          y_true (numpy.ndarray): actual class
         """
         if self.drift_state == "drift":
             self.reset()
 
-        super().update()
+        super().update(X, y_true, y_pred)
         classifier_result = int(y_pred == y_true)
         self._s += classifier_result
 
diff --git a/src/menelaus/data_drift/cdbd.py b/src/menelaus/data_drift/cdbd.py
index d46f9d25..6cf309db 100644
--- a/src/menelaus/data_drift/cdbd.py
+++ b/src/menelaus/data_drift/cdbd.py
@@ -161,37 +161,41 @@ def __init__(
             subsets=subsets,
         )
 
-    def set_reference(self, reference_batch):
+    def set_reference(self, X, y_true=None, y_pred=None):
         """
         Initialize detector with a reference batch. After drift, reference batch is
         automatically set to most recent test batch. Option for user to specify
         alternative reference batch using this method.
 
         Args:
-            reference_batch (DataFrame): initial baseline dataset
+            X (pandas.DataFrame): initial baseline dataset
+            y_true (numpy.array): true labels for dataset - not used by CDBD
+            y_pred (numpy.array): predicted labels for dataset - not used by CDBD
         """
 
-        super().set_reference(reference_batch)
+        super().set_reference(X, y_true, y_pred)
 
         # Ensure only being used with 1 variable in reference
         if self._num_features != 1:
             raise ValueError("CDBD should only be used to monitor 1 variable")
 
-    def update(self, test_batch):
+    def update(self, X, y_true=None, y_pred=None):
         """
         Update the detector with a new test batch. If drift is detected, new
         reference batch becomes most recent test batch. If drift is not
         detected, reference batch is updated to include most recent test batch.
 
         Args:
-          test_batch (DataFrame): next batch of data to detect drift on.
+          X (DataFrame): next batch of data to detect drift on.
+          y_true (numpy.ndarray): true labels of next batch - not used in HDDDM
+          y_pred (numpy.ndarray): predicted labels of next batch - not used in HDDDM
         """
 
         # Ensure only being used with 1 variable in test
-        if test_batch.shape[1] != 1:
+        if X.shape[1] != 1:
             raise ValueError("CDBD should only be used to monitor 1 variable")
 
-        super().update(test_batch)
+        super().update(X, y_true, y_pred)
 
     def reset(self):
         """
diff --git a/src/menelaus/data_drift/hdddm.py b/src/menelaus/data_drift/hdddm.py
index 11034edb..f5193c3c 100644
--- a/src/menelaus/data_drift/hdddm.py
+++ b/src/menelaus/data_drift/hdddm.py
@@ -219,33 +219,37 @@ def __init__(
             subsets=subsets,
         )
 
-    def set_reference(self, reference_batch):
+    def set_reference(self, X, y_true=None, y_pred=None):
         """
         Initialize detector with a reference batch. After drift, reference batch is
         automatically set to most recent test batch. Option for user to specify
         alternative reference batch using this method.
 
         Args:
-            reference_batch (DataFrame): initial baseline dataset
+            X (pandas.DataFrame): initial baseline dataset
+            y_true (numpy.array): true labels for dataset - not used by HDDDM
+            y_pred (numpy.array): predicted labels for dataset - not used by HDDDM
         """
 
-        super().set_reference(reference_batch)
+        super().set_reference(X, y_true, y_pred)
 
-    def update(self, test_batch):
+    def update(self, X, y_true=None, y_pred=None):
         """
         Update the detector with a new test batch. If drift is detected, new
         reference batch becomes most recent test batch. If drift is not
         detected, reference batch is updated to include most recent test batch.
 
         Args:
-          test_batch (DataFrame): next batch of data to detect drift on.
+          X (DataFrame): next batch of data to detect drift on.
+          y_true (numpy.ndarray): true labels of next batch - not used in HDDDM
+          y_pred (numpy.ndarray): predicted labels of next batch - not used in HDDDM
         """
 
         # Ensure batch datasets have equivalent features
-        if not self._num_features == test_batch.shape[1]:
+        if not self._num_features == X.shape[1]:
             raise ValueError("Batch features must match")
 
-        super().update(test_batch)
+        super().update(X, y_true, y_pred)
 
     def reset(self):
         """
diff --git a/src/menelaus/data_drift/histogram_density_method.py b/src/menelaus/data_drift/histogram_density_method.py
index a4789dcb..e5e20697 100644
--- a/src/menelaus/data_drift/histogram_density_method.py
+++ b/src/menelaus/data_drift/histogram_density_method.py
@@ -241,22 +241,24 @@ def __init__(
         self.epsilon_values = {}
         self.thresholds = {}
 
-    def set_reference(self, reference_batch):
+    def set_reference(self, X, y_true=None, y_pred=None):
         """
         Initialize detector with a reference batch. After drift, reference batch is
         automatically set to most recent test batch. Option for user to specify
         alternative reference batch using this method.
 
         Args:
-            reference_batch (DataFrame): initial baseline dataset
+            X (pandas.DataFrame): initial baseline dataset
+            y_true (numpy.array): true labels for dataset - not used by HDM
+            y_pred (numpy.array): predicted labels for dataset - not used by HDM
         """
 
         # Initialize attributes
-        self.reference = reference_batch
+        self.reference = X
         self._num_features = self.reference.shape[1]
         self.reset()
 
-    def update(self, test_batch):
+    def update(self, X, y_true=None, y_pred=None):
 
         """
         Update the detector with a new test batch. If drift is detected, new
@@ -264,25 +266,27 @@ def update(self, test_batch):
         detected, reference batch is updated to include most recent test batch.
 
         Args:
-          test_batch (DataFrame): next batch of data to detect drift on.
+            X (DataFrame): next batch of data to detect drift on.
+            y_true (numpy.ndarray): true labels of next batch - not used in HDM
+            y_pred (numpy.ndarray): predicted labels of next batch - not used in HDM
         """
 
         if self._drift_state == "drift":
             self.reset()
 
-        super().update()
-        test_n = test_batch.shape[0]
+        super().update(X, y_true, y_pred)
+        test_n = X.shape[0]
 
         # Estimate reference and test histograms
         mins = []
         maxes = []
         for f in range(self._num_features):
             reference_variable = self.reference.iloc[:, f]
-            test_variable = test_batch.iloc[:, f]
+            test_variable = X.iloc[:, f]
             mins.append(np.concatenate((reference_variable, test_variable)).min())
             maxes.append(np.concatenate((reference_variable, test_variable)).max())
         self._reference_density = self._build_histograms(self.reference, mins, maxes)
-        test_density = self._build_histograms(test_batch, mins, maxes)
+        test_density = self._build_histograms(X, mins, maxes)
 
         # Divergence metric
         total_distance = 0
@@ -338,13 +342,13 @@ def update(self, test_batch):
                         }
 
                     self._drift_state = "drift"
-                    self.reference = test_batch
+                    self.reference = X
                     self._lambda = self.total_updates
 
         if self._drift_state != "drift":
             self._prev_distance = self.current_distance
             self._prev_feature_distances = feature_distances
-            self.reference = pd.concat([self.reference, test_batch])
+            self.reference = pd.concat([self.reference, X])
             self.reference_n = self.reference.shape[0]
             # number of bins for histogram, from reference batch
             self._bins = int(np.floor(np.sqrt(self.reference_n)))
diff --git a/src/menelaus/data_drift/kdq_tree.py b/src/menelaus/data_drift/kdq_tree.py
index 36e7f590..e31a333c 100644
--- a/src/menelaus/data_drift/kdq_tree.py
+++ b/src/menelaus/data_drift/kdq_tree.py
@@ -396,7 +396,7 @@ def reset(self):
         KdqTreeDetector.reset(self)
         self._drift_counter = 0  # samples consecutively in the drift region
 
-    def update(self, data):
+    def update(self, X, y_true=None, y_pred=None):
         """
         Update the detector with a new sample point. Constructs the reference
         data's kdqtree; then, when sufficient samples have been received, puts
@@ -408,16 +408,18 @@ def update(self, data):
         reference windows will be constructed once sufficient samples are received.
 
         Args:
-            data (pandas.DataFrame or numpy array): If just reset/initialized,
+            X (pandas.DataFrame or numpy array): If just reset/initialized,
             the reference data. Otherwise, a new sample to put into the test
             window.
+            y_true (numpy.ndarray): true labels of input data - not used in KdqTree
+            y_pred (numpy.ndarray): predicted labels of input data - not used in KdqTree
         """
-        ary = self._prepare_data(data)
+        ary = self._prepare_data(X)
 
         if self.drift_state == "drift":
             self.reset()
 
-        StreamingDetector.update(self)
+        StreamingDetector.update(self, X, y_true, y_pred)
         KdqTreeDetector._evaluate_kdqtree(self, ary, "stream")
 
 
@@ -513,28 +515,30 @@ def reset(self):
         BatchDetector.reset(self)
         KdqTreeDetector.reset(self)
 
-    def set_reference(self, data):
+    def set_reference(self, X, y_true=None, y_pred=None):
         """
         Initialize detector with a reference batch. The user may specify an
         alternate reference batch than the one maintained by kdq-Tree. This will
         reset the detector.
 
         Args:
-            data (pandas.DataFrame or numpy array): baseline dataset
+            X (pandas.DataFrame or numpy.array): baseline dataset
+            y_true (numpy.array): actual labels of dataset - not used in KdqTree
+            y_pred (numpy.array): predicted labels of dataset - not used in KdqTree
         """
-        if isinstance(data, pd.DataFrame):
+        if isinstance(X, pd.DataFrame):
             # XXX - notice how inner_set calling KLD requires us to continue
             #       branching on input_type, which is not ideal - Anmol Srivastava
-            self._inner_set_reference(data.values, input_type="batch")
-            self.input_cols = data.columns
-        elif isinstance(data, np.ndarray):
-            self._inner_set_reference(data, input_type="batch")
+            self._inner_set_reference(X.values, input_type="batch")
+            self.input_cols = X.columns
+        elif isinstance(X, np.ndarray):
+            self._inner_set_reference(X, input_type="batch")
         else:
             raise ValueError(
                 "This method is only available for data inputs in the form of a Pandas DataFrame or a Numpy Array."
             )
 
-    def update(self, data):
+    def update(self, X, y_true=None, y_pred=None):
         """
         Update the detector with a new batch. Constructs the reference
         data's kdqtree; then, when sufficient samples have been received, puts
@@ -547,18 +551,20 @@ def update(self, data):
         batches to ``update``.
 
         Args:
-            data (pandas.DataFrame or numpy array): If just reset/initialized,
+            X (pandas.DataFrame or numpy array): If just reset/initialized,
             the reference data. Otherwise, a new batch of data to be compared
             to the reference window.
+            y_true (numpy.ndarray): true labels of input data - not used in KdqTree
+            y_pred (numpy.ndarray): predicted labels of input data - not used in KdqTree
         """
-        ary = self._prepare_data(data)
+        ary = self._prepare_data(X)
 
         if self.drift_state == "drift":
             self.set_reference(self.ref_data)
 
-        BatchDetector.update(self)
+        BatchDetector.update(self, X, y_true, y_pred)
         KdqTreeDetector._evaluate_kdqtree(self, ary, "batch")
 
         # if _evaluate_kdqtree resulted in drift for batch data, do a redundant check
-        if self.drift_state == "drift" and isinstance(data, pd.DataFrame):
-            self.input_cols = data.columns
+        if self.drift_state == "drift" and isinstance(X, pd.DataFrame):
+            self.input_cols = X.columns
diff --git a/src/menelaus/data_drift/pca_cd.py b/src/menelaus/data_drift/pca_cd.py
index eae3c5ae..2b6804bb 100644
--- a/src/menelaus/data_drift/pca_cd.py
+++ b/src/menelaus/data_drift/pca_cd.py
@@ -42,7 +42,7 @@ class PCACD(DriftDetector):
             the specified ``ev_threshold`` parameter.
     """
 
-    input_type = "batch"
+    input_type = "streaming"
 
     def __init__(
         self,
@@ -115,11 +115,13 @@ def __init__(
         self._density_reference = {}
         self._change_score = [0]
 
-    def update(self, next_obs):
+    def update(self, X, y_true=None, y_pred=None):
         """Update the detector with a new observation.
 
         Args:
-            next_obs: next observation, as a ``pandas`` ``Series``
+            X (numpy.ndarray): next observation, as a ``pandas`` ``Series``
+            y_true (numpy.ndarray): true label of observation - not used in PCACD
+            y_pred (numpy.ndarray): predicted label of observation - not used in PCACD
         """
 
         if self._build_reference_and_test:
@@ -138,13 +140,11 @@ def update(self, next_obs):
 
             elif len(self._reference_window) < self.window_size:
                 self._reference_window = pd.concat(
-                    [self._reference_window, pd.DataFrame(next_obs)]
+                    [self._reference_window, pd.DataFrame(X)]
                 )
 
             elif len(self._test_window) < self.window_size:
-                self._test_window = pd.concat(
-                    [self._test_window, pd.DataFrame(next_obs)]
-                )
+                self._test_window = pd.concat([self._test_window, pd.DataFrame(X)])
 
             if len(self._test_window) == self.window_size:
                 self._build_reference_and_test = False
@@ -204,7 +204,7 @@ def update(self, next_obs):
 
             # Add new obs to test window
             if self.online_scaling is True:
-                next_obs = pd.DataFrame(self._reference_scaler.transform(next_obs))
+                next_obs = pd.DataFrame(self._reference_scaler.transform(X))
             self._test_window = pd.concat([self._test_window.iloc[1:, :], next_obs])
 
             # Project new observation onto PCs
@@ -271,15 +271,13 @@ def update(self, next_obs):
                 change_score = max(change_scores)
                 self._change_score.append(change_score)
 
-                self._drift_detection_monitor.update(
-                    next_obs=change_score, obs_id=next_obs.index.values[0]
-                )
+                self._drift_detection_monitor.update(X=change_score)
 
                 if self._drift_detection_monitor.drift_state is not None:
                     self._build_reference_and_test = True
                     self.drift_state = "drift"
 
-        super().update()
+        super().update(X, y_true, y_pred)
 
     def reset(self):
         """Initialize the detector's drift state and other relevant attributes.
diff --git a/src/menelaus/drift_detector.py b/src/menelaus/drift_detector.py
index a62bc98c..9268d4f5 100644
--- a/src/menelaus/drift_detector.py
+++ b/src/menelaus/drift_detector.py
@@ -20,8 +20,15 @@ def __init__(self, *args, **kwargs):
         self._drift_state = None
 
     @abstractmethod
-    def update(self, *args, **kwargs):
-        """Update detector with new sample (data point)."""
+    def update(self, X, y_true, y_pred):
+        """
+        Update detector with new sample (data point).
+
+        Args:
+            X (numpy.ndarray): input data
+            y_true (numpy.ndarray): if applicable, true labels of input data
+            y_pred (numpy.ndarray): if applicable, predicted labels of input data
+        """
         self.total_samples += 1
         self.samples_since_reset += 1
 
@@ -74,11 +81,30 @@ def __init__(self, *args, **kwargs):
         self._drift_state = None
 
     @abstractmethod
-    def update(self, *args, **kwargs):
-        """Update detector with new batch of data"""
+    def update(self, X, y_true, y_pred):
+        """
+        Update detector with new batch of data
+
+        Args:
+            X (numpy.ndarray): input data
+            y_true (numpy.ndarray): if applicable, true labels of input data
+            y_pred (numpy.ndarray): if applicable, predicted labels of input data
+        """
         self.total_batches += 1
         self.batches_since_reset += 1
 
+    @abstractmethod
+    def set_reference(self, X, y_true, y_pred):
+        """
+        Initialize detector with a reference batch.
+
+        Args:
+            X (pandas.DataFrame or numpy.array): baseline dataset
+            y_true (numpy.array): actual labels of dataset
+            y_pred (numpy.array): predicted labels of dataset
+        """
+        raise NotImplementedError
+
     @abstractmethod
     def reset(self, *args, **kwargs):
         """
@@ -142,8 +168,15 @@ def __init__(self, *args, **kwargs):
         self._input_type = None
 
     @abstractmethod
-    def update(self, *args, **kwargs):
-        """Update the detector with a new sample or batch."""
+    def update(self, X, y_true, y_pred):
+        """
+        Update the detector with a new sample or batch.
+
+        Args:
+            X (numpy.ndarray): input data
+            y_true (numpy.ndarray): if applicable, true labels of input data
+            y_pred (numpy.ndarray): if applicable, predicted labels of input data
+        """
         self.total_updates += 1
         self.updates_since_reset += 1
 
diff --git a/src/menelaus/utils/_locate.py b/src/menelaus/utils/_locate.py
index 28a8959b..2c8e0e24 100644
--- a/src/menelaus/utils/_locate.py
+++ b/src/menelaus/utils/_locate.py
@@ -12,9 +12,6 @@
 def find_git_root(search_dirs=(".git",)):
     """Find the root directory for the git repo, so that we don't have to
     fool with strange filepaths.
-
-    This is a strange place to have this utility function, but we don't have any
-    other such functions at the moment.
     """
     test_dir = os.getcwd()
     prev_dir, test_dir = None, os.path.abspath(test_dir)


Test patch:
diff --git a/tests/menelaus/test_adwin.py b/tests/menelaus/test_adwin.py
index b4a9522b..02d104a5 100644
--- a/tests/menelaus/test_adwin.py
+++ b/tests/menelaus/test_adwin.py
@@ -10,9 +10,9 @@ def test_compression():
     max_buckets = 1
     det = ADWIN(max_buckets=max_buckets, delta=0.000001)
     n_samples = 10
-    for element in [1, 2, 3]:
+    for element in [(1, 0), (0, 0), (1, 1)]:
         for _ in range(n_samples):
-            det.update(element)
+            det.update(y_true=element[0], y_pred=element[1])
     curr = det._bucket_row_list.head
 
     overflow = False
@@ -35,7 +35,7 @@ def test_mean():
     assert det.mean() == 0
     n_samples = 5
     for _ in range(n_samples):
-        det.update(1)
+        det.update(y_true=1, y_pred=1)
     assert det.mean() == 1
 
 
@@ -45,8 +45,8 @@ def test_variance():
     assert det.variance() == 0
     n_samples = 10
     for _ in range(n_samples):
-        det.update(1)
-        det.update(0)
+        det.update(y_true=1, y_pred=1)
+        det.update(y_true=0, y_pred=1)
     assert det.variance() == 0.25
 
 
@@ -56,13 +56,13 @@ def test_drift():
     - ADWIN.retraining_recs takes on the proper values before and after drift
     """
     det = ADWIN(new_sample_thresh=2)
-    n_samples = 20
+    n_samples = 50
     for _ in range(n_samples):
-        det.update(0.1)
+        det.update(y_true=1, y_pred=1)
 
     drift_found = False
     for _ in range(n_samples):
-        det.update(100)
+        det.update(y_true=1, y_pred=0)
         drift_found = (det.drift_state == "drift") or drift_found
         if drift_found:
             break
@@ -73,7 +73,7 @@ def test_drift():
     assert det.retraining_recs is not None
 
     # do we reset properly?
-    det.update(100)
+    det.update(y_true=1, y_pred=0)
     assert det.drift_state is None
     assert det.retraining_recs == [None, None]
 
@@ -83,13 +83,13 @@ def test_conservative_bound():
     very drifting stream.
     """
     det = ADWIN(new_sample_thresh=2, conservative_bound=True)
-    n_samples = 20
+    n_samples = 50
     for _ in range(n_samples):
-        det.update(0.1)
+        det.update(y_true=1, y_pred=1)
 
     drift_found = False
     for _ in range(n_samples):
-        det.update(100)
+        det.update(y_true=1, y_pred=0)
         drift_found = (det.drift_state == "drift") or drift_found
         if drift_found:
             break
diff --git a/tests/menelaus/test_ddm.py b/tests/menelaus/test_ddm.py
index fec92b19..f33e57f9 100644
--- a/tests/menelaus/test_ddm.py
+++ b/tests/menelaus/test_ddm.py
@@ -26,7 +26,7 @@ def test_warning():
 
     # enter the "warning" region
     for _ in range(n_error_samples):
-        det.update(1, 0)
+        det.update(y_true=0, y_pred=1)
     assert det.drift_state == "warning"
 
     # leave the "warning" region
diff --git a/tests/menelaus/test_drift_detector.py b/tests/menelaus/test_drift_detector.py
index 7e0c57dd..e3a37eae 100644
--- a/tests/menelaus/test_drift_detector.py
+++ b/tests/menelaus/test_drift_detector.py
@@ -9,6 +9,12 @@
 BatchDetector.__abstractmethods__ = set()
 
 
+def test_set_reference_batch():
+    '''Ensure error on call to set_reference from batch ABC'''
+    det = BatchDetector()
+    with pytest.raises(NotImplementedError) as _:
+        det.set_reference(None, None, None)
+
 def test_validation_streaming():
     '''Ensure ValueError on invalid drift_state for StreamingDetector'''
     det = StreamingDetector()
diff --git a/tests/menelaus/test_eddm.py b/tests/menelaus/test_eddm.py
index cd930938..0959a7a9 100644
--- a/tests/menelaus/test_eddm.py
+++ b/tests/menelaus/test_eddm.py
@@ -11,11 +11,11 @@ def test_drift():
         det.update(1, 1)
 
     for _ in range(n_error_samples):
-        det.update(1, 0)
+        det.update(y_true=0, y_pred=1)
 
     # drift and reset
     assert det.drift_state == "drift"
-    det.update(1, 0)
+    det.update(y_true=0, y_pred=1)
     assert det.drift_state is None
 
 
@@ -29,12 +29,12 @@ def test_warning():
 
     # enter warning region
     for _ in range(n_error_samples):
-        det.update(1, 0)
+        det.update(y_true=0, y_pred=1)
     assert det.drift_state == "warning"
 
     # increase distance between errors, leave warning region
     det.update(1, 1)
-    det.update(0, 1)
+    det.update(y_true=1, y_pred=0)
     assert det.drift_state is None
 
 
diff --git a/tests/menelaus/test_lfr.py b/tests/menelaus/test_lfr.py
index e3463742..414a667e 100644
--- a/tests/menelaus/test_lfr.py
+++ b/tests/menelaus/test_lfr.py
@@ -12,10 +12,10 @@ def test_build_no_drift():
     detParallel = LinearFourRates(burn_in=burn_in, num_mc=15, parallelize=True)
     for _ in range(burn_in + 50):
         # values that ostensibly should not indicate drift
-        det.update(y_pred=1, y_true=1)
-        det.update(y_pred=0, y_true=1)
-        det.update(y_pred=1, y_true=0)
-        det.update(y_pred=0, y_true=0)
+        det.update(y_true=1, y_pred=1)
+        det.update(y_true=1, y_pred=0)
+        det.update(y_true=0, y_pred=1)
+        det.update(y_true=0, y_pred=0)
         assert det.drift_state is None
         assert det.retraining_recs == [None, None]
 
@@ -53,30 +53,29 @@ def test_build_with_warning_drift():
     # have warning and drift significance levels be spread out
     for i in range(burn_in + 20):
         if i < burn_in:
-            #det.update(1,np.random.binomial(n=1,p=0.6))
-            det.update(1,1)
+            det.update(y_true=1, y_pred=1)
             assert det.drift_state is None
             assert det.retraining_recs == [None, None]
             np.random.seed(123)
-            detParallel.update(1,1)
+            detParallel.update(y_true=1, y_pred=1) # x=1, y_true=1
             assert detParallel.drift_state is None
             assert detParallel.retraining_recs == [None, None]
         elif i == burn_in: # get this to be warning
-            det.update(1,1)
+            det.update(y_true=1, y_pred=1)
             assert det.drift_state == "warning"
             np.random.seed(123)
-            detParallel.update(1,1)
+            detParallel.update(y_true=1, y_pred=1)
             assert detParallel.drift_state == "warning"
         elif i == burn_in + 1: # get this to be drift
-            det.update(1,0)
+            det.update(y_true=0, y_pred=1)
             assert det.drift_state == "drift"
             np.random.seed(123)
-            detParallel.update(1,0)
+            detParallel.update(y_true=0, y_pred=1)
             assert detParallel.drift_state == "drift"
         else:
-            det.update(1,1)
+            det.update(y_true=1, y_pred=1)
             np.random.seed(123)
-            detParallel.update(1,1)
+            detParallel.update(y_true=1, y_pred=1)
 
 
 # need a test that goes straight from no drift to drift (bypassing warning)
@@ -90,8 +89,8 @@ def test_build_with_drift():
         burn_in=burn_in, num_mc=1, detect_level=0.5, parallelize=True # get it to drift asap
     ) 
     for i in range(burn_in + 1):
-        det.update(1, 1)
-        detParallel.update(1, 1)
+        det.update(y_true=1, y_pred=1)
+        detParallel.update(y_true=1, y_pred=1)
         if i >= burn_in:
             assert det.drift_state == "drift"
             assert det._retraining_recs == [burn_in,burn_in]
@@ -107,7 +106,7 @@ def test_drift_no_rates_tracked():
         burn_in=burn_in, num_mc=1, detect_level=0.5, rates_tracked=[] # get it to drift asap
     )
     for i in range(burn_in + 1):
-        det.update(1, 1)
+        det.update(y_true=1, y_pred=1)
         if i >= burn_in:
             assert det.drift_state is None
             assert det._retraining_recs == [None, None]
@@ -189,19 +188,19 @@ def test_drift_rate_dependence():
             detNPV.update(1,1)
             assert detNPV.drift_state is None
         elif i == burn_in + 1: # get this to be drift
-            detTPR.update(1,0)
+            detTPR.update(y_true=0, y_pred=1)
             assert detTPR.drift_state == "warning"
 
             np.random.seed(123)
-            detTNR.update(1,0)
+            detTNR.update(y_true=0, y_pred=1)
             assert detTNR.drift_state == "warning"
 
             np.random.seed(123)
-            detPPV.update(1,0)
+            detPPV.update(y_true=0, y_pred=1)
             assert detPPV.drift_state == "drift"
 
             np.random.seed(123)
-            detNPV.update(1,0)
+            detNPV.update(y_true=0, y_pred=1)
             assert detNPV.drift_state is None
         else:
             detTPR.update(1,1)
diff --git a/tests/menelaus/test_md3.py b/tests/menelaus/test_md3.py
index 9d093d14..8a0f5454 100644
--- a/tests/menelaus/test_md3.py
+++ b/tests/menelaus/test_md3.py
@@ -29,7 +29,7 @@ def test_set_reference():
     clf = svm.SVC(kernel='linear')
     clf.fit(X_train, y_train)
     det = MD3(clf=clf, k=2)
-    det.set_reference(df, 'y')
+    det.set_reference(df, target_name='y')
     assert X_train.equals(det.reference_batch_features)
     assert df.loc[:, df.columns == 'y'].equals(det.reference_batch_target)
     assert det.reference_distribution["len"] == len(data)
@@ -52,7 +52,7 @@ def test_update():
     clf = svm.SVC(kernel='linear')
     clf.fit(X_train, y_train)
     det = MD3(clf=clf, k=2)
-    det.set_reference(df, 'y')
+    det.set_reference(df, target_name='y')
     
     data_update1 = [[-4, 8], [3, 4]]
     df_update1 = pd.DataFrame(data_update1, columns=['var1', 'var2'])
@@ -80,7 +80,7 @@ def test_give_oracle_label():
     clf = svm.SVC(kernel='linear')
     clf.fit(X_train, y_train)
     det = MD3(clf=clf, sensitivity=0.5, k=3)
-    det.set_reference(df, 'y')
+    det.set_reference(df, target_name='y')
     
     labeled_sample = [[1, 2, 0]]
     labeled_df = pd.DataFrame(labeled_sample, columns=['var1', 'var2', 'y'])