Repository: JacksonBurns/astartes (language: python)

Issue:
Constant Integration for Example Notebook
After the example Jupyter notebook is added, add a GitHub action to do CI on it in the same manner as [this action](https://github.com/JacksonBurns/ArClOTs/blob/main/.github/workflows/check_execution.yml). 
 - [x] #31 
Add Green Group Reaction dataset Example Notebook
Edit the example Jupyter notebook, demonstrating the efficacy of non-random (deterministic) splitting.


Discussion / hints:

Fix patch:
diff --git a/.github/workflows/ipynb_ci.yml b/.github/workflows/ipynb_ci.yml
new file mode 100644
index 0000000..d64ed97
--- /dev/null
+++ b/.github/workflows/ipynb_ci.yml
@@ -0,0 +1,40 @@
+name: Check Jupyer Notebook Execution
+on:
+  push:
+    branches: [main]
+  pull_request:
+    branches: [main]
+
+  workflow_dispatch:
+
+jobs:
+  build:
+    strategy:
+      fail-fast: false
+      matrix:
+        nb-file:
+          ["RDB7_barrier_prediction_example", "train_val_test_split_example"]
+    runs-on: ubuntu-latest
+    defaults:
+      run:
+        shell: bash -el {0}
+    name: Check ${{ matrix.nb-file }} Notebook Execution
+    steps:
+      - uses: actions/checkout@v3
+      - uses: mamba-org/provision-with-micromamba@main
+        with:
+          environment-file: false
+          environment-name: temp
+          channels: defaults,conda-forge
+          channel-priority: flexible
+          extra-specs: |
+            python=3.11
+      - name: Install dependencies
+        run: |
+          python -m pip install -e .[molecules]
+          python -m pip install notebook
+      - name: Test Execution
+        run: |
+          cd examples
+          jupyter nbconvert --to script ${{ matrix.nb-file }}.ipynb
+          ipython ${{ matrix.nb-file }}.py
diff --git a/README.md b/README.md
index 3fc41dc..ecd7e3f 100644
--- a/README.md
+++ b/README.md
@@ -35,10 +35,19 @@ X_train, X_test, y_train, y_test = train_test_split(
 )
 ```
 
+### Example Notebooks
+
+Click the badges in the table below to be taken to a live, interactive demo of `astartes`:
+
+| Demo | Link |
+|:---:|---|
+| Using `train_val_test_split` with the `sklearn` example datasets | [![Binder](https://mybinder.org/badge_logo.svg)](https://mybinder.org/v2/gh/JacksonBurns/astartes/main?labpath=examples%2Ftrain_val_test_split_example.ipynb) |
+| Cheminformatics sample set partitioning with `astartes` | [![Binder](https://mybinder.org/badge_logo.svg)](https://mybinder.org/v2/gh/JacksonBurns/astartes/main?labpath=examples%2FRDB7_barrier_prediction_example.ipynb) |
+
 ### Rational Splitting Algorithms
-While much machine learning is done with a random choice between training/test/validation data, an alternative is the use of so-called "rational" splitting algorithms. These approaches use some similarity-based algorithm to divide data into sets. Some of these algorithms include Kennard-Stone, minimal test set dissimilarity, and sphere exclusion algorithms [as discussed by Tropsha et. al](https://pubs.acs.org/doi/pdf/10.1021/ci300338w) as well as the DUPLEX, OptiSim, D-optimal, as discussed in [Applied Chemoinformatics: Achievements and Future Opportunities](https://www.wiley.com/en-us/Applied+Chemoinformatics%3A+Achievements+and+Future+Opportunities-p-9783527806546). Some clustering-based splitting techniques have also been introduced, such as [DBSCAN](http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.1016.890&rep=rep1&type=pdf).
+While much machine learning is done with a random choice between training/test/validation data, an alternative is the use of so-called "rational" splitting algorithms. These approaches use some similarity-based algorithm to divide data into sets. Some of these algorithms include Kennard-Stone, minimal test set dissimilarity, and sphere exclusion algorithms [as discussed by Tropsha et. al](https://pubs.acs.org/doi/pdf/10.1021/ci300338w) as well as the OptiSim as discussed in [Applied Chemoinformatics: Achievements and Future Opportunities](https://www.wiley.com/en-us/Applied+Chemoinformatics%3A+Achievements+and+Future+Opportunities-p-9783527806546). Some clustering-based splitting techniques have also been introduced, such as [DBSCAN](http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.1016.890&rep=rep1&type=pdf).
 
-There are two broad categories of sampling algorithms implemented in `astartes`: extrapolative and interpolative. The former will force your model to predict on out-of-smaple data, effectively asking a 'harder question' than interpolative sampling.
+There are two broad categories of sampling algorithms implemented in `astartes`: extrapolative and interpolative. The former will force your model to predict on out-of-smaple data, effectively asking a 'harder question' than interpolative sampling. See the table below for all of the sampling approaches currently implemented in `astartes`.
 
 #### Implemented Sampling Algorithms
 
diff --git a/examples/astartes_example.ipynb b/examples/astartes_example.ipynb
deleted file mode 100644
index fae714b..0000000
--- a/examples/astartes_example.ipynb
+++ /dev/null
@@ -1,33 +0,0 @@
-{
- "cells": [
-  {
-   "cell_type": "code",
-   "execution_count": null,
-   "id": "c71ab290",
-   "metadata": {},
-   "outputs": [],
-   "source": []
-  }
- ],
- "metadata": {
-  "kernelspec": {
-   "display_name": "Python 3 (ipykernel)",
-   "language": "python",
-   "name": "python3"
-  },
-  "language_info": {
-   "codemirror_mode": {
-    "name": "ipython",
-    "version": 3
-   },
-   "file_extension": ".py",
-   "mimetype": "text/x-python",
-   "name": "python",
-   "nbconvert_exporter": "python",
-   "pygments_lexer": "ipython3",
-   "version": "3.7.0"
-  }
- },
- "nbformat": 4,
- "nbformat_minor": 5
-}
diff --git a/postBuild b/postBuild
new file mode 100644
index 0000000..3c6e86c
--- /dev/null
+++ b/postBuild
@@ -0,0 +1,9 @@
+#!/bin/bash
+
+# This file is needed to make the binder demonstrations work
+# because we use a pyproject.toml file, Binder won't build
+# the environment unless we manually specify it here
+
+set -e
+# Work-around until pip stops making this broken in requirement.txt
+pip install -e .[molecules]
\ No newline at end of file


Test patch:
diff --git a/examples/train_val_test_split_example.ipynb b/examples/train_val_test_split_example.ipynb
new file mode 100644
index 0000000..c83cbdf
--- /dev/null
+++ b/examples/train_val_test_split_example.ipynb
@@ -0,0 +1,274 @@
+{
+ "cells": [
+  {
+   "attachments": {},
+   "cell_type": "markdown",
+   "id": "9feb8e56",
+   "metadata": {},
+   "source": [
+    "# Using `train_val_test_split` for Rigorous Modeling\n",
+    "Thanks to [this](https://towardsdatascience.com/automatic-hyperparameter-tuning-with-sklearn-gridsearchcv-and-randomizedsearchcv-e94f53a518ee) article for providing some code used below in the automatic hyperparameter tuning.\n",
+    "\n",
+    "First, we load example data from sklearn into our X and y arrays, where X are the features and y is the response, aka target."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 1,
+   "id": "91321870",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "from sklearn import datasets\n",
+    "diabetes_X, diabetes_y = datasets.load_diabetes(return_X_y=True)"
+   ]
+  },
+  {
+   "attachments": {},
+   "cell_type": "markdown",
+   "id": "e884e357",
+   "metadata": {},
+   "source": [
+    "Next we split the data into training, testing, and validation sets. We will use the training and validation set to develop our model, and then only use the test set at the very end to see how well the tuning worked.\n",
+    "\n",
+    "We will stick with random sampling for this example, which we can specify by adding `sampler=\"random\"` to the call to `train_val_test_split` (`'random'` is also the default option). There are many other samplers available -- checkout [this documentation](https://github.com/JacksonBurns/astartes#implemented-sampling-algorithms) for a complete table."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 2,
+   "id": "be4de1fa",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "from astartes import train_val_test_split\n",
+    "X_train, X_val, X_test, y_train, y_val, y_test = train_val_test_split(\n",
+    "    diabetes_X,\n",
+    "    diabetes_y,\n",
+    "    sampler=\"random\",\n",
+    "    hopts = {\n",
+    "        \"random_state\": 42,\n",
+    "    }\n",
+    ")"
+   ]
+  },
+  {
+   "attachments": {},
+   "cell_type": "markdown",
+   "id": "0c288eb8",
+   "metadata": {},
+   "source": [
+    "By default, `train_val_test_split` will use a validation size of 0.1 (10% of the dataset), test size of 0.1, and train size of 0.8. You can override these with `val_size`, `test_size`, and `train_size` keyword arguments.\n",
+    "\n",
+    "Now we create a baseline model without tuning it for better performance on our data:"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 3,
+   "id": "6b73362c",
+   "metadata": {},
+   "outputs": [
+    {
+     "data": {
+      "text/html": [
+       "<style>#sk-container-id-1 {color: black;background-color: white;}#sk-container-id-1 pre{padding: 0;}#sk-container-id-1 div.sk-toggleable {background-color: white;}#sk-container-id-1 label.sk-toggleable__label {cursor: pointer;display: block;width: 100%;margin-bottom: 0;padding: 0.3em;box-sizing: border-box;text-align: center;}#sk-container-id-1 label.sk-toggleable__label-arrow:before {content: \"▸\";float: left;margin-right: 0.25em;color: #696969;}#sk-container-id-1 label.sk-toggleable__label-arrow:hover:before {color: black;}#sk-container-id-1 div.sk-estimator:hover label.sk-toggleable__label-arrow:before {color: black;}#sk-container-id-1 div.sk-toggleable__content {max-height: 0;max-width: 0;overflow: hidden;text-align: left;background-color: #f0f8ff;}#sk-container-id-1 div.sk-toggleable__content pre {margin: 0.2em;color: black;border-radius: 0.25em;background-color: #f0f8ff;}#sk-container-id-1 input.sk-toggleable__control:checked~div.sk-toggleable__content {max-height: 200px;max-width: 100%;overflow: auto;}#sk-container-id-1 input.sk-toggleable__control:checked~label.sk-toggleable__label-arrow:before {content: \"▾\";}#sk-container-id-1 div.sk-estimator input.sk-toggleable__control:checked~label.sk-toggleable__label {background-color: #d4ebff;}#sk-container-id-1 div.sk-label input.sk-toggleable__control:checked~label.sk-toggleable__label {background-color: #d4ebff;}#sk-container-id-1 input.sk-hidden--visually {border: 0;clip: rect(1px 1px 1px 1px);clip: rect(1px, 1px, 1px, 1px);height: 1px;margin: -1px;overflow: hidden;padding: 0;position: absolute;width: 1px;}#sk-container-id-1 div.sk-estimator {font-family: monospace;background-color: #f0f8ff;border: 1px dotted black;border-radius: 0.25em;box-sizing: border-box;margin-bottom: 0.5em;}#sk-container-id-1 div.sk-estimator:hover {background-color: #d4ebff;}#sk-container-id-1 div.sk-parallel-item::after {content: \"\";width: 100%;border-bottom: 1px solid gray;flex-grow: 1;}#sk-container-id-1 div.sk-label:hover label.sk-toggleable__label {background-color: #d4ebff;}#sk-container-id-1 div.sk-serial::before {content: \"\";position: absolute;border-left: 1px solid gray;box-sizing: border-box;top: 0;bottom: 0;left: 50%;z-index: 0;}#sk-container-id-1 div.sk-serial {display: flex;flex-direction: column;align-items: center;background-color: white;padding-right: 0.2em;padding-left: 0.2em;position: relative;}#sk-container-id-1 div.sk-item {position: relative;z-index: 1;}#sk-container-id-1 div.sk-parallel {display: flex;align-items: stretch;justify-content: center;background-color: white;position: relative;}#sk-container-id-1 div.sk-item::before, #sk-container-id-1 div.sk-parallel-item::before {content: \"\";position: absolute;border-left: 1px solid gray;box-sizing: border-box;top: 0;bottom: 0;left: 50%;z-index: -1;}#sk-container-id-1 div.sk-parallel-item {display: flex;flex-direction: column;z-index: 1;position: relative;background-color: white;}#sk-container-id-1 div.sk-parallel-item:first-child::after {align-self: flex-end;width: 50%;}#sk-container-id-1 div.sk-parallel-item:last-child::after {align-self: flex-start;width: 50%;}#sk-container-id-1 div.sk-parallel-item:only-child::after {width: 0;}#sk-container-id-1 div.sk-dashed-wrapped {border: 1px dashed gray;margin: 0 0.4em 0.5em 0.4em;box-sizing: border-box;padding-bottom: 0.4em;background-color: white;}#sk-container-id-1 div.sk-label label {font-family: monospace;font-weight: bold;display: inline-block;line-height: 1.2em;}#sk-container-id-1 div.sk-label-container {text-align: center;}#sk-container-id-1 div.sk-container {/* jupyter's `normalize.less` sets `[hidden] { display: none; }` but bootstrap.min.css set `[hidden] { display: none !important; }` so we also need the `!important` here to be able to override the default hidden behavior on the sphinx rendered scikit-learn.org. See: https://github.com/scikit-learn/scikit-learn/issues/21755 */display: inline-block !important;position: relative;}#sk-container-id-1 div.sk-text-repr-fallback {display: none;}</style><div id=\"sk-container-id-1\" class=\"sk-top-container\"><div class=\"sk-text-repr-fallback\"><pre>RandomForestRegressor(n_estimators=5)</pre><b>In a Jupyter environment, please rerun this cell to show the HTML representation or trust the notebook. <br />On GitHub, the HTML representation is unable to render, please try loading this page with nbviewer.org.</b></div><div class=\"sk-container\" hidden><div class=\"sk-item\"><div class=\"sk-estimator sk-toggleable\"><input class=\"sk-toggleable__control sk-hidden--visually\" id=\"sk-estimator-id-1\" type=\"checkbox\" checked><label for=\"sk-estimator-id-1\" class=\"sk-toggleable__label sk-toggleable__label-arrow\">RandomForestRegressor</label><div class=\"sk-toggleable__content\"><pre>RandomForestRegressor(n_estimators=5)</pre></div></div></div></div></div>"
+      ],
+      "text/plain": [
+       "RandomForestRegressor(n_estimators=5)"
+      ]
+     },
+     "execution_count": 3,
+     "metadata": {},
+     "output_type": "execute_result"
+    }
+   ],
+   "source": [
+    "from sklearn.ensemble import RandomForestRegressor\n",
+    "rfr_baseline = RandomForestRegressor(n_estimators=5)\n",
+    "rfr_baseline.fit(X_train, y_train)"
+   ]
+  },
+  {
+   "attachments": {},
+   "cell_type": "markdown",
+   "id": "60a4a5d4",
+   "metadata": {},
+   "source": [
+    "To judge how good this baseline model is, we use the `score` method, which returns the coefficient of determination for the inputs using the model as a predictor:"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 4,
+   "id": "283c1843",
+   "metadata": {},
+   "outputs": [
+    {
+     "data": {
+      "text/plain": [
+       "0.36233207787448163"
+      ]
+     },
+     "execution_count": 4,
+     "metadata": {},
+     "output_type": "execute_result"
+    }
+   ],
+   "source": [
+    "rfr_baseline.score(X_val, y_val)"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "bb00d17a",
+   "metadata": {},
+   "source": [
+    "Now try and find some better model parameters by tuning the model, in this case with an automatic tuner:"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 5,
+   "id": "a3258b4e",
+   "metadata": {},
+   "outputs": [
+    {
+     "name": "stdout",
+     "output_type": "stream",
+     "text": [
+      "Fitting 3 folds for each of 50 candidates, totalling 150 fits\n"
+     ]
+    },
+    {
+     "data": {
+      "text/plain": [
+       "0.44237143491088005"
+      ]
+     },
+     "execution_count": 5,
+     "metadata": {},
+     "output_type": "execute_result"
+    }
+   ],
+   "source": [
+    "from sklearn.model_selection import RandomizedSearchCV\n",
+    "import numpy as np\n",
+    "\n",
+    "rdr_tuned = RandomForestRegressor()\n",
+    "\n",
+    "n_estimators = np.arange(5, 50, step=5)\n",
+    "max_depth = list(np.arange(2, 20, step=2)) + [None]\n",
+    "\n",
+    "param_grid = {\n",
+    "    \"n_estimators\": n_estimators,\n",
+    "    \"max_depth\": max_depth,\n",
+    "}\n",
+    "\n",
+    "random_cv = RandomizedSearchCV(\n",
+    "    rdr_tuned, param_grid, cv=3, n_iter=50, scoring=\"r2\", n_jobs=-1, verbose=1, random_state=1\n",
+    ")\n",
+    "random_cv.fit(X_train, y_train)\n",
+    "random_cv.score(X_val, y_val)"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "5fc16d62",
+   "metadata": {},
+   "source": [
+    "In the real world, you might have done this by changing a hyperparameter a bit, training the model, and then evaluating how it worked on the validation set. In this way, you are tuning the model to work _specifically_ on the validation set itself.\n",
+    "\n",
+    "Before celebrating our substantial improvement, let's make sure that the model performs well on the test set:"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 6,
+   "id": "1cad8cd4",
+   "metadata": {},
+   "outputs": [
+    {
+     "data": {
+      "text/plain": [
+       "0.3867129657498729"
+      ]
+     },
+     "execution_count": 6,
+     "metadata": {},
+     "output_type": "execute_result"
+    }
+   ],
+   "source": [
+    "random_cv.score(X_test, y_test)"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "6ff82d2e",
+   "metadata": {},
+   "source": [
+    "The performance is lower on the test set than it is on the validation set, which is a sign that the results might not be generalizable. If future measurements were to be taken, we cannot be sure what the performance would be of this trained model. We should try to improve our model further or re-evaluate our modeling approach!\n",
+    "\n",
+    "_Side Note:_\n",
+    "For completeness, we can also look at how the baseline model performs on the test set:"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 7,
+   "id": "27a9cde7",
+   "metadata": {},
+   "outputs": [
+    {
+     "data": {
+      "text/plain": [
+       "0.2761820693760284"
+      ]
+     },
+     "execution_count": 7,
+     "metadata": {},
+     "output_type": "execute_result"
+    }
+   ],
+   "source": [
+    "rfr_baseline.score(X_test, y_test)"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "7d9f5f88",
+   "metadata": {},
+   "source": [
+    "We did improve model performance on the test set by tuning the hyperparameters, but the improvement is not the same amount as we observed on the validation set so the tuned model is not completely generalizable."
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "0cb5bf22",
+   "metadata": {},
+   "source": []
+  }
+ ],
+ "metadata": {
+  "kernelspec": {
+   "display_name": "Python 3 (ipykernel)",
+   "language": "python",
+   "name": "python3"
+  },
+  "language_info": {
+   "codemirror_mode": {
+    "name": "ipython",
+    "version": 3
+   },
+   "file_extension": ".py",
+   "mimetype": "text/x-python",
+   "name": "python",
+   "nbconvert_exporter": "python",
+   "pygments_lexer": "ipython3",
+   "version": "3.11.0"
+  }
+ },
+ "nbformat": 4,
+ "nbformat_minor": 5
+}