Repository: microsoft/archai (language: python)

Issue:
[BUG] algos.ipynb does its own train-validation split?
**Describe the bug**

Why is this:

```python
        # Loads the dataset
        tr_data = dataset_provider.get_train_dataset()
        
        # Train-validation split
        tr_data, val_data = torch.utils.data.random_split(
            tr_data, lengths=[len(tr_data) - 1_000, 1_000], 
            generator=torch.Generator().manual_seed(42)
        )
```

not just this?

```python
        # Loads the datasets
        tr_data = dataset_provider.get_train_dataset()
        val_data = dataset_provider.get_val_dataset()
```


Discussion / hints:
These notebooks were written prior the new DatasetProvider abstraction. At that time, we had to manually create the validation split because the old abstraction did not have support for it.

I will fix it.

Fix patch:
diff --git a/docs/getting_started/notebooks/cv/pl_trainer.ipynb b/docs/getting_started/notebooks/cv/pl_trainer.ipynb
index 5189b0b0d..19ebb3c63 100644
--- a/docs/getting_started/notebooks/cv/pl_trainer.ipynb
+++ b/docs/getting_started/notebooks/cv/pl_trainer.ipynb
@@ -7,7 +7,7 @@
    "source": [
     "# Training a CV-based Model\n",
     "\n",
-    "Training a CV-based model with PyTorch-Lightning is a simplified process, where the model architecture, loss function, and training process are defined using the `LightningModule`. Archai's offers a set of dataset providers to load and pre-process the data. Additionally, Archai provides a `PlTrainer` which wraps the `TrainerBase` abstraction and exposes proper name for methods that are invoked in the searches' evaluators."
+    "Training a CV-based model with PyTorch-Lightning is a simplified process, where the model architecture, loss function, and training process are defined using the `LightningModule`. Archai offers a set of dataset providers to load and pre-process the data. Additionally, Archai provides a `PlTrainer` which wraps the `TrainerBase` abstraction and renames methods so they fit in the search interface."
    ]
   },
   {
@@ -24,7 +24,7 @@
   },
   {
    "cell_type": "code",
-   "execution_count": 11,
+   "execution_count": 10,
    "metadata": {},
    "outputs": [],
    "source": [
@@ -55,7 +55,7 @@
   },
   {
    "cell_type": "code",
-   "execution_count": 12,
+   "execution_count": 11,
    "metadata": {},
    "outputs": [],
    "source": [
@@ -64,6 +64,7 @@
     "import torch.nn.functional as F\n",
     "from torch import nn\n",
     "\n",
+    "\n",
     "class Model(pl.LightningModule):\n",
     "    def __init__(self):\n",
     "        super().__init__()\n",
@@ -97,9 +98,7 @@
     "\n",
     "    def configure_optimizers(self):\n",
     "        optimizer = torch.optim.Adam(self.parameters(), lr=1e-3)\n",
-    "        return optimizer\n",
-    "\n",
-    "model = Model()"
+    "        return optimizer"
    ]
   },
   {
@@ -114,7 +113,7 @@
   },
   {
    "cell_type": "code",
-   "execution_count": 13,
+   "execution_count": 12,
    "metadata": {},
    "outputs": [
     {
@@ -142,7 +141,7 @@
     {
      "data": {
       "application/vnd.jupyter.widget-view+json": {
-       "model_id": "dba6a99f6dc347e7b42285253da35bf3",
+       "model_id": "fdea4e8f500a48ffbb8b2f83c88f2d63",
        "version_major": 2,
        "version_minor": 0
       },
@@ -163,7 +162,7 @@
     {
      "data": {
       "application/vnd.jupyter.widget-view+json": {
-       "model_id": "7c88693a408544a89ae3504015f12bdb",
+       "model_id": "0d6371e9c99d46bb80add4afe33cd8f7",
        "version_major": 2,
        "version_minor": 0
       },
@@ -175,23 +174,33 @@
      "output_type": "display_data"
     },
     {
-     "name": "stdout",
-     "output_type": "stream",
-     "text": [
-      "────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────\n",
-      "       Test metric             DataLoader 0\n",
-      "────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────\n",
-      "        val_loss             2.335841178894043\n",
-      "────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────\n"
-     ]
+     "data": {
+      "text/html": [
+       "<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\">┏━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━┓\n",
+       "┃<span style=\"font-weight: bold\">        Test metric        </span>┃<span style=\"font-weight: bold\">       DataLoader 0        </span>┃\n",
+       "┡━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━┩\n",
+       "│<span style=\"color: #008080; text-decoration-color: #008080\">         val_loss          </span>│<span style=\"color: #800080; text-decoration-color: #800080\">     2.466376781463623     </span>│\n",
+       "└───────────────────────────┴───────────────────────────┘\n",
+       "</pre>\n"
+      ],
+      "text/plain": [
+       "┏━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━┓\n",
+       "┃\u001b[1m \u001b[0m\u001b[1m       Test metric       \u001b[0m\u001b[1m \u001b[0m┃\u001b[1m \u001b[0m\u001b[1m      DataLoader 0       \u001b[0m\u001b[1m \u001b[0m┃\n",
+       "┡━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━┩\n",
+       "│\u001b[36m \u001b[0m\u001b[36m        val_loss         \u001b[0m\u001b[36m \u001b[0m│\u001b[35m \u001b[0m\u001b[35m    2.466376781463623    \u001b[0m\u001b[35m \u001b[0m│\n",
+       "└───────────────────────────┴───────────────────────────┘\n"
+      ]
+     },
+     "metadata": {},
+     "output_type": "display_data"
     },
     {
      "data": {
       "text/plain": [
-       "[{'val_loss': 2.335841178894043}]"
+       "[{'val_loss': 2.466376781463623}]"
       ]
      },
-     "execution_count": 13,
+     "execution_count": 12,
      "metadata": {},
      "output_type": "execute_result"
     }
@@ -200,6 +209,7 @@
     "from torch.utils.data import DataLoader\n",
     "from archai.trainers.cv.pl_trainer import PlTrainer\n",
     "\n",
+    "model = Model()\n",
     "trainer = PlTrainer(max_steps=1, limit_train_batches=1, limit_test_batches=1, limit_predict_batches=1)\n",
     "\n",
     "trainer.train(model, DataLoader(train_dataset))\n",
diff --git a/docs/getting_started/notebooks/discrete_search/algos.ipynb b/docs/getting_started/notebooks/discrete_search/algos.ipynb
index c90d1fc73..f81a69e79 100644
--- a/docs/getting_started/notebooks/discrete_search/algos.ipynb
+++ b/docs/getting_started/notebooks/discrete_search/algos.ipynb
@@ -329,12 +329,7 @@
     "    def evaluate(self, model, dataset_provider, budget = None) -> float:\n",
     "        # Loads the dataset\n",
     "        tr_data = dataset_provider.get_train_dataset()\n",
-    "        \n",
-    "        # Train-validation split\n",
-    "        tr_data, val_data = torch.utils.data.random_split(\n",
-    "            tr_data, lengths=[len(tr_data) - 1_000, 1_000], \n",
-    "            generator=torch.Generator().manual_seed(42)\n",
-    "        )\n",
+    "        val_data = dataset_provider.get_val_dataset()\n",
     "        \n",
     "        tr_dl = torch.utils.data.DataLoader(tr_data, batch_size=16, shuffle=True, num_workers=4)\n",
     "        val_dl = torch.utils.data.DataLoader(val_data, batch_size=16, shuffle=False, num_workers=4)\n",
@@ -2062,7 +2057,7 @@
    "name": "python",
    "nbconvert_exporter": "python",
    "pygments_lexer": "ipython3",
-   "version": "3.7.13 (default, Mar 28 2022, 08:03:21) [MSC v.1916 64 bit (AMD64)]"
+   "version": "3.7.16 (default, Jan 17 2023, 16:06:28) [MSC v.1916 64 bit (AMD64)]"
   },
   "vscode": {
    "interpreter": {
diff --git a/docs/index.rst b/docs/index.rst
index 8954449a7..a7d1f0c30 100644
--- a/docs/index.rst
+++ b/docs/index.rst
@@ -56,7 +56,7 @@ If you use Archai in a scientific publication, please consider citing it:
 
    First Time Contributor <contributing/first_contribution>
    Documentation <contributing/documentation>
-   Unitary Tests <contributing/unitary_tests>
+   Unit Tests <contributing/unit_tests>
 
 .. toctree::
    :hidden:


Test patch:
diff --git a/.github/workflows/lint-run-tests.yaml b/.github/workflows/lint-run-unit-tests.yaml
similarity index 75%
rename from .github/workflows/lint-run-tests.yaml
rename to .github/workflows/lint-run-unit-tests.yaml
index 1f5ac11b3..67bf5e55d 100644
--- a/.github/workflows/lint-run-tests.yaml
+++ b/.github/workflows/lint-run-unit-tests.yaml
@@ -1,4 +1,4 @@
-name: Linter and Tests Runner
+name: Linter and Unit Tests Runner
 
 on:
   push:
@@ -15,8 +15,8 @@ on:
       - "tests/**"
 
 jobs:
-  lint-run:
-    name: Lints with Flake8 and run tests with Pytest
+  lint-run-tests:
+    name: Lints with Flake8 and run unit tests with Pytest
     runs-on: ubuntu-latest
     strategy:
       fail-fast: false
@@ -37,21 +37,17 @@ jobs:
           sudo apt install -y graphviz
           python -m pip install --user -e .[dev]
           python -m pip install graphviz
-      - name: Finds Python syntax errors and undefined names
+      - name: Finds syntax errors and undefined names
         shell: bash -l {0}
         run: |
-          # Stops the build if there are Python syntax errors or undefined names
+          # Stops the build if there are syntax errors or undefined names
           flake8 . --count --select=E9,F63,F7,F82 --show-source --statistics
       - name: Lints the files
         shell: bash -l {0}
         run: |
           # Exit-zero treats all errors as warnings (GitHub editor is 127 chars wide)
           flake8 . --count --exit-zero --max-complexity=10 --max-line-length=127 --statistics
-      - name: Runs unitary tests
+      - name: Runs unit tests
         shell: bash -l {0}
         run: |
           pytest tests
-      - name: Tests documentation notebooks
-        shell: bash -l {0}
-        run: |
-          pytest --nbval-lax docs/getting_started/notebooks
diff --git a/.github/workflows/run-notebook-tests.yml b/.github/workflows/run-notebook-tests.yml
new file mode 100644
index 000000000..854c4a50d
--- /dev/null
+++ b/.github/workflows/run-notebook-tests.yml
@@ -0,0 +1,41 @@
+name: Notebook Tests Runner
+
+on:
+  push:
+    branches:
+      - main
+    paths:
+      - "docs/getting_started/notebooks/**"
+  pull_request:
+    branches:
+      - main
+    paths:
+      - "docs/getting_started/notebooks/**"
+
+jobs:
+  run-tests:
+    name: Run notebook tests with `nbval`
+    runs-on: ubuntu-latest
+    strategy:
+      fail-fast: false
+      matrix:
+        python-version: ["3.7", "3.8", "3.9", "3.10"]
+    steps:
+      - name: Pulls the repository
+        uses: actions/checkout@v3
+        with:
+          fetch-depth: 0
+      - name: Sets up Python ${{ matrix.python-version }}
+        uses: actions/setup-python@v4
+        with:
+          python-version: ${{ matrix.python-version }}
+      - name: Installs the requirements
+        shell: bash -l {0}
+        run: |
+          sudo apt install -y graphviz
+          python -m pip install --user -e .[dev]
+          python -m pip install graphviz
+      - name: Tests notebooks
+        shell: bash -l {0}
+        run: |
+          pytest --nbval-lax docs/getting_started/notebooks
diff --git a/docs/contributing/unitary_tests.rst b/docs/contributing/unit_tests.rst
similarity index 82%
rename from docs/contributing/unitary_tests.rst
rename to docs/contributing/unit_tests.rst
index 773540aa4..a760aec36 100644
--- a/docs/contributing/unitary_tests.rst
+++ b/docs/contributing/unit_tests.rst
@@ -1,13 +1,13 @@
-Unitary Tests
-=============
+Unit Tests
+==========
 
-The Archai project welcomes contributions through the implementation of unitary tests using Pytest. If you are interested in contributing to the project in this way, please follow these steps:
+The Archai project welcomes contributions through the implementation of unit tests using Pytest. If you are interested in contributing to the project in this way, please follow these steps:
 
 #. Ensure that Pytest is installed. You can install it using ``pip install archai[tests]``.
 
 #. Check out the Archai codebase and create a new branch for your changes. This will allow for easy submission of your code as a pull request upon completion.
 
-#. Create a ``.py`` test file in the :github:`tests` directory. For example, if writing a unitary test for the :github:`archai/trainers/nlp/hf_training_args.py` file, the corresponding path would be :github:`tests/trainers/nlp/test_hf_training_args.py`.
+#. Create a ``.py`` test file in the :github:`tests` directory. For example, if writing a unit test for the :github:`archai/trainers/nlp/hf_training_args.py` file, the corresponding path would be :github:`tests/trainers/nlp/test_hf_training_args.py`.
 
 #. Write test functions inside the created file. These functions should be named using the pattern: ``test_<name of function being tested>``.