Repository: JohnSnowLabs/langtest (language: python)

Issue:
Docs/website-nbs-updates
## Checklist:
<!--- Go over all the following points, and put an `x` in all the boxes that apply. -->
- [ ] I've added Google style docstrings to my code.
- [ ] I've used `pydantic` for typing when/where necessary.
- [ ] I have linted my code
- [ ] I have added tests to cover my changes.



Discussion / hints:

Fix patch:
diff --git a/README.md b/README.md
index 3cfa7ccde..f67b5746f 100644
--- a/README.md
+++ b/README.md
@@ -40,7 +40,7 @@ Take a look at our official page for user documentation and examples: [langtest.
 
 ```python
 # Install langtest
-!pip install langtest
+!pip install langtest transformers==4.28.1
 
 # Import and create a Harness object
 from langtest import Harness
diff --git a/demo/tutorials/llm_notebooks/AI21_QA_Summarization_Testing_Notebook.ipynb b/demo/tutorials/llm_notebooks/AI21_QA_Summarization_Testing_Notebook.ipynb
index a3e703a76..02553389e 100644
--- a/demo/tutorials/llm_notebooks/AI21_QA_Summarization_Testing_Notebook.ipynb
+++ b/demo/tutorials/llm_notebooks/AI21_QA_Summarization_Testing_Notebook.ipynb
@@ -57,6 +57,22 @@
         "!pip install langtest"
       ]
     },
+    {
+      "cell_type": "markdown",
+      "metadata": {},
+      "source": [
+        "## Installing required dependencies"
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {},
+      "outputs": [],
+      "source": [
+        "!pip install \"langtest[evaluate,ai21,langchain]\" transformers==4.28.1"
+      ]
+    },
     {
       "attachments": {},
       "cell_type": "markdown",
diff --git a/demo/tutorials/llm_notebooks/Azure_OpenAI_QA_Summarization_Testing_Notebook.ipynb b/demo/tutorials/llm_notebooks/Azure_OpenAI_QA_Summarization_Testing_Notebook.ipynb
index d1ef7cd0b..69aaa7c1d 100644
--- a/demo/tutorials/llm_notebooks/Azure_OpenAI_QA_Summarization_Testing_Notebook.ipynb
+++ b/demo/tutorials/llm_notebooks/Azure_OpenAI_QA_Summarization_Testing_Notebook.ipynb
@@ -57,6 +57,22 @@
         "!pip install langtest"
       ]
     },
+    {
+      "cell_type": "markdown",
+      "metadata": {},
+      "source": [
+        "## Installing required dependencies"
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {},
+      "outputs": [],
+      "source": [
+        "!pip install \"langtest[evaluate,langchain,openai]\" transformers==4.28.1"
+      ]
+    },
     {
       "attachments": {},
       "cell_type": "markdown",
diff --git a/demo/tutorials/llm_notebooks/Cohere_QA_Summarization_Testing_Notebook.ipynb b/demo/tutorials/llm_notebooks/Cohere_QA_Summarization_Testing_Notebook.ipynb
index ccdc6ca6b..eb5d1e744 100644
--- a/demo/tutorials/llm_notebooks/Cohere_QA_Summarization_Testing_Notebook.ipynb
+++ b/demo/tutorials/llm_notebooks/Cohere_QA_Summarization_Testing_Notebook.ipynb
@@ -57,6 +57,22 @@
         "!pip install langtest"
       ]
     },
+    {
+      "cell_type": "markdown",
+      "metadata": {},
+      "source": [
+        "## Installing required dependencies"
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {},
+      "outputs": [],
+      "source": [
+        "!pip install \"langtest[evaluate,cohere,langchain]\" transformers==4.28.1"
+      ]
+    },
     {
       "attachments": {},
       "cell_type": "markdown",
diff --git a/demo/tutorials/llm_notebooks/HuggingFaceHub_QA_Summarization_Testing_Notebook.ipynb b/demo/tutorials/llm_notebooks/HuggingFaceHub_QA_Summarization_Testing_Notebook.ipynb
index 55f150856..99febb82c 100644
--- a/demo/tutorials/llm_notebooks/HuggingFaceHub_QA_Summarization_Testing_Notebook.ipynb
+++ b/demo/tutorials/llm_notebooks/HuggingFaceHub_QA_Summarization_Testing_Notebook.ipynb
@@ -66,7 +66,32 @@
       },
       "outputs": [],
       "source": [
-        "!pip install langtest"
+        "!pip install \"langtest[evaluate,langchain]\" transformers==4.28.1"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {},
+      "source": [
+        "## Installing required dependencies"
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {},
+      "outputs": [],
+      "source": [
+        "!pip install \"langtest[transformers,johnsnowlabs]\""
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {},
+      "source": [
+        "# Harness and Its Parameters\n",
+        "\n",
+        "The Harness class is a testing class for Natural Language Processing (NLP) models. It evaluates the performance of a NLP model on a given task using test data and generates a report with test results.Harness can be imported from the LangTest library in the following way."
       ]
     },
     {
diff --git a/demo/tutorials/llm_notebooks/OpenAI_QA_Summarization_Testing_Notebook.ipynb b/demo/tutorials/llm_notebooks/OpenAI_QA_Summarization_Testing_Notebook.ipynb
index 10c4ccbf9..25b58bd11 100644
--- a/demo/tutorials/llm_notebooks/OpenAI_QA_Summarization_Testing_Notebook.ipynb
+++ b/demo/tutorials/llm_notebooks/OpenAI_QA_Summarization_Testing_Notebook.ipynb
@@ -57,6 +57,22 @@
         "!pip install langtest"
       ]
     },
+    {
+      "cell_type": "markdown",
+      "metadata": {},
+      "source": [
+        "## Installing required dependencies"
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {},
+      "outputs": [],
+      "source": [
+        "!pip install \"langtest[evaluate,langchain,openai]\" transformers==4.28.1"
+      ]
+    },
     {
       "attachments": {},
       "cell_type": "markdown",
diff --git a/demo/tutorials/llm_notebooks/Toxicity_NB.ipynb b/demo/tutorials/llm_notebooks/Toxicity_NB.ipynb
index 6d708c2ea..6f145b436 100644
--- a/demo/tutorials/llm_notebooks/Toxicity_NB.ipynb
+++ b/demo/tutorials/llm_notebooks/Toxicity_NB.ipynb
@@ -53,6 +53,22 @@
         "!pip install langtest"
       ]
     },
+    {
+      "cell_type": "markdown",
+      "metadata": {},
+      "source": [
+        "## Installing required dependencies"
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {},
+      "outputs": [],
+      "source": [
+        "!pip install \"langtest[evaluate,langchain,openai]\" transformers==4.28.1"
+      ]
+    },
     {
       "attachments": {},
       "cell_type": "markdown",
@@ -1160,7 +1176,7 @@
       "name": "python",
       "nbconvert_exporter": "python",
       "pygments_lexer": "ipython3",
-      "version": "3.10.11"
+      "version": "3.9.13"
     }
   },
   "nbformat": 4,
diff --git a/demo/tutorials/llm_notebooks/dataset-notebooks/BBQ_dataset.ipynb b/demo/tutorials/llm_notebooks/dataset-notebooks/BBQ_dataset.ipynb
index f261c1eaf..e249049a9 100644
--- a/demo/tutorials/llm_notebooks/dataset-notebooks/BBQ_dataset.ipynb
+++ b/demo/tutorials/llm_notebooks/dataset-notebooks/BBQ_dataset.ipynb
@@ -50,6 +50,22 @@
         "!pip install langtest"
       ]
     },
+    {
+      "cell_type": "markdown",
+      "metadata": {},
+      "source": [
+        "## Installing required dependencies"
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {},
+      "outputs": [],
+      "source": [
+        "!pip install \"langtest[langchain,openai]\" transformers==4.28.1"
+      ]
+    },
     {
       "attachments": {},
       "cell_type": "markdown",
diff --git a/demo/tutorials/llm_notebooks/dataset-notebooks/HellaSwag_Question_Answering.ipynb b/demo/tutorials/llm_notebooks/dataset-notebooks/HellaSwag_Question_Answering.ipynb
index 2fd2f7818..a33ba3e08 100644
--- a/demo/tutorials/llm_notebooks/dataset-notebooks/HellaSwag_Question_Answering.ipynb
+++ b/demo/tutorials/llm_notebooks/dataset-notebooks/HellaSwag_Question_Answering.ipynb
@@ -51,6 +51,22 @@
         "!pip install langtest"
       ]
     },
+    {
+      "cell_type": "markdown",
+      "metadata": {},
+      "source": [
+        "## Installing required dependencies"
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {},
+      "outputs": [],
+      "source": [
+        "!pip install \"langtest[langchain,openai]\" transformers==4.28.1"
+      ]
+    },
     {
       "attachments": {},
       "cell_type": "markdown",
diff --git a/demo/tutorials/llm_notebooks/dataset-notebooks/NarrativeQA_Question_Answering.ipynb b/demo/tutorials/llm_notebooks/dataset-notebooks/NarrativeQA_Question_Answering.ipynb
index ef45c3ebd..9906a6357 100644
--- a/demo/tutorials/llm_notebooks/dataset-notebooks/NarrativeQA_Question_Answering.ipynb
+++ b/demo/tutorials/llm_notebooks/dataset-notebooks/NarrativeQA_Question_Answering.ipynb
@@ -51,6 +51,22 @@
         "!pip install langtest"
       ]
     },
+    {
+      "cell_type": "markdown",
+      "metadata": {},
+      "source": [
+        "## Installing required dependencies"
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {},
+      "outputs": [],
+      "source": [
+        "!pip install \"langtest[langchain,openai]\" transformers==4.28.1"
+      ]
+    },
     {
       "attachments": {},
       "cell_type": "markdown",
diff --git a/demo/tutorials/llm_notebooks/dataset-notebooks/OpenbookQA_dataset.ipynb b/demo/tutorials/llm_notebooks/dataset-notebooks/OpenbookQA_dataset.ipynb
index 48fbb1658..b963307a4 100644
--- a/demo/tutorials/llm_notebooks/dataset-notebooks/OpenbookQA_dataset.ipynb
+++ b/demo/tutorials/llm_notebooks/dataset-notebooks/OpenbookQA_dataset.ipynb
@@ -50,6 +50,22 @@
         "!pip install langtest"
       ]
     },
+    {
+      "cell_type": "markdown",
+      "metadata": {},
+      "source": [
+        "## Installing required dependencies"
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {},
+      "outputs": [],
+      "source": [
+        "!pip install \"langtest[langchain,openai]\" transformers==4.28.1"
+      ]
+    },
     {
       "attachments": {},
       "cell_type": "markdown",
diff --git a/demo/tutorials/llm_notebooks/dataset-notebooks/TruthfulQA_dataset.ipynb b/demo/tutorials/llm_notebooks/dataset-notebooks/TruthfulQA_dataset.ipynb
index ac635619a..880709a0c 100644
--- a/demo/tutorials/llm_notebooks/dataset-notebooks/TruthfulQA_dataset.ipynb
+++ b/demo/tutorials/llm_notebooks/dataset-notebooks/TruthfulQA_dataset.ipynb
@@ -50,6 +50,22 @@
         "!pip install langtest"
       ]
     },
+    {
+      "cell_type": "markdown",
+      "metadata": {},
+      "source": [
+        "## Installing required dependencies"
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {},
+      "outputs": [],
+      "source": [
+        "!pip install \"langtest[langchain,openai]\" transformers==4.28.1"
+      ]
+    },
     {
       "attachments": {},
       "cell_type": "markdown",
diff --git a/demo/tutorials/llm_notebooks/dataset-notebooks/mmlu_dataset.ipynb b/demo/tutorials/llm_notebooks/dataset-notebooks/mmlu_dataset.ipynb
index 26f3af355..91d098235 100644
--- a/demo/tutorials/llm_notebooks/dataset-notebooks/mmlu_dataset.ipynb
+++ b/demo/tutorials/llm_notebooks/dataset-notebooks/mmlu_dataset.ipynb
@@ -51,6 +51,22 @@
         "!pip install langtest"
       ]
     },
+    {
+      "cell_type": "markdown",
+      "metadata": {},
+      "source": [
+        "## Installing required dependencies"
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {},
+      "outputs": [],
+      "source": [
+        "!pip install \"langtest[langchain,openai]\" transformers==4.28.1"
+      ]
+    },
     {
       "attachments": {},
       "cell_type": "markdown",
diff --git a/demo/tutorials/llm_notebooks/dataset-notebooks/quac_dataset.ipynb b/demo/tutorials/llm_notebooks/dataset-notebooks/quac_dataset.ipynb
index 4570e39f4..454938d86 100644
--- a/demo/tutorials/llm_notebooks/dataset-notebooks/quac_dataset.ipynb
+++ b/demo/tutorials/llm_notebooks/dataset-notebooks/quac_dataset.ipynb
@@ -50,6 +50,22 @@
         "!pip install langtest"
       ]
     },
+    {
+      "cell_type": "markdown",
+      "metadata": {},
+      "source": [
+        "## Installing required dependencies"
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {},
+      "outputs": [],
+      "source": [
+        "!pip install \"langtest[langchain,openai]\" transformers==4.28.1"
+      ]
+    },
     {
       "attachments": {},
       "cell_type": "markdown",
diff --git a/demo/tutorials/misc/Augmentation_Control_Notebook.ipynb b/demo/tutorials/misc/Augmentation_Control_Notebook.ipynb
index 4a26f3060..e9a5ad4c4 100644
--- a/demo/tutorials/misc/Augmentation_Control_Notebook.ipynb
+++ b/demo/tutorials/misc/Augmentation_Control_Notebook.ipynb
@@ -71,7 +71,7 @@
       },
       "outputs": [],
       "source": [
-        "!pip install johnsnowlabs"
+        "!pip install langtest[johnsnowlabs] transformers==4.28.1"
       ]
     },
     {
diff --git a/demo/tutorials/misc/Comparing_Models_Notebook.ipynb b/demo/tutorials/misc/Comparing_Models_Notebook.ipynb
index 5d763da73..3a3e782b6 100644
--- a/demo/tutorials/misc/Comparing_Models_Notebook.ipynb
+++ b/demo/tutorials/misc/Comparing_Models_Notebook.ipynb
@@ -175,7 +175,7 @@
       },
       "outputs": [],
       "source": [
-        "!pip install johnsnowlabs transformers"
+        "!pip install langtest[johnsnowlabs] transformers==4.28.1"
       ]
     },
     {
@@ -900,7 +900,7 @@
       },
       "outputs": [],
       "source": [
-        "!pip install johnsnowlabs spacy"
+        "!pip install \"langtest[spacy,johnsnowlabs]\" transformers==4.28.1"
       ]
     },
     {
diff --git a/demo/tutorials/misc/Different_Report_formats.ipynb b/demo/tutorials/misc/Different_Report_formats.ipynb
index bbc91cf64..cbc868a6c 100644
--- a/demo/tutorials/misc/Different_Report_formats.ipynb
+++ b/demo/tutorials/misc/Different_Report_formats.ipynb
@@ -44,6 +44,22 @@
     "pip install langtest"
    ]
   },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "### Installing required dependencies"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "!pip install langtest[spacy] transformers==4.28.1"
+   ]
+  },
   {
    "attachments": {},
    "cell_type": "markdown",
diff --git a/demo/tutorials/misc/Editing_TestCases_Notebook.ipynb b/demo/tutorials/misc/Editing_TestCases_Notebook.ipynb
index 768f66467..c4cd9fbee 100644
--- a/demo/tutorials/misc/Editing_TestCases_Notebook.ipynb
+++ b/demo/tutorials/misc/Editing_TestCases_Notebook.ipynb
@@ -77,6 +77,22 @@
         "from langtest import Harness"
       ]
     },
+    {
+      "cell_type": "markdown",
+      "metadata": {},
+      "source": [
+        "## Installing required dependencies"
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {},
+      "outputs": [],
+      "source": [
+        "!pip install langtest transformers==4.28.1"
+      ]
+    },
     {
       "attachments": {},
       "cell_type": "markdown",
diff --git a/demo/tutorials/misc/HuggingFace_Dataset_Notebook.ipynb b/demo/tutorials/misc/HuggingFace_Dataset_Notebook.ipynb
index 402e8d288..48bbda4e8 100644
--- a/demo/tutorials/misc/HuggingFace_Dataset_Notebook.ipynb
+++ b/demo/tutorials/misc/HuggingFace_Dataset_Notebook.ipynb
@@ -1,1 +1,1 @@
-{"cells":[{"cell_type":"markdown","metadata":{"id":"e7PsSmy9sCoR"},"source":["![image.png](data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAAUgAAABcCAYAAAAMJCwKAAAgAElEQVR4nOy9f5gcZ3Xn+znnra5pjcfKZCyNfqDIQgghZMdxZMfGxpbbwhjM2g4h2Ak/Nol3Aw5xEsLu5eHh8vCofNl9uFluLhiwhUi4zib3ZomcZBMgARsjt4RxbGIritcSsiyE0GpleSQLMYxHPd1V59w/qnq6Z6ZnNJJG/Ej6+zw9PW911fueeqvq1Pn9CucASZJokkzZaudirC666KKLcwWZ+y4TveyWJeW4/lKZYYD5mI2m8+YdH61Wk3Tux+uiiy66ODeYYwaZaKUysNSI7xSVtfj4MCPi9t8WLhzY+sADt9fndswuuuiii3ODaO66ShQSM7lvvYj8B6A8/pMIiM4/evToTuDI3I3ZRRdddHHuMIcMMocgC9ysFwx3DBzVyFzCQBpF8VyP10UXXXRxrjDnDBJygdFyl4wiTS3egJPnYrguuuiii3MCPRedem57NHBk3A6pwLxzMVwXXXTRxTnBnEmQSZJ/xP2gaDjhrv00vTSigB12tVqSJNrcf/p+uiFBXXTRxY8ec+7Fvuqq+f1RT/ktgl40PogwbKn/XQgv7KhUsJwBJjNIr10G2UUXXfzocU7iICsV9AfnL4k5nG85//zYKpXv1pMksStv+uT8eKy0RtyWqU9U8U1cU5e9Mb17qtU7anNPWxdddNHF7HEOGOTUTJpKBa1UsC271kYLjh79zyL6bnefP3F4b5JzxLEPvrhw4Z/v7sZMdtFFFz9CnBMGORW5On1V5YLVsUT/CNJrlnXcUzXg+JfU7c5K5ehQ1x7ZRRdd/KhwTsJ8JqMpTW7dzlJc+swykBZ3HpcdAfcMkVAGLVerKHl8UBdddNHFDx3nJMxn2sHMFYrEmrbtPyQxtosuuujitPBDlSDXbwgqDo4grUTtCRJkF1100cWPC+aIQc4uZMdMLAhtzDH/lo7KdhdddNHFjxZzwCATXbuWCNZO8/sWBgdfUvhuCh75hN8mM8P2djfKp4suuvjR4iwYZKLXvq7/YrGeD7jbIBxF3NskyZZ/JTc9LkyBBdP5XNxBwETV8OwwcKJSwarVM6ewiy666OJscEb6bJIkWq0uXOkS/ptqaZ1ZSqsoxQxwU/f28J7Jxzil6LwnG/aDD2zf+rtbz4S2Lrrooou5whlLkCa+LmjP8ix9KXUkEloWxBm+TaTwnDsmok+L6iHcIxcxaBzP0h98bnvlxe1szetLnu0JdtFFF12cKc6YQbprjLgiolKECzXlwVN9Fz2kmdumyPyhNLhGmRhEI9XqnceongFzLIpg0A0s76KLLuYILQaZJAobIZFZMphsgnQ4W7g7ICaAqp2oXHfs4K5dREePthsnZ2BySdPOWS2+K5bTvLG5rcsgu+iiizlBziCTRyIWDpY5ursO5PnPic8QunM3ofgvZ46T2eSp2tB04iRJYkmSpDOmFCau44x77e6II3GZ0s+U0bEyvq+PTc/2Ic8tw5fGJL5l9ky+iy666GJ65AxyydJVuN7OYh/lM88OIQwjz42QygjKMJ6OYlajhzqhd5Q7qFPJO/Ai7Lv5fx7VOHO7CfdZZPJsPtwLe9fxmb2D4H286IuJWYTqAvS8BbgsRmwAGCTL9gFb5mhuuuiii3/lyBlkqsuZN+8OsvogIaqhOgqhRikbJUtHca2TpaM0pE5afzBJNn5m/bb7VGkP8p74/3TtcSapBhODIjvDvj9I+fy7kbCGtF7GrBfPYtwUc8vXd3AIEdC5AEYXXXTRxZkgZ5Alt9yg6BH1sX5gfsHbNOdnriBQ7jVOvpRWqH72rHVYY3bGSytFNBqLkXSQrFFInN70hBffbmiYZYdddNFFF7NDIUECJcgZjytNxtiEA7iRpYqQTu2mubPMsi2AIGKz5LMCmOKmHeMtu3yxiy66OAeI2v6eIthbirVlRGGyq3imlMHJ7bbM60ICzMuatSrsTlmXRrFZqeNddNFFF3OIXEXtIBNOz5CauvfZQ0TqANXqRH47qyK5XYbZRRddnGNMlCDbMUWY7MyR2r3Ys4XjiKC4r61UPnMQsrJpi0lm+olDpfTE4Wo16cS6p6Gviy666GJuMZE1+mTD4/RcyFWsGcRzOpCWAKogHzGyjwATdPbg8QF06d2Vyv2fn75WRbc0WhdddHFuMclJAy3GM7lG4xSHSwp5QLa7W3uwT4t1easHkem1cqHVrWMi0XIXeY9Qa/LHtmOno+cnH801wydt6wa9d9HFjwgdVOxTOVya8N2W1YdE4wXi2YxH5BFERidm5u75/sVPDmAZIEsta/QC9YnHdex9GhrPHJ2YVbH9HDCsRG+6aaCvWg29k3+pVDanlcrzx//lMMr2eW2d08SVMP+lnOuPEdoz485Vptnk7LvTHSdxhbvJ04anw91nXm+hSV87XaeYl4kqdrsXe4oGOy7iWZWKVbJtu2HwfZlnG8VZPC1RCuLgbgMg/ePVfMaHLAZpfakI5gBxTOvHSUzwHGrY0zHHczXWU08tKZ8YyX4f918uwt5VwAwipfF0tbrkvUmS/EQzyZwBJkYClSo6NFRELly0FtjNll1Q1P+05vz/JJ9vF2eARGxqrYV2VIqaC8nE9ONT9lvUmWj2u2VXG9/bDbuHLO+bKf1Ob4OcUqpxIiOrVLAk+e2HIdl62WVLykuXTkfd8wCcGB78UAjRfzCrRyAzVBGapTR4jpjjbbdtiavVY+sybIUIRhaADIJHiB4DHprrMYeGxqK4HF6uIbrYLVMpXgiRBixr1EulenzKTn5skWilglarS/qvrty7LFTlNSby6gWLfJkg/Rw7rrB4FOG4kR1av97/6aGq7CXWw5VKcnxGR10Xs8Omb61A9l0OGXhQPv2tnfzOq/fOWf/JIxFLll2CPbsq3yCK6yj3f2c7d7z8xCmP37Ir5lhpGZEuxp5dCroAedl8JJQR78ElxTmJ7x0G389nnjuI7B0i8eP5+DMwysSVnzown/i5FaitI7rwSk74UpA+xFPcj7P0woPw3C42P/c0YfcBEj/R7HN6RuU+KS6yybgKKRVyzpwk9tRTjD711LQUKsC111nqba6Yyd7vZnvWPvEp9J09KpUkOjR8qC/WeXeKh7fnGToOLghR5GZPcg4Y5Lx5wTL31C2z3BSRM0jLR09H53rAHwKaUmC1urA3w25Q4ZYS4Ro3WyUiKqJ4YcMW0DyyIeBqtZLqARq+AwY/BTz+Iz2Rn2Q0JSd/7mpCuAejTKlkYB8C5oZBJolywZJBotIHSeVW8BSIEB2hkd4BfKHJJzof78rRby9nXvmjZI31CPNxi0GLpBAthCEDF0PCMCE6hNsOFu39Mg39exIfmZZJLn52HRq/DS29kbSxGhFFFEQUHBzDHUxSotJBTP+SZbs/1mSSE+MgRVpSZJP5TG5PqEp2ahWoZVcquivY38QCFq32KVleJ/rm0ATZM3aeQkCQCCd2J3aIEVVkJsn37CCtOyEPgZrgiPrJxBe/uKScuX44aM/HwX8NfBU47hlmDSyr5x+r45ZinoEQ46zGeKuJLYcfrsnjXxaaaqUoqhEiMVEMOoPD9ExQ0lVIuJjcfFYGIkLUj+hNwKn5hKS9qCwDGaD5rIWIfBGWDDzL81OiHiWEftzW4PZOeno/TmQbedm+pR2rj21+9hqi8iZEfhv31WgUIZr32RiDtFgJQRVEIpxVGOsIvdOo2DBVahxvnzkXShL42rai+0nGw9MNE+pM31w7aQzM8WbON27F2+aHgJ9873zTrnre+endIfT8dpaNxTiKoHnWapvtuWi3NRRxQ+WAethd9Ne1RZ4NJrAOn7uKqYkra3dHHLN1pPXlxeJTxRgZmN/A//vcfN75yuHpO7kb5J2FFJfm6cRwgKzxNwj/E6eGiaLWh6SvxFmPllbgBo2xBcQ9v0Wj3s/CAx8i8aFxO+aSfZcS9XycrL4OMyOUFLLDGF/CfRduI0BMlr4c90twW8d5fQsYPvY1vvuq4dxZNNmL3ZTOxnmYTGqfBQwIs+lqMmMYyw+cvEs7fXMNV/WiMlBLqJbTZ+b/SrFlF9HCkfR3Qii/O01PxiIStU+d5Kq1tiWdGoKKY/nLCEXYWS8xVKkkUdcOORdwxl/ycyk/vhAW0Ft+HZmVUVXS9CuUoktxHyREqxitryfxvwdmthU26z3kmtROTD7KC684NuWY+7/TT73+a2j0XsxXkDViSvHtZNn/4MIDnyHxlEXfHsDlA5hdipmhoY5nW8jC3bzn5QemjJ24sujAcn7w4luw7AtTnTQT4iCZJtJnbpjDqXtpqdo5q+yZ0OrYyU+usNUBk+M8f7JQLOi2lhDdlqVjfcJEdU5EUxE9CLbHPT3miKlIHxIGUF2M23KgTJb+c2znDXdXtpwrTHSyzgkSMe57bjlZdmmxxRC/n6h0F5ktQAOkfhNUv0Jy/Wm85DwizSKuQ0naH+674bsrhlny/B+TvZQSlT5CI+1HrZcQ3sBIbQtUh5CfWUccX06jDhqBsJVG9hGGXnFw2kLgL6w4SCL/9+TNp1Gs4sxQVAxXhe+rBMuQIrB8qoMGwAUTFBEZcer5pJ6qNNo5oHvSALPeczycZdK24vuslZvJ/Z+q79kEn7diECfHJZ4+vdUqmrpfEcxX57p06zeRAOJfERu7B0r76uXGcM+YGMRlPOuzLBuUwKVo6UqX8Pj1679bb94/pzqHs6F5ch/5N0yOx5yu/5lspDPRM/m4TmOeaozZn2+bdjgXKnYzHCYK1yC6ODdLZUOkPEpmr8eya8hSRaPXMPiy5SR+4LTjIrdhU45JNirPL6mx8MBfo+k7CKXX5GdkawjxAi5ccZyxxsWk9aW4QVwe4eTI3zH0qoP58dPQMA3j7BzmM9lDfJYe4yRJ7NprP/Gwp/V3hKh86cyKtqu51zJPv9DosSPAYO5JnkRnRw/73KEps+aUztx/O5NKinbTNzXl+5QPcbOo8ERUq2iSJIz3P8n5Nf3DO3176kOXKLPstxOSJNEvPzHQW66Fi9ysb9zmSG6gcLNhj/QDgeN7Ad5wVf6oVquMAMe2b0/23XbbliePHv3eFqE80hw3/y5oSzoO3U7EeJhFqyrU7BaBa55ra15a85Mk01/D6embpRNz/LgZmanl3uDmhsljnQpzrJWMMxq/CRUgMpxvsqh+jO/V/wcS1fAsJu5dRnbychLZf0rypqDDGlOJ5PNwdOMQS57bQ6nnNaR1cPqwrJ8fSMw8/Rncy+ApwgjoPujAbDuez0RMVLHbvdhNJjQeG3l2TOjrX//9pyuVe/+NWe0t7lZkjDTvvxZt4sFcbU9w2f7El39vhJvfNJinNLbR1ZG+uUXrwW6Xb6dWLE+SRLfsWhsNHj0yuH7Dp1bLtvCaRwivuA4WQBY/4jricOhasn/m2vt2fPnL6QFg+HSlnaEh9KuP9i+9Juu5YSty5XUbfCnmPLJN9nuWfSPL0scrleRwXhkp77dS2bQiwy/11FJVVVOxrdsye+3rP7Xz9a998UheZm7higy9/LrruQp0BdssAj3yCPbPlcq926vV3j1JktRnS2vISmURHURzb7XguIuJBpzs4Ne/dmRPMXPtqvN43xddtDtNkuRYs33ZZZt7zz+/foUZ860qputVATz69KEXLxh8ZvDobhsbmz9fe3rWbt2u16x3+XnB5rNBRrZW/cA1lU8+GNGzE5ITM9kyK5UkeuihRQPr19+76pFtevl118urcJaSe2VrW6scuZb0Wat86tFqNT5QqeT9VSr3l2H0cjMbaNJnKqbmCvcc2779vY91GqvOwou3bpPl11TMqIKuV0313oOPVe/aOXX/+8uZ1i6Rbb6Y9cWEVc2iikZZ+OTer3/t93af+so0X/fMnQ3yvj2X4H4NaUMRMdz/jtsvqrP52R2E6ABuq0nTAcRfxyef+wrHV00fjnMmj7Fbffx/kTpRGOWkKm5Riy+IgkzJUJstpqYaTpYUJ4f7nAWq1buOAPedar9WDF2HHzvSdy6NkNImQU50FiVJol/9av+yhfHRm116flHcLgcGkOZNEEAEcVdcUonCgbLKX1+74dN/Ua0e250kSZ0OaB9RALFQvmBwwVvUone523rRkN/iWkjiwm9GpWg7LL4HfusrkEuYW7dlG5Tojzx4DUHVzUTiUW003l+tLvxLM26UEL1PsHUQehGseY754pPRPhi9p1rt2wIc60DqjBhfkUhcPU9HXXbttYMXv+51Q8/kNHZUVydsmzcvW+we/YEIl6q4oYCLikd/0//9F38XLlhe6gn/HuRmcVla1CzNRxZXNfl3HvE3kl2wqVJJdnZikle94Y8HsrGxDaUe/SWMG9xYIKoTGEkeiqcaiR5w2Oos+KvLLttchXqvubwHid6q5PSpuEnQ2C3aWakkV7WPmSSJfvUbFwyW0ujDbtnNiqSIqASNStjDwE3ttFUqj0Rp2LU8ePRRd7+6SZO6mmsoq/EeYBYMsg1z5cVWuYFSOSIdM5BDYE8CUPf9SGMvImuwFOLyJdjoCrj7mbkZeCMs291PI1pNVoTqiB7ETx6j96U6dv4xJKQgkGXzwS7jwgMPkST1001TnL4e5GScczvfRJyWLekcO2m8k/yfJFqtXrA6RPGnIPrP4De4eb+54Vkzxq+BZ3XcU8AjsJUov68S3Zux4M1ffGpJOZfiOp9MMeWxpPZOJXwUZL27q2f1vN+sgWcNwMuOvxENH69U7nvNuBqdaU01KEgZJ0aIVUOs7ksz+A2Nev4Q/Grce90LWpv9muFuKyF8xCj/1k03fXL+bOIR43qtbm7H3a3wSkPLbCD9ov7Rr1YHr9iya+2kJYc7I4rE0JCiGmHEOLEEjZQwX+q22qV0r4j+O5ylbpm25iWPrQTvF5O3u0QfzbKB1ZP7r1TuXRzX7UMq0cfBf9VhgWOYNcav43if7ubmy8F/TSW+5/zz7feGFv70sKg+JSKG5/RhRSygyKpG44LBibdNYpr5MlFdKSqtawORO5dWKpsXTKRvm6mzGMIyEYnHx4AyeE1cpkioM6KIvT4rJIly/3f6gdcXy6AoIjtI64dJXHnx+SHcniCKR4EU95WIrJ05x7oN0wljSaLjtsK0VKHUs5YsNZAU9ypmx3j+sjruu4ii44hAWu8lKr2Z2tjVrL0tym2ns4+rzXecHObzI8aPX9zb1HmpVC9YnRE2icrNbul890wR0yYrLbJFtJ25upu6W+yZXy4e/vC8kcbNUyWacS++uhuOrBb0P7r7cstSLVxammcESB5bKK7uZu7Zmgzf+NBDixbkc+i1PI7eQUxx1KwRu8htKuH95o1lZinuZjjmbX2Cq3umjs8XLb3rByd1PcwmaPv7I0L2zyI6MjHeFXAzRG6MNHzugqGhjZXKp9aQd2rkJocpfTcaYybjBUscxNUtU7N0tbr/IcgVbhYVvNha8yKKgONq1oiRaL2WSu+f2HuirtHHReTd7tni/HwzBVcBXFAR1bbzUMSa46+QEH9w4dDQ73iWPSOqRxAMseJ6ZIjo/FJJV7aGK87RwnJ3W+qeX5e2/QfNGmsLm2lrPlJdhtsCt2J/DNEA5nvghT0zX49JmCsnTb1+MaXyGiw1oEaWfoOFHM+LSVyfYjwOHMctIksHiEpXMbCvb+blpAtMJ4s1+cLi564h6vkAWTqAqqL6NHbyAY4+MAoYFu3A/BmcCDMQ1hJKH+NY/MbChpnHSs6Clok7zCgl/ngwz444x8JtK+snI0kSrVQ2rXDCx1R0vecXILeL5a/nVELphIjsNfc9IcRDImEiE/RMRWWxEG2+9nX3XXLyZKaTw2HGz0noBe/L/1VUo1SQnKG17SqCmmdpFHpeE+L0LUmSqKnXJ3QoqHtWBrnULFuGmZL3aaKKeMs+JCKIiLplkWe2LEjpjmp14eBkp087kiSxSgUT9+2CPi46yd6UF0lWz7I1IcT/u0v0j9dtuO/Prq3c9+bXfnXJsi1b1kaTmWSppOZNHWe80ImD+EoRvcIsNQRVVUSDFT/bhIQrcfWsHrn7r61ff+/VkOhll23uXV8Z/AOV8KtZNtYLFo2fN2IaolGVsB9nt4TosGioC0W/goJFWVbrDaXeD6Csc2cvIupe3C3uphppBs0QGBLy1Etcf8GzbAGeL4ZXVLMy1aAeqOQ25MSqVbRaXdiL+s+6Zf15VpxAca+4yN9Xq0n6Q800ShKF65RM14MMgqRE8X5UHmf32nSciVn9ScZGnyaKQQKIVuixaSs2FCgW4ZMyJZayaPEyNn1rBfftXcnmZ9fw2b03sOQ7mwjRf8fSy9EIgj6O1d/LnWt35IxPjLtW7SPLPkb5vL2okku5cimBv+Wz+/8rn917Awt3D0JVT8UoO8dBdsT0XChx1yLwfE6QnKtyTKeBiT5yz62CrrlDRl+8WQjXFA/nuKoooiaqO71R36QavknGaCb1derhXaJhvVsWk8cwqVlmqqV+Se0DIZTeZ3gqjk728I8nZmrY75buMOe4qi4vJKeBPPOkuZdHZo35SrjuoccW/XUkmRVse1IuRe52EpW6oI+aNQ4gUtYQXeKWXTJZzc+7tyvAlkFy5NRe4Rf3Zb7gc0HjNe4sds90vB6ooI5hWcMQ6ROJ3i6kb45i/+bCRcf/qlod+AJwqOmpbzTESrGk3kZ38yxwN5HIVGSve7bTzU5I0NWIrMOy/lawQ26nVonVqN8CyWPnnffpimjp7WluP8sZjjuCGnAo8+xz5tnfSxSOq9sKcf6tiLzV3fpaHmGP0sbYAkF/CU+HNET1jCxu7w+4qDlfCfDahs0v9ZTWuhvuaZt06nlMs8vP33LL5t4vfvH5WrWKXX2j9pbSsAo3xX2cRvdsGPWvz3wXT4OzYqcb4WX7FuPhKtJ6nKuxjd00xiZ6qe+6aIRNzz6I6M1kYyC6CgmXksie6SvxCGCgcjla2gyhmTgQgffhtpigfWQpwGG88RUyPs6RVROl6MSVIzzEon0fpjzvD2iMrSgkXSPSd5Lpmyj1PsqSpV9G9lQ5fGR/EfIwTbmzM1GxN26EJOETu04ul2dH3+S/IhHuhoQzn37PDAKf+NWxR39/Tc/TZ9zPHKAV4tPGpAQbPHpk0CX+JfD5tN9qriYiJ9wb/3HDhmOPNjfv2rX20JEXXzyo5veAXOHuxUPratYwDfE1sTQuMbfc09tWetidIutEdpqnH80auj2ObbQRxgaiLHqnavR+t6y/RbXg5mgUrQhZulhdzCfFIgKIYwh1N/usRX5P5DIE9ahhsiYS+SOQi/OiGQV7dVPQxYJeDDyZJFPDh5oowmSoVuVLnjUGRMNHRaI+LyQ9mhlJuRqf21CFPjeviMrlaPn69Rs+/alq9dhjlQo0GuDixaJtE9ITTTQC829CfaNQ3yk6r4bbYkPuFA3vxrK+1jUS3DMQW1epbF7gkv0i7oMTcyDERMOwe/qpejn77BNfPj5S/HCgUhnYax56VUu3uzVyVb4ZDKa6yiwbVbeaIHFz3twzcF9dqfzU/GolGSZJrFTZNGDua5quxXH2KCi5mr36e99rLAP2QWKa3dcHvpKiDB5Cs97CHjLfe0axn2cjfiRibPrWKuKe1aR1I4pr1Eef4OjQMZKLWiXDAHTvw2SNEZBeNJSx7A3A508dD6n9aLSu+D9/EIpsXxr1lHweTiD+jwhD42M2+22mG76w6i9Z8u06qncRxVcDZRpjIKEfsVuReAORfpNFS/8W+/W/hOTI5MIas3fStIjPaSharqzE5f0CH0T0g4h/UNo+p9NG9QOi9gF3W3c6FJ17FGxSvJYSLnbzy3MnRpukpaqI/7Xasceq1evG4yIvumh3uviCC3YiPCAhGqG4PXMV1k1hIHO7HogmhDMB4KYhOu6SbQr0fimOXzherRwd/cbDJw6JN+7DssdEI9zb46QwdwZClg20r/Mz3qNDblPXrZbJPVE2dLBaPToK3x95fWXom5h/yt1TL9TUNptqZMgrZjNbuap9dHRkJPoTJ/tdYK+GWIubfeI5NhklmbpZn3t2q0rPPSkL3ghAb/uuzZNonoupB7sbjldh5ESlcnQUjh5Q5L+CPENbFXvH86ElLDUdW6caX+JmOm4eaaq41tiRxvqnN13ZZI5JEat5/DCBexxLc2bbJMrVzfpBBtzTWq5mA1DYFcNSiBZX8pU71Sxbi2XL3QxcwN3cyRMn3Ey1NKAlXdOkO8p8qbstd2tZs91NPfUdUDsx1ck3C5ypCJO4cv93yki4nLS+vAinOU4WHodKEaeZaDOPmedX78PZQVTKGZzZhsK5MzM8HSUdO0ha309aP0BaP0jWOIGIUe6NCAFCWM28+R/B5HMsfnbdxFqStOIan/+fX6KR3oll7ydLdxL1KFFJMQNPe0nTDcTzPkKJTWzad3F+bMtkMdFJMytPdfHMFXMgSorIqED+cUZo+0xoU7RpfSb9PuowKh3X3v7hYrKKXbzv64peJyrz80IWkjNJF3PLhh17II+N22btQc4PPLA7bbhvxX1IhOYDhLtoljV6Bb8cvJ/2cnCOiahmWX3Ig26tVr9br1aTwsaTWLX6vhMmfFk1dApk70uRPjWxKdIjmCg1cftiFA0drFQo+kvSJEksy6wqovtVWyFN7m6ImogOMkskSWK33PJ8bfsjd/1pGuQNZul/EtHdGnpG8WAgaev9InnxCnE1y2K37OJI40/Bomva+2wG0DuF9CiyY/vWux6qVpO0SX+lgp1/vu53T3eIaJ2mKNw80r2XNLrW8pTGCVCNMOVvH3voPUNF8HdxbP7/9q13PYbzpIQSTAjeFVWVsjsHRQPgzegzk1CanyKrxvcN4ToJIXYc1Qjwb6roweZS9OY+X+DSSmWccV+C+4LcOQOCpqLhmEn29Wrl+8OTVwSdHs2XPGcnQY6MDRDF16MaUeqBsZM7iE7sbDk/ig9AIinIA2SZkaVQ6lnOWHrD9J27FXRuh3Ataf3nSMd+lpPRzxHkZ2nUr4lUAr8AACAASURBVOXkS/8HIjuAlNEf9FMq3Uyp9//js/tvnVJkNxEjuT5l6JUHOLzyM8ThtaT1X6Y+9nlK8UE0GGZG/eR8gt5KpA+y6G2Xw8ZxJjnNu8QnqduT2y2IuYGnhtfBUnJ5tPPH2769rQ0pWNGWVPxUl3ASPefAf9SxSyNCfDWiJmBN+5yoIqqHTfwAdPbC+1jPQbf0cBFnaOMrO4orooOO9I+rn+MQBEZcs1pnlVYONetHTiyI45GgEaRtFq6m1wIDHcnwY3n17ok9RlGoC+SFSGWCGwiE0yrc25yHbzx858Ht1aGN4v4rno19VFQeEo0Oi2hK4RgaL3snglmmDstd+DCjcVSYGZjw2hJBjCPFSBPu48sue76myAtISPPzLc5B8nMQZRVu88enq/g2S8F9GtNOPoaITPrdEcFAyiqyF3dEirAmwRR6BVlRrWJr1xLltlyMgkE6uh2V/VLEznrWKLv5RbCkH8Al/KxoZDhWOHNURA+QsTe/dKeTauhn96wkYvREK/BsXe5gQlGG8f71fGbPGyd8Fu99I5959k14I8ZtBFFDxBC/iS27TnEfSUqqdY6uHeWui0Z438tP8K5XHuLoXzzO0OGP4GPvIEv/BNE6acOwdDUiG1my7JKOITxNafKOl9c48ud/g/a9i3r9DtLGnxLFJ9AI6jXQsJhS+WMs3bOqGZI0UcX2JuMZt8xPbY+jzSvj1BCpC1ITpCZyZh+EGlBDfHoJshN959SLPSFPPHZncOJdVgwucjzKQsfAb0isp+fQMHBMVWkvC+wO4tILEkNhMyzGbf2djjKvNfdoUz+104RMYbyGTX64kiTRRqTmkp9H03c/V2+gavWF3SLH/ou4v8fTsd8F+WNURmj6porxRFDPUhC9JoR0DWitKfw0YwUACFNfpM30wsyzurTJSs1XiLur4QvcPPY2ppFL9lkaEXUMiG97kRwZZw5FzwV6Ef8ndxsZZ+aOmmW94K+47JYl5YGBwWU4a1pFkQ1RnkD0ADC+sJ1GpeVZyJYmSaK4r83PurjOKlia7g2hdPA0pr5F55nGQTbVV/cKyCCWKY0xQ/RWouiPCD2fm/iJ/yj/lN6PWx9uSqMGGl/B96KVM4fYOJTHtPOyC9uMw2v2kcUfAdtCFEd5LCSXIvqOZsjYVPrb7J53Lh3lhVXbKcfvx+obCeEQGnImKXI5pu/gwgMxietEFRumMsJTqN2ipDmDo+ZCzdXqLlZ3L75ltm3qAjXwus2kBHSi7xxGII0/jrnEGkkeqNuyXTVvXJd6o6EdCysAVKuYIB0YqBgaVCZyiVlh5uq92Sn3mA06BsmfEZqmgSStVF44uGHDi19qjI1+yN3vEuFA4T0eH89xVKLY1K91UqWI5/TCwTPZMz89/cW3FDpsXso8br2AJrhL0jRk07zkmpCxcRW6SamBO+UU9uCyVzQycTcH3LNYkRXn/yCdLxGXiJb6MENENEsbdXWextLv5jZJDMHcWCoNX/zEE6v6EFbiha3U3VTDCGL/dGYLuZ3FszLOYPQNSGFL1qBEpQFgGSJLO390MSGKgNzuV4oW4375zI4agU5l9NvV96MrhsjsHiwbHY+Qc7uVe3f1zZgt01L/jRUHRvDz/gRr3IOEEUQhrZcpla9mNFsGc/AEpSmIWj2gGJh625uh+aKcZdudVHBcT9MGOUfPcLWKVSpphER9orlHeFzykkLddclVhZz28ZqGDr2lkk3jUUy0Urkwdk72NVlqy/nh6m41F6nLhBqJZ4hxlTLMvN8s0KJzbkX05hxVKsnw0MJlWwaODcVBo4+5Wb9IW9FVHHHWgMduTRUcaIsBPRXG59llvOakC3VEwFrsMZckJY4yZszbdbfzRbStXsr4CGnJ5TBBtnor9lFxjBAPYukCsNeqKJm4iUQK2d5K5ej+rdsu2Ccan3DL+t1dRWxQRFaMjIwckuCL3VtXwtyPoZxe9kzz/Jrc8UxtkPfuvRT8NWSN3K5kthfP9mAetdJrOw3tA2i4FKxMo94P0ev4+D99ie+fGMkXy/r26dHRYq5P80f7dhNK64qCFSuQsJIkyVMaT/UCuf76lOQRWPgzX6As/waXDQgpqsvRxjIS2TdRxT6ddMKNG4tDPBWRmkNNoO5IzZGaS/E5jTbqNReti4fTu4RzJEHmapSWaa7SKC0lU3Nj4xFROdQ+Ty0Hji2uYx09dEkCjdLIgIsvNjOgXfoUHDuheYXjlq3wNJhS59PPOM3whNPs/9Q4VQBztZqkg0d3W+S6WzU6RFtgeZ6P7gAxPiGb5bTombCvkJfTcx8SpD6+zEfBdTVEajbVeVOcSxF9wEpErKm+53lNggjHwWrm2T+4pXVENF9SRUxF+qGxGPe1ZllhRwSQJ5MkMXU9KKJDCCaCOl520VeGYKtVS3mWkGOiQS2r71Orn17udfPkzxYRNxKXI/KMpRouG3n+lb+Enn8bPaXpP0HuIpSeyV9KppTii+ntWwnbjLMNoHbJFwVzz71sQeaf4ohJqBiMHaFeP4Bqmj/O3otob37Krb9nhsjNTWuKmEEuR07Rfjrxu6nPjpF7XSU79xLkxLp/UKmgSZKk69dvWolk42EW446/nA8edOGo5OEhxc+Cu6mIDqpwCbBzciB1ksD6DaxRiRabp4wvN5BXuUnF0n2GRHqGrOicmmDPoP9OZdSa8zxRwk40l9qzMnh5siMwd1n5CYR+0dzHebr0tDQANHegaOruB1TCCcda0qKTB4wrVyVJ8qVOmkClcm+fua+T9vvZx42jB8BHXMMeNfYDa8wzlTy4e74RLhVhZV60Q3C31Mi+AZAGORwsPYSzGjBRAdFV7vYDFaWotI5IhEj69Wr1fSfOrIiwnNnNkiTKsn/fT+Pk68kaoAFE9yAndwDw/JJa5wML5jfwjv301J9Gw7p8jRlbidvFcN0cxDrnWWb5v2ago62c71nWg4t+2vAf1HKeZNY+SR1Y48RMjqntAm2MXyH1fGU6y4qU2BwtBaa1TSe1WxARyzNWbAYJshN9p4/JD0ClklCpJLr1Eb9LVPvNsjw+zwsmaKkiPEua7XMNI7j0uuQ5u7ntSGNxfxvwp8UImveLwoVRaiOvV2WBu1vTGC+CqZaGU8+eELefZ8JbY/bnNc0V4mwtKGf2LCVarS5a7mK3O/5MpXL/1mr1jmm88HDllQN9mcstkqYrEJ9EsIDotwS5zJuhQPlmbb+zZsbE2VEJqWm6C5FDIEvHexHUrAGU3vjwwwvur1SS/fnSxq2eTLhRJVpheXC7FhRansrOznovwyHzuro+jdvaptfZ3frEea2jA4ghqoAcDsiTAFHmQ+bZXtFSxTyFzFXUVpl5LJKNu/TMGmTIGdZXPxsv9kZo7LuEnvJqxk6ChgjsSYLlDq0Z6ywmyvFVIyx69h+Ie9/C2EvzcesnlK/ip1Z8gUsPjHB62eQth9GSvQO4ryJLc6btNkw9O3L65/eDXlwGsbQo2yajICMwOdVwfIXA5k0jrfY0T4umpRTSmqOWhzugrcfcaQmUxcbJAmZ72y0X1CSawYvdib7ZY+3aJB4cXHS1iS/1NN3nrieiKMRbt/pKUb9DVG81y3TcvuS5ucXhYObp0yX1Iy6lRxG/Ec8lcgTFUtMQ3bi+cu//1hjr+X96eg4VMWoLyyYnbw3S83bL0phchcpVJtHIspMHAjxs8PNeLHrkM7C8TpjgZsgdSLTbICevHHk6aB07OyRJYus33Ls60vPuzGxsmVntmfWVz2zH7B9V2Z8GhqJMLAvSGzJfaeLvwv1N7lY4UYq5QcnS2qiKPezwC+30nO55tJ+/4+oi+ywd+6ZoWGd56FbO7NxNlLUhkg/Coru3bHnhcJKQVqsXxnnNR/+ISRp5U5b1XMbVEO03sr+76crjI7t2ra0NHRv6Bwi34pTzQPJ0PrABsd7WlZKdwJE8E+aukfXXf/op1WjY0rQ/L4jhqwVZbtbIox60hFu2uyRHnzytk++E5vM203KsTSSee5Nl6XqcBagaGp2g0djG80PD8MDMYyWJkWxULNpO/eRhRPoRNczWMy9dyrZte1j0zkkHzeKhXvJ8GdffptSzgEbNiGIwHuPFVUdy73el5c2eaclZqkr2skvp6bmYRj1Pa/TsAMYhEtepSy6cUT1IrUsza2Py8ZM16RnahhgK0YTg3kk4i3qQuXTzU72m4VfE7TcJ0Ql1GTUhQhlAQtkss0lDGGAisr3k8QGIR8xH/0IlrMN1QdOp4DmTBJcPx3Hj1akt3HbttYxmLlep6O2epUvBtWlbaxaeyCz9XP1kOtRT1gjBcLS9HuRsMZVlZMW8hDNijNB8lGdPS5IkumULkWSsymx00N0jCdGlAusMUhOGg8mwo6mYlc19UDXEmRW1KNqcHqKKW/b5RoPDUezllg9b8NNw0sCkF4N7/gIJ/ldCuFHUV7lleYiNoG5ZJITbHR+8YHDwi1+r+rGgtVWWydtEdY2bjWsADiaqdcuyh+aVSzvzEKPd6QvbFz0j6BHwFYVwoUBuG3Mxx8zddo6OlIab8/a17faMWXZCkCKHXGKYGHcqKtXqI8k06uypZ2EqNkIyUzTARqCqLBlcisZXktbLedSF7CewO2dC15/aX5CIkTxygMVLHyOetzZP99OVqFxBkuxm0+3ka08V8OKZvo4iYHsjucpaqM6Lvr0Az94KelcRagRuJzC7H6rK4LLL0W/3k922k7suOjI1pKjoKxHj3r2XEOR3SRurwYxo3ijpS9tYYIcY6iRBTodpHDgaxtLM4xqSV0M5mzx4AcMhUzk9G+RpPC31uBzHKQs89zAOoDIghSrtZHnwdrPb3GZlInoos/pfBV48AZDFi/5eG/yChNJveFYvN1W+/CR8vov8RkDfCpK6WX9epqrlnRUXE1V1S78QGPt8Z4/zGbpG5Ix9lB26On0MDv5Ur6Gvxr0XUMtSy/3FROLaj0o/4uNOmMzSybdWKqqK2ZMe/F5ixnn9mUnAHc6jAcdeHHx84cKhTaLh4+QRNCYi6oJC1gv6JhWtAKPu3gfEZqZ5EXsHxDSUEOdxs9q9Dz74nuMA1eojkbL7oIscQFg5ZXwRUwnHzPyfb7nl+RrkNuqr3pDuK9X0gGi0sjBUNZlwbj7FasC2fP8zWXvHARRLI5yL2LT3ZngO/Fe1df81K+Y3289C9DLDWIPIxUVoD2SN3YTy1NUBZ0Jyfcpn9j6IZe/GHUKIsfQm4E8mO+EQYsT72D04zIW/njK6OyJ6Wxn2LiCTdZTC67HoTbgtAIworuPp54nqW7lwRR+mb0PCrdT9m2za8yD+rd2kpUMMMMxL56WE28qk+xZz395LifRdIFdjmVEqK86TpKUt7H5FSlIwtdmZqjo/sHWLLcJriMbkthhMMHVTkyh32bppvq1gPqKFimJKsX+zPwXIZggU74RZPjdJkthrX7u5TMziwnsMnqdw5fbrdkkjV/5D6BnNvPG5gD7ctpzB0A03fOIPGo3yAo3i2y2tNyWaXDV3U3fpQ9wQz+v3FZKPoIiqmttXAvLhavX7w5XKwl6bUUL/yUA+v5+YX4rDxS5mZm0vnPwFpLl0MEntzf/Ns0tCrJ6lzxD8w4svGHzm8IkXFnQebXbocGtYCKndfvvu9IknBv7kpZPyStHwW+T1N1NBiqfBcJMyeWFammuku+dZPSGU1PG9Da+//xtfP76nybSq1W122WVLDp/Xlz4jGq5xyyLaXroI6iIHVdnfnDOAN1yVnPhadeGOoGFDXui3FWCV2yzZL954uv2Y00I+x0paLxNKt1OK3zTrl3CWlUkb/eBQikcYe+kJDi87cdqLcIlvJ02PoNFg7qxhPZv2DY4vP49ofhvI5YSwGWSYWqNOiCKM+USlBZRKg2SNATzLmWpcTmmMfYGGf5yja0+waM9yovJrEF+KyFuJz9uAZ8fRxnFG/BiM1ElLfYQwSFxaSv1kwWR7FPchxkY/xNE1+5vnNlHgG1dX2yeu2e7MhcolTOCkZz7q4qPuPiomNXcZFfOamNda2/Lf3bzmxfb8t3w/cR91l9FsxjjITvTNHqVSvdexQciZFS4mxSdPe5O0CKlINcRDDat/eNEFA/8lL4TQujGvuebEIZEjv25p/ZOi4VirTmOzVqNT2NVM0BTHVCOTEB9yz/6vQPquavU9z7Q7AYq0RcPF2p+pjkGzraMoDMtN+ovtgbT15kvHf5dgrRTCTjjJeICqF7RIUQl4Fo9DVupRkFS1NKIarIitMRFJBTWcPG3O1fJ2HjKjoZRq6DnmWf2PLbLbtq8/+vBFF+1uuw/yfvL9i3Oc1eOpNK9JM60xyyIFuPLK4yPnzcs+hGXvFaI9QeNiPClSIL2Nkef0qqppKJ2wrLElqzdu+Ub1xR2txcEAEnvqqedruD2hWjohzb5a18c8G9sD9XEJrOn1D/A1MwMN7fsX9gd/cmysMTQ5rXLWEPL7BAHL+qifXEy9NrtPkzlqgLQxhPmjpx2ek7hy56uOoeEhQpQ7Yks9g3h6I9Rb9ImmqPQTQoWo52ZKpbcQ4lsJ0QbMLqZRGwSUuHcUZD+1l95Pze7k6CtypqZaJkQpUZybIhq1ftJ0JSJXEKI3EUpvRsONWHYJjbEBRCGeN4LZwzTGfpGjax5vJ7tDPcjJjHBm8axu5BWfFdP8T4H266gdtnVoN3OwZ7JBdqLvtKSvKBL0sKiWTaQPtzJ54QkDqSMyjPsQlu0Usb94tPrbDwM8MMkWXTwQtUrl/g+kfvKL6nabhJ5LgWW49UlegFVB6yI6jNgRS9OnTep/dnxo0WO33747bYZqnH9+ZN//QXZYNX7aMFQL35UEGo2TB0qlUsfsjgaMlDXeIRN0VDFERyRNR4AR1Z4draI2CrghOuI6Ntxxek6GNJSj/aj0mQYTXB1MpaSucqjt3Dvi8eoLB6+5ZvBOVasgvFajaK0QBtyZD152L7SWfC2WuiDH3bMhz+o7UR5UOfbQhmuxR5PEEhK9+sYoVQ0HBN1pmk2gJ5NakW43MaQqSUA0OhZC/DRCLG03mkjpsPjJ0eYSq0mSjFSrfLbuCx8LJreFKGxwD0vzXG0rjpVUJIwAx9zGnvEs+++qjYe2P/q+E52X+YVqlR0i4fEQlZY1tzuYalxv1EYeqX69FarTCpy/d6e7PR6intjVinPNXyBpdvJrPT3DwzOVmpsWlg0T9T4DVj4jI5ijBUNTRr/3GPN69p7u2i7jCPwVIaxFepSe82Cs9mpMHqdU3oPQh3kZiPHm85NnF0GooTJKo3GcNN2PNZ5ArMp7Xr13Qmrh86v3snTPHWR6IyLXEc9bBT6AWR9mEZiimiLRKBKOU39pH7XRv0PCF3jPq4YmO67yJ+uze2+g1LuZdGw5WTadwp3r6I3aX/Kq//W2ZFvFkkTs4986uQLxN6vPQV5b4eixzKvvW3teHmN1775V9ER/i9uaYvW0Dge6EfVAlj3N83922UwXr1K5v5yFk6s9s+UqMmDIAnWPwVLxMOyeHVHVg8C+SuXo6GzVmZtu+uT8kZFohUS+SmCxYX3iquJ+3NWPqLf6hElMJkn0tV/tX1YqlQbaOWFQVxdGouzY/k6LTV150yfnxyO6KgstVScGsiAWsrGDJ08Gi+Ppf69W33dicp+33bYlfv740Apx+jJrHRfU1cZKx77xjTtPmQPcZBqVyr19WQjLQ9YYNNEBy7yfQF4d3RkVYVjdh0APQe+havWOGsWSuW3ZNhEsXJGpz59MTzAZrlbv2teJhqtv3DQY123p1DeLpmPn6/6nvnjnuFzelOB27VobHTl+fJVYusKdpYL3g0YOI2I+BHJo3ryePQ8++JvHTzUHt922JT569IWVmUpvO90A3jN28B8e/A8d+kj06spPrw1ZiJvX7FTXa1b4410D1MMymqnFTWGoUXzP1G7/PxJljCF+75WHzogOgHt39SHzVhIKPpPKML3hEA1bTqO+gCjqwzxGPcI9ArW8iogWoTc+hDeGOLo2v36d1PymY2fZoX7Sl1biuhjxAdA+3CPUR3E5TqZH0Jf28Z6fG5qO3JzbbNqzgZ6+zaS1FTmX7Yj8DdKo/w090duS766oJ4nYJ58bXeaZ3+yEGMfOyktjBqpIJtX3ru3J04U2P7sGjf8WfNW0DNLdKPWAZzt41yt+YeoOE9G+/nG+ZOtLOjT0Xbv9dtL2dZFP19bTYgxJBBcW8/jdZimufK3safucSXWa/phKBW0vedUsk9XcNt3veYzf6fU78zEdeimqgrevTz15/NYa3zP1e/r05BELE49p+3WasI8Wc06SRHftIjp69EJtv4ZF37Ocg6nX9NTzOPGY2V2vU5Exi3VgZoWqwjY7Y+lxCj3NcJxpajlOe9wM+0zYv2CUrf4Vqkwc8+4ZUxJzbrP52Wso9W6mMbYan4FBaqRY+ijiv8Tzq4+TiG1+1hec9Nobxa0X1bP0oBpmmhJk+/f//P88kCSJsenZKwjRF4EFZOn0EmRpHmTpdt698vrZj9fK8ICm6jIXC4ZN7vfHbRGyHxXaM2pgbub63GFittWPN61dzAKniovsACFxZelzl1Cat5n62OXj3qGOfhkB1b1kY7/MC6/eTSJ27y7vS8NL17iEQU5Zx/HUUPfR1OZVhx/gRJKIsXnv2xG9H/N4gkNmAn1uxL2QNv6ad6+8bVYBsF100UUXp0CzWMUwaTact8fTuXJMKExrRqmnHymtgbtJ3PXoEDVTjoh7TfC647Uz/Yh4aipDw0O0ORDCL6AhHndZji9X10afA5aBUtjHZrn+bhdddNHFDMgZZNw4QTZ2pChZNFHymqzSZul84Cou/PU4AZLrJY0bHBHXE47XBK1LpnWh7XPKttcFr5tRH3Pbz7a7cxru/04ZYUPhYe6cqSPFtiyFzJ6d+ynqoosu/rUiZ5CH1p7A2UUUj+YS2jRhMyJKlsbEPeupp2uboVBHh847JioH1b2mntZUqam3fU7ZDjXB63h04OSreo/AxrwOx8n6G9FwMWld8WncP05RXUSOIeSOnblcg7aLLrr4V4vWUonC0+CdY+Pa4Q5ZuhbRm1m4u5ck0eR6SV+M4wOWlo5khLq518y9ZqH4tP/f3m7bniHHYi/tTUQsgTzfslS6sxhzyuJTEyGgYTcuh7r2xy666GKu0JLKgj5NOnaIEGkH70wbXHEvA/8WDVfkbnTX5OVSmzcW71NPjyleV3wio/S2Txtz1NTrkqbH5WR939G1jJK4suSpMpK9EwmvIa3TvnznFIgYuGHZDsbsBFw3RyENXXTRxb92FG5vMf7XoSNktpWoB5gpk4XcIQIr///27ifEruoO4Pj3d869972ZvsQYnTCRYEIYUpmFRBoGXdVAd13ZVpe1QWiKWVYLUkrvUIrYLooUq6YuFARtCy5aKaWbDLRKrS66KLY0dkwlZpKZMB3j+ObNfef+jov73sub/2/GSSPl94FhOMx973Bn8eOce3/n98P5H7L/vapgZR7d6RPS/O++xrRGuaROm1LGIJIUErQQ6fsJWlR/06IUuVxvNqY/Or7vWt7dGWvjXlz2CGW7AVvkcImAS66i5RvMjy2Sn7zpLWONMf8fVi4Vf/HPu3H+LYQM7ZSFiquu7tWHFCWtKaF4lVA8ztzs1W4CZh6jOzhDPSx/spdm0mg5XHSFYxnqaaaFoknQlk+GFubGaeYiSn4ugfuVQ++fILpniXo3ZTtZVeVj1ePRCN4r4v9AaJ3hyl0fbPsAvTHGbGDtXvr5f7+C9w91muC4zXfbUcnqBWX7t8TiKW6Nf+fd8dAfpPJzMeEIyUhzLoER5marPtj5SQnXM+MnYeTBYZyfIKs/g8a7KNsbTLpq/trwAq3mE8wee2GrrHhjjNmO6+Gv+3Lj7L++giQvEXWUUjcPkFW2tuLTgJbvoPpL2vIa82OLOZOdjhAb5CT2H/85cP5OvDyE84+AHKVsb/0cMaIkCSBTEB7mw7FLtno0xuymleEvzx2HH95LO/wY5Nuods4vbkkRgbQ2S2vpjzh+Ra35JqfuWVj3HGg3kD3z/ii++Bo++zqRE8Sy0TvJM8iczjtUH+Ty2GsrvtcYY3bB2kiUR8fBfxwn3fNzQjGBbljdp09nJQmQZAqySFieBvkLTt6mHS+RyiKxdJRxP94fBb5EZILa0CHay/XqxU/cOjjG7vPPuqLlr/mweQpWbuuNMWY3rB8gc1GeO/8NstrPCMVoFSQHLNsdY7Wa9KnDewgBNFR9dKvVaB2fgnMQ2lAG3TSNZ+0EikuA+FdieYqZV3Zem84YYzax/vY3jw75wu9pffIsiEOcDlyUVsQRoyMUyvKSom065wHrIBkxQnsZlpd08ODYPd0TOw165AKqP2UmTG/jXo0xZls2Xhbm0XHLhb0Mhadx8k1Uldh5ntjrM9qp5r3huG+K6+lBdBqUDPD5vjFU5eLTbJ6y/AHt1svMjTdta22MuVE2Xr3lonx05Bqe76O8iEsCzmkv6PWauMsm41U5jL1CE4N+vvsVUq0c01qL0H6C1L3I3G8sOBpjbqitHyzm0THy7gF88jhJ7Vto2IeuetPcW+XJjRgr3iuRi8T4JKfHzu74bo0xZhu2fv6XizI3PovwJGUxSZJdxGdVWbQYtfNWmV7zrN0aRxSRquct7k20/C4Mv3xD/xvGGNNnsLfHuSgzx+bJ0rOE9hkiUyRZwCeuU0OyIn1b452Pq+CbZHRSh14gLJ1hf/t1Zg62dnSXxhizA37gK6cmI/fcqnz8wHka8+dQvQJ6lNrQHlQFYlldGGVNy4beKrFroz7bUqXwJGmLMryDxu8RWs8xO36JuRG1Z47GmP+lwQMkwNRU5H4RFh+4xmO3vcFXH/0dZXsJn9ZIa/Wqx7QH5yIinf1ylPWDo4A4xbkqenrfojZ0haL1JzT8BIk/4jvH3mbiQCA/qUxNbqf5tTHGfGYDZn+vo9eshxRnXwAAALtJREFU+8uOO0aPojIBch/p8HGkPEQobyfGYbzXNdNEdagqIk18chHVC4Tib0TewvNnTn/xam8OSwI3xtwkOw+QcD2Adc9b73+vQcYhXLyDUu9E/GHSZBTxDaJmAGhs4uICoZyB+AGlTEOcxV+7zMzrrV4fW2OMuck+W4Bcrb8Rd34u4fCRhI9Dxp7EsdC5xgfFF8rwcOA/RwK5hF4tSAuMxpjPkd0NkP16W3BYWfJssjPu/LagaIz5nPoUBSp4D1AF9yMAAAAASUVORK5CYII=)"]},{"cell_type":"markdown","metadata":{"id":"3o5sAOfwL5qd"},"source":["[![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/JohnSnowLabs/langtest/blob/main/demo/tutorials/misc/HuggingFace_Dataset_Notebook.ipynb)"]},{"cell_type":"markdown","metadata":{"id":"WJJzt3RWhEc6"},"source":["**LangTest** is an open-source python library designed to help developers deliver safe and effective Natural Language Processing (NLP) models. Whether you are using **John Snow Labs, Hugging Face, Spacy** models or **OpenAI, Cohere, AI21, Hugging Face Inference API and Azure-OpenAI** based LLMs, it has got you covered. You can test any Named Entity Recognition (NER), Text Classification model using the library. We also support testing LLMS for Question-Answering and Summarization tasks on benchmark datasets. The library supports 50+ out of the box tests. These tests fall into robustness, accuracy, bias, representation and fairness test categories.\n","\n","Metrics are calculated by comparing the model's extractions in the original list of sentences against the extractions carried out in the noisy list of sentences. The original annotated labels are not used at any point, we are simply comparing the model against itself in a 2 settings."]},{"cell_type":"markdown","metadata":{"id":"26qXWhCYhHAt"},"source":["# Getting started with LangTest on John Snow Labs"]},{"cell_type":"code","execution_count":null,"metadata":{"colab":{"base_uri":"https://localhost:8080/"},"executionInfo":{"elapsed":23927,"status":"ok","timestamp":1689532651217,"user":{"displayName":"Prikshit Sharma","userId":"02603035510215936699"},"user_tz":-330},"id":"azUb114QhOsY","outputId":"30f8035b-250e-4bbf-da25-65cd552f700e"},"outputs":[],"source":["!pip install langtest"]},{"cell_type":"markdown","metadata":{"id":"yR6kjOaiheKN"},"source":["# Harness and Its Parameters\n","\n","The Harness class is a testing class for Natural Language Processing (NLP) models. It evaluates the performance of a NLP model on a given task using test data and generates a report with test results.Harness can be imported from the LangTest library in the following way."]},{"cell_type":"code","execution_count":2,"metadata":{"executionInfo":{"elapsed":11992,"status":"ok","timestamp":1689532663205,"user":{"displayName":"Prikshit Sharma","userId":"02603035510215936699"},"user_tz":-330},"id":"lTzSJpMlhgq5"},"outputs":[],"source":["#Import Harness from the LangTest library\n","from langtest import Harness"]},{"cell_type":"markdown","metadata":{"id":"JFhJ9CcbsKqN"},"source":["# HuggingFace Datasets Testing For `text-classification`\n","\n","In this section, we dive into testing of HuggingFace Models for different HuggingFace Datasets."]},{"cell_type":"markdown","metadata":{"id":"iO7jyI9F8DQ8"},"source":["## Glue - `sst2` Dataset Testing\n","-------------------"]},{"cell_type":"markdown","metadata":{"id":"ZtqqWrqO8DQ8"},"source":["The provided code initializes an instance of the Harness class, which is designed to handle text classification tasks using Hugging Face. The Harness class accepts a data parameter, which can also be specified as a `dictionary` with several attributes.\n","\n","The `data` prameter also takes a dictionary which contains the following attributes:\n","\n","```python\n","{\n","   \"name\": \"\",\n","   \"subset\": \"\",\n","   \"feature_column\": \"\",\n","   \"target_column\": \"\",\n","   \"split\": \"\"\n","}\n","```\n","<br/>\n","\n","\n","| Key  | Description |\n","| - | - |\n","|**name**     |Represents the name of the dataset being used.|\n","|**subset**     |Indicates the subset of the dataset being considered.\n","|**feature_column**       |Specifies the column that contains the input features.\n","|**target_column**     |Represents the column that contains the target labels or categories.\n","|**split**       |Denotes which split of the dataset should be used.|\n","\n","<br/>\n","<br/>\n","\n","`It's important to note that the default values for the split, feature_column, and target_column attributes are \"test\", \"text\", and \"label\", respectively.`"]},{"cell_type":"markdown","metadata":{"id":"swaYPW-wPlku"},"source":["### Setup and Configure Harness"]},{"cell_type":"code","execution_count":null,"metadata":{"id":"JaarBdfe8DQ8"},"outputs":[],"source":["harness = Harness(task=\"text-classification\", hub=\"huggingface\",\n","                   model=\"distilbert-base-uncased-finetuned-sst-2-english\",\n","                  data={\"name\":'glue',\n","                  \"subset\":\"sst2\",\n","                  \"feature_column\":\"sentence\",\n","                  \"target_column\":'label',\n","                  \"split\":\"train\"\n","                  })"]},{"cell_type":"markdown","metadata":{"id":"jWPAw9q0PwD1"},"source":["We have specified task as `text-classification` , hub as `huggingface` and model as `distilbert-base-uncased-finetuned-sst-2-english`\n","\n","For dataset we used `sst2` which is a subset of glue dataset.\n","\n","You can find more HuggingFace Benchmark Datasets [here](https://huggingface.co/datasets?task_categories=task_categories:text-classification&sort=downloads)\n"]},{"cell_type":"markdown","metadata":{"id":"MSktjylZ8DQ9"},"source":["For tests we used lowercase and uppercase. Other available robustness tests are:\n","* `add_context`\n","* `add_contraction`\n","* `add_punctuation`\n","* `add_typo`\n","* `add_ocr_typo`\n","* `american_to_british`\n","* `british_to_american`\n","* `lowercase`\n","* `strip_punctuation`\n","* `titlecase`\n","* `uppercase`\n","* `number_to_word`\n","* `add_abbreviation`\n","* `add_speech_to_text_typo`\n","* `add_slangs`\n","* `dyslexia_word_swap`\n","* `multiple_perturbations`\n","* `adjective_synonym_swap`\n","* `adjective_antonym_swap`"]},{"cell_type":"markdown","metadata":{"id":"zCP1nGeZ8DQ9"},"source":["Bias tests:\n","\n","* `replace_to_male_pronouns`\n","* `replace_to_female_pronouns`\n","* `replace_to_neutral_pronouns`\n","* `replace_to_high_income_country`\n","* `replace_to_low_income_country`\n","* `replace_to_upper_middle_income_country`\n","* `replace_to_lower_middle_income_country`\n","* `replace_to_white_firstnames`\n","* `replace_to_black_firstnames`\n","* `replace_to_hispanic_firstnames`\n","* `replace_to_asian_firstnames`\n","* `replace_to_white_lastnames`\n","* `replace_to_sikh_names`\n","* `replace_to_christian_names`\n","* `replace_to_hindu_names`\n","* `replace_to_muslim_names`\n","* `replace_to_inter_racial_lastnames`\n","* `replace_to_native_american_lastnames`\n","* `replace_to_asian_lastnames`\n","* `replace_to_hispanic_lastnames`\n","* `replace_to_black_lastnames`\n","* `replace_to_parsi_names`\n","* `replace_to_jain_names`\n","* `replace_to_buddhist_names`\n","\n","Representation tests:\n","\n","* `min_gender_representation_count`\n","* `min_ethnicity_name_representation_count`\n","* `min_religion_name_representation_count`\n","* `min_country_economic_representation_count`\n","* `min_gender_representation_proportion`\n","* `min_ethnicity_name_representation_proportion`\n","* `min_religion_name_representation_proportion`\n","* `min_country_economic_representation_proportion`\n","\n","\n","Accuracy tests:\n","\n","* `min_exact_match_score`\n","* `min_bleu_score`\n","* `min_rouge1_score`\n","* `min_rouge2_score`\n","* `min_rougeL_score`\n","* `min_rougeLsum_score`\n","\n","\n","Fairness tests:\n","\n","* `max_gender_rouge1_score`\n","* `max_gender_rouge2_score`\n","* `max_gender_rougeL_score`\n","* `max_gender_rougeLsum_score`\n","* `min_gender_rouge1_score`\n","* `min_gender_rouge2_score`\n","* `min_gender_rougeL_score`\n","* `min_gender_rougeLsum_score`"]},{"cell_type":"code","execution_count":null,"metadata":{"id":"5DNBjDLp8DQ9","outputId":"b184a72c-447f-45a0-f77f-19782fb4a15f"},"outputs":[{"data":{"text/plain":["{'tests': {'defaults': {'min_pass_rate': 0.65},\n","  'robustness': {'lowercase': {'min_pass_rate': 0.66},\n","   'uppercase': {'min_pass_rate': 0.66}}}}"]},"execution_count":8,"metadata":{},"output_type":"execute_result"}],"source":["harness.configure(\n","{\n"," 'tests': {'defaults': {'min_pass_rate': 0.65},\n","           'robustness': {'lowercase': {'min_pass_rate': 0.66},\n","                          'uppercase': {'min_pass_rate': 0.66},\n","                        }\n","          }\n"," }\n"," )"]},{"cell_type":"markdown","metadata":{"id":"ZPU46A7WigFr"},"source":["Here we have configured the harness to perform two robustness tests (uppercase and lowercase) and defined the minimum pass rate for each test."]},{"cell_type":"markdown","metadata":{"id":"KVolf25u_OFV"},"source":["➤ You can adjust the level of transformation in the sentence by using the \"`prob`\" parameter, which controls the proportion of words to be changed during robustness tests.\n","\n","➤ **NOTE** : \"`prob`\" defaults to 1.0, which means all words will be transformed.\n","```\n","harness.configure(\n","{\n"," 'tests': {\n","    'defaults': {'min_pass_rate': 0.65},\n","      'robustness': {\n","        'lowercase': {'min_pass_rate': 0.66, 'prob': 0.50},\n","        'uppercase':{'min_pass_rate': 0.60, 'prob': 0.70},\n","      }\n","  }\n","})\n","\n","```"]},{"cell_type":"code","execution_count":null,"metadata":{"id":"kvHcQi3M8DQ-"},"outputs":[],"source":["# Limit the data to the first 2000 samples\n","harness.data = harness.data[:2000]"]},{"cell_type":"markdown","metadata":{"id":"i6kPvA13F7cr"},"source":["### Generating the test cases."]},{"cell_type":"code","execution_count":null,"metadata":{"colab":{"base_uri":"https://localhost:8080/"},"id":"mdNH3wCKF9fn","outputId":"cd348490-7ade-40fa-d870-dc059f5aa647"},"outputs":[{"name":"stderr","output_type":"stream","text":["Generating testcases...: 100%|██████████| 1/1 [00:00<?, ?it/s]\n"]},{"data":{"text/plain":[]},"execution_count":11,"metadata":{},"output_type":"execute_result"}],"source":["harness.generate()"]},{"cell_type":"markdown","metadata":{"id":"nyjDdYLeGCmM"},"source":["harness.generate() method automatically generates the test cases (based on the provided configuration)"]},{"cell_type":"code","execution_count":null,"metadata":{"colab":{"base_uri":"https://localhost:8080/","height":641},"id":"c0jL1_G7F_p6","outputId":"502c1525-9000-4041-823b-3b04f6650892"},"outputs":[{"data":{"text/html":["<div>\n","<style scoped>\n","    .dataframe tbody tr th:only-of-type {\n","        vertical-align: middle;\n","    }\n","\n","    .dataframe tbody tr th {\n","        vertical-align: top;\n","    }\n","\n","    .dataframe thead th {\n","        text-align: right;\n","    }\n","</style>\n","<table border=\"1\" class=\"dataframe\">\n","  <thead>\n","    <tr style=\"text-align: right;\">\n","      <th></th>\n","      <th>category</th>\n","      <th>test_type</th>\n","      <th>original</th>\n","      <th>test_case</th>\n","      <th>expected_result</th>\n","    </tr>\n","  </thead>\n","  <tbody>\n","    <tr>\n","      <th>0</th>\n","      <td>robustness</td>\n","      <td>lowercase</td>\n","      <td>hide new secretions from the parental units</td>\n","      <td>hide new secretions from the parental units</td>\n","      <td>NEGATIVE</td>\n","    </tr>\n","    <tr>\n","      <th>1</th>\n","      <td>robustness</td>\n","      <td>lowercase</td>\n","      <td>contains no wit , only labored gags</td>\n","      <td>contains no wit , only labored gags</td>\n","      <td>NEGATIVE</td>\n","    </tr>\n","    <tr>\n","      <th>2</th>\n","      <td>robustness</td>\n","      <td>lowercase</td>\n","      <td>that loves its characters and communicates som...</td>\n","      <td>that loves its characters and communicates som...</td>\n","      <td>POSITIVE</td>\n","    </tr>\n","    <tr>\n","      <th>3</th>\n","      <td>robustness</td>\n","      <td>lowercase</td>\n","      <td>remains utterly satisfied to remain the same t...</td>\n","      <td>remains utterly satisfied to remain the same t...</td>\n","      <td>NEGATIVE</td>\n","    </tr>\n","    <tr>\n","      <th>4</th>\n","      <td>robustness</td>\n","      <td>lowercase</td>\n","      <td>on the worst revenge-of-the-nerds clichés the ...</td>\n","      <td>on the worst revenge-of-the-nerds clichés the ...</td>\n","      <td>NEGATIVE</td>\n","    </tr>\n","    <tr>\n","      <th>...</th>\n","      <td>...</td>\n","      <td>...</td>\n","      <td>...</td>\n","      <td>...</td>\n","      <td>...</td>\n","    </tr>\n","    <tr>\n","      <th>3995</th>\n","      <td>robustness</td>\n","      <td>uppercase</td>\n","      <td>when there 's nothing else happening</td>\n","      <td>WHEN THERE 'S NOTHING ELSE HAPPENING</td>\n","      <td>NEGATIVE</td>\n","    </tr>\n","    <tr>\n","      <th>3996</th>\n","      <td>robustness</td>\n","      <td>uppercase</td>\n","      <td>on cable</td>\n","      <td>ON CABLE</td>\n","      <td>NEGATIVE</td>\n","    </tr>\n","    <tr>\n","      <th>3997</th>\n","      <td>robustness</td>\n","      <td>uppercase</td>\n","      <td>it with ring ,</td>\n","      <td>IT WITH RING ,</td>\n","      <td>POSITIVE</td>\n","    </tr>\n","    <tr>\n","      <th>3998</th>\n","      <td>robustness</td>\n","      <td>uppercase</td>\n","      <td>far from a groundbreaking endeavor</td>\n","      <td>FAR FROM A GROUNDBREAKING ENDEAVOR</td>\n","      <td>NEGATIVE</td>\n","    </tr>\n","    <tr>\n","      <th>3999</th>\n","      <td>robustness</td>\n","      <td>uppercase</td>\n","      <td>that these women are spectacular</td>\n","      <td>THAT THESE WOMEN ARE SPECTACULAR</td>\n","      <td>POSITIVE</td>\n","    </tr>\n","  </tbody>\n","</table>\n","<p>4000 rows × 5 columns</p>\n","</div>"],"text/plain":["        category  test_type  \\\n","0     robustness  lowercase   \n","1     robustness  lowercase   \n","2     robustness  lowercase   \n","3     robustness  lowercase   \n","4     robustness  lowercase   \n","...          ...        ...   \n","3995  robustness  uppercase   \n","3996  robustness  uppercase   \n","3997  robustness  uppercase   \n","3998  robustness  uppercase   \n","3999  robustness  uppercase   \n","\n","                                               original  \\\n","0          hide new secretions from the parental units    \n","1                  contains no wit , only labored gags    \n","2     that loves its characters and communicates som...   \n","3     remains utterly satisfied to remain the same t...   \n","4     on the worst revenge-of-the-nerds clichés the ...   \n","...                                                 ...   \n","3995              when there 's nothing else happening    \n","3996                                          on cable    \n","3997                                    it with ring ,    \n","3998                far from a groundbreaking endeavor    \n","3999                  that these women are spectacular    \n","\n","                                              test_case expected_result  \n","0          hide new secretions from the parental units         NEGATIVE  \n","1                  contains no wit , only labored gags         NEGATIVE  \n","2     that loves its characters and communicates som...        POSITIVE  \n","3     remains utterly satisfied to remain the same t...        NEGATIVE  \n","4     on the worst revenge-of-the-nerds clichés the ...        NEGATIVE  \n","...                                                 ...             ...  \n","3995              WHEN THERE 'S NOTHING ELSE HAPPENING         NEGATIVE  \n","3996                                          ON CABLE         NEGATIVE  \n","3997                                    IT WITH RING ,         POSITIVE  \n","3998                FAR FROM A GROUNDBREAKING ENDEAVOR         NEGATIVE  \n","3999                  THAT THESE WOMEN ARE SPECTACULAR         POSITIVE  \n","\n","[4000 rows x 5 columns]"]},"execution_count":12,"metadata":{},"output_type":"execute_result"}],"source":["harness.testcases()"]},{"cell_type":"markdown","metadata":{"id":"NOJ8BAU2GGzd"},"source":["harness.testcases() method displays the produced test cases in form of a pandas data frame."]},{"cell_type":"markdown","metadata":{"id":"3CwhQw6hGR9S"},"source":["### Running the tests"]},{"cell_type":"code","execution_count":null,"metadata":{"colab":{"base_uri":"https://localhost:8080/"},"id":"aguX6-aFGOnP","outputId":"bb014811-522b-4f07-fa8a-bf3d1c906d7f"},"outputs":[{"name":"stderr","output_type":"stream","text":["Running testcases... : 100%|██████████| 4000/4000 [05:29<00:00, 12.14it/s]\n"]},{"data":{"text/plain":[]},"execution_count":14,"metadata":{},"output_type":"execute_result"}],"source":["harness.run()"]},{"cell_type":"markdown","metadata":{"id":"191O2oaUGWrH"},"source":["Called after harness.generate() and is to used to run all the tests.  Returns a pass/fail flag for each test."]},{"cell_type":"code","execution_count":null,"metadata":{"id":"XDbd1mpREWR5","outputId":"872d7612-e0dc-435f-932c-3e74406f38e3"},"outputs":[{"data":{"text/html":["<div>\n","<style scoped>\n","    .dataframe tbody tr th:only-of-type {\n","        vertical-align: middle;\n","    }\n","\n","    .dataframe tbody tr th {\n","        vertical-align: top;\n","    }\n","\n","    .dataframe thead th {\n","        text-align: right;\n","    }\n","</style>\n","<table border=\"1\" class=\"dataframe\">\n","  <thead>\n","    <tr style=\"text-align: right;\">\n","      <th></th>\n","      <th>category</th>\n","      <th>test_type</th>\n","      <th>original</th>\n","      <th>test_case</th>\n","      <th>expected_result</th>\n","      <th>actual_result</th>\n","      <th>pass</th>\n","    </tr>\n","  </thead>\n","  <tbody>\n","    <tr>\n","      <th>0</th>\n","      <td>robustness</td>\n","      <td>lowercase</td>\n","      <td>hide new secretions from the parental units</td>\n","      <td>hide new secretions from the parental units</td>\n","      <td>NEGATIVE</td>\n","      <td>NEGATIVE</td>\n","      <td>True</td>\n","    </tr>\n","    <tr>\n","      <th>1</th>\n","      <td>robustness</td>\n","      <td>lowercase</td>\n","      <td>contains no wit , only labored gags</td>\n","      <td>contains no wit , only labored gags</td>\n","      <td>NEGATIVE</td>\n","      <td>NEGATIVE</td>\n","      <td>True</td>\n","    </tr>\n","    <tr>\n","      <th>2</th>\n","      <td>robustness</td>\n","      <td>lowercase</td>\n","      <td>that loves its characters and communicates som...</td>\n","      <td>that loves its characters and communicates som...</td>\n","      <td>POSITIVE</td>\n","      <td>POSITIVE</td>\n","      <td>True</td>\n","    </tr>\n","    <tr>\n","      <th>3</th>\n","      <td>robustness</td>\n","      <td>lowercase</td>\n","      <td>remains utterly satisfied to remain the same t...</td>\n","      <td>remains utterly satisfied to remain the same t...</td>\n","      <td>NEGATIVE</td>\n","      <td>NEGATIVE</td>\n","      <td>True</td>\n","    </tr>\n","    <tr>\n","      <th>4</th>\n","      <td>robustness</td>\n","      <td>lowercase</td>\n","      <td>on the worst revenge-of-the-nerds clichés the ...</td>\n","      <td>on the worst revenge-of-the-nerds clichés the ...</td>\n","      <td>NEGATIVE</td>\n","      <td>NEGATIVE</td>\n","      <td>True</td>\n","    </tr>\n","    <tr>\n","      <th>...</th>\n","      <td>...</td>\n","      <td>...</td>\n","      <td>...</td>\n","      <td>...</td>\n","      <td>...</td>\n","      <td>...</td>\n","      <td>...</td>\n","    </tr>\n","    <tr>\n","      <th>3995</th>\n","      <td>robustness</td>\n","      <td>uppercase</td>\n","      <td>when there 's nothing else happening</td>\n","      <td>WHEN THERE 'S NOTHING ELSE HAPPENING</td>\n","      <td>NEGATIVE</td>\n","      <td>NEGATIVE</td>\n","      <td>True</td>\n","    </tr>\n","    <tr>\n","      <th>3996</th>\n","      <td>robustness</td>\n","      <td>uppercase</td>\n","      <td>on cable</td>\n","      <td>ON CABLE</td>\n","      <td>NEGATIVE</td>\n","      <td>NEGATIVE</td>\n","      <td>True</td>\n","    </tr>\n","    <tr>\n","      <th>3997</th>\n","      <td>robustness</td>\n","      <td>uppercase</td>\n","      <td>it with ring ,</td>\n","      <td>IT WITH RING ,</td>\n","      <td>POSITIVE</td>\n","      <td>POSITIVE</td>\n","      <td>True</td>\n","    </tr>\n","    <tr>\n","      <th>3998</th>\n","      <td>robustness</td>\n","      <td>uppercase</td>\n","      <td>far from a groundbreaking endeavor</td>\n","      <td>FAR FROM A GROUNDBREAKING ENDEAVOR</td>\n","      <td>NEGATIVE</td>\n","      <td>NEGATIVE</td>\n","      <td>True</td>\n","    </tr>\n","    <tr>\n","      <th>3999</th>\n","      <td>robustness</td>\n","      <td>uppercase</td>\n","      <td>that these women are spectacular</td>\n","      <td>THAT THESE WOMEN ARE SPECTACULAR</td>\n","      <td>POSITIVE</td>\n","      <td>POSITIVE</td>\n","      <td>True</td>\n","    </tr>\n","  </tbody>\n","</table>\n","<p>4000 rows × 7 columns</p>\n","</div>"],"text/plain":["        category  test_type  \\\n","0     robustness  lowercase   \n","1     robustness  lowercase   \n","2     robustness  lowercase   \n","3     robustness  lowercase   \n","4     robustness  lowercase   \n","...          ...        ...   \n","3995  robustness  uppercase   \n","3996  robustness  uppercase   \n","3997  robustness  uppercase   \n","3998  robustness  uppercase   \n","3999  robustness  uppercase   \n","\n","                                               original  \\\n","0          hide new secretions from the parental units    \n","1                  contains no wit , only labored gags    \n","2     that loves its characters and communicates som...   \n","3     remains utterly satisfied to remain the same t...   \n","4     on the worst revenge-of-the-nerds clichés the ...   \n","...                                                 ...   \n","3995              when there 's nothing else happening    \n","3996                                          on cable    \n","3997                                    it with ring ,    \n","3998                far from a groundbreaking endeavor    \n","3999                  that these women are spectacular    \n","\n","                                              test_case expected_result  \\\n","0          hide new secretions from the parental units         NEGATIVE   \n","1                  contains no wit , only labored gags         NEGATIVE   \n","2     that loves its characters and communicates som...        POSITIVE   \n","3     remains utterly satisfied to remain the same t...        NEGATIVE   \n","4     on the worst revenge-of-the-nerds clichés the ...        NEGATIVE   \n","...                                                 ...             ...   \n","3995              WHEN THERE 'S NOTHING ELSE HAPPENING         NEGATIVE   \n","3996                                          ON CABLE         NEGATIVE   \n","3997                                    IT WITH RING ,         POSITIVE   \n","3998                FAR FROM A GROUNDBREAKING ENDEAVOR         NEGATIVE   \n","3999                  THAT THESE WOMEN ARE SPECTACULAR         POSITIVE   \n","\n","     actual_result  pass  \n","0         NEGATIVE  True  \n","1         NEGATIVE  True  \n","2         POSITIVE  True  \n","3         NEGATIVE  True  \n","4         NEGATIVE  True  \n","...            ...   ...  \n","3995      NEGATIVE  True  \n","3996      NEGATIVE  True  \n","3997      POSITIVE  True  \n","3998      NEGATIVE  True  \n","3999      POSITIVE  True  \n","\n","[4000 rows x 7 columns]"]},"execution_count":15,"metadata":{},"output_type":"execute_result"}],"source":["harness.generated_results()"]},{"cell_type":"markdown","metadata":{"id":"TKB8Rsr2GZME"},"source":["This method returns the generated results in the form of a pandas dataframe, which provides a convenient and easy-to-use format for working with the test results. You can use this method to quickly identify the test cases that failed and to determine where fixes are needed."]},{"cell_type":"markdown","metadata":{"id":"PBSlpWnUU55G"},"source":["### Final Results"]},{"cell_type":"markdown","metadata":{"id":"umnEgUHM8DRA"},"source":["We can call `.report()` which summarizes the results giving information about pass and fail counts and overall test pass/fail flag."]},{"cell_type":"code","execution_count":null,"metadata":{"colab":{"base_uri":"https://localhost:8080/","height":112},"id":"gp57HcF9yxi7","outputId":"b893072f-102a-45a6-be03-d737996e659c"},"outputs":[{"data":{"text/html":["<div>\n","<style scoped>\n","    .dataframe tbody tr th:only-of-type {\n","        vertical-align: middle;\n","    }\n","\n","    .dataframe tbody tr th {\n","        vertical-align: top;\n","    }\n","\n","    .dataframe thead th {\n","        text-align: right;\n","    }\n","</style>\n","<table border=\"1\" class=\"dataframe\">\n","  <thead>\n","    <tr style=\"text-align: right;\">\n","      <th></th>\n","      <th>category</th>\n","      <th>test_type</th>\n","      <th>fail_count</th>\n","      <th>pass_count</th>\n","      <th>pass_rate</th>\n","      <th>minimum_pass_rate</th>\n","      <th>pass</th>\n","    </tr>\n","  </thead>\n","  <tbody>\n","    <tr>\n","      <th>0</th>\n","      <td>robustness</td>\n","      <td>lowercase</td>\n","      <td>0</td>\n","      <td>2000</td>\n","      <td>100%</td>\n","      <td>66%</td>\n","      <td>True</td>\n","    </tr>\n","    <tr>\n","      <th>1</th>\n","      <td>robustness</td>\n","      <td>uppercase</td>\n","      <td>0</td>\n","      <td>2000</td>\n","      <td>100%</td>\n","      <td>66%</td>\n","      <td>True</td>\n","    </tr>\n","  </tbody>\n","</table>\n","</div>"],"text/plain":["     category  test_type  fail_count  pass_count pass_rate minimum_pass_rate  \\\n","0  robustness  lowercase           0        2000      100%               66%   \n","1  robustness  uppercase           0        2000      100%               66%   \n","\n","   pass  \n","0  True  \n","1  True  "]},"execution_count":16,"metadata":{},"output_type":"execute_result"}],"source":["harness.report()"]},{"cell_type":"markdown","metadata":{"id":"5N0cKfKiLsiQ"},"source":["## `Imdb` Dataset Testing\n","-------------------\n"]},{"cell_type":"markdown","metadata":{"id":"l5H75bwe8DRA"},"source":["We can also use another dataset to test"]},{"cell_type":"markdown","metadata":{"id":"Ny0585_H8DRA"},"source":["### Harness and Its Parameters"]},{"cell_type":"code","execution_count":null,"metadata":{"id":"oDh3Zaa9EDfZ"},"outputs":[],"source":["harness = Harness(task=\"text-classification\", hub=\"huggingface\",\n","                    model=\"lvwerra/distilbert-imdb\",\n","                    data={\"name\":'imdb'})"]},{"cell_type":"markdown","metadata":{"id":"LIK5jh0x8DRB"},"source":["We have specified task as `text-classification` , hub as `huggingface` and model as `lvwerra/distilbert-imdb`\n","\n","For dataset we used `imdb`. With default parameters for feature_column, target_column and split\n","\n","You can find more HuggingFace Benchmark Datasets [here](https://huggingface.co/datasets?task_categories=task_categories:text-classification&sort=downloads)"]},{"cell_type":"markdown","metadata":{"id":"uTbZ_qJV8DRB"},"source":["### Setup and Configure Harness"]},{"cell_type":"code","execution_count":null,"metadata":{"colab":{"base_uri":"https://localhost:8080/"},"id":"ZnLWJkPVEDmg","outputId":"92ca0633-a1c6-4de3-f9fd-c77e6bcb5374"},"outputs":[{"data":{"text/plain":["{'tests': {'defaults': {'min_pass_rate': 0.65},\n","  'robustness': {'lowercase': {'min_pass_rate': 0.66},\n","   'uppercase': {'min_pass_rate': 0.66}}}}"]},"execution_count":28,"metadata":{},"output_type":"execute_result"}],"source":["harness.configure(\n","{\n"," 'tests': {'defaults': {'min_pass_rate': 0.65},\n","           'robustness': {'lowercase': {'min_pass_rate': 0.66},\n","                          'uppercase': {'min_pass_rate': 0.66},\n","                        }\n","          }\n"," }\n"," )"]},{"cell_type":"code","execution_count":null,"metadata":{"id":"VdLgXi968DRB"},"outputs":[],"source":["# Limit the data to the first 2000 samples\n","harness.data = harness.data[:2000]"]},{"cell_type":"code","execution_count":null,"metadata":{"colab":{"base_uri":"https://localhost:8080/"},"id":"A3U0kM62EG6B","outputId":"1ad54c30-3371-41b6-e85c-4dc69ffcd8aa"},"outputs":[{"name":"stderr","output_type":"stream","text":["Generating testcases...: 100%|██████████| 1/1 [00:00<?, ?it/s]\n"]},{"data":{"text/plain":[]},"execution_count":31,"metadata":{},"output_type":"execute_result"}],"source":["harness.generate()"]},{"cell_type":"markdown","metadata":{"id":"JR7kIfZY8DRJ"},"source":["### Generating the test cases."]},{"cell_type":"code","execution_count":null,"metadata":{"colab":{"base_uri":"https://localhost:8080/","height":423},"id":"0ipwMsy0EHYY","outputId":"a3e9c1fb-22dc-416d-8cf6-043ac4b28432"},"outputs":[{"data":{"text/html":["<div>\n","<style scoped>\n","    .dataframe tbody tr th:only-of-type {\n","        vertical-align: middle;\n","    }\n","\n","    .dataframe tbody tr th {\n","        vertical-align: top;\n","    }\n","\n","    .dataframe thead th {\n","        text-align: right;\n","    }\n","</style>\n","<table border=\"1\" class=\"dataframe\">\n","  <thead>\n","    <tr style=\"text-align: right;\">\n","      <th></th>\n","      <th>category</th>\n","      <th>test_type</th>\n","      <th>original</th>\n","      <th>test_case</th>\n","      <th>expected_result</th>\n","    </tr>\n","  </thead>\n","  <tbody>\n","    <tr>\n","      <th>0</th>\n","      <td>robustness</td>\n","      <td>lowercase</td>\n","      <td>I love sci-fi and am willing to put up with a ...</td>\n","      <td>i love sci-fi and am willing to put up with a ...</td>\n","      <td>NEGATIVE</td>\n","    </tr>\n","    <tr>\n","      <th>1</th>\n","      <td>robustness</td>\n","      <td>lowercase</td>\n","      <td>Worth the entertainment value of a rental, esp...</td>\n","      <td>worth the entertainment value of a rental, esp...</td>\n","      <td>NEGATIVE</td>\n","    </tr>\n","    <tr>\n","      <th>2</th>\n","      <td>robustness</td>\n","      <td>lowercase</td>\n","      <td>its a totally average film with a few semi-alr...</td>\n","      <td>its a totally average film with a few semi-alr...</td>\n","      <td>NEGATIVE</td>\n","    </tr>\n","    <tr>\n","      <th>3</th>\n","      <td>robustness</td>\n","      <td>lowercase</td>\n","      <td>STAR RATING: ***** Saturday Night **** Friday ...</td>\n","      <td>star rating: ***** saturday night **** friday ...</td>\n","      <td>NEGATIVE</td>\n","    </tr>\n","    <tr>\n","      <th>4</th>\n","      <td>robustness</td>\n","      <td>lowercase</td>\n","      <td>First off let me say, If you haven't enjoyed a...</td>\n","      <td>first off let me say, if you haven't enjoyed a...</td>\n","      <td>POSITIVE</td>\n","    </tr>\n","    <tr>\n","      <th>...</th>\n","      <td>...</td>\n","      <td>...</td>\n","      <td>...</td>\n","      <td>...</td>\n","      <td>...</td>\n","    </tr>\n","    <tr>\n","      <th>3995</th>\n","      <td>robustness</td>\n","      <td>uppercase</td>\n","      <td>A rather disappointing film. The club scenes w...</td>\n","      <td>A RATHER DISAPPOINTING FILM. THE CLUB SCENES W...</td>\n","      <td>NEGATIVE</td>\n","    </tr>\n","    <tr>\n","      <th>3996</th>\n","      <td>robustness</td>\n","      <td>uppercase</td>\n","      <td>There were so many reasons why this movie coul...</td>\n","      <td>THERE WERE SO MANY REASONS WHY THIS MOVIE COUL...</td>\n","      <td>NEGATIVE</td>\n","    </tr>\n","    <tr>\n","      <th>3997</th>\n","      <td>robustness</td>\n","      <td>uppercase</td>\n","      <td>After Kenneth Opel's rousing story of the invi...</td>\n","      <td>AFTER KENNETH OPEL'S ROUSING STORY OF THE INVI...</td>\n","      <td>NEGATIVE</td>\n","    </tr>\n","    <tr>\n","      <th>3998</th>\n","      <td>robustness</td>\n","      <td>uppercase</td>\n","      <td>Having already seen the original \"Jack Frost\",...</td>\n","      <td>HAVING ALREADY SEEN THE ORIGINAL \"JACK FROST\",...</td>\n","      <td>NEGATIVE</td>\n","    </tr>\n","    <tr>\n","      <th>3999</th>\n","      <td>robustness</td>\n","      <td>uppercase</td>\n","      <td>Ill-conceived sequel(..the absurd idea of havi...</td>\n","      <td>ILL-CONCEIVED SEQUEL(..THE ABSURD IDEA OF HAVI...</td>\n","      <td>NEGATIVE</td>\n","    </tr>\n","  </tbody>\n","</table>\n","<p>4000 rows × 5 columns</p>\n","</div>"],"text/plain":["        category  test_type  \\\n","0     robustness  lowercase   \n","1     robustness  lowercase   \n","2     robustness  lowercase   \n","3     robustness  lowercase   \n","4     robustness  lowercase   \n","...          ...        ...   \n","3995  robustness  uppercase   \n","3996  robustness  uppercase   \n","3997  robustness  uppercase   \n","3998  robustness  uppercase   \n","3999  robustness  uppercase   \n","\n","                                               original  \\\n","0     I love sci-fi and am willing to put up with a ...   \n","1     Worth the entertainment value of a rental, esp...   \n","2     its a totally average film with a few semi-alr...   \n","3     STAR RATING: ***** Saturday Night **** Friday ...   \n","4     First off let me say, If you haven't enjoyed a...   \n","...                                                 ...   \n","3995  A rather disappointing film. The club scenes w...   \n","3996  There were so many reasons why this movie coul...   \n","3997  After Kenneth Opel's rousing story of the invi...   \n","3998  Having already seen the original \"Jack Frost\",...   \n","3999  Ill-conceived sequel(..the absurd idea of havi...   \n","\n","                                              test_case expected_result  \n","0     i love sci-fi and am willing to put up with a ...        NEGATIVE  \n","1     worth the entertainment value of a rental, esp...        NEGATIVE  \n","2     its a totally average film with a few semi-alr...        NEGATIVE  \n","3     star rating: ***** saturday night **** friday ...        NEGATIVE  \n","4     first off let me say, if you haven't enjoyed a...        POSITIVE  \n","...                                                 ...             ...  \n","3995  A RATHER DISAPPOINTING FILM. THE CLUB SCENES W...        NEGATIVE  \n","3996  THERE WERE SO MANY REASONS WHY THIS MOVIE COUL...        NEGATIVE  \n","3997  AFTER KENNETH OPEL'S ROUSING STORY OF THE INVI...        NEGATIVE  \n","3998  HAVING ALREADY SEEN THE ORIGINAL \"JACK FROST\",...        NEGATIVE  \n","3999  ILL-CONCEIVED SEQUEL(..THE ABSURD IDEA OF HAVI...        NEGATIVE  \n","\n","[4000 rows x 5 columns]"]},"execution_count":32,"metadata":{},"output_type":"execute_result"}],"source":["harness.testcases()"]},{"cell_type":"markdown","metadata":{"id":"1WtdwEZL8DRJ"},"source":["### Running the tests"]},{"cell_type":"code","execution_count":null,"metadata":{"colab":{"base_uri":"https://localhost:8080/"},"id":"0Nic5HRZEJu5","outputId":"dbbf911a-413e-479c-996b-98430920f0b5"},"outputs":[{"name":"stderr","output_type":"stream","text":["Running testcases... : 100%|██████████| 4000/4000 [43:06<00:00,  1.55it/s]\n"]},{"data":{"text/plain":[]},"execution_count":33,"metadata":{},"output_type":"execute_result"}],"source":["harness.run()"]},{"cell_type":"code","execution_count":null,"metadata":{"id":"BjZc-ZcCELbU","outputId":"5913de81-5f5d-4978-a1dc-f6cc1f0f2e7d"},"outputs":[{"data":{"text/html":["<div>\n","<style scoped>\n","    .dataframe tbody tr th:only-of-type {\n","        vertical-align: middle;\n","    }\n","\n","    .dataframe tbody tr th {\n","        vertical-align: top;\n","    }\n","\n","    .dataframe thead th {\n","        text-align: right;\n","    }\n","</style>\n","<table border=\"1\" class=\"dataframe\">\n","  <thead>\n","    <tr style=\"text-align: right;\">\n","      <th></th>\n","      <th>category</th>\n","      <th>test_type</th>\n","      <th>original</th>\n","      <th>test_case</th>\n","      <th>expected_result</th>\n","      <th>actual_result</th>\n","      <th>pass</th>\n","    </tr>\n","  </thead>\n","  <tbody>\n","    <tr>\n","      <th>0</th>\n","      <td>robustness</td>\n","      <td>lowercase</td>\n","      <td>I love sci-fi and am willing to put up with a ...</td>\n","      <td>i love sci-fi and am willing to put up with a ...</td>\n","      <td>NEGATIVE</td>\n","      <td>NEGATIVE</td>\n","      <td>True</td>\n","    </tr>\n","    <tr>\n","      <th>1</th>\n","      <td>robustness</td>\n","      <td>lowercase</td>\n","      <td>Worth the entertainment value of a rental, esp...</td>\n","      <td>worth the entertainment value of a rental, esp...</td>\n","      <td>NEGATIVE</td>\n","      <td>NEGATIVE</td>\n","      <td>True</td>\n","    </tr>\n","    <tr>\n","      <th>2</th>\n","      <td>robustness</td>\n","      <td>lowercase</td>\n","      <td>its a totally average film with a few semi-alr...</td>\n","      <td>its a totally average film with a few semi-alr...</td>\n","      <td>NEGATIVE</td>\n","      <td>NEGATIVE</td>\n","      <td>True</td>\n","    </tr>\n","    <tr>\n","      <th>3</th>\n","      <td>robustness</td>\n","      <td>lowercase</td>\n","      <td>STAR RATING: ***** Saturday Night **** Friday ...</td>\n","      <td>star rating: ***** saturday night **** friday ...</td>\n","      <td>NEGATIVE</td>\n","      <td>NEGATIVE</td>\n","      <td>True</td>\n","    </tr>\n","    <tr>\n","      <th>4</th>\n","      <td>robustness</td>\n","      <td>lowercase</td>\n","      <td>First off let me say, If you haven't enjoyed a...</td>\n","      <td>first off let me say, if you haven't enjoyed a...</td>\n","      <td>POSITIVE</td>\n","      <td>POSITIVE</td>\n","      <td>True</td>\n","    </tr>\n","    <tr>\n","      <th>...</th>\n","      <td>...</td>\n","      <td>...</td>\n","      <td>...</td>\n","      <td>...</td>\n","      <td>...</td>\n","      <td>...</td>\n","      <td>...</td>\n","    </tr>\n","    <tr>\n","      <th>3995</th>\n","      <td>robustness</td>\n","      <td>uppercase</td>\n","      <td>A rather disappointing film. The club scenes w...</td>\n","      <td>A RATHER DISAPPOINTING FILM. THE CLUB SCENES W...</td>\n","      <td>NEGATIVE</td>\n","      <td>NEGATIVE</td>\n","      <td>True</td>\n","    </tr>\n","    <tr>\n","      <th>3996</th>\n","      <td>robustness</td>\n","      <td>uppercase</td>\n","      <td>There were so many reasons why this movie coul...</td>\n","      <td>THERE WERE SO MANY REASONS WHY THIS MOVIE COUL...</td>\n","      <td>NEGATIVE</td>\n","      <td>NEGATIVE</td>\n","      <td>True</td>\n","    </tr>\n","    <tr>\n","      <th>3997</th>\n","      <td>robustness</td>\n","      <td>uppercase</td>\n","      <td>After Kenneth Opel's rousing story of the invi...</td>\n","      <td>AFTER KENNETH OPEL'S ROUSING STORY OF THE INVI...</td>\n","      <td>NEGATIVE</td>\n","      <td>NEGATIVE</td>\n","      <td>True</td>\n","    </tr>\n","    <tr>\n","      <th>3998</th>\n","      <td>robustness</td>\n","      <td>uppercase</td>\n","      <td>Having already seen the original \"Jack Frost\",...</td>\n","      <td>HAVING ALREADY SEEN THE ORIGINAL \"JACK FROST\",...</td>\n","      <td>NEGATIVE</td>\n","      <td>NEGATIVE</td>\n","      <td>True</td>\n","    </tr>\n","    <tr>\n","      <th>3999</th>\n","      <td>robustness</td>\n","      <td>uppercase</td>\n","      <td>Ill-conceived sequel(..the absurd idea of havi...</td>\n","      <td>ILL-CONCEIVED SEQUEL(..THE ABSURD IDEA OF HAVI...</td>\n","      <td>NEGATIVE</td>\n","      <td>NEGATIVE</td>\n","      <td>True</td>\n","    </tr>\n","  </tbody>\n","</table>\n","<p>4000 rows × 7 columns</p>\n","</div>"],"text/plain":["        category  test_type  \\\n","0     robustness  lowercase   \n","1     robustness  lowercase   \n","2     robustness  lowercase   \n","3     robustness  lowercase   \n","4     robustness  lowercase   \n","...          ...        ...   \n","3995  robustness  uppercase   \n","3996  robustness  uppercase   \n","3997  robustness  uppercase   \n","3998  robustness  uppercase   \n","3999  robustness  uppercase   \n","\n","                                               original  \\\n","0     I love sci-fi and am willing to put up with a ...   \n","1     Worth the entertainment value of a rental, esp...   \n","2     its a totally average film with a few semi-alr...   \n","3     STAR RATING: ***** Saturday Night **** Friday ...   \n","4     First off let me say, If you haven't enjoyed a...   \n","...                                                 ...   \n","3995  A rather disappointing film. The club scenes w...   \n","3996  There were so many reasons why this movie coul...   \n","3997  After Kenneth Opel's rousing story of the invi...   \n","3998  Having already seen the original \"Jack Frost\",...   \n","3999  Ill-conceived sequel(..the absurd idea of havi...   \n","\n","                                              test_case expected_result  \\\n","0     i love sci-fi and am willing to put up with a ...        NEGATIVE   \n","1     worth the entertainment value of a rental, esp...        NEGATIVE   \n","2     its a totally average film with a few semi-alr...        NEGATIVE   \n","3     star rating: ***** saturday night **** friday ...        NEGATIVE   \n","4     first off let me say, if you haven't enjoyed a...        POSITIVE   \n","...                                                 ...             ...   \n","3995  A RATHER DISAPPOINTING FILM. THE CLUB SCENES W...        NEGATIVE   \n","3996  THERE WERE SO MANY REASONS WHY THIS MOVIE COUL...        NEGATIVE   \n","3997  AFTER KENNETH OPEL'S ROUSING STORY OF THE INVI...        NEGATIVE   \n","3998  HAVING ALREADY SEEN THE ORIGINAL \"JACK FROST\",...        NEGATIVE   \n","3999  ILL-CONCEIVED SEQUEL(..THE ABSURD IDEA OF HAVI...        NEGATIVE   \n","\n","     actual_result  pass  \n","0         NEGATIVE  True  \n","1         NEGATIVE  True  \n","2         NEGATIVE  True  \n","3         NEGATIVE  True  \n","4         POSITIVE  True  \n","...            ...   ...  \n","3995      NEGATIVE  True  \n","3996      NEGATIVE  True  \n","3997      NEGATIVE  True  \n","3998      NEGATIVE  True  \n","3999      NEGATIVE  True  \n","\n","[4000 rows x 7 columns]"]},"execution_count":34,"metadata":{},"output_type":"execute_result"}],"source":["harness.generated_results()"]},{"cell_type":"markdown","metadata":{"id":"aQw2X-IG8DRK"},"source":["### Final Report\n","\n","We can call `.report()` which summarizes the results giving information about pass and fail counts and overall test pass/fail flag."]},{"cell_type":"code","execution_count":null,"metadata":{"colab":{"base_uri":"https://localhost:8080/","height":112},"id":"PlrAxK1eENmh","outputId":"7fd59473-20ac-402b-a39b-e5e3e29cf1f4"},"outputs":[{"data":{"text/html":["<div>\n","<style scoped>\n","    .dataframe tbody tr th:only-of-type {\n","        vertical-align: middle;\n","    }\n","\n","    .dataframe tbody tr th {\n","        vertical-align: top;\n","    }\n","\n","    .dataframe thead th {\n","        text-align: right;\n","    }\n","</style>\n","<table border=\"1\" class=\"dataframe\">\n","  <thead>\n","    <tr style=\"text-align: right;\">\n","      <th></th>\n","      <th>category</th>\n","      <th>test_type</th>\n","      <th>fail_count</th>\n","      <th>pass_count</th>\n","      <th>pass_rate</th>\n","      <th>minimum_pass_rate</th>\n","      <th>pass</th>\n","    </tr>\n","  </thead>\n","  <tbody>\n","    <tr>\n","      <th>0</th>\n","      <td>robustness</td>\n","      <td>lowercase</td>\n","      <td>0</td>\n","      <td>2000</td>\n","      <td>100%</td>\n","      <td>66%</td>\n","      <td>True</td>\n","    </tr>\n","    <tr>\n","      <th>1</th>\n","      <td>robustness</td>\n","      <td>uppercase</td>\n","      <td>0</td>\n","      <td>2000</td>\n","      <td>100%</td>\n","      <td>66%</td>\n","      <td>True</td>\n","    </tr>\n","  </tbody>\n","</table>\n","</div>"],"text/plain":["     category  test_type  fail_count  pass_count pass_rate minimum_pass_rate  \\\n","0  robustness  lowercase           0        2000      100%               66%   \n","1  robustness  uppercase           0        2000      100%               66%   \n","\n","   pass  \n","0  True  \n","1  True  "]},"execution_count":35,"metadata":{},"output_type":"execute_result"}],"source":["harness.report()"]},{"cell_type":"markdown","metadata":{"id":"emrRp2vlF1T1"},"source":["# HuggingFace Datasets Testing For `summarization`\n","\n","In this section, we dive into testing of HuggingFace Models for different HuggingFace Datasets."]},{"cell_type":"markdown","metadata":{"id":"EsCtdb6cF9IN"},"source":["## samsum - Dataset Testing\n","-------------------"]},{"cell_type":"markdown","metadata":{"id":"vzC8J9SxFnqP"},"source":["### Set environment for OpenAI"]},{"cell_type":"code","execution_count":null,"metadata":{},"outputs":[],"source":["!pip install openai"]},{"cell_type":"code","execution_count":32,"metadata":{"executionInfo":{"elapsed":6,"status":"ok","timestamp":1689533812752,"user":{"displayName":"Prikshit Sharma","userId":"02603035510215936699"},"user_tz":-330},"id":"rNeyF-FC_j82"},"outputs":[],"source":["import os\n","\n","import openai\n","\n","os.environ[\"OPENAI_API_KEY\"] = \"<YOUR_API_KEY>\""]},{"cell_type":"markdown","metadata":{"id":"B01bfV9pFhg-"},"source":["### Setup and configure harness"]},{"cell_type":"code","execution_count":33,"metadata":{"colab":{"base_uri":"https://localhost:8080/"},"executionInfo":{"elapsed":559,"status":"ok","timestamp":1689533813306,"user":{"displayName":"Prikshit Sharma","userId":"02603035510215936699"},"user_tz":-330},"id":"8WMCEFwT_d1P","outputId":"98954e80-b979-4f14-91b1-e3fd6863de4c"},"outputs":[{"name":"stdout","output_type":"stream","text":["Test Configuration : \n"," {\n"," \"model_parameters\": {\n","  \"temperature\": 0.2,\n","  \"max_tokens\": 64\n"," },\n"," \"tests\": {\n","  \"defaults\": {\n","   \"min_pass_rate\": 1.0\n","  },\n","  \"robustness\": {\n","   \"add_typo\": {\n","    \"min_pass_rate\": 0.7\n","   },\n","   \"lowercase\": {\n","    \"min_pass_rate\": 0.7\n","   }\n","  }\n"," }\n","}\n"]}],"source":["harness = Harness(task=\"summarization\", hub=\"openai\",\n","                   model=\"text-davinci-003\",\n","                  data={\"name\":'samsum',\n","                  \"feature_column\":\"dialogue\",\n","                  \"target_column\":'summary',\n","                  \"split\":\"test\"\n","                  })"]},{"cell_type":"markdown","metadata":{"id":"qRtXmy4GFXwP"},"source":["### Configure the Tests\n","We can use the .configure() method to manually configure the tests we want to perform."]},{"cell_type":"code","execution_count":34,"metadata":{"colab":{"base_uri":"https://localhost:8080/"},"executionInfo":{"elapsed":2,"status":"ok","timestamp":1689533813901,"user":{"displayName":"Prikshit Sharma","userId":"02603035510215936699"},"user_tz":-330},"id":"dhmCGPALAPJv","outputId":"a164dfea-6b0d-4080-f548-156a49867907"},"outputs":[{"data":{"text/plain":["{'tests': {'defaults': {'min_pass_rate': 0.65},\n","  'robustness': {'uppercase': {'min_pass_rate': 0.66},\n","   'lowercase': {'min_pass_rate': 0.6}}}}"]},"execution_count":34,"metadata":{},"output_type":"execute_result"}],"source":["harness.configure(\n","{\n"," 'tests': {'defaults': {'min_pass_rate': 0.65},\n","           'robustness': {'uppercase': {'min_pass_rate': 0.66},\n","                          'lowercase':{'min_pass_rate': 0.60},\n","                        }\n","          }\n"," }\n"," )"]},{"cell_type":"markdown","metadata":{"id":"qoAgrQQbGMC2"},"source":["Here we have configured the harness to perform two robustness tests (uppercase and lowercase) and defined the minimum pass rate for each test."]},{"cell_type":"code","execution_count":35,"metadata":{"executionInfo":{"elapsed":2,"status":"ok","timestamp":1689533817937,"user":{"displayName":"Prikshit Sharma","userId":"02603035510215936699"},"user_tz":-330},"id":"PYQGX2OdBDd1"},"outputs":[],"source":["harness.data=harness.data[0:5]"]},{"cell_type":"markdown","metadata":{"id":"jtHbCs1kFNYn"},"source":["### Generating the test cases."]},{"cell_type":"code","execution_count":36,"metadata":{"colab":{"base_uri":"https://localhost:8080/"},"executionInfo":{"elapsed":4,"status":"ok","timestamp":1689533818349,"user":{"displayName":"Prikshit Sharma","userId":"02603035510215936699"},"user_tz":-330},"id":"LxqMY_FjA_Pp","outputId":"3b5c6e44-9174-4143-98b9-00ba4f823de7"},"outputs":[{"name":"stderr","output_type":"stream","text":["\n","Generating testcases...: 100%|██████████| 1/1 [00:00<00:00, 6269.51it/s]\n"]},{"data":{"text/plain":[]},"execution_count":36,"metadata":{},"output_type":"execute_result"}],"source":["harness.generate()"]},{"cell_type":"markdown","metadata":{"id":"N-jdxDdBFKgl"},"source":["harness.generate() method automatically generates the test cases (based on the provided configuration)"]},{"cell_type":"code","execution_count":37,"metadata":{"colab":{"base_uri":"https://localhost:8080/","height":363},"executionInfo":{"elapsed":3,"status":"ok","timestamp":1689533819321,"user":{"displayName":"Prikshit Sharma","userId":"02603035510215936699"},"user_tz":-330},"id":"Yo5Q6VfVBASF","outputId":"a5af73e9-616e-49e9-f0d3-6846c10185ab"},"outputs":[{"data":{"text/html":["\n","\n","  <div id=\"df-d32e911c-ee6d-4c2e-af4b-1520ea542d58\">\n","    <div class=\"colab-df-container\">\n","      <div>\n","<style scoped>\n","    .dataframe tbody tr th:only-of-type {\n","        vertical-align: middle;\n","    }\n","\n","    .dataframe tbody tr th {\n","        vertical-align: top;\n","    }\n","\n","    .dataframe thead th {\n","        text-align: right;\n","    }\n","</style>\n","<table border=\"1\" class=\"dataframe\">\n","  <thead>\n","    <tr style=\"text-align: right;\">\n","      <th></th>\n","      <th>category</th>\n","      <th>test_type</th>\n","      <th>original</th>\n","      <th>test_case</th>\n","    </tr>\n","  </thead>\n","  <tbody>\n","    <tr>\n","      <th>0</th>\n","      <td>robustness</td>\n","      <td>uppercase</td>\n","      <td>Hannah: Hey, do you have Betty's number?\\nAman...</td>\n","      <td>HANNAH: HEY, DO YOU HAVE BETTY'S NUMBER? AMAND...</td>\n","    </tr>\n","    <tr>\n","      <th>1</th>\n","      <td>robustness</td>\n","      <td>uppercase</td>\n","      <td>Eric: MACHINE!\\r\\nRob: That's so gr8!\\r\\nEric:...</td>\n","      <td>ERIC: MACHINE! ROB: THAT'S SO GR8! ERIC: I KNO...</td>\n","    </tr>\n","    <tr>\n","      <th>2</th>\n","      <td>robustness</td>\n","      <td>uppercase</td>\n","      <td>Lenny: Babe, can you help me with something?\\r...</td>\n","      <td>LENNY: BABE, CAN YOU HELP ME WITH SOMETHING? B...</td>\n","    </tr>\n","    <tr>\n","      <th>3</th>\n","      <td>robustness</td>\n","      <td>uppercase</td>\n","      <td>Will: hey babe, what do you want for dinner to...</td>\n","      <td>WILL: HEY BABE, WHAT DO YOU WANT FOR DINNER TO...</td>\n","    </tr>\n","    <tr>\n","      <th>4</th>\n","      <td>robustness</td>\n","      <td>uppercase</td>\n","      <td>Ollie: Hi , are you in Warsaw\\r\\nJane: yes, ju...</td>\n","      <td>OLLIE: HI , ARE YOU IN WARSAW JANE: YES, JUST ...</td>\n","    </tr>\n","    <tr>\n","      <th>5</th>\n","      <td>robustness</td>\n","      <td>lowercase</td>\n","      <td>Hannah: Hey, do you have Betty's number?\\nAman...</td>\n","      <td>hannah: hey, do you have betty's number? amand...</td>\n","    </tr>\n","    <tr>\n","      <th>6</th>\n","      <td>robustness</td>\n","      <td>lowercase</td>\n","      <td>Eric: MACHINE!\\r\\nRob: That's so gr8!\\r\\nEric:...</td>\n","      <td>eric: machine! rob: that's so gr8! eric: i kno...</td>\n","    </tr>\n","    <tr>\n","      <th>7</th>\n","      <td>robustness</td>\n","      <td>lowercase</td>\n","      <td>Lenny: Babe, can you help me with something?\\r...</td>\n","      <td>lenny: babe, can you help me with something? b...</td>\n","    </tr>\n","    <tr>\n","      <th>8</th>\n","      <td>robustness</td>\n","      <td>lowercase</td>\n","      <td>Will: hey babe, what do you want for dinner to...</td>\n","      <td>will: hey babe, what do you want for dinner to...</td>\n","    </tr>\n","    <tr>\n","      <th>9</th>\n","      <td>robustness</td>\n","      <td>lowercase</td>\n","      <td>Ollie: Hi , are you in Warsaw\\r\\nJane: yes, ju...</td>\n","      <td>ollie: hi , are you in warsaw jane: yes, just ...</td>\n","    </tr>\n","  </tbody>\n","</table>\n","</div>\n","      <button class=\"colab-df-convert\" onclick=\"convertToInteractive('df-d32e911c-ee6d-4c2e-af4b-1520ea542d58')\"\n","              title=\"Convert this dataframe to an interactive table.\"\n","              style=\"display:none;\">\n","\n","  <svg xmlns=\"http://www.w3.org/2000/svg\" height=\"24px\"viewBox=\"0 0 24 24\"\n","       width=\"24px\">\n","    <path d=\"M0 0h24v24H0V0z\" fill=\"none\"/>\n","    <path d=\"M18.56 5.44l.94 2.06.94-2.06 2.06-.94-2.06-.94-.94-2.06-.94 2.06-2.06.94zm-11 1L8.5 8.5l.94-2.06 2.06-.94-2.06-.94L8.5 2.5l-.94 2.06-2.06.94zm10 10l.94 2.06.94-2.06 2.06-.94-2.06-.94-.94-2.06-.94 2.06-2.06.94z\"/><path d=\"M17.41 7.96l-1.37-1.37c-.4-.4-.92-.59-1.43-.59-.52 0-1.04.2-1.43.59L10.3 9.45l-7.72 7.72c-.78.78-.78 2.05 0 2.83L4 21.41c.39.39.9.59 1.41.59.51 0 1.02-.2 1.41-.59l7.78-7.78 2.81-2.81c.8-.78.8-2.07 0-2.86zM5.41 20L4 18.59l7.72-7.72 1.47 1.35L5.41 20z\"/>\n","  </svg>\n","      </button>\n","\n","\n","\n","    <div id=\"df-af14a5b7-83c5-4e4f-849b-01bc45cd0167\">\n","      <button class=\"colab-df-quickchart\" onclick=\"quickchart('df-af14a5b7-83c5-4e4f-849b-01bc45cd0167')\"\n","              title=\"Suggest charts.\"\n","              style=\"display:none;\">\n","\n","<svg xmlns=\"http://www.w3.org/2000/svg\" height=\"24px\"viewBox=\"0 0 24 24\"\n","     width=\"24px\">\n","    <g>\n","        <path d=\"M19 3H5c-1.1 0-2 .9-2 2v14c0 1.1.9 2 2 2h14c1.1 0 2-.9 2-2V5c0-1.1-.9-2-2-2zM9 17H7v-7h2v7zm4 0h-2V7h2v10zm4 0h-2v-4h2v4z\"/>\n","    </g>\n","</svg>\n","      </button>\n","    </div>\n","\n","<style>\n","  .colab-df-quickchart {\n","    background-color: #E8F0FE;\n","    border: none;\n","    border-radius: 50%;\n","    cursor: pointer;\n","    display: none;\n","    fill: #1967D2;\n","    height: 32px;\n","    padding: 0 0 0 0;\n","    width: 32px;\n","  }\n","\n","  .colab-df-quickchart:hover {\n","    background-color: #E2EBFA;\n","    box-shadow: 0px 1px 2px rgba(60, 64, 67, 0.3), 0px 1px 3px 1px rgba(60, 64, 67, 0.15);\n","    fill: #174EA6;\n","  }\n","\n","  [theme=dark] .colab-df-quickchart {\n","    background-color: #3B4455;\n","    fill: #D2E3FC;\n","  }\n","\n","  [theme=dark] .colab-df-quickchart:hover {\n","    background-color: #434B5C;\n","    box-shadow: 0px 1px 3px 1px rgba(0, 0, 0, 0.15);\n","    filter: drop-shadow(0px 1px 2px rgba(0, 0, 0, 0.3));\n","    fill: #FFFFFF;\n","  }\n","</style>\n","\n","    <script>\n","      async function quickchart(key) {\n","        const containerElement = document.querySelector('#' + key);\n","        const charts = await google.colab.kernel.invokeFunction(\n","            'suggestCharts', [key], {});\n","      }\n","    </script>\n","\n","      <script>\n","\n","function displayQuickchartButton(domScope) {\n","  let quickchartButtonEl =\n","    domScope.querySelector('#df-af14a5b7-83c5-4e4f-849b-01bc45cd0167 button.colab-df-quickchart');\n","  quickchartButtonEl.style.display =\n","    google.colab.kernel.accessAllowed ? 'block' : 'none';\n","}\n","\n","        displayQuickchartButton(document);\n","      </script>\n","      <style>\n","    .colab-df-container {\n","      display:flex;\n","      flex-wrap:wrap;\n","      gap: 12px;\n","    }\n","\n","    .colab-df-convert {\n","      background-color: #E8F0FE;\n","      border: none;\n","      border-radius: 50%;\n","      cursor: pointer;\n","      display: none;\n","      fill: #1967D2;\n","      height: 32px;\n","      padding: 0 0 0 0;\n","      width: 32px;\n","    }\n","\n","    .colab-df-convert:hover {\n","      background-color: #E2EBFA;\n","      box-shadow: 0px 1px 2px rgba(60, 64, 67, 0.3), 0px 1px 3px 1px rgba(60, 64, 67, 0.15);\n","      fill: #174EA6;\n","    }\n","\n","    [theme=dark] .colab-df-convert {\n","      background-color: #3B4455;\n","      fill: #D2E3FC;\n","    }\n","\n","    [theme=dark] .colab-df-convert:hover {\n","      background-color: #434B5C;\n","      box-shadow: 0px 1px 3px 1px rgba(0, 0, 0, 0.15);\n","      filter: drop-shadow(0px 1px 2px rgba(0, 0, 0, 0.3));\n","      fill: #FFFFFF;\n","    }\n","  </style>\n","\n","      <script>\n","        const buttonEl =\n","          document.querySelector('#df-d32e911c-ee6d-4c2e-af4b-1520ea542d58 button.colab-df-convert');\n","        buttonEl.style.display =\n","          google.colab.kernel.accessAllowed ? 'block' : 'none';\n","\n","        async function convertToInteractive(key) {\n","          const element = document.querySelector('#df-d32e911c-ee6d-4c2e-af4b-1520ea542d58');\n","          const dataTable =\n","            await google.colab.kernel.invokeFunction('convertToInteractive',\n","                                                     [key], {});\n","          if (!dataTable) return;\n","\n","          const docLinkHtml = 'Like what you see? Visit the ' +\n","            '<a target=\"_blank\" href=https://colab.research.google.com/notebooks/data_table.ipynb>data table notebook</a>'\n","            + ' to learn more about interactive tables.';\n","          element.innerHTML = '';\n","          dataTable['output_type'] = 'display_data';\n","          await google.colab.output.renderOutput(dataTable, element);\n","          const docLink = document.createElement('div');\n","          docLink.innerHTML = docLinkHtml;\n","          element.appendChild(docLink);\n","        }\n","      </script>\n","    </div>\n","  </div>\n"],"text/plain":["     category  test_type                                           original  \\\n","0  robustness  uppercase  Hannah: Hey, do you have Betty's number?\\nAman...   \n","1  robustness  uppercase  Eric: MACHINE!\\r\\nRob: That's so gr8!\\r\\nEric:...   \n","2  robustness  uppercase  Lenny: Babe, can you help me with something?\\r...   \n","3  robustness  uppercase  Will: hey babe, what do you want for dinner to...   \n","4  robustness  uppercase  Ollie: Hi , are you in Warsaw\\r\\nJane: yes, ju...   \n","5  robustness  lowercase  Hannah: Hey, do you have Betty's number?\\nAman...   \n","6  robustness  lowercase  Eric: MACHINE!\\r\\nRob: That's so gr8!\\r\\nEric:...   \n","7  robustness  lowercase  Lenny: Babe, can you help me with something?\\r...   \n","8  robustness  lowercase  Will: hey babe, what do you want for dinner to...   \n","9  robustness  lowercase  Ollie: Hi , are you in Warsaw\\r\\nJane: yes, ju...   \n","\n","                                           test_case  \n","0  HANNAH: HEY, DO YOU HAVE BETTY'S NUMBER? AMAND...  \n","1  ERIC: MACHINE! ROB: THAT'S SO GR8! ERIC: I KNO...  \n","2  LENNY: BABE, CAN YOU HELP ME WITH SOMETHING? B...  \n","3  WILL: HEY BABE, WHAT DO YOU WANT FOR DINNER TO...  \n","4  OLLIE: HI , ARE YOU IN WARSAW JANE: YES, JUST ...  \n","5  hannah: hey, do you have betty's number? amand...  \n","6  eric: machine! rob: that's so gr8! eric: i kno...  \n","7  lenny: babe, can you help me with something? b...  \n","8  will: hey babe, what do you want for dinner to...  \n","9  ollie: hi , are you in warsaw jane: yes, just ...  "]},"execution_count":37,"metadata":{},"output_type":"execute_result"}],"source":["harness.testcases()"]},{"cell_type":"markdown","metadata":{"id":"q-KYrFMWGSw1"},"source":["harness.testcases() method displays the produced test cases in form of a pandas data frame."]},{"cell_type":"markdown","metadata":{"id":"SkHPaAN7FBSG"},"source":["### Running the tests"]},{"cell_type":"code","execution_count":38,"metadata":{"colab":{"base_uri":"https://localhost:8080/"},"executionInfo":{"elapsed":367293,"status":"ok","timestamp":1689534188020,"user":{"displayName":"Prikshit Sharma","userId":"02603035510215936699"},"user_tz":-330},"id":"Pvqtr_G7BBR0","outputId":"dcf96ef6-c0f2-4e5d-958b-f4a896d5b42a"},"outputs":[{"name":"stderr","output_type":"stream","text":["Running testcases... : 100%|██████████| 10/10 [06:07<00:00, 36.71s/it]\n"]},{"data":{"text/plain":[]},"execution_count":38,"metadata":{},"output_type":"execute_result"}],"source":["harness.run()"]},{"cell_type":"markdown","metadata":{"id":"eQ_ufKmrGVqt"},"source":["Called after harness.generate() and is to used to run all the tests.  Returns a pass/fail flag for each test."]},{"cell_type":"markdown","metadata":{"id":"FpG0SFmnE7Nt"},"source":["### Generated Results"]},{"cell_type":"code","execution_count":42,"metadata":{"colab":{"base_uri":"https://localhost:8080/","height":581},"executionInfo":{"elapsed":4219,"status":"ok","timestamp":1689540121904,"user":{"displayName":"Prikshit Sharma","userId":"02603035510215936699"},"user_tz":-330},"id":"CFXsxZHJDKtj","outputId":"0e37cb99-f2ce-4dde-a237-867e0bca29dd"},"outputs":[{"data":{"text/html":["\n","\n","  <div id=\"df-e0648226-1f58-4c80-9617-d96af6ab7c50\">\n","    <div class=\"colab-df-container\">\n","      <div>\n","<style scoped>\n","    .dataframe tbody tr th:only-of-type {\n","        vertical-align: middle;\n","    }\n","\n","    .dataframe tbody tr th {\n","        vertical-align: top;\n","    }\n","\n","    .dataframe thead th {\n","        text-align: right;\n","    }\n","</style>\n","<table border=\"1\" class=\"dataframe\">\n","  <thead>\n","    <tr style=\"text-align: right;\">\n","      <th></th>\n","      <th>category</th>\n","      <th>test_type</th>\n","      <th>original</th>\n","      <th>test_case</th>\n","      <th>expected_result</th>\n","      <th>actual_result</th>\n","      <th>eval_score</th>\n","      <th>pass</th>\n","    </tr>\n","  </thead>\n","  <tbody>\n","    <tr>\n","      <th>0</th>\n","      <td>robustness</td>\n","      <td>uppercase</td>\n","      <td>Hannah: Hey, do you have Betty's number?\\nAman...</td>\n","      <td>HANNAH: HEY, DO YOU HAVE BETTY'S NUMBER? AMAND...</td>\n","      <td>Hannah is looking for Betty's phone number, b...</td>\n","      <td>Hannah is looking for Betty's number, but Ama...</td>\n","      <td>0.969697</td>\n","      <td>True</td>\n","    </tr>\n","    <tr>\n","      <th>1</th>\n","      <td>robustness</td>\n","      <td>uppercase</td>\n","      <td>Eric: MACHINE!\\r\\nRob: That's so gr8!\\r\\nEric:...</td>\n","      <td>ERIC: MACHINE! ROB: THAT'S SO GR8! ERIC: I KNO...</td>\n","      <td>Eric and Rob are discussing a stand-up comedy...</td>\n","      <td>Eric and Rob are discussing a stand-up comedy...</td>\n","      <td>0.413793</td>\n","      <td>False</td>\n","    </tr>\n","    <tr>\n","      <th>2</th>\n","      <td>robustness</td>\n","      <td>uppercase</td>\n","      <td>Lenny: Babe, can you help me with something?\\r...</td>\n","      <td>LENNY: BABE, CAN YOU HELP ME WITH SOMETHING? B...</td>\n","      <td>Lenny was unsure which trousers to buy and as...</td>\n","      <td>Lenny is trying to decide which pair of trous...</td>\n","      <td>0.152381</td>\n","      <td>False</td>\n","    </tr>\n","    <tr>\n","      <th>3</th>\n","      <td>robustness</td>\n","      <td>uppercase</td>\n","      <td>Will: hey babe, what do you want for dinner to...</td>\n","      <td>WILL: HEY BABE, WHAT DO YOU WANT FOR DINNER TO...</td>\n","      <td>Will and Emma are having a conversation about...</td>\n","      <td>Will and Emma are having a conversation about...</td>\n","      <td>0.851852</td>\n","      <td>True</td>\n","    </tr>\n","    <tr>\n","      <th>4</th>\n","      <td>robustness</td>\n","      <td>uppercase</td>\n","      <td>Ollie: Hi , are you in Warsaw\\r\\nJane: yes, ju...</td>\n","      <td>OLLIE: HI , ARE YOU IN WARSAW JANE: YES, JUST ...</td>\n","      <td>Ollie and Jane are arranging to meet for lunc...</td>\n","      <td>Ollie and Jane are making plans to meet up fo...</td>\n","      <td>0.352941</td>\n","      <td>False</td>\n","    </tr>\n","    <tr>\n","      <th>5</th>\n","      <td>robustness</td>\n","      <td>lowercase</td>\n","      <td>Hannah: Hey, do you have Betty's number?\\nAman...</td>\n","      <td>hannah: hey, do you have betty's number? amand...</td>\n","      <td>Hannah is looking for Betty's number, but Ama...</td>\n","      <td>Hannah is looking for Betty's number, but Ama...</td>\n","      <td>0.920000</td>\n","      <td>True</td>\n","    </tr>\n","    <tr>\n","      <th>6</th>\n","      <td>robustness</td>\n","      <td>lowercase</td>\n","      <td>Eric: MACHINE!\\r\\nRob: That's so gr8!\\r\\nEric:...</td>\n","      <td>eric: machine! rob: that's so gr8! eric: i kno...</td>\n","      <td>Eric and Rob are discussing a stand-up comedy...</td>\n","      <td>Eric and Rob are discussing a Russian stand-u...</td>\n","      <td>0.288889</td>\n","      <td>False</td>\n","    </tr>\n","    <tr>\n","      <th>7</th>\n","      <td>robustness</td>\n","      <td>lowercase</td>\n","      <td>Lenny: Babe, can you help me with something?\\r...</td>\n","      <td>lenny: babe, can you help me with something? b...</td>\n","      <td>Lenny was unsure which trousers to buy, so he...</td>\n","      <td>Lenny is trying to decide which pair of trous...</td>\n","      <td>0.303571</td>\n","      <td>False</td>\n","    </tr>\n","    <tr>\n","      <th>8</th>\n","      <td>robustness</td>\n","      <td>lowercase</td>\n","      <td>Will: hey babe, what do you want for dinner to...</td>\n","      <td>will: hey babe, what do you want for dinner to...</td>\n","      <td>Will and Emma are discussing dinner plans for...</td>\n","      <td>Will and Emma are discussing dinner plans for...</td>\n","      <td>0.825688</td>\n","      <td>True</td>\n","    </tr>\n","    <tr>\n","      <th>9</th>\n","      <td>robustness</td>\n","      <td>lowercase</td>\n","      <td>Ollie: Hi , are you in Warsaw\\r\\nJane: yes, ju...</td>\n","      <td>ollie: hi , are you in warsaw jane: yes, just ...</td>\n","      <td>Ollie and Jane are arranging to meet for lunc...</td>\n","      <td>Ollie and Jane are making plans to meet up. O...</td>\n","      <td>0.183486</td>\n","      <td>False</td>\n","    </tr>\n","  </tbody>\n","</table>\n","</div>\n","      <button class=\"colab-df-convert\" onclick=\"convertToInteractive('df-e0648226-1f58-4c80-9617-d96af6ab7c50')\"\n","              title=\"Convert this dataframe to an interactive table.\"\n","              style=\"display:none;\">\n","\n","  <svg xmlns=\"http://www.w3.org/2000/svg\" height=\"24px\"viewBox=\"0 0 24 24\"\n","       width=\"24px\">\n","    <path d=\"M0 0h24v24H0V0z\" fill=\"none\"/>\n","    <path d=\"M18.56 5.44l.94 2.06.94-2.06 2.06-.94-2.06-.94-.94-2.06-.94 2.06-2.06.94zm-11 1L8.5 8.5l.94-2.06 2.06-.94-2.06-.94L8.5 2.5l-.94 2.06-2.06.94zm10 10l.94 2.06.94-2.06 2.06-.94-2.06-.94-.94-2.06-.94 2.06-2.06.94z\"/><path d=\"M17.41 7.96l-1.37-1.37c-.4-.4-.92-.59-1.43-.59-.52 0-1.04.2-1.43.59L10.3 9.45l-7.72 7.72c-.78.78-.78 2.05 0 2.83L4 21.41c.39.39.9.59 1.41.59.51 0 1.02-.2 1.41-.59l7.78-7.78 2.81-2.81c.8-.78.8-2.07 0-2.86zM5.41 20L4 18.59l7.72-7.72 1.47 1.35L5.41 20z\"/>\n","  </svg>\n","      </button>\n","\n","\n","\n","    <div id=\"df-6094f0a5-cdbd-42bb-bfc8-d84bef19b5ec\">\n","      <button class=\"colab-df-quickchart\" onclick=\"quickchart('df-6094f0a5-cdbd-42bb-bfc8-d84bef19b5ec')\"\n","              title=\"Suggest charts.\"\n","              style=\"display:none;\">\n","\n","<svg xmlns=\"http://www.w3.org/2000/svg\" height=\"24px\"viewBox=\"0 0 24 24\"\n","     width=\"24px\">\n","    <g>\n","        <path d=\"M19 3H5c-1.1 0-2 .9-2 2v14c0 1.1.9 2 2 2h14c1.1 0 2-.9 2-2V5c0-1.1-.9-2-2-2zM9 17H7v-7h2v7zm4 0h-2V7h2v10zm4 0h-2v-4h2v4z\"/>\n","    </g>\n","</svg>\n","      </button>\n","    </div>\n","\n","<style>\n","  .colab-df-quickchart {\n","    background-color: #E8F0FE;\n","    border: none;\n","    border-radius: 50%;\n","    cursor: pointer;\n","    display: none;\n","    fill: #1967D2;\n","    height: 32px;\n","    padding: 0 0 0 0;\n","    width: 32px;\n","  }\n","\n","  .colab-df-quickchart:hover {\n","    background-color: #E2EBFA;\n","    box-shadow: 0px 1px 2px rgba(60, 64, 67, 0.3), 0px 1px 3px 1px rgba(60, 64, 67, 0.15);\n","    fill: #174EA6;\n","  }\n","\n","  [theme=dark] .colab-df-quickchart {\n","    background-color: #3B4455;\n","    fill: #D2E3FC;\n","  }\n","\n","  [theme=dark] .colab-df-quickchart:hover {\n","    background-color: #434B5C;\n","    box-shadow: 0px 1px 3px 1px rgba(0, 0, 0, 0.15);\n","    filter: drop-shadow(0px 1px 2px rgba(0, 0, 0, 0.3));\n","    fill: #FFFFFF;\n","  }\n","</style>\n","\n","    <script>\n","      async function quickchart(key) {\n","        const containerElement = document.querySelector('#' + key);\n","        const charts = await google.colab.kernel.invokeFunction(\n","            'suggestCharts', [key], {});\n","      }\n","    </script>\n","\n","      <script>\n","\n","function displayQuickchartButton(domScope) {\n","  let quickchartButtonEl =\n","    domScope.querySelector('#df-6094f0a5-cdbd-42bb-bfc8-d84bef19b5ec button.colab-df-quickchart');\n","  quickchartButtonEl.style.display =\n","    google.colab.kernel.accessAllowed ? 'block' : 'none';\n","}\n","\n","        displayQuickchartButton(document);\n","      </script>\n","      <style>\n","    .colab-df-container {\n","      display:flex;\n","      flex-wrap:wrap;\n","      gap: 12px;\n","    }\n","\n","    .colab-df-convert {\n","      background-color: #E8F0FE;\n","      border: none;\n","      border-radius: 50%;\n","      cursor: pointer;\n","      display: none;\n","      fill: #1967D2;\n","      height: 32px;\n","      padding: 0 0 0 0;\n","      width: 32px;\n","    }\n","\n","    .colab-df-convert:hover {\n","      background-color: #E2EBFA;\n","      box-shadow: 0px 1px 2px rgba(60, 64, 67, 0.3), 0px 1px 3px 1px rgba(60, 64, 67, 0.15);\n","      fill: #174EA6;\n","    }\n","\n","    [theme=dark] .colab-df-convert {\n","      background-color: #3B4455;\n","      fill: #D2E3FC;\n","    }\n","\n","    [theme=dark] .colab-df-convert:hover {\n","      background-color: #434B5C;\n","      box-shadow: 0px 1px 3px 1px rgba(0, 0, 0, 0.15);\n","      filter: drop-shadow(0px 1px 2px rgba(0, 0, 0, 0.3));\n","      fill: #FFFFFF;\n","    }\n","  </style>\n","\n","      <script>\n","        const buttonEl =\n","          document.querySelector('#df-e0648226-1f58-4c80-9617-d96af6ab7c50 button.colab-df-convert');\n","        buttonEl.style.display =\n","          google.colab.kernel.accessAllowed ? 'block' : 'none';\n","\n","        async function convertToInteractive(key) {\n","          const element = document.querySelector('#df-e0648226-1f58-4c80-9617-d96af6ab7c50');\n","          const dataTable =\n","            await google.colab.kernel.invokeFunction('convertToInteractive',\n","                                                     [key], {});\n","          if (!dataTable) return;\n","\n","          const docLinkHtml = 'Like what you see? Visit the ' +\n","            '<a target=\"_blank\" href=https://colab.research.google.com/notebooks/data_table.ipynb>data table notebook</a>'\n","            + ' to learn more about interactive tables.';\n","          element.innerHTML = '';\n","          dataTable['output_type'] = 'display_data';\n","          await google.colab.output.renderOutput(dataTable, element);\n","          const docLink = document.createElement('div');\n","          docLink.innerHTML = docLinkHtml;\n","          element.appendChild(docLink);\n","        }\n","      </script>\n","    </div>\n","  </div>\n"],"text/plain":["     category  test_type                                           original  \\\n","0  robustness  uppercase  Hannah: Hey, do you have Betty's number?\\nAman...   \n","1  robustness  uppercase  Eric: MACHINE!\\r\\nRob: That's so gr8!\\r\\nEric:...   \n","2  robustness  uppercase  Lenny: Babe, can you help me with something?\\r...   \n","3  robustness  uppercase  Will: hey babe, what do you want for dinner to...   \n","4  robustness  uppercase  Ollie: Hi , are you in Warsaw\\r\\nJane: yes, ju...   \n","5  robustness  lowercase  Hannah: Hey, do you have Betty's number?\\nAman...   \n","6  robustness  lowercase  Eric: MACHINE!\\r\\nRob: That's so gr8!\\r\\nEric:...   \n","7  robustness  lowercase  Lenny: Babe, can you help me with something?\\r...   \n","8  robustness  lowercase  Will: hey babe, what do you want for dinner to...   \n","9  robustness  lowercase  Ollie: Hi , are you in Warsaw\\r\\nJane: yes, ju...   \n","\n","                                           test_case  \\\n","0  HANNAH: HEY, DO YOU HAVE BETTY'S NUMBER? AMAND...   \n","1  ERIC: MACHINE! ROB: THAT'S SO GR8! ERIC: I KNO...   \n","2  LENNY: BABE, CAN YOU HELP ME WITH SOMETHING? B...   \n","3  WILL: HEY BABE, WHAT DO YOU WANT FOR DINNER TO...   \n","4  OLLIE: HI , ARE YOU IN WARSAW JANE: YES, JUST ...   \n","5  hannah: hey, do you have betty's number? amand...   \n","6  eric: machine! rob: that's so gr8! eric: i kno...   \n","7  lenny: babe, can you help me with something? b...   \n","8  will: hey babe, what do you want for dinner to...   \n","9  ollie: hi , are you in warsaw jane: yes, just ...   \n","\n","                                     expected_result  \\\n","0   Hannah is looking for Betty's phone number, b...   \n","1   Eric and Rob are discussing a stand-up comedy...   \n","2   Lenny was unsure which trousers to buy and as...   \n","3   Will and Emma are having a conversation about...   \n","4   Ollie and Jane are arranging to meet for lunc...   \n","5   Hannah is looking for Betty's number, but Ama...   \n","6   Eric and Rob are discussing a stand-up comedy...   \n","7   Lenny was unsure which trousers to buy, so he...   \n","8   Will and Emma are discussing dinner plans for...   \n","9   Ollie and Jane are arranging to meet for lunc...   \n","\n","                                       actual_result  eval_score   pass  \n","0   Hannah is looking for Betty's number, but Ama...    0.969697   True  \n","1   Eric and Rob are discussing a stand-up comedy...    0.413793  False  \n","2   Lenny is trying to decide which pair of trous...    0.152381  False  \n","3   Will and Emma are having a conversation about...    0.851852   True  \n","4   Ollie and Jane are making plans to meet up fo...    0.352941  False  \n","5   Hannah is looking for Betty's number, but Ama...    0.920000   True  \n","6   Eric and Rob are discussing a Russian stand-u...    0.288889  False  \n","7   Lenny is trying to decide which pair of trous...    0.303571  False  \n","8   Will and Emma are discussing dinner plans for...    0.825688   True  \n","9   Ollie and Jane are making plans to meet up. O...    0.183486  False  "]},"execution_count":42,"metadata":{},"output_type":"execute_result"}],"source":["harness.generated_results()"]},{"cell_type":"markdown","metadata":{"id":"WBda2qn1GaLl"},"source":["This method returns the generated results in the form of a pandas dataframe, which provides a convenient and easy-to-use format for working with the test results. You can use this method to quickly identify the test cases that failed and to determine where fixes are needed."]},{"cell_type":"markdown","metadata":{"id":"3Ko-gZISE3oW"},"source":["### Report of the tests"]},{"cell_type":"markdown","metadata":{"id":"Ir8VwGYzGdE1"},"source":["We can call `.report()` which summarizes the results giving information about pass and fail counts and overall test pass/fail flag."]},{"cell_type":"code","execution_count":40,"metadata":{"colab":{"base_uri":"https://localhost:8080/","height":112},"executionInfo":{"elapsed":4062,"status":"ok","timestamp":1689534196772,"user":{"displayName":"Prikshit Sharma","userId":"02603035510215936699"},"user_tz":-330},"id":"qu5TUU2kE0nb","outputId":"ed425397-cd1f-4b34-9423-7d66e2e260df"},"outputs":[{"data":{"text/html":["\n","\n","  <div id=\"df-43c904ee-e7b3-4a35-9556-72e5b6ec6680\">\n","    <div class=\"colab-df-container\">\n","      <div>\n","<style scoped>\n","    .dataframe tbody tr th:only-of-type {\n","        vertical-align: middle;\n","    }\n","\n","    .dataframe tbody tr th {\n","        vertical-align: top;\n","    }\n","\n","    .dataframe thead th {\n","        text-align: right;\n","    }\n","</style>\n","<table border=\"1\" class=\"dataframe\">\n","  <thead>\n","    <tr style=\"text-align: right;\">\n","      <th></th>\n","      <th>category</th>\n","      <th>test_type</th>\n","      <th>fail_count</th>\n","      <th>pass_count</th>\n","      <th>pass_rate</th>\n","      <th>minimum_pass_rate</th>\n","      <th>pass</th>\n","    </tr>\n","  </thead>\n","  <tbody>\n","    <tr>\n","      <th>0</th>\n","      <td>robustness</td>\n","      <td>uppercase</td>\n","      <td>3</td>\n","      <td>2</td>\n","      <td>40%</td>\n","      <td>66%</td>\n","      <td>False</td>\n","    </tr>\n","    <tr>\n","      <th>1</th>\n","      <td>robustness</td>\n","      <td>lowercase</td>\n","      <td>3</td>\n","      <td>2</td>\n","      <td>40%</td>\n","      <td>60%</td>\n","      <td>False</td>\n","    </tr>\n","  </tbody>\n","</table>\n","</div>\n","      <button class=\"colab-df-convert\" onclick=\"convertToInteractive('df-43c904ee-e7b3-4a35-9556-72e5b6ec6680')\"\n","              title=\"Convert this dataframe to an interactive table.\"\n","              style=\"display:none;\">\n","\n","  <svg xmlns=\"http://www.w3.org/2000/svg\" height=\"24px\"viewBox=\"0 0 24 24\"\n","       width=\"24px\">\n","    <path d=\"M0 0h24v24H0V0z\" fill=\"none\"/>\n","    <path d=\"M18.56 5.44l.94 2.06.94-2.06 2.06-.94-2.06-.94-.94-2.06-.94 2.06-2.06.94zm-11 1L8.5 8.5l.94-2.06 2.06-.94-2.06-.94L8.5 2.5l-.94 2.06-2.06.94zm10 10l.94 2.06.94-2.06 2.06-.94-2.06-.94-.94-2.06-.94 2.06-2.06.94z\"/><path d=\"M17.41 7.96l-1.37-1.37c-.4-.4-.92-.59-1.43-.59-.52 0-1.04.2-1.43.59L10.3 9.45l-7.72 7.72c-.78.78-.78 2.05 0 2.83L4 21.41c.39.39.9.59 1.41.59.51 0 1.02-.2 1.41-.59l7.78-7.78 2.81-2.81c.8-.78.8-2.07 0-2.86zM5.41 20L4 18.59l7.72-7.72 1.47 1.35L5.41 20z\"/>\n","  </svg>\n","      </button>\n","\n","\n","\n","    <div id=\"df-4b6aa484-a426-47cd-b207-855a05a6a3ef\">\n","      <button class=\"colab-df-quickchart\" onclick=\"quickchart('df-4b6aa484-a426-47cd-b207-855a05a6a3ef')\"\n","              title=\"Suggest charts.\"\n","              style=\"display:none;\">\n","\n","<svg xmlns=\"http://www.w3.org/2000/svg\" height=\"24px\"viewBox=\"0 0 24 24\"\n","     width=\"24px\">\n","    <g>\n","        <path d=\"M19 3H5c-1.1 0-2 .9-2 2v14c0 1.1.9 2 2 2h14c1.1 0 2-.9 2-2V5c0-1.1-.9-2-2-2zM9 17H7v-7h2v7zm4 0h-2V7h2v10zm4 0h-2v-4h2v4z\"/>\n","    </g>\n","</svg>\n","      </button>\n","    </div>\n","\n","<style>\n","  .colab-df-quickchart {\n","    background-color: #E8F0FE;\n","    border: none;\n","    border-radius: 50%;\n","    cursor: pointer;\n","    display: none;\n","    fill: #1967D2;\n","    height: 32px;\n","    padding: 0 0 0 0;\n","    width: 32px;\n","  }\n","\n","  .colab-df-quickchart:hover {\n","    background-color: #E2EBFA;\n","    box-shadow: 0px 1px 2px rgba(60, 64, 67, 0.3), 0px 1px 3px 1px rgba(60, 64, 67, 0.15);\n","    fill: #174EA6;\n","  }\n","\n","  [theme=dark] .colab-df-quickchart {\n","    background-color: #3B4455;\n","    fill: #D2E3FC;\n","  }\n","\n","  [theme=dark] .colab-df-quickchart:hover {\n","    background-color: #434B5C;\n","    box-shadow: 0px 1px 3px 1px rgba(0, 0, 0, 0.15);\n","    filter: drop-shadow(0px 1px 2px rgba(0, 0, 0, 0.3));\n","    fill: #FFFFFF;\n","  }\n","</style>\n","\n","    <script>\n","      async function quickchart(key) {\n","        const containerElement = document.querySelector('#' + key);\n","        const charts = await google.colab.kernel.invokeFunction(\n","            'suggestCharts', [key], {});\n","      }\n","    </script>\n","\n","      <script>\n","\n","function displayQuickchartButton(domScope) {\n","  let quickchartButtonEl =\n","    domScope.querySelector('#df-4b6aa484-a426-47cd-b207-855a05a6a3ef button.colab-df-quickchart');\n","  quickchartButtonEl.style.display =\n","    google.colab.kernel.accessAllowed ? 'block' : 'none';\n","}\n","\n","        displayQuickchartButton(document);\n","      </script>\n","      <style>\n","    .colab-df-container {\n","      display:flex;\n","      flex-wrap:wrap;\n","      gap: 12px;\n","    }\n","\n","    .colab-df-convert {\n","      background-color: #E8F0FE;\n","      border: none;\n","      border-radius: 50%;\n","      cursor: pointer;\n","      display: none;\n","      fill: #1967D2;\n","      height: 32px;\n","      padding: 0 0 0 0;\n","      width: 32px;\n","    }\n","\n","    .colab-df-convert:hover {\n","      background-color: #E2EBFA;\n","      box-shadow: 0px 1px 2px rgba(60, 64, 67, 0.3), 0px 1px 3px 1px rgba(60, 64, 67, 0.15);\n","      fill: #174EA6;\n","    }\n","\n","    [theme=dark] .colab-df-convert {\n","      background-color: #3B4455;\n","      fill: #D2E3FC;\n","    }\n","\n","    [theme=dark] .colab-df-convert:hover {\n","      background-color: #434B5C;\n","      box-shadow: 0px 1px 3px 1px rgba(0, 0, 0, 0.15);\n","      filter: drop-shadow(0px 1px 2px rgba(0, 0, 0, 0.3));\n","      fill: #FFFFFF;\n","    }\n","  </style>\n","\n","      <script>\n","        const buttonEl =\n","          document.querySelector('#df-43c904ee-e7b3-4a35-9556-72e5b6ec6680 button.colab-df-convert');\n","        buttonEl.style.display =\n","          google.colab.kernel.accessAllowed ? 'block' : 'none';\n","\n","        async function convertToInteractive(key) {\n","          const element = document.querySelector('#df-43c904ee-e7b3-4a35-9556-72e5b6ec6680');\n","          const dataTable =\n","            await google.colab.kernel.invokeFunction('convertToInteractive',\n","                                                     [key], {});\n","          if (!dataTable) return;\n","\n","          const docLinkHtml = 'Like what you see? Visit the ' +\n","            '<a target=\"_blank\" href=https://colab.research.google.com/notebooks/data_table.ipynb>data table notebook</a>'\n","            + ' to learn more about interactive tables.';\n","          element.innerHTML = '';\n","          dataTable['output_type'] = 'display_data';\n","          await google.colab.output.renderOutput(dataTable, element);\n","          const docLink = document.createElement('div');\n","          docLink.innerHTML = docLinkHtml;\n","          element.appendChild(docLink);\n","        }\n","      </script>\n","    </div>\n","  </div>\n"],"text/plain":["     category  test_type  fail_count  pass_count pass_rate minimum_pass_rate  \\\n","0  robustness  uppercase           3           2       40%               66%   \n","1  robustness  lowercase           3           2       40%               60%   \n","\n","    pass  \n","0  False  \n","1  False  "]},"execution_count":40,"metadata":{},"output_type":"execute_result"}],"source":["harness.report()"]}],"metadata":{"accelerator":"TPU","colab":{"machine_shape":"hm","provenance":[],"toc_visible":true},"gpuClass":"standard","kernelspec":{"display_name":"Python 3","name":"python3"},"language_info":{"codemirror_mode":{"name":"ipython","version":3},"file_extension":".py","mimetype":"text/x-python","name":"python","nbconvert_exporter":"python","pygments_lexer":"ipython3","version":"3.9.13"}},"nbformat":4,"nbformat_minor":0}
+{"cells":[{"cell_type":"markdown","metadata":{"id":"e7PsSmy9sCoR"},"source":["![image.png](data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAAUgAAABcCAYAAAAMJCwKAAAgAElEQVR4nOy9f5gcZ3Xn+znnra5pjcfKZCyNfqDIQgghZMdxZMfGxpbbwhjM2g4h2Ak/Nol3Aw5xEsLu5eHh8vCofNl9uFluLhiwhUi4zib3ZomcZBMgARsjt4RxbGIritcSsiyE0GpleSQLMYxHPd1V59w/qnq6Z6ZnNJJG/Ej6+zw9PW911fueeqvq1Pn9CucASZJokkzZaudirC666KKLcwWZ+y4TveyWJeW4/lKZYYD5mI2m8+YdH61Wk3Tux+uiiy66ODeYYwaZaKUysNSI7xSVtfj4MCPi9t8WLhzY+sADt9fndswuuuiii3ODaO66ShQSM7lvvYj8B6A8/pMIiM4/evToTuDI3I3ZRRdddHHuMIcMMocgC9ysFwx3DBzVyFzCQBpF8VyP10UXXXRxrjDnDBJygdFyl4wiTS3egJPnYrguuuiii3MCPRedem57NHBk3A6pwLxzMVwXXXTRxTnBnEmQSZJ/xP2gaDjhrv00vTSigB12tVqSJNrcf/p+uiFBXXTRxY8ec+7Fvuqq+f1RT/ktgl40PogwbKn/XQgv7KhUsJwBJjNIr10G2UUXXfzocU7iICsV9AfnL4k5nG85//zYKpXv1pMksStv+uT8eKy0RtyWqU9U8U1cU5e9Mb17qtU7anNPWxdddNHF7HEOGOTUTJpKBa1UsC271kYLjh79zyL6bnefP3F4b5JzxLEPvrhw4Z/v7sZMdtFFFz9CnBMGORW5On1V5YLVsUT/CNJrlnXcUzXg+JfU7c5K5ehQ1x7ZRRdd/KhwTsJ8JqMpTW7dzlJc+swykBZ3HpcdAfcMkVAGLVerKHl8UBdddNHFDx3nJMxn2sHMFYrEmrbtPyQxtosuuujitPBDlSDXbwgqDo4grUTtCRJkF1100cWPC+aIQc4uZMdMLAhtzDH/lo7KdhdddNHFjxZzwCATXbuWCNZO8/sWBgdfUvhuCh75hN8mM8P2djfKp4suuvjR4iwYZKLXvq7/YrGeD7jbIBxF3NskyZZ/JTc9LkyBBdP5XNxBwETV8OwwcKJSwarVM6ewiy666OJscEb6bJIkWq0uXOkS/ptqaZ1ZSqsoxQxwU/f28J7Jxzil6LwnG/aDD2zf+rtbz4S2Lrrooou5whlLkCa+LmjP8ix9KXUkEloWxBm+TaTwnDsmok+L6iHcIxcxaBzP0h98bnvlxe1szetLnu0JdtFFF12cKc6YQbprjLgiolKECzXlwVN9Fz2kmdumyPyhNLhGmRhEI9XqnceongFzLIpg0A0s76KLLuYILQaZJAobIZFZMphsgnQ4W7g7ICaAqp2oXHfs4K5dREePthsnZ2BySdPOWS2+K5bTvLG5rcsgu+iiizlBziCTRyIWDpY5ursO5PnPic8QunM3ofgvZ46T2eSp2tB04iRJYkmSpDOmFCau44x77e6II3GZ0s+U0bEyvq+PTc/2Ic8tw5fGJL5l9ky+iy666GJ65AxyydJVuN7OYh/lM88OIQwjz42QygjKMJ6OYlajhzqhd5Q7qFPJO/Ai7Lv5fx7VOHO7CfdZZPJsPtwLe9fxmb2D4H286IuJWYTqAvS8BbgsRmwAGCTL9gFb5mhuuuiii3/lyBlkqsuZN+8OsvogIaqhOgqhRikbJUtHca2TpaM0pE5afzBJNn5m/bb7VGkP8p74/3TtcSapBhODIjvDvj9I+fy7kbCGtF7GrBfPYtwUc8vXd3AIEdC5AEYXXXTRxZkgZ5Alt9yg6BH1sX5gfsHbNOdnriBQ7jVOvpRWqH72rHVYY3bGSytFNBqLkXSQrFFInN70hBffbmiYZYdddNFFF7NDIUECJcgZjytNxtiEA7iRpYqQTu2mubPMsi2AIGKz5LMCmOKmHeMtu3yxiy66OAeI2v6eIthbirVlRGGyq3imlMHJ7bbM60ICzMuatSrsTlmXRrFZqeNddNFFF3OIXEXtIBNOz5CauvfZQ0TqANXqRH47qyK5XYbZRRddnGNMlCDbMUWY7MyR2r3Ys4XjiKC4r61UPnMQsrJpi0lm+olDpfTE4Wo16cS6p6Gviy666GJuMZE1+mTD4/RcyFWsGcRzOpCWAKogHzGyjwATdPbg8QF06d2Vyv2fn75WRbc0WhdddHFuMclJAy3GM7lG4xSHSwp5QLa7W3uwT4t1easHkem1cqHVrWMi0XIXeY9Qa/LHtmOno+cnH801wydt6wa9d9HFjwgdVOxTOVya8N2W1YdE4wXi2YxH5BFERidm5u75/sVPDmAZIEsta/QC9YnHdex9GhrPHJ2YVbH9HDCsRG+6aaCvWg29k3+pVDanlcrzx//lMMr2eW2d08SVMP+lnOuPEdoz485Vptnk7LvTHSdxhbvJ04anw91nXm+hSV87XaeYl4kqdrsXe4oGOy7iWZWKVbJtu2HwfZlnG8VZPC1RCuLgbgMg/ePVfMaHLAZpfakI5gBxTOvHSUzwHGrY0zHHczXWU08tKZ8YyX4f918uwt5VwAwipfF0tbrkvUmS/EQzyZwBJkYClSo6NFRELly0FtjNll1Q1P+05vz/JJ9vF2eARGxqrYV2VIqaC8nE9ONT9lvUmWj2u2VXG9/bDbuHLO+bKf1Ob4OcUqpxIiOrVLAk+e2HIdl62WVLykuXTkfd8wCcGB78UAjRfzCrRyAzVBGapTR4jpjjbbdtiavVY+sybIUIRhaADIJHiB4DHprrMYeGxqK4HF6uIbrYLVMpXgiRBixr1EulenzKTn5skWilglarS/qvrty7LFTlNSby6gWLfJkg/Rw7rrB4FOG4kR1av97/6aGq7CXWw5VKcnxGR10Xs8Omb61A9l0OGXhQPv2tnfzOq/fOWf/JIxFLll2CPbsq3yCK6yj3f2c7d7z8xCmP37Ir5lhpGZEuxp5dCroAedl8JJQR78ElxTmJ7x0G389nnjuI7B0i8eP5+DMwysSVnzown/i5FaitI7rwSk74UpA+xFPcj7P0woPw3C42P/c0YfcBEj/R7HN6RuU+KS6yybgKKRVyzpwk9tRTjD711LQUKsC111nqba6Yyd7vZnvWPvEp9J09KpUkOjR8qC/WeXeKh7fnGToOLghR5GZPcg4Y5Lx5wTL31C2z3BSRM0jLR09H53rAHwKaUmC1urA3w25Q4ZYS4Ro3WyUiKqJ4YcMW0DyyIeBqtZLqARq+AwY/BTz+Iz2Rn2Q0JSd/7mpCuAejTKlkYB8C5oZBJolywZJBotIHSeVW8BSIEB2hkd4BfKHJJzof78rRby9nXvmjZI31CPNxi0GLpBAthCEDF0PCMCE6hNsOFu39Mg39exIfmZZJLn52HRq/DS29kbSxGhFFFEQUHBzDHUxSotJBTP+SZbs/1mSSE+MgRVpSZJP5TG5PqEp2ahWoZVcquivY38QCFq32KVleJ/rm0ATZM3aeQkCQCCd2J3aIEVVkJsn37CCtOyEPgZrgiPrJxBe/uKScuX44aM/HwX8NfBU47hlmDSyr5x+r45ZinoEQ46zGeKuJLYcfrsnjXxaaaqUoqhEiMVEMOoPD9ExQ0lVIuJjcfFYGIkLUj+hNwKn5hKS9qCwDGaD5rIWIfBGWDDzL81OiHiWEftzW4PZOeno/TmQbedm+pR2rj21+9hqi8iZEfhv31WgUIZr32RiDtFgJQRVEIpxVGOsIvdOo2DBVahxvnzkXShL42rai+0nGw9MNE+pM31w7aQzM8WbON27F2+aHgJ9873zTrnre+endIfT8dpaNxTiKoHnWapvtuWi3NRRxQ+WAethd9Ne1RZ4NJrAOn7uKqYkra3dHHLN1pPXlxeJTxRgZmN/A//vcfN75yuHpO7kb5J2FFJfm6cRwgKzxNwj/E6eGiaLWh6SvxFmPllbgBo2xBcQ9v0Wj3s/CAx8i8aFxO+aSfZcS9XycrL4OMyOUFLLDGF/CfRduI0BMlr4c90twW8d5fQsYPvY1vvuq4dxZNNmL3ZTOxnmYTGqfBQwIs+lqMmMYyw+cvEs7fXMNV/WiMlBLqJbTZ+b/SrFlF9HCkfR3Qii/O01PxiIStU+d5Kq1tiWdGoKKY/nLCEXYWS8xVKkkUdcOORdwxl/ycyk/vhAW0Ft+HZmVUVXS9CuUoktxHyREqxitryfxvwdmthU26z3kmtROTD7KC684NuWY+7/TT73+a2j0XsxXkDViSvHtZNn/4MIDnyHxlEXfHsDlA5hdipmhoY5nW8jC3bzn5QemjJ24sujAcn7w4luw7AtTnTQT4iCZJtJnbpjDqXtpqdo5q+yZ0OrYyU+usNUBk+M8f7JQLOi2lhDdlqVjfcJEdU5EUxE9CLbHPT3miKlIHxIGUF2M23KgTJb+c2znDXdXtpwrTHSyzgkSMe57bjlZdmmxxRC/n6h0F5ktQAOkfhNUv0Jy/Wm85DwizSKuQ0naH+674bsrhlny/B+TvZQSlT5CI+1HrZcQ3sBIbQtUh5CfWUccX06jDhqBsJVG9hGGXnFw2kLgL6w4SCL/9+TNp1Gs4sxQVAxXhe+rBMuQIrB8qoMGwAUTFBEZcer5pJ6qNNo5oHvSALPeczycZdK24vuslZvJ/Z+q79kEn7diECfHJZ4+vdUqmrpfEcxX57p06zeRAOJfERu7B0r76uXGcM+YGMRlPOuzLBuUwKVo6UqX8Pj1679bb94/pzqHs6F5ch/5N0yOx5yu/5lspDPRM/m4TmOeaozZn2+bdjgXKnYzHCYK1yC6ODdLZUOkPEpmr8eya8hSRaPXMPiy5SR+4LTjIrdhU45JNirPL6mx8MBfo+k7CKXX5GdkawjxAi5ccZyxxsWk9aW4QVwe4eTI3zH0qoP58dPQMA3j7BzmM9lDfJYe4yRJ7NprP/Gwp/V3hKh86cyKtqu51zJPv9DosSPAYO5JnkRnRw/73KEps+aUztx/O5NKinbTNzXl+5QPcbOo8ERUq2iSJIz3P8n5Nf3DO3176kOXKLPstxOSJNEvPzHQW66Fi9ysb9zmSG6gcLNhj/QDgeN7Ad5wVf6oVquMAMe2b0/23XbbliePHv3eFqE80hw3/y5oSzoO3U7EeJhFqyrU7BaBa55ra15a85Mk01/D6embpRNz/LgZmanl3uDmhsljnQpzrJWMMxq/CRUgMpxvsqh+jO/V/wcS1fAsJu5dRnbychLZf0rypqDDGlOJ5PNwdOMQS57bQ6nnNaR1cPqwrJ8fSMw8/Rncy+ApwgjoPujAbDuez0RMVLHbvdhNJjQeG3l2TOjrX//9pyuVe/+NWe0t7lZkjDTvvxZt4sFcbU9w2f7El39vhJvfNJinNLbR1ZG+uUXrwW6Xb6dWLE+SRLfsWhsNHj0yuH7Dp1bLtvCaRwivuA4WQBY/4jricOhasn/m2vt2fPnL6QFg+HSlnaEh9KuP9i+9Juu5YSty5XUbfCnmPLJN9nuWfSPL0scrleRwXhkp77dS2bQiwy/11FJVVVOxrdsye+3rP7Xz9a998UheZm7higy9/LrruQp0BdssAj3yCPbPlcq926vV3j1JktRnS2vISmURHURzb7XguIuJBpzs4Ne/dmRPMXPtqvN43xddtDtNkuRYs33ZZZt7zz+/foUZ860qputVATz69KEXLxh8ZvDobhsbmz9fe3rWbt2u16x3+XnB5rNBRrZW/cA1lU8+GNGzE5ITM9kyK5UkeuihRQPr19+76pFtevl118urcJaSe2VrW6scuZb0Wat86tFqNT5QqeT9VSr3l2H0cjMbaNJnKqbmCvcc2779vY91GqvOwou3bpPl11TMqIKuV0313oOPVe/aOXX/+8uZ1i6Rbb6Y9cWEVc2iikZZ+OTer3/t93af+so0X/fMnQ3yvj2X4H4NaUMRMdz/jtsvqrP52R2E6ABuq0nTAcRfxyef+wrHV00fjnMmj7Fbffx/kTpRGOWkKm5Riy+IgkzJUJstpqYaTpYUJ4f7nAWq1buOAPedar9WDF2HHzvSdy6NkNImQU50FiVJol/9av+yhfHRm116flHcLgcGkOZNEEAEcVdcUonCgbLKX1+74dN/Ua0e250kSZ0OaB9RALFQvmBwwVvUone523rRkN/iWkjiwm9GpWg7LL4HfusrkEuYW7dlG5Tojzx4DUHVzUTiUW003l+tLvxLM26UEL1PsHUQehGseY754pPRPhi9p1rt2wIc60DqjBhfkUhcPU9HXXbttYMXv+51Q8/kNHZUVydsmzcvW+we/YEIl6q4oYCLikd/0//9F38XLlhe6gn/HuRmcVla1CzNRxZXNfl3HvE3kl2wqVJJdnZikle94Y8HsrGxDaUe/SWMG9xYIKoTGEkeiqcaiR5w2Oos+KvLLttchXqvubwHid6q5PSpuEnQ2C3aWakkV7WPmSSJfvUbFwyW0ujDbtnNiqSIqASNStjDwE3ttFUqj0Rp2LU8ePRRd7+6SZO6mmsoq/EeYBYMsg1z5cVWuYFSOSIdM5BDYE8CUPf9SGMvImuwFOLyJdjoCrj7mbkZeCMs291PI1pNVoTqiB7ETx6j96U6dv4xJKQgkGXzwS7jwgMPkST1001TnL4e5GScczvfRJyWLekcO2m8k/yfJFqtXrA6RPGnIPrP4De4eb+54Vkzxq+BZ3XcU8AjsJUov68S3Zux4M1ffGpJOZfiOp9MMeWxpPZOJXwUZL27q2f1vN+sgWcNwMuOvxENH69U7nvNuBqdaU01KEgZJ0aIVUOs7ksz+A2Nev4Q/Grce90LWpv9muFuKyF8xCj/1k03fXL+bOIR43qtbm7H3a3wSkPLbCD9ov7Rr1YHr9iya+2kJYc7I4rE0JCiGmHEOLEEjZQwX+q22qV0r4j+O5ylbpm25iWPrQTvF5O3u0QfzbKB1ZP7r1TuXRzX7UMq0cfBf9VhgWOYNcav43if7ubmy8F/TSW+5/zz7feGFv70sKg+JSKG5/RhRSygyKpG44LBibdNYpr5MlFdKSqtawORO5dWKpsXTKRvm6mzGMIyEYnHx4AyeE1cpkioM6KIvT4rJIly/3f6gdcXy6AoIjtI64dJXHnx+SHcniCKR4EU95WIrJ05x7oN0wljSaLjtsK0VKHUs5YsNZAU9ypmx3j+sjruu4ii44hAWu8lKr2Z2tjVrL0tym2ns4+rzXecHObzI8aPX9zb1HmpVC9YnRE2icrNbul890wR0yYrLbJFtJ25upu6W+yZXy4e/vC8kcbNUyWacS++uhuOrBb0P7r7cstSLVxammcESB5bKK7uZu7Zmgzf+NBDixbkc+i1PI7eQUxx1KwRu8htKuH95o1lZinuZjjmbX2Cq3umjs8XLb3rByd1PcwmaPv7I0L2zyI6MjHeFXAzRG6MNHzugqGhjZXKp9aQd2rkJocpfTcaYybjBUscxNUtU7N0tbr/IcgVbhYVvNha8yKKgONq1oiRaL2WSu+f2HuirtHHReTd7tni/HwzBVcBXFAR1bbzUMSa46+QEH9w4dDQ73iWPSOqRxAMseJ6ZIjo/FJJV7aGK87RwnJ3W+qeX5e2/QfNGmsLm2lrPlJdhtsCt2J/DNEA5nvghT0zX49JmCsnTb1+MaXyGiw1oEaWfoOFHM+LSVyfYjwOHMctIksHiEpXMbCvb+blpAtMJ4s1+cLi564h6vkAWTqAqqL6NHbyAY4+MAoYFu3A/BmcCDMQ1hJKH+NY/MbChpnHSs6Clok7zCgl/ngwz444x8JtK+snI0kSrVQ2rXDCx1R0vecXILeL5a/nVELphIjsNfc9IcRDImEiE/RMRWWxEG2+9nX3XXLyZKaTw2HGz0noBe/L/1VUo1SQnKG17SqCmmdpFHpeE+L0LUmSqKnXJ3QoqHtWBrnULFuGmZL3aaKKeMs+JCKIiLplkWe2LEjpjmp14eBkp087kiSxSgUT9+2CPi46yd6UF0lWz7I1IcT/u0v0j9dtuO/Prq3c9+bXfnXJsi1b1kaTmWSppOZNHWe80ImD+EoRvcIsNQRVVUSDFT/bhIQrcfWsHrn7r61ff+/VkOhll23uXV8Z/AOV8KtZNtYLFo2fN2IaolGVsB9nt4TosGioC0W/goJFWVbrDaXeD6Csc2cvIupe3C3uphppBs0QGBLy1Etcf8GzbAGeL4ZXVLMy1aAeqOQ25MSqVbRaXdiL+s+6Zf15VpxAca+4yN9Xq0n6Q800ShKF65RM14MMgqRE8X5UHmf32nSciVn9ScZGnyaKQQKIVuixaSs2FCgW4ZMyJZayaPEyNn1rBfftXcnmZ9fw2b03sOQ7mwjRf8fSy9EIgj6O1d/LnWt35IxPjLtW7SPLPkb5vL2okku5cimBv+Wz+/8rn917Awt3D0JVT8UoO8dBdsT0XChx1yLwfE6QnKtyTKeBiT5yz62CrrlDRl+8WQjXFA/nuKoooiaqO71R36QavknGaCb1derhXaJhvVsWk8cwqVlmqqV+Se0DIZTeZ3gqjk728I8nZmrY75buMOe4qi4vJKeBPPOkuZdHZo35SrjuoccW/XUkmRVse1IuRe52EpW6oI+aNQ4gUtYQXeKWXTJZzc+7tyvAlkFy5NRe4Rf3Zb7gc0HjNe4sds90vB6ooI5hWcMQ6ROJ3i6kb45i/+bCRcf/qlod+AJwqOmpbzTESrGk3kZ38yxwN5HIVGSve7bTzU5I0NWIrMOy/lawQ26nVonVqN8CyWPnnffpimjp7WluP8sZjjuCGnAo8+xz5tnfSxSOq9sKcf6tiLzV3fpaHmGP0sbYAkF/CU+HNET1jCxu7w+4qDlfCfDahs0v9ZTWuhvuaZt06nlMs8vP33LL5t4vfvH5WrWKXX2j9pbSsAo3xX2cRvdsGPWvz3wXT4OzYqcb4WX7FuPhKtJ6nKuxjd00xiZ6qe+6aIRNzz6I6M1kYyC6CgmXksie6SvxCGCgcjla2gyhmTgQgffhtpigfWQpwGG88RUyPs6RVROl6MSVIzzEon0fpjzvD2iMrSgkXSPSd5Lpmyj1PsqSpV9G9lQ5fGR/EfIwTbmzM1GxN26EJOETu04ul2dH3+S/IhHuhoQzn37PDAKf+NWxR39/Tc/TZ9zPHKAV4tPGpAQbPHpk0CX+JfD5tN9qriYiJ9wb/3HDhmOPNjfv2rX20JEXXzyo5veAXOHuxUPratYwDfE1sTQuMbfc09tWetidIutEdpqnH80auj2ObbQRxgaiLHqnavR+t6y/RbXg5mgUrQhZulhdzCfFIgKIYwh1N/usRX5P5DIE9ahhsiYS+SOQi/OiGQV7dVPQxYJeDDyZJFPDh5oowmSoVuVLnjUGRMNHRaI+LyQ9mhlJuRqf21CFPjeviMrlaPn69Rs+/alq9dhjlQo0GuDixaJtE9ITTTQC829CfaNQ3yk6r4bbYkPuFA3vxrK+1jUS3DMQW1epbF7gkv0i7oMTcyDERMOwe/qpejn77BNfPj5S/HCgUhnYax56VUu3uzVyVb4ZDKa6yiwbVbeaIHFz3twzcF9dqfzU/GolGSZJrFTZNGDua5quxXH2KCi5mr36e99rLAP2QWKa3dcHvpKiDB5Cs97CHjLfe0axn2cjfiRibPrWKuKe1aR1I4pr1Eef4OjQMZKLWiXDAHTvw2SNEZBeNJSx7A3A508dD6n9aLSu+D9/EIpsXxr1lHweTiD+jwhD42M2+22mG76w6i9Z8u06qncRxVcDZRpjIKEfsVuReAORfpNFS/8W+/W/hOTI5MIas3fStIjPaSharqzE5f0CH0T0g4h/UNo+p9NG9QOi9gF3W3c6FJ17FGxSvJYSLnbzy3MnRpukpaqI/7Xasceq1evG4yIvumh3uviCC3YiPCAhGqG4PXMV1k1hIHO7HogmhDMB4KYhOu6SbQr0fimOXzherRwd/cbDJw6JN+7DssdEI9zb46QwdwZClg20r/Mz3qNDblPXrZbJPVE2dLBaPToK3x95fWXom5h/yt1TL9TUNptqZMgrZjNbuap9dHRkJPoTJ/tdYK+GWIubfeI5NhklmbpZn3t2q0rPPSkL3ghAb/uuzZNonoupB7sbjldh5ESlcnQUjh5Q5L+CPENbFXvH86ElLDUdW6caX+JmOm4eaaq41tiRxvqnN13ZZI5JEat5/DCBexxLc2bbJMrVzfpBBtzTWq5mA1DYFcNSiBZX8pU71Sxbi2XL3QxcwN3cyRMn3Ey1NKAlXdOkO8p8qbstd2tZs91NPfUdUDsx1ck3C5ypCJO4cv93yki4nLS+vAinOU4WHodKEaeZaDOPmedX78PZQVTKGZzZhsK5MzM8HSUdO0ha309aP0BaP0jWOIGIUe6NCAFCWM28+R/B5HMsfnbdxFqStOIan/+fX6KR3oll7ydLdxL1KFFJMQNPe0nTDcTzPkKJTWzad3F+bMtkMdFJMytPdfHMFXMgSorIqED+cUZo+0xoU7RpfSb9PuowKh3X3v7hYrKKXbzv64peJyrz80IWkjNJF3PLhh17II+N22btQc4PPLA7bbhvxX1IhOYDhLtoljV6Bb8cvJ/2cnCOiahmWX3Ig26tVr9br1aTwsaTWLX6vhMmfFk1dApk70uRPjWxKdIjmCg1cftiFA0drFQo+kvSJEksy6wqovtVWyFN7m6ImogOMkskSWK33PJ8bfsjd/1pGuQNZul/EtHdGnpG8WAgaev9InnxCnE1y2K37OJI40/Bomva+2wG0DuF9CiyY/vWux6qVpO0SX+lgp1/vu53T3eIaJ2mKNw80r2XNLrW8pTGCVCNMOVvH3voPUNF8HdxbP7/9q13PYbzpIQSTAjeFVWVsjsHRQPgzegzk1CanyKrxvcN4ToJIXYc1Qjwb6roweZS9OY+X+DSSmWccV+C+4LcOQOCpqLhmEn29Wrl+8OTVwSdHs2XPGcnQY6MDRDF16MaUeqBsZM7iE7sbDk/ig9AIinIA2SZkaVQ6lnOWHrD9J27FXRuh3Ataf3nSMd+lpPRzxHkZ2nUr4lUAr8AACAASURBVOXkS/8HIjuAlNEf9FMq3Uyp9//js/tvnVJkNxEjuT5l6JUHOLzyM8ThtaT1X6Y+9nlK8UE0GGZG/eR8gt5KpA+y6G2Xw8ZxJjnNu8QnqduT2y2IuYGnhtfBUnJ5tPPH2769rQ0pWNGWVPxUl3ASPefAf9SxSyNCfDWiJmBN+5yoIqqHTfwAdPbC+1jPQbf0cBFnaOMrO4orooOO9I+rn+MQBEZcs1pnlVYONetHTiyI45GgEaRtFq6m1wIDHcnwY3n17ok9RlGoC+SFSGWCGwiE0yrc25yHbzx858Ht1aGN4v4rno19VFQeEo0Oi2hK4RgaL3snglmmDstd+DCjcVSYGZjw2hJBjCPFSBPu48sue76myAtISPPzLc5B8nMQZRVu88enq/g2S8F9GtNOPoaITPrdEcFAyiqyF3dEirAmwRR6BVlRrWJr1xLltlyMgkE6uh2V/VLEznrWKLv5RbCkH8Al/KxoZDhWOHNURA+QsTe/dKeTauhn96wkYvREK/BsXe5gQlGG8f71fGbPGyd8Fu99I5959k14I8ZtBFFDxBC/iS27TnEfSUqqdY6uHeWui0Z438tP8K5XHuLoXzzO0OGP4GPvIEv/BNE6acOwdDUiG1my7JKOITxNafKOl9c48ud/g/a9i3r9DtLGnxLFJ9AI6jXQsJhS+WMs3bOqGZI0UcX2JuMZt8xPbY+jzSvj1BCpC1ITpCZyZh+EGlBDfHoJshN959SLPSFPPHZncOJdVgwucjzKQsfAb0isp+fQMHBMVWkvC+wO4tILEkNhMyzGbf2djjKvNfdoUz+104RMYbyGTX64kiTRRqTmkp9H03c/V2+gavWF3SLH/ou4v8fTsd8F+WNURmj6porxRFDPUhC9JoR0DWitKfw0YwUACFNfpM30wsyzurTJSs1XiLur4QvcPPY2ppFL9lkaEXUMiG97kRwZZw5FzwV6Ef8ndxsZZ+aOmmW94K+47JYl5YGBwWU4a1pFkQ1RnkD0ADC+sJ1GpeVZyJYmSaK4r83PurjOKlia7g2hdPA0pr5F55nGQTbVV/cKyCCWKY0xQ/RWouiPCD2fm/iJ/yj/lN6PWx9uSqMGGl/B96KVM4fYOJTHtPOyC9uMw2v2kcUfAdtCFEd5LCSXIvqOZsjYVPrb7J53Lh3lhVXbKcfvx+obCeEQGnImKXI5pu/gwgMxietEFRumMsJTqN2ipDmDo+ZCzdXqLlZ3L75ltm3qAjXwus2kBHSi7xxGII0/jrnEGkkeqNuyXTVvXJd6o6EdCysAVKuYIB0YqBgaVCZyiVlh5uq92Sn3mA06BsmfEZqmgSStVF44uGHDi19qjI1+yN3vEuFA4T0eH89xVKLY1K91UqWI5/TCwTPZMz89/cW3FDpsXso8br2AJrhL0jRk07zkmpCxcRW6SamBO+UU9uCyVzQycTcH3LNYkRXn/yCdLxGXiJb6MENENEsbdXWextLv5jZJDMHcWCoNX/zEE6v6EFbiha3U3VTDCGL/dGYLuZ3FszLOYPQNSGFL1qBEpQFgGSJLO390MSGKgNzuV4oW4375zI4agU5l9NvV96MrhsjsHiwbHY+Qc7uVe3f1zZgt01L/jRUHRvDz/gRr3IOEEUQhrZcpla9mNFsGc/AEpSmIWj2gGJh625uh+aKcZdudVHBcT9MGOUfPcLWKVSpphER9orlHeFzykkLddclVhZz28ZqGDr2lkk3jUUy0Urkwdk72NVlqy/nh6m41F6nLhBqJZ4hxlTLMvN8s0KJzbkX05hxVKsnw0MJlWwaODcVBo4+5Wb9IW9FVHHHWgMduTRUcaIsBPRXG59llvOakC3VEwFrsMZckJY4yZszbdbfzRbStXsr4CGnJ5TBBtnor9lFxjBAPYukCsNeqKJm4iUQK2d5K5ej+rdsu2Ccan3DL+t1dRWxQRFaMjIwckuCL3VtXwtyPoZxe9kzz/Jrc8UxtkPfuvRT8NWSN3K5kthfP9mAetdJrOw3tA2i4FKxMo94P0ev4+D99ie+fGMkXy/r26dHRYq5P80f7dhNK64qCFSuQsJIkyVMaT/UCuf76lOQRWPgzX6As/waXDQgpqsvRxjIS2TdRxT6ddMKNG4tDPBWRmkNNoO5IzZGaS/E5jTbqNReti4fTu4RzJEHmapSWaa7SKC0lU3Nj4xFROdQ+Ty0Hji2uYx09dEkCjdLIgIsvNjOgXfoUHDuheYXjlq3wNJhS59PPOM3whNPs/9Q4VQBztZqkg0d3W+S6WzU6RFtgeZ6P7gAxPiGb5bTombCvkJfTcx8SpD6+zEfBdTVEajbVeVOcSxF9wEpErKm+53lNggjHwWrm2T+4pXVENF9SRUxF+qGxGPe1ZllhRwSQJ5MkMXU9KKJDCCaCOl520VeGYKtVS3mWkGOiQS2r71Orn17udfPkzxYRNxKXI/KMpRouG3n+lb+Enn8bPaXpP0HuIpSeyV9KppTii+ntWwnbjLMNoHbJFwVzz71sQeaf4ohJqBiMHaFeP4Bqmj/O3otob37Krb9nhsjNTWuKmEEuR07Rfjrxu6nPjpF7XSU79xLkxLp/UKmgSZKk69dvWolk42EW446/nA8edOGo5OEhxc+Cu6mIDqpwCbBzciB1ksD6DaxRiRabp4wvN5BXuUnF0n2GRHqGrOicmmDPoP9OZdSa8zxRwk40l9qzMnh5siMwd1n5CYR+0dzHebr0tDQANHegaOruB1TCCcda0qKTB4wrVyVJ8qVOmkClcm+fua+T9vvZx42jB8BHXMMeNfYDa8wzlTy4e74RLhVhZV60Q3C31Mi+AZAGORwsPYSzGjBRAdFV7vYDFaWotI5IhEj69Wr1fSfOrIiwnNnNkiTKsn/fT+Pk68kaoAFE9yAndwDw/JJa5wML5jfwjv301J9Gw7p8jRlbidvFcN0cxDrnWWb5v2ago62c71nWg4t+2vAf1HKeZNY+SR1Y48RMjqntAm2MXyH1fGU6y4qU2BwtBaa1TSe1WxARyzNWbAYJshN9p4/JD0ClklCpJLr1Eb9LVPvNsjw+zwsmaKkiPEua7XMNI7j0uuQ5u7ntSGNxfxvwp8UImveLwoVRaiOvV2WBu1vTGC+CqZaGU8+eELefZ8JbY/bnNc0V4mwtKGf2LCVarS5a7mK3O/5MpXL/1mr1jmm88HDllQN9mcstkqYrEJ9EsIDotwS5zJuhQPlmbb+zZsbE2VEJqWm6C5FDIEvHexHUrAGU3vjwwwvur1SS/fnSxq2eTLhRJVpheXC7FhRansrOznovwyHzuro+jdvaptfZ3frEea2jA4ghqoAcDsiTAFHmQ+bZXtFSxTyFzFXUVpl5LJKNu/TMGmTIGdZXPxsv9kZo7LuEnvJqxk6ChgjsSYLlDq0Z6ywmyvFVIyx69h+Ie9/C2EvzcesnlK/ip1Z8gUsPjHB62eQth9GSvQO4ryJLc6btNkw9O3L65/eDXlwGsbQo2yajICMwOdVwfIXA5k0jrfY0T4umpRTSmqOWhzugrcfcaQmUxcbJAmZ72y0X1CSawYvdib7ZY+3aJB4cXHS1iS/1NN3nrieiKMRbt/pKUb9DVG81y3TcvuS5ucXhYObp0yX1Iy6lRxG/Ec8lcgTFUtMQ3bi+cu//1hjr+X96eg4VMWoLyyYnbw3S83bL0phchcpVJtHIspMHAjxs8PNeLHrkM7C8TpjgZsgdSLTbICevHHk6aB07OyRJYus33Ls60vPuzGxsmVntmfWVz2zH7B9V2Z8GhqJMLAvSGzJfaeLvwv1N7lY4UYq5QcnS2qiKPezwC+30nO55tJ+/4+oi+ywd+6ZoWGd56FbO7NxNlLUhkg/Coru3bHnhcJKQVqsXxnnNR/+ISRp5U5b1XMbVEO03sr+76crjI7t2ra0NHRv6Bwi34pTzQPJ0PrABsd7WlZKdwJE8E+aukfXXf/op1WjY0rQ/L4jhqwVZbtbIox60hFu2uyRHnzytk++E5vM203KsTSSee5Nl6XqcBagaGp2g0djG80PD8MDMYyWJkWxULNpO/eRhRPoRNczWMy9dyrZte1j0zkkHzeKhXvJ8GdffptSzgEbNiGIwHuPFVUdy73el5c2eaclZqkr2skvp6bmYRj1Pa/TsAMYhEtepSy6cUT1IrUsza2Py8ZM16RnahhgK0YTg3kk4i3qQuXTzU72m4VfE7TcJ0Ql1GTUhQhlAQtkss0lDGGAisr3k8QGIR8xH/0IlrMN1QdOp4DmTBJcPx3Hj1akt3HbttYxmLlep6O2epUvBtWlbaxaeyCz9XP1kOtRT1gjBcLS9HuRsMZVlZMW8hDNijNB8lGdPS5IkumULkWSsymx00N0jCdGlAusMUhOGg8mwo6mYlc19UDXEmRW1KNqcHqKKW/b5RoPDUezllg9b8NNw0sCkF4N7/gIJ/ldCuFHUV7lleYiNoG5ZJITbHR+8YHDwi1+r+rGgtVWWydtEdY2bjWsADiaqdcuyh+aVSzvzEKPd6QvbFz0j6BHwFYVwoUBuG3Mxx8zddo6OlIab8/a17faMWXZCkCKHXGKYGHcqKtXqI8k06uypZ2EqNkIyUzTARqCqLBlcisZXktbLedSF7CewO2dC15/aX5CIkTxygMVLHyOetzZP99OVqFxBkuxm0+3ka08V8OKZvo4iYHsjucpaqM6Lvr0Az94KelcRagRuJzC7H6rK4LLL0W/3k922k7suOjI1pKjoKxHj3r2XEOR3SRurwYxo3ijpS9tYYIcY6iRBTodpHDgaxtLM4xqSV0M5mzx4AcMhUzk9G+RpPC31uBzHKQs89zAOoDIghSrtZHnwdrPb3GZlInoos/pfBV48AZDFi/5eG/yChNJveFYvN1W+/CR8vov8RkDfCpK6WX9epqrlnRUXE1V1S78QGPt8Z4/zGbpG5Ix9lB26On0MDv5Ur6Gvxr0XUMtSy/3FROLaj0o/4uNOmMzSybdWKqqK2ZMe/F5ixnn9mUnAHc6jAcdeHHx84cKhTaLh4+QRNCYi6oJC1gv6JhWtAKPu3gfEZqZ5EXsHxDSUEOdxs9q9Dz74nuMA1eojkbL7oIscQFg5ZXwRUwnHzPyfb7nl+RrkNuqr3pDuK9X0gGi0sjBUNZlwbj7FasC2fP8zWXvHARRLI5yL2LT3ZngO/Fe1df81K+Y3289C9DLDWIPIxUVoD2SN3YTy1NUBZ0Jyfcpn9j6IZe/GHUKIsfQm4E8mO+EQYsT72D04zIW/njK6OyJ6Wxn2LiCTdZTC67HoTbgtAIworuPp54nqW7lwRR+mb0PCrdT9m2za8yD+rd2kpUMMMMxL56WE28qk+xZz395LifRdIFdjmVEqK86TpKUt7H5FSlIwtdmZqjo/sHWLLcJriMbkthhMMHVTkyh32bppvq1gPqKFimJKsX+zPwXIZggU74RZPjdJkthrX7u5TMziwnsMnqdw5fbrdkkjV/5D6BnNvPG5gD7ctpzB0A03fOIPGo3yAo3i2y2tNyWaXDV3U3fpQ9wQz+v3FZKPoIiqmttXAvLhavX7w5XKwl6bUUL/yUA+v5+YX4rDxS5mZm0vnPwFpLl0MEntzf/Ns0tCrJ6lzxD8w4svGHzm8IkXFnQebXbocGtYCKndfvvu9IknBv7kpZPyStHwW+T1N1NBiqfBcJMyeWFammuku+dZPSGU1PG9Da+//xtfP76nybSq1W122WVLDp/Xlz4jGq5xyyLaXroI6iIHVdnfnDOAN1yVnPhadeGOoGFDXui3FWCV2yzZL954uv2Y00I+x0paLxNKt1OK3zTrl3CWlUkb/eBQikcYe+kJDi87cdqLcIlvJ02PoNFg7qxhPZv2DY4vP49ofhvI5YSwGWSYWqNOiCKM+USlBZRKg2SNATzLmWpcTmmMfYGGf5yja0+waM9yovJrEF+KyFuJz9uAZ8fRxnFG/BiM1ElLfYQwSFxaSv1kwWR7FPchxkY/xNE1+5vnNlHgG1dX2yeu2e7MhcolTOCkZz7q4qPuPiomNXcZFfOamNda2/Lf3bzmxfb8t3w/cR91l9FsxjjITvTNHqVSvdexQciZFS4mxSdPe5O0CKlINcRDDat/eNEFA/8lL4TQujGvuebEIZEjv25p/ZOi4VirTmOzVqNT2NVM0BTHVCOTEB9yz/6vQPquavU9z7Q7AYq0RcPF2p+pjkGzraMoDMtN+ovtgbT15kvHf5dgrRTCTjjJeICqF7RIUQl4Fo9DVupRkFS1NKIarIitMRFJBTWcPG3O1fJ2HjKjoZRq6DnmWf2PLbLbtq8/+vBFF+1uuw/yfvL9i3Oc1eOpNK9JM60xyyIFuPLK4yPnzcs+hGXvFaI9QeNiPClSIL2Nkef0qqppKJ2wrLElqzdu+Ub1xR2txcEAEnvqqedruD2hWjohzb5a18c8G9sD9XEJrOn1D/A1MwMN7fsX9gd/cmysMTQ5rXLWEPL7BAHL+qifXEy9NrtPkzlqgLQxhPmjpx2ek7hy56uOoeEhQpQ7Yks9g3h6I9Rb9ImmqPQTQoWo52ZKpbcQ4lsJ0QbMLqZRGwSUuHcUZD+1l95Pze7k6CtypqZaJkQpUZybIhq1ftJ0JSJXEKI3EUpvRsONWHYJjbEBRCGeN4LZwzTGfpGjax5vJ7tDPcjJjHBm8axu5BWfFdP8T4H266gdtnVoN3OwZ7JBdqLvtKSvKBL0sKiWTaQPtzJ54QkDqSMyjPsQlu0Usb94tPrbDwM8MMkWXTwQtUrl/g+kfvKL6nabhJ5LgWW49UlegFVB6yI6jNgRS9OnTep/dnxo0WO33747bYZqnH9+ZN//QXZYNX7aMFQL35UEGo2TB0qlUsfsjgaMlDXeIRN0VDFERyRNR4AR1Z4draI2CrghOuI6Ntxxek6GNJSj/aj0mQYTXB1MpaSucqjt3Dvi8eoLB6+5ZvBOVasgvFajaK0QBtyZD152L7SWfC2WuiDH3bMhz+o7UR5UOfbQhmuxR5PEEhK9+sYoVQ0HBN1pmk2gJ5NakW43MaQqSUA0OhZC/DRCLG03mkjpsPjJ0eYSq0mSjFSrfLbuCx8LJreFKGxwD0vzXG0rjpVUJIwAx9zGnvEs+++qjYe2P/q+E52X+YVqlR0i4fEQlZY1tzuYalxv1EYeqX69FarTCpy/d6e7PR6intjVinPNXyBpdvJrPT3DwzOVmpsWlg0T9T4DVj4jI5ijBUNTRr/3GPN69p7u2i7jCPwVIaxFepSe82Cs9mpMHqdU3oPQh3kZiPHm85NnF0GooTJKo3GcNN2PNZ5ArMp7Xr13Qmrh86v3snTPHWR6IyLXEc9bBT6AWR9mEZiimiLRKBKOU39pH7XRv0PCF3jPq4YmO67yJ+uze2+g1LuZdGw5WTadwp3r6I3aX/Kq//W2ZFvFkkTs4986uQLxN6vPQV5b4eixzKvvW3teHmN1775V9ER/i9uaYvW0Dge6EfVAlj3N83922UwXr1K5v5yFk6s9s+UqMmDIAnWPwVLxMOyeHVHVg8C+SuXo6GzVmZtu+uT8kZFohUS+SmCxYX3iquJ+3NWPqLf6hElMJkn0tV/tX1YqlQbaOWFQVxdGouzY/k6LTV150yfnxyO6KgstVScGsiAWsrGDJ08Gi+Ppf69W33dicp+33bYlfv740Apx+jJrHRfU1cZKx77xjTtPmQPcZBqVyr19WQjLQ9YYNNEBy7yfQF4d3RkVYVjdh0APQe+havWOGsWSuW3ZNhEsXJGpz59MTzAZrlbv2teJhqtv3DQY123p1DeLpmPn6/6nvnjnuFzelOB27VobHTl+fJVYusKdpYL3g0YOI2I+BHJo3ryePQ8++JvHTzUHt922JT569IWVmUpvO90A3jN28B8e/A8d+kj06spPrw1ZiJvX7FTXa1b4410D1MMymqnFTWGoUXzP1G7/PxJljCF+75WHzogOgHt39SHzVhIKPpPKML3hEA1bTqO+gCjqwzxGPcI9ArW8iogWoTc+hDeGOLo2v36d1PymY2fZoX7Sl1biuhjxAdA+3CPUR3E5TqZH0Jf28Z6fG5qO3JzbbNqzgZ6+zaS1FTmX7Yj8DdKo/w090duS766oJ4nYJ58bXeaZ3+yEGMfOyktjBqpIJtX3ru3J04U2P7sGjf8WfNW0DNLdKPWAZzt41yt+YeoOE9G+/nG+ZOtLOjT0Xbv9dtL2dZFP19bTYgxJBBcW8/jdZimufK3safucSXWa/phKBW0vedUsk9XcNt3veYzf6fU78zEdeimqgrevTz15/NYa3zP1e/r05BELE49p+3WasI8Wc06SRHftIjp69EJtv4ZF37Ocg6nX9NTzOPGY2V2vU5Exi3VgZoWqwjY7Y+lxCj3NcJxpajlOe9wM+0zYv2CUrf4Vqkwc8+4ZUxJzbrP52Wso9W6mMbYan4FBaqRY+ijiv8Tzq4+TiG1+1hec9Nobxa0X1bP0oBpmmhJk+/f//P88kCSJsenZKwjRF4EFZOn0EmRpHmTpdt698vrZj9fK8ICm6jIXC4ZN7vfHbRGyHxXaM2pgbub63GFittWPN61dzAKniovsACFxZelzl1Cat5n62OXj3qGOfhkB1b1kY7/MC6/eTSJ27y7vS8NL17iEQU5Zx/HUUPfR1OZVhx/gRJKIsXnv2xG9H/N4gkNmAn1uxL2QNv6ad6+8bVYBsF100UUXp0CzWMUwaTact8fTuXJMKExrRqmnHymtgbtJ3PXoEDVTjoh7TfC647Uz/Yh4aipDw0O0ORDCL6AhHndZji9X10afA5aBUtjHZrn+bhdddNHFDMgZZNw4QTZ2pChZNFHymqzSZul84Cou/PU4AZLrJY0bHBHXE47XBK1LpnWh7XPKttcFr5tRH3Pbz7a7cxru/04ZYUPhYe6cqSPFtiyFzJ6d+ynqoosu/rUiZ5CH1p7A2UUUj+YS2jRhMyJKlsbEPeupp2uboVBHh847JioH1b2mntZUqam3fU7ZDjXB63h04OSreo/AxrwOx8n6G9FwMWld8WncP05RXUSOIeSOnblcg7aLLrr4V4vWUonC0+CdY+Pa4Q5ZuhbRm1m4u5ck0eR6SV+M4wOWlo5khLq518y9ZqH4tP/f3m7bniHHYi/tTUQsgTzfslS6sxhzyuJTEyGgYTcuh7r2xy666GKu0JLKgj5NOnaIEGkH70wbXHEvA/8WDVfkbnTX5OVSmzcW71NPjyleV3wio/S2Txtz1NTrkqbH5WR939G1jJK4suSpMpK9EwmvIa3TvnznFIgYuGHZDsbsBFw3RyENXXTRxb92FG5vMf7XoSNktpWoB5gpk4XcIQIr///27ifEruoO4Pj3d869972ZvsQYnTCRYEIYUpmFRBoGXdVAd13ZVpe1QWiKWVYLUkrvUIrYLooUq6YuFARtCy5aKaWbDLRKrS66KLY0dkwlZpKZMB3j+ObNfef+jov73sub/2/GSSPl94FhOMx973Bn8eOce3/n98P5H7L/vapgZR7d6RPS/O++xrRGuaROm1LGIJIUErQQ6fsJWlR/06IUuVxvNqY/Or7vWt7dGWvjXlz2CGW7AVvkcImAS66i5RvMjy2Sn7zpLWONMf8fVi4Vf/HPu3H+LYQM7ZSFiquu7tWHFCWtKaF4lVA8ztzs1W4CZh6jOzhDPSx/spdm0mg5XHSFYxnqaaaFoknQlk+GFubGaeYiSn4ugfuVQ++fILpniXo3ZTtZVeVj1ePRCN4r4v9AaJ3hyl0fbPsAvTHGbGDtXvr5f7+C9w91muC4zXfbUcnqBWX7t8TiKW6Nf+fd8dAfpPJzMeEIyUhzLoER5marPtj5SQnXM+MnYeTBYZyfIKs/g8a7KNsbTLpq/trwAq3mE8wee2GrrHhjjNmO6+Gv+3Lj7L++giQvEXWUUjcPkFW2tuLTgJbvoPpL2vIa82OLOZOdjhAb5CT2H/85cP5OvDyE84+AHKVsb/0cMaIkCSBTEB7mw7FLtno0xuymleEvzx2HH95LO/wY5Nuods4vbkkRgbQ2S2vpjzh+Ra35JqfuWVj3HGg3kD3z/ii++Bo++zqRE8Sy0TvJM8iczjtUH+Ty2GsrvtcYY3bB2kiUR8fBfxwn3fNzQjGBbljdp09nJQmQZAqySFieBvkLTt6mHS+RyiKxdJRxP94fBb5EZILa0CHay/XqxU/cOjjG7vPPuqLlr/mweQpWbuuNMWY3rB8gc1GeO/8NstrPCMVoFSQHLNsdY7Wa9KnDewgBNFR9dKvVaB2fgnMQ2lAG3TSNZ+0EikuA+FdieYqZV3Zem84YYzax/vY3jw75wu9pffIsiEOcDlyUVsQRoyMUyvKSom065wHrIBkxQnsZlpd08ODYPd0TOw165AKqP2UmTG/jXo0xZls2Xhbm0XHLhb0Mhadx8k1Uldh5ntjrM9qp5r3huG+K6+lBdBqUDPD5vjFU5eLTbJ6y/AHt1svMjTdta22MuVE2Xr3lonx05Bqe76O8iEsCzmkv6PWauMsm41U5jL1CE4N+vvsVUq0c01qL0H6C1L3I3G8sOBpjbqitHyzm0THy7gF88jhJ7Vto2IeuetPcW+XJjRgr3iuRi8T4JKfHzu74bo0xZhu2fv6XizI3PovwJGUxSZJdxGdVWbQYtfNWmV7zrN0aRxSRquct7k20/C4Mv3xD/xvGGNNnsLfHuSgzx+bJ0rOE9hkiUyRZwCeuU0OyIn1b452Pq+CbZHRSh14gLJ1hf/t1Zg62dnSXxhizA37gK6cmI/fcqnz8wHka8+dQvQJ6lNrQHlQFYlldGGVNy4beKrFroz7bUqXwJGmLMryDxu8RWs8xO36JuRG1Z47GmP+lwQMkwNRU5H4RFh+4xmO3vcFXH/0dZXsJn9ZIa/Wqx7QH5yIinf1ylPWDo4A4xbkqenrfojZ0haL1JzT8BIk/4jvH3mbiQCA/qUxNbqf5tTHGfGYDZn+vo9eshxRnXwAAALtJREFU+8uOO0aPojIBch/p8HGkPEQobyfGYbzXNdNEdagqIk18chHVC4Tib0TewvNnTn/xam8OSwI3xtwkOw+QcD2Adc9b73+vQcYhXLyDUu9E/GHSZBTxDaJmAGhs4uICoZyB+AGlTEOcxV+7zMzrrV4fW2OMuck+W4Bcrb8Rd34u4fCRhI9Dxp7EsdC5xgfFF8rwcOA/RwK5hF4tSAuMxpjPkd0NkP16W3BYWfJssjPu/LagaIz5nPoUBSp4D1AF9yMAAAAASUVORK5CYII=)"]},{"cell_type":"markdown","metadata":{"id":"3o5sAOfwL5qd"},"source":["[![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/JohnSnowLabs/langtest/blob/main/demo/tutorials/misc/HuggingFace_Dataset_Notebook.ipynb)"]},{"cell_type":"markdown","metadata":{"id":"WJJzt3RWhEc6"},"source":["**LangTest** is an open-source python library designed to help developers deliver safe and effective Natural Language Processing (NLP) models. Whether you are using **John Snow Labs, Hugging Face, Spacy** models or **OpenAI, Cohere, AI21, Hugging Face Inference API and Azure-OpenAI** based LLMs, it has got you covered. You can test any Named Entity Recognition (NER), Text Classification model using the library. We also support testing LLMS for Question-Answering and Summarization tasks on benchmark datasets. The library supports 50+ out of the box tests. These tests fall into robustness, accuracy, bias, representation and fairness test categories.\n","\n","Metrics are calculated by comparing the model's extractions in the original list of sentences against the extractions carried out in the noisy list of sentences. The original annotated labels are not used at any point, we are simply comparing the model against itself in a 2 settings."]},{"cell_type":"markdown","metadata":{"id":"26qXWhCYhHAt"},"source":["# Getting started with LangTest on John Snow Labs"]},{"cell_type":"code","execution_count":null,"metadata":{"colab":{"base_uri":"https://localhost:8080/"},"executionInfo":{"elapsed":23927,"status":"ok","timestamp":1689532651217,"user":{"displayName":"Prikshit Sharma","userId":"02603035510215936699"},"user_tz":-330},"id":"azUb114QhOsY","outputId":"30f8035b-250e-4bbf-da25-65cd552f700e"},"outputs":[],"source":["!pip install langtest"]},{"cell_type":"markdown","metadata":{},"source":["### Installing required dependencies"]},{"cell_type":"code","execution_count":null,"metadata":{},"outputs":[],"source":["!pip install transformers==4.28.1"]},{"cell_type":"markdown","metadata":{"id":"yR6kjOaiheKN"},"source":["# Harness and Its Parameters\n","\n","The Harness class is a testing class for Natural Language Processing (NLP) models. It evaluates the performance of a NLP model on a given task using test data and generates a report with test results.Harness can be imported from the LangTest library in the following way."]},{"cell_type":"code","execution_count":2,"metadata":{"executionInfo":{"elapsed":11992,"status":"ok","timestamp":1689532663205,"user":{"displayName":"Prikshit Sharma","userId":"02603035510215936699"},"user_tz":-330},"id":"lTzSJpMlhgq5"},"outputs":[],"source":["#Import Harness from the LangTest library\n","from langtest import Harness"]},{"cell_type":"markdown","metadata":{"id":"JFhJ9CcbsKqN"},"source":["# HuggingFace Datasets Testing For `text-classification`\n","\n","In this section, we dive into testing of HuggingFace Models for different HuggingFace Datasets."]},{"cell_type":"markdown","metadata":{"id":"iO7jyI9F8DQ8"},"source":["## Glue - `sst2` Dataset Testing\n","-------------------"]},{"cell_type":"markdown","metadata":{"id":"ZtqqWrqO8DQ8"},"source":["The provided code initializes an instance of the Harness class, which is designed to handle text classification tasks using Hugging Face. The Harness class accepts a data parameter, which can also be specified as a `dictionary` with several attributes.\n","\n","The `data` prameter also takes a dictionary which contains the following attributes:\n","\n","```python\n","{\n","   \"name\": \"\",\n","   \"subset\": \"\",\n","   \"feature_column\": \"\",\n","   \"target_column\": \"\",\n","   \"split\": \"\"\n","}\n","```\n","<br/>\n","\n","\n","| Key  | Description |\n","| - | - |\n","|**name**     |Represents the name of the dataset being used.|\n","|**subset**     |Indicates the subset of the dataset being considered.\n","|**feature_column**       |Specifies the column that contains the input features.\n","|**target_column**     |Represents the column that contains the target labels or categories.\n","|**split**       |Denotes which split of the dataset should be used.|\n","\n","<br/>\n","<br/>\n","\n","`It's important to note that the default values for the split, feature_column, and target_column attributes are \"test\", \"text\", and \"label\", respectively.`"]},{"cell_type":"markdown","metadata":{"id":"swaYPW-wPlku"},"source":["### Setup and Configure Harness"]},{"cell_type":"code","execution_count":null,"metadata":{"id":"JaarBdfe8DQ8"},"outputs":[],"source":["harness = Harness(task=\"text-classification\", hub=\"huggingface\",\n","                   model=\"distilbert-base-uncased-finetuned-sst-2-english\",\n","                  data={\"name\":'glue',\n","                  \"subset\":\"sst2\",\n","                  \"feature_column\":\"sentence\",\n","                  \"target_column\":'label',\n","                  \"split\":\"train\"\n","                  })"]},{"cell_type":"markdown","metadata":{"id":"jWPAw9q0PwD1"},"source":["We have specified task as `text-classification` , hub as `huggingface` and model as `distilbert-base-uncased-finetuned-sst-2-english`\n","\n","For dataset we used `sst2` which is a subset of glue dataset.\n","\n","You can find more HuggingFace Benchmark Datasets [here](https://huggingface.co/datasets?task_categories=task_categories:text-classification&sort=downloads)\n"]},{"cell_type":"markdown","metadata":{"id":"MSktjylZ8DQ9"},"source":["For tests we used lowercase and uppercase. Other available robustness tests are:\n","* `add_context`\n","* `add_contraction`\n","* `add_punctuation`\n","* `add_typo`\n","* `add_ocr_typo`\n","* `american_to_british`\n","* `british_to_american`\n","* `lowercase`\n","* `strip_punctuation`\n","* `titlecase`\n","* `uppercase`\n","* `number_to_word`\n","* `add_abbreviation`\n","* `add_speech_to_text_typo`\n","* `add_slangs`\n","* `dyslexia_word_swap`\n","* `multiple_perturbations`\n","* `adjective_synonym_swap`\n","* `adjective_antonym_swap`"]},{"cell_type":"markdown","metadata":{"id":"zCP1nGeZ8DQ9"},"source":["Bias tests:\n","\n","* `replace_to_male_pronouns`\n","* `replace_to_female_pronouns`\n","* `replace_to_neutral_pronouns`\n","* `replace_to_high_income_country`\n","* `replace_to_low_income_country`\n","* `replace_to_upper_middle_income_country`\n","* `replace_to_lower_middle_income_country`\n","* `replace_to_white_firstnames`\n","* `replace_to_black_firstnames`\n","* `replace_to_hispanic_firstnames`\n","* `replace_to_asian_firstnames`\n","* `replace_to_white_lastnames`\n","* `replace_to_sikh_names`\n","* `replace_to_christian_names`\n","* `replace_to_hindu_names`\n","* `replace_to_muslim_names`\n","* `replace_to_inter_racial_lastnames`\n","* `replace_to_native_american_lastnames`\n","* `replace_to_asian_lastnames`\n","* `replace_to_hispanic_lastnames`\n","* `replace_to_black_lastnames`\n","* `replace_to_parsi_names`\n","* `replace_to_jain_names`\n","* `replace_to_buddhist_names`\n","\n","Representation tests:\n","\n","* `min_gender_representation_count`\n","* `min_ethnicity_name_representation_count`\n","* `min_religion_name_representation_count`\n","* `min_country_economic_representation_count`\n","* `min_gender_representation_proportion`\n","* `min_ethnicity_name_representation_proportion`\n","* `min_religion_name_representation_proportion`\n","* `min_country_economic_representation_proportion`\n","\n","\n","Accuracy tests:\n","\n","* `min_exact_match_score`\n","* `min_bleu_score`\n","* `min_rouge1_score`\n","* `min_rouge2_score`\n","* `min_rougeL_score`\n","* `min_rougeLsum_score`\n","\n","\n","Fairness tests:\n","\n","* `max_gender_rouge1_score`\n","* `max_gender_rouge2_score`\n","* `max_gender_rougeL_score`\n","* `max_gender_rougeLsum_score`\n","* `min_gender_rouge1_score`\n","* `min_gender_rouge2_score`\n","* `min_gender_rougeL_score`\n","* `min_gender_rougeLsum_score`"]},{"cell_type":"code","execution_count":null,"metadata":{"id":"5DNBjDLp8DQ9","outputId":"b184a72c-447f-45a0-f77f-19782fb4a15f"},"outputs":[{"data":{"text/plain":["{'tests': {'defaults': {'min_pass_rate': 0.65},\n","  'robustness': {'lowercase': {'min_pass_rate': 0.66},\n","   'uppercase': {'min_pass_rate': 0.66}}}}"]},"execution_count":8,"metadata":{},"output_type":"execute_result"}],"source":["harness.configure(\n","{\n"," 'tests': {'defaults': {'min_pass_rate': 0.65},\n","           'robustness': {'lowercase': {'min_pass_rate': 0.66},\n","                          'uppercase': {'min_pass_rate': 0.66},\n","                        }\n","          }\n"," }\n"," )"]},{"cell_type":"markdown","metadata":{"id":"ZPU46A7WigFr"},"source":["Here we have configured the harness to perform two robustness tests (uppercase and lowercase) and defined the minimum pass rate for each test."]},{"cell_type":"markdown","metadata":{"id":"KVolf25u_OFV"},"source":["➤ You can adjust the level of transformation in the sentence by using the \"`prob`\" parameter, which controls the proportion of words to be changed during robustness tests.\n","\n","➤ **NOTE** : \"`prob`\" defaults to 1.0, which means all words will be transformed.\n","```\n","harness.configure(\n","{\n"," 'tests': {\n","    'defaults': {'min_pass_rate': 0.65},\n","      'robustness': {\n","        'lowercase': {'min_pass_rate': 0.66, 'prob': 0.50},\n","        'uppercase':{'min_pass_rate': 0.60, 'prob': 0.70},\n","      }\n","  }\n","})\n","\n","```"]},{"cell_type":"code","execution_count":null,"metadata":{"id":"kvHcQi3M8DQ-"},"outputs":[],"source":["# Limit the data to the first 2000 samples\n","harness.data = harness.data[:2000]"]},{"cell_type":"markdown","metadata":{"id":"i6kPvA13F7cr"},"source":["### Generating the test cases."]},{"cell_type":"code","execution_count":null,"metadata":{"colab":{"base_uri":"https://localhost:8080/"},"id":"mdNH3wCKF9fn","outputId":"cd348490-7ade-40fa-d870-dc059f5aa647"},"outputs":[{"name":"stderr","output_type":"stream","text":["Generating testcases...: 100%|██████████| 1/1 [00:00<?, ?it/s]\n"]},{"data":{"text/plain":[]},"execution_count":11,"metadata":{},"output_type":"execute_result"}],"source":["harness.generate()"]},{"cell_type":"markdown","metadata":{"id":"nyjDdYLeGCmM"},"source":["harness.generate() method automatically generates the test cases (based on the provided configuration)"]},{"cell_type":"code","execution_count":null,"metadata":{"colab":{"base_uri":"https://localhost:8080/","height":641},"id":"c0jL1_G7F_p6","outputId":"502c1525-9000-4041-823b-3b04f6650892"},"outputs":[{"data":{"text/html":["<div>\n","<style scoped>\n","    .dataframe tbody tr th:only-of-type {\n","        vertical-align: middle;\n","    }\n","\n","    .dataframe tbody tr th {\n","        vertical-align: top;\n","    }\n","\n","    .dataframe thead th {\n","        text-align: right;\n","    }\n","</style>\n","<table border=\"1\" class=\"dataframe\">\n","  <thead>\n","    <tr style=\"text-align: right;\">\n","      <th></th>\n","      <th>category</th>\n","      <th>test_type</th>\n","      <th>original</th>\n","      <th>test_case</th>\n","      <th>expected_result</th>\n","    </tr>\n","  </thead>\n","  <tbody>\n","    <tr>\n","      <th>0</th>\n","      <td>robustness</td>\n","      <td>lowercase</td>\n","      <td>hide new secretions from the parental units</td>\n","      <td>hide new secretions from the parental units</td>\n","      <td>NEGATIVE</td>\n","    </tr>\n","    <tr>\n","      <th>1</th>\n","      <td>robustness</td>\n","      <td>lowercase</td>\n","      <td>contains no wit , only labored gags</td>\n","      <td>contains no wit , only labored gags</td>\n","      <td>NEGATIVE</td>\n","    </tr>\n","    <tr>\n","      <th>2</th>\n","      <td>robustness</td>\n","      <td>lowercase</td>\n","      <td>that loves its characters and communicates som...</td>\n","      <td>that loves its characters and communicates som...</td>\n","      <td>POSITIVE</td>\n","    </tr>\n","    <tr>\n","      <th>3</th>\n","      <td>robustness</td>\n","      <td>lowercase</td>\n","      <td>remains utterly satisfied to remain the same t...</td>\n","      <td>remains utterly satisfied to remain the same t...</td>\n","      <td>NEGATIVE</td>\n","    </tr>\n","    <tr>\n","      <th>4</th>\n","      <td>robustness</td>\n","      <td>lowercase</td>\n","      <td>on the worst revenge-of-the-nerds clichés the ...</td>\n","      <td>on the worst revenge-of-the-nerds clichés the ...</td>\n","      <td>NEGATIVE</td>\n","    </tr>\n","    <tr>\n","      <th>...</th>\n","      <td>...</td>\n","      <td>...</td>\n","      <td>...</td>\n","      <td>...</td>\n","      <td>...</td>\n","    </tr>\n","    <tr>\n","      <th>3995</th>\n","      <td>robustness</td>\n","      <td>uppercase</td>\n","      <td>when there 's nothing else happening</td>\n","      <td>WHEN THERE 'S NOTHING ELSE HAPPENING</td>\n","      <td>NEGATIVE</td>\n","    </tr>\n","    <tr>\n","      <th>3996</th>\n","      <td>robustness</td>\n","      <td>uppercase</td>\n","      <td>on cable</td>\n","      <td>ON CABLE</td>\n","      <td>NEGATIVE</td>\n","    </tr>\n","    <tr>\n","      <th>3997</th>\n","      <td>robustness</td>\n","      <td>uppercase</td>\n","      <td>it with ring ,</td>\n","      <td>IT WITH RING ,</td>\n","      <td>POSITIVE</td>\n","    </tr>\n","    <tr>\n","      <th>3998</th>\n","      <td>robustness</td>\n","      <td>uppercase</td>\n","      <td>far from a groundbreaking endeavor</td>\n","      <td>FAR FROM A GROUNDBREAKING ENDEAVOR</td>\n","      <td>NEGATIVE</td>\n","    </tr>\n","    <tr>\n","      <th>3999</th>\n","      <td>robustness</td>\n","      <td>uppercase</td>\n","      <td>that these women are spectacular</td>\n","      <td>THAT THESE WOMEN ARE SPECTACULAR</td>\n","      <td>POSITIVE</td>\n","    </tr>\n","  </tbody>\n","</table>\n","<p>4000 rows × 5 columns</p>\n","</div>"],"text/plain":["        category  test_type  \\\n","0     robustness  lowercase   \n","1     robustness  lowercase   \n","2     robustness  lowercase   \n","3     robustness  lowercase   \n","4     robustness  lowercase   \n","...          ...        ...   \n","3995  robustness  uppercase   \n","3996  robustness  uppercase   \n","3997  robustness  uppercase   \n","3998  robustness  uppercase   \n","3999  robustness  uppercase   \n","\n","                                               original  \\\n","0          hide new secretions from the parental units    \n","1                  contains no wit , only labored gags    \n","2     that loves its characters and communicates som...   \n","3     remains utterly satisfied to remain the same t...   \n","4     on the worst revenge-of-the-nerds clichés the ...   \n","...                                                 ...   \n","3995              when there 's nothing else happening    \n","3996                                          on cable    \n","3997                                    it with ring ,    \n","3998                far from a groundbreaking endeavor    \n","3999                  that these women are spectacular    \n","\n","                                              test_case expected_result  \n","0          hide new secretions from the parental units         NEGATIVE  \n","1                  contains no wit , only labored gags         NEGATIVE  \n","2     that loves its characters and communicates som...        POSITIVE  \n","3     remains utterly satisfied to remain the same t...        NEGATIVE  \n","4     on the worst revenge-of-the-nerds clichés the ...        NEGATIVE  \n","...                                                 ...             ...  \n","3995              WHEN THERE 'S NOTHING ELSE HAPPENING         NEGATIVE  \n","3996                                          ON CABLE         NEGATIVE  \n","3997                                    IT WITH RING ,         POSITIVE  \n","3998                FAR FROM A GROUNDBREAKING ENDEAVOR         NEGATIVE  \n","3999                  THAT THESE WOMEN ARE SPECTACULAR         POSITIVE  \n","\n","[4000 rows x 5 columns]"]},"execution_count":12,"metadata":{},"output_type":"execute_result"}],"source":["harness.testcases()"]},{"cell_type":"markdown","metadata":{"id":"NOJ8BAU2GGzd"},"source":["harness.testcases() method displays the produced test cases in form of a pandas data frame."]},{"cell_type":"markdown","metadata":{"id":"3CwhQw6hGR9S"},"source":["### Running the tests"]},{"cell_type":"code","execution_count":null,"metadata":{"colab":{"base_uri":"https://localhost:8080/"},"id":"aguX6-aFGOnP","outputId":"bb014811-522b-4f07-fa8a-bf3d1c906d7f"},"outputs":[{"name":"stderr","output_type":"stream","text":["Running testcases... : 100%|██████████| 4000/4000 [05:29<00:00, 12.14it/s]\n"]},{"data":{"text/plain":[]},"execution_count":14,"metadata":{},"output_type":"execute_result"}],"source":["harness.run()"]},{"cell_type":"markdown","metadata":{"id":"191O2oaUGWrH"},"source":["Called after harness.generate() and is to used to run all the tests.  Returns a pass/fail flag for each test."]},{"cell_type":"code","execution_count":null,"metadata":{"id":"XDbd1mpREWR5","outputId":"872d7612-e0dc-435f-932c-3e74406f38e3"},"outputs":[{"data":{"text/html":["<div>\n","<style scoped>\n","    .dataframe tbody tr th:only-of-type {\n","        vertical-align: middle;\n","    }\n","\n","    .dataframe tbody tr th {\n","        vertical-align: top;\n","    }\n","\n","    .dataframe thead th {\n","        text-align: right;\n","    }\n","</style>\n","<table border=\"1\" class=\"dataframe\">\n","  <thead>\n","    <tr style=\"text-align: right;\">\n","      <th></th>\n","      <th>category</th>\n","      <th>test_type</th>\n","      <th>original</th>\n","      <th>test_case</th>\n","      <th>expected_result</th>\n","      <th>actual_result</th>\n","      <th>pass</th>\n","    </tr>\n","  </thead>\n","  <tbody>\n","    <tr>\n","      <th>0</th>\n","      <td>robustness</td>\n","      <td>lowercase</td>\n","      <td>hide new secretions from the parental units</td>\n","      <td>hide new secretions from the parental units</td>\n","      <td>NEGATIVE</td>\n","      <td>NEGATIVE</td>\n","      <td>True</td>\n","    </tr>\n","    <tr>\n","      <th>1</th>\n","      <td>robustness</td>\n","      <td>lowercase</td>\n","      <td>contains no wit , only labored gags</td>\n","      <td>contains no wit , only labored gags</td>\n","      <td>NEGATIVE</td>\n","      <td>NEGATIVE</td>\n","      <td>True</td>\n","    </tr>\n","    <tr>\n","      <th>2</th>\n","      <td>robustness</td>\n","      <td>lowercase</td>\n","      <td>that loves its characters and communicates som...</td>\n","      <td>that loves its characters and communicates som...</td>\n","      <td>POSITIVE</td>\n","      <td>POSITIVE</td>\n","      <td>True</td>\n","    </tr>\n","    <tr>\n","      <th>3</th>\n","      <td>robustness</td>\n","      <td>lowercase</td>\n","      <td>remains utterly satisfied to remain the same t...</td>\n","      <td>remains utterly satisfied to remain the same t...</td>\n","      <td>NEGATIVE</td>\n","      <td>NEGATIVE</td>\n","      <td>True</td>\n","    </tr>\n","    <tr>\n","      <th>4</th>\n","      <td>robustness</td>\n","      <td>lowercase</td>\n","      <td>on the worst revenge-of-the-nerds clichés the ...</td>\n","      <td>on the worst revenge-of-the-nerds clichés the ...</td>\n","      <td>NEGATIVE</td>\n","      <td>NEGATIVE</td>\n","      <td>True</td>\n","    </tr>\n","    <tr>\n","      <th>...</th>\n","      <td>...</td>\n","      <td>...</td>\n","      <td>...</td>\n","      <td>...</td>\n","      <td>...</td>\n","      <td>...</td>\n","      <td>...</td>\n","    </tr>\n","    <tr>\n","      <th>3995</th>\n","      <td>robustness</td>\n","      <td>uppercase</td>\n","      <td>when there 's nothing else happening</td>\n","      <td>WHEN THERE 'S NOTHING ELSE HAPPENING</td>\n","      <td>NEGATIVE</td>\n","      <td>NEGATIVE</td>\n","      <td>True</td>\n","    </tr>\n","    <tr>\n","      <th>3996</th>\n","      <td>robustness</td>\n","      <td>uppercase</td>\n","      <td>on cable</td>\n","      <td>ON CABLE</td>\n","      <td>NEGATIVE</td>\n","      <td>NEGATIVE</td>\n","      <td>True</td>\n","    </tr>\n","    <tr>\n","      <th>3997</th>\n","      <td>robustness</td>\n","      <td>uppercase</td>\n","      <td>it with ring ,</td>\n","      <td>IT WITH RING ,</td>\n","      <td>POSITIVE</td>\n","      <td>POSITIVE</td>\n","      <td>True</td>\n","    </tr>\n","    <tr>\n","      <th>3998</th>\n","      <td>robustness</td>\n","      <td>uppercase</td>\n","      <td>far from a groundbreaking endeavor</td>\n","      <td>FAR FROM A GROUNDBREAKING ENDEAVOR</td>\n","      <td>NEGATIVE</td>\n","      <td>NEGATIVE</td>\n","      <td>True</td>\n","    </tr>\n","    <tr>\n","      <th>3999</th>\n","      <td>robustness</td>\n","      <td>uppercase</td>\n","      <td>that these women are spectacular</td>\n","      <td>THAT THESE WOMEN ARE SPECTACULAR</td>\n","      <td>POSITIVE</td>\n","      <td>POSITIVE</td>\n","      <td>True</td>\n","    </tr>\n","  </tbody>\n","</table>\n","<p>4000 rows × 7 columns</p>\n","</div>"],"text/plain":["        category  test_type  \\\n","0     robustness  lowercase   \n","1     robustness  lowercase   \n","2     robustness  lowercase   \n","3     robustness  lowercase   \n","4     robustness  lowercase   \n","...          ...        ...   \n","3995  robustness  uppercase   \n","3996  robustness  uppercase   \n","3997  robustness  uppercase   \n","3998  robustness  uppercase   \n","3999  robustness  uppercase   \n","\n","                                               original  \\\n","0          hide new secretions from the parental units    \n","1                  contains no wit , only labored gags    \n","2     that loves its characters and communicates som...   \n","3     remains utterly satisfied to remain the same t...   \n","4     on the worst revenge-of-the-nerds clichés the ...   \n","...                                                 ...   \n","3995              when there 's nothing else happening    \n","3996                                          on cable    \n","3997                                    it with ring ,    \n","3998                far from a groundbreaking endeavor    \n","3999                  that these women are spectacular    \n","\n","                                              test_case expected_result  \\\n","0          hide new secretions from the parental units         NEGATIVE   \n","1                  contains no wit , only labored gags         NEGATIVE   \n","2     that loves its characters and communicates som...        POSITIVE   \n","3     remains utterly satisfied to remain the same t...        NEGATIVE   \n","4     on the worst revenge-of-the-nerds clichés the ...        NEGATIVE   \n","...                                                 ...             ...   \n","3995              WHEN THERE 'S NOTHING ELSE HAPPENING         NEGATIVE   \n","3996                                          ON CABLE         NEGATIVE   \n","3997                                    IT WITH RING ,         POSITIVE   \n","3998                FAR FROM A GROUNDBREAKING ENDEAVOR         NEGATIVE   \n","3999                  THAT THESE WOMEN ARE SPECTACULAR         POSITIVE   \n","\n","     actual_result  pass  \n","0         NEGATIVE  True  \n","1         NEGATIVE  True  \n","2         POSITIVE  True  \n","3         NEGATIVE  True  \n","4         NEGATIVE  True  \n","...            ...   ...  \n","3995      NEGATIVE  True  \n","3996      NEGATIVE  True  \n","3997      POSITIVE  True  \n","3998      NEGATIVE  True  \n","3999      POSITIVE  True  \n","\n","[4000 rows x 7 columns]"]},"execution_count":15,"metadata":{},"output_type":"execute_result"}],"source":["harness.generated_results()"]},{"cell_type":"markdown","metadata":{"id":"TKB8Rsr2GZME"},"source":["This method returns the generated results in the form of a pandas dataframe, which provides a convenient and easy-to-use format for working with the test results. You can use this method to quickly identify the test cases that failed and to determine where fixes are needed."]},{"cell_type":"markdown","metadata":{"id":"PBSlpWnUU55G"},"source":["### Final Results"]},{"cell_type":"markdown","metadata":{"id":"umnEgUHM8DRA"},"source":["We can call `.report()` which summarizes the results giving information about pass and fail counts and overall test pass/fail flag."]},{"cell_type":"code","execution_count":null,"metadata":{"colab":{"base_uri":"https://localhost:8080/","height":112},"id":"gp57HcF9yxi7","outputId":"b893072f-102a-45a6-be03-d737996e659c"},"outputs":[{"data":{"text/html":["<div>\n","<style scoped>\n","    .dataframe tbody tr th:only-of-type {\n","        vertical-align: middle;\n","    }\n","\n","    .dataframe tbody tr th {\n","        vertical-align: top;\n","    }\n","\n","    .dataframe thead th {\n","        text-align: right;\n","    }\n","</style>\n","<table border=\"1\" class=\"dataframe\">\n","  <thead>\n","    <tr style=\"text-align: right;\">\n","      <th></th>\n","      <th>category</th>\n","      <th>test_type</th>\n","      <th>fail_count</th>\n","      <th>pass_count</th>\n","      <th>pass_rate</th>\n","      <th>minimum_pass_rate</th>\n","      <th>pass</th>\n","    </tr>\n","  </thead>\n","  <tbody>\n","    <tr>\n","      <th>0</th>\n","      <td>robustness</td>\n","      <td>lowercase</td>\n","      <td>0</td>\n","      <td>2000</td>\n","      <td>100%</td>\n","      <td>66%</td>\n","      <td>True</td>\n","    </tr>\n","    <tr>\n","      <th>1</th>\n","      <td>robustness</td>\n","      <td>uppercase</td>\n","      <td>0</td>\n","      <td>2000</td>\n","      <td>100%</td>\n","      <td>66%</td>\n","      <td>True</td>\n","    </tr>\n","  </tbody>\n","</table>\n","</div>"],"text/plain":["     category  test_type  fail_count  pass_count pass_rate minimum_pass_rate  \\\n","0  robustness  lowercase           0        2000      100%               66%   \n","1  robustness  uppercase           0        2000      100%               66%   \n","\n","   pass  \n","0  True  \n","1  True  "]},"execution_count":16,"metadata":{},"output_type":"execute_result"}],"source":["harness.report()"]},{"cell_type":"markdown","metadata":{"id":"5N0cKfKiLsiQ"},"source":["## `Imdb` Dataset Testing\n","-------------------\n"]},{"cell_type":"markdown","metadata":{"id":"l5H75bwe8DRA"},"source":["We can also use another dataset to test"]},{"cell_type":"markdown","metadata":{"id":"Ny0585_H8DRA"},"source":["### Harness and Its Parameters"]},{"cell_type":"code","execution_count":null,"metadata":{"id":"oDh3Zaa9EDfZ"},"outputs":[],"source":["harness = Harness(task=\"text-classification\", hub=\"huggingface\",\n","                    model=\"lvwerra/distilbert-imdb\",\n","                    data={\"name\":'imdb'})"]},{"cell_type":"markdown","metadata":{"id":"LIK5jh0x8DRB"},"source":["We have specified task as `text-classification` , hub as `huggingface` and model as `lvwerra/distilbert-imdb`\n","\n","For dataset we used `imdb`. With default parameters for feature_column, target_column and split\n","\n","You can find more HuggingFace Benchmark Datasets [here](https://huggingface.co/datasets?task_categories=task_categories:text-classification&sort=downloads)"]},{"cell_type":"markdown","metadata":{"id":"uTbZ_qJV8DRB"},"source":["### Setup and Configure Harness"]},{"cell_type":"code","execution_count":null,"metadata":{"colab":{"base_uri":"https://localhost:8080/"},"id":"ZnLWJkPVEDmg","outputId":"92ca0633-a1c6-4de3-f9fd-c77e6bcb5374"},"outputs":[{"data":{"text/plain":["{'tests': {'defaults': {'min_pass_rate': 0.65},\n","  'robustness': {'lowercase': {'min_pass_rate': 0.66},\n","   'uppercase': {'min_pass_rate': 0.66}}}}"]},"execution_count":28,"metadata":{},"output_type":"execute_result"}],"source":["harness.configure(\n","{\n"," 'tests': {'defaults': {'min_pass_rate': 0.65},\n","           'robustness': {'lowercase': {'min_pass_rate': 0.66},\n","                          'uppercase': {'min_pass_rate': 0.66},\n","                        }\n","          }\n"," }\n"," )"]},{"cell_type":"code","execution_count":null,"metadata":{"id":"VdLgXi968DRB"},"outputs":[],"source":["# Limit the data to the first 2000 samples\n","harness.data = harness.data[:2000]"]},{"cell_type":"code","execution_count":null,"metadata":{"colab":{"base_uri":"https://localhost:8080/"},"id":"A3U0kM62EG6B","outputId":"1ad54c30-3371-41b6-e85c-4dc69ffcd8aa"},"outputs":[{"name":"stderr","output_type":"stream","text":["Generating testcases...: 100%|██████████| 1/1 [00:00<?, ?it/s]\n"]},{"data":{"text/plain":[]},"execution_count":31,"metadata":{},"output_type":"execute_result"}],"source":["harness.generate()"]},{"cell_type":"markdown","metadata":{"id":"JR7kIfZY8DRJ"},"source":["### Generating the test cases."]},{"cell_type":"code","execution_count":null,"metadata":{"colab":{"base_uri":"https://localhost:8080/","height":423},"id":"0ipwMsy0EHYY","outputId":"a3e9c1fb-22dc-416d-8cf6-043ac4b28432"},"outputs":[{"data":{"text/html":["<div>\n","<style scoped>\n","    .dataframe tbody tr th:only-of-type {\n","        vertical-align: middle;\n","    }\n","\n","    .dataframe tbody tr th {\n","        vertical-align: top;\n","    }\n","\n","    .dataframe thead th {\n","        text-align: right;\n","    }\n","</style>\n","<table border=\"1\" class=\"dataframe\">\n","  <thead>\n","    <tr style=\"text-align: right;\">\n","      <th></th>\n","      <th>category</th>\n","      <th>test_type</th>\n","      <th>original</th>\n","      <th>test_case</th>\n","      <th>expected_result</th>\n","    </tr>\n","  </thead>\n","  <tbody>\n","    <tr>\n","      <th>0</th>\n","      <td>robustness</td>\n","      <td>lowercase</td>\n","      <td>I love sci-fi and am willing to put up with a ...</td>\n","      <td>i love sci-fi and am willing to put up with a ...</td>\n","      <td>NEGATIVE</td>\n","    </tr>\n","    <tr>\n","      <th>1</th>\n","      <td>robustness</td>\n","      <td>lowercase</td>\n","      <td>Worth the entertainment value of a rental, esp...</td>\n","      <td>worth the entertainment value of a rental, esp...</td>\n","      <td>NEGATIVE</td>\n","    </tr>\n","    <tr>\n","      <th>2</th>\n","      <td>robustness</td>\n","      <td>lowercase</td>\n","      <td>its a totally average film with a few semi-alr...</td>\n","      <td>its a totally average film with a few semi-alr...</td>\n","      <td>NEGATIVE</td>\n","    </tr>\n","    <tr>\n","      <th>3</th>\n","      <td>robustness</td>\n","      <td>lowercase</td>\n","      <td>STAR RATING: ***** Saturday Night **** Friday ...</td>\n","      <td>star rating: ***** saturday night **** friday ...</td>\n","      <td>NEGATIVE</td>\n","    </tr>\n","    <tr>\n","      <th>4</th>\n","      <td>robustness</td>\n","      <td>lowercase</td>\n","      <td>First off let me say, If you haven't enjoyed a...</td>\n","      <td>first off let me say, if you haven't enjoyed a...</td>\n","      <td>POSITIVE</td>\n","    </tr>\n","    <tr>\n","      <th>...</th>\n","      <td>...</td>\n","      <td>...</td>\n","      <td>...</td>\n","      <td>...</td>\n","      <td>...</td>\n","    </tr>\n","    <tr>\n","      <th>3995</th>\n","      <td>robustness</td>\n","      <td>uppercase</td>\n","      <td>A rather disappointing film. The club scenes w...</td>\n","      <td>A RATHER DISAPPOINTING FILM. THE CLUB SCENES W...</td>\n","      <td>NEGATIVE</td>\n","    </tr>\n","    <tr>\n","      <th>3996</th>\n","      <td>robustness</td>\n","      <td>uppercase</td>\n","      <td>There were so many reasons why this movie coul...</td>\n","      <td>THERE WERE SO MANY REASONS WHY THIS MOVIE COUL...</td>\n","      <td>NEGATIVE</td>\n","    </tr>\n","    <tr>\n","      <th>3997</th>\n","      <td>robustness</td>\n","      <td>uppercase</td>\n","      <td>After Kenneth Opel's rousing story of the invi...</td>\n","      <td>AFTER KENNETH OPEL'S ROUSING STORY OF THE INVI...</td>\n","      <td>NEGATIVE</td>\n","    </tr>\n","    <tr>\n","      <th>3998</th>\n","      <td>robustness</td>\n","      <td>uppercase</td>\n","      <td>Having already seen the original \"Jack Frost\",...</td>\n","      <td>HAVING ALREADY SEEN THE ORIGINAL \"JACK FROST\",...</td>\n","      <td>NEGATIVE</td>\n","    </tr>\n","    <tr>\n","      <th>3999</th>\n","      <td>robustness</td>\n","      <td>uppercase</td>\n","      <td>Ill-conceived sequel(..the absurd idea of havi...</td>\n","      <td>ILL-CONCEIVED SEQUEL(..THE ABSURD IDEA OF HAVI...</td>\n","      <td>NEGATIVE</td>\n","    </tr>\n","  </tbody>\n","</table>\n","<p>4000 rows × 5 columns</p>\n","</div>"],"text/plain":["        category  test_type  \\\n","0     robustness  lowercase   \n","1     robustness  lowercase   \n","2     robustness  lowercase   \n","3     robustness  lowercase   \n","4     robustness  lowercase   \n","...          ...        ...   \n","3995  robustness  uppercase   \n","3996  robustness  uppercase   \n","3997  robustness  uppercase   \n","3998  robustness  uppercase   \n","3999  robustness  uppercase   \n","\n","                                               original  \\\n","0     I love sci-fi and am willing to put up with a ...   \n","1     Worth the entertainment value of a rental, esp...   \n","2     its a totally average film with a few semi-alr...   \n","3     STAR RATING: ***** Saturday Night **** Friday ...   \n","4     First off let me say, If you haven't enjoyed a...   \n","...                                                 ...   \n","3995  A rather disappointing film. The club scenes w...   \n","3996  There were so many reasons why this movie coul...   \n","3997  After Kenneth Opel's rousing story of the invi...   \n","3998  Having already seen the original \"Jack Frost\",...   \n","3999  Ill-conceived sequel(..the absurd idea of havi...   \n","\n","                                              test_case expected_result  \n","0     i love sci-fi and am willing to put up with a ...        NEGATIVE  \n","1     worth the entertainment value of a rental, esp...        NEGATIVE  \n","2     its a totally average film with a few semi-alr...        NEGATIVE  \n","3     star rating: ***** saturday night **** friday ...        NEGATIVE  \n","4     first off let me say, if you haven't enjoyed a...        POSITIVE  \n","...                                                 ...             ...  \n","3995  A RATHER DISAPPOINTING FILM. THE CLUB SCENES W...        NEGATIVE  \n","3996  THERE WERE SO MANY REASONS WHY THIS MOVIE COUL...        NEGATIVE  \n","3997  AFTER KENNETH OPEL'S ROUSING STORY OF THE INVI...        NEGATIVE  \n","3998  HAVING ALREADY SEEN THE ORIGINAL \"JACK FROST\",...        NEGATIVE  \n","3999  ILL-CONCEIVED SEQUEL(..THE ABSURD IDEA OF HAVI...        NEGATIVE  \n","\n","[4000 rows x 5 columns]"]},"execution_count":32,"metadata":{},"output_type":"execute_result"}],"source":["harness.testcases()"]},{"cell_type":"markdown","metadata":{"id":"1WtdwEZL8DRJ"},"source":["### Running the tests"]},{"cell_type":"code","execution_count":null,"metadata":{"colab":{"base_uri":"https://localhost:8080/"},"id":"0Nic5HRZEJu5","outputId":"dbbf911a-413e-479c-996b-98430920f0b5"},"outputs":[{"name":"stderr","output_type":"stream","text":["Running testcases... : 100%|██████████| 4000/4000 [43:06<00:00,  1.55it/s]\n"]},{"data":{"text/plain":[]},"execution_count":33,"metadata":{},"output_type":"execute_result"}],"source":["harness.run()"]},{"cell_type":"code","execution_count":null,"metadata":{"id":"BjZc-ZcCELbU","outputId":"5913de81-5f5d-4978-a1dc-f6cc1f0f2e7d"},"outputs":[{"data":{"text/html":["<div>\n","<style scoped>\n","    .dataframe tbody tr th:only-of-type {\n","        vertical-align: middle;\n","    }\n","\n","    .dataframe tbody tr th {\n","        vertical-align: top;\n","    }\n","\n","    .dataframe thead th {\n","        text-align: right;\n","    }\n","</style>\n","<table border=\"1\" class=\"dataframe\">\n","  <thead>\n","    <tr style=\"text-align: right;\">\n","      <th></th>\n","      <th>category</th>\n","      <th>test_type</th>\n","      <th>original</th>\n","      <th>test_case</th>\n","      <th>expected_result</th>\n","      <th>actual_result</th>\n","      <th>pass</th>\n","    </tr>\n","  </thead>\n","  <tbody>\n","    <tr>\n","      <th>0</th>\n","      <td>robustness</td>\n","      <td>lowercase</td>\n","      <td>I love sci-fi and am willing to put up with a ...</td>\n","      <td>i love sci-fi and am willing to put up with a ...</td>\n","      <td>NEGATIVE</td>\n","      <td>NEGATIVE</td>\n","      <td>True</td>\n","    </tr>\n","    <tr>\n","      <th>1</th>\n","      <td>robustness</td>\n","      <td>lowercase</td>\n","      <td>Worth the entertainment value of a rental, esp...</td>\n","      <td>worth the entertainment value of a rental, esp...</td>\n","      <td>NEGATIVE</td>\n","      <td>NEGATIVE</td>\n","      <td>True</td>\n","    </tr>\n","    <tr>\n","      <th>2</th>\n","      <td>robustness</td>\n","      <td>lowercase</td>\n","      <td>its a totally average film with a few semi-alr...</td>\n","      <td>its a totally average film with a few semi-alr...</td>\n","      <td>NEGATIVE</td>\n","      <td>NEGATIVE</td>\n","      <td>True</td>\n","    </tr>\n","    <tr>\n","      <th>3</th>\n","      <td>robustness</td>\n","      <td>lowercase</td>\n","      <td>STAR RATING: ***** Saturday Night **** Friday ...</td>\n","      <td>star rating: ***** saturday night **** friday ...</td>\n","      <td>NEGATIVE</td>\n","      <td>NEGATIVE</td>\n","      <td>True</td>\n","    </tr>\n","    <tr>\n","      <th>4</th>\n","      <td>robustness</td>\n","      <td>lowercase</td>\n","      <td>First off let me say, If you haven't enjoyed a...</td>\n","      <td>first off let me say, if you haven't enjoyed a...</td>\n","      <td>POSITIVE</td>\n","      <td>POSITIVE</td>\n","      <td>True</td>\n","    </tr>\n","    <tr>\n","      <th>...</th>\n","      <td>...</td>\n","      <td>...</td>\n","      <td>...</td>\n","      <td>...</td>\n","      <td>...</td>\n","      <td>...</td>\n","      <td>...</td>\n","    </tr>\n","    <tr>\n","      <th>3995</th>\n","      <td>robustness</td>\n","      <td>uppercase</td>\n","      <td>A rather disappointing film. The club scenes w...</td>\n","      <td>A RATHER DISAPPOINTING FILM. THE CLUB SCENES W...</td>\n","      <td>NEGATIVE</td>\n","      <td>NEGATIVE</td>\n","      <td>True</td>\n","    </tr>\n","    <tr>\n","      <th>3996</th>\n","      <td>robustness</td>\n","      <td>uppercase</td>\n","      <td>There were so many reasons why this movie coul...</td>\n","      <td>THERE WERE SO MANY REASONS WHY THIS MOVIE COUL...</td>\n","      <td>NEGATIVE</td>\n","      <td>NEGATIVE</td>\n","      <td>True</td>\n","    </tr>\n","    <tr>\n","      <th>3997</th>\n","      <td>robustness</td>\n","      <td>uppercase</td>\n","      <td>After Kenneth Opel's rousing story of the invi...</td>\n","      <td>AFTER KENNETH OPEL'S ROUSING STORY OF THE INVI...</td>\n","      <td>NEGATIVE</td>\n","      <td>NEGATIVE</td>\n","      <td>True</td>\n","    </tr>\n","    <tr>\n","      <th>3998</th>\n","      <td>robustness</td>\n","      <td>uppercase</td>\n","      <td>Having already seen the original \"Jack Frost\",...</td>\n","      <td>HAVING ALREADY SEEN THE ORIGINAL \"JACK FROST\",...</td>\n","      <td>NEGATIVE</td>\n","      <td>NEGATIVE</td>\n","      <td>True</td>\n","    </tr>\n","    <tr>\n","      <th>3999</th>\n","      <td>robustness</td>\n","      <td>uppercase</td>\n","      <td>Ill-conceived sequel(..the absurd idea of havi...</td>\n","      <td>ILL-CONCEIVED SEQUEL(..THE ABSURD IDEA OF HAVI...</td>\n","      <td>NEGATIVE</td>\n","      <td>NEGATIVE</td>\n","      <td>True</td>\n","    </tr>\n","  </tbody>\n","</table>\n","<p>4000 rows × 7 columns</p>\n","</div>"],"text/plain":["        category  test_type  \\\n","0     robustness  lowercase   \n","1     robustness  lowercase   \n","2     robustness  lowercase   \n","3     robustness  lowercase   \n","4     robustness  lowercase   \n","...          ...        ...   \n","3995  robustness  uppercase   \n","3996  robustness  uppercase   \n","3997  robustness  uppercase   \n","3998  robustness  uppercase   \n","3999  robustness  uppercase   \n","\n","                                               original  \\\n","0     I love sci-fi and am willing to put up with a ...   \n","1     Worth the entertainment value of a rental, esp...   \n","2     its a totally average film with a few semi-alr...   \n","3     STAR RATING: ***** Saturday Night **** Friday ...   \n","4     First off let me say, If you haven't enjoyed a...   \n","...                                                 ...   \n","3995  A rather disappointing film. The club scenes w...   \n","3996  There were so many reasons why this movie coul...   \n","3997  After Kenneth Opel's rousing story of the invi...   \n","3998  Having already seen the original \"Jack Frost\",...   \n","3999  Ill-conceived sequel(..the absurd idea of havi...   \n","\n","                                              test_case expected_result  \\\n","0     i love sci-fi and am willing to put up with a ...        NEGATIVE   \n","1     worth the entertainment value of a rental, esp...        NEGATIVE   \n","2     its a totally average film with a few semi-alr...        NEGATIVE   \n","3     star rating: ***** saturday night **** friday ...        NEGATIVE   \n","4     first off let me say, if you haven't enjoyed a...        POSITIVE   \n","...                                                 ...             ...   \n","3995  A RATHER DISAPPOINTING FILM. THE CLUB SCENES W...        NEGATIVE   \n","3996  THERE WERE SO MANY REASONS WHY THIS MOVIE COUL...        NEGATIVE   \n","3997  AFTER KENNETH OPEL'S ROUSING STORY OF THE INVI...        NEGATIVE   \n","3998  HAVING ALREADY SEEN THE ORIGINAL \"JACK FROST\",...        NEGATIVE   \n","3999  ILL-CONCEIVED SEQUEL(..THE ABSURD IDEA OF HAVI...        NEGATIVE   \n","\n","     actual_result  pass  \n","0         NEGATIVE  True  \n","1         NEGATIVE  True  \n","2         NEGATIVE  True  \n","3         NEGATIVE  True  \n","4         POSITIVE  True  \n","...            ...   ...  \n","3995      NEGATIVE  True  \n","3996      NEGATIVE  True  \n","3997      NEGATIVE  True  \n","3998      NEGATIVE  True  \n","3999      NEGATIVE  True  \n","\n","[4000 rows x 7 columns]"]},"execution_count":34,"metadata":{},"output_type":"execute_result"}],"source":["harness.generated_results()"]},{"cell_type":"markdown","metadata":{"id":"aQw2X-IG8DRK"},"source":["### Final Report\n","\n","We can call `.report()` which summarizes the results giving information about pass and fail counts and overall test pass/fail flag."]},{"cell_type":"code","execution_count":null,"metadata":{"colab":{"base_uri":"https://localhost:8080/","height":112},"id":"PlrAxK1eENmh","outputId":"7fd59473-20ac-402b-a39b-e5e3e29cf1f4"},"outputs":[{"data":{"text/html":["<div>\n","<style scoped>\n","    .dataframe tbody tr th:only-of-type {\n","        vertical-align: middle;\n","    }\n","\n","    .dataframe tbody tr th {\n","        vertical-align: top;\n","    }\n","\n","    .dataframe thead th {\n","        text-align: right;\n","    }\n","</style>\n","<table border=\"1\" class=\"dataframe\">\n","  <thead>\n","    <tr style=\"text-align: right;\">\n","      <th></th>\n","      <th>category</th>\n","      <th>test_type</th>\n","      <th>fail_count</th>\n","      <th>pass_count</th>\n","      <th>pass_rate</th>\n","      <th>minimum_pass_rate</th>\n","      <th>pass</th>\n","    </tr>\n","  </thead>\n","  <tbody>\n","    <tr>\n","      <th>0</th>\n","      <td>robustness</td>\n","      <td>lowercase</td>\n","      <td>0</td>\n","      <td>2000</td>\n","      <td>100%</td>\n","      <td>66%</td>\n","      <td>True</td>\n","    </tr>\n","    <tr>\n","      <th>1</th>\n","      <td>robustness</td>\n","      <td>uppercase</td>\n","      <td>0</td>\n","      <td>2000</td>\n","      <td>100%</td>\n","      <td>66%</td>\n","      <td>True</td>\n","    </tr>\n","  </tbody>\n","</table>\n","</div>"],"text/plain":["     category  test_type  fail_count  pass_count pass_rate minimum_pass_rate  \\\n","0  robustness  lowercase           0        2000      100%               66%   \n","1  robustness  uppercase           0        2000      100%               66%   \n","\n","   pass  \n","0  True  \n","1  True  "]},"execution_count":35,"metadata":{},"output_type":"execute_result"}],"source":["harness.report()"]},{"cell_type":"markdown","metadata":{"id":"emrRp2vlF1T1"},"source":["# HuggingFace Datasets Testing For `summarization`\n","\n","In this section, we dive into testing of HuggingFace Models for different HuggingFace Datasets."]},{"cell_type":"markdown","metadata":{"id":"EsCtdb6cF9IN"},"source":["## samsum - Dataset Testing\n","-------------------"]},{"cell_type":"markdown","metadata":{},"source":["### Installing required dependencies"]},{"cell_type":"code","execution_count":null,"metadata":{},"outputs":[],"source":["!pip install \"langtest[evaluate,langchain,openai]\" transformers==4.28.1"]},{"cell_type":"markdown","metadata":{"id":"vzC8J9SxFnqP"},"source":["### Set environment for OpenAI"]},{"cell_type":"code","execution_count":32,"metadata":{"executionInfo":{"elapsed":6,"status":"ok","timestamp":1689533812752,"user":{"displayName":"Prikshit Sharma","userId":"02603035510215936699"},"user_tz":-330},"id":"rNeyF-FC_j82"},"outputs":[],"source":["import os\n","\n","import openai\n","\n","os.environ[\"OPENAI_API_KEY\"] = \"<YOUR_API_KEY>\""]},{"cell_type":"markdown","metadata":{"id":"B01bfV9pFhg-"},"source":["### Setup and configure harness"]},{"cell_type":"code","execution_count":33,"metadata":{"colab":{"base_uri":"https://localhost:8080/"},"executionInfo":{"elapsed":559,"status":"ok","timestamp":1689533813306,"user":{"displayName":"Prikshit Sharma","userId":"02603035510215936699"},"user_tz":-330},"id":"8WMCEFwT_d1P","outputId":"98954e80-b979-4f14-91b1-e3fd6863de4c"},"outputs":[{"name":"stdout","output_type":"stream","text":["Test Configuration : \n"," {\n"," \"model_parameters\": {\n","  \"temperature\": 0.2,\n","  \"max_tokens\": 64\n"," },\n"," \"tests\": {\n","  \"defaults\": {\n","   \"min_pass_rate\": 1.0\n","  },\n","  \"robustness\": {\n","   \"add_typo\": {\n","    \"min_pass_rate\": 0.7\n","   },\n","   \"lowercase\": {\n","    \"min_pass_rate\": 0.7\n","   }\n","  }\n"," }\n","}\n"]}],"source":["harness = Harness(task=\"summarization\", hub=\"openai\",\n","                   model=\"text-davinci-003\",\n","                  data={\"name\":'samsum',\n","                  \"feature_column\":\"dialogue\",\n","                  \"target_column\":'summary',\n","                  \"split\":\"test\"\n","                  })"]},{"cell_type":"markdown","metadata":{"id":"qRtXmy4GFXwP"},"source":["### Configure the Tests\n","We can use the .configure() method to manually configure the tests we want to perform."]},{"cell_type":"code","execution_count":34,"metadata":{"colab":{"base_uri":"https://localhost:8080/"},"executionInfo":{"elapsed":2,"status":"ok","timestamp":1689533813901,"user":{"displayName":"Prikshit Sharma","userId":"02603035510215936699"},"user_tz":-330},"id":"dhmCGPALAPJv","outputId":"a164dfea-6b0d-4080-f548-156a49867907"},"outputs":[{"data":{"text/plain":["{'tests': {'defaults': {'min_pass_rate': 0.65},\n","  'robustness': {'uppercase': {'min_pass_rate': 0.66},\n","   'lowercase': {'min_pass_rate': 0.6}}}}"]},"execution_count":34,"metadata":{},"output_type":"execute_result"}],"source":["harness.configure(\n","{\n"," 'tests': {'defaults': {'min_pass_rate': 0.65},\n","           'robustness': {'uppercase': {'min_pass_rate': 0.66},\n","                          'lowercase':{'min_pass_rate': 0.60},\n","                        }\n","          }\n"," }\n"," )"]},{"cell_type":"markdown","metadata":{"id":"qoAgrQQbGMC2"},"source":["Here we have configured the harness to perform two robustness tests (uppercase and lowercase) and defined the minimum pass rate for each test."]},{"cell_type":"code","execution_count":35,"metadata":{"executionInfo":{"elapsed":2,"status":"ok","timestamp":1689533817937,"user":{"displayName":"Prikshit Sharma","userId":"02603035510215936699"},"user_tz":-330},"id":"PYQGX2OdBDd1"},"outputs":[],"source":["harness.data=harness.data[0:5]"]},{"cell_type":"markdown","metadata":{"id":"jtHbCs1kFNYn"},"source":["### Generating the test cases."]},{"cell_type":"code","execution_count":36,"metadata":{"colab":{"base_uri":"https://localhost:8080/"},"executionInfo":{"elapsed":4,"status":"ok","timestamp":1689533818349,"user":{"displayName":"Prikshit Sharma","userId":"02603035510215936699"},"user_tz":-330},"id":"LxqMY_FjA_Pp","outputId":"3b5c6e44-9174-4143-98b9-00ba4f823de7"},"outputs":[{"name":"stderr","output_type":"stream","text":["\n","Generating testcases...: 100%|██████████| 1/1 [00:00<00:00, 6269.51it/s]\n"]},{"data":{"text/plain":[]},"execution_count":36,"metadata":{},"output_type":"execute_result"}],"source":["harness.generate()"]},{"cell_type":"markdown","metadata":{"id":"N-jdxDdBFKgl"},"source":["harness.generate() method automatically generates the test cases (based on the provided configuration)"]},{"cell_type":"code","execution_count":37,"metadata":{"colab":{"base_uri":"https://localhost:8080/","height":363},"executionInfo":{"elapsed":3,"status":"ok","timestamp":1689533819321,"user":{"displayName":"Prikshit Sharma","userId":"02603035510215936699"},"user_tz":-330},"id":"Yo5Q6VfVBASF","outputId":"a5af73e9-616e-49e9-f0d3-6846c10185ab"},"outputs":[{"data":{"text/html":["\n","\n","  <div id=\"df-d32e911c-ee6d-4c2e-af4b-1520ea542d58\">\n","    <div class=\"colab-df-container\">\n","      <div>\n","<style scoped>\n","    .dataframe tbody tr th:only-of-type {\n","        vertical-align: middle;\n","    }\n","\n","    .dataframe tbody tr th {\n","        vertical-align: top;\n","    }\n","\n","    .dataframe thead th {\n","        text-align: right;\n","    }\n","</style>\n","<table border=\"1\" class=\"dataframe\">\n","  <thead>\n","    <tr style=\"text-align: right;\">\n","      <th></th>\n","      <th>category</th>\n","      <th>test_type</th>\n","      <th>original</th>\n","      <th>test_case</th>\n","    </tr>\n","  </thead>\n","  <tbody>\n","    <tr>\n","      <th>0</th>\n","      <td>robustness</td>\n","      <td>uppercase</td>\n","      <td>Hannah: Hey, do you have Betty's number?\\nAman...</td>\n","      <td>HANNAH: HEY, DO YOU HAVE BETTY'S NUMBER? AMAND...</td>\n","    </tr>\n","    <tr>\n","      <th>1</th>\n","      <td>robustness</td>\n","      <td>uppercase</td>\n","      <td>Eric: MACHINE!\\r\\nRob: That's so gr8!\\r\\nEric:...</td>\n","      <td>ERIC: MACHINE! ROB: THAT'S SO GR8! ERIC: I KNO...</td>\n","    </tr>\n","    <tr>\n","      <th>2</th>\n","      <td>robustness</td>\n","      <td>uppercase</td>\n","      <td>Lenny: Babe, can you help me with something?\\r...</td>\n","      <td>LENNY: BABE, CAN YOU HELP ME WITH SOMETHING? B...</td>\n","    </tr>\n","    <tr>\n","      <th>3</th>\n","      <td>robustness</td>\n","      <td>uppercase</td>\n","      <td>Will: hey babe, what do you want for dinner to...</td>\n","      <td>WILL: HEY BABE, WHAT DO YOU WANT FOR DINNER TO...</td>\n","    </tr>\n","    <tr>\n","      <th>4</th>\n","      <td>robustness</td>\n","      <td>uppercase</td>\n","      <td>Ollie: Hi , are you in Warsaw\\r\\nJane: yes, ju...</td>\n","      <td>OLLIE: HI , ARE YOU IN WARSAW JANE: YES, JUST ...</td>\n","    </tr>\n","    <tr>\n","      <th>5</th>\n","      <td>robustness</td>\n","      <td>lowercase</td>\n","      <td>Hannah: Hey, do you have Betty's number?\\nAman...</td>\n","      <td>hannah: hey, do you have betty's number? amand...</td>\n","    </tr>\n","    <tr>\n","      <th>6</th>\n","      <td>robustness</td>\n","      <td>lowercase</td>\n","      <td>Eric: MACHINE!\\r\\nRob: That's so gr8!\\r\\nEric:...</td>\n","      <td>eric: machine! rob: that's so gr8! eric: i kno...</td>\n","    </tr>\n","    <tr>\n","      <th>7</th>\n","      <td>robustness</td>\n","      <td>lowercase</td>\n","      <td>Lenny: Babe, can you help me with something?\\r...</td>\n","      <td>lenny: babe, can you help me with something? b...</td>\n","    </tr>\n","    <tr>\n","      <th>8</th>\n","      <td>robustness</td>\n","      <td>lowercase</td>\n","      <td>Will: hey babe, what do you want for dinner to...</td>\n","      <td>will: hey babe, what do you want for dinner to...</td>\n","    </tr>\n","    <tr>\n","      <th>9</th>\n","      <td>robustness</td>\n","      <td>lowercase</td>\n","      <td>Ollie: Hi , are you in Warsaw\\r\\nJane: yes, ju...</td>\n","      <td>ollie: hi , are you in warsaw jane: yes, just ...</td>\n","    </tr>\n","  </tbody>\n","</table>\n","</div>\n","      <button class=\"colab-df-convert\" onclick=\"convertToInteractive('df-d32e911c-ee6d-4c2e-af4b-1520ea542d58')\"\n","              title=\"Convert this dataframe to an interactive table.\"\n","              style=\"display:none;\">\n","\n","  <svg xmlns=\"http://www.w3.org/2000/svg\" height=\"24px\"viewBox=\"0 0 24 24\"\n","       width=\"24px\">\n","    <path d=\"M0 0h24v24H0V0z\" fill=\"none\"/>\n","    <path d=\"M18.56 5.44l.94 2.06.94-2.06 2.06-.94-2.06-.94-.94-2.06-.94 2.06-2.06.94zm-11 1L8.5 8.5l.94-2.06 2.06-.94-2.06-.94L8.5 2.5l-.94 2.06-2.06.94zm10 10l.94 2.06.94-2.06 2.06-.94-2.06-.94-.94-2.06-.94 2.06-2.06.94z\"/><path d=\"M17.41 7.96l-1.37-1.37c-.4-.4-.92-.59-1.43-.59-.52 0-1.04.2-1.43.59L10.3 9.45l-7.72 7.72c-.78.78-.78 2.05 0 2.83L4 21.41c.39.39.9.59 1.41.59.51 0 1.02-.2 1.41-.59l7.78-7.78 2.81-2.81c.8-.78.8-2.07 0-2.86zM5.41 20L4 18.59l7.72-7.72 1.47 1.35L5.41 20z\"/>\n","  </svg>\n","      </button>\n","\n","\n","\n","    <div id=\"df-af14a5b7-83c5-4e4f-849b-01bc45cd0167\">\n","      <button class=\"colab-df-quickchart\" onclick=\"quickchart('df-af14a5b7-83c5-4e4f-849b-01bc45cd0167')\"\n","              title=\"Suggest charts.\"\n","              style=\"display:none;\">\n","\n","<svg xmlns=\"http://www.w3.org/2000/svg\" height=\"24px\"viewBox=\"0 0 24 24\"\n","     width=\"24px\">\n","    <g>\n","        <path d=\"M19 3H5c-1.1 0-2 .9-2 2v14c0 1.1.9 2 2 2h14c1.1 0 2-.9 2-2V5c0-1.1-.9-2-2-2zM9 17H7v-7h2v7zm4 0h-2V7h2v10zm4 0h-2v-4h2v4z\"/>\n","    </g>\n","</svg>\n","      </button>\n","    </div>\n","\n","<style>\n","  .colab-df-quickchart {\n","    background-color: #E8F0FE;\n","    border: none;\n","    border-radius: 50%;\n","    cursor: pointer;\n","    display: none;\n","    fill: #1967D2;\n","    height: 32px;\n","    padding: 0 0 0 0;\n","    width: 32px;\n","  }\n","\n","  .colab-df-quickchart:hover {\n","    background-color: #E2EBFA;\n","    box-shadow: 0px 1px 2px rgba(60, 64, 67, 0.3), 0px 1px 3px 1px rgba(60, 64, 67, 0.15);\n","    fill: #174EA6;\n","  }\n","\n","  [theme=dark] .colab-df-quickchart {\n","    background-color: #3B4455;\n","    fill: #D2E3FC;\n","  }\n","\n","  [theme=dark] .colab-df-quickchart:hover {\n","    background-color: #434B5C;\n","    box-shadow: 0px 1px 3px 1px rgba(0, 0, 0, 0.15);\n","    filter: drop-shadow(0px 1px 2px rgba(0, 0, 0, 0.3));\n","    fill: #FFFFFF;\n","  }\n","</style>\n","\n","    <script>\n","      async function quickchart(key) {\n","        const containerElement = document.querySelector('#' + key);\n","        const charts = await google.colab.kernel.invokeFunction(\n","            'suggestCharts', [key], {});\n","      }\n","    </script>\n","\n","      <script>\n","\n","function displayQuickchartButton(domScope) {\n","  let quickchartButtonEl =\n","    domScope.querySelector('#df-af14a5b7-83c5-4e4f-849b-01bc45cd0167 button.colab-df-quickchart');\n","  quickchartButtonEl.style.display =\n","    google.colab.kernel.accessAllowed ? 'block' : 'none';\n","}\n","\n","        displayQuickchartButton(document);\n","      </script>\n","      <style>\n","    .colab-df-container {\n","      display:flex;\n","      flex-wrap:wrap;\n","      gap: 12px;\n","    }\n","\n","    .colab-df-convert {\n","      background-color: #E8F0FE;\n","      border: none;\n","      border-radius: 50%;\n","      cursor: pointer;\n","      display: none;\n","      fill: #1967D2;\n","      height: 32px;\n","      padding: 0 0 0 0;\n","      width: 32px;\n","    }\n","\n","    .colab-df-convert:hover {\n","      background-color: #E2EBFA;\n","      box-shadow: 0px 1px 2px rgba(60, 64, 67, 0.3), 0px 1px 3px 1px rgba(60, 64, 67, 0.15);\n","      fill: #174EA6;\n","    }\n","\n","    [theme=dark] .colab-df-convert {\n","      background-color: #3B4455;\n","      fill: #D2E3FC;\n","    }\n","\n","    [theme=dark] .colab-df-convert:hover {\n","      background-color: #434B5C;\n","      box-shadow: 0px 1px 3px 1px rgba(0, 0, 0, 0.15);\n","      filter: drop-shadow(0px 1px 2px rgba(0, 0, 0, 0.3));\n","      fill: #FFFFFF;\n","    }\n","  </style>\n","\n","      <script>\n","        const buttonEl =\n","          document.querySelector('#df-d32e911c-ee6d-4c2e-af4b-1520ea542d58 button.colab-df-convert');\n","        buttonEl.style.display =\n","          google.colab.kernel.accessAllowed ? 'block' : 'none';\n","\n","        async function convertToInteractive(key) {\n","          const element = document.querySelector('#df-d32e911c-ee6d-4c2e-af4b-1520ea542d58');\n","          const dataTable =\n","            await google.colab.kernel.invokeFunction('convertToInteractive',\n","                                                     [key], {});\n","          if (!dataTable) return;\n","\n","          const docLinkHtml = 'Like what you see? Visit the ' +\n","            '<a target=\"_blank\" href=https://colab.research.google.com/notebooks/data_table.ipynb>data table notebook</a>'\n","            + ' to learn more about interactive tables.';\n","          element.innerHTML = '';\n","          dataTable['output_type'] = 'display_data';\n","          await google.colab.output.renderOutput(dataTable, element);\n","          const docLink = document.createElement('div');\n","          docLink.innerHTML = docLinkHtml;\n","          element.appendChild(docLink);\n","        }\n","      </script>\n","    </div>\n","  </div>\n"],"text/plain":["     category  test_type                                           original  \\\n","0  robustness  uppercase  Hannah: Hey, do you have Betty's number?\\nAman...   \n","1  robustness  uppercase  Eric: MACHINE!\\r\\nRob: That's so gr8!\\r\\nEric:...   \n","2  robustness  uppercase  Lenny: Babe, can you help me with something?\\r...   \n","3  robustness  uppercase  Will: hey babe, what do you want for dinner to...   \n","4  robustness  uppercase  Ollie: Hi , are you in Warsaw\\r\\nJane: yes, ju...   \n","5  robustness  lowercase  Hannah: Hey, do you have Betty's number?\\nAman...   \n","6  robustness  lowercase  Eric: MACHINE!\\r\\nRob: That's so gr8!\\r\\nEric:...   \n","7  robustness  lowercase  Lenny: Babe, can you help me with something?\\r...   \n","8  robustness  lowercase  Will: hey babe, what do you want for dinner to...   \n","9  robustness  lowercase  Ollie: Hi , are you in Warsaw\\r\\nJane: yes, ju...   \n","\n","                                           test_case  \n","0  HANNAH: HEY, DO YOU HAVE BETTY'S NUMBER? AMAND...  \n","1  ERIC: MACHINE! ROB: THAT'S SO GR8! ERIC: I KNO...  \n","2  LENNY: BABE, CAN YOU HELP ME WITH SOMETHING? B...  \n","3  WILL: HEY BABE, WHAT DO YOU WANT FOR DINNER TO...  \n","4  OLLIE: HI , ARE YOU IN WARSAW JANE: YES, JUST ...  \n","5  hannah: hey, do you have betty's number? amand...  \n","6  eric: machine! rob: that's so gr8! eric: i kno...  \n","7  lenny: babe, can you help me with something? b...  \n","8  will: hey babe, what do you want for dinner to...  \n","9  ollie: hi , are you in warsaw jane: yes, just ...  "]},"execution_count":37,"metadata":{},"output_type":"execute_result"}],"source":["harness.testcases()"]},{"cell_type":"markdown","metadata":{"id":"q-KYrFMWGSw1"},"source":["harness.testcases() method displays the produced test cases in form of a pandas data frame."]},{"cell_type":"markdown","metadata":{"id":"SkHPaAN7FBSG"},"source":["### Running the tests"]},{"cell_type":"code","execution_count":38,"metadata":{"colab":{"base_uri":"https://localhost:8080/"},"executionInfo":{"elapsed":367293,"status":"ok","timestamp":1689534188020,"user":{"displayName":"Prikshit Sharma","userId":"02603035510215936699"},"user_tz":-330},"id":"Pvqtr_G7BBR0","outputId":"dcf96ef6-c0f2-4e5d-958b-f4a896d5b42a"},"outputs":[{"name":"stderr","output_type":"stream","text":["Running testcases... : 100%|██████████| 10/10 [06:07<00:00, 36.71s/it]\n"]},{"data":{"text/plain":[]},"execution_count":38,"metadata":{},"output_type":"execute_result"}],"source":["harness.run()"]},{"cell_type":"markdown","metadata":{"id":"eQ_ufKmrGVqt"},"source":["Called after harness.generate() and is to used to run all the tests.  Returns a pass/fail flag for each test."]},{"cell_type":"markdown","metadata":{"id":"FpG0SFmnE7Nt"},"source":["### Generated Results"]},{"cell_type":"code","execution_count":42,"metadata":{"colab":{"base_uri":"https://localhost:8080/","height":581},"executionInfo":{"elapsed":4219,"status":"ok","timestamp":1689540121904,"user":{"displayName":"Prikshit Sharma","userId":"02603035510215936699"},"user_tz":-330},"id":"CFXsxZHJDKtj","outputId":"0e37cb99-f2ce-4dde-a237-867e0bca29dd"},"outputs":[{"data":{"text/html":["\n","\n","  <div id=\"df-e0648226-1f58-4c80-9617-d96af6ab7c50\">\n","    <div class=\"colab-df-container\">\n","      <div>\n","<style scoped>\n","    .dataframe tbody tr th:only-of-type {\n","        vertical-align: middle;\n","    }\n","\n","    .dataframe tbody tr th {\n","        vertical-align: top;\n","    }\n","\n","    .dataframe thead th {\n","        text-align: right;\n","    }\n","</style>\n","<table border=\"1\" class=\"dataframe\">\n","  <thead>\n","    <tr style=\"text-align: right;\">\n","      <th></th>\n","      <th>category</th>\n","      <th>test_type</th>\n","      <th>original</th>\n","      <th>test_case</th>\n","      <th>expected_result</th>\n","      <th>actual_result</th>\n","      <th>eval_score</th>\n","      <th>pass</th>\n","    </tr>\n","  </thead>\n","  <tbody>\n","    <tr>\n","      <th>0</th>\n","      <td>robustness</td>\n","      <td>uppercase</td>\n","      <td>Hannah: Hey, do you have Betty's number?\\nAman...</td>\n","      <td>HANNAH: HEY, DO YOU HAVE BETTY'S NUMBER? AMAND...</td>\n","      <td>Hannah is looking for Betty's phone number, b...</td>\n","      <td>Hannah is looking for Betty's number, but Ama...</td>\n","      <td>0.969697</td>\n","      <td>True</td>\n","    </tr>\n","    <tr>\n","      <th>1</th>\n","      <td>robustness</td>\n","      <td>uppercase</td>\n","      <td>Eric: MACHINE!\\r\\nRob: That's so gr8!\\r\\nEric:...</td>\n","      <td>ERIC: MACHINE! ROB: THAT'S SO GR8! ERIC: I KNO...</td>\n","      <td>Eric and Rob are discussing a stand-up comedy...</td>\n","      <td>Eric and Rob are discussing a stand-up comedy...</td>\n","      <td>0.413793</td>\n","      <td>False</td>\n","    </tr>\n","    <tr>\n","      <th>2</th>\n","      <td>robustness</td>\n","      <td>uppercase</td>\n","      <td>Lenny: Babe, can you help me with something?\\r...</td>\n","      <td>LENNY: BABE, CAN YOU HELP ME WITH SOMETHING? B...</td>\n","      <td>Lenny was unsure which trousers to buy and as...</td>\n","      <td>Lenny is trying to decide which pair of trous...</td>\n","      <td>0.152381</td>\n","      <td>False</td>\n","    </tr>\n","    <tr>\n","      <th>3</th>\n","      <td>robustness</td>\n","      <td>uppercase</td>\n","      <td>Will: hey babe, what do you want for dinner to...</td>\n","      <td>WILL: HEY BABE, WHAT DO YOU WANT FOR DINNER TO...</td>\n","      <td>Will and Emma are having a conversation about...</td>\n","      <td>Will and Emma are having a conversation about...</td>\n","      <td>0.851852</td>\n","      <td>True</td>\n","    </tr>\n","    <tr>\n","      <th>4</th>\n","      <td>robustness</td>\n","      <td>uppercase</td>\n","      <td>Ollie: Hi , are you in Warsaw\\r\\nJane: yes, ju...</td>\n","      <td>OLLIE: HI , ARE YOU IN WARSAW JANE: YES, JUST ...</td>\n","      <td>Ollie and Jane are arranging to meet for lunc...</td>\n","      <td>Ollie and Jane are making plans to meet up fo...</td>\n","      <td>0.352941</td>\n","      <td>False</td>\n","    </tr>\n","    <tr>\n","      <th>5</th>\n","      <td>robustness</td>\n","      <td>lowercase</td>\n","      <td>Hannah: Hey, do you have Betty's number?\\nAman...</td>\n","      <td>hannah: hey, do you have betty's number? amand...</td>\n","      <td>Hannah is looking for Betty's number, but Ama...</td>\n","      <td>Hannah is looking for Betty's number, but Ama...</td>\n","      <td>0.920000</td>\n","      <td>True</td>\n","    </tr>\n","    <tr>\n","      <th>6</th>\n","      <td>robustness</td>\n","      <td>lowercase</td>\n","      <td>Eric: MACHINE!\\r\\nRob: That's so gr8!\\r\\nEric:...</td>\n","      <td>eric: machine! rob: that's so gr8! eric: i kno...</td>\n","      <td>Eric and Rob are discussing a stand-up comedy...</td>\n","      <td>Eric and Rob are discussing a Russian stand-u...</td>\n","      <td>0.288889</td>\n","      <td>False</td>\n","    </tr>\n","    <tr>\n","      <th>7</th>\n","      <td>robustness</td>\n","      <td>lowercase</td>\n","      <td>Lenny: Babe, can you help me with something?\\r...</td>\n","      <td>lenny: babe, can you help me with something? b...</td>\n","      <td>Lenny was unsure which trousers to buy, so he...</td>\n","      <td>Lenny is trying to decide which pair of trous...</td>\n","      <td>0.303571</td>\n","      <td>False</td>\n","    </tr>\n","    <tr>\n","      <th>8</th>\n","      <td>robustness</td>\n","      <td>lowercase</td>\n","      <td>Will: hey babe, what do you want for dinner to...</td>\n","      <td>will: hey babe, what do you want for dinner to...</td>\n","      <td>Will and Emma are discussing dinner plans for...</td>\n","      <td>Will and Emma are discussing dinner plans for...</td>\n","      <td>0.825688</td>\n","      <td>True</td>\n","    </tr>\n","    <tr>\n","      <th>9</th>\n","      <td>robustness</td>\n","      <td>lowercase</td>\n","      <td>Ollie: Hi , are you in Warsaw\\r\\nJane: yes, ju...</td>\n","      <td>ollie: hi , are you in warsaw jane: yes, just ...</td>\n","      <td>Ollie and Jane are arranging to meet for lunc...</td>\n","      <td>Ollie and Jane are making plans to meet up. O...</td>\n","      <td>0.183486</td>\n","      <td>False</td>\n","    </tr>\n","  </tbody>\n","</table>\n","</div>\n","      <button class=\"colab-df-convert\" onclick=\"convertToInteractive('df-e0648226-1f58-4c80-9617-d96af6ab7c50')\"\n","              title=\"Convert this dataframe to an interactive table.\"\n","              style=\"display:none;\">\n","\n","  <svg xmlns=\"http://www.w3.org/2000/svg\" height=\"24px\"viewBox=\"0 0 24 24\"\n","       width=\"24px\">\n","    <path d=\"M0 0h24v24H0V0z\" fill=\"none\"/>\n","    <path d=\"M18.56 5.44l.94 2.06.94-2.06 2.06-.94-2.06-.94-.94-2.06-.94 2.06-2.06.94zm-11 1L8.5 8.5l.94-2.06 2.06-.94-2.06-.94L8.5 2.5l-.94 2.06-2.06.94zm10 10l.94 2.06.94-2.06 2.06-.94-2.06-.94-.94-2.06-.94 2.06-2.06.94z\"/><path d=\"M17.41 7.96l-1.37-1.37c-.4-.4-.92-.59-1.43-.59-.52 0-1.04.2-1.43.59L10.3 9.45l-7.72 7.72c-.78.78-.78 2.05 0 2.83L4 21.41c.39.39.9.59 1.41.59.51 0 1.02-.2 1.41-.59l7.78-7.78 2.81-2.81c.8-.78.8-2.07 0-2.86zM5.41 20L4 18.59l7.72-7.72 1.47 1.35L5.41 20z\"/>\n","  </svg>\n","      </button>\n","\n","\n","\n","    <div id=\"df-6094f0a5-cdbd-42bb-bfc8-d84bef19b5ec\">\n","      <button class=\"colab-df-quickchart\" onclick=\"quickchart('df-6094f0a5-cdbd-42bb-bfc8-d84bef19b5ec')\"\n","              title=\"Suggest charts.\"\n","              style=\"display:none;\">\n","\n","<svg xmlns=\"http://www.w3.org/2000/svg\" height=\"24px\"viewBox=\"0 0 24 24\"\n","     width=\"24px\">\n","    <g>\n","        <path d=\"M19 3H5c-1.1 0-2 .9-2 2v14c0 1.1.9 2 2 2h14c1.1 0 2-.9 2-2V5c0-1.1-.9-2-2-2zM9 17H7v-7h2v7zm4 0h-2V7h2v10zm4 0h-2v-4h2v4z\"/>\n","    </g>\n","</svg>\n","      </button>\n","    </div>\n","\n","<style>\n","  .colab-df-quickchart {\n","    background-color: #E8F0FE;\n","    border: none;\n","    border-radius: 50%;\n","    cursor: pointer;\n","    display: none;\n","    fill: #1967D2;\n","    height: 32px;\n","    padding: 0 0 0 0;\n","    width: 32px;\n","  }\n","\n","  .colab-df-quickchart:hover {\n","    background-color: #E2EBFA;\n","    box-shadow: 0px 1px 2px rgba(60, 64, 67, 0.3), 0px 1px 3px 1px rgba(60, 64, 67, 0.15);\n","    fill: #174EA6;\n","  }\n","\n","  [theme=dark] .colab-df-quickchart {\n","    background-color: #3B4455;\n","    fill: #D2E3FC;\n","  }\n","\n","  [theme=dark] .colab-df-quickchart:hover {\n","    background-color: #434B5C;\n","    box-shadow: 0px 1px 3px 1px rgba(0, 0, 0, 0.15);\n","    filter: drop-shadow(0px 1px 2px rgba(0, 0, 0, 0.3));\n","    fill: #FFFFFF;\n","  }\n","</style>\n","\n","    <script>\n","      async function quickchart(key) {\n","        const containerElement = document.querySelector('#' + key);\n","        const charts = await google.colab.kernel.invokeFunction(\n","            'suggestCharts', [key], {});\n","      }\n","    </script>\n","\n","      <script>\n","\n","function displayQuickchartButton(domScope) {\n","  let quickchartButtonEl =\n","    domScope.querySelector('#df-6094f0a5-cdbd-42bb-bfc8-d84bef19b5ec button.colab-df-quickchart');\n","  quickchartButtonEl.style.display =\n","    google.colab.kernel.accessAllowed ? 'block' : 'none';\n","}\n","\n","        displayQuickchartButton(document);\n","      </script>\n","      <style>\n","    .colab-df-container {\n","      display:flex;\n","      flex-wrap:wrap;\n","      gap: 12px;\n","    }\n","\n","    .colab-df-convert {\n","      background-color: #E8F0FE;\n","      border: none;\n","      border-radius: 50%;\n","      cursor: pointer;\n","      display: none;\n","      fill: #1967D2;\n","      height: 32px;\n","      padding: 0 0 0 0;\n","      width: 32px;\n","    }\n","\n","    .colab-df-convert:hover {\n","      background-color: #E2EBFA;\n","      box-shadow: 0px 1px 2px rgba(60, 64, 67, 0.3), 0px 1px 3px 1px rgba(60, 64, 67, 0.15);\n","      fill: #174EA6;\n","    }\n","\n","    [theme=dark] .colab-df-convert {\n","      background-color: #3B4455;\n","      fill: #D2E3FC;\n","    }\n","\n","    [theme=dark] .colab-df-convert:hover {\n","      background-color: #434B5C;\n","      box-shadow: 0px 1px 3px 1px rgba(0, 0, 0, 0.15);\n","      filter: drop-shadow(0px 1px 2px rgba(0, 0, 0, 0.3));\n","      fill: #FFFFFF;\n","    }\n","  </style>\n","\n","      <script>\n","        const buttonEl =\n","          document.querySelector('#df-e0648226-1f58-4c80-9617-d96af6ab7c50 button.colab-df-convert');\n","        buttonEl.style.display =\n","          google.colab.kernel.accessAllowed ? 'block' : 'none';\n","\n","        async function convertToInteractive(key) {\n","          const element = document.querySelector('#df-e0648226-1f58-4c80-9617-d96af6ab7c50');\n","          const dataTable =\n","            await google.colab.kernel.invokeFunction('convertToInteractive',\n","                                                     [key], {});\n","          if (!dataTable) return;\n","\n","          const docLinkHtml = 'Like what you see? Visit the ' +\n","            '<a target=\"_blank\" href=https://colab.research.google.com/notebooks/data_table.ipynb>data table notebook</a>'\n","            + ' to learn more about interactive tables.';\n","          element.innerHTML = '';\n","          dataTable['output_type'] = 'display_data';\n","          await google.colab.output.renderOutput(dataTable, element);\n","          const docLink = document.createElement('div');\n","          docLink.innerHTML = docLinkHtml;\n","          element.appendChild(docLink);\n","        }\n","      </script>\n","    </div>\n","  </div>\n"],"text/plain":["     category  test_type                                           original  \\\n","0  robustness  uppercase  Hannah: Hey, do you have Betty's number?\\nAman...   \n","1  robustness  uppercase  Eric: MACHINE!\\r\\nRob: That's so gr8!\\r\\nEric:...   \n","2  robustness  uppercase  Lenny: Babe, can you help me with something?\\r...   \n","3  robustness  uppercase  Will: hey babe, what do you want for dinner to...   \n","4  robustness  uppercase  Ollie: Hi , are you in Warsaw\\r\\nJane: yes, ju...   \n","5  robustness  lowercase  Hannah: Hey, do you have Betty's number?\\nAman...   \n","6  robustness  lowercase  Eric: MACHINE!\\r\\nRob: That's so gr8!\\r\\nEric:...   \n","7  robustness  lowercase  Lenny: Babe, can you help me with something?\\r...   \n","8  robustness  lowercase  Will: hey babe, what do you want for dinner to...   \n","9  robustness  lowercase  Ollie: Hi , are you in Warsaw\\r\\nJane: yes, ju...   \n","\n","                                           test_case  \\\n","0  HANNAH: HEY, DO YOU HAVE BETTY'S NUMBER? AMAND...   \n","1  ERIC: MACHINE! ROB: THAT'S SO GR8! ERIC: I KNO...   \n","2  LENNY: BABE, CAN YOU HELP ME WITH SOMETHING? B...   \n","3  WILL: HEY BABE, WHAT DO YOU WANT FOR DINNER TO...   \n","4  OLLIE: HI , ARE YOU IN WARSAW JANE: YES, JUST ...   \n","5  hannah: hey, do you have betty's number? amand...   \n","6  eric: machine! rob: that's so gr8! eric: i kno...   \n","7  lenny: babe, can you help me with something? b...   \n","8  will: hey babe, what do you want for dinner to...   \n","9  ollie: hi , are you in warsaw jane: yes, just ...   \n","\n","                                     expected_result  \\\n","0   Hannah is looking for Betty's phone number, b...   \n","1   Eric and Rob are discussing a stand-up comedy...   \n","2   Lenny was unsure which trousers to buy and as...   \n","3   Will and Emma are having a conversation about...   \n","4   Ollie and Jane are arranging to meet for lunc...   \n","5   Hannah is looking for Betty's number, but Ama...   \n","6   Eric and Rob are discussing a stand-up comedy...   \n","7   Lenny was unsure which trousers to buy, so he...   \n","8   Will and Emma are discussing dinner plans for...   \n","9   Ollie and Jane are arranging to meet for lunc...   \n","\n","                                       actual_result  eval_score   pass  \n","0   Hannah is looking for Betty's number, but Ama...    0.969697   True  \n","1   Eric and Rob are discussing a stand-up comedy...    0.413793  False  \n","2   Lenny is trying to decide which pair of trous...    0.152381  False  \n","3   Will and Emma are having a conversation about...    0.851852   True  \n","4   Ollie and Jane are making plans to meet up fo...    0.352941  False  \n","5   Hannah is looking for Betty's number, but Ama...    0.920000   True  \n","6   Eric and Rob are discussing a Russian stand-u...    0.288889  False  \n","7   Lenny is trying to decide which pair of trous...    0.303571  False  \n","8   Will and Emma are discussing dinner plans for...    0.825688   True  \n","9   Ollie and Jane are making plans to meet up. O...    0.183486  False  "]},"execution_count":42,"metadata":{},"output_type":"execute_result"}],"source":["harness.generated_results()"]},{"cell_type":"markdown","metadata":{"id":"WBda2qn1GaLl"},"source":["This method returns the generated results in the form of a pandas dataframe, which provides a convenient and easy-to-use format for working with the test results. You can use this method to quickly identify the test cases that failed and to determine where fixes are needed."]},{"cell_type":"markdown","metadata":{"id":"3Ko-gZISE3oW"},"source":["### Report of the tests"]},{"cell_type":"markdown","metadata":{"id":"Ir8VwGYzGdE1"},"source":["We can call `.report()` which summarizes the results giving information about pass and fail counts and overall test pass/fail flag."]},{"cell_type":"code","execution_count":40,"metadata":{"colab":{"base_uri":"https://localhost:8080/","height":112},"executionInfo":{"elapsed":4062,"status":"ok","timestamp":1689534196772,"user":{"displayName":"Prikshit Sharma","userId":"02603035510215936699"},"user_tz":-330},"id":"qu5TUU2kE0nb","outputId":"ed425397-cd1f-4b34-9423-7d66e2e260df"},"outputs":[{"data":{"text/html":["\n","\n","  <div id=\"df-43c904ee-e7b3-4a35-9556-72e5b6ec6680\">\n","    <div class=\"colab-df-container\">\n","      <div>\n","<style scoped>\n","    .dataframe tbody tr th:only-of-type {\n","        vertical-align: middle;\n","    }\n","\n","    .dataframe tbody tr th {\n","        vertical-align: top;\n","    }\n","\n","    .dataframe thead th {\n","        text-align: right;\n","    }\n","</style>\n","<table border=\"1\" class=\"dataframe\">\n","  <thead>\n","    <tr style=\"text-align: right;\">\n","      <th></th>\n","      <th>category</th>\n","      <th>test_type</th>\n","      <th>fail_count</th>\n","      <th>pass_count</th>\n","      <th>pass_rate</th>\n","      <th>minimum_pass_rate</th>\n","      <th>pass</th>\n","    </tr>\n","  </thead>\n","  <tbody>\n","    <tr>\n","      <th>0</th>\n","      <td>robustness</td>\n","      <td>uppercase</td>\n","      <td>3</td>\n","      <td>2</td>\n","      <td>40%</td>\n","      <td>66%</td>\n","      <td>False</td>\n","    </tr>\n","    <tr>\n","      <th>1</th>\n","      <td>robustness</td>\n","      <td>lowercase</td>\n","      <td>3</td>\n","      <td>2</td>\n","      <td>40%</td>\n","      <td>60%</td>\n","      <td>False</td>\n","    </tr>\n","  </tbody>\n","</table>\n","</div>\n","      <button class=\"colab-df-convert\" onclick=\"convertToInteractive('df-43c904ee-e7b3-4a35-9556-72e5b6ec6680')\"\n","              title=\"Convert this dataframe to an interactive table.\"\n","              style=\"display:none;\">\n","\n","  <svg xmlns=\"http://www.w3.org/2000/svg\" height=\"24px\"viewBox=\"0 0 24 24\"\n","       width=\"24px\">\n","    <path d=\"M0 0h24v24H0V0z\" fill=\"none\"/>\n","    <path d=\"M18.56 5.44l.94 2.06.94-2.06 2.06-.94-2.06-.94-.94-2.06-.94 2.06-2.06.94zm-11 1L8.5 8.5l.94-2.06 2.06-.94-2.06-.94L8.5 2.5l-.94 2.06-2.06.94zm10 10l.94 2.06.94-2.06 2.06-.94-2.06-.94-.94-2.06-.94 2.06-2.06.94z\"/><path d=\"M17.41 7.96l-1.37-1.37c-.4-.4-.92-.59-1.43-.59-.52 0-1.04.2-1.43.59L10.3 9.45l-7.72 7.72c-.78.78-.78 2.05 0 2.83L4 21.41c.39.39.9.59 1.41.59.51 0 1.02-.2 1.41-.59l7.78-7.78 2.81-2.81c.8-.78.8-2.07 0-2.86zM5.41 20L4 18.59l7.72-7.72 1.47 1.35L5.41 20z\"/>\n","  </svg>\n","      </button>\n","\n","\n","\n","    <div id=\"df-4b6aa484-a426-47cd-b207-855a05a6a3ef\">\n","      <button class=\"colab-df-quickchart\" onclick=\"quickchart('df-4b6aa484-a426-47cd-b207-855a05a6a3ef')\"\n","              title=\"Suggest charts.\"\n","              style=\"display:none;\">\n","\n","<svg xmlns=\"http://www.w3.org/2000/svg\" height=\"24px\"viewBox=\"0 0 24 24\"\n","     width=\"24px\">\n","    <g>\n","        <path d=\"M19 3H5c-1.1 0-2 .9-2 2v14c0 1.1.9 2 2 2h14c1.1 0 2-.9 2-2V5c0-1.1-.9-2-2-2zM9 17H7v-7h2v7zm4 0h-2V7h2v10zm4 0h-2v-4h2v4z\"/>\n","    </g>\n","</svg>\n","      </button>\n","    </div>\n","\n","<style>\n","  .colab-df-quickchart {\n","    background-color: #E8F0FE;\n","    border: none;\n","    border-radius: 50%;\n","    cursor: pointer;\n","    display: none;\n","    fill: #1967D2;\n","    height: 32px;\n","    padding: 0 0 0 0;\n","    width: 32px;\n","  }\n","\n","  .colab-df-quickchart:hover {\n","    background-color: #E2EBFA;\n","    box-shadow: 0px 1px 2px rgba(60, 64, 67, 0.3), 0px 1px 3px 1px rgba(60, 64, 67, 0.15);\n","    fill: #174EA6;\n","  }\n","\n","  [theme=dark] .colab-df-quickchart {\n","    background-color: #3B4455;\n","    fill: #D2E3FC;\n","  }\n","\n","  [theme=dark] .colab-df-quickchart:hover {\n","    background-color: #434B5C;\n","    box-shadow: 0px 1px 3px 1px rgba(0, 0, 0, 0.15);\n","    filter: drop-shadow(0px 1px 2px rgba(0, 0, 0, 0.3));\n","    fill: #FFFFFF;\n","  }\n","</style>\n","\n","    <script>\n","      async function quickchart(key) {\n","        const containerElement = document.querySelector('#' + key);\n","        const charts = await google.colab.kernel.invokeFunction(\n","            'suggestCharts', [key], {});\n","      }\n","    </script>\n","\n","      <script>\n","\n","function displayQuickchartButton(domScope) {\n","  let quickchartButtonEl =\n","    domScope.querySelector('#df-4b6aa484-a426-47cd-b207-855a05a6a3ef button.colab-df-quickchart');\n","  quickchartButtonEl.style.display =\n","    google.colab.kernel.accessAllowed ? 'block' : 'none';\n","}\n","\n","        displayQuickchartButton(document);\n","      </script>\n","      <style>\n","    .colab-df-container {\n","      display:flex;\n","      flex-wrap:wrap;\n","      gap: 12px;\n","    }\n","\n","    .colab-df-convert {\n","      background-color: #E8F0FE;\n","      border: none;\n","      border-radius: 50%;\n","      cursor: pointer;\n","      display: none;\n","      fill: #1967D2;\n","      height: 32px;\n","      padding: 0 0 0 0;\n","      width: 32px;\n","    }\n","\n","    .colab-df-convert:hover {\n","      background-color: #E2EBFA;\n","      box-shadow: 0px 1px 2px rgba(60, 64, 67, 0.3), 0px 1px 3px 1px rgba(60, 64, 67, 0.15);\n","      fill: #174EA6;\n","    }\n","\n","    [theme=dark] .colab-df-convert {\n","      background-color: #3B4455;\n","      fill: #D2E3FC;\n","    }\n","\n","    [theme=dark] .colab-df-convert:hover {\n","      background-color: #434B5C;\n","      box-shadow: 0px 1px 3px 1px rgba(0, 0, 0, 0.15);\n","      filter: drop-shadow(0px 1px 2px rgba(0, 0, 0, 0.3));\n","      fill: #FFFFFF;\n","    }\n","  </style>\n","\n","      <script>\n","        const buttonEl =\n","          document.querySelector('#df-43c904ee-e7b3-4a35-9556-72e5b6ec6680 button.colab-df-convert');\n","        buttonEl.style.display =\n","          google.colab.kernel.accessAllowed ? 'block' : 'none';\n","\n","        async function convertToInteractive(key) {\n","          const element = document.querySelector('#df-43c904ee-e7b3-4a35-9556-72e5b6ec6680');\n","          const dataTable =\n","            await google.colab.kernel.invokeFunction('convertToInteractive',\n","                                                     [key], {});\n","          if (!dataTable) return;\n","\n","          const docLinkHtml = 'Like what you see? Visit the ' +\n","            '<a target=\"_blank\" href=https://colab.research.google.com/notebooks/data_table.ipynb>data table notebook</a>'\n","            + ' to learn more about interactive tables.';\n","          element.innerHTML = '';\n","          dataTable['output_type'] = 'display_data';\n","          await google.colab.output.renderOutput(dataTable, element);\n","          const docLink = document.createElement('div');\n","          docLink.innerHTML = docLinkHtml;\n","          element.appendChild(docLink);\n","        }\n","      </script>\n","    </div>\n","  </div>\n"],"text/plain":["     category  test_type  fail_count  pass_count pass_rate minimum_pass_rate  \\\n","0  robustness  uppercase           3           2       40%               66%   \n","1  robustness  lowercase           3           2       40%               60%   \n","\n","    pass  \n","0  False  \n","1  False  "]},"execution_count":40,"metadata":{},"output_type":"execute_result"}],"source":["harness.report()"]}],"metadata":{"accelerator":"TPU","colab":{"machine_shape":"hm","provenance":[],"toc_visible":true},"gpuClass":"standard","kernelspec":{"display_name":"Python 3","name":"python3"},"language_info":{"codemirror_mode":{"name":"ipython","version":3},"file_extension":".py","mimetype":"text/x-python","name":"python","nbconvert_exporter":"python","pygments_lexer":"ipython3","version":"3.9.13"}},"nbformat":4,"nbformat_minor":0}
diff --git a/demo/tutorials/misc/Multiple_Variations_Notebook.ipynb b/demo/tutorials/misc/Multiple_Variations_Notebook.ipynb
index 89bdab331..e7ea3e0b3 100644
--- a/demo/tutorials/misc/Multiple_Variations_Notebook.ipynb
+++ b/demo/tutorials/misc/Multiple_Variations_Notebook.ipynb
@@ -178,7 +178,7 @@
       },
       "outputs": [],
       "source": [
-        "!pip install johnsnowlabs"
+        "!pip install langtest[johnsnowlabs] transformers==4.28.1"
       ]
     },
     {
diff --git a/demo/tutorials/misc/RuntimeTest_Notebook.ipynb b/demo/tutorials/misc/RuntimeTest_Notebook.ipynb
index 887eac49e..713b8a146 100644
--- a/demo/tutorials/misc/RuntimeTest_Notebook.ipynb
+++ b/demo/tutorials/misc/RuntimeTest_Notebook.ipynb
@@ -53,6 +53,22 @@
         "!pip install langtest"
       ]
     },
+    {
+      "cell_type": "markdown",
+      "metadata": {},
+      "source": [
+        "### Installing required dependencies"
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {},
+      "outputs": [],
+      "source": [
+        "!pip install transformers==4.28.1"
+      ]
+    },
     {
       "attachments": {},
       "cell_type": "markdown",
diff --git a/demo/tutorials/task-specific-notebooks/Translation_Notebook.ipynb b/demo/tutorials/task-specific-notebooks/Translation_Notebook.ipynb
index 1fb4842fe..b82dd7ba7 100644
--- a/demo/tutorials/task-specific-notebooks/Translation_Notebook.ipynb
+++ b/demo/tutorials/task-specific-notebooks/Translation_Notebook.ipynb
@@ -49,6 +49,15 @@
         "!pip install langtest"
       ]
     },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {},
+      "outputs": [],
+      "source": [
+        "!pip install transformers==4.28.1"
+      ]
+    },
     {
       "cell_type": "markdown",
       "metadata": {
@@ -78,7 +87,7 @@
       ],
       "source": [
         "# Import Harness from the LangTest library\n",
-        "from langtest import Harness\n"
+        "from langtest import Harness"
       ]
     },
     {
@@ -149,7 +158,7 @@
       ],
       "source": [
         "harness = Harness(task=\"translation\",  model='t5-base',\n",
-        "                  hub=\"huggingface\", data=\"Translation-test\")\n"
+        "                  hub=\"huggingface\", data=\"Translation-test\")"
       ]
     },
     {
diff --git a/docs/_data/navigation.yml b/docs/_data/navigation.yml
index cbbda59ca..ae77c6d8b 100644
--- a/docs/_data/navigation.yml
+++ b/docs/_data/navigation.yml
@@ -10,6 +10,9 @@ header:
   - title: Tests
     url: /docs/pages/tests/test
     key: tests
+  - title: Blog
+    url: https://www.johnsnowlabs.com/responsible-ai-blog/
+    key: blog
 
 
 docs-menu:
diff --git a/docs/_layouts/landing.html b/docs/_layouts/landing.html
index 08f132697..a821c1f91 100644
--- a/docs/_layouts/landing.html
+++ b/docs/_layouts/landing.html
@@ -94,6 +94,8 @@ <h3 class="grey h3_title">{{ _section.title }}</h3>
                   <div class="tabs-item">
                     <div class="highlight-box">
                       {% highlight python %}
+!pip install langtest[johnsnowlabs] transformers==4.28.1
+
 from langtest import Harness
 
 # Create a Harness object
@@ -107,6 +109,8 @@ <h3 class="grey h3_title">{{ _section.title }}</h3>
                   <div class="tabs-item">
                     <div class="highlight-box">
                       {% highlight python %}
+!pip install langtest transformers==4.28.1
+
 from langtest import Harness
 
 # Create a Harness object
@@ -120,6 +124,9 @@ <h3 class="grey h3_title">{{ _section.title }}</h3>
                   <div class="tabs-item">
                     <div class="highlight-box">
                       {% highlight python %}
+
+!pip install "langtest[transformers,langchain,openai]"
+                 
 from langtest import Harness
 
 # Set API keys
@@ -136,7 +143,9 @@ <h3 class="grey h3_title">{{ _section.title }}</h3>
                   <div class="tabs-item">
                     <div class="highlight-box">
                       {% highlight python %}
-from langtest import Harness
+
+!pip install langtest[spacy] transformers==4.28.1
+from langtest import Harness                    
 
 # Create a Harness object
 h = Harness(task='ner', model='en_core_web_sm', hub='spacy')
diff --git a/docs/index.md b/docs/index.md
index 8221ab154..d24c5f1c0 100644
--- a/docs/index.md
+++ b/docs/index.md
@@ -1,7 +1,7 @@
 ---
 layout: landing
 title: 'LangTest <span>Deliver Safe & Effective Language Models</span>'
-excerpt: <br> 50+ Tests Evaluating Language Models
+excerpt: <br> 50+ Test Types for Comparing LLM & NLP Models on Accuracy, Bias, Fairness, Robustness & More
 permalink: /
 header: true
 article_header:
diff --git a/docs/pages/docs/data.md b/docs/pages/docs/data.md
index ec875a14e..7e74f3917 100644
--- a/docs/pages/docs/data.md
+++ b/docs/pages/docs/data.md
@@ -133,7 +133,7 @@ In the Harness, we specify the data input in the following way:
 from langtest import Harness
 
 harness = Harness(task="text-classification", hub="huggingface",
-                   model="distilbert-base-uncased-finetuned-sst-2-english",
+                  model="distilbert-base-uncased-finetuned-sst-2-english",
                   data={"name":'glue',
                   "subset":"sst2",
                   "feature_column":"sentence",
@@ -225,6 +225,26 @@ harness = Harness(task='summarization',
                   hub ='openai',
                   data='XSum-test-tiny')
 ```
+
+#### Passing a Hugging Face Dataset for Summarization to the Harness
+
+In the Harness, we specify the data input in the following way:
+
+```python
+# Import Harness from the LangTest library
+from langtest import Harness
+
+harness = Harness(task="summarization", 
+                  hub="openai",
+                  model="text-davinci-003",
+                  data={"name":'samsum',
+                  "feature_column":"dialogue",
+                  "target_column":'summary',
+                  "split":"test"
+                  })
+```
+</div><div class="h3-box" markdown="1">
+
 ### Toxicity
 
 This test checks the toxicity of the completion., the user is meant to select a benchmark dataset from the following list:
diff --git a/docs/pages/docs/model.md b/docs/pages/docs/model.md
index 3254f6bd0..02dc180d1 100644
--- a/docs/pages/docs/model.md
+++ b/docs/pages/docs/model.md
@@ -17,7 +17,7 @@ The `Harness` `model` parameter accepts either a pretrained model or pipeline fr
 #### Pretrained Models
 
 ```bash
-pip install johnsnowlabs
+!pip install johnsnowlabs
 ```
 
 ```python
@@ -84,6 +84,10 @@ h.generate().run().report()
 
 #### Pretrained Models
 
+```bash
+!pip install langtest[transformers]
+```
+
 ```python
 from langtest import Harness
 
@@ -112,6 +116,10 @@ h.generate().run().report()
 
 Using any large language model from the [OpenAI API](https://platform.openai.com/docs/models/overview):
 
+```bash
+!pip install "langtest[langchain,openai,transformers]"
+```
+
 ```python
 from langtest import Harness
 
@@ -131,6 +139,10 @@ h.generate().run().report()
 
 #### Pretrained Models
 
+```bash
+!pip install spacy
+```
+
 ```python
 from langtest import Harness
 
@@ -158,6 +170,10 @@ h.generate().run().report()
 
 #### Pretrained Models
 
+```bash
+!pip install "langtest[transformers,langchain,cohere]"
+```
+
 ```python
 from langtest import Harness
 
@@ -177,6 +193,10 @@ h.generate().run().report()
 
 #### Pretrained Models
 
+```bash
+!pip install "langtest[transformers,langchain,ai21]"
+```
+
 ```python
 from langtest import Harness
 
@@ -195,6 +215,10 @@ h.generate().run().report()
 
 #### Pretrained Models
 
+```bash
+!pip install "langtest[transformers,langchain,openai]"
+```
+
 ```python
 from langtest import Harness
 
@@ -219,6 +243,10 @@ h.generate().run().report()
 
 #### Pretrained Models
 
+```bash
+!pip install "langtest[transformers,langchain,huggingface-hub]"
+```
+
 ```python
 from langtest import Harness
 
diff --git a/docs/pages/docs/one_liner.md b/docs/pages/docs/one_liner.md
index 14f65dc92..268a6a863 100644
--- a/docs/pages/docs/one_liner.md
+++ b/docs/pages/docs/one_liner.md
@@ -26,8 +26,9 @@ Try out the LangTest library on the following default model-dataset combinations
     <div class="tabs-item">
       <div class="highlight-box">
         {% highlight python %}
-from langtest import Harness
+!pip install langtest[johnsnowlabs] transformers==4.28.1
 
+from langtest import Harness
 # Make sure to specify data='path_to_data' when using custom models
 h = Harness(task='ner', model='ner.dl', hub='johnsnowlabs')
 
@@ -39,6 +40,8 @@ h.generate().run().report()
     <div class="tabs-item">
       <div class="highlight-box">
         {% highlight python %}
+!pip install langtest transformers==4.28.1
+
 from langtest import Harness
 
 # Make sure to specify data='path_to_data' when using custom models
@@ -52,6 +55,8 @@ h.generate().run().report()
     <div class="tabs-item">
       <div class="highlight-box">
         {% highlight python %}
+!pip install langtest[spacy] transformers==4.28.1
+
 from langtest import Harness
 
 # Make sure to specify data='path_to_data' when using custom models
@@ -79,6 +84,8 @@ Try out the LangTest library on the following default model-dataset combinations
     <div class="tabs-item">
       <div class="highlight-box">
         {% highlight python %}
+!pip install langtest[johnsnowlabs] transformers==4.28.1
+
 from langtest import Harness
 
 # Make sure to specify data='path_to_data' when using custom models
@@ -92,7 +99,7 @@ h.generate().run().report()
     <div class="tabs-item">
       <div class="highlight-box">
         {% highlight python %}
-from langtest import Harness
+!pip install langtest transformers==4.28.1
 
 # Make sure to specify data='path_to_data' when using custom models
 h = Harness(task='text-classification', model='lvwerra/distilbert-imdb', hub='huggingface')
@@ -105,6 +112,8 @@ h.generate().run().report()
     <div class="tabs-item">
       <div class="highlight-box">
         {% highlight python %}
+!pip install langtest[spacy] transformers==4.28.1
+
 from langtest import Harness
 
 # Make sure to specify data='path_to_data' when using custom models
@@ -131,6 +140,8 @@ Try out the LangTest library on the following default model-dataset combinations
     <div class="tabs-item">
       <div class="highlight-box">
         {% highlight python %}
+!pip install "langtest[transformers,langchain,openai]"
+
 from langtest import Harness
 
 # Set API keys
@@ -160,6 +171,8 @@ Try out the LangTest library on the following default model-dataset combinations
     <div class="tabs-item">
       <div class="highlight-box">
         {% highlight python %}
+!pip install "langtest[evaluate,transformers,langchain,openai]"
+
 from langtest import Harness
 
 # Set API keys
@@ -188,6 +201,8 @@ Try out the LangTest library on the following default model-dataset combinations
     <div class="tabs-item">
       <div class="highlight-box">
         {% highlight python %}
+!pip install "langtest[evaluate,transformers,langchain,openai]"
+
 from langtest import Harness
 
 # Set API keys
@@ -216,6 +231,7 @@ To compare different models (either from same or different hubs) on the same tas
     <div class="tabs-item">
       <div class="highlight-box">
         {% highlight python %}
+!pip install "langtest[spacy,johnsnowlabs]" transformers==4.28.1
 from langtest import Harness
 
 # Define the dictionary
@@ -245,6 +261,8 @@ Try out the LangTest library on the following default model-dataset combinations
     <div class="tabs-item">
       <div class="highlight-box">
         {% highlight python %}
+!pip install langtest[transformers]
+
 from langtest import Harness
 
 # Create a Harness object


Test patch:
diff --git a/demo/tutorials/test-specific-notebooks/Accuracy_Demo.ipynb b/demo/tutorials/test-specific-notebooks/Accuracy_Demo.ipynb
index 18ebf4888..7c89fb2c3 100644
--- a/demo/tutorials/test-specific-notebooks/Accuracy_Demo.ipynb
+++ b/demo/tutorials/test-specific-notebooks/Accuracy_Demo.ipynb
@@ -86,7 +86,7 @@
     },
     {
       "cell_type": "code",
-      "execution_count": 3,
+      "execution_count": 4,
       "metadata": {
         "id": "w1g27-uxl1AA"
       },
@@ -1061,7 +1061,7 @@
       "name": "python",
       "nbconvert_exporter": "python",
       "pygments_lexer": "ipython3",
-      "version": "3.9.16"
+      "version": "3.11.4"
     }
   },
   "nbformat": 4,
diff --git a/demo/tutorials/test-specific-notebooks/Fairness_Demo.ipynb b/demo/tutorials/test-specific-notebooks/Fairness_Demo.ipynb
index c12accb32..76fcaf406 100644
--- a/demo/tutorials/test-specific-notebooks/Fairness_Demo.ipynb
+++ b/demo/tutorials/test-specific-notebooks/Fairness_Demo.ipynb
@@ -41,26 +41,6 @@
         "# Getting started with LangTest on John Snow Labs"
       ]
     },
-    {
-      "cell_type": "code",
-      "execution_count": null,
-      "metadata": {
-        "id": "kJ-dxTWu7bcA"
-      },
-      "outputs": [],
-      "source": [
-        "!pip install langtest"
-      ]
-    },
-    {
-      "cell_type": "markdown",
-      "metadata": {
-        "id": "VVVWrtnu77eU"
-      },
-      "source": [
-        "# John Snow Labs setup"
-      ]
-    },
     {
       "cell_type": "code",
       "execution_count": null,
@@ -69,7 +49,7 @@
       },
       "outputs": [],
       "source": [
-        "!pip install johnsnowlabs"
+        "!pip install langtest[johnsnowlabs] transformers==4.28.1"
       ]
     },
     {
diff --git a/docs/pages/tests/robustness/strip_all_punctuation.md b/docs/pages/tests/robustness/strip_all_punctuation.md
new file mode 100644
index 000000000..07a8c0c61
--- /dev/null
+++ b/docs/pages/tests/robustness/strip_all_punctuation.md
@@ -0,0 +1,37 @@
+
+<div class="h3-box" markdown="1">
+
+## Strip All Punctuation
+
+This test checks if the NLP model can handle sentences with no punctuations.
+
+**alias_name:** `strip_all_punctuation`
+
+<i class="fa fa-info-circle"></i>
+<em>To test QA models, we are using QAEval from Langchain where we need to use the model itself or other ML model for evaluation, which can make mistakes.</em>
+
+</div><div class="h3-box" markdown="1">
+
+#### Config
+```yaml
+strip_all_punctuation:
+    min_pass_rate: 0.7
+    prob: 0.5 # Defaults to 1.0, which means all words will be transformed.
+```
+<i class="fa fa-info-circle"></i>
+<em>You can adjust the level of transformation in the sentence by using the "`prob`" parameter, which controls the proportion of words to be changed during `strip_all_punctuation` test.</em>
+
+- **min_pass_rate (float):** Minimum pass rate to pass the test.
+- **prob (float):** Controls the proportion of words to be changed.
+
+</div><div class="h3-box" markdown="1">
+
+#### Examples
+
+{:.table2}
+|Original|Test Case|
+|-|
+|Dutasteride 0.5 mg Capsule Sig : One ( 1 ) Capsule PO once a day.|Dutasteride 0.5 mg Capsule Sig  One ( 1 ) Capsule PO once a day|
+|In conclusion , RSDS is a relevant osteoarticular complication in patients receiving either anticalcineurinic drug ( CyA or tacrolimus ) , even under monotherapy or with a low steroid dose.|In conclusion  RSDS is a relevant osteoarticular complication in patients receiving either anticalcineurinic drug ( CyA or tacrolimus )  even under monotherapy or with a low steroid dose|
+
+</div>
diff --git a/docs/pages/tests/test.md b/docs/pages/tests/test.md
index a0160b230..6a3684fcf 100644
--- a/docs/pages/tests/test.md
+++ b/docs/pages/tests/test.md
@@ -99,6 +99,7 @@ The following tables give an overview of the different categories and tests.
 |[Robustness](robustness)		            |[Multiple Perturbations](robustness#multiple-perturbations)                                                        |`text-classification`, `question-answering`, `summarization`, `translation`
 |[Robustness](robustness)		            |[Adjective Synonym Swap](robustness#adjective-synonym-swap)                                                 |`ner`, `text-classification`, `question-answering`, `summarization`, `translation`
 |[Robustness](robustness)		            |[Adjective Antonym Swap](robustness#adjective-antonym-swap)                                                 |`ner`, `text-classification`, `question-answering`, `summarization`, `translation`
+|[Robustness](robustness)		            |[Strip All Punctution](robustness#strip-all-punctuation)                                                 |`ner`, `text-classification`, `question-answering`, `summarization`, `translation`
 |[Toxicity](toxicity)		            |[Offensive](toxicity#Offensive)                                                        |`toxicity`
 
 </div></div>
\ No newline at end of file