Repository: pegasystems/pega-datascientist-tools (language: python)

Issue:
Small things
Couple of things when GitHub-installing the package on another system. Just works in general but small glitches

1. There is no vignette to describe the PMML export functionality, only the scripts in "extra" which should probably go there
2. The main README should probably list the (few) vignettes that we support if not a quick example straight there
3. The import of XML or jsonlite is for some reason required when running the adm2pmml code; something funky with the dependencies somewhere
4. ADM-Reporting vignette has an ugly header showing the vignette meta data
5. The graphs in the adm-datasetanalysis vignette are really small (too small) but the ones in reporting are ok - probably because the header got screwed up as per previous point
6. [FIXED] Filtering in the JSON export version is currently not supported
7. It looks like import of "fasttime" is also mandatory when using getModelsFromDatamart - should not be the case
8. A bunch of palette functions would be nice see https://drsimonj.svbtle.com/creating-corporate-colour-palettes-for-ggplot2
9. options that take the last snapshot only should do group by model ID not assume all snapshots are the same


Discussion / hints:
Obsolete mostly - will file separate issues

Fix patch:
diff --git a/.gitignore b/.gitignore
index 5dafbb8a..f2e5348c 100644
--- a/.gitignore
+++ b/.gitignore
@@ -18,4 +18,5 @@ r/tests/testthat/*.pdf
 build/
 *.egg-info/
 
-.DS_Store
\ No newline at end of file
+.DS_Store
+.idea
\ No newline at end of file
diff --git a/data/SampleHDS.json b/data/SampleHDS.json
new file mode 100644
index 00000000..75488beb
--- /dev/null
+++ b/data/SampleHDS.json
@@ -0,0 +1,7 @@
+{"Context_Name":"FirstMortgage30yr","Customer_MaritalStatus":"Married","Customer_CLV":1460,"Customer_City":"Port Raoul","IH_Web_Inbound_Accepted_pxLastGroupID":"Account","Decision_Outcome":"Rejected"}
+{"Context_Name":"FirstMortgage30yr","Customer_MaritalStatus":"Unknown","Customer_CLV":669,"Customer_City":"Laurianneshire","IH_Web_Inbound_Accepted_pxLastGroupID":"AutoLoans","Decision_Outcome":"Accepted"}
+{"Context_Name":"MoneyMarketSavingsAccount","Customer_MaritalStatus":"No Resp+","Customer_CLV":1174,"Customer_City":"Jacobshaven","IH_Web_Inbound_Accepted_pxLastGroupID":"Account","Decision_Outcome":"Rejected"}
+{"Context_Name":"BasicChecking","Customer_MaritalStatus":"Unknown","Customer_CLV":1476,"Customer_City":"Lindton","IH_Web_Inbound_Accepted_pxLastGroupID":"Account","Decision_Outcome":"Rejected"}
+{"Context_Name":"BasicChecking","Customer_MaritalStatus":"Married","Customer_CLV":1211,"Customer_City":"South Jimmieshire","IH_Web_Inbound_Accepted_pxLastGroupID":"DepositAccounts","Decision_Outcome":"Accepted"}
+{"Context_Name":"UPlusFinPersonal","Customer_MaritalStatus":"No Resp+","Customer_CLV":533,"Customer_City":"Bergeville","IH_Web_Inbound_Accepted_pxLastGroupID":null,"Decision_Outcome":"Rejected"}
+{"Context_Name":"BasicChecking","Customer_MaritalStatus":"No Resp+","Customer_CLV":555,"Customer_City":"Willyville","IH_Web_Inbound_Accepted_pxLastGroupID":"DepositAccounts","Decision_Outcome":"Rejected"}
diff --git a/examples/hds/Example_Data_Anonymization.ipynb b/examples/hds/Example_Data_Anonymization.ipynb
new file mode 100644
index 00000000..adf8f011
--- /dev/null
+++ b/examples/hds/Example_Data_Anonymization.ipynb
@@ -0,0 +1,1450 @@
+{
+ "cells": [
+  {
+   "attachments": {},
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "# Example data anonymization\n",
+    "\n",
+    "In Pega CDH 8.5 and up, it's now possible to record the historical data as seen by the Adaptive Models. See [this academy challenge](https://academy.pega.com/challenge/exporting-historical-data/v4) for reference. This historical data can be further used to experiment with offline models, but also to fine-tune the OOTB Gradient Boosting model. However, sharing this information with Pega can be sensitive as it contains raw predictor data. \n",
+    "\n",
+    "To this end, we provide a simple and transparent script to fully anonimize this dataset.\n",
+    "\n",
+    "The DataAnonymization script is now part of pdstools, and you can import it directly as such."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 1,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "from pdstools import ADMDatamart\n",
+    "from pdstools import Config, DataAnonymization\n",
+    "import polars as pl"
+   ]
+  },
+  {
+   "attachments": {},
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "# Input data\n",
+    "\n",
+    "To demonstrate this process, we're going to anonymise this toy example dataframe:"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 2,
+   "metadata": {},
+   "outputs": [
+    {
+     "data": {
+      "text/html": [
+       "<div>\n",
+       "<style>\n",
+       ".pl-dataframe > thead > tr > th {\n",
+       "  text-align: right;\n",
+       "}\n",
+       "</style>\n",
+       "\n",
+       "<table border=\"1\" class=\"dataframe pl-dataframe\">\n",
+       "<small>shape: (7, 6)</small>\n",
+       "<thead>\n",
+       "<tr>\n",
+       "<th>\n",
+       "Context_Name\n",
+       "</th>\n",
+       "<th>\n",
+       "Customer_MaritalStatus\n",
+       "</th>\n",
+       "<th>\n",
+       "Customer_CLV\n",
+       "</th>\n",
+       "<th>\n",
+       "Customer_City\n",
+       "</th>\n",
+       "<th>\n",
+       "IH_Web_Inbound_Accepted_pxLastGroupID\n",
+       "</th>\n",
+       "<th>\n",
+       "Decision_Outcome\n",
+       "</th>\n",
+       "</tr>\n",
+       "<tr>\n",
+       "<td>\n",
+       "str\n",
+       "</td>\n",
+       "<td>\n",
+       "str\n",
+       "</td>\n",
+       "<td>\n",
+       "i64\n",
+       "</td>\n",
+       "<td>\n",
+       "str\n",
+       "</td>\n",
+       "<td>\n",
+       "str\n",
+       "</td>\n",
+       "<td>\n",
+       "str\n",
+       "</td>\n",
+       "</tr>\n",
+       "</thead>\n",
+       "<tbody>\n",
+       "<tr>\n",
+       "<td>\n",
+       "&quot;FirstMortgage3...\n",
+       "</td>\n",
+       "<td>\n",
+       "&quot;Married&quot;\n",
+       "</td>\n",
+       "<td>\n",
+       "1460\n",
+       "</td>\n",
+       "<td>\n",
+       "&quot;Port Raoul&quot;\n",
+       "</td>\n",
+       "<td>\n",
+       "&quot;Account&quot;\n",
+       "</td>\n",
+       "<td>\n",
+       "&quot;Rejected&quot;\n",
+       "</td>\n",
+       "</tr>\n",
+       "<tr>\n",
+       "<td>\n",
+       "&quot;FirstMortgage3...\n",
+       "</td>\n",
+       "<td>\n",
+       "&quot;Unknown&quot;\n",
+       "</td>\n",
+       "<td>\n",
+       "669\n",
+       "</td>\n",
+       "<td>\n",
+       "&quot;Laurianneshire...\n",
+       "</td>\n",
+       "<td>\n",
+       "&quot;AutoLoans&quot;\n",
+       "</td>\n",
+       "<td>\n",
+       "&quot;Accepted&quot;\n",
+       "</td>\n",
+       "</tr>\n",
+       "<tr>\n",
+       "<td>\n",
+       "&quot;MoneyMarketSav...\n",
+       "</td>\n",
+       "<td>\n",
+       "&quot;No Resp+&quot;\n",
+       "</td>\n",
+       "<td>\n",
+       "1174\n",
+       "</td>\n",
+       "<td>\n",
+       "&quot;Jacobshaven&quot;\n",
+       "</td>\n",
+       "<td>\n",
+       "&quot;Account&quot;\n",
+       "</td>\n",
+       "<td>\n",
+       "&quot;Rejected&quot;\n",
+       "</td>\n",
+       "</tr>\n",
+       "<tr>\n",
+       "<td>\n",
+       "&quot;BasicChecking&quot;\n",
+       "</td>\n",
+       "<td>\n",
+       "&quot;Unknown&quot;\n",
+       "</td>\n",
+       "<td>\n",
+       "1476\n",
+       "</td>\n",
+       "<td>\n",
+       "&quot;Lindton&quot;\n",
+       "</td>\n",
+       "<td>\n",
+       "&quot;Account&quot;\n",
+       "</td>\n",
+       "<td>\n",
+       "&quot;Rejected&quot;\n",
+       "</td>\n",
+       "</tr>\n",
+       "<tr>\n",
+       "<td>\n",
+       "&quot;BasicChecking&quot;\n",
+       "</td>\n",
+       "<td>\n",
+       "&quot;Married&quot;\n",
+       "</td>\n",
+       "<td>\n",
+       "1211\n",
+       "</td>\n",
+       "<td>\n",
+       "&quot;South Jimmiesh...\n",
+       "</td>\n",
+       "<td>\n",
+       "&quot;DepositAccount...\n",
+       "</td>\n",
+       "<td>\n",
+       "&quot;Accepted&quot;\n",
+       "</td>\n",
+       "</tr>\n",
+       "<tr>\n",
+       "<td>\n",
+       "&quot;UPlusFinPerson...\n",
+       "</td>\n",
+       "<td>\n",
+       "&quot;No Resp+&quot;\n",
+       "</td>\n",
+       "<td>\n",
+       "533\n",
+       "</td>\n",
+       "<td>\n",
+       "&quot;Bergeville&quot;\n",
+       "</td>\n",
+       "<td>\n",
+       "null\n",
+       "</td>\n",
+       "<td>\n",
+       "&quot;Rejected&quot;\n",
+       "</td>\n",
+       "</tr>\n",
+       "<tr>\n",
+       "<td>\n",
+       "&quot;BasicChecking&quot;\n",
+       "</td>\n",
+       "<td>\n",
+       "&quot;No Resp+&quot;\n",
+       "</td>\n",
+       "<td>\n",
+       "555\n",
+       "</td>\n",
+       "<td>\n",
+       "&quot;Willyville&quot;\n",
+       "</td>\n",
+       "<td>\n",
+       "&quot;DepositAccount...\n",
+       "</td>\n",
+       "<td>\n",
+       "&quot;Rejected&quot;\n",
+       "</td>\n",
+       "</tr>\n",
+       "</tbody>\n",
+       "</table>\n",
+       "</div>"
+      ],
+      "text/plain": [
+       "shape: (7, 6)\n",
+       "┌────────────────┬────────────────┬──────────────┬────────────────┬────────────────┬───────────────┐\n",
+       "│ Context_Name   ┆ Customer_Marit ┆ Customer_CLV ┆ Customer_City  ┆ IH_Web_Inbound ┆ Decision_Outc │\n",
+       "│ ---            ┆ alStatus       ┆ ---          ┆ ---            ┆ _Accepted_pxLa ┆ ome           │\n",
+       "│ str            ┆ ---            ┆ i64          ┆ str            ┆ stGr...        ┆ ---           │\n",
+       "│                ┆ str            ┆              ┆                ┆ ---            ┆ str           │\n",
+       "│                ┆                ┆              ┆                ┆ str            ┆               │\n",
+       "╞════════════════╪════════════════╪══════════════╪════════════════╪════════════════╪═══════════════╡\n",
+       "│ FirstMortgage3 ┆ Married        ┆ 1460         ┆ Port Raoul     ┆ Account        ┆ Rejected      │\n",
+       "│ 0yr            ┆                ┆              ┆                ┆                ┆               │\n",
+       "│ FirstMortgage3 ┆ Unknown        ┆ 669          ┆ Laurianneshire ┆ AutoLoans      ┆ Accepted      │\n",
+       "│ 0yr            ┆                ┆              ┆                ┆                ┆               │\n",
+       "│ MoneyMarketSav ┆ No Resp+       ┆ 1174         ┆ Jacobshaven    ┆ Account        ┆ Rejected      │\n",
+       "│ ingsAccount    ┆                ┆              ┆                ┆                ┆               │\n",
+       "│ BasicChecking  ┆ Unknown        ┆ 1476         ┆ Lindton        ┆ Account        ┆ Rejected      │\n",
+       "│ BasicChecking  ┆ Married        ┆ 1211         ┆ South          ┆ DepositAccount ┆ Accepted      │\n",
+       "│                ┆                ┆              ┆ Jimmieshire    ┆ s              ┆               │\n",
+       "│ UPlusFinPerson ┆ No Resp+       ┆ 533          ┆ Bergeville     ┆ null           ┆ Rejected      │\n",
+       "│ al             ┆                ┆              ┆                ┆                ┆               │\n",
+       "│ BasicChecking  ┆ No Resp+       ┆ 555          ┆ Willyville     ┆ DepositAccount ┆ Rejected      │\n",
+       "│                ┆                ┆              ┆                ┆ s              ┆               │\n",
+       "└────────────────┴────────────────┴──────────────┴────────────────┴────────────────┴───────────────┘"
+      ]
+     },
+     "execution_count": 2,
+     "metadata": {},
+     "output_type": "execute_result"
+    }
+   ],
+   "source": [
+    "pl.read_ndjson('../../data/SampleHDS.json')"
+   ]
+  },
+  {
+   "attachments": {},
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "As you can see, this dataset consists of regular predictors, IH predictors, context keys and the outcome column. Additionally, some columns are numeric, others are strings. Let's first initialize the DataAnonymization class."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 3,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "anon = DataAnonymization(hdr_folder='../../data')"
+   ]
+  },
+  {
+   "attachments": {},
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "By default, the class applies a set of anonymisation techniques:\n",
+    "- Column names are remapped to a non-descriptive name\n",
+    "- Categorical values are hashed with a random seed\n",
+    "- Numerical values are normalized between 0 and 1\n",
+    "- Outcomes are mapped to a binary outcome.\n",
+    "\n",
+    "To apply these techniques, simply call `.process()`:"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 4,
+   "metadata": {},
+   "outputs": [
+    {
+     "data": {
+      "text/html": [
+       "<div>\n",
+       "<style>\n",
+       ".pl-dataframe > thead > tr > th {\n",
+       "  text-align: right;\n",
+       "}\n",
+       "</style>\n",
+       "\n",
+       "<table border=\"1\" class=\"dataframe pl-dataframe\">\n",
+       "<small>shape: (7, 6)</small>\n",
+       "<thead>\n",
+       "<tr>\n",
+       "<th>\n",
+       "PREDICTOR_0\n",
+       "</th>\n",
+       "<th>\n",
+       "PREDICTOR_1\n",
+       "</th>\n",
+       "<th>\n",
+       "PREDICTOR_2\n",
+       "</th>\n",
+       "<th>\n",
+       "CK_PREDICTOR_0\n",
+       "</th>\n",
+       "<th>\n",
+       "IH_PREDICTOR_0\n",
+       "</th>\n",
+       "<th>\n",
+       "OUTCOME\n",
+       "</th>\n",
+       "</tr>\n",
+       "<tr>\n",
+       "<td>\n",
+       "f64\n",
+       "</td>\n",
+       "<td>\n",
+       "str\n",
+       "</td>\n",
+       "<td>\n",
+       "str\n",
+       "</td>\n",
+       "<td>\n",
+       "str\n",
+       "</td>\n",
+       "<td>\n",
+       "str\n",
+       "</td>\n",
+       "<td>\n",
+       "bool\n",
+       "</td>\n",
+       "</tr>\n",
+       "</thead>\n",
+       "<tbody>\n",
+       "<tr>\n",
+       "<td>\n",
+       "0.983033\n",
+       "</td>\n",
+       "<td>\n",
+       "&quot;98204418375119...\n",
+       "</td>\n",
+       "<td>\n",
+       "&quot;13648188728621...\n",
+       "</td>\n",
+       "<td>\n",
+       "&quot;13393724272505...\n",
+       "</td>\n",
+       "<td>\n",
+       "&quot;99152283876968...\n",
+       "</td>\n",
+       "<td>\n",
+       "false\n",
+       "</td>\n",
+       "</tr>\n",
+       "<tr>\n",
+       "<td>\n",
+       "0.144221\n",
+       "</td>\n",
+       "<td>\n",
+       "&quot;40194439864190...\n",
+       "</td>\n",
+       "<td>\n",
+       "&quot;34289627369783...\n",
+       "</td>\n",
+       "<td>\n",
+       "&quot;13393724272505...\n",
+       "</td>\n",
+       "<td>\n",
+       "&quot;14569848895294...\n",
+       "</td>\n",
+       "<td>\n",
+       "true\n",
+       "</td>\n",
+       "</tr>\n",
+       "<tr>\n",
+       "<td>\n",
+       "0.679745\n",
+       "</td>\n",
+       "<td>\n",
+       "&quot;74996261949779...\n",
+       "</td>\n",
+       "<td>\n",
+       "&quot;82530010984907...\n",
+       "</td>\n",
+       "<td>\n",
+       "&quot;58821356644788...\n",
+       "</td>\n",
+       "<td>\n",
+       "&quot;99152283876968...\n",
+       "</td>\n",
+       "<td>\n",
+       "false\n",
+       "</td>\n",
+       "</tr>\n",
+       "<tr>\n",
+       "<td>\n",
+       "1.0\n",
+       "</td>\n",
+       "<td>\n",
+       "&quot;17508757597135...\n",
+       "</td>\n",
+       "<td>\n",
+       "&quot;34289627369783...\n",
+       "</td>\n",
+       "<td>\n",
+       "&quot;12710223883496...\n",
+       "</td>\n",
+       "<td>\n",
+       "&quot;99152283876968...\n",
+       "</td>\n",
+       "<td>\n",
+       "false\n",
+       "</td>\n",
+       "</tr>\n",
+       "<tr>\n",
+       "<td>\n",
+       "0.718982\n",
+       "</td>\n",
+       "<td>\n",
+       "&quot;64355859976386...\n",
+       "</td>\n",
+       "<td>\n",
+       "&quot;13648188728621...\n",
+       "</td>\n",
+       "<td>\n",
+       "&quot;12710223883496...\n",
+       "</td>\n",
+       "<td>\n",
+       "&quot;67832225186728...\n",
+       "</td>\n",
+       "<td>\n",
+       "true\n",
+       "</td>\n",
+       "</tr>\n",
+       "<tr>\n",
+       "<td>\n",
+       "0.0\n",
+       "</td>\n",
+       "<td>\n",
+       "&quot;91446674384419...\n",
+       "</td>\n",
+       "<td>\n",
+       "&quot;82530010984907...\n",
+       "</td>\n",
+       "<td>\n",
+       "&quot;16136027877032...\n",
+       "</td>\n",
+       "<td>\n",
+       "null\n",
+       "</td>\n",
+       "<td>\n",
+       "false\n",
+       "</td>\n",
+       "</tr>\n",
+       "<tr>\n",
+       "<td>\n",
+       "0.02333\n",
+       "</td>\n",
+       "<td>\n",
+       "&quot;20772971755365...\n",
+       "</td>\n",
+       "<td>\n",
+       "&quot;82530010984907...\n",
+       "</td>\n",
+       "<td>\n",
+       "&quot;12710223883496...\n",
+       "</td>\n",
+       "<td>\n",
+       "&quot;67832225186728...\n",
+       "</td>\n",
+       "<td>\n",
+       "false\n",
+       "</td>\n",
+       "</tr>\n",
+       "</tbody>\n",
+       "</table>\n",
+       "</div>"
+      ],
+      "text/plain": [
+       "shape: (7, 6)\n",
+       "┌─────────────┬──────────────────┬──────────────────┬──────────────────┬─────────────────┬─────────┐\n",
+       "│ PREDICTOR_0 ┆ PREDICTOR_1      ┆ PREDICTOR_2      ┆ CK_PREDICTOR_0   ┆ IH_PREDICTOR_0  ┆ OUTCOME │\n",
+       "│ ---         ┆ ---              ┆ ---              ┆ ---              ┆ ---             ┆ ---     │\n",
+       "│ f64         ┆ str              ┆ str              ┆ str              ┆ str             ┆ bool    │\n",
+       "╞═════════════╪══════════════════╪══════════════════╪══════════════════╪═════════════════╪═════════╡\n",
+       "│ 0.983033    ┆ 9820441837511987 ┆ 1364818872862190 ┆ 1339372427250519 ┆ 991522838769687 ┆ false   │\n",
+       "│             ┆ 503              ┆ 1152             ┆ 9285             ┆ 1670            ┆         │\n",
+       "│ 0.144221    ┆ 4019443986419030 ┆ 3428962736978388 ┆ 1339372427250519 ┆ 145698488952941 ┆ true    │\n",
+       "│             ┆ 17               ┆ 17               ┆ 9285             ┆ 51342           ┆         │\n",
+       "│ 0.679745    ┆ 7499626194977968 ┆ 8253001098490770 ┆ 5882135664478858 ┆ 991522838769687 ┆ false   │\n",
+       "│             ┆ 811              ┆ 655              ┆ 592              ┆ 1670            ┆         │\n",
+       "│ 1.0         ┆ 1750875759713543 ┆ 3428962736978388 ┆ 1271022388349645 ┆ 991522838769687 ┆ false   │\n",
+       "│             ┆ 920              ┆ 17               ┆ 4773             ┆ 1670            ┆         │\n",
+       "│ 0.718982    ┆ 6435585997638671 ┆ 1364818872862190 ┆ 1271022388349645 ┆ 678322251867283 ┆ true    │\n",
+       "│             ┆ 881              ┆ 1152             ┆ 4773             ┆ 3228            ┆         │\n",
+       "│ 0.0         ┆ 9144667438441906 ┆ 8253001098490770 ┆ 1613602787703215 ┆ null            ┆ false   │\n",
+       "│             ┆ 466              ┆ 655              ┆ 5916             ┆                 ┆         │\n",
+       "│ 0.02333     ┆ 2077297175536527 ┆ 8253001098490770 ┆ 1271022388349645 ┆ 678322251867283 ┆ false   │\n",
+       "│             ┆ 093              ┆ 655              ┆ 4773             ┆ 3228            ┆         │\n",
+       "└─────────────┴──────────────────┴──────────────────┴──────────────────┴─────────────────┴─────────┘"
+      ]
+     },
+     "execution_count": 4,
+     "metadata": {},
+     "output_type": "execute_result"
+    }
+   ],
+   "source": [
+    "anon.process()"
+   ]
+  },
+  {
+   "attachments": {},
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "To trace back the columns to their original names, the class also contains a mapping, which does not have to be provided."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 5,
+   "metadata": {},
+   "outputs": [
+    {
+     "data": {
+      "text/plain": [
+       "{'Customer_CLV': 'PREDICTOR_0',\n",
+       " 'Customer_City': 'PREDICTOR_1',\n",
+       " 'Customer_MaritalStatus': 'PREDICTOR_2',\n",
+       " 'Context_Name': 'CK_PREDICTOR_0',\n",
+       " 'IH_Web_Inbound_Accepted_pxLastGroupID': 'IH_PREDICTOR_0',\n",
+       " 'Decision_Outcome': 'OUTCOME'}"
+      ]
+     },
+     "execution_count": 5,
+     "metadata": {},
+     "output_type": "execute_result"
+    }
+   ],
+   "source": [
+    "anon.column_mapping"
+   ]
+  },
+  {
+   "attachments": {},
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "# Configs\n",
+    "\n",
+    "Each capability can optionally be turned off - see below for the full list of config options, and refer to the API reference for the full description."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 6,
+   "metadata": {},
+   "outputs": [
+    {
+     "data": {
+      "text/plain": [
+       "{'config_file': None,\n",
+       " 'hdr_folder': '.',\n",
+       " 'use_datamart': False,\n",
+       " 'datamart_folder': 'datamart',\n",
+       " 'output_format': 'ndjson',\n",
+       " 'output_folder': 'output',\n",
+       " 'mapping_file': 'mapping.map',\n",
+       " 'mask_predictor_names': True,\n",
+       " 'mask_context_key_names': True,\n",
+       " 'mask_ih_names': True,\n",
+       " 'mask_outcome_name': True,\n",
+       " 'mask_predictor_values': True,\n",
+       " 'mask_context_key_values': True,\n",
+       " 'mask_ih_values': True,\n",
+       " 'mask_outcome_values': True,\n",
+       " 'context_key_label': 'Context_*',\n",
+       " 'ih_label': 'IH_*',\n",
+       " 'outcome_column': 'Decision_Outcome',\n",
+       " 'positive_outcomes': ['Accepted', 'Clicked'],\n",
+       " 'negative_outcomes': ['Rejected', 'Impression'],\n",
+       " 'special_predictors': ['Decision_DecisionTime', 'Decision_OutcomeTime']}"
+      ]
+     },
+     "execution_count": 6,
+     "metadata": {},
+     "output_type": "execute_result"
+    }
+   ],
+   "source": [
+    "dict(zip(Config.__init__.__code__.co_varnames[1:], Config.__init__.__defaults__))"
+   ]
+  },
+  {
+   "attachments": {},
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "It's easy to change these parameters by just passing the keyword arguments. In the following example, we\n",
+    "- Keep the IH predictor names\n",
+    "- Keep the outcome values\n",
+    "- Keep the context key values\n",
+    "- Keep the context key predictor names"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 7,
+   "metadata": {},
+   "outputs": [
+    {
+     "data": {
+      "text/html": [
+       "<div>\n",
+       "<style>\n",
+       ".pl-dataframe > thead > tr > th {\n",
+       "  text-align: right;\n",
+       "}\n",
+       "</style>\n",
+       "\n",
+       "<table border=\"1\" class=\"dataframe pl-dataframe\">\n",
+       "<small>shape: (7, 6)</small>\n",
+       "<thead>\n",
+       "<tr>\n",
+       "<th>\n",
+       "PREDICTOR_0\n",
+       "</th>\n",
+       "<th>\n",
+       "PREDICTOR_1\n",
+       "</th>\n",
+       "<th>\n",
+       "PREDICTOR_2\n",
+       "</th>\n",
+       "<th>\n",
+       "Context_Name\n",
+       "</th>\n",
+       "<th>\n",
+       "IH_Web_Inbound_Accepted_pxLastGroupID\n",
+       "</th>\n",
+       "<th>\n",
+       "OUTCOME\n",
+       "</th>\n",
+       "</tr>\n",
+       "<tr>\n",
+       "<td>\n",
+       "f64\n",
+       "</td>\n",
+       "<td>\n",
+       "str\n",
+       "</td>\n",
+       "<td>\n",
+       "str\n",
+       "</td>\n",
+       "<td>\n",
+       "str\n",
+       "</td>\n",
+       "<td>\n",
+       "str\n",
+       "</td>\n",
+       "<td>\n",
+       "str\n",
+       "</td>\n",
+       "</tr>\n",
+       "</thead>\n",
+       "<tbody>\n",
+       "<tr>\n",
+       "<td>\n",
+       "0.983033\n",
+       "</td>\n",
+       "<td>\n",
+       "&quot;13971681906009...\n",
+       "</td>\n",
+       "<td>\n",
+       "&quot;15176930337713...\n",
+       "</td>\n",
+       "<td>\n",
+       "&quot;FirstMortgage3...\n",
+       "</td>\n",
+       "<td>\n",
+       "&quot;24782110777461...\n",
+       "</td>\n",
+       "<td>\n",
+       "&quot;Rejected&quot;\n",
+       "</td>\n",
+       "</tr>\n",
+       "<tr>\n",
+       "<td>\n",
+       "0.144221\n",
+       "</td>\n",
+       "<td>\n",
+       "&quot;69577294999355...\n",
+       "</td>\n",
+       "<td>\n",
+       "&quot;55473831953643...\n",
+       "</td>\n",
+       "<td>\n",
+       "&quot;FirstMortgage3...\n",
+       "</td>\n",
+       "<td>\n",
+       "&quot;30104198515712...\n",
+       "</td>\n",
+       "<td>\n",
+       "&quot;Accepted&quot;\n",
+       "</td>\n",
+       "</tr>\n",
+       "<tr>\n",
+       "<td>\n",
+       "0.679745\n",
+       "</td>\n",
+       "<td>\n",
+       "&quot;14691701592044...\n",
+       "</td>\n",
+       "<td>\n",
+       "&quot;14505132909284...\n",
+       "</td>\n",
+       "<td>\n",
+       "&quot;MoneyMarketSav...\n",
+       "</td>\n",
+       "<td>\n",
+       "&quot;24782110777461...\n",
+       "</td>\n",
+       "<td>\n",
+       "&quot;Rejected&quot;\n",
+       "</td>\n",
+       "</tr>\n",
+       "<tr>\n",
+       "<td>\n",
+       "1.0\n",
+       "</td>\n",
+       "<td>\n",
+       "&quot;14082263256326...\n",
+       "</td>\n",
+       "<td>\n",
+       "&quot;55473831953643...\n",
+       "</td>\n",
+       "<td>\n",
+       "&quot;BasicChecking&quot;\n",
+       "</td>\n",
+       "<td>\n",
+       "&quot;24782110777461...\n",
+       "</td>\n",
+       "<td>\n",
+       "&quot;Rejected&quot;\n",
+       "</td>\n",
+       "</tr>\n",
+       "<tr>\n",
+       "<td>\n",
+       "0.718982\n",
+       "</td>\n",
+       "<td>\n",
+       "&quot;35742398451852...\n",
+       "</td>\n",
+       "<td>\n",
+       "&quot;15176930337713...\n",
+       "</td>\n",
+       "<td>\n",
+       "&quot;BasicChecking&quot;\n",
+       "</td>\n",
+       "<td>\n",
+       "&quot;41946855300578...\n",
+       "</td>\n",
+       "<td>\n",
+       "&quot;Accepted&quot;\n",
+       "</td>\n",
+       "</tr>\n",
+       "<tr>\n",
+       "<td>\n",
+       "0.0\n",
+       "</td>\n",
+       "<td>\n",
+       "&quot;39219711095332...\n",
+       "</td>\n",
+       "<td>\n",
+       "&quot;14505132909284...\n",
+       "</td>\n",
+       "<td>\n",
+       "&quot;UPlusFinPerson...\n",
+       "</td>\n",
+       "<td>\n",
+       "null\n",
+       "</td>\n",
+       "<td>\n",
+       "&quot;Rejected&quot;\n",
+       "</td>\n",
+       "</tr>\n",
+       "<tr>\n",
+       "<td>\n",
+       "0.02333\n",
+       "</td>\n",
+       "<td>\n",
+       "&quot;14857210303222...\n",
+       "</td>\n",
+       "<td>\n",
+       "&quot;14505132909284...\n",
+       "</td>\n",
+       "<td>\n",
+       "&quot;BasicChecking&quot;\n",
+       "</td>\n",
+       "<td>\n",
+       "&quot;41946855300578...\n",
+       "</td>\n",
+       "<td>\n",
+       "&quot;Rejected&quot;\n",
+       "</td>\n",
+       "</tr>\n",
+       "</tbody>\n",
+       "</table>\n",
+       "</div>"
+      ],
+      "text/plain": [
+       "shape: (7, 6)\n",
+       "┌─────────────┬──────────────────┬──────────────────┬─────────────────┬─────────────────┬──────────┐\n",
+       "│ PREDICTOR_0 ┆ PREDICTOR_1      ┆ PREDICTOR_2      ┆ Context_Name    ┆ IH_Web_Inbound_ ┆ OUTCOME  │\n",
+       "│ ---         ┆ ---              ┆ ---              ┆ ---             ┆ Accepted_pxLast ┆ ---      │\n",
+       "│ f64         ┆ str              ┆ str              ┆ str             ┆ Gr...           ┆ str      │\n",
+       "│             ┆                  ┆                  ┆                 ┆ ---             ┆          │\n",
+       "│             ┆                  ┆                  ┆                 ┆ str             ┆          │\n",
+       "╞═════════════╪══════════════════╪══════════════════╪═════════════════╪═════════════════╪══════════╡\n",
+       "│ 0.983033    ┆ 1397168190600999 ┆ 1517693033771346 ┆ FirstMortgage30 ┆ 247821107774611 ┆ Rejected │\n",
+       "│             ┆ 6647             ┆ 7668             ┆ yr              ┆ 0568            ┆          │\n",
+       "│ 0.144221    ┆ 6957729499935594 ┆ 5547383195364360 ┆ FirstMortgage30 ┆ 301041985157128 ┆ Accepted │\n",
+       "│             ┆ 351              ┆ 789              ┆ yr              ┆ 5528            ┆          │\n",
+       "│ 0.679745    ┆ 1469170159204448 ┆ 1450513290928487 ┆ MoneyMarketSavi ┆ 247821107774611 ┆ Rejected │\n",
+       "│             ┆ 9991             ┆ 977              ┆ ngsAccount      ┆ 0568            ┆          │\n",
+       "│ 1.0         ┆ 1408226325632662 ┆ 5547383195364360 ┆ BasicChecking   ┆ 247821107774611 ┆ Rejected │\n",
+       "│             ┆ 6892             ┆ 789              ┆                 ┆ 0568            ┆          │\n",
+       "│ 0.718982    ┆ 3574239845185297 ┆ 1517693033771346 ┆ BasicChecking   ┆ 419468553005781 ┆ Accepted │\n",
+       "│             ┆ 445              ┆ 7668             ┆                 ┆ 4628            ┆          │\n",
+       "│ 0.0         ┆ 3921971109533293 ┆ 1450513290928487 ┆ UPlusFinPersona ┆ null            ┆ Rejected │\n",
+       "│             ┆ 491              ┆ 977              ┆ l               ┆                 ┆          │\n",
+       "│ 0.02333     ┆ 1485721030322218 ┆ 1450513290928487 ┆ BasicChecking   ┆ 419468553005781 ┆ Rejected │\n",
+       "│             ┆ 0226             ┆ 977              ┆                 ┆ 4628            ┆          │\n",
+       "└─────────────┴──────────────────┴──────────────────┴─────────────────┴─────────────────┴──────────┘"
+      ]
+     },
+     "execution_count": 7,
+     "metadata": {},
+     "output_type": "execute_result"
+    }
+   ],
+   "source": [
+    "anon = DataAnonymization(hdr_folder='../../data', mask_ih_names=False, mask_outcome_values=False, mask_context_key_values=False, mask_context_key_names=False)\n",
+    "anon.process()"
+   ]
+  },
+  {
+   "attachments": {},
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "The configs can also be written and read as such:"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 8,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "anon.config.save_to_config_file('config.json')"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 9,
+   "metadata": {},
+   "outputs": [
+    {
+     "data": {
+      "text/html": [
+       "<div>\n",
+       "<style>\n",
+       ".pl-dataframe > thead > tr > th {\n",
+       "  text-align: right;\n",
+       "}\n",
+       "</style>\n",
+       "\n",
+       "<table border=\"1\" class=\"dataframe pl-dataframe\">\n",
+       "<small>shape: (7, 6)</small>\n",
+       "<thead>\n",
+       "<tr>\n",
+       "<th>\n",
+       "PREDICTOR_0\n",
+       "</th>\n",
+       "<th>\n",
+       "PREDICTOR_1\n",
+       "</th>\n",
+       "<th>\n",
+       "PREDICTOR_2\n",
+       "</th>\n",
+       "<th>\n",
+       "Context_Name\n",
+       "</th>\n",
+       "<th>\n",
+       "IH_Web_Inbound_Accepted_pxLastGroupID\n",
+       "</th>\n",
+       "<th>\n",
+       "OUTCOME\n",
+       "</th>\n",
+       "</tr>\n",
+       "<tr>\n",
+       "<td>\n",
+       "f64\n",
+       "</td>\n",
+       "<td>\n",
+       "str\n",
+       "</td>\n",
+       "<td>\n",
+       "str\n",
+       "</td>\n",
+       "<td>\n",
+       "str\n",
+       "</td>\n",
+       "<td>\n",
+       "str\n",
+       "</td>\n",
+       "<td>\n",
+       "str\n",
+       "</td>\n",
+       "</tr>\n",
+       "</thead>\n",
+       "<tbody>\n",
+       "<tr>\n",
+       "<td>\n",
+       "0.983033\n",
+       "</td>\n",
+       "<td>\n",
+       "&quot;13749488081303...\n",
+       "</td>\n",
+       "<td>\n",
+       "&quot;26147176613051...\n",
+       "</td>\n",
+       "<td>\n",
+       "&quot;FirstMortgage3...\n",
+       "</td>\n",
+       "<td>\n",
+       "&quot;14921383665438...\n",
+       "</td>\n",
+       "<td>\n",
+       "&quot;Rejected&quot;\n",
+       "</td>\n",
+       "</tr>\n",
+       "<tr>\n",
+       "<td>\n",
+       "0.144221\n",
+       "</td>\n",
+       "<td>\n",
+       "&quot;88612884249396...\n",
+       "</td>\n",
+       "<td>\n",
+       "&quot;71226778007428...\n",
+       "</td>\n",
+       "<td>\n",
+       "&quot;FirstMortgage3...\n",
+       "</td>\n",
+       "<td>\n",
+       "&quot;98996640880559...\n",
+       "</td>\n",
+       "<td>\n",
+       "&quot;Accepted&quot;\n",
+       "</td>\n",
+       "</tr>\n",
+       "<tr>\n",
+       "<td>\n",
+       "0.679745\n",
+       "</td>\n",
+       "<td>\n",
+       "&quot;15135653830369...\n",
+       "</td>\n",
+       "<td>\n",
+       "&quot;10609058199223...\n",
+       "</td>\n",
+       "<td>\n",
+       "&quot;MoneyMarketSav...\n",
+       "</td>\n",
+       "<td>\n",
+       "&quot;14921383665438...\n",
+       "</td>\n",
+       "<td>\n",
+       "&quot;Rejected&quot;\n",
+       "</td>\n",
+       "</tr>\n",
+       "<tr>\n",
+       "<td>\n",
+       "1.0\n",
+       "</td>\n",
+       "<td>\n",
+       "&quot;86084059998400...\n",
+       "</td>\n",
+       "<td>\n",
+       "&quot;71226778007428...\n",
+       "</td>\n",
+       "<td>\n",
+       "&quot;BasicChecking&quot;\n",
+       "</td>\n",
+       "<td>\n",
+       "&quot;14921383665438...\n",
+       "</td>\n",
+       "<td>\n",
+       "&quot;Rejected&quot;\n",
+       "</td>\n",
+       "</tr>\n",
+       "<tr>\n",
+       "<td>\n",
+       "0.718982\n",
+       "</td>\n",
+       "<td>\n",
+       "&quot;94223062024558...\n",
+       "</td>\n",
+       "<td>\n",
+       "&quot;26147176613051...\n",
+       "</td>\n",
+       "<td>\n",
+       "&quot;BasicChecking&quot;\n",
+       "</td>\n",
+       "<td>\n",
+       "&quot;84282893704966...\n",
+       "</td>\n",
+       "<td>\n",
+       "&quot;Accepted&quot;\n",
+       "</td>\n",
+       "</tr>\n",
+       "<tr>\n",
+       "<td>\n",
+       "0.0\n",
+       "</td>\n",
+       "<td>\n",
+       "&quot;16594334543608...\n",
+       "</td>\n",
+       "<td>\n",
+       "&quot;10609058199223...\n",
+       "</td>\n",
+       "<td>\n",
+       "&quot;UPlusFinPerson...\n",
+       "</td>\n",
+       "<td>\n",
+       "null\n",
+       "</td>\n",
+       "<td>\n",
+       "&quot;Rejected&quot;\n",
+       "</td>\n",
+       "</tr>\n",
+       "<tr>\n",
+       "<td>\n",
+       "0.02333\n",
+       "</td>\n",
+       "<td>\n",
+       "&quot;12843537306871...\n",
+       "</td>\n",
+       "<td>\n",
+       "&quot;10609058199223...\n",
+       "</td>\n",
+       "<td>\n",
+       "&quot;BasicChecking&quot;\n",
+       "</td>\n",
+       "<td>\n",
+       "&quot;84282893704966...\n",
+       "</td>\n",
+       "<td>\n",
+       "&quot;Rejected&quot;\n",
+       "</td>\n",
+       "</tr>\n",
+       "</tbody>\n",
+       "</table>\n",
+       "</div>"
+      ],
+      "text/plain": [
+       "shape: (7, 6)\n",
+       "┌─────────────┬──────────────────┬──────────────────┬─────────────────┬─────────────────┬──────────┐\n",
+       "│ PREDICTOR_0 ┆ PREDICTOR_1      ┆ PREDICTOR_2      ┆ Context_Name    ┆ IH_Web_Inbound_ ┆ OUTCOME  │\n",
+       "│ ---         ┆ ---              ┆ ---              ┆ ---             ┆ Accepted_pxLast ┆ ---      │\n",
+       "│ f64         ┆ str              ┆ str              ┆ str             ┆ Gr...           ┆ str      │\n",
+       "│             ┆                  ┆                  ┆                 ┆ ---             ┆          │\n",
+       "│             ┆                  ┆                  ┆                 ┆ str             ┆          │\n",
+       "╞═════════════╪══════════════════╪══════════════════╪═════════════════╪═════════════════╪══════════╡\n",
+       "│ 0.983033    ┆ 1374948808130327 ┆ 2614717661305103 ┆ FirstMortgage30 ┆ 149213836654383 ┆ Rejected │\n",
+       "│             ┆ 028              ┆ 405              ┆ yr              ┆ 13722           ┆          │\n",
+       "│ 0.144221    ┆ 8861288424939620 ┆ 7122677800742885 ┆ FirstMortgage30 ┆ 989966408805598 ┆ Accepted │\n",
+       "│             ┆ 524              ┆ 129              ┆ yr              ┆ 6879            ┆          │\n",
+       "│ 0.679745    ┆ 1513565383036905 ┆ 1060905819922389 ┆ MoneyMarketSavi ┆ 149213836654383 ┆ Rejected │\n",
+       "│             ┆ 5267             ┆ 6320             ┆ ngsAccount      ┆ 13722           ┆          │\n",
+       "│ 1.0         ┆ 8608405999840033 ┆ 7122677800742885 ┆ BasicChecking   ┆ 149213836654383 ┆ Rejected │\n",
+       "│             ┆ 033              ┆ 129              ┆                 ┆ 13722           ┆          │\n",
+       "│ 0.718982    ┆ 9422306202455803 ┆ 2614717661305103 ┆ BasicChecking   ┆ 842828937049669 ┆ Accepted │\n",
+       "│             ┆ 189              ┆ 405              ┆                 ┆ 2376            ┆          │\n",
+       "│ 0.0         ┆ 1659433454360869 ┆ 1060905819922389 ┆ UPlusFinPersona ┆ null            ┆ Rejected │\n",
+       "│             ┆ 7598             ┆ 6320             ┆ l               ┆                 ┆          │\n",
+       "│ 0.02333     ┆ 1284353730687158 ┆ 1060905819922389 ┆ BasicChecking   ┆ 842828937049669 ┆ Rejected │\n",
+       "│             ┆ 8322             ┆ 6320             ┆                 ┆ 2376            ┆          │\n",
+       "└─────────────┴──────────────────┴──────────────────┴─────────────────┴─────────────────┴──────────┘"
+      ]
+     },
+     "execution_count": 9,
+     "metadata": {},
+     "output_type": "execute_result"
+    }
+   ],
+   "source": [
+    "anon = DataAnonymization(config=Config(config_file='config.json'))\n",
+    "anon.process()"
+   ]
+  },
+  {
+   "attachments": {},
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "# Exporting\n",
+    "Two functions export:\n",
+    "- `create_mapping_file()` writes the mapping file of the predictor names\n",
+    "- `write_to_output()` writes the processed dataframe to disk\n",
+    "\n",
+    "Write to output accepts the following extensions: `[\"ndjson\", \"parquet\", \"arrow\", \"csv\"]`"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 16,
+   "metadata": {},
+   "outputs": [
+    {
+     "name": "stdout",
+     "output_type": "stream",
+     "text": [
+      "Customer_CLV=PREDICTOR_0\n",
+      "Customer_City=PREDICTOR_1\n",
+      "Customer_MaritalStatus=PREDICTOR_2\n",
+      "Context_Name=CK_PREDICTOR_0\n",
+      "IH_Web_Inbound_Accepted_pxLastGroupID=IH_PREDICTOR_0\n",
+      "Decision_Outcome=OUTCOME\n",
+      "\n"
+     ]
+    }
+   ],
+   "source": [
+    "anon.create_mapping_file()\n",
+    "with open('mapping.map') as f:\n",
+    "    print(f.read())"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 11,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "anon.write_to_output(ext='arrow')"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 12,
+   "metadata": {},
+   "outputs": [
+    {
+     "data": {
+      "text/html": [
+       "<div>\n",
+       "<style>\n",
+       ".pl-dataframe > thead > tr > th {\n",
+       "  text-align: right;\n",
+       "}\n",
+       "</style>\n",
+       "\n",
+       "<table border=\"1\" class=\"dataframe pl-dataframe\">\n",
+       "<small>shape: (7, 6)</small>\n",
+       "<thead>\n",
+       "<tr>\n",
+       "<th>\n",
+       "PREDICTOR_0\n",
+       "</th>\n",
+       "<th>\n",
+       "PREDICTOR_1\n",
+       "</th>\n",
+       "<th>\n",
+       "PREDICTOR_2\n",
+       "</th>\n",
+       "<th>\n",
+       "Context_Name\n",
+       "</th>\n",
+       "<th>\n",
+       "IH_Web_Inbound_Accepted_pxLastGroupID\n",
+       "</th>\n",
+       "<th>\n",
+       "OUTCOME\n",
+       "</th>\n",
+       "</tr>\n",
+       "<tr>\n",
+       "<td>\n",
+       "f64\n",
+       "</td>\n",
+       "<td>\n",
+       "str\n",
+       "</td>\n",
+       "<td>\n",
+       "str\n",
+       "</td>\n",
+       "<td>\n",
+       "str\n",
+       "</td>\n",
+       "<td>\n",
+       "str\n",
+       "</td>\n",
+       "<td>\n",
+       "str\n",
+       "</td>\n",
+       "</tr>\n",
+       "</thead>\n",
+       "<tbody>\n",
+       "<tr>\n",
+       "<td>\n",
+       "0.983033\n",
+       "</td>\n",
+       "<td>\n",
+       "&quot;13709236032899...\n",
+       "</td>\n",
+       "<td>\n",
+       "&quot;65280124093365...\n",
+       "</td>\n",
+       "<td>\n",
+       "&quot;FirstMortgage3...\n",
+       "</td>\n",
+       "<td>\n",
+       "&quot;24209164365759...\n",
+       "</td>\n",
+       "<td>\n",
+       "&quot;Rejected&quot;\n",
+       "</td>\n",
+       "</tr>\n",
+       "<tr>\n",
+       "<td>\n",
+       "0.144221\n",
+       "</td>\n",
+       "<td>\n",
+       "&quot;61523008642021...\n",
+       "</td>\n",
+       "<td>\n",
+       "&quot;84822406214008...\n",
+       "</td>\n",
+       "<td>\n",
+       "&quot;FirstMortgage3...\n",
+       "</td>\n",
+       "<td>\n",
+       "&quot;70814434193761...\n",
+       "</td>\n",
+       "<td>\n",
+       "&quot;Accepted&quot;\n",
+       "</td>\n",
+       "</tr>\n",
+       "<tr>\n",
+       "<td>\n",
+       "0.679745\n",
+       "</td>\n",
+       "<td>\n",
+       "&quot;73828690479369...\n",
+       "</td>\n",
+       "<td>\n",
+       "&quot;17057876247860...\n",
+       "</td>\n",
+       "<td>\n",
+       "&quot;MoneyMarketSav...\n",
+       "</td>\n",
+       "<td>\n",
+       "&quot;24209164365759...\n",
+       "</td>\n",
+       "<td>\n",
+       "&quot;Rejected&quot;\n",
+       "</td>\n",
+       "</tr>\n",
+       "<tr>\n",
+       "<td>\n",
+       "1.0\n",
+       "</td>\n",
+       "<td>\n",
+       "&quot;14834594955202...\n",
+       "</td>\n",
+       "<td>\n",
+       "&quot;84822406214008...\n",
+       "</td>\n",
+       "<td>\n",
+       "&quot;BasicChecking&quot;\n",
+       "</td>\n",
+       "<td>\n",
+       "&quot;24209164365759...\n",
+       "</td>\n",
+       "<td>\n",
+       "&quot;Rejected&quot;\n",
+       "</td>\n",
+       "</tr>\n",
+       "<tr>\n",
+       "<td>\n",
+       "0.718982\n",
+       "</td>\n",
+       "<td>\n",
+       "&quot;14416422926530...\n",
+       "</td>\n",
+       "<td>\n",
+       "&quot;65280124093365...\n",
+       "</td>\n",
+       "<td>\n",
+       "&quot;BasicChecking&quot;\n",
+       "</td>\n",
+       "<td>\n",
+       "&quot;10444479079610...\n",
+       "</td>\n",
+       "<td>\n",
+       "&quot;Accepted&quot;\n",
+       "</td>\n",
+       "</tr>\n",
+       "<tr>\n",
+       "<td>\n",
+       "0.0\n",
+       "</td>\n",
+       "<td>\n",
+       "&quot;15868705575342...\n",
+       "</td>\n",
+       "<td>\n",
+       "&quot;17057876247860...\n",
+       "</td>\n",
+       "<td>\n",
+       "&quot;UPlusFinPerson...\n",
+       "</td>\n",
+       "<td>\n",
+       "null\n",
+       "</td>\n",
+       "<td>\n",
+       "&quot;Rejected&quot;\n",
+       "</td>\n",
+       "</tr>\n",
+       "<tr>\n",
+       "<td>\n",
+       "0.02333\n",
+       "</td>\n",
+       "<td>\n",
+       "&quot;32076854049289...\n",
+       "</td>\n",
+       "<td>\n",
+       "&quot;17057876247860...\n",
+       "</td>\n",
+       "<td>\n",
+       "&quot;BasicChecking&quot;\n",
+       "</td>\n",
+       "<td>\n",
+       "&quot;10444479079610...\n",
+       "</td>\n",
+       "<td>\n",
+       "&quot;Rejected&quot;\n",
+       "</td>\n",
+       "</tr>\n",
+       "</tbody>\n",
+       "</table>\n",
+       "</div>"
+      ],
+      "text/plain": [
+       "shape: (7, 6)\n",
+       "┌─────────────┬──────────────────┬──────────────────┬─────────────────┬─────────────────┬──────────┐\n",
+       "│ PREDICTOR_0 ┆ PREDICTOR_1      ┆ PREDICTOR_2      ┆ Context_Name    ┆ IH_Web_Inbound_ ┆ OUTCOME  │\n",
+       "│ ---         ┆ ---              ┆ ---              ┆ ---             ┆ Accepted_pxLast ┆ ---      │\n",
+       "│ f64         ┆ str              ┆ str              ┆ str             ┆ Gr...           ┆ str      │\n",
+       "│             ┆                  ┆                  ┆                 ┆ ---             ┆          │\n",
+       "│             ┆                  ┆                  ┆                 ┆ str             ┆          │\n",
+       "╞═════════════╪══════════════════╪══════════════════╪═════════════════╪═════════════════╪══════════╡\n",
+       "│ 0.983033    ┆ 1370923603289908 ┆ 6528012409336537 ┆ FirstMortgage30 ┆ 242091643657598 ┆ Rejected │\n",
+       "│             ┆ 2893             ┆ 592              ┆ yr              ┆ 6605            ┆          │\n",
+       "│ 0.144221    ┆ 6152300864202120 ┆ 8482240621400855 ┆ FirstMortgage30 ┆ 708144341937619 ┆ Accepted │\n",
+       "│             ┆ 458              ┆ 154              ┆ yr              ┆ 1576            ┆          │\n",
+       "│ 0.679745    ┆ 7382869047936979 ┆ 1705787624786035 ┆ MoneyMarketSavi ┆ 242091643657598 ┆ Rejected │\n",
+       "│             ┆ 708              ┆ 8620             ┆ ngsAccount      ┆ 6605            ┆          │\n",
+       "│ 1.0         ┆ 1483459495520256 ┆ 8482240621400855 ┆ BasicChecking   ┆ 242091643657598 ┆ Rejected │\n",
+       "│             ┆ 9655             ┆ 154              ┆                 ┆ 6605            ┆          │\n",
+       "│ 0.718982    ┆ 1441642292653037 ┆ 6528012409336537 ┆ BasicChecking   ┆ 104444790796108 ┆ Accepted │\n",
+       "│             ┆ 5948             ┆ 592              ┆                 ┆ 98409           ┆          │\n",
+       "│ 0.0         ┆ 1586870557534238 ┆ 1705787624786035 ┆ UPlusFinPersona ┆ null            ┆ Rejected │\n",
+       "│             ┆ 224              ┆ 8620             ┆ l               ┆                 ┆          │\n",
+       "│ 0.02333     ┆ 3207685404928967 ┆ 1705787624786035 ┆ BasicChecking   ┆ 104444790796108 ┆ Rejected │\n",
+       "│             ┆ 890              ┆ 8620             ┆                 ┆ 98409           ┆          │\n",
+       "└─────────────┴──────────────────┴──────────────────┴─────────────────┴─────────────────┴──────────┘"
+      ]
+     },
+     "execution_count": 12,
+     "metadata": {},
+     "output_type": "execute_result"
+    }
+   ],
+   "source": [
+    "pl.read_ipc('output/hds.arrow')"
+   ]
+  }
+ ],
+ "metadata": {
+  "kernelspec": {
+   "display_name": "Python 3",
+   "language": "python",
+   "name": "python3"
+  },
+  "language_info": {
+   "codemirror_mode": {
+    "name": "ipython",
+    "version": 3
+   },
+   "file_extension": ".py",
+   "mimetype": "text/x-python",
+   "name": "python",
+   "nbconvert_exporter": "python",
+   "pygments_lexer": "ipython3",
+   "version": "3.11.0"
+  },
+  "orig_nbformat": 4,
+  "vscode": {
+   "interpreter": {
+    "hash": "aee8b7b246df8f9039afb4144a1f6fd8d2ca17a180786b69acc140d282b71a49"
+   }
+  }
+ },
+ "nbformat": 4,
+ "nbformat_minor": 2
+}
diff --git a/python/pdstools/__init__.py b/python/pdstools/__init__.py
index 3e8ae723..1cb862ea 100644
--- a/python/pdstools/__init__.py
+++ b/python/pdstools/__init__.py
@@ -1,5 +1,6 @@
 from .adm.ADMDatamart import ADMDatamart
 from .adm.ADMTrees import ADMTrees, MultiTrees
 from .valuefinder.ValueFinder import ValueFinder
-from .utils import cdh_utils
-from .utils import datasets
\ No newline at end of file
+from .utils import cdh_utils, datasets, hds_utils
+from .utils.hds_utils import Config, DataAnonymization
+from .utils.datasets import CDHSample, SampleTrees, SampleValueFinder
\ No newline at end of file
diff --git a/python/pdstools/utils/hds_utils.py b/python/pdstools/utils/hds_utils.py
new file mode 100644
index 00000000..86959309
--- /dev/null
+++ b/python/pdstools/utils/hds_utils.py
@@ -0,0 +1,426 @@
+import os
+import re
+import glob
+import json
+from pathlib import Path
+import polars as pl
+
+from random import randint
+from typing import Literal, Optional
+from pdstools import ADMDatamart
+
+
+class Config:
+    """Configuration file for the data anonymizer.
+
+    Parameters
+    ----------
+    config_file: str = None
+        An optional path to a config file
+    hdr_folder: Path = "."
+        The path to the hdr files
+    use_datamart: bool = False
+        Whether to use the datamart to infer predictor types
+    datamart_folder: Path = "datamart"
+        The folder of the datamart files
+    output_format: Literal["ndjson", "parquet", "arrow", "csv"] = "ndjson"
+        The output format to write the files in
+    output_folder: Path = "output"
+        The location to write the files to
+    mapping_file: str = "mapping.map"
+        The name of the predictor mapping file
+    mask_predictor_names: bool = True
+        Whether to mask the names of regular predictors
+    mask_context_key_names: bool = True
+        Whether to mask the names of context key predictors
+    mask_ih_names: bool = True
+        Whether to mask the name of Interaction History summary predictors
+    mask_outcome_name: bool = True
+        Whether to mask the name of the outcome column
+    mask_predictor_values: bool = True
+        Whether to mask the values of regular predictors
+    mask_context_key_values: bool = True
+        Whether to mask the values of context key predictors
+    mask_ih_values: bool = True
+        Whether to mask the values of Interaction History summary predictors
+    mask_outcome_values: bool = True
+        Whether to mask the values of the outcomes to binary
+    context_key_label: str = "Context_*"
+        The pattern of names for context key predictors
+    ih_label: str = "IH_*"
+        The pattern of names for Interaction History summary predictors
+    outcome_column: str = "Decision_Outcome"
+        The name of the outcome column
+    positive_outcomes: list = ["Accepted", "Clicked"]
+        Which positive outcomes to map to True
+    negative_outcomes: list = ["Rejected", "Impression"]
+        Which negative outcomes to map to False
+    special_predictors: list = ["Decision_DecisionTime", "Decision_OutcomeTime"]
+        A list of special predictors which are not touched
+    """
+
+    def __init__(
+        self,
+        config_file: Optional[str] = None,
+        hdr_folder: Path = ".",
+        use_datamart: bool = False,
+        datamart_folder: Path = "datamart",
+        output_format: Literal["ndjson", "parquet", "arrow", "csv"] = "ndjson",
+        output_folder: Path = "output",
+        mapping_file: str = "mapping.map",
+        mask_predictor_names: bool = True,
+        mask_context_key_names: bool = True,
+        mask_ih_names: bool = True,
+        mask_outcome_name: bool = True,
+        mask_predictor_values: bool = True,
+        mask_context_key_values: bool = True,
+        mask_ih_values: bool = True,
+        mask_outcome_values: bool = True,
+        context_key_label: str = "Context_*",
+        ih_label: str = "IH_*",
+        outcome_column: str = "Decision_Outcome",
+        positive_outcomes: list = ["Accepted", "Clicked"],
+        negative_outcomes: list = ["Rejected", "Impression"],
+        special_predictors: list = ["Decision_DecisionTime", "Decision_OutcomeTime"],
+    ):
+        self._opts = {key: value for key, value in vars().items() if key != "self"}
+
+        if config_file is not None:
+            self.load_from_config_file(config_file)
+        for key, value in self._opts.items():
+            setattr(self, key, value)
+        self.validate_paths()
+
+    def load_from_config_file(self, config_file: Path):
+        """Load the configurations from a file.
+
+        Parameters
+        ----------
+        config_file : Path
+            The path to the configuration file
+        """
+        if not os.path.exists(config_file):
+            raise ValueError("Config file does not exist.")
+        with open(config_file) as f:
+            self._opts = json.load(f)
+
+    def save_to_config_file(self, file_name: str = None):
+        """Save the configurations to a file.
+
+        Parameters
+        ----------
+        file_name : str
+            The name of the configuration file
+        """
+        with open("config.json" if file_name is None else file_name, "w") as f:
+            json.dump(self._opts, f)
+
+    def validate_paths(self):
+        """Validate the outcome folder exists."""
+        if not os.path.exists(self.output_folder):
+            os.mkdir(self.output_folder)
+
+
+class DataAnonymization:
+    def __init__(
+        self,
+        config: Optional[Config] = None,
+        df: Optional[pl.LazyFrame] = None,
+        datamart: Optional[ADMDatamart] = None,
+        **config_args,
+    ):
+        """Anonymise a historical data recording dataset
+
+        Parameters
+        ----------
+        config : Optional[Config]
+            Override the default configurations with the Config class
+        df : Optional[pl.LazyFrame]
+            Manually supply a Polars lazyframe to anonymize
+        datamart : Optional[ADMDatamart]
+            Manually supply a Datamart file to infer predictor types
+
+        Keyword arguments
+        -----------------
+        See :Class:`.Config`
+        """
+        self.config = Config(**config_args) if config is None else config
+
+        if df is None:
+            self.df = self.load_hdr_files()
+        elif isinstance(df, pl.DataFrame):
+            self.df = df.lazy()
+        elif isinstance(df, pl.LazyFrame):
+            self.df = self.df
+        else:
+            raise ValueError("No data found.")
+        self.df_out = None
+        if self.config.use_datamart:
+            self.predictors_by_type = self.read_predictor_type_from_datamart(
+                config.datamart_folder, datamart
+            )
+        else:
+            self.predictors_by_type = self.read_predictor_type_from_file(self.df)
+
+        (
+            self.symbolic_predictors_to_mask,
+            self.numeric_predictors_to_mask,
+            self.column_mapping,
+        ) = self.get_predictors_mapping()
+
+    def write_to_output(
+        self,
+        df: Optional[pl.DataFrame] = None,
+        ext: Literal["ndjson", "parquet", "arrow", "csv"] = None,
+    ):
+        """Write the processed dataframe to an output file.
+
+        Parameters
+        ----------
+        df : Optional[pl.DataFrame]
+            Dataframe to write.
+            If not provided, runs `self.process()`
+        ext : Literal["ndjson", "parquet", "arrow", "csv"]
+            What extension to write the file to
+
+        """
+        out_filename = f"{self.config.output_folder}/hds"
+        out_format = self.config.output_format if ext is None else ext
+        if df is None:
+            df = self.process()
+        if out_format == "ndjson":
+            df.write_ndjson(f"{out_filename}.json")
+        elif out_format == "parquet":
+            df.write_parquet(f"{out_filename}.parquet")
+        elif out_format == "arrow":
+            df.write_ipc(f"{out_filename}.arrow")
+        else:
+            df.write_csv(f"{out_filename}.csv")
+
+    def create_mapping_file(self):
+        """Create a file to write the column mapping"""
+        with open(self.config.mapping_file, "w") as wr:
+            for key, mapped_val in self.column_mapping.items():
+                wr.write(f"{key}={mapped_val}")
+                wr.write("\n")
+
+    def load_hdr_files(self):
+        """Load the hdr files from the `config.hdr_folder` location."""
+        files = glob.glob(f"{self.config.hdr_folder}/*.json")
+        if len(files) < 1:
+            raise FileNotFoundError(
+                """Files not found. Please check the `hdr_folder` 
+                and check if there are `.json` files in there. 
+                If preferred, you can also supply a polars Dataframe or Lazyframe 
+                directly with the `df` argument."""
+            )
+        out = []
+        for file in files:
+            out.append(pl.scan_ndjson(file))
+
+        df = pl.concat(out, how="diagonal")
+        return df
+
+    @staticmethod
+    def read_predictor_type_from_file(df: pl.LazyFrame):
+        """Infer the types of the preditors from the data.
+
+        This is non-trivial, as it's not ideal to pull in all data to memory for this.
+        For this reason, we sample 1% of data, or all data if less than 50 rows,
+        and try to cast it to numeric. If that fails, we set it to categorical,
+        else we set it to numeric.
+
+        It is technically supported to manually override this, by just overriding
+        the `symbolic_predictors_to_mask` & `numeric_predictors_to_mask` properties.
+
+        Parameters
+        ----------
+        df : pl.LazyFrame
+            The lazyframe to infer the types with
+
+        """
+        types = {}
+
+        import numpy as np
+
+        def sample_it(s: pl.Series) -> pl.Series:
+            out = pl.Series(
+                values=np.random.binomial(1, 0.01, s.len()),
+                dtype=pl.Boolean,
+            )
+            if out.len() < 50:
+                out = pl.Series(values=[True] * out.len(), dtype=pl.Boolean)
+            return out
+
+        df_ = (
+            df.lazy()
+            .with_columns(pl.first().map(sample_it).alias("_sample"))
+            .filter(pl.col("_sample"))
+            .drop("_sample")
+            .collect()
+        )
+
+        for col in df_.columns:
+            try:
+                df_.get_column(col).cast(pl.Float64)
+                types[col] = "numeric"
+            except:
+                types[col] = "symbolic"
+
+        return types
+
+    @staticmethod
+    def read_predictor_type_from_datamart(
+        datamart_folder: Path, datamart: ADMDatamart = None
+    ):
+        """The datamart contains type information about each predictor.
+        This function extracts that information to infer types for the HDS.
+
+        Parameters:
+        datamart_folder : Path
+            The path to the datamart files
+        datamart : ADMDatamart
+            The direct ADMDatamart object
+        """
+        dm = ADMDatamart(path=datamart_folder) if datamart is None else datamart
+        df = pl.DataFrame(dm.predictorData)
+        df = (
+            df.filter(pl.col("EntryType") != "Classifier")
+            .unique(subset="PredictorName")
+            .select(["PredictorName", "Type"])
+        )
+        df_dict = dict(zip(df.get_column("PredictorName"), df.get_column("Type")))
+        df_dict = dict(
+            (key.replace(".", "_"), value) for (key, value) in df_dict.items()
+        )
+        return df_dict
+
+    def get_columns_by_type(self):
+        """Get a list of columns for each type."""
+        columns = self.df.columns
+        columns_ = ",".join(columns)
+
+        exp = "[^,]+"
+        context_keys_t = re.findall(f"{self.config.context_key_label}{exp}", columns_)
+        ih_predictors_t = re.findall(f"{self.config.ih_label}{exp}", columns_)
+
+        special_predictors_t = []
+        for exc in self.config.special_predictors:
+            for e in re.findall(exc, columns_):
+                special_predictors_t.append(e)
+
+        outcome_column_t = [self.config.outcome_column]
+        predictors_t = list(
+            set(columns)
+            - set(
+                context_keys_t
+                + ih_predictors_t
+                + special_predictors_t
+                + outcome_column_t
+            )
+        )
+        return context_keys_t, ih_predictors_t, special_predictors_t, predictors_t
+
+    def get_predictors_mapping(self):
+        """Map the predictor names to their anonymized form."""
+        symbolic_predictors_to_mask, numeric_predictors_to_mask = [], []
+
+        (
+            context_keys_t,
+            ih_predictors_t,
+            special_predictors_t,
+            predictors_t,
+        ) = self.get_columns_by_type()
+        outcome_t = self.config.outcome_column
+
+        outcome_column = {}
+        context_keys = {}
+        ih_predictors = {}
+        predictors = {}
+
+        for idx, val in enumerate(context_keys_t):
+            if self.config.mask_context_key_values:
+                if self.predictors_by_type[val] == "symbolic":
+                    symbolic_predictors_to_mask.append(val)
+                else:
+                    numeric_predictors_to_mask.append(val)
+            context_keys[val] = (
+                f"CK_PREDICTOR_{idx}" if self.config.mask_context_key_names else val
+            )
+
+        for idx, val in enumerate(ih_predictors_t):
+            if self.config.mask_ih_values:
+                if self.predictors_by_type[val] == "symbolic":
+                    symbolic_predictors_to_mask.append(val)
+                else:
+                    numeric_predictors_to_mask.append(val)
+            ih_predictors[val] = (
+                f"IH_PREDICTOR_{idx}" if self.config.mask_ih_names else val
+            )
+
+        for idx, val in enumerate(predictors_t):
+            if self.config.mask_predictor_values:
+                if self.predictors_by_type[val] == "symbolic":
+                    symbolic_predictors_to_mask.append(val)
+                else:
+                    numeric_predictors_to_mask.append(val)
+            predictors[val] = (
+                f"PREDICTOR_{idx}" if self.config.mask_predictor_names else val
+            )
+
+        special_predictors = {x: x for x in special_predictors_t}
+
+        outcome_column[outcome_t] = (
+            "OUTCOME" if self.config.mask_outcome_name else outcome_t
+        )
+
+        column_mapping = (
+            predictors
+            | context_keys
+            | ih_predictors
+            | special_predictors
+            | outcome_column
+        )
+
+        return symbolic_predictors_to_mask, numeric_predictors_to_mask, column_mapping
+
+    def process(self):
+        """Anonymize the dataset."""
+
+        def to_hash(cols):
+            return (
+                pl.when(
+                    (pl.col(cols).is_not_null())
+                    & (pl.col(cols) != "")
+                    & (pl.col(cols).is_in(["true", "false"]).is_not())
+                )
+                .then(pl.col(cols).hash(seed=randint(1, 100)))
+                .otherwise(pl.col(cols))
+            )
+
+        def to_normalize(cols):
+            return (pl.col(cols) - pl.col(cols).min()) / (
+                pl.col(cols).max() - pl.col(cols).min()
+            )
+
+        def to_boolean(col, positives):
+            return (
+                pl.when(pl.col(col).is_in(positives)).then(True).otherwise(False)
+            ).alias(col)
+
+        df = self.df.with_columns(
+            pl.col(self.numeric_predictors_to_mask).cast(pl.Float64)
+        ).with_columns(
+            [
+                to_hash(self.symbolic_predictors_to_mask),
+                to_normalize(self.numeric_predictors_to_mask),
+            ]
+        )
+        if self.config.mask_outcome_values:
+            df = df.with_columns(
+                to_boolean(self.config.outcome_column, self.config.positive_outcomes)
+            )
+        df = (
+            df.select(self.column_mapping.keys()).rename(self.column_mapping)
+        ).collect()
+        return df


Test patch:
diff --git a/python/tests/test_anonymizer.py b/python/tests/test_anonymizer.py
new file mode 100644
index 00000000..f89bb3df
--- /dev/null
+++ b/python/tests/test_anonymizer.py
@@ -0,0 +1,57 @@
+import pytest
+import sys
+import polars as pl
+
+sys.path.append("python")
+from pdstools.utils.hds_utils import Config, DataAnonymization
+
+
+@pytest.fixture
+def sampleInput():
+    """Fixture to serve as input dataframe"""
+    from io import StringIO
+
+    input = StringIO(
+        """{"Context_Name":"FirstMortgage30yr","Customer_MaritalStatus":"Married","Customer_CLV":1460,"Customer_City":"Port Raoul","IH_Web_Inbound_Accepted_pxLastGroupID":"Account","Decision_Outcome":"Rejected"}
+        {"Context_Name":"FirstMortgage30yr","Customer_MaritalStatus":"Unknown","Customer_CLV":669,"Customer_City":"Laurianneshire","IH_Web_Inbound_Accepted_pxLastGroupID":"AutoLoans","Decision_Outcome":"Accepted"}
+        {"Context_Name":"MoneyMarketSavingsAccount","Customer_MaritalStatus":"No Resp+","Customer_CLV":1174,"Customer_City":"Jacobshaven","IH_Web_Inbound_Accepted_pxLastGroupID":"Account","Decision_Outcome":"Rejected"}
+        {"Context_Name":"BasicChecking","Customer_MaritalStatus":"Unknown","Customer_CLV":1476,"Customer_City":"Lindton","IH_Web_Inbound_Accepted_pxLastGroupID":"Account","Decision_Outcome":"Rejected"}
+        {"Context_Name":"BasicChecking","Customer_MaritalStatus":"Married","Customer_CLV":1211,"Customer_City":"South Jimmieshire","IH_Web_Inbound_Accepted_pxLastGroupID":"DepositAccounts","Decision_Outcome":"Accepted"}
+        {"Context_Name":"UPlusFinPersonal","Customer_MaritalStatus":"No Resp+","Customer_CLV":533,"Customer_City":"Bergeville","IH_Web_Inbound_Accepted_pxLastGroupID":null,"Decision_Outcome":"Rejected"}
+        {"Context_Name":"BasicChecking","Customer_MaritalStatus":"No Resp+","Customer_CLV":555,"Customer_City":"Willyville","IH_Web_Inbound_Accepted_pxLastGroupID":"DepositAccounts","Decision_Outcome":"Rejected"}
+        """
+    )
+    return pl.read_ndjson(input)
+
+
+def testDefault(sampleInput):
+    processed = DataAnonymization(df=sampleInput).process()
+    cols = [
+        "PREDICTOR_0",
+        "PREDICTOR_1",
+        "PREDICTOR_2",
+        "CK_PREDICTOR_0",
+        "IH_PREDICTOR_0",
+        "OUTCOME",
+    ]
+    assert processed.columns == cols
+    processed = processed.select(cols)
+    assert processed.select(pl.col(pl.Float64)).to_series().to_list() == [
+        0.9830328738069989,
+        0.14422057264050903,
+        0.679745493107105,
+        1.0,
+        0.71898197242842,
+        0.0,
+        0.02332979851537646,
+    ]
+    assert processed[5, 4] is None
+    assert processed.get_column("OUTCOME").to_list() == [
+        False,
+        True,
+        False,
+        False,
+        True,
+        False,
+        False,
+    ]