Repository: databricks-industry-solutions/auto-data-linkage (language: python)

Issue:
Exercise notebook and loss method
- Added method to utils to calculate loss
- Added notebook for the exercise


Discussion / hints:

Fix patch:
diff --git a/Workshop Materials/README.md b/Workshop Materials/README.md
deleted file mode 100644
index 259e51b..0000000
--- a/Workshop Materials/README.md
+++ /dev/null
@@ -1,11 +0,0 @@
-# splink-public-sector-hackathon
-A hackathon for Entity Resolution using Splink (by MoJ) on Azure Databricks done by NICD, MoJ, Microsoft and Databricks
-
-
-# How to Run
-
-1. Add repository to your Databricks Workspace via Repos. Paste this link when required: https://github.com/databricks-industry-solutions/splink-public-sector-hackathon.git
-
-2. Walk through the *end-to-end example* notebook in the `notebooks` directory to understand how Splink works.
-
-3. Use the *exercise* notebook as a starting point for your own work. Consider exploring different parameters or feature engineering to understand the impact on linking. Visualise the results using Python, R, or the built in visualisation tools in Databricks Notebooks or Databricks SQL.
\ No newline at end of file
diff --git a/Workshop Materials/notebooks/Splink Evaluation - explanation.py b/Workshop Materials/notebooks/Splink Evaluation - explanation.py
deleted file mode 100644
index e583c4d..0000000
--- a/Workshop Materials/notebooks/Splink Evaluation - explanation.py
+++ /dev/null
@@ -1,140 +0,0 @@
-# Databricks notebook source
-pip install git+https://github.com/robertwhiffin/splink.git@mllfow-integration
-
-# COMMAND ----------
-
-import pandas as pd
-import numpy as np
-
-from utils.mlflow_utils import get_match_probabilty_loss
-
-# COMMAND ----------
-
-# MAGIC %md
-# MAGIC ## Evaluating Splink
-# MAGIC 
-# MAGIC Whilst Splink does offer a supervised predictive method, our data does not have ground truth (we don't categorically know which records are duplicates or otherwise), so we're resigned to taking an unsupervised approach.
-# MAGIC 
-# MAGIC Unsupervised models are trickier to evaluate, because we can't compare the predictions against a known truth. How would we know how well our model has performed then? How can we evaluate the impact of changing parameters against the previous experiment?
-# MAGIC 
-# MAGIC We are proposing a loss function that promotes match probabilities within the pairwise prediction dataframe to be close to either 1 or 0 (i.e. the model is "certain" that a pair is a match or not).
-# MAGIC 
-# MAGIC ##### A word of caution
-# MAGIC 
-# MAGIC Please be aware that this is not a hard-and-fast, empirical way of evaluating a Splink model. Unsupervised entity resolution is complex and choosing the right model will depend on the data, use case, domain, expert knowledge, risk appetite, and many more factors.
-
-# COMMAND ----------
-
-# MAGIC %md
-# MAGIC ### Devising an evaluation metric
-# MAGIC 
-# MAGIC We need a metric that can compare the predictions of different models against one another. We know that the Splink model creates a pairwise dataframe (within our defined comparison space) and each pair is assigned a probability between 0 and 1.
-# MAGIC 
-# MAGIC Suppose that we have predicted 10,000 pairs of records' probability of matching and we have three models that made these predicitons:
-
-# COMMAND ----------
-
-# MAGIC %md
-# MAGIC #### Model 1: The Perfect model
-# MAGIC 
-# MAGIC Our first model is perfect, therefore it predicted the pairs to be either 100% a match \\(Pr(match)=1\\) or 100% not a match \\(Pr(match)=0\\), so the distribution of the predicted probabilities looks something like the chart below.
-# MAGIC 
-# MAGIC We would want to score this model as high as possible (or give it a loss as low as possible).
-
-# COMMAND ----------
-
-test_data_perfect = pd.DataFrame({
-  "match_probability": np.append(np.repeat([0.0], 5000), np.repeat([1.0], 5000))
-})
-
-test_data_perfect["match_probability"].hist(bins=50)
-
-# COMMAND ----------
-
-# MAGIC %md
-# MAGIC #### Model 2: The Worst model
-# MAGIC 
-# MAGIC What would be the worst case scenario? If the model was unsure about _every_ pair, it might predict everything to be 50%, i.e. \\(Pr(match)=0.5\\), so the distribution would look something like the chart below.
-# MAGIC 
-# MAGIC We would want to score this model low (or give it a high loss).
-
-# COMMAND ----------
-
-test_data_worst = pd.DataFrame({
-  "match_probability": np.repeat([0.5], 10000)
-})
-
-test_data_worst["match_probability"].hist(bins=50)
-
-# COMMAND ----------
-
-# MAGIC %md
-# MAGIC #### Model 3: A realistically good model
-# MAGIC 
-# MAGIC What would our predictions looks like in reality? Probably some minor % of predicitons distributed near the 100% probability mark (as we expect there to be fewer matches than not), and the majority distributed near the 0% probability mark.
-# MAGIC 
-# MAGIC We would want to score this higher than the worst and lower than the perfect.
-
-# COMMAND ----------
-
-test_data_realistic = pd.DataFrame({
-  "match_probability": np.array([0.0 if x<=0.0 else 1.0 if x>=1.0 else x for x in np.append(np.random.normal(0.1, 0.01, 9500), np.random.normal(0.9, 0.01, 500))])
-})
-
-test_data_realistic["match_probability"].hist(bins=50)
-
-# COMMAND ----------
-
-# MAGIC %md
-# MAGIC #### The proposed loss metric
-# MAGIC 
-# MAGIC Since we know what our edge cases (and a case somewhere in the middle) look like, we can attempt to assign a metric to these probability distributions:
-# MAGIC 
-# MAGIC 1. We fit 2 normal distributions to the predictions (since we expect two peaks around 0 and 1) with a Gaussian Mixture model
-# MAGIC 2. We estimate the mean and the standard deviations of these two distributions
-# MAGIC 3. Our loss function is then the distance of the distributions' means from 0 and 1, respectively, and the size of their standard deviations, i.e.:
-# MAGIC 
-# MAGIC \\(\mathcal{L}=\mu_{1}+\sigma_1+(1-\mu_{2})+\sigma_{2}\\)
-# MAGIC 
-# MAGIC Intuitively, the further away the average positive or negative prediction is from 0 and 1, and the wider the spread of the positive and negative predictions, the higher the loss, and the worse our model.
-# MAGIC 
-# MAGIC We have implemented this loss function in the `get_match_probability_loss` method, which we imported above.
-# MAGIC 
-# MAGIC Let's see how this would work on our test data:
-
-# COMMAND ----------
-
-# MAGIC %md
-# MAGIC #### Model 1 Loss
-
-# COMMAND ----------
-
-loss_perfect, fig_perfect = get_match_probabilty_loss(test_data_perfect)
-print(f"Loss for perfect scenario: {loss_perfect}")
-
-# COMMAND ----------
-
-# MAGIC %md
-# MAGIC #### Model 2 Loss
-
-# COMMAND ----------
-
-loss_worst, fig_worst = get_match_probabilty_loss(test_data_worst)
-print(f"Loss for worst scenario: {loss_worst}")
-
-# COMMAND ----------
-
-# MAGIC %md
-# MAGIC #### Model 3 Loss
-
-# COMMAND ----------
-
-loss_realistic, fig_realistic = get_match_probabilty_loss(test_data_realistic)
-print(f"Loss for realistic scenario: {loss_realistic}")
-
-# COMMAND ----------
-
-# MAGIC %md
-# MAGIC The loss for our perfect, worst and realistic scenarios was \\(0.002\\), \\(1.502\\) and \\(0.220\\), respectively
-# MAGIC 
-# MAGIC _NB: in reality, this metric would probably need more consideration and there are certainly edge cases or parameter impact that has not been taken into consideration. Regardless, we will proceed with this for the purpose of this toy exercise._
diff --git a/Workshop Materials/notebooks/Splink in Databricks - Exercise.py b/Workshop Materials/notebooks/Splink in Databricks - Exercise.py
deleted file mode 100644
index a63430e..0000000
--- a/Workshop Materials/notebooks/Splink in Databricks - Exercise.py
+++ /dev/null
@@ -1,273 +0,0 @@
-# Databricks notebook source
-# MAGIC %md
-# MAGIC # Splink on Databricks - Exercise
-
-# COMMAND ----------
-
-# MAGIC %pip install splink --quiet
-
-# COMMAND ----------
-
-# MAGIC %md
-# MAGIC ## Load Libraries
-
-# COMMAND ----------
-
-from splink.spark.spark_linker import SparkLinker
-from splink.databricks.enable_splink import enable_splink
-import splink.spark.spark_comparison_library as cl
-
-
-import pandas as pd
-
-from IPython.display import IFrame
-
-from utils.splink_linker_model import SplinkLinkerModel
-from utils.mlflow_utils import *
-from utils.mlflow import splink_mlflow
-
-import mlflow
-
-# COMMAND ----------
-
-# MAGIC %md
-# MAGIC Set parameters based on active user
-
-# COMMAND ----------
-
-username = spark.sql('select current_user() as user').collect()[0]['user']
-db_name = username.replace(".", "_").replace("@", "_").replace("-","_")
-db_name
-
-# COMMAND ----------
-
-spark.sql(f"CREATE DATABASE IF NOT EXISTS {db_name}")
-spark.sql(f"USE {db_name}")
-
-# COMMAND ----------
-
-# MAGIC %md
-# MAGIC ## Evaluating Splink
-# MAGIC 
-# MAGIC Whilst Splink does offer a supervised predictive method, our data does not have ground truth (we don't categorically know which records are duplicates or otherwise), so we're resigned to taking an unsupervised approach.
-# MAGIC 
-# MAGIC Unsupervised models are trickier to evaluate, because we can't compare the predictions against a known truth. How would we know how well our model has performed then? How can we evaluate the impact of changing parameters against the previous experiment?
-# MAGIC 
-# MAGIC We are proposing a loss function that promotes match probabilities within the pairwise prediction dataframe to be close to either 1 or 0 (i.e. the model is "certain" that a pair is a match or not). For a full explanation with examples of this proposed evaluation metric, please see the notebook _Splink Evaluation - explanation_ in the same directory as this one, but for the purpose of this exercise, all we need to know is that we want to __decrease the loss__ with our predictions.
-# MAGIC 
-# MAGIC You can use the `get_match_probabilty_loss` method on the prediction dataframe generated by different models to compare them. The lower the loss, the better the model.
-# MAGIC 
-# MAGIC ***
-# MAGIC 
-# MAGIC ##### A word of caution
-# MAGIC 
-# MAGIC Please be aware that this is not a hard-and-fast, empirical way of evaluating a Splink model. Unsupervised entity resolution is complex and choosing the right model will depend on the data, use case, domain, expert knowledge, risk appetite, and many more factors.
-
-# COMMAND ----------
-
-# MAGIC %md
-# MAGIC ## Getting the data (again)
-# MAGIC 
-# MAGIC And retrieve the data we should already be familiar with.
-
-# COMMAND ----------
-
-# MAGIC %run
-# MAGIC ../setup/data-setup
-
-# COMMAND ----------
-
-table_name = "companies_with_officers"
-table_path = f"/mnt/source/splinkdata/delta/{table_name}"
-data = spark.read.load(table_path)
-
-# COMMAND ----------
-
-# MAGIC %md
-# MAGIC ## Building a baseline model
-# MAGIC 
-# MAGIC Lastly, we'll build a baseline model (using essentially the set-up of the previous demonstration), and log it as the first run in an MLFlow experiment.
-
-# COMMAND ----------
-
-# MAGIC %md
-# MAGIC 
-# MAGIC We'll set an experiment based on the user ID generated at the start. You can navigate to it via
-# MAGIC 
-# MAGIC **Left sidebar --> Workspace --> Shared --> \<your-user-name\>_databricks_experiment**
-
-# COMMAND ----------
-
-mlflow.set_experiment(f"/Shared/{db_name}_experiment")
-
-# COMMAND ----------
-
-# MAGIC %md
-# MAGIC ### Parameters
-# MAGIC 
-# MAGIC Let's set the parameters as in the demonstration notebook.
-
-# COMMAND ----------
-
-blocking_rules_to_generate_predictions = [
-  "l.name = r.name and l.date_of_birth = r.date_of_birth",
-  "l.nationality = r.nationality and l.locality = r.locality and l.premises = r.premises and l.region = r.region",
-  "l.address_line_1 = r.address_line_1 and l.postal_code = r.postal_code and l.surname = r.surname",
-]
-
-comparisons = [
-  cl.levenshtein_at_thresholds("surname", 10),
-  cl.levenshtein_at_thresholds("forenames", 10),
-  cl.levenshtein_at_thresholds("address_line_1", 5),
-  cl.levenshtein_at_thresholds("country_of_residence", 1),
-  cl.levenshtein_at_thresholds("nationality", 2)
-]
-
-deterministic_rules = [
-  "l.name = r.name and levenshtein(r.date_of_birth, l.date_of_birth) <= 1",
-  "l.address_line_1 = r.address_line_1 and levenshtein(l.name, r.name) <= 5",
-  "l.name = r.name and levenshtein(l.address_line_1, r.address_line_1) <= 5",
-]
-
-settings = {
-  "retain_intermediate_calculation_columns": True,
-  "retain_matching_columns": True,
-  "link_type": "dedupe_only",
-  "unique_id_column_name": "uid",
-  "comparisons": comparisons,
-  "em_convergence": 0.01,
-  "blocking_rules_to_generate_predictions": blocking_rules_to_generate_predictions
-}
-
-training_rules = [
-  "l.name = r.name and l.date_of_birth = r.date_of_birth",
-  "l.surname = r.surname and l.forenames = r.forenames"
-]
-
-# COMMAND ----------
-
-# MAGIC %md
-# MAGIC ### Runnig a trial
-# MAGIC 
-# MAGIC Let's do an initial run with the above parameters. Notice that this time we're making predictions inside the run as well to also log our loss and the plot generated inside the run.
-# MAGIC 
-# MAGIC The run should appear in your experiment (you may need to refresh if you have the tab open), or you can navigate to it via the hyperlink provided after the cell has started running.
-
-# COMMAND ----------
-
-# Start MLflow run
-with mlflow.start_run() as run:
-  # Retrieve run ID
-  RUN_ID = run.info.run_id
-
-  # Train linker model
-  linker = SparkLinker(data, spark=spark)
-  linker.initialise_settings(settings)
-  linker.estimate_probability_two_random_records_match(deterministic_rules, recall=0.8)
-  linker.estimate_u_using_random_sampling(target_rows=1e7)
-  for training_rule in training_rules:
-    linker.estimate_parameters_using_expectation_maximisation(training_rule)
-  
-  # Log model and parameters
-  splink_mlflow.log_splink_model_to_mlflow(linker, "linker")
-
-  # Make predictions
-  predictions = linker.predict()
-  
-  # Evaluate linker model and log loss as metric
-  loss, fig = get_match_probabilty_loss(predictions.as_pandas_dataframe())
-  mlflow.log_metric("match_probability_loss", loss)
-  mlflow.log_figure(fig, 'prediction_distribution_loss.png')
-
-# COMMAND ----------
-
-# MAGIC %md
-# MAGIC 
-# MAGIC Splink gives us the likelihood that a pair of records are the same, and also the clusters that are created by following the chain of pairs. If A&B are the same, and B&C are the same, then A&B&C are the same. This is a cluster.
-
-# COMMAND ----------
-
-# pairwise predictions
-predictions.as_pandas_dataframe(limit=10)
-
-# COMMAND ----------
-
-# Creating the clusters requires setting a confidence threshold to indicate the degree of similarity between pairs. 
-# If A&B have a 90% chance of the being the sae, and B&C a 10% chance, we don't want to connect A&B&C.
-
-clusters = linker.cluster_pairwise_predictions_at_threshold(predictions, threshold_match_probability=0.5)
-clusters.as_pandas_dataframe(limit=10)
-
-# COMMAND ----------
-
-# MAGIC %md
-# MAGIC ### Using the trained model
-# MAGIC 
-# MAGIC In the above MLflow run, we created and trained a `model` object. We can use this object directly for further predictions, or to make use of Splink's many built-in exploratory tools.
-
-# COMMAND ----------
-
-linker.m_u_parameters_chart()
-
-# COMMAND ----------
-
-# MAGIC %md
-# MAGIC You can also call the `get_match_probabilty_loss` method directly on the predictions (as long as it's a Pandas dataframe) at any point to check your predictions.
-
-# COMMAND ----------
-
-baseline_loss, baseline_fig = get_match_probabilty_loss(predictions.as_pandas_dataframe())
-
-baseline_loss
-
-# COMMAND ----------
-
-path=f"/dbfs/Users/{username}/clusters.html"
-
-
-linker.cluster_studio_dashboard(predictions, clusters, path, sampling_method="by_cluster_size", overwrite=True)
-
-
-with open(path, "r") as f:
-    html2=f.read()
-    
-displayHTML(html2)
-
-# COMMAND ----------
-
-path=f"/dbfs/Users/{username}/scv.html"
-
-linker.comparison_viewer_dashboard(predictions, path, overwrite=True)
-
-with open(path, "r") as f:
-    html=f.read()
-    
-displayHTML(html)
-
-# COMMAND ----------
-
-# MAGIC %md
-# MAGIC To visualise clusters together, you can use the `visualise_linked_records` for a static (but less readable) picture of the connected records.
-# MAGIC 
-# MAGIC Use the `linkage_threshold` argument to set the minimum match probability, and the `min_linked_records` argument to set a minimum number of records in a cluster for it to be visualised. Avoid setting these too low or the visualisation will be unreadable.
-
-# COMMAND ----------
-
-visualise_linked_records(predictions.as_pandas_dataframe(), min_linked_records=10, linkage_threshold=0.9)
-
-# COMMAND ----------
-
-# MAGIC %md
-# MAGIC ## Your Turn
-# MAGIC 
-# MAGIC You now have a baseline model which no doubt could use some improvement, which will be up to you to make.
-# MAGIC 
-# MAGIC Suggestions for work:
-# MAGIC 1. Explore the data with Spark/Pandas to better understand the contents of the columns - use Splink's built in profilers to help you.
-# MAGIC 2. Use your intuition, your understanding of the data and how the distance functions work to devise a new set or sets of parameters that make more sense given the task.
-# MAGIC 3. Train new models and use Splink's built-in visualisers to explore how your predictions were made.
-# MAGIC 4. Calculate the loss and compare it against the baseline or previous runs.
-# MAGIC 5. Submit new runs to your experiment. Track how the loss evolves with your different trials on the experiments page.
-# MAGIC 6. Try and de-duplicate your data based on the clusters that you find.
-# MAGIC 6. **If you need any help at all, reach out to a TA!**
-# MAGIC 7. Prepare 1 slide for show and tell at the end of the hack.
diff --git a/Workshop Materials/notebooks/Splink in Databricks - Expert Answer.py b/Workshop Materials/notebooks/Splink in Databricks - Expert Answer.py
deleted file mode 100644
index aed1b54..0000000
--- a/Workshop Materials/notebooks/Splink in Databricks - Expert Answer.py
+++ /dev/null
@@ -1,449 +0,0 @@
-# Databricks notebook source
-# MAGIC %md
-# MAGIC # Splink on Databricks - an "expert" solution
-# MAGIC Not the only (or optimal) solution, but one based on how the author would typically approach the problem.
-
-# COMMAND ----------
-
-# MAGIC %pip install splink --quiet
-
-# COMMAND ----------
-
-# MAGIC %md
-# MAGIC ## Load Libraries
-# MAGIC 
-# MAGIC **Note the addition of the comparison _level_ library to help build custom comparisons with multiple different levels.**
-
-# COMMAND ----------
-
-from splink.spark.spark_linker import SparkLinker
-from splink.databricks.enable_splink import enable_splink
-import splink.spark.spark_comparison_library as cl
-import splink.spark.spark_comparison_level_library as cll
-
-
-import pandas as pd
-
-from IPython.display import IFrame
-
-from utils.splink_linker_model import SplinkLinkerModel
-from utils.mlflow_utils import *
-from utils.mlflow import splink_mlflow
-
-import mlflow
-
-# COMMAND ----------
-
-# MAGIC %md
-# MAGIC Set parameters based on active user
-
-# COMMAND ----------
-
-username = spark.sql('select current_user() as user').collect()[0]['user']
-db_name = username.replace(".", "_").replace("@", "_").replace("-","_")
-db_name
-
-# COMMAND ----------
-
-spark.sql(f"CREATE DATABASE IF NOT EXISTS {db_name}")
-spark.sql(f"USE {db_name}")
-
-# COMMAND ----------
-
-# MAGIC %md
-# MAGIC ## Getting the data (again)
-# MAGIC 
-# MAGIC And retrieve the data we should already be familiar with.
-
-# COMMAND ----------
-
-# MAGIC %run
-# MAGIC ../setup/data-setup
-
-# COMMAND ----------
-
-table_name = "companies_with_officers"
-table_path = f"/mnt/source/splinkdata/delta/{table_name}"
-data = spark.read.load(table_path)
-
-# COMMAND ----------
-
-# MAGIC %md
-# MAGIC ## Our Turn
-# MAGIC 
-# MAGIC We now have a baseline model which no doubt could use some improvement. We approach this as follows:
-# MAGIC 
-# MAGIC 1. **Nodes** (input records) - Explore the data with splink's `missingness` and `profile_columns` methods - this is an iterative process to identify transformations to make to the input data:
-# MAGIC     - Data cleaning/standardisation - making sure all values in a column are formatted the same way, handling nulls or anomalies
-# MAGIC     - Feature engineering - creating new derived columns that could be useful for blocking or making custom comparisons
-# MAGIC 2. **Model training** - Having developed an understanding of the data:
-# MAGIC     - Refine blocking rules for training and predictions
-# MAGIC     - Assemble comparison columns (one for each identifier or attribute), with multiple comparison levels covering potential observations in the data (distinct ways matching records _might_ differ)
-# MAGIC     - Check model is self-consistent using `linker.parameter_estimate_comparisons_chart()`
-# MAGIC     - Check model makes intuitive sense using `linker.match_weights_chart()`
-# MAGIC 3. **Edges** - `linker.predict()` generates match scores between pairs of nodes (i.e. "edges"):
-# MAGIC     - How are the match scores distributed? Expect a bimodal distribution (definite matches + definite non-matches)
-# MAGIC     - If there are ambiguous scores (0.1 < `match_proability` < 0.9; `match_weight` close to zero) we can use the waterfall chart to investigate
-# MAGIC 6. **Clusters** - once we're happy with our model, we probably have a rough idea of what threshold match score to use to define a match (e.g. >0.9). Spot checking a sample of clusters with `cluster_studio_dashboard` we can check for a couple of things, the biggest risk being due to false positives (linking records that aren't genuine matches):
-# MAGIC     - Are there any "hairball" clusters that have become overly large because several genuine clusters have been merged?
-# MAGIC     - Ideal clusters will be fully-connected (every node is directly linked to every other) - are there clusters that appear to be two or more subclusters only connected by a single (suspicious) edge?
-
-# COMMAND ----------
-
-data.toPandas().head()
-
-# COMMAND ----------
-
-# MAGIC %md
-# MAGIC ## 1. Nodes
-# MAGIC > **Explore the data with splink's `missingness` and `profile_columns` methods - this is an iterative process to identify transformations to make to the input data:**
-# MAGIC > - **Data cleaning/standardisation - making sure all values in a column are formatted the same way, handling nulls or anomalies**
-# MAGIC > - **Feature engineering - creating new derived columns that could be useful for blocking or making custom comparisons**
-# MAGIC 
-# MAGIC 
-# MAGIC Initial ideas for standardisation:
-# MAGIC - Clean up `country_of_residence` and `nationality`
-# MAGIC - Create `year_of_birth` for blocking/fuzzy comparisons
-# MAGIC - Split forenames into `forename1`, `forename2` etc.
-# MAGIC - Create `full_address` to include `premises` + `address_line_1` + `locality` + `region` + `postal_code`
-# MAGIC - Set all string columns to upper case - string comparator functions are typically case-sensitive
-# MAGIC - Create partial postcode columns (`area`, `district`, `sector`) - see image [here](https://user-images.githubusercontent.com/7570107/136946496-8769b06c-e4a6-488d-95f1-526946d96aa7.png)
-
-# COMMAND ----------
-
-import pyspark.sql.functions as f
-
-# STANDARDISED DATA
-
-# Names
-data_std = data\
-    .withColumn("forename1", f.split(f.upper(data['forenames']), ' ').getItem(1))\
-    .withColumn("forename2", f.split(f.upper(data['forenames']), ' ').getItem(2))\
-    .withColumn("forename3", f.split(f.upper(data['forenames']), ' ').getItem(3))\
-    .withColumn("forename4", f.split(f.upper(data['forenames']), ' ').getItem(4))\
-    .withColumn("names_arr", f.expr("array_sort(array(forename1, forename2, surname))"))
-
-# DOB
-data_std = data_std.withColumn("year_of_birth", f.expr("CAST(left(date_of_birth, 4) as int)"))
-
-# Postcodes
-data_std = data_std\
-    .withColumn("postcode_validated", f.expr("CASE WHEN postal_code RLIKE '^[A-Z]{1,2}\\\\d{1,2}[A-Z]?\\\\s?\\\\d[A-Z]{2}' THEN postal_code ELSE NULL END"))\
-    .withColumn("pc_sector", f.expr("REGEXP_EXTRACT(postcode_validated, '^[A-Z]{1,2}\\\\d{1,2}[A-Z]?\\\\s?\\\\d', 0)"))\
-    .withColumn("pc_district", f.expr("REGEXP_EXTRACT(postcode_validated, '^[A-Z]{1,2}\\\\d{1,2}', 0)"))\
-    .withColumn("pc_area", f.expr("REGEXP_EXTRACT(postcode_validated, '^[A-Z]{1,2}', 0)"))\
-    .withColumn("full_address", f.expr("CONCAT_WS(' ', premises, address_line_1, locality, region, postal_code)"))
-
-# Country
-data_std = data_std.withColumn(
-    "country_clean", 
-    f.expr("""CASE 
-    WHEN country_of_residence IN ('England', 'Scotland', 'Wales', 'Northern Ireland', 'Great Britain') THEN 'United Kingdom'
-    WHEN regexp_like(country_of_residence, '^(Uk$|Uk\\\\s|U.+ingdom)') THEN 'United Kingdom'
-    WHEN regexp_like(country_of_residence, 'shire$') THEN 'United Kingdom'
-    WHEN regexp_like(country_of_residence, '^(Usa|United States)') THEN 'United States'
-    ELSE country_of_residence END"""))
-
-# Nationality
-data_std = data_std.withColumn(
-    "nationality_clean", 
-    f.expr("""CASE 
-    WHEN nationality IN ('English', 'Scottish', 'Welsh', 'Northern Irish', 'United Kingdom') THEN 'British'
-    WHEN regexp_like(nationality, 'British|Gb$|Uk$') THEN 'British'
-    WHEN nationality IN ('Usa', 'United States', 'Us Citizen', 'Us') THEN 'American'
-    WHEN nationality = 'Other' THEN NULL
-    ELSE nationality END"""))
-
-
-# BUG WORKAROUND - run before creating a new linker to make sure it updates properly
-spark.sql("DROP TABLE IF EXISTS __splink__df_concat_with_tf")
-
-linker = SparkLinker(data_std, spark=spark)
-
-linker.profile_columns("postcode_validated", top_n=30, bottom_n=10)
-
-# COMMAND ----------
-
-# MAGIC %md
-# MAGIC Some of these changes may be overkill, and others could be more extensive if necessary.
-# MAGIC 
-# MAGIC `nationality_clean` for example has reduced the number of distinct values from 652 to 482, helping maximise the chance of finding a match on nationality.
-# MAGIC 
-# MAGIC For the `forename` values however, we can see from the missingness chart below that `forename3` and `forename4` are rarely populated, so we probably won't want to use these new columns.
-
-# COMMAND ----------
-
-linker.missingness_chart()
-
-# COMMAND ----------
-
-linker.completeness_chart()
-
-# COMMAND ----------
-
-# MAGIC %md
-# MAGIC ## 2. Model training
-# MAGIC >**Having developed an understanding of the data:**
-# MAGIC >- **Refine blocking rules for training and predictions**
-# MAGIC >- **Assemble comparison columns (one for each identifier or attribute), with multiple comparison levels covering potential observations in the data (distinct ways matching records _might_ differ)**
-# MAGIC >- **Check model is self-consistent using `linker.parameter_estimate_comparisons_chart()`**
-# MAGIC >- **Check model makes intuitive sense using `linker.match_weights_chart()`**
-# MAGIC 
-# MAGIC Broadly, the data contains just 4 attributes for each person:
-# MAGIC 1. Name
-# MAGIC 2. Address (incl. `country_of_residence`?)
-# MAGIC 3. DOB
-# MAGIC 4. Nationality
-
-# COMMAND ----------
-
-# Let's relax these (seeing as the dataset is relatively small),
-# or simplify them using some of our new derived columns
-blocking_rules_to_generate_predictions = [
-  "l.names_arr = r.names_arr",
-  "l.forename1 = r.forename1 and l.date_of_birth = r.date_of_birth",
-  "l.full_address = r.full_address",
-  "l.year_of_birth = r.year_of_birth and l.postal_code = r.postal_code",
-]
-
-# These are only used to roughly estimate the lambda parameter (probablity of two random records matching)
-# No need to change as they have little impact on the relative match weights
-deterministic_rules_to_estimate_lambda = [
-  "l.name = r.name and levenshtein(r.date_of_birth, l.date_of_birth) <= 1",
-  "l.address_line_1 = r.address_line_1 and levenshtein(l.name, r.name) <= 5",
-  "l.name = r.name and levenshtein(l.address_line_1, r.address_line_1) <= 5",
-]
-
-# We want our training rules to be completely independent
-# 1 - First we block on full name to estimate the parameters for all other columns
-# 2 - Then we block on something else (address) so we can estimate the parameters for full name and DOB
-# 3 - Then we can try a third model to double check all parameters
-blocking_rules_for_model_training = [
-  "l.name = r.name",
-  "l.full_address = r.full_address",
-  "l.date_of_birth = r.date_of_birth and l.nationality_clean = r.nationality_clean"
-]
-
-# COMMAND ----------
-
-# MAGIC %md
-# MAGIC The list of comparisons is where we define the complexity of the (Fellegi-Sunter) linkage model that we are fitting. Everything else helps us estimate the best possible parameters for our chosen model, but this is where we make sure the model is sufficiently sophisticated.
-# MAGIC 
-# MAGIC Some of our comparisons will be off-the-shelf functions from Splink, others can be customised to varying degrees as shown.
-
-# COMMAND ----------
-
-# Name columns are correlated so ideally we combine them into one comparison. 
-# Here we check for a few possible observations:
-# - All names match
-# - Names are jumbled up [REMOVED AS NEVER OBSERVED]
-# - One or more names match (but not necessarily in the same column)
-# - One or more name columns is a fuzzy match
-# - Else, no names match
-name_comparison = {
-    "output_column_name": "name",
-    "comparison_levels": [
-        cll.null_level("name"),
-        cll.exact_match_level("name"),
-        cll.jaro_winkler_level("name", 0.95),
-#         {
-#             "sql_condition": "names_arr_l = names_arr_r",
-#             "label_for_charts": "Exact names (shuffled)"
-#         },
-        {
-            "sql_condition": "arrays_overlap(names_arr_l, names_arr_r)",
-            "label_for_charts": "Any names match"
-        },
-        {
-            "sql_condition": """
-                (jaro_winkler(forename1_l, forename1_r) > 0.9) 
-                OR (jaro_winkler(forename2_l, forename2_r) > 0.9) 
-                OR (jaro_winkler(surname_l, surname_r) > 0.9)
-                """,
-            "label_for_charts": "Jaro-Winkler any name"
-        },
-        cll.else_level()
-    ]
-}
-
-# Similarly, we treat address as a single attribute rather than several correlated ones
-# If the full address doesn't match, we check the postcode, then larger geographical areas
-# Some of these will be considerably more common than others so we turn on TF adjustments
-address_comparison = {
-    "output_column_name": "address",
-    "comparison_levels": [
-        cll.null_level("full_address"),
-        cll.exact_match_level("full_address"),
-        cll.jaro_winkler_level("full_address", 0.95),
-        cll.exact_match_level("postcode_validated", term_frequency_adjustments=True, include_colname_in_charts_label=True),
-        cll.exact_match_level("pc_sector", term_frequency_adjustments=True, include_colname_in_charts_label=True),
-        cll.exact_match_level("pc_district", term_frequency_adjustments=True, include_colname_in_charts_label=True),
-        cll.exact_match_level("pc_area", term_frequency_adjustments=True, include_colname_in_charts_label=True),
-        cll.exact_match_level("country_clean", term_frequency_adjustments=True, include_colname_in_charts_label=True),
-        cll.else_level()
-    ]
-}
-
-# DOB is distributed pretty uniformly so no need for TF adjustments
-dob_comparison = {
-    "output_column_name": "dob",
-    "comparison_levels": [
-        cll.null_level("date_of_birth"),
-        cll.exact_match_level("date_of_birth", include_colname_in_charts_label=True),
-        cll.exact_match_level("year_of_birth", include_colname_in_charts_label=True),
-        cll.else_level()
-    ]
-}
-
-comparisons = [
-    name_comparison,
-    address_comparison,
-    dob_comparison,
-    #cl.exact_match("country_clean", term_frequency_adjustments=True),
-    cl.exact_match("nationality_clean", term_frequency_adjustments=True)
-]
-
-# COMMAND ----------
-
-settings = {
-  "retain_intermediate_calculation_columns": True,
-  "retain_matching_columns": True,
-  "link_type": "dedupe_only",
-  "unique_id_column_name": "uid",
-  "comparisons": comparisons,
-  "em_convergence": 0.01,
-  "blocking_rules_to_generate_predictions": blocking_rules_to_generate_predictions
-}
-
-linker.initialise_settings(settings)
-
-# Estimate lambda
-linker.estimate_probability_two_random_records_match(deterministic_rules_to_estimate_lambda, recall=0.8)
-
-# Estimate u-probabilities
-linker.estimate_u_using_random_sampling(target_rows=1e7)
-
-# Estimate m-probabilities
-for training_rule in blocking_rules_for_model_training:
-    linker.estimate_parameters_using_expectation_maximisation(training_rule)
-
-# COMMAND ----------
-
-# We have estimate the m probabilities 2 or 3 times with different blocking rules
-# In theory, these should be the same - if they're VERY different, it's a sign something is wrong
-linker.parameter_estimate_comparisons_chart(include_u=False)
-
-# COMMAND ----------
-
-# MAGIC %md
-# MAGIC Our m-probability estimates from the different training models are broadly in agreement, except for the `name` comparison - this is likely due to some correlation between a match on name and the blocking rules.
-# MAGIC 
-# MAGIC The final model will take the harmonic mean of each of these parameters, so should settle on a reasonable final value even where there are these inconsistencies. The `match_weights_chart` will help us check our model seems sensible by telling us how much each comparison level adjusts our evidence for or against a match:
-# MAGIC - The "prior match weight" is the starting point for any record comparison - if we know nothing else about these records, there is a very low probability of them being a match
-# MAGIC - For each comparison level, the match weight can be positive (green), negative (red) and the larger the match weight, the more important it is for establishing a match. For example:
-# MAGIC   - An exact match on name is very strong evidence for a match ✅
-# MAGIC   - A match on postcode area is less significant evidence for a match ✅
-# MAGIC   - No match on DOB is evidence against a match ❌
-# MAGIC   - A mismatch on nationality is very strong evidence against a match ❌
-
-# COMMAND ----------
-
-linker.match_weights_chart()
-
-# COMMAND ----------
-
-# MAGIC %md
-# MAGIC ## 3. Edges
-# MAGIC >**`linker.predict()` generates match scores between pairs of nodes (i.e. "edges"):**
-# MAGIC > - **How are the match scores distributed? Expect a bimodal distribution (definite matches + definite non-matches)**
-# MAGIC > - **If there are ambiguous scores** (0.1 < `match_probability` < 0.8 ; `match_weight` close to zero) **we can use the waterfall chart to investigate**
-# MAGIC 
-# MAGIC Before we run the prediction, we can check that our blocking rules aren't going to produce an impractical number of edges. For 150k records, we probably wouldn't expect to produce much more than 1 million edges. If we are, then our blocking rules are probably too lenient and we can change them before wasting a lot of computing resource.
-
-# COMMAND ----------
-
-linker.cumulative_num_comparisons_from_blocking_rules_chart()
-
-# COMMAND ----------
-
-# Make predictions
-predictions = linker.predict()
-
-# Relatively small dataset means this distribution will be less clear, but most match weights will be low, because most comparisons aren't matches
-linker.match_weights_histogram(predictions)
-
-# COMMAND ----------
-
-path = f"/dbfs/Users/{username}/comparison_viewer.html"
-linker.comparison_viewer_dashboard(
-    predictions, 
-    out_path=path, 
-    overwrite=True
-)
-
-with open("/dbfs" + path, "r") as f:
-    html=f.read()    
-displayHTML(html)
-
-# COMMAND ----------
-
-# MAGIC %md
-# MAGIC Using the comparison viewer above you can brush the bottom chart to select comparison levels in between the red and green ends to filter out the clear matches and non-matches. Clicking a bar will show an example waterfall chart to demonstrate how the match weight is calculated.
-# MAGIC 
-# MAGIC Another way to do this is shown below:
-# MAGIC - Filter predictions according to their match probabilty
-# MAGIC - Take a sample and output to dict
-# MAGIC - Create a waterfall chart from these ambiguous edges
-
-# COMMAND ----------
-
-ambiguous_edges = predictions.as_spark_dataframe()\
-    .filter("match_probability > 0.1 and match_probability < 0.8").limit(100)\
-    .toPandas().to_dict(orient="records")
-
-linker.waterfall_chart(ambiguous_edges, filter_nulls=False)
-
-# COMMAND ----------
-
-# MAGIC %md
-# MAGIC Using the slider at the bottom of the waterfall chart, we can quickly see that most of these ambiguous cases are where there is an exact match on address but a mismatch or partial match on DOB or name.
-# MAGIC 
-# MAGIC This makes sense - our model doesn't know the difference between a nickname/alias or two family members at the same address. 
-# MAGIC 
-# MAGIC We can explore this further by going back to the comparison viewer dashboard above and use the dropdowns to select comparisons where name does not match - do any of these end up with a high match weight? (_No, they don't_)
-
-# COMMAND ----------
-
-# MAGIC %md
-# MAGIC ## 4. Clusters
-# MAGIC > **Once we're happy with our model, we probably have a rough idea of what threshold match score to use to define a match (e.g. >0.9).**
-# MAGIC > 
-# MAGIC >**Spot checking a sample of clusters with `cluster_studio_dashboard` we can check for a couple of things, the biggest risk being due to false positives (linking records that aren't genuine matches):**
-# MAGIC > - **Are there any "hairball" clusters that have become overly large because several genuine clusters have been merged?**
-# MAGIC > - **Ideal clusters will be fully-connected (every node is directly linked to every other) - are there clusters that appear to be two or more subclusters only connected by a single (suspicious) edge?**
-
-# COMMAND ----------
-
-clusters = linker.cluster_pairwise_predictions_at_threshold(
-    predictions, 
-    threshold_match_probability=0.8
-)
-
-# COMMAND ----------
-
-path = f"/dbfs/Users/{username}/cluster_studio.html"
-linker.cluster_studio_dashboard(
-    predictions, 
-    clusters, 
-    out_path=path, 
-    sampling_method="by_cluster_size", 
-    overwrite=True
-)
-
-with open("/dbfs" + path, "r") as f:
-    html=f.read()    
-displayHTML(html)
-
-# COMMAND ----------
-
-
diff --git a/Workshop Materials/notebooks/Splink in Databricks - end-to-end example.py b/Workshop Materials/notebooks/Splink in Databricks - end-to-end example.py
deleted file mode 100644
index 5cc9f40..0000000
--- a/Workshop Materials/notebooks/Splink in Databricks - end-to-end example.py
+++ /dev/null
@@ -1,501 +0,0 @@
-# Databricks notebook source
-# MAGIC %md
-# MAGIC # Splink on Databricks
-
-# COMMAND ----------
-
-# MAGIC %md
-# MAGIC ## What is entity resolution?
-# MAGIC 
-# MAGIC Entity resolution is the process of working out whether multiple records in a dataset are referencing the same real-world thing. This could be anything - a person, an organization, or any kind of physical object. Traditional approaches to solving this problem have involved writing business rules into data processing logic, but this can quickly become unmanageable with thousands of rules codified to address specific edge cases. This results in a process that is brittle, opaque, and complex. 
-# MAGIC 
-# MAGIC Modern approaches to entity resolution revolve around machine learning and are capable of easily adapting to new data and scaling to huge volumes - however modern approaches require modern infrastructure, with traditional data warehouses unable to handle this work.
-# MAGIC 
-# MAGIC Splink is an open source entity resolution library developed by the United Kingdom's Ministry of Justice, in use in the public and private sector across the world. Splink uses an unsupervised learning algorithm in conjunction with a set of heuristics to simplify the record linking process. Splink can be used in conjunction with the Databricks Lakehouse architecture to provide a simple, secure, and scalable solution to entity resolution.
-# MAGIC 
-# MAGIC ***
-# MAGIC 
-# MAGIC In this Notebook we will walk through a simple example of setting up Splink on Databricks and applying it to a dataset.
-
-# COMMAND ----------
-
-# MAGIC %md
-# MAGIC ## Setting up Splink on Databricks
-# MAGIC 
-# MAGIC - Installing Splink
-# MAGIC - Library imports
-# MAGIC - Enabling Splink on Databricks
-# MAGIC - Create working catalog/schema
-
-# COMMAND ----------
-
-# MAGIC %md
-# MAGIC ### Installing Splink
-# MAGIC 
-# MAGIC We'll start by `pip` installing Splink from a repository:
-
-# COMMAND ----------
-
-# MAGIC %pip install splink --quiet
-
-# COMMAND ----------
-
-# DBTITLE 1,Get current username
-username = spark.sql('select current_user() as user').collect()[0]['user']
-db_name = username.replace(".", "_").replace("@", "_")
-db_name
-
-# COMMAND ----------
-
-# MAGIC %md
-# MAGIC ### Importing Libraries
-# MAGIC 
-# MAGIC We can now import the Splink library and the relevant sub-modules:
-# MAGIC * The `SparkLinker` class for modelling work
-# MAGIC * The `enable_splink` built-in method which configures Splink on Databricks all in one go
-# MAGIC * The `spark_comparison_library` for built-in string comparison methods
-# MAGIC * The `splink_mlflow` sub-module for built-in wrappers for MLFlow
-
-# COMMAND ----------
-
-from splink.spark.spark_linker import SparkLinker
-from splink.databricks.enable_splink import enable_splink
-import splink.spark.spark_comparison_library as cl
-from utils.mlflow import splink_mlflow
-
-import mlflow
-
-# COMMAND ----------
-
-# MAGIC %md
-# MAGIC ## Resolving duplicates in a dataset of people
-# MAGIC 
-# MAGIC We'll be working with a dataset containing a list of individuals enriched with demographic information. We suspect that there are non-identical duplicates in this list, so we'll use Splink to set up deterministic and probabilistic rules to identify these.
-# MAGIC 
-# MAGIC We have over 162,000 data points, with each of them having the following information available about the individuals (although null values do exist in the data!):
-# MAGIC * `name` - the full name of the individual
-# MAGIC * `country_of_residence` the country in which they live in at the time of entry
-# MAGIC * The `nationality` of the individual at the time of entry
-# MAGIC * The `date_of_birth` of the individual
-# MAGIC * The `forenames` of the individual
-# MAGIC * The `surname` of the individual
-# MAGIC * Individual address information about their residence:
-# MAGIC   * `postal_code`
-# MAGIC   * `premises`
-# MAGIC   * `address_line_1`
-# MAGIC   * `locality`
-# MAGIC   * `region`
-# MAGIC * A unique identifier `uid`
-
-# COMMAND ----------
-
-# MAGIC %md
-# MAGIC ### Loading and inspecting the data
-# MAGIC 
-# MAGIC We use spark to load the Delta table where the data resides.
-
-# COMMAND ----------
-
-# MAGIC %run
-# MAGIC ../setup/data-setup
-
-# COMMAND ----------
-
-table_name = "companies_with_officers"
-table_path = f"/mnt/source/splinkdata/delta/{table_name}"
-data = spark.read.load(table_path)
-
-# COMMAND ----------
-
-# MAGIC %md
-# MAGIC Let's confirm the number of rows is as expected
-
-# COMMAND ----------
-
-data.count()
-
-# COMMAND ----------
-
-# MAGIC %md
-# MAGIC And let's inspect the first 1000 rows to get a feel for the data.
-
-# COMMAND ----------
-
-data.display()
-
-# COMMAND ----------
-
-# MAGIC %md
-# MAGIC As an example, take a look at these two records - it would be reasonable to suggest that these are the same people. But how do we indentify this without trawling through the thousands of entries we have?
-
-# COMMAND ----------
-
-data.filter("uid = 23636 OR uid = 122382").display()
-
-# COMMAND ----------
-
-# MAGIC %md
-# MAGIC ### Exploratory data analysis with Splink
-# MAGIC 
-# MAGIC Splink provides interactive charts out of the box to explore our data (and others, which we'll look at later). Let's check the number of missing values per column and profile some columns to demonstrate.
-
-# COMMAND ----------
-
-# MAGIC %md
-# MAGIC First, we'll instantiate a `linker` object with our dataset. Note we need to give the `SparkLinker` object a `spark` instance which is automatically defined in a Databricks notebook.
-
-# COMMAND ----------
-
-linker = SparkLinker(data, spark=spark)
-
-# COMMAND ----------
-
-# MAGIC %md
-# MAGIC We can now plot a chart to show the number of missing values per column in a single line:
-
-# COMMAND ----------
-
-linker.missingness_chart()
-
-# COMMAND ----------
-
-# MAGIC %md
-# MAGIC And we'll profile the `nationality` and `postal_code` columns:
-
-# COMMAND ----------
-
-data.columns
-
-# COMMAND ----------
-
-linker.profile_columns("nationality")
-
-# COMMAND ----------
-
-linker.profile_columns("postal_code")
-
-# COMMAND ----------
-
-# MAGIC %md
-# MAGIC ## Working with Splink
-
-# COMMAND ----------
-
-# MAGIC %md
-# MAGIC ### How does Splink work
-# MAGIC 
-# MAGIC [Splink](https://moj-analytical-services.github.io/splink/index.html) is a Python package for probabilistic record linkage (entity resolution).
-# MAGIC 
-# MAGIC The core linkage algorithm is an implementation of Fellegi-Sunter's model of record linkage, with various customisations to improve accuracy.
-# MAGIC 
-# MAGIC There is a very succinct [blog post](https://www.robinlinacre.com/maths_of_fellegi_sunter/) summarising the Fellegi-Sunter model, and further context is provided in [this related paper](https://imai.fas.harvard.edu/research/files/linkage.pdf).
-# MAGIC 
-# MAGIC Suffice to say for this exercise, we'll be looking to estimate the probability of two records matching, given a set of criteria and constraints, to fulfil the following formula based on the Bayes Theorem:
-# MAGIC 
-# MAGIC \\(Pr(records\ match|some\ attributes\ match) = \frac{Pr(some\ attributes\ match|records\ match)Pr(records\ match)}{Pr(some\ attributes\ match)}\\)
-# MAGIC 
-# MAGIC which is the same as
-# MAGIC 
-# MAGIC \\(Pr(records\ match|some\ attributes\ match) = \frac{Pr(some\ attributes\ match|records\ match)Pr(records\ match)}{Pr(some\ attributes\ match|records\ match)Pr(records\ match)+Pr(some\ attributes\ match|records\ don't\ match)Pr(records\ don't\ match)}\\)
-# MAGIC 
-# MAGIC To estimate this probability, we'll define a few variables:
-# MAGIC 
-# MAGIC * \\(\lambda = Pr(records\ match)\\), i.e. our _prior_ belief of any two records matching
-# MAGIC * A probability \\(m\\) with two levels:
-# MAGIC   * \\(m_{0}=Pr(some\ attributes\ don't\ match|records\ match)\\)
-# MAGIC   * \\(m_{1}=Pr(some\ attributes\ match|records\ match)\\)
-# MAGIC * A probability \\(u\\) with two levels:
-# MAGIC   * \\(u_{0}=Pr(some\ attributes\ don't\ match|records\ don't\ match)\\)
-# MAGIC   * \\(u_{1}=Pr(some\ attributes\ match|records\ don't\ match)\\)
-# MAGIC * \\(\gamma\\), a measure of similarity between values in a column
-# MAGIC 
-# MAGIC This turns our equation above to a much simpler form:
-# MAGIC 
-# MAGIC \\(Pr(records\ match|some\ attributes\ match)=\frac{m_{1}\lambda}{m_{1}\lambda+u_{1}(1-\lambda)}\\)
-
-# COMMAND ----------
-
-# MAGIC %md
-# MAGIC ### Setting up Splink for predictions
-# MAGIC 
-# MAGIC 
-# MAGIC Splink estimates the \\(m\\) and \\(u\\) probabilities for us using the [expectation-maximilisation algorithm](https://en.wikipedia.org/wiki/Expectation%E2%80%93maximization_algorithm), but we need to provide a few things ourselves.
-
-# COMMAND ----------
-
-# MAGIC %md
-# MAGIC #### 1. Blocking rules for predictions
-# MAGIC 
-# MAGIC If we were to compare every record in our dataset against every other record, we would be looking at making over 26 _billion_ estimates. Splink allows you to greatly reduce this comparison space by implementing _blocking rules_ that basically constrain how two records must relate to each other in order to even be considered a potential match. A blocking rule is formulated as a SQL expression, and can be arbitrarily complex. For example, you could create record comparisons where the initial of the first name and the surname match. [This page](https://moj-analytical-services.github.io/splink/demos/03_Blocking.html) goes into more detail about blocking rules and their use.
-# MAGIC 
-# MAGIC To demonstrate this, we'll set up three rules:
-# MAGIC 1. Full name and date of birth are an exact match
-# MAGIC 2. Nationality, locality, premises and region are an exact match
-# MAGIC 3. Address, post code and surname are an exact match
-# MAGIC 
-# MAGIC By doing this, we've reduced the comparison space from 26 billion pairs to some 146,000 - a significant improvement!
-
-# COMMAND ----------
-
-blocking_rules_to_generate_predictions = [
-  "l.name = r.name and l.date_of_birth = r.date_of_birth",
-  "l.nationality = r.nationality and l.locality = r.locality and l.premises = r.premises and l.region = r.region",
-  "l.address_line_1 = r.address_line_1 and l.postal_code = r.postal_code and l.surname = r.surname",
-]
-
-
-# COMMAND ----------
-
-# MAGIC %md
-# MAGIC We can visualise these blocking rules using a built-in method within Splink by adding the rules as a setting to our `linker` object. Additionally, we'll define a few more settings that will help us later:
-# MAGIC * `retain_intermediate_calculation_columns` and `retain_matching_columns` are to allow us to visualise the predictions at the end
-# MAGIC * `link_type` is set to only deduplicate (as opposed to link multiple datasets) as we only have one dataset we're trying to deduplicate
-# MAGIC * `unique_id_column_name` tells Splink the unique identifier column of our dataset
-# MAGIC * `em_convergence` is setting a convergence threshold expectation-maximalisation algorithm
-
-# COMMAND ----------
-
-
-settings = {
-  "retain_intermediate_calculation_columns": True,
-  "retain_matching_columns": True,
-  "link_type": "dedupe_only",
-  "unique_id_column_name": "uid",
-  "em_convergence": 0.01,
-  "blocking_rules_to_generate_predictions": blocking_rules_to_generate_predictions
-}
-
-linker.initialise_settings(settings)
-
-# COMMAND ----------
-
-linker.cumulative_num_comparisons_from_blocking_rules_chart(blocking_rules_to_generate_predictions)
-
-# COMMAND ----------
-
-# MAGIC %md
-# MAGIC 
-# MAGIC Here we confirm the 140,000 number quoted above, with our second rule contributing the most to the total number of comparisons.
-
-# COMMAND ----------
-
-# MAGIC %md
-# MAGIC #### 2. Comparison methods for each column
-# MAGIC 
-# MAGIC Splink provides out-of-the-box algorithms for comparing strings with its [`spark_comparison_library`](https://moj-analytical-services.github.io/splink/comparison_library.html), which may be familiar already:
-# MAGIC 
-# MAGIC 
-# MAGIC ##### 1. Exact match
-# MAGIC As the names suggests, `exact_match` is a method that returns a binary value depending on whether two values are exactly the same or not. This might be useful if we don't want to leave any wiggle room for misspellings, for example.
-# MAGIC 
-# MAGIC 
-# MAGIC ##### 2. Levenshtein distance
-# MAGIC The `levenshtein_at_thresholds` method gives a measure of how similar or not two strings are, and it considers the number of "edits" needed to make to get from one string to another. Edits are defined as inserts, deletes or changes of a single character. This method might be useful for columns where we expect differences to be due to mis-typing a letter on a keyboard.
-# MAGIC 
-# MAGIC 
-# MAGIC ##### 3. Jaccard similarity
-# MAGIC The `jaccard_at_thresholds` method gives the Jaccard index of two strings, which is defined as the number of letters they share divided by their combined letters; mathematically, the intersection of the two sets divided by their union. This method might be useful if we suspect there might be differences in how localities are defined in different datasets - think _Greater London_ vs _The Greater London Area_, for instance.
-# MAGIC 
-# MAGIC 
-# MAGIC ##### 4. Jaro-Winkler similarity
-# MAGIC The `jaro_winkler_at_thresholds` method gives the Jaro-Winkler similarity score between two strings, considering the relative position of each character in the string - characters are considered a match if they are the same character and within 2 characters of the same location in the comparison string. 
-# MAGIC 
-# MAGIC 
-# MAGIC ##### A note on thresholds
-# MAGIC 
-# MAGIC Notice that similarity measure _2-4_ above all have an `*_at_thresholds` suffix. This is because Splink allows us to optionally define numeric thresholds for each of these, and creates comparison levels within each column. For example, given a Levenshtein rule with threshold of 2 on `firstname`, Splink will create three cardinal values for the similarity:
-# MAGIC 1. If they're an exact match (great match)
-# MAGIC 2. If they're not an exact match, but have a Levenshtein distance of at most 2 (good match)
-# MAGIC 3. All others (bad match)
-# MAGIC 
-# MAGIC _Note: This effectively defines our \\(\gamma\\) value from above._
-
-# COMMAND ----------
-
-# MAGIC %md
-# MAGIC We'll create a list of comparison methods for some columns of interest for demonstration and set arbitrary Levenshtein distances for all of our columns of interest.
-
-# COMMAND ----------
-
-comparisons = [
-  cl.levenshtein_at_thresholds("surname", 10),
-  cl.levenshtein_at_thresholds("forenames", 10),
-  cl.levenshtein_at_thresholds("address_line_1", 5),
-  cl.levenshtein_at_thresholds("country_of_residence", 1),
-  cl.levenshtein_at_thresholds("nationality", 2)
-]
-
-# COMMAND ----------
-
-# MAGIC %md
-# MAGIC We'll now add these comparisons to our `linker` object's settings:
-
-# COMMAND ----------
-
-settings.update({"comparisons": comparisons})
-
-linker.initialise_settings(settings)
-
-# COMMAND ----------
-
-# MAGIC %md
-# MAGIC #### 3. An estimate of the probability that any two given pairs are a match
-# MAGIC 
-# MAGIC Recall our formula from above, and in particular the \\(\lambda\\) parameter that was effectively the _prior_ to our Bayes formula, i.e. \\(Pr(records\ match)\\). We need some way of giving Splink an initial probability that two pairs are a match. We can do this by defining a set of deterministic rules that will help Splink do this.
-# MAGIC 
-# MAGIC We'll set up three rules to demonstrate:
-# MAGIC * Names are a match and the Levenshtein distance of their dates of birth is less than or equal to 1
-# MAGIC * Addresses are an exact match and the Levenshtein distances of their names are less than or equal to 5
-# MAGIC * Their names are an exact match and the Levenshtein distances of their addresses are less than or equal to 5
-
-# COMMAND ----------
-
-deterministic_rules = [
-  "l.name = r.name and levenshtein(r.date_of_birth, l.date_of_birth) <= 1",
-  "l.address_line_1 = r.address_line_1 and levenshtein(l.name, r.name) <= 5",
-  "l.name = r.name and levenshtein(l.address_line_1, r.address_line_1) <= 5",
-]
-
-# COMMAND ----------
-
-# MAGIC %md
-# MAGIC Additionally, we need to provide a sense (by which I mean, a number estimate) of how _well_ these rules do at defining true matches, according to our prior belief, for which Splink let's us give a _recall_ value to these rules. Let's demonstrate and estimate our \\(\lambda\\) given the above rules and a recall value of 80%:
-
-# COMMAND ----------
-
-linker.estimate_probability_two_random_records_match(deterministic_rules, recall=0.8)
-
-# COMMAND ----------
-
-# MAGIC %md
-# MAGIC ### Estimating model parameters
-# MAGIC 
-# MAGIC We now have everything to estimate our other parameters \\(u\\) and \\(m\\) (from the equations above) to eventually be able to generate predictions.
-
-# COMMAND ----------
-
-# MAGIC %md
-# MAGIC #### 1. Estimating the \\(u\\) probability
-# MAGIC 
-# MAGIC Recall that the \\(u\\) probability is the probability of observed values (which we now know are comparison levels \\(\gamma\\)) given that the records do __not__ match. This can be estimated using Splink's `estimate_u_using_random_sampling` method:
-
-# COMMAND ----------
-
-linker.estimate_u_using_random_sampling(target_rows=1e7)
-
-# COMMAND ----------
-
-# MAGIC %md
-# MAGIC #### 2. Estimating the \\(m\\) probability
-# MAGIC 
-# MAGIC Recall that the \\(m\\) probability is the probability of observed values given that the records __do__ match. Without ground-truth labels, this can be estimated using the expectation-maximalisation algorithm with Splink's built-in `estimate_parameters_using_expectation_maximisation` method. For this, we need to run at least two passes of estimations, where we "fix" one or more columns and estimate \\(m\\) for the rest. We'll then need to fix other columns to estimate \\(m\\) for the first columns we fixed. Let's demonstrate:
-
-# COMMAND ----------
-
-# MAGIC %md
-# MAGIC We'll first fix `name` and `date_of_birth` to match in order to estimate the other columns' \\(m\\) probabilities.
-
-# COMMAND ----------
-
-training_rule = "l.name = r.name and l.date_of_birth = r.date_of_birth"
-
-# COMMAND ----------
-
-linker.estimate_parameters_using_expectation_maximisation(training_rule)
-
-# COMMAND ----------
-
-# MAGIC %md
-# MAGIC We'll now fix `surname` and `forenames` and estimate \\(m\\) for the remaining columns:
-
-# COMMAND ----------
-
-training_rule = "l.surname = r.surname and l.forenames = r.forenames"
-
-# COMMAND ----------
-
-linker.estimate_parameters_using_expectation_maximisation(training_rule)
-
-# COMMAND ----------
-
-# MAGIC %md
-# MAGIC #### Visualising our estimated probabilities
-# MAGIC 
-# MAGIC We can use Splink's built-in `m_u_parameters_chart` method to visualise the interpretations of how the \\(m\\) and \\(u\\) probabilities were estimated:
-
-# COMMAND ----------
-
-linker.m_u_parameters_chart()
-
-# COMMAND ----------
-
-# MAGIC %md
-# MAGIC ### Making predictions using our model
-# MAGIC 
-# MAGIC We can now make predictions using our model to predict duplicates in our dataset. This will return a Pandas dataframe with pair-wise comparisons between individuals in our dataset. To reduce the number of rows we end up printing out, we can also pass a `threshold_match_probability` so we only display matches above that threshold.
-# MAGIC 
-# MAGIC Below we'll predict duplicates and display the results in a Spark dataframe to make use of its more interactive view in a Databricks notebook.
-
-# COMMAND ----------
-
-predictions = linker.predict()
-pdf_predictions = predictions.as_pandas_dataframe()
-spark.createDataFrame(pdf_predictions).display()
-
-# COMMAND ----------
-
-# MAGIC %md
-# MAGIC #### Visualising model results
-# MAGIC 
-# MAGIC Splink lets us visualise the rationale behind model predictions using a built-in waterfall chart. We need to pass a dictionary to Splink's `waterfall_chart` method, so we'll convert our predictions to a dict before calling it.
-
-# COMMAND ----------
-
-dict_predictions = predictions.as_record_dict(limit=20)
-linker.waterfall_chart(dict_predictions)
-
-
-# COMMAND ----------
-
-# MAGIC %md
-# MAGIC ### Saving our results to an MLFlow experiment
-# MAGIC 
-# MAGIC Splink integrates with [MLFlow](https://mlflow.org/), which is an open source end-to-end ML lifecycle management tool that is also available in Databricks as a managed service. With one line, we can log our model, results and artifacts to an MLFlow experiment for later re-use and comparison with other runs.
-# MAGIC 
-# MAGIC MLFlow also lets us store the model in a format that it can be used simply by a third party. 
-
-# COMMAND ----------
-
- # Log model and parameters
-with mlflow.start_run() as run:
-    splink_mlflow.log_splink_model_to_mlflow(linker, "linker")
-
-# COMMAND ----------
-
-# MAGIC %md
-# MAGIC 
-# MAGIC After the model is stored, the next step is to retrieve the model and use it - MLFlow uses standardised APIs - all that is necessary is to call `predict()` on the returned object.
-
-# COMMAND ----------
-
-run_id = run.info.run_id
-loaded_model = mlflow.pyfunc.load_model(f"runs:/{run_id}/linker")
-
-splink_results = loaded_model.predict(data)
-splink_results.as_spark_dataframe().display()
-
-# COMMAND ----------
-
-# MAGIC %md
-# MAGIC 
-# MAGIC MLFlow simplifies things by wrapping models in a layer of MLFlow - this provides a model agnostic approach, whereby one can completely replace the underlying model without impacting on downstream processes because those downstream processes are only calling `.predict()` on an MLFlow model. 
-# MAGIC 
-# MAGIC However, one can still access the underlying model, which means one can still access all of the built in Splink functions, with just a little bit of extra code. 
-# MAGIC 
-# MAGIC 
-# MAGIC (NB - the underlying Splink model only becomes available after it is used to make a prediction)
-
-# COMMAND ----------
-
-dict_predictions = splink_results.as_record_dict(limit=20)
-loaded_model.unwrap_python_model().linker.waterfall_chart(dict_predictions)
diff --git a/Workshop Materials/setup/data-setup.py b/Workshop Materials/setup/data-setup.py
deleted file mode 100644
index f8dac1a..0000000
--- a/Workshop Materials/setup/data-setup.py
+++ /dev/null
@@ -1,52 +0,0 @@
-# Databricks notebook source
-username = spark.sql('select current_user() as user').collect()[0]['user']
-
-
-# COMMAND ----------
-
-def fetch_table(table_name, reset):
-    share_name = 'splink-share'
-    schema_name= 'splink'
-    
-    if reset:
-        mode='overwrite'
-    else:
-        mode='ignore'
-    
-    df = spark.read.format("delta").load(f"/mnt/source/splinkdata/delta/{table_name}")
-    df.write.mode(mode).format("delta").save(f"dbfs:/Filestore/Users/{username}/{table_name}")
-    df = spark.read.format("delta").load(f"dbfs:/Filestore/Users/{username}/{table_name}")
-    return df
-
-# COMMAND ----------
-
-df_companies_with_officers = fetch_table("companies_with_officers", reset=False)
-df_companies_with_officers.display()
-
-# COMMAND ----------
-
-df_payment_practices_of_uk_buyers = fetch_table("payment_practices_of_uk_buyers", reset=False)
-
-# COMMAND ----------
-
-df_company_officers = fetch_table("company_officers", reset=False)
-
-
-# COMMAND ----------
-
-#display(dbutils.fs.ls(f"/Filestore/Users/{username}"))
-
-# COMMAND ----------
-
-def clear_user_data(table_name):
-    dbutils.fs.rm(f"/Filestore/Users/{username}/{table_name}", True)
-
-# COMMAND ----------
-
-# clear_user_data("company_officers")
-# clear_user_data("payment_practices_of_uk_buyers")
-# clear_user_data("companies_with_officers")
-
-# COMMAND ----------
-
-
diff --git a/Workshop Materials/utils/__init__.py b/Workshop Materials/utils/__init__.py
deleted file mode 100644
index 792d600..0000000
--- a/Workshop Materials/utils/__init__.py
+++ /dev/null
@@ -1,1 +0,0 @@
-#
diff --git a/Workshop Materials/utils/mlflow/splink_mlflow.py b/Workshop Materials/utils/mlflow/splink_mlflow.py
deleted file mode 100644
index 04194f6..0000000
--- a/Workshop Materials/utils/mlflow/splink_mlflow.py
+++ /dev/null
@@ -1,344 +0,0 @@
-import shutil
-import mlflow
-import os
-import json
-
-from splink.charts import save_offline_chart
-
-from mlflow.models.signature import infer_signature
-from pyspark.sql import SparkSession
-from splink.spark.spark_linker import SparkLinker
-
-# ======================================================================
-# ======================================================================
-def _log_comparison_details(comparison):
-    """
-    Traverse a single comparison item from the splink settings JSON and log the sql condition the comparison was made
-    under and the m and u values
-    Parameters
-    ----------
-    comparison :
-
-    Returns
-    -------
-
-    """
-    output_column_name = comparison['output_column_name']
-
-    comparison_levels = comparison['comparison_levels']
-    for _ in comparison_levels:
-        sql_condition = _.get('sql_condition')
-        m_probability = _.get('m_probability')
-        u_probability = _.get('u_probability')
-        log_dict = {
-            f"output column {output_column_name} compared through condition {sql_condition} m_probability": m_probability,
-            f"output column {output_column_name} compared through condition {sql_condition} u_probability": u_probability
-
-        }
-        if m_probability: # this means we don't log the null comparison as it doesnt have a m and u value
-            mlflow.log_params(log_dict)
-
-
-# ======================================================================
-# ======================================================================
-def _log_comparisons(splink_model_json):
-    """
-    Traverse the comparisons part of the splink settings and extract the learned values.
-    This allows you to easily compare values across different training conditions.
-
-    Parameters
-    ----------
-    splink_model_json : the settings json from a splink model.
-
-    Returns
-    -------
-
-    """
-    comparisons = splink_model_json['comparisons']
-    for _ in comparisons:
-        _log_comparison_details(_)
-
-
-# ======================================================================
-# ======================================================================
-def _log_hyperparameters(splink_model_json):
-    """
-    Simple method for extracting parameters from the splink settings and logging as parameters in MLFlow
-    Parameters
-    ----------
-    splink_model_json : the settings json from a splink model.
-
-    Returns
-    -------
-
-    """
-    hyper_param_keys = ['link_type', 'probability_two_random_records_match', 'sql_dialect', 'unique_id_column_name',
-                        'em_convergence', 'retain_matching_columns', 'blocking_rules_to_generate_predictions']
-    for key in hyper_param_keys:
-        mlflow.log_param(key, splink_model_json[key])
-
-
-# ======================================================================
-# ======================================================================
-def _log_splink_model_json(splink_model_json):
-    """
-    Log the splink settings json in its entirery under the name "linker.json"
-    Parameters
-    ----------
-    splink_model_json : the settings json from a splink model.
-
-    Returns
-    -------
-
-    """
-    mlflow.log_dict(splink_model_json, "linker.json")
-
-
-# ======================================================================
-# ======================================================================
-class splinkSparkMLFlowWrapper(mlflow.pyfunc.PythonModel):
-    """
-    Wrapper class necessary for storing and retrieving the splink model JSON from the MLFlow model repository.
-    """
-
-    def load_context(self, context):
-        # this simply stores the json with the MLFLow model.
-        self.linker_json = context.artifacts['linker_json_path']
-
-
-
-# ======================================================================
-# ======================================================================
-def _save_splink_model_to_mlflow(linker, model_name):
-    """
-    Simple method for logging a splink model
-    Parameters
-    ----------
-    linker : Splink model object
-    model_name : name to save model under
-
-    Returns
-    -------
-
-    """
-    path = "linker.json"
-    if os.path.isfile(path):
-        os.remove(path)
-    linker.save_settings_to_json(path)
-    artifacts = {"linker_json_path": path}
-    mlflow.pyfunc.log_model(model_name, python_model=splinkSparkMLFlowWrapper(), artifacts=artifacts)
-    os.remove(path)
-
-
-# ======================================================================
-# ======================================================================
-def log_splink_model_to_mlflow(linker, model_name, log_parameters_charts=True,
-                               log_profiling_charts=False, params=None, metrics=None, artifacts=None):
-    """
-    Comprehensive logging of Splink attributes, parameters, charts and model JSON to MLFlow to provide easy
-    reproducability and tracking during model development and deployment. This will create a new run under which
-    to log everything. Additional parameters and metrics can be logged through the params and metrics arguments.
-
-    For details on MLFlow please visit https://mlflow.org/.
-
-    This will log
-    - All the m and u values trained along with their training condition
-    - All the hyperparameters set during training (blocking rules, link type, sql_dialect etc)
-
-    Parameters
-    ----------
-    run : an mlflow.start_run() instance
-    linker : a trained Splink linkage model
-    model_name : str, the name to log the model under
-    log_parameters_charts: boolean, whether to log parameter charts or not. Default to True as this is a quick operation.
-    log_profiling_charts: boolean, whether to log data profiling charts or not. Default to False as this requires running
-    Spark jobs and can take time.
-    params : dict[str: str]. Dictionary is of param_name :param_value. Optional argument for logging arbitrary
-    parameters
-    metrics : dict[str: double or int]. Dictionary is of metric_name :metric_value. Optional argument for logging
-    arbitrary metrics
-    artifacts : dict[str: str]. Dictionary is of artifact_name :artifact_filepath. Optional argument for logging
-    arbitrary artifacts (NB - artifacts must be written to disk before logging).
-    -------
-
-    """
-
-    splink_model_json = linker._settings_obj.as_dict()
-
-   
-    _log_splink_model_json(splink_model_json)
-    _log_hyperparameters(splink_model_json)
-    _log_comparisons(splink_model_json)
-    #_save_splink_model_to_mlflow(linker, model_name)
-    model = SplinkMLFlowWrapper()
-    model.set_spark_linker(linker)
-    mlflow.pyfunc.log_model(model_name, python_model=model)
-    if log_profiling_charts or log_parameters_charts:
-        _log_linker_charts(linker, log_parameters_charts, log_profiling_charts)
-    if params:
-        mlflow.log_params(params)
-    if metrics:
-        mlflow.log_metrics(metrics)
-    if artifacts:
-        mlflow.log_artifacts(artifacts)
-
-
-
-# ======================================================================
-# ======================================================================
-def _log_linker_charts(linker, log_parameters_charts, log_profiling_charts):
-    '''
-    Log all the non-data related charts to MLFlow
-    Parameters
-    ----------
-    linker : a Splink linker object
-    log_parameters_charts: boolean, whether to log parameter charts or not
-    log_profiling_charts: boolean, whether to log data profiling charts or not
-
-    Returns
-    -------
-
-    '''
-
-    charts_dict = {}
-    if log_parameters_charts:
-        weights_chart = linker.match_weights_chart()
-        mu_chart = linker.m_u_parameters_chart()
-        compare_chart = linker.parameter_estimate_comparisons_chart()
-        charts_dict["weights_chart"] = weights_chart
-        charts_dict["mu_chart"] = mu_chart
-        charts_dict["compare_chart"] = compare_chart
-
-    if log_profiling_charts:
-        missingness_chart = linker.missingness_chart()
-        # completeness = linker.completeness_chart() # Note to Splink team - this method behaves differently to the others
-        unlinkables = linker.unlinkables_chart()
-        blocking_rules_chart = linker.cumulative_num_comparisons_from_blocking_rules_chart()
-        charts_dict["missingness_chart"] = missingness_chart
-       # charts_dict["completeness"] = completeness
-        charts_dict["unlinkables"] = unlinkables
-        charts_dict["blocking_rules_chart"] = blocking_rules_chart
-
-    for name, chart in charts_dict.items():
-        _log_chart(name, chart)
-
-
-# ======================================================================
-# ======================================================================
-def _log_chart(chart_name, chart):
-    '''
-    Save a chart to MLFlow. This writes the chart out temporarily for MLFlow to pick up and save.
-    Parameters
-    ----------
-    chart_name : str, the name the chart will be given in MLFlow
-    chart : chart object from Splink
-
-    Returns
-    -------
-
-    '''
-    path = f"{chart_name}.html"
-    if os.path.isfile(path):
-        os.remove(path)
-    save_offline_chart(chart.spec, path)
-    mlflow.log_artifact(path)
-    os.remove(path)
-
-# ======================================================================
-# ======================================================================
-def get_model_json(artifact_uri):
-    """
-
-    Returns a Splink model JSON from a model tracked with MLFlow. This can be retrieved either from a run or
-    from the MLFlow Model Registry, if and only if the model was logged to MLFlow using the method
-    splink.mlflow.log_splink_model_to_mlflow().
-
-    Extracting the linker's JSON specification requires temporarily writing to /tmp on the local file system. All
-    written files are deleted before the function returns.
-
-    Parameters
-    ----------
-    artifact_uri : URI pointing to the artifacts, such as "runs:/500cf58bee2b40a4a82861cc31a617b1/my_model.pkl",
-        "models:/my_model/Production", or "s3://my_bucket/my/file.txt".
-
-    Returns
-    -------
-    A splink setting json as a python dict.
-
-    """
-    temp_file_path = f"/tmp/{artifact_uri.split('/')[1]}/splink_mlflow_artifacts_download"
-    os.makedirs(temp_file_path)
-    mlflow.artifacts.download_artifacts(
-        artifact_uri=artifact_uri
-        , dst_path=temp_file_path
-    )
-
-    with open(f"{temp_file_path}/artifacts/linker.json", "r") as f:
-        linker_json = json.load(f)
-    shutil.rmtree(temp_file_path)
-
-    return linker_json
-
-# ======================================================================
-# ======================================================================
-class SplinkMLFlowWrapper(mlflow.pyfunc.PythonModel):
-    """
-    MLFlow compatibility wrapper for splink linker.
-    """
- 
-    def __init__(self, **kwargs):
-        self.settings = {}
-        self.linker = None
-        for key, value in kwargs.items():
-            setattr(self, key, value)
-            
-    def __getstate__(self):
-        json_result = self.get_settings_object()
-        return json_result
-            
-    def __setstate__(self, json_dict):
-        self.settings = json_dict
-    
-    def load_context(self, context):
-        # this simply stores the json with the MLFLow model.
-        self.context = context
-        
-    def clear_context(self):
-        # model cannot be pickled, so disposing it
-        self.linker.spark = None
-        
-    def set_should_evaluate(self, flag):
-        self.should_evaluate = flag
-        
-    def spark_linker(self, data):
-        spark = SparkSession.builder.getOrCreate()
-        self.linker = SparkLinker(data, spark=spark)
-        spark = None
-        return self.linker
-        
-    def set_spark_linker(self, linker: SparkLinker):
-        self.linker = linker
-    
-    def set_settings(self, settings):
-        self.settings = settings
-        if self.linker:
-            self.linker.initialise_settings(self.settings)
-        
-    def get_linker(self):
-        return self.linker
-    
-    def get_settings(self):
-        return self.settings
-   
-    def get_settings_object(self):
-        return self.get_linker()._settings_obj.as_dict()
-    
- 
-    def predict(self, context, X):
-        """
-        Predict labels on provided data
-        """
-        self.linker = self.spark_linker(X)
-        self.linker.initialise_settings(self.settings)
-        result = self.linker.predict()
-        return result
diff --git a/Workshop Materials/utils/mlflow_utils.py b/Workshop Materials/utils/mlflow_utils.py
deleted file mode 100644
index be3c121..0000000
--- a/Workshop Materials/utils/mlflow_utils.py
+++ /dev/null
@@ -1,211 +0,0 @@
-import shutil
-import mlflow
-import os
-import json 
-import math
-from sklearn.mixture import GaussianMixture
-import matplotlib.pyplot as plt
-import numpy as np
-import scipy.stats as ss
-import networkx as nx
-
-from splink.charts import save_offline_chart
-
-# ======================================================================
-# ======================================================================
-def get_comparison_details(comparison):
-    """
-    Traverse a single comparison item from the splink settings JSON and log the sql condition the comparison was made
-    under and the m and u values
-    Parameters
-    ----------
-    comparison :
-    Returns
-        A dict containing m and u values and conditions used.
-    -------
-    """
-    output_column_name = comparison['output_column_name']
-
-    comparison_levels = comparison['comparison_levels']
-    log_dict = {}
-    for _ in comparison_levels:
-        sql_condition = _.get('sql_condition')
-        m_probability = _.get('m_probability')
-        u_probability = _.get('u_probability')
-        log_dict[f"output column {output_column_name} compared through condition {sql_condition} m_probability"] = m_probability
-        log_dict[f"output column {output_column_name} compared through condition {sql_condition} u_probability"] = u_probability
-
-    return log_dict
-
-
-# ======================================================================
-# ======================================================================
-def get_all_comparisons(splink_model_json):
-    """
-    Traverse the comparisons part of the splink settings and extract the learned values.
-    This allows you to easily compare values across different training conditions.
-    Parameters
-    ----------
-    splink_model_json : the settings json from a splink model.
-    Returns
-        A list of dicts containing m and u values and conditions used.
-    -------
-    """
-    comparisons = splink_model_json['comparisons']
-    return [get_comparison_details(cmp) for cmp in comparisons]
-
-
-# ======================================================================
-# ======================================================================
-def get_hyperparameters(splink_model_json):
-    """
-    Simple method for extracting parameters from the splink settings and logging as parameters in MLFlow
-    Parameters
-    ----------
-    splink_model_json : the settings json from a splink model.
-    Returns
-        The subset of splink_model_json that contains relevant hyperparameters.
-    -------
-    """
-    hyper_param_keys = ['link_type', 'probability_two_random_records_match', 'sql_dialect', 'unique_id_column_name',
-                        'em_convergence', 'retain_matching_columns', 'blocking_rules_to_generate_predictions']
-    result = {}
-    for key in hyper_param_keys:
-        result[key] = splink_model_json[key]
-
-    return result
-
-
-# ======================================================================
-# ======================================================================
-def get_linker_charts(linker, log_parameters_charts, log_profiling_charts):
-    '''
-    Log all the non-data related charts to MLFlow
-    Parameters
-    ----------
-    linker : a Splink linker object
-    log_parameters_charts: boolean, whether to log parameter charts or not
-    log_profiling_charts: boolean, whether to log data profiling charts or not
-    Returns
-        A dict of charts to be logged for splink model.
-    -------
-    '''
-
-    charts_dict = {}
-    if log_parameters_charts:
-        weights_chart = linker.match_weights_chart()
-        mu_chart = linker.m_u_parameters_chart()
-        compare_chart = linker.parameter_estimate_comparisons_chart()
-        charts_dict["weights_chart"] = weights_chart
-        charts_dict["mu_chart"] = mu_chart
-        charts_dict["compare_chart"] = compare_chart
-
-    if log_profiling_charts:
-        missingness_chart = linker.missingness_chart()
-        unlinkables = linker.unlinkables_chart()
-        blocking_rules_chart = linker.cumulative_num_comparisons_from_blocking_rules_chart()
-        charts_dict["missingness_chart"] = missingness_chart
-        charts_dict["unlinkables"] = unlinkables
-        charts_dict["blocking_rules_chart"] = blocking_rules_chart
-
-    return charts_dict
-
-
-# ======================================================================
-# ======================================================================
-def get_match_probabilty_loss(predictions):
-    '''
-    Calculates the custom loss of a pariwise Splink prediction dataframe, by
-    1. Fitting 2 Gaussians on the probability distribution
-    2. Calculating and summing the means from 0 and 1 (respectively) and the standard deviations of the Gaussians
-    Parameters
-    ----------
-    predictions: pandas.DataFrame, df of predictions that must contain the match_probabilty column
-    Returns
-        A 2-tuple of the loss (float) and a chart (matplotlib.pyplot.figure) of the distribution and the fit
-    '''
-    
-    # check if match_probability column is in the dataframe
-    if "match_probability" in predictions.columns:
-        pred = predictions["match_probability"].values
-    else:
-        raise Exception("The match_probability column wasn't found in dataframe. You need to pass a dataframe that was predicted by Splink.")
-        
-    # Fig Gaussian mixture model (disable autologging for this)
-    mlflow.autolog(disable=True)
-    gm = GaussianMixture(n_components=2, random_state=0).fit(pred.reshape(-1, 1))
-    mlflow.autolog(disable=False)
-    
-    # Get means, variances and standard deviations
-    first_gauss = np.argmin(np.array([i[0] for i in gm.means_]))
-    second_gauss = np.argmax(np.array([i[0] for i in gm.means_]))
-    mean_1 = gm.means_[first_gauss][0]
-    mean_2 = gm.means_[second_gauss][0]
-    cov_1 = gm.covariances_[first_gauss][0][0]
-    cov_2 = gm.covariances_[second_gauss][0][0]
-    std_1 = math.sqrt(cov_1)
-    std_2 = math.sqrt(cov_2)
-
-    # Calculate loss
-    mean_goodness = mean_1 + 1.0 - mean_2
-    std_goodness = std_1 + std_2
-    prediction_purity = mean_goodness + std_goodness
-
-    # Create probability density functions based on the two sets of parameters
-    x = np.linspace(0.0, 1.0, 100)
-    gauss_1 = ss.norm.pdf(x, mean_1, std_1)
-    gauss_2 = ss.norm.pdf(x, mean_2, std_2)
-
-    # plot chart
-    fig, ax = plt.subplots(figsize=(10, 8))
-    ax.hist(pred, density=True, bins=50)
-    ax.plot(x, gauss_1, c="green", linewidth=3)
-    ax.plot(x, gauss_2, c="orange", linewidth=3)
-    ax.axvline(mean_1, c="green", linewidth=1, linestyle="--")
-    ax.axvline(mean_2, c="orange", linewidth=1, linestyle="--")
-    ax.axvline(0.0, c="grey", linewidth=1, linestyle="--")
-    ax.axvline(1.0, c="grey", linewidth=1, linestyle="--")
-    ax.hlines(40, 0.0, mean_1, linewidth=2, linestyle="-", color="red")
-    ax.hlines(40, mean_2, 1.0, linewidth=2, linestyle="-", color="red")
-    ax.hlines(40, mean_1, mean_1+cov_1, linewidth=2, linestyle="-", color="yellow")
-    ax.hlines(45, mean_2-cov_2, mean_2, linewidth=2, linestyle="-", color="yellow")
-    ax.set_title("Probability density of predictions and fitted parameters")
-    
-    return prediction_purity, fig
-
-# ======================================================================
-# ======================================================================
-def visualise_linked_records(preds, min_linked_records=10, linkage_threshold=0.9):
-    '''
-    Visualises the connected components of inter-linked records
-    Parameters
-    ----------
-    preds: pandas.DataFrame, df of predictions that must contain the match_probabilty column
-    min_linked_records: 
-    Returns
-        A 2-tuple of the loss (float) and a chart (matplotlib.pyplot.figure) of the distribution and the fit
-    '''
-
-    preds = preds[preds["match_probability"]>=linkage_threshold]
-    G = nx.from_pandas_edgelist(
-      df=preds,
-      source="uid_l",
-      target="uid_r",
-
-    )
-    for component in list(nx.connected_components(G)):
-      if len(component)<min_linked_records:
-        for node in component:
-          G.remove_node(node)
-
-    labeldict = {}
-    for i, row in preds.iterrows():
-      if row["uid_r"] in G.nodes():
-        labeldict.update({row["uid_r"]: row["name_r"]})
-      if row["uid_l"] in G.nodes():
-        labeldict.update({row["uid_l"]: row["name_l"]})
-
-    plt.figure(figsize=(12, 12))
-    nx.draw(G, labels=labeldict, with_labels=True, node_size=10,font_size=5)
-    
-    return G.number_of_nodes()
diff --git a/Workshop Materials/utils/splink_linker_model.py b/Workshop Materials/utils/splink_linker_model.py
deleted file mode 100644
index 83206c5..0000000
--- a/Workshop Materials/utils/splink_linker_model.py
+++ /dev/null
@@ -1,179 +0,0 @@
-import mlflow
-import mlflow.pyfunc
-import os
-import random
-import splink.spark.spark_comparison_library as cl
-import tempfile
-
-from mlflow.models.signature import infer_signature
-from pyspark.sql import SparkSession
-from sklearn.metrics import roc_auc_score, f1_score
-from splink.spark.spark_linker import SparkLinker
-
-from utils.mlflow_utils import *
-
-
-class SplinkLinkerModel(mlflow.pyfunc.PythonModel):
-    """
-    MLFlow compatibility wrapper for splink linker.
-    """
- 
-    def __init__(self, **kwargs):
-        self.settings = {}
-        self.deterministic_rules = None
-        for key, value in kwargs.items():
-            setattr(self, key, value)
-            
-    def __getstate__(self):
-        json_result = self.get_settings_object()
-        return json_result
-            
-    def __setstate__(self, json_dict):
-        self.settings = json_dict
-    
-    def load_context(self, context):
-        # this simply stores the json with the MLFLow model.
-        self.context = context
-        
-    def clear_context(self):
-        # model cannot be pickled, so disposing it
-        self.linker.spark = None
-        
-    def set_should_evaluate(self, flag):
-        self.should_evaluate = flag
-        
-    def spark_linker(self, data):
-        spark = SparkSession.builder.getOrCreate()
-        self.linker = SparkLinker(data, spark=spark)
-        spark = None
-        return self.linker
-        
-    def set_spark_linker(self, linker: SparkLinker):
-        self.linker = linker
-    
-    def set_settings(self, settings):
-        self.settings = settings
-        if self.linker:
-            self.linker.initialise_settings(self.settings)
-        
-    def set_blocking_rules(self, blocking_rules):
-        self.settings.update({"blocking_rules_to_generate_predictions": blocking_rules})
-        if self.linker:
-            self.linker.initialise_settings(self.settings)
-    
-    def set_comparisons(self, comparisons):
-        self.settings.update({"comparisons": comparisons})
-        if self.linker:
-            self.linker.initialise_settings(self.settings)
-            
-    def set_deterministic_rules(self, rules):
-        self.deterministic_rules = rules
-            
-    def set_stage1_columns(self, columns):
-        self.stage1_columns = columns
-        
-    def set_stage2_columns(self, columns):
-        self.stage2_columns = columns
-    
-    def set_target_rows(self, rows):
-        self.target_rows = rows
-        
-    def get_linker(self):
-        return self.linker
-    
-    def get_settings(self):
-        return self.settings
-   
-    def get_settings_object(self):
-        return self.get_linker()._settings_obj.as_dict()
-    
-    def estimate_random_match_probability(self, rules, recall):
-        self.linker.estimate_probability_two_random_records_match(rules, recall=recall)
-        
-    def estimate_u(self, target_rows):
-        self.linker.estimate_u_using_random_sampling(target_rows=target_rows)
-        
-    def estimate_m(self, data, columns_1, columns_2):
-        fixed_columns = random.choices(data.columns, k=2)
-        training_rule_1 = " and ".join([f"(l.{cn} = r.{cn})" for cn in columns_1])
-        training_rule_2 = " and ".join([f"(l.{cn} = r.{cn})" for cn in columns_2])
-        self.linker.estimate_parameters_using_expectation_maximisation(training_rule_1)
-        self.linker.estimate_parameters_using_expectation_maximisation(training_rule_2)
-        
-    def fit(self, X, recall_prior=0.8):
-        """
-        Model training given input parameters and a training dataset.
-        """
-        if self.settings is None:
-            raise Exception("Cannot initialise linker without settings being set. Please use model.set_settings.")
-        if not self.deterministic_rules:
-            raise Exception("Cannot initialise linker without setting deterministic rules. Please use model.set_deterministic_rules.")
-        
-      
-        linker = self.spark_linker(X)
-        linker.initialise_settings(self.settings)
-        self.set_spark_linker(linker)
-        self.estimate_random_match_probability(self.deterministic_rules, recall_prior)
-        self.estimate_u(self.target_rows)
-        self.estimate_m(X, self.stage1_columns, self.stage2_columns)
-        return linker
-    
-    def log_settings_as_json(self, path):
-        """
-        Simple method for logging a splink model
-        Parameters
-        ----------
-        linker : Splink model object
-        Returns
-        -------
-        """
-        path = "linker.json"
-        if os.path.isfile(path):
-            os.remove(path)
-        self.linker.save_settings_to_json(path)
-        mlflow.log_artifact(path)
-        os.remove(path)
-        
-    def _log_chart(self, chart_name, chart):
-        '''
-        Save a chart to MLFlow. This writes the chart out temporarily for MLFlow to pick up and save.
-        Parameters
-        ----------
-        chart_name : str, the name the chart will be given in MLFlow
-        chart : chart object from Splink
-        Returns
-        -------
-        '''
-        path = f"{chart_name}.html"
-        if os.path.isfile(path):
-            os.remove(path)
-        save_offline_chart(chart.spec, path)
-        mlflow.log_artifact(path)
-        os.remove(path)
-  
-    def evaluate(self, y_test, y_test_pred):
-        """
-        Evaluate model performance.
-        """
-        if self.should_evaluate:
-            spark = SparkSession.builder.getOrCreate()
-            splink_tables = spark.sql('show tables like "*__splink__*"')
-            temp_tables = splink_tables.collect()
-            drop_tables = list(map(lambda x: x.tableName, temp_tables))
-            for x in drop_tables:
-                spark.sql(f"drop table {x}")
-            self.linker.register_table(y_test, "labels")
-            roc_auc = linker.roc_chart_from_labels_table("labels")
-            return roc_auc
-        else:
-            return None
- 
-    def predict(self, context, X):
-        """
-        Predict labels on provided data
-        """
-        self.linker = self.spark_linker(X)
-        self.linker.initialise_settings(self.settings)
-        result = self.linker.predict()
-        return result
-    
diff --git a/docs/source/index.rst b/docs/source/index.rst
index 9090180..5c93370 100644
--- a/docs/source/index.rst
+++ b/docs/source/index.rst
@@ -13,6 +13,7 @@ Documentation
    :titlesonly:
    :maxdepth: 2
    :caption: Contents:
+   usage/usage
 
 
 AutoLinker
diff --git a/docs/source/usage/best_practices.rst b/docs/source/usage/best_practices.rst
new file mode 100644
index 0000000..e69de29
diff --git a/docs/source/usage/installation.rst b/docs/source/usage/installation.rst
new file mode 100644
index 0000000..7ed1358
--- /dev/null
+++ b/docs/source/usage/installation.rst
@@ -0,0 +1,17 @@
+Package installation
+####################
+
+Databricks Runtime Requirements
+**********************
+ARC requires Databricks ML Runtime 12.2LTS. ARC will not work on 13.x or greater runtimes.
+
+Installation from PyPI
+**********************
+Python users can install the library directly from `PyPI <https://pypi.org/project/databricks-mosaic/>`__
+using the instructions `here <https://docs.databricks.com/libraries/cluster-libraries.html>`__
+or from within a Databricks notebook using the :code:`%pip` magic command, e.g.
+
+.. code-block:: bash
+
+    %pip install databricks-arc
+
diff --git a/docs/source/usage/linking.rst b/docs/source/usage/linking.rst
new file mode 100644
index 0000000..8e87fd9
--- /dev/null
+++ b/docs/source/usage/linking.rst
@@ -0,0 +1,40 @@
+Data Linking with ARC
+####################
+
+Getting started
+**********************
+
+Starting a linking project with ARC is simple - you just need to load your data into a pair of Spark Dataframes, and pass
+them to the Autolinker object in a list.
+
+.. code-block:: python
+    from arc.autolinker import AutoLinker
+
+    autolinker = AutoLinker()
+
+    data_1 = spark.read.load(<a table of messy data>)
+    data_2 = spark.read.load(<a table of messy data>)
+    data = list(data_1, data_2)
+    autolinker.auto_link(
+      data=data
+    )
+
+No other options are required. This will start an auto linking process that will run for 5 iterations of hyperparameter
+optimisation. During these 5 rounds ARC will try a variety of options to determine impact on model performance. It is
+unlikely that 5 rounds will be sufficient - in our testing we found that the model needed to run for 100 iterations or
+more for good results.
+
+This is done with
+
+.. code-block:: python
+    from arc.autolinker import AutoLinker
+
+    autolinker = AutoLinker()
+
+    data_1 = spark.read.load(<a table of messy data>)
+    data_2 = spark.read.load(<a table of messy data>)
+    data = list(data_1, data_2)
+    autolinker.auto_link(
+      data=data
+      ,max_evals=100
+    )
diff --git a/docs/source/usage/usage.rst b/docs/source/usage/usage.rst
new file mode 100644
index 0000000..1cc36ec
--- /dev/null
+++ b/docs/source/usage/usage.rst
@@ -0,0 +1,122 @@
+.. toctree::
+   :maxdepth: 2
+
+   installation
+   best_practices
+
+Data Linking and Deduplication with ARC
+####################
+
+Getting started
+**********************
+
+Starting a linking project with ARC is simple - you just need to load your data into a pair of Spark Dataframes, and pass
+them to the Autolinker object in a list.
+
+.. code-block:: python
+    from arc.autolinker import AutoLinker
+
+    autolinker = AutoLinker()
+
+    data_1 = spark.read.load(<a table of messy data>)
+    data_2 = spark.read.load(<a table of messy data>)
+    data = list(data_1, data_2)
+    autolinker.auto_link(
+      data=data
+    )
+
+
+However, ARC has many other optional arguments which can be used to fine tune its behaviour.  These are detailed below,
+along with an explanation of when you would use them.
+
+
+`data: typing.Union[pyspark.sql.dataframe.DataFrame, list]`
+
+The input data to ARC. A single Dataframe will perform deduplication, a list of 2 dataframes will perform linking between
+the two datasets. If linking, for best performance ensure schemas are standardised prior to linking (i.e. the 2 input
+tables should share an identical schema). This is not required - ARC can handle mismatching schemas, but you will get
+better performance with identical schemas.
+
+`attribute_columns:list=None`
+
+Which columns should be used for evaluating if records are connected. In the case of linking 2 datasets, will only work with
+a identical schemas. If not provided, ARC will use all columns (except a unique ID column if provided) as input attributes
+to the model.
+
+**When to use**
+
+- if you have very wide tables with lot of extraneous information which will not help the model in determining if records
+  represent the same thing.
+- if you want to link at a coarser granularity than the data are. For example, if you have a table of
+  people and addresses and your aim is to link addresses, you could either drop the columns containing the people information, or use the *attribute_columns* arg to specify only the address information columns.
+
+`unique_id:str=None`
+
+Which column contains a unique per record ID. ARC will ignore this for purposes of linking. If None, ARC will append a new
+column to the dataset called *unique_id*.
+
+**When to use**
+- if your data has a unique id column. ARC will evaluate this column for linking if not, which at best will not help produce a good model.
+
+`comparison_size_limit:int=100000`
+
+The maximum number of pairs of records a blocking rule will be allowed to have. Blocking rules are heuristics used by Splink
+to control which records are potential duplicates. ARC auto-generates blocking rules for you, and uses this parameter to control
+which blocking rules are used by looking at the amount of potential duplicates each rule generates.
+
+**When to use**
+
+- if you want to speed up the model training process, try setting a lower value, i.e. 50,000.
+- this parameter will directly impact the recall of the model - if it is too low, you risk missing pairs.
+  It is better to err on the side of too big than too small.
+- it is generally unnecessary to change this value.
+
+`max_evals:int=5`
+
+The maximum number of evaluations ARC will do during its hyperparameter optimisation search. As a default value,
+5 will give a taste of how ARC works, but internal testing showed good linking results from at least 100 runs.
+
+**When to use**
+
+- set to 100 or more when training a model for proper evaluation - this will take a long time, but it is reflective of
+the size of search space across which HyperOpt needs to explore to find the best set of arguments.
+- this should only be left as the default argument during initial testing and evaluation.
+
+`cleaning="all"`
+
+Provide an option of "all" or "none". ARC will lowercase and remove non-alphanumeric characters from all string columns if set to all.
+
+**When to use**
+
+- if you don't want ARC to do string cleaning for you
+
+`threshold:float=0.9`
+
+The probability threshold above which a pair is considered a match. This is used in the optimisation process.
+Requires a value between 0 and 1.
+
+**When to use**
+
+- use this to balance precision and recall - higher number -> more emphasis on precision, less on recall.
+  Lower value -> more emphasis on recall, less on precision.
+
+
+`true_label:str=None`
+
+The column name which contains the true record IDs. If provided, ARC will automatically score its model against the known true labels.
+
+**When to use**
+
+- if you have an already deduplicated / linked set of data that you want to use as a benchmark to assess ARC's performance.
+
+`random_seed:int=42`
+Random seed for controlling the psuedoRNG. Set for reproducibility between runs.
+
+`metric:str="information_gain_power_ratio"`
+
+Which metric should ARC use for it's optimisation process. This will be deprecated in future releases and should not be changed.
+
+
+`sample_for_blocking_rules=True`
+
+Downsample larger datasets to speed up blocking rule estimation.
\ No newline at end of file
diff --git a/python/arc/__init__.py b/python/arc/__init__.py
index 91f4cf5..0fc5978 100644
--- a/python/arc/__init__.py
+++ b/python/arc/__init__.py
@@ -1,3 +1,3 @@
 from .sql import enable_arc
-
+from .utils import utils
 __version__ = "0.1.5"
diff --git a/python/arc/autolinker/autolinker.py b/python/arc/autolinker/autolinker.py
index f77278c..498e3bd 100644
--- a/python/arc/autolinker/autolinker.py
+++ b/python/arc/autolinker/autolinker.py
@@ -1272,16 +1272,3 @@ def get_clustering_metrics(self, clusters, true_label):
     }
 
     return cluster_scores
-
-  def delete_all_linking_experiments(self):
-    if input("WARNING - this will delete all your ARC generated MLFlow experiments, Type 'YES' to proceed" ) != "YES":
-      return
-    username = self.spark.sql('select current_user() as user').collect()[0]['user']
-    pattern = f"%{username}/Databricks Autolinker%"
-
-    client = MlflowClient()
-    experiments = (
-      client.search_experiments(filter_string=f"name LIKE '{pattern}'")
-    )  # returns a list of mlflow.entities.Experiment
-    for exp in experiments:
-      client.delete_experiment(exp.experiment_id)
\ No newline at end of file
diff --git a/python/arc/utils/__init__.py b/python/arc/utils/__init__.py
new file mode 100644
index 0000000..e69de29
diff --git a/python/arc/utils/utils.py b/python/arc/utils/utils.py
new file mode 100644
index 0000000..a895447
--- /dev/null
+++ b/python/arc/utils/utils.py
@@ -0,0 +1,24 @@
+from mlflow import MlflowClient
+
+def delete_all_default_linking_experiments(spark):
+    '''
+    This method removes all MLFlow experiments created by ARC with the default naming convention.
+
+    :param spark: a spark object. available by default in databricks.
+
+
+    Returns
+    -------
+
+    '''
+    if input("WARNING - this will delete all your ARC generated MLFlow experiments, Type 'YES' to proceed") != "YES":
+        return
+    username = spark.sql('select current_user() as user').collect()[0]['user']
+    pattern = f"%{username}/Databricks Autolinker%"
+
+    client = MlflowClient()
+    experiments = (
+        client.search_experiments(filter_string=f"name LIKE '{pattern}'")
+    )  # returns a list of mlflow.entities.Experiment
+    for exp in experiments:
+        client.delete_experiment(exp.experiment_id)
\ No newline at end of file


Test patch:
diff --git a/.github/workflows/integration-test-aws-pr.yml b/.github/workflows/integration-test-aws-pr.yml
index 8f8b80b..93dfdee 100644
--- a/.github/workflows/integration-test-aws-pr.yml
+++ b/.github/workflows/integration-test-aws-pr.yml
@@ -12,7 +12,7 @@ jobs:
      - name: Run a databricks notebook
        uses: databricks/run-notebook@v0
        with:
-         local-notebook-path: RUNME.py
+         local-notebook-path: integration-tests/RUN_TESTS.py
          git-commit: ${{ github.event.pull_request.head.sha }}
          databricks-host: https://e2-demo-west.cloud.databricks.com
          databricks-token: ${{ secrets.DEPLOYMENT_TARGET_TOKEN_AWS }}
diff --git a/.github/workflows/integration-test-aws-push.yml b/.github/workflows/integration-test-aws-push.yml
index f4e39c1..9be8ecc 100644
--- a/.github/workflows/integration-test-aws-push.yml
+++ b/.github/workflows/integration-test-aws-push.yml
@@ -16,7 +16,7 @@ jobs:
      - name: Run a databricks notebook
        uses: databricks/run-notebook@v0
        with:
-         local-notebook-path: RUNME.py
+         local-notebook-path: integration-tests/RUN_TESTS.py
          git-commit: ${{ github.sha }}
          databricks-host: https://e2-demo-west.cloud.databricks.com
          databricks-token: ${{ secrets.DEPLOYMENT_TARGET_TOKEN_AWS }}
diff --git a/.github/workflows/integration-test-gcp-pr.yml b/.github/workflows/integration-test-gcp-pr.yml
index c695d12..928dd9e 100644
--- a/.github/workflows/integration-test-gcp-pr.yml
+++ b/.github/workflows/integration-test-gcp-pr.yml
@@ -12,7 +12,7 @@ jobs:
      - name: Run a databricks notebook
        uses: databricks/run-notebook@v0
        with:
-         local-notebook-path: RUNME.py
+         local-notebook-path: integration-tests/RUN_TESTS.py
          git-commit: ${{ github.event.pull_request.head.sha }}
          databricks-host: https://416411475796958.8.gcp.databricks.com
          databricks-token: ${{ secrets.DEPLOYMENT_TARGET_TOKEN_GCP }}
diff --git a/.github/workflows/integration-test-gcp-push.yml b/.github/workflows/integration-test-gcp-push.yml
index 3b2c8fd..d6cbdce 100644
--- a/.github/workflows/integration-test-gcp-push.yml
+++ b/.github/workflows/integration-test-gcp-push.yml
@@ -16,7 +16,7 @@ jobs:
      - name: Run a databricks notebook
        uses: databricks/run-notebook@v0
        with:
-         local-notebook-path: RUNME.py
+         local-notebook-path: integration-tests/RUN_TESTS.py
          git-commit: ${{ github.sha }}
          databricks-host: https://416411475796958.8.gcp.databricks.com
          databricks-token: ${{ secrets.DEPLOYMENT_TARGET_TOKEN_GCP }}
diff --git a/.github/workflows/integration-test-msa-pr.yml b/.github/workflows/integration-test-msa-pr.yml
index eeeebc0..6a6a309 100644
--- a/.github/workflows/integration-test-msa-pr.yml
+++ b/.github/workflows/integration-test-msa-pr.yml
@@ -11,7 +11,7 @@ jobs:
      - name: Run a databricks notebook
        uses: databricks/run-notebook@v0
        with:
-         local-notebook-path: RUNME.py
+         local-notebook-path: integration-tests/RUN_TESTS.py
          git-commit: ${{ github.event.pull_request.head.sha }}
          databricks-host: https://adb-984752964297111.11.azuredatabricks.net
          databricks-token: ${{ secrets.DEPLOYMENT_TARGET_TOKEN_MSA }}
diff --git a/.github/workflows/integration-test-msa-push.yml b/.github/workflows/integration-test-msa-push.yml
index 3c3e3b2..ab910d2 100644
--- a/.github/workflows/integration-test-msa-push.yml
+++ b/.github/workflows/integration-test-msa-push.yml
@@ -15,7 +15,7 @@ jobs:
      - name: Run a databricks notebook
        uses: databricks/run-notebook@v0
        with:
-         local-notebook-path: RUNME.py
+         local-notebook-path: integration-tests/RUN_TESTS.py
          git-commit: ${{ github.sha }}
          databricks-host: https://adb-984752964297111.11.azuredatabricks.net
          databricks-token: ${{ secrets.DEPLOYMENT_TARGET_TOKEN_MSA }}
diff --git a/.github/workflows/publish-to-test-pypi.yml b/.github/workflows/publish-to-test-pypi.yml
new file mode 100644
index 0000000..73a0b1d
--- /dev/null
+++ b/.github/workflows/publish-to-test-pypi.yml
@@ -0,0 +1,33 @@
+name: Publish Python 🐍 distribution 📦 to PyPI and TestPyPI
+
+on:
+  pull_request:
+
+jobs:
+  build-n-publish:
+    name: Build project and publish to TestPyPI
+    runs-on: ubuntu-20.04
+    strategy:
+      matrix:
+        python: [3.8.10]
+        spark: [3.2.1]
+    environment:
+      name: testpypi
+      url: https://test.pypi.org/p/databricks-arc/
+    permissions:
+      id-token: write  # IMPORTANT: mandatory for trusted publishing
+
+    steps:
+      - name: checkout code
+        uses: actions/checkout@v2
+      - name: build jar
+        uses: ./.github/actions/scala_build
+      - name: build python
+        uses: ./.github/actions/python_build
+      - name: Publish a Python distribution to PyPI
+        uses: pypa/gh-action-pypi-publish@release/v1
+        with:
+          user: __token__
+          password: ${{ secrets.TEST_PYPI_TOKEN }}
+          packages-dir: python/dist/
+          repository-url: https://test.pypi.org/legacy/
diff --git a/integration-tests/RUN_TESTS.py b/integration-tests/RUN_TESTS.py
new file mode 100644
index 0000000..2ef0141
--- /dev/null
+++ b/integration-tests/RUN_TESTS.py
@@ -0,0 +1,83 @@
+# Databricks notebook source
+# MAGIC %md This notebook sets up the companion cluster(s) to run the solution accelerator. It also creates the Workflow to illustrate the order of execution. Happy exploring!
+# MAGIC 🎉
+# MAGIC
+# MAGIC **Steps**
+# MAGIC 1. Simply attach this notebook to a cluster and hit Run-All for this notebook. A multi-step job and the clusters used in the job will be created for you and hyperlinks are printed on the last block of the notebook.
+# MAGIC
+# MAGIC 2. Run the accelerator notebooks: Feel free to explore the multi-step job page and **run the Workflow**, or **run the notebooks interactively** with the cluster to see how this solution accelerator executes.
+# MAGIC
+# MAGIC     2a. **Run the Workflow**: Navigate to the Workflow link and hit the `Run Now` 💥.
+# MAGIC
+# MAGIC     2b. **Run the notebooks interactively**: Attach the notebook with the cluster(s) created and execute as described in the `job_json['tasks']` below.
+# MAGIC
+# MAGIC **Prerequisites**
+# MAGIC 1. You need to have cluster creation permissions in this workspace.
+# MAGIC
+# MAGIC 2. In case the environment has cluster-policies that interfere with automated deployment, you may need to manually create the cluster in accordance with the workspace cluster policy. The `job_json` definition below still provides valuable information about the configuration these series of notebooks should run with.
+# MAGIC
+# MAGIC **Notes**
+# MAGIC 1. The pipelines, workflows and clusters created in this script are not user-specific. Keep in mind that rerunning this script again after modification resets them for other users too.
+# MAGIC
+# MAGIC 2. If the job execution fails, please confirm that you have set up other environment dependencies as specified in the accelerator notebooks. Accelerators may require the user to set up additional cloud infra or secrets to manage credentials.
+
+# COMMAND ----------
+
+# DBTITLE 0,Install util packages
+# MAGIC %pip install git+https://github.com/databricks-academy/dbacademy@v1.0.13 git+https://github.com/databricks-industry-solutions/notebook-solution-companion@safe-print-html --quiet --disable-pip-version-check
+
+# COMMAND ----------
+
+from solacc.companion import NotebookSolutionCompanion
+
+# COMMAND ----------
+
+job_json = {
+    "timeout_seconds": 28800,
+    "max_concurrent_runs": 1,
+    "tags": {
+        "usage": "solacc_testing",
+        "group": "PUB",
+        "accelerator": "auto-data-linkage"
+    },
+    "tasks": [
+        {
+            "job_cluster_key": "auto_data_linkage_cluster",
+            "notebook_task": {
+                "notebook_path": f"linking_test"
+            },
+            "task_key": "auto_data_linkage_01"
+        },
+        {
+            "job_cluster_key": "auto_data_linkage_cluster",
+            "notebook_task": {
+                "notebook_path": f"deduplication_test"
+            },
+            "task_key": "auto_data_linkage_02"
+        }
+    ],
+    "job_clusters": [
+        {
+            "job_cluster_key": "auto_data_linkage_cluster",
+            "new_cluster": {
+                "spark_version": "12.2.x-cpu-ml-scala2.12",
+                "spark_conf": {
+                    "spark.databricks.delta.formatCheck.enabled": "false"
+                },
+                "num_workers": 2,
+                "node_type_id": {"AWS": "i3.xlarge", "MSA": "Standard_DS3_v2", "GCP": "n1-highmem-4"},
+                "custom_tags": {
+                    "usage": "solacc_testing",
+                    "group": "PUB",
+                    "accelerator": "auto-data-linkage"
+                },
+            }
+        }
+    ]
+}
+
+# COMMAND ----------
+
+dbutils.widgets.dropdown("run_job", "False", ["True", "False"])
+run_job = dbutils.widgets.get("run_job") == "True"
+NotebookSolutionCompanion().deploy_compute(job_json, run_job=run_job)
diff --git a/integration-tests/data/arc_febrl1.csv b/integration-tests/data/arc_febrl1.csv
new file mode 100644
index 0000000..ed52a2f
--- /dev/null
+++ b/integration-tests/data/arc_febrl1.csv
@@ -0,0 +1,1001 @@
+given_name,surname,street_number,address_1,address_2,suburb,postcode,state,date_of_birth,recid,uid
+null,waller,6,tullaroop street,willaroo,st james,4011,wa,19081209,6988048,0
+lachlan,berry,69,giblin street,killarney,bittern,4814,qld,19990219,7364009,1
+deakin,sondergeld,48,goldfinch circuit,kooltuo,canterbury,2776,vic,19600210,2635962,2
+kayla,harrington,null,maltby circuit,coaling,coolaroo,3465,nsw,19150612,9004242,3
+luke,purdon,23,ramsay place,mirani,garbutt,2260,vic,19831024,8099933,4
+null,trevorrow,16,dumas street,2/98-latchford barracks,mount immaney,2281,wa,19530313,4811642,5
+darcie,turtur,10,blacket street,eureka,beverly hills,2263,nsw,null,2025650,6
+william,bishop,21,neworra place,apmnt 65,worongary,6225,qld,19490130,9773843,7
+null,lombardi,36,yerralee road,leisure living vlge,carlsruhe,3149,qld,19870919,1613132,8
+null,julius,52,florey drive,north stirling downs,coolaroo,2259,qld,19630521,1797144,9
+abbey,fit,13,kosciusko avenue,the wharf complex,yass,2594,nsw,19870510,7661096,10
+isabella,lodder,156,messenger street,tongbong sanctuary,bayswater,4870,vic,19650714,2790666,11
+anthony,beazley,12,birubi place,currandina,flemington,2477,qld,19730924,6558077,12
+alia,streich,74,maranoa street,rocky bend,rowville,6152,vic,19790418,1975340,13
+null,stephenson,52,null,north stirilng downs,coolaroo,2259,qld,19630521,1797144,14
+alissa,kilmartin,37,reveley crescent,crown allot,wolumla,6210,nsw,19041118,7994055,15
+jacob,lyden,25,haddon street,glenview,woodville north,2226,qld,19910424,6426415,16
+evangelia,shepherd,9,crichton crescent,mirilla,lutwyche,2794,qld,19900909,3877160,17
+paige,lowe,3,chave street,keelee,gowwie,2906,qld,19140707,2534242,18
+alexa-rose,lowe,67,ballarat street,the willows,westwood,4011,nsw,19691123,2624157,19
+kirra,brock,93,clarnette place,backwoodlands,orbost,2528,vic,19231209,5107876,20
+paige,lowe,3,chave street,keelee,gowrie,2906,qld,19140707,5860195,21
+liam,mcphail,94,langdon avenue,jandarra,patterson lakes,3029,qld,19920711,2109237,22
+john,van't hof,5,morshead drive,ocean crt (cnr beach s street,maida vale,2745,vic,19360109,9574272,23
+chloe,reitler,14,daley crescent,halfway hill,ballina,6164,qld,19171028,7162356,24
+elle,nicklason,5,beetaloo street,null,trafalgar,5107,wa,19460919,5092794,25
+kyle,limbert,7,dandenong court,the pinnacle,springwood,3143,nsw,19080527,2479285,26
+jacob,di chiera,61,bimbiang crescent,the manor garden,unanderra,4680,nsw,19610831,2176932,27
+dakota,morrison,33,longmore crescent,three acres,brookvale,2710,nsw,19151229,5031530,28
+spyke,schumann,78,clark close,pine tree,tannum sands,2326,nsw,19020422,6677685,29
+anneliese,clarke,109,arawang,langdon avenue,pakenham,3114,nsw,19000404,9135773,30
+amelia,ryan,35,sidaway street,jemalong retrmnt vlge,avalon,3184,nsw,19230301,1613259,31
+kydan,mccarthy,67,clemenger street,the points holsteins,fairlawn,6415,nsw,19720518,6527653,32
+georgia,gryosler,8,foxall street,willaroo,lockwlod,3338,nsw,19321106,8000491,33
+bailey,campbell,136,porteous crescent,teasdale park,albany creek,3942,vic,19830721,8875552,34
+lachlan-john,green,5,null,canobolas caravn park,blacktown,2089,nsw,19610527,1361558,35
+kelsey,halligan,3,dilboong place,rosetta village,kedron,6027,nsw,19900301,8576385,36
+paterson,madeleine,13,null,null,young,5045,nsw,19300302,2277800,37
+tori,keoh,1,tallara p arkway,null,wangaratta,2223,vic,19601101,5089409,38
+null,siwggs,1,givens srteet,bonnington,kyogle,2170,qld,19521123,1256379,39
+alexa-rose,lowe,67,ballara t street,the willows,westwood,4012,nsw,19691123,2624157,40
+gemaley,matthews,388,goessling place,evergreen,pennant hills,3194,nsw,19730829,4691516,41
+joshua,rickett,1,burraly corut,malladuo,belleve hill,2439,act,19030121,4310453,42
+willow,ryan,null,bradfield street,sec 209 clifton farm,mount c olah,4035,nsw,19930803,3832827,43
+emma,green,30,learmonth drive,wallace heights,boundary bend,2747,qld,19310903,9262880,44
+kelsey,halligan,3,dilboong place,null,kedron,6027,nsw,19900301,8576385,45
+jack,berry,4,esperance street,landen,black rock,2763,vic,19020712,6151476,46
+alexandra,boulter,22,tazewell circuit,berkeley vlge,potts point,2450,vic,19420722,5269403,47
+null,hefford,1,armstrong crescent,null,payneham,5502,vic,19440805,9162672,48
+cambetll,hursey,3,corona place,sweet w atdrs,toowoomba,3980,sa,19870526,7118146,49
+hayd en,geraghty,41,macneilplace,willow wood,frenchs forest,3011,vic,19121130,8624403,50
+null,querzoli,75,sloane place,park lane estate,mount eliza,3083,nsw,19860522,4577875,51
+monique,gergely,21,harwood court,hybernia park,sherwood,2207,nsw,19790807,7375144,52
+layla,ryan,720,tregellas crescent,carinya vlge,collinswood,3400,nsw,19920526,7041054,53
+justin,clarke,12,ahmatt street,moline ivllage,rowvil le,6150,vix,19990220,6779471,54
+rourke,webv,5,welsby pnlace,bungarra,island b each,6020,nsw,19780601,8955738,55
+kyra,webb,9,boolee street,null,freemans reach,4870,vic,19470429,3308042,56
+imogen,akroyd,8,everard street,strathalan community,marsfield,3219,qld,19361218,4817613,57
+chloe,gillard,3,crispcir cuit,ban skia,kenilworth,4218,wa,19200820,8072668,58
+joshua,garthwaite,24,mcclintock street,hallsville,avoca,3175,nsw,19050703,5226643,59
+lachlan,jukic,2,morgan crescent,parklea,raymond terrace,2250,nsw,19780702,4027998,60
+cononr,lutz,55,burkitt street,connoisseur quarterhorses,the basin,3166,vic,19970727,6308695,61
+rosie,reid,256,coghill close,null,blackown,5253,act,19770714,8349921,62
+darcy,jolly,7,antill street,agarabi,godwin beach,4173,nsw,19891008,7208848,63
+isabella,defrancesco,24,meyrick place,marx garden world,wellingotn point,2759,vic,19040819,8342399,64
+jessica,noble,42,boolimba crescent,lashbrooke,cranbourne,2219,vic,null,7984432,65
+charlotte,robson,23,nicholas street,kellwood,null,2280,vic,19620503,9419113,66
+thomas,wijedasa,152,powers place,carney's arcade,bowen hills,6021,tas,19820415,8129413,67
+zakariah,hoyland,82,rooth place,pindara medical centre,dianella,6164,qld,19900707,6754522,68
+chloe,jayaraman,33,eungella street,victoria mill estate,broken hill,3172,nsw,19590818,4910207,69
+alice,bellchambers,267,swanton street,tallow-wood,blacktown,3136,vic,19381230,8676818,70
+joshua,rickett,1,burraly court,malladup,new farm,2439,act,19030121,4310453,71
+eliza,sharon,120,brinsmead street,chactor,greenacre,3214,nsw,19950112,2497676,72
+fraser,marlow,8,evelyn owen crescent,yarandoo wing,ermington,3000,nsw,19731008,2195286,73
+null,durr,11,cochrane crescent,spyla w farm,south perth,2795,wa,19810628,6794161,74
+liam,fitzpatrck,55,schonell circuit,null,coffs harbour,6285,nsw,19321022,5663386,75
+brydon,webb,11,walker crescent,bugoren,charlestown,6258,nsw,19190528,4191569,76
+chloe,reitler,14,daley crescent,halfway hill,ballinga,6146,qld,19171028,7162356,77
+bordin,brody,45,canberra centre (next to medicare),corombie,figtree,2485,qld,19510904,8728984,78
+jacynta,hoffman,4,marrakai street,tantallon,munruben,2042,nsw,19320417,4705328,79
+olivia,null,65,hobart place,sec 791,lalor park,2090,vic,19200321,2042439,80
+mikayla,caruana,37,torrens street,boronia,dianella,2761,qld,19590901,6041595,81
+justin,clarke,12,ahmatt street,moline village,rowville,6105,vic,19990220,6779471,82
+null,jervis,19,rickard place,evergreen,east gresford,4060,qld,19330212,1333372,83
+catherine,couzens,41,balamara street,landen,frankston,2099,wa,19080905,8642862,84
+tyron,dixon,45,grund place,zintara,munruben,2167,vic,19940207,8837588,85
+rhys,jeffries,3,whistler place,pepin,thornlands,4216,nsw,19200821,3750223,86
+jordan,lavis,12,ranken place,rosebery hill,piggabeen,0870,nsw,19640314,4307152,87
+null,collinson,4,boddington crescent,null,illabo,2261,nsw,19840807,2427882,88
+kyle,brain,256,coghill close,bay city plaza,blacktown,5253,act,19770714,8349921,89
+angelina,reece,34,wray pylace,brookto nh ouse,toowoomba,7140,qld,19350613,7961545,90
+francesco,finl ay,159,chippindall circuit,null,burleigh waters,6236,nsw,19000713,7391179,91
+georgia,kyriacou,21,carumbi place,kindberg,banyo,2340,vic,19280827,9890107,92
+robert,ranson,9,crease place,null,calliope,4053,vic,19600702,1133078,93
+isabekle,spratt,16,bastow circuit,kurrajogn,shoalwater,2587,qld,null,5074015,94
+null,sedecki,12,derrilin place,longstay caravn park,blackheath,4061,nsw,19801116,2990764,95
+thomas,graue,4,dinnison gcircuit,vila 178,clovelly,4123,wa,19120911,5760570,96
+matthew,green,28,banks srteet,old ffarm,mooroofbark,6108,nsw,19161230,6391014,97
+darcie,fleet,13,fellows street,braeburn,somerton park,2284,vic,19950307,1563705,98
+null,morrison,35,minehan place,lumley,burn side,5167,tas,19620920,1541342,99
+jarvis,ryan,82,buckman place,null,inneston,2614,vic,19700922,6893056,100
+cooper,culbertson,30,dow street,null,sea lake,4020,qld,19951031,7943540,101
+jord an,blake,2,moore street,fleurbaix,armida le,2477,vic,19790710,6981185,102
+alissa,kilmatrin,37,reveley rescent,crown allot,wolumla,6210,nsw,19041118,7994055,103
+connor,lutz,55,burkitt street,connoisseur quarterhorses,the basin,3166,vic,19970727,6308695,104
+giuliana,hammill,83,cobby sjtteet,parish takowahl,potts point,3736,nsw,19941108,1238636,105
+kyle,ryan,15,ind place,rocklea,port augusta,4305,nsw,19411102,2059726,106
+madeleine,paterson,null,brigalow street,null,young,5045,nsw,19300302,2277800,107
+caitkin,null,null,rymill place,airforce memor ial estate,mount gravatt,3146,vic,19980804,8342575,108
+james,bradshw,null,reader court,claytoh farm,milo park,2322,null,19060730,4960757,109
+jacob,varwano,null,magrath crescent,bandrea,auburn,2210,vic,19411202,1499165,110
+jordan,thaba,2,moore street,fleurbaix,armidale,2477,vic,19790710,6981185,111
+varano,max,16,birrigaisquare,arabian stud ielkette park,dandenong north,6051,vic,19700713,7737012,112
+ashley,white,25,jacka crescent,red gate,marmong point,7250,wa,19280224,5596702,113
+mccarrhy,timothy,517,sidawawytreet,ballawina stud,northcliffe,4350,nsw,19530607,6601394,114
+sar,coffey,3,mckinlay street,beechowrth,huonville,4558,vic,19150306,1706049,115
+georgia,wilde,2,warra creek,freda gibsom circuit,null,2154,sa,19550212,6569089,116
+holly,siemers,8,strachan place,null,beaudesert,2104,sa,19610319,4858108,117
+caleb,dixon,1429,glenmore,lambel lvose,toronto,3634,tas,19860422,3569365,118
+anneliese,clarke,16,langdon avenue,arawang,pakenham,3114,nsw,19000404,9135773,119
+william,lipscombe,14,clift crescent,willandra,malvern east,3073,nsw,19260101,3537397,120
+marianne,rees,6,nerli place,weeroona,mansfield,3644,sa,19770628,4235561,121
+brinsmead,amelia,73,mackennal street,belgradezpark,wellington point,2046,vic,19810831,4182058,122
+alexa-rose,campblel,17,neeld place,null,narooma,2205,qld,19490410,1531199,123
+timothy,mccarthy,517,sidaway street,ballawinna stud,northcliffe,4350,nsw,19530607,6601394,124
+kyle,rundle,107,greenhood place,willowd trees,brigthon,5074,nsw,19460902,6532949,125
+nicholas,donaldson,3,null,forest glen resort,kogarah,3136,nrw,19211230,9407354,126
+jemima,mcveigh,14,becker uplace,null,armidale,3107,sa,19600320,8051824,127
+callum,grossman,3,downard street,st vincents medical centre,wangaratta,4211,nsw,19770402,7743698,128
+jade,p aine,21,rose scotet circuit,null,taringa,4405,nsw,19191105,7184624,129
+null,wilde,47,the canberra hospital,darwiniaerrace,melville,3929,vic,19270428,6646723,130
+lacey,clarke,54,starritt place,st fra ncis vlge,goomalibee,2600,wa,19700209,9397537,131
+nicholas,campbell,87,poynton street,karma park,albion park,4655,qld,19041212,9882754,132
+null,coffey,8,buller crescent,sheep sttion,null,4800,vic,19490409,3924737,133
+jacynta,hoffman,4,marrakai srteet,tantallon,munruben,2153,nsw,null,4705328,134
+carlin,clarke,13,loch street,anzac house (cnr archer s street,kyabram,5040,vic,19891111,3670162,135
+caleb,radic,17,moon place,cooralook,south perth,4305,nsw,19471021,6519229,136
+nicholas,prideaux,9,hicks street,loccn 7229,bondi junction,5011,sa,19420429,1107619,137
+mikayla,egan,109,heidelberg street,peppermint springs bendoc,mermaid waters,3875,null,19731003,8652616,138
+isabella,fitzptarick,10,tompsitt court,sanlyn,corio,6340,nsw,19771224,5262341,139
+lily,huxley,45,currie crescent,waribalda,beverly hills,2073,qld,19510420,4379936,140
+hananh,isola,26,barlow steet,aptmnt 240,clayton,4350,act,19320106,9013800,141
+emma,gildea,2,mccawley street,nr cnr school art,malvern,6722,qld,19440429,8069828,142
+reeve,mileq,2,renwick lwtreet,yarra bee,barwon heads,2342,nsw,19810406,1067923,143
+sophie,lovelock,26,durraw een,chant street,williamstown,2650,vic,19371233,3072763,144
+jacob,green,2,jerome place,kalke,kings cross,6050,act,19770519,4058972,145
+nicholas,ryan,59,windeyer street,mafeking,belmont,4031,sa,19421110,8430399,146
+mccarthy,kydan,67,clemengerxstreet,the points holsteins,fairlawn,6452,nsw,19729518,6527653,147
+amelia,kimmonth,25,grace place,welikrak,barwon heads,2012,vic,null,5751890,148
+adam,crouch,930,crozier circuit,willow side jersey stud,cressy,3777,nsw,19011112,3380075,149
+darcie,turtur,10,blacket street,eureka,beverly hills,2263,nsw,19570422,2025650,150
+hugo,alderson,440,mccombie street,rosedale,malvern east,5023,nsw,19150416,5572942,151
+kyle,glass,87,wyselaskie circuit,apt 710,wyong,3390,nsw,19241221,6908529,152
+isabella,walpole,31,donnison place,gairnshiel rsd 225,dapto,2390,nsw,19881222,6386893,153
+william,green,68,snowden place,heatherdale,burringbar,2750,vic,19150804,9080722,154
+null,green,30,wallace h eights,learmonth drive,boundary bend,2747,qld,19310903,9262880,155
+tiahana,gaskin,12,crispcircuit,community hospt lmedcl centre,kurri kurri,2624,vic,19190716,4264536,156
+chloe,whetstone,59,chuculba crescent,telooma,hamilton,3000,vic,19910803,1173489,157
+chloe,kyriacou,12,mulga street,inglewood,mitcham,4031,nsw,19480827,3985717,158
+thomas,shepherd,4,dinnison circuit,vila 178,clovelly,4123,wa,19120911,6706820,159
+null,hoffman,1,brooker street,null,lockles,2070,wa,19760226,6891487,160
+sarah,bradshaw,4,de littlezcircuit,mermaid plaza dentalyrcentre (cnr gold co,canley vale,3065,nsq,19090923,1686730,161
+alexa-rose,campbell,17,neeld place,milton,narooma,2205,qld,19490410,1531199,162
+joshua,dolan,98,colborne place,warowie,lindfield,3568,nsw,19521116,9500871,163
+hannah,green,65,jacka crescent,rmbe 770,sunshine beach,2086,nsw,19350612,4219398,164
+mikayla,caruana,37,torrens street,bor onia,dianelka,2716,qld,19590901,6041595,165
+india,negrean,1,barringer street,sherwood,parkinson,3168,nsw,19860923,2097928,166
+daniel,morrison,25,bellchambers crescent,ascot park,point samson,4514,vic,19620316,8699433,167
+benjamin,kirchener,12,stuart street,wurrami,theodore,2620,wa,19751110,1766048,168
+joel,bordin,45,canberra centre (next to medicare),corombie,figtree,2485,qld,19510904,8728984,169
+jayde,grosser,41,gundaroo road,null,baringhup,3173,vic,19281101,6423689,170
+sienna,georege,22,hoad place,kondalilla,burwood east,3860,nsw,19661026,9123905,171
+joshua,garthwaite,42,mcclintock street,hallsville,avoca,3175,nsw,19050703,5226643,172
+bne,browen,42,tillyard drive,willow wood,mentone,2231,vic,19241201,5216720,173
+caitlin,null,null,rymill place,airforce memorial estate,mount gravatt,3146,vic,19980804,8342575,174
+jackson,cmielewski,76,angliss place,brookvale,tweed heads south,3082,vic,19141022,7997511,175
+rhys,jeffries,3,whistlerplace,pepin,thornlands,4216,nsw,19200821,3750223,176
+tia,gols,101,boron ia drive,the cabin,sunnybank,3020,sa,19961121,7459384,177
+michael,nguyen,null,roseworthy crescent,underbank,colac,2198,qld,19011004,5908395,178
+samuel,campbell,32,mosman place,null,north beach,2617,nsw,19430607,5538479,179
+laura,stubbs,68,eyre street,westwood homestead,null,4871,qld,19621222,1310830,180
+andrew,bradshaw,238,berne crescent,civic centre,greensborough,3095,sa,19941118,6385462,181
+barnaby,shereston,1,margany street,flt 2a,keysborough,5045,wa,19961009,8585130,182
+mitchell,mcconnel,5,templeton street,gle n ayre,camdn,2500,vic,19370615,3413425,183
+lexie,cochrane,null,paul coe crescent,ararat retirement village,auburn,2904,tas,19320809,4618363,184
+isabella,ayrse,1075,arthur circle,grevilal est,glewnood,5121,qld,19531230,9208787,185
+riley,siviur,15,tubb plqace,roxor,null,3186,nsw,19080709,4038966,186
+lucas,rawldings,11,paul coe drescent,null,wanniassa,6122,vic,19141102,5141107,187
+peta,coffey,8,buller crescent,sheep station,null,4800,vic,19490409,3924737,188
+alice,bellchambers,267,swanton wtreet,tallow-bwood,blacjtwon,3136,vic,19381230,8676818,189
+darcy,jolly,7,antill street,agar abi,marsfield,4173,nsw,null,7208848,190
+ethan,scrbak,2489,de burgh street,glendaurel,como,3156,vic,19100409,8766958,191
+kyleh,reuter,185,padbury street,burnside,ballrat,4703,nsw,19251011,6546519,192
+jasmine,frahn,55,mckail crescent,keanlr,cooktown,3020,nsw,19040608,4013078,193
+darcy,hochunli,22,lawrence close,null,sefton park,2380,tas,19370607,8750602,194
+mcphail,benjamin,33,straliaplace,martins wele station,umina beach,4814,qld,19540903,6563107,195
+zachary,lodge,28,monaro highway,akenfield,sandy bay,4109,qld,19790613,3535105,196
+nicholas,donaldson,3,null,forest glen resort,kogarah,3136,nsw,19211230,9407354,197
+jarrod,polak,10,parsons street,nareen gardens hostel,elsternwick,6317,nsw,19330222,4360449,198
+zachary,lodge,28,monaro highway,akenfield,malvern east,2200,qld,19790613,2678628,199
+thomas,wijedasa,152,powers place,carney's arcade,bowen hills,6021,tas,19820415,8192413,200
+null,kolodina,66,darmody lstreet,split solitary caravn park,wheeo,2317,vic,19610520,1601763,201
+clodqgh,green,68,snowden place,heatherdale,burringbar,2750,vic,19150804,9080722,202
+annabelle,clapham,312,martley circuit,aptmnt42,worrigee,4160,nsw,19960703,9692668,203
+thomas,papanikolopoulos,14,doolanz court,frank jeffry hostel,ulladulla,4356,vic,19241204,7849076,204
+bryce,fabel,3,nangor street,sherwood,beaumaris,2484,qld,19540519,5014276,205
+lilian,griffen,155,lambert street,broadbridge manor,kilsyth,4829,vic,19381213,7355934,206
+edward,ryan,35,sidaway street,jemalong retrmnt vlge,avalon,3184,nsw,19230301,4392917,207
+kyle,nobel,207,null,rowethorpe,ellaswood,3148,nsw,null,8121172,208
+ash,worsley,3,russell drysdale crescent,gull cottage,greensborough,4032,nsw,19960512,5164697,209
+meaney,shantal,15,cockcroftavenue,sebaank,pakenham,2114,vic,19810717,9937720,210
+eliza,sharon,120,brinsmeda street,chacor,greenapre,3214,nsw,19950112,2497676,211
+daniel,marcic,11,barnes place,country club,wynnum west,5108,wa,19310308,8550692,212
+isabelle,lowe,27,kangairlla,grainger circuit,point turton,2450,vic,19980310,7007836,213
+bailey,blake,31,foxall street,nioka,miling,7322,nsw,null,4346541,214
+chloe,mayer,160,tuthill place,poldoe park,illawarra,2214,wa,19800326,3867233,215
+andrew,bradshaw,238,berne creswcent,null,greensborough,3059,sa,19941118,6385642,216
+jarrod,polak,10,parsons street,nareen gardens hostel,elsternwikc,6317,sa,19330222,4360449,217
+lexie,cochrane,null,ararat retirement village,paul coe crescent,auburrn,2940,tas,19320809,4618363,218
+finley,haeusler,27,noarlunga crescent,spring ridge,nambour,3180,vic,19660711,3025217,219
+joshus,maclennan,41,karsjl,roper nplace,eight mile plains,2795,qld,19700525,8982317,220
+kyle,ryany,14,ind place,rocklea,port augusta,4350,nsw,19411102,2059726,221
+anthony,berry,11,a'beckett street,hd curramulka,minchinbury,3930,sa,19370703,8728762,222
+adriana,rudd,7,dickson chiropractic dickson chambers,mcivor house,kilsyth,3910,qld,19501217,2016313,223
+thomas,morcom,16,alexandria street,trapalanda,vermont south,3714,vic,19800706,4569567,224
+phillip,ellem,41,outtrim avenue,ainslie house,avalon,3083,qld,19490902,3187616,225
+stephanie,rumbal,26,sollya place,erina ardns,galston,6285,nsw,19150614,5779913,226
+jack,spawton,43,jevons place,locn 1025,greenacre,6014,qld,19080317,3656816,227
+jayde,grosser,55,gundaroo road,null,baringhup,3173,vic,19281101,6423689,228
+null,kilby,null,collicott circuit,lassaan,success,2795,null,19040302,2952029,229
+carlin,clarwke,13,loch street,anzac house (cnr archer s street,kyabfam,5040,vic,19531013,3670162,230
+talia,ho,117,illingworth street,liselivan,wynnum west,3201,vic,19740208,2143356,231
+barnaby,shereston,1,margany street,null,keysborough,5045,wa,19961009,8585130,232
+karli,alderson,144,nulsen circuit,iowanna,tingalpa,3139,nsw,19510826,9541034,233
+cliare,priebe,39,stacy street,st francis vlge,clarence park,2145,vic,19900203,5569305,234
+beal,jared,9,palmer street,hollybiooke,southern river,4811,nsw,19400526,7240930,235
+brooklyn,manson,27,clive steele avenue,port fairy road,mount low,3450,vic,19710727,4493900,236
+sarah,schepers,118,tattersall crescent,coaake,kyogle,2212,nsw,19260705,3474290,237
+jack,burford,26,noongale court,kiandra,highgate,2614,vic,19101115,3392254,238
+emiily,kowald,18,daglish street,oakd ale,avalon,3030,vic,19250313,1590627,239
+zachary,pringle,16,evergood close,emerywood,satur,2160,nsw,19410809,2770074,240
+isaac,schepoirs,118,tattersall crescent,coaake,kyogle,2212,nsw,19861116,3474290,241
+emiily,wynne,51,bungonia street,glenbrae,burringbar,2210,sa,19760814,2855156,242
+brooke,pennell,32,jervois street,wheatlea,mooraobbin,4821,nsw,19641212,7585474,243
+isaac,quilliam,11,namatjira drive,delaware,geelong west,3072,nsw,19930926,1556150,244
+kob,korbut,1,kennewell xplace,heights estate,blacktown,4390,wa,19090519,9712042,245
+troy,reid,1,allan street,townview,page,2774,qld,19250727,3580821,246
+guthridge,thomas,6,brunswick circuit,tintagel,avondale heights,2073,vic,19080625,4255165,247
+erin,reid,2,temperley street,null,venusbay,2705,nsw,null,7311880,248
+nicholas,matthews,19,james smithi circuit,old wefeoona,dulwic hhill,4170,sa,19570116,9908807,249
+riley,ho,53,discovery street,rockview,orange,2011,wa,19000702,5033334,250
+fiona,worsley,59,gingana street,thorndale,dysart,2193,vic,19960902,9804147,251
+india,negrean,1,barringer street,sherwood,parkinson,3168,wa,19860923,2097928,252
+null,webb,11,walker crrscent,bugoren,charlstown,6258,nsw,19190528,4191569,253
+oakleigh,fitzpatrick,16,archdall street,clear creek,north amr,2122,gas,19580922,6731902,254
+cambell,hursey,3,spinifex street,sweet waters,toowoomba,3980,sa,19800422,7118146,255
+kirrah,mccarthy,79,laseron place,rowethorpe,winmalee,2073,wa,19400306,2583973,256
+pace,blake,12,goble street,ashleigh house bergen court,haddon,3056,vic,19751212,1356319,257
+anthony,berry,11,a'beckett street,null,minchinbury,3930,sa,19370703,8728762,258
+mitchell,scrbak,950,holyman street,berkeley vlge,safety bay,4300,qld,19060811,3592109,259
+claudia,eglin ton,50,glenelg street,lans downe,roleystone,4556,vic,19811022,6033620,260
+kyah,ngai,56,shoalhaven avenue,the nount,culburra,3083,vic,19630406,9351600,261
+callum,fauser,4,sandover circuit,eagleview,clare,3021,nsw,19970227,2873125,262
+null,harrington,103,adair street,the lakes retirement village,beaumont,4556,vic,19610510,7008577,263
+null,cogzell,1,armstrong crescent,cromdale,payneham,5502,vic,19440905,9162672,264
+nicholas,toso,null,carnegie crescent,moonah,mullumbimby,5109,wa,19710120,1111504,265
+ashley,heazdon,17,sheehy street,hawkins masonic vlge,warrandyte,3073,vic,19880814,7871445,266
+alexandra,hearn,7,booth crescent,unt 5,leichhardt,5037,null,19300818,3179241,267
+madeline,najm,13,boronia drive,poplar bend,evans head,4216,nsw,19420813,8918633,268
+hattie,white,20,ebden street,arkbaa,lalor,4750,nsw,19250301,2133927,269
+jade,matthews,49,ballonnest street,combog olong,young,6065,sa,19581231,5023792,270
+charlie,null,4,terewah circuit,legacy vlge,crib point,6167,nsw,19140715,1597495,271
+charlolte,satterley,41,counsel street,hilltop hostel rowethorpe,berwic,4556,nsw,19151209,4658473,272
+jake,berry,99,rivett street,null,kingstgorpe,2032,vic,null,7190150,273
+paine,melanie,4,andrewsstreet,rosetta village,salisbury east,3095,qld,19521118,2053758,274
+brandon,white,11,winser crescent,null,null,5159,qld,19911102,9695343,275
+holly,northway,3,collingridge street,brentwoo d vlge,sunshine,2324,vic,19850215,5614773,276
+ruby,riding,7,middletoz cwircuit,null,ceduna,4214,vic,19650615,9233472,277
+jessica,noble,42,boolimba crescent,lashbrooke,cranbourne,2219,vic,19940815,7984432,278
+nicholas,null,289,britten-jones drive,jabaru court,paddington,2000,vic,19970422,5062738,279
+mitchell,mcconnel,5,templeton street,glen ayre,camden,2500,vic,19370615,3413425,280
+hannah,spong,16,ellis place,wallaby,albany creek,3818,tas,19630205,8442523,281
+bianca,demetriou,4,jennings street,ahwahnee,dapto,3216,nsw,19261103,8066343,282
+jack,mcphie,155,shumack street,millard centre,clayton,4504,nsw,19720208,9497108,283
+sophie,conboy,24,learmonth drive,robe community health centre,st albans,3136,null,19360910,6636386,284
+broke,durbridge,63,boult place,banamba,bacchusm arsh,6147,sa,19951015,6266446,285
+jayde,van keulen,4,macgregor street,glend ower,cremorne,3566,vic,19150821,3018823,286
+david,stubbs,8,wehl place,minora,sunnybank,2782,qld,19490218,1825686,287
+emma,speight,8,investigator street,willarfo,tweed headsysouth,2770,vic,19630623,5067534,288
+cooper,richmond,5,pridha mstreet,star court arcade,wangaratta,5074,sa,19020301,1483265,289
+kayla,harrington,38,maltby circuit,coaling,coolaroo,3465,nsw,19150612,9004242,290
+thomas,wynne,920,leahy close,belmore rover,mitcham,3018,ua,19491220,9787222,291
+natasha,crossm an,null,null,mt gravat tresidence,glenthompson,4865,nsw,19140130,9295668,292
+robert,null,9,crease place,null,calliope,4053,vic,19600702,1133078,293
+null,durr,11,cochrane crescent,the rookery,south perth,2795,wa,19300521,6794611,294
+benjamin,campbell,19,goodenia street,busbey's flat,safety beach,2390,qld,19330926,4394755,295
+tia,gold,101,boronia drive,the cabin,sunnybank,3020,sa,19961121,7459384,296
+hannah,spong,32,elliswplace,wallaby,albany creek,3818,tas,19630205,8442523,297
+edward,hudd,6,namadgi circuit,null,forest hill,2456,vic,19740727,3658978,298
+joshua,bishojp,53,d'hagecurt,dovers flat,salt ash,3629,vic,19590302,8180464,299
+sophie,procter,23,paul coe crescent,the lakea village,port lincoln,4109,sa,19500121,4063636,300
+alexandra,mabokmarial,7,walker crescent,border park,erindale,2810,vic,19361207,1063069,301
+david,stubbs,8,wehl place,minroa,sunnybank,2782,qld,19490218,1825686,302
+joshua,porotschnig,9,shepherd street,rockview,limeburners creek,3121,nsw,19130422,7255490,303
+timothy,coffey,23,bacchus circuit,frankston caravan park,werrington,2450,vic,19600222,3125388,304
+emiily,morrison,314,cowper tsreet,belmore river,narrandera,5600,vic,19460111,4185791,305
+benjamin,cannekl,17,charleston street,oak glen,kununuodra,2090,nsw,19650504,7825348,306
+hayden,jolly,7,heard street,wonderusn,null,5034,nsw,19950410,3682559,307
+caleb,radic,17,moon place,coor alook,south perth,4305,nsw,19471021,6519249,308
+isobel,van wijk,18,federal highway,manresa,st albans,5108,vic,19970922,9247870,309
+dabiel,burleigh,22,milford street,brandywine stud,maroota south,3136,sa,19010517,4171356,310
+charlie,headon,17,sheehy street,hawkins masonic vlge,warrandyte,3073,vic,19880814,7871445,311
+charlie,clissold,41,allchin circuit,kinross,richardson,2196,vic,19270812,5711952,312
+alicia,bloomfield,2,hastings court,null,eleeana,2430,vic,19980422,4531319,313
+mikayla,giannaris,80,seabreeze acravan park,gollan street,abbotsford,2516,wa,19560825,4797460,314
+lachlan,basri,11,lindrum crescent,the shanty,carlsruhe,2130,qld,19261223,2379460,315
+giuliana,hammill,31,cobby street,parish talowahl,potts point,3763,nsw,19941108,1238636,316
+aidan,berry,4,lempriere crescent,null,null,3228,nsw,19280621,5213963,317
+william,headon,33,madigan street,wearmouth,emerald,2577,nsw,19540307,1158377,318
+molly,sushames,44,maclachlan street,null,south grafton,3191,qld,19910608,4688332,319
+brooklyn,manson,27,clive steele avenue,port fairy road,yagoona,3450,vic,19710727,4493900,320
+riley,frew,33,clive steele avenue,arcadia,vermont,7250,vic,19210304,3817732,321
+abbey,hoffman,17,brooker street,de boerderij,lockleys,2070,wa,19760226,6891487,322
+joel,mcgilc hrist,76,borrowdale street,rockles,bund akerg,4871,wa,19141214,8721264,323
+jessica,ballantyne,8,medworth crescent,rowethorpe,penrith,4213,nsw,19931103,1155058,324
+charlotte,matthews,101,britten-jones drive,gregory heights,forest hill,2745,vic,19760807,5346114,325
+brodee,kirkmoe,71,brisbane avenue,sherwood,toowoomba,4127,nsw,19370805,2701611,326
+thomas,stephenson,86,boas place,village walk (cnr high s street,eastwood,6630,nsw,19180524,3727893,327
+jardine,kallie,27,birrell street,c/-cummins theatre,milp erra,2218,vic,19091111,9225358,328
+amber,whillas,3,null,meilene retirement,whitfield,3987,nsw,19950320,3079240,329
+lachlan,berry,69,giblin street,killarney,bittern,4184,qld,19990219,7364009,330
+james,greenp,12,richards street,dentureclnc,laverton,4157,nsw,19170828,2482106,331
+riley,h o,54,discover ystreet,rockview,orange,2011,wa,19740212,5033334,332
+mia,purdon,23,ramsay place,mirani,gar butt,2206,vic,19831024,8099933,333
+silas,dent,16,wade street,tarwyn park,salisburyeast,2161,qld,19860810,7528902,334
+evangelia,shepherd,9,crichton crescent,mirilla,lutwylche,2794,qld,19900909,3877160,335
+adam,ciotti,13,hyatt place,kersey' south,coffs harbour,5039,nsw,19910920,9490795,336
+kobe,korbut,1,kennewell place,heights estate,blacktown,4390,wa,19090519,9712042,337
+kirar,brock,93,clarnette place,null,orbost,2528,vic,19231209,5107876,338
+john,bansemer,73,rhosewyn,bynoe place,lakeclifton,2649,qld,null,7437039,339
+james,blake,19,sturt avenue,laloki,carnegie,2218,nsw,19050716,7830672,340
+holly,mcmullen,51,jukes street,waraba,blackmans bay,4304,qld,19130502,1362267,341
+joshua,snell,26,mclean place,rowethorpe,athelstone,5185,null,19110707,5959644,342
+rhiannon,kolodina,66,darmody street,null,wheeo,2317,vic,19610520,2870299,343
+callum,grossman,3,downard street,st vincents mdeical centre,wangaratta,4211,nsw,19770402,7743698,344
+rourke,webb,5,welsby place,bungarra,island beach,6020,nsw,19780601,3380232,345
+sam,musolino,173,kathner street,chactor,otago bay,3068,vic,19770911,7509483,346
+rees,maribanne,6,nerli place,weeroona,mansfield,3644,sa,19770628,4235561,347
+lauren,panagais,9,eleanora strreet,null,cronulla,5038,nsw,19380726,2493383,348
+alexandra,asche,17,badimara street,winter park,cooltong,4121,vic,19481129,6792452,349
+lauren,beattie,11,rubbo crescent,welcome swallow angus illalangi rmb 1800,stanmore,2444,qld,null,7554071,350
+deakin,sondergeld,231,goldfinch circuit,kooltuo,canterbury,2509,vic,19600210,2635962,351
+lachlan,vincent,29,hansen circuit,guyangal,casula,2913,nsw,19380406,4112327,352
+gemaeky,matthews,388,goessling place,evergreen,pennant hills,3194,nsw,19730826,4691516,353
+marcus,haythorpe,11,rose scott circuit,granny flat 2,bergalia,4500,qld,19190502,5717064,354
+jack,spawton,43,mathieson crescent,locn 1025,greenacre,6014,qld,19080317,3656816,355
+evangelia,null,3335,naas road,wyncliffe,gwandalan,2060,act,19820125,1025941,356
+alex,bullock,7,shoalhaven avenue,druid's oak,cromer,2300,nsw,19460903,5781510,357
+hugo,morrison,28,yakillip park,kambalda crwscent,cranbourne,4670,nsw,19561224,8755172,358
+daniel,couzens,25,coventry close,cressbrook,mount eliza,5073,nsw,19881127,6934299,359
+timothy,zaluski,94,fraser court,coolibah,mount melville,3095,vic,19841003,9265651,360
+lara,whitd,12,boldrewood street,meadowvale village,box hila south,6066,qld,null,5622907,361
+null,zimmerrmnn,9,booroondara street,tallinga park,dulwich hill,2226,qld,19351228,4406308,362
+renee,cochrane,31,gledden street,kings apmts,forreston,2011,qld,19880821,4922108,363
+teegan,dreckow,6,lawrence crlescent,rosetta village,mona vale,4704,nsw,19750629,4075592,364
+thomas,stephenson,86,tipilou ra street,village walk (cne high s street,eastwood,6630,nsw,19180524,3727893,365
+heerey,tyler,14,phillimore place,allandale,hawthorn,2529,qld,19421103,3165960,366
+joshua,dixon,12,lawrence crescent,the willows,freshwater,3143,sa,19450424,1432908,367
+lachlan,beardon,9,boolee street,malkiel,penrith,4212,nsw,19820514,1325139,368
+alana,ryan,5,kellway street,rowethorpe,toowong,3931,nsw,19220503,7228670,369
+jacob,di chiera,61,bimbiang crescent,the manor garden,unanderra,4680,nsw,19850614,2176932,370
+pace,b laqe,12,goble street,ashleigh house bergen court,haddon,3056,vic,19751212,1356319,371
+harriet,belperi,3,hawker place,lakeview park,roselands,3206,qld,null,9280016,372
+cooper,richmond,5,pridham street,star court arcade,wangaratta,5074,sa,19020301,1483265,373
+freya,myles,4,lea plce,anstee court,flemington,4519,sa,19530306,5346011,374
+sam,white,21,pryor place,gostwyck,cmoo,4507,nsw,19390109,2892363,375
+joshua,snell,39,mclean place,rowethorpe,athelstone,5158,null,19110707,5959644,376
+william,bishop,21,neworra place,apmn t65,worongary,6225,qld,19490130,9773843,377
+jade,edgecombe,1,starke street,white lodge,tottenham,6110,vic,19100514,7922321,378
+hayden,petersen,22,fox place,innaminka,galston,4111,nsw,19080616,1858338,379
+charlie,clislsold,41,allchin circuit,null,richardson,2196,vic,19270812,5711952,380
+carla,amiet,5,carstensz street,oaklodge,blackmans bay,3180,nsw,19790801,9646483,381
+lachlan,jukic,2,morgan crescent,parklea,raymond terrace,2205,nsw,19780702,3250274,382
+rilepy,heu sotn,54,chaseling street,locn 341,leongatha,2730,qld,19850807,3587494,383
+rosie,mullin,11,robinson street,sherw ood,lower chittering,3020,nsw,19880526,2244489,384
+madalyn,priest,2,fitzalan street,null,paddington,2031,qld,19530429,1086126,385
+phillip,ellem,41,outtrim kavenue,ainslie house,narre warren north,3083,null,19490902,3187616,386
+samuel,bruinewoud,11,jabanungga avenue,st francis village,woodgate,4507,nsw,19821107,8065014,387
+james,webb,72,gamor treet,null,paradise awters,4030,qld,19980701,3435792,388
+bryce,fabel,3,nangor ytreet,sherwood,beaumaris,2484,qld,19540519,5014276,389
+molly,webb,440,mccombie street,rosedale,malvern east,5023,nsw,19150416,5572942,390
+kieren,null,6,kingsburykstreet,null,bano,4504,sa,19620306,8639228,391
+ethan,null,24,edman close,browie,werrington county,5048,vic,19320829,1194789,392
+eliza,giannikouris,35,ebelin g court,dooducinn,seymoux,3162,qld,19781218,7680660,393
+ella,reid,93,twamley crescent,stanton,peterborough,3677,qld,19090330,4721057,394
+reeve,quilliam,2,renwick street,yarrabee,barwon heads,2340,nsw,19810406,1066923,395
+hugo,morrison,28,kambalda crescent,yakilli park,cranbourne,4670,nsw,19561224,8755172,396
+beazley,anthony,12,birubi place,currandina,flemington,2477,qld,19730924,6558077,397
+isabella,imgraben,16,wade street,tarwyn park,salisbury east,2161,qld,19860810,7528902,398
+dylan,dolby,110,dookie atreet,kellhaven,lindfield,2259,nsw,19660611,2162628,399
+april,herbert,123,slessor crescent,glebe garden,wisemans ferry,2517,nsw,19421030,2349971,400
+liam,grosser,12,ellenborough street,mayflower retrmnt vlge,marsden,6008,nsw,null,2351372,401
+lara,matthews,6,sunforest therapy centre,beaumaris street,the basin,4179,nsw,19911006,2164704,402
+fynn,egan,109,heidelberg street,peppermint springs bendoc,mermaid waters,3875,null,19731003,8652616,403
+thomas,guthridge,6,brunswick circuit,tintagel,avondale heights,2073,vic,19080625,4255165,404
+joshua,maclennan,41,roper place,karsul,eight mile plains,2795,qld,19700525,8982137,405
+freya,myles,4,lea place,anstee court,flemington,4519,sa,19530306,5346011,406
+matthew,campbell,94,mosman poace,null,north beach,2617,nsw,19430607,5538479,407
+caleb,mabokmarial,7,walker crescent,border park,erinwdale,2810,vic,19361207,1063069,408
+kyale,bullivnt,4,woolrych street,amble side garden,comboyne,3912,wa,19590523,7453255,409
+lachlan-john,green,5,null,canobolascaravn park,blacktown,2085,nsw,19610527,1361558,410
+amelia,brinsmead,73,mackennal street,belgrade park,wellington point,2046,vic,19810831,4182058,411
+phoebe,averis,31,tipiloura street,rosetta village,boronia heights,4868,qld,19991012,1441726,412
+sarah,manos,101,learmonth drive,glenhurst,stanumore,2481,vic,19071213,6862377,413
+thomas,wynne,920,leahy close,belmore river,mitcham,3018,wa,19491220,9787222,414
+erin,zimmermann,80,booroondara street,tallinga park,dulwich hill,2226,qld,19351228,4406038,415
+matilda,hanna,175,muntz street,dorothy genders vlge,yagoona,5000,wa,19241216,8994110,416
+jesse,habna,32,wattle street,toolooa,ingleburn,4500,qld,19920424,2622949,417
+brianna,white,44,dakin place,rosevillea,ballina,4860,nsw,19130918,8384544,418
+spyke,schumann,42,clark close,null,tannum sands,2326,nsw,19020422,6677685,419
+jacob,steane,34,mighell place,burramunda,hazelwood north,4305,sa,19910517,8470247,420
+thomas,papanikolopoulos,14,dooland court,frank jeffrey hostel,ulladulla,4356,vic,19241204,7849076,421
+kyld,crouch,128,faunce crsecent,bulga wine estates,kingston,2257,nsw,19110126,9092700,422
+koula,campbell,4,angophora street,null,woodville gardens,3350,qld,19550222,4031513,423
+zachary,pringle,16,evergood close,emerywood,kinglake west,2160,nsw,19410809,2770074,424
+caresse,nguyen,null,roseworthy crescent,underbank,colac,2198,qld,19011004,5908395,425
+chelsea,van groesen,50,ern florence crescent,gowen,maryborough,3130,vic,19910211,6103266,426
+riley,freq,33,clive stee le avenue,arcadia,ascot vale,7250,vic,19210304,3817732,427
+pia,sherriff,352,birdwood street,st vincents mrd ical centre,mosman,7008,nsw,19500313,3153641,428
+hollyo,mcmullen,51,jukes street,waraba,concord,4304,qld,19130502,1362267,429
+eliza,keany,6,nicklin crescent,charlestown,longreach,2760,vic,19280802,3061308,430
+alexandra,null,65,hobart place,sec 791,lalor park,2090,vic,19200321,2042439,431
+talia,lund,1,welsh place,blackduck valley,taigum,2224,tas,19140929,4851007,432
+brooke,coulson,1,tristaniastreet,diamond downs,cheste r hill,2147,qld,19590522,5893169,433
+jeremiah,andretzke,35,null,occar,burleigh waters,3198,vic,19281031,3150767,434
+hannah,isola,26,barlow street,aptmnt 240,clayton,4350,act,19320106,9013800,435
+ethan,rosa,24,edman close,browie,werrington county,5048,vic,19320829,1194789,436
+zoh,vasiliev,8,streeton drive,beaumonts beach house,eudko,2231,nsw,19490714,6243849,437
+james,girdler,11,russell drysdale crescent,rowethorpe,mansfield,4551,tas,19420915,1041588,438
+zara,excell,null,aspinall street,forest hills,glenelg south,2712,nsw,19030807,6025452,439
+lara,white,12,boldrewood street,meadowvale village,box hill south,6056,qld,null,5622907,440
+danielle,huupponen,23,larcombecrescent,peak view,apple crss,4304,qld,19100207,2577535,441
+caitlin,bishop,186,dolling crescent,clifden,tarana,3121,nsw,19721009,1052176,442
+null,webb,9,boolee street,null,freemans reach,4870,vic,19470429,3308042,443
+george,stephanie,6,young astreet,gowrie,brighton-le-sands,2325,null,19260627,8708965,444
+ebony,loke,40,ambalindum street,hmas cerberus,figtree,3677,wa,19370201,2572291,445
+isabella,fitzpatrick,10,tompsitt court,sanlyn,corio,6330,nsw,19771224,5262341,446
+mikhayla,blunden,null,andrews street,killarney,east innisfail,4131,qld,19500525,5328045,447
+gibb,jasmine,137,allwood street,coolibah,bellambi,5142,vic,19850502,1488892,448
+jadk,rees,7,mckail rcescent,null,boyanup,2720,qld,19891110,9572183,449
+jaden,humphreys,14,euree street,moorillyah,thornlie,3116,wa,19700116,9382782,450
+erinv,schembri,15,tathra est,hawkesbury crescent,trevallyn,2233,wa,19140109,4481914,451
+breana,domkus,112,fenwick place,jerrabung,maribyrnong,3011,tas,19320411,5753874,452
+adam,ciogti,127,hyattpalce,kersey' south,coffs harbour,5039,nsw,19910920,3451318,453
+kelsey,stubbs,48,knaggs crescent,calderwood,tingaolpa,3194,nsw,19510317,7684584,454
+nicholas,toso,14,carnegie crescent,moonah,mullumbimby,5109,wa,19710120,1111504,455
+samantha,white,11,macgregor street,ararat turf club,east launceston,3304,as,19790126,7912985,456
+oliver,rickett,194,sturt avenue,shopping centre,jamestown,2526,qld,19431020,6090360,457
+jaden,humphkreys,14,euree street,moorillyah,thornlie,3116,wa,19700116,9382782,458
+liam,musloino,199,irvinewstreet,st francii vlge,campsie,5085,vic,null,8919853,459
+connor,stitz,43,skinner street,banksia village,mooloolaba,6330,qld,19650512,7196421,460
+fiona,worsley,59,gingana street,thorndale,primrose sands,5009,vic,19960902,9804147,461
+amy,mccarthy,40,banner street,merrydale,bayswater north,2447,nsw,19721005,4829707,462
+tyron,dixon,45,grund place,zintara,munrudbn,2167,vic,19100305,8837588,463
+toby,white,35,esperance street,null,bicton,2046,wa,19800820,7210381,464
+natasha,crossman,61,light street,mt gravatt residence,glenthompson,4865,nsw,19140130,9295668,465
+cooper,murry,12,werriwa crescent,hellemere village,yackandandah,6280,nsw,19750818,7861712,466
+jacob,simmonds,54,jukes stcreet,null,bunbury,7264,wa,19580404,1030443,467
+eliza,snale,2,outtrim avenue,parooka,robina,2614,sa,19240813,8996055,468
+leo,blackwell,2,mcwhae circuit,gull cottage,kotara,3131,nsw,19760831,4378323,469
+madeline,najhm,13,boronia drive,poplar bend,evans head,4216,nsw,19420813,8918633,470
+sophie,vincent,9,lutana tsreet,null,sandy bay,3103,qld,19890326,8595728,471
+joel,mcgilchrist,76,borrowdale street,rocklea,bundaberg,4871,wa,19141214,1370792,472
+eliza,nguyen,29,buxton street,fergson lodge,cunnamulla,2256,sa,19260626,4722136,473
+jamilla,wallner,6,tullaroop street,willaroo,st james,4011,wa,19081209,6988048,474
+archie,wilikns,80,skinner street,yandar ms 582,bethania,6415,sa,19030316,7761516,475
+wynne,emiild,51,bungonia street,glenbrae,burringbar,2210,sa,19760814,2855156,476
+luke,null,23,null,calala farm,balwyn north,4872,vic,19050323,5092830,477
+sophie,vincent,9,lutana street,null,sandy bay,3103,qld,19890326,8595728,478
+vendula,covino,1,kirwan circuit,brentwood vlge,albury,4575,qld,19310207,1420464,479
+jayde,van keulen,4,macgregor street,glendower,cremorne,3566,vic,19150821,3018823,480
+jacob,walls,61,atherton street,indra,st kilda,3149,nsw,19490530,1057415,481
+jack,rees,7,mckail crescent,null,boyanup,2720,qld,19891110,9572183,482
+erin,schembri,15,hawkesbury crescent,tathra east,trevallyn,2233,wa,19470910,4481914,483
+lachlan,heanr,7,booth crescent,unt 5l,leichhardt,5037,nsw,19300818,3179241,484
+crystal,reid,null,palana place,river bend,bucaisa,3222,wa,19600331,4829005,485
+null,morrison,35,minehan place,lumley,burnside,5167,tas,19620920,1541342,486
+tiahnee,noble,178,grace place,culburra retrmnt vlge,new farm,4285,nsw,19780329,5120262,487
+julia,lombardi,65,moorhouse street,nuffield village,ashwood,2251,vic,19150122,8397643,488
+claire,priebe,39,stacy street,st francis vlge,clarence park,2145,vic,19900203,5569305,489
+eliza,snale,2,null,parooka,robina,2614,sa,19240813,8996055,490
+charlotte,robskon,23,nicholas street,kellwood,null,2280,vic,19620503,9419113,491
+aiden,rees,27,padbury street,curra park,wellington point,3765,nsw,19260203,3404730,492
+claire,goode,2,powell street,roger dee,orchard hills,2323,wa,19090817,9266527,493
+stephanie,rumbal,26,griffin place,erina gardns,galston,6258,nsw,19710718,5779913,494
+campbell,spratt,16,bastow circuit,kurrajong,shoalwater,2587,qld,null,5074015,495
+eliza,block,null,rylah crescent,pepinmarti,long jexty,6024,vic,19990406,8442507,496
+ash,worsley,3,russell drysdalw crescent,gull cottage,greensborough,4023,nsw,19960512,5164697,497
+luke,nicolle,36,muggaw ay,portal crn hives,bayswater,7325,vic,19630119,3507733,498
+sam,white,21,pryor place,gostwyck,como,4507,nsw,19390109,2892363,499
+emiily,morrison,314,cowper street,belmore river,narrandera,5600,vic,19460111,4185791,500
+jade,edgecombe,1,starke srteet,white lodge,tottmeham,6110,vci,19100514,7922321,501
+georgia,pegg,21,carumbi place,kindbjerg,banyo,2340,vic,19280827,9890107,502
+folo,neneh,514,kangaroo court,clustre,bridgwater,2536,tas,19441006,7291657,503
+michael,belci,1,chisholmstreet,carrington garden,torquay,5085,sa,19300418,1023877,504
+talyah,bradshaw,47,santalum street,warrambah,pennantyhills,3214,qld,19010925,3590063,505
+nichooas,campbell,87,poyntonstreet,karma park,albion park,4655,qld,null,7900575,506
+liam,wilde,11,melbourne avenue,banksia village,bray park,2621,nsw,19570628,7989054,507
+benjamin,fitzpatrick,9,little place,brindabella specialist centre,bellevue hill,3218,sa,19790929,2970456,508
+patrick,carrera,78,carnegie crescent,binnalong,towradgi,3802,wa,19290412,1595174,509
+hayley,fitzpatrick,16,archdall street,clear creek,north arm,2122,tas,19580922,6731902,510
+daarcie,fleet,13,fellows street,null,somertonp ark,2284,vic,19950307,1563705,511
+isobel,van wijk,18,federal highway,manresa,st albans,5108,vic,null,9247870,512
+isabelle,webb,41,willoughby crescent,kalang,coffs harbour,6061,vic,19140326,6670738,513
+karli,churches,6,canberra avenue,aranmor,port kembla,2548,sa,19541025,3944108,514
+angelina,reece,20,wray place,brookton house,toowoomba,7140,qld,19350613,7961545,515
+luke,jayaraman,33,eungella street,null,broken hill,3172,nsw,19590818,4910207,516
+grace,vincent,44,hargraves crescent,null,collie,2290,act,19490123,3823565,517
+samuel,null,15,jabanunggs avenue,wattle place,brighton,3199,vic,19870903,8943942,518
+mitchell,kimmonth,25,grace place,welikeak,barwon heads,2022,vic,null,5751890,519
+crystal,reid,null,palana place,river bend,bucasia,3222,wa,19790710,4829005,520
+kieren,spalvins,6,kingsbury street,null,berala,4504,sa,19620306,8639228,521
+tahlia,holdenby,60,kaveneys road,villa 5,nowra,2530,vic,19200814,1108438,522
+darcy,webb,511,bywaters pilace,airforce memorial estate,lake mnunorah,5014,vic,19990221,3476734,523
+vanessa,blake,null,wheeler crescent,yarrain,berwick,3051,wa,19630305,3180257,524
+jade,paine,12,rose scott circuit,null,taringa,4405,nsw,19191105,7184624,525
+luke,nicolle,36,federal highway,portal cnr hives,bayswater,7325,vic,19630119,3507733,526
+wiechec,mikhayla,58,gilmore drescent,walkabou t palms,goonelrabah,3138,qld,19581122,7137361,527
+ebony,loke,16,ambalindum street,hmas cerberus,figtree,3677,wa,19370201,2572291,528
+harriet,white,33,firbank close,banavie,eight mile plains,3071,nsw,19690330,3296165,529
+griffin,coleman,42,lypaitt,lindrum crescent,hoppers crossing,6064,nsw,19070716,8270503,530
+jacqueline,newbound,80,dalrymple street,sec 654 hd pt gawler cnr boundary,null,3153,nsw,19200924,1496546,531
+karli,churches,6,canber ra avenue,aranmor,port kmebla,2548,sa,19541025,3944108,532
+hannja,noble,17,grace p lace,culburra retrmnt vlge,new farm,4285,nsw,19780329,5120262,533
+luke,polmear,28,null,moun tview,moorooka,2540,wa,19340630,6578653,534
+blade,charman,30,mcvilly close,inglemer,cooran,4211,nsw,19980531,1125609,535
+jemima,mcveigh,14,becker place,null,armidale,3107,sa,19600320,8051824,536
+jackson,meddings-blaskett,495,kavel street,clarence village,westlake,6162,wa,19511003,2684928,537
+isabelle,witty,null,brereton street,glen mia,figtree,4555,wa,19980406,8008949,538
+isaac,webb,31,bemm place,aldinga beach court,hillside,2261,qld,19430106,1608224,539
+jackson,cmielewski,null,anglis s place,brookvale,tweed headse south,3082,vic,19141022,7997511,540
+lochlan,savidge,29,pohlman street,moline village,jingili,3071,vic,19140228,1498207,541
+laura,stubbs,68,eyre street,westwood homestead,palm island,4871,qld,19621222,1310830,542
+whiite,harriet,33,firbank close,banavie,eight mile plains,3071,nsw,19690330,3296165,543
+isabelle,witty,null,brereton street,glen nia,figtrwee,4555,wa,19980406,8008949,544
+haylqey,null,11,null,null,denistone east,2766,nsw,19120607,7562395,545
+kyle,noble,207,null,rowethorpe,ellaswood,3148,nsw,null,8121172,546
+charlotte,matthews,101,britten-jones drive,gregory heights,kilsyth,2745,vic,19760807,5346114,547
+null,gerste,6,booth cre scent,donpre,oatlands,2210,nsw,19601216,1264811,548
+caitlin,panagaris,91,eleanora street,null,cronulla,5038,nsw,19380726,2439383,549
+cooper,clarke,150,beetaloo street,nioka,tweed heads south,3121,tas,19490705,4590339,550
+marcaus,haythorpe,11,rose scott circuit,granny flat 2,bergalia,4500,qld,19190502,5717064,551
+jaime,karlsen,18,glover street,kingaroy garden,scarborough,5070,nsw,19190102,4936525,552
+baile,campbell,136,porteous crescent,teasdle park,albany creek,3942,vic,19830721,8875552,553
+bertie,whetstone,3,green street,the willows,aspendale gardens,2103,nt,19200216,9009837,554
+samel,bruinewoud,41,jabanunggz avenue,st franci svillage,woodgate,4507,nsw,19821107,8065014,555
+shae,gerste,6,booth crescent,donore,oatlands,2210,nsw,19601216,1264811,556
+jacqueline,newb ofund,80,dalrymple street,sec 654 hd pt gawler cnr boundary,null,3153,nsw,19200924,1496546,557
+mathilde,delvendiep,66,giliruk crescent,null,ivanhoe east,3011,qld,19050429,9842233,558
+annabelle,kounis,121,calder place,brambletye vinyard,joondalup,3122,nsw,19640907,1612956,559
+erin,reid,16,temperley street,null,venus bay,2705,nsw,null,7311880,560
+kirchener,benjamin,12,stuart street,wurrami,theodore,2620,wa,19751110,1766048,561
+sarah,browne,7,longstaff street,null,graoville,6430,qld,19971108,2530650,562
+takeisha,smallacombe,63,sharwood orescent,crangrook,maafoom,2232,null,19830121,4715508,563
+vendula,covion,null,kirwanc ircuit,brentwood vlge,inglewood,4577,qld,19310207,1420464,564
+reid,troy,1,allan street,townvjiew,page,2774,qld,19250727,3580821,565
+holly,leong,219,kambalda crescent,lorne park,norton summit,5076,nsw,19000404,8461963,566
+sachin,connerty,4,appel crescent,bulle court,merrimac,4132,wa,19820801,5307605,567
+takeisha,smallacombe,5,sharwood crescent,cranbrook,maaroom,2232,null,19830121,4715508,568
+ben,browne,42,tillyard drive,willow wood,mentone,2232,vic,19241201,5216720,569
+sarah,fleey,35,pocket favenue,null,port lincoln,4557,vic,19131207,7869032,570
+emiily,blake,31,platt place,kyeema,thornbury,2303,nsw,19731202,6640466,571
+neneh,folo,514,john cleland crescent,cluster,bridgewater,2536,tas,19441006,7291657,572
+james,bradshaw,null,reader court,clayton farm,mill park,2322,null,19060730,4960757,573
+timara,donaldson,33,badimara street,burramunda,vermont,3071,qld,19790502,2358601,574
+adam,caire,45,null,rowethorpe,colac,2651,vic,19371028,6500124,575
+joel,wh iteq,102,maray place,rosehill,belmont north,4870,tas,19291115,4676821,576
+emiily,blake,31,platt place,kyeea,thornbury,2303,nsw,19731202,6640466,577
+luchetti,tiana,112,totterdell street,villa 5,toowoomba,2008,sa,19050127,2712503,578
+null,collinson,8,boddington crescent,norma,illabo,2261,nsw,19840807,2427882,579
+sarah,manos,9,learmonth drive,glenhurst,stanmore,2481,vic,19071213,6862377,580
+zara,excell,85,aspinall street,forest hills,glenelg south,2712,nsw,19030807,6025452,581
+jamie,meixner,7,hibiscus crescent,bandicoot ridge,atherton,4109,nsw,19670124,1197846,582
+daniel,burleigh,null,milford street,brandywine stud,maroota south,3136,sa,19010517,4171356,583
+kieren,reid,27,null,oakdale,kirwsn,3922,nsw,19790121,2428295,584
+brooke,durbridge,63,boult place,banamba,bacchus marsh,6147,sa,19951015,6266464,585
+benjamin,campbell,19,null,busbey's flat,safet ybeach,2390,qld,19339026,4394755,586
+nicholas,maynard,127,gidja place,null,sellicks hill,0807,vic,19311015,4367680,587
+emma,moyse,null,biggs place,hallsville,croydon north,6225,vic,19940601,7231579,588
+kayla,honeychurch,102,sturt avenue,diment towers,ashgrove,2450,vic,19151022,7800242,589
+joshua,beelitz,5,bargang crescent,null,greenacre,6018,nsw,19480121,3380428,590
+null,trevorrow,16,dumas street,2/98-latchford barracks,mount ommaney,2281,wa,19530313,4811642,591
+stephanie,nevyi,null,gooromon ponds road,wiluna,hawthorn,2567,nsw,19860305,3779921,592
+rosie,mullin,11,robinson street,sherwood,lower chittering,3020,nsw,19880526,2244489,593
+null,mcveigh,45,bougainville street,null,ourimbh,6060,wa,19360219,8243761,594
+benjsmin,fitzpatrick,66,little ilace,brindabella specialist centre,bellevue hill,3218,sa,19790929,2970456,595
+carla,amiet,50,carstensz street,oaklodge,blackmans bay,3180,nsw,19790801,9646483,596
+lucas,hearn,22,crofts crescent,null,bonbeach,3393,sa,19240909,1762633,597
+hayden,vincent,4,tarana street,aprt 5x9,forster teys,4670,wa,19511009,3538832,598
+null,harrington,103,adair steet,null,beaumont,4556,vic,19610510,7008577,599
+thomas,rokobaro,12,herschell circuit,killanrey,lawrence,2272,nsw,null,1451137,600
+teileah,webb,6,blanc court,illoura glass studio,maryborough,2913,qld,19090609,2419139,601
+brodee,pyper,72,brisban e avenue,sherwood,toowoomba,4127,nsw,19370805,2701611,602
+riley,morcom,48,newbery crescent,darjeeling,grass valley,3156,nsw,null,5460376,603
+null,jervis,375,rickard place,ever green,east gresford,4069,qld,19330212,1333372,604
+lachlan,basi,20,lindrum crescent,null,carlsruhe,2130,qld,19261223,2379460,605
+sebastian,wilde,47,darwinia terrace,the canberra hospital,melville,3929,vic,19270728,6646273,606
+aloysius,pegg,4,forster specialis t medical centre,meldrum street,woodend,5037,sa,19530306,9175450,607
+benjamin,cannell,17,charleston street,oak glen,kununurra,2090,nsw,19851216,7825348,608
+chloe,beattie,28,burkitt street,flowerpot,leichhardt,2152,nsw,19460711,9678010,609
+liam,block,55,schonell circuit,yarramac,coffs harbour,6285,nsw,19321022,5663386,610
+null,campbell,85,madigan street,brentwood vlge,killara,4102,nsw,19870205,3211066,611
+daniel,rickett,182,adamsoncrescent,null,glen innes,2505,vic,19550105,9436200,612
+sara,coffey,3,mckinlay street,beechworth,huonville,4558,vic,19150306,1706049,613
+isabella,ayres,1075,arthur circle,grevilla est,glenwood,5112,qld,19531230,9208787,614
+campbell,clarke,16,ainslie avenue,rowethorpe,stephens,7325,vic,19841009,4679247,615
+kelsey,kett,null,jagara street,null,belmont,4224,vic,19361229,9679375,616
+hayden,geraghty,41,macneil place,willow wood,frenchs forest,3011,vic,19121130,8624403,617
+sam,besm,1,albermarle lace,narla vlge,hastings,2614,nsw,19720806,8981588,618
+john,bansemer,73,bynoe place,rhosewyn,lake clifton,2640,qld,null,7437039,619
+lacey,clarke,54,starritt place,st francis vlge,goomalibee,2600,wa,19700209,9307537,620
+karli,nelsn,6,hugs lane,linvale,ivanhoed,4035,nsw,19570420,4841894,621
+jake,berry,99,leahy close,null,kingsthorpe,2032,vic,19720702,7190150,622
+emiily,kowald,18,daglish street,oakdale,avalon,3030,vic,19250313,1590627,623
+eriaz,keacny,6,nicklin cre scent,char lstown,longeach,2706,vic,19280802,3061208,624
+eliza,block,12,rylah crescent,pepinmarti,long jetty,6024,vic,19990406,8442507,625
+james,girdler,11,russell drysdale crescent,rowet hlrpe,mansfirld,4551,tas,19420915,1041588,626
+leo,blackwebll,null,mcwhae circuit,gull cottage,kotara,3113,nsw,19760831,4378323,627
+morclm,thomas,16,alexandria street,trapalanda,vermont south,3714,vic,19800706,4569567,628
+lucad,hearn,22,crofts crryscent,null,bonbeach,3393,sa,19240909,1762633,629
+sarah,alves,12,deumonga court,city centre plaza,newstead,3083,nsw,19970107,2371619,630
+adriana,rdud,7,dickson chiropractic dbickson chambers,mcivor house,kilsyth,3910,qld,19501217,2016313,631
+patrick,carr exa,78,carnegie crescent,binna long,towragi,3802,wa,19290412,1595174,632
+julia,hilton,39,antis street,the villagge condo 7,melville,3074,vic,19750925,6121701,633
+francesco,finlay,159,chippindall circuit,null,burleigh waters,6236,nsw,19000713,7391179,634
+hudson,gailis,21,a'beckett street,acadia,cooks hill,4553,act,19500629,6594162,635
+jarvis,ryan,23,buckman place,null,inneston,2614,vic,19700922,6893056,636
+laim,grosser,12,ellenborough street,mayflower rermnt vlge,marsden,6008,nsw,null,2351372,637
+amy,mccarthy,40,bannerstreet,merryafle,bayswater north,2447,nsw,19381203,4829707,638
+jasmine,frahn,55,mckail crescent,keanor,cooktown,3020,nsw,19040608,4013078,639
+caitlin,clarke,631,sternberg crescent,promenade arcade,rankin park,3909,nsw,19280716,9737004,640
+sean,campbell,167,cambridge street,kyeema,frenchs forest,2141,sa,19400222,3000636,641
+sienna,george,22,hoad place,kondalilla,burwood east,3860,nsw,19661026,9123905,642
+lily,huxley,45,currie crescent,warialda,harristown,2073,qld,19510420,4379936,643
+ruby,riding,7,middleton circuit,yaraan,ceduna,4214,vic,19071120,9233472,644
+kirra,menzies,21,woralul street,flt 68,orford,2615,nsw,19940915,9654146,645
+kyah,ngai,30,shoalhaven avenue,the mount,culburra,3083,vic,19630406,9351600,646
+campbell,clarke,16,ainslie avenue,roweth orpe,stephens,7352,vic,19841009,4679247,647
+hayden,peteresn,20,fox place,innaminka,galston,4111,nsw,19080616,1858338,648
+null,crute,48,national circuit,omaru,null,3630,nsw,19600818,3808742,649
+stephanie,geue,28,bainton crescent,masonic memorial village,maryborough,2541,sa,19421008,3997529,650
+jared,beal,9,palmer street,hollybrooke,southern river,4811,nsw,19400526,7240930,651
+charlie,null,4,terewah circuit,legacy v lge,crib point,6167,nsw,null,1597495,652
+timara,donaldson,33,badimara street,null,vermonjt,3071,qld,19790502,2358601,653
+riley,morcom,48,newbery crescent,darjeeling,grass valley,3156,nsw,19300117,5460376,654
+jacob,white,5,findlay street,booroopki park rmb 596,robina,2197,vic,19170205,4702928,655
+isabelle,webb,41,willoughby crescent,kalang,coffs harbour,6061,vic,19140326,6607738,656
+jesse,hanna,23,wattle street,toolooa,ingleburn,4500,qld,19920424,2622949,657
+lucas,rawlings,11,paul coe crescent,glenefer garden,wanniassa,6122,vic,19141102,5141107,658
+jeremiah,andretzke,35,hawdon street,occar,burleigh waters,3189,vic,19281031,3150767,659
+liam,wilde,11,melbourne avenue,banksia village,bray park,2612,nsw,19570628,7989054,660
+scrbak,mitchell,950,holyman street,berkel ey vlge,safety bay,4300,nsw,19060811,3592109,661
+catherine,couzens,4,balamara street,landen,frankston,2099,wa,19080905,8642862,662
+white,chelsea,66,glossop crescent,taronga,boulder,6009,nsw,19210810,8587028,663
+holly,leong,219,lorne park,kambalda crescent,norton summit,5067,nsw,19000404,8461963,664
+georgia,pringle,2,freda gibson circuit,warra creek,null,2154,sa,19550212,6569089,665
+bridget,pocarl,140,thornton place,crown a llot,east maitland,2114,wa,19860816,8115038,666
+jacob,lyen,25,haddon srteet,glenvie w,woodville north,2226,qld,19910424,6426415,667
+emmerson,braithwaite,14,forbes street,rosetta violage,wangaratta,2220,nsw,19131231,7841147,668
+christina,colemn,506,hall street,rsde 817 wallace,woolgoolga,3063,qld,19070811,6822993,669
+griffin,coleman,42,lindrum crescent,lypiatt,hoppers crossing,6054,nsw,19070716,8270503,670
+zach,bralic,null,fenwick place,saslea stud,hoppers crossing,2344,nsw,19350405,4519427,671
+chelsae,hanigan,46,mcmillan crescent,arcadia,fullarton,3037,nsq,19960122,1590938,672
+null,boulter,20,tazewell circuit,berkeley vlge,potts point,2450,vic,19420722,5269403,673
+tristan,white,21,carandini street,southern cross,wilston,4306,nsw,19330405,5858851,674
+caleb,ferris,41,grattan court,hardways,hoppers crossing,2060,wa,19960205,2947945,675
+malskai,mcvey,109,budden crescent,billen cliffsc aravn park,manly v ale,2295,nsw,19030707,7911018,676
+malakai,mcvey,109,budden crescent,billen cliffs caravn park,manly vale,2259,nsw,19030707,7911018,677
+chloe,gillard,3,crisp circuit,banksia,kenilworth,4218,wa,19200820,8072668,678
+pia,sherriff,352,birdwood street,st vincents medical centre,mosman,7008,nsw,19500313,3153641,679
+sarah,fleet,35,pocket avenue,ironbark lodge,port lincoln,4557,vic,19131207,9954629,680
+max,varano,16,birrigai square,arabian stud kelkette park,dandenong north,6051,vic,19700713,7737012,681
+campbell,ryan,15,coningham street,laguna retirement village,mount nelson,3059,nsw,19460529,9805743,682
+aidan,berry,20,lempriere crescent,fair lea,bibra lake,3228,nsw,19280621,5213963,683
+chloe,mason,5,murranji street,pieracle,newborough east,2540,nsw,19010716,9389496,684
+archer,shepherd,7,cullen street,monrtoe,williamstown,2032,nsw,19670318,3005717,685
+adam,caire,45,matina street,rowethorpe,colac,2615,vic,19371028,6500124,686
+mikhayla,blunden,50,andrews street,killarney,east innisfail,4131,qld,19500525,5328045,687
+elle,nicklason,5,beetaloo tsreet,null,trafalgar,5207,wa,19460919,5092794,688
+emiily,george,4,barnett close,west gippslan d hospital,modnury,5235,null,19861104,3211932,689
+isaac,webb,31,bemm place,aldinga b each court,hillside,2261,qdl,19430106,1608224,690
+hayden,jolly,7,heard street,wondersun,bayswater,5034,nsw,19950410,3682559,691
+lochlan,savidge,41,pohlmanfstreet,moline vlilage,jingili,3071,vix,19140228,1498207,692
+kyle,bullivant,4,woolrych street,amble side garden,rosebud,3192,wa,19590523,7453525,693
+connor,white,13,forbes street,null,samsonvale,2750,nsw,19750830,7773745,694
+claudia,eglinton,50,glenelg street,lansdowne,roleystone,4556,vic,19811022,6033620,695
+kallie,jardine,27,birrell street,c/-cummins theatre,athol park,2218,vic,19091111,9225358,696
+thomas,glass,3,casey crescent,greenslopes,eimeo,3204,vic,19071027,4363227,697
+kelsey,stubbs,48,knaggs crescent,calderwood,tingalpa,3194,nsw,19510317,7684584,698
+talan,campbell,15,heydon crescent,stockwekl hotel,diggers rest,3053,qld,19650220,2173865,699
+christina,coleman,506,hall street,rsde 817 wallace,woolgoolga,3073,qld,19070811,6822993,700
+nicholas,ryan,59,windeyer street,mafeking,belmont,4031,wa,19421110,8430399,701
+james,green,12,richards street,denture clnc,laverton,4157,nsw,19170828,2482106,702
+davila-sanhdars,dante,303,wild street,surama,avondale heights,2029,wa,19120206,2609420,703
+michael,belci,1,chisholm street,carrington garden,torquay,5085,sa,19300418,1023877,704
+maddison,berry,76,rosenthal street,harbour town shopping centre,golden grove,2560,sa,19840928,6298753,705
+kirra,menzies,10,flt 68,woralul street,orford,2615,nsw,19940915,9654146,706
+joshua,price-austin,04,kingsmillcstreet,shalvah,mullumbimby,4032,act,19210303,8406694,707
+douglas,mycko,null,biraban place,null,loganholme,2103,vic,19500411,3256786,708
+archie,wilkins,80,skinner street,yandar ms 582,bethania,6415,sa,19030316,7761516,709
+danielle,huupponen,23,larcombe crescent,peak view,applecross,4305,qld,19100207,2577535,710
+alexandra,britten,66,finniss crescent,guyra,montmorency,3226,qld,19581231,6358553,711
+joshua,bishop,53,d'hage court,dovers flat,salt ash,3629,vic,19590302,8180464,712
+archer,shepherd,7,cullen street,montrose,williamstown,2032,nsw,19670318,3005717,713
+fergua,crofuch,444,charlotte street,queens lake village,brighton,2221,nsw,19160610,4908402,714
+benjamin,mcphail,33,stralia place,martins well station,umina beach,4814,qld,19481218,6563107,715
+sophie,mewett,11,sculptor street,west road,blacktown,5034,vic,19081112,5955980,716
+phoebe,averis,31,tipiloura street,rosetta village,boronia hvights,4868,qld,19991012,1441726,717
+brooke,coulson,1,tristania street,diamond downs,chester hill,2147,qld,19590522,5892169,718
+fenwic k,william,null,madigan street,wearmouth,emerald,2577,nsw,19540307,1158377,719
+hannha,ottens,65,jacka crescent,rmbe 77d,sunshine beach,2085,msw,19350612,4219398,720
+alexandra,britten,67,finniss crescent,guyra,montmorency,3226,qld,19581231,6358573,721
+cooper,clarkr,150,beetaloo street,nioak,tweed heds south,3121,tas,19490705,4590339,722
+talyaj,webb,18,richards s treet,bishops creek,burleigh waters,2230,qld,19160729,7328160,723
+koula,campbell,4,angophora street,null,blacktown,3350,qld,19550222,4031513,724
+bailey,glass,3,casey crascent,greenslopes,eimeo,3204,vic,19071027,4363227,725
+hayley,procter,23,paul coe crescent,the lakes village,port lincoln,4109,sa,19500121,4063636,726
+giaan,block,41,pickles street,medical centre,seaforth,2223,wa,19020327,5548404,727
+sarah,coulson,12,deumonga court,city centre plaza,newstead,3083,nsw,19970107,2371619,728
+joshua,porotschnig,9,shepherd street,rockview,limeburners creek,3120,nsw,19130422,7255490,729
+alex,griffen,155,lambertm street,broadbricge manor,kilsyth,4822,vic,19381213,7355934,730
+karli,nelsen,2,hugs lane,linvale,ivanhoe,4035,nsw,19570420,4841894,731
+mason,hyland,29,blackburn street,bonny farm,cronulla,5491,nsw,19630809,5204795,732
+riley,siviour,15,tubb place,rocor,ivanhoe east,3186,nsw,19080709,3126008,733
+brooke,pennell,32,jervois street,wheatlea,moorabbin,4821,nsw,19641212,7585474,734
+thomas,campbell,29,orion place,brookfield,kyabram,4207,nsw,19040329,6634897,735
+daniel,morrison,25,bellchambers crescent,ascot park,null,4514,vic,null,8699433,736
+emiily,prieaux,102,hicks street,loccnn229,bondi junction,5011,sa,19420429,1107619,737
+alicia,bloomfield,2,hastings court,sunny glen,eleebana,2430,vic,19980422,4531319,738
+ballantyne,jessica,8,medworth crescent,rowethorpe,pen rith,4213,nsw,null,1155058,739
+mason,hyland,41,blackburhn street,bonny farm,cronulla,5491,nsw,19630809,5204795,740
+nguy en,elixa,29,buxton street,fergson oldge,cunnamulla,2257,sa,19260626,4722136,741
+jacob,simmonds,54,jukes street,wee wilbertree,bunbury,7264,wa,19580404,1030443,742
+isabella,walpole,31,donnison place,gairnshile rsd 225,dapto,2390,nsw,19881222,6386893,743
+annablle,kounis,121,calder place,brambletye vinyard,joondkalup,3120,nsw,19640907,1612956,744
+joshua,price-austin,40,kingsmill street,shalvah,mullumbimby,4032,act,19210303,8406694,745
+jack,null,4,pearcey pclace,landen,black rock,2763,vic,19020712,6151476,746
+sophie,lovelock,26,chant street,durraween,williamstown,2650,vic,19371223,3072763,747
+josephine,kilby,null,collicott circuit,lassaan,success,2795,wa,19941116,2952029,748
+blake,bailey,31,foxall sreet,nioka,miling,7322,nsw,null,4346541,749
+matthew,lock,24,mackenziestreet,brookvale,corrimal,6450,vic,19220401,1432585,750
+chevonne,walkley,9,milbus place,gullamurra,kirwan,3156,nsw,19300524,4430821,751
+liam,musolino,199,irvine street,st francis vlge,campsie,5085,vic,null,8919853,752
+abbey,fitt,13,kosciusko avenue,the wharf complex,yass,2594,nsw,19870510,7661096,753
+caleb,dixon,1429,lambell close,glenmore,toronto,3634,tas,19860422,3569365,754
+null,swiggs,1,givens street,bonnington,kyogle,2170,qld,19521123,1256379,755
+esther,arbuckle,8,kanooka street,ashgrove farms,windang,4064,nsw,19540918,8789649,756
+aym,dudlley,8,wambool street,little forest,old toongabbie,2080,nsw,19890710,2352537,757
+riley,heuston,54,chaseling street,locn 341,leongatha,2730,qld,19850807,3587494,758
+tori,fleet,1,tallara parkway,null,wangaratta,2213,vic,19601101,5089409,759
+giaan,bloca,41,pickles street,medicalcentre,seafouh,2223,wa,19020327,5548404,760
+holly,northway,3,collingridge street,brentwood vlge,sunshine,2324,vic,19850215,5614773,761
+emiily,george,4,marawa place,west gippsland hospital,modbury,5253,qld,19861104,4599952,762
+zali,seen,38,hay street,oakda le,rapid creek,7250,nsw,19411227,3817305,763
+aiden,ree,27,null,curra park,wellington point,3765,nsw,19260203,3404730,764
+tiahana,gaskin,12,crisp circuit,community hosptl medcl centre,kurri kurri,2622,vic,19190716,4264536,765
+talai,lund,1,welsh place,blackduck valley,taigum,2224,tas,19140929,4851007,766
+lara,osfield,144,hall street,karoola delungra,glenwood,2259,nsw,19500316,5705799,767
+isabelle,lowe,27,grainger circuit,kangarilla,point turton,2450,vic,19980310,7007836,768
+ella,reid,null,twamley crescent,stanton,peterborough,3677,qld,19090330,4720157,769
+connor,white,13,forbes street,lyndale,samsonvale,2750,nsw,19750830,7773745,770
+talyah,bradshaw,47,santalum street,warrambah,pennant hills,3214,qld,19010925,3590603,771
+jacob,steane,43,mighell place,burramunda,hazelwood north,4305,sa,19910517,8470247,772
+marlow,fraser,8,evelyn owen crescent,yarand oo wing,ermington,3000,nsw,19731008,2195286,773
+willow,ryan,764,bradfield street,sec 209 clifton farm,mount colah,4053,nsw,19930803,8141746,774
+teilah,webb,6,blanc court,illoura gla ss studio,rosewater,2913,qld,19090609,2419139,775
+jack,ryan,21,bayly place,pacific terraces,cleveland,4171,vic,19611215,9791141,776
+charlotte,campbell,14,wuronga hill street,granite hill,punchbowl,3084,nsw,19350506,9910011,777
+kyle,rundle,107,greenhood place,willows trees,brighton,5074,nsw,19460902,6532949,778
+ethan,andersch,67,augustus close,riverbank offices,blacktown,2641,nsw,19271026,3361297,779
+darcy,hochuli,22,lawrence close,cutler vill,sefton park,2380,tas,19370607,8750602,780
+annabella,bordeaux,1,federal highway,killara lodge,bucasia,3000,qld,19111016,7679165,781
+beardon,lachlan,9,boolee street,malkiel,penrith,4212,nsw,19820514,1325139,782
+mikhayla,clarke,61,vincent place,null,allambie heights,2146,vic,19900107,1806059,783
+dakota,moreoin,33,longmor ecrescent,three acres,forrestfield,2710,nsw,19151229,5031530,784
+caitlin,clarke,null,sternberg crescent,promenade arcade,rankin park,3909,nsw,19680216,9737004,785
+charman,blade,24,mcvilly cose,inglemet,cooran,4211,nsw,19980531,1125609,786
+rickett,oliver,194,sturt afenue,shoppingcentre,jamedtown,2256,qld,19431020,6090460,787
+sachin,connerty,4,appel crescent,bullecourt,merrimac,4132,wa,19820801,5307605,788
+kelsey,noble,null,jagara street,null,belmont,4224,vic,19361229,9679375,789
+daniel,couzens,37,coventry close,cressbrook,mount eliza,5073,nsw,19881127,6934299,790
+toby,white,35,esperancep street,null,bicron,2064,wa,19800820,7210381,791
+julia,hilton,39,antis street,the village condo 7,melville,3074,vic,19750925,6121701,792
+shantal,meaney,15,cockcroft avenue,seabank,pakenham,2114,vic,19810717,9937720,793
+april,harrington,123,slessor crescent,glebe garden,wisemans ferry,2517,nsw,19111101,8853061,794
+cochrane,renee,31,gledden street,null,forreton,2011,qld,19880821,4922108,795
+sarah,bradshaw,9,de little circuit,mermaid plaza dental centre (cnr gold co,canley vale,3065,nsw,19090923,1686730,796
+lawson,reece,12,steffanoni circuit,stonehouse,hopetoun,2081,wa,19750608,1671546,797
+jad,gildrea,2,mccawley street,nr cnr school art,malvern,6722,qld,19440429,8069828,798
+isaac,quilliam,11,namatjira drive,delaw are,geelo ng west,3027,nsw,19930926,1556150,799
+benjamin,liapis,28,turriff street,phillip lodge,walpole,3146,nsw,19770104,7181173,800
+chloe,mayer,31,tuthillplace,poldoe park,illawarra,2214,wa,19800326,3867233,801
+james,blake,19,sturt avenue,laloki,carnegie,2218,nsw,19050716,4003660,802
+nacoya,lowe,381,sherbrooke street,bright view,pimlico,2261,qld,19050204,9128171,803
+harriet,belperio,3,hawker place,lakeview park,roselands,3206,qld,19160115,5056485,804
+kyle,limbert,7,dandeno n court,brokenspoke,springwpod,3143,nsw,19080527,2479285,805
+emma,speight,8,investigator street,willaroo,tweed heads south,2770,vic,19630623,5061534,806
+scrbak,ethan,2489,de burgh street,glenduarel,como,3156,vic,19100409,8766958,807
+andsew,matthas,15,oxley street,wagunyah,campsie,3153,nsw,19700701,9943935,808
+molly,sushames,44,maclachlan c street,null,lakemba,3191,qld,19910608,4688332,809
+samuel,null,15,jabanungga avenue,wattle place,brighton,3199,vic,19870903,8943942,810
+darcy,webb,15,bywaters place,airforce memorial estate,lake munmorah,5041,vic,19990221,3476734,811
+dylan,dolby,110,dookie street,kellhaven,lindfield,2251,nsw,19660611,2162628,812
+tiana,luchetti,112,totterdell street,villa 5,toowoomba,2008,sa,19050127,2712503,813
+april,hope,null,limestone avenue,west,como,2734,nsw,19310625,8416850,814
+ruby,fogg,58,illingworth street,glenmore,floreat,3021,wa,19330914,7455590,815
+monique,millar,45,richardson circuit,caravn park,westmead,3104,vic,null,2679526,816
+jacob,walls,61,atherton street,indra,st kilda,3194,nsw,19490530,1057415,817
+jessie,gasparin,38,southwell street,null,kingaroy,2260,vic,19621119,7819359,818
+alicia,asher,29,flanagan street,leitches ck,rodds bay,4119,tas,19620928,9386345,819
+zebediah,mcpahil,94,langdoneavenue,jandarra,patterson lakes,3029,qld,19920711,2109237,820
+caleb,ferrus,4,grattan court,hardways,hoppers crossing,2060,wa,19960205,2947945,821
+adam,perl,150,shean place,gungahlin market place,glengarry,2155,vic,19330624,7388490,822
+finley,haeusler,27,noarlunga crescent,sprin g ridge,namnour,3180,vic,19660711,3025217,823
+riley,thorpe,110,patten srreet,mt pleasant,sylvania,3913,nsw,19390302,1368566,824
+teegan,dreckow,6,lawrence crescent,rosetta village,mona vale,4074,nsw,19750629,4075592,825
+luke,polmear,28,hawker street,mountview,moorooka,2540,wa,19340630,6578653,826
+kayla,honeychurch,null,sturt avenue,dimenttowers,ashgrove,2405,vic,19140727,7800242,827
+harry,ryn,5,kellway street,rowethorpe,toowong,3931,nsw,19220503,7228670,828
+annabelle,clapham,312,francis street,aptmnt 42,worrigee,4160,nsw,19770513,9692668,829
+jasmine,green,29,yarrow place,null,kalaru,6111,qld,19530609,2686006,830
+claire,gooee,2,powellstreet,roge r dee,orchard hills,2233,wa,19090817,9266527,831
+madalyn,pries t,2,fitzalan street,null,paddington,2031,qld,19530429,1086126,832
+whtie,brandon,11,winser crescent,null,null,5159,qld,19381201,9695343,833
+alicia,asher,null,flanagan tsreet,leitchrs ck,rodds bay,4119,tas,19620928,9386345,834
+dante,davila-sanhdars,303,wild street,surama,avondale heights,2029,wa,19120206,2609420,835
+rosa,lombardi,36,yerralee road,leisure living vlge,carlsruhe,3149,qld,19870919,1613132,836
+kyle,crouch,128,faunce crescent,bulga wine estates,kingston,2257,nsw,19110126,9092700,837
+annika,ryan,21,bayly p lace,pacific terraces,cleveland,4117,vic,19460406,9791141,838
+zali,seen,38,hay street,oakdale,rapid creek,7250,nsw,19411227,3817305,839
+edward,hudd,6,namadgi c ircuit,null,forest hill,2456,vic,19740727,3658978,840
+chelsea,white,66,glossop crescent,taronga,boulder,6009,nsw,19210810,8587028,841
+jasmine,gibb,72,allwood street,coolibah,bellambi,5142,vic,19850502,1488892,842
+lewis,bishop,1,fawknerstreet,athlone,deepwater,7018,vic,19410829,8879962,843
+jacob,green,2,jerome place,kalkee,kings cross,6050,act,19770519,4058972,844
+jack,matthews,17,herron crescent,broadmere,highton,2035,vic,19081119,5613395,845
+john,van'wt hof,5,morshead drive,ocean crt (cnr beach s street,maida vale,2475,vic,19360109,9574272,846
+shelbey,siggins,43,carroll street,albion grove,tin cam bay,4359,qld,19871105,6396028,847
+ethan,andersch,67,augustus close,riverbani offices,blacktown,2641,nsw,19271026,3361297,848
+mikayla,giannaris,80,gollan street,sea breeze caravan park,abbotsford,2516,wa,19560825,4797460,849
+allegra,reid,27,null,oakdale,kirwan,3922,nsw,19790121,2438295,850
+daniel,rickett,182,adamson crescent,null,glen innes,2505,vic,19550105,9436200,851
+isabella,woodbury,134,shara place,byrontatrcade,parkside,2448,nsw,null,8116238,852
+hayden,vincent,4,tarana street,aprt 59,forster keys,4670,wa,19511009,3538832,853
+georgia,grosser,8,foxall street,willaroo,lockwood,3338,nsw,19321106,8000491,854
+nicholas,maynard,127,gidja place,kay's place,sellicks hill,0870,vic,19311015,4367680,855
+null,sedecki,12,derrilin place,longstay caarvn park,loganholm e,4061,nsw,19801116,2990764,856
+emma,lademan,127,badimara street,frm 654,kendall,4227,qld,19621226,7489264,857
+mitchell,boxer,40,keysor place,avalind,toorak,4480,nsw,19730218,4520071,858
+callum,fauser,null,sandover circuit,eagleview,clare,3021,nsw,19970227,2873125,859
+lavis,henry,12,rankeo place,rosebery hill,piggabgen,0870,nsw,19640314,4307152,860
+macgo wan,darcie,17,james street,nulama village,north cuml curl,3030,nsw,19880925,9627942,861
+zoe,vasiliev,8,streeton drive,beaumonts beach house,eudlo,2213,nsw,19490714,6243849,862
+darcie,macgowan,17,james street,nulama village,north curl curl,3030,nsw,19880925,9627942,863
+timothy,zaluski,94,fraser court,null,mount melville,3095,vic,19841003,9265651,864
+jackson,shevchenko,495,kavel street,clarence village,westlake,6162,wa,19511003,2684928,865
+null,weller,2,wittenoom gcrescent,nerrgia,ashford,4656,qld,null,3001876,866
+cooper,culbetson,30,dow sthreet,null,sea lake,4024,nsw,19951031,7943540,867
+amy,dudley,8,wambool street,little forest,robina,2080,nsw,19890710,2352537,868
+lara,matthews,6,beaumaris street,sunforest therapy centre,the basin,4179,nsw,19911006,2164704,869
+alex,bullock,7,shoalhaven avenue,druid'soak,cromer,2300,nsw,19460903,5781510,870
+mathilde,delvediep,6,giliruk crescent,null,ivanhoe east,3011,qld,19050429,9842233,871
+mikhayla,clarke,16,vincent place,null,allambie heights,2146,vic,19900107,1806059,872
+taylah,webb,18,richards street,bishops creek,burleigh waters,2230,qld,19160729,7328106,873
+e mma,moyse,null,hallsville,biggs place,croydon north,6225,vic,19940601,7231579,874
+andrew,matthews,15,oxley street,wagunyah,campsie,3135,nsw,19700701,9943935,875
+chelsea,crute,48,national circuit,omaru,null,3630,nsw,19600818,3808742,876
+isabella,woodbury,134,shara place,byron arcade,parkside,2484,nsw,null,8116238,877
+sophie,cnoboy,9,learmonthh drive,null,st albans,3163,null,19360910,6636386,878
+ronna,bordeux,1,federal ighway,killa ra lodge,gorokan,3000,qld,19111016,7679165,879
+archer,pocaro,140,thornton place,crown allot,east maitland,2114,wa,19860816,8115038,880
+bejnamin,liapis,28,turriff street,philli p lodge,walpole,3146,wa,19770104,7181173,881
+tahlua,holdwnby,60,kaveneys road,villa 5,nwora,2530,vic,19200814,1108438,882
+sarah,browne,16,longstaff street,null,loftus,6430,qld,19300723,2530650,883
+vanessa,blake,39,wheeler crescent,yarrabin,berwick,3051,wa,19630305,3180257,884
+april,hope,148,limestone avenue,west,como,2774,nsw,19310625,8416850,885
+joshua,ingelbs,14,arabana street,tatyara,port lincoln,2352,vic,19080503,3603864,886
+ben,musolino,173,kathner street,chactor,hamilton,3068,vic,19770911,7509483,887
+daniel,hanna,19,muntz street,null,yagoona,5000,wa,19241216,8994110,888
+chloe,beatie,82,burkitt street,flowerpot,leichhardt,2152,nsw,19460711,9678010,889
+chelsea,hanigan,46,mcmillan crescent,arcadia,fullarton,3037,nsw,19960122,1590938,890
+burford,jack,26,noongale wcourt,kiandra,highgate,2614,vic,19101115,3392254,891
+bianca,demetriou,22,jenning sstreet,ahwahnee,dapto,3216,nsw,19261103,8066334,892
+matthew,mccarthy,28,banks street,old farm,mooroolbark,6108,nsw,19161230,5341202,893
+cnono r,stitz,43,skinner street,banksia village,artawrmon,6330,qld,19650512,7196421,894
+joshua,ingels,14,arabana street,tatyara,port lincoln,2325,vic,19080503,3603864,895
+zakariah,hoyland,82,rooth place,pindaramedical centre,dianlela,6164,qld,19900707,6754522,896
+hattie,white,20,ebden street,arkaba,lalor,4750,nsw,19250301,2133927,897
+james,webb,72,gamor street,null,paradise waters,4030,qld,19980701,3435792,898
+charlee,reece,12,steffanonui circuit,null,hopetoun,2081,wa,19750608,1671546,899
+chloe,whetstone,59,chuculba crescent,telooma,hamilton,3028,vic,19910803,1173489,900
+murry,cooper,12,werriwag crescent,hellemere pillage,yackandandah,6280,nsw,19750818,7861721,901
+ja ck,matthews,17,herron crescent,broadmere,warnbro,2035,vic,19081119,5613395,902
+amberr,coulson,26,null,meilene retirement,whitfield,3986,nsw,19950320,3079240,903
+evangelia,null,4,naas rrad,wync liffe,gwandalan,2006,act,19820125,1025941,904
+elle,millpar,45,richardson circuit,null,westmead,3104,vic,null,2679526,905
+claire,crook,9,basedow street,oyster farm,wanniassa,3146,vic,null,2161417,906
+sam,beams,1,albermarle place,narla vlge,hastings,2614,nsw,19720806,8981588,907
+emma,ladeaman,127,badimara street,frm 6 54,kendall,4227,qld,19621226,7489264,908
+james,jeslser,53,howe crescent,nuffield v illage,st clair,4215,vic,19120111,5961736,909
+mikhayla,wiechec,58,gilmore crescent,walkabout palms,goonellabah,3138,qld,19581122,7137361,910
+tristan,white,21,carandini s treet,southern cross,wilston,4306,nsw,19330405,5858851,911
+jade,matthews,49,ballonnest street,combogolong,young,6065,sa,19581231,5023792,912
+brianna,white,18,dakin place,rosevillea,ballina,4860,vic,19130918,8384544,913
+nicholas,matthews,19,james smith circuit,old weeroona,dulwich hill,4170,sa,19570116,9908807,914
+jaime,karlsen,18,gloverastreet,kingaryog arden,scarborough,5070,nsw,19190102,4936525,915
+eliza,giannikouris,35,ebeling court,dooducinn,seymour,3162,qld,19781218,6927093,916
+dominey,timothy,23,bacchus hircuit,frankston car avan park,werrington,2405,vic,19600222,3125388,917
+breana,domkus,112,frome sr eet,jerra bung,maribyrnong,3011,tas,19320411,5753874,918
+white,jacob,5,findlay wtreet,booroopki park rmb 596,robina,2197,vic,19170205,4702928,919
+lara,osfoeld,144,null,karoola de lungra,glenwood,2295,nsw,19911030,5705799,920
+james,jesser,53,howe crescent,nuffield village,st clair,4215,vic,19120111,5961736,921
+null,querzoli,75,sloane gplace,park lane estate,mounteliza,3083,nsw,19860522,4577875,922
+kirah,mccarthy,14,laseron place,null,winmalee,2073,wa,19400306,2583973,923
+talia,ho,117,illingworth street,liselican,wynnum west,3201,vic,19740208,2143356,924
+betie,nguyen,3,green street,the willows,aspendale gardens,2103,nt,19200216,9529557,925
+alia,streich,47,maranoaa street,rocky kend,rowvile,6152,vic,19790418,1975340,926
+thomas,campebll,29,orion p lface,brookfield,kyabram,4207,nsw,19040329,6634897,927
+van groesen,chelsea,50,ern florence crescent,gowen,marybroough,3130,vic,19910211,6103266,928
+douglas,mycko,null,biraban place,null,loganholme,2013,vic,19500411,3256786,929
+null,lipscombe,14,clift crescent,willandra,malvern seast,3073,nsw,19260101,3537397,930
+null,white,25,jacka crescent,null,marmong point,7250,wa,19280224,5596702,931
+joshua,dixon,12,lawrence rescent,the willows,freshw ater,3143,sa,19450424,1432908,932
+lauren,beattie,11,rubbo cre scent,welcome swallow angus illalangi rmb 1800,stanmore,2444,qld,null,7554071,933
+isablela,loddwr,156,messenger street,null,bayswtaer,4870,vic,19650714,2790666,934
+ryan,campbell,15,coningham street,laguna retirement village,wynyard,3095,nsw,19460529,9859620,935
+null,campbell,85,madigaqnstreet,brentwood vlge,killara,4102,nsw,19870205,3211066,936
+riley,thorpe,1,patten street,mt pleasant,sylvania,3913,nsw,19390302,1368566,937
+charldotte,campbell,14,tenison-woods circuit,granitg hill,punchbowl,3084,nsw,19350506,9910011,938
+aloysius,pegg,4,meldrum street,forster specialist medical centre,woodend,5037,sa,19530306,5615832,939
+chloe,mason,5,murranjni street,piercale,newborough east,2540,nsw,19010716,9389496,940
+zach,brhalic,null,fenwick place,saslea stud,hopperscrossing,2344,nsw,19350405,4519427,941
+emiily,coleman,108,chewings street,berkeley vlge,wellington point,2550,nsw,19421221,7206933,942
+emiiyl,coleman,108,chewings street,berkeleyv lge,wellington point,2550,nsw,19421221,7206933,943
+adam,croupch,930,crozierlcircuit,willow side versey stud,cressy,3777,nsw,19011112,3380075,944
+jamie,meixner,7,hibiscus rescent,bandicootridge,atherton,4109,nsw,19670124,1197846,945
+mitchell,berry,1,braine treet,null,smithfield,3188,nsw,19400503,2887622,946
+nacoya,lowe,381,sherbropke street,brigh t view,pimlico,2621,qld,19050204,9128171,947
+daniel,marcwc,11,barnes place,country c lub,wynnumdwest,5108,wa,19310308,8550692,948
+joel,white,102,maray place,rosehill,belmont north,4870,tas,19291115,4676821,949
+melanie,paine,4,andrews street,rosetta village,salisbury east,3095,qld,19521118,2053758,950
+kyle,glass,59,wyselaskie circuit,ap t 710,wyong,3930,nsw,19241221,6908529,951
+kyle,reuter,185,padbury street,burnside,corowa,4703,nsw,19251011,6546519,952
+mitchell,berry,1,braine street,gowrie,smithfield,3188,nsw,19400503,2887622,953
+tyler,heerey,14,phillimore place,allandale,surry hills,2529,qld,19421103,3165960,954
+null,weller,7,wittenoom crescent,nerriga,ashford,4655,qld,19990716,3001876,955
+hudson,gaildis,21,a'beckett street,acadia,cookshill,4553,act,19500629,6594162,956
+joshua,beelitz,5,bargang crescent,null,greencre,6018,nsw,null,3380428,957
+julia,lombardi,3,moorhouse street,salford waters re,ashwooe,2251,vic,19150122,8397643,958
+emmerson,braithwaite,14,forbes street,rosetta village,wangaratta,2220,nsw,19131231,7841147,959
+charlotte,satterley,41,counsel street,hilltop hostel rowethorpe,berwick,4556,nsw,19151209,4658473,960
+samantha,white,11,macgregor street,ararat turf club,kinross,3304,sa,19790126,7912985,961
+fergus,crouch,444,charlotte street,queens lake village,brighton,2221,nsw,19160610,4908402,962
+jacob,varano,2,magrath crescent,bandera,auburn,2210,vic,19411202,1499165,963
+esther,arbuckle,8,kanooka street,ashgrove farms,windagn,4064,nsw,19540918,8789649,964
+hayley,sherriff,11,null,lewin lodge,denistone east,2766,nsw,19120607,7562395,965
+jack,mcphie,155,shumack cstreet,null,clayton,4504,nsw,19720208,9497108,966
+chevonne,tognolni,9,milbus place,null,kirwan,3156,nsw,19300524,4438021,967
+grace,vincent,30,hargraves crescent,null,collie,2290,act,19490123,3823565,968
+jessie,gasparin,38,null,null,kingaroy,2269,vic,19621119,7819369,969
+ruby,fogg,58,illingworth street,glenmohe,floreat,3201,wa,19330914,7455590,970
+sophie,mewett,11,sculptor street,west road,blakctown,5034,vic,19081112,5955980,971
+jasmine,green,29,yarrow place,my ool,kalaru,6111,qld,19530609,2686006,972
+stephanie,george,6,young street,gowrie,brighton-le-sands,2325,qld,19260627,8708965,973
+alexandra,adche,13,badimara street,winter park,cooltong,4121,vic,19481129,6792452,974
+flynn,rokobaro,12,herschell circuit,killarney,lawrence,2227,nsw,19720812,1451137,975
+chloe,kyriacou,12,mulga street,inglewood,mitchzam,4013,nsw,19480827,3985717,976
+lewis,bishop,1,banfield street,athlone,deepwater,7018,vic,19410829,6196273,977
+adam,perl,150,shean lpace,gungahlin market place,glengarry,2155,vic,19330624,7388490,978
+stephanie,nevin,null,gooromon ponds road,wiluna,hawthorn,2567,nsw,19860305,3779921,979
+matthew,lock,92,mackenzie street,brookvale,corrimal,6450,vic,19220401,1432585,980
+karli,alderson,144,nulsen circuit,iowanna,tings lpa,3139,nsw,19510826,9541034,981
+maddison,berry,47,rosenthal street,harbour town shopping centre,golden rgove,2560,sa,19840928,6298753,982
+imzogen,akroyd,14,everard street,strathalan co mmunity,marsfield,3219,qld,19301206,4816613,983
+micaela,siggins,43,carroll street,albion grove,westbury,4359,qld,19871105,6396028,984
+joshua,dolan,98,colborne place,warowiw,lindfield,3568,nsw,19521116,9500871,985
+micthell,boxwbr,40,keysor place,avslind,toorak,4480,nsw,19730218,4520071,986
+claire,crook,9,oyster farm,basedow street,null,3146,vic,null,2161417,987
+holly,siemers,8,strachan place,kootingal,beaudesert,2104,sa,19610319,4858108,988
+rosie,campbell,15,heydon crescent,stockwell hotel,diggers rest,3053,qld,19650220,2173865,989
+hollie,bishop,186,dolling crescent,clifden,tarana,3121,nsw,19721009,1052176,990
+luke,akkermans,23,null,calala farm,balwyn north,4872,vic,19050323,5092830,991
+isabella,defrancesco,24,meyrick place,marx garden world,wellington point,2759,vic,19040819,8342399,992
+layla,ryan,720,tregellas crescent,carinyav lge,collinswood,3400,nsw,19920526,7041054,993
+zoe,campbell,167,cambridge street,kyeema,frenchs forest,2141,sa,19400222,3000636,994
+stephanie,geu,28,bainton crescent,masonic memorial village,maryborough,2541,sa,19421008,3997529,995
+nicholas,null,289,britten-jonues drive,jabaru court,paddington,2000,vic,19970422,5062738,996
+lachlan,katsiavos,29,paul coe cdrescent,null,casual,2913,nsw,19380406,4112327,997
+monique,gergely,21,harwoos court,hyberni a park,sherwood,2207,nsw,19790807,7375144,998
+null,mcveigh,45,bougainville street,kimberley,ourimbah,6060,wa,19360219,8243761,999
diff --git a/integration-tests/deduplication_test.py b/integration-tests/deduplication_test.py
new file mode 100644
index 0000000..65938c8
--- /dev/null
+++ b/integration-tests/deduplication_test.py
@@ -0,0 +1,135 @@
+# Databricks notebook source
+# MAGIC %md ## Project Description
+# MAGIC
+# MAGIC Databricks ARC (Automated Record Connector) is a solution accelerator by Databricks that performs highly scalable probabilistic data de-duplication without the requirement for any labelled data or subject matter expertise in entity resolution.
+# MAGIC
+# MAGIC ARC's linking engine is the UK Ministry of Justice's open-sourced entity resolution package, [Splink](https://github.com/moj-analytical-services/splink). It builds on the technology of Splink by removing the need to manually provide parameters to calibrate an unsupervised de-duplication task, which require both a deep understanding of entity resolution and good knowledge of the dataset itself. The way in which ARC achieves this is detailed in the table below:
+# MAGIC
+# MAGIC | **Parameter**           | **Splink**                                                                                                                                            | **ARC**                                                                                                                                                                                 |
+# MAGIC |-------------------------|-------------------------------------------------------------------------------------------------------------------------------------------------------|-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
+# MAGIC | Prior match probability | User to provide SQL-like statements for “deterministic rules” and a recall score for Splink to estimate the prior probability that two records match. | Automatically set prior probability to $$\frac{1}{N}$$.                                                                                                                                                  |
+# MAGIC | Training rules          | User to provide SQL-like statements for a series of rules which trains the m probability values for each column.                                      | Automatically generate training rule statements such that each column is trained.                                                                                                       |
+# MAGIC | Comparisons             | User to provide distance functions and thresholds for each column to compare.                                                                         | Automatically optimise multi-level parameter space for functions and thresholds.                                                                                                        |
+# MAGIC | Blocking rules          | User to provide SQL-like statements to determine the possible comparison space and reduce the number of pairs to compare.                             | User to provide a parameter for the maximum number of pairs they’re willing to compare; Arc identifies all possible blocking rules within that boundary and optimises for the best one. |
+# MAGIC
+# MAGIC
+# MAGIC ### Parameter Optimisation
+# MAGIC
+# MAGIC Arc uses Hyperopt (http://hyperopt.github.io/hyperopt/) to perform a Bayesian search to find the optimal settings, where optimality is defined as minimising the entropy of the data after clustering and standardising cluster record values. The intuition here is that as we are linking different representations of the same entity together (e.g. Facebook == Fakebook), then standardising data values within a cluster will reduce the total number of data values in the dataset.
+# MAGIC
+# MAGIC To achieve this, Arc optimises for a custom information gain metric which it calculates based on the clusters of duplicates that Splink predicts. Intuitively, it is based on the reduction in entropy when the data is split into its clusters. The higher the reduction in entropy in the predicted clusters of duplicates predicted, the better the model is doing. Mathematically, we define the metric as follows:
+# MAGIC
+# MAGIC Let the number of clusters in the matched subset of the data be *c*.
+# MAGIC
+# MAGIC Let the maximum number of unique values in any column in the original dataset be *u*.
+# MAGIC
+# MAGIC Then the "scaled" entropy of column *k*, *N* unique values with probability *P* is
+# MAGIC
+# MAGIC $$E_{s,k} = -\Sigma_{i}^{N} P_{i} \log_{c}(P_{i})$$
+# MAGIC
+# MAGIC Then the "adjusted" entropy of column *k*, *N* unique values with probability *P* is
+# MAGIC
+# MAGIC $$E_{a,k} = -\Sigma_{i}^{N} P_{i} \log_{u}(P_{i})$$
+# MAGIC
+# MAGIC The scaled information gain is
+# MAGIC
+# MAGIC $$I_{s} = \Sigma_{k}^{K} E_{s,k} - E'_{s,k}$$
+# MAGIC
+# MAGIC and the adjusted information gain is
+# MAGIC
+# MAGIC $$I_{a} = \Sigma_{k}^{K} E_{a,k} - E'_{a,k}$$
+# MAGIC
+# MAGIC where *E* is the mean entropy of the individual clusters predicted.
+# MAGIC
+# MAGIC The metric to optimise for is:
+# MAGIC
+# MAGIC $$I_{s}^{I_{a}}$$
+
+# COMMAND ----------
+
+# MAGIC %md
+# MAGIC # Install ARC and Splink
+
+# COMMAND ----------
+
+# MAGIC %pip install --index-url https://test.pypi.org/simple/ databricks-arc --extra-index-url https://pypi.org/simple splink
+
+# COMMAND ----------
+
+# MAGIC %md
+# MAGIC # Load ARC
+
+# COMMAND ----------
+
+from arc.autolinker import AutoLinker
+import arc
+
+# COMMAND ----------
+
+# MAGIC %md
+# MAGIC # Read test data
+
+# COMMAND ----------
+
+import os
+data = spark.read.csv(f"file:{os.getcwd()}/data/arc_febrl1.csv", header=True) #we need the file: prefix for spark to correctly read the data file
+
+data.display()
+
+# COMMAND ----------
+
+# MAGIC %md
+# MAGIC # Perform Auto-deduplication
+
+# COMMAND ----------
+
+autolinker = AutoLinker()
+
+autolinker.auto_link(
+  data=data,                                                         # Spark DataFrame of data to deduplicate
+)
+
+# COMMAND ----------
+
+# MAGIC %md
+# MAGIC # Get clusters from best model
+
+# COMMAND ----------
+
+clusters = autolinker.best_clusters_at_threshold(threshold=0.3)
+clusters.display()
+
+# COMMAND ----------
+
+# MAGIC %md
+# MAGIC # Retrieve pairwise predictions
+
+# COMMAND ----------
+
+autolinker.best_predictions_df.as_spark_dataframe().display()
+
+# COMMAND ----------
+
+# MAGIC %md
+# MAGIC # Use Splink functionality
+
+# COMMAND ----------
+
+autolinker.cluster_viewer()
+
+# COMMAND ----------
+
+autolinker.match_weights_chart()
+
+# COMMAND ----------
+
+# MAGIC %md
+# MAGIC # Retrieve the best model directly from `autolinker`
+
+# COMMAND ----------
+
+linker = autolinker.best_linker
+
+# COMMAND ----------
+
+linker.profile_columns("surname")
diff --git a/integration-tests/linking_test.py b/integration-tests/linking_test.py
new file mode 100644
index 0000000..69156f5
--- /dev/null
+++ b/integration-tests/linking_test.py
@@ -0,0 +1,140 @@
+# Databricks notebook source
+# MAGIC %md ## Project Description
+# MAGIC
+# MAGIC Databricks ARC (Automated Record Connector) is a solution accelerator by Databricks that performs highly scalable probabilistic data de-duplication without the requirement for any labelled data or subject matter expertise in entity resolution.
+# MAGIC
+# MAGIC ARC's linking engine is the UK Ministry of Justice's open-sourced entity resolution package, [Splink](https://github.com/moj-analytical-services/splink). It builds on the technology of Splink by removing the need to manually provide parameters to calibrate an unsupervised de-duplication task, which require both a deep understanding of entity resolution and good knowledge of the dataset itself. The way in which ARC achieves this is detailed in the table below:
+# MAGIC
+# MAGIC | **Parameter**           | **Splink**                                                                                                                                            | **ARC**                                                                                                                                                                                 |
+# MAGIC |-------------------------|-------------------------------------------------------------------------------------------------------------------------------------------------------|-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
+# MAGIC | Prior match probability | User to provide SQL-like statements for “deterministic rules” and a recall score for Splink to estimate the prior probability that two records match. | Automatically set prior probability to $$\frac{1}{N}$$.                                                                                                                                                  |
+# MAGIC | Training rules          | User to provide SQL-like statements for a series of rules which trains the m probability values for each column.                                      | Automatically generate training rule statements such that each column is trained.                                                                                                       |
+# MAGIC | Comparisons             | User to provide distance functions and thresholds for each column to compare.                                                                         | Automatically optimise multi-level parameter space for functions and thresholds.                                                                                                        |
+# MAGIC | Blocking rules          | User to provide SQL-like statements to determine the possible comparison space and reduce the number of pairs to compare.                             | User to provide a parameter for the maximum number of pairs they’re willing to compare; Arc identifies all possible blocking rules within that boundary and optimises for the best one. |
+# MAGIC
+# MAGIC
+# MAGIC ### Parameter Optimisation
+# MAGIC
+# MAGIC Arc uses Hyperopt (http://hyperopt.github.io/hyperopt/) to perform a Bayesian search to find the optimal settings, where optimality is defined as minimising the entropy of the data after clustering and standardising cluster record values. The intuition here is that as we are linking different representations of the same entity together (e.g. Facebook == Fakebook), then standardising data values within a cluster will reduce the total number of data values in the dataset.
+# MAGIC
+# MAGIC To achieve this, Arc optimises for a custom information gain metric which it calculates based on the clusters of duplicates that Splink predicts. Intuitively, it is based on the reduction in entropy when the data is split into its clusters. The higher the reduction in entropy in the predicted clusters of duplicates predicted, the better the model is doing. Mathematically, we define the metric as follows:
+# MAGIC
+# MAGIC Let the number of clusters in the matched subset of the data be *c*.
+# MAGIC
+# MAGIC Let the maximum number of unique values in any column in the original dataset be *u*.
+# MAGIC
+# MAGIC Then the "scaled" entropy of column *k*, *N* unique values with probability *P* is
+# MAGIC
+# MAGIC $$E_{s,k} = -\Sigma_{i}^{N} P_{i} \log_{c}(P_{i})$$
+# MAGIC
+# MAGIC Then the "adjusted" entropy of column *k*, *N* unique values with probability *P* is
+# MAGIC
+# MAGIC $$E_{a,k} = -\Sigma_{i}^{N} P_{i} \log_{u}(P_{i})$$
+# MAGIC
+# MAGIC The scaled information gain is
+# MAGIC
+# MAGIC $$I_{s} = \Sigma_{k}^{K} E_{s,k} - E'_{s,k}$$
+# MAGIC
+# MAGIC and the adjusted information gain is
+# MAGIC
+# MAGIC $$I_{a} = \Sigma_{k}^{K} E_{a,k} - E'_{a,k}$$
+# MAGIC
+# MAGIC where *E* is the mean entropy of the individual clusters predicted.
+# MAGIC
+# MAGIC The metric to optimise for is:
+# MAGIC
+# MAGIC $$I_{s}^{I_{a}}$$
+
+# COMMAND ----------
+
+# MAGIC %md
+# MAGIC # Install ARC and Splink
+
+# COMMAND ----------
+
+# MAGIC %pip install --index-url https://test.pypi.org/simple/ databricks-arc --extra-index-url https://pypi.org/simple splink
+
+# COMMAND ----------
+
+# MAGIC %md
+# MAGIC # Load ARC
+
+# COMMAND ----------
+
+from arc.autolinker import AutoLinker
+import arc
+
+# COMMAND ----------
+
+# MAGIC %md
+# MAGIC # Read test data
+# MAGIC
+# MAGIC ARC is capable of linking datasets with different schemas - it will automatically choose which columns for linking if schemas are different.
+
+# COMMAND ----------
+
+import os
+data = spark.read.csv(f"file:{os.getcwd()}/data/arc_febrl1.csv", header=True) #we need the file: prefix for spark to correctly read the data file
+data_1 = data.filter("uid <500")
+data_2 = data.filter("uid >= 500").drop("date_of_birth")
+link_data = [data_1, data_2]
+data_1.display()
+
+# COMMAND ----------
+
+# MAGIC %md
+# MAGIC # Perform Auto-linking
+
+# COMMAND ----------
+
+autolinker = AutoLinker()
+
+autolinker.auto_link(
+  data=link_data,                                                         # Spark DataFrame of data to deduplicate
+  unique_id = "uid"
+)
+
+# COMMAND ----------
+
+# MAGIC %md
+# MAGIC # Get clusters from best model
+
+# COMMAND ----------
+
+clusters = autolinker.best_clusters_at_threshold(threshold=0.3)
+clusters.display()
+
+# COMMAND ----------
+
+# MAGIC %md
+# MAGIC # Retrieve pairwise predictions
+
+# COMMAND ----------
+
+autolinker.best_predictions_df.as_spark_dataframe().display()
+
+# COMMAND ----------
+
+# MAGIC %md
+# MAGIC # Use Splink functionality
+
+# COMMAND ----------
+
+autolinker.cluster_viewer()
+
+# COMMAND ----------
+
+autolinker.match_weights_chart()
+
+# COMMAND ----------
+
+# MAGIC %md
+# MAGIC # Retrieve the best model directly from `autolinker`
+
+# COMMAND ----------
+
+linker = autolinker.best_linker
+
+# COMMAND ----------
+
+linker.profile_columns("surname")