Splink is a PySpark package that implements the Fellegi-Sunter model of record linking, and enables parameters to be estimated using the Expectation Maximisation algorithm. The package is fully open source and can be found on GitHub. It is accompanied by a set of interactive demos to illustrate its functionality, whereby users can run real record linking jobs in their web browser. The package closely follows the approach described in fastLink. In particular it implements the same mathematical model and likelihood functions described in the fastLink paper (see pages 354 to 357), with a comprehensive suite of tests to ensure correctness of the implementation. In addition, Splink introduces a number of innovations:
Comprehensive graphical output showing parameter estimates and iteration history make it easier to understand the model and diagnose convergence issues.
An ‘intuition report’, which can be generated for any record pair, which explains the estimated match probability in words.
Support for deduplication, linking, and a combination of both, including support for deduplicating and linking multiple data sets.
Greater customisability of record comparisons, including the ability to specify custom, user defined comparison functions.
Term frequency adjustments on any number of columns.
It’s possible to save a model once it’s been estimated - enabling a model to be estimated, quality assured, and then reused as new data becomes available.
A companion website provides a complete description of the various configuration options, and examples of how to achieve different linking objectives. In addition, the source code is fully documented using Google’s recommended documentation style. So far, the MoJ has used it to tackle record linkage problems up to around 15 million records with a runtime of less than an hour, but it is anticipated that the approach can scale to substantially larger data sets. A clerically labelled data set has been used to assess the performance of Splink. This has shown Splink has improved accuracy (that is, an unambiguously better ROC curve) compared to the previous rules-based approach used by the MoJ.