Ensemble Semi-Supervised Approach for Unsupervised Record Linkage Problem
By: Ahmed Magdy
Tags: Unsupervised Learning, Machine Learning, Semi Supervised Learning, Record Linkage

In this article I’m going to explain and trying to implement an approach proposed in A Novel Ensemble Learning Approach to Unsupervised Record Linkage Anna Jurek, Jun Hong, Yuan Chi, Weiru Liu paper for using semi supervised learning in matching databases with automatic seed selection.

Record linkage is a process of identifying records that refer to the same real-world entity. Many existing approaches to record linkage apply supervised machine learning techniques to generate a classification model that classifies a pair of records as either match or non-match.

Record linkage is necessary when joining data sets based on entities that may or may not share a common identifier example joining hospital data sets for medical applications or Hotel data for mapping different suppliers' data for OTA.

The main requirement of such an approach is a labeled training dataset. In many real-world applications, no labeled dataset is available hence manual labeling is required to create a sufficiently sized training dataset for a supervised machine learning algorithm.

To avoid the manual labeling process this will lead us to deal with the problem either using Unsupervised approaches like clustering or using the semi-supervised approach with few labeled data, here we are going to talk about semi-supervised or Self Learning approaches.

The proposed approach is to train with self-learning algorithm which takes small labeled seed at the start and train on it then predict on the other unlabeled data and then select the most confident points and join them to the labeled seed and start the training process again and repeat until no unlabeled data exists.

the difference here is the automatic seed selection, to avoid the time needed to select the right seed for matching and not matching algorithm.

the whole process is done in 6 steps :

calculate different distance metrics for each given feature. make a combination of all the features from the metrics and store them as feature schema select the most diverse sachems as features to our classifiers, so that each classifier is deciding from a different perspective than other classifiers (ensemble learning). for each feature schema, select the seed for matching and not matching classes. make several classifiers = number of selected feature sachems and start the self-learning process. the last step is to calculate the difference between each classifier and the other classifier so that if there is a classifier that has a big difference with others we remove it from the final prediction.

Let's talk about each one in detail and see how to implement it with python.

1- process the data and create features :

This step actually depends on the data you have so for example you might have data with (Name-Email-phone number ), so this step is done when you create similarity distance for each one like (Levenshtein distance — Jaro-Winkler — fuzzy matching) for strings and (Hamming distance ) for phone number.

As I said it depends on the data, so the code is different but here is a sample from the result you should have at this point and some helpful python libraries.

2- generate the feature schema

The target of the ensemble classifiers is to make each classifier look at different features from the other, so in this step we want to generate a pool of schemas each schema has different metrics for each feature like this

to do that we are going to use Cosine similarity to calculate the similarity between each metric and the other for each feature.

let's stick with the (name — email — phone) example, so the algorithm is an iterating process where we loop through each feature and get the lowest pair of metrics in term of cosine similarity by using this function:

from itertools import combinations

def get_lowest(df,cols):

low = 0

pair1 = 0

pair2 = 0

for k in combinations(cols,2):

cos=spatial.distance.cosine(df[k[0]].values,df[k[1]].values)

print(cos,k)

if cos>low:

low = cos

pair1 = k[0]

pair2 = k[1]

return pair1,pair2

I’m using the itertools library to generate all the pairs.

Now after we got the lowest pair we start to compare all the other metrics for the same feature to these pairs and select metrics in which no cosine similarity between two similarity measures is greater than a threshold.

from collections import defaultdict

def generate_pool(df,p):

features=defaultdict(list)

for feature in features:

vf = [col for col in df.columns if feature in col]

V = []

pair1,pair2=get_lowest(df,vf)

V.append(pair1)

V.append(pair2)

features[feature].append(pair1)

features[feature].append(pair2)

vf.remove(pair1)

vf.remove(pair2)

while(len(vf)>0):

f=vf.pop()

c=0

for item in V:

num = spatial.distance.cosine(df[f].values,df[item].values)

if (num > p) & (f not in features):

c+=1;

if(c==len(V)):

V.append(f)

features[feature].append(f)

return features

Now let's create a list combination of the selected similarities

features=dict(features)

a=list(features.values())

import itertools

features_schema=list(itertools.product(*a))

Now we have our schemas let's start the initial training seed selection process

3- Automatic Seed selection:

we need our algorithm to find the number of matching rows and mismatching rows to do so we will first normalize all our similarity measures so that

{1: means perfect match, 0: mean perfect mismatch}

then we will calculate the Manhattan distance between 1 vector for matching seed and 0 vector for mismatching seed.

as an example, we have row values like this <100,80,92> we first normalize it to be <1,0.8,0.92> and then subtract it from 1 vector <1,1,1> and the result is <0,0.2,0.08> by the sum over axis 1 we got 0.28 and the smaller value we got the more matching seed we have.

for now, this will help us get a good initial seed, but what if there is a feature as an example the Name helps us getting more matching seed than others, for the email or address it might be hard to find a perfect similarity metric for them.

To solve this we will make a field weighting algorithm, which gives more weight to a feature if it provides more good seeds while selecting the seeds

The algorithm will be like this we will first give every field equal weight = 1/num of fields, then we will calculate the Manhattan distance for the match and mismatch seed and update our weights based on specific equations I will talk about it, and if the difference between the new weights and the old weights is smaller than a threshold we stop.

An example of the non-match seed selection process with 2-dimensional similarity vectors. The process is performed in 3 iterations. In the first iteration (blue) the weights of the fields are ( 1 2, 1 2 ). In the second iteration (red) the weights were updated to ( 2 3, 1 3 ). In the last iteration (green) the weights were set as (34, 1 4 ).

so how we update our weights?

The Formula is more explained in the paper but the idea is that we sum all the distances between the matching seed and the 1 vector and the non-matching seed and 0 vector for each field and if the result = 0 then this field is having more power in selecting right seed more than others so it gets a weight of 1 and everything else is 0, but if we have more than one field with result = 0 then giving them all equal weights and the other field 0, else if there is no field have result = 0 then we assign them according to the formula above.

here is the function the calculate the weights

def calculate_weights(df,features,Xm,Xu,w):

ls=[]

djs=[]

for index,f in enumerate(features):

match =(df.loc[Xm][f]-1).sum(axis=0)

notmatch =(df.loc[Xu][f]-0).sum(axis=0)

dj = match + notmatch

djs.append(dj)

if dj == 0:

ls.append(index)

if len(ls)>0:

w = np.zeros(w.shape)

w[ls] = 1/len(ls)

else :

s = sum(1/d for d in djs)

djs = [round(1/(dj*s),5) for dj in djs]

w = np.asarray(djs).reshape(w.shape)

return w

and this is the function that used to get the seeds

def automatic_seed_selection(df,Mm,Mu,e,w):

Xm = set()

Xu = set()

tm,tu = 0,0

while(len(Xm)<Mm):

t = np.dot(abs(df[~df.index.isin(Xm)].values-1),w)

Xm.update(df[~df.index.isin(Xm)][t<=tm].head(Mm-len(Xm)).index) ### fill the seed until we reach Mm without repeating

tm+ = 0.05

while(len(Xu)<Mu):

t = np.dot(abs(df[~df.index.isin(Xu)].values-0),w)

inde = set(df[(~df.index.isin(Xu))][t<=tu].head(Mu-len(Xu)).index) ### make sure that no matching point is selected for not matching point

Xu.update(inde-Xm)

tu+ = 0.05

wnew = calculate_weights(df,df.columns,Xm,Xu,w)

while( np.array(abs(wnew-w)>e).any()):

Xm = set()

Xu = set()

tm,tu = 0,0

w = wnew

while(len(Xm)<Mm):

t = np.dot(abs(df[~df.index.isin(Xm)].values-1),w)

Xm.update(df[~df.index.isin(Xm)][t<=tm].head(Mm-len(Xm)).index)

tm+ = 0.05

while(len(Xu)<Mu):

t = np.dot(abs(df[~df.index.isin(Xu)].values-0),w)

inde = set(df[(~df.index.isin(Xu))][t<=tu].head(Mu-len(Xu)).index)

Xu.update(inde-Xm)

tu+ = 0.05

wnew = calculate_weights(df,df.columns,Xm,Xu,w)

return Xm,Xu

Mm is the number of rows we want in the matching seed, Mu is the same for the mismatching seed, the function starts with tolerance = 0 which mean getting the perfect match and mismatch and increase by 0.05 each iteration until we fill the Mm and Mu.

I’m using python sets to make sure that no example is provided in matching and non-matching at the same time.

gathering all this together we get the indices of the seeds for each feature schema

Xm =[]

Xu =[]

for schema in sachems:

print(schema)

z = len(schema)

w = np.full((z,1),1/z)

x1,x2 = automatic_seed_selection(pairs[schema],200,8000,0.5,w)

Xm.append(x1)

Xu.append(x2)

4- Selecting highly diverse sets of seeds

For each schema we now have a small number of labelled examples (seeds). Consequently, any binary classifier can be trained using the selected seeds to classify the remaining unlabelled similarity vectors. With ensemble learning it is a common practice to select a collection of Binary Classifiers that has the highest diversity to form an ensemble.

example :

The table above shows a simplified example of two ensembles, each consisting of three BCs. The output of each classifier is denoted as 1 for being correct and 0 for being wrong. The final prediction of the ensemble is determined as the mode of the three individual predictions. It can be seen that in Ensemble I each of the BCs makes mistakes on different examples. As a consequence the combined classification is better than any of the BCs. In Ensemble II, every BC misclassified the same example. Therefore, combining them does not make any overall improvement.

therefore we are going to calculate the Q static value which is a pairwise diversity measure that is defined based on a 2 × 2 table representing the relationship between predictions of two classifiers.

and it is calculated like this

The value of Q1, 2 is between −1 and 1. The classifiers with high values of N00 and N11 (classifiers that make the same predictions) will have a positive value. On the other hand, the classifiers with high values of N10 and N01 (classifiers that have different classifications on the same examples) will have a negative value.

here is the code for it

## combine all the selected maching seed from each features schema

ALL = set().union(*Xm) def calculate_Q(set0,set1):

S00 = set0.intersection(set1)

S11 = ALL-(set0.union(set1))

S01 = set0-set1

S10 = set0-set1

Q =((len(S00)*len(S11))-(len(S01)*len(S10)))/((len(S00)*len(S11))+(len(S01)*len(S10)))

return Q from itertools import combinations

Qs=[]

for f in combinations(range(0,len(Xm)),2):

Qs.append((calculate_Q(Xm[f[0]],Xm[f[1]]),f[0],f[1])) pd.DataFrame(Qs).sort_values(by=0)

5- Training

now we finally reached the training step as we said before for the semi-supervised we train on the seed, predict all the data, select the most confident one and add them to the seed, and iterate.

Thank god there are some awesome people made that for us here is the repo for the self-learning library.

and this is the code for this step

from frameworks.SelfLearning import *

from sklearn.linear_model import LogisticRegression, SGDClassifier

models=[]

for i in range (0,len(features_schema)):

model=SelfLearningModel(LogisticRegression(tol=1e-3))

models.append(model) X_features=[]

for schema in schema:

X_features.append(schema)

i=0

for model in models:

X = pairs[X_features[i]].values

name = 'match_'+str(i)

pairs[name] = -1 ##unlabeled sympol

pairs.loc[Xm[i],name] = 1

pairs.loc[Xu[i],name] = 0

y = pairs[name].values

model.fit(X,y)

pairs.loc[pairs[name] == -1,name] = model.predict(pairs[pairs[name] == -1][X_features[i]].values)

i+=1

if i == len(models):

break

6- Selecting the final ensemble using the contribution ratios of BCs

Since the proposed method is fully unsupervised we are not able to evaluate how good each of classification models is. Therefore, there is a risk of including classifiers with very poor accuracy (i.e., below 0.5) which are not valid in general, into the ensemble. In order to address this issue we propose a statistic which takes into account the contribution ratio of each individual BC to the final output of the ensemble. Each BC makes a prediction on each record pair as match or non-match. Following this, the mode of all the predictions by all the BCs is taken as the prediction of the ensemble.

and the contribution ratio is calculated from this formula

and here is the code by making mode prediction on except the initial seed

def calculate_CR(df):

CRS=[]

for col in ['match_0','match_1','match_2']:

ls = [abs(~(i^j)) for i,j in (zip(predicted[col],predicted['mode']))] CRS.append(ls.count(1)/len(ls))

return CRS

Conclusion

the problem already have other solution like genetic algorithms but this approach is very helpful when you are not familiar with the genetic approach and you don’t have the time or the resources for labeling the data, the paper already provided its results and I already tested it on different data set and the result was good.

This is my first article hope u like it, and please don’t hesitate to comment with any advice u have for improvements.

Thanks