An Ensemble Blocking Approach for Entity Resolution of Heterogeneous Datasets

Entity Resolution, also called record linkage or deduplication, refers to the process of identifying and merging duplicate versions of the same entity into a unified representation. The standard practice is to use a Rule based or Machine Learning based model that compares pairs of records and assigns a score to represent the pairs’ Match/Non-Match status. However, performing an exhaustive pair-wise comparison on all pairs of records leads to quadratic matcher complexity and hence a Blocking step is performed before the Matching to group similar entities into smaller blocks that the matcher can then examine exhaustively. Several blocking schemes have been developed to efficiently and effectively block the input dataset into manageable groups. At our organization, we perform deduplication on massive datasets of people profiles collected from disparate sources with varying informational content. We observed that, employing a single blocking technique did not cover the base for all possible scenarios due to high heterogeneity in our data sources. In this paper, we describe our ensemble approach to blocking that combines two different blocking techniques to leverage their respective strengths. In the online recruitment domain, recruiters source qualified candidates for their clients’ job openings from a pool of both active and passive candidates. Active candidates are actively looking for work and hence they keep their personal brand updated. Passive candidates on the other hand are not likely to update their digital profile information, such as, resume and online social profiles, as frequently as active candidates, if at all. For proactive recruitment, recruiters usually leverage candidate search solutions from disparate vendors to identify qualified talent for their clients. Since each vendor may provide a different facet of a candidates’ profile (e.g., social, technical, professional), recruiters usually want to aggregate such profile information to get a complete view of a candidate. At our organization, we aim to develop a one-stop search shop for recruiters that presents an enhanced view of a candidate by consolidating the information contained in a candidate’s resume in our resume database, with their open web profiles. We aggregate candidate information and online activity from over 100 different sources and identify the ones Copyright c © 2017, Association for the Advancement of Artificial Intelligence (www.aaai.org). All rights reserved. that belong to the same candidate. Entity Resolution (ER), also known as Deduplication or Record Linkage, is a long-established challenge in the Artificial Intelligence (AI) domain, where the goal is to provide accurate, fast and scalable solutions to identify and match duplicate entities. The core of the deduplication framework is the Matcher component, that functions as the logical decision unit to determine if the compared entities are a match or a non-match. The naive approach to deduplication involves the matcher performing a pair-wise comparison on all available profiles, the results of which are aggregated together to represent duplicate entities. However, due to the massive volumes of data typically involved (we currently process about 0.5 billion profiles), conducting such exhaustive comparisons is not a scalable option. To tackle this, traditionally, a Blocking process is applied, where, seemingly similar profiles are grouped together into smaller sized blocks, which are then exhaustively explored by a Matcher component. The Blocker forms a key component in the entire ER pipeline, as it determines the entities that would be examined by the matcher, hence directly influencing the efficiency and effectiveness of the ER process. Several blocking schemes have already been proposed and widely used to reduce matching complexity, without impacting the recall of the ER framework. However, in our application, since we procure data not only from multiple sources, but also with each source representing different aspects of a person (Social, Professional, Technical), employing a single blocking method proved insufficient to guarantee comprehensive duplicate coverage. Running time is also a concern, especially when dealing with huge volumes of data. We experimented with two popular blocking techniques and found that, using either one of the blocking techniques resulted in loss of a few potential duplicate pairs from being fed into the Matcher stage, thereby reducing the effectiveness of the entire deduplication process. While one blocking method was well suited for sparse, heterogeneous datasets, it required apriori knowledge of the relative importance of attributes. Also, the run time complexity prevented us from employing it extensively. The other technique, although less complex and generic, does not guarantee complete pair coverage for large datasets. In this paper, we propose an ensemble blocking approach that combines two popular blocking strategies to leverage their respective Proceedings of the Thirtieth International Florida Artificial Intelligence Research Society Conference