Dataset deduplication using spark’s MLlib | by Ronald Ángel | Towards Data Science

A deduplication process is always important for companies with a huge amount of data. From one thing, deduplication minimizes the amount of space required to store business data and will bring a lower infrastructure cost and better performance for our pipelines. From another thing, reducing the number of duplicates will reduce the pipelines complexity and will increase the time to business via continuing integration and continuous delivery (CI/CD).
Sometimes a deduplication process consists of a simple text to text matching and you can simply choose either a CRC32-Checksum or an MD5 matching. However, there are some situations where the dataset rows differ only for some small text discrepancies on some of the columns, even though they represent the same entity. Thus, this article shows an entities recognition and linking process using two different spark approaches over a specific dataset of products collected by scrapping e-commerce websites will be used.
The entire code and process describe following could be found here:
github.com
The general process could be found on this trait (... Yes, I use scala for data science !!!):
package com.sample.utilsimport org.apache.spark.sql.DataFrametrait OperationsHelper {  def ds: DataFrame  def preparedDataSet()(df: DataFrame): DataFrame  def deduplicateDataSet()(df: DataFrame): DataFrame  def resultsDataFrame()(df: DataFrame): DataFrame}
As you will see the idea behind this helper will be having a functional pipe from where easily chain dataframe transforms could be called.
Techniques to reduce dimensionality are widely used by the data science community to get a smaller set of features to analyze and obtain better performance while training and evaluating models. The PCA method allows dimensionality reduction while keeping those features that describe a large amount of the information. Consequently, this pre-processing stage follows these steps:
Data Cleaning: Cleaning the data to have a common scale. For the case of the products, dataset consists of a simple text cleaning for cases, white spaces, encoding, and symbols.
Features Selection: Using the PCA technic a set of features are selected.(“titleChunk”, “contentChunk”, “color”, “ productType”)
The content found on the features above contains most of the discrepancies for candidate duplicate products.
Locality-sensitive hashing is a technic used for entity resolution, then records that represent the same entity will be found. The spark MLlib has a custom LSH implementation used here to find duplicates as follow:
First, hashes are generated using a concatenation of selected features (PC above). For a real-world example hashes for each feature could be generated. However, for this example and in order to get results faster a simple concatenated column is used.
Then, this column is used for generating LSH vectors as follow:
— A tokenizer generates the list of words for a record using a words stopper.
— A CountVectorizerModel creates the vectors with hashes and buckets (similar hashes) for the LSH algorithm.
val pccTokenizer = new Tokenizer()  .setInputCol(OperationsHelperLSH.ConcatComments)  .setOutputCol(OperationsHelperLSH.ColumnWordsArray)val wordsArrayDF = pccTokenizer.transform(df)val remover = new StopWordsRemover()  .setCaseSensitive(false)  .setStopWords(OperationsHelperLSH.stopWords)  .setInputCol(OperationsHelperLSH.ColumnWordsArray)  .setOutputCol(OperationsHelperLSH.ColumnFilteredWordsArray)val wordsFiltered = remover.transform(wordsArrayDF)val validateEmptyVector = udf({ v: Vector => v.numNonzeros > 0 }, DataTypes.BooleanType)val vectorModeler: CountVectorizerModel = new CountVectorizer()  .setInputCol(OperationsHelperLSH.ColumnFilteredWordsArray)  .setOutputCol(OperationsHelperLSH.ColumnFeaturesArray)  .setVocabSize(VocabularySHLSize)  .setMinDF(10)  .fit(wordsFiltered)val vectorizedProductsDF = vectorModeler.transform(wordsFiltered)  .filter(validateEmptyVector(col(OperationsHelperLSH.ColumnFeaturesArray)))  .select(col(OperationsHelperWindowStrategy.ConcatComments),    col(OperationsHelperLSH.ColumnUniqueId),    col(OperationsHelperLSH.ColumnFilteredWordsArray),    col(OperationsHelperLSH.ColumnFeaturesArray))(vectorizedProductsDF, vectorModeler)
Class: com.sample.products.OperationsHelperLSH.scala
In order to finish the training step, a MinHashLSHModel is used to train the products data, generating the final buckets of similar products.
Finally, using KNN similar hashes for a category could be found.
  /**    * Uses the dataset to train the model.    *    */  def deduplicateDataSet(df: DataFrame): (DataFrame, MinHashLSHModel) = {    val minLshConfig = new MinHashLSH().setNumHashTables(hashesNumber)      .setInputCol(OperationsHelperLSH.ColumnFeaturesArray)      .setOutputCol(OperationsHelperLSH.hashValuesColumn)    val lshModel = minLshConfig.fit(df)    (lshModel.transform(df), lshModel)  }  /**    * Applies KNN to find similar records.    *    */  def filterResults(df: DataFrame,                    vectorModeler: CountVectorizerModel,                    lshModel: MinHashLSHModel,                    categoryQuery: (String, String)                   ): DataFrame = {    val key = Vectors.sparse(VocabularySHLSize,      Seq((vectorModeler.vocabulary.indexOf(categoryQuery._1), 1.0),        (vectorModeler.vocabulary.indexOf(categoryQuery._2), 1.0)))    lshModel.approxNearestNeighbors(df, key, nearNeighboursNumber).toDF()}
To run an example: Go to the test com.sample.processor.products.ProcessorProductsLshTest and you will see a complete flow running.
Input Params:
category → color = ‘negro’ and productType = ‘tdi’.
nearNeighboursNumber → 4
hashesNumber → 3 (More hashes more precision but more computing cost).
Pros:
Accurate: If a complete set of fields (representing the string) is used, the correct value for hashes and neighbors could detect almost all the repeated values.
Faster: compared with other ML strategies as Term-frequency-inverse, etc.
Cons :
A cluster with good resources is needed.
Needs a process for data cleaning.
Levenshtein is an algorithm used for strings fuzzy matching. Basically, this method measures the difference between two strings. Furthermore, the spark windows functions allow dataset analytics function in a concise way, avoiding multiple groupBy and Join operations. Thus, this method defines a 2-level window to group similar data and then applies Levenshtein to values in the same windows to discover duplicates. The process is described here:
First of all, a set of records described as non-fuzzy is selected. The list contains columns that represent categories and are were free of errors most of the times in the PCA process: (“productType”, “city”, “country”, “region”, “year”).
This window represents the general windows hash for the analysis.
Secondly, a second window to discover quite similar records is applied. This list represents records that are neither part from the fuzzy list (PCA) nor from the non-fuzzy list: (“doors”, “fuel”, “make”, “mileage”, “model”, “color”, “price”)
Note: the “date” field helps to order and get only the most recent.
Then, For each group applies levenshtein (string difference only in the second window) over the concatenated most fuzzy fields from PCA results: (“titleChunk”, “contentChunk”).
As you can see an MD5 representation of the columns is used instead of each String to have better performance:
keyhash: MD5 for the category column set. The picture below shows many products within the same category.
hashDiff: MD5 hash that represents the non-fuzzy set. The picture below shows products within the same category but with a different description (> levenshteinThreshold) and also those with a small levenshtein (< levenshteinThreshold) difference having the same hashDiff.
Finally, the values with the same hashes (both) and rank only change the row_num. Filtering row_num == 1 is possible to get the deduplicate Data set.
/**  * Applies windows functions and Levenshtein to group similar  categories.  *  */override def deduplicateDataSet()(df: DataFrame): DataFrame = {  df    .withColumn(OperationsHelperWindowStrategy.ColumnRank, dense_rank().over(windowProductKeyHash))    .withColumn(OperationsHelperWindowStrategy.ColumnHashWithDiff,      concat(col(OperationsHelperWindowStrategy.ColumnCategoryFieldsHash),        when(levenshtein(          first(OperationsHelperWindowStrategy.ConcatComments).over(windowProductsCategoryRank),          col(OperationsHelperWindowStrategy.ConcatComments)) >= levenshteinThreshold, lit("1"))          .otherwise(lit(""))))    .withColumn(OperationsHelperWindowStrategy.ColumnRowNum, row_number().over(windowProductsCategoryRank))}
Class: com.sample.products.OperationsHelperWindowStrategy.scala
To run an example: Go to the test com.sample.processor.products.ProcessorProductsWindowsTest and you will see a complete flow running.
Input Params: levenshteinThreshold → 6
Results:
The results are deduplicated after filtering rn == 1. This removes > 1/3 of the data in the sample dataset.
Pros:
More control in the spark partitioner and functions.
Cons :
Could have much more false positives.
A deduplication process depends always on the company needs and the amount of data to analyze. This article describes two different strategies. As a result, Levenshtein with windows functions is good enough for small dimensionality problems; otherwise, LSH is always the best option