Finding Value in Unlabeled Data
By: Zack Miller
Tags: Machine Learning, String Matching, Unsupervised Learning, Unlabeled Data, Data Science

Photo by Agence Olloweb on Unsplash

“Machine learning” conjures up images of complex neural networks and massive data warehouses, but you probably already have all you need to start reaping the benefits.

For businesses that are new to machine learning, it can sound like an overly complex buzzword that requires massive amounts of specialized data, and even more talent and expertise. In reality, machine learning and data science are just about generating insight by recognizing patterns in all kinds of data. With the right approach, companies can ease into things by extracting value from data and assets that already exist within their organization. In this post, I’ll discuss how the application of data science and machine learning techniques helped transform metadata from an existing content management system into a new QA process.

The Situation

A client was looking to automate some quality controls for their content pipeline. In this instance, the concern was a portal where users could upload a file and include a short description. If the description did not match the contents of the file, they wanted to raise a warning so that the error could be corrected. The portal had been in operation for quite some time, which meant that the client had access to plenty of existing descriptions and file content.

The initial request was to find a string matching algorithm that would take in the file contents and descriptions and return a confidence score for whether or not the inputs were a match. This would involve selecting several candidate algorithms, applying them across the dataset, and comparing their results. However, because the pipeline previously had no quality controls, there was nothing in the dataset to indicate whether the description for each file did, in fact, match the content. This type of data is referred to as unlabeled data.

Unlabeled data is a common snag in many machine learning projects. One remedy is to manually review the data and create labels for each row, but this can quickly become expensive and time-consuming. There are also heuristic tools, like Snorkel, that allow developers to write custom labeling code to enrich their dataset — but even this process can require a high level of effort and domain expertise. Labeled data can be used to train supervised learning models, but it is also possible to train certain models without ever labeling the data. This is called unsupervised learning.

The Data

The format of the existing dataset contained a unique identifier for each file, along with a string description and a list of strings extracted from the file contents via ASR (automatic speech recognition) and OCR (optical character recognition).

Sample rows in the client’s media database

In the example above, row 1 should match because the description is paired well to the contents of the file. The description for row 2 is unlikely to pass QA because it does not relate to the contents.

The Solution

Six different fuzzy string matching algorithms were selected as candidates and applied to the dataset using the fuzzywuzzy Python package to find the best scoring match among the extracted file contents. The results looked like this:

Sample scores generated by several string matching algorithms

By examining the difference in score distribution between each of the candidate functions, we can see that there is not an immediately obvious consensus about which rows do or do not match. Heuristic labeling alone does not appear to be enough to label our dataset.

Distributions of scores from each string matching algorithm

This is where unsupervised learning can help. If we treat the output scores of each algorithm as an input feature, we can train a classifier model to divide the input rows into several possible classes or categories. One such model is called a k-means classifier and allows us to choose the number of clusters we will generate. Since we are trying to create a binary pass-fail label for the data, we can train a k-means model to find 2 clusters in the model and review the distribution of the scores in each. If we do that, the scores look something like this:

Score distribution for two clusters generated by k-means classification

The scores in the above clusters are approximately bimodal, with the mean of the potential match cluster averaging a score of 0.8 and the potential reject cluster averaging 0.47. This is a meaningful result, but deploying such a classifier in a production pipeline would mean running 6 separate matching functions against every upload and passing the results to a seventh process to determine whether to accept or reject the file description. It would be less cumbersome to apply a single scoring method that best represents the distinction between the two labeled classes.

The SHAP package in Python allows us to analyze the inputs and outputs of various machine learning models using an “optimal credit allocation” approach. This effectively tells us which inputs most greatly influence the output of a model. Applied to the labeled clusters above, we obtain the following:

Influence of each matching algorithm, as determined by SHAP

Method 6 is the most discriminating scoring function, and gives us enough confidence to include it as a standalone part of the quality control pipeline. It is important to note that this post does not touch on several critical elements of the data science process, such as data cleaning, centering, model tuning, and model evaluation. In applied machine learning, like with many other software projects, a number of other contextual factors can inform a final decision.

For example, this particular client planned to use a human-augmented approach which would automatically approve matches above a very high threshold, reject matches below a moderately low threshold, and refer the remaining records to a manual review process. When enough data has been manually reviewed, the pipeline will be analyzed all over again using a supervised approach with the newly available labeled data. For this reason, we were comfortable choosing a matching algorithm that was sufficiently discriminatory without further investigating confusion metrics and other parameters that you’d expect in a more comprehensive analysis.

Conclusion

Machine learning may carry the magical aura of a modern business buzzword, but following the powerful and proven fundamentals of data science can set almost any project on the right track. I hope this post has introduced you to some new tools and concepts for dealing with unlabeled data, and I encourage you to reach out to REDspace to see what unrealized opportunities your data might be hiding.