String Similarity Matching for Big Data using Distributed Cloud Computations | by Georgios Ntanakas | Towards Data Science

Our story — like most stories — starts with an ambitious undertaking! We are offered two large data-sets (coming from two tables in BigQuery). One consists of all the app names from the Google Play and iOS App Stores — about 14 million entries— which we shall call the “clean” app names. The other contains about 5K “dirty” app names. Dirtiness refers to missing words, misspelling of certain words, additional words referring to the OS or other info etc. Examples can be found by looking at the section “Putting everything together and running it”.
Along with the data-sets we are offered a blue and a red pill. We think we are brave enough, decide to take the red pill, accept the challenge and seek the brutal truth.
The request is to match entries from the “dirty” data-set to entries in the “clean” one and then store the results as a lookup table.
Before becoming a man and facing t̶h̶e̶ ̶r̶e̶a̶l̶ ̶w̶o̶r̶l̶d̶ production, a boy needs to develop his skills in the playground. A fancy one is Google Colaboratory, a Jupyter notebook hosted on the cloud, offering free, powerful computing resources (including GPUs/TPUs) and that’s where the first tests were run.
A computer scientist would classify our problem as an approximate string matching problem. Traditional solutions to these problems use different string metrics to come up with a distance between two given sequences of characters. Examples are the Jaro–Winkler distance and the Levenshtein distance.
Our first attempt was to employ fuzzy wuzzy, a Python library that implements the Levenshtein distance. (Please take the rough numbers that follow — rather than an appropriate bench-marking — with a pinch of salt. The final differences are so significant that perfect precision is of little importance.) Fuzzy wuzzy needed about 15 min to compute the distance between a single entry from the “dirty” data-set and all the entries from the “clean” data-set. Repeating the process 5K times translates to about 52 days, which is slightly more than we would like for a process that runs at least once daily. If time is money, we would be broke by now!
To avoid crippling loans and bankruptcy, a slightly different approach was employed: increasing performance through the power of numbers. The string data-sets are transformed into vectors based on tf–idf in order to use the cosine similarity. The core of the algorithm has been used before for similar problems here, here and here. A few modifications are introduced in this article to enable running the computations distributed on a cloud Dask client in order to further improve performance.
Why Dask?/What is Dask?
Dask is a library that provides ways to scale Pandas, Scikit-Learn, and Numpy workflows natively, with minimal rewriting. Thus, ordinary Python operations can be distributed/parallelized between different nodes/CPU cores. It also can (but doesn’t have to) run on a distributed cluster. I.e., Dask is doing nothing more than providing the infrastructure to split data, such as a Pandas DataFrame, into different nodes and then perform operations using multiple cores and ensuring communication between the split data.
Note: Below the clean and dirty name conventions are used for the baseline and the to-be-matched data-set respectively.
Here is a list of the basic libraries that were used for this project:
To get data from BigQuery, we are using the BigQuery Client Library for Python. The data is loaded into a Pandas DataFrame by the client, which is immediately transformed into a Dask DataFrame. The Dask DataFrame is nothing more than a composition of Pandas DataFrames distributed over a number of partitions (npartitions), depending on the CPU architecture.
Transforming the DataFrames from strings to vectors using tf-idf needs literally 3 lines of code:
The vectorizing function is initialized using the TfidfVectorizer function from Scikit-Learn and by defining the analyzer function (see info below). The clean DataFrame is used to define the feature vector (based on the analyzer) and compute the clean tf-idf matrix at the same time (fit_transform) and then the dirty tf-idf matrix is computed (transform).
The analyzer is the function that is used to split each app name (string) into smaller bits. Gathering all these smaller bits from all entries of the clean DataFrame defines the feature vector. If a specific app name contains this bit, then when transforming into a vector it will get a float value at the corresponding position — if not a 0. The float value depends on the number of times this bit appears within the app name and the value is modified so that the vector for this app name is normalized (vector norm equal to 1 unit). To give an example: If the analyzer is defined as below
and the clean DataFrame is simply the following,
then the feature vector has this form:
['ACE', 'AGR', 'BOO', 'CEB', 'EBO', 'FAC', 'GRA', 'INS', 'NST', 'OOK', 'RAM', 'STA', 'TAG']
and the clean DataFrame is transformed to the following vector matrix (the output matrix is given in sparse matrix (CSR) format but displayed as an array here)
[[0.40824829 0.         0.40824829 0.40824829 0.40824829 0.40824829  0.         0.         0.         0.40824829 0.         0.  0.        ] [0.         0.37796447 0.         0.         0.         0.  0.37796447 0.37796447 0.37796447 0.         0.37796447 0.37796447  0.37796447]]
As soon as clean and dirty data-sets are in vector mode, we can proceed with getting the cosine similarity scores matrix. Performing the dot product between the clean and dirty vectorized matrices is enough to give us the cosine since the vectors are normalized. I.e., the dot product coincides with the cosine (similarity).
To do so, the following code is used:
Here, the sparse_dot_topn package is used, which provides a fast way to perform a sparse matrix multiplication followed by top-n multiplication result selection using Cython. The cosine similarities will be given in a sparse matrix form with rows corresponding to the dirty data-set and columns to the clean one.
Using this similarity matrix, we can extract the entries matched between clean and dirty and their similarity score using:
Last but not least, we need to upload the result of our matching process back to BiqQuery as a lookup table. The code to do so looks like this:
Would you prefer a picture rather than a thousand words of the algorithm? Here is a simplified diagram of it:
Section written by Gaiar Baimuratov
Dask is such a versatile tool that can benefit you even by running it locally using its drop-in Pandas API support. However, it shows it’s full power when it runs on a distributed system — aka cluster. In collaboration with the DevOps engineers, we got access to a managed Kubernetes (k8s) cluster deployed on Google Cloud Platform (GCP). In most cases, after getting access to the cluster, Dask can easily be deployed on k8s using the helm package manager by running the command:
helm install —name my-dask stable/dask
After a while, you’ll get 5 pods deployed:
the scheduler — main Dask task manager
3 workers — to process data and apply operations
the Jupyter pod —a pre-setup Jupiter Lab pod as a nice way to interact with Dask
To get the list of the pods, once deployed, we’ve created a simple shell script. Run it and it will show the list of IP addresses for all freshly created pods:
In case you don’t need it anymore or you broke something and want a fresh start you can run:
helm delete dask —purge
Does it look too easy to be true? You’re right; universal solutions rarely work.
Here comes a bit of customization to make it work for our needs.
After the initial test, we’ve discovered a couple of issues when using Dask helm as-is.
It had several bugs. Luckily all fixes existed as pending pull requests (PRs) in the official repo but were not merged.Default images were missing for some of the packages we needed.One package didn’t have a binary version in pip and was failing at pip install ... part.Default pod configuration wasn’t beefy enough to process our large DataFrames.
It had several bugs. Luckily all fixes existed as pending pull requests (PRs) in the official repo but were not merged.
Default images were missing for some of the packages we needed.
One package didn’t have a binary version in pip and was failing at pip install ... part.
Default pod configuration wasn’t beefy enough to process our large DataFrames.
Moving on to the solutions:
1. Bugs in helm configurationFortunately, the bugs were fixed already in the official repo via pending PRs. To apply them, we had to make a local clone of the repo and merge all the PRs with the solutions as patches. That is certainly not the ideal solution but it is a quick way to make it work.Another problem was that the Jupyter Lab version defined in the repo had some UI troubles. To overcome them, it had to be upgraded by specifying the package version. The way to do it is included in the solution of the 2nd issue.
2. Missing packagesAs our data pipeline is based on GCP, we needed to include the SDK packages and some other libraries. Fortunately, Dask helm chart supports the usage of a YAML config file. Thus, it can be specified which pip or conda packages to be included, how much RAM to allocate to workers and much more than that.In our case, we’ve added the following:
Most of the packages will be installed via conda using the channels conda-forge and gaiar(see below why this channel is needed too). The rest is installed via pip.
3. Missing binary package versionFor speeding up the sparse matrix multiplication, we are using sparse_dot_topn, as mentioned in the algorithm section, which relies on Cython. Packages like this are usually supplied in two ways: either as binary, pre-compiled packages or when library compiles some of it is parts on installation host. In our case, we were missing the binary distribution and this meant that we should add compilation tools in our already “not-so-light” image. Instead, we chose another way. I’ve already had experience in building and maintaining conda packages in my own channel. A significant advantage of conda is that it ships packages in a binary way and that’s exactly what we needed.Defining the YAML file with conda recipe based on the pip distribution is relatively easy:
With that recipe, you run the conda build command and upload the package to the channel you have access to. Then, the package can be installed with conda -c CHANNEL_NAME PACKAGE_NAME or in our case conda -c gaiar sparse_dot_topn. I prepared two versions of it — one for Linux and another for MacOS (as it also had some troubles with the installation).
4. Limited resources for podsThis part is case specific and depends on what kind of tasks Dask is running and which are the constraints. In our case, we needed more RAM and wanted to define also the upper limit that every pod can consume. This is done again via a config YAML file:
This translates to: the lower limit for every worker is 3G, but it can request up to 6G if it needs to.
Here is a simple demo of Dask working on k8s on GCP:
Combining the algorithm and the deployment mentioned above, our code performs the matching process in about 20–30 mins; quite a significant speed up compared to the original 52 days. 😆
Here is a small part from the output table of one of our runs:
This table and the execution time needed to obtain it is an indication that the red pill might have been the wiser decision this time.
Let’s try to give it in one sentence: Tf-idf vectorization and cosine similarity combined with efficient sparse matrix computations and parallel execution of operations using the distribution capabilities of Dask, deployed in Kubernetes proved to be a very efficient approach for handling string similarity problems of large scale.
Special thanks to Gaiar Baimuratov for making the deployment happen and writing the corresponding section.