Embeeding and clustering combining Knime and Python | Towards Data Science

Knime is a free and open-source data analytics, reporting and integration platform. KNIME integrates various components for machine learning and data mining through its modular data pipelining concept. For people like me, who do not have a strong coding background, Knime has been the open door to end to end data science experience. For the last two years working with Knime, I have been able to understand the global picture of data science, I left my fears behind and now I am stepping deeper and deeper in the challenging and amazing world of data science.
A few days ago I read an article in TowardsDataScience by Rosaria Silipo and Mischa Lisovyi showing a very nice Knime workflow involving t-SNE applied on MNIST dataset. I had done a similar workflow in the past with UMAP with nice results.
Encouraged by Rosaria Silipo in Linkedin I decided to publish my first story.
The raw material for this article is the MNIST dataset from OpenML of handwritten numbers from 0 to 9 by Yan LeCun. The dataset has 70000 handwritten numbers of 28x28 pixels of 0–255 grey scale levels. That is to say that we have 70000 vectors wich dimension is 784.
That data is fetched in a Python script within Knime and the UMAP algorithm for dimensional reduction is applied, also on that Python script, so we can map each image of the data set in a point on a lower dimensional space. For the aim of representing the results, logically, I used 2D and 3D as target dimensions.
After that, in order to have more fun, with the resulting UMAP embedded data ignoring lables, I applied a DBSCAN node within Knime for clustering results. We will check results later.
Uniform Manifold Approximation and Projection (UMAP) is a dimension reduction technique that can be used for visualisation similarly to t-SNE, but also for general non-linear dimension reduction. The algorithm is founded on three assumptions about the data
-The data is uniformly distributed on Riemannian manifold;
-The Riemannian metric is locally constant (or can be approximated as such);
-The manifold is locally connected.
What is a Riemannian manifold? Wait!! What is a manifold? A manifold is a topological object of n-dimension that locally, on the neighborhood of a point, behaves like an n-dimensional euclidean space. A good layman’s example would be the earth. A Rieamannian manifold is a smooth differentiable manifold in which notions like length, curves, angles, volumes, gradient of functions can be defined. For more information on manifolds or Riemannian manifolds, try here and here.
UMAP has some parameters to control its performance.
n_neighbors: The bigger the n_neighbors the bigger neighborhoods that will be sought by the algorithm and will be tending to global overall structures.The lower the n_neighbors, the more detailed structures that UMAP will be focused on.
minimum distance: Defines the minimum distance that the algorithm will consider to pack points together in the final reducted space. That means that low values of minimum distance will result in smaller cluster embeddings. On the other side, high values of minimum distance will lead UMAP algorithm to point to global structures.
metric: We can define the metric with which the algorithm will calculate the distance in the original space.
Hard to digest! I know. Nearly magic for engineers. Coming out of that darkness, specifically (and simply!!), UMAP models each high-dimensional object by a n-dimensional point (n being down to 2) in such a way that similar objects are modeled by nearby points and dissimilar objects are modeled by distant points with high probability.
DBSCAN, Density-based spatial clustering of applications with noise, is a clustering algorithm considered among unsupervised lerning methods. Based on three input parameters the algorithm is able to clusterized a set of data by grouping together points that are closely packed together (points with a certain number of nearby neighbors), marking as noise those points that lie alone and away in low-density regions (whose nearest neighbors are too far away).
As you can imagine, the parameters ruling the performance of the DBSCAN algorithm are related to the distance to decide whether a point is near or not to one another, epsilon, and are also related to the minimum number of nearby points, minimum points, necessary to considered one point is a “neighborhood” or core point. A cluster is the sum of all core points and their neighbors. Finally, in KNIME node for DBSCAN included in the Knime extension KNIME Distance Matrix, we can see that a metric input is needed so the algorithm has the necessary Distance model to calculate de distances between points on the dataset.
So summarizing:
epsilon: Specifies distance between points so they can be be considered a part of a cluster. Two points will be considered neighbors belonging to the same cluster if the distance between them is lower than epsilon.
minimum Points: Minimum number of points reachable from a point p so that p point can be considered a core point.
distance model: As said before, it is necessary to inform DBSCAN node with a distance model to evaluate distances between points in data set. To do that we use the Mumeric Distance node.
Significative advantages of of DBSCAN algorithm are:
DBSCAN does not need the user to specify the number of clusters.
DBSCAN manages noise concept and discards and marks noise points
Both Knime workflows are published and available in Knime Hub:
UMAP DBSCAN MNIST 2D Python. UMAP 2D dimension embedding
UMAP DBSCAN MNIST 3D Python. UMAP 3D dimension embedding
I will only focuse on 2D explanation since both workflows are virtually identical. However, I will show comparative results for both in the end.
Following you can see the whole workflow and later I will explain it block by block.
Following you can find a detail picture of the first block with which data is fetched and UMAP is calculated within the Python Source node from Knime Python extension.
Orange block in the center is the Python Source node that includes the script used to fetch data from OpenML and to perform UMAP technique. Two variable nodes, “Metric” and “Number of rows” are merged and passed like inputs for the UMAP algorithm. For the final results I used euclidean metrics and regarding the “Number of rows” Integer input node, I just used it to change to low numbers while testing. Just a matter of time. On the right, I use the Column Rename node to name columns with understandable names during the workflow.
Here you can find the code within the Python Source node:
from pandas import DataFramefrom sklearn.datasets import fetch_openmlimport numpy as npimport matplotlib.pyplot as pltimport seaborn as sns# Dimension reduction and clustering librariesimport umapin_batch = flow_variables[‘input_batch’]umap_metric = flow_variables[‘metric’]#Varias metricas posibles # mnist_784 # Fashion-MNISTmnist = fetch_openml(‘mnist_784’, version=1, as_frame=False)mnist.target = mnist.target.astype(int)standard_embedding = umap.UMAP( n_neighbors=30, min_dist=0.0, n_components=2, metric=umap_metric, random_state=42 ).fit_transform(mnist.data[:in_batch, 1:])#Añadimos los targetarray2D_1=standard_embeddingarray2D_2=mnist.target[:in_batch]array2D_2=np.reshape(array2D_2,(in_batch,1))array_to_save=np.concatenate((array2D_1,array2D_2),axis=1)# Output table for the nodeoutput_table = DataFrame(array_to_save)
After the UMAP algorithm is performed, we have a 2D mapping for the MNIST digits. I used two different approaches to plotting just to compare results.
UMAP output Scatter Plot (Plotly) combined with Color Manager
Scatter Plot from Plotly Knime extension is a very nice node. Result is interactive and output colors can be configured with the Color Manager node so dataset point within the reducted 2D space can be labelled according to the digit they were within the original space before being embedded.
Alternatively, in 3D, instead of Scatter Plot node we can use Scatter 3D and the result is the following.
Please note that the same ten clearly differentiate clusters have been detected and mapped both in 2D and in 3D by the UMAP algorithm.
UMAP output with Python View
Using a Python Source node and a piece of code, similar results can also be found.
Following you can find de pieces of code to do both plots within the Python view node of Knime.
For 2D
from io import BytesIOimport matplotlib as mpltimport matplotlib.pyplot as pltimport numpy as nplabel_array=input_table['Col2']colors = ['red', 'blue', 'green', 'yellow', 'orange', 'peru', 'lime', 'pink', 'steelblue', 'olive']plt.scatter(input_table['Col0'], input_table['Col1'],   c=label_array, alpha=0.5, marker='o',   cmap=mplt.colors.ListedColormap(colors),   s=0.1,            label="Label")plt.xlabel("X")plt.ylabel("Y")plt.legend(loc='best')buffer=BytesIO()plt.savefig(buffer,format="svg")output_image = buffer.getvalue()
For 3D:
from io import BytesIOimport matplotlib as mpltimport matplotlib.pyplot as pltimport numpy as npfrom mpl_toolkits.mplot3d import Axes3Dlabel=input_table['Col3']colors = ['red', 'blue', 'green', 'yellow', 'orange', 'peru', 'lime', 'pink', 'steelblue', 'olive']fig = plt.figure(figsize=(12,12))ax = fig.add_subplot(111, projection='3d')ax.scatter(input_table['Col0'],input_table['Col1'] ,input_table['Col2'],           c=input_table['Col3'],           cmap=mplt.colors.ListedColormap(colors),           marker='o',           s=1)ax.set_xlabel('X Label')ax.set_ylabel('Y Label')ax.set_zlabel('Z Label')buffer=BytesIO()plt.savefig(buffer,format="svg")output_image = buffer.getvalue()
So, up to this point we have a set of 2D (or 3D) points representing a set of 700000 written images. We know which original number corresponds to each point of the reducted space since original labels have been added to the UMAP output as you can see here:
But imagine we had not original label. We would have a mass of points with no further information like this:
It is time to use DBSCAN and try to find what is the structure lying beneath that scatter nonsense plot.
We drive the output of the UMAP to the DBSCAN node. Additionally we feed the rest of the DBSCAN inputs with variables although they coud be configured inside the node.
For 2D DBSCAN I used Epsilon=0.15 and Minimum points=50. Metric, within the Numeric Distances node, was configured to Euclidean.
The output, after some string manipulation, is a table with clustered data.
What will we find if we plot points within the reducted space in 2D or 3D labelled with the clustering resulting from DBSCAN?
Let’s see.
In the following images you can see that DBSCAN identified clusters are the same clusters that UMAP delivered according to original images. That is pretty satisfying and nice.
Are results of DBSCAN good enough related to original labels?
Accuracy of DBSCAN clustering related to original labels is 95,9%.
All this work has been done within Knime combining native Knime nodes and embedding Python code on its workflows showing, hence, the capabilities of Knime environement.
Next story I am thinking about, will be on how to to perform a similar analysis but reaching the embedded state with a neural autoencoder. Will keep you informed.