Entity embedding using t-SNE. The well known dimensionality reduction... | by Michael Larionov, PhD | Towards Data Science

We have been discussing some of the possible ways to embed categorical features: Kernel PCA and Spectral Encoding. The goal of such embedding is to map categorical features to vectors in a low dimensional space. The advantage is this mapping dramatically reduce overfitting, compared to 1-hot encoding. However we can lose information and make the learning more difficult if the embedding is chosen incorrectly. In order to increase the quality of embedding, we use the category similarity information (that we can set a priori or by computing the similarity of conditional probability distribution). Kernel PCA method is also using the probability distribution of the categorical variable, whereas the Spectral Encoding does not. For completeness, we will also use t-SNE method and we can discuss its advantages and disadvantages.
t-SNE stands for t-distributed stochastic neighbor embedding, and was originally proposed for visualization of high-dimensional data. As opposed to PCA, it is a non-linear method, and its goal is to preserve similarity between the data points. It introduces a non-linear transformation from the original data x to a low-dimensional vectors y, and it models the similarity between data points using Student t-Distribution:
The set of points y is found by minimizing KL divergence of the distribution Q from the original distribution P:
The minimization can be done using Gradient Descent.
To demonstrate how the method works we will use synthetic data set with only one categorical variable Day of Week, with 0 being Monday and 6 being Sunday:
array([ 23,  19,  15,  19,  22, 429, 473])
Here we deliberately added more data for Saturday and Sunday to see how t-SNE will take into account the probability distribution of the original data. To remind you, Laplacian method is not influenced by the the source data probability distribution, whereas PCA and kernel PCA are.
The Kernel function is the same as in the previous examples
array([[1. , 0.9, 0.8, 0.7, 0.5, 0.1, 0.3],       [0.9, 1. , 0.9, 0.8, 0.5, 0.1, 0.2],       [0.8, 0.9, 1. , 0.9, 0.5, 0.1, 0.2],       [0.7, 0.8, 0.9, 1. , 0.6, 0.1, 0.2],       [0.5, 0.5, 0.5, 0.6, 1. , 0.7, 0.5],       [0.1, 0.1, 0.1, 0.1, 0.7, 1. , 0.8],       [0.3, 0.2, 0.2, 0.2, 0.5, 0.8, 1. ]])
Scikit-learn implementation of t-SNE actually wants me to provide metric function instead, so we transform our kernel matrix back to distance matrix
array([[-0.        ,  0.21072103,  0.4462871 ,  0.71334989,  1.38629436,  4.60517019,  2.40794561],       [ 0.21072103, -0.        ,  0.21072103,  0.4462871 ,  1.38629436,  4.60517019,  3.21887582],       [ 0.4462871 ,  0.21072103, -0.        ,  0.21072103,  1.38629436,  4.60517019,  3.21887582],       [ 0.71334989,  0.4462871 ,  0.21072103, -0.        ,  1.02165125,  4.60517019,  3.21887582],       [ 1.38629436,  1.38629436,  1.38629436,  1.02165125, -0.        ,  0.71334989,  1.38629436],       [ 4.60517019,  4.60517019,  4.60517019,  4.60517019,  0.71334989, -0.        ,  0.4462871 ],       [ 2.40794561,  3.21887582,  3.21887582,  3.21887582,  1.38629436,  0.4462871 , -0.        ]])
Sigma here is an important hyperparameter that needs to be tuned. Another important hyperparameter of the t-SNE method is perplexity score, but in our case it can be either left to default or set to sigma.
In this first example we used t-SNE without a kernel, meaning that the distance between all days of the week is 1. This is what you would get by just using 1-hot encoding. Note, that the embeddings seem to be spread randomly, and this is a difference from PCA, which even in the absence of the kernel uses the initial data probability distribution to embed the data in such a way, that would maximize their linear separability. Let’s see what results we get if we use kernel function:
Here results are more meaningful. Monday through Thursday are grouped together to preserve their similarity. Friday, Saturday and Sunday are far apart from each other and from the other days. It is interesting to see that Friday is closer to Sunday than Saturday.
t-SNE is one of the options for category embedding. Generally, t-SNE does not generalize (because it will not be able to map unknown data), but for categorical features it does not matter, because the new data will have one of the existing categories. A bigger problem is that it will not work without a kernel, as opposed to PCA which will give reasonable results even if kernel is not given. Just like Kernel PCA, it is using probability distribution to map the data, even though in a different way. If Kernel PCA is trying to achieve smoothness in the resulting vector space, t-SNE cares more about preserving similarity. This will probably guide your decision which method is more appropriate for your model.
You can find the code for this story in my github repository.