Why you should be using PHATE for dimensionality reduction | by Jamshaid Shahir | Towards Data Science

As data scientists, we often work with high-dimensional data with more than 3 features, or dimensions, of interest. In supervised machine learning, we may use this data for training and classification for example and may reduce the dimensions to speed up the training. In unsupervised learning, we use this type of data for visualization and clustering. In single-cell RNA sequencing (scRNA-seq), for example, we accumulate measurements of tens of thousands of genes per cell for upwards of a million cells. That’s a lot of data that provides a window into the cell’s identity, state, and other properties. More importantly, these properties put it in relation with the myriad of other cells in the dataset. Nevertheless, this creates a massive matrix of 1,000,000 cells and 10,000 genes, with each gene representing a dimension or axis. How do we interpret such data? As humans in a 3-D world, we cannot see anything beyond the three physical dimensions and need a way to capture the essence of the datasets like these while not losing anything of value.
How can we compress this dataset to 2 or 3 dimensions such that the essential information is retained? Your first instinct may be to use PCA or tSNE to project the dataset into a 2-D embedding. However, there are key tradeoffs with each of these methods that may lead to erroneous conclusions about the dataset in question during downstream analysis. With PCA, you lose local relationships between data-points (or cells in our case) at the cost of capturing a global representation. Furthermore, PCA is a linear method, which does not accurately capture complex datasets such as single-cell RNA sequencing data, where we can have a myriad of cell types defined by distinct gene expression portfolios while also undergoing various processes such as cell division, differentiation (specialization of stem cells into more mature cell types like neurons), metabolism, and so forth. tSNE, on the other hand, is a non-linear method that does a better job preserving local relationships but at the cost of shattering the global picture. For example, in stem cell differentiation, stem cells do not automatically turn into, say, a neuron, at the flick of switch (or set of genes in this case), but rather go through a continuum of changes defined by gradual changes in its transcriptomic profile. We often describe this continuum as a trajectory. The problem with tSNE is that it shatters these trajectories, resulting in disjointed clusters of cells, with little information on how one cell type is related to another.
So now that I’ve illustrated the downstream issues with this tradeoff, how can we resolve it? That is where PHATE enters the picture. PHATE — which stands for Potential of Heat diffusion for Affinity-based Transition Embedding — is a newcomer onto the world of dimensionality reduction. Like tSNE, it is a non-linear, unsupervised technique. However, unlike tSNE, which preserves local structure of higher-dimensional data often at the cost of the global structure, PHATE captures the best of both worlds of PCA and tSNE, preserving both local and global relationships between data-points to accurately reflect the high-dimensional dataset in question.
We will continue to use scRNA-seq data as an example. With this dataset, we essentially have an m-by-n matrix of m cells (rows) and n genes (columns), which represent discrete counts of messenger RNA molecules. We first compute a square matrix of the Euclidean distance between each of these cells, which is simply the length of the line segment between two points based on the Cartesian coordinates. In the context of scRNA-seq, these Cartesian coordinates are the gene expression measurements, so intuitively, we would expect cells a short distance apart to be very similar in gene expression, and hence similar cell types, whereas cells further apart to have very different gene expression patterns, and hence reflect starkly different cell types (e.g., a neuron and a red blood cell). However, this metric doesn’t always lend well to that interpretation. This is because of the Curse of Dimensionality, where too many dimensions can cause the data-points in your dataset to appear equidistant from all the others, makes it hard to draw conclusions of trends in your data, derive meaningful clusters and local neighborhoods, and determine other types of patterns. To address this problem, we convert our distances to affinities, which quantify local similarities between observations in our data. (This is where the “A” in PHATE for “Affinity-based” comes from!) These affinities are inversely proportional to the distances such that the further apart two observations are in Euclidean space, the smaller their affinity; likewise, the closer they are, the greater their affinity. Affinities are commonly calculated by using a kernel function to transform your Euclidean distances. In very simple terms, this is your probability mass or density function minus the factors/coefficients that normalize it to ensure that probabilities are between 0 and 1. They are used often in other Machine Learning problems such as support vector machines. One popular kernel you can guess is the Gaussian kernel:
Where x and y are coordinates in a high-dimensional space X, and ε is a bandwidth metric that measures the “spread” or radius of neighborhoods captured by this kernel. The authors of the PHATE paper use a slightly more advanced kernel function that does a better job of quantifying similarities, whose details I will spare for brevity. While this is a handy trick to preserve the local structure of our dataset, just embedding these alone will fracture the global structure as in the case of tSNE. Hence, in addition to retaining the local structure, PHATE’s other objective it to maintain the global relationships across the data. To achieve this, PHATE uses the affinities to “diffuse” through the data via a Markov random-walk. When we say diffuse, we mean the net spread from a region of high concentration to that of low concentration. In the context of affinities, this can be thought of as going from high affinities (i.e., a cluster of cells in our high-dimensional dataset) to lower affinities (i.e., more spread-out cells). More intuitively, we can think of this is the spread of heat in a room going from a warm source (e.g., a fireplace) to a less warm area (e.g., you on your couch), which can be modeled mathematically as the heat equation and whose solution is the heat kernel. This is where the “H” in PHATE, Heat diffusion, comes from. As for a random walk, this represents a trail of successive random steps through our high-dimensional space (i.e., transitioning from cell to cell), where each possible step or transition has a defined probability of going down said route. You can think of this as the heat in room random spreading in the direction of one corner of the room initially then switching directions. This is known more broadly as a stochastic or random process. In our context, we have the probability of cell i going to cell j depends on the last cell we visited. Now with all that terminology out of the way, let’s see how this works in action!
We first calculate initial probabilities of our random walk by normalizing our previously calculated affinities. This produces the following:
where
This gives us an N-by-N transition probability matrix of moving from cell x to cell y within a single time step.
A simpler term for this matrix, used in the PHATE paper, is the diffusion operator. Mathematically, to attain effective diffusion, we raise the diffusion operator to an optimal number of steps t to learn the global structure of our data. This gives us the probability of transitioning from cell x to cell y in t time steps. With larger t, we can cover more distance in the high-dimensional space and learn more about the global structure without getting bogged down by the locality from our single-time step affinity-based probabilities. You can think of this as scouting a hiking trail to build a map of the surroundings: every few steps, you lay down a marker to note your previous position, generally at notable areas (e.g., a large tree, a river bank, etc.), rather than setting markers with every step. This lets you build a general map of the area without getting bogged down by every single twig and branch. For sake of brevity, I’ll spare the technical details of how to calculate this optimal step t, which is described in the PHATE paper, using von Neumann entropy, but this all feeds into the “T” of our algorithm for “Transition”.
Alright, we’ve got our affinities to capture local relationships between nearby cells in one hand and a powered diffusion operator to learn our global space in another. Now what? Embed this powered diffusion operator? Not so fast. One limitation is that the resulting probability-based distances between cells via this operator (or diffusion distances, as the authors define them) are not very sensitive to distances between far away points and can suffer stability issues when we consider boundary points of our high-dimensional space (the paper goes into more detail on these shortcomings). This can be resolved however via the first letter of our acronym, “P”, which stands for Potential. The authors define this clever metric called potential distance, inspired by information theory and stochastic dynamics, where we measure the distance between log-transformed probabilities from the powered diffusion operator. This increases the sensitivity of our resulting distances, and it enables PHATE to preserve and local and global architecture for visualization purposes. Mathematically this is defined as follows:
where
pxt refers to row x of our diffusion operator, which you’ll recall is our transition probability matrix raised to the t’th power.
When we say that this distance metric is more sensitive, suppose the transition probability from cell a to cell b is 0.04, while for cell a to c it’s 0.05. Under these diffusion distances, they are not very sensitive to fold-changes. The distance, 0.01, may suffer sensitivity issues in trying to encapsulate this relationship in a lower dimension projection. However, if we log transform these probabilities and take their distance, we obtain a larger distance of 0.223, which is the same as if these probabilities had been 0.4 and 0.5 respectively (recall that log a - log b = log (a/b)). Pretty neat!
Alright, we’re ready for the “E” part of PHATE, “embedding”! Typically, with these diffusion metrics, we commonly perform an eigen-decomposition (i.e., break it up by eigenvectors of the matrix in question, in this case our powered diffusion operator) to derive a diffusion map of the data, which were a popular approach for studying differentiation trajectories in scRNA-seq data. The problem with this approach, however, is that it shatters trajectories into a myriad of eigenvectors reflecting the diffusion components. This high intrinsic dimensionality consequently renders them unamenable for visualization. To bypass this limitation, the authors embed the potential distance matrix using metric Multidimensional Scaling (metric MDS). This is an embedding method tailored for distance matrices as input by minimizing what’s known as a “stress” function:
While this equation may look intimidating, what it’s essentially doing is measuring the goodness of fit, in this case, how well the embedded coordinates fit the higher-dimensional data we seek to visualize. The smaller the stress, the greater fit. Thus, if the stress of these embedded points is zero, then the data has been successfully captured in this MDS embedding. Small, non-zero values may occur as a result of noise or a small number of MDS dimensions (i.e., m = 2 or 3). However, this level of distortion is generally tolerable so long as trajectories and other key features are preserved.
Everything I’ve discussed up until this point can be visually summarized in this figure from the authors of the PHATE paper:
If you’ve stuck around this long, congratulations! We’re now ready to learn how to implement PHATE in Python and see it in action! For simplicity, we’ll use the smaller version of the popular MNIST dataset where we have 8-by-8 images of hand-drawn digits (you can find an example from the authors of the paper applying PHATE to real scRNA-seq data here).
Install the PHATE library using pip or pip3:
pip install phate
Create a new Python file in your favorite editor or a notebook in Jupyter notebooks and run the following:
Let’s run this program to see the results
python3 mnist_phate.py
Looking at the PHATE embedding, you can see that we have generally clear separation of the digits, while the clusters themselves have unique shapes and distributions. Let’s see how the this compares to PCA and tSNE
PCA (left) overcrowds the data, making it difficult to draw concrete conclusions. tSNE (center) on the other hand does a nice job of separating them out into balled-up clusters, at the cost of a clear global structure of how the digits relate to another, as well as suppressing the unique spread of data within clusters. PHATE (right) reconciles these issues where you can clearly make out the clusters and get a sense of how they relate to one another (e.g., 3 and 9 having a lot of similarity). For the case of digits, this isn’t such a big deal, but for single-cell biology when we’re studying continuous processes such as turning stem cells into neurons, having that added detail of how cells transition from one cell type to another is very useful.
In this article, we learned about the PHATE algorithm, how it works, and how to implement it in Python. As you can see, it does a good job of preserving the local and global aspects of data. For more information on the math behind the PHATE algorithm, as well as more examples and comparisons to other methods like tSNE and UMAP, I encourage you to check out the original publication as well as some tutorials on PHATE and single-cell data analysis from the authors.
I hope you enjoyed this article. Thanks for reading and happy coding!
References:
[1] K. Moon, D. van Dijk, Z. Wang, S. Gigante, D. Burkhart, W. Chen, K. Yim, A. van den Elzen, M.J. Hirn, R.R. Coifman, N.B. Ivanova, G. Wolf, and S. Krishnaswamy, Visualizing structure and transitions in high-dimensional biological data (2019), Nature Biotechnology