SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Hugues Van Assel
ENS de Lyon, CNRS
UMPA UMR 5669
hugues.van_assel@ens-lyon.fr
Titouan Vayer
Univ. Lyon, ENS de Lyon, UCBL, CNRS, Inria
LIP UMR 5668
titouan.vayer@inria.fr
École polytechnique, IP Paris, CNRS
CMAP UMR 7641
Université Bretagne Sud, CNRS
IRISA UMR 6074
nicolas.courty@irisa.fr
Many approaches in machine learning rely on a weighted graph to encode the similarities between samples in a dataset. Entropic affinities (EAs), which are notably used in the popular Dimensionality Reduction (DR) algorithm t-SNE, are particular instances of such graphs. To ensure robustness to heterogeneous sampling densities, EAs assign a kernel bandwidth parameter to every sample in such a way that the entropy of each row in the affinity matrix is kept constant at a specific value, whose exponential is known as perplexity. EAs are inherently asymmetric and row-wise stochastic, but they are used in DR approaches after undergoing heuristic symmetrization methods that violate both the row-wise constant entropy and stochasticity properties. In this work, we uncover a novel characterization of EA as an optimal transport problem, allowing a natural symmetrization that can be computed efficiently using dual ascent. The corresponding novel affinity matrix derives advantages from symmetric doubly stochastic normalization in terms of clustering performance, while also effectively controlling the entropy of each row thus making it particularly robust to varying noise levels. Following, we present a new DR algorithm, SNEkhorn, that leverages this new affinity matrix. We show its clear superiority to existing approaches with several indicators on both synthetic and real-world datasets.
Exploring and analyzing high-dimensional data is a core problem of data science that requires building low-dimensional and interpretable representations of the data through dimensionality reduction (DR). Ideally, these representations should preserve the data structure by mimicking, in the reduced representation space (called latent space), a notion of similarity between samples. We call affinity the weight matrix of a graph that encodes this similarity. It has positive entries and the higher the weight in position  $(i,j)$ , the higher the similarity or proximity between samples  $i$  and  $j$ . Seminal approaches relying on affinities include Laplacian eigenmaps [2], spectral clustering [50] and semi-supervised learning [55]. Numerous methods can be employed to construct such affinities. A common choice is to use a kernel (e.g., Gaussian) derived from a distance matrix normalized by a bandwidth parameter that usually has a large influence on the outcome of the algorithm. Indeed, excessively small kernel
bandwidth can result in solely capturing the positions of closest neighbors, at the expense of large-scale dependencies. Inversely, setting too large a bandwidth blurs information about close-range pairwise relations. Ideally, one should select a different bandwidth for each point to accommodate varying sampling densities and noise levels. One approach is to compute the bandwidth of a point based on the distance from its  $k$ -th nearest neighbor [53]. However, this method fails to consider the entire distribution of distances. In general, selecting appropriate kernel bandwidths can be a laborious task, and many practitioners resort to greedy search methods. This can be limiting in some settings, particularly when dealing with large sample sizes.
Entropic Affinities and SNE/t-SNE. Entropic affinities (EAs) were first introduced in the seminal paper Stochastic Neighbor Embedding (SNE) [16]. It consists in normalizing each row  $i$  of a distance matrix by a bandwidth parameter  $\varepsilon_{i}$  such that the distribution associated with each row of the corresponding stochastic (i.e., row-normalized) Gaussian affinity has a fixed entropy. The value of this entropy, whose exponential is called the perplexity, is then the only hyperparameter left to tune and has an intuitive interpretation as the number of effective neighbors of each point [49]. EAs are notoriously used to encode pairwise relations in a high-dimensional space for the DR algorithm t-SNE [46], among other DR methods including [6]. t-SNE is increasingly popular in many applied fields [20, 32] mostly due to its ability to represent clusters in the data [27, 5]. Nonetheless, one major flaw of EAs is that they are inherently directed and often require post-processing symmetrization.
Doubly Stochastic Affinities. Doubly stochastic (DS) affinities are non-negative matrices whose rows and columns have unit  $\ell_1$  norm. In many applications, it has been demonstrated that DS affinity normalization (i.e., determining the nearest DS matrix to a given affinity matrix) offers numerous benefits. First, it can be seen as a relaxation of k-means [51] and it is well-established that it enhances spectral clustering performances [10, 52, 1]. Additionally, DS matrices present the benefit of being invariant to the various Laplacian normalizations [50]. Recent observations indicate that the DS projection of the Gaussian kernel under the KL geometry is more resilient to heteroscedastic noise compared to its stochastic counterpart [23]. It also offers a more natural analog to the heat kernel [30]. These properties have led to a growing interest in DS affinities, with their use expanding to various applications such as smoothing filters [33], subspace clustering [25] and transformers [41].
Contributions. In this work, we study the missing link between EAs, which are easy to tune and adaptable to data with heterogeneous density, and DS affinities which have interesting properties in practical applications as aforementioned. Our main contributions are as follows. We uncover the convex optimization problem that underpins classical entropic affinities, exhibiting novel links with entropy-regularized Optimal Transport (OT) (Section 3.1). We then propose in Section 3.2 a principled symmetrization of entropic affinities. The latter enables controlling the entropy in each point, unlike t-SNE's post-processing symmetrization, and produces a genuinely doubly stochastic affinity. We show how to compute this new affinity efficiently using a dual ascent algorithm. In Section 4, we introduce SNEkhorn: a DR algorithm that couples this new symmetric entropic affinity with a doubly stochastic kernel in the low-dimensional embedding space, without sphere concentration issue [29]. We finally showcase the benefits of symmetric entropic affinities on a variety of applications in Section 5 including spectral clustering and DR experiments on datasets ranging from images to genomics data.
Notations.  $\llbracket n\rrbracket$  denotes the set  $\{1,\dots,n\}$ . exp and log applied to vectors/matrices are taken elementwise.  $\mathbf{1} = (1,\dots,1)^{\top}$  is the vector of 1.  $\langle \cdot ,\cdot \rangle$  is the standard inner product for matrices/vectors.  $\mathcal{S}$  is the space of  $n\times n$  symmetric matrices.  $\mathbf{P}_{i}$ : denotes the  $i$ -th row of a matrix  $\mathbf{P}$ .  $\odot$  (resp.  $\oslash$ ) stands for element-wise multiplication (resp. division) between vectors/matrices. For  $\alpha, \beta \in \mathbb{R}^n$ ,  $\alpha \oplus \beta \in \mathbb{R}^{n\times n}$  is  $(\alpha_{i} + \beta_{j})_{ij}$ . The entropy of  $\mathbf{p}\in \mathbb{R}_+^n$  is  ${}^{1}\mathrm{H}(\mathbf{p}) = -\sum_{i}p_{i}(\log (p_{i}) - 1) = -\langle \mathbf{p},\log \mathbf{p} - \mathbf{1}\rangle$ . The Kullback-Leibler divergence between two matrices  $\mathbf{P},\mathbf{Q}$  with nonnegative entries such that  $Q_{ij} = 0\Rightarrow P_{ij} = 0$  is  $\mathrm{KL}(\mathbf{P}|\mathbf{Q}) = \sum_{ij}P_{ij}\left(\log (\frac{P_{ij}}{Q_{ij}}) - 1\right) = \langle \mathbf{P},\log (\mathbf{P}\oslash \mathbf{Q}) - \mathbf{1}\mathbf{1}^{\top}\rangle$ .
2 Entropic Affinities, Dimensionality Reduction and Optimal Transport
Given a dataset  $\mathbf{X} \in \mathbb{R}^{n \times p}$  of  $n$  samples in dimension  $p$ , most DR algorithms compute a representation of  $\mathbf{X}$  in a lower-dimensional latent space  $\mathbf{Z} \in \mathbb{R}^{n \times q}$  with  $q \ll p$  that faithfully captures and represents pairwise dependencies between the samples (or rows) in  $\mathbf{X}$ . This is generally achieved
<sup>1</sup>With the convention  $0\log 0 = 0$
by optimizing  $\mathbf{Z}$  such that the corresponding affinity matrix matches another affinity matrix defined from  $\mathbf{X}$ . These affinities are constructed from a matrix  $\mathbf{C} \in \mathbb{R}^{n \times n}$  that encodes a notion of "distance" between the samples, e.g., the squared Euclidean distance  $C_{ij} = \|\mathbf{X}_{i:} - \mathbf{X}_{j:}\|_2^2$  or more generally any cost matrix  $\mathbf{C} \in \mathcal{D} := \{\mathbf{C} \in \mathbb{R}_+^{n \times n} : \mathbf{C} = \mathbf{C}^\top$  and  $C_{ij} = 0 \iff i = j\}$ . A commonly used option is the Gaussian affinity that is obtained by performing row-wise normalization of the kernel  $\exp(-\mathbf{C} / \varepsilon)$ , where  $\varepsilon > 0$  is the bandwidth parameter.
Entropic Affinities (EAs). Another frequently used approach to generate affinities from  $\mathbf{C} \in \mathcal{D}$  is to employ entropic affinities [16]. The main idea is to consider adaptive kernel bandwidths  $(\varepsilon_i^\star)_{i \in [[n]]}$  to capture finer structures in the data compared to constant bandwidths [47]. Indeed, EAs rescale distances to account for the varying density across regions of the dataset. Given  $\xi \in [[n - 1]]$ , the goal of EAs is to build a Gaussian Markov chain transition matrix  $\mathbf{P}^{\mathrm{e}}$  with prescribed entropy as
$$
\forall i, \forall j, P _ {i j} ^ {\mathrm {e}} = \frac {\exp \left(- C _ {i j} / \varepsilon_ {i} ^ {\star}\right)}{\sum_ {\ell} \exp \left(- C _ {i \ell} / \varepsilon_ {i} ^ {\star}\right)} \tag {EA}
$$
$$
\text {w i t h} \varepsilon_ {i} ^ {\star} \in \mathbb {R} _ {+} ^ {*} \text {s . t .} \mathrm {H} \left(\mathbf {P} _ {i:} ^ {\mathrm {e}}\right) = \log \xi + 1.
$$
The hyperparameter  $\xi$ , which is also known as perplexity, can be interpreted as the effective number of neighbors for each data point [49]. Indeed, a perplexity of  $\xi$  means that each row of  $\mathbf{P}^{\mathrm{e}}$  (which is a discrete probability since  $\mathbf{P}^{\mathrm{e}}$  is row-wise stochastic) has the same entropy as a uniform distribution over  $\xi$  neighbors. Therefore, it provides the practitioner with an interpretable parameter specifying which scale of dependencies the affinity matrix should faithfully capture. In practice, a root-finding algorithm is used to find the bandwidth parameters  $(\varepsilon_i^\star)_{i\in [[n]]}$  that satisfy the constraints [49]. Hereafter, with a slight abuse of language, we call  $e^{\mathrm{H}(\mathbf{P}_{i:}) - 1}$  the perplexity of the point  $i$ .
Dimension Reduction with SNE/t-SNE. One of the main applications of EAs is the DR algorithm SNE [16]. We denote by  $\mathbf{C}_{\mathbf{X}} = \left(\| \mathbf{X}_{i:} - \mathbf{X}_{j:}\|_{2}^{2}\right)_{ij}$  and  $\mathbf{C}_{\mathbf{Z}} = \left(\| \mathbf{Z}_{i:} - \mathbf{Z}_{j:}\|_{2}^{2}\right)_{ij}$  the cost matrices derived from the rows (i.e., the samples) of  $\mathbf{X}$  and  $\mathbf{Z}$  respectively. SNE focuses on minimizing in the latent coordinates  $\mathbf{Z} \in \mathbb{R}^{n \times q}$  the objective KL( $\mathbf{P}^{\mathrm{e}}|\mathbf{Q}_{\mathbf{Z}}$ ) where  $\mathbf{P}^{\mathrm{e}}$  solves (EA) with cost  $\mathbf{C}_{\mathbf{X}}$  and  $[\mathbf{Q}_{\mathbf{Z}}]_{ij} = \exp(-[\mathbf{C}_{\mathbf{Z}}]_{ij}) / (\sum_{\ell} \exp(-[\mathbf{C}_{\mathbf{Z}}]_{\ell \ell}))$ . In the seminal paper [46], a newer proposal for a symmetric version was presented, which has since replaced SNE in practical applications. Given a symmetric normalization for the similarities in latent space  $[\widetilde{\mathbf{Q}}_{\mathbf{Z}}]_{ij} = \exp(-[\mathbf{C}_{\mathbf{Z}}]_{ij}) / \sum_{\ell, t} \exp(-[\mathbf{C}_{\mathbf{Z}}]_{\ell t})$  it consists in solving
$$
\min  _ {\mathbf {Z} \in \mathbb {R} ^ {n \times q}} \mathrm {K L} (\overline {{\mathbf {P} ^ {\mathrm {e}}}} | \widetilde {\mathbf {Q}} _ {\mathbf {Z}}) \quad \text {w h e r e} \quad \overline {{\mathbf {P} ^ {\mathrm {e}}}} = \frac {1}{2} \left(\mathbf {P} ^ {\mathrm {e}} + \mathbf {P} ^ {\mathrm {e} \top}\right). \quad \text {(S y m m e t r i c - S N E)}
$$
In other words, the affinity matrix  $\overline{\mathbf{P}^{\mathrm{e}}}$  is the Euclidean projection of  $\mathbf{P}^{\mathrm{e}}$  on the space of symmetric matrices  $S$ :  $\overline{\mathbf{P}^{\mathrm{e}}} = \operatorname{Proj}_{S}^{\ell_2}(\mathbf{P}^{\mathrm{e}}) = \arg \min_{\mathbf{P} \in S} \| \mathbf{P} - \mathbf{P}^{\mathrm{e}} \|_2$  (see Appendix A.1). Instead of the Gaussian kernel, the popular extension t-SNE [46] considers a different distribution in the latent space  $[\widetilde{\mathbf{Q}}_{\mathbf{Z}}]_{ij} = (1 + [\mathbf{C}_{\mathbf{Z}}]_{ij})^{-1} / \sum_{\ell, t} (1 + [\mathbf{C}_{\mathbf{Z}}]_{\ell t})^{-1}$ . In this formulation,  $\widetilde{\mathbf{Q}}_{\mathbf{Z}}$  is a joint Student  $t$ -distribution that accounts for crowding effects: a relatively small distance in a high-dimensional space can be accurately represented by a significantly greater distance in the low-dimensional space.
Considering symmetric similarities is appealing since the proximity between two points is inherently symmetric. Nonetheless, the Euclidean projection in (Symmetric-SNE) does not preserve the construction of entropic affinities. In particular,  $\overline{\mathbf{P}^{\mathrm{e}}}$  is not stochastic in general and  $\mathrm{H}(\overline{\mathbf{P}_{i:}^{\mathrm{e}}})\neq (\log \xi +1)$  thus the entropy associated with each point is no longer controlled after symmetrization (see the bottom left plot of Figure 1). This is arguably one of the main drawbacks of the approach. By contrast, the  $\mathbf{P}^{\mathrm{se}}$  affinity that will be introduced in Section 3 can accurately set the entropy in each point to the desired value  $\log \xi +1$ . As shown in Figure 1 this leads to more faithful embeddings with better separation of the classes when combined with the t-SNEkhorn algorithm (Section 4).
Figure 1: Top: COIL [34] embeddings with silhouette scores produced by t-SNE and t-SNEkhorn (our method introduced in Section 4) for  $\xi = 30$ . Bottom:  $e^{\mathrm{H}(\mathbf{P}_{i:}) - 1}$  (perplexity) for each point  $i$ .
Symmetric Entropy-Constrained Optimal Transport. Entropy-regularized OT [37] and its connection to affinity matrices are crucial components in our solution. In the special case of uniform marginals, and for  $\nu > 0$ , entropic OT computes the minimum of  $\mathbf{P} \mapsto \langle \mathbf{P}, \mathbf{C} \rangle - \nu \sum_{i} \mathrm{H}(\mathbf{P}_{i:})$  over the space of doubly stochastic matrices  $\{\mathbf{P} \in \mathbb{R}_+^{n \times n} : \mathbf{P}\mathbf{1} = \mathbf{P}^\top \mathbf{1} = \mathbf{1}\}$ . The optimal solution is the unique doubly stochastic matrix  $\mathbf{P}^{\mathrm{ds}}$  of the form  $\mathbf{P}^{\mathrm{ds}} = \mathrm{diag}(\mathbf{u})\mathbf{K} \mathrm{diag}(\mathbf{v})$  where  $\mathbf{K} = \exp(-\mathbf{C} / \nu)$  is the Gibbs energy derived from  $\mathbf{C}$  and  $\mathbf{u}$ ,  $\mathbf{v}$  are positive vectors that can be found with the celebrated Sinkhorn-Knopp's algorithm [8, 42]. Interestingly, when the cost  $\mathbf{C}$  is symmetric (e.g.,  $\mathbf{C} \in \mathcal{D}$ ) we can take  $\mathbf{u} = \mathbf{v}$  [17, Section 5.2] so that the unique optimal solution is itself symmetric and writes
$$
\mathbf {P} ^ {\mathrm {d s}} = \exp \left(\left(\mathbf {f} \oplus \mathbf {f} - \mathbf {C}\right) / \nu\right) \text {w h e r e} \mathbf {f} \in \mathbb {R} ^ {n}. \tag {DS}
$$
In this case, by relying on convex duality as detailed in Appendix A.2, an equivalent formulation for the symmetric entropic OT problem is
$$
\min  _ {\mathbf {P} \in \mathbb {R} _ {+} ^ {n \times n}} \langle \mathbf {P}, \mathbf {C} \rangle \quad \text {s . t .} \quad \mathbf {P} \mathbf {1} = \mathbf {1}, \mathbf {P} = \mathbf {P} ^ {\top} \text {a n d} \sum_ {i} \mathrm {H} (\mathbf {P} _ {i:}) \geq \eta , \tag {EOT}
$$
where  $0 \leq \eta \leq n(\log n + 1)$  is a constraint on the global entropy  $\sum_{i} \mathrm{H}(\mathbf{P}_{i})$  of the OT plan  $\mathbf{P}$  which happens to be saturated at optimum (Appendix A.2). This constrained formulation of symmetric entropic OT will provide new insights into entropic affinities, as detailed in the next sections.
3 Symmetric Entropic Affinities
In this section, we present our first major contribution: symmetric entropic affinities. We begin by providing a new perspective on EAs through the introduction of an equivalent convex problem.
3.1 Entropic Affinities as Entropic Optimal Transport
We introduce the following set of matrices with row-wise stochasticity and entropy constraints:
$$
\mathcal {H} _ {\xi} := \left\{\mathbf {P} \in \mathbb {R} _ {+} ^ {n \times n} \text {s . t .} \mathbf {P 1} = \mathbf {1} \text {a n d} \forall i, \mathrm {H} \left(\mathbf {P} _ {i}\right) \geq \log \xi + 1 \right\}. \tag {1}
$$
This space is convex since  $\mathbf{p} \in \mathbb{R}_+^n \mapsto \mathrm{H}(\mathbf{p})$  is concave, thus its superlevel set is convex. In contrast to the entropic constraints utilized in standard entropic optimal transport which set a lower-bound on the global entropy, as demonstrated in the formulation (EOT),  $\mathcal{H}_{\xi}$  imposes a constraint on the entropy of each row of the matrix  $\mathbf{P}$ . Our first contribution is to prove that EAs can be computed by solving a specific problem involving  $\mathcal{H}_{\xi}$  (see Appendix A for the proof).
Proposition 1. Let  $\mathbf{C} \in \mathbb{R}^{n \times n}$  without constant rows. Then  $\mathbf{P}^{\mathrm{e}}$  solves the entropic affinity problem (EA) with cost  $\mathbf{C}$  if and only if  $\mathbf{P}^{\mathrm{e}}$  is the unique solution of the convex problem
$$
\min  _ {\mathbf {P} \in \mathcal {H} _ {\xi}} \langle \mathbf {P}, \mathbf {C} \rangle . \quad \text {(E A a s O T)}
$$
Interestingly, this result shows that EAs boil down to minimizing a transport objective with cost  $\mathbf{C}$  and row-wise entropy constraints  $\mathcal{H}_{\xi}$  where  $\xi$  is the desired perplexity. As such, (EA as OT) can be seen as a specific semi-relaxed OT problem [39, 14] (i.e., without the second constraint on the marginal  $\mathbf{P}^{\top}\mathbf{1} = \mathbf{1}$ ) but with entropic constraints on the rows of  $\mathbf{P}$ . We also show that the optimal solution  $\mathbf{P}^{\star}$  of (EA as OT) has saturated entropy i.e.,  $\forall i$ ,  $\mathrm{H}(\mathbf{P}_{i:}^{\star}) = \log \xi + 1$ . In other words, relaxing the equality constraint in (EA) as an inequality constraint in  $\mathbf{P} \in \mathcal{H}_{\xi}$  does not affect the solution while it allows reformulating entropic affinity as a convex optimization problem. To the best of our knowledge, this connection between OT and entropic affinities is novel and is an essential key to the method proposed in the next section.
Remark 2. The kernel bandwidth parameter  $\varepsilon$  from the original formulation of entropic affinities (EA) is the Lagrange dual variable associated with the entropy constraint in (EA as OT). Hence computing  $\varepsilon^{\star}$  in (EA) exactly corresponds to solving the dual problem of (EA as OT).
Remark 3. Let  $\mathbf{K}_{\sigma} = \exp (-\mathbf{C} / \sigma)$ . As shown in Appendix A.5, if  $\varepsilon^{\star}$  solves (EA) and  $\sigma \leq \min (\varepsilon^{\star})$ , then  $\mathbf{P}^{\mathrm{e}} = \mathrm{Proj}_{\mathcal{H}_{\xi}}^{\mathrm{KL}}(\mathbf{K}_{\sigma}) = \arg \min_{\mathbf{P}\in \mathcal{H}_{\xi}}\mathrm{KL}(\mathbf{P}|\mathbf{K}_{\sigma})$ . Therefore  $\mathbf{P}^{\mathrm{e}}$  can be seen as a KL Bregman projection [3] of a Gaussian kernel onto  $\mathcal{H}_{\xi}$ . Hence the input matrix in (Symmetric-SNE) is  $\overline{\mathbf{P}^{\mathrm{e}}} = \mathrm{Proj}_S^{\ell_2}(\mathrm{Proj}_{\mathcal{H}_{\xi}}^{\mathrm{KL}}(\mathbf{K}_{\sigma}))$  which corresponds to a surprising mixture of KL and orthogonal projections.
Figure 2: Samples from a mixture of three Gaussians with varying standard deviations. The edges' strength is proportional to the weights in the affinities  $\mathbf{P}^{\mathrm{ds}}$  (DS) and  $\mathbf{P}^{\mathrm{se}}$  (SEA) computed with  $\xi = 5$  (for  $\mathbf{P}^{\mathrm{ds}}$ ,  $\xi$  is the average perplexity such that  $\sum_{i}\mathrm{H}(\mathbf{P}_{i:}^{\mathrm{ds}}) = \sum_{i}\mathrm{H}(\mathbf{P}_{i:}^{\mathrm{se}})$ ). Points' color represents the perplexity  $e^{\mathrm{H}(\mathbf{P}_{i:}) - 1}$ . Right plot: smallest eigenvalues of the Laplacian for the two affinities.
3.2 Symmetric Entropic Affinity Formulation
Based on the previous formulation we now propose symmetric entropic affinities: a symmetric version of EAs that enables keeping the entropy associated with each row (or equivalently column) to the desired value of  $\log \xi + 1$  while producing a symmetric doubly stochastic affinity matrix. Our strategy is to enforce symmetry through an additional constraint in (EA as OT), in a similar fashion as (EOT). More precisely we consider the convex optimization problem
$$
\min  _ {\mathbf {P} \in \mathcal {H} _ {\xi} \cap \mathcal {S}} \left\langle \mathbf {P}, \mathbf {C} \right\rangle . \tag {SEA}
$$
where we recall that  $S$  is the set of  $n \times n$  symmetric matrices. Note that for any  $\xi \leq n - 1$ ,  $\frac{1}{n}\mathbf{1}\mathbf{1}^{\top} \in \mathcal{H}_{\xi} \cap S$  hence the set  $\mathcal{H}_{\xi} \cap S$  is a non-empty and convex set. We first detail some important properties of problem (SEA) (the proofs of the following results can be found in Appendix A.4).
Proposition 4 (Saturation of the entropies). Let  $\mathbf{C} \in \mathcal{S}$  with zero diagonal, then (SEA) with cost  $\mathbf{C}$  has a unique solution that we denote by  $\mathbf{P}^{\mathrm{se}}$ . If moreover  $\mathbf{C} \in \mathcal{D}$ , then for at least  $n - 1$  indices  $i \in [[n]]$  the solution satisfies  $\mathrm{H}(\mathbf{P}_{i:}^{\mathrm{se}}) = \log \xi + 1$ .
In other words, the unique solution  $\mathbf{P}^{\mathrm{se}}$  has at least  $n - 1$  saturated entropies i.e., the corresponding  $n - 1$  points have exactly a perplexity of  $\xi$ . In practice, with the algorithmic solution detailed below, we have observed that all  $n$  entropies are saturated. Therefore, we believe that this proposition can be extended with a few more assumptions on  $\mathbf{C}$ . Accordingly, problem (SEA) allows accurate control over the point-wise entropies while providing a symmetric doubly stochastic matrix, unlike  $\overline{\mathbf{P}^{\mathrm{e}}}$  defined in (Symmetric-SNE), as summarized in Table 1. In the sequel, we denote by  $\mathrm{H_r}(\mathbf{P}) = (\mathrm{H}(\mathbf{P}_{i:}))_i$  the vector of row-wise entropies of  $\mathbf{P}$ . We rely on the following result to compute  $\mathbf{P}^{\mathrm{se}}$ .
Proposition 5 (Solving for SEA). Let  $\mathbf{C} \in \mathcal{D}$ ,  $\mathcal{L}(\mathbf{P},\boldsymbol{\gamma},\boldsymbol{\lambda}) = \langle \mathbf{P},\mathbf{C}\rangle + \langle \boldsymbol{\gamma},(\log \xi +1)\mathbf{1} - \mathrm{H}_{\mathrm{r}}(\mathbf{P})\rangle + \langle \boldsymbol{\lambda},\mathbf{1} - \mathbf{P}\mathbf{1}\rangle$  and  $q(\boldsymbol{\gamma},\boldsymbol{\lambda}) = \min_{\mathbf{P} \in \mathbb{R}_+^{n\times n} \cap \mathcal{S}} \mathcal{L}(\mathbf{P},\boldsymbol{\gamma},\boldsymbol{\lambda})$ . Strong duality holds for (SEA). Moreover, let  $\boldsymbol{\gamma}^{\star},\boldsymbol{\lambda}^{\star} \in \operatorname{argmax}_{\boldsymbol{\gamma} \geq 0,\boldsymbol{\lambda}} q(\boldsymbol{\gamma},\boldsymbol{\lambda})$  be the optimal dual variables respectively associated with the entropy and marginal constraints. Then, for at least  $n - 1$  indices  $i \in [[n]]$ ,  $\gamma_i^\star > 0$ . When  $\forall i \in [[n]]$ ,  $\gamma_i^\star > 0$  then  $\mathrm{H}_{\mathrm{r}}(\mathbf{P}^{\mathrm{se}}) = (\log \xi + 1)\mathbf{1}$  and  $\mathbf{P}^{\mathrm{se}}$  has the form