SlowGuess/NeurIPS_2023_OCR huggingface dataset:
[47] David Van Dijk, Roshan Sharma, Juozas Nainys, Kristina Yim, Pooja Kathail, Ambrose J Carr, Cassandra Burdziak, Kevin R Moon, Christine L Chaffer, Diwakar Pattabiraman, et al. Recovering gene interactions from single-cell data using data diffusion. Cell, 174(3):716-729, 2018.
[48] Jarkko Venna and Samuel Kaski. Neighborhood preservation in nonlinear projection methods: An experimental study. In Artificial Neural Networks—ICANN 2001: International Conference Vienna, Austria, August 21–25, 2001 Proceedings 11, pages 485–491. Springer, 2001.
[49] Max Vlademyrov and Miguel Carreira-Perpinan. Entropic affinities: Properties and efficient numerical computation. In International Conference on Machine Learning (ICML), pages 477–485. PMLR, 2013.
[50] Ulrike Von Luxburg. A tutorial on spectral clustering. Statistics and computing, 17(4):395-416, 2007.
[51] Ron Zass and Amnon Shashua. A unifying approach to hard and probabilistic clustering. In Tenth IEEE International Conference on Computer Vision (ICCV'05) Volume 1, volume 1, pages 294-301. IEEE, 2005.
[52] Ron Zass and Amnon Shashua. Doubly stochastic normalization for spectral clustering. In B. Scholkopf, J. Platt, and T. Hoffman, editors, Neural Information Processing Systems (NeurIPS). MIT Press, 2006.
[53] Lihi Zelnik-Manor and Pietro Perona. Self-tuning spectral clustering. Advances in neural information processing systems, 17, 2004.
[54] Shuangfei Zhai, Tatiana Likhomanenko, Etai Littwin, Jason Ramapuram, Dan Busbridge, Yizhe Zhang, Jiatao Gu, and Joshua M. Susskind.  $\S\Sigma\Sigma\text{重}$  reparam: Stable transformer training with spectral reparametrization. 2023.
[55] Dengyong Zhou, Olivier Bousquet, Thomas Lal, Jason Weston, and Bernhard Schölkopf. Learning with local and global consistency. Neural Information Processing Systems (NeurIPS), 16, 2003.
A.1 Euclidean Projection onto  $S$
For the problem  $\arg \min_{\mathbf{P} \in S} \| \mathbf{P} - \mathbf{K} \|_2^2$ , the Lagrangian takes the form, with  $\mathbf{W} \in \mathbb{R}^{n \times n}$
$$
\mathcal {L} (\mathbf {P}, \mathbf {W}) = \| \mathbf {P} - \mathbf {K} \| _ {2} ^ {2} + \langle \mathbf {W}, \mathbf {P} - \mathbf {P} ^ {\top} \rangle . \tag {3}
$$
Cancelling the gradient of  $\mathcal{L}$  with respect to  $\mathbf{P}$  gives  $2(\mathbf{P}^{\star} - \mathbf{K}) + \mathbf{W} - \mathbf{W}^{\top} = \mathbf{0}$ . Thus  $\mathbf{P}^{\star} = \mathbf{K} + \frac{1}{2}\left(\mathbf{W}^{\top} - \mathbf{W}\right)$ . Using the symmetry constraint on  $\mathbf{P}^{\star}$  yields  $\mathbf{P}^{\star} = \frac{1}{2}\left(\mathbf{K} + \mathbf{K}^{\top}\right)$ . Hence we have:
$$
\underset {\mathbf {P} \in \mathcal {S}} {\arg \min } \| \mathbf {P} - \mathbf {K} \| _ {2} ^ {2} = \frac {1}{2} (\mathbf {K} + \mathbf {K} ^ {\top}). \tag {4}
$$
A.2 From Symmetric Entropy-Constrained OT to Sinkhorn Iterations
In this section, we derive Sinkhorn iterations from the problem (EOT). Let  $\mathbf{C} \in \mathcal{D}$ . We start by making the constraints explicit.
$$
\min  _ {\mathbf {P} \in \mathbb {R} _ {+} ^ {n \times n}} \langle \mathbf {P}, \mathbf {C} \rangle \tag {5}
$$
$$
\text {s . t .} \quad \sum_ {i \in [ n ]} \mathrm {H} \left(\mathbf {P} _ {i:}\right) \geq \eta \tag {6}
$$
$$
\mathbf {P} \mathbf {1} = \mathbf {1}, \quad \mathbf {P} = \mathbf {P} ^ {\top}. \tag {7}
$$
For the above convex problem the Lagrangian writes, where  $\nu \in \mathbb{R}_+$ ,  $\mathbf{f} \in \mathbb{R}^n$  and  $\Gamma \in \mathbb{R}^{n \times n}$ :
$$
\mathcal {L} (\mathbf {P}, \mathbf {f}, \nu , \Gamma) = \left\langle \mathbf {P}, \mathbf {C} \right\rangle + \left\langle \nu , \eta - \sum_ {i \in [ n ]} \mathrm {H} \left(\mathbf {P} _ {i}\right) \right\rangle + 2 \left\langle \mathbf {f}, \mathbf {1} - \mathbf {P} \mathbf {1} \right\rangle + \left\langle \boldsymbol {\Gamma}, \mathbf {P} - \mathbf {P} ^ {\top} \right\rangle . \tag {8}
$$
Strong duality holds and the first order KKT condition gives for the optimal primal  $\mathbf{P}^{\star}$  and dual  $(\nu^{\star},\mathbf{f}^{\star},\Gamma^{\star})$  variables:
$$
\nabla_ {\mathbf {P}} \mathcal {L} \left(\mathbf {P} ^ {\star}, \mathbf {f} ^ {\star}, \nu^ {\star}, \boldsymbol {\Gamma} ^ {\star}\right) = \mathbf {C} + \nu^ {\star} \log \mathbf {P} ^ {\star} - 2 \mathbf {f} ^ {\star} \mathbf {1} ^ {\top} + \boldsymbol {\Gamma} ^ {\star} - \boldsymbol {\Gamma} ^ {\star \top} = \mathbf {0}. \tag {9}
$$
Since  $\mathbf{P}^{\star},\mathbf{C}\in S$  we have  $\Gamma^{\star} - \Gamma^{\star \top} = \mathbf{f}^{\star}\mathbf{1}^{\top} - \mathbf{1}\mathbf{f}^{\star \top}$ . Hence  $\mathbf{C} + \nu^{\star}\log \mathbf{P}^{\star} - \mathbf{f}^{\star}\oplus \mathbf{f}^{\star} = \mathbf{0}$ . Suppose that  $\nu^{\star} = 0$  then the previous reasoning implies that  $\forall (i,j),C_{ij} = f_i^\star +f_j^\star$ . Using that  $\mathbf{C}\in \mathcal{D}$  we have  $C_{ii} = C_{jj} = 0$  thus  $\forall i,f_i^\star = 0$  and thus this would imply that  $\mathbf{C} = 0$  which is not allowed by hypothesis. Therefore  $\nu^{\star}\neq 0$  and the entropy constraint is saturated at the optimum by complementary slackness. Isolating  $\mathbf{P}^{\star}$  then yields:
$$
\mathbf {P} ^ {\star} = \exp \left(\left(\mathbf {f} ^ {\star} \oplus \mathbf {f} ^ {\star} - \mathbf {C}\right) / \nu^ {\star}\right). \tag {10}
$$
$\mathbf{P}^{\star}$  must be primal feasible in particular  $\mathbf{P}^{\star}\mathbf{1} = \mathbf{1}$ . This constraint gives us the Sinkhorn fixed point relation for  $\mathbf{f}^{\star}$ :
$$
\forall i \in [   [ n ]   ], \quad \left[ \mathbf {f} ^ {\star} \right] _ {i} = - \nu^ {\star} \operatorname {L S E} \left(\left(\mathbf {f} ^ {\star} - \mathbf {C} _ {: i}\right) / \nu^ {\star}\right), \tag {11}
$$
where for a vector  $\alpha$ , we use the notation  $\mathrm{LSE}(\alpha) = \log \sum_{k} \exp(\alpha_k)$ .
We recall the result
$$
\min  _ {\mathbf {P} \in \mathcal {H} _ {\xi}} \langle \mathbf {P}, \mathbf {C} \rangle . \quad (\text {E A a s O T})
$$
Proof. We begin by rewriting the above problem to make the constraints more explicit.
$$
\min_{\substack{\mathbf{P}\in \mathbb{R}_{+}^{n\times n}}}\langle \mathbf{P},\mathbf{C}\rangle
$$
$$
\text {s . t .} \quad \forall i, \mathrm {H} \left(\mathbf {P} _ {i:}\right) \geq \log \xi + 1
$$
$$
\mathrm {P} 1 = 1.
$$
By concavity of entropy, one has that the entropy constraint is convex thus the above primal problem is a convex optimization problem. Moreover, the latter is strictly feasible for any  $\xi \in [[n - 1]]$ . Therefore Slater's condition is satisfied and strong duality holds.
Introducing the dual variables  $\pmb{\lambda} \in \mathbb{R}^n$  and  $\varepsilon \in \mathbb{R}_+^n$ , the Lagrangian of the above problem writes:
$$
\mathcal {L} (\mathbf {P}, \boldsymbol {\lambda}, \varepsilon) = \langle \mathbf {P}, \mathbf {C} \rangle + \langle \varepsilon , (\log \xi + 1) \mathbf {1} - \mathrm {H} _ {\mathrm {r}} (\mathbf {P}) \rangle + \langle \boldsymbol {\lambda}, \mathbf {1} - \mathbf {P} \mathbf {1} \rangle , \tag {12}
$$
where we recall that  $\mathrm{H_r}(\mathbf{P}) = (\mathrm{H}(\mathbf{P}_{i:}))_i$ . Note that we will deal with the constraint  $\mathbf{P}\in \mathbb{R}_+^{n\times n}$  directly, hence there is no associated dual variable. Since strong duality holds, for any solution  $\mathbf{P}^{\star}$  to the primal problem and any solution  $(\varepsilon^{\star},\lambda^{\star})$  to the dual problem, the pair  $\mathbf{P}^{\star},(\varepsilon^{\star},\lambda^{\star})$  must satisfy the Karush-Kuhn-Tucker (KKT) conditions. The first-order optimality condition gives:
$$
\nabla_ {\mathbf {P}} \mathcal {L} \left(\mathbf {P} ^ {\star}, \varepsilon^ {\star}, \boldsymbol {\lambda} ^ {\star}\right) = \mathbf {C} + \operatorname {d i a g} \left(\varepsilon^ {\star}\right) \log \mathbf {P} ^ {\star} - \boldsymbol {\lambda} ^ {\star} \mathbf {1} ^ {\top} = \mathbf {0}. \quad (\text {f i r s t - o r d e r})
$$
Assume that there exists  $\ell \in [[n]]$  such that  $\varepsilon_{\ell}^{\star} = 0$ . Then (first-order) gives that the  $\ell^{th}$  row of  $\mathbf{C}$  is constant which is not allowed by hypothesis. Therefore  $\varepsilon^{\star} > 0$  (i.e.,  $\varepsilon^{\star}$  has positive entries). Thus isolating  $\mathbf{P}^{\star}$  in the first order condition results in:
$$
\mathbf {P} ^ {\star} = \operatorname {d i a g} (\mathbf {u}) \exp \left(- \operatorname {d i a g} \left(\varepsilon^ {\star}\right) ^ {- 1} \mathbf {C}\right) \tag {13}
$$
where  $\mathbf{u} = \exp (\lambda^{\star}\oslash \varepsilon^{\star})$ . This matrix must satisfy the stochasticity constraint  $\mathbf{P1} = \mathbf{1}$ . Hence one has  $\mathbf{u} = \mathbf{1}\oslash (\exp (\mathrm{diag}(\varepsilon^{\star})^{-1}\mathbf{C})\mathbf{1})$  and  $\mathbf{P}^{\star}$  has the form
$$
\forall (i, j) \in [   [ n ]   ] ^ {2}, \quad P _ {i j} ^ {\star} = \frac {\exp \left(- C _ {i j} / \varepsilon_ {i} ^ {\star}\right)}{\sum_ {\ell} \exp \left(- C _ {i \ell} / \varepsilon_ {i} ^ {\star}\right)}. \tag {14}
$$
As a consequence of  $\varepsilon^{\star} > 0$ , complementary slackness in the KKT conditions gives us that for all  $i$ , the entropy constraint is saturated i.e.,  $\mathrm{H}(\mathbf{P}_{i:}^{\star}) = \log \xi + 1$ . Therefore  $\mathbf{P}^{\star}$  solves the problem (EA). Conversely any solution of (EA)  $P_{ij}^{\star} = \frac{\exp(-C_{ij} / \varepsilon_i^{\star})}{\sum_\ell \exp(-C_{ie} / \varepsilon_i^{\star})}$  with  $(\varepsilon_{i}^{\star})$  such that  $\mathrm{H}(\mathbf{P}_{i:}^{\star}) = \log \xi + 1$  gives an admissible matrix for  $\min_{\mathbf{P} \in \mathcal{H}_{\xi}} \langle \mathbf{P}, \mathbf{C} \rangle$  and the associated variables satisfy the KKT conditions which are sufficient conditions for optimality since the problem is convex.
A.4 Proof of Proposition 4 and Proposition 5
The goal of this section is to prove the following results:
Proposition 4 (Saturation of the entropies). Let  $\mathbf{C} \in S$  with zero diagonal, then (SEA) with cost  $\mathbf{C}$  has a unique solution that we denote by  $\mathbf{P}^{\mathrm{se}}$ . If moreover  $\mathbf{C} \in \mathcal{D}$ , then for at least  $n - 1$  indices  $i \in [[n]]$  the solution satisfies  $\mathrm{H}(\mathbf{P}_{i:}^{\mathrm{se}}) = \log \xi + 1$ .
The unicity of the solution in Proposition 4 is a consequence of the following lemma
Lemma 7. Let  $\mathbf{C} \neq 0 \in S$  with zero diagonal. Then the problem  $\min_{\mathbf{P} \in \mathcal{H}_{\xi} \cap S} \langle \mathbf{P}, \mathbf{C} \rangle$  has a unique solution.
Proof. Making the constraints explicit, the primal problem of symmetric entropic affinity takes the following form
$$
\min  _ {\mathbf {P} \in \mathcal {P} _ {x}} \quad \langle \mathbf {P}, \mathbf {C} \rangle
$$
$$
\mathbf {P} \in \mathbb {R} _ {+} ^ {n \times n}
$$
$$
\text {s . t .} \forall i, \mathrm {H} \left(\mathbf {P} _ {i:}\right) \geq \log \xi + 1 \tag {SEA}
$$
$$
\mathbf {P} \mathbf {1} = \mathbf {1}, \quad \mathbf {P} = \mathbf {P} ^ {\top}.
$$
Suppose that the solution is not unique i.e., there exists a couple of optimal solutions  $(\mathbf{P}_1, \mathbf{P}_2)$  that satisfy the constraints of (SEA) and such that  $\langle \mathbf{P}_1, \mathbf{C} \rangle = \langle \mathbf{P}_2, \mathbf{C} \rangle$ . For  $i \in [n]$ , we denote the
function  $f_{i}:\mathbf{P}\to (\log \xi +1) - \mathrm{H}(\mathbf{P}_{i:})$ . Then  $f_{i}$  is continuous, strictly convex and the entropy conditions of (SEA) can be written as  $\forall i\in [[n]],f_i(\mathbf{P})\leq 0$
Now consider  $\mathbf{Q} = \frac{1}{2} (\mathbf{P}_1 + \mathbf{P}_2)$ . Then clearly  $\mathbf{Q}\mathbf{1} = \mathbf{1},\mathbf{Q} = \mathbf{Q}^{\top}$ . Since  $f_{i}$  is strictly convex we have  $f_{i}(\mathbf{Q}) = f_{i}(\frac{1}{2}\mathbf{P}_{1} + \frac{1}{2}\mathbf{P}_{2}) < \frac{1}{2} f_{i}(\mathbf{P}_{1}) + \frac{1}{2} f(\mathbf{P}_{2})\leq 0$ . Thus  $f_{i}(\mathbf{Q}) < 0$  for any  $i\in [[n]]$ . Take any  $\varepsilon >0$  and  $i\in [[n]]$ . By continuity of  $f_{i}$  there exists  $\delta_i > 0$  such that, for any  $\mathbf{H}$  with  $\| \mathbf{H}\| _F\leq \delta_i$ , we have  $f_{i}(\mathbf{Q} + \mathbf{H}) < f_{i}(\mathbf{Q}) + \varepsilon$ . Take  $\varepsilon >0$  such that  $\forall i\in [[n]],0 < \varepsilon < - \frac{1}{2} f_{i}(\mathbf{Q})$  (this is possible since for any  $i\in [[n]],f_i(\mathbf{Q}) < 0$ ) and  $\mathbf{H}$  with  $\| \mathbf{H}\| _F\leq \min_{i\in [[n]]}\delta_i$ . Then for any  $i\in [[n]],f_{i}(\mathbf{Q} + \mathbf{H}) < 0$ . In other words, we have proven that there exists  $\eta >0$  such that for any  $\mathbf{H}$  such that  $\| \mathbf{H}\| _F\leq \eta$ , it holds:  $\forall i\in [[n]],f_{i}(\mathbf{Q} + \mathbf{H}) < 0$ .
Now let us take  $\mathbf{H}$  as the Laplacian matrix associated to  $\mathbf{C}$  i.e., for any  $(i,j)\in [[n]]^2$ ,  $H_{ij} = -C_{ij}$  if  $i\neq j$  and  $\sum_l C_{il}$  otherwise. Then we have  $\langle \mathbf{H},\mathbf{C}\rangle = -\sum_{i\neq j}C_{ij}^{2} + 0 = -\sum_{i\neq j}C_{ij}^{2} < 0$  since  $\mathbf{C}$  has zero diagonal (and is nonzero). Moreover,  $\mathbf{H} = \mathbf{H}^{\top}$  since  $\mathbf{C}$  is symmetric and  $\mathbf{H1} = \mathbf{0}$  by construction. Consider for  $0 < \beta \leq \frac{\eta}{\|\mathbf{H}\|_F}$ , the matrix  $\mathbf{H}_{\beta} := \beta \mathbf{H}$ . Then  $\| \mathbf{H}_{\beta}\| _F = \beta \| \mathbf{H}\| _F\leq \eta$ . By the previous reasoning one has:  $\forall i\in [[n]],f_i(\mathbf{Q} + \mathbf{H}_\beta) < 0$ . Moreover,  $(\mathbf{Q} + \mathbf{H}_{\beta})^{\top} = \mathbf{Q} + \mathbf{H}_{\beta}$  and  $(\mathbf{Q} + \mathbf{H}_{\beta})\mathbf{1} = \mathbf{1}$ . For  $\beta$  small enough we have  $\mathbf{Q} + \mathbf{H}_{\beta}\in \mathbb{R}_{+}^{n\times n}$  and thus there is a  $\beta$  (that depends on  $\mathbf{P}_1$  and  $\mathbf{P}_2$ ) such that  $\mathbf{Q} + \mathbf{H}_{\beta}$  is admissible i.e., satisfies the constraints of (SEA). Then, for such  $\beta$ ,
$$
\begin{array}{l} \langle \mathbf {C}, \mathbf {Q} + \mathbf {H} _ {\beta} \rangle - \langle \mathbf {C}, \mathbf {P} _ {1} \rangle = \frac {1}{2} \langle \mathbf {C}, \mathbf {P} _ {1} + \mathbf {P} _ {2} \rangle + \langle \mathbf {C}, \mathbf {H} _ {\beta} \rangle - \langle \mathbf {C}, \mathbf {P} _ {1} \rangle \tag {15} \\ = \langle \mathbf {C}, \mathbf {H} _ {\beta} \rangle = \beta \langle \mathbf {H}, \mathbf {C} \rangle <   0. \\ \end{array}
$$
Thus  $\langle \mathbf{C},\mathbf{Q} + \mathbf{H}_{\beta}\rangle <  \langle \mathbf{C},\mathbf{P}_1\rangle$  which leads to a contradiction.
We can now prove the rest of the claims of Proposition 4 and Proposition 5.
Proof. Let  $\mathbf{C} \in \mathcal{D}$ . We first prove Proposition 4. The unicity is a consequence of Lemma 7. For the saturation of the entropies we consider the Lagrangian of the problem (SEA) that writes
$$
\mathcal {L} (\mathbf {P}, \boldsymbol {\lambda}, \gamma , \Gamma) = \langle \mathbf {P}, \mathbf {C} \rangle + \langle \gamma , (\log \xi + 1) \mathbf {1} - \mathrm {H} _ {r} (\mathbf {P}) \rangle + \langle \boldsymbol {\lambda}, \mathbf {1} - \mathbf {P} \mathbf {1} \rangle + \langle \boldsymbol {\Gamma}, \mathbf {P} - \mathbf {P} ^ {\top} \rangle
$$
for dual variables  $\gamma \in \mathbb{R}_+^n$ ,  $\lambda \in \mathbb{R}^n$  and  $\Gamma \in \mathbb{R}^{n\times n}$ . Strong duality holds by Slater's conditions because  $\frac{1}{n}\mathbf{11}^\top$  is strictly feasible for  $\xi \leq n - 1$ . Since strong duality holds, for any solution  $\mathbf{P}^{\star}$  to the primal problem and any solution  $(\gamma^{\star},\lambda^{\star},\Gamma^{\star})$  to the dual problem, the pair  $\mathbf{P}^{\star}$ ,  $(\gamma^{\star},\lambda^{\star},\Gamma^{\star})$  must satisfy the KKT conditions. They can be stated as follows:
$$
\begin{array}{l} \mathbf {C} + \operatorname {d i a g} \left(\boldsymbol {\gamma} ^ {\star}\right) \log \mathbf {P} ^ {\star} - \boldsymbol {\lambda} ^ {\star} \mathbf {1} ^ {\top} + \boldsymbol {\Gamma} ^ {\star} - \boldsymbol {\Gamma} ^ {\star \top} = \mathbf {0} \\ \mathbf {P} ^ {\star} \mathbf {1} = \mathbf {1}, \mathrm {H} _ {r} (\mathbf {P} ^ {\star}) \geq (\log \xi + 1) \mathbf {1}, \mathbf {P} ^ {\star} = \mathbf {P} ^ {\star^ {\top}} \tag {KKT-SEA} \\ \gamma^ {\star} \geq 0 \\ \forall i, \gamma_ {i} ^ {\star} (\mathrm {H} (\mathbf {P} _ {i:} ^ {\star}) - (\log \xi + 1)) = 0. \\ \end{array}
$$
Let us denote  $I = \{\ell \in [[n]]$  s.t.  $\gamma_{\ell}^{\star} = 0\}$ . For  $\ell \in I$ , using the first-order condition, one has for  $i \in [[n]]$ ,  $C_{\ell i} = \lambda_{\ell}^{\star} - \Gamma_{\ell i}^{\star} + \Gamma_{i\ell}^{\star}$ . Since  $\mathbf{C} \in \mathcal{D}$ , we have  $C_{\ell \ell} = 0$  thus  $\lambda_{\ell}^{\star} = 0$  and  $C_{\ell i} = \Gamma_{i\ell}^{\star} - \Gamma_{\ell i}^{\star}$ . For  $(\ell, \ell') \in I^{2}$ , one has  $C_{\ell \ell'} = \Gamma_{\ell' \ell}^{\star} - \Gamma_{\ell \ell'}^{\star} = -(\Gamma_{\ell \ell'}^{\star} - \Gamma_{\ell' \ell}^{\star}) = -C_{\ell' \ell}$ .  $\mathbf{C}$  is symmetric thus  $C_{\ell \ell'} = 0$ . Since  $\mathbf{C}$  only has null entries on the diagonal, this shows that  $\ell = \ell'$  and therefore  $I$  has at most one element. By complementary slackness condition (last row of the KKT-SEA conditions) it holds that  $\forall i \neq \ell, \mathrm{H}(\mathbf{P}_{i:}^{\star}) = \log \xi + 1$ . Since the solution of (SEA) is unique  $\mathbf{P}^{\star} = \mathbf{P}^{\mathrm{se}}$  and thus  $\forall i \neq \ell, \mathrm{H}(\mathbf{P}_{i:}^{\mathrm{se}}) = \log \xi + 1$  which proves Proposition 4 but also that for at least  $n - 1$  indices  $\gamma_i^{\star} > 0$ . Moreover, from the KKT conditions we have
$$
\forall (i, j) \in [   [ n ]   ] ^ {2}, \Gamma_ {j i} ^ {\star} - \Gamma_ {i j} ^ {\star} = C _ {i j} + \gamma_ {i} ^ {\star} \log P _ {i j} ^ {\star} - \lambda_ {i} ^ {\star}. \tag {16}
$$
Now take  $(i,j)\in [[n]]^2$  fixed. From the previous equality  $\Gamma_{ji}^{\star} - \Gamma_{ij}^{\star} = C_{ij} + \gamma_{i}^{\star}\log P_{ij}^{\star} - \lambda_{i}^{\star}$  but also  $\Gamma_{ij}^{\star} - \Gamma_{ji}^{\star} = C_{ji} + \gamma_{j}^{\star}\log P_{ji}^{\star} - \lambda_{j}^{\star}$ . Using that  $\mathbf{P}^{\star} = (\mathbf{P}^{\star})^{\top}$  and  $\mathbf{C}\in S$  we get  $\Gamma_{ij}^{\star} - \Gamma_{ji}^{\star} = C_{ij} + \gamma_{j}^{\star}\log P_{ij}^{\star} - \lambda_{j}^{\star}$ . But  $\Gamma_{ij}^{\star} - \Gamma_{ji}^{\star} = -(\Gamma_{ji}^{\star} - \Gamma_{ij}^{\star})$  which gives
$$
C _ {i j} + \gamma_ {j} ^ {\star} \log P _ {i j} ^ {\star} - \lambda_ {j} ^ {\star} = - \left(C _ {i j} + \gamma_ {i} ^ {\star} \log P _ {i j} ^ {\star} - \lambda_ {i} ^ {\star}\right). \tag {17}
$$
$$
\forall (i, j) \in [   [ n ]   ] ^ {2}, 2 C _ {i j} + \left(\gamma_ {i} ^ {\star} + \gamma_ {j} ^ {\star}\right) \log P _ {i j} ^ {\star} - \left(\lambda_ {i} ^ {\star} + \lambda_ {j} ^ {\star}\right) = 0. \tag {18}
$$
Consequently, if  $\gamma^{\star} > 0$  we have the desired form from the above equation and by complementary slackness  $\mathrm{H_r}(\mathbf{P^{se}}) = (\log \xi +1)\mathbf{1}$  which proves Proposition 5. Note that otherwise, it holds
$$
\forall (i, j) \neq (\ell , \ell), P _ {i j} ^ {\star} = \exp \left(\frac {\lambda_ {i} ^ {\star} + \lambda_ {j} ^ {\star} - 2 C _ {i j}}{\gamma_ {i} ^ {\star} + \gamma_ {j} ^ {\star}}\right). \tag {19}
$$
A.5 EA and SEA as a KL projection
We prove the characterization as a projection of (EA) in Lemma 8 and of (SEA) in Lemma 9.
Lemma 8. Let  $\mathbf{C} \in \mathcal{D}, \sigma > 0$  and  $\mathbf{K}_{\sigma} = \exp(-\mathbf{C} / \sigma)$ . Then for any  $\sigma \leq \min_{i} \varepsilon_{i}^{\star}$ , it holds  $\mathbf{P}^{\mathrm{e}} = \operatorname{Proj}_{\mathcal{H}_{\xi}}^{\mathrm{KL}}(\mathbf{K}_{\sigma}) = \arg \min_{\mathbf{P} \in \mathcal{H}_{\xi}} \mathrm{KL}(\mathbf{P}|\mathbf{K}_{\sigma})$ .
Proof. The KL projection of  $\mathbf{K}$  onto  $\mathcal{H}_{\xi}$  reads
$$
\min  _ {\mathbf {P} \in \mathbb {R} _ {+} ^ {n \times n}} \quad \operatorname {K L} (\mathbf {P} | \mathbf {K}) \tag {20}
$$
$$
s. t. \quad \forall i, \mathrm {H} \left(\mathbf {P} _ {i:}\right) \geq \log \xi + 1 \tag {21}
$$
$$
\mathbf {P} \mathbf {1} = \mathbf {1}. \tag {22}
$$
Introducing the dual variables  $\pmb{\lambda} \in \mathbb{R}^n$  and  $\pmb{\kappa} \in \mathbb{R}_+^n$ , the Lagrangian of this problem reads:
$$
\mathcal {L} (\mathbf {P}, \boldsymbol {\lambda}, \boldsymbol {\kappa}) = \operatorname {K L} (\mathbf {P} | \mathbf {K}) + \left\langle \boldsymbol {\kappa}, (\log \xi + 1) \mathbf {1} - \mathrm {H} (\mathbf {P}) \right\rangle + \left\langle \boldsymbol {\lambda}, \mathbf {1} - \mathbf {P} \mathbf {1} \right\rangle \tag {23}
$$
Strong duality holds hence for any solution  $\mathbf{P}^{\star}$  to the above primal problem and any solution  $(\kappa^{\star},\lambda^{\star})$  to the dual problem, the pair  $\mathbf{P}^{\star},(\kappa^{\star},\lambda^{\star})$  must satisfy the KKT conditions. The first-order optimality condition gives:
$$
\nabla_ {\mathbf {P}} \mathcal {L} \left(\mathbf {P} ^ {\star}, \boldsymbol {\kappa} ^ {\star}, \boldsymbol {\lambda} ^ {\star}\right) = \log \left(\mathbf {P} ^ {\star} \oslash \mathbf {K}\right) + \operatorname {d i a g} \left(\boldsymbol {\kappa} ^ {\star}\right) \log \mathbf {P} ^ {\star} - \boldsymbol {\lambda} ^ {\star} \mathbf {1} ^ {\top} = \mathbf {0}. \tag {24}
$$
Solving for  $\lambda^{\star}$  given the stochasticity constraint and isolating  $\mathbf{P}^{\star}$  gives
$$
\forall (i, j) \in [   [ n ]   ] ^ {2}, \quad P _ {i j} ^ {\star} = \frac {\exp \left(\left(\log K _ {i j}\right) / \left(1 + \kappa_ {i} ^ {\star}\right)\right)}{\sum_ {\ell} \exp \left(\left(\log K _ {i \ell}\right) / \left(1 + \kappa_ {i} ^ {\star}\right)\right)}. \tag {25}
$$
We now consider  $\mathbf{P}^{\star}$  as a function of  $\kappa$ . Plugging this expression back in  $\mathcal{L}$  yields the dual function  $\kappa \mapsto \mathcal{G}(\kappa)$ . The latter is concave as any dual function and its gradient reads:
$$
\nabla_ {\kappa} \mathcal {G} (\kappa) = (\log \xi + 1) \mathbf {1} - \mathrm {H} \left(\mathbf {P} ^ {\star} (\kappa)\right). \tag {26}
$$
Denoting by  $\pmb{\rho} = \mathbf{1} + \pmb{\kappa}$  and taking the dual feasibility constraint  $\kappa \geq 0$  into account gives the solution: for any  $i$ ,  $\rho_{i}^{\star} = \max (\varepsilon_{i}^{\star},1)$  where  $\varepsilon^{\star}$  solves (EA) with cost  $\mathbf{C} = -\log \mathbf{K}$ . Moreover we have that  $\sigma \leq \min (\varepsilon^{\star})$  where  $\varepsilon^{\star}\in (\mathbb{R}_{+}^{*})^{n}$  solves (EA). Therefore for any  $i\in [n]$ , one has  $\varepsilon_i^\star /\sigma \geq 1$ . Thus there exists  $\kappa_{i}^{\star}\in \mathbb{R}_{+}$  such that  $\sigma (1 + \kappa_i^{\star}) = \varepsilon_i^{\star}$ .
This  $\kappa^{\star}$  cancels the above gradient i.e.,  $(\log \xi +1)\mathbf{1} = \mathrm{H}(\mathbf{P}^{\star}(\pmb {\kappa}^{\star}))$  thus solves the dual problem. Therefore given the expression of  $\mathbf{P}^{\star}$  we have that  $\operatorname{Proj}_{\mathcal{H}_\xi}^{\mathrm{KL}}(\mathbf{K}) = \mathbf{P}^{\mathrm{e}}$
Lemma 9. Let  $\mathbf{C} \in \mathcal{D}, \sigma > 0$  and  $\mathbf{K}_{\sigma} = \exp(-\mathbf{C} / \sigma)$ . Suppose that the optimal dual variable  $\gamma^{\star}$  associated with the entropy constraint of (SEA) is positive. Then for any  $\sigma \leq \min_{i} \gamma_{i}^{\star}$ , it holds  $\mathbf{P}^{\mathrm{se}} = \operatorname{Proj}_{\mathcal{H}_{\xi} \cap \mathcal{S}}^{\mathrm{KL}}(\mathbf{K}_{\sigma})$ .
Proof. Let  $\sigma > 0$ . The KL projection of  $\mathbf{K}$  onto  $\mathcal{H}_{\xi} \cap \mathcal{S}$  boils down to the following optimization problem.
$$
\min_{\substack{\mathbf{P}\in \mathbb{R}_{+}^{n\times n}}\quad \mathrm{KL}(\mathbf{P}|\mathbf{K}_{\sigma})
$$
$$
\text {s . t .} \quad \forall i, \mathrm {H} \left(\mathbf {P} _ {i:}\right) \geq \log \xi + 1 \tag {SEA-Proj}
$$
$$
\mathbf {P} \mathbf {1} = \mathbf {1}, \quad \mathbf {P} ^ {\top} = \mathbf {P}.
$$