SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Guan-Horng Liu, Tianrong Chen, Evangelos A. Theodorou†, Molei Tao†  
Georgia Institute of Technology, USA
{ghliu, tianrong.chen, evangelos.theodorou, mtao}@gatech.edu
Modern successes of diffusion models in learning complex, high-dimensional data distributions are attributed, in part, to their capability to construct diffusion processes with analytic transition kernels and score functions. The tractability results in a simulation-free framework with stable regression losses, from which reversed, generative processes can be learned at scale. However, when data is confined to a constrained set as opposed to a standard Euclidean space, these desirable characteristics appear to be lost based on prior attempts. In this work, we propose Mirror Diffusion Models (MDM), a new class of diffusion models that generate data on convex constrained sets without losing any tractability. This is achieved by learning diffusion processes in a dual space constructed from a mirror map, which, crucially, is a standard Euclidean space. We derive efficient computation of mirror maps for popular constrained sets, such as simplices and  $\ell_2$ -balls, showing significantly improved performance of MDM over existing methods. For safety and privacy purposes, we also explore constrained sets as a new mechanism to embed invisible but quantitative information (i.e., watermarks) in generated data, for which MDM serves as a compelling approach. Our work brings new algorithmic opportunities for learning tractable diffusion on complex domains.
MDM: learns tractable diffusions in dual spaces
Figure 1: Mirror Diffusion Models (MDM) is a new class of diffusion models for convex constrained manifolds  $\mathcal{M} \subseteq \mathbb{R}^d$ . (left) Instead of learning score-approximate diffusions on  $\mathcal{M}$ , MDM applies a mirror map  $\nabla \phi$  and learns tractable diffusions in its unconstrained dual-space  $\nabla \phi(\mathcal{M}) = \mathbb{R}^d$ . (right) We also present MDM for watermarked generation, where generated contents (e.g., images) live in a high-dimensional token constrained set  $\mathcal{M}$  that is certifiable only from the private user.
Diffusion models [1-3] have emerged as powerful generative models with their remarkable successes in synthesizing high-fidelity data such as images [4, 5], audio [6, 7], and 3D geometry [8, 9]. These models work by progressively diffusing data to noise, and learning the score functions (often
Equal advising.
Table 1: Comparison of different diffusion models for constrained generation on  $\mathcal{M} \subseteq \mathbb{R}^d$ . Rather than learning reflected diffusions on  $\mathcal{M}$  with approximate scores, our Mirror Diffusion constructs a mirror map  $\nabla \phi$  and lift the diffusion processes to the unconstrained dual space  $\nabla \phi(\mathcal{M}) = \mathbb{R}^d$ , inheriting favorable features from standard Euclidean-space diffusion models. Constraints are satisfied by construction via the inverse map  $\nabla \phi^*$ ; that is,  $\nabla \phi^*(y) \in \mathcal{M}$  for all  $y \in \nabla \phi(\mathcal{M})$ .
Diffusion models	Domain	Tractable conditional score	Simulation-free training	Regression objective
Reflected Diffusion
Fishman et al. [18]	M	X	X	X
Lou and Ermon [25]	M	X	△²	✓
Mirror Diffusion
This work (ours)	∇φ(M)	✓	✓	✓
parameterized by neural networks) to reverse the processes [10]; the reversed processes thereby provide transport maps that generate data from noise. Modern diffusion models [11-13] often employ diffusion processes whose transition kernels are analytically available. This characteristic enables simulation-free training, bypassing the necessity to simulate the underlying diffusion processes by directly sampling the diffused data. It also leads to tractable conditional score functions and simple regression objectives, facilitating computational scalability for high-dimensional problems. In standard Euclidean spaces, tractability can be accomplished by designing the diffusion processes as linear stochastic differential equations, but doing so for non-Euclidean spaces is nontrivial.
Recent progress of diffusion models has expanded their application to non-Euclidean spaces, such as Riemannian manifolds [14, 15], where data live in a curved geometry. The development of generative models on manifolds has enabled various new applications, such as modeling earth and climate events [16] and learning densities on meshes [17]. In this work, we focus on generative modeling on constrained sets (also called constrained manifolds [18]) that are defined by a set of inequality constraints. Such constrained sets, denoted by  $\mathcal{M} \subseteq \mathbb{R}^d$ , are also ubiquitous in several scientific fields such as computational statistics [19], biology [20], and robotics [21, 22].
Previous endeavors in the direction of constrained generation have primarily culminated in reflected diffusions [23, 24], which reflect the direction at the boundary  $\partial \mathcal{M}$  to ensure that samples remain inside the constraints. Unfortunately, reflected diffusions do not possess closed-form transition kernels [18, 25], thus necessitating the approximation of the conditional score functions that can hinder learning performance. Although simulation-free training is still possible $^2$ , it comes with a computational overhead due to the use of geometric techniques [25]. From a theoretical standpoint, reflected Langevin dynamics are widely regarded for their extended mixing time [26, 27]. This raises practical concerns when simulating reflected diffusions, as prior methods often adopted special treatments such as thresholding and discretization [18, 25].
An alternative diffusion for constrained sampling is the Mirror Langevin Dynamics (MLD [28]). MLD, as a subclass of Riemannian Langevin Dynamics [29, 30], is tailored specifically to convex constrained sampling. Specifically, MLD constructs a strictly convex function  $\phi$  so that its gradient map, often referred to as the mirror map  $\nabla \phi : \mathcal{M} \to \mathbb{R}^d$ , defines a nonlinear mapping from the initial constrained set to an unconstrained dual space. Despite extensive investigation of MLD [31-34], its potential as a foundation for designing diffusion generative models remains to be comprehensively examined. Thus, there is a need to explore the potential of MLD for designing new, effective, and tailored diffusion models for constrained generation.
With this in mind, we propose Mirror Diffusion Models (MDM), a new class of diffusion generative models for convex constrained sets. Similar to MLD, MDM utilizes mirror maps  $\nabla \phi$  that transform data distributions on  $\mathcal{M}$  to its dual space  $\nabla \phi (\mathcal{M})$ . From there, MDM learns a dual-space diffusion between the mirrored distribution and Gaussian prior in the dual space. Note that this is in contrast to MLD, which constructs invariant distributions on the initial constrained set; MDM instead considers dual-space priors. Since the dual space is also unconstrained Euclidean space  $\nabla \phi (\mathcal{M}) = \mathbb{R}^d$ , MDM
$^{2}$ Lou and Ermon [25] proposed a semi-simulation-free training with the aid of geometric techniques.
Figure 2: MDM for watermarked generation: (left) We first construct a constrained set  $\mathcal{M}$  based on a set of user-defined tokens private to other users. (right) MDM can be instantiated by either learning the corresponding dual-space diffusions, or projecting pretrained, i.e., unwatermarked, diffusion models onto  $\mathcal{M}$ . In both cases, MDM embeds watermarks that are certifiable only from the user.
preserves many important features of standard diffusion models, such as simulation-free training and tractable conditional scores, which yields simple regression objectives; see Table 1. After learning the dual-space distribution, MDM applies the inverse mapping  $\nabla \phi^{*}$ , transforming mirrored samples back to the constrained set. We propose efficient construction of mirror maps and demonstrate that MDM outperforms prior reflected diffusion models on common constrained sets, such as  $\ell_{2}$ -balls and simplices.
Moreover, we show that MDM also stands as a novel approach for watermarked generation, a technique that aims to embed undetectable information (i.e., watermarks) into generated contents to prevent unethical usage and ensure copyright protection. This newly emerging goal nevertheless attracted significant attention in natural language generation [35-37], while what we will present is a general algorithmic strategy demonstrated for image generations.
To generate watermarked contents that are private to individual users, we begin by constructing a constrained set  $\mathcal{M}$  based on user-defined tokens (see Figure 2). MDM can then be instantiated to learn the dual-space distribution. Alternatively, MDM can distill pretrained diffusion models [38] by projecting the generated data onto the constraint set. In either case, MDM generates watermarked contents that can be certified only by users who know the tokens. Our empirical results suggest that MDM serves as a competitive approach to prior watermarking techniques for diffusion models, such as those proposed by Zhao et al. [39], by achieving lower (hence better) FID values.
In summary, we present the following contributions:
- We introduce Mirror Diffusion Models (MDM), a new class of diffusion models that utilizes mirror maps to enable constrained generation with Euclidean-space diffusion models.
- We propose efficient computation of mirror maps on common constrained sets, including balls and simplices. Our results demonstrate that MDM consistently outperforms previous reflected diffusion models.
- As a novel application of constrained generation, we explore MDM's potential for watermarking and copyrighting diffusion models where constraints serve as private tokens. We show that it yields competitive performance compared to prior methods.
2 Preliminary on Euclidean-space Diffusion Models
Notation  $\mathcal{M} \subseteq \mathbb{R}^d$  denotes a convex constrained set. We preserve  $x \in \mathcal{M}$  and  $y \in \mathbb{R}^d$  to better distinguish variables on constrained sets and standard  $d$ -dimensional Euclidean space.  $\pmb{I} \in \mathbb{R}^{d \times d}$  denotes the identity matrix.  $x_i$  denotes the  $i$ -th element of a vector  $x$ . Similarly,  $[A]_{ij}$  denotes the element of a matrix  $A$  at  $i$ -th row and  $j$ -th column.
Diffusion process Given  $y_0 \in \mathbb{R}^d$  sampled from some Euclidean-space distribution, conventional diffusion models define the diffusion process as a forward Markov chain with the joint distribution:
$$
q \left(y _ {1: T} \mid y _ {0}\right) = \prod_ {t = 1} ^ {T} q \left(y _ {t} \mid y _ {t - 1}\right), \quad q \left(y _ {t} \mid y _ {t - 1}\right) := \mathcal {N} \left(y _ {t}; \sqrt {1 - \beta_ {t}} y _ {t - 1}, \beta_ {t} I\right), \tag {1}
$$
which progressively injects Gaussian noise to  $y_0$  such that, for a sufficiently large  $T$  and a properly chosen noise schedule  $\beta_t \in (0,1)$ ,  $y_T$  approaches an approximate Gaussian, i.e.,  $q(y_T) \approx \mathcal{N}(0,\pmb{I})$ .
Equation (1) has tractable marginal densities, and two of which that are of particular interest are:
$$
q \left(y _ {t} \mid y _ {0}\right) = \mathcal {N} \left(y _ {t}; \sqrt {\bar {\alpha} _ {t}} y _ {0}, \left(1 - \alpha_ {t}\right) I\right), \quad q \left(y _ {t - 1} \mid y _ {t}, y _ {0}\right) = \mathcal {N} \left(y _ {t - 1}; \tilde {\mu} _ {t} \left(y _ {t}, y _ {0}\right), \tilde {\beta} _ {t} I\right), \tag {2}
$$
$$
\tilde {\mu} _ {t} (y _ {t}, y _ {0}) := \frac {\sqrt {\bar {\alpha} _ {t - 1}} \beta_ {t}}{1 - \bar {\alpha} _ {t}} y _ {0} + \frac {\sqrt {1 - \beta_ {t}} (1 - \bar {\alpha} _ {t - 1})}{1 - \bar {\alpha} _ {t}} y _ {t}, \quad \tilde {\beta} _ {t} := \frac {1 - \bar {\alpha} _ {t - 1}}{1 - \bar {\alpha} _ {t}} \beta_ {t}, \quad \bar {\alpha} _ {t} := \prod_ {s = 0} ^ {t} (1 - \beta_ {s}).
$$
The tractable marginal  $q(y_{t}|y_{0})$  enables direct sampling of  $y_{t}$  without simulating the forward Markov chain (1), i.e.,  $y_{t}|y_{0}$  can be sampled in a simulation-free manner. It also suggests a closed-form score function  $\nabla \log q(y_{t}|y_{0})$ . The marginal  $q(y_{t - 1}|y_t,y_0)$  hints the optimal reverse process given  $y_{0}$ .
Generative process The generative process, which aims to reverse the forward diffusion process (1), is defined by a backward Markov chain with the joint distribution:
$$
p _ {\theta} \left(y _ {0: T}\right) = p _ {T} \left(y _ {T}\right) \prod_ {t = 1} ^ {T} p _ {\theta} \left(y _ {t - 1} \mid y _ {t}\right), \quad p _ {\theta} \left(y _ {t - 1} \mid y _ {t}\right) := \mathcal {N} \left(y _ {t - 1}; \mu_ {\theta} \left(y _ {t}, t\right), \tilde {\beta} _ {t} I\right). \tag {3}
$$
Here, the mean  $\mu_{\theta}$  is typically parameterized by some DNNs, e.g., U-net [40], with  $\theta$ . Equations (1) and (3) imply an evidence lower bound (ELBO) for maximum likelihood training, indicating that the optimal prediction of  $\mu_{\theta}(y_t,t)$  should match  $\tilde{\mu}_t(y_t,y_0)$  in Equation (2).
Parameterization and training There exists many different ways to parameterize  $\mu_{\theta}(y_t, t)$ , and each of them leads to a distinct training objective. For instance, Ho et al. [2] found that
$$
\mu_ {\theta} \left(y _ {t}, t\right) := \frac {1}{\sqrt {1 - \beta_ {t}}} \left(y _ {t} - \frac {\beta_ {t}}{\sqrt {1 - \bar {\alpha} _ {t}}} \epsilon_ {\theta} \left(y _ {t}, t\right)\right), \tag {4}
$$
achieves better empirical results compared to directly predicting  $y_{0}$  or  $\tilde{\mu}_{t}$ . This parameterization aims to predict the "noise" injected to  $y_{t} \sim q(y_{t}|y_{0})$ , as the ELBO reduces to a simple regression:
$$
\mathcal {L} (\theta) := \mathbb {E} _ {t, y _ {0}, \epsilon} [ \lambda (t) \| \epsilon - \epsilon_ {\theta} (y _ {t}, t) ] ]. \tag {5}
$$
where  $y_{t} = \sqrt{\bar{\alpha}_{t}} y_{0} + \sqrt{1 - \alpha_{t}}\epsilon$ ,  $\epsilon \sim \mathcal{N}(0, I)$ , and  $t$  sampled uniformly from  $\{1, \dots, T\}$ . The weighting  $\lambda(t)$  is often set to 1. Once  $\epsilon_{\theta}$  is properly trained, we can generate samples in Euclidean space by simulating Equation (3) backward from  $y_{T} \sim \mathcal{N}(0, I)$ .
3 Mirror Diffusion Models (MDM)
In this section, we present a novel class of diffusion models, Mirror Diffusion Models, which are designed to model probability distributions  $p_{\mathrm{data}}(x)$  supported on a convex constrained set  $\mathcal{M} \subseteq \mathbb{R}^d$  while retraining the computational advantages of Euclidean-space diffusion models.
3.1 Dual-Space Diffusion
Mirror map Following the terminology in Li et al. [34], let  $\phi : \mathcal{M} \to \mathbb{R}$  be a twice differentiable<sup>3</sup> function that is strictly convex and satisfying that  $\lim_{x \to \partial \mathcal{M}} \| \nabla \phi(x) \| \to \infty$  and  $\nabla \phi(\mathcal{M}) = \mathbb{R}^d$ . We call its gradient map  $\nabla \phi : \mathcal{M} \to \mathbb{R}^d$  the mirror map and its image,  $\nabla \phi(\mathcal{M}) = \mathbb{R}^d$ , as its dual/mirror space. Let  $\phi^* : \mathbb{R}^d \to \mathbb{R}$  be the dual function of  $\phi$  defined by
$$
\phi^ {*} (y) = \sup  _ {x \in \mathcal {M}} \langle x, y \rangle - \phi (x) \tag {6}
$$
A notable property of this dual function  $\phi^{*}(y)$  is that its gradient map reverses the mirror map. In other words, we have  $\nabla \phi^{*} = (\nabla \phi)^{-1}$ , implying that
$$
\nabla \phi^ {*} (\nabla \phi (x)) = x \text {f o r a l l} x \in \mathcal {M}, \quad \nabla \phi (\nabla \phi^ {*} (y)) = y \text {f o r a l l} y \in \nabla \phi (\mathcal {M}) = \mathbb {R} ^ {d}.
$$
The functions  $(\nabla \phi, \nabla \phi^{*})$  define a nonlinear, yet bijective, mapping between the convex constrained set  $\mathcal{M}$  and Euclidean space  $\mathbb{R}^d$ . As a result, MDM does not require building diffusion models on  $\mathcal{M}$ , as done in previous approaches, but rather they learn a standard Euclidean-space diffusion model in the corresponding dual space  $\nabla \phi(\mathcal{M}) = \mathbb{R}^d$ .
<sup>3</sup>We note that standard mirror maps require twice differentiability to induce a (Riemannian) metric on the mirror space for constructing MLD. For our MDM, however, twice differentiability is needed only for Equation (8), and continuous differentiability, i.e.,  $C^1$ , would suffice for training.
Figure 3: Examples of how mirror maps  $\nabla \phi$  pushforward constrained distributions to unconstrained ones, for (left) an  $\ell_2$ -ball where  $\mathcal{M} := \{x \in \mathbb{R}^2 : \| x \|_2^2 < R\}$  and (right) a simplex  $\Delta_3$  where  $\mathcal{M} := \{x \in \mathbb{R}^2 : \sum_{i=1}^{2} x_i \leq 1, x_i \geq 0\}$ . Note that for simplex,  $x_3 = 1 - x_1 - x_2$  is a redundant coordinate; see Section 3.2 for more details.
Essentially, the goal of MDM is to model the dual-space distribution:
$$
\tilde {p} _ {\text {d a t a}} (y) := \left(\left[ \nabla \phi_ {\sharp} \right] p _ {\text {d a t a}}\right) (x), \tag {7}
$$
where  $\sharp$  is the push-forward operation and  $y = \nabla \phi (x)$  is the mirror of data  $x\in \mathcal{M}$ .
Training and generation MDM follows the same training procedure of Euclidean-space diffusion models, except with an additional push-forward given by Equation (7). Similarly, its generation procedure includes an additional step that maps the dual-space samples  $y_0$ , generated from Equation (3), back to the constrained set via  $\nabla \phi^{*}(y_{0})\in \mathcal{M}$
Tractable variational bound Given a datapoint  $x \in \mathcal{M}$ , its mirror  $y_0 = \nabla \phi(x) \in \mathbb{R}^d$ , and the dual-space Markov chain  $\{y_1, \dots, y_T\}$ , the ELBO of MDM can be computed by
$$
L _ {\mathrm {E L B O}} (x) := \log | \det \nabla^ {2} \phi^ {*} (y _ {0}) | + \tilde {L} _ {\mathrm {E L B O}} (y _ {0}), \tag {8}
$$
where we apply the change of variables theorem [41] w.r.t. the mapping  $x = \nabla \phi^{*}(y_{0})$ . The dual-space ELBO  $\tilde{L}_{\mathrm{ELBO}}(y_0)$  shares the same formula with the Euclidean-space ELBO [42, 43],
$$
\tilde {L} _ {\mathrm {E L B O}} \left(y _ {0}\right) := D _ {\mathrm {K L}} \left(q \left(y _ {T} \mid y _ {0}\right) \mid p \left(y _ {T}\right)\right) + \sum_ {t = 1} ^ {T - 1} D _ {\mathrm {K L}} \left(q \left(y _ {t} \mid y _ {t + 1}, y _ {0}\right) \mid p _ {\theta} \left(y _ {t} \mid y _ {t + 1}\right)\right) - \log p _ {\theta} \left(y _ {0} \mid y _ {1}\right). \tag {9}
$$
It should be noted that (8,9) provide a tractable variational bound for constrained generation. This stands in contrast to concurrent methods relying on reflected diffusions, whose ELBO entails either intractable [18] or approximate [25] scores that could be restricted when dealing with more complex constrained sets. While, in practice, MDM is trained using the regression objective in Equation (5), computing tractable ELBO might be valuable for independent purposes such as likelihood estimation.
3.2 Efficient Computation of Mirror Maps
What remains to be answered pertains to the construction of mirror maps for common constraint sets such as  $\ell_2$ -balls, simplices, polytopes, and their products. In particular, we seek efficient, preferably closed-form, computation for  $\nabla \phi$ ,  $\nabla \phi^*$  and  $\nabla^2\phi^*$ .
$\ell_2$ -Ball Let  $\mathcal{M} \coloneqq \{x \in \mathbb{R}^d : \| x \|_2^2 < R\}$  denote the  $\ell_2$ -ball of radius  $R$  in  $\mathbb{R}^d$ . We consider the common log-barrier function:
$$
\phi_ {\text {b a l l}} (x) := - \gamma \log \left(R - \| x \| _ {2} ^ {2}\right). \tag {10}
$$
where  $\gamma \in \mathbb{R}^{+}$ . The mirror map, its inverse, and the Hessian can be computed analytically by
$$
\nabla \phi_ {\mathrm {b a l l}} (x) = \frac {2 \gamma x}{R - \| x \| _ {2} ^ {2}}, \qquad \nabla \phi_ {\mathrm {b a l l}} ^ {*} (y) = \frac {R y}{\sqrt {R \| y \| _ {2} ^ {2} + \gamma^ {2}} + \gamma},
$$
$$
\nabla^ {2} \phi_ {\text {b a l l}} ^ {*} (y) = \frac {R}{R \sqrt {\| y \| _ {2} ^ {2} + \gamma^ {2}} + \gamma} \left(\boldsymbol {I} - \frac {R}{\left(R \sqrt {\| y \| _ {2} ^ {2} + \gamma^ {2}} + \gamma\right) \sqrt {R \| y \| _ {2} ^ {2} + \gamma^ {2}}} y y ^ {\top}\right). \tag {11}
$$
We refer to Appendix A for detailed derivation. Figure 3 visualizes such a mirror map for an  $\ell_2$ -ball in 2D. It is worth noting that while the mirror map of log-barriers generally does not admit an analytical inverse, this is not the case for the  $\ell_2$ -ball constraints.
Simplex Given a  $(d + 1)$ -dimensional simplex,  $\Delta_{d + 1} \coloneqq \{x \in \mathbb{R}^{d + 1} : \sum_{i = 1}^{d + 1} x_i = 1, x_i \geq 0\}$ , we follow standard practices [18, 25] by constructing the constrained set  $\mathcal{M} \coloneqq \{x \in \mathbb{R}^d : \sum_{i = 1}^d x_i \leq 1, x_i \geq 0\}$  and define  $x_{d + 1} \coloneqq 1 - \sum_{i = 1}^d x_i$ .<sup>4</sup>
While conventional log-barriers may remain a viable option, a preferred alternative that is widely embraced in the MLD literature for enforcing simplices [28, 44] is the entropic function [45, 46]:
$$
\phi_ {\text {s i m p l e x}} (x) := \sum_ {i = 1} ^ {d} x _ {i} \log \left(x _ {i}\right) + \left(1 - \sum_ {i = 1} ^ {d} x _ {i}\right) \log \left(1 - \sum_ {i = 1} ^ {d} x _ {i}\right). \tag {12}
$$
The mirror map, its inverse, and the hessian of the dual function can be computed analytically by
$$
\left[ \nabla \phi_ {\text {s i m p l e x}} (x) \right] _ {i} = \log x _ {i} - \log \left(1 - \sum_ {i = 1} ^ {d} x _ {i}\right), \quad \left[ \nabla \phi_ {\text {s i m p l e x}} ^ {*} (y) \right] _ {i} = \frac {e ^ {y _ {i}}}{1 + \sum_ {i = 1} ^ {d} e ^ {y _ {i}}}, \tag {13}
$$
$$
\left[ \nabla^ {2} \phi_ {\mathrm {s i m p l e x}} ^ {*} (y) \right] _ {i j} = \frac {e ^ {y i}}{1 + \sum_ {i = 1} ^ {d} e ^ {y _ {i}}} \mathbb {I} (i = j) - \frac {e ^ {y _ {i}} e ^ {y _ {j}}}{\left(1 + \sum_ {i = 1} ^ {d} e ^ {y _ {i}}\right) ^ {2}},
$$
where  $\mathbb{I}(\cdot)$  is an indicator function. Again, we leave the derivation of Equation (13) to Appendix A. Figure 3 visualizes the entropic mirror map for a 3-dimensional simplex  $\Delta_3$ .
Polytope Let the constrained set be  $\mathcal{M} \coloneqq \{x \in \mathbb{R}^d : c_i < a_i^\top x < b_i, \forall i \in \{1, \dots, m\}\}$ , where  $c_i, b_i \in \mathbb{R}$  and  $a_i \in \mathbb{R}^d$  are linearly independent to each other. We consider the standard log-barrier:
$$
\phi_ {\text {p o l y t o p e}} (x) := - \sum_ {i = 1} ^ {m} \left(\log \left(\left\langle a _ {i}, x \right\rangle - c _ {i}\right) + \log \left(b _ {i} - \left\langle a _ {i}, x \right\rangle\right)\right) + \sum_ {j = m + 1} ^ {d} \frac {1}{2} \left\langle a _ {j}, x \right\rangle^ {2}, \tag {14}
$$
$$
\nabla \phi_ {\text {p o l y t o p e}} (x) = \sum_ {i = 1} ^ {m} s _ {i} (\langle a _ {i}, x \rangle) a _ {i} + \sum_ {j = m + 1} ^ {d} \langle a _ {j}, x \rangle a _ {j}, \tag {15}
$$
where the monotonic function  $s_i:(c_i,b_i)\to \mathbb{R}$  is given by  $s_i = \frac{-1}{\langle a_i,x\rangle - c_i} +\frac{-1}{\langle a_i,x\rangle - b_i}$ , and  $\{a_j\}$  do not impose any constraints and can be chosen arbitrarily as long as they span  $\mathbb{R}^d$  with  $\{a_i\}$  (see below for more details); hence  $\nabla \phi_{\mathrm{polytope}}(\mathcal{M}) = \mathbb{R}^d$ . While its inverse,  $\nabla \phi_{\mathrm{polytope}}^*$ , does not admit closed-form in general, it does when all  $d$  constraints are orthonormal:
$$
\nabla \phi_ {\text {p o l y t o p e}} ^ {*} (y) = \sum_ {i = 1} ^ {m} s _ {i} ^ {- 1} (\langle a _ {i}, y \rangle) a _ {i} + \sum_ {j = m + 1} ^ {d} \langle a _ {j}, y \rangle a _ {j}. \tag {16}
$$
Figure 4: Comparison between  $s_i$  induced by standard log-barriers vs. hyperbolic tangents in Eq. (18).
Here,  $s_i^{-1} : \mathbb{R} \to (c_i, b_i)$  is the inverse of  $s_i$ . Essentially, (15,16) manipulate the coefficients of the bases from which the polytope set is constructed. These manipulations are nonlinear yet bijective, defined uniquely by  $s_i$  and  $s_i^{-1}$ , for each orthonormal basis  $a_i$  or, equivalently, for each constraint.