SlowGuess/NeurIPS_2023_OCR huggingface dataset:
$$
\boldsymbol {J} _ {L} = \left[ \begin{array}{c c c c c c c c c c c} u _ {2} & u _ {3} & u _ {4} & & & & & & & & \\ & & & u _ {1} & u _ {3} & u _ {4} & & & & & \\ & & & & & & u _ {1} & u _ {2} & u _ {4} & & \\ & & & & & & & & & u _ {1} & u _ {2} & u _ {3} \end{array} \right].
$$
It is crucial to notice that the  $i$ th row does not include  $u_{i}$ . This means that, the matrix  $\pmb{J}_s^\top \pmb{J}_L$  has entries that are either 0, or  $\phi'(s_i) u_i u_j$  for  $i \neq j$ , which is  $\mathbb{E} \phi'(s_i) u_i u_j = 0$  by Assumption 1. Therefore,
$$
\mathbb {E} J _ {s} ^ {\top} J _ {L} = \mathbf {0}.
$$
Finally, the elements of  $\pmb{J}_L^\top \pmb{J}_L$  are
$$
\mathbb {E} \sum_ {i = 0} ^ {d} \frac {\partial \mathcal {T} _ {i} (\boldsymbol {\lambda} ; \boldsymbol {u})}{\partial L _ {j k}} \frac {\partial \mathcal {T} _ {i} (\boldsymbol {\lambda} ; \boldsymbol {u})}{\partial L _ {l m}} = \mathbb {E} \sum_ {i = 0} ^ {d} u _ {k} u _ {m} \mathbb {1} _ {i = j} \mathbb {1} _ {i = l} = \mathbb {1} _ {j = l} (\mathbb {E} u _ {k} u _ {m}) = \mathbb {1} _ {j = l} \mathbb {1} _ {k = m},
$$
where the last equality follows from Assumption 1, which forms an identity matrix as
$$
\mathbb {E} J _ {L} ^ {\top} J _ {L} = \mathbf {I}.
$$
Therefore, the expected-squared Jacobian is now
$$
\begin{array}{l} \mathbb {E} \left(\frac {\partial \mathcal {T} _ {\lambda} (\boldsymbol {u})}{\partial \lambda}\right) ^ {\top} \frac {\partial \mathcal {T} _ {\lambda} (\boldsymbol {u})}{\partial \lambda} = \left[ \begin{array}{c c c} \mathbf {I} & \mathbb {E} \boldsymbol {J} _ {s} & \mathbb {E} \boldsymbol {J} _ {L} \\ \mathbb {E} \boldsymbol {J} _ {s} ^ {\top} & \mathbb {E} \boldsymbol {J} _ {s} ^ {\top} \boldsymbol {J} _ {s} & \mathbb {E} \boldsymbol {J} _ {s} ^ {\top} \boldsymbol {J} _ {L} \\ \mathbb {E} \boldsymbol {J} _ {L} ^ {\top} & \mathbb {E} \boldsymbol {J} _ {L} ^ {\top} \boldsymbol {J} _ {s} & \mathbb {E} \boldsymbol {J} _ {L} ^ {\top} \boldsymbol {J} _ {L} \end{array} \right] \\ = \left[ \begin{array}{c c c} \mathbf {I} & & \\ & \operatorname {d i a g} \left(\boldsymbol {\phi} (\boldsymbol {s})\right) ^ {2} & \\ & & \mathbf {I} \end{array} \right], \\ \end{array}
$$
which, conveniently, is a diagonal matrix. The maximal singular value of a block-diagonal matrix is the maximal singular value of each block. And since each block is diagonal with only positive entries, the largest element forms the maximal singular value. As we assume that  $\phi$  is 1-Lipchitz, the element of all blocks is lower-bounded by 0 and upper-bounded by 1. Therefore, the maximal singular value of the expected-squared Jacobian is bounded by 1.
By the Leibniz and chain rule, the Hessian of the energy  $f$  follows as
$$
\begin{array}{l} \nabla^ {2} f (\boldsymbol {\lambda}) = \mathbb {E} \nabla_ {\boldsymbol {\lambda}} ^ {2} \ell \left(\mathcal {T} _ {\boldsymbol {\lambda}} (\boldsymbol {u})\right) \\ = \underbrace {\mathbb {E} \left(\frac {\partial \mathcal {T} _ {\lambda} (\boldsymbol {u})}{\partial \lambda}\right) ^ {\top} \nabla^ {2} \ell \left(\mathcal {T} _ {\lambda} (\boldsymbol {u})\right) \frac {\partial \mathcal {T} _ {\lambda} (\boldsymbol {u})}{\partial \lambda}} _ {\triangleq T _ {\mathrm {l i n}}} + \underbrace {\mathbb {E} \sum_ {i = 1} ^ {d} \mathrm {D} _ {i} \ell \left(\mathcal {T} _ {\lambda} (\boldsymbol {u})\right) \frac {\partial^ {2} \mathcal {T} _ {i} (\boldsymbol {\lambda} ; \boldsymbol {u})}{\partial \boldsymbol {\lambda} ^ {2}}} _ {\triangleq T _ {\mathrm {n o n}}}. \\ \end{array}
$$
When  $\mathcal{T}$  is linear with respect to  $\lambda$ , it is clear that we have
$$
\frac {\partial^ {2} \mathcal {F} _ {i} (\lambda ; \boldsymbol {u})}{\partial \lambda^ {2}} = \boldsymbol {0}. \tag {5}
$$
Then,  $T_{\mathrm{non}}$  is zero. In contrast,  $T_{\mathrm{lin}}$  appears for both the linear and nonlinear cases. Therefore,  $T_{\mathrm{non}}$  fully characterizes the effect of nonlinearity in the reparameterization function.
Now, the triangle inequality yields
$$
\| \nabla^ {2} f (\lambda) \| _ {2} = \| T _ {\mathrm {l i n}} + T _ {\mathrm {n o n}} \| _ {2} \leq \| T _ {\mathrm {l i n}} \| _ {2} + \| T _ {\mathrm {n o n}} \| _ {2},
$$
where equality is achieved when either term is 0. On the contrary, the reverse triangle inequality states that
$$
\left| \left\| T _ {\mathrm {l i n}} \right\| _ {2} - \left\| T _ {\mathrm {n o n}} \right\| _ {2} \right| \leq \| \nabla^ {2} f (\lambda) \| _ {2}.
$$
This implies that, if either  $T_{\mathrm{lin}}$  or  $T_{\mathrm{non}}$  is unbounded, the Hessian is not bounded. Thus, ensuring that  $T_{\mathrm{lin}}$  and  $T_{\mathrm{non}}$  are bounded is sufficient and necessary to establish that  $f$  is smooth.
Proof of (i) The bound on the linear part,  $T_{\mathrm{lin}}$ , follows from Lemma 2 as
$$
\begin{array}{l} \left\| T _ {\mathrm {l i n}} \right\| _ {2} = \left\| \mathbb {E} \left(\frac {\partial \mathcal {T} _ {\lambda} (\boldsymbol {u})}{\partial \lambda}\right) ^ {\top} \nabla^ {2} \ell \left(\mathcal {T} _ {\lambda} (\boldsymbol {u})\right) \frac {\partial \mathcal {T} _ {\lambda} (\boldsymbol {u})}{\partial \lambda} \right\| _ {2} \\ \leq L _ {\ell} \left\| \mathbb {E} \left(\frac {\partial \mathcal {T} _ {\lambda} (\boldsymbol {u})}{\partial \lambda}\right) ^ {\top} \frac {\partial \mathcal {T} _ {\lambda} (\boldsymbol {u})}{\partial \lambda} \right\| _ {2}, \\ \end{array}
$$
and from the 1-Lipschitzness of  $\phi$ , Lemma 9 yields
$$
\leq L _ {\ell}.
$$
When  $\phi$  is linear, it immediately follows from Equation (5) that
$$
\left\| \nabla^ {2} f (\lambda) \right\| _ {2} = \left\| T _ {\mathrm {l i n}} \right\| _ {2} \leq L _ {\ell},
$$
which is tight as shown by Domke (2020, Theorem 6).
Proof of (ii) For the nonlinear part  $T_{\mathrm{non}}$ , we use the fact that  $\mathcal{T}_i(\lambda ;\pmb {u})$  is given as
$$
\mathcal {T} _ {i} (\boldsymbol {\lambda}; \boldsymbol {u}) = m _ {i} + \phi (s _ {i}) u _ {i} + \sum_ {j \neq i} L _ {i j} u _ {j}.
$$
The second derivative of  $\mathcal{F}_i$  is clearly non-zero only for the nonlinear part involving  $s_1, \ldots, s_d$ . Thus,  $T_{\mathrm{non}}$  follows as
$$
\begin{array}{l} T _ {\mathrm {n o n}} = \mathbb {E} \sum_ {i = 1} ^ {d} \mathrm {D} _ {i} \ell \left(\mathcal {F} _ {\lambda} (\boldsymbol {u})\right) \frac {\partial^ {2} \mathcal {F} _ {i} (\boldsymbol {\lambda} ; \boldsymbol {u})}{\partial \lambda^ {2}} \\ = \mathbb {E} \sum_ {i = 1} ^ {d} g _ {i} (\boldsymbol {\lambda}; \boldsymbol {u}) \frac {\partial^ {2} \mathcal {T} _ {i} (\boldsymbol {\lambda} ; \boldsymbol {u})}{\partial \boldsymbol {\lambda} ^ {2}} \\ = \left[ \begin{array}{c c c} \cdot & \cdot & \cdot \\ \cdot & \mathbb {E} \sum_ {i = 1} ^ {d} g _ {i} (\boldsymbol {\lambda}; \boldsymbol {u}) \frac {\partial^ {2} \mathcal {T} _ {i} (\boldsymbol {\lambda} ; \boldsymbol {u})}{\partial s ^ {2}} & \cdot \\ \cdot & \cdot & \cdot \end{array} \right]. \\ \end{array}
$$
Furthermore, the second-order derivatives with respect to  $s_1, \ldots, s_d$  are given as
$$
\frac {\partial^ {2} \mathcal {T} _ {i} (\boldsymbol {\lambda} ; \boldsymbol {u})}{\partial s _ {j} ^ {2}} = \mathbb {1} _ {i = j} \phi^ {\prime \prime} (s _ {j}).
$$
Considering this, the only non-zero block of  $T_{\mathrm{non}}$  forms a diagonal matrix as
$$
\begin{array}{l} \mathbb {E} \sum_ {i = 1} ^ {d} g _ {i} (\boldsymbol {\lambda}; \boldsymbol {u}) \frac {\partial^ {2} \mathcal {F} _ {i} (\boldsymbol {\lambda} ; \boldsymbol {u})}{\partial \boldsymbol {s}} = \left[ \begin{array}{c c c} \mathbb {E} g _ {1} (\boldsymbol {\lambda}; \boldsymbol {u}) \frac {\partial^ {2} \mathcal {F} _ {1} (\boldsymbol {\lambda} ; \boldsymbol {u})}{\partial s _ {1} ^ {2}} & & \\ & \ddots & \\ & & \mathbb {E} g _ {d} (\boldsymbol {\lambda}; \boldsymbol {u}) \frac {\partial^ {2} \mathcal {F} _ {d} (\boldsymbol {\lambda} ; \boldsymbol {u})}{\partial s _ {d} ^ {2}} \end{array} \right] \\ = \left[ \begin{array}{c c c} \mathbb {E} g _ {1} \left(\boldsymbol {\lambda}; \boldsymbol {u}\right) \phi^ {\prime \prime} \left(s _ {1}\right) u _ {1} & & \\ & \ddots & \\ & & \mathbb {E} g _ {d} \left(\boldsymbol {\lambda}; \boldsymbol {u}\right) \phi^ {\prime \prime} \left(s _ {d}\right) u _ {d} \end{array} \right] \\ \end{array}
$$
This implies that the only non-zero entries of  $T_{\mathrm{non}}$  lie on its diagonal. Since the  $\ell_2$  norm of a diagonal matrix is the value of the maximal diagonal element,
$$
\left\| T _ {\text {n o n}} \right\| _ {2} \leq \max  _ {i = 1, \dots , d} \mathbb {E} g _ {i} (\boldsymbol {\lambda}; \boldsymbol {u}) \phi^ {\prime \prime} (s _ {i}) u _ {i} \leq L _ {s},
$$
where  $L_{s}$  is finite constant if Assumption 3 holds. On the contrary, if a finite  $L_{s}$  does not exist,  $\| T_{\mathrm{non}}\| _2$  cannot be bounded. Therefore, the energy is smooth if and only if Assumption 3 holds. When it does, the energy  $f$  is  $L_{f} + L_{s}$  smooth.
Example 2. Let  $\ell(z) = (1/2)z^{\top}Az$  and the diagonal conditioner be  $\phi(x) = \text{softplus}(x)$ . Then,
(i) if  $\mathbf{A}$  is dense and the variational family is the mean-field family or
(ii) if  $\mathbf{A}$  is diagonal and the variational family is the Cholesky family,
Assumption 3 holds with  $L_{s}\approx 0.26034\left(\max_{i = 1,\dots ,d}A_{ii}\right)$
Proof. Since the gradient is
$$
\nabla \ell (z) = A z,
$$
combined with reparameterization, we have
$$
g (\lambda ; \boldsymbol {u}) = \boldsymbol {A} (\boldsymbol {C u} + \boldsymbol {m})
$$
Then, for each coordinate  $i = 1,\dots ,d$  we have
$$
\begin{array}{l} \mathbb {E} g _ {i} (\boldsymbol {\lambda}; \boldsymbol {u}) u _ {i} \phi^ {\prime \prime} (s _ {i}) = \mathbb {E} \left(\sum_ {j} \sum_ {k \leq j} A _ {i j} C _ {j k} u _ {k} + \sum_ {j} A _ {i j} m _ {j}\right) u _ {i} \phi^ {\prime \prime} (s _ {i}) \\ = \sum_ {j} \sum_ {k \leq j} A _ {i j} C _ {j k} \mathbb {E} u _ {k} u _ {i} \phi^ {\prime \prime} (s _ {i}) + \sum_ {j} A _ {i j} m _ {j} \mathbb {E} u _ {i} \phi^ {\prime \prime} (s _ {i}), \\ \end{array}
$$
and from Assumption 1,
$$
\begin{array}{l} = \phi^ {\prime \prime} (s _ {i}) \sum_ {j} \sum_ {k \leq j} A _ {i j} C _ {j k} \mathbb {I} _ {k = i} \\ = \phi^ {\prime \prime} (s _ {i}) \sum_ {j} A _ {i j} C _ {j i}. \\ \end{array}
$$
Furthermore, the diagonal of  $C$  involves  $\phi$  such that
$$
\mathbb {E} g _ {i} \left(\boldsymbol {\lambda}; \boldsymbol {u}\right) u _ {i} \phi^ {\prime \prime} (s _ {i}) = \underbrace {A _ {i i} \phi \left(s _ {i}\right) \phi^ {\prime \prime} \left(s _ {i}\right)} _ {T _ {\mathrm {d i a g}}} + \underbrace {\sum_ {j <   i} A _ {i j} C _ {j i} \phi^ {\prime \prime} (s _ {i})} _ {T _ {\mathrm {o f f}}}.
$$
For the softplus function, we have
$$
0 <   \phi^ {\prime \prime} (s) <   1
$$
for any finite  $s$ , and we have
$$
\sup  _ {s} \phi (s) \phi^ {\prime \prime} (s) \approx 0. 2 6 0 3 4,
$$
where the supremum was numerically approximated. Then, it is clear that  $T_{\mathrm{diag}}$  is finite as long as the diagonals of  $A$  are finite. Furthermore, we have the following:
(i) If  $\mathbf{A}$  is diagonal, then  $T_{\mathrm{off}}$  is 0.
(ii) If  $\mathbf{A}$  is dense but  $\mathbf{C}$  is diagonal due to the use of the mean-field family,  $T_{\mathrm{off}}$  is again 0.
(iii) However, when both  $\mathbf{A}$  and  $\mathbf{C}$  are not diagonal,  $T_{\mathrm{off}}$  can be made arbitrarily large.
F.2.2 Convexity
Lemma 10. Let  $\ell$  be convex. Then, for a convex nonlinear  $\phi$ , the inequality
$$
\langle \nabla_ {\lambda} f (\lambda), \lambda - \lambda^ {\prime} \rangle \leq \mathbb {E} \langle \nabla g (\lambda ; \boldsymbol {u}), \mathcal {T} _ {\lambda} (\boldsymbol {u}) - \mathcal {T} _ {\lambda^ {\prime}} (\boldsymbol {u}) \rangle
$$
holds for all  $\lambda \in \Lambda$  if and only if Assumption 4 holds. For the linear parameterization, the inequality becomes equality.
Proof. First, notice that the left-hand side is
$$
\left\langle \nabla_ {\lambda} f (\lambda), \lambda - \lambda^ {\prime} \right\rangle = \sum_ {i = 1} ^ {p} \left\langle \nabla_ {\lambda} \mathbb {E} \ell \left(\mathcal {T} _ {\lambda} (\boldsymbol {u})\right), \lambda - \lambda^ {\prime} \right\rangle = \mathbb {E} \sum_ {i = 1} ^ {p} \left\langle \left(\frac {\partial \mathcal {T} _ {\lambda} (\boldsymbol {u})}{\partial \lambda_ {i}}\right) g (\lambda ; \boldsymbol {u}), \lambda - \lambda^ {\prime} \right\rangle .
$$
By restricting us to the location-scale family, we then get
$$
\begin{array}{l} = \mathbb {E} \left(\underbrace {\sum_ {i} \left(\frac {\partial \mathcal {T} _ {\lambda} (\boldsymbol {u})}{\partial m _ {i}}\right) g \left(\lambda ; \boldsymbol {u}\right) \left(m _ {i} - m _ {i} ^ {\prime}\right)} _ {\text {c o n v e x i t y w i t h r e s p e c t t o} \boldsymbol {m}} + \underbrace {\sum_ {i j} \left(\frac {\partial \mathcal {T} _ {\lambda} (\boldsymbol {u})}{\partial L _ {i j}}\right) g \left(\lambda ; \boldsymbol {u}\right) \left(L _ {i j} - L _ {i j} ^ {\prime}\right)} _ {\text {c o n v e x i t y w i t h r e s p e c t t o} \boldsymbol {L}}\right) \\ + \underbrace {\sum_ {i} \mathbb {E} \left(\frac {\partial \mathcal {T} _ {\boldsymbol {\lambda}} (\boldsymbol {u})}{\partial s _ {i}}\right) g \left(\boldsymbol {\lambda} ; \boldsymbol {u}\right) \left(s _ {i} - s _ {i} ^ {\prime}\right)} _ {\text {c o n v e x i t y w i t h r e s p e c t t o s}}, \\ \end{array}
$$
and plugging the derivatives of the reparameterization function,
$$
= \mathbb {E} \left(\sum_ {i} g _ {i} (\boldsymbol {\lambda}; \boldsymbol {u}) (m _ {i} - m _ {i} ^ {\prime}) + \sum_ {i} \sum_ {j <   i} u _ {j} g _ {i} (\boldsymbol {\lambda}; \boldsymbol {u}) (L _ {i j} - L _ {i j} ^ {\prime}) + \sum_ {i} \phi^ {\prime} (s _ {i}) u _ {i} g _ {i} (\boldsymbol {\lambda}; \boldsymbol {u}) (s _ {i} - s _ {i} ^ {\prime})\right).
$$
On the other hand, the right-hand side follows as
$$
\begin{array}{l} \mathbb {E} \left\langle \nabla g (\boldsymbol {\lambda}; \boldsymbol {u}), \mathcal {T} _ {\boldsymbol {\lambda}} (\boldsymbol {u}) - \mathcal {T} _ {\boldsymbol {\lambda} ^ {\prime}} (\boldsymbol {u}) \right\rangle \\ = \mathbb {E} \left(\left\langle g (\lambda ; \boldsymbol {u}), \boldsymbol {m} - \boldsymbol {m} ^ {\prime} \right\rangle + \left\langle g (\lambda ; \boldsymbol {u}), \left(\boldsymbol {L} - \boldsymbol {L} ^ {\prime}\right) \boldsymbol {u} \right\rangle + \left\langle g (\lambda ; \boldsymbol {u}), \left(\boldsymbol {\Phi} (\boldsymbol {s}) - \boldsymbol {\Phi} (\boldsymbol {s} ^ {\prime})\right) \boldsymbol {u} \right\rangle\right) \\ = \mathbb {E} \left(\sum_ {i} g _ {i} (\boldsymbol {\lambda}; \boldsymbol {u}) (m _ {i} - m _ {i} ^ {\prime}) + \sum_ {i} \sum_ {j <   i} u _ {j} g _ {i} (\boldsymbol {\lambda}; \boldsymbol {u}) (L _ {i j} - L _ {i j} ^ {\prime}) + \sum_ {i} g _ {i} (\boldsymbol {\lambda}; \boldsymbol {u}) u _ {i} (\phi (s _ {i}) - \phi (s _ {i} ^ {\prime}))\right). \\ \end{array}
$$
The convexity with respect to the  $m$  and  $L$  is clear from the first two terms; they are equal. The statement is now up to the last term. That is, the statement holds if
$$
\mathbb {E} \sum_ {i} g _ {i} (\boldsymbol {\lambda}; \boldsymbol {u}) u _ {i} \phi^ {\prime} (s _ {i}) \left(s _ {i} - s _ {i} ^ {\prime}\right) \leq \mathbb {E} \sum_ {i} g _ {i} (\boldsymbol {\lambda}; \boldsymbol {u}) u _ {i} \left(\phi \left(s _ {i}\right) - \phi \left(s _ {i} ^ {\prime}\right)\right). \tag {6}
$$
For this, we will show that Assumption 4 is both necessary and sufficient.
Proof of sufficiency Equation (6) holds if
$$
\mathbb {E} g _ {i} (\lambda ; \boldsymbol {u}) u _ {i} \geq 0
$$
for all  $i = 1, \dots, d$ , which is non other than Assumption 4.
Proof of necessity Suppose that the inequality
$$
\langle \nabla_ {\lambda} f (\lambda), \lambda - \lambda^ {\prime} \rangle \leq \mathbb {E} \langle g (\lambda ; \boldsymbol {u}), \mathcal {T} _ {\lambda} (\boldsymbol {u}) - \mathcal {T} _ {\lambda^ {\prime}} (\boldsymbol {u}) \rangle
$$
holds for all  $\lambda \in \Lambda$ , implying
$$
\sum_ {i} \mathbb {E} u _ {i} g _ {i} (\boldsymbol {\lambda}; \boldsymbol {u}) \phi^ {\prime} (s _ {i}) (s _ {i} - s _ {i} ^ {\prime}) \leq \sum_ {i} \mathbb {E} u _ {i} g _ {i} (\boldsymbol {\lambda}; \boldsymbol {u}) ((\phi (s _ {i}) - \phi (s _ {i} ^ {\prime})).
$$
For any  $\lambda$ , we are free to set any  $\lambda' \in \Lambda$  and check whether we can retrieve Assumption 4 for this specific  $\lambda$ . Now, for each axis  $i$ , set  $s_j' = s_j$  for all  $j \neq i$ , then
$$
\mathbb {E} u _ {i} g _ {i} (\boldsymbol {\lambda}; \boldsymbol {u}) \phi^ {\prime} (s _ {i}) \left(s _ {i} - s _ {i} ^ {\prime}\right) \leq \mathbb {E} u _ {i} g _ {i} (\boldsymbol {\lambda}; \boldsymbol {u}) \left(\phi \left(s _ {i}\right) - \phi \left(s _ {i} ^ {\prime}\right)\right).
$$
Since  $\phi$  is assumed to be convex such that
$$
\phi^ {\prime} \left(s _ {i}\right) \left(s _ {i} - s _ {i} ^ {\prime}\right) \leq \phi \left(s _ {i}\right) - \phi \left(s _ {i} ^ {\prime}\right).
$$
$$
\mathbb {E} u _ {i} g _ {i} (\lambda ; \boldsymbol {u}) \geq 0. \tag {7}
$$
Therefore, for any  $\lambda \in \Lambda$  it must be that Assumption 4 holds.
Proof. For (i), the key property is the monotonicity of the gradient.
Proof of (i) For the mean-field family, recall that
$$
C _ {i i} = \phi (s _ {i}).
$$
Also, observe that
$$
\pmb {C} \pmb {u} + \pmb {m} = (C _ {1 1} u _ {1} + m _ {1}, \dots , C _ {d d} u _ {d} + m _ {d}).
$$
By the property of convex functions,  $\nabla \ell$  is monotone such that
$$
\langle \nabla \ell (\boldsymbol {z}) - \nabla \ell (\boldsymbol {z} ^ {\prime}), \boldsymbol {z} - \boldsymbol {z} ^ {\prime} \rangle \geq 0.
$$
Now, by setting  $\pmb{z} = \pmb{C}\pmb{u} + \pmb{m}$  and  $\pmb{z}' = \pmb{C}\pmb{u} + \pmb{m} - C_{ii}u_i\pmb{e}_i$ , we obtain
$$
\langle \nabla \ell (\boldsymbol {C} \boldsymbol {u} + \boldsymbol {m}) - \nabla \ell (\boldsymbol {C} \boldsymbol {u} + \boldsymbol {m} - C _ {i i} u _ {i} \mathbf {e} _ {i}), C _ {i i} u _ {i} \mathbf {e} _ {i} \rangle \geq 0
$$
for every  $i = 1,\ldots ,d$
For the mean-field family,  $\pmb{C}\pmb{u} + \pmb{m} - C_{ii}u_i\pmb{e}_i$  is now independent of  $u_{i}$ . Thus,
$$
\begin{array}{l} \mathbb {E} C _ {i i} u _ {i} \mathrm {D} _ {i} \ell (\boldsymbol {C} \boldsymbol {u} + \boldsymbol {m}) \geq \mathbb {E} C _ {i i} u _ {i} \mathrm {D} _ {i} \ell (\boldsymbol {C} \boldsymbol {u} + \boldsymbol {m} - \mathbf {e} _ {i} C _ {i i} u _ {i}) \\ = C _ {i i} \left(\mathbb {E} u _ {i}\right) \left(\mathbb {E} \mathrm {D} _ {i} f \left(\boldsymbol {C} \boldsymbol {u} + \boldsymbol {m} - \mathbf {e} _ {i} C _ {i i} u _ {i}\right)\right) \\ = 0, \\ \end{array}
$$
where  $\mathrm{D}_i f$  denotes the  $i$ th axis of  $\nabla f$ . Since  $C_{ii} > 0$  by design,
$$
\mathbb {E} C _ {i i} u _ {i} \mathrm {D} _ {i} f (\boldsymbol {C} \boldsymbol {u} + \boldsymbol {m}) > 0 \quad \Leftrightarrow \quad \mathbb {E} u _ {i} \mathrm {D} _ {i} f (\boldsymbol {C} \boldsymbol {u} + \boldsymbol {m}) > 0,
$$
which is Assumption 4.
Proof of (ii) We provide an example that proves the statement. Let  $\ell(\pmb{z}) = \frac{1}{2}\pmb{z}^{\top}\pmb{A}\pmb{z}$ . Then,
$$
g (\lambda ; \boldsymbol {u}) = \ell \left(\mathcal {T} _ {\lambda} (\boldsymbol {u})\right) = \boldsymbol {A} (\boldsymbol {C u} + \boldsymbol {m}) = \boldsymbol {A C u} + \boldsymbol {A m}.
$$
Suppose that we choose  $\lambda$  such that
$$
\boldsymbol {C} = \left[ \begin{array}{l l} 1 & 0 \\ 1 & 1 \end{array} \right]
$$
and  $\pmb{m} = \mathbf{0}$ . Also, setting
$$
\boldsymbol {A} = \left[ \begin{array}{c c} 1 & - 2 \\ - 2 & 5 \end{array} \right],
$$
we get a strongly convex function  $\ell$ . Then,
$$
g \left(\lambda ; \boldsymbol {u}\right) = \boldsymbol {A} \boldsymbol {C} \boldsymbol {u} = \left[ \begin{array}{c c} 1 & - 2 \\ - 2 & 5 \end{array} \right] \left[ \begin{array}{c c} 1 & 0 \\ 1 & 1 \end{array} \right] \left[ \begin{array}{c} u _ {1} \\ u _ {2} \end{array} \right] = \left[ \begin{array}{c c} - 1 & - 2 \\ 3 & 5 \end{array} \right] \left[ \begin{array}{c} u _ {1} \\ u _ {2} \end{array} \right] = \left[ \begin{array}{c} - u _ {1} - 2 u _ {2} \\ 3 u _ {1} + 5 u _ {2} \end{array} \right]
$$
$$
\mathbb {E} g _ {1} (\boldsymbol {\lambda}; \boldsymbol {u}) u _ {1} = \mathbb {E} (- u _ {1} - 2 u _ {2}) u _ {1} = - 1 <   0,
$$
which violates Assumption 4.
Lemma 11. For any function  $f \in C^{1}(\mathbb{R},\mathbb{R}_{+})$ , there is no constant  $0 < L < \infty$  such that
$$
| f (x) - f (y) | \geq L | x - y |.
$$
Proof. Suppose for the sake of contradiction that such  $L > 0$  exists. Letting  $y \to x$  gives  $|f'(x)| \geq L$  for all  $x \in \mathbb{R}$ . For each  $x$ , either  $f'(x) \leq -L$  or  $f'(x) \geq L$  holds. We discuss two cases based on the value of  $f'(0)$ .
If  $f^{\prime}(0) \geq L$ , we claim that  $f^{\prime}(x) \geq L$  for all  $x \in \mathbb{R}$ . Otherwise,  $f^{\prime}(x) < L$  for some  $x$  implies  $f^{\prime}(x) \leq -L$ . By the intermediate value theorem ( $f'$  is continuous), there exists a point  $y$  between 0 and  $x$  that attains the value  $f^{\prime}(y) = 0$ , which is a contradiction.
Now that  $f'(x) \geq L > 0$  for all  $x, f$  is an increasing function. For any  $x < 0$ , we have
$$
\begin{array}{l} f (x) = f (x) - f (0) + f (0) \\ = - | f (x) - f (0) | + f (0) \\ \leq - L | x | + f (0). \\ \end{array}
$$
Here, we can plug  $x' = -\frac{f(0)}{L}$  as
$$
f \left(x ^ {\prime}\right) = - L \left| - \frac {f (0)}{L} \right| + f (0) = - | f (0) | + f (0) \leq 0,
$$
which implies that  $f(x') \notin \mathbb{R}_+$ , which is a contradiction.
Now we discuss the second case  $f'(0) \leq -L$ . By a similar argument,  $f'(x) \leq -L$  for all  $x \in \mathbb{R}$ . Thus,  $f$  is a decreasing function. For any  $x > 0$ , we have
$$
\begin{array}{l} f (x) = f (x) - f (0) + f (0) \\ = - | f (x) - f (0) | + f (0) \\ \leq - L x + f (0). \\ \end{array}
$$
Picking  $x' = \frac{f(0)}{L}$  results in  $f(x') \notin \mathbb{R}_+$ , which is a contradiction.
Proof. The special case (i) is proven by Domke (2020, Theorem 9). We focus on the general statement (ii).
If  $\ell$  is  $\mu$ -strongly convex, the inequality
$$
\ell (\boldsymbol {z}) - \ell \left(\boldsymbol {z} ^ {\prime}\right) \geq \left\langle \nabla \ell \left(\boldsymbol {z} ^ {\prime}\right), \boldsymbol {z} - \boldsymbol {z} ^ {\prime} \right\rangle + \frac {\mu}{2} \| \boldsymbol {z} - \boldsymbol {z} ^ {\prime} \| _ {2} ^ {2} \tag {8}
$$
holds, where the general convex case is obtained as a special case with  $\mu = 0$ . The goal is to relate this to the ( $\mu$ -strong-)convexity of the energy with respect to the variational parameters given by
$$
f (\lambda) - f (\lambda^ {\prime}) \geq \langle \nabla_ {\lambda} f (\lambda^ {\prime}), \lambda - \lambda^ {\prime} \rangle + \frac {\mu}{2} \| \lambda - \lambda^ {\prime} \| _ {2} ^ {2}.
$$
Proof of (ii) Plugging the reparameterized latent variables to Equation (8) and taking the expectation, we have