SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Aaron J. Havens*¹ Alexandre Araujo*² Siddharth Garg² Farshad Khorrami² Bin Hu¹
$^{1}$  CSL & ECE, University of Illinois Urbana-Champaign  
 $^{2}$  ECE, New York University
Recently, deep equilibrium models (DEQs) have drawn increasing attention from the machine learning community. However, DEQs are much less understood in terms of certified robustness than their explicit network counterparts. In this paper, we advance the understanding of certified robustness of DEQs via exploiting the connections between various Lipschitz network parameterizations for both explicit and implicit models. Importantly, we show that various popular Lipschitz network structures, including convex potential layers (CPL), SDP-based Lipschitz layers (SLL), almost orthogonal layers (AOL), Sandwich layers, and monotone DEQs (MonDEQ) can all be reparameterized as special cases of the Lipschitz-bounded equilibrium networks (LBEN) without changing the prescribed Lipschitz constant in the original network parameterization. A key feature of our reparameterization technique is that it preserves the Lipschitz prescription used in different structures. This opens the possibility of achieving improved certified robustness of DEQs via a combination of network reparameterization, structure-preserving regularization, and LBEN-based finetuning. We also support our theoretical understanding with new empirical results, which show that our proposed method improves the certified robust accuracy of DEQs on classification tasks. All codes and experiments are made available at https://github.com/AaronHavens/ExploitingLipschitzDEQ.
Recently, deep equilibrium models (DEQs) have drawn increasing attention from the deep learning community [3]. In DEQs, the output is defined as the solution to an input-dependent fixed point equation. Specifically, consider a DEQ with input  $x$  and output  $y$ . Then  $y$  is typically determined from  $x$  as follows
$$
z = \sigma (W z + U x + b _ {z}), \quad y = G z + b _ {y}, \tag {1}
$$
where  $\sigma$  denotes some nonlinear activation function, e.g. ReLU. DEQs can be viewed as an implicit model that directly solves the fixed point of an infinitely-deep network. Since DEQs have implicit depth, they have the potential to reduce the large memory footprint required to train finite-depth feed-forward networks via explicit back-propagation. Noticeably, DEQs have shown great promise in achieving comparable performance on computer vision tasks [4, 33, 6, 28, 23].
Despite great potential, there are many open issues to be explored and understood for DEQs. There has been a body of recent works focusing on developing and understanding DEQ structures for addressing well-posedness, training stability, and adversarial robustness [47, 35, 31, 7, 30, 46, 5, 17, 25, 48, 13].
For example, the monotone DEQ (MonDEQ) uses the following parameterization:
$$
z = \sigma \left(((1 - m) I - A ^ {\top} A + B - B ^ {\top}) z + U x + b _ {z}\right), y = G z + b _ {y},
$$
where  $(A,B,U,G,b_z,b_y)$  are the decision variables to be trained, and  $m$  is a hyperparameter [47]. In other words, MonDEQ does not treat  $W$  as the variable to be directly trained and just parameterizes it as  $W = (1 - m)I - A^{\top}A + B - B^{\top}$ . The advantage of this parameterization is that well-posedness and  $\frac{\|U\|_2}{m}$ -Lipschitzness (from  $x$  to  $z$ ) are automatically ensured [47, 31]. Another popular DEQ structure for controlling the Lipschitz constant is given by the so-called Lipschitz-bounded equilibrium network (LBEN) parameterization which specifies the weight matrix  $W$  in (1) as follows
$$
W = I - \Psi \left(\left(G ^ {\top} G + \Psi^ {- 1} U U ^ {\top} \Psi^ {- 1}\right) / (2 L) + V ^ {\top} V + S - S ^ {\top} + \epsilon I\right), \tag {2}
$$
where  $\Psi$  is a diagonal positive definite matrix, and  $(V,S)$  are unconstrained free variables [35]. Using the LBEN parameterization, one can train  $L$ -Lipschitz DEQs by solving an unconstrained optimization problem.
This paper focuses on advancing the understanding of DEQ parameterizations in the context of certified robustness. It is well known that neural networks are susceptible to imperceptible adversarial input perturbations [40]. Hence, there is a need to develop certifiably robust models for safety-critical applications. For explicit networks, there has been significant progress on improving certified robustness by using various 1-Lipschitz layer structures such as orthogonal layers [41, 24, 37, 49], almost orthogonal layers (AOL) [34], convex potential layers (CPL) [27], SDP-based Lipschitz layers (SSL) [2], and Sandwich layers [43]. However, how to train DEQs with reasonable certified robustness remains largely open [31]. In this paper, we improve the certified robustness of DEQs via exploiting the connections between various Lipschitz structures for both explicit and implicit models. Importantly, we show that various popular Lipschitz network structures, including convex potential layers (CPL), SDP-based Lipschitz layers (SLL), almost orthogonal layers (AOL), Sandwich layers, and monotone DEQs (MonDEQ) can all be reparameterized as special cases of the Lipschitz-bounded equilibrium networks (LBEN) without changing the prescribed Lipschitz constant in the original network parameterizations. A key feature of our reparameterization technique is that it preserves the Lipschitz prescription used in different structures. This feature is particular relevant to training certifiably robust DEQs, and opens the possibility of achieving improved certified robustness of DEQs via a combination of network reparameterization, structure-preserving regularization, and LBEN-based fine-tuning. We provide empirical results to show that our reparameterization technique improves the certified robust accuracy of DEQs on classification benchmarks. Our work sheds new light on the principles for developing 1-Lipschitz DEQ structures and serves as a meaningful step towards achieving better certified robustness for DEQs.
Additional Related Work. There are also several recent results on certified robustness of DEQs under  $\ell_{\infty}$  perturbations [22, 30, 46, 11, 18, 10]. Our work focuses on certified robustness under  $\ell_{2}$  perturbations and hence naturally complements these existing works. In this paper, we mainly study the impacts of network parameterizations on certified robustness of DEQs. For explicit models, many other methods for designing certifiably robust models are also available [8, 36, 21, 16, 38, 15]. It is an interesting future task to study how to tailor these different methods for improving the certified robust accuracy of DEQs.
Notation. The set of  $n$ -dimensional real vectors is denoted by  $\mathbb{R}^n$ . For any vector  $x \in \mathbb{R}^n$ , we denote the associated Euclidean norm as  $\|x\|$ . In addition, we denote the  $n \times n$  identity matrix and the  $n \times n$  zero matrix as  $I_n$  and  $0_n$ , respectively. The subscripts will be omitted when the dimension is clear from the context. When a matrix  $P$  is negative semidefinite (definite), we will use the notation  $P \preceq (\prec)0$ . When a matrix  $P$  is positive semidefinite (definite), we will use the notation  $P \succeq (\succ)0$ . Given a collection of scalars  $\{a_i\}_{i=1}^n$ , the  $n \times n$  diagonal matrix whose  $(i,i)$ -th entry is  $a_i$  is denoted as  $\mathrm{diag}(a_i)$ . For a matrix  $A$ , let  $A^\top$  and  $\|A\|_2$  denote its transpose and largest singular value, respectively.
2.1 Lipschitz Property and Certified Robustness
Suppose we have a prediction model given by an input-output mapping  $y = f(x)$ . We say that  $f$  is  $L$ -Lipschitz (with respect to the  $\ell_2$  norm) if  $\| f(x_1) - f(x_2) \| \leq L \| x_1 - x_2 \|$  for any  $(x_1, x_2)$ .
The Lipschitz constant can be combined with the prediction margin of  $f$  to give certified robust accuracy [42, Proposition 1]. In the past, 1-Lipschitz models have shown great promise in achieving improved certified robustness on classification benchmark problems[29, 41, 24, 37, 27, 34, 2]. In this paper, we focus on designing certifiably robust DEQs via using novel 1-Lipschitz parameterizations.
2.2 SDPs for Well-posedness and Lipschitz Bounds of DEQs
Given a DEQ in the general form (1), the following existing sufficient conditions are useful for testing its well-posedness and Lipschitz properties.
Proposition 2.1 (Theorem 1 in Revay et al. [35]). Consider the DEQ model (1) with  $\sigma$  being sloped-restricted on  $[0,1]$ . If there exists a positive definite diagonal matrix  $\Gamma$  such that the following SDP holds,
$$
2 \Gamma - \Gamma W - W ^ {\top} \Gamma \succ 0, \tag {3}
$$
then the DEQ model (1) is well-posed and has a finite Lipschitz bound from  $x$  to  $y$ .
Proposition 2.2 (Theorem 2 in Revay et al. [35]). Consider the DEQ model (1) with  $\sigma$  be slope-restricted on  $[0,1]$ . If there exists a positive definite diagonal matrix  $\Lambda$  such that
$$
2 \Lambda - \Lambda W - W ^ {\top} \Lambda - \frac {1}{L} G ^ {\top} G - \frac {1}{L} \Lambda U U ^ {\top} \Lambda \succ 0, \tag {4}
$$
then the  $DEQ$  (1) is well-posed and  $L$ -Lipschitz from  $x$  to  $y$ .
The above propositions can be proved using standard control-theoretic arguments borrowed from the quadratic constraint theory [26, 12]. See [35] for detailed proofs. In general, matrix inequality conditions have been widely adopted for addressing the Lipschitz properties of neural networks [12, 32, 14, 44, 2, 45, 43], and (4) can be viewed as the extension of the LipSDP condition [12] to the DEQ setting. Notice that we use different notations for the decision variables in the testing conditions (3) and (4) to emphasize the fact that the solution to (3) may not be a solution to (4) even for the same DEQ model. For fixed  $W$ , the condition (3) is an SDP that can be efficiently verified by existing solvers. For given  $(W,U,G)$ , the condition (4) is bilinear in  $\Lambda$  and hence not an SDP. However, one can easily convexify (4) into an SDP form using the Schur complement lemma. Later, we will develop a variant of Proposition 2.2, which can be used to unify the developments of various Lipschitz implicit and explicit models.
2.3 DEQ Parameterizations: MonDEQ and LBEN
Now we briefly review several existing DEQ parameterizations for inducing well-posedness and the Lipschitz property.
MonDEQ. The well-posedness issue of DEQs was addressed in [47] using the MonDEQ parameterization derived from the monotone operator theory. In addition, operator splitting approaches can be used to efficiently compute the unique fixed point of MonDEQ. Recall that MonDEQ adopts the following weight parameterization:
$$
W = (1 - m) I - A ^ {\top} A + B - B ^ {\top} \tag {5}
$$
where  $m > 0$ , and  $(A,B)$  are free decision variables to be trained. As long as  $W$  is parameterized as in (5), the resultant DEQ model (1) is well-posed [47]. Based on examining the unrolled operator splitting iterates, it has been shown [31] that MonDEQ is  $\frac{\|U\|_2}{m}$ -Lipschitz from  $x$  to  $z$  (and hence  $\frac{\|U\|_2\|G\|_2}{m}$ -Lipschitz from  $x$  to  $y$ ). If  $G = I$ , then one can enforce MonDEQ to be 1-Lipschitz by adding the constraint  $\|U\|_2 \leq m$ .
Generalized MonDEQ (G-MonDEQ). Revay et al. [35] generalizes the MonDEQ parameterization as follows
$$
W = I - \Phi \left(A ^ {\top} A + B ^ {\top} - B + m I\right), \tag {6}
$$
where  $\Phi$  is a positive definite diagonal matrix to be trained. Obviously, the MonDEQ model is a special case of (6) with  $\Phi = I$ . For the G-MonDEQ parameterization (6), the well-posedness can also be guaranteed. Specifically, one can verify that the SDP condition (3) with  $W$  given by (6) holds if we choose  $\Gamma = \Phi^{-1}$ . In this case, the left side of (3) becomes  $2A^{\top}A + 2mI\succ 0$ . The Lipschitz constant for the G-MonDEQ parameterization (6) has not been derived before. Later we will present an explicit Lipschitz bound for (6) via constructing an analytical solution to a variant of (4).
LBEN. Revay et al. [35] has also proposed the following LBEN parameterization with free decision variables  $(G,U,V,S)$ , a diagonal matrix variable  $\Psi \succ 0$ , and a hyperparameter  $\varepsilon > 0$ :
$$
W = I - \Psi \left(\frac {1}{2 L} G ^ {\top} G + \frac {1}{2 L} \Psi^ {- 1} U U ^ {\top} \Psi^ {- 1} + V ^ {\top} V + S ^ {\top} - S + \varepsilon I\right). \tag {7}
$$
The LBEN parameterization guarantees the DEQ to be well-posed and  $L$ -Lipschitz. Specifically, one can verify that the condition (4) holds with  $W$  being defined by (7) and  $\Lambda = \Psi^{-1}$ . In this case, the left side of (4) becomes  $2V^{\top}V + 2\varepsilon I\succ 0$ . The above parameterization allows one to train  $L$ -Lipschitz DEQs in an unconstrained manner.
2.4 1-Lipschitz Explicit Feed-forward Networks: AOL, SLL and Sandwich Layers
For traditional explicit models, there exist several 1-Lipschitz parameterizations which can be used for efficient training of certifiably robust feed-forward networks. Now we briefly discuss several state-of-the-art 1-Lipschitz feed-forward parameterizations.
Orthogonal and Almost Orthogonal Layers. Consider the following standard explicit feedforward network:
$$
x _ {0} = x, \quad x _ {n + 1} = \sigma \left(W _ {n} x _ {n} + b _ {n}\right), \quad y = W _ {N} x _ {N} + b _ {N},
$$
where  $n = 0,1,\dots ,N - 1$ . Since the compositions of 1-Lipschitz functions are also 1-Lipschitz, one can just parameterize every explicit layer  $x_{n + 1} = \sigma (W_n x_n + b_n)$  to be 1-Lipschitz, and the resultant feed-forward network is 1-Lipschitz. If  $\sigma$  is 1-Lipschitz<sup>1</sup>, one only needs to ensure  $\| W_{n}\|_{2}\leq 1$  for all  $n$ . An important issue in training such Lipschitz layers for deep explicit networks is gradient vanishing, and this motivates the development of various techniques which introduce gradient norm preservation based on orthogonality. The orthogonality parameterizations just enforce  $W_{n}$  to satisfy  $W_{n}^{\top}W_{n} = I$  [41, 24, 37]. In contrast, the Almost-Orthogonal-layer (AOL) [34] performs a weight normalization  $W_{n} = \tilde{W}_{n}D_{n}$ , where  $\tilde{W}_n$  is the free decision variable to be trained, and  $D_{n}$  is a scaling matrix which is diagonal and defined as<sup>2</sup>
$$
D _ {n} = \operatorname {d i a g} \left(\sum_ {j} | \tilde {W} _ {n} ^ {\top} \tilde {W} _ {n} | _ {i j}\right) ^ {- \frac {1}{2}}. \tag {8}
$$
AOL ensures  $\| W_{n}\|_{2} \leq 1$ . In [34], it has been empirically demonstrated that the trained weight  $W_{n}$  tends to be "almost orthogonal."
Lipschitz Residual Networks and SLL. Another way to address the gradient vanishing issue in training Lipschitz networks is to use a residual structure [27, 2]. A general result is given in [2] and shows that the following residual network is guaranteed to be 1-Lipschitz given  $W_{n}^{\top}W_{n}\preceq T_{n}$ :
$$
x _ {0} = x, \quad x _ {n + 1} = x _ {n} - 2 W _ {n} T _ {n} ^ {- 1} \sigma \left(W _ {n} ^ {\top} x _ {n} + b _ {n}\right), \quad y = x _ {N}.
$$
For CPL, the choice of  $T_{n}$  is simple, i.e.  $T_{n} = \| W_{n}\|_{2}^{2}I$ . To achieve better certified robust accuracy, SLL [2] uses the following more delicate choice of  $T_{n}$ :
$$
T _ {n} = \operatorname {d i a g} \left(\sum_ {j} \left| W _ {n} ^ {\top} W _ {n} \right| _ {i j} \frac {q _ {j} ^ {(n)}}{q _ {i} ^ {(n)}}\right) \tag {9}
$$
where  $\{q_i^{(n)}\}$  are free positive scalars to be trained.
Sandwich Layers. Recently, the sandwich layer has been proposed in [43], and achieves very competitive performances with relatively small models. Based on [43], the following network is guaranteed to be  $L$ -Lipschitz:
$$
x _ {0} = \sqrt {L} x, \quad x _ {n + 1} = \sqrt {2} A _ {n} ^ {\top} \Phi_ {n} \sigma (\sqrt {2} \Phi_ {n} ^ {- 1} B _ {n} x _ {n} + b _ {n}), \quad y = \sqrt {L} B _ {N} x _ {N} + b _ {N},
$$
where  $A_{n}A_{n}^{\top} + B_{n}B_{n}^{\top} = I$ ,  $B_{N}^{\top}B_{N} = I$ , and  $\Phi_{n}$  is a free diagonal matrix whose entries are restricted to be non-negative. One can parameterize  $[A_{n}\quad B_{n}]^{\top}$  to satisfy  $A_{n}A_{n}^{\top} + B_{n}B_{n}^{\top} = I$  using the generalized Cayley transformation [19], and train the above network efficiently.
1 Almost all the activation functions that are commonly used in practice are 1-Lipschitz.
2If one has  $\sum_{j}|\tilde{W}_{n}^{\top}\tilde{W}_{n}|_{ij} = 0$  for  $i$ , then the  $(i,i)$ -th entry of  $D_{n}$  is just set to be 0.
3 Network Reparameterizations with Preserved Lipschitz Prescriptions
In this section, we show that various Lipschitz structures such as MonDEQ, G-MonDEQ, SLL, Sandwich, and AOL can all be reparameterized as special cases of LBEN without changing the prescribed Lipschitz constant in the original parameterizations. The feature of preserving the Lipschitz constant prescription between different parameterizations is particularly relevant to training certifiably robust DEQs, and we will elaborate on this point later. We emphasize that our result does not indicate that we should abandon all other Lipschitz structures and only use the LBEN parameterization for training certifiably robust DEQs. As a matter of fact, when implementing these different parameterizations for classification tasks, one can potentially incorporate quite different convolution structures and the inductive bias can be very different. The true implication of our theory is that one can potentially fine-tune the trained models from MonDEQ, SLL, or other explicit Lipschitz layers by reparameterizing the networks as LBEN and initializing the LBEN training from these reparameterized models. The key to this implication is that the prescribed Lipschitz constant has to be preserved during the reparameterization process. Our theory opens the possibility of achieving improved robustness of DEQs via a combination of network reparameterization and LBEN-based fine-tuning.
3.1 Warm-up: Issues in Reparameterizing MonDEQ as LBEN and a Fix
In this section, we reveal a deeper connection between MonDEQ and LBEN. As mentioned in [35, Remark 2], any MonDEQ immediately leads to an LBEN for sufficiently large  $L$ . However, this trivial reparameterization of LBEN from MonDEQ does not preserve the prescribed Lipschitz constant in the original MonDEQ parameterization. Specifically, the original MonDEQ controls its Lipschitz constant by  $\frac{\|U\|_2}{m}$ . After the trivial reparameterization based on [35, Remark 2], the resultant Lipschitz constant is no longer  $\frac{\|U\|_2}{m}$ . In contrast, it can be arbitrarily large. This is problematic for training certifiably robust DEQs, since the increase in the Lipschitz constant will decrease the certified robust accuracy.
Notice that the construction of LBEN relies on Proposition 2.2. The above technical issue hinges upon the fact that Proposition 2.2 cannot be directly applied to prove the Lipschitz constant of MonDEQ. To gain some insight into how to fix this issue, we first discuss how to modify Proposition 2.2 to recover the previous Lipschitz bound of MonDEQ. Specifically, we will use a variant of [35, Theorem 2] (which has been restated as Proposition 2.2). Now we state this new condition.
Theorem 3.1. Suppose the DEQ model (1) is well-posed, and  $\sigma$  is slope-restricted on  $[0,1]$ . If there exists a positive definite diagonal matrix  $\Lambda$  such that the following non-strict matrix inequality holds,
$$
2 \Lambda - \Lambda W - W ^ {\top} \Lambda - \frac {1}{L} G ^ {\top} G - \frac {1}{L} \Lambda U U ^ {\top} \Lambda \succeq 0, \tag {10}
$$
then the  $DEQ$  (1) is  $L$ -Lipschitz from  $x$  to  $y$ .
Proof. We only need to modify the quadratic constraint argument used in [35, Theorem 2]. Originally, a strict matrix inequality is needed to ensure well-posedness. However, well-posedness has been assumed here, and a non-strict matrix inequality is sufficient to ensure the  $L$ -Lipschitz property. For completeness, a detailed proof is presented in the appendix.  $\square$
The above variant of the strict matrix inequality (4) gives us a non-strict condition, which can then be used to derive Lipschitz properties of MonDEQ, G-MonDEQ, and other explicit models in a unified manner. It turns out that such a non-strict variant is quite crucial for our proposed reparameterization techniques. Recall that originally the LBEN parameterization (7) is derived using Proposition 2.2. Since Theorem 3.1 is the non-strict variant of Proposition 2.2, it is not that surprising that we can use Proposition 2.1 to take care of the well-posedness issue and then combine it with Theorem 3.1 to prove the desired Lipschitz property. We will show that the Lipschitz properties of MonDEQ, G-MonDEQ, and explicit layers such as SLL can also be obtained by providing analytical solutions to (10). More importantly, these new analytical solutions will guide us to find a systematic way to reparameterize these structures as LBEN without changing the prescribed Lipschitz constant in the original parameterizations.
First, we will apply our new theorem to recover the original Lipschitz bound from [31]. The original analysis in [31] relies on examining the operator splitting iterates and bounding the difference between outputs based on their inputs. Alternatively, we can recover this result using the non-strict condition (10). Specifically, we can set  $G = I$  (recalling that we are interested in the Lipschitz constant from  $x$  to  $z$ ),  $\Lambda = \frac{1}{\|U\|_2} I$ ,  $L = \frac{\|U\|_2}{m}$ , and  $W$  as defined by the MonDEQ parameterization (5). Then we can show that the left side of (10) becomes
$$
\begin{array}{l} \frac {2 m}{\| U \| _ {2}} I + \frac {2}{\| U \| _ {2}} A ^ {\top} A - \frac {1}{L} I - \frac {1}{L \| U \| ^ {2}} U U ^ {\top} = \frac {1}{L} I + \frac {2}{\| U \| _ {2}} A ^ {\top} A - \frac {1}{L \| U \| _ {2} ^ {2}} U U ^ {\top} \\ = \frac {2}{\| U \| _ {2}} A ^ {\top} A + \frac {1}{L} \left(I - \frac {1}{\| U \| _ {2} ^ {2}} U U ^ {\top}\right) \succeq 0. \\ \end{array}
$$
Therefore, the condition holds with  $\Lambda = \frac{1}{\|U\|_2} I$  and  $L = \| U\| _2 / m$ . Notice that MonDEQ is known to be well-posed. Hence Theorem 3.1 can be directly applied to give the conclusion that MonDEQ is  $\frac{\|U\|_2}{m}$ -Lipschitz from  $x$  to  $z$ . If we have  $G = I$  and want to enforce MonDEQ to be 1-Lipschitz, then we only need to require  $\| U\| _2\leq m$ . There are many ways to parameterize  $U$  such that this norm constraint is automatically satisfied. For example, one can enforce  $U / \sqrt{m}$  to be an orthogonal matrix. Similarly, one can also use the scaling trick in AOL to ensure  $\| U\| _2\leq m$ . See the appendix for more discussions on how to parameterize 1-Lipschitz MonDEQ.
Next, we discuss how to address the previous issue in reparameterization such that we can reparameterize 1-Lipschitz MonDEQ as LBEN by exactly setting  $L = 1$  in the LBEN reparameterization. We will leverage the construction of  $\Lambda$  above. Recall that the free variables in MonDEQ are  $(A, B, U)$ . The matrices  $(W, G)$  are given by
$$
W = (1 - m) I - A ^ {\top} A + B - B ^ {\top}, G = I
$$
Now we confine our scope to 1-Lipschitz MonDEQ, i.e.  $\| U \|_2 \leq m$ . We will show that given a MonDEQ model (5) with any fixed  $(A, B, U)$ , we can always generate an LBEN parameterization in the form of (7) which gives exactly the same input-output mapping behavior from  $x$  to  $y$ . This means that any trained MonDEQ model can be reparameterized in the form of LBEN with some choice of  $(\Psi, G, U, V, S, \epsilon)$ . We will provide explicit formulas for such a choice of  $(\Psi, G, U, V, S, \epsilon)$ . We have the following formal statement.
Theorem 3.2. Given a 1-Lipschitz MonDEQ model with  $(A,B,U)$  satisfying  $\| U\| _2\leq m$  , the following choice of  $(\Psi ,G,U,V,S,\epsilon)$  with  $U$  being the same one used for MonDEQ gives a valid 1-Lipschitz LBEN model which generates the same input-output mapping:
$$
\Psi = | | U | | _ {2} I, \quad G = I, \quad S = \frac {B}{\| U \| _ {2}}, \quad \epsilon = 0
$$
$$
V ^ {\top} V = \left(\frac {m}{\| U \| _ {2}} - \frac {1}{2}\right) I + \frac {A ^ {\top} A}{\| U \| _ {2}} - \frac {1}{2 \| U \| _ {2} ^ {2}} U U ^ {\top} \succeq 0
$$