SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Martin Ryner
Vironova AB, Stockholm, Sweden
Division of Numerical Analysis, Optimization and Systems Theory,
KTH Royal Institute of Technology, Stockholm, Sweden
martin.ryner@vironova.com, martinrr@kth.se
Jan Kronqvist
jankr@kth.se
Johan Karlsson
johan.karlsson@math.kth.se
This paper presents a framework for computing the Gromov-Wasserstein problem between two sets of points in low dimensional spaces, where the discrepancy is the squared Euclidean norm. The Gromov-Wasserstein problem is a generalization of the optimal transport problem that finds the assignment between two sets preserving pairwise distances as much as possible. This can be used to quantify the similarity between two formations or shapes, a common problem in AI and machine learning. The problem can be formulated as a Quadratic Assignment Problem (QAP), which is in general computationally intractable even for small problems. Our framework addresses this challenge by reformulating the QAP as an optimization problem with a low-dimensional domain, leveraging the fact that the problem can be expressed as a concave quadratic optimization problem with low rank. The method scales well with the number of points, and it can be used to find the global solution for large-scale problems with thousands of points. We compare the computational complexity of our approach with state-of-the-art methods on synthetic problems and apply it to a near-symmetrical problem which is of particular interest in computational biology.
Many important applications in machine learning deal with comparing sequences, images, and higher dimensional data, where the data is unstructured and not directly comparable. In physics, chemistry, biology, music, and linguistics, objects with greatly different properties often appear in symmetrical variations characterized by concepts such as isomerisms, chirality, harmonies, and alternations. Understanding, and being able to analyze, these types of variations can be truly critical
as some variations in chemicals and biologicals may be toxic or even lethal. The Gromov-Wasserstein framework [15, 16, 17], has shown to be a powerful approach for comparing and matching such data, as it is invariant to translations and rotation. The Gromov-Wasserstein framework has, for example, been successfully applied to domain adaptation [29], graph matching [28], metric alignment [9], single-cell alignment [7], and word embedding [1].
The task of evaluating the Gromov-Wasserstein problem is in general considered to be intractable. Typically, the computational burden grows exponentially with the number of points describing the compared objects. In fact, a Gromov-Wasserstein problem can be formulated as quadratic assignment problem (QAP) [14, 5, 4], which is known to be NP-Hard. Naturally, there has been plenty of research on local and approximate methods for solving Gromov-Wasserstein and QAP problems [20, 23, 22, 27, 2, 24, 25]. However, objects containing symmetries or repeated patterns are particularly challenging for local optimization methods and may lead to significant errors in the estimated discrepancy as matching such objects with local optimization methods may accidentally find the sub-optimal reflections and rotations. The inability to detect such phenomena can have a great impact on the discovery of isomerisms and subsequently attributes of crucial importance.
In this paper, we develop a rigorous method for globally optimizing Gromov-Wasserstein problems by calculating a sequence of iteratively improving upper- and lower bounds. We consider a general class of Gromov-Wasserstein discrepancy problems where the points, representing the objects, belong to a Euclidean space. We show that such Gromov-Wasserstein problems can be formulated exactly as low-rank QAPs. We build upon this low-rank QAP representation to develop an algorithm that scales well with the number of points. The proposed algorithm can be characterized as a so-called cutting plane method [12, 10] where we solve a sequence of relaxed problems that are iteratively strengthened by generating and accumulating valid linear inequality constraints, i.e., cutting planes. The optimum of the relaxed problem provides a valid lower bound for the optimum of the Gromov-Wasserstein problem in each iteration. By solving a computationally cheap optimal transportation problem [18, 26, 6], we obtain both an upper-bound and a new cutting plane to strengthen the relaxation. We prove convergence for the proposed algorithm, and present a computational study that clearly shows the algorithm's efficiency and that the performance scales well with the number of points.
The main contribution of the paper can be summarized as:
- We identify a general class of Gromov-Wasserstein problems, for point clouds embedded in low dimensional Euclidean spaces, that can be exactly represented as a concave low-rank QAP. In particular, mappings of images fits well within our framework.
- We develop a method for solving this class of Gromov-Wasserstein problems by solving a sequence of alternating sub-problems, which are either low-dimensional or linear.
- We prove that the proposed algorithm converges to a global optimal solution. The algorithm produces an optimality certificate in each iteration, in the form of upper- and lower bounds, which informs us of the potential suboptimality if the algorithm is terminated early.
- We present a numerical study, showing the efficiency of the proposed algorithm by comparing to other global optimization methods. We also illustrate the importance of globally solving Gromov-Wasserstein problems on a problem in computational biology.
In Section 2 we introduce the Gromov-Wasserstein problems and how it can be written as a QAP. In Section 3 we identify a class of Gromov-Wasserstein discrepancy problems that can be written as a concave relaxed QAPs problem, and in Section 4 we present the main methodology and an algorithm for solving this class of problems. Finally, in Section 5 we present numerical results and an application in computational biology.
2 The Gromov-Wasserstein discrepancy problem
Let  $x_{1},\ldots ,x_{n}\in \mathcal{X}$  and  $y_{1},\ldots ,y_{n}\in \mathcal{Y}$  be two sets of points and consider the problem of finding an assignment  $\pi$  between the point sets such that the pairwise distances  $d_{\mathcal{X}}(x_i,x_{i'})$  and  $d_{\mathcal{Y}}(y_{\pi (i)},y_{\pi (i')})$  are as close as possible for  $i,i' = 1,\dots ,n$ , where  $d_{\mathcal{X}}$  and  $d_{\mathcal{Y}}$  represents a notion of distance on the sets  $\mathcal{X}$  and  $\mathcal{Y}$ , respectively. This can be formulated as the discrete Gromov-Wasserstein discrepancy
$$
\min  _ {\Gamma \in P} \frac {1}{2} \sum_ {i, i ^ {\prime}, j, j ^ {\prime} = 1} ^ {n} \left(d _ {\mathcal {X}} \left(x _ {i}, x _ {i ^ {\prime}}\right) - d _ {\mathcal {Y}} \left(y _ {j}, y _ {j ^ {\prime}}\right)\right) ^ {2} \Gamma_ {i, j} \Gamma_ {i ^ {\prime}, j ^ {\prime}}, \tag {1}
$$
and where the assignment  $\pi$  is represented by a permutation matrix  $\Gamma$  and  $P$  is the set of all  $n\times n$  permutation matrices. The corresponding relaxed problem, where instead  $\Gamma$  is in the set of doubly stochastic matrices, denoted by  $\overline{P}$ , is often referred to as the Gromov-Wasserstein problem [20]. In these formulations we note that
$$
\begin{array}{l} \sum_ {i, i ^ {\prime}, j, j ^ {\prime} = 1} ^ {n} (d _ {\mathcal {X}} (x _ {i}, x _ {i ^ {\prime}}) - d _ {\mathcal {Y}} (y _ {j}, y _ {j ^ {\prime}})) ^ {2} \Gamma_ {i, j} \Gamma_ {i ^ {\prime}, j ^ {\prime}} \\ = \sum_ {i, i ^ {\prime}, j, j ^ {\prime} = 1} ^ {n} (d _ {\mathcal {X}} (x _ {i}, x _ {i ^ {\prime}}) ^ {2} - 2 d _ {\mathcal {X}} (x _ {i}, x _ {i ^ {\prime}}) d _ {\mathcal {Y}} (y _ {j}, y _ {j ^ {\prime}}) + d _ {\mathcal {Y}} (y _ {j}, y _ {j ^ {\prime}}) ^ {2}) \Gamma_ {i, j} \Gamma_ {i ^ {\prime}, j ^ {\prime}} \\ = \langle C _ {x}, C _ {x} \rangle - 2 \langle C _ {x} \Gamma , \Gamma C _ {y} \rangle + \langle C _ {y}, C _ {y} \rangle \\ \end{array}
$$
where  $C_x = [d_{\mathcal{X}}(x_i, x_{i'})]_{i,i' = 1}^n$ ,  $C_y = [d_{\mathcal{Y}}(y_j, y_{j'})]_{j,j' = 1}^n$ , and  $\langle \cdot, \cdot \rangle$  denotes the standard (Frobenius) inner product. Since the first and third sums are independent of  $\Gamma$ , solving the discrete Gromov-Wasserstein problem (1) is the same as solving a quadratic assignment problem (QAP) on a simplified Koopmans-Beckmann form [5], namely as
$$
\min  _ {\Gamma \in P} - \left\langle C _ {x} \Gamma , \Gamma C _ {y} \right\rangle + \frac {1}{2} \left(\left\langle C _ {x}, C _ {x} \right\rangle + \left\langle C _ {y}, C _ {y} \right\rangle\right). \tag {2}
$$
This problem is in general NP-hard, and the number of variables scales with the number of data points, making (2) computationally intractable for problems of relevant size. Here, we focus on instances where the matrices  $C_x, C_y$  are positive definite and low rank. By utilizing this structure, we develop an algorithm that is guaranteed to find a globally optimal solution and scales well with the number of points.
3 The Gromov-Wasserstein problem and low rank QAP
An important special case of the Gromow-Wasserstein problem, considered in [20, 22], is when the point clouds belong to the Euclidean space and the squared Euclidean distance is used as discrepancy. That is, when the set of points are  $x_{1},\ldots ,x_{n}\in \mathbb{R}^{\ell_{x}}$  and  $y_{1},\ldots ,y_{n}\in \mathbb{R}^{\ell_{y}}$ , which we represent by the matrices
$$
X = \left(x _ {1}, x _ {2}, \dots , x _ {n}\right) \in \mathbb {R} ^ {\ell_ {x} \times n}, \quad Y = \left(y _ {1}, y _ {2}, \dots , y _ {n}\right) \in \mathbb {R} ^ {\ell_ {y} \times n}.
$$
In this case it can be noted that the distance matrices  $C_x$  and  $C_y$  has a rank bounded by  $\ell_x + 2$  and  $\ell_y + 2$ , respectively, which can be seen from the identities
$$
C _ {x} = \left(\left\| x _ {i} - x _ {j} \right\| _ {2} ^ {2}\right) _ {i, j = 1} ^ {n} = \mathbf {1} m _ {x} ^ {T} - 2 X ^ {T} X + m _ {x} \mathbf {1} ^ {T}, \tag {3a}
$$
$$
C _ {y} = \left(\left\| y _ {i} - y _ {j} \right\| _ {2} ^ {2}\right) _ {i, j = 1} ^ {n} = \mathbf {1} m _ {y} ^ {T} - 2 Y ^ {T} Y + m _ {y} \mathbf {1} ^ {T}, \tag {3b}
$$
where  $m_x = (\| x_1\|^2, \| x_2\|^2, \ldots, \| x_n\|^2)^T$ ,  $m_y = (\| y_1\|^2, \| y_2\|^2, \ldots, \| y_n\|^2)^T$ , and  $\mathbf{1} \in \mathbb{R}^{n \times 1}$  is a column vector of ones. This observation was also used in [22] for formulating the Gromov-Wasserstein problem as a quadratic problem of rank  $(\ell_x + 2)(\ell_y + 2)$  and developing fast algorithms for the problem. However, the rank can be even further reduced and the corresponding Gromov-Wasserstein problem can be formulated as a QAP problem of rank  $\ell_x \ell_y$  [25, Lemma 4.2.3] (cf. [20, Proposition 1]).
Proposition 1. [25, Lemma 4.2.3] Let  $\Gamma$  be a doubly stochastic matrix and the matrices  $C_x$  and  $C_y$  given by (3), then it holds that
$$
\langle C _ {x} \Gamma , \Gamma C _ {y} \rangle = \langle 2 X \Gamma Y ^ {T}, 2 X \Gamma Y ^ {T} \rangle + \langle L, \Gamma \rangle + 2 \mathbf {1} ^ {T} m _ {y} \mathbf {1} ^ {T} m _ {x},
$$
where  $L = 2nm_{x}m_{y}^{T} - 4m_{x}\mathbf{1}^{T}Y^{T}Y - 4X^{T}X\mathbf{1}m_{y}^{T}$ .
Proof. The proposition follows by straightforward computations, where the expressions are simplified using  $\mathbf{1} = \Gamma \mathbf{1} = \Gamma^T\mathbf{1}$ . See the appendix for the proof.
Hence, the low rank QAP formulation of the discrete Gromov-Wasserstein problem can be stated as
$$
\min  _ {\Gamma \in P} - \left\langle 2 X \Gamma Y ^ {T}, 2 X \Gamma Y ^ {T} \right\rangle - \left\langle L, \Gamma \right\rangle + c _ {0} \tag {4}
$$
where  $L = 2nm_{x}m_{y}^{T} - 4m_{x}\mathbf{1}^{T}Y^{T}Y - 4X^{T}X\mathbf{1}m_{y}^{T}$ , and  $c_{0} = (\langle C_{x},C_{x}\rangle +\langle C_{y},C_{y}\rangle -4\mathbf{1}^{T}m_{y}\mathbf{1}^{T}m_{x}) / 2$ . The Gromov-Wasserstein problem can also be rewritten similarly and formulated as
$$
\min  _ {\Gamma \in \bar {P}} - \left\langle 2 X \Gamma Y ^ {T}, 2 X \Gamma Y ^ {T} \right\rangle - \left\langle L, \Gamma \right\rangle + c _ {0}, \tag {5}
$$
and since the objective function is concave, any optimal solution of (4) is also an optimal solution of the relaxed problem.
Proposition 2. Any optimal solution of the discrete Gromov-Wasserstein problem (4), is also an optimal solution to the Gromov-Wasserstein problem (5). Conversely, problem (5) always has an optimal solution in one extreme point, $^{1}$  and any optimal extreme point to (5) is also an optimal solution to (4).
Proof. Since (5) is the minimization of a concave objective function over a convex sets  $\overline{P}$ , it attains the optimal value in an extreme point of the feasible set. Since the permutation matrices are the extreme points to the doubly stochastic matrices, i.e.,  $P = \mathrm{ext}(\overline{P})$ , (5) attains its minimum on  $P$ . Further, the set of points in  $P$  for which (5) attains its minimum are the optimal solutions of (4). To show the converse statement, note that a minimum exists since  $\overline{P}$  is compact and the objective function is continuous. Further, since the objective function is concave, an optimum must be at an extreme point. Finally, since the extreme points of  $\overline{P}$  is the permutation matrices  $P$ , any optimal extreme point of (5) is also feasible and optimal to (4).
In the next section we will propose a methodology and an algorithm for solving this problem.
4 A cutting plane algorithm utilizing the low rank structure
By Proposition 2, we know that an optimal solution to the discrete Gromov-Wasserstein problem (4) can be obtained by solving the relaxed problem (5). However, the relaxed problem (5) is still a high-dimensional non-convex QP, which is NP-hard [19]. The high dimensionality can, in particular, be a limiting factor in solving the problem. For example, it is known that the performance of spatial branch-and-bound, one of the main approaches for globally optimizing nonconvex problems [10], can scale poorly with the number of variables. Thus, directly optimizing either (1) or (5) by spatial branch-and-bound is not computationally tractable for larger instances. Our idea is to use the low-rank formulation of the Gromov-Wasserstein problem and perform the optimization in a projected subspace of dimension  $\ell_x\ell_y + 1$  by solving a sequence of relaxed problems.
First, we note that problem (5) can be written as
$$
\min  _ {W \in \mathbb {R} ^ {\ell_ {x} \times \ell_ {y}}, w \in \mathbb {R}, \Gamma \in \overline {{P}}} - \| W \| _ {F} ^ {2} - w + c _ {0} \tag {6a}
$$
$$
\text {s u b j e c t} \quad W = 2 X \Gamma Y ^ {T}, w = \langle L, \Gamma \rangle . \tag {6b}
$$
Equivalence of problems (5) and (6) is shown by simply inserting the expressions for  $w$  and  $W$  into the objective function. Next, we project out the  $\Gamma$  variables, and we define the feasible set in the  $(w, W)$ -space as  $\mathcal{F} = \mathrm{Proj}_{W,w}\left(W \in \mathbb{R}^{\ell_x \times \ell_y}, w \in \mathbb{R}, \Gamma \in \overline{P} \mid W = 2X\Gamma Y^T, w = \langle L, \Gamma \rangle\right)$ .
Constructing an H-representation of the polytope  $\mathcal{F}$ , i.e., representing it by linear constraints of the form  $\langle Z_r, W \rangle + \alpha_r w \leq \beta_r$ , is not trivial and the number of constraints can grow exponentially with the number of data points. Therefore, we propose an algorithm based on a cutting plane scheme to optimize over  $\mathcal{F}$ .
An extreme point of a convex set is a point in the set which does not lie in any open line segment joining two points of the set.
Instead of directly optimizing the objective in (6a) over the feasible set  $\mathcal{F}$ , which we don't have a tractable representation for, we relax the problem as
$$
\min  _ {W \in \mathbb {R} ^ {\ell_ {x} \times \ell_ {y}}, w \in \mathbb {R}} - \| W \| _ {F} ^ {2} - w + c _ {0} \tag {7a}
$$
$$
\text {s u b j e c t} \quad \langle Z _ {r}, W \rangle + \alpha_ {r} w \leq \beta_ {r}, \quad \text {f o r} r = 1, \dots , N. \tag {7b}
$$
The linear constraints (7b) are supporting hyperplanes of the feasible set  $\mathcal{F}$ , which we will generate iteratively. The goal is to force the minimizer of problem (7) into the feasible set  $\mathcal{F}$  by using relatively few linear constraints. Keep in mind, we don't need a full representation of set  $\mathcal{F}$ , we only need to capture the shape of  $\mathcal{F}$  in some areas of interest, e.g., the constraints defining the faces of  $\mathcal{F}$  at the optimal solution of problem (6) would suffice. The main advantage of the relaxation in problem (7) is that it contains far fewer variables than both problems (5) and (6), and the dimensionality is independent of the number of data points. Problem (7) can, therefore, be solved much more efficiently, especially in early iterations when the number of constraints is low. We will show that the constraints can be determined, as needed, by solving optimal transport problems. Based on this, we will develop an iterative approach that sequentially solves problem (7) and adds a constraint until the solutions is the same as (6).
To initialize the search we determine a bounding box of  $\mathcal{F}$  and use this to define a set of constraints (7b). The bounding box is determined by the (elementwise) minimum and maximum of the variables  $w$  and  $W$  given by
$$
\min  _ {\Gamma \in \bar {P}} 2 \left(X \Gamma Y ^ {T}\right) _ {i, j} \leq W _ {i, j} \leq \max  _ {\Gamma \in \bar {P}} 2 \left(X \Gamma Y ^ {T}\right) _ {i, j} \quad \text {f o r} i = 1, \dots , \ell_ {x}; j = 1, \dots , \ell_ {y} \tag {8a}
$$
$$
\min  _ {\Gamma \in \bar {P}} \langle L, \Gamma \rangle \leq w \leq \max  _ {\Gamma \in \bar {P}} \langle L, \Gamma \rangle , \tag {8b}
$$
which can each be computed efficiently by solving a standard optimal transport problem. Initializing the set of constraints by the bounding box ensures that (7) is well-defined and bounded.
If the minimizer of problem (7) is within  $\mathcal{F}$ , then we can stop as the solution is optimal for (6).<sup>2</sup> Otherwise, we improve the outer approximation of  $\mathcal{F}$  by adding new a constraint defined by  $Z_{N + 1}$ ,  $\alpha_{N + 1}$  and  $\beta_{N + 1}$ . Let  $(w_{N},W_{N})$  be the current optimal solution of (7), and assume that  $(w_{N},W_{N}) \notin \mathcal{F}$ , then we form a new constraint, a so-called cutting plane, that excludes  $(w_{N},W_{N})$  from the feasible set of (7).
We form a new constraint based on the gradient of the objective function (7a), which is given by
$$
\nabla_ {(w, \operatorname {v e c} (W) ^ {T})} (- \| W \| _ {F} ^ {2} - w,) = (- 1, - 2 \operatorname {v e c} (W) ^ {T}).
$$
By letting  $\alpha_{N + 1} = 1$  and  $Z_{N + 1} = 2W_{N}$ , the hyperplane defining the new constraint will have the (negative) gradient in the optimum  $(w_{N}, W_{N})$  as normal vector. Then we select  $\beta_{N + 1}$  such that the new constraint forms a supporting hyperplane of  $\mathcal{F}$  (7b). This can be found by solving the following optimal transport problem
$$
\beta_ {N + 1} := \max  _ {W \in \mathbb {R} ^ {\ell_ {x} \times \ell_ {y}}, w \in \mathbb {R}, \Gamma \in \overline {{P}}} \langle Z _ {N + 1}, W \rangle + \alpha_ {N + 1} w = \max  _ {\Gamma \in \overline {{P}}} \left\langle 4 X ^ {T} W _ {N} Y + L, \Gamma \right\rangle . \tag {9}
$$
$$
\text {s u b j e c t} W = 2 X \Gamma Y ^ {T}, w = \langle L, \Gamma \rangle
$$
When solving this problem, we also obtain a solution  $\Gamma_N$  which is a doubly stochastic matrix (generically also a permutation matrix), which gives an upper bound for (6) and a candidate for the optimal solution. In the following subsection, we prove that the that algorithm, described in Algorithm 1, converges to a globally optimal solution.
A geometrical illustration of the algorithm is given in Figure (1). For illustrative purposes, we have used one-dimensional data resulting in a two-dimensional problem in the  $(W,w)$ -space. The data sets consist of 6 points each where one of the data sets has a reflective symmetry. This results in two global optima and 6! projected permutations in  $P$ . The first solution of (7) is located at one of the corners of the bounding box and marked with a "1" (the subsequent solutions are marked "2" - "5"). The infeasible point "1" is excluded from the search space by a cutting plane (red line, marked with an "A"). Following the same procedure we obtain point "2", and cutting plane "B". Adding further cutting planes excludes "3" and subsequently "4", resulting in the feasible and optimal point "5". Note that the cutting planes from iterations 3 and 4 almost overlap the cutting planes "A" and "B", since the gradients in the points 1 and 3 are very similar (the same for points 2 and 4).
2 Remember, we are minimizing the objective (6a) over an outer approximation of the feasible set.
Algorithm 1 Gromov-Wasserstein problem
Input  $X\in \mathbb{R}^{\ell_x\times n},Y\in \mathbb{R}^{\ell_y\times n},\epsilon >0$  (Define point clouds and give tolerance level)  $L_{\mathrm{bound}}\gets -\infty$  , and  $U_{\mathrm{bound}}\gets \infty$  (Set lower and upper bounds)  $(Z_{r},\alpha_{r},\beta_{r})$  for  $r = 1,\ldots ,N$  from (8), where  $N = 2\ell_x\ell_y + 2$  (Set initial constraints) while  $U_{\mathrm{bound}} - L_{\mathrm{bound}} > \epsilon$  do  $(w_N,W_N)\gets$  Optimal solution to (7) (Solve (7))  $L_{\mathrm{bound}}\gets -\| W_N\| _F^2 -w_N + c_0$  (Update lower bound)  $\Gamma_N\gets$  Optimal solution to (9) (Solve (9))  $U_{\mathrm{bound}}\gets \min (U_{\mathrm{bound}}, - \| 2X\Gamma_NY^T\| _F^2 -\langle L,\Gamma_N\rangle +c_0)$  (Update upper bound)  $(Z_{N + 1},\alpha_{N + 1},\beta_{N + 1})\gets (2W_N,1,\langle 4X^T W_NY + L,\Gamma_N\rangle)$  (Calculate new constraints)  $N\gets N + 1$  (Update iteration number)   
end while
Figure 1: Left image: An illustrative example of the method on one-dimensional data. Right image: The area around the two global optima highlighting the sequence of optimal extreme points in the approximate cover and generation of cutting planes.
4.1 Proof of convergence of Algorithm 1
The main result considering convergence is presented in the following theorem.
Theorem 1. The gap between the upper bound and lower bound in Algorithm 1 converges to 0 (if the tolerance is  $\epsilon = 0$ ).
Proof. Consider the  $N$ th iteration in Algorithm (1), let  $(w_{N},W_{N})$  be an optimal solution to (7), and  $\Gamma_N$  is an optimal solution to (9) with corresponding points  $(\hat{w},\hat{W}) = (\langle L,\Gamma_N\rangle ,2X\hat{\Gamma}_NY^T)$ , in the  $(w,W)$ -space. Assume that the gap in the objective function between those two points is
$$
\epsilon_ {N} = \left\| W _ {N} \right\| _ {F} ^ {2} + w _ {N} - \left\| \hat {W} \right\| _ {F} ^ {2} - \hat {w}. \tag {10}
$$
The new constraint is then defined by  $\langle Z_{N + 1},W\rangle +w\leq \beta_{N + 1}$  where  $Z_{N + 1} = 2W_{N}$  and  $\beta_{N + 1} = 2\langle W_N,\hat{W}\rangle +\hat{w}$ , and thus for any point  $(w,W)$  that satisfy the constraint it must hold that
$$
0 \leq 2 \langle W _ {N}, \hat {W} - W \rangle + \hat {w} - w.
$$
By substituting  $\hat{w}$  from (10), we obtain
$$
\begin{array}{l} \epsilon_ {N} \leq 2 \langle W _ {N}, \hat {W} - W \rangle - w + \| W _ {N} \| _ {F} ^ {2} + w _ {N} - \| \hat {W} \| _ {F} ^ {2} \\ = w _ {N} - w + 2 \left\langle W _ {N}, W _ {N} - W \right\rangle - \| \hat {W} - W _ {N} \| _ {F} ^ {2} \\ \leq w _ {N} - w + 2 \langle W _ {N}, W _ {N} - W \rangle \\ \leq \left(\left| w _ {N} - w \right| ^ {2} + \left\| W _ {N} - W \right\| _ {F} ^ {2}\right) ^ {1 / 2} \left(1 + 4 \left\| W _ {N} \right\| _ {F} ^ {2}\right) ^ {1 / 2} \\ \end{array}
$$
where we in the last step have used the Cauchy-Schwarz inequality.