SlowGuess/NeurIPS_2023_OCR huggingface dataset:
[27] D. Kim and J. A. Fessler. Another look at the fast iterative shrinkage/thresholding algorithm (FISTA). SIAM Journal on Optimization, 28(1):223-250, 2018.
[28] D. Kim and J. A. Fessler. Generalizing the optimized gradient method for smooth convex minimization. SIAM Journal on Optimization, 28(2):1920-1950, 2018.
[29] D. Kim and J. A. Fessler. Optimizing the efficiency of first-order methods for decreasing the gradient of smooth convex functions. Journal of Optimization Theory and Applications, 188(1):192-219, 2021.
[30] J. Kim and I. Yang. Unifying nesterov's accelerated gradient methods for convex and strongly convex objective functions: From continuous-time dynamics to discrete-time algorithms. ICML, 2023.
[31] J. Lee, C. Park, and E. K. Ryu. A geometric structure of acceleration and its role in making gradients small fast. NeurIPS, 2021.
[32] B. Li, M. Coutiño, and G. B. Giannakis. Revisit of estimate sequence for accelerated gradient methods. ICASSP, 2020.
[33] O. L. Mangasarian and J. Ponceau, Minmax and duality in nonlinear programming. Journal of Mathematical Analysis and Applications, 11:504-518, 1965.
[34] A. S. Nemirovski. Optimization II: Numerical methods for nonlinear continuous optimization., 1999.
[35] A. S. Nemirovsky. On optimality of Krylov's information when solving linear operator equations. Journal of Complexity, 7(2):121-130, 1991.
[36] A. S. Nemirovsky. Information-based complexity of linear operator equations. Journal of Complexity, 8(2):153-175, 1992.
[37] Y. Nesterov. A method for unconstrained convex minimization problem with the rate of convergence  $\mathcal{O}(1 / k^2)$ . Proceedings of the USSR Academy of Sciences, 269:543-547, 1983.
[38] Y. Nesterov. Smooth minimization of non-smooth functions. Mathematical Programming, 103(1):127-152, 2005.
[39] Y. Nesterov. Accelerating the cubic regularization of Newton's method on convex problems. Mathematical Programming, 112(1):159-181, 2008.
[40] Y. Nesterov. Efficiency of coordinate descent methods on huge-scale optimization problems. SIAM Journal on Optimization, 22(2):341-362, 2012.
[41] Y. Nesterov. How to make the gradients small. Optima. Mathematical Optimization Society Newsletter, (88):10-11, 2012.
[42] Y. Nesterov. Lectures on Convex Optimization. Springer, 2nd edition, 2018.
[43] C. Park, J. Park, and E. K. Ryu. Factor- $\sqrt{2}$  acceleration of accelerated gradient methods. Applied Mathematics and Optimization, 2023.
[44] C. Park and E. K. Ryu. Optimal first-order algorithms as a function of inequalities. arXiv:2110.11035, 2021.
[45] G. B. Passty. Ergodic convergence to a zero of the sum of monotone operators in Hilbert space. Journal of Mathematical Analysis and Applications, 72(2):383-390, 1979.
[46] R. T. Rockafellar. Convex Analysis. Princeton university press, 1970.
[47] R. T. Rockafellar. Conjugate Duality and Optimization. CBMS-NSF Regional Conference Series in Applied Mathematics. Society for Industrial and Applied Mathematics, 1974.
[48] E. K. Ryu and W. Yin. Large-Scale Convex Optimization. Cambridge University Press, 2022.
[49] J. Stoer. Duality in nonlinear programming and the minimax theorem. Numerische Mathematik, 5(1):371-379, 1963.
[50] W. Su, S. Boyd, and E. Candes. A differential equation for modeling Nesterov's accelerated gradient method: Theory and insights. NeurIPS, 2014.
[51] J. J. Suh, G. Roh, and E. K. Ryu. Continuous-time analysis of accelerated gradient methods via conservation laws in dilated coordinate systems. ICML, 2022.
[52] A. B. Taylor. Convex interpolation and performance estimation of first-order methods for convex optimization. 2017.
[53] A. B. Taylor and F. Bach. Stochastic first-order methods: non-asymptotic and computer-aided analyses via potential functions. *COLT*, 2019.
[54] A. B. Taylor, J. M. Hendrickx, and F. Glineur. Exact worst-case performance of first-order methods for composite convex optimization. SIAM Journal on Optimization, 27(3):1283-1313, 2017.
[55] A. B. Taylor, J. M. Hendrickx, and F. Glineur. Smooth strongly convex interpolation and exact worst-case performance of first-order methods. Mathematical Programming, 161(1-2):307-345, 2017.
[56] A. Wibisono, A. C. Wilson, and M. I. Jordan. A variational perspective on accelerated methods in optimization. Proceedings of the National Academy of Sciences of the United States of America, 113(47):E7351-E7358, 2016.
[57] P. Wolfe. A duality theorem for non-linear programming. Quarterly of applied mathematics, 19(3):239-244, 1961.
[58] S. Yamanaka and N. Yamashita. Duality of optimization problems with gauge functions. Optimization, pages 1-31, 2022.
[59] S. Zhao, L. Lessard, and M. Udell. An automatic system to detect equivalence between iterative algorithms. arXiv:2105.04684, 2021.
[60] K. Zhou, L. Tian, A. M.-C. So, and J. Cheng. Practical schemes for finding near-stationary points of convex finite-sums. AISTATS, 2022.
Reformulate (C1) and (C2) into  $\mathbf{U}$  and  $V$ . In this paragraph, we will show
$$
\left. \quad \Leftrightarrow \quad \left[ \mathbf {U} \geq 0, \quad \left(\forall \nabla f (x _ {0}), \dots , \nabla f (x _ {N}) \in \mathbb {R} ^ {d}\right) \right] \right. \tag {C1}
$$
$$
\quad \Leftrightarrow \quad \left[ \mathbf {V} \geq 0, \quad \left(\forall \nabla f (y _ {0}), \dots , \nabla f (y _ {N}) \in \mathbb {R} ^ {d}\right) \right]. \tag {C2}
$$
Recall the definition of  $\mathbf{U}$  and  $\mathbf{V}$ .
$$
\mathbf {U} \colon = \mathcal {U} _ {N} - u _ {N} \left(f \left(x _ {N}\right) - f _ {\star}\right) - \frac {L}{2} \left\| x _ {\star} - x _ {0} + \frac {1}{L} \sum_ {i = 0} ^ {N} \left(u _ {i} - u _ {i - 1}\right) \nabla f \left(x _ {i}\right) \right\| ^ {2} \tag {11}
$$
$$
\mathbf {V} \colon = \mathcal {V} _ {N} - \frac {1}{2} \| \nabla f (y _ {N}) \| ^ {2}. \tag {12}
$$
First we calculate  $\mathcal{U}_N - u_N(f(x_N) - f_\star)$ .
$$
\begin{array}{l} \mathcal {U} _ {N} - u _ {N} \left(f \left(x _ {N}\right) - f _ {\star}\right) \\ = \frac {L}{2} \| x _ {0} - x _ {\star} \| ^ {2} + \sum_ {i = 0} ^ {N} (u _ {i} - u _ {i - 1}) \left(f (x _ {i}) - f _ {\star} + \langle \nabla f (x _ {i}), x _ {\star} - x _ {i} \rangle + \frac {1}{2 L} \| \nabla f (x _ {i}) \| ^ {2}\right) \\ + \sum_ {i = 0} ^ {N - 1} u _ {i} \left(f (x _ {i + 1}) - f (x _ {i}) + \langle \nabla f (x _ {i + 1}), x _ {i} - x _ {i + 1} \rangle + \frac {1}{2 L} \| \nabla f (x _ {i}) - \nabla f (x _ {i + 1}) \| ^ {2}\right) - u _ {N} \left(f (x _ {N}) - f _ {\star}\right) \\ \stackrel {(\circ)} {=} \frac {L}{2} \| x _ {0} - x _ {\star} \| ^ {2} + \sum_ {i = 0} ^ {N} (u _ {i} - u _ {i - 1}) \left(\langle \nabla f (x _ {i}), x _ {\star} - x _ {i} \rangle + \frac {1}{2 L} \| \nabla f (x _ {i}) \| ^ {2}\right) \\ + \sum_ {i = 0} ^ {N - 1} u _ {i} \left(\langle \nabla f (x _ {i + 1}), x _ {i} - x _ {i + 1} \rangle + \frac {1}{2 L} \| \nabla f (x _ {i}) - \nabla f (x _ {i + 1}) \| ^ {2}\right) \\ = \frac {L}{2} \left\| x _ {0} - x _ {\star} + \sum_ {i = 0} ^ {N} \frac {u _ {i} - u _ {i - 1}}{L} \nabla f (x _ {i}) \right\| ^ {2} - \frac {1}{2 L} \left\| \sum_ {i = 0} ^ {N} (u _ {i} - u _ {i - 1}) \nabla f (x _ {i}) \right\| ^ {2} \\ + \sum_ {i = 0} ^ {N} \left(u _ {i} - u _ {i - 1}\right) \left(\left\langle \nabla f \left(x _ {i}\right), x _ {0} - x _ {i} \right\rangle + \frac {1}{2 L} \| \nabla f (x _ {i}) \| ^ {2}\right) \\ + \sum_ {i = 0} ^ {N - 1} u _ {i} \left(\langle \nabla f (x _ {i + 1}), x _ {i} - x _ {i + 1} \rangle + \frac {1}{2 L} \| \nabla f (x _ {i}) - \nabla f (x _ {i + 1}) \| ^ {2}\right). \\ \end{array}
$$
Note that all function value terms are deleted at  $\circ$ . Therefore,
$$
\begin{array}{l} \mathbf {U} = - \frac {1}{2 L} \left\| \sum_ {i = 0} ^ {N} (u _ {i} - u _ {i - 1}) \nabla f (x _ {i}) \right\| ^ {2} + \sum_ {i = 0} ^ {N} (u _ {i} - u _ {i - 1}) \left(\langle \nabla f (x _ {i}), x _ {0} - x _ {i} \rangle + \frac {1}{2 L} \| \nabla f (x _ {i}) \| ^ {2}\right) \\ + \sum_ {i = 0} ^ {N - 1} u _ {i} \left(\left\langle \nabla f \left(x _ {i + 1}\right), x _ {i} - x _ {i + 1} \right\rangle + \frac {1}{2 L} \| \nabla f (x _ {i}) - \nabla f \left(x _ {i + 1}\right) \| ^ {2}\right) \tag {13} \\ \end{array}
$$
$$
\begin{array}{l} \mathcal {U} _ {N} - u _ {N} \left(f \left(x _ {N}\right) - f _ {\star}\right) \\ = \mathbf {U} + \frac {L}{2} \left\| x _ {0} - x _ {\star} + \sum_ {i = 0} ^ {N} \frac {u _ {i} - u _ {i - 1}}{L} \nabla f (x _ {i}) \right\| ^ {2}. \\ \end{array}
$$
Since  $(x_0 - x_i)$ ,  $(x_i - x_{i+1}) \in \operatorname{span}\left\{\nabla f(x_0), \ldots, \nabla f(x_N)\right\}$ , the value of  $\mathbf{U}$  is independent with  $x_0, x_\star$ . Thus the only term that depends on  $x_0$  and  $x_\star$  is  $\frac{L}{2}\left\|x_0 - x_\star + \sum_{i=0}^{N} \frac{u_i - u_{i-1}}{L} \nabla f(x_i)\right\|^2$ . Next, since  $x_0, x_\star$  can have any value, we can take  $x_0 - x_\star = \frac{u_i - u_{i-1}}{L} \nabla f(x_i)$ . Thus it gives the fact that (C1) is equivalent to  $\left[\mathbf{U} \geq 0, \left(\forall \nabla f(x_0), \ldots, \nabla f(x_N) \in \mathbb{R}^d\right)\right]$ .
Now we calculate  $\mathcal{V}_N - \frac{1}{2}\left\| \nabla f(y_N)\right\|^2$
$$
\begin{array}{l} \mathcal {V} _ {N} - \frac {1}{2} \| \nabla f (y _ {N}) \| ^ {2} \\ = v _ {0} \left(f _ {\star} - f \left(y _ {N}\right) + \frac {1}{2 L} \| \nabla f \left(y _ {N}\right) \| ^ {2}\right) + v _ {0} \left(f \left(y _ {0}\right) - f _ {\star}\right) \\ + \sum_ {i = 0} ^ {N - 1} v _ {i + 1} \left(f \left(y _ {i + 1}\right) - f \left(y _ {i}\right) + \left\langle \nabla f \left(y _ {i + 1}\right), y _ {i} - y _ {i + 1} \right\rangle + \frac {1}{2 L} \| \nabla f (y _ {i}) - \nabla f \left(y _ {i + 1}\right) \| ^ {2}\right) \\ + \sum_ {i = 0} ^ {N - 1} \left(v _ {i + 1} - v _ {i}\right) \left(f \left(y _ {i}\right) - f \left(y _ {N}\right) + \left\langle \nabla f \left(y _ {i}\right), y _ {N} - y _ {i} \right\rangle + \frac {1}{2 L} \left\| \nabla f \left(y _ {i}\right) - \nabla f \left(y _ {N}\right) \right\| ^ {2}\right) - \frac {1}{2 L} \| \nabla f \left(y _ {N}\right) \| ^ {2} \\ \stackrel {(\circ)} {=} \frac {v _ {0}}{2 L} \| \nabla f (y _ {N}) \| ^ {2} + \sum_ {i = 0} ^ {N - 1} v _ {i + 1} \left(\langle \nabla f (y _ {i + 1}), y _ {i} - y _ {i + 1} \rangle + \frac {1}{2 L} \| \nabla f (y _ {i}) - \nabla f (y _ {i + 1}) \| ^ {2}\right) \\ + \sum_ {i = 0} ^ {N - 1} \left(v _ {i + 1} - v _ {i}\right) \left(\left\langle \nabla f \left(y _ {i}\right), y _ {N} - y _ {i} \right\rangle + \frac {1}{2 L} \left\| \nabla f \left(y _ {i}\right) - \nabla f \left(y _ {N}\right) \right\| ^ {2}\right) - \frac {1}{2 L} \| \nabla f \left(y _ {N}\right) \| ^ {2}. \\ \end{array}
$$
Note that all function values are deleted at  $(\circ)$ . By the calculation result,
$$
\begin{array}{l} \mathbf {V} = \frac {v _ {0}}{2 L} \| \nabla f (y _ {N}) \| ^ {2} + \sum_ {i = 0} ^ {N - 1} v _ {i + 1} \left(\langle \nabla f (y _ {i + 1}), y _ {i} - y _ {i + 1} \rangle + \frac {1}{2 L} \| \nabla f (y _ {i}) - \nabla f (y _ {i + 1}) \| ^ {2}\right) \\ + \sum_ {i = 0} ^ {N - 1} \left(v _ {i + 1} - v _ {i}\right) \left(\left\langle \nabla f \left(y _ {i}\right), y _ {N} - y _ {i} \right\rangle + \frac {1}{2 L} \left\| \nabla f \left(y _ {i}\right) - \nabla f \left(y _ {N}\right) \right\| ^ {2}\right) - \frac {1}{2 L} \| \nabla f \left(y _ {N}\right) \| ^ {2}. \tag {14} \\ \end{array}
$$
(C2) is equivalent to  $\left[\mathbf{V} \geq 0, \quad \left(\forall \nabla f(y_0), \dots, \nabla f(y_N) \in \mathbb{R}^d\right)\right]$ . To establish Theorem 1, demonstrating
$$
\left[ \mathbf {U} \geq 0, \quad (\forall \nabla f (x _ {0}), \dots , \nabla f (x _ {N}) \in \mathbb {R} ^ {d}) \right] \quad \Leftrightarrow \quad \left[ \mathbf {V} \geq 0, \quad (\forall \nabla f (y _ {0}), \dots , \nabla f (y _ {N}) \in \mathbb {R} ^ {d}) \right] \tag {15}
$$
would suffice.
Transforming U and V into a trace. Define
$$
g _ {x} \colon = \left[ \nabla f (x _ {0}) | \nabla f (x _ {1}) | \dots | \nabla f (x _ {N}) \right] \in \mathbb {R} ^ {d \times (N + 1)},
$$
$$
g _ {y} \colon = [ \nabla f (y _ {0}) | \nabla f (y _ {1}) | \dots | \nabla f (y _ {N}) ] \in \mathbb {R} ^ {d \times (N + 1)}.
$$
In this paragraph, we convert the  $\mathbf{U}$  of (11) and the  $\mathbf{V}$  of (12) into the trace of symmetric matrices. The key idea is: For each  $\langle a,b\rangle$  term where  $a,b\in \operatorname {span}\{\nabla f(x_0),\ldots ,\nabla f(x_N)\}$ , we can write  $a = g_{x}\mathbf{a}$ ,  $b = g_{x}\mathbf{b}$  for some  $\mathbf{a},\mathbf{b}\in \mathbb{R}^{(N + 1)\times 1}$ . Then
$$
\langle a, b \rangle = \left\langle g _ {x} \mathbf {a}, g _ {x} \mathbf {b} \right\rangle = \mathbf {b} ^ {\intercal} g _ {x} ^ {\intercal} g _ {x} \mathbf {a} = \operatorname {T r} \left(\mathbf {b} ^ {\intercal} g _ {x} ^ {\intercal} g _ {x} \mathbf {a}\right) = \operatorname {T r} \left(\mathbf {a b} ^ {\intercal} g _ {x} ^ {\intercal} g _ {x}\right) = \operatorname {T r} \left(g _ {x} \mathbf {a b} ^ {\intercal} g _ {x} ^ {\intercal}\right). \tag {16}
$$
Also note that  $(x_0 - x_i)$ ,  $(x_i - x_{i + 1})\in \operatorname {span}\{\nabla f(x_0),\ldots ,\nabla f(x_N)\}$  and  $(y_{i} - y_{N})$ $(y_{i}-$ $y_{i + 1})\in \operatorname {span}\{\nabla f(y_0),\dots ,\nabla f(y_N)\}$ . By using this technique, we observe that there exists  $S(H,u)$ ,  $\mathcal{T}(H^A,v)$  that satisfy
$$
(1 1) = \operatorname {T r} \left(g _ {x} \mathcal {S} (H, u) g _ {x} ^ {\intercal}\right), \qquad (1 2) = \operatorname {T r} \left(g _ {y} \mathcal {T} \left(H ^ {A}, v\right) g _ {y} ^ {\intercal}\right).
$$
From now, we specifically calculate  $\mathcal{S}(H,u)$  and  $\mathcal{T}(H^A,v)$ . Denote  $\{\mathbf{e}_i\}_{i=0}^N \in \mathbb{R}^{(N+1)\times 1}$  as a unit vector which  $(i+1)$ -th component is 1,  $\mathbf{e}_{-1} = \mathbf{e}_{N+1} = \mathbf{0}$  and define  $\mathcal{H}$  as
$$
\mathcal {H} = \left[ \begin{array}{c c} 0 & 0 \\ H & 0 \end{array} \right] \in R ^ {(N + 1) \times (N + 1)}.
$$
Then, by the definition of  $g_{x}$  and  $\mathcal{H}$ , we have
$$
g _ {x} \mathbf {e} _ {i} = \nabla f (x _ {i}) \quad 0 \leq i \leq N, \quad \frac {1}{L} g _ {x} \mathcal {H} ^ {\intercal} \mathbf {e} _ {0} = 0, \tag {17}
$$
$$
\frac {1}{L} g _ {x} \mathcal {H} ^ {\intercal} \mathbf {e} _ {i + 1} = \frac {1}{L} \sum_ {j = 0} ^ {i} h _ {i, j} g _ {x} \mathbf {e} _ {j} = \frac {1}{L} \sum_ {j = 0} ^ {i} h _ {i, j} \nabla f (x _ {j}) = x _ {i} - x _ {i + 1} \quad 0 \leq i \leq N - 1. \tag {18}
$$
Therefore, we can express (11) with  $\mathcal{H}$ ,  $\{u_i\}_{i=0}^N$ ,  $g_x$  and  $\{\mathbf{e}_i\}_{i=0}^N$  using (17) and (18) as
$$
\begin{array}{l} = - \frac {1}{2 L} \left\| \sum_ {i = 0} ^ {N} (u _ {i} - u _ {i - 1}) \nabla f (x _ {i}) \right\| ^ {2} + \sum_ {i = 0} ^ {N} (u _ {i} - u _ {i - 1}) \left(\langle \nabla f (x _ {i}), x _ {0} - x _ {i} \rangle + \frac {1}{2 L} \| \nabla f (x _ {i}) \| ^ {2}\right) \\ + \sum_ {i = 0} ^ {N - 1} u _ {i} \left(\langle \nabla f (x _ {i + 1}), x _ {i} - x _ {i + 1} \rangle + \frac {1}{2 L} \| \nabla f (x _ {i}) - \nabla f (x _ {i + 1}) \| ^ {2}\right) \\ = - \frac {1}{2 L} \left\| \sum_ {i = 0} ^ {N} \left(u _ {i} - u _ {i - 1}\right) g _ {x} \mathbf {e} _ {i} \right\| ^ {2} + \sum_ {i = 0} ^ {N} \left(u _ {i} - u _ {i - 1}\right) \left(\left\langle g _ {x} e _ {i}, \frac {1}{L} g _ {x} \mathcal {H} ^ {\intercal} \left(\mathbf {e} _ {0} + \dots + \mathbf {e} _ {i}\right) \right\rangle + \frac {1}{2 L} \| g _ {x} \mathbf {e} _ {i} \| ^ {2}\right) \\ + \sum_ {i = 0} ^ {N - 1} u _ {i} \left(\left\langle g _ {x} \mathbf {e} _ {i + 1}, \frac {1}{L} g _ {x} \mathcal {H} ^ {\intercal} \mathbf {e} _ {i + 1} \right\rangle + \frac {1}{2 L} \left\| g _ {x} (\mathbf {e} _ {i} - \mathbf {e} _ {i + 1}) \right\| ^ {2}\right). \\ \end{array}
$$
Using (16) induces  $(11) = \operatorname{Tr}\left(g_{x}S(H,u)g_{x}^{\intercal}\right)$  where
$$
\begin{array}{l} \mathcal {S} (H, u) \\ = - \frac {1}{2 L} \left(\sum_ {i = 0} ^ {N} \left(u _ {i} - u _ {i - 1}\right) \mathbf {e} _ {i}\right) \left(\sum_ {i = 0} ^ {N} \left(u _ {i} - u _ {i - 1}\right) \mathbf {e} _ {i}\right) ^ {\intercal} \\ + \frac {1}{2 L} \mathcal {H} ^ {\intercal} \left[ \sum_ {i = 0} ^ {N} u _ {i} \left(\mathbf {e} _ {0} + \dots + \mathbf {e} _ {i}\right) \left(\mathbf {e} _ {i} - \mathbf {e} _ {i + 1}\right) ^ {\intercal} \right] + \frac {1}{2 L} \left[ \sum_ {i = 0} ^ {N} u _ {i} \left(\mathbf {e} _ {i} - \mathbf {e} _ {i + 1}\right) \left(\mathbf {e} _ {0} + \dots + \mathbf {e} _ {i}\right) ^ {\intercal} \right] \mathcal {H} \\ + \frac {1}{2 L} \left[ \sum_ {i = 0} ^ {N} u _ {i} \left(\left(\mathbf {e} _ {i} - \mathbf {e} _ {i + 1}\right) \mathbf {e} _ {i} ^ {\intercal} + \mathbf {e} _ {i} \left(\mathbf {e} _ {i} - \mathbf {e} _ {i + 1}\right) ^ {\intercal}\right) - u _ {N} \mathbf {e} _ {N} \mathbf {e} _ {N} ^ {\intercal} \right]. \tag {19} \\ \end{array}
$$
Similarly, we calculate (12). Define  $\mathcal{H}^A$  as a anti-transpose matrix of  $\mathcal{H}$ :
$$
\mathcal {H} ^ {A} = \left[ \begin{array}{c c} 0 & 0 \\ H ^ {A} & 0 \end{array} \right] \in R ^ {(N + 1) \times (N + 1)}.
$$
Then, by the definition of  $g_{y}$  and  $\mathcal{H}^A$ , we have
$$
g _ {y} \mathbf {e} _ {i} = \nabla f (y _ {i}) \quad 0 \leq i \leq N, \quad \frac {1}{L} g _ {y} \left(\mathcal {H} ^ {A}\right) ^ {\top} \mathbf {e} _ {0} = 0, \tag {20}
$$
$$
\frac {1}{L} g _ {y} \left(\mathcal {H} ^ {A}\right) ^ {\intercal} \mathbf {e} _ {i + 1} = \frac {1}{L} \sum_ {j = 0} ^ {i} h _ {i, j} g _ {y} \mathbf {e} _ {j} = \frac {1}{L} \sum_ {j = 0} ^ {i} h _ {i, j} \nabla f (y _ {j}) = y _ {i} - y _ {i + 1} \quad 0 \leq i \leq N - 1. \tag {21}
$$
Therefore, we can express (12) with  $\mathcal{H}^A$ ,  $\{v_i\}_{i=0}^N$ ,  $g$  and  $\{\mathbf{e}_i\}_{i=0}^N$  using (20) and (21) as (12)
$$
\begin{array}{l} = \frac {v _ {0} - 1}{2 L} \| \nabla f (y _ {N}) \| ^ {2} + \sum_ {i = 0} ^ {N - 1} v _ {i + 1} \left(\langle \nabla f (y _ {i + 1}), y _ {i} - y _ {i + 1} \rangle + \frac {1}{2 L} \| \nabla f (y _ {i}) - \nabla f (y _ {i + 1}) \| ^ {2}\right) \\ + \sum_ {i = 0} ^ {N - 1} \left(v _ {i + 1} - v _ {i}\right) \left(\left\langle \nabla f \left(y _ {i}\right), y _ {N} - y _ {i} \right\rangle + \frac {1}{2 L} \left\| \nabla f \left(y _ {i}\right) - \nabla f \left(y _ {N}\right) \right\| ^ {2}\right) \\ = \frac {v _ {0} - 1}{2 L} \| g _ {y} \mathbf {e} _ {N} \| ^ {2} + \sum_ {i = 0} ^ {N - 1} v _ {i + 1} \left(\left\langle g _ {y} \mathbf {e} _ {i + 1}, \frac {1}{L} g _ {y} \left(\mathcal {H} ^ {A}\right) ^ {\top} \mathbf {e} _ {i + 1} \right\rangle + \frac {1}{2 L} \| g _ {y} (\mathbf {e} _ {i} - \mathbf {e} _ {i + 1}) \| ^ {2}\right) \\ + \sum_ {i = 0} ^ {N - 1} (v _ {i + 1} - v _ {i}) \left(- \left\langle g _ {y} \mathbf {e} _ {i}, \frac {1}{L} g _ {y} (\mathcal {H} ^ {A}) ^ {\intercal} (\mathbf {e} _ {i + 1} + \dots + \mathbf {e} _ {N}) \right\rangle + \frac {1}{2 L} \| g _ {y} (\mathbf {e} _ {i} - \mathbf {e} _ {N}) \| ^ {2}\right). \\ \end{array}
$$
We can write  $(12) = \operatorname{Tr}\left(g_y\mathcal{T}(H^A,v)g_y^\intercal\right)$  where
$$
\begin{array}{l} \mathcal {T} (H ^ {A}, v) \\ = \frac {1}{2 L} \left[ \sum_ {i = 0} ^ {N} v _ {i} \left((\mathbf {e} _ {i - 1} - \mathbf {e} _ {i}) (\mathbf {e} _ {i - 1} - \mathbf {e} _ {N}) ^ {\intercal} + (\mathbf {e} _ {i - 1} - \mathbf {e} _ {N}) (\mathbf {e} _ {i - 1} - \mathbf {e} _ {i}) ^ {\intercal}\right) \right] - \frac {v _ {0}}{2 L} \mathbf {e} _ {0} \mathbf {e} _ {0} ^ {\intercal} - \frac {1}{2 L} \mathbf {e} _ {N} \mathbf {e} _ {N} ^ {\intercal} \\ + \frac {1}{2 L} \left[ \sum_ {i = 0} ^ {N} v _ {i} \left(\left(\mathcal {H} ^ {A}\right) ^ {\intercal} \left(\mathbf {e} _ {i} + \dots + \mathbf {e} _ {N}\right) \left(\mathbf {e} _ {i} - \mathbf {e} _ {i - 1}\right) ^ {\intercal} + \left(\mathbf {e} _ {i} - \mathbf {e} _ {i - 1}\right) \left(\mathbf {e} _ {i} + \dots + \mathbf {e} _ {N}\right) ^ {\intercal} \mathcal {H} ^ {A}\right) \right] \\ = \frac {1}{2 L} \left[ \sum_ {i = 0} ^ {N} \frac {1}{u _ {N - i}} \left(\left(\mathbf {e} _ {i - 1} - \mathbf {e} _ {i}\right) \left(\mathbf {e} _ {i - 1} - \mathbf {e} _ {N}\right) ^ {\intercal} + \left(\mathbf {e} _ {i - 1} - \mathbf {e} _ {N}\right) \left(\mathbf {e} _ {i - 1} - \mathbf {e} _ {i}\right) ^ {\intercal}\right) \right] - \frac {1}{2 u _ {N} L} \mathbf {e} _ {0} \mathbf {e} _ {0} ^ {\intercal} - \frac {1}{2 L} \mathbf {e} _ {N} \mathbf {e} _ {N} ^ {\intercal} \\ + \frac {1}{2 L} \left[ \sum_ {i = 0} ^ {N} \frac {1}{u _ {N - i}} \left(\left(\mathcal {H} ^ {A}\right) ^ {\intercal} \left(\mathbf {e} _ {i} + \dots + \mathbf {e} _ {N}\right) \left(\mathbf {e} _ {i} - \mathbf {e} _ {i - 1}\right) ^ {\intercal} + \left(\mathbf {e} _ {i} - \mathbf {e} _ {i - 1}\right) \left(\mathbf {e} _ {i} + \dots + \mathbf {e} _ {N}\right) ^ {\intercal} \mathcal {H} ^ {A}\right) \right]. \tag {22} \\ \end{array}
$$
Finding auxiliary matrix  $\mathcal{M}(u)$  that gives the relation between  $S(H,u)$  and  $\mathcal{T}(H^A,v)$ . We can show that there exists an invertible  $M(u)\in \mathbb{R}^{(N + 1)\times (N + 1)}$  such that
$$
\mathcal {S} (H, u) = \mathcal {M} (u) ^ {\top} \mathcal {T} \left(H ^ {A}, v\right) \mathcal {M} (u). \tag {23}
$$
If we assume the above equation,
$$
\operatorname {T r} \left(g _ {x} \mathcal {S} (H, u) g _ {x} ^ {\intercal}\right) = \operatorname {T r} \left(g _ {y} \mathcal {T} \left(H ^ {A}, v\right) g _ {y} ^ {\intercal}\right) \tag {24}
$$
with  $g_{y} = g_{x}\mathcal{M}(u)^{\top}$ . Since  $\mathcal{M}(u)$  is invertible,
$$
\{g | g \in \mathbb {R} ^ {d \times (N + 1)} \} = \left\{g \mathcal {M} (u) ^ {\intercal} | g \in \mathbb {R} ^ {d \times (N + 1)} \right\}. \tag {25}
$$
$$
\left[ \mathbf {U} \geq 0 \quad \forall \nabla f (x _ {0}), \dots , \nabla f (x _ {N}) \right] \quad \Leftrightarrow \quad \left[ \operatorname {T r} \left(g _ {x} \mathcal {S} (H, u) g _ {x} ^ {\intercal}\right) \geq 0 \quad \forall g _ {x} \in \mathbb {R} ^ {d \times (N + 1)} \right]
$$
$$
\left[ \mathbf {V} \geq 0 \quad \forall \nabla f (y _ {0}), \ldots , \nabla f (y _ {N}) \right] \Leftrightarrow \left[ \operatorname {T r} \left(g _ {y} \mathcal {T} (H ^ {A}, u) g _ {y} ^ {\intercal}\right) \geq 0 \quad \forall g _ {y} \in \mathbb {R} ^ {d \times (N + 1)} \right].
$$
By combining (24) and (25), we obtain
$$
\begin{array}{l} \left[ \operatorname {T r} \left(g _ {x} \mathcal {S} (H, u) g _ {x} ^ {\intercal}\right) \geq 0 \quad \forall g _ {x} \in \mathbb {R} ^ {d \times (N + 1)} \right] \\ \Leftrightarrow \left[ \operatorname {T r} \left(g _ {y} \mathcal {S} (H, u) g _ {y} ^ {\intercal}\right) \geq 0 \quad \forall g _ {x} \in \mathbb {R} ^ {d \times (N + 1)}, g _ {y} = g _ {x} \mathcal {M} (u) ^ {\intercal} \right] \\ \Leftrightarrow \left[ \operatorname {T r} \left(g _ {y} \mathcal {S} (H, u) g _ {y} ^ {\intercal}\right) \geq 0 \quad \forall g _ {y} \in \mathbb {R} ^ {d \times (N + 1)} \right]. \\ \end{array}
$$
To sum up, we obtain (15)
$$
\left[ \mathbf {U} \geq 0 \quad \forall \nabla f (x _ {0}), \dots , \nabla f (x _ {N}) \right] \quad \Leftrightarrow \quad \left[ \mathbf {V} \geq 0 \quad \forall \nabla f (y _ {0}), \dots , \nabla f (y _ {N}) \right],
$$
Explicit form of  $\mathcal{M}(u)$  and justification of (23). Explicit form of  $\mathcal{M}(u)$  is
$$
\mathcal {M} = \left[ \begin{array}{c c c c c} 0 & \dots & 0 & 0 & u _ {N} \\ 0 & \dots & 0 & u _ {N - 1} & u _ {N} - u _ {N - 1} \\ 0 & \dots & u _ {N - 2} & u _ {N - 1} - u _ {N - 2} & u _ {N} - u _ {N - 1} \\ \vdots & \vdots & \vdots & \vdots & \vdots \\ u _ {0} & \dots & u _ {N - 2} - u _ {N - 3} & u _ {N - 1} - u _ {N - 2} & u _ {N} - u _ {N - 1} \end{array} \right] \in \mathbb {R} ^ {(N + 1) \times (N + 1)}. \tag {26}
$$
Now, we express  $\mathcal{M}(u) = \sum_{0\leq i,j\leq N}m_{ij}(u)\mathbf{e}_i\mathbf{e}_j^\intercal$ $\mathcal{S}(H,u) = \sum_{0\leq i,j\leq N}s_{ij}\mathbf{e}_i\mathbf{e}_j^\intercal$  and  $\mathcal{T}(H^A,v) = \sum_{0\leq i,j\leq N}t_{ij}\mathbf{e}_i\mathbf{e}_j^\intercal$  Calculating  $\mathcal{M}^{\intercal}(u)\mathcal{T}(H^{A},v)\mathcal{M}(u)$  gives