SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Robin Manhaeve, Sebastijan Dumancic, Angelika Kimmig, Thomas Demeester, and Luc De Raedt. Deepproblog: Neural probabilistic logic programming. advances in neural information processing systems, 31, 2018.
Qing Li, Siyuan Huang, Yining Hong, Yixin Chen, Ying Nian Wu, and Song-Chun Zhu. Closed loop neural-symbolic learning via integrating neural perception, grammar parsing, and symbolic reasoning. In International Conference on Machine Learning, pages 5884-5894. PMLR, 2020.
Lieven Vandenberghe and Stephen Boyd. Convex optimization (Tutorial), volume 1. Cambridge university press Cambridge, 2004.
Charles R Henderson. Best linear unbiased estimation and prediction under a selection model. Biometrics, pages 423-447, 1975.
Åke Björck. Least squares methods. Handbook of numerical analysis, 1:465-652, 1990.
Trevor Hastie, Robert Tibshirani, Jerome H Friedman, and Jerome H Friedman. The elements of statistical learning: data mining, inference, and prediction, volume 2. Springer, 2009.
Suriya Gunasekar, Blake E Woodworth, Srinadh Bhojanapalli, Behnam Neyshabur, and Nati Srebro. Implicit regularization in matrix factorization. Advances in Neural Information Processing Systems, 30, 2017.
Samuel L Smith, Benoit Dherin, David GT Barrett, and Soham De. On the origin of implicit regularization in stochastic gradient descent. arXiv preprint arXiv:2101.12176, 2021.
Alnur Ali, Edgar Dobriban, and Ryan Tibshirani. The implicit regularization of stochastic gradient flow for least squares. In International conference on machine learning, pages 233-244. PMLR, 2020.
Emile van Krieken, Thiviyan Thanapalasingam, Jakub M Tomczak, Frank van Harmelen, and Annette ten Teije. A-nesi: A scalable approximate method for probabilistic neurosymbolic inference. arXiv preprint arXiv:2212.12393, 2022.
Pham Dinh Tao and Le Thi Hoai An. Convex analysis approach to dc programming: theory, algorithms and applications. Acta mathematica vietnamica, 22(1):289-355, 1997.
Alan L Yuille and Anand Rangarajan. The concave-convex procedure. Neural computation, 15(4): 915-936, 2003.
Thomas Lipp and Stephen Boyd. Variations and extension of the convex-concave procedure. Optimization and Engineering, 17:263-287, 2016.
Le Thi Hoai An and Pham Dinh Tao. Dc programming and dca: thirty years of developments. Mathematical Programming, 169(1):5-68, 2018.
Dimitri Bertsekas. Convex optimization algorithms. Athena Scientific, 2015.
Pierre Hansen, Brigitte Jaumard, Michèle Ruiz, and Junjie Xiong. Global minimization of indefinite quadratic functions subject to box constraints. Naval Research Logistics (NRL), 40(3):373-392, 1993.
Hoai An Le Thi and Pham Ding Tao. A continuous approach for globally solving linearly constrained quadratic. Optimization, 50(1-2):93-120, 2001.
Meisam Razaviyayn. Successive convex approximation: Analysis and applications. PhD thesis, University of Minnesota, 2014.
Amir Beck and Marc Teboulle. Global optimality conditions for quadratic optimization problems with binary constraints. SIAM journal on optimization, 11(1):179-188, 2000.
Yong Xia. New optimality conditions for quadratic optimization problems with binary constraints. Optimization letters, 3:253-263, 2009.
A Proofs of DC technique
Notations. We define  $\pmb{S} \coloneqq (\pmb{Q}^{\mathsf{T}}\pmb{Q} + \tau \pmb{I})$ ,  $s \coloneqq (\pmb{Q}^{\mathsf{T}}\pmb{q}_1 + \tau \pmb{q}_2)$ , and denote the largest eigenvalues and largest diagonal element of  $\pmb{S}$  by  $\sigma_{\mathrm{max}}$  and  $\delta_{\mathrm{max}}$ , respectively. Hence, the two problems can be equivalently rewritten as
$$
\left(\mathbf {P}\right) \min  _ {\boldsymbol {u} \in \{0, 1 \} ^ {n}} \boldsymbol {u} ^ {\top} \boldsymbol {S} \boldsymbol {u} - 2 s ^ {\top} \boldsymbol {u},
$$
$$
\left(\mathrm {P} _ {\mathrm {t}}\right) \min  _ {\boldsymbol {u} \in [ 0, 1 ] ^ {n}} \boldsymbol {u} ^ {\top} (\boldsymbol {S} - t \boldsymbol {I}) \boldsymbol {u} - (2 \boldsymbol {s} - t \boldsymbol {e}) ^ {\top} \boldsymbol {u}.
$$
Proof. The results are primarily based on Bertsekas [2015, Proposition 1.3.4]: the minima of a strictly concave function cannot be in the relative interior of the feasible set.
We first show that if  $t_0 \geq \sigma_{\mathrm{max}}$ , then the two problems are equivalent [Le Thi and Ding Tao, 2001, Theorem 1]. Specifically, since  $S - tI$  is negative definite, problem  $(\mathbf{P}_{\mathrm{t}})$  is strictly concave. Therefore, the minima should be in the vertex set of the feasible domain, which is consistent with problem (P).
We can further generalize this result to the case  $t_0 \geq \delta_{\mathrm{max}}$  [Hansen et al., 1993, Proposition 1]. In this case, considering the  $i$ -th component of  $\mathbf{u}$ , its second-order derivative in problem  $(\mathrm{P_t})$  is  $2(S_{ii} - t)$ . Similarly, the strict concavity of  $\mathbf{u}_i$  ensures a binary solution, indicating the equivalence of problems (P) and  $(\mathrm{P_t})$ .
Proof. The Karush-Kuhn-Tucker (KKT) conditions of the problem  $(\mathrm{P_t})$  are as follows.
$$
[ 2 \boldsymbol {S} \boldsymbol {u} - 2 t \boldsymbol {u} - 2 \boldsymbol {s} + t \boldsymbol {e} ] _ {i} - \alpha_ {i} + \beta_ {i} = \mathbf {0};
$$
$$
\boldsymbol {u} _ {i} \in [ 0, 1 ] ^ {n}; \quad \boldsymbol {\alpha} _ {i} \geq \mathbf {0}, \beta_ {i} \geq \mathbf {0};
$$
$$
\boldsymbol {\alpha} _ {i} \boldsymbol {u} _ {i} = 0, \quad \boldsymbol {\beta} _ {i} (\boldsymbol {u} _ {i} - 1) = 0; \quad i = 1, \ldots , n.
$$
where  $\alpha$  and  $\beta$  are multiplier vector. For  $u\in \{0,1\} ^n$ , the KKT condition is equivalent to
$$
\boldsymbol {\alpha} _ {i} = \left[ 2 \boldsymbol {S} \boldsymbol {u} - 2 t \boldsymbol {u} - 2 \boldsymbol {s} + t \boldsymbol {e} \right] _ {i} (1 - \boldsymbol {u} _ {i}) \geq 0, \quad \beta_ {i} = \left[ 2 \boldsymbol {S} \boldsymbol {u} - 2 t \boldsymbol {u} - 2 \boldsymbol {s} + t \boldsymbol {e} \right] _ {i} \boldsymbol {u} _ {i} \leq 0.
$$
By using  $(1 - 2\pmb{u}_i)\in \{-1,1\}$ , we can further combine the above two inequalities, and obtain
$$
2 [ \boldsymbol {S} \boldsymbol {u} - \boldsymbol {s} ] _ {i} (1 - 2 \boldsymbol {u} _ {i}) + t \geq 0, \quad i = 1, \dots , n.
$$
On the other hand, if  $2[\mathbf{S}\mathbf{u} - \mathbf{s}]_i(1 - 2\mathbf{u}_i) + t \geq 0$  holds for each  $i = 1, \dots, n$ , it is easy to check that  $\alpha \geq 0$  and  $\beta \geq 0$ , which proves the first part of the proposition.
The proof of the second part is a direct result of Beck and Teboulle [2000, Theorem 2.4]. To be specific, if  $\pmb{u}$  achieves a global minimum of (P), then  $q(\pmb{u}) \leq q(\pmb{u}^{\prime})$  for any  $\pmb{u}^{\prime} \in \{0,1\}^{n}$ . Hence, we only flip the  $i$ -th value of  $\pmb{u}$ , i.e., considering  $\pmb{u}_i$  and  $\pmb{u}_i^{\prime} = 1 - \pmb{u}_i$ , and it holds that
$$
\begin{array}{l} \boldsymbol {u} ^ {\top} \boldsymbol {S} \boldsymbol {u} - 2 \boldsymbol {s} ^ {\top} \boldsymbol {u} \leq (\boldsymbol {u} ^ {\prime}) ^ {\top} \boldsymbol {S} \boldsymbol {u} ^ {\prime} - 2 \boldsymbol {s} ^ {\top} \boldsymbol {u} ^ {\prime} \\ = \left(\boldsymbol {u} ^ {\top} \boldsymbol {S} \boldsymbol {u} - 2 \boldsymbol {s} ^ {\top} \boldsymbol {u}\right) + 2 \left[ \boldsymbol {S} \boldsymbol {u} - \boldsymbol {s} \right] _ {i} \left(1 - 2 \boldsymbol {u} _ {i}\right) + \boldsymbol {S} _ {i i}. \\ \end{array}
$$
Rearranging the inequality, we obtain
$$
2 [ \boldsymbol {S} \boldsymbol {u} - \boldsymbol {s} ] _ {i} (1 - 2 \boldsymbol {u} _ {i}) \geq - \boldsymbol {S} _ {i i}, \quad i = 1, \dots , n,
$$
Proof. Notations. We use  $\|\cdot\|$  to denote the  $\ell_2$  norm for vectors and Frobenius norm for matrices. We define
$$
\varphi (\boldsymbol {\phi}, \boldsymbol {\theta}, \mathbf {Z}, \mathbf {Y}) := \| \mathbf {Z} \boldsymbol {w} _ {u} + \mathbf {Y} \boldsymbol {w} _ {v} - \boldsymbol {b} \| ^ {2} + \alpha \| (\mathbf {Z}, \mathbf {Y}) - (f _ {\boldsymbol {\theta}} (\mathbf {X}), \mathbf {Y}) \| ^ {2} + \lambda \| \boldsymbol {w} - \boldsymbol {w} ^ {0} \| ^ {2}.
$$
For the loss functions of logic programming and network training, we assume  $\ell_1(\pmb{\theta})$  and  $\ell_2(\phi)$  to be  $\mu_{\theta}$  and  $\mu_{\phi}$  smooth, respectively. For ease of presentation, we define  $\Delta^k = f_{\pmb{\theta}^k}(\mathbf{X})\pmb{w}_u^k + \mathbf{Y}\pmb{w}_v^k - \pmb{b}$ ,
and let  $c_{\mathrm{max}}$  be the upper bound of  $\| \Delta^k\|$ . Furthermore, by using the Woodbury identity formula, we can compute
$$
\begin{array}{l} \left(\mathbf {Z} ^ {k}; \mathbf {Y} ^ {k}\right) = \arg \min  _ {\left(\mathbf {Z}, \mathbf {Y}\right)} \left\| \mathbf {Z} \boldsymbol {w} _ {u} ^ {k} + \mathbf {Y} \boldsymbol {w} _ {v} ^ {k} - \boldsymbol {b} \right\| ^ {2} + \alpha \left\| \left(\mathbf {Z}, \mathbf {Y}\right) - \left(f _ {\boldsymbol {\theta} ^ {k}} (\mathbf {X}), \mathbf {Y}\right) \right\| ^ {2} + \lambda \| \boldsymbol {w} - \boldsymbol {w} ^ {0} \| ^ {2} \\ = \left(f _ {\boldsymbol {\theta} ^ {k}} (\mathbf {X}); \mathbf {Y}\right) - \beta^ {k} \Delta^ {k} \left(\boldsymbol {w} ^ {k}\right) ^ {\top}, \quad \text {w h e r e} \quad \beta^ {k} = \frac {1}{\alpha + \| \boldsymbol {w} ^ {k} \| ^ {2}}. \\ \end{array}
$$
Let  $\rho^k \coloneqq (\alpha \beta^k)$ , we have
$$
\begin{array}{l} \varphi \left(\phi^ {k}, \boldsymbol {\theta} ^ {k}, f _ {\boldsymbol {\theta} ^ {k}} (\mathbf {X}), \mathbf {Y}\right) - \varphi \left(\phi^ {k}, \boldsymbol {\theta} ^ {k}, \mathbf {Z} ^ {k}, \mathbf {Y} ^ {k}\right) = \left(1 - \left(\left(\alpha \beta^ {k}\right) ^ {2} + \left(1 - \alpha \beta^ {k}\right) ^ {2}\right) \| \Delta^ {k} \| ^ {2} \right. \\ = 2 \rho^ {k} (1 - \rho^ {k}) \| \Delta^ {k} \| ^ {2}. \\ \end{array}
$$
Update of  $\phi$ . We consider the single rule case (multiple rules can be directly decomposed), i.e.,  $\overline{\phi = (\boldsymbol{w},\boldsymbol{b})}$  and  $\boldsymbol{b} = (b;\ldots ;b)$ . The update of  $\phi$  is conducted on the loss function
$$
\ell_ {2} ^ {k} (\boldsymbol {w}, \boldsymbol {b}) = \varphi \left(\phi^ {k}, \boldsymbol {\theta} ^ {k}, f _ {\boldsymbol {\theta} ^ {k}} (\mathbf {X}), \mathbf {Y}\right) = \| f _ {\boldsymbol {\theta} ^ {k}} (\mathbf {X}) \boldsymbol {w} _ {u} + \mathbf {Y} \boldsymbol {w} _ {v} - \boldsymbol {b} \| ^ {2} + \lambda \| \boldsymbol {w} - \boldsymbol {w} ^ {0} \| ^ {2}.
$$
The smallest and the largest eigenvalues of  $(f_{\pmb{\theta}^k}(\mathbf{X}),\mathbf{Y})^\top (f_{\pmb{\theta}^k}(\mathbf{X}),\mathbf{Y}) + \lambda I$  are denoted by  $\sigma_{\mathrm{min}}$  and  $\sigma_{\mathrm{max}}$ , respectively.
The PPA method updates  $\pmb{w}$  by
$$
\boldsymbol {w} ^ {k + 1} = \arg \min _ {\boldsymbol {w}} \ell_ {2} ^ {k} (\boldsymbol {w}, \boldsymbol {b}) + \frac {1}{\gamma} \| \boldsymbol {w} - \boldsymbol {w} ^ {k} \| ^ {2},
$$
which can be reduced to
$$
\boldsymbol {w} ^ {k + 1} - \boldsymbol {w} ^ {k} = - M ^ {k} \delta^ {k}, \quad \boldsymbol {\delta} ^ {k} = \left(f _ {\boldsymbol {\theta} ^ {k}} (\mathbf {X}), \mathbf {Y}\right) ^ {\top} \Delta = \nabla_ {\boldsymbol {w}} \ell_ {2} ^ {k} (\boldsymbol {w}, \boldsymbol {b}),
$$
$$
\boldsymbol {M} ^ {k} = \left(\left(f _ {\boldsymbol {\theta} ^ {k}} (\mathbf {X}), \mathbf {Y}\right) ^ {\top} \left(f _ {\boldsymbol {\theta} ^ {k}} (\mathbf {X}), \mathbf {Y}\right) + \lambda \boldsymbol {I} + \frac {1}{\gamma} \boldsymbol {I}\right) ^ {- 1}.
$$
The  $(2 / \gamma)$ -strongly convexity of the proximal term implies the Polyak-Łojasiewicz (PL) inequality, which derives that
$$
\begin{array}{l} \varphi \left(\boldsymbol {\phi} ^ {k}, \boldsymbol {\theta} ^ {k}, \mathbf {Z} ^ {k}, \mathbf {Y} ^ {k}\right) = \varphi \left(\boldsymbol {\phi} ^ {k}, \boldsymbol {\theta} ^ {k}, f _ {\boldsymbol {\theta} ^ {k}} (\mathbf {X}), \mathbf {Y}\right) - 2 \rho^ {k} \left(1 - \rho^ {k}\right) \| \Delta \| ^ {2} \\ = \ell_ {2} ^ {k} (\boldsymbol {w} ^ {k}, \boldsymbol {b}) - 2 \rho^ {k} (1 - \rho^ {k}) \| \Delta^ {k} \| ^ {2} \geq \ell_ {2} ^ {k} (\boldsymbol {w} ^ {k + 1}, \boldsymbol {b}) - 2 \rho^ {k} (1 - \rho^ {k}) \| \Delta \| ^ {2} + \frac {2}{\gamma} \| \boldsymbol {w} ^ {k + 1} - \boldsymbol {w} ^ {k} \| ^ {2}. \\ \end{array}
$$
Plugging  $\pmb{w}^{k + 1} - \pmb{w}^k = -\pmb{M}^k\delta^k$  into the inequality, we have
$$
\begin{array}{l} \varphi \left(\boldsymbol {\phi} ^ {k}, \boldsymbol {\theta} ^ {k}, \mathbf {Z} ^ {k}, \mathbf {Y} ^ {k}\right) \geq \ell_ {2} ^ {k} (\boldsymbol {w} ^ {k + 1}, \boldsymbol {b}) + \frac {2}{\gamma} \left(\boldsymbol {\delta} ^ {k}\right) ^ {\top} \left(\boldsymbol {M} ^ {k}\right) ^ {2} \boldsymbol {\delta} ^ {k} - 2 \rho^ {k} \left(1 - \rho^ {k}\right) \| \Delta^ {k} \| ^ {2} \\ \geq \varphi (\pmb {\phi} ^ {k + 1}, \pmb {\theta} ^ {k}, \pmb {Z} ^ {k + \frac {1}{2}}, \pmb {Y} ^ {k + \frac {1}{2}}) + \frac {2}{\gamma} (\pmb {\delta} ^ {k}) ^ {\top} (\pmb {M} ^ {k}) ^ {2} \pmb {\delta} ^ {k} - 2 \rho^ {k} (1 - \rho^ {k}) \| \Delta^ {k} \| ^ {2}, \\ \end{array}
$$
$$
\begin{array}{l} \left(\mathbf {Z} ^ {k + \frac {1}{2}}; \mathbf {Y} ^ {k + \frac {1}{2}}\right) = \arg \min  _ {(\bar {\mathbf {Z}}, \bar {\mathbf {Y}})} \| \bar {\mathbf {Z}} \boldsymbol {w} _ {u} ^ {k + 1} + \bar {\mathbf {Y}} \boldsymbol {w} _ {v} ^ {k + 1} - \boldsymbol {b} \| ^ {2} + \alpha \| (\bar {\mathbf {Z}}, \bar {\mathbf {Y}}) - (f _ {\boldsymbol {\theta} ^ {k}} (\mathbf {X}), \mathbf {Y}) \| ^ {2} \\ = \left(f _ {\boldsymbol {\theta} ^ {k}} (\mathbf {X}); \mathbf {Y}\right) - \beta^ {k + \frac {1}{2}} \Delta^ {k + \frac {1}{2}} \left(\boldsymbol {w} ^ {k + 1}\right) ^ {\mathsf {T}}, \quad \text {w h e r e} \quad \beta^ {k + \frac {1}{2}} = \frac {1}{\alpha + \| \boldsymbol {w} ^ {k + 1} \| ^ {2}}. \\ \end{array}
$$
Note that  $(M^k)^2$  has the smallest eigenvalue  $\gamma^2 / (1 + \gamma \sigma_{\max})^2$ , and thus we have
$$
\varphi (\boldsymbol {\phi} ^ {k}, \boldsymbol {\theta} ^ {k}, \mathbf {Z} ^ {k}, \mathbf {Y} ^ {k}) \geq \varphi (\boldsymbol {\phi} ^ {k + 1}, \boldsymbol {\theta} ^ {k}, \mathbf {Z} ^ {k + \frac {1}{2}}, \mathbf {Y} ^ {k + \frac {1}{2}}) + \frac {2 \gamma}{(1 + \gamma \sigma_ {\max }) ^ {2}} \| \nabla_ {\boldsymbol {\phi}} \ell_ {2} (\boldsymbol {w}, \boldsymbol {b}) \| ^ {2} - 2 \rho^ {k} (1 - \rho^ {k}) c _ {\max }.
$$
Update of  $\theta$ . The update of  $\theta$  is conducted on the loss function
$$
\ell_ {1} ^ {k} (\pmb {\theta}) = \| \mathbf {Z} ^ {k + \frac {1}{2}} - f _ {\pmb {\theta}} (\mathbf {X}) \| ^ {2}.
$$
By using  $\mu_{\theta}$ -smooth of  $\ell_1^k$ , we obtain that
$$
\begin{array}{l} \varphi \left(\phi^ {k + 1}, \boldsymbol {\theta} ^ {k}, \mathbf {Z} ^ {k + \frac {1}{2}}, \mathbf {Y} ^ {k + \frac {1}{2}}\right) - \varphi \left(\phi^ {k + 1}, \boldsymbol {\theta} ^ {k + 1}, \mathbf {Z} ^ {k + \frac {1}{2}}, \mathbf {Y} ^ {k + \frac {1}{2}}\right) = \ell_ {1} ^ {k} \left(\boldsymbol {\theta} ^ {k + 1}\right) - \ell_ {1} ^ {k} \left(\boldsymbol {\theta} ^ {k}\right) \\ \geq - \left\langle \nabla_ {\boldsymbol {\theta}} \ell_ {1} ^ {k} \left(\boldsymbol {\theta} ^ {k}\right), \boldsymbol {\theta} ^ {k + 1} - \boldsymbol {\theta} ^ {k} \right\rangle - \frac {\mu_ {\theta}}{2} \left\| \boldsymbol {\theta} ^ {k + 1} - \boldsymbol {\theta} ^ {k} \right\| ^ {2} \geq \frac {1}{2} \eta \left\| \nabla_ {\boldsymbol {\theta}} \ell_ {1} ^ {k} \left(\boldsymbol {\theta} ^ {k}\right) \right\| ^ {2}. \\ \end{array}
$$
Letting  $\mathbf{Z}^{k + 1} = \arg \min_{\mathbf{Z}}\varphi (\phi^{k + 1},\pmb{\theta}^{k + 1},\mathbf{Z})$  , we conclude
$$
\varphi (\boldsymbol {\phi} ^ {k + 1}, \boldsymbol {\theta} ^ {k}, \mathbf {Z} ^ {k + \frac {1}{2}}, \mathbf {Y} ^ {k + \frac {1}{2}}) \geq \varphi (\boldsymbol {\phi} ^ {k + 1}, \boldsymbol {\theta} ^ {k + 1}, \mathbf {Z} ^ {k + 1}, \mathbf {Y} ^ {k + 1}) + \frac {1}{2} \eta \| \nabla_ {\boldsymbol {\theta}} \ell_ {1} ^ {k} (\boldsymbol {\theta} ^ {k}) \| ^ {2}.
$$
Convergent result. By combining the update of  $\phi$  and  $\theta$ , we have
$$
\begin{array}{l} \varphi \left(\boldsymbol {\phi} ^ {k}, \boldsymbol {\theta} ^ {k}, \mathbf {Z} ^ {k}, \mathbf {Y} ^ {k}\right) - \varphi \left(\boldsymbol {\phi} ^ {k + 1}, \boldsymbol {\theta} ^ {k + 1}, \mathbf {Z} ^ {k + 1}, \mathbf {Y} ^ {k + 1}\right) \\ \geq \frac {1}{2} \eta \| \nabla_ {\boldsymbol {\theta}} \ell_ {1} ^ {k} (\boldsymbol {\theta} ^ {k}) \| ^ {2} + \frac {2 \gamma}{(1 + \gamma \sigma_ {\max }) ^ {2}} \| \nabla_ {\phi} \ell_ {2} (\boldsymbol {\phi} ^ {k}) \| ^ {2} - 2 \rho^ {k} (1 - \rho^ {k}) c _ {\max }. \\ \end{array}
$$
Taking a telescopic sum over  $k$ , we obtain
$$
\begin{array}{l} \varphi \left(\boldsymbol {\phi} ^ {0}, \boldsymbol {\theta} ^ {0}, \mathbf {Z} ^ {0}, \mathbf {Y} ^ {0}\right) - \varphi \left(\boldsymbol {\phi} ^ {K}, \boldsymbol {\theta} ^ {K}, \mathbf {Z} ^ {K}, \mathbf {Y} ^ {K}\right) \\ \geq \sum_ {i = 1} ^ {K} \frac {1}{2} \eta \| \nabla_ {\boldsymbol {\theta}} \ell_ {1} ^ {k} (\boldsymbol {\theta} ^ {k}) \| ^ {2} + \frac {2 \gamma}{(1 + \gamma \sigma_ {\max }) ^ {2}} \| \nabla_ {\phi} \ell_ {2} (\boldsymbol {\phi} ^ {k}) \| ^ {2} - 2 \rho^ {k} (1 - \rho^ {k}) c _ {\max }. \\ \end{array}
$$
Since  $\rho^k (1 - \rho^k)\leq \kappa_\rho /(K + 1)^2$  , we have
$$
\mathbb {E} [ \| \nabla_ {\pmb {\theta}} \ell_ {2} ^ {k} (\pmb {\theta} ^ {k}) \| ^ {2} ] \leq \frac {2}{(K + 1) \eta} \big ((\varphi (\phi^ {0}, \pmb {\theta} ^ {0}, \mathbf {Z} ^ {0}, \mathbf {Y} ^ {0}) - \min \varphi) + 2 \kappa c _ {\max} \big),
$$
$$
\mathbb {E} [ \| \nabla_ {\phi} \ell_ {2} (\boldsymbol {\phi} ^ {k}) \| ^ {2} ] \leq \frac {(1 + \gamma \sigma_ {\max }) ^ {2}}{2 (K + 1)} \big ((\varphi (\boldsymbol {\phi} ^ {0}, \boldsymbol {\theta} ^ {0}, \mathbf {Z} ^ {0}, \mathbf {Y} ^ {0}) - \min \varphi) + 2 \kappa c _ {\max } \big).
$$
Stochastic version. We first introduce an additional assumption: the gradient estimate is unbiased and has bounded variance [Bottou et al., 2018, Sec. 4], i.e.,
$$
\mathbb {E} _ {\boldsymbol {\xi}} [ \tilde {\nabla} _ {\boldsymbol {\theta}} \ell_ {1} ^ {k} (\boldsymbol {\theta} ^ {k}) ] = \nabla_ {\boldsymbol {\theta}} \ell_ {1} ^ {k} (\boldsymbol {\theta} ^ {k}), \quad \mathbb {E} _ {\boldsymbol {\xi}} [ \tilde {\nabla} _ {\boldsymbol {\theta}} \ell_ {2} ^ {k} (\boldsymbol {\phi} ^ {k}) ] = \nabla_ {\boldsymbol {\theta}} \ell_ {2} ^ {k} (\boldsymbol {\phi} ^ {k}),
$$
$$
\mathbb {V} _ {\xi} \left[ \tilde {\nabla} _ {\boldsymbol {\theta}} \ell_ {1} ^ {k} \left(\boldsymbol {\theta} ^ {k}\right) \right] \leq \zeta + \zeta_ {v} \| \nabla_ {\boldsymbol {\theta}} \ell_ {1} ^ {k} \left(\boldsymbol {\theta} ^ {k}\right) \| ^ {2}, \quad \mathbb {V} _ {\xi} \left[ \tilde {\nabla} _ {\phi} \ell_ {1} ^ {k} \left(\boldsymbol {\theta} ^ {k}\right) \right] \leq \zeta + \zeta_ {v} \| \nabla_ {\phi} \ell_ {2} ^ {k} \left(\boldsymbol {\phi} ^ {k}\right) \| ^ {2}.
$$
This assumption derives the following inequalities hold for  $\zeta_g = \zeta_v + 1$ :
$$
\mathbb {E} _ {\boldsymbol {\xi}} \left[ \| \tilde {\nabla} _ {\boldsymbol {\theta}} \ell_ {1} ^ {k} (\boldsymbol {\theta} ^ {k}) \| ^ {2} \right] \leq \zeta + \zeta_ {g} \| \nabla_ {\boldsymbol {\theta}} \ell_ {1} ^ {k} (\boldsymbol {\theta} ^ {k}) \| ^ {2}, \quad \mathbb {E} _ {\boldsymbol {\xi}} \left[ \| \tilde {\nabla} _ {\boldsymbol {\theta}} \ell_ {2} ^ {k} (\boldsymbol {\phi} ^ {k}) \| ^ {2} \right] \leq \zeta + \zeta_ {g} \| \nabla_ {\boldsymbol {\phi}} \ell_ {2} ^ {k} (\boldsymbol {\phi} ^ {k}) \| ^ {2},
$$
For the update of  $\theta$ , we have
$$
\varphi \left(\boldsymbol {\phi} ^ {k + 1}, \boldsymbol {\theta} ^ {k}, \mathbf {Z} ^ {k + \frac {1}{2}}, \mathbf {Y} ^ {k + \frac {1}{2}}\right) - \mathbb {E} _ {\xi} \left[ \varphi \left(\boldsymbol {\phi} ^ {k + 1}, \boldsymbol {\theta} ^ {k + 1}, \mathbf {Z} ^ {k + 1}, \mathbf {Y} ^ {k + 1}\right) \right] \geq \frac {\eta_ {k}}{2} \| \nabla_ {\boldsymbol {\theta}} \ell_ {1} ^ {k} \left(\boldsymbol {\theta} ^ {k}\right) \| ^ {2} - \frac {\eta_ {k} ^ {2} \mu_ {\boldsymbol {\theta}}}{2} \zeta .
$$
For the update of  $\phi$ , using the  $\mu_{\phi}$ -smooth, and taking the total expectation:
$$
\begin{array}{l} \varphi \left(\phi^ {k}, \boldsymbol {\theta} ^ {k}, \mathbf {Z} ^ {k}, \mathbf {Y} ^ {k}\right) - \mathbb {E} _ {\xi} \left[ \varphi \left(\phi^ {k + 1}, \boldsymbol {\theta} ^ {k}, \mathbf {Z} ^ {k + \frac {1}{2}}, \mathbf {Y} ^ {k + \frac {1}{2}}\right) \right] + 2 \rho^ {k} \left(1 - \rho^ {k}\right) \| \Delta^ {k} \| ^ {2} \\ \geq \left(\nabla_ {\phi} \ell_ {2} ^ {k} \left(\phi^ {k}\right)\right) ^ {\top} M ^ {k} \left(\nabla_ {\phi} \ell_ {2} ^ {k} \left(\phi^ {k}\right)\right) - \frac {\mu_ {\phi}}{2} \mathbb {E} _ {\xi} [ \| \tilde {M} ^ {k} \tilde {\nabla} _ {\phi} \ell_ {2} ^ {k} \left(\phi^ {k}\right) \| ^ {2} ] \\ \geq \frac {1}{\epsilon^ {k} + \sigma_ {\max}} \| \nabla_ {\phi} \ell_ {2} ^ {k} (\phi^ {k}) \| ^ {2} - \frac {\mu_ {\phi}}{2 (\epsilon^ {k} + \sigma_ {\min}) ^ {2}} (\zeta + \zeta_ {g} \| \nabla_ {\phi} \ell_ {2} ^ {k} (\phi^ {k}) \| ^ {2}), \\ \end{array}
$$
where we define  $\epsilon^k = 1 / \gamma^k$  for simplicity. Now, let  $\gamma$  be sufficiently small (that is, satisfying  $(\epsilon^k + \sigma_{\min})^2 \geq \mu_\phi (\epsilon^k + \sigma_{\max}))$ , we obtain
$$
\begin{array}{l} \varphi \left(\phi^ {k}, \boldsymbol {\theta} ^ {k}, \mathbf {Z} ^ {k}, \mathbf {Y} ^ {k}\right) - \mathbb {E} _ {\xi} \left[ \varphi \left(\phi^ {k + 1}, \boldsymbol {\theta} ^ {k}, \mathbf {Z} ^ {k + \frac {1}{2}}, \mathbf {Y} ^ {k + \frac {1}{2}}\right) \right] + 2 \rho^ {k} \left(1 - \rho^ {k}\right) \| \Delta^ {k} \| ^ {2} \\ \geq \frac {1}{2 \left(\epsilon^ {k} + \sigma_ {\max}\right)} \| \nabla_ {\phi} \ell_ {2} ^ {k} (\phi^ {k}) \| ^ {2} - \frac {\mu_ {\phi}}{2 \left(\epsilon^ {k} + \sigma_ {\min}\right) ^ {2}} \zeta . \\ \end{array}
$$
Putting the updates of  $\theta$  and  $\phi$  together, we have
$$
\begin{array}{l} \varphi \left(\boldsymbol {\phi} ^ {k}, \boldsymbol {\theta} ^ {k}, \mathbf {Z} ^ {k}, \mathbf {Y} ^ {k}\right) - \mathbb {E} _ {\xi} \left[ \varphi \left(\boldsymbol {\phi} ^ {k + 1}, \boldsymbol {\theta} ^ {k + 1}, \mathbf {Z} ^ {k + 1}, \mathbf {Y} ^ {k + 1}\right) \right] + 2 \rho^ {k} \left(1 - \rho^ {k}\right) \| \Delta^ {k} \| ^ {2} \\ \geq \frac {1}{2} \eta_ {k} \| \nabla_ {\pmb {\theta}} \ell_ {1} ^ {k} (\pmb {\theta} ^ {k}) \| ^ {2} - \frac {1}{2} \eta_ {k} ^ {2} \mu_ {\pmb {\theta}} \zeta + \frac {1}{2 (\epsilon^ {k} + \sigma_ {\max})} \| \nabla_ {\phi} \ell_ {2} ^ {k} (\phi^ {k}) \| ^ {2} - \frac {\mu_ {\phi}}{2 (\epsilon^ {k} + \sigma_ {\min}) ^ {2}} \zeta . \\ \end{array}
$$
Now, setting  $\eta^k\leq \kappa_\theta /\sqrt{K + 1}$  and  $\gamma^k\leq \kappa_\phi /\sqrt{K + 1}$ , we can conclude
$$
\mathbb {E} [ \| \nabla_ {\boldsymbol {\theta}} \ell_ {1} ^ {k} (\boldsymbol {\theta} ^ {k}) \| ^ {2} ] = \mathcal {O} (\frac {1}{\sqrt {K + 1}}), \quad \mathbb {E} [ \| \nabla_ {\boldsymbol {\phi}} \ell_ {2} (\boldsymbol {\phi} ^ {k}) \| ^ {2} ] = \mathcal {O} (\frac {1}{\sqrt {K + 1}}).
$$
Proof. We consider the following problem,
$$
\left(\mathrm {P} _ {\xi}\right) \min  _ {\boldsymbol {u} \in \{0, 1 \} ^ {n}} q _ {\xi} (\boldsymbol {u}) := \boldsymbol {u} ^ {\top} (\boldsymbol {S} + \lambda \boldsymbol {I}) \boldsymbol {u} - 2 (\boldsymbol {s} + \lambda \boldsymbol {\xi}) ^ {\top} \boldsymbol {u}.
$$
For given  $t \geq 0$ , the corresponding stationary points of  $(\mathrm{P}_{\xi})$  satisfy
$$
2 [ \boldsymbol {S} \boldsymbol {u} - \boldsymbol {s} ] _ {i} (1 - 2 \boldsymbol {u} _ {i}) + 2 \lambda (\boldsymbol {u} _ {i} - \boldsymbol {\xi} _ {i}) (1 - 2 \boldsymbol {u} _ {i}) + t \geq 0, i = 1, \dots , n.
$$
$$
(\boldsymbol {u} _ {i} - \boldsymbol {\xi} _ {i}) (1 - 2 \boldsymbol {u} _ {i}) = \left\{ \begin{array}{l l} - \boldsymbol {\xi} _ {i} & \text {i f} \quad \boldsymbol {u} _ {i} = 0; \\ \boldsymbol {\xi} _ {i} - 1 & \text {i f} \quad \boldsymbol {u} _ {i} = 1. \end{array} \right.
$$
For given  $\pmb{u} \in \{0,1\}^n$ , we denote  $\varrho_{i} = 2[\pmb{S}\pmb{u} - \pmb{s}]_{i}(1 - 2\pmb{u}_{i})$ . Then, the probability that  $(\mathrm{P}_{\xi})$  has the stationary point  $\pmb{u}$  can be computed as
$$
\Pr (\boldsymbol {u}) = \prod_ {i = 1} ^ {n} \Pr \left(\varrho_ {i} + 2 \lambda \left(\boldsymbol {u} _ {i} - \boldsymbol {\xi} _ {i}\right) (1 - 2 \boldsymbol {u} _ {i}) + t \geq 0\right),
$$
$$
\Pr (2 \lambda (\boldsymbol {u} _ {i} - \boldsymbol {\xi} _ {i}) (1 - 2 \boldsymbol {u} _ {i}) + \varrho_ {i} + t \geq 0) = \min  (\frac {1}{2 \lambda} (t + \varrho_ {i}), 1).
$$