SlowGuess/NeurIPS_2023_OCR huggingface dataset:
[33] Giulia Luise, Alessandro Rudi, Massimiliano Pontil, and Carlo Ciliberto. Differential properties of sinkhorn approximation for learning with Wasserstein distance. In NeurIPS, volume 31, 2018.
[34] Jelena Luketina, Mathias Berglund, Klaus Greff, and Tapani Raiko. Scalable gradient-based tuning of continuous regularization hyperparameters. In ICML, pages 2952-2960, 2016.
[35] Sheheryar Mehmood and Peter Ochs. Automatic differentiation of some first-order methods in parametric optimization. In International Conference on Artificial Intelligence and Statistics, pages 1584-1594. PMLR, 2020.
[36] Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas Kopf, Edward Yang, Zachary DeVito, Martin Raison, Alykhan Tejani, Sasank Chilamkurthy, Benoit Steiner, Lu Fang, Junjie Bai, and Soumith Chintala. Pytorch: An imperative style, high-performance deep learning library. In Advances in Neural Information Processing Systems 32, pages 8024-8035, 2019.
[37] Edouard Pauwels and Samuel Vaiter. The derivatives of sinkhorn-knopp converge. SIAM J Optim, 33(3):1494-1517, 2023.
[38] Boris Polyak. Introduction to optimization. In Optimization Software, Publications Division. Citeseer, 1987.
[39] Zaccharie Ramzi, Florian Mannel, Shaojie Bai, Jean-Luc Starck, Philippe Ciuciu, and Thomas Moreau. SHINE: SHaring the INverse estimate from the forward pass for bi-level optimization and implicit models. In International Conference on Learning Representations, 2022.
[40] Subham Sekhar Sahoo, Anselm Paulus, Marin Vlastelica, Vít Musil, Volodymyr Kuleshov, and Georg Martius. Backpropagation through combinatorial algorithms: Identity with projection works. In The Eleventh International Conference on Learning Representations, 2023.
[41] Damien Scieur, Gauthier Gidel, Quentin Bertrand, and Fabian Pedregosa. The curse of unrolling: Rate of differentiating through optimization. Advances in Neural Information Processing Systems, 35:17133-17145, 2022.
[42] Amirreza Shaban, Ching-An Cheng, Nathan Hatch, and Byron Boots. Truncated back-propagation for bilevel optimization. In AISTATS, pages 1723-1732. PMLR, 2019.
[43] Lieven Vandenberghe. The cvxopt linear and quadratic cone program solvers. Online: http://cvxopt.org/documentation/conepprog.pdf, 2010.
[44] R. E. Wengert. A simple automatic derivative evaluation program. Communications of the ACM, 7(8):463-464, August 1964.
[45] Matt Johnson Zico Kolter, David Duvenaud. Deep implicit layers - neural odes, deep equilibrium models, and beyond, 2020. NeurIPS Tutorial.
A Proof of Section 3
Proof of Corollary 4: We have  $\nabla_{\theta}(g\circ \bar{x})(\theta) = J_{\theta}\bar{x} (\theta)^{T}\nabla g(\bar{x} (\theta))$  , and
$$
\begin{array}{l} J _ {\theta} \bar {x} (\theta) ^ {T} \nabla g (\bar {x} (\theta)) - J ^ {\mathrm {O S}} x _ {k} (\theta) ^ {T} \nabla g (x _ {k}) = \left(J _ {\theta} \bar {x} (\theta) ^ {T} - J ^ {\mathrm {O S}} x _ {k} (\theta) ^ {T}\right) \nabla g (\bar {x} (\theta)) \\ - J ^ {\mathrm {O S}} x _ {k} (\theta) ^ {T} \left(\nabla g (x _ {k}) - \nabla g (\bar {x} (\theta))\right). \\ \end{array}
$$
The result follows from the triangular inequality combined with Corollary 1 and the following
$$
\begin{array}{l} \left\| \left(J _ {\theta} \bar {x} (\theta) ^ {T} - J ^ {\mathrm {O S}} x _ {k} (\theta) ^ {T}\right) \nabla g (\bar {x} (\theta)) \right\| \leq \left\| J _ {\theta} \bar {x} (\theta) - J ^ {\mathrm {O S}} x _ {k} (\theta) \right\| _ {\mathrm {o p}} \| \nabla g (\bar {x} (\theta)) \| \\ \leq l _ {g} \| J _ {\theta} \bar {x} (\theta) - J ^ {\mathrm {O S}} x _ {k} (\theta) \| _ {\mathrm {o p}}, \\ \end{array}
$$
$$
\| J ^ {\mathrm {O S}} x _ {k} (\theta) ^ {T} (\nabla g (x _ {k}) - \nabla g (\bar {x} (\theta)) \| \leq \| J ^ {\mathrm {O S}} x _ {k} (\theta) \| _ {\mathrm {o p}} \| \nabla g (x _ {k}) - \nabla g (\bar {x} (\theta)) \| \leq L _ {F} l _ {\nabla} \| \bar {x} (\theta) - x _ {k} \|
$$
Proof of Lemma 3: We have, using the "descent lemma" (see [7]), for all  $l \in 0 \dots K$
$$
\begin{array}{l} h \left(\theta_ {l + 1}\right) - h \left(\theta_ {l}\right) \leq \left\langle \nabla h \left(\theta_ {l}\right), \theta_ {l + 1} - \theta_ {l} \right\rangle + \frac {L}{2} \left\| \theta_ {l + 1} - \theta_ {l} \right\| ^ {2} \\ = \frac {L}{2} \left(2 \left\langle \frac {\nabla h \left(\theta_ {l}\right)}{L}, \theta_ {l + 1} - \theta_ {l} \right\rangle + \left\| \theta_ {l + 1} - \theta_ {l} \right\| ^ {2}\right) \\ = \frac {L}{2} \left(\left\| \theta_ {l + 1} - \theta_ {l} + \frac {\nabla h (\theta_ {l})}{L} \right\| ^ {2} - \left\| \frac {\nabla h (\theta_ {l})}{L} \right\| ^ {2}\right) \\ \leq \frac {1}{2 L} \left(\epsilon^ {2} - \min _ {l = 0, \dots , K} \| \nabla h (\theta_ {l}) \| ^ {2}\right). \\ \end{array}
$$
Summing for  $l = 0,\dots ,K$  ，we have
$$
h ^ {*} - h (\theta_ {0}) \leq h (\theta_ {K + 1}) - h (\theta_ {0}) \leq \frac {K + 1}{2 L} \left(\epsilon^ {2} - \min  _ {l = 0, \dots , K} \| \nabla h (\theta_ {l}) \| ^ {2}\right).
$$
and we deduce, by using concavity of the square root, that
$$
\min  _ {l = 0, \dots , K} \| \nabla h (\theta_ {l}) \| ^ {2} \leq \epsilon^ {2} + \frac {2 L (h (x _ {0}) - h ^ {*})}{K + 1}.
$$
Lemma 4 Let  $A \colon \mathbb{R}^d \to \mathbb{R}^{n \times m}$  and  $B \colon \mathbb{R}^d \to \mathbb{R}^{m \times p}$ , be  $M_A$  and  $M_B$  bounded and  $L_A$  and  $L_B$  Lipschitz respectively, in operator norm, then  $A \cdot B$  is  $M_AL_B + L_AM_B$  Lipschitz in operator norm.
Proof: We have for all  $x, y \in \mathbb{R}^d$
$$
\begin{array}{l} \| A (x) B (x) - A (y) B (y) \| _ {\mathrm {o p}} = \| (A (x) - A (y)) B (x) - A (y) (B (y) - B (x)) \| _ {\mathrm {o p}} \\ \leq \| (A (x) - A (y)) B (x) \| _ {\mathrm {o p}} + \| A (y) (B (y) - B (x)) \| _ {\mathrm {o p}} \\ \leq \| A (x) - A (y) \| _ {\mathrm {o p}} \cdot \| B (x) \| _ {\mathrm {o p}} + \| A (y) \| _ {\mathrm {o p}} \cdot \| B (y) - B (x) \| _ {\mathrm {o p}} \\ \leq \left(L _ {A} M _ {B} + M _ {A} L _ {B}\right) \| x - y \| \\ \end{array}
$$
Lemma 5 The function  $M \mapsto (I - M)^{-1}$  is  $(1 - \rho)^{-2}$  Lipschitz in operator norm on the set of square matrices  $M$  such that  $\| M \|_{\mathrm{op}} \leq \rho < 1$ .
Proof: For any two square matrices  $A, B$  with operator norm bounded by  $\rho < 1$ ,  $I - A$  and  $I - B$  are invertible and the operator norm of the inverse is at most  $(1 - \rho)^{-1}$  (see proof of
Lemma 2). We have
$$
\begin{array}{l} (I - A) \left((I - A) ^ {- 1} - (I - B) ^ {- 1}\right) (I - B) = A - B \\ (I - A) ^ {- 1} - (I - B) ^ {- 1} = (I - A) ^ {- 1} (A - B) (I - B) ^ {- 1} \\ \left\| (I - A) ^ {- 1} - (I - B) ^ {- 1} \right\| _ {\mathrm {o p}} = \left\| (I - A) ^ {- 1} (A - B) (I - B) ^ {- 1} \right\| _ {\mathrm {o p}} \\ \leq \| (I - A) ^ {- 1} \| _ {\mathrm {o p}} \cdot \| (A - B) \| _ {\mathrm {o p}} \cdot \| (I - B) ^ {- 1} \| _ {\mathrm {o p}} \\ \leq \frac {\| (A - B) \| _ {\mathrm {o p}}}{(1 - \rho) ^ {2}} \\ \end{array}
$$
Proof of Corollary 5: Let us evaluate a Lipschitz constant of  $\nabla_{\theta}(g \circ \bar{x})$ . We have
$$
\begin{array}{l} \nabla_ {\theta} (g \circ \bar {x}) (\theta) = J _ {\theta} \bar {x} (\theta) ^ {T} \nabla g (\bar {x} (\theta)) \\ = \left(I - J _ {x} F (\bar {x}, \theta)\right) ^ {- 1} J _ {\theta} F (\bar {x}, \theta) ^ {T} \nabla g (\bar {x} (\theta)). \\ \end{array}
$$
This can be seen as a composition of a matrix function with  $\bar{x}$ . Combining Lemma 4 and Lemma 5 with the fact that  $J_{x}F$  and  $J_{\theta}F$  are both  $L_{J}$  Lipschitz and  $L_{F}$  bounded, the function  $(x,\theta)\mapsto (I - J_{x}F(x,\theta))^{-1}J_{\theta}f(x,\theta)$  is  $\frac{L_JL_F}{(1 - \rho)^{-2}} +\frac{L_J}{1 - \rho} = \frac{L_J}{1 - \rho}\left(\frac{L_F}{1 - \rho} +1\right)$  Lipschitz.
Invoking Lemma 4 and using the fact that the operator norm of a vector is the euclidean norm, we have that the function  $(x,\theta)\mapsto (I - J_{x}F(x,\theta))^{-1}J_{\theta}f(x,\theta)\nabla g(x)$  is  $\frac{L_J}{1 - \rho}\left(\frac{L_F}{1 - \rho} +1\right)l_g + l_\nabla \frac{L_F}{1 - \rho}$  Lipschitz.
From the implicit function theorem,  $\| J_{\theta}\bar{x}(\theta)\|_{\mathrm{op}} \leq L_F / (1 - \rho)$  so that  $\bar{x}$  is  $L_F / (1 - \rho)$ . Lipschitz and  $\nabla_{\theta}(g \circ \bar{x})$  is  $\left(\frac{L_J}{1 - \rho}\left(\frac{L_F}{1 - \rho} + 1\right)l_g + l_\nabla \frac{L_F}{1 - \rho}\right)\frac{L_F}{1 - \rho}$ . Lipschitz so the choice of  $\alpha$  is such that  $1 / \alpha$  a Lipschitz constant for the gradient.
Using Corollary 4, we are in the setting of Lemma 3 and the result follows.
B Experimental details Section 4
B.1 QP experiment
Implicit differentiation: We explicitly formed the Jacobians of the iterative process (function  $f$ ) and solved the underlying linear systems using a solver in jax. The inversion was not done based on VJP (using fixed point iterations or conjugate gradient) because, for the relatively small scale of our problems, pure linear algebra was more efficient. We did not form the full implicit Jacobian in equation (3) using full matrix inversion; we only solved a linear system involving a left incoming vector in equation (3). In this sense, we did implement VJP for implicit differentiation although we explicitly formed Jacobians because this was the most efficient.
Generation of random QPs: To generate the QP instances, the quantities  $n, m, p$  are varied on a logarithmic scale. We set  $Q = M^T M$  where  $M$  is of size  $n \times n$  with entries uniform in  $[-1, 1]$ . Constraint matrices  $A$  and  $G$  also have entries uniform in  $[-1, 1]$ , and we chose  $m$  and  $p$  to be smaller than  $n$  so that feasibility is generic and occurs with probability one.