BPQP: A Differentiable Convex Optimization Framework for Efficient End-to-End Learning
Authors: Jianming Pan
Zeqi Ye
Xiao Yang
Xu Yang
Weiqing Liu
Lewen Wang
Jiang Bian
Venue: NeurIPS 2024 spotlight (2024)
Keywords: Differentiable optimization layers
machine learning
end-to-end learning
Primary area: optimization

Abstract: Data-driven decision-making processes increasingly utilize end-to-end learnable deep neural networks to render final decisions. Sometimes, the output of the forward functions in certain layers is determined by the solutions to mathematical optimization problems, leading to the emergence of differentiable optimization layers that permit gradient back-propagation.
However, real-world scenarios often involve large-scale datasets and numerous constraints, presenting significant challenges. Current methods for differentiating optimization problems typically rely on implicit differentiation, which necessitates costly computations on the Jacobian matrices, resulting in low efficiency.
In this paper, we introduce BPQP, a differentiable convex optimization framework designed for efficient end-to-end learning. To enhance efficiency, we reformulate the backward pass as a simplified and decoupled quadratic programming problem by leveraging the structural properties of the Karush–Kuhn–Tucker (KKT) matrix. This reformulation enables the use of first-order optimization algorithms in calculating the backward pass gradients, allowing our framework to potentially utilize any state-of-the-art solver. As solver technologies evolve, BPQP can continuously adapt and improve its efficiency.
Extensive experiments on both simulated and real-world datasets demonstrate that BPQP achieves a significant improvement in efficiency—typically an order of magnitude faster in overall execution time compared to other differentiable optimization layers. Our results not only highlight the efficiency gains of BPQP but also underscore its superiority over differential optimization layer baselines.

BPQP: A Differentiable Convex Optimization Framework for Efficient End-to-End Learning
Jianming Pan'; Zeqi Ye
Xiao Yang", Xu Yang', Weiqing Liu Lewen Wang Jiang Bian
University of California, Berkeley, Nankai University
Miirosooft Research Asia
paamingg_annberkeley edu, 1iamyzq@gmail. com (xiao.yan yang, xuyang1, weiqing liu, lewen. wang, jiang. bian)mmiii wwwccrooot.ccom
Abstract
Data-driven decision-making processes increasingly utilize end-to-end learnable deep neural networks to render final decisions. Sometimes, the output of the for- ward functions in certain layers is determined by the solutions to mathematical optimization problems, leading the emergence of differentiable optimization layers that permit gradient fack-propagation. However, real-world scenarios of- ten involve large-scale datasets and numerous constraints, presenting significant challenges. Current methods for differentiating optimization problems typically rely on implicit differentiation, which necessitates costly computations on the Jacobian matrices, resulting in low efficiency. In this paper, we introduce BPQP, differentiable convex optimization framework designed for efficient end-to-end learning. To enhance efficiency, we reformulate the backward pass simplified and decoupled quadratic programming problem by leveraging the structural proper- ties of the Maanss-KKhnnTTeehee (KKT) matrix. This reformulation enables the use of first-order optimization algorithms calculating the backward pass gradients, allowing our framework to potentially utilize any state-of-the-an solver. As solver technologies evolve, BPQP can continuously adapt and improve its efficiency. Extensive experiments on both simulated and real-world datasets demonstrate that BPQP achieves significant improvement efficiency -tyniccllyyy an order of magnitude faster in overall execution time compared to other differentiable optimization layers. Our results not only highlight the efficiency gains of BPQP but also underscore its superiority over differential optimization layer baselines.
Introduction
In recent years, deep neural networks have increasingly been used to address data-driven decision- making problems and to generate final decisions for end-to-end learning tasks. Beyond explicit forward functions, some layers of the network may characterized by behaviors of implicit outputs, such as the solutions to mathematical optimization problems, which can be described as differentiable optimization layers M. These can be treated as implicit functions where inputs are mapped to optimal solutions. In this manner, the network can incorporate useful inductive biases, including domain-spectie knowledge, physical structures, and priors, thereby enabling more accurate and reliable decision decision-making. This approach has been integrated into deep declarative networks [2] for end-to and-to-end learning and has proven effective in various applications, such as energy minimization and predict-then-opting
3 4
[ 6 problems. Here, we focus convex optimization due to
"Wokk done during an internship at Microsoft. 'Correpoondigg Author
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

its broad applications in fields such as portfolio optimization 77 control systems processing [ among others.
g and signal
Optimization problems typically lack a general closed-form solution; therefore, calculating gradients for relevant parameters requires more sophisticated methods. These methods can be categorized based on whether an explicit computational graph is constructed, namely into explicit unrolling and implicit methods. Explicit methods [4] [0H12] involve unrolling the iterations of the optimization process, which incurs additional computational costs. Conversely, implicit methods leverage the Implicit Function Theorem [3] to derive gradients. Some of these methods [4 [5] are tailored for specific problems, thus restricting the options for forward optimization and reducing efficiency. Alternatively, other approaches [2 offer more general solutions for deriving gradients but face inefficiencies during the backward pass. There remains substantial potential for enhancing efficiency in these methodologies.
To enable rapid, tractable differentiation within convex optimization layers and further enhance the capabilities of the end-to-end learning paradigm, we propose general, first-order differentiable convex optimization framework, which we refer to as the Backward Pass as a Quadratic Programming (BPQP). Specifically, BPQP simplify the backward pass for the parameters of the optimization layer, by reformulating first-order condition matrix into a simpler Quadratic Programming (QP) problem. This decouples the forward and backward passes and creates a framework that can leverage existing efficient solvers (with the first-order algorithm, Alternating Direction Method of Multipliers (ADMM) []], as the default) that do not require differentiabiiity both passes. Simplifying and decoupling the backward pass significantly reduces the computational costs in both the forward and backward passes. This key idea summarized in Fig.
Layer input
BPQP Difirrnntiblle Convex Optinization Framwwork Effciint End-to-End Leaming
Layer output
Pass
1/1
Convex Opt Problem
Arbitrary Solvar
zi
Prodlem Paromotias
Mosa Lu
cryep Aira
Erephay Restorre 35.01a
Salatin
1/2
forward and backward decoupled a a
Backward Pass
Arbitrary
Pass OPs
Differentiate KKT
1/P
Solver
conditions
Brokay
doficit
Reformable ayattes snople OP?
aL
Bj
Figure 1: The learning process of BPQP: the previous layer outputs and generates the optimal solution 2" in the forward pass; the backward pass propagates the loss gradient for end-to-end learning; the process is accelerated by reformulating and simplifying the problem first and then adopting efficient solvers.
Our proposed framework has several theoretical and practical advantages:
Novel Backward Pass Reformulation: Our framework BPQP decouples the backward pass from the forward pass of the convex optimization layer, then reforms the backward pass process to a simple QP problem. The method avoids solving the linear system involving the Karush-Kuhn Tucker (KKT) matrix and enables large-scale gradients computation via ADMM. It leverages structural traits such as sparsity, solution polishing [16), and active-sets accurate gradients computation.
17 for efficient and
Efficient Gradients Computation: Empirically, BPQP significantly improves the overall com- putational time, achieving up 13.54x, 21.02x, and 1.67x faster performance over existing differentiable layers on 100-dimension Linear Programming, Quadratic Programming, and Second Order Cone Programming, respectively. Such efficiency improvements pave the way for BPQP's application in large-scale real-world end-to-end learning scenarios, such as portfolio optimization. By adopting BPQP, the enhancement of the Sharpe ratio has increased from 0.65 (+0.25) to 1.28 (+0.43) compared to the adoete two-stage approach methods.

---

Flexible Solver Choice: BPQP accommouigg any general-pure purpose convex solver to integrate differentiablee
I differennoo layer for end-to-end training. This flexibility in solver choice allows for better
the differentiable matching of solver capabilities with specific problem structures, potentially leading to improved i
c efficiency and performance.
2 Related works
Explinna methods
Optimization problems typically do not have general closed-form solution formula that expresses the decision variable in terms of other parameters. To address this challenge, expliable methods [ [T] [T] umroll the iterations of the optimization process and use the decision variable from the final iteration as a proxy solution for the optimization problem. This constructs an computational graph from the parameters the proxy parameters, then calculate relevant tradieetty
explict
unconsstan prxxy
directly Typically, these methods are designed for unconstrained optimizations. Applying them decision
to constrained optimizations is computational expensive because it requires projecting variables into feasible region. Alt-Diff [12) is a novel unrolling solution that decouples constraints from the optimization and significantly reduces the computad cost. While advanced unrolling methods continue to improve their efficiency, they require an additional cost in the unrolled computine computational graph that increases with the number of optimization iterations.
Implicit methods In contrast, implicit methods use the Implicit Function Theorem to relate the decision variable to other parameters. These methods specifically apply the theorem KKT conditions in convex optimization. Specificity-drivel Sercificity-dddari approaches are often tailored for particular problems in convex optimization such as QP, offering good efficiency but sacrificing generality. OptNet [1] presented a differentiable batched-GPU QP solver. An ADMM layer is also developed for QP 18. It facilitates implicit differentiation through a custom fixed-point mapping, reducing the cost by reducing the dimensions of the linear system to be resolved. Generality-focae focuuse approaches pursued more generalized solutions but faced limitations in efficiency, performing poorly on large- scale optimization problems. Their solution processes often rely on specific methods (such as coupled forward and backward passes), which prevent the application of various optimizers, thus leading to the aforementioned efficiency issues. Diffep 11 considers computing the derivative of convex cone program by implicitly differentiating the residual map for its homogeneous self-dual embedding. Open-source convex solver CVXPY [19] adopts a similar method and computes gradients by SCS [20). JaxOpt [10) proposes a simple approach to adding implicit differentiation on top of any existing solver, which significantly lowers the barrier to using implicit differentiation. Our work, BPQP, based on implicit methods, can be efficiently and broadly applied to convex optimization problems. We simplify the backward pass by reformulating it into a simpler, decoupled QP problem. This decon
recouppiie grants BPQP the freedom to choose an optimizer and, in conjunction with problem simplification, greatly reduces the comeetationns computationas cost in both the forward and backward passes. Notably, some work also examine discrete optimization challenge 21 22), which are outside the purview of our study.
3 Background
3.1 Differentiable convex optimization layers
ioviide some background information for the differentiable convex optimization layers.
We will now provide
Suppose the differentiable didierentiable
convex convex optimization layer has its input y E IR,, output =" E IRd, we define the layer Soverwoe with the standard form of a convex optimization problem parameterized by y.
optimizzann
Definition 1 deefferenttinn Convex Optimization Layer). Given the input y E RP, output =" E Rd. Sommizzzziiin layer is defined
a differentiable convex ooteenittttiiiii d
x (3) = arg min (z(2) me
s.t. (2) 0,
9, (2) 0,
where y is the parameter vector of the objective functions and the constraints, 2 is the optimizatiin pariable, z" is the optimal solution to the problem; The functions f Rº - R.. and
E²" - R
3

---

convex, and the function h I + R is affine. The functions f. h, and are continuously differentiable w.r.f both y and enabling the computation of 0z" By
The gradient of the parameter vector y can be computed by combining the chain rule with the Implicit Function Theorem (IFT) [33. Within the deep learning architecture, optimization layers are integrated alongside explicit layers within end-to-end framework. Given the global loss function C, the derivative of C w.r.t y can written as
0c 0C 0z dy 02 dy
We can easily obtain the derivative ac through conventional automatic diffcrentiation techniques 2s the gradient of an implicit applied to explicit layers. However, challenges arise calculating "h
optimization layer. Considering the layer as an implicit function F(=", y) = O, recent studies such as OptNet [4] employ the IFT on the KKT conditions of the convex optimization problem to derive 0s" We state IFT here as a lemma for reference. By
Lemma 1 (Implicit Function Theorem), Consider a continuously differentiable function F(z, v) with F("", y) O, and suppose the Jacobian matrix of is invertible at a small neighborhood at (2" ;), we have
Oz -
Qy
where Jz(2) and Fy) are respectively the Jacobian matrix of F w.r. z and y.
It should be emphasized that the differentiation of the KKT conditions requires calculating the optimal value in the forward pass and necessitates solving linear systems involving the Jacobian matrix in the backward pass. Both phases phases-forward and backward-an are notably computationally intensive, particularly for large-scale problems. As result, differentiating through KKT conditions directly does not scale efficiently to extensive optimization challenges.
3.2 Differentiating Through KKT Conditions
To differentiate KKT conditions more efficiently, CvxpyLayer [19] has adopted the LSQR technique to accelerate implicit differentiation for sparse optimization problems. However, this method may be efficient for more general cases, which might not exhibit sparsity. Although OptNet [14 employs primal-dual interior point method in the forward pass, making its backward pass relatively straightforward, it suitable only for quadratic optimization problems.
In this paper, our main target is to develop new method that can increase the computational speed of the differentiation procedure especially for general large-scale convex optimization problems.
consider a general convex problem as defined Definition I To compute the derivative of the solution 2 parameter y. follow the procedure of [14) differentiate the KKT conditions using techniques from matrix differential calculus. Following this method, the Lagrangian is given by (omitting y).
L(z,v,A) f(2) + h(=) v X g()),
(I)
where E 9T and X E RR", 2 0 respectively denotes the dual variables on the equality
and inequality constraints. The sufficient and necessary conditions for optimality of the convex optimization problem are KKT conditions. Applying the IFT (Lemma [ to the KKT conditions and
let P( v² A+) f²f(2*) + Vhh(2") + V2g(2*) X A(2*) Vh(2") and G(z*) Vg((²))
Let q(2 2 V X²) + +V((2*) + Vg(2") V)(=")>>)//0 )r)//yy b(²*) ah(2* Bh(z")/Oy and c(2* X²)
a(D(X* (g(=" Oy.. Then the matrix form of the linear system can be written as:
8x
P(2t.v'))) ((2)) A(x*)
G(2
A
By
9(2 1 (7)
(gg(z)))
c(z
(2)
By
0
O
b(zz)
Oy
D(-): R - Rax represents a diagonal matrix that formed from a vector and =", v² X denotes the optimal primal and dual variables. Left-hand side is the KKT matrix of the original optimization

---

problem times the Jacobian matrix of primal and dual variables to e.g., 0z' E Rdxy, Right-hand side is the negative partial derivatives of KKT conditions to the
We can then backpropagate losses by solving the linear system in Eq. 2 In practice, however, explicitly computing the actual Jacobian matrices da' is not desirable due space complexity; by dc 8z
instead, [4] products previous pass gradient vectors
E R to reform it by notations [
R.
IR R"] (see Appendix A.3
p(zt.urX'") G(E))(
G(=n) (g((t))
A(=1)
2
ac ( o=
0
Ā
(3)
A(z*)
0
0
And the direct gradients V,L RP
1a(2" vz c()" ((2))((22(2)))0)) ][ DT
4 Methodology
4.1 Backward Pass as QPs
Our approach solves Eq. (3) using reformulation. Consider a general class of QPs that have decision
variables, m equality constraints and inequality constraints:
minimize / T p'z + s.t A'B=8, GE
where P $4, o' E Rd, A' Rmxd b' Rm, G' Raxd and d E RR. KKT conditions write down matrix form:
G¹ D(G's-0 - c)
A'
z od'
D(AG A'
D(ij
(5)
i
We note that Eq. 5 equivalent to Eq. A(z*), D(ĀGG' D(A*)G(=*), D(A)c, - 0,,0 and (ii) P( , x²) is positive semi-definite. However, D(A)G' = D(>))G(==). which contains the unknown variable X, may not hold. As the backward pass solves after the forward pass, we can change inequality constraints accurate active-set (i.e., of binding constraints) of equality conditions, and then condition (i) always holds for the equality-constraine obtained (The detailed proof can be found in Appendix A.2
3 if and only if: (i) P P( A'
QP. From this, the following theorem can be
Theorem 1. Suppose that the convex optimization problem in Definition ! is not primal infeasible and the corresponding Jacobian vector v,C exists. It is given by V, 1a(2*, "ia",v", ,,)), X²), (²).b(2") b(2")[2, TT and 2, X, v is the optimal solution of following equality constrained Quadratie Problem:
minimize = TT'++ s.t. A'2 b. G.8
(6)
Where P' P'= A' = A(2*), G2= === and (-d, 1- (BC) 0,
and 5 only keep the elements according to the active-set after rewriting the inequality constraints to equality constraints. We did not create new notations for the sake of simplicity.
Though our BPQP procedure described above also applies to Jacobians with forms other than vectors, e.g., matrices, in these cases where each I-dimension column in [ž, X, 7) right multiply the same KKT matrix and can be viewed as QPs packed in multi multi-dimmnsions, directly calculating the inverse of the KKT matrix may be more appropriate, especially when it contains special structure like OptNet 14 and SATNet (23]
The intuition of BPQP is that the linearity of IFT requires the KKT matrix left-multiply homogeneous linear partial derivative variables. Theorem highlights special situation that considers gradients at the optimal point (where KKT conditions satisfied). Generally, BPQP
General Gradients
5

---

provides perspective to define gradients in cerumetterssllttionn space that preserves KKT norm. Let us consider a series of vectors denoting the ktth iteration norn value of KKT conditions:
|| (y(4 = -11
(" d'uutt Toent: prim Ck.
(k)
(7)
Where y.(k) E Rd+m+n the KKT conditions in kth iteration and C R the norm value. The
series {Cu, Ci.... ) converges to 0 if the iteration algorithm is contraction operator. Let Q(A)
denote standard QP problem w.r.t. parameter Pk. qr. Ak. by, Gk. Cj and decision variable 2g. At each iteration, BPQP yields V v(((k) that preserves ||r((]]] = C2.- (See in Appendix A.4
Time Complexity and constraints which is the same level directly solving the linear system Eq. (3). However, reformulation as QP provides substantial structures that can be exploited for efficiency, such that (we cover them in Section 4.2) sparse matrix, solution polishing (16). active-sets, and first-order methods, etc. Cleverly implementing BPQP, experiments at fairly large-scale dimensions in practice highlight BPQP's capacity in comparison to the state-of-art differentiable solver and NN-based optimization layers. Intuitively, BPQP is more efficient than previous methods because it utilizes the convex QP structural trait in the backward pass.
The time complexity of solving such QP is ((N)) in the number variables
4.2 Efficiently Solve Backward Pass Problem with OSQP
The solver we referenced is OSQP [6], which incorporates the sparse matrix method and uses a first-order ADMM method to solve QPs, which we summarize below. On each iteration, it refines a solution from an initialization point for vectors (0)) E Rd X e IR",, and z/() E R And then iteratively computes the values for the k + 1th iterates solving the following linear system:
P P+a al A A
][
2(++1) w(++1) |
(2(1) 4 X [=[x0]
diag(p)-
diag(p) - (1)
(8)
And then performing the following updates:
((++1) + (A) + diag(p) (00-'(((()))) (D(A+l)-v(()) v(k))
(k+11 + IT (x (x(++1) + diag(p) -1y(k))
(9)
(k+1) - (b) + diag(p) (Î(*+]) - (k++1))
where a E R2 and E ET are the step-size parameters, and II 1 + T denotes the Euclidean projection onto constraints When the primal and dual residual vectors small enough in norm after kth iterations, 2k++1), (k++1) and p((+)) converges to exact solution 8 X and v²
In particular, given backward pass problem Eq. OSQP, we form KKT matrix below
(6 with known active constraints, as stated in
P+
81
GT
A
G
81
0
X
=
(10)
4
0
81
As the original KKT matrix not always inveritible, e.g., if has one or more redundant constraints, we modify it to be more robust for QPs of all kinds by adding a small regularization parameter D(P 8T. -87, -ōT) (in Eq. (10) as default 5 z 10-6 We could then solve with the aforementioned ADMM procedure obtain a candidate solution, denoted as f and recover the exact solution from the perturbed KKT conditions (K + AK) g by iteratively solving:
(K + AK) = K
(11)
where f++ = f* + A and it converges to & quickly in practice [66 for only one backward and one forward solve, which helps BPQP solve backward pass problems in a general but efficient way.
We have implemented BPQP with some of the use cases and have released it in the open-source
library Qlib (https://github.. http:://itthbb.o//mmeoooo////// com/microsoot/glii)).
G+ G(zī) has the same of active-set g(24) = is the number of active sets.

---

4.3 Examples: Differentiable QP and SOCP
Below we provide examples for differentiable QP and SOCP oracles (i.e. solutions) using BPQP. The general procedure is to first write down KKT matrix of the original decision making problem. And then apply Theorem |
Assuming the optimal solution 2" is already obtained in forward pass.
Differentiable QP With a slight abuse of notation, given the standard QP problem with parameters P. q, A, b, G,C as in Eq. (4) The result is exactly the same OptNet [14 since both approaches for accurate gradients. But BPQP is capable of efficiently solving large-scale QP forward- backard pass via ADMM [16). as shown Section 5.1
voL (22*T 22227 ViL
v,C Vc, C D(A2)A2* +1
VAC= D2*T + +
(12)
Ve, C -D(()A
And [2, P, A] solves
1
dc Oz+
minimize
P2
ă s.t. A 0, G 2 =. 0.
(13)
2
Differentiable SOCP The second-order cone programming (SOCP) of our interest is the problem of robust linear program [25
minimize qT= s.t. a. 2 + ||=1|2 - b; i= 1,2, m.
(14)
where q Rd, a, Rd, and by R. With m inequality constraints in L2 norm, we give the gradients w.r.t. above parameters.
vyc=2 Val+ - -= 14+ d12* Vat C= dr, 1= 1,2,..., m.
(15)
And [2, D. AJ are given by (f1 = E; Xi+ and to = =11²112)
minimize
1 2T
t1 I
t
2t
8 + OC & s.t. (aT + -t i= 1,2, m. (16)
2
to
to
Oz*
to
Experiments
In this section, present several experimental results that highlight the capabilities of the BPQP. To be precise, we evaluate for (i) large-scale computational efficiency over existing solvers on random- generated constrained optimization problems including QP, LP, and SOCP, and (ii) performance real-world end-to-end portfolio optimization task that challenging for existing end-to-end learning approaches.
5.1 Simulated Large-scale Constrained Optimization
We randomly generate three datasets (e.g. simulated constrained optimization) for QPs, LPs, and SOCPs respectively. The datasets cover diverse scales of problems. The problem scale includes 10 5, 50 x 10, 100 x 20, 500 x 100 (e.g., 10 x represents the scale of 10 variables, 5 equality constraints, and 5 inequality constraints). Please refer more experiment details in Appendix A..
QPs Dataset The format of generated QPs follows Eq. (6) which the notations in the following descriptions align. We take q as the learnable parameter to be differentiated and L = 1 T =² in Eq. (3)
To generate positive semi-definite matrix P, P' P' + 5I is assigned P where P' E Rdxd is randomly generated dense matrix, 5/ is a small regularization matrix, and 5 = 10-6, Potentially, we set Gz', G E Rmx* 2 RR" avoid large slackness values that lead inaccurate results. All other random variables are drawn i.i.d. from standard normal distribution N N(0, 1).
LPs Dataset The IP problems are generated in the format below
minimize 0T z <||=1| s.t. A= Gz < h.
(17)
where E RRd is the learnable parameter to be differentiated, E IRdd E Raxd E RD, G E Rmxd h 1 and e R., All random variables drawn from the same distribution as the QPs dataset.
7

---

It is noteworthy that it contains an extra item el|=|12 compared with traditional LP. This item is added to make the optimal solution 2* differentiable with respect to 9. Without this item, P(=*, v2",, X+) is always zero and thus the left-hand side matrix becomes singular in Eq. 2 This is trick adopted by previous work 77, and here we set e = 10-6 as default.
SOCPs Dataset For SOCP in Eq. [14) we consider a specific simple case, i.c. a, = 0 vi and this relaxations results in m 1. As QP and LP, we take q as differentiable parameter and set loss function C = 1=2,, but all variables are drawn i..d. from standard Gaussian distribution N (0,, 1).
Compared Methods To demonstrate the effectiveness of BPQP, evaluate the efficiency and accuracy of otate-of-the-art differentiable convex optimizers, as well as BPQP, on the datasets mentioned above. The following methods compared: CVXPY [T5], qpth/OptNet (14), Alt-Diff' 12 JAXOpt [1] and Exact [2]. Exact adopts the same algorithm as BPQP for the forward pass, but attempts to calculate exact gradients using direct matrix inversion the KKT matrix during the backward pass.
Evaluation and Metrics To evaluate the efficiency of the compared methods, the runtime in seconds is used for each forward pass, backward pass, and total process. To evaluate the accuracy, we first get a target solution _/xeact with high-accuracy method and then calculate the cos similarity with compared methods (CosSim- = 2 _Fxuct 2 methhod /[| * || methhdd
(])). We ran each instance 200 times for average and standard deviation (marked in brackets) of the metries.
Table I: Efficiency evaluation of methods by runtime in seconds based on 200 runs, with lower numbers indicating better performance.
Bockword 50c00
Forsond
Rackword)
10e5
100+30
500.100
10s5
50x 10
100x20
500s
danseet
metr
method
OP
abs, line
Exsat
37.26=19:01
181.4155522 2)
5027 3027-119421-81
38 21
187,91255.0 DI
484.2 (21775)
3495.5 (2446.1)
(mmll))
(scale 1.06-04)
CVXPY
41.10-153
140 2012 5)
4154 S)
4116.201888 ()
26.3655.7
35
41.36=11.3)
58.3=2265)
())
(5055433444
254.76222.1
HPUP TAXO
2.
11.01A87)
106:2142.5
2.050.55)
9,3632.9
$71.66113077
(scale 1.0e-00)
S.1:42.4)
6.0k23.)
28.5
383. 383.7
6.1123.0)
12:4(45.2)
12.11:13.6)
729.8)239.5)
LP
als, line
Essus
50. (6.14:664)
266 T) 7) 595 106.3/116144)
32
40 40.311226-4)
20033 (0
62002216644 164-4)
2040, (:378.5)
(scale 1.3e 04)
CVXPY
39:342243
48.712511
72.5155.5
$50.9:45.2
Att10.0)
350.11t33..2
13013)2110033
Tr
20
28 76
16 U)
KS4 (041003)
WMU 21+1900
MS 11433100)
BPQP
0.5
1.760.2)
4.1128.4)
25.6117.5)
1.7t00.0
5.3
29.5:26.7)
318.712106.4)
SOCP
linz 1.te-00)
Exsat
2.3/45.1)
4.3136.3)
12.8022.22
110.7111168)
47.5
52.0118.4)
73.3)22423
200.2
CVXPY
31.1650.31
64. (4.1)251))
511 (111143.44
105
354,3155
BPQP
0.2:20.0)
0.71x0.0
2.3
53.4/20.3
45.42:4.)
4k.5(a2.6)
63.1(116)
242.9t2.7)
Table 2: Large-seale comparison of efficiency evaluation of methods by runtime in seconds based on 10 runs, with lower numbers indicating better performance.
stage
Backwaard
Total|Forward Backward)) 3000x1000
size
500x200
1500x500
1000x1000
5000x2000
500x200
1500x500
5000x2000
datascc
metric
method
QP
abs, time (Scale 1.0e-01)
Exact
4.1(11.6)
28.5(18.7)
86.2 (111.)
249.0 (117.3)
(2901 1.5(21.6)
150.3 (110.0)
189, 1 (122 5)
(large)
Alt-Diff
7.0k+1.3)
42 42.56=3.2)
282 (02.61155.85 8)
1820 (00.) 4)
BPQP
0.1(20.0)
1.5(=0.))
6.3(10.5)
20.4(20.4)
0.5(20.0)
10.0e50.)
70.66+4.7)
262.8(+17.0)
The results for efficiency evaluation are shown in Table T The evaluation covers three typical optimization problems with different problem scales. The results start from the QP dataset. Compared with ofate-of-the-ar art accurate methods, BPQP achieves tens to thousands of times of speedup in total time. We visualize part of the results of Table Soblleiinn Figure and perform sensitivity analysis under 500x 100 setting in Figure 3, where the horizontal axis represents (tolerance, maximum iterations) of the methods. These results demonstrates the efficiency and robustness of BPQP. When the problem becomes large, such as 5000x2000, previous methods fail to generate results. CVXPY extremely much slower because reformulates the QP as a conic program and the reformulation is slow and has to be done repeatedly when the problem parameters change [16]. It is worth noting that BPQP is faster even in the backward pass, where CVXPY and qpth/OptNet share information from the forward pass to reduce computational costs. Sharing this information will limit the available forward solvers and result in a coupled design. Exact falls back to a simpler implementation that does not involve sharing information between designs. It solves the KKT matrix (i.e., Eq. ( in the backward pass via matrix inverse method without relying on information from the forward pass. Although Exact uses relatively efficient implementation in the forward pass (i.e., a first-order method, same as BPQP), the fallback backward implementation becomes a bottleneck for efficiency. The results of the LP dataset lead to similar conclusions those of the QP dataset.
Results
8

---

In the evaluation of the SOCP dataset, qpth/OptNet and Dit- focus on QP and are excluded from this non-QP setting, Due the specialty of SOCP, CVXPY does not require problem reformulation into conic programs, giving it an advantage. BPQP still outperforms other options in terms of total time across all problem scales.
Table Backward accuracy of methods on simulated QP and non non-QP(SOCT) dataset
QP
SOCP
method
BPQP
CVXPY
qpthoOpttNet 0.989(40.12)
Aidi
JAXOpt 0.831(20.14)
BPQP
CVXPY
Avg. CosSim.
0.992(40.09)
0.924(40.15)
1.
L.0041)
1.00441.3e
The accuracy evaluation results are shown in Table I In the forward pass, all solvers give nearly the same results, which are not shown in the table. When evaluating the backward accuracy, we use a matrix inverse method with high precision solve Eq. (3) directly to get target solution(i.e. B and compare solutions from evaluated methods against it. The CosSim. is relatively higher than that in the forward pass due accumulated computational errors, Among them, the CosSim. of our method BPQP is the highest in QP. The Sis of all methods are small enough for SOCP.
Tobe Time mertormence of Defferent Methods on OF (Ararage 203 times)
Sondiciity Aealyss of M0foret Nuthods O' Divragge 50 Siman
1523
2549
183
Figure 3: Sensitivity analysis under 500x 10 ser- ting.
Figure 2: Table results visualization.
5.2 Real-world End-to-End Portfolio Optimization
Portfolio optimization is fundamental problem for asset allocation in finance. It involves constructing and balancing the investment portfolio periodically to maximize profit and minimize risk. The problem is an important case of end-to-end learning and can also be solved utilizing differentiable convex optimization layers [26]. We now show how to apply BPQP to the problem of end-to-end portfolio optimization (more experiment details in Appendix (A6))
Mean- Variane Optimization (MVO) 27 is a basic portfolio optimization model that maximizes risk-adjusted returns and requires long only and budget constraints.
maximize w w T Ew subject to 1T w = 1.
(18)
2
where variables w Rd represent the portfolio weight, 7 E R > 0, the risk aversion coefficient, and E R2 the expected returns are the parameters to be predicted (under our setting, the input to the optimization layer) of the convex optimization problem. The covariance matrix, Σ, of all assets can be learned end-to-end by BPQP. However, it preserves more stable characteristic than returns in time-series 288) Therefore, we set it as a constant.
Benchmarks We evaluate BPQP based on the most widely used predictive baseline neural network, MLP. For the learning approach, we compared the separately coo-stgee(Two-Staee and differentiable convex optimization layer upproachesqptth/Opteeee The optimization problem in the experiment has a variable scale 500, which cannot be handled by other layers based on CVXPY and JAXOpt. We found the tolerance level for truncation in Alt-Diff hard to satisfy the 500 inequality constraints and yield relatively longer training time (588 minutes per training epoch) than the above benchmarks. Our implementation substantially lowers the barrier to using convex optimization layers.

---

Table 4: Prediction and (pertioo(ootffllio) metrics evaluation of different methods in portfolio optimization. Speed is evaluated by training time per epoch (minute).
Prediction Metrics
Portfolio Metrics
Optimization Metries Regret Speed]
1
ICIR f
Ann.Ret.(%)
Sharpe
Two-Stage qpth/OptNet 0.026(#0.003)
0.033(#0.004)
0.32()0.03)
9.28(43.466
0.65(=0.25)
0.0283(=0.0271)
0.11
0.38(+0.12)
1.25(#0.42) 16.54(+7.51)
0.0176(#0.0049) 1.28(+0.43)
21.2
BPQP
0.026(#0.002) 0.28(+0.03)
17.67(#6.11)
0.0129(#0.0020)
7.7
Results The overall results are shown in Table 4 As we can see in the prediction metrics, Two- Stage performs best. Instead of minimizing multiple objectives without non-competing guarantee, Two-Stage only focuses on minimizing the prediction error and thus avoids the trade-off between different objectives. However, achieving the best prediction performance does not equal the best decision performance. BPQP outperforms Two-Stage in all portfolio metrics, although its prediction performance is slightly compromised. qpth/OptNNet shows comparable performance with BPQP. But the average training time of BPQP is 2.75x faster than OptNet. These experiments demonstrate the superiority of end-to-end learning, which minimizes the ultimate decision error, over separate two-stage learning.
6 Further discussion
In this section, we discuss the potential for BPQP to be applied in non-convex problems.
When addressing non-convex problems, we may encounter two challenges. Firstly, the solution only a local minimum. Secondly, the solution represents only a proximate solution near local minimum. If an effective non-convex method (e.g. Improved SVRG (29) is employed in the forward pass, BPQP is still equipped to reformulate the backward pass a QP. This is because our derivations and theoretical analysis are equally applicable to non-convex scenarios.
BPQP allows for the derivation of gradients that preserve the KKT norm, elaborated in Section 4.1 under "General Gradients." which means that when KKT norm is small, BPQP can derive a high quality gradient. Therefore, when non-convex solver used in the forward pass successfully achieves a solution that is close to or even reaches a local or global minimum, BPQP can still compute well-behaved gradients effectively. This capability underscores the robustness of BPQP and adaptability in handling the complexities associated with non-convex optimization problems.
Additionally, many non-convex problems can be transformed into convex problems, making our approach applicable in a broader range of scenarios.
While its hard to perform experiments on non-convex problem due to the lack of baselines, we hope that future work can employ BPQP to do further analysis.
7 Conclusion
We have introduced a differentiable convex optimization framework for efficient end-to-end learning. Prior work in this area can be divided into explicit and implicit methods, based on the construction of an explicit computational graph.Explicit methods unroll the iterations of the optimization process, incurring additional costs. Conversely, implicit methods often struggle to achieve overall efficiency in both computing the optimal decision variable during the forward pass and solving the linear system involving KKT matrix during the backward pass. Our approach, BPQP, is grounded in implicit methods and simplify the backward pass by reformulating into simpler decoupled QP problem, which greatly reduces the computational cost in both the forward and backward passes. Extensive experiments on both simulated and real-wor world datasets have been conducted, demonstrating a considerable improvement in terms of efficiency.
References
[1] Akshay Agrawal, Brandon Amos, Shane Barratt, Stephen Boyd, Steven Diamond, and J Zico Kolter. Differentiable convex optimization layers. Advances in neural information processing
10

---

systems, 32, 2019.
[2] Stephen Gould, Richard Hartley, and Dylan Campbell. Deep declarative networks. IEEE Transactions on Pattern Analysis and Machine Intelligence, 44(8):3988 4004, 2021.
[3] Yann LeCun, Sumit Chopra, Raia Hadsell, M Ranzato, and Fujie Huang. A tutorial on energy- based learning. Predicting structured data, 1(0), 2006.
[4] Justin Domke. Generic methods for aptimization-basee modeling. In Artificial Intelligence and Statistics, pages 318-326. PMLR, 2012.
[5] Jayanta Mandi, Peter Stuckey, Tias Guns, et Smart predict-and-optimii for hard combina- torial optimization problems, In Proceedings of the AAAI Conference on Artificial Intelligence, volume 34, pages 1603-1610, 2020.
[6] Adam N Elmachtoub and Paul Grigas. Smart "predict, then optimize". Management Science, 68(1):9-26 2022.
[7] Bryan Wilder, Bistra Dilkina, and Milind Tambe. Melding the data-decisions pipeline: Decision- focused learning for combinatorial optimization. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 33, pages 1658-1665 2019.
[8] Lei Guo and Hong Wang. Stochastic distribution control system design: a convex optimization approach. Springer, 2010.
[9] John Mattingley and Stephen Boyd. Real-time convex optimization in signal processing. IEEE Signal Processing Magazine, 27(3):50-61, 2010.
[10] Mathieu Blondel, Quentin Berthet, Marco Cuturi, Roy Frostig, Stephan Hoyer, Felipe Llinares- López, Fabian Pedregosa, and Jean- Philipph Vert. Efficient and modular implicit differentiation. arXiv preprint 200. 2021.
[11] Chuan-sheng Foo, Andrew Ng, al. Efficient multiple hyperparameter learning for log-linear models, In Advances in neural information processing systems, volume 20, 2007.
[12] Haixiang Sun, Ye Shi, Jingya Wang, Hoang Duong Tuan, H Vincent Poor, and Dacheng Tao. Alternating differentiation for optimization layers. In The Eleventh International Conference on Learning Representations, 2022.
[13] Jittomtrum, An implicit function theorem. Journal of Optimization Theory and Applications, 25(4):575-577, 1978.
[14] Brandon Amos and J Zico Kolter. Optnet:
Differentiable optimization as layer in neural networks. International Conference on Machine Learning, pages 136-145. PMLR, 2017.
[15] Akshay Agrawal, Shane Barratt, Stephen Boyd, Enzo Busseti, and Walaa M Moursi. Differenti- ating through cone program. arXiv preprint arXiv: 1904. 09043, 2019.
[16] Bartolomeo Stellato, Goran Banjac, Paul Goulart, Alberto Bemporad, and Stephen Boyd. Osqp:
An operator splitting solver for quadratic programs. Mathematical Programming Computation, 12(4):637-672, 2020.
[17] Philip Wolfe. The simplex method for quadratic programming. Econometrica: Journal of the Econometric Society, pages 382-398, 1959.
[18] Andrew Butler and Roy H Kwon. Efficient differentiable quadratic programming layers: an admm approach. Computational Optimization and Applications, 84(2):449-476, 2023.
[19] Steven Diamond and Stephen Boyd. Cvxpy: A python-embedded modeling language for convex optimization. The Journal of Machine Learning Research, 17(1):2909-29) 2016.
[20] Brendan O'donoghue, Eric Chu, Neal Parikh, and Stephen Boyd. Conic optimization via operator splitting and homogeneous self-dual embedding. Journal of Optimization Theory and Applications, 169:1042-1066 2016.

---

[21] Aaron Ferber, Bryan Wilder, Bistra Dilkina, and Milind Tambe. Mipaal: Mixed integer program as layer. Proceedings of the AAAI Conference on Artificial Intelligence, volume 34, pages 1504-1511 1511, 2020.
[22] Mathias Niepert, Pasquale Minervini, and Luca Franceschi. Implicit mle:
backpropagating through discrete exponential family distributions. In Advances in Neural Information Processing Systems, 2021.
[23] Po- Po-Wei Wang, Priya Donti, Bryan Wilder, and Zico Kolter. Satnet: Bridging deep learning and logical reasoning using differentiable satisfiability solver. In International Conference Machine Learning, pages 6545-6554. PMLR, 2019.
[24] Xiao Yang, Weiqing Liu, Dong Zhou, Jiang Bian, and Tie- Yan Liu. Qlib: An ai-oriented quantitative investment platform. arXiv preprint arXiv:2009. 2020.
[25] Kristin P Bennett and Olvi L Mangasarian. Robust linear programming discrimination of two linearly inseparable sets. Optimization methods and software, 1(1):23-34 34, 1992.
[26] Sinem Uysal, Xiaoyue Li, and John M Mulvey. End-to-end risk budgeting portfolio optimiza- tion with neural networks. Annals of Operations Research, pages 1-30, 2023.
[27] H. M. Markowitz. Portfolio selection. The journal of finance, 7(1):77-91, 1952.
[28] Thomas Lux and Michele Marchesi. Volatility clustering in financial markets: a microsimulation of interacting agents. International journal of theoretical and applied finance, 304))6675-702 2000.
[29] Zeyuan Allen-Zhu and Yang Yuan. Improved svrg for non-strongly-conee sum-of-non- convex objectives. In International conference on machine learning, pages 1080-1089, PMLR, 2016.
[30] Priya L Donti, David Rolnick, and Zico Kolter. De3: A learning method for optimization with hard constraints. arXiv preprint arXiv:2 12225, 2021.
[31] Rares Cristian, Pavithra Harsha, Georgia Perakis, Brian I. Quanz, and Ioannis Spantidakis. End-to-end learning for optimization via constraint-enfforingg approximators. I Proceedings of the AAAI Conference on Artificial Intelligence, volume 37, pages 7253-7260, 2023.
[32] Lingkai Kong, Jiaming Cui, Yuchen Zhuang, Rui Feng, B Aditya Prakash, and Chao Zhang End-to-end stochastic optimization with energy- model. Advances in Neural Information Processing Systems, 35:1 1341-11354, 2022.
[33] Chaitanya K Joshi, Quentin Cappart, Louis-Martin Rousseau, and Thomas Laurent. Learning the travelling salesperson problem requires rethinking generalization. Constraints, pages 1-29, 2022.
[34] Elias Khalil, Hanjun Dai, Yuyu Zhang, Bistra Dilkina, and Le Song. Learning combinatorial optimization algorithms over graphs. Advances in neural information processing systems, 30, 2017.
(35) Qiang Ma, Suwen Ge, Danyang He, Darshan Thaker, and Iddo Drori. Combinatorial opti- mization by graph pointer networks and hierarchical reinforcement learning. arXiv preprint 1x1iv:1911.04990 2019.
[36] Wouter Kool, Herke Van Hoof. and Max Welling. Attention, learn solve routing problems! arXiv preprint 180i 088475 2018.
[37] Ayse Sinem Uysal, Xiaoyue Li, and John M Mulvey. End-to-end risk budgeting portfolio optimization with neural networks. arXiv preprint arXiv:2 ien.2200774646 2021.
[38] Sanket Shah, Kai Wang, Bryan Wilder, Andrew Perrault, and Milind Tambe. Decision-focused learning without differentiable optimization: Learning locally optimized decision losses. arXiv preprint 22X3 2022.
12

---

[39] Kai Wang. Bryan Wilder, Andrew Perrault, and Milind Tambe. Automatically learning compact quality-aware surrogates for optimization problems. Advances Neural Information Processing Systems, 33:9586-9596, 2020.
[40] Aaron M Ferber, Taoan Huang, Daochen Zha, Martin Schubert, Benoit Steiner, Bistra Dilkina, and Yuandong Tian. Surco: Learning linear surrogates for combinatorial nonlinear optimization problems. In International Conference on Machine Learning, pages 10034-10052. PMLR, 2023.
[41] Arman Zharmagambetov, Brandon Amos, Aaron Ferber, Taoan Huang, Bistra Dilkina, and Yuandong Tian. Landscape surrogate: Learning decision losses for mathematical optimization under partial information. Advances in Neural Information Processing Systems, 36, 2024.
[42] Brendan O'Donoghue, Eric Chu, Neal Parikh, and Stephen Boyd. Conic optimization via operator splitting and homogeneous self-dual embedding. Journal of Optimization Theory and Applications, 169(3):1042- 0406-006,,,, June 2016.
[43] Brendan O' O'Donoghue. Operator splitting for homogencous embedding of the linear comple- mentarity problem. SIAM Journal on Optimization, 31:1999-2023, August 2021.
[44] Erhan Beyaz, Firat Tekiner, Xiao-jun Zeng. and John Keane. Comparing technical and fun- damental indicators in stock price forecasting. In 2018 IEEE 20th International Confer- ence on High Performance Computing and Communications; IEEE 16th International Con- ference Smart City; IEEE 4th International Conference Data Science and Systems (IIPCC)SmaCCit)DSS)) pages 1607-1613. IEEE, 2018
13

---

A
Appendix
A.1 Additional Related Works.
In this section, we discuss additional related works which focus on scenarios that regard optimiza- tion problem as terminal objectives. When optimization solutions solely constitute the terminal output of an end-to-end learning rrocess-rather than intermediates within neural network architec- additiona methodologies emerge.
A.1.1 Learn-to-optinize
coontto-opiimiz has relatively low accuracy, which means it can only support some problems with a high error tolerance. DC3 [30] and ProjectNet [31] research works in this direction. They leverage the universal approximation ability of neural networks and choose error correction algorithms modify the output solution into the feasible region. [3] exploits energy-based model
for decision-focused learning.
As a comparison to compute for exact gradients, existing work on Learn-to-Optimize trains an approximated solver network via SGD (e.g. DC3 (30) or RL policy gradients (33H36] to solve constrained optimization problems that have true graphical structure e.g. TSP, VRP, Minimum Vertex Cover, Max-Cut, and their variants. Leveraging the strong representation ability of state-of- the-art graph-based networks, RL obtains final solutions or intermediate results to be polished by searching or optimization algorithms, When optimizing convex and hard constraints in real-world scenarios, the underlying graph is typically fully connected and the accuracy tolerance is lower [37] This presents a restriction on the widespread usage of works based on approximated solvers.
A.1.2 Surrogate Loss
Alternatively, surrogate loss methods, arises from predict-ten--otiiee problem setting, present another avenue. LODL (38 introduces a generalizable surrogate loss, albeit at a high computational cost, limiting its widespread applicability. Some work employ linear, low-dimensional repre- sentations of convex problems, yield approximate surrogate losses but at the expense of precision. Both SurCo 40 and SPO 66 explore linear surrogate models, yet their inability capture the full complexity of original problems limits their effectiveness. LANCER [41 proposes a smooth, learn- able landscape surrogate that extends to novel optimization challenges, though it may compromise accuracy. Our approach, BPQP, served as differentiable layer which is capable of back-propagating exact gradients, offering greater flexibility and utility in practical applications.
A.2
Proof of Theorem 1
In this section, we'll demonstrate the proof of Theorem
Proof. After the forward pass, the active sets of the original set are known. Thus, the original optimization problem in can be reformulated as
arg min f(2) subject h(2)) 0, g+) O,
(19)
ZER
Where is omitted for simplifying notations, 9+ has the same row of the active set as the original inequality constraints 9g.
Accordingly, the matrix form of the KKT conditions, as shown in 2 can be rewritten as follows:
0="
u+ G4(2*)
G+(=+) 0
A(**)
dg
(+(2,,0
}.
ax;
=
(+(2))
(20)
A(z*)
0
b(2*)
Dy
where X1 represents the Lagrangian multiplier for the equality constraints g+, and only contains the dimensions of the active-set of gy. The following equations modified accordingly:
14

---

Vah(2') v (1) V +
V'
G+(=") (=+(=*)
94(= v x4) = -(V[(=*) + Vh(=*) +346377)))))))
c( 0()+(x"))/Oy
The direct gradients become v,L E R = =1= v 11) C+ (+)))((2)))))) (2*), M|,, X4, TJ then converted to:
Equation is
p x1) G_(z")
G+(z*)
A(=*)
0C 00
)T
0
X =
0
(21)
A(21)
0
Finally, the linear Eq. below:
21 system equal to the KKT condition of the QP theorem [
displayed
P
G
T
A'
+
-
G
0
_
(22)
A'
0
1
Please note that Theorem does not create new notations for XT, doo but reuses X Å for simplicity.
A.3 Differentiate Through KKT Conditions Using the Implicit Function Theorem
In this section, we give detailed discussion on Eq. optimality for are KKT conditions:
The sufficient and necessary conditions for
V V + Vg(2 ) =
1(2*) D ((X') (g(z"))
(23)
X 2*20. 0,
Applying the Implicit Function Theorem to the KKT conditions and let P(=", ,,' f(*")>
(²h(=*) )rr + V²g(2") A(2*) Vh(=*) and G(=*) Vg(2*) yields Eq. 2 We can then backpropagate losses by solving the linear system. In practice, however, explicitly computing the actual Jacobian matrices 0="
By is not desirable due to space complexity; instead, we product some
previous pass gradient vectors 02 E Rd to reform it by noting that
0C
02 Oy
0x*
02"
v,f
0
(24)
oy
Oy
0
The first term of left hand side is the transposed solution of Eq. 2 and above can be reformulated as
P( vz x*)
G(x))((2*)
A(z*)
BC
T
(oz")
VyC [q. c, bl
G(2²)
D(((z*))
0
(25)
A(2*)
0
0
BFQP solution: [=.
A.4 Preserve KKT Norm Gradients
In a typical optimization algorithm, each stage of the iteration gives primal-dual conditions (k)) we follow the procedures of BPQP and solve the corresponding QP problem O(k) to define general gradients V ((k), The key difference here is that instead of using the optimal solution to derive BPQP, plug in the intermediate points. By IFT,
(r()) (k) = K (d) dA, duj + (x(k) Oy dy = O.
(26)
15

---

where (())) is the Hessian matrix (KKT matrix) at points (Zk, Ag. Vk). The general gradients V, Vy.(M) is given by dr(k) = 0 and therefore ||r((]]] = C preserves KKT norm.
A.5 Simulation Experiment
Compared Methods
In Section 5.1 we randomly generate simulated constrained optimization datasets with uniform
distributions and varying scales. We use these datasets to evaluate the efficiency and accuracy of state-of-the thr-rtt art differentiable convex optimizers as well as BPQP. The methods of comparison briefly introduced previously are now detailed below:
CVXPY is a universal differentiable convex solver 19 accelerate the gradients calculation process.
15
SCS
42
solver is employed to
qpth/OptNet qpth a GPU-based differentiable optimizer, OptNet 14 is a differenntablee neural network layer that wraps qpth as the internal optimizer.
BPQP is our proposed method. Its forward and backward passes are implemented way. It adopts the OSQP [16] as the forward pass solver. In the backward pass, it reformulates the backward pass as an equivalent simplified gquality-constraieg QP. OSQP is also adopted in the backward pass to solve the QP.
decoupled
Exact the same forward pass solver as BPQP. The optimization algorithm used for the forward pass is the OSQP [16), which a first-order optimization algorithm that does not share differential structure information. In the backward pass, without using reformulation via BPQP, the Eq. (3) solved using the matrix inversion method like (2] As a result, this approach fails to achieve overall efficiency.
JAXOpt 10 is an open-sourced optimization package that supports hardware accelerated, catchable training and differentiable backward pass. Optimization problem solutions can be differentiated with respect to their inputs either implicitly or via autodiff of unrolled algorithm iterations.
Alt-Diff 12] adopts ADMM in specializing in solving QP problems with exact solutions as well as gradients w.r.t. parameters.
Hardware Setting
All results were obtained unloaded 16-core Intel(R) Xeon(R) CPU E5-2630 v3 @ 2.40GHz. qpth runs on an NVIDIA GeForce GTX TITAN X.
Choice of Solvers of BPQP
BPQP decoupled the forward and backward pass and provides flexibility of choosing solvers. Nor- mally, the first-order solver is greatly preferred when the problem scale becomes large and is also robust for small problem scale. Therefore, the first-order solver is good enough default value, which also employed by our framework and experiments.
A.6 Portfolio Optimization Experiment
Statistical Risk Model (SRM) used generate the covariance matrix of MVO in Section 5.2 takes the first 10 components with the largest eigenvalues by applying PCA on stock returns in the last 240 trading days. SRM shows the best performance of the traditional data-driven approach for learning latent risk factors.
Dataset & Metries
This section provides a more detailed introduction to the datasets and metrics used in the experiments described in Section 5.2 The dataset is from Qlib (24 and consists of 158 sequences, each containing OHI.C-based time- meries technical features 44 from 2008 to 2020 in daily frequency. Our experiment is conducted on CSI 500 universe which contains at most 500 different stocks each day.
16

---

For the predictive metrics, we evaluate IC (Information Coefficient) and ICIR (IC Information Ratio) of predictive model baselines. IC measures the correlation coefficient between the predicted stock returns 9 and the ground truth y. At each timestamp t, IC(() corr(g f(t) in which
Ex (y: 5)
y)rr(x,y) V2. - 8)22 2, (ii 0)²
We report average IC across instances. ICIR= Mean(IC)
std(JC) measures both the average and stability of IC. A well-trained predictive model is expected to have higher IC and ICIR. For portfolio metrics, which measure the performance of investment strategies in the real market, indicators, Aet. (Annualized Return) and Sharpe (Sharpe Ratio) which are the ultimate criteria widely used in quantitative investment. Ann.Ret. indicates the return of given portfolios each year. Sharpe = Ann,Ret. higher Sharpe, portfolios are expected maximize the total return and minimize the volatility of the daily returns. Transaction costs are not considered in our portfolio metrics to align with the regret loss and more stably demonstrate the effectiveness of end-to-ent learning without being distracted by unconsidered random factors.
key include
Vol. in which Ann.Vol indicates the annualizeed volatility. To achieve
Loss Construction
The loss of Portfolio Optimization in end-to-end learning form can be constructed as
Lro EryDD Fl (2t) fu (2t) |
SERD [lv 011"
ta
Freg (0).
Decision Error: Regret
Prediction Error
prior
where x is the input data and 0 is the parameter of the network. For we have ground truth values for the returns under this problem setting, we use the notation ý to denote the input to the optimization layer, also the predicted returns, and y for realized returns. On selecting 3, we choose 3 - E (0, 1)
that denotes the ratio variances between the random parameter and the decision error. Since the empirical regret suffers more severe fluctuation over y (oa >> oy 0) in convex optimization [5), prediction error should dominate in the end-to-end loss. However, we use an empiric distribution to approximate the Dirac distribution and set B3 to small (3 0.1 in portfolio optimization experiment) but not zero value.
Under normality assumption, the MAP loss can be written
arg max(p(0 regret, y, x)) x
(ug 2,13 203
arg max
Be
(27)
OdV 2π
Tg
2T
That is
0²
I1y arg min E11, (zē)
Sy (zt)
(28)
Compared Methods
Here is more detailed explanation of the compared methods in this experiment.
Two-Stage separately learns a prediction MLP model to predict expected returns (i.e. μ)) and then generates decisions based on Eq. (8). All other methods below share the same prediction MLP model and only differ in the learning paradigm.
qpth/OptNet performs similar to BPQP, but with sightly lower performance in portfolio metrics and regret as it approaches exact gradient with lower accuracy, shown in Table 3
BPQP is our proposed method. All the accurate approaches (e.g. CVXPY, JAXOpt) have similar high- quality solutions in both forward and backward passes and are expected to have similar performance. Among them, only BPQP can efficiently handle the problem size of 500 variables(refer to Table T).. and thus BPQP are selected.
17

---

MiEtpeedictoo loso
Ro: regrottddcisnn looss)
MSEiprediction loos)
Regrettdeeciiinn loos)
1.60-
1.60-
1.55
2
1.55
MSElprediction loss)
MSElprediction loss]
1.50
1.50
Regretidecision loss)
Regretidecision less)
1.45-
1.45
10
10
epoch
epoch
(a) Two Stage
(b) BPQP
Figure 4: The prediction and decision error/loss of methods with different objectives
BPQP trained using the loss function described in Section A.I.2
To gain a deeper understanding of how end-to-end regret loss works, Figure demonstrates the detailed learning curve of Two-Stage and BPQP. For each subfigure, the x-axis represents the number of epochs during training, and the y-axis represents the training loss of prediction and decision, respectively. Two-Stage aims minimize the prediction loss, which ultimately smaller than BPQP. However, the decision loss remains at a high level, resulting in suboptimal decision. BPQP aims to minimize both prediction loss and decision loss. Both losses decrease initially, and then they start to compete in the later epochs. However, the decision error remains at much lower value than Two-Stage, resulting in better decisions in the final evaluation.
Experiment Setting
Here are the detailed search space for model architecture and hyper-parameters.
We the same tolerance parameters for simulations experiments: Dual infeasibility tolerance:
le-04, Primal infeasibility tolerance: le-04, Check termination interval: 25, Absolute tolerance: 1e-03, Relative tolerance: 1e-03, ADMM relaxation parameter: 1.6, Maximum number of iterations: 4000.
We use lower tolerance parameter for real-world portfolio optimization experiments, due to the long-only strategy, we do not want small negative weight in the portfolio: Absolute tolerance: 1e-05, Relative tolerance: le-05, Dual infeasibility tolerance: le-05, Primal infeasibility tolerance: 1e-05.
MLP predictor: feature size: 153, hidden layer size: 256, number of layersr: 3, dropout rate: O.Training: number of epoch: 30, learning rate: le-4, optimizer: Adam, frequency of rebalancing portfolio: days, risk aversion coefficient: 1, early stopping rounds: 5, the inverse of beta (line 112): 0.1.
DC3: hidden size of solver net: 512, max stock size: 530, corrEps: le-4, cor 10, softWeightEqFrac: 0.5, commomentum:
Additional experiments of approximate methods
In this section, present additional experiments results for typical learning-basse based approximate optimizer, DC3. DC3 are trained using the loss function described in Section A.1.2] We train the solver net (i.e. optimizer) with 500 epochs, 10000 samples, and 10 correction Test Max Steps for each type of QP and LP entries.
The computational cost scales linearly with the problem size and the number of model parameters. So, is very efficient, especially when the scale is large. When the problem size is small (10x5 or 50x10), BPQP is still tens of times faster than DC3. However, DC3 becomes 5-10 times faster than BPQP when the problem size becomes large (500x100). Table the more detailed result of DC3 for both QP & LP (their problem size and number of model parameters the same, the time is nearly the same).
For large-scale real-worid portfolio optimization, approximate methods such as DC3 can be practical solution terms of efficiency. The experiment results are shown Table Although DC3 computationally efficient and thus applicable to large-scale real-world datasets, it performs poorly decision metries. This due to the inaceurate gradient that deteriorates the learned model based on DC3. Therefore, accuracy is an important feature in end-to-end learning.
18

---

size
10x5
50x10
100x20
500x100
forward + backward time(s)
1.4e-02
1.5e-02
1.7e-02
1.7e-02
Table 5: Efficiency evaluation the learnto-opptmmiee method DC3 by runtime in seconds.
Table 6: Prediction and (ecision(portfolio) metrics evaluation of DC3 in portfolio optimization. Speed is evaluated by training time per epoch (minute).
Prediction Metrics
Portfolio Metrics
Optimization Metrics
IC f
ICIR
Ann.Ret((%)
Sharpe f
Speed]
DC3
0.033(+0.001) 0.31(+0.01)
-0.40(+0.97)
-0. 00(4600))))
0.43
19

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification:
The paper's contribution is accurately reflected in the main claims.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
It is fine to include aspirational goals as motivation long is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors? Answer: [Yes]
Justification: We state that our method can be only applied to convex problems, and some further limitations are discussed in the experiment section.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g.. independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
the paper. The authors should use their best
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [Yes]
20

---

Justification:
We provide full set of assumptions, and the complete proof of the main theorem is in the appendix.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Yes]
Justification: The code of our experiments is provided, and the details are included in the experiment section and the appendix.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution a specific model and empirical evaluation, it may be necessary either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of large language model), releasing of a model checkpoint, or other means that appropriate to the research performed.
While NeurlPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how reproduce that algorithm. (b) If the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome deseribe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
21

---

reproducible, they Question: Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [Yes]
Justification: The code of our experiments is provided (also data simulation method), and the details are included in the experiment section and the appendix.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips.c cc/
public/guides/ Staalees//ooeSSuusmsiiiiiie
for more details.
While we encourage the release of code and data, we understand that this might be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https
/
Cotee//ddeeeSbmmmcia ssi onPolicy) for more details.
The authors should provide instructions data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code is permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: Full details of our experiments are provided in the experiment section and the appendix.
Guidelines:
The answer NA means that the paper does include experiments.
The experimental setting should be presented in the core the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment Statistical Significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer:
[Yes]
Justification:
We provide the confidence intervals of our results.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall with given experimental conditions).
22

---

The method for calculating the error bars should be explained (closed form formula, call a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
It should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma enror bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are range (e.g. negative enror rates).
If error bars reported in tables plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Question:
Answer: [Yes]
Justification:
We provide information on our hardware setting the appendix.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make into the paper).
9. Code Of Ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics
Answer: [Yes]
Justification:
The research conducted in the paper conform with the NeurIPS Code of Ethics.
Guidelines:
The answer NA means that the authors have not reviewed the NeurlPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., there special consid- eration due to laws or regulations in their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [No]
Justification: This paper presents work whose goal is advance the field of Machine Learning, There are many potential societal consequences of our work, none which we feel must be specifically highlighted here.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
23

---

Examples of negative societal impacts include potential malicious unintended uses (e.g., disinformation, generating fake profiles, surveillance), faireess considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: Our paper does not have such risks.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g.. code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: The original papers of models and algorithms used in our paper are properly cited.
Guidelines:
The answer NA means that the paper does use existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
24

---

If assets are released, the license, copyright information, and terms of use in the thcode. has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
package should be provided. For popular datasets, paperswi
batasets.papersstt
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information is available online, the authors encouraged to reach out the asset's s creators.
13. New Assets
Question: Are assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification: Our newly proposed convex optimization layer is well documented (code is provided).
Guidelines:
The answer NA means that the paper does not release assets.
Researchers should communicate the details of the dataset/code/mooel as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsoureing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification:
The paper does not involve crowdsourcing nor research with human subjects
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then much detail as possible should be included the main paper.
According to the NeurlPS Code of Ethics, workers involved in data collection, curation, other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification:
The paper does not involve crowdsourcing nor research with human subjects
Guidelines:
The answer NA means that the paper does involve crowdsourcing nor research with human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
25

---

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors adhere the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
26


Reviews:
[{"review_id": "9ZotqZ0JKI", "reviewer": "Reviewer_wmCS", "summary": "This paper proposes BPQP, a differentiable convex optimization framework to perform efficient end-to-end learning on convex problems using KKT condition. Compared with existing OptNet baselines, BPQP achieves similar performance but much faster speed.", "questions": "My main question is: Is the portfolio experiment shown in section 5.2 enough to showcase the performance of the proposed method, or do we need more experiments? For the baseline, are the two-stage and OptNet enough, or do we still need more baselines?", "limitations": "1. More experiments, or at least, more datasets should be included in the results section to showcase the performance of BPQP.\n2. In the experiments, the author should also add more baselines to compare with.", "rating": 7, "confidence": 5, "soundness": 2, "presentation": 2, "contribution": 3, "strengths": "1. This paper proposed BPQP, which utilizes the linearity of IFT to speed up the differential layers in an end-to-end framework. The intuition is good, and the performance of BPQP is good (2.75x faster than OptNet).\n\n2. The paper is clearly written, and the theory part is well-explained with detailed proofs.", "weaknesses": "1. The BPQP framework proposed in this paper is solely built on KKT conditions, which means BPQP can only be applied to convex end-to-end problems. However, most real-world decision-making problems are more complex and non-convex, making BPQP less applicable.\n\n2. In the experiment section, the author only performed two experiments: one is synthetic, and the other one is the portfolio optimization problem. However, I am a little worried if only the portfolio problem is enough to showcase the effectiveness of the proposed method. Besides, the author only compared with OptNet and two-stage, ignoring the other baselines in this area.\n\n3. Despite the clear mathematical theorem and proof of BPQP, many of the mathematical aspects are actually similar to the OptNet work, which makes the paper less novel.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your review. We will address each of the weaknesses and questions you have raised in detail.\n\n## Q1. Extension to non-convex problems\n\n> most real-world decision-making problems are more complex and non-convex\n\nThank you for highlighting this important limitation. While it is true that many real-world problems are non-convex and this poses significant challenges, BPQP provides a viable approach even in these contexts. \n\nWhen addressing non-convex problems, we may encounter two challenges. Firstly, the solution might only reach a local minimum. Secondly, the solution may be near a local minimum but not strictly satisfy the KKT conditions, representing only a proximate solution. If an effective non-convex method (e.g. Improved SVRG proposed in AllenZhu-Hazan, 2016a) is employed in the forward pass, capable of partially handling some the second issue (like reach the local minimum in high efficiency & accuracy), BPQP is still equipped to reformulate the backward pass as a Quadratic Program (QP). This is because our derivations and theoretical analysis are equally applicable to non-convex scenarios.\n\nBPQP's framework allows for the derivation of gradients that preserve the KKT norm, as elaborated in Section 4.1 under \"General Gradients.\" , which means that when KKT norm is small, BPQP can derive a high quality gradient. Therefore, when a non-convex solver used in the forward pass successfully achieves a solution that is reasonably close to or even reaches a local or global minimum, BPQP can still compute well-behaved gradients effectively. This capability underscores BPQP's robustness and adaptability in handling the complexities associated with non-convex optimization problems.\n\nAdditionally, many non-convex problems can be transformed into convex problems, making our approach applicable in a broader range of scenarios.\n\n## Q2. Real world experiments\n\n> Is the portfolio experiment shown in section 5.2 enough to showcase the performance of the proposed method?\n\nThe portfolio optimization problem selected for our experiments is a quintessential example in convex optimization. It is notably scalable and versatile, making it an excellent proxy for testing the efficiency and accuracy of end-to-end learning methods. Many real-world problems are fundamentally simple convex optimizations, such as QP, Linear Programming (LP) and Second Order Cone Programming (SOCP) we mentioned in this paper. Therefore, a comprehensive experiment on portfolio optimization is sufficient to demonstrate BPQP's capabilities in deriving a accurate result efficiently.\n\n> For the baseline, are the two-stage and OptNet enough, or do we still need more baselines?\n\nRegarding additional baselines, besides the two-stage and OptNet (Amos & Kolter, 2017) comparisons already discussed, we have evaluated other prominent methods such as CVXPY, JAXOpt (Blondel et al., 2021), and Alt-diff (Sun et al., 2022). These methods were unable to complete the task on the large-scale data of the portfolio problem (Section 5.2, Lines 312-318). Furthermore, we have explored the 'learn-to-optimize' approach, another method for addressing end-to-end learning issues. However, our results indicate that BPQP, along with the two-stage and OptNet methods, significantly outperforms the 'learn-to-optimize' approach DC3 (Donti et al., 2021), as detailed in Appendix A.6.\n\n## Q3. Similarity to OptNet in mathematical aspects\n\n> many of the mathematical aspects are actually similar to the OptNet work\n\nThe differentiation of the KKT matrix and the application of the Implicit Function Theorem (IFT) to these conditions are common steps in many convex optimization methods. However, our main contribution with BPQP lies in how we handle the output from these processes. Unlike conventional approaches, after applying the IFT to the KKT conditions, we reformulate the resulting complex linear systems into simpler, more manageable quadratic problems, which enables efficient large-scale gradients computation. Also, BPQP completely decouples the backward pass from the forward pass, leading to the flexibility in solver choice, which allows for bettermatching of solver capabilities with specific problem structures, potentially leading to improved efficiency and performance.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "cUkTgiLzVQ", "reviewer": "Reviewer_qYw9", "summary": "The paper provides a novel approach for handling differentiable optimization layers when their forward corresponds to the solution to a convex constrained optimization problem. The authors show that the gradient of such a layer corresponds to the solution of a QP, thus enabling a tractable backward through the use of QP solvers (in contrast to previous approaches which require solving a linear system). The authors evaluate their proposed algorithm on synthetic data tasks and a portfolio optimization problem. The results show a significant improvement in efficiency with respect to baselines.", "questions": "1. Would the proposed method work if the forward problem is solved approximately, thus yielding an imprecise estimate of the optimal solution and Lagrange multipliers? Note that this often happens in practice due to computational cost considerations (which are the main motivation of the submission). Did the authors consider a sensitivity analysis of the precision of the solution to the forward problem?\n2. Can the proposed algorithm be extended to non-convex problems?", "limitations": "* The authors do not mention any limitations of their work. \n* There is no discussion on future work.\n* No broader impact statement is provided in the paper.", "rating": 7, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "The high-level story of the paper is easy to follow. The proposed idea is simple and neat. Unlike existing approaches for implicit layers, it allows for the forward and backward passes to be decoupled algorithmically, thus allowing for extra flexibility since off-the-shelf solvers can be used for solving both the forward and backward problems. The results show a significant improvement in efficiency with respect to baselines.", "weaknesses": "* Although the proposed method applies when the forward passes correspond to general convex constrained optimization problems, only QPs are considered in the experiments section. Does the proposed algorithm provide as significant a speedup when the considered problem is less well-behaved?\n* The paper does not include any comments on the limitations of the proposed method or potential future works. Note that there is a difference between scoping (e.g. saying that the problem applies only to the convex setting) and identifying a limitation (e.g. saying that the gains of an approach are less significant as the dimension of the problem increases). \n* Tables 1 and 2 should include performance measures instead of just the computational time. Even if these synthetic problems are relatively trivial to solve with high precision, it is important to report the quality of the solution to ensure a fair comparison across approaches. \t\n* The paper does not include any ablation experiments. See Question 1 for a potential ablation. \n\nMinor points\n* The authors could do a better job at motivating the field in the introduction: in which practical context have differentiable optimization layers been used and how (concretely) have they proven more effective than alternative approaches?\n* Figure 1 is too small.\n* The main paper results are only presented through tables. Plots help illustrate the results much better. Moreover, they make it easier to identify trends (e.g. as the size of the problem changes) and make it more evident if confidence intervals overlap across methods.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your review. We will address each of the weaknesses and questions you have raised in detail.\n\n## Q1. Effectiveness in less well-behaved problems\n\n> Only QPs are considered in the experiments section. Does the proposed algorithm provide as significant a speedup when the considered problem is less well-behaved?\n\nWhile Quadratic Programming (QP) is featured prominently in our experiments, we have also explored Linear Programming (LP) and Second Order Cone Programming (SOCP) in our Simulated Constrained Optimization experiments. The results, detailed in Table 1, demonstrate BPQP’s efficiency and generalization across different convex optimization problems, affirming the framework's robustness and adaptability. For our larger-scale simulated experiments, we followed the settings used in Alt-diff, and focused solely on QP to ensure comparability. These specific results are detailed in Table 2.\n\n## Q2. Performance metrics\n\n> Tables 1 and 2 should include performance measures instead of just the computational time.\n\nThank you for your suggestion. While Tables 1 and 2 focused on computational time, we have included gradient accuracy metrics in Table 3. These results show that BPQP achieves the highest accuracy in deriving backward pass gradients for QP and SOCP.\n\n## Q3. Extension to non-convex problems & Sensitivity analysis\n\n> Can the proposed algorithm be extended to non-convex problems?\n\nWhen addressing non-convex problems, we often encounter two specific challenges. Firstly, the solution might only reach a local minimum. Secondly, the solution may be near the local minimum but not strictly satisfy the KKT conditions, representing only a proximate solution near a KKT point. If an effective non-convex method (e.g. Improved SVRG proposed in AllenZhu-Hazan, 2016a) is employed in the forward pass, capable of partially handling the second issue (like reach the KKT point in high efficiency & accuracy), BPQP is still equipped to reformulate the backward pass as a simple QP, as BPQP's framework allows for the derivation of gradients that preserve the KKT norm. Our derivations and theoretical analysis are equally applicable to non-convex scenarios.\n\nWhile we cannot ensure that the forward pass solver always finds a global minimum or strictly satisfies the KKT conditions, we can explore scenarios where the solver outputs a solution close to a stationary point and assess the quality of the resulting gradient. This aspect aligns closely with sensitivity analysis. Therefore, we will conduct sensitivity analysis on convex optimization problems to simultaneously evaluate BPQP's performance in scenarios resembling near-local optima within nonconvex optimization contexts.\n\n> Would the proposed method work if the forward problem is solved approximately, thus yielding an imprecise estimate of the optimal solution and Lagrange multipliers? Did the authors consider a sensitivity analysis of the precision of the solution?\n\nThank you for raising this important point. Indeed, in practical settings, we often implement early stopping. In light of the challenges associated with non-convex roblems, we conducted a sensitivity analysis to explore scenarios where the forward pass only reaches near KKT points. Specifically, we analyzed the sensitivity of BPQP, OptNet (Amos & Kolter, 2017), and CVXPY (Agrawal et al., 2019b) under settings involving 500-dimensional variables with 100 equality and 100 inequality constraints, adjusting algorithm tolerance and maximum iterations. Please refer to Figure R2 in the pdf document. The results indicate that even when the forward pass solution is approximate—merely near stationary points—BPQP maintains high accuracy in computing the backward pass gradients, demonstrating robustness and effectiveness. This is largely due to BPQP's capability to preserve the KKT norm during the computation of backward pass gradients. From this, we can infer that in non-convex optimization scenarios, if the forward pass solver yields a reasonably good solution, using BPQP to compute the backward pass gradient would likely result in favorable outcomes.\n\n## Q4. Limitations\n> The paper does not include any comments on the limitations of the proposed method or potential future works. \n\nThank you for highlighting this oversight. The performance and limitations of BPQP largely depend on the choice of solver, as the framework completely decouples the forward and backward pass processes. As indicated in Table 1, while the forward pass time dominates the total computing duration for large-scale problems, the backward pass—formulated as a straightforward QP problem—is relatively quick to solve, showcasing our method's strength. However, both the efficiency and accuracy of BPQP are still contingent on the solver's capabilities.\n\nWith a solver adept at handling large-scale convex optimization, BPQP's performance could be significantly enhanced. Moreover, if a robust non-convex solver is employed, BPQP is capable of deriving high-quality gradients, as mentioned earlier. Moving forward, we hope to see more research integrating advanced solvers within the BPQP framework to test its efficacy across diverse problem settings. This will help in further elucidating the scope and scalability of our approach.\n\n## Q5. Minor points\n\nThank you for your valuable feedback in minor points. To better motivate the introduction of BPQP, we will clarify the efficiency and effectiveness challenges in large-scale portfolio optimization.\n\nWe have also addressed the concern regarding the size of Figure 1 by simplifying its design and adjusting its dimensions for better clarity. Please refer to the revised Figure R1 in the PDF document.\n\nAdditionally, to enhance the presentation of our results, we have visualized the solving time of different methods for 500 variables with 100 equality and inequality constraints. This visualization can be found in Figure R3 in the PDF document.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "OVWUXpCO1f", "reviewer": "Reviewer_3Ly2", "summary": "This paper develops a technique to use deep learning models to solve convex optimization problems that offers speedups and space benefits over the current state of the art. Rather than using conventional implicit layers to predict optimal solutions, the authors consider the Karush-Kuhn Tucker (KKT) conditions for optimality in a different light in order to avoid costly and large Jacobian matrix computations. Here, they formulate this backward pass as a quadratic program (QP), which can be efficiently solved using first derivatives and requires less space and time than differentiating through the system of equations resulting from the KKT conditions. The KKT conditions are still used to form the backward pass as a QP, but solving the large system of equations is avoided.", "questions": "During inference, if the input to the model is actually infeasible (no solution exists that satisfies the corresponding KKT conditions), is there any indication of infeasibility produced by the model?\n\nDo the authors have an idea of how this would extend to nonconvex problems? The KKT conditions would provide local minima, but perhaps the same reformulation of the backward pass as a QP would be more complicated.", "limitations": "The authors have adequately addressed the limitations and I do not see any potential negative societal impacts.", "rating": 8, "confidence": 4, "soundness": 4, "presentation": 3, "contribution": 4, "strengths": "The BPQP framework, due to avoiding solving the linear system of equations introduced by the KKT optimality conditions, offers significant speedups versus existing differentiable layers for LP, QP, and SOC problems. This is a very relevant area of research as NN-based end-to-end optimization solutions are gaining a lot of attention for the extreme speed-ups over using conventional optimization solvers for convex programs.\n\nThe comparisons show very promising results against a wide variety of other solutions, both conventional (CVXPY) and NN-based (OptNet).\n\nThe reformulation of the backward pass into a format which has efficient solution algorithms rather than naively solving the linear system of equations introduced by the optimality conditions is simple but quite clever, and the results speak for themselves.", "weaknesses": "I would prefer if BPQP was defined in the abstract (also, shouldn't it either be \"as a Quadratic Program\" or \"as Quadratic Programming\" to be grammatically correct?)\n\nThere are quite a few grammatical errors in the paper. Not to the level of affecting readability, but the authors may want to go through and polish this.\n\nA bit nitpicky, but ADMM is not a \"solver\", it is an algorithm (page 2).\n\nThe text in Figure 1 in the forward and backward pass is very small.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your review. We will address each of the weaknesses and questions you have raised in detail.\n\n## Q1. Infeasible input\n\n> During inference, if the input to the model is actually infeasible (no solution exists that satisfies the corresponding KKT conditions), is there any indication of infeasibility produced by the model?\n\nThank you for raising this important issue. In situations where the predicted parameter $y$ influences the constraints of the optimization problem, there indeed exists the possibility that the constraints could become too restrictive, resulting in an empty feasible set. \n\nAlternatively, the objective function might become unbounded with certain values of $y$. We acknowledge the significance of this concern and appreciate your insight. We will contemplate strategies to address these potential scenarios. \n\nHowever, it is important to note that in most practical applications and real-world scenarios, the formulation of the problem typically involves $y$ appearing only within a well-defined objective function. The constraints of the optimization problem generally encompass fixed physical constraints that do not change based on $y$ (as seen in methods like Alt-diff (Sun et al., 2022) and OptNet (Amos & Kolter, 2017)). Under these common conditions, the issue of input infeasibility due to $y$ influencing the constraints is unlikely to occur.\nOnce again, thank you for your comment. We will continue to explore and refine our approaches to ensure robustness across all possible inputs.\n\n## Q2. Extension to non-convex problems\n\n> Do the authors have an idea of how this would extend to nonconvex problems? The KKT conditions would provide local minima, but perhaps the same reformulation of the backward pass as a QP would be more complicated.\n\nThank you for your insightful question regarding non-convex problems. When dealing with such problems, the solution might only reach a stationary point even without necessarily representing the local minimum. While achieving the global minimum is contingent the properties of the objective function, our framework, BPQP, is designed to be robust even in non-convex scenarios that a solution near a local minimum is provided. When adopting a efficient non-convex optimization method (e.g. Improved SVRG proposed in AllenZhu-Hazan, 2016a) achieves an approximate local minimum (also a KKT point), the gradient derived by BPQP may still hold good properties. \n\nOur theoretical analysis and derivations ensure that BPQP can still reformulate the backward pass as a simple Quadratic Program (QP). Also, BPQP can maintain nice gradient properties across both convex and non-convex contexts, which are detailed in Section 4.1 under \"General Gradients\", that the backward pass solution can preserve the KKT norm.\n\nWe recognize the importance of ongoing advancements in non-convex solvers, as improvements in these solvers will enhance our ability to tackle more complex problems effectively. We are optimistic that as these solvers evolve, they will expand BPQP's applicability and efficiency in handling non-convex challenges.\n\n## Q3. Minor points\n> I would prefer if BPQP was defined in the abstract.\n\n> There are quite a few grammatical errors in the paper. Not to the level of affecting readability, but the authors may want to go through and polish this.\n\n> A bit nitpicky, but ADMM is not a \"solver\", it is an algorithm (page 2).\n\n> The text in Figure 1 in the forward and backward pass is very small.\n\nThank you for your valuable feedback. We appreciate your attention to detail and will define BPQP clearly in the abstract, correct terminology around ADMM, and address grammatical errors throughout the paper. Additionally, we'll ensure that the text in Figure 1 is enlarged for better readability, please refer to Figure R1 in the pdf document as a refined version. Your insights are instrumental in enhancing the quality of our manuscript.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "WU8DPg4H6S", "reviewer": "Reviewer_Vrnz", "summary": "The authors introduce BPQP, a differentiable convex optimization framework designed for efficient end-to-end learning. The core of this work lies in simplifying and decoupling the KKT matrix for the backward pass and solving it with a first-order solver to improve the overall efficiency of the module.", "questions": "See above.", "limitations": "N/A", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. The paper is well-organized and well-written, making it relatively easy to read.\n\n2. The proposed strategy performs well in the experiments.", "weaknesses": "1. The differences between BPQP and OptNet seem to be only reflected between Equation 3 and Equation 5. The rationale, necessity, and justification for this change may need more explanation.\n\n2. For Equation 3, OptNet uses the interior-point method, while for Equation 5, BPQP uses ADMM. What is the motivation for using ADMM? The impact of using the interior-point method or various first-order/second-order optimization algorithms on performance (convergence, accuracy, time cost) also needs to be reflected in the experiments.\n\n3. It is a good practice to learn from OptNet and conduct experiments on richer tasks such as Total Variation Denoising and Sudoku.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your review. We will address each of the weaknesses and questions you have raised in detail.\n\n## Q1&Q2. Differnece between BPQP and OptNet & The motivation of using ADMM\n\n> The differences between BPQP and OptNet seem to be only reflected between Equation 3 and Equation 5. \n\n> For Equation 3, OptNet uses the interior-point method, while for Equation 5, BPQP uses ADMM. What is the motivation for using ADMM? The impact of using the interior-point method or various first-order/second-order optimization algorithms on performance also needs to be reflected in the experiments.\n\nThank you for your insightful questions regarding the distinctions between BPQP and OptNet (Amos & Kolter, 2017) and our choice of using ADMM in BPQP.\n\nThe differentiation of the KKT matrix and the application of the Implicit Function Theorem (IFT) to these conditions are indeed standard steps in many convex optimization methods (as shown in Equations 1 and 2). Our primary contribution with BPQP, however, lies in our novel approach to the output of these processes. After applying the IFT to the KKT conditions, we reformulate the complex linear systems into simpler, more manageable quadratic problems (QPs) (illustrated in Equations 3, 4, 5, and Theorem 1). This reformulation enables the efficient computation of large-scale gradients and allows BPQP to completely decouple the backward pass from the forward pass. This decoupling provides significant flexibility in solver choice, enhancing the match between solver capabilities and specific problem structures, which potentially improves both efficiency and overall performance.\n\nMoreover, since BPQP fully decouples the backward pass from the forward pass, its performance is almost entirely dependent on the efficacy of the chosen solvers, both during the forward and backward passes. This decoupling underscores BPQP's adaptability and evolutionary capacity within a general framework for deriving backward pass gradients. We opted for the OSQP solver employing the ADMM method due to its strong performance in our framework. However, for other convex optimization challenges, methods like the interior-point or other first/second-order methods might be more suitable, and we remain open to using these alternatives. While we anticipate future work to explore replacing OSQP with more advanced solvers within the BPQP framework for potentially superior outcomes, conducting comparative experiments of different solvers within BPQP is not our current focus due to the vast array of choices and because it diverges from our primary research objectives.\n\n## Q3. More real-world scenario experiments\n\n> It is a good practice to learn from OptNet and conduct experiments on richer tasks such as Total Variation Denoising and Sudoku.\n\nThank you for your suggestion. OptNet indeed represents pioneering work in the field of differentiable convex optimization, and the use of tasks like Total Variation Denoising and Sudoku underscores its versatility in various real-world scenarios. However, these tasks typically involve relatively simple convex optimizations—such as small-scale QP, Linear Programming (LP), and Second Order Cone Programming (SOCP)—which we have already addressed through simulated experiments. These types of tasks require lower computational power and are less challenging, which means that, similar to OptNet, BPQP can easily handle them. However, such tasks do not fully demonstrate BPQP’s advantages when dealing with large-scale optimization problems.\n\nIn contrast, our choice to focus on the portfolio optimization problem is deliberate. This problem is not only a quintessential example of convex optimization but also notably scalable, making it particularly suitable for transforming into the type of large-scale optimization problems that other convex optimization layers struggle with. This makes the portfolio optimization problem an excellent test case for evaluating the efficiency and accuracy of end-to-end learning methods like BPQP. Therefore, a comprehensive experiment on portfolio optimization adequately showcases BPQP's capabilities in efficiently deriving accurate results, thus aligning with our focus on demonstrating the framework's performance in more complex and large-scale scenarios.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
We would like to thank all reviewers for their constructive feedback. Below, we summarize the major concerns raised and provide our  explanations.

## Q1. Can the proposed algorithm be extended to non-convex problems?
BPQP can provide a viable approach even in non-convex scenarios. 

When addressing non-convex problems, we may encounter two challenges. Firstly, the solution is only a local minimum. Secondly, the solution represents only a proximate solution near a local minimum. If an effective non-convex method (e.g. Improved SVRG proposed in AllenZhu-Hazan, 2016a) is employed in the forward pass, BPQP is still equipped to reformulate the backward pass as a Quadratic Program (QP). This is because our derivations and theoretical analysis are equally applicable to non-convex scenarios.

BPQP allows for the derivation of gradients that preserve the KKT norm, as elaborated in Section 4.1 under "General Gradients." , which means that when KKT norm is small, BPQP can derive a high quality gradient. Therefore, when a non-convex solver used in the forward pass successfully achieves a solution that is close to or even reaches a local or global minimum, BPQP can still compute well-behaved gradients effectively. This capability underscores BPQP's robustness and adaptability in handling the complexities associated with non-convex optimization problems.

Additionally, many non-convex problems can be transformed into convex problems, making our approach applicable in a broader range of scenarios.

While its hard to perform experiments on non-convex problem due to the lack of baselines, we hope that future work can employ BPQP to do further analysis.

In fact, the scenarios where the solver outputs a solution close to a stationary point aligns closely with sensitivity analysis. Therefore, we will conduct sensitivity analysis on convex optimization problems to simultaneously evaluate BPQP's performance in scenarios resembling near-local optima within nonconvex optimization contexts.

## Q2. Did the authors consider a sensitivity analysis of the precision of the solution?

Indeed, in practical settings, we often implement early stopping due to computational cost considerations. We conducted a sensitivity analysis to explore scenarios where the forward pass only reaches near stationary points. Specifically, we analyzed the sensitivity of BPQP, OptNet (Amos & Kolter, 2017), and CVXPY (Agrawal et al., 2019b) under settings involving 500-dimensional variables with 100 equality and 100 inequality constraints, adjusting algorithm tolerance and maximum iterations. Please refer to Figure R2 in the pdf document. The results indicate that even when the forward pass solution is approximate, BPQP maintains high accuracy in computing the backward pass gradients. This is due to BPQP's capability to preserve the KKT norm during the computation of backward pass gradients. From this, we can infer that in non-convex optimization scenarios, if the forward pass solver yields a reasonable solution, using BPQP to compute the backward pass gradient would likely result in favorable outcomes.

## Q3. What is the main difference between BPQP and OptNet, especially in mathematical aspect?

The differentiation of the KKT matrix and the application of the Implicit Function Theorem (IFT) to these conditions, highlighted in OptNet, are now becoming standard steps in similar methods (as shown in Equations 1 and 2). Our primary contribution with BPQP, however, lies in our novel approach to the output of these processes. After applying the IFT to the KKT conditions, we reformulate the complex linear systems into simpler QPs (illustrated in Equations 3, 4, 5, and Theorem 1). This reformulation enables the efficient computation of large-scale gradients and allows BPQP to completely decouple the backward pass from the forward pass. This decoupling provides flexibility in solver choice, enhancing the match between solver capabilities and specific problem structures, which potentially improves both efficiency and overall performance.

## Q4. Do we need more real-world scenarios and baseline methods?

The portfolio optimization problem selected for our experiments is a typical example in convex optimization. It is notably scalable, making it an excellent proxy for testing the efficiency and accuracy of end-to-end learning methods. Many real-world problems are fundamentally simple convex optimizations, such as QP, Linear Programming and Second Order Cone Programming we mentioned in this paper. Therefore, a comprehensive experiment on portfolio optimization is sufficient to demonstrate BPQP's capabilities in deriving a accurate result efficiently.

Regarding additional baselines, besides the two-stage and OptNet comparisons already discussed, we have evaluated other prominent methods such as CVXPY, JAXOpt (Blondel et al., 2021), and Alt-diff (Sun et al., 2022). These methods were hard to handle the large-scale  portfolio problem (Section 5.2, Lines 312-318). Furthermore, we have explored the 'learn-to-optimize' approach DC3 (Donti et al., 2021). However, our results indicate that BPQP, along with the two-stage and OptNet methods, significantly outperforms DC3, as detailed in Appendix A.6.

## List of Figures

In response to the reviewers' suggestions, we have conducted additional experiments and redesigned three figures, which are included in the pdf document:
- Figure R1 is the refined version of Figure 1 for clarity: it presents with more readable fonts.
- Figure R2 shows our Sensitivity Analysis. We evaluated the robustness and performance of BPQP, OptNet, and CVXPY in settings involving 500-dimensional variables with 100 equality and inequality constraints, adjusting algorithm tolerance and maximum iterations. The results highlight BPQP's robustness and effectiveness.
- Figure R3 visualizes the total computation time under the same setting in Figure R2. This visualization demonstrates the trend and underscores BPQP's superior performance.

Meta-review: There is consensus among the expert referees that the submission should be accepted. The issues raised in the initial reviews were addressed in the detailed rebuttal, which should be incorporated in the paper.
Decision: Accept (spotlight)