SlowGuess/NeurIPS_2023_OCR huggingface dataset:
We thank the anonymous referees for their constructive comments. In addition, Xinyi Hu and Jimmy H.M. Lee acknowledge the financial support of a General Research Fund (RGC Ref. No. CUHK 14206321) by the University Grants Committee, Hong Kong. Jasper C.H. Lee was supported in part by the generous funding of a Croucher Fellowship for Postdoctoral Research, NSF award DMS-2023239, NSF Medium Award CCF-2107079 and NSF AiTF Award CCF-2006206.
[1] A. Agrawal, B. Amos, S. Barratt, S. Boyd, S. Diamond, and J. Z. Kolter. Differentiable convex optimization layers. Advances in neural information processing systems, 32, 2019.
[2] B. Amos and J. Z. Kolter. Optnet: Differentiable optimization as a layer in neural networks. In International Conference on Machine Learning, pages 136-145. PMLR, 2017.
[3] B. Chen, P. L. Donti, K. Baker, J. Z. Kolter, and M. Bergés. Enforcing policy feasibility constraints through differentiable projection for energy optimization. In Proceedings of the Twelfth ACM International Conference on Future Energy Systems, pages 199-210, 2021.
[4] E. Demirovic, P. J. Stuckey, J. Bailey, J. Chan, C. Leckie, K. Ramamohanarao, and T. Guns. An investigation into Prediction+Optimisation for the knapsack problem. In International Conference on Integration of Constraint Programming, Artificial Intelligence, and Operations Research, pages 241-257. Springer, 2019.
[5] E. Demirovic, P. J. Stuckey, J. Bailey, J. Chan, C. Leckie, K. Ramamohanarao, and T. Guns. Predict+Optimise with ranking objectives: Exhaustively learning linear functions. Proceedings of the Twenty-Eighth International Joint Conference on Artificial Intelligence, pages 1078-1085, 2019.
[6] E. Demirovic, P. J. Stuckey, T. Guns, J. Bailey, C. Leckie, K. Ramamohanarao, and J. Chan. Dynamic programming for Predict+Optimise. In Proceedings of the Thirty-Fourth AAAI Conference on Artificial Intelligence, pages 1444-1451, 2020.
[7] A. N. Elmachtoub and P. Grigas. Smart "Predict, then Optimize". Management Science, 68(1):9-26, 2022.
[8] A. N. Elmachtoub, J. C. N. Liang, and R. McNellis. Decision trees for decision-making under the predict-then-optimize framework. In Proceedings of the 37th International Conference on Machine Learning, pages 2858–2867, 2020.
[9] J. Friedman, T. Hastie, and R. Tibshirani. The elements of statistical learning. Springer series in statistics New York, 2001. Volume 1, Number 10.
[10] A. U. Guler, E. Demirović, J. Chan, J. Bailey, C. Leckie, and P. J. Stuckey. A divide and conquer algorithm for Predict+Optimize with non-convex problems. In Proceedings of the Thirty-Sixth AAAI Conference on Artificial Intelligence, 2022.
[11] Gurobi Optimization, LLC. Gurobi Optimizer Reference Manual, 2023.
[12] X. Hu, J. C. H. Lee, and J. H. M. Lee. Predict+Optimize for packing and covering LPs with unknown parameters in constraints. In Proceedings of the AAAI Conference on Artificial Intelligence, 2022.
[13] X. Hu, J. C. H. Lee, J. H. M. Lee, and A. Z. Zhong. Branch & Learn for recursively and iteratively solvable problems in Predict+Optimize. In Advances in Neural Information Processing Systems, 2022.
[14] K. B. Kabir and I. Mahmud. Study of erosion-corrosion of stainless steel, brass and aluminum by open circuit potential measurements. Journal of Chemical Engineering, pages 13-17, 2010.
[15] N. Kahraman, B. Gülenç, and F. Findik. Joining of titanium/stainless steel by explosive welding and effect on interface. Journal of Materials Processing Technology, 169(2):127-133, 2005.
[16] B. Maenhout and M. Vanhoucke. Branching strategies in a Branch-and-Price approach for a multiple objective nurse scheduling problem. Journal of scheduling, 13(1):77-93, 2010.
[17] J. Mandi, V. Bucarey, M. M. K. Tchomba, and T. Guns. Decision-focused learning: Through the lens of learning to rank. In International Conference on Machine Learning, pages 14935-14947. PMLR, 2022.
[18] J. Mandi and T. Guns. Interior point solving for LP-based Prediction+Optimisation. Advances in Neural Information Processing Systems, 33:7272-7282, 2020.
[19] M. Mulamba, J. Mandi, M. Diligenti, M. Lombardi, V. Bucarey, and T. Guns. Contrastive losses and solution caching for Predict-and-Optimize. arXiv preprint arXiv:2011.05354, 2020.
[20] R. Muniyan, R. Ramalingam, S. S. Alshamrani, D. Gangodkar, A. Dumka, R. Singh, A. Gehlot, and M. Rashid. Artificial bee colony algorithm with Nelder-Mead method to solve nurse scheduling problem. Mathematics, 10(15):2576, 2022.
[21] Y. Nandwani, R. Ranjan, P. Singla, et al. A solver-free framework for scalable learning in neural ilp architectures. Advances in Neural Information Processing Systems, 35:7972-7986, 2022.
[22] A. Paszke, S. Gross, F. Massa, A. Lerer, J. Bradbury, G. Chanan, T. Killeen, Z. Lin, N. Gimelshein, L. Antiga, A. Desmaison, A. Kopf, E. Yang, Z. DeVito, M. Raison, A. Tejani, S. Chilamkurthy, B. Steiner, L. Fang, J. Bai, and S. Chintala. Pytorch: An imperative style, high-performance deep learning library. In Advances in Neural Information Processing Systems 32, pages 8024-8035. 2019.
[23] A. Paulus, M. Rolínek, V. Musil, B. Amos, and G. Martius. Comboptnet: Fit the right NP-hard problem by learning integer programming constraints. In International Conference on Machine Learning, pages 8443-8453. PMLR, 2021.
[24] F. Pedregosa, G. Varoquaux, A. Gramfort, V. Michel, B. Thirion, O. Grisel, M. Blondel, P. Prettenhofer, R. Weiss, V. Dubourg, J. Vanderplas, A. Passos, D. Cournapeau, M. Brucher, M. Perrot, and E. Duchesnay. Scikit-learn: Machine learning in Python. Journal of Machine Learning Research, 12:2825–2830, 2011.
[25] D. E. Rumelhart, G. E. Hinton, and R. J. Williams. Learning representations by back-propagating errors. nature, 323(6088):533-536, 1986.
[26] M. Vanhoucke and B. Maenhout. Nsplib-a nurse scheduling problem library: A tool to evaluate (meta-) heuristic procedures. In Operational research for health policy: making better decisions, proceedings of the 31st annual meeting of the working group on operations research applied to health services, pages 151-165, 2007.
[27] B. Wilder, B. Dilkina, and M. Tambe. Melding the data-decisions pipeline: Decision-focused learning for combinatorial optimization. In Proceedings of the Thirty-Third AAAI Conference on Artificial Intelligence, pages 1658–1665, 2019.
A Detailed discussion on the Two-Stage Predict+Optimize framework
A.1 Problem modelling using the framework
As mentioned in Section 3, the proposed Two-Stage Prediction+Optimize framework is phrased differently from some other two-stage problem formulations. The proposed framework phrases Stage 1 solutions as soft commitments, and corrects Stage 1 solutions with penalty in Stage 2. On the other hand, some two-stage frameworks phrase Stage 1 solutions as hard commitments, and include explicit recourse variables in both stages of OP to model the correction in Stage 2. Some optimization problems are more natural to express according to one perspective than the other, while some problems might be straightforward to express in either. This section aims to show that our framework, while explicitly stated and motivated according to the first perspective, is in fact general enough to also easily model the second perspective of hard commitments and recourse actions. In what follows, we first describe different types of variables and how our framework can capture them. Then, we give two example problems that respectively use the soft/hard commitment perspectives, and we detail how the problem can be modelled.
Soft commitment variables: These are variables which represent decisions that correspond to soft commitments made in Stage 1 in an application, namely decisions that may be modified once the true parameters are revealed, but at a cost or penalty. The discussion in Section 3 is tailored for this kind of variables—simply define such a variable in Stage 1 and use a finite penalty function to model the cost of changing this soft commitment in Stage 2.
Hard commitment variables: These are variables  $x_{\mathrm{hard}}^*$  which represent hard commitments made in Stage 1, meaning that after commitment, they absolutely cannot change in Stage 2. To model these variables in our framework, simply write a penalty function that is infinite whenever Stage 1 and Stage 2 solutions for these variables are different. Explicitly, add a term  $\infty \cdot \mathbb{1}[x_{\mathrm{hard},1}^* \neq x_{\mathrm{hard},2}^*]$ . This way, no Stage 2 solution will change these variables from what they were committed to in Stage 1.
Recourse/other variables: These are variables which represent explicit actions/decisions taken only in Stage 2, once the true parameters are revealed. These variables are necessary, for example, when Stage 1 actions are all hard commitment variables, to ensure that we have a mechanism for corrective action if the hard Stage 1 decisions are in any way "incompatible" with the revealed parameters. These corrective actions also typically come at a cost. Thus, to model these variables, simply include them in both Stages 1 and 2, and incorporate their cost into the objective of the optimization problem. There should also be 0 penalty for modifying these variables between the stages.
To summarize, Stage 1 actions can be classified as either soft or hard commitments, depending on whether they can be changed in Stage 2 (at a finite penalty). Stage 2 actions are classified as "recourse" variables, which are simply variables that have no penalty from changing between Stage 1 to Stage 2. The above discussion shows how our framework captures all these possibilities. We now give two example applications: the first one is more naturally expressed via the soft commitment perspective, and the second one is more natural to phrase using hard commitments+recourse. We give also their explicit formulations to demonstrate how the modelling is done in our framework.
We first show an example problem which is naturally modelled using soft commitment variables and penalty functions. Consider the product-stocking problem in Example 1 again, where regular orders have to be placed two weeks ahead of monthly deliveries. We aim to maximize the net profit by selling stocked products, under the constraint that the available storage space is limited. Each product  $i$  has a purchase price  $p_i^u$  (the price of purchasing the product from the wholesale company) and a selling price  $p_i^s$  (the price of selling the product to customers), and needs  $s_i$  space to be stocked. Let  $x_i$  denote whether the product  $i$  is ordered. In Stage 1, i.e., two weeks before the delivery, the available storage space  $Sp$  at the time of delivery is unknown, and we place the order  $x$  based on estimated space. In Stage 2, i.e., the night before the delivery, the precise available space is revealed, and we ask the wholesale company to change the order but need to pay a surcharge for last-minute changes. Assume the surcharge for the last-minute change in the order of product  $i$  is  $c_i$ . In this example,  $x_i$  is thus a soft commitment variable, and we model the surcharge  $c_i$  using the penalty function of the framework.
The proposed framework can naturally model this problem. The Stage 1 OP can be formulated as:
$$
x _ {1} ^ {*} = \underset {x} {\arg \max} \sum_ {i} (p _ {i} ^ {s} - p _ {i} ^ {u}) x _ {i}
$$
$$
\mathrm {s . t .} \sum_ {i} s _ {i} x _ {i} \leq \hat {S p}, \quad x \in \{0, 1 \}
$$
In Stage 2, the order  $x_{1}^{*}$  can be changed with surcharges, which can be modelled as a penalty function:
$$
P e n (x _ {1} ^ {*} \to x) = \sum_ {i} c _ {i} | x _ {1} ^ {*} - x _ {i} |
$$
Then the Stage 2 OP can be formulated as:
$$
x _ {2} ^ {*} = \underset {x} {\arg \max} \sum_ {i} (p _ {i} ^ {s} - p _ {i} ^ {u}) x _ {i} - \sum_ {i} c _ {i} | x _ {1} ^ {*} - x _ {i} |
$$
$$
\text {s . t .} \sum_ {i} s _ {i} x _ {i} \leq S p, \quad x \in \{0, 1 \}
$$
Next, we give an example problem which is more naturally modelled using hard commitment variables and recourse variables. Consider a production-planning problem: a company owns a set of facilities and provides services to a set of customers. Each facility  $i$  can provide a fixed amount of services  $m_{i}$  and has a fixed operating cost  $f_{i}$  in the standard working mode. The company aims to meet customer demands  $d$  at the minimum operating costs. In Stage 1, the company decides which facilities to open for production based on the estimated demands  $\hat{d}$ . This is a binary decision variable  $x_{i}$  for each facility  $i$ . In Stage 2, the orders from customers arrive and the demands  $d$  are revealed. If the services provided by the operating facilities in the standard mode cannot meet demands, the company will ask some facilities that are already operating (i.e.  $x_{i} = 1$ ) to work overtime, but naturally need to pay high overtime fees. Let  $o_{i}$  denote the unit overtime fee for producing service in facility  $i$ , and  $\sigma_{i}$  denote the amount of service provided by overtime working in facility  $i$ .
This example is naturally modelled using hard commitment variables and recourse variables. Which facilities to operate,  $x$ , is a vector of 0/1 hard commitment variables. The amount of service,  $\sigma$ , provided by the overtime working mode of operating facilities can be modeled by recourse variables, and the recourse costs are the overtime fees  $o$ . Using hard commitment variables and recourse variables, the Stage 1 OP can be formulated as:
$$
x _ {1} ^ {*}, \sigma_ {1} ^ {*} = \arg \min  _ {x, \sigma} \sum_ {i} f _ {i} x _ {i} + \sum_ {i} o _ {i} \sigma_ {i}
$$
$$
s. t. \sum_ {i} (m _ {i} + \sigma_ {i}) x _ {i} \geq \hat {d}, \quad x \in \{0, 1 \}, \quad \sigma \geq 0
$$
In Stage 2, we include a term  $\infty \cdot \mathbb{1}[x_1^* \neq x]$  in the penalty function part of the Stage 2 objective to make sure that  $x$  cannot be changed, while the penalty for changing  $\sigma$  is zero since it is a recourse variable. The Stage 2 OP is formulated as:
$$
x _ {2} ^ {*}, \sigma_ {2} ^ {*} = \arg \min  _ {x, \sigma} \sum_ {i} f _ {i} x _ {i} + \sum_ {i} o _ {i} \sigma_ {i} + \infty \cdot \mathbb {1} \left[ x _ {1} ^ {*} \neq x \right]
$$
$$
s. t. \sum_ {i} (m _ {i} + \sigma_ {i}) x _ {i} \geq d, \quad x \in \{0, 1 \}, \quad \sigma \geq 0
$$
In summary, we discussed how to model in our framework soft and hard commitment actions in Stage 1, as well as recourse/other actions in Stage 2. We gave two concrete examples to demonstrate how such modelling can be done.
A.2 What if correction/recourse is not possible in the application?
The motivating premise of this paper is that the application scenario at hand allows for some post-hoc corrective action once the true parameters are revealed. One natural question is: what if such
corrective action (Stage 2 actions) is not actually possible in the application? For example, in our running example of the product-stocking problem, we considered a wholesale company that allows for order changes the night before. Other wholesalers may not allow such a correction/modification. Our framework can essentially still model these scenarios: just set the penalty of modification to infinity (or at least, very large numbers for practice). Concretely, use the penalty function  $\infty \cdot \mathbb{1}[x_2^* \neq x_1^*]$  (or replace  $\infty$  with a very large number). This penalty function encourages the learning algorithm to learn conservative predictions that maximize the chances of yielding Stage 1 decisions that remain feasible in Stage 2.
To show this, we ran another quick experiment, using the 0-1 knapsack problem setting in the paper (with knapsack capacity  $= 100$ ). This time, as we varied the magnitude of the penalty function, we measure at test time the empirical fraction of Stage 1 solutions that remain feasible under the true parameters. The results in Table 6 demonstrate our claim that as the penalty term increases, the predictions get more and more likely to remain feasible, making it a reasonable way to train a predictor even when Stage 2 correction mechanisms do not actually exist in the application.
Table 6: Mean and standard deviation of empirical fraction of Stage 1 solutions that remain feasible in Stage 2, for the 0-1 knapsack problem when capacity is 100 using the Two-Stage Predict+Optimize framework.
Penalty Factor	Feasibility%
0.05	0.00%±0.00%
0.25	0.00%±0.00%
0.5	1.73%±0.52%
1	50.93%±1.92%
2	51.63%±1.22%
4	99.07%±0.31%
A.3 Two-Stage Predict+Optimize vs Prior Hu et al. Framework
As mentioned earlier in Section 3, Two-Stage Predict+Optimize is technically mathematically equivalent to the prior framework of Hu et al. [12], in the sense of expressiveness, ignoring differentiability issues. On the one hand, we can regard the Stage 2 optimization as a form of correction function, and hence Two-Stage Predict+Optimize can be considered as a special case of the Hu et al. [12] framework. On the other hand, given a correction function as in the Hu et al. [12] framework, we can simply modify the penalty function such that we keep the penalty value of the corrected solution, and make the penalty value infinite for all other potential Stage 2 solutions. This forces the Stage 2 optimization to always emulate the correction function. In this sense, our Two-Stage Predict+Optimize framework can also emulate the Hu et al. [12] framework, meaning that the two frameworks are technically equivalent.
Nevertheless, the Two-Stage Predict+Optimize framework is both conceptually simpler and easier to apply. In the main paper, we showed how to perform end-to-end neural network training within this new framework whenever both stages of optimization can be phrased as MILPs, and also give empirical experimental results. Together, they demonstrate the much more general applicability of the Two-Stage Predict+Optimize framework.
We end this appendix with the statement and short proof that, conditioned on the same penalty function and prediction model, Two-Stage Predict+Optimize always outputs at least as good a final solution as the prior framework using any correction function.
Proposition A.1. Consider an arbitrary minimization Para-OP  $P$ , penalty function Pen, correction function  $x_{corr}^{*}$ , estimated parameters  $\hat{\theta}$  and true parameters  $\theta$ . Let  $x^{*}(\hat{\theta})$  and  $x_{1}^{*}(\hat{\theta})$  both denote the estimated solution from the estimated parameters  $\hat{\theta}$ ,  $x_{2}^{*}(\hat{\theta},\theta)$  be the output final solution from the Two-Stage Predict+Optimize framework, and  $x_{corr}^{*}(\hat{\theta},\theta)$  be the output corrected solution from the prior framework of Hu et al. Then,
$$
o b j (x _ {2} ^ {*} (\hat {\theta}, \theta), \theta) + P e n (x _ {1} ^ {*} (\hat {\theta}) \to x _ {2} ^ {*} (\hat {\theta}, \theta)) \leq o b j (x _ {c o r r} ^ {*} (\hat {\theta}, \theta), \theta) + P e n (x ^ {*} (\hat {\theta}) \to x _ {c o r r} ^ {*} (\hat {\theta}, \theta))
$$
Proof. Observe that both sides of the inequality are the objective of the Stage 2 optimization problem, evaluated at  $x_{2}^{*}$  and  $x_{corr}^{*}$  respectively. Since  $x_{2}^{*}$  is the optimal solution to the minimization problem, the inequality follows directly.
B Gradient Calculations for Problem (5)
Approximating  $\frac{\partial x_1^*}{\partial\hat{\theta}}$ . In the context of the MILP, the unknown parameter  $\hat{\theta}$  may either be  $c, A, b, G$  or  $h$ . Using the solution  $x$  and the barrier weight  $\mu$  returned from solving Problem (5), we can compute the relevant derivatives of  $\frac{\partial x_1^*}{\partial\hat{c}}$ . The case of  $c$  has already been derived by Mandi and Guns [18] (see Appendix A.1 and A.2 in their paper). Problem (5) can be rewritten as:
$$
x ^ {*} = \underset {x ^ {\prime}} {\arg \min } c ^ {\prime \top} x ^ {\prime} - \mu \sum_ {i = 1} ^ {d + q} \ln \left(x _ {i} ^ {\prime}\right) \tag {6}
$$
$$
\mathrm {s . t .} A ^ {\prime} x ^ {\prime} = b ^ {\prime}
$$
$$
c ^ {\prime} = \left[ \begin{array}{c c} c & 0 \end{array} \right] \in \mathbb {R} ^ {d + q}
$$
$$
x ^ {\prime} = \left[ \begin{array}{c c} x & \bar {s} \end{array} \right] \in \mathbb {R} ^ {d + q}
$$
$$
A ^ {\prime} = \left[ \begin{array}{c c} A & 0 \\ G & - I \end{array} \right] \in \mathbb {R} ^ {(p + q) \times (d + q)}
$$
$$
b ^ {\prime} = \left[ \begin{array}{c} b \\ h \end{array} \right] \in \mathbb {R} ^ {p + q}
$$
Fact B.1. Consider the LP relaxation (6), defining  $x'$  as a function of  $c', A'$  and  $b'$ . Then, according to Mandi and Guns [18], under this definition of  $x^*$ ,
$$
\left[ \begin{array}{c c c} - X ^ {\prime - 1} T & A ^ {\prime \top} & - c ^ {\prime} \\ A ^ {\prime} & 0 & - b ^ {\prime} \\ - c ^ {\prime \top} & b ^ {\prime \top} & \frac {\kappa}{\tau} \end{array} \right] \left[ \begin{array}{c} \frac {\partial x ^ {\prime}}{\partial c ^ {\prime}} \\ \frac {\partial y ^ {\prime}}{\partial c ^ {\prime}} \\ \frac {\partial \tau}{\partial c ^ {\prime}} \end{array} \right] = \left[ \begin{array}{c} \tau I \\ 0 \\ x ^ {\top} \end{array} \right]
$$
where  $X' = \text{diag}(x')$ ,  $t = \mu X'^{-1} e$ ,  $T = \text{diag}(t)$ ,  $y'$  is the lagrangian multiplier of Problem (6), and  $\kappa$  and  $\tau$  are additional variables added by Mandi and Guns [18] to represent the duality gap. The gradient  $\frac{\partial x_1^*}{\partial \hat{c}}$  can be obtained by solving this system of equalities.
Define the notation  $f(x, c, G, h) = c^\top x - \mu \sum_{i=1}^d \ln(x_i) - \mu \sum_{i=1}^q \ln(G_i^\top x - h_i)$ . Then, Problem (5) can be expressed as finding:
$$
x ^ {*} = \underset {x} {\arg \min } f (x, c, G, h) \text {s . t .} A x = b \tag {7}
$$
Using this notation, we write down the following four lemmas on computing  $\frac{\partial x^{*}}{\partial G},\frac{\partial x^{*}}{\partial h},\frac{\partial x^{*}}{\partial A}$ , and  $\frac{\partial x^{*}}{\partial b}$  approximately.
Lemma B.2. Consider the LP relaxation (7), defining  $x^{*}$  as a function of  $c, A, b, G$  and  $h$ . Then, under this definition of  $x^{*}$ ,
$$
\frac {\partial x ^ {*}}{\partial G} = \left(H ^ {- 1} A ^ {\top} \left(A H ^ {- 1} A ^ {\top}\right) ^ {- 1} A H ^ {- 1} - H ^ {- 1}\right) f _ {G x} (x, c, G, h)
$$
where  $H = f_{xx}(x, c, G, h)$  denotes the matrix of second derivatives of  $f$  with respect to different coordinates of  $x$ , and similarly for other subscripts, and explicitly:
$$
f _ {x _ {k} x _ {j}} (x, c, G, h) = \left\{ \begin{array}{c} \mu x _ {j} ^ {- 2} + \mu \sum_ {i = 1} ^ {q} G _ {i j} ^ {2} / \left(G _ {i} ^ {\top} x - h _ {i}\right) ^ {2}, \quad j = k \\ \mu \sum_ {i = 1} ^ {q} G _ {i j} G _ {i k} / \left(G _ {i} ^ {\top} x - h _ {i}\right) ^ {2}, \quad j \neq k \end{array} \right. \tag {8}
$$
$$
f _ {G _ {\ell r} x _ {j}} (x, c, G, h) = \left\{ \begin{array}{l l} \mu G _ {\ell j} x _ {j} / (G _ {\ell} ^ {\top} x - h _ {\ell}) ^ {2} - \mu / (G _ {\ell} ^ {\top} x - h _ {\ell}) & r = j \\ \mu G _ {\ell j} x _ {r} / (G _ {\ell} ^ {\top} x - h _ {\ell}) ^ {2} & r \neq j \end{array} \right.
$$
Note that when there are no equality constraints, i.e.,  $A = 0$ , we have
$$
\frac {\partial x ^ {*}}{\partial G} = - H ^ {- 1} f _ {G x} (x, c, G, h)
$$
which is the same as the Lemma 3 in [12].
Proof. Using the Lagrangian multiplier  $y$ , the Lagrangian relaxation of Problem (7) can be written as
$$
\mathbb {L} (x, y; c, G, h) = f (x, c, G, h) + y ^ {\top} (b - A x) \tag {9}
$$
Since  $x^{*} = \arg \min_{x} f(x, c, G, h)$  s.t.  $Ax = b$  is an optimum,  $x^{*}$  must obey the Karush-Kuhn-Tucker (KKT) conditions, obtained by setting the partial derivative of Equation (9) with respect to  $x$  and  $y$  to 0. Let  $f_{x}(x, c, G, h)$  denotes the vector of first derivatives of  $f$  with respect to different coordinates of  $x$ ,  $f_{xx}(x, c, G, h)$  denotes the matrix of second derivatives of  $f$  with respect to different coordinates of  $x$ , we obtain:
$$
\begin{array}{c} f _ {x} (x, c, G, h) - A ^ {\top} y = 0 \\ A x - b = 0 \end{array} \tag {10}
$$
The implicit differentiation of these KKT conditions with respect to  $G$  allows us to get the following system of equalities: