Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 3: Reject: For instance, a paper with technical flaws, weak evaluation, inadequate reproducibility and incompletely addressed ethical considerations.
review_content: summary: This paper proposes LODL as a surrogate to replace the original optimization loss while approximately providing gradient information of the original optimization loss. The key argument is that the gradient information may be difficult to obtain when solving complex optimization problems, such as non-convex optimization. In LODL, a surrogate loss is constructed based on variants of MSE/quadratic functions over a dataset sampled around each training sample, such that it approximates the original optimization loss and is easily differentiable.
strengths_and_weaknesses: Pros:
+ Constructing surrogates to obtain the gradients of the downstream optimization with respect to the predictions is important for decision-focused learning.

Cons:
- The computational complexity of LODL is high. It's true that we reduce the dimensionality (Line 126), but the value of $N$ can be very large (which is typically the case in decision-focused learning where the number of training samples is large). Also, we have to learn a surrogate to approximate DL for each training sample. The total complexity will be way much higher than existing methods.

- LODL is still an approximation-based method to differentiate the optimizer. There's no analytical/theoretical evidence to show that LODL can approximate the gradient of the original optimizer with respect to the predictions with sufficiently high accuracy. LODL is actually approximating DL, but this doesn't mean the gradients of DL with respect to the prediction $y$ is still well approximated by the gradient of LODL. One can easily construct counter examples in which two functions have similar values but dramatically different gradients in a neighborhood.

- The way to sample $y$ to train LODL and learn $\phi_n$ around each training sample $y_n$ is questionable. The samples are randomly generated based on a prior distribution in stage 1, but in stage 2 of learning the prediction model, the output --- predictions --- can follow a different distribution than the assumed prior distribution in stage 1. A direct consequence is that $\phi_n$ may not be accurate for the new distribution of predictions in stage 2, raising further concerns with the use of pre-trained $\phi_n$. This is also against the core idea of decision-focused learning where we want to learn the predictions by considering the entire decision pipeline as a single process.

- Some analysis of LODL would be useful, e.g., sampling complexity and generalization bounds.

- The authors are suggested to highlight the targeted scenario of LODL rather than general decision-focused learning. For example, in convex problems, we can efficiently and accurately differentiate the optimizer with respect to predictions, and so LODL is not needed or advantageous. 
questions: - The computational complexity of LODL is high. It's true that we reduce the dimensionality (Line 126), but the value of $N$ can be very large (which is typically the case in decision-focused learning where the number of training samples is large). Also, we have to learn a surrogate to approximate DL for each training sample. The total complexity will be way much higher than existing methods.

- LODL is still an approximation-based method to differentiate the optimizer. There's no analytical/theoretical evidence to show that LODL can approximate the gradient of the original optimizer with respect to the predictions with sufficiently high accuracy. LODL is actually approximating DL, but this doesn't mean the gradients of DL with respect to the prediction $y$ is still well approximated by the gradient of LODL. One can easily construct counter examples in which two functions have similar values but dramatically different gradients in a neighborhood.

- The way to sample $y$ to train LODL and learn $\phi_n$ around each training sample $y_n$ is questionable. The samples are randomly generated based on a prior distribution in stage 1, but in stage 2 of learning the prediction model, the output --- predictions --- can follow a different distribution than the assumed prior distribution in stage 1. A direct consequence is that $\phi_n$ may not be accurate for the new distribution of predictions in stage 2, raising further concerns with the use of pre-trained $\phi_n$. This is also against the core idea of decision-focused learning where we want to learn the predictions by considering the entire decision pipeline as a single process.
limitations: Yes.
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: This paper proposes methods to approximate the decision-focused loss function that quantifies the quality of a prediction function by the quality of its induced decision. The proposed method considers several classes of locally parameterized loss functions at each label value in the sample. These loss functions are convex functions of the prediction. The parameters in the loss function for each label value in the sample are estimated by minimizing the loss approximation error at randomly sampled prediction values around the corresponding label truth. 
strengths_and_weaknesses: Originality: The idea of assigning different parameters to the loss function at different label values appears novel. This is an interesting idea to improve the expressive power of the loss function class that builds on relatively simple parametric losses (to guarantee convexity). 

Quality: The writing quality of this paper is overall very good and the proposed idea in this paper is well executed. But some important limitations of the proposed approach (e.g., computation and scalability) lack discussion. 

Clarity: I found this paper is overall well-written and the high-level idea of this paper is easy to grasp. One exception is that the experiment section seems to lack some important details. 

Significance: Although this paper proposes an interesting idea, I am somewhat skeptical of its practicability considering the challenges in parameter tuning and the computational scalability of the proposed method. 
questions: 1. The proposed method learns the loss function parameters using samples generated by perturbing the label values (e.g., by adding gaussian noises). The magnitude of perturbation is obviously a very important tuning parameter. But how should this tuning parameter be selected? The selection of this tuning parameter is not discussed at all. Even the experiment section does not mention this. 

2. In the WeightedMSE and DirectedWeightedMSE losses, are the weights required to be nonnegative? Line 170 to 171 suggest that the weights are free parameters without any constraints. But negative weights along some dimensions mean that large prediction errors on those dimensions can actually decrease the loss, which doesn't seem desirable. I think negative weights may also encourage predictions that are far away from the labels in the training sets, where the loss function is least accurately estimated. This may also cause problems for the idea of using locally perturbed samples. 

3. The proposed approach needs to learn parameters for each label value separately. So if there are N training data points and we sample K local data points for each label value, then computing the DL loss function would require solving KN optimization problems. Isn't this very challenging for even moderately large-scale problems? Unfortunately, the computation limitation is not explicitly discussed in this paper and the problem sizes in the experiment section aren't clear about this either. For example, the perturbation process, the local sample size $K$ for the results in Table 1, and the running time information are not provided. 

Some other comments:
- Is the $\hat y^\top Q\hat y$ term in portfolio optimization a type? 
- The uses of the letters $n, N$ are confusing. From the equation below the line (77), $n$ refers to training data indices, and $N$ refers to training data sample size. However, in Section 5, $N$ seems to refer to the dimension of the uncertain variables, e.g.,  the number of resources or the number of stocks. 
- The processes of generating data based on real datasets in web advertising and portfolio optimization experiments are not clearly described. 
- Figure 2 seems interesting but I do not totally get the reason for the observed patterns. Can the authors provide some explanations? 
- The authors may hope to cite some additional literature on decision-focused learning, such as 
  - Elmachtoub, Adam, Jason Cheuk Nam Liang, and Ryan McNellis. "Decision trees for decision-making under the predict-then-optimize framework." International Conference on Machine Learning. PMLR, 2020.
  - Kallus, Nathan, and Xiaojie Mao. "Stochastic optimization forests." Management Science (2022).
  - Hu, Yichun, Nathan Kallus, and Xiaojie Mao. "Fast rates for contextual linear optimization." Management Science (2022). 
  - Grigas, Paul, and Meng Qi. "Integrated conditional estimation-optimization." arXiv preprint arXiv:2110.12351 (2021).
  - Qi, Meng, et al. "A practical end-to-end inventory management model with deep learning." Available at SSRN 3737780 (2020).
limitations: See the comments in the box above.
review_title: null

final_score: rating: 5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.
review_content: summary: This paper proposes a novel locally optimized decision loss (LODL) for decision focused learning (DFL). The LODL loss is a parameterized function trained with the true decision loss as the target. The LODL loss is a relatively faithful measure of the decision quality, and can provide informative gradient for the DFL training.  The authors run the experiments on various optimization tasks including Linear Model, Web Advertising, and Portfolio Optimization, verifying the effectiveness of LODL. Also, the authors product some ablation studies and shows how well the LODL represents the decision quality.
strengths_and_weaknesses: ### Originality 
The idea of training a parameterized function to approximate the decision loss is interesting. Comparing with the MSE loss in 2-stage method, the LODL is more faithful to the decision quality. Comparing with the surrogate loss in DFL, the LODL can be easier to design. 

### Significance
The idea is novel for DFL.  

### Quality
The proposed method is compared with multiple baselines including  2-stage, DFL, and the LODL by NN on various resource allocation problems, which verifies the advantages of LODL for some problems.

### Clarity
The paper is well-written and easy to follow.  


questions: 1. The design of LODL is interesting, but I have a concern about the complexity of fitting the local loss function.  The design fits one loss function for each sample by MSE. When the prediction space is large and needs a large number of samples, the training complexity would be very high.  The authors may discuss more about the training complexity.

2. The authors show the fitting quality of LODL by measuring the mean absolute error (MAE). From the results, we also know that the LODL may not easily fit some kinds of task loss for some inputs. It would be interesting to have some study about the relationship between the fitting quality, the gradient, and the task loss or different inputs. This may give more observations supporting the claims.
limitations: The limitations of the design is clearly listed.
review_title: null

final_score: rating: 8: Strong Accept: Technically strong paper, with novel ideas, excellent impact on at least one area, or high-to-excellent impact on multiple areas, with excellent evaluation, resources, and reproducibility, and no unaddressed ethical considerations.
review_content: summary: This paper considers learning losses for predictive models that
are used in a downstream optimization task.
Section 2 summarizes the basic setup where there is a predictive
model $\hat y=M_\theta(x)$ that creates predictions used to
parameterize an optimization process.
The baselines considered are 1) 2-stage learning, which trains
the predictive model with an intermediate loss, and
2) decision-focused learning, which seeks to optimize a decision loss
with the predictive model, defined with the objective
of the optimization problem.
The locally-optimized decision losses (LODL) proposed in this paper
seek to learn the parameters of surrogate intermediate losses
to match the decision loss.
strengths_and_weaknesses: Strengths:
+ The idea of parameterizing surrogate/intermediate losses makes a lot
  of sense in these two-stage settings, and the formulation considered
  here nicely shows the benefits of learning non-Euclidean regression losses.
  I can imagine learned losses to be a crucial,
  impactful, and long-lasting contribution in these settings.
+ The experimental results clearly demonstrate that the LODL is
  able to learn a reasonable surrogate for the tasks considered

Weaknesses:
+ If we take the LODL to be the MSE loss parameterized by weights on
  each dimension, I do not understand the objective on L191:
  my interpretation is that it tries to make the weighted
  MSE loss match the decision loss around the optimal prediction
  by changing the weights.
  Since the MSE loss and decision loss are very different quantities,
  it like this objective will not be possible to optimize.
+ Even though the paper experimentally shows that LODL works, the
  results are difficult to contextualize and compare to related research.
  For example, the web advertising task starting
  at L222 takes one of the settings from
  [Wilder et al., (2019)](https://ojs.aaai.org/index.php/AAAI/article/view/3982),
  but does not provide or present the results in a way that is
  comparable to the results in that paper. The submission would be
  significantly easier to evaluate in comparison to this work
  if it reproduces exactly Table 1 of Wilder et al., (2019) and
  adds additional lines showing how well LODL performs in
  comparison.

Related work:
+ It could be interesting to connect the work to learned losses
  used in meta-learning, such as in
  [Bechtle et al. (2021)](https://arxiv.org/pdf/1906.05374.pdf),
  which takes a much more black-box perspective to learning a
  latent loss function.
questions: I would be very willing to re-evaluate my assessment after a discussion
around the following questions on weaknesses I've listed above.

1. Can you clarify how the LODL loss on line 191 should be interpreted? Did you consider alternatives to this?
2. Can you comment on how the experimental results connect to
   established experimental results in DFL settings?
limitations: The paper does not clearly discuss limitations in a dedicated section.
While parameterizing and learning an intermediate loss seems appealing,
it seems limited by needing to specify and learn the right
parameterization.
review_title: null

Meta-review: metareview: This paper considers the problem of making decision-focused learning (DFL) more usable for both researchers and practitioners. It proposes a novel approach referred to as locally-optimized decision losses (LODL) which learns the parameters of surrogate intermediate losses to match the decision loss. Experimental results clearly demonstrate that LODL approach is able to learn effective surrogate for the considered tasks.

All the reviewers appreciated the LODL idea, but also raised a number of concerns. There was a lot of discussion and authors' have addressed most of the concerns and also acknowledged some limitations pointed out by some reviewers'. One expert reviewer who deeply engaged with the authors to both clarify and improve the paper was willing to strongly champion the paper. In their words: "It's a brilliant idea that will be foundational in the space and will be engaging and thought-provoking at the conference." Couple of reviewers' raised few points beyond the author-reviewer discussion which authors' could not see/respond to. However, I think the overall strengths of the paper outweigh these concerns.

Therefore, I recommend accepting the paper. I strongly encourage the authors' to improve the paper in terms of clarity, exposition, and additional experimental results to reflect the discussion with reviewers.
confidence: Certain
Decision: Accept