Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: The authors propose using the structured energy network as a loss to achieve faster and better performance on structured prediction problems. They demonstrate that their approach achieves competitive results, where a dynamic component of the loss can adapt to specific characteristics of the problem at hand.
strengths_and_weaknesses: The authors present a novel framework where a trainable loss function is used to train a network to perform the inference through a forward pass. The paper is well structured, and the main concepts are clearly conveyed to the reader. The authors created many experiments and considered the ceteris paribus principle to ensure the results could be attributed to a particular change or aspect of their framework, ensuring the quality of research. They demonstrated their approach on various tasks and datasets, obtaining good results. Nevertheless, we miss code to ensure reproducibility or a statement ensuring such code would be made available upon acceptance.
questions: 1) The authors measure the performance of their model using F1 scores for classification. While the results are promising and valid, would it be possible to compute the AUC ROC too? This would provide a measurement agnostic to the cut thresholds, providing a better assessment of the classification performance. Furthermore, clarifications should be provided regarding how the cut threshold is computed to obtain the F1 score.

2) The authors report the mean IoU metric for image segmentation. It would make sense to compute some additional metrics (e.g., https://cocodataset.org/#detection-eval) to make it easier to compare the results against other well-known models? Furthermore, would it make sense to execute additional experiments regarding image segmentation on a well-known benchmark dataset (e.g., PASCAL VOC)? 

3) In the description provided for Table 1, please consider: "are marked with underline" -> "are underlined"

4) The authors may be interested in the following research work, at least for the literature review section: Bechtle, Sarah, et al. "Meta learning via learned loss." 2020 25th International Conference on Pattern Recognition (ICPR). IEEE, 2021.

5) In Table 1, the authors report the inference speed, but no information is provided regarding the hardware on which such models were executed for inference. Could such data be provided? We consider hardware specifications to provide context to put execution times in perspective.
limitations: We did not find potential negative societal impacts in the present work. The authors did not provide details regarding the limitations of the proposed approach.
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: Energy based networks have to run gradient based inference (GBI), which can be quite expensive. In this paper, the authors propose to use an energy based network to develop a structured prediction loss. Thereafter (or simultaneously for the dynamic approach), a feedforward network is learned that tries to match the energy network's output (modeling structure) and a cross-entropy loss w.r.t. ground truth outputs. Combined, the proposed technique gives the inference efficiency of feedforward networks, yet is able to model structure similar to an energy network. 

strengths_and_weaknesses: ### Strengths

The proposed method is quite simple, yet effective. The idea is fairly obvious, but getting the details right is laudable. The paper is decently well-written and well-motivated. 

### Weaknesses

* Limitations should be better discussed: The use of a feedforward network to map the input to the GBI output of the energy based network can be thought of as "learning to do gradient-based inference". It is unclear to what extent the strategy can be expected to generalize. 

* Relation with InfNet: The authors argue InfNet is a special case of their proposed method. While I understand that there is a train-test mismatch for InfNet, the follow-up by Tu et al. (2020) where two separate networks are used for learning an adversarial sampler and for performing inference gets very close to the proposed method. It would be helpful to focus more on the discussion separating these methods. Currently, the discussion makes it seem like the proposed SEAL method is akin to a slight extension. 

Post rebuttal: The authors justifiably addressed my concerns and I have accordingly raised my scores.

questions: * Is the InfNet model in Table 1 from Tu et al. (2020)? That is, is the updated and closer to the current method version of the baseline being used for comparison?

* How expensive is SEAL's training? How does it fare in comparison to other methods? I suspect that twice the model size (due to a Feedforward network and an energy based network) plus having to run GBI to get the energy-based network's output will yield a significant training cost. 

* Alternating minimization optimization problems can often be sensitive and hard to tune. Did that cause any difficulty for the dynamic model? Regardless, I think this point should be discussed.


#### Minor comments:

* The global energy in Equation 2 does not have a dependence on x. Is this correct?
limitations: It will be valuable to discuss the limitations of using a feedforward network to "learn to do GBI".
review_title: null

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: The paper proposes the combined use of structured prediction energy networks (loss-net) as a loss function that considers relations in the output. This loss function is then used to train a feed forward network (task-net) to speed up inference at test time. They proposed both a static and dynamic approach to train these two networks. 
strengths_and_weaknesses: The paper is well written and easy to follow, it also does a good job in placing itself in the literature and, as the authors state themselves, it is a generalization of previous work [Tu et al, 2020] aimed at reducing inference time for energy networks. Results show remarkable improvement over the state of the art. 
questions: 
- The loss function of task-net is a weighted average with parameters $\lambda_1$ and $\lambda_2$, the authors fix the latter to 1 and perform bayesian search to determine the best $\lambda_1$. The variability of this parameter is high and for certain tasks or energy losses it sets to be quite small. For example, Table 7 in Appendix C.2.1 shows that for genbase (the best performing task for the presented model) the influence of loss-net is quite small compared to simple BCE. Nonetheless, the model performs substantially better than CE. Could authors provide an intuition of why this is happening? It seems to be consistently true for margin as an energy loss across tasks and there are other interesting patterns for different tasks. A discussion on this might help in future usability of the proposed approach. 
- Could the author provide confidence intervals for the results?
- Given the closeness of the results for SEAL-static and SEAL-dynamic which one would the authors recommend for use? Is there any advantage in one above the other based on context? 
- What’s the behavior in training of SEAL-dynamic? I imagine that given the alternating minimization approach training would be highly variable in early stages. Does it require more epochs to be trained due to its alternating minimization nature? Could the author provide details in this regard? I could not find any mention of it either in the main text nor the appendix. 
- Similar to the question above, how is batch size selected? It seems it should be a crucial parameter in stabilizing training and performances but it is not mentioned. 

limitations: See questions.
review_title: null

Meta-review: metareview: This work proposes using structured energy networks as loss functions for training feed forward networks to solve structured prediction tasks. The reviewers find the paper to be well written and easy to follow. The contribution is well positioned with respect to the literature and empirical results are strong. During the discussion period the authors addressed the concerns of the most negative reviewer sufficiently for them to increase their score. I can therefore recommend accepting this paper.
confidence: Certain
Decision: Accept