Venue: ICLR 2022 Conference

Reviews:

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: The paper presents Structured Energy As Loss (SEAL) that uses structured energy networks as a trainable loss for a feedforward network. SEAL is a general framework that supports multiple loss functions, including margin-based loss, regression-based loss, and NCE-based loss, and generalizes the recent approach in [Tu et al., 2020]. The paper presents extensive experiments on multi-label classification on a range of feature-based and text-based datasets. SEAL shows significant improvement over standard cross entropy trained networks, as well as approaches based on Structured Prediction Energy Networks (SPEN).
main_review: Strengths:
- Interesting approach for solving structured prediction problems that shows significant improvement in performance (F1 score) on feature-based datasets
- A thorough experimental analysis, including important ablation study, that demonstrates the strengths of the proposed approach.


Weaknesses:
- The proposed approach is a generalization of [Tu et al., 2020] (that is represented by the margin-based SEAL version). The proposed approach is not restricted to margin-based loss and indeed the new loss functions outperform margin-based loss, however the performance gain is smaller compared to the performance gain over non-SEAL approaches.
- I have some concerns about the reporting of the experimental results:
	1) The description of the experimental results is a bit misleading due to the focus on the performance gain over cross entropy (e.g., "+50 F1") in  Section 4 and the abstract. CE is not the state-of-the-art, and the performance gain over the state of the art should be reported (which, as far as I understand from the work, is margin-based SEAL that appears in [Tu et al., 2020]).
	2) Results on large text datasets only contain SEAL with NCEranking loss. It is not clear why the other losses (including the one that represents [Tu et al., 2020]) and other existing approaches like SPEN and DVN are not reported. It is hard to evaluate the gain of SEAL over these approaches.
- The paper only considers one type of structured prediction benchmarks: multi-label classification. It would be interesting to show the performance on additional tasks like POS tagging and NER.
- Some related literature that is missing: recent works that have used NCE for training energy based models for text-based datasets [1, 2].


Minor comments, typos, etc:
- page 1 abstract line 1 "this"
- page 1 abstract "compared" -> "Compared"
- page 2 "network  using": looks like double space
- page 2 "as efficient at inference as well as a feed-forward ....": "as well as" -> "as"
- page 2 "Despite these efforts, .... remain relatively inefficient" and "are finicky to train" - need more details or relevant citations
- page 4 "(Ma & Collins)" - missing year
- page 9 conclusion: "BLEURTSellam"


[1] Bhattacharyya, S., Rooshenas, A., Naskar, S., Sun, S., Iyyer, M., & McCallum, A. (2020). Energy-based reranking: Improving neural machine translation using energy-based models. arXiv preprint arXiv:2009.13267.

[2] Bakhtin, A., Deng, Y., Gross, S., Ott, M., Marc'Aurelio Ranzato, & Szlam, A. (2021). Residual Energy-Based Models for Text. J. Mach. Learn. Res., 22, 40-1.

summary_of_the_review: Overall, I think the paper presents an interesting approach that shows significant improvement in multi-label classification. However, the experimental analysis could be strengthen by improving the presentation of the results, and by considering more benchmarks.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper proposes a framework named SEAL that can adopt structed energy network as both a trainable and adaptable loss function for training feedforward networks. 7 feature-based 3 text-based datasets are used for testing SEAL and showed better results and even better with combinations. Rich ablation study was given as well.
main_review: The general idea of SEAL is interesting since various energy-network-learning objectives can be plugged into SEAL. Such as various losses of margin-based, regression-based and contrastive-sampling-based. Figure 1 is intuitive for understanding the major updates of the network structures in SEAL. Generally, this paper is well written with good motivations, technical details and rich experiments.



Detailed questions and comments:
1.	In algorithm 1, “kaiming initialization” stands for? Do you mean Kaiming He’s method? Can you give a citation as well? What will happen if you use other types of initialization methods?
2.	In table 2, any detailed information about parameter sizes, training time and inference time comparisons among the baselines and SEAL variants? Any other types of networks that are not related to energy network to solve the tasks in table 2?  [the appended information from the authors answered this question in detail and thanks for that.]

summary_of_the_review: Strong:
1.	The idea of SEAL is novel and effective to a number of tasks;
2.	This paper is well written and the direction of using of dynamic loss functions is an interesting direction.

Weak:
1.	Less aware of the signicient technical aspects of using energy network for the target tasks. Any other baselines of not using energy related networks in table 2? [thanks the authors' responses for the detailed explanation of this point. Now I have a better understanding of the benefits of applying "energy-based methods" to the corresponding tasks.]
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper proposes a framework to use Structured Prediction Energy Network (SPEN, a prior work) as a loss function. Previously SPEN is a deep architecture to learn an energy function of candidate labels, which captures structural dependencies between labels that would lead to intractable graphical models. The author of this paper finds out that it is effective to jointly learn the parameters in SPEN and the parameters in the feed-forward network (MLP). In the training time, the model alternatingly minimizes the parameters in SPEN by fixing the parameters in MLP; and then minimizes the parameters in MLP by fixing the parameters in SPEN. Empirical results in 10 multi-label datasets are given, which shows good performance boost from using the proposed methods. 
main_review: The motivation of this paper is reasonable as the author pointed out in the introduction section: "The key idea is to provide the feed-forward network access to rich relationships in the output space through a learned loss function, and do this in a dynamic manner by updating the loss based on the current outputs of the feed-forward network".

General Significance: The main significance of the model is an adoption of STEN as a trainable loss function for learning an MLP, which could achieve better model quality with this combination. Regarding training stability and computation efficiency, from the discussion of the paper, it seems that back-propagation of gradient can be used in alternating fashion to optimizes the energy network (STEN) and the MLP. This seems easier and more stable (from the attached implementation based on AllenNLP library - pytorch) than training an energy network alone. The impact of this paper is limited because the experimental dataset are mostly on a label cardinality under 1000. It would be more interesting if the author could use a larger dataset (regarding number of train/val/test examples) and a larger label cardinality. Could take a look at Extreme datasets (http://manikvarma.org/downloads/XC/XMLRepository.html).

Novelty: This method is largely based on the STEN work (reused most building component of STEN, e.g., the energy network is built the same way), but it adds interesting extension which could open opportunities for the application of energy-based learning. Regarding related work, could you also discuss how "label embedding" (https://arxiv.org/abs/1503.08677) and "joint learning of label embedding and the feedforward network" (e.g., https://arxiv.org/pdf/1805.04174.pdf) relate to your proposed work here.

Technical Quality: The overall technical quality is on the fence. It would be nice if the author could provide some more explanations on the experiment results:
1. Why the ranking-based methods (in table 2) are significantly better than others in delicious and cal500 dataset?
2. Why the SEAL-based methods  lead by a large margin on genbase dataset? (is it something related to the nature that this is a very small dataset?)
3. Why DVN performs so well on cal500? (seems to be another small dataset).
4. What if you replace the adapter-based BERT encoder with a full finetuning on BERT?
5. Mentioned above, provide benchmark results on larger datasets.

Clarity: The paper is well-structured, with clear relationships between each section. The writing is good. I especially like Figure 1, which is very illustrative thats explains a good amount of details with a simple graph. 

Minor typos (issues):
* Page 5. Section 3 Learning with dynamic loss. should probably say "Lastly, these models were designed for multi-class classification" (remove "do").

summary_of_the_review: I am more inclined to marginally accept the paper given its current status.
review_title: null

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: This paper proposes a method to utilize a SPEN as a trainable loss function for training a neural network. In particular, the authors propose to train the energy function utilizing NCE loss compared to other loss functions used by prior works, and show that such a choice improves performance of the resultant classifier.


main_review: The paper proposes an interesting problem -- capturing the higher level dependencies in data utilizing an energy function compared to assuming conditionally independent predictions.

However I have several major concerns about the paper.

The first major concern is that the underlying presentation of the paper is somewhat poor. For example, one of the major claims in the paper are not supported. The authors states such that an energy function is interpretted as a dynamic loss function, but the method section does not really specify what this means.

The paper further appears to be in a rushed state and there are a large number of different typos -- for example

noise contarstive estimation -> noise contrastive estimation.  
hyptothesize -> hypothesize.    
avergae -> average.    
f1 -> F1.   

In addition, the empirical comparisons appear to be limited and evaluated only on low dimensional datasets. Can the authors show a practical use case for the proposed approach on an existing large scale benchmark? Or compare utilizing existing datasets from past works?

The underlying technical novelty of the approach also seems a bit poor -- the main contribution appears to be applying an InfoNCE loss to train an energy function. It is also not clear from the text why this loss is beneficial during training. For example, the authors claim that their new loss prevents adversarial training. I don't follow this as both models are still optimized at the same time -- with one model encourage to have lower energy generations and another to increase the energy of generated samples.  For furthermore, why is it that the proposed loss would leave to a gain of performance?
summary_of_the_review: Due to poor presentation, a lack of empirical comparison, and no theoretical analysis in the underlying paper, I believe this paper is not in a suitable form for publication right now.
review_title: null

Meta-review: This paper proposes a new method for multi-label classification, which leverages the advantage of the emery-based model. However, one reviewer and the area chair have two serious concerns on the experiments: (1) The proposed method is only evaluated on low dimensional datasets; (2) Some important baselines methods are missing, which makes the comparison not convincing. I suggest the authors to evaluate their methods on more datasets, and add the results from well known multi-label classification method for comparison.
Decision: Reject