Structured Energy Network as a dynamic loss function. Case study. A case study with multi-label Classification
Authors: Jay-Yoon Lee
Dhruvesh Patel
Purujit Goyal
Andrew McCallum
Venue: ICLR 2022 Submitted (2022)
Keywords: Structured Prediction
Energy network
Energy-based models
Loss-function learning
Dynamic loss function

Abstract: We propose SEAL which utilizes this energy network as a trainable loss function for a simple feedfoward network. Structured prediction energy networks (SPENs) (Belanger & McCallum, 2016; Gygli et al., 2017) have shown that a neural network (i.e. energy network) can learn a reasonable energy function over the candidate structured outputs. We find that rather than using SPEN as a prediction network, using it as a trainable loss function is not only computationally efficient but also results in higher performance. compared to SPENs in both training and inference time. As the energy loss function is trainable, we propose SEAL to be dynamic which can adapt energy function to focus on the region where feedforward model will be affected most. We find this to be effective in ablation study comparing SEAL to the static version (§4) where energy function is fixed after pretraining. We show the relation to previous work on the joint optimization model of energy network and feedforward model (INFNET) as we show that it is equivalent to SEAL using margin-based loss if INFNET relaxes their loss function. Based on the unique architecture of SEAL, we further propose a variant of SEAL that utilizes noise contrastive ranking (NCE) loss that by itself does not perform well as a structured energy network, but embodied in SEAL, it shows the greatest performance among the variants we study. We demonstrate the effectiveness of SEAL on 7 feature-based and 3 text-based multi-label classification datasets. The best version of SEAL that uses NCE ranking method achieves close to +2.85, +2.23 respective F1 point gain in average over cross-entropy and INFNET on the feature-based datasets, excluding one outlier that has an excessive gain of +50.0 F1 points. Lastly, examining whether the proposed framework is effective on a large pre-trained model as well, we observe SEAL achieving +0.87 F1 point gain in average on top of BERT-based adapter model o text datasets.

Under review as a conference paper at ICLR 2022
STRUCTURED ENERGY NETWORK AS A DYNAMIC LOSS FUNCTION A CASE STUDY WITH MULTI-LABEL CLASSIFICATION
Anonymous authors Paper under double-bline review
ABSTRACT
We propose SEAL which utilizes an energy network as trainable loss function for a simple feedfoward network. Structured prediction energy networks (SPENs) (Belanger & McCallum 2016 Gygli al. 2017) have shown that neural network (i.e. energy network) can learn reasonable energy function over the candidate structured outputs. We find that rather than using SPENs as prediction network, using it as trainable loss function is not only computationally more efficient but also better performing. As the energy function is trainable, we propose SEAL to be dynamic so it can adapt the energy function to focus on the region where feed-forward models will be affected the most at the time point of model update. We find this to be effective in ablation study comparing SEAL to the static version $ where the energy function is fixed after pretraining. We show the relation to previous work the joint optimization model of energy network and feedforward model (INFNET) as we show that it is equivalent to SEAL using margin-based loss if INFNET relaxes their loss function. Based on the unique architecture of SEAL, we further propose a variant of SEAL that utilizes noise contrastive ranking (NCE) loss that by itself does not perform well structured energy network, but embodied in SEAL, it shows the greatest performance among the variants we study. We demonstrate the effectiveness SEAL on 7 reature-based and 3 text-based multi-label classification datasets. The best version of SEAL that uses NCE ranking method (SEAL-NCE) achieves close to 2.85, 2.23 respective Fl point gain average over cross-entropy and INFNET for feature-based based datasets, excluding one dataset that has an excessively large gain. Lastly, examining whether the proposed framework is effective
large pre-trained model well, we observe SEAL-NCE achieving 0.87, 0.68 respective Fl point gain in average over cross-entropy and INFNET with BERT-based adapter model on text datasets.
INTRODUCTION
Structured prediction is a popular machine learning task wherein the model learns a mapping function from input
multivariate-strutture output y. Popular examples of this include image segmentation (Müller 2014), extracting parse trees or semantic role labels from a text (Palmer et al., 2010), and multi-label classification (Belanger & McCallum 2016;; Gygli et al. 2017). In structured prediction, the output space ) is often extremely large. For example, in multi-label classification, the size of y is 24 where model needs to predict output y E {0, 1) There are two key aspects that model for structured prediction needs to balance: statistical efficiency and computational efficiency. Most models for structured prediction can be categorized into two categories: the forward approach, wherein one learns a neural network that models the direct mapping between the input and the structured output, and the structured approach, in which the model explicitly models the interactions (structure) in the output space.
The feed-forward approach learns all dimensions of y jointly in a conditionally independent manner given input x, relying on the representational power of the network to capture dependencies in output structure implicitly. While the forward approach computationally efficient, since does not capture relationships in the label space, it lacks statistical efficiency. The traditional structured approaches, on the other hand, model joint probability distribution p(x.y) that can capture the label relationships. However, due the intractability of modeling the full joint distribution which

---

Under review as a conference paper at ICLR 2022
captures every possible interaction of output space, these approaches resort to limiting the interaction terms to local subsets of the output space (Lafferty et al. 2001; Ghamrawi & McCallum 2005). In order to make inference for structured models more efficient. a recent line of work replaces the joint probability with a structured energy function E(x. (LeeCnn et al. 2006), which can be thought of unnormalized probability distribution, allowing the model to learn arbitrary global dependencies in the output space. There have been several works that propose efficient approximate inference procedures for structured energy networks using gradient based inference (Belanger et al. 2017]] Gygli et al. 2017; Rooshenas et al. 2019), hence also called 'prediction networks'. These models showed noticeable gains in predictive performance over feed-forward models and graphical models that assume partial structure such as full pairwise potential (Chen et al. 2013 [Schwing & Urtasun 2015). Despite these efforts, the inference for energy-based models still remains relatively inefficient when compared to the feed-forward approach (Tu & Gimpel 2019). Moreover, in our experience, the models using gradient based inference (GBI) are finicky to train as the training also utilizes GBI, requiring numerous hyperparaneters: step size, number of iterations for GBL, and initial point to begin GBL. This raises a question: Can the energy network be used in a way that is as expressive as a full joint probabiliry, as efficient at inference train? We believe that using structured energy networks as a parameterized dynamic loss function for feed-forward networks, instead of prediction network, can fulfill all these requirements.
feed-forward approach, and also stable and easy to
In this paper, we propose the Structured Energy As Loss (SEAL) framework that uses a trainable structured energy network (SEN as loss function guiding the training of feed-forward network. The key idea is to provide the feed-forward network access to rich relationships in the output space through learned loss function. We also propose to learn SEN in a dynamic fashion by adjusting the energy function to be confident with the most up-to-date outputs of the teed-forward network. We show that learning the loss function dynamically leads to more efficient, more stable, and better performance. SEAL can be viewed as general-purposs framework where one can plugin various loss functions and architectures to train SEN as well as the feed-forward network. Through experiments ($4). we analyze the effect of applying different energy losses (e.g. margin -bssddd regression- based) within SEAL framework. We also propose the noise-contrastive ranking loss (NCEranking) for SEN which performs the best within the SEAL framework.
To summarize, we introduce a general framework SEAL that interprets structured energy networks (SEN) as a dynamic loss functions. Through empirical evaluation on the task of multi-label clas- sification, we analyse the impact of various loss functions for updating SEN. Finally, we propose an NCE ranking loss that is uniquely suited for the SEAL framework, and demonstrate its superior performance on 7 feature-based as well as 3 text-based multi-label classification datasets, when compared to simple feed-forward approach and various energy based models.
Lp ④
LE(
Algorithm 1: SEAL Algorithm
Require: yx Training Instance
Ee
Eg
Require: Fe: Feedforward Network
Require: sampling: True/False flag
Require: optimizere, optimizere
Require: T: No. of steps
F5
F
t + 0
Ou. t + Random initialization while do
X
sampling lnn then
$ + ;i
Kly ( ~ Fo, (x))
else
Figure 1: Overview of the SEAL framework. The figure on the right shows the update to the en- engy network (equation 4 and the figure the left shows the update to the feed-forward network (equation
S { (Fr, (x))
singleton set
+ t t+1
Update θ as Eqn.
Jusing ý E S and optimizere usiig optimizere
Update &b as Eqn.
'The term comes from structured prediction energy network, minus 'prediction', as we do not use it for prediction anymore. For brevity, use terms 'energy network' and 'streutured energy newtork' interchangeably.
2

---

Under review as a conference paper at ICLR 2022
2 STRUCTURED ENERGY NETWORK AS LOsS (SEAL)
This section first describes the proposed SEAL framework. Let X' denote the input space, y = {0, 1 the output space, and 5 [0, 1] the continuous relaxation of V. Then the structured energy Fe X' x ) + IR is defined to be parameterized function of the input and the continuous relaxation of the output. The feedforward network F&: X - y is defined to be neural network that maps an input to the continuous relaxation of the output space. We denote j-th training instance pair as (x(1), y()) D and y: E {0, 1}: I E [o, 1] to denote i-th label dimension where (I....,4} L).
As shown in Figure 1. SEAL consists two loss functions: the energy loss Lg that trains structured energy network (+), and feedforward loss Ly that guides the training of the feedforward network (.). We first discuss how structured energy network implicitly affects training of the feedforward network by defining L.) Given a training instance (x. y), the feedforward loss is defined as:
LF( = A1 Fo(x)) - A2 Σ lus log (x(x); + (1 - y1) log (1 Fo(x):)
(1)
Here, the first term involving the energy captures interaction across label space whereas binary cross-entropy considers the predictions of each label independently (More details in Appendix BB)
Since the training of feedforward network depends on the quality of the parameterized energy network, it is critical to find the parameters θ that produce the best loss surface for training the feedforward network. This can be done in two ways: by first training the energy separately, by training the energy and the feedforward network simultaneously. SEAL uses the latter, which is shown to perform better (see $5) than the former. We denote the former method SEAL-static for distinction.
In SEAL-static, we first estimate the energy network parameter C over training data and optimize + by plugging fixed ê into Lp. Given (x.y) D, trained as equation
min
s.t. e argmin LF D
Σ [L((x,Y,30)) e |D]
1
D
We finally present SEAL which learns θ dynamically at step t. To do so, we alternate the optimization steps of and Φ Given a training instance (x, y) E Br, a complete training step is given as:
O O.-1- Ve [B
1
Σ
Fo. LE Fx. ()
B.
1
Φ <- Φ1
V
Lr(
(4)
B.
B.
Note that, from 2 to 3 arbitrary ý got replaced with Fg.., (x) in estimating θ so that estimation model output Fo. (x). Further, the update step for feedforward network & relies on O, rather than static θ. In both SEAL and SEAL-static, test-time predictions are performed solely using the feedfoward network ÿz 1 (2:); 0.5).
of energy surface depends
The motivation for proposing dynamic loss function in SEAL is as following. It would be useful to learn perfect energy surface, if possible, by estimating θ that is static. However, doing so is challenging as we are estimating energy surface on the joint space of x, high-dimensional continuous space with limited data and resource. Instead, we hypothesize that it is more important to concentrate resource in depicting accurate energy surface around the current input, output pair (x, (x)). In SEAL, utilizing the fact that we have access to trainable loss function, dynamically adapt energy surface so that the region of interest for training feedforward network is well represented.
We now discuss various energy loss imargin-based regression-based, and contrastive-sapllngg -basd) that can be plugged into Lg in SEAL. For brevity, we use (e) and Lg y: θ) interchangeably where again (x.y) denotes labeled data and ý denotes probability vector. We defer the description of specific energy network structure to the experiment section ($4) as SEAL framework can work with arbitrary network structures.
3

---

Under review as a conference paper at ICLR 2022
Margin-based (Lg-me margin) SPEN (Belanger & McCallum 2016) learned structured energy net- work with SSVM loss (Taskar et al. 2004 T'occanttridhs et al. (2044) so that energy functions learn to have sufficient energy difference, larger than the margin A (y, y), between arbitrary output y and true output y. To examine the effect of the margin-based loss in SEAL, we follow SPEN and utilize the SSVM loss as
-E-margin
Σ x.Y
max [A (},y) Ee (x, + Ee (x,y)1+
(5)
rrggrsssoon) Deep Value Network (DVN) GGygi et al. 2017) attempts to learn energy network which directly outputs score that is similar the metric s(y. y) of interest, such as F1 score, that compares arbitrary y with true output y. Following DVN, by making score a( and (-) be between [0, 1J. we express regression loss as cross entropy
Regression-based
regresssion = (y, y) log - Ex( - (1 s(), log (1 (x(
(6)
Noise-contrastive ranking (Lg NCC-rankinn and (FErrnkkigg) In SEAL, are interested in captur- ing the output region that feedforward network has a high probability. We ask, rather than just taking single point y Fgg. (x) from feedforward network, whether sampling many discrete binary vectors from y and taking all those samples into consideration for learning Ee could lead in better estimation of energy surface. Motivated from noise contrastive estimation (NCE) (Ma & Collins 2018), we ask whether energy network Ee trained to contrast the true output y from groups of K negative samples drawn from feedforward output Fo (x) can induce a good loss for teaching $ The intuition is as the
y Fo (x) becomes better and better, the loss surface teaching feedforward network should become more fine grained as y and samples from y will be already close to the true y.
- -CCEnninn of our choice, we first review original form of NCE ranking loss from Ma & Collins 20188). For K samples (*) PN, k = 1..... K drawn from noise distribution PN and rewriting y as y without loss of generality, NCE ranking loss is defined as
Before we discuss LE
exp (s(x, y ());
log Σ k= exp (sxx, O))
where s(x,y; - y) log Ps(y).
Minimization of this loss makes P(ylz; Evey exp( Es(2,y)) as an unbiased estimator of true
expí (o(a,))
distribution [X) Thus, given perfect energy E(.), minimizing - (xx with respect to y in LF would be equivalent maximizing the estimate of log x).
From the presented NCE ranking method, we propose to use P(y|x; 2.) = II, P(y; |x; D,) in place of (y). The novelty of this proposal is that we view the output of feedforward network that train as a noise distribution, and that the energy model that contrasts noise distribution teaches the noise distribution as well through our SEAL framework (equation 3. In short, we get -C-NCEnnking when we plug in sx,,;;
y; () = (x, - log Pa (y) 22) into equation 7. i.e.
NCEnnkinnn log exp (s(x, +)) Ek exp (s(x, y(k); 0))
(x. y: Q) yx)(x,y) - loo Po Poly|x; T.).
(8)
This achieves two benefits that we have in mind in using NCE. First benefit is that it captures the region of interest by sampling from the output distribution of Φ,, Second benefit is that it brings more efficient NCE method. NCE method is known to work best when the noise distribution is close to the data distribution (Gutmann & Hirayamal 2012) but not exactly the same. In our SEAL learning procedure, we hypothesize that P(y|x; (b) becomes closer to the data distribution as training steps proceed.
Lastly, motivated that NCE can estimate the true distribution, we propose yet another loss function that can estimate the difference between the true probability P(y(x) (x) and feedforward network probability. We show in appendix A if we use plain ranking loss that sets s(x, () = Eel Ex y) (y) exp(-Es (x.y))
in equation unbiased estimator of P(y!x) now becomes Σyey Py (x) exp( - Fie (x.)) (9). This
II this paper, we adopt soft Fl score s(ý.y) from Gygli et al. 2017 that is defined on the continuous y E 1)
4

---

Under review as a conference paper at ICLR 2022
means if we take a derivative of withth respect to y, then we are taking derivative with respect to - log P(y|x) + log Pv (y). With the trick of replacing the (y with P(y|x; D1) in SEAL, we hypothesize that loss captured by energy network can focus only the difference between true and feedforward probability which may end up being more informative surface. In short, we get plain
ranking loss L-E-ranning by plugging y: Fi x y) into equation 7 resulting in
exp (- (x. y (0); +1)
La-rangnng
log
(10)
Ex= k= exp - Exx (k); 0))
RELATED WORK
Learning with dynamic loss: Wu et al. (2018) and Huang et al. (019)) attempt to learn a dynamic loss defined by neural network that is tailored to the task-specific metric m, such as BLEU or 0-1 accuracy. Both approaches try to to learn a loss function that can directly increase model's metric score on the validation set. Since is usually non-differentiable, |Huan et al. (2019) view this problem as a reinforcement learning problem and attempts to learn a loss function that maximizes the expected reward: higher m score in validation set. On the other hand. Wu al. (2018) relax the to be differentiable and view the feed-forward model update and loss-model update as a connected chain of computation graph, that loss function parameter (0) gets updated in relation to ferd-forward model (Ø) improving the relaxed metric score.
There are three major differences between SEAL and the previous work. First, energy surface of SEAL is not limited to a specific parametric form, oly" θ log I's (x) as previous papers do. Instead, SEAL simply utilizes a neural network to express the loss function as (.) which allows it to represent a much larger function class. Second, previous works require manual design of state vector. For example, the state vector used by Wu et al. (2018) consists of iteration number, validation accuracy, and current loss-function parameter O In contrast, SEAL only requires the input and output pair (x, F&(x)) of the feed-forward network. Lastly, since these models were designed for multi-class classification, it is non-trivial to extend them to the multi-label classification or to the general structured prediction task wherein large number of labels interact.
Structured Prediction Energy Networks: Structured prediction energy networks (Belanger & McCallum 2016) and its variants (Gygli et al. 2017 Rooshenas et al.) (2099) (referred to as SPENs
from here on) learn an energy network Ee : X' x y - I and predict the output using gradient-based inference (GBI) with the objective y = arg miny E(x, y). To train energy network, SPENs apply GBI adversarially in order to find x, y pair that violates the minimality of the energy the most. We found this training procedure to be a unstable as its success depends on the initial point of GBI. We discuss the loss function for SPEN (Belanger & McCallum 2016) and DVN (Gygli et al. 2017) in 2 and use these models as basclines in this work.
Jointly learning O and Φ: The Inference network (INFNET) proposed in Tu & Gimpel (2018) is the first work that tried to learn structured energy network and feedforward network jointly. As the name suggests, INFNET proposes a model that mimics the behaviour of SPEN by replacing the expensive gradient-based inference with feedforward networks. However, just like SPENs, INFNET also resorts to finding adversarial point that has the maximum energy violation, resulting in an adversarial framework similar to GAN (Goodfellow et al. 2014), where INFNET maximizes, and energy network minimizes the same margin based loss given in equation [5] INFNET has to serve as the adversarial sampler for training energy network. While the end goal of INFNET is efficient inference for finding output with lowest energy, having to serve two roles, INFNET suffers from train-, test-time objective mismatch and Tu et al. (2020) resolves this by augmenting INFNET structure; Two separate feed-forward networks are used, one for prediction and the other as the adversarial sampler. However, INFNET's reliance on margin-based losses (marign-rescaled higne and perceptron) still leaves the approach to be adversarial, i.e. ming maxe Lx(x, y, (x)) θ).
In contrast, SEAL has completely independent loss functions LE and Lr for updating the energy and feed-forward network. On feed-forward (Φ) update, SEAL simply update the model in the direction
"The papers (Wu et al. consistent manner with this paper.
2018 Huang et al. 2019) use O instead of θ. We change it to θ put notations in
5

---

Under review as a conference paper at ICLR 2022
#lnstances
Inpor Type
Label
Dataset
Domain
#Labels
Taxonomy
Train
Val
Test
Expe
Onlology
1636
500
Continnous
Spo FUN
Gene Ontology
1600
837
1266
500
Continuous
Forest
Bibtex
Text
4407
1497
159
Binary
Cai500
Music
283
105
114
174
Continuous
Delicious
Text
9690
3207
3194
983
Binary
Genbase
Biology
398
27
Binary
Eurlex-ev
Text
11557
3876
3881
3993
Binary
BGC
Text
58715
14785
18394
142
Raw Text
Forcat
RCV
Text
13890
9260
781265
104
Raw Text
DAG
NYT
Text
175299
177067
180659
2109
Raw Text
Table
Statistics of the datasets used in the experiments.
that reduces energy defined by structured energy network; Energy network is only used tool for evaluation of loss and SEAL framework is not interested in mimicking the test-time behaviour of SPEN as INFNET tries to. Despite the difference in the end goal. mechanically speaking. INFNET is a special case of SEAL (i.e. SEAL with Lg-margin) if the margin is removed from the SSVM loss for the feedforward model udpate. In fact. Tuu et al. (2020) reports that this special case leads more efficient and stable learning of feedforward network in their ablation study. We use this special case of INFNET represent SEAL with marging in the main experiment.
Learning label embeddings for multi-label classification Utilizing label representation has been a popular approach for multi-label classification (Wang et al. 2018 Xiao et al.. 2019; Zhang et al.) 2021). These works embed labels as vectors, allowing for richer representations, and then utilizing cross-entropy loss which models individual label probability independently. While the energy network in SEAL represents an output probability, its main purpose is to allow the gradient for each label to depend on other label probabilities, a feature which is absent from these prior works, for which label interactions must be latently encoded in the embedding space.
Eenrgy-based models used for reranking Recently, mergy-bbseed models (EBM) have been applied to text generation problems to first generate and rerank the generated sentences by EBM score (Deng al., 2020: Bhattacharyya al., (222I) The residual energy modeling of of et al. (2020) shares some similarity to the equation 9,ap from the minor difference that they utilize frozen language model for PN whereas SEAL has PN that constantly learns from the energy models. The major difference is that these previous works have simply used the scores of the energy network, whereas SEAL views the energy network loss function, leveraging the gradient from the energy network to further train another neural network.
4 EXPERIMENTS
In our experiments, we use 10 multi-label classification datasets (shown in Table covering varied label space, size, well as input characteristics. The first are small and medium sized feature based datasets, while the last three Blurb Genre Collection (BGC), RCV (Reuters I), and New York Times (NYT) are large datasets with raw text as the input
Energy Network: We use the same structure for the energy network (s y) as described in Be
langer & Mccallum 2016), wherein the energy is the sum of local Flgeal (ar, y) and global F.globall (x, y)
energies defined as:
L Σ (x(x)T(())
Elocal y)
Eglobal
-
(11)
where b;, v, M, Te contain learnable parameters, (=) log(1 e²) is the softplus activation function, and TE is feature network for structured energy network. Note that due to the presence of
"Due to their enormous size, we subset of NYT training set, which is still the largest dataset in experiments.

---

Under review as a conference paper at ICLR 2022
Table Performance of SEAL framework (row 5-9) compared feedforward network trained with cross-entropy (row 1) and sturctured energy networks (row 2-4) that are learned with Lgdescribed in earlier section. We observe feedforward network learned with SEAL is almost always better than cross-entropy learned model. We also observe while margigiiannd Læ regression not show significant difference, LE- NFFNCFrannnng making stronger performance in general.
discrete input datasets
continuous input datasets eurlexev
method
samples
bibtex
delicious
genbase
cal500
expr fun
spo_fun
cross-entropy
42.40
30.16
47.37
33.58
42.19
37.48
27.97
energy only
SPEN DVN NCE
42.99
26.76
32.50
39.36
41.75
36.43
27.20
42.73
29.59
78.43
48.21
28.90
32.19
29.97
17.93
16.78
20.64
38.87
0.19
27.12
19.33
SEAL
margin
42.86
29.75
96.53
36.69
41,83
37,91
28.43 43
regression
43.74
29.79
96.95
37.97
41.65
38.12
28.89
regression-s
44.53
29.87
96.81
38.95
42.32
37.84
28.02
NCEranking
44.76
34.79
97.32
41.62
42.78
38.21
28.70
ranking
44.20
36.04
96.60
40.71
42.78
38.03
29.68
Egl term in LF, the gradient for feedforward network, aLe can capture the dependency of ye to ws all L dimensions of y. The expression for this is provided in Appendix B
Feature Network: We use the same feature network structure for both the energy and feedforward network denoted as Tg, T'p respectively. For the the feature-based datasets, the feature network : h consists of multi-layer perceptron with with softplus nonlinearities. For the raw text pre-trained BERT (Devlin et al. 2019) with adapter (Houlsby al. 2019
based datasets, Pfeiffer et al.) 2020) as the feature network.
Feedforward Network: Given feature network Tp that generates features in R². the feedforward network (x)) (I:), where G E Lx is matrix consisting learnable embeddings, one row of embedding for each label.
Training: The models for feature-based based datasets are trained on a single TitanX GPU with 14GB CPU memory and the models for BCG, RCV and NYT trained m40 GPU. We use separte ADAM optimizer (Kingma & Ba 2014) for energy network (0) and for feedforward network (Φ) which optimizes parameters in alternating fashion For each minimization loop, we take ngg n gradient steps respectively for estimating O2 and d,. We leave these ng. R.F as hyperparameters to be tuned. The best hyper-panc parameters for each model are found using Bayesian searct and reported in Appendix We also report train, inference time and parameter size of different methods in Appendix
D
4.1 RESULTS FOR FEATURE BASED DATASETS
The experiment results across 7 casture-based dataset are presented Table 2 The table consists of feedforward network only trained with cross- eross-entropy (row 1), energy networks (row 2-4) described in previous section that is evaluated with GBI, and lastly feedforward network trained with SEAL framework with different types losses (row 5-9).
SEAL outperforms cross-entropy loss (CE) on almost all datasets with average gains over CE ranging from +0.6 (SEAL-L-g margin) +2.85 (SEAL-LE (NCC--nnnn))gg F1 points. We excluded excessive gain of genbase in computing the average as we believe it is an outlier. Further analysis on the performance on genbase is provided in Appendix F
'The code we train evaluate our models available a https://anonymouu 4open
science//n wence/rr//e.. T/SEAL///////////// README_ md
WWe use Weights & Biases (Biewald 2020 for hyperparameter search

---

Under review as a conference paper at ICLR 2022
Between LE types, Lg- -CCEraningg and La-naninng performs the best while Lg margin and Lg -egerssssion performs similarly on average. Lg- NCCEanking and -unnknning are the only ones that always outperforms CE in all datasets whereas the others sometimes perform slightly worse than CE.
With the high gains of Lg -NCErankkngg and Lannkinggg we are likely to conclude using samples rather than single point can capture the output surface of the feedforward network better. To examine this hypothesis in more detail, we test sampling approaches on regression- based loss as well. We define LE Legressssson - Lyes L regresssioon (x, y y; θ) given sample set S. We take two approaches in collecting these samples: 1. discrete binary vectors drawn from probability vector y done in LE -NE-NnCagningg and 2. continuous perturbation of y with gaussian noise. We found that discrete samples did not make any notable changes between regresoosoo and Lg -egeesssiinn-- however, found that using continuous samples made +0,4 Fl improvements in average and report Lg- -egerrsssoon as the version which uses continuous sample. We conleude that using samples are helpful in capturing better energy surface for traininig feedforward network, however, effect and characteristics of sample might differ per types of Lg. We believe the sampling is more effective on Lgg-NEEnnking Errniiig as groups of samples contribute in relative manner in learning surface in contrast to LE regression
To further study effect of different L.g, we examine whether there is a correlation in performance between the isolated structured energy network and SEAL which utilizes idential Lgg For training isolated structured energy network with margin-based based and regression- based energy loss, we follow training approaches of SPEN (Belanger & McCallum 2016) and DVN (Gygli et al. 2017). For NCE, we need access to noise distribution PN. we set pretrained feedforward network probability P( X 4) as PN (Y). We do not much of correlation between the two, in other words, charac- teristics of LE that trains best structured energy network and Lg that trains best feedforward network different. While NCE seems perform worst when utilizing L Dr=-Nirann in isolated manner, the constrastive characteristic of NCCCnnnnnn and LE- -ankinng mankingnnthhtt that is tailored to feedforward network seems to be helpful. Regardless of Lg types, we also obesreve that it is much more stable and usually higher-pen perhrrrprforming to train with SEAL than to train energy network separately. In Appendix (C, we also show compare individual energy network as prediction model compared SEAL in detail.
4.2 RESULTS FOR LARGE TEXT DATASETS
Table 3: Test Fl for text datasets.
method datasets
BGC
RCV
NYT
cross-entropy
81.15
87.18
77.4
SEAL-margin
81.14
87.01
78.13
SEAL-NCCranking
81.64
87.82
78.87
In order to test the effectiveness of SEAL in the scope of large pre-trained models, we examine text datasets with pre-trained BERT (Devlin al. 2019) with adapter (Houlsby et al.) 2019 Pfeiffer et al.) 2020) as the feature network Te and FF as shown in Table Considering the computational expense of running
searches on BERT, based on Table 2 we choose the best rest-performing SEAL (NCEranking) to compare with baselines: SEAL-margin and cross-entropy. The NYT subset we used has almost 17x larger training data than the largest feature-based data and SEAL framework outperforms cross-entropy fine-tuning. Overall, the Table 3 shows that SEAL, especially SEAL- NCEranking, is effective on pre-trained models that utilizes large text datasets as well. On average, SEAL SEAL-NCEranking gains 0.87, 0.68 Fl points over cross entropy and SEAL-margin, respectively, on text datasets as shown in Table
FURTHER ANALYSIS
SEAL without cross entropy: Figure 2a shows the performance of SEAL when the cross-entropy loss (CE) is completely removed from equation] We first observe that surprisingly that SEAL itself without resorting to CE learns a model that sometimes outperforms model trained with CE (cal500 and spo funn. However, we also observe that SEAL without CE is unstable as performance varies lot across different datasets. This tells us that SEAL requires a signal that can bring the model (.).) into reasonable region first in order to demonstrate stable performance shown in Table 2 Another
8

---

Under review as a conference paper at ICLR 2022
cross ectossy B margin
regression
8
NCEranking
regression-s
NCEraking
035
avg.
ca1500
1pu
eurlex
genbuse
bibiux
delicious
ca1500
spo
eurios geabase biblex delicious increase
(a)
(b)
Figure 2: (a) The performance of SEAL variants without cross-entropy loss term on Lp, i.e. when A1 1, X2 in equation compared the model that only learns with cross-entropy. (b) Change in test Fl when moving from SEAL-static to SEAL-dynamic.
interesting observation is that SEAI-NCEranking model when is often the weakest without CE can give the largest improvement when combined with CE together in SEAL as shown Table 2 In short, we find CE to bring a stable synergy with energy network loss within SEAL: surpassing each individual's performance significantly when combined together. While there could be multiple techniques to stabilize learning of SEAL besides adding CE loss, such pre-training techniques, we leave those interesting studies as future work.
Effect of dynamic loss computation: We now compare the effect of dyanmic loss learning by comparing SEAL to SEAL-static in Figure 2b We observe that pretrained energy networks with SEAL-static also results in competitve results, often better performing than the simple cross-entropy loss. However, we oberve that in average, the dynamic loss learnig of SEAL outperforms SEAL-static for Lg Eegressioon and - -CErnkinn
Effeet of applying ranking loss directly on Fs With the large improvement that Lg- -NCCrnnnng and L-g-ranking has, we examine whether the gain is actually coming from SEAL framework or simply from the power of ranking loss itself which can be applied to feedforward network Fg. directly. We conduct this ablation study over relatively small datasets: genbase, cal500 and delicious. While we observed near +10 Fl point in genbase, we decrease in performance in cal500 (-1.0 FI) and delicious (-2.3 F1) compared to plain cross-entropy model. With these experiments, we notice two trends. First, again observe that genbase is outlier where capturing strueture increase the performance although not as much as SEAL framework helped. Two, we confirm that the ranking loss without energy network is very effective in training feedforward model Fg. This again confirms that the energy network Fe plays a major role to train Fg. in the SEAL framework.
greatly
CONCLUSION
We propose SEAL: a framework that can adopt structured energy network as a trainable loss function for training feedforward network. Through examining different energy losses that trains energy network, we show that SEAL is general framework which is effective over different loss functions and different architectures, such as MLP and BERT-base adapter. Through extensive result on feature-based and 3 text-based datasets, we show that SEAL brings synergy between isolated energy newtork and cross entropy function: performing better and in more stable manner when combined together. Lastly, through ablation study, we examine the benefit of dyanmic loss learing which in average brings gain over static version SEAL, i.c. SEAL-static. This research opens up doors for various future work, including but not limited to, application of learned energy loss in unsupervised dataset and exploring different architectures of energy network. the long horizon, the authors are interested in whether general pretrained scoring neural networks, such as BLEURT (Sellam et al.) 2020) and BERT-score (Zhang et al.) 2019), could serve as a loss function when it can provide a backpropagatable gradients to the output space.

---

Under review as a conference paper at ICLR 2022
REFERENCES
David Belanger and Andrew Mccallum. Structured Prediction Energy Networks. Technical report, 2016.
David Belanger and Andrew McCallum. Structured prediction energy networks. In International Conference Machine Learning, pp. 983-992. PMLR, 2016.
David Belanger, Bishan Yang, and Andrew Mccallum. End-to-End Learning for Structured Prediction Energy Networks. Technical report, 2017.
Sumanta Bhattacharyya, Pedram Rooshenas, Subhajit Naskar, Simeng Sun, Mohit Iyyer, and Andrew McCallum. Energy-based reranking: Improving neural machine translation using energy-based models. In Association for Computational Linguistics, 2021.
Lukas Biewald. Experiment tracking with weights and biases, 2020. URL htttp::///wwtt wandb com/ Software available from wandb.com.
Liang-Chieh Chen, Alexander Schwing, Alan Yuille, and Raquel Urtasun. Learning deep structured models. In International Conference on Machine Learning, pp. 1785-1794, PMLR, 2015.
Yuntian Deng, Anton Bakhtin, Myle Out, Arthur Szlam, and Marc' Aurelio Ranzato. Residual energy- based models for text generation. In International Conference on Learning Representations, 2020.
URL
Jacob Devlin, Ming Wei Chang, Kenton Lee, and Kristina Toutanova. BERT: Pre-training of deep
bidirectional transformers for language understanding. NAACL HLT 2019 2019 Conference of the North American Chupter of the Association for Computational Linguistics: Human Language Technologies- Proceedings of the Conference, 1:4171-41 2019. URL http: http::///axxvv.orr
abs 11111111110 04805
Nadia Ghamrawi and Andrew McCallum. Collective multi-label classification. In Proceedings of the 14th ACM international conference on Information and knowledge management, pp. 195-200, 2005.
Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial nets. Advances in neural information processing systems, 27, 2014.
Michael Gutmann and Jun-ichiro Hirayama. Bregman divergence as general framework to estimate unnormalized statistical models. arXiv preprint arXiv: 1202.377. 2012.
Michael Gygli, Mohammad Norouzi, and Anelia Angelova. Deep value networks learn to evaluate and iteratively refine structured outputs. In 34th International Conference Machine Learning, ICML 2017, volume 3, pp. 2160-2170, 2017. ISBN 9781510855144, URL httpp::///ggg /goo///ooo/o/ gl/
80Lufh
Neil Houlsby, Andrei Giurgiu, Stanislaw Jastrzebski, Bruna Morrone, Quentin De Laroussilhe, Andrea Gesmundo, Mona Attariyan, and Sylvain Gelly. Parameter-efficient transfer learning for NLP. In Kamalika Chaudhuri and Ruslan Salakhutdinov (eds.), Proceedings of the 36th International Conference Machine Learning, volume 97 of Proceedings of Machine Learning Research, pp. 2790-2799. PMLR, 09-15 Jun 2019. URL press/ v9 7 abyl 9a. html
mlr
Chen Huang, Shuangfei Zhai, Walter Talbott, Miguel Bautista Martin, Shih- Yu Sun, Carlos Guestrin, and Josh Susskind. Addressing the loss-metric mismatch with adaptive loss alignment. In International Conference on Machine Learning, pp. 2891- PMLR, 2019.
Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint 1:Xiv:1412.6980 2014.
John Lafferty, Andrew McCallum, and Fernando CN Pereira. Conditional random fields: Probabilistic models for segmenting and labeling sequence data. 2001.
10

---

Under review as a conference paper at ICLR 2022
Yann LeCun, Sumit Chopra, Raia Hadsell, M Ranzato, and F Huang. A tutorial energy- based learning. Predicting structured data, 1(0), 2006.
Zhuang Ma and Michael Collins. Noise contrastive estimation and negative sampling for conditional
models: Consistency and statistical efficiency. In Proceedings of the 2018 Conference on Empirical
Methods in Natural Language Processing, pp. 3698-3707, Brussels, Belgium, October-Novemble
2018. Association for Computational Linguistics. doi: 10.18653/v1/D18. 1 URL https 7/aclanthologe www.oyy.rggg//-e/ee
Andreas Christian Müller. Methods for learning structured prediction in semantic segmentation of natural images. 2014.
Martha Palmer, Daniel Gildea, and Nianwen Xue. Semantic role labeling. Synthesis Lectures on Human Language Technologies, 3(1):1-103, 2010.
Jonas Pfeiffer, Andreas Rücklé, Clifton Poth, Aishwarya Kamath, Ivan Vulić, Sebastian Ruder, Kyunghyun Cho, and Iryna Gurevych. Adapterhub: A framework for adapting transformers. In Proceedings of the 2020 Conference on Empirical Methods Natural Language Processing: System Demonstrations, pp. 46-54, 2020.
Amirmohammad Rooshenas, Dongxu Zhang, Gopal Sharma, and Andrew McCallum. Search- guided, lightly-supervised training of structured prediction energy networks. Advances in Neural Information Processing Systems, 32:13522-13532, 2019.
Alexander Schwing and Raquel Urtasun. Fully connected deep structured networks. arXiv preprint arXiv:1503.02351, 2015.
Thibault Sellam, Dipanjan Das, and Ankur P Parikh. Bleurt: Learning robust metrics for text generation. In Proceedings ACL, 2020. URL https:///rxiv..t http://awwwv...gaa////0...00
Ben Taskar, Carlos Guestrin, and Daphne Koller. Max-Margin Markov Networks. Advances in neural information processing systems 16 (2004): 25, 2004.
Ioannis Tsochantaridis, Thomas Hofmann, Thorsten Joachims, and Yasemin Altun. Support vector machine learning for interdependent and structured output spaces. In Proceedings of the twenty-first international conference on Machine learning, pp. 104, 2004.
Lifu Tu and Kevin Gimpel. Learning approximate inference networks for structured prediction, 2018 ISSN 23318422.
Lifu Tu and Kevin Gimpel. Benchmarking approximate inference methods for neural structured pre- diction. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume (Long and Short Papers), pp. 3313-3324, Minneapolis, Minnesota, June 2019. Association for Computational Linguistics. doi:
Lifu Tu, Richard Yuanzhe Pang, and Kevin Gimpel. Improving joint training of inference networks and structured prediction energy networks. In SPNLP, 2020.
Guoyin Wang, Chunyuan Li, Wenlin Wang, Yizhe Zhang. Dinghan Shen, Xinyuan Zhang, Ricardo Henao, and Lawrence Carin. Joint embedding of words and labels for text classification. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume I: Long Papers), pp. 2321 2331, Melbourne, Australia, July 2018. Association for Computational Linguistics. doi: 10.18653/v1/Pt 1216. URL http://caaanttt http://aaacaaaatoooooro/////////
Lijun WU, Fei Tian, Yingce Xia, Yang Fan, Tao Qin, Lai Jian-Huang, and Tie-Yan Yan Liu. Learning to Teach with Dynamic Loss Functions. In Advances in Neural Information Processing Systems, vol- ume 31. Curran Associates, Inc., 2018 URL / nips. cttp.ccccpppcpp////21111

---

Under review as a conference paper at ICLR 2022
Lin Xiao, Xin Huang, Boli Chen, and Liping Jing. Label-specifie document representation for multi-label text classification. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLPP-JCNIL)), pp. 466-475, Hong Kong, China, November 2019. Association for Computational Linguistics. doi: 10. 11653//11D11910444 URL htttss/t:://aat antho logy
org/D19-1044
Tianyi Zhang, Varsha Kishore, Felix Wu, Kilian Q Weinberger, and Yoav Artzi. Bertscore: Evaluating text generation with bert. arXiv preprint arXiv: /904.09675 2019.
Ximing Zhang, Qian-Wen Zhang, Zhao Yan, Ruifang Liu, and Yunbo Cao. Enhancing label cor- relation feedback in multi-label text classification via multi-task learning. In Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021. pp. 1190-1200, Online, August 2021. Association for Computational Linguussess. doi: 10. 10.33v1111120 findings--cl. 101. URL
12

---

Under review as a conference paper at ICLR 2022
APPENDIX
A NCE RANKING METHOD WITH y; = -E)
As pointed out by Ma & Collins this this learning with (x,,) model estimate P(y|x xx with
(018 well, If we set s(x,y; - Ee y), regard
(x y) log (y).. As pointed out earlier, this leads the
exp (- y)) Eyey exp (- E v))
Plugging in Fig' as y) + log (y once again, we can see that P(Y[X) is estimated by
(y exp Ex)(
Evey (y) Eo (x)
B GRADIENT TO THE LABEL REPRESENTATIONS
As described in section the energy component of loss LF, for the feedforward network, the sum of following two parts:
Ellccal (x,y) Σ T.b.TTEU
and
Lighbbll (x,y) =To
(12)
The gradient of Felobal
w.r.t is given by the following vector of length
b Felobbll
= MM Diag(o'((MM)))
(13)
oy
While the gradient of cross entropy ((v.8) w.r.t y given by the following vector of length
01
1 1-y
y 2 y
(1 (1-y")
&
(14)
dy
where & element-wise product.
Note the dependency of the gradient for particular component ys of other component in the expressions above. The gradient cross entropy ( at only depends on its label dimension yi and
Og..
V In contrast, in the case of the energy function loss (SEAL), the gradient ay.
öLe depends all L
dimensions of y due to the product term My in @gdas
By:
C COMPARISON OF SEAL AND SEPARATE ENERGY NETWORK
In the following figures, we compare the Fl scores of SEAL framework with Le-ngression (Figure 3 NCE-annknn (Figure and Lg-mariin (Figure 5) on validation set with its energy-only counterpart. We further report validation Fl scores for the output of GBI starting from the samples drawn from and randomly. We show this comparison to evaluate the performance the energy network on points around ý and random points.
13

---

Under review as a conference paper at ICLR 2022
0.2
000
bibtex
(b) cal500
D725
0.10
0.0
20
1003
2000
3300
756
33.00
12500
1900
1756
Stop
(c) genbase
(d) delicious
0.30
0.2-
0.88
0.30
0.28
0.:0
1003
20:0
30.0
5001
2000
1000
D00
1900
210
20op
(e) expr fun
(f) spo fun
Figure 3: Validation Fl performance of SEAL framework with Lg- reg--ressooon (Fa) compared energy- only (DVN). Further, we compare validation F1 performance of GBI starting from samples drawn from feedforward network's ' output (f4+GGi +GB) and samples drawn randomly (random + GBI).
14

---

Under review as a conference paper at ICLR 2022
4001
033
130
3353
422
180
130
210
36
Stap
bibtex
(b) ca1500
4GB0
1.20
0.30
0.00
010
1291
5.00
10000
13:01
2000
Stop
(c) genbase
(d) delicious
038
020
066
1003
2003
196
126
33.0
390
21p
280
(c) expr__eee fun
(f) fun
Figure Validation F1 performance of SEAL framework with Lg- -CEankang (Fa) compared to energy-only (NCE). Further, we compare validation Fl performance of GBI starting from samples drawn from feedforward network's output (F++GBI) and samples drawn randomly (random GBI).
15

---

Under review as a conference paper at ICLR 2022
SPGN
2% GDI
622
6.0
0.2
035
bibtex
(b) cal500
SPEEN
11
0.2
4000
203
1306
1800
13000
2000
15:00
(c) genbase
(d) delicious
Pg
0.25
0.15
0.18
3/5
-80
1200
000
2003
1200
210
20ep
(e) expr__fre fun
(f) spo_ppofo fun
Figure 5: Validation Fl performance of SEAL framework with Lg- mariin F compared to energy- only (SPEN). Further, we compare validation Fl performance of GBI starting from samples drawn from feedforward network's 's output (F4 (f++GBI) and samples drawn randomly (random + GBI).
D HYPERPARAMETERS
The following tables tabulate the parammerrsmeters for SEAL framework for each dataset. We further fix the number of layers, layer- layer-dimensions and dropout in feed-forward network based on the configuration of the best model trained using cross- cross-entropy. We fix the X2 = in equation to reduce degree of freedom in hyperparameter search.
16

---

Under review as a conference paper at ICLR 2022
method
X
num samples
1r
1/r
margin
0.001
n/a
0.0002
0.001
regression
0.002
n/a
0.0003
0.001
regression-s
0.03
20
0.0004
0.001
NCEranking
0.002
100
0.0005
0.001
ranking
0.002
100
0.000015
0.001
Table genbase
method
A1
num samples
Oir
Pir
margin
0.001
n/a
0.0005
0,008
regression
1.4
n/a
0.006
0.007
regression-s
5
0.008
0,006
NCEranking
0.4
40
0.007
0.005
ranking
80
0.003
0.0001
Table 5: cal500
method
num samples
Oir
1r
margin
0.0002
n/a
0.002
0.004
regression
8
n/a
0.0001
0.00015
regression-s
0.0001
0.003
NCEranking
0.9
20
0.002
0.001
ranking
60
0.0008
0.0001
Table 6: delicious
method
A1
num samples
1₁
1tr
margin
0.001
n/a
0.0003
0.002
regression
0.002
n/a
0.0006
0.004
regression-s
0.01
20
0.0004
0,007
NCEranking
0.5
40
0,001
0.002
ranking
0.3
40
0.0003
0.002
Table 7: eurlex
method
num samples
Oi
dir
margin
0.01
n/a
0.00003
0.001
regression
0.02
n/a
0.0003
0.0006
regression-s
30
0.007
0.002
NCEranking
80
0.00005
0,0002
ranking
40
0.008
0.0005
Table 8: fun
17

---

Under review as a conference paper at ICLR 2022
method
X
num samples
Gr
margin
0.08
n/a
0.0001
0.002
regression
n/a
0.001
0.00015
regression-s
0.02
10
0.0002
0.0009
NCEranking
0.7
60
0.008
0.0005
ranking
1.5
40
0.004
0,002
Table 9: spo fun
method
X
num samples
2.
11r
margin
0.0002
n/a
0.00005
0.001
regression
n/a
0,005
0.003
regression-s
10
0.005
0.001
NCEranking
20
0.0004
0.001
ranking
40
0.00005
0.001
Table 10: bibtex
ANALYSIS ON PARAMETER SIZE AND SPEED OF INFERENCE AND TRAINING
Here, we present the analysis on number of parameters (Table inference speed (Table and training speed (Table 13) that different methods utilize dataset.
In summary, SEAL requires approximately twice number of parameters compared to entropy loss on training time, however it requires the same amount of parameter on inference time. Similarly. train time (sec) per epoch of SEAL slower than that cross-entropy and energy network as SEAL requires multiple backpropagation steps for both energy network and feedforward network. Nonetheless, again, at inference time, the speed of CE and SEAL is the same and is much faster (2x-7x) than the inference of energy network.
The analysis we present below are on the seven feature-based dataset, however the trend similar for text-based dataset well: twice parameter size training time but equal parameter and runtime in inference time with CE method. The only difference the text-based MLC is, due to running on a much larger BERT-baed models, that we only run one step for each update of feedforward network and energy network. Thus, unlike Table 13 train time of SEAL is approximately twice that of CE.
Parameter size
Dataset Methods
Energy network
SEAL (Train-time)
SEAL (Inference-time)
CE
(SPEN, DVN, NCE)
EXPR_FUN
333000
533800
866800
333000
SPO_FUN
447000
597600
1044600
447000
Bibtex
958800
991000
1949800
958800
Cal500
1158700
1123500
2282200
1158700
Delicious
754000
95 951000
1705000
754000
Genbase
485600
491400
977000
485600
Eurlex-ev
4747500
5546500
10294000
4747500
Table The number of parameters required during train time for SEAL is approximately double
the size of feedforward (CE column) while energy network and feedforward sizes are comparable. However, in the inference time, SEAL has an equal amount of parameters to the CE column as only feedforward network is utilized during inference time.
18

---

Under review as a conference paper at ICLR 2022
Speed ratio
Inference time (sec)
Inference speed (examples/sec)
Dataset
(CE and SEAL/
Energy network
CE and SEAL
Energy network
CE and SEAL
EnergyNetworr)
EXPR_FUN
1.33
0.22
638
3801
5.96
SPO FUN
1.18
0.16
709
5231
7.38
Bibtex
3.6
1.78
414
840
2.03
Cal500
0.24
0.10
438
1080
2.47
Delicious
5.35
1.39
599
2307
3.85
Genbase
0.23
0.13
574
1005
1.75
Eurlex-ev
24
12.22
162
317
1.96
Table 12: We simply average inference time for CE and SEAL variants as they are very similar. Likewise, average the inference time of different energy networks. Here, inference time (sec) is recorded for the whole validation set. We also present speed per sec (example/sec) and speed ratio. The last column shows that CE and SEAL methods are 2x-7x faster in inference time than the energy networks.
Train time (sec/epoch)
Methods Datasets
EXPRFUN
SPOLFUN
Bibtex
Cal500
Delicious
Genbase
Eurlex-ev
CE
6.29
5.46
22.67
1.00
33.01
2.12
114.84
SPEN
10.99
11.01
28.04
2.53
37.31
3.57
136. 10
DVN
10.95
10.02
32.12
1.87
55.24
2.79
129.94
NCE
3.71
3.83
13.97
2.82
22.03
3.68
89.33
SEAL-margin
27.96
35.24
212.94
5.78
44.60
8.73
260.10
SEAL-regression
46.32
56.97
27.78
8.07
143.90
13.91
352.63
SEAL-regression-s
42.87
77.92
179.97
8.48
45.33
10.79
221.33
SHAL-NCHranking
72.43
40.34
131.73
16.58
158.97
17.77
431.92
SEAL-Ranking
41.24
26.83
218.37
7.04
317.63
10.74
408.79
Table 13:
The training time per epoch presented per dataset and per loss function used. Due to different gpu types and node status, there are some outliers. Furthermore, as SEAL runs multiple number of backpropagation steps for energy network and feedforward network in the alternating optimization, direct comparison of training time is not really available. However, the general trend in terms of training time per epoch CE Energy Networks SEAL.
F DATA-SPECIFIC ANALYSIS
In this section, we analyze why certain datasets behaves differently.
F.1 GENBASE
Here, analyze the factors behind SEAL achieving almost perfect Fl score on the genbase dataset. seems genbase stands out with its small label size (27 which is the smallest among Table T and with very clear pattern in the label space. Upon analysis, out of 27 labels, we found that 6 labels only occur as singleton (by itself), 10 labels only occur as non-singlcton, and 7 labels do not participate at all. This leaves only labels occurring by themselves as well as with others. Not only that, only 7% (35/500) of training instances have more than one active label. In this peculiar setting, we conjecture feedforward networks with cross-entropy (CE) loss can easily learn singletons, and energy networks would easily learn co-occurrence, but learning both might be confusing for these two models at the opposite end of the spectrum. We believe the synergy of CE and a loss from the energy network enables capturing the best of both worlds and achieving near cear-perfect score that none of the approaches by itself achieves. As can seen in Table neither cross-ents cross-entropy nor energy networks are nearly as strong SEAL methods.
19

---

Under review as a conference paper at ICLR 2022
F.2 DELICIOUS AND CAL500
In Table 2 it seems SEAL-NCEranking and SEAL-Ranking are particularly well performing on cal500 and delicious. It turns out that cal500 and delicious have very high diversity of and 0.981 (Reference: MLC data repo). Diversity of means that each data point holds unique label set. seems that the exposure to multiple samples and evaluations of their relative scores in NCEranking loss function can be helpful in a high-diversity setting.
20


Reviews:
[{"review_id": "sH9r3VT4csX", "reviewer": "Reviewer_CpcQ", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The motivation of this paper is reasonable as the author pointed out in the introduction section: \"The key idea is to provide the feed-forward network access to rich relationships in the output space through a learned loss function, and do this in a dynamic manner by updating the loss based on the current outputs of the feed-forward network\".\n\nGeneral Significance: The main significance of the model is an adoption of STEN as a trainable loss function for learning an MLP, which could achieve better model quality with this combination. Regarding training stability and computation efficiency, from the discussion of the paper, it seems that back-propagation of gradient can be used in alternating fashion to optimizes the energy network (STEN) and the MLP. This seems easier and more stable (from the attached implementation based on AllenNLP library - pytorch) than training an energy network alone. The impact of this paper is limited because the experimental dataset are mostly on a label cardinality under 1000. It would be more interesting if the author could use a larger dataset (regarding number of train/val/test examples) and a larger label cardinality. Could take a look at Extreme datasets (http://manikvarma.org/downloads/XC/XMLRepository.html).\n\nNovelty: This method is largely based on the STEN work (reused most building component of STEN, e.g., the energy network is built the same way), but it adds interesting extension which could open opportunities for the application of energy-based learning. Regarding related work, could you also discuss how \"label embedding\" (https://arxiv.org/abs/1503.08677) and \"joint learning of label embedding and the feedforward network\" (e.g., https://arxiv.org/pdf/1805.04174.pdf) relate to your proposed work here.\n\nTechnical Quality: The overall technical quality is on the fence. It would be nice if the author could provide some more explanations on the experiment results:\n1. Why the ranking-based methods (in table 2) are significantly better than others in delicious and cal500 dataset?\n2. Why the SEAL-based methods  lead by a large margin on genbase dataset? (is it something related to the nature that this is a very small dataset?)\n3. Why DVN performs so well on cal500? (seems to be another small dataset).\n4. What if you replace the adapter-based BERT encoder with a full finetuning on BERT?\n5. Mentioned above, provide benchmark results on larger datasets.\n\nClarity: The paper is well-structured, with clear relationships between each section. The writing is good. I especially like Figure 1, which is very illustrative thats explains a good amount of details with a simple graph. \n\nMinor typos (issues):\n* Page 5. Section 3 Learning with dynamic loss. should probably say \"Lastly, these models were designed for multi-class classification\" (remove \"do\").\n", "summary_of_the_paper": "This paper proposes a framework to use Structured Prediction Energy Network (SPEN, a prior work) as a loss function. Previously SPEN is a deep architecture to learn an energy function of candidate labels, which captures structural dependencies between labels that would lead to intractable graphical models. The author of this paper finds out that it is effective to jointly learn the parameters in SPEN and the parameters in the feed-forward network (MLP). In the training time, the model alternatingly minimizes the parameters in SPEN by fixing the parameters in MLP; and then minimizes the parameters in MLP by fixing the parameters in SPEN. Empirical results in 10 multi-label datasets are given, which shows good performance boost from using the proposed methods. ", "summary_of_the_review": "I am more inclined to marginally accept the paper given its current status.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "jJ617M74xHG", "reviewer": "Reviewer_8Y6b", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The paper proposes an interesting problem -- capturing the higher level dependencies in data utilizing an energy function compared to assuming conditionally independent predictions.\n\nHowever I have several major concerns about the paper.\n\nThe first major concern is that the underlying presentation of the paper is somewhat poor. For example, one of the major claims in the paper are not supported. The authors states such that an energy function is interpretted as a dynamic loss function, but the method section does not really specify what this means.\n\nThe paper further appears to be in a rushed state and there are a large number of different typos -- for example\n\nnoise contarstive estimation -> noise contrastive estimation.  \nhyptothesize -> hypothesize.    \navergae -> average.    \nf1 -> F1.   \n\nIn addition, the empirical comparisons appear to be limited and evaluated only on low dimensional datasets. Can the authors show a practical use case for the proposed approach on an existing large scale benchmark? Or compare utilizing existing datasets from past works?\n\nThe underlying technical novelty of the approach also seems a bit poor -- the main contribution appears to be applying an InfoNCE loss to train an energy function. It is also not clear from the text why this loss is beneficial during training. For example, the authors claim that their new loss prevents adversarial training. I don't follow this as both models are still optimized at the same time -- with one model encourage to have lower energy generations and another to increase the energy of generated samples.  For furthermore, why is it that the proposed loss would leave to a gain of performance?", "summary_of_the_paper": "This paper proposes a method to utilize a SPEN as a trainable loss function for training a neural network. In particular, the authors propose to train the energy function utilizing NCE loss compared to other loss functions used by prior works, and show that such a choice improves performance of the resultant classifier.\n\n", "summary_of_the_review": "Due to poor presentation, a lack of empirical comparison, and no theoretical analysis in the underlying paper, I believe this paper is not in a suitable form for publication right now.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "OZCyXeBDGhw", "reviewer": "Reviewer_vt8f", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The general idea of SEAL is interesting since various energy-network-learning objectives can be plugged into SEAL. Such as various losses of margin-based, regression-based and contrastive-sampling-based. Figure 1 is intuitive for understanding the major updates of the network structures in SEAL. Generally, this paper is well written with good motivations, technical details and rich experiments.\n\n\n\nDetailed questions and comments:\n1.\tIn algorithm 1, “kaiming initialization” stands for? Do you mean Kaiming He’s method? Can you give a citation as well? What will happen if you use other types of initialization methods?\n2.\tIn table 2, any detailed information about parameter sizes, training time and inference time comparisons among the baselines and SEAL variants? Any other types of networks that are not related to energy network to solve the tasks in table 2?  [the appended information from the authors answered this question in detail and thanks for that.]\n", "summary_of_the_paper": "This paper proposes a framework named SEAL that can adopt structed energy network as both a trainable and adaptable loss function for training feedforward networks. 7 feature-based 3 text-based datasets are used for testing SEAL and showed better results and even better with combinations. Rich ablation study was given as well.", "summary_of_the_review": "Strong:\n1.\tThe idea of SEAL is novel and effective to a number of tasks;\n2.\tThis paper is well written and the direction of using of dynamic loss functions is an interesting direction.\n\nWeak:\n1.\tLess aware of the signicient technical aspects of using energy network for the target tasks. Any other baselines of not using energy related networks in table 2? [thanks the authors' responses for the detailed explanation of this point. Now I have a better understanding of the benefits of applying \"energy-based methods\" to the corresponding tasks.]", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "LC4ZDroRMek", "reviewer": "Reviewer_5BUz", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths:\n- Interesting approach for solving structured prediction problems that shows significant improvement in performance (F1 score) on feature-based datasets\n- A thorough experimental analysis, including important ablation study, that demonstrates the strengths of the proposed approach.\n\n\nWeaknesses:\n- The proposed approach is a generalization of [Tu et al., 2020] (that is represented by the margin-based SEAL version). The proposed approach is not restricted to margin-based loss and indeed the new loss functions outperform margin-based loss, however the performance gain is smaller compared to the performance gain over non-SEAL approaches.\n- I have some concerns about the reporting of the experimental results:\n\t1) The description of the experimental results is a bit misleading due to the focus on the performance gain over cross entropy (e.g., \"+50 F1\") in  Section 4 and the abstract. CE is not the state-of-the-art, and the performance gain over the state of the art should be reported (which, as far as I understand from the work, is margin-based SEAL that appears in [Tu et al., 2020]).\n\t2) Results on large text datasets only contain SEAL with NCEranking loss. It is not clear why the other losses (including the one that represents [Tu et al., 2020]) and other existing approaches like SPEN and DVN are not reported. It is hard to evaluate the gain of SEAL over these approaches.\n- The paper only considers one type of structured prediction benchmarks: multi-label classification. It would be interesting to show the performance on additional tasks like POS tagging and NER.\n- Some related literature that is missing: recent works that have used NCE for training energy based models for text-based datasets [1, 2].\n\n\nMinor comments, typos, etc:\n- page 1 abstract line 1 \"this\"\n- page 1 abstract \"compared\" -> \"Compared\"\n- page 2 \"network  using\": looks like double space\n- page 2 \"as efficient at inference as well as a feed-forward ....\": \"as well as\" -> \"as\"\n- page 2 \"Despite these efforts, .... remain relatively inefficient\" and \"are finicky to train\" - need more details or relevant citations\n- page 4 \"(Ma & Collins)\" - missing year\n- page 9 conclusion: \"BLEURTSellam\"\n\n\n[1] Bhattacharyya, S., Rooshenas, A., Naskar, S., Sun, S., Iyyer, M., & McCallum, A. (2020). Energy-based reranking: Improving neural machine translation using energy-based models. arXiv preprint arXiv:2009.13267.\n\n[2] Bakhtin, A., Deng, Y., Gross, S., Ott, M., Marc'Aurelio Ranzato, & Szlam, A. (2021). Residual Energy-Based Models for Text. J. Mach. Learn. Res., 22, 40-1.\n", "summary_of_the_paper": "The paper presents Structured Energy As Loss (SEAL) that uses structured energy networks as a trainable loss for a feedforward network. SEAL is a general framework that supports multiple loss functions, including margin-based loss, regression-based loss, and NCE-based loss, and generalizes the recent approach in [Tu et al., 2020]. The paper presents extensive experiments on multi-label classification on a range of feature-based and text-based datasets. SEAL shows significant improvement over standard cross entropy trained networks, as well as approaches based on Structured Prediction Energy Networks (SPEN).", "summary_of_the_review": "Overall, I think the paper presents an interesting approach that shows significant improvement in multi-label classification. However, the experimental analysis could be strengthen by improving the presentation of the results, and by considering more benchmarks.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper proposes a new method for multi-label classification, which leverages the advantage of the emery-based model. However, one reviewer and the area chair have two serious concerns on the experiments: (1) The proposed method is only evaluated on low dimensional datasets; (2) Some important baselines methods are missing, which makes the comparison not convincing. I suggest the authors to evaluate their methods on more datasets, and add the results from well known multi-label classification method for comparison.
Decision: Reject