Year: 2021

Abstract:
We propose SEAL which utilizes an energy network as a trainable loss function for a simple feedfoward network. Structured prediction energy networks (SPENs) (Belanger & McCallum, 2016; Gygli et al., 2017) have shown that a neural network (i.e. energy network) can learn a reasonable energy function over the candidate structured outputs. We find that rather than using SPENs as a prediction network, using it as a trainable loss function is not only computationally more efficient but also better performing. As the energy function is trainable, we propose SEAL to be dynamic so it can adapt the energy function to focus on the region where feed-forward models will be affected the most at the time point of model update. We find this to be effective in an ablation study comparing SEAL to the static version (§5) where the energy function is fixed after pretraining. We show the relation to previous work on the joint optimization model of energy network and feedforward model (INFNET) as we show that it is equivalent to SEAL using margin-based loss if INFNET relaxes their loss function. Based on the unique architecture of SEAL, we further propose a variant of SEAL that utilizes noise contrastive ranking (NCE) loss that by itself does not perform well as a structured energy network, but embodied in SEAL, it shows the greatest performance among the variants we study. We demonstrate the effectiveness of SEAL on 7 feature-based and 3 text-based multi-label classification datasets. The best version of SEAL that uses NCE ranking method (SEAL-NCE) achieves close to 2.85, 2.23 respective F1 point gain on average over cross-entropy and INFNET for feature-based datasets, excluding one dataset that has an excessively large gain. Lastly, examining whether the proposed framework is effective on a large pre-trained model as well, we observe SEAL-NCE achieving 0.87, 0.68 respective F1 point gain in average over cross-entropy and INFNET with BERT-based adapter model on text datasets.

heading: 1 INTRODUCTION
text: Structured prediction is a popular machine learning task wherein the model learns a mapping function from an input x to a multivariate-structured output y. Popular examples of this include image segmentation (Müller, 2014), extracting parse trees or semantic role labels from a text (Palmer et al., 2010), and multi-label classification (Belanger & McCallum, 2016; Gygli et al., 2017). In structured prediction, the output space Y is often extremely large. For example, in multi-label classification, the size of Y is 2L where model needs to predict output y ∈ {0, 1}L. There are two key aspects that a model for structured prediction needs to balance: statistical efficiency and computational efficiency. Most models for structured prediction can be categorized into two categories: the feed-forward approach, wherein one learns a neural network that models the direct mapping between the input and the structured output, and the structured approach, in which the model explicitly models the interactions (structure) in the output space.
The feed-forward approach learns all dimensions of y jointly in a conditionally independent manner given input x, relying on the representational power of the network to capture dependencies in output structure implicitly. While the feed-forward approach is computationally efficient, since it does not capture relationships in the label space, it lacks statistical efficiency. The traditional structured approaches, on the other hand, model a joint probability distribution P (x,y) that can capture the label relationships. However, due to the intractability of modeling the full joint distribution which
captures every possible interaction of output space, these approaches resort to limiting the interaction terms to local subsets of the output space (Lafferty et al., 2001; Ghamrawi & McCallum, 2005). In order to make inference for structured models more efficient, a recent line of work replaces the joint probability with a structured energy function E(x,y) (LeCun et al., 2006), which can be thought of as an unnormalized probability distribution, allowing the model to learn arbitrary global dependencies in the output space. There have been several works that propose efficient approximate inference procedures for structured energy networks using gradient based inference (Belanger et al., 2017; Gygli et al., 2017; Rooshenas et al., 2019), hence also called ’prediction networks’. These models showed noticeable gains in predictive performance over feed-forward models and graphical models that assume partial structure such as full pairwise potential (Chen et al., 2015; Schwing & Urtasun, 2015). Despite these efforts, the inference for energy-based models still remains relatively inefficient when compared to the feed-forward approach (Tu & Gimpel, 2019). Moreover, in our experience, the models using gradient based inference (GBI) are finicky to train as the training also utilizes GBI, requiring numerous hyperparameters: step size, number of iterations for GBI, and initial point to begin GBI. This raises a question: Can the energy network be used in a way that is as expressive as a full joint probability, as efficient at inference as a feed-forward approach, and also stable and easy to train? We believe that using structured energy networks as a parameterized dynamic loss function for feed-forward networks, instead of a prediction network, can fulfill all these requirements.
In this paper, we propose the Structured Energy As Loss (SEAL) framework that uses a trainable structured energy network (SEN)1 as a loss function guiding the training of a feed-forward network. The key idea is to provide the feed-forward network access to rich relationships in the output space through a learned loss function. We also propose to learn SEN in a dynamic fashion by adjusting the energy function to be confident with the most up-to-date outputs of the feed-forward network. We show that learning the loss function dynamically leads to more efficient, more stable, and better performance. SEAL can be viewed as general-purpose framework where one can plugin various loss functions and architectures to train SEN as well as the feed-forward network. Through experiments (§4), we analyze the effect of applying different energy losses (e.g. margin-based, regression-based) within SEAL framework. We also propose the noise-contrastive ranking loss (NCEranking) for SEN which performs the best within the SEAL framework.
To summarize, we introduce a general framework SEAL that interprets structured energy networks (SEN) as a dynamic loss functions. Through empirical evaluation on the task of multi-label classification, we analyse the impact of various loss functions for updating SEN. Finally, we propose an NCE ranking loss that is uniquely suited for the SEAL framework, and demonstrate its superior performance on 7 feature-based as well as 3 text-based multi-label classification datasets, when compared to simple feed-forward approach and various energy based models.
Algorithm 1: SEAL Algorithm
Require: (x,y): Training Instance Require: FΦ: Feedforward Network Require: sampling: True/False flag Require: optimizerΘ, optimizerΦ Require: T : No. of steps
t← 0 Θ0,Φ0 ← Random initialization while t < T do
if sampling then S ← {y(i), i = 1, . . . ,K|y(i) ∼ FΦt(x)} else S ← {FΦt(x)} . singleton set t← t + 1 Update Θ as Eqn. 3 using ỹ ∈ S and optimizerΘ Update Φ as Eqn. 4 using optimizerΦ
1The term comes from structured prediction energy network, minus ’prediction’, as we do not use it for prediction anymore. For brevity, we use terms ’energy network’ and ’strcutured energy newtork’ interchangeably.
2 STRUCTURED ENERGY NETWORK AS LOSS (SEAL)
This section first describes the proposed SEAL framework. LetX denote the input space, Y = {0, 1}L the output space, and Ỹ = [0, 1]L the continuous relaxation of Y . Then the structured energy EΘ : X ×Ỹ → R is defined to be a parameterized function of the input and the continuous relaxation of the output. The feedforward network FΦ : X → Ỹ is defined to be a neural network that maps an input to the continuous relaxation of the output space. We denote j-th training instance pair as (x(j),y(j)) ∈ D and yi ∈ {0, 1}, ỹi ∈ [0, 1] to denote i-th label dimension where i ∈ {1, . . . , L}. As shown in Figure 1, SEAL consists two loss functions: the energy loss LE that trains structured energy network (Θ), and feedforward loss LF that guides the training of the feedforward network (Φ). We first discuss how structured energy network implicitly affects training of the feedforward network by defining LF (Φ). Given a training instance (x,y), the feedforward loss is defined as:
LF (Φ) = λ1EΘ(x, FΦ(x))− λ2 L∑
j=1
[yi logFΦ(x)i + (1− yi) log (1− FΦ(x)i)] . (1)
Here, the first term involving the energy captures interaction across label space whereas binary cross-entropy considers the predictions of each label independently (More details in Appendix B).
Since the training of feedforward network depends on the quality of the parameterized energy network, it is critical to find the parameters Θ that produce the best loss surface for training the feedforward network. This can be done in two ways: by first training the energy separately, or by training the energy and the feedforward network simultaneously. SEAL uses the latter, which is shown to perform better (see §5) than the former. We denote the former method as SEAL-static for distinction .
In SEAL-static, we first estimate the energy network parameter Θ̂ over training data and optimize Φ by plugging fixed Θ̂ into LF . Given (x,y) ∈ D, Φ is trained as equation 2.
min Φ
1 |D| ∑ D LF (Φ) s.t. Θ̂ = arg min Θ 1 |D| ∑ D LE(x,y, ỹ; Θ) (2)
We finally present SEALwhich learns Θ̂t dynamically at step t. To do so, we alternate the optimization steps of Θ and Φ. Given a training instance (x,y) ∈ Bt, a complete training step is given as:
Θt ← Θt−1 −∇Θ 1 |Bt| ∑ Bt LE ( x,y, FΦt−1(x); Θ ) (3)
Φt ← Φt−1 −∇Φ 1 |Bt| ∑ Bt LF (Φ) (4)
Note that, from (2) to (3), arbitrary ỹ got replaced with FΦt−1(x) in estimating Θt so that estimation of energy surface depends on model output FΦt−1(x). Further, the update step for feedforward network Φ relies on Θt rather than static Θ̂. In both SEAL and SEAL-static, test-time predictions are performed solely using the feedfoward network as ŷi = 1(FΦ(x)i ≥ 0.5). The motivation for proposing dynamic loss function in SEAL is as following. It would be useful to learn perfect energy surface, if possible, by estimating Θ̂ that is static. However, doing so is challenging as we are estimating energy surface on the joint space of x, ỹ, a high-dimensional continuous space with limited data and resource. Instead, we hypothesize that it is more important to concentrate resource in depicting accurate energy surface around the current input, output pair (x, FΦ(x)). In SEAL, utilizing the fact that we have access to trainable loss function, we dynamically adapt energy surface so that the region of interest for training feedforward network is well represented.
We now discuss various energy loss (margin-based, regression-based, and contrastive-sampling-based) that can be plugged into LE in SEAL. For brevity, we use LE(Θ) and LE(x,y, ỹ; Θ) interchangeably where again (x,y) denotes labeled data and ỹ denotes probability vector. We defer the description of specific energy network structure to the experiment section (§4) as SEAL framework can work with arbitrary network structures.
Margin-based (LE−margin) SPEN (Belanger & McCallum, 2016) learned structured energy network with SSVM loss (Taskar et al., 2004; Tsochantaridis et al., 2004) so that energy functions learn to have sufficient energy difference, larger than the margin ∆ (ỹ,y), between arbitrary output ỹ and true output y. To examine the effect of the margin-based loss in SEAL, we follow SPEN and utilize the SSVM loss as
LE−margin = ∑ x,y max ỹ [∆ (ỹ,y)− EΘ (x, ỹ) + EΘ (x,y)]+ (5)
Regression-based (LE−regression) Deep Value Network (DVN) (Gygli et al., 2017) attempts to learn an energy network which directly outputs a score that is similar to the metric s(ỹ,y) of interest, such as F1 score, that compares arbitrary ỹ with true output y. Following DVN, by making score s(·)2 and −E(·) to be between [0, 1], we express regression loss as cross entropy
LE−regression = −s(ỹ,y) log−EΘ(x, ỹ)− (1− s(ỹ,y)) log (1 + EΘ(x, ỹ)) . (6)
Noise-contrastive ranking (LE−NCEranking and LE−ranking) In SEAL, we are interested in capturing the output region that feedforward network has a high probability. We ask, rather than just taking a single point ỹ = FΦ(x) from a feedforward network, whether sampling many discrete binary vectors from ỹ and taking all those samples into consideration for learning EΘ could lead in better estimation of energy surface. Motivated from noise contrastive estimation (NCE) (Ma & Collins, 2018), we ask whether energy network EΘ trained to contrast the true output y from groups of K negative samples drawn from feedforward output FΦ(x) can induce a good loss for teaching Φ. The intuition is as the ỹ = FΦ(x) becomes better and better, the loss surface teaching feedforward network should become more fine grained as ỹ and samples from ỹ will be already close to the true y.
Before we discuss LE−NCEranking of our choice, we first review original form of NCE ranking loss from Ma & Collins (2018). For K samples y(k) ∼ PN , k = 1, . . . ,K drawn from noise distribution PN and rewriting y as y(0) without loss of generality, NCE ranking loss is defined as
log exp
( s(x,y(0); Θ) )∑K k=1 exp ( s(x,y(k); Θ)
) , where s(x,y; Θ) = −EΘ(x,y)− logPN(y). (7) Minimization of this loss makes P (y|x; Θ) = exp(−EΘ(x,y))∑
y∈Y exp(−EΘ(x,y)) as an unbiased estimator of true
distribution P (Y |X). Thus, given perfect energy E(·), minimizing −EΘ(x,y) with respect to y in LF would be equivalent to maximizing the estimate of − logP (y|x). From the presented NCE ranking method, we propose to use P (y|x; Φt) = ∏ i P (yi|x; Φt) in place of PN (y). The novelty of this proposal is that we view the output of feedforward network that we train as a noise distribution, and that the energy model that contrasts noise distribution teaches the noise distribution as well through our SEAL framework (equation 3). In short, we get LE−NCEranking when we plug in s(x,y; Θ) = −EΘ(x,y)− logPΦ(y|x; Φt) into equation 7 , i.e .
LE−NCEranking = log exp
( s(x,y(0); Θ) )∑K k=1 exp ( s(x,y(k); Θ)
) , s(x,y; Θ) = −EΘ(x,y)− logPΦ(y|x; Φt). (8) This achieves two benefits that we have in mind in using NCE. First benefit is that it captures the region of interest by sampling from the output distribution of Φt. Second benefit is that it brings more efficient NCE method. NCE method is known to work best when the noise distribution is close to the data distribution (Gutmann & Hirayama, 2012) but not exactly the same. In our SEAL learning procedure, we hypothesize that P (y|x; Φt) becomes closer to the data distribution as training steps proceed.
Lastly, motivated that NCE can estimate the true distribution, we propose yet another loss function that can estimate the difference between the true probability P (y|x) and feedforward network probability. We show in appendix A, if we use plain ranking loss that sets s(x,y; Θ) = −EΘ(x,y) in equation 7, unbiased estimator of P (y|x) now becomes PN (y) exp(−EΘ(x,y))∑
y∈Y PN (y) exp(−EΘ(x,y)) (9). This
2In this paper, we adopt soft F1 score s(ỹ,y) from Gygli et al. (2017) that is defined on the continuous ỹ ∈ [0, 1]L.
means if we take a derivative of EΘwith respect to y, then we are taking derivative with respect to − logP (y|x) + logPN (y). With the trick of replacing the PN (y) with P (y|x; Φt) in SEAL, we hypothesize that loss captured by energy network can focus only on the difference between true and feedforward probability which may end up being more informative surface. In short, we get plain ranking loss LE−ranking by plugging s(x,y; Θ) = −EΘ(x,y) into equation 7 , resulting in
LE−ranking = log exp
( −E(x,y(0); Θ) )∑K k=1 exp ( −E(x,y(k); Θ)
) . (10)

heading: 3 RELATED WORK
text: Learning with dynamic loss: Wu et al. (2018) and Huang et al. (2019) attempt to learn a dynamic loss defined by neural network that is tailored to the task-specific metric m, such as BLEU or 0-1 accuracy. Both approaches try to to learn a loss function that can directly increase model’s metric score on the validation set. Since m is usually non-differentiable, Huang et al. (2019) view this problem as a reinforcement learning problem and attempts to learn a loss function that maximizes the expected reward: higher m score in validation set. On the other hand, Wu et al. (2018) relax the m to be differentiable and view the feed-forward model update and loss-model update as a connected chain of computation graph, so that loss function parameter (Θ) gets updated in relation to feed-forward model (Φ) improving the relaxed metric score.
There are three major differences between SEAL and the previous work. First, energy surface of SEAL is not limited to a specific parametric form, σ(yT Θ logFΦ(x))3, as previous papers do. Instead, SEAL simply utilizes a neural network to express the loss function as −EΘ(·) which allows it to represent a much larger function class. Second, previous works require manual design of state vector. For example, the state vector used by Wu et al. (2018) consists of iteration number, validation accuracy, and current loss-function parameter Θ. In contrast, SEAL only requires the input and output pair (x, FΦ(x)) of the feed-forward network. Lastly, since these models were designed for multi-class classification, it is non-trivial to extend them to the multi-label classification or to the general structured prediction task wherein large number of labels interact.
Structured Prediction Energy Networks: Structured prediction energy networks (Belanger & McCallum, 2016) and its variants (Gygli et al., 2017; Rooshenas et al., 2019) (referred to as SPENs from here on) learn an energy network EΘ : X × Ỹ → R and predict the output using gradient-based inference (GBI) with the objective y = arg miny E(x,y). To train energy network, SPENs apply GBI adversarially in order to find x, y pair that violates the minimality of the energy the most. We found this training procedure to be a bit unstable as its success depends on the initial point of GBI. We discuss the loss function for SPEN (Belanger & McCallum, 2016) and DVN (Gygli et al., 2017) in §2 and use these models as baselines in this work.
Jointly learning Θ and Φ: The Inference network (INFNET) proposed in Tu & Gimpel (2018) is the first work that tried to learn structured energy network and feedforward network jointly. As the name suggests, INFNET proposes a model that mimics the behaviour of SPEN by replacing the expensive gradient-based inference with feedforward networks. However, just like SPENs, INFNET also resorts to finding adversarial point that has the maximum energy violation, resulting in an adversarial framework similar to GAN (Goodfellow et al., 2014), where INFNET maximizes, and energy network minimizes the same margin based loss given in equation 5. INFNET has to serve as the adversarial sampler for training energy network. While the end goal of INFNET is an efficient inference for finding output with lowest energy, having to serve two roles, INFNET suffers from train-, test-time objective mismatch and Tu et al. (2020) resolves this by augmenting INFNET structure; Two separate feed-forward networks are used, one for prediction and the other as the adversarial sampler. However, INFNET’s reliance on margin-based losses (marign-rescaled higne and perceptron) still leaves the approach to be adversarial, i.e. minΘ maxΦ LE(x,y, FΦ(x); Θ).
In contrast, SEAL has completely independent loss functions LE and LF for updating the energy and feed-forward network. On feed-forward (Φ) update, SEAL simply update the model in the direction
3The papers (Wu et al., 2018; Huang et al., 2019) use Φ instead of Θ. We change it to Θ to put notations in consistent manner with this paper.
that reduces energy defined by structured energy network; Energy network is only used as a tool for evaluation of loss and SEAL framework is not interested in mimicking the test-time behaviour of SPEN as INFNET tries to. Despite the difference in the end goal, mechanically speaking, INFNET is a special case of SEAL (i.e. SEAL with LE−margin) if the margin is removed from the SSVM loss for the feedforward model udpate. In fact, Tu et al. (2020) reports that this special case leads to more efficient and stable learning of feedforward network in their ablation study. We use this special case of INFNET to represent SEAL with LE−marginin the main experiment.
Learning label embeddings for multi-label classification Utilizing label representation has been a popular approach for multi-label classification (Wang et al., 2018; Xiao et al., 2019; Zhang et al., 2021). These works embed labels as vectors, allowing for richer representations, and then utilizing cross-entropy loss which models individual label probability independently. While the energy network in SEAL represents an output probability, its main purpose is to allow the gradient for each label to depend on other label probabilities, a feature which is absent from these prior works, for which label interactions must be latently encoded in the embedding space.
Eenrgy-based models used for reranking Recently, energy-based models (EBM) have been applied to text generation problems to first generate and rerank the generated sentences by EBM score (Deng et al., 2020; Bhattacharyya et al., 2021). The residual energy modeling of Deng et al. (2020) shares some similarity to the equation 9,apart from the minor difference that they utilize frozen language model for PN whereas SEAL has PN that constantly learns from the energy models. The major difference is that these previous works have simply used the scores of the energy network, whereas SEAL views the energy network as a loss function, leveraging the gradient from the energy network to further train another neural network.

heading: 4 EXPERIMENTS
text: In our experiments, we use 10 multi-label classification datasets (shown in Table 1) covering varied label space, size, as well as input characteristics. The first 7 are small and medium sized feature based datasets, while the last three Blurb Genre Collection (BGC), RCV (Reuters 1), and New York Times (NYT) are large datasets with raw text as the input.4
Energy Network: We use the same structure for the energy network EΘ(x, y) as described in Belanger & Mccallum (2016), wherein the energy is the sum of localElocalΘ (x, y) and globalE global Θ (x, y) energies defined as:
ElocalΘ (x,y) = L∑ i=1 yib > i TE(x) , E global Θ (x,y) = v >σ(My), (11)
where bi,v,M, TE contain learnable parameters, σ(z) = log(1 + ez) is the softplus activation function, and TE is a feature network for structured energy network. Note that due to the presence of
4Due to their enormous size, we use a subset of NYT training set, which is still the largest dataset in our experiments.
EglobalΘ term in LF , the gradient for feedforward network, ∂LF yi
, can capture the dependency of yi to all L dimensions of y. The expression for this is provided in Appendix B.
Feature Network: We use the same feature network structure for both the energy and feedforward network denoted as TE , TF respectively. For the the feature-based datasets, the feature network T : Rd → Rh consists of multi-layer perceptron with with softplus nonlinearities. For the raw text based datasets, we use pre-trained BERT (Devlin et al., 2019) with adapter (Houlsby et al., 2019; Pfeiffer et al., 2020) as the feature network.
Feedforward Network: Given a feature network TF that generates features in Rh, the feedforward network FΦ(x) = GTF (x), where G ∈ RL×h is a matrix consisting of learnable embeddings, one row of embedding for each label.
Training: The models for feature-based datasets are trained on a single TitanX GPU with 14GB CPU memory and the models for BCG, RCV and NYT are trained on m40 GPU. We use separte ADAM optimizer (Kingma & Ba, 2014) for energy network (Θ) and for feedforward network (Φ) which optimizes parameters in alternating fashion5. For each minimization loop, we take nE , nF gradient steps respectively for estimating Θt and Φt. We leave these nE , nF as hyperparameters to be tuned. The best hyper-parameters for each model are found using Bayesian search6 and reported in Appendix D. We also report train, inference time and parameter size of different methods in AppendixE.

heading: 4.1 RESULTS FOR FEATURE BASED DATASETS
text: The experiment results across 7 feature-based dataset are presented in Table 2. The table consists of feedforward network only trained with cross-entropy (row 1), energy networks (row 2-4) described in previous section that is evaluated with GBI, and lastly feedforward network trained with SEAL framework with different types losses (row 5-9).
SEAL outperforms cross-entropy loss (CE) on almost all datasets with average gains over CE ranging from +0.6 (SEAL-LE−margin) to +2.85 (SEAL-LE−NCEranking) F1 points. We excluded excessive gain of genbase in computing the average as we believe it is an outlier. Further analysis on the performance on genbase is provided in Appendix F.
5The code we used to train and evaluate our models is available a https://anonymous.4open. science/r/SEAL/README.md.
6We use Weights & Biases (Biewald, 2020) for hyperparameter search
Between LE types, LE−NCEranking and LE−ranking performs the best while LE−margin and LE−regression performs similarly on average. LE−NCEranking and LE−ranking are the only ones that always outperforms CE in all 7 datasets whereas the others sometimes perform slightly worse than CE.
With the high gains of LE−NCEranking and LE−ranking, we are likely to conclude using samples rather than a single point can capture the output surface of the feedforward network better. To examine this hypothesis in more detail, we test sampling approaches on regression-based loss as well. We define LE−regression-s = ∑ ỹ∈S LE−regression(x,y
∗, ỹ; Θ) given sample set S. We take two approaches in collecting these samples: 1. discrete binary vectors drawn from probability vector ỹ as done in LE−NCEranking and 2. continuous perturbation of ỹ with gaussian noise. We found that discrete samples did not make any notable changes between LE−regression and LE−regression-s, however, found that using continuous samples made +0.4 F1 improvements in average and report LE−regression-s as the version which uses continuous sample. We conlcude that using samples are helpful in capturing better energy surface for traininig feedforward network, however, effect and characteristics of sample might differ per types of LE . We believe the sampling is more effective on LE−NCEranking, LE−ranking as groups of samples contribute in relative manner in learning surface in contrast to LE−regression.
To further study effect of different LE , we examine whether there is a correlation in performance between the isolated structured energy network and SEAL which utilizes idential LE . For training isolated structured energy network with margin-based and regression-based energy loss, we follow training approaches of SPEN (Belanger & McCallum, 2016) and DVN (Gygli et al., 2017). For NCE, we need access to noise distribution PN . we set pretrained feedforward network probability P (Y |X; Φ) as PN (Y ). We do not see much of correlation between the two, in other words, characteristics of LE that trains best structured energy network and LE that trains best feedforward network are different. While NCE seems to perform worst when utilizing LE−NCEranking in isolated manner, the constrastive characteristic of LE−NCEranking and LE−rankingthat is tailored to feedforward network seems to be helpful. Regardless of LE types, we also obesreve that it is much more stable and usually higher-performing to train with SEAL than to train energy network separately. In Appendix C, we also show compare individual energy network as a prediction model compared to SEAL in detail.

heading: 4.2 RESULTS FOR LARGE TEXT DATASETS
text: In order to test the effectiveness of SEAL in the scope of large pre-trained models, we examine text datasets with pre-trained BERT (Devlin et al., 2019) with adapter (Houlsby et al., 2019; Pfeiffer et al., 2020) as the feature network TE and TF as shown in Table 3. Considering the computational expense of running hyper-parameter searches on BERT, based on Table 2, we choose the best-performing SEAL (NCEranking) to compare with baselines: SEAL-margin and cross-entropy. The NYT subset we used has almost 17x larger training data than the largest feature-based data and SEAL framework outperforms cross-entropy fine-tuning. Overall, the Table 3 shows that SEAL, especially SEALNCEranking, is effective on pre-trained models that utilizes large text datasets as well. On average, SEAL-NCEranking gains 0.87, 0.68 F1 points over cross entropy and SEAL-margin, respectively, on text datasets as shown in Table 3.

heading: 5 FURTHER ANALYSIS
text: SEAL without cross entropy: Figure 2a shows the performance of SEAL when the cross-entropy loss (CE) is completely removed from equation 1. We first observe that surprisingly that SEAL itself without resorting to CE learns a model that sometimes outperforms model trained with CE ( cal500 and spo fun). However, we also observe that SEAL without CE is unstable as performance varies a lot across different datasets. This tells us that SEAL requires a signal that can bring the model FΦ(·) into reasonable region first in order to demonstrate stable performance shown in Table 2. Another
interesting observation is that SEAL-NCEranking model when it is often the weakest without CE can give the largest improvement when combined with CE together in SEAL as shown in Table 2. In short, we find CE to bring a stable synergy with energy network loss within SEAL: surpassing each individual’s performance significantly when combined together. While there could be multiple techniques to stabilize learning of SEAL besides adding CE loss, such as pre-training techniques, we leave those interesting studies as future work.
Effect of dynamic loss computation: We now compare the effect of dyanmic loss learning by comparing SEAL to SEAL-static in Figure 2b We observe that pretrained energy networks with SEAL-static also results in competitve results, often better performing than the simple cross-entropy loss. However, we oberve that in average, the dynamic loss learnig of SEAL outperforms SEAL-static for LE−regression, and LE−NCEranking.
Effect of applying ranking loss directly on FΦ With the large improvement that LE−NCEranking and LE−ranking has, we examine whether the gain is actually coming from SEAL framework or simply from the power of ranking loss itself which can be applied to feedforward network FΦ directly. We conduct this ablation study over relatively small datasets: genbase, cal500 and delicious. While we observed a near +10 F1 point in genbase, we saw decrease in performance in cal500 (-1.0 F1) and delicious (-2.3 F1) compared to plain cross-entropy model. With these experiments, we notice two trends. First, we again observe that genbase is an outlier where capturing structure can greatly increase the performance although not as much as SEAL framework helped. Two, we confirm that the ranking loss without energy network is not very effective in training feedforward model FΦ. This again confirms that the energy network EΘ plays a major role to train FΦ in the SEAL framework.

heading: 6 CONCLUSION
text: We propose SEAL: a framework that can adopt structured energy network as a trainable loss function for training feedforward network. Through examining different energy losses that trains energy network, we show that SEAL is a general framework which is effective over different loss functions and different architectures, such as MLP and BERT-based adapter. Through extensive result on 7 feature-based and 3 text-based datasets, we show that SEAL brings synergy between isolated energy newtork and cross entropy function: performing better and in more stable manner when combined together. Lastly, through ablation study, we examine the benefit of dyanmic loss learing which in average brings a gain over static version of SEAL, i.e. SEAL-static. This research opens up doors for various future work, including but not limited to, application of learned energy loss in unsupervised dataset and exploring different architectures of energy network. In the long horizon, the authors are interested in whether general pretrained scoring neural networks, such as BLEURT (Sellam et al., 2020) and BERT-score (Zhang et al., 2019), could serve as a loss function when it can provide a backpropagatable gradients to the output space.

heading: B GRADIENT TO THE LABEL REPRESENTATIONS
text: As described in section 4, the energy component of loss LF , for the feedforward network, is the sum of following two parts:
ElocalΘ (x,y) = L∑ i=1 yib > i TE(x) and E global Θ (x,y) = v >σ(My). (12)
The gradient of Eglobal w.r.t y is given by the following vector of length L
∂Eglobal
∂y = MT Diag(σ′(My))v. (13)
While the gradient of cross entropy l(y, y∗) w.r.t. y is given by the following vector of length L
∂l ∂y = y∗ ⊗ 1 y − (1− y∗)⊗ 1 1− y , (14)
where ⊗ is element-wise product. Note the dependency of the gradient for a particular component yi of y on other component in the expressions above. The gradient of cross entropy ( ∂l∂yi ) only depends on its label dimension i: yi and y∗i . In contrast, in the case of the energy function as loss (SEAL), the gradient ∂LF ∂yi depends on all L dimensions of y due to the product term My in ∂E global
∂yi .
C COMPARISON OF SEAL AND SEPARATE ENERGY NETWORK
In the following figures, we compare the F1 scores of SEAL framework with LE−regression (Figure 3), LE−NCEranking (Figure 4) and LE−margin (Figure 5) on validation set with its energy-only counterpart. We further report validation F1 scores for the output of GBI starting from the samples drawn from ŷ and randomly. We show this comparison to evaluate the performance of the energy network on points around ŷ and random points.

heading: D HYPERPARAMETERS
text: The following tables tabulate the hyper-parameters for SEAL framework for each dataset. We further fix the number of layers, layer-dimensions and dropout in feed-forward network based on the configuration of the best model trained using cross-entropy. We fix the λ2 = 1 in equation 1 to reduce degree of freedom in hyperparameter search.

heading: E ANALYSIS ON PARAMETER SIZE AND SPEED OF INFERENCE AND TRAINING
text: Here, we present the analysis on number of parameters (Table 11), inference speed (Table 12)), and training speed (Table 13) that different methods utilize per dataset.
In summary, SEAL requires approximately twice number of parameters compared to cross-entropy loss on training time, however it requires the same amount of parameter on inference time. Similarly, train time (sec) per epoch of SEAL is slower than that of cross-entropy and energy network as SEAL requires multiple backpropagation steps for both energy network and feedforward network. Nonetheless, again, at inference time, the speed of CE and SEAL is the same and is much faster (2x-7x) than the inference of energy network.
The analysis we present below are on the seven feature-based dataset, however the trend is similar for text-based dataset as well: twice parameter size on training time but equal parameter and runtime in inference time with CE method. The only difference in the text-based MLC is, due to running on a much larger BERT-baed models, that we only run one step for each update of feedforward network and energy network. Thus, unlike Table 13, train time of SEAL is approximately twice that of CE.

heading: F DATA-SPECIFIC ANALYSIS
text: In this section, we analyze why certain datasets behaves differently.
F.1 GENBASE
Here, we analyze the factors behind SEAL achieving almost perfect F1 score on the genbase dataset. It seems genbase stands out with its small label size (27 which is the smallest among Table 1) and with very clear pattern in the label space. Upon analysis, out of 27 labels, we found that 6 labels only occur as a singleton (by itself), 10 labels only occur as non-singleton, and 7 labels do not participate at all. This leaves only 4 labels occurring by themselves as well as with others. Not only that, only 7% (35/500) of training instances have more than one active label. In this peculiar setting, we conjecture feedforward networks with cross-entropy (CE) loss can easily learn singletons, and energy networks would easily learn co-occurrence, but learning both might be confusing for these two models at the opposite end of the spectrum. We believe the synergy of CE and a loss from the energy network enables capturing the best of both worlds and achieving a near-perfect score that none of the approaches by itself achieves. As can seen in Table 2, neither cross-entropy nor energy networks are nearly as strong as SEAL methods.
F.2 DELICIOUS AND CAL500
In Table 2, it seems SEAL-NCEranking and SEAL-Ranking are particularly well performing on cal500 and delicious. It turns out that cal500 and delicious have a very high diversity of 1 and 0.981 (Reference: MLC data repo). Diversity of 1 means that each data point holds a unique label set. It seems that the exposure to multiple samples and evaluations of their relative scores in NCEranking loss function can be helpful in a high-diversity setting.

Review summaries:
This paper proposes a framework named SEAL that can adopt structed energy network as both a trainable and adaptable loss function for training feedforward networks. 7 feature-based 3 text-based datasets are used for testing SEAL and showed better results and even better with combinations. Rich ablation study was given as well.

This paper proposes a framework to use Structured Prediction Energy Network (SPEN, a prior work) as a loss function. Previously SPEN is a deep architecture to learn an energy function of candidate labels, which captures structural dependencies between labels that would lead to intractable graphical models. The author of this paper finds out that it is effective to jointly learn the parameters in SPEN and the parameters in the feed-forward network (MLP). In the training time, the model alternatingly minimizes the parameters in SPEN by fixing the parameters in MLP; and then minimizes the parameters in MLP by fixing the parameters in SPEN. Empirical results in 10 multi-label datasets are given, which shows good performance boost from using the proposed methods. 

The paper presents Structured Energy As Loss (SEAL) that uses structured energy networks as a trainable loss for a feedforward network. SEAL is a general framework that supports multiple loss functions, including margin-based loss, regression-based loss, and NCE-based loss, and generalizes the recent approach in [Tu et al., 2020]. The paper presents extensive experiments on multi-label classification on a range of feature-based and text-based datasets. SEAL shows significant improvement over standard cross entropy trained networks, as well as approaches based on Structured Prediction Energy Networks (SPEN).