Title: IMPROVING AND STABILIZING DEEP ENERGY-BASED LEARNING Anonymous

Abstract: Deep energy-based models are powerful, but pose challenges for learning and inference (Belanger & McCallum, 2016) . Tu & Gimpel (2018) developed an efficient framework for energy-based models by training "inference networks" to approximate structured inference instead of using gradient descent. However, their alternating optimization approach suffers from instabilities during training, requiring additional loss terms and careful hyperparameter tuning. In this paper, we contribute several strategies to stabilize and improve this joint training of energy functions and inference networks for structured prediction. We design a compound objective to jointly train both cost-augmented and test-time inference networks along with the energy function. We propose joint parameterizations for the inference networks that encourage them to capture complementary functionality during learning. We empirically validate our strategies on two sequence labeling tasks, showing easier paths to strong performance than prior work, as well as further improvements with global energy terms.

1. INTRODUCTION:

Energy-based modeling (LeCun et al., 2006) associates a scalar compatibility measure to each configuration of input and output variables. Belanger & McCallum (2016) formulated deep energybased models for structured prediction, which they called structured prediction energy networks (SPENs). SPENs use arbitrary neural networks to define the scoring function over input/output pairs. However, this flexibility leads to challenges for learning and inference. The original work on SPENs used gradient descent for structured inference (Belanger & McCallum, 2016; Belanger et al., 2017) . Tu & Gimpel (2018; 2019) found improvements in both speed and accuracy by replacing the use of gradient descent with a method that trains a neural network (called an "inference network") to do inference directly. Their formulation, which jointly trains the inference network and energy function, is similar to training in generative adversarial networks (Goodfellow et al., 2014) , which is known to suffer from practical difficulties in training due to the use of alternating optimization (Salimans et al., 2016) . To stabilize training, Tu & Gimpel (2018) experimented with several additional terms in the training objectives, finding performance to be dependent on their inclusion. Also, when using the approach of Tu & Gimpel (2018) , there is a mismatch between the training and test-time uses of the trained inference network. During training with hinge loss, the inference network is actually trained to do "cost-augmented" inference. However, at test time, the goal is to simply minimize the energy without any cost term. Tu & Gimpel (2018) fine-tuned the cost-augmented network to match the test-time criterion, but found only minimal change from this fine-tuning. This suggests that the cost-augmented network was mostly acting as a test-time inference network by convergence, which may be hindering the potential contributions of cost-augmented inference in max-margin structured learning (Tsochantaridis et al., 2004; Taskar et al., 2004) .

In this paper, we contribute a new training objective for SPENs that addresses the above concern and also contribute several techniques for stabilizing and improving learning. We design a compound objective to jointly train both cost-augmented and test-time inference networks along with the energy function. In the context of the new objective, we propose shared parameterizations for the two inference networks that encourage them to capture complementary functionality while reducing the total number of parameters being trained. Quantitative and qualitative analysis shows clear differences in the characteristics of the trained cost-augmented and test-time inference networks. We also present three methods to streamline and stabilize training that help with both the old and new objectives. We empirically validate our strategies on two sequence labeling tasks from natural language processing (NLP), namely part-of-speech tagging and named entity recognition. We show easier paths to strong performance than prior work, and further improvements with global energy terms.

While SPENs have been used for multiple NLP tasks, including multi-label classification (Belanger & McCallum, 2016) , part-of-speech tagging (Tu & Gimpel, 2018) , and semantic role labeling (Belanger et al., 2017) , they are not widely used in NLP. Structured prediction is extremely common in NLP, but is typically approached using methods that are more limited than SPENs (such as conditional random fields) or models that suffer from a train/test mismatch (such as most auto-regressive models). SPENs offer a maximally expressive framework for structured prediction while avoiding the train/test mismatch and therefore offer great potential for NLP. However, the training and inference difficulties have deterred NLP researchers. Our hope is that our methods can enable SPENs to be applied to a larger set of applications, including generation tasks.

2. BACKGROUND:

We denote the input space by X . For an input x ∈ X , we denote the structured output space by Y(x). The entire space of structured outputs is denoted Y = ∪ x∈X Y(x). A SPEN (Belanger & McCallum, 2016) defines an energy function E Θ : X × Y → R parameterized by Θ that computes a scalar energy for an input/output pair. At test time, for a given input x, prediction is done by choosing the output with lowest energy:

EQUATION

However, solving equation ( 1) requires combinatorial algorithms because Y is a structured, discrete space. This becomes intractable when E Θ does not decompose into a sum over small "parts" of y. Belanger & McCallum (2016) relax this problem by allowing the discrete vector y to be continuous; Y R denotes the relaxed output space. They solve the relaxed problem by using gradient descent to iteratively minimize the energy with respect to y. The energy function parameters Θ are trained using a structured hinge loss which requires repeated cost-augmented inference during training.

Using gradient descent for the repeated cost-augmented inference steps is time-consuming and makes learning unstable (Belanger et al., 2017) . Tu & Gimpel (2018) propose an alternative that replaces gradient descent with a neural network trained to do inference, i.e., to mimic the function performed in equation (1). This "inference network" A Ψ : X → Y R is parameterized by Ψ and trained with the goal that

EQUATION

When training the energy function parameters Θ, Tu & Gimpel (2018) replaced the cost-augmented inference step in the structured hinge loss from Belanger & McCallum (2016) with a cost-augmented inference network F Φ and trained the energy function and inference network parameters jointly:

EQUATION

)

where D is the set of training pairs, [h] + = max(0, h), and is a structured cost function that computes the distance between its two arguments. Tu & Gimpel (2018) alternatively optimized Θ and Φ, which is similar to training in generative adversarial networks (Goodfellow et al., 2014) . As alternating optimization can be difficult in practice (Salimans et al., 2016) , Tu & Gimpel experimented with including several additional terms in the above objective to stabilize training. We adopt the same learning framework as Tu & Gimpel of jointly learning the energy function and inference network, but we propose a novel objective function that jointly trains a cost-augmented inference network, a test-time inference network, and the energy function.

The energy functions we use for our sequence labeling tasks are taken from Tu & Gimpel (2018) and are described in detail in the appendix. 

3. AN OBJECTIVE FOR JOINT LEARNING OF INFERENCE NETWORKS:

We now describe our "compound" objective that combines two widely-used losses in structured prediction. We first present it without inference networks:

min Θ xi,yi ∈D max y ( (y, y i )-E Θ (x i , y)+E Θ (x i , y i )) + +λ max y (-E Θ (x i , y) + E Θ (x i , y i )) +

This objective contains two different inference problems, which are also the two inference problems that must be solved in structured max-margin learning, whether during training or at test time. Eq. ( 1) shows the test-time inference problem. The other one is cost-augmented inference, defined as follows:

EQUATION

where y * is the gold standard output. This inference problem involves finding an output with low energy but high cost relative to the gold standard. Thus, it is not well-aligned with the test-time inference problem. Tu & Gimpel (2018) used the same inference network for solving both problems, which led them to fine-tune the network at test-time with a different objective. We avoid this issue by training two inference networks, A Ψ for test-time inference and F Φ for cost-augmented inference:

EQUATION

As indicated, this loss can be viewed as the sum of the margin-rescaled and perceptron losses for SPEN training with inference networks. We treat this optimization problem as a minmax game and find a saddle point for the game similar to Tu & Gimpel (2018) and Goodfellow et al. (2014) . We alternatively optimize Θ, Φ, and Ψ. The objective for the energy function parameters is:

Θ ← arg min Θ (F Φ (x), y i )-E Θ (x i , F Φ (x))+E Θ (x i , y i ) + +λ -E Θ (x i , A Ψ (x i ))+E Θ (x i , y i ) +

When we remove 0-truncation (see Sec. 4.1), the objective for the inference network parameters is:

Ψ, Φ ← arg max Ψ,Φ (F Φ (x), y i ) -E Θ (x i , F Φ (x)) -λE Θ (x i , A Ψ (x i ))

Joint Parameterizations. If we were to train independent inference networks A Ψ and F Φ , this new objective could be much slower than the original approach of Tu & Gimpel (2018) . However, the compound objective offers several natural options for defining joint parameterizations of the two inference networks. We consider three options which are visualized in Figure 1 and described below:

• separated: F Φ and A Ψ are two independent networks with their own architectures and parameters as shown in Figure 1 (a). • shared: F Φ and A Ψ share a "feature" network as shown in Figure 1(b) . We consider this option because both F Φ and A Ψ are trained to produce output labels with low energy. However F Φ also needs to produce output labels with high cost (i.e., far from the gold standard).

• stacked: the cost-augmented network F Φ is a function of the output of the test-time network A Ψ and the gold standard output y. That is, F Φ = q(A Ψ (x), y) where q is a parameterized function. This is depicted in Figure 1(c ). Note that we block the gradient at A Ψ when updating Φ.

For the q function in the stacked option, we use an affine transform on the concatenation of the inference network label distribution and the gold standard one-hot vector. That is, denoting the vector at position t of the cost-augmented network output by F Φ (x, y) t , we have:

F Φ (x, y) t = softmax(W q [A Ψ (x) t ; y(t)] + b q )

where semicolon (;) is vertical concatenation, y(t) (position t of y) is an L-dimensional one-hot vector, A Ψ (x) t is the vector at position t of A Ψ (x), W q is an L × 2L matrix, and b q is a bias.

One motivation for these parameterizations is to reduce the total number of parameters in the procedure. Generally, the number of parameters is expected to decrease when moving from separated to shared to stacked. We will compare the three options empirically in our experiments, in terms of both accuracy and number of parameters.

Another motivation, specifically for the third option, is to distinguish the two inference networks in terms of their learned functionality. With all three parameterizations, the cost-augmented network will be trained to produce an output that differs from the gold standard, due to the presence of the (F Φ (x), y i ) term in the combined objective. However, Tu & Gimpel (2018) found that the trained cost-augmented network was barely affected by fine-tuning for the test-time inference objective. This suggests that the cost-augmented network was mostly acting as a test-time inference network by the time of convergence. With the stacked parameterization, however, we explicitly provide the gold standard y to the cost-augmented network, permitting it to learn to change the predictions of the test-time network in appropriate ways to improve the energy function.

4. TRAINING STABILITY AND EFFECTIVENESS:

We now discuss several methods that simplify and stabilize training SPENs with inference networks. When describing them, we will illustrate their impact by showing training trajectories for the Twitter part-of-speech tagging task described in Section 6 and the appendix.

4.1. REMOVING ZERO TRUNCATION:

Tu & Gimpel (2018) used the following objective for the cost-augmented inference network (maximizing it with respect to Φ):

l 0 = [ (F Φ , y) -E Θ (x, F Φ ) + E Θ (x, y)] +

where [h] + = max(0, h). However, there are two potential reasons why l will equal zero and therefore trigger no gradient update. First, E Θ (the energy function, corresponding to the discriminator in a GAN) may already be well-trained, and it does a good job separating the gold standard output and the cost-augmented inference network output. Or, it may be the case that the cost-augmented inference network (corresponding to the generator in a GAN) is so poorly trained that the energy of its output is extremely large, leading the margin constraints to be satisfied and l 0 to be zero.

In standard margin-rescaled max-margin learning in structured prediction (Taskar et al., 2004; Tsochantaridis et al., 2004) , the cost-augmented inference step is performed exactly (or approximately with reasonable guarantee of effectiveness), ensuring that when l 0 is 0, the energy parameters are well trained. However, in our case, l 0 may be zero simply because the cost-augmented inference network is undertrained, which will be the case early in training. Then, when using zero truncation, the gradient of the inference network parameters will be 0. This is likely why Tu & Gimpel (2018) found it important to add several stabilization terms to the l 0 objective. We find that by instead removing the truncation, learning stabilizes and becomes less dependent on these additional terms. Note that we retain the truncation at zero when updating the energy parameters Θ.

As shown in Figure 3 (a) in the appendix, without any stabilization terms and with truncation, the inference network will barely move from its starting point and learning fails overall. However, without truncation, the inference network can work well even without any stabilization terms. 

4.2. LOCAL CROSS ENTROPY (CE) LOSS:

Tu & Gimpel (2018) proposed adding a local cross entropy loss, which is the sum of the label cross entropy losses over all positions in the sequence, to stabilize inference network training. We similarly find this term to help speed up convergence and improve accuracy. 

4.3. MULTIPLE INFERENCE NETWORK UPDATE STEPS:

When training SPENs with inference networks, the inference network parameters are nested within the energy function. We found that the gradient components of the inference network parameters consequently have smaller absolute values than those of the energy function parameters. So, we alternate between k ≥ 1 steps of optimizing the inference network parameters ("I steps") and one step of optimizing the energy function parameters ("E steps"). We find this strategy especially helpful when using complex inference network architectures.

To analyze this, we compute the cost-augmented loss l 1 = (F Φ , y) -E Θ (x, F Φ ) and the marginrescaled loss ∂Φ after the I steps. With k = 1, the inference network lags behind the energy, making the energy parameter updates very small, as shown by the small norms in Fig. 2(c ). The inference network gradient norm (Fig. 2(d )) remains high, indicating underfitting. However, increasing k too much also harms learning, as evidenced by the "plateau" effect in the l 1 curves for k = 50; this indicates that the energy function is lagging behind the inference network. Using k = 5 leads to more of a balance between l 1 and l 0 and gradient norms that are mostly decreasing during training. We treat k as a hyperparameter that is tuned in our experiments.

l 0 = [ (F Φ , y) -E Θ (x, F Φ ) + E Θ (x,

5. GLOBAL ENERGIES FOR SEQUENCE LABELING:

In addition to new training strategies, we also experiment with several global energy terms for sequence labeling. Eq. ( 7) in the appendix shows the base energy. To capture long-distance dependencies, we include global energy (GE) terms in the form of Eq. ( 8). Tu & Gimpel (2018) pretrained their tag language model (TLM) on a large, automatically-tagged corpus and fixed its parameters when optimizing Θ. We instead do not pretrain the TLM and learn its parameters when training Θ.

We also propose new global energy terms. Define y t = h(y 0 , . . . , y t-1 ) where h is an LSTM TLM that takes a sequence of labels as input and returns a distribution over next labels. First, we add a TLM in the backward direction (denoted y t analogously to the forward TLM). Second, we include words as additional inputs to forward and backward TLMs. We define y t = g(x 0 , ..., x t-1 , y 0 , ..., y t-1 ) where g is a forward LSTM TLM. We define the backward version similarly (denoted y t ). The global energy is therefore

E GE (y) = - T +1

t=1 log(y t y t ) + log(y t y t ) + γ log(y t y t ) + log(y t y t )

Here γ is a hyperparameter that is tuned. We experiment with three settings for the global energy: GE(a): forward TLM as in Tu & Gimpel (2018) ; GE(b): forward and backward TLMs (γ = 0); GE(c): all four TLMs in Eq. 6.

6. EXPERIMENTAL SETUP:

We consider two sequence labeling tasks: Twitter part-of-speech (POS) tagging (Gimpel et al., 2011) and named entity recognition (NER; Tjong Kim Sang & De Meulder, 2003) , described in detail in the appendix. We consider three NER modeling configurations. NER uses only words as input and pretrained, fixed GloVe embeddings. NER+ uses words, the case of the first letter, POS tags, and chunk labels, as well as pretrained GloVe embeddings with fine-tuning. NER++ includes everything in NER+ as well as character-based word representations obtained using a convolutional network over the character sequence in each word. Unless otherwise indicated, our SPENs use the energy in Eq. 7. As a baseline, we use a BiLSTM tagger trained only with the local CE term.

7. RESULTS AND ANALYSIS:

Effect of Removing Truncation. Table 1 shows results for the margin-rescaled and perceptron losses when considering the removal of zero truncation and its interaction with the use of the local Table 3 : Left: differences in accuracy/F1 between test-time inference networks A Ψ and costaugmented networks F Φ (on development sets). The "margin-rescaled" row uses a SPEN with the local CE term and without zero truncation, where A Ψ is obtained by fine-tuning F Φ as done by Tu & Gimpel (2018) . Right: most frequent output differences between A Ψ and F Φ on the development set.

CE term. Training fails for both tasks when using zero truncation without the CE term. Removing truncation makes learning succeed and leads to effective models even without using CE. However, when using the local CE term, truncation has little effect on performance. The importance of CE in prior work (Tu & Gimpel, 2018) is likely due to the fact that truncation was being used.

Effect of Local CE. The local cross entropy (CE) term is useful for both tasks, though it appears more helpful for tagging. This may be because POS tagging is a more local task. Regardless, for both tasks, as shown in Section 4.2, the inclusion of the CE term speeds convergence and improves training stability. For example, on NER, using the CE term reduces the number of epochs chosen by early stopping from ∼100 to ∼25. On Twitter POS Tagging, using the CE term reduces the number of epochs chosen by early stopping from ∼150 to ∼60.

Effect of Compound Objective and Joint Parameterizations. The compound objective is the sum of the margin-rescaled and perceptron losses, and outperforms them both (see Table 2 ). Across all tasks, the shared and stacked parameterizations are more accurate than the previous objectives.

For the separated parameterization, the performance drops slightly for NER, likely due to the larger number of parameters. The shared and stacked options also have fewer parameters to train than the separated option, and the stacked version processes examples at the fastest rate during training. receives as an additional input the gold standard label sequence, which leads to the largest differences as the cost-augmented network can explicitly favor incorrect labels.1 

The left part of:

The right part of Table 3 shows qualitative differences between the two inference networks. On the POS development set, we count the differences between the predictions of A Ψ and F Φ when A Ψ makes the correct prediction. 2 The most frequent combinations show that F Φ tends to output tags that are highly confusable with those output by A Ψ . For example, it often outputs proper noun when the gold standard is common noun or vice versa. It also captures the noun-verb ambiguity and ambiguities among adverbs, adjectives, and prepositions.

Global Energies. The results are shown in Table 4 . Adding the backward (b) and word-augmented TLMs (c) improves over only using the forward TLM from Tu & Gimpel (2018) . With the global energies, our performance is comparable to several strong results (cf. 90.94 of Lample et al., 2016 and 91.37 of Ma & Hovy, 2016) . However, it is still lower than the state of the art (Akbik et al., 2018; Devlin et al., 2019) , likely due to the lack of contextualized embeddings.

but significant challenges remain, specifically in terms of defining appropriate inference network architectures to enable efficient learning.

A APPENDIX A.1 ENERGY FUNCTIONS AND INFERENCE NETWORKS FOR SEQUENCE LABELING:

Our experiments in this paper consider sequence labeling tasks, so the input x is a length-T sequence of tokens where x t denotes the token at position t. The output y is a sequence of labels also of length T . We use y t to denote the output label at position t, where y t is a vector of length L (the number of labels in the label set) and where y t,j is the jth entry of the vector y t . In the original output space Y(x), y t,j is 1 for a single j and 0 for all others. In the relaxed output space Y R (x), y t,j can be interpreted as the probability of the tth position being labeled with label j. We then use the following energy for sequence labeling (Tu & Gimpel, 2018 ):

EQUATION

where U j ∈ R d is a parameter vector for label j and the parameter matrix W ∈ R L×L contains label pair parameters. Also, b(x, t) ∈ R d denotes the "input feature vector" for position t. We define it to be the d-dimensional BiLSTM (Hochreiter & Schmidhuber, 1997) hidden vector at t. The full set of energy parameters Θ includes the U j vectors, W , and the parameters of the BiLSTM.

Tu & Gimpel (2018) also added a global energy term that they referred to as a "tag language model" (TLM). We use h to denote an LSTM TLM that takes a sequence of labels as input and returns a distribution over next labels. We define y t = h(y 0 , . . . , y t-1 ). Then, the energy term is:

EQUATION

where y 0 is the start-of-sequence symbol and y T +1 is the end-of-sequence symbol. This energy returns the negative log-likelihood under the TLM of the candidate output y.

For inference networks, we use architectures similar to those used by Tu & Gimpel (2018) . In particular, we choose BiLSTMs as the inference network architectures in our experiments. We also use BiLSTMs for the baselines and both the inference networks and baseline models use the same hidden sizes.

A.2 EXPERIMENTAL SETUP DETAILS:

Twitter Part-of-Speech (POS) Tagging. We use the Twitter POS data from Gimpel et al. (2011) and Owoputi et al. (2013) which contains 25 tags. We use 100-dimensional skip-gram (Mikolov et al., 2013) embeddings from Tu et al. (2017) . Like Tu & Gimpel (2018) , we use a BiLSTM to compute the input feature vector for each position, using hidden dimension of size 100. We also use BiLSTMs for the inference networks. The output of the inference network is a softmax function, so the inference network will produce a distribution over labels at each position. The ∆ is L1 distance. We train the inference network using stochastic gradient descent (SGD) with momentum and train the energy parameters using Adam (Kingma & Ba, 2014) . We also explore training the inference network using Adam when we do not use the local CE loss. 4 In experiments with the local CE term, its weight is set to 1.

Named Entity Recognition (NER). We use the CoNLL 2003 English dataset (Tjong Kim Sang & De Meulder, 2003; Ma & Hovy, 2016; Lample et al., 2016) . We use the BIOES tagging scheme, following previous work (Ratinov & Roth, 2009) , resulting in 17 NER labels. We use 100-dimensional pretrained GloVe embeddings (Pennington et al., 2014) . The task is evaluated using F1 score computed with the conlleval script. The architectures for the feature networks in the energy function and inference networks are all BiLSTMs. The architectures for tag language models are LSTMs. We use a dropout keep-prob of 0.7 for all LSTM cells. The hidden size for all LSTMs is 128. We use Adam (Kingma & Ba, 2014 ) and do early stopping on the development set.

The hyperparameter k (the number of I steps) is tuned over the set {1, 2, 5, 10, 50}. γ is tuned over the set {0, 0.5, 1}. 

:

RELATED WORKAside from the relevant work discussed already, there are several efforts aimed at stabilizing and improving learning in adversarial frameworks, for example those developed for generative adversarial networks (GANs)(Goodfellow et al., 2014;Salimans et al., 2016;Zhao et al., 2016;Arjovsky et al., 2017). Progress in training GANs has come largely from overcoming learning difficulties by modifying loss functions and optimization, and GANs have become more successful and popular as a result. Notably, Wasserstein GANs(Arjovsky et al., 2017) provided the first convergence measure in GAN training using Wasserstein distance. To compute Wasserstein distance, the discriminator uses weight clipping, which limits network capacity. Weight clipping was subsequently replaced with a gradient norm constraint(Gulrajani et al., 2017).Miyato et al. (2018) proposed a novel weight normalization technique called spectral normalization. These methods may be applicable to the similar optimization problems solved in learning SPENs. Another direction may be to explore alternative training objectives for SPENs, such as those that use weaker supervision than complete structures(Rooshenas et al., 2018).

We also tried a BiLSTM in the final layer of the stacked parameterization but results were similar to the simpler affine architecture, so we only report results here with the affine architecture.

For this analysis we used the BiLSTM version of the stacked parameterization.

When comparing methods on the Switchboard-NXT(Calhoun et al., 2010) dataset, the seq2seq baseline achieves 82.80 F1 on the development set and the SPEN (stacked parameterization) achieves 83.11.

We find that Adam works better than SGD when training the inference network without the local cross entropy term.

Figures and Tables:

Figure 1: Joint parameterizations for cost-augmented inference network F Φ and test-time inference network A Ψ .

Figure 2: Training trajectories with different numbers of I steps. The three curves in each setting correspond to different random seeds. (a) cost-augmented loss after I steps; (b) margin-rescaled loss after I steps; (c) gradient norm of energy function after E steps; (d) gradient norm of test-time inference network after I steps. Tu & Gimpel (2018) use one I step after each E step.

Figure 3(b) shows faster convergence to high accuracy when adding the local CE term. More comparisons are in Section 7.

y)] + averaged over all training pairs (x, y) after each set of I steps. The I steps seek to maximize these terms and the E steps seek to minimize them. Figs. 2(a) and (b) show l 1 and l 0 during training for different numbers of I steps for every one E step. Fig. 2(c) shows the norm of ∂EΘ(x,AΨ) ∂Ψ after the E steps, and Fig. 2(d) shows the norm of

∂l0

Figure 3: Training trajectories with different settings. The three curves for each setting correspond to different random seeds. Tu & Gimpel (2018) use truncation and CE during training.

Test set results for Twitter POS tagging and NER of several SPEN configurations. Results with * correspond to the setting ofTu & Gimpel (2018).

Table content: <table><tr><td/><td>zero</td><td/><td>POS</td><td>NER</td><td>NER+</td></tr><tr><td/><td colspan="5">trunc. CE acc (%) F1 (%) F1 (%)</td></tr><tr><td/><td>yes</td><td>no</td><td>13.9</td><td>3.91</td><td>3.91</td></tr><tr><td>margin-</td><td>no</td><td>no</td><td>87.9</td><td>85.1</td><td>88.6</td></tr><tr><td>rescaled</td><td>yes</td><td>yes</td><td>89.4*</td><td>85.2*</td><td>89.5*</td></tr><tr><td/><td>no</td><td>yes</td><td>89.4</td><td>85.2</td><td>89.5</td></tr><tr><td>perceptron</td><td>no no</td><td>no yes</td><td>88.2 88.6</td><td>84.0 84.7</td><td>88.1 89.0</td></tr></table>

Test set results for Twitter POS tagging and NER. |T | is the number of trained parameters; |I| is the number of parameters needed during the inference procedure. Training speeds (examples/second) are shown for joint parameterizations to compare them in terms of efficiency. Best setting (highest performance with fewest parameters and fastest training) is in boldface.

Table content: <table><tr><td/><td/><td/><td>POS</td><td/><td/><td>NER</td><td>NER+</td></tr><tr><td/><td colspan="3">acc (%) |T |</td><td>|I|</td><td colspan="2">speed F1 (%) |T |</td><td>|I|</td><td>speed</td><td>F1 (%)</td></tr><tr><td>BiLSTM</td><td/><td>88.8</td><td colspan="2">166K 166K</td><td>-</td><td colspan="2">84.9 239K 239K</td><td>-</td><td>89.3</td></tr><tr><td colspan="7">SPENs with inference networks (Tu &amp; Gimpel, 2018):</td></tr><tr><td colspan="2">margin-rescaled</td><td>89.4</td><td colspan="2">333K 166K</td><td>-</td><td colspan="2">85.2 479K 239K</td><td>-</td><td>89.5</td></tr><tr><td>perceptron</td><td/><td>88.6</td><td colspan="2">333K 166K</td><td>-</td><td colspan="2">84.4 479K 239K</td><td>-</td><td>89.0</td></tr><tr><td colspan="8">SPENs with inference networks, compound objective, CE, no zero truncation (this paper):</td></tr><tr><td>separated</td><td/><td>89.7</td><td colspan="2">500K 166K</td><td>66</td><td colspan="2">85.0 719K 239K</td><td>32</td><td>89.8</td></tr><tr><td>shared</td><td/><td>89.8</td><td colspan="2">339K 166K</td><td>78</td><td colspan="2">85.6 485K 239K</td><td>38</td><td>90.1</td></tr><tr><td>stacked</td><td/><td>89.8</td><td colspan="2">335K 166K</td><td>92</td><td colspan="2">85.6 481K 239K</td><td>46</td><td>90.1</td></tr><tr><td/><td/><td/><td/><td/><td/><td>test-time (AΨ)</td><td>cost-augmented (FΦ)</td></tr><tr><td/><td/><td/><td/><td/><td/><td>common noun</td><td>proper noun</td></tr><tr><td/><td/><td/><td>POS</td><td colspan="2">NER</td><td>proper noun</td><td>common noun</td></tr><tr><td/><td/><td colspan="2">AΨ -FΦ</td><td colspan="2">AΨ -FΦ</td><td>common noun</td><td>adjective</td></tr><tr><td colspan="2">margin-rescaled</td><td/><td>0.2</td><td>0</td><td/><td>proper noun</td><td>proper noun + possessive</td></tr><tr><td/><td colspan="2">separated</td><td>2.2</td><td colspan="2">0.4</td><td>adverb</td><td>adjective</td></tr><tr><td>compound</td><td>shared</td><td/><td>1.9</td><td colspan="2">0.5</td><td>preposition</td><td>adverb</td></tr><tr><td/><td>stacked</td><td/><td>2.6</td><td colspan="2">1.7</td><td>adverb</td><td>preposition</td></tr><tr><td/><td/><td/><td/><td/><td/><td>verb</td><td>common noun</td></tr><tr><td/><td/><td/><td/><td/><td/><td>adjective</td><td>verb</td></tr></table>

Table3shows how the performance of the test-time inference network A Ψ and the cost-augmented inference network F Φ vary when using the new compound objective. The differences between F Φ and A Ψ are larger than in the baseline configuration, showing that the two are learning complementary functionality. With the stacked parameterization, the cost-augmented network F Φ NER test F1 scores with global energy terms. ‡: We took the best configuration from NER/NER+ and evaluated it in the NER++ setting.

Table content: <table><tr><td>NER NER+ NER++</td></tr></table>

Peer review (iclr2020):
*Summary* SPENs are an interesting structured prediction technique. They used a trained deep energy function (like the discriminator in a GAN) to reward high-quality structured outputs. Prior work (Tu and Gimpel 2018) contributed an *amorized inference* method for training SPENs that appears to be considerably more practical than the training methods originally proposed for SPENs. This paper provides additional improvements to the amortized inference training method that make it simpler and perform better. Directly address a weakness of Tu and Gimpel: the amortized predictor was trained for a different objective (loss augmented inference) than it will be used for at test time. They explore a variety of ways to tie two separate amortized predictors together. The most interesting one is where the train-time predictor has access to the ground truth label. *High-level assessment* Overall, the authors do a good job of providing careful ablation analysis of their contributions. This level of analysis is absent from many submitted papers. The paper*s contributions also seem like an important step in making SPENs practical. I can imagine that other reviewers may be critical of the size of the paper*s contributions: it mainly presents a collection of tricks of the trade to make SPEN training work in practice. Much of the ICLR community is very interested in structured prediction, but It appears that SPENs have not had much traction. I think that this paper is important because it may teach practitioners that they should consider SPENs. *Additional Comments* Sec 4.2: your paper should be self contained. What is this loss? Paper would be improved if it helped explain why people should care about SPENs. I*d include more of a sales pitch for the benefits of them. Also, your approach is substantially more complex than training a straightforward model with some deep representations. Is it worth the complexity? How do you justify removing the hinge term in your max-margin objective when updating the inference network? Is there some principle that would suggest that this is OK? The idea of having the inference network condition on the ground truth label is cool. This is similar in RL to how people will use *asymmetric critics*, where the critic has access to more information than the actor. For example, the critic may have access to a scene*s actual 3-D geometry, while the actor just has a 2D camera view of it. It might be worth drawing this connection.