Venue: ICLR 2018 conference

Reviews:

final_score: rating: 7: Good paper, accept
review_content: = Quality = 
Overall, the authors do a good job of placing their work in the context of related research, and employ a variety of non-trivial technical details to get their methods to work well. 

= Clarity = 

Overall, the exposition regarding the method is good. I found the setup for the sequence tagging experiments confusing, tough. See more comments below.

= Originality / Significance = 

The paper presents a clever idea that could help make SPENs more practical. The paper's results also suggest that we should be thinking more broadly about how to using complicated structured distributions as teachers for model compression.

= Major Comment =

I'm concerned by the quality of your results and the overall setup of your experiments. In particular, the principal contribution of the sequence tagging experiments seems top be different than what is advertised earlier on in the paper. 

Most of your empirical success is obtained by taking a pretrained CRF energy function and using this as a teacher model to train a feed-forward inference network. You have have very few experiments using a SPEN energy function parametrization that doesn't correspond to a CRF, even though you could have used an arbitrary convnet, RNN, etc. The one exception is when you use the tag language model. This is a good idea, but it is pretrained, not trained using the saddle-point objective you introduce. In fact, you don't have any results demonstrating that the saddle-point approach is better than simpler alternatives.

It seems that you could have written a very different paper about model compression with CRFs that would have been very interesting and you could've have used many of the same experiments. It's unclear why SPENs are so important. The idea of amortizing inference is perhaps more general. My recommendation is that you either rebrand the paper to be more about general methods for amortizing structured prediction inference using model compression or do more fine-grained experiments with SPENs that demonstrate empirical gains that leverage their flexible deep-network-based energy functions.


= Minor Comments = 

* You should mention 'Energy Based GANs"

* I don't understand "This approach performs backpropagation through each step of gradient descent, permitting more stable training but also evidently more overfitting." Why would it overfit more? Simply because training was more stable? Couldn't you prevent overfitting by regularizing more?

* You spend too much space talking about specific hyperparameter ranges, etc. This should be moved to the appendix. You should also add a short summary of the TLM architecture to the main paper body.

* Regarding your footnote discussing using a positive vs. negative sign on the entropy regularization term, I recommend checking out "Regularizing neural networks by penalizing confident output distributions."

* You should add citations for the statement "In these and related settings, gradient descent has started to be replaced by inference networks."

* I didn't find Table 1 particularly illuminating. All of the approaches seem to perform about the same. What conclusions should I make from it?

* Why not use KL divergence as your \Delta function?

* Why are the results in Table 5 on the dev data?

* I was confused by Table 4. First of all, it took me a very long time to figure out that the middle block of results corresponds to taking a pretrained CRF energy and amortizing inference by training an inference network. This idea of training with a standard loss (conditional log lik.) and then amortizing inference post-hoc was not explicitly introduced as an alternative to the saddle point objective you put forth earlier in the paper. Second, I was very surprised that the inference network outperformed Viterbi (89.7 vs. 89.1 for the same CRF energy). Why is this?

* I'm confused by the difference between Table 6 and Table 4? Why not just include the TLM results in Table 4?







review_title: Amortized inference for SPENs

final_score: rating: 9: Top 15% of accepted papers, strong accept
review_content: This paper proposes an improvement in the speed of training/inference with structured prediction energy networks (SPENs) by replacing the inner optimization loop with a network trained to predict its outputs.

SPENs are an energy-based structured prediction method, where the final prediction is obtained by optimizing min_y E_theta(f_phi(x), y), i.e., finding the label set y with the least energy, as computed by the energy function E(), using a set of computed features f_phi(x) which comes from a neural network. The key innovation in SPENs was representing the energy function E() as an arbitrary neural network which takes the features f(x) and candidate labels y and outputs a value for the energy. At inference time y can be optimized by gradient descent steps. SPENs are trained using maximum-margin loss functions, so the final optimization problem is max -loss(y, y') where y' = argmin_y E(f(x), y).

The key idea of this paper is to replace the minimization of the energy function min_y E(f(x), y) with a neural network which is trained to predict the resulting output of this minimization. The resulting formulation is a min-max problem at training time with a striking similarity to the GAN min-max problem, where the y-predicting network learns to predict labels with low energy (according to the E-computing network) and high loss while the energy network learns to assign a high energy to predicted labels which have a higher loss than true labels (i.e. the y-predicting network acts as a generator and the E-predicting network acts as a discriminator).

The paper explores multiple loss functions and techniques to train these models. They seem rather finnicky, and the experimental results aren't particularly strong when it comes to improving the quality over SPENs but they have essentially the same test-time complexity as simple feedforward models while having accuracy comparable to full inference-requiring energy-based models. The improved understanding of SPENs and potential for further work justify accepting this paper.
review_title: Interesting reinterpretation of SPENs

final_score: rating: 5: Marginally below acceptance threshold
review_content: The paper proposes training ``inference networks,'' which are neural network structured predictors. The setup is analogous to generative adversarial networks, where the role of the discriminator is played by a structured prediction energy network (SPEN) and the generator is played by an inference network.

The idea is interesting. It could be viewed as a type of adversarial training for large-margin structured predictors, where counterexamples, i.e., structures with high loss and low energy, cannot be found by direct optimization. However, it remains unclear why SPENs are the right choice for an energy function.

Experiments suggest that it can result in better structured predictors than training models directly via backpropagation gradient descent. However, the experimental results are not clearly presented. The clarity is poor enough that the paper might not be ready for publication.

Comments and questions:

1) It is unclear whether this paper is motivated by training SPENs or by training structured predictors. The setup focuses on using SPENs as an inference network, but this seems inessential. Experiments with simpler energy functions seem to be absent, though the experiments are unclear (see below).

2) The confusion over the motivation is confounded by the fact that the experiments are very unclear. Sometimes predictions are described as the output of SPENs (Tables 2, 3, 4, and 7), sometimes as inference networks (Table 5), and sometimes as a CRF (Tables 4 and 6). In 7.2.2 it says that a BiLSTM is used for the inference network in Twitter POS tagging, but Tables 4 and 6 indicate both CRFs and BiLSTMS? It is also unclear when a model, e.g., BiLSTM or CRF is the energy function (discriminator) or inference network (generator).

3) The third and fourth columns of Table 5 are identical. The presentation should be made consistent, either with dev/test or -retuning/+retuning as the top level headers.

4) It is also unclear how to compare Tables 4 and 5. The second to bottom row of Table 5 seems to correspond with the first row of Table 5, but other methods like slack rescaling have higher performance. What is the takeaway from these two tables supposed to be?

5) Part of the motivation for the work is said to be the increasing interest in inference networks: "In these and related settings, gradient descent has started to be replaced by inference networks. Our results below provide more evidence for making this transition." However, no other work on inference networks is directly cited.
review_title: New idea for training structured predictors, but unclear motivation and evaluation

Meta-review: The submission modifies the SPEN framework for structured prediction by adding an inference network in place of the usual combinatorial optimization based inference.  The resulting architecture has some similarity to a GAN, and significantly increases the speed of inference.

The submission provides links between two seemingly different frameworks: SPENs and GANs.  By replacing inference with a network output, the connection is made, but importantly, this massively speeds up inference and may mark an important step forward in structured prediction with deep learning.  
Decision: Accept (Poster)