Venue: ICLR 2017 conference

Reviews:

final_score: rating: 3: Clear rejection
review_content: This paper proposes a way of enforcing constraints (or penalizing violations of those constraints) on outputs in structured prediction problems, while keeping inference unconstrained. The idea is to tweak the neural network parameters to make those output constraints hold. The underlying model is that of structured prediction energy networks (SPENs), recently proposed by Belanger et al. 

Overall, I didn't find the approach very convincing and the paper has a few problems regarding the empirical evaluation. There's also some imprecisions throughout. The proposed approach (secs 6 and 7) looks more like a "little hack" to try to make it vaguely similar to Lagrangian relaxation methods than something that is theoretically well motivated.

Before eq. 6: "an exponential number of dual variables" -- why exponential? it's not one dual variable per output.

From the clarification questions:
- The accuracy reported in Table 1 needs to be explained. 
- for the parsing experiments it would be good to report the usual F1 metric of parseval, and to compare with state of the art systems.
- should use the standard training/dev/test splits of the Penn Treebank.
The reported conversion rate in Table 1 does not tell us how many violations are left by the unconstrained decoder to start with. It would be good to know what happens in highly structured problems where these violations are frequent, since these are the problems where the proposed approach could be more beneficial.


Minor comments/typos:
- sec.1: "there are" -> there is?
- sec 1: "We find that out method is able to completely satisfy constraints on 81% of the outputs." -> at this point, without specifying the problem, the model, and the constraints, this means very little. How many constrains does the unconstrained method satisfies?
- sec 2 (last paragraph): "For RNNs, each output depends on hidden states that are functions of previous output values" -- this is not very accurate, as it doesn't hold for general RNNs, but only for those (e.g. RNN decoders in language modeling) where the outputs are fed back to the input in the next time frame. 
- sec 3: "A major advantage of neural networks is that once trained, inference is extremely efficient." -- advantage over what? also, this is not necessarily true, depends on the network and on its size.
- sec 3: "our goal is take advantage" -> to take advantage
- last paragraph of sec 6: "the larger model affords us" -> offers?

review_title: Not very convincing

final_score: rating: null
review_content: What is the accuracy reported in Table 1? Since the output is of variable length, it's not clear how this accuracy is computed. 

Also, for the parsing experiments it would be good to report the usual F1 metric of parseval, and to compare with state of the art systems. Why didn't you use the standard training/dev/test splits of the Penn Treebank?

The reported conversion rate in Table 1 does not tell us how many violations are left by the unconstrained decoder to start with. It would be good to know what happens in highly structured problems where these violations are frequent, since these are the problems where the proposed approach could be more beneficial.
review_title: A few questions

final_score: rating: null
review_content: My understanding is that most shift/reduce parsers have a set of valid actions at each point in time, and are hence already constrained not to produce invalid sequences. 

Also the choice of context-free languages for constraints is surprising, since I assume it'd rule out non-projective parsing. Can the paper clarify whether this is the case?

The specific gradient of the constrained loss and how it is computed can be clarified as well.
review_title: About the parsing algorithms used

final_score: rating: 4: Ok but not good enough - rejection
review_content: This paper attempted to solve an interesting problem -- incorporating hard constraints in seq2seq model. The main idea is to modify the weight of the neural network in order to find a feasible solution. Overall, the idea presented in the paper is interesting, and it tries to solve an important problem. However, it seems to me the paper is not ready to publish yet.

Comments:

- The first section of the paper is clear and well-motivated. 

- The authors should report test running time. The proposed approach changes the weight matrix. As a result, it needs to reevaluate the values of hidden states and perform the greedy search for each iteration of optimizing Eq (7). This is actually pretty expensive in comparison to running the beam search or other inference methods. Therefore, I'm not convinced that the proposed approach is a right direction for solving this problem (In table, 1, the authors mention that they run 100 steps of SGD).   

- If I understand correctly, Eq (7) is a noncontinuous function w.r.t W_\lambda and the simple SGD algorithm will not be able to find its minimum.

- For dependency parsing, there are standard splits of PTB. I would suggest the authors follow the same splits of train, dev, and test in order to compare with existing results. 


Minor comments: several sentences are misleading and should be rewritten carefully. 

- Beginning of Section 3: "A major advantage of neural network is that once trained, inference is extremely efficient." This sentence is not generally right,  and I guess the authors mean if using greedy search as inference method, the inference is efficient. 

- The description in the end of section 2 is awkward. To me,  feed-forward and RNN  are general families that cover many specific types of neural networks, and the training procedures are not necessarily to aim to optimize Eq. (2). Therefore, the description here might not be true. In fact, I don't think there is a need to bring up feed-forward networks here; instead, the authors should provide more details the connection between RNN and Eq (2) here.

- The second paragraph of section 3 is related to [1], where it shows the search space of the inference can be represented as an imperative program. 
	


[1] Credit assignment compiler for joint prediction, NIPS 2016

review_title: null

final_score: rating: 3: Clear rejection
review_content: This paper proposes a dual-decomposition-inspired technique for enforcing constraints in neural network prediction systems.

Many things don't quite make sense to me:
 1. Most seq2seq models (such as those used for parsing) have substantially better performance when coupled with beam search than greedy search, and exact search is infeasible. This is because these models are trained to condition on discrete values of past outputs in each timestamp, and hence the problem of finding the highest-scoring total sequence of outputs is not solvable efficiently. It's unclear what kind of model this paper is using which allows for greedy decoding, and how well it compares to the state-of-the-art, specially when constraint-aware beam search is used. This comparison is specially interesting because both constrained beam search and this dual-decomposition-like approach require multiple computations of the model's score.
 2. It's unclear (to me at least) how to differentiate the constraint term g() in the objective function in the general case (though the particular example used here is understandable)
 3. The paper claims that "Lagrangian relaxation methods for NLP have multipliers for each output variable that can be combined with linear models [...] . Since our non-linear functions and global constraints do not afford us the same ability" but it is possible to add linear terms to the outputs of neural networks, possibly avoiding rerunning all the expensive inference terms.

Moreover, the justification for the particular method is hand-wavy at best, with inconvenient terms from equations ignored or changed at will. At this point it might be better to omit the attempted theoretical explanation and just present this method as a heuristic which is likely to achieve the desired result.

This, plus the concerns around lack of clear comparisons with baselines on benchmark problems lead me to recommend rejection. Further explanation of how this compares with beam search, how this relates to the state-of-the-art, and a better explanation for how to come up with differentiable constraint sets, are probably required for acceptance.
review_title: Reject

Meta-review: The program committee appreciates the authors' response to the clarifying questions. Unfortunately, all reviewers are leaning against accepting the paper. Authors are encouraged to incorporate reviewer feedback in future iterations of this work.
Decision: Reject