Venue: ICLR 2018 conference

Reviews:

final_score: rating: 8: Top 50% of accepted papers, clear accept
review_content: The paper studies procedural language, which can be very useful in applications such as robotics or online customer support. The system is designed to model knowledge of the procedural task using actions and their effect on entities. The proposed solution incorporates a structured representation of domain-specific knowledge that appears to improve performance in two evaluated tasks: tracking entities as the procedure evolves, and generating sentences to complete a procedure. The method is interesting and presents a good amount of evidence that it works, compared to relevant baseline solutions. 

The proposed tasks of tracking entities and generating sentences are also interesting given the procedural context, and the authors introduce a new dataset with dense annotations for evaluating this task. Learning happens in a weakly supervised manner, which is very interesting too, indicating that the model introduces the right bias to produce better results.

The manual selection and curation of entities for the domain are reasonable assumptions, but may also limit the applicability or generality from the learning perspective. This selection may also explain part of the better performance, as the right bias is not just in the model, but in the construction of the "ontologies" to make it work.
review_title: Interesting model to incorporate domain-specific knowledge in procedural language understanding

final_score: rating: 6: Marginally above acceptance threshold
review_content: Summary

This paper presents Neural Process Networks, an architecture for capturing procedural knowledge stated in texts that makes use of a differentiable memory, a sentence and word attention mechanism, as well as learning action representations and their effect on entity representations. The architecture is tested for tracking entities in recipes, as well as generating the natural language description for the next step in a recipe. It is compared against a suit of baselines, such as GRUs, Recurrent Entity Networks, Seq2Seq and the Neural Checklist Model. While I liked the overall paper, I am worried about the generality of the model, the qualitative analysis, as well as a fair comparison to Recurrent Entity Networks and non-neural baselines.

Strengths

I believe the authors made a good effort in comparing against existing neural baselines (Recurrent Entity Networks, Neural Checklist Model) *for their task*. That said, it is unclear to me how generally applicable the method is and whether the comparison against Recurrent Entity Networks is fair (see Weaknesses).
I like the ablation study.

Weaknesses

While I find the Neural Process Networks architecture interesting and I acknowledge that it outperforms Recurrent Entity Networks for the presented tasks, after reading the paper it is not clear to me how generally applicable the architecture is. Some design choices seem rather tailored to the task at hand (manual collection of actions MTurk annotation in section 3.1) and I am wondering where else the authors see their method being applied given that the architecture relies on all entities and actions being known in advance. My understanding is that the architecture could be applied to bAbI and CBT (the two tasks used in the Recurrent Entity Networks paper). If that is the case, a fair comparison to Recurrent Entity Networks would have been to test against Recurrent Entity Networks on these tasks too. If they the architecture cannot be applied in these tasks, the authors should explain why.
I am not convinced by the qualitative analysis. Table 2 tells me that even for the best model the entity selection performance is rather unreliable (only 55.39% F1), yet all examples shown in Table 3 look really good, missing only the two entities oil (1) and sprinkles (3). This suggests that these examples were cherry-picked and I would like to see examples that are sampled randomly from the dev set. I have a similar concern regarding the generation task. First, it is not mentioned where the examples in Table 6 are taken from – is it the train, dev or test set? Second, the overall BLEU score seems quite low even for the best model, yet the examples in Table 6 look really good. In my opinion, a good qualitative analysis should also discuss failure cases. Since the BLEU score is so low here, you might also want to compare perplexity of the models.
The qualitative analysis in Table 5 is not convincing either. In Appendix A.1 it is mentioned that word embeddings are initialized from word2vec trained on the training set. My suspicion is that one would get the clustering in Table 4 already from those pretrained vectors, maybe even when pretrained on the Google news corpus. Hence, it is not clear what propagating gradients through the Neural Process Networks into the action embeddings adds, or put differently, why does it have to be a differentiable architecture when an NLP pipeline might be enough? This could easily be tested by another ablation where action embeddings are pretrained using word2vec and then fixed during training of the Neural Process Network. Moreover, in 3.3 it is mentioned that even the Action Selection is pretrained, which makes me wonder what is actually trained jointly in the architecture and what is not.
I think the difficulty of the task at hand needs to be discussed at some point, ideally early in the paper. Until examples on page 7 are shown, I did not have a sense for why a neural architecture is chosen. For example, in 2.3 it is mentioned that for "wash and cut" the two functions fwash and fcut need to be selected. For this example, this seems trivial as the functions have the same name (and you could even have a function per name!). As far as I understand, the point of the action selector is to only have a fixed number of learned actions and multiple words (cut, slice etc.) should select the same action fcut. Otherwise (if there is little language ambiguity) I would not see the need for a complex neural architecture. Related to that, a non-neural baseline for the entity selection task that in my opinion definitely needs to be added is extracting entities using a pretrained NER system and returning all of them as the selection.
p2 Footnote 1: So if I understand this correctly, this work builds upon a dataset of over 65k recipes from Kiddon et al. (2016), but only for 875 of those detailed annotations were created?

Minor Comments

p1: The statement "most natural language understanding algorithms do not have the capacity …" should be backed by reference.
p2: "context representation ht" – I would directly mention that this is a sentence encoding.
p3: 2.4: I have the impression what you are describing here is known in the literature as entity linking.
p3 Eq.3: Isn't c3*0 always a vector of zeros?
p4 Eq.6: W4 is an order-3 tensor, correct?
p4 Eq.8: What is YC and WC here and what are their dimensions? I am confused by the softmax, as my understanding (from reading the paragraph on the Action Selection Loss on p.5) was that the expression in the softmax here is a scalar (as it is done for every possible action), so this should be a sigmoid to allow for multiple actions to attain a probability of 1?
p5: "See Appendix for details" -> "see Appendix C for details"
p5 3.3: Could you elaborate on the heuristic for extracting verb mentions? Is only one verb mention per sentence extracted?
p5: "trained to minimize cross-entropy loss" -> "trained to minimize the cross-entropy loss"
p5 3.3: What is the global loss?
p6: "been read (§2.5." -> "been read (§2.5)."
p6: "We encode these vectors using a bidirectional GRU" – I think you composing a fixed-dimensional vector from the entity vectors? What's eI?
p7: For which statement is (Kim et al. 2016) the reference? Surely, they did not invent the Hadamard product.
p8: "Our model, in contrast" use" -> "Our model, in contrast, uses".
p8 Related Work: I think it is important to mention that existing architectures such as Memory Netwroks could, in principle, learn to track entities and devote part of their parameters to learn the effect of actions. What Neural Process Networks are providing is a strong inductive bias for tracking entities and learning the effect of actions that is useful for the task considered in this paper. As mentioned in the weaknesses, this might however come at the price of a less general model, which should be discussed.

# Update after the rebuttal
Thanks for the clarifications and updating the paper. I am increasing my score by two points and expect to see the ablations as well as the NER baseline mentioned in the rebuttal in the next revision of the paper. Furthermore, I encourage the authors to include the analysis of pretrained word2vec embeddings vs the embeddings learned by this architecture into the paper. 
review_title: Worried about the generality of the model, the qualitative analysis, as well as a fair comparison to Recurrent Entity Networks and non-neural baselines

final_score: rating: 9: Top 15% of accepted papers, strong accept
review_content: SUMMARY.

The paper presents a novel approach to procedural language understanding.
The proposed model reads food recipes and updates the representation of the entities mentioned in the text in order to reflect the physical changes of the entities in the recipe.
The authors also propose a manually annotated dataset where each passage of a recipe is annotated with entities, actions performed over the entities, and the change in state of the entities after the action.
The authors tested their model on the proposed dataset and compared it with several baselines.


----------

OVERALL JUDGMENT
The paper is very well written and easy to read.
I enjoyed reading this paper, I found the proposed architecture very well thought for the proposed task.
I would have liked to see a little bit more of analysis on the results, it would be interesting to see what are the cases the model struggles the most.

I am wondering how the model would perform without intermediate losses i.e., entity selection loss and action selection loss.
It would also be interesting to see the impact of the amount of 'intermediate' supervision on the state change prediction.

The setup for generation is a bit unclear to me.
The authors mentioned to encode entity vectors with a biGRU, do the authors encode it in order of appearance in the text? would not it be better to encode the entities with some structure-agnostic model like Deep Sets?


review_title: good paper

Meta-review: this submission proposes a novel extension of existing recurrent networks that focus on capturing long-term dependencies via tracking entities/their statesand tested it on a new task. there's a concern that the proposed approach is heavily engineered toward the proposed task and may not be applicable to other tasks, which i fully agree with. i however find the proposed approach and the authors' justification to be thorough enough, and for now, recommend it to be accepted.
Decision: Accept (Poster)