Venue: ICLR 2020 Conference

Reviews:

final_score: rating: 3: Weak Reject
review_content: This paper proposes the incorporation of “prior knowledge” which enters in the form of the relations between training instances in neural network training. The proposed method is tested on VQA problem, bringing improvements upon the popular soft regularizer. 
The authors claim that their method is a general technique but in fact, the constraints are drawn from specific tasks (VQA for example). So, I believe the contribution is rather domain-dependent and not general. Can you explain more how this method can be applied to general problems?

Other than that, I have some concerns:
1. Although the authors claim that they are the first to bring these annotations to VQA, I see their training procedure is closely related to cycle-consistent learning. Recent work in VQA also applied cycle consistency as an online data-augmentation technique (See Shah et al. 2019).
“Shah, M., Chen, X., Rohrbach, M., & Parikh, D. (2019). Cycle-consistency for robust visual question answering.”
2. In Section 2, the authors say “constraints on the parameter space of a model are often non-intuitive”. How are they "non-intuitive" and why the proposed method is more intuitive in terms of theory? Please clarify this.
3. Each question in Hud et al. is associated with a functional program, therefore, questions are compositional. However, arbitrary questions don’t need to strictly follow this constraint. Natural language is not exactly suited to functional programming I think. I have doubts about the claim in Section 4 “Our method can use partial annotations and should more easily extend to other datasets and human-produced annotations”. Also, the definition “A question is defined as a set of operations” does not seem correct. A question can be translated into a program that is composed of a set of operations.
4. Experimental results are not strong enough for such strong claims I believe. Regarding GQA dataset, the authors should compare the proposed method with more works, for example, Hu et al. 2019 and Hudson et al. 2019 achieve much favorable performance upon MAC.
"Hu, R., Rohrbach, A., Darrell, T., & Saenko, K. (2019). Language-Conditioned Graph Networks for Relational Reasoning." 
"Hudson, D. A., & Manning, C. D. (2019). Learning by abstraction: The neural state machine."

Minor comments: The paper is not really well written. I even found a wrong reference (Section 3).

review_title: Official Blind Review #3

final_score: rating: 6: Weak Accept
review_content: The paper argues for encoding external knowledge in the (linguistic) embedding layer of a multimodal neural network, as a set of hard constraints. The domain that the method is applied to is VQA, with various relations on the questions translated into hard constraints on the embedding space. A technique which involves distillation is used to satisfy those constraints during learning.

The question of how to encode external knowledge in neural networks is a crucial one, and the limitations of end-to-end learning with supervised data is well-made. Overall I feel that this is a potentially interesting paper, addressing an important question in a novel way, but I found the current version a highly-frustrating read (and I read the paper carefully a number of times); in fact, so frustrating that it is hard for me to recommend acceptance in its current form. More detailed comments below.

Major comments
--
The main problem I have with the paper lies with the first part of section 3, which is a key section describing the main method by which the constraints are satisfied during learning. This is very confusing. The need for the two-step procedure, in particular, and the importance of distillation needs much more explanation, and not relegated to the Appendix (which reviewers are not required to read - see call for papers). I'm not suggesting that the whole of the appendix needs moving to the body of the paper, but I would suggest perhaps 1/2 a page.

A related comment is the use of the distillation technique. This looks crucial, but I don't believe distillation is mentioned at all until the end of the related work section, and even there it comes as a bit of a surprise since there's no mention anywhere of this technique in the introduction.

I would say a little more about the distinction between the embedding space and parameter space, since you say that the external knowledge is encoded in the former and not the latter, and this is important to the overall method. Since embeddings are typically learned (or at least fine-tuned) it's not clear where the boundary is here. Another comment is that embedding space in this paper means the linguistic embedding space. Since this is ICLR and not, eg, ACL, I would make clear what you mean by embedding space.

I don't understand the diagram in Fig. 3 of the architecture, nor the explanation. What's an operation here? Is it *, or *6? I don't get why 3 is embedded by itself in the diagram, and then combined with the remainder using the MLP. Why not just run the RNN over the sequence?

Why are the training instances {3,+1...} and {4,*2,...} equivalent. I stared at this a while, and still have no idea. Also, how are these "known to be equivalent" - what's the procedure?

Minor comments including typos etc.
--
The paper has the potential to be really nicely written and well-presented. Currently it reads like it was thrown together just before the deadline (which only adds to the overall frustration as a reader).

In fig. 1 the second equivalent question example is interesting, since strictly speaking "box" and "rectangular container" are not synonyms (e.g. boxes can be round). Since strict synonymy is hard to find, does that matter? (I realise the dataset already exists and was presented elsewhere, but this might be worth a footnote).

missing (additional) right bracket after Herbert (2016)

Not sure footnote 1 needs to be a footnote. It's already been said, I think, but if it does need repeating it probably deserves to be in the body of the text.

between pairs questions

see Fig.3 -> figure 2?

see Fig.1 -> Tab. 1? (on p.5)

footnote 1 missing a right bracket

usually involve -> involves

+9]) - extraneous bracket

Fig. 4.1 -> Fig. 3? (p.6)

p.7 wastes a lot of space. In order to bring some of the appendix into the main body, I would do away with the very large bulleted list. (I don't mean lose the content - just present it more efficiently)

Remember than

Finally in Fig. 4.2 - some other figure

due of the long chains
review_title: Official Blind Review #2

final_score: rating: 3: Weak Reject
review_content: The authors propose a framework to incorporate additional semantic prior knowledge into the traditional training of deep learning models such that the additional knowledge acts as both soft and hard constraints to regularize the embedding space instead of the parameter space. To illustrate the idea, the authors use 3 different annotated knowledge that are already available in a public dataset that contains equivalent statements, entailed statements as well as functional programs and show that the final performance indeed increases. 

In general, the paper is well-written and easy to follow. The motivation is clear, i.e., to boost the performance of supervised learning tasks with additional knowledge constraints in a hard way. Compared with the existing models that treat the constraints as soft regularizers, the authors propose to additionally distill the knowledge using teacher-student framework. And this paper contributes in a novel way to incorporate the constraints with both soft and hard training strategies. However, there are several considerations which limits the contribution of this paper:

1. As a teach-student distillation framework, there are several papers using a posterior regularizer with hard constraints, e.g., "Harnessing deep neural networks with logic rules", "Constrained Convolutional Neural Networks for Weakly Supervised Segmentation". More discussions and comparisons with these models should be addressed, and even experimental comparisons if possible, since they also use knowledge distillation to convey the knowledge expressed in the constraints.

2. The proposed model differs with other soft-regularization-based methods in terms of an additional distillation process. The authors state that the combination of task loss with soft regularization lead to over-fitting. To my point of view, the distillation step actually makes similar effect with the case when only optimize the regularizer without the task loss. Hence, I am wondering what's the performance of first using the combined loss and then fix the subsequent layers to only optimize the embedding layers using only the regularization loss. This could demonstrate the difference between the distillation process and the regularization process.

3. Many recent models for VQA have been proposed, e.g, "The Neuro-Symbolic Concept Learner: Interpreting Scenes, Words, and Sentences from Natural Supervision" which also combines extra knowledge as symbolic reasoning. The authors should also compare with such models.

4. It seems the model need to sample a pair of data each time at training to compute the regularizer and also conducting the distillation process. In this case, the time cost should be non-trivial because the distillation process requires optimizing the distance between the current embedding with the hard constraint. Then the question comes as how's the time complexity of the model? What's the convergence speed?
review_title: Official Blind Review #1

Meta-review: The paper proposes a technique for incorporating prior knowledge as relations between training instances.

The reviewers had a mixed set of concerns, with one common one being an insufficient comparison with / discussion of related work. Some reviewers also found the clarity lacking, but were satisfied with the revision. One reviewer found the claim of the approach being general but only tested and valid for the VQA dataset problematic.

Following the discussion, I recommend rejection at this time, but encourage the authors to take the feedback into account and resubmit to another venue.
Decision: Reject