Venue: ICLR 2021 Conference

Reviews:

final_score: rating: 5: Marginally below acceptance threshold
review_content: The paper integrates a neural encoding of first-order logic with deep learning architectures, supplementing training data with declarative knowledge. The approach is experimentally evaluated on MNIST and visual predicate detection, demonstrating  a reduction in data requirements. 

The paper deals with an important topic and provides very promising results. However, I would've liked to see a more extensive experimental evaluation. Also, which performance gain does additional knowledge produce for visual predicate detection, compared to the current SOTA for that task. So, are the first-order constraints always satisfied by the produced results? 

After rebuttal: The long discussion with the authors to clarify the questions in my review and further related questions actually shows that the paper is not really clear and would still benefit from a substantially improved presentation. So, I slightly downgraded my overall rating of the paper.
review_title: An interesting paper

final_score: rating: 3: Clear rejection
review_content: This paper focuses on the problem of augmenting the user-provided knowledge into Neural networks. The core premise of the work is that the existing approaches tend to replace the True and False with values in [0,1] and then use maximum/minimum operators which have vanishing gradients. The key idea is to use logit representation and therefore, the authors hope to avoid the vanishing gradients problem. The paper presents empirical results to argue that embedding knowledge helps. 

I think its an interesting paper but somehow the comparison with prior work is not properly contrasted, which makes me believe that the paper has probably reinvented quite a bit of aspects. An important missing work is [1], which in fact uses product and addition instead of minimum/maximum.  (It is well known that exists and forall operators can be expressed using disjunction and conjunction)

Also, Xie et al (NeurIPS 2020) build on [1], and  do perform empirical on the same VRD dataset and achieve significant improvements; their numbers seem better than what is reported in Table 1.

In light of the above remarks, it is hard to understand the new contribution of the paper. 


[1] https://arxiv.org/abs/1711.11157 (Published at ICML 2018)
review_title: Interesting work but lack of empirical comparison and contrast with prior work 

final_score: rating: 6: Marginally above acceptance threshold
review_content: The paper describes an approach to combine first-order knowledge bases with deep networks. The idea is to represent logical operators as neural components that produce continuous values (using a logistic output) instead of boolean values. The proposed approach is formalized using Tarski models. Here, it is shown that if we consider each parameterization as a model, the learning samples models from the logical theory. Results are proven that such an approach is both sound and complete for any first-order logic theory.

Neuro-symbolic models seem very promising and therefore the paper addresses a very significant problem. The approach seems similar to Rockstachel et al. 2017’s approach for theorem proving in first-order logic. It seems like the main contribution though is that their model is more general and they prove soundness and completeness of their approach. Maybe a little bit more on why the approach is more general than existing approaches can help motivate the novelty aspect more clearly.


The experiments seem to be well thought-out though only 1 real domain and 1 toy domain are explored. However, I was not sure how the knowledge bases are constructed for the VRD problems. The single example seems clear. But how is it done for any problem in general? What types of logical formulas were pre-specified corresponding to visual relationships since I imagine this can be quite extensive. Is there any impact on training time/effort as the formulas become more complex since the architecture of the network becomes more complex. Also, does the expert knowledge change for each image? I think some more details on this task will help clarify the experiments and make the results more easily reproducible.

In summary, this seems like a nice paper but a little fuzzy on the details of the experiments which can be clarified better.
review_title: Unifies logical and neural network models. Seems like a good paper that can be stronger by clarifying some details.

final_score: rating: 5: Marginally below acceptance threshold
review_content: This paper introduces Deep Adaptive Semantic Logic (DASL) as a novel framework that automates the generation of deep neural networks on the basis of some background domain knowledge. A first-order logic formalism (which exploits Tarski models) is used to represent knowledge, and, subsequently, to model deep networks on the basis of such knowledge. This resembles ideas coming from the neuro-symbolic community, such as the pioneering work by Towel and Shavlik on KBANNs, and more recent approaches such as Neural Tensor Networks, DeepProblog, Lyrics.

The model is evaluated on a toy problem (extracted from MNIST) and the task of visual relationship detection, that aims to predict relations between objects in the scene.

Although the work is interesting, and perfectly in scope with the conference, focusing on a very hot topic in AI, I have to raise some concerns regarding the experimental evaluation, which basically does not compare against similar approaches, nor it considers similar tasks in the literature.

In particular, no comparison is presented with other approaches from the Neuro-Symbolic community: more precisely, maybe some tasks could have been borrowed from previous works. For example, the DeepProbLog paper presents a similar task regarding sums with MNIST digits? Or, in the setting of image classification, the framework of "Learning from constraints" has been evaluated on a task which considers exploiting class hierarchies and commonsense (e.g., see "Image Classification Using Deep Learning and Prior Knowledge", AAAI Workshops, 2018).

Other comments:

* What is the state-of-the-art for the VRD task? Is DASL competitive?

* The way references are used in the text should be made consistent. In Sections 1 and 2, there is no parentheses before references (e.g., "...estimated from data LeCun et al. (2015)") whereas in Section 3 there are parentheses, which is to be preferred (e.g., "...by boolean Tarski models [Chang & Keisler (1973), Weiss & D’Mello (1997)]").

- "techniques such conditional random fields" -> "techniques such as conditional random fields"
- "data scarce conditions" -> "conditions of data scarcity"





review_title: Deep adaptive semantic logic as a framework for knowledge-driven neural network design

Meta-review: The reviewers and AC appreciate the improvements made to the paper and thank the authors for engaging with the reviewer questions.
There are now quite a few neuro-symbolic approaches, and they are all rather similar. This places a larger burden on the authors to have a thorough and systematic experimental comparison and related work discussion. Reviewers also believe the clarity of the paper should still be improved. The revised paper already made good progress in addressing these concerns, yet the reviewers still believe the paper would strongly benefit from another round of revisions.
Decision: Reject