Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper:  In this paper, a new framework for learning with logical constraints is proposed. The shortcut satisfaction issue is addressed by introducing dual variables for logical connectives, encoding how the constraint is satisfied. A variational framework is proposed where the logical constraint is expressed as a distributional loss that is compatible with the model’s original training loss. Theoretical analysis shows that the proposed approach bears some nice properties, and experimental evaluations demonstrate its superior performance in both model generalizability and constraint satisfaction.
strength_and_weaknesses: The work successfully addressed the shortcut satisfaction issue and proposed a variational framework where the logical constraint is expressed as a distributional loss that is compatible with the model’s original training loss. However the design lacks theoretical support.
clarity,_quality,_novelty_and_reproducibility: The paper is clearly state and the content is well organized. 
summary_of_the_review:  In this paper, a new framework for learning with logical constraints is proposed. The shortcut satisfaction issue is addressed by introducing dual variables for logical connectives, encoding how the constraint is satisfied. A variational framework is proposed where the logical constraint is expressed as a distributional loss that is compatible with the model’s original training loss. Theoretical analysis shows that the proposed approach bears some nice properties, and experimental evaluations demonstrate its superior performance in both model generalizability and constraint satisfaction. The paper is clearly state and the content is well organized. The work successfully addressed the shortcut satisfaction issue and proposed a variational framework where the logical constraint is expressed as a distributional loss that is compatible with the model’s original training loss.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: The paper proposes a new way of integrating logical constraints in the training of deep neural networks. In particular, the authors try to solve the problem that the current models often learn to satisfy the constraints by learning the *obvious solution* (e.g., given $A \to B$ the models learn to set $A = \bot$). The logical constraints taken in considerations by the authors are written in CNF and the atoms in the formula are $v \cdot c$, where: 
- $v$ is a variable
- $\cdot \in \{ \le, \ge, <, >, =, \not = \}$
- $c$ is a constant. 
In order to avoid the obvious solution the authors propose to associate a a dual variable to each logical connective.

strength_and_weaknesses: **Strength:**

1) The paper identifies an interesting problem in the current state-of-the-art models 
2) The solution proposed seems very interesting

**Weaknesses:**

1) I don't know variational learning well, but I know Neurosymbolic learning quite well and I struggled to follow the paper. A bit more explanations on the variational learning framework could really help somebody with my background.
2) The theoretical part seems to be disconnected from the experimental analysis. Indeed, in the theoretical analysis the authors state that all the atoms in the formulas are of the type $v \cdot c$, but then in the experimental analysis we have constraints of the type $v_1 +v_2 \cdot c$. Suppose we simply set $v = v_1 + v_2$ in the constraint $v \cdot c$ do all the theorems still hold? Do they also hold if we have multiple constraints sharing the same variables (e.g., $v_1 + v_2 < c$ and $v_3 - v_2 > c$). 
3) In the experimental analysis the authors compare only with PD and DL2. I would suggest to also add as baseline DeepProbLog [1] as it is one of the most well known methods for including the constraints. 
4) In Table 2 the authors report the results with 2% and 5% of the data. This is a very little amount. Could the authors produce the results also for 10% 20% and 50%?
5) The authors are missing some related works which they should discuss. In particular: [2,3,4,5,6]. 
6) Are the experiments run for only one seed? Ideally we should have 5 runs and we should have average and std. (I am asking because from what I know variational learning based methods tend to be quite unstable, so it would be useful to asses the stability of the proposed model)


**References:**

[1] Robin Manhaeve, Sebastijan Dumancic, Angelika Kimmig, Thomas Demeester, and Luc De Raedt. DeepProbLog: Neural probabilistic logic programming. In Proc. of NeurIPS, 2018.

[2] Samy Badreddine, Artur d’Avila Garcez, Luciano Serafini, and Michael Spranger. Logic tensor networks. Artif. Intell., 303, 2022.

[3] Eleonora Giunchiglia and Thomas Lukasiewicz. Multi-label classification neural networks with hard logical constraints. JAIR, 72, 2021.

[4] Tao Li and Vivek Srikumar. Augmenting neural networks with first-order logic. In Proc. of ACL, 2019.

[5] Zhun Yang, Adam Ishay, and Joohyung Lee. NeurASP: Embracing neural networks into answer set programming. In Proc. of IJCAI, 2020.

[6] Ahmed, Kareem, Stefano Teso, Kai-Wei Chang, Guy Van den Broeck, and Antonio Vergari. Semantic Probabilistic Layers for Neuro-Symbolic Learning. arXiv preprint arXiv:2206.00426, 2022.
clarity,_quality,_novelty_and_reproducibility: Here I am just giving score, see above for details

**Clarity**: 6/10
**Quality:** 8/10
**Novelty:** 9/10
**Reproducibility:** 10/10 (code is already available, I didn't try to run it though)
summary_of_the_review: TL;DR: The paper seems novel and interesting, it needs to improve on clarity
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: The paper tackles the problem of "shortcut satisfaction" in the context of neuro-symbolic integration (their results are however of interest for statistical relational learning at large).  Shortcut satisfaction, in this context, refers to models allocating most probability mass to "easy to satisfy" variable configurations.  This is a well-known and central (but sadly overlooked) problem in NeSy and SRL.  The authors propose a technique reminiscent of T-norm approaches to constraint satisfaction, which however involves injecting dual variables into the formulation of the knowledge-based loss which control what parts of the various formulas are active.  The authors also derive an alternating SGD optimization algorithm.  Experimental results in four tasks against SOTA competitors illustrate the efficacy of the proposed approach in avoiding shortcut satisfaction while retaining high accuracy and even outperforming the competitors.
strength_and_weaknesses: PROS
----

- Crisply written, if a bit dense in parts.
- Tackles an underappreciated but very central issue in NeSy integration and SRL.
- Provides a very clear formulation and diagnosis of the problem.
- Proposed technique is creative and non-trivial.
- Very sensible (and surprisingly intuitive) optimization algorithm.
- Contribution is carefully compared to relevant literature.
- Empirical results are very convincing.

CONS
----

- Some parts of the text are a bit dense and would benefit from in-line concrete examples.
- Does not entirely explore limits of the proposed approach.
clarity,_quality,_novelty_and_reproducibility: **Clarity**: The text is very well written, although a bit dense in parts.  I am referring especially to Section 2.1.  The meaning of the dual variables is not easy to grasp, and a couple of concrete examples would help the reader wad through that section.  The rest of the paper is easier to read, also thanks to the examples in the appendix.  The formalization is crisp.

**Quality**: This is a very solid contribution:  it tackles an important (at least to my eyes) open problem, it provides a very elegant modelization and solution procedure, a clear optimization algorithm, and plenty of evidence that the proposed approach (which, by the way, should be given a **name** for ease of reference) works as intended.  The choice of experimental task and competitors is also very appropriate.

One minor complaint is that, although the authors do mention a key limitation of their proposed method in the conclusion (namely, time complexity), they do not address possible limitations of the variational setup.  In fact, it is not entirely clear to me whether "spreading out" the model's probability mass over all logical terms/clauses is always the right approach.  This would be worth discussing briefly in the conclusion. 

**Originality**:  To the best of my knowledge, no prior work has tackled shortcut satisfaction under knowledge constraints.  I believe this work will have quite an impact within the SRL and NeSy communities.


**Reproducibility**:  Experimental details are given in the appendix.  It is not clear to me whether the code will be made available upon publication, though.
summary_of_the_review: Very solid and significant contribution that tackles an overlooked open problem.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: In the setting of constrained deep learning, a key challenge is to inject logical constraints as loss functions while preserving the semantics of their truth tables. Notably, some logical constraints such as material implications have *preferred tuples* which should be reached whenever possible. This paper aims at addressing this challenge by providing a new translation of CNF formulas into loss functions and a variational learning framework. One of the key ideas of the translation is to map simple conjunctions and disjunctions as optimization functions into corresponding simplices. This translation satisfies desirable properties such as monotonicity, and interpretability. The variational learning framework takes the form of a nonzero-sum game for which local Nash equilibria are searched using a stochastic gradient-ascent algorithm with min-oracle. Experiments on various problems corroborate the effectiveness of this approach.

strength_and_weaknesses: Overall, the paper is well-motivated and well-written. I found the logic-to-loss translation in Section 2 very intuitive. Based on this translation, the use of a stochastic gradient descent-ascent learning algorithm is quite natural. The convergence analysis is, however, much more difficult to establish, and as far as I could check, the proof looks correct. Finally, experimental results reported for various constrained learning tasks highlight the merits of this framework. 

I did not find any real weakness in this paper. I just have some minor recommendations, detailed below.

clarity,_quality,_novelty_and_reproducibility: The related work section is well-detailed, but it is missing some recent results about translating logical constraints into loss functions using straight-through estimators (e.g. Yang et. al., 2022). Yet, the present approach that uses dual variables and simplices is conceptually more simple to implement and analyze. 

The clarity of the paper is remarkable. In Section 2.2, I would here suggest replacing *General Formulas* with *Clausal Formulas* (or CNF formulas). Indeed, general (negation normal form) formulas are beyond the reach of this translation method. For Theorem 3, it would be nice to add a brief comment about $\kappa$, $\gamma$, and $\Delta_0$. 

**Reference:**

Yang, Z., Lee, J. and Park, C.. (2022). Injecting Logical Constraints into Neural Networks via Straight-Through Estimators. Proceedings of the 39th International Conference on Machine Learning, 2022.

summary_of_the_review: This is a conceptually simple, flexible, and robust approach to neural network learning with logical constraints. In a nutshell, a good paper.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: This paper proposes a new neuro-symbolic technique that pushes the state of the art in terms of supporting inequality constraints (i.e., relational constraints) and avoiding trivial ways of satisfying implication bodies. Presentation is clear, with clear contributions and empirical results. Obvious accept.
justification_for_why_not_higher_score: It is a very strong paper, but perhaps some of the related work discussion and comparison could be stronger for it to become an oral. There are many neuro-symbolic baselines.
justification_for_why_not_lower_score: Best in my batch.
Decision: Accept: notable-top-25%