Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: This work introduces a new neural architecture (FOLNet) that incorporates a first-order logical inductive bias. Taking inspiration from the forward-chaining algorithm, the architecture is recursively applying learnable Kernel operations on unary and binary representations of tokens.

Input and outputs are traditional token ids, making FOLNet an easy plug-and-play replacement from traditional Transformers. In addition, traditional Transformers can still be represented by a FOLNet making the proposed architecture more flexible and likely more powerful.

Experiments on GLUE, SQuAD2.0, and FOLIO show that this new architecture performs better than comparable baselines such as BERT, RoBERTa, ALBERT, and Megatron.

strength_and_weaknesses: **strengths**

This is a strong paper that introduces a novel architecture with inductive biases from first-order logic. Experimental results are very promising and the fact that logical tasks remain a challenge for traditional Transformers can make this work impactful.
The paper is well written and surprisingly easy to read. However the methodology could benefit from some clarifications listed below:

**weaknesses**

1. Methodology : After pre-training on large datasets, were the models fine-tuned on individual tasks (GLUE, SQuAD, FOLIO) ? If so, for how long? it should be mentioned somewhere.

2. Methodology : It is not clear how the final representations output from the model u_L(x) and u_L(x, y) are used in the computation of any loss. How does one go from these two representations to say a softmax layer to predict the masked token in MLM? Similarly for the other losses NSP and SOP.

*Question*: Related to the point above, is it safe to assume that FOLNets are encoder type networks and cannot be used in an decoder-only or encoder-decoder fashion? If this assumption is wrong, then the paper should also describe how u_L(x) and u_L(x, y) are used to predict next words.

3. Literature : Some influential previous work should be further discussed. Neural Theorem Provers (NeurIPS’17), and in particular Greedy Neural Theorem Provers (AAAI’20) applied logical modules on text before. More recently, Edge Transformers (NeurIPS’21) proposed a triangular attention mechanism between pairs of tokens in an effort to better represent logical behaviors between tokens. Comparing FOLNets to these works would help contextualize the work better.

4. Experiments : Right now the paper shows improvements on “classical” language tasks. It would be a nice bonus to also show that the FOL inductive bias is indeed useful for more reasoning heavy tasks such as GSM8k, ProofWriter, CLUTRR, etc…

clarity,_quality,_novelty_and_reproducibility: The paper is clear and easy to read.
Experimental details are clear and should be reproducible.
Some methodology clarifications are needed (see section above).

summary_of_the_review: strong paper. more previous work should be discussed and some clarifications are needed.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: The paper proposes a new family of transformer-style architectures inspired by inductive logic programming, with the goal of improving natural language reasoning capabilities. They connect this new family of architectures to existing commonly used modifications. They show good results on widely used benchmarks, modulo some concerns I raise later.
strength_and_weaknesses: Method:

The problem of learning new transformer architectures to aid reasoning is of broad interest to the community.

The model is well motivated by comparing to things like relative position encoding, and trying to use logical formalisms to derive new architectural variants on transformers.

How much the new architectural variants actually maintain the spirit of the logical formalisms is a bit harder to parse,  for example in 3.1 we are replacing discrete booleans with arbitrary real numbers and stating that “it characterizes the extent to which the atom is true”, which usually implies some value in [0,1] or some kind of increasing transformation, which I do not think you have here.

Experiments: 

The experiments are quite thorough, but are often confounded by the proposed method having more parameters than the baselines. Additionally, experiments would be better if they included your own reruns of BERT/Roberta experiments using the same settings to make it apples to apples in some cases, this is especially helpful for the situations where gains are small. This would also make it easier to make apples-to-apples comparisons in terms of # of parameters, but it would be a lot of work.

The ablation experiments in Table #3 are nice and quite appreciated. It would be good to have another column saying how many parameters each ablated model has.

Having some more concrete examples / toy examples of how these new architectures specifically aid reasoning on concrete examples would be really helpful.


clarity,_quality,_novelty_and_reproducibility: The paper is generally well written, aside from some typographical nitpicks which I will list here. The experiments and mathematical formalism are quite thorough, and the comparisons are to widely used benchmarks. While this is not the first work to apply ideas from inductive logic programming to deep learning, they make a solid contribution.


Various copy-editing nits from experiment section:

“Learns better language representations” should be “learn better language representations”

“Many different variants of widely encoder-only” should be “many different variants of widely used encoder-only”

“To demonstrates its advantage” should be “to demonstrate its advantage”

“We first pretaining” should be “we first pretrain”


Issues with experiment presentation:

ALBERT XXL+ value of 96.9 is not bold but FOLNet Large of 96.8 is bold in the same column

Advantage of dual branch, Benefits of larger D2, contributions of larger D2, often are referencing table 4 when you mean to reference table 3

In 4.3, “line #14 on table 9” is referring to the wrong table, since table 9 is in an appendix and has no line #14. These are supposed to refer to table 4 I believe 

In table #9 for MNLI, both 50.8’s should be bolded
summary_of_the_review: This is a well-motivated and well-executed paper addressing a problem of wide interest to the community. It could benefit from more experiments giving intuition of how the architectures work in practice, and some very careful apples-to-apples comparisons, but the ablations already go a good way in that direction. This paper merits acceptance.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper introduce a novel logical inductive bias which treats the language representation learning problem as logic programming. We then develop a fully-differentiable neural architecture (FOLNet) that forward-chains a set of neural logic operators, which effectively encodes this inductive bias. The proposed FOLNet architecture has the same input-output interface as the transformer models and can be pretrained over large-scale text data. Experimental results demonstrate that the FOLNet architecture significantly outperforms different variants of transformer models under the same pretraining losses. The results further show that the inherent dual-branch architecture has many advantages over the single-branch version in transformers.
strength_and_weaknesses: Strength: 

1. This paper develops a novel neural architecture named FOLNet (First-Order Logic Network), to encode this new logic inductive bias.

2. This paper finds that the self-attention module in transformers can be composed by two of our neural logic operators, which probably explains their strong reasoning performance.

3. The extensive experimental results support their  claims and validates the superiority.

Weaknesses：

1. It is not clear why FOLNet  could encode  logic inductive bias?
clarity,_quality,_novelty_and_reproducibility: It is novel and of high quality.
summary_of_the_review: Interesting and useful method.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: This paper proposes a new neural architecture inspired by first-order logic and the authors have conducted extensive experiments to show the effectiveness of their approach. 

I do share the concern with Reviewer dLtS: While the model architecture is inspired by first-order logic, there is no evidence that the model is performing ACTUAL first-order logic reasoning. The proposed model is still a black box and it's not completely clear what kind of inductive bias that the architecture brings in.
justification_for_why_not_higher_score: The proposed approach isn't as groundbreaking as the scores as. Although it's inspired by first-order logic, it's still a blackbox machinery and doesn't perform actual reasoning. 
justification_for_why_not_lower_score: The idea of mimicking first-order logic reasoning in the neural architecture design is refreshing, so I would recommend acceptance.
Decision: Accept: poster