Learning Language Representations with Logical Inductive Bias
Authors: Jianshu Chen
Venue: ICLR 2023 poster (2023)
Keywords: Pretraining
model architecture
language representation learning
inductive bias

Abstract: Transformer architectures have achieved great success in solving natural language tasks, which learn strong language representations from large-scale unlabeled texts. In this paper, we seek to go further beyond and explore a new logical inductive bias for better language representation learning. Logic reasoning is known as a formal methodology to reach answers from given knowledge and facts. Inspired by such a view, we develop a novel neural architecture named FOLNet (First-Order Logic Network), to encode this new inductive bias. We construct a set of neural logic operators as learnable Horn clauses, which are further forward-chained into a fully differentiable neural architecture (FOLNet). Interestingly, we find that the self-attention module in transformers can be composed by two of our neural logic operators, which probably explains their strong reasoning performance. Our proposed FOLNet has the same input and output interfaces as other pretrained models and thus could be pretrained/finetuned by using similar losses. It also allows FOLNet to be used in a plug-and-play manner when replacing other pretrained models. With our logical inductive bias, the same set of ``logic deduction skills'' learned through pretraining are expected to be equally capable of solving diverse downstream tasks. For this reason, FOLNet learns language representations that have much stronger transfer capabilities. Experimental results on several language understanding tasks show that our pretrained FOLNet model outperforms the existing strong transformer-based approaches.

Published as a conference paper at ICLR 2023
LEARNING LANGUAGE REPRESENTATIONS WITH LOGICAL INDUCTIVE BIAS
Jianshu Chen
Tencent AI Lab, Bellevue, WA 98004, USA j annhucchen@gl oball teeneett.ccc... com
ABSTRACT
Transformer architectures have achieved great success in solving natural language tasks, which learn strong language representations from large-scale unlabeled texts. In this paper, we seek to go further beyond and explore a new logical inductive bias for better language representation learning. Logic reasoning is known as a formal methodology to reach answers from given knowledge and facts. Inspired by such view, we develop a novel neural architecture named FOLNet (First- Order Logic Network), to encode this new inductive bias. We construct a set of neural logic operators as learnable Horn clauses, which are further forward-chainel into fully differentiable neural architecture (FOLNet). Interestingly, we find that the self-attention module in transformers can be composed by two of our neural logic operators, which probably explains their strong reasoning performance. Our proposed FOLNet has the same input and output interfaces as other pretrained models and thus could be pretrained/finetuned by using similar losses. It also allows FOLNet to be used in a plug-and-play manner when replacing other pretrained models. With our logical inductive bias, the same set of "logic deduction skills" learned through pretraining are expected to be equally capable of solving diverse downstream tasks. For this reason, FOLNet learns language representations that have much stronger transfer capabilities. Experimental results on several language understanding tasks show that our pretrained FOLNet model outperforms the existing strong transformer-bar approaches
INTRODUCTION
Pretrained transformer models have achieved great success in solving natural language tasks, which learn strong language representations from large-scale unlabeled texts. The learned representations can be easily transferred to different downstream tasks by finetuning over limited amount of labeled
data (Radford al. 2018 Devlin et al. 2018 Lan et al. 2019 Liu et al. 2019 Yang et al. 2019).
They even exhibit strong zero-shot or few shot shot generalization capability without finetuning when further scaling up the model size (Radford et al. 2019 Brown al., 2020 Chowdhery et al.) (022). Besides large-scale models and training data, one important reason for the success 18 the strong relational inductive bias encoded in the transformer architecture (Vaswani al. (017); it effectively models the pairwise relations between tokens and use it to compute the language representations.
In this paper, seek go beyond the inductive bias in transformer models and explore inductive bias for better language representation learning. The main idea is to view the computation of language representations as a logic reasoning process; that the language representations are deduced via logic reasoning see-bb--ttep from the original discrete token sequences. Specifically, we treat the tokens the input sequence as the terms in logic programming, and treat their properties and relations as the predicates of different arities. Then, the final language representations are derived as the advanced properties and relations from the basic input properties and relations (e.g., token ids and relative distances). Most importantly, we require the construction of such deduction process to follow the principles of first-order logic, in order to encode such logical inductive bias.
logical
Following the above logical inductive bias, we derive principled neural architecture, named FOLNet (First-Orde Logic Network), for learning language representations. Specifically, we construct set
TThe code along with the pretrained model checkpoints will be relcased publicly.

---

Published as a conference paper at ICLR 2023
of neural logic operators learnable Horn clauses, which are further forward-chained into fully differentiable neural architecture. In particular, the FOLNet architecture consists of two interacting branches responsible for unary and binary relational reasoning, respectively. Interestingly, find that the self-attention mechanism can be constructed by two of our developed neural logic operators, and the entire transformer architecture can be understood newly discovered connection might partially explain the surprisingly strong reasoning performance of the transformer architecture (Wei et al. 2022 Lewkowyez al. 2022). As we will demonstrate in our experiments, such dual-branch architecture has several significant advantages that are essential for learning better language representations. Furthermore, we also establish a new unified understanding of different positional encoding strategies with our logical inductive bias. For instance, we find that the existing popular relative positional encoding can be constructed by the degenerated version of our two neural logic operators. More importantly, it also allows us to develop a new principled relative positional encoding that is simple yet quite effective in practice. Notably, our proposed FOLNet has the same input and output interfaces other pretrained transformer models (e.g.. BERT) and thus could be trained by using similar losses. It also allows FOLNet to be used in plug-and-play manner when replacing other pretrained models in solving downstream tasks. Our logical inductive bias assumes that the "logic deduction skills" are shared across all natural language tasks; that is, these skills learned during pretraining should be equally applicable to solving diverse downstream tasks. For this reason, FOLNet learns language representations that have much stronger transfer generalization capabilities. Experimental results on several language understanding tasks (GLUE, SQuAD 2.0 and FOLIO) show that our FOLNet model outperforms the transformer architecture by a arree-argin when they are pretrained using similar losses. The results clearly show that advantage of using the logical inductive bias for learning language representations.
single-branch version of FOLNet. This
2 LOGICAL INDUCTIVE BIAS FOR LANGUAGE REPRESENTATIONS
Natural language text can be viewed sequence of discrete symbols, and language representations learning considers the problem of mapping the discrete symbols into certain more computable forms. One widely used approach is distributed representation, which maps the discrete token ids into dense vectors (Mikolov et al.] 2013) Penniinnnnn et al. [2014] Peters et al. 2018). Many different functional forms, such as LSTM (Hcchretter & Schmidhuber] 1997). and more recently, transformer models (Vaswani et al. 2017). have been used to implement such mappings. They generally encode different kinds of inductive bias for modeling natural languages. For example, RNNs use the same set of model parameters to update the hidden states over time, which encodes cansllatio-invariannt over time (Battaglia et al. 2018). These forms of inductive bias continuously push the ttate-of-the-arts in solving natural language tasks. In this section, we introduce new form of inductive bias, named logical inductive bias, which will work together with distributed representations to design more effective representation mappings. Our main idea is to view the language representation mapping as a logic reasoning process; that is, the language representations are deduced step-by-step from the original discrete token sequences. Specifically, we treat the tokens in the input sequence as terms (or objects) in logic programming, and treat their properties and relations as predicates of different ariiee.. In light of logical inductive bias, the language representations that we seek to compute are the (advanced) properties and relations that can be deduced from these input (basic) properties and relations. Most importantly, we require the construction of such deduction process to follow the principles of first-order logic, in order to encode the logical inductive bias into the representation learning process. We now formulate the language representation learning as logic programming problem by adopting similar (logic programming) notations used in Evans & Grefenstette (2018).
We consider first-order logic system without function symbols, so that terms can only be variables or constant. They are used to represent general objects or a particular object of interest, respectively. In the context of language representation learning, we model each instance of text sequence x (of length T') as a collection of constants x = ( J. where each token T, is a constant (t 1, I,.... T". We use lower-case letters to denote constants and upper case for variables as in logic programming. For example, X is a variable to represent a general object (e.g., token).
Terms:
Atoms: For each term, we will define its properties and relations as an r-ary predicate p (XX....,,..), which takes the value of T (True) or F (False) depending tain property/relation regarding (X,,...,X,) holds or not. For example, whether the a token a takes the w-th id in the vocabulary is unary predicate TokenID,.()) for u 1, 1,..., V, where V
whether a cer-
2

---

Published as a conference paper at ICLR 2023
Language representations
Logic programming
FOLNet
Tokens 2; in the text sequence
Constant: 2tt
The argument 2; in tensor u (2ze)
Token ids, relative distances, etc Final langauge representation
Input (basic) atoms: Co = B
Input tensors: (uo(2), (2, y)
Deduced (advanced) atoms: C2
Output tensors: {ug uz (a. v]
Representation mapping (partial) Representation mapping (partial)
Modus Ponens using generic clause X
Neural logic operator: see Table
1-step deduction: C conve conv((
Forward pass: I-layer
Representation mapping (full)
L-step deduction: whwand-
3
Forward pass: L-layer
Table 1: Identification of language representation learning as a logic programming problem.
is the vocabulary size, and whether the distance between two tokens and is equal to d is a binary predicate Dista(a, b). for |d] < T. An atom is ground if it has no variables, e.g., the above TokenID,,(a) and Dista(a, b) are all ground atoms.
Clauses: The reasoning process is constructed by set of "if-then" clauses in the form of:
- P1 A... A Pm:
where Pi..... P... pm and q are the body atoms and head atoms, respectively, and A denotes conjunction (i.e.. logical AND). These atoms play the roles of premises and conclusions: pi..... Pm are true, then q is also true. Clauses of the above form are known as definite Horn clauses (Horn) 1951). We call a clause a ground rule all its variables are substituted by constants. For example, when applying a substitution 0 _ {a/X X, b/Y} to a clause q Y) < p(X, Y), we get a ground rule: q(a, b) < p(a, b). It can be viewed as applying general clause to particular instantiation.
Our objective is learn collection of clauses and compose them into mapping from input predicates (e.g., TokenID, (zx)) and RelDista 2,,) )) to language representations. Specifically, let R be a of clauses and ground(R) be the corresponding set of ground rules. We define the immediate consequences of applying the ground rules in ground to a set of ground atoms X' as
conR (X) = X' {gla p ...... Am ground (R), A A
(2)
can be understood as a set of ground atoms that can be deduced from X' together with X' itself. Given set of input ground atoms B,
repeatedly apply the ground rules in R for L steps:
C = conR (C--),,
Co = B and / I.
(3)
Then, Cr. is all the possible ground atoms (predicates) that can be deduced from B (including B itself) in steps. The above procedure is known as forward-chaining: deduces all the possible conclusions from the input premises (i.e., B3) by repeatedly applying (i.e.. chaining) clauses. If want to verify whether a predicate q' holds (i.e., can be entailed), it suffices to check if q' is in Cz. In language representation learning, we start, for example, from the following input (basic) atoms:
B [TokenI,, RelDista x,),... |t, I,..., T),
(4)
and deduce C₁, as the final representations by forward-chaining our (learned) clauses. For example, in solving an (extractive) question answering problem, whether a certain token is the beginning (or end) of the answer span is modeled as advanced deduced property of this token, i.c.,
AnswerEndsAt(nn C₁ [AnserStartsstt Ansaatretttttt..sseeeeeaeAAAAA
1,...,
(5)
In autoregressive language modeling, the advanced deduced property becomes the next token ids. Ta- ble summarizes the above identifications between language representations and logic programming. Next, we will develop a neural architecture to encode such logical inductive bias. Throughout the paper, we will use boldface letters to denote vectors (lowercase) and matrices (uppercase).
FOLNET: A NEURAL ARCHITECTURE WITH LOGICAL INDUCTIVE BIAS
In this section, we develop novel neural architecture, named First-Order Logic Network (FOLNet). which encodes the logical inductive bias presented Section 2 Specifically, we focus on: (i) how to represent the atoms as continuous vectors (Section 3.1. (ii) how to devise neural inference step that approximates (2) (Section (.2),, and (iii) how to forward-chand them into a fully dilleeeifferettiablee architecture based on 33) (Section 3.3). The overall neural architecture and its correspondence to the logical inductive bias are shown in Figure Next, we will discuss step-by by-step how we devise the architecture, its advantages, and also some important connections to existing approaches.
3

---

Published as a conference paper at ICLR 2023
H-co = B+ ar)
C =
C₁
C- fola + Pı A pm ground (i) Nimi
C-1)
(x,()))
LayerNorm
LayerNorm
Miser Ops
LayerNorm
XL
un(sr,y)
(2.(2.)))
LayerNorm
Mixer Ops
LayerNorrm
Boolean
LayerNorm
Figure 1: Overview of the FOLNet architecture and how it encodes the logical inductive bias. The neural logic operators model the clauses, which forward-chained into differentiable model. The "mixer ops" refer to the operators C, j, m, and t in Table 2 as they reduce over the object dimension.
3.1 VECTOR REPRESENTATIONS OF ATOMS
Recall that we use an r-ary ground atom Pa( xTy)) to characterize whether the d-th prop- erty/relation holds for tuple of tokens (I1,-.., where d = 1,..., D,. To overcome the difficulty of learning these discrete-valuee atoms, which takes values in (T,F). we introduce ua(zi,...... .y, E R its continuous representation and characterizes the extent to which the atom pa is true. For example, ProbLog (De Raedt et al. 2007 De Raedt Kimmig 2015 Fierens et al. 2012 Manhaeve et al. 2018), ud(-) gives the probability of the atom pz(-) being true. In this paper, consider u)(-) to be the logit of the corresponding atom, i.c., Pr(pa(-) T} 1/(1 + ) a larger value of u.(() implies a higher chance the atom Pa(-) being true. And we can also easily verify that the logit for the negated atom of pa(-) is simply -ua(-). Let u(r1,...,, (.......).)))).... E R D. be a Dy-ddimensional vector that collects Ue MM ........,,,, .r.)) as its d-th element. Then, u(zi,...,. Ty) will be continuous (logit) vector representation for D, atoms with arity r. For example, for input text sequence x = (II,.... ITT), we can use u(rz) to represent Dj (unary) properties for each token Ty, and use u F.,) ) characterize D:2 (binary) relations between any pair of tokens. Both the input (basic) properties/relations and the deduced (advanced) ones will be represented in the same logit vector space, where the deduction process will be carried out. For convenience, we may also directly refer to set of atoms by their logit vector representation 11--).
3.2 NEURAL MODUS PONENS INFERENCE
now develop set of neural operators for implementing the deduction step characterized (2) To begin with, we first introduce the Modus Ponens (MP) rule from first-order logic (Andrews) 2013
which states that if clause + A and statement A hold, then B is true:
+ [B + A and A}.
(6)
In the context of when choosing A P1 ... Pm and = the deduction in 2 can be viewed as applications of the MP inference using all the ground clauses in ground (R). In this paper, we restrict our focus to the setting where all the atoms have arity of either one or two. That is, we will only consider atoms of the form u(r:) and ux,,, (represented in their vector forms), respectively Then, we will need to develop the MP inference from set of atoms {v(a), v(a, 6)} to another set of atoms {u(x), u(x, } which can be categorized into the following four groups:
u(x) + Rav, v(a); u(x) RuB, v(a, b); u(x, v) + Reu, v(a); u(r, v) RaB ((a b)
where Ruy, Rus, Rev and RæB denote the sets of rules that deduce atoms of certain arity from another arity. For example, Ray defines a collection of clauses that derives binary (B) atoms from unary (U) atoms. We now proceed to model these clauses and their inference logit space. Given set of premise atoms ......,Ph, we consider N clauses of the following generic form:
- -Pm (
Qn
Pm
N,
mEM.,t
m.M..
*Generalizing our work to higher arities is relatively straighforward in principle but will lead to high computation complexities in practice. We leave such extension future work.
4

---

Published as a conference paper at ICLR 2023
Sym.
Ops.
Typing
B-dim.
R-dim.
Neural operator in logit space
Kernel act.
Remarks
b
bool
c- U
[b,(x)
(e Wa,(2) Kaalajva
Identity
cjoin
U
U B
Softmax.
j
join
U
B U
h
Wh, (2)
Kn(E, ()vs
Softmax.
Self-attention
m
mu
U <
B X
Wbr(2) (x)
5 Kula, (a,,
Softmax,
General RPE
a
assoc
B 4--
U x
h
(a 9) Kau (ar) (a.
(a
Identity
Self-attention
P
prod
B
X B
W
Wn (2. 9)
Identity
General RPE
trans
B
B
[z E. Ku(e, (a.
Softmax.
Table 2: List of all our neural logic operators with restricted kernels (see Appendix A.1 for our naming protocols). Note that each operator has a unique batching dimension (B-dim) and a unique reduction dimension (R-dim). The typing of the operator defines the arities of the kernel, the premise and the output atom. For example, U < B x U means the arities of the kernel, the input atom, and the output atom are 2, 1 and 1, respectively. We also list the activation functions that are used to compute the corresponding kernels, where the normalization dimension of Softmax is listed in its subscript.
where Qn is the head atom, - denotes logical negation (i.e., NOT), and Ma,+ and Mn,- are two subsets of M fl....; : M) with M M,,-- 0. Then, the logit vector for the head atoms {Qx} can be approximately inferred from the logit vector v of the premises {Pm] by a matrix multiplication followed by an (optional) elementwise nonlinear activation function (Appendix D):
u (Kv),
where K an x M kernel matrix that represents the clauses in (8) and (z) = In + 2c*) is the activation function. Notably, each row of K characterizes the conjunction pattern on the right-hand side of (8): a positive (negative) value of its (n, m m)-th element, Knm, means that m is more likely in Mn.+ (Mn.-) ) for the n-th clause. It follows that the kernels for Rov. RuB, Rao and Rae in 7 would be in the form of (x, a). KuB (r. a, b). K y, a) and Kaa y, a, b), respectively, which are
D1 D1, D1 x D2, D2 D2 and D2 D2 matrices. And (9) would become matrix multiplications between them and their corresponding premises (i.e., v(a). v(a, b). v(a) or v(a, b)) followed by a
summation over a and 6 whoever appear therein (see (30) (33 in Appendix D). Finally, the activation function af) can be dropped when "t" is replaced with
where A = B iff A < B and B < A.
One major limitation of directly implementing (9) for the inference rules in 77 the high memory and computation costs. For example, the kernel KBs (r, a, b) needs O ( to store its value. And the MP inference (.. which now becomes u(r, y) Ea,b K(x, y, a, (a, b), also has a computation complexity of (D)TTA) Therefore, we have to restrict the size of the kernel and reduce the overall complexity by using different methods, such as sharing the values of se (ær, y, a, b) across different dimensions. We now provide systematic approach based on the following principles:
1. We restrict all the kernels to be in the form of (K(w), K(w, )) i.e., the arity r 1, 2. We pick one reduction dimension and one batching dimension in the matrix multiplication.
With the above assumption, we factor the predicate dimensions of unary kernels and unary atoms so that (a(w) = Kne(w) and ua(x) = (z), where d = (h -1)S + s with h = 1.... H and .., S. This is inspired by the multi-head attention mechanism (Vaswani et al. 2017). where h is akin to the head index, H is the numher of heads and 5 is the size of the head. Then, we enumerate all possible neural logic operators that can compatibly multiply a kernel from (K(w), K(w, v)} with a premise from (v(a), v(a, b)} by properly choosing different reduction and the batching dimensions. With this, we list the resulting neural logic operators for each typing in Table 2 which are further discussed in Appendix A. for their different roles in (restricted) Modus Ponens reasoning.
Connection with transformers Interestingly, we find that the j-operator and the a-operator share similar forms as the self-attention mechanism in transformers, where the a-operator computes the self- attention scores and the j-operator performs the self-attention operation. Furthermore, the m-operator and the p-operator indeed generalize the existing relative positional encodings developed in (Shaw ett al. 2018), which are widely used in different transformer variants, such as in T5 models (Raffel et al. 2020). Specifically, we show in Appendix E that by making V,, y) instance-independees the p p-operator computes the second term in equation (5) of (Shaw et al. (2018), where V (z. y) play the role of ats. And by setting vs( a) instance-inded indeeeeden,,,,, the m-operator computes the second
5

---

Published as a conference paper at ICLR 2023
term in equation (3) of Shaw et al. (2018), where v, a) plays the role of a That is, under such degenerated settings, these two operators will compose the relative positional encoding developed therein. Note that their a and a static learnable embeddings, while our (w, y) and vz a) are dynamically computed for each instance (as we will discuss in Section [.3]. Therefore, our m-op and p-op can also be viewed as more adaptive relative positional encoding, whose advantages will be further demonstrated in our experiments (Section 4).
3.3 FORWARD-CHAINING AND DIFFERENTIABLE LEARNING
Note that, in general, a logic operator takes a kernel K and a premise V to infer an outcome u (Table 2). Specifically, it "neuralizes" the logic deduction in (2) for a particular typing (c.g.. U < U). Applying all the neural logic operators amounts to have a full execution of 2). which is one recursion step in (3) that maps a set of (1/-1()) (z), 1 y)} into (z), (z y) Therefore, we can naturally forward -chin L stages of them together to create a fully- sully-differentiable architecture that models the reasoning chain (3). Figure depicts such a forward-chaining process and also how encodes the logical inductive bias described in Section 2 One remaining problem is how to obtain the kernels K-)) and the premises v(.) from our FOLNet architecture in Figure ! We do this simply by applying two linear projections (one for the premise and one for the kernel) to the previous layer's output {u, (z), (x, y))- For the kernel, we may further apply an activation function after the linear projection for each operator). In other words, we parameterize the kernels K(-) and the premises v(-) by (the intermediate deduction results of) FOLNet itself. This based on the observation that clauses are themselves predicates since A <- B is defined as A V -B (Andrews disjunction (logical OR). We now describe the input and the output of FOLNet in Figure [T At the input, we can encode the discrete token ids for token Te into vectors of the form (x(2)) by standard embedding lookup. Likewise, we also convert the (discrete) relative distance between two tokens Ie and into a vector of the form representations of the base atoms B and fed into the FOLNet model (Figure D After L layers (i.c., L steps of deduction), the output {ur (zze,, uy (1te -r) Jt.r becomes the vector representations of Cr. in (3). which used as the final language representations. Therefore, our FOLNet model has the same input-output interface as other transformer models and can be used in plug-and aod--an- manner for solving downstream tasks. Because of this, our model can also be pretrained over large-scale texts using the same losses (e.g., MLM, NSP, SOP, etc) as other encoder-only models see Appendix [A.4] for how to compute these losses from {ur, (Zte, uy (I+, -rr],,r- Our model can also be extended to the decoder-only and the encoder-decoder versions by slightly modifying the neural logic operators (see Appendix AA.)). which can then be pretrained to predict the next words auto-regressively, We will conelude this section by discussing several important properties of FOLNet architecture.
compute the kernel (see Table 2 for the list of kernel activation functions
2013), where V denotes
The (xn( Uo( z>)]=r will be used as vector
The dual-branch architecture Note that the FOLNet model in Figure 1 has two branches: (i) unary predicate branch for reasoning over (z),, and (ii) a binary predicate branch for reasoning over ux(x,)) This is in sharp contrast to the single-branch architecture of the transformer models (Figure il Appendix A.2 We further note that when FOLNet is only loaded with j-operator and a-operator, degenerates into dual-branch variant of the transformer architecture. In our experiments, we will show that, even in such degenerated setting, FOLNet still outperforms transformer. This is probably because the binary predicate branch explicitly maintains the pairwise relations uz (2: y) throughout the reasoning process. In addition, the explicit binary predicate branch also allows us to directly input the relative distance knowledge into the model without performing the less-intuitive operations as in existing RPEs (Shaw et al. 2018). In experiments in Section 4. we will demonstrate the ad vantagg such a simple yet effective approach for consuming the relative positional information, along with some other advantages of the dual-branch architecture of FOLNet.
EXPERIMENTS
4.1 EXPERIMENTAL SETTINGS
We now evaluate FOLNet models under different settings and seek to answer the following question: Can the neural architecture (FOLNet) that encodes the logical inductive bias learn better language representations than the transformer models? ? To this end, we need to eliminate other
6

---

Published as a conference paper at ICLR 2023
Model
Params
PE
D2
Less
MNLI-m/mm
QQP
QNLI
SST-2
CoLA
STS-B
MRPC
RTE
Avg
1 BERT
H10
APF:
MI M.NS
84.5/-
91.3
91.7
93.2
58.9
89.5
87.3
68.6
83.1
2
BERT (ours)
110M
APE
MLM.NSP
83,9/84.1
90.9
88.2
92.6
61.5
89.2
88.2
66.8
82.7
Net: ja
110M
APR
MIMSN
84.9/84.5
91.1
91.6
92.2
61,1
89.6
89.5
72.2
84.0
4
FOLNet: ja
109M
RPE
12
MLMNNSP
85.0/84.9
91.4
91.4
93.5
63.8
90.1
89.9
72.9
84.7
FOLNet: ja
110M
RPE"
12
MLM.NSP
84.7/84.9
91.4
91.5
93.8
63.4
89.8
90.6
72.6
84.7
FOLNet: jm.ap
123M
RPE
12
MLM.NSP
85.7/85.3
91.6
91.8
93.8
65.7
90.4
91.0
73.5
85.4
FOLNot
109M
RPE
16
MLM.NSP
85.2/85.2
91.3
91.8
93.7
64.1
89.9
89.3
71.8
84.6
8
FOLNet: ja
109M
RPE
32
MLM.NSP
85.8/8577
91.4
92.0
93.7
64.2
90.0
90.5
71,8
84.9
FOLNet: ja
110M
MLMINSP
85.7/85.5
91.4
91.8
63.5
90.2
10 FOLNet: jat
110M
RPE
64
MLM.NSP
86.7/86.6
91.6
92.8
93.1
63.6
91.1
89.9
80.9
86.2
FOLNot: jatp
117M
RPE
64
MLM.NSP
87.4/87.4
91.9
93.3
94.0
62.9
91.4
81.6
86.7
12 FOLNet: jm.atp
124M
RPE
64
MLM.NSP
88.1/87.6
91.7
93,9
94,2
64.7
91.2
91.4
83.2
87.3
FOLNeE: jin.alp
138M
RPE
64
88.2/87.9
91.9
94.1
66.9
91.6
91.5
87.7
14 FOLNet: jmc.arp
137M
RPE
12
MI MNP
85.9/86.3
91.6
92.7
93.6
63.4
90.5
91.0
75.8
85.6
15 FOLNet: jms.atp
138M
RPB
64
MOM
88.3/87.9
91.8
94.2
94.7
65.6
91.1
91.1
83.2
87.5
Table 3: Analysis of FOLNet the development sets of GLUE benchmark. All the results are medians of five random seeds. From top to bottom, the first block shows the advantage of dual-branch architecture and compares
new positional encoding to others, the second block analyzes the influence of the binary predicate dimension, the third block performs ablation study of all the logic operators, and the last block shows the results of other pretraining losses for FOLNet. APE stands for absolute positional encoding, RPE* denotes the relative positional encoding used by T5 (Shaw et al.) 2018 Raffel al. 2020), and RPE means our proposed relative positional encoding. We have also pretrained a BERT (base) model (line #2) by using the same settings as FOLNet for fair comparison.
confouding factors and make sure the only difference lies in the model architecture itself. First, we choose pretrain our FOLNet model using the widely used masked language modeling (MLM) loss (Devlin et al. 2018), and add an extra loss of either the next sentence prediction (NSP) (Devlin et al. 2018) or sentence-order prediction (SOP) (Lan al. 2019 Many different variants of widely used encoder-only transformer models such as BERT, RoBERTa, ALBERT, DeBERTa and Megatron LM are pretrained with these losses. Therefore, we will also use these models as our primary baselines. Although there could be other more efficient pretraining losses such as the ones in (Bao et al.) 2020
Clark et al., 2020 Yang et al. 2019 Meng et al. 202I). we believe that developing new model architecture with a better inductive bias is an orthogonal line of research. Therefore, we leave the exploration of other pretraining loss for FOLNet as future work. In addition, we consider two settings of pretraining dataset: (i) Wikipedia + BookCorpus (Zhu et al., 2015) (16GB in texts) and (ii) a larger set of 160GB texts consisting of Wikipedia, BookCorpus2, OpenWeTTeeee (extracted from the Pile dataset (Gao et al.) (020)). We use the BERT tokenizer with 32,764 768 uncased BPE vocabulary (Sennrich et 2016) throughout our experiments 3) We consider FOLNet models of two different sizes: FOLNetgase and FOLNetLagge which are comparable in size to the base (e.g.. BERT and large models (c.g.. BERT Large) in literature. Finally, the FOLNet model will always be pretrained with sequence length of 128 tokens, although it will be evaluated on different downstream tasks with longer sequence lengths (e.g., 384 or 512). For evaluation, we consider three benchmarks: GLUE (Wang et al. 2019), SQuAD 2.0 (Rajpurkar al., (0066),, and FOLIO (Han al., (0022). where we finetune our pretrained models on each individual task separately for evaluation. More details about these downstream tasks and param--rrseeters can be found in Appendix
and Pile-CC
4.2 ANALYSIS OF THE FOLNET ARCHITECTURE
We begin with in-depth analysis of FOLNet to demonstrate its advantage over the transformer architecture. To this end, we first pretrain our FOLNelgase model on the dataset of Wikipedia and BookCorpus, which is the same as the one used by BERT. And we further use MLM and NSP as pretraining losses to make consistent with BERT. We analyze FOLNet from different aspects on GLUE benchmark and report the results in Table We now proceed to discuss the results below.
3 "Although Liu et (019)) pointed out that it would be more ideal to use the byte-level BPE for preprocessing the much larger 160GB texts, we use the same BERT tokenizer (based on character-level BPE) to process the
160GB text to simplify our logistics, and it already demonstrates the strong performance of our models, Using the hyte-level BPF tokenizer with larger 50K vocabulary in (Liu et al. 2019 Radford et al. 2019 Brown al. 2020) may further improve our FOL.eet models that are pretrained on the 166G texts.
7

---

Published as a conference paper at ICLR 2023
GLUE
SQuAD 2.0
FOLIO
Model
Params
MNI.Im/mm
QQP
QNLI
SST-2
CoLA
STS-B
MRPC.
RTE
EM
F1
Acc
BERT Bane
110M
84.5/-
91.3
87.3
68.6
76.3
RoRERTiBee
10M
85.8/85.5
91.3
92.0
93.7
60.1
88.5
87.3
68.2
83.3
77.7
80.5
DeBERTBBue
134M
86.3/86.2
79.3
82.5
FOLNetm
138M
88.2/87.9
91.9
94.1
94.5
66.9
91.6
91.5
83,5
87.7
84.7
87.9
64.2
BERT Large
340M
86.6/-
91.3
92.3
93.2
60.6
90.0
88.0
70.4
84.1
79.0
81.8
62.3
RoBERTans
125M
87.6/-
91.9
92.8
90.2
78.7
83.7
DeBERTigee
134M
88.8/88.5
83.1
86.2
FOI Fef.Ne
138M
89.4/89.7
88.6
RoBERTaLane
356M
90.2/90.2
92.2
94.7
96.4
68.0
92.4
90.9
86.6
88.9
86.5
89.4
67.7
DeBERTa arga
384M
91.1/91.1
92.3
95.3
96.8
70.5
88.0
90.7
ALBERTxNi.
235M
90:4/-
92.0
95.2
96.8
68.7
92.7
90.2
88.1
89.3
87.2
89.9
ALBERT
235M
90.8/-
92.2
95.3
96.9
71.4
93.0
90.9
89.2
89.9
87.4
9n.2
437M
91.2/91.3
92.5
95.8
96.8
71.5
92.2
93.5
91.1
90.6
88.5
91.5
70.6
Megatren:com
1.3B
90.9/91.0
92.6
87.1
90.2
Megatromy un
91.4/9144
92.7
91.2
Table 4: Overall results on the development sets of GLUE,, SQuAD 2.0 and FOLIO. The upper block (separated by the solid line) of the table shows the results of the models pretrained on Wikipedia + BookCorpus (16GB), and the lower block are the models pretrained on extended data (160GB). We use dashed lines to separate models of different sizes within each block. All the results are medians of five random seeds. The baseline results of FOLIO are provided by the authors of Han et al. 2022). Here we use ALBERT xXL to refer to the ALBERT model pretrained by 1M steps and use LBBETxxXXL. to refer to the ALBERTxx1 xxI. modeled pretrained by 1.5M steps.
The advantage of the dual-branch architecture Recall that when FOLNet only has the join and assoc operators, it can be viewed as dual-branch version of the transformer architecture. To have fair comparison, we pretrain FOLNet model with the same absolute positional encoding (APE) as BERT (line #3 of Table 3). Note that, even in such overly degenerated case, FOLNet still noticeably outperforms BERT (RPE) (line #4 of Table we will outperform BERT by 2 points on average. Notably, it achieves on par (or slightly better) average performance compared to the one with T5 relative positional encoding (RPE* in line #5). As we discussed in carlier section, the T5 RPE are degenerated version of our mu and prod operators. Line #6 of Table show that adding mu and prod operators to the FOLNet would further boost the performance by a noticeable amount.
average. When equipping FOLNet with our new relative positional encoding
The benefits of a larger D2 We can see (lines #7-9 of Table that increasing the dimension D2 will steadily improves the performance. As we will reveal soon, when FOLNet fully loaded with all the operators, having a larger D2 is essential to unleash their full power; that is, the performance improvement from increasing D, would be even larger for fully-loaded FOLNet.
Contribution of the logic operators We now analyze the contributions of the logic operators by adding them one-by-one into FOLNet until being fully loaded. We see from line #9 to line #13 Table that this drastically improves the average performance. In line #14, we evaluate a fully-loaded FOLNet with D2 decreased from 64 to 12, which shows significant performance drop. This confirms the importance of having a relatively large D2 in order to store the deduced relations.
4.3 OVERALL PERFORMANCE
Having closely examined various aspects of FOLNet architecture, we now proceed to evaluate it comprehensively on three benchmarks (GLUE, SQuAD 2.0, and FOLIO). We will also examine its performance when we further scale up the pretraining data size and model size. To pretrain the model on a larger (160GB) dataset, we find it more efficient to generate the pretraining data with SOP losses. This is because NSP losses require begin with, we verify the performance by pretraining a FOLNetgase with SOP loss on Wikipedia and BookCorpus. The result (line #15 in Table 3) shows that it could slightly degrade the performance compared to the one with NSP (line #13). However, this is relatively tolerable given its convenience when pretraining large corpus. Therefore, we will replace NSP with SOP when pretraining
to sample negative sentences from another document. To
8

---

Published as a conference paper at ICLR 2023
FOLNetgase and Lree on the 160GB dataset. We show results the GLUE benchmark in Table [ and compare them with other baseline methods. Observe that our FOLNetBase models pretrained both 16GB data and 160GB significantly outperform other transformer- based models on all three benchmarks. Our FOLNetBasee pretrained on 16GB data outperforms BERT Large model that is 3x larger in model size. In addition, it even outperforms RoBERTag.se that is pretrained data. When pretraining our FOLNelBase model on 160G data, we even surpass the RoBERTa model (89.3 vs 88.9) on GLUE benchmark. Likewise, our FOLNetLarge model also significantly outperforms all other baselines. It even outperforms ALBERT XXL+ that is pretrained by 1.5M steps (i.e.. 50% more tokens during pretraining). Notably, our FOLNet model achieves comparable performance as the Megatrona.99 model on both GLUE and SQuAD 2.0 benchmark, which is 10 times larger than our model. Furthermore, although our FOLNet model is not designed for solving reasoning problem (but incorporating reasoning as an inductive bias at the token-level), our model still consistently demonstrates stronger first-order logic reasoning capability on FOLIO task (c.g., +3.9 over RoBERTaLage) Finally, we would like to highlight that all the FOLNetBase and FOLNetLagee models are pretrained with sequence length of 128, in contrast to 512 as in other baselines. However, they are evaluated on GLUE, SQuAD 2.0 and FOLIO with sequence length of 128, 384 and 512, respectively. In particular, by finetuning with merely 1,004 training examples on FOLIO, it is able to generalize to much longer sequences (512), which have never been seen during pretraining.
160GB
5 RELATED WORKS
Transformer language models There have been a long line of research on neural language models since Bengio et al. (2000). Recently, it has achieved great success by exploring different variants of pretrained transtormer models /Vaswani et al. 2017) for solving downstream language tasks, such as with finetuning (Radford et al. 2018: Devlin et al. 2018 Lan et al. 2019; Liu et al. [019 Yang et al.
2019) or with zero/few-shor shot learning using large language models (Radford et al. 2209 Brown et al.
2020: Chowdhery et al. 2022). Another line of active research focuses on developing more effective
pretraiming losses Yang et al. 2019; Clark et al. 2020 Bao et al., 2020 Tay et all (0022) beyond the widely used autoregressive or masked language modeling objectives. There have been limited works on developing new neural architectures for learning better language representations. In this paper,
seek to move in this direction and develop a new neural architecture based on logical inductive bias.
Logic programming and neural reasoning Our logical inductive bias is inspired by logic pro- grammings (Hom) 1951; De Raedt et al., 2007 De Raedt & Kimmig, 2015 Fierens et al. 2012 Manhaeve et al. 2018 8) and inductive logic programming (Evans & Grefenstette 2018 Muggleton
1991; Friedman et al. 1999). Different from these works, we do not directly work on reasoning problems. Instead, we seek to encode the logical inductive bias into the neural model to learn better language representations. Another line related works focuses on developing neural models that can perform reasoning in broad sense. For example, different variants of memory augmented networks are developed (Le et al. 2020; Santoro et al. 2018) Graves et al. 2014 2016 Santoro et al., 2016 Lc et al. 2019), which augment control network with memory units and the associated neural read/ wead/write modules. Besides, RRokkäächel & Riedel (2017) and Minervini et al. (0000) consider the problem of proving structured queries to knowledge base, by constructing differentiable neural networks via mackward-chainingg [Bergen et al.] (2021) develop a triangular attention to deduce relations from other transformer with single relational branch. These methods are effective in solving (relatively small-scale) reasoning tasks. However, it remains unclear whether they can be effectively pretrained on large-scale texts to solve diverse downstream natural language tasks.
relations, which can be viewed
6 CONCLUSION
We introduce a novel logical inductive bias, which treats language representation learning as logic programming problem. We then develop a fully-differentiable neural architecture (FOLNet) that effectively encodes this inductive bias by forward-chaining a rich set of neural logic operators. The proposed FOLNet architecture has the same input-output interface as the transformer models and can be pretrained over large-scale text data. Experimental results demonstrate that the FOLNet architecture significantly outperforms different variants of transformer models, and has many inherent advantages due to the new dual-branch architecture along with its rich set of neural logic operators.
9

---

Published as a conference paper at ICLR 2023
REFERENCES
Peter B Andrews. An introduction to mathematical logic and type theory: to truth through proof. volume 27. Springer Science & Business Media, 2013.
Hangbo Bao, Li Dong, Furu Wei, Wenhui Wang. Nan Yang, Xiaodong Liu, Yu Wang, Jianfeng Gao, Songhao Piao, Ming Zhou, et al. Unilmv2: Pseudo-masker language models for unified language model pre-training. In International Conference on Machine Learning, pp. 642-652. PMLR, 2020.
Peter W Battaglia, Jessica B Hamrick, Vietor Bapst, Alvaro Sanchez-Gonzaler Vinicius Zambaldi, Mateusz Malinowski, Andrea Tacchetti, David Raposo, Adam Santoro, Ryan Faulkner, et al. Relational inductive biases, deep learning, and graph networks. arXiv preprint arXiv: 180.002600. 2018.
Yoshua Bengio, Réjean Ducharme, and Pascal Vincent. A neural probabilistic language model. In T. Leen, T. Dietterich, and V. Tresp (eds.), Advances in Neural Information Processing Systems, volume 13. MIT Press, 2000. URL ht
Luisa Bentivogli, Peter Clark, Ido Dagan, and Danilo Giampiccolo. The fifth pascal recognizing textual entailment challenge. In TAC, 2009.
Leon Bergen, Timothy O' O'Donnell, and Dzmitry Bahdanau. Systematic generalization with edge transformers. Advances in Neural Information Processing Systems, 34:
2021.
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot leamners Advances in neural information processing systems, 33:1877-1901, 2020. 2020.
Daniel Cer, Mona Diab, Eneko Agirre, lñigo Lopez-Gazpio, and Lucia Specia. SemEval-2011 task 1: Semantic textual similarity multilingual and crosslingual focused evaluation. In Proceedings of the 1 11th International Workshop Semantic Evaluation (SemEvall22017) pp. 1-14, Vancouver, Canada, August 2017. 7. Association for Computational Linguistics. doi: 10. 10.18663/11SS17-2000
URL https:://acaatttottet wtt.://toooooo..oo11110000000000
Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann, et al. Palm: Scaling language modeling with pathways. arXiv preprint arXiv:2204.02.11 2022.
Kevin Clark, Minh-Thang Luong, Quoe V Le, and Christopher 1) Manning. Electra: Pre-training text encoders as discriminators rather than generators. arXiv preprint arXiv:2003. 2020.
Ido Dagan, Oren Glickman, and Bernardo Magnini. The pascal recognising textual entailment challenge. In Machine learning challenges workshop, pp. 177-190. Springer, 2005.
Luc De Raedt and Angelika Kimmig. Probabilistic (logic) programming concepts. Machine Learning, 100(1):5 47, 2015.
Lue De Raedt, Angelika Kimmig, and Hannu Toivonen. Problog: probabilistic prolog and its application in link discovery. In IJCAI, volume 7, pp. 2462-2467. Hyderabad, 2007.
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of NAACL-HILT, pp. 4171- 4186. 2018.
William B. Dolan and Chris Brockett. Automatically constructing a corpus of sentential paraphrases. In Proceedings of the Third International Workshop on Paraphrasing (IWP2005), 2005. URL
Richard Evans and Edward Grefenstette. Learning explanatory rules from noisy data. Journal of
Artificial Intelligence Research, 61:1-64, 2018.
Daan Fierens, Guy Van den Broeck, Maurice Bruynooghe, and Luc De Raedt. Constraints for probabilistic logic programming. Proceedings of the NIPS probabilistic programming workshop. pp. 1-4, 2012.
10

---

Published as a conference paper at ICLR 2023
Nir Friedman, Lise Getoor, Daphne Koller, and Avi Pfeffer. Learning probabilissicc relational models. In Proc. IJCAI, 1999.
Leo Gao, Stella Biderman, Sid Black, Laurence Golding, Travis Hoppe, Charles Foster, Jason Phang, Horace He, Anish Thite, Noa Nabeshima, et al. The pile: An 800gb dataset of diverse text for language modeling. arXiv preprint SKKi:2201.00022 2020.
Tianyu Gao, Adam Fisch, and Danqi Chen. Making pre-trained language models better few-shot learners. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume I: Long Papers). pp. 3816-3830, 2021.
Danilo Giampiccolo, Bernardo Magnini, Ido Dagan, and Bill Dolan. The third PASCAL recognizing textual entailment challenge. In Proceedings of the ACL-PASCAL Workshop on Textual Entailment and Paraphrasing, pp. 9, Prague, June 2007. Association for Computational Linguistics. URL
htttp:://aclanthhr oogy org/ wwtp://///00000 NO7-1401
Alex Graves, Greg Wayne, and Ivo Danihelka. arXiv:1410.5401,220 2014.
Neural turing machines. arXiv preprint
Alex Graves, Greg Wayne, Malcolm Reynolds, Tim Harley, Ivo Danihelka, Agnieszka Grabska- Barwiñsska Sergio Gómez Colmenarejo, Edward Grefenstette, Tiago Ramalho, John Agapiou, et al. Hybrid computing using a neural network with dynamic external memory. Nature, 538(7626): 471-476, 2016.
Bar Haim, Ido Dagan, Bill Dolan, Lisa Ferro, Danilo Giampiccolo, Bernardo Magnini, and Idan Szpektor. The second pascal recognising textual entailment challenge. In Proceedings of the Second PASCAL Challenges Workshop on Recognising Textual Entailment, volume 2006.
Simeng Han, Hailey Schoelkopf. Yilun Zhao, Zhenting Qi, Martin Riddell, Luke Benson, Lucy Sun, Ekaterina Zubova, Yujie Qiao, Matthew Burtell, et al. Folio: Natural language reasoning with first-order logic. arXiv preprint arXiv:2209. 008400 2022.
Sepp Hochreiter and Jürgen Schmidhuber. Long short-term memory. Neural computation, 9())
1735-1780, 1997.
Alfred Horn. On sentences which are true of direct unions of algebrasl. The Journal of Symbolic Logic, 16(1):14-21, 1951
Shankar Iyer, Nikhil Dandekar, and Kornl Csernai. First quora dataset release: Question
pairs. In Quora, January 2017. URL https://www hwtp://wwww..w../oo.o////// com/ // quoradata/
Zhenzhong Lan, Mingda Chen, Sebastian Goodman, Kevin Gimpel, Piyush Sharma, and Radu Soricut. Albert: A lite bert for self-supervised learning of language representations. arXiv preprint arXiv: 1 1942, 2019.
Hung Le, Truyen Tran, and Svetha Venkatesh. Neural stored-program memory. In International Conference on Learning Representations, 2019.
Hung Le, Truyen Tran, and Svetha Venkatesh. Self-attentive associative memory. In International Conference on Machine Learning, pp. 5691. PMLR, 2020.
Hector Levesque, Ernest Davis, and Leora Morgenstern. The winograd schema challenge. In Thirteenth international conference on the principles of knowledge representation and reasoning, 2012.
Aitor Lewkowycz, Anders Andreassen, David Dohan, Ethan Dyer, Henryk Michalewski, Vinay Ramasesh, Ambrose Slone, Cem Anil, Imanol Schlag, Thco Gutman-Sol Yuhuai Wu, Behnam Neyshabur, Guy Gur-Ari, and Vedant Misra. Solving quantitative reasoning problems with language models, 2022. URL http:///....tttrea///aaaoooo/oo.
2206. 14858
Percy Liang. Lambda dependeency composstional semantics. CoRR, abs/ 1309,4408, 2013. URL

---

Published as a conference paper at ICLR 2023
Peter Liu, Mohammad Saleh, Etienne Pot, Ben Goodrich, Ryan Sepassi, Lukasz Kaiser, and Noam Shazeer. Generating wikipedia by summarizing long sequences. In International Conference on Learning Representations, 2018.
Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. Roberta: A robustly optimized bert pretraining approach, arXiv preprint arXiv: / 1692, 2019.
Robin Manhaeve, Sebastijan Dumancic, Angelika Kimmig, Thomas Demeester, and Lue De Raedt. Deepproblog: Neural probabilistic logic programming. Advances Neural Information Processing Systems, 31, 2018.
Yu Meng, Chenyan Xiong, Payal Bajaj. Paul Bennett, Jiawei Han. Xia Song, et al. Coco-Im: Correcting and contrasting text sequences for language model pretraining. Advances in Neural Information Processing Systems, 34:23102-23114 2021.
Paulius Micikevicius, Sharan Narang. Jonah Alben, Gregory Diamos, Erich Elsen, David Garcia, Boris Ginsburg, Michael Houston, Oleksii Kuchaiev, Ganesh Venkatesh, and Hao Wu. Mixed precision training. In International Conference Learning Representations, 2018. URL https
7/openreview net htte:/foommmm=rrlssssssee
Tomas Mikolov, Ilya Sutskever, Kai Chen, Greg S Corrado, and Jeff Dean. Distributed representations of words and phrases and their compositionality. Advances in neural information processing systems, 26, 2013.
Pasquale Minervini, Matko Bošnjak, Tim Rocktäschel, Sebastian Riedel, and Edward Grefenstette. Differentiable reasoning on large knowledge bases and natural language. In Proceedings of the AAAI conference on artificial intelligence, pp. 5182-5190, 2020.
Stephen Muggleton. Inductive logic programming. New Gener. Comput., 844)2295-318 1991.
Stephen Muggleton. Stochastic logic programs. Advances in Inductive Logic Programming, 32: 254 264, 1996.
Nvidia. A pytorch extension: Tools for easy mixed precision and distributed training in pytorch. Available Online: http:://tttcs///ttttbbo//o///d/t
2019.
Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, et al. Pytorch: An imperative style, high-pent peghorerffrmance deep learning library. Advances in neural information processing systems, 32, 2019.
Jeffrey Pennington, Richard Socher, and Christopher Manning. Glove: Global vectors for word representation. In Proceedings of the 2014 conference on empirical methods in natural language processing (EMNLP), pp. 1532,1543, 2014.
Matthew E. Peters, Mark Neumann, Mohit lyyer, Matt Gardner, Christopher Clark, Kenton Lee. and Luke Zettlemoyer. Deep contextualized word representations. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers), pp. 2227-2237, New Orleans, Louisiana, June 2018. Association for Computational Linguistics. doi: 10. 16.18653/11/N18- 1202. URL https
7/acl antho I bogy..rrg/ae
1202
Alec Radford, Karthik Narasimhan, Tim Salimans, Ilya Sutskever, al. Improving language understanding by generative pre-training. OpenAl, 2018.
Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, llya Sutskever, et al. Language models are unsupervised multitask learners. OpenAl blog, 1(8):9, 2019.
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, Peter J Liu, et al. Exploring the limits of transfer learning with a unified text-to-text transformer. J. Mach. Learn. Res., 21(140):1-67, 2020.
12

---

Published as a conference paper at ICLR 2023
Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang. SQuAD: 100,000+ questions for machine comprehension of text. In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, pp. 2883-2392, Austin, Texas, November 2016a. Association for Computational Linguistics. doi: 10. (0.18633/11/D1-- 1264. URL htt.h::t///et.. antho logy
arg/D16-126h
Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang. Squad: 100,000+ questions for machine comprehension of text. In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, pp. 238322392, 2016b.
Tim Rocktäschel and Sebastian Riedel. End-to-end differentiable proving. Advances in neural information processing systems, 30, 2017.
Adam Santoro, Sergey Bartunov, Matthew Botvinick, Daan Wierstra, and Timothy Lillicrap. Meta- learning with mummory-augmented neural networks. In International conference machine learning, pp. 1842-1850. PMLR, 2016.
Adam Santoro, Ryan Faulkner, David Raposo, Jack Rae, Mike Chrzanowski, Theophane Weher, Daan Wierstra, Oriol Vinyals, Razvan Pascanu, and Timothy Lillicrap. Relational recurrent neural networks. Advances in Neural Information Processing Systems, 31:7299-7310, 2018.
Rico Sennrich, Barry Haddow, and Alexandra Birch. Neural machine translation of rare words with subword units. In 54th Annual Meeting of the Association for Computational Linguistics, pp. 1715-1725. Association for Computational Linguistics (ACL), 2016.
Peter Shaw, Jakob Uszkoreit, and Ashish Vaswani. Self-attention with relative posiioon represstttttsssnnnnnnn.... In NAACI-HLI (2), pp. 464 468, 2018. URL httpp:://ccccc ho logy. info/paperea
718-2074/n18-2077
Koustuv Sinha, Shagun Sodhani, Jin Dong, Joelle Pineau, and William I. Hamilton. CLUTRR: diagnostic benchmark for inductive reasoning from text. In Proceedings of the Conference on Empirical Methods Natural Language Processing (EMNLP), pp. 4506-4511 2019.
Richard Socher, Alex Perelygin, Jean Wu, Jason Chuang, Christopher D Manning, Andrew Ng, and Christopher Potts. Recursive deep models for semantic compositionality over sentiment treebank. In Proceedings of the 2013 conference on empirical methods in natural language processing, pp. 1631-1642, 2013.
Yi Tay, Mostafa Dehghami, Vinh Q Tran, Xavier Garcia, Dara Bahri, Tal Schuster, Huaixiu Steven Zheng, Neil Houlsby, and Donald Metzler. Unifying language learning paradigms. arXiv preprint arXiv:2205.05133, 2022.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention all you need. Advances in neural information processing systems, 30, 2017.
Petar Veličković, Guillem Cucurull, Arantxa Casanova, Adriana Romero, Pietro Liò, and Yoshua Bengio. Graph attention networks. In International Conference on Learning Representations. 2018.
Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R Bowman. Glue: A multi-task benchmark and analysis platform for natural language understanding, In 7th International Conference on Learning Representations, ICLR 2019, 2019.
Alex Warstadt, Amanpreet Singh, and Samuel Bowman. Neural network acceptability judgments. Transactions of the Association for Computational Linguistics, 7:625-641, 2019.
Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Ed Chi, Quoe Le, and Denny Zhou Chain of thought prompting elicits reasoning in large language models. arXiv preprint arXiv:2201.1 11903, 2022.
13

---

Published as a conference paper at ICLR 2023
Adina Williams, Nikita Nangia, and Samuel Bowman. A broad-coverage challenge corpus for sentence understanding through inference. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguisties: Human Lan- guage Techmologies, Volume (Long Pupers), pp. 1112-1122, New Orleans, Louisiana, June 2018. Association for Computational Linguistics. doi: 10.18653/11/118-10001 https lww.lannhooog..oog/ooooooooooooe
Zhilin Yang, Zihang Dai, Yiming Yang, Jaime Carbonell, Russ R Salakhutdinov, and Quoc V Le. XI.Net: Generalized autoregressive pretraining for language understanding. Advances in neural information processing systems, 32, 2019.
Yang You, Jing Li, Sashank Reddi, Jonathan Hseu, Sanjiv Kumar, Srinadh Bhojanapalli, Xiaodan Song, James Demmel, Kurt Keutzer, and Cho-Jui Hsich. Large batch optimization for deep learning: Training bert in 76 minutes. In International Conference on Learning Representations, 2020.
Yukun Zhu, Ryan Kiros, Rich Zemel, Ruslan Salakhutdinov. Raquel Urtasun, Antonio Torralba, and Sanja Fidler. Aligning books and movies: Towards story-like visual explanations by watching movies and reading books. In Proceedings of the IEEE International Conference on Computer Vision (ICCV), December 2015.
14

---

Published as a conference paper at ICLR 2023
Supplementary Materials
A ADDITIONAL DETAILS OF THE FOLNET ARCHITECTURE
A.1 MORE DISCUSSIONS ON THE NEURAL LOGIC OPERATORS
Modus Ponens inference with restricted kernels. First, it is straightforward to show that the neural logic operators in Table 2 can be implemented as batch matrix multiplications (Figure 2 where multiple slices of matrix multiplications are executed in parallel to obtain the outputs. Different operators pick their own reduction dimensions (R-dim) and batching dimensions (B-dim) for the matrix multiplication (see Table 2). For example, the join operator picks the a-dimension as the R-dim and the h--imension as the B-dim so that matrix multiplications are carried out in parallel over h. According to ( each slice of the matrix multiplication can be viewed as an independent neural Modus Ponens inference based on its own set of clauses. For example, in the join operator, let Kjain. denote matrix that collects K2 (Ir, a) as its (z, a)-th element. Then, join, is the h-th kernel slice that represents particular group of clauses for the join operator. In addition, recall that the values at each row the kernel slice characterize the conjunction pattern on the right-hand side of (8. with positive (or negative) values determine whether the original premise Pm. (or the negated premise -P,)) should be used for conjunction (see Section 3.2 and Appendix D) Therefore, the R-dim in the matrix multiplication shall be understood the conjunction dimension in (8). In the join operator, it implies that the conjunction operation is over the a-dimension of the premises (a.(a), with the conjunction pattern determined by the æ-th row of Kj if we want infer (ns In contrast, a full Modus Ponens infernece from unary atoms to unary atoms (the first expression in 7 shall perform its conjunction over the joint dimensions of (h, 8, a). which is much higher in complexity. Therefore, the join operator controls the complexity of Modus Ponens inference by restricting the conjunction operation over the a-dimension. Furthermore, it is noteworthy that the join operator is applying the same kernel slice to premises of different s; that is, it implements kernel-sharing across the s-dimension. This is another complexity reduucion strategy resulted from the principles in Section 3.2 Likewise, the same conclusion holds for all other operators, who pick their own conjunction-dimene dimenssons and kernel-sharing dimensions (see Figure 2
join IU B B (x) = E. Ka (a(a,a)vaale
3550c (B c- U UX) us( v) E, (eo
hy
X
x
mu
10 B B
(a) E. Ki(s, (2,
prod (B < x B) (a(a,))
En (x)
X
X
h
cjoin
10 U B): (x) = E. Kas (a)
trans (B 4- B
(u((e.))
E. Ka(s, a)va
h
X
x
X
Figure 2: Our neural logic operators can be efficiently implemented as batch matrix multiplications by using hardware accelerators (c.g., GPUs and TPUs). Each slice of matrix multiplications represents an independent neural Modus Ponens inference based on its own set of clauses.
Naming protocols and examples. We now explain the naming protocols for the logic operators along with concrete examples to demonstrate how each of them help reasoning in different aspects.
15

---

Published as a conference paper at ICLR 2023
bool: It operates the predicate dimension and plays the role of Boolean functions set of imput truth values, which leads to its name of bool-operator. It used to deduce a property of from other properties regarding the same via certain Boolean operations. For example, it can model inferences like Edible + IsMushroom(r) A -IsToxic(r)), where the conjunction pattern the right-hand side is determined by the kernel.
join: We name this operator as join-operator because it is in similar form as the join operation in X-DCS (Liang 2013). For example, when the kernel represents IsBornIn (2: y). it can be used to infer EitizenOfUSA(s from premise CountryIsUSA(g) where the kernel determines the conjunction pattern of CountryIsUSA.
over yy-
cjoin: Since it simply swaps the roles of the kernel and the premise in the join-operator, name as the conjugate join operator. It plays a similar role as the join operator.
mu: We name it mu-operator because it is be viewed a more general-form of the mu-abstraction operation in X-DCS (Liang 2013). For example, when the kernel models ApplyJobAt(z. y). can deduce HasAJob(z from premise ReceiveOfferFrom y), where the kernel determines the conjunction pattern of ReceiveOfferFrom
y) over
assoc: name as assoc-operator because can be viewed as computing the association between two vectors. For example, when the kernel represents LosAngeles(y)) it can be used to infer SameStateAs(r, y) from SanFrancisco(z)) where the kernel determines the conjunction pattern of SanFrancisco(s) over the predicate dimension. In this example, only have one premise atom over the predicate dimension. In the general case, a particular conjunction pattern would be applied to multiple premise atoms yield the output.
prod: We name it as prod-operator because it is in resemblance of computing the product over the predicate dimension. For example, when the kernel models Graduated(x)). it can be used to infer TraduatedFromt. x. y) from StudyAt(x, x. y). where the kernel determines the conjunction pattern StudyAt (x, y) over the predicate dimension. premise atom over the predicate dimension. In the general case, particular conjunction pattern would be applied to multiple premise atoms to yield the output.
this example, we only have one
trans: We name it trans-operator because its form is in reminiscent of reasoning with transitive properties. For example, when the kernel models FatherOf (æ,, 2). it can be used to infer GrandFatherOff( (x, y) from ParentOf (z, y), where the kernel determines the conjunction pattern of ParentOf (2, y) over the dimension 2.
Note that we do not have logic operator corresponding to the typing B <- B x U because it will be identical to the prod operator when the kernel action function is chosen to be the identity mapping. In addition, the above examples further show that the kernels (i.e., the clauses) themselves are also atoms or can be computed from certain atoms. This justifies our strategy of parameterizing the kernels by (the intermediate deduction results of) FOLNet itself in Section
A.2 COMPARING TO THE TRANSFORMER ARCHITECTURE
Figure 3 show that the transformer architecture can be understood single-branch version of the FOLNet architecture with only cin-opeeator and assoc-operator.
(x)
(p())
LayerNorm
LayenNorm
Join
LayerNorm
Boolean
XL
Assoc
Figure 3: Overview of the Transformer architecture. It can be understood as single-branch branch version of the FOLNet architecture with only join-operator and assoc-operator.
16

---

Published as a conference paper at ICLR 2023
A.3 THE BOOLEAN OPERATOR
In FOLNet, we do not chain the Boolean operators in parallel with other operators but chain it with others in cascade manner. This is akin cascading the FFN with the self-attention module in transformers. Nevertheless, we may also place it in parallel to other operators just as making FFN in parallel self-attention in transformer, which done in Chowdhery et al. (2022). In addition, we also adopt the same FFN architecture (i.e.. a multilayer perceptron with GeLU units) for the Boolcan operator in order to make FOLNet directly comparable with transformer architectures.
A.4 THE INPUT-OUTPUT INTERFACE
Computing the pretraining losses (e.g., MLM, NSP and SOP) from the output atoms As pointed out, FOLNet models will have a similar input-output interface as the cxisting transformer models, that we seamless adopt existing pretraining losses (e.g., MLM, NSP and SOP) by computing them from (uz(r)} Recall that uz (x+) is the vector that represents the derived (advanced) properties for the object (token) Ti, where f I....,.. For masked token It, uz (1r,) contains its properties that could be deduced from other tokens via their input properties and relations. Therefore, these deduced properties in uz (Ire can be used predict the original masked token. For example, we can apply a linear classifier followed by softmax operator to compute probability distribution over the vocabulary and the MI.M loss. Likewise, to compute NSP and SOP losses, we add a special " "[CLS]" " token at the beginning of the input sequence, so that the ur (z( that corresponds to the [CISS] token will be fed into binary classifier for computing the NSP or SOP loss. The usage of {uz { in downstream tasks (such as sequence classification, multiple-choice and sequence labeling) are also similar. On the other hand, we have not used the output binary predicates {ur (22, y+)) } for computing any losses. The main reason is that would like to adopt the existing fff-the-shelf pretraining losses for an apple-to-apple comparison regarding the proposed model architecture. Since most of these losses are developed for the transformer architecture, which is a single-branch model with only unary predicates on its main pathway (Figure 3). it is not surprising that these losses are mainly computed from the unary properties {ur (z+)). Nevertheless, we believe that our newly introduced binary predicate branch could open new avenue for developing additional pretraining losses using un (Tte M/) (i.c., the token relations). For example, we may randomly swap two tokens and 2, and use uz(2z, x) to predict whether they have been swapped or not. We will leave the development of more effective pretraining losses for FOLNet future work.
The input base atoms in B and their logit vector representations Throughout our work. only consider plain texts as the input, which is similar to other pretrained language models. Therefore, the base atoms at the input should only encode the information that is directly available from the plain text, which include the token ids in the input sequence (denoted by Token ID, (x:)) and the relative distance between any of the two tokens (denoted by RelDist, xr)) Specifically, TokenID,, the token Te takes the v-th id in the vocabulary for I,...;M V. and TokenID, = F otherwise. When the input sequence consists of pair of sequences, we will construct the input sequence as:
T if
" "[CLS] Sequence #0 [SEP] Sequence #1 [SEP] [PAD] ... [PAD]".
which is similar to the input format used in BERT and other transformer models. In this case, we will introduce additional base atom SeqID (1r)) characterize whether token s; belongs the 8-th sequence (s = 0, 1); that is, SeqID, (xr) = T if token o, belongs to the s-th sequence and is F otherwise. This atom is akin to the segment ids (or token type ids) in existing pretrained transformer models. Furthermore, the relative positional atom RelDista(22 x,) = T if d = dist (I,, xr)) and F otherwise, where dist a (r) ) a clipped distance function with a clipping parameter A:
max(1 1 - A min(r (on(T-t,A-1)
t> 0, > and SeqID, (xx) = SeqID, (I+) t> 0, > 0 and SeqID, (xx) # SeqID, (I+)
A + 1
dista
A
t= 0, 0
A
1> 0, 0
t= 0, =0
The above clipped distance function clamps the relative distance to be within [-A + 1, A 1] when the two tokens are from the same sequence. And it also assigns special distance id whenever they
17

---

Published as a conference paper at ICLR 2023
belong to two different sequences. In other words, the distance between any two tokens would be the same if they belong to two separate sequences. Likewise, we also characterize the distances between the [CLS] token (t 0 or r 0) and the other tokens (T > 0 or t > 0) with two special distance ids, which sets all the regular tokens to have the same distance towards (or from) the special [CLS] token. Such relative positional encoding (atom) preserves the translational invariance of the text sequence and generalizes better than the absolute positional encodings used in BERT and RoBERTa. Notably, it allows FOLNet to generalize to much longer sequences that are unseen during pretraining. In summary, we consider the following base atoms as the inputs to the FOLNet model:
(10) By stacking TokenID..(r.) and SeqID, (3t,) over and 8, stacking RelDista(Te, xx)) over d. and using the values of 1 and to represent and F, respectively, we will have the 0-1 vectors to represent all the input atoms in B. Then, we may further convert them into dense vector represen- tations (u)( uo ( 22 2z)],,r via embedding lookup. The embedding lookup process can also be understood as finding a set of more fine- grained learnable properties (or relations) to characterize )),, where these properties (or relations) are represented in the same logit space as the neural logic operators in Table 2 which carry out Modus Ponens inferences. Finally, the above base atoms are just one particular design for our FOLNet model, and there could be other alternatives for encoding the input information. And when there is extra information available besides plain texts, we may also encode it as the new unary or binary base atoms in B.
B (TokenID, (zz). SeqID, , (zz)), RelDista( It, r= I.....
given token de (or a token-pair
A.5 EXTENDING FOLNET TO TEXI-TO-TEXI VERSIONS
So far, we have mainly focused on the encoder-only version of the FOLNet model. We now show that it can be extended to the decoder-only and the encoder-decoder counterparts in relatively straightforward manner. Such extension would be useful for text-to-text generation tasks.
Decoder-only To develop the decoder-only version of FOLNet, we need to let the model auto- regressively generate the output tokens. In the context of FOLNet, this requires the unary and binary atoms {u(z,), ux), } to be inferred only from the premises in the past: {v(I)), v( TT)) w S t). In addition, we further restrict the binary atoms to have a causal pattern, i.e., u(z,, = 0 and u( = O, whenever t < 7 and w v. Figure illlustaass these patterns for the unary and the binary atoms. Note that the causal structure for the binary atoms translates into a lower triangular pattern for the nonzeros. In particular, the dark blue atoms are directly responsible for the generation of the next token (word), and the auto-regressive generation requires the model to update them only from the light blue atoms. We enforce such retrereereesvve property by multiplying a proper 0-1 mask to the binary kernels and premises in the neural logic operators. In the last column of Table 5 we list the kernels and the premises that should be masked, and in Figure 4bb we show the masks that correspond to two variants of decoder-only models: (i) Causal Langauge Model (CausalLM) (Radford et al., 2018 2019 Brown et al. 2020 Chowdhery et al. 2022), and (ii) Prefix Langauge Model (M) ( et al. 2018 Raffel et al. 2020). The CausalLM has a lower triangular mask for the binary atoms so that the information pattern is always unidirectional. On the other hand, the PrefixLM will have a bi-directional mask for the input segment (the top-left part in Figure 4(b) and a unidirectional mask for the output (target) segment (the bottom-right part in Figure 4(b). Accordingly, the binary atoms of the PrefiXL.M version will share similar pattern its mask in Figure [bb] which models the relatons for the prefix and the output segments separately. With such simple modifications, our decoder-only FOLNet model will have the same input-output interface as the decoder-only transformers; it can be pretrained to predict the next tokens using linear classifier over the unary atoms uy (It)). After the training, it can generate tokens in an auto-regressive manner.
-eecddrr For the encoder-decoder variant, we use two separate stacks of FOLNet for the encoder and decoder, where each of them has the same overall architecture as in Figure with its own set of atoms (the green and blue blocks in Figure ) In particular, the encoder will be identical to the encoder-only version that we have thoroughly discussed earlier in the main paper. Meanwhile, the decoder part will be similar to the decoder-only variant with few additional modifications. First, the decoder needs to maintain a slightly different version of binary atoms (Figure 4(c).. Specifically, besides the relations between the output tokens, the decoder also has to model the (unidirectional) relations from the input tokens to the output tokens (i.c., the bottom-left part of un (z, y) in Figure HC).. These relations are crucial in deducing the output tokens from the input ones, which plays
18

---

Published as a conference paper at ICLR 2023
27
Causal Language Model
Prefix Language Model
(a) Atoms for decoder-only FOLNet
(b) Masks for decoder-only FOLNet
B
Eacnda-Deendkr
(c) Atoms for encoder-decoder FOLNet
(d) Masks for encoder-decoder FOLNet
Figure 4: Extending FOLNet to text-to-text versions. The top row illustrates the atoms and the masking strategy for the decoder-only version, and the bottom shows the encoder-decoder variant. On the left side, we visualize the unary and the binary atoms (u,(z), un (Ir (xr for these cases, where The green and blue colors characterize the nonzero patterns for the atoms of the
d == (h- 1)5 encoder and the decoder, respectively. The green-colored atoms are associated with the encoder part the encoder-decoder variant. The dark blue color represents the atoms that are directly responsible for generating the next token in the decoder. Notably, they are deduced only from the light blue atoms, so that the entire generation process retains an auto-regressive nature. On the right side, the white and the orange blocks represent the 0-1 masking positions, where the light orange part are associated with the prefix segment Lr. Likewise, the encoder-decoder variant has separate sets of masks for the encoder and the decoder, respectively, where the mask for the encoder part are designed to have bi-directional information pattern. Similar design also holds for the prefix part in PrefixLM.
Sym.
Ops.
Typing
B-dim.
R-dim.
Neural operator in logit space
Mask
bool
U +- U
T
w
(.)( > (.
cjoin
U + U B
h
(a) = (Knlaavva
va a)
join
U 4 B x
h
(a) = Ka(a,
Kx(2, a)
m
mu
U 4 B x
(n)(() = Ka(a, (1c
Kr(2, a), (s a)
assoc
U
h
ua ua(r, pu
(z)
prod
B B U
Khu vw
(2, Kx(z, (a,
trans
B B-
h
Ua (a(
Table The binary kernels and premises be masked for decoder-only versions FOLNet. In the encoder-decoder variant, similar part of the kernels and premises would be masked in its decoder.
similar role as the cross- cooss-tttetionn scores in transformers. Accordingly, we also need to adjust the masks to handle these relations separately (see Figure ((d)). Second, neural logic operators in Table shhould also be slightly adjusted in order to incorporate the additional premise atoms from the encoder output. For example, the premise Vhs (a) in join-operator should now a concatenation of the unary atoms from the encoder output (with linear projection) and the decoder premises. Likewise, the premise Vhw (y) in assoc-operator should also be a concatenation of the encoder output (with linear projection) and the decoder premises. These two modified operators share similar spirit as the self-attention and the cross-attention mechanisms in transformer decoders. The cjoin operator is similar join operator, except now the concatenation happens the kernel (as(a) (with
19

---

Published as a conference paper at ICLR 2023
linear projection). The mu-operator and the prod-operator stay the same before. Meanwhile, the trans-operator for the decoder needs to be implemented according to the following new expression:
[uET Kn a) (a. Un (xx, v) - [uuz (z, (1(2 (a. + ( VEI
VET
where T and I are defined be the target and the input sequences, respectively, the kernel K,, a) obtained by applying linear projection to the binary atoms u(æ, a) in the decoder, and the superscript E or D in the premises Vn (a, y) denotes whether they are from the encoder or the decoder. Notably, we observe that the encoder-decode version of FOLNet retains the dual-branch architecture in its decoder module as well. This is in sharp contrast to the standard transformer decoder, which is single- single-branch architecture with only unary atoms on its main pathway.
B EXPERIMENTAL DETAILS
B.1 OVERVIEW OF THE DOWNSTREAM TASKS
The GLUE benchmark (Wan et al,, 2019) consists of 9 tasks: MNLI (Williams et al. 2018 QQP [yer et al. 2017). QNLI Raiiuurkar et al. 2016a). SST-2 (Socher et al. 2013). CoLA
GLUE
WWrstadt et al. 2019). STS-B (Cer et al. 2017). MRPC Dolan & Brockett 2005). ) RTE (Dagan et al.
2005 HHam et al. 2006 Giampiccolo al. [007 Beetivogli al. at. 2009), WNLI (Levesque et al. 2012). They cover a wide range of natural language understanding tasks such as natural language inference, paraphrasing, linguistic acceptability, and sentiment analysis. We finetune our FOLNet models by following the same procedures from BERT (Devlin et al. 20188. We do not evaluate our model on WNLI because it generally needs special procedures. The basic description of all the tasks in GLUE (including their evaluation metrics) can be found in Table
SQuAD 2.0 SQuAD 2.0 (Rajpurkar et al. 2016b) is an extractive question answering dataset built from Wikipedia. The objective of the task is predict an answer span from the context paragraphs. SQuAD 2.0 is an updated version that adds additional 50,000 unanswerable questions to the original SQuAD 1.1 version. The performance metrics are exact match (EM) and Fl scores.
FOLIO FOLIO (Han et al. 2022) is a natural language reasoning dataset that contains first-order logic reasoning problems. It requires the models to decide whether conclusion statement is correct or not given a world defined by a set of premises. It is formulated as a 3-class classification problem with the labels being "True", "False", "Unknown". We follow the same procedure as Han al. (2022) by formulating the problem as sequence pair classification problem. Specifically, we concatenate
all the the premises into one sequence (i.e., sequence A), and then further concatenate it with the conclusion (i.e., sequence B), where the two sequences are separated by a [SEP]. ]. In addition, we add [CLS] token at the beginning and a [SEP] in the end before padding (in the end). By doing so, the task has the same format as a natural language inference task (e.g.. MNLI). Table (quoted from the original FOLIO paper (Han et al. b022)) shows an example from the FOLIO dataset, which demonstrate that it requires strong first-order reasoning capabilities to solve the problem. The dataset has an official train/validation/ess split with 1,004/204/227 examples, respectively. By the time of this submission, the test set is not available and thus we only report performance the validation set.
MNLI
QQP
QNLI
SST-2
CoLA
RTE
MRPC
STS-B
Size Task Metric(s) #Classes
393K Inference Accuracy
364K Similarity Accuray/F11
108K OA/Inference Accuracy
67K Sentiment Accuracy 2
8.5K Acceplability Mathhws corr.
2.5K Inference Accuracy 2
3.7K Paraphrase Accuracy/F1 2
5.7K Similarity
Pearson/Secarman. (regression)
2
Table Basic information about different tasks in GLUE benchmark.
B.2 IMPLEMENTATION DETAILS AND EYPER-PARAMETEE
We implement both pretraining and finetuning pipelines using PyTorch (Paszke et al. (019) and automatic mixed precision (AMP) learning (Micikevicius et al., 2018) based on the Apex library (Nvidia 2019). For pretraining, use large-batch training (with batch-size 131K) using LAMB
20

---

Published as a conference paper at ICLR 2023
FOLIO esample based on the Wild Turkey Wikipodia page
NL premises
NL Conclusion Labels
turkey. Mornians is wild tirkey, Rio Grande wild turkey, and the Ocellated wild turkey
types wild turkes:
turkey. Gould's wild turkey
Ocellated turkey. 1>
B. Tom is an Eastern wild turkey. -> Falsn
Tom is Eastern wild turkey.
Jocy wild turkey. Unknown
is
wild turkey.
Tom is also not Gould's is wild mirkey, or Merria''s 's wild turkey, or Rio Grande wild turkey.
5. Tom is wild tarkey.
FOL
FOL conclusion
1. Var(W1dTarkaar - (Eastarn(a) v DacesTa(ss V Goulda(a)
A. (eallated((rmm)) > True
Nerriane (ar) Riogrande(s) Ocellsted(ss))
Bastern(/om)) False
"Wildtakkkyyttom Bastern(tom)
Vildfurkey(jrr)) Unknown
3. 1(2111 aararky A (aceola(tom))
VildTarkey((nn (fom) + (Goulde (fom) /Merrians (fom) Riogranda((onn))
MildTurkey(tom)
Table 7:
We show example from the FOLIO dataset, which is quoted directly from the original FOLIO paper (Han et al. 2022). It demonstrates that it requires strong first-order reasoning capabil- ities to solve the problem. "NL" stands for "natural language" and "FOL" stands for "First-Order Logic". We only use the natural language part to solve the task in our experiments.
optimizer You al. 2020). The pretraining of FOLNetBase Wikipedia + BookCorpus (16GB) for 8K steps takes about 12 hours using 512 V100 GPUs. The pretraining of FOLNet on 160GB data for 128K steps takes days using 512 V GPUs. And the pretraining of FOLNet,aage data for 128K steps takes 19 days using 512 V100 GPUs. For the finetuning of all downstream tasks, we also use AMP learning based on Apex, and the optimizers are Adam from Apex library.
160GB
The hypper-parameters for finetuning different downstream tasks included in Table The hyper-parameters for the finetuning tasks are searched per task, and the results are the median of five random seeds.
We report the hyper-parameters of pretraining FOLNet in Table 8
Hyperparams
FOLNcIBae
FOLNelBaw
FOLNa
Pretraining data size
16G
160G
160G
Number of Layers (L.
12
12
24
Unary Hidden size (D2)
768
768
1024
Binary Hidden size (D2)
64
64
64
Unary Boolean (FFN) intermediate size
3072
3072
4096
Binary Boolean (FFN) intermediate size
256
256
256
Number of Attention heads (H)
12
12
16
Attention head size (S)
64
64
64
RPE clipping parameter (A)
64
64
64
Dropout rate
0.1
0.1
0.1
Attention Dropout rate
0.1
0.1
0.1
Warmup Ratio
1%
1%
1%
Peak Learning Rate
le-2
2c-3
1.6e-3
Batch Size
131,072
131,072
131,072
Weight Decay
0.01
0.01
0.01
Max Steps
8K
128K
128K
Learning rate Decay
Linear
Linear
Linear
LAMB
le-6
le-6
1c-6
LAMB $1
0.9
0.9
0.9
LAMB 32
0.999
0.999
0.999
Gradient Clipping
0.0
0.0
0.0
Sequence length (I)
128
128
128
Table 8: The hyper-parameters for pretraining FOLNet in different settings.
C ADDITIONAL EXPERIMENTS
We evaluate the zero-shot performance of our FOLNet model on GLUE benchmark. Specifically, perform zeros-shot predictions by using the same method and prompt templates from Gao et al. (02TT. We only consider the FOLNet model and compare it RoBERTa arge, which similar model-size, pretraining dataset and pretraining losses. In
Zero-shot performance on GLUE
21

---

Published as a conference paper at ICLR 2023
Hyperparams
GLUUE-smal/big
SQuAD 2.0
FOLIO
Max epochs
(5, 10, 20) / [2, 5] {6c-5, 8c-5 Ic-4, 2c-4)
12, 3)
[60, 80]
Peak Ir for Base (16G):
{ 18--5, 9c-5, 1o-4
6c-5, 70-5, 8c-5)
Peak Ir for Base/Large (160G);
(lc-5, 2c-5, 3e-5, 4c-5)
(1c-5, 2c-5, 3c-5, 4o-5)
{ lc-5, 2c-5, 3e-5)
Batch size
{16, 32) / [32)
(16, 32) Linear
{16, 32) Linear
Learning rate decay
Linear (6%, 25%) ) [6%)
Warmup ratio
6%
(6%, 25%)
Sequence length
128
384
512
Adam e
le-6
le-6
le-6
Adam &
0.9
0.9
0.9
Adam B2
0.999
0.999
0.999
Gradient clipping
0,0
0.0
0.0
Dropout rate
0.1
0.1
0.1
Weight decay
0.01
0.01
0.01
Table 9: The hyperparameters for finetuning on GLUE, SQuAD 2.0, and FOLIO tasks. GLUl-small
refers to CoLA, STS-B, MRPC and RTE. GLUE-big stands for MNLI, QQP, QNLI and SST-2.
MNLI-m/mm
QQP
QNLI
SST-2 Ace
CoLA STS-B
MRPC
RTE
Model
Method
Data
Avg
Ace
F1
Ace
mee
Pear.
F1
Ace
Majority guess
32.7/33.0
0.0
49,5
50.9
0.0
81.2
52.7
33.3
RoBERTaLage
MLM
160G
50.8/51.7
49.7
50.8
83.6
2.0
-3.2
61.9
51.3
44.3
FOLNetLage
MLM
160G
50.8/52.6
55.4
59.7
79.5
6,4
23.6
77.2
58.2
51.5
Table 10: The zero-shot performance on the GLUE development set. "Acc" stands for accuracy, "mee" means Matthews's i's correlation coefficient, and "Pear." is short for Pearson's 's correlation coefficient.
addition, the zero-shot prediction methods are also similar: they both predict the label words using their own MLM heads. The results are summarized in Table which demonstrates that FOLNeer,arge outperforms RoBERTaLage on 7 out of 8 tasks with average gain of 7.2 points.
Performance on CLUTRR To further examine the reasoning capabilities, we evaluate our FOL.- Net models on the CLUTRR (Compositional Language Understanding and Text-based Relational Reasoning) dataset (Sinha et al. 2019). CLUTRR is semi-synthetic diagnostic benchmark designed evaluate the systematic generalization ability of model. Specifically, for a given natural language story, a model is asked to infer the (implicit) relationship between two family members. To solve the problem, it has to extract relationships between entities and master the logical rules governing these relat relationships. CLUTRR examines the systematic generalization by testing model on stories that contain unseen combinations of logical rules as well as stories that require more reasoning steps. Following the same setting as (Sinaa et al. (2019), we first finetune our FOL.Net models with clauses of length h = 2.3 and = 2,3, 3, 4, respectively, and then we evaluate them on clauses of length k 2.... 10. Specifically, for each input instance, we concatenate the natural language story with the text query (separated by a [SEP] ] token), and cast the problem as a sequence -pair classification. In addition, we add [ [CLS] token at the beginning and append a [SEP] ] token in the end. However, unlike Sinha et al. (201)). for simplicity, we do not replace entities with special task-specific embed- dings but treat them as regular English words. And we leave such entity representation techniques as a future work, which may further improve our performance. We finetune FOLNetgase for 100 epochs using an Adam optimizer with a learning rate of 10 5 batch-size of 16 and warmup ratio of 6%. In our experiment, we mainly compare to transformer- language inputs. Specifically, we compare to the results of BERT Base and BERT-L-TM from Sinha et (2199), which share the same model size and pretraining corpus. There is also rich set of approaches that work directly on the symbolic inputs from CLUTRR, such as the Graph Attention Network (GAT) (Veličković al. 2018). Since these methods directly use the structured logical graph underlying the story as their input, they circumvent the difficulty of parsing natural language stories and generally perform much better than the text-based counterparts. We include GAT reference to examine whether our FOLNet model could narrow such a performance gap with the help of our logical inductive bias. The full results are reported in Figure 5 First, our FeL performs
basclines that also use the natural
22

---

Published as a conference paper at ICLR 2023
100
BERT
BERT-LSTM
BERT-LSTM
40-
GAT
56-
GAT FOLNe(
FOLNefguse
Accuracy (%)
60-
Accuracy (N)
60-
40-
40
20-
20-
Length
Relation Lengrh
(a) Training clause length 2,3
(b) Training clause length = 2,3,
Figure 5: Systematic generalization performance on CLUTRR benchmark.
significantly better than both text-based methods (i.e., BERT Base and BERT-LSTM) by large margin. In particular, when the testing out the training set, we improve BERT Base by as much as 8% ~ 41% (in absolute improvement). Meanwhile, this advantage further improves to 54% 81% when the testing & has been seen during training. and FOLNetgase achieves competitive performance achieves substantially better performance (10% 20%) than GAT when the testing 10. The results confirms the benefit of encoding logical inductive bias.
GAT. Notably, FOLNetgase
D NEURAL MODUS PONENS
In this section, we show that generic Modus Ponens inference using clauses of the form ( can be expressed as matrix multiplication followed by optional nonlinear activation function.
Suppose .l...., Pas are premise atoms and Q is the head (i.e., conclusion) atom. partition set M = (1..... M) into three noneoverlg sets M4, Mo and M_, and define the following general clause (where we have dropped the subscript n in (8) for simplicity of notation):
A
Pm
A
APm
A
P
(11)
mEM+
mEM
Note that the expression 11 can be used to represent general clause that infers Q from any subset of the premises in (P,..... PM, PP..... PMM) where the sets M4 and M_ contain the indexes of the selected original and negated premises, respectively, and the set Mo indexes the ignored premises. Therefore, each particular partition M M4 U MMg MM also corresponds
unique
clause. To apply Modus Ponens inference with the clause (II). we can proceed in two steps: (i)
evaluate the body of the clause (i.e., the hight-hand side of (11) according to
P=
A Pm mEM.
)
A PP MEM_
(12)
and (ii) apply the generic Modus Ponens inference rule: Q + 1Q+ P and P). In the following two subsections, will derive the neural operations that implement these two steps, respectively. Specifically, the neural operations will be carried out in the logit space under the ProbLog setting.
D.1 THE MATRIX MULTIPLICATION FOR THE COMPOSITIONS OF THE BODY ATOMS
To derive the operations for the logic expression 12 we define the probabilities of the atoms P. Pi...., Par being true in the following logistic form:
Pr[P T I+e-2
Pr[Pm =T 1 -m-0m
1, I,... M,
(13)
23

---

Published as a conference paper at ICLR 2023
where 2 and um are the logits corresponding to the atoms P and Pm. respectively. In addition, for each m, we further introduce a variable Wn, = +1,0, 0, -1 to indicate whether m is in M4, Mo and M_, respectiivyy Then, (I2) can be characterized by the following conditional probability:
{
Am= [(Wm = 1AP T) - --1PPm = (W =0] (14) otherwisc
Pr{
where W - {Wi...., W and P A Pi...., PM are independent of each other and are also independent of Wi...., WM. In addition, for given set of of = (Wi...., War,, we introduce the following random event of Pi...., Par:
We further assume that the variables
M
8 (P (P...., PM)
A [m==1A Pm = T) (W - P. =F) (W,,. -V }. (15)
m=1
along with its indicator function I((P)...., Pas) E E). An indicator function 1(-) is one if the expression inside its parenthesis is true and zero otherwise. Then, we can derive Pr(P = TIW} as:
Pr(P T(W Σ [
PM|W)
(o)
= TIP PM1
P
Σ
I((P,..... PM) E P) ................................ Park
M T((P,,... PM) E II m=1
(0)
Pr (m))
P
( Σ
II
Pr[Pm) II
Pr[Pm.
II
Pr(Pm]
(P (Pan.,Pa))EE mEM+
M.
mE.Ma
Σ
II
Pr[Pm = T II
Pr(Pm =]
II
Pr[Pm.
(P PMOEEE mEM+
mEM_
mEMo
II Pr(P m =T-
II Pr(P.. n =]
Σ
II
Pr(Pm)
mEM+
MEM-
{Pm: mE me.o
II Pr(P T
II
Pr(P.. FF
mEM,
mEM
II mEMi
II MEM
I
I emm
(I + e Vm Ih
vm
(1 + emm)
M
(1 + )m)
(1 + evi )mm
m=1
SV
1
(16)
1te EM -1m)
where step (a) uses the independence between Pı...., PM and WI...., WAr, step (b) substitutes the definition of the conditional probability (14), step (c) uses the assumption that Pı...., PM are independent each other, step (d) substitutes the decomposition M = M n Mg n A__, step (e) is obtained by following the definition (15) for the event & (i.e., within event E, we have P.. = T for m E M4. P... = F for m E M_ and Pm. being arbitrary value in {T, F)). step (f) takes out the common factors of the summands and keeps the remaining summation over all possible values in (Pm : m E Mo} (based on the definition (15), step (g) uses the fact that the total probability of the event (Pm E MM) is one, step (h) substitutes the second logistic expression in (13). step
24

---

Published as a conference paper at ICLR 2023
(i) introduces variables I & I(m E MM+) and In e I(m MM-,, and the upper bound in step (i) is obtained by expanding the multiplications in the denominator and dropping all the cross-terms, all of which are nonnegative. Note that the upper bound (T6) is tight when only one unique I or is nonzero. The above result in (66 is conditioned
Im (among all "i.."................... ... Iy and IT.... ,...., M particular realization of W W..... WM. which are discrete variables. Therefore, it is difficult to learn them directly in differentiable manner. To address this issue, we further assume that W1, 1i..... Wa are random variables. Then, taking expectation over WI,..., WM, we have:
1
Pr[P= = SE lite Emai (Im (1m--m)) Wm
(17)
where the randomness inside the expectation comes from 1tt and mm which are further from Wi.-.., Was- We now adopt simple yet effective strategy to approximate the right-hand side in order to obtain a differentiable implementation. Specifically, we use the following approximation:
(18)
EXXEX
which becomes more accurate when the distribution of X gets more concentrated around a single peak (i.e., becoming determinisitic). Using the above approximation, we obtain
=
(19)
+ Em (Mmh Nom 10
where K/th _ E Pr(W,,, m = +1} and Kin
- El Pr(W m = -1}. Substituting the first expression (13 into the left-hand side of we conclude that the logit of Pr(P = T) can be approximated via:
M (mith
)m (Ke, v)
(20)
172=1
where k and denote the vectors that collect with Km and Um as their m-th elements, respectively. The above expression implies that the logit 2 for the atom can be computed by a simple inner product operation between the two vectors k. and Further recall that different realizations of WI.... WM correspond to different partitions of M = MM UMo U-M, which further defines different logic expressions for P in (12). Since K is a vector that collects Pr(W,, m +1} Pr( - as its m-th element, it can also be viewed as a signature vector of the logic expression (12). which the body of the clause (TT). Therefore,
also the signature vector for the clause (IT) as determines the logic compositions among the body atoms. When we have multiple logic expressions, represented by Ki,..., KN.. that compose N logic expressions from the atoms Pi..... PM. then we can compute the logits of the output logic expressions by the following matrix multiplication:
z Kv,
(21)
where z a logit vector for the N output atoms, and K is matrix consists of K., its n2th row.
D.2 THE NONLINEAR ACTIVATION FOR THE IMPLICATION OPERATION
In this subsection, Q = [Q < P and P). in the logit space. Likewise, we consider the ProbLog setting, where the atoms P and Q will be assigned with probabilities Pr P T} and Pr(Q T), respectively, which characterize the chances of them being true. Let C (Q t P) be the clause. Our objective is to infer the outcome probability Pr(Q = T} from Pr(P = T} based on the fact that C' = T.
proceed to derive the the neural operator for generic Modus Ponens inference,
To this end, we first derive the conditional probability Pr(Q TJP. =]. We will first need to establish the conditional probability Pr(C' = TIP.Q) based on the definition of the logical implication "z" (Andrews) 2013) and then apply Bayes rule. Note that C = (Q + P) is defined as
C é (Q v PP), which can be expressed as the following conditional probability:
0
if P = T and Q =
Pr(C = T
-1
(22)
otherwise
25

---

Published as a conference paper at ICLR 2023
That is, the clause C will be false only when the premise P is true and the conclusion Q is false. To proceed, we further assume that Pr(Q = T]P] = Pr[Q = P) = 1/2. The intuition of the assumption is that we do not have any prior knowledge about the outcome Q when only given the input premise P (without O). Then, by Bayes rule, we have
= T) = Pr(Q -
Pr[C' = T
= Pr(Q -TP.)
EQE(F.T) Pr(Q1P)Pr(C-=THH =
Pr(C=
Pr[C (P
(M) [1
P=
(23)
0.5 i=
where steps (a) and (b) substitute Pr[Q TIP) = Pr(Q = -IP)= = 1/2 and 22 respectively.
Next, we derive (r(Q- = TIC = Т} with the assumption that the premise P is independent of the clause C' that is used in the current Modus Ponens inference step. We have
Pr(Q =
Σ PE(T.F)
Pr(Q= - TTP:(
Σ PE(T.F)
Pr(Q = =HQQ=TP,C=T]P:112
1 Pr(PP=
Tr[P=T]+ + +
1 2 Pr
(24)
where step (a) uses the assumption that P is independent of C, and step (b) substitutes (23)
Finally, we derive the expression for the conditional probability (24) in the logit space. Specifically, let q and p be the logits for Q and P, respectively, which parameterize their probabilities:
Pr(P T) 1
Pr(Q T|C T) Ite-"
(25)
Substituting the first expression in (5 into 24
followed by some simple algebra, obtain
Pr(Q= T) = e In(1+e==)
(26)
Comparing the right-han hand side of 26 with that of 25 we obtain the logit for P as
In(1 + 2e
(27)
vThe above expression implies that, to implement Modus Ponens in logit space, we only need to apply the above simple nonlinear activation function to the input premise logit p. To gain further insights into the above logit-space Modus Ponens inference,
carry out further analysis of the above nonlinear activation function. Note that In(1 + e²)) is a non-negative convex function. By using Jensen's is inequality and the non-negativity, we can prove that the above nonlinear activation function can be lower bounded as In(1 + 2e*) > ReLU(( + In 2), where the right-hand side is indeed good approximation of the left-hand side. Therefore, we can implement the generic Modus Ponens by applying the (shifted) ReLU function in the logit space with much lower computation complexity.
D.3 PUTTING EVERYTHING TOGETHER
Given N different clauses of the form [I] and M input premises Pt,...,PMM the deduction of the N outcome atoms Q ,,,...,C QN using Modus Ponens rule can be implemented (approximately) via
Kv
(28)
26

---

Published as a conference paper at ICLR 2023
==In(1 + +2c"),
(29)
where K is an N x M matrix with its n-th row being the signature vector of the n-th clause, v an M-dimensional vector consisting of the logits for the input premises Pi...., PM. u is an N- N-dimensional vector that contains the logits of the output atoms Q1.... QN. Note that the Modus Ponens inference using N clauses can be implemented in parallel by first multiply matrix K to the left of the logit vector v and then pass through special element- wise nonlinear activation function (which can be approximated by a shifted ReLU function). For this reason, we call K the kernel-ef-clauses (or kernely for short) in this paper. Furthermore, we note that, when the implication " in (TT) is replaced by the logical equivalence "=", the original clause (II) becomes (12) so that the activation function in (29) can be dropped. Finally, it is straightforward to show that the neural Modus Ponens inference 28 for the four categories of rules in (7 can be expressed (cquivalently) as:
n(x)= Σ (w((za))v(a)
(30)
n(x)
Σ aa)
(31)
a
u(z, Σ (r, (I,
(32)
u(z, = Σ KBB y, b)v((,
(33)
a.b
where Kvv( a). Kup(z, a, b), (z y, a) and K (z, y, a. b) are D1 D1. D1 Dz, D2 D1 and D2 x D2 matrices that correspond to Ryu, Rue, Reu and Res, respectively.
E COMPOSITION OF EXISTING RELATIVE POSITIONAL ENCODING
We now show that we can compose the existing relative positional encoding (denoted as RPE* in our paper) from our m-operator and p-operator. To begin with, we first write the expressions of
existing RPE* from Shaw et al. (2018) by using their original notation. Specifically, RPE' computes the (multi-head) self-attention outputs according to the following expressions (with the notation of self-attention head h being dropped for simplicity):
" = Σ Cay WV +
2; =
Oig WV + +
Σ asjay
(34)
J=1
WKYT + + 5T 3. x, x, W aar
Cty =
(35)
Vd
Vd.
where zi is the self-attention output at the i-th token, egy is the unnormalized self-attention scores, Onl is the self-attention probability (which is obtained by applying softmax to Eij. normalized over f), 2n is the vector of the i-th token at the attention input, (r//k//W' are the weight matrices for query/key/value in the self-attention operation, respectively, n is the sequence length, and d, the dimension of each head. Notably, a and at are the learnable relative positional embedding vectors, defined as
a w K
K
(ciip(jii.k)
dj = w (dip(j-i,
v
clip(r, k) = max(- minkk
That is, a and at are the embedding vectors corresponding to the (clipped) relative distance between the i-th and the j-th tokens, where & is the clipping threshold. Therefore, the second terms in both (34) (35) are the RPE' bias terms that seep into the computations of self-attention mechanism. We now proceed to show that these two terms can be viewed as the degenerated form of our m-operator and p-operator in Table 2 For convenience, we first rewrite the expressions for these two operators:
m Uhs (x) =
Σ Kr (x
(36)
27

---

Published as a conference paper at ICLR 2023
p: (1
Σ K (
(37)
We first show that the second terms in can be composed from the m-operator (36 To see this, note that tokens æ and a in (36) can be identified as and j in (34, respectively. addition, the kernel K (zz a) and the premise (z a) can be identified as the self-attention probability dıj and the relative positional embedding vector a'j, respectively. Therefore, the m-operator shares the same form as the second term in (34), except our "head" index h. Likewise, we can show that the second term in (35) can be composed from the p-operator. Observe that the second term in 33) is inner product between vector 2: Wº and vector at Let kg, (a:) be a vector that collects / (x) as its w-th element and let v(z, y) be a vector that collects (w, y) as its w-th element. Then, the p-operator can also be viewed an inner product between the vectors kn (x) and vax, y). By identifying kn. (a) and (az, y) as the vectors x; W/ and a. respectively, we conclude that the second term in (35) can be composed from the p-operator. Besides these similarities, our m-operator and p-operator are more general and powerful than the original RPE* in the following aspects. First, recall from Section 3.3 that both the kernels K and are parametrized by the FOLNet (by linearly projecting the intermedhate representations (ur( (I } followed by possible activation functions). Therefore, our vo a) and Va (I. y) are instance-depee deepndd and are different across input instances. contrast, d and K "1 in (4 (35) are static embedding vectors that are the same for all input instances. Furthermore, our m-operator and p-operator are adaptive layerwise manner; that is, each layer will compute their own m-operator and p-operator adaptively based on their own intermediate representations (u,( (xr y)}, whereas in RPE* the vectors a and at are generally identical across layers. Therefore, the existing operations in RPE* can be viewed as the degenerated special cases that are composable from our m-operator and p-operator.
28


Reviews:
[{"review_id": "jphKHENi0Z8", "reviewer": "Reviewer_93RW", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper proposes a new family of transformer-style architectures inspired by inductive logic programming, with the goal of improving natural language reasoning capabilities. They connect this new family of architectures to existing commonly used modifications. They show good results on widely used benchmarks, modulo some concerns I raise later.", "summary_of_the_review": "This is a well-motivated and well-executed paper addressing a problem of wide interest to the community. It could benefit from more experiments giving intuition of how the architectures work in practice, and some very careful apples-to-apples comparisons, but the ablations already go a good way in that direction. This paper merits acceptance.", "strength_and_weaknesses": "Method:\n\nThe problem of learning new transformer architectures to aid reasoning is of broad interest to the community.\n\nThe model is well motivated by comparing to things like relative position encoding, and trying to use logical formalisms to derive new architectural variants on transformers.\n\nHow much the new architectural variants actually maintain the spirit of the logical formalisms is a bit harder to parse,  for example in 3.1 we are replacing discrete booleans with arbitrary real numbers and stating that “it characterizes the extent to which the atom is true”, which usually implies some value in [0,1] or some kind of increasing transformation, which I do not think you have here.\n\nExperiments: \n\nThe experiments are quite thorough, but are often confounded by the proposed method having more parameters than the baselines. Additionally, experiments would be better if they included your own reruns of BERT/Roberta experiments using the same settings to make it apples to apples in some cases, this is especially helpful for the situations where gains are small. This would also make it easier to make apples-to-apples comparisons in terms of # of parameters, but it would be a lot of work.\n\nThe ablation experiments in Table #3 are nice and quite appreciated. It would be good to have another column saying how many parameters each ablated model has.\n\nHaving some more concrete examples / toy examples of how these new architectures specifically aid reasoning on concrete examples would be really helpful.\n\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "U2z29tce81R", "reviewer": "Reviewer_dLtS", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper introduce a novel logical inductive bias which treats the language representation learning problem as logic programming. We then develop a fully-differentiable neural architecture (FOLNet) that forward-chains a set of neural logic operators, which effectively encodes this inductive bias. The proposed FOLNet architecture has the same input-output interface as the transformer models and can be pretrained over large-scale text data. Experimental results demonstrate that the FOLNet architecture significantly outperforms different variants of transformer models under the same pretraining losses. The results further show that the inherent dual-branch architecture has many advantages over the single-branch version in transformers.", "summary_of_the_review": "Interesting and useful method.", "strength_and_weaknesses": "Strength: \n\n1. This paper develops a novel neural architecture named FOLNet (First-Order Logic Network), to encode this new logic inductive bias.\n\n2. This paper finds that the self-attention module in transformers can be composed by two of our neural logic operators, which probably explains their strong reasoning performance.\n\n3. The extensive experimental results support their  claims and validates the superiority.\n\nWeaknesses：\n\n1. It is not clear why FOLNet  could encode  logic inductive bias?", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "8drDwzJMb6q", "reviewer": "Reviewer_nDcE", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This work introduces a new neural architecture (FOLNet) that incorporates a first-order logical inductive bias. Taking inspiration from the forward-chaining algorithm, the architecture is recursively applying learnable Kernel operations on unary and binary representations of tokens.\n\nInput and outputs are traditional token ids, making FOLNet an easy plug-and-play replacement from traditional Transformers. In addition, traditional Transformers can still be represented by a FOLNet making the proposed architecture more flexible and likely more powerful.\n\nExperiments on GLUE, SQuAD2.0, and FOLIO show that this new architecture performs better than comparable baselines such as BERT, RoBERTa, ALBERT, and Megatron.\n", "summary_of_the_review": "strong paper. more previous work should be discussed and some clarifications are needed.\n", "strength_and_weaknesses": "**strengths**\n\nThis is a strong paper that introduces a novel architecture with inductive biases from first-order logic. Experimental results are very promising and the fact that logical tasks remain a challenge for traditional Transformers can make this work impactful.\nThe paper is well written and surprisingly easy to read. However the methodology could benefit from some clarifications listed below:\n\n**weaknesses**\n\n1. Methodology : After pre-training on large datasets, were the models fine-tuned on individual tasks (GLUE, SQuAD, FOLIO) ? If so, for how long? it should be mentioned somewhere.\n\n2. Methodology : It is not clear how the final representations output from the model u_L(x) and u_L(x, y) are used in the computation of any loss. How does one go from these two representations to say a softmax layer to predict the masked token in MLM? Similarly for the other losses NSP and SOP.\n\n*Question*: Related to the point above, is it safe to assume that FOLNets are encoder type networks and cannot be used in an decoder-only or encoder-decoder fashion? If this assumption is wrong, then the paper should also describe how u_L(x) and u_L(x, y) are used to predict next words.\n\n3. Literature : Some influential previous work should be further discussed. Neural Theorem Provers (NeurIPS’17), and in particular Greedy Neural Theorem Provers (AAAI’20) applied logical modules on text before. More recently, Edge Transformers (NeurIPS’21) proposed a triangular attention mechanism between pairs of tokens in an effort to better represent logical behaviors between tokens. Comparing FOLNets to these works would help contextualize the work better.\n\n4. Experiments : Right now the paper shows improvements on “classical” language tasks. It would be a nice bonus to also show that the FOL inductive bias is indeed useful for more reasoning heavy tasks such as GSM8k, ProofWriter, CLUTRR, etc…\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works.", "empirical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works."}, "extra_text": {}}]

Author rebuttal:

Decision: Accept: poster