Logic Pre-Training of Language Models
Authors: Siru Ouyang
Zhuosheng Zhang
hai zhao
Venue: ICLR 2022 Submitted (2022)
Keywords: Language Models
Pre-training
Logical Reasoning
Natural Language Understanding

Abstract: Pre-trained language models (PrLMs) have been shown useful for enhancing a broad range of natural language understanding (NLU) tasks. However, the capacity for capturing logic relations in challenging NLU still remains a bottleneck even for state-of-the-art PrLM enhancement, which greatly stalled their reasoning abilities. Thus we propose logic pre-training of language models, leading to the logic reasoning ability equipped PrLM, \textsc{Prophet}. To let logic pre-training perform on a clear, accurate, and generalized knowledge basis, we introduce \textit{fact} instead of the plain language unit in previous PrLMs. The \textit{fact} is extracted through syntactic parsing in avoidance of unnecessary complex knowledge injection. Meanwhile, it enables training logic-aware models to be conducted on a more general language text. To explicitly guide the PrLM to capture logic relations, three pre-training objectives are introduced: 1) logical connectives masking to capture sentence-level logics, 2) logical structure completion to accurately capture facts from the original context, 3) logical path prediction on a logical graph to uncover global logic relationships among facts. We evaluate our model on a broad range of NLP and NLU tasks, including natural language inference, relation extraction, and machine reading comprehension with logical reasoning. Results show that the extracted fact and the newly introduced pre-training tasks can help \textsc{Prophet} achieve significant performance in all the downstream tasks, especially in logic reasoning related tasks. 

Under review as a conference paper at ICLR 2022
LOGIC PRE-TRAINING OF LANGUAGE MODELS
Anonymous authors Paper under double-blind review
ABSTRACT
Pre-trained language models (PrLMs) have been shown useful for enhancing broad range of natural language understanding (NLU) tasks. However, the capacity for capturing logic relations in challenging NLU still remains a bottleneck even for state-of-ther thr-t-rrtt PrLM enhancement, which greatly stalled their reasoning abilities. Thus we propose logic pre-training of language models, leading to the logic reasoning ability equipped PrLM, PROPHET. To let logic pre-training perform on a clear, accurate, and generalized knowledge basis, we introduce fact instead of the plain language unit in previous Pr..s. The fact extracted through syntactic parsing in avoidance of unnecessary complex knowledge injection. Meanwhile, it enables training logic-aware models to be conducted on a more general language text. To explicitly guide the PrLM to capture logic relations, three pre-training objectives are introduced: 1) logical connectives masking to capture sentence-level logics, 2) logical structure completion accurately capture facts from the original context, 3) logical path prediction on a logical graph to uncover global logic relationships among facts. We evaluate our model on broad range of NLP and NLU tasks, including natural language inference, relation extraction, and machine reading comprehension with logical reasoning. Results show that the extracted fact and the newly introduced pre-training tasks can help PROPHET achieve significant performance in all the downstream tasks, especially in logic reasoning related tasks.
INTRODUCTION
Machine reasoning in natural language understanding (NLU) aims to teach machines to understand human languages by building and analyzing the connections between the facts, events, and observations using logical analysis techniques like deduction and induction, which is one of the ultimate goals towards human- paatty intelligence. Although pre-trained language models (PrLMs), such as BERT (Devlin al., 2018), GPT (Radford et al., 2018), XLNet (Yang et al.] 2019) and RoBERTa (Liu al.) 2019), have established state-of-the-art performance on various aspects in NLU, they are still short complex language understanding tasks that involve reasoning (Helwe et al. 2021). The major reason behind this is that they insufficiently capable of capturing logic relations such as negation (Kassner& & Schütze 2019), factual knowledge (Poerner et al.) (019). events (Rogers et al. 2020), and so Many previous studies (Sun et al. 2021 Xiong al. 2019, Wang al. 2020) are then motivated to inject knowledge into pre-trained models like BERT and RoBERTa. However, they too much rely on massive external knowledge sources and ignore that language itself is a natural knowledge carrier as the basis of acquiring logic reasoning ability (Ouyang et al. 2021)). Taking the context in Figure [ as an example, previous approaches tend to focus on entities such as the definition of "govvenment" and the concepts related to it like "governor", but overlook the exact relations inherent in this example, thus failing model the complex reasoning process.
Given the fact that PrLMs are the key supporting components in natural language understanding, in this work, we propose a fundamental solution by empowering the PrL.M with the capacity of capturing logic relations, which is necessary for logical reasoning. However, logical reasoning can only be implemented on the basis of clear, accurate, and generalized knowledge. Therefore, we leverage fact as the conceptual knowledge unit to serve the basis for logic relation extraction. Fact is organized as a triplet, i.e., in the form of predicate-aggument structures, to represent the meaning such as "who-did- did-ddat--ttto whha-ttowhhmm " and "who-is- what". Compared with existing studies that inject complex knowledge like knowledge graphs, the knowledge structure based on fact is far less complicated and more general in representing events and relations in languages.

---

Under review as a conference paper at ICLR 2022
On top of the fact-based knowledge structure, we present PROPHET, logic-aware pre-trained language model to learn the logic-aware relations in a universal way from very large texts. In detail, we introduce three novel pre-training objectives based on the newly introduced knowledge structure basis fact: 1) logical connectives masking for learning sentence-level logic connection. 2) logical structure completion task top of facts for regularization, aligning extracted fact with the original context. 3) logical path prediction to capture the logic relationship between facts. PROPHET is evaluated on broad range of language understanding tasks: natural language inference, semantic similarity, machine reading comprehension, etc. Experimental results show that the fact is useful as the carrier for knowledge modeling, and the newly introduced pre-training tasks can improve PROPHET and achieves significant performance on downstream tasks |
RELATED WORK
2.1 PRE-TRAINED LANGUAGE MODELS IN NLP
Large pre-trained language models (Devlin et al. 2018 Liu et al. 2019 Radford et al.] 2018 have brought dramatic empirical improvements on almost every NLP task in the past few years. classical norm of pre-training is train neural models on a large corpus with self-supervised pre-training objectives. "Self-superviied" means that the supervision provided in the training process automatically generated from the raw text instead of manually generation. Designing effective criteria for language modeling is one of the major topics in training pre-trained models, which decides how the model captures the knowledge from large-scale unlabeled data. The most popular pre-training objective used today is masked language modeling (MLM), initially used in BERT (Devlin et al. 2018), which randomly masks out tokens, and the model is asked to uncover it given surrounding context. Recent studies have investigated diverse variants of denoising strategies (Raffel et al. 2020
Lewis et al.) 2020), model architecture (Yang et al. 2019). and auxiliary objectives (Lan et al. 2019; Joshi al. (020)) to improve the model strength during pre-training. Although the existing techniques have shown effectiveness in capturing syntactic and semantic information after large-scale pre-training, they perform sensitivity role reversal and struggles with pragmatic inference and role-based event knowledge (Rogers et al. 2020). which are critical to the ultimate goal of complex reasoning that requires to uncover logical structures. However, it is difficult for pre-trained language models to capture the logical structure inherent in the texts since logical supervision is rarely available during pre-training. Therefore, we are motivated to explicitly guide the model to capture such clues via our newly introduced self-supervised tasks.
2.2 REASONING ABILITY FOR PRE-TRAINED LANGUAGE MODELS
There is lot of work in the research line of enhancing reasoning abilities in pre-trained language models via injecting knowledge. The existing approaches mainly design novel pre-training objectives and leverage abundant knowledge sources such as WordNet (Miller 1995).
Notably, ERNIE 3.0 (Sun et al. T uses broad range of pre-training objectives from word-aware structure-aware to knowledge-awarr tasks, based on a 4TB corpus consisting of plain texts and a large-scale knowledge graph. WKLM (Xiong et al. 2019) replaces entity mentions in the document with other entities of the same type, and the objective is to distinguish the replaced entity from the original ones. KEPLER (Wang et al. 202 encodes textual entity descriptions using embeddings from a PrLM to take full advantage of the abundant textual information. K-Adapter (Wang et al. 2020) designs neural adapters to distinguish the type of knowledge sources to capture various knowledge.
Our proposed method differs from previous studies in three aspects. Firstly, our model does not require any external knowledge resources like previous methods that WordNet, WikiData, etc. We only use small-scale textual sources following the standard PrLMs like BERT (Devlin et al.) 2018), along with off-the-shelf dependency parser to extract facts. Secondly, previous works only consider triplet-level pre-training objectives. We proposed multi-granularity pre-training strategy, considering not only triplet-level information but also sentence-level and global knowledge to enhance logic reasoning. Finally, we propose new training mechanism apart from masked language modeling (MLM), hoping to shed light on more logic pre-training strategies in this research line.
OOr codes have heen uploaded supplemental material, which will he open after the double review period.
2

---

Under review as a conference paper at ICLR 2022
Fact
Logical Graph
opposite
anarchists, participated, revolution revolution, opposite, movement
participated
movement
anarchists
revolution
after
they, met, suppression
coraf
suppression
stabalized
suppresion, after, stabilized
they
was
government, was, stabilized
government
stabalized
Despite concemn, anarchists participated in the Russian Revolution in opposition to White movement However they met harsh suppression after the Rolshovik government stahilzed.
Text
Figure I: How the facts and logical graph constructed from raw text inputs. Edges in red denotes additional edges added in the logical graph, while text with green indicates the sentence-level logical connectives which will be mentioned in
3 PRELIMINARIES
In this section, will introduce the concept of fact and logical graph, which is the basis of PROPHET. We will also describe extracting the fact for logical graph construction, as an example shown in Figure
3.1 FACT
Following Nakashole & Mitchell (2014) and Ouyang et al. (2021), extract facts which triplets represented as = [Ai, P, A2 where A₁ and A are the arguments and / is the predicate between them. well represent broad range of facts, reflecting the notion of wwho-dd--wha-to-wwihe and "who-is-we
what", "who-is-what,, etc.
We extract such facts in a syntactic way, which makes our approach generic and easy to apply. Given a document, we first split the document into multiple sentences. For each sentence, we conduct dependency parsing using StanfordCoreNL. (Manning et al. 2014) For the analyzed dependencies, basically, we consider verb phrases and some prepositions in the sentences as "predicates", and then we search for their corresponding actors and actees as the "arguments".
3.2 LOGICAL GRAPII
logical graph an undirected (but is not required to be connected) graph that represents logical dependency relation between components in facts. In logical graphs, nodes represent argumend/predicates the fact, and edges indicate whether two nodes have relations in a fact. Such structure can well unveil and organize semantic information captured by facts. Besides, logical graph supports considerations among long-range dependencies via connecting arguments and their relations in different facts across different spans.
We further show how to construct such graphs based on facts. Despite given relations facts, we design another two types of edges based on identical mentions and coreference information. (1) There can be identical mentions in different sentences, resulting in repeated nodes in facts. We connect nodes corresponding to the same non-pronoun arguments by edges with edge type same. (2) We conduct coreference resolution
context using an off-to-shelf model to identify arguments in facts that refer the same one We add edges with type coref between them. The final logical graph is denoted as S (V, E), where V A; U.U. P and E (1,2).
we also tried to use OpenIE directly; however,
the performance is not satisfactory. https://githan comub.commhhuii http.////wugggiiggac///eeeerreee

---

Under review as a conference paper at ICLR 2022
anarchists, participated, revolution revolution, opposite, movement they, met, suppression suppresion, after, stabilized govermment stabilized
after
stabilized
Fact
pre-training with fact-aware logics
Text
Text Encoder
However, they harsh suppression
after the Bolshevik government...
logical prediction
masking
fact unit aignment
Logical Graph
apposito
anarchists
participated
Levonution
movementt
[CLS) suppression (MASK] the Bolshovik... [SEPP
suppression, after revolution, government
mat corof
s.pprrassion
suppression after the Boishevik..
suporession, efter, stabilized)
revolution, government
they WOS
gevernment
stabalized
samee
Text
Fact Unit
Node Pairs
Figure 2: An illustration about pre-training methods used in PROPHET. The model takes the text,
extracted fact and the randomly sampled node pairs in the logical graph as the input. The model is pre- pre-trained with three novel objectives. One is the standard masked language modeling applied to sententious connectives, the others are fact alignment and logical path prediction.
PROPHET
4.1 MODEL ARCHITECTURE
We follow BERT (Devlin et al. 2018) and use a multi-layer bidirectional Transformer (Vaswani et al. (017) as the model architecture of PROPHET. For keeping the focus the newly introduced techniques, we will not review the ubiquitous Transformer architecture in detail. We develop PROPIET by using exactly the same model architecture as BERT-base, where the model consists of 12 transformer layers, with 768 hidden size, 12 attention heads, and 1 11MM model parameters in total.
4.2 LOGIC-AWARE PRE-TRAINING TASKS
We describe three pre-training tasks used for pre-training PROPHET in this section. Figure is illustration of PROPHET pre-training. The first task is logical connectives masking (I.M) generalized from masked language modeling (Devlin et al.) 2018) for logical connectives to learn sentence-level representation. The second task is logical structure completion (LSC) for learning logic relationship inside fact, where we first randomly mask arguments in facts, and then predict those items. Finally, logical path prediction (LPP) task is proposed for recognizing the logical relations of randomly selected node pairs.
Logical Connective Masking Logical connective masking is an extension of the masked language modeling (MLM) pre-training objective in Devlin al. (2018 8), with particular focus connective indication tokens. We use the Penn Discourse Trec 2.0 (PDTB) (Prasad et al.) 2008) to draw the logical relations among sentences. Specifically, PDTB 2.0 contains relations that manually annotated on the million Wall Street Journal (WSJ) corpus and are broadly characterized into "Explicit" and "Implicit" connectives. We use the "Explicit" type (in total 100 such connectives), which apparently presents in sentences such as discourse adverbial "instead" or subordinating conjunction "because". Taking all the identified connectives and some randomly sampled other tokens (for a total 15% of the tokens of the original context), we replace them with a [MASK] token 80% of the time, with a random token 10% of the time and leave them unchanged 10% of the time. The MLM objective is to predict the original tokens of these sampled tokens, which has proven effective in previous works (Devlin et al., 2018 Liu et al. 2019).. In this way, the model learns to recover the logical relations for two given sentences, which helps language understanding. The objective of this task is denoted as Lconn.
Logical Structure Completion To align representation between the context and the extracted fact, introduce pre-training task of logical structure completion. The motivation here is to encourage
4

---

Under review as a conference paper at ICLR 2022
the model learn the structure representation that encodes the Aetto-id-What--ee What-to-Whom" like meanings for better language understanding. To speak in detail, we randomly select a specific proportion À of the total facts (A 20% in this work), from given context. For each chosen fact, either ask the model to complete Argument-Predicatee
or "Argument-?-Ar A-rgummemm (the templates selected based equal probability). We denote all the blanks that need to be completed as m and m², denoting arguments and predicates, respectively, In our implementation, this objective is the same masked language modeling to keep simplicity, by using the original loss following Devlin et al. 2018).
Lulign =
Σ
log (( m,"))
kEaUp
where the discriminator predicts a token from large vocabulary.
Logical Path Prediction To learn representation from the constructed logical graph, thus endowing the model with global logical reasoning ability, we propose the pre-training task of predicting whether there exists path between two selected nodes in the logical graph. In this way, the model learns to look at logical relations across a long distance of arguments and predicates in different facts.
We randomly sample 20% nodes from logical graph to form set V', there are total T node pairs. We have maximum number mazp of node pairs to predict. To avoid bias the training process, we try to make sure that mazz are positive samples and the rest are negative samples, thus balancing negative ratios. If the number of positive/negative samples is less than " we just keep original pairs. Formally, the pre-training objective of this task calculated as below following Guo et al. (2020):
Path =
Σ oog uj] + (1 - 8) log(1 [us, v,])],
where 5 is when i; and vy have connected path and 0 otherwise. [ug, By] denotes the concatenation of representations of u and vy.
The final training objective is the weighted sum of the above mentioned three losses.
C = Leonn + Lalign + Patth
4.3 PRE-TRAINING DETAILS
We use the English Wikipedia (1.1 million articles in total), we sample the train and valid datasets with split ratio of 19: 1 on the original datasets. We omit the "Reference" and "Literature" part in a document to ensure data quality. Following the previous practice (Devlin al.) 2018), we limit the length of sentences in each batch up 512 tokens and the batch size is 128. We use Adam (Kingma& & Ba 2014) with 31 = 0.9, 3g = 0.98 and e = le 6, and weight decay is set as 0.01. We pre-train our model for 500k steps. We use 8 NVIDIA V100 32G GPUs, with FP1 and deepspeed for training acceleration. Initialized by the pre-trained weights of BERT basT, we continue training models for 200k steps.
EXPERIMENTS
5.1 TASKS AND DATASETS
Our experiments are conducted on a broad range of language understanding tasks, including natural language inference, machine reading comprehension, semantic similarity, and text classification. Some of these tasks are part of GLUE (Wagg et al.) 2018) benchmark. We also extend our experiments to DocRED (Yao et al. 2019). widely used benchmark of document-level relation extraction for generalizability. To verify model's reasoning abilities of logic, perform experiments on two recent logical reasoning datasets in the form machine reading comprehension, ReClor (Yu et al. 2020) and LogiQA (Liu al. 2020).
5

---

Under review as a conference paper at ICLR 2022
Model
Classification
Language Inference
Semantic Similarity Avg.
CoLAA SST-2
MNLI
QNLI RIE
MRPC
QQP
STS-B
In literature
BERT base
52.1
93.5
84.6/83.4
90.5
66.4
88.9
71.2
85.8
79.6
SemBERT base
57.8
93.5
84.4/84.0
90.9
69.3
88.2
71.8
87.3
80.8
Our implementation
BERT base
53.6
93.5
84.6/83.4
90.9
66.6
88.6
71.2
85.8
79.8
PROPHET
57.0
93.9
85.3/84.3
91.4
69.8
89.5
72.0
86.0
81.1
Table Leaderboard results on GLUE benchmark. The number below each task denotes the number of training examples. Fl scores are reported for QQP and MRPC, Spearman correlations are reported for STS-B, and accuracy scores are reported for the other tasks.
Model
ReClor
LogiQA
Dev
Test
Test-E
Test-H
Dev
Test
Human Performance*
63.0
57.1
67.2
86.0
In literature
FOCAL REASONER (Oyann LReasoner (Wang et al. 2021a
2021
78.6
73.3
86.4
63.0
47.3
45.8
74.6
71.8
83.4
62.7
45,8
43.3
DAGN (Huang et 202T
65.8
58.3
75.9
44.5
36.9
39.3
BERT large Devlin et al. 2018)
53.8
49.8
72.0
32.3
34.1
31.0
XLNetiarge Yang et al. 2019)
62.0
56.0
75.7
40.5
RoBERTatarge
Liu et al
2019
62.6
55.6
75.5
40.0
35.0
35.3
DeBERTatarge Tatarge
(He al.
2020)
74.4
68.9
83.4
57.5
44.4
41.5
Our implementation
BERT base
51.2
47.3
71.6
28.2 32.2
33.8
32.1
PROPHET
53.4
48.8
72.4
35.2
34.1
Table 2: Accuracy on ReClor and LogiQA dataset. The public methods are based on large models.
5.2 RESULTS
Table shows results the GLUE benchmark datasets. We have the following observations from the above results.
(1) PROPHET obtains substantial gains over the BERT baseline (continual trained for 200K steps for a fair comparison), indicating that our model can work well in a general sense of language understanding.
(2) PROPHET performs particularly well on language inference tasks including MNLI, QNLI, and RTE which indicates our model's ability reasoning.
(3) Whether it is large-scale datasets such as QQP and MNLI or small datasets like COLA and SST-B, our model demonstrates a consistent improvement, indicating its robustness.
(4) From Table we can see that PROPHET improves the logical reasoning ability of BERT baseline by large margın. Especially, armed with our approach, the results on the datasets for the BERT-base model are comparable or even surpass those with BERT-large results.
In addition, we conducted experiments on large-scale uuman-annotated dataset for document-level relation extraction (Yao et al. 2019). The results are shown in Table 3 From the table, we can see that PROPHET still does well for relation extraction for documents by outperforming the baseline
WWe exclude the problematic WNLI set.
We only report the results for Ign Fl in the annotated setting the distant supervision is slow train.

---

Under review as a conference paper at ICLR 2022
Dev
Test
Model
F1
Intra-F1
Inter-F1
F1
basT
Devlin et al. 2018)
54.2
61.6
47.2
53.2
Two-Phase BERT " (Wang el 2019
54.4
61.8
47.3
53.9
PROPHET
54.8 (10.6) 62.4 (f0.8)
47.5 (10.3) 54.3 (11.1)
Table Main results on the dev and test set for DocRED. indicates that the results are taken from Nan et al. (2020). Intra- and Inter-Fl indicates Fl scores for the intra- and inter-sentence relations following the setting of Nan al. (2020)
Model
Classification
Language Inference
Semantic Similarity
Avg.
CoLA
SST-2
MNLI
QNLI
RTE
MRPC
QQP
STS-B
PROPHET
57.0
93.9
85.3/84.3
91.4
69.8
89.5
72.0
86.0
81.1
w/o LCM
53.6
93.5
85.1/84.0
90.9
68.2
88.6
71.2
85.8
80.1
w/o LSC
53.6
93.6
85.0/84.1
91.3
69.0
88.9
71.4
85.9
80.3
w/o LPP
52.1
93.0
84.6/83.4
90.9
66.4
88.6
71.2
85.8
79.6
Table 4: Ablation studies of PROPHET the test set of GLUE dataset.
substantially. even surpasses the two-phase BERT. Also, our model is especially good at coping with inter-sentence relations compared with baseline models, which means that our model is indeed capable of synthesizing the information across multiple sentences of document, verifying the effectiveness of leveraging sententious and global information.
6 ANALYSIS
6.1 ABLATION STUDY
To investigate the impacts of different objectives introduced, evaluate three variants of PROPHET as described Section 4.2] 1) the w/o LCM model adopts a substitute without logical connectives masking the pre-training objective, 2) the w/o LSC model is such that it leaves the logical structure completion objective, and 3) the w/o LPP model only uses the objectives of connective masking and structure completion. The results are shown in Table
Based on the ablation studies, we come to the following conclusions. Firstly, all three components contribute the performance removing any one of them causes performance drop the average score. Especially, the average point drops the most as we remove the logical path prediction objective, which sheds light on the importance of modeling chain-like relations of events. Secondly, we can see that logical path prediction contributes the most to the reasoning abilities as the performance on language inference improves the most when we add the sententious connective masking objective and the task of logical path prediction.
6.2 COMPARISON BETWEEN FACT AND ENTITY-LIKE KNOWLEDGE
We also replace the injected fact with common practice using entity-like knowledge, which is using named entities. In detail, change the arguments in facts into named entities recognized by StanfordCoreNI..
and leave the predicates extracted unchanged, resulting in the form < N E,, predictte N Egg > (NE stands for named entity). If fact not recognized with any named entities, we just leave it out.
The results are shown in Table We can that the performance is hurt a lot, even worse than vanilla BERT. This is quite intuitive as the number of named entities is far less than our obtained fact,
https::/stanfordnn

---

Under review as a conference paper at ICLR 2022
Model
Classification
Language Inference
Semantic Similarity
Avg.
CoLA
SST-2
MNLI
QNLI
RTE
MRPC
QQP
STS-B
PROPHET
57.0
93.6
85.0/84.1
91.4
69.8
89.2
71.4
86.0
81.0
w/ named entities
50.4
93.2
84.9/84.2
90.8
68.7
88.4
71.0
84.9
79.3
Table 5: Results on GLUE test set when replacing facts with named entities and key the relations unchanged.
missing lot of information inherent in the context. Whereas our introduced fact can well capture the knowledge used in the reasoning process, providing fundamental reasoning basis.
6.3 ATTENTION MATRIX HEATMAP
We plot the attention matrix in token level to see how our model interprets the context using heatmap shown in Figure g
10
06
kell
14
mon
Figure 3: Heatmap of the attention matrix of vanilla BERT and our implemented PROPHET for the sentence "However, they met harsh suppression after the Bolthevik government was stabilized.". Weights are selected from the first head of the last attention layer.
From the figure, we can see that the vanilla BERT attends to delimiters, particularly punctuation suggested in Clark al. (2019). In comparison, our model exhibits quite different attention distribution. Firstly, our model clearly decreases the influences introduced by punctuation. Secondly, model pays more attention to tokens representing discourse-level information, such as "however" and "after", which is consistent with our motivation. It also well captures the relations of pronouns. The event characteristics are also illustrated
from the "after suppression" phrases.
6.4 EFFECT OF DIFFERENT CONTEXT LENGTH
We group samples into ten subsets according to an equal amount of samples (around 1000 samples per interval) by context length since the majority of the samples concentrate on the interval of under 60. The statistics of calculate the accuracy the baseline and PROPHET per group for both matched and mismatched set, as shown in Figure. 4 We observe that the performance of the baseline groups drops dramatically when encountered with long contexts, especially for those longer than 45 tokens, while our model performs more robustly on those intervals (the slope of the dashed line is more gentle).
and ANLL-mismatcher dev sets are shown in Table Then we
6.5
CASE STUDY
We also give a case study to demonstrate that PROPHET could enhance the reasoning process in language understanding. Given two sentences, PROPHET and BERT-base predict whether
8

---

Under review as a conference paper at ICLR 2022
Table 6: Distribution of context length dev set of MINII-matched and MNLI- mismatched dataset.
Dataset
(0,29)
[30, 59)
[60, 89)
[90, 119)
[120, 1.49)
[150, 179)
[180, 209)
[210, 239)
MNNI-matched
39.2%
49.8%
9.6%
1.0%
0.12%
0.12%
0.02%
0.06%
MNLL-mismatched
33.7%
55.0%
9.7%
1.7%
0.3%
0.1%
0.1%
0%
87
86.5
87
85
84.5
84
83
835
varile BERT
83
81
vanila BERT
(016)]
(29,29 130, (51 (6) ml
10181 (24.28) (28.22) (2236)) 66420 (4550) 19.00
Contect Length
Context Length
Figure 4: Accuracy of different context length on MNLI-match (left) and MNLI-mismatch (right) dev set. There approximate 1000 samples in each intervals.
the sentences entailed not. Results are shown in Figure 5 To see the language understanding ability of our model, we made two subtle changes in the original training sample. Firstly, we change the entity referred to in the sentence. We between entities than BERT-base model. Additionally, we add a negation the sentence. Although this change small, it completely changes the semantic of the sentence, and leads a reversal of the ground-truth labels. We can see that PROPHET is good at all the samples given, indicating that is not only good reasoning in language understanding, but also is more robust than baseline models.
that PROPHET learns better alignment relations
Unchanged
Entity change
Negation
Sentence
that S8B, CFF and FFS stand for the main railway company, in German, French and Italian..
Sentence 1: Note that SB8, CFF and FFS stand out for the main railway company, in German, French and Italian..
Note that SBB, CFF and FFS stand out for the railway company, in German, French and Italian..
Sentence 1:
Input
Sentence 2:
The French rallway company is called SNCF.
Sentence 2:
The French railway company is called SBB..
Sentence 2:
The French railway company is not called SNCF.
Label
not entailment
not entailment
entailment
Prophet: entailment /
Prophet: not entailment /
Prophet: nentailment /
Prediction
BERT-base:
entailment
BERT-base: entailment
BERT-base: not entailment x
Figure 5: We take example from RTE dataset, and use PROPHET and BERT-base to predict the label of the relations among two given sentences.
7 CONCLUSION
In this paper, we leverage fact in newly pre-trained language model PROPHET to capture logic relations essentially, in consideration of the fundamental role of PrLM serving NLP and NLU tasks. We introduce three novel pre-training tasks and show that PROPHET achieves significant improvement over various logic reasoning involved NLP and NLU downstream tasks, including language inference. sentence classification, semantic similarity, and machine reading comprehension. Further analysis shows that our model can well interpret the inner logical structure the context to the reasoning process.

---

Under review as a conference paper at ICLR 2022
REFERENCES
Kevin Clark, Urvashi Khandelwal, Omer Levy, and Christopher D) Manning. What does bert look at? an analysis of bert's attention. arXiv preprint arXiv: 04341, 2019.
Jacob Devlin, Ming-Wai Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.0480 2018.
Daya Guo, Shuo Ren, Shuai Lu, Zhangyin Feng. Duyu Tang, Shujie Liu, Long Zhou, Nan Duan, Alexey Svyatkovskiy, Shengyu Fu, et Graphcodebert: Pre-training code representations with data flow. arXiv preprint arXiv:2009.0885
2020.
Pengcheng He, Xiaodong Liu, Jianfeng Gao, and Weizhu Chen. Deberta: Decoding-enhanced bert with disentangled attention. arXiv preprint arXiv:2006.03654, 2020.
Chadi Helwe, Chloé Clavel, and Fabian M Suchanek. Reasoning with transformer-based models: Deep learning, but shallow reasoning. In 3rd Conference Automated Knowledge Base Construction, 2021.
Yinya Huang, Meng Fang, Yu Cao, Liwei Wang, and Xiaodan Liang. DAGN: Discourse-aware graph network for logical reasoning. In NAACL, 2021.
Mandar Joshi, Dangi Chen, Yinhan Liu, Daniel Weld, Luke Zettlemoyer, and Omer Levy. SpanBERT: Improving pre-training by representing and predicting spans. Transactions of the Association for Computational Linguistics, 8:64-77, 2020.
Nora Kassner and Hinrich Schütze. Negated and misprimed probes for pretrained language models: Birds can talk, but cannot fly. arXiv preprint
2019.
Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint 142.691412.69900 2014.
Zhenzhong Lan, Mingda Chen, Sehastian Goodman, Kevin Gimpel, Piyush Sharma, and Radu Soricut. ALBERT: A lite BERT for self-supervised learning of language representations. In International Conference on Learning Representations, 2019. URL httpt:tt/////....t. http::///oppeeeeeeeeeee. htw.eeewwiw..eeeeeeeeeeeeeeeeeee nwtp
H1cA7AEtvS
Mike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, Abdelrahman Mohamed, Omer Levy, Veselin Stoyanov, and Luke Zettlemoyer. Bart: Denoising sequence-to-sequenee pre-training for natural language generation, translation, and comprehension. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pp. 7871-7880, 2020.
Jian Liu, Leyang Cui, Hanmeng Liu, Dandan Huang, Yile Wang, and Yue Zhang. Logiqa: A challenge dataset for machine reading comprehension with logical reasoning. In Christian Bessiere (ed.), Proceedings of the Twenty-Nint International Joint Conference Artificial Intelligence, IJCAI-20, pp. 3622-3628 International Joint Conferences on Artificial Intelligence Organization, 7 2020. doi: 10. 0t.24963/jcai.22220555 /tt.:t/taee//ooooooo/////22222/0 Main track.
Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. RoBERTa: Robustly Optimized BERT Pretraining Approach. arXiv e-prints, art. 19X.:1997..1192.. July 2019.
Yinhan Liu. Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. Roberta: A robustly optimized bert pretraining approach. arXiv preprint arXiv: /907./ 1690, 2019.
Christopher D Manning, Mihai Surdeanu, John Bauer, Jenny Rose Finkel, Steven Bethard, and David McClosky. The stanford corenlp natural language processing toolkit. In Proceedings of 52nd annual meeting of the association for computational linguistics: system demonstrations, pp. 55-60, 2014.
George A Miller. Wordnet: a lexical database for english. Communications of the ACM, 38(11): 39-41, 1995.
10

---

Under review as a conference paper at ICLR 2022
Ndapandula Nakashole and Tom Mitchell. Language-awer -ware aware truth assessment of fact candidates. In Proceedings of the 52nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 1009-1019, 2014.
Guoshun Nan, Zhijiang Guo, Ivan Sekulić, and Wei Lu. Reasoning with latent structure refinement for document-level relation extraction. arXiv preprint arXiv:2005.06312. 2020.
Siru Ouyang, Zhuosheng Zhang, and Hai Zhao. Fact-driven logical reasoning. arXiv preprint arXiv:2105. 10334, 2021.
Nina Poerner, Ulli Waltinger, and Hinrich Schütze. Bert is not knowledge base (yet): Factual knowledge vs. name-based reasoning in unsupervised qa. arXiv preprint /rXiv:/911.0300 2019.
Rashmi Prasad, Nikhil Dinesh, Alan Lee, Eleni Miltsakaki, Livio Robaldo, Aravind K Joshi, and Bonnie L Webber. The penn discourse treebank 2.0. In LREC. Citeseer, 2008
Alec Radford, Karthik Narasimhan, Tim Salimans, and Ilya Sutskever. Improving language understanding by generative pre- pre-training. 2018.
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. Exploring the limits of transfer learning with a unified text-to-text transformer. Journal of Machine Learning Research, 21:1-67, 2020.
Anna Rogers, Olga Kovaleva, and Anna Rumshisky. A primer in bertology: What we know about how bert works. Transactions of the Association for Computational Linguisties, 8:842-866, 2020.
Yu Sun, Shuohuan Wang, Shikun Feng, Siyu Ding, Chao Pang, Junyuan Shang, Jiaxiang Liu, Xuyi Chen, Yanbin Zhao, Yuxiang Lu, et al. Ernie 3.0: Large-scale knowledge enhanced pre-training for language understanding and generation. arXiv preprint arXiv:2107.021377 2021.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Advances in neural information processing systems, pp. 5998-6008, 2017.
Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R Bowman. Glue: A multi-task benchmark and analysis platform for natural language understanding. arXiv preprint arXiv: 080..80440.6.. 2018
Hong Wang, Christfried Focke, Rob Sylvester, Nilesh Mishra, and William Wang. Fine-tune bert for docred with two-step process. arXiv preprint arXiv: 11898, 2019.
Ruize Wang, Duyu Tang, Nan Duan, Zhongyu Wei, Xuanjing Huang, Guihong Cao. Daxin Jiang, Ming Zhou, al. K-adapter: Infusing knowledge into pre-trained models with adapters. arXiv preprint arXiv:2002.00 01 808, 2020.
Siyuan Wang, Wanjun Zhong, Duyu Tang, Zhongyu Wei, Zhihao Fan, Daxin Jiang, Ming Zhou, and Nan Duan. Logic-driven context extension and data augmentation for logical reasoning of text. arXiv preprint arXiv:2105.03655 2021a.
Xiaozhi Wang, Tianyu Gao, Zhaocheng Zhu, Zhengyan Zhang. Zhiyuan Liu, Juanzi Li, and Jian Tang. Kepler: A unified model for knowledge embedding and pre-trained language representation. Transactions of the Association for Compulational l Linguistics, 9:176-194, 2021b.
Wenhan Xiong, Jingfei Du, William Yang Wang, and Veselin Stoyanov. Pretrained encyclopedia: pretrraie language model. arXiv preprint arXiv:1912.09637, 2019.
Weakly supervised
Zhilin Yang, Zihang Dai, Yiming Yang, Jaime Carbonell, Russ R Salakhutdinov, and Quoc V Le. Xlnet: Generalized autoregressive pretraining for language understanding. Advances in neural information processing systems, 32, 2019.
Yuan Yao, Deming Ye, Peng Li, Xu Han, Yankai Lin, Zhenghao Liu, Zhiyuan Liu, Lixin Huang, Jie Zhou, and Maosong Sun. Docred: A large-scale document-level relation extraction dataset. arXiv preprint arXiv:I906 06170,1117000 2019.
Weihao Yu, Zihang Jiang, Yanfei Dong, and Jiashi Feng. Reclor: A reading comprehension dataset requiring logical reasoning. In International Conference on Learning Representations (ICLR), April 2020.
11


Reviews:
[{"review_id": "uqC7N-fcEzG", "reviewer": "Reviewer_rHDy", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "This paper is intended to inject knowledge into pre-trained language models without relying on external knowledge. Instead, it proposes to leverage knowledge in the corpus itself via dependency parsing and pre-trains a new language model with 3 designed pre-training tasks. \n\nThe authors should be cautious when using big words such as “logic reasoning”. The writing in sections 1 and 3 gives the feeling of overclaiming. It does not specify the exact meaning of the word \"logic\". The related work section is far from being exhaustive, missing a large portion of PrTM with syntax information. The pretraining tasks lack some details. Is LSC another modified MLM? Does it consider context? This work is based on BERT-base, which has limited capacity. It is not convincing of where the improvement is from, and it is unlikely that the proposed base-size model will have a broader impact. In short, I find the overall performance of the method not impressive. Additionally, in the analysis section, the attention maps are clearly selected biased over the multi-heads. It is not difficult to select the desired heat map from multiple heads. \n\nStrengths: \n1. This paper proposes a new method to enhance the logical reasoning ability of PrTM via syntactic parsing that avoids complex knowledge injection. \n2. Experiments show the effectiveness of the method, and the writing is clear and easy to understand.\n\nWeakness:\n1. The proposed method is not novel enough when compared with other works using syntactic parsing to enhance PrTMs.\n2. It is likely that there are only very limited logic relations contained in texts, making PROPHET not substantially beneficial to downstream NLP tasks.\n3. The experimental results are not convincing enough. Firstly, this paper only conducts experiments on BERT-base, which is an undertrained model. It is worth at least reporting both BERT-base and BERT-large to demonstrate the scalability of PROPHET.\n\n\n", "summary_of_the_paper": "To enable pre-trained language models to capture logic relations contained in natural language, this paper presents a pre-trained language model, PROPHET, which aims to encode logic relations (facts) with three pre-training objectives. Experiments show that the proposed method outperforms its baseline BERT-base in a broad range of NLP tasks.", "summary_of_the_review": "Injecting knowledge into pre-trained models without resorting to external knowledge sources is interesting, and the effect is confirmed by empirical results. However, my concern is the limited novelty of the proposed model.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "sNyCe5_HzEt", "reviewer": "Reviewer_K1pp", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "I think this paper has the following weak points:\n1. The word \"logic\" is misused. Obviously, the paper considers the additional information of dependency parsing rather than logic. Secondly, the information in this paper is not \"fact\", they should be called word dependency. Of course, the naming is not a very serious issue. I just hope that the author's naming will not mislead the readers.\n2. The main drawback of this paper is that it does not convince me that dependency parsing is a very important additional information for pre-training language models. According to previous studies [1], the pre-training of BERT actually already contains morphology, syntax, and semantics information. And the current state-of-the-art dependency parser is even built based on pre-trained language models. This is what I suspect motivates the further introduction of dependency information. In particular, this paper does not specify what exactly the introduction of dependency parsing information solves for the original masked language modeling task. For reference, [2] introduces span information (simpler morphology information than dependency), thus making pre-training more reasonable and difficult.\n3. Equation (1) should be rewriten. Since ma and mp are blanks, why can they be used as conditions?\n4. The logical connective masking proposed by the authors is very strange. It is irrelevant to their motivation based on fact and logical graph as stated in the previous sections.\n5. I do not think that the authors' experiments support their motivation very well. First, with reference to SpanBERT, which introduces span information, the effect of PROPHET is degraded. Second, the authors do not experimentally verify what exactly the newly introduced dependency parsing solves. The example in Figure 5 does not give an in-depth insight to me.\n6. In the experiments of Figure 3, it is not reasonable to compare the attention matrix of token level only. Because some dependency information may be represented by higher levels of BERT.\n7. in equation 3, the authors did not consider the original masked language modeling as the loss, which I think should be helpful.\n\n\n[1] Bert rediscovers the classical nlp pipeline\n[2] SpanBERT: Improving Pre-training by Representing and Predicting Spans", "summary_of_the_paper": "This paper proposes to add the dependency parsing information of sentences as part of the pre-training objectives to the pre-training of BERT. In particular, it designs three pre-training tasks: logical connective masking, logical structure completion, and logical path prediction, to introduce the dependency parsing information.", "summary_of_the_review": "In summary, I think the motivation for adding additional dependency parsing has not been adequately articulated. And the experiments have not fully verified what the dependency parsing information actually solves. The simpler SpanBERT is in fact better than this paper. Therefore, I think this paper does not reach the threshold of ICLR.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "1: The contributions are neither significant nor novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "dIvlQkOdExm", "reviewer": "Reviewer_DQM2", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "## The good\n\nThe paper attempts to build logic-aware models by pre-training on relation prediction and link prediction objectives. Since natural language corpuses are not annotated with facts (triples of source, sink and relation), the paper proposes a dependency parser based extraction mechanism. The paper investigates their method on GLUE benchmark, and also investigates it on two additional datasets which require logical reasoning. The paper also provides an ablation study and attempts to glance at the inner working of the model by qualitative analysis of attentions. The proposed model appears to work better on long contexts, which is also a plus.\n\n## The bad\n\nIt is clear that the paper required multiple revisions in the writing process before submission, as it is rife with many grammatical issues and weird constructions. Even if I choose to ignore that fact, a big issue in the framing of the paper is its complete lack of awareness of relation prediction and link prediction literature. A fact introduced is essentially a triple of source, sink and the relation, which is extensively studied in the Knowledge Base (KB) and Graph Neural Networks community. The related work misses to relate to this large body of work. Preliminaries also miss out on important nomenclature of the logical facts being triples and how it relates to having a knowledge base. In fact, the logic aware pre-training tasks are also variations of popular knowledge base reasoning tasks, such as relation prediction (\"Logical Connective Masking\" in the paper) and link prediction (\"Logical Structure Completion\"). The authors should acknowledge this vast area of research, and relate their tasks with the existing literature appropriately.\n\nThe proposed pre-training technique appears to work well in downstream tasks, but the paper raises more questions of the entire experimental results. Firstly, it is not clear whether the authors pre-trained the whole model on their pre-training objectives, or just continued pre-training on a publicly available checkpoint. If it is the latter, then the claim that their pre-training objectives are able to inject logical reasoning awareness is significantly weakened, as the downstream performance could be just an artefact of regularization through continual training. The presented results are also incomplete, as no information is provided on whether its the mean or median (typically downstream experiments are reported on a median of five runs, see Liu at al 2019 https://arxiv.org/abs/1907.11692 .) The authors tend to investigate primarily on GLUE benchmark, but it has been shown extensively in the literature that many tasks in this benchmark does not require logical reasoning, as the data consists of annotation artefacts which are leveraged by the model during training. (Gururangan et al 2018 https://arxiv.org/abs/1803.02324, Poliak et al 2019 https://aclanthology.org/S18-2023/, Tsuchiya et al 2018 https://aclanthology.org/L18-1239/, McCoy et al. 2019 https://arxiv.org/abs/1902.01007) Interestingly enough, recent research indicates these tasks might not even require the knowledge of word order, either during fine-tuning (Gupta et al 2021 https://ojs.aaai.org/index.php/AAAI/article/view/17531, Pham et al 2021 https://arxiv.org/abs/2012.15180) or in the pre-trained representation (Sinha et al 2021 https://arxiv.org/abs/2104.06644.) Thus, authors could have instead doubled down on more tasks that require explicit logical reasoning, such as CFQ (https://openreview.net/pdf?id=SygcCnNKwr), CLUTRR (https://arxiv.org/abs/1908.06177), TextWorld (https://arxiv.org/abs/1806.11532), etc.\n\nThe analysis presented in the paper (Attention Matrix Heatmap and Case Study) is not convincing and instead raises questions on the motivation and methodology used. Both are cases of extreme cherry picking, which results in a weak argument and does not answer the question of whether the proposed model really imbibes logical reasoning abilities adequately.\n\n## Questions & Comments\n\n### Section 3.1\n\n- Does the fact only represent the two notions \"who-did-what-to-whom\" and \"who-is-what\"? There are more categories of relations in knowledge base triples.\n\n### Section 4.2\n\n- \"apparently presents in sentences\" -> what does this mean?\n- The authors had previously defined a fact in Section 3.1 (T={A_1, P, A_2}), yet they do not re-use them in the text (\"Argument-Predicate-?\" could have been T={A_1, P, ?}). Thus, it does not make sense when new notations are introduced ($m^{a}$ and $m^{p}$)\n\n### Section 4.3\n\n- This section really puzzled me for a while. The authors note \"We pre-train our model for 500k steps\", and in the next to next sentence they say \"Initialized by pre-trained weights of BERT_{base}, we continue training our models for 200k steps.\" Which one is true? Do the authors just used a pre-trained model (BERT_{base}) and then trained on their logic pre-training objective for 200k steps? If so, then that severely impacts the contribution of the paper as then the benefit of the pre-training tasks are masked by the initialization!\n\n### Table 1, Table 2, Table 5\n\n- It is not clear what these numbers represent - is it mean or median of multiple runs? Or is it a single run on the fine-tuning scores? The authors can consult Liu et al 2019. (RoBERTa) for a good reference on how to report the numbers.\n\n### Section 5.2 Results\n\n- \"continual trained for 200K steps for a fair comparison\" - this isn't clear. Does the authors mean they trained the baseline BERT_{base} for 200k steps?\n- Performing better on GLUE benchmark does not indicate a models better ability for reasoning, as these tasks have been extensively studied in the literature to contain annotation artefacts which the models exploit (Gururangan et al 2018, McCoy et al 2019, etc).\n- The claim that the model is robust as it gains consistent improvements on small and large datasets is also not clear to me. Robustness is typically measured by the model's ability to comprehend on different adversarial examples.\n\n### Section 6.2\n\n- Yet another introduction of a new notation, without re-using the ones already defined\n\n### Section 6.3\n\n- I have severe issues with this analysis. First of all, its a cherry-picked result of a single sentence. Secondly, attentions behave differently in different heads of the originally pre-trained BERT model. From the same paper authors cite (Clark et al 2019), the last layers of BERT typically act as a \"no-op\" operation. It does not make sense to compare the last layer with that of the proposed model to claim the model pays more attention to discourse level information, as due to new training objectives the attention maps might be simply re-arranged in different layers. Attention map comparison of a multi-headed BERT model to display linguistic prowess is also incorrect - there exists other methods to quantify the same, such as the probing literature.\n\n### Section 6.4\n\n- \"slope of the dashed line is more gentle\" -> how is it quantified? Is it purely based on visual inspection? A metric would be more convincing to quantify the rate of decline.\n- Similarly as of Table 1 and 2, the results of Figure 4 cannot be parsed properly as they lack error bars.\n\n### Section 6.5\n\n- The case study is also an instance of cherry-picking. It is good to see the proposed model being more robust to negation and entity change. However, it is hardly convincing as it is a single instance. If the authors want to establish this as a strength of the model they should perform rigorous examination of the entire evaluation set, using these intervention methods.\n\n## Grammatical issues\n\n### Abstract\n\n- \"logic reasoning ability\" -> \"logical reasoning ability\"\n- \"knowledge basis\" -> \"knowledge bases\"\n- \"logic reasoning\" -> \"logical reasoning\" (lots of instances in the whole paper)\n- \"We evaluate our model on a broad range of NLP and NLU tasks\" - NLU is a subset of NLP, so this statement doesn't make sense. Either \"broad range of NLP\" tasks or \"broad range of NLU tasks\"\n\n### Section 2.2\n\n- \"multi-granularity pre-training\" -> \"granular pre-training\"\n\n### Section 3.1\n\n- Is \"actees\" even a word?\n\n### Section 6.2\n\n- \"performance is hurt a lot\" -> usage of colloquial english is generally discouraged in scientific literature. Consider replacing by \"perform is hurt significantly\"\n\n", "summary_of_the_paper": "The paper proposes a new pre-training technique to induce a logical prior in the language model representation. Concretely, they propose pre-training on facts, represented as knowledge base triples (source, sink, relation) (knowledge-base completion) and link prediction, alongside traditional masked language modeling objective. Their proposed method achieves some improvement over downstream tasks, including a subset of GLUE benchmark and a couple of relation prediction datasets.\n\n", "summary_of_the_review": "Overall, the paper requires extensive re-writing and better result presentation to be considered for acceptance in the conference. While the paper introduces a nice pre-training technique, it raises way more questions than it answers. Ultimately, the empirical results (and especially the presentation of those results) are not convincing enough to support the claim that the proposed pre-training techniques at all induces logical reasoning capabilities to BERT model. At this stage, unfortunately I cannot recommend acceptance.\n", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "0IA3D9P1rgp", "reviewer": "Reviewer_cXhg", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths\n- Good and clear ablations to verify each pre-training objective, along with an assessment of longer term dependencies by context length\n- Approach and findings are important for the community in that this work uses no external knowledge bases for incorporating logical reasoning, positioned well relative to previous work\n\nWeaknesses\n- The emphasis on self-supervision could be stronger in the paper to make clear that it is a self-supervised approach to logical reasoning\n- The submission has areas where the technical explanation could be clearer (see questions below), and even the naming convention of the proposed pre-training objectives is not consistent\n- Several methodological choices were not justified (see questions below)\n- The attention matrix heatmap analysis is not convincing due to certain methodological choices\n\nQuestions\n- How exactly do you convert the dependency parse to the triplet of {$A_1, P, A_2$}? Please illustrate with an example\n- Why do you choose to model the logical graph as edges defined over relations between argument-predicates and between coreferents only? Are other relationships possible? Why did you not choose them?\n- In equation 3, there is no indication that the final training objective is weighted? Was a weighted sum used? What weights did you try and why?\n- In 4.3, it is unclear which model is pre-trained for 500k steps and which is fine-tuned from BERT for 200k steps. \n- If the baseline model is fine-tuned for 200k steps, why is that a sufficiently fair comparison with your pre-trained model with 500k steps?\n- In table 2: why are PROPHET's results clearly better than BERT large on LogiQA but perform worse / about the same on ReClor?\n- Why do you only choose the first head of the last attention layer for the attention matrix? What do you see for the other heads or other layers? I'm not convinced by this methodological choice\n\nPresentation concerns\n- Spelling of stabilized in Figure 1\n- If you specify what V is in 3.2, it is also good to specify what E is\n- The names used in Figure 2 for pre-training objective should correspond with those in 4.2\n- In 5.2, advise to keep Table 2 results separate from the list of Table 1 results\n- Suggest you show the BERT baseline in Table 4 for comparison with the ablations\n- In Table 6, are the figures from vanilla BERT or Prophet?\n- In Figure 5, typo \"nentailment\"\n", "summary_of_the_paper": "The goal of the paper is to incorporate logical relations into pre-training of language models to solve the reliance of existing reasoning-enabled language models on external knowledge bases. This is done in a self-supervised way - facts (tuple of 2 arguments and a predicate) are parsed using dependency parsing, and then a logical graph is created to denote relationships between coreferents, and between predicates and arguments. Three pre-training objectives are presented over the facts and logical graph: logical connective masking, logical structure completion and logical path prediction.\n\nThe author's claimed contributions are the following:\n- 3 new pre-training objectives: logical connective masking, logical structure completion, logical path prediction\n- The model Prophet which \"achieves significant improvement over various logic reasoning involved NLP and NLU downstream tasks\"\n- An analysis that verifies how Prophet is using the context for logical reasoning", "summary_of_the_review": "Overall, I think this paper is very marginally below the acceptance threshold. I like the self-supervised approach to logical reasoning in pre-training. My major concern is the clarity of the paper, which hinders some explanation of methodological choices and proper understanding of the technical approach. I am willing to move this paper above the acceptance threshold if the authors can address my questions appropriately.\n", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper proposes a pre-training technique for improving the logical abilities of pre-trained language models.
Reviewers point to many issues with clarity and experimental evaluation. No response was given by authors.
Decision: Reject