Logic-aware Pre-training of Language Models
Authors: Siru Ouyang
Zhuosheng Zhang
hai zhao
Venue: Submitted to ICLR 2023 (2023)

Abstract: Pre-trained language models (PrLMs) have been shown useful for enhancing a broad range of natural language understanding (NLU) tasks. However, the capacity for capturing logic relations in challenging NLU still remains a bottleneck even for state-of-the-art PrLM enhancement, which greatly stalls their reasoning abilities. To bridge the gap, we propose logic pre-training of language models to equip PrLMs with logical reasoning ability. To let logic pre-training perform on a clear, accurate, and generalized knowledge basis, we introduce \textit{fact} instead of the plain language unit in previous PrLMs. The \textit{fact} is extracted through syntactic parsing in avoidance of unnecessary complex knowledge injection. Meanwhile, it enables training logic-aware models to be conducted on a more general language text. To explicitly guide the PrLM to capture logic relations, three complementary self-supervised pre-training objectives are introduced: 1) logical structure completion to accurately capture fact-level logic from the original context, 2) logical path prediction on a logical graph to uncover global logic relationships among facts, 3) logical connectives masking to capture discourse-level for fact groups. We evaluate our model on a broad range of NLP tasks, including natural language inference, relation extraction, and machine reading comprehension with logical reasoning. Experimental results show that our model achieves significant performance in all the downstream tasks, especially in logical reasoning-related tasks. 

Under review as a conference paper at ICLR 2023
LOGIC PRE-TRAINING OF LANGUAGE MODELS
Anonymous authors Paper under double-blind review
ABSTRACT
Pre-trained language models (PrLMs)) have been shown useful for enhancing a broad range of natural language understanding (NLU) tasks. However, the capacity for capturing logic relations in challenging NLU still remains a bottleneck even for state-of-the-an PrLM enhancement, which greatly stalls their reasoning abilities. To bridge the gap, we propose logic pre-training of language models to equip PrLMs with logical reasoning ability. To let logic pre-training perform on clear, accurate, and generalized knowledge basis, we introduce fact instead of the plain language unit in previous PrLMs. The fact is extracted through syntactic parsing in avoidance of unnecessary complex knowledge injection. Meanwhile, it enables training logic-aware models to be conducted on a more general language text. To explicitly guide the PrLM to capture logic relations, three complementary self- supervised pre-training objectives are introduced: 1) logical structure completion accurately capture fact-level logic from the original context, 2) logical path prediction on logical graph to uncover global logic relationships among facts, 3) logical connectives masking capture discourse-level for fact groups. We evaluate our model on a broad range of NLP tasks, including natural language inference, relation extraction, and machine reading comprehension with logical reasoning. Experimental results show that our model achieves significant performance in all the downstream tasks, especially in logical reasoning related tasks.
INTRODUCTION
Machine reasoning in natural language understanding (NLU) aims to teach machines to understand human languages by building and analyzing the connections between the facts, events, and observations using logical analysis techniques like deduction and induction, which is one of the ultimate goals towards human- paanty intelligence. Although pre-trained language models (PrLMs), such as BERT (Devlin al., 2018), GPT (Radford et al., 2018), XLNet (Yang et al.] 2019) and RoBERTa (Liu al. 2019), have established state-of-the-art performance on various aspects in NLU, they are still short complex language understanding tasks that involve reasoning (Helwe et al. 2021). The major reason behind this is that they insufficiently capable of capturing logic relations such as negation (Kassner& & Schütze 2019), factual knowledge (Poerner et al.) 2019). events (Rogers al. 2020), and so Many previous studies (Sun et al. 2021 Xiong al. 2019, Wang al. 2020) are then motivated to inject knowledge into pre-trained models like BERT and RoBERTa. However, they too much rely on massive external knowledge sources and ignore that language itself is a natural knowledge carrier as the basis of acquiring logic reasoning ability (Ouyang et al. 20211). Taking the context in Figure [ as an example, previous approaches tend to focus on entities such as the definition of "goovenment" and the concepts related to it like "governor", but overlook the exact relations inherent in this example, thus failing model the complex reasoning process.
Given the fact that PrLMs are the key supporting components in natural language understanding, in this work, we propose a fundamental solution by empowering the PrLMS with the capacity of capturing logic relations, which is necessary for logical reasoning. However, logical reasoning can only be implemented on the basis of clear, accurate, and generalized knowledge. Therefore, we leverage fact as the conceptual knowledge unit to serve the basis for logic relation extraction. Fact is organized as a triple, i.e., in the form of madiicate-aguumen structures, to represent the meaning such "who-did- mhno-did-wha--eee what-to- whht-ttowhhmm and "who-is-wer whht". Compared with existing studies that inject complex knowledge like knowledge graphs, the knowledge structure based on fact far less complicated and more general in representing events and relations in languages.

---

Under review as a conference paper at ICLR 2023
On top of the fact-based knowledge structure, we present PROPHET, logic-aware pre-trained language model to learn the logical relations in a universal way from very large corpus. Given syntactically extracted fact, instead of simply syntactic-level structure of text (Xu et al. 2020), we leverage multi-dimensional logical information for better representations. In detail, we introduce three novel pre-training objectives based the newly introduced knowledge structure basis fact: 1) logical connectives masking for learning sentence-level logic connection. 2) logical structure completion task on top of facts for regularization, aligning extracted fact with the original context. 3) logical path prediction to capture the logic relationship between facts. PROPHET is based on Transformer( Vaswani et al. 2017) neural architecture.
We pre-train PROPHET on the traditional English Wikipedia corpus, which includes 1.1M articles in total. We evaluate the model on a broad range of language understanding tasks: natural language inference, semantic similarity, machine reading comprehension, etc. Experimental results show that the fact is useful as the carrier for knowledge modeling, and the newly introduced pre-training tasks can improve PROPHET and achieves significant performance on downstream tasks
In summary, the contributions of this paper are: (1) PROPHET takes the initial attempt to pre- train language models that leverages the conceptual knowledge unit fact to equip PrLMs with logical reasoning ability. (2) We introduce three novel pre-training tasks for learning logic-aware representations from texts and facts. (3) PROPHET provides significant improvement on various downstream tasks, including logical reasoning of texts, general natural language understanding, and document-level relation extraction.
RELATED WORK
2.1 PRE-TRAINED LANGUAGE MODELS IN NLP
Large pre-trained language models (Devlin et al., 2018 Liu et al. 2019 Radford et al. 2018) have brought dramatic empirical improvements on almost every NLP task the past few years. A classical norm of pre-training is to train neural models on a large corpus with self- pre-training objectives. "Self-supervised" means that the supervision provided the training process is automatically generated from the raw text instead of manually generation. Designing effective criteria for language modeling is one of the major topics in training pre-trained models, which decides how the model captures the knowledge from large-scale unlabeled data. The most popular pre-training objective used today is masked language modeling (MLM), initially used in BERT (Devlin et al. 2018), which randomly masks out tokens, and the model is asked to uncover it given surrounding context. Recent studies have investigated diverse variants of denoising strategies (Raffel et al. 2020 Lewis al., 2020), model architecture (Yang et al. 2019), and auxiliary objectives (Lan al. Joshi et al.) 2020) to improve the model strength during pre-training. Although the existing techniques have shown effectiveness in capturing syntactic and semantic information after large-scale pre-training, they perform sensitivity to role reversal and struggles with pragmatic inference and role-based event knowledge (Rogers et al. 2020), which are critical to the ultimate goal of complex reasoning that requires to uncover logical structures. However, it is difficult for pre-trained language models to capture the logical structure inherent in the texts since logical supervision is rarely available during pre-training. Therefore, we are motivated to explicitly guide the model to capture such clues via our newly introduced self-supervised tasks.
2.2 REASONING ABILITY FOR PRE-TRAINED LANGUAGE MODELS
Previous works have extensively investigated the possibility of injecting specific type of reasoning via pre-training. For reasoning types, Pi et al. (2022 Geva et al. (2020) focuses on numerical reasoning;
Staliunatt et al. (202T): T'abborrino et al. (2020) target at commonse reasoning; /Zhng et al. (2022)
deals with multi- reasoning. Different from them, PROPHET concentrates on logical reasoning, which plays a fundamental role in general reasoning via natural language.
There is also a lot of work in the research line of enhancing general reasoning abilities in pre-trained language models. The existing approaches mainly design novel pre-training objectives and leverage abundant knowledge sources such as WordNet (Miller 1995). Notably, ERNIE 3,0 (Sun al. 202
Our codes have heen uploaded supplemental material, which will he open after the double review period.
2

---

Under review as a conference paper at ICLR 2023
Fact
Logical Graph
opposite
anarchists, participated, revolution revolution, opposite, movement
participated
movement
anarchists
revolution
after
they, met, suppression
coraf
suppression
stabalized
suppresion, after, stabilized
they
was
same
government, was, stabilized
government
stabalized
Despite concern, anarchists participated in the Russian Revolution in opposition to White movement However they met harsh suppression after the Rolshovik government stahilzedi
Text
Figure I: How the facts and logical graph constructed from raw text inputs. Edges in red denotes additional edges added in the logical graph, while text with green indicates the sentence-level logical connectives which will be mentioned in $4
uses broad range of pre-training objectives from word-aware, structure-aware to knowledge-awee tasks, based on 4TB corpus consisting of plain texts and large-scale knowledge graph. WKLM (Xiong et al. 2019) replaces entity mentions in the document with other entities of the same type, and the objective is to distinguish the replaced entity from the original ones. KEPLER (Wang et al. 202I) encodes textual entity descriptions using embeddings from PrLM to take full advantage of the abundant textual information. K-Adapter (Wang et al. (020)) designs neural adapters to distinguish the type of knowledge sources to capture various knowledge. Our proposed method differs from previous studies three aspects. Firstly, our model does not require any external knowledge resources like previous methods that WordNet, Wikilata, etc. We only use small-scale textual sources following the standard PrLMs like BERT (Devlin et al. 2018). along with an off-the-shelf dependency parser extract facts. Secondly, previous works only consider triplet-level pre-training objectives. We proposed multi-dimen dimensional pre-training strategy, considering not only triplet-level information but also sentence-level and global knowledge enhance logic reasoning. Finally, we propose a new training mechanism apart from masked language modeling (MLM), hoping to shed light on more logic pre-training strategies in this research line.
PRELIMINARIES
In this section, we will introduce the concept fact and logical graph, which serves the basis of PROPHET. We will also describe extracting the facts for logical graph construction from the plain texts, as an example shown Figure]
3.1 FACT
Following Nakashole Mitchell (0014) and Ouyang et al. (2021), extract facts which are triplets represented as 7 [A,, P, A2 where A₁ and Ag are the arguments and P is the predicate between them. can well represent broad range of facts, reflecting the notion of whatddd----tt-wwieeo and stho-is-what"" etc.
We extract such facts in a syntactic way, which makes our approach generic and easy to apply. Given document, we first split the document into multiple sentences. For each sentence, we conduct dependency parsing using StanfordCoreNIPP (Manning et al. 2014) For the analyzed dependencies, basically, we consider verb phrases and some prepositions in the sentences as "predicates", and then we search for their corresponding actors and actees as the "arguments".
3.2 LOGICAL GRAPH
logical graph an undirected (but is required be connected) graph that represents logical dependency relation between components in facts. In logical graphs, nodes represent
hhttps://stanfordmn
italb..c///rree /c/Ceee/P/ we also tried to OpenIE directly; however,
the performance is satisfactory.
3

---

Under review as a conference paper at ICLR 2023
argument/predicates in the fact, and edges indicate whether two nodes have relations in fact. Such a structure can well unveil and organize semantic information captured by facts. Besides, a logical graph supports considerations among long-range dependencies via connecting arguments and their relations in different facts across different spans.
We further show how construct such graphs based on facts. Despite given relations in facts, design another two types of edges based on identical mentions and coreference information. (1) There can be identical mentions in different sentences, resulting repeated nodes facts. We connect nodes corresponding to the same non-pronoun arguments by edges with edge type same. (2) We conduct coreference resolution on context using an off-to-shelf model to identify arguments in facts that refer to the same one We add edges with type coref between them. The final logical graph is denoted as S (V, F.,, where A; and E (1,
anarchists, participated, revolution
revolution, opposite, movement
after
stabilized
they, met, suppression auppresion, after, stabilioed government was, stabilized
Fact
pre-training with fact-aware logics
Text
However, they met harsh suppression after the Bolshevik government...
Text Encoder
senterce cornective masking
fact unit aignment
Ingical path prediction
Logical Graph
sppcsite
participated
movement
(CLS) suppression (MASK] the Bolshevik.... ISEP)
suppression, after, revolution, government
anarchists
after
suporession
suppression
Baishewk...
suporrssion, after, stabilized)
revolution, government
they geveriment
stabalzed
Mame
Text
Fact Unit
Node Pairs
Figure 2: An illustration about pre-training methods used in PROPHET. The model takes the text,
extracted fact and the randomly sampled node pairs in the logical graph as the imput. The model is pre-trained with three novel objectives. One is the standard masked language modeling applied to sententious connectives, the others are fact alignment and logical path prediction.
PROPHET PRE-TRAINING OBJECTIVES
We describe three pre-training tasks used for pre-training PROPHET in this section. Figure is an illustration of PROPIIE pre-training. The three pre-training tasks are based plain texts, extracted fact units, and constructed logical graphs respectively. The first task is logical connectives masking (LCM) generalized from masked language modeling (Devlin et al., 2018) for logical connectives to learn sentence-leve representation. The second task is logical structure completion (LSC) for learning logic relationship inside fact, where we first randomly mask arguments in facts, and then predict those items. Finally, logical path prediction (LPP) task is proposed for recognizing the logical relations of randomly selected node pairs.
Logical Connective Masking Logical connective masking is an extension of the masked language modeling (MI.) pre-training objective in DDevlin et al. (208)) 8), with a particular focus on connective indication tokens. We use the Penn Discourse TreeBank 2.0 (PDTB) (Prasad et al. 2008) to draw the logical relations among sentences. Specifically, PDTB 2.0 contains relations that are manually annotated on the million Wall Street Journal (WSJ) corpus and broadly characterized into "Explicit" and "Implicit" connectives. We use the "Explicit" type (in total 100 such connectives), which apparently presents in sentences such as discourse adverbial "instead" subordinating conjunction "because". Taking all the identified connectives and some randomly sampled other tokens (for a total 15% of the tokens of the original context), we replace them with a [MASK] token 80% of the time, with a random token 10% of the time and leave them unchanged 10% of the time. The MLM objective is to predict the original tokens of these sampled tokens, which has proven effective in previous works (Devlin et al. 2018 Liu et al. 2019). In this way, the model learns
https:///gii thtb. http:////ggttttbccc//////ggna//a coref
4

---

Under review as a conference paper at ICLR 2023
to recover the logical relations for two given sentences, which helps language understanding. The objective of this task is denoted as ccon..
Logical Structure Completion To align representation between the context and the extracted fact, we introduce pre-training task of logical structure completion. The motivation here is to encourage the model learn the structure-aware representation that encodes the WWh-did--hat-tto --otto"-h"o like meanings for better language understanding. To speak in detail, we randomly select a specific proportion of the total facts (A 20% in this work), from a given context. For each chosen fact, we either ask the model to complete are selected based on equal probability). We denote all the blanks that need to be completed as m and m², denoting arguments and predicates, respectively, In implementation, this objective is the same as masked language modeling to keep simplicity, by using the original loss following Devlin
Eegument-Predictte.
or "Argument-?" A-grrmmmn (the templates
al. 2018).
Lulign
Σ
log D(x |m mm"m"";
sEaUp
where is the discriminator to predicts a token from large vocabulary.
Logical Path Prediction To learn representation from the constructed logical graph, thus endowing the model with global logical reasoning ability, we propose the pre-training task of predicting whether there exists path between two selected nodes in the logical graph. In this way, the model learns to look at logical relations across a long distance of arguments and predicates in different facts.
We randomly sample 20% nodes from logical graph to form set V there are total CPe node pairs. have maximum number maz, of node pairs to predict. To avoid bias the training process, we try to make sure that - are positive samples and the rest are negative samples, thus balancing positive-negaiie negative ratios. If the number of positive/negative samples is less than "aae we just keep the original pairs. Formally, the pre-training objective of this task calculated as below following
Guo et al. (2020):
Path =
Σ VLyEV"
18 log olv, uj] + (1 - 8) log(1 ofv, e,])],
where d is when i; and vy have connected path and 0 otherwise. [u), By] denotes the concatenation of representations of e and Vy.
The final training objective is the weighted sum of the above mentioned three losses.
C = Leonn + Lalign + Patth-
(3)
5 EXPERIMENTS
5.1 MODEL. ARCHITECTURE
We follow BERT (Devlin et al. 2018) and use a multi-layer bidirectional Transformer (Vaswani et al. 2017) as the model architecture of PROPHET. For keeping the focus the newly introduced techniques, we will not review the ubiquitous Transformer architecture PROPHET by using exactly the same model architecture BERT-base, where the model consists of 12 transformer layers, with 768 hidden size, 12 attention heads, and 11M model parameters total.
detail. We develop
5.2 PRE-TRAINING SETUP
We use the English Wikipedia (1.1 million articles in total), we sample the train and valid datasets with split ratio of 19 1 on the original datasets. We omit the "Reference" and "Literature" part in a document to ensure data quality. Following the previous practice (Devlin et al.) 2018), we limit the length of sentences in each batch up to 512 tokens and the batch size is 128. We use Adam Kingma& & Ba 2014) with 31 = 0.9, 3g = 0.98 and e = le - 6, and weight decay is set as 0.01. We pre-train deepspeed for training acceleration. Initialized by the pre-trained weights of BERT bases we continue training models for 2001 steps.
model for 500k steps. We use 8 NVIDIA V100 32G GPUs, with FP16 and
5

---

Under review as a conference paper at ICLR 2023
5.3 TASKS AND DATASETS
Our experiments are conducted on a broad range of language understanding tasks, including natural language inference, machine reading comprehension. semantic similarity. and text classification. Some of these tasks are a part of GLUE (Wang et al.) 2018) benchmark. We also extend our experiments to DocRED (Yao et al.) 2019). widely used benchmark of document-level relation extraction for generalizability. To verify our model's reasoning abilities of logic, we perform experiments on two recent logical reasoning datasets the form machine reading comprehension, ReClor Yu et al. 2020) and LogiQA (Liu et al. 2020).
Model
Classification
Language Inference
Semantic Similarity
Avg.
CoLA SST-2
MNLI QNLI RTE
MRPC QQP
STS-B
In literalure
BERT bese
52.1
93.5
84.6/83.4
90.5
66.4
88.9
71,2
85.8
79.6
SemBERT base
57.8
93.5
84.4/84.0
90.9
69.3
88.2
71.8
87.3
80.8
Our implementation
BERT
baae base
53.6
93.5
84.6/83.4
90.9
66.6
88.6
71,2
85.8
79.8
PROPHET base
57.0
93.9
85.3/84.3
91.4
69.8
89.5
72.0
86.0
81.1(+1.3)
BERTlarge
60.5
94.9
86.7/85.9
92.7
70.1
89.3
72.1
86.5
82.1
PROPHET\arge 64.1
95.3
87.1/86.5
93.0
72.8
90.1
72.7
87.3
83.2 (11.1)
Table 1: Leaderboard results on GLUE benchmark. The number below each task denotes the number of training examples. Fl scores are reported for QQP and MRPC, Spearman correlations are reported for STS-B, and accuracy scores are reported for the other tasks.
Model
ReClor
LogiQA
Dev
Test
Test-E
Test-H
Dev
Test
Human Performance*
63.0
57.1
67.2
86.0
BERT base
51.2
47.3
71.6
28.2
33.8
32.1
PROPHETOase buse
53.4 (12.2)
48.8 (11.5)
72.4 (f0.8)
32.2 (14.0)
35.2 (f1.4) 34.1 (12.0)
BERT large
53.8
49.8
72.0
32.3
31.0 35.4 (f12 34.5 (13.5)
34.1
PROPHETTTagge
55.8 (12.0)
51.5 (t1.7) 73.0 (+10) 36.6 (14.3)
Table 2: Accuracy on ReClor and LogiQA dataset. The public methods are based on large models.
5.4 RESULTS
Table ! show results on several benchmark datasets for measuring the language understanding and logical reasoning abilities of PROPHET. We have the following observations.
(1) PROPHET obtains substantial gains over the BERT baseline models (we continual trained the BERT baseline for 200K steps for fair comparison), indicating that our model is effective in general sense of language understanding. In addition, our model is able to generalize to different model scales as it can perform well on both BERT base and BERT models.
(2) PROPHET performs particularly well on language inference tasks including MNLI, QNLI, and RTE compared with other tasks such as MRPC for semantic similarity. This indicates our model's ability of reasoning in natural language understanding.
(3) Whether it is large-scale datasets such as QQP and MNLI or small datasets like COLA and SST-B, our model demonstrates consistent improvement, indicating its generalization across different scale and scope NLU tasks.
We exclude the prohlematic WNLI set.

---

Under review as a conference paper at ICLR 2023
Dev
Test
Model
F1
Intra-F1
Inter-F1
F1
BERTbasd
Devlin et al. 2018)
54.2
61.6
47.2
53.2
Two-Phase BERT " (Wang el 2019
54.4
61.8
47.3
53.9
PROPHET
54.8 (10.6) 62.4 (10.8)
47.5 (10.3) 54.3 (11.1)
Table Main results on the dev and test set for DocRED. * indicates that the results are taken from Nan et al. (2020). Intra- and Inter-Fl indicates Fl scores for the intra- and inter-sentence relations following the setting of Nan al. (2020).
we observe that PROPHET improves the logical reasoning ability of BERT baseline by large margin on the benchmark datasets ReClor and LogiQA. Especially, armed with our approach, even on the backbone of the BERT-base model, the performance of our model is on par with or even surpasses the BERT--arge baseline. Also, we noticed that PROPHET's results are clearly better on LogiQA but perform worse or about the same on ReClor with BERT large- This is due to the more abstractive logical relations inherent in LogiQA datasets compared with ReClor, which needs explicit logical representations to cope with. The better improvement on LogiQA proves the effectiveness the proposed PROPHET for better logical reasoning abilities. Also, PROPHET even demonstrates larger performance gains with BERTlerge backbone, which shows its generalizability on different model scales for downstream logical reasoning tasks.
(4) From Table
(5) In addition, we conducted experiments level relation extraction Yao et al. 2019). The results are shown in Table 3 From the table, we see that PROPHET still does well for relation extraction for documents by outperforming the baseline substantially. It even surpasses the two-phase BERT Also, our model is especially good at coping with inter-sentence relations compared with baseline models, which means that model is indeed capable of synthesizing the information across multiple sentences of a document, verifying the effectiveness of global information.
large-scale human-annotated dataset for document-
ANALYSIS
6.1 ABLATION STUDY
To investigate the impacts of different objectives introduced, we evaluate three variants of PROPHET as described in Section 4 1) the w/o LCM model adopts substitute without logical connectives masking as the pre-training objective, 2) the w/o LSC model is such that it leaves out the logical structure completion objective, and 3) the w/o LPP model only uses the objectives of connective masking and structure completion. The results are shown in Table
Based on the ablation studies, we come to the following conclusions. Firstly, all three components contribute to the performance removing any one of them causes performance drop the average score. Especially, the average point drops the most as we remove the logical path prediction objective, which sheds light on the importance of modeling chain-like relations of events. Secondly, we see that logical path prediction contributes the most to the reasoning abilities as the performance on language inference improves the most when we add the sententious connective masking objective and the task of logical path prediction.
6.2 COMPARISON BETWEEN FACT AND ENTITY-LIKE KNOWLEDGE
We also replace the injected fact with common practice using entity-like knowledge, which is using named entities. In detail, we change the arguments in facts into named entities recognized by StanfordCoreNlL.P and leave the predicates extracted unchanged, resulting the form of < N predicate N Eg > (NE stands for named entity). If fact is not recognized with any named entities, we just leave it out.
We only report the results for Ign F1 in the annotated setting as the distant supervision is too slow to train.
thu.. i C/CeNN/
7

---

Under review as a conference paper at ICLR 2023
Model
Classification
Language Inference
Semantic Similarity
Avg.
CoI.A
SST-2
MNLI
QNLI
RTH
MRPC
QQP
STS-B
PROPHET
57.0
93.9
85.3/84.3
91.4
69.8
89.5
72.0
86.0
81.1
w/o LCM
53.6
93.5
85.1/84.0
90.9
68.2
88.6
71.2
85.8
80.1
w/o LSC
53.6
93.6
85.0/84.1
91.3
69.0
88.9
71.4
85.9
80.3
w/o LPP
52.1
93.0
84.6/83.4
90.9
66.4
88.6
71.2
85.8
79.6
Table 4: Ablation studies of PROPHET on the test set of GLUE dataset.
Model
Classification
Language Inference
Semantic Similarity
Avg.
CoLA
SST-2
MNLI
QNLI
RTE
MRPC
QQP
STS-B
PROPHET
57.0
93.6
85.0/84.1
91.4
69.8
89.2
71.4
86.0
81.0
w/ named entities
50.4
93.2
84.9/84.2
90.8
68.7
88.4
71.0
84.9
79.3
Table 5: Results on GLUE test set when replacing facts with named entities and key the relations unchanged.
The results shown in Table 5 We vanilla BERT. This is quite intuitive as the number named entities is far less than our obtained fact, missing lot of information inherent in the context. Whereas introduced fact can well capture the knowledge used in the reasoning process, providing a fundamental reasoning basis.
that the performance is hurt a lot, even worse than
6.3 INTERPRETATION VIA ATTENTION MATRIX HEATMAP
We plot the attention matrix in token level to see how our model interprets the context using heatmap shown in Figure
of
11
Figure 3: Heatmap of the attention matrix (left) of vanilla BERT and our implemented PROPHET for the sentence, "However, they met harsh suppression after the Bolthevik government was stabilized." Weights are selected from the first head of the last attention layer. Accuracy (right) of different context length on MNLI-match (left) and MNLI-mismatch (right) dev set (about 1000 samples in each interval).
From the figure, we can see that the vanilla BERT attends to delimiters, particularly punctuation suggested in Clark al. (2019). In comparison, model exhibits quite different attention distribution. Firstly, our model clearly decreases the influences introduced by punctuation. Secondly, our model pays more attention tokens representing discourse-level information, such as "however" and "after", which is consistent with our motivation. It also well captures the relations of pronouns. The event characteristics are also illustrated as seen from the "after suppression' " phrases.
6.4 EFFECT OF DIFFERENT CONTEXT LENGTII
We group samples into ten subsets according to an equal amount of samples (around 1000 samples per interval) by context length since the majority of the samples concentrate the interval of under 60. The statistics of MNLI-matchar and miimmmismt dev sets are shown in Table Then we calculate the accuracy of the baseline and PROPHET per group for both the matched and mismatched
8

---

Under review as a conference paper at ICLR 2023
shown in Figure. We observe that the performance of the baseline groups drops dramatically when encountered with long contexts, especially for those longer than 45 tokens, while our model performs more robustly on those intervals (the slope of the dashed line is more gentle).
Table 6: Distribution of context length on dev set of MNLI-matched and MNLI- mismatched dataset.
Dataset
[0, 29)
[30, 59)
[60, 89)
[90, 119)
[120, 1499
[150, 179)
[180, 209)
[210, 239)
MNLI-matched
39.2%
49.8%
9.6%
1.0%
0.12%
0.12%
0.02%
0.06%
MNLI-mismatched
33.7%
55.0%
9.7%
1.7%
0.3%
0.1%
0.1%
0%
6.5
CASE STUDY
We also give case study to demonstrate that PROPHET could enhance the reasoning process, in language understanding. Given two sentences, we use PROPHET and BERT-base to predict whether the sentences are entailed not. Results are shown in Figure 4 To see the language understanding ability of our model, we made two subtle changes in the original training sample. Firstly, we change the entity referred to in the sentence. We between entities than BERT-base model. Additionally, we add a negation this change is small, it completely changes the semantic of the sentence, and leads a reversal of the ground-trutt labels. We can see that PROPHET is good at all the samples given, indicating that is not only good reasoning in language understanding, but also is more robust than baseline models.
that PROPHET learns better alignment relations the sentence. Although
Unchanged
Entity change
Negation
Sentence 1: Note that SBB, CFF and FFS stand out for the main railway company, in German, French and Italian..
Sentence 1: Note that SB8, CFF and FFS stand out for the main railway company, in German, French and Italian..
Sentence 1: Note that SBB, CFF and FFS stand out for the main railway company, in German, French and Italian..
Input
Sentence 2: The French railway company called SNCF.
Sentence The French railway company called SBB..
Sentence 2:
The French railway company is not called SNCF.
Label
not entailment
not entailment
entailment
Prophet: not entailment /
Prophet: not entailment
Prophet: nentailment /
Prediction
BERT-base: not entailment /
BERT-base: entailment
BERT-base: not entailment x
Figure 4: We take an example from RTE dataset, and use PROPHET and BERT-base to predict the label of the relations among two given sentences.
6.6 MULTI-TASK FINE-TUNING
To test whether the proposed pre-training objectives could be used for fine-tuning already pre- trained model, we directly train the BERT-base models along with our pre-training objectives. We conduct experiments on ReClor dataset. PROPHET achieves 48.0 on the Test set, with 71.9 and 31.0 on Test-E and Test-H set respectively. This shows another direct application of using our proposed objectives improve performance in the downstream tasks.
CONCLUSION
In this paper, we leverage fact in newly pre-trained language model PROPHET to capture logic relations essentially, in consideration of the fundamental role of PrLM serving NLP and NLU tasks. We introduce three novel pre-training tasks and show that PROPHET achieves significant improvement over various logic reasoning involved NLP and NLU downstream tasks, including language inference. sentence classification, semantic similarity, and machine reading comprehension. Further analysis shows that our model can well interpret the inner logical structure the context to aid the reasoning process.

---

Under review as a conference paper at ICLR 2023
REFERENCES
Kevin Clark, Urvashi Khandelwal, Omer Levy, and Christopher D) Manning. What does bert look at?
an analysis of bert's attention. arXiv preprint arXiv: 2019,
Jacob Devlin, Ming-Wai Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805 2018.
Mor Geva, Ankit Gupta, and Jonathan Berant. Injecting numerical reasoning skills into language models. arViv preprint arXiv:2004. 04487, 2020.
Daya Guo. Shuo Ren, Shuai Lu, Zhangyin Feng. Duyu Tang, Shujie Liu, Long Zhou, Nan Duan, Alexey Svyatkovskiy, Shengyu Fu, et al. Graphcodebert: Pre-training code representations with data flow. arXiv preprint 2r0i 08366, 2020.
Chadi Helwe, Chloé Clavel, and Fabian M Suchanek. Reasoning with transformer-based models: Deep learning, but shallow reasoning. In 3rd Conference Automated Knowledge Base Construction, 2021.
Mandar Joshi, Danqi Chen, Yinhan Liu, Daniel S. Weld, Luke Zettlemoyer, and Omer Levy. SpanBERT: Improving pre-training by representing and predicting spans. Transactions of the Association for Computational Linguistics, 8:64-77, 2020.
Nora Kassner and Hinrich Schütze. Negated and misprimed probes for pretrained language models: Birds can talk, but cannot fly. arXiv preprint arXiv: 1.0334,, 2019.
Diederik P Kingma and Jimmy Ba. Adam: method for stochastic optimization. arXiv preprint arXiv:1412.6980, 2014.
Zhenzhong Lan, Mingda Chen, Sebastian Goodman, Kevin Gimpel, Piyush Sharma, and Radu Soricut.
ALBERT: A lite BERT for self-supervised learning of language representations. In International Conference Learning Representations, 2019. URL httpas/////penneeeeeeeeeee/eeees
EleA7AEtv3
Mike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, Abdelrahman Mohamed, Omer Levy, Veselin Stoyanov, and Luke Zettlemoyer Bart: Denoising sequence-to-sequenee pre-training for natural language generation, translation, and comprehension. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pp. 7871-7880, 2020.
Jian Liu, Leyang Cui, Hanmeng Liu, Dandan Huang, Yile Wang, and Yue Zhang. Logiqa: challenge dataset for machine reading comprehension with logical reasoning. In Christian Bessiere (ed.), Proceedings of the Twenty-Ninth International Joint Conference on Artificial Intelligence, IJCAI-20, pp. 3622-3628. International Joint Conferences Artificial Intelligence Organization, 7 2020. doi: 10.24963/iccai.2220550. URL Utttttss///aa..rr///22228888/ Main track.
Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. Roberta: A robustly optimized bert pretraining approach. arXiv preprini arXiv: 1907. 1692, 2019.
Christopher D Manning, Mihai Surdeanu, John Bauer, Jenny Rose Finkel, Steven Bethard, and David McClosky. The stanford corenlp natural language processing toolkit. In Proceedings of 52nd annual meeting of the association for computational linguistics: system demonstrations, pp. 55 60, 2014.
George A Miller. Wordnet: a lexical database for english. Communications of the ACM, 38(11): 39-41, 1995.
Ndapandula Nakashole and Tom Mitchell. Language-aware truth assessment of fact candidates. In Proceedings of the 52nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 1009-1019, 2014.
Guoshun Nan, Zhijiang Guo, Ivan Sekulić, and Wei Lu. Reasoning with latent structure refinement for document-level relation extraction. arXiv preprint arXivv2005.063/2. 2020.
10

---

Under review as a conference paper at ICLR 2023
Siru Ouyang, Zhuosheng Zhang, and Hai Zhao. Fact-driven logical reasoning. arXiv preprint arXiv:210S.10334 2021.
Xinyu Pi, Qian Liu, Bei Chen, Morteza Ziyadi, Zeqi Lin, Yan Gao, Qiang Fu, Weizhu Chen. Reasoning like program executors. arXiv preprint arXiv:2201.1177,, 2022.
Lou, and
Nina Poerner, Ulli Waltinger, and Hinrich Schütze. Bert is not a knowledge base (yet): Factual knowledge vs. name-based reasoning in unsupervised qa. arXiv preprint arXiv:1911.03653 2019.
Rashmi Prasad, Nikhil Dinesh, Alan Lee, Eleni Miltsakaki, Livio Robaldo, Aravind K. Joshi, and Bonnie L Webber. The penn discourse treebank 2.0. In LREC. Citeseer, 2008
Alec Radford, Karthik Narasimhan, Tim Salimans, and Ilya Sutskever. Improving language understanding by generative pre-training. 2018
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. Exploring the limits of transfer learning with a unified text-to-text transformer. Journal of Machine Learning Research, 21:1-67, 2020.
Anna Rogers, Olga Kovaleva, and Anna Rumshisky. A primer in bertology: What we know about how bert works. Transactions of the Association for Computational Linguisties, 8:842-866 866, 2020.
leva Staliūnaite, Philip John Gorinski, and Ignacio Iacobacci. Improving commonsense causal reasoning by adversarial training and data augmentation. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 35, pp. 13834-13842, 2021.
Yu Sun, Shuohuan Wang, Shikun Feng, Siyu Ding, Chao Pang, Junyuan Shang, Jiaxiang Liu, Xuyi Chen, Yanbin Zhao, Yuxiang Lu, al. Ernie 3.0: Large-scale knowledge enhanced pre-training for language understanding and generation. arXiv preprint arXiv:2 107. (0.0000...77. 137, 2021.
Alexandre Tamborrino, Nicola Pellicano, Baptiste Pannier, Pascal Voitot, and Louise Naudin. Pre- training is (almost) all you need: An application to commonsense reasoning. arXiv preprint arXiv:2004. 14074, 2020.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Advances in neural information processing systems, pp. 5998-6008, 2017.
Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R Bowman. Glue: multi-task benchmark and analysis platform for natural language understanding. arXiv preprint 1rXhv:1804.0746: 2018.
Hong Wang. Christfried Focke, Rob Sylvester, Nilesh Mishra, and William Wang. Fine-tune bert for docred with two-step process. arXiv preprint 190.v:1909.1189 2019.
Ruize Wang, Duyu Tang, Nan Duan, Zhongyu Wei, Xuanjing Huang, Guihong Cao. Daxin Jiang, Ming Zhou, et al. K-adapter: Infusing knowledge into pre-trained models with adapters. arXiv preprint ar arXiv:2002.0 0180 2020.
Xiaozi Wang, Tianyu Gao, Zhaocheng Zhu, Zhengyan Zhang, Zhiyuan Liu, Juanzi Li, and Jian Tang. Kepler: A unified model for knowledge embedding and pre-trained language representation. Transactions of the Association for Computational Linguistics, 9:176-194, 2021.
Wenhan Xiong, Jingfei Du, William Yang Wang, and Veselin Stoyanov. Pretrained encyclopedia: language model. arXiv preprint arXiv: 2019.
Weakly supervised
Zenan Xu, Daya Guo, Duyu Tang, Qinliang Su, Linjun Shou, Ming Gong, Wanjun Zhong, Xiaojun Quan, Nan Duan, and Daxin Jiang. Syntax-enhanced pre-trained model. arXiv preprint orXiv:2012.141166 2020.
Zhilin Yang, Zihang Dai, Yiming Yang, Jaime Carbonell, Russ R Salakhutdinov, and Quoc V Le. Xlnet: Generalized autoregressive pretraining for language understanding. Advances in neural information processing systems, 32, 2019.
11

---

Under review as a conference paper at ICLR 2023
Yuan Yao, Deming Ye, Peng Li, Xu Han, Yankai Lin, Zhenghao Liu, Zhiyuan Liu, Lixin Huang, Jie Zhou, and Maosong Sun. Docred: A large-scale document-level relation extraction dataset. arXiv preprint MXXYv:196600122. 2019.
Weihao Yu, Zihang Jiang, Yanfei Dong, and Jiashi Feng. Reclor: A reading comprehension dataset requiring logical reasoning. In International Conference on Learning Representations (ICLR), April 2020.
Wanjun Zhong, Junjie Huang. Qian Liu, Ming Zhou, Jiahai Wang. Jian Yin, and Nan Duan. Reasoning over hybrid chain for table-and-text open domain qa. arXiv preprint arXiv:2201.0588 2022.
12


Reviews:
[{"review_id": "ubNwdIGw2DO", "reviewer": "Reviewer_kqSf", "summary": "", "questions": "", "limitations": "", "rating": 1, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper proposes a new pre-training technique to induce a logical prior in the language model representation. Concretely, they propose pre-training on facts, represented as knowledge base triples (source, sink, relation) (knowledge-base completion) and link prediction, alongside traditional masked language modeling objective. Their proposed method achieves some improvement over downstream tasks, including a subset of GLUE benchmark and a couple of relation prediction datasets.\n\n", "summary_of_the_review": "Overall, I believe the paper cannot be recommended for acceptance in any form, given the authors:\n\n- Did not change their text based on copious reviewer feedback\n- Did not respond to the questions raised by the reviewers\n- Made a mockery of the open, peer review system by resubmitting a paper with minimal changes to the same conference **one year** later.\n\nThis is highly unfortunate, as the paper does have some interesting contributions which could have been useful for the community at large, had the authors engaged in the scientific discussion and updated their manuscript accordingly.", "strength_and_weaknesses": "Unluckily for the authors, I was one of the reviewers who have reviewed the same paper when it was submitted to ICLR 2022. [Link to the ICLR 2022 forum](https://openreview.net/forum?id=1gEb_H1DEqZ&referrer=%5BReviewer%20Console%5D(%2Fgroup%3Fid%3DICLR.cc%2F2022%2FConference%2FReviewers%23assigned-papers)), which contains the reviews on the previous version of the draft.\n\nChanges in the current submission compared to the previous version:\n\n- Minor writing improvements in abstract, introduction.\n- Addition of a few more related works\n- Addition of BERT large and Prophet large experiments\n- Addition of multi-task finetuning results\n\nI would encourage the readers to refer to the reviews from the previous forum to get an idea about this work, as this version has barely changed. I'm adding a [diff of the two versions](https://draftable.com/compare/KSXHLMKbYBxe) for comparison. \n\nI'm surprised and sad that the authors only chose to perform minor modifications in their paper, given they had a full year to make revisions and incorporate the reviewer feedback. The authors also did not respond to the questions raised by several reviewers in the initial draft. What is shocking is that many of those questions _still remain unanswered_ in the draft the authors resubmitted **one year later**. For instance, majority of the reviewers had issues and with the presentation of the paper, involving the notation abuse in the theoretical background, motivation behind the methodology, implementation details, and explanation of the results (`cXhg`, `DQM2`, `K1pp`). The preliminaries and the model description are virtually untouched in the new version (check the diff). Reviewers also had severe issues regarding the analysis, especially the cherry picking of the heatmap and case study (`DQM2`, `K1pp`, `cXHg`). Those sections remains unchanged as well. Heck, **even typos** pointed out by the reviewers remain the same!! (\"nentailment\" in Figure 5, as pointed out previously by cXHg).\n\nRegarding the addition of new results, they do indeed add more data point on the effectiveness of the approach compared to BERT large baseline. However, these results do not change the fundamental questions about the work which has been asked previously.\n\n\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "mYLy9soiB0", "reviewer": "Reviewer_5kDz", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper introduces PROPHET which is a BERT-like model but pre-trained with three new pre-training objective:\n\n(1) Logical Connective Masking: Masking but with more chance on masking connectives (discourse adverbials, subordniate conjuctions etc.). \n\n(2) Logical Structure Completion: (closer to fact-triplet completion) \"Arg1 predicate Arg2\" forms of fact triplets are recovered from dependency parses (from off-the-shelf tools) (combined with co-reference resolution edges and such). Part of it may be masked and the model has to precict it. \n\n(3) Logical Path Prediction: (closer to syntactic dependency paths) randomly select nodes from dependency parses (from off-the-shelf tools) and predict whether the nodes are connected by a path in the parsed tree or not (binary class prediction using binary cross entropy). \n\nA decent amount of empirical experiments on different tasks, and ablations show effectiveness of the components and superiority over basic MLM-based BERT. ", "summary_of_the_review": "The paper introduces PROPHET: a BERT-like model trained with three new pre-training objectives created based on connectives and syntactic dependencies. Decent performance, but issues with clarity in technical exposition. ", "strength_and_weaknesses": "Strength: \n\n1. The proposed modeling strategies are not too bad. They look interesting.\n\n2. Improved performance over baseline BERT trained by standard MLM in multiple tasks and settings. \n\nWeaknesses:\n\n1. While the clarity is not too bad, it seems to fall a bit flat when it concerns specific and concrete details about the implementation. There are several parts that appeared unclear to me. Particularly it's hard for me to even verify the fairness of the experiments given unclear descriptions about how PROPHET and BERT-base were pre-trained. For different confusions/concerns, see my questions in the next section of the review. (This is my primary concern. With better clarity on some key details, I think this would be an overall acceptable paper).\n\n2. Generally the scales of the loss from generation and path classification can be different. Typically in these kinds of contexts, a scalar multiplier is used as a hyperparameter to scale up/down the losses from different objectives. \n\n3. In a sense, PROPHET is still just \"another\" new pre-training objective that improves a bit over BERT on several tasks. I am not sure how much we can take from that. There are still multiple other BERT-variants (SpanBERT, StructBERT, ReasonBERT, BERT-NCE, MC-BERT etc.)with endless different objective that gets a bit better than BERT on different NLU tasks. There are also pre-training strategies introduced for LogiQA/ReClor like LogiGAN. It's not clear if empirical-performance-wise PROPHET offer something distinctly much better than those or stack with others. Authors also attempt to motivate something about prior methods being sensitive to role reversal (section 2.1); but it's not discussed too deeply and I am not sure any empirical results in this paper demonstrate that PROPHET perform distinctly better than others in some special aspect.\n\n4. Even if PROPHET may perform better than BERT under the same training steps, PROPHET can require additional computational budget for parsing. BERT may be trainable further for the same budget and provide better results. \n\n5. Minor: a single anecdote in case study doesn't reveal much. It could be more interesting to show statistically, if PROPHET is more robust to such kind of changes as considered in the case study. Similarly, there's limited conclusion we can get from a single attention heatmap. It may be better to remove those parts (or move to the appendix) and focus more on clarifying/elaborating the other sections. ", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "UOHQXOwsfF", "reviewer": "Reviewer_X9Xi", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposes to enhance a pre-trained language model (PrLM) by three self-supervised pre-training tasks, namely logical structure completion, logical path prediction and logical connective masking. Experimental results demonstrate that the enhanced PrLM achieve significant performance in several NLP tasks including natural language inference, relation extraction and machine reading comprehension, compared with the basic PrLM BERT.", "summary_of_the_review": "The novelty of the paper lies in three new pre-training tasks, which are shown to be effective compared with the baseline BERT. However, the paper can still be improved by clarifying notations in method descriptions and the crucial term logical reasoning. More importantly, the comparisons with other enhanced PrLMs are probably needed by considering that all of them adding new pre-training tasks to BERT.", "strength_and_weaknesses": "Strength:\n\n(1) The proposed three pre-training tasks for enhancing PrLMs are novel as far as I know.\n\n(2) The enhancements are shown to be effective compared with the PrLM baseline BERT.\n\nWeaknesses:\n(1) There is a lack of comparisons with other enhancements for PrLM. In particular, none of the enhanced PrLMs mentioned in Related Work, namely ERNIE, WKLM, KEPLER and K-Adapter, is compared in experiments.\n\n(2) The presentation is not good enough and can be improved, especially in clarifying notations given in paper. In the last sentence above Figure 2, $E$ is not defined in $S=(V,E)$, and $A_i$ and $P$ are not defined in the context. In Equation 1, the symbols $a$ and $p$ are not defined, nor the whole notation $D(x_i\\mid m^a, m^p)$; especially, the meaning of the symbol $\\mid$ is unclear. In Equation 2, the function symbol $\\sigma$ used in $\\sigma[v_i, v_j]$ is not defined.\n\n(3) The declaration that the enhanced PrLM improves the ability of logical reasoning is problematic. The proposed three pre-training tasks do not deal with classical logical reasoning such as deduction and abduction. What kinds of logical reasoning that benefit from the proposal should be clarify.\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "GGAZ50HFSa", "reviewer": "Reviewer_NY6R", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "Based on the knowledge structure fact, this paper proposes a logic-aware pre-trained language model PROPHET, which can then learn logical relations more generally from larger corpus. Extract fact from a given syntax to leverage multidimensional logical information for better representation. In this paper, three new pre-training methods based on fact are introduced in detail: 1) Logical connective masking for learning sentence-level logical connections. 2) Complete the task based on logical structure, and align the extracted facts with the original context. 3) Logical path prediction, which captures the logical relationship between facts.\n\nContributions：\n(1) PROPHET takes the initial attempt to pre-train language models that leverages the conceptual knowledge unit fact to equip PrLMs with logical reasoning ability. (2)This article introduce three novel pre-training tasks for learning logic-aware \nrepresentations from texts and facts. (3) The model performs in text classification, language reasoning, semantic similarity and other tasks better than BERT, especially in logical reasoning. The model can also model the semantic relationship between sentences.\n", "summary_of_the_review": "In this paper, by using a new pre-trained language model PROPHET to process logic, significant improvements have been made in downstream tasks involving NLP and NLU, which can well explain the internal logic structure of the context to assist the reasoning process. It has strong stability and robustness. The description of experimental details and the treatment of ablation experiments are also more detailed.", "strength_and_weaknesses": "Strength：\n(1) Compared with existing studies that inject complex knowledge like knowledge graphs, the knowledge structure based on fact is far less complicated and more general in representing events and relations in languages.\n(2) The paper performs ablation experiments that show the effect of each pre-training task.\n(3) Compared with the lack of logical supervision in the existing pre-trained models, the proposed method can capture the inherent logical structure in the text well.\n(4) The overall structure of the article is relatively clear.\n(5) The paper performs ablation experiments that show the effect of each pre-training task.\n\nWeaknesses:\n(1) The specific technical implementation of this model is largely based on existing studies. Includes criteria for some of these parameters and measures of testing.\n(2) The experimental data and parameter selection in this paper lack necessary explanations.\nThe Prophet proposed in the paper did not show a significant advantage.\n(3) No significant improvement was reported on any of the other tasks, except for the logical reasoning task.\n(4) In the third column of Figure 4, the Prophet: nentailment maybe Prophet: entailment.\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "CLW8WZcMEMe", "reviewer": "Reviewer_i2UV", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper provides an approach to pre-train language models by infusing them with logic information to enhance their performance on logical reasoning. They propose to use a \"fact\" which is extracted from the text itself and propose to use three pre-training tasks 1) Logical connective masking, 2) Logical structure completion, 3) Logical path prediction. Their facts are created using dependency parse information and do not require any additional Knowledge base. Their approach outperforms BERT base and BERT large on multiple datasets.", "summary_of_the_review": "While the idea of using dependency parse to extract triplets from the text and feeding them as part of a pre-training objective seems novel, i'm not sure how practical it is since it's very costly to run dependency parse / constituency parse. There are some experiments that seem too much cherry picked and are not convincing to support the claim they make. This paper is also lacking crucial literature review of relevant tasks like relation prediction, link prediction etc. The baseline used in this paper is also old and more recent \"logic induced\" baselines have not been used properly. There are some design choices in the experiments that raise questions (like why not use Bert trained for 200k more steps for all experiments, why not pre-train using your objectives from scratch), etc. Some of the datasets chosen to evaluate their approach (GLUE) are known widely to be poor benchmarks to evaluate logical reasoning.\nIn any case, I'm willing to bump up my rating to borderline accept if the authors have a satisfactory reply to some of my questions (I'm aware it might be tough to run new baselines) but some other explanations can definitely be provided.", "strength_and_weaknesses": "Strengths:\n* The idea to use dependency parse information to pre-train a language model such that it has logical reasoning capabilities is nice. It alleviates the need to use external manually curated knowledge bases.\n* They provide a detailed analysis talking about various aspects of their approach including ablation, comparison between entity based knowledge and showing how their method is good for longer length text.\n* They show good results when compared to a BERT baseline on multiple GLUE benchmarks and DocRED, ReClor and LogiQA.\n\nWeakness and Questions:\n* Terminology of \"fact\" is certainly not novel. It is exactly the definition of a \"triple\" that is used widely in KG + graph literature. The \"logical graph\" is also exactly a knowledge graph -- just with knowledge extracted automatically using dependency parse. Many automatic KG creation techniques also extensively use dependency parse information.\n* In the abstract, it is stated: \"it enables training logic-aware models to be conducted on a more general language text\" - not sure what \"general language\" here this means. \n* In the introduction they claim that capacity to capture logic relations is necessary for logical reasoning but provide no citation / justification for the same.\n* The three pre-training tasks they claim as novel are well known tasks. \"Logical structure completion\" and \"Logical Path Prediction\" are essentially relation / entity prediction and link prediction.\n* Baseline and Benchmarks:\n  * The authors have chosen a very old baseline - There are more recent papers that use logical information (or KGs) in pretrained LMs (Eg: Methods present in A Survey of Knowledge Enhanced Pre-trained Models - Yang et al, 2021 and more recently, LogiGAN Pi et al, 2022).\n  * The paper lacks comparison with such other baselines. The authors have not compared with other more recent LMs like T5 too. The authors did compare with SemBERT on GLUE benchmark, but omitted the model for other comparisons in Table 2 and 3.\n  * Was SemBERT also further trained for 200k steps? In Table 3, why were original numbers used and not the numbers of BERT-base trained with further 200k steps?\n  * It is well known that GLUE benchmark is not ideal to compare models for logical reasoning capabilities (Gururangan et al, 2018). It is recommended to add more benchmarks to verify the reasoning capabilities like CLUTRR. It is also recommended to add standard relation prediction / link prediction benchmarks. \n* What all types of relations do you use (Number)?\n* What happens if you just pretrain with your objectives without using BERT initialized model?\n* Great job in specifying all experimental details. Can you also talk about the time it takes to pretrain according to your method? I feel extracting dependency parse + coref information is costly, and only a very significant jump in performance warrants the need to spend compute on calculating those. \n* Please provided statistical significance with the results.\n* In 5.4 results, you only talk about BERT but not SemBERT.\n* Table 3 why didn't you use BERT trained 200k more steps? Is 0.3 especially on the original BERT statistically significant?\n* Section 6.2 - How were Named entities used? Are all NEs paired with all other NEs? Or do you just filter your set of triples such that both entities are NEs?\n* The analysis of attention map is not at all convincing. Why just use one attention map? This experiment just not do justice to the claim.\n\n\nGrammar and nit picks:\n* The title in the paper doesn't match the title here in open-review (aware missing).\n* Intro para 1 - \"However they too much rely on massive\" -> \"However, they rely too much on massive\"\n* Logic reasoning -> logical reasoning\n* Figure 1 stabalized -> stabilized\n* In Equation 1- you mean conditioned on the input with masks and not just the masks right?\n* In section 6.1 you can rephrase this as currently both of your points talk about link prediction being most effective. \n* Describe what MNLI-matched and mismatched is in Section 6.4. Are the results in Table 6 for your method? What are the scores for BERT? \n* Figure 4 - nentailment -> entailment.\n* Conclusion last line - \"our model can well interpret ..... of the context to aid\" -> \"our model can interpret .... of the context well, to aid\"\n* Some equations are missing descriptions of the variables used. There's no need to define new variables if they are not used. What are the variables in Eq 2? \n* Basis -> bases", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Decision: Reject