Pre-training Language Models for Comparative Reasoning
Authors: Mengxia Yu
Zhihan Zhang
Wenhao Yu
Meng Jiang
Venue: EMNLP 2023 Main (2023)
Keywords: language model pre-training
question answering
question generation
summarization
comparative reasoning
TL;DR: This paper introduces a novel framework to pre-train language models for enhancing the ability of comparison reasoning over texts, and the proposed method shows significant improvement on tasks including comparative QA, QG and summarization.

Abstract: Comparative reasoning is a process of comparing objects, concepts, or entities to draw conclusions, which constitutes a fundamental cognitive ability. In this paper, we propose a novel framework to pre-train language models for enhancing their abilities of comparative reasoning over texts. While there have been approaches for NLP tasks that require comparative reasoning, they suffer from costly manual data labeling and limited generalizability to different tasks. Our approach introduces a novel method of collecting scalable data for text-based entity comparison, which leverages both structured and unstructured data. Moreover, we present a framework of pre-training language models via three novel objectives on comparative reasoning. Evaluation on downstream tasks including comparative question answering, question generation, and summarization shows that our pre-training framework significantly improves the comparative reasoning abilities of language models, especially under low-resource conditions. This work also releases the first integrated benchmark for comparative reasoning.

Pre-training Language Models for Comparative Reasoning
Mengxia Yu', Zhihan Zhang', Wenhao Yu! Meng Jiang'
UUniversity of Notre Dame, "Tencent AI Seattle Lab
(myu2, mjiang2)end edu, wenhaowyueglobal. wtw.:g/..llbateeett...ommm tencent.com
Abstract
identifying comparative sentences (Jindal and Liu, 2006), mining comparable entities (Li et al., 201 I), identifying comparative aspects from a set of ques- tions (Bondarenko et al., 2022; Beloucif et al., 2022), extracting comparative summaries (Bista et al., 2019), and summarizing different opinions (Iso et al., 2022). Yet, the data collection for these tasks relies expensive and time-consuming man- ual annotation. As result, low-resource scenarios are common when it comes to new comparative tasks (Iso et al., 2022). Moreover, the task-specific design such models limits their general compar- ative reasoning abilities. Meanwhile, pre-trained language models (PLMs) such as BART (Lewis et al., 2020) and T5 (Raffel et al., 2020) exhibit generalizability several NLP tasks. However, existing pre-training methods such as masked lan- guage modeling and span in-filling fail to empower language models (LMs) with strong comparative reasoning abilities due to the lack of explicit train- ing on comparisons.
Comparative reasoning process of com- paring objects, concepts, or entities to draw conclusions, which constitutes fundamental cognitive ability. In this paper, we propose a novel framework to pre-train language mod- els for enhancing their abilities of comparative reasoning over texts, While there have been approaches for NLP tasks that require compar- ative reasoning, they suffer from costly man- ual data labeling and limited generalizability different tasks. Our approach introduces novel method of collecting scalable data for text-based entity comparison, which leverages both structured and unstructured data. More- over, we present a framework pre-training language models via three novel objectives comparative reasoning. Evaluation down- stream tasks including comparative question answering, question generation, and summa- rization shows that our pre-training framework significantly improves the comparative reason- ing abilities of language models, especially un- der low-resource conditions. This work also releases the first integrated benchmark for com- parative reasoning.
To address these challenges, we propose a novel pre-training framework to enhance the compara- tive reasoning abilities of LMs. Specifically, it trains LMs to capture the comparison information between entities from paired documents. Our ap- proach pilots around a scalable, labor-free data col- lection method that gathers documents entity descriptions and a wealth of facts for entity compar- ison by combining structured (i.e., Wikidata) and unstructured data (i.e., news and Wikipedia), We represent these comparisons of facts as quintuples, which consist of a pair of entities and the corre- sponding values of their shared property. To em- power LMs with comparative reasoning abilities on such data, given two comparable entities and their corresponding descriptive documents, we design three novel pre-training tasks including the genera- tion of comparative answers, question-answer pairs, and summaries. Pre-training data of these tasks are obtained through automatic textualization of fac-
1 Introduction
Comparative reasoning constitutes a fundamental cognitive ability that plays crucial role in decision- making. It refers to comparing objects, concepts, entities draw conclusions make informed decisions. For example, consumers often compare products on their features such as price, quality, and user reviews before placing an order. Policy- makers weigh the advantages and disadvantages of different policy proposals to address pressing issues. Regarding textual documents, comparative reasoning commonly needed in identifying differ- ences between research studies. contrasting news articles from different sources, synthesizing ar- guments of opposing viewpoints in a debate.
Recent research has developed models for a few NLP tasks related to comparing texts, including

---

Step 1: Collecting Quintuples for Entity Comparison
The show, with book by the semenwwitr Diablo Codyy ("Juno") ) and staging by director Paulus ("Waitress" ") takes on the good casking new musicals to do: the work of singing about real things
always
News
human
(2,,
instunce
instunce of
Diablo Cody
eg: Diane Paulus
p work
p. work
Wikidata
Screenwriter
Vg director
Step Textualizing Comparison Knowledge
Step 3: Seq-to-seq Pre-training
Wikipedia
D1, Q D1,
D1,
(ei, ez, Ui, v2)
D2
D2
Template
Dala2Text
Brook Maurio (Dusey; June 1978), pon name Diablo Cody, American writer producer. She guined recegnitiion for her candid blog and subsequent memoir
is Ametican theater und opera director who is currently the Terric and Realley Bloom Artistic Director of American Reperiory Theater Harvard University. Paulus was
Paulus 1966)
LM
A
Q
A
S
(Q, A)
S
Figure I: The framework of pre-training LMs for comparative reasoning abilities. In Step 1, we collect quintuples for entity comparison by combining structured knowledge base (i.e., Wikidata) and unstructured text corpora (i.c., Gigawords, CC-News, Wikipedia). Details are in $ 3.2.2. In Step 2, to obtain text-based pre-training data, we textualize the quintuples into synthetic QA pairs with set of templates, and convert the quintuples into summaries with off-the-shelf data-to-text model. We gather Wikipedia documents in $3.2.3. In Step 3, we design novel pre-training tasks for the LMs. Details are described in $3.3.
descriptions of entities. Details are
tual quintuples, to prevent expensive manual annotation. Subsequently, the pre-training tasks uniformly formatted with natural language prompts perform multi-task pre-training on the LMs. To our best knowledge, this work is the first to pre- train LMs for comparative reasoning.
shot setting, the improvement becomes as high 13.99 points average. These results highlight the effectiveness of our pre-training framework which empowers LMs with impressive abilities of com- parative reasoning even when zero or few examples are available. Besides, we analyze the effect of the pre-training data size on the model performance, and provide case study to better understand the benefits of our pre-training.
To comprehensively evaluate the comparative reasoning abilitics of LMs, we introduce new benchmark with suite of comparative reasoning tasks. It contains: (1) comparative question an- swering (QA), sourced from subsets HotpotQA and 2WikiQA datasets (Yang et al., 2018; Ho et al., 2020); (2) comparative question generation (QG), including HotporQG and 2WikiQG which are con- verted from the QA datasets; (3) comparative sum- marization, including the Diffen dataset that we crawled and the existing CocoTrip dataset (Iso
Our contributions are summarized follows:
We propose a scalable method of collecting and designing training data for entity compar- ison, using both structured and unstructured data sources that are publicly accessible.
We present novel framework for pre-training LMs enhance their comparative reasoning abilities on multiple related objectives.
al., 2022).
We provide the first benchmark for entity com- parison over texts, serving as a foundation for future research in this topic.
With this benchmark, we conduct extensive ex- periments with vanilla PLMs (i.e., BART and T5) and their counterparts trained by our proposed framework. Results demonstrate notable improve- ment in the performance of these PLMs on different comparative reasoning scenarios, especially under low-resource settings. Specifically, under the few- shot setting, the BART model pre-trained with our framework outperforms the vanilla BART by an av- erage of 17 points on all datasets. Under the zero-
Related Work
2.1 Comparative Reasoning
The academic landscape of comparative reason- ing tasks has seen a significant progression. Early research primarily focused on mining explicit com- parative information from massive corpora, such

---

identifying comparative sentences (Jindal and Liu, 2006), extracting comparable entities (Li et al., 2011), and classifying components of comparison (Beloucif et al., 2022). Recent work focused more on text generation tasks such as generating argu- ments to answer comparative questions (Chekalina et al 2021), generating comparable questions from news (Beloucif al., 2022), and summarizing com- parative opinions (Lerman and McDonald, 2009; Iso et al., 2022). The existing techniques were designed for specific tasks and could not general- ize across all types of comparative reasoning tasks, Moreover, they suffered from the scarcity of la- belled data in low-resource settings. Our approach aims to address these two challenges.
entities. Specifically, LMs are given a pair of docu- ments, each describing an entity, and are trained generate target sequences which require compari- son between the entities. We consider three types of target sequences: an answer to a comparative question, a question-answe pair that requires com- parative reasoning, and comparative summary of entities. Correspondingly, we design three text-to- text pre-training tasks that require the LMs to si- multaneously attend to both documents and extract information for pairwise comparison. This frame- work enables them to handle various downstream scenarios that require comparative reasoning.
To collect data for large-scale pre-training, we extract comparable entity pairs with their proper- ties by combining structured and unstructured data. The extracted data are first formulated as quintu- ples (elaborated in $3.1 to show their comparative nature), which are later used for text-to-texi pre- training.
2.2 Language Model Pre-training
It is worth exploring use both structured and unstructured data language model pre-training. Early work proposed to fuse knowledge graphs and textual information by encoding entities or nodes as a part of the input (Zhang et al., 2019;
3.1 Notations
Wang et al., 2021; Yu et al., 2022; Ke et al., 2021; Liu et al., 2021; Hu et al., 2022). For example. Hu et al. (2022) integrated graph-based knowledge augmented modules bring structured knowledge into generative LMs. Another branch of work in- corporated entity information (Xiong et al., 2020; Zhang et al., 2022) or relational information (Qin et al., 2021; Hu et al., 2021) without modifying the structure of the LM. While these pre-trained models delivered encouraging outcomes across a multitude of downstream tasks, they were not tai- lored for the needs of comparative reasoning. A novel design of pre-training objectives is necessary, which has not been inherently presented in these models. Regarding the collection of pre-training data, RGPT-QA (Hu et al., 2021) combined Wiki- data and Wikipedia to generate synthetic QA pairs for pre-training. However, such a set of pre-training data only comprised statements of individual enti- ties, so the trained model is not effective for multi- hop comparative questions. MQA-QG (Pan al., 2021) and MuSiQue (Trivedi et al., 2022) generate synthetic data for unsupervised multi-hop QA, but they did not consider other comparative tasks.
A Wikidata statement is denoted as (e, p, v). Here, signifies the entity, which is the subject of the statement. p refers to the property, describing the aspect of the entity that the statement addresses. represents the value, which is the object en- tity or specific value associated with the property. We define a quintuple as pair of Wikidata state- ments of two comparable entities on a common property. Formally, quintuple is represented (e1, E2, p, U1 122), where is common property of 61 and 62, and en and U2 are the corresponding values. Such quintuples enable the comparison on shared properties, reflecting the similarity or differ- ence between the corresponding property values or tail entities.
In our framework, the input sequence constitutes two documents D1 and D2 e1 and e2 respec- tively. The target sequences are textualized forms of the quintuple, such as question-answw pairs (de- noted by (Q, A)), and summaries (denoted by S).
3.2 Pre-training Data Preparation
3.2.1 Data Sources
Structured data is reliable source for obtaining tity information. We use Wikidata, a collaborative knowledge base that stores data in a structured for- mat. Wikidata contains numerous statements that describe entities, where each statement includes a property of the entity and a value. Each entity and
3 Pre-training Framework
To enhance the ability of LMs in comparative rea- soning, we introduce a novel framework for pre- training LMs on a collected corpus of comparative

---

All:
is linked a sentence if the aliases of e, p. and v all appear in the sentence. Next, we pair (#1, Pl, v1) and (e2, P2, ria:
Q: Do ej and e2 have the same/different value of p?
A: Yes/No
(2,22) U2) if they satisfy the following crite-
Q: Do e and ey both have the value of bn in terms of p?
A: Yes/No
What are they p e and e2?
1. and é2 belong to the same category, e.g.. both have the value human for property in- stance of. This ensures the entities are analo- gous each other.
A: en 12
ef ø2:
Q: Which one of the following entity's i vs? en or eu?
A: e1
2. pl = p2. This follows the common practice that comparisons are usually made on a shared property between two entities.
Q: Is e1 p or ez?
A:
If 01 = 12:
3. The sentences linked to (e1.P1, and (02, 12) co-occur in a from news or Wikipedia. Being mentioned together indi- cates implicit entity comparison.
Q: Which entity has the same value in terms of p?
e2
Q: and are known for what (value) of p?
01/122
We denote such statement pair as quintuple (e1, 62, p. U1, U2). By following the above criteria, such quintuples store necessary information for comparing entities ei and e2, which plays a critical role in our pre- pre-training task design.
Table 1: Synthetic QA templates. All indicates the templates are applied to all quintuples. The templates under If en A U2: or If fj U2: are applied to quintuples whose U1 and 12 are different or the same, respectively.
property is associated with a set of aliases.
3.2.3 Quintuple Textualization
Unstructured data including news sources (i.e., Gigawords, CC-News) and encyclopedia (i.e..
In order to empower the LM with the ability of comparative reasoning in various language genera- tion scenarios, we pre-train the LM in a text-to-text manner. To achieve this, the first step is to repre- sent the comparative information inherent in the quintuples in a textual form.
Wikipeda) offer an abundance of information for determining the comparability of entities and rele- vant properties. For example, a sentence in a piece of news from New York Times like "The show, with a book by the screemwriter Diablo Cody ( ('Juno') and staging by director Diane Paulus ( ('Waitress') ') takes on the good work ...." indicates that Diablo Cody and Diane Paulus can be compared on the property of work (values: screenwriter vs. director). Besides, Wikipedia contains vast collection of articles per- taining large set of entities. A Wikidata entity uniquely corresponds to a Wikipedia article whose title matches the entity's s surface form.
To begin with, extract descriptive documents Dj and D2 for each pair of entities E and eg texts in our pre-training. First, we find Wikipedia articles of e1 and e2 by the links from Wikidata. To ensure the information within the quintuple can be inferred from the context, we filter the ar- ticles based on whether any sentence within the article can be linked to statements (e1, U1) and (c2, P. 02). We link the statements based on two heuristics: (1) Within article pertaining to en- tity e, sentences as their subject; (2) sentence in a Wikipedia article of e mentions both e and v from Wikidata statement (e, p, v) then the sentence likely to describe the fact of (c,p, v). Thus, we link the statements to sentences whenever v) or (p, v) can be matched. To assess the linking quality, we randomly sampled 100 statement-sentene links and performed manual inspection. The linking accuracy exceeds 95%, indicating the Wikidata statements are effectively linked the sentences. Finally, due to length limit of LMs, we split the original article into 10-sentence segments, and use the segment that contains the linked sentence as the
3.2.2 Quintuple Collection
highly probable discuss
In this section, we elaborate the process of collect- ing quintuples by combining structured data (i.e.. Wikidata) and unstructured data (i.c., news and Wikipedia). Intuitively, when a pair of statements concerning the same property of related entities co- occur in a textual context, there is high probability that these statements are indeed comparable.
To extract this comparability information, first sample paragraph from the news or Wikipedia. Then, we link Wikidata statements to the sentences in the paragraph by identifying the mentions of entity e, property p. and value u using string matching. Specifically, a statement (c,p. p, u)

---

Pre-training Task
Source -> Target
Comparative Answer Generation
Answer the comparative question. Question: (Q) Context: (D1] [SEP] (D2)
Comparative QA Pairs Generation
Generate a comparative question-answer pair. Context: [Di] [SEP] (Da) - Q:A
Comparative Summary Generation
Generate comparative summary. Context: (D1) [SEP] ( (D2) +
Text Infilling
(corrupted D1 [SEP] /corrupted D2] + (Di [SEP] (Dz)
Table 2: Task names and the format of source-target sequence format in each pre-training task.
(John Lewis. Hank Johnson, member of po- litical party, Democratic Party, Democratic Party)
Quintuple:
3.3 Pre-training Tasks and Objectives
In this section, we describe three comparative pre- training tasks used to train LMs. They are all text generation tasks, which align seamlessly with ar- chitectures of widely used language models such as BART and T5. We unify them with task speciffc prompts in multi-task setting, shown in Table 2.
D2: John Robert Lewis (February 21, 1940July 17, 2020) American statesman and civil rights activist who served in the United States House of Representatives for from 1987 until his death in 2020. He was the chairman of the Student Nonviolent Coordinating Committee (SNCC) from 1963 1966. ... While in the House, Lewis was of the leaders of the Democratic Party, serving from 1991
3.3.1 Comparative Answer Generation To train the LM with the ability answer com- parative questions, given a comparative question, we concatenate with the documents D1, D2 as input, and then train the model to generate the cor- responding answer. This task not only requires the model to find relevant contexts to the question in each single document, more importantly, it requires the interaction between both documents to make the comparison. We define the loss function
D2 Henry Calvin Johnson Jr. (born October 2. 1954) American lawyer and politician serving the U.S. representative for since 2007. He is member of the Democratic Party....
Synthetic QA pairs Q A:
I. Q: Do John Lewis and Hank Johnson have the same value for member of political party? A: Yes
2. Q: Do John Lewis and Hank Johnson both have the value of Democratic Party in terms of member of political
party? A: Yes
3. Q: What are the member of political party of John
Lewis and Hank Johnson? A: Democratic Party
4. Q: Which entity has the same value John Lewis in terms of member of political party? A: Hank Johnson
LQA LQA=-
Σ log P(A:Q., D1, D2)
Q: John Lewis and Hank Johnson are known for what value of member of political party? A: Democratic Party
(Q.,A)TT
Synthetic Summary S: John Lewis is member of the Democratic Party. as is Hank Johnson.
in which T is a of QA pairs derived from the templates. and P(.) is the predicted probability.
3.3.2 Comparative QA Pairs Generation
Table 3: A quintuple for comparison between John Lewis and Hank Johnson on their shared property mem- ber of political party. The example consist of the textu- alized data used in pre-training, including the entities' descriptive documents D2 and D2, QA pairs (Q, A) synthesized with designed templates, and the synthetic summary S generated by data-to-text model with two the two Wikidata statements as input.
Given two documents, the model is required to generate comparative questions and answers. With this objective, the model learns to attend to both documents, identify the comparable propertics of two entities and ask meaningful questions:
LQAG Σ log P(Q;, A A, Dz) (Q.,))TT
descriptive document Dj for e, (or D2 for ez).
3.3.3 Comparative Summary Generation
Next, we convert the comparison knowledge en- capsulated within the quintuples into comparative texts, namely, QA pairs and summaries. To syn- thesize comparative QA pairs A). we design a diverse set of templates shown in Table 1. To generate synthetic comparative summaries S, utilize off-the-shed data-to-text model (Ribeiro al., 2021) fine-tuned on DART (Nan al., 2021) dataset. This allows us transform quintuples into concise declarative sentences. An example of a textualized quintuple is provided Table 3.
Comparative summarization aims at generating summaries that highlight the similarities or differ- ences between two entities given their descriptions. Given two documents, the model tasked with gen- erating short comparative summaries that represent the comparable statements:
LsUM =
Σ log P(S D2) SCS
where S is the set of summaries from quintuple textualization.

---

3.3.4 Prompt-based Multi-task Training
datasets collected from English Wikipedia. These datasets require multi-hop reasoning on different entities before reaching the correct answer. To fo- cus on comparative ability, we obtain the subset of comparative questions by their question type an- notations in the original dataset. As result, the train and validation set of HotpotQA consist of 17, 17,456 and 1,487 instances, respectively. Likewise, 2WikiQA comprises 51,693 and 3,040 instances in training and validation set, respectively. We report results in validation sets.
Inspired by the prompt-based multi-task training methods utilized by previous text-to-text transform- ers (Raffel et al., 2020; Sanh et al., 2022), we jointly train the aforementioned pre-training tasks by uni- fying their input sequences with natural language prompts. The detailed format of source and tar- get sequences shown in Table 2. The model is jointly optimized for all tasks, which encourages the model to learn generalizable representations that are beneficial across tasks. To preserve its general language modeling ability, we employ the proposed pre-training tasks along with the text in- filling (TT) task, where the model is required to re- construct the documents corrupted with randomly masked spans, described in Lewis et al. (2020). We denote the loss function for text infilling as as CTI- Hence, the overall objective as follows: LQA + fQAG LSUM + Cr--
4.1.2 Comparative Question Generation
Comparative QG aims at generating questions that draw comparisons between the shared properties of two entities, given their textual descriptions. We convert the aforementioned QA datasets to QG by using the evidence passages as input and the com- parative question as output. We report the results on validation sets using overall BLEU (Papineni et al., 2002) and ROUGE-L (Lin, 2004) metrics.
We denote the proposed multi-task pre-training for comparison as +CMP. To analyze the effects of each pre-training task, we define single-task vari- ants: +CMPQA for comparative answer generation, +CMPQAG for comparative QA pairs generation, and +CMPsUM for summary generation.
4.1.3 Comparative Summarization
Comparative summarization aims at generating summaries that highlight the similarities or differ- ences between two entities given their descriptions. Following the convention in text summarization (Zhang et al., 2020), we evaluate the generated summaries with ROUGE-2 and ROUGE-L scores.
4 Experiments
4.1 Datasets and Evaluation Metrics
To evaluate our proposed method, we consider downstream tasks involving comparative reasoning, including comparative question answering (QA), comparative question generation (QG) and compar- ative summarization. In this section, we introduce the downstream datasets and evaluation metrics.
CocoTrip. We collect data from the common opinion summarization setting of the CocoTrip dataset (Iso et al., 2022), which involves summa- rizing the shared opinions from two sets of reviews about hotels. The dataset consists of 20, 10, and 18 instances for training, validation and test set, respectively. Since the test data is available, we report the results on the test set. We concatenate both reviews as the input context.
4.1.1 Comparative Question Answering
Comparative QA requires the comparison of two entities on their shared properties. Since our focus on comparison over documents instead of knowl- edge retrieval, we do not include distractor pas- sages but directly use the gold evidence passages as the context for question answering. For evaluation, we calculate the exact match (EM) score between the predicted answer and the ground-truth answer, after necessary normalization (Chen et al., 2017). Besides, unigram F-1 scores are also calculated as a complementary metric.
To address the lack of available datasets for the comparative summarization of two entities, we create a new dataset from Diffen.com a web- site recognized for offering high-quality, human- authored comparisons between different people or objects to help people make informed decisions. Comparison articles on Diffen.com typically in- clude brief introduction summarizing the similar- ities and differences. We manually collect these introductory paragraphs as comparative summaries. To gather input sources, we obtain Wikipedia arti- cles for each entity. The resulting dataset comprises 20 instances for training and 100 instances for vali-
Diffen.
HotpotQA and 2WikiQA. HotpotQA (Yang et al., 2018) and 2WikiMultihopQA (2WikiQA)
(Ho et al., 2020) are factual question answering

---

Comparative QA
Comparative QG
Comparative Summarization
Ho
2WikiQA
2WikiQG
Crip
Diffen
EM
F1
EM
F1
BLEU
R-L
BLEU
R-L
R-2
R-L
R-2
R-L
AVG
ChatGPT
73.45
62.68
82.95
74.33
7.36
29.16
10.89
33.02
6.80
18.59
9.29
21.69
35.85
BART
69.27
75.70
91.87
92.43
16.29
43.41
35.28
61.94
23.63
44.99
10.04
24.39
49.10
Full-data
CMP
69.26
75.43
91.81
92.30
17.18
43.66
35.82
62.13
27.60
47.90
12.11
26.69
50.16
T5
73.16
79.20
87.40
89.67
17.57
44.70
36.02
62.70
29.18
45.57
9.21
24.03
49.87
CMP
72.69
78.83
88.75
91.08
17.26
44.65
36.12
63.18
30.48
49.19
8.12
23.04
50.28
BART
33.82
39.70
37.65
39.67
11.38
39.04
30.02
57.14
23.63
44.99
10.04
24.39
32.62
Few-shot
+ CMP
44.31
52.15
57.58
58.49
12.75
39.33
30.29
56.28
27.60
47.90
12.11
26.69
39.09
T5
48.89
54.71
43.85
45.63
6.48
30.95
6.71
28.44
29.18
45.57
9.21
24.03
31.14
CMP
50,50
58.29
56.51
58.33
8.18
33.88
12.12
37.95
30.48
49.19
8.12
23.04
35.55
BART
0.00
11.93
0.00
19.30
1.70
18.53
3.45
20.34
4.09
18.32
6.32
17.90
10.16
Zero-shot
+ CMP
31,47
39.04
40.55
42.47
6.86
29.13
9.21
32.23
6.11
24.43
8.02
20.22
24.15
T5
20.44
28.87
20.88
26.92
1.21
18.70
2.38
18.53
8.94
25.28
5.61
17.65
16.28
CMP
44.25
52.62
54.34
56.30
7.24
28.99
5.83
32.31
8.63
28.19
5.72
18.43
28.57
Table 4: Main results. Our pre-trained models denoted by +CMP, bring significant performance gain to BART and T5 in zero-shot (e.g., relatively +82% and +220% of Fl on HotpotQA) and few-shot (e.g.. relatively +29% and +52% of F1 on 2WikiQA) settings across all tasks. In full-data settings that assume a huge number of labeled examples are available, our approach makes smaller improvements on the two models.
dation. The task aims at generating comparative summary based on the given text descriptions of two entities. The input sequence consists of con- catenated entity descriptions, with each description truncated to the first 512 tokens.
merge the full-data and few-shot settings for these two datasets. See training details in Appendix A.3.
4.3
Experimental Results
4.3.1 Effects of Comparative Pre-training
In the comprehensive evaluation across the afore- mentioned six datasets, we compare LMs trained with our method against the vanilla BART and T5. Main results are listed in Table 4.
4.2 Experimental Setup
As a pilot study on pre-training for comparative reasoming, we adopt the pre-trained BART (Lewis et 2020) and T5 (Raffel et al., 2020) as base- lines. Models further pre-trained our compar- ative objectives are denoted BART+CMP and T5+CMP, respectively. See training details in A.2, We also conduct zero-shot experiments with Chat- GPT (gpt3.5-turbo), where the details are in 3.2.3. Since ChatGPT is pre-trained on much scale data, and the downstream datasets might have leaked to its training data, it is not a comparable baseline. We provide ChatGPT (OpenAI, 2021) !) as reference to the performance of one of the most advanced large language models.
When abundant data are available, both our pro- posed models, BART+CMP and T5+CMP, achieve competitive performance which improves their corresponding baselines by -1 point on average. However, in low-resource scenarios, the superi- ority of our method over the baselines becomes clearly evident. Specifically, for the few-shot set- ting, our BART+CMP achieves an average score of 39.09, showing an relative improvement of +19.8% compared to BART's score of 32.62. Similarly, our T5+CMP achieves an average score of 35.55, which improves + +14.2% relatively over T5. Among three tasks, our models show the most significant improvement on comparative QA, demonstrating the effectiveness of our synthetic QA pre-training. In zero-shot setting, BART+CMP and T5+CMP also consistently surpass their baselines by large margins. For instance, BART+CMP achieves an av- erage score of 24.15, 15, which outperforms BART by +13.99 (+ 137% relatively). Likewise, T5+CMP
To test the comparative reasoning ability of mod- els under low-resource scenarios, we compare the models in sew-shot and zero-shot settings in addi- tion to the conventional full-data fine-tuning. In the few-shot setting, we randomly selected 100 in- stances from the training set. However, given the limited number of training instances available in CocoTrip and Diffen (only 20 instances each), we

---

Comparative QA
Comparative QG
Comparative Summarization
HA
2WikiQA
HotpotQA
2WikiQA
CocoTrip
Diffen
EM
F1
EM
F1
BLEU
R-L
BLEU
R-L
R-2
R-L
R-2
R-L
AVG
BART
33.82
39.70
37.65
39.67
11.38
39.04
30.02
57.14
23.63
44.99
10,04
24.39
32.62
Few.
+ CMP
44.31
52.15
57.58
58.49
12.75
39.33
30.29
56.28
27.60
47.90
12.11
26.69
39.09
+ CMMPA
45.25
52.46
55.96
57.22
11.76
39.51
25.73
52.43
24.91
45.36
12.36
25.92
37.41
Shot
+ CMPQAG
32.21
38.17
45.13
46.63
12.57
39.29
26.89
52.74
27.22
42.55
12.28
26.85
33.54
SMP
32.41
37.81
32.34
34.33
12.50
39.40
33.04
59.29
30.54
47.84
12.10
26.69
33.19
BART
0.00
11.93
0.00
19.30
1.70
18.53
3.45
20.34
4.09
18.32
6.32
17.90
10.16
Zero-1 shot Zen
+ CMP
31.47
39.04
40.55
42.47
6,86
29.13
9.21
32.23
6.11
24.43
8.02
20.22
24.15
+ QMP
34.50
42.41
43.85
45.83
1.57
19.39
3.31
20.27
0.00
5.14
1.61
6.72
18.71
+ +CMPAAG
0.00
13.79
0.00
20.85
6.16
27.89
8.00
29.44
1.39
16.62
6.13
18.22
12.37
+ +MMPSUM
0.00
16.56
0.00
20.02
5.33
32.78
6.76
32.78
5.53
25.79
8.34
20.00
14.40
Table S: Few-shot and Zero-shot results of models with multi-task pre-training (denoted by +CMP) vs. single-task pre-training (denoted by +CMP +CMPQAG. and +CMP
achieves average score of 28.57, improving +75% over T5 relatively. These results indicate that proposed pre-training method greatly enhances LM's performance in low resource scenarios, while retaining competitive performance when abundant training data are available.
60
55
EM
F1
4.3.2 Effects of Pre-training Tasks
10
103
10a
104
10a
10a
Number of Quintuples
To further explore the benefits of multi-task pre- training, we compare the performance of our mod- els pre-trained on any single task (i.e., QA, QAG SUM) with the unified models pre-trained on all proposed tasks. Results are shown in Table 5. single task,
Figure 2: Few-shot performance (measured by FI) of BART+CMP on 2WikiQA, when the model is pre- trained on different number of quintuples.
eralized representations and to effectively transfer the shared knowledge across different tasks.
When the model is pre-trained we observe significant improvement in perfor- mance on the downstream task that closely resem- bled the pre-training task. However, such mod- els do not exhibit similar improvements on other tasks that are less similar in nature. For exam- ple, BART+CMPOA improves over BART by large margin on few-shot comparative QA (+11.43 points in Fl HotpotQA), but performs at simi- lar lower level BART QG (+0.38 points in BLEU on HotpotQG and -4.29 points BLEU on 2WikiQG), On the other hand, the unified model BART+CMP exhibits substantial improvements across all downstream tasks and therefore achieves the best overall performance. The improvements brought by multi-task pre-trained on each task is comparable to the gains achieved through the cor- responding task-specific pre-training. These results suggest that pre-training on a single task enhances the model's ability to transfer knowledge only to tasks with similar characteristics, while multi-task pre-training enables the model to learn more gen-
4.4 Effects of Size Pre-training Data
In Figure 2, plot the few-shot performance of BART+CMP on 2WikiQA according to the num- ber of quintuples used in pre- pre-training. We observe that when the number of quintuples increases on a logarithmic scale. the performance grows lincarly. The analysis reveals that scaling the pre-training data benefits the downstream tasks, affirming the effectiveness of the proposed method for gathering large scale pre-training data. Further discussion on the effects of entity coverage is shown in Ap- pendix 4.4.1.
4.4.1 Effects of Entity Coverage
To study the effects of the downstream entity cover- age in pre-training data, we count the overlapping entities between pre-training quintuples and Hot- potQA validation set. We proceed by removing the overlapping entities from pre-training data, fol-

---

lowed by pre-training the model and evaluating on the same HotpoiQA validation set. Our findings, as depicted in Table 6, demonstrate that even with the removal of overlapping entities, where 97.6% of the pre-training data persists, the model's s few- shot performance remains strong. Hence, it can be concluded that the observed performance im- provement is not attributed to downstream entity coverage, instead, it is attributed to the task adap- tation, indicating the potential generalizability to unseen entities.
D1: Airsoft a team game which participants climi- nate opposing players by lagging them out of play with spherical plastic projectiles shot with mock air weapons called airsoft guns. (446 words left)
D2: Paintball a competitive team shooting sport in which players climinate opponents from play by hitting them with spherical dye-filled gelatin capsules called painthalls that break upon impact. (472 words left)
Gold: Airsoft is a popular combat simulation game where participants are climinated when hit by pellets launched from guns that resemble real firearms. In paint- ball participants try to hit each other with paintballs launched from special paintball marker/gun. While airsoft is cheaper and provides a more m warfare experience, paintball is more more_popular, more organized and has larger events.
# Quintuples
Covered Ent.
EM
Fl
warfare
219,598
471 (18%)
44.92
52.41
21 214,327
0
44.86
52.38
BART (R-L: 18.66, R-2: 4.39)
Airsoft is a team shooting sport in which participants eliminate opponents by hitting them with airsoft guns. Airsoft guns are shaped like basketballs or baseball bats and are equipped with series round-shaped projectiles called paintballs.
Table 6: Few-shot performance of HotpotQA. The quin- tuples in our overall pre-training cover 18% of the down- stream entities. When downstream entities are excluded from pre-training. substantial amount (97.6%) of pre- training data remains, and the downstream performance remains robust.
BART+CMP (R-L: 19.17, R-2: 8.62)
Airsoft and Paintball are two of the most popular shooting sports all time. Airsoft is a shooting sport that involves hitting opponents with airsoft guns, while Paintball is a more aggressive game that uses a softer. more aggressive, ball-shaped paintball.
4.5
Case Study
To intuitively show the comparative reasoning abil- ity of our pre-trained model, we present an example of comparative summarization in Table 7. Given documents describing airsoft and paintball, models are expected to generate a summary comparing the commonalities and differences of these two games. However, without exhaustive fine-tuning, the gen- erated summary of BART fails to describe the cor- rect relationship between these two entities. On the contrary, after pre-trained on various compara- tive reasoning objectives, our model generates high quality comparative summaries based on the pro- vided documents under the few-shot setting. The generated summary includes that both games are popular shooting sports while also comparing their differences in their equipment.
Table 7: A test example of Diffen dataset. BART and BART+CMP refer to the model predictions under few- shot fine-tuning tuning. BART+CMP generated the similarities and differences between airsoft and paintball.
of entity comparison over texts, we release bench- mark for future research.
Limitations
In our pre-training framework, we generate syn- thetic data with templates to collect comparative question-answer pairs, which may cause fluency issues on some synthetic questions. Such noise in the pre- training data might affect the downstream performance. Similarly, since the synthetic sum- maries were generated by an off-the-shelff data-to- text model, the language of generated summaries can be rigid and lack of diversity and flexibility. Future work can adopt more advanced approaches to convert quintuples into more fluent and diverse texts for pre-training. Another limitation is that BART and T5 have maximum input token limit of 1,024. When dealing with longer documents complex comparative scenarios, this limation may lead truncation of relevant context, potentially affecting the model's performance. Future work can explore LMs that can handle longer texts.
5 Conclusion
In this paper, we presented a novel framework for pre-training language models for comparative rea- soning. It obtained quintuples for entity compari- son by combining structured and unstructured data, converted the quintuples into textual components, and employed them in three novel sequence-to- sequence pre-training tasks. We demonstrated the effects of the pre-training tasks on six downstream datasets, especially in limited- resource scenarios. To facilitate the assessment of models' capability

---

Acknowledgements
Ziniu Hu, Yichong Xu. Wenhao Yu. Shuohang Wang. Ziyi Yang, Chenguang Zhu, Kai- Chang, and Yizhou Sun. 2022. Empowering language models with knowledge graph reasoning for open-domain question answering. In Proceedings of the 2022 Con- ference on Empirical Methods in Natural Language Processing, pages 9562-9581, Abu Dhabi, United Arab Emirates. Association for Computational Lin- guistics.
This work was supported by NSF I1S-2119531, IIS-213739,, UIS-2142227. IIS-2234058, CCF- 1901059, and ONR N00014-22-1-2507, Wenhao Yu is also supported by Bloomberg Data Science Ph.D Fellowship.
Hayate Iso, Xiaolan Wang, Stefanos Angelidis, and Yoshihiko Suhara. 2022. Comparative opinion sum- marization via collaborative decoding. In Findings of the Association for Computational Linguistics: ACL 2022, pages 3307-3324, Dublin, Ireland. Association for Computational Linguistics.
References
Meriem Beloucif, Seid Muhie Yimam, Steffen Stahlhacke, and Chris Biemann. 2022. Elvis vs. M. Jackson: Who has more albums? classification and identification of elements in comparative questions. In Proceedings the Thirteenth Language Resources and Evaluation Conference. pages 3771-3777 Mar- seille, France. European Language Resources Asso- ciation.
Nitin Jindal and Bing Liu. 2006. Identifying compar- ative sentences in text documents. In Proceedings of the 29th anmual international ACM SIGIR confer- ence on Research and development in information retrieval, pages 244-251.
Umanga Bista, Alexander Mathews, Minjeong Shin, Aditya Krishna Menon, and Lexing Xie. 2019. Com- parative document summarisation via classification. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 33, pages 20-28.
Pei Ke, Haozhe Ji, Yu Ran, Xin Cui, Liwei Wang, Lin- feng Song. Xiaoyan Zhu, and Minlic Huang. 2021. Jointgt: Graph-text joint representation learning for text generation from knowledge graphs. In Find- ings of the Association for Computational Linguis- tics: ACL/IICNLP 2021. Online Event, August 1-6, 2021, volume ACL/UJCNLP 2021 of Findings of ACL,, pages 2526-2533 Association for Computational Linguistics.
Alexander Bondarenko, Yamen Ajjour, Valentin Dittmar, Niklas Homann, Pavel Braslavski, and Matthias Hagen. 2022. Towards understanding and answering comparative questions. In Proceedings of the Fifieenth ACM International Conference on Web Search and Data Mining, pages 66-74.
Kevin Lerman and Ryan McDonald. 2009. Contrastive summarization: An experiment with consumer re- views. In Proceedings of Human Language Tech- nologies: The 2009 Annual Conference of the North American Chapter of the Association for Computa- tional Linguistics, Companion Volume: Short Papers, pages 113-116, Boulder, Colorado. Association for Computational Linguistics.
Viktoriia Chekalina, Alexander Bondarenko, Chris Bie- mann, Meriem Beloucif, Varvara Logacheva, and Alexander Panchenko. 2021. Which is better for deep learning: Python or MATLAB? answering compara- tive questions in natural language. In Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: System Demonstrations, pages 302-311, Online. Association for Computational Linguistics.
Mike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad. Abdelrahman Mohamed, Omer Levy, Veselin Stoyanov, and Luke Zettlemoyer. 2020. BART: Denoising sequence-to-sequene pre-training for natural language generation, translation, and com- prehension. In Proceedings of the 58th Annual Meet- ing of the Association for Computational Linguistics, pages 7871-7880, Online. Association for Computa- tional Linguistics.
Danqi Chen, Adam Fisch, Jason Weston, and Antoine Bordes. 2017. Reading wikipedia to answer open- domain questions. In Proceedings the 55th Annual Meeting of the Association for Computational Lin- guistics, ACL 2017, pages 1870-1879. Association for Computational Linguistics.
Xanh Ho, Knh-Khoa Duong Nguyen, Saku Sugawara, and Akiko Aizawa. 2020. Constructing multi- hop QA dataset for comprehensive evaluation of reasoning steps. In Proceedings of the 28th Inter- national Conference on Computational Linguistics, pages 6609-6625, Barcelona, Spain (Online). Inter- national Committee on Computational Linguistics.
Shasha Li, Chin-Yew Lin, Young-In Song, and Zhoujun Li. 201 I. Comparable entity mining from compara- tive questions. IEEE transactions on knowledge and data engineering. 25(7):1498-1500
Chin-Yew Lin. 2004. ROUGE: A package for auto- matic evaluation of summaries. In Text Summariza- tion Branches Out, pages 74-81, Barcelona, Spain. Association for Computational Linguisties.
Ziniu Hu, Yizhou Sun, and Kai-Wei Chang. 2021. Relation- guided pre-training for open-domain ques- tion answering. In Findings of the Association for Computational Linguistics: EMNLP 2021, pages 3431-3 3448, Punta Cana, Dominican Republic. Asso- ciation for Computational Linguistics.
Y Liu, Y Wan, I. He, H Peng, and PS Yu. 2021. Kg- bart: Knowledge graph-augmented bart for genera- tive commonsense reasoning. In Proceedings of the AAAI Conference on Artificial Intellgence.

---

Linyong Nan, Dragomir Radev, Rui Zhang, Amrit Rau, Abhinand Sivaprasad. Chiachun Hsieh, Xi- angru Tang, Aadit Vyas, Neha Verma, Pranav Kr- ishna, Yangxiaokang Liu, Nadia Irwanto, Jessica Pan, Faiaz Rahman, Ahmad Zaidi, Mutethia Mutuma, Yasin Tarabar, Ankit Gupta, Tao Yu, Yi Chern Tan, Xi Victoria Lin, Caiming Xiong, Richard Socher, and Nazneen Fatema Rajani. 2021. DART: Open- domain structured data record to text generation. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computa- fional L Linguistics: Human Language Technologies, pages 432-447, Online. Association for Computa- tional Linguistics.
Leonardo F. Ribeiro. Martin Schmitt, Hinrich Schütze, and Iryna Gurevych. 2021. Investigating pretrained language models for graph-to-tex genera- tion. In Proceedings of the 3rd Workshop on Natural Language Processing for Conversational AI, pages 211-227, Online. Association for Computational Lin- guistics.
Victor Sanh, Albert Webson, Colin Raffel. Stephen H. Bach, Lintang Sutawika, Zaid Alyafeai, Antoine Chaffin, Arnaud Stiegler, Arun Raja, Manan Dey, M Saiful Bari, Canwen Xu, Urmish Thakker, Shanya Sharma Sharma, Eliza Szezechla, Taewoon Kim, Gunjan Chhablani, Nihal V. Nayak, Debajyoti Datta, Jonathan Chang. Mike Tian-Jian Jiang. Han Wang. Mattco Manica, Sheng Shen, Zheng Xin Yong, Harshit Pandey, Rachel Bawden, Thomas Wang, Tr- ishala Neeraj. Jos Rozen, Abheesht Sharma, An- drea Santilli, Thibault Févry, Jason Alan Fries, Ryan Teehan, Teven Le Scao, Stella Biderman, Leo Gao, Thomas Wolf, and Alexander M. Rush. 2022. Multi- task prompted training enables zero-shot task gener- alization. In The Tenth International Conference Learning Representations, ICLR 2022, Virtual Event, April 25-29, 2022. OpenReview.net.
OpenAL. 2021.
Chatgpt: A large-scale generative language model. chatgpt. Accessed: 2023-06-01.
com/research//
Liangming Pan. Wenhu Chen, Wenhan Xiong, Min- Yen Kan, and William Yang Wang. 2021. Unsupervised multi-hop question answering by question generation. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computa- tional Länguistics: Human Language Technologies. pages 5866-5880, Online. Association for Computa- tional Linguistics.
Harsh Trivedi. Niranjan Balasubramanian. Tushar Khot, and Ashish Sabharwal. 2022. MuSiQue: Multi- hop Questions via Single-hop Question Composition. Transactions of the Association for Computational Linguistics, 10:539 554.
Kishore Papineni, Salim Roukos, Todd Ward, and Wei- Jing Zhu. 2002. Bleu: a method for automatic evalu- ation of machine translation. In Proceedings of the 40th Annual Meeting of the Association for Compu- lational Linguistics, pages 311-318, Philadelphia, Pennsylvania, USA. Association for Computational Linguistics.
Xiaozhi Wang. Tianyu Gao, Zhaocheng Zhu, Zhengyan Zhang, Zhiyuan Liu, Juanzi Li, and Jian Tang. 2021. KEPLER: A unified model for knowledge embed- ding and pre- pre-trained language representation. Trans- actions of the Association for Computational Linguis- tics, 9: 194.
Fabio Petroni, Aleksandra Piktus, Angela Fan, Patrick Lewis, Majid Yazdani, Nicola De Cao, James Thorne, Yacine Jernite, Vladimir Karpukhin, Jean Maillard, Vassilis Plachouras. Tim Rocktäschel, and Sebastian Riedel. 2021. KILT: a benchmark for knowledge intensive language tasks. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 2523- 2244,, Online. Association for Computational Linguistics.
Wenhan Xiong, Jingfei Du William Yang Wang, and Veselin Stoyanov. 2020. Pretrained encyclopedia: Weakly supervised prowtedge-reetraieed language model. In International Conference on Learning Representations.
Zhilin Yang, Peng Qi, Saizheng Zhang, Yoshua Bengio, William Cohen, Ruslan Salakhutdinov, and Christo- pher D. Manning. 2018. HotpotQA: A dataset for diverse, explainable multi-hop question answering. In Proceedings of the 2018 Conference on Empiri- cal Methods in Natural Language Processing, pages 2369-2380, Brussels, Belgium. Association for Com- putational Linguistics.
Yujia Qin, Yankai Lin, Ryuichi Takanobu, Zhiyuan Liu, Peng Li, Heng Ji, Minlie Huang, Maosong Sun, and Jie Zhou. 1.21. ERICA: Improving entity and rela- tion understanding for pre-trained language models via contrastive learning. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguisties and the 11th International Joint Confer- ence on Natural Language Processing (Volume 1: Online. Association for Computational I Linguistics.
Donghan Yu, Chenguang Zhu, Yiming Yang, and Michael Zeng. 2022. Jaket: Joint pre-training of knowledge graph and language understanding. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 36, pages 11630-11638,
Long Papers). pages
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee. Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. 2020. Exploring the limits of transfer learning with unified text-to-text trans- former. The Journal of Machine Learning Research, (1()::485-555)
Jingqing Zhang, Yao Zhao, Mohammad Saleh, and Pe- ter Liu. 2020. Pegasus: Pre-training with extracted gap-sentences for abstractive summarization. In In- ternational Conference on Machine Learning, pages 11328-11339, PMLR

---

Zhengyan Zhang, Xu Han, Zhiyuan Liu, Xin Jiang.
A.3 Downstream Experimental Details
Maosong Sun, and Qun Liu. 2019. ERNIE: En hanced language representation with informative en- tities. In Proceedings the 57th Annual Meeting of the Association for Computational Linguistics, pages 1441-1451, Florence, Italy. Association for Compu- tational Linguistics.
For downstream experiments, we fine-tune the models with a batch size 64, and search for learning rates among lc-5, 3c-5, lc-4. For QA and QG, we set the max input length as 512 tokens and max output length as 32. For summarization, the max input length as 1,024 tokens and the output length 128 tokens. For comparative QA, we select the best checkpoints by the high- est FI. For comparative QG, we select the best checkpoints by the highest BLEU. For comparative summarization, the best checkpoints are selected by the highest ROUGE-L. For evaluation metrics, we adopt the implementations of Exact Match, un- igram F1, and ROUGE-L by KILT (Petroni et al., 2021), and the BLEU implemented by the Hugging Face evaluate library (v0.3.4). For all downstream datasets, we report the average scores of three run with different random seeds.
Zhihan Zhang, Wenhao Yu, Chenguang Zhu, and Meng Jiang. 2022. unified encoder-decoder framework with entity memory. In Proceedings of the 2022 Con- ference Empirical Methods in Natural Language Processing, EMNLP.
A Experiment Details
Batch size
192
Learning rate
3e-5
Warmup ratio
1%
Max source length
512
Max target length
512
Training steps
70k
A.4 Experimental Details of ChatGPT
Table 8: Pre-training hyperparameters.
Since ChatGPT is a much larger model with much more pre-trained data, and the downstream datasets might have leaked to its training data, we provide the result of ChatGPT only performance of one of the most advanced large lan- guage models. We prompt ChatGPT (gpt3.5-turbo) in zero-shot setting and the prompts are shown in the below table. We modify the prompts from the ones we use with T5/BART and empirically choose the ones that work effectively, as shown in Table 10.
A.1 Pre-training Data
reference to the
For Wikidata, we use the dump wikidata-20220103 all. For Gigawords we the dump from https: //catalog. 1dc. upwnu. edu/LDc2e11T07. For CC-News we download the data from https: https://huggiggffce co/datasets/cccnnee new... We randomly sample small portion data as validation set. An example of pre-training data is shown in Table 3.
Train
Validation
All
219.598
6,252
Gigawords 110,399
3,179
Wikipedia 90,420
2,557
CC-News 18,779
516
Table 9: Number of quintuples in pre-training data from each unstructured data sources.
A.2 Pre-training Details
Our models are initialized from checkpoints facebook/bart-base and t5-base and further trained with pre-training tasks. All models are implemented with Hugging Face Transformers 4.17. The hyperparameters used in pre-training are listed Table 8. For text infilling, we mask 30% the tokens. The pre- pre-trained model checkpoints are selected by the lowest validation loss.

---

Task
Prompt
QA
Paragraph: [D,. D2]
(HotpotQA, 2WikiQA)
Question: [Q]
Only output the short answer.
QG
Paragraph: [D, D2]
(HotpotQG, 2WikiQG)
Given the paragraphs above, please output factoid question that requires
comparison of the two entities on some shared property.
Summarization (CocoTrip)
Reviews of Hotel 1: [D1]
Reviews of Hotel 2: [/2]
Given the reviews of the two hotels above, please output summary the
common opinions about both hotels, which only contains subjective information that is commonly described in both sets of the reviews. The summary should be than 30 words.
Comparative Summ (Diffen)
Paragraph I: [11]]
Paragraph 2: [D2] Based on the paragraphs above, output a comparative summary of the two entities. The summary should be less than 100 words.
Table 10: Prompts of comparative downstream tasks for ChatGPT.


Reviews:
[{"review_id": "xsP9RPXpIR", "reviewer": "Reviewer_EaYH", "summary": "", "questions": "1) How do you do zero-shot and few-shot with T5-base? I am surprised it works at all since it's trained with a denoising objective. Moreover, the base model is pretty small comparatively to models conventionally used for zero-shot which are in the billions of parameters.\n2) You have a ChatGPT baseline, but I do not see details on how it was done. For instance, was it zero-shot or few-shot?\n3) Looking at Table 7, it seems the Q/A has redundant information. For instance Q2 implies Q1 and Q3 and Q5 are essentially the same. Do you think that affects the model quality? At the least it could cause slower training.\n4) How much training data was possible? The pretraining set is fairly small (around 200k examples), was this the entirety of what could be obtained? When I think of LM pretraining, I think of scales at least in the orders of millions, this seems more like pre-fine-tuning contrary to the framing in the paper.\n5) What about catastrophic forgetting? Do the pertained T5/BART models retain their abilities?", "limitations": "", "rating": 3, "confidence": 4, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for your thoughtful review!\n\n- **Weakness 1, 2**: According to the biggest concern on whether our pre-trained models have general utility for other tasks, we would like to clarify that our model is pre-trained for comparative reasoning generation, not for general language modeling. Therefore, it's unrealistic and unnecessary to require the model to have general utility to be fine-tuned for other language tasks. Please see reponse to Q4 for further clarification.\n- **Q1, Weakness 5**: *\"How do you do zero-shot and few-shot with T5-base?\"*  T5 is pre-trained with a multi-task mixture of unsupervised and supervised tasks, and all tasks are converted to a text-to-text format using different prompts. Therefore, T5 models have the ability to perform zero-shot and few-shot learning as long as carefully designed prompts are given, such as the ones we used in Table 2.  We adopt the prefixes during pre-training as well as downstream fine-tuning. Due to the cost of pre-training large LMs on large corpora, we leave it to future work.\n- **Q2**: *Experimental details of ChatGPT.* It's zero-shot with ChatGPT (gpt3.5-turbo). We modify the prompts from the ones we use with T5/BART and empirically choose the ones that work effectively, otherwise ChatGPT will generate too lengthy responses. Since ChatGPT is a much larger model with much more pre-trained data, and the downstream datasets might have leaked to its training data, it is not a comparable baseline. We provide ChatGPT as a reference to the performance of one of the most advanced large language models. We appreciate your comment and will add experimental details to the appendix in our revised version. \n\nHere's a table showing the prompts used with ChatGPT.\n| Task | Prompt |\n| ----- | ---- | \n| Comparative QA (HotpotQA, 2WikiQA) |  Paragraph: [D1, D2] \\n Question: [Q] \\n Only output the short answer. |\n| Comparative QG (HotpotQG, 2WikiQG) | Paragraph: [D1, D2] \\n Given the paragraphs above, please output a factoid question that requires comparison of the two entities on some shared property. |\n|  Comparative Summ (CocoTrip) |  Reviews of Hotel 1: [D1] \\n Reviews of Hotel 2: [D2] \\n Given the reviews of the two hotels above, please output a summary of the common opinions about both hotels, which only contains subjective information that is commonly described in both sets of the reviews. The summary should be less than 30 words. |\n| Comparative Summ (Diffen) | Paragraph 1: [D1] \\n Paragraph 2: [D2] \\n Based on the paragraphs above, output a comparative summary of the two entities. The summary should be less than 100 words. | \n\n- **Q3**: *\"For instance Q2 implies Q1 and Q3 and Q5 are essentially the same.\"* We adopted templates with diversified language to represent the same knowledge quintuple for better generalization, inspired by Flan-T5 (Chung et al, 2022).\n- **Q4**: *\"The pretraining set is fairly small (around 200k examples), was this the entirety of what could be obtained?\"* Yes. The amount of quintuples depends on the size and coverage of unstructured data. With Gigawords (V5), CC-News (Hugging Face version) and Wikipedia, the current amount (219K) of quintuples is the entire set that we collect. However, the data size does not limit our contribution. First, we propose a labor-free framework to collect knowledge for entity comparison instead of a perfectly pre-trained model. The data size can be easily expanded by applying our framework to other corpora. Second, Fig 2 shows the effect of pre-training data size. The performance grows linearly as the data increases on a logarithmic scale, as discussed in Section 4.4.\n- **Q4**: *\"The pre-training in the paper is more like pre-fine-tuning\"* We appreciate your careful consideration regarding the terminology \"pre-training\". We had the same concern and carefully surveyed it before using the term. To our knowledge, pre-training is training on a large dataset using self-supervised or unsupervised objectives, aiming to learn representations that can be later fine-tuned for downstream tasks. General language modeling pre-training enables the model to perform language related tasks, but pre-training doesn't have to be restricted to language modeling. Task-adaptive pre-training on a smaller but directly task-relevant corpus has been studied and widely accepted (see ''Don't Stop Pretraining: Adapt Language Models to Domains and Tasks'' by S Gururangan et al, 2020). We have discussed related work that inject task-related knowledge into LMs through pre-training in Section 2.2. On the contrary, pre-finetuning is massively multi-task learning using human labeled data. Thus, we believe that \"pre-training\" would better describe our proposed framework.\n- **Q5**: *\"What about catastrophic forgetting? Do the pertained T5/BART models retain their abilities?\"* Again, we want to emphasize that we aim to pre-train LMs for comparative reasoning, not for general language modeling, so the ability to be fine-tuned for other language tasks is not in our consideration. To avoid catastrophic forgetting, we employ text infilling as the denoising task that aligns with the model's original pre-training, as described in Section 3.3.4.\n- **Weekness 6**: *\"There are no comparisons or mentions of any related work or alternative methods.\"* Comparative reasoning is an important type of intelligence that humans have and humans would like machines to have. It involves various NLP applications. We are the pilot study to enable pre-trained LM with such ability.\n- **Weekness 4**: *\"The summaries used in pertaining are generated from another pertained model. This is more like distillation than pretraining.\"* For data collection, we use an off-the-shelf data-to-text model to generate synthetic summaries given two **tuples** from our collected quintuples. This does not involve complex comparative reasoning since the data-to-text model is already given the correct tuples. In contrast, we pre-train our LM to generate this summary given two **documents**. This requires much harder comparative reasoning ability because the LM needs to attend to both documents, identify the comparable properties of two entities and generate a fluent and meaningful summary about the entities' commonalities and differences. In short, the collection of synthetic summaries is different from the pre-training objective of comparable summarization, and thus this is not distillation.\n\nHope our response addresses your concern! We appreciate your feedback and are open to further discussions.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1) The paper makes contributions in three different areas related to comparative reasoning: 1) An approach to automatically create training data using Wikipedia and Wikidata for training large language models. 2) Experiments showing the utility of using the curated data and 3) An evaluation benchmark for comparative reasoning involving 3 tasks and 4 datasets.\n2) The results show that pertaining on the their data creates models with stronger comparison reasoning abilities.\n3) The paper is very thorough, I like the experiments on showing that few-shot performance scales with the amount of training and improvements aren not simply from entity matching are examples of the paper's completeness.", "reasons_to_reject": "1) The pre-training in the paper is more like pre-fine-tuning as the dataset is fairly small ~200k examples and there are no experiments showing that the models retain their original abilities. Catastrophic forgetting could have happened for instance, making the models more like specially-tuned models for the tasks evaluated in the paper.\n2) The tasks used for evaluation are the same as those used in pre-training (QA, generated QA, summaries). This paper feels a little like overfitting to a specific set of tasks via pretaining and my concern is that the models are no longer general and wouldn't have much utility.\n2) Finetuning makes things very very close between the original models and the CMP ones which again argues that the models wouldn't have utility for other applications.\n4) The summaries used in pertaining are generated from another pertained model. This is more like distillation than pretraining. One could use pretrained models for many of these tasks which would hamper the interestingness of the work.\n5) No experiments are done with larger models, only base models are used. Do the results hold up with larger models? Few-shot and zero-shot are pretty weak for base models (and Id imagine even more so for models like T5 which are trained on a denoising objecting). The training data is pretty small, so experiments could likely be done on larger models to see if the results hold.\n6) There are no comparisons or mentions of any related work or alternative methods. Also the fact that the authors had to create all of their evaluations concerns me if this problem is very niche or already covered by other tasks.", "questions_to_authors": "1) How do you do zero-shot and few-shot with T5-base? I am surprised it works at all since it's trained with a denoising objective. Moreover, the base model is pretty small comparatively to models conventionally used for zero-shot which are in the billions of parameters.\n2) You have a ChatGPT baseline, but I do not see details on how it was done. For instance, was it zero-shot or few-shot?\n3) Looking at Table 7, it seems the Q/A has redundant information. For instance Q2 implies Q1 and Q3 and Q5 are essentially the same. Do you think that affects the model quality? At the least it could cause slower training.\n4) How much training data was possible? The pretraining set is fairly small (around 200k examples), was this the entirety of what could be obtained? When I think of LM pretraining, I think of scales at least in the orders of millions, this seems more like pre-fine-tuning contrary to the framing in the paper.\n5) What about catastrophic forgetting? Do the pertained T5/BART models retain their abilities?", "paper_topic_and_main_contributions": "This paper proposes an approach to collect data automatically for comparative reasoning using both structured and unstructured data. They use this data as pre-training for T5 and BART and show improvements across three areas 1) fine-tuning 2) few-shot 3) zero-shot.  With the largest gains coming from zero-shot and small gains coming from fine-tuning. For evaluation, they make another contribution by creating a benchmark of 3 tasks (QA, QA generation, and summarization) for comparative reasoning.", "excitement": 3, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {"Typos_Grammar_Style_and_Presentation_Improvements": "1) In Appendix B: \"demonstrate that even with the removal of overlapping entities, which constituted 97.6% of the pre-training data, the model’s few-shot performance remains strong.\" -> I interpret this as 97.6% of the data was removed (but I see further down that is not the case). I think this could be clearer.\n2) In Section 3.2 \"An example of a textualized quintuple is provided in Appendix A.1.\" -> There is no example here, did you mean Table 7?"}}, {"review_id": "DbyJPk2ynH", "reviewer": "Reviewer_5Nu8", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for acknowledging the value of our proposed collection pipeline of entity comparison data and our gathered downstream benchmark. We are also excited about comparative reasoning applications like the Diffen dataset.\n\nResponse to reviewer's concerns:\n- *\"For example, it seems to blur the lines between multi-hop QA and comparative reasoning - what is the difference? It seems like all multi-hop QA would be comparative.\"* First, we need to clarify that our research problem is not just multihop QA, but a wide range of applications involving comparative reasoning. Comparative reasoning is an important intelligent ability that is needed in the comparison multihop QA but also in many other tasks beyond QA. For example, we have included comparative QG and summarization in this study, but our collected quintuples for entity comparison can be easily adapted to other comparative reasoning tasks, which is valuable for future research. To further clarify, existing research on multi-hop QA mainly focused on retrieving the relevant document that contains the final answer. Differently, our work focuses particularly on the improvement of comparative reasoning ability of the LM itself over the retrieved relevant documents. Thus, our work is orthogonal to existing multihop QA work and our findings can be added to their work to yield further improvement.   \n- *\"There are no comparisons to other works on synthetic QA generation\"* *\" it is not very useful to know that using Hotpot-like training data would improve performance on Hotpot\"* Thanks for pointing out these relevant papers in synthetic QA generation, whose findings of the effectiveness of QA templates well align with ours. However, our work is in a different line from theirs: Our main contribution is the philosophy and the novel pipeline of collecting quintuples that store entity comparison knowledge for comparative reasoning. We design multiple approaches to textualize the knowledge for LM pre-training for multiple downstreams. Table 4 shows all 3 pre-training tasks are effective, not just QA generation. Furthermore, our collected quintuples can be easily adapted to other comparative reasoning tasks by varying the textualization approaches. However, we highly appreciate the references you listed and we find L Pan 2020 is the most relevant. They use similar QA templates but only on 4 pre-defined properties, while we propose a novel pipeline to mine the entities comparison knowledge from news and Wikipedia, which covers a much wider range of properties. What could make a fair comparison is: their T5 based comparison-only model reaches 38.2/45.0 on EM/F1 while ours reaches 44.25/52.62 on EM/F1. Other papers (Alberti et al, Lyu et al, and Lewis et al) didn't work on multi-hop QA so we considered them as related work but not baseline methods. Again, these works are **orthogonal** to ours and they can be developed upon our approach. We will cite the papers and add above discussions in our next revised version.\n- *\"The Diffen dataset is not discussed much and no examples are given\"* Thanks for acknowledging the value of Diffen dataset. An example is given in Table 5. We will give more examples in the appendix in the next revised version.\n- *\"The Diffen dataset and pre-training/mining pipeline is not mentioned to be open-sourced?\"* We have attached the code in additional materials for review session. We'll release the code and dataset upon acceptance. \n\nHope our response addresses your concerns! We appreciate your feedback and are open to further discussion.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- The paper is well written and the results clearly support the view that this approach improves over the baseline models\n- The Diffen dataset looks particularly useful for future work on this topic\n- They gather a large group of datasets to serve as a benchmark for comparative reasoning", "reasons_to_reject": "- The framing of this paper suggests that comparative reasoning is a unique phenomena that merits specialized evaluation and pre-training. I think that could be a valid premise, but I’m not sure the evidence in this paper establishes that. For example, it seems to blur the lines between multi-hop QA and comparative reasoning - what is the difference? It seems like all multi-hop QA would be comparative.\n- There are no comparisons to other works on synthetic QA generation - my guess is any other technique would perform similarly. My biggest concern is this lack of baselines, given the number of papers on the topic, I would hope to see at least one comparison (see references). It would also be important for the authors to cite some of these.\n- The “pre-training” pipeline is very similar to some of the datasets evaluated on. For example, HotpotQA (and 2WikaQA) was constructed by having humans annotate questions in nearly the same manner. Thus, it is not very useful to know that using Hotpot-like training data would improve performance on Hotpot (the more interesting results on CoCoTrip or Diffen, I think). Plenty of papers have shown that more synthetic similar data is better (see references). The authors provide a great analysis of entity overlap in the appendix, but I think that’s a distinct issue from whether having more data is better. Without the improvements from 2Wiki and Hotpot the  methods gains are much more minor.\n- The Diffen dataset is not discussed much and no examples are given. Unclear as to the quality of the dataset.\n- The Diffen dataset and pre-training/mining pipeline is not mentioned to be open-sourced? Happy to remove this point if the authors are doing that", "questions_to_authors": "", "paper_topic_and_main_contributions": "This paper proposes a new method for pre-training to improve on comparative question answering, question generation, and summarization. They do this by mining entity co-occurrences from various sources (Wikipedia, Gigaword, etc.) and use those mentions and the entities Wikipedia pages to automatically construct synthetic pre-training data. \n\nThey pre-train T5 and BART on this data and show improved performance over the base models with no pre-training.", "excitement": 3, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {"Missing_References": "These works represent the most influential ones I can think of for synthetic question generation. I would advise reading and following their citation trails as well:\n- Unsupervised Multi-hop Question Answering by Question Generation by Pan et. al 2020\n- Improving Unsupervised Question Answering via Summarization-Informed Question Generation by Lyu et. al 2021\n- Synthetic QA Corpora Generation with Roundtrip Consistency by Alberti et. al 2019\n- PAQ: 65 Million Probably-Asked Questions and What You Can Do With Them by Lewis et. al. 2021", "Typos_Grammar_Style_and_Presentation_Improvements": "The data example in the appendix should be moved up (IMO) when the authors have more space - not an issue right now. It was very helpful!\n\nVery minor: it could be interesting to make templates that are more diverse, perhaps ChatGPT generated templates. Could improve the results - but less important than having baselines."}}, {"review_id": "0SAy5jmal2", "reviewer": "Reviewer_W4mp", "summary": "", "questions": "*Question A:* I didn’t quite understand the objectives presented in 3.3.1 and 3.3.2. Is the model learning to generate these questions and answers in an unsupervised manner through the given objective, or did you collect ground truth question data?\n\n*Question B:* There isn’t much said about the ChatGPT evaluation in Table 3. What is the purpose of including ChatGPT in the evaluation? How exactly is it evaluated, e.g., did you use greedy decoding? Were some examples given in the prompt? What is the message of the ChatGPT results?", "limitations": "", "rating": 4, "confidence": 3, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for acknowledging the value of our study! In terms of your concerns, here are our clarifications:\n- **Weakness A & Question B**: Since ChatGPT is a much larger model with much more pre-trained data, and the downstream datasets might have leaked to its training data, it is not a comparable baseline. We provide ChatGPT as a reference to the performance of one of the most advanced large language models.\nWe prompt ChatGPT (gpt3.5-turbo) in zero-shot setting and the prompts are shown in the below table. We modify the prompts from the ones we use with T5/BART and empirically choose the ones that work effectively, otherwise ChatGPT will generate too lengthy responses. We appreciate your comment and will add experimental details to the appendix. \n\nHere's a table showing the prompts used with ChatGPT.\n| Task | Prompt |\n| ----- | ---- | \n| Comparative QA (HotpotQA, 2WikiQA) |  Paragraph: [D1, D2] \\n Question: [Q] \\n Only output the short answer. |\n| Comparative QG (HotpotQG, 2WikiQG) | Paragraph: [D1, D2] \\n Given the paragraphs above, please output a factoid question that requires comparison of the two entities on some shared property. |\n|  Comparative Summ (CocoTrip) |  Reviews of Hotel 1: [D1] \\n Reviews of Hotel 2: [D2] \\n Given the reviews of the two hotels above, please output a summary of the common opinions about both hotels, which only contains subjective information that is commonly described in both sets of the reviews. The summary should be less than 30 words. |\n| Comparative Summ (Diffen) | Paragraph 1: [D1] \\n Paragraph 2: [D2] \\n Based on the paragraphs above, output a comparative summary of the two entities. The summary should be less than 100 words. | \n\n- **Question A**: *\"Is the model learning to generate these questions and answers in an unsupervised manner through the given objective, or did you collect ground truth question data?\"*  In Section 3.3.1 and 3.3.2, we pre-train the model with the data collected by our labor-free and scalable framework. The QA pairs are textualized from the entity comparison quintuples using pre-defined templates so it doesn't involve human-labeled ground truth. Detailed description about the textualization can be found in Section 3.2.3.\n\nHope our response addresses your concerns! We appreciate your feedback and are open to further discussion.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "*Strength A.* The paper presents substantial resources for research in comparative reasoning, building a foundation for future research on a problem that is seemingly difficult for current SOTA language models.\n\n*Strength B.* The proposed LM pre-training paradigms demonstrate sharp performance improvements on comparative reasoning tasks over vanilla counterparts (and sometimes even ChatGPT). Performance gains are especially apparent in lower-resource settings, which may be common as comparative reasoning annotation is expensive to collect and can be domain-specific. The evaluations used for comparative reasoning seem mostly sound.\n\n*Strength C.* Analysis of pre-training objectives shown in Table 4 and Section 3.4 demonstrate the benefit brought by each objective, further strengthening the authors’ contribution.", "reasons_to_reject": "*Weakness A.* While the evaluations show that the authors’ proposed pre-training paradigms help improve the performance of T5 and BART, it’s unclear how to contextualize results with the current paradigm of using larger language models, e.g., ChatGPT, in zero-shot and in-context learning settings to tackle language tasks. Based on the results in Tables 3-4, it seems that their proposed approach really sees big benefits when applied in a zero-shot setting on downstream comparative reasoning tasks. This finding may be relevant to this paradigm, but the authors don't give any details or discussion about their evaluation of ChatGPT in Table 3, which makes it difficult to judge whether it’s comparable to their zero-shot results.", "questions_to_authors": "*Question A:* I didn’t quite understand the objectives presented in 3.3.1 and 3.3.2. Is the model learning to generate these questions and answers in an unsupervised manner through the given objective, or did you collect ground truth question data?\n\n*Question B:* There isn’t much said about the ChatGPT evaluation in Table 3. What is the purpose of including ChatGPT in the evaluation? How exactly is it evaluated, e.g., did you use greedy decoding? Were some examples given in the prompt? What is the message of the ChatGPT results?", "paper_topic_and_main_contributions": "This paper focuses on and builds a foundation for research in comparative reasoning, a problem with limited resources. The authors make three main contributions: \n1. A scalable data collection method for comparative reasoning that mines entities from structured resources such that the entities have common properties over which they can be compared, then automatically retrieves text documents describing these entities and their properties through a heuristic approach.\n2. A new pre-training paradigm for LMs that enhances their ability to compare entities in text using text-to-text generation objectives based on the data collected through (1). Specifically, they train an LM to learn to generate comparative questions, question-answer pairs, and summaries, while balancing this with a text infilling task over documents collected through (1).\n3. A novel benchmark for this task, which was extracted from subsets of other benchmark datasets and websites.\n\nThis paper is mostly successful in demonstrating that their data collection approach can improve the performance of LMs on comparative reasoning. While I am struggling to contextualize the findings of this work with the advances in the most recent LLMs, I hope the authors will provide some clarity.", "excitement": 4, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Missing_References": "None I can think of."}, "extra_text": {"Typos_Grammar_Style_and_Presentation_Improvements": "The abstract may be more clear if it starts with a sentence about what comparative reasoning is and why it’s important, e.g., “Comparative reasoning is the process of  … which can enable …”.\n\nL023: do you have a citation for this?\n\nIn the results tables, it may help to explain in captions that R-L and R-2 refer to ROUGE metrics.\n\nIn some spots, e.g., Table 3 caption, you mention performance differences are “significant”. You may want to provide some statistical significance testing to justify this, as not all performance differences are huge."}}]

Author rebuttal:

Meta-review: After several rounds of discussion with the authors, the reviews lean positive on this paper (both reviewers who left 3 "soundness" scores indicated they were really 3.5s in in their comments). The reviewers highlight the thoroughness of the the analysis and the usefulness of this work as a resource for others working in comparative reasoning. There are some concerns about comparative reasoning as a task, specifically whether it should come at the expense of other quality aspects and how it relates to other tasks and datasets (that could potentially make a stronger evaluation).
Decision: Accept-Main