Causal Intervention for Abstractive Related Work Generation
Authors: Jiachang Liu
Qi Zhang
Chongyang Shi
Usman Naseem
Shoujin Wang
Liang Hu
Ivor Tsang
Venue: EMNLP 2023 Findings (2023)
Keywords: Related work generation
text summarization
causal intervention
TL;DR: An abstractive related work generation method that incorporates causal intervention to improve the quality of the generated text.

Abstract: Abstractive related work generation has attracted increasing attention in generating coherent related work that helps readers grasp the current research. However, most existing models ignore the inherent causality during related work generation, leading to spurious correlations which downgrade the models’ generation quality and generalizability. In this study, we argue that causal intervention can address such limitations and improve the quality and coherence of generated related work. To this end, we propose a novel Causal Intervention Module for Related Work Generation (CaM) to effectively capture causalities in the generation process. Specifically, we first model the relations among the sentence order, document (reference) correlations, and transitional content in related work generation using a causal graph. Then, to implement causal interventions and mitigate the negative impact of spurious correlations, we use do-calculus to derive ordinary conditional probabilities and identify causal effects through CaM. Finally, we subtly fuse CaM with Transformer to obtain an end-to-end related work generation framework. Extensive experiments on two real-world datasets show that CaM can effectively promote the model to learn causal relations and thus produce related work of higher quality and coherence.

Causal Intervention for Abstractive Related Work Generation
Jiachang Liu'**, Qi Zhang 2.5,x Chongyang Shili
Usman Naseem SShoujin Wang', Liang Hu Ivor W. Tsang"?
'Beijing Institute of Technology, China Tongji University, China
James Cook University, Australia *University of Technology Sydney, Australia DDeepBlue Academy of Science, China
CFAR, Agency for Science, Technology and Research, Singapore 'IHPC Agency for Science, Technology and Research, Singapore [jj_lii, cy_shijebe t. edu. cn "zhanggi_csetongeee eedu.cc cn
Abstract
summarization task (Li and Ouyang, 2022). Dis- tinct from multi- multi-docummnt summarization, related work generation entails comparison after the sum- marization of a set of references and needs to sort out the similarities and differences between these references (Agarwal et al., 2011)
Abstractive related work generation has at- tracted increasing attention generating co- herent related work that helps readers grasp the current research. However, most existing models ignore the inherent causality during re- lated work generation, leading to spurious cor- relations which downgrade the models' genera- tion quality and generalizability. In this study, we argue that causal intervention can address such limitations and improve the quality and coherence of generated related work. To this end, we propose a novel Causal Intervention Module for Related Work Generation (CaM) to effectively capture causalities in the generation process. Specifically, we first model the rela- tions among the sentence order, document (ref- erence) correlations, and transitional content in related work generation using a causal graph. Then, implement causal interventions and mitigate the negative impact of spurious corre- lations, we use do-calculus derive ordinary conditional probabilities and identify causal effects through CaM. Finally, we subtly fuse CaM with Transformer obtain an end-to-end related work generation framework. Extensive experiments on two real-world datasets show that CaM can effectively promote the model to learn causal relations and thus produce related work higher quality and coherence.
Recently, various abstractive text generation methods have been proposed to generate related work based on the abstracts of references. For ex- ample, Xing et al. (2020a) used the context of cita- tion and the abstract of each cited paper the input to generate related work. Ge al. (2021) encoded the citation network and used edge to generate related work, Chen et (2022) proposed a target-aware related work generator that captures the relations between reference papers and the target paper through target-centered attention mechanism. Equipped with well-designed encod- ing strategies, external knowledge, or novel train- ing techniques, these studies have made promising progress in generating coherent related work.
external knowl-
However, those models are inclined to explore and exploit spurious correlations, such high- frequency word/phrase patterns, writing habits, presentation skills, to build superficial shortcuts between reference papers and the related work of the target paper. Such spurious correlations may harm the quality the generated related work, es- pecially when distribution shift exists between the testing set and training set. This is because spuri- ous correlations are different from genuine causal relations. They often do not intrinsically contribute to the related work generation and easily cause the robustness problem and impair the models' gener- alizability (Arjovsky et al., 2019).
1 Introduction
A comprehensive related work usually covers abun- dant reference papers, which costs authors plenty of time in reading and summarization and even forces authors to pursue ever-updating advanced work (Hu and Wan, 2014). Fortunately, the task of related work generation emerged and attracted increasing attention from the community of text summariza- tion and content analysis recent years (Chen et al., 2021, 2022). Related work generation can be considered as a variant the
Figure illustrates the difference between causality and spurious correlation. The phrases "for example" and "later" are often used to bridge two sentences in related work. Their usage may be attributed to writers' presentation habits about or- ganizing sentence orders or the reference document
The authors contribute equally to this work.
Corresponding author.

---

nents: 1) Primitive Intervention cuts off the connec- tion that induces spurious correlations in the causal graph by leveraging do-calculus and buckdoor cri- terion (Pearl, 2009a), 2) Context-aware Remapping smoothens the distribution of intervened embed- dings and injects contextual information, and 3) Optimal Intensity Learning learns the best intensity of overall intervention by controlling the output from different parts. Finally, strategically fuse CaM with Transformer (Vaswani et al., 2017) de- liver an end-to-end causal related work generation model. Our main contributions are as follows:
For ccomple [1] take in set keywords
amagal in hicnechicaal fashion to
TR
darive..
exploits PI PIASA mnddl aplit sertence set of multiple reference
RELATION
pepers itto...
Training Set
Howerer
2nd
SENTENCE ORDER
For example
Later
Test Set
G25: Geneml Specific Th: Temporal Relation Con: Contralive Pro: Progreséive Sparioas Carrelation
Causality
Figure I:
An illustration of the effect difference be-
tween causality (solid arrows) and spurious correlations (dashed arrows) in related work generation.
To the best of our knowledge, this work the first attempt to introduce causality theory into the related work generation task.
relations corresponding to the sentences. Ideally. a related work generation model is expected to learn the reference relation and distinguish it from the writing habits. However, previous generation models easily capture the superficial habitual sen- tence organization (a spurious correlation) instead of learning complex causal reference relations, es- pecially when the habitual patterns frequently oc- cur in the training set. In this case, the transitional phrases generated mainly based on writing habits are likely to be unsuitable and subsequently af- feet the content generation of related work during testing when the training and testing sets are not distributed uniformly.
We propose a novel Causal Intervention Module for Related Work Generation (CaM) which uti- lizes causal intervention mitigate the impact of spurious correlations. CaM is subtly fused with Transformer derive an end-to-end causal related work generation model, enabling the prop- agation of intervened information.
Extensive experiments on two real-world bench- mark datasets demonstrate that our proposed model can generate related works of high qual- ity and verify the effectiveness and rationality of bringing causality theory into the related work generation task.
Fortunately, causal intervention can effectively remove spurious correlations and focus causal relations by intervening in the learning process. It not only observes the impact of the sentence order and document relation on generating transitional content, but also probes the impact of each possi- ble order on the whole generation of related work, thereby removing the spurious correlations (Pearl, 2009a). Accordingly, causal intervention serving an effective solution allows causal relations to exert greater impact and instruct the model to produce the correct content.
2 Problem Formulation
Given a set of reference papers = {r..... Miii. we assume the ground truth related work Y (W1, W2.... WM), where r; = (wf. wa...., w(()) denotes a single cited paper, w/ is the j-th word in rús and wy is the j-th word in related work Y. Generally, the related work generation task can be formulated as generating related work sec- tion Y (w1, 2.,,..,, u (y) based on the reference input D and minimizing the difference between Y and Y. Considering that the abstract section is usually well-drafted to provide a concise paper summarization (Hu and Wan, 2014), we use the abstract section to represent each reference paper.
Accordingly, to address the aforementioned gaps in existing work for related work generation, we propose a novel Causal Intervention Module for Related Work Generation (CaM), which effectively removes spurious correlations by performing the causal intervention. Specifically, we first model the relations among sentence order, document relation, and transitional content in related work generation and identify the confounder that raises spurious correlations (see Figure 2). Then, we implement causal intervention that consists of three compo-
3 Methodology
We first analyze the causalities in related work gen- eration, identify the confounder that raises spurious correlations, and use a causal graph to model these relations. Then, we implement CaM enhance the quality of related work through causal intervention. Finally, we describe how CaM, as an intervention

---

of both + y and +
do(x)
3.2 Causal Intervention Module for Related Work Generation
x
x: document relation c sentence order y: transitional content
The proposed CaM contains three parts as shown Figure 3: Primitive Intervention performs causal intervention and preliminarily removes the spuri- ous correlations between sentence order and tran- sitional content. SeelienttContex-tte Context-aware Remapping cap- tures and fuses contextual information, facilitating the smoothing of the intervened embeddings. Op- timal Intensity Learning learns the best intensity of holistic causal intervention.
Figure 2: Causal graph for related work generation. By applying do-calculus, path is cut off and the impact of spurious correlation c + x + y is mitigated.
module, integrated with Transformer to inter- vene the entire generation process. The overall structure of our model shown in Figure 3.
3.1 Causal Modeling for Related Work Generation
3.2.1 Primitive Intervention
Based on the causal graph G shown in Figure 2, first perform the following derivation using do- calculus and backdoor criterion.
We believe that three aspects play significant roles in related work generation for better depicting the relations between different references, namely, sen- tence order c, document relation æ, and transitional content y (illustrated in Figure 2). In many cases, sentence order is independent the specified con- tent and directly establishes relations with transi- tional content. For example, we tend to use "firstly" at the beginning and "finally" at the end while com- posing paragraph, regardless of what exactly is in between. This relation corresponds to path + y. and should be preserved as a writing experience or habit. Mcanwhile, there a lot of transitional content that portrays the relations between referred papers based on the actual content, at this time, models need to analyze and use these relations. The corresponding path is -> y.
p(yl do(()))()))))))))))))))))) (c)
= ()(((lc,))((cdo((z))
(1)
EEp(yll,
In short, the do-calculus is mathematical represen- tation of an intervention, and the backdoor criterion can help identify the causal effect of x y (Pearl, 2009b). As result, by taking into consideration the effect of each possible value of sentence order c on transitional content y, c stops affecting docu- ment relation æ when using x to estimate y, which means path e + x is cut off (see the arrow-pointed graph in Figure 2). Next, we will explain how to estimate separatcly P(y c) and p(c) using deep learning models and finally obtain p(y ((y)do(z)).
Though idcally, sentence order and document re- lation can instruct the generation of transitional con- tent based on practical writing needs, deep learning models are usually unable to trade off the influence of these two aspects correctly but prioritize sen- tence order. This can be attributed to the fact that sentence order information is casily accessible and learnable. In Figure 2, such relation corresponds to c > x > y. In this case, sentence order e the confounder that raises a spurious correlation with transitional content y. Although performing well on the training set, once a data distribution shift ex- ists between the test set and training set where the test set focuses more document relations, the transitional content instructed by sentence order can be quite unreliable. To mitigate the impact of the spurious correlation, we need to cut off the path c - I, enabling the model to generate transitional content based on the correct and reliable causality
Let E E g denote the input embeddings corresponding to M-sized related work and Eite c TRM xd denote the output embeddings Primitive Intervention. We first integrate the sentence order information into the input embeddings:
odr(s) Linear(e," (eart O oj), efri e;
Eort (2)
where 1...., 8, is the total number of sen- adr()) tences in the generated related work and e, denotes the order-enhanced embedding for the i-th word. We take oj = (Ig (j 1), .g (j + 1)) with the same dimension layer (i.e., Linear) further projects the concate- nated embedding to odr with the same dimension as cori Accordingly, we have the estimation plyl x. c) == cadr Then, we feed the subsequence
cors The linear

---

Primitive Intorveation
Fridctdd Lziehdagg
Mirs
Remupping
ud
POlK
polas
aurbutty
Enender
de
M
Transfurmer Black
PDfR)
POTR RAPCO)
Maga
S4rAi
Nora
CaM
Ooderizt
XN
Transfienner
Add & Niem
Maak
CaM
Furwand Layee
Optimal Inteasity Learning
W2
Figure 3: The structure of CaM fused with the Transformer in the decoder. CaM consists of three parts: Intervention. Context-aware Remapping and Optimal Intensity Learning.
Primitive
E1 1 to feed-forward network to predict the sen- tence position probability of the current decoding word:
where WIN(-) returns a consecutive subsequence of Fit at length n.w- Then, we follow the process of Multi-head Attention Mechanism (Vaswani al.,
h; Soltmax(PPN(ReLUTT) i-1
Eite ))
2017) update the embeddings in B;:
BTp MultiHead Bi, Bi) (e. rmpp
(6)
where each hr E h; denotes the probability. Thus, we estimate the sentence position probability of each decoding word p(c) h. After obtaining the estimation of ply|2r, e) and p(c). the final embed- ding with primitive causal intervention achieved:
(1)
Even though all embeddings in B; are updated, we only keep the renewed e (Th /2) E B, as the output, and leave the rest unchanged. Since WIN(.) scans the entire sequence step by step, ev- ery embedding will have the chance to update. The output is denoted ErmPp R. Mxd
1i
odr(s) x 12 1/ E hy (4)
odrli) x hi multiplying sentence order prob- where ej
ability with order-enhanced embeddings is exactly plul,, ()p(c) in Equation! 1. Since most transitions are rendered by start words, our CaM intervenes only with these words, namely part of edtv E being equal e gori For simplicity, still use Eite in the following.
3.2.3 Optimal Intensity Learning
There is no guarantee that causal intervention with maximum (unaltered) intensity will improve model performance, especially when combined with pre- trained models (Brown et al., 2020; Lewis et al., 2020), as the intervention may conflict with the pre-training strategies. To guarantee performance, propose the Optimal Intensity Learning. By applying Primitive Intervention and Context- aware Remapping, we have three types of embed- dings, E and Errp, To figure out their respective importance to the final output, we derive the output intensity corresponding to each of them:
3.2.2 Context-aware Remapping
Two problems may exist in Primitive Intervention: The lack of learnable parameters may lead to the intervened embeddings and the original ones being apart and obstructs the subsequent decoding process. 2) Intervention in individual words may damage the context along with the order-enhanced embedding. To solve the two problems, we propose Context-aware Remapping mechanism. First, scan Esto with context window of fixed size ney:
rri oWori. cort
g jtn (Wite ari
(8)
mmp = a (W CorL)
(9)
B: = WIN(E
(5)
ori
stv mm fo(ig n0 .g' .grmp))
(10)

---

Statistic
S2ORC
Delve
trained minimize the cross-entropy loss between the predicted Y and the ground-truth Y, the vocabulary index for wi E Y:
Pairs
126k/5k/5k
72k/3k/3k
source
5.02
3.69
words/sent(doc)) #
1079/45
626/26
wsrds/sent(summ) #
148/6.69
181/7.88
C - - MM (og
(15)
vocab size #
377,431
190,381
Table I: Statistics of the datasets
4 Experiments
where o-) the sigmoid function, fs(-) is the softmax function. Combining car, ctt crmp,
4.1 Datasets
we can obtain the optimal intervention inten- sity and the final word embedding set -opm opm
Following the settings in Chen al. (2021, 2022), we adopt two publicly available datasets derived from the scholar corpora S20RC (Lo et al., 2020) and Delve (Akujuobi and Zhang, 2017) respec- tively to evaluate our proposed method in related work generation. S2ORC consists of scientific pa- pers from multiple domains, and Delve focuses on the computer domain. The datasets summarized in Table 1, where the corresponding ratios of the training/validaatioonnee pairs are detailed !
opm) with causal intervention: N
e opm crrigrri + clue give + e
(11)
3.3 Fusing CaM with Transformer
To derive an end-to-ent causal generation model and ensure that the intervened information can be propagated, we choose to integrate CaM with Transformer (Vaswani et al., 2017). However, un- like the RNN-based models that generate words recurrently (Nallapati et al., 2016), the attention mechanism computes the embeddings of all words in parallel, while the intervention is performed on the sentence start words.
4.2 Settings
In experiments, we incorporate CaM into the Transformer decoder (see Figure and evaluate our model using the resultant encoder-decoder ar- chitecture. We utilize pre-trained weights from BERT (Devlin et al., 2019) for both the encoder and decoder of the architecture, as described in Rothe et al. (2020) 2. Also, when CaM is removed from the decoder in the following experiments, the remaining Transformer model evaluate still em- ploys pre-trained weights.
To tackle this challenge, we perform vocabulary mapping on word embeddings before intervention and compare the result with sentence start token [CLS] obtain Mas::
= arggaax[Lneetrroo [Err)]]
(12)
Mask = (,, IDCLs)
(13)
In the Transformer architecture we use, the di- mension of word embedding is 768, both the num- ber of attention heads and hidden layers in the en- coder and decoder are 12, and the intermediate size is 3072. We implement our model with PyTorch on NVIDIA 3080Ti GPU. The maximum reference paper number is set to 5, i.e., [ 5. We select the first |40/| words in each reference paper ab- stract and concatenate them to obtain the model input sequence. The total number of sentences target related work is set to 6, i.e., s = 6. We use beam search for decoding, with beam size of 4 and maximum decoding step of 200. We use SGD as the optimizer with learning rate le 3. We use ROUGE-I, ROUGE-2, and ROUGE-L on Fl as the metries (Lin, 2004; Jiang al., 2022).
I contains the vocabulary index of each word. 5(-) compares the values of the two parameters, and returns 1 if the same, 0 otherwise. Mask indicates whether the word is sentence start word. There- fore, Evom can be calculated as:
Fopm Foom Form Mask F0r Mask) (14)
The @ operation multiplies each embedding with the corresponding 1) values, and ~ denotes the inverse operation. Note that we omit Mask for conciseness in Section 3.2.3. Mask helps re- store the non-sentence-star word embeddings and preserve the intervened sentence-start ones.
As illustrated in Figure 3, we put CaM between the Transformer layers in the decoder. The anal- ysis of the amount and location settings will be discussed in detail in Section 4.6. The model is

---

S2ORC
Delve
Model
ROUGE-1
ROUGE-2
ROUGE-L
ROUGE-1
ROUGE-2
ROUGE-L
TextRank
22.36
2.65
19.73
25.25
3.04
22.14
BertSumEXT
24.62
3.62
21.88
28.43
3.98
24.71
MGSum-eXI
24.10
3.19
20.87
27.85
3.95
24.28
TransformerABS
21.65
3.64
20.43
26.89
3.92
23.64
BertSumABS
23.63
4.17
21.69
28.02
3.50
24.74
MGSum-abs
23.94
4.58
21.57
28.13
24.95
GS
23.92
4.51
22.05
28.27
4.36
25.08
TS-base
23.20
4.01
21.41
26.38
5.69
24.35
BART-base
23.36
4.13
21.08
26.96
5.33
24.42
longformer
26.00
4.96
23.20
28.05
5.20
25.65
RRG
25.46
4.93
22.97
29.10
4.94
26,29
NG-Abs
25.06
5.18
22.33
27.49
5.93
24.56
TAG
25.04
5.68
23,02
27.82
6.16
25,50
CaM (ours)
26.65
5.40
24.62
29.31
6.17
26.61
Table 2: ROUGE scores comparison between CaM and the baselines.
4.2.1
Extractive Methods
28
(1) TextRank (Mihalcea and Tarau, 2004): A graph-based text ranking model that can be used in multi-document sentence extraction. (2) Bert- SumEXT (Liu and Lapata, 2019): An extractive summarization model that extends BERT by in- serting multiple [CLS] tokens. (3) MGSum-ext (Jin et al., 2020): multi-granulaan network that jointly learns different semantic representations. Abstractive Methods: (1) Transformer
ROVDEL
RO0003
Figure Ablation results on S2ORC.
(Vaswani et al., 2017): An abstractive summa- rization model based on Transformer. (2) Bert- SumABS (Liu and Lapata, 2019): An abstractive model based BERT with designed two-stage fine-tuning approach. (3) MGSum-abs (Jin al.,
ROUSEL
2020): graniiiranularity interaction network that can be utilized for abstractive document summa- rization.(4) GS (Li et al., 2020): An abstractive summarization model that utilizes special graphs to encode documents capture cross-document re- lations. (5) T5-base (Raffel et al., 2020): A text-to- text generative language model that leverages trans- (6) BART-base (Lewis to-sucuen---
Sc0ms ROUSE:
on
Figure 5: Ablation results Delve.
fer learning techniques.
TAG (Chen et al., 2022): takes the paper that lated work belongs to as the target and employs a target-centered attention mechanism to generate related work.
al., 2020): A powerful model that combines the benefits autoregressive and denoising pretraining objectives. (7) Long- former (Beltagy et al., 2020): A transformer-based model that can efficiently process long-range de- pendencies in text. (8) RGG (Chen et al., 2021) An encoder-decoder model specifically tailored for related work generation, which constructs and re- fines the relation graph of reference papers. (9)
4.3 Overall Performance
It can be found in Table 2 that abstractive models have attracted more attention in recent years and usually outperform extractive ones. Among the generative models, pretrained model T5 and BART achieve promising results our task without ad- ditional design. Mcanwhile, Longformer, which
NG-Abs (Zhu al., 2023): A BART model that optimized jointly with the cross-entropy loss and the proposed differentiable N-gram objectives. (10)
is good at handling long text input, also achieves

---

lavorable results. However, the performance of these models is limited by the complexity of the academic content in the dataset.
ROUGED WET ROLGE4
Scone ROUGES
Our proposed CaM achieves the best perfor- mance on both datasets. Due to fusing CaM with Transformer, its large scale ensures that our model still effectively capture document relations without additional modeling. Accordingly, CaM enables the model obviate the impact of spurious correlations through causal intervention and pro- motes the model to learn more robust causalities to achieve the best performance.
24
26
Nanber of calls fased transfomer
Figure 6: Performance analysis on the number of CaMs fused with Transformer.
ROUSE
ROUSE
ROUGE-L
4.4 Ablation Study
0.5%
24.70
24.52
To analyze the contribution of the different com- ponents of CaM, we separately control the use of Primitive Intervention (PI), Context-aware Remap- ping (RMP) and Optimal Intensity Learning (OPT). Figure and Figure 5 show the performance parison between different variants of CaM.
25.73
2111
22.%
6.45
35%
45%
Trandfomer
.5%%
Figure Comparison between Transformer and CaM original and reordered samples.
First, we observe that Transformer already guar- antees a desirable base performance. When only PI is used, the model generally shows a slight perfor- mance drop. PI+RMP outperforms RMP, showing the necessity of the PI and the effectiveness of RMP. PI+RMP+OPT achieves optimal results, indicating that OPT can effectively exploit the information across different representations.
4.6 Fusing Strategy Comparison
In our setting, the Transformer decoder consists 12 layers, so there are multiple locations to fuse different number of CaMs. For cach scenario, CaMs are placed evenly among the Transformer decoder layers, and one will always be placed at the end of the entire model. The results of all cases are shown in Figure 6. can be observed that the model performs best when the number of CaM is 4 both on S2ORC and Delve. With small number CaMs, the model may underperform the benchmark model and fail to achieve optimal performance due to the lack of sufficient continuous intervention. If there too many CaMs, the distance between different CaMs will be too short, leaving insuf- ficient learning process for the entire fused model, and this might cause the CaMs bring the noise.
4.5 Human Evaluation
inf
coh
QA
CaM
2.21
2.38
41.6
RRG
2.07
2.10
2.05
34.1
Transformer
2.11
1.97
1.92
38.3
Table 3: Human evaluation result
We evaluate the quality of related work gener- ated by the CaM, RRG, and Transformer from three perspectives (informativenesss, coherence, and succinctness) by randomly selecting forty samples from S2ORC and rating the generated results by 15 PhD students. In the QA task, three PhD students posed three questions for each sample, ensuring that the answers existed in ground truth. Partici- pants need to answer these questions after reading the generated text and we use accuracy the met- ric. As table shows, our method achieves the best in informativeness, coherence, and the QA task. However, succinctness is slightly lower than RRG, probably due to the output length limit.
4.7 Robustness Analysis
4.7.1 Testing with Reordered Samples
We randomly select 50 samples (15 from S2ORC and 35 from Delve) and manually rearrange the order of the cited papers and the order of their cor- responding sentences in each Transstiraniiiionnll content in related work is also removed since the reordering damages the original logical relations. Figure 7 shows that CaM has better performance no matter whether the samples have been reordered or not. Regarding the reordered samples, the per-

---

6.73
F1
13
perforzancee
algorihmm
Da aos
ISOMLAP iangen! grypt algorithes
calutecuant non-leed study fnrger effoct two flodings behavioni
Sinctsional roduction
kernel mainx:
approximaat" rofonn
compant
0.133
works
work
Compare document rebations
0.333
Geuerate "howover" causally
0.323
tel grounds low
tie higlty anilyss
keneel adapting
leamed that
Figure 8: The result of migrating test set from Delve S2ORC (trained on Delve).
Unclear deciion
Generate "the" empirically
formance of Transformer decreases on all three metries, but CaM only decreases on ROUGE-1 and ROUGE-2 at a much lower rate. Particularly, com- pared to Transformer, CaM makes improvement on ROUGE-L when tested with reordered samples. The result indicates that CaM is able to tackle the noise disturbance caused by reordering, and the generated content maintains better coherence.
Figure Visualization of the generating process within CaM and Transformer.
corresponding to "however" and "the" to the input content using different color shades (Figure 10) to explore what information these two words rely More details of the above two experiments be found in Appendix B.
We picked out the words that "however" and "the" focused on the most and analyzed the implica- tions of these words in the context of the input. The results are shown in Figure 9. It can be found that the words highlighted by CaM have their respec- tive effects in the cited papers. When generating "however", the model aggregates this information, comparing the relations between the documents and producing the correct result. However, there is no obvious connection between the words focused on by Transformer, hence there is no clear decision process after combining the information, and the generated word "the" is simply a result obtained from learned experience and preference. Through causality visualization, it can be observed very cretely how CaM improves model performance by conducting causal intervention.
4.7.2 Testing with Migrated Test Set
We train the models on Delve and test them on S2ORC, which is challenging task and signif- icant for robustness analysis. As expected, the performances of all models drop, but we can still obtain credible conclusions. Since CaM outper- forms Transformer initially, simply comparing the ROUGE scores after migrating the test set is not informative. To this end, we use Relative Ouiper- formance Rate (ROR) for evaluation:
ROR (ScaM STF)/STF
(16)
SCaM and STF are the ROUGE scores of CaM and Transformer, respectively. ROR computes the advantage of CaM over Transformer.
Figure 8 reports that CaM outperforms Trans- former regardless of migrating from Delve to S20RC for testing. In addition, comparing the change of ROR, we observe that although migra- tion brings performance drop, CaM not only main- tains its advantage over Transformer but also en- larges it. The above two experiments demonstrate that the CaM effectively learns causalities to im- prove model robustness.
Conclusions
In this paper, we propose a Causal Intervention Module for Related Work Generation (CaM) to capture causalities in related work generation. We first model the relations in related work generation using causal graph. The proposed CaM imple- ments causal intervention and enables the model to capture causality. We subtly fuse CaM with Trans- former to obtain end-to-end model to integrate the intervened information throughout the gener- ation process. Extensive experiments show the superiority of CaM and demonstrate our method's effectiveness.
4.8 Causality Visualization
To visualize how causal intervention works in the generation process, we compare the related work generated by Transformer and CaM with a case study. Specifically, we map their cross attention

---

Limitations
Target- aware abstractive related work generation with contrastive learning. In Proceedings of the 45th In- ternational ACM SIGIR Conference on Research and Development in Information Retrieval, SIGIR '22, page 373-383, New York, NY, USA. Association for Computing Machinery.
Although extensive experiments have demonstrated that CaM can effectively improve the performance of the generation model, as mentioned above, since the intervention occurs the sentence start words, it is inconclusive that CaM can bring improvement if the generation of sentence start words is inaccu- rate. That is, if is to be combined with small-scale models without any pre-trained knowledge, then the effectiveness of the model might not be ensured. This will also be a direction of improvement for our future work.
Xiuying Chen, Hind Alamro, Mingzhe Li, Shen Gao, Xi- angliang Zhang, Dongyan Zhao, and Rui Yan. 2021. Capturing relations between scientific papers: An abstractive model for related work section generation. In P Proceedings of the 59th Annual Meeting of the A Association for Computational Linguistics and the 11th International Joint Conference on Natural Lan- guage Processing (Volume I: Long Pupers), pages 6068-6077, Online. Association for Computational Linguistics.
Acknowledgements
Jacob Devlin, Ming-Was Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language under- standing. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguisties: Humun Lunguage Tech- nologies, Volume (Long and Short Papers), pages 4171-41866 Minneapolis, Minnesota. Association for Computational Linguistics.
This work is supported by the Fundamental Re- search Funds for the Central Universities.
References
Nitin Agarwal, Ravi Shankar Reddy, Kiran Gvr, and Carolyn Penstein Rosé. 2011. SeiSumm: A multi- document summarization system for scientific arti- cles. In Proceedings of the ACL-HLT 2011 System Demonstrations, pages 115-120, Portland, Oregon. Association for Computational Linguistics.
Fuli Feng. Jizhi Zhang, Xiangnan He, Hanwang Zhang. and Tat-Seng Chua. 2021. Empowering language un- derstanding with counterfactual reasoning. In Find- ings of the Association for Computational Linguis- rics: ACL-IJCNLP 2021, pages 2226-2236, Online, Association for Computational Linguisties.
Uchenna Akujuobi and Xiangliang Zhang. 2017. Delve: A dataset-driven scholarly search and analysis system. 19(2):36-46.
Yubin Ge. Ly Dinh, Xiaofeng Liu, Jinsong Su, Ziyao Lu, Ante Wang, and Jana Diesner. 2021. BACO: A background knowledge- and content-based frame- work for citing sentence generation. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume I: Long Papers), pages 1466-1478, Online. Association for Computational Linguisties.
Martin Arjovsky, Léon Bottou, Ishaan Gulrajani, and David Lopez-Paz. 2019. Invariant risk minimization.
Peter C Austin. 2011. An introduction to propensity score methods for reducing the effects of confound- ing in observational studies. Multivariate Behav Res, 46(3):399-424,
Iz Beltagy, Matthew E. Peters, and Arman Cohan. 2020. Longformer: The long- cong-document transformer. arXiv:2004.05/500
Cong Duy Vu Hoang and Min-Yer Kan. 2010. Towards automated related work summarization. In Coling 2010: Posters, pages 427-435, Beijing, China. Col- ing 2010 Organizing Committee.
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neclakantan, Pranav Shyam, Girish Sustry, Amanda Askell, Sandhini Agarwal, Ariel Serbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel Ziegler, Jeffrey Wu, Clemens Winter, Chris Hesse, Mark Chen, Eric Sigler, Ma- teusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever. and Dario Amodei, 2020. Language models are few-shot learners. In Ad- vances in Neural Information Processing Systems, volume 33, pages 1877-1901. Curran Associates, Inc.
Yue Hu and Xiaojun Wan. 2014. Automatic generation of related work sections in scientific papers: An opti- mization approach. Proceedings of the 2014 Con- ference on Empirical Methods in Natural Language Processing (EMNLP), Doha, Qatar. Association for Computational Linguistics.
Xinyu Jiang, Qi Zhang, Chongyang Shi, Kaiying Jiang. Liang Hu, and Shoujin Wang. 2022. An ion exchange mechanism inspired story ending generator for dif- ferent characters. In Machine Learning and Knowl- edge Discovery in Databases- European Conference, ECML. PKDD 2022, Grenoble, France, September 19-23, 2022, Proceedings, Part II, volume 13714 of Lecture Notes Computer Science, pages 553-570. Springer.
Xiuying Chen, Hind Alamro, Mingzhe Li, Shen Gao, Rui Yan, Xin Gao, and Xiangliang Zhang. 2022.

---

Hanqi Jin, Tianming Wang, and Xiaojun Wan. 2020. Multi-ge Multi-graaullartyy interaction network for extractive and abstractive multi-document
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. 2020. Exploring the limits of transfer learning with a unified text-to-text trans- former.
summarization. In Proceedings of the 58th Annual Meeting of the Asso- ciation for Computational Linguistics, pages 6244- 6254, Online. Association for Computational Lin- guistics.
Sascha Rothe, Shashi Narayan, and Aliaksei Severyn. 2020. Leveraging pre-trained checkpoints for se- quence generation tasks. Transactions of the Associ- ation for Computational Linguistics, 8:264-280.
Mike Lewis. Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, AbdeIrahman Mohamed, Omer Levy. Veselin Stoyanov, and Luke Zettlemoyer. 2020. BART: Denoising suuuence-t--squeene pre-training for natural language generation, translation, and com- prehension. In Proceedings of the 58th Annual Meet- ing of the Association for Computational Linguistics, pages 7871-7880 Online. Association for Computa- tional Linguisties.
Ashish Vaswami, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, E ukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In Advances in Neural Information Pro- cessing Systems, volume 30. Curran Associates, Inc.
Pancheng Wang. Shasha Li, Haifang Zhou, Jintao Tang. and Ting Wang. 2020. Toc-rwg: Explore the com- bination of topic model and citation information for automatic related work generation. IEEE Access, 8:13043 13055.
Wei Li, Xinyan Xiao, Jiachen Liu, Hua Wu, Haifeng Wang, and Junping Du. 2020. Leveraging graph improve abstractive multi-document summariza- tion. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 6232 6243, Online. Association for Computational Linguistics.
Yichao Wang. Huifeng Guo, Bo Chen, Weiwen Liu, Zhi- rong Liu, Qi Zhang, Zhicheng He, Hongkun Zheng, Weiwei Yao, Muyu Zhang. Zhenhua Dong, and Ruim- ing Tang. 2022a. Causalint: Causal inspired inter- vention for multi-scenario recommendation, In Pro- ceedings of the 28th ACM SIGKDD Conference on Knowledge Discovery and Data Mining, KDD '22, page 4090-4099, New York, NY, USA. Association for Computing Machinery.
Xiangei Li and Jessica Ouyang. 2022. Automatic re- lated work generation: A meta study.
Yew Lin. 2004. ROUGE: package for auto- matic evaluation of summaries. In Text Summariza- tion Branches Out, pages 74-81, Barcelona, Spain. Association for Computational Linguistics.
Zhao Wang and Aron Culotta. 2020. Identifying spu- rious correlations for robust text classification. Findings of the Association for Computational Lin- guisties: EMNLP 2020, pages 3431 -4400, Online. Association for Computational Linguisties.
Yang Liu and Mirella Lapata. 2019. Text summariza- tion with pretrained encoders. In Proceedings of the 2019 Conference on Empirical Methods in Natu- ral Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 3730-3740, Hong Kong, China. Association for Computational Linguistics.
Zhenlei Wang. Shiqi Shen, Zhipeng Wang, Bo Chen, Xu Chen, and Ji-Rong Wen. 2022b. Unbiased se- quential recommendation with latent confounders. WWW '22, page 2195-2204, New York, NY, USA. Association for Computing Machinery.
Kyle Lo, Lucy Lu Wang, Mark Neumann, Rodney Kin- ney, and Daniel Weld. 2020. S2ORC: The semantic scholar open research corpus. In Proceedings of the 58th Annual Meeting of the Association for Compu- tational Linguistics, pages 4969-4983, Online. Asso- ciation for Computational Linguistics.
Xinyu Xing, Xiaosheng Fan, and Xiaojun Wan. 2020a. Automatic generation of citation texts in scholarly papers: A pilot study. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguisticss pages 6181-6190, Online. Association for Computational Linguistics.
Rada Mihalcea and Paul Tarau. 2004. TextRank: Bring- ing order into text. In Proceedings of the 2004 Con- ference on Empirical Methods in Natural Language Processing, pages 404-411 411, Barcelona, Spain. Asso- ciation for Computational Linguistics.
Xinyu Xing, Xiaosheng Fan, and Xiaojun Wan. 2020b. Automatic generation of citation texts in scholarly papers: A pilot study. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 6181-6190, Online. Association for Computational Linguistics.
Ramesh Nallapati, Bowen Zhou, Cicero Nogueira dos santos, Caglar Gulcehre, and Bing Xiang. 2016. Abstractive text summarization using sequence-to- sequence rnns and beyond.
Yunqi Zhu, Xuebing Yang, Yuanyuan Wu, Mingjin Zhu, and Wensheng Zhang. 2023. Differentiable n n-gram objective on abstractive summarization. Expert Sys- tems with Applications, 215:
Judea Pearl. 2009a. Causal inference in statistics: An overview. Statistics Surveys, 3(none):96 146.
Judea Pearl. 2009b. Causality: Models, Reasoning and Inference. Cambridge University Press.

---

A Related Work
B Experiment Result for Causal Visualization
A.1 Related Work Generation
In this section, we will give an extra analysis of the experiments introduced in Section 4.8.
The related work generation task can viewed as a variant of the multi-document summarization task, and its methods can be categorized as extractive or abstractive. Most of the early studies use extractive methods. The work of Hoang and Kan (2010) is one of the first attempts. They propose heuristic approach generate general and specific content separately given a topic tree. Wang et al. (2020) train the model extract cited text spans through a specific training set and use greedy algorithm to select the most suitable candidate sentences to compose related work. Most recent studies focus on abstractive approaches. Xing et al. (2020b) use the citation context and the abstract of the cited papers together as inputs to generate citation text. Chen al. (2021) construct relation graph of the cited papers during the encoding process and update them iteratively. The relation graph is used as an auxiliary information for decoding. The most recent work is done by Chen et al. (2022), in which they take the paper that related work belongs to as the target and employ target-centered attention mechanism to generate informative related work.
B.1 Generated Related Work Comparison
From Table 4, we can notice that CaM generates riched content and its meaning is closer ground truth compared to Transformer. Crucially, when pointing out the problems of previous approaches and presenting the new oncs(sentence marked in green), CaM correctly generates "however" at the beginning of the sentence and the entire sentence has more accurate expression, making the transi- tions more seamless. But Transformer only gener- ates very high-frequency word "the" at the same position. It be perceived that in this process Transformer is not making effective decisions, but simply generating with preference and experience.
B.2 Visualization Result Analysis on Full Text
Figure 10 visualizes the cross attention of words "however" and "the" in CaM and Transformer. Dif- ferent cited papers are split with vertical lines. The deeper blue color denotes the higher attention re- ccived by the input source word. Judging from the overall coloring situation, can find that in CaM, there is more deep blue text, as well as more light-colored text. This means the information that "however" focuses on more targeted and more important, and CaM is capable to produce correct content by accurately capturing document relations and avoid distractions from the confounder. In the result of Transformer, both light and deep blue text become less visible, and the coverage of normal blue increases greatly, indicating that "the" focuses on wider range of information but lacks emphasis. indicates that the decision process in Transformer is unclear and ineffective.
A.2 Causal Intervention
recent years, causality theory has attracted in- creasing attention in various domains. In the field recommendation system, Wang et al. (2022a) use the causal graph to model multi-scenario rec- ommendation and solve the problem of existing sys- tems that may introduce unnecessary information from other scenarios. Wang et al. (2022b) propose framework for sequential recommendation that can perceive data biases by reweighing training data and using inverse propensity scores(Austin, 2011), the field of natural language processing, Feng et al. (2021) introduce counterfactual reasoning into the sentiment analysis task and leverage the knowledge of both factual and counterfactual sam- ples. Wang and Culotta (2020) propose method for identifying spurious correlations in the text clas- sification task. The method extracts the words with the highest relevance to the category and uses an estimator to determine whether the correlation is a spurious correlation.
Detailed analysis of the exact words they focus on and the decision process of the models is pre- sented in Section 4.8.

---

Many dimension reduction techniques are proposed based on the vector forms, which are generally divided into two parts, linear and nonlinear. The classical methods of principal component analysis and multi- dimensional scaling are linear, since the outputs returned by these methods are related to the input patterns by a simple linear transformation.
Ground Truth
However. when the imput patterns lie on or near low dimensional sub of the input space, that the structure of the data set may be highly nonlinear, then linear methods are bound to fail. As the research for manifold learning, several graph- based nonlinear methods have been proposed, such as locally linear em. Reduction methods have been proposed on the dimensional space, such are divided into two categories: linear and nonlinear. The first method are the component analysis, the dimensional of the methods are linear to the kernel data. The data of the input dimensional space are not lincar to the large dimensional space. The data space dimensional of the data be the nonlinear, and are not used. The graph-based non- lincar methods have been proposed. Including as the lincar kemel, and entropy.
Transformer
Reduction methods have been proposed on the kernel space, such are divided into two categories: linear and nonlinear. The first approach component analysis are linear and dimensional analysis are based the kernel of the methods. Data of the input are not represented to the low dimensional space.
CaM
However, the data are not on low dimensional space. The data space is more nonlincar, and the methods can not be used. The graph-based nonlinear methods have been proposed. Including as the linear entropy.
Table 4: Related work generated by CaM and Transformer. Analysis of the bolded words in Section 4.8.
order to evaluste the fits of pc. md and ise
in order to eveluste the fits of pc, md and so on co compaaabl grounds wo use the residual variance [UNK)] -. the mat is of auclidean distances in the ow em rer recovered by each algorithm [UNK]] is each algorithm bbst estimana of the intrinsic manifolld distancess for iso. this is the graph distance matrix for ac and
on comparable grounds. we use the residual verience [UNK].. i
the maini of
dislances
low rec avvered by algorithm
isNK] each aigorithmm best estimate of rinsic manifold distances
fio iso, this the graph distance
pc
md,
sucidman inputs distance matri excast with
mid.. in the eucliccon inputs distance matrix except with the hand
where md use the tangent distance. is standard innar correlaiin conmicints taklen over all ontrees [UNK]and each
where mi uses the tangent distance is the stancard linear correlation coefficien taken over all entries of [UNK] and in each sequence shown, the three intermediete images are those closest..ight rrihtt, and light and
sequence shown,
three intermediste images are those closest. right,, and superir frnnall left rightttt..... and
superior for fronal
current study. previous study with larger onhancement non trend was highly significent.. subjects study. enelyzed
antouui the improvement performaace
althouuh the Improvement performance with cho enhancement was non freend current study i provious study larçer sample effect was highly sianificant... data six seven subjects because the behaviorsl data
current study, we analyzed
beh dehaviral
subject findings
difference significance
subject unavail due computer lure, difference the sign ficence
the difference
inding simply result diffrrence sample sizes
analysis shows the
difforance and
arelysi algerihim size the rt iffrencc veriability
(wo algorthm thm nonlinear dimensional recuction
semi
Jow
reducctin
matrix factor algariti dimensional manifold. in
kemel matrix
prog
the
high
conrotonall
dimensional
kamei matris learned dimensional manifold.
the variance feature distances and between nearest neighbors. graphs. show that full kemel matrix be approximate product the kemn matrix this way.
the kerne
variance feature
while
presenving distances angles between nearest noighorss this adapting
adapting recent ideas
show
that full matrix be well
product
smaller matrices... reform
smaller matrices, representing the kerell matrit way we reform
CaM
Transformer
Figure 10: Raw visualization result from CaM on the word "however" and Transformer on the word "the".


Reviews:
[{"review_id": "DrbIBR4gVe", "reviewer": "Reviewer_nTMu", "summary": "", "questions": "The transformer model used in the ablations – I assume this is using the BERT architecture just without the CaM add-ons, and that the “transformer” model was still trained on the same data as the full CaM model with the same settings.  Is that the case? ", "limitations": "", "rating": 4, "confidence": 3, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely appreciate you taking the time and providing valuable comments. In the following, we provide a detailed response to address all of your concerns.\n\n**1. Sections 3.2.1 and 3.2.2 can be a bit confusing and need to be expanded more (having a running example might help with this). Some of the variable naming conventions were difficult to keep track of (especially when there are multiple different subscripts and superscripts).**\n   \n> + Thanks very much for your suggestion. Section 3.2.1 and Section 3.2.2 introduce the key parts of the causal intervention module. To provide a better understanding of the two sections, we will provide examples based on Figure 1 or Figure 2 in Sections 3.2.1 and 3.2.2.\n> + Regarding the variable naming conventions, we will carefully check and refine the notations to guarantee consistency in the paper. In addition, we have substituted the complex superscripts and subscripts with single characters, for example changing **ori, itv, rmp, and opm** with **b, a, c, o**, to provide clear/simplified formations of our proposed model.\n\n**2. Only one experiment is done with human evaluations and the performance differences are much smaller in the human evals. Expanding the human evaluation would strengthen the results. Discussion of results felt a bit condensed. I would love to see this expanded on.**\n\n> + Note that in this paper, we conducted human evaluation from three aspects, informativeness, coherence, and succinctness [1,2]. Our model scores the highest point on informativeness and coherence, and just a little bit lower on succinctness compared to RRG (probably due to the max output length).\n> + To address your concern, we expanded the human evaluation and designed an additional QA task. Specifically, three Ph.D. students posed three questions for each of the sampled 40 instances, ensuring that the answers existed in the ground truth. We invited participants to answer these questions after reading the text generated by CaM, RRG, and Transformer, and the PhD students would judge the correctness. The result precision for each of the three models are as follows:**CaM-41.6%, Transformer-38.3%, RRG-34.1%**\n> We will update the human evaluation in the final version.\n\n[1] https://aclanthology.org/2021.acl-long.473/\n\n[2] https://dl.acm.org/doi/10.1145/3477495.3532065\n\n**3. Discussion of results felt a bit condensed. I would love to see this expanded on.**\n\n> Thanks for your suggestion. We will polish the discussion of our experimental results in the final version to deliver a high-quality evaluation analysis.\n\n**Q1. The transformer model used in the ablations – I assume this is using the BERT architecture just without the CaM add-ons,  and that the “transformer” model was still trained on the same data as the full CaM model with the same settings. Is that the case?**\n\n> - Yes. The “Transformer” model in the ablations is the one without CaM add-ons, but it is a transformer-based encoder-decoder generation model and is not exactly the same as BERT. Specifically, the encoder can be seen as a BERT model without the classification layer, and its output is the input of the decoder. The decoder can be seen as a BERT model with extra cross-attention layers in each block, and the final classification layer is kept. Both the encoder and decoder are loaded with BERT pre-trained weights, and the cross-attention layers are randomly initialized \n(ref: https://huggingface.co/docs/transformers/main/en/model_doc/encoder-decoder#transformers.EncoderDecoderModel).\n> - We have tried to integrate CaM into T5 or BART decoders, but the improvement is limited. We believe this is because BERT is mainly trained at the token level (masked LM), and our interventions are also performed at the token level, such that the granularity match allows CaM to be more effective.\n> - The rationale behind the design of T5 or other LLMs is to model multiple types of NLP tasks as sequence-to-sequence tasks, which makes the fine-tuning effect of the model more dependent on the design of the input sequences (including prefix or prompt technique), and the incorporation of certain sub-modules within the model is likely to have a negative effect.\n\n**Typos Grammar Style And Presentation Improvements**\n\n> Thanks for your valuable suggestions and comments. We will redraw Figure 8 to report the results of ROR and draw another graph to report the results of ROUGE in the appendices.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- Interesting and very novel approach to this problem\n- New causal Intervention Module which could potentially have applications in other tasks with similar confounding factors\n- Promising quantitative improvements with multiple metrics with ablations and analysis of the results to show the utility of each component and the robustness of the technique.", "reasons_to_reject": "- Section 3.2.1 and 3.2.2 can be a bit confusing and need to be expanded more (having a running example might help with this).  Some of the variable naming conventions were difficult to keep track of (especially when there are multiple different subscripts and superscripts) .\n- Only one experiment is done with human evaluations and the performance differences are much smaller in the human evals.  Expanding the human evaluation would strengthen the results\n- Discussion of results felt a bit condensed.  I would love to see this expanded on.", "questions_to_authors": "The transformer model used in the ablations – I assume this is using the BERT architecture just without the CaM add-ons, and that the “transformer” model was still trained on the same data as the full CaM model with the same settings.  Is that the case? ", "paper_topic_and_main_contributions": "This paper introduces a new approach to related work generation that separates the spurious correlations associated with sentence order that may confound transitional language (as opposed to understanding nuanced differences in how the works compare to each other).  They introduce the CaM module that does this using primitive intervention and a few other context-related modeling techniques and integrate this within a transformer architecture by alternating with transformer blocks. They achieve improvements in human and automatic metrics on two datasets and perform ablations.  They also demonstrate more robustness to adversarial augmentation (reordering sentences).", "excitement": 4, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Typos_Grammar_Style_and_Presentation_Improvements": "Figure 8, the rouge&ROR on the same axis was a bit hard to read.  I might suggest just an ror graph would be better with the rouge numbers pushed to the appendices."}, "extra_text": {}}, {"review_id": "1fFLihJrpI", "reviewer": "Reviewer_DEd8", "summary": "", "questions": "1. The authors chose to fuse CaM with Transformer to obtain an end-to-end model, why not other modules. The authors should explain more. \n2. As the authors said, CaM is incorprated into the Transformer decoder and utilize pre-trained weights from BERT. As BERT and Transformer has different architectures, how can the pretrained weights of BERT be applied to Transformer?\n3. As the authors asid, among the generative models, pretrained model T5 and BART achieve promising results in our task without additional design. why not fuse CAM with T5 or BART?\n4. The authors claimed that the proposed CaM achieves the best performance on both datasets. However, as listed in Table 2, ROUGE-2 for TAG on S2ORC is 5.68, higher than 5.40 of CAM.", "limitations": "", "rating": 3, "confidence": 4, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks very much for your comments.\n\n**1. The authors chose to fuse CaM with Transformer to obtain an end-to-end model, why not other modules. The authors should explain more.**\n\n> - According to [1], there are over 70 mainstream transformer-based architectures available now. The attention mechanism allows the transformer model to accelerate computation in parallel and far outperforms RNN-based models in terms of both network depth and supported sequence lengths.\n> - As discussed in Limitations, the causal intervention mainly happens on sentence start words. That is, if the sentence start words are not accurately generated in the first place, it is inconclusive that CaM can still bring improvements. Public pre-training research based on transformers is very well developed, and this makes it easier to choose pre-training weights if transformers are used as the backbone model.\n> - The characteristics of transformer layer stacking and the consistency in the dimensionality of input and output information make our packaged CaM easy to integrate with.\n\n> [1] https://arxiv.org/abs/2302.07730\n \n**2. As the authors said, CaM is incorprated into the Transformer decoder and utilize pre-trained weights from BERT. As BERT and Transformer has different architectures, how can the pretrained weights of BERT be applied to Transformer?**\n\n> - As mentioned earlier, we chose the transformer because of its superior performance, more easily integrated structure, and support for pre-trained weights. The pre-trained weights can ensure a relatively favorable generation of initial sentence-initial words, which sets the ground for CaM to exert further effects.\n> - In the experiments, the “Transformer” model in the ablations is a transformer-based encoder-decoder generation model. Specifically, its encoder can be seen as a BERT model without the classification layer, and its decoder can be seen as a BERT model with extra cross-attention layers in each block with the final classification layer being kept. Both the encoder and decoder are loaded with BERT pre-trained weights, and the cross-attention layers are randomly initialized.\n> (ref: https://huggingface.co/docs/transformers/main/en/model_doc/encoder-decoder#transformers.EncoderDecoderModel).\n\n**3. As the authors said, among the generative models, pretrained model T5 and BART achieve promising results in our task without additional design. Large models may perform better after prompt tuning, why not fuse CAM with T5 or other LLM?**\n\n> Since our backbone model can be considered as stacking two BERT-base modules, the total number of parameters is basically the same as the T5-base (220M). We have tried to integrate CaM into T5 or BART decoders, but the improvement is limited. We believe this is because BERT is mainly trained at the token level (masked LM), and our interventions are also performed at the token level, such that the granularity match allows CaM to be more effective. However, the rationale behind the design of T5 or other LLMs is to model multiple types of NLP tasks as sequence-to-sequence tasks, which makes the fine-tuning effect of the model more dependent on the design of the input sequences(including prefix or prompt technique), and the incorporation of certain sub-modules within the model is likely to have a negative effect.\n\n**4. The authors claimed that the proposed CaM achieves the best performance on both datasets. However, as listed in Table 2, ROUGE-2 for TAG on S2ORC is 5.68, higher than 5.40 of CAM.**\n\n> We are sorry for the mistake. We will fix it in the final version.\n\nThank you for your comments and valuable suggestions. We hope our explanations can address all your concerns.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "Extensive experiments on two real-world datasets show that CaM can effectively promote the model to learn causal relations and thus produce related work of higher quality and coherence.", "reasons_to_reject": "1. The authors chose to fuse CaM with Transformer to obtain an end-to-end model, why not other modules. The authors should explain more. \n2. As the authors said, CaM is incorprated into the Transformer decoder and utilize pre-trained weights from BERT. As BERT and Transformer has different architectures, how can the pretrained weights of BERT be applied to Transformer?\n3. As the authors said, among the generative models, pretrained model T5 and BART achieve promising results in our task without additional design. Large models may perform better after prompt tuning, why not fuse CAM with T5 or other LLM?\n4. The authors claimed that the proposed CaM achieves the best performance on both datasets. However, as listed in Table 2, ROUGE-2 for TAG on S2ORC is 5.68, higher than 5.40 of CAM.", "questions_to_authors": "1. The authors chose to fuse CaM with Transformer to obtain an end-to-end model, why not other modules. The authors should explain more. \n2. As the authors said, CaM is incorprated into the Transformer decoder and utilize pre-trained weights from BERT. As BERT and Transformer has different architectures, how can the pretrained weights of BERT be applied to Transformer?\n3. As the authors asid, among the generative models, pretrained model T5 and BART achieve promising results in our task without additional design. why not fuse CAM with T5 or BART?\n4. The authors claimed that the proposed CaM achieves the best performance on both datasets. However, as listed in Table 2, ROUGE-2 for TAG on S2ORC is 5.68, higher than 5.40 of CAM.", "paper_topic_and_main_contributions": "This paper proposed a Causal Intervention Module for Related Work Generation (CaM) to effectively capture causalities in the generation\n process. This model takes the inherent causality during related work generation into consideration, sovling spurious correlations which downgrade the models’ generation quality and generalizability.", "excitement": 3, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "aq8yedEeLa", "reviewer": "Reviewer_WopL", "summary": "", "questions": "How many different types of causality relations are considered in this paper? Is there any specific design for temporal and logical relations in CAM framework? The dependencies among c, x, and y need to be carefully explained, is there any observations or theoretical evidence to prompt the design of causal graphs?", "limitations": "", "rating": 2, "confidence": 3, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We much appreciate your time and valuable comments. We provide the following explanations to address your concerns.\n\n**1. The descriptions of some key concepts need to be clarified, e.g., causality, spurious correlations, and paths.**\n> - In our context, causality can be regarded as cause-and-effect relationships between two variables, and there are no confounding variables that could distort the result.\n> - If a correlation between two variables is affected or enhanced by a third variable, then this correlation is a spurious correlation.\n> - A directed path from $a$ to $b$ can be considered as a causal relation without other unobserved variables entangled.\n> - We will provide more explicit definitions of these concepts in the final version.\n\n**2. The automatic evaluation of Rouge-based metrics is not convincing enough, more metrics to measure semantic similarity, correctness, distinction, and repetition should be considered.**\n> - We followed prior related work generation research [1, 2] to choose our automated metrics. Since ROUGE-based metrics mainly measure n-gram content matching, they do lack in the evaluation of aspects such as semantics and repetition. \n> - In addition, we adopt another three metrics, i.e., informativeness, coherence, and succinctness, in human evaluation to alleviate the limitations of the ROUGE metrics to some extent.\n> - To better evaluate the effectiveness of the model, we will add more automated evaluation metrics in the final version.\n> [1] https://aclanthology.org/2021.acl-long.473/\n> [2] https://dl.acm.org/doi/10.1145/3477495.3532065\n\n**3. It is a bit overclaimed about the contribution of the causality intervention by cutting off the dependency of spurious correlation, the results show a subtle improvement compared with TAG.**\n> - Note that in **Table 2**, compared to the TAG, our model achieves significantly higher ROUGE-1 and ROUGE-L scores and gets comparable ROUGE-2 scores on both two datasets. These results indicate the superiority of our proposed CAM and verify the contribution of the causality intervention. In addition, we have provided visualization experiments to interpret the effectiveness of our proposed model, which also verifies the contribution of the causality intervention in improving the interpretability of the related work generation.\n\n**1. How many different types of causality relations are considered in this paper?**\n\n> Two types of causality are considered. They are, respectively, from sentence order to transitional content and from document relations to transitional content.\n\n**2. Is there any specific design for temporal and logical relations in CAM framework?**\n\n> - Among the two causal relations we expect the model to learn, the one from sentence order to transitional content covers a significant amount of temporal relations, since the multi-document summary content organized on the basis of temporal relations tends to be presented according to the order of the current sentence (e.g., using *in early time*, *later*, *recently* for each sentence in order). \n> - The causal relation from document relations to transitional content covers a large number of logical relations since we want the model to decide what transitional content to generate by analyzing the logical relations between documents.\n> - Overall, the design of temporal and logical relations is present in CaM, but not deliberately explicit, and instead causal interventions enable the model to autonomously learn the importance of each of these relations in different scenarios.\n\n**3. The dependencies among c, x, and y need to be carefully explained, is there any observations or theoretical evidence to prompt the design of causal graphs?**\n\n> - As analyzed in [1], if the model's learning process is not interfered with, then the model is prone to learn the correlations that exist in the training data but are unreliable and underperforms on out-of-distribution samples (e.g., since the background of pictures of Huskies is basically snowy, then given a picture of a Chihuahua in the snow, the model will most likely misclassify it as a Husky). Our analysis of the spurious correlations existing in the related work generation process and the modeling of causal graphs are inspired by it to some extent. We hope that causal intervention will guide the model to perform equally well on out-of-distribution samples. Based on this goal and our prior knowledge, we designed the causal graph and derived the causal intervention process in the hope that the model can learn more reliable causal relations motivated by this process.\n> - Although we mentioned that a large number of repeated patterns of transitional words in the training set would cause the model to simply learn a shortcut without analyzing the document relations anymore, we did not have statistics on the real situation in the training set, and we do not think that this is necessary either. On the one hand, when the theoretical modeling is complete, we need the model's generalization ability to learn certain aspects that the theory fails to fully take into account, and on the other hand, detailed ablation experiments can illustrate the effectiveness of our design both theoretically and in terms of implementation details.\n> [1] https://dl.acm.org/doi/10.1145/2939672.2939778\n\n**Typos Grammar Style And Presentation Improvements**\n> - Thanks for your suggestions. We will try to elaborate on our motivation and concepts in our final version.\n> - We are sorry for the second mistake. We will fix it in the final version.\n\nThank you for your comments and valuable suggestions. We hope our explanations can address all your concerns.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. The idea to produce causal intervention for generating related work is very interesting\n2. The framework is introduced clearly and sounds reasonable.\n3. Proper ablation studies demonstrates the effectiveness.", "reasons_to_reject": "1. The descriptions of some key concepts need to be clarified, e.g., causality, spurious correlations, and paths.\n2. The automatic evaluation of Rouge-based metrics is not convincing enough, more metrics to measure semantic similarity, correctness, distinction, and repetition should be considered.\n3. It is a bit overclaimed about the contribution of the causality intervention by cutting off the dependency of spurious correlation, the results show a subtle improvement compared with TAG.", "questions_to_authors": "How many different types of causality relations are considered in this paper? Is there any specific design for temporal and logical relations in CAM framework? The dependencies among c, x, and y need to be carefully explained, is there any observations or theoretical evidence to prompt the design of causal graphs?", "paper_topic_and_main_contributions": "The article proposes a novel approach for improving the quality and coherence of abstractive related work generation using causal intervention. The proposed Causal Intervention Module for Related Work Generation (CaM) effectively captures causality in the generation process by mitigating the negative impact of spurious correlations and injecting contextual information. CaM is fused with Transformer to deliver an end-to-end causal related work generation model. Extensive experiments demonstrate that CaM outperforms the state-of-the-art approaches and verifies the effectiveness of bringing causality theory into the related work generation task.", "excitement": 2, "reproducibility": 2, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {"Typos_Grammar_Style_and_Presentation_Improvements": "The writing needs to be improved to provide more details about the motivation and foundation of the proposed approach. Standard formulations and definitions would aid the understanding of concepts. \nIn Table 2, on S2ORC, the rouge-2 value of CaM is 5.40 which is lower than that of TAG, while CaM is marked as the best model with 5.40 bolded."}}]

Author rebuttal:

Meta-review: The paper improves the performance of related work generation by introducing casual intervention that addresses specific confounds (e.g. sentence order and transitional languages) to transformer-based pretrained models. Experiments demonstrate that it works well compared to state-of-the-art models. Most reviewers thought the proposed approach is novel and interesting, and that the experiments are extensive in that it covers two datasets and there are various tests and analyses to measure the utility of each component. That said, there are some concerns that reviewers have identified:  (1) from the discussions it appears that the proposed casual intervention technique mostly benefits BERT-based models and not other pretrained models - while the authors had a reasonable speculation (owing to the differences in their pretraining objectives), this is a significant limitation that needs to be properly discussed in the paper; (2) reviewers would appreciate if the human evaluation is done more thoroughly; (3) the performance gain appears to be less than what it claims - authors should probably tone this down to avoid misleading the readers; and (4) reviewers thought certain parts of the paper are unclear or difficult to follow (e.g. key concepts and the baseline transformer encoder-decoder model needs more clarification).
Decision: Accept-Findings