Probing Graph Decomposition for Argument Pair Extraction
Year: 2023

Abstract:
Argument pair extraction (APE) aims to extract interactive argument pairs from two passages within a discussion. The key challenge of APE is to effectively capture the complex contextaware interactive relations of arguments between the two passages. In this paper, we elicit relational semantic knowledge from large-scale pre-trained language models (PLMs) via a probing technique. The induced sentence-level relational probing graph can help capture rich explicit interactive relations between argument pairs effectively. Since the relevance score of a sentence pair within a passage is generally larger than that of the sentence pair from different passages, each sentence would prefer to propagate information within the same passage and under-explore the interactive relations between two passages. To tackle this issue, we propose a graph decomposition method to decompose the probing graph into four subgraphs from intraand inter-passage perspectives, where the intra-passage graphs can help detect argument spans within each passage and the inter-passage graphs can help identify the argument pairs between the review and rebuttal passages. Experimental results on two benchmark datasets show that our method achieves substantial improvements over strong baselines for APE.

heading: 1 Introduction
text: Dialogical argumentation, which focuses on the analysis of argumentation in debates or discussions, has rapidly emerged as a hot research topic in recent years. Argument pair extraction (APE) (Cheng et al., 2020) is a new and challenging task in the field of dialogical argumentation, which aims to extract interactive argument pairs from two argumentative passages within a discussion. As illustrated in Figure 1, a peer review process involves rich interactive arguments with each argument consisting
∗ Min Yang and Ruifeng Xu are corresponding authors.
of several consecutive sentences. An argument in the review can form an argument pair with the corresponding argument in the rebuttal that discusses the same topic.
The core of APE is to detect the arguments within each passage and construct the relations between interactive arguments in the two passages. Most existing works (Cheng et al., 2020, 2021; Bao et al., 2022) apply powerful encoders, such as table encoders and attention mechanisms, to learn the sentence-level semantic representation for implicitly modeling relationships between argument pairs. However, as revealed in (Cheng et al., 2021), the sentence representations learned by pure attentionbased methods are difficult to effectively capture
13075
the complicated relations between the sentences from different passages. Bao et al. (2021) explicitly established argument links based on co-occurring words within the sentence pairs and verified the importance of word-level relations among arguments. Nevertheless, they ignore the fact that the sentence pairs within the argument pairs generally contain semantically similar words, such as “space” and “interval” in the sentence pair, as illustrated in Figure 2. Although the semantic-aware relational information is already contained in the continuous representations of PLMs, neural networks lack an optimal mechanism to benefit from such information.
To address the aforementioned issues, we propose a novel ProbIng Graph dEcompositiON (PIGEON) framework for argument pair extraction by exploiting explicit semantic knowledge induced from large-scale PLMs. Specifically, we employ a two-stage masked language modeling process to construct sentence-level relation graphs between sentences (global linguistic properties) based on the number of highly similar word pairs within each sentence pair. The key idea behind this probing method is that we can obtain similar sentence pair representations when we mask out one word from a word pair with high similarity. The sentence-level relation graph is essential to effectively identify argument pairs.
Since the review and rebuttal passages have different writing styles and word distributions, the learned sentence-level probing graph may underexplore the interactive relations between the two passages. In particular, the relevance score of each sentence pair within a passage is generally larger than sentence pairs from different passages. For example, as shown in Figure 1, the tenth review sentence contains more semantically similar words with the eleventh review sentence than the second rebuttal sentence1. Consequently, the review sentences would prefer to propagate information within the same passage and under-explore the interactive relation between two passages. To effectively capture argument relations of different passages, we decompose the sentence-level probing graph into four sub-graphs from intra- and interpassage perspectives. The intra-passage graph can help detect the argument spans within each passage, while the inter-passage graph is used to identify the argument pairs from different passages. To fur-
1Further analysis is presented in Appendix A.1
ther improve the performance of our method, we also design an auxiliary graph contrastive loss to weaken the impact of noisy edges brought by the probing procedure.
Our contributions can be summarized as follows. (1) We propose a probing technique to elicit semantic-aware relational knowledge from largescale PLMs for constructing sentence-level probing graph. (2) We decompose the sentence-level probing graph into four sub-graphs from intra- and inter-passage perspectives so as to effectively detect argument spans within each passage via intrapassage graphs and identify argument pairs from two passages via inter-passage graphs. (3) We conduct experiments on two APE benchmark datasets, and the results show that our method outperforms the strong baselines by a noticeable margin.

heading: 2 Methodology
text: Following previous works (Cheng et al., 2020; Bao et al., 2021), we aim to automatically extract interactive argument pairs from the review and rebuttal passages by casting the argument mining and argument pairs extraction as two sentencelevel sequence labeling problems using the standard BIOES scheme (Ratinov and Roth, 2009). Formally, given a review passage Sv = {sv1, . . . , svm} consisting of m sentences and a rebuttal passage Su = {su1 , . . . , sun} consisting of n sentences, we first identify argument spans within the review and rebuttal passages, and obtain a review argument spans set Xv = {xv1, xv2, . . .} and a rebuttal argument spans set Xu = {xu1 , xu2 , . . .}, where xvi and xuj are sentence-level spans in review and rebuttal passages, respectively. Then, we extract the paired arguments from review and rebuttal passages, and a set of interactive argument pairs P = {p1, p2, . . .} can be collected, where p = (xvi , x u j ) is an interactive argument pair. As illustrated in Figure 2, PIGEON contains four components, including a sentence representation learning module, a probing graph construction module, a graph decomposition module, and an argument pair prediction module. Next, we will describe each component of PIGEON in detail.

heading: 2.1 Sentence Representation Learning
text: We first apply BERT (Devlin et al., 2018) to obtain the word-level context representation of each sentence within the review and rebuttal passages. Then, we use a shared bidirectional LSTM (BiL-
STM) (Hochreiter and Schmidhuber, 1997) to encode the sentence-level dependencies within each passage. Specifically, we feed each sentence si into BERT and obtain the sentence embedding ei ∈ Rdb by mean pooling over all token representations, where db is the vector dimension of the last layer of BERT. A BiLSTM is then utilized to encode the sentence representations {e1, e2, . . .} within each passage into the contextual sentence representations {h1,h2, . . .}, where hi ∈ Rd and d is the hidden size of BiLSTM. We denote the contextual representations of review and rebuttal as Hv = {hv1,hv2, . . . ,hvm} and Hu = {hu1 ,hu2 , . . . ,hun}, respectively.

heading: 2.2 Probing Graph Construction
text: To effectively capture the relation between each sentence pair, we need to detect the semantically similar words from the sentence pair. So far, there are many possible ways to derive semantically similar words, such as WordNet (Miller, 1995), Word2Vec (Mikolov et al., 2013), and GloVe (Pennington et al., 2014). However, these methods generally focus on context-free word similarity and ignore the context, failing to deal with words with multiple meanings in different contextual scenarios. In this paper, we elicit explicit semantic knowledge from large-scale pre-trained language models (PLMs) and build a relation graph between each sentence pair through a probing procedure. It is worth noting that such semantic knowledge is extracted in an unsupervised and off-the-line manner.
Formally, given a sentence pair (si, sj), we first concatenate (si, sj) and obtain a single sequence s = {[CLS], si,[SEP], sj ,[SEP]},
where [CLS] and [SEP] represent the classification and separation tokens respectively. Then, we propose a probing approach with a masking technique to learn the semantic similarity between arbitrary word pairs from each sentence pair. We employ a two-stage masked language modeling (MLM) process to measure the impact a context word has on predicting another word. After the probing process, we can construct sentence-level relation graphs based on the number of highly similar word pairs with each sentence pair. The key idea behind the probing process is that we can obtain similar sentence pairs representations when we mask out one word from a word pair with high similarity. Concretely, we replace the k-th token wik in sentence si with a special mask token [mask] and feed the obtained new sequence s/wik into BERT. We can obtain the contextualized representation h/wik of the k-th token. To calculate the correlation between wik and the t-th word w j t in sentence sj , we further mask out w j t from s/wik to obtain the second corrupted sequence s /wikw j t
and feed it into BERT. We use h
/wikw j t
to denote the new rep-
resentation of word wik when both w i k and w j t are masked out simultaneously. After that, we measure the distance f(wik, w j t ) between h/wik and h/wikw j t to induce the semantically correlation between the k-th word wik of si and the t-th word wjt of sj . Here, we use the Euclidean distance metric to implement the distance function f(·) due to its simplicity and effectiveness as follows:
f(wik, w j t ) = ||h/wi k − h /wi k w j t ||2 (1)
Note that one word may be split into multiple tokens, we mask all tokens for each split-up word and apply a mean pooling over the token representations to obtain the word representation.
Word-level Similarity Matrix By repeating the two-stage MLM process on each pair of words (wik, wjt ) of the sentence pair (si, sj), we can obtain a word-level similarity graph M = {Mk,t}|s1|,|s2|k=1,t=1 for the sentence pair (si, sj), where Mk,t = f(wik, w j t ) denotes the relation between word pair (wik, w j t ). Then, we utilize the min-max normalization to reduce the impact of the range of correlation scores. The normalized word-level similarity matrix M̂k,t is computed by:
M̂k,t = Mk,t −min max−min (2)
where max and min is the maximum and minimum similarity scores of all word pairs in the review and rebuttal passages.
Sentence-level Probing Graph We construct the sentence-level probing graph in which nodes are sentences. The sentence-level relation matrix can be derived from the word-level relation matrices of all sentence pairs using the probing procedure. Specifically, we compute the relevance between each sentence pair (si, sj) based on the word pairs with high semantic similarity. Formally, we compute the relation between the sentence pair (si, sj) as follows:
Ri,j =
|si|∑
k=1
|sj |∑
t=1
I(M̂k,t > σ) (3)
where σ is a pre-defined threshold. I is an indicator function. By traversing all sentence pairs, we can obtain the symmetrical sentence-level relation matrix R = {Ri,j}|m+n|,|m+n|i=1,j=1 for the review and rebuttal passages. Intuitively, if two sentences have many semantically similar word pairs, the corresponding edge will have a large weight.

heading: 2.3 Graph Decomposition
text: The review and rebuttal passages have different writing styles and word distributions. The relevance score of each sentence pair within a passage is generally larger than the sentence pair from different passages. To effectively capture argument relations of different passages, we decompose the sentence-level probing graph into four sub-graphs from intra- and inter-passage perspectives. The
intra-passage graph can help detect the argument spans within the review (or rebuttal) passage, while the inter-passage graph is used to identify the argument pairs from the review and rebuttal passages. Formally, we decompose the sentence-level relation matrix R ∈ R(m+n)×(m+n) of two passages (Sv, Su) into four sub-matrices Rvv ∈ Rm×m, Ruu ∈ Rn×n, Rvu ∈ Rm×n and Ruv ∈ Rn×m, as illustrated in the left part of Figure 2. Among these, Rvv and Ruu represent the intra-passage relation matrices of the review and rebuttal passages, respectively. Rvu and Ruv denote the inter-passage relation matrices.
Intra-passage Graph Construction The intrapassage graph of each passage takes the sentences within the passage as vertices. The embeddings of the vertices are initialized with the corresponding sentence representations. Then, we refine the edges and the corresponding weights by the relative positions between sentences and intra-passage relevance matrices. Specifically, given a review passage Sv (or a rebuttal passage Su), the edge weight Avvi,j for each sentence pair (si, sj) can be computed by:
Avvi,j =
{ Rvvi,j
max(Rvvi ) if |i− j| ≤ τ
0 otherwise (4)
where τ is a pre-defined threshold. max(Rvvi ) represents the maximum value of the i-th row of intrapassage relation matrix Rvv.
Inter-passage Graph Construction The interpassage graph for each passage is a bipartite graph, where each edge only exists between two sentences from different passages. The inter-passage adjacency matrix Avu ∈ Rm×n of passage Sv can be computed by:
Avui,j = Rvui,j
max(Rvu) (5)
where max(Rvv) is the maximum value of the sentence-level relation matrix Rvv. Consequently, each passage can derive two different graphs (i.e. intra-passage and inter-passage graphs). Note that the intra- and inter-passage graphs of each passage are mutually independent though they are derived from the same passage.
Relation-aware Sentence Representations We use graph convolutional networks (GCNs) to learn the representations of intra- and inter-passage relation graphs. Each node in the graphs is updated
according to the hidden representations of its neighborhoods based on the adjacency matrices of intrapassage and inter-passage graphs. Given the intraand inter-passage graphs of passage Sv, the update of the l-th GCN block is defined as follows:
Gvv = ReLU(ÂvvZv,l−1W l1 + b l 1) Gvu = ReLU(ÂvuZu,l−1W l2 + b l 2)
Zv,l = Gvv +Gvu + Zv,l−1 (6)
where Âvv (or Âvu) is the normalized adjacency matrix learned from Avv (or Avu), and we have Âvvi = Avvi∑ j A vv i,j+1
. Zv,l−1 represents the sentence representations of the review passage evolved in the l-1-th GCN block. Here, the node representations of the first GCN layer are defined as Zv,0 = Hv. For simplicity, we denote the final output of the GCN blocks as Zv. In this way, we can obtain the updated representation Zvi for the i-th sentence of passage Sv by integrating the representations of its neighbouring nodes within intra- and inter-passage graphs. Similarly, we can compute the updated sentence representations Zu of passage Su.

heading: 2.4 Argument Pair Prediction
text: After decomposing the probing graphs, the updated sentence representations are used for argument mining and argument pair extraction following previous works (Cheng et al., 2020; Bao et al., 2021).
Argument Mining We adopt a BiLSTM sequence tagger followed by a CRF sequence tagger to identify all potential arguments. Concretely, we feed the sentence representations Zv and Zu into the BiLSTM tagger to learn output hidden states Ov = {ov1, . . . ,ovm} and Ou = {ou1 , . . . ,oun}. Then, Ov and Ou are put into the CRF tagger to predict the argument labels Ŷ v = {ŷv1 , . . . , ŷvm} and Ŷ u = {ŷu1 , . . . , ŷun} for review and rebuttal respectively, where yi is the BIOES label for the i-th sentence. Based on these two label sequences Ŷ v and Ŷ u, we can further parse the potential argument spans for the review and rebuttal passages, i.e. X̂v = {x̂v1, x̂v2, . . .} and X̂u = {x̂u1 , x̂u2 , . . .}, where x̂i is the i-th predicted argument span. The sequence labeling loss LAM for each instance is defined as follows:
LAM = −[log p(Yv|ov) + log p(Yu|ou)] (7)
where Yv and Yv are the ground-truth sequence labels of review and rebuttal.
Argument Pair Extraction With the learned argument spans sets (Xv and Xb) and the argumentspecific sentence representations (Ẑv = Zv +Ov and Ẑu = Zu + Ou), we can extract argument pairs with dual sequence taggers. Specifically, we first produce the representation avk =
1 ek−bk+1 ∑ek i=bk
Ẑvi of the k-th extracted argument span xvk = (bk, ek) by mean pooling over the corresponding argument-specific sentence representations. Then, we concatenate avk to the argumentspecific sentence representation Ẑu of rebuttal Su to obtain the argument xvk-specific sentence representations {[avk, Ẑu1 ], . . . , [avk, Ẑun ]}, where [·, ·] is the concatenation operation. We feed the learned representations into a BiLSTM tagger and a CRF tagger to predict the argument label sequences Yuk with its paired arguments from the rebuttal passage. Similarly, we can perform the same procedure to capture its paired arguments from review for the k-th rebuttal argument by predicting the label sequences Yvk with another BiLSTM and CRF tagger. For APE, its sequence labeling loss LAPE in each instance is defined as: LAPE = − ∑
k
log p(Yuk |avk, Ẑu)− ∑
k
log p(Yvk |auk , Ẑv)
(8)
Graph Contrastive Loss We introduce an auxiliary graph contrastive loss to weaken the impact of the introduced noisy edges brought by the probing procedure. Taking the intra- and inter-passage graphs of review Sv as an example, we follow an i.i.d. uniform distribution to randomly drop the noisy edges (non-argument pairs) in the graph with probability μ and generate auxiliary graph views with adjacency matrices Ãvv = drop(Âvv) and Ãvu = drop(Âvu) from the original graph. Noted that the removal probabilities of the edges for the argument pairs are zero. Then, we feed the learn auxiliary graph views into GCNs and produce the auxiliary updated node representations Z̃v and Z̃u for the review Sv and rebuttal Su respectively. After that, we employ a contrastive objective LGCL to distinguish the representations of different views of the same node from the representations of different nodes:
LGCL = − ∑
i
log exp(ψ(Zi, Z̃i))
exp( ∑ j =i ψ(Zi, Zj) + ∑
j ψ(Zi, Z̃j)) (9)
where Z = [Zv, Zu] represent the updated node representation matrices of passages Sv and Su. ψ(·, ·) denotes the cosine similarity and g(·) is a
two-layer perceptron.
Joint Training Objective We minimize the joint loss function Ljoint by summing up the three training objectives as:
Ljoint = LAM + LAPE + λLGCL (10)
where λ is a tuned hyper-parameter controlling the impact of LGCL.

heading: 3 Experimental Setup
text: Datasets We conduct experiments on the ReviewRebuttal (RR) dataset, which is a benchmark dataset proposed by (Cheng et al., 2020). The RR dataset includes 4,764 review-rebuttal pairs collected from ICLR 2013 to ICLR 2020. There are two versions provided: RR-Passage-v1 (RR-P) and RR-Submission-v2 (RR-S). Both RR-P and RR-S are split by the ratio of 8:1:1 into train, dev, and test sets. In the RR-P dataset, different review-rebuttal passage pairs of the same paper submission could be put into different sets, while in the RR-S dataset, multiple review-rebuttal passage pairs of the same submission are included in the same set. Since RRS is more challenging than RR-P, we conduct further experiments on RR-S. The detailed statistics of RR-P and RR-S are summarized in Appendix A.2.
Evaluation Metrics Following previous works (Cheng et al., 2020; Bao et al., 2021), we adopt precision (Prec.), recall (Rec.) and F1 scores to measure the performance of both argument mining (AM) and argument pair extraction (APE).
Baselines To evaluate the effectiveness of PIGEON, we compare it with several strong baselines. PL-H-LSTM-CRF (Cheng et al., 2020) learns separate sequence labeling and sentence relation classification models, and then combines two results together to predict the argument pairs. Similar to PL-H-LSTM-CRF, MT-H-LSTM-CRF (Cheng et al., 2020) trains two subtasks via a shared feature encoder in the multi-task learning manner. MLMC (Cheng et al., 2021) is an attentionguided model based on a table-filling approach. MGF (Bao et al., 2021) proposes a mutual guidance framework with an inter-sentence relation graph. MRC-APE (Bao et al., 2022) applies machine reading comprehension framework with a Longformer (Beltagy et al., 2020) as the encoder, which is the state-of-the-art method on RR.
Implementation Details PIGEON is implemented in PyTorch on an NVIDIA TITAN RTX GPU. We apply the uncased BERT base2 as our PLM. The AdamW optimizer (Loshchilov and Hutter, 2018) is employed for parameter optimization, and the initial learning rates for the BERT layer and other layers are set to 1e-5 and 1e-3, respectively. Similar to previous works (Cheng et al., 2021), we set the batch size as 1 due to the limited memory. The maximum number of the GCN blocks on RR-S and RR-P are set to 5 and 3, respectively. The hidden size of BiLSTMs is set to 256. In addition, the parameters of BiLSTMs and CRFs used in the three taggers are not shared 3. All experiments are performed five times with different random seeds, and we report the averaged scores. Our code and data are available at https: //github.com/syiswell/PIGEON.

heading: 4 Experimental Results

heading: 4.1 Overall Performance
text: We report the overall performance of our proposed framework and baseline methods in Table 1. Our method achieves the best performance on both RRS and RR-P. On RR-S, our method outperforms the current state-of-the-art method (i.e., MRC-APE) by 2.94% in terms of F1 score on the APE subtask. On RR-P, our model also exceeds MRC-APE and obtains about 3.05% higher F1 score on the APE subtask. The experimental results verify the superiority of our method on the APE subtask. In addition, our PIGEON is also more efficient than baselines, as shown in Appendix A.4.
We also observe that the pipeline method (i.e., PLH-LSTM-CRF) perform more poorly than the other end-to-end baselines because it may lead to error propagation. The attention-based method (i.e., MLMC) achieves significant improvement over MT-H-LSTM-CRF, since MLMC can implicitly model the argument correlation. The graph-based method (i.e., MGF) surpasses MLMC but underperforms MRC-APE. This may be because MGF only considers the word overlap and explicitly constructs the incomplete argument correlation without considering semantic information. Our PIGEON performs better than all baselines by probing semantic knowledge from PLMs.
2We implement BERT using huggingface toolkit: https://huggingface.co/
3More details about hyperparameter settings can be found in Appendix A.3.

heading: 4.2 Ablation Study
text: To analyze the impact of different components in our proposed PIGEON method, we conduct ablation test in terms of removing probing graph (w/o PG), removing graph decomposition (w/o GD), removing intra-passage graphs (w/o Intra-PG), removing inter-passage graphs (w/o Inter-PG), and removing graph contrastive learning (w/o GCL), respectively.The results are reported in Table 2. We can observe that both w/o Inter-PG and w/o InterPG degrade the performance substantially, verifying that the sentence relations within both the intrapassage and inter-passage graphs are important for the APE subtask. Removing graph decomposition (w/o GD) leads to performance drops significantly, demonstrating that probing graph decomposition can help alleviate the problem that the review and rebuttal passages have different styles and word distributions. It is no surprise that graph contrastive learning also contribute to the effectiveness of our method by reducing the impact of noisy edges in probing graphs.

heading: 4.3 Graph Parameters Analysis
text: We analyze the hyperparameters used in the processes of probing graph construction and decom-
position, i.e., the threshold σ for determining word pairs with high semantic similarity, the distance threshold τ for constructing the intra-passage graph, and the number l of GCN blocks. The results on RR-S are illustrated in Figure 3. (i) From Figure 3a, we observe that the best results can be obtained when σ = 0.5. The small σ tend to introduce infrequent word pairs and lead to noisy argument relations. (ii) From Figure 3b, our method achieves the best performance when τ equals 3. A too large distance threshold may introduce incorrect edges in the graph, while a too small threshold will discard the correct relations between sentences. (iii) We vary the block number l in GCNs from 1 to 10, and illustrate the results in Figure 3c. We can achieve the best results when l = 5. While the performance tends to decrease as the number of GCN blocks increases when the block number is greater than 5. This implies that roughly increasing the number of GCN blocks is vulnerable to slash the learning ability of the model owing to the sharp increase of the model parameters.

heading: 4.4 Different Word Similarity Measures
text: We compare our probing method with several popular word similarity measures (e.g., Word2Vec,
GloVe and WordNet) for detecting word pairs with high semantic similarity. The co-occurrence based method focuses on string matching. The Word2Vec and GloVe methods obtain the word pair similarity by computing the cosine similarity of their word vectors. The WordNet method computes the word pair similarity based on the shortest path connecting the two word senses in a taxonomy. After learning the similarity of arbitrary word pairs, we construct the inter- and intra-passage graphs similar to our PIGEON method. We report the AM and APE results in Table 3. Our probing method performs significantly better than the compared word similarity measures by eliciting context-aware semantic knowledge from large-scale PLMs.

heading: 4.5 Case Study
text: We provide an exemplary case that is selected from RR-S test set by visualizing the adjacent matrix of the inter-passage graphs, where the distribution of edge weights is similar to the distribution of ground-truth argument pair labels. As shown in Figure 4b, the probing method can catch the seman-
tic similarity between the word pair “observation” and “visible” with the help of elicited knowledge from PLMs. We believe that our PIGEON can probe rich semantic knowledge from PLMs, helping detect argument relations for APE.

heading: 5 Related Work
text: Argument Pair Extraction Most existing argument mining methods focus on modeling the arguments in monologues, such as argumentation structure parsing (Stab and Gurevych, 2014; Morio et al., 2020), argument quality assessment (Lauscher et al., 2020), and argumentation strategies modeling (Al Khatib et al., 2017). However, in real-life scenarios, arguments are often in the form of dialogues. Several prior studies detect agreement and disagreement in online debates and discussions (Morio and Fujita, 2018; Chakrabarty et al., 2019; Ji et al., 2021). Subsequently, Cheng et al. (2020) introduced a new argument pair extraction (APE) task in the domain of peer review and rebuttal, aiming to extract the argument pairs from the review and rebuttal passages simultaneously. Cheng et al. (2021) applied an attention mechanism and a table-filling approach to implicitly model the interaction between argument pairs. To explicitly model the relations between argument pairs, Bao et al. (2021) proposed a mutual guidance framework with an inter-sentence graph for APE. Bao et al. (2022) explored a bidirectional machine reading comprehension (MRC) to capture the interactions between argument pairs. Different from previous works, we explicitly capture the relations between argument pairs by eliciting context-aware semantic knowledge from PLM. In addition, we propose a graph decomposition method to deal with the issue that the review and rebuttal passages have different styles and word distribution.
Probing Knowledge from PLMs Recently, the success of PLMs has led to plenty of studies applying the probing technique to elicit rich knowledge from large-scale PLMs (Jawahar et al., 2019; Clark et al., 2019; Wu et al., 2020; Wang et al., 2022). A typical probing study is to investigate the knowledge and linguistic properties contained in PLMs, such as morphology (Belinkov et al., 2017), word sense (Reif et al., 2019), syntax (Hewitt and Manning, 2019; Dai et al., 2021). The key idea behind these works is to define a precise task, and then design a simple model (called a probe) to solve the task using the contextualized representations provided by PLMs. There are also some studies (Petroni et al., 2019; Zhong et al., 2021) that seek to answer to what extent PLMs store factual, relational and commonsense knowledge.

heading: 6 Conclusion
text: In this paper, we designed a probing technique to elicit semantic-aware relational knowledge from large-scale PLMs, which captured rich explicit interactive relations between argument pairs. In addition, we proposed a graph decomposition method to decompose the probing graph into four sub-graphs from intra- and inter-passage perspectives, which could alleviate the issue that different participants might have different writing styles and word distributions for the APE task. Experimental results on two benchmark datasets showed that our method outperformed strong baselines significantly.
Limitation
To better understand the limitations of the proposed model, we carry out an analysis of the errors made by PIGEON. Specifically, we randomly select 100 instances that are incorrectly predicted by PIGEON and summarize the primary types of error. The first error category is boundary prediction error. Since we modeled the APE task as a sequence labeling, our model may only recognize a part of an argument. Thus, multiple consecutive arguments may be identified as a single argument. The second type of error is caused by the absence of semantically similar words in the argument pairs. In this case, the proposed probing graphs cannot model the relations between argument pairs. Third, another error category occurs when semantically similar words are also presented in non-matching argument pairs. The argument relation may be misled by these words. It suggests that certain relation
modeling method needs to be devised in the future so as to better infer argument relation. For example, we may leverage the high-level topic information over argument pairs to guide the learning of relation-specific features.
In addition, the proposed probing approach may be computationally expensive and we can alleviate this problem by saving the similarity of all word pairs for one time for the entire dataset. We will address this issue in future work.

heading: Acknowledgements
text: We thank the anonymous reviewers for their valuable suggestions to improve the quality of this work. This work was partially supported by the National Natural Science Foundation of China (62006062, 62176076), Natural Science Foundation of GuangDong 2023A1515012922 the Shenzhen Foundational Research Funding (JCYJ20220818102415032), the Major Key Project of PCL2021A06, Guangdong Provincial Key Labo-ratory of Novel Security Intelligence Technologies 2022B1212010005. Min Yang was partially supported by the National Key Research and Development Program of China (2022YFF0902100), Shenzhen Science and Technology Innovation Program (KQTD20190929172835662), Shenzhen Basic Research Foundation (JCYJ20210324115614039 and JCYJ20200109113441941).

heading: A Appendix
text: A.1 Sentence Relevance Analysis
In Figure 5, we randomly select 100 samples from the RR-S test set and show the distributions of the number of similar words and similarities of sentences from intra- and inter-passages. The number of similar words between sentences is computed by our probing procedure, while the sentence similarities are measured by the cosine similarity of sentence representations obtained from BERT. We can observe that the number of similar words in the sentences within a passage is larger than that in the sentences from different passages, as shown in Figure 5a. In addition, the sentence representations in the intra-passage, on average, have higher similarity than that in the inter-passage, as illustrated in Figure 5b. This may be because the review and rebuttal passages have different writing styles and word distributions.
A.2 Data Statistics
The detailed statistics of the RR-S and RR-P datasets are summarized in Table 4. Both RR-S and RR-P dataset contain 4,764 review-rebuttal pairs collected from ICLR 2013 to ICLR 2020, which are split by the ratio of 8:1:1 into train, dev, and test sets. In the RR-P dataset, different review-rebuttal passage pairs of the same paper submission could be put into different sets, while in the RR-S dataset,
multiple review-rebuttal passage pairs of the same submission are included in the same set. Thus, RR-S is more challenging than RR-P.
A.3 Hyperparameter Settings We manually tune the hyperparameter values (e.g., the weight λ for graph contrastive loss and the drop probability μ of noisy edges) on the RR-S. We report the results in Table 5 and Table 6. The weight λ of graph contrastive loss is tuned from 0.001 to 1 with a ratio of 10. The drop probability μ is tuned from 0.1 to 0.5 with a step size of 0.1. From the results in Table 5 and Table 6, we set the value of λ to 0.01 and the value of μ to 0.1.
A.4 Computational Cost Table 7 shows the training time, the testing time, the number of parameters, and the APE results of our model on the RR-S development set. As the number of GCN blocks increases, the performance on the development set improves yet the performance on the testing set becomes worse. It implies our model might suffer from the overfitting issue with the increasing layers of GCN blocks. In addition, our PIGEON is more efficient than baselines during inference owing to the fewer model parameters. Note that MRC-APE with fewer parameters
requires the most running and testing time because it needs to repeatedly implement multi-turn machine reading comprehension if there are several arguments within an instance.
A.5 Different Graph Augmentation Methods
We also explore an additional random dropping method for graph augmentation. The results are provided in Table 8. The random dropping method randomly removes a certain percentage of edges, without considering whether the edge is a noisy edge or not. We find that the random dropping method performs worse than the noisy edge drop-
ping method for graph contrastive learning. This may be because the random dropping method removes the important edges and corrupts the true argument relations. By contrast, the noisy edge dropping method only focuses on the noisy edges within non-argument pairs and achieves better performance.
A.6 Different Sentence Similarity Measures We also explore two sentence similarity measures i.e., SBERT and SROBERTA (Reimers and Gurevych, 2019). The results of SBERT, SROBERTA and our method on RR-S are presented in Table 9. We can observe that our method significantly outperforms both SBERT and SROBERT. The reason may be that sentences with a high degree of similarity are not certainly in the same argument pair, but may also be expressing similar opinions on different topics and then incorrectly establishing argument relations.
ACL 2023 Responsible NLP Checklist

heading: A For every submission:
text: 3 A1. Did you describe the limitations of your work?
Section Limitation
7 A2. Did you discuss any potential risks of your work? Our work does not have potential risks.
3 A3. Do the abstract and introduction summarize the paper’s main claims? Section Abstract and Introduction
7 A4. Have you used AI writing assistants when working on this paper? Left blank.
B 7 Did you use or create scientific artifacts? Left blank.
B1. Did you cite the creators of artifacts you used? No response.
B2. Did you discuss the license or terms for use and / or distribution of any artifacts? No response.
B3. Did you discuss if your use of existing artifact(s) was consistent with their intended use, provided that it was specified? For the artifacts you create, do you specify intended use and whether that is compatible with the original access conditions (in particular, derivatives of data accessed for research purposes should not be used outside of research contexts)? No response.
B4. Did you discuss the steps taken to check whether the data that was collected / used contains any information that names or uniquely identifies individual people or offensive content, and the steps taken to protect / anonymize it? No response.
B5. Did you provide documentation of the artifacts, e.g., coverage of domains, languages, and linguistic phenomena, demographic groups represented, etc.? No response.
B6. Did you report relevant statistics like the number of examples, details of train / test / dev splits, etc. for the data that you used / created? Even for commonly-used benchmark datasets, include the number of examples in train / validation / test splits, as these provide necessary context for a reader to understand experimental results. For example, small differences in accuracy on large test sets may be significant, while on small test sets they may not be. No response.
C 3 Did you run computational experiments? section 4
3 C1. Did you report the number of parameters in the models used, the total computational budget (e.g., GPU hours), and computing infrastructure used? section Appendix
The Responsible NLP Checklist used at ACL 2023 is adopted from NAACL 2022, with the addition of a question on AI writing assistance.
3 C2. Did you discuss the experimental setup, including hyperparameter search and best-found hyperparameter values? section 3 and section Appendix
3 C3. Did you report descriptive statistics about your results (e.g., error bars around results, summary statistics from sets of experiments), and is it transparent whether you are reporting the max, mean, etc. or just a single run? section 3
3 C4. If you used existing packages (e.g., for preprocessing, for normalization, or for evaluation), did you report the implementation, model, and parameter settings used (e.g., NLTK, Spacy, ROUGE, etc.)? section 3
D 7 Did you use human annotators (e.g., crowdworkers) or research with human participants? Left blank.
D1. Did you report the full text of instructions given to participants, including e.g., screenshots, disclaimers of any risks to participants or annotators, etc.? No response.
D2. Did you report information about how you recruited (e.g., crowdsourcing platform, students) and paid participants, and discuss if such payment is adequate given the participants’ demographic (e.g., country of residence)? No response.
D3. Did you discuss whether and how consent was obtained from people whose data you’re using/curating? For example, if you collected data via crowdsourcing, did your instructions to crowdworkers explain how the data would be used? No response.
D4. Was the data collection protocol approved (or determined exempt) by an ethics review board? No response.
D5. Did you report the basic demographic and geographic characteristics of the annotator population that is the source of the data? No response.

heading: Future Work
text: It suggests that certain relation
modeling method needs to be devised in the future so as to better infer argument relation. For example, we may leverage the high-level topic information over argument pairs to guide the learning of relation-specific features. In addition, the proposed probing approach may be computationally expensive and we can alleviate this problem by saving the similarity of all word pairs for one time for the entire dataset. We will address this issue in future work