Hi-ArG: Exploring the Integration of Hierarchical Argumentation Graphs in Language Pretraining
Authors: Jingcong Liang
Rong Ye
Meng Han
Qi Zhang
Ruofei Lai
Xinyu Zhang
Zhao Cao
Xuanjing Huang
zhongyu wei
Venue: EMNLP 2023 Main (2023)
Keywords: computational argumentation
knowledge graph
abstract meaning representation

Abstract: The knowledge graph is a structure to store and represent knowledge, and recent studies have discussed its capability to assist language models for various applications. Some variations of knowledge graphs aim to record arguments and their relations for computational argumentation tasks. However, many must simplify semantic types to fit specific schemas, thus losing flexibility and expression ability. In this paper, we propose the **Hi**erarchical **Ar**gumentation **G**raph (Hi-ArG), a new structure to organize arguments. We also introduce two approaches to exploit Hi-ArG, including a text-graph multi-modal model GreaseArG and a new pre-training framework augmented with graph information. Experiments on two argumentation tasks have shown that after further pre-training and fine-tuning, GreaseArG supersedes same-scale language models on these tasks, while incorporating graph information during further pre-training can also improve the performance of vanilla language models. Code for this paper is available at <https://github.com/ljcleo/Hi-ArG>.

Hi-ArG: Exploring the Integration of Hierarchical Argumentation Graphs in Language Pretraining
Jingcong Liang', Rong Ye Meng Han', Qi Zhang', Ruofei Lai?, Xinyu Zhang', Zhao Cao', Xuanjing Huang' Zhongyu Wei
Fudan University, HHuawei Poisson Lab, *ByteDance jcl jcliang22@m fudan. edu cn, yewongeeyyeeanceee...o com,
edu. cn, tha caozhao Jehuawei com
{gz,
Abstract
We shoudd
Guns are killing people.
ban guns.
The knowledge graph is a structure to store represent knowledge, and recent studies have discussed its capability assist language models for various applications. Some vari- ations of knowledge graphs aim record ar- guments and their relations for computational argumentation tasks. However, many must sim- plify semantic types fit specific schemas, thus losing flexibility and expression ability. paper, we propose the Hierarchical Argumentation Graph (Hi-ArG), ture to organize arguments. We also introduce two approaches to exploit Hi-ArG, including a text-graph multi-modal model GreaseArG and a new pre-training framework augmented with graph information. Experiments two argumentation tasks have shown that after fur- ther pre pre-training and fine-tuning, GreaseArG supersedes same-scale language models these tasks, while incorporating graph infor- mation during further pre-training can also improve the performance of vanilla language models. Code for this paper is available at https://githuu: thwbb.tttthu.... com/ http//jjleoHHiAAA
Support
TOP0
TOP1
per- son
rec.
G0
G1
Attack
ban
kill
G2
G3
pre-
recc.
vent
struc-
TOP2
TOP3
Support
shouldn't '
Guns can prevent killing everyonne has one.
guns.
Figure 1: An example of Hi-ArG extracted from the con- versation between Alice and Bob. TOPO to TOP3 rep- resent top nodes corresponding different arguments. GO to G3 represent intra- aarra-argument sub-graphs whose detailed structures are omitted. rec. and -rec. repre- sent recommend and not recommend respectively.
between arguments and semantic relations between mentioned entities. To explicitly provide these re- lation links, Al-Khatib et al. (2020) borrowed the form of knowledge graphs and introduced the Argu- mentation Knowledge Graph (AKG) to represent entity relations extracted from arguments. They also applied AKG to fine-tune language model for argument generation tasks (Al Khatib et al.. 2021).
1 Introduction
Debating a fundamental formal process to find solutions and gain consensus among groups of peo- ple with various opinions. It is widely accepted in politics (Park al., 2015; Lippi and Torroni, 2016),
education (Stab and Gurevych, 2014, 2017) and online discussions (Habernal and Gurevych, 2017). As more and more formal and informal debates are launched and recorded on Internet. organiz- ing arguments within them has become crucially valuable for automated debate preparation and even participation (Slonim et al., 2021).
Despite the effectiveness AKG, we argue that semantics within arguments can be better mod- eled. In this paper, we propose the Hierarchical Argumentation Graph (Hi-ArG), graph structure to organize arguments. This new structure tain more semantics within arguments the lower (intra-argument) level and record relations between arguments the upper (inter-argument) level. We claim that Hi-ArG provides more explicit informa- tion from arguments and debates, which benefits
Recently, language models have shown domi- nating advantages in various argumentation tasks (Gretz et al., 2020; Rodrigues and Branco, 2022). However, such tasks often rely on logical relations
*Corresponding author.

---

language models argumentation tasks. For instance, consider the following conversa- tion about whether guns should be banned:
2017; Eger al., 2017), stance detection (Augen- stein et al., 2016; Kobbe al., 2020), argumenta- tion quality assessment (Wachsmuth et al., 2017: El Baff et al., 2020), argument generation (Wang and Ling, 2016; Schiller et al., 2021; Alshomary et al., 2022) and automated debating (Slonim et al., 2021). A majority of these tasks are boosted with language models recently (Gretz et al., 2020; Fried- man et al., 2021; Rodrigues and Branco, 2022), and studies have discussed various ways inject external argumentation structure knowledge such models (Bao et al., 2021; Dutta et al., 2022).
Alice: We should ban guns. Guns are killing people.
Bob: No, guns can prevent killing if everyone has one, so we shouldn't ban them.
Alice and Bob have opposite opinions, yet their ev- idence focuses on the topic's ' same aspect (killing). In Hi-ArG (see Figure 1), all their claims and premises can live together and connect through inter-arg (supporting/attcciing)) and intra-arg (se- mantic relations between concepts like guns and killing) links.
Meanwhile, several works have discussed proper methods to retrieve arguments from data sources (Levy al., 2018; Stab al., 2018; Ajjour et al., 2019). yet majority of them keep the text form unchanged, resulting in loose connections between arguments. One of the methods to manage textual information is to represent it as knowledge graphs; however, popular knowledge graphs mainly record concept relations (Auer et al., 2007; Bollacker et al., 2008; Speer et al., 2017). To cope with this issue, Heindorf et al. (2020) proposed a knowledge graph that stores causal connections between concepts. AI-Khatib et al. (2020) further developed this idea and introduced the argumentation knowledge graph (AKG), which can be seen as the first approach to organize arguments structurally, and has been employed in a few downstream tasks (Al Khatib et al., 2021).
To validate the power of Hi-ArG, we propose two approaches to exploit information from the graph. First, introduce GreaseArG, text-graph multi-modal model based on GreaseLM (Zhang et al., 2022) to process texts with sub- sub-graphs from Hi-ArG simultaneously. Second, we design new pre-training method assisted by Hi-ArG, which is available for both GreaseArG and vanilla lan- guage models. To examine the effectiveness of these approaches, we conduct experiments on two downstream argumentation tasks namely Key Point Matching (KPM) and Claim Extraction with Stance Classification (CESC).
In general, our contributions are:
A flaw of the AKG that relations between con- cepts are highly simplified, losing rich semantic in- formation. The representation semantic relations has been long studied as the task of semantic role la- beling (SRL, Palmer et al. 2005; Bonial et al. 2014; Li et al. 2018; Shi and Lin 2019). While traditional SRL keeps the linear form of sentences, Banarescu et al. (2013) proposed abstract meaning represen- tation (AMR), graph-based structure express semantic relations. Being similar knowledge graphs, the idca becomes natural to record argu- ments AMR graphs and organize them, then uti- lize them for downstream tasks in similar way as typical knowledge graphs, especially accompanied with language models (Yang et al., 2021; Zhang et al., 2022; Amayuelas et al., 2022).
We propose Hi-ArG, new graph structure to organize various arguments, where semantic and logical relations are recorded at intra- and inter-argument levels.
We design two potential methods use the structural information from Hi-ArG, including multi-modal model and new pre-training framework.
We validate the above methods two down- stream tasks prove that Hi-ArG can enhance language models on argumentation scenarios with further analysis.
2 Related Work
The Hi-ArG Framework
Computational argumentation is an active field in NL.P focusing on argumentation-related tasks, such as argument mining (Stab and Gurevych, 2014; Persing and Ng, 2016; Habernal and Gurevych,
In this section, we introduce the detailed structure of Hi-ArG. We also propose a fully automated con- struction procedure applicable to different corpora.

---

3.1 Structure
adding edges from such information is straightfor- ward; hence, we will not cover the details here.
As described in Scction 1, Hi-ArG can be divided into the lower angra-ar level and the upper inter-argument level. In the remaining part of this paper, these two levels will be inferred intra-arg and inter-arg graphs, respectively.
The construction pipeline can be split into three stages: (1) extracting, (2) parsing, and (3) merging.
Extracting Stage In the extracting stage, docu- ments in the corpus are cut into sentences, and valid sentences are treated as arguments. The standard of valid arguments can be adjusted according specific task requirements.
Intra-arg Graph This level represents the se- mantic relations within a particular argument, mainly related to semantic analyses like Abstract Meaning Representation (AMR, Banarescu et al. 2013). Since AMR is highly abstract structure suitable for organizing semantic relations among concepts, it is introduced as the primary backbone the lower level of the argumentation graph. More specifically, nodes in the intra-arg graph represent single semantic units connected with directed edges following the AMR schema. Arguments are linked to node naming the top node, whose respective sub-graph fully represents the argument. More de- tails of AMR concepts and their relation intra-arg graph components can be found in Appendix A.
Parsing Stage As soon as arguments are ex- tracted from the corpus, they are parsed into sepa- rate AMR graphs in the parsing stage. AMR pars- ing models can be applied automate this stage. Besides parsing, additional operations can be per- formed during this stage, such as graph-text align- ing (Pourdamghani et al., 2014).
Merging Stage After obtaining AMR graphs for each argument, the final merging stage will com- bine all these separate graphs into one Hi-ArG. This can be realized by iteratively merging isomorphic nodes or, more specifically, eliminating nodes with common attributes and neighbors.
Inter-arg Graph The inter-arg level represents logical relations among arguments like supporting or attacking. Relation edges can connect arguments (top nodes) directly or through proxy nodes if mul- tiple arguments need to be packed together. Formal logic and traditional argumentation theories (like the ones in Toulmin, 2003 Budzynska and Reed, 201 1) can be applied to this level.
3.3 Adapting Exploitation Scenarios
A few issues exist when applying Hi-ArG to more specific models and tasks. In this section, in- troduce adaptations of the Hi-ArG structure to two common scenarios where the original one could cause problems. For instance, we apply these two adaptations to Grea one of our exploitation methods proposed in Section
The Hi-ArG structure has several advantages. First, note that different arguments can share one single top node if they have identical meanings. Such design allows efficient argument organization across paragraphs or documents. Second, if parts of sub-graphs of different top nodes are isomorphic, these parts can be merged, and thus the sub-graphs become connected. This way, arguments about common concepts may be grouped more densely, and debates around specific topics are likely to be associated.
Incorporating Text Two issues arise when pro- cessing text information and its corresponding Hi- ArG sub-graph. On the one hand, this sub-graph may not be connected; on the other hand, multi- ple sentences in one text can point to the same top node, which could confuse each other. To resolve these problems, an extra link node is added for each sentence, along with single edge connecting it to the corresponding top node, Furthermore, all link nodes are connected by another root node to ensure that the final graph connected.
3.2 Construction Pipeline
Based on the structure described above, We pro- pose brief construction pipeline for intra-arg graphs, illustrated in Figure 2. Once the intra-arg graph is created, the inter-arg graph can be easily attached to the corresponding top nodes. Construct- ing inter-arg graphs usually depends on external annotations or argumentation mining results, and
Message Passing A significant problem when processing Hi-ArG sub-graphs in message-passing models like graph neural networks (GNN) is that such graphs are directed, and directed edges cannot pass information in the reverse direction. To make message- passsig algorithms effective, each edge is accompanied by a reversed edge, whose attribute is

---

①
Extracting
Parsing
recommend
Merging
ban-01
Extractor & Filter
We should ban guns.
han guns.
AMR Parser
AARBB /
we
Y ARRG
gun
Iterative Merge
recomnend-01
kill-81
Guns kill people.
SARGR /ARST A
ARGA
Guns kill people.
kill-81
we
gum
person
gun
person
Figure 2: construction pipeline for Hi-ArG intra-arg graphs.
also a reverse of the original edge. This is possible for intra-arg graphs because AMR can change the direction of edges by adding to or removing from the edge label the passive mark -of, and for inter- arg graphs, similar approach can be used.
nected to the root nodes of each sub-graph, and an extra root node R connected to all link nodes, as shown in Figure 3; furthermore, reversed edges (labels not displayed; with embeddings 4 ci,...) are added the connected directed graph allow message passing GNN.
4 Exploiting Hi-ArG
Instead of using a specific prediction head as in GreaseLM, in GreaseArG, a cross- moosll atten- tion layer is appended after the output of the last LM/GNN layer, generating the final representation vectors of both text tokens and graph nodes. An- other critical difference between vanilla language models and GreaseArG is that GreaseArG needs to generate word embeddings for special tokens in the AMR graph like ARGo and ban-01. This is achieved by increasing the vocabulary and adding slots for them.
Knowledge graphs can participate in various stages of standard NLP pipelines, such as information retrieval and model training. In this section, we in- troduce two methods of exploiting Hi-ArG on the model side. First, present model structure sim- ilar to GreaseLM (Zhang et al., 2022), referred to GreaseArG, that simultaneously digests text and Hi-ArG data. We also propose further pre-training tasks specially designed for GreaseArG. Second, we develop a novel pre-training framework that modifies the method to construct training samples with another self- supervupe pre-training task.
In this paper, we adopt RoBERTa' (Liu et al., 2019) as the LM backbone of GreaseArG, and Graph Transformer (Dwivedi and Bresson, 2020) as the GNN backbone.
4.1 GreaseArG
Processing and understanding graph structures like argumentation graphs can challenge language mod- els since they can only handle linear informa- tion like text segments. Following GreaseLM, GreaseArG uses graph neural network (GNN) lay- add-ons to facilitate this issue.
4.2 Pre-training Tasks for GreaseArG
Before heading towards the second exploitation method, we introduce more details about the pre- training tasks for GreaseArG. Since GreaseArG handles information from two modalities, we ar- gue that pre-training this model should cover both. Here, we propose six tasks categorized into two classes: masking tasks and graph structure tasks. Each of the six tasks corresponds loss; these losses nal pre-training loss. Figure 4 illustrates the two categories of tasks with inputs from Figure
Figure 3 illustrates the structure of Are. Like GreaseL.M. GreaseArG concatenates GNN to transformer-bast language models per layer and inserts a modality interaction layer between layers of LM/GNN. An interaction token from the text side and an interaction node from the graph side exchange information each interaction layer.
specific task summed up to form the fi-
Hi-ArG adaptations mentioned in Section 3.3 are applied so GreaseArG can properly process Hi- ArG sub- -rbbggrahh. More specifically, for series of sub-gran aub-graphs, we create link nodes Si, S2.... con-
Masking Tasks Following Liu et al. (2019), we use Masked Language Modeling (MLM) as the
Unlless explicitly stated, RoBERTa will always refer to RoBERTa-basc.

---

Teat shoald Ean gits. Gana Kill pesple,
LM Blck
+DX
(MEaan
Inkesber Jakkn Iaabed, Luger
Tidee Eacheliigs
Takes Endedh
Takes Ented,
Telt
Denesdecan Spekkaiees
lapat
Wo dordeihnngges Cuecid peots
Enteddings
Last
Sesleer
Groph laput
N RAy
Layer
Modal Aiss Laye
Suner Juandon
Reprosttatio ful Gail
C Lacennmple
Nedea Eleae
C
GNN
Figure 3: GreaseArG model structure when processing text with its Hi-ArG sub-graph. {to, f1.... are token embeddings generated by the token embedding layer, whose transformed versions after least one LM layer are denoted by (fo. t. }. S₁ and S2 are link nodes added to connect subgraph roots recommend-01 and kill-81, while the extra root node added to connect link nodes S S2; labels of reversed edges are not displayed for clarity. (e1, 12, ug....) } node embeddings generated by the node embedding layer, whose transformed versions after at least one GNN layer are denoted by V fur, i2 ..,-...) }. E = {eoo e/s e1, c1.-.-) } are edge embeddings generated by the edge embedding layer (prime marks indicate reversed edges), which remain unchanged throughout the process. The embedded graph through one or more GNN layers is noted as G (V, Kt) in the joint block.
text pre-training task. For AMR-based intra-arg graphs, Masked Components Modeling (MCM, Xia et al. 2022) can treated as the graph ver- sion of MLM, where the model needs to predict attributes of masked components. In this paper, we consider masked node modeling (MNM) and masked edge modeling (MEM).
topic-related sentences from Hi-ArG called rela- tives. In an augmented sample, each relative is linked to a specific sentence in one of the sampled documents and is inserted after that particular docu- ment. This provides another approach to generating pre-training samples.
For any specific sentence, its relatives are sam- pled from related sentences found in Hi-ArG, weighted by the two-hop similarity in the sentence- node graph Gen- Gen is bipartite graph between link nodes of sentences and nodes from the orig- inal Hi-ArG sub-graph, where each link node is connected to nodes that form the semantic of its sentence; original edges that appeared in Hi-ArG are not included. The two-hop similarity between two sentences (link nodes) is the probability that a random walk starting from one link node ends at the other link node in two steps.
Graph Structure Tasks Graph structure tasks, focusing on learning structural information, consist Graph Contrastive Learning (GCL), Top Order Prediction (TOP), and Edge Direction Prediction (DIR). GCL requires the model to discriminate nodes with randomly permuted attributes, follow- ing the modified GCL version from Zheng et al. (2022). TOP aims to predict the relative order of consecutive link nodes from document. DIR requires the model to determine the original direc- tion of each edge in the bi-directed graph.
4.3 Augmenting Pre- rre-Training with Relatives
Most language models today are pre-trained with chunks of text data, where continuous sentences from one or more documents are sampled and con- catenated (Radford et al., 2018; Liu et al., 2019). When further pre-training GreaseArG, documents can refer to debates and other argumentation docu- ments in the training corpus. Thus, sentences are sampled according to their original order continu- ously.
Relative Stance Detection (RSD) Suppose the topic and stance of sampled documents and rela- tives are known beforehand. that case, we can classify each relative as supporting, attacking, or non-relevant concerming the document it relates to. Based on this observation, we propose a new pre-training task called Relative Stance Detection (RSD), In RSD, the model must predict the above stance relation between documents and their rela- tives. The final pre-training loss will include the corresponding loss of RSD, which is also computed by cross-entropy.
However, such a method does not exploit struc- tural knowledge Hi-ArG, perfect tool for find- ing attacking or supporting statement pairs. Hence, we can augment original training samples with

---

Masking Tasks
Text Input
Graph Structure Tasks
MLM
Mi
should
ban
<MASK>


<NAS
kill
people


GCL
<MASIC>
Graph Inputt
kill-31
Graph Input
we
NO
gaant
Guns
recommend-81
R
MNN
.ere
bene
recomennd-5
SMASK
kill-31
ARGYY
gun
DIR
MEM
ban 81
APG1
:ARGI
-ARG-
DNASO ABG2
<MASK>
per son
person
ban--61
Figure 4: Pre-training Tasks for GreaseArG. MLM: Predict masked tokens in input. MNM: Predict masked node attributes in graph input. MEM: Predict masked edge attributes in graph input. GCL: Predict whether node belongs to the original or permuted graph input. TOP: Predict the relative order of consecutive link nodes. DIR: Predict the original edge direction. Note that the permuted graph input is only used by GCL.
5 Experiments
5.2 Dataset
In this section, we introduce the dataset used for further pre-training and task-related datasets for fine-tuning and evaluation. We construct each dataset's corresponding Hi-ArG using the graph- construction pipeline mentioned in Section 3.2. Ap- pendix B describes the details of this procedure across datasets.
To examine approaches exploit Hi-ArG, apply them to selected argumentation tasks and compare the results with previous work. All mod- els are further pre-trained on argumentation corpus with generated Hi-ArG and then fine-tuned on spe- cific downstream tasks. We report the main results at the end of this section.
5.2.1 Further Pre-training
5.1 Downstream Tasks
Few works concern high-quality argumentation corpus sufficiently large for effective further pre- training. Among them, the args.me corpus (Ajjour et al., 2019) contains sufficient documents and ar- guments for further pre-training. Therefore, construct the Hi-ArG of the args.me corpus gen- erate further pre-training samples.
The performance of GreaseArG on argumentation tasks is evaluated on two downstream tasks - key point matching (KPM) and claim extraction with stance classification (CESC), covering argument pairing, extraction, and classification.
Because KPM and CESC are highly related inter-arg graphs, we only further pre-train models with intra-arg graphs to avoid bias towards these downstream tasks. Details about the construction of Hi-ArG for further pre-training can be found in Appendix C.
Key Point Matching This task (Bar-Haim et al., 2020a.b) aims to match long arguments with sets of shorter ones called key points. It is evaluated by the mean AP (mAP) of predicted scores between each candidate pair of arguments and key points. Some pairs are labeled as undecided if the relation is vague. Therefore, the mAP is further split into strict and relaxed mAP based on how to treat predictions on such pairs.
5.2.2 Fine-tuning and Evaluation
We choose a specific dataset for each down- stream task and construct its Hi-ArG, following the pipeline in Section 3.2. The training, dev, and test sets are processed separately, and we only construct intra-arg graphs like when building the further pre- training dataset.
Claim Extraction with Stance Classification This (Cheng et al., 2022) is integrated task that merges claim extraction and stance classification. In this task, the model needs to extract claims from a series of articles under specific topic and then identify their stances. This task is evaluated as a 3-class classification task.
KPM: ArgKP-2021 The 2021 Key Point Analy- sis (KPA-2021) shared task (Friedman al., 2021) provides a dataset for KPM called ArgKP-200 2021,, covering 31 topics. Since they also report results

---

KPM
CESC
Model
Striet mAP
Relaxed mAP
Mean of mAP
Macro Fi
Miero Fi
KPM
ModemnTalk
75.4
90.2 90.2
82.8 82.4
Baseline
Mat
74.5
CESC
Pipeline Multi-label
55.95
88.56
Baseline
60.25
91.22
no FP
71.3
87.7+3.66
79.5
1..01
89.00
w/o aug.
72. 72.642.1
88.2
80,4
59.59+1 1.29
89.58+0..e
RoBERTa
w/ aug.
+2.7
87.9
80.3
60.08
89.36
FP w/ mix
72.2+2.8
87.6+1.5
79.9+2.1
61.07 10.80
90.22 0.077
no w/o aug.
72.5+3.0
88.842.1
80.642.5
58.81 20.82
88. 78..55g..m 1.35
73.611.3
89.3
81.410.
59.03 62
89.76 $99.6610.0 34
GreaseArG
w/ aug.
75.8+17
89.5+1.2
82.6+1.2
59.65 1.27
89.61 1:42
FP w/ mix
71.5+2.1
85.7+1.5
78.6+1.7
60.52 10.28
89.83 20.23
KPM
SMatchToPR
78.9
92.7
85.8
Baseline
NLP@UIT UIT
74.6
93.0
83.8
(large)
Enigma
73.9
92.8
83.3
Table 1: Main result on KPM and CESC. FP: further pre-training: aug.: relative-augmente samples; mix: FP samples can either plain or augmented, selected randomly during FP. The model group shows KPM results from models with large-scale backbones. Bold font indicates the best results (except for the last group) under cach evaluation metric; underscores indicate the best results within the model group (RoBERTa(GreaseArGG) We also provide standard deviations of each metric across runs, shown to the average.
from other models on this dataset, we use it to conduct experiments on KPM.
we compare our models. These methods can be categorized into two groups: sentence pair classifi- cation predicts results on concatenated input, and contrastive learning uses the similarity between argument and key point representations.
CESC: IAM Besides the integrated task, Cheng et al. (2022) also proposed a benchmark dataset called IAM for training and evaluation. This dataset is generated from articles from 123 differ- ent topics. We adopt this dataset for CESC experi- ments.
Cheng et al. (2022) proposed and exam- ined two approaches for CESC, pipeline and multi- label. The pipeline approach first extracts claims from articles and then decides their stances. The multi-label one treats this task 3-class classifi- cation task, where each candidate argument can be positive, negative, or unrelated to given topic.
CESC
5.3 Implementation
We conduct all experiments on NVIDIA GeForce RTX 4090 and repeat 4 times under different ran- dom seeds. In each run, the best model is chosen according to the performance on the dev set; the final result on the test set is the average of the ones from the best models in all 4 runs. Model and train- ing configurations, including checkpointing rules, are listed in Appendix D.
5.5 Main Results
Table lists the main results on both downstream tasks. We use pair classification for our models in KPM experiments, which gives better results than contrastive learning. For KPM, we select best results from Friedman et al. (2021): SMatchToPR (Alshomary et al., 2021) and MatchTstm (Phan et al., 2021) use contrastive learning; NLP@ UIT, Enigma (Kapadnis et al., 2021) and ModernTalk (Reimer et al., 2021) use pair classification. For CESC, we select the results of both approaches mentioned in Cheng al. (2022).
To obtain further pre-trainin mr-ttrinnng samples, especially augmented ones, we apply a few more operations on the pre-training dataset, details described in Ap- pendix C. Note that documents from args.me me nat- urally contain stance labels towards their conclu- sions. Therefore, we include the RSD task when pre-training with relative- augmented samples.
GreaseArG, after further pre-training with por- tion of samples augmented, showed comparable (on KPM) or better (on CESC) performance se- lected base-scale baselines on either task. While
5.4 Baseline Models
KPM A handful of participants KPA-2021 have described their method for KPM, with which

---

Model
Support
Contest Fi
Model
Macro Fi
Micro Fi
RoBERTa 40.43
47.74
GreaseArG, FP w/
60.52
89.83
GreaseArG 40.59
46.25
w/ MNM
59.87( 0.65)
(0665)
89.54( (5..9)
w/o MEM
59.83( -0.69)
89.78( -0.05)
Table 2: Fi scores on support/eontest claim class on CESC. Here, only show results from FP-with-mix models.
w/o GCL
60.36( 0.16)
89.92( +0009)
w/o TOP
59.76( 00.76)
89.91 (+.08)
w/o DIR
59.67( -0.85)
89.48( 00335
Table Pre-training task ablation results on CESC, based on GreaseArG, further pre-trained with mixed samples. Only graph-related tasks are considered.
Model
Strict
Relaxed Mean
MatchTstm
74.5
90.2
82.4
no FP
67.3
83.9
75.6
RoBERTa
FP w/o aug.
69.7
86.4
78.0
Model
Support Fi
Contest Fi
FP w/ aug.
69.1
85.1
77.1
FP w/ mix
70.0
86.3
78.1
GreaseArG, FP w/ mix
40.59
46.25
no FP
63.7
81.1
72.4
w/o GCL. w/o TOP
40.04( 00.55
16.26(++001)
FP w/o aug.
65.6
81.7
73.7
38.51 - 2.08)
45.90 45.96( 0.299
GreaseArG
FP aug.
69.6
86.6
78.1
FP mix
68.3
84.3
76.3
Table 5: F1 scores support/contest claim class on CESC when ablating GCL/TOP.
Table 3: Evaluation result on KPM, using contrastive learning instead of pair classification. Bold font indi- cates the best results within each group.
6.2 Relative-Augmennted Further Pre-training As shown in Table 1, adding samples augmented with relatives generally helps improve further pre- training. This phenomenon is more significant with GreaseArG and the CESC task. Inter- estingly, models could prefer various portions of augmented samples for different downstream tasks: both vanilla RoBERTa and GreaseARG pre- fer mixed samples during further pre-training on CESC, yet this strategy has shown adverse effects on KPM.
vanilla RoBERTa gives worse results² than baseline models, on CESC it gains significant boost when further pre-trained with mixed samples. Our re- sults being compared the mean of runs; when taking random fluctuations into account, further pre-trained GreaseArG can easily supersede base- scale models and even be on par with large-scale baselines.
6 Analysis
6.3 Alternative Approach for KPM
We further investigate the impact of different down- stream approaches. Table 3 illustrates the results on KPM under the same setup but using contrastive learning as the downstream approach. Since the model computes the representation of arguments and key points separately, cannot exploit the potential graph connections between them, which could be why GreaseArG is worse than RoBERTa. Nonetheless, further pre-training benefits both models, where the gap between GreaseArG and RoBERTa nearly disappeared. Although the base- scale MatchTssm (Phan et al., 2021) from Fried- man al. (2021) show better results, this model has introduced other strategies to improve model performance and thus cannot be compared directly.
6.1
GreaseArG Vanilla LM
As seen in Table 1, under the same further pre- training settings, GreaseArG significantly outper- forms vanilla RoBERTa on all metrics of KPM. However, CESC, the order is reversed. Table demonstrates the Fi scores of each stance class (support and contest). While GreaseArG super- sedes RoBERTa concerning supporting claims, its performance on contesting claims is not as good as RoBERTa's. This result aligns with the ones on KPM, where GreaseArG shows an advantage since matching pairs of arguments and key points must have the same stance toward the topic.
FFor CESC, RoBERTa without further pre-training is equivalent to the multi-label model. However, we use different random seods and average results across all runs; hence, the final Fj scores are different from what is reported by Cheng et al. (2022).
6.4 Contributions of Pre-training Tasks
We conduct ablation experiments on CESC us- ing GreaseArG, further pre-trained with mixed

---

Hi-ArG Jaccard
RoBERTa, no FP
GreaseArG, w/ aug.
Argument
Candidate Key Point
Gold Label
Mandatory vaccination contradicts busic rights
non-matching
0.0417
0.001
1.263
agree force children without parental consent should be fair
The parents and not the state should decide
matching
00.132
1.609
Table 6: KPM sample where GreaseArG corrects vanilla RoBERTa. Hi-ArG Jaccard is the Jaccard Similarity between intra-arg node sets of argument and key point. Note that predicted scores across models are not comparable.
KPM (mAP)
CESC (75)
ments, 94 have seen the label of their predicted key point changed, of which 58 or 61.7% have positive changes, Table demonstrates such a positive ex- ample. We can see that the argument and candidate key points have common Hi-ArG nodes, which GreaseArG can exploit.
Model
R.
M.
Macro
Micro
RoBERTa FP w/ mix
72.2 87.6 79.9 61.07 90.28
GreaseArG FP w/ aug.
75.8
89.5 82.6 59.65 89.61
Direct
24.0
35.2
29.6
34.29
56.73
ChatGPT
Explain
43.5
58.8 51.2 41.60 69.02
Table 7: ChatGPT performance KPM and CESC.
6.6 Comparing with LLM
Direct: the model generates the prediction directly. Ex- plain: the model generates an explanation before the final prediction. Table headers are the same as Table 1.
Large language models (LLMs) like ChatGPT and GPT-4 have recently shown competitive perfor- mance various applications without fine-tuning (OpenAT, 2023). In response to this trend, we duct primitive experiments that apply ChatGPT 3ppt-3.5-turbe on KPM and CESC, using in- context learning only, For KPM, we ask the model to give a matching score to maintain the evaluation process. For CESC, we asked the model to treat the task as a 3-class classification and predict the class label. All prompts consist of the task definition and several examples.
samples, to measure the contributions of various graph-related pre-training tasks during further pre- training (the importance of MLM is obvious). Re- sults are listed in table 4. All ablated models per- form worse concerning the macro F1. At the same time, a few of them have higher micro Fi, indi- cating that such ablation improves performance in some classes yet harms others. Tasks related to graph components contribute more; conversely, re- moving Graph Contrastive Learning (GCL) causes a less significant effect.
Table demonstrates the results, comparing them with our best results. ChatGPT gives poor results when asked to predict directly, even with examples given. Asking the model to explain be- fore answering can improve its performance, but it has yet to reach our models and other baselines. While these primitive experiments have not in- volved further prompt engineering and techniques. they have shown that more efforts (such as Hi-ArG) are needed for LLMs handle computational ar- gumentation tasks.
To further investigate the increasing micro Fi when ablating GCL and Top Order Prediction (TOP), table illustrates Fi scores on claim classes when ablating either task. Ablation caused sig- nificant drop support Fi yet did not significantly increase contest F1. This indicates that the higher micro Fi is due to the imbalance between claims and non-claims (527 against 6, 538): ablating these tasks might slightly improve claim extraction but could harm stance classification.
Conclusion
In this paper, we propose a new graph structure for argumentation knowledge, Hi-ArG. We design an automated construction pipeline to generate the lower intra-arg graph. To exploit information from Hi-ArG, we introduce a text-graph multi-modal model, GreaseArG, and a novel pre-training frame- work. On two different argumentation tasks (KPM and CESC), these approaches create models super- seding current models under the same scale.
6.5 Hi-ArG Information Benefit
To examine the benefits of Hi-ArG information on downstream tasks, compare model predictions the KPM task, which is the candidate key point with the highest predicted score for each argument.
We chose RoBERTa without further pre-training, and GreaseArG further pre-traine pre-trained with augmented samples as target models. Among all 723 argu-

---

Limitations
Alfonso Amayuelas. Shuai Zhang, Xi Susie Rao, and Ce Zhang. 2022. Neural methods for logical reason- ing over knowledge graphs. In International Confer- ence on Learning Representations.
Despite the above results, this paper has a few limi- tations for which we appreciate future studies. First, the quality of Hi-ArG can be further improved us- ing more powerful models and tools. Second, we only test small language models due resource limitation, yet, we conjecture that integrating Hi- ArG and LLMs could also be a potential solution to some current problems in LLMs, such as hallu- cination.
Sören Auer, Christian Bizer, Georgi Kobilarov, Jens Lehmann. Richard Cygamiak, and Zachary Ives. 2007. Dbpedia: A nucleus for web of open data. In The Semantic Web, pages 722-735, Berlin, Heidelherg. Springer Berlin Heidelberg.
Isabelle Augenstein, Tim Rocktäschel, Andreas Vla- chos, and Kalina Bontcheva. 2016. Stance detection with bidirectional conditional encoding. In Proceed- ings of the 2016 Conference on Empirical Methods in Natural Language Processing, pages 876-885, Austin, Texas. Association for Computational Lin- guistics.
Acknowledgments
This work is supported by National Natural Sci- ence Foundation of China (No. 6217020551) and Science and Technology Commission of Shanghai Municipality Grant (No.21QA1400600)
Xuefeng Bai, Yulong Chen, and Yue Zhang. 2022. Graph pre-training for AMR parsing and generation. In Proceedings of the 60th Annual Meeting of the Association for Compulational Linguistics (Volume
References
1: Long Papers), pages 6001-6015, Dublin, Ireland. Association for Computational Linguistics.
Yamen Ajjour, Henning Wachsmuth, Johannes Kiesel, Martin Potthast, Matthias Hagen, and Benno Stein. 2019. Data acquisition for argument search: The args.me corpus. In KI 2019: Advances in Artificial Intelligence. pages 48 59, Cham. Springer Interna- tional Publishing.
Laura Banarescu, Claire Bonial, Shu Cai, Madalina Georgescu, Kira Griffitt, Ulf Hermjakob. Kevin Knight, Philipp Koehn, Martha Palmer, and Nathan Schneider. 2013, Abstract Meaning Representation for sembanking. In Proceedings of the 7th Linguistic Annotation Workshop and Interoperability with Dis- course, pages 178-186, Sofia, Bulgaria. Association for Computational Linguistics.
Khalid AI-Khatib, Yufang Hou, Henning Wachsmuth, Charles Jochim, Francesca Bonin, and Benno Stein. 2020. End-to-end argumentation knowledge graph construction. Proceedings of the AAAI Conference on Artificial Intelligence, 34005)736777374.
Jianzhu Bao, Chuang Fan, Jipeng Wu, Yixue Dang, Ji- achen Du, and Ruifeng Xu. 2021. A neural transition- based model for argumentation mining. In Proceed- ings of the 59th Annual Meeting of the Association for Computational Linguistics and the 1 11th International Joint Conference (Volume I: Long Papers). pages 6354-6364, Online. Association for Computational Linguistics.
Khalid Al Khatib, Lukas Trautner, Henning Wachsmuth, Yufang Hou, and Benno Stein. 2021. Employing ar- gumentation knowledge graphs for neural argument generation. In Proceedings of the 59th Annual Meet- ing of the Association for Computational Linguistics and the 11th International Joint Conference on Natu- ral Language Processing (Volume I: Long Popers), pages 4744-4754, Online. Association for Computa- tional Linguistics.
Natural Language Processing
Roy Bai-Haim, Lilach Eden. Roni Friedman, Yoav Kan- tor, Dan Lahav, and Noam Slonim. 2020a. From ar- guments to key points: Towards automatic argument summarization. In Proceedings of the 58th Annual Meeting of the Association for Computational Lin- guistics, pages 4029-4039, Online. Association for Computational Linguistics.
Milad Alshomary, Roxanne Ell Balf, Timon Gurcke, and Henning Wachsmuth. 2022. The moral debater: study on the computational generation of morally framed arguments. In Proceedings of the 60th An- nual Meeting of the Association for Computational Linguistics (Volume 1: Long Pupers), pages 8782- 8797, Dublin, Ireland, Association for Computational Linguistics.
Roy Bar-Haim, Yoav Kantor, Lilach Eden, Roni Fried- man, Dan Lahav, and Noam Slonim. 2020b. Quanti- tative argument summarization and beyond: Cross- domain key point analysis. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 39-49, On- line. Association for Computational Linguistics.
Milad Alshomary, Timon Gurcke, Shahbaz Syed, Philipp Heinisch, Maximilian Splicthöver, Philipp Cimiano. Martin Potthast, and Henning Wachsmuth. 2021. Key point analysis via contrastive learning and extractive argument summarization. In Proceedings of the 8th Workshop on Argument Mining, pages 184- 189, Punta Cana, Dominican Republic. Association for Computational Linguistics.
Kurt Bollacker, Colin Evans, Praveen Paritosh, Tim Sturge, and Jamie Taylor. 2008 Freebase: A col- laboratively created graph database for structuring human knowledge. In Proceedings of the 2008 ACM SIGMOD International Conference on Management

---

of Data, SIGMOD '08, pages 1247-1250, New York, NY, USA. Association for Computing Machinery.
Stefan Heindorf, Yan Scholten, Henning Wachsmuth, Axel- Beel-Cyrille Ngonga Ngomo, and Martin Potthast. 2020. Causenet: Towards a causality graph extracted from the web. In Proceedings of the 29th ACM Inter- national Conference on Information & Knowledge Management, CIKM '20, pages 3023-3030, New York, NY, USA. Association for Computing Machin- ery.
Claire Bonial, Julia Bonn, Kathryn Conger, Jena D. Hwang, and Martha Palmer. 2014. PropBank: Se- mantics of new predicate types. In Proceedings of the Ninth International Conference on Language Resources and Evaluation (LREC' pages 3013- 3019, Reykjavik, Iceland. European Language Re- sources Association (ELRA).
Manav Kapadnis, Sohan Patnaik, Siba Panigrahi, Varun Madhavan, and Abhilash Nandy. 2021. Team enigma at EigMining-EMNN language models for key point matching. In Pro- ceedings of the 8th Workshop on Argument Mining, pages 200-205, Punta Cana, Dominican Republic. Association for Computational Linguisties.
Katarzyna Budzynska and Chris Reed. 2011. Whence inference. Technical report, University of Dundee.
2021: Leveraging pre-trained
Liying Cheng, Lidong Bing, Ruidan He, Qian Yu, Yan Zhang, and Luo Si. 2022. IAM: A comprehensive and large-scale dataset for integrated argument min- ing tasks. In Proceedings of the 60th Annual Meet- ing of the Association for Computational Linguistics (Volume 1: Long Papers), pages 2277-2287, Dublin, Ireland. Association for Computational Linguisties.
Jonathan Kobbe, Ioana Hulpus, and Heiner Stucken- schmidt. 2020. Unsupervised stance detection for arguments from consequences. In Proceedings of the 2020 Conference on Empirical Methods in Nat- ural Language Procexsing (EMNLP), pages 50-60, Online. Association for Computational Linguistics.
Subhabrata Dutta, Jeevesh Juneja, Dipankar Das, and Tanmoy Chakraborty. 2022. Can unsupervised knowledge transfer from social discussions help ar- gument mining? In Proceedings of the 60th Annual Meeting of the Association for Computational Lin- guistics
Ran Levy, Ben Bogin, Shai Gretz, Ranit Aharonov, and Noam Slonim. 2018 Towards an argumentative con- tent search engine using weak supervision. In Pro- ceedings of the 27th International Conference on Computational Linguistics, pages 2066-2081, Santa Fe. New Mexico, USA. Association for Computa- tional Linguistics.
( (Volume 1: Long Papers), pages 7774-7786, Dublin, Ireland. Association for Computational Lin- guistics.
Vijay Prakash Dwivedi and Xavier Bresson. 2020. generalization of transformer networks to graphs.
Mike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, Abdelrahman Mohamed, Omer Levy, Veselin Stoyanov, and Luke Zettlemoyer. 2020. BART: Denoising soqueecce-t--eqqeeee for natural language generation, translation, and com- prehension. In Proceedings of the 58th Annual Meet- ing of the Association for Computational Linguistics, pages 7871-7880, Online, Association for Computa- tional Linguistics.
Steffen Eger, Johannes Daxenberger, and Iryna Gurevych. 2017. Neural end-to-end learning for computational argumentation mining. In Proceed- ings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Pa- pers). pages 11 11-22, Vancouver, Canada. Association for Computational Linguistics.
pre-training
Roxanne El Baff. Henning Wachsmuth, Khalid AI Khatib, and Benno Stein. 2020. Analyzing the Per- suasive Effect of Style in News Editorial Argumenta- tion. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 3154-3160, Online. Association for Computational Linguistics.
Zuchao Li, Shexia He, Jiaxun Cai, Zhuosheng Zhang. Hai Zhao, Gongshen Liu, Linlin Li, and Luo Si. 2018. A unified syntax-aware framework for semantic role labeling. In Proceedings of the 2018 Conference Empirical Methods in Natural Language Processing. pages 2401-2411, Brussels, Belgium. Association for Computational Linguistics.
Roni Friedman, Lena Dankin, Yufang Hou, Ranit Aharonov, Yoav Katz, and Noam Slonim. 2021. Overview of the 2021 key point analysis shared task. In Proceedings of the 8th Workshop on Argument Mining, pages 154-164, Punta Cana, Dominican Re- public. Association for Computational Linguistics.
Marco Lippi and Paolo Torroni. 2016. Argument min- ing from speech: Detecting claims in political de- bates. Proceedings of the AAAI Conference on Artifi- cial Intelligence, 30(1).
Shai Gretz, Yonatan Bilu, Edo Cohen- and Noam Slonim. 2020. The workweek is the best time start a family - a study of GPT-2 hased claim generation. In Findings of the Association for Computational Linguistics: EMNLP 2020, pages 528-544, Online. Association for Computational Linguisties.
Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Man- dar Joshi, Danqi Chen, Omer Levy. Mike Lewis. Luke Zettlemoyer, and Veselin Stoyanov. 2019. Roberta: A robustly optimized bert pretraining ap- proach.
OpenAL. 2023. Gpt-4 technical report.
Ivan Habernal and Tryna Gurevych. 2017. Argumenta- tion mining in user-generated web discourse. Com- putatoonal Linguistics, 43(1) 177.
Martha Palmer, Daniel Gildea, and Paul Kingsbury. 2005. The Proposition Bank: An annotated corpus of

---

semantic roles. Computational Linguisties, 31(1):71- 106.
Noam Slonim, Yonatan Bilu, Carlos Alzate, Roy Bar-Haim, Ben Bogin, Francesca Bonin, Leshem Choshen, Edo Cohen-Karlik Karlik, Lena Dankin, Lilach Edelstein, Liat Ein-Dor, Roni Fricdman-Melamees Assaf Gavron, Ariel Gera, Martin Gleize, Shai Gretz, Dan Gutfreund, Alon Halfon, Daniel Hershcovich, Ron Hoory, Yufang Hou, Shay Hummel, Michal Ja- covi, Charles Jochim, Yoav Kantor, Yoav Katz, David Konopnicki, Zvi Kons, Lili Kotlerman, Dalia Krieger, Dan Lahav, Tamar Lavee, Ran Levy, Naftali Liber- man, Yosi Mass, Amir Menczel. Shachar Mirkin, Guy Moshkowich, Shila Koif-Ko,,,,n, Matan Or- bach, Ella Rabinovich, Ruty Rinott, Slava Shechtman, Dafna Sheinwald, Eyal Shnarch, llya Shnayderman. Aya Soffer, Artem Spector, Benjamin Sznajder, As- saf Toledo, Orith Toledo-Ronen, Elad Venezian, and Ranit Aharonov. 2021. An autonomous debating system. Nature, 591(7850):379-3 388443433
Joonsuk Park, Cheryl Blake, and Claire Cardie. 2015. Toward machine-assisted participation in erulemak- ing: An argumentation model of evaluability. In Proceedings of the I 15th International Conference on Artificial Intelligence and Law, ICAIL 11, pages 206-210, New York, NY, USA. Association for Com- puting Machinery.
Isaac Persing and Vincent Ng. 2016. End-to-end argu- mentation mining in student essays. In Proceedings of the 2016 Conference of the North American Chap- ter of the Association for Computational Linguis- tics: Human Language Technologies, pages 1384- 1394, San Diego, California. Association for Compu- tational Linguistics.
Robyn Speer, Joshua Chin, and Catherine Havasi. 2017. Conceptnet 5.5: An open multilingual graph of gen- eral knowledge. Proceedings of the AAAI Conference Artificial Intelligence, 31(1).
Hoang Phan, Long Nguyen, Long Nguyen, and Khanh Doan. 2021. Matching the statements: A simple and accurate model for key point analysis. In Pro- ceedings of the 8th Workshop on Argument Mining, pages 155-174, Punta Cana, Dominican Republic. Association for Computational Linguistics.
Christian Stab, Johannes Daxenberger, Chris Stahlhut, Tristan Miller, Benjamin Schiller, Christopher Tauch- mann, Steffen Eger, and Iryna Gurevych. 2018. Ar- gumenTexx: Searching for arguments in heteroge- neous sources. In Proceedings of the 2018 Confer- ence of the North American Chapter of the Associa- tion for Computational Linguistics: Demonstrations, pages 21 11-25, New Orleans, Louisiana, Association for Computational Linguistics.
Nima Pourdamghani, Yang Gao, Ulf Hermjakob, and Kevin Knight. 2014. Aligning English strings with Abstract Mcaning Representation graphs. In Proceedings of the 2014 Conference Empirical Methods in Natural Language Processing (EMNLP), pages 425 429, Doha, Qatar. Association for Com- putational Linguisties.
Christian Stab and Iryna Gurevych. 2014. Identifying argumentative discourse structures in persuasive es- says. In Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 46-56, Doha, Qatar. Association for Computational Linguistics.
Alec Radford, Karthik Narasimhan, Tim Salimans, Nya Sutskever, et al. 2018. Improving language under- standing by generative pre- pre-training, OpenAl blog.
Jan Heinrich Reimer, Thi Kim Hanh Luu, Max Henze, and Yamen Ajjour. 2021. Modern talking in key point analysis: Key point matching using pretrained encoders. In Proceedings of the 8th Workshop on Argument Mining, pages 175-183, Punta Cana, Do- minican Republic. Association for Computational Linguistics.
Christian Stab and Tryna Gurevych. 2017. Parsing argu- mentation structures in persuasive essays. Computa- tional Linguistics, 43(3)619-659.
Stephen E, Toulmin. 2003. The Uses of Argument. Cam- bridge University Press.
João António Rodrigues and António Branco. 2022. Transferring confluent knowledge to argument min- ing. In Proceedings of the 29th International Con- ference on Computational Linguistics, pages 6859- 6874, Gyeongju, Republic of Korea. International Committee on Computational Linguistics.
Henning Wachsmuth, Nona Naderi, Yufang Hou, Yonatan Bilu, Vinodkumar Prabhakaran, Tim Alberd- ingk Thijm, Graeme Hirst, and Benno Stein. 2017. Computational argumentation quality assessment in natural language. In Proceedings of the 15th Con- ference of the European Chapter of the Association for Computational Linguistics: Volume 1, Long Pa- pers, pages 176-187, Valencia, Spain. Association for Computational Linguistics.
Benjamin Schiller, Johannes Daxenberger, and Iryna Gurevych. 2021. Aspect-controlled neural argument generation. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies. pages 380-396, Online. Association for Computational Linguistics.
Lu Wang and Wang Ling, 2016. Neural network-based abstract generation for opinions and arguments. In Proceedings of the 2016 Conference of the North American Chapter of the Association for Computa- rional Linguistics: Human Language Technologies, pages 47-57, San Diego, California. Association for Computational Linguistics.
Peng Shi and Jimmy Lin. 2019. Simple bert models for relation extraction and semantic role labeling.

---

Jun Xia, Yangiao Zhu, Yuanqi Du, and Stan Z. Li. 2022. A survey of pretraining on graphs: Taxonomy, meth- ods, and applications.
Guns kill
kill-01
people.
:RGG
ARG1
People are killed by guns.
Jian Yang. Gang Xiao, Yulong Shen, Wei Jiang, Xinyu Hu, Ying Zhang, and Jinghui Peng. 2021. A survey of knowledge enhanced pre-trained models.
gun
person
Figure 5: An example of an AMR graph representing two sentences: "Guns kill people" and "People are killed by guns."
Xikun Zhang. Antoine Bosselut, Michihiro Yasunaga. Hongyu Ren, Percy Liang, Christopher D. Manning, and Jure Leskovec. 2022. GreaseLM: Graph REA- Soning enhanced language models. International Conference on Learning Reprexentations.
thus, we choose this root node as the top node of the argument so that its descending subgraph is exactly the AMR graph of the argument. Finally, we force every AMR graph recorded in Hi-ArG to be acyclic. following the strict definition of AMR graphs, since every edge in an AMR graph can be reverted by appending to or removing from its relation label passive mark -of.
Yizhen Zheng, Shirui Pan, Vincent Lee, Yu Zheng, and Philip Yu. 2022. Rethinking and scaling up graph contrastive learning: An extremely efficient approach with group discrimination. In Advances in Neural Information Processing Systems.
A AMR and Hi-ArG Intra-arg Graphs
semantic analysis, Abstract Mcaning Represen- tation (AMR, Banarescu et al. 2013) is a highly abstract structure representing semantic informa- tion of sentences and documents. An AMR is rooted, directed graph with labeled nodes and edges, where nodes record concepts predicates and edges record semantic relations between them. this way, sentences semantically identical can be assigned to the same AMR graph, even if they are syntactically different.
B Hi-ArG Construction Details
B.1 Extracting Stage
We use spaCy² to cut documents into sentences. Some basic rules can be applied to filter out in- valid sentences for most corpus, such as limiting the minimum number of words or forcing all char- acters to be printable. While advanced filtering based on argumentation mining models can also be applied to enhance argument quality, in this paper, we only focus on basic filters, which are easier to implement.
In a general AMR graph, concepts (as nodes) can be English words, PropBank (Palmer et al., 2005) framesets (a predicate labeled with semantic arguments), or predefined keywords such nega- tion (-); relations (as directed edges) can be frame arguments defined PropBank, or other relations that predicate phrases cannot cover (domain, topic. quantity, date/time, list element, etc.).
B.2 Parsing Stage
AMR parsing can be tedious when conducted man- ually; however, the recent development of language models has made automatic AMR parsing possi- ble on large corpora. Although the AMR graphs generated by these models could contain noise, the negative effect is acceptable specific applications such as pre-training (Bai et al., 2022). In this paper, we apply a BART--based (Lewis et al., 2020) model implemented by amrlib' to this stage to generate AMR graphs for all datasets.
For example, in Figure 5, both sentences can be represented by the given AMR graph. Concepts in the graph include gun, person, and kill-01, where kill is PropBank frameset and kill-01 is one of its role sets, meaning "causing to die". Relations in the graph include :RGG and ARG1, following the roleset definition of kill-01 where
B.3 Merging Stage
: :RGG the killer and :RGG the corpse.
Due to its capability to express the semantics of multiple sentences compactly, we introduce AMR the primary backbone of the intra-arg graph of Hi-ArG. More specifically, an argument is stored in the intra-arg graph in its AMR form, keeping all nodes and edges unchanged. Moreover, each AMR graph has a root node that can visit all of the nodes in the graph through the directed edges;
We implement the iterative merging method in this stage based node isomorphism. To identify iso- morphic nodes, each node is labeled by its attribute (text), its child nodes, and the attributes of edges between them (referred to as child edges). Two nodes are called directly isomorphic they share
"https://spacy.io"

---

# sent.
token
node
edge
top.
Sentences must not appear close in the source corpus: there should be at least L sen- tences between them.
6.08M
141M
29.4M
61.0M
5.23M
Table Statistics of the pre-training dataset constructed from the args.me corpus. The number of tokens is com- puted based on the RoBERTa tokenizer.
4. Only nodes connected to no more than S sen- tences are considered.
5. Leading hint sentences ("..." is right/wrong.) will not have sampled relatives, though they can be relatives of other sentences.
the same node attribute, child nodes, and child edge attributes therefore must have identical labels. Note that merging nodes may only change the la- bel of their entrance nodes, as AMR graphs are acyclic. Meanwhile, merging directly isomorphic nodes can make these nodes, if isomorphic, directly isomorphic. Thus, all isomorphic sub-gran aub-graphs can be merged by repeatedly merging directly isomor- phic nodes.
The third constraint is prevent relatives from ap- pearing the training sample twice, and the fourth is to exclude pairs with only weak connections through "public" nodes representing common con- cepts. In this paper, we use = 31 and S = 500. Although several sentences cannot find any relative under these constraints, 32.10% of all sentences have at least one candidate match.
C Further Pre-training Details
Due the limitation of computational resources. we sample relatives before pre-training, among which 45.88% are of the same stance. This ratio is balanced enough for a valid RSD task for further pre-training.
C.1 Hi-ArG Construction
In args.me, documents are recorded as conclusion- premise pairs, where the premises can be for or against the conclusion. During the extracting stage, premise sentences shorter than words contain unprintable characters excluded; the conclusion sentence, leading the whole document, is rewritten as "..." is right/wron.. The parsing and merging stages are the same as described in Section 3.2, yet we also include an extra graph-text aligning using amb for multi-modal mask generation.
C.3 Multi-modal Mask Generation
Although random masking can help language mod- els learn text representations, improving mask se- lection with extra information from Hi-ArG is pos- sible. Here we propose special masking proce- dure to generate more valuable masks on both texts and graphs (nodes and edges):
The final further pre-training dataset contains 6, 080, 249 sentences and large Hi-ArG. More detailed statistics of the dataset are listed in Table 8. Note that several sentences share the same meaning. Hence each group of such sentences corresponds to one single top node.
1.
Masks are generated on graphs, where the probability of each node being masked is pro- portional to the number of sentences it is lated to, and of each edge, the weight product of nodes on both sides. In this way, common nodes and edges across sentences are more likely to be masked and Iet the model focus on such intersections. The overall mask ratio can still be controlled at specific level.
C.2 Relative Searching
As mentioned in Section 4.3, we can search for relatives Hi-ArG with the help of the sentence- node graph. Since the full sentence-node graph of args.me is too large to perform weighted sampling for all sentences, we introduce few constraints on candidate sentence pairs reduce computation:
2. A pre-mask text is computed based on graph masks, achieved using graph-text align- ments generated beforehand. Each masked node and cach node with a masked edge looks for any aligned token spans and mask them, creating a base text mask that shall not leak information between text and graph.
1. Sentences must not share one top node in the original Hi-ArG.
2. Sentences must come from documents under the same topic or conclusion.
3. If the pre-mask does not reach the desired mask ratio for text, extra masks will be gen- erated on non mask tokens to satisfy the re- quirement.
'For instance, if a document supports the conclusion We should bon guns, then the first sentence extracted from the document will be "We should ban guns" is right.

---

Model Configuration
Value
Fine-tuning Configuration
KPM
CESC
Maximum # Tokens
512
Batch Size
32
128
Node PE Dimension
Optimizer
Adam
Adam W
# Joint Blocks
Learning Rate
10-3
x 10-5
GNN Hidden Size
Decay
inverse_soe sqri
inverse_sqrt
Graph Transformer Attention Heads
Warmup Ratio
0.1
# Mix Layers
Total Epochs
20
Layer Hidden Size
Cross-modal Attention Heads
'abble 11: Fine-tuning configuration for KPM and CESC.
Table Model configuration of GreaseArG, Configura- tions mentioned here inherit values from RoBERTa- base and Graph Transformer.
checkpoints will be used base models during fine-tuning. In the fine-tuning stage, after each fine- tuning epoch, evaluation will be conducted on dev and test sets. Test results from the model with the highest gold metric on the dev set will be chosen as the final test result of the current random seed. The mean of strict and relaxed mAP is selected for KPM as the gold metrie; for CESC, the macro Fi is selected.
Pre Pre-training Configuration
Value
Batch Size
32
Tokens per Sample
512
Optimizer
AdamW x 10
LM Learning Rate
Non-LM Learning Rate
x 10
LR Decay
inverse sqrt
Warmup Steps
2 500
Total Steps
180, 000
Table Pre-training configuration for both RoBERTa and GreaseArG.
D Implementation Details
D.1 Model
Table lists model configuration parameters ap- plied to all experiments. Graph Transformer re- quires positional embeddings (PE) for every node to encode positional information. Following the suggestions by Dwivedi and Bresson (2020), lapla- cian eigenvectors are chosen node PEs, calcu- lated on bi-directed graph removing all edge at- tributes.
D.2 Further Pre-training
Table lists parrmeerrreeters used during fur- ther pre-training, applied to both RoBERTa and GreaseArG. Training samples are guaranteed to fill up 512 tokens by concatenating many sentences possible. The total training step is approximately epoch concerning the training set (95% of all pre-training data), rounding to ten thousand for checkpoint generation.
D.3 Fine-tuning
Fine-tuning typer-parameters for the two down- stream tasks are listed in Table 11.
D.4 Checkpointing Rules
For each pre-training model and configuration, checkpoints are saved every 000 steps, and these


Reviews:
[{"review_id": "uoOnk03F58", "reviewer": "Reviewer_zpwA", "summary": "", "questions": "A. Why did you bold the performance excluding large-scale models' performance in Table 2? I think it's really confusing to reading the table. \nB. Why is the performance drops when ablating GCL, TOP pre-training tasks in Table 6?", "limitations": "", "rating": 4, "confidence": 4, "soundness": 5, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank you for taking the time to review our submission and for your valuable feedback.\n\nIn response to your comments and questions, we would like to provide the following rebuttal:\n\n## Q1: Clarity in notations, especially in Figure 3\n\nWe thank you for pointing out the need for more clarity in the notation. In Figure 3, $\\tilde{t}$ denotes token embeddings went through at least one LM layer, distinguishing from the original $t$ generated by the token embedding layer. $G=\\langle\\tilde{V},E\\rangle$ denotes the embedded graph went through at least one GNN layer: $\\tilde{V}=\\{\\tilde{v_1},\\tilde{v}_2,\\tilde{v}_3,\\dots,\\}$ are node embeddings transformed by at least one GNN layer, also distinguishing from the original $V=\\{v_1,v_2,v_3,\\dots\\}$ generated by the node embedding layer; $E=\\{e_0,e'_0,e_1,e'_1,\\dots\\}$ are edge embeddings generated by the edge embedding layer (prime marks indicate reversed edges), which remain unchanged throughout the process.\n\nIn response to your comment, we will add further explanations of several notations that may confuse, including the ones you have mentioned, to the revised version of our paper.\n\n## Q2: Further explanation with visualization for pre-training tasks\n\nWe appreciate your advice concerning visualization. We will add more details about pre-training tasks to the Appendix in the revision, including necessary text and figures.\n\n## Q3: Notations and format of Table 2\n\nWe mainly compared our model with base-scale baseline models because they have the same backbone capability and pre-training knowledge. Meanwhile, the KPM-2021 shared task's ranking contains both base-scale and large-scale models, and we found that our model performs on par with some large-scale models. Therefore, we decided to include these baseline results in Table 2.\n\nYet, in response to your concern, we will split the KPM baselines into base-scale and large-scale groups in the revised version of Table 2 to further clarify it.\n\n## Q4: Model performance when ablating GCL or TOP\n\nWe suppose you are concerned about the increasing micro $F_1$ when ablating GCL and TOP. However, the increments are statistically insignificant, and their macro $F_1$ decreases more (especially for TOP). We suggest this is due to the imbalance between claims and non-claims (much more than claims): ablating these tasks might *slightly* improve **claim extraction** but could harm **stance classification**.\n\nFurther investigations have shown that when considering claim stance classes, ablating either task caused a significant drop in support $F_1$, yet did not significantly increase contest $F_1$. See the table below for evaluation details. These results support our claim above.\n\n| Model                | Macro $F_1$       | Micro $F_1$       | Support $F_1$     | Contest $F_1$     |\n| -------------------- | ----------------- | ----------------- | ----------------- | ----------------- |\n| GreaseArG, FP w/ mix | $60.52$           | $89.83$           | $40.59$           | $46.25$           |\n| ... w/o GCL          | $60.36$ ($-0.16$) | $89.92$ ($+0.09$) | $40.04$ ($-0.55$) | $46.26$ ($+0.01$) |\n| ... w/o GCL          | $59.76$ ($-0.76$) | $89.91$ ($+0.08$) | $38.51$ ($-2.08$) | $45.96$ ($-0.29$) |\n\n## Q5: Missing reference, typo, and misplaced Table 1\n\nWe thank you for notifying us about these referencing and writing issues. We will fix them in the revised version of our paper.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1) This work suggests Hi-ArG, and its graph structure is sufficient to retain more semantics within the arguments by borrowing AMR's structure and concepts. \n2) Also, the exploitation methods of the Hi-ArG are well explained, and the experiments and analyses show its potential. \n3) The paper also analyzes the degradation of the GreaseArG in detail in the analysis section which lead the way of further research.", "reasons_to_reject": "1) The paper needs more clarity in notation. Especially, the explanation of the notations in Figure 3 is missing such as t^tilde, G, E, and V. Even the notations are widely used, please indicate what they are in the paper. \n2) Visualization of graph structure tasks for the pre-training would be helpful. The three pre-training tasks need more explanation along with visualization for clear understanding.", "questions_to_authors": "A. Why did you bold the performance excluding large-scale models' performance in Table 2? I think it's really confusing to reading the table. \nB. Why is the performance drops when ablating GCL, TOP pre-training tasks in Table 6?", "paper_topic_and_main_contributions": "1) This paper introduces a Hierarchical Augmentation Graph to retain more semantics within the arguments at the intra-argument level and inter-argument level. \n2) Also, this work suggests two approaches to exploit Hi-ArG, which are the text-graph multi-modal model GreaseArG and the pre-training framework. \n3) The claim is well supported by the experiments and the analyses suggest diverse insights of exploiting Hi-ArG in KPM and CESC tasks.", "excitement": 4, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Missing_References": "[1] SpaCy in Appendix A.1", "Typos_Grammar_Style_and_Presentation_Improvements": "1) line 966, comma is missing in number. \n2) Since Table 1 is mentioned in Appendix for the first time, so please move the Table to Appendix as well."}, "extra_text": {}}, {"review_id": "9Cfnh7IESk", "reviewer": "Reviewer_nF6N", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 3, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank you for taking the time to review our submission and for your valuable feedback.\n\nIn response to your comments and questions, we would like to provide the following rebuttal:\n\n## Q1: Insignificant performance boost\n\nWe understand your concern regarding the significance of the performance boost. Yet we would like to provide a potential explanation on this point.\n\nWe repeated all experiments four times with different seeds and averaged their results as the final ones demonstrated in our paper; however, the baseline results to which our results are compared may follow a different setting. For instance, in CESC, our RoBERTa, without further pre-training, achieved a lower average performance than the multi-label baseline, even though both models are *essentially the same*.\n\n## Q2: Evaluation of automatically generated Hi-ArG\n\nWe appreciate your advice regarding the correctness of the generated Hi-ArG.\n\nThe quality of generated intra-arg and inter-arg graphs can be measured using the correctness of their corresponding sub-tasks (e.g., AMR parsing and argument mining). We believe current methods are powerful and robust enough to ensure *in-level* quality.\n\nHowever, as we understand, evaluating the *entire* Hi-ArG requires novel approaches since no existing metric exists. A potential way we would like to propose here is to use a small fraction of the corpus and measure the similarity between manually annotated and automatically generated graphs.\n\nThe above method is only one of many possible approaches, and we encourage more research on this topic. Nevertheless, the evaluation task would be crucial in future work to improve Hi-ArG quality.\n\n## Q3: LLM performance on downstream tasks\n\nWe appreciate your advice concerning LLM's performance on our downstream tasks.\n\nWe have conducted a few primitive experiments that apply `gpt-3.5-turbo` (GPT) on KPM and CESC, using in-context learning only. For KPM, we asked GPT to give a matching score so that the evaluation process remained the same. For CESC, we asked GPT to treat the task as a 3-class classification and predict the class label.\n\nWe observed that GPT gave poor results when asked to predict directly, even with examples given. Asking GPT to explain before answering improved its performance, but it has yet to reach our models and other baselines.\n\n| Model                | (KPM) Strict mAP | Relaxed mAP | Mean of mAP | (CESC) Macro $F_1$ | Micro $F_1$ |\n| -------------------- | ---------------- | ----------- | ----------- | ------------------ | ----------- |\n| RoBERTa FP w/ mix    | $72.2$           | $87.6$      | $79.9$      | $61.07$            | $90.28$     |\n| GreaseArG FP w/ aug. | $75.8$           | $89.5$      | $82.6$      | $59.65$            | $89.61$     |\n| GPT (direct predict) | $24.0$           | $35.2$      | $29.6$      | $34.29$            | $56.73$     |\n| GPT (explain first)  | $43.5$           | $58.8$      | $51.2$      | $41.60$            | $69.02$     |\n\nWhile these primitive experiments have not involved further prompt engineering and techniques like chain of thoughts, they have shown that more efforts are needed for LLM to handle computational argumentation tasks like KPM and CESC. We hope that Hi-ArG can become one of these efforts in the future.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- A new structure to represent and store argumentations, which can facilitate future research on argumentation tasks.\n  \n- Two potential methods to use the structural information from Hi-ArG\n  \n- Intensive experiment to show the effectiveness of the proposed methods", "reasons_to_reject": "- The performance boost brought by the proposed method is not very significant,\n  \n- Besides, I do not see major weaknesses that prevent the paper from being accepted. Yet this paper can be improved by providing the following results:\n  \n  - evaluation on the correctness of machine-generated Hi-ArG\n  - the performance of LLMs such GPT3.5 (possibly combined with in-context learning) on the two evaluation tasks", "questions_to_authors": "", "paper_topic_and_main_contributions": "This paper proposes a new hierarchical structure Hi-ArG to represent argumentations. It can retain more semantics within arguments at the intra-argument level and record relations between arguments at the inter-argument level.\n\nTo exploit Hi-ArG, they propose several pretraining tasks to augment a text-graph model with the information from the argumentation graphs. Experiments on two argumentation tasks, Key Point Matching and Claim Extraction with Stance Classification, have shown that further pre-training and fine-tuning improve performance.", "excitement": 4, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "Vu2kQJNRhG", "reviewer": "Reviewer_TQxT", "summary": "", "questions": "1. What does \"sampled documents\" in line 343 refer to? I assume that in each training instance, there is only one document.\n\n2. What is the effect of link node? I can think of an alternative w/o link node to solve two issues mentioned in Section 3.3: for the \"disconnected\" case, we just connect top nodes to the Root node, and for the \"same top node\" case, we replicate the top nodes and intermediate nodes in sub-graph if needed, then connect top nodes to the Root node.", "limitations": "", "rating": 4, "confidence": 4, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank you for taking the time to review our submission and for your valuable feedback.\n\nIn response to your comments and questions, we would like to provide the following rebuttal:\n\n## Q1: Re-introduction of AMR and related concepts\n\nWe appreciate your advice on a clear introduction to AMR concepts. In our paper, we followed the original definitions of AMR graphs so that we could apply existing parsers to automated Hi-ArG construction; yet, in response to your comment, we will include essential information (text and figures if necessary) around AMR and related concepts in the revised version of our paper.\n\n## Q2: Clarity of section 3.3 and advantages of link nodes\n\nWe thank you for pointing out the potential confusion in section 3.3 and will add the necessary explanations for this section. Yet, we would like to emphasize the advantages of link nodes introduced in our paper.\n\nAs explained, link nodes are necessary to distinguish multiple appearances of a single argument (possibly with nuances) in the given text. While Hi-ArG does not store transitional information as the graph is context-independent, such contextual information could become important when cooperating with text data. In this case, link nodes can handle this information while leaving context-independent information (e.g., semantics and logic) to Hi-ArG nodes.\n\nAnother advantage of link nodes is efficiency. Link nodes could reduce the number of new nodes and edges, especially when the sub-graph referred to multiple times contains a lot of nodes and edges. They also allow the sharing of information through one common sub-graph.\n\nFinally, both using link nodes and copying nodes need an extra root node to ensure graph connectivity, so we suggest no difference in this part.\n\n## Q3: Necessity of inter-arg graphs\n\nWe understand your concern about the inter-arg level of Hi-ArG.\n\nDue to the nature of the two downstream tasks we selected, we did not include inter-arg graphs in our downstream experiments to avoid leaking information. However, we would like to propose two potential applications of inter-arg graphs not mentioned in this paper.\n\nFirst, inter-arg graphs can also participate in the pre-training stage for more general uses so that pre-trained models can learn logical relations between arguments. Furthermore, when applying our proposed pre-training method with relatives, we can sample the relatives of a given argument from both intra-arg and inter-arg levels of Hi-ArG as long as other constraints are satisfied.\n\nSecond, inter-arg graphs can provide clues for generative tasks like argument generation. These tasks require claims and evidence from multiple aspects, as well as multi-step induction/deduction among arguments. The Inter-arg graph is a proper source to retrieve such arguments and logical relations.\n\n## Q4: Comparison between Hi-ArG and AKG\n\nWe understand your comment concerning the comparison between Hi-ArG and AKG.\n\nBefore discussing this topic, we would like to clarify that GreaseLM is not the AKG counterpart of GreaseArG: GreaseLM is a model designed for questioning answering tasks, whose authors used ConceptNet and a self-constructed graph for their experiment but not AKG; in our paper, we modified the structure of GreaseLM for pre-training purposes, resulting in GreaseArG, which outputs text/node representations for general use.\n\nFor the main issue, because Hi-ArG and AKG have completely different structures, it would be better to compare these two graphs through downstream applications. However, we understand that AKG is currently not applied to the two downstream tasks we introduced (we only found applications on argument generation). Therefore, it requires novel methods to exploit AKG for certain tasks where Hi-ArG can also be applied to make a meaningful comparison possible. We would like to include this subject in future studies concerning more downstream applications of Hi-ArG.\n\n## Q5: Definition of \"sampled documents\"\n\nAs mentioned in the same section, a document consists of multiple sentences. When appearing in one training instance, these sentences form a continuous segment as we follow the sample construction method of RoBERTa.\n\nIn our experiments, a document is usually a debate speech or something similar, and most of the documents can fit in a single training instance. Therefore, when we mention \"sampled documents\" or other similar terms, we refer to the documents within the training instance, complete or split at the beginning/end.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "This paper introduces a brand-new way to represent text as knowledge graphs and use it in argumentation tasks, which opens a highly potential direction to apply it in various tasks, including generation tasks such as question answering and knowledge-grounded generation.\n\nThe data structure (general or adapted) and the construction of corresponding Hi-ArG knowledge graph is clearly described. The experimental setup, including the new model architecture, pretraining tasks, and data augmentation method is also clear.\n\nAlthough the improvement brought by Hi-ArG is not large and in fact it doesn't beat the SOTA task models, the improvement is significant over the same-scale models. Furthermore, the paper does extensively analyze the effect of each proposed components in the further-pretraining stage, help readers understand behaviors of the new model in evaluation tasks and the contribution of each proposed component.", "reasons_to_reject": "The building block AMR is not re-introduced in the paper with important notes of authors, making the paper hard to follow continuously and hard to link AMR to the concepts in section 3.1 and 3.3, e.g top nodes.\nAlso, although the description of method is clear, because text is inherently not the best way to describe a structure, the lack of reference between Section 3.3 and Figure 3 makes it harder to understand how to add the link nodes.\nNonetheless, the reason \"multiple sentences in one text can point to the same top node, which could confuse each other\" to introduce link node is not clearly justified.\n\nThe paper does introduce the inter-arg level of Hi-ArG, but never construct Hi-ArG with it or use it in pretraining/inference. That confuses readers about the necessity of the inter-arg level of Hi-ArG, at least in the scope of this paper.\n\nSince the paper focus on Hi-ArG, readers expect to see the comparison between Hi-ArG and AKG. However, the paper does not show the experiment result of GreaseLM - which is the AKG counterpart of GreaseArG - on the two evaluation tasks. Thus, it's unclear about the superiority of Hi-ArG over AKG.", "questions_to_authors": "1. What does \"sampled documents\" in line 343 refer to? I assume that in each training instance, there is only one document.\n\n2. What is the effect of link node? I can think of an alternative w/o link node to solve two issues mentioned in Section 3.3: for the \"disconnected\" case, we just connect top nodes to the Root node, and for the \"same top node\" case, we replicate the top nodes and intermediate nodes in sub-graph if needed, then connect top nodes to the Root node.", "paper_topic_and_main_contributions": "Problem: This paper deals with the problem of incorporating knowledge graph in argumentation tasks, including Key Point Matching and Claim Extraction with Stance Classification.\n\nContribution:\n1. It proposes Hi-ArG, a graph structure based on Abstract Meaning Representation (AMR) which is better than well-known Argumentation Knowledge Graph to retain both intra-level and inter-level info of arguments in documents and the whole corpus.\n2. It exploits the new structure in two ways, model architecture and data augmentation. In term of model architecture, they modify GreaseLM to form GreaseArG model which facilitates the use of Hi-ArG in the GNN component of the model, as long as propose a new set of further-pretraining graph tasks for GreaseArG. In term of data augmentation, they utilize graph structure of Hi-ArG to find text relevant to input documents (called \"relatives\") which facilitates another pre-training objective for GreaseArG.\n3. It conducts experiments, including constructing Hi-ArG graphs for further-pretraining/evaluation and training GreaseArG. The results show the improvement over same-scale models on the considered evaluation tasks.", "excitement": 4, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Missing_References": "n/a", "Typos_Grammar_Style_and_Presentation_Improvements": "n/a"}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper proposes a new hierarchical structure Hi-ArG to represent argumentations. It can retain more semantics within arguments at the intra-argument level and record relations between arguments at the inter-argument level. To exploit Hi-ArG, they propose several pre-training tasks to augment a text-graph model with the information from the argumentation graphs. Experiments on two argumentation tasks, Key Point Matching and Claim Extraction with Stance Classification, have shown that further pre-training and fine-tuning improve performance.

In general, the authors found the proposed Hi-ArG to be an interesting method, the claim that it retains more semantics within the arguments at the intra-/inter-argument level is well-supported and the intensive experiment shows the effectiveness of proposed methods and the effects of each proposed component.

The more significant reasons to reject seem to be related to clarification and requests for additional results.
* Requests for clarification: the rebuttal provided a decent expansion on the issues brought up by reviewers.
* Requests for additional results: the authors provided additional results for one request (results on LLMs on the two evaluation tasks), however other requests were deferred to future work.
Decision: Accept-Main