Constituency Tree Representation for Argument Unit Recognition
Authors: Samuel Guilluy
Florian Méhats
Billal Chouli
Venue: ICLR 2022 Submitted (2022)
Keywords: transformer
attention
bert
graph attention network
constituency parsing
deep learning

Abstract: The extraction of arguments from sentences is usually studied by considering only the neighbourhood dependencies of words. Such a representation does not rely on the syntactic structure of the sentence and can lead to poor results especially in languages where grammatical categories are scattered in the sentence. In this paper, we investigate the advantages of using a constituency tree representation of sentences for argument discourse unit (ADU) prediction. We demonstrate that the constituency structure is more powerful than simple linear dependencies between neighbouring words in the sentence. Our work was organised as follows: First, we compare the maximum depth allowed for our constituency trees. This first step allows us to choose an optimal maximum depth. Secondly, we combine this structure with graph neural networks, which are very successful in neural network tasks. Finally, we evaluate the benefits of adding a conditional random field to model global dependencies between labels, given local dependency rules. We improve the current best models for argument unit recognition at token level and also present an explainability method to evaluate the suitability of our model architecture.

Under review as a conference paper at ICLR 2022
CONSTITUENCY TREE REPRESENTATION FOR ARGU- MENT UNIT RECOGNITION
Anonymous authors Paper under double-blind review
ABSTRACT
The extraction of arguments from sentences is usually studied by considering only the neighbourhood dependencies of words. Such a representation does not rely on the syntactic structure of the sentence and can lead to poor results especially in languages where grammatical categories are scattered in the sentence. In this pa- per, we investigate the advantages of using a constituency tree representation of sentences for argument discourse unit (ADU) prediction. We demonstrate that the constituency structure is more powerful than simple linear dependencies between neighbouring words in the sentence, Our work was organised as follows: First, we compare the maximum depth allowed for our constituency trees. This first step allows us to choose an optimal maximum depth. Secondly, we combine this structure with graph neural networks, which are very successful in neural network tasks. Finally, we evaluate the benefits of adding a conditional random field to model global dependencies between labels, given local dependency rules. We im- prove the current best models for argument unit recognition at token level and also use explainability methods evaluate the suitability of our model architecture.
INTRODUCTION
Argument Identification inside documents is the first step to study rhetorical speech processes or political debates. It is conerete NLP application that interests the media community during the electoral periods.
The goal is precisely identify arguments discourse unit (ADU) (defined the minimal analysis units) in sentence and predict its stance, given topic.
Former works on argument analysis at the token level Trautmann (0220) has used language models such as the Bidirectional Encoder Representations from Transformers model (BERT) Devlin et al. (2099, associated to probabilistic models such as Conditional Random Field (CRF) Lafferty et al. 2001.. This combination enables to increase the global prediction coherence with restricted fine- tuning.
Language models are essential to incorporate statistical knowledge without having to train huge datasets. However, these models were recently criticized for their lack transparency on how training data are reused in predictions Bender al. (2021). Thus, the creation of collaborations between linguists and data researchers into collaborative project Gauthier et al. (2020) are an actual research topic which aims to create more linguistic oriented language models.
In this work, we study the impact of incorporating grammatical structure into the BERT-CRF model. We focus on the constituency tree representation of the sentence. This representation is composed of internal nodes (also called NT for Non Terminal) representing the grammatical structure and leaf nodes (nodes without children) corresponding the sentence words.
The main goals this paper are
evaluate GNNs models combined with CRF layer in order use the syntactic infor- mation contained in the constituency tree representation;
'The code will be publicly available on GitHub after the blind review process. It contains download script of the dataset, the model code, the interpretability models and the hyper parameter optimization scripts.

---

Under review as a conference paper at ICLR 2022
to use interpretability models to evaluate the contribution of the tree structure the final results.
RELATED WORKS
In this section, related works about Argument Mining and tree structure representation in NLP, Graph Neural Network and Conditional Random Field reviewed.
ARGUMENT MINING
The precise definition of the concept of argument is an important step when creating dataset anno- tation rules. A previous work on Argument Mining Lawrence & Reed (2020) has identified two main scales of argument units: Elementary Discourse Units (HDUss) which are the non-overlapping text spans corresponding to the atomic discourse units, and Argumentative Discourse Units (ADUs), defined as minimal analysis units. As explained in Peldszus & Stede (2013). "when two EDUs are joined by some coherence relation that irrelevant for argumentation, the resulting complex might be the better ADU". With this definition, one notice that EDUs and ADUs not defined the sentence level. Thus it is more relevant to use a dataset labeled at the token level instead of sentence level. The current biggest dataset with argument unit ADUs labels the token level is AURC Trautmann et al. (202)).
TREE STRUCTURE REPRESENTATION IN NLP
The goal of using a tree representation of sentence is to exploit the syntactic hierarchical de- pendencies of the sentence with graph neural network. Thus, the constituency tree representation is more appropriate than POS taggers which provide one tag per token (according to the Univer- sal POS tags categories). We prefer the constituency parsing over the dependency parsing because dependency grammars phrasal constituents and phrase-structure rules do not play direct role in dependency parsing. Thus, the distance (in terms of number of edges of the tree) between words of the same grammatical class is longer in the dependency parsing of the sentence.
Pioneer work (Gildea Palmer 2002
presenting the benefits of using constituency tree represen- tation has failed to scale into production. This is caused by the lack of a reliable model to generate constituency tree representation of the sentences. However, recent promising results were made in consistency and dependency parsing, Zhhang (2020)
For our preprocessing step we used a neural network model called Berkeley Neural Parser (BENEPAR) (Kitaev & Klein 2018), which is trained on 11 languages. We use this model with the weights provided by these designers. Other possibilities for this task are the StanfordParser with Stanza Qi et al. (2020) NLTK We chose BENEPAR because it is integrated with Spacy and it different languages.
can be used
Other papers have recently studied the use of tree structure to incorporate syntactic information to their models. Marcheggiani Titov| (017)) uses Graph Convolutional Networks (GCNs) based on the dependency tree structure of the sentence for semantic role labeling. Beck et al. (2018) uses GNNs for generation tasks from abstract meaning representation.
GRAPH NEURAL NETWORK
Graph Neural Networks refer to the various methods which extend Deep Learning data organized in graphs. GNNs were introduced by Gori et al. (2005) as an extension of Recursive Neural Net- works (RNN). GNNs use a neural network to propagate information within the dataset geometrical structure.
In this article, the Graph ATtention (GAT) layer Veličković et al. (2018) which appears to be the most adapted message passing layer to propagate information following a self-attention strategy is used. The attention mechanism has been widely used in NLP tasks since good results translation tasks presented in Vaswani et al. 2017 The main benefits of attention mechanisms are that they
Thhe dataset publicly available wttptttt///gghg
ctm/ http//ttmm/aeee AURC
2

---

Under review as a conference paper at ICLR 2022
allow to deal with variable size imputs and enable to specify different weights to different neighbours focusing on the most relevant parts the input to make decisions.
CONDITIONAL RANDOM FIELD
Conditional Random Fields were introduced by Hammersley & Clifford (1971) as probabilisticc models for magnetic field modeling. Their goal was to exploit the Markov assumption of lattice to derive the log probability of the label distribution formula. Since then, CRFs have been widely used in machine learning thanks to the iterative scaling algorithms presented by Lafferty et al. (2001) to find the parameter that maximizes the log-likelihood of the training data. Many interactions have been made between CRFs and GNNs. Gao et al. (2019) proposed a new CRF layer for GCNs, Cui & Zhang (2019) proposed neural network alternative CRFs motivated by the fact that the Markov assumption is not always respected for linear chain representation of the sentence.
MODEL PRESENTATION
3.1
BERT & CRF
This model has been introduced by Trautmann et al. (2020). It is composed of two steps. In the first step, the sentence is tokenized following the BERT tokenizer and the BERT model is fine-tuned for token classification, where the output of the last layer matches the number of classes of the dataset. the second step, a linear chain Conditional Random Field is applied to estimate the probability of each label class from BERT.
The main idea of this model leverage the BERT attention knowledge and then to improve the results by incorporating linear chain dependency structure. This takes advantage of neighbours dependency relations between words.
The good results of this model lead to use it as competitive benchmark for approach based on constituency trees as input representations of sentences.
3.2 BERT, GNN & CRF
A major difficulty in choosing graph neural layer is that each sentence has a different tree represen- tation. Hence, the model needs be agnostic the lack of completeness the tree structures from the training set. The message passing design enables to share the model weights among the network nodes, thus the results do not depend on the upfront global tree structure access. also enables to deal with directed and undirected graphs as the information flow is conditioned to the existence of directed edge between two nodes.
GRAPH ATTENTION LAYER
The Graph Attention Layer Veličković et al. (2018) allows to combine the attention mechanism with the graph structure, preserving the syntactic structural information of the sentence. In order to improve the model stability, adding multi-head attention layers might be beneficial to the training step. The different heads are then aggregated in order to provide the next hidden states the neural network. As opposed to Graph Convolution Layer, this layer allows to assign different importances to nodes of same neighbourhood. Furthermore, analyzing the learned attentional weights may benefit to the model interpretability,
extract information from attention weight matrices.
In order to take advantage the dependency structure of the sentence, we incorporate GNN between the use of BERT and the linear chain CRF.
The idea behind this architecture the following. First, the BERT language model outputs the sentence hidden representation. Next, the information is spread the graph neighbours at each iteration. In that way, we expect to reach a better consistency between neighbour nodes when we train restricted dataset.
3

---

Under review as a conference paper at ICLR 2022
4 EXPERIMENTAL SETUP
4.1 DATA SOURCE AND ANNOTATION RULES
The dataset used in study has been introduced by Trautmann et al. (2020). This dataset appears to be one of the few datasets available that annotates argument unit at the token level. It is composed of 8000 sentences, equally divided into 8 topics, and can be used for the task of argument unit identification well as argument polarity identification.
The authors distinguished between PRO (supporting), CON (opposing) arguments and NON (non- argumentative) words.
In the cross domain setting, the topics in the train dataset are not presented in the test dataset. In the in domain setting, the topics in the train dataset are also present (with other test sentences) inside the test dataset. This enables to evaluate how important the domain knowledge are when identifying ADU. In our preprocessing step, we need to label the internal nodes NT of the constituency tree in order learn their representation. As these annotations are not provided by the dataset, we have chosen to annotate the internal nodes with the same labeling rules the one described above for the sentences labeling. This enables to preserve the internal logic of the labeling process.
4.2 CONSTITUENCY TREE CONSTRUCTION
One of the main advantages of adding a constituency tree to classical methods is the greater proxim- ity of the words which belong the same grammatical class than the words which only neigh- bours in a linear representation of the sentence. We can also illustrate this fact with the constituency tree presented in Figure [ For this sentence, it can be seen that the ADU units are distributed fol- lowing the grammatical structure of the sentence. For instance, the words "inflation" and "the" are neighbours in the sentence but are further in the constituency tree structure. This leads better identification of boundaries between ADUs.
However
SBAR
when
the
now
federal
ADUP
wage
Itrough
most
partod
from
1961
Figure 1: Constituency tree representation of the sentence, according to the Universal POS tags categories (with a maximum depth of 3): "However, when adjusted for inflation, the new federal minimum is still less than the minimum wage through most of the period from 1961 to 1981." about the topic "minimum wage". The green nodes represents words or spans with "PRO" label and the grey nodes represents words and spans with "NON" label.
For the construction of the constituency tree, we use Berkeley Neural Parser (BENEPAR) Kitaev & Klein (2018). This is multilingual constituency parser which benefits from unsupervised pre- training across variety of languages. This model provides the weights for 12 languages and simple API incorporated to Spacy, and thus a good choice in order to have a good accuracy, integrated into larger pipeline. This additional pre-processing step requires to have an exact match between the tokenizations used by BENEPAR and BERT models. This constraint imposed us to remove the unmatched sentences. In addition, the samples from the in-domain test set were also excluded in the cross-domain train and dev sets. As result, there were 4000 samples reduced to 3960 in train, 800 reduced to 790 in dev and 2000 in test reduced 1959 for the cross- doms-domain split; and 4200 reduced to 4157 in train, 600 reduced to 593 dev and 1200 reduced to 1159 in test for the in-domain split, At the end, less than 2 % of the dataset removed.
4

---

Under review as a conference paper at ICLR 2022
Maximal depth the tree
No tree
3
4
Category
Neighbour similarity
97.1
98.3%
98.4%
98.6%
Similarity of grammatical classes and words
90.2%
91.3%
84.2%
Similarity of grammatical classes
88.5%
93.3%
92.9%
Table 1: Evaluation of three proportions to assess the appropriateness of using the constituency tree representation. The computation was performed on the train dataset under the cross-domain setting.
4.3 EVALUATION OF THE NODE SIMILARITY
One of the main advantages of adding a constituency tree to argument identification methods is the greater proximity of words that belong the same grammatical class. Indeed, noticed that words belonging to the same grammatical class have more often the same label than words that are only neighbours in the sense of the linear representation of the sentence.
However, when the tree is too deep, the distance between words of the same grammatical class might be longer than between words of different grammatical classes. This lead us to cap the maximal tree depth allowed.
To measure this fact, we compute three statistics about the proximity of the nodes inside con- stituency tree and distinguish the results regarding to the maximal depth accepted for the con- stituency tree. The three proportions computed, summarized in Table are the following.
Neighbour similarity: the proportion of nodes which have the same label and are neigh- bours in the linear chain sentence representation. For the cases (columns 2, 4) where we have also a constituency tree representation, we additionally restrict this set to the nodes which share the same grammatical class (i.e. which have the same parent node).
Grammatical class and word similarity: the proportion of leaf nodes which have the same label as their grammatical class.
Grammatical class similarity: the proportion of internal nodes which are linked by an edge and share the same label.
Table [ shows a stronger proximity between neighbour words the same grammatical class than neighbours words when the grammatical structure is not taken into account. We can also notice that the constituency tree with a maximal depth shows a stronger node similarity than the two others. However, is not necessarily the best depth as the constituency tree with maximal depth of 3 shows better results concerning the grammatical class similarity. This lead us to try our models with the maximum depth of 3 and 4.
4.4 EVALUATION SCORE
In this study, we use the Fi-score the evaluation metric of the models. The Fi-score refers the equally weighted harmonic mean of the precision and enables to take into account the precision and recall.
There are two ways for computing the Fl-score. The micro average calculates metrics globally by counting the total true positives, false negatives and false positives. The maero average calculates metrics for each label and finds their unweighted mean. This does not take label imbalance into count. Therefore, the micro Fl-score is preferable for our studies as the dataset is inss-immllanced regarding the non-argument label NON at the token level.
The F1-score is computed at the token level the BERT token. Since some words are split into two subtokens with the BERT tokenizer (in order separate suffixes), accuracy and recall evaluations are slightly biased. One way to minimize this effect is to merge the labels of the subtokens of word that has been split.
5

---

Under review as a conference paper at ICLR 2022
5 MODELS RESULTS
5.1 HYPERPARAMETER, OPTIMIZATION
The BERT-GAT-CRF model have many more hyperparameters than the BERT-CRF model. This is due to the large amount of hyperparameters of GAT network: number of layer, units per layer, number of heads.
In order to choose the best hyperparameters for this model, we use the library Optuma (Akiba et al., 2019). Optuna is framework for cost-effectiveness of hyperparameters optimization. The two main advantages of Optuna for our study are the following.
Searching strategy: Optuna implements a relational sampling method that can identify the underlying concurrence relations after some independent samplings.
Pruning strategy: regarding the intermediate F1 score values, Optuna interrupts the non promising trials based on the Successive Halving algorithm Li et al. (2020), which contin- ues the trials if only the actual F1-scores is among the best intermediate results.
The importance of the different hyperparameters of our model has been evaluated in Table 2 It can be observed that the most important hyperparameters are the learning rate and the maximum allowed value for the gradient. From empirical point of view, we notice that when we do not constrain the gradient, the model converges to a local optimum which the assignment of the label NO to each word. This local optimum exists because the dataset is unbalanced towards the absence of an argument.
Test Intervals
Best values
Parameters importance
Learning rate
10 to 10-3
2.8 10=b
30
Maximum gradient allowed
10 to 102
9.7
49 %
Number of GAT layers
1 to 3
2
2 %
Number of unit per GAT layers
to 300
290 and 100
Number of heads per GAT layers
1 to
3 and 3
7 %
Number linear layers
1 to 3
Number of unit per linear layers
50 to 250
100 and 100
%
Table 2: Feature importance of the BERT-GAT-CRF model in Cross Domain prediction
One difficulty encountered during the development of this model was the management of batches. In Pytorch Geometric, we associate graph with each input. This graph is represented by two- dimensional matrix that contains the edges between the nodes. Thus, in order to allow different sizes of graphs, the batches matrix used in Pytorch is unfolded into list and list of indices links the input data to the batch indices. Thus, as we have to switch between the two representations order to use BERT and GNNs within model, this reduces the computational speed.
The global training step requires loading the BERT model and a batch of data into memory. Accord- ing our implementation, this requires 30 GB of memory (GPU CPU). We therefore performed our global training experiments on the CPU, which took about 2 hours per iteration and led to about 30 iterations per model configuration. The transfer learning model training required about 7 GB and could therefore be run on the GPU, We therefore had about 200 trials per model optimisation.
5.2 MODELS EVALUATION
Consider first the performance of the models when the BERT model is not fine tuned. In this case, the model BERT + CRF shows poor result: 31% Fl score probably due to the low number of parameters (we only train here the last layer of BERT and the parameters of the CRF). The BERT+GNN+CRB model reaches 50% of F1 score on the Cross Domain dataset.
In second step, performed global training of the models. In the initial paper from Trautmann et al. (2020). the following metrics were presented token level, span level and sentence level. Our model focuses improving the argument border recognition and not the argument stance

---

Under review as a conference paper at ICLR 2022
In Domain
Cross Domain
BERT large
68 %
59 %
BERT large- Linear chain CRF
69 %
61 %
BERT large GAT CRF Maximum depth allowed for Con-
72.8 %
68.2 %
stituency tree is
BERT large GAT CRF Maximum depth allowed for Con-
73.5 %
68.7 %
stituency tree is 4
Table 3: FI- score of the different models at token level on the test dataset (the results with star (*) have been directly taken from (Trautmann et al. 2020
identification. Thus, it improves the state of the art the token level (but achieves similar results on the sentence level and the span level).
The results corresponding the token level classification are presented in Table +] As suggested by Table [ we computed our models with a maximum depth of and 4. We reached our best results with the model composed of BERT-GNN CRF when we allowed maximal depth of 4. This achieves 74,1 1 % on token level F1 score miero in cross-domain setting and 73.8 % on token level F1 score micro for the in-domain setup.
5.3 MODEL INTERPRETABILIT
In this subsection, we study the interpretability of the model BERT-GNN-CRE
Consider the problem of attributing the model prediction its input features. First, we use the Integrated Gradient method (Sundararajan et al. 2017) which computes a score for every input of the model compared the input representation from the baseline to our input. When this gradient is higher, our input data more relevant for prediction. In the BERT model, this baseline is the token "[PAD]" (commonly used for padding) which indicates the absence of word.
baseline value. The score calculated by integrating the gradient along
We used the Captum framework (oohlikkann interpretability models. Figure 2 output. In this example, it can be seen that the word contributions follow the grammatical structure (expect for the word "then", due to the fact that labeling did not follow strictly the grammar).
al., 2020) which implements a wide variety of presents the contribution of the words of the sentence to the final
NT NT NT NT NT you have fear ofloosing your important data to hard drive failure, then you can use this drive clone tool to clone your mac hard drive or volume and restore the lost data in case of media failure
Figure 2: Word Importance of the Integrated Gradient Method for the sentence: you have fear of losing your important data to hard drive failure, then you can use this drive clone tool to clone your mac hard drive or volume and restore the lost data, in case of media failure.". " The "NT" token refers to the sentence internal nodes. The words highlighted in green (resp. red) have positive (resp. negative) impact on the final outputs.
Then, we study the importance of the edges for the model prediction. The Integrated Gradient method cannot be directly applied our graph study as we cannot integrate through the edges weights (the edges not weighted). We compute a simple algorithm to measure the importance of the different edges of our graph. We iteratively hide one edge from the graph and compute the measure the necessity of
model to see how the result varies. By doing with all the edges every edges taken independently. The algorithm is described the Algorithm T
The results of the interpretability edges be found in Figure It globally shows that the words which are labeled with an argument (in color) furnish more information to their related edges. This confirm the utility of adding the constituency representation to model.

---

Under review as a conference paper at ICLR 2022
Algorithm Measure of the impact of the graph edges.
Input: sentence analyse, The edges of the constituency tree of the sentence, The labels of the sentence
Output: matrix containing the importance weight of each edge
1: Run the model on the input sentence
2: for Every edges of the graph do
3: Remove one edge of the graph
4: Run the model with the new input
5: Compute difference between the previous loss and the new loss
6: Store this value as the importance of the edge
7: end for
Figure 3:
Tree representation the edges interpretability methods. The red edges refer to edges which have positive impact on the prediction. The green edges refer to edges which have negative impact on the prediction. (This figure has been generated by NetworkX)
6 CONCLUSION
In this study, we established a new method for identifying the ADU boundary using the sentence constituent representation. This model allows the information be spread across the graph and performs well on a small dataset.
The main errors previously observed this dataset were:
the span of an argumentative segments was not correctly recognized.
the stances are not correctly classified.
In this paper, we focus on the span detection problem and improve the method for identifying the boundaries of an ADU. However, we do not have strict adherence grammatical correctness. As mentioned in Trautmann et al.) (2020), this would require that all spans containing arguments be clauses. There are examples sentences where subjects are absent, which limits the performance of models using grammatical structure. This problem could be solved by specifying annotation rules that more strictly follow the syntactic structure of the sentence.
Regarding the second type of error, the problem of position identification is mainly related to the performance of the BERT model. The dataset only provides sentences with a maximum length of 64. This implies that only narrow context of the ADU is available, which limits the capability of our model. Many arguments require more domain knowledge to understand what is at stake.
They model arguments self-contained pieces of information that can be verified relevant argu- ments for a given topic with no or minimal surrounding context.
A disadvantage of using message- message-passing GNNs is that, since not all words have the same depth in tree representations, message passing is treated identically across words. A word with shallow depth will pass information to much more distant part of the tree than node with greater depth.
In future work, we will explore other GNNs to overcome this problem. We plan to continue our study argument recognition by working datasets with more surrounding context.
8

---

Under review as a conference paper at ICLR 2022
REFERENCES
Takuya Akiba, Shotaro Sano, Toshihiko Yanase, Takeru Ohta, and Masanori Koyama. Optuna: A Next-generation Hyperparameter Optimization Framework. In Proceedings of the 25th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining, KDD '19, pp. 2623-2631, New York. NY, USA. July 2019. Association for Computing Machinery. ISBN doi: 10.1 10.1145/22225003333701. URL https::
1145/
0222500033307700
Daniel Beck, Gholamreza Haffari, and Trevor Cohn. Graph-to-Sequence Learning using Gated Graph Neural Networks. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume I: Long Papers), pp. 273-283. Melbourne, Australia, July 2018. Association for Computational Linguistics. doi: 10. 10.18663//1/118-11 1026.
URL https:
7/www. wwwweb. org/ wttp://///nnnnooo///-----1-0
Emily M. Bender, Timnit Gebru, Angelina McMillan-Majon and Shmargaret Shmitchell. On the
Dangers of Stochastic Parrots: Can Language Models Be Too Big? In Proceedings of the 202/ ACM Conference on Fairness, Accountability, and Transparency, pp. 610-623, Virtual Event Canada, March 2021. ACM. ISBN 978- 0-015-503--3090 doi: (0.114453442188.44444
URL
https:///l acm. wwtpw//wg.oooooooooootoo 1ttp://1..1115334420 24444444.888. 3445922
Leyang Cui and Yue Zhang.
Saaerehiicaly-RReiiee Label Attention Network for Sequence Label- ing. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Pro- cessing and the 9th International Joint Conference on Natural Language Processing (EMNLP- IJCNLP), pp. 4115-4128, Hong Kong, China, November 2019. Association for Computa- tional Linguistics. doi: 10. 10.18653/1//11--1422.
URL https:://aaee htaps:://aclathooooyy. orr//
D19-1422
Jacob Devlin, Ming- Wei Chang, Kenton Lee, and Kristina Toutanova. BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding. In Proceedings of the 2019 Confer- ence of the North American Chapter of the Association for Computational Linguisties: Human Language Technologies. Volume I (Long and Short Papers), pp. 4171 Minneapolis, Min- nesota, June 2019. Association for Computational Linguistics. doi: 10.18653/11/N19-11 1423. URL httpss://cct antho ogy.
Hongchang Gao, Jian Pei, and Heng Huang. Conditional Random Field Enhanced Graph Con- volutional Neural Networks. In Proceedings of the 25th ACM SIGKDD International Con- ference on Knowledge Discovery & Data Mining, KDD '19, pp. 276-2844 New York, NY, USA, July 2019. Association for Computing Machinery. ISBN 978-1-4503-6201-6 doi: 10. 10.1145/3292500.33330
URL httphhttp///ooii. wwt.rorggg.oo....
Jon Gauthier, Jennifer Hu, Ethan Wilcox, Peng Qian, and Roger Levy. SyntaxGym: An Online Plat- form for Targeted Evaluation of Language Models. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics: System Demonstrations, pp. 70-76, Online, July 2020. Association for Computational Linguistics. doi: (0.186653/1/2002..lddddemmmm
URL
https://ace antaanaoooooo oogy. wrgg//20, awt.:o...llcce adwl-.eoss...... 10
Daniel Gildea and Martha Palmer. The Necessity of Parsing for Predicate Argument Recognition. In Proceedings of the 40th Annual Meeting of the Association for Computational Linguistics, pp- 239-246, Philadelphia, Pennsylvania, USA, July 2002. Association for Computational Linguis- URL hwtpt:ttt//..mm http::::s//caaaatthhhoto
tics. doi:
wwtp/02-1110000 1031
M. Gori, G. Monfardini, and F. Scarselli. A new model for learning in graph domains. In Pro- ceedings. 2005 IEEE International Joint Conference on Neural Networks, 2005., volume 2, pp- 729-734 vol. July 2005, doi: 10. N0.10091CCNN.000.555550
ISSN: 2161-4407
J.M Hammersley and P Clifford. Markov Fields On Finite Graphs And Lattices. 1971. URL
htttp://ww.st... http::/www..aaalaaa.aaa..ca..
lab. cam. wwwp:u///ggggggggogog/eoo h estt /hamm cl iff .dff
Nikita Kitaev and Dan Klein. Constituency Parsing with a Self-Attentive Encoder. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 2676-2686, Melbourne, Australia, July 2018, Association for Computational Lin- guisties. doi: 10. 00.18653/v1/P11 1249. URL. httpstss:: / acl anthol anthelogy.oor ogw.org/p18-1145

---

Under review as a conference paper at ICLR 2022
Narine Kokhlikyan, Vivek Miglani, Miguel Martin, Edward Wang, Bilal Alsallakh, Jonathan Reynolds, Alexander Melnikov, Natalia Kliushkina, Carlos Araya, Siqi Yan, and Orion Reblitz- Richardson. Captum: A unified and generic model interpretability library for PyTorch. arXiv:2009.0788 fcs, stat), September 2020. URL http:://axilv.orgabbss..
07896
arXiv: 2009.07896.
John D. Lafferty, Andrew McCallum, and Fernando C. N. Pereira. Conditional Random Fields: Probabilistic Models for Segmenting and Labeling Sequence Data. In Proceedings of the Eigh-
teenth International Conference on Machine Learning, ICML '01, pp. 282-289. San Francisco, CA, USA, June 2001. Morgan Kaufmann Publishers Inc. ISBN 778-1-55860-778-1
John Lawrence and Chris Reed. Argument Mining: A Survey. Computational Linguistics, 45 (4):765 818, January 2020. ISSN 0891-2017, 1530-9312. doi: 1221166l URL wew11116ooll 00364
https:/ www
org. ottp:/aaaa..rg/////////////////a
Liam Li, Kevin Jamieson, Afshin Rostamizadeh, Ekaterina Gonina, Jonathan Ben-tzur, System for Massively Par- allel Hyperparameter Tuning. Proceedings of Machine Learning and Systems, 2:230 246, March 2020. URL /ttps::/proceeddiii ngs www.gmlmss.......moo............ mlsys. orgg/paper/222///att/ FBbSecc/Da0FFff555000099368
Moritz Hardt, Benjamin Recht, and Ameet Talwalkar.
rast html
Diego Marcheggiani and Ivan Titov. Encoding Sentences with Graph Convolutional Networks for
Semantic Role Labeling. In Proceedings of the 2017 Conference on Empirical Methods in Natu- ral Language Processing, pp. 1506-1515, Copenhagen, Denmark, September 2017. Association for Computational Linguistics. doi: 10.18653/1/D117-115555 htthstss::/ www.www...m aclweb.
erc/antthology/117--1
Andreas Peldszus and Manfred Stede. From Argument Diagrams Argumentation Mining in Texts: A Survey. International Journal of Cognitive Informatics and Natural Intelligence, 7(1):1-31, January 2013. ISSN 1557-3958. doi: 201008//iiii.22111111111 URL (Rthhsts:///oo org/
10. 00400/35e11 2013010101
Peng Qi, Yuhao Zhang, Yuhui Zhang, Jason Bolton, and Christopher D. Manning. Stanza: A Python Natural Language Processing Toolkit for Many Human Languages. arXiv:2003.070 07002 [cs]. April 2020. http:
2003. C7082
Xrv:: 2003.07082.
Mukund Sundararajan, Ankur Taly, and Qiqi Yan. Axiomatic Attribution for Deep Networks. In International Conference on Machine Learning, pp. 3319-3328. PMLR, July 2017. URL/http: //proceedings waagggsmmll...o mwr. www.speeessseee /v00 http////nnnaaaaaaaaaaaaaaaaa html ISSN: 2640-3498.
Dietrich Trautmann. Aspect-Based Argument Mining. In Proceedings of the 7th Workshop on Ar- gument Mining, pp. 41-52, Online, December 2020. Association for Computational Linguistics. URL
org/ 2020 www.m..n......n..........
Dietrich Trautmann, Johannes Daxenberger, Christian Stab, Hinrich Schütze, and Iryna Gurevych. Fine-Grained Argument Unit Recognition and Classification. Proceedings of the AAAI Con- ference on Artificial Intelligence, 34(05) 9056, April 2020. ISSN 2374-3468. doi: 10. 1609/aaai, v34i05.643 66438. URL https:: aaai. org/ index. wwtpp/ppp///ttaaiill/aaa Number: 05.
view/ 6438
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Fukasz. Kaiser, and Illia Polosukhin. Attention is All you Need. Guyon, U. V. Luxburg, S. Bengio. H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett (eds.), Advances in Neu- ral Information Processing Systems 30, pp. 5998-6008 Curran Associates, Inc., 2017. URL pdf
Petar Veličković, Guillem Cucurull, Arantxa Casanova, Adriana Romero, Pietro Liò, and Yoshua Bengio. Graph Attention Networks. February 2018. URL httpp://gpeneeview.nnest
forum? JXM kCZ
Meishan Zhang. A Survey of Pentactic-Semantee Parsing Based on Constituent and Dependency Structures. arXiv:2006.1 11056 [cs], June 2020. URL http://axxiv.org/ www.xiv..rg/abb.... orw.s//000/bb//000.. 11056 arXiv: 2006.1 1106..
10


Reviews:
[{"review_id": "d2wHyNbErIt", "reviewer": "Reviewer_sFFn", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The method used in this paper is more alike a connection of three existing methods, bert, gat, and crf. Currently, could find little novel methods other than bert-gat-crf in this paper. The target questions is rather limited to a specific NLP task of semantic parsing field. Also semantic role labeling and related baselines/datasets should be involved for some direct comparison of this “argument unit recognition” task. \n\n\n\n\nDetailed questions and comments:\n1.\tFigure 1, why use 0 to 6 as higher level nodes? I think using POS-tags and phrase-tags are more readable, such as article tag for “The”, verb tag for “is” and noun phrase for “the time”.\n2.\tPrefer to see detailed examples that are improved by this bert-gat-crf method for intuitive understanding.\n3.\tSome cites can be improved, such as the bottom lines in page 5. With Roberta liu et al. (2019)? -> Roberta (liu et al. 2019).\n4.\tWhat are the reasons of not using dependency trees and give a comparison as well for this task? Dependency tree structure are shallow parsing and can be understood easily and adapted for semantic role labeling of argument unit detection and also predicate-argument structure prediction as well.\n", "summary_of_the_paper": "\nThis paper presents a bert-gat-crf framework for recognizing argument units in sentences through constituency tree representation. Examples of what is a consistency tree of short and long sequences are given. Experiments on small datasets proved the performance of the bert-gat-crf algorithm.\n", "summary_of_the_review": "Strong: \n1.\tUsing of bert-gat-crf for argument unit detection and achieved improvements in small-scale datsets;\n\nWeak:\n1.\tThe method of bert-gat-crf is less novel for this paper;\n2.\tThe target question is a pure NLP task and also related to other tasks such as semantic role labeling that better mention and compare with their baselines as well.\n", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "ckwjrp1CcG", "reviewer": "Reviewer_cyhs", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "\nSome comments:\n\nThe concept of Argument from Sentences is not properly described until Section 2, first paragraph. The entire abstract and introduction you are wondering what the paper will be about. Also, the description of Argument mining will be much better with an example.\nthe abstract starts too ambiguous. It is not even clear what \"arguments\" from sentences are and why neighboring words are relevant.\n\n- \"However, these models were recently criticized for their lack of transparency on how training data are reused in predictions.\" \nThis claim requires a citation.\n\n- \"However, these models were recently criticized for their lack of transparency on how training data are reused in predictions. Thus, model architectures based on linguistic concepts seem o be needed to improve current models and increase their explainability. \" \nWhy are you reaching this conclusion? The two sentences seems a bit disconnected.\n\n- \"the most commonly used models do not do not take into account the grammatical structure of sentences, which leads to a gap between current NLP models and linguistic analysis.\" \nThere is in fact a lot of work on showing how NLP models are able to understand linguistic structure, this is one place to start: https://aclanthology.org/N19-1419.pdf\n\n-\"This is caused by the lack of a reliable model to generate constituency tree representation of the sentences. However, recent promising results were made in consistency and dependency parsing\" \nAgain, these require citations. The first statement and the second one, both of them. \n\nThe related work section contains technical details about your experimental setup. It is hard to follow. \n\n\"The attention mechanism was introduced by Vaswani et al. (2017) for translation tasks in NLP and has since been widely used in many other tasks. \" The attention mechanism was not introduced by Vaswani et al. (2017), I would recommend to thoroughly read Vaswani et al. (2017) to find out more.  --> Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. Neural machine translation by jointly\nlearning to align and translate. CoRR, abs/1409.0473, 2014.\n\nThere are many more like this, please make sure that when you make a claim that is not supported by your own results, you cite relevant work, also make sure to find the appropriate paper(s) to cite for relevant concepts.\n\n\n\n", "summary_of_the_paper": "This paper presents how to extract arguments (Elementary Discourse Units (EDUs), which are the non-overlapping text spans corresponding to the atomic discourse units, and Argumentative Discourse Units (ADUs), defined as minimal analysis units) from sentences by incorporating constituency tree representations (using BENEPAR parser, Kitaev and Klein) in a BERT with a- Graph neural network and a subsequent CRF (explained in Sect 4.2).\n\nThe authors show that the constituency tree representations are relevant since the ADU units are distributed according to the grammatical structure of the sentence which is captured by the tree, and this is supported by empirical results.\n\n\n\n", "summary_of_the_review": "The paper lacks many details that, in my opinion, preclude it from publication at this stage. It makes bold claims that are not supported by citations or results (see Main review section). \n\nThe results show improvements over a BERT model that does not use the syntactic information nor CRF. I would recommend the authors to train multiple models (e.g. 5) with different random seeds and show average scores and standard deviation, it would help to ensure that your improvements are not just for the initialization you chose.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "bC3FjZm0Xh", "reviewer": "Reviewer_ExWX", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths:\n1. This paper proposes a novel method in addressing argument discourse unit detection, by incorporating constituency parser output via Graph Attention Layer. Although the reported results are better than the baseline, the scores are not comparable. Unlike previous works (Trautmann et al., 2020), the paper also does not report the complete result (see weaknesses below). \n\nWeaknesses:\n1. This paper has a poor literature review, motivation, and presentation, making it hard to read. For instance, a) In Section 1, many references are missing for argument analysis definition and examples, previous works in par3, and reference to the SPMRL dataset. Par4 is also confusing. The motivation to incorporate syntax into this task is also very weak. I suggest providing an example to show why syntax can be a good feature for ADU detection; b) In Section 2, par1-2 have a bad flow; par4 mentions “recent promising results”, but which one? It needs some references. c) It’s strange to have a Section of Experimental set-up prior to model architecture. Section 4.1 also has a bad naming (I suggest renaming it as the model name); d) The proposed method needs mathematical/formal definition which is completely missing in this paper.\n2. Although the dataset is based on Trautmann et al., 2020, the paper does not clearly state the original train/dev/test split of data, as well as general statistics of stance labels. I don't think the reader needs to check the original paper for this basic information.\n3. The motivation for pruning the constituency tree is not properly presented since Section 1. For instance, a) in Figure 1, why is the depth of the tree capped to a maximal value of three? b) in Section 3.2, “Maximum depth” is not clearly explained and poorly presented. What does the author mean by “columns 2, 3,4”? Which data partition is used to determine optimal “maximum depth”? It sounds to me that this value is determined based on all the data, which potentially harms the methodology.\n4. Unclear motivation and setting of using BENEPAR constituency parser. The authors mention that BENEPAR is a multilingual parser that can parse French and English sentences, however, in the reported result, there is no experimental result for both languages. Instead, the results seem to be in 1 language.\n5. The authors discard 1300 samples of data due to tokenization requirements of BERT and BANEPAR, which I think is a very strange motivation. Moreover, the main experimental result in Table 3 uses baseline results from Trautmann et al., 2020, which can be misleading as the data is not comparable anymore.\n6. The reported result does not have a clear description of in-domain and cross-domain data, nor further explanation or analysis.\n7. This paper does not report the full result of experiments, only token-level F1-score. In Trautmann et al., 2020, there are three different metrics: token-level, segment-level, and sentence-level", "summary_of_the_paper": "This paper demonstrates the utility of constituency parser for improving argumentative discourse unit (ADU) detection. ADU is defined as a “minimal unit of analysis” in argument mining and the task is to first segment the ADU spans, and then predict the stance label. This paper uses Trautmann et al., 2020 dataset that comprises 8000 sentences with 3 stance labels: PRO (supporting), CON (opposing), and NON (non-argumentative). The paper alters the architecture of BERT-CRF in the previous works, by adding Graph Attention Layer (GAT) prior to CRF. GAT is computed based on pruned constituency trees, yielding improved performance compared to BERT-Large.", "summary_of_the_review": "Although the paper has a novel and interesting idea, I argue it is not ready for publication due to the aforementioned weaknesses above. The critical problems are incomparable results with baseline as well as its incompleteness issues. \n", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "PfuDIZYwqd", "reviewer": "Reviewer_iNq3", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths\n\nThe paper presents a simple method for incorporating syntactic information for the task of ADU recognition. The hyperparameters are well documented, and strong empirical results.\n\nInteresting analysis for understanding which substructure of constituency tree is important.\n\nWeaknesses\n\nThe main weakness is that the relationship between ADU and constituency tree is not clearly described. Is it true that ADU are often phrases that occur in the constituency tree? How often does this happen? Does the new BERT-based model adhere to constituency constraints? Is the BENEPAR parser appropriate for this data? Based on Trautmann et al.’s comments on grammaticality and clauses, my intuition is that an ADU is almost always a phrase in the tree, in which case it is somewhat less surprising that it helps this task (and maybe it should be helping even more). If this is the case, it’s worth considering related work in distant supervision to include.\n\nThe treatment of constituency trees is haphazard. 1) At times it is not clear if dependency or constituency trees are being used; 2) Table 1 should be made more clear, what do the percentage values indicate, and why not use the full tree at all; 3) It is worth adding to the related work more work in constituency tree representation, such as but not limited to Yang and Deng 2020 that also use GNN to represent constituency tree.\n\n(low priority) Some technical details are concerning as described. For example, it is true that batching heterogeneous graphs may be somewhat more challenging than batching similar length sequences, but it is hard to believe this is one of the “major difficulties” of this work. If it is such a challenge, then it may have warranted further discussion about tradeoffs in architecture selection and impact on speed or performance.\n", "summary_of_the_paper": "The authors present results using syntactic information (primarily through constituency trees) on the task of recognizing argument discourse units (abbreviated to ADU). Their results are strong the paper is clearly lacking in multiple dimensions: a) it is not made clear the similarities between ADUs and constituency trees, and the lack of this clarity makes the research story less compelling; b) in general, the paper is difficult to follow (e.g. it is not clear whether dependency structure or constituency, or both, are being used in the model); and c) the paper does not present any new methods, primarily relying on the previously published GAT model --- although, the lack of novelty is not the sole issue with this work. For these reasons, I do not recommend to accept this paper at this time.\n", "summary_of_the_review": "I recommend to reject this paper for the following reasons: a) it does not sufficiently motivate the use of syntax, and does not make it clear relation between ADU and constituency; b) it is haphazard in its use and description of its main feature, constituency trees; c) the writing is lacking, it does not tell a particularly compelling research story, nor cover related work particularly well. In summary, it is not particularly strong in any dimension, and the main result does beat the baseline, but can easily be explained by the connection between constituency and ADU.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper present results using syntactic information (primarily through constituency trees) on the task of recognizing argument discourse units. No reviewer recommends acceptance of the paper:
- The empirical results appear strong, though the reviewers raise questions about some of the experimental choices. 
- The writing is unclear and reviewers point out many missing or incorrect references in the bibliography.
- There is little methodological novelty - known techniques are applied to a topic that has not been studied much.
Overall, the area chair agrees with the reviewers that this work does not yet meet the bar for ICLR.
Decision: Reject