Constituency Tree Representation for Argument Unit Recognition

1 INTRODUCTION . Argument Identification inside documents is the first step to study rhetorical speech processes or political debates . It is a concrete NLP application that interests the media community during the electoral periods . The goal is to precisely identify arguments discourse unit ( ADU ) ( defined as the minimal analysis units ) in a sentence and predict its stance , given a topic . Former works on argument analysis at the token level Trautmann ( 2020 ) has used language models such as the Bidirectional Encoder Representations from Transformers model ( BERT ) Devlin et al . ( 2019 ) , associated to probabilistic models such as Conditional Random Field ( CRF ) Lafferty et al . ( 2001 ) . This combination enables to increase the global prediction coherence with restricted finetuning . Language models are essential to incorporate statistical knowledge without having to train on huge datasets . However , these models were recently criticized for their lack of transparency on how training data are reused in predictions Bender et al . ( 2021 ) . Thus , the creation of collaborations between linguists and data researchers into collaborative project Gauthier et al . ( 2020 ) are an actual research topic which aims to create more linguistic oriented language models . In this work , we study the impact of incorporating grammatical structure into the BERT-CRF model . We focus on the constituency tree representation of the sentence . This representation is composed of internal nodes ( also called NT for Non Terminal ) representing the grammatical structure and leaf nodes ( nodes without children ) corresponding to the sentence words . The main goals of this paper are • to evaluate GNNs models combined with a CRF layer in order to use the syntactic information contained in the constituency tree representation ; 1 1The code will be publicly available on GitHub after the blind review process . It contains a download script of the dataset , the model code , the interpretability models and the hyper parameter optimization scripts . • to use interpretability models to evaluate the contribution of the tree structure to the final results . 2 RELATED WORKS . In this section , related works about Argument Mining and tree structure representation in NLP , Graph Neural Network and Conditional Random Field are reviewed . ARGUMENT MINING The precise definition of the concept of argument is an important step when creating dataset annotation rules . A previous work on Argument Mining Lawrence & Reed ( 2020 ) has identified two main scales of argument units : Elementary Discourse Units ( EDUs ) , which are the non-overlapping text spans corresponding to the atomic discourse units , and Argumentative Discourse Units ( ADUs ) , defined as minimal analysis units . As explained in Peldszus & Stede ( 2013 ) , ” when two EDUs are joined by some coherence relation that is irrelevant for argumentation , the resulting complex might be the better ADU ” . With this definition , one can notice that EDUs and ADUs are not defined at the sentence level . Thus it is more relevant to use a dataset labeled at the token level instead of sentence level . The current biggest dataset with argument unit ADUs labels at the token level is AURC 2 Trautmann et al . ( 2020 ) . TREE STRUCTURE REPRESENTATION IN NLP The goal of using a tree representation of the sentence is to exploit the syntactic hierarchical dependencies of the sentence with graph neural network . Thus , the constituency tree representation is more appropriate than POS taggers which provide one tag per token ( according to the Universal POS tags categories ) . We prefer the constituency parsing over the dependency parsing because dependency grammars phrasal constituents and phrase-structure rules do not play a direct role in dependency parsing . Thus , the distance ( in terms of number of edges of the tree ) between words of the same grammatical class is longer in the dependency parsing of the sentence . Pioneer work ( Gildea & Palmer , 2002 ) presenting the benefits of using constituency tree representation has failed to scale into production . This is caused by the lack of a reliable model to generate constituency tree representation of the sentences . However , recent promising results were made in consistency and dependency parsing . Zhang ( 2020 ) For our preprocessing step we used a neural network model called Berkeley Neural Parser ( BENEPAR ) ( Kitaev & Klein , 2018 ) , which is trained on 11 languages . We use this model with the weights provided by these designers . Other possibilities for this task are the StanfordParser with Stanza Qi et al . ( 2020 ) or NLTK . We chose BENEPAR because it is integrated with Spacy and it can be used in 11 different languages . Other papers have recently studied the use of tree structure to incorporate syntactic information to their models . Marcheggiani & Titov ( 2017 ) uses Graph Convolutional Networks ( GCNs ) based on the dependency tree structure of the sentence for semantic role labeling . Beck et al . ( 2018 ) uses GNNs for generation tasks from abstract meaning representation . GRAPH NEURAL NETWORK Graph Neural Networks refer to the various methods which extend Deep Learning to data organized in graphs . GNNs were introduced by Gori et al . ( 2005 ) as an extension of Recursive Neural Networks ( RNN ) . GNNs use a neural network to propagate information within the dataset geometrical structure . In this article , the Graph ATtention ( GAT ) layer Veličković et al . ( 2018 ) which appears to be the most adapted message passing layer to propagate information following a self-attention strategy is used . The attention mechanism has been widely used in NLP tasks since its good results in translation tasks presented in Vaswani et al . ( 2017 ) . The main benefits of attention mechanisms are that they 2The dataset is publicly available at https : //github.com/trtm/AURC allow to deal with variable size inputs and enable to specify different weights to different neighbours focusing on the most relevant parts of the input to make decisions . CONDITIONAL RANDOM FIELD Conditional Random Fields were introduced by Hammersley & Clifford ( 1971 ) as probabilistic models for magnetic field modeling . Their goal was to exploit the Markov assumption of a lattice to derive the log probability of the label distribution formula . Since then , CRFs have been widely used in machine learning thanks to the iterative scaling algorithms presented by Lafferty et al . ( 2001 ) to find the parameter that maximizes the log-likelihood of the training data . Many interactions have been made between CRFs and GNNs . Gao et al . ( 2019 ) proposed a new CRF layer for GCNs . Cui & Zhang ( 2019 ) proposed a neural network alternative to CRFs motivated by the fact that the Markov assumption is not always respected for linear chain representation of the sentence . 3 MODEL PRESENTATION . 3.1 BERT & CRF . This model has been introduced by Trautmann et al . ( 2020 ) . It is composed of two steps . In the first step , the sentence is tokenized following the BERT tokenizer and the BERT model is fine-tuned for token classification , where the output of the last layer matches the number of classes of the dataset . In the second step , a linear chain Conditional Random Field is applied to estimate the probability of each label class from BERT . The main idea of this model is to leverage the BERT attention knowledge and then to improve the results by incorporating a linear chain dependency structure . This takes advantage of neighbours dependency relations between words . The good results of this model lead us to use it as a competitive benchmark for our approach based on constituency trees as input representations of sentences . 3.2 BERT , GNN & CRF . A major difficulty in choosing a graph neural layer is that each sentence has a different tree representation . Hence , the model needs to be agnostic to the lack of completeness of the tree structures from the training set . The message passing design enables to share the model weights among the network nodes , thus the results do not depend on the upfront global tree structure access . It also enables to deal with directed and undirected graphs as the information flow is conditioned to the existence of a directed edge between two nodes . GRAPH ATTENTION LAYER The Graph Attention Layer Veličković et al . ( 2018 ) allows to combine the attention mechanism with the graph structure , preserving the syntactic structural information of the sentence . In order to improve the model stability , adding multi-head attention layers might be beneficial to the training step . The different heads are then aggregated in order to provide the next hidden states of the neural network . As opposed to Graph Convolution Layer , this layer allows to assign different importances to nodes of a same neighbourhood . Furthermore , analyzing the learned attentional weights may benefit to the model interpretability , as we can extract information from attention weight matrices . In order to take advantage of the dependency structure of the sentence , we incorporate a GNN between the use of BERT and the linear chain CRF . The idea behind this architecture is the following . First , the BERT language model outputs the sentence hidden representation . Next , the information is spread to the graph neighbours at each iteration . In that way , we expect to reach a better consistency between neighbour nodes when we train on a restricted dataset . 4 EXPERIMENTAL SETUP . 4.1 DATA SOURCE AND ANNOTATION RULES . The dataset used in our study has been introduced by Trautmann et al . ( 2020 ) . This dataset appears to be one of the few datasets available that annotates argument unit at the token level . It is composed of 8000 sentences , equally divided into 8 topics , and can be used for the task of argument unit identification as well as argument polarity identification . The authors distinguished between PRO ( supporting ) , CON ( opposing ) arguments and NON ( nonargumentative ) words . In the cross domain setting , the topics in the train dataset are not presented in the test dataset . In the in domain setting , the topics in the train dataset are also present ( with other test sentences ) inside the test dataset . This enables to evaluate how important the domain knowledge are when identifying ADU . In our preprocessing step , we need to label the internal nodes NT of the constituency tree in order to learn their representation . As these annotations are not provided by the dataset , we have chosen to annotate the internal nodes with the same labeling rules as the one described above for the sentences labeling . This enables to preserve the internal logic of the labeling process . 4.2 CONSTITUENCY TREE CONSTRUCTION . One of the main advantages of adding a constituency tree to classical methods is the greater proximity of the words which belong to the same grammatical class than the words which only are neighbours in a linear representation of the sentence . We can also illustrate this fact with the constituency tree presented in Figure 1 . For this sentence , it can be seen that the ADU units are distributed following the grammatical structure of the sentence . For instance , the words ” inflation ” and ” the ” are neighbours in the sentence but are further in the constituency tree structure . This leads to a better identification of boundaries between ADUs . For the construction of the constituency tree , we use Berkeley Neural Parser ( BENEPAR ) Kitaev & Klein ( 2018 ) . This is a multilingual constituency parser which benefits from unsupervised pretraining across a variety of languages . This model provides the weights for 12 languages and a simple API incorporated to Spacy , and is thus a good choice in order to have a good accuracy , integrated into a larger pipeline . This additional pre-processing step requires to have an exact match between the tokenizations used by BENEPAR and BERT models . This constraint imposed us to remove the unmatched sentences . In addition , the samples from the in-domain test set were also excluded in the cross-domain train and dev sets . As a result , there were 4000 samples reduced to 3960 in train , 800 reduced to 790 in dev and 2000 in test reduced to 1959 for the cross-domain split ; and 4200 reduced to 4157 in train , 600 reduced to 593 in dev and 1200 reduced to 1159 in test for the in-domain split . At the end , less than 2 % of the dataset were removed .

Summary: This paper demonstrates the utility of constituency parser for improving argumentative discourse unit (ADU) detection. ADU is defined as a “minimal unit of analysis” in argument mining and the task is to first segment the ADU spans, and then predict the stance label. This paper uses Trautmann et al., 2020 dataset that comprises 8000 sentences with 3 stance labels: PRO (supporting), CON (opposing), and NON (non-argumentative). The paper alters the architecture of BERT-CRF in the previous works, by adding Graph Attention Layer (GAT) prior to CRF. GAT is computed based on pruned constituency trees, yielding improved performance compared to BERT-Large.