Constituency Tree Representation for Argument Unit Recognition

1 INTRODUCTION . Argument Identification inside documents is the first step to study rhetorical speech processes or political debates . It is a concrete NLP application that interests the media community during the electoral periods . The goal is to precisely identify arguments discourse unit ( ADU ) ( defined as the minimal analysis units ) in a sentence and predict its stance , given a topic . Former works on argument analysis at the token level Trautmann ( 2020 ) has used language models such as the Bidirectional Encoder Representations from Transformers model ( BERT ) Devlin et al . ( 2019 ) , associated to probabilistic models such as Conditional Random Field ( CRF ) Lafferty et al . ( 2001 ) . This combination enables to increase the global prediction coherence with restricted finetuning . Language models are essential to incorporate statistical knowledge without having to train on huge datasets . However , these models were recently criticized for their lack of transparency on how training data are reused in predictions Bender et al . ( 2021 ) . Thus , the creation of collaborations between linguists and data researchers into collaborative project Gauthier et al . ( 2020 ) are an actual research topic which aims to create more linguistic oriented language models . In this work , we study the impact of incorporating grammatical structure into the BERT-CRF model . We focus on the constituency tree representation of the sentence . This representation is composed of internal nodes ( also called NT for Non Terminal ) representing the grammatical structure and leaf nodes ( nodes without children ) corresponding to the sentence words . The main goals of this paper are • to evaluate GNNs models combined with a CRF layer in order to use the syntactic information contained in the constituency tree representation ; 1 1The code will be publicly available on GitHub after the blind review process . It contains a download script of the dataset , the model code , the interpretability models and the hyper parameter optimization scripts . • to use interpretability models to evaluate the contribution of the tree structure to the final results . 2 RELATED WORKS . In this section , related works about Argument Mining and tree structure representation in NLP , Graph Neural Network and Conditional Random Field are reviewed . ARGUMENT MINING The precise definition of the concept of argument is an important step when creating dataset annotation rules . A previous work on Argument Mining Lawrence & Reed ( 2020 ) has identified two main scales of argument units : Elementary Discourse Units ( EDUs ) , which are the non-overlapping text spans corresponding to the atomic discourse units , and Argumentative Discourse Units ( ADUs ) , defined as minimal analysis units . As explained in Peldszus & Stede ( 2013 ) , ” when two EDUs are joined by some coherence relation that is irrelevant for argumentation , the resulting complex might be the better ADU ” . With this definition , one can notice that EDUs and ADUs are not defined at the sentence level . Thus it is more relevant to use a dataset labeled at the token level instead of sentence level . The current biggest dataset with argument unit ADUs labels at the token level is AURC 2 Trautmann et al . ( 2020 ) . TREE STRUCTURE REPRESENTATION IN NLP The goal of using a tree representation of the sentence is to exploit the syntactic hierarchical dependencies of the sentence with graph neural network . Thus , the constituency tree representation is more appropriate than POS taggers which provide one tag per token ( according to the Universal POS tags categories ) . We prefer the constituency parsing over the dependency parsing because dependency grammars phrasal constituents and phrase-structure rules do not play a direct role in dependency parsing . Thus , the distance ( in terms of number of edges of the tree ) between words of the same grammatical class is longer in the dependency parsing of the sentence . Pioneer work ( Gildea & Palmer , 2002 ) presenting the benefits of using constituency tree representation has failed to scale into production . This is caused by the lack of a reliable model to generate constituency tree representation of the sentences . However , recent promising results were made in consistency and dependency parsing . Zhang ( 2020 ) For our preprocessing step we used a neural network model called Berkeley Neural Parser ( BENEPAR ) ( Kitaev & Klein , 2018 ) , which is trained on 11 languages . We use this model with the weights provided by these designers . Other possibilities for this task are the StanfordParser with Stanza Qi et al . ( 2020 ) or NLTK . We chose BENEPAR because it is integrated with Spacy and it can be used in 11 different languages . Other papers have recently studied the use of tree structure to incorporate syntactic information to their models . Marcheggiani & Titov ( 2017 ) uses Graph Convolutional Networks ( GCNs ) based on the dependency tree structure of the sentence for semantic role labeling . Beck et al . ( 2018 ) uses GNNs for generation tasks from abstract meaning representation . GRAPH NEURAL NETWORK Graph Neural Networks refer to the various methods which extend Deep Learning to data organized in graphs . GNNs were introduced by Gori et al . ( 2005 ) as an extension of Recursive Neural Networks ( RNN ) . GNNs use a neural network to propagate information within the dataset geometrical structure . In this article , the Graph ATtention ( GAT ) layer Veličković et al . ( 2018 ) which appears to be the most adapted message passing layer to propagate information following a self-attention strategy is used . The attention mechanism has been widely used in NLP tasks since its good results in translation tasks presented in Vaswani et al . ( 2017 ) . The main benefits of attention mechanisms are that they 2The dataset is publicly available at https : //github.com/trtm/AURC allow to deal with variable size inputs and enable to specify different weights to different neighbours focusing on the most relevant parts of the input to make decisions . CONDITIONAL RANDOM FIELD Conditional Random Fields were introduced by Hammersley & Clifford ( 1971 ) as probabilistic models for magnetic field modeling . Their goal was to exploit the Markov assumption of a lattice to derive the log probability of the label distribution formula . Since then , CRFs have been widely used in machine learning thanks to the iterative scaling algorithms presented by Lafferty et al . ( 2001 ) to find the parameter that maximizes the log-likelihood of the training data . Many interactions have been made between CRFs and GNNs . Gao et al . ( 2019 ) proposed a new CRF layer for GCNs . Cui & Zhang ( 2019 ) proposed a neural network alternative to CRFs motivated by the fact that the Markov assumption is not always respected for linear chain representation of the sentence . 3 MODEL PRESENTATION . 3.1 BERT & CRF . This model has been introduced by Trautmann et al . ( 2020 ) . It is composed of two steps . In the first step , the sentence is tokenized following the BERT tokenizer and the BERT model is fine-tuned for token classification , where the output of the last layer matches the number of classes of the dataset . In the second step , a linear chain Conditional Random Field is applied to estimate the probability of each label class from BERT . The main idea of this model is to leverage the BERT attention knowledge and then to improve the results by incorporating a linear chain dependency structure . This takes advantage of neighbours dependency relations between words . The good results of this model lead us to use it as a competitive benchmark for our approach based on constituency trees as input representations of sentences . 3.2 BERT , GNN & CRF . A major difficulty in choosing a graph neural layer is that each sentence has a different tree representation . Hence , the model needs to be agnostic to the lack of completeness of the tree structures from the training set . The message passing design enables to share the model weights among the network nodes , thus the results do not depend on the upfront global tree structure access . It also enables to deal with directed and undirected graphs as the information flow is conditioned to the existence of a directed edge between two nodes . GRAPH ATTENTION LAYER The Graph Attention Layer Veličković et al . ( 2018 ) allows to combine the attention mechanism with the graph structure , preserving the syntactic structural information of the sentence . In order to improve the model stability , adding multi-head attention layers might be beneficial to the training step . The different heads are then aggregated in order to provide the next hidden states of the neural network . As opposed to Graph Convolution Layer , this layer allows to assign different importances to nodes of a same neighbourhood . Furthermore , analyzing the learned attentional weights may benefit to the model interpretability , as we can extract information from attention weight matrices . In order to take advantage of the dependency structure of the sentence , we incorporate a GNN between the use of BERT and the linear chain CRF . The idea behind this architecture is the following . First , the BERT language model outputs the sentence hidden representation . Next , the information is spread to the graph neighbours at each iteration . In that way , we expect to reach a better consistency between neighbour nodes when we train on a restricted dataset . 4 EXPERIMENTAL SETUP . 4.1 DATA SOURCE AND ANNOTATION RULES . The dataset used in our study has been introduced by Trautmann et al . ( 2020 ) . This dataset appears to be one of the few datasets available that annotates argument unit at the token level . It is composed of 8000 sentences , equally divided into 8 topics , and can be used for the task of argument unit identification as well as argument polarity identification . The authors distinguished between PRO ( supporting ) , CON ( opposing ) arguments and NON ( nonargumentative ) words . In the cross domain setting , the topics in the train dataset are not presented in the test dataset . In the in domain setting , the topics in the train dataset are also present ( with other test sentences ) inside the test dataset . This enables to evaluate how important the domain knowledge are when identifying ADU . In our preprocessing step , we need to label the internal nodes NT of the constituency tree in order to learn their representation . As these annotations are not provided by the dataset , we have chosen to annotate the internal nodes with the same labeling rules as the one described above for the sentences labeling . This enables to preserve the internal logic of the labeling process . 4.2 CONSTITUENCY TREE CONSTRUCTION . One of the main advantages of adding a constituency tree to classical methods is the greater proximity of the words which belong to the same grammatical class than the words which only are neighbours in a linear representation of the sentence . We can also illustrate this fact with the constituency tree presented in Figure 1 . For this sentence , it can be seen that the ADU units are distributed following the grammatical structure of the sentence . For instance , the words ” inflation ” and ” the ” are neighbours in the sentence but are further in the constituency tree structure . This leads to a better identification of boundaries between ADUs . For the construction of the constituency tree , we use Berkeley Neural Parser ( BENEPAR ) Kitaev & Klein ( 2018 ) . This is a multilingual constituency parser which benefits from unsupervised pretraining across a variety of languages . This model provides the weights for 12 languages and a simple API incorporated to Spacy , and is thus a good choice in order to have a good accuracy , integrated into a larger pipeline . This additional pre-processing step requires to have an exact match between the tokenizations used by BENEPAR and BERT models . This constraint imposed us to remove the unmatched sentences . In addition , the samples from the in-domain test set were also excluded in the cross-domain train and dev sets . As a result , there were 4000 samples reduced to 3960 in train , 800 reduced to 790 in dev and 2000 in test reduced to 1959 for the cross-domain split ; and 4200 reduced to 4157 in train , 600 reduced to 593 in dev and 1200 reduced to 1159 in test for the in-domain split . At the end , less than 2 % of the dataset were removed .

Summary:  This paper presents a bert-gat-crf framework for recognizing argument units in sentences through constituency tree representation. Examples of what is a consistency tree of short and long sequences are given. Experiments on small datasets proved the performance of the bert-gat-crf algorithm.