Venue: ICLR 2022 Conference

Reviews:

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: 
This paper presents a bert-gat-crf framework for recognizing argument units in sentences through constituency tree representation. Examples of what is a consistency tree of short and long sequences are given. Experiments on small datasets proved the performance of the bert-gat-crf algorithm.

main_review: The method used in this paper is more alike a connection of three existing methods, bert, gat, and crf. Currently, could find little novel methods other than bert-gat-crf in this paper. The target questions is rather limited to a specific NLP task of semantic parsing field. Also semantic role labeling and related baselines/datasets should be involved for some direct comparison of this “argument unit recognition” task. 




Detailed questions and comments:
1.	Figure 1, why use 0 to 6 as higher level nodes? I think using POS-tags and phrase-tags are more readable, such as article tag for “The”, verb tag for “is” and noun phrase for “the time”.
2.	Prefer to see detailed examples that are improved by this bert-gat-crf method for intuitive understanding.
3.	Some cites can be improved, such as the bottom lines in page 5. With Roberta liu et al. (2019)? -> Roberta (liu et al. 2019).
4.	What are the reasons of not using dependency trees and give a comparison as well for this task? Dependency tree structure are shallow parsing and can be understood easily and adapted for semantic role labeling of argument unit detection and also predicate-argument structure prediction as well.

summary_of_the_review: Strong: 
1.	Using of bert-gat-crf for argument unit detection and achieved improvements in small-scale datsets;

Weak:
1.	The method of bert-gat-crf is less novel for this paper;
2.	The target question is a pure NLP task and also related to other tasks such as semantic role labeling that better mention and compare with their baselines as well.
review_title: null

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: This paper demonstrates the utility of constituency parser for improving argumentative discourse unit (ADU) detection. ADU is defined as a “minimal unit of analysis” in argument mining and the task is to first segment the ADU spans, and then predict the stance label. This paper uses Trautmann et al., 2020 dataset that comprises 8000 sentences with 3 stance labels: PRO (supporting), CON (opposing), and NON (non-argumentative). The paper alters the architecture of BERT-CRF in the previous works, by adding Graph Attention Layer (GAT) prior to CRF. GAT is computed based on pruned constituency trees, yielding improved performance compared to BERT-Large.
main_review: Strengths:
1. This paper proposes a novel method in addressing argument discourse unit detection, by incorporating constituency parser output via Graph Attention Layer. Although the reported results are better than the baseline, the scores are not comparable. Unlike previous works (Trautmann et al., 2020), the paper also does not report the complete result (see weaknesses below). 

Weaknesses:
1. This paper has a poor literature review, motivation, and presentation, making it hard to read. For instance, a) In Section 1, many references are missing for argument analysis definition and examples, previous works in par3, and reference to the SPMRL dataset. Par4 is also confusing. The motivation to incorporate syntax into this task is also very weak. I suggest providing an example to show why syntax can be a good feature for ADU detection; b) In Section 2, par1-2 have a bad flow; par4 mentions “recent promising results”, but which one? It needs some references. c) It’s strange to have a Section of Experimental set-up prior to model architecture. Section 4.1 also has a bad naming (I suggest renaming it as the model name); d) The proposed method needs mathematical/formal definition which is completely missing in this paper.
2. Although the dataset is based on Trautmann et al., 2020, the paper does not clearly state the original train/dev/test split of data, as well as general statistics of stance labels. I don't think the reader needs to check the original paper for this basic information.
3. The motivation for pruning the constituency tree is not properly presented since Section 1. For instance, a) in Figure 1, why is the depth of the tree capped to a maximal value of three? b) in Section 3.2, “Maximum depth” is not clearly explained and poorly presented. What does the author mean by “columns 2, 3,4”? Which data partition is used to determine optimal “maximum depth”? It sounds to me that this value is determined based on all the data, which potentially harms the methodology.
4. Unclear motivation and setting of using BENEPAR constituency parser. The authors mention that BENEPAR is a multilingual parser that can parse French and English sentences, however, in the reported result, there is no experimental result for both languages. Instead, the results seem to be in 1 language.
5. The authors discard 1300 samples of data due to tokenization requirements of BERT and BANEPAR, which I think is a very strange motivation. Moreover, the main experimental result in Table 3 uses baseline results from Trautmann et al., 2020, which can be misleading as the data is not comparable anymore.
6. The reported result does not have a clear description of in-domain and cross-domain data, nor further explanation or analysis.
7. This paper does not report the full result of experiments, only token-level F1-score. In Trautmann et al., 2020, there are three different metrics: token-level, segment-level, and sentence-level
summary_of_the_review: Although the paper has a novel and interesting idea, I argue it is not ready for publication due to the aforementioned weaknesses above. The critical problems are incomparable results with baseline as well as its incompleteness issues.
review_title: null

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: The authors present results using syntactic information (primarily through constituency trees) on the task of recognizing argument discourse units (abbreviated to ADU). Their results are strong the paper is clearly lacking in multiple dimensions: a) it is not made clear the similarities between ADUs and constituency trees, and the lack of this clarity makes the research story less compelling; b) in general, the paper is difficult to follow (e.g. it is not clear whether dependency structure or constituency, or both, are being used in the model); and c) the paper does not present any new methods, primarily relying on the previously published GAT model --- although, the lack of novelty is not the sole issue with this work. For these reasons, I do not recommend to accept this paper at this time.

main_review: Strengths

The paper presents a simple method for incorporating syntactic information for the task of ADU recognition. The hyperparameters are well documented, and strong empirical results.

Interesting analysis for understanding which substructure of constituency tree is important.

Weaknesses

The main weakness is that the relationship between ADU and constituency tree is not clearly described. Is it true that ADU are often phrases that occur in the constituency tree? How often does this happen? Does the new BERT-based model adhere to constituency constraints? Is the BENEPAR parser appropriate for this data? Based on Trautmann et al.’s comments on grammaticality and clauses, my intuition is that an ADU is almost always a phrase in the tree, in which case it is somewhat less surprising that it helps this task (and maybe it should be helping even more). If this is the case, it’s worth considering related work in distant supervision to include.

The treatment of constituency trees is haphazard. 1) At times it is not clear if dependency or constituency trees are being used; 2) Table 1 should be made more clear, what do the percentage values indicate, and why not use the full tree at all; 3) It is worth adding to the related work more work in constituency tree representation, such as but not limited to Yang and Deng 2020 that also use GNN to represent constituency tree.

(low priority) Some technical details are concerning as described. For example, it is true that batching heterogeneous graphs may be somewhat more challenging than batching similar length sequences, but it is hard to believe this is one of the “major difficulties” of this work. If it is such a challenge, then it may have warranted further discussion about tradeoffs in architecture selection and impact on speed or performance.

summary_of_the_review: I recommend to reject this paper for the following reasons: a) it does not sufficiently motivate the use of syntax, and does not make it clear relation between ADU and constituency; b) it is haphazard in its use and description of its main feature, constituency trees; c) the writing is lacking, it does not tell a particularly compelling research story, nor cover related work particularly well. In summary, it is not particularly strong in any dimension, and the main result does beat the baseline, but can easily be explained by the connection between constituency and ADU.
review_title: null

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: This paper presents how to extract arguments (Elementary Discourse Units (EDUs), which are the non-overlapping text spans corresponding to the atomic discourse units, and Argumentative Discourse Units (ADUs), defined as minimal analysis units) from sentences by incorporating constituency tree representations (using BENEPAR parser, Kitaev and Klein) in a BERT with a- Graph neural network and a subsequent CRF (explained in Sect 4.2).

The authors show that the constituency tree representations are relevant since the ADU units are distributed according to the grammatical structure of the sentence which is captured by the tree, and this is supported by empirical results.




main_review: 
Some comments:

The concept of Argument from Sentences is not properly described until Section 2, first paragraph. The entire abstract and introduction you are wondering what the paper will be about. Also, the description of Argument mining will be much better with an example.
the abstract starts too ambiguous. It is not even clear what "arguments" from sentences are and why neighboring words are relevant.

- "However, these models were recently criticized for their lack of transparency on how training data are reused in predictions." 
This claim requires a citation.

- "However, these models were recently criticized for their lack of transparency on how training data are reused in predictions. Thus, model architectures based on linguistic concepts seem o be needed to improve current models and increase their explainability. " 
Why are you reaching this conclusion? The two sentences seems a bit disconnected.

- "the most commonly used models do not do not take into account the grammatical structure of sentences, which leads to a gap between current NLP models and linguistic analysis." 
There is in fact a lot of work on showing how NLP models are able to understand linguistic structure, this is one place to start: https://aclanthology.org/N19-1419.pdf

-"This is caused by the lack of a reliable model to generate constituency tree representation of the sentences. However, recent promising results were made in consistency and dependency parsing" 
Again, these require citations. The first statement and the second one, both of them. 

The related work section contains technical details about your experimental setup. It is hard to follow. 

"The attention mechanism was introduced by Vaswani et al. (2017) for translation tasks in NLP and has since been widely used in many other tasks. " The attention mechanism was not introduced by Vaswani et al. (2017), I would recommend to thoroughly read Vaswani et al. (2017) to find out more.  --> Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. Neural machine translation by jointly
learning to align and translate. CoRR, abs/1409.0473, 2014.

There are many more like this, please make sure that when you make a claim that is not supported by your own results, you cite relevant work, also make sure to find the appropriate paper(s) to cite for relevant concepts.




summary_of_the_review: The paper lacks many details that, in my opinion, preclude it from publication at this stage. It makes bold claims that are not supported by citations or results (see Main review section). 

The results show improvements over a BERT model that does not use the syntactic information nor CRF. I would recommend the authors to train multiple models (e.g. 5) with different random seeds and show average scores and standard deviation, it would help to ensure that your improvements are not just for the initialization you chose.
review_title: null

Meta-review: The paper present results using syntactic information (primarily through constituency trees) on the task of recognizing argument discourse units. No reviewer recommends acceptance of the paper:
- The empirical results appear strong, though the reviewers raise questions about some of the experimental choices. 
- The writing is unclear and reviewers point out many missing or incorrect references in the bibliography.
- There is little methodological novelty - known techniques are applied to a topic that has not been studied much.
Overall, the area chair agrees with the reviewers that this work does not yet meet the bar for ICLR.
Decision: Reject