Venue: NAACL 2019

# Argument Mining for Understanding Peer Reviews

Xinyu Hua, Mitko Nikolov, Nikhil Badugu, Lu Wang

Khoury College of Computer Sciences

Northeastern University

Boston, MA 02115

{hua.x, nikolov.m, badugu.n}@husky.neu.edu

luwang@ccs.neu.edu

###### Abstract

Peer-review plays a critical role in the scientific writing and publication ecosystem. To assess the efficiency and efficacy of the reviewing process, one essential element is to understand and evaluate the reviews themselves. In this work, we study the content and structure of peer reviews under the argument mining framework, through automatically detecting (1) argumentative propositions put forward by reviewers, and (2) their types (e.g., evaluating the work or making suggestions for improvement). We first collect \(14.2\)K reviews from major machine learning and natural language processing venues. \(400\) reviews are annotated with \(10,386\) propositions and corresponding types of Evaluation, Request, Fact, Reference, or Quote. We then train state-of-the-art proposition segmentation and classification models on the data to evaluate their utilities and identify new challenges for this new domain, motivating future directions for argument mining. Further experiments show that proposition usage varies across venues in amount, type, and topic.

+
Footnote †: journal: Proceedings of NAACL-HLT 2019

1
Footnote 1: _Proceedings of NAACL-HLT 2019_, pages 2131–2137 Minneapolis, Minnesota, June 2 - June 7, 2019. ©2019 Association for Computational Linguistics

## 1 Introduction

Peer review is a process where domain experts scrutinize the quality of research work in their field, and it is a cornerstone of scientific discovery [1, 16, 1]. In 2015 alone, approximately \(63.4\) million hours were spent on peer reviews [17]. To maximize their benefit to the scientific community, it is crucial to understand and evaluate the construction and limitation of reviews themselves. However, minimal work has been done to analyze reviews' content and structure, let alone to evaluate their qualities.

As seen in Figure 1, peer reviews resemble arguments: they contain **argumentative propositions** (henceforth propositions) that convey reviewers' interpretation and evaluation of the research. Constructive reviews, e.g., review \(\#2\), often contain in-depth analysis as well as concrete suggestions. As a result, automatically identifying propositions and their types would be useful to understand the composition of peer reviews.

Therefore, we propose _an argument mining-based approach to understand the content and structure of peer reviews_. Argument mining studies the automatic detection of argumentative components and structure within discourse [16]. Specifically, argument types (e.g. evidence and reasoning) and their arrangement are indicative of argument quality [1, 15]. In this work, we focus on two specific tasks: (1) **proposition segmentation**--detecting elementary argumentative discourse units that are

Figure 1: Sample ICLR review excerpts. Propositions are annotated with types, such as Fact (fact), Eval (evaluation), and Req (request). Review #2 contains in-depth evaluation and actionable suggestion, thus is perceived to be of a higher quality.



propositions, and (2) **proposition classification--** labeling the propositions according to their types (e.g., evaluation vs. request).

Since there was no annotated dataset for peer reviews, as part of this study, we first collect \(14.2\)K reviews from major machine learning (ML) and natural language processing (NLP) venues. We create a dataset, **AMPERE** (Argument Mining for PEer REviews), by annotating \(400\) reviews with \(10,386\) propositions and labeling each proposition with the type of Evaluation, Request, Fact, Reference, Quote, or Non-arg.1 Significant inter-annotator agreement is achieved for proposition segmentation (Cohen's \(\kappa=0.93\)), with good consensus level for type annotation (Krippendorf's \(\alpha_{U}=0.61\)).

Footnote 1: Dataset and annotation guideline can be found at [http://xinyuhua.github.io/Resources/naacl19/](http://xinyuhua.github.io/Resources/naacl19/).

We benchmark our new dataset with state-of-the-art and popular argument mining models to better understand the challenges posed in this new domain. We observe a significant drop of performance for proposition segmentation on AMPERE, mainly due to its different argument structure. For instance, \(25\%\) of the sentences contain more than one proposition, compared to that of \(8\%\) for essays [7], motivating new solutions for segmentation and classification.

We further investigate review structure difference across venues based on proposition usage, and uncover several patterns. For instance, ACL reviews tend to contain more propositions than those in ML venues, especially with more requests but fewer facts. We further find that reviews with extreme ratings, i.e., strong reject or accept, tend to be shorter and make much fewer requests. Moreover, we probe the salient words for different proposition types. For example, ACL reviewers ask for more "examples" when making requests, while ICLR reviews contain more evaluation of "network" and how models are "trained".

## 2 Ampere Dataset

We collect review data from three sources: (1) openreview.net--an online peer reviewing platform for ICLR 2017, ICLR 2018, and UAI 2018 2; (2) reviews released for accepted papers at NeurIPS from 2013 to 2017; and (3) opted-in reviews for ACL 2017 from Kang et al. kangetal18.

Footnote 2: ICLR reviews are downloaded from the public API: [https://github.com/iesl/openreview-py](https://github.com/iesl/openreview-py). UAI reviews are collected by the OpenReview team.

In total, \(14,202\) reviews are collected (ICLR: \(4,057\); UAI: \(718\); ACL: \(275\); and NeurIPS: \(9,152\)). All venues except NeurIPS have paper rating scores attached to the reviews.

Annotation ProcessFor proposition segmentation, we adopt the concepts from Park et al. parketal15 and instruct the annotators to identify elementary argumentative discourse units on sentence or subsentence level, based on their discourse functions and topics. They then classify the propositions into five types with an additional non-argument category, as explained in Table 1.

\(400\) ICLR 2018 reviews are sampled for annotation, with similar distributions of length and rating to those of the full dataset. Two annotators who are fluent English speakers first label the \(400\) reviews with proposition segments and types, and a third annotator then resolves disagreements.

We calculate the inter-annotator agreement between the two annotators. A Cohen's \(\kappa\) of \(0.93\) is achieved for proposition segmentation, with each review treated as a BIO sequence. For classification, unitized Krippendorf's \(\alpha_{U}\)[10], which considers disagreements among segmentation, is calculated per review and then averaged over all samples, and the value is \(0.61\). Among the exactly matched proposition segments, we report a Cohen's \(\kappa\) of \(0.64\).

StatisticsTable 2 shows comparison between AMPERE and some other argument min

\begin{table}
\begin{tabular}{l c c c} \hline \hline
**Evaluation**: Subjective statements, often containing qualitative judgment. Ex: “_This paper shows nice results on a number of small tasks._”**Request**: Statements suggesting a course of action. Ex: “_The authors should compare with the following methods._”**Fact**: Objective information of the paper or commonsense knowledge. Ex: “_Existing works on multi-task neural networks typically use hand-tuned weights..._”**Reference**: Citations and URLs. Ex: “_see MuseGAN [20]__, MidiNet [29] etc._ ”**Quote**: Quotations from the paper. Ex: “_The author wrote ’where r is lower bound of feature norm!._”**Non-arg**: Non-argumentative statements. Ex: “_Aha, now I understand._”**& \\ \hline \hline  & **\#Doc** & **\#Sent** & **\#Prop** \\ \hline Comments [21] & 731 & 3,994 & 4,931 \\ Essays [7] & 402 & 7,116 & 6,089 \\ News [1] & 300 & 11,754 & 1,3413 \\ Web [1] & 340 & 3,899 & 1,882 \\ AMPERE & 400 & 8,030 & 10,386 \\ \hline \hline \end{tabular}
\end{table}
Table 2: Statistics for AMPERE and some argument mining corpora, including # of annotated propositions.

 

ing datasets of different genres. We also show the number of propositions in each category in Table 3. The most frequent types are evaluation (\(38.3\%\)) and fact (\(36.5\%\)).

## 3 Experiments with Existing Models

We benchmark AMPERE with popular and state-of-the-art models for proposition segmentation and classification. Both tasks can be treated as sequence tagging problems with the setup similar to Schulz et al. (2018). For experiments, \(320\) reviews (\(7,999\) propositions) are used for training and \(80\) reviews (\(2,387\) propositions) are used for testing. Following Niculae et al. (2017), \(5\)-fold cross validation on the training set is used for hyperparameter tuning. To improve the accuracy of tokenization, we manually replace mathematical formulas, variables, URL links, and formatted citation with special tokens such as <EQN>, <VAR>, <URL>, and <CIT>. Parameters, lexicons, and features used for the models are described in the supplementary material.

### Task I: Proposition Segmentation

We consider three baselines. **FullSent**: treating each sentence as a proposition. **PDTB-conn**: further segmenting sentences when any discourse connective (collected from Penn Discourse Treebank Prasad et al. (2007)) is observed. **RST-parser**: segmenting discourse units by the RST parser in Feng and Hirst (2014).

For learning-based methods, we start with Conditional Random Field (**CRF**) Lafferty et al. (2001) with features proposed by Stab and Gurevych ((2017), Table 7), and **BiLSTM-CRF**, a bidirectional Long Short-Term Memory network (BiLSTM) connected to a CRF output layer and further enhanced with ELMo representation Peters et al. (2018). We adopt the BIO scheme for sequential tagging Ramshaw and Marcus (1999), with O corresponding to Non-arg. Finally, we consider **jointly modeling** segmentation and classification by appending the proposition types to BI tags, e.g., B-fact, with CRF (**CRF-joint**) and BiLSTM-CRF (**BiLSTM-CRF-joint**).

Table 4 shows that BiLSTM-CRF outperforms other methods in F1. More importantly, the performance on reviews is lower than those reached on existing datasets, e.g., an F1 of \(86.7\) is obtained by CRF for essays Stab and Gurevych (2017). This is mostly due to essays' better structure, with frequent use of discourse connectives.

### Task II: Proposition Classification

With given proposition segments, predicted or gold-standard, we experiment with proposition-level models to label proposition types.

We utilize two baselines. **Majority** simply assigns the majority type in the training set. **PropLexicon** matches the following lexicons for different proposition types in order, and returns the first corresponding type with a match; if no lexicon is matched, the proposition is labeled as NonARG:

* Reference: <URL>, <CIT>
* Quote: ",","
* Request: _should, would be nice, why, please, would like to, need_
* Evaluation: _highly, very, unclear, clear, interesting, novel, well, important, similar, clearly, quite, good_
* Fact: _author, authors, propose, present, method, parameters, example, dataset, same, incorrect, correct_

\begin{table}
\begin{tabular}{l c c c} \hline \hline  & Prec. & Rec. & F1 \\ \hline FullSent & 73.68 & 56.00 & 63.64 \\ PDTB-conn & 51.11 & 49.71 & 50.40 \\ RST-parser & 30.28 & 43.00 & 35.54 \\ CRF & 66.53 & 52.92 & 58.95 \\ BiLSTM-CRF & **82.25** & **79.96** & **81.09*** \\ CRF-joint & 74.99 & 63.33 & 68.67 \\ BiLSTM-CRF-joint & 81.12 & 78.42 & 79.75 \\ \hline \hline \end{tabular}
\end{table}
Table 4: Proposition segmentation results. Result that is significantly better than all comparisons is marked with \(*\) (\(p<10^{-6}\), McNemar test).

\begin{table}
\begin{tabular}{l l l l l l l} \hline \hline Eval & Rep & Fact & Ref & Quot & Non-A & Total \\ \hline
3,982 & 1,911 & 3,786 & 207 & 161 & 339 & 10,386 \\ \hline \hline \end{tabular}
\end{table}
Table 3: Number of propositions per type in AMPERE.

\begin{table}
\begin{tabular}{l r r r r r} \hline \hline  & Overall & Eval & Req & Fact & Ref & Quot \\ \hline _With Gold-Standard Segments_ & & & & & \\ Majority & 40.75 & 57.90 & – & – & – & – \\ PropLexicon & 36.83 & 40.42 & 36.07 & 32.23 & 59.57 & 31.28 \\ SVM & 60.98 & 63.88 & **69.02** & 54.74 & **69.47** & 7.69 \\ CNN & **66.56*** & **69.02** & 63.26 & **66.17** & 67.44 & **52.94** \\ \hline \multicolumn{5}{l}{_With Predicted Segments_} \\ Majority & 33.30 & 47.60 & – & – & – & – \\ PropLexicon & 23.21 & 22.45 & 23.97 & 23.73 & 35.96 & 16.67 \\ SVM & 51.46 & 54.05 & 48.16 & 52.77 & 52.27 & 4.71 \\ CNN & 55.48 & 57.75 & 53.71 & 55.19 & 48.78 & 33.33 \\ CRF-joint & 50.69 & 46.78 & 55.74 & 52.27 & **55.77** & 26.47 \\ BiLSTM-CRF- & **62.64*** & **62.36*** & **67.31*** & **61.86** & 54.74 & **37.36** \\ joint & & & & & \\ \hline \hline \end{tabular}
\end{table}
Table 5: Proposition classification F1 scores. Results that are significant better than other methods are marked with \(*\) (\(p<10^{-6}\), McNemar test).

 

For supervised models, we employ linear **SVM** with a squared hinge loss and group Lasso regularizer Yuan and Lin (2006). It is trained with the top \(500\) features selected from Table 9 in Stab and Gurevych (2017) by \(\chi^{2}\) test. We also train a convolutional neural network (**CNN**) proposed by Kim Kim (2014), with the same setup and pre-trained word embeddings from word2vec Mikolov et al. (2013). Finally, results by joint models of CRF and BiLSTM-CRF are also reported.

F1 scores for all propositions and each type are reported in Table 5. A prediction is correct when both segment and type are matched with the true labels. CNN performs better for types with significantly more training samples, i.e., evaluation and fact, indicating the effect of data size on neural model's performance. Joint models (CRF-joint and BiLSTM-CRF-joint) yield the best F1 scores for all categories when gold-standard segmentation is unavailable.

## 4 Proposition Analysis by Venues

Here we leverage the BiLSTM-CRF-joint model trained on the annotated AMPERE data to identify propositions and their types in unlabeled reviews from the four venues (ICLR, UAI, ACL, and NeurIPS), to understand the content and structure of peer reviews at a larger scale.

**Proposition Usage by Venue and Rating.** Figure 2 shows the average number of propositions per review, grouped by venue and rating. Scores in \(1-10\) are scaled to \(1-5\) by \(\lceil x/2\rceil\), with \(1\) as strong reject and \(5\) as strong accept. ACL and NeurIPS have significantly more propositions than ICLR and UAI. Ratings, which reflect a reviewer's judgment of paper quality, also affect proposition usage. We find that reviews with extreme ratings, i.e., \(1\) and \(5\), tend to have fewer propositions.

We further study the distribution of proposition type in each venue. As observed in Figure 3, ACL reviews contain more requests but fewer facts than other venues. Specifically, we find that \(94.6\%\) of ACL reviews have at least one request proposition, compared to \(81.5\%\) for ICLR and \(84.7\%\) for UAI. We also show proposition type distribution based on ratings in Figure 4. Reviews with the highest rating tend to use fewer evaluation and reference, while reviews with ratings of \(3-4\) (borderline or weak accept) contain more requests. We further observe a sharp decrease of Quote usage in rating group \(4\), and a surge of Non-arg for rating group \(5\), while Fact remains consistent across rating ranges.

**Proposition Structure.** Argumentative structure, which is usually studied as support and attack relations, reveals how propositions are organized into coherent text. According to Park and Cardie Park and Cardie (2018), \(75\%\) of support relations happen between adjacent propositions in user comments. We thus plot the proposition transition probability matrix in Figure 5, to show the argument structure in AMPERE. The high probabilities along the diagonal line imply that propositions of the same type are often constructed consecutively, with the exception of quote, which is more likely to be followed by evaluation.

**Proposition Type and Content.** We also probe the salient words used for each proposition type, and the difference of their usage across venues. For each venue, we utilize log-likelihood ratio test Lin and Hovy (2000) to identify the represen

Figure 4: Distribution of proposition type per rating (in \(\%\)) on AMPERE.

Figure 3: Distribution of proposition type per venue.

Figure 2: Proposition number in reviews. Differences among venues are all significant except UAI vs. ICLR and ACL vs. NeurIPS (\(p<10^{-6}\), unpaired \(t\)-test).

 

tative words in each proposition type compared to other types. Table 6 shows both the commonly used salient words across venues and the unique words with top frequencies for each venue (\(\alpha=0.001\), \(\chi^{2}\) test). For evaluation, all venues tend to focus on clarity and contribution, with ICLR discussing more about "network" and NeurIPS often mentioning equations. ACL reviews then frequently request for "examples".

## 5 Related Work

There is a growing interest in understanding the content and assessing the quality of peer reviews. Authors' feedback such as satisfaction and helpfulness have been adopted as quality indicators Latu and Everett (2000); Hart-Davidson et al. (2010); Xiong and Litman (2011). Nonetheless, they suffer from author subjectivity and are often influenced by acceptance decisions Weber et al. (2002). Evaluation by experts or editors proves to be more reliable and informative van Rooyen et al. (1999), but requires substantial work and knowledge of the field. Shallow linguistic features, e.g., sentiment words, are studied in Bornmann et al. Bornmann et al. (2012) for analyzing languages in peer reviews. To the best of our knowledge, our work is the first to understand the content and structure of peer reviews via argument usage.

Our work is also in line with the growing body of research in argument mining Teufel et al. (1999); Palau and Moens (2009). Most of the work focuses on arguments in social media posts Park and Cardie (2014); Wei et al. (2016); Habernal and Gurevych (2016), online debate portals or Oxfordstyle debates Wachsmuth et al. (2017); Hua and Wang (2017); Wang et al. (2017), and student essays Persing and Ng (2015); Ghosh et al. (2016). We study a new domain of peer reviews, and identify new challenges for existing models.

## 6 Conclusion

We study the content and structure of peer reviews under the argument mining framework. AMPERE, a new dataset of peer reviews, is collected and annotated with propositions and their types. We benchmark AMPERE with state-of-the-art argument mining models for proposition segmentation and classification. We leverage the classifiers to analyze the proposition usage in reviews across ML and NLP venues, showing interesting patterns in proposition types and content.

## Acknowledgements

This research is based upon work supported in part by National Science Foundation through Grants IIS-1566382 and IIS-1813341. We are grateful to the OpenReview team, especially Michael Spector, for setting up the API to facilitate review data collection. We also thank three anonymous reviewers for their constructive suggestions.

Figure 5: Proposition transition prob. on AMPERE.

 



## References

* Al Khatib et al. (2016) Khalid Al Khatib, Henning Wachsmuth, Johannes Kiesel, Matthias Hagen, and Benno Stein. 2016. A news editorial corpus for mining argumentation strategies. In _Proceedings of COLING 2016, the 26th International Conference on Computational Linguistics: Technical Papers_, pages 3433-3443. The COLING 2016 Organizing Committee.
* Bornmann et al. (2012) Lutz Bornmann, Markus Wolf, and Hans-Dieter Daniel. 2012. Closed versus open reviewing of journal manuscripts: how far do comments differ in language use? _Scientometrics_, 91(3):843-856.
* Feng and first (2014) Vanessa Wei Feng and Graeme Hirst. 2014. A linear-time bottom-up discourse parser with constraints and post-editing. In _Proceedings of the 52nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_, pages 511-521. Association for Computational Linguistics.
* Ghosh et al. (2016) Debanjan Ghosh, Aquila Khanam, Yubo Han, and Smaranda Muresan. 2016. Coarse-grained argumentation features for scoring persuasive essays. In _Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers)_, pages 549-554. Association for Computational Linguistics.
* Habernal and Gurevych (2016) Ivan Habernal and Iryna Gurevych. 2016. Which argument is more convincing? analyzing and predicting convincingness of web arguments using bidirectional lstrn. In _Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_, pages 1589-1599. Association for Computational Linguistics.
* Habernal and Gurevych (2017) Ivan Habernal and Iryna Gurevych. 2017. Argumentation mining in user-generated web discourse. _Computational Linguistics_, 43(1):125-179.
* Hart-Davidson et al. (2010) William Hart-Davidson, Michael McLeod, Christopher Klerkx, and Michael Wojcik. 2010. A method for measuring helpfulness in online peer review. In _Proceedings of the 28th ACM international conference on design of communication_, pages 115-121. ACM.
* Hettich and Pazzani (2006) Seth Hettich and Michael J Pazzani. 2006. Mining for proposal reviewers: lessons learned at the national science foundation. In _Proceedings of the 12th ACM SIGKDD international conference on Knowledge discovery and data mining_, pages 862-871. ACM.
* Hua and Wang (2017) Xinyu Hua and Lu Wang. 2017. Understanding and detecting supporting arguments of diverse types. In _Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers)_, pages 203-208, Vancouver, Canada. Association for Computational Linguistics.
* Kang et al. (2018) Dongyeop Kang, Waleed Ammar, Bhavana Dalvi, Madeleine van Zuylen, Sebastian Kohlmeier, Eduard Hovy, and Roy Schwartz. 2018. A dataset of peer reviews (peerread): Collection, insights and nlp applications. In _Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers)_, pages 1647--1661. Association for Computational Linguistics.
* Kelly et al. (2014) Jacalyn Kelly, Tara Sadeghieh, and Khosrow Adeli. 2014. Peer review in scientific publications: benefits, critiques, & a survival guide. _EJIFCC_, 25(3):227.
* Kim (2014) Yoon Kim. 2014. Convolutional neural networks for sentence classification. In _Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP)_, pages 1746-1751. Association for Computational Linguistics.
* Kovanis et al. (2016) Michail Kovanis, Raphael Porcher, Philippe Ravaud, and Ludovic Trinquart. 2016. The global burden of journal peer review in the biomedical literature: Strong imbalance in the collective enterprise. _PLoS One_, 11(11):e0166387.
* Krippendorff (2004) Klaus Krippendorff. 2004. Measuring the reliability of qualitative text analysis data. _Quality and Quantity_, 38:787-800.
* Lafferty et al. (2001) John Lafferty, Andrew McCallum, and Fernando CN Pereira. 2001. Conditional random fields: Probabilistic models for segmenting and labeling sequence data.
* Latu and Everett (2000) Tavite M Latu and Andre M Everett. 2000. _Review of satisfaction research and measurement approaches_. Citeseer.
* Lin and Hovy (2000) Chin-Yew Lin and Eduard Hovy. 2000. The automated acquisition of topic signatures for text summarization. In _COLING 2000 Volume 1: The 18th International Conference on Computational Linguistics_.
* Mikolov et al. (2013) Tomas Mikolov, Ilya Sutskever, Kai Chen, Greg S Corrado, and Jeff Dean. 2013. Distributed representations of words and phrases and their compositionality. In _Advances in neural information processing systems_, pages 3111-3119.
* Niculae et al. (2017) Vlad Niculae, Joonsuk Park, and Claire Cardiie. 2017. Argument mining with structured svms and rnns. In _Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_, pages 985-995. Association for Computational Linguistics.
* Palau and Moens (2009) Raquel Mochales Palau and Marie-Francine Moens. 2009. Argumentation mining: the detection, classification and structure of arguments in text. In _Proceedings of the 12th international conference on artificial intelligence and law_, pages 98-107. ACM.
* Park et al. (2015) Joonsuk Park, Cheryl Blake, and Claire Cardiie. 2015. Toward machine-assisted participation in erulemaking: An argumentation model of evaluability. In _Proceedings of the 15th International Conference on Artificial Intelligence and Law_, pages 206-210. ACM

Joonsuk Park and Claire Cardie. 2014. Identifying appropriate support for propositions in online user comments. In _Proceedings of the First Workshop on Argumentation Mining_, pages 29-38. Association for Computational Linguistics.
* Park and Cardie (2018) Joonsuk Park and Claire Cardie. 2018. A corpus of erulemaking user comments for measuring evaluability of arguments. In _Proceedings of the Eleventh International Conference on Language Resources and Evaluation (LREC-2018)_. European Language Resource Association.
* Peldszus and Stede (2013) Andreas Peldszus and Manfred Stede. 2013. From argument diagrams to argumentation mining in texts: A survey. _International Journal of Cognitive Informatics and Natural Intelligence (IJCINI)_, 7(1):1-31.
* Persing and Ng (2015) Isaac Persing and Vincent Ng. 2015. Modeling argument strength in student essays. In _Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural Language Processing (Volume 1: Long Papers)_, pages 543-552. Association for Computational Linguistics.
* Peters et al. (2018) Matthew Peters, Mark Neumann, Mohit Iyyer, Matt Gardner, Christopher Clark, Kenton Lee, and Luke Zettlemoyer. 2018. Deep contextualized word representations. In _Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers)_, pages 2227-2237. Association for Computational Linguistics.
* Prasad et al. (2007) Rashmi Prasad, Eleni Miltsakaki, Nikhil Dinesh, Alan Lee, Aravind Joshi, Livio Robaldo, and Bonnie L Webber. 2007. The penn discourse treebank 2.0 annotation manual.
* Price and Flach (2017) Simon Price and Peter A Flach. 2017. Computational support for academic peer review: A perspective from artificial intelligence. _Communications of the ACM_, 60(3):70-79.
* Ramshaw and Marcus (1999) Lance A Ramshaw and Mitchell P Marcus. 1999. Text chunking using transformation-based learning. In _Natural language processing using very large corpora_, pages 157-176. Springer.
* van Rooyen et al. (1999) Susan van Rooyen, Nick Black, and Fiona Godlee. 1999. Development of the review quality instrument (rqi) for assessing peer reviews of manuscripts. _Journal of clinical epidemiology_, 52(7):625-629.
* Schulz et al. (2018) Claudia Schulz, Steffen Eger, Johannes Daxenberger, Tobias Kahse, and Iryna Gurevych. 2018. Multitask learning for argumentation mining in low-resource settings. In _Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 2 (Short Papers)_, pages 35-41. Association for Computational Linguistics.
* Stab and Gurevych (2017) Christian Stab and Iryna Gurevych. 2017. Parsing argumentation structures in persuasive essays. _Computational Linguistics_, 43(3):619-659.
* Teufel et al. (1999) Simone Teufel et al. 1999. _Argumentative zoning: Information extraction from scientific text_. Ph.D. thesis, Citeseer.
* Wachsmuth et al. (2017) Henning Wachsmuth, Nona Naderi, Ivan Habernal, Yufang Hou, Graeme Hirst, Iryna Gurevych, and Benno Stein. 2017. Argumentation quality assessment: Theory vs. practice. In _Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers)_, pages 250-255. Association for Computational Linguistics.
* Wang et al. (2017) Lu Wang, Nick Beauchamp, Sarah Shugars, and Kechen Qin. 2017. Winning on the merits: The joint effects of content and style on debate outcomes. _Transactions of the Association for Computational Linguistics_, 5:219-232.
* Weber et al. (2002) Ellen J Weber, Patricia P Katz, Joseph F Waeckerle, and Michael L Callaham. 2002. Author perception of peer review: impact of review quality and acceptance on satisfaction. _JAMA_, 287(21):2790-2793.
* Wei et al. (2016) Zhongyu Wei, Yang Liu, and Yi Li. 2016. Is this post persuasive? ranking argumentative comments in online forum. In _Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers)_, pages 195-200. Association for Computational Linguistics.
* Xiong and Litman (2011) Wenting Xiong and Diane Litman. 2011. Automatically predicting peer-review helpfulness. In _Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies_, pages 502-507. Association for Computational Linguistics.
* Yuan and Lin (2006) Ming Yuan and Yi Lin. 2006. Model selection and estimation in regression with grouped variables. _Journal of the Royal Statistical Society: Series B (Statistical Methodology)_, 68(1):49-67.