SlowGuess/NAACL_2021_OCR huggingface dataset:
Thanks to the anonymous reviewers for their constructive feedback, and to all the #NLProc Twitter people for discussion.
ACL Reviewing Committee. 2020a. ACL rolling review proposal. Archived by the Internet Archive on June 20, 2020.
ACL Reviewing Committee. 2020b. Short-term reform proposals for acl reviewing. Adopted by the ACL Exec on June 8 as an initial step to improve reviewing. Archived by the Internet Archive on October 30, 2020.
Emily M. Bender. 2011. On achieving and evaluating language-independence in NLP. Linguistic Issues in Language Technology, 6(3):1-26.
Emily M. Bender, Leon Derczynski, and Pierre Isabelle, editors. 2018. Proceedings of the 27th International Conference on Computational Linguistics. Association for Computational Linguistics, Santa Fe, New Mexico, USA.
Emily M. Bender and Batya Friedman. 2018. Data statements for natural language processing: Toward mitigating system bias and enabling better science. Transactions of the Association for Computational Linguistics, 6:587-604.
Sarai Blincoe and Stephanie Buchert. 2020. Research preregistration as a teaching and learning tool in undergraduate psychology courses. *Psychology Learning & Teaching*, 19(1):107-115.
António Branco, Nicoletta Calzolari, Piek Vossen, Gertjan Van Noord, Dieter van Uytvanck, João Silva, Luís Gomes, André Moreira, and Willem Elbers. 2020. A shared task of a new, collaborative type to foster reproducibility: A first exercise in the area of language science and technology with REPROLANG2020. In Proceedings of the 12th Language Resources and Evaluation Conference, pages 5539-5545, Marseille, France. European Language Resources Association.
Xuan-Nga Cao, Cyrille Dakhlia, Patricia Del Carmen, Mohamed-Amine Jaouani, Malik Ould-Arbi, and Emmanuel Dupoux. 2018. BabyCloud, a technological platform for parents and researchers. In Proceedings of the Eleventh International Conference on Language Resources and Evaluation (LREC 2018), Miyazaki, Japan. European Language Resources Association (ELRA).
Chris Chambers. 2019. What's next for registered reports? Nature, 573:187-189.
K. Bretonnel Cohen, Jingbo Xia, Pierre Zweigenbaum, Tiffany Callahan, Orin Hargraves, Foster
Goss, Nancy Ide, Aurélie Névél, Cyril Grouin, and Lawrence E. Hunter. 2018. Three dimensions of reproducibility in natural language processing. In Proceedings of the Eleventh International Conference on Language Resources and Evaluation (LREC 2018), Miyazaki, Japan. European Language Resources Association (ELRA).
Kevin Cohen, Karen Fort, Margot Mieskes, and Aurelie Néveol. 2020. Reviewing natural language processing research. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics: Tutorial Abstracts, pages 16-18, Online. Association for Computational Linguistics.
Jesse Dodge, Suchin Gururangan, Dallas Card, Roy Schwartz, and Noah A. Smith. 2019. Show your work: Improved reporting of experimental results. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 2185-2194, Hong Kong, China. Association for Computational Linguistics.
Rotem Dror, Gili Baumer, Segev Shlomov, and Roi Reichart. 2018. The hitchhiker's guide to testing statistical significance in natural language processing. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1383-1392, Melbourne, Australia. Association for Computational Linguistics.
Rotem Dror, Lotem Peled-Cohen, Segev Shlomov, and Roi Reichart. 2020. Statistical Significance Testing for Natural Language Processing. Human Language Technologies. Morgan & Claypool.
Jason Eisner. 2010. Write the paper first. Advice published on Jason Eisner's academic webpage, at Johns Hopkins University.
Fiona Fidler and John Wilcox. 2018. Reproducibility of Scientific Results. In Edward N. Zalta, editor, The Stanford Encyclopedia of Philosophy, winter 2018 edition. Metaphysics Research Lab, Stanford University.
Antske Fokkens, Marieke van Erp, Marten Postma, Ted Pedersen, Piek Vossen, and Nuno Freire. 2013. Offspring from reproduction problems: What replication failure teaches us. In Proceedings of the 51st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1691-1701, Sofia, Bulgaria. Association for Computational Linguistics.
Richard Futrell and Roger P. Levy. 2019. Do RNNs learn human-like abstract word order preferences? In Proceedings of the Society for Computation in Linguistics (SCiL) 2019, pages 50-59.
Timnit Gebru, Jamie Morgenstern, Briana Vecchione, Jennifer Wortman Vaughan, Hanna Wallach, Hal Daumé Iii, and Kate Crawford. 2018.
Datasheets for Datasets. In Proceedings of the 5th Workshop on Fairness, Accountability, and Transparency in Machine Learning, Stockholm, Sweden.
Andrew Gelman and Eric Loken. 2013. The garden of forking paths: Why multiple comparisons can be a problem, even when there is no "fishing expedition" or "p-hacking" and the research hypothesis was posited ahead of time. *Department of Statistics*, Columbia University.
Andrew Gelman and Eric Loken. 2014. The statistical crisis in science: data-dependent analysis-a" garden of forking paths"-explains why many statistically significant comparisons don't hold up. American scientist, 102(6):460-466.
Tamarinde L. Haven and Dr. Leonie Van Grootel. 2019. Preregistering qualitative research. Accountability in Research, 26(3):229-244. PMID: 30741570.
David M. Howcroft, Anya Belz, Miruna-Adriana Clinciu, Dimitra Gkatzia, Sadid A. Hasan, Saad Mahamood, Simon Mille, Emiel van Miltenburg, Sashank Santhanam, and Verena Rieser. 2020. Twenty years of confusion in human evaluation: NLG needs evaluation sheets and standardised definitions. In Proceedings of the 13th International Conference on Natural Language Generation, pages 169–182, Dublin, Ireland. Association for Computational Linguistics.
Min-Yen Kan. 2018. "research fast and slow". Keynote presented at COLING 2018, Santa Fe, NM, USA. Slides available through http://bit.ly/kan-coling18.
Wuwei Lan and Wei Xu. 2018. Neural network models for paraphrase identification, semantic textual similarity, natural language inference, and question answering. In Proceedings of the 27th International Conference on Computational Linguistics, pages 3890-3902, Santa Fe, New Mexico, USA. Association for Computational Linguistics.
Alessandro Liberati, Douglas G. Altman, Jennifer Tetzlaff, Cynthia Mulrow, Peter C. Goetzche, John P. A. Ioannidis, Mike Clarke, P. J. Devereaux, Jos Kleijnen, and David Moher. 2009. The prisma statement for reporting systematic reviews and meta-analyses of studies that evaluate health care interventions: Explanation and elaboration. PLOS Medicine, 6(7):1-28.
Malcolm R Macleod, Susan Michie, Ian Roberts, Ulrich Dirnagl, Iain Chalmers, John P A Ioannidis, Rustam Al-Shahi Salman, An-Wen Chan, and Paul Glasziou. 2014. Biomedical research: increasing value, reducing waste. Lancet, 383(9912):101-104.
Sandya Mannarswamy and Shourya Roy. 2018. Evolving ai from research to real life - some challenges and suggestions. In Proceedings of the Twenty-Seventh International Joint Conference on Artificial
Intelligence, IJCAI-18, pages 5172-5179. International Joint Conferences on Artificial Intelligence Organization.
Margaret Mitchell, Simone Wu, Andrew Zaldivar, Parker Barnes, Lucy Vasserman, Ben Hutchinson, Elena Spitzer, Inioluwa Deborah Raji, and Timnit Gebru. 2019. Model Cards for Model Reporting. In Proceedings of the Conference on Fairness, Accountability, and Transparency - FAT* '19, pages 220-229, Atlanta, GA, USA. ACM Press.
David Moher, Alessandro Liberati, Jennifer Tetzlaff, Douglas G. Altman, and The PRISMA Group. 2009. Preferred reporting items for systematic reviews and meta-analyses: The prisma statement. PLOS Medicine, 6(7):1-6.
Andrew Moore and Paul Rayson. 2018. Bringing replication and reproduction together with generalisability in NLP: Three reproduction studies for target dependent sentiment analysis. In Proceedings of the 27th International Conference on Computational Linguistics, pages 1132-1144, Santa Fe, New Mexico, USA. Association for Computational Linguistics.
Marcus R. Munafò, Brian A. Nosek, Dorothy V. M. Bishop, Katherine S. Button, Christopher D. Chambers, Nathalie Percie du Sert, Uri Simonsohn, Eric-Jan Wagenmakers, Jennifer J. Ware, and John P. A. Ioannidis. 2017. A manifesto for reproducible science. Nature Human Behaviour, 1(1):0021.
Brian A. Nosek, Charles R. Ebersole, Alexander C. DeHaven, and David T. Mellor. 2018. The preregistration revolution. Proceedings of the National Academy of Sciences, 115(11):2600-2606.
Open Science Collaboration. 2015. Estimating the reproducibility of psychological science. Science, 349(6251).
Benjamin Recht, Rebecca Roelofs, Ludwig Schmidt, and Vaishaal Shankar. 2019. Do ImageNet classifiers generalize to ImageNet? In Proceedings of Machine Learning Research, volume 97, pages 5389-5400, Long Beach, California, USA. PMLR.
Ehud Reiter. 2018. A structured review of the validity of BLEU. Computational Linguistics, 44(3):393-401.
Timo Roettger. 2020. Preregistration in experimental linguistics: Applications, challenges, and limitations. Preprint available at: https://psyarxiv.com/vc9hu/. DOI: https://doi.org/10.31234/osf.io/vc9hu.
Anna Rogers and Isabelle Augenstein. 2020. What can we do to improve peer review in NLP? In Findings of the Association for Computational Linguistics: EMNLP 2020, pages 1256-1262, Online. Association for Computational Linguistics.
Josef Ruppenhofer, Michael Wiegand, Rebecca Wilm, and Katja Markert. 2018. Distinguishing affixoid formations from compounds. In Proceedings of the 27th International Conference on Computational Linguistics, pages 3853-3865, Santa Fe, New Mexico, USA. Association for Computational Linguistics.
Anastasia Shimorina and Anya Belz. 2021. The human evaluation datasheet 1.0: A template for recording details of human evaluation experiments in nlp.
Joe Simmons, Leif Nelson, and Uri Simonsohn. 2017. How to properly preregister a study. Data Colada.
Emma Strubell, Ananya Ganesh, and Andrew McCallum. 2019. Energy and policy considerations for deep learning in NLP. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 3645-3650, Florence, Italy. Association for Computational Linguistics.
Aba Szollosi, David Kellen, Danielle J Navarro, Richard Shiffrin, Iris van Rooij, Trisha Van Zandt, and Chris Donkin. 2020. Is preregistration worthwhile? Trends in cognitive sciences, 24(2):94-95.
Laure Thompson and David Mimno. 2018. Authorless topic models: Biasing models away from known structure. In Proceedings of the 27th International Conference on Computational Linguistics, pages 3903-3914, Santa Fe, New Mexico, USA. Association for Computational Linguistics.
Chris van der Lee, Albert Gatt, Emiel van Miltenburg, and Emiel Krahmer. 2021. Human evaluation of automatically generated text: Current trends and best practice guidelines. Computer Speech & Language, 67:101151.
Chris van der Lee, Albert Gatt, Emiel van Miltenburg, Sander Wubben, and Emiel Krahmer. 2019. Best practices for the human evaluation of automatically generated text. In Proceedings of the 12th International Conference on Natural Language Generation, pages 355-368, Tokyo, Japan. Association for Computational Linguistics.
Emiel van Miltenburg, Ákos Kárdár, Ruud Koolen, and Emiel Krahmer. 2018. DIDEC: The Dutch image description and eye-tracking corpus. In Proceedings of the 27th International Conference on Computational Linguistics, pages 3658-3669, Santa Fe, New Mexico, USA. Association for Computational Linguistics.
Anna Elisabeth van 't Veer and Roger Giner-Sorolla. 2016. Pre-registration in social psychology—a discussion and suggested template. Journal of Experimental Social Psychology, 67:2-12. Special Issue: Confirmatory.
A Preregistration forms
This appendix provides preregistration forms for different kinds of paper types. These forms are preliminary, and they are mainly meant as a starting point for discussions of preregistration in NLP. We are happy to admit that there may be flaws in this appendix (either in the forms or in our reasoning). Future work should investigate whether these forms are complete (i.e. limit researcher degrees of freedom as much as possible) and appropriate for different kinds of NLP research.
A.1 Preface: data availability in NLP
Preregistration is a means to avoid hindsight bias, because you have to specify your expectations upfront, when your perspective is not yet colored by your experience with the data. But for NLP studies it is unclear what 'the data' is. We can distinguish three kinds of data: 1. The training/validation/test sets, 2. The model output, 3. Human judgments.
In an ideal situation, preregistration would occur before any kind of data has been obtained. The problem is that this is often not the case; there are many canonical datasets for which the data is publicly available. Of course one could collect an additional test set (as we suggested above), but the community often judges new approaches based on their performance for established datasets. So what should we do? Still preregister! Arguably the training, validation, and test data is usually not central to the work. What matters is how a particular system performs. So even if we don't usually find ourselves in the ideal situation where none of the data is available yet, it is typically fine to preregister your study if the train/eval/test data is available but system outputs and evaluation scores are not. When authors are transparent in their data sharing policy, we can reconstruct the timeline of events before and after the preregistration, to see how much their knowledge about the data may have influenced them.
A.2 Computationally aided linguistic analysis
This paper type corresponds to several different setups, ranging from experiments with human subjects, to corpus analyses to see if particular generalisations from the literature hold up. Preregistration has been discussed from a linguistics perspective by Roettger (2020). For experiments with human participants, readers may refer to the standard preregistration forms from AsPredicted (see our Table 1),
OSF, or the questions from Roettger's Figure 1.
For more corpus-oriented studies (e.g. Ruppenhofer et al. 2018), we should consider a mix of the quantitative and qualitative questions from our Table 1. Usually these kinds of studies do require some data collection, so authors should ask:
1. What is the goal of this study?
2. What are the main questions/hypotheses?
3. What kind of data will be collected?
4. How will this data be collected?
5. What sampling strategy will be used? Why?
6. How much data are you planning to collect?
(Is there any target or stopping criterion?)
7. How will the data be analysed?
(a) If automatic: what analysis tool will you use, and how will it be configured?
(b) If manual: what is the background of the annotators? How will you ensure reliability and validity of the analysis?
8. What statistical tests will be used, if any?
9. Anything else you'd like to preregister?
A.3 NLP Engineering experiment paper
NLP engineering experiments are like experiments in the social sciences, except that the subjects are NLP models and the performance data is model output. So the standard social science questions do not need to be modified that much to fit NLP experiments.
1. What is the goal of your study?
2. What are your hypotheses/ key assumptions?
3. What are the (in)dependent variables?
4. How will these variables be measured?
5. Is there just one condition, or more?
6. What software libraries will you use?
7. What hardware will you use?
8. What parameter settings will you use?
9. What data set will you use?
10. If the data set does not already exist, see  $\S A.6$  If it does:
(a) How familiar are you with the data?
(b) To what extent are your hypotheses informed by yourself or others interacting with this data? To what extent does this hinder the generalisability of your approach?
(c) Are you planning to collect additional data to validate your approach?
11. Why this data? What are its key properties?
12. How is the data split in train/val/test?
13. How will you analyse the results and test the hypotheses?
(a) If automatic: what metric(s) (including implementation) will you use, and how will they be configured?
(b) If human judgments: see  $\S A.8.1$
14. Will you carry out an error analysis? If so, see §A.8.2.
15. Anything else you'd like to preregister?
A.4 Position paper
Position papers typically do not need to be preregistered, since they often do not provide any new data, but rely on the author's experience. These kinds of papers also usually signal that they are more opinionated than other kinds of papers.
A.5 Reproduction paper
For a reproduction paper, the questions are a mix of the questions above (§A.3) with reproduction-specific questions.
1. What results do you aim to reproduce?
2. What kinds of experiments does this involve?
3. What is the goal?
(a) What constitutes a successful reproduction?
(b) What constitutes an unsuccessful reproduction?
(c) What is the margin of error?
4. Do you expect to be successful? Why (not)?
5. How are you planning to reproduce the original results?
(a) Will you use the same soft/hardware?
(b) Will you use the same data?
(c) Will you use the same codebase?
(d) If human participants are used: will you target the same demographic, and use the same experimental settings?
(e) Will you contact the authors?
(f) How much time do authors have to respond to your queries?
(g) How much time/effort are you willing to spend?
6. Will you carry out an error analysis? If so, see §A.8.2.
7. Anything else you'd like to preregister?
A.6 Resource paper
It is at least a bit unexpected to promote preregistration for resource papers. After all, if all you do is data collection, then there are no hypotheses to test. But since the goal of this appendix is to provide a starting point for discussion, we are taking the stance that no study is free from biases or initial expectations. As such, it is useful to at least document what you aim to collect, for what reasons, and how you are planning to do so.
1. What is the goal of this study?
2. What kind of data will be collected?
3. How will this data be collected?
4. What is the intended application for the data you plan to collect?
5. What sampling strategy will be used? Why?
6. How much data are you planning to collect? (Is there any target or stopping criterion?)
8. What properties should the data have?
9. How will you ensure that the data will have those properties?
10. Anything else you'd like to preregister?
A.7 Survey paper
We would recommend that authors follow the PRISMA guidelines (Moher et al., 2009; Liberati et al., 2009)) for their surveys. This requires authors to develop a review protocol, which means authors should answer the following questions before initiating their study:
1. What is the goal of this study?
2. What is the rationale behind this study?
3. What questions do you hope to answer?
4. What types of articles are relevant to answer your question?
(a) What are the inclusion criteria?
(b) What are the exclusion criteria?
(c) What languages are included?
5. How will you decide which articles are relevant? (e.g. judging by the title/abstract)
6. What search engines will you use?
7. What search queries will you use?
8. What are the variables of interest?
9. How will you synthesize the results?
10. How will you ensure the reliability and validity of your study?
11. Anything else you'd like to preregister?
A.8 Other kinds of preregistrations
Instead of preregistering a full study, one might also preregister part of a study, e.g. a human evaluation or error analysis.
A.8.1 Human evaluation
Human evaluation studies often do not report all the necessary details to reproduce their work (Howcroft et al., 2020). Thus Shimorina and Belz (2021) developed a datasheet for recording all the necessary details. This datasheet can mostly be filled in before the study is carried out. A selection of their questions is provided below (see the paper for more details and additional questions).
1. What type of input(s) does the system have?
2. What type of output does the system produce?
3. What task is the system supposed to carry out?
4. What languages are involved?
5. How many systems/outputs per system are being evaluated?
6. How are the outputs selected?
7. What is the statistical power of the sample size?
8. What kind of evaluators are being used?
9. What training is given to the evaluators?
10. What is the background if the evaluators?
11. How are responses collected?
12. What quality assurance measures are used?
13. What do evaluators see when carrying out evaluations?
14. How free are evaluators regarding when and how quickly they are supposed to evaluate the results.
15. Can evaluators provide feedback or not?
16. What are the experimental conditions like?
17. What type of quality is assessed in the evaluation?
18. How is this quality assessed?
19. How are the responses processed?
20. What are the ethical implications of your work?
A.8.2 Error analysis
Error analysis is similar to human evaluation, except that it is typically more qualitatively oriented. This does not mean that there cannot be a quantitative component (e.g. counting the number of errors, comparing this number between different systems), but often systems are also just analysed by themselves, and we just want to know what future researchers still ought to improve about the system.
1. What is the goal of the error analysis?
2. What type of input(s) does the system have?
3. What type of output does the system produce?
4. What task is the system supposed to carry out?
5. What languages are involved?
6. What do you know from the literature about system errors?
7. When does something count as an error?
8. What kinds of errors do you expect to find?
9. How many outputs will you analyse?
10. How will you sample the outputs to analyse?
11. Do you also consider the input in your sampling strategy?
12. How do you plan to analyse the output?
13. How many judges will assess the output?
14. What is the background of the judges?
15. What training do the judges receive?
16. How is the reliability of the judges assessed?
17. How will their responses be processed?
18. Is there a fixed error categorisation scheme or not?