SlowGuess/NAACL_2021_OCR huggingface dataset:
The 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies
NAACL-HLT 2021
June 6 - 11, 2021
Diamond Sponsors:
Platinum Sponsors:
Gold Sponsors:
Silver Sponsors:
Bronze Sponsors:
Diversity and Inclusion Champions:
©2021 The Association for Computational Linguistics
ISBN 978-1-954085-46-6
It is my pleasure to welcome you to the virtual NAACL-HLT 2021 conference! Although conditions did not allow us to hold the conference in Mexico City as originally planned, we hope our rich program of presentations, poster sessions, discussions, and social events will enable fruitful scientific exchange and increase our connectedness as a community. The increased affordability could also allow us to welcome new members that would not be able to attend a physical conference.
In this message I would first like to highlight a few initiatives and sessions at NAACL 2021 and then acknowledge the many people on our organizing committee and those of prior conferences that were critical to making it happen.
Following suggestions by Dan Jurafsky and members of the NAACL executive board, this year we followed a more well-defined process for ethics reviews<sup>1</sup>, proposed by the ethics co-chairs Emily Bender and Karen Fort, and refined and implemented in coordination with the program co-chairs and the demo and industry track chairs. Key changes were allowing additional space in submissions to discuss ethical considerations and establishing a category of papers accepted conditionally on addressing ethical concerns together with a timeline and process for an additional stage of review of re-submissions. We are grateful to Emily and Karen for setting these foundations and hope our community will continue to improve its process of education about and review of the ethical implications of our research.
Another initiative this year that Graham Neubig suggested, Luciana Benotti, Thamar Solario, Smaranda Muresan, and other members of the NAACL Exec significantly contributed to, and Pranav A and the rest of the D&I committee strongly advocated for and worked hard on was the D&I Grant Initiative.2 With the help of our sponsors, we were able to waive registration and membership fees for authors from underrepresented developing countries, as well as many others that could not have attended otherwise.
Also new this year was a software package developed by the publication chairs Ryan Cotterell, Steven Bethard, Yichao Zhou, Iz Beltagy, and Tanmoy Chakroborty to automatically check and report formatting violations in an easy-to-understand way. This significant contribution to the infrastructure used by the ACL community went above and beyond the duties of publication chairs for a single conference.
The NAACL virtual conference will host 6 tutorials, 17 system demonstrations, 39 industry track papers, 499 main conference and CL/TACL papers, 6 plenary invited talks, 2 panels, 22 workshops, and a large set of social and thematic gatherings. This was made possible by the hard work of the many members of our organizing committee:
- Anna Rumshisky, Luke Zettlemoyer, and Dilek Hakkani-Tur, our program co-chairs, have contributed the most by leading the selection of the scientific content for the main conference.
- Priscilla Rasmussen arranged our transition to a virtual presence and provided guidance on nearly every aspect of the organization.
- Industry Track Chairs (Owen Rambow, Yunyao Li, and Young-Bum Kim), who advocated for the inclusion of this track and led the selection of 39 papers and additional invited talks and panels.
- Demonstration Track Chairs (Avi Sil and Victoria Lin), who organized the selection of 17 system demonstrations.
Workshop Chairs (Bhavana Dalvi, Mamoru Komachi, and Michel Galley) who led an efficient and organized process for the workshops despite the uncertainty of the conference format.
<https://2021.naacl.org/ethics/faq/
$^{2}$ https://2021.naacl.org/blog/dei-grants/
- Tutorial Chairs (Greg Kondrak, Kalina Bontcheva, and Dan Gillick) who selected and coordinated the presentation of six tutorials.
- Student Research Workshop Chairs (Esin Durmus, Nelson Liu, Vivek Gupta) and Faculty Advisors (Nanyun Peng and Yu Su) who selected 22 research papers and thesis proposals.
- Ethics Chairs (Emily Bender and Karen Fort), who substantially improved the process for ethics review and education.
- Publication Chairs (Ryan Cotterell, Steven Bethard, Yichao Zhou, Iz Beltagy, and Tanmoy Chakroborty) who improved the publication infrastructure through the effort described above.
- Diversity and Inclusion Chairs (Pranav A, Samira Shaikh, Pat Verga, Murathan Kurfali, Khyati Mahajan, and Prathyusha Jwalapuram) and Social Chairs (Luca Soldaini and Sabine Weber) who established the D&I grant initiative, took steps to strengthen the presence of affinity groups at NAACL, coordinated the organization of many socials, mentoring events, and topical discussions, ensured improved accessibility, and distributed financial support to community members in need.
- Publicity Chairs (Sarah Wegreffe, Enrico Santus, Peng Qi, and Danqi Chen) who made it possible for the program co-chairs and general chair to not have to check Twitter regularly, enabled efficient communication, and initiated a creative way for members to be introduced to the community via a PeopleOfNLPProc Blog.
- Volunteer Chair (Hao Cheng) who took the important responsibility of coordinating the work of more than a hundred volunteers.
- Virtual Infrastructure Committee (Deepak Ramachandran, Mauricio Mazuecos, Martin Villalba) for stepping up to secure the foundations of the virtual conference and Advisors (Jan-Christoph Klie, Hao Fang, and Gisela Vallejo) for taking time to point us in the right direction.
- Website Chairs (Ice Pasupat and Iulia Turc) who posted information to the website extremely quickly and in beautiful arrangement.
- Volunteers: More than a hundred volunteers that will help lead the live sessions and ensure information on the website is correct.
I am also grateful to Bonnie Webber, general chair of EMNLP, and Anna Rogers for helping me set expectations on the difficulty of organizing a virtual conference and major challenges to watch out for, and Donia Scott, Horacio Saggion, and Leo Wanner for sharing their experience with Underline, our virtual conference provider, with us. I would also like to acknowledge Sol Rosenberg and Daniel Luise from Underline for arranging everything on a short timeline.
Colin Cherry, David Yarowsky, and other members of the NAACL exec provided valuable advice at multiple decision points.
Finally, I would like to thank all authors of papers, invited talks, and panels, area chairs, and reviewers, and the volunteers organizing and chairing sessions, and all attendees and readers of this volume for engaging with the content and the community.
Kristina Toutanova
NAACL 2021 General Chair
Welcome to the 2021 Annual Conference of the North American Association for Computational Linguistics! NAACL-HLT 2021 is a completely virtual conference, in response to the COVID-19 pandemic, which will still be felt around the world for some time to come. We are grateful for the support and contributions of the entire NAACL 2021 team. Without them, it would not have been possible to organize an exciting and memorable event during very stressful global times.
We largely adopted the best practices of other recent virtual conferences, while modifying the format somewhat to treat all papers equally, as described in more detail below. Our paper review process followed the recent trend of a hierarchical organization, with senior area chairs (SACs) that organized coherent research tracks and area chairs (ACs) who shepherded smaller batches of papers within each track. We made an effort to balance the tracks. Recent NLP conferences have had many tracks that received well over 200 submissions, making them mini conferences of their own. To make the senior area chairs jobs more manageable, we split the machine learning track by areas (Classification and Structured Prediction Models and Language Modeling and Sequence to Sequence Models) and separated Machine Translation from Multilingual. We also wrote a brief guide to authors to help them decide which track was most appropriate for their work.<sup>3</sup> Overall, we felt that this did make the organization more manageable and that other tracks, including NLP applications, could possibly benefit from further splitting in future conferences. Otherwise, we followed recent traditions in track selection, including keeping some of the smaller, more recent additions (e.g. Green NLP and Ethics).
We also had a special theme for the conference, which we called "New Challenges in NLP: Tasks, Methods, Positions." This theme was selected to recognize that we have made significant progress in NLP over the last five years, and that the community could benefit from thinking about the new problems and upcoming challenges we should focus on next. Despite the general applicability of the unsupervised pre-training/fine-tuning paradigm, many problems are still very challenging for current models. At the same time, given the recent progress, there are likely broad new classes of problems that can now be studied for the first time. What tasks or capabilities should we focus on next? What new classes of models should we be investigating? We envisioned papers falling into this theme including (but not limited to) (1) empirical and dataset papers that propose new challenges that bring us closer to human-level language understanding and generation, and (2) position papers framing an important direction or highlighting an understudied research problem.
We recruited reviewers through a centralized process, designed to minimize workload for senior area chairs (SACs) without sacrificing review quality. We collected a list of likely qualified reviewers based on the reviewer and author pool of other recent NLP conferences. These candidates were invited to sign up to review, and were required to fill out a profile that allowed us to better assess their potential area fit and experience levels. We were fortunate enough to have enough volunteers to not need everyone, and were able to bias the final selection towards more senior reviewers, although many junior and first time reviewers remained in the pool. The final program included 54 SACs, 267 ACs, and 1941 committee members. We greatly appreciate the incredible amount of work they all did, and also thank all of the volunteers who were not selected in the end.
NAACL-HLT 2021 received 1797 submissions—a record for our conference! We accepted 477 papers, including 350 long and 127 short, for an overall acceptance rate of  $26\%$ . The acceptance rate for long papers was higher than short papers (28% vs. 23%), although this gap was smaller than in other recent conferences. From the accepted papers, and based on the nominations from SACs and review by the best paper committee, we selected best papers in the long and short paper categories, as well as a small number of outstanding papers in each category. NAACL-HLT 2021 will also feature 18 papers that were published at Transactions of the Association for Computational Linguistics (TACL) and 4 papers from
<sup>3</sup>https://2021.naacl.org/calls/area-descriptions/
the journal of Computational Linguistics (CL).
Our virtual format includes both interactive talk sessions and posters in Gather.Town, following recent successful online events. A 12-minute video will also be available for long papers and a 7-minute video for short papers. We made a significant change by removing the distinction between oral and poster papers, and instead treat every paper the same. The papers are grouped into sessions, where each session has two parts. The first  $50\mathrm{min}$  will be a live Zoom-like session with oral pitches for each paper, including 5 minutes for long papers and 3 minutes for short papers, followed by 3 minutes of Q&A for each paper. The remainder of the interactive session will happen in Gather.Town, with poster presentations for each paper. The goal is to allow the audience to engage into an in-depth discussion with the authors if desired. This is new model that likely will not scale to very large conference, but will hopefully allow for rich interactions at the conference while only requiring each author to be present for a single session. The sessions were also arranged across time zones to, as much as possible, be in normal waking hours for the main contact authors of each paper. Finally, another highlight of our program is keynote talks from Dhruv Batra, Shakir Mohamed, Hinrich Schütze, and Thamar Solorio. These four excellent speakers were chosen to provide exciting and thought-provoking perspectives from both within and outside of our the core NLP.
NAACL 2021 is truly a community-run effort. We want to second all of Kristina's acknowledgements in the General Chair statement. We have also had the pleasure to work with these folks, and appreciate all of their help. We would additionally like to thank:
- The Senior Area Chairs, who were incredibly organized and responsive, in every step from assigning reviewers to making final decisions. They were a crucial support for the very large area chair and reviewer pools.
- The Area Chairs who led paper review discussions and wrote meta-reviews.
- The primary reviewers and secondary reviewers who provided valuable feedback to the authors. Special thanks to those who stepped in at the last minute to serve as emergency reviewers.
- Our distinguished Best Paper Committee: Isabelle Augenstein, Marco Baroni, Jacob Eisenstein, Hanna Hajishirzi, Omer Levy, Jessy Li, Yang Liu, Chris Quirk, Barbara Di Eugenio, and Bonnie Webber.
- The authors who submitted their work to NAACL-HLT 2021. Although we could not accept many of the strong papers, we know that most of it will end up at other amazing venues, and hope we were able to provide some useful feedback.
- TACL editors-in-chief Mark Johnson, Ani Nenkova, and Brian Roark, TACL Editorial Assistant Cindy Robinson, and CL Editor-in-Chief Hwee Tou Ng for coordinating TACL and CL presentations with us.
- The Program co-Chairs of NAACL 2019, Christy Doran and Thamar Solorio; of ACL 2020, Joyce Chai, Natalie Schluter, and Joel Tetreault; and of EMNLP 2020, Trevor Cohn, Yulan He, and Yang Liu. You were all amazingly available and generous with your time to answer the very many questions we had about how to run a successful program committee.
And last but not least, our General Chair Kristina Toutanova. She has done an excellent job of steering a large ship in very challenging times!
We sincerely appreciate your help, and hope you will enjoy the NAACL-HLT 2021 conference!
Dilek Hakkani-Tur, Anna Rumshisky, and Luke Zettlemoyer
NAACL 2021 Program Committee Co-Chairs
Kristina Toutanova, Google Research
Anna Rumshisky, University of Massachusetts Lowell Luke Zettlemoyer, University of Washington and Facebook AI Research Dilek Hakkani-Tur, Amazon Alexa AI
Local Chair:
Priscilla Rasmussen, Association for Computational Linguistics
Diversity & Inclusion Senior Chairs:
Pranav A, Miro AI  
Samira Shaikh, University of North Carolina - Charlotte  
Pat Verga, Google Research
Diversity & Inclusion Student Chairs:
Murathan Kurfali, Stockholm University  
Khyati Mahajan, University of North Carolina - Charlotte  
Prathyusha Jwalapuram, Nanyang Technological University
Diversity & Inclusion Social Chairs:
Luca Soldaini, Amazon Alexa AI Sabine Weber, University of Edinburgh
Avi Sil, IBM Research AI Victoria Lin, Facebook AI
Ethics Chairs:
Emily Bender, University of Washington  
Karen Fort, Sorbonne Université
Industry Track Chairs:
Owen Rambow, Stony Brook University  
Yunyao Li, IBM Research  
Young-Bum Kim, Amazon
Iz Beltagy, AI2  
Steven Bethard, University of Arizona  
Ryan Cotterell, ETH Zurich and University of Cambridge  
Tanmoy Chakraborty, IIIT-Delhi  
Yichao Zhou, University of California, Los Angeles
SarahWiegrefe,Georgia Institute of Technology   
Enrico Santus,Bayer   
Peng Qi,JD AI   
Danqi Chen,Princeton
Esin Durmus, Cornell University  
Nelson Liu, Stanford University  
Vivek Gupta, University of Utah
Student Research Workshop Faculty Advisors:
Nanyun Peng, University of California, Los Angeles  
Yu Su, Ohio State University
Greg Kondrak, University of Alberta  
Kalina Bontcheva, University of Sheffield  
Dan Gillick, Google Research
Volunteer Chairs:
Hao Cheng, Microsoft Research
Deepak Ramachandran, Google Research
Virtual Infrastructure Committee Advisors:
Jan-Christoph Klie, UKP Lab, Technical University of Darmstadt  
Hao Fang, Microsoft Semantic Machines, USA  
Gisela Vallejo, Independent research
Website Chairs:
Ice Pasupat, Google Research  
Iulia Turc, Google Research
Bhavana Dalvi, AI2  
Mamoru Komachi, Tokyo Metropolitan University  
Michel Galley, Microsoft Research
Senior Program Committee
Computational Social Science and Cultural Analytics:
SACs: David Bamman, Chenhao Tan  
ACs: Dallas Card, Paramveer Dhillon, Lucie Flek, Kenny Joseph, David Mimno, Dong Nguyen, Brendan O'Connor, Daniel Preotiac-Pietro, Sara Tonelli, Svetlana Volkova, Justine Zhang
SACs: Maxine Eskenazi, Kallirroi Georgila, Jiwei Li  
ACs: Ryuichiro Higashinaka, Casey Kennington, Kazunori Komatani, Sungjin Lee, Pierre Lison, Jose David Lopes, Ramesh Manuvinakurike, Michael McTear, Alexandros Papangelis, Rebecca Passonneau, Gabriel Skantze, David Traum, Stefan Ultes, Nigel Ward, Jason Williams, Koichiro Yoshino, Kai Yu, Tiancheng Zhao
SACs: Giuseppe Carenini, Junyi (Jessy) Li  
ACs: Malihe Alikhani, Shafiq Joty, Sujian Li, Annie Louis, Deyi Xiong
Ethics, Bias, and Fairness:
SACs: Kai-Wei Chang  
ACs: Vinodkumar Prabhakaran, Rachel Rudinger
Green NLP:
SACs: Roy Schwartz  
ACs: Niranjan Balasubramanian, Emma Strubell
Language Generation:
SACs: Asli Celikyilmaz, Nanyun Peng, Wei Xu  
ACs: Antoine Bosselut, Greg Durrett, Claire Gardent, Iryna Gurevych, Srinivasan Iyer, Mohit Iyyer, Yangfeng Ji, Mirella Lapata, Mike Lewis, Ankur Parikh, Laura Perez-Beltrachini, Sudha Rao, Hannah Rashkin, Mark Riedl, John Wieting
SACs: Heng Ji, Zornitsa Kozareva, William Wang  
ACs: Gerard de Melo, Luciano Del Corro, Bhuwan Dhingra, Ruihong Huang, Lifu Huang, Arzoo Katiyar, Ni Lao, Yi Luan, Ndapa Nakashole, Vincent Ng, Thien Huu Nguyen, Qiang Ning, Sujith Ravi, Siliang Tang, Derry Tanti Wijaya, Bishan Yang, Dian Yu, Sheng Zhang
SACs: Eugene Agichtein, Alessandro Moschitti
ACs: Qingyao Ai, Ahmed Hassan Awadallah, Simone Filice, Karthik Raman, Oleg Rokhlenko, Luca Soldaini, Yi Tay, Olga Uryupina, Thuy Vu, Hamed Zamani
SACs: Yonatan Belinkov, Grzegorz Chrupala, Anders Søgaard
ACs: Jasmijn Bastings, Allyson Ettinger, Katja Filippova, Antske Fokkens, Dieuwke Hupkes, Hung-yi Lee, Zachary Lipton, Lilja Øvrelid, Anna Rogers, Benjamin Roth, Hassan Sajjad, Wojciech Samek, Ian Tenney, Xin (Eric) Wang, Willem Zuidema
SACs: Jason Baldridge, Mohit Bansal
ACs: Aishwarya Agrawal, Joyce Chai, Spandana Gella, David Harwath, Lisa Anne Hendricks, Angeliki Lazaridou, Karthik Narasimhan, Jesse Thomason, Mark Yatskar
Language Resources and Evaluation:
SACs: Sam Bowman, Matt Gardner
ACs: Pradeep Dasigi, Kyle Gorman, Joakim Nivre, Saku Sugawara, Alane Suhr, Reut Tsarfaty, Aaron White, Adina Williams
Linguistic Theories, Cognitive Modeling and Psycholinguistics:
SACs: William Schuler
ACs: Richard Futrell, Kasia Hitzenko, Emily Prud'hommeaux, Marten van Schijndel
Machine Learning for NLP: Classification and Structured Prediction Models:
SACs: Alan Ritter, Sasha Rush, Vivek Srikumar
ACs: Yoav Artzi, Isabelle Augenstein, Mathieu Blondel, Shay Cohen, Caio Corro, Dan Goldwasser, Yoon Kim, Lingpeng Kong, Paul Michel, Lili Mou, Vlad Niculae, Gabriel Stanovsky, Karl Stratos, Swabha Swayamdipta
Machine Learning for NLP: Language Modeling and Sequence to Sequence Models:
SACs: Kevin Gimpel, Omer Levy
ACs: Roee Aharoni, Michael Auli, Taylor Berg-Kirkpatrick, Li Dong, Tatsunori Hashimoto, Andre Martins, Matthew Peters, Colin Raffel, Rico Sennrich, Lei Yu
SACs: George Foster, Maja Popovic, Matt Post
ACs: Antonis Anastasopoulos, Eleftherios Avramidis, Michael Denkowski, Mark Fishel, Alexander Fraser, Markus Freitag, Yvette Graham, Paco Guzmán, Christian Hardmeier, Carolina Scarton, Dimitar Shterionov, Michel Simard, Marco Turchi, Eva Vanmassenhove, David Vilar, Rui Wang
Multilinguality:
SACs: Manaal Faruqui, Radu Florian
ACs: Marine Carpuat, Orhan First, Dan Garrette, Katharina Kann, Sara Rosenthal, Sakriani Sakti, Ves Stoyanov, Shyam Upadhyay, Zhiguo Wang, Imed Zitouni
SACs: Yun-Nung Vivian Chen, Jing Huang, Nitin Madnani, Byron Wallace
ACs: Andrew O. Arnold, Aoife Cahill, Chris Donahue, Vivek Kulkarni, Rohit Kumar, Cheng-Te Li, Yang Liu, Tristan Naumann, Hoifung Poon, Peng Qi, Nazneen Fatema Rajani, Vikram Ramanarayanan, Kevin Small, Gokhan Tur, Sowmya Vajjala, Karin Verspoor, Guangtao Wang, Youzheng Wu, Bing Xiang, Pengtao Xie, Helen Yannakoudakis, Torsten Zesch, Yuhao Zhang
SACs: Ryan Cotterell
ACs: Xuanjing Huang, Francis Tyers, Ekaterina Vylomova
SACs: Danqi Chen, Hannaneh Hajishirzi
ACs: Jonathan Berant, Jordan Boyd-Graber, Kyunghyun Cho, Eunsol Choi, Nan Duan, Kelvin Guu, Robin Jia, Daniel Khashabi, Tushar Khot, Jimmy Lin, Xiang Ren, Minjoon Seo, Pontus Stenetorp, Huan Sun
Semantics: Lexical Semantics:
SACs: Ivan Vulić
ACs: Goran Glavaš, Taher Pilehvar, Roi Reichart, Steven Schockaert, Ekaterina Shutova, Aline Villavicencio
Semantics: Sentence-level Semantics and Textual Inference:
SACs: Daniel Cer, Douwe Kiela, Wenpeng Yin
ACs: Heike Adel, Roy Bar-Haim, Iz Beltagy, Eduardo Blanco, Shiyu Chang, Daniel Hershcovich, Adam Poliak, Christopher G. Potts, Siva Reddy, Marek Rei, German Rigau, Cicero Nogueira dos Santos, Yangqiu Song, Yinfei Yang, Mo Yu
Sentiment Analysis and Stylistic Analysis:
SACs: Preslav Nakov, Diyi Yang
ACs: Tanmoy Chakraborty, Lun-Wei Ku, Soujanya Poria, Paolo Rosso, Thamar Solorio, Aixin Sun, Arkaitz Zubiaga, Yulan He, Roman Klinger
Speech:
SAC: Eric Fosler-Lussier
ACs: Erica Cooper, Yao Qian, Yuxuan Wang
SACs: Michael Elhadad, Yue Zhang
ACs: Arman Cohan, Sebastian Gehrmann, Logan Lebanoff, Fei Liu, Pengfei Liu, Shashi Narayan, Manabu Okumura, Xiaojun Wan, Rui Zhang
Syntax: Tagging, Chunking, and Parsing:
SAC: Kenji Sagae
ACs: Marco Kuhlmann, Miryam de Lhoneux, Jennifer Foster, David Chiang, Weiwei Sun, Miguel Ballesteros
Special Theme: New Challenges in NLP:
SAC: Kenneth Church
ACs: Chris Callison-Burch, Valia Kordoni, Sabine Schulte im Walde, Costanza Navarretta
Conflict of Interest (COI):
SAC: Graham Neubig
ACs: Monoji Choudhury, Anna Korhonen, Wei Lu, Kathy McKeown, Noah A. Smith
Ethics Committee:
Alain Couillault, Alvin Grissom II, Amanda Stent, Annemarie Friedrich, Aurélie Néveol, Paris Saclay, Chen Li, Claudia Borg, Dan Jurafsky, Darja Fiser, Dirk Hovy, George K. Acquaah-Mensah, Gilles ADDA, Jade Abbott, Jean-Yves Antoine, Jin-Dong Kim, Jingbo Xia, Jochen L. Leidner, José Ochoa-Luna, Karl Pichotta, Kevin Bretonnel Cohen, Laura Alonso Alemany, Luciana Benotti, Luis Chiruzzo, Malvina Nissim, Manny Rayner, Margot Mieskes, Maxime Amblard, Min-Yen Kan, Mona Diab, Natalie Schluter, Nyalleng Moorosi, Qingcai Chen, Robert Monarch, Ryan Georgi, Te Rutherford, Thiago Castro Ferreira, Vukosi Marivate, Wondwissen Mulugeta