Credits
This document has been adapted from the instructions for earlier ACL and NAACL proceedings, including those for ACL 2018 by Shay Cohen, Kevin Gimpel, and Wei Lu, NAACL 2018 by Margaret Michell and Stephanie Lukin, 2017/2018 (NA)ACL bibtex suggestions from Jason Eisner, ACL 2017 by Dan Gildea and Min-Yen Kan, NAACL 2017 by Margaret Mitchell, ACL 2012 by Maggie Li and Michael White, those from ACL 2010 by Jing-Shing Chang and Philipp Koehn, those for ACL 2008 by JohannaD. Moore, Simone Teufel, James Allan, and Sadaoki Furui, those for ACL 2005 by Hwee Tou Ng and Kemal Oflazer, those for ACL 2002 by Eugene Charniak and Dekang Lin, and earlier ACL and EACL formats. Those versions were written by several people, including John Chen, Henry S. Thompson and Donald Walker. Additional elements were taken from the formatting instructions of the International Joint Conference on Artificial Intelligence and the Conference on Computer Vision and Pattern Recognition.
Introduction
The following instructions are directed to authors of papers submitted to NAACL-HLT 2019 or accepted for publication in its proceedings. All authors are required to adhere to these specifications. Authors are required to provide a Portable Document Format (PDF) version of their papers. The proceedings are designed for printing on A4 paper.
General Instructions
Manuscripts must be in two-column format. Exceptions to the two-column format include the title, authors' names and complete addresses, which must be centered at the top of the first page, and any full-width figures or tables (see the guidelines in Subsection "The First Page" ). Type single-spaced. Start all pages directly under the top margin. See the guidelines later regarding formatting the first page. The manuscript should be printed single-sided and its length should not exceed the maximum page limit described in Section "Length of Submission" . Pages are numbered for initial submission. However, do not number the pages in the camera-ready version.
By uncommenting \aclfinalcopy at the top of this document, it will compile to produce an example of the camera-ready formatting; by leaving it commented out, the document will be anonymized for initial submission. When you first create your submission on softconf, please fill in your submitted paper ID where *** appears in the \def\aclpaperid{***} definition at the top.
The review process is double-blind, so do not include any author information (names, addresses) when submitting a paper for review. However, you should maintain space for names and addresses so that they will fit in the final (accepted) version. The NAACL-HLT 2019 style will create a titlebox space of 2.5in for you when \aclfinalcopy is commented out.
The author list for submissions should include all (and only) individuals who made substantial contributions to the work presented. Each author listed on a submission to NAACL-HLT 2019 will be notified of submissions, revisions and the final decision. No authors may be added to or removed from submissions to NAACL-HLT 2019 after the submission deadline.
The Ruler
The NAACL-HLT 2019 style defines a printed ruler which should be presented in the version submitted for review. The ruler is provided in order that reviewers may comment on particular lines in the paper without circumlocution. If you are preparing a document without the provided style files, please arrange for an equivalent ruler to appear on the final output pages. The presence or absence of the ruler should not change the appearance of any other content on the page. The camera ready copy should not contain a ruler. ( users may uncomment the \aclfinalcopy command in the document preamble.)
Reviewers: note that the ruler measurements do not align well with lines in the paper – this turns out to be very difficult to do well when the paper contains many figures and equations, and, when done, looks ugly. In most cases one would expect that the approximate location will be adequate, although you can also use fractional references (e.g., the first paragraph on this page ends at mark $108.5$ ).
Electronically-available resources
NAACL-HLT provides this description in 2e (naaclhlt2019.tex) and PDF format (naaclhlt2019.pdf), along with the 2e style file used to format it (naaclhlt2019.sty) and an ACL bibliography style (acl_natbib.bst) and example bibliography (naaclhlt2019.bib). These files are all available at http://naacl2019.org/downloads/ naaclhlt2019-latex.zip. We strongly recommend the use of these style files, which have been appropriately tailored for the NAACL-HLT 2019 proceedings.
Format of Electronic Manuscript
For the production of the electronic manuscript you must use Adobe's Portable Document Format (PDF). PDF files are usually produced from using the pdflatex command. If your version of produces Postscript files, you can convert these into PDF using ps2pdf or dvipdf. On Windows, you can also use Adobe Distiller to generate PDF.
Please make sure that your PDF file includes all the necessary fonts (especially tree diagrams, symbols, and fonts with Asian characters). When you print or create the PDF file, there is usually an option in your printer setup to include none, all or just non-standard fonts. Please make sure that you select the option of including ALL the fonts. Before sending it, test your PDF by printing it from a computer different from the one where it was created. Moreover, some word processors may generate very large PDF files, where each page is rendered as an image. Such images may reproduce poorly. In this case, try alternative ways to obtain the PDF. One way on some systems is to install a driver for a postscript printer, send your document to the printer specifying “Output to a file”, then convert the file to PDF.
It is of utmost importance to specify the A4 format (21 cm x 29.7 cm) when formatting the paper. When working with dvips, for instance, one should specify -t a4. Or using the command \special{papersize=210mm,297mm} in the latex preamble (directly below the \usepackage commands). Then using dvipdf and/or pdflatex which would make it easier for some.
Print-outs of the PDF file on A4 paper should be identical to the hardcopy version. If you cannot meet the above requirements about the production of your electronic submission, please contact the publication chairs as soon as possible.
Layout
Format manuscripts two columns to a page, in the manner these instructions are formatted. The exact dimensions for a page on A4 paper are:
Left and right margins: 2.5 cm
Top margin: 2.5 cm
Bottom margin: 2.5 cm
Column width: 7.7 cm
Column height: 24.7 cm
Gap between columns: 0.6 cm
Papers should not be submitted on any other paper size. If you cannot meet the above requirements about the production of your electronic submission, please contact the publication chairs above as soon as possible.
Fonts
For reasons of uniformity, Adobe's Times Roman font should be used. In 2e this is accomplished by putting
\usepackage{times}
\usepackage{latexsym}
in the preamble. If Times Roman is unavailable, use Computer Modern Roman (2e's default). Note that the latter is about 10% less dense than Adobe's Times Roman font.
The First Page
Center the title, author's name(s) and affiliation(s) across both columns. Do not use footnotes for affiliations. Do not include the paper ID number assigned during the submission process. Use the two-column format only when you begin the abstract.
Title: Place the title centered at the top of the first page, in a 15-point bold font. (For a complete guide to font sizes and styles, see Table 1 ) Long titles should be typed on two lines without a blank line intervening. Approximately, put the title at 2.5 cm from the top of the page, followed by a blank line, then the author's names(s), and the affiliation on the following line. Do not use only initials for given names (middle initials are allowed). Do not format surnames in all capitals (e.g., use “Mitchell” not “MITCHELL”). Do not format title and section headings in all capitals as well except for proper names (such as “BLEU”) that are conventionally in all capitals. The affiliation should contain the author's complete address, and if possible, an electronic mail address. Start the body of the first page 7.5 cm from the top of the page.
The title, author names and addresses should be completely identical to those entered to the electronical paper submission website in order to maintain the consistency of author information among all publications of the conference. If they are different, the publication chairs may resolve the difference without consulting with you; so it is in your own interest to double-check that the information is consistent.
Abstract: Type the abstract at the beginning of the first column. The width of the abstract text should be smaller than the width of the columns for the text in the body of the paper by about 0.6 cm on each side. Center the word Abstract in a 12 point bold font above the body of the abstract. The abstract should be a concise summary of the general thesis and conclusions of the paper. It should be no longer than 200 words. The abstract text should be in 10 point font.
Text: Begin typing the main body of the text immediately after the abstract, observing the two-column format as shown in the present document. Do not include page numbers.
Indent: Indent when starting a new paragraph, about 0.4 cm. Use 11 points for text and subsection headings, 12 points for section headings and 15 points for the title.
Sections
Headings: Type and label section and subsection headings in the style shown on the present document. Use numbered sections (Arabic numerals) in order to facilitate cross references. Number subsections with the section number and the subsection number separated by a dot, in Arabic numerals. Do not number subsubsections.
Citations: Citations within the text appear in parentheses as BIBREF0 or, if the author's name appears in the text itself, as Gusfield Gusfield:97. Using the provided style, the former is accomplished using \cite and the latter with \shortcite or \newcite. Collapse multiple citations as in BIBREF0 , BIBREF1 ; this is accomplished with the provided style using commas within the \cite command, e.g., \cite{Gusfield:97,Aho:72}. Append lowercase letters to the year in cases of ambiguities. Treat double authors as in BIBREF1 , but write as in BIBREF2 when more than two authors are involved. Collapse multiple citations as in BIBREF0 , BIBREF1 . Also refrain from using full citations as sentence constituents.
We suggest that instead of
“ BIBREF0 showed that ...”
you use
“Gusfield Gusfield:97 showed that ...”
If you are using the provided and Bib style files, you can use the command \citet (cite in text) to get “author (year)” citations.
If the Bib file contains DOI fields, the paper title in the references section will appear as a hyperlink to the DOI, using the hyperref package. To disable the hyperref package, load the style file with the nohyperref option:
\usepackage[nohyperref]{naaclhlt2019}
Digital Object Identifiers: As part of our work to make ACL materials more widely used and cited outside of our discipline, ACL has registered as a CrossRef member, as a registrant of Digital Object Identifiers (DOIs), the standard for registering permanent URNs for referencing scholarly materials. As of 2017, we are requiring all camera-ready references to contain the appropriate DOIs (or as a second resort, the hyperlinked ACL Anthology Identifier) to all cited works. Thus, please ensure that you use Bib records that contain DOI or URLs for any of the ACL materials that you reference. Appropriate records should be found for most materials in the current ACL Anthology at http://aclanthology.info/.
As examples, we cite BIBREF3 to show you how papers with a DOI will appear in the bibliography. We cite BIBREF4 to show how papers without a DOI but with an ACL Anthology Identifier will appear in the bibliography.
As reviewing will be double-blind, the submitted version of the papers should not include the authors' names and affiliations. Furthermore, self-references that reveal the author's identity, e.g.,
“We previously showed BIBREF0 ...”
should be avoided. Instead, use citations such as
“ BIBREF0 Gusfield:97 previously showed ... ”
Any preliminary non-archival versions of submitted papers should be listed in the submission form but not in the review version of the paper. NAACL-HLT 2019 reviewers are generally aware that authors may present preliminary versions of their work in other venues, but will not be provided the list of previous presentations from the submission form.
Please do not use anonymous citations and do not include when submitting your papers. Papers that do not conform to these requirements may be rejected without review.
References: Gather the full set of references together under the heading References; place the section before any Appendices. Arrange the references alphabetically by first author, rather than by order of occurrence in the text. By using a .bib file, as in this template, this will be automatically handled for you. See the \bibliography commands near the end for more.
Provide as complete a citation as possible, using a consistent format, such as the one for Computational Linguistics or the one in the Publication Manual of the American Psychological Association BIBREF5 . Use of full names for authors rather than initials is preferred. A list of abbreviations for common computer science journals can be found in the ACM Computing Reviews BIBREF6 .
The and Bib style files provided roughly fit the American Psychological Association format, allowing regular citations, short citations and multiple citations as described above.
Example citing an arxiv paper: BIBREF7 .
Example article in journal citation: BIBREF8 .
Example article in proceedings, with location: BIBREF9 .
Example article in proceedings, without location: BIBREF10 .
See corresponding .bib file for further details.
Submissions should accurately reference prior and related work, including code and data. If a piece of prior work appeared in multiple venues, the version that appeared in a refereed, archival venue should be referenced. If multiple versions of a piece of prior work exist, the one used by the authors should be referenced. Authors should not rely on automated citation indices to provide accurate references for prior and related work.
Appendices: Appendices, if any, directly follow the text and the references (but see above). Letter them in sequence and provide an informative title: Appendix A. Title of Appendix.
Footnotes
Footnotes: Put footnotes at the bottom of the page and use 9 point font. They may be numbered or referred to by asterisks or other symbols. Footnotes should be separated from the text by a line.
Graphics
Illustrations: Place figures, tables, and photographs in the paper near where they are first discussed, rather than at the end, if possible. Wide illustrations may run across both columns. Color illustrations are discouraged, unless you have verified that they will be understandable when printed in black ink.
Captions: Provide a caption for every illustration; number each one sequentially in the form: “Figure 1. Caption of the Figure.” “Table 1. Caption of the Table.” Type the captions of the figures and tables below the body, using 10 point text. Captions should be placed below illustrations. Captions that are one line are centered (see Table 1 ). Captions longer than one line are left-aligned (see Table 2 ). Do not overwrite the default caption sizes. The naaclhlt2019.sty file is compatible with the caption and subcaption packages; do not add optional arguments.
Accessibility
In an effort to accommodate people who are color-blind (as well as those printing to paper), grayscale readability for all accepted papers will be encouraged. Color is not forbidden, but authors should ensure that tables and figures do not rely solely on color to convey critical distinctions. A simple criterion: All curves and points in your figures should be clearly distinguishable without color.
Translation of non-English Terms
It is also advised to supplement non-English characters and terms with appropriate transliterations and/or translations since not all readers understand all such characters and terms. Inline transliteration or translation can be represented in the order of: original-form transliteration “translation”.
Length of Submission
The NAACL-HLT 2019 main conference accepts submissions of long papers and short papers. Long papers may consist of up to eight (8) pages of content plus unlimited pages for references. Upon acceptance, final versions of long papers will be given one additional page – up to nine (9) pages of content plus unlimited pages for references – so that reviewers' comments can be taken into account. Short papers may consist of up to four (4) pages of content, plus unlimited pages for references. Upon acceptance, short papers will be given five (5) pages in the proceedings and unlimited pages for references. For both long and short papers, all illustrations and tables that are part of the main text must be accommodated within these page limits, observing the formatting instructions given in the present document. Papers that do not conform to the specified length and formatting requirements are subject to be rejected without review.
NAACL-HLT 2019 does encourage the submission of additional material that is relevant to the reviewers but not an integral part of the paper. There are two such types of material: appendices, which can be read, and non-readable supplementary materials, often data or code. Do not include this additional material in the same document as your main paper. Additional material must be submitted as one or more separate files, and must adhere to the same anonymity guidelines as the main paper. The paper must be self-contained: it is optional for reviewers to look at the supplementary material. Papers should not refer, for further detail, to documents, code or data resources that are not available to the reviewers. Refer to Appendix "Appendices" and Appendix "Supplemental Material" for further information.
Workshop chairs may have different rules for allowed length and whether supplemental material is welcome. As always, the respective call for papers is the authoritative source.
Acknowledgments
The acknowledgments should go immediately before the references. Do not number the acknowledgments section. Do not include this section when submitting your paper for review.
Preparing References:
Include your own bib file like this: \bibliographystyle{acl_natbib} \begin{thebibliography}{40}
Gabor Angeli, Melvin Jose Johnson Premkumar, and Christopher D Manning. 2015. Leveraging linguistic structure for open domain information extraction. In Proc. ACL '15/IJCNLP '15, pages 344–354.
Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. 2015. Neural machine translation by jointly learning to align and translate. In Proc. ICLR '15.
Michele Banko, Michael J. Cafarella, Stephen Soderland, Matt Broadhead, and Oren Etzioni. 2007. Open information extraction from the web. In Proc. IJCAI '07, pages 2670–2676.
Jonathan Berant, Andrew Chou, Roy Frostig, and Percy Liang. 2013. Semantic parsing on freebase from question-answer pairs. In Proc. EMNLP '13, pages 1533–1544.
Nikita Bhutani, HV Jagadish, and Dragomir Radev. 2016. Nested propositions in open information extraction. In Proc. EMNLP '16, pages 55–64.
Antoine Bordes, Nicolas Usunier, Alberto Garcia-Duran, Jason Weston, and Oksana Yakhnenko. 2013. Translating embeddings for modeling multi-relational data. In Proc. NIPS '13, pages 2787–2795.
Kyunghyun Cho, Bart van Merrienboer, Caglar Gulcehre, Dzmitry Bahdanau, Fethi Bougares, Holger Schwenk, and Yoshua Bengio. 2014. Learning phrase representations using RNN encoder–decoder for statistical machine translation. In Proc. EMNLP '14, pages 1724–1734.
Lei Cui, Furu Wei, and Ming Zhou. 2018. Neural open information extraction. In Proc. ACL '18, pages 407–413.
Dorottya Demszky, Kelvin Guu, and Percy Liang. 2018. Transforming question answering datasets into natural language inference datasets. arXiv preprint arXiv:1809.02922.
Anthony Fader, Stephen Soderland, and Oren Etzioni. 2011. Identifying relations for open information extraction. In Proc. EMNLP '11, pages 1535–1545.
Ben Hixon, Peter Clark, and Hannaneh Hajishirzi. 2015. Learning knowledge graphs for question answering through conversational dialog. In Proc. NAACL-HLT '15, pages 851–861.
Zhiheng Huang, Wei Xu, and Kai Yu. 2015. Bidirectional LSTM-CRF models for sequence tagging. arXiv preprint arXiv:1508.01991.
Srinivasan Iyer, Ioannis Konstas, Alvin Cheung, Jayant Krishnamurthy, and Luke Zettlemoyer. 2017. Learning a neural semantic parser from user feedback. In Proc. ACL '17, pages 963–973.
Prachi Jain, Shikhar Murty, Mausam, and Soumen Chakrabarti. 2018. Mitigating the effect of out-of-vocabulary entity pairs in matrix factorization for KB inference. In Proc. IJCAI '18, pages 4122–4129.
Guillaume Klein, Yoon Kim, Yuntian Deng, Jean Senellart, and Alexander Rush. 2017. OpenNMT: Open-source toolkit for neural machine translation. In Proc. ACL '17 (System Demonstrations), pages 67–72.
Minh-Thang Luong, Quoc V Le, Ilya Sutskever, Oriol Vinyals, and Lukasz Kaiser. 2015a. Multi-task sequence to sequence learning. In Proc. ICLR '16.
Minh-Thang Luong, Hieu Pham, and Christopher D. Manning. 2015b. Effective approaches to attention-based neural machine translation. In Proc. EMNLP '15, pages 1412–1421.
Laurens van der Maaten and Geoffrey Hinton. 2008. Visualizing data using $t$ -SNE. Journal of Machine Learning Research, 9(Nov):2579–2605.
Mausam, Michael Schmitz, Robert Bart, Stephen Soderland, Oren Etzioni, et al. 2012. Open language learning for information extraction. In Proc. EMNLP '12, pages 523–534.
Julian McAuley and Alex Yang. 2016. Addressing complex and subjective product-related queries with customer reviews. In Proc. WWW '16, pages 625–635.
Maximilian Nickel, Lorenzo Rosasco, and Tomaso Poggio. 2016. Holographic embeddings of knowledge graphs. In Proc. AAAI '16, pages 1955–1961.
Jeffrey Pennington, Richard Socher, and Christopher Manning. 2014. GloVe: Global vectors for word representation. In Proc. EMNLP '14, pages 1532–1543.
Nils Reimers and Iryna Gurevych. 2017. Reporting score distributions makes a difference: Performance study of LSTM-networks for sequence tagging. In Proc. EMNLP '17, pages 338–348.
Subhashree S and P Sreenivasa Kumar. 2018. Enriching domain ontologies using question-answer datasets. In Proc. CoDS-COMAD '18, pages 329–332.
Swarnadeep Saha, Harinder Pal, et al. 2017. Bootstrapping for numerical open ie. In Proc. ACL '17, pages 317–323.
Denis Savenkov, Wei-Lwun Lu, Jeff Dalton, and Eugene Agichtein. 2015. Relation extraction from community generated question-answer pairs. In Proc. NAACL-HLT '15, pages 96–102.
Gabriel Stanovsky and Ido Dagan. 2016. Creating a large benchmark for open information extraction. In Proc. EMNLP '16.
Gabriel Stanovsky, Julian Michael, Luke Zettlemoyer, and Ido Dagan. 2018. Supervised open information extraction. In Proc. ACL '18, pages 885–895.
Antonio Toral and Víctor M. Sánchez-Cartagena. 2017. A multifaceted evaluation of neural versus phrase-based machine translation for 9 language directions. In Proc. EACL '17, pages 1063–1073.
Oriol Vinyals, Meire Fortunato, and Navdeep Jaitly. 2015. Pointer networks. In Proc. NIPS '15, pages 2692–2700.
Mengting Wan and Julian McAuley. 2016. Modeling ambiguity, subjectivity, and diverging viewpoints in opinion question answering systems. In Proc. ICDM '16, pages 489–498.
Quan Wang, Zhendong Mao, Bin Wang, and Li Guo. 2017. Knowledge graph embedding: A survey of approaches and applications. IEEE Transactions on Knowledge and Data Engineering, 29(12):2724–2743.
Zeqiu Wu, Xiang Ren, Frank F. Xu, Ji Li, and Jiawei Han. 2018. Indirect supervision for relation extraction using question-answer pairs. In Proc. WSDM '18, pages 646–654.
Chunyang Xiao, Marc Dymetman, and Claire Gardent. 2016. Sequence-based structured prediction for semantic parsing. In Proc. ACL '16, pages 1341–1350.
Caiming Xiong, Victor Zhong, and Richard Socher. 2017. Dynamic coattention networks for question answering. In Proc. ICLR '17.
Alexander Yates, Michael Cafarella, Michele Banko, Oren Etzioni, Matthew Broadhead, and Stephen Soderland. 2007. TextRunner: Open information extraction on the web. In Proc. NAACL-HLT '07 (Demonstrations), pages 25–26.
Pengcheng Yin and Graham Neubig. 2017. A syntactic neural model for general-purpose code generation. In Proc. ACL '17, pages 440–450.
Biao Zhang, Deyi Xiong, and Jinsong Su. 2016. Cseq2seq: Cyclic sequence-to-sequence learning. arXiv preprint arXiv:1607.08725.
Yaoyuan Zhang, Zhenxu Ye, Yansong Feng, Dongyan Zhao, and Rui Yan. 2017. A constrained sequence-to-sequence neural model for sentence simplification. arXiv preprint arXiv:1704.02312.
Barret Zoph and Kevin Knight. 2016. Multi-source neural translation. In Proc. NAACL-HLT '16, pages 30–34.
|
where naaclhlt2019 corresponds to a naaclhlt2019.bib file. Appendices Appendices are material that can be read, and include lemmas, formulas, proofs, and tables that are not critical to the reading and understanding of the paper. Appendices should be uploaded as supplementary material when submitting the paper for review. Upon acceptance, the appendices come after the references, as shown here. Use \appendix before any appendix section to switch the section numbering over to letters. Supplemental Material Submissions may include non-readable supplementary material used in the work and described in the paper. Any accompanying software and/or data should include licenses and documentation of research review as appropriate. Supplementary material may report preprocessing decisions, model parameters, and other details necessary for the replication of the experiments reported in the paper. Seemingly small preprocessing decisions can sometimes make a large difference in performance, so it is crucial to record such decisions to precisely characterize state-of-the-art methods. Nonetheless, supplementary material should be supplementary (rather than central) to the paper. Submissions that misuse the supplementary material may be rejected without review. Supplementary material may include explanations or details of proofs or derivations that do not fit into the paper, lists of features or feature templates, sample inputs and outputs for a system, pseudo-code or source code, and data. (Source code and data should be separate uploads, rather than part of the paper). The paper should not rely on the supplementary material: while the paper may refer to and cite the supplementary material and the supplementary material will be available to the reviewers, they will not be asked to review the supplementary material.

Question: How did they evaluate the system?
Answer: Unanswerable

Question: Where did they get training data?
Answer: AmazonQA and ConciergeQA datasets

Question: What extraction model did they use?
Answer: Multi-Encoder, Constrained-Decoder model

Question: Which datasets did they experiment on?
Answer: ConciergeQA and AmazonQA

Question: What types of facts can be extracted from QA pairs that can't be extracted from general text?
Answer: Unanswerable