The ACL OCL Corpus: Advancing Open Science in Computational Linguistics
Authors: Shaurya Rohatgi
Yanxia Qin
Benjamin Aw
Niranjana Anand Unnithan
Min-Yen Kan
Venue: EMNLP 2023 Main (2023)
Keywords: scholarly corpus
computational linguistics
topic trend analysis
acl anthology
TL;DR: We present ACL OCL (https://huggingface.co/datasets/WINGNUS/ACL-OCL), a scholarly corpus derived from the ACL Anthology to assist Open scientific research in the Computational Linguistics domain.

Abstract: We present ACL OCL, a scholarly corpus derived from the ACL Anthology to assist Open scientific research in the Computational Linguistics domain. Integrating and enhancing the previous versions of the ACL Anthology, the ACL OCL contributes metadata, PDF files, citation graphs and additional structured full texts with sections, figures, and links to a large knowledge resource (Semantic Scholar). The ACL OCL spans seven decades, containing 73K papers, alongside 210K figures. 

We spotlight how ACL OCL applies to observe trends in computational linguistics. By detecting paper topics with a supervised neural model, we note that interest in “Syntax: Tagging, Chunking and Parsing” is waning and “Natural Language Generation” is resurging. Our dataset is available from HuggingFace (https://huggingface.co/datasets/WINGNUS/ACL-OCL).

The ACL OCL Corpus: Advancing Open Science in Computational Linguisties
Shaurya Rohatgi Yanxia Qin Benjamin Aw Niranjana Unnithan Min-Yen Kan
College of Information Sciences and Technology, Pennsylvania State University Schooo of Computing, National University of Singapore
szr207@psu.edee edu, {qinyx, kanmy)@comp..nus edu.sg sg
Abstract
Mefa
Metadata
PDF
PDF files
Full text type
We present ACL OCL, a scholarly corpus derived from the ACL Anthology to assist Open scientific research in the Computational Linguistics domain. Integrating and enhanc- ing the previous versions of the ACL Anthol- ogy, the ACL OCL contributes metadata, PDF files, citation graphs and additional structured full texts with sections, figures, and links to large knowledge resource (Semantic Scholar). ACL. OCI. spans seven decades, containing 73K papers, alongside 210K figures.
Figure
Linked KG
Citation graph
Meta
struct
OCL (73.3K)
2022
ACL Anthology
2007
2009
2016
ARC (10.9K)
AAN (25K)
We spotlight how ACL OCL applies to ob- serve trends in computational linguistics. By detecting paper topies with a supervised neural model, note that interest in "Syntax: Täg- ging. Chunking and Parsing" is waning and "Natural Language Generation" is resurging. Our dataset is available from HuggingFace'.
Meta
plain
PDF
Meta
plain
The ACL Anthology and the corpora built upon it. Compared to ARC and AAN, our OCL includes more data with additional structured full text, figures, and links to the Semantic Scholar academic graph.
Figure 1:
1 Introduction
Building scholarly corpora for open research ac- celerates scientific progress and promotes repro- ducibility research by providing researchers with accessible and standardized data resources. Driven by advancements in natural language processing and machine learning technologies, the computa- tional linguistics (CL. discipline has experienced rapid growth in recent years. This rapid growth un- derpins the importance of having scholarly corpus in the CL domain for ensuring sustainable progress.
Author Network (AAN; Radev al., 2009), ex- tend its utility by providing full texts, citation and collaboration networks. However, both are becom- ing obsolete due to their outdated text extraction methods and insufficient updates.
We present the ACL OCL (or OCL for short), an enriched and contemporary scholarly corpus that builds upon the strengths of its predecessors while addressing their limitations. The OCL cor- pus includes 73,285 papers hosted by the ACL An- thology published from 1952 to September 2022. The OCL further provides higher-quality structured full texts for all papers, instead of previous string- formatted ones, enabling richer textual analyses. For instance, higher-quality full texts better foster the development of document-leved information traction tasks (Jain al., 2020; Das al., 2022). The structured information in full texts, such as sec- tions and paragraphs, facilitates section-wie tasks such as related work generation (Hoang and Kan, 2010) and enables fine- fine-grained grained linguistic analyses
The ACI Anthology² is an important resource that digitally archives all scientific papers the CL domain, including metadata, PDF files, and sup- plementary materials. Previous scholarly corpora built on it, such the Anthology Reference Cor- pus (ARC: Bird et al., 2008) and the Anthology
"Equal contribution
Correspondence author
Work done during internship NUS.
'https: hnttps:/huugingfaeee http://ggiinn.a....oaaatateesese wttpee/..ttaatttee//iaaaaa/
ACL-OCL
Phttps: /taktpps//aclanthooogg..oo

---

Name
#Doc
Text Type
Linked KG
Fig.
Peer
Source www
Domain
RefSeer (Huang et al., 2015)
1.0M
string
CiteSeerX
partial
multi
S2ORC (Lo ct al., 2020)
8.1M
structured
S2AG
partial
multi
multi
CSL (Li et al., 2022)
396K
self
all
CCJ
multi
unarXive (Saicr ct al., 2023)
1.9M
structured
MAG
partial
arXiv
multi
ACL ARC (Bird et al., 2008)
10.9K
string
self
all
ACL
CL
AAN (Radev et al., 2009)
25K
string
self
all
ACL
CL
ACL OCL (Ours)
73.3K
structured
S2AG
all
ACL
CL
Table 1:
Comparison between ACL OCL and existing scholar corpora. Text type means the type of full text. "Peer" means whether the scientific document peer-reviewed. N.B., S2ORC contains papers from multiple sources including arv, ACL (42K), and PMC. CCJ is short for Chinese Core Journal.
(Jiang et al., 2020a).
Related Work
In addition, to advance multimodal research such figure caption generation (Hsu al., 2021), OCL extracts 210K figures from its source documents. Furthermore, we link OCL to large-scale scientific knowledge graphs to enrich OCL with external information. In particular, we consider informa- tion such as citation data from larger scholarly database (e.g.. Semantic Scholar) and linkage to other platforms (e-g., arXiv).
Scholarly datasets typically fall into two categories: task-specific and open research-oriented The for- mer, designed to serve one task, includes selective information of scientific papers such as abstract and citation strings, paired with task-specific outputs such as entities (Hou et al., 2021), summaries (Ca- chola et al., 2020) and citation intent labels (Cohan et al., 2019). In contrast, open research- -erercc- scholarly datasets aim to provide comprehensive and fundamental information about scientific pa- pers, such metadata and full text. The open scholarly datasets not only facilitate researchers in refining their task-specific data but also aid analyzing the characteristics of scientific papers or groups of them. Our work in line with the open scholarly datasct construction to serve a wide range of applications.
To showcase the scientific value of the OCL cor- pus, illustrate its utility through downstream application of temporal topic trends (Hall al., 2008; Gollapalli and Li, 2015) within the CL do- main. We first train a pre-trained language model (PLM) based topic classification method on a sub- set of 2,545 scientific papers with ground truth topic labels. We then extrapolate and integrate the model predictions silver-labeled topic informa- tion in the OCL.
Similar other datasets, the OCL features publi- cation metadata, staple in open scholarly datasets. This can enhance metadata analysis and biblio- graphic research within the CL domain. compar- ison of the OCL with existing datasets, excluding metadata aspects, is presented in Table 1. The OCL distinguishes itself from other corpora by the target domain, focusing on the computational lin- guistic domain same ARC and AAN3. Their common source (i.e., ACL Anthology) provides higher-quality scientific papers, which are all peer- reviewed by domain experts. In contrast them, the enriched and updated OCL corpus includes more papers and information.
The contributions of this paper are as follows:
We construct the ACL OCL, which augments the source ACL Anthology. The OCL pro- vides structured full text for 73.3K CL papers, enriches metadata originated from Anthology by linking to an external knowledge graph, and extracts 210K figures. We also analyze the OCL, disclosing its statisties and the qual- ity of full texts.
We conduct a case study on OCL's temporal topic trends, showing the emergence of new topies like "Ethics" and witnessing the past glory of "Machine Translation". We validate the importance of supervised data in topic clas- sification. Model- Mooeel-peddiced silver topic labels are released together with OCL.
Inspired by and following S2ORC, the OCL pro- vides structured full texts with the scientific docu- ments' discourse structure (i.c., sections), which en- ables more extensive textual analysis. In contrast corpora that rely solely on internal papers for cita-
/https://clair. secs, umich. edu/ http//anninndx.pppe

---

tion networks and thus limit their completeness, the OCL is linked to a large knowledge graph to over- come the constraints. Furthermore, multi-modal features such as figures are extracted to foster re- search in document layout and multi-modality.
Field
Description
acl_id
ACL Anthology ID
title
Title of paper
abstract
Abstract from Anthology
bib_key
ACL, bibliographic key
pdf_hash
SHAI hash of PDF file
3 The ACL OCL Corpus
plain_text
Full text in string format S2ORC-ISON object S2 corpus ID
full_text
We start by crawling the PDF files and metadata from the source ACL Anthology. We then pass the PDF files through the full-text extraction process. We enhance this data using Semantic Scholar's API to fetch additional information about OCL papers.
corpus_paaer__i id
arXiv_paper_id citation_count language
arXiv paper ID
Citation from S2
Language written in
predicted_topic Model-predicted CL topic
3.1 Data Acquisition
Table Simplified OCL schema, showing 12 fields. "S2" refers to Semantic Scholar. Associated figure schema not shown.
From the ACL Anthology, we design crawlers to fetch all the PDF documents and metadata from its website. Meta information is included, as the web- site's version is more accurate than that obtained by PDF extraction, given its author-driven nature. We remove PDF files longer than 50 pages, which mostly conference volume introductions' or hand- books. We then obtain 73,285 conference journal and workshop papers in the CL domain.
figures and tables from PDF files using PDFFigures (Clark and Divvala, 2016) following Karishma al. (2023). Each extracted figure is associated with its caption texts, which show the figure ID and textual description in the paper. The figures are stored separately from the textual data in OCL.
The dataset undergoes annual updates, during which we download and process papers recently added to the ACL Anthology. For these updates, we identify new additions by monitoring changes in URLs.
3.3 Knowledge Graph Linking
We link the OCL corpus with knowledge graphs to enrich OCL with external information such as citations. We choose the recently released Seman- tic Scholar Academic Graph (S2AG, Kinney et al., 2023), which includes the most recent data. We use its Graph API to connect OCL document to its corresponding document in S2AG. While the S2AG Graph API offers general information such as metadata, authors, and citations, does not pro- vide full texts of the articles. This is where ACL OCL steps in to supply the complete structured full text, thereby expanding the range of information and providing opportunities for more analysis.
3.2 Full-text Extraction
Different from other platforms with easier-to- extract resources (e.g., ITT the ACL Anthol- ogy only provides PDF files, which we can use to extract full texts for OCL. After an extensive com- parison with open-source toolkits such as PDFBox and pdfminer.six, we use GROBID? for the full- text extraction from PDF files. Our study validates findings from Meuschke et al. (2023) which found GROBID outperforms the other aveelllavaillblle tools in metadata, reference, and general text extrac- tion tasks from academic PDF documents. We take the S2ORC-ISON format used by Lo et al. (2020) and Wang et al. (2020) for our full-text schema, which includes complete information parsed from PDF files, such as metadata, authors, and body text with citations, references, sections and etc.
Citation Network. Different from the previous
AAN corpus, we construct the citation network us- ing the citation information from both inside and outside of OCL. Following AAN, we use in-degree to denote the number of incoming citations internal to OCL. We use citations to mention the number of references to OCL articles beyond OCL. In par- the citationCount field provided by the S2AG Graph Paper Lookup API via ACL An- thology ID for external citation information. In
Due to the limitations of GROBID in formatting information extraction such figures, extract
ticular,
hnttps::/accanttholeg org/2022.ac1. acl-long.@0 pdp hhttps://pdfbox.co aww.hfeeo....cc....omm org/
hhttps://github. http://tttu..oo//o//aaeoa wwtp///www/.//ee///////m/co htt.m/ienne/emmmmiee.iii
hnctoss//bo..emaaticchhllal semanticscholar http:/chhh.a......ppp.iaa.p.ppa. wwwp/org///p//...ooo
ttttps::/gitthub... wttpe//b.commkemmttt/grrrrooii
graph

---

Papers in the ACL-OCL corpus
Distribution of language detected in the ACL-OCL corpus
100k
Full Text Available
6000
Full Text Missing
5000
Dog
4000
3000
2000
236
0965
E993
336
Z66t
566t
9661
1006
1006
2022
oroz
20o2
91oz
stec
Z2oz
Desectted language
69
00
Sa
Figure 3: Distribution of the top languages processed in the OCL corpus. Note that the scale of the y-axis is logarithmic.
Figure 2: Growth in publication rates in the OCL. We
observe that our full-text extraction fail mostly for pa-
pers between the years 2000 2015.
4.1 Statisties
total, we have 669,650 internal directed connec- tions among the 73K papers in ACL OCL.
Annual Publication Growth. The annual quan- tity of published papers is computed and presented in Figure 2. The trend of annual publications shows an exponential growth pattern, especially after the year 2000. This escalating trend indicates the CL community is experiencing a rapid cra of devel- opment, which underscores the need for resources like OCL to manage and leverage this rapidly ex- panding knowledge base. The noticeable spike in publications during even years, relative to odd years, be primarily attributed to the scheduling of certain conferences that exclusively occurred in even years, such as COLING.
3.4 Data Schema
Our dataset adheres the standard Semantic Scholar schema² (c.f., Table 2; complete listing in Appendix A) that resembles scientific documents, ensuring an organized and consistent structure. We note that certain fields are only valid on a portion of the papers. For example, only 16.8% of OCL papers have prxiv_paaper.de indicating corre- sponding version in arXiv.org.
Linguistic Distribution. We also investigate the distribution of written languages of CL publica- tions. Figure illustrates the distribution of the top 27 out 96 languages within the corpus. As ex- pected, English the most dominant language pro- cessed in the ACL OCL. However, it is noteworthy that languages such as Latin and Gujarati are also present. Prior research (Ranathunga and de Silva, 2022) the representation of low-resource lan- guages the ACL Anthology found that numerous languages remain underrepresented. Even among languages in the same language group, there is a significant disparity in coverage.
We add automatically detected fields such as language and topic to enable further analysis. To detect the language in which a document is writ- ten, utilize public language detection toolkit (Shuyo, 2010) with its title and abstract inputs. We discuss research topic detection next in Sec- tions 5 and 6.1.
In addition to the standard CSV format, we pro- vide our dataset in Apache Parquet, an open-source, column-oriented data format, to facilitate efficient downstream processing.
4 Dataset Analysis
4.2 Full-text Quality
A Although we utilize an accurate full-text extraction toolkit GROBID, its inherent limitations do influ- ence the quality of full texts in OCL. We manually
We present statistical analyses of the OCL corpus, including the distribution of papers across years and linguistic distribution. We further highlight the quality full texts and citation graph analysis.
phttps://apii swwanticscholar aataaaar.rrr///pp arg/api-docs/ "https:///t thub. htm//Wimino666/lnggdeee.

---

Title
Rdeg
Reil
Diff
Year
Topic
BERT: Pre-training of Deep Bidirectional Transformers for Language...
2019
ML
Bleu: Method for Automatic Evaluation of Machine Translation
2
3
2002
Summ
GloVe: Global Vectors for Word Representation
3
-1
2014
LexSem
Moses: Open Source Toolkit for Statistical Machine Translation
4
14
10
2007
MT
Deep Contextualized Word Representations
5
8
2018
LexSem
Building Large Annotated Corpus of English: The Penn Treebank Neural Machine Translation of Rare Words with Subword Units A Systematic Comparison of Various Statistical Alignment Models Minimum Error Rate Training in Statistical Machine Translation The Mathematics of Statistical Machine Translation: Parameter Est...
1993
Syntax
7
17
10
2016
MT
8
19
11
2003
MT
9
30
2003
MT
10
N
N
1993
MT
Statistical Phrase-Basee Translation
11
25
14
2003
MT
Learning Phrase Representations using RNN Snecoder-Decoder for
-8
2014
MT
ROUGE: A Package for Automatic Evaluation of Summaries Enriching Word Vectors with Subword Information
13
10
-3
2004
Summ
14
11
-3
2017
LexSem
Convolutional Neural Networks for Sentence Classification
15
-9
2014
ML
Table 3: Top 15 OCL papers ranked by in-degree (Rarg), together with their citation ranking (Roi). the ranking discrepancy (Diff = Roit Rdng), publication year, and model-predicter silver CL topics. N means a number larger than 50.
checked 21 documents to assess the quality of full texts, especially three aspects including metadata, general texts, and contents in specialized formats, Incorrect metadata such as author names exceed- ing three tokens, a common issue. In response, the OCL uses the metadata from ACL Anthology. For general text extraction, some common issues include missing section names, merged paragraphs, the mixing of footnotes with texts, and misidentify- ing texts as table captions. Most of the above errors stem from the challenge of extracting formatting information from PDF files. As with other toolkits, GROBID also struggles with the extraction of ta- bles and figures, especially in-line equations. Quan- titative assessments are provided Appendix D.
and RNN Encoder-Decoder (Cho et al., 2014) are interesting contrasts. Out of these 15 top-ranked papers, there are 7 papers (bolded) published after 2010, indicating the most recent research interests in CL, namely neural models. By analyzing the research topics of these top-cited papers, machine translation (7 of 15) is still the dominant research topic in CL. Note that the system pereem--eeccted topics are silver data, which we detail later.
5 Objective Topic Classification
Topic information serves as a crucial attribute in retrieving scientific papers. We focus on objective topies (Prabhakaran et al., 2016), which are used to denote specific research tasks such as machine translation or text generation. Notably, even though authors submit topics during manuscript submis- sion, this information remains invisible on the ACL Anthology website. We aim to assign the most appropriate objective topic to each CL paper and further explore how this topic information efit the development of the CL community. The single-label topic classification setting adopted this paper for simplicity; multi-label classification is left to future work.
4.3 Citation Graph
Table 3 displays the most frequently cited OCL pa- pers ranked by their in-degrees (Rdeg). which refer the internal citations within OCL. Citations from papers beyond OCL are denoted as citations (Ret). We show the discrepancy between in-degrees and citations with their difference. By analyzing in- degrees and citations, we can gain insights into the research interests of communities besides CL and compare them with the priorities of the CL commu- nity itself. From Table 3, it observed that seminal works in the CL domain such as Moses (Koehn et al., 2007) and minimum error rate (Och, 2003) are not influential outside of CL. On the other hand, the CNN for sentence classification (Kim, 2014)
ben-
Given scientific document d, with its textual information such title, abstract, and full text, objective topic classification aims to assign a task topic label I E T. to I. the topic label set taken from the submission topics (c.g., "Genera- tion". "Question Answering") of the ACL confer-

---

ences. The complete list of 21 topics is presented in Appendix B.
Source
#Doc
# Unique Topic
ACL 2020
705
21
Based on the amount of supervised information
EMNLP 2020
681
19
used for training, we explore three classes of meth-
ACL-IICNLP 2021
470
21
ods for topic classification: unsupervised, semi- supervised, and supervised methods.
EACL 2021
295
NAACL 2021
394
21
Total
2545
21
5.1 NLI-based Un- and Semi-supervised Methods
Table 4: Statistics of the topic corpus, STop,
Given the absence of large-scale topic-labeled data, our initial investigation focuses on zero-shot doc- ument classification methods. Yin et al. (2019) fine-tuned BART (Lewis et al., 2020) on natural language inference datasets, thus achieving zero- shot prediction of many tasks including document vari-
the most suitable topic. We explore different key- word extraction methods including TF-IDF and Yake! (Campos et al., 2020), both of which are simple and efficient.
classification. We follow their work and ant model BarT--arge-MNLL" to model the topic classification task as an inference task. To identify the topic label of a document d, the BaRT-large- MNLI model predicts the probability d) that the hypothesis label / is entailed from the premise document d. We denote this unsupervised method BART-NLI-Oshot.
5.3 PLM-based Supervised Method
Given the proven success of pre-trained language models (PLMs) in multiple NLP tasks, particularly with training data in a small scale, we utilize a based classification framework for our task. The framework employs pre-trained language model to encode the input document and a softmax classification layer atop it for topic label predic- tion. In addition, we consider pre-trained language models trained from scientific documents, namely SciBERT (Maheshwari et al., 2021) and SPECTER (Cohan et al., 2020), to take advantage of their strong encoding power of domain-specific docu- ments.
Inspired by the label-partially-seen experimental settings Yin et al. (2019), we establish a semi- supervised setup leveraging the limited labeled data ($6) for improved performance. Specifically, we fine-tuned the BART-large-MNLJ model with the labeled data, which is tailored to fit the NLI task. We refer to this semi-supervised method as BART- NLI-FT.
Experiments
5.2 Keyword-based Supervised Method
Topic Data Curation. We crawl published pa- pers of several online held CL conferences (e.g.. ACL 2020, EACL 2021) between 2020 and 2022. together with their topics from those websites. Af- ter aligning those papers with the data the ACL OCL, we obtained 2545 documents classified in 21 topics in total, present in Table 4. These documents together with their topics are used as our training and testing data. We use 5-fold cross-validation across all experiments, randomly selecting 2,036 (80%) papers balanced in each topic our training set, and the remaining 509 (20%) as test. We use Macro F1, Weighted F1, and Accuracy (aka. Micro FI) as evaluation metrics for the multi-class topic classification. We rely on Accuracy to compare systems' performances.
After obtaining over 2000 documents with ground- truth topic labels, we train supervised model for topic classification. As salient information of docu- ments, keywords are shown to be helpful for many tasks such as classification (Zhang et al., 2021), clustering (Chiu et al., 2020), summarization (Lit- vak and Last, 2008; Liu al., 2021), etc. Inspired by these findings, we design keyword-basse su- pervised methods for topic classification. Initially, keywords are extracted from each document of the training set. Subsequently, we select the top 10 topie-represenntatiee keywords!2 for each topic by TF-IDF. During inference, given test docu- ment d, the topic containing the most matching toppe-representative keywords in d is considered
/nttps://hugginnffce..oooo wttpe//aaccccoaccoooookoooo
6.1 Topic Classification
bart-large-mn..
The number of keywords is selected from [10, 20, 30, 40) via hyper-parameter test.
Table shows that performance improves over the full range- - from zero-shot, semi-supervised, su-

---

Syntex
Resource
Discourse
Speech
Topic
KI
0.2
0.16
0.14
0.14
0.12
0.12
0.1
0.08
0.06
0.05
0.34
0.04
0.02
0.02
2000
2005
2010
2015
2020
2000
2005
2010
2015
2020
(a) Waning topics
(b) Machine translation, increasing and declining later
Topic
NLG
Topic
Ethics
VaRRobo
It CompSocial
SeriSirm
Sentimerr
0.00
0.08
0.05
0.04
0.00
2000
2005
2010
2015
2070
2000
2005
2010
2015
2020
(c) Resurging underrepresented topics
(d) Increasing underrepresentted topies
Figure Plot research trend of topies, grouped by patterns. The y-axis represents a topic's publication percentage in each year (x-axis). To identify the patterns, remove the scattered data points and only show the topies' smoothed trending lines. Best viewed color.
Method
Mac. F1
Wei. Fl
Acc.
how different input text selection methods in- fluence the task, namely Abstract and Introduc- tion+Conclusion (Tabel 6). We adopt the I+C set- ting which has better performance.
BART-NLI-Oshot
38%
43%
45%
BART-NLI-FT
44%
50%
51%
Keyword-TFIDF
51%
54%
53%
Keyword- Yake!
38%
42%
39%
Case Study. From the last column in Table 3 66%
PLM-SciBERT 58%
6.%%
(system-predieee predicted topics), observe 13 correct la- bels out of 15 documents. Two works in Resource & Evaluation, namely BLEU (Papineni et al., 2002) and ROUGE (Lin, 2004), are incorrectly predicted due to insufficient training samples and high over- lap with other topics. The 87% accuracy is higher than an expected 69% accuracy tested on the STop test set in Table 5, mainly because of the bias in the distribution of top-cited papers towards domi- nant topics (e.g., MT). Interestingly, three papers from lexical semantics including GloVe (Penning- ton et al., 2014) are correctly identified, perhaps due to the strong indication from words in their titles (i.c., "word representation" and "word vec- tors").
PLM-SPECTER 66%
68%
69%
Table 5: Performances of topic classification models.
Method Abstract
I+C
Diff.
SciBERT 66%
67%
1%t
SPECTER 64%
68%
4%
Table 6: Performances (Accuracy) of different input texts, Abstract VS. I+C Introduction=Conclusioo))
pervised, to PLM-based methods. The trends high- light the importance of supervised data, even at small scales. The best performance (achieved by the PLM-based supervised method) is comparable those reported in the state-of-the-art topic clas- sification tasks on the scientific domain (Lo al., 2020; Li et al., 2022). But the challenges of pro- cessing scientific data still remain, as performance is still much lower than those (F1 > 80%) in the news domain (Wang et al., 2022).
6.2 Topic Trend Analysis
We analyze the trend of model-predicted research topics in OCL starting from 2000 to 2021. Figure 4 presents the popularity (estimated by publication percentage) of all topics across years, subgrouped into recognizable trend patterns. We first introduce waning topics in Figure 4a, including both pre-
Inspired by (Meng et al., 2021), we explore

---

dominant ones like "Syntax" and "Resource" and underrepresented ones such as "Discourse" and "Speech". Another predominant topic in CL "MT". shown in Figure 4b, which peaks (around 19%) in the 00 and declines in the latter years.
abstracts and related work texts can be used direct references for summarization and re- lated work generation (Hoang and Kan, 2010; Hu and Wan, 2014) tasks, respectively.
Structures in full texts such as sections and paragraphs provide opportunities for informa- tion/knowledge extraction tasks from specific sections, such as contribution extraction from Introductions and Conclusions, future work prediction from Conclusions, and analysis of generalization ability of NLP models from Experiments.
From all the remaining underrepresented topics, we further classify them into three types, only two types are shown in Figure 4. Figure 4c shows resur- gent topics including "Dialogue", "Interpret", "ML"" and "NLG", which have declining/low interests be- fore 2015 but increased afterward. Figure 4d shows topics including "CompSocial", "IE", "Sentiment" and "SenSem", which are underrepresented histor- ically and become noticeable later. Among them, "Ethics" is a very new and small topic starting in 2016. In contrast, "QA", "LexSem", "Summ" and "IR" (not shown) relatively stable research top- ies with mild corrections in recent years. Among all topics, the popularity of "Syntax" drops the most (2%%++%) while "CompSocial" increases the most (1 (1%)...
Links to external platforms, including arXiv. are beneficial as it allows to access various versions of a paper in the ACL Anthology, As a result, this opens up opportunities for further analysis, such as examining the modifications made to the pre-print version of a paper before final publication, among other possibilities.
As an up-to-date corpus spanning decades, OCL helps to analyze historical language change 14 in the CL domain, such vocab- ulary change (Tahmasebi al., 2021). To illustrate, consider the term 'prompt'. Tra- ditionally in computer science, it referred to a signal on a computer screen (e.g., 'C:/>') that indicates the system is ready for user in- put. However, after 2020, its interpretation broadened to encompass natural language instruction query presented to Al system (Jiang al., 2020b). This newer definition is documented, for instance, the Cambridge Dictionary
7 Downstream Applications
We previously highlighted one example application of ACL OCL, topic trend analysis. By nature, ACL OCL is a scholarly corpus aiming to enable and benefit a wide range of research. We now further depict a few existing research directions that could benefit from ACL. OCL, predicting opportunities enabled by its characteristics.
Additional forms of topic analyses and topic- aided tasks are now feasible with the domain- specific, fine-grained topic information pro- vided by OCL, such as emerging topic detec- tion (Asooja et al., 2016), evolution detection of topics (Uban al., 2021), !), paper-reviewer matching (Thorn Jakobsen and Rogers, 2022) via topics and etc.
With the combinations of citation network, metadata, and full text, OCL can facilitate the construction of knowledge graph, in- cluding papers, authors, topics, datasets, mod- cls, claims, and their relationships. Such a knowledge graph can enable more intelli- gent academic search engine beyond keyword matching, facilitating information retrieval on multiple aspects.
OCL aids tasks that demand complete tex- tual content from scientific papers, such as document-level terminology detection, coref- erence resolution, pre-training scientific large language models like Galactica (Taylor et al., 2022) or fine-tuning a general LLM like Llama (Touvron et al., 2023). Building sci- entific assistants for the CL domain, capable of responding to domain or task-specific ques- tions (Lu et al., 2022) is also possible.
The full texts and figures provision multi- modal research tasks such as summarization,
"nttps://ww..chrggg http://wwww.cc.neeikk.....c www.aaaeieeeorreeeeeeeeeeeeeee org/event/
mm2=-emmp-lchenge//
High-quality texts are more suited for task- specific supervised data. For example, the
english/prompt

---

which integrates text summaries with figures and citation visualizations for more holistic understanding.
guage models (LLMs), we suggest that LLMs can play an instrumental role in generating high-quality, large-scale silver labels (Yu et al., 2023). Moreover, human-AT collaborative annotations (Liu et al., 2022) provide an effective strategy for complex tasks like natural language inference.
8 Conclusion
We introduce ACL OCL, a scholarly corpus aiming to advance open research in the computational lin- guistics domain. The structured full texts, enriched metadata from Semantic Scholar Academic Graph, and figures provided by OCL can benefit existing research tasks well enable more opportunities. We highlight the utility of OCL in temporal topic trends analysis in the CL domain. The topics are generated by a trained neural model with small yet effective scientific topic dataset. By analyz- ing the topics' popularity, we ask for more atten- tion on the emerging new topics such as "Ethics of NLP" and the declining underrepresented topics like "Discourse and Pragmatics" and "Phonology, Morphology and Word Segmentation". In the future, we will work on the data currency OCL, which aims to keep OCL up-to-date with the ACL Anthology data. We plan to update OCL by year to keep it alive. The ultimate solution is to provide full-text extraction and information extraction APIS to ACL Anthology, thus hosting the OCL data ACL Anthology itself.
A Acknowledgements
This research is supported by the Singapore Min- istry of Education Academic Research Fund Tier 1 (251RES2216, 25 251RES2037). We thank Sergey Feldman from Allen Al for providing Semantic Scholar data access.
References
Kartik Asooja, Georgeta Bordea, Gabriela Vuleu, and Paul Buitelaar. 2016. Forecasting emerging trends from scientific literature. In Proceedings of the Tenth International Conference on Language Resources and Evaluation (IREE')16), pages 417-420, Portorož, Slovenia. European Language Resources Association (ELRA).
Steven Bird, Robert Dale, Bonnie Dorr, Bryan Gibson, Mark Joseph, Min Kan, Dongwon Lec, Brett Powley, Dragomir Radev, and Yee Fan Tan. 2008 The ACI. Anthology reference corpus: reference dataset for bibliographic research in computational linguistics. In Proceedings of the Sixth International Conference on Language Resources and Evaluation (LREC'08), Marrakech, Morocco. European Lan- guage Resources Association (ELRA).
9 Limitations
The OCL corpus is small-scale collection of doc- uments specifically focusing on peer-reviewed and open-access CL papers. As a result, is not a com- prehensive corpus like S2ORC (Lo et al., 2020), since it does not include any other sources beyond the ACL Anthology. In the future, the OCL could be expanded by incorporating CL papers on Xrviv (e.g., CL), which is related to unarXive (Saier et al., 2023). The challenge is how to filter out arXiv papers of low quality.
Isabel Cachola, Kyle Lo, Arman Cohan, and Daniel Weld. 2020. TLDR: Extreme summarization of sci- entific documents. In Findings of the Association for Computational Linguistics: EMNLP 2020, pages 4766-4777, Online. Association for Computational Linguisties.
Ricardo Campos, Vítor Mangaravite, Arian Pasquali, Alípio Jorge, Célia Nunes, and Adam Jatowt. 2020. YAKE! Keyword extraction from single documents using multiple local features. Information Sciences, 509:257-289.
To ensure the extraction of high-quality full texts from the provided PDF files, the OCL corpus uti- lizes the most advanced open-sourced PDF2text toolkit, GROBID. Due to constraints on budget, only open-source toolkits are considered, although it is acknowledged that some paid PDF2text ser- vices might yield higher-quality full texts. In addition, previous work such as unarXive use ETEX files as source documents avoid PDF2text, The OCL corpus is an unlabeled resource, lack- ing annotations for specific tasks that require labels. Given the demonstrated capabilities of large lan-
Billy Chiu, Sunil Kumar Sahu, Derek Thomas, Neha Sengupta, and Mohammady Mahdy. 2020. Autoen- coding keyword correlation graph for document clus- tering. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 3974-3981, Online. Association for Computational Linguistics.
Kyunghyun Cho, Bart van Merriënboer, Caglar Gul- cehre, Dzmitry Bahdanau, Fethi Bougares, Holger Schwenk, and Yoshua Bengio. 2014. Learning phrase representations using RNN encoder for statistical machine translation. In Proceedings

---

of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 1724- 1734, Doha, Qatar. Association for Computational Linguistics.
Linguistics: EMNLP 2021, pages 3258-3264, Punta Cana, Dominican Republic. Association for Compu- tational Linguistics.
Yue Hu and Xiaojun Wan. 2014. Automatic gener- ation of related work sections in scientific papers: An optimization approach. In Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 1624-1633, Doha, Qatar. Association for Computational Linguis- tics.
Christopher Clark and Santosh Divvala. 2016. Paffig- ures 2.0: Mining figures from research papers.
Arman Cohan, Waleed Ammar, Madeleine van Zuylen, and Field Cady. 2019. Structural scaffolds for ci- tation intent classification in scientific publications. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computa- tional Linguistics: Human Language Technologies, Volume (Long and Short Papers), pages 3586-3596, Minneapolis, Minnesota. Association for Computa- tional Linguistics.
Wenyi Huang, Zhaohui Wu, Chen Liang, Prasenjit Mi- tra, and C. Lee Giles. 2015. A neural probabilistic model for context based citation recommendation. In AAAI Conference on Artificial Intellignce.
Sarthak Jain, Madeleine van Zuylen. Hannaneh Ha- jishirzi, and Iz Beltagy. 2020. SeiREX: A challenge dataset for document-leve information extraction. In Proceedings of the 58th Annual Meeting of the Asso- ciation for Computational Linguistics, pages 7506- 7516, Online. Association for Computational Lin- guistics.
Arman Cohan, Sergey Feldman, Iz Beltagy, Doug Downey, and Daniel Weld. 2020. SPECTER: Document-level representation learning using citation-informed transformers. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 2270-2282, Online. Association for Computational Linguistics.
Feng Jiang, Xiaomin Chu, Peifeng Li, Fang Kong, and Qiaoming Zhu. 2020a. Chinese paragraph-Ievel dis- course parsing with global backward and local re- verse reading. In Proceedings of the 28th Inter- national Conference on Computational Linguistics, pages 7749-5759, Barcelona, Spain (Online). Inter- national Committee on Computational Linguistics.
Aliva Das, Xinya Du, Barry Wang. Kejian Shi, Jiayuan Gu, Thomas Porter, and Claire Cardie. 2022. Auto- matic error analysis for document-level information extraction. In Proceedings of the 60th Anmual Meet- ing of the Association for Computational Linguistics (Volume 1: Long Papers), pages -960-,975, Dublin, Ireland. Association for Computational I Linguistics.
Zhengbao Jiang. Frank F. Xu, Jun Araki, and Graham Neubig. 2020b. How can we know what language models know? Transactions of the Association for Computational Linguistics, 4383-448.
Sujatha Das Gollapalli and Xiaoli Li. 2015. EMNLP versus ACI: Analyzing NI.P research over time. In Proceedings of the 2015 Conference Empirical Methods Natural Language Processing, pages 2002-2006, Lisbon, Portugal. Association for Com- putational Linguistics.
Zeba Karishma, Shaurya Rohatgi. Kavya Shrinivas Pu- ranik, Jian Wu, and C. Lee Giles. 2023. Acl-fig: A dataset for scientific figure classification.
David Hall, Daniel Jurafsky, and Christopher D. Man- ning. 2008. Studying the history of ideas using topic models. In Proceedings of the 2008 Conference on Empirical Methods in Natural Language Processing, pages 363-371, Honolulu, Hawaii. Association for Computational Linguistics.
Yoon Kim. 2014. Convolutional neural networks for sentence classification. In Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 1746-1751, Doha, Qatar. Association for Computational Linguis- tics.
Cong Duy Vu Hoang and Min-Yon Yen Kan. 2010. Towards automated related work summarization. In Coling 2010: Poxters, pages 427-435, Beijing, China. Col- ing 2010 Organizing Committee.
Rodney Kinney, Chloe Anastasiades, Russell Authur, Iz Beltagy. Jonathan Bragg, Alexandra Buraczyn- ski, Isabel Cachola, Stefan Candra, Yoganand Chan- drasekhar, Arman Cohan, et al. 2023. The se- mantic scholar open data platform. arXiv preprint arXiv:2301.1011 10140..
Yufang Hou, Charles Jochim, Martin Gleize, Francesca Bonin, and Debasis Ganguly. 2021. TDMSci: A spe- cialized corpus for scientific literature entity tagging of tasks datasets and metrics. Proceedings of the 16th Conference of the European Chapter of the Asso- ciation for Computational Linguistics: Main Volume, pages 707-714, Online. Association for Computa- tional Linguistics.
Philipp Koehn, Hieu Hoang, Alexandra Birch, Chris Callison-Burch Marcello Federico, Nicola Bertoldi Brooke Cowan, Wade Shen, Christine Moran, Richard Zens, Chris Dyer, Ondfej Bojar, Alexandra Constantin, and Evan Herbst. 2007. Moses: Open source toolkit for statistical machine tramslation. In Proceedings of the 45th Annual Meeting of the As- sociation for Computational Linguistics Companion Volume Proceedings of the Demo and Poster Sessions,
Ting-Yao Hsu, C Lee Giles, and Tang-Hao Huang, 2021. SciCap: Generating captions for scientific figures. In Findings of the Association for Computational

---

pages 177-180, Prague, Czech Republic. Association for Computational Linguistics.
Processing, pages 130-133, Online. Association for Computational Linguistics.
Mike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, Abdelrahman Mohamed, Omer Levy, Veselin Stoyanov, and Luke Zettlemoyer. 2020. pre-training
Rui Meng, Khushboo Thaker, Lei Zhang. Yue Dong,
Xingdi Yuan, Tong Wang, and Daqing He. 2021. Bringing structure into summaries: a faceted sum- marization dataset for long scientific documents. In Proceedings of the 59th Annual Meeting of the Asso- ciation for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 2: Short Papers), pages 1080- 1089, Online. Association for Computational Linguis- tics.
BART: Denoising for natural language generation, translation, and com- prehension. In Proceedings of the 58th Annual Meet- ing of the Association for Computational Linguistics, pages 7871 1-710, Online, Association for Computa- tional Linguistics.
Yudong Li, Yuqing Zhang, Zhe Zhao, Linlin Shen, Wei- jie Liu, Weiquan Mao, and Hui Zhang. 2022. CSL: A large-scale Chinese scientific literature dataset. In Proceedings of the 29th International Conference on Computational Linguistics, pages 3917-3923, Gyeongju, Republic of Korea. International Com- mittee on Computational Linguistics.
Norman Meuschke, Apurva Jagdale. Timo Spinde, Jc- lena Mitrović, and Bela Gipp. 2023. A benchmark of pdf information extraction tools using a multi- task and multi-domain evaluation framework for aca- demic documents. In Information for a Better World: Normality, Virtuality, Physicality, Inclusiviy: 18th International Conference, iConference 2023, Virtual Event, March 13-17, 2023, Proceedings, Part II, pages 383 405.405.65 Springer.
Yew Lin. 2004. ROUGE: A package for auto- matic evaluation of summaries. In Text Summariza- tion Branches Out, pages 74-81, Barcelona, Spain. Association for Computational Linguistics.
Franz Josef Och. 2003. Minimum error rate training in statistical machine translation. In Proceedings of the 41st Anmual Meeting of the Association for Computational Linguistics, pages 160-167, Sapporo, Japan. Association for Computational Linguistics.
Marina Litvak and Mark Last. 2008. Graph-based key- word extraction for single-documens summarization. In Coling 2008: Proceedings of the workshop Multi- source Multilingual Information Extraction and Sum- marization, pages 17-24, Manchester, UK. Coling 2008 Organizing Committee.
Kishore Papineni. Salim Roukos, Todd Ward, and Wei- Jing 7hu. 2002. Bleu: a method for automatic evalu- ation of machine translation. In Proceedings of the 40th Annual Meeting of the Association for Compu- tational Linguistics, pages 111-118, Philadelphia, Pennsylvania, USA, Association for Computational Linguistics.
Alisa Liu, Swabha Swayamdipta, Noah A. Smith, and Yejin Choi. 2022. WANLIE Worker and AT collabora- tion for natural language inference dataset creation. In Findings of the Association for Computational Linguistics: EMNLP 2022. pages 6826-6847, Abu Dhabi, United Arab Emirates. Association for Com- putational Linguistics.
Jeffrey Pennington, Richard Socher, and Christopher Manning. 2014. GloVe: Global vectors for word representation. In Proceedings of the 2014 Confer- ence on Empirical Methods in Natural Language Pro- cessing (EMNLP), pages 1532-1543, Doha, Qatar. Association for Computational Linguistics.
Shuaiqi Liu, Jiannong Cao, Ruosong Yang, and Zhiyuan Wen. 2021. gighiigghttransfrmmer I Leveraging key phrase aware attention to improve abstractive multi- document summarization. In Findings of the Associ- ation for Computational Linguistics: ACL-IJCNLP 2021. pages 5021-5027, Online. Association for Computational Linguistics.
Vinodkumar Prabhakaran, William L. Hamilton, Dan McFarland, and Dan Jurafsky. 2016. Predicting the rise and fall of scientific topics from trends in their rhetorical framing. In Proceedings of the 54th An- nual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1170- 1180, Berlin, Germany. Association for Computa- tional Linguistics.
Kyle Lo, Lucy Lu Wang, Mark Neumann, Rodney Kin- ney, and Daniel Weld. 2020. S2ORC: The semantic scholar open research corpus. In Proceedings of the 58th Annual Meeting of the Association for Compu- tational Linguistics, pages 4969-4983, Online. Asso- ciation for Computational Linguistics.
Pan Lu. Swaroop Mishra, Tony Xia, Liang Qiu, Kai- Wei Chang, Song-Chun Zhu, Oyvind Tafjord, Peter Clark, and Ashwin Kalyan. 2022. Learn to explain: Multimodal reasoning via thought chains for science question answering. In The 36th Conference on Neu- ral Information Processing Systems (NeurIPS).
Dragomir R. Radev, Pradeep Muthukrishnan, and Va- hed Qazvinian. 2009. The ACL Anthology net- work. In Proceedings of the 2009 Workshop on Text and Citation Analysis for Scholarly Digital Libraries (NLPIR4DL.), pages 54-61, Suntec City, Singapore. Association for Computational Linguistics.
Himanshu Maheshwari, Bhavyajeet Singh, and Va- sudeva Varma. 2021. SciBERT sentence representa- tion for citation context classification. Proceed- ings of the Second Workshop on Scholarly Document
Surangika Ranathunga and Nisansa de Silva. 2022. Some languages are more equal than others: Prob- ing deeper into the linguistic disparity in the NLP world. In Proceedings of the 2nd Conference of the

---

Pacifac Chapter of the Association for Compu- tational Linguistics and the 12th International Joint Conference on Natural Language Processing (Vol- ume I: Long Papers), pages 823-848, Online only. Association for Computational Linguistics.
of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). pages 7109-7119, Dublin, Ireland. Association for Computational Linguistics.
Wenpeng Yin, Jamaal Hay, and Dan Roth. 2019. Bench- marking zero-shot text classification: Datasets, eval- uation and entailment approach. In Proceedings of the 2019 Conference on Empirical Methods in Natu- ral Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNILP)) pages 3914-3923, Hong Kong, China. Association for Computational Linguistics.
Tarek Saier, Johan Krause, and Michael Färber. 2023. unarxive 2022: All arxiv publications pre- pre-processed for nip. including structured full-text and citation network.
Nakatani Shuyo. 2010. Language detection library for java.
Nina Tahmasebi, Lars Borin, and Adam Jatowt. 2021. Survey of computational approaches to lexical se- mantic change detection.
Danni Yu, Luyang Li, Hang Su, and Matteo Fuoli. 2023. Assessing the potential of ai-assisted pragmatic an- notation: The case of apologies.
Ross Taylor, Marcin Kardas, Guillem Cucurull, Thomas Scialom, Anthony Hartshorn, Elvis Saravia, Andrew Poulton, Viktor Kerkez, and Robert Stojnic. 2022. Galactica: A large language model for science.
Lu Zhang, Jiandong Ding, Yi Xu, Yingyao Liu, and Shuigeng Zhou. 2021. vieaklysuupervise text classi- fication based on keyword graph. In Proceedings of the 2021 Conference on Empirical Methods in Natu- ral Language Processing, pages 2803-2813, 13, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics.
Terne Thom Jakobsen and Anna Rogers, 2022, What factors should paper-reviewer assignments rely on? community perspectives on issues and ideals con- ference peer-review. In Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics; Human Seattle,
A Full Data Schema
Language Technologies, pages United States. Association for Computational Lin- guistics.
The full data schema including 27 fields, is shown in Table 7.
Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, Aurelien Rodriguez, Armand Joulin, Edouard Grave, and Guillaume Lample. 2023, Llama: Open and efficient foundation language models.
Field
Description
acl_id
Unique Anthology II)
title
Title of paper
author
List of authors
abstract
Abstract from ACL Anthology
url
Publication link
Ana Sabina Uban, Cornelia Caragea, and Liviu P. Dinu. 2021. Studying the evolution of scientific topies and their relationships. In Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021, pages 1908-1922, Online. Association for Computa- tional Linguisties.
year
Publication year
month
Publication month
booktitle
Publication venue
pages
Page range
address
Address of
DOI number
journal
Journal name
volume
Volume number
Lucy Lu Wang, Kyle Lo, Yoganand Chandrasekhar, Russell Reas, Jiangjiang Yang. Doug Burdick, Darrin Eide, Kathryn Funk, Yannis Katsis, Rodney Michael Kinney, Yunyao Li, Ziyang Liu, William Merill, Paul Mooney, Dewey A. Murdick, Devvret Rishi, Jerry Sheehan, Zhihong Shen, Brandon Stilson, Alex D. Wade, Kuansan Wang, Nancy Xin Ru Wang, Christopher Wilhelm, Boya Xie, Douglas M. Ray- mond, Daniel S. Weld, Oren Etzioni, and Sebastian Kohlmeier. 2020. CORD-19: The COVID-19 open research dataset. In Proceedings of the 1st Work- shop NLP for COVID-19 at ACL 2020, Online. Association for Computational Linguistics.
number
Issue number
editor
Editor name
isbn
ISBN number
ENTITYTYPE
Publication type
bib_key
ACL bihliographic key
note
Notes from authors
hash
SHAI hash of the PDF file
plxin_text
Full text in string format
full_ text corpus_ _id erXiv_paper__t ci citation_count language predicted_tapic
S2ORC-ISON object with sections S2 corpus ID
arXiv paper ID)
Number of citations from $2 Language written in Model-predicted CL topic
Zihan Wang, Peiyi Wang, Lianzhe Huang, Xin Sun, and Houfeng Wang. 2022. Incorporating hierarchy into text encoder: a contrastive learning approach for hierarchical text classification. In Proceedings
Table 7: Full data schema. S2 refers to Semantic Scholar.

---

B Topics in CL domain
E
Reading the corpus
We construct a taxonomy of objective topics in the CL domain, shown Table 8, by taking and re-organizing the submission topics in major CL conferences (i.e., ACL, EMNLP, COLING). We surveyed their call for papers from 2000 to 2022. and include all topics in the taxonomy. We remove four broad coverage topies, including "Theme", "Student Research Workshop", "System Demon- strations", and "NLP Applications".
The corpus file can read and analyzed with the following Python commands:
df = pandas. read_parquete ocl.parquet" df shape # for size df. columns # for data schema
Scientific Topic Dataset
We create the Scientific Topic Dataset, STop, by crawling scientific papers along with their topic labels from the following conferences:
ACL 2020: org org/papers, html
https://virtull.acl.22 ac12000
EMNLP 2020: html
https://virtual. 2020.
ACL-UCCLL 2021: https://2021. aclweb. org/progrramoverriien
EACL 2021: https:/www.vvirtual222. eacl. facl.org/index.r html
NAACL 2021: https: //2021. naacl. orgp/confereeee-prorramm// prggr program. html.
D Text quality assessment
Missing sections. 19 sections are not detected from 21 scientific papers. For example, "4.1 Analysis of Stemming", "Limitations. Conclu- sions, and Future Work".
Footnote, All footnote numbers are concate- nated to the body text, such "Google Could Natural Language APII". There are 25 foot- notes missing out of 92.
Figures. There are 28.78% (19/66) figures detected.
Tables. There are 76 tables in total, 31.58% (24) of them are detected and 46.06% (35) are detected but with either wrong contents or titles.

---

ID
Abbr
Topic name
1
CompSocial
Computational Social Science and Social Media
2
Dialogue
Dialogue and Interactive Systems
3
Discourse
Discourse and Pragmatics
4
Ethies
Ethics and NLP
5
NLG
Generation
6
IE
Information Extraction
7
IR
Information Retrieval and Text Mining
8
Interpret
Interpretability and Analysis of Models for NL.P
9
VisRobo
Language Grounding to Vision, Robotics and Beyond
10
LingThhorg
Linguistic Theories, Cognitive Modeling and Psycholinguistics
11
ML
Machine Learning for NLP
12
MT
Machine Translation and Multilinguality
13
WS
Phonology, Morphology and Word Segmentation
14
QA
Question Answering
15
Resource
Resources and Evaluation
16
LexSem
Semanties: Lexical Semanties
17
SenSem
Semanties: Sentence-level Semantics, Textual Inference
18
Sentiment
Sentiment Analysis, Stylistic Analysis, and Argument Mining
19
Speech
Speech and Multimodality
20
Summ
Summarization
21
Syntax
Syntax: Tagging, Chunking and Parsing
Table 8: Objective topics of papers in CL domain, together with defined abbreviations.


Reviews:
[{"review_id": "b8Sl2iFy3X", "reviewer": "Reviewer_q3Sr", "summary": "", "questions": "1. Did you try making use of Large Language Models to generate the keywords instead of using TF-IDF and Yake?", "limitations": "", "rating": 4, "confidence": 4, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We would like to express our gratitude to Reviewer q3Sr for recognizing the potential of our work as a valuable resource in advancing research in the scientific domain. We also appreciate the identification of the typos, which will be addressed in the final manuscript. \n\n> Reject Reason A. *My only concern is that this corpus will be a source of unlabeled data. While the authors do provide topic classification labels for a small subset, it is not extendable without a significant amount of human effort. As a result, it is unclear how easy/feasible/suitable it will be for downstream applications.*\n\nThanks for raising this pertinent concern. We would like to discuss certain promising methodologies for deriving labeled data beneficial for downstream applications, highlighting the corpus's potential versatility. In our topic classification demonstration, we propose an approach that leverages a model, fine-tuned on a limited amount of 'golden data', to predict 'silver topic labels' for the rest of unlabeled data in the OCL corpus. This 'golden data' can be curated through human annotations, ensuring precision. Given the demonstrated capabilities of large language models (LLM), we believe that LLMs can play an instrumental role in generating high-quality, large-scale silver labels [1]. Moreover, human-AI collaborative annotations [2] provide an effective strategy for complex tasks like natural language inference. We'll incorporate this discussion into the finalized version of the paper.\n\n\n\n> QA. *Did you try making use of Large Language Models to generate the keywords instead of using TF-IDF and Yake?*\n\nNo, we did not. We initially focused on TF-IDF and Yake! due to their unsupervised approach, ease of use, and efficiency. Nevertheless, your suggestion holds merit. For example, employing prompt engineering to tailor Large Language Models (LLM), such as GPT-4 [3] or Llama2 [4], might offer more nuanced and high-quality keywords compared to the currently adopted methods. While we won't transit to LLM-based strategies in this version of the paper, we acknowledge the potential and will incorporate LLM into our topic classification system in subsequent iterations.\n\n\nReferences:\n\n[1] Yu, Danni, Luyang Li, Hang Su, and Matteo Fuoli. 2023. “Using LLM-Assisted Annotation for Corpus Linguistics: A Case Study of Local Grammar Analysis.” arXiv. http://arxiv.org/abs/2305.08339.\n\n[2] Liu, Alisa, Swabha Swayamdipta, Noah A. Smith, and Yejin Choi. 2022. “WANLI: Worker and AI Collaboration for Natural Language Inference Dataset Creation.” In Findings of the Association for Computational Linguistics: EMNLP 2022, 6826–47. Abu Dhabi, United Arab Emirates: Association for Computational Linguistics. https://doi.org/10.18653/v1/2022.findings-emnlp.508.\n\n[3] OpenAI. 2023. “GPT-4 Technical Report.” arXiv. http://arxiv.org/abs/2303.08774.\n\n[4] Touvron, Hugo, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, et al. 2023. “Llama 2: Open Foundation and Fine-Tuned Chat Models.” arXiv. http://arxiv.org/abs/2307.09288.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. The proposed corpus will be a valuable resource for the progress of research topics dealing with scientific text.\n2. The structured data (e.g., title, abstract, section-wise full-text, figures, etc.) will be valuable for a multitude of research areas.\n3. The corpus contains a large number of papers.\n4. The data extraction/cleaning/curation methods seem very sound and completed with care.\n5. The authors plan to maintain and regularly update the corpus.", "reasons_to_reject": "My only concern is that this corpus will be a source of unlabeled data. While the authors do provide topic classification labels for a small subset, it is not extendable without a significant amount of human effort. As a result, it is unclear how easy/feasible/suitable it will be for downstream applications. However, providing labels for downstream tasks is probably out of scope for this paper, as the authors mention that this is an \"open research-oriented\" dataset.", "questions_to_authors": "1. Did you try making use of Large Language Models to generate the keywords instead of using TF-IDF and Yake?", "paper_topic_and_main_contributions": "The authors of this paper primarily address the limitations of available corpora containing scholarly articles in the computational linguistic domains such as ARC (Bird et al., 2008) and AAN (Radev et al., 2009). In doing so, they introduce ACL OCL, a corpus containing peer reviewed scholarly articles published in various computational linguistic conferences and workshops. The corpus contains 73K papers which is a significant step up from available resources (ARC and AAN).  In addition, the topical labels for a subset of the papers are also made available, which can aid in topic classification research. As an example use case of the corpus, the authors provide an analysis of the trends of research topics over the years (e.g., waning and resurgence). Furthermore, a number of future use cases of the corpus is also outlined.", "excitement": 4, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Typos_Grammar_Style_and_Presentation_Improvements": "line499: upopportunities"}, "extra_text": {}}, {"review_id": "BlD5jSIr2d", "reviewer": "Reviewer_s1m2", "summary": "", "questions": "Lines 144-145: Why is the version of metadata from the web site more accurate than that obtained by PDF extraction?\nLines 239-241 (Linguistic Distributeion): There is a difference between the paper being written IN a language and ABOUT a language, and both are interesting as metadata. Have you considered adding both types of metadata? In this respect the claim that \"numerous languages remain underrepresented\" is rather unclear. One can write about Icelandic in English or in Icelandic about English.\nLine 250: When checking the 50 documents for text quality, the resulting claims are rather general - could you provide more accurate assessment (in numbers)?\nLine 357: Is top 10 topic-representative keywords a number chosen arbitrarily or based on some tests?\n", "limitations": "", "rating": 3, "confidence": 3, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer s1m2 for the thoughtful feedback and appreciation of our effort for this important resource. We are excited that the reviewer identified our motivation to build this resource and its value to advance 'Downstream Applications' as detailed in Section 7. \n\n> Reject Reason A. *Corpus building in itself is not a very exciting topic any more and there is a feeling that work was done hastily to some extent - the authors admit themselves that they worked under budget constraints and used whatever was available freely (which is not a problem in itself).*\n\nWe agree with the statement that corpus building, while perhaps not seen as 'very exciting', remains 'valuable' and 'important', thus indispensable to the research community. We wish to clarify that the constraints on budget did not compromise the quality of our work. Our budgetary limitations indeed motivated us on some of our design choices, but these decisions were taken after careful consideration and not out of haste. For example, although we restricted ourselves to open-source toolkits for PDF text extraction, we undertook an evaluation of available options, eventually opting for the most advanced one. We would appreciate any specific examples of hasty decisions that you believe were made, so that we can address them appropriately.\n\n\n> Reject Reason B. *The other issue is the choice of Parquet format - I believe this rather limits the use of the corpus to the CL community. However, there is a large corpus linguistics community which usually uses CSV or other formats for corpora.*\n\nWe acknowledge that Parquet is not as widely recognized as CSV or JSON. However, we opted for Parquet primarily because of its efficient compression and encoding, which reduces required storage space making it more accessible to researchers. Its columnar storage allows for faster query results and better compatibility with data processing tools like pandas.  In addition, Parquet files can be easily transformed to CSV files with a pandas command `parquet_data.to_csv('filename.csv')`. Although, we recognize CSV as a more popular option, thus we will offer a supplementary CSV file containing metadata for all 73K papers, while storing full texts in distinct files.\n\n> QA. *Lines 144-145: Why is the version of metadata from the web site more accurate than that obtained by PDF extraction?* \n\nThe website-sourced metadata is more accurate because it is directly provided by the authors, ensuring its authenticity and precision. In contrast, extracting metadata from PDFs can sometimes introduce errors or miss out on certain data points. Thus, we take the website-sourced metadata in our corpus. We will clarify the rationale in Section 3.1 (Data Acquisition).\n\n> QB. *Lines 239-241 (Linguistic Distributeion): There is a difference between the paper being written IN a language and ABOUT a language, and both are interesting as metadata. Have you considered adding both types of metadata? In this respect the claim that \"numerous languages remain underrepresented\" is rather unclear. One can write about Icelandic in English or in Icelandic about English.* \n\nWe appreciate this insightful suggestion of incorporating both written-IN and written-ABOUT languages. We will consider this as future work and note it in the conclusion section. The assertion that \"numerous languages remain underrepresented\" pertains specifically to the written-IN language.\n\n> QC. *Line 250: When checking the 50 documents for text quality, the resulting claims are rather general - could you provide more accurate assessment (in numbers)?* \n\nThanks for the suggestion. We will add a detailed assessment in the appendix.\n\n> QD. *Line 357: Is top 10 topic-representative keywords a number chosen arbitrarily or based on some tests?*\n\nThe selection was determined through a hyper-parameter search test from {10, 20, 30, 40}. We will add the details to the final version.\n\n\n> Typos Grammar Style And Presentation Improvements: \n*Lines 111-114: I don't understand the grammar of the sentence - it would be better to rephrase. Line 118: \"same like\" -> \"same as\" Line 127: \"more textual analysis\" -> more extensive? (something is missing) Line 146: mostlyconference -> mostly conference*\n\nWe appreciate the reviewer's attention to detail in identifying the typos and offering writing suggestions.\nLines 111-114 have been revised to: \"Similar to other datasets, the OCL features publication metadata, a staple in open scholarly datasets. This can enhance metadata analysis and bibliographic research within the CL domain. A comparison of the OCL with existing datasets, excluding metadata aspects, is presented in Table 1.\" We will carefully revise the draft.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "I think ACL OCL is an important resource and should be presented at EMNLP 2023. As authors note the last similar effort was done some time ago and those corpora are now more or less obsolete, and the information extraction technology available then is outdated. It is positive that someone undertook a rather laborious corpus-building task which is perphaps of more infrastructural nature and less exciting research-wise but the result is quite valuable for reasons that are listed in the \"Downstream Applications\" section in the paper. ", "reasons_to_reject": "Corpus building in itself is not a very exciting topic any more and there is a feeling that work was done hastily to some extent - the authors admit themselves that they worked under budget constraints and used whatever was available freely (which is not a problem in itself). The other issue is the choice of Parquet format - I believe this rather limits the use of the corpus to the CL community. However, there is a large corpus linguistics community which usually uses CSV or other formats for corpora. ", "questions_to_authors": "Lines 144-145: Why is the version of metadata from the web site more accurate than that obtained by PDF extraction?\nLines 239-241 (Linguistic Distributeion): There is a difference between the paper being written IN a language and ABOUT a language, and both are interesting as metadata. Have you considered adding both types of metadata? In this respect the claim that \"numerous languages remain underrepresented\" is rather unclear. One can write about Icelandic in English or in Icelandic about English.\nLine 250: When checking the 50 documents for text quality, the resulting claims are rather general - could you provide more accurate assessment (in numbers)?\nLine 357: Is top 10 topic-representative keywords a number chosen arbitrarily or based on some tests?\n", "paper_topic_and_main_contributions": "The paper is about a corpus of texts from the ACL Anthology web site called Advancing Open Science\nin Computational Linguistics (ACL OCL). Full texts are extracted from PDF files, together with internal structure (sections), metadata is added with automatic topic detection, citation graphs are also available, with links to Semantic Scholar. The corpus is available under open source licence (CC BY-NC 4.0) in Apache Parquet format.\n\n", "excitement": 3, "reproducibility": 2, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Missing_References": "I think references are ok. "}, "extra_text": {"Typos_Grammar_Style_and_Presentation_Improvements": "Lines 111-114: I don't understand the grammar of the sentence - it would be better to rephrase.\nLine 118: \"same like\" -> \"same as\"\nLine 127: \"more textual analysis\" -> more extensive? (something is missing)\nLine 146: mostlyconference -> mostly conference"}}, {"review_id": "Dk92UIdy96", "reviewer": "Reviewer_v8uW", "summary": "", "questions": "A) Is the list of metadata you provide fixed, or will it be possible to include in the future additional metadata also referencing data or software provided?\nB) You do mention the problem of extracting tables from pdf, but it remains unclear how you deal with tables in OCL. Can you please comment.\nC) You also mention TF/IDF as a helpful method to extract possible topics. However, how do you deal with multi-word topics? Have you considered using terminology extraction tools to generate a first pool of possible topics?\n\n\n\n", "limitations": "", "rating": 4, "confidence": 4, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We are heartened that reviewer v8uW recognizes our work as a significant advancement in providing valuable resources and analyses to the community. We are grateful for the insightful comments and the suggested references on research data infrastructures and measurement of the evolution of a field. We find resources in Common Language Resources and Technology Infrastructure (CLARIN) and German text+ are useful guidelines for designing metadata. We will enhance our work by incorporating the recommended references in the revised manuscript.\n\n> QA. *Is the list of metadata you provide fixed, or will it be possible to include in the future additional metadata also referencing data or software provided?*\n\nThe current list of metadata provided is not rigid and can be easily extended. We understand the importance and benefits of including additional metadata, especially those referencing data or software resources related to scholarly work. To facilitate such updates, we can conveniently execute a pandas command. For example, to add a new field named 'lang_about', simply execute the command `metadata['lang-about'] = languages_list`.\n\n\n> QB. *You do mention the problem of extracting tables from pdf, but it remains unclear how you deal with tables in OCL. Can you please comment.*\n\nWe acknowledge the challenges that arise from extracting tables directly from PDFs. As a result, we are currently working with a method wherein tables are included as images and metadata. This information is extracted using 'pdffigures', a tool that detects figures, tables, and other graphical material in scholarly documents. While this might not provide a text-based representation of table contents, it ensures the preservation and availability of table information for in-depth review. We continue to explore more efficient techniques for table extraction and hope to improve this feature in future iterations of the dataset.\n\n\n> QC. *You also mention TF/IDF as a helpful method to extract possible topics. However, how do you deal with multi-word topics? Have you considered using terminology extraction tools to generate a first pool of possible topics?*\n\nNo, we did not use terminology extraction tools to generate a first pool of keywords, instead, we used a multi-word keyword extraction toolkit Yake! for this purpose.  We also consider both unigrams and bi-grams in TF-IDF based keyword extraction method to include multi-word keywords. We will revise Section 5.2 (Keyword-based Supervised Method) with more details.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "The corpus comprises a state of the art textual resource based on the ACL Anthology (spanning almost 70 years of research in the field) with structured full texts, helpful metadata, links to the Semantic Scholar Academic Graph, and figures contained in the papers. It thus marks an important step forward in making available to the community the publication history of ACL papers.\nThe case studies are straightforward and well describe the usefulness, a.o., of citation graphs as well as the detection of research topics (based on unsupervised, semi-supervised, and supervised methods for topic classification). ", "reasons_to_reject": "I do not see strong reasons to reject the paper, neverthess I would recommend to include some references in the state-of-the-art section to \n- scientometrics \n- research data infrastructures, in particular the European CLARIN and the German text+ research data infrastructures on text and linguistic data\n- metadata harvesting", "questions_to_authors": "A) Is the list of metadata you provide fixed, or will it be possible to include in the future additional metadata also referencing data or software provided?\nB) You do mention the problem of extracting tables from pdf, but it remains unclear how you deal with tables in OCL. Can you please comment.\nC) You also mention TF/IDF as a helpful method to extract possible topics. However, how do you deal with multi-word topics? Have you considered using terminology extraction tools to generate a first pool of possible topics?\n\n\n\n", "paper_topic_and_main_contributions": "The paper presents an open corpus derived from the ACL Anthology to assist Open scientific research in the Computational Linguistics/NLP  domain as a replacement of the older ACL ARC and AAN corpora. It comprises metadata, PDF files, citation graphs and additional structured full texts with sections, figures, and links to a large knowledge resource (Semantic Scholar). \nTo demonstrate the usefulness of the corpus, the paper also reports an interesting case study on temporal research trends in the field.", "excitement": 4, "reproducibility": 5, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {"Missing_References": "David Jurgens, Srijan Kumar, Raine Hoover, Dan Mc-Farland, and Dan Jurafsky. 2018. Measuring the evolution of a scientific field through citation frames.Transactions of the Association for Computational Linguistics, 6:391–406."}}, {"review_id": "q9a1HLP5eA", "reviewer": "Reviewer_XBuc", "summary": "", "questions": "N/A.", "limitations": "", "rating": 3, "confidence": 4, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We extend our gratitude to reviewer XBuc for succinctly highlighting our key contributions. We are heartened by the acknowledgment of our resource's potential value and the utility of our topic labels for paper searches. A special thanks is for the meticulous identification of typos, ill-formed citations, overclaims, and ambiguities. We are committed to refining our draft based on these invaluable suggestions.\n\nWe appreciate the reviewer's constructive feedback regarding the length and readability 'very long (hard-to-read at times)' of our draft. We are keen to make it more concise and reader-friendly. We recognize the challenges presented in lines 503-511 and are committed to revising it for clarity. We would be grateful for specific recommendations on which tedious section to condense. \n\n> Reject Reason A. *The main contribution of the paper is the ACL OCL corpus and the anotation of papers with their topic(s). This could have been discussed in a short paper. The rest of the contributions of the paper are minor or look more like ongoing work. In particular, section 7 (Downstream Applications) are considered by this reviewer wishful thinking; the actual implementation of some of the proposals in this section would have meant a greater contribution to the field, up to the standards of an EMNLP long paper.*\n\nWe respectfully disagree with the reviewer's assertion that our work could be condensed into a short paper. The primary contribution, the ACL OCL corpus, is notably richer than previous offerings. Our structured full texts, figures, and links to Semantic Scholar were meticulously curated, offering significant potential for various downstream applications outlined in Section 7. Moreover, our topic labels for OCL papers provide an exemplar of how to derive valuable annotations from the corpus.\n\nWhile we acknowledge the primary focus on the corpus, we assert that the accompanying analyses, such as corpus analysis and trending topic analysis, hold inherent value. For instance, the highlighted exponential growth of annual publications not only signifies the community's progression but also underscores the need for sustainable development. This hints at broadening our research horizons to encompass documents outside our current purview, disseminate significant CL research, and encourage exploration into new or underrepresented topics.\n\nSection 7, \"Downstream Applications\", is intended to illuminate potential applications of the corpus. While it does not present fully-fledged results, it is pivotal in guiding future research trajectories. Predicting such potential directions is a hallmark of corpus studies.\n\nIn conclusion, our work, spanning the comprehensive ACL OCL corpus, methodical topic detection, pertinent analyses, and proposed applications, collectively forms a substantial contribution to the CL domain. We believe this aligns well with the rigors of an EMNLP long paper.\n\n\n> Reject Reason B. *Another interesting contribution would have been setting up the procedures to keep the corpus up-to-date, by means of a sustainable initiative. However, this is simply commented as future work in the paper.*\n\nWe recognize the importance of keeping the corpus current and appreciate the reviewer's emphasis on this aspect. To address this concern, we will incorporate the following strategies in the final version: (1) Open Sourcing the Code. By making our codebase publicly available, we facilitate transparency and encourage collaborative contributions from the research community. (2) Regular updates, including incrementally downloading new or changed data, processing it, and updating the repository. (3) Engaging the Community. We will actively seek community contributions to adding new papers, enriching metadata, or enhancing existing entries.\n\n\n> Typos Grammar Style and Presentation Improvements\n\nFor the data schema, we adhere to the Semantic Scholar's standard schema, designed and maintained by their team. The schema can be accessed at https://api.semanticscholar.org/api-docs/.\n\nWe explain the discussion in Lines 503-511 as follows: As a corpus spanning decades, OCL helps to study diachronic language change in the CL domain (Tahmasebi et al., 2021). To illustrate, consider the term 'prompt'. Traditionally in computer science, it referred to a signal on a computer screen (e.g., 'C:/>') that indicates the system is ready for user input. However, after 2020, its interpretation broadened to encompass a natural language instruction or query presented to an AI system [1]. This newer definition is documented, for instance, in the Cambridge Dictionary (https://dictionary.cambridge.org/dictionary/english/prompt).\n\n\nReference:\n\n[1] Zhengbao Jiang, Frank F. Xu, Jun Araki, and Graham Neubig. 2020. How Can We Know What Language Models Know?. Transactions of the Association for Computational Linguistics, 8:423–438.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "The ACL OCL corpus, built by the authors is an interesting and potentially valuable resource for the CL community. Its topic labels can be quite useful in paper searches by topic within the ACL Anthology in case both resources are eventually and effectively interconnected.", "reasons_to_reject": "The main contribution of the paper is the ACL OCL corpus and the anotation of papers with their topic(s). This could have been discussed in a short paper. The rest of the contributions of the paper are minor or look more like ongoing work. In particular, section 7 (Downstream Applications) are considered by this reviewer wishful thinking; the actual implementation of some of the proposals in this section would have meant a greater contribution to the field, up to the standards of an EMNLP long paper. Another interesting contribution would have been setting up the procedures to keep the corpus up-to-date, by means of a sustainable initiative. However, this is simply commented as future work in the paper.", "questions_to_authors": "N/A.", "paper_topic_and_main_contributions": "This very long (hard-to-read at times) paper provides three main contributions to the field of Computational Linguistics (CL). Firstly, it presents a new open source corpus (namely the ACL OCL corpus), based on the materials included in the ACL Anthology, enhancing this collection with \"metadata [including the annotation of papers with their corresponding (main) topic, not present in the original resource], PDF files, citation graphs and additional structured full texts with sections, figures, and links to [...] Semantic Scholar\". Secondly, some basic descriptive statistics methods are applied on the ACL OCL corpus in order to showcase its usefulness, i.e., by identifying trends in the topics being discussed in the papers in the ACL Anthology. And thirdly, it discusses potential scenarios where this new corpus could be applied for further research.", "excitement": 3, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Missing_References": "N/A."}, "extra_text": {"Typos_Grammar_Style_and_Presentation_Improvements": "- Perhaps the statement \"The ACL Anthology is the key resource that digitally archives *all* scientific papers in the CL 032 domain\", in lines 030-032, should be conveniently hedged. There is life in CL beyond the ACL Anthology... Or so they say.\n\n- There seems to be a typo in Table 1: the \"k\" in \"73.3k\" should be perhaps capitalized?\n\n- There is space missing in line 147, between \"handbooks.\" and \"We\"; and also in line 499, between \"up\" and \"opportunities\".\n\n- In lines 198-199, the authors mention the total number of \"internal directed connections among the 73K papers in ACL OCL\". However, the number of external connections is missing. Mentioning this other number would be highly appreciated.\n\n-  In line 202, the authors state that their \"dataset adheres to a standard schema\", but they fail to mention what standard it is, by whom it was developed, and/or where this standard can be retrieved.\n\n- In line 338, there is an ill-formed inline citation, that is, \"in (Yin et al., 2019)\", which should be cited as \"in Yin et al. (2019)\" instead. Please, check the rest of the inline references in the paper, as there might be some other occurrences that need fixing this way.  \n\n- In line 471, \" and\" should be replaced by a comma (\",\").\n\n- The discussion in lines 503-511 should be conveniently rephrased. To start with, the description of 'prompt' is quite arguable, and so is its description in relation with current AI agents querying.\n\n- Thre seems to be a tpo in line 554 - \"any\" may have been meant instead of \"and\"?"}}]

Author rebuttal:

Meta-review: This paper presents a scholarly corpus containing 73K peer reviewed publications from various computational linguistics conferences and workshops from the past seven decades. It massively extends upon existing corpora, whose contents may now be considered obsolete and includes topic labels for a subset of publications that may benefit downstream research on topic classification. Reviewers were pleased with the design of the corpus, the data collection and curation methodology (two 3s, two 4s for Soundness), and the inclusion of case studies demonstrating its utility and impact (e.g. the corpus may be used to observe waning and resurgence trends among particular topics). There were some concerns over the format used (Parquet) and the lack of labels for every item in the corpus, but these were addressed to reviewer satisfaction during the author rebuttal period. Lastly, though corpus creation is not particularly exciting research, the Excitement scores were still relatively high (two 3s, two 4s). One reviewer also suggested this submission would be better as a short paper, but I believe the authors’ defense to be sufficient. I would, however,  advise authors to consider extending their bullet point list of contributions to also include those mentioned in the rebuttal.

In summary, only minor revisions, addressing reviewers’ comments and questions, need to be made to ensure this paper is camera ready.
Decision: Accept-Main