NYTAC-CC: A Climate Change Subcorpus based on New York Times Articles
Authors: Francesca Grasso, Ronny Patz, Manfred Stede (Università di Torino, University of Potsdam, 2024)
Topic: Lexical and Semantic Resources and Analysis

Abstract
Over the past decade, the analysis of discourses on climate change (CC) has gained increased interest within the social sciences and the NLP community. Textual resources are crucial for understanding how narratives about this phenomenon are crafted and delivered. However, %while there is growing attention on social media resources, there still is a scarcity of datasets that cover CC in news media in a representative way. This paper presents a CC-specific subcorpus of 3,630 articles extracted from the 1.8 million New York Times Annotated Corpus, marking the first CC analysis on this data. The subcorpus was created by combining different methods for text selection to ensure representativeness and reliability, which is validated using ClimateBERT. To provide initial insights into the CC subcorpus, we discuss the results of a topic modeling experiment (LDA). These show the diversity of contexts in which CC is discussed in news media over time. with \verb|| and ends with \verb||. This is an abstract.

Introduction
We present NYTAC-CC, a topic-specific subcorpus with 3,630 articles addressing climate change (CC), derived from the New York Times Annotated Corpus. This subcorpus covers a 20-year period, drawing from NYTAC’s collection of 1.8 million articles published between 1987 and 2007, which is available through the Linguistic Data Consortium. The original corpus, and thus also the subcorpus, includes a variety of metadata such as the `desk' (the newspaper branch) and both manually- and automatically-labeled content categories, with many articles also featuring hand-written summaries. %tags that categorize the content. Furthermore, a sizable subset of the articles includes hand-written summaries. The extensive use of NYTAC in NLP research over the last 15 years (e.g., AUTHOR) benefits CC researchers, allowing for detailed historical analysis of CC discussions in news media. This includes exploring how CC debates were interwoven with topics like domestic and foreign policy, science reporting, and arts and culture coverage. Unlike other CC-focused resources that often contain shorter documents, the NYTAC-CC subcorpus offers a diverse array of articles with varying lengths and complex content, making it a unique resource for investigating the evolution of CC narratives over time. The contribution of this paper is threefold: (i) We present the NYTAC-CC subcorpus %(by publishing the filenames) and its construction using blending of dictionary-based and supervised methods in order to ensure representativeness as well as validity and reliability, which are key in social science research AUTHOR. This hybrid approach addresses the challenges of refining a topic-specific subcorpus from a larger corpus, aiming to mitigate the limitations of traditional keyword-based sampling that often results in false positives. %This hybrid approach is designed to address the challenges associated with refining a topic-specific subcorpus or extracting relevant information from a larger, existing corpus. It aims to overcome the limitations of traditional sampling techniques, which often involve retrieving articles via a small set of keywords or bigrams and can lead to the inclusion of false positives in the datasets. %we propose a more laborate method for text extraction. %We describe a method for extracting a CC topic-specific subcorpus from a large article collection (here: NYTAC). Previous work was usually restricted to retrieving articles via a small set of keywords or bigrams; we propose here a more elaborate method that can contribute to the important goal of representativeness of data sets used in social science research (besides validity and reliability; cf., for example, AUTHOR). (ii) To demonstrate the validity of the subcorpus, and thus its reliability for further downstream tasks, we illustrate the results of a classification experiment using ClimateBERT AUTHOR. %a BERT-based model specifically trained on CC-related texts, While this experiment further validates that the articles in our NYTAC-CC subcorpus are, indeed, true positives, it also shows limitations of ClimateBERT. As ClimateBERT falsely classifies a number of true positives from our subcorpus as (false) negatives, we demonstrate that our approach achieves better results in ensuring recall of relevant CC articles from the NYTAC corpus. (iii) To gain initial insights into the CC subcorpus coverage, we use keyword analysis and topic modeling (specifically LDA) to track specifics of CC reporting over the 1987-2007 time span. The results show important trends over time, including key periods of reporting and a large variety of contexts in which CC is discussed. %Given this data source (subcorpus), we apply topic modeling (in particular LDA) to track the development of climate change reporting, specifically the coverage of subtopics, in the NYT for the time span 1987-2007. Thus, our goal is to provide a substantively new and relevant subcorpus, developed and validated in multiple iterations, and to then provide a first overview of the NYT's coverage of climate change during the time period covered in our corpus. Although several studies have explored U.S. print media’s reporting on anthropogenic CC, we cover an important 20-year period in which much of today's climate change discourse evolved. The rest of the paper is structured as follows: Section sct:relwork summarizes relevant related work. Section sct:corpus describes the process of creating the topic-specific subcorpus from the NYTAC through the aforementioned combined method, as well as results from classification task using ClimateBERT to validate the NYTAC-CC subcorpus. Section sct:corpstat offers key observations on the content and distribution of NYTAC-CC articles, including the outcome of an LDA experiment for unsupervised sub-topic exploration within the subcorpus. We conclude in Section sct:conc with suggestions for future work based on the subcorpus. %Section sct:vanilla reports on our LDA experiments, CEUR-WS's article template provides a consistent \LaTeX{} style for use across CEUR-WS publications, and incorporates accessibility and metadata-extraction functionality. This document will explain the major features of the document class. If you are new to publishing with CEUR-WS, this document is a valuable guide to the process of preparing your work for publication. The ``\verb|ceurart|'' document class can be used to prepare articles for any CEUR-WS publication, and for any stage of publication, from review to final ``camera-ready'' copy with {\itshape very} few changes to the source. This class depends on the following packages for its proper functioning: \item \verb|natbib.sty| for citation processing; \item \verb|geometry.sty| for margin settings; \item \verb|graphicx.sty| for graphics inclusion; \item \verb|hyperref.sty| optional package if hyperlinking is required in the document; \item \verb|fontawesome5.sty| optional package for bells and whistles. All the above packages are part of any standard \LaTeX{} installation. Therefore, the users need not be bothered about downloading any extra packages.

Conclusion
In this paper, we introduced the NYTAC-CC, a specialized subcorpus of 3,630 climate change articles from the New York Times Annotated Corpus spanning 1987 to 2007, marking the first CC analysis with this dataset. Addressing the lack of available news-based textual resources for NLP tasks, we employed a hybrid method combining keyword-based prefiltering and automatic classification to optimize the corpus construction. The representativeness of the subcorpus was confirmed using ClimateBERT, but additional manual inspection of ClimateBERT's classification of a relevant amount of true positives as (false) negatives also showed the model's limitations and the benefits of the hybrid approach chosen. Initial analyses of the subcorpus, including statistics, keyword searches, and topic modeling, highlight the corpus's potential for detailed diachronic and subtopic exploration. Thus, the NYTAC-CC subcorpus can be a useful resource for examining the historical narrative of climate change in news media. As it builds on the NYTAC corpus, it adds to previous work on this data, providing valuable insights for social science research. It also serves as a beneficial dataset for developing NLP applications that require a deep understanding of climate-related discourse. While the size of the subcorpus may restrict certain quantitative analyses, its rich, concentrated content is ideal for qualitative studies. Furthermore, it offers the potential for expansion and further integration with additional sources to enhance its utility and relevance for ongoing climate change research. Future work will expand on these findings with advanced topic modeling techniques and integrate more recent articles to enrich the diachronic analysis. In this paper, we introduced the NYTAC-CC, a topic-specific subcorpus of 3,630 articles on climate change (CC) %of variable length and heterogeneous content, drawn from the New York Times Annotated Corpus covering the span from 1987 to 2007. This marks the first CC analysis using this data set. Our work addresses the scarcity of available news-based textual resources for climate change, crucial for many NLP downstream tasks. %by providing a representative and reliable collection. We constructed the corpus using a hybrid approach that combines keyword-based prefiltering with automatic classification, effectively optimizing the extraction process. %and overcoming methodological limitations while leveraging on their strengths. The representativeness of the subcorpus is validated through the application of ClimateBERT, with classification results revealing both the model's intrinsic limitations %(primarily due to its limited context length) and the topic-consistency of the subcorpus. %in addressing CC-related news. Initial explorations, including basic statistics, keyword analysis, and topic modeling, have already highlighted the potential for nuanced diachronic analysis and more fine-grained subtopic exploration. As future work, we plan to extend these preliminary findings by employing advanced topic modeling techniques, such as structured topic modeling that systematically incorporates time and dynamic topic modeling. %These efforts aim to further enrich our understanding of the evolving narratives on climate change. Newspaper text documents and corpora can be complex in terms of their "inner logic" and metadata assignment. We worked with the NYTAC, which contains certain types of documents that involve a set of potentially very different news items (making content analysis more difficult) as well as noise in the manually-assigned topic terms. Building a reliable topic-specific subcorpus is therefore not trivial, and we experimented with different methods before settling on our "climate change" subcorpus. Our procedure contrasts with the typical of approach of previous research, which either uses just a small number of query terms (often only only the CC bigram) or involves manual confirmation. The former has problems for both precision and recall, and the latter is feasible only when the overall corpus is reasonably small (as in the case of AUTHOR. We see our method of keyword-based pre-filtering and subsequent supervised classification as a compromise between quality and efficiency requirements, suitable for relatively large document collections. We showed basic statistics on the patterns of climate change coverage in the NYT over a 20 year period, and then extended this by running topic modeling (LDA) on the texts when reduced to just their nominal terms. The resulting set of 18 topics seems to us to reflect the subtopic "landscape" of climate change quite well, and the topic-document assignment can now be used for more specific content analyses. The IDs of the texts that we identified as the NYTAC CC subcorpus will be made available for research by other licensees of the corpus. For our own future work, we plan to do further experiments with topic modeling, including more recent algorithms than LDA, such as structured topic modeling (where the time can modeled systematically) and dynamic topic modeling. Future Work: \item apply Classification Experiments on the whole corpus (?) \item perform both qualitative and quantitative dyachronic analysis on the NYTAC-CC over time both in terms of language use and of more fine-grained analysis on subtopics overtime