GenAI4ELab/papercli-papers huggingface dataset:
Sociocultural Considerations in Monitoring Anti-LGBTQ+ Content on Social Media
Authors: Sidney Wong
Venue: ACL 2024 (Proceedings of the 2nd Workshop on Cross-Cultural Considerations in NLP)

Abstract: The purpose of this paper is to ascertain the influence of sociocultural factors (i.e., social, cultural, and political) in the development of hate speech detection systems. We set out to investigate the suitability of using open-source training data to monitor levels of anti-LGBTQ+ content on social media across different national-varieties of English. Our findings suggests the social and cultural alignment of open-source hate speech data sets influences the predicted outputs. Furthermore, the keyword-search approach of anti-LGBTQ+ slurs in the development of open-source training data encourages detection models to overfit on slurs; therefore, anti-LGBTQ+ content may go undetected. We recommend combining empirical outputs with qualitative insights to ensure these systems are fit for purpose.
Are Generative Language Models Multicultural? A Study on Hausa Culture and Emotions using ChatGPT
Authors: Ibrahim Ahmad; Shiran Dudy; Resmi Ramachandranpillai; Kenneth Church
Venue: ACL 2024 (Proceedings of the 2nd Workshop on Cross-Cultural Considerations in NLP)

Abstract: Large Language Models (LLMs), such as ChatGPT, are widely used to generate content for various purposes and audiences. However, these models may not reflect the cultural and emotional diversity of their users, especially for low-resource languages. In this paper, we investigate how ChatGPT represents Hausa’s culture and emotions. We compare responses generated by ChatGPT with those provided by native Hausa speakers on 37 culturally relevant questions. We conducted experiments using emotion analysis. We also used two similarity metrics to measure the alignment between human and ChatGPT responses. We also collect human participants ratings and feedback on ChatGPT responses. Our results show that ChatGPT has some level of similarity to human responses, but also exhibits some gaps and biases in its knowledge and awareness of Hausa culture and emotions. We discuss the implications and limitations of our methodology and analysis and suggest ways to improve the performance and evaluation of LLMs for low-resource languages.
Computational Language Documentation: Designing a Modular Annotation and Data Management Tool for Cross-cultural Applicability
Authors: Alexandra O’Neil; Daniel Swanson; Shobhana Chelliah
Venue: ACL 2024 (Proceedings of the 2nd Workshop on Cross-Cultural Considerations in NLP)

Abstract: While developing computational language documentation tools, researchers must center the role of language communities in the process by carefully reflecting on and designing tools to support the varying needs and priorities of different language communities. This paper provides an example of how cross-cultural considerations discussed in literature about language documentation, data sovereignty, and community-led documentation projects can motivate the design of a computational language documentation tool by reflecting on our design process as we work towards developing an annotation and data management tool. We identify three recurring themes for cross-cultural consideration in the literature - Linguistic Sovereignty, Cultural Specificity, and Reciprocity - and present eight essential features for an annotation and data management tool that reflect these themes.
Climate Policy Transformer: Utilizing NLP to track the coherence of Climate Policy Documents in the Context of the Paris Agreement
Authors: Prashant Singh; Erik Lehmann; Mark Tyrrell
Venue: ACL 2024 (Proceedings of the 1st Workshop on Natural Language Processing Meets Climate Change (ClimateNLP 2024))

Abstract: Climate policy implementation is pivotal inglobal efforts to mitigate and adapt to climatechange. In this context, this paper explores theuse of Natural Language Processing (NLP) as atool for policy advisors to efficiently track andassess climate policy and strategies, such asNationally Determined Contributions (NDCs).These documents are essential for monitoringcoherence with the Paris Agreement, yet theiranalysis traditionally demands significant la-bor and time. We demonstrate how to leverageNLP on existing climate policy databases totransform this process by structuring informa-tion extracted from these otherwise unstruc-tured policy documents and opening avenuesfor a more in-depth analysis of national and re-gional policies. Central to our approach is thecreation of a machine-learning (ML) dataset’CPo-CD’, based on data provided by the Inter-national Climate Initiative (IKI) and ClimateWatch (CW). The CPo-CD dataset is utilizedto fine-tune Transformer Models on classify-ing climate targets, actions, policies, and plans,along with their sector, mitigation-adaptation,and greenhouse gas (GHG) components. Wepublish our model and dataset on a HuggingFace repository.
Informing climate risk analysis using textual information - A research agenda
Authors: Andreas Dimmelmeier; Hendrik Doll; Malte Schierholz; Emily Kormanyos; Maurice Fehr; Bolei Ma; Jacob Beck; Alexander Fraser; Frauke Kreuter
Venue: ACL 2024 (Proceedings of the 1st Workshop on Natural Language Processing Meets Climate Change (ClimateNLP 2024))

Abstract: We present a research agenda focused on efficiently extracting, assuring quality, and consolidating textual company sustainability information to address urgent climate change decision-making needs. Starting from the goal to create integrated FAIR (Findable, Accessible, Interoperable, Reusable) climate-related data, we identify research needs pertaining to the technical aspects of information extraction as well as to the design of the integrated sustainability datasets that we seek to compile. Regarding extraction, we leverage technological advancements, particularly in large language models (LLMs) and Retrieval-Augmented Generation (RAG) pipelines, to unlock the underutilized potential of unstructured textual information contained in corporate sustainability reports. In applying these techniques, we review key challenges, which include the retrieval and extraction of CO2 emission values from PDF documents, especially from unstructured tables and graphs therein, and the validation of automatically extracted data through comparisons with human-annotated values. We also review how existing use cases and practices in climate risk analytics relate to choices of what textual information should be extracted and how it could be linked to existing structured data.
My Climate Advisor: An Application of NLP in Climate Adaptation for Agriculture
Authors: Vincent Nguyen; Sarvnaz Karimi; Willow Hallgren; Ashley Harkin; Mahesh Prakash
Venue: ACL 2024 (Proceedings of the 1st Workshop on Natural Language Processing Meets Climate Change (ClimateNLP 2024))

Abstract: Climate adaptation in the agricultural sector necessitates tools that equip farmers and farm advisors with relevant and trustworthy information to help increase their resiliency to climate change. We introduce My Climate Advisor, a question-answering (QA) prototype that synthesises information from different data sources, such as peer-reviewed scientific literature and high-quality, industry-relevant grey literature to generate answers, with references, to a given user’s question. Our prototype uses open-source generative models for data privacy and intellectual property protection, and retrieval augmented generation for answer generation, grounding and provenance. While there are standard evaluation metrics for QA systems, no existing evaluation framework suits our LLM-based QA application in the climate adaptation domain. We design an evaluation framework with seven metrics based on the requirements of the domain experts to judge the generated answers from 12 different LLM-based models. Our initial evaluations through a user study via domain experts show promising usability results.
Generative Debunking of Climate Misinformation
Authors: Francisco Zanartu; Yulia Otmakhova; John Cook; Lea Frermann
Venue: ACL 2024 (Proceedings of the 1st Workshop on Natural Language Processing Meets Climate Change (ClimateNLP 2024))

Abstract: Misinformation about climate change causes numerous negative impacts, necessitating corrective responses. Psychological research has offered various strategies for reducing the influence of climate misinformation, such as the fact-myth-fallacy-fact-structure. However, practically implementing corrective interventions at scale represents a challenge. Automatic detection and correction of misinformation offers a solution to the misinformation problem. This study documents the development of large language models that accept as input a climate myth and produce a debunking that adheres to the fact-myth-fallacy-fact (“truth sandwich”) structure, by incorporating contrarian claim classification and fallacy detection into an LLM prompting framework. We combine open (Mixtral, Palm2) and proprietary (GPT-4) LLMs with prompting strategies of varying complexity. Experiments reveal promising performance of GPT-4 and Mixtral if combined with structured prompts. We identify specific challenges of debunking generation and human evaluation, and map out avenues for future work. We release a dataset of high-quality truth-sandwich debunkings, source code and a demo of the debunking system.
Decoding Climate Disagreement: A Graph Neural Network-Based Approach to Understanding Social Media Dynamics
Authors: Ruiran Su; Janet Pierrehumbert
Venue: ACL 2024 (Proceedings of the 1st Workshop on Natural Language Processing Meets Climate Change (ClimateNLP 2024))

Abstract: This paper presents the ClimateSent-GAT Model, a novel approach that combines Graph Attention Networks (GATs) with natural language processing techniques to accurately identify and predict disagreements within Reddit comment-reply pairs. Our model classifies disagreements into three categories: agree, disagree, and neutral. Leveraging the inherent graph structure of Reddit comment-reply pairs, the model significantly outperforms existing benchmarks by capturing complex interaction patterns and sentiment dynamics. This research advances graph-based NLP methodologies and provides actionable insights for policymakers and educators in climate science communication.
Evaluating ChatNetZero, an LLM-Chatbot to Demystify Climate Pledges
Authors: Angel Hsu; Mason Laney; Ji Zhang; Diego Manya; Linda Farczadi
Venue: ACL 2024 (Proceedings of the 1st Workshop on Natural Language Processing Meets Climate Change (ClimateNLP 2024))

Abstract: This paper introduces and evaluates ChatNetZero, a large-language model (LLM) chatbot developed through Retrieval-Augmented Generation (RAG), which uses generative AI to produce answers grounded in verified, climate-domain specific information. We describe ChatNetZero’s design, particularly the innovation of anti-hallucination and reference modules designed to enhance the accuracy and credibility of generated responses. To evaluate ChatNetZero’s performance against other LLMs, including GPT-4, Gemini, Coral, and ChatClimate, we conduct two types of validation: comparing LLMs’ generated responses to original source documents to verify their factual accuracy, and employing an expert survey to evaluate the overall quality, accuracy and relevance of each response. We find that while ChatNetZero responses show higher factual accuracy when compared to original source data, experts surveyed prefer lengthier responses that provide more context. Our results highlight the importance of prioritizing information presentation in the design of domain-specific LLMs to ensure that scientific information is effectively communicated, especially as even expert audiences find it challenging to assess the credibility of AI-generated content.
Using LLMs to Build a Database of Climate Extreme Impacts
Authors: Ni Li; Shorouq Zahra; Mariana Brito; Clare Flynn; Olof Görnerup; Koffi Worou; Murathan Kurfali; Chanjuan Meng; Wim Thiery; Jakob Zscheischler; Gabriele Messori; Joakim Nivre
Venue: ACL 2024 (Proceedings of the 1st Workshop on Natural Language Processing Meets Climate Change (ClimateNLP 2024))

Abstract: To better understand how extreme climate events impact society, we need to increase the availability of accurate and comprehensive information about these impacts. We propose a method for building large-scale databases of climate extreme impacts from online textual sources, using LLMs for information extraction in combination with more traditional NLP techniques to improve accuracy and consistency. We evaluate the method against a small benchmark database created by human experts and find that extraction accuracy varies for different types of information. We compare three different LLMs and find that, while the commercial GPT-4 model gives the best performance overall, the open-source models Mistral and Mixtral are competitive for some types of information.
Envisioning NLP for intercultural climate communication
Authors: Steven Bird; Angelina Aquino; Ian Gumbula
Venue: ACL 2024 (Proceedings of the 1st Workshop on Natural Language Processing Meets Climate Change (ClimateNLP 2024))

Abstract: Climate communication is often seen by the NLP community as an opportunity for machine translation, applied to ever smaller languages. However, over 90% the world’s linguistic diversity comes from languages with ‘primary orality’ and mostly spoken in non-Western oral societies. A case in point is the Aboriginal communities of Northern Australia, where we have been conducting workshops on climate communication, revealing shortcomings in existing communication practices along with new opportunities for improving intercultural communication. We present a case study of climate communication in an oral society, including the voices of many local people, and draw several lessons for the research program of NLP in the climate space.
EnClaim: A Style Augmented Transformer Architecture for Environmental Claim Detection
Authors: Diya Saha; Manjira Sinha; Tirthankar Dasgupta
Venue: ACL 2024 (Proceedings of the 1st Workshop on Natural Language Processing Meets Climate Change (ClimateNLP 2024))

Abstract: Across countries, a noteworthy paradigm shift towards a more sustainable and environmentally responsible economy is underway. However, this positive transition is accompanied by an upsurge in greenwashing, where companies make exaggerated claims about their environmental commitments. To address this challenge and protect consumers, initiatives have emerged to substantiate green claims. With the proliferation of environmental and scientific assertions, a critical need arises for automated methods to detect and validate these claims at scale. In this paper, we introduce EnClaim, a transformer network architecture augmented with stylistic features for automatically detecting claims from open web documents or social media posts. The proposed model considers various linguistic stylistic features in conjunction with language models to predict whether a given statement constitutes a claim. We have rigorously evaluated the model using multiple open datasets. Our initial findings indicate that incorporating stylistic vectors alongside the BERT-based language model enhances the overall effectiveness of environmental claim detection.
LEAF: Predicting the Environmental Impact of Food Products based on their Name
Authors: Bas Krahmer
Venue: ACL 2024 (Proceedings of the 1st Workshop on Natural Language Processing Meets Climate Change (ClimateNLP 2024))

Abstract: Although food consumption represents a sub- stantial global source of greenhouse gas emis- sions, assessing the environmental impact of off-the-shelf products remains challenging. Currently, this information is often unavailable, hindering informed consumer decisions when grocery shopping. The present work introduces a new set of models called LEAF, which stands for Linguistic Environmental Analysis of Food Products. LEAF models predict the life-cycle environmental impact of food products based on their name. It is shown that LEAF models can accurately predict the environmental im- pact based on just the product name in a multi- lingual setting, greatly outperforming zero-shot classification methods. Models of varying sizes and capabilities are released, along with the code and dataset to fully reproduce the study.
Large Scale Narrative Messaging around Climate Change: A Cross-Cultural Comparison
Authors: Haiqi Zhou; David G Hobson; Derek Ruths; Andrew Piper
Venue: ACL 2024 (Proceedings of the 1st Workshop on Natural Language Processing Meets Climate Change (ClimateNLP 2024))

Abstract: In this study, we explore the use of Large Language Models (LLMs) such as GPT-4 to extract and analyze the latent narrative messaging in climate change-related news articles from North American and Chinese media. By defining “narrative messaging” as the intrinsic moral or lesson of a story, we apply our model to a dataset of approximately 15,000 news articles in English and Mandarin, categorized by climate-related topics and ideological groupings. Our findings reveal distinct differences in the narrative values emphasized by different cultural and ideological contexts, with North American sources often focusing on individualistic and crisis-driven themes, while Chinese sources emphasize developmental and cooperative narratives. This work demonstrates the potential of LLMs in understanding and influencing climate communication, offering new insights into the collective belief systems that shape public discourse on climate change across different cultures.
Challenges in End-to-End Policy Extraction from Climate Action Plans
Authors: Nupoor Gandhi; Tom Corringham; Emma Strubell
Venue: ACL 2024 (Proceedings of the 1st Workshop on Natural Language Processing Meets Climate Change (ClimateNLP 2024))

Abstract: Gray policy literature such as climate action plans (CAPs) provide an information-rich resource with potential to inform analysis and decision-making. However, these corpora are currently underutilized due to the substantial manual effort and expertise required to sift through long and detailed documents. Automatically structuring relevant information using information extraction (IE) would be useful for assisting policy scientists in synthesizing vast gray policy corpora to identify relevant entities, concepts and themes. LLMs have demonstrated strong performance on IE tasks in the few-shot setting, but it is unclear whether these gains transfer to gray policy literature which differs significantly to traditional benchmark datasets in several aspects, such as format of information content, length of documents, and inconsistency of document structure. We perform a case study on end-to-end IE with California CAPs, inspecting the performance of state-of-the-art tools for: (1) extracting content from CAPs into structured markup segments; (2) few-shot IE with LLMs; and (3) the utility of extracted entities for downstream analyses. We identify challenges at several points of the end-to-end IE pipeline for CAPs, and we provide recommendations for open problems centered around representing rich non-textual elements, document structure, flexible annotation schemes, and global information. Tackling these challenges would make it possible to realize the potential of LLMs for IE with gray policy literature.
Structuring Sustainability Reports for Environmental Standards with LLMs guided by Ontology
Authors: Aida Usmanova; Ricardo Usbeck
Venue: ACL 2024 (Proceedings of the 1st Workshop on Natural Language Processing Meets Climate Change (ClimateNLP 2024))

Abstract: Following the introduction of the European Sustainability Reporting Standard (ESRS), companies will have to adapt to a new policy and provide mandatory sustainability reports. However, implementing such reports entails a challenge, such as the comprehension of a large number of textual information from various sources. This task can be accelerated by employing Large Language Models (LLMs) and ontologies to effectively model the domain knowledge. In this study, we extended an existing ontology to model ESRS Topical Standard for disclosure. The developed ontology would enable automated reasoning over the data and assist in constructing Knowledge Graphs (KGs). Moreover, the proposed ontology extension would also help to identify gaps in companies’ sustainability reports with regard to the ESRS requirements.Additionally, we extracted knowledge from corporate sustainability reports via LLMs guided with a proposed ontology and developed their KG representation.
Unlearning Climate Misinformation in Large Language Models
Authors: Michael Fore; Simranjit Singh; Chaehong Lee; Amritanshu Pandey; Antonios Anastasopoulos; Dimitrios Stamoulis
Venue: ACL 2024 (Proceedings of the 1st Workshop on Natural Language Processing Meets Climate Change (ClimateNLP 2024))

Abstract: Misinformation regarding climate change is a key roadblock in addressing one of the most serious threats to humanity. This paper investigates factual accuracy in large language models (LLMs) regarding climate information. Using true/false labeled Q&A data for fine-tuning and evaluating LLMs on climate-related claims, we compare open-source models, assessing their ability to generate truthful responses to climate change questions. We investigate the detectability of models intentionally poisoned with false climate information, finding that such poisoning may not affect the accuracy of a model’s responses in other domains. Furthermore, we compare the effectiveness of unlearning algorithms, fine-tuning, and Retrieval-Augmented Generation (RAG) for factually grounding LLMs on climate change topics. Our evaluation reveals that unlearning algorithms can be effective for nuanced conceptual claims, despite previous findings suggesting their inefficacy in privacy contexts. These insights aim to guide the development of more factually reliable LLMs and highlight the need for additional work to secure LLMs against misinformation attacks.
Statements: Universal Information Extraction from Tables with Large Language Models for ESG KPIs
Authors: Lokesh Mishra; Sohayl Dhibi; Yusik Kim; Cesar Berrospi Ramis; Shubham Gupta; Michele Dolfi; Peter Staar
Venue: ACL 2024 (Proceedings of the 1st Workshop on Natural Language Processing Meets Climate Change (ClimateNLP 2024))

Abstract: Environment, Social, and Governance (ESG) KPIs assess an organization’s performance on issues such as climate change, greenhouse gas emissions, water consumption, waste management, human rights, diversity, and policies. ESG reports convey this valuable quantitative information through tables. Unfortunately, extracting this information is difficult due to high variability in the table structure as well as content. We propose Statements, a novel domain agnostic data structure for extracting quantitative facts and related information. We propose translating tables to statements as a new supervised deep-learning universal information extraction task. We introduce SemTabNet - a dataset of over 100K annotated tables. Investigating a family of T5-based Statement Extraction Models, our best model generates statements which are 82% similar to the ground-truth (compared to baseline of 21%). We demonstrate the advantages of statements by applying our model to over 2700 tables from ESG reports. The homogeneous nature of statements permits exploratory data analysis on expansive information found in large collections of ESG reports.