<figure><img src="image_1.png" /></figure>

HAL
open science

# Benchmarking the Benchmarks: Reproducing Climate-Related NLP Tasks

Tom Calamai, Oana Balalau, Fabian M Suchanek

<figure><img src="image_2.png" /></figure>

## To cite this version:

Tom Calamai, Oana Balalau, Fabian M Suchanek. Benchmarking the Benchmarks: Reproducing Climate-Related NLP Tasks. ACL 2025 - The 63rd Annual Meeting of the Association for Computational Linguistics, ACL, Jul 2025, Vienne, Austria. hal-05115308

HAL Id: hal-05115308

https://hal.science/hal-05115308v1

Submitted on 16 Jun 2025

HAL is a multi-disciplinary open access archive for the deposit and dissemination of scientific research documents, whether they are published or not. The documents may come from teaching and research institutions in France or abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est destinée au dépôt et à la diffusion de documents scientifiques de niveau recherche, publiés ou non, émanant des établissements d'enseignement et de recherche français ou étrangers, des laboratoires publics ou privés.

<figure><img src="image_3.png" /></figure>

Distributed under a Creative Commons Attribution 4.0 International License
# Benchmarking the Benchmarks: Reproducing Climate-Related NLP Tasks

**Tom Calamai**

**Oana Balalau**

**Fabian M. Suchanek**

Amundi,

Inria,

Télécom Paris,

Télécom Paris, Inria,

Institut Polytechnique de Paris,

Institut Polytechnique de Paris,

Institut Polytechnique de Paris,

France

France

France

oana.balalau@inria.fr

suchanek@telecom-paris.fr

tom.calamai@amundi.com

## Abstract

Significant efforts have been made in the NLP community to facilitate the automatic analysis of climate-related corpora by tasks such as climate-related topic detection, climate risk classification, question answering over climate topics, and many more. In this work, we perform a reproducibility study on 8 tasks and 29 datasets, testing 6 models. We find that many tasks rely heavily on surface-level keyword patterns rather than deeper semantic or contextual understanding. Moreover, we find that 96% of the datasets contain annotation issues, with 16.6% of the sampled wrong predictions of a zero-shot classifier being actually clear annotation mistakes, and 38.8% being ambiguous examples. These results call into question the reliability of current benchmarks to meaningfully compare models and highlight the need for improved annotation practices. We conclude by outlining actionable recommendations to enhance dataset quality and evaluation robustness.

## 1 Introduction

As climate change becomes a more and more urgent problem (The World Meteorological Organization (WMO)), there has been a surge of interest in the field of climate-related natural language processing (NLP). Over 60 recent works study tasks such as detecting climate-related discourse, identifying green claims in corporate communications, answering climate-related questions, and detecting deceptive patterns in climate-related communication (Calamai et al., 2025). And yet, the progress in this area is hard to gauge, for several reasons: First, the approaches are evaluated on different datasets even if they treat the same tasks; second, the evaluations differ in their metrics, which makes the results incomparable; third, the approaches are rarely compared to simple TF-IDF baselines, so that it is unclear to what degree the proposed problems can be solved by simpler methods as well; and finally,

few of the works use zero-shot LLMs as competitors, so that it is hard to judge to what degree the proposed methods have become obsolete with the arrival of generative models.

In this paper, we set out to make a transversal and unified comparison of climate-related NLP tasks. We compiled 29 climate-related NLP datasets and evaluated a suite of baselines that cover traditional machine learning approaches such as TF-IDF with logistic regression, as well as fine-tuned Transformer models, and open and closed source LLMs in a standardized setting – effectively building a benchmark. We also performed an in-depth error analysis, with more than 500 manually annotated errors.

Our works yields the following insights:

1. TF-IDF, finetuned models, and zero-shot LLMs perform similarly, with finetuned models performing slightly better. While this confirms that current models are capable of solving these tasks, it also questions the difficulty of the tasks. If TF-IDF performs well, as we systematically show, then this means that the datasets consist of simple examples that can be predicted using word-frequency only.

2. 96% of our datasets contain annotation issues, with 16.6% of the sampled wrong predictions of a zero-shot classifier being actually clear annotation mistakes, and 38.8% being ambiguous examples where multiple labels could fit or where it is not clear which label to assign.

3. Our manual analysis shows that finetuned models may perform better not because they generalize better, but because they overfit on annotation noise, bias, or unclear guidelines.

These findings highlight the limitations of the datasets to meaningfully compare models, because their simplicity and inconsistencies compress performance into a narrow band, masking real differences in model capability.

We believe our findings motivate the need for
more challenging datasets, with examples that go
beyond keywords, require reasoning, and come
with clearer, more robust annotation schemes. We
propose recommendations for construction such
datasets at the end of our paper.

We make all cleaned-up datasets, baselines, and
a Python library to run comparisons publicly avail-
able at https://github.com/tcalamai/acl_
climateNLPtoolbox.

## 2 Related Work

**Climate-related Benchmarks.** To the best of our knowledge, only three benchmarks have been proposed for climate-related tasks: **ClimaBench** (Spokoyny et al., 2023) contains the existing datasets climateStance, climatext, climateEng, climateFEVER, SciDCC, and introduces 3 additional datasets, ClimaINS, ClimaQA, and ClimaTOPIC. The authors evaluate multiple fine-tuned transformers and simpler baselines on ClimaBench. **ClimateGPT** (Thulke et al., 2024) is a benchmark that consists of ClimaBench, Pira 2.0 MCQ (Pirozelli et al., 2023) and CC-Contrarian Claims. Several LLMs were evaluated in a few-shot setting. Finally, **ClimRetrieve** (Schimanski et al., 2024b) is a benchmark restricted to climate-related information retrieval. We improve upon these efforts as follows:

1. Our dataset collection is significantly larger than previous ones, with 19 more datasets than ClimaBench and 17 more datasets than ClimateGPT

2. We evaluate simple baselines, fined tuned trans-
formers, and recent LLMs in zero-shot settings,
thereby contextualizing all performances

3. We conduct an in-depth analysis of the errors

**Model Variability and Annotation Errors.** Re-
producibility and significance of results is a major
issue in the scientific literature, especially in the
machine learning community (Bouthillier et al.,
2021; Gundersen et al., 2022; Ruffinelli et al.,
2020). One challenge is that not all papers share the
code, the experimental settings, the datasets created
and the dataset annotation details. This issue has
recently gained a lot of attention in the community,
and conferences guidelines now often explicitly
ask authors for these elements. Another source of
irreproducibility comes from the machine learn-
ing algorithms themselves, and the design choices
that can be made. Gundersen et al. (2022) identi-
fied 41 design choices that can affect reproducibly,

among which: hyperparameter tuning that is costly
and is often done manually (Bouthillier and Varo-
quaux, 2020); dataset issues such as preprocess-
ing, data splits and annotation quality; a lack of
sufficient baselines; and a lack of confidence in-
tervals for results. Annotation quality is another
major issue in the machine learning community,
and solutions such as automatic detection of er-
roneous labels (Klie et al., 2023a) and datasheets
for datasets (Gebru et al., 2021) have been pro-
posed. In this work, we apply these considerations
to climate-related NLP tasks.

3 Tasks, Datasets, and Models

3.1 Tasks

In previous work (Calamai et al., 2025), we have
identified the following climate-related NLP tasks:

**Task 1. Climate-Related Topic Detection:** Given an input sentence or a paragraph, output a binary label, "climate-related" or "not climate-related".

**Task 2. Thematic Analysis:** Given an input sentence or a paragraph, output a subtopic related to climate change. The categories can be the four categories of the Task Force on Climate-related Financial Disclosures (TCFD), the categories of Environment, Social and Governance (ESG), or custom-made categories.

**Task 3. Climate Risk Classification:** Given an input sentence or a paragraph, output the label "opportunity" (if the input talks about a positive effect of climate-change for the company) or "risk" (if the input talks about a negative effect). Some works focus only on risks, classifying them into types of risks (e.g., physical risk, reputational risk, regulatory risk, or transition risk).

**Task 4. Green Claim Detection:** Given an input sentence or a paragraph, output a binary label, "green claim" or "not green claim". Green claims refer to the practice of suggesting or otherwise creating the impression that a product or a service is environmentally friendly (Stammbach et al., 2023).

**Task 5. Green Claim Characteristics:** Given an input sentence or a paragraph labeled as a green claim, output a more fine-grained characterization of the claim. This is a multi-label classification task; the labels can be about the form (e.g. specificity) or the substance (e.g. action, targets, facts).

**Task 6. Green Stance Detection:** Given two input sentences or paragraphs, one labeled as the claim and one as the evidence, predict if the evidence
supports the claim, refutes the claim, or is neutral
towards the claim. Some studies fix the claim (e.g.
the claim is always "Climate change poses a severe
threat") and aim to predict if the evidence supports
or refutes that claim.

**Task 7. Climate-Related Question Answering:**
Given an input question and a set of resources (paragraphs or documents), produce an answer to the question.

**Task 8. Classification of Deceptive Techniques:**
Given a statement, classify it into argumentative categories, such as fallacies, types of arguments, or rhetorical techniques.

## 3.2 Datasets

For each of the tasks, our previous work (Calamai et al., 2025) listed all related datasets. For this work, We collected all those datasets that are openly available to reproduce the original results. Table 1 shows all these datasets with their description from Calamai et al. (2025). These datasets suffered from several issues, and hence we subjected them to a data cleaning pipeline. We provide here an overview of the issues we encountered, and we give the detailed statistics of each cleaning step in Appendix A.

**Duplicate Removal.** We identified many duplicates in the datasets, some of which became visible only after correcting formatting issues such as differing numbers of spaces. We differentiate between exact duplicates (same text and same label) and those with conflicting labels. Exact duplicates were removed to keep only one instance. For duplicates with conflicting labels, when possible we reconstructed the dataset to avoid them (e.g., Spokoyny et al. (2023) provide the source questionnaires for ClimaQA and ClimaINS); otherwise, we removed them. Some duplicates were intrinsic to the task, in which case we did not remove them (e.g. ClimateFEVER evidence contains pairs of claim-evidence; the same evidence could be found for multiple claims).

**Noisy text.** We investigated the text quality of datasets using a gibberish detection model<fnref n="1" /> and a language detection model². We found that some datasets have noisy text samples or non-English

text, but in small proportion (<0.5%). Moreover, noisy text is usually labeled with the negative label. For example, for detecting “climate-related” vs “not climate-related”, the noisy text had the label “not climate-related”. The few positively labeled noisy texts were mostly false positives – they were actually not noisy. Finally, even noisy examples are part of real-world texts. We therefore did not remove noisy samples.

**Input text length.** BERT-like models usually have a limited context window (e.g. 4096 tokens for Longformer). In multiple datasets, we found some excessively long texts (text with more than 4000 tokens), often due to formatting issues (e.g. spaces between all characters). To address this, we applied a text cleaning step to remove formatting and encoding issues using the clean-text<fnref n="3" /> Python library. Some datasets still contained very long texts, so we removed text longer than 4000 tokens to fit within Longformer’s context window. This impacted only 6 datasets and removed less than 0.3% of the data per dataset. We also identified very short texts resulting from parsing errors in PDF documents. These were typically page numbers, escape sequences, or table fragments. To remove these spurious occurrences, we removed all texts with fewer than 5 tokens.

**Dataset size.** After data cleaning, 7 of our 29 datasets still contained a large number of samples, ranging from 500 to over 150K. To reduce computational costs and experiment duration, we limited the training and development splits to 10,000 samples, while keeping **the original size of the test splits intact**. During down-sampling, we ensured that label distribution was preserved through stratification. For heavily imbalanced datasets, we adjusted the sampling to improve the balance.

**Dataset splits.** We split each dataset into 80% train, 10% test and 10% development. We kept the original splits when they were available in these proportions. However, some existing splits exhibited train-test contamination, in which case we re-created the splits.

## 3.3 Models and Measures

**Models.** We evaluated a range of models on each dataset: random baselines, a traditional approach (TF-IDF with logistic regression), fine-tuned transformers (distilRoBERTa, Longformer), and large

<footnote n="1">¹https://huggingface.co/madhunjindal/autonlp-Gibberish-Detector-492513457
²https://huggingface.co/papluca/xlm-roberta-base-language-detection</footnote>

<footnote n="3">³https://github.com/prasanthg3/cleantext</footnote>
<table><caption>Table 1: Description of the datasets we collected.</caption><thead><tr><th>Dataset</th><th>Input</th><th>Labels</th></tr></thead><tbody><tr><td colspan="3"><strong>Climate-Related Topic Detection</strong></td></tr><tr><td>ClimateBug-data, Yu et al. (2024)</td><td>sentences from Banks' reports</td><td>relevant/irrelevant: Climate change and sustainability (including ESG, SDGs related to the environment, recycling and more)</td></tr><tr><td>ClimateBERT's climate detection, Bingler et al. (2023)</td><td>paragraphs from reports</td><td>1/0: Climate policy, climate change or an environmental topic</td></tr><tr><td>Climatext (Wikipedia, 10-K, claims), Varini et al. (2020)</td><td>sentences from Wikipedia, 10-Ks or web scraping</td><td>1/0: Directly related to climate-change</td></tr><tr><td>Climatext (wiki-doc), Varini et al. (2020)</td><td>sentences from a Wikipedia page</td><td>1/0: Extracted from a Wikipedia page related to climate-change</td></tr><tr><td>Sustainable signals's reviews, Lin et al. (2023)</td><td>online product reviews (user comments)</td><td>relevant/irrelevant: Contains terms related to sustainability</td></tr><tr><td colspan="3"><strong>Thematic Analysis</strong></td></tr><tr><td>TCFD rec., Bingler et al. (2021)</td><td>paragraphs from corporate annual reports</td><td>Metrics and Targets, Risk Management, Strategy, Governance and General: TCFD 4 main categories</td></tr><tr><td>ESGBERT's ESG, Schimanski et al. (2023b)</td><td>Sentences from reports and corporate news</td><td>Environment, Social, Governance and None: Environmental criteria comprise a company's energy use, waste management, pollution, as well as compliance with governmental regulations. Special areas of interest are climate change and environmental sustainability.</td></tr><tr><td>ESGBERT's Nature, Schimanski et al. (2024a)</td><td>Paragraphs from reports</td><td>General, Nature, Biodiversity, Forest, Water: Multi-label Nature-related topics</td></tr><tr><td>SciDCC, Mishra and Mittal (2021)</td><td>News articles (Title, Summary, Body)</td><td>Environment, Geology, Animals, Ozone Layer, Climate, etc.: Category in which the article was published (Automatic Label)</td></tr><tr><td>ClimateEng, Vaid et al. (2022)</td><td>Tweets posted during COP25 filtered by keywords (relevant to climate-change)</td><td>Ocean/Water, Politics, Disaster, Agriculture/Forestry, General: Sub-categories of climate-change</td></tr><tr><td>ClimaTOPIC, Spokoyny et al. (2023)</td><td>CDP responses (short texts)</td><td>Adaptation, Buildings, Climate Hazards, Emissions, Water, etc.: Category of the question (Automatic Label)</td></tr><tr><td colspan="3"><strong>Climate Risk Classification</strong></td></tr><tr><td>ClimateBERT's Sentiment, Bingler et al. (2023)</td><td>Paragraphs from companies' annual reports</td><td>Risk, Opportunity, Neutral: Risk or threat that negatively impacts an entity of interest (negative sentiment); or Opportunity arising due to climate change (positive sentiment); Neutral otherwise.</td></tr><tr><td colspan="3"><strong>Green Claim Detection</strong></td></tr><tr><td>Green Claims, Woloszyn et al. (2022)</td><td>Marketing Tweets</td><td>Green Claim/Not Green: Environmental (or green) advertisements refer to all appeals that include ecological, environmental sustainability, or nature-friendly messages that target the needs and desires of environmentally concerned stakeholders.</td></tr><tr><td>Environmental Claims, Stammbach et al. (2023)</td><td>Paragraph from reports</td><td>Yes/No: Environmental claims refer to the practice of suggesting or otherwise creating the impression [...] that a product or a service is environmentally friendly (i.e., it has a positive impact on the environment) or is less damaging to the environment than competing goods or services [...] In our case, claims relate to products, services, or specific corporate environmental performance.</td></tr><tr><td colspan="3"><strong>Green Claim Characteristics</strong></td></tr><tr><td>Implicit/Explicit Green Claims, Woloszyn et al. (2022)</td><td>Marketing Tweets</td><td>Implicit green claims raise the same ecological and environmental concerns as explicit green claims (see definition in Section C.6), but without showing any commitment from the company. If the tweet does not contain a green claim then No Claim.</td></tr><tr><td>Specificity, Bingler et al. (2023)</td><td>Paragraph from reports</td><td>Specific, Non-specific: A paragraph is Specific if it includes clear, tangible, and firm-specific details about events, goals, actions, or explanations that directly impact or clarify the firm's operations, strategy, or objectives. Non-specific otherwise.</td></tr><tr><td>Commitments and Actions, Bingler et al. (2023)</td><td>Paragraph from reports</td><td>Yes/No: A paragraph is a commitment or an action if it contains targets for the future or actions already taken in the past.</td></tr><tr><td>Net Zero/Reduction, Schimanski et al. (2023a)</td><td>Paragraph from Net Zero Tracker, Lang et al. (2023)</td><td>Net-zero, Reduction, None: The paragraph contains either a Net-Zero target, a Reduction target or no target (None)</td></tr><tr><td colspan="3"><strong>Green Stance Detection</strong></td></tr><tr><td>ClimateFEVER (evidence), Diggelmann et al. (2020)</td><td>A claim and an evidence sentence from Wikipedia</td><td>Support, Refutes, Not Enough Information: Determines the relation between a claim and a single evidence sentence</td></tr><tr><td>LobbyMap (Stance), Morio and Manning (2023)</td><td>Page from a company communications (report, press release, ...)</td><td>Strongly supporting, Supporting, No or mixed position, Not supporting, Opposing: Given the policy and the page, classifies the stance</td></tr><tr><td>Global Warming Stance Detection (GWSD), Luo et al. (2020)</td><td>Sentences from news about global warming</td><td>Stance of the evidence (Agree, Disagree, Neutral) toward the claim: Climate-Change is a serious concern.</td></tr><tr><td>ClimateStance, Vaid et al. (2022)</td><td>Tweets posted during COP25 filtered by keywords (relevant to climate-change)</td><td>Stance towards climate change prevention: Favor, Against, Ambiguous. (Stance used as a broad notion including sentiment, evaluation, appraisal, ...)</td></tr><tr><td>ClimateFEVER (claim), Diggelmann et al. (2020)</td><td>A claim and multiple evidence sentences from Wikipedia</td><td>Support, Refutes, Debated, Not Enough Information: Determines if a claim is supported by a set of retrieved evidence sentences</td></tr><tr><td>LobbyMap (Page), Morio and Manning (2023)</td><td>Page from a company communications (report, press release, ...)</td><td>1/0: Contains a stance on a remediation policy</td></tr><tr><td>LobbyMap (Query), Morio and Manning (2023)</td><td>Page from a company communications (report, press release, ...)</td><td>GHG emission regulation, Renewable energy, Carbon tax, ...: Classifies the remediation policy</td></tr><tr><td colspan="3"><strong>Climate-Related Question Answering</strong></td></tr><tr><td>ClimaQA, Spokoyny et al. (2023)</td><td>The text from a response to one of the CDP questions; and one of the questions from the CDP questionnaire</td><td>1: the response answers this question<br>0: The response does not answer this question, but another one</td></tr><tr><td>ClimaNS, Spokoyny et al. (2023)</td><td>The text from a response to one of the questions from the NAIC questionnaire</td><td>MANAGE, RISK PLAN, MITIGATE, ENGAGE, ASSESS, RISKS: The labels correspond to the 8 questions asked in the NAIC questionnaires</td></tr><tr><td colspan="3"><strong>Classification of Deceptive Techniques</strong></td></tr><tr><td>LogicClimate, Jin et al. (2022)</td><td>texts from climatefeedback.org</td><td>Faulty Generalization, Ad Hominem, Ad Populum, False Causality, etc.: Classifies fallacies (Multi-label)</td></tr><tr><td>Contrarian Claims, Coan et al. (2021)</td><td>paragraphs from conservative think tank</td><td>No Claim, Global Warming is not happening, Climate Solutions won't work, Climate impacts are not bad, etc.: Classifies arguments into super/sub-categories of climate science denier's arguments</td></tr></tbody></table>
language models (LLMs) in a zero-shot setting (GPT-4o-mini, Llama 3.1 – 8B and 70B).

**Evaluation.** All models are evaluated on the test datasets. We computed the macro F1-score for each dataset, even if the original papers may have reported other scores such as accuracy or the micro F1-score.

**Statistical Significance.** To compare models, we computed a 95% confidence interval (CI) of the F1-scores based on bootstrapping (Efron and Tibshirani, 1994). This interval measures the variability based on the data sampling. We consider a difference *significant* if the CIs are disjoint.

**Parameters.** Throughout our experiments, we used fixed parameters following Spokoyny et al. (2023)'s parameter choices and seeds to control for randomness (detailed parameters in Appendix B).

# 4 Results and Analysis

## 4.1 Results

Table 2 shows the macro-F1 of all models on all datasets. We also show the most relevant performance score from the original studies, which is not necessarily the macro F1-score and thus not comparable to our results (see details in the table caption). We observe that the finetuned Transformer models generally perform best, with an average F1-score of 74.73% for distilRoBERTa and 74.04% for Longformer. Most datasets (70%) do not contain long text, which limits the benefit of using Longformer compared to distilRoBERTa. Secondly, the zero-shot LLMs performed competitively, with an average F1-score reaching 69.78% for GPT-4o-mini. As expected, the larger Llama 3.1 70B outperformed Llama 3.1 8B, although not by a large margin. Finally, the TF-IDF baseline performed surprisingly well, with an average F1-score of 69.18%, out-performing the Llama models, and coming close to GPT-4o-mini. This indicates that the tasks studied in these datasets can be largely solved using only term-frequency features.

## 4.2 Error Analysis

It is surprising that the top performing LLMs perform worse than the fine-tuned models, despite being significantly more complex and having demonstrated high performance on many other benchmarks (Meta, 2024). We hypothesize that a significant portion of the model's errors might in fact be

issues in the datasets. To test this hypothesis, we analyzed a sample of 503 errors. We investigated the errors of GPT-4o-mini, the best-performing LLM, to minimize the number of genuine errors. However, our findings are relevant for the other LLMs as well, as we found that 70% of the sampled errors are also errors made by the Llama's models, and 70% of Llama's errors are also GPT-4o-mini errors (Table 7 in Appendix C).

We analyzed the discrepancies between the model's prediction and the gold standard to understand both model and dataset limitations. We distinguish mislabeled from debatable instances to align with existing frameworks that distinguish between annotation errors from Human Label Variation (Weber-Genzel et al., 2024).

We sampled between 20 and 30 errors per dataset. Some datasets had fewer than 20 errors, in which case we reviewed them all. Given the input text, the true label, and the prediction of the model, we classified them based in type of error.

**Model errors.** Incorrect predictions made by the zero-shot GPT-4o-mini approach account for 44.6% of the sampled errors – these are clear misclassifications, not cases of ambiguity or errors in the gold standard. They are caused by different factors such as hallucinations, misunderstanding of implicit or indirect relations, anchoring to a specific word in the prompt, not following part of the instructions, or having no identifiable cause. Many of these errors could most likely be fixed by using a larger model, and will not be detailed here (see Appendix C.11 for details).

**Annotation mistakes.** Evaluation datasets can contain annotation errors that bias the performances and the conclusions than can be drawn from experiments (Weber-Genzel et al., 2024; Rücker and Akbik, 2023; Bowman and Dahl, 2021; Klie et al., 2023b). We found obvious annotation errors in 60% of the datasets. This accounts for 16.6% of the sampled errors.

**Debatable Errors.** Finally, we find debatable examples in almost all datasets (96%). We identified three main causes: statements that are out-of-context; statements that are ambiguous or implicit; and definitions of labels that are not exhaustive or detailed enough, leaving room for interpretation. 38.8% of the sampled errors are debatable.
<table><caption>Table 2: Macro F1 score of baselines and methods on each dataset. The bootstrap confidence interval is displayed as ± F1<sub>95%</sub> - F1<sub>5%</sub>. For each Reference value, we add a clarifying number if the metric is not macro F1: 1. binary F1-score, 2. average F1-score (macro/micro is not specified), 3. weighted average F1-score. In addition, we specify the method in the reference via letters: a. SVM, c. climateBERT, d. Contaminated split, SciBERT, e. ESGBERT, f. climateBUG-LM, g. EnvironmentalBERT, h. Human Baseline using annotations, i. Filtered Split (by removing disputed claims), j. RoBERTa + Logistic Regression, k. Electra, l. Longformer, r. RoBERTa, t. BERT. ▲: weakly labeled datasets, ▲: small datasets (<1000 in training dataset), ✓: larger, human-annotated datasets. In red: performance not significantly higher than random; In gray: performance not significantly higher than TF-IDF baseline; In green: fine-tuned models performing significantly better than all zero-shot approaches (and inversely). A performance is considered significantly higher if 0 is not included in the bootstrap 95% confidence interval of the difference of F1-score.</caption><thead><tr><th>Dataset</th><th>Random</th><th>TF-IDF</th><th>Longformer</th><th>DistilRoBERTaGPT-4o-mini</th><th>Llama 8B</th><th>Llama 70B</th><th>Reference</th></tr></thead><tbody><tr><td colspan="8"><strong>Climate Topic Classification</strong></td></tr><tr><td>✓ climateBUG-data</td><td>49.6±0.6</td><td>86.4±0.4</td><td>90.5±0.4</td><td>90.5±0.3</td><td>89.2±0.4</td><td>79.0±0.5</td><td>88.3±0.3</td><td>91.36<sup>f</sup></td></tr><tr><td>✓ ClimateBERT Cl. det.</td><td>42.8±4.9</td><td>79.3±4.8</td><td>95.8±2.4</td><td>94.0±3.0</td><td>93.2±3.1</td><td>88.3±3.8</td><td>93.4±3.2</td><td>99.1<sup>3, c</sup></td></tr><tr><td>▲ climatext (Wiki-doc)</td><td>49.3±1.5</td><td>80.4±1.2</td><td>85.6±1.1</td><td>83.8±1.2</td><td>84.0±1.2</td><td>86.3±1.1</td><td>89.8±1.0</td><td>-</td></tr><tr><td>✓ climatext (10k)</td><td>46.9±5.4</td><td>91.0±4.0</td><td>97.0±2.4</td><td>96.5±2.6</td><td>95.4±3.2</td><td>90.9±4.2</td><td>96.0±2.8</td><td>95±2<sup>1, t</sup></td></tr><tr><td>✓ climatext (claim)</td><td>48.2±3.1</td><td>74.7±2.8</td><td>67.8±3.2</td><td>75.4±2.9</td><td>82.2±2.4</td><td>83.6±2.2</td><td>85.7±2.0</td><td>83±1<sup>1, t</sup></td></tr><tr><td>✓ climatext (Wikipedia)</td><td>40.1±4.8</td><td>83.5±7.7</td><td>83.9±7.2</td><td>88.2±6.5</td><td>84.7±7.2</td><td>82.8±7.6</td><td>87.4±6.5</td><td>80±6<sup>1, t</sup></td></tr><tr><td>▲ Sustainable Signals rev.</td><td>37.6±11.0</td><td>65.8±11.1</td><td>73.2±10.0</td><td>73.7±9.9</td><td>69.7±11.0</td><td>63.5±10.7</td><td>73.3±9.9</td><td>-</td></tr><tr><td colspan="8"><strong>Climate Sub-thematic topic classification</strong></td></tr><tr><td>▲ ClimaTOPIC</td><td>6.0±0.5</td><td>46.6±1.2</td><td>55.8±1.4</td><td>54.8±1.3</td><td>35.8±1.0</td><td>34.5±1.0</td><td>30.5±1.0</td><td>65.22<sup>r</sup></td></tr><tr><td>✓ climateEng</td><td>13.2±2.9</td><td>58.4±9.3</td><td>70.5±8.5</td><td>67.3±8.6</td><td>65.8±9.2</td><td>53.9±9.4</td><td>61.1±8.5</td><td>74.58<sup>r</sup></td></tr><tr><td>✓ esgbert Biodiversity</td><td>41.1±6.4</td><td>91.3±5.5</td><td>91.3±5.1</td><td>89.0±5.5</td><td>91.8±4.9</td><td>88.8±5.6</td><td>82.3±5.9</td><td>92.29<sup>1, g</sup></td></tr><tr><td>✓ esgbert Forest</td><td>41.4±5.9</td><td>92.9±4.8</td><td>97.2±2.8</td><td>97.0±2.8</td><td>91.6±4.9</td><td>87.4±5.6</td><td>86.2±5.8</td><td>95.37<sup>1, g</sup></td></tr><tr><td>✓ esgbert Nature</td><td>51.5±6.5</td><td>82.9±5.0</td><td>89.8±3.9</td><td>89.9±4.0</td><td>82.9±5.2</td><td>86.7±4.5</td><td>88.0±4.6</td><td>94.19<sup>1, g</sup></td></tr><tr><td>✓ esgbert Water</td><td>42.7±6.4</td><td>84.6±6.6</td><td>95.3±3.4</td><td>93.9±3.9</td><td>93.8±4.1</td><td>89.4±5.3</td><td>93.5±3.9</td><td>95.10<sup>1, c</sup></td></tr><tr><td>▲ sciDCC</td><td>3.7±1.0</td><td>42.6±3.9</td><td>39.8±2.8</td><td>42.1±3.3</td><td>29.2±2.3</td><td>20.5±2.2</td><td>30.0±2.4</td><td>54.79<sup>l</sup></td></tr><tr><td>✓ Climate TCFD rec.</td><td>16.5±3.7</td><td>56.5±5.6</td><td>69.4±5.2</td><td>68.5±5.2</td><td>47.5±5.4</td><td>43.2±5.2</td><td>50.2±5.4</td><td>-</td></tr><tr><td>✓ ESGBERT E</td><td>42.6±7.1</td><td>88.1±5.1</td><td>95.8±3.1</td><td>95.8±3.2</td><td>95.0±3.5</td><td>95.1±3.5</td><td>94.6±3.4</td><td>93.19<sup>1, e</sup></td></tr><tr><td>✓ ESGBERT G</td><td>53.5±7.1</td><td>80.2±6.3</td><td>78.2±6.4</td><td>83.7±6.4</td><td>83.5±5.7</td><td>75.6±6.1</td><td>62.6±6.9</td><td>78.86<sup>1, e</sup></td></tr><tr><td>✓ ESGBERT S</td><td>54.4±6.9</td><td>82.6±5.6</td><td>88.3±4.4</td><td>89.6±4.2</td><td>78.1±5.8</td><td>77.9±6.0</td><td>73.6±6.2</td><td>91.90<sup>1, e</sup></td></tr><tr><td colspan="8"><strong>Sentiment Analysis</strong></td></tr><tr><td>✓ climate sentiment</td><td>35.9±5.2</td><td>69.0±5.5</td><td>79.9±4.8</td><td>77.7±4.8</td><td>77.3±4.9</td><td>71.4±5.3</td><td>70.5±5.4</td><td>83.8<sup>3, c</sup></td></tr><tr><td colspan="8"><strong>Claim detection</strong></td></tr><tr><td>✓ Environmental Claims</td><td>42.8±5.5</td><td>80.5±5.4</td><td>90.5±3.8</td><td>91.2±3.8</td><td>86.8±4.5</td><td>81.4±5.0</td><td>76.6±5.4</td><td>84.9<sup>1, r</sup></td></tr><tr><td>▲ Green Claims</td><td>46.1±10.9</td><td>86.1±8.0</td><td>94.5±5.0</td><td>97.2±3.5</td><td>91.5±6.8</td><td>90.0±7.3</td><td>86.8±7.3</td><td>92.08<sup>1, r</sup></td></tr><tr><td colspan="8"><strong>Claim characteristics</strong></td></tr><tr><td>✓ Commitments&Actions</td><td>48.5±5.5</td><td>72.7±5.4</td><td>76.7±5.1</td><td>81.9±4.7</td><td>67.2±5.4</td><td>64.6±5.2</td><td>50.7±5.6</td><td>81<sup>3, c</sup></td></tr><tr><td>✓ Climate Specificity</td><td>48.9±5.3</td><td>72.4±5.0</td><td>77.3±4.8</td><td>77.5±5.1</td><td>71.6±4.9</td><td>76.5±4.6</td><td>68.0±5.2</td><td>77<sup>3, c</sup></td></tr><tr><td>▲ ESGBERT action500</td><td>44.4±14.3</td><td>82.5±11.0</td><td>85.9±10.2</td><td>89.1±8.8</td><td>76.0±12.6</td><td>67.0±13.2</td><td>63.0±14.2</td><td>-</td></tr><tr><td>▲ Implicit/Explicit Claims</td><td>37.2±9.9</td><td>70.3±12.8</td><td>63.0±13.1</td><td>81.2±11.9</td><td>81.6±9.6</td><td>68.5±13.2</td><td>75.7±11.3</td><td>81.45<sup>2, r</sup></td></tr><tr><td>✓ Net-Zero/Reduction</td><td>29.8±4.9</td><td>94.8±2.4</td><td>97.8±1.5</td><td>97.7±1.8</td><td>97.3±1.8</td><td>93.7±2.8</td><td>95.6±2.2</td><td>98.7<sup>1, c</sup></td></tr><tr><td colspan="8"><strong>Stance classification</strong></td></tr><tr><td>✓ climateF. claim (our split)</td><td>15.4±5.1</td><td>35.0±8.3</td><td>31.9±7.1</td><td>32.5±7.4</td><td>19.3±5.9</td><td>30.0±7.7</td><td>26.3±6.5</td><td>80.7<sup>i, t</sup></td></tr><tr><td>✓ climateF. claim (agg.)</td><td>-</td><td>-</td><td>42.5±7.8</td><td>41.3±8.6</td><td>50.4±8.1</td><td>-</td><td>-</td><td>60.10±8.6<sup>h</sup></td></tr><tr><td>✓ climateFEVER evidence</td><td>28.4±3.1</td><td>46.2±4.0</td><td>52.7±4.1</td><td>51.3±3.9</td><td>60.0±3.7</td><td>52.5±3.7</td><td>63.7±3.9</td><td>68.03±12.9<sup>h</sup></td></tr><tr><td>✓ climateStance</td><td>21.3±3.7</td><td>49.6±6.5</td><td>56.6±6.8</td><td>56.1±7.7</td><td>56.4±5.9</td><td>48.5±5.9</td><td>58.0±6.3</td><td>59.69<sup>r</sup></td></tr><tr><td>✓ Global-Warming Stance</td><td>29.5±6.5</td><td>59.2±7.4</td><td>69.3±6.8</td><td>75.3±6.5</td><td>69.8±6.6</td><td>64.4±6.8</td><td>72.5±6.5</td><td>73<sup>t</sup></td></tr><tr><td>▲ LobbyMap (Pages)</td><td>46.3±0.8</td><td>73.1±0.8</td><td>71.3±0.8</td><td>73.6±0.8</td><td>63.2±0.8</td><td>62.5±0.8</td><td>60.5±0.8</td><td>-</td></tr><tr><td>▲ LobbyMap (Query)</td><td>12.4±0.5</td><td>49.3±1.8</td><td>57.3±2.1</td><td>52.1±2.1</td><td>36.4±1.8</td><td>27.6±1.6</td><td>32.7±1.6</td><td>-</td></tr><tr><td>▲ LobbyMap (Stance)</td><td>19.2±1.3</td><td>43.6±1.8</td><td>46.7±1.6</td><td>44.7±1.9</td><td>30.0±1.6</td><td>26.9±1.5</td><td>24.1±1.5</td><td>-</td></tr><tr><td colspan="8"><strong>Question answering</strong></td></tr><tr><td>▲ ClimalNS (our split)</td><td>12.8±1.8</td><td>81.2±2.1</td><td>77.6±4.2</td><td>75.8±4.5</td><td>58.5±2.6</td><td>45.4±2.5</td><td>48.4±2.6</td><td>-</td></tr><tr><td>▲ climaQA (our split)</td><td>50.5±1.1</td><td>50.3±1.0</td><td>89.5±0.6</td><td>89.4±0.6</td><td>78.8±0.8</td><td>57.4±1.1</td><td>76.1±1.0</td><td>-</td></tr><tr><td colspan="8"><strong>Deceptive techniques</strong></td></tr><tr><td>✓ CC-Contrarian Claims</td><td>3.2±0.7</td><td>62.7±2.5</td><td>71.6±2.7</td><td>71.2±2.5</td><td>58.8±2.9</td><td>40.2±3.1</td><td>55.8±2.9</td><td>79<sup>j</sup></td></tr><tr><td>▲ logicClimate</td><td>13.3±2.0</td><td>13.3±5.0</td><td>15.6±5.6</td><td>9.4±3.2</td><td>27.4±7.8</td><td>10.3±4.3</td><td>22.1±6.4</td><td>29.37<sup>k</sup></td></tr><tr><td><strong>Average</strong></td><td>34.24</td><td>69.18</td><td>74.04</td><td>74.73</td><td>69.78</td><td>65.03</td><td>67.39</td><td>-</td></tr></tbody></table>
## 4.3 Dataset Issues

In the following, we discuss possible causes of the annotation mistakes we observed in our study (This section contains a condensed analysis of errors, we detail error per dataset in Appendix C).

**Ambiguous annotations guidelines.** For each of the datasets, we designed the prompts based on the annotation guidelines. However, we observed that many prediction errors from GPT-4o-mini were caused by ambiguities in the guidelines:

(1) In ClimateBUG, the guideline says that “Sustainability not related to the environment (i.e. sustainable profits)” should be classified as “Not Climate”. However, some sentences do not specify whether the sustainability is related to the environment or not (e.g. “They are the source of jobs, innovation, sustainability, and prosperity”).

(2) In ESGBERT Nature, it is not clear if the “water” label, which focuses on “water management, consumption, and pollution”, includes water-related natural disasters such as tsunamis.

(3) In climate sentiment, it is not clear if corporate ambitions should be classified as “Opportunities” as they “associate specific positive objectives to the anticipated, past, or present developments and topics covered” (e.g. “TD recently launched a bold and ambitious climate action plan to address the challenges of climate change. This includes a target to achieve net-zero greenhouse gas emissions in our operations and financing activities by 2050”) or as a neutral statement as aligning with climate objectives will not necessary bring growth to a company.

(4) Environmental Claims does not provide instructions on implicit claims.

(5) Green Claims is not specific on how to annotate claims suggesting that a product is better, healthier, or has good properties thanks to a natural ingredient.

(6) In Commitments&Actions, it is not clear how to annotate descriptions of company values, governance structures, or descriptions of existing processes.

(7) Climate Specificity does not describe how to handle the granularity of the specificity (e.g. “For our sustainable strategy range, we incorporate a series of proprietary ‘red lines’ in

order to ensure the poorest-performing companies from an ESG perspective are not eligible for investment”).

(8) In ClimateFEVER evidence, it is not clear if “SUPPORTING” means that the evidence is sufficient to entail the claim, or if it means that the evidence is merely in line with the claim.

(9) In ClimateStance, it is not clear how to annotate a tweet that is in “opposition to climate change policies”, but simultaneously acknowledges the urgency of climate change.

(10) In CC-Contrarian Claims, some labels are difficult to differentiate such as “Weather is cold/snowing” and “Ice/permafrost/snow cover isn’t melting”.

(11) In LogicClimate, the “intentional” label is broad and encompasses all other fallacy types. These issues impact many datasets and a large part of the sampled errors of GPT-4o-mini (4/9 on ESGBERT Water, 5/20 on ClimateBUG, 5/10 on climate sentiment, 5/14 on Environmental Claims, 5/6 on Woloszyn et al., 2022), 9/20 Commitments and Actions, 4/20 on specificity, 20/20 additional samples<fnref n="4" /> ClimateFEVER evidence, 2/10 in ClimateStance, 2/10 in CC-Contrarian Claims, 3/10 in LogicClimate).

**Implicitness.** Many errors were due to the implicit nature of the text. This represents a large portion of the sampled errors in multiple datasets (4/17 on Climate detection, 3/10 on climate TCFD recommendations, 2/8 on ESGBERT, 3/14 on Environmental claims, 2/10 on ClimateStance, 7/10 on GWSD). Interestingly, we even found examples where the model understood the indirect link but not the annotators (2/10 on climate sentiment). A notable implied reference to climate change, which the model struggles on, is categorizing statements about energy as climate/environment-related (4/17 Climate detection, 2/8 on ESGBERT, 3/14 on ClimateText (wiki)). These cases are prediction errors by GPT-4o-mini; however, due to the implicit nature of the statements, the labels are debatable.

**Multi-label.** In many different contexts, multiple labels might be valid (Weber-Genzel et al., 2024). This is what we observed in 26% of the single-label datasets, and 8.75% of the sampled errors. For example, “Reduce GHG emissions [...] by 55%

<footnote n="4">⁴We randomly sampled 10 errors of GPT-4o-mini predicting “REFUTES” instead of “NOT ENOUGH INFO” and 10 errors of it predicting “NOT ENOUGH INFO” instead of “SUPPORTS”.</footnote>
by FY2030 and to zero by FY2050 [...]” contains
both a reduction target and a net-zero target. We
believe this issue arises from trying to simplify the
annotation process and build a single-label dataset
in settings that are intrinsically multi-label.

**Weak labels.** Zero-shot LLMs performed signif-
icantly worse than finetuned models on weakly
labeled datasets than were not annotated through a
controlled annotation procedure. The weak labels
are useful to create large datasets without requiring
human annotations; however, they lack the rigor-
ous definitions that manual labels provide, which
are derived from strict annotation guidelines. This
automatic generation can introduce inconsistencies
such as evolving label definitions over time or unex-
pected responses in survey data. This makes these
labels noisy and less reliable for classification tasks.
This explains the lower performances on SciDCC,
ClimaINS, ClimaTOPIC and ClimaQA. Addition-
ally, since the labels do not follow strict guidelines,
designing a prompt for these tasks requires guess-
ing the actual label meaning, which might be differ-
ent from the label intended purpose. For example,
the Climatext task is to identify climate-related
sentences. However, to create the datasets anno-
tated by humans, Varini et al. (2020) relied on a
weakly labeled dataset to filter potentially climate-
related sentences. Therefore, the intended purpose
is climate-relatedness, but the actual meaning of the
label is “a sentence extracted from a climate-related
Wikipedia page” (which is the prompt used to reach
F1-scores above 80%). Despite these non-intuitive
definitions, the labels are actually linguistically dis-
tinguishable, as shown by the performances of the
fine-tuned models and the TF-IDF baselines.

**Exhaustivity.** This is a particular case of issues
arising from weakly labeled datasets. Because they
do not rely on rigorous annotation guidelines in-
tended for classification, the annotations are not
necessarily exhaustive. The LobbyMap dataset,
e.g., is constructed using data from LobbyMap.org,
a website that identifies companies' stances on cli-
mate change mitigation policies and quotes compa-
nies' documents with evidence about their stance.
Therefore, for each policy, they identify evidence,
but they do not necessarily collect all the evidence
about that stance. Inversely, the model is asked to
identify, given a document, what is the list of policy
stances mentioned. GPT-4o-mini tends to predict
more stances per document than the original labels.
Moreover, in the sampled errors, we found that the

predictions were mostly reasonable. This suggests
that the annotations might not be exhaustive.

4.4 Discussion

The weakly labeled datasets are interesting as
they build a foundation for relevant tasks: e.g.
LobbyMap for identifying stance on climate-
remediation policies, SciDCC to classify news, or,
ClimaQA and ClimaINS-from **Task 7**- and Cli-
maTOPIC to structure company documents; how-
ever, the poor performance and the error analysis
reveal that the datasets need further annotations.

**Task 1 to 5** focus on topic classification, risk
and green claim. When excluding weakly labeled
datasets, performances are high for fine-tuned mod-
els, but also for TF-IDF, showing that these tasks
are highly based on vocabulary. Some datasets re-
main challenging: TCFD recommendations, Clima-
teEng and, to a lesser extent, climate sentiment and
specificity. However, improving the guidelines to
resolve ambiguous cases would improve label con-
sistency, and most likely performances, particularly
for LLMs which rely more heavily on guidelines,
while finetuned models rely on the annotations.

**Task 6** focuses on stance detection. Models
demonstrate that they have predictive power, but
the performance is relatively low (F1-score un-
der 75%). However, the task is inherently hard,
as shown by the low performance of humans on
ClimateFEVER, or moderate IAA (Krippendorf's
α =0.54-0.64) reported by Luo et al. (2020).

**Task 8** focuses on deceptive techniques. On
CC-contrarian claims, given the high number of
classes, the performances are high, even for TF-
IDF. For fallacy detection, the task is inherently
subjective (Helwe et al., 2024). This is particularly
the case when using real-world examples where
multiple types of fallacies could fit.

4.5 Recommendations

As shown through our error analysis, many false
predictions of a zero-shot model are actually mis-
labeled or debatable examples. Such examples
are problematic for two reasons: First, the perfor-
mance differences between models are often mini-
mal (sometimes less than 1%), making it difficult
to draw meaningful comparisons when the datasets
are noisy. Second, it becomes unclear whether
observed performance gains stem from genuine
model improvements or from overfitting to annota-
tion biases and errors. Fine-tuned models may per-
form well on these datasets, but that does not mean
that the datasets are clean: The models may pick
up patterns in the data that have been informally
discussed between the annotators, but that are not
codified in the guidelines. To address these issues
and improve the reliability of model evaluation, we
propose the following set of recommendations.

**Use Precise and Exhaustive Annotation Guidelines.** Ambiguous or under-specified guidelines result in inconsistent annotations, reducing dataset reliability and skewing model evaluation. To mitigate this, guidelines should be:

(1) Precise: Clearly define all terms and concepts.

(2) Exhaustive: Address edge cases and ambiguities.

(3) Unambiguous: Avoid vague phrasing and provide concrete examples.

Additionally, we recommend not using automati-
cally labeled datasets, as they do not follow well-
defined annotation procedures and can easily pro-
duce ill-defined tasks.

**Design Datasets with Ambiguous or Implicit Statements.** Many current datasets are too easily solved with keyword-based heuristics (e.g., detecting terms like “GHG” or “climate change”). However, LLMs struggle with more implicit or ambiguous cases. To assess true model understanding, future datasets should focus on examples that require inference and contextual reasoning.

**Include Simple Baselines to Assess Task Difficulty.** We observed that many tasks can be solved with simple models such as word frequency or TF-IDF. Including these baselines is essential to gauge the added value of advanced models. Ideally, a subset of the dataset should be challenging for such baselines, indicating that genuine language understanding is required.

**Quantify Annotation Error Rates and Their Impact.** When model performance exceeds 90%, even a small number of annotation errors can meaningfully distort evaluation. We recommend estimating the annotation error rate through manual review or IAA agreement analysis. This allows researchers to define a performance margin below which observed differences may fall within the noise of the dataset, and thus not reflect genuine model improvement. Such quantification is essential for drawing reliable conclusions from small performance gains. This also highlights the importance of computing meaningful confidence intervals measuring uncertainty of performances due to factors such as dataset sampling or initialization.

**Prioritize Dataset Quality Over Quantity.** As

model accuracy rises, even small annotation er-
rors can significantly impact evaluation. We rec-
ommend focusing on clean, high-quality test sets
rather than large weakly labeled datasets, which of-
ten lack robust guidelines. Poor LLM performance
on such datasets may reflect structural flaws rather
than model limitations.

5 Conclusion

In this work, we curated, cleaned, and standardized
29 climate-related NLP datasets, ensuring consis-
tency across tasks. We systematically evaluated a
diverse range of models. Our results indicate that
all approaches achieved competitive performance.
However, since even the TF-IDF baseline performs
well, the datasets may overall be too simple. Our
analysis also revealed that nearly all datasets con-
tain annotation inconsistencies, which may intro-
duce noise in model evaluations. These findings
highlight the importance of dataset quality in bench-
marking and call for more rigorous annotation pro-
tocols in climate-related NLP research.

All our data, scripts, and models are publicly
available at at https://github.com/tcalamai/
acl_climateNLPtoolbox. In particular, we pro-
vide a Python library that can run all models on
entire files by parsing the PDFs, splitting them
into paragraphs, doing model inference on each
paragraph (individually or as an ensemble), and
aggregating the results into statistics (example in
Appendix D).

**Acknowledgement.** This work was performed using HPC resources from GENCI-IDRIS (Grant AD011014244R1). This work is supported by Amundi Technology and the ANRT with a CIFRE fellowship.

6 Limitations

Our study has several limitations. First, it relies
on publicly available datasets, which may not fully
represent the entire scientific corpus of datasets
related to climate-related tasks.

Second, while we accounted for variability in
training through uncertainty estimation, our results
remain influenced by factors such as hyperparame-
ter selection. These factors can significantly impact
model performance, and while our findings align
with those of previous studies, further optimization
could improve performance.

Third, we designed our prompts to maximize
model performance under a zero-shot setting with
a small chain-of-thought (CoT) component. However, alternative prompting strategies, such as few-shot learning, self-reflection, and iterative reasoning, could enhance LLM performance. Additionally, our study was limited to smaller language models, and larger models, such as LLaMA 3 405B, could yield improved results.

Regarding annotation, error analysis was conducted by the authors of this study. To facilitate interpretability, we exposed both the gold label and the model's prediction during the annotation process, allowing us to analyze the model's reasoning. Despite this, we believe our error analysis provides meaningful insights into model behavior. Moreover, we computed the IAA on a subset of the datasets for 2 annotators; we found a Cohen's κ 0.395 for the 3 label classification: “actual error”, “debatable”, “mislabeled”. This is a weak agreement, making the exact figures mentioned in this paper approximations. However, the agreement is better on classifying errors as “actual errors” or not (κ = 0.591). Moreover, the disagreement between the annotators was mostly with the “debatable” label. This further confirms that the wrong predictions partly stem from HLV.

Finally, we acknowledge a broader ethical consideration: developing tools that analyze how entities communicate about climate-related issues may inadvertently enable strategic adaptation by these entities. Specifically, organizations could leverage such tools either to evade detection or to optimize their messaging to increase perceived compliance. This risk underscores the importance of continuous evaluation and responsible deployment of NLP systems in this domain.

## References

Alix Auzepy, Elena Tönjes, David Lenz, and Christoph Funk. 2023. Evaluating tcfd reporting—a new application of zero-shot analysis to climate-related financial disclosures. *PLOS ONE*, 18(11):e1–23.

Iz Beltagy, Matthew E. Peters, and Arman Cohan. 2020. Longformer: The long-document transformer. *arXiv:2004.05150*.

Julia Bingler, Mathias Kraus, and Markus Leippold. 2021. Cheap Talk and Cherry-Picking: What ClimateBert has to say on Corporate Climate Risk Disclosures. *Social Science Research Network*.

Julia Bingler, Mathias Kraus, Markus Leippold, and Nicolas Webersinke. 2023. How cheap talk in climate disclosures relates to climate initiatives, corpo-

rate emissions, and reputation risk. Working paper, Available at SSRN 4000708.

Xavier Bouthillier, Pierre Delaunay, Mirko Bronzi, As-sya Trofimov, Brennan Nichyporuk, Justin Szeto, Nazanin Mohammadi Sepahvand, Edward Raff, Kanika Madan, Vikram Voleti, et al. 2021. Accounting for variance in machine learning benchmarks. *Proceedings of Machine Learning and Systems*, 3:747–769.

Xavier Bouthillier and Gaël Varoquaux. 2020. *Survey of machine-learning experimental methods at NeurIPS2019 and ICLR2020*. Ph.D. thesis, Inria Saclay Ile de France.

Samuel R. Bowman and George Dahl. 2021. What will it take to fix benchmarking in natural language understanding? In *Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies*, pages 4843–4855, Online. Association for Computational Linguistics.

Tom Calamai, Oana Balalau, Théo Le Guenedal, and Fabian M. Suchanek. 2025. Corporate green-washing detection in text - a survey. *Preprint*, arXiv:2502.07541.

Travis G. Coan, Constantine Boussalis, John Cook, and Mirjam O. Nanko. 2021. Computer-assisted classification of contrarian claims about climate change. *Scientific Reports*, 11(1):22320.

Thomas Diggelmann, Jordan Boyd-Graber, Jannis Bulian, Jannis Bulian, Massimiliano Ciaramita, and Markus Leippold. 2020. CLIMATE-FEVER: A Dataset for Verification of Real-World Climate Claims. *Tackling Climate Change with Machine Learning workshop at NeurIPS 2020, Online, 11 December 2020 - 11 December 2020*.

Bradley Efron and Robert J Tibshirani. 1994. *An introduction to the bootstrap*. Chapman and Hall/CRC.

David Friederich, Lynn H. Kaack, Alexandra Luccioni, and Bjarne Steffen. 2021. Automated Identification of Climate Risk Disclosures in Annual Corporate Reports. Papers 2108.01415, arXiv.org.

Eduardo C. Garrido-Merchán, Cristina González-Barthe, and María Coronado Vaca. 2023. Fine-tuning climatebert transformer with climatext for the disclosure analysis of climate-related financial risks. *Preprint*, arXiv:2303.13373.

Timnit Gebru, Jamie Morgenstern, Briana Vecchione, Jennifer Wortman Vaughan, Hanna Wallach, Hal Daumé Iii, and Kate Crawford. 2021. Datasheets for datasets. *Communications of the ACM*, 64(12):86–92.

Odd Erik Gundersen, Kevin Coakley, Christine Kirkpatrick, and Yolanda Gil. 2022. Sources of irreproducibility in machine learning: A review. *arXiv preprint arXiv:2204.07610*.
Chadi Helwe, Tom Calamai, Pierre-Henri Paris, Chloé Clavel, and Fabian Suchanek. 2024. MAFALDA: A benchmark and comprehensive study of fallacy detection and classification. In *Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers)*, pages 4810–4845, Mexico City, Mexico. Association for Computational Linguistics.

Zhijing Jin, Abhinav Lalwani, Tejas Vaidhya, Xiaoyu Shen, Yiwen Ding, Zhiheng Lyu, Mrinmaya Sachan, Rada Mihalcea, and Bernhard Schoelkopf. 2022. Logical fallacy detection. In *Findings of the Association for Computational Linguistics: EMNLP 2022*, pages 7180–7198, Abu Dhabi, United Arab Emirates. Association for Computational Linguistics.

Jan-Christoph Klie, Bonnie Webber, and Iryna Gurevych. 2023a. Annotation error detection: Analyzing the past and present for a more coherent future. *Computational Linguistics*, 49(1):157–198.

Jan-Christoph Klie, Bonnie Webber, and Iryna Gurevych. 2023b. Annotation Error Detection: Analyzing the Past and Present for a More Coherent Future. *Computational Linguistics*, 49(1):157–198.

John Lang, Camilla Hyslop, Natasha Lutz, Natalie Short, Richard Black, Peter Chalkley, Thomas Hale, Frederic Hans, Nick Hay, Niklas Höhne, Angel Hsu, Takeshi Kuramochi, Silke Mooldijk, and Steve Smith. 2023. Net zero tracker. Energy and Climate Intelligence Unit, Data-Driven EnviroLab, NewClimate Institute, Oxford Net Zero.

Tong Lin, Tianliang Xu, Amit Zac, and Sabina Tomkins. 2023. Sustainablesignals: An ai approach for inferring consumer product sustainability. In *Proceedings of the Thirty-Second International Joint Conference on Artificial Intelligence, IJCAI-23*, pages 6067–6075. International Joint Conferences on Artificial Intelligence Organization. AI for Good.

Yiwei Luo, Dallas Card, and Dan Jurafsky. 2020. Detecting Stance in Media On Global Warming. In *Findings of the Association for Computational Linguistics: EMNLP 2020*, pages 3296–3315, Online. Association for Computational Linguistics.

Meta. 2024. Introducing llama 3.1: Our most capable models to date.

Prakamya Mishra and Rohan Mittal. 2021. Neuralnere: Neural named entity relationship extraction for end-to-end climate change knowledge graph construction. In *ICML 2021 Workshop on Tackling Climate Change with Machine Learning*.

Gaku Morio and Christopher D Manning. 2023. An NLP benchmark dataset for assessing corporate climate policy engagement. In *Thirty-seventh Conference on Neural Information Processing Systems Datasets and Benchmarks Track*.

Gabrijela Perković, Antun Drobnjak, and Ivica Botički. 2024. Hallucinations in llms: Understanding and addressing challenges. In *2024 47th MIPRO ICT and Electronics Convention (MIPRO)*, pages 2084–2088.

Paulo Pirozelli, Marcos M. José, Igor Silveira, Flávio Nakasato, Sarajane M. Peres, Anarosa A. F. Brandão, Anna H. R. Costa, and Fabio G. Cozman. 2023. Benchmarks for pirá 2.0, a reading comprehension dataset about the ocean, the brazilian coast, and climate change. *Preprint*, arXiv:2309.10945. https://github.com/C4AI/Pira.

Susanna Rücker and Alan Akbik. 2023. CleanCoNLL: A nearly noise-free named entity recognition dataset. In *Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing*, pages 8628–8645, Singapore. Association for Computational Linguistics.

Daniel Ruffinelli, Samuel Broscheit, and Rainer Gemulla. 2020. You can teach an old dog new tricks! on training knowledge graph embeddings.

Rylen Sampson, Aysha Cotterill, and Quoc Tien Au. 2022. Tcfd-nlp: Assessing alignment of climate disclosures using nlp for the financial markets. In *NeurIPS 2022 Workshop on Tackling Climate Change with Machine Learning*.

Victor Sanh, Lysandre Debut, Julien Chaumond, and Thomas Wolf. 2019. Distilbert, a distilled version of bert: smaller, faster, cheaper and lighter. *ArXiv*, abs/1910.01108.

Tobias Schimanski, Julia Bingler, Camilla Hyslop, Mathias Kraus, and Markus Leippold. 2023a. Climatebert-netzero: Detecting and assessing net zero and reduction target. *Swiss Finance Institute Research Paper*, (23-110).

Tobias Schimanski, Chiara Colesanti Senni, Glen Gostlow, Jingwei Ni, Tingyu Yu, and Markus Leippold. 2024a. Exploring nature: Datasets and models for analyzing nature-related disclosures. *SSRN Electronic Journal*.

Tobias Schimanski, Jingwei Ni, Roberto Spacey Martín, Nicola Ranger, and Markus Leippold. 2024b. ClimRetrieve: A benchmarking dataset for information retrieval from corporate climate disclosures. In *Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing*, pages 17509–17524, Miami, Florida, USA. Association for Computational Linguistics.

Tobias Schimanski, Andrin Reding, Nico Reding, Julia Bingler, Mathias Kraus, and Markus Leippold. 2023b. Bridging the Gap in ESG Measurement: Using NLP to Quantify Environmental, Social, and Governance Communication.

Freda Shi, Xinyun Chen, Kanishka Misra, Nathan Scales, David Dohan, Ed Chi, Nathanael Schärli, and Denny Zhou. 2023. Large language models can be easily distracted by irrelevant context. *Preprint*, arXiv:2302.00093.
Daniel Spokoyny, Tanmay Laud, Tom Corringham, and Taylor Berg-Kirkpatrick. 2023. Towards answering climate questionnaires from unstructured climate reports. *Preprint*, arXiv:2301.04253.

Dominik Stammbach, Nicolas Webersinke, Julia Bin-
gler, Mathias Kraus, and Markus Leippold. 2023. En-
vironmental claim detection. In *Proceedings of the
61st Annual Meeting of the Association for Compu-
tational Linguistics (Volume 2: Short Papers)*, pages
1051–1066, Toronto, Canada. Association for Com-
putational Linguistics.

The World Meteorological Organization (WMO). Wmo confirms 2024 as warmest year on record at about 1.55°c above pre-industrial level. https://wmo.int/news/media-centre/wmo-confirms-2024-warmest-year-record-about-155degc-above-pre-industrial-level. [Online; accessed 03-February-2025].

David Thulke, Yingbo Gao, Petrus Pelser, Rein Brune, Rricha Jalota, Floris Fok, Michael Ramos, Ian van Wyk, Abdallah Nasir, Hayden Goldstein, Taylor Tragemann, Katie Nguyen, Ariana Fowler, Andrew Stanco, Jon Gabriel, Jordan Taylor, Dean Moro, Evgenii Tsymbalov, Juliette de Waal, Evgeny Matusov, Mudar Yaghi, Mohammad Shihadah, Hermann Ney, Christian Dugast, Jonathan Dotan, and Daniel Erasmus. 2024. *Climategpt: Towards ai synthesizing interdisciplinary research on climate change*. *Preprint*, arXiv:2401.09646.

Saeid Vaghefi, Veruska Muccione, Christian Huggel, Hamed Khashehchi, and Markus Leippold. 2022. Deep climate change: A dataset and adaptive domain pre-trained language models for climate change related tasks. In *NeurIPS 2022 Workshop on Tackling Climate Change with Machine Learning*.

Roopal Vaid, Kartikey Pant, and Manish Shrivastava. 2022. Towards fine-grained classification of climate change related social media text. In *Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics: Student Research Workshop*, pages 434–443, Dublin, Ireland. Association for Computational Linguistics.

Francesco S. Varini, Jordan Boyd-Graber, Massimiliano Ciaramita, and Markus Leippold. 2020. ClimaText: A Dataset for Climate Change Topic Detection. *Tackling Climate Change with Machine Learning workshop at NeurIPS 2020, Online, 11 December 2020 - 11 December 2020.*

Gengyu Wang, Lawrence Chillrud, and Kathleen McKe-
own. 2021. Evidence based automatic fact-checking
for climate change misinformation. In *ICWSM Work-
shops*.

Leon Weber-Genzel, Siyao Peng, Marie-Catherine De Marneffe, and Barbara Plank. 2024. VariErr NLI: Separating annotation error from human label variation. In Proceedings of the 62nd Annual Meeting of

the Association for Computational Linguistics (Volume 1: Long Papers), pages 2256–2269, Bangkok, Thailand. Association for Computational Linguistics.

Nicolas Webersinke, Mathias Kraus, Julia Bingler, and Markus Leippold. 2022. ClimateBERT: A Pretrained Language Model for Climate-Related Text.

Vinicius Woloszyn, Joseph Kobti, and Vera Schmitt. 2022. Towards automatic green claim detection. In *Proceedings of the 13th Annual Meeting of the Forum for Information Retrieval Evaluation*, FIRE '21, page 28–34, New York, NY, USA. Association for Computing Machinery.

Kun Xiang and Akihiro Fujii. 2023. DARE: Distill and Reinforce Ensemble Neural Networks for Climate-Domain Processing. *Entropy*, 25(4):643.

Yinan Yu, Samuel Scheidegger, Jasmine Elliott, and Asa Lofgren. 2024. climateBUG: A data-driven framework for analyzing bank reporting through a climate lens. *Expert Systems with Applications*, 239:122162.

A Datasets

**Summary of the construction of the dataset** To construct the datasets used for the reproducibility study, we followed the following steps:

1. We collected the datasets: we present statistics on their size in Table 3;

2. We explored each dataset to assess data quality (language, gibberish, text length, language), understand the labels (single label or multi-label, relation classification or multi-class classification);

3. We removed text longer than 4000 tokens, and smaller than 5 tokens, for details on which datasets were affected see Table 4;

4. We cleaned the text for formatting, punctuation, space, and encoding issues, and we removed duplicates: we report the statistics in Table 5;

5. We tried to keep existing splits if they are provided in the original source. However, if there was data contamination between the train set and the test set and the instances of contamination are rare, we remove the contamination. If there is a dataset construction issue, we rebuilt the dataset to remove the contamination.

6. We constructed based on the previous steps a training and development dataset, each smaller than 10k examples (as it is intended
for fine-tuning). We also construct a test dataset, however for the test dataset we keep the maximum size (to stay as close as possible to the original paper). During down-sampling, we ensured that label distribution was preserved through stratification. For heavily imbalanced datasets, we adjusted the sampling to improve the balance. In Table 6 we detail the ratio between the most frequent and least frequent label in the datasets, in the original datasets and in the newly created datasets.

<table><caption>Table 3: Original datasets: length of each split (train, test, dev) for each dataset, and size of the largest examples for each dataset (in number of tokens).</caption><thead><tr><th rowspan="2">Dataset</th><th colspan="3">Size</th><th>max.</th></tr><tr><th>train</th><th>dev</th><th>test</th><th>length</th></tr></thead><tbody><tr><td>ESGBERT action500</td><td>400</td><td>50</td><td>50</td><td>139</td></tr><tr><td>Green Claims</td><td>618</td><td>77</td><td>78</td><td>93</td></tr><tr><td>SUSTAINABLESIGNALS reviews</td><td>623</td><td>78</td><td>78</td><td>246</td></tr><tr><td>Implicit/Explicit Green Claims</td><td>618</td><td>77</td><td>78</td><td>93</td></tr><tr><td>climateFEVER claim (our split)</td><td>1228</td><td>154</td><td>153</td><td>78</td></tr><tr><td>logicClimate</td><td>679</td><td>218</td><td>179</td><td>316</td></tr><tr><td>ESGBERT G</td><td>1600</td><td>200</td><td>200</td><td>200</td></tr><tr><td>Global-Warming Stance (GWSD)</td><td>1890</td><td>210</td><td>200</td><td>70</td></tr><tr><td>ESGBERT S</td><td>1600</td><td>200</td><td>200</td><td>200</td></tr><tr><td>ESGBERT E</td><td>1600</td><td>200</td><td>200</td><td>200</td></tr><tr><td>esgbert Biodiversity</td><td>1760</td><td>220</td><td>220</td><td>445</td></tr><tr><td>esgbert Forest</td><td>1760</td><td>220</td><td>220</td><td>445</td></tr><tr><td>esgbert Nature</td><td>1760</td><td>220</td><td>220</td><td>445</td></tr><tr><td>esgbert Water</td><td>1760</td><td>220</td><td>220</td><td>445</td></tr><tr><td>Environmental Claims</td><td>2117</td><td>265</td><td>265</td><td>76</td></tr><tr><td>climatext (Wiki-doc) 10k</td><td>6000</td><td>300</td><td>300</td><td>282</td></tr><tr><td>climatext (Wiki-doc) wiki</td><td>3000</td><td>300</td><td>300</td><td>282</td></tr><tr><td>climate sentiment</td><td>800</td><td>200</td><td>320</td><td>698</td></tr><tr><td>Climate Specificity</td><td>800</td><td>200</td><td>320</td><td>698</td></tr><tr><td>Commitments And Actions</td><td>800</td><td>200</td><td>320</td><td>698</td></tr><tr><td>Net-Zero/Reduction</td><td>2752</td><td>345</td><td>344</td><td>1436</td></tr><tr><td>climateEng</td><td>2871</td><td>354</td><td>355</td><td>345</td></tr><tr><td>climateStance</td><td>2871</td><td>354</td><td>355</td><td>345</td></tr><tr><td>ClimateBERT's Climate detection</td><td>1040</td><td>260</td><td>400</td><td>701</td></tr><tr><td>Climate TCFD recommendations</td><td>1040</td><td>260</td><td>400</td><td>701</td></tr><tr><td>climateFEVER evidence</td><td>6140</td><td>765</td><td>770</td><td>441</td></tr><tr><td>climatext (Wiki-doc) claim</td><td>6000</td><td>300</td><td>1000</td><td>282</td></tr><tr><td>sciDCC</td><td>9224</td><td>1154</td><td>1161</td><td>503</td></tr><tr><td>ClimaINS</td><td>13755</td><td>1710</td><td>1710</td><td>5426</td></tr><tr><td>CC-Contrarian Claims</td><td>23436</td><td>2605</td><td>2898</td><td>4677</td></tr><tr><td>LobbyMap (Query)</td><td>11728</td><td>1320</td><td>3817</td><td>16743</td></tr><tr><td>climatext (Wiki-doc)</td><td>115847</td><td>3618</td><td>3826</td><td>317</td></tr><tr><td>LobbyMap (Stance)</td><td>15038</td><td>1660</td><td>4718</td><td>16743</td></tr><tr><td>climaQA</td><td>71478</td><td>8934</td><td>8936</td><td>1065</td></tr><tr><td>ClimaTOPIC</td><td>46803</td><td>8771</td><td>8984</td><td>1056</td></tr><tr><td>LobbyMap (Pages)</td><td>67091</td><td>7289</td><td>15755</td><td>16743</td></tr><tr><td>climateBUG-data</td><td>96852</td><td>24214</td><td>29551</td><td>4567</td></tr></tbody></table>

**Duplicates in ClimaQA** ClimaQA's original construction method inadvertently produced many duplicates. The task of ClimaQA is to determine whether a given a CDP response answers a specific CDP question. The dataset contains correct response-question pairs (positive pairs) and mismatching response-question pairs (negative pairs). For the negative pairs, responses were randomly selected from the pool of all other responses, excluding the correct one.

<table><caption>Table 4: Number of text larger than 4000 tokens</caption><thead><tr><th>Dataset</th><th>>4000 tokens</th></tr></thead><tbody><tr><td>Lobbymap (Pages)</td><td>66 0.073%</td></tr><tr><td>Lobbymap (Query)</td><td>48 0.285%</td></tr><tr><td>Lobbymap (Stance)</td><td>65 0.304%</td></tr><tr><td>ClimaINS</td><td>18 0.105%</td></tr><tr><td>CC-Contrarian Claims</td><td>1 0.003%</td></tr><tr><td>ClimateBUG-data</td><td>2 0.001%</td></tr></tbody></table>

This naive approach led to mislabeled examples due to all companies responding to the same set of questions. Additionally, the issue of duplication is exacerbated as companies often reuse or copy responses across multiple questions, increasing the likelihood of selecting an appropriate answer as a negative example. The original ClimaQA dataset led to poor performances; we therefore rebuilt the dataset making sure to avoid those issues.

**Duplicates in ClimateFEVER** For Climate-FEVER, the original dataset does not have a split. However, in ClimaBench (Spokoyny et al., 2023), the authors created the split at the evidence level. The same claim can be found in the training and the testing split but associated to different evidences. This resulted in partial contamination. We experimented with both evidence and claim-based splits (ClimateFEVER (climabench split) and Climate-FEVER (our split)).

## B Experimental Settings

Each dataset was divided into train, test, and development splits. When already available, we kept the initial split; otherwise, we used proportions of 80%, 10%, 10%. The dataset construction is detailed in Appendix A. The datasets with original test split are climatext (wiki-doc, wikipedia, 10k, claim), Environmental Claims, CC-Contrarian Claims, Lobbymap (Pages, Query, Stance), ClimateEng, ClimateStance, LogicClimate, ClimaTOPIC. For ClimatINS and ClimaQA we experimented with both the original splits and our splits.

To establish a benchmark for comparison, we computed several baselines using different statistical and machine-learning approaches:

**Random Baseline** A model that randomly predicts the label (following a uniform distribution).
<table><caption>Table 5: Number of duplicates in the unfiltered datasets. *text duplicates* correspond to duplicates of the un-processed texts, *cleaned text duplicates* correspond to duplicates of text after the formatting and encoding cleaning, *exact duplicates* correspond to duplicates of both the input text and the label. Therefore, the difference between *cleaned text duplicates* and *exact duplicates* correspond to duplicates with mismatching labels.</caption><thead><tr><th>dataset</th><th>text duplicates</th><th>cleaned text duplicates</th><th>exact duplicates</th><th>Dataset Size</th></tr></thead><tbody><tr><td>ESGBERT E</td><td>61</td><td>79</td><td>73</td><td>2000</td></tr><tr><td>CC-Contrarian Claims</td><td>20</td><td>32</td><td>30</td><td>28939</td></tr><tr><td>climateBUG-data</td><td>17445</td><td>20965</td><td>18815</td><td>150617</td></tr><tr><td>ClimaINS</td><td>2538</td><td>6986</td><td>6778</td><td>17175</td></tr><tr><td>climateFEVER evidence</td><td>3473</td><td>3473</td><td>2625</td><td>7675</td></tr><tr><td>logicClimate</td><td>458</td><td>458</td><td>113</td><td>1351</td></tr><tr><td>Net-Zero/Reduction</td><td>0</td><td>14</td><td>14</td><td>3441</td></tr><tr><td>Green Claims</td><td>0</td><td>0</td><td>0</td><td>773</td></tr><tr><td>LobbyMap (Stance)</td><td>7837</td><td>7843</td><td>4365</td><td>21416</td></tr><tr><td>Commitments And Actions</td><td>0</td><td>0</td><td>0</td><td>1320</td></tr><tr><td>climaQA</td><td>74190</td><td>74584</td><td>33580</td><td>89348</td></tr><tr><td>Environmental Claims</td><td>0</td><td>0</td><td>0</td><td>2647</td></tr><tr><td>SUSTAINABLESIGNALS reviews</td><td>4</td><td>4</td><td>2</td><td>779</td></tr><tr><td>esgbert Biodiversity</td><td>79</td><td>79</td><td>69</td><td>2200</td></tr><tr><td>climateStance</td><td>0</td><td>6</td><td>4</td><td>3580</td></tr><tr><td>Implicit/Explicit Green Claims</td><td>0</td><td>0</td><td>0</td><td>773</td></tr><tr><td>esgbert Water</td><td>79</td><td>79</td><td>79</td><td>2200</td></tr><tr><td>esgbert Nature</td><td>79</td><td>79</td><td>79</td><td>2200</td></tr><tr><td>climatext (Wiki-doc)</td><td>2597</td><td>2650</td><td>2595</td><td>123291</td></tr><tr><td>climatext (Wikipedia)</td><td>49</td><td>55</td><td>55</td><td>3600</td></tr><tr><td>climatext (10k)</td><td>299</td><td>327</td><td>327</td><td>6600</td></tr><tr><td>climatext (claim)</td><td>292</td><td>322</td><td>322</td><td>7300</td></tr><tr><td>esgbert Forest</td><td>79</td><td>79</td><td>73</td><td>2200</td></tr><tr><td>climateEng</td><td>0</td><td>6</td><td>4</td><td>3580</td></tr><tr><td>sciDCC</td><td>87</td><td>87</td><td>37</td><td>11539</td></tr><tr><td>LobbyMap (Pages)</td><td>2182</td><td>2303</td><td>2067</td><td>90135</td></tr><tr><td>Climate TCFD recommendations</td><td>0</td><td>0</td><td>0</td><td>1700</td></tr><tr><td>ESGBERT G</td><td>36</td><td>54</td><td>45</td><td>2000</td></tr><tr><td>ClimaTOPIC</td><td>42</td><td>717</td><td>675</td><td>64558</td></tr><tr><td>ESGBERT action500</td><td>19</td><td>21</td><td>21</td><td>500</td></tr><tr><td>Global-Warming Stance (GWSD)</td><td>266</td><td>270</td><td>268</td><td>2300</td></tr><tr><td>ESGBERT S</td><td>5</td><td>23</td><td>23</td><td>2000</td></tr><tr><td>ClimateBERT's Climate detection</td><td>0</td><td>0</td><td>0</td><td>1700</td></tr><tr><td>climateFEVER claim (our split)</td><td>0</td><td>2</td><td>2</td><td>1535</td></tr><tr><td>climate sentiment</td><td>0</td><td>0</td><td>0</td><td>1320</td></tr><tr><td>Climate Specificity</td><td>0</td><td>0</td><td>0</td><td>1320</td></tr><tr><td>climateFEVER evidence</td><td>3473</td><td>3473</td><td>2625</td><td>7675</td></tr></tbody></table>

<table><caption>Table 6: Ratio between the most frequent and least frequent label in the dataset. The ratio is reported for the original dataset, then after filtering for token length and dataset size, as well as for the test set. For heavily imbalanced datasets, we used a weighted loss during training.</caption><thead><tr><th rowspan="2">dataset</th><th colspan="3">Imbalance ratio</th><th rowspan="2">Weighted Training</th></tr><tr><th>Original</th><th>Filtered</th><th>Test</th></tr></thead><tbody><tr><td>climaQA</td><td>1.0</td><td>1.0</td><td>1.0</td><td></td></tr><tr><td>climatext (Wiki-doc)</td><td>1.0</td><td>1.0</td><td>1.0</td><td></td></tr><tr><td>esgbert Nature</td><td>1.1</td><td>1.1</td><td>1.0</td><td></td></tr><tr><td>SUSTAINABLESIGNALS reviews</td><td>1.3</td><td>1.3</td><td>1.3</td><td></td></tr><tr><td>ESGBERT action500</td><td>1.4</td><td>1.4</td><td>1.3</td><td></td></tr><tr><td>Commitments And Actions</td><td>1.4</td><td>1.4</td><td>2.3</td><td></td></tr><tr><td>ESGBERT S</td><td>1.5</td><td>1.5</td><td>1.5</td><td></td></tr><tr><td>Climate Specificity</td><td>1.5</td><td>1.5</td><td>1.6</td><td></td></tr><tr><td>Net-Zero/Reduction</td><td>1.6</td><td>1.6</td><td>1.6</td><td></td></tr><tr><td>climate sentiment</td><td>1.6</td><td>1.6</td><td>3.2</td><td></td></tr><tr><td>ClimaINS</td><td>1.7</td><td>1.7</td><td>2.1</td><td></td></tr><tr><td>Green Claims</td><td>1.9</td><td>1.9</td><td>1.9</td><td></td></tr><tr><td>ESGBERT E</td><td>2.0</td><td>2.1</td><td>2.2</td><td></td></tr><tr><td>Global-Warming Stance (GWSD)</td><td>2.2</td><td>2.2</td><td>2.3</td><td></td></tr><tr><td>climateBUG-data</td><td>2.3</td><td>2.3</td><td>1.3</td><td></td></tr><tr><td>LobbyMap (Stance)</td><td>2.5</td><td>2.3</td><td>2.7</td><td></td></tr><tr><td>ESGBERT G</td><td>2.7</td><td>2.7</td><td>2.8</td><td></td></tr><tr><td>Environmental Claims</td><td>3.0</td><td>3.0</td><td>3.0</td><td></td></tr><tr><td>esgbert Water</td><td>3.1</td><td>3.1</td><td>3.4</td><td></td></tr><tr><td>ClimateBERT's Climate detection</td><td>3.3</td><td>3.3</td><td>4.0</td><td></td></tr><tr><td>esgbert Forest</td><td>4.0</td><td>4.1</td><td>4.3</td><td></td></tr><tr><td>climateFEVER claim (our split)</td><td>4.3</td><td>4.2</td><td>4.0</td><td></td></tr><tr><td>Implicit/Explicit Green Claims</td><td>4.3</td><td>4.3</td><td>4.2</td><td></td></tr><tr><td>esgbert Biodiversity</td><td>4.7</td><td>4.8</td><td>4.9</td><td></td></tr><tr><td>LobbyMap (Pages)</td><td>4.7</td><td>4.5</td><td>3.0</td><td></td></tr><tr><td>climateFEVER evidence</td><td>6.3</td><td>2.4</td><td>5.6</td><td>✓</td></tr><tr><td>Climate TCFD recommendations</td><td>8.3</td><td>8.3</td><td>7.6</td><td>✓</td></tr><tr><td>climateStance</td><td>8.5</td><td>8.5</td><td>8.1</td><td>✓</td></tr><tr><td>climatext (wiki)</td><td>10.5</td><td>10.3</td><td>8.1</td><td></td></tr><tr><td>climateEng</td><td>12.0</td><td>12.1</td><td>23.1</td><td>✓</td></tr><tr><td>climatext (claim)</td><td>17.8</td><td>17.2</td><td>1.0</td><td></td></tr><tr><td>climatext (10k)</td><td>17.8</td><td>17.2</td><td>3.5</td><td></td></tr><tr><td>logicClimate</td><td>34.8</td><td>34.8</td><td>45.0</td><td>✓</td></tr><tr><td>ClimaTOPIC</td><td>44.3</td><td>3.4</td><td>26.7</td><td></td></tr><tr><td>sciDCC</td><td>49.2</td><td>48.8</td><td>49.5</td><td>✓</td></tr><tr><td>LobbyMap (Query)</td><td>65.3</td><td>65.3</td><td>47.0</td><td></td></tr><tr><td>CC-Contrarian Claims</td><td>110.9</td><td>19.5</td><td>83.2</td><td>✓</td></tr></tbody></table>

bution) implemented with sklearn's Dummy-Classifier.

**TF-IDF** A logistic regression using TF-IDF features to predict the class, implemented with sklearn's LogisticRegression and TidyVectorizer.

**DistilRoBERTa** A DistilRoBERTa (Sanh et al., 2019) fine-tuned on each task implemented with transformers' AutoModelForSequenceClassification for 10 epochs, with AdamW optimizer and a learning rate of 5e-5.

**Longformer** A Longformer (Beltagy et al., 2020) fine-tuned on each task implemented with transformers' AutoModelForSequenceClassification for 10 epochs, with AdamW optimizer and a learning rate of 5e-5.

**GPT-4o-mini** GPT-4o-mini used as a zero-shot
classifier implemented with OpenAI's API
(temperature of 0.1, top_p of 1)

**Llama 3.1 8B** Llama 3.1 8B used as a zero-shot
classifier (temperature of 0.1, top_p of 1)

**Llama 3.1 70B** Llama 3.1 70B used as a zero-shot
classifier (temperature of 0.1, top_p of 1)

Additionally, we reported the performances
reached in the original studies. Compiling and com-
paring reported performances from various papers
revealed significant heterogeneity in performance
metrics (Calamai et al., 2025). We reported the
value in the result tables; however, they are not
necessarily comparable.

**Finetuning Parameters** We finetuned for each dataset a Longformer and a distilRoBERTa for 10 epochs (with early stopping based on the validation F1-score). To speed up the training, we used half-precision floating-point (fp16). To maximize the GPU resources, we used a batch of 16 for distil-RoBERTa and 7 for Longformer (with an accumulation step of 2). We set the seed/random state to 42 for every library (both to create the dataset and train the models). We tried setting parameters to deterministic, however, Longformer was not compatible with PyTorch deterministic algorithms. The models are fine-tuned on the training dataset, evaluated at each epoch on the development dataset, and the best model is selected using the development dataset. We used a learning rate of 5e-5 (warmup ratio of 0.1, a weight decay of 0.01) following Spokoyny et al. (2023)'s parameters choices.

**Baselines** For the simple baselines, we used the seed 42. We used the scikit-learn implementations of the random classifier, the TF-IDF vectorizer, and the logistic regression. For the logistic regression, we used the weights to handle imbalanced datasets. For the relation classification, we parse both texts using the TF-IDF vectorizer and then concatenate the 2 vectors for the logistic regression. The model is fitted on the training dataset.

**Zero-shot** For the zero-shot models, we collected
the description of the labels from each paper and a
description of the annotation task. When missing,
we wrote a description. We then designed a corre-
sponding prompt template. Using GPT-4, we gen-
erated all the instruction prompts for each dataset.
We then generated the prediction for a small subset
of each dataset (50 examples). Using the outputs,

we validated the format of the prompts and ran the
experiment on a larger dataset. We identified issues
with some prompts that were manually updated.
Finally, we experimented with both zero-shot and
Chain-of-thought (CoT). If the performances on the
subset were significantly improved, we selected the
CoT prompt. The results using the CoT prompts
are: *climateFEVER*, *climateStance*, *climatext* (10K,
wiki, claims, wiki-doc), GWSD, LobbyMap.

C Detailed Analysis of Performance

In this section, we deep dive into the performances
of models and errors analysis for each dataset. For
each task, we provide a general result analysis, both
on the performance of the models and on the error
annotation that we conducted.

C.1 Climate-Related Topic Detection

**Task Description** Given an input sentence or a paragraph, output a binary label, “*climate-related*” or “*not climate-related*”.

**Experiment** We collected all available datasets: *climateBUG-Data* (Yu et al., 2024), *ClimateBERT's climate detection* (Bingler et al., 2023), *ClimaText* (Varini et al., 2020), and *SUSTAINABLES-IGNALS reviews* (Lin et al., 2023). *Climatext* is composed of *climatext wiki-doc* composed of text weakly labeled; and *climatext wikipedia*, *10k* and *claim* which are annotated datasets. For our experiments, we trained our models on *climatext wikipedia* training split as it was the larger annotated one. For *climatext 10k* we also used the training split for 10-Ks which contains only 58 positive samples. *climatext claim* did not contain a training split, so we used the same as for *climatext 10k*. Our cleaning and processing steps affected the *climateBUG-data* and all *climatext* datasets by removing duplicates, smaller/larger texts, and encoding issues. The results of the experiments are shown in Table 8.

**Analysis** We found that the TF-IDF baseline achieves strong performance, with values between 65.8% and 86.4%. This is expected as some words are heavily associated with climate change such as “greenhouse gas” or “climate”. However, the baseline is outperformed by the fine-tuned transformers, with macro F1-scores between 73.7% and 95.8%. This difference is statistically significant for all datasets except *SUSTAINABLESIGNALS reviews*. **This shows that (1) the task relies on vocabu-**
<table><caption>Table 7: This table presents the overlap between errors made by GPT-4o-mini and Llama models (8B and 70B). The first two columns shows the number of errors of Llama models among the sample of error of GPT-4o-mini (third column). The overall overlapping errors between GPT-4o-mini and Llama (8B) / Llama (70B) are reported in the following columns, along with the total number of errors for each model. (We do not include *LogicClimate* and *LobbyMap* because they are multilabel datasets)</caption><thead><tr><th rowspan="2">dataset</th><th colspan="3">Errors in sample</th><th colspan="5">Overlapping Errors</th></tr><tr><th>8B</th><th>70B</th><th>GPT</th><th>8B∩GPT</th><th>8B</th><th>70B∩GPT</th><th>70B</th><th>GPT</th></tr></thead><tbody><tr><td>climate sentiment</td><td>9</td><td>7</td><td>10</td><td>56</td><td>84</td><td>54</td><td>86</td><td>68</td></tr><tr><td>Environmental Claims</td><td>12</td><td>11</td><td>14</td><td>26</td><td>43</td><td>26</td><td>57</td><td>29</td></tr><tr><td>Climate TCFD recommendations</td><td>7</td><td>7</td><td>10</td><td>179</td><td>230</td><td>166</td><td>207</td><td>211</td></tr><tr><td>Implicit/Explicit Green Claims</td><td>5</td><td>9</td><td>12</td><td>5</td><td>16</td><td>9</td><td>16</td><td>12</td></tr><tr><td>climatext (10k)</td><td>8</td><td>5</td><td>9</td><td>8</td><td>17</td><td>5</td><td>8</td><td>9</td></tr><tr><td>Climate Specificity</td><td>8</td><td>11</td><td>16</td><td>58</td><td>75</td><td>75</td><td>102</td><td>91</td></tr><tr><td>Green Claims</td><td>4</td><td>2</td><td>6</td><td>4</td><td>7</td><td>2</td><td>10</td><td>6</td></tr><tr><td>climateBUG-data</td><td>17</td><td>14</td><td>20</td><td>81</td><td>238</td><td>74</td><td>128</td><td>112</td></tr><tr><td>ClimaNS (our split)</td><td>9</td><td>9</td><td>11</td><td>123</td><td>195</td><td>117</td><td>172</td><td>142</td></tr><tr><td>Global-Warming Stance (GWSD)</td><td>10</td><td>7</td><td>10</td><td>48</td><td>71</td><td>35</td><td>56</td><td>61</td></tr><tr><td>ESGBERT action500</td><td>8</td><td>7</td><td>10</td><td>8</td><td>15</td><td>7</td><td>17</td><td>10</td></tr><tr><td>sciDCC</td><td>9</td><td>10</td><td>10</td><td>597</td><td>750</td><td>573</td><td>670</td><td>646</td></tr><tr><td>ClimateBERT's Climate detection</td><td>11</td><td>12</td><td>17</td><td>11</td><td>33</td><td>12</td><td>16</td><td>17</td></tr><tr><td>climatext (Wikipedia)</td><td>14</td><td>10</td><td>14</td><td>16</td><td>18</td><td>12</td><td>14</td><td>16</td></tr><tr><td>climatext (Wiki-doc)</td><td>18</td><td>15</td><td>23</td><td>100</td><td>123</td><td>63</td><td>101</td><td>145</td></tr><tr><td>Net-Zero/Reduction</td><td>5</td><td>6</td><td>8</td><td>5</td><td>19</td><td>6</td><td>14</td><td>8</td></tr><tr><td>climateStance</td><td>5</td><td>9</td><td>10</td><td>96</td><td>158</td><td>88</td><td>113</td><td>120</td></tr><tr><td>CC-Contrarian Claims</td><td>7</td><td>8</td><td>10</td><td>359</td><td>588</td><td>322</td><td>473</td><td>424</td></tr><tr><td>Commitments And Actions</td><td>15</td><td>10</td><td>20</td><td>84</td><td>110</td><td>82</td><td>157</td><td>100</td></tr><tr><td>esgbert Forest</td><td>9</td><td>10</td><td>10</td><td>9</td><td>16</td><td>19</td><td>10</td><td>10</td></tr><tr><td>esgbert Water</td><td>9</td><td>4</td><td>9</td><td>9</td><td>15</td><td>4</td><td>10</td><td>9</td></tr><tr><td>climateFEVER evidence</td><td>7</td><td>8</td><td>10</td><td>218</td><td>357</td><td>182</td><td>251</td><td>286</td></tr><tr><td>ClimaTOPIC</td><td>10</td><td>10</td><td>10</td><td>424</td><td>496</td><td>424</td><td>461</td><td>471</td></tr><tr><td>ESGBERT E</td><td>5</td><td>5</td><td>8</td><td>5</td><td>8</td><td>5</td><td>9</td><td>8</td></tr><tr><td>climateEng</td><td>8</td><td>6</td><td>10</td><td>61</td><td>106</td><td>71</td><td>99</td><td>101</td></tr><tr><td>climaQA</td><td>13</td><td>10</td><td>20</td><td>96</td><td>379</td><td>94</td><td>222</td><td>188</td></tr></tbody><tfoot><tr><td>Total</td><td>242</td><td>219</td><td>317</td><td>2686</td><td>4167</td><td>2515</td><td>3488</td><td>3300</td></tr></tfoot></table>

<table><caption>Table 8: Performances of baselines for each dataset using the train, test described in section 3.3. The performance computed is the macro F1-score. The 95% confidence interval is computed using a 1000 bootstrap of sampled with replace as True of the size of the original test set. The reference performance is the best relevant performances reported in the original papers. 1. binary F1-score, 3. weighted average F1-score, c. climateBERT, f. climateBUG-LM, t. BERT.</caption><thead><tr><th>Dataset</th><th>Random</th><th>TF-IDF</th><th>Longformer</th><th>DistilRoBERTa</th><th>GPT-4o-mini</th><th>Llama</th><th>Llama 70B</th><th>Reference</th></tr></thead><tbody><tr><td>climateBUG-data(Yu et al., 2024)</td><td>49.6(49.0-50.1)</td><td>86.4(86.0-86.8)</td><td>90.5(90.1-90.8)</td><td>90.5(90.2-90.8)</td><td>89.2(88.8-89.5)</td><td>79.0(78.5-79.4)</td><td>88.3(87.9-88.6)</td><td>91.36<sup>f</sup></td></tr><tr><td>ClimateBERT Cl. det.(Bingler et al., 2023)</td><td>42.8(38.0-47.8)</td><td>79.3(74.3-83.8)</td><td>95.8(93.2-98.0)</td><td>94.0(91.0-96.9)</td><td>93.2(90.0-96.3)</td><td>88.3(84.3-91.8)</td><td>93.4(90.1-96.4)</td><td>99.1<sup>c</sup></td></tr><tr><td>climatext (Wiki-doc)(Varini et al., 2020)</td><td>49.3(47.8-50.8)</td><td>80.4(79.2-81.7)</td><td>85.6(84.5-86.8)</td><td>83.8(82.6-85.0)</td><td>84.0(82.8-85.2)</td><td>86.3(85.1-87.3)</td><td>89.8(88.8-90.7)</td><td>-</td></tr><tr><td>climatext (10k)(Varini et al., 2020)</td><td>46.9(41.5-52.2)</td><td>91.0(86.9-95.0)</td><td>97.0(94.3-99.0)</td><td>96.5(93.6-98.7)</td><td>95.4(91.8-98.2)</td><td>90.9(86.3-94.8)</td><td>96.0(93.0-98.5)</td><td>95.3<sup>c</sup></td></tr><tr><td>climatext (claim)(Varini et al., 2020)</td><td>48.2(45.1-51.2)</td><td>74.7(71.7-77.3)</td><td>67.8(64.6-70.9)</td><td>75.4(72.5-78.2)</td><td>82.2(79.7-84.6)</td><td>83.6(81.4-85.8)</td><td>85.7(83.7-87.8)</td><td>83.3<sup>c</sup></td></tr><tr><td>climatext (Wikipedia)(Varini et al., 2020)</td><td>40.1(35.3-44.9)</td><td>83.5(75.0-90.4)</td><td>83.9(76.0-90.4)</td><td>88.2(81.0-94.0)</td><td>84.7(77.1-91.5)</td><td>82.8(74.8-89.9)</td><td>87.4(80.4-93.5)</td><td>80.3<sup>c</sup></td></tr><tr><td>Sustainable Signals rev.(Lin et al., 2023)</td><td>37.6(27.0-49.0)</td><td>65.8(54.4-76.6)</td><td>73.2(63.2-83.3)</td><td>73.7(62.9-82.7)</td><td>69.7(57.6-79.7)</td><td>63.5(52.5-73.9)</td><td>73.3(62.6-82.4)</td><td>-</td></tr></tbody></table>

## lary, but (2) vocabulary is not enough to identify perfectly if the statement is about climate.

All zero-shot approaches reach performances above 75% showing that they can identify text as climate-related. The best zero-shot approaches even reached performances similar to the fine-tuned models. GPT-4o-mini and Llama 3.1 (70B) outperformed the fine-tuned models on *climaText (Wiki)*, and slightly underperformed on *climateBUG-data*, *ClimateBERT's Climate Detection* and *SUSTAINABLE SIGNALS reviews*. However, the difference between fine-tuning and zero-shot is not significant, except for Llama 3.1 70B outperforming significantly all fine-tuned models on *climaText (Wiki)*.

The reference performances remain higher, with F1-scores above 90% (Garrido-Merchán et al., 2023; Bingler et al., 2023; Yu et al., 2024).

### C.1.1 Error analysis

**climatext (Wiki-Doc)** When classifying sentences as climate-related or not, 91% of errors from GPT-4o-mini are FN. There are only 13 FP. 7 of them come from pages on topics that are indeed somewhat related to climate change, but that do not count as climate-related in *Climatext's* methodology – for example, pages about lakes that will be impacted by climate change (Example C.1). Of the remaining FP, 4 are out-of-context statements and 2 are actual errors from GPT-4o-mini. As for the sampled FN, 6/10 are text that is only indirectly linked to climate change. For example, a page about the lithosphere (Example C.2) counts as climate-related in *climatext's* methodology, but not in GPT-4o-mini's view. In 4/10 FN are out-of-context statements. For example, a statistics about politics (Example C.3) counts as "climate-related"
in *climatext's* methodology, but not in the view of **GPT. It is important to note that the climatext (Wiki-Doc) is weakly labeled, therefore the actual task is to identify if a sentence is extracted from a Wikipedia page related to climate change.**
Some sentences are therefore out-of-context and impossible to correctly classify. This dataset could be better used as a only a training dataset.

**Example C.1** (Not Climate-related). IISD Experimental Lakes Area (IISD-ELA) is a natural laboratory consisting of 58 small lakes and their watersheds set aside for scientific research.

**Example C.2** (Climate-related). There are two types of lithosphere:

**Example C.3** (Climate-related). The Senate vote throughout the states was between 10 and 20 percent.

**ClimaText (Wikipedia/10-Ks)** For *climatext (Wikipedia)* and *climatext (10-Ks)* the majority of errors are FN (respectively 75% and 89%).

There are actually only 4 FP for *climatext (Wikipedia)* and 1 for *climatext (10-Ks)*. All the FP are linked to environmental issues, and depending on the context, could be linked to climate change. Either directly such as Example C.5 or indirectly such as Example C.4. We believe those errors are all ambiguous and debatable.

For the FN in *climatext (wiki)*, we found 3 examples that are annotation errors (e.g. Example C.6 as "*Acronyms, potentially well connected to climate change, must be mentioned along with some mechanism/cause/effect of climate change*`). Those annotation errors occurred on examples that are ambiguous, yet precisely described in the guidelines. *This shows that Human do also struggle with ambiguous statements.* We found 2 errors that are due to our prompt which is a simplified version of the guidelines, therefore missing some nuance. 3 are due to indirect links to climate-change. The remaining 2 sampled errors are debatable. In the guidelines, it is mentioned that "*Just mentioning clean energy, emissions, fossil fuels, etc. is not sufficient: rather it must be connected to an environmental (CO2) or societal aspect (divestment, Kyoto treaty) of climate change.*". Example C.7 was classified by the model as "not climate-related" while the annotator decided it was "climate-related". While this example is definitely climate-related with the use of "zero-carbon", when strictly following the guidelines, this could be labeled "not climate-related".

For the FN in *climatext (10-Ks)*, due to the high performances, there are actually only 8 FN. There are 2 actual errors due to the following reasoning: "*However, it does not directly discuss the mechanisms, causes, or effects of climate change itself*" (Example C.8). *The model was anchored to the part of the prompt mentioning mechanisms, causes, or effects.* 2 errors are debatable due to the issue mentioned for *climatext (wiki)*. The remaining 4 errors are obvious annotation errors, most likely due to the Active Learning approach used to create the dataset. Those examples contain words such as "political climate" or "economic climate" (e.g. Example C.9). They were mislabeled in the dataset but correctly classified by GPT-4o-mini.

**Example C.4** (Not Climate-related). Think-tanks such as the World Pensions Council (WPC) argued that the keys to success lay in convincing officials in the U.S. and China, by far the two largest national emitters:

**Example C.5** (Not Climate-related). "Indirect effects include the fact that aerosols can act as cloud condensation nuclei, stimulating cloud formation."

**Example C.6** (Climate-related). The location of UNFCCC talks is rotated by regions throughout United Nations countries.

**Example C.7** (Climate-related). As of 2012, France generated over 90% of its electricity from zero carbon sources, including nuclear, hydroelectric, and wind.

**Example C.8** (Climate-related). In addition, we cannot control the actions of our third party manufacturers or the public's perceptions of them, nor can we assure that these manufacturers will conduct their businesses using climate change proactive or sustainable practices.

**Example C.9** (Climate-related). The current economic climate, especially in Europe, may have an adverse effect in the markets in which we operate.

**ClimateBERT's climate detection** For ClimateBERT's climate detection dataset, there are 11 FP and 6 FN. The majority of FP (7/11) stem from the use of the ambiguous word "sustainable". Indeed, "sustainable" can be used in a business context meaning that the business is economically viable (Example C.10). However, some of those FP (3) mentioning "sustainable" are actually debatable (e.g. Example C.11 in which "sustainable concepts" is likely to refer to environmental sustainability).
In the remaining FP, 2 are actually mislabeled, and 1 is debatable.

Of the 6 FN, 4 are indirectly but not explicitly linked to climate change or the environment. For example, the dataset labels as climate-related some statements about the fossil fuel/energy industry that do not mention anything regarding the environmental impact, policy, or governance (Example C.12). It also labels as positive statements about the governance of sustainability-related activity (Example C.13).

**Example C.10** (Not Climate-related). With passion and integrity at the heart of everything we do, we aim to build a sustainable business that makes a positive difference for all, now and in the future.

**Example C.11** (Not Climate-related). We have a long heritage of innovation and strive to provide athletes with the best by creating high-performance and competitive products. In 2020, we continued to serve consumers with innovative technologies and sustainable concepts built into our products.

**Example C.12** (Climate-related). On January 5, 2021, Centrica plc. closed a transaction to sell its entire ownership interest in Direct Energy to NRG Energy Inc. (NRG). Effective January 5, 2021, NRG provided a $300 million guarantee, supported by a $300 million letter of credit for Direct Energy's obligations to ATCO Gas and ATCO Electric under the transaction agreements.

**Example C.13** (Climate-related). The diagnosis of organizational culture represents the internal scenario and it is one of the elements considered/ analyzed in the definition of the drivers. Since 2017, after reviewing the research strategy by our organizational behavior area, only the Organizational Climate and Engagement Survey was applied annually, as presented in the table below.

**ClimateBUG** In this dataset, GPT produces FN (54%) and FP (46%) in roughly equal proportion. In the sampled errors, we found that 2 FN are statements where the link to climate comes from the context – which is however not given (Example C.14). As in *ClimateBERT's climate detection*, the word “sustainability” caused problems. However, in this case, the annotation guidelines are still ambiguous. It mentions that “Sustainability not related to the environment (i.e. sustainable profits)” should be classified as “Not Climate”. However, when not given sufficient context, the word can still be ambiguous. Some examples mention “sustainability”

in a broader sense (without explicitly mentioning environmental/climate sustainability), and they are labeled positive (Example C.15) – although GPT believes they are negative. Vice versa, other broad references to sustainability are labeled as negative (Example C.16) – and GPT believes they are positive. This issue accounts 5/10 FP. This shows that both annotators and GPT struggle with this ambiguity and highlights inconsistencies in the annotations. The remaining FP (5/10) are climate-related tables that GPT-4o-mini classified as positives (therefore not following the annotation guidelines to classify tables as negatives). However, we also found at least 1 table mislabeled in the gold standard (Examples C.17). This also shows the importance of well-defined labels and label consistency during annotation, as it can introduce mislabeled examples. This is particularly important when comparing performances of 88.8% and 90.5%.

**Example C.14** (Climate-related). This also gives us the opportunity to exchange knowledge and experiences with representatives from different spheres of society and to implement the principles and reach our targets.

**Example C.15** (Climate-related). After all, they are the source of jobs, innovation, sustainability, and prosperity.

**Example C.16** (Not Climate-related). Firstly, Nordic customers are relatively advanced in their sustainability considerations, and they expect their bank to be able to support sustainable development across segments.

**Example C.17** (Climate-related). SCOPE 2—Electricity consumed (market-based method)—Electricity consumed (location-based method)3, 8 67.

## C.2 Climate-Related Subtopic Detection

**Task Description** Given an input sentence or a paragraph, output a subtopic related to climate change. This is a multiclass classification task.

**Experiment** We collected the *ClimaTOPIC* (Spokoyny et al., 2023), *esgbert Nature* (Schimanski et al., 2024a), *ClimateEng* (Vaid et al., 2022) and *SciDCC* (Mishra and Mittal, 2021) datasets. The results are in Table 9. Our cleaning and processing steps impacted all datasets: it removed duplicated examples from the *esgbert Nature* and *ClimaTOPIC* test sets, and it removed the URLs in *climateEng*. And for *SciDCC*, we made sure that each
category was represented in the test set; however, some categories are severely under-represented (Geology appears only 28 times overall, and therefore only 3 times in the test set). We note that no original split is given for *SciDCC*.

**Result Analysis** In our experiment, the best-performing models are usually the fine-tuned models (except on Biodiversity label from *esgbert Nature*). While the performance of the fine-tuned models improved over the TF-IDF baseline, the difference is statistically significant only for *ClimaTOPIC*, *esgbert Nature* (nature and water labels). **Overall, this shows that the classes are distinguishable using mostly the vocabulary.**

The zero-shot approaches perform comparably to the fine-tuned transformers on most datasets. While the fine-tuned models outperformed the zero-shot approaches, the difference is statistically significant only on *SciDCC*, *ClimaTOPIC*. Those two datasets are constructed automatically and not annotated by humans; therefore, building a comprehensive prompt is difficult, while identifying linguistic patterns remains effective.

In terms of order of magnitude, our models achieved performance levels comparable to those reported in the original studies. However, we observed a notable performance gap on *ClimaTOPIC* and *SciDCC*, where our models underperformed relative to the results presented in prior work. For *ClimaTOPIC* this could be explained by the smaller training dataset. For *SciDCC*, we provide only the summary of the article, but the dataset also contains the content of the article, which can help increase performance.

### C.2.1 Error analysis

**SciDCC** We have reported the confusion matrix of GPT-4o-mini predictions in figure 1a. We observe several types of errors: (1) hallucinations ("Marine Biology" and "Archaeology" are not labels in this dataset) (2) predicting too frequently the label "Climate" (3) mixing related labels ("biology", "Biotechnology" and "Genetically Modified", or "Geology" and "Earthquakes")

All these labels are so close in meaning because they are categories that changed over time as shown in Figure 1b. We can observe the topic shift over time: "Ozone Holes", "GMOs" and then "Global Warming". Categories not only appeared but also changed over time. For example, "endangered animals" topics might have later been included in

"Animals" or "Zoology". Moreover, multiple categories might fit a specific topic such as "Extinction" and "Climate" as one is the consequence of the other. Moreover, in the original paper, they did not provide a clear definition of each label. Examples of surprising gold labels include: Example C.18 classified as "Geography" and not "Pollution", or Example C.19 classified as "Geography" and not "Geology".

When reviewing sampled errors by GPT-4o-mini, we identified that 11/20 are due to text with multiple possible labels (Examples C.21 which could fit in "Animals", "Endangered Animals", "Extinction" or "Climate"), 7/20 contain debatable labels (Examples C.20 labeled "Earthquakes" but could fit in "Geology"). The latter can be explained as in our experiment we only provide the summary, but the article's content might be linked to the original label. **While this dataset might be interesting for studying the evolution of environmental topics, it would require human annotations to re-label the data to make it reliable.**

**Example C.18 (Geography).** Shipping traffic can be a major source of tiny plastic particles floating in the sea, especially out in the open ocean. In a paper published in the scientific journal

**Example C.19 (Geography).** New research led by the University of Cambridge has found rare evidence – preserved in the chemistry of ancient rocks from Greenland – which tells of a time when Earth was almost entirely molten.

**Example C.20 (Earthquakes).** Diamonds, those precious, sparkling jewels, are known as the hardest materials on Earth. They are a high-pressure form of carbon and found deep in the ground.

**Example C.21 (Animals).** Many species might be left vulnerable in the face of climate change, unable to adapt their physiologies to respond to rapid global warming. According to a team of international researchers, species evolve heat tolerance more slowly than cold tolerance, and the level of heat they can adapt to has limits.

**ESGBERT Nature** GPT-4o-mini performs well overall, but when errors occur, they tend to be FN (90% for forest, 77% for water and 60% for biodiversity), especially in cases involving indirect references. There are 1 FP and 9 FN for the "forest" dataset. All 9 FN were indeed related to forests, but they were misclassified due to a lack of explicit references to forest management. Instead,
<figure><img src="image_4.png" /><figcaption>(a) Confusion matrix of GPT-4o-mini predictions on *SciDCC* dataset. “Archeology” and “Marine Biology” are not labels from *SciDCC*.</figcaption></figure>

<figure><img src="image_5.png" /><figcaption>(b) Evolution of proportion of labels over time in *SciDCC* dataset (Mishra and Mittal, 2021)</figcaption></figure>
<table><caption>Table 9: Performances of baselines for each dataset using the train, test described in section 3.3. The performance computed is the macro F1-score. The 95% confidence interval is computed using a 1000 bootstrap of sampled with replace as True of the size of the original test set. The reference performance is the best relevant performances reported in the original papers. *l.* binary F1-score, *c.* climateBERT, *g.* EnvironmentalBERT, *l.* Longformer, *r.* RoBERTa</caption><thead><tr><th>Dataset</th><th>Random</th><th>TF-IDF</th><th>Longformer</th><th>DistilRoBERTa</th><th>GPT-4o-mini</th><th>Llama</th><th>Llama 70B</th><th>Reference</th></tr></thead><tbody><tr><td>ClimaTOPIC(Spokoyny et al., 2023)</td><td>6.0(5.6-6.5)</td><td>46.6(45.4-47.8)</td><td>55.8(54.4-57.2)</td><td>54.8(53.5-56.1)</td><td>35.8(34.8-36.7)</td><td>34.5(33.4-35.4)</td><td>30.5(29.5-31.5)</td><td>65.22<sup>r</sup></td></tr><tr><td>climateEng(Vaid et al., 2022)</td><td>13.2(10.4-16.2)</td><td>58.4(48.2-66.8)</td><td>70.5(60.8-77.7)</td><td>67.3(58.3-75.4)</td><td>65.8(55.5-73.9)</td><td>53.9(43.5-62.2)</td><td>61.1(51.9-68.9)</td><td>74.58<sup>r</sup></td></tr><tr><td>esgbert Biodiversity(Schimanski et al., 2024a)</td><td>41.1(35.0-47.7)</td><td>91.3(85.3-96.2)</td><td>91.3(85.6-95.7)</td><td>89.0(83.3-94.2)</td><td>91.8(86.4-96.2)</td><td>88.8(83.0-94.3)</td><td>82.3(75.9-87.7)</td><td>92.29<sup>1-g</sup></td></tr><tr><td>esgbert Forest(Schimanski et al., 2024a)</td><td>41.4(35.4-47.2)</td><td>92.9(87.4-97.1)</td><td>97.2(93.8-99.3)</td><td>97.0(93.8-99.4)</td><td>91.6(86.4-96.2)</td><td>87.4(81.2-92.5)</td><td>86.2(80.3-91.8)</td><td>95.37<sup>1-g</sup></td></tr><tr><td>esgbert Nature(Schimanski et al., 2024a)</td><td>51.5(45.0-58.0)</td><td>82.9(77.7-87.7)</td><td>89.8(85.8-93.6)</td><td>89.9(85.8-93.8)</td><td>82.9(77.6-87.9)</td><td>86.7(82.0-91.0)</td><td>88.0(83.3-92.4)</td><td>94.19<sup>1-g</sup></td></tr><tr><td>esgbert Water(Schimanski et al., 2024a)</td><td>42.7(36.6-49.5)</td><td>84.6(77.5-90.8)</td><td>95.3(91.7-98.6)</td><td>93.9(89.7-97.5)</td><td>93.8(89.4-97.6)</td><td>89.4(83.4-94.1)</td><td>93.5(89.1-96.9)</td><td>95.10<sup>1-c</sup></td></tr><tr><td>sciDCC(Mishra and Mittal, 2021)</td><td>3.7(2.7-4.8)</td><td>42.6(39.0-46.7)</td><td>39.8(37.1-42.7)</td><td>32.1(39.2-45.9)</td><td>29.2(26.8-31.4)</td><td>20.5(18.3-22.7)</td><td>30.0(27.5-32.2)</td><td>54.79<sup>j</sup></td></tr></tbody></table>

these statements included discussions about plants, wood usage, the economic impact of wildfires, and the effects of climate change—topics inherently connected to forestry (Example C.22).

**Example C.22.** While gross loss reserve estimates for the 2018 California wildfires were also reduced, this was largely offset by a reduction in reinsurance recoverables resulting in very little change to estimated net losses from those wildfires. [Forest]

There are 2 FP and 7 FN for the “water” dataset. The 2 FP are both debatable as they explicitly talk about water-related topics (Example C.23). As part of the FN, we found 2 focusing on financial consequences, therefore indirectly linked to “water”. The “water” label: “The topic of water centers around water management, consumption, and pollution. [...]” could be understood as focusing only on freshwater, and not include water-related natural disasters such as tsunami. We found that 4 FN are related to those topics (Example C.24). The last FN is an actual error by GPT-4o-mini.

**Example C.23.** 70% of the water in the world is used for agriculture. [Not Water]

**Example C.24.** In addition, Intel employees responded to these events with great generosity, contributing $1.8 million toward tsunami relief, $1.6 million to help the victims of Hurricane Katrina and $475,000 for those affected by the earthquake in Pakistan.

While this dataset is well-built, these errors highlight the need to have precise definitions and the ambiguity of indirect relation to the topic. This is further emphasized by the inter-annotator agreement (Fleiss' $\kappa = 80\%$) showing some disagreement between annotators.

**ClimateEng** Firstly we see that the performance of the GPT-4o-mini model is close to the performances of the finetuned models. We have reported the confusion matrix in figure 2, we see that a large

portion of issues come from the labels "General" and "Politics" being mixed-up.

<figure><img src="image_6.png" /><figcaption>Figure 2: Confusion matrix of GPT-4o-mini predictions on ClimateEng dataset.</figcaption></figure>

When investigating the sampled errors, we found that 12/30 are actual errors, 4/30 are debatable and 12/30 are mislabeled examples (Example C.25). We found that the actual errors are related to the very specific definitions of the labels. For example, the political label focuses only on leaders, political organizations and awareness about climate change and not on political activism or opinions. However, we see inconsistencies in the annotations for political awareness: Example C.27 was annotated as “Politics” - while GPT-4o-mini predicted “General” - and the Example C.28 was annotated as “General” - while GPT-4o-mini predicted “Politics”. 2 tweets contain hashtags about COP25, which could fit into the “Politics” category but were annotated as “General”. In the guidelines, the “Ocean/Water” label is specific toward “biodiversity”, but GPT-4o-mini predicted Example C.26 as “Ocean/Water”, therefore not following the instructions. *Overall, we observe that the labels are really specific and*
counter-intuitive compared to the usual definition
of the categories. GPT-4o-mini tend to struggle
with this re-definition of terms. However, the inter-
annotator agreement (Cohen's κ = 0.739) also
shows that people do have disagreements and also
struggle with those definitions. Once again, this
shows the importance of having well-defined la-
bels.

**Example C.25** (Agriculture/Forestry). Contracts awarded for Scottish offshore wind farm <URL>#energy #sustainability #climatechange

**Example C.26** (General). Scientists fear that a
change in ocean circulation could profoundly alter
climate patterns. <URL>

**Example C.27** (Politics). U.S. students excel in global warming awareness, social justice activism, racial angst, and gender sensitivity while China's teens waste their time gaining proficiency in reading, math and science, according to global education study. <URL>

**Example C.28** (General). On December 11, 2019, the Intergenerational Dialogue was successfully held with the participation of Youth Delegation and a series of climate experts. The aim of the dialogue is to exchange personal experience and guide the delegates to the most fertile areas of climate change. <URL>

**ClimaTOPIC** As the topics are categories of
questions from CDP questionnaires, many cate-
gories are actually related. The Example C.29 and
C.30 both deal with emissions of buildings, how-
ever, one is from the “Emissions” category and the
other is from the “Building” category. This might
be improved by giving more context to the model,
in particular, the type of questions available in each
category.

From the sample of errors, we found that 9/10
errors are similar mistakes. From the confusion ma-
trix (Figure 3), we see that the model fails most of-
ten on “Governance and Data Management”, “Op-
portunities” and “Strategy” labels which are non-
specific categories. While we cannot consider the
data to be mislabeled—since the labels correspond
to the section headers from which the responses
were extracted—it is evident that the model’s pre-
predictive performance is constrained by the broad
and non-specific nature of these labels.

**Example C.29** (Emissions). The Direct emission
from institutional buildings occur but we are not

able to estimate it on this inventory, due to financial
constraints and limited manpower.

**Example C.30** (Building). So as to decrease GHG emissions, under the climate action plan for reducing GHG emissions, the target of the energy upgrade of municipal buildings has been set, tackling energy efficiency. The emission reduction target has been set to 27,804 tnCO2eq annually.

C.3 Detecting Climate-Related Financial Disclosure

**Task Description** This is a particular case of sub-
topic classification. Given a paragraph or a sen-
tence, output the TCFD recommendation associ-
ated with the content of the text.

**Experiment** We were not able to collect any of the datasets; however, the authors of Bingler et al. (2021) released a smaller TCFD annotated dataset alongside their follow-up work (Bingler et al., 2023). We identify it as *Climate TCFD recommendations*. So we evaluated this dataset with the setting described in section B and reported the results in Table 10. Our pre-processing did not alter this dataset.

**Result Analysis** The best performing approaches were the fine-tuned transformers reaching F1-score above 68%, significantly out-performing the zero-shot approaches and the TF-IDF baseline. The TF-IDF baseline is performing relatively well given the nature of the task. Moreover, it performed better than the zero-shot approaches.

As this dataset is not part of a previous study, we
rely on other works for performance comparison.
Bingler et al. (2021) observed that paragraph-level
approaches underperformed compared to methods
that aggregate sentence-level features, such as ma-
jority voting or logistic regression. In contrast,
Sampson et al. (2022) found that classical models,
such as TF-IDF-based random forests and stacked
approaches, outperformed fine-tuned transformers,
which performed well but not optimally. Interest-
ingly, the zero-shot approach evaluated by Auzepy
et al. (2023) yielded surprisingly strong results
given the complexity of the task.

C.3.1 Error analysis

**climate TCFD recommendations** The results, presented in the confusion matrix in Figure 4, revealed that 62% of the model's classification errors are associated with the "Strategy" label. Additionally, the model tends to over-predict the "None"
<figure><img src="image_7.png" /><figcaption>Figure 3: Confusion matrix of GPT-4o-mini predictions on ClimaTOPIC dataset.</figcaption></figure>

<table><caption>Table 10: Performances of baselines for each dataset using the train, test described in section 3.3. The performance computed is the macro F1-score. The 95% confidence interval is computed using a 1000 bootstrap of sampled with replace as True of the size of the original test set. The reference performance is the best relevant performances reported in the original papers.</caption><thead><tr><th>Dataset</th><th>Random</th><th>TF-IDF</th><th>Longformer</th><th>DistilRoBERTa</th><th>GPT-4o-mini</th><th>Llama</th><th>Llama 70B</th><th>Reference</th></tr></thead><tbody><tr><td>Climate TCFD rec.(Bingler et al., 2023)</td><td>16.5(12.9-20.3)</td><td>56.5(50.8-61.9)</td><td>69.4(64.0-74.4)</td><td>68.5(63.0-73.5)</td><td>47.5(42.2-53.0)</td><td>43.2(37.5-48.0)</td><td>50.2(44.4-55.2)</td><td>-</td></tr></tbody></table>

category, which accounts for 47.4% of the total errors.

In our sampled errors, we found that 15/30 misclassifications are genuine errors, with 4/30 stemming from the model's inability to correctly identify strategic actions or cases where the strategic nature of the text is implied but not explicitly stated. The use of a larger model (GPT-4) mitigated this issue to some extent, reducing the rate of strategy-related errors by 75%.

Furthermore, there is notable confusion between the “Strategy” and “Risk” categories. This can be explained by the guidelines. The “strategy” label includes descriptions of the climate-related risks, while “risk” includes more specifically risk management, which therefore exclude descriptions

of the risks. Moreover, it seems that the “strategy” label is used as “default” label for vague statement mentioning climate-related topics (2/30). Example C.32 and C.33 are both classified as “strategy” in the gold standard, yet they do not describe climate-related risks and impacts on the company. Overall, we believe that there is a significant overlap between the strategy and risk labels, as describing the risk and describing how the risk is managed are often discussed in the same sentences.

We found 5 examples of texts not focusing on climate-related disclosure but more broadly about sustainability. These examples seem to indicate that this should be considered as related to climate-disclosure. However, even if it mentions sustainability, Example C.31 is mainly about plastic pol-
lution, which we considered as an annotation error.

We also found the issues mentioned on other task: mentioning energy is classified as not climate-related by gpt-4o-mini but systematically climate-related in the dataset (2/30).

**Example C.31** (governance). Report 2019 (SR 2019), as requested by the shareholders. The hard-copies will be delivered once they are made available to the Company. * Nevertheless, we hope you would consider the environment before you decide to print the above reports or request for the printed copy of the IR 2019, GFR 2019 and SR 2019. The environmental concerns like global warming, deforestation, climate change and many more affect every human, animal and nation on this planet.

**Example C.32** (strategy). The Sustainability Report is based upon the internationally recognized Global Reporting Initiative (GRI) Standards. Our reporting is also guided by the Sustainability Accounting Standards Board (SASB) and the Financial Stability Board's Task Force on Climate-related Financial Disclosures' (TCFD) recommendations.

**Example C.33** (strategy). The progress toward all Green Company targets is tracked through an environmental data reporting system and is disclosed in detail in our annual Green Company Report, available on our corporate website as of spring 2021. ! > ADIDAS-GROUP.COMSENVIRONMENTAL-APPROACH Own operations: Progress toward 2020 targets

**Example C.34** (governance). Report 2019 (SR 2019), as requested by the shareholders. The hard-copies will be delivered once they are made available to the Company. * Nevertheless, we hope you would consider the environment before you decide

to print the above reports or request for the printed copy of the IR 2019, GFR 2019 and SR 2019. The environmental concerns like global warming, deforestation, climate change and many more affect every human, animal and nation on this planet.

## C.4 Detecting Environmental, Social, and Governance Disclosure

**Task Description** This is a particular case of subtopic classification. The topics are “Environment”, “Social”, and “Governance” categories.

**Experiment** The only openly available datasets are *ESGBERT E* (Environmental) *ESGBERT S* (Social) and *ESGBERT G* (Governance) datasets (Schimanski et al., 2023b). We evaluated this dataset with the setting described in section B and reported the results in Table 11. Our pre-processing pipeline slightly impacted the dataset when removing duplicates.

**Result Analysis** In our experiments, we found that for the *ESGBERT E*, *S* and *G* datasets, the best performing model is DistilRoBERTa. As we reported the macro average, we also computed the binary F1-score 95%, 88%, 76% (respectively E, S and G), which are close to the performance reported in the original study: 93%, 92%, 79%. This shows that fine-tuned models can identify text relevant to those E, S and G categories. However, we also see that the TF-IDF baseline performances is over 80% for each dataset, showing that those thematic have largely different vocabulary, making them easily identifiable. Finally, the zero-shot approaches also performed well on those datasets, with performances above 78% for GPT-4o-mini. Showing that zero-shot models can also identify those categories. Interestingly, the only dataset where the fine-tuned model performed significantly better than GPT-4o-mini is the “Social” dataset.

Moreover, Schimanski et al. (2023b) measured a high inter-annotator agreement (Fleiss' κ of more than 86%). Overall, this shows that those labels are quite easy to distinguish.

### C.4.1 Error analysis

**ESGBERT E** We are only reviewing errors from *ESGBERT E* as it is the most relevant to our topic. GPT-4o-mini made only 8 errors. 6 FN and 2 FP. 2/6 FN are text mentioning the energy industry (Example C.35) but not directly talking about the environment. 1/6 FN is truncated, 1/6 FN is out-of-context, which makes them difficult to classify.

<figure><img src="image_8.png" /><figcaption>Figure 4: Confusion matrix of GPT-4o-mini predictions on *Climate TCFD recommendations* dataset.</figcaption></figure>
<table><caption>Table 11: Performances of baselines for each dataset using the train, test described in section 3.3. The performance computed is the macro F1-score. The 95% confidence interval is computed using a 1000 bootstrap of sampled with replace as True of the size of the original test set. The reference performance is the best relevant performances reported in the original papers. 1. binary F1-score, e. ESGBERT</caption><thead><tr><th>Dataset</th><th>Random</th><th>TF-IDF</th><th>Longformer</th><th>DistilRoBERTa</th><th>GPT-4o-mini</th><th>Llama</th><th>Llama 70B</th><th>Reference</th></tr></thead><tbody><tr><td>ESGBERT E(Schimanski et al., 2023b)</td><td>42.6(35.1-49.3)</td><td>88.1(82.5-92.8)</td><td>95.8(92.4-98.7)</td><td>95.8(92.3-98.7)</td><td>95.0(91.3-98.2)</td><td>95.1(91.2-98.2)</td><td>94.6(90.8-97.6)</td><td>93.19<sup>1, e</sup></td></tr><tr><td>ESGBERT G(Schimanski et al., 2023b)</td><td>53.5(46.4-60.6)</td><td>80.2(73.5-86.1)</td><td>78.2(71.7-84.5)</td><td>83.7(77.0-89.7)</td><td>83.5(77.4-88.8)</td><td>75.6(69.0-81.3)</td><td>62.6(55.3-69.0)</td><td>78.86<sup>1, e</sup></td></tr><tr><td>ESGBERT S(Schimanski et al., 2023b)</td><td>54.4(47.0-60.8)</td><td>82.6(76.8-88.1)</td><td>88.3(83.7-92.6)</td><td>89.6(85.2-93.7)</td><td>78.1(72.4-84.1)</td><td>77.9(71.5-83.4)</td><td>73.6(67.3-79.8)</td><td>91.90<sup>1, e</sup></td></tr></tbody></table>

We also found 1/6 FN mislabeled example talking about the "office environment" (Example C.36). Overall, the errors stem from statement that are systematically classified as related to the environment such as anything related to energy, even if the statement is not explicitly talking about the environment. For the 2 FP, 1/2 is related to the use of "sustainable" in an economic context, and 1/2 describes the impact of floods on mining operations which was classified by GPT-4o-mini as "Environment" but not in the gold standard.

**Example C.35 (Environment).** e Natural gas liquids for Russia are included in crude oil.

**Example C.36 (Environment).** The pandemic may also have long-term effects on the nature of the office environment and remote working, which may result in increased costs and present operational and workplace culture challenges that may also adversely affect our business.

## C.5 In-depth Disclosure: Climate Risk Classification

**Task Description** Given an input sentence or a paragraph, output "opportunity" or "risk" label.

**Experiment** We collected climateBERT's climate sentiment dataset (Bingler et al., 2023). We evaluated this dataset with the setting described in section B and reported the results in Table 12. Our pre-processing did not alter this dataset.

**Result Analysis** In our experiments, as reported in Table 12, on climate sentiment, the best performing solutions are the fine-tuned transformers, with Longformer reaching a macro F1-score of 79.9%. The performance reported in the original study was similar, with a weighted F1-score of 83.8% (Bingler et al., 2023), within the confidence interval of our best performing model. The TF-IDF baseline is performing lower but still good, with a macro F1-score of 69%. This performance shows that the vocabulary for risk and opportunity is quite characteristic and identifiable. The zero-shot approaches' performances varied from 70.5% for Llama 3.1

70B to 77.3% for GPT-4o-mini approaching the performance of finetuned models.

As shown in the confusion matrix Figure 5, only 4 out of the 68 errors do not include the label neutral (5 out of 68 for distilRoBERTa). This shows that discriminating between polar opposite is easy, but the edge cases can be more challenging. If the majority class is the neutral class, then the performances are not representative of the actual performance of the model. This is further confirmed by the relatively lower inter-annotator agreement compared to previous tasks Friederich et al. (2021) reported a low IAA ($\alpha$ = 0.20) and Bingler et al. (2023) reported a moderate IAA (Krippendorff's $\alpha$ = 0.61).

### C.5.1 Error analysis

**climate sentiment** As shown in the confusion matrix displayed in Figure 5, most of the errors are on texts originally labeled as "neutral" (65% of errors). And the model tends to preferably output the "opportunity" label (43% of errors) reducing its precision.

In the sampled errors, we identified several patterns of misclassifications. 2/30 instances involved discussions of corporate ambitions (Example C.37) that were labeled as "opportunity" by GPT-4o-mini, but categorized as "neutral" in the gold standard. These texts are not specifically opportunities, but the mention of "ambitious targets" could be considered "positive" in the guidelines due to: "2) and/or about positive impact of an entity's activities on the society/environment 3) and/or associates specific positive adjectives to the anticipated, past or present developments and topics covered". 3/30 sampled errors contained discussions of both risks and opportunities (Example C.39), highlighting the lack of clarity in the annotation guidelines for such mixed content. Interestingly, 2/30 examples referenced indirect risks (Examples C.40 and C.41), which were labeled as "neutral" in the gold standard, yet identified as "risk" by the model. Lastly, we found 14/30 instances that were likely mis
<table><caption>Table 12: Performances of baselines for each dataset using the train, test describe in section 3.3. The performance computed is the macro F1-score. The 95% confidence interval is computed using a 1000 bootstrap of sampled with replace as True of the size of the original test set. The reference performance is the best relevant performances reported in the original papers. 3. weighted average F1-score, c. climateBERT.</caption><thead><tr><th>Dataset</th><th>Random</th><th>TF-IDF</th><th>Longformer</th><th>DistilRoBERTa</th><th>GPT-4o-mini</th><th>Llama</th><th>Llama 70B</th><th>Reference</th></tr></thead><tbody><tr><td>climate sentiment(Bingler et al., 2023)</td><td>35.9(30.6-41.1)</td><td>69.0(63.4-74.4)</td><td>79.9(74.8-84.4)</td><td>77.7(72.6-82.2)</td><td>77.3(72.4-82.2)</td><td>71.4(65.8-76.4)</td><td>70.5(64.7-75.5)</td><td>83.8<sup>3,c</sup></td></tr></tbody></table>

<figure><img src="image_9.png" /><figcaption>Figure 5: Confusion matrix of GPT-4o-mini predictions on ClimateBERT's *climate sentiment* dataset.</figcaption></figure>

labeled. We found 5/30 examples of texts that describe a risk-related structure (e.g. Examples C.42), which can reasonably be classified as neutral. While the dataset's labels are generally well-defined, these observations suggest that further refinement of definitions for certain edge cases could help improve consistency in classification.

**Example C.37 (neutral).** TD recently launched a bold and ambitious climate action plan to address the challenges of climate change. This includes a target to achieve net-zero greenhouse gas emissions in our operations and financing activities by 2050. We backed this commitment with the creation of a new Sustainable Finance and Corporate Transitions Group to support clients around the world, and an Environmental, Social and Governance (ESG) Centre of Expertise to participate in the global efforts required to deliver on this long-term target.

**Example C.38 (neutral).** To exclude the least energy efficient hydrocarbons and those that pose the greatest threat to the environment, because these are incompatible with the goal of combating climate change and they represent an economic risk for investors. This means turning down projects and companies that do the majority of their business in: o Oil sands production, o Oil extracted from the Arctic region (off-shore and on-shore production), o Shale gas or oil production involving excessive flaring or venting, o Infrastructure projects mainly intended for schemes covered by the exclusion criteria set out above, Credit Agricole S.A. is

committed to offsetting the Group's entire direct carbon footprint until 2040 via the

**Example C.39 (risk).** Sustainability aspects and climate-related risks and opportunities are integrated into the Group-wide risk management process at Daimler. They are understood to be conditions, events, or developments involving environmental, social or governance factors (ESG), the occurrence of which may have an actual or potential impact on the Daimler Group's profitability, cash flows and financial position, as well as on its reputation. ESG-related risks and opportunities that are very likely to have a serious negative impact on non-financial aspects in accordance with the CSR Directive Implementation Act (CSR-RUG) can be found in the respective categories of the Risk and Opportunity Report according to their cause. Furthermore, Daimler follows the recommendations of the Task Force on Climate-related Financial Disclosures (TCFD) with regard to climate-related risks and opportunities.

**Example C.40 (neutral).** In responding to Resolution 6.1 we have: z approved and published an Energy Policy consisting of financing of fossil-fuel-related activities, encompassing thermal coal, upstream oil and upstream gas, fossil-fuelled power generation, and renewable and embedded energy solutions.

**Example C.41 (neutral).** We also look at the way banks facilitate financing by others, for example by arranging the issue of green bonds. For each
of these categories we look at the bank's current
lending, historical trends and lending targets. There
are also 'no go projects'. We won't invest in any
bank which lends to an Adani Carmichael coal
mine. Internationally we won't invest in any bank
which lends to a Keystone XL pipeline transporting
oil from the tar sands of Canada.

**Example C.42** (risk). Our Emerging Risk Committee, constituted by our executive directors, discusses various risks that may affect our business, and includes environmental considerations such as climate change as a standing agenda item. In addition, Newton's Operating Committee is responsible for the management of how our business is run, and also considers relevant risks. Both committees ultimately report to the Executive Committee, with the outputs from the Emerging Risk Committee being reviewed at the Board Risk Committee.

C.6 Green Claim Detection

**Task Description** Given an input sentence or a paragraph, output a binary label, "green claim" or "not green claim".

**Experiment** We collected the datasets on *Environmental Claims* (Stammbach et al., 2023), and on *Green Claims* (Woloszyn et al., 2022) and reproduced the experiments with the setting described in section B and reported the results in Table 13. Our pre-processing pipeline altered the *Green Claims* dataset when removing URLs that are important in tweets.

**Results Analysis** As shown in Table 13, the best performing models are the fine-tuned transformers reaching F1-scores up to 91.2% on *Environmental Claims* and 97.2% on *Green Claims*. The performances of the zero-shot models are slightly worse by around 5%. Finally, the TF-IDF baseline underperformed the other approaches but still reached macro F1-scores above 80%.

Stammbach et al. (2023) reported a binary F1-score of 84.9%. In our experiments, GPT-4o-mini reached a binary F1-score of 81% and Distil-RoBERTa reached 87%.

Similarly, Woloszyn et al. (2022) reported a binary F1-score of 92.08%. In our experiments, GPT-4o-mini reached a binary F1-score of 89% and DistilRoBERTa reached 98%. When accounting for the confidence interval (94.84%-100%), our fine-tuned DistilRoBERTa outperformed their best model (fine-tuned RoBERTa). This might be explained by our cleaning process (e.g. removing

URLs) that removed noise from the original dataset,
but it could also be due to our split. (Woloszyn
et al., 2022) showed that fine-tuned models trained
on their dataset are prone to adversarial attacks,
which our cleaning process might have exacer-
bated.

C.6.1 Error analysis

**Environmental Claims** We found that most of the errors are FP (86%). Most of the errors sampled (9/14) are actual errors by GPT-4o-mini. There are only 4 FN. 3/4 FN are implicit claims. They are descriptions of environment-related actions (Example C.43) but they do not explicitly suggest that the company has a positive environmental impact. We also found 2 examples of implicit claims in the FP (Example C.44). *In the gold standard, implicit claims are labeled sometime positively and sometime negatively. This shows an ambiguity in the annotation guidelines for implicit or indirect claims.*

Most of the sampled FP (7/10) are statements
out-of-context that require the annotator to make
assumptions. Such statements are well-described in
the definition and should be classified as negatives,
which makes them actual errors from the model.

**Example C.43** (Environmental Claim). We have developed a Climate Change Strategy Roadmap outlining our climate change governance framework.

**Example C.44** (Not Environmental Claim). In terms of initiatives regarding energy issues, the AGC Group works to reduce the energy involved in its production activities.

**Example C.45** (Not Environmental Claim). This reduces KLP's climate risk and makes it clear that our work has an effect on the climate.

**Example C.46** (Not Environmental Claim). Obviously, it's without a doubt, the environment continues to improve, and that's all good.

**Green Claims** Due to the size of the dataset and the high model performance, we observed only 6 classification errors: 3 FN and 3 FP.

When reviewing the errors, we found that 2 FP
(e.g. Example C.47 with “Sea buckthorn extracts”)
and 1 FN (Example C.49 with “Coconut Oil”) fall
into the “natural claim” category - suggesting that a
product is better, healthier, or has properties thanks
to a natural ingredient. It is not clear how these
claims should be annotated. In the guidelines, they
<table><caption>Table 13: Performances of baselines for each dataset using the train, test described in section 3.3. The performance computed is the macro F1-score. The 95% confidence interval is computed using a 1000 bootstrap of sampled with replace as True of the size of the original test set. The reference performance is the best relevant performances reported in the original papers. 1. binary F1-score, r. RoBERTa</caption><thead><tr><th>Dataset</th><th>Random</th><th>TF-IDF</th><th>Longformer</th><th>DistilRoBERTa</th><th>GPT-4o-mini</th><th>Llama</th><th>Llama 70B</th><th>Reference</th></tr></thead><tbody><tr><td>Environmental Claims (Stammbach et al., 2023)</td><td>42.8(37.4-48.4)</td><td>80.5(74.8-85.5)</td><td>90.5(86.5-94.0)</td><td>91.2(86.9-94.6)</td><td>86.8(82.0-90.9)</td><td>81.4(76.0-86.1)</td><td>76.6(71.1-81.9)</td><td>84.9<sup>1,r</sup></td></tr><tr><td>Green Claims (Woloszyn et al., 2022)</td><td>46.1(34.9-56.7)</td><td>86.1(77.2-93.3)</td><td>94.5(88.7-98.7)</td><td>97.2(93.0-100.0)</td><td>91.5(83.9-97.4)</td><td>90.0(82.3-96.9)</td><td>86.8(78.9-93.5)</td><td>92.08<sup>1,r</sup></td></tr></tbody></table>

could be considered “nature-friendly messages that target the needs and desires of environmentally concerned stakeholders” but they could also be considered as “Claims that address issues not related to the environment, such as health or equality, [which] are not considered green claims”. (Woloszyn et al., 2022) reported that: “natural ingredients” and “naturally derived” were the most frequent bigrams in "Explicit Green Claims". We found that the tweets containing these bigrams are “natural claim”, therefore we considered that “natural claims” are part of “green claims”. This implies that the 2 FP are mislabeled examples, and the FN and error by GPT-4o-mini.

The 2 other FN, are out-of-context, generic statements that do not refer to any specific product (Example C.48). Whether such statements should be classified as “green claims” is debatable. If we consider that any communication on environmental topics is an attempt to portray the company in a more environmentally friendly light, then these statements could be interpreted as green claims. However, this approach is overly simplistic and risks categorizing any environmental reference as a green claim, which would be an over-extension of the definition.

The remaining FP represents an actual misclassification by the model.

**Example C.47 (Not Green).** He needs protection from harsh weather too. Sea buckthorn extracts fortify + invigorate for smooth, energized skin: <URL> <URL>

**Example C.48 (Green Claim).** Environmental solutions come in all shapes and sizes.

**Example C.49 (Green Claim).** TELL US: what do you love the most about #WholeBlends Smoothing with #Coconut Oil and #Cocoa Butter Extracts? #haircare #coconutoil <URL>

## C.7 Green Claim Characteristics

**Task Description** Given an input sentence or a paragraph labeled as green claim, output a more fine-grained characterization of the claim. This is

a multi-label classification task; the labels can be about the form (e.g. specificity) or the substance (e.g. action, targets, facts).

**Experiment** All the previously mentioned datasets are available. We collected them and reproduced the experiments with the setting described in section B and reported the results in Table 14. *Implicit/Explicit Green Claims* (Woloszyn et al., 2022) was impacted by our pre-processing pipeline when we removed URLs.

**Results Analysis** As shown in Table 14, we observed the same patterns for each dataset. The best-performing models are fine-tuned transformers reaching performances above 77%. The zero-shot approaches underperformed the fine-tuned models by 1 to 5%. The TF-IDF baseline performed significantly better than random, underperforming the best models by between 2 to 10% only. It is important to note that the difference in performance with the TF-IDF baseline is significant only for DistilRoBERTa on *Commitments And Actions* and for Longformer on *Net-zero/Reduction*. This suggests that the characteristics mentioned, both in terms of form and content, are expressed with a specific vocabulary that can be used as a decent predictor. On *Implicit/Explicit Green Claims* and *Net-zero/Reduction*, GPT-4o-mini reached similar performances as the fine-tuned models. On *Climate Specificity* and *ESGBERT action500* GPT-4o-mini underperformed both the TF-IDF baseline and the fine-tuned models. On *Commitments And Actions*, the fine-tuned models significantly outperform the zero-shot approaches.

The performance reached by fine-tuned models in our experiments is similar to those reached in the original studies.

### C.7.1 Error analysis

**Commitments And Actions** The majority of all errors are FP (77%). Another ambiguous aspect is the description of company philosophy/values/mindset and the description of governance structures. We found examples of philoso-
<table><caption>Table 14: Performances of baselines for each dataset using the train, test described in section 3.3. The performance computed is the macro F1-score. The 95% confidence interval is computed using a 1000 bootstrap of sampled with replace as True of the size of the original test set. The reference performance is the best relevant performances reported in the original papers. 1. binary F1-score, 2. average F1-score (macro/micro is not specified), 3. weighted average F1-score, c. climateBERT, r. RoBERTa</caption><thead><tr><th>Dataset</th><th>Random</th><th>TF-IDF</th><th>Longformer</th><th>DistilRoBERTa</th><th>GPT-4o-mini</th><th>Llama</th><th>Llama 70B</th><th>Reference</th></tr></thead><tbody><tr><td>Commitments&Actions(Bingler et al., 2023)</td><td>48.5(42.9-53.9)</td><td>72.7(67.4-78.2)</td><td>76.7(71.2-81.5)</td><td><strong>81.9(77.1-86.5)</strong></td><td>67.2(61.6-72.4)</td><td>64.6(59.4-69.7)</td><td>50.7(45.1-56.2)</td><td>81<sup>3,c</sup></td></tr><tr><td>Climate Specificity(Bingler et al., 2023)</td><td>48.9(43.6-54.3)</td><td>72.4(67.6-77.5)</td><td>77.3(72.4-81.9)</td><td>77.5(72.1-82.2)</td><td>71.6(66.6-76.3)</td><td>76.5(71.5-80.8)</td><td>68.0(62.8-73.2)</td><td>77<sup>3,c</sup></td></tr><tr><td>ESGBERT action500(Schimanski et al., 2023b)</td><td>44.4(29.9-58.5)</td><td>82.5(71.1-93.1)</td><td>85.9(75.4-95.8)</td><td>89.1(80.2-97.8)</td><td>76.0(62.2-87.4)</td><td>67.0(53.3-79.7)</td><td>63.0(47.7-76.1)</td><td>-</td></tr><tr><td>Implicit/Explicit Green Claims(Woloszyn et al., 2022)</td><td>37.2(26.3-46.1)</td><td>70.3(56.5-82.0)</td><td>63.0(48.8-74.9)</td><td>81.2(68.1-91.8)</td><td><strong>81.6(71.2-90.4)</strong></td><td>68.5(54.7-81.0)</td><td>75.7(63.4-86.0)</td><td>81.45<sup>2,r</sup></td></tr><tr><td>Net-Zero/Reduction(Schimanski et al., 2023a)</td><td>29.8(24.9-34.8)</td><td>94.8(92.2-97.1)</td><td>97.8(96.0-99.1)</td><td>97.7(95.8-99.3)</td><td>97.3(95.5-99.0)</td><td>93.7(90.7-96.2)</td><td>95.6(93.3-97.8)</td><td>98.7<sup>1,c</sup></td></tr></tbody></table>

phy with the positive and negative labels (Examples C.51, C.52) and examples of governance in both (Examples C.53, C.52). Those kinds of errors represent 3/10 sampled FN and 7/10 of FP. Indeed expressing company values could be considered as commitments toward these values or a simple description of them; similarly, describing a governance structure could be considered as an action (setting up the governance structure) or a simple description. This ambiguity on what constitutes a commitment and an action is also reflected in the moderate inter-annotator agreement among human annotators (Krippendorff's α of 0.5354).

Additionally, this task is not climate-specific, and therefore the guidelines include both “business or climate actions”. GPT-4o-mini understood the prompt as only “climate actions”. 2 FN are due to GPT-4o-mini being anchored to a word in the prompt. Using a larger model or updating the prompt might resolve this issue. There is also 1 FN that was mislabeled (Example C.54), as the text describes a process and does not contain an action or a commitment. The remaining FN (4/10) are actual errors.

Of the 3 remaining FP sampled, 2 are mislabeled, such as Example C.50, which describes an action about recycling but is labeled as “No” in the gold standard - and as an “action” by GPT-4o-mini.

**Example C.50 (No).** 1 Recycling: To meet the environmental policies of major countries, we operate a recycling process utilizing non-reusable dead batteries and scraps generated during battery production. We have a strategic cooperation relationship with major partners for realizing a closed-loop. Through this system, we extract raw materials of batteries such as nickel, cobalt, and lithium by crushing and dissolving battery waste or scraps.

**Example C.51 (Yes).** In our view, true sustainable investing cannot be achieved by simply voting a proxy, adding a director of sustainability or even divesting from an asset class. Because traditional

models of finance and investing often fail to appropriately integrate sustainability issues, we've had to build it into our thinking from the ground up. It requires integration across our products, across our product teams and across our entire organization.

**Example C.52 (No).** Climate risk has emerged as one of the top environmental risks for the Bank. This includes physical risks related to the chronic and acute physical impacts of climate change (e.g., shifts in climate norms, and extreme weather events such as hurricanes, wildfires and floods), and transition risks associated with the global transition to a low-carbon economy (e.g., climate-related policy actions and litigation claims, technological innovations, and shifts in supply and demand for certain commodities, products and services). Both physical and transition risks could result in strategic, credit, operational, legal, and reputational risks for the Bank and its clients in climate sensitive sectors. TD supports Canada's objectives to meet the goals of the Paris Agreement and recognizes the Bank's responsibility to contribute by integrating climate considerations across its business. The Bank continues to monitor industry and regulatory developments and assess the potential impacts of climate change and related risks on its operations, lending portfolios, investments, and businesses.

**Example C.53 (Yes).** Our sustainability efforts are underpinned by strong corporate governance, which we continue to reinforce in line with recommendations of the Malaysian Code on Corporate Governance 2017 (MCCG 2017). This saw us set up a new Board Risk Committee to further enhance our risk oversight, adding to the existing Board Audit Committee and Nomination and Remuneration Committee. The Board as a whole comprises four Independent Non-Executive Directors (INEDs), who make up 50% of its composition, two of whom are foreign Directors. On a related note, I represented the Group in signing an Integrity Pact with the Malaysian Anti-Corruption Commission
(MACC), underlining our commitment to observ-
ing integrity in all our dealings with stakeholders.

**Example C.54** (Yes). D+ This client does not (yet) meet Rabobank's sustainability policy on one or more points or has not responded adequately to key questions. Specific agreements are made about a possible solution and timelines are established. Once the customer meets the sustainability policy of Rabobank, it is classified in category A, B or C.

**Example C.55** (No). Governance structure A robust governance structure ensures timely and direct execution of programs that drive the achievement of our set goals for 2020 as well as of our new set of targets for 2025. The head of Sustainability is responsible for the development, coordination and execution of our sustainability strategy and reports to the member of the Executive Board responsible for Global Operations. He or she also leads the sustainability Sponsor Board, which is composed of senior representatives from Global Brands, Global Operations, Digital, Sales, Finance, Corporate Communication, and other relevant functions across the company. The Sponsor Board ensures cross-functional alignment, transparent end-to-end management and execution of agreed-upon sustainability goals within their functions. This includes reviewing and signing-off on policies as required. We also maintain a separate compliance function which is operated as the Social & Environmental Affairs Team (SEA) to evaluate supplier-facing social and environmental compliance performance and human rights impacts, reporting, through the General Counsel, to the CEO.

*This dataset would require guidelines that are more detailed and specific for multiple ambiguities (that are not clear neither in the guidelines nor in throughout the annotations):*

* Setting a governance structure or process an action ?

* Is describing an existing process (industrial, governance, ...) an action ?

* Is saying that the company want to implement sustainability across all their products an action/commitment ?

**ESGBERT's Actions500** GPT-4o-mini produced only 10 errors, 9 of which were FP. Since this dataset is not part of a publication, we lack access to detailed annotation guidelines, leaving certain

edge cases, such as statements about commitments,
company philosophy, or exploration of potential
solutions, open to interpretation. Notably, 13/19
of these errors involve cases that are debatable,
suggesting that clearer guidelines could improve
consistency.

**Climate Specificity** In GPT-4o-mini predictions for *Climate Specificity*, most of the errors are FP (93%). There are actually only 6 FN.

When investigating the FN, we found 2 examples of part of the legend of a figure such as Example C.56. In the guidelines, the legend “provides firm-specific detailed explanations to enable readers to better understand the overall information reported” and is therefore considered “specific”. We also found 2 examples which were mislabeled as the statement applies to the whole industry, which is considered “non-specific” in the guidelines. We classified Examples C.57 as out-of-context, as they mention "A similar approach could be used" which we argue is not specific. But more context could make it specific.

Out of the sampled FP, we found 4/10 debatable examples. *This is due to the granularity of the specificity*. A paragraph might contain a list of specific “Key Priorities”. However, as the priorities are strategic axes, they are not specific about actions or targets. For instance, Example C.58 is specific about using "red lines" but does not describe specifically what the "red lines" are. Similarly to *Commitments and Actions*, the statements about governance can be precise governance points - such as the role of the CEO in overseeing the climate-related issue - but not specific about actions or targets. This is the case for 2 of the sampled errors.

The dataset contains annotation for identification of “specific actions/targets specific to the company”. This task is particularly difficult, as it needs to be precisely defined and even with a precise definition, we believe that some statements remain ambiguous due to the granularity of the specificity. This is further confirmed by the low IAA (Krippendorff’s α of 0.1703)

**Example C.56** (specific). [1] Data calculated on the same reporting perimeter as 2018, excluding Abertis Group. [2] Data on the diesel consumed by the gensets in Chile in 2018 are not available. [3] Figure updated following a consolidation subsequent to the close of the Integrated Report 2018 on ETC data. [4] Data updated following a restatement of the income statement 2018. [5] Location-based
emissions.

**Example C.57** (specific). A similar approach could be used for allocating emissions in the fossil fuel electricity supply chain between coal miners, transporters and generators. We don't invest in fossil fuel companies, but those investors who do should account properly for their role in the production of dangerous emissions from burning fossil fuels.

**Example C.58** (not-specific). Sustainable strategy 'red lines' For our sustainable strategy range, we incorporate a series of proprietary 'red lines' in order to ensure the poorest-performing companies from an ESG perspective are not eligible for investment.

**Implicit/Explicit Green Claims** As the dataset is small, there are only 12 errors. As shown in figure 6, the most frequent error, is a statement without claims classified as an “implicit claim”.

<figure><img src="image_10.png" /><figcaption>Figure 6: Confusion matrix of GPT-4o-mini predictions on *Explicit/Implicit Green Claims* dataset.</figcaption></figure>

As discussed in Section C.6, it is not clear if green claims include “natural claim” - suggesting that a product is better, healthier, or has properties thanks to a natural ingredient. Among the 12 errors, 5 contain “natural claims” (Example C.60). We also found 2 errors that are debatable, and 2 annotation errors. For instance, the annotator classified Example C.59 as “not green”. They most likely understood "Irish Spring" as the season instead of the soap brand. On the contrary, GPT-4o-mini correctly classified it as “implicit green claim”. This shows that LLMs can have broader general knowledge than the original annotators. *The distinction between explicit and implicit is highly relevant, as it define straightforward cases from ambiguous cases which are harder to classify.* Despite those errors, overall both fine-tuned models and the zero-shot approaches reached good performances. This is

aligned with the good IAA reported in the original study (Woloszyn et al., 2022) (Krippendorff $\alpha$ = 0.8223).

**Example C.59** (not green). Irish Spring is green in more ways than one... <URL>

**Example C.60** (not green). Give your skin a mid-day refresh with the Yes To Grapefruit Unicorn Brightening Mist! It's naturally packed with vitamin C to help give your skin a boost of GLOW! Thanks, @POPSUGAR <URL>

**Example C.61** (green). A healthy lifestyle becomes ever more important for many of us - also within our beauty routine. Stay tuned for our new permanent hair color featuring natural ingredients such as soy protein, oat milk and argan oil! Available from February. #headsup #WhatsComingNext #ProductNews <URL>

**Net-zero/Reduction** On this dataset, the model failed only on 8 instances as shown in Figure 7. As the performances of models and TF-IDF baseline are really high, we also experimented with using only a few keywords. Using [“net”, “zero”, “neutrality”] for the “net-zero” class and [“reduction”, “reduce”] for the “reduction” class, we reached an F1-score of 78.03%.

<figure><img src="image_11.png" /><figcaption>Figure 7: Confusion matrix of GPT-4o-mini predictions on *Net-zero/Reduction* dataset.</figcaption></figure>

Among the errors, 1 is a simple statement classified as targets in the gold standard (Example C.62). Another example is labeled “net-zero” while dealing with a reduction of 95% (Example C.64), raising the question: is 95% reduction the same as net-zero? Finally, half of the errors are statements with both “reduction” and “net-zero” targets (Example C.63). Overall, we see that both fine-tuned models and the zero-shot approaches are proficient at identifying targets. However, we point out that without addressing the labeling error mentioned, we can't conclude on the performance differences. While
the task appears straightforward for models, annotators do not achieve perfect inter-annotator agreement, despite a high score (Cohen's κ = 0.931). This indicates that even seemingly simple tasks can still be subject to occasional disagreement among annotators.

**Example C.62** (net-zero). Based on our track record and plans up to 2030 we are confident that we are well-placed to make significant progress beyond 2030 and up to 2050. (Pg. 27)

**Example C.63** (reduction). Reduce GHG emissions (Scopes 1+2) (on a market basis*1) by 55% by FY2030 and to zero by FY2050 (compared to FY2017)

**Example C.64** (net-zero). Amsterdam aims for a 95% emmissions reduction by 2050 (reference year 1990). Unclear how this becomes climate neutrality though.

## C.8 Green Stance Detection

**Task Description** Given two input sentences or paragraphs, one labeled as the claim and one as the evidence, predict the stance between the two: supports, refutes or neutral. Some studies fix the claim and only vary the evidence (e.g. the claim is always *Climate change poses a severe threat*), training a model to predict the stance of the evidence in respect to the fixed claim. Other studies train a model to predict the relation between varying claims/evidences.

**Experiment** We collected *ClimateFEVER* (Diggelmann et al., 2020), *climateStance* (Vaid et al., 2022), *GWSD* (Luo et al., 2020) and *LobbyMap* (Morio and Manning, 2023) and reproduced the experiments with the setting described in section B and reported the results in Table 15. For *ClimateFEVER* the dataset is designed to predict the label of claim-evidence pair relation and then aggregate the predictions to infer if the claim is supported or not. Therefore, we split the tasks into 3. The first is to predict the claim label only, without the evidence (*ClimateFEVER claim (our split)*). The second task is to predict the claim-evidence label (*ClimateFEVER evidence (our split)*). Finally, the third task is given the prediction of the claim-evidence pair, predict the claim label (*ClimateFEVER claim (our split, aggregated)*). We also reported the performance for a train-test split similar to climabench(Spokoyny et al., 2023). For *Lobbymap*, we evaluated each

intermediary task (Page, Query, Stance). Our pre-processing impacted *ClimateFEVER* and *Lobbymap* as we resampled the datasets and removed longer texts. It also altered *climateStance*, as we removed URLs which are important in tweets.

**Results Analysis** When focusing on the results for *ClimateFEVER claim (our split, aggregated)*, *ClimateFEVER evidence (our split)*, *ClimateStance*, *GWSD* and *LobbyMap* datasets; our experiments results show performances below 75% for the best-performing models. This shows that those tasks are challenging for models. For most datasets, the best-performing approaches are the fine-tuned models. However, for *ClimateFEVER*, it is the zero-shot approach.

When using a split similar to Spokoyny et al. (2023) on *climateFEVER evidence (climabench split)*, distilRoBERTa reached an F1-score of 59.9% (comparable to theirs, with 62.7% for (Spokoyny et al., 2023)). However, as mentioned in Appendix A, the split provided in their benchmark contains partial contamination (the same claim is both in the train and test splits, but with different evidences). When using our split, the performance of the best-performing fine-tuned transformer drops to 52.7%. On *ClimateFEVER claim* - classify if the claim is supported by evidences - we first evaluated if the claim contains a predictive power in itself. Both fine-tuned models and zero-shot approaches struggle on this task. This is likely due to the way the labels are constructed: the claim label is aggregated on the claim-evidence labels. Therefore, the claim's label is dependent on the evidence collected previously and not on its inherent factuality. When using aggregated claim-evidence prediction to infer the claim's label, the performance increases to 41.3%. Still, this is far from the performances reported in other papers (Wang et al., 2021; Vaghefi et al., 2022; Xiang and Fujii, 2023; Webersinke et al., 2022; Vaid et al., 2022) reaching performances above 80%. However, they are not comparable as they removed claims with the "DISPUTED" label.

We achieved performance levels comparable to those reported in the original studies across all other datasets except for *LobbyMap*. This discrepancy can be attributed to the reduced size of our training dataset, which was limited to 10,000 samples. Given the complexity of the task, a larger training set or extended training duration could po-
<table><caption>Table 15: Performances of baselines for each dataset using the train, test described in section 3.3. The performance computed is the macro F1-score. The 95% confidence interval is computed using a 1000 bootstrap of sampled with replace as True of the size of the original test set. The reference performance is the best relevant performances reported in the original papers. *d*. Contaminated split, *SciBERT*, *h*. Human Baseline using annotations, *i*. Filtered Split (by removing disputed claims), *r*. *RoBERTa*, *t*. *BERT*</caption><thead><tr><th>Dataset</th><th>Random</th><th>TF-IDF</th><th>Longformer</th><th>DistilRoBERTa</th><th>GPT-4o-mini</th><th>Llama</th><th>Llama 70B</th><th>Reference</th></tr></thead><tbody><tr><td>climateF claim (our split)(Diggelmann et al., 2020)</td><td>15.4(10.4-20.7)</td><td>35.0(26.4-43.0)</td><td>31.9(24.7-38.9)</td><td>32.5(24.9-39.6)</td><td>19.3(13.2-25.0)</td><td>30.0(21.9-37.2)</td><td>26.3(19.8-32.8)</td><td>80.7<sup>f</sup></td></tr><tr><td>climateF claim (agg.) (Diggelmann et al., 2020)</td><td>-</td><td>-</td><td>42.5(34.2-49.9)</td><td>41.3(32.8-49.9)</td><td>50.4(42.0-58.2)</td><td>-</td><td>-</td><td>60.10<sup>f,h</sup></td></tr><tr><td>climateF claim (climabench split, agg.) (Diggelmann et al., 2020)</td><td>-</td><td>-</td><td>11.9 (10.7-12.8)</td><td>48.8 (45.3-52.4)</td><td>-</td><td>-</td><td>-</td><td>-</td></tr><tr><td>climateFEVER evidence (Diggelmann et al., 2020)</td><td>28.4(25.2-31.4)</td><td>46.2(42.2-50.2)</td><td>52.7(48.5-56.7)</td><td>51.3(47.3-55.1)</td><td>60.0(56.2-63.6)</td><td>52.5(48.4-55.9)</td><td>63.7(59.7-67.4)</td><td>68.03<sup>f,h</sup></td></tr><tr><td>climateFEVER evidence (climabench split)(Diggelmann et al., 2020)</td><td>30.4 (27.4-33.5)</td><td>57.0 (52.6-60.7)</td><td>26.3 (25.5-27.1)</td><td>59.9 (55.5-64.0)</td><td>-</td><td>-</td><td>-</td><td>62.68<sup>d</sup></td></tr><tr><td>climateFEVER evidence (climabench split)(Vaid et al., 2022)</td><td>21.3(17.7-25.1)</td><td>49.6(42.9-55.9)</td><td>56.6(49.4-63.0)</td><td>56.1(47.9-63.2)</td><td>56.4(50.4-62.2)</td><td>48.5(42.4-54.2)</td><td>58.0(51.2-63.8)</td><td>59.69<sup>f</sup></td></tr><tr><td>Global-Warming Stance (GWSD)(Luo et al., 2020)</td><td>29.5(23.0-36.0)</td><td>59.2(51.5-66.2)</td><td>69.3(62.2-75.8)</td><td>75.3(68.5-81.6)</td><td>69.8(63.1-76.3)</td><td>64.4(57.5-71.1)</td><td>72.5(65.8-78.8)</td><td>73<sup>f</sup></td></tr><tr><td>LobbyMap (Pages)(Morio and Manning, 2023)</td><td>46.3(45.5-47.1)</td><td>73.1(72.3-73.9)</td><td>71.3(70.4-72.1)</td><td>73.6(72.8-74.4)</td><td>63.2(62.4-64.0)</td><td>62.5(61.7-63.3)</td><td>60.5(59.6-61.3)</td><td>-</td></tr><tr><td>LobbyMap (Query)(Morio and Manning, 2023)</td><td>12.4(12.0-12.9)</td><td>49.3(47.3-50.9)</td><td>57.3(55.0-59.2)</td><td>52.1(50.0-54.2)</td><td>36.4(34.5-38.0)</td><td>27.6(26.0-29.2)</td><td>32.7(31.1-34.2)</td><td>-</td></tr><tr><td>LobbyMap (Stance)(Morio and Manning, 2023)</td><td>19.2(17.8-20.5)</td><td>43.6(41.7-45.2)</td><td>46.7(45.0-48.3)</td><td>44.7(42.8-46.5)</td><td>30.0(28.5-31.7)</td><td>26.9(25.4-28.4)</td><td>24.1(22.6-25.6)</td><td>-</td></tr></tbody></table>

tentially enhance model performance by providing more comprehensive coverage of the data distribution and enabling the model to learn more nuanced patterns.

### C.8.1 Error analysis

**ClimateFEVER** We will not investigate errors on the claim classification task as the performance is really poor and the task is not designed for this.

<figure><img src="image_12.png" /><figcaption>Figure 8: Confusion matrix of GPT-4o-mini predictions on *ClimateFEVER* (*evidence*) dataset.</figcaption></figure>

On the claim-evidence pair relation classification task, we firstly see that the model rarely mixes 'SUPPORT' and 'REFUTES' (less than 5% of errors) as reported in Figure 8. Then we see that GPT-4o-mini tend to predict "REFUTES" instead of "NOT ENOUGH INFO"; and "NOT ENOUGH INFO" instead of "SUPPORTS".

Upon examining the errors, we found that 12/30 are mislabeled in the gold standard. Most of these annotation mistakes (9/12) are evidence that can be classified as SUPPORT/REFUTES but were classified as NOT ENOUGH INFO. This shows that many examples are ambiguous as they are at the frontier of NOT ENOUGH INFO's label, and this is also aligned with the LLM's error pattern. We

found that 15/30 errors were genuine misclassifications by GPT-4o-mini, and three instances being potentially debatable. These errors are likely to be mitigated through prompt engineering, for instance, by emphasizing that the evidence was retrieved by a model with potential limitations. Consequently, although the evidence may appear related to the claim, additional verification is required to ensure its relevance. For example, in "He concluded", "He" is not necessarily the person mentioned in the claim (Example C.65).

As mentioned, we found some errors that are debatable, and Diggelmann et al. (2020) reported a low Cohen's κ = 0.334. This shows that the task is difficult even for humans, and there are numerous disagreements. As the individual annotations were provided in the dataset, we also computed the performances of each annotator. We found that the average macro F1-score on the claim-evidence task is 60%, ranging from 54% to 69%. The model is well within the range of the human annotator performances.

### Example C.65 (NOT ENOUGH INFO).

*Claim:* "When you read Phil Jones' actual words, you see he's saying there is a warming trend

*Evidence:* "From this, he concluded that The post-1980 global warming trend from surface thermometers is not credible."

Additionally, we sampled 10 examples labeled "SUPPORTS" predicted as "NOT ENOUGH INFO". All 10 are claims supporting climate change, and evidence indeed supports the claim. However, the evidence is not sufficient on its own (supporting only a part of the claim or indirectly supporting the claim). We also sampled 10 examples labeled "NOT ENOUGH INFO" predicted as "REFUTES". 8 claims deny climate change, and 2 are neutral. Except for 2, which are out of context, they all either refute or partly refute (refuting
part of the claim or indirectly refuting the claim). The stance can be interpreted as: "the evidence is sufficient to support/refute the claim" or "the evidence is aligned with the claim and can be part of an argument that supports/refutes the claim". The annotators seem to struggle with this distinction, while GPT-4o-mini tends to predict "REFUTES" for arguments refuting climate change denial arguments, and "NOT ENOUGH INFO" for arguments supporting climate change arguments.

**Climate Stance** This dataset is heavily imbalanced toward the “Favor” label. Therefore, most of the errors are on this label. However, as shown in Figure 9 the rate of error is not significantly higher.

<figure><img src="image_13.png" /><figcaption>Figure 9: Confusion matrix of GPT-4o-mini predictions on *Climate Stance* dataset.</figcaption></figure>

The definition of the stance provided by Vaid et al. (2022) contains some gray area. For example, the “Against” label includes: “opposition to climate change policies”. Therefore, a statement that includes acknowledging climate change, but denying the actions that are required to limit it, is difficult to classify (Example C.66).

In the sampled errors, 7/30 are statements implicitly expressing concerns about climate change, such as promoting an IPCC conference or making a joke (Example C.67) which are misclassified by GPT-4o-mini. The other sampled errors (23/30) are debatable or mislabeled. They are 11 statements out-of-context and could be interpreted (e.g. Example C.68), or 4 statements clearly mislabeled (e.g. Example C.66 labeled “Favor” in the dataset but correctly classified as “Against” by GPT-4o-mini).

Contrary to other datasets, there are many errors where the model predicts “Against” instead of “Favor”. Therefore, we investigated a sample of 10 errors of this type. We found 6 mislabeled examples (e.g. Example C.69), 1 tweet with an ambiguous stance, and 3 that are actual errors by GPT-4o-mini. This suggests that a lot of those errors are

actually annotation mistakes, most likely caused by the reference to personalities (e.g. “Christopher Monckton”, “Corbyn”) and implicit stance (e.g. ""carbon border tax" will damage global efforts to tackle #ClimateChange”) which are not trivial to understand.

**Example C.66** (Favor). My point is China uses it devolving status to get an easy ride - while Australia who is making reductions - continues down a silly track of higher energy prices for no real change in outcomes for this so called climate emergency

**Example C.67** (Favor). living through climate change, I know what it's like to be a townsperson in an RPG who goes, "there sure are a lot more wandering monsters than there used to be!"

**Example C.68** (Favor). But are solutions towards zero emissions as promising as they seem? This well-documented story by @johncarlosbaez contains some sobering numbers: <URL> via @NautilusMag #ClimateChange #ClimateCrisis 2/2

**Example C.69** (Favor). Power curve doesn't exist just like global warming

**GWSD** In this dataset the “disagrees” label is under-represented. As shown in the confusion matrix (Figure 10), most of the errors involve the label “neutral”. Confusions between “agrees” and “disagrees” represent less than 15% of the errors.

<figure><img src="image_14.png" /><figcaption>Figure 10: Confusion matrix of GPT-4o-mini predictions on GWSD dataset.</figcaption></figure>

With our prompt, the models often identified factual statements as non-opinionated, leading them to incorrectly assign the “neutral” label to statements that clearly conveyed agreement with the target opinion. In multiple examples (10), such as Example C.70, the model correctly recognized that the fact implied the “seriousness” of the situation. However, because it did not contain an explicit opinion, the model classified it as “neutral”. This
pattern accounts for 14 of the 30 sampled errors (with 10 correct reasonings). Interestingly, this type of answer (correct reasoning but wrong label) is inconsistent, in similar situation the model does answer with agrees/desagrees labels for implicit statements.

Additionally, we identified 7 instances of incorrect labeling in the original dataset. One such example, Example C.71), discusses the attention given to global warming, suggesting that it does not match the severity of the issue. While using a prompt that emphasizes implicit opinions did address some of these errors, it did not lead to overall performance improvements. Although it corrected the previously identified mistakes, it introduced new errors in the opposite direction. These newly introduced errors include misclassified examples, such as Example C.72, as well as statements that do not clearly convey the authors' opinions. This demonstrates the challenge of dealing with implicit statements, where the inherent ambiguity makes it difficult for the model to assign the correct label.

**Example C.70** (agrees). Global temperatures in 2014 shattered earlier records, making 2014 the hottest year since record-keeping began in 1880.

**Example C.71** (neutral). The stark truth is that severe weather events alone will not cause global warming to pop to the top of the national agenda.

**Example C.72** (neutral). This summer is seeing record lows for Arctic ice.

**LobbyMap** Firstly, it is important to note that this dataset is constructed based on LobbyMap.org. While LobbyMap.org is curated by experts, the site does not claim to be exhaustive. As a result, it is possible that the collected stances are not fully comprehensive either. For this dataset, the most important metric, therefore, is the recall.

<table><caption>Table 16: Detailed performances of GPT-4o-mini on the LobbyMap dataset different tasks.</caption><thead><tr><th></th><th>Precision</th><th>Recall</th><th>F1-score</th></tr></thead><tbody><tr><td><em>binary</em></td><td></td><td></td><td></td></tr><tr><td>Page</td><td>50%</td><td>94%</td><td>65%</td></tr><tr><td><em>macro</em></td><td></td><td></td><td></td></tr><tr><td>Query</td><td>34%</td><td>54%</td><td>40%</td></tr><tr><td>Stance</td><td>30%</td><td>29%</td><td>24%</td></tr><tr><td>Stance (relieved)</td><td>47%</td><td>46%</td><td>45%</td></tr></tbody></table>

As shown in Table 16, the recall is significantly higher for both the Query and Page tasks. This

is expected, as the dataset is non-exhaustive, leading to lower precision. However, this limitation does not impact the stance task. Since the page and policy query are predefined, the annotators from LobbyMap.org might have overlooked a page containing a stance on a policy. However, given a specific page and policy, there is only one correct stance to identify. In fact, on the Query task, the GPT-4o-mini outputs for each page on average 2.7 queries, while the original labels contain on average 1.6.

As this dataset contains 3 tasks, and that the texts are relatively long, we analyzed the Page and Query task simultaneously. We sampled 10 instances. For the Page task: 3 were TN (true negatives), 1 was a TP (true positive), 1 was a FP. The FP is due to lack of context (Example C.74). The 5 others are texts that actually contain one or more pieces of evidence about the stance of the company. For instance, Example C.73 was not classified as containing a stance in the original dataset; however, the statement contains evidence about the stance of the company on "Energy transition & zero carbon technologies" through the mention of electric vehicles.

For Stance task, it appears that GPT-4o-mini almost never output the “supporting” label. Using the relaxed labels from the original paper, we reach an F1-score of 41%, which is still very low. We also sampled 10 errors. Half of them are labels that are close (“Supporting” and “Strongly Supporting”). We also found 2 instances where the annotators assigned the “not supporting” instead of the “no position or mixed position” label to statements with no position about a policy. The stance is actually expressed in numerical value (between -2 and +2) on LobbyMap.org, but was translated in labels (e.g “supporting (+1)”) in LobbyMap (Stance). The rest of the errors are genuine mistakes.

**Example C.73.** We'll show how we've revolutionized and electrified some of the most popular, iconic vehicles, helping to shape the future of zero-emissions transportation [No position]

**Example C.74.** David Maxwell, CEO of east coast gas producer Cooper Energy, described the policy as "draconian" [No position]

In our benchmark, we measure F1-scores for each task individually; however, (Morio and Manning, 2023) proposed custom metrics that we reported in Table 17. We observe that the performances of the Most-Frequent and the Linear Model
<table><caption>Table 17: Performance comparison of various models across different evaluation metrics.</caption><thead><tr><th rowspan="2"></th><th rowspan="2">Model</th><th colspan="3">Document</th><th colspan="3">Page Overlap</th><th colspan="3">Strict</th></tr><tr><th>P</th><th>Q</th><th>S</th><th>P</th><th>Q</th><th>S</th><th>P</th><th>Q</th><th>S</th></tr></thead><tbody><tr><td rowspan="6">Morio and Manning (2023)</td><td>Most-frequent</td><td>46.7</td><td>52.6</td><td>36.8</td><td>51.8</td><td>25.6</td><td>19.8</td><td>41.2</td><td>19.6</td><td>17.5</td></tr><tr><td>Linear</td><td>66.0</td><td>61.9</td><td>50.3</td><td>71.4</td><td>44.5</td><td>36.1</td><td>52.0</td><td>31.2</td><td>27.0</td></tr><tr><td>BERT-base</td><td>71.0</td><td>63.5</td><td>51.6</td><td>73.6</td><td>48.1</td><td>37.2</td><td>50.2</td><td>31.9</td><td>25.8</td></tr><tr><td>ClimateBERT</td><td>71.8</td><td>64.0</td><td>52.8</td><td>74.4</td><td>48.9</td><td>39.0</td><td>50.2</td><td>32.2</td><td>26.8</td></tr><tr><td>RoBERTa-base</td><td>71.6</td><td>64.5</td><td>53.1</td><td>73.8</td><td>49.6</td><td>38.3</td><td>50.4</td><td>33.4</td><td>26.6</td></tr><tr><td>Longformer-base</td><td>73.7</td><td>66.9</td><td>54.6</td><td>75.9</td><td>53.0</td><td>40.8</td><td>52.5</td><td>36.1</td><td>28.6</td></tr><tr><td rowspan="4">Ours</td><td>Longformer-large</td><td>73.9</td><td>68.8</td><td>57.3</td><td>76.5</td><td>55.0</td><td>44.1</td><td>53.6</td><td>38.7</td><td>31.5</td></tr><tr><td>GPT-4o-mini</td><td>58.6</td><td>33.5</td><td>39.7</td><td>68.6</td><td>22.6</td><td>23.9</td><td>41.0</td><td>11.6</td><td>14.2</td></tr><tr><td>distilRoBERTa</td><td>61.2</td><td></td><td></td><td>-</td><td></td><td></td><td>39.23</td><td></td><td></td></tr><tr><td>TF-IDF</td><td>63.5</td><td>57.4</td><td>50.2</td><td>65.9</td><td>42.6</td><td>34.7</td><td>39.3</td><td>25.5</td><td>20.9</td></tr><tr><td></td><td>Most Frequent</td><td>46.7</td><td>52.6</td><td>36.8</td><td>52.0</td><td>25.7</td><td>19.8</td><td>41.2</td><td>19.6</td><td>17.5</td></tr></tbody></table>

were easily reproducible. However, our finetuned distilRoBERTa could not reach the same performances as Morio and Manning (2023)'s finetuned models. Moreover, GPT-4o-mini could not output better results than the TF-IDF/linear baseline.

## C.9 Question Answering

**Task Description** Given an input question and a set of resources (paragraphs or documents), output an answer to the question.

**Experiments** We collected ClimaQA and ClimaINS and reproduced the experiments with the setting described in Section B and reported the results in Table 18. Our pre-processing step heavily altered the datasets as they contained many problematic duplicates. We found that 10% ClimaQA's dataset are duplicates with mismatching labels (a given question and its answer would appear with both positive and negative labels). We had to rebuild the dataset from the original CDP questionnaires answered by the companies. This dataset is called ClimaQA (our split) in the evaluation. ClimaINS also contained many duplicates due to the companies answering similarly each year, and companies answering similarly to the parent-company (reducing the dataset size by 25% and avoiding contamination). Around 2.4% of the remaining answers are duplicates with mismatching labels. We clean the dataset by removing all problematic examples, we name this dataset ClimaINS (our split). We framed ClimaQA as a classification task, to fit the settings of our experiments.

**Results Analysis** Firstly, for ClimaINS, in the original paper the authors reported a performance of 84.4%, which we could reproduce with the same settings, even reaching 89.3% with DistilRoBERTa.

However, as described previously, we significantly altered the dataset, removing duplicates and train-test contamination. This resulted in lower performances from fine-tuned models, both Longformer and DistilRoBERTa, as shown in Table 18. With this split, all models are under-performing the TF-IDF baseline. This suggests that while each question induces a different vocabulary, understanding the responses actually makes it harder to identify the source question.

For ClimaQA, with the original split the models could not reach more than an F1-score of 50%. This was due to the duplicates with mismatching labels introducing noise. However, when re-constructing the dataset, the fine-tuned models can reach excellent performances, around 89% as shown in Table 18. Contrary to ClimaINS, the zero-shot approaches do not under-perform the TF-IDF baseline. Moreover, the gap with the fine-tuned models is smaller (around 9%).

In the original paper, the authors framed ClimaQA as a retrieval task, using models to rank the answers. The best model is MiniLM, reaching a MRR (Mean Reciprocal Ranking) of 0.755. We also computed the MRR of the distilRoBERTa fine-tuned during our classification experiment, as well as a random baseline and a distribution baseline in Table 19. The distribution baseline always ranks the question based on the frequency of the question in the training set. As reported in the table, the fine-tuned model reached a MRR of only 0.636. We believe that the performance might be improved by using a larger training dataset than ours (limited to 10,000 samples).
<table><caption>Table 18: Performances of baselines for each dataset using the train, test described in section 3.3. The performance computed is the macro F1-score. The 95% confidence interval is computed using a 1000 bootstrap of sampled with replace as True of the size of the original test set. The reference performance is the best relevant performances reported in the original papers. a. SVM</caption><thead><tr><th></th><th>Dataset</th><th>Random</th><th>TF-IDF</th><th>Longformer</th><th>DistilRoBERTa</th><th>GPT-4o-mini</th><th>Llama</th><th>Llama 70B</th><th>Reference</th></tr></thead><tbody><tr><td>ClimaINS(Spokoyny et al., 2023)</td><td></td><td>12.3 (10.6-14.0)</td><td>77.5 (75.3-79.5)</td><td>84.7 (82.9-86.4)</td><td>89.3 (55.6-100.0)</td><td>-</td><td>-</td><td>-</td><td>86.00<sup>a</sup></td></tr><tr><td>ClimaINS (our split)(Spokoyny et al., 2023)</td><td></td><td>12.8 (11.1-14.6)</td><td>81.2 (79.0-83.2)</td><td>77.6 (72.9-81.4)</td><td>75.8 (71.3-80.2)</td><td>58.5 (55.9-61.0)</td><td>45.4 (42.7-47.8)</td><td>48.4 (45.9-51.0)</td><td>-</td></tr><tr><td>climaQA (Spokoyny et al., 2023)</td><td></td><td>45.5 (44.9-46.1)</td><td>47.7 (47.1-48.3)</td><td>44.3 (44.1-44.4)</td><td>-</td><td>-</td><td>-</td><td>-</td><td>-</td></tr><tr><td>climaQA (our split)(Spokoyny et al., 2023)</td><td></td><td>50.5 (49.4-51.6)</td><td>50.3 (49.2-51.3)</td><td>89.5 (88.9-90.2)</td><td>89.4 (88.7-90.0)</td><td>78.8 (77.9-79.6)</td><td>57.4 (56.3-58.4)</td><td>76.1 (75.1-77.0)</td><td>-</td></tr></tbody></table>

<table><caption>Table 19: Performances on the task of ranking question based on the answer for ClimaQA (Spokoyny et al., 2023). random ranks randomly each question, distribution ranks the question from the most frequent to the least frequent based on the train dataset, distilRoBERTa use the outputs of distilRoBERTa for each question/answer pairs to rank the questions. Finally, ClimateBERT and MiniLM are the performances reported by Spokoyny et al. (2023).</caption><thead><tr><th></th><th>random</th><th>Ours<br>distribution</th><th>distilRoBERTa</th><th colspan="2">Spokoyny et al. (2023)</th></tr><tr><th></th><th></th><th></th><th></th><th>ClimateBERT</th><th>MiniLM</th></tr></thead><tbody><tr><th>MRR</th><td>0.106 (0.10 - 0.11)</td><td>0.343 (0.33 - 0.35)</td><td>0.636 (0.63, 0.65)</td><td>0.753</td><td>0.755</td></tr></tbody></table>

### C.9.1 Error Analysis

**ClimaINS** This dataset contains labels associated to each question, for example “EMISSION” corresponds to “Does the company have a plan to assess, reduce or mitigate its emissions in its operations or organizations? If yes, please summarize”. It is balanced, except for “EMISSIONS” being twice as frequent. As shown in Figure 11, the easiest label to identify is the “EMISSION” label, and the hardest are “ASSESS”, “MITIGATE” and “RISK PLAN”.

<figure><img src="image_15.png" /><figcaption>Figure 11: Confusion matrix of GPT-4o-mini predictions on ClimaINS dataset.</figcaption></figure>

In the 11 sampled errors, we found 6 instances that are cropped or out-of-context; therefore, they are harder to classify, as they provide fewer context-

tual hints about the original question. Additionally, the average length of text misclassified is shorter than the length of correctly labeled text (respectively 635 and 1091 characters), and more than 54% of text misclassified are shorter than 300 characters compared to only 25% of the correctly classified. This shows that this task is significantly harder with less context and details. We also found one statement referring to its parent company, making it impossible to predict, as well as three statements that are partly or fully off-topic (Example C.75). Overall, we found that most of the sampled errors could answer multiple questions. The dataset is derived from a mandatory survey, which may have led to varying levels of response quality from the companies. Some companies provided uniform responses across all their subsidiaries, while others submitted minimalistic answers, making it difficult to discern which specific questions were addressed.

**Example C.75** (MITIGATE: Summarize steps the company has taken to encourage policyholders to reduce the losses caused by climate change-influenced events.). However, Lincoln Financial has adopted electronic document delivery and document delivery suppression initiatives to reduce paper and energy usage.

**ClimaQA** Instead of using each question as a label, as in ClimaINS, in this dataset is a binary classification: given a response and a question, the model predicts if the response answers the question. Therefore, we reported the F1-scores per question in Figure 12. We see that Q29 to Q38 have F1-scores of either 0% or 1%. This is due to the small size of the subset (less than 5 instances). When fo-
cusing on larger subsets, we see that most questions have F1-scores above 60%, with the exception of Q13 and Q21. For Q13, the recall is around 25%, and for Q21 the recall is 0%. This is because those questions are about “other targets” which make these questions more open, resulting in varied content. For instance, in Example C.76 the company provided a target about waste management, which is indirectly related to climate-change.

<figure><img src="image_16.png" /><figcaption>Figure 12: F1-scores (macro) of GPT-4o-mini on CLIMAQA dataset, grouped by question. In blue: The F1-scores. In orange: the size of the subset.</figcaption></figure>

70% of FN are questions that actually contain sub-questions. For example, Q3: "Which of the following risk types are considered in your organization's climate-related risk assessments?" has sub-questions specific to "Market Risk", "Chronic physical Risk", etc. When the company answers that a risk is considered, they provide a text specific to that sub-category. The response in Example C.77 was answering the sub-categories: "Acute physical". However, given the question Q3, one might expect the answer to list all the risks considered, prompting them to classify it as "No (not answering the question)" - as did GPT-4o-mini. For this kind of question, we should construct the dataset differently in order to emphasize that the answer is about a sub-category and often a justification of a value previously reported.

**Example C.76.** [Yes]

Q13: "Provide details of other key climate-related targets not already reported in question C4.1/a/b."
Answer: "We are targeting a 65% waste diversion rate by 2022 associated in our real estate portfolio."

**Example C.77.** [Yes]

Q3: "Which of the following risk types are considered in your organization's climate-related risk

assessments? (Current regulation, Market, Acute physical, Upstream, Emerging regulation, Technology, Legal, Reputation, Downstream, Chronic physical)

Answer: "[...] At this point, it is difficult to predict and assess the probability of potential risks related to a global warming trend on Dow specifically. Concerns have been raised that climate change may result in sea level rise or average temperature rise. Dow has operations and supply chains in coastal areas that potentially could be affected by these. To mitigate risks associated with severe weather, Dow has engineered the facilities to better withstand climate related events."

## C.10 Classification of deceptive techniques:

**Task Description** The goal is to classify statements into argumentative categories: fallacies, types of arguments, or rhetorical techniques.

**Experiment** We collected *CC-Contrarian Claims* and *LogicClimate* and reproduced the experiments with the setting described in section B and reported the results in Table 20. Our pre-processing steps did not alter the datasets significantly.

**Result Analysis** As shown in Table 20, the performances on *LogicClimate* and on *CC-Contrarian Claims* datasets are widely different. The performances on *CC-Contrarian Claims* for fine-tuned models reach F1-scores of up to 71%. On the contrary, for *LogicClimate* the best-performing approaches are the zero-shot models reaching only 27%. In both cases we reached performances in the same order of magnitude as in the original studies, but our approaches underperformed those reported in the original studies.

On *LogicClimate*, our fine-tuned models performed poorly (F1-score between 9-15%), underperforming the random classifier. In the original study, they reported an F1-score of 29.37%. This can be explained by the fact that we only used the domain-specific dataset. Its small size, combined with the label diversity and difficulty, makes this dataset alone less suitable for fine-tuning. Jin et al. (2022) actually fine-tuned the model on general fallacy detection first. For the label difficulty, our TF-IDF baseline performed similarly to the random classifier (13%) suggesting that the labels are not distinguishable through vocabulary only. All this can highlight that the dataset is really challenging, or it could also indicate that there is an issue in
<table><caption>Table 20: Performances of baselines for each dataset using the train, test described in section 3.3. The performance computed is the macro F1-score. The 95% confidence interval is computed using a 1000 bootstrap of sampled with replace as True of the size of the original test set. The reference performance is the best relevant performances reported in the original papers. *j*. RoBERTa + Logistic Regression, *k*. Electra</caption><thead><tr><th>Dataset</th><th>Random</th><th>TF-IDF</th><th>Longformer</th><th>DistilRoBERTa</th><th>GPT-4o-mini</th><th>Llama</th><th>Llama 70B</th><th>Reference</th></tr></thead><tbody><tr><td>CC-Contrarian Claims(Coan et al., 2021)</td><td>3.2(2.6-4.0)</td><td>62.7(60.1-65.1)</td><td>71.6(68.5-73.9)</td><td>71.2(68.6-73.6)</td><td>58.8(55.5-61.3)</td><td>40.2(37.0-43.2)</td><td>55.8(52.6-58.4)</td><td>79<sup>j</sup></td></tr><tr><td>logicClimate(Jin et al., 2022)</td><td>13.3(11.4-15.3)</td><td>13.3(8.1-18.0)</td><td>15.6(9.6-20.9)</td><td>9.4(6.0-12.4)</td><td>27.4(15.8-31.4)</td><td>10.3(5.4-14.0)</td><td>22.1(15.3-28.1)</td><td>29.37<sup>k</sup></td></tr></tbody></table>

the dataset itself. Fallacy detection is highly subjective and IAA agreements are often quite low in that context (Helwe et al., 2024) making them difficult to annotate. Their best-performing models, which can reach 60% on general-domain fallacy detection, only reached 29% on this domain-specific dataset. Our zero-shot approaches reached similar performances (macro F1-score of 27% and micro F1-score of 26% for GPT-4o-mini). The performance gap can also be explained by the source. Indeed, the climate fallacies are taken from real arguments, extracted from a fact-checking website, which are therefore context-dependent and not necessarily explicit, compared to general fallacies which are toy examples extracted from quizzes.

On the contrary, on *CC-Contrarian Claims*, the fine-tuned transformers performed well reaching performances above 70%. This is lower than the performances reported in the original study (79%). This performance gap might be due to the downsizing of the training dataset to 10,000 examples. As there are 18 labels, it resulted in only around 500 examples per label (compared to 1300 originally) and 4 labels appeared less than 200 times in our training dataset. Our TF-IDF baseline reached an F1-score of 62.7%, which is extremely high when considering that there are 18 labels. This shows that the vocabulary used in each argument category is quite specific. In a pilot study, Coan et al. (2021) measured a really low IAA, $κ = 0.19$, showing that the task of classifying in a large number of categories might be really difficult for humans. They limited the disagreement by using a decision tree for annotations. They also computed the accuracy of the human annotators on a subset of the dataset for the classification of super-claims. They found an average accuracy above 95% for 4 categories, 86% for the super-claim “Climate movement/science is unreliable”, and only 50% for “No Claim”. This shows that while annotators disagree on the fine-grained categories, they can easily distinguish between broader categories. We also found that most of the errors of GPT-4o-mini occur between sub-claims of the same category (as shown in Figure

13). Both the models and the annotators struggle to classify an argument as “No Claim”. Interestingly, GPT-4o-mini and humans are not biased in the same manner. The model tends to over-classify as “No Claim”, while humans over-classified as “Contrarian Claims”.

### C.10.1 Error analysis

**CC-Contrarian Claims** As shown in figure 13 most of the errors are GPT-4o-mini predicting "No Claim" (42.45%), or errors between the claim types of the same category (31.60%). Only 25.94% of errors are unrelated.

While investigating the errors, we found that 9/30 of statements are multi-faceted, and could fit into multiple categories. Usually, one category is more dominant. Example C.79 could fit in the category of “CO2 is not a pollutant”, but the core argument is “we see no impact of higher CO2 levels on species”. They could also fit into multiple categories because labels are closely linked such as “Weather is cold/snowing” and “Ice/permafrost/snow cover isn’t melting”. In our sample (30 errors), we also found 4 FN which would require more context, such as Example C.78 which could be a contrarian argument by a large emitter advocating against carbon tax; however, it could also be a statement by an activist advocating for stricter regulations. It is not possible to choose without more context. We found 11 texts contain no explicit climate change contrarian claim, however, the arguments can lead to a contrarian claim. For instance, Example C.80 might be part of an unrelated scientific publication, but it could be used in a climate-denial rhetoric to argue that CO2 is actually beneficial.

In 4 cases the model provide the correct reasoning, but decided to use the “No Claim” label anyway. *Overall the labels are really detailed and well defined, but due to the granularity the labels are really close together.* While we can use the *Super-claims labels for evaluation instead of the Sub-Claims*, this task could also be a multi-label classification tasks in order to keep the nuanced
<figure><img src="image_17.png" /><figcaption>Figure 13: Confusion matrix of GPT-4o-mini predictions on *CC-Contrarian Claims* dataset.</figcaption></figure>

labels.

**Example C.78 (No Claim).** While a price on GHG emissions would encourage some helpful kinds of innovation, it will not generate other kinds. Coping with climate change will require major breakthroughs in basic science. Such breakthroughs are often elusive, and seeking them is an inherently high-risk venture. The private sector finds it difficult to capture the economic rewards of funding basic science, and placing a price on GHG emissions will not correct this bias. As a result, the private sector usually does not make large, sustained investments in basic science; yet that kind of investment is the key to achieving the breakthroughs needed in climate policy.

**Example C.79 (Species/plants/reefs aren't showing climate impacts/are benefiting from climate change).** The ocean acidification story depends

only on a chemical hypothesis whereas biological factors can overcome this and create conditions that allow calcification to continue. This is corroborated by the historical record of millions of years of success in much higher CO2 environments.

**Example C.80 (CO2 is beneficial/not a pollutant).** Although elevated CO2 did not significantly increase total seedling biomass, it did increase it by 14% when averaged across all temperature and fertilization regimes. However, elevated temperature did significantly increase seedling biomass by 55%, when averaged across all CO2 and fertilization treatments, as did fertilization, by 157%, when averaging across all CO2 and temperature levels. With respect to root exudation, a similar pattern emerged. Elevated CO2 did not significantly increase total dissolved organic carbon compounds exuded from seedling roots over a 24-hour period, yet plants