Benchmarking the Benchmarks: Reproducing Climate-Related NLP Tasks
Authors: Tom Calamai, Oana Balalau, Fabian M. Suchanek
Accepted at: acl25

Benchmarking the Benchmarks: Reproducing Climate-Related NLP Tasks

Tom Calamai
Amundi,
Télécom Paris, Inria, Institut Polytechnique de Paris,
France
tom.calamai@amundi.com

Oana Balalau
Inria,
Institut Polytechnique de Paris, Institut Polytechnique de Paris,
France
oana.balalau@inria.fr

Fabian M. Suchanek
Télécom Paris,
France
suchanek@telecom-paris.fr

Abstract
Significant efforts have been made in the NLP community to facilitate the automatic analysis of climate-related corpora by tasks such as climate-related topic detection, climate risk classification, question answering over climate topics, and many more. In this work, we perform a reproducibility study on 8 tasks and 29 datasets, testing 6 models. We find that many tasks rely heavily on surface-level keyword patterns rather than deeper semantic or contextual understanding. Moreover, we find that 96% of the datasets contain annotation issues, with 16.6% of the sampled wrong predictions of a zero-shot classifier being actually clear annotation mistakes, and 38.8% being ambiguous examples. These results call into question the reliability of current benchmarks to meaningfully compare models and highlight the need for improved annotation practices. We conclude by outlining actionable recommendations to enhance dataset quality and evaluation robustness.

1 Introduction
As climate change becomes a more and more urgent problem (The World Meteorological Organization (WMO)), there has been a surge of interest in the field of climate-related natural language processing (NLP). Over 60 recent works study tasks such as detecting climate-related discourse, identifying green claims in corporate communications, answering climate-related questions, and detecting deceptive patterns in climate-related communication (Calamai et al., 2025). And yet, the progress in this area is hard to gauge, for several reasons: First, the approaches are evaluated on different datasets even if they treat the same tasks; second, the evaluations differ in their metrics, which makes the results incomparable; third, the approaches are rarely compared to simple TF-IDF baselines, so that it is unclear to what degree the proposed problems can be solved by simpler methods as well; and finally, few of the works use zero-shot LLMs as competitors, so that it is hard to judge to what degree the proposed methods have become obsolete with the arrival of generative models.

In this paper, we set out to make a transversal and unified comparison of climate-related NLP tasks. We compiled 29 climate-related NLP datasets and evaluated a suite of baselines that cover traditional machine learning approaches such as TF-IDF with logistic regression, as well as fine-tuned Transformer models, and open and closed source LLMs in a standardized setting – effectively building a benchmark. We also performed an in-depth error analysis, with more than 500 manually annotated errors.

Our works yields the following insights:
(1) TF-IDF, finetuned models, and zero-shot LLMs perform similarly, with finetuned models performing slightly better. While this confirms that current models are capable of solving these tasks, it also questions the difficulty of the tasks. If TF-IDF performs well, as we systematically show, then this means that the datasets consist of simple examples that can be predicted using word-frequency only.
(2) 96% of our datasets contain annotation issues, with 16.6% of the sampled wrong predictions of a zero-shot classifier being actually clear annotation mistakes, and 38.8% being ambiguous examples where multiple labels could fit or where it is not clear which label to assign.
(3) Our manual analysis shows that finetuned models may perform better not because they generalize better, but because they overfit on annotation noise, bias, or unclear guidelines.

These findings highlight the limitations of the datasets to meaningfully compare models, because their simplicity and inconsistencies compress performance into a narrow band, masking real differences in model capability.

We believe our findings motivate the need for
more challenging datasets, with examples that go beyond keywords, require reasoning, and come with clearer, more robust annotation schemes. We propose recommendations for construction such datasets at the end of our paper.

We make all cleaned-up datasets, baselines, and a Python library to run comparisons publicly available at https://github.com/tcalamai/acl_climateNLPToolbox.

2 Related Work

Climate-related Benchmarks. To the best of our knowledge, only three benchmarks have been proposed for climate-related tasks: ClimaBench (Spokoyny et al., 2023) contains the existing datasets climateStance, climatext, climateEng, climateFEVER, SciDCC, and introduces 3 additional datasets, ClimaINS, ClimaQA, and ClimaTOPIC. The authors evaluate multiple fine-tuned transformers and simpler baselines on ClimaBench. ClimateGPT (Thulke et al., 2024) is a benchmark that consists of ClimaBench, Pira 2.0 MCQ (Pirozelli et al., 2023) and CC-Contrarian Claims. Several LLMs were evaluated in a few-shot setting. Finally, ClimRetrieve (Schimanski et al., 2024b) is a benchmark restricted to climate-related information retrieval. We improve upon these efforts as follows:
1. Our dataset collection is significantly larger than previous ones, with 19 more datasets than ClimaBench and 17 more datasets than ClimateGPT
2. We evaluate simple baselines, fine tuned transformers, and recent LLMs in zero-shot settings, thereby contextualizing all performances
3. We conduct an in-depth analysis of the errors

Model Variability and Annotation Errors. Reproducibility and significance of results is a major issue in the scientific literature, especially in the machine learning community (Bouthillier et al., 2021; Gundersen et al., 2022; Ruffinelli et al., 2020). One challenge is that not all papers share the code, the experimental settings, the datasets created and the dataset annotation details. This issue has recently gained a lot of attention in the community, and conferences guidelines now often explicitly ask authors for these elements. Another source of irreproducibility comes from the machine learning algorithms themselves, and the design choices that can be made. Gundersen et al. (2022) identified 41 design choices that can affect reproducibly, among which: hyperparameter tuning that is costly and is often done manually (Bouthillier and Varoquaux, 2020); dataset issues such as preprocessing, data splits and annotation quality; a lack of sufficient baselines; and a lack of confidence intervals for results. Annotation quality is another major issue in the machine learning community, and solutions such as automatic detection of erroneous labels (Klie et al., 2023a) and datasheets for datasets (Gebru et al., 2021) have been proposed. In this work, we apply these considerations to climate-related NLP tasks.

3 Tasks, Datasets, and Models

3.1 Tasks

In previous work (Calamai et al., 2025), we have identified the following climate-related NLP tasks:

Task 1. Climate-Related Topic Detection: Given an input sentence or a paragraph, output a binary label, “climate-related” or “not climate-related”.

Task 2. Thematic Analysis: Given an input sentence or a paragraph, output a subtopic related to climate change. The categories can be the four categories of the Task Force on Climate-related Financial Disclosures (TCFD), the categories of Environment, Social and Governance (ESG), or custom-made categories.

Task 3. Climate Risk Classification: Given an input sentence or a paragraph, output the label “opportunity” (if the input talks about a positive effect of climate-change for the company) or “risk” (if the input talks about a negative effect). Some works focus only on risks, classifying them into types of risks (e.g., physical risk, reputational risk, regulatory risk, or transition risk).

Task 4. Green Claim Detection: Given an input sentence or a paragraph, output a binary label, “green claim” or “not green claim”. Green claims refer to the practice of suggesting or otherwise creating the impression that a product or a service is environmentally friendly (Stammbach et al., 2023).

Task 5. Green Claim Characteristics: Given an input sentence or a paragraph labeled as a green claim, output a more fine-grained characterization of the claim. This is a multi-label classification task; the labels can be about the form (e.g. specificity) or the substance (e.g. action, targets, facts).

Task 6. Green Stance Detection: Given two input sentences or paragraphs, one labeled as the claim and one as the evidence, predict if the evidence
supports the claim, refutes the claim, or is neutral towards the claim. Some studies fix the claim (e.g., the claim is always “Climate change poses a severe threat”) and aim to predict if the evidence supports or refutes that claim.

Task 7. Climate-Related Question Answering. Given an input question and a set of resources (paragraphs or documents), produce an answer to the question.

Task 8. Classification of Deceptive Techniques: Given a statement, classify it into argumentative categories, such as fallacies, types of arguments, or rhetorical techniques.

3.2 Datasets

For each of the tasks, our previous work (Calamai et al., 2025) listed all related datasets. For this work, We collected all those datasets that are openly available to reproduce the original results. Table 1 shows all these datasets with their description from Calamai et al. (2025). These datasets suffered from several issues, and hence we subjected them to a data cleaning pipeline. We provide here an overview of the issues we encountered, and we give the detailed statistics of each cleaning step in Appendix A.

Duplicate Removal. We identified many duplicates in the datasets, some of which became visible only after correcting formatting issues such as differing numbers of spaces. We differentiate between exact duplicates (same text and same label) and those with conflicting labels. Exact duplicates were removed to keep only one instance. For duplicates with conflicting labels, when possible we reconstructed the dataset to avoid them (e.g., Spokoyny et al. (2023) provide the source questionnaires for ClimaQA and ClimaINS); otherwise, we removed them. Some duplicates were intrinsic to the task, in which case we did not remove them (e.g. ClimateFEVER evidence contains pairs of claim-evidence; the same evidence could be found for multiple claims).

Noisy text. We investigated the text quality of datasets using a gibberish detection model1 and a language detection model2. We found that some datasets have noisy text samples or non-English text, but in small proportion (<0.5%). Moreover, noisy text is usually labeled with the negative label. For example, for detecting “climate-related” vs “not climate-related”, the noisy text had the label “not climate-related”. The few positively labeled noisy texts were mostly false positives – they were actually not noisy. Finally, even noisy examples are part of real-world texts. We therefore did not remove noisy samples.

Input text length. BERT-like models usually have a limited context window (e.g. 4096 tokens for Longformer). In multiple datasets, we found some excessively long texts (text with more than 4000 tokens), often due to formatting issues (e.g. spaces between all characters). To address this, we applied a text cleaning step to remove formatting and encoding issues using the clean-text3 Python library. Some datasets still contained very long texts, so we removed text longer than 4000 tokens to fit within Longformer’s context window. This impacted only 6 datasets and removed less than 0.3% of the data per dataset. We also identified very short texts resulting from parsing errors in PDF documents. These were typically page numbers, escape sequences, or table fragments. To remove these spurious occurrences, we removed all texts with fewer than 5 tokens.

Dataset size. After data cleaning, 7 of our 29 datasets still contained a large number of samples, ranging from 500 to over 150K. To reduce computational costs and experiment duration, we limited the training and development splits to 10,000 samples, while keeping the original size of the test splits intact. During down-sampling, we ensured that label distribution was preserved through stratification. For heavily imbalanced datasets, we adjusted the sampling to improve the balance.

Dataset splits. We split each dataset into 80% train, 10% test and 10% development. We kept the original splits when they were available in these proportions. However, some existing splits exhibited train-test contamination, in which case we recreated the splits.

3.3 Models and Measures

Models. We evaluated a range of models on each dataset: random baselines, a traditional approach (TF-IDF with logistic regression), fine-tuned transformers (distilRoBERTa, Longformer), and large

1https://huggingface.co/madhurjindal/autonlp-Gibberish-Detector-492513457
2https://huggingface.co/papluca/xlm-roberta-base-language-detection
3https://github.com/prasanthg3/cleantext
<table>
  <tr>
    <th>Dataset</th>
    <th>Input</th>
    <th>Labels</th>
  </tr>
  <tr>
    <th colspan="3">Climate-Related Topic Detection</th>
  </tr>
  <tr>
    <td>ClimateBug-data, Yu et al. (2024)</td>
    <td>sentences from Banks’ reports</td>
    <td><i>relevant/irrelevant:</i> Climate change and sustainability (including ESG, SDGs related to the environment, recycling and more)</td>
  </tr>
  <tr>
    <td>ClimateBERT’s climate detection, Bingler et al. (2023)</td>
    <td>paragraphs from reports</td>
    <td><i>I/O:</i> Climate policy, climate change or an environmental topic</td>
  </tr>
  <tr>
    <td>ClimateX (Wikipedia, 10-K, Clima), Varni et al. (2020)</td>
    <td>sentences from Wikipedia, 10-Ks or web scraping</td>
    <td><i>I/O:</i> Directly related to climate-change</td>
  </tr>
  <tr>
    <td>ClimateX (wiki-doc), Varni et al. (2020)</td>
    <td>sentences from a Wikipedia page</td>
    <td><i>I/O:</i> Extracted from a Wikipedia page related to climate-change</td>
  </tr>
  <tr>
    <td>Sustainable signals’s reviews, Lin et al. (2023)</td>
    <td>online product reviews (user comments)</td>
    <td><i>relevant/irrelevant:</i> Contains terms related to sustainability</td>
  </tr>
  <tr>
    <th colspan="3">Thematic Analysis</th>
  </tr>
  <tr>
    <td>TCFD rec., Bingler et al. (2021)</td>
    <td>paragraphs from corporate annual reports</td>
    <td><i>Metrics and Targets, Risk Management, Strategy, Governance and General:</i> TCFD 4 main categories.<br><i>Environment, Social, Governance and None:</i> Environmental criteria comprise a company’s energy use, waste management, pollution, as well as compliance with governmental regulations. Special areas of interest are climate change and environmental sustainability.<br><i>General, Nature, Biodiversity, Forest, Water:</i> Multi-label Nature-related topics</td>
  </tr>
  <tr>
    <td>ESGBERT’s ESG, Schimanski et al. (2023b)</td>
    <td>Sentences from reports and corporate news</td>
    <td><i>Environment, Social, Governance and None:</i> Environmental criteria comprise a company’s energy use, waste management, pollution, as well as compliance with governmental regulations. Special areas of interest are climate change and environmental sustainability.<br><i>General, Nature, Biodiversity, Forest, Water:</i> Multi-label Nature-related topics</td>
  </tr>
  <tr>
    <td>ESGBERT’s Nature, Schimanski et al. (2024a)</td>
    <td>Paragraphs from reports</td>
    <td><i>Environment, Geology, Animals, Ozone Layer, Climate, etc.:</i> Category in which the article was published (Automatic Label)</td>
  </tr>
  <tr>
    <td>SciCDC, Mishra and Mittal (2021)</td>
    <td>News articles (Title, Summary, Body)</td>
    <td><i>Ocean/Water, Politics, Disaster, Agriculture/Forestry, General:</i> Sub-categories of climate-change<br><i>Adaptation, Buildings, Climate Hazards, Emissions, Water, etc.:</i> Category of the question (Automatic Label)</td>
  </tr>
  <tr>
    <td>ClimateEng, Vaid et al. (2022)</td>
    <td>Tweets posted during COP25 filtered by keywords (relevant to climate-change)</td>
    <td></td>
  </tr>
  <tr>
    <td>ClimaTOPIC, Spokoyny et al. (2023)</td>
    <td>CDP responses (short texts)</td>
    <td></td>
  </tr>
  <tr>
    <th colspan="3">Climate Risk Classification</th>
  </tr>
  <tr>
    <td>ClimateBERT’s Sentiment, Bingler et al. (2023)</td>
    <td>Paragraphs from companies’ annual reports</td>
    <td><i>Risk, Opportunity, Neutral:</i> Risk or threat that negatively impacts an entity of interest (negative sentiment); or Opportunity arising due to climate change (positive sentiment); Neutral otherwise.</td>
  </tr>
  <tr>
    <th colspan="3">Green Claim Detection</th>
  </tr>
  <tr>
    <td>Green Claims, Woloszyn et al. (2022)</td>
    <td>Marketing Tweets</td>
    <td><i>Green Claim/Not Green:</i> Environmental (or green) advertisements refer to all appeals that include ecological, environmental sustainability, or nature-friendly messages that target the needs and desires of environmentally conscious stakeholders.<br><i>Yes/No:</i> A green claim is one that presents something interesting or otherwise creating the impression [...] that a product or a service is environmentally friendly (i.e., it has a positive impact on the environment) or is less damaging to the environment than competing goods or services [...] In our case, claims relate to products, services, or specific corporate environmental performance.</td>
  </tr>
  <tr>
    <td>Environmental Claims, Stammbach et al. (2023)</td>
    <td>Paragraph from reports</td>
    <td></td>
  </tr>
  <tr>
    <th colspan="3">Green Claim Characteristics</th>
  </tr>
  <tr>
    <td>Implicit/Explicit Green Claims, Woloszyn et al. (2022)</td>
    <td>Marketing Tweets</td>
    <td><i>Implicit green claims</i> raise the same ecological and environmental concerns as <i>explicit green claims</i>, but do not explicitly state them, but rather rely on any commitment from the company. If the tweet does not contain a green claim then <i>No Claim</i>.<br><i>Specific, Non-specific:</i> A paragraph is <i>Specific</i> if it includes clear, tangible, and firm-specific details about events, goals, actions, or explanations that directly impact or clarify the firm’s operations, strategy, or objectives. <i>Non-specific</i> otherwise.<br><i>Yes/No:</i> A paragraph is a commitment or an action if it contains targets for the future or actual achievements in the past.</td>
  </tr>
  <tr>
    <td>Specificity, Bingler et al. (2023)</td>
    <td>Paragraph from reports</td>
    <td></td>
  </tr>
  <tr>
    <td>Commitments and Actions, Bingler et al. (2023)</td>
    <td>Paragraph from reports</td>
    <td></td>
  </tr>
  <tr>
    <td>Net Zero/Reduction, Schimanski et al. (2023a)</td>
    <td>Paragraph from Net Zero Tracker, Lang et al. (2023)</td>
    <td><i>Net-zero, Reduction, None:</i> The paragraph contains either a <i>Net-Zero</i> target, a <i>Reduction</i> target or no target (<i>None</i>)</td>
  </tr>
  <tr>
    <th colspan="3">Green Stance Detection</th>
  </tr>
  <tr>
    <td>ClimateFEVER (evidence), Diggelmann et al. (2020)</td>
    <td>A claim and an evidence sentence from Wikipedia</td>
    <td><i>Support, Refutes, Not Enough Information:</i> Determines the relation between a claim and a single evidence sentence.<br><i>Strongly Agreeing, Supporting, No or mixed position, Not supporting, Opposing:</i> Given the position of the party, classifies the stance<br><i>Stance of the evidence (Agree, Disagree, Neutral)</i> toward the claim: Climate-Change is a serious concern.<br><i>Stance towards climate change prevention: Favor, Against, Ambiguous.</i> (Stance used as a broad category including sentiment, evaluation, appraisal, ...) <br><i>Support, Refutes, Debated, Not Enough Information:</i> Determines if a claim is supported by a set of retrieved evidence sentences<br><i>I/O:</i> Contains a stance on a remediation policy</td>
  </tr>
  <tr>
    <td>LobbyMap (Stance), Morio and Manning (2023)</td>
    <td>Page from a company communications (report, press release, ...)</td>
    <td></td>
  </tr>
  <tr>
    <td>Global Warming Stance Detection (GWSD), Luo et al. (2020)</td>
    <td>Sentences from news about global warming</td>
    <td></td>
  </tr>
  <tr>
    <td>ClimateStance, Vaid et al. (2022)</td>
    <td>Tweets posted during COP25 filtered by keywords (relevant to climate-change)</td>
    <td></td>
  </tr>
  <tr>
    <td>ClimateFEVER (claim), Diggelmann et al. (2020)</td>
    <td>a claim and multiple evidence sentences from Wikipedia</td>
    <td></td>
  </tr>
  <tr>
    <td>LobbyMap (Page), Morio and Manning (2023)</td>
    <td>Page from a company communications (report, press release, ...)</td>
    <td></td>
  </tr>
  <tr>
    <td>LobbyMap (Query), Morio and Manning (2023)</td>
    <td>Page from a company communications (report, press release, ...)</td>
    <td></td>
  </tr>
  <tr>
    <th colspan="3">Climate-Related Question Answering</th>
  </tr>
  <tr>
    <td>ClimaQA, Spokoyny et al. (2023)</td>
    <td>The text from a response to one of the CDP questions; and one of the questions from the CDP questionnaire</td>
    <td><i>1:</i> the response answers this question.<br><i>0:</i> The response does not answer this question, but another one</td>
  </tr>
  <tr>
    <td>ClimaINS, Spokoyny et al. (2023)</td>
    <td>The text from a response to one of the questions from the NAIC questionnaire</td>
    <td><i>MANAGE, RISK PLAN, MITIGATE, ENGAGE, ASSESS, RISKS:</i> The labels correspond the 8 questions asked in the NAIC questionnaires</td>
  </tr>
  <tr>
    <th colspan="3">Classification of Deceptive Techniques</th>
  </tr>
  <tr>
    <td>LogicClimate, Jin et al. (2022)</td>
    <td>texts from climatefeedback.org</td>
    <td><i>Faulty Generalization, Ad Hominem, Ad Populum, False Causality, etc.</i>: Classifies fallacies (Multi-label)<br><i>No Claim, Global Warming is not happening, Climate Solutions won’t work, Climate impacts are not bad, etc.</i>: Classifies arguments into super/sub-categories of climate science denier’s arguments</td>
  </tr>
</table>

Table 1: Description of the datasets we collected.
language models (LLMs) in a zero-shot setting (GPT-4o-mini, Llama 3.1 – 8B and 70B).

Evaluation. All models are evaluated on the test datasets. We computed the macro F1-score for each dataset, even if the original papers may have reported other scores such as accuracy or the micro F1-score.

Statistical Significance. To compare models, we computed a 95% confidence interval (CI) of the F1-scores based on bootstrapping (Efron and Tibshirani, 1994). This interval measures the variability based on the data sampling. We consider a difference significant if the CIs are disjoint.

Parameters. Throughout our experiments, we used fixed parameters following Spokoyny et al. (2023)’s parameter choices and seeds to control for randomness (detailed parameters in Appendix B).

4 Results and Analysis

4.1 Results

Table 2 shows the macro-F1 of all models on all datasets. We also show the most relevant performance score from the original studies, which is not necessarily the macro F1-score and thus not comparable to our results (see details in the table caption). We observe that the finetuned Transformer models generally perform best, with an average F1-score of 74.73% for distilRoBERTa and 74.04% for Longformer. Most datasets (70%) do not contain long text, which limits the benefit of using Longformer compared to distilRoBERTa. Secondly, the zero-shot LLMs performed competitively, with an average F1-score reaching 69.78% for GPT-4o-mini. As expected, the larger Llama 3.1 70B outperformed Llama 3.1 8B, although not by a large margin. Finally, the TF-IDF baseline performed surprisingly well, with an average F1-score of 69.18%, out-performing the Llama models, and coming close to GPT-4o-mini. This indicates that the tasks studied in these datasets can be largely solved using only term-frequency features.

4.2 Error Analysis

It is surprising that the top performing LLMs perform worse than the fine-tuned models, despite being significantly more complex and having demonstrated high performance on many other benchmarks (Meta, 2024). We hypothesize that a significant portion of the model’s errors might in fact be issues in the datasets. To test this hypothesis, we analyzed a sample of 503 errors. We investigated the errors of GPT-4o-mini, the best-performing LLM, to minimize the number of genuine errors. However, our findings are relevant for the other LLMs as well, as we found that 70% of the sampled errors are also errors made by the Llama’s models, and 70% of Llama’s errors are also GPT-4o-mini errors (Table 7 in Appendix C).

We analyzed the discrepancies between the model’s prediction and the gold standard to understand both model and dataset limitations. We distinguish mislabeled from debatable instances to align with existing frameworks that distinguish between annotation errors from Human Label Variation (Weber-Genzel et al., 2024).

We sampled between 20 and 30 errors per dataset. Some datasets had fewer than 20 errors, in which case we reviewed them all. Given the input text, the true label, and the prediction of the model, we classified them based in type of error.

Model errors. Incorrect predictions made by the zero-shot GPT-4o-mini approach account for 44.6% of the sampled errors – these are clear misclassifications, not cases of ambiguity or errors in the gold standard. They are caused by different factors such as hallucinations, misunderstanding of implicit or indirect relations, anchoring to a specific word in the prompt, not following part of the instructions, or having no identifiable cause. Many of these errors could most likely be fixed by using a larger model, and will not be detailed here (see Appendix C.11 for details).

Annotation mistakes. Evaluation datasets can contain annotation errors that bias the performances and the conclusions than can be drawn from experiments (Weber-Genzel et al., 2024; Rücker and Akbik, 2023; Bowman and Dahl, 2021; Klie et al., 2023b). We found obvious annotation errors in 60% of the datasets. This accounts for 16.6% of the sampled errors.

Debatable Errors. Finally, we find debatable examples in almost all datasets (96%). We identified three main causes: statements that are out-of-context; statements that are ambiguous or implicit; and definitions of labels that are not exhaustive or detailed enough, leaving room for interpretation. 38.8% of the sampled errors are debatable.
<table>
  <tr>
    <th>Dataset</th>
    <th>Random</th>
    <th>TF-IDF</th>
    <th>Longformer</th>
    <th>DistilRoBERTaGPT-4o-mini</th>
    <th>Llama 8B</th>
    <th>Llama 70B</th>
    <th>Reference</th>
  </tr>
  <tr>
    <th colspan="8">Climate Topic Classification</th>
  </tr>
  <tr>
    <td>climateBUG-data</td>
    <td>49.6±0.6</td>
    <td>86.4±0.4</td>
    <td style="background-color:#b3e5f5;">90.5 ± 0.4</td>
    <td style="background-color:#b3e5f5;">90.5 ± 0.1</td>
    <td>89.2±0.4</td>
    <td>79.0±0.5</td>
    <td>88.3±0.3</td>
    <td>91.36<sup>f</sup></td>
  </tr>
  <tr>
    <td>ClimateBERT Cl. det.</td>
    <td>42.8±4.9</td>
    <td>79.3±4.8</td>
    <td>95.8±2.4</td>
    <td>94.0±3.0</td>
    <td>93.2±3.1</td>
    <td>88.3±3.8</td>
    <td>93.4±3.2</td>
    <td>99.1<sup>c,e</sup></td>
  </tr>
  <tr>
    <td>climatext (Wiki-doc)</td>
    <td>49.3±1.5</td>
    <td>80.4±1.2</td>
    <td>85.6±1.1</td>
    <td>83.8±1.2</td>
    <td>84.0±1.2</td>
    <td>86.3±1.1</td>
    <td style="background-color:#b3e5f5;">89.8 ± 1.0</td>
    <td>-</td>
  </tr>
  <tr>
    <td>climatext (10k)</td>
    <td>46.9±5.4</td>
    <td>91.0±4.0</td>
    <td>97.0±2.4</td>
    <td>96.5±2.6</td>
    <td>95.4±3.2</td>
    <td>90.9±4.2</td>
    <td>96.0±2.8</td>
    <td>95±2.1<sup>t</sup></td>
  </tr>
  <tr>
    <td>climatext (claim)</td>
    <td>48.2±3.1</td>
    <td>74.7±2.8</td>
    <td>67.8±3.2</td>
    <td>75.4±2.9</td>
    <td style="background-color:#b3e5f5;">82.2 ± 2.4</td>
    <td style="background-color:#b3e5f5;">83.6 ± 2.2</td>
    <td style="background-color:#b3e5f5;">85.7 ± 2.0</td>
    <td>83±1.1<sup>t</sup></td>
  </tr>
  <tr>
    <td>climatext (Wikipedia)</td>
    <td>40.1±4.8</td>
    <td>83.5±7.7</td>
    <td>83.9±7.2</td>
    <td>88.2±6.5</td>
    <td>84.7±7.2</td>
    <td>82.8±7.6</td>
    <td>87.4±6.5</td>
    <td>80±6.1<sup>t</sup></td>
  </tr>
  <tr>
    <td>Sustainable Signals rev.</td>
    <td>37.6±11.0</td>
    <td>65.8±11.1</td>
    <td>73.2±10.0</td>
    <td>73.7±9.9</td>
    <td>69.7±11.0</td>
    <td>63.5±10.7</td>
    <td>73.3±9.9</td>
    <td>-</td>
  </tr>
  <tr>
    <th colspan="8">Climate Sub-thematic topic classification</th>
  </tr>
  <tr>
    <td>ClimaTOPIC</td>
    <td>6.0±0.5</td>
    <td>46.6±1.2</td>
    <td style="background-color:#b3e5f5;">55.8 ± 1.1</td>
    <td style="background-color:#b3e5f5;">54.8 ± 1.1</td>
    <td>35.8±1.0</td>
    <td>34.5±1.0</td>
    <td>30.5±1.0</td>
    <td>65.2<sup>r</sup></td>
  </tr>
  <tr>
    <td>climateEng</td>
    <td>13.2±2.9</td>
    <td>58.4±9.3</td>
    <td>70.5±8.5</td>
    <td>67.3±8.6</td>
    <td>65.8±9.2</td>
    <td>53.9±9.4</td>
    <td>61.1±8.5</td>
    <td>74.5<sup>r</sup></td>
  </tr>
  <tr>
    <td>esgbert Biodiversity</td>
    <td>41.1±6.4</td>
    <td>91.3±5.5</td>
    <td>91.1±5.1</td>
    <td>89.0±5.5</td>
    <td>91.8±4.9</td>
    <td>88.8±5.6</td>
    <td>82.3±5.9</td>
    <td>92.29<sup>r</sup></td>
  </tr>
  <tr>
    <td>esgbert Forest</td>
    <td>41.4±5.9</td>
    <td>92.9±4.8</td>
    <td>97.2±2.8</td>
    <td>97.0±2.8</td>
    <td>91.6±4.9</td>
    <td>87.4±5.6</td>
    <td>86.2±5.8</td>
    <td>95.7<sup>r</sup></td>
  </tr>
  <tr>
    <td>esgbert Nature</td>
    <td>51.5±6.5</td>
    <td>82.9±5.0</td>
    <td>89.8±1.9</td>
    <td>89.9±4.0</td>
    <td>82.9±5.2</td>
    <td>86.7±4.5</td>
    <td>88.0±4.6</td>
    <td>94.19<sup>r</sup></td>
  </tr>
  <tr>
    <td>esgbert Water</td>
    <td>42.7±6.4</td>
    <td>84.6±6.6</td>
    <td>95.3±3.4</td>
    <td>93.9±3.9</td>
    <td>93.8±4.1</td>
    <td>89.4±5.3</td>
    <td>93.5±3.9</td>
    <td>95.10<sup>c</sup></td>
  </tr>
  <tr>
    <td>sciDCD</td>
    <td>3.7±1.0</td>
    <td>42.6±3.9</td>
    <td style="background-color:#b3e5f5;">10.3 ± 2.8</td>
    <td style="background-color:#b3e5f5;">12.1 ± 2.7</td>
    <td>29.2±2.3</td>
    <td>20.5±2.2</td>
    <td>30.0±2.4</td>
    <td>54.79<sup>r</sup></td>
  </tr>
  <tr>
    <td>Climate TCFD rec.</td>
    <td>16.5±3.7</td>
    <td>56.5±5.6</td>
    <td style="background-color:#b3e5f5;">69.4 ± 5.2</td>
    <td style="background-color:#b3e5f5;">68.5 ± 5.2</td>
    <td>47.5±5.4</td>
    <td>43.2±5.2</td>
    <td>50.2±5.4</td>
    <td>-</td>
  </tr>
  <tr>
    <td>ESGBERT E</td>
    <td>42.6±7.1</td>
    <td>88.1±5.1</td>
    <td>95.8±3.2</td>
    <td>95.0±3.5</td>
    <td>95.1±3.5</td>
    <td>94.6±3.4</td>
    <td>93.19<sup>c</sup></td>
    <td>-</td>
  </tr>
  <tr>
    <td>ESGBERT G</td>
    <td>53.5±7.1</td>
    <td>80.2±6.3</td>
    <td>78.2±6.4</td>
    <td>83.7±6.4</td>
    <td>83.5±5.7</td>
    <td>75.6±6.1</td>
    <td>62.6±6.9</td>
    <td>78.86<sup>c</sup></td>
  </tr>
  <tr>
    <td>ESGBERT S</td>
    <td>54.4±6.9</td>
    <td>82.6±5.6</td>
    <td>88.3±4.4</td>
    <td style="background-color:#b3e5f5;">89.6 ± 4.2</td>
    <td>78.1±5.8</td>
    <td>77.9±6.0</td>
    <td>73.6±6.2</td>
    <td>91.90<sup>c</sup></td>
  </tr>
  <tr>
    <th colspan="8">Sentiment Analysis</th>
  </tr>
  <tr>
    <td>climate sentiment</td>
    <td>35.9±5.2</td>
    <td>69.0±5.5</td>
    <td>79.9±4.8</td>
    <td>77.7±4.8</td>
    <td>77.3±4.9</td>
    <td>71.4±5.3</td>
    <td>70.5±5.4</td>
    <td>83.8<sup>c,e</sup></td>
  </tr>
  <tr>
    <th colspan="8">Claim detection</th>
  </tr>
  <tr>
    <td>Environmental Claims</td>
    <td>42.8±5.5</td>
    <td>80.5±5.4</td>
    <td>90.5±3.8</td>
    <td>91.2±3.8</td>
    <td>86.8±4.5</td>
    <td>81.4±5.0</td>
    <td>76.6±5.4</td>
    <td>84.9<sup>c,r</sup></td>
  </tr>
  <tr>
    <td>Green Claims</td>
    <td>46.1±10.9</td>
    <td>86.1±8.0</td>
    <td>94.3±4.0</td>
    <td>97.2±3.3</td>
    <td>91.5±6.8</td>
    <td>90.0±7.3</td>
    <td>86.8±7.3</td>
    <td>92.08<sup>c,r</sup></td>
  </tr>
  <tr>
    <th colspan="8">Claim characteristics</th>
  </tr>
  <tr>
    <td>Commitments&Actions</td>
    <td>48.5±5.5</td>
    <td>72.7±5.4</td>
    <td>76.7±5.1</td>
    <td style="background-color:#b3e5f5;">81.9 ± 5.7</td>
    <td>67.2±5.4</td>
    <td>64.6±5.2</td>
    <td>50.7±5.6</td>
    <td>81<sup>c,e</sup></td>
  </tr>
  <tr>
    <td>Climate Specificity</td>
    <td>48.9±5.3</td>
    <td>72.4±5.0</td>
    <td>77.3±4.8</td>
    <td>77.5±5.1</td>
    <td>71.6±4.9</td>
    <td>76.5±4.6</td>
    <td>68.0±5.2</td>
    <td>77<sup>c</sup></td>
  </tr>
  <tr>
    <td>ESGBERT action500</td>
    <td>44.4±14.3</td>
    <td>82.5±11.0</td>
    <td>85.9±10.2</td>
    <td>89.1±8.8</td>
    <td>76.0±12.6</td>
    <td>67.0±13.2</td>
    <td>63.0±14.2</td>
    <td>-</td>
  </tr>
  <tr>
    <td>Implicit/Explicit Claims</td>
    <td>37.2±9.9</td>
    <td>70.3±12.8</td>
    <td>63.0±13.1</td>
    <td>81.2±11.9</td>
    <td>81.6±9.6</td>
    <td>68.5±13.2</td>
    <td>75.7±11.3</td>
    <td>81.45<sup>c,r</sup></td>
  </tr>
  <tr>
    <td>Net-Zero/Reduction</td>
    <td>29.8±4.9</td>
    <td>94.8±2.4</td>
    <td>97.8±1.5</td>
    <td>97.7±1.8</td>
    <td>97.3±1.8</td>
    <td>93.7±2.8</td>
    <td>95.6±2.2</td>
    <td>98.7<sup>c</sup></td>
  </tr>
  <tr>
    <th colspan="8">Stance classification</th>
  </tr>
  <tr>
    <td>climateF: claim (our split)</td>
    <td>15.4±5.1</td>
    <td>35.0±8.3</td>
    <td>31.9±7.1</td>
    <td>32.5±7.4</td>
    <td>19.3±5.9</td>
    <td>30.0±7.7</td>
    <td>26.3±6.5</td>
    <td>80.7<sup>c,t</sup></td>
  </tr>
  <tr>
    <td>climateF: claim (agg.)</td>
    <td>-</td>
    <td>-</td>
    <td>42.5±7.8</td>
    <td>41.3±6.6</td>
    <td>50.4±8.1</td>
    <td>-</td>
    <td>-</td>
    <td>60.10<sup>c,t</sup></td>
  </tr>
  <tr>
    <td>climateFEVER evidence</td>
    <td>28.4±3.1</td>
    <td>46.2±4.0</td>
    <td>52.7±4.1</td>
    <td>51.3±3.9</td>
    <td>60.0±3.7</td>
    <td>52.5±3.7</td>
    <td style="background-color:#b3e5f5;">63.7 ± 3.9</td>
    <td>68.03<sup>c,t</sup></td>
  </tr>
  <tr>
    <td>climateStance</td>
    <td>21.3±3.7</td>
    <td>49.6±6.5</td>
    <td>56.0±6.8</td>
    <td>56.1±7.7</td>
    <td>56.4±5.9</td>
    <td>48.5±5.9</td>
    <td>58.0±6.3</td>
    <td>59.69<sup>c</sup></td>
  </tr>
  <tr>
    <td>Global-Warming Stance</td>
    <td>29.5±6.5</td>
    <td>59.2±7.4</td>
    <td>69.3±6.8</td>
    <td>73.5±6.3</td>
    <td>69.8±6.6</td>
    <td>64.4±6.8</td>
    <td>72.5±6.5</td>
    <td>73<sup>c</sup></td>
  </tr>
  <tr>
    <td>LobbyMap (Pages)</td>
    <td>46.3±0.8</td>
    <td>73.1±0.8</td>
    <td style="background-color:#b3e5f5;">71.3 ± 0.8</td>
    <td style="background-color:#b3e5f5;">73.0 ± 0.8</td>
    <td>63.2±0.8</td>
    <td>62.5±0.8</td>
    <td>60.0±0.8</td>
    <td>-</td>
  </tr>
  <tr>
    <td>LobbyMap (Query)</td>
    <td>12.4±0.5</td>
    <td>49.3±1.8</td>
    <td style="background-color:#b3e5f5;">57.3 ± 2.1</td>
    <td style="background-color:#b3e5f5;">52.1 ± 2.1</td>
    <td>36.4±1.8</td>
    <td>27.6±1.6</td>
    <td>32.7±1.6</td>
    <td>-</td>
  </tr>
  <tr>
    <td>LobbyMap (Stance)</td>
    <td>19.2±1.3</td>
    <td>43.6±1.8</td>
    <td style="background-color:#b3e5f5;">46.7 ± 1.6</td>
    <td style="background-color:#b3e5f5;">44.7 ± 1.9</td>
    <td>30.0±1.6</td>
    <td>26.9±1.5</td>
    <td>24.1±1.5</td>
    <td>-</td>
  </tr>
  <tr>
    <th colspan="8">Question answering</th>
  </tr>
  <tr>
    <td>ClimaINS (our split)</td>
    <td>12.8±1.8</td>
    <td>81.2±2.1</td>
    <td style="background-color:#b3e5f5;">77.6 ± 4.2</td>
    <td style="background-color:#b3e5f5;">75.8 ± 4.5</td>
    <td>58.5±2.6</td>
    <td>45.4±2.5</td>
    <td>48.4±2.6</td>
    <td>-</td>
  </tr>
  <tr>
    <td>climaQA (our split)</td>
    <td>50.5±1.1</td>
    <td>50.3±1.0</td>
    <td style="background-color:#b3e5f5;">89.5 ± 0.6</td>
    <td style="background-color:#b3e5f5;">89.4 ± 0.6</td>
    <td>78.8±0.8</td>
    <td>57.4±1.1</td>
    <td>76.1±1.0</td>
    <td>-</td>
  </tr>
  <tr>
    <th colspan="8">Deceptive techniques</th>
  </tr>
  <tr>
    <td>CC-Contrarian Claims</td>
    <td>3.2±0.7</td>
    <td>62.7±2.5</td>
    <td style="background-color:#b3e5f5;">71.6 ± 2.7</td>
    <td style="background-color:#b3e5f5;">71.2 ± 2.5</td>
    <td>58.8±2.9</td>
    <td>40.2±3.1</td>
    <td>55.8±2.9</td>
    <td>79<sup>j</sup></td>
  </tr>
  <tr>
    <td>logicClimate</td>
    <td>13.3±2.0</td>
    <td>13.3±5.0</td>
    <td>15.6±5.6</td>
    <td>9.4±3.2</td>
    <td>27.4±7.8</td>
    <td>10.3±4.3</td>
    <td>22.1±6.4</td>
    <td>29.37<sup>k</sup></td>
  </tr>
  <tr>
    <td>Average</td>
    <td>34.24</td>
    <td>69.18</td>
    <td>74.04</td>
    <td>74.73</td>
    <td>69.78</td>
    <td>65.03</td>
    <td>67.39</td>
    <td>-</td>
  </tr>
</table>

Table 2: Macro F1 score of baselines and methods on each dataset. The bootstrap confidence interval is displayed as \( \pm \frac{F_{1\text{95%}} - F_{1\text{5%}}}{2} \). For each Reference value, we add a clarifying number if the metric is not macro F1: 1. binary F1-score, 2. average F1-score (macro/micro is not specified), 3. weighted average F1-score. In addition, we specify the method in the reference via letters: a. SVM, c. climateBERT, d. Contaminated split, SciBERT, e. ESGBERT, f. climateBUG-LM, g. EnvironmentalBERT, h. Human Baseline using annotations, i. Filtered Split (by removing disputed claims), j. RoBERTa + Logistic Regression, k. Electra, l. Longformer, r. RoBERTa, t. BERT. ▲ weakly labeled datasets; ▲ small datasets (<1000 in training dataset), ✓ larger, human-annotated datasets. In red: performance not significantly higher than random; In gray: performance not significantly higher than TF-IDF baseline; In green: fine-tuned models performing significantly better than all zero-shot approaches (and inversely). A performance is considered significantly higher if 0 is not included in the bootstrap 95% confidence interval of the difference of F1-score.
4.3 Dataset Issues

In the following, we discuss possible causes of the annotation mistakes we observed in our study (This section contains a condensed analysis of errors, we detail error per dataset in Appendix C).

Ambiguous annotations guidelines. For each of the datasets, we designed the prompts based on the annotation guidelines. However, we observed that many prediction errors from GPT-4o-mini were caused by ambiguities in the guidelines:
(1) In ClimateBUG, the guideline says that “Sustainability not related to the environment (i.e. sustainable profits)” should be classified as “Not Climate”. However, some sentences do not specify whether the sustainability is related to the environment or not (e.g. “They are the source of jobs, innovation, sustainability, and prosperity”).
(2) In ESGBERT Nature, it is not clear if the “water” label, which focuses on “water management, consumption, and pollution”, includes water-related natural disasters such as tsunamis.
(3) In climate sentiment, it is not clear if corporate ambitions should be classified as “Opportunities” as they “associate specific positive adjectives to the anticipated, past, or present developments and topics covered” (e.g. “TD recently launched a bold and ambitious climate action plan to address the challenges of climate change. This includes a target to achieve net-zero greenhouse gas emissions in our operations and financing activities by 2050”) or as a neutral statement as aligning with climate objectives will not necessary bring growth to a company.
(4) Environmental Claims does not provide instructions on implicit claims.
(5) Green Claims is not specific on how to annotate claims suggesting that a product is better, healthier, or has good properties thanks to a natural ingredient.
(6) In Commitments&Actions, it is not clear how to annotate descriptions of company values, governance structures, or descriptions of existing processes.
(7) Climate Specificity does not describe how to handle the granularity of the specificity (e.g. ‘For our sustainable strategy range, we incorporate a series of proprietary ’red lines’ in order to ensure the poorest-performing companies from an ESG perspective are not eligible for investment’).
(8) In ClimateFEVER evidence, it is not clear if “SUPPORTING” means that the evidence is sufficient to entail the claim, or if it means that the evidence is merely in line with the claim.
(9) In ClimateStance, it is not clear how to annotate a tweet that is in “opposition to climate change policies”, but simultaneously acknowledges the urgency of climate change.
(10) In CC-Contrarian Claims, some labels are difficult to differentiate such as “Weather is cold/snowing” and “Ice/permafrost/snow cover isn’t melting”.
(11) In LogicClimate, the “intentional” label is broad and encompasses all other fallacy types.
These issues impact many datasets and a large part of the sampled errors of GPT-4o-mini (4/9 on ESGBERT Water, 5/20 on ClimateBUG, 5/10 on climate sentiment, 5/14 on Environmental Claims, 5/6 on (Woloszyn et al., 2022), 9/20 Commitments and Actions, 4/20 on specificity, 20/20 additional samples4 ClimateFEVER evidence, 2/10 in ClimateStance, 2/10 in CC-Contrarian Claims, 3/10 in LogicClimate).

Implicitness. Many errors were due to the implicit nature of the text. This represents a large portion of the sampled errors in multiple datasets (4/17 on Climate detection, 3/10 on climate TCFD recommendations, 2/8 on ESGBERT, 3/14 on Environmental claims, 2/10 on ClimateStance, 7/10 on GWSD). Interestingly, we even found examples where the model understood the indirect link but not the annotators (2/10 on climate sentiment). A notable implied reference to climate change, which the model struggles on, is categorizing statements about energy as climate/environment-related (4/17 Climate detection, 2/8 on ESGBERT, 3/14 on Climatext (wiki)). These cases are prediction errors by GPT-4o-mini; however, due to the implicit nature of the statements, the labels are debatable.

Multi-label. In many different contexts, multiple labels might be valid (Weber-Genzel et al., 2024). This is what we observed in 26% of the single-label datasets, and 8.75% of the sampled errors. For example, “Reduce GHG emissions [...] by 55%

4We randomly sampled 10 errors of GPT-4o-mini predicting “REFUTES” instead of “NOT ENOUGH INFO” and 10 errors of it predicting “NOT ENOUGH INFO” instead of “SUPPORTS”.
by FY2030 and to zero by FY2050 [...]” contains both a reduction target and a net-zero target. We believe this issue arises from trying to simplify the annotation process and build a single-label dataset in settings that are intrinsically multi-label.

Weak labels. Zero-shot LLMs performed significantly worse than finetuned models on weakly labeled datasets that were not annotated through a controlled annotation procedure. The weak labels are useful to create large datasets without requiring human annotations; however, they lack the rigorous definitions that manual labels provide, which are derived from strict annotation guidelines. This automatic generation can introduce inconsistencies such as evolving label definitions over time or unexpected responses in survey data. This makes these labels noisy and less reliable for classification tasks. This explains the lower performances on SciDCC, ClimaINS, ClimaTOPIC and ClimaQA. Additionally, since the labels do not follow strict guidelines, designing a prompt for these tasks requires guessing the actual label meaning, which might be different from the label intended purpose. For example, the Climatext task is to identify climate-related sentences. However, to create the datasets annotated by humans, Varini et al. (2020) relied on a weakly labeled dataset to filter potentially climate-related sentences. Therefore, the intended purpose is climate-relatedness, but the actual meaning of the label is “a sentence extracted from a climate-related Wikipedia page” (which is the prompt used to reach F1-scores above 80%). Despite these non-intuitive definitions, the labels are actually linguistically distinguishable, as shown by the performances of the fine-tuned models and the TF-IDF baselines.

Exhaustivity. This is a particular case of issues arising from weakly labeled datasets. Because they do not rely on rigorous annotation guidelines intended for classification, the annotations are not necessarily exhaustive. The LobbyMap dataset, e.g., is constructed using data from LobbyMap.org, a website that identifies companies’ stances on climate change mitigation policies and quotes companies’ documents with evidence about their stance. Therefore, for each policy, they identify evidence, but they do not necessarily collect all the evidence about that stance. Inversely, the model is asked to identify, given a document, what is the list of policy stances mentioned. GPT-4o-mini tends to predict more stances per document than the original labels. Moreover, in the sampled errors, we found that the predictions were mostly reasonable. This suggests that the annotations might not be exhaustive.

4.4 Discussion
The weakly labeled datasets are interesting as they build a foundation for relevant tasks: e.g. LobbyMap for identifying stance on climate-remediation policies, SciDCC to classify news, or, ClimaQA and ClimaINS–from Task 7– and ClimaTOPIC to structure company documents; however, the poor performance and the error analysis reveal that the datasets need further annotations.

Task 1 to 5 focus on topic classification, risk and green claim. When excluding weakly labeled datasets, performances are high for fine-tuned models, but also for TF-IDF, showing that these tasks are highly based on vocabulary. Some datasets remain challenging: TCFD recommendations, ClimateEng and, to a lesser extent, climate sentiment and specificity. However, improving the guidelines to resolve ambiguous cases would improve label consistency, and most likely performances, particularly for LLMs which rely more heavily on guidelines, while finetuned models rely on the annotations.

Task 6 focuses on stance detection. Models demonstrate that they have predictive power, but the performance is relatively low (F1-score under 75%). However, the task is inherently hard, as shown by the low performance of humans on ClimateFEVER, or moderate IAA (Krippendorf’s \( \alpha = 0.54-0.64 \)) reported by Luo et al. (2020).

Task 8 focuses on deceptive techniques. On CC-contrarian claims, given the high number of classes, the performances are high, even for TF-IDF. For fallacy detection, the task is inherently subjective (Helwe et al., 2024). This is particularly the case when using real-world examples where multiple types of fallacies could fit.

4.5 Recommendations
As shown through our error analysis, many false predictions of a zero-shot model are actually mislabeled or debatable examples. Such examples are problematic for two reasons: First, the performance differences between models are often minimal (sometimes less than 1%), making it difficult to draw meaningful comparisons when the datasets are noisy. Second, it becomes unclear whether observed performance gains stem from genuine model improvements or from overfitting to annotation biases and errors. Fine-tuned models may perform well on these datasets, but that does not mean
that the datasets are clean: The models may pick up patterns in the data that have been informally discussed between the annotators, but that are not codified in the guidelines. To address these issues and improve the reliability of model evaluation, we propose the following set of recommendations.

Use Precise and Exhaustive Annotation Guidelines. Ambiguous or under-specified guidelines result in inconsistent annotations, reducing dataset reliability and skewing model evaluation. To mitigate this, guidelines should be:
(1) Precise: Clearly define all terms and concepts.
(2) Exhaustive: Address edge cases and ambiguities.
(3) Unambiguous: Avoid vague phrasing and provide concrete examples.
Additionally, we recommend not using automatically labeled datasets, as they do not follow well-defined annotation procedures and can easily produce ill-defined tasks.

Design Datasets with Ambiguous or Implicit Statements. Many current datasets are too easily solved with keyword-based heuristics (e.g., detecting terms like “GHG” or “climate change”). However, LLMs struggle with more implicit or ambiguous cases. To assess true model understanding, future datasets should focus on examples that require inference and contextual reasoning.

Include Simple Baselines to Assess Task Difficulty. We observed that many tasks can be solved with simple models such as word frequency or TF-IDF. Including these baselines is essential to gauge the added value of advanced models. Ideally, a subset of the dataset should be challenging for such baselines, indicating that genuine language understanding is required.

Quantify Annotation Error Rates and Their Impact. When model performance exceeds 90%, even a small number of annotation errors can meaningfully distort evaluation. We recommend estimating the annotation error rate through manual review or IAA agreement analysis. This allows researchers to define a performance margin below which observed differences may fall within the noise of the dataset, and thus not reflect genuine model improvement. Such quantification is essential for drawing reliable conclusions from small performance gains. This also highlights the importance of computing meaningful confidence intervals measuring uncertainty of performances due to factors such as dataset sampling or initialization.

Prioritize Dataset Quality Over Quantity. As model accuracy rises, even small annotation errors can significantly impact evaluation. We recommend focusing on clean, high-quality test sets rather than large weakly labeled datasets, which often lack robust guidelines. Poor LLM performance on such datasets may reflect structural flaws rather than model limitations.

5 Conclusion

In this work, we curated, cleaned, and standardized 29 climate-related NLP datasets, ensuring consistency across tasks. We systematically evaluated a diverse range of models. Our results indicate that all approaches achieved competitive performance. However, since even the TF-IDF baseline performs well, the datasets may overall be too simple. Our analysis also revealed that nearly all datasets contain annotation inconsistencies, which may introduce noise in model evaluations. These findings highlight the importance of dataset quality in benchmarking and call for more rigorous annotation protocols in climate-related NLP research.

All our data, scripts, and models are publicly available at at https://github.com/tcalamai/acl_climateNLPToolbox. In particular, we provide a Python library that can run all models on entire files by parsing the PDFs, splitting them into paragraphs, doing model inference on each paragraph (individually or as an ensemble), and aggregating the results into statistics (example in Appendix D).

Acknowledgement. This work was performed using HPC resources from GENCI-IDRIS (Grant AD011014244R1). This work is supported by Amundi Technology and the ANRT with a CIFRE fellowship.

6 Limitations

Our study has several limitations. First, it relies on publicly available datasets, which may not fully represent the entire scientific corpus of datasets related to climate-related tasks.

Second, while we accounted for variability in training through uncertainty estimation, our results remain influenced by factors such as hyperparameter selection. These factors can significantly impact model performance, and while our findings align with those of previous studies, further optimization could improve performance.

Third, we designed our prompts to maximize model performance under a zero-shot setting with
a small chain-of-thought (CoT) component. However, alternative prompting strategies, such as few-shot learning, self-reflection, and iterative reasoning, could enhance LLM performance. Additionally, our study was limited to smaller language models, and larger models, such as LLaMA 3 405B, could yield improved results.

Regarding annotation, error analysis was conducted by the authors of this study. To facilitate interpretability, we exposed both the gold label and the model’s prediction during the annotation process, allowing us to analyze the model’s reasoning. Despite this, we believe our error analysis provides meaningful insights into model behavior. Moreover, we computed the IAA on a subset of the datasets for 2 annotators; we found a Cohen’s \( \kappa \) 0.395 for the 3 label classification: “actual error”, “debatable”, “mislabeled”. This is a weak agreement, making the exact figures mentioned in this paper approximations. However, the agreement is better on classifying errors as “actual errors” or not (\( \kappa = 0.591 \)). Moreover, the disagreement between the annotators was mostly with the “debatable” label. This further confirms that the wrong predictions partly stem from HLV.

Finally, we acknowledge a broader ethical consideration: developing tools that analyze how entities communicate about climate-related issues may inadvertently enable strategic adaptation by these entities. Specifically, organizations could leverage such tools either to evade detection or to optimize their messaging to increase perceived compliance. This risk underscores the importance of continuous evaluation and responsible deployment of NLP systems in this domain.

References

Alix Auzepy, Elena Tönjes, David Lenz, and Christoph Funk. 2023. Evaluating tcfd reporting—a new application of zero-shot analysis to climate-related financial disclosures. PLOS ONE, 18(11):1–23.

Iz Beltagy, Matthew E. Peters, and Arman Cohan. 2020. Longformer: The long-document transformer. arXiv:2004.05150.

Julia Bingler, , Mathias Kraus, and Markus Leippold. 2021. Cheap Talk and Cherry-Picking: What ClimateBert has to say on Corporate Climate Risk Disclosures. Social Science Research Network.

Julia Bingler, Mathias Kraus, Markus Leippold, and Nicolas Webersinke. 2023. How cheap talk in climate disclosures relates to climate initiatives, corporate emissions, and reputation risk. Working paper, Available at SSRN 4000708.

Xavier Bouthillier, Pierre Delaunay, Mirko Bronzi, Assya Trofimov, Brennan Nichyporuk, Justin Szeto, Nazanin Mohammadi Sepahvand, Edward Raff, Kanika Madan, Vikram Voleti, et al. 2021. Accounting for variance in machine learning benchmarks. Proceedings of Machine Learning and Systems, 3:747–769.

Xavier Bouthillier and Gaël Varoquaux. 2020. Survey of machine-learning experimental methods at NeurIPS2019 and ICLR2020. Ph.D. thesis, Inria Saclay Ile de France.

Samuel R. Bowman and George Dahl. 2021. What will it take to fix benchmarking in natural language understanding? In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 4843–4855, Online. Association for Computational Linguistics.

Tom Calamai, Oana Balalau, Théo Le Guenadal, and Fabian M. Suchanek. 2025. Corporate greenwashing detection in text - a survey. Preprint, arXiv:2502.07541.

Travis G. Coan, Constantine Boussalis, John Cook, and Mirjam O. Nanko. 2021. Computer-assisted classification of contrarian claims about climate change. Scientific Reports, 11(1):22320.

Thomas Digelmann, Jordan Boyd-Graber, Jannis Bulian, Jannis Bulian, Massimiliano Ciaramita, and Markus Leippold. 2020. CLIMATE-FEVER: A Dataset for Verification of Real-World Climate Claims. Tackling Climate Change with Machine Learning workshop at NeurIPS 2020, Online, 11 December 2020 - 11 December 2020.

Bradley Efron and Robert J Tibshirani. 1994. An introduction to the bootstrap. Chapman and Hall/CRC.

David Friederich, Lynn H. Kaack, Alexandra Lucioni, and Bjarne Steffen. 2021. Automated Identification of Climate Risk Disclosures in Annual Corporate Reports. Papers 2108.01415, arXiv.org.

Eduardo C. Garrido-Merchán, Cristina González-Barthe, and María Coronado Vaca. 2023. Fine-tuning climatebert transformer with climatext for the disclosure analysis of climate-related financial risks. Preprint, arXiv:2303.13373.

Timnit Gebru, Jamie Morgenstern, Briana Vecchione, Jennifer Wortman Vaughan, Hanna Wallach, Hal Daumé III, and Kate Crawford. 2021. Datasheets for datasets. Communications of the ACM, 64(12):86–92.

Odd Erik Gundersen, Kevin Coakley, Christine Kirkpatrick, and Yolanda Gil. 2022. Sources of irreproducibility in machine learning: A review. arXiv preprint arXiv:2204.07610.
Chadi Helwe, Tom Calamai, Pierre-Henri Paris, Chloé Clavel, and Fabian Suchanek. 2024. MAFALDA: A benchmark and comprehensive study of fallacy detection and classification. In Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers), pages 4810–4845, Mexico City, Mexico. Association for Computational Linguistics.

Zhijing Jin, Abhinav Lalwani, Tejas Vaidhya, Xiaoyu Shen, Yiwen Ding, Zhiheng Lyu, Mrinmaya Sachan, Rada Mihalcea, and Bernhard Schoelkopf. 2022. Logical fallacy detection. In Findings of the Association for Computational Linguistics: EMNLP 2022, pages 7180–7198, Abu Dhabi, United Arab Emirates. Association for Computational Linguistics.

Jan-Christoph Klie, Bonnie Webber, and Iryna Gurevych. 2023a. Annotation error detection: Analyzing the past and present for a more coherent future. Computational Linguistics, 49(1):157–198.

Jan-Christoph Klie, Bonnie Webber, and Iryna Gurevych. 2023b. Annotation Error Detection: Analyzing the Past and Present for a More Coherent Future. Computational Linguistics, 49(1):157–198.

John Lang, Camilla Hyslop, Natasha Lutz, Natalie Short, Richard Black, Peter Chalkley, Thomas Hale, Frederic Hans, Nick Hay, Niklas Höhne, Angel Hsu, Takeshi Kuramochi, Silke Mooldijk, and Steve Smith. 2023. Net zero tracker. Energy and Climate Intelligence Unit, Data-Driven EnviroLab, NewClimate Institute, Oxford Net Zero.

Tong Lin, Tianliang Xu, Amit Zac, and Sabina Tomkins. 2023. Sustainable signals: An ai approach for inferring consumer product sustainability. In Proceedings of the Thirty-Second International Joint Conference on Artificial Intelligence, IJCAI-23, pages 6067–6075. International Joint Conferences on Artificial Intelligence Organization. AI for Good.

Yiwei Luo, Dallas Card, and Dan Jurafsky. 2020. Detecting Stance in Media On Global Warming. In Findings of the Association for Computational Linguistics: EMNLP 2020, pages 3296–3315, Online. Association for Computational Linguistics.

Meta. 2024. Introducing Ilama 3.1: Our most capable models to date.

Prakamya Mishra and Rohan Mittal. 2021. Neuralnere: Neural named entity relationship extraction for end-to-end climate change knowledge graph construction. In ICML 2021 Workshop on Tackling Climate Change with Machine Learning.

Gaku Morio and Christopher D Manning. 2023. An NLP benchmark dataset for assessing corporate climate policy engagement. In Thirty-seventh Conference on Neural Information Processing Systems Datasets and Benchmarks Track.

Gabrijela Perković, Antun Drobnjak, and Ivica Botički. 2024. Hallucinations in llms: Understanding and addressing challenges. In 2024 47th MIPRO ICT and Electronics Convention (MIPRO), pages 2084–2088.

Paulo Pirozelli, Marcos M. José, Igor Silveira, Flávio Nakasato, Sarajane M. Peres, Anarosa A. F. Brandão, Anna H. R. Costa, and Fabio G. Cozman. 2023. Benchmarks for pirá 2.0, a reading comprehension dataset about the ocean, the brazilian coast, and climate change. Preprint, arXiv:2309.10945. https://github.com/C4AI/Pira.

Susanna Rücker and Alan Akbik. 2023. CleanCoNLL: A nearly noise-free named entity recognition dataset. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 8628–8645, Singapore. Association for Computational Linguistics.

Daniel Ruffinelli, Samuel Broscheit, and Rainer Gemulla. 2020. You can teach an old dog new tricks! on training knowledge graph embeddings.

Rylen Sampson, Aysha Cotterill, and Quoc Tien Au. 2022. Tcfd-nlp: Assessing alignment of climate disclosures using nlp for the financial markets. In NeurIPS 2022 Workshop on Tackling Climate Change with Machine Learning.

Victor Sanh, Lysandre Debut, Julien Chaumond, and Thomas Wolf. 2019. Distilbert, a distilled version of bert: smaller, faster, cheaper and lighter. ArXiv, abs/1910.01108.

Tobias Schimanski, Julia Bingler, Camilla Hyslop, Mathias Kraus, and Markus Leippold. 2023a. Climatebert-netzero: Detecting and assessing net zero and reduction target. Swiss Finance Institute Research Paper, (23-110).

Tobias Schimanski, Chiara Colesanti Senni, Glen Gostlow, Jingwei Ni, Tingyu Yu, and Markus Leippold. 2024a. Exploring nature: Datasets and models for analyzing nature-related disclosures. SSRN Electronic Journal.

Tobias Schimanski, Jingwei Ni, Roberto Spacey Martín, Nicola Ranger, and Markus Leippold. 2024b. ClimRetrieve: A benchmarking dataset for information retrieval from corporate climate disclosures. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pages 17509–17524, Miami, Florida, USA. Association for Computational Linguistics.

Tobias Schimanski, Andrín Reding, Nico Reding, Julia Bingler, Mathias Kraus, and Markus Leippold. 2023b. Bridging the Gap in ESG Measurement: Using NLP to Quantify Environmental, Social, and Governance Communication.

Freda Shi, Xinyun Chen, Kanishka Misra, Nathan Scales, David Dohan, Ed Chi, Nathanael Schärli, and Denny Zhou. 2023. Large language models can be easily distracted by irrelevant context. Preprint, arXiv:2302.00093.
Daniel Spokoyny, Tanmay Laud, Tom Corringham, and Taylor Berg-Kirkpatrick. 2023. Towards answering climate questionnaires from unstructured climate reports. Preprint, arXiv:2301.04253.

Dominik Stammnbach, Nicolas Webersinke, Julia Bingler, Mathias Kraus, and Markus Leippold. 2023. Environmental claim detection. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pages 1051–1066, Toronto, Canada. Association for Computational Linguistics.

The World Meteorological Organization (WMO). Wmo confirms 2024 as warmest year on record at about 1.55°C above pre-industrial level. https://wmo.int/news/media-centre/wmo-confirms-2024-warmest-year-record-about-155degc-above-pre-industrial-level. [Online; accessed 03-February-2025].

David Thulke, Yingbo Gao, Petrus Pelser, Rein Brune, Rricha Jalota, Floris Fok, Michael Ramos, Ian van Wyk, Abdallah Nasir, Hayden Goldstein, Taylor Tragemann, Katie Nguyen, Ariana Fowler, Andrew Stanco, Jon Gabriel, Jordan Taylor, Dean Moro, Evgenii Tsyymbalov, Juliette de Waal, Evgeny Matusov, Mudar Yaghi, Mohammad Shihadah, Hermann Ney, Christian Dugast, Jonathan Dotan, and Daniel Erasmus. 2024. Climategpt: Towards ai synthesizing interdisciplinary research on climate change. Preprint, arXiv:2401.09646.

Saeid Vaghefi, Veruska Muccione, Christian Huggel, Hamed Khashehchi, and Markus Leippold. 2022. Deep climate change: A dataset and adaptive domain pre-trained language models for climate change related tasks. In NeurIPS 2022 Workshop on Tackling Climate Change with Machine Learning.

Roopal Vaid, Kartikey Pant, and Manish Shrivastava. 2022. Towards fine-grained classification of climate change related social media text. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics: Student Research Workshop, pages 434–443, Dublin, Ireland. Association for Computational Linguistics.

Francesco S. Varini, Jordan Boyd-Graber, Massimiliano Ciaramita, and Markus Leippold. 2020. ClimaText: A Dataset for Climate Change Topic Detection. Tackling Climate Change with Machine Learning workshop at NeurIPS 2020, Online, 11 December 2020 - 11 December 2020.

Gengyu Wang, Lawrence Chillrud, and Kathleen McKeown. 2021. Evidence based automatic fact-checking for climate change misinformation. In ICWSM Workshops.

Leon Weber-Genzel, Siyao Peng, Marie-Catherine De Marneffe, and Barbara Plank. 2024. VariErr NLI: Separating annotation error from human label variation. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 2256–2269, Bangkok, Thailand. Association for Computational Linguistics.

Nicolas Webersinke, Mathias Kraus, Julia Bingler, and Markus Leippold. 2022. ClimateBERT: A Pretrained Language Model for Climate-Related Text.

Vinicius Woloszyn, Joseph Kobti, and Vera Schmitt. 2022. Towards automatic green claim detection. In Proceedings of the 13th Annual Meeting of the Forum for Information Retrieval Evaluation, FIRE ’21, page 28–34, New York, NY, USA. Association for Computing Machinery.

Kun Xiang and Akihiro Fujii. 2023. DARE: Distill and Reinforce Ensemble Neural Networks for Climate-Domain Processing. Entropy, 25(4):643.

Yinan Yu, Samuel Scheidegger, Jasmine Elliott, and Asa Lofgren. 2024. climateBUG : A data-driven framework for analyzing bank reporting through a climate lens. Expert Systems with Applications, 239:122162.

A Datasets

Summary of the construction of the dataset To construct the datasets used for the reproducibility study, we followed the following steps:

1. We collected the datasets: we present statistics on their size in Table 3;

2. We explored each dataset to assess data quality (language, gibberish, text length, language), understand the labels (single label or multi-label, relation classification or multi-class classification);

3. We removed text longer than 4000 tokens, and smaller than 5 tokens, for details on which datasets were affected see Table 4;

4. We cleaned the text for formatting, punctuation, space, and encoding issues, and we removed duplicates: we report the statistics in Table 5 ;

5. We tried to keep existing splits if they are provided in the original source. However, if there was data contamination between the train set and the test set and the instances of contamination are rare, we remove the contamination. If there is a dataset construction issue, we rebuilt the dataset to remove the contamination.

6. We constructed based on the previous steps a training and development dataset, each smaller than 10k examples (as it is intended
for fine-tuning). We also construct a test dataset, however for the test dataset we keep the maximum size (to stay as close as possible to the original paper). During down-sampling, we ensured that label distribution was preserved through stratification. For heavily imbalanced datasets, we adjusted the sampling to improve the balance. In Table 6 we detail the ratio between the most frequent and least frequent label in the datasets, in the original datasets and in the newly created datasets.

<table>
  <tr>
    <th>Dataset</th>
    <th>Size<br>train</th>
    <th>dev</th>
    <th>test</th>
    <th>max.<br>length</th>
  </tr>
  <tr>
    <td>ESGBERT action500</td>
    <td>400</td>
    <td>50</td>
    <td>139</td>
    <td>78</td>
  </tr>
  <tr>
    <td>Green Claims</td>
    <td>618</td>
    <td>77</td>
    <td>78</td>
    <td>93</td>
  </tr>
  <tr>
    <td>SUSTAINABLESIGNS reviews</td>
    <td>623</td>
    <td>78</td>
    <td>78</td>
    <td>246</td>
  </tr>
  <tr>
    <td>Implicit/Explicit Green Claims</td>
    <td>618</td>
    <td>77</td>
    <td>78</td>
    <td>93</td>
  </tr>
  <tr>
    <td>climateFEVER claim (our split)</td>
    <td>1228</td>
    <td>154</td>
    <td>153</td>
    <td>78</td>
  </tr>
  <tr>
    <td>logicClimate</td>
    <td>679</td>
    <td>218</td>
    <td>179</td>
    <td>316</td>
  </tr>
  <tr>
    <td>ESGBERT G</td>
    <td>1600</td>
    <td>200</td>
    <td>200</td>
    <td>200</td>
  </tr>
  <tr>
    <td>Global-Warming Stance (GWSD)</td>
    <td>1890</td>
    <td>210</td>
    <td>200</td>
    <td>70</td>
  </tr>
  <tr>
    <td>ESGBERT S</td>
    <td>1600</td>
    <td>200</td>
    <td>200</td>
    <td>200</td>
  </tr>
  <tr>
    <td>ESGBERT E</td>
    <td>1600</td>
    <td>200</td>
    <td>200</td>
    <td>200</td>
  </tr>
  <tr>
    <td>esgbert Biodiversity</td>
    <td>1760</td>
    <td>220</td>
    <td>220</td>
    <td>445</td>
  </tr>
  <tr>
    <td>esgbert Forest</td>
    <td>1760</td>
    <td>220</td>
    <td>220</td>
    <td>445</td>
  </tr>
  <tr>
    <td>esgbert Nature</td>
    <td>1760</td>
    <td>220</td>
    <td>220</td>
    <td>445</td>
  </tr>
  <tr>
    <td>esgbert Water</td>
    <td>1760</td>
    <td>220</td>
    <td>220</td>
    <td>445</td>
  </tr>
  <tr>
    <td>Environmental Claims</td>
    <td>2117</td>
    <td>265</td>
    <td>265</td>
    <td>76</td>
  </tr>
  <tr>
    <td>climatex (Wiki-doc) 10k</td>
    <td>6000</td>
    <td>300</td>
    <td>300</td>
    <td>282</td>
  </tr>
  <tr>
    <td>climatex (Wiki-doc) wiki</td>
    <td>3000</td>
    <td>300</td>
    <td>300</td>
    <td>282</td>
  </tr>
  <tr>
    <td>climate sentiment</td>
    <td>800</td>
    <td>200</td>
    <td>320</td>
    <td>698</td>
  </tr>
  <tr>
    <td>Climate Specificity</td>
    <td>800</td>
    <td>200</td>
    <td>320</td>
    <td>698</td>
  </tr>
  <tr>
    <td>Commitments And Actions</td>
    <td>800</td>
    <td>200</td>
    <td>320</td>
    <td>698</td>
  </tr>
  <tr>
    <td>Net-Zero/Reduction</td>
    <td>2752</td>
    <td>345</td>
    <td>344</td>
    <td>1436</td>
  </tr>
  <tr>
    <td>climateEng</td>
    <td>2871</td>
    <td>354</td>
    <td>355</td>
    <td>345</td>
  </tr>
  <tr>
    <td>climateStance</td>
    <td>2871</td>
    <td>354</td>
    <td>355</td>
    <td>345</td>
  </tr>
  <tr>
    <td>ClimateBERT’s Climate detection</td>
    <td>1040</td>
    <td>260</td>
    <td>400</td>
    <td>701</td>
  </tr>
  <tr>
    <td>Climate TCFD recommendations</td>
    <td>1040</td>
    <td>260</td>
    <td>400</td>
    <td>701</td>
  </tr>
  <tr>
    <td>climateFEVER evidence</td>
    <td>6140</td>
    <td>765</td>
    <td>770</td>
    <td>441</td>
  </tr>
  <tr>
    <td>climatex (Wiki-doc) claim</td>
    <td>6000</td>
    <td>1000</td>
    <td>1000</td>
    <td>282</td>
  </tr>
  <tr>
    <td>sdCC</td>
    <td>9224</td>
    <td>1154</td>
    <td>1161</td>
    <td>503</td>
  </tr>
  <tr>
    <td>ClimatINS</td>
    <td>13755</td>
    <td>1710</td>
    <td>1710</td>
    <td>5426</td>
  </tr>
  <tr>
    <td>CC-Contrarian Claims</td>
    <td>23436</td>
    <td>2605</td>
    <td>2898</td>
    <td>4677</td>
  </tr>
  <tr>
    <td>LobbyMap (Query)</td>
    <td>11728</td>
    <td>1320</td>
    <td>3817</td>
    <td>16743</td>
  </tr>
  <tr>
    <td>climatex (Wiki-doc)</td>
    <td>115847</td>
    <td>3618</td>
    <td>3826</td>
    <td>317</td>
  </tr>
  <tr>
    <td>LobbyMap (Stance)</td>
    <td>15038</td>
    <td>1660</td>
    <td>4718</td>
    <td>1643</td>
  </tr>
  <tr>
    <td>climaQA</td>
    <td>7148</td>
    <td>8934</td>
    <td>8936</td>
    <td>1065</td>
  </tr>
  <tr>
    <td>ClimaTOPIC</td>
    <td>46803</td>
    <td>8771</td>
    <td>8984</td>
    <td>1056</td>
  </tr>
  <tr>
    <td>LobbyMap (Pages)</td>
    <td>67091</td>
    <td>7289</td>
    <td>15755</td>
    <td>16743</td>
  </tr>
  <tr>
    <td>climateBUG-data</td>
    <td>96852</td>
    <td>24214</td>
    <td>29551</td>
    <td>4567</td>
  </tr>
</table>

Table 3: Original datasets: length of each split (train, test, dev) for each dataset, and size of the largest examples for each dataset (in number of tokens).

Duplicates in ClimaQA ClimaQA’s original construction method inadvertently produced many duplicates. The task of ClimaQA is to determine whether a given a CDP response answers a specific CDP question. The dataset contains correct response-question pairs (positive pairs) and mismatching response-question pairs (negative pairs). For the negative pairs, responses were randomly selected from the pool of all other responses, excluding the correct one. This naive approach led to mislabeled examples due to all companies responding to the same set of questions. Additionally, the issue of duplication is exacerbated as companies often reuse or copy responses across multiple questions, increasing the likelihood of selecting an appropriate answer as a negative example. The original ClimaQA dataset led to poor performances; we therefore rebuilt the dataset making sure to avoid those issues.

Duplicates in ClimateFEVER For ClimateFEVER, the original dataset does not have a split. However, in ClimaBench (Spokony et al., 2023), the authors created the split at the evidence level. The same claim can be found in the training and the testing split but associated to different evidences. This resulted in partial contamination. We experimented with both evidence and claim -based splits (ClimateFEVER (climabench split) and ClimateFEVER (our splt)).

B Experimental Settings

Each dataset was divided into train, test, and development splits. When already available, we kept the initial split; otherwise, we used proportions of 80%, 10%, 10%. The dataset construction is detailed in Appendix A. The datasets with original test split are climatex (wiki-doc, wikipedia, 10k, claim), Environmental Claims, CC-Contrarian Claims, Lobbymap (Pages, Query, Stance), ClimateEng, ClimateStance, LogicClimate, ClimatOPIC. For ClimatINS and ClimaQA we experimented with both the original splits and our splits.

To establish a benchmark for comparison, we computed several baselines using different statistical and machine-learning approaches:

Random Baseline A model that randomly predicts the label (following a uniform distri-
<table>
  <tr>
    <th rowspan="2">dataset</th>
    <th colspan="3">text duplicates</th>
    <th colspan="2">cleaned text duplicates</th>
    <th colspan="2">exact duplicates</th>
    <th colspan="2">Dataset Size</th>
  </tr>
  <tr>
    <th>text duplicates</th>
    <th>cleaned text duplicates</th>
    <th>exact duplicates</th>
    <th>text duplicates</th>
    <th>cleaned text duplicates</th>
    <th>exact duplicates</th>
    <th>text duplicates</th>
    <th>cleaned text duplicates</th>
    <th>exact duplicates</th>
  </tr>
  <tr>
    <td>ESGBERT E</td><td>61</td><td>79</td><td>73</td><td>2000</td><td></td><td></td><td></td><td></td><td></td>
  </tr>
  <tr>
    <td>CC-Contrarian Claims</td><td>20</td><td>32</td><td>30</td><td>28939</td><td></td><td></td><td></td><td></td><td></td>
  </tr>
  <tr>
    <td>climateBUG-data</td><td>17445</td><td>20965</td><td>18815</td><td>150617</td><td></td><td></td><td></td><td></td><td></td>
  </tr>
  <tr>
    <td>ClimaINS</td><td>2538</td><td>6986</td><td>6778</td><td>17175</td><td></td><td></td><td></td><td></td><td></td>
  </tr>
  <tr>
    <td>climateFEVER evidence</td><td>3473</td><td>3473</td><td>2625</td><td>7675</td><td></td><td></td><td></td><td></td><td></td>
  </tr>
  <tr>
    <td>logicClimate</td><td>458</td><td>458</td><td>113</td><td>1351</td><td></td><td></td><td></td><td></td><td></td>
  </tr>
  <tr>
    <td>Net-Zero/Reduction</td><td>0</td><td>14</td><td>14</td><td>3441</td><td></td><td></td><td></td><td></td><td></td>
  </tr>
  <tr>
    <td>Green Claims</td><td>0</td><td>0</td><td>0</td><td>773</td><td></td><td></td><td></td><td></td><td></td>
  </tr>
  <tr>
    <td>LobbyMap (Stance)</td><td>7837</td><td>7843</td><td>4365</td><td>21416</td><td></td><td></td><td></td><td></td><td></td>
  </tr>
  <tr>
    <td>Commitments And Actions</td><td>0</td><td>0</td><td>0</td><td>1320</td><td></td><td></td><td></td><td></td><td></td>
  </tr>
  <tr>
    <td>climaQA</td><td>74190</td><td>74584</td><td>33580</td><td>89348</td><td></td><td></td><td></td><td></td><td></td>
  </tr>
  <tr>
    <td>Environmental Claims</td><td>0</td><td>0</td><td>0</td><td>2647</td><td></td><td></td><td></td><td></td><td></td>
  </tr>
  <tr>
    <td>SUSTAINABLESIGNS reviews</td><td>4</td><td>4</td><td>2</td><td>779</td><td></td><td></td><td></td><td></td><td></td>
  </tr>
  <tr>
    <td>esgbert Biodiversity</td><td>79</td><td>79</td><td>69</td><td>2200</td><td></td><td></td><td></td><td></td><td></td>
  </tr>
  <tr>
    <td>climateStance</td><td>0</td><td>6</td><td>4</td><td>3580</td><td></td><td></td><td></td><td></td><td></td>
  </tr>
  <tr>
    <td>Implicit/Explicit Green Claims</td><td>0</td><td>0</td><td>0</td><td>773</td><td></td><td></td><td></td><td></td><td></td>
  </tr>
  <tr>
    <td>esgbert Water</td><td>79</td><td>79</td><td>79</td><td>2200</td><td></td><td></td><td></td><td></td><td></td>
  </tr>
  <tr>
    <td>esgbert Nature</td><td>79</td><td>79</td><td>79</td><td>2200</td><td></td><td></td><td></td><td></td><td></td>
  </tr>
  <tr>
    <td>climatext (Wiki-doc)</td><td>259</td><td>2650</td><td>2595</td><td>123291</td><td></td><td></td><td></td><td></td><td></td>
  </tr>
  <tr>
    <td>climatext (Wikipedia)</td><td>49</td><td>55</td><td>55</td><td>3600</td><td></td><td></td><td></td><td></td><td></td>
  </tr>
  <tr>
    <td>climatext (10k)</td><td>299</td><td>327</td><td>327</td><td>6600</td><td></td><td></td><td></td><td></td><td></td>
  </tr>
  <tr>
    <td>climatext (claim)</td><td>292</td><td>322</td><td>322</td><td>7300</td><td></td><td></td><td></td><td></td><td></td>
  </tr>
  <tr>
    <td>esgbert Forest</td><td>79</td><td>79</td><td>79</td><td>2200</td><td></td><td></td><td></td><td></td><td></td>
  </tr>
  <tr>
    <td>climateEng</td><td>0</td><td>6</td><td>4</td><td>3580</td><td></td><td></td><td></td><td></td><td></td>
  </tr>
  <tr>
    <td>sciDCC</td><td>87</td><td>87</td><td>37</td><td>11539</td><td></td><td></td><td></td><td></td><td></td>
  </tr>
  <tr>
    <td>LobbyMap (Pages)</td><td>2182</td><td>2303</td><td>2067</td><td>90135</td><td></td><td></td><td></td><td></td><td></td>
  </tr>
  <tr>
    <td>Climate TCFD recommendations</td><td>0</td><td>0</td><td>0</td><td>1700</td><td></td><td></td><td></td><td></td><td></td>
  </tr>
  <tr>
    <td>ESGBERT G</td><td>36</td><td>54</td><td>45</td><td>2000</td><td></td><td></td><td></td><td></td><td></td>
  </tr>
  <tr>
    <td>ClimaTOPIC</td><td>42</td><td>717</td><td>675</td><td>64558</td><td></td><td></td><td></td><td></td><td></td>
  </tr>
  <tr>
    <td>ESGBERT action500</td><td>19</td><td>21</td><td>21</td><td>500</td><td></td><td></td><td></td><td></td><td></td>
  </tr>
  <tr>
    <td>Global-Warming Stance (GWSD)</td><td>266</td><td>270</td><td>268</td><td>2300</td><td></td><td></td><td></td><td></td><td></td>
  </tr>
  <tr>
    <td>ESGBERT S</td><td>5</td><td>23</td><td>23</td><td>2000</td><td></td><td></td><td></td><td></td><td></td>
  </tr>
  <tr>
    <td>ClimateBERT’s Climate detection</td><td>0</td><td>0</td><td>0</td><td>1700</td><td></td><td></td><td></td><td></td><td></td>
  </tr>
  <tr>
    <td>climateFEVER claim (our split)</td><td>0</td><td>2</td><td>2</td><td>1535</td><td></td><td></td><td></td><td></td><td></td>
  </tr>
  <tr>
    <td>climate sentiment</td><td>0</td><td>0</td><td>0</td><td>1320</td><td></td><td></td><td></td><td></td><td></td>
  </tr>
  <tr>
    <td>Climate Specificity</td><td>0</td><td>0</td><td>0</td><td>1320</td><td></td><td></td><td></td><td></td><td></td>
  </tr>
  <tr>
    <td>climateFEVER evidence</td><td>3473</td><td>3473</td><td>2625</td><td>7675</td><td></td><td></td><td></td><td></td><td></td>
  </tr>
</table>

Table 5: Number of duplicates in the unfiltered datasets. text duplicates correspond to duplicates of the unprocessed texts, cleaned text duplicates correspond to duplicates of text after the formatting and encoding cleaning, exact duplicates correspond to duplicates of both the input text and the label. Therefore, the difference between cleaned text duplicates and exact duplicates correspond to duplicates with mismatching labels.

<table>
  <tr>
    <th>dataset</th>
    <th colspan="2">Imbalance ratio</th>
    <th colspan="2">Weighted</th>
  </tr>
  <tr>
    <th></th>
    <th>Original</th>
    <th>Filtered</th>
    <th>Test</th>
    <th>Training</th>
  </tr>
  <tr>
    <td>climaQA</td><td>1.0</td><td>1.0</td><td>1.0</td><td>1.0</td>
  </tr>
  <tr>
    <td>climatext (Wiki-doc)</td><td>1.0</td><td>1.0</td><td>1.0</td><td>1.0</td>
  </tr>
  <tr>
    <td>esgbert Nature</td><td>1.1</td><td>1.1</td><td>1.0</td><td>1.0</td>
  </tr>
  <tr>
    <td>SUSTAINABLESIGNS reviews</td><td>1.3</td><td>1.3</td><td>1.3</td><td>1.3</td>
  </tr>
  <tr>
    <td>ESGBERT action500</td><td>1.4</td><td>1.4</td><td>1.3</td><td>1.3</td>
  </tr>
  <tr>
    <td>Commitments And Actions</td><td>1.4</td><td>1.4</td><td>2.3</td><td>2.3</td>
  </tr>
  <tr>
    <td>ESGBERT S</td><td>1.5</td><td>1.5</td><td>1.5</td><td>1.5</td>
  </tr>
  <tr>
    <td>Climate Specificity</td><td>1.5</td><td>1.5</td><td>1.6</td><td>1.6</td>
  </tr>
  <tr>
    <td>Net-Zero/Reduction</td><td>1.6</td><td>1.6</td><td>1.6</td><td>1.6</td>
  </tr>
  <tr>
    <td>climateStance</td><td>1.6</td><td>1.6</td><td>1.6</td><td>1.6</td>
  </tr>
  <tr>
    <td>ClimaINS</td><td>1.7</td><td>1.7</td><td>2.1</td><td>2.1</td>
  </tr>
  <tr>
    <td>Green Claims</td><td>1.9</td><td>1.9</td><td>1.9</td><td>1.9</td>
  </tr>
  <tr>
    <td>ESGBERT E</td><td>2.0</td><td>2.1</td><td>2.2</td><td>2.2</td>
  </tr>
  <tr>
    <td>Global-Warming Stance (GWSD)</td><td>2.2</td><td>2.2</td><td>2.3</td><td>2.3</td>
  </tr>
  <tr>
    <td>climateBUG-data</td><td>2.3</td><td>2.3</td><td>1.3</td><td>1.3</td>
  </tr>
  <tr>
    <td>LobbyMap (Stance)</td><td>2.5</td><td>2.3</td><td>2.7</td><td>2.7</td>
  </tr>
  <tr>
    <td>ESGBERT G</td><td>2.7</td><td>2.7</td><td>2.8</td><td>2.8</td>
  </tr>
  <tr>
    <td>Environmental Claims</td><td>3.0</td><td>3.0</td><td>3.0</td><td>3.0</td>
  </tr>
  <tr>
    <td>esgbert Water</td><td>3.1</td><td>3.1</td><td>3.4</td><td>3.4</td>
  </tr>
  <tr>
    <td>ClimateBERT’s Climate detection</td><td>3.3</td><td>3.3</td><td>4.0</td><td>4.0</td>
  </tr>
  <tr>
    <td>esgbert Forest</td><td>4.1</td><td>4.1</td><td>4.3</td><td>4.3</td>
  </tr>
  <tr>
    <td>climateFEVER claim (our split)</td><td>4.3</td><td>4.2</td><td>4.0</td><td>4.0</td>
  </tr>
  <tr>
    <td>Implicit/Explicit Green Claims</td><td>4.3</td><td>4.3</td><td>4.2</td><td>4.2</td>
  </tr>
  <tr>
    <td>esgbert Biodiversity</td><td>4.7</td><td>4.8</td><td>4.9</td><td>4.9</td>
  </tr>
  <tr>
    <td>LobbyMap (Pages)</td><td>4.7</td><td>4.5</td><td>3.0</td><td>3.0</td>
  </tr>
  <tr>
    <td>climateFEVER evidence</td><td>6.3</td><td>2.4</td><td>5.6</td><td>5.6</td>
  </tr>
  <tr>
    <td>Climate TCFD recommendations</td><td>8.3</td><td>8.3</td><td>7.6</td><td>7.6</td>
  </tr>
  <tr>
    <td>climateStance</td><td>8.5</td><td>8.5</td><td>8.1</td><td>8.1</td>
  </tr>
  <tr>
    <td>climatext (wiki)</td><td>10.5</td><td>10.3</td><td>8.1</td><td>8.1</td>
  </tr>
  <tr>
    <td>climatext (eng)</td><td>12.0</td><td>12.1</td><td>23.1</td><td>23.1</td>
  </tr>
  <tr>
    <td>climatext (claim)</td><td>17.8</td><td>17.2</td><td>1.0</td><td>1.0</td>
  </tr>
  <tr>
    <td>climatext (10k)</td><td>17.8</td><td>17.2</td><td>3.5</td><td>3.5</td>
  </tr>
  <tr>
    <td>logcfev (wiki)</td><td>34.8</td><td>34.8</td><td>45.0</td><td>45.0</td>
  </tr>
  <tr>
    <td>ClimaTOPIC</td><td>44.3</td><td>34.4</td><td>26.7</td><td>26.7</td>
  </tr>
  <tr>
    <td>sciDCC</td><td>49.2</td><td>48.8</td><td>49.5</td><td>49.5</td>
  </tr>
  <tr>
    <td>LobbyMap (Query)</td><td>65.3</td><td>65.3</td><td>47.0</td><td>47.0</td>
  </tr>
  <tr>
    <td>CC-Contrarian Claims</td><td>110.9</td><td>19.5</td><td>83.2</td><td>83.2</td>
  </tr>
</table>

Table 6: Ratio between the most frequent and least frequent label in the dataset. The ratio is reported for the original dataset, then after filtering for token length and dataset size, as well as for the test set. For heavily imbalanced datasets, we used a weighted loss during training.

bution) implemented with sklearn’s DummyClassifier.

TF-IDF A logistic regression using TF-IDF features to predict the class, implemented with sklearn’s LogisticRegression and TfidfVectorizer.

DistilRoBERTa A DistilRoBERTa (Sanh et al., 2019) fine-tuned on each task implemented with transformers’ AutoModelForSequenceClassification for 10 epochs, with AdamW optimizer and a learning rate of 5e-5.

Longformer A Longformer (Beltagy et al., 2020) fine-tuned on each task implemented with transformers’ AutoModelForSequenceClassification for 10 epochs, with AdamW optimizer and a learning rate of 5e-5.

GPT-4o-mini GPT-4o-mini used as a zero-shot
classifier implemented with OpenAI’s API (temperature of 0.1, top_p of 1)

Llama 3.1 8B  Llama 3.1 8B used as a zero-shot classifier (temperature of 0.1, top_p of 1)

Llama 3.1 70B  Llama 3.1 70B used as a zero-shot classifier (temperature of 0.1, top_p of 1)

Additionally, we reported the performances reached in the original studies. Compiling and comparing reported performances from various papers revealed significant heterogeneity in performance metrics (Calamai et al., 2025). We reported the value in the result tables; however, they are not necessarily comparable.

Finetuning Parameters  We finetuned for each dataset a Longformer and a distilRoBERTa for 10 epochs (with early stopping based on the validation F1-score). To speed up the training, we used half-precision floating-point (fp16). To maximize the GPU resources, we used a batch of 16 for distil-RoBERTa and 7 for Longformer (with an accumulation step of 2). We set the seed/random state to 42 for every library (both to create the dataset and train the models). We tried setting parameters to deterministic, however, Longformer was not compatible with PyTorch deterministic algorithms. The models are fine-tuned on the training dataset, evaluated at each epoch on the development dataset, and the best model is selected using the development dataset. We used a learning rate of 5e-5 (warmup ratio of 0.1, a weight decay of 0.01) following Spokoyny et al. (2023)’s parameters choices.

Baselines  For the simple baselines, we used the seed 42. We used the scikit-learn implementations of the random classifier, the TF-IDF vectorizer, and the logistic regression. For the logistic regression, we used the weights to handle imbalanced datasets. For the relation classification, we parse both texts using the TF-IDF vectorizer and then concatenate the 2 vectors for the logistic regression. The model is fitted on the training dataset.

Zero-shot  For the zero-shot models, we collected the description of the labels from each paper and a description of the annotation task. When missing, we wrote a description. We then designed a corresponding prompt template. Using GPT-4, we generated all the instruction prompts for each dataset. We then generated the prediction for a small subset of each dataset (50 examples). Using the outputs, we validated the format of the prompts and ran the experiment on a larger dataset. We identified issues with some prompts that were manually updated. Finally, we experimented with both zero-shot and Chain-of-thought (CoT). If the performances on the subset were significantly improved, we selected the CoT prompt. The results using the CoT prompts are: climateFEVER, climateStance, climatext (10K, wiki, claims, wiki-doc), GWSD, LobbyMap.

C  Detailed Analysis of Performance

In this section, we deep dive into the performances of models and errors analysis for each dataset. For each task, we provide a general result analysis, both on the performance of the models and on the error annotation that we conducted.

C.1  Climate-Related Topic Detection

Task Description  Given an input sentence or a paragraph, output a binary label, “climate-related” or “not climate-related”.

Experiment  We collected all available datasets: climateBUG-Data (Yu et al., 2024), Climate-BERT’s climate detection (Bingler et al., 2023), ClimaText (Varini et al., 2020), and SUSTAINABLESIGNALS reviews (Lin et al., 2023). Climatext is composed of climatext wiki-doc composed of text weakly labeled; and climatext wikipedia, 10k and claim which are annotated datasets. For our experiments, we trained our models on climatext wikipedia training split as it was the larger annotated one. For climatext 10k we also used the training split for 10-Ks which contains only 58 positive samples. climatext claim did not contain a training split, so we used the same as for climatext 10k. Our cleaning and processing steps affected the climateBUG-data and all climatext datasets by removing duplicates, smaller/larger texts, and encoding issues. The results of the experiments are shown in Table 8.

Analysis  We found that the TF-IDF baseline achieves strong performance, with values between 65.8% and 86.4%. This is expected as some words are heavily associated with climate change such as “greenhouse gas” or “climate”. However, the baseline is outperformed by the fine-tuned transformers, with macro F1-scores between 73.7% and 95.8%. This difference is statistically significant for all datasets except SUSTAINABLESIGNALS reviews.

This shows that (1) the task relies on vocabu-
<table>
  <tr>
    <th rowspan="2">dataset</th>
    <th colspan="3">Errors in sample</th>
    <th colspan="6">Overlapping Errors</th>
  </tr>
  <tr>
    <th>SB</th>
    <th>70B</th>
    <th>GPT</th>
    <th>sSB</th>
    <th>GPT</th>
    <th>SB</th>
    <th>70B</th>
    <th>GPT</th>
  </tr>
  <tr><td>climate sentiment</td><td>9</td><td>7</td><td>10</td><td>56</td><td>84</td><td>54</td><td>86</td><td>68</td></tr>
  <tr><td>Environmental Claims</td><td>12</td><td>11</td><td>14</td><td>26</td><td>43</td><td>26</td><td>57</td><td>29</td></tr>
  <tr><td>Climate TCDF recommendations</td><td>7</td><td>7</td><td>10</td><td>179</td><td>230</td><td>166</td><td>207</td><td>211</td></tr>
  <tr><td>Implicitly Green Claims</td><td>5</td><td>9</td><td>12</td><td>5</td><td>16</td><td>9</td><td>16</td><td>12</td></tr>
  <tr><td>climatext (10k)</td><td>8</td><td>5</td><td>9</td><td>8</td><td>17</td><td>8</td><td>9</td><td>8</td></tr>
  <tr><td>Climate Specificity</td><td>8</td><td>11</td><td>16</td><td>58</td><td>75</td><td>75</td><td>102</td><td>91</td></tr>
  <tr><td>Green Claims</td><td>4</td><td>2</td><td>6</td><td>4</td><td>7</td><td>2</td><td>10</td><td>6</td></tr>
  <tr><td>climate data</td><td>12</td><td>14</td><td>20</td><td>81</td><td>238</td><td>71</td><td>124</td><td>112</td></tr>
  <tr><td>ClimaINS (our split)</td><td>9</td><td>9</td><td>11</td><td>123</td><td>195</td><td>117</td><td>172</td><td>142</td></tr>
  <tr><td>Global-Warming Stance (GWSD)</td><td>10</td><td>7</td><td>10</td><td>48</td><td>71</td><td>35</td><td>56</td><td>61</td></tr>
  <tr><td>ESGBERT actions500</td><td>8</td><td>7</td><td>10</td><td>8</td><td>15</td><td>7</td><td>17</td><td>10</td></tr>
  <tr><td>scDCS</td><td>9</td><td>10</td><td>10</td><td>597</td><td>726</td><td>570</td><td>700</td><td>666</td></tr>
  <tr><td>ClimateBERT’s Climate detection</td><td>11</td><td>12</td><td>17</td><td>11</td><td>33</td><td>12</td><td>16</td><td>17</td></tr>
  <tr><td>climatext (Wikipedia)</td><td>14</td><td>10</td><td>14</td><td>16</td><td>18</td><td>12</td><td>14</td><td>16</td></tr>
  <tr><td>climatext (Wiki-doc)</td><td>18</td><td>15</td><td>21</td><td>100</td><td>125</td><td>65</td><td>101</td><td>145</td></tr>
  <tr><td>No-Zero-Action</td><td>5</td><td>6</td><td>8</td><td>5</td><td>19</td><td>6</td><td>14</td><td>8</td></tr>
  <tr><td>climateStance</td><td>5</td><td>9</td><td>10</td><td>96</td><td>158</td><td>88</td><td>113</td><td>120</td></tr>
  <tr><td>CC-Contrarian Claims</td><td>7</td><td>8</td><td>10</td><td>359</td><td>588</td><td>322</td><td>473</td><td>424</td></tr>
  <tr><td>Committments And Actions</td><td>15</td><td>10</td><td>20</td><td>84</td><td>110</td><td>82</td><td>157</td><td>100</td></tr>
  <tr><td>esgbert Fossil</td><td>9</td><td>7</td><td>9</td><td>9</td><td>16</td><td>7</td><td>19</td><td>10</td></tr>
  <tr><td>esgbert Water</td><td>9</td><td>4</td><td>9</td><td>9</td><td>15</td><td>4</td><td>10</td><td>9</td></tr>
  <tr><td>climateFEVER evidence</td><td>7</td><td>8</td><td>10</td><td>218</td><td>357</td><td>182</td><td>251</td><td>286</td></tr>
  <tr><td>ClimateTOPIC</td><td>10</td><td>10</td><td>10</td><td>424</td><td>496</td><td>424</td><td>464</td><td>471</td></tr>
  <tr><td>ESGBERT E</td><td>5</td><td>5</td><td>8</td><td>5</td><td>8</td><td>5</td><td>9</td><td>8</td></tr>
  <tr><td>climateEng</td><td>8</td><td>6</td><td>10</td><td>61</td><td>106</td><td>71</td><td>99</td><td>101</td></tr>
  <tr><td>climaQA</td><td>13</td><td>10</td><td>20</td><td>96</td><td>379</td><td>94</td><td>222</td><td>188</td></tr>
  <tr><td>Total</td><td>242</td><td>219</td><td>317</td><td>2686</td><td>4167</td><td>2515</td><td>3488</td><td>3300</td></tr>
</table>

Table 7: This table presents the overlap between errors made by GPT-4o-mini and Llama models (8B and 70B). The first two columns shows the number of errors of Llama models among the sample of error of GPT-4o-mini (third column). The overall overlapping errors between GPT-4o-mini and Llama (8B) / Llama (70B) are reported in the following columns, along with the total number of errors for each model. (We do not include LogicClimate and LobbyMap because they are multilabel datasets)

<table>
  <tr>
    <th>Dataset</th>
    <th>Random</th>
    <th>TF-IDF</th>
    <th>Longformer</th>
    <th>DistilBERT</th>
    <th>GPT-4o-mini</th>
    <th>Llama</th>
    <th>Llama 70B</th>
    <th>Reference</th>
  </tr>
  <tr>
    <td>climateBUG-data (Yu et al., 2024)</td>
    <td>49.4(9.0-50.1)</td>
    <td>86.4(86.0-86.8)</td>
    <td>90.5(90.1-90.8)</td>
    <td>90.5(90.2-90.8)</td>
    <td>89.2(88.8-89.5)</td>
    <td>79.0(78.5-79.4)</td>
    <td>88.3(87.9-88.6)</td>
    <td>91.3(6)</td>
  </tr>
  <tr>
    <td>ClimateBERT CL det. (Bingler et al., 2023)</td>
    <td>42.8(38.0-47.8)</td>
    <td>79.3(74.3-83.8)</td>
    <td>95.8(93.9-98.0)</td>
    <td>94.0(91.6-96.9)</td>
    <td>93.2(90.0-96.3)</td>
    <td>88.3(84.3-91.8)</td>
    <td>93.4(90.1-96.4)</td>
    <td>99.1(5)</td>
  </tr>
  <tr>
    <td>climatext (Wiki-doc) (Varian et al., 2020)</td>
    <td>49.4(37.8-58.5)</td>
    <td>77.9(71.9-81.7)</td>
    <td>85.6(84.8-86.8)</td>
    <td>83.8(82.6-85.0)</td>
    <td>84.8(82.8-85.2)</td>
    <td>86.3(85.1-87.3)</td>
    <td>89.8(88.5-90.7)</td>
    <td>-</td>
  </tr>
  <tr>
    <td>climatext (10k) (Varian et al., 2020)</td>
    <td>48.2(45.1-51.2)</td>
    <td>74.7(71.7-77.3)</td>
    <td>97.0(96.6-97.9)</td>
    <td>75.4(72.5-78.2)</td>
    <td>82.7(79.7-84.6)</td>
    <td>83.6(81.1-85.1)</td>
    <td>85.7(83.7-87.3)</td>
    <td>95.1(4)</td>
  </tr>
  <tr>
    <td>climatext (Wikipedia) (Varian et al., 2020)</td>
    <td>40.1(35.4-44.9)</td>
    <td>83.5(75.9-90.4)</td>
    <td>81.9(76.0-90.4)</td>
    <td>88.2(81.4-94.0)</td>
    <td>84.7(77.1-91.5)</td>
    <td>82.8(74.8-89.9)</td>
    <td>87.4(80.4-93.5)</td>
    <td>80.1(4)</td>
  </tr>
  <tr>
    <td>Sustainable Signals rev. (Lin et al., 2023)</td>
    <td>37.6(27.0-49.0)</td>
    <td>65.8(54.7-76.6)</td>
    <td>73.2(63.2-83.3)</td>
    <td>73.7(62.6-82.7)</td>
    <td>69.7(57.6-79.7)</td>
    <td>63.5(52.5-73.9)</td>
    <td>73.3(62.6-82.4)</td>
    <td>-</td>
  </tr>
</table>

Table 8: Performances of baselines for each dataset using the train, test describe in section 3.3. The performance computed is the macro F1-score. The 95% confidence interval is computed using a 1000 bootstrap of sampled with replace as True of the size of the original test set. The reference performance is the best relevant performances reported in the original papers. 1. binary F1-score, 3. weighted average F1-score, c. climateBERT, f. climateBUG-LM, t. BERT.

lary, but (2) vocabulary is not enough to identify perfectly if the statement is about climate.

All zero-shot approaches reach performances above 75% showing that they can identify text as climate-related. The best zero-shot approaches even reached performances similar to the fine-tuned models. GPT-4o-mini and Llama 3.1 (70B) outperformed the fine-tuned models on climaText (Wiki), and slightly underperformed on climateBUG-data, ClimateBERT’s Climate Detection and SUSTAINABLESIGNS reviews. However, the difference between fine-tuning and zero-shot is not significant, except for Llama 3.1 70B outperforming significantly all fine-tuned models on climaText (Wiki).

The reference performances remain higher, with F1-scores above 90% (Garrido-Merchán et al., 2023; Bingler et al., 2023; Yu et al., 2024).

C.1.1 Error analysis

climatext (Wiki-Doc) When classifying sentences as climate-related or not, 91% of errors from GPT-4o-mini are FN. There are only 13 FP. 7 of them come from pages on topics that are indeed somewhat related to climate change, but that do not count as climate-related in Climatext’s methodology – for example, pages about lakes that will be impacted by climate change (Example C.1). Of the remaining FP, 4 are out-of-context statements and 2 are actual errors from GPT-4o-mini. As for the sampled FN, 6/10 are text that is only indirectly linked to climate change. For example, a page about the lithosphere (Example C.2) counts as climate-related in climatext’s methodology, but not in GPT-4o-mini’s view. In 4/10 FN are out-of-context statements. For example, a statistics about politics (Example C.3) counts as “climate-related”
in climatext’s methodology, but not in the view of GPT. It is important to note that the climatext (Wiki-Doc) is weakly labeled, therefore the actual task is to identify if a sentence is extracted from a Wikipedia page related to climate change. Some sentences are therefore out-of-context and impossible to correctly classify. This dataset could be better used as a only a training dataset.

Example C.1 (Not Climate-related). IISD Experimental Lakes Area (IISD-ELA) is a natural laboratory consisting of 58 small lakes and their watersheds set aside for scientific research.

Example C.2 (Climate-related). There are two types of lithosphere:

Example C.3 (Climate-related). The Senate vote throughout the states was between 10 and 20 percent.

ClimaText (Wikipedia/10-Ks) For climatext (Wikipedia) and climatext (10-Ks) the majority of errors are FN (respectively 75% and 89%).

There are actually only 4 FP for climatext (Wikipedia) and 1 for climatext (10-Ks). All the FP are linked to environmental issues, and depending on the context, could be linked to climate change. Either directly such as Example C.5 or indirectly such as Example C.4. We believe those errors are all ambiguous and debatable.

For the FN in climatext (wiki), we found 3 examples that are annotation errors (e.g. Example C.6 as “Acronyms, potentially well connected to climate change, must be mentioned along with some mechanism/cause/effect of climate change”). Those annotation errors occurred on examples that are ambiguous, yet precisely described in the guidelines. This shows that Human do also struggle with ambiguous statements. We found 2 errors that are due to our prompt which is a simplified version of the guidelines, therefore missing some nuance. 3 are due to indirect links to climate-change. The remaining 2 sampled errors are debatable. In the guidelines, it is mentioned that “Just mentioning clean energy, emissions, fossil fuels, etc. is not sufficient; rather it must be connected to an environmental (CO2) or societal aspect (divestment, Kyoto treaty) of climate change.”. Example C.7 was classified by the model as “not climate-related” while the annotator decided it was “climate-related”. While this example is definitely climate-related with the use of “zero-carbon”, when strictly following the guidelines, this could be labeled “not climate-related”.

For the FN in climatext (10-Ks), due to the high performances, there are actually only 8 FN. There are 2 actual errors due to the following reasoning: “However, it does not directly discuss the mechanisms, causes, or effects of climate change itself” (Example C.8). The model was anchored to the part of the prompt mentioning mechanisms, causes, or effects. 2 errors are debatable due to the issue mentioned for climatext (wiki). The remaining 4 errors are obvious annotation errors, most likely due to the Active Learning approach used to create the dataset. Those examples contain words such as “political climate” or “economic climate” (e.g. Example C.9). They were mislabeled in the dataset but correctly classified by GPT-4o-mini.

Example C.4 (Not Climate-related). Think-tanks such as the World Pensions Council (WPC) argued that the keys to success lay in convincing officials in the U.S. and China, by far the two largest national emitters:

Example C.5 (Not Climate-related). "Indirect effects include the fact that aerosols can act as cloud condensation nuclei, stimulating cloud formation."

Example C.6 (Climate-related). The location of UNFCCC talks is rotated by regions throughout United Nations countries.

Example C.7 (Climate-related). As of 2012, France generated over 90% of its electricity from zero carbon sources, including nuclear, hydroelectric, and wind.

Example C.8 (Climate-related). In addition, we cannot control the actions of our third party manufacturers or the public’s perceptions of them, nor can we assure that these manufacturers will conduct their businesses using climate change proactive or sustainable practices.

Example C.9 (Climate-related). The current economic climate, especially in Europe, may have an adverse effect in the markets in which we operate.

ClimateBERT’s climate detection For ClimateBERT’s climate detection dataset, there are 11 FP and 6 FN. The majority of FP (7/11) stem from the use of the ambiguous word “sustainable”. Indeed, “sustainable” can be used in a business context meaning that the business is economically viable (Example C.10). However, some of those FP (3) mentioning “sustainable” are actually debatable (e.g. Example C.11 in which “sustainable concepts” is likely to refer to environmental sustainability).
In the remaining FP, 2 are actually mislabeled, and 1 is debatable.

Of the 6 FN, 4 are indirectly but not explicitly linked to climate change or the environment. For example, the dataset labels as climate-related some statements about the fossil fuel/energy industry that do not mention anything regarding the environmental impact, policy, or governance (Example C.12). It also labels as positive statements about the governance of sustainability-related activity (Example C.13).

Example C.10 (Not Climate-related). With passion and integrity at the heart of everything we do, we aim to build a sustainable business that makes a positive difference for all, now and in the future.

Example C.11 (Not Climate-related). We have a long heritage of innovation and strive to provide athletes with the best by creating high-performance and competitive products. In 2020, we continued to serve consumers with innovative technologies and sustainable concepts built into our products.

Example C.12 (Climate-related). On January 5, 2021, Centrica plc. closed a transaction to sell its entire ownership interest in Direct Energy to NRG Energy Inc. (NRG). Effective January 5, 2021, NRG provided a $300 million guarantee, supported by a $300 million letter of credit for Direct Energy’s obligations to ATCO Gas and ATCO Electric under the transaction agreements.

Example C.13 (Climate-related). The diagnosis of organizational culture represents the internal scenario and it is one of the elements considered/analyzed in the definition of the drivers. Since 2017, after reviewing the research strategy by our organizational behavior area, only the Organizational Climate and Engagement Survey was applied annually, as presented in the table below.

ClimateBUG In this dataset, GPT produces FN (54%) and FP (46%) in roughly equal proportion. In the sampled errors, we found that 2 FN are statements where the link to climate comes from the context – which is however not given (Example C.14). As in ClimateBERT’s climate detection, the word “sustainability” caused problems. However, in this case, the annotation guidelines are still ambiguous. It mentions that “Sustainability not related to the environment (i.e. sustainable profits)” should be classified as “Not Climate”. However, when not given sufficient context, the word can still be ambiguous. Some examples mention “sustainability” in a broader sense (without explicitly mentioning environmental/climate sustainability), and they are labeled positive (Example C.15) – although GPT believes they are negative. Vice versa, other broad references to sustainability are labeled as negative (Example C.16) – and GPT believes they are positive. This issue accounts 5/10 FP. This shows that both annotators and GPT struggle with this ambiguity and highlights inconsistencies in the annotations. The remaining FP (5/10) are climate-related tables that GPT-4o-mini classified as positives (therefore not following the annotation guidelines to classify tables as negatives). However, we also found at least 1 table mislabeled in the gold standard (Examples C.17). This also shows the importance of well-defined labels and label consistency during annotation, as it can introduce mislabeled examples. This is particularly important when comparing performances of 88.8% and 90.5%.

Example C.14 (Climate-related). This also gives us the opportunity to exchange knowledge and experiences with representatives from different spheres of society and to implement the principles and reach our targets.

Example C.15 (Climate-related). After all, they are the source of jobs, innovation, sustainability, and prosperity.

Example C.16 (Not Climate-related). Firstly, Nordic customers are relatively advanced in their sustainability considerations, and they expect their bank to be able to support sustainable development across segments.

Example C.17 (Climate-related). SCOPE 2—Electricity consumed (market-based method)—Electricity consumed (location-based method)3, 8 67.

C.2 Climate-Related Subtopic Detection

Task Description Given an input sentence or a paragraph, output a subtopic related to climate change. This is a multiclass classification task.

Experiment We collected the ClimaTOPIC (Spokoyny et al., 2023), esgbert Nature (Schimanski et al., 2024a), ClimateEng (Vaid et al., 2022) and SciDCC (Mishra and Mittal, 2021) datasets. The results are in Table 9. Our cleaning and processing steps impacted all datasets: it removed duplicated examples from the esgbert Nature and ClimaTOPIC test sets, and it removed the URLs in climateEng. And for SciDCC, we made sure that each
category was represented in the test set; however, some categories are severely under-represented (Geology appears only 28 times overall, and therefore only 3 times in the test set). We note that no original split is given for SciDCC.

Result Analysis In our experiment, the best-performing models are usually the fine-tuned models (except on Biodiversity label from esgbert Nature). While the performance of the fine-tuned models improved over the TF-IDF baseline, the difference is statistically significant only for ClimaTOPIC, esgbert Nature (nature and water labels). Overall, this shows that the classes are distinguishable using mostly the vocabulary.

The zero-shot approaches perform comparably to the fine-tuned transformers on most datasets. While the fine-tuned models outperformed the zero-shot approaches, the difference is statistically significant only on SciDCC, ClimaTOPIC. Those two datasets are constructed automatically and not annotated by humans; therefore, building a comprehensive prompt is difficult, while identifying linguistic patterns remains effective.

In terms of order of magnitude, our models achieved performance levels comparable to those reported in the original studies. However, we observed a notable performance gap on ClimaTOPIC and SciDCC, where our models underperformed relative to the results presented in prior work. For ClimaTOPIC this could be explained by the smaller training dataset. For SciDCC, we provide only the summary of the article, but the dataset also contains the content of the article, which can help increase performance.

C.2.1 Error analysis

SciDCC We have reported the confusion matrix of GPT-4o-mini predictions in figure 1a. We observe several types of errors: (1) hallucinations (“Marine Biology” and “Archaeology” are not labels in this dataset) (2) predicting too frequently the label “Climate” (3) mixing related labels (“biology”, “Biotechnology” and “Genetically Modified”, or “Geology” and “Earthquakes”)

All these labels are so close in meaning because they are categories that changed over time as shown in Figure 1b. We can observe the topic shift over time: “Ozone Holes”, “GMOs” and then “Global Warming”. Categories not only appeared but also changed over time. For example, “endangered animals” topics might have later been included in “Animals” or “Zoology”. Moreover, multiple categories might fit a specific topic such as “Extinction” and “Climate” as one is the consequence of the other. Moreover, in the original paper, they did not provide a clear definition of each label. Examples of surprising gold labels include: Example C.18 classified as “Geography” and not “Pollution”, or Example C.19 classified as “Geography” and not “Geology”.

When reviewing sampled errors by GPT-4o-mini, we identified that 11/20 are due to text with multiple possible labels (Examples C.21 which could fit in “Animals”, “Endangered Animals”, “Extinction” or “Climate”), 7/20 contain debatable labels (Examples C.20 labeled “Earthquakes” but could fit in “Geology”). The latter can be explained as in our experiment we only provide the summary, but the article’s content might be linked to the original label. While this dataset might be interesting for studying the evolution of environmental topics, it would require human annotations to re-label the data to make it reliable.

Example C.18 (Geography). Shipping traffic can be a major source of tiny plastic particles floating in the sea, especially out in the open ocean. In a paper published in the scientific journal

Example C.19 (Geography). New research led by the University of Cambridge has found rare evidence – preserved in the chemistry of ancient rocks from Greenland – which tells of a time when Earth was almost entirely molten.

Example C.20 (Earthquakes). Diamonds, those precious, sparkling jewels, are known as the hardest materials on Earth. They are a high-pressure form of carbon and found deep in the ground.

Example C.21 (Animals). Many species might be left vulnerable in the face of climate change, unable to adapt their physiologies to respond to rapid global warming. According to a team of international researchers, species evolve heat tolerance more slowly than cold tolerance, and the level of heat they can adapt to has limits.

ESGBERT Nature GPT-4o-mini performs well overall, but when errors occur, they tend to be FN (90% for forest, 77% for water and 60% for biodiversity), especially in cases involving indirect references. There are 1 FP and 9 FN for the “forest” dataset. All 9 FN were indeed related to forests, but they were misclassified due to a lack of explicit references to forest management. Instead,
(a) Confusion matrix of GPT-4o-mini predictions on SciDCC dataset. “Archeology” and “Marine Biology” are not labels from SciDCC.

(b) Evolution of proportion of labels over time in SciDCC dataset (Mishra and Mittal, 2021)
<table>
  <tr>
    <th>Dataset</th>
    <th>Random</th>
    <th>TF-IDF</th>
    <th>Longformer</th>
    <th>DistilRoBERTa</th>
    <th>GPT-4o-mini</th>
    <th>Llama</th>
    <th>Llama 7B</th>
    <th>Reference</th>
  </tr>
  <tr>
    <td>ClimaTOPIC (Strokosky et al., 2023)</td>
    <td>6.0/5-6.6/5</td>
    <td>46.6/45.8-46.9/48.1</td>
    <td><b>55.8/54.3-57.2/57.1</b></td>
    <td><b>54.8/53.5-56.1/56.1</b></td>
    <td>55.8/54.3-56.7/56.7</td>
    <td>34.5/33.4-35.4/35.4</td>
    <td>30.5/29.5-31.5/31.5</td>
    <td>65.22*</td>
  </tr>
  <tr>
    <td>climateEng (Vaid et al., 2023)</td>
    <td>4.1/4.1-16.2/16.2</td>
    <td>58.4/58.4-36.8/36.8</td>
    <td>70.5/60.8-47.7/47.7</td>
    <td>67.3/58.3-37.5/37.5</td>
    <td>67.3/58.3-37.5/37.5</td>
    <td>61.1/59.1-46.2/46.2</td>
    <td>61.1/59.1-46.2/46.2</td>
    <td>54.58*</td>
  </tr>
  <tr>
    <td>esgbert Biodiversity (Schmanksi et al., 2024a)</td>
    <td>41.1/35.4-47.7/47.7</td>
    <td>91.3/85.3-39.6/39.6</td>
    <td>91.3/85.3-39.6/39.6</td>
    <td>91.3/85.3-39.6/39.6</td>
    <td>91.3/85.3-39.6/39.6</td>
    <td>88.8/83.0-37.8/37.8</td>
    <td>88.8/83.0-37.8/37.8</td>
    <td>92.29*</td>
  </tr>
  <tr>
    <td>esgbert Forest (Schmanksi et al., 2024a)</td>
    <td>41.4/35.4-47.2/47.2</td>
    <td>92.9/87.4-49.1/49.1</td>
    <td>97.2/93.8-39.3/39.3</td>
    <td>97.0/93.8-39.4/39.4</td>
    <td>97.0/93.8-39.4/39.4</td>
    <td>87.4/81.2-39.5/39.5</td>
    <td>86.2/80.3-39.1/39.1</td>
    <td>95.37*</td>
  </tr>
  <tr>
    <td>esgbert Nature (Schmanksi et al., 2024a)</td>
    <td>51.5/45.0-58.0/58.0</td>
    <td>82.9/77.7-57.7/57.7</td>
    <td>89.8/85.8-39.3/39.3</td>
    <td>89.9/85.8-39.3/39.3</td>
    <td>89.9/85.8-39.3/39.3</td>
    <td>86.7/82.0-39.0/39.0</td>
    <td>88.0/83.3-39.2/39.2</td>
    <td>94.19*</td>
  </tr>
  <tr>
    <td>esgbert Water (Schmanksi et al., 2024a)</td>
    <td>42.7/36.6-49.5/49.5</td>
    <td>84.6/77.9-50.8/50.8</td>
    <td>95.3/91.7-39.6/39.6</td>
    <td>93.9/89.7-39.5/39.5</td>
    <td>93.8/89.7-39.6/39.6</td>
    <td>89.4/83.4-39.1/39.1</td>
    <td>93.5/89.1-39.6/39.6</td>
    <td>95.10*</td>
  </tr>
  <tr>
    <td>sciDCC(Mishra and Mittal, 2021)</td>
    <td>3.7/2.7-4.8/4.8</td>
    <td>42.6/39.0-46.7/46.7</td>
    <td>42.6/39.0-46.7/46.7</td>
    <td>42.6/39.0-46.7/46.7</td>
    <td>29.2/26.8-31.4/31.4</td>
    <td>20.5/18.3-22.7/22.7</td>
    <td>30.0/27.5-32.2/32.2</td>
    <td>54.79*</td>
  </tr>
</table>

Table 9: Performances of baselines for each dataset using the train, test describe in section 3.3. The performance computed is the macro F1-score. The 95% confidence interval is computed using a 1000 bootstrap of sampled with replace as True of the size of the original test set. The reference performance is the best relevant performances reported in the original papers. l. binary F1-score, c. climateBERT, g. EnvironmentalBERT, l. Longformer, r. RoBERTa

these statements included discussions about plants, wood usage, the economic impact of wildfires, and the effects of climate change—topics inherently connected to forestry (Example C.22).

Example C.22. While gross loss reserve estimates for the 2018 California wildfires were also reduced, this was largely offset by a reduction in reinsurance recoverables resulting in very little change to estimated net losses from those wildfires. [Forest]

There are 2 FP and 7 FN for the “water” dataset. The 2 FP are both debatable as they explicitly talk about water-related topics (Example C.23). As part of the FN, we found 2 focusing on financial consequences, therefore indirectly linked to “water”. The “water” label: “The topic of water centers around water management, consumption, and pollution. [...]” could be understood as focusing only on freshwater, and not include water-related natural disasters such as tsunami. We found that 4 FN are related to those topics (Example C.24). The last FN is an actual error by GPT-4o-mini.

Example C.23. 70% of the water in the world is used for agriculture. [Not Water]

Example C.24. In addition, Intel employees responded to these events with great generosity, contributing $1.8 million toward tsunami relief, $1.6 million to help the victims of Hurricane Katrina and $475,000 for those affected by the earthquake in Pakistan.

While this dataset is well-built, these errors highlight the need to have precise definitions and the ambiguity of indirect relation to the topic. This is further emphasized by the inter-annotator agreement (Fleiss’ \( \kappa = 80\% \)) showing some disagreement between annotators.

ClimateEng Firstly we see that the performance of the GPT-4o-mini model is close to the performances of the finetuned models. We have reported the confusion matrix in figure 2, we see that a large portion of issues come from the labels "General" and "Politics" being mixed-up.

![Confusion matrix of GPT-4o-mini predictions on ClimateEng dataset.](page_1012_1342_377_246.png)

Figure 2: Confusion matrix of GPT-4o-mini predictions on *ClimateEng* dataset.

When investigating the sampled errors, we found that 12/30 are actual errors, 4/30 are debatable and 12/30 are mislabeled examples (Example C.25). We found that the actual errors are related to the very specific definitions of the labels. For example, the political label focuses only on leaders, political organizations and awareness about climate change and not on political activism or opinions. However, we see inconsistencies in the annotations for political awareness: Example C.27 was annotated as “Politics” - while GPT-4o-mini predicted “General” - and the Example C.28 was annotated as “General” - while GPT-4o-mini predicted “Politics”. 2 tweets contain hashtags about COP25, which could fit into the “Politics” category but were annotated as “General”. In the guidelines, the “Ocean/Water” label is specific toward “biodiversity”, but GPT-4o-mini predicted Example C.26 as “Ocean/Water”, therefore not following the instructions. *Overall, we observe that the labels are really specific and
counter-intuitive compared to the usual definition of the categories. GPT-4o-mini tend to struggle with this re-definition of terms. However, the inter-annotator agreement (Cohen’s \( \kappa = 0.739 \)) also shows that people do have disagreements and also struggle with those definitions. Once again, this shows the importance of having well-defined labels.

Example C.25 (Agriculture/Forestry). Contracts awarded for Scottish offshore wind farm <URL> #energy #sustainability #climatechange

Example C.26 (General). Scientists fear that a change in ocean circulation could profoundly alter climate patterns. <URL>

Example C.27 (Politics). U.S. students excel in global warming awareness, social justice activism, racial angst, and gender sensitivity while China’s teens waste their time gaining proficiency in reading, math and science, according to global education study. <URL>

Example C.28 (General). On December 11, 2019, the Intergenerational Dialogue was successfully held with the participation of Youth Delegation and a series of climate experts. The aim of the dialogue is to exchange personal experience and guide the delegates to the most fertile areas of climate change. <URL>

ClimaTOPIC As the topics are categories of questions from CDP questionnaires, many categories are actually related. The Example C.29 and C.30 both deal with emissions of buildings, however, one is from the “Emissions” category and the other is from the “Building” category. This might be improved by giving more context to the model, in particular, the type of questions available in each category.

From the sample of errors, we found that 9/10 errors are similar mistakes. From the confusion matrix (Figure 3), we see that the model fails most often on “Governance and Data Management”, “Opportunities” and “Strategy” labels which are non-specific categories. While we cannot consider the data to be mislabeled—since the labels correspond to the section headers from which the responses were extracted—it is evident that the model’s predictive performance is constrained by the broad and non-specific nature of these labels.

Example C.29 (Emissions). The Direct emission from institutional buildings occur but we are not able to estimate it on this inventory, due to financial constraints and limited manpower.

Example C.30 (Building). So as to decrease GHG emissions, under the climate action plan for reducing GHG emissions, the target of the energy upgrade of municipal buildings has been set, tackling energy efficiency.The emission reduction target has been set to 27,804 tCO2eq annually.

C.3 Detecting Climate-Related Financial Disclosure

Task Description This is a particular case of sub-topic classification. Given a paragraph or a sentence, output the TCFD recommendation associated with the content of the text.

Experiment We were not able to collect any of the datasets; however, the authors of Bingler et al. (2021) released a smaller TCFD annotated dataset alongside their follow-up work (Bingler et al., 2023). We identify it as Climate TCFD recommendations. So we evaluated this dataset with the setting describe in section B and reported the results in Table 10. Our pre-processing did not alter this dataset.

Result Analysis The best performing approaches were the fine-tuned transformers reaching F1-score above 68%, significantly out-performing the zero-shot approaches and the TF-IDF baseline. The TF-IDF baseline is performing relatively well given the nature of the task. Moreover, it performed better than the zero-shot approaches.

As this dataset is not part of a previous study, we rely on other works for performance comparison. Bingler et al. (2021) observed that paragraph-level approaches underperformed compared to methods that aggregate sentence-level features, such as majority voting or logistic regression. In contrast, Sampson et al. (2022) found that classical models, such as TF-IDF-based random forests and stacked approaches, outperformed fine-tuned transformers, which performed well but not optimally. Interestingly, the zero-shot approach evaluated by Auzepy et al. (2023) yielded surprisingly strong results given the complexity of the task.

C.3.1 Error analysis
climate TCFD recommendations The results, presented in the confusion matrix in Figure 4, revealed that 62% of the model’s classification errors are associated with the "Strategy" label. Additionally, the model tends to over-predict the "None"
Figure 3: Confusion matrix of GPT-4o-mini predictions on ClimaTOPIC dataset.

<table>
  <tr>
    <th>Dataset</th>
    <th>Random</th>
    <th>TF-IDF</th>
    <th>Longformer</th>
    <th>DistilRoBERTa</th>
    <th>GPT-4o-mini</th>
    <th>Llama</th>
    <th>Llama 70B</th>
    <th>Reference</th>
  </tr>
  <tr>
    <td>Climate TCFD rec.(Bingler et al., 2023)</td>
    <td>16.5(12.9-20.3)</td>
    <td>56.5(50.8-61.9)</td>
    <td><b>69.4(64.0-74.4)</b></td>
    <td><b>68.5(61.0-73.5)</b></td>
    <td>47.5(42.2-53.0)</td>
    <td>43.2(37.5-48.0)</td>
    <td>50.2(44.4-55.2)</td>
    <td>-</td>
  </tr>
</table>

Table 10: Performances of baselines for each dataset using the train, test describe in section 3.3. The performance computed is the macro F1-score. The 95% confidence interval is computed using a 1000 bootstrap of sampled with replace as True of the size of the original test set. The reference performance is the best relevant performances reported in the original papers.

category, which accounts for 47.4% of the total errors.

In our sampled errors, we found that 15/30 misclassifications are genuine errors, with 4/30 stemming from the model’s inability to correctly identify strategic actions or cases where the strategic nature of the text is implied but not explicitly stated. The use of a larger model (GPT-4) mitigated this issue to some extent, reducing the rate of strategy-related errors by 75%.

Furthermore, there is notable confusion between the “Strategy” and “Risk” categories. This can be explained by the by the guidelines. The “strategy” label includes descriptions of the climate-related risks, while “risk” includes more specifically risk management, which therefore exclude descriptions of the risks. Moreover, it seems that the “strategy” label is used as “default” label for vague statement mentioning climate-related topics (2/30). Example C.32 and C.33 are both classified as “strategy” in the gold standard, yet they do not describe climate-related risks and impacts on the company. Overall, we believe that there is a significant overlap between the strategy and risk labels, as describing the risk and describing how the risk is managed are often discussed in the same sentences.

We found 5 examples of texts not focusing on climate-related disclosure but more broadly about sustainability. These examples seem to indicate that this should be considered as related to climate-disclosure. However, even if it mentions sustainability, Example C.31 is mainly about plastic pol-
lution, which we considered as an annotation error.

We also found the issues mentioned on other task: mentioning energy is classified as not climate-related by gpt-4o-mini but systematically climate-related in the dataset (2/30).

Example C.31 (governance). Report 2019 (SR 2019), as requested by the shareholders. The hard-copies will be delivered once they are made available to the Company. * Nevertheless, we hope you would consider the environment before you decide to print the above reports or request for the printed copy of the IR 2019, GFR 2019 and SR 2019. The environmental concerns like global warming, deforestation, climate change and many more affect every human, animal and nation on this planet.

Example C.32 (strategy). The Sustainability Report is based upon the internationally recognized Global Reporting Initiative (GRI) Standards. Our reporting is also guided by the Sustainability Accounting Standards Board (SASB) and the Financial Stability Board’s Task Force on Climate-related Financial Disclosures’ (TCFD) recommendations.

Example C.33 (strategy). The progress toward all Green Company targets is tracked through an environmental data reporting system and is disclosed in detail in our annual Green Company Report, available on our corporate website as of spring 2021. ! > ADIDAS-GROUP.COMENVIRONMENTAL-APPROACH Own operations: Progress toward 2020 targets

Example C.34 (governance). Report 2019 (SR 2019), as requested by the shareholders. The hard-copies will be delivered once they are made available to the Company. * Nevertheless, we hope you would consider the environment before you decide to print the above reports or request for the printed copy of the IR 2019, GFR 2019 and SR 2019. The environmental concerns like global warming, deforestation, climate change and many more affect every human, animal and nation on this planet.

C.4 Detecting Environmental, Social, and Governance Disclosure

Task Description This is a particular case of sub-topic classification. The topics are “Environment”, “Social”, and “Governance” categories.

Experiment The only openly available datasets are ESGBERT E (Environmental) ESGBERT S (Social) and ESGBERT G (Governance) datasets (Schimanski et al., 2023b). We evaluated this dataset with the setting described in section B and reported the results in Table 11. Our pre-processing pipeline slightly impacted the dataset when removing duplicates.

Result Analysis In our experiments, we found that for the ESGBERT E, S and G datasets, the best performing model is DistilRoBERTa. As we reported the macro average, we also computed the binary F1-score 95%, 88%, 76% (respectively E, S and G), which are close to the performance reported in the original study: 93%, 92%, 79%. This shows that fine-tuned models can identify text relevant to those E, S and G categories. However, we also see that the TF-IDF baseline performances is over 80% for each dataset, showing that those thematic have largely different vocabulary, making them easily identifiable. Finally, the zero-shot approaches also performed well on those datasets, with performances above 78% for GPT-4o-mini. Showing that zero-shot models can also identify those categories. Interestingly, the only dataset where the fine-tuned model performed significantly better than GPT-4o-mini is the “Social” dataset.

Moreover, Schimanski et al. (2023b) measured a high inter-annotator agreement (Fleiss’ \( \kappa \) of more than 86%). Overall, this shows that those labels are quite easy to distinguish.

C.4.1 Error analysis

ESGBERT E We are only reviewing errors from ESGBERT E as it is the most relevant to our topic. GPT-4o-mini made only 8 errors. 6 FN and 2 FP. 2/6 FN are text mentioning the energy industry (Example C.35) but not directly talking about the environment. 1/6 FN is truncated, 1/6 FN is out-of-context, which makes them difficult to classify.

![Confusion matrix of GPT-4o-mini predictions on Climate TCFD recommendations dataset.](page_1012_1342_340_246.png)

Figure 4: Confusion matrix of GPT-4o-mini predictions on *Climate TCFD recommendations* dataset.
<table>
  <tr>
    <th>Dataset</th>
    <th>Random</th>
    <th>TF-IDF</th>
    <th>Longformer</th>
    <th>DistilRoBERTa</th>
    <th>GPT-4o-mini</th>
    <th>Llama</th>
    <th>Llama 70B</th>
    <th>Reference</th>
  </tr>
  <tr>
    <td>ESGBERT (Schmanskii et al., 2023b)</td>
    <td>42.6/35.1-49.3</td>
    <td>88.1/82.5-92.8</td>
    <td>95.8/92.3-98.7</td>
    <td>95.8/92.3-98.7</td>
    <td>95.0/91.3-98.2</td>
    <td>95.1/91.2-98.2</td>
    <td>94.6/90.8-97.6</td>
    <td>93.19<sup>a</sup></td>
  </tr>
  <tr>
    <td>ESGBERT (Schmanskii et al., 2023b)</td>
    <td>53.5/46.4-60.6</td>
    <td>80.2/73.5-86.1</td>
    <td>78.2/71.4-85.5</td>
    <td>83.5/77.0-89.7</td>
    <td>83.5/77.0-89.7</td>
    <td>73.6/69.0-81.3</td>
    <td>62.6/53.5-69.0</td>
    <td>78.86<sup>c</sup></td>
  </tr>
  <tr>
    <td>ESGBERT (Schmanskii et al., 2023b)</td>
    <td>54.4/47.0-60.8</td>
    <td>82.6/76.8-88.1</td>
    <td>88.3/83.7-92.6</td>
    <td><b>89.6/85.3-93.7</b></td>
    <td>78.1/72.4-84.1</td>
    <td>77.9/71.5-83.4</td>
    <td>73.6/67.3-79.8</td>
    <td>91.90<sup>c</sup></td>
  </tr>
</table>

Table 11: Performances of baselines for each dataset using the train, test describe in section 3.3. The performance computed is the macro F1-score. The 95% confidence interval is computed using a 1000 bootstrap of sampled with replace as True of the size of the original test set. The reference performance is the best relevant performances reported in the original papers. * binary F1-score, e. ESGBERT

We also found 1/6 FN mislabeled example talking about the "office environment" (Example C.36).
Overall, the errors stem from statement that are systematically classified as related to the environment such as anything related to energy, even if the statement is not explicitly talking about the environment. For the 2 FP, 1/2 is related to the use of “sustainable” in an economic context, and 1/2 describes the impact of floods on mining operations which was classified by GPT-4o-mini as “Environment” but not in the gold standard.

Example C.35 (Environment). e Natural gas liquids for Russia are included in crude oil.

Example C.36 (Environment). The pandemic may also have long-term effects on the nature of the office environment and remote working, which may result in increased costs and present operational and workplace culture challenges that may also adversely affect our business.

C.5 In-depth Disclosure: Climate Risk Classification

Task Description Given an input sentence or a paragraph, output “opportunity” or “risk” label.

Experiment We collected climateBERT’s climate sentiment dataset (Bingler et al., 2023). We evaluated this dataset with the setting described in section B and reported the results in Table 12. Our pre-processing did not alter this dataset.

Result Analysis In our experiments, as reported in Table 12, on climate sentiment, the best performing solutions are the fine-tuned transformers, with Longformer reaching a macro F1-score of 79.9%. The performance reported in the original study was similar, with a weighted F1-score of 83.8% (Bingler et al., 2023), within the confidence interval of our best performing model. The TF-IDF baseline is performing lower but still good, with a macro F1-score of 69%. This performance shows that the vocabulary for risk and opportunity is quite characteristic and identifiable. The zero-shot approaches’ performances varied from 70.5% for Llama 3.1 70B to 77.3% for GPT-4o-mini approaching the performance of finetuned models.

As shown in the confusion matrix Figure 5, only 4 out of the 68 errors do not include the label neutral (5 out of 68 for distilRoBERTa). This shows that discriminating between polar opposite is easy, but the edge cases can be more challenging. If the majority class is the neutral class, then the performances are not representative of the actual performance of the model. This is further confirmed by the relatively lower inter-annotator agreement compared to previous tasks Friederich et al. (2021) reported a low IAA (\( \alpha = 0.20 \)) and Bingler et al. (2023) reported a moderate IAA (Krippendorff’s \( \alpha = 0.61 \)).

C.5.1 Error analysis

climate sentiment As shown in the confusion matrix displayed in Figure 5, most of the errors are on texts originally labeled as “neutral” (65% of errors). And the model tends to preferably output the “opportunity” label (43% of errors) reducing its precision.

In the sampled errors, we identified several patterns of misclassifications. 2/30 instances involved discussions of corporate ambitions (Example C.37) that were labeled as “opportunity” by GPT-4o-mini, but categorized as “neutral” in the gold standard. These texts are not specifically opportunities, but the mention of “ambitious targets” could be considered “positive” in the guidelines due to: “2) and/or about positive impact of an entity’s activities on the society/environment 3) and/or associates specific positive adjectives to the anticipated, past or present developments and topics covered”. 3/30 sampled errors contained discussions of both risks and opportunities (Example C.39), highlighting the lack of clarity in the annotation guidelines for such mixed content. Interestingly, 2/30 examples referenced indirect risks (Examples C.40 and C.41), which were labeled as “neutral” in the gold standard, yet identified as “risk” by the model. Lastly, we found 14/30 instances that were likely mis-
<table>
  <tr>
    <th>Dataset</th>
    <th>Random</th>
    <th>TF-IDF</th>
    <th>Longformer</th>
    <th>DistilRoBERTa</th>
    <th>GPT-4o-mini</th>
    <th>Llama</th>
    <th>Llama 70B</th>
    <th>Reference</th>
  </tr>
  <tr>
    <td>climate sentiment (Bingler et al., 2023)</td>
    <td>35.9(30.6-41.1)</td>
    <td>69.0(63.4-74.4)</td>
    <td>79.9(74.8-84.4)</td>
    <td>77.7(72.6-82.2)</td>
    <td>77.3(72.4-82.2)</td>
    <td>71.4(65.8-76.4)</td>
    <td>70.5(64.7-75.5)</td>
    <td>83.8<sup>ac</sup></td>
  </tr>
</table>

Table 12: Performances of baselines for each dataset using the train, test describe in section 3.3. The performance computed is the macro F1-score. The 95% confidence interval is computed using a 1000 bootstrap of sampled with replace as True of the size of the original test set. The reference performance is the best relevant performances reported in the original papers. 3. weighted average F1-score, c. climateBERT.

![Confusion matrix of GPT-4o-mini predictions on ClimateBERT’s climate sentiment dataset.](page_324_613_700_246.png)

Figure 5: Confusion matrix of GPT-4o-mini predictions on ClimateBERT’s climate sentiment dataset.

labeled. We found 5/30 examples of texts that describe a risk-related structure (e.g. Examples C.42), which can reasonably be classified as neutral. While the dataset’s labels are generally well-defined, these observations suggest that further refinement of definitions for certain edge cases could help improve consistency in classification.

Example C.37 (neutral). TD recently launched a bold and ambitious climate action plan to address the challenges of climate change. This includes a target to achieve net-zero greenhouse gas emissions in our operations and financing activities by 2050. We backed this commitment with the creation of a new Sustainable Finance and Corporate Transitions Group to support clients around the world, and an Environmental, Social and Governance (ESG) Centre of Expertise to participate in the global efforts required to deliver on this long-term target.

Example C.38 (neutral). To exclude the least energy efficient hydrocarbons and those that pose the greatest threat to the environment, because these are incompatible with the goal of combating climate change and they represent an economic risk for investors. This means turning down projects and companies that do the majority of their business in: o Oil sands production, o Oil extracted from the Arctic region (off-shore and on-shore production), o Shale gas or oil production involving excessive flaring or venting, o Infrastructure projects mainly intended for schemes covered by the exclusion criteria set out above, Credit Agricole S.A. is committed to offsetting the Group’s entire direct carbon footprint until 2040 via the

Example C.39 (risk). Sustainability aspects and climate-related risks and opportunities are integrated into the Group-wide risk management process at Daimler. They are understood to be conditions, events, or developments involving environmental, social or governance factors (ESG), the occurrence of which may have an actual or potential impact on the Daimler Group’s profitability, cash flows and financial position, as well as on its reputation. ESG-related risks and opportunities that are very likely to have a serious negative impact on non-financial aspects in accordance with the CSR Directive Implementation Act (CSR-RUG) can be found in the respective categories of the Risk and Opportunity Report according to their cause. Furthermore, Daimler follows the recommendations of the Task Force on Climate-related Financial Disclosures (TCFD) with regard to climate-related risks and opportunities.

Example C.40 (neutral). In responding to Resolution 6.1 we have: z approved and published an Energy Policy consisting of financing of fossil-fuel-related activities, encompassing thermal coal, upstream oil and upstream gas, fossil-fuelled power generation, and renewable and embedded energy solutions.

Example C.41 (neutral). We also look at the way banks facilitate financing by others, for example by arranging the issue of green bonds. For each
of these categories we look at the bank’s current lending, historical trends and lending targets. There are also ‘no go projects’. We won’t invest in any bank which lends to an Adani Carmichael coal mine. Internationally we won’t invest in any bank which lends to a Keystone XL pipeline transporting oil from the tar sands of Canada.

Example C.42 (risk). Our Emerging Risk Committee, constituted by our executive directors, discusses various risks that may affect our business, and includes environmental considerations such as climate change as a standing agenda item. In addition, Newton’s Operating Committee is responsible for the management of how our business is run, and also considers relevant risks. Both committees ultimately report to the Executive Committee, with the outputs from the Emerging Risk Committee being reviewed at the Board Risk Committee.

C.6 Green Claim Detection

Task Description Given an input sentence or a paragraph, output a binary label, “green claim” or “not green claim”.

Experiment We collected the datasets on Environmental Claims (Stammbach et al., 2023), and on Green Claims (Woloszyn et al., 2022) and reproduced the experiments with the setting described in section B and reported the results in Table 13. Our pre-processing pipeline altered the Green Claims dataset when removing URLs that are important in tweets.

Results Analysis As shown in Table 13, the best performing models are the fine-tuned transformers reaching F1-scores up to 91.2% on Environmental Claims and 97.2% on Green Claims. The performances of the zero-shot models are slightly worse by around 5%. Finally, the TF-IDF baseline underperformed the other approaches but still reached macro F1-scores above 80%.

Stammbach et al. (2023) reported a binary F1-score of 84.9%. In our experiments, GPT-4o-mini reached a binary F1-score of 81% and DistilRoBERTa reached 87%.

Similarly, Woloszyn et al. (2022) reported a binary F1-score of 92.08%. In our experiments, GPT-4o-mini reached a binary F1-score of 89% and DistilRoBERTa reached 98%. When accounting for the confidence interval (94.84%-100%), our fine-tuned DistilRoBERTa outperformed their best model (fine-tuned RoBERTa). This might be explained by our cleaning process (e.g. removing URLs) that removed noise from the original dataset, but it could also be due to our split. (Woloszyn et al., 2022) showed that fine-tuned models trained on their dataset are prone to adversarial attacks, which our cleaning process might have exacerbated.

C.6.1 Error analysis

Environmental Claims We found that most of the errors are FP (86%). Most of the errors sampled (9/14) are actual errors by GPT-4o-mini. There are only 4 FN. 3/4 FN are implicit claims. They are descriptions of environment-related actions (Example C.43) but they do not explicitly suggest that the company has a positive environmental impact. We also found 2 examples of implicit claims in the FP (Example C.44). In the gold standard, implicit claims are labeled sometime positively and sometime negatively. This shows an ambiguity in the annotation guidelines for implicit or indirect claims..

Most of the sampled FP (7/10) are statements out-of-context that require the annotator to make assumptions. Such statements are well-described in the definition and should be classified as negatives, which makes them actual errors from the model.

Example C.43 (Environmental Claim). We have developed a Climate Change Strategy Roadmap outlining our climate change governance framework.

Example C.44 (Not Environmental Claim). In terms of initiatives regarding energy issues, the AGC Group works to reduce the energy involved in its production activities.

Example C.45 (Not Environmental Claim). This reduces KLP’s climate risk and makes it clear that our work has an effect on the climate.

Example C.46 (Not Environmental Claim). Obviously, it’s without a doubt, the environment continues to improve, and that’s all good.

Green Claims Due to the size of the dataset and the high model performance, we observed only 6 classification errors: 3 FN and 3 FP.

When reviewing the errors, we found that 2 FP (e.g. Example C.47 with “Sea buckthorn extracts”) and 1 FN (Example C.49 with “Coconut Oil”) fall into the “natural claim” category – suggesting that a product is better, healthier, or has properties thanks to a natural ingredient. It is not clear how these claims should be annotated. In the guidelines, they
<table>
  <tr>
    <th>Dataset</th>
    <th>Random</th>
    <th>TF-IDF</th>
    <th>Longformer</th>
    <th>DistilRoBERTa</th>
    <th>GPT-4o-mini</th>
    <th>Llama</th>
    <th>Llama 70B</th>
    <th>Reference</th>
  </tr>
  <tr>
    <td>Environmental Claims (Stummelbach et al., 2023)</td>
    <td>42.8(37.4-48.4)</td>
    <td>80.5(74.8-85.5)</td>
    <td>90.6(86.5-94.0)</td>
    <td>91.2(86.9-94.6)</td>
    <td>86.8(82.0-90.9)</td>
    <td>81.4(76.8-86.1)</td>
    <td>76.6(71.1-81.9)</td>
    <td>84.9<sup>a</sup></td>
  </tr>
  <tr>
    <td>Green Claims (Woloszyn et al., 2022)</td>
    <td>46.1(34.9-56.7)</td>
    <td>86.1(77.2-93.3)</td>
    <td>94.5(88.7-98.7)</td>
    <td>97.2(93.0-100.0)</td>
    <td>91.5(86.9-97.1)</td>
    <td>90.0(85.2-94.6)</td>
    <td>86.6(78.9-93.5)</td>
    <td>92.0<sup>b</sup></td>
  </tr>
</table>

Table 13: Performances of baselines for each dataset using the train, test describe in section 3.3. The performance computed is the macro F1-score. The 95% confidence interval is computed using a 1000 bootstrap of sampled with replace as True of the size of the original test set. The reference performance is the best relevant performances reported in the original papers. * binary F1-score, r. RoBERTa

could be considered “nature-friendly messages that target the needs and desires of environmentally concerned stakeholders” but they could also be considered as “Claims that address issues not related to the environment, such as health or equality, [which] are not considered green claims”. (Woloszyn et al., 2022) reported that: “natural ingredients” and “naturally derived” were the most frequent bigrams in "Explicit Green Claims". We found that the tweets containing these bigrams are “natural claim”, therefore we considered that “natural claims” are part of “green claims”. This implies that the 2 FP are mislabeled examples, and the FN and error by GPT-4o-mini.

The 2 other FN, are out-of-context, generic statements that do not refer to any specific product (Example C.48). Whether such statements should be classified as “green claims” is debatable. If we consider that any communication on environmental topics is an attempt to portray the company in a more environmentally friendly light, then these statements could be interpreted as green claims. However, this approach is overly simplistic and risks categorizing any environmental reference as a green claim, which would be an over-extension of the definition.

The remaining FP represents an actual misclassification by the model.

Example C.47 (Not Green). He needs protection from harsh weather too. Sea buckthorn extracts fortify + invigorate for smooth, energized skin: <URL> <URL>

Example C.48 (Green Claim). Environmental solutions come in all shapes and sizes.

Example C.49 (Green Claim). TELL US: what do you love the most about #WholeBlends Smoothing with #Coconut Oil and #Cocoa Butter Extracts? #haircare #coconutoil <URL>

C.7 Green Claim Characteristics

Task Description Given an input sentence or a paragraph labeled as green claim, output a more fine-grained characterization of the claim. This is a multi-label classification task; the labels can be about the form (e.g. specificity) or the substance (e.g. action, targets, facts).

Experiment All the previously mentioned datasets are available. We collected them and reproduced the experiments with the setting describe in section B and reported the results in Table 14. Implicit/Explicit Green Claims (Woloszyn et al., 2022) was impacted by our pre-processing pipeline when we removed URLs.

Results Analysis As shown in Table 14, we observed the same patterns for each dataset. The best-performing models are fine-tuned transformers reaching performances above 77%. The zero-shot approaches underperformed the fine-tuned models by 1 to 5%. The TF-IDF baseline performed significantly better than random, underperforming the best models by between 2 to 10% only. It is important to note that the difference in performance with the TF-IDF baseline is significant only for DistilRoBERTa on Commitments And Actions and for Longformer on Net-zero/Reduction. This suggests that the characteristics mentioned, both in terms of form and content, are expressed with a specific vocabulary that can be used as a decent predictor. On Implicit/Explicit Green Claims and Net-zero/Reduction, GPT-4o-mini reached similar performances as the fine-tuned models. On Climate Specificity and ESGBERT action500 GPT-4o-mini underperformed both the TF-IDF baseline and the fine-tuned models. On Commitments and Actions, the fine-tuned models significantly outperform the zero-shot approaches.

The performance reached by fine-tuned models in our experiments is similar to those reached in the original studies.

C.7.1 Error analysis

Commitments And Actions The majority of all errors are FP (77%). Another ambiguous aspect is the description of company philosophy/values/mindset and the description of governance structures. We found examples of philoso-
<table>
  <tr>
    <th>Dataset</th>
    <th>Random</th>
    <th>TF-IDF</th>
    <th>Longformer</th>
    <th>DistilRoBERTa</th>
    <th>GPT-4o-mini</th>
    <th>Llama</th>
    <th>Llama 70B</th>
    <th>Reference</th>
  </tr>
  <tr>
    <td>Commitments&Actions(Bingler et al., 2021)</td>
    <td>48.9/32.9/53.9</td>
    <td>72.7/67.4/78.2</td>
    <td>72.7/67.4/78.2</td>
    <td><b>81.9/77.1/86.5</b></td>
    <td>67.2/61.6/72.4</td>
    <td>64.6/58.7/69.7</td>
    <td>70.7/51.5/62.1</td>
    <td>81<sup>a</sup></td>
  </tr>
  <tr>
    <td>Climate Specificity(Bingler et al., 2021)</td>
    <td>48.9/32.9/53.9</td>
    <td>72.7/67.4/78.2</td>
    <td>77.3/72.4/84.9</td>
    <td>81.9/77.1/86.5</td>
    <td>76.8/67.1/80.8</td>
    <td>68.0/57.8/69.3</td>
    <td>77.9/51.6/67.1</td>
    <td>77<sup>b</sup></td>
  </tr>
  <tr>
    <td>ESGaction500(Schimanski et al., 2023b)</td>
    <td>44.4/29.9/58.5</td>
    <td>82.5/71.1/93.1</td>
    <td>83.5/75.4/95.8</td>
    <td>89.1/80.2/97.8</td>
    <td>76.0/65.2/87.4</td>
    <td>67.0/53.7/78.0</td>
    <td>63.0/47.7/61.1</td>
    <td>-</td>
  </tr>
  <tr>
    <td>Implicit/Explicit Green Claims(Woloszyn et al., 2022)</td>
    <td>37.2/26.3/46.1</td>
    <td>70.3/56.8/82.0</td>
    <td>63.0/48.8/74.9</td>
    <td>81.2/68.1/91.8</td>
    <td>81.6/71.2/90.4</td>
    <td>68.5/54.7/81.0</td>
    <td>75.7/63.4/86.0</td>
    <td>81.45<sup>c</sup></td>
  </tr>
  <tr>
    <td>Net-Zero/Reduction(Schimanski et al., 2023a)</td>
    <td>29.8/24.9/34.8</td>
    <td>94.8/92.2/97.1</td>
    <td>97.8/96.0/99.1</td>
    <td>97.7/95.8/99.3</td>
    <td>97.3/95.5/99.0</td>
    <td>93.7/90.7/96.2</td>
    <td>95.6/93.3/97.8</td>
    <td>98.7<sup>d</sup></td>
  </tr>
</table>

Table 14: Performances of baselines for each dataset using the train, test describe in section 3.3. The performance computed is the macro F1-score. The 95% confidence interval is computed using a 1000 bootstrap of sampled with replace as True of the size of the original test set. The reference performance is the best relevant performances reported in the original papers. 1. binary F1-score, 2. average F1-score (macro/micro is not specified), 3. weighted average F1-score, c. climateBERT, r. RoBERTa

phy with the positive and negative labels (Examples C.51, C.52) and examples of governance in both (Examples C.53, C.52). Those kinds of errors represent 3/10 sampled FN and 7/10 of FP. Indeed expressing company values could be considered as commitments toward these values or a simple description of them; similarly, describing a governance structure could be considered as an action (setting up the governance structure) or a simple description. This ambiguity on what constitutes a commitment and an action is also reflected in the moderate inter-annotator agreement among human annotators (Krippendorff’s α of 0.5354).

Additionally, this task is not climate-specific, and therefore the guidelines include both “business or climate actions”. GPT-4o-mini understood the prompt as only “climate actions”. 2 FN are due to GPT-4o-mini being anchored to a word in the prompt. Using a larger model or updating the prompt might resolve this issue. There is also 1 FN that was mislabeled (Example C.54), as the text describes a process and does not contain an action or a commitment. The remaining FN (4/10) are actual errors.

Of the 3 remaining FP sampled, 2 are mislabeled, such as Example C.50, which describes an action about recycling but is labeled as “No” in the gold standard - and as an “action” by GPT-4o-mini.

Example C.50 (No). 1 Recycling: To meet the environmental policies of major countries, we operate a recycling process utilizing non-reusable dead batteries and scraps generated during battery production. We have a strategic cooperation relationship with major partners for realizing a closed-loop. Through this system, we extract raw materials of batteries such as nickel, cobalt, and lithium by crushing and dissolving battery waste or scraps.

Example C.51 (Yes). In our view, true sustainable investing cannot be achieved by simply voting a proxy, adding a director of sustainability or even divesting from an asset class. Because traditional models of finance and investing often fail to appropriately integrate sustainability issues, we’ve had to build it into our thinking from the ground up. It requires integration across our products, across our product teams and across our entire organization.

Example C.52 (No). Climate risk has emerged as one of the top environmental risks for the Bank. This includes physical risks related to the chronic and acute physical impacts of climate change (e.g., shifts in climate norms, and extreme weather events such as hurricanes, wildfires and floods), and transition risks associated with the global transition to a low-carbon economy (e.g., climate-related policy actions and litigation claims, technological innovations, and shifts in supply and demand for certain commodities, products and services). Both physical and transition risks could result in strategic, credit, operational, legal, and reputational risks for the Bank and its clients in climate sensitive sectors. TD supports Canada’s objectives to meet the goals of the Paris Agreement and recognizes the Bank’s responsibility to contribute by integrating climate considerations across its business. The Bank continues to monitor industry and regulatory developments and assess the potential impacts of climate change and related risks on its operations, lending portfolios, investments, and businesses.

Example C.53 (Yes). Our sustainability efforts are underpinned by strong corporate governance, which we continue to reinforce in line with recommendations of the Malaysian Code on Corporate Governance 2017 (MCCG 2017). This saw us set up a new Board Risk Committee to further enhance our risk oversight, adding to the existing Board Audit Committee and Nomination and Remuneration Committee. The Board as a whole comprises four Independent Non-Executive Directors (INEDs), who make up 50% of its composition, two of whom are foreign Directors. On a related note, I represented the Group in signing an Integrity Pact with the Malaysian Anti-Corruption Commission
(MACC), underlining our commitment to observing integrity in all our dealings with stakeholders.

Example C.54 (Yes). D+ This client does not (yet) meet Rabobank’s sustainability policy on one or more points or has not responded adequately to key questions. Specific agreements are made about a possible solution and timelines are established. Once the customer meets the sustainability policy of Rabobank, it is classified in category A, B or C.

Example C.55 (No). Governance structure A robust governance structure ensures timely and direct execution of programs that drive the achievement of our set goals for 2020 as well as of our new set of targets for 2025. The head of Sustainability is responsible for the development, coordination and execution of our sustainability strategy and reports to the member of the Executive Board responsible for Global Operations. He or she also leads the sustain- ability Sponsor Board, which is composed of senior representatives from Global Brands, Global Operations, Digital, Sales, Finance, Corporate Communication, and other relevant functions across the company. The Sponsor Board ensures cross-functional alignment, transparent end-to-end management and execution of agreed-upon sustainability goals within their functions. This includes reviewing and signing-off on policies as required. We also maintain a separate compliance function which is operated as the Social & Environmental Affairs Team (SEA) to evaluate supplier-facing social and environmental com- pliance performance and human rights impacts, reporting, through the General Counsel, to the CEO.

This dataset would require guidelines that are more detailed and specific for multiple ambiguities (that are not clear neither in the guidelines nor in throughout the annotations):

• Setting a governance structure or process an action ?

• Is describing a existing process (industrial, governance, ...) an action ?

• Is saying that the company want to implement sustainability across all their products an action/commitment ?

ESGBERT’s Actions500 GPT-4o-mini produced only 10 errors, 9 of which were FP. Since this dataset is not part of a publication, we lack access to detailed annotation guidelines, leaving certain edge cases, such as statements about commitments, company philosophy, or exploration of potential solutions, open to interpretation. Notably, 13/19 of these errors involve cases that are debatable, suggesting that clearer guidelines could improve consistency.

Climate Specificity In GPT-4o-mini predictions for Climate Specificity, most of the errors are FP (93%). There are actually only 6 FN.

When investigating the FN, we found 2 examples of part of the legend of a figure such as Example C.56. In the guidelines, the legend “provides firm-specific detailed explanations to enable readers to better understand the overall information reported” and is therefore considered “specific”. We also found 2 examples which were mislabeled as the statement applies to the whole industry, which is considered “non-specific” in the guidelines. We classified Examples C.57 as out-of-context, as they mention "A similar approach could be used" which we argue is not specific. But more context could make it specific.

Out of the sampled FP, we found 4/10 debatable examples. This is due to the granularity of the specificity. A paragraph might contain a list of specific “Key Priorities”. However, as the priorities are strategic axes, they are not specific about actions or targets. For instance, Example C.58 is specific about using "red lines" but does not describe specifically what the "red lines" are. Similarly to Commitments and Actions, the statements about governance can be precise governance points - such as the role of the CEO in overseeing the climate-related issue - but not specific about actions or targets. This is the case for 2 of the sampled errors.

The dataset contains annotation for identification of “specific actions/targets specific to the company”. This task is particularly difficult, as it needs to be precisely defined and even with a precise definition, we believe that some statements remain ambiguous due to the granularity of the specificity. This is further confirmed by the low IAA (Krippendorff’s α of 0.1703)

Example C.56 (specific). [1] Data calculated on the same reporting perimeter as 2018, excluding Abertis Group. [2] Data on the diesel consumed by the gensets in Chile in 2018 are not available. [3] Figure updated following a consolidation subsequent to the close of the Integrated Report 2018 on ETC data. [4] Data updated following a restatement of the income statement 2018. [5] Location-based
Example C.57 (specific). A similar approach could be used for allocating emissions in the fossil fuel electricity supply chain between coal miners, transporters and generators. We don’t invest in fossil fuel companies, but those investors who do should account properly for their role in the production of dangerous emissions from burning fossil fuels.

Example C.58 (not-specific). Sustainable strategy ‘red lines’ For our sustainable strategy range, we incorporate a series of proprietary ‘red lines’ in order to ensure the poorest- performing companies from an ESG perspective are not eligible for investment.

Implicit/Explicit Green Claims As the dataset is small, there are only 12 errors. As show in figure 6, the most frequent error, is a statement without claims classified as an “implicit claim”.

![Confusion matrix of GPT-4o-mini predictions on Explicit/Implicit Green Claims dataset.](page_324_1012_393_246.png)

Figure 6: Confusion matrix of GPT-4o-mini predictions on Explicit/Implicit Green Claims dataset.

As discussed in Section C.6, it is not clear if green claims include “natural claim” - suggesting that a product is better, healthier, or has properties thanks to a natural ingredient. Among the 12 errors, 5 contain “natural claims” (Example C.60). We also found 2 errors that are debatable, and 2 annotation errors. For instance, the annotator classified Example C.59 as “not green”. They most likely understood "Irish Spring" as the season instead of the soap brand. On the contrary, GPT-4o-mini correctly classified it as “implicit green claim”. This shows that LLMs can have broader general knowledge than the original annotators. The distinction between explicit and implicit is highly relevant, as it define straightforward cases from ambiguous cases which are harder to classify. Despite those errors, overall both fine-tuned models and the zero-shot approaches reached good performances. This is aligned with the good IAA reported in the original study (Woloszyn et al., 2022) (Krippendorff \( \alpha = 0.8223 \)).

Example C.59 (not green). Irish Spring is green in more ways than one... <URL>

Example C.60 (not green). Give your skin a mid-day refresh with the Yes To Grapefruit Unicorn Brightening Mist! It’s naturally packed with vitamin C to help give your skin a boost of GLOW! Thanks, @POPSUGAR <URL>

Example C.61 (green). A healthy lifestyle becomes ever more important for many of us - also within our beauty routine. Stay tuned for our new permanent hair color featuring natural ingredients such as soy protein, oat milk and argan oil! Available from February. #headsup #WhatsComingNext #ProductNews <URL>

Net-zero/Reduction On this dataset, the model failed only on 8 instances as shown in Figure 7. As the performances of models and TF-IDF baseline are really high, we also experimented with using only a few keywords. Using ["net", "zero", "neutrality"] for the “net-zero” class and ["reduction", "reduce"] for the “reduction” class, we reached an F1-score of 78.03%.

![Confusion matrix of GPT-4o-mini predictions on Net-zero/Reduction dataset.](page_1012_1012_393_246.png)

Figure 7: Confusion matrix of GPT-4o-mini predictions on Net-zero/Reduction dataset.

Among the errors, 1 is a simple statement classified as targets in the gold standard (Example C.62). Another example is labeled “net-zero” while dealing with a reduction of 95% (Example C.64), raising the question: is 95% reduction the same as net-zero? Finally, half of the errors are statements with both “reduction” and “net-zero” targets (Example C.63). Overall, we see that both fine-tuned models and the zero-shot approaches are proficient at identifying targets. However, we point out that without addressing the labeling error mentioned, we can’t conclude on the performance differences. While
the task appears straightforward for models, annotators do not achieve perfect inter-annotator agreement, despite a high score (Cohen’s \( \kappa = 0.931 \)). This indicates that even seemingly simple tasks can still be subject to occasional disagreement among annotators.

Example C.62 (net-zero). Based on our track record and plans up to 2030 we are confident that we are well-placed to make significant progress beyond 2030 and up to 2050. (Pg. 27)

Example C.63 (reduction). Reduce GHG emissions (Scopes 1+2) (on a market basis*1) by 55% by FY2030 and to zero by FY2050 (compared to FY2017)

Example C.64 (net-zero). Amsterdam aims for a 95% emissions reduction by 2050 (reference year 1990). Unclear how this becomes climate neutrality though.

C.8 Green Stance Detection

Task Description Given two input sentences or paragraphs, one labeled as the claim and one as the evidence, predict the stance between the two: supports, refutes or neutral. Some studies fix the claim and only vary the evidence (e.g. the claim is always *Climate change poses a severe threat*), training a model to predict the stance of the evidence in respect to the fixed claim. Other studies train a model to predict the relation between varying claims/evidences.

Experiment We collected *ClimateFEVER* (Diggelmann et al., 2020), *climateStance* (Vaid et al., 2022), *GWSD* (Luo et al., 2020) and *LobbyMap* (Morio and Manning, 2023) and reproduced the experiments with the setting described in section B and reported the results in Table 15. For *ClimateFEVER* the dataset is designed to predict the label of claim-evidence pair relation and then aggregate the predictions to infer if the claim is supported or not. Therefore, we split the tasks into 3. The first is to predict the claim label only, without the evidence (*ClimateFEVER claim (our split)*). The second task is to predict the claim-evidence label (*ClimateFEVER evidence (our split)*). Finally, the third task is given the prediction of the claim-evidence pair, predict the claim label (*ClimateFEVER claim (our split, aggregated)*). We also reported the performance for a train-test split similar to climabench(Spokoyny et al., 2023). For *Lobbymap*, we evaluated each intermediary task (Page, Query, Stance). Our pre-processing impacted *ClimateFEVER* and *Lobbymap* as we resampled the datasets and removed longer texts. It also altered *climateStance*, as we removed URLs which are important in tweets.

Results Analysis When focusing on the results for *ClimateFEVER claim (our split, aggregated)*, *ClimateFEVER evidence (our split)*, *ClimateStance*, *GWSD* and *LobbyMap* datasets; our experiments results show performances below 75% for the best-performing models. This shows that those tasks are challenging for models. For most datasets, the best-performing approaches are the fine-tuned models. However, for *ClimateFEVER*, it is the zero-shot approach.

When using a split similar to Spokoyny et al. (2023) on *climateFEVER evidence (climabench split)*, distilRoBERTa reached an F1-score of 59.9% (comparable to theirs, with 62.7% for (Spokoyny et al., 2023)). However, as mentioned in Appendix A, the split provided in their benchmark contains partial contamination (the same claim is both in the train and test splits, but with different evidences). When using our split, the performance of the best-performing fine-tuned transformer drops to 52.7%. On *ClimateFEVER claim* - classify if the claim is supported by evidences - we first evaluated if the claim contains a predictive power in itself. Both fine-tuned models and zero-shot approaches struggle on this task. This is likely due to the way the labels are constructed: the claim label is aggregated on the claim-evidence labels. Therefore, the claim’s label is dependent on the evidence collected previously and not on its inherent factuality. When using aggregated claim-evidence prediction to infer the claim’s label, the performance increases to 41.3%. Still, this is far from the performances reported in other papers (Wang et al., 2021; Vaghefi et al., 2022; Xiang and Fujii, 2023; Webersinke et al., 2022; Vaid et al., 2022) reaching performances above 80%. However, they are not comparable as they removed claims with the "DISPUTED" label.

We achieved performance levels comparable to those reported in the original studies across all other datasets except for *LobbyMap*. This discrepancy can be attributed to the reduced size of our training dataset, which was limited to 10,000 samples. Given the complexity of the task, a larger training set or extended training duration could po-
<table>
  <tr>
    <th>Dataset</th>
    <th>Random</th>
    <th>TF-IDF</th>
    <th>Longformer</th>
    <th>DistilRoBERTa</th>
    <th>GPT-4o-mini</th>
    <th>Llama</th>
    <th>Llama 70B</th>
    <th>Reference</th>
  </tr>
  <tr>
    <td>climateF claim (one split) (Diggelmann et al., 2020)</td>
    <td>15.4 (10.4–20.7)</td>
    <td>35.8 (20.4–43.0)</td>
    <td>39.6 (24.4–54.8)</td>
    <td>42.5 (34.2–49.9)</td>
    <td>41.8 (32.4–49.9)</td>
    <td>50.4 (42.0–58.2)</td>
    <td>30.0 (21.9–37.2)</td>
    <td>26.8 (19.8–32.8)</td>
    <td>60.7<sup>a</sup></td>
  </tr>
  <tr>
    <td>climateF claim (agg.) (Diggelmann et al., 2020)</td>
    <td>28.4 (25.5–31.0)</td>
    <td>46.4 (24.2–58.2)</td>
    <td>57.0 (52.6–66.7)</td>
    <td>51.8 (47.5–55.1)</td>
    <td>60.0 (56.2–63.6)</td>
    <td>52.5 (48.5–55.9)</td>
    <td>63.7 (59.7–67.4)</td>
    <td>68.0<sup>b</sup></td>
    <td>60.10 ...<sup>c</sup></td>
  </tr>
  <tr>
    <td>climateF claim (climabench split, agg. @ Diggelmann et al., 2020)</td>
    <td>30.4 (27.4–33.5)</td>
    <td>46.4 (24.2–58.2)</td>
    <td>57.0 (52.6–66.7)</td>
    <td>51.8 (47.5–55.1)</td>
    <td>60.0 (56.2–63.6)</td>
    <td>52.5 (48.5–55.9)</td>
    <td>63.7 (59.7–67.4)</td>
    <td>68.0<sup>b</sup></td>
    <td>60.10 ...<sup>c</sup></td>
  </tr>
  <tr>
    <td>climateFEVER evidence (climateSanct/Vaid et al., 2021)</td>
    <td>21.3 (17.7–25.1)</td>
    <td>49.4 (42.5–55.9)</td>
    <td>56.4 (49.4–63.0)</td>
    <td>56.4 (50.4–62.2)</td>
    <td>48.5 (42.2–54.2)</td>
    <td>58.0 (51.2–64.8)</td>
    <td>58.0 (51.2–64.8)</td>
    <td>59.6<sup>d</sup></td>
    <td>52.6<sup>e</sup></td>
  </tr>
  <tr>
    <td>climateSanct/Vaid (GWSID) (Liu et al., 2020)</td>
    <td>29.5 (23.0–36.0)</td>
    <td>59.2 (51.6–66.9)</td>
    <td>69.3 (62.2–75.8)</td>
    <td>75.3 (68.6–81.6)</td>
    <td>69.8 (63.1–76.3)</td>
    <td>64.4 (57.5–71.1)</td>
    <td>72.5 (66.8–78.8)</td>
    <td>73<sup>f</sup></td>
    <td>-</td>
  </tr>
  <tr>
    <td>GlobaWarming Stance (GWSID) (Liu et al., 2020)</td>
    <td>48.1 (37.9–58.3)</td>
    <td>73.1 (65.9–79.9)</td>
    <td>73.1 (65.9–79.9)</td>
    <td>73.1 (65.9–79.9)</td>
    <td>73.1 (65.9–79.9)</td>
    <td>73.1 (65.9–79.9)</td>
    <td>73.1 (65.9–79.9)</td>
    <td>73<sup>f</sup></td>
    <td>-</td>
  </tr>
  <tr>
    <td>LobbyMap (Stances/Morin and Manning, 2023)</td>
    <td>12.4 (12.0–12.9)</td>
    <td>47.4 (35.0–59.2)</td>
    <td>57.3 (50.5–65.9)</td>
    <td>52.1 (50.0–54.2)</td>
    <td>36.4 (34.5–38.0)</td>
    <td>27.6 (26.2–29.1)</td>
    <td>32.7 (31.3–34.2)</td>
    <td>-</td>
    <td>-</td>
  </tr>
  <tr>
    <td>LobbyMap (Stances/Morin and Manning, 2023)</td>
    <td>19.2 (17.8–20.5)</td>
    <td>43.6 (41.1–47.5)</td>
    <td>46.7 (45.0–48.3)</td>
    <td>47.4 (45.7–48.9)</td>
    <td>30.0 (28.5–31.7)</td>
    <td>26.9 (25.4–28.4)</td>
    <td>24.1 (22.8–25.6)</td>
    <td>-</td>
    <td>-</td>
  </tr>
</table>

Table 15: Performances of baselines for each dataset using the train, test describe in section 3.3. The performance computed is the macro F1-score. The 95% confidence interval is computed using a 1000 bootstrap of sampled with replace as True of the size of the original test set. The reference performance is the best relevant performances reported in the original papers. d. Contaminated split, SciBERT, h. Human Baseline using annotations, i. Filtered Split (by removing disputed claims), r. RoBERTa, t. BERT

tentially enhance model performance by providing more comprehensive coverage of the data distribution and enabling the model to learn more nuanced patterns.

C.8.1 Error analysis

ClimateFEVER We will not investigate errors on the claim classification task as the performance is really poor and the task is not designed for this.

![Confusion matrix of GPT-4o-mini predictions on ClimateFEVER (evidence) dataset.](page_1012_1012_384_384.png)

Figure 8: Confusion matrix of GPT-4o-mini predictions on ClimateFEVER (evidence) dataset.

On the claim-evidence pair relation classification task, we firstly see that the model rarely mixes ‘SUPPORT’ and ‘REFUTES’ (less than 5% of errors) as reported in Figure 8. Then we see that GPT-4o-mini tend to predict “REFUTES” instead of “NOT ENOUGH INFO”; and “NOT ENOUGH INFO” instead of “SUPPORTS”.

Upon examining the errors, we found that 12/30 are mislabeled in the gold standard. Most of these annotation mistakes (9/12) are evidence that can be classified as SUPPORT/REFUTES but were classified as NOT ENOUGH INFO. This shows that many examples are ambiguous as they are at the frontier of NOT ENOUGH INFO’s label, and this is also aligned with the LLM’s error pattern. we found that 15/30 errors were genuine misclassifications by GPT-4o-mini, and three instances being potentially debatable. These errors are likely to be mitigated through prompt engineering, for instance, by emphasizing that the evidence was retrieved by a model with potential limitations. Consequently, although the evidence may appear related to the claim, additional verification is required to ensure its relevance. For example, in “He concluded”, “He” is not necessarily the person mentioned in the claim (Example C.65).

As mentioned, we found some errors that are debatable, and Diggelmann et al. (2020) reported a low Cohen’s \( \kappa = 0.334 \). This shows that the task is difficult even for humans, and there are numerous disagreements. As the individual annotations were provided in the dataset, we also computed the performances of each annotator. We found that the average macro F1-score on the claim-evidence task is 60%, ranging from 54% to 69%. The model is well within the range of the human annotator performances.

Example C.65 (NOT ENOUGH INFO).
Claim: “When you read Phil Jones’ actual words, you see he’s saying there is a warming trend
Evidence: "From this, he concluded that The post-1980 global warming trend from surface thermometers is not credible."

Additionally, we sampled 10 examples labeled “SUPPORTS” predicted as “NOT ENOUGH INFO”. All 10 are claims supporting climate change, and evidence indeed supports the claim. However, the evidence is not sufficient on its own (supporting only a part of the claim or indirectly supporting the claim). We also sampled 10 examples labeled “NOT ENOUGH INFO” predicted as “REFUTES”. 8 claims deny climate change, and 2 are neutral. Except for 2, which are out of context, they all either refute or partly refute (refuting
part of the claim or indirectly refuting the claim). The stance can be interpreted as: “the evidence is sufficient to support/refute the claim” or “the evidence is aligned with the claim and can be part of an argument that supports/refutes the claim”. The annotators seem to struggle with this distinction, while GPT-4o-mini tends to predict “REFUTES” for arguments refuting climate change denial arguments, and “NOT ENOUGH INFO” for arguments supporting climate change arguments.

Climate Stance This dataset is heavily imbalanced toward the “Favor” label. Therefore, most of the errors are on this label. However, as shown in Figure 9 the rate of error is not significantly higher.

![Confusion matrix of GPT-4o-mini predictions on Climate Stance dataset.](page_328_670_353_181.png)

Figure 9: Confusion matrix of GPT-4o-mini predictions on *Climate Stance* dataset.

The definition of the stance provided by Vaid et al. (2022) contains some gray area. For example, the “Against” label includes: “opposition to climate change policies”. Therefore, a statement that includes acknowledging climate change, but denying the actions that are required to limit it, is difficult to classify (Example C.66).

In the sampled errors, 7/30 are statements implicitly expressing concerns about climate change, such as promoting an IPCC conference or making a joke (Example C.67) which are misclassified by GPT-4o-mini. The other sampled errors (23/30) are debatable or mislabeled. They are 11 statements out-of-context and could be interpreted (e.g. Example C.68), or 4 statements clearly mislabeled (e.g. Example C.66 labeled “Favor” in the dataset but correctly classified as “Against” by GPT-4o-mini).

Contrary to other datasets, there are many errors where the model predicts “Against” instead of “Favor”. Therefore, we investigated a sample of 10 errors of this type. We found 6 mislabeled examples (e.g. Example C.69), 1 tweet with an ambiguous stance, and 3 that are actual errors by GPT-4o-mini. This suggests that a lot of those errors are actually annotation mistakes, most likely caused by the reference to personalities (e.g. “Christopher Monckton”, “Corbyn”) and implicit stance (e.g. “‘carbon border tax’ will damage global efforts to tackle #ClimateChange”) which are not trivial to understand.

Example C.66 (Favor). My point t is China uses it devolving status to get an easy ride - while Australia who is making reductions - continues down a silly track of higher energy prices for no real change in outcomes for this so called climate emergency

Example C.67 (Favor). living through climate change, I know what it’s like to be a townsper- son in an RPG who goes, "there sure are a lot more wandering monsters than there used to be!"

Example C.68 (Favor). But are solutions towards zero emissions as promising as they seem? This well-documented story by @johncarlosbaez contains some sobering numbers: <URL> via @Nau- tilusMag #ClimateChange #ClimateCrisis 2/2

Example C.69 (Favor). Power curve doesn’t exist just like global warming

GWSD In this dataset the “disagrees” label is under-represented. As shown in the confusion matrix (Figure 10), most of the errors involve the label “neutral”. Confusions between “agrees” and “disagrees” represent less than 15% of the errors.

![Confusion matrix of GPT-4o-mini predictions on GWSD dataset.](page_328_1202_353_181.png)

Figure 10: Confusion matrix of GPT-4o-mini predictions on *GWSD* dataset.

With our prompt, the models often identified factual statements as non-opinionated, leading them to incorrectly assign the “neutral” label to statements that clearly conveyed agreement with the target opinion. In multiple examples (10), such as Example C.70, the model correctly recognized that the fact implied the “seriousness” of the situation. However, because it did not contain an explicit opinion, the model classified it as “neutral”. This
pattern accounts for 14 of the 30 sampled errors (with 10 correct reasonings). Interestingly, this type of answer (correct reasoning but wrong label) is inconsistent, in similar situation the model does answer with agrees/disagrees labels for implicit statements.

Additionally, we identified 7 instances of incorrect labeling in the original dataset. One such example, Example C.71), discusses the attention given to global warming, suggesting that it does not match the severity of the issue. While using a prompt that emphasizes implicit opinions did address some of these errors, it did not lead to overall performance improvements. Although it corrected the previously identified mistakes, it introduced new errors in the opposite direction. These newly introduced errors include misclassified examples, such as Example C.72, as well as statements that do not clearly convey the authors’ opinions. This demonstrates the challenge of dealing with implicit statements, where the inherent ambiguity makes it difficult for the model to assign the correct label.

Example C.70 (agrees). Global temperatures in 2014 shattered earlier records, making 2014 the hottest year since record-keeping began in 1880.

Example C.71 (neutral). The stark truth is that severe weather events alone will not cause global warming to pop to the top of the national agenda.

Example C.72 (neutral). This summer is seeing record lows for Arctic ice.

LobbyMap Firstly, it is important to note that this dataset is constructed based on LobbyMap.org. While LobbyMap.org is curated by experts, the site does not claim to be exhaustive. As a result, it is possible that the collected stances are not fully comprehensive either. For this dataset, the most important metric, therefore, is the recall.

<table>
  <tr>
    <th>binary</th>
    <th>Precision</th>
    <th>Recall</th>
    <th>F1-score</th>
  </tr>
  <tr>
    <td>Page</td>
    <td>50%</td>
    <td>94%</td>
    <td>65%</td>
  </tr>
  <tr>
    <th colspan="4">macro</th>
  </tr>
  <tr>
    <td>Query</td>
    <td>34%</td>
    <td>54%</td>
    <td>40%</td>
  </tr>
  <tr>
    <td>Stance</td>
    <td>30%</td>
    <td>29%</td>
    <td>24%</td>
  </tr>
  <tr>
    <td>Stance (relieved)</td>
    <td>47%</td>
    <td>46%</td>
    <td>45%</td>
  </tr>
</table>

Table 16: Detailed performances of GPT-4o-mini on the LobbyMap dataset different tasks.

As shown in Table 16, the recall is significantly higher for both the Query and Page tasks. This is expected, as the dataset is non-exhaustive, leading to lower precision. However, this limitation does not impact the stance task. Since the page and policy query are predefined, the annotators from LobbyMap.org might have overlooked a page containing a stance on a policy. However, given a specific page and policy, there is only one correct stance to identify. In fact, on the Query task, the GPT-4o-mini outputs for each page on average 2.7 queries, while the original labels contain on average 1.6.

As this dataset contains 3 tasks, and that the texts are relatively long, we analyzed the Page and Query task simultaneously. We sampled 10 instances. For the Page task: 3 were TN (true negatives), 1 was a TP (true positive), 1 was a FP. The FP is due to lack of context (Example C.74). The 5 others are texts that actually contain one or more pieces of evidence about the stance of the company. For instance, Example C.73 was not classified as containing a stance in the original dataset; however, the statement contains evidence about the stance of the company on "Energy transition & zero carbon technologies" through the mention of electric vehicles.

For Stance task, it appears that GPT-4o-mini almost never output the “supporting” label. Using the relaxed labels from the original paper, we reach an F1-score of 41%, which is still very low. We also sampled 10 errors. Half of them are labels that are close (“Supporting” and “Strongly Supporting”). We also found 2 instances where the annotators assigned the “not supporting” instead of the “no position or mixed position” label to statements with no position about a policy. The stance is actually expressed in numerical value (between -2 and +2) on LobbyMap.org, but was translated in labels (e.g “supporting (+1)” in LobbyMap (Stance). The rest of the errors are genuine mistakes.

Example C.73. We’ll show how we’ve revolutionized and electrified some of the most popular, iconic vehicles, helping to shape the future of zero-emissions transportation [No position]

Example C.74. David Maxwell, CEO of east coast gas producer Cooper Energy, described the policy as "draconian" [No position]

In our benchmark, we measure F1-scores for each task individually; however, (Morio and Manning, 2023) proposed custom metrics that we reported in Table 17. We observe that the performances of the Most-Frequent and the Linear Model
<table>
  <tr>
    <th rowspan="2">Model</th>
    <th colspan="3">Document</th>
    <th colspan="3">Page Overlap</th>
    <th colspan="3">Strict</th>
  </tr>
  <tr>
    <th>P</th><th>Q</th><th>S</th>
    <th>P</th><th>Q</th><th>S</th>
    <th>P</th><th>Q</th><th>S</th>
  </tr>
  <tr>
    <td>Most-frequent</td><td>46.7</td><td>52.6</td><td>36.8</td><td>51.8</td><td>25.6</td><td>19.8</td><td>41.2</td><td>19.6</td><td>17.5</td>
  </tr>
  <tr>
    <td>Linear</td><td>66.0</td><td>61.9</td><td>50.3</td><td>71.4</td><td>44.5</td><td>36.1</td><td>52.0</td><td>31.2</td><td>27.0</td>
  </tr>
  <tr>
    <td>BERT-base</td><td>71.0</td><td>63.5</td><td>51.6</td><td>73.6</td><td>48.1</td><td>37.2</td><td>50.2</td><td>31.9</td><td>25.8</td>
  </tr>
  <tr>
    <td>ClimateBERT</td><td>71.8</td><td>64.0</td><td>52.8</td><td>74.4</td><td>48.9</td><td>39.0</td><td>50.2</td><td>32.2</td><td>26.8</td>
  </tr>
  <tr>
    <td>RoBERTa-base</td><td>71.6</td><td>64.5</td><td>53.1</td><td>73.8</td><td>49.6</td><td>38.3</td><td>50.4</td><td>33.4</td><td>26.6</td>
  </tr>
  <tr>
    <td>Longformer-base</td><td>73.7</td><td>66.9</td><td>54.6</td><td>75.9</td><td>53.0</td><td>40.8</td><td>52.5</td><td>36.1</td><td>28.6</td>
  </tr>
  <tr>
    <td>Longformer-large</td><td>73.9</td><td>68.8</td><td>57.3</td><td>76.5</td><td>55.0</td><td>44.1</td><td>53.6</td><td>38.7</td><td>31.5</td>
  </tr>
  <tr>
    <td>GPT-4o-mini</td><td>58.6</td><td>33.5</td><td>39.7</td><td>68.6</td><td>22.6</td><td>23.9</td><td>41.0</td><td>11.6</td><td>14.2</td>
  </tr>
  <tr>
    <td>distilRoBERTa</td><td>61.2</td><td>-</td><td>-</td><td>-</td><td>-</td><td>-</td><td>39.23</td><td>-</td><td>-</td>
  </tr>
  <tr>
    <td>TF-IDF</td><td>63.5</td><td>57.4</td><td>50.2</td><td>65.9</td><td>42.6</td><td>34.7</td><td>39.3</td><td>25.5</td><td>20.9</td>
  </tr>
  <tr>
    <td>Most Frequent</td><td>46.7</td><td>52.6</td><td>36.8</td><td>52.0</td><td>25.7</td><td>19.8</td><td>41.2</td><td>19.6</td><td>17.5</td>
  </tr>
</table>

Table 17: Performance comparison of various models across different evaluation metrics.

were easily reproducible. However, our finetuned distilRoBERTa could not reach the same performances as Morio and Manning (2023)’s finetuned models. Moreover, GPT-4o-mini could not output better results than the TF-IDF/linear baseline.

C.9 Question Answering

Task Description Given an input question and a set of resources (paragraphs or documents), output an answer to the question.

Experiments We collected ClimaQA and CimaINS and reproduced the experiments with the setting described in Section B and reported the results in Table 18. Our pre-processing step heavily altered the datasets as they contained many problematic duplicates. We found that 10% ClimaQA’s dataset are duplicates with mismatching labels (a given question and its answer would appear with both positive and negative labels). We had to re-build the dataset from the original CDP questionnaires answered by the companies. This dataset is called ClimaQA (our split) in the evaluation. CimaINS also contained many duplicates due to the companies answering similarly each year, and companies answering similarly to the parent-company (reducing the dataset size by 25% and avoiding contamination). Around 2.4% of the remaining answers are duplicates with mismatching labels. We clean the dataset by removing all problematic examples, we name this dataset CimaINS (our split). We framed ClimaQA as a classification task, to fit the settings of our experiments.

Results Analysis Firstly, for CimaINS, in the original paper the authors reported a performance of 84.4%, which we could reproduce with the same settings, even reaching 89.3% with DistilRoBERTa. However, as described previously, we significantly altered the dataset, removing duplicates and train-test contamination. This resulted in lower performances from fine-tuned models, both Longformer and DistilRoBERTa, as shown in Table 18. With this split, all models are under-performing the TF-IDF baseline. This suggests that while each question induces a different vocabulary, understanding the responses actually makes it harder to identify the source question.

For ClimaQA, with the original split the models could not reach more than an F1-score of 50%. This was due to the duplicates with mismatching labels introducing noise. However, when re-constructing the dataset, the fine-tuned models can reach excellent performances, around 89% as shown in Table 18. Contrary to CimaINS, the zero-shot approaches do not under-perform the TF-IDF baseline. Moreover, the gap with the fine-tuned models is smaller (around 9%).

In the original paper, the authors framed ClimaQA as a retrieval task, using models to rank the answers. The best model is MiniLM, reaching a MRR (Mean Reciprocal Ranking) of 0.755. We also computed the MRR of the distilRoBERTa fine-tuned during our classification experiment, as well as a random baseline and a distribution baseline in Table 19. The distribution baseline always ranks the question based on the frequency of the question in the training set. As reported in the table, the fine-tuned model reached a MRR of only 0.636. We believe that the performance might be improved by using a larger training dataset than ours (limited to 10,000 samples).
<table>
  <tr>
    <th>Dataset</th>
    <th>Random</th>
    <th>TF-IDF</th>
    <th>Longformer</th>
    <th>DistilRoBERTa</th>
    <th>GPT-4o-mini</th>
    <th>Llama</th>
    <th>Llama 70B</th>
    <th>Reference</th>
  </tr>
  <tr>
    <td>ClimaINS(Spokoyny et al., 2023)</td>
    <td>12.1(10.6-14.0)</td>
    <td>77.5 (75.3-79.5)</td>
    <td>84.7 (82.5-86.4)</td>
    <td>89.3 (85.4-100.0)</td>
    <td>-</td>
    <td>-</td>
    <td>-</td>
    <td>88.60*</td>
  </tr>
  <tr>
    <td>ClimaINS (our split)(Spokoyny et al., 2023)</td>
    <td>12.8(11.1-14.6)</td>
    <td>81.2 (79.0-83.2)</td>
    <td>84.7 (82.5-86.4)</td>
    <td>89.3 (85.4-100.0)</td>
    <td>58.5 (55.5-61.0)</td>
    <td>45.4(42.7-47.8)</td>
    <td>48.4(45.9-51.0)</td>
    <td>-</td>
  </tr>
  <tr>
    <td>climaQA (Spokoyny et al., 2023)</td>
    <td>45.5 (44.9-46.1)</td>
    <td>47.7 (47.1-48.3)</td>
    <td>44.3 (44.1-44.4)</td>
    <td>-</td>
    <td>-</td>
    <td>-</td>
    <td>-</td>
    <td>-</td>
  </tr>
  <tr>
    <td>climaQA (our split)(Spokoyny et al., 2023)</td>
    <td>50.5(49.4-51.6)</td>
    <td>50.3(49.2-51.3)</td>
    <td>89.5(88.9-90.2)</td>
    <td>89.4(88.7-90.0)</td>
    <td>78.8(77.9-79.6)</td>
    <td>57.4(56.3-58.4)</td>
    <td>76.1(75.1-77.0)</td>
    <td>-</td>
  </tr>
</table>

Table 18: Performances of baselines for each dataset using the train, test describe in section 3.3. The performance computed is the macro F1-score. The 95% confidence interval is computed using a 1000 bootstrap of sampled with replace as True of the size of the original test set. The reference performance is the best relevant performances reported in the original papers. * SVM

<table>
  <tr>
    <th></th>
    <th colspan="2">Ours</th>
    <th colspan="2">Spokoyny et al. (2023)</th>
  </tr>
  <tr>
    <th></th>
    <th>random</th>
    <th>distribution</th>
    <th>ClimateBERT</th>
    <th>MiniLM</th>
  </tr>
  <tr>
    <th>MRR</th>
    <td>0.106 (0.10 - 0.11)</td>
    <td>0.343 (0.33 - 0.35)</td>
    <td>0.636 (0.63, 0.65)</td>
    <td>0.753</td>
    <td>0.755</td>
  </tr>
</table>

Table 19: Performances on the task of ranking question based on the answer for ClimaQA(Spokoyny et al., 2023). random ranks randomly each question, distribution ranks the question from the most frequent to the least frequent based on the train dataset, distilRoBERTa use the outputs of distilRoBERTa for each question/answer pairs to rank the questions. Finally, ClimateBERT and MiniLM are the performances reported by Spokoyny et al. (2023).

C.9.1 Error Analysis

ClimaINS This dataset contains labels associated to each question, for example “EMISSION” correspond to “Does the company have a plan to assess, reduce or mitigate its emissions in its operations or organizations? If yes, please summarize”. It is balanced, except for “EMISSIONS” being twice as frequent. As shown in Figure 11, the easiest label to identify is the “EMISSION” label, and the hardest are “ASSESS”, “MITIGATE” and “RISK PLAN”.

![Confusion matrix of GPT-4o-mini predictions on ClimaINS dataset.](page_1012_1342_388_312.png)

Figure 11: Confusion matrix of GPT-4o-mini predictions on ClimaINS dataset.

In the 11 sampled errors, we found 6 instances that are cropped or out-of-context; therefore, they are harder to classify, as they provide fewer contextual hints about the original question. Additionally, the average length of text misclassified is shorter than the length of correctly labeled text (respectively 635 and 1091 characters), and more than 54% of text misclassified are shorter than 300 characters compared to only 25% of the correctly classified. This shows that this task is significantly harder with less context and details. We also found one statement referring to its parent company, making it impossible to predict, as well as three statements that are partly or fully off-topic (Example C.75). Overall, we found that most of the sampled errors could answer multiple questions. The dataset is derived from a mandatory survey, which may have led to varying levels of response quality from the companies. Some companies provided uniform responses across all their subsidiaries, while others submitted minimalistic answers, making it difficult to discern which specific questions were addressed.

Example C.75 (MITIGATE: Summarize steps the company has taken to encourage policyholders to reduce the losses caused by climate change-influenced events.). However, Lincoln Financial has adopted electronic document delivery and document delivery suppression initiatives to reduce paper and energy usage.

ClimaQA Instead of using each question as a label, as in ClimaINS, in this dataset is a binary classification: given a response and a question, the model predicts if the response answers the question. Therefore, we reported the F1-scores per question in Figure 12. We see that Q29 to Q38 have F1-scores of either 0% or 1%. This is due to the small size of the subset (less than 5 instances). When fo-
cusing on larger subsets, we see that most questions have F1-scores above 60%, with the exception of Q13 and Q21. For Q13, the recall is around 25%, and for Q21 the recall is 0%. This is because those questions are about “other targets” which make these questions more open, resulting in varied content. For instance, in Example C.76 the company provided a target about waste management, which is indirectly related to climate-change.

![Bar chart showing macro F1-score and size for each question in the ClimmaQA dataset](page_184_670_579_377.png)

Figure 12: F1-scores (macro) of GPT-4o-mini on ClimmaQA dataset, grouped by question. In blue: The F1-scores. In orange: the size of the subset.

70% of FN are questions that actually contain sub-questions. For example, Q3: “Which of the following risk types are considered in your organization’s climate-related risk assessments?” has sub-questions specific to “Market Risk”, “Chronic physical Risk”, etc. When the company answers that a risk is considered, they provide a text specific to that sub-category. The response in Example C.77 was answering the sub-categories: “Acute physical”. However, given the question Q3, one might expect the answer to list all the risks considered, prompting them to classify it as “No (not answering the question)” - as did GPT-4o-mini. For this kind of question, we should construct the dataset differently in order to emphasize that the answer is about a sub-category and often a justification of a value previously reported.

Example C.76. [Yes]
Q13: “Provide details of other key climate-related targets not already reported in question C4.1/a/b.”
Answer: "We are targeting a 65% waste diversion rate by 2022 associated in our real estate portfolio."

Example C.77. [Yes]
Q3: "Which of the following risk types are considered in your organization’s climate-related risk assessments? (Current regulation, Market , Acute physical , Upstream , Emerging regulation, Technology , Legal , Reputation , Downstream , Chronic physical )"
Answer: "[...] At this point, it is difficult to predict and assess the probability of potential risks related to a global warming trend on Dow specifically. Concerns have been raised that climate change may result in sea level rise or average temperature rise. Dow has operations and supply chains in coastal areas that potentially could be affected by these. To mitigate risks associated with severe weather, Dow has engineered the facilities to better withstand climate related events."

C.10 Classification of deceptive techniques:
Task Description The goal is to classify statements into argumentative categories: fallacies, types of arguments, or rhetorical techniques.

Experiment We collected CC-Contrarian Claims and LogicClimate and reproduced the experiments with the setting described in section B and reported the results in Table 20. Our pre-processing steps did not alter the datasets significantly.

Result Analysis As shown in Table 20, the performances on LogicClimate and on CC-Contrarian Claims datasets are widely different. The performances on CC-Contrarian Claims for fine-tuned models reach F1-scores of up to 71%. On the contrary, for LogicClimate the best-performing approaches are the zero-shot models reaching only 27%. In both cases we reached performances in the same order of magnitude as in the original studies, but our approaches underperformed those reported in the original studies.

On LogicClimate, our fine-tuned models performed poorly (F1-score between 9-15%), underperforming the random classifier. In the original study, they reported an F1-score of 29.37%. This can be explained by the fact that we only used the domain-specific dataset. Its small size, combined with the label diversity and difficulty, makes this dataset alone less suitable for fine-tuning. Jin et al. (2022) actually fine-tuned the model on general fallacy detection first. For the label difficulty, our TF-IDF baseline performed similarly to the random classifier (13%) suggesting that the labels are not distinguishable through vocabulary only. All this can highlight that the dataset is really challenging, or it could also indicate that there is an issue in
<table>
  <tr>
    <th>Dataset</th>
    <th>Random</th>
    <th>TF-IDF</th>
    <th>Longformer</th>
    <th>DistilRoBERTa</th>
    <th>GPT-4o-mini</th>
    <th>Llama</th>
    <th>Llama 70B</th>
    <th>Reference</th>
  </tr>
  <tr>
    <td>CC-Contrarian Claims (Coan et al., 2021)</td>
    <td>3.2(2.6-4.0)</td>
    <td>62.7(60.1-65.1)</td>
    <td><b>71.6(68.5-73.9)</b></td>
    <td><b>71.2(68.6-73.6)</b></td>
    <td>58.8(55.6-61.3)</td>
    <td>40.2(37.0-43.2)</td>
    <td>55.8(52.6-58.4)</td>
    <td>79/</td>
  </tr>
  <tr>
    <td>logicClimate (Jin et al., 2022)</td>
    <td>13.3(11.4-15.3)</td>
    <td>13.8(11.8-18.0)</td>
    <td>15.6(9.6-20.9)</td>
    <td>9.4(6.0-12.4)</td>
    <td>27.4(15.8-31.4)</td>
    <td>10.9(5.4-14.0)</td>
    <td>22.1(15.3-28.1)</td>
    <td>29.37*</td>
  </tr>
</table>

Table 20: Performances of baselines for each dataset using the train, test describe in section 3.3. The performance computed is the macro F1-score. The 95% confidence interval is computed using a 1000 bootstrap of sampled with replace as True of the size of the original test set. The reference performance is the best relevant performances reported in the original papers. J. RoBERTa + Logistic Regression, k. Electra

the dataset itself. Fallacy detection is highly subjective and IAA agreements are often quite low in that context (Helwe et al., 2024) making them difficult to annotate. Their best-performing models, which can reach 60% on general-domain fallacy detection, only reached 29% on this domain-specific dataset. Our zero-shot approaches reached similar performances (macro F1-score of 27% and micro F1-score of 26% for GPT-4o-mini). The performance gap can also be explained by the source. Indeed, the climate fallacies are taken from real arguments, extracted from a fact-checking website, which are therefore context-dependent and not necessarily explicit, compared to general fallacies which are toy examples extracted from quizzes.

On the contrary, on CC-Contrarian Claims, the fine-tuned transformers performed well reaching performances above 70%. This is lower than the performances reported in the original study (79%). This performance gap might be due to the downsizing of the training dataset to 10,000 examples. As there are 18 labels, it resulted in only around 500 examples per label (compared to 1300 originally) and 4 labels appeared less than 200 times in our training dataset. Our TF-IDF baseline reached an F1-score of 62.7%, which is extremely high when considering that there are 18 labels. This shows that the vocabulary used in each argument category is quite specific. In a pilot study, Coan et al. (2021) measured a really low IAA, \( \kappa = 0.19 \), showing that the task of classifying in a large number of categories might be really difficult for humans. They limited the disagreement by using a decision tree for annotations. They also computed the accuracy of the human annotators on a subset of the dataset for the classification of super-claims. They found an average accuracy above 95% for 4 categories, 86% for the super-claim “Climate movement/science is unreliable”, and only 50% for “No Claim”. This shows that while annotators disagree on the fine-grained categories, they can easily distinguish between broader categories. We also found that most of the errors of GPT-4o-mini occur between sub-claims of the same category (as shown in Figure 13). Both the models and the annotators struggle to classify an argument as “No Claim”. Interestingly, GPT-4o-mini and humans are not biased in the same manner. The model tends to over-classify as “No Claim”, while humans over-classified as “Contrarian Claims”.

C.10.1 Error analysis
CC-Contrarian Claims As shown in figure 13 most of the errors are GPT-4o-mini predicting "No Claim" (42.45%), or errors between the claim types of the same category (31.60%). Only 25.94% of errors are unrelated.

While investigating the errors, we found that 9/30 of statements are multi-faceted, and could fit into multiple categories. Usually, one category is more dominant. Example C.79 could fit in the category of “CO2 is not a pollutant”, but the core argument is “we see no impact of higher CO2 levels on species”. They could also fit into multiple categories because labels are closely linked such as “Weather is cold/snowing” and “Ice/permafrost/snow cover isn’t melting”. In our sample (30 errors), we also found 4 FN which would require more context, such as Example C.78 which could be a contrarian argument by a large emitter advocating against carbon tax; however, it could also be a statement by an activist advocating for stricter regulations. It is not possible to choose without more context. We found 11 texts contain no explicit climate change contrarian claim, however, the arguments can lead to a contrarian claim. For instance, Example C.80 might be part of an unrelated scientific publication, but it could be used in a climate-denial rhetoric to argue that CO2 is actually beneficial.

In 4 cases the model provide the correct reasoning, but decided to use the “No Claim” label anyway. Overall the labels are really detailed and well define, but due to the granularity the labels are really close together. While we can use the Super-claims labels for evaluation instead of the Sub-Claims, this task could also be a multi-label classification tasks in order to keep the nuanced
Figure 13: Confusion matrix of GPT-4o-mini predictions on CC-Contrarian Claims dataset.

labels.
Example C.78 (No Claim). While a price on GHG emissions would encourage some helpful kinds of innovation, it will not generate other kinds. Coping with climate change will require major breakthroughs in basic science. Such breakthroughs are often elusive, and seeking them is an inherently high-risk venture. The private sector finds it difficult to capture the economic rewards of funding basic science, and placing a price on GHG emissions will not correct this bias. As a result, the private sector usually does not make large, sustained investments in basic science; yet that kind of investment is the key to achieving the breakthroughs needed in climate policy.
Example C.79 (Species/plants/reefs aren’t showing climate impacts/are benefiting from climate change). The ocean acidification story depends only on a chemical hypothesis whereas biological factors can overcome this and create conditions that allow calcification to continue. This is corroborated by the historical record of millions of years of success in much higher CO2 environments.
Example C.80 (CO2 is beneficial/not a pollutant). Although elevated CO 2 did not significantly increase total seedling biomass, it did increase it by 14% when averaged across all temperature and fertilization regimes. However, elevated temperature did significantly increase seedling biomass by 55%, when averaged across all CO 2 and fertilization treatments, as did fertilization, by 157%, when averaging across all CO 2 and temperature levels. With respect to root exudation, a similar pattern emerged. Elevated CO 2 did not significantly increase total dissolved organic carbon compounds exuded from seedling roots over a 24-hour period, yet plants
grown in elevated CO 2 exuded 20% more such compounds than ambiently-grown plants did, when averaged across all temperature and fertilization treatments. And once again, elevated temperature and fertilization significantly increased root exudation by 71 and 55%, respectively, when averaged across the other main effect variables.

LogicClimate LOGIC Climate is a dataset that was built to challenge models trained on LOGIC dataset. Therefore, some labels are quite rare, such as circular reasoning (appears once in the test set), equivocation (appears 4 times in the test set) or fallacy of extension (appears 5 times in the test set) which can bias the macro F1-score.

Correctly defining a list of fallacies is not a trivial task, as fallacies have different granularity and can easily overlap (Helwe et al., 2024). Fallacies such as “intentional fallacy” and “Deductive Fallacy” could fit most cases as fallacies are by essence flawed reasoning. While the annotators identified 44 “intentional” fallacies, GPT-4o-mini predicted this fallacy only twice. Moreover, Helwe et al. (2024) concluded that the task of identifying fallacies at a granular level is extremely difficult because of the expertise required and the subjectivity of the task. This explains the low F1-score of both fine-tuned models and zero-shot approaches. Among the errors, we found 3 instances which are not arguments but statements or claims alone. Multiple instances are out-of-context (40%), therefore, missing important information (Example C.81). However, this dataset is particularly interesting due to its construction process relying on fact-checking websites and therefore expert annotations.

Example C.81. Global sea level rose permanently by 1.5 millimeters as a result . [faulty generalization]

C.11 Summary of actual errors

This section contains the details of the actual errors of GPT-4o-mini. These errors are known downfalls of LLMs such as hallucination or being distracted by elements in the prompt (Perković et al., 2024; Shi et al., 2023), but also genuine errors.

Hallucinations and Anchoring We observed a few instances of hallucinations in GPT-4o-mini predictions. It predicted non-existent labels in SciDCC, and provided a nonsensical reasoning for a claim-evidence relation prediction in ClimateFEVER evidence. Despite observing some, in our experiments, the model rarely hallucinates. Additionally, GPT-4o-mini is sometimes anchored to a specific word in the prompt (2/9 on climateext (10K)) making it misunderstand the instructions.

Redefining terms GPT-4o-mini also struggles when the terms use an alternative definition of known terms (2/16 climate specificity, 6/10 in ClimateEng, 5/20 on ClimateBUG Data). (1) In ClimateEng, the label “Politics” is restricted to “leaders, political organizations, policies” therefore excluding activists/protesters. Similarly the label “Ocean/Water” is specific to biodiversity, therefore excluding sea rise. (2) In climate specificity, the label “specific” includes all captions and footnotes because they enable readers understand the overall information reported.

Bias In two of the datasets we observed potential biases. On climate sentiment, when ambiguity arises (“neutral”), GPT-4o-mini, with our prompt, tends to output preferably the “Opportunity” label (15%) compared to the “Risk” label (12%). The precision for “Opportunity” is 59%, while it is above 80% for both “Neutral” and “Risk”. This is not the case for distilRoBERTa - on “neutral” texts, it predicts “Opportunity” (11%) less than “Risk” (16%). This tends to indicate that the model, with our prompt, has a positive bias. We observed another systematic bias, on ClimateFEVER. We observed that GPT-4o-mini, with our prompt, tends to predict disproportionately (1) “NOT ENOUGH INFO” instead of “SUPPORTS”, and (2) “REFUTES” instead of “NOT ENOUGH INFO”. In (1) the claims are all support the existence of climate change; and in (2) the claims all deny climate change. GPT-4o-mini, with our prompt, avoid classifying as “NOT ENOUGH INFO” evidence refuting climate denial claims, but not arguments supporting the seriousness of climate change.

Polysemy On recurrent error is the misunderstanding of the word “sustainable” which is polysemic. It was initially used to mean “consistent” or “viable”, such as in “This business model is sustainable”. However, it is more and more frequently meaning “environmentally and/or socially responsible”. When the term can be interpreted both ways (e.g. “Sustainable business”), GPT-4o-mini, with our prompt, sometimes interprets it as “environmentally and/or socially responsible” (7/17 on Climate detection, 5/20 on ClimateBUG, 1/8 on ESGBERT E). This can be caused by an intrinsic
bias of the model, a more dominant definition in the model’s knowledge, or due to the model being anchored by the prompt which focuses on climate.

Ambiguity   We found that the model tends to fail on labels that are more ambiguous, and performs better on labels that are clear-cut binary opposites. (1) 65% of errors in climate sentiment occurs on texts with the “neutral” label. (2) The precision of the model on “implicit green claims” is significantly lower than on “explicit green claims” or on “not green” on Implicit/Explicit Green Claims. It reached a precision of 55% on “implicit green claims” while all other recall and precision are above 80%. (3) 95% of errors involves “NOT ENOUGH INFO” in ClimateFEVER evidence. (4) 70% of errors include “Ambiguous” on ClimateStance. (5) 85% of errors involve “neutral” label on GWSD. (6) 32% of errors involve labels in the same super-category (compared to 26% different super-category, and 42% for with “no claim”) in CC-Contrarian Claims

Context   Many of the datasets consist of sentences or paragraphs which often miss context. We found the model struggle with the lack of context as 12.5% of the errors that we sampled were statements out of context or truncated. Moreover, in ClimaINS, we found that shorter texts are harder to classify for GPT-4o-mini: misclassified examples are on average shorter than correctly classified ones (635 vs 1091 characters), and 54% of the misclassified examples were shorter than 300 characters, compared to 25% in the correctly classified ones.

D Toolbox

The script is available at https://github.com/tcalamai/acl_climateNLPToolbox. We provide here an example of the toolbox outputs in Figure 14. This was generated with the following simple script:

from src.analysis import process_folder,
    visualize

# Path of the folder containing the PDF files
path="case_study/data/"

# Transform PDF into a machine-readable dataframe:
process_folder(model_name="tfidf", path=path)

# Run all models and generate statistics:
visualize(model_name="tfidf")
(a) Company 1                (b) Company 2

Figure 14: Visualization of the statistics generated for 2 companies.

Metareview score: 3
Metareview: The authors conduct a reproducibility study of climated-related NLP tasks, identifying and correcting issues with datasets, such as duplicates and annotation errors, and applying and evaluating six models. They release the cleaned datasets, evaluation code, and a software pipeline for applying the models.  

The reviewers recognised the wide scope of the benchmarking effort, and the potential value of a unified and cleaned benchmark. The paper also provides interesting findings about model performance and error analysis. While it does not propose any methodological innovations, the benchmark and key results will be of interest to the community.
Summary of reasons to publish: This is a growing field of research, and the authors present a unified, reliable benchmark with a useful description of the tasks and datasets in Climate NLP. Reviewers noted the comprehensiveness, with a wide range of datasets and eight task categories. This is a much broader range than previous benchmarks. The analysis covers a good breadth of methods (simple baselines up to recent LLMs) and provides preprocessing/evaluation scripts and prompts for reproducibility. Reviewers appreciated the contribution to reproducibility and dataset quality, particularly the identification of annotation errors with a strong error analysis process.  The volume of information provided shows a high level of care in the experiments.