grown in elevated CO 2 exuded 20% more such compounds than ambiently-grown plants did, when averaged across all temperature and fertilization treatments. And once again, elevated temperature and fertilization significantly increased root exudation by 71 and 55%, respectively, when averaged across the other main effect variables.

**LogicClimate** LOGIC Climate is a dataset that was built to challenge models trained on LOGIC dataset. Therefore, some labels are quite rare, such as circular reasoning (appears once in the test set), equivocation (appears 4 times in the test set) or fallacy of extension (appears 5 times in the test set) which can bias the macro F1-score.

Correctly defining a list of fallacies is not a trivial task, as fallacies have different granularity and can easily overlap (Helwe et al., 2024). Fallacies such as “intentional fallacy” and “Deductive Fallacy” could fit most cases as fallacies are by essence flawed reasoning. While the annotators identified 44 “intentional” fallacies, GPT-4o-mini predicted this fallacy only twice. Moreover, Helwe et al. (2024) concluded that the task of identifying fallacies at a granular level is extremely difficult because of the expertise required and the subjectivity of the task. This explains the low F1-score of both fine-tuned models and zero-shot approaches. Among the errors, we found 3 instances which are not arguments but statements or claims alone. Multiple instances are out-of-context (40%), therefore, missing important information (Example C.81). However, this dataset is particularly interesting due to its construction process relying on fact-checking websites and therefore expert annotations.

**Example C.81.** Global sea level rose permanently by 1.5 millimeters as a result. [faulty generalization]

## C.11 Summary of actual errors

This section contains the details of the actual errors of GPT-4o-mini. These errors are known downfalls of LLMs such as hallucination or being distracted by elements in the prompt (Perković et al., 2024; Shi et al., 2023), but also genuine errors.

**Hallucinations and Anchoring** We observed a few instances of hallucinations in GPT-4o-mini predictions. It predicted non-existent labels in *SciDCC*, and provided a nonsensical reasoning for a claim-evidence relation prediction in *Climate-FEVER evidence*. Despite observing some, in our

experiments, the model rarely hallucinates. Additionally, GPT-4o-mini is sometimes anchored to a specific word in the prompt (2/9 on *climatext* (10K)) making it misunderstand the instructions.

**Redefining terms** GPT-4o-mini also struggles when the terms use an alternative definition of known terms (2/16 *climate specificity*, 6/10 in *ClimateEng*, 5/20 on *ClimateBUG Data*). (1) In *ClimateEng*, the label “Politics” is restricted to “leaders, political organizations, policies” therefore excluding activists/protesters. Similarly the label “Ocean/Water” is specific to biodiversity, therefore excluding sea rise. (2) In *climate specificity*, the label “specific” includes all captions and footnotes because they enable readers understand the overall information reported.

**Bias** In two of the datasets we observed potential biases. On *climate sentiment*, when ambiguity arises (“neutral”), GPT-4o-mini, with our prompt, tends to output preferably the “Opportunity” label (15%) compared to the “Risk” label (12%). The precision for “Opportunity” is 59%, while it is above 80% for both “Neutral” and “Risk”. This is not the case for *distilRoBERTa* - on “neutral” texts, it predicts “Opportunity” (11%) less than “Risk” (16%). This tends to indicate that the model, with our prompt, has a positive bias. We observed another systematic bias, on *ClimateFEVER*. We observed that GPT-4o-mini, with our prompt, tends to predict disproportionately (1) “NOT ENOUGH INFO” instead of “SUPPORTS”, and (2) “REFUTES” instead of “NOT ENOUGH INFO”. In (1) the claims are all support the existence of climate change; and in (2) the claims all deny climate change. GPT-4o-mini, with our prompt, avoid classifying as “NOT ENOUGH INFO” evidence refuting climate denial claims, but not arguments supporting the seriousness of climate change.

**Polysemy** On recurrent error is the misunderstanding of the word “sustainable” which is polysemic. It was initially used to mean “consistent” or “viable”, such as in “This business model is sustainable”. However, it is more and more frequently meaning “environmentally and/or socially responsible”. When the term can be interpreted both ways (e.g. “Sustainable business”), GPT-4o-mini, with our prompt, sometimes interprets it as “environmentally and/or socially responsible” (7/17 on *Climate detection*, 5/20 on *ClimateBUG*, 1/8 on *ESGBERT E*). This can be caused by an intrinsic
bias of the model, a more dominant definition in
the model’s knowledge, or due to the model being
anchored by the prompt which focuses on climate.

**Ambiguity** We found that the model tends to fail on labels that are more ambiguous, and performs better on labels that are clear-cut binary opposites. (1) 65% of errors in *climate sentiment* occurs on texts with the “neutral” label. (2) The precision of the model on “implicit green claims” is significantly lower than on “explicit green claims” or on “not green” on *Implicit/Explicit Green Claims*. It reached a precision of 55% on “implicit green claims” while all other recall and precision are above 80%. (3) 95% of errors involves “NOT ENOUGH INFO” in *ClimateFEVER evidence*. (4) 70% of errors include “Ambiguous” on *ClimateStance*. (5) 85% of errors involve “neutral” label on *GWSD*. (6) 32% of errors involve labels in the same super-category (compared to 26% different super-category, and 42% for with “no claim”) in *CC-Contrarian Claims*

**Context** Many of the datasets consist of sentences or paragraphs which often miss context. We found the model struggle with the lack of context as 12.5% of the errors that we sampled were statements out of context or truncated. Moreover, in *ClimaINS*, we found that shorter texts are harder to classify for GPT-4o-mini: misclassified examples are on average shorter than correctly classified ones (635 vs 1091 characters), and 54% of the misclassified examples were shorter than 300 characters, compared to 25% in the correctly classified ones.

D Toolbox

The script is available at https://github.com/tcalamai/acl_climateNLPtoolbox. We provide here an example of the toolbox outputs in Figure 14. This was generated with the following simple script:

from src import process folder,
↔ visualize

# Path of the folder containing the PDF files
path ="case study/data/"

# Transform PDF into a machine-readable
↔ dataframe:
process folder(model_name="tfidf", path=path)

# Run all models and generate statistics:
visualize(model_name="tfidf")
<figure><img src="image_18.png" /><figcaption>Figure 14: Visualization of the statistics generated for 2 companies.</figcaption></figure>