Environmental Claim Detection
Year: 2023

Abstract:
To transition to a green economy, environmental claims made by companies must be reliable, comparable, and verifiable. To analyze such claims at scale, automated methods are needed to detect them in the first place. However, there exist no datasets or models for this. Thus, this paper introduces the task of environmental claim detection. To accompany the task, we release an expert-annotated dataset and models trained on this dataset. We preview one potential application of such models: We detect environmental claims made in quarterly earning calls and find that the number of environmental claims has steadily increased since the Paris Agreement in 2015.

heading: 1 Introduction
text: In the face of climate change, we witness a transition towards a more sustainable and green economy. This change is driven by changes in regulation, public opinion, and investor attitudes. For example, global assets managed under a sustainability label are on track to exceed $53 trillion by 2025, more than a third of total assets under management. However, unfortunately, the boom has been accompanied by rampant greenwashing, with companies boasting about their environmental credentials.1 Because of this surge in environmental claims and to protect consumers, initiatives on substantiating green claims are developed.2 Due to an ever-growing amount of text, there is a need for automated methods to detect environmental claims. Detecting such claims at scale can assist policy-makers, regulators, journalists, activists, the research community, and an informed public in analyzing and scrutinizing environmental claims made by companies and facilitating the transition to a green economy.
1See, e.g., The Economist, May 22nd, 2021. 2For example an EU initiative on green claims:
https://ec.europa.eu/environment/eussd/smgp/ initiative_on_green_claims.htm
Thus, we introduce the task of environmental claim detection. Environmental claim detection is a sentence-level classification task with the goal of predicting whether a sentence contains an environmental claim or not. Often, environmental claims are made in a clear and concise matter on a sentence level, with the intention to convey to a consumer or stakeholder that a company or product is environmentally friendly.
To facilitate future research on environmental claim detection, we release an expert-annotated dataset containing real-world environmental claims and models which can be used by practitioners. For constructing the dataset, we were inspired by the European Commission (EC), which defines such claims as follows: Environmental claims refer to the practice of suggesting or otherwise creating the impression (in the context of a commercial communication, marketing or advertising) that a product or a service is environmentally friendly (i.e., it has a positive impact on the environment) or is less damaging to the environment than competing goods or services.3 While such claims can be truthful and made in good faith, boasting about environmental credentials can also be monetized (de Freitas Netto
3From the Commission Staff Working Document, Guidance on the implementation/application of Directive 2005/29/EC on Unfair Commercial practices, Brussels, 3 December 2009 SEC(2009) 1666. See section 2.5 on misleading environmental claims.
1051
et al., 2020). For example, consumers are willing to spend more money on environmentally friendly products (Nielsen Media Research, 2015). The Commission states if environmental claims are too vague, unclear, or misleading, we are confronted with an instance of "greenwashing" (this definition is given in the same Commission Staff Working Document).
We situate environmental claim detection at the intersection of claim detection (e.g., Arslan et al., 2020) and pledge detection (Subramanian et al., 2019; Fornaciari et al., 2021). An environmental claim is typically made to increase the environmental reputation of a firm or a product. We show that models trained on the current claim and pledge detection datasets perform poorly at detecting environmental claims, hence the need for this new dataset. We make our dataset, code and models publicly available.4 Lastly, we envision computerassisted detection of greenwashing in future work, i.e., the automatic determination if an environmental claim is false, too vague, non-verifiable, or misleading. To make progress on automated greenwashing detection, it is mandatory to first detect environmental claims at scale.

heading: 2 Related Work
text: This work is part of an ongoing effort at the intersection of environmental and climate changerelated topics and natural language processing (Stede and Patz, 2021). Resulting datasets and methods can help regulators, policy-makers, journalists, the research community, activists, and an informed public investigate such topics at scale with the help of computer assistance. Methods include ClimateBERT (Webersinke et al., 2021), and ClimateGPT (Vaghefi et al., 2022), two language models pre-trained on climate-related text. NLP tasks and datasets include climate change topic detection (Varini et al., 2020) and detecting media stance on global warming (Luo et al., 2020). Duong et al. (2022) collect climate change opinions at scale from social platforms, Al-Rawi et al. (2021) analyze fake news Tweets around climate change. In a similar direction, Coan et al. (2021) analyze contrarian claims about climate change and (Piskorski et al., 2022) explore data augmentation techniques for climate change denial classification.
4We host all code, data and models on https://github. com/dominiksinsaarland/environmental_claims. The dataset can also be accessed as a hugginface dataset, and our model is available on the huggingface model hub.
Further, there exists work about claim verification of climate change related claims (Diggelmann et al., 2020), detecting media stance on global warming (Luo et al., 2020), collecting climate change opinions at scale from social platforms (Duong et al., 2022), and finally, the analysis of regulatory disclosures (Friederich et al., 2021; Kölbel et al., 2022).
In this broader context of applying NLP methods for climate change-related topics, We situate environmental claim detection at the intersection of claim spotting and pledge detection, covering the domain of text produced by companies with the goal of boosting their environmental credentials. Claim spotting is the task of finding fact-check worthy claims (Arslan et al., 2020; Atanasova et al., 2018; Barron-Cedeno et al., 2020). Pledge detection aims to detect pledges made in, for example, political campaigns (Subramanian et al., 2019; Fornaciari et al., 2021). Environmental claims state an environmental benefit (claim) or convey the intention (pledge) for a material impact, i.e., some environmental benefit, which pleases the audience (consumers or stakeholders) of the claim.

heading: 3 Dataset
text: Our dataset contains environmental claims made by listed companies. We collected text from sustainability reports, earning calls, and annual reports of listed companies and annotated 3’000 sentences. After discarding tied annotations, our resulting dataset contains 2’647 examples.5 We provide dataset statistics in Table 1 and a text length histogram in Appendix Figure 4.
The dataset is annotated by 16 domain experts.6
5In the GitHub repository, we also include a link to all 3’000 sentences, with the 4 individual annotations for each datapoint (including ties), in case this additional information is useful for follow-up research.
6All annotators passed a core course on sustainable investing with a high grade. This course is part of the executive education program for the Master of Advanced Studies in Sustainable Finance, offered by the University of Zurich. Most of the annotators have prior work experience in the financial sector.
The authors drafted annotation guidelines in an iterative process and added examples of clear and borderline environmental claims to the guidelines. In Appendix B, we list the complete guidelines available to the annotators, along with examples and rationales that the authors discussed in pilot annotation rounds.
To extract the sentences annotated in our dataset, we use a preliminary model to sample candidate sentences from various text sources produced by firms. Furthermore, we randomly sample sentences from different clusters obtained with k-means to increase the coverage of the domain. We describe the sampling process of the dataset in detail in Appendix A and provide further information on the data sources in Appendix C.
While we do not release a large-scale dataset, this is the result of a conscious decision to prioritize quality over quantity. We employed domain experts to annotate the data, which results in costly annotations. In Appendix D, we show that the performance of models converges after being trained on more than 60% of the training set, and we find diminishing marginal utility of including more sentences. Hence our decision to stop annotation here and release an annotated dataset with 2’647 examples.
We assigned each sentence to four annotators. The annotations are aggregated by majority vote. 60% of the 3’000 samples was decided unanimously by the annotators, and 88.3% of the annotations made were part of a majority decision. 353 sentences received tied annotations (11.7% of the samples), and we discarded these examples from the dataset.The overall inter-annotator agreement measured in Krippendorff’s alpha is 0.47, indicating moderate agreement.

heading: 4 Experiments
text: We conduct two types of experiments: (1) We examine the performance of various models on our dataset, among them pre-trained claim and pledge detection models and fine-tuned environmental claim detection transformer models (such as, e.g. Devlin et al., 2019; Liu et al., 2019; Sanh et al., 2019; Webersinke et al., 2021). (2) we apply our models to the text produced by listed companies, which leads to a small case study demonstrating one of the intended use cases of the dataset.

heading: 4.1 Environmental Claim Detection Models
text: We report various metrics on a 5-fold crossvalidation split of the whole dataset, the development, and the test set in Table 2. We present two poorly performing baselines: majority, where we assign the not-a-claim label to all examples, and random, where we randomly assign one of the two labels to each example. Next, we fine-tune a RoBERTabase model on the ClaimBuster dataset (Arslan et al., 2020), and use this model to detect environmental claims in the dataset.7 While achieving rather high recall, the model does not cope well with the domain shift and fails to detect environmental claims reliably. Similar findings hold for a RoBERTabase model trained on a Pledge Detection dataset (Subramanian et al., 2019).8 These results highlight the need for a dedicated dataset.
Furthermore, we train two SVM models. The first one uses tf-idf bag-of-word features, the sec-
7We train the model to distinguish fact-check-worthy claims vs. all other claims. The model works exceptionally well on the ClaimBuster test set with a micro-F1 of 97.9% and a macro-F1 of 97.0%.
8The model achieves a 67% F1 score and 78% accuracy on a held-out split of the Pledge Detection but also fails to adapt to detect environmental claims.
ond is based on character n-gram features. Both models achieve an acceptable F1 score between 65% and 71% on all dataset splits. These results indicate that environment-related keywords or ngrams are somewhat predictive of whether a sentence is an environmental claim or not. However, all transformer models explored in this study outperform the SVM, hence the presence of environmental keywords alone is not sufficient for predicting such claims. Especially for recall, we find a large gap between transformer and SVM models of up to 25% points. We interpret this gap as evidence that not all environmental claims contain distinguishing environmental keywords.
Lastly, we fine-tune various transformer models (Liu et al., 2019; Sanh et al., 2019; Webersinke et al., 2021). They all achieve an F1 score higher than 82% on all different dataset splits, a vast performance increase compared to the other models examined so far. We observe only minor differences between these models. The biggest model RoBERTalarge achieves the best scores overall, followed by ClimateBERT, a DistilBert-like language model further pre-trained on over 1.6 million climate-related paragraphs. Hence, further pretraining on climate-related text seems beneficial to detect environmental claims.
For training our models, we use Hugging Face (Wolf et al., 2020) and standard RoBERTa hyperparameters. We use the Adam optimizer with a learning rate of 2e-5, a batch size of 16, and train models for 3 epochs. To minimize compute and environmental footprint of our experiments and due to consistent results over different dataset splits, we did not explore other hyper-parameters in more detail and reported only results of single runs.

heading: 4.2 Earning Calls
text: We use our trained model to detect environmental claims in corporate earning calls between 2012 and 2020. These are conference calls between the management of a publicly traded company, analysts, investors, and the media to discuss the company’s financial results and other topics for a given reporting period (mainly quarterly). The conference calls consist of different segments, of which the segment with questions and answers is the most interesting for our purposes. Therefore, we focus on the management responses, which consist of 12 million sentences from 3,361 unique companies. All earnings conference call transcripts are obtained from Refinitiv Company Events Coverage. Due to the size of the data and computational constraints, we use our ClimateBERT model, finetuned on detecting environmental claims instead of the RoBERTalarge model.
We would expect that the amount of environmental claims made by corporations and business leaders has steadily increased since the Paris Agreement in 2015. In Figure 2, we find that this is indeed the case. The amount of environmental claims is not only increasing, but the increase is also accelerating. In 2019, the share of environmental claims is twice as high as in 2015. Not only the amount of environmental claims made in earning calls is increasing, but also the share of companies who makes such claims increased by 33%, and in 2019, one in ten companies makes at least one environmental claim in the answer sections of an earning call.
In Figure 3, we display word clouds for the most important words classified as non-claims (on the
left), and the most important words for environmental claims (on the right). It is evident that the sentences classified as claims contain more environmental-related keywords; We see that these keywords cover different environmental aspects, e.g., recycling and waste, carbon and emissions, renewables, water, etc. In Appendix Table 6, we additionally list the 5 highest and lowest scoring sentences based on our model. Our model effectively identifies environmental claims as the predominant category at the upper end of the distribution, whereas it appears that such claims are absent in the lower end of the distribution.
This small case study illustrates one of the intended use cases of our dataset and the associated models: We present a tool that allows us to detect environmental claims at scale. Having access to environmental claims at scale makes it possible to analyze and scrutinize them in future work.

heading: 5 Conclusion
text: The vast and ever-growing volume of corporate disclosures, regulatory filings, and statements in the news calls for an algorithmic approach to detect environmental claims made by companies at scale. Thus, we introduce the NLP task of detecting environmental claims, a dataset containing such claims and associated models which can detect these claims in the wild. Our dataset is annotated by domain experts and thus of high quality. We describe the dataset and its construction process and present various models for detecting environmental claims in our dataset and a small case study.
We envision several directions for future work. First, we plan to investigate "greenwashing", the practice of making a false, vague, unclear, or mis-
leading environmental claim. To make progress on this front, it is mandatory that we can detect environmental claims in the first place. Second, models trained on detecting environmental claims have merits of their own, as previewed in our case study. We plan to explore more such applications in detail, e.g., analyzing annual reports and TCFD9 reports at scale. For example, it would be interesting to see in which sections of TCFD reports firms make environmental claims. Lastly, we expect an increase of contributions at the intersection of environmental topics, climate change, and NLP in the near future. This work contributes to such efforts.
Limitations
We find several limitations in this work. First, we acknowledge that the technical novelty of this work is limited: We introduce a sequence classification task, and we investigate rather standard models in our experiment section (i.e., state-of-the-art transformer language models). Nevertheless, we believe that there is a gap in the literature for the task presented in this work, hence our introduction of the environmental claim detection task, the dataset, and models.
Second, we collect data from sustainability reports, earning calls, and annual reports. However, this does not cover the universe of text where environmental claims are made, e.g., company websites and product descriptions. Also, environmental claims can be made about environmental improvements on a wide range of topics such as carbon emissions, water pollution, and recycling, among others. We discussed creating different datasets, where each dataset is dedicated to one specific is-
9Task Force on Climate-Related Financial Disclosures
sue. However, we leave this to future work. Third, sometimes it is necessary to have access to more context to determine whether a sentence is an environmental claim. We discussed whether it would be beneficial to annotate whole paragraphs instead. However, the trade-off would be exploding annotation work and costs, hence our decision to introduce environmental claims as a sentence-level classification task (and we specifically asked annotators to reject ambiguous cases as environmental claims). Nevertheless, given a unlimited budget, we would have pursued annotating whole paragraphs instead (or annotating all environmental claims in a paragraph).
Our data sources, e.g., sustainability reports, are mostly published by European and US-listed companies, which is reflected in our dataset. We crawled these reports from the SEC10, hence our dataset contains mostly claims made by (a) big firms and (b) firms from developed countries. It is conceivable that smaller firms and firms from nondeveloped countries make different environmental claims, and models trained on our dataset might not be suitable to detect these claims.
Moreover, our work is subject to all concerns raised in the Ethics Statement below. We find it important to keep all these perspectives in mind when reading and discussing our work.
Ethics Statement
Intended Use: This dataset will benefit journalists, activists, the research community, and an informed public analyzing environmental claims made by listed companies at scale. Also, we see this as a first step towards algorithmic greenwashing detection using NLP methods. It might also be useful to policy-makers and regulators in both the financial sector and the legal domain. Next, we hope companies are inspired by our work to produce more carefully drafted environmental claims. To conclude, we envision that the dataset and related models bring a large positive impact by encouraging truly environmentally friendly actions and less verbose boasting about environmental credentials.
Misuse Potential: Although we believe the intended use of this research is largely positive, there exists the potential for misuse. For example, it is possible that for-profit corporations will exploit AI models trained on this dataset while drafting
10https://www.sec.gov/
environmental claims.
Model Bias: Although the performance of NLP models usually achieves an F1 score of above 80%, it is widely known that ML models suffer from picking up spurious correlations from data. Furthermore, it has been shown that large pre-trained language models such as ClimateBERT suffer from inherent biases present in the pre-training data leading to biased models – and we believe our models presented in this work also suffer from these biases.
Data Privacy: The data used in this study are mostly public textual data provided by companies and public databases. There is no user-related data or private data involved.
Annotator Salary: We paid standard research assistant salaries of around $30 per hour, which is common practice at the University of Zurich. We were upfront in disclosing to annotators that their annotations will lead to a dataset and models which can automatically detect environmental claims. We found that this goal motivated annotators. We speculate (and hope) annotators interpreted the dataset creation process and the goal of releasing the resulting dataset and models as an AI4Good application. The feedback was overwhelmingly positive, and many annotators have asked whether it is possible to participate in follow-up annotation work related to greenwashing detection.

heading: A Sample Selection
text: The basis for selecting samples are documents from four domains in the text produced by companies. We consider TCFD reports that are voluntarily selfdisclosed by firms about their environmental impact, but not legally binding. Furthermore, we consider annual reports, comprehensive reports about activities conducted by a firm in a given year. We also consider corporate earnings calls (only the answer sections), which are conference calls between the management of a public company, analysts, investors, and the media to discuss the company’s financial results and other business-relevant topics during a given reporting period. Earnings conference call transcripts are obtained from Refinitiv Company Events Coverage (formerly Thomson Reuters StreetEvents). Lastly, we include the language data on environmental risks, targets, and performance from the CDP disclosure questionnaire responses from 2021. We denote the universe of these documents by Dlarge. In Table 3, we show many sentences we have from each of these sources (first row), and the distribution of these sources in our final dataset (second row).
In pilot studies, we decided to only keep sentences having more than 10 and less than 40 words. Shorter sentences rarely ever are environmental claims, but a combination of section titles, filler
sentences, and table descriptions. Longer sentences usually are the result of a failure in preprocessing.
A random selection of sentences from these documents would lead to a high number of sentences not related to the environment, thus, is impracticable. We also decided against using a keyword search to pre-filter Dlarge for two reasons. If we use a keyword set that is too narrow, we might have dataset artifacts. On the other hand, if we use a set that is too loose, we might again end up with too many non-climate-related sentences, which again is impracticable.
As a remedy, we start with a handpicked selection of 250 environmental claims used in a recent marketing study about greenwashing in French investment funds by 2DII, an independent, non-profit think tank working to align financial markets and regulations with the Paris Agreement goals. We also consider 200 non-environmental claims as negative samples, randomly sampled from company websites. The authors translated them to English (if necessary) and loosely annotated these sentences to double-check their quality and to help come up with annotation guidelines. However, these 450 sentences do not appear in the final version of the dataset. Next, we train a preliminary RoBERTabase model on this dataset and use this trained model to compute the likelihood of each sentence from Dlarge being an environmental claim. Using this likelihood, we use the following strategy to select both samples with a high chance of being environmental claims, samples with a low chance of being environmental claims, and samples that are semantically similar but lead to very different results compared to our base transformer model:
1. First 300 samples were sampled, which are adjacent to our starting selection of 250 environmental claims in SBERT embedding space (Reimers and Gurevych, 2019), but for which the base transformer model assigned a small score of being an environmental claim.
2. Then, 1500 samples with a score greater than 0.7 from our preliminary transformer model are selected.
3. Next, 500 samples with a score between 0.2 and 0.5 from our preliminary transformer model are selected.
4. Then, we selected 200 samples with a score
lower than 0.2 from our preliminary transformer model.
5. Finally, all encoded samples from SBERT are clustered into 2000 clusters using k-means. The largest clusters, from which no sample was selected in steps 1-4, are then represented by a random sample from the cluster. This way we increase the coverage of the whole domain by our selected samples. We selected 500 samples with that strategy.
While we tried to maximize domain coverage using this sampling procedure, given the limited annotation budget, it is likely that we missed lots of utterances of environmental claims. Also, the sample is somewhat biased toward our preliminary model, which we used to sample environmental claims from. Moreover, we did not include all domains of text produced by listed companies. For example, company websites and advertisements are not included in our universe of documents.

heading: B Annotation Guidelines
text: Your task is to label sentences. The information we need is whether they are environmental claims (yes or no).
A broad definition for such a claim is given by the European Commission: Environmental claims refer to the practice of suggesting or otherwise creating the impression [...] that a product or a service is environmentally friendly (i.e., it has a positive impact on the environment) or is less damaging to the environment than competing goods or services [...]
In our case, claims relate to products, services OR specific corporate environmental performance.
General annotation procedure/principles :
• You will be presented with a sentence and have to decide whether the sentence contains an explicit environmental claim.
• Do not rely on implicit assumptions when you decide on the label. Base your decision on the information that is available within the sentence.
• However, if a sentence contains an abbreviation, you could search online for the meaning of the abbreviation before assigning the label.
• In case a sentence is too technical/complicated and thus not easily understandable, it usually does not suggest to the average consumer that a product or a service is environmentally friendly and thus can be rejected.
• Likewise, if a sentence is not specific about having an environmental impact for a product or service, it can be rejected.
• Final goal: We will train a classifier on these annotations and apply it to massive amounts of financial text to explore which companies/sectors at which time make how many environmental claims. Does the number of environmental claims correlate with sectors/companies reducing their environmental footprint?
• The annotation task is not trivial in most cases. Borderline decisions are often the case. If you are uncertain about your decisions, copy-paste the sentence and add an explanatory note to the sentence. We will then cross-check it in case needed.
In Table 4 and 5, we show examples that were discussed within the author team.
We presented each sentence in our sample to four annotators to determine a label. In the case of a clear majority of the annotators for a sentence (4:0, 3:1, 1:3, or 0:4), the sentence is annotated as such. In case of no majority (2:2), the sentence is discarded and excluded from our final dataset. The rationale behind this is that a sentence annotated as positive accuses the writer to claim something. This accusation should be agreed on by the majority of readers (in dubio pro reo - in doubt, rule for the accused).

heading: C Data Sources
text: We crawled TCFD and annual reports from the SEC (the U.S. Securities and Exchange Commission), specifically from www.annualreports.com www.responsibilityreports.com. Given that sustainability reports are mostly published by European and US firms, there is not an even global coverage in our sample, but a tendency for firms in developed countries. For the reports we collected, we show a distribution of Countries in Figure 5a and Industries in Figure 5b. For the earning calls data, we show a distribution over sectors in Figure 5c.
Un ite
d Ki
ng do m Fr an ce
Un ite
d St
at es
o f A
m er
ica Ja pa n
Au st
ra lia Sp ai n Ne th er la nd s Ge rm an y Ca na da Br az il Ita ly So ut h Af ric a In di a De nm ar k Sw ed en No rw ay Sw itz er la nd M ex ico Ph ilip pi ne s Fin la nd Po rtu ga l Au st ria Co lo m bi a Ru ss ia Ta iw an Be lg iu m
Re pu
bl ic
of Ir
el an d Si ng ap or e
Lu xe
m bo
ur g
M al
ay sia
Pa pa
N ew
G ui
ne a
Lit hu
an ia
0
10
20
30
40
50
60
70
(a) Geographical Distribution for annual and TCFD Reports in our Data.
Ba nk s El ec tri c an d Ga s U til iti es Oi l a nd G as M et al s a nd M in in g
IT a
nd C
om m
un ica
tio ns
S er
vi ce s As se t M an ag er s In su ra nc e Ca pi ta l G oo ds Re al E st at e M an ag em en t a nd D ev el op m en t Au to m ob ile s a nd C om po ne nt s Co ns um er G oo ds a nd A pp ar el Di ve rs ifi ed Fi na nc ia ls Ch em ica ls Pa ss en ge r A ir Tr an sp or ta tio n Co ns tru ct io n M at er ia ls As se t O wn er s M ul tiUt ilit ie s M ar iti m e Tr an sp or ta tio n Pa pe r, Pa ck ag in g an d Fo re st P ro du ct s Re ta il Ph ar m ac eu tic al W at er U til iti es Be ve ra ge s Pa ck ag ed F oo ds a nd M ea ts
Co ns
um er
G oo
ds a
nd A
pp ar
el , D
iv er
sif ie
d Fin
an cia
ls, M
ul ti-
Ut ilit
ie s
Co m
m er
cia l a
nd P
ro fe
ss io
na l S
er vi
ce s
Pa ck
ag ed
F oo
ds a
nd M
ea ts
, P ha
rm ac
eu tic al Ra il Tr an sp or ta tio n To ba cc o
Di ve
rs ifi
ed Fi
na nc
ia ls,
IT a
nd C
om m
un ica
tio ns
S er
vi ce
s, M
ul ti-
Ut ilit
ie s,
Re al
E st
at e
M an
ag em
en t a
nd D
ev el
op m
en t
Co ns
tru ct
io n
M at
er ia
ls, C
on su
m er
G oo
ds a
nd A
pp ar
el , R
ea l E
st at
e M
an ag
em en
t a nd
D ev
el op
m en
t, Re
ta il
Ch em
ica ls,
E le
ct ric
a nd
G as
U til
iti es
0
20
40
60
80
(b) Distribution over Sectors for annual and TCFD Reports in our Data.
he al
th c
ar e
in fo
rm at
io n
te ch
no lo
gy
fin an
cia ls
in du
st ria
ls
co ns
um er
d isc
re tio
na ry
en er
gy
re al
e st
at e
co m
m un
ica tio
n se
rv ice
s
m at
er ia
ls
co ns
um er
st ap
le s
ut ilit
ie s
0
100
200
300
400
500
600
(c) Distribution over Sectors for quarterly earning calls in our Data.

heading: D Dataset Size
text: Figure 6 shows that model performance as a function of dataset size converges quickly. We fine-tune a ClimateBERT model on different subsets of the training data, e.g. on 10%, on 20%, etc. In Figure 6, we find diminishing marginal utility after having fine-tuned a model on more than 60% of the dataset.
Hence, we believe that our dataset is sufficient in size and we do not expect model performance to increase drastically anymore if we were to annotate more data points.

heading: E Environmental Impact
text: In this section, following (Hershcovich et al., 2022) we describe the environmental impact of our dataset construction and experiments. All experiments were conducted on a carbon-neutral computing cluster in Switzerland, using a single Nvidia GeForce GTX 1080 Ti GPU with a TDP of 250 W.While the computing cluster we performed the experiments on is superficially carbon-neutral, there are still emissions for the production and shipping of the hardware used. Also, the energy used for our experiments could replace power produced by fossil fuel somewhere else. Therefore, we calculate emissions based on the country’s energy mix.
Running the main experiments took less than 1 hour combined. Detecting environmental claims in the quarterly earning calls took an additional 3 hours. For preliminary experiments, we trained a battery of transformer models on loosely annotated data (we used scores assigned by our "best" model to sample the sentences in the dataset). This took roughly 48 hours. Also, we embedded all sentences with SBERT for two additional hours. In total, we spent about 60 hours of computation time.

heading: F Funding
text: This paper has received funding from the Swiss National Science Foundation (SNSF) under the project (Grant Agreement No. 207800).
ACL 2023 Responsible NLP Checklist

heading: A For every submission:
text: 3 A1. Did you describe the limitations of your work?
in the dedicated "Limitations" section after the conclusion
3 A2. Did you discuss any potential risks of your work? in the dedicated "Ethics Statement" after the conclusion
3 A3. Do the abstract and introduction summarize the paper’s main claims? view the Abstract + Introduction
7 A4. Have you used AI writing assistants when working on this paper? Left blank.
B 3 Did you use or create scientific artifacts? in Section "3. Dataset"
3 B1. Did you cite the creators of artifacts you used? in Section "4. Experiments", we use existing datasets to train models which we evaluate in a zero-shot setting on our newly created dataset. We cite the authors of the artifacts invovled in this process appropriately.
7 B2. Did you discuss the license or terms for use and / or distribution of any artifacts? We will host our dataset and models upon publication on huggingface hub and github. We provide the license and terms for use and/or distribution of our artifacts on the huggingface hub and github, instead of mentioning this in the paper.
7 B3. Did you discuss if your use of existing artifact(s) was consistent with their intended use, provided that it was specified? For the artifacts you create, do you specify intended use and whether that is compatible with the original access conditions (in particular, derivatives of data accessed for research purposes should not be used outside of research contexts)? We used two existing artifacts – two datasets associated with a research paper, the first one containing claims, the second containing pledges. For both artifacts, we did not find an intended use in the paper. However, we assume that it is fine to use these artifacts for follow-up research (given the datasets are associated with a research paper and the datasets are freely accessible).
3 B4. Did you discuss the steps taken to check whether the data that was collected / used contains any information that names or uniquely identifies individual people or offensive content, and the steps taken to protect / anonymize it? in the dedicated "Ethics Statement" after the conclusion
3 B5. Did you provide documentation of the artifacts, e.g., coverage of domains, languages, and linguistic phenomena, demographic groups represented, etc.? in Appendix Section "D Data Sources"
3 B6. Did you report relevant statistics like the number of examples, details of train / test / dev splits, etc. for the data that you used / created? Even for commonly-used benchmark datasets, include the number of examples in train / validation / test splits, as these provide necessary context for a reader to understand experimental results. For example, small differences in accuracy on large test sets may be significant, while on small test sets they may not be. in Section "3. Dataset"
The Responsible NLP Checklist used at ACL 2023 is adopted from NAACL 2022, with the addition of a question on AI writing assistance.
C 3 Did you run computational experiments? in Section "4. Experiments"
3 C1. Did you report the number of parameters in the models used, the total computational budget (e.g., GPU hours), and computing infrastructure used? in Section "4. Experiments" and In Appendix Section "E Environmental Impact"
3 C2. Did you discuss the experimental setup, including hyperparameter search and best-found hyperparameter values? in Section "4. Experiments"
3 C3. Did you report descriptive statistics about your results (e.g., error bars around results, summary statistics from sets of experiments), and is it transparent whether you are reporting the max, mean, etc. or just a single run? in Section "4. Experiments"
C4. If you used existing packages (e.g., for preprocessing, for normalization, or for evaluation), did you report the implementation, model, and parameter settings used (e.g., NLTK, Spacy, ROUGE, etc.)? Not applicable. Left blank.
D 3 Did you use human annotators (e.g., crowdworkers) or research with human participants? in Section "3. Dataset"
3 D1. Did you report the full text of instructions given to participants, including e.g., screenshots, disclaimers of any risks to participants or annotators, etc.? in Appendix "B Annotation Guidelines"
3 D2. Did you report information about how you recruited (e.g., crowdsourcing platform, students) and paid participants, and discuss if such payment is adequate given the participants’ demographic (e.g., country of residence)? in Section "3. Dataset" and in the "Ethics Statement"
7 D3. Did you discuss whether and how consent was obtained from people whose data you’re using/curating? For example, if you collected data via crowdsourcing, did your instructions to crowdworkers explain how the data would be used? in the "Ethics Statement"
7 D4. Was the data collection protocol approved (or determined exempt) by an ethics review board? there was no need for an approval by an ethics review board for our data collection protocol
3 D5. Did you report the basic demographic and geographic characteristics of the annotator population that is the source of the data? in Section "3. Dataset"

heading: Future Work
text: We envision several directions for future work. First, we plan to investigate "greenwashing", the practice of making a false, vague, unclear, or mis-
leading environmental claim. To make progress on this front, it is mandatory that we can detect environmental claims in the first place. Second, models trained on detecting environmental claims have merits of their own, as previewed in our case study. We plan to explore more such applications in detail, e. g. , analyzing annual reports and TCFD9 reports at scale. For example, it would be interesting to see in which sections of TCFD reports firms make environmental claims. Lastly, we expect an increase of contributions at the intersection of environmental topics, climate change, and NLP in the near future. This work contributes to such efforts. Limitations
We find several limitations in this work. First, we acknowledge that the technical novelty of this work is limited: We introduce a sequence classification task, and we investigate rather standard models in our experiment section (i. e. , state-of-the-art transformer language models). Nevertheless, we believe that there is a gap in the literature for the task presented in this work, hence our introduction of the environmental claim detection task, the dataset, and models. Second, we collect data from sustainability reports, earning calls, and annual reports. However, this does not cover the universe of text where environmental claims are made, e. g. , company websites and product descriptions. Also, environmental claims can be made about environmental improvements on a wide range of topics such as carbon emissions, water pollution, and recycling, among others. We discussed creating different datasets, where each dataset is dedicated to one specific is-
9Task Force on Climate-Related Financial Disclosures
sue. However, we leave this to future work. Third, sometimes it is necessary to have access to more context to determine whether a sentence is an environmental claim. We discussed whether it would be beneficial to annotate whole paragraphs instead. However, the trade-off would be exploding annotation work and costs, hence our decision to introduce environmental claims as a sentence-level classification task (and we specifically asked annotators to reject ambiguous cases as environmental claims). Nevertheless, given a unlimited budget, we would have pursued annotating whole paragraphs instead (or annotating all environmental claims in a paragraph). Our data sources, e. g. , sustainability reports, are mostly published by European and US-listed companies, which is reflected in our dataset. We crawled these reports from the SEC10, hence our dataset contains mostly claims made by (a) big firms and (b) firms from developed countries. It is conceivable that smaller firms and firms from nondeveloped countries make different environmental claims, and models trained on our dataset might not be suitable to detect these claims