ClimateBERT-NetZero: Detecting and Assessing Net Zero and Reduction Targets
Authors: Tobias Schimanski
Julia Bingler
Mathias Kraus
Camilla Hyslop
Markus Leippold
Venue: EMNLP 2023 Main (2023)
Keywords: ClimateBERT
BERT
climate change
net zero
TL;DR: This study develops a dataset and methods to detect and analyse emission reduction and net zero targets in texts.

Abstract: Public and private actors struggle to assess the vast amounts of information about sustainability commitments made by various institutions. To address this problem, we create a novel tool for automatically detecting corporate and national net zero and reduction targets in three steps. First, we introduce an expert-annotated data set with 3.5K text samples. Second, we train and release ClimateBERT-NetZero, a natural language classifier to detect whether a text contains a net zero or reduction target. Third, we showcase its analysis potential with two use cases: We first demonstrate how ClimateBERT-NetZero can be combined with conventional question-answering (Q\&A) models to analyze the ambitions displayed in net zero and reduction targets. Furthermore, we employ the ClimateBERT-NetZero model on quarterly earning call transcripts and outline how communication patterns evolve over time. Our experiments demonstrate promising pathways for extracting and analyzing net zero and emission reduction targets at scale.

ClimateBERT-NetZerro: Detecting and Assessing Net Zero and Reduction Targets
Tobias Schimanski Julia Bingler Camilla Hyslop24 Mathias Kraus² Markus Leippold
'University of Zurich UUniversity of Oxford Counciil on Economic Policies
*Net Zero Tracker 'FAU Erlangen-Nürnberg 'Swiss Finance Institute (SFI)
tobias. schimankki@ff wwwponsssifffuu.......m wzw. ch
Abstract
the extraction of information about net zero and reduction targets.
Public and private actors struggle assess the vast amounts of information about sustainabil- ity commitments made by various institutions. To address this problem, create novel tool for automatically detecting corporate, national, and regional zero and reduction targets in three steps. First, we introduce expert- annotated data set with 3.5K text samples. Second, train and release ClimateBERT- NetZero, natural language classifier to detect whether text contains net zero or reduction target. Third, showcase its analysis poten- tial with two use cases: We first demonstrate how ClimateBEKT-NefZaa with conventional question-answeriin (Q&A) models to analyze the ambitions displayed in net zero and reduction targets. Furthermore. we employ the HlimatBBERT-NeIZee model on quarterly earning call transcripts and outline how communication patterns evolve over time. Our experiments demonstrate promising path- ways for extracting and analyzing net zero and emission reduction targets at scale.
Contribution. As a remedy, this study deliv- ers a threefold contribution at the intersection of climate change and NLP by creating tool that automatically extracts and assesses net zero and reduction target information from various sources. Our first contribution is introducing expert-annotated data with 3.5K text samples, which builds on the Net Zero Tracker project (Lang et al., 2023). Second, we develop and publish ClimateBERT-NetZerro (based on Climate- BERT (Webersinke et al., 2022)), a comparatively lightweight and, therefore, less NLP model that can effectively classify net zero and reduction targets in text samples.' Third, we provide two real-world use cases: We demonstrate how extend the model capabilities and analyze the ambitions of the net 7ero and reduction tar- gets. Finally, we utilize ClimateBERT-Neteerro to analyze the appearance of net zero and reduction claims in companies' carning calls transcripts from 2003 to 2022.
can be combined
1 Introduction
Results. In this context, we show that CiimateBERT-NeeZerro achieves better predictive performance than larger models. Moreover, we demonstrate that we can effectively detect the tar- get year for net zero claims; as well as the target year, bascline year, and reduction targets in percent for general emission reduction claims. Finally, we provide evidence for the rise in target communi- cations by analyzing communicated net zero and reduction targets in companies' carning calls. Implications. The implications of our study are significant for both practice and research. Ana- lyzing net zero and reduction targets represents a crucial step in understanding and acting on transi- tion plans. Since the tool can operate on a large scale, we lower the barriers to data accessibility.
Motivation. "Zero tolerance for net zero green- washing" is the central aim of UN General Secre- tary Antonio Guterres at the GOP27 (UN, 2022). To limit global warming to 1.5%C, massive ef- forts in emission reduction are necessary (Klaassen and Steffen, 2023). Consequently, an increasing amount of net zero and emission reduction tar- gets are introduced, communicated, or updated by private and public institutions (Höhne et al., 2021). With the help of Natural Language Process- ing (NLP) methods, it is possible to automatically assess large chunks of textual data and gain struc- tured insights about climate-related information (e.g.. Webersinke al., 2022; Bingler et al., 2022a; Stammbach al., 2022). However, none of the previous works have studied the arguably most rel- evant and best tractable objective for institutions
Model and data are available on https://hugiingfaae co/cl wwe.climmtbbertt wtw.eebrrreeeeeeeeeeeettttttteie

---

Acknowledging that further research is needed to hold companies responsible for their commitments and analyze potential greenwashing patterns, our approach presents an important first step towards "zero tolerance for net zero greenwashing".
has up 175 billion parameters (OpenAI, 2023). Hence, climate-aware decisions for model training and deployment must be carefully considered. Availability of Climate Change Data. In light of continuously increasing societal and regulatory pressure for firms to disclose their climate change activities, data availability will drastically increase in the upcoming years. For instance, the EU re- quires companies to rigorously disclose risks and opportunities arising from environmental and so- cial issues from 2025 on (EU, 2023). An important role is attributed to credible and feasible transi- tion plans aligning with the ambition to restrict global warming to 1.5 degrees (Höhne et al., 2021; Fankhauser et al., 2022). Thus, assessing firms' re- duction or net zero commitments plays an essential role. To this end, the present study introduces a data set and methods to automatically detect and assess reduction and net zero statements by private and public institutions.
2 Background
Large Language Models. In previous years, Large Language Models (LLMs) continuously gained im- portance. LLMs have proved to possess human- like abilities in a variety of domains (Radford et al., 2019; Brown et al., 2020; Ouyang et al., 2022). These models are primarily based on subtypes of the transformer architecture and are trained on mas- sive amounts of data to learn patterns within the training data (Vaswani et al., 2017). As result of the training process, the models can solve mul- titude of downstream tasks. One particular down- stream task represents text classification. For these tasks, commonly used models are based on the Bidirectional Encoder Representations from Trans- formers (BERT) architecture (Devlin et al., 2019). NLP in the Climate Change Domain. Our ongoing effort to employ LLMs support climate action by developing tar- geted datasets and models. Previously released work includes the verification of environmental claims (Stammbach et al., 2022), climate change topic detection (Varini et al., 2021), and the analy- sis of climate risks in regulatory disclosures (Köl- bel et al., 2022). A particular way to improve the performance of models on downstream tasks is the creation of domain-specific models, such as Cli- mateBERT. With further pre-training on climate- related text samples, this model outperforms its peers trained on general domain language (We- bersinke et al., 2022). ClimateBERT has already been applied in recent studies that analyze corpo- rate disclosures to investigate patterns around com- panies' cheap talk (Bingler et al., 2022a.b).
3 Data
The raw data for this study originates from col- laboration with the Net Zero Tracker project (Lang et al., 2023). The Net Zero Tracker assesses tar- gets for reduction and net zero emissions or similar aims (e.g., zero carbon, climate neutral, net neg- ative). Reduction targets are claims that refer to an absolute or relative reduction of emissions, often accompanied by a baseline year to which the reduc- tion target is compared. Net zero targets represent a special case of reduction targets where an institu- tion states to bring its emissions balance down to additional net emissions by certain year. The Net Zero Tracker assesses claims of over 1,500 countries, regions, cities, or companies. For this project, we created an expert-annotated 3.5K data set. The data set differentiates between three classes: net zero, reduction, and no targets. For the reduction and net zero targets, we fur- ther process the claims gathered by the Net Zero Tracker. If necessary, we clean and hand-check the text samples and correct the pre-assigned labels by the Net Zero Tracker. Moreover, we enhance the data with a variety of non-target text samples. All of these samples originate from annotated and re- viewed datasets of previous projects in the climate domain (Stammbach et al., 2022; Webersinke et al.,
work contributes
Apart from model-centric considerations and in- novations, previous research and public attention have also shifted to the climate impact of training and deploying LLMs (Stokel- Wallker and Noorden, 2023; Hao, 2019). With the ongoing development of new models, the models' number of parameters steadily rises. Consequently, this increases electric- ity requirements and, therefore, the environmental footprint of training and using the models. For in- stance, while ClimateBERT possesses 82 million parameters (Webersinke et al., 2022), ChatGPT
Collectivelyy, the dataset contains 273 claims by citics. 1396 claims by companies, 205 claims by countries, and 159 claims by regions.

---

Net Zero
Reduction No Target
Model
Model size
Acc (std.)
990
1005
1522
ClimateBERT
82 M
0.966 (.004)
DistilRoBERIa
82 M
0.959 (.007)
Table 1: Distribution of the labels in the text samples
RoBERTa-base
125 M
0.963 (.006)
GPT-3.5-turbo
175 B
0.938
2022) and are again hand-checked by the authors (for more details, Appendix A). Ultimately, construct dataset with 3.5K text samples (see Ta- ble 1).
Table 2: Accuracy and model size of tested models
ClimateBERT-NetZero also outperforms GPT-3.5- turbo on this task.
4 ClimateBERTT-NtZeeww
To further evaluate the performance of the ClimateBERT-NeeZeroo a real-world setting, we conduct a hand-check on the output of the model on 300 sentences sampled from sustainability re- ports. We find that the model classifies 98% of the sentences right. This solidifies the robustness of the approach (see Appendix E for more details). Since net zero targets are special case of re- duction targets, we also create model called ClimateBERT-Reducee
We use the data set to train and test the capabilities of several classifier models. We particularly want focus training model that is both effective in performance and efficient in the use of resources. Thus, as a base model for fine-tuning, we utilize Cli- mateBERT (Webersinke et al., 2022). This model is based on DistilRoBERTa which is a smaller and thus less resource-demanding version of RoBERTa (Sanh et al., 2020). ClimateBERT is further pre- trained on a large corpus of climate-related lan- guage and thus outperforms conventional models a variety of downstream tasks (Webersinke et al.. 2022). We further fine-tune ClimateBERT clas- sify whether a text contains a net zero target, a reduction target, or none of them. We christen this model ClimateBERT-NetZaaee sults into context, we rerun the fine-tuning with its base architecture BEETia well as with the larger model RoBERTa (Liu et al., 2019). Fur- thermore, also test how generative, much more resource-intensive, but also more general models perform in comparison to ClimateBERT-NetZooe To create this comparison, we prompt GPT-3.5- turbo to classify the texts (see Appendix for the detailed prompt).
This model simply dif- ferentiates between general reduction targets (in- cluding net zero targets) and no targets. We re- peat the previous setup to train and test the model. ClimateBERRT-Recuctioo consistently reaches per- formance measures of ca. 98% (for more details, see Appendix F).
To set the re-
Use Cases
To showcase further capabilities of ClimateBERT- NetZero, we develop two potential use cases, First, we showcase how the model can be used perform a deeper analysis net zero and reduction targets by analyzing the underlying ambitions of the state- ments. Second, we demonstrate how ClimateBERT- NetZero can detect patterns in company communi- cation by analyzing carning call transcripts.
To evaluate the performance of the BERT-family models, we five-fold cross-validation and con- sult the performance measures accuracy, Fl-score, precision, and recall (see Appendix C for all ex- plored variations of hyperparameters and Appendix for full results and description of the validation set). As Table shows, ClimateBERTT-NeZeerr slightly outperforms all other BERT-famil mod- els. With a consistent accuracy of over 96%, ClimateBERT-NetZerro is able to distinguish be- tween the different types of targets effectively. Fur- thermore, we evaluate the performance of 5PT-3.5- turbo by prompting it to classify all texts, trans- form the output to numeric labels, and calculate the performance metrics. As Table 2 indicates,
Measure Ambitions with Q&A Models. In this use case, we study methods assess the ambitions of institutions further. For net zero claims, the pri- mary measure ambition is given by the year the institution wants to achieve net zero. For reduction claims, we use the target year of the reduction, the percentage of how much the institution wants to de- crease its emissions, and the baseline year against which the reduction is measured.
For evaluating the ambitions in the net zero and reduction claims, we employ the Question Answer- ing (Q&A) model et al., 2016). This model delivers concise answers
(Rajpurkar
'Model and data are available on https://huggingfant co/cl imatebert wwwteeeerereuuttiiiii

---

questions posed sample. Therefore, develop questions extract the relevant informa- tion for each dimension of ambition in the net zero and reduction claims (see Appendix G for detailed questions).
Accuracy
Task
Data
(% of data)
net zero target year
raw/optimal
0.95 (1.0)
confidence-tumee
0.97 (0.5)
raw
0.84 (1.0)
reduction
To evaluate this approach, again use the data compiled by the Net Zero Tracker project (Lang al., 2023). The data set contains cuman-exxtactedd data for each of the researched dimensions of am- bitions. To test the ambitions, we preprocess and hand-check 750 text samples.
optimal
0.90 (0.97)
target year
confidence-tuned
0.90 (0.8)
raw
0.68 (1.0)
reduction base year
optimal
0.80 (0.84)
confidence-tunee
0.82 (0.62)
raw
0.88 (1.0)
reduction
Finally, we rum the Q&A model on the text sam- ples. The model typically outputs a single number. We assess the accuracy of the model by checking how many answers contain the correct number. Ta- ble 3 shows the results each dimension in three steps. First, we evaluate the question on the raw data set. For approximately 95% of the net zero targets, the model can detect by which year net zero is aimed to be achieved. In the second step, we reduce the data cluded if the text actually contains the information we search for. This gives the upper bound accuracy the model. For instance, this reduces the data set for the baseline year detection of reduction targets to 84% while increasing the accuracy from 67% to 80%. This underlines the potential for improving Q&A results with more fine-grained data cleaning. In the third step, again use the raw data and improve the performance by using the confidence of the Q&A model. Optimizing the confidence rep- resents a trade-off between boosting the accuracy and preserving an amount of data where these con- fident decisions can be taken (see Appendix for more details). Table reports the results using confidence of 0.3. Increasing the model confidence consistently improves the accuracy.
optimal
0.91 (0.96)
percentage
confidence-tuned
0.93 (0.35)
Table 3: Accuracy of the Q& approach in assessing
the quantitative properties of the net zero and reduction targets
they seem to be discussed throughout the whole period. These results underline both the rising im- portance of the topic in recent times well as the value of differentiating between net zero and duction targets (for further details, see Appendix I).
that a target text is only in-
0.6
D30
508
2000
2009
Figure 1: Communication of net zero and reduction target sentences percentage of all sentences over time
Detection Net Zero and Reduction Targets in Earning Call Transeripts. In this use case, we em- ploy ClimateBERT-NetZZern on transcripts of quar- terly earning calls from publicly-listed US compa- nies from 2003-2022 to demonstrate its large-scale analysis potential. We create yearly index by building an average of all quarters of year. As Figure 1 indicates, the net zero target discussion has experienced a sharp increase since 2019.4 While the same observation true for reduction targets,
Conclusion
In conclusion, this paper demonstrates the develop- ment and exemplary employment of ClimateBERT- NetZero, a model that automatically detects net zero and reduction targets in textual data. We show that the model can effectively classify texts and even outperforms larger, more energy-consuming architectures. We further demonstrate a more fine- grained analysis method by assessing the ambitions of the targets as well as demonstrating the large- scale analysis potentials by classifying earning call
'This observation largely coincides with the general large increase in sustainable invest- ments in 2019-2022 (sec. for instance https: //www. morningstar www.iinnstar.oo/////// hom/sustaanaall wecllle-iveettin investing/ 2022-us-sun www.usssssssnaalll.eee.e fundd------d--- landscape- 5-charts)

---

Julian Kölbel, Markus Leippold, Jordy Rillaerts, and transcripts. By releasing the dataset and model, we deliver an important contribution to the intersection of climate change and NLP research. Identifying and analyzing reduction and net zero targets repre- sent an important first step to controlling set com- mitments and further comparing them with other real-world indicators of corporate, national, or re- gional actors. Especially in the realm of greenwash- ing, the abundance or failure to achieve set targets while benefitting from a differently communicated public image are key factors to investigate.
learners. Advances neural information processing systems, 33:1877-1901.
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. Bert: Pre-training of deep bidirectional transformers for language understand- ing. arXiv preprint 071.v:1810.04805
EU. 2023. Corporate sustainability reporting.
Sam Fankhauser, Stephen M. Smith, Myles Allen, Kaya Axelsson, Thomas Hale, Cameron Hepburn. J. Michael Kendall, Radhika Khosla, Javier Lezaun, Eli Mitchell-Larson, Michael Obersteiner, Lavanya Rajamani, Rosalind Rickaby, Nathalie Seddon, and Thom Wetzer. 2022. The meaning of net zero and how to get right. Nature Climate Change, 12:15- 21.
7 Limitations
Like all studies, ours also has limitations. First, the dataset size is limited to 3.5K samples. While this could entail lower generalizability, we show that the model performance is already at a very high level. The data size limitation also applies to the first use case, where further data could increase the validity. Second, this study predominately focuses on BERT-family models. In a strive to employ ac- cessible models of smaller size, we do not employ larger, more recent architectures such as LLaMa (Touvron et al., 2023). While the impact on the classification task might be limited, as shown with GPT-3.5-turbo, especially for detecting ambitions, instruction-finetune models could improve the re- sults.
Karen Hao. 2019, Training single ai model can emit as much carbon as five cars in their lifetimes. MIT Technology Review, 75:103.
Niklas Höhne, Matthew Gidden, Michel den Elzen, Frederic Hans, Claire Fyson, Andreas Geiges. Louise Jeffery, Sofia Gonzales- Zuñiga, Silke Mooldijk, William Hare, and Joeri Rogelj. 2021. Wave of net emission targets opens window meeting the paris agreement. Nature Climate Change, 11:820- 822.
Lena Klaassen and Bjarn Steffen. 2023. Meta-analysis on necessary investment shifts to reach net zero path- ways in Europe. Nature Climate Change, 13:58-66.
Acknowledgements
Qian Wang. 2022. Ask BERT: How Regulatory Dis- closure of Transition and Physical Climate Risks Af- feets the CDS Term Structure*. Journal of Financial Econometrics.
This paper has received funding from the Swiss National Science Foundation (SNSF) under the project 'How sustainable is sustainable finance? Impact evaluation and automated greenwashing de- tection' (Grant Agreement No. (00018_207800)
John Lang, Camilla Hyslop, Zhi Yi Yeo, Richard Black, Thomas Hale Peter Chalkley, Frederic Hans, Nick Hay, Niklas Höhne, Angel Hsu, Takeshi Kuramochi, Silke Mooldijk, and Steve Smith. 2023. Net zero tracker. Energy and Climate Intelligence Unit, Data- Driven Envirolab,, New Institute, Oxford Net Zero.
References
Julia Anna Bingler, Mathias Kraus, Markus Leippold, and Nicolas Webersinke. 2022a. Cheap talk and cherry-picking: What ClimateBert has to say cor- porate climate risk disclosures. Finance Research Letters, page 102776.
Yinhan Liu, Myle Oit, Naman Goyal, Jingfei Du, Man- dar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019. Roberta: A robustly optimized bert pretraining ap- proach. arXiv preprint 1907
Julia Anna Bingler, Mathias Kraus, Markus Leippold, and Nicolas Wehersinke. 2022b. Cheap talk in cor- porate climate commitments: The role of active insti- tutional ownership, signaling, materiality, and senti- ment. Swiss Finance Institute Research Paper.
OpenAL. 2023. Instructgpt: Ai for generating instruc- tions. htw.:ppppe....ooo//eccccacnnnnna
Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. 2022. Training language models to follow instruc- tions with human feedback. Advances in Neural Information Processing Systems, 35:27730-2774
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. 2020. Language models are few-shot

---

Alec Radford, Jeff Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. 2019. Language models are unsupervised multitask learners. OpenAl Blog.
project tracks the commitments of over 1,500 com- panies, countries, regions, and cities. 5 We use their raw data and process it in three steps.
First, we transform the raw data in several ways. The Net Zero Tracker follows the general classi- fication of reduction and net zero targets but uses more fine-granular classification. For instance. net-zero targets could be zero carbon, climate neu- tral, net negative. We aggregate all sub-labels into their two main categories. Furthermore, we preprocess the corresponding texts. We remove sentences shorter than five words, URLs, and other special characters. However, these cleaning steps only result in marginal changes in the data charac- teristics, reducing the overall dataset by 97 samples (see Table A.1).
Pranav Rajpurkar. Jian Zhang, Konstantin Lopyrev, and Percy Liang. 2016. SQuAD: 100,000+ questions for machine comprehension of text. In Proceedings of the 2016 Conference on Empirical Methods in Natu- ral Language Processing, pages 2383-2392 Austin,
Texas. Association for Computational Linguisties.
Victor Sanh, Lysandre Debut, Julien Chaumond, and Thomas Wolf. 2020. Distilbert, a distilled version of bert: smaller, faster, cheaper and lighter. arXiv preprint 1rXiv:1910.011000
Dominik Stammbach, Nicolas Webersinke, Julia Anna Bingler, Mathias Kraus, and Markus Leippold. 2022. A dataset for detecting real-world environmental claims. Xrviv preprint arXiv:2209.005000
Second, hand-check the labels of the remaining text samples. Since the texts are already assigned with labels gathered by expert annotators of the Net Zero Tracker project. o The aim of this step is to increase validity by hand- handcchekiing the labels. This step creates an inter-labeler agreement (see Table A.2). The results show high inter-labeler agreement with Cohen's 's Kappa of 93%. We correct labels if necessary remove text samples that don't correspond to their labels or are meaningless. Each text sample is hand-checked by at least one author. If the label is found to be differrnt from the expert annotators of the Net Zero Tracker, the decisions are marked and decided upon by the team. Thus, we create two labels: First, the labels assigned by our data collaborators of the Net Zero Tracker, and second, the labels assigned by the author team. For unified decision-making process, the following labeling guidelines are used:
Chris Stokel- Walker and Richard Van Noorden. 2023. What chatgpt and generative ai mean for science. Nature, 61 61:2114-216.
Hugo Touvron, Thibaut Lavril. Gautier Izacard, Xavier Martinet, Marie-Aune Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, al. 2023. Llama: Open and effi- cient foundation language models. arXiv preprint arXiv:2302.FF 13971.
UN. 2022. Cop27:
'zero tolerance for greenwashing', guterres says as new report cracks down on empty net-zero pledges.
Francesco S. Varini, Jordan Boyd-Graber, Massimiliano Ciaramita, and Markus Leippold. 2021. Climatext: A dataset for climate change topic detection. arXiv preprint arXiv:2012.0048:
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017. Attention all you need. In Advances in Neural Information Pro- cessing Systems, pages
Reduction targets are claims that refer to an absolute or relative reduction of emissions, often accompanied by a baseline year to which the reduction target is compared.
Nicolas Webersinke, Mathias Kraus, Julia Bingler, and Markus Leippold. 2022. Climatebert: A pretrained language model for climate-related text. In Proceed- ings of AAAI 2022 Fall Symposium: The Role Af Al in Responding to Climate Challenges.
Net zero targets represent special case of reduction targets where an institution states bring its emissions balance down to no additional net emissions by a certain year.
Appendix
A Details on Training Data Creation
If both targets appear in the text, the main focus of
Further information can be found on Net Zero Tracker website https:///eeotrrackr net/.
The training data consists of three parts: reduction targets, net-zero targets, and non-target texts.
"The Net Zero Tracker a combined initiative of the Uni- versity of Oxford Net Zero, the New Climate Institute, Data Driven Envirolah, and the Energy & Climate Intelligence Unit. Thus, the annotators we collaborate with in this project are domain experts.
Reduction and Net-Zero Targets. For reduc- tion and net-zero targets, we collaborate with the Net Zero Tracker project (Lang et al., 2023). The

---

the text is decisive. For instance, most reduction targets serve as intermediary steps for the fi- nal goal of net zero. Thus, the focus lies on net zero.
Your task is to classify a previded text whether it contains claims about Reduction Net Zero targets or none
The following examples provide an impression of reduction and net zero target:
Radustion targets clains that refer
absolute or relative reduction of anissions, aften acconpanied the reduction target is compared.
baseline
to which
Net zaro targets represent special case of institution states to bring its enissions balance down to no additianal
Reduction target:
reduce our abselute Scope and 2 GHG erissions fron non- loconotive operations (including enissions assoctated with our buildings and facilities) by 27.5% 2030."
reduction targets
emissions by If bath targets appear in the text, the main focus of the text is decisive. For instance, reduction targets serve internediary steps for the final goal of net zero. Thus, the focus net zero.
year.
Net zero target: "Me are connitted to net-zero energy across our businesses and achieved 100% carbon neutrality by purchasing carbon offsats with Renewable Energy Certificates (REC).'
operations
the provided text, please respond with "Reduction for reduction targets, 'Net Zero for Net Zero targets 'None' if no category applias.
Third, we use human-in-the-loop validation process to enhance the validity of the data further. We use rre-folddrrossvalldation with the baseline ClimateBERT model on the entirety of the dataset and save the missclassifiaatinn of each fold. This gives us the opportunity to train the model with the data and, at the same time, store the wrongly identified samples within the training process. We then involve human annotator to check the mis- classifications and readjust labels if necessary. This way, the machine identify edge cases and the human interfere if needed. We repeat this human-in -n-te--oop validation three times.
Provided text: ""[text)**> ...
C Hyperparamters for Fine-Tuning
Table C.3 displays the base case of the configured hyperparameters for fine-tuning ClimateBERT- NetZero, DistilRoBERTa and RoBERTa-base. The performance metrics created using five-fold crossvalidation.
Furthermore, Table I.9 systematically explores the variation of the hyperparameters, such as the learning rate, batch size, and epochs. The results show that ClimateBERT consistently outperforms its peers. However, it also shows that the choice of the hyperparameters only marginally affects the final outcome accuracy.
Non-Target Samples. For non-target text samples, we use publicly available datasets in the climate domain (Stammbach et al., 2022; Webersinke al., 2022). We aim to increase the gemeralizability of the models by creating well-balanced dataset. Including climate-related samples that are not targets is crucial for the models learn the specific differentiation between targets and general sentences in the climate domain. Furthermore, the datasets enable us also to include generie, non-climate-related sentences. We follow the same procedure de- scribed above to improve the validity of the dataset.
D Full Performance Metrics of All Models
For the full performance metrics of all models with the base hyperparameter setting, see Table I.8. Cli- mateBERT consistently outperforms all other mod- els for the performance metrics accuracy, Fl-score, precision, and recall.
Furthermore, the high F1, precision, and recall values suggest that all three labels are consistently predicted well with the models. Figure I.1 solidifies this notion by displaying the confusion Matrix for the cross validatiio results of the ClimateBERT model, taking the predictions of each validation set at the end of a fold. All three labels are consistently predicted on very satisfactory level. The most confusion is visible between reduction and net zero targets, which are naturally semantically close. In particular, outstanding performance is achieved on the Non- -argea label, which is very important
Ultimately, we create dataset with 3.5K text samples. Table A.l gives an overview of the de- scriptive statistics of the text length.
B GPT-3.5-turbo Prompt
To create comparison between BERT-family and larger instruction-finetune models, also use GPT-3.5-turbo classify all texts. We use the following prompt for zero-shot classifying the dataset to contain a reduction, net zero, or no target at all:

---

Processing count
mean
std
min
max
25%
75%
before
3614
38.5
59.0
1
1057
16
40
after
3517
39.5
59.5
5
1057
17
40
Table A.1: Characteristics of the label data text length before and after the processing steps
Labler Agreement Cohen's 's Kappa
sentences (237). We read every sentence and assign a label to it. The evaluation is largely in line with the overall evaluation metries. The F1 scores are 98% for net zero, 90% for reduction, and 99% for non-target. The comparatively lower value for reduction results from some confusion between the reduction of waste or water and the reduction of emissions, ie., a confusion between reduction and no target. Further details can be scen the corresponding confusion matrix in Figure 1.2.
0.952
0.931
Table A.2: Statistics on labeler agreement
Hyperparameter
Value
Epochs
10
Batch Size
32
Gradient
2
Accumulation
Warmup ratio
0.1
Learning rate
5c-5
F Performance Metries for CiimatBEERT-Reducttoo
Patience
5
Table C.3: Hyperparamters for training the models with five-fold cross-validation
Table F.5 displays the evaluation metrics for the ClimateBERT-RReuuction model that defines net zero and reduction targets as a combined class. With consistent measures over 98%, the model can successfully differentiate between the classes
for real-world applications where this label is likely the overwhelming majority class.
For overview of the validation data used in the five-fold cross-vall voos--valdation,, Table D.4. Since stratified split, the distribution in each fold is nearly equal.
Metric
Score
Parameters
82 million
Accuracy (std.)
0.985 (0.005)
FI-Score (std.)
0.987 (0.005)
Fold
Non-Tane Target
Reduction
Net-Zero
Precision (std.)
0.990 (0.004)
0
305
201
198
Recall (std.)
0.988 (0.005)
1
305
201
198
Table F.5:
Performance metrics of ClimateBERT-
2 304
201
198
Reduction
3
304
201
198
4
304
201
198
Table D.4: Label distribution in the validation sets of the five-fold cross-validation
G Questions for Evaluating the Ambition of Targets
These questions are posed to the text samples to evaluate the ambition of the net zero and reduction targets.
E Hand-Evaluatton of Sustainability Reports
In this test, we hand-evaluate 300 sentences of 2022 sustainability reports. We take the sustainability reports of Pfizer, JP Morgan, and Apple (to have varicty of data sources), classify the entirety with ClimateBEERT-clmaaee and ClimateBERTT-NeZZeen. and sample 300 sentences for hand-checking. In this test set, we include all sentences labeled as first climate and second net zero or reduction target (63) and fill up the rest the test set with random
For detecting the target year of net zero tar- gets:
When does the organizstion want to achieva zaro?'
For reduction targets:
For detecting the target year:

---

seems crucial to first identify climate-related sen- tences. Otherwise, the chance of false positives might be high. ClimateBERTT-eeteerro might pick up general reduction targets that are semantically similar to non-climate reduction targets. Finally, we calculate the mean for every ECC and aggregate it per year.
which does the organization want to reduce its enissions?"
For detecting the baseline year:
'What is the baseline year or level target to reduction target is compared
The results are displayed in Figure 1. As the Figure indicates, the general discussion of reduc- tion targets occurs throughout the sample period. even reached its peak around 2010 and has ex- perienced a stcady decrease since then. Net zero targets are rarely discussed the earlier sample pe- riods. However, in 2019, both the reduction and net zero target discussion increased drastically. This likely originates from the general mainstreaming of climate issues in recent years. However, this analysis represents preliminary step and should inspire other rescarchers use the models and find patterns. demonstrates the large-scale analysis capabilities of the ClimateBERT-eerie
For detecting the relative or absolute re- duction target:
What is the reduction target of the organization terns?
These questions posed to the Q&A model Koberta-base-squee (Rajpurkar al., 2016).
Model Accuracy and Dataset Size as Function of Model Confidence
The graphs in H.4 display the model accuracy and the corresponding remaining fraction the dataset for given confidence level of a model. The model accuracy predominately increases when the model confidence increases. At the same time, the size of the dataset is reduced because there are fewer samples for which the model confident. For instance, for confidence of 0.3, the accuracy of detecting the baseline year of the reduction target rises from 67% to 82% while still analyzing 62% of the data set.
I Details on the Earning Call Transcript Evaluation
The base data for the evaluation are earning con- ference call (ECC) transcripts originating from Re- finitiv. We use all available ECC data for pub- licly listed firms in the US from 2003-2022. The raw documents are split into single sentences. We assess a total of 17825 earning call events. The summary statistics of the number sentences per event can be seen in Table 16.. After creating the sentence-level data, we use two-step process to identify net zero and reduction targets. We first clas- sify whether the sentences climate-related or not with the ClimateBERT climate detector model.? Then, we use ClimateBERT-NeeZeero climate-related sentences as either net zero, reduc- tion, or no target. Especially for the reduction class,
classify all
"See https:/ / whwhhhhgggggaaccc acee. co/coo//ccc///o imatebert/ distilroberten base waw.ttteeaaeaaaaaeeeee.. -wweeeteeeeeeeeooorr detector for the model.

---

nome
1506
Nuv labe
reduction
net-2ero
nona
reduction Predicted label
net zeno
Figure 111: Confusion Matrix for the cross-validation of the ClimateBERT-NeZZon model extracting the results on the validation set for each fold
200
232
150
True label
reduction
32
50
none
reduction Predicted label
net-zero
Figure 1.2: Confusion Matrix for the hand-checking of the classification results of ClimatcBERT-NetZzrroo on 2022 sustainability reports from Pfizer, Morgan and Apple

---

Sentences per Event
# events
mean
std
min
max
25
50%
75 %
17825
329
149
5
1799
217
323
428
Table 1.6: Summary statistics of the number of sentences in an carning call event
04
38-
0.
34
0.2
Rachaa the delace!
Pactas dabaset
Accurscy
Accuracy
1.2
the
3.-
3.5
Accunacy
Medol
confidincce
Table 1.7: Top left: Accuracy of the net zero target year detection as a function of the model confidence, Top right: Accuracy of the baseline year detection a function of the model confidence, Bottom left: Accuracy of the reduction target year detection as function of the model confidence, Bottom right: Accuracy of the relative or absolute target detection as function of the model confidence
Model
Model size
Accuracy (std.)
F1 -Score (std.)
Precision (std.)
Recall (std.)
ClimateBERT
82 million
0.966 (.004)
0,962 (.004)
0,962 (.004)
0.963 (.004)
BoERil
82 million
0.949 (.007)
0.944 (.005)
0.955 (.007)
0.946 (.01)
RoBERTa-base
125 million
0.963 (.006)
0.958 (.006)
0.959 (.007)
0.958 (.006)
GPT-3.5-turbo
175 billion
0.938
0.920
0.932
0.901
Table L88 Performance metrics of tested models

---

Hyperparamters
Model Accuracy (std.)
Learning rate
Epochs
Batch Size
ClimateBERT
DosRil
RoBERTa-base
3c-05
16
0.965 (.005)
0.963 (.005)
0.965 (.005)
3e-05
5
32
0.971 (.005)
0.963 (.005)
0.965 (.005)
3e-05
10
16
0.963 (.005)
0.958 (.005)
0.956 (.003)
3e-05
10
32
0.968 (.002)
0.961 (.003)
0.964 (.003)
5c-05
5
16
0.965 (.003)
0.963 (.007)
0.962 (.004)
5e-05
5
32
0.967 (.006)
0.963 (.005)
0.964 (.005)
5e-05
10
16
0.963 (.002)
0.957 (.005)
0.961 (.006)
5c-05
10
32
0.966 (.004)
0.959 (.007)
0.963 (.006)
7e-05
16
0.966 (.003)
0.962 (.006)
0.962 (.007)
7e-05
5
32
0.966 (.004)
0.960 (.005)
0.966 (.007)
7e-05
10
16
0.961 (.005)
0.962 (.006)
0.959 (.006)
7c-05
10
32
0.965 (.001)
0.963 (.002)
0.964 (.005)
Table 1.9: Performance metrics of the models for grid of varying hyperparameters learning rate, batch size and
epochs


Reviews:
[{"review_id": "Z1dbnPjiXa", "reviewer": "Reviewer_Qtwv", "summary": "", "questions": "[Question A]: To my understanding, ClimateBERT-NetZero is a classification model right? How do you apply it to the task of ambition prediction in a Q&A fashion?", "limitations": "", "rating": 2, "confidence": 3, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Hello dear Reviewer,\n\nthank you for your feedback. Indeed, ClimateBERT-NetZero \"just\" serves the task of identifying reduction and net zero claims. However, we argue that this task and dataset represent an important contribution because they address an important real-world problem with much-needed nice language, i.e. climate change disclosure language. In an abundance of specific datasets in the realm of climate change, corresponding challenges cannot be solved. Specifically, adaptation and transition disclosures (i.e., net zero and reduction ambitions) will play an essential role in the real-world economy and in fighting climate change. Thus, we build on existing work in this domain (e.g. ClimateBERT) to create the best possible domain-specific solution.\n\nIn acknowledgment of your comments, we created new metrics and information for the data creation process that would be included in the final version of the paper. I will briefly outline three central components that address your points about data quality and transparency. We add a more detailed view of the data quality and make the data creation process more transparent by including details on the dataset creation. One label was assigned by the Net Zero Tracker and controlled by the author team to reach an inter-labeler agreement. This agreement is very high with a Cohen's Kappa of 93%. When disagreements arose, they were discussed with the team. Second, more details on the additional human-in-the-loop control process could mitigate concerns about data quality problems. One central element that is further described is that we used the five-fold cross-validation with ClimateBERT and saved all misclassifications of each fold. Then, the author team discussed them and thereby refined the dataset. This way, the dataset ran through multiple human and machine checks. Third, we included additional multiple statistics about text length before and after preprocessing, the label distributions during training and validation as well as samples of the dataset in the paper to address your concerns about the opacity of how the dataset looks.\n\nRegarding your question (Question A): we don't use ClimateBERT-NetZero for the QA approach. As described in the paper, we use Roberta-base-squad2 and create the evaluation dataset for the questions. \n\nWe hope these comments clarify some concerns about the paper.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. The motivation of this paper seems clear and sound.\n2. The empirical results also give evidence of the effectiveness of the proposal of this paper.", "reasons_to_reject": "1. The data creation process lacks details. This paper put more details of the training data creation in the appendix, which is fine. But it never mentions anything about quality control (such as IAA), how samples in the dataset looks like.\n2. This work lacks novelty. In my opinion, it just looks like an off-the-shelf BERT-based language model tuned on domain-specific data without any modification to make the model better fit this certain domain or task.", "questions_to_authors": "[Question A]: To my understanding, ClimateBERT-NetZero is a classification model right? How do you apply it to the task of ambition prediction in a Q&A fashion?", "paper_topic_and_main_contributions": "In order to detect and assess the net zero and reduction targets, this paper introduces a classification model named ClimateBERT-NetZero, with ClimateBERT as the base model. This paper also introduces a dataset containing 3.5k text samples labeled by Net-Zero/Reduction/No Target as the dataset ClimateBERT finetuned on. Empirical results of this model applied on two subsequent tasks show the evidence of the effectiveness of the proposal of this paper.", "excitement": 2, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Missing_References": "N/A", "Typos_Grammar_Style_and_Presentation_Improvements": "line 165: 1.500 → 1500"}, "extra_text": {}}, {"review_id": "9mTFzVNrQM", "reviewer": "Reviewer_8w4H", "summary": "", "questions": "- Any thoughts about A1 above?\n- Any thoughts about A2 above?\n- What is the nature of \"expert annotator\"?\n- How did the authors decide the number of NoTarget samples. \n- The evaluation results show extremely high classification performance of the model. This suggests that the task is extremely easy. Is it possible that the model is making decisions due to specific biases in the dataset?", "limitations": "", "rating": 4, "confidence": 5, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Hello dear Reviewer, \n\nthank you very much for the detailed and comprehensive feedback. We find the points very good and will answer each of them briefly. All the background and details mentioned in the answers would also be included in the final paper version. \n\nRegarding A1 & A2 / Lack of discussion for the motivation & Lack of discussion for the task design: \nIndeed, the types of greenwashing are underspecified in the paper. We don't view ClimateBERT-NetZero yet as a tool for identifying greenwashing (of any type) but as a very important first step. This tool can help humans to fully or partially automate the net zero and reduction claim assessment. Thus, we aim to shed light on the fuzzy communication side. In the second step, this information has to be compared with real-world data (e.g., realized emissions) to identify greenwashing enterprises. With this paper and the publication of models and data, we want to extend the capabilities of a broader research and practice community. Hence, we also included the use cases of ambition assessment and large-scale analysis. \n\nRegarding Dataset issues (B1 and additional question): \n- To address the opacity of the label distribution, we added a Table with descriptive statistics of each validation set in the five-fol cross-validation. We used a stratified split. Therefore, the label distribution of the validation sets follows the general distribution (more or less 305 non-targets and 200 reduction and net zero targets).\n- Expert annotators are employed by the Net Zero Tracker which is a combined initiative of Oxford Net Zero, the New Climate Institute, Data Driven Envirolab, and the Energy & Climate Intelligence Unit. Thus, the annotation steps are done by either the authors or persons with domain know-how.\n- The number of No Target label sentences was set to balance the reduction and net zero claims. More specifically, it is mainly constructed with other climate-related sentences. The reason for this is that we propose to use the detection of climate-related sentences as introduced in the ClimateBERT paper as a preliminary step before finding net zero and reduction targets. So, this makes the actual real-world problem less imbalanced. However, also looking at the results, besides the accuracy, the F1 score is also very high indicating that all labels are predicted well. Actually, we also added the confusion matrix to the paper where we can see that the Non-Target is predicted with 99% accuracy giving evidence that the approach works particularly well on this important label.\n- We included sample texts in the paper to show what the labels look like. Furthermore, the classification problem is rather short paragraphs that include sentences.\n\nWith the feedback of Reviewer 3, we addressed further data issues to make the contribution more solid. I refrain from restating them here but it complements your feedback and thus might be of interest to read for you.\n\nIrrespective of the outcome, I want to thank you again for providing feedback. As mentioned, the problem we're trying to solve is nothing abstract but rather a pressing real-world task. This is why we are happy that you made meaningful comments to improve this work.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- This paper proposes a novel task and dataset on a net-zero perspective of climate change. The dataset is valuable to researchers because this field is nascent and lacks datasets and task definitions. The authors will release the dataset and models, which will be useful for researchers in this domain.\n- They appear to be collaborating with domain experts to create the dataset", "reasons_to_reject": "The major drawback is that the paper lacks detail in the discussion and dataset.\nIt is especially important to discuss motives because this paper takes on a new challenge.\nSee below comments.\n\n### High-level comment\n\nA1. Lack of discussion for the motivation\n\nThe authors begin their motivation with a reference to greenwashing. Indeed, claims of net zero have sometimes been accused of greenwashing, so I agree with the importance of this motivation.\nWhile the study seems to focus on extracting net-zero targets, there is no insight of what can be done to suppress greenwashing as a result.\n\nThere are various typologies of greenwashing, e.g., net zero may be dependent on carbon offsetting, there may be no clear intermediate target, it may not follow IPCC guidance, etc.\n\nIt is not clear what typologies of greenwashing the authors have in mind or whether they have extracted the necessary information to do so. It might be good to see these discussions in a related work, introduction, or discussion section.\n\n\nA2. Lack of discussion for the task design\n\nSimilar to above, I would like to know the motivation behind the task definition.\nThe net-zero claim is a complex matter in practice. For example,\n- Is the scope of the net-zero claim related to the company or for a specific project? Is it for Scope 1, 2, or 3 emissions?\n- The difference between intermediary and long-term goals.\n- Differences in units and calculation methods\n\nThe Appendix presents some of the guidelines, but they are oversimplified and lack an explanation of why the task was defined in the way.\n\n\n### Low-level comment\n\nB1. Detailed description of the dataset is missing.\n\n- The label distribution for both train and test data should be described in Table 1.\n- Please specify the nature of \"expert annotator\"\n- The design of the NoTarget label is questionable. These samples may be different from the domains of the NetZero and Reduction labels. Thus, they may overestimate the classification performance of the model that can use such \"bias\". In fact, the models are extremely accurate. Furthermore, the reason for selecting the number of NoTarget labels is ambiguous. For example, there should be far more NoTarget labels in a sustainability report or other data source.\n- The sample text of the dataset is not presented in the paper, making it difficult for readers to understand.\n- It is not clear whether the annotation is in a paragraph or a sentence -level.", "questions_to_authors": "- Any thoughts about A1 above?\n- Any thoughts about A2 above?\n- What is the nature of \"expert annotator\"?\n- How did the authors decide the number of NoTarget samples. \n- The evaluation results show extremely high classification performance of the model. This suggests that the task is extremely easy. Is it possible that the model is making decisions due to specific biases in the dataset?", "paper_topic_and_main_contributions": "The major contribution of this paper is the proposed dataset.\n\nNet-zero (\"net zero\" GHG emissions)  claims by firms and countries have been on the rise recently, but are sometimes pointed to as greenwashing.\nThis paper proposes a task and dataset for detecting net-zero claims from text.\nThe authors provided a 3.5K text annotated corpus including three labels: NetZero, Reduction, and NoTarget.\nThe fine-tuned ClimateBERT outperformed baselines.\nThe authors also experimented the reduction target year extraction by using QA approach.\nFinally, a case study on the earning call transcripts analysis suggests that the net-zero target discussion has experienced a sharp increase since 2019.", "excitement": 4, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Typos_Grammar_Style_and_Presentation_Improvements": "- By removing the bold text (\"Motivation\", \"Contribution\", \"Results\", and \"Implications\") in Introduction, more paper space can be used?\n- Table 1 should include train and test statistics."}, "extra_text": {"Missing_References": "I recommend citing and discussing some net-zero or greenwashing related papers.\nBelow is an example of papers I know of, but there must be others.\n\n- An Integrated Framework to Assess Greenwashing. Sustainability. 2022.\n- The meaning of net zero and how to get it right. Nature Climate Change. 2022."}}, {"review_id": "XM0nrfLQ9R", "reviewer": "Reviewer_vRSM", "summary": "", "questions": "Q1: What is the distribution of data between companies, countries, regions, and cities categories? Table 1 doesn’t provide detail on this. \nQ2: Appendix lines 451-455: How does performed preprocessing change the descriptive statistics of the data? Does it induce any bias? \nQ3: Appendix lines 456-463: If the data was already labeled by Net Zero Tracker then how can paper stat contribution as introducing new data? It is more appropriate to say refine the data. \nQ4: Appendix lines 463-464: What do you mean by “at least one author”? What was the average number of annotators used for each data point? How it was decided how many annotators to use for a particular data point. \nQ5: The paper claims that data was annotated by experts. What was the qualification of the annotators?\nQ6: What was an inter-annotator agreement? \nQ7: Appendix lines 482-491: The entire process of human-in-the-loop is not clear. Which model was used to identify misclassification? Filtering sentences for readjusting labels based on the model’s output might induce some kind of bias. If the model is wrong on certain data points which can be also wrongly labeled in the first place as authors are refining them, it contaminates the labeled dataset. \nQ8: What is the accuracy of the model in detecting Net Zero and Reduction Targets in Earnings Call Transcripts? \nQ9: In Figure 1, does y-axis values are multiplied by 100 to show the % value or it is just a fraction?\nQ10: In the second use case, does the sample of companies each year stays the same or it changes over time? If it changes over time, then one can’t say if the trend is changing because of the sample of earnings call transcript or some companies actually moving from not committing to climate goals to committing to them and vice versa. Also, it doesn’t list from which country these companies are located. The sharp increase since 2019 can be driven by just a change of policy in a few countries. \nQ11: As earnings call is a quarterly reporting not yearly, how and why data was aggregated to a yearly frequency in Figure 1? \nQ12: Why the guidelines listed in the appendix (lines 469-480) is not included in ChatGPT zero-shot prompt and rather a different prompt is used? \nQ13: How does model performance ranking change if hyperparameters different from the one listed in Table C.2 are used?", "limitations": "", "rating": 3, "confidence": 4, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Hello dear Reviewer, \n\nWe are very grateful for the detailed feedback you provided. Since the feedback was very constructive, we tried to address the issues and the corresponding answers and analyses would also be part of the final paper. I will give a brief version of the answers.\n\nAnswer to the questions:\n- Q1: The dataset contains 273 claims by cities, 1396 claims by companies, 205 claims by countries, and 159 claims by regions.\n- Q2: The descriptive statistics barely change after the preprocessing. We only exclude 97 data points (from 3614 to 3517) in the preprocessing. An overview of the descriptive statistics before and after preprocessing is added to the paper.\n- Q3, 4, 6: We not only collaborated with the Net Zero Tracker but one of the authors is also responsible for the data collection at the Net Zero Tracker. The data contained a collection of claims and the authors added an additional layer of labeling by checking each sample and the label. Thus, we can also create an inter-labeler agreement (Cohen's Kappa of around 93%). Upon disagreement, the data point was discussed with the team. Hence, at least two persons labeled a data point. \n- Q5: The annotators were employed by the Net Zero Tracker which is a combined initiative of Oxford Net Zero, the New Climate Institute, Data Driven Envirolab, and the Energy & Climate Intelligence Unit. Thus, the annotation steps are done by either the authors or persons with domain know-how.\n- Q7: For the additional human-in-the-loop quality check, we also used the five-fold cross-validation with ClimateBERT. We save the misclassifications of each fold and then an author decides whether the model is right or wrong. Thus, the model's decision boundary is always slightly altered and we reapt this process three times. This way, we assess samples where the model itself has trouble identifying the right label. Thus, we refine the model with the hard samples. This technique leans on Data Boosting which won an Andrew Ng challenge for deep vision systems.\n- Q8: The net-zero and reduction claim analysis in the earning calls is a use case, rather than an evaluation. Thus, the models are applied rather than tested with it and no accuracy is given. However, the patterns coincide with the general interest in sustainable investing since 2019 (see https://www.morningstar.com/sustainable-investing/2022-us-sustainable-funds-landscape-5-charts)\n- Q9: This was a fault, thank you.\n- Q10: We used the earning calls of all publicly listed US companies from 2003-2022. While I agree that the companies might change and there are likely some key events that influence the analysis (see link above), this use case mainly serves to show the large-scale analysis potentials and is not yet a full analysis for itself. It should give practitioners an idea of how the models could be used. Further descriptive statistics about for instance the number of events, companies, and analyzed sentences are added in the final paper version.\n- Q11: We build a yearly average of all quarters in a year to form a year index.\n- Q12: Initially, we wanted to create a simple prompt. I repeated the evaluation with the labeling guidelines in the prompt and achieved slightly better results. However, the overall picture of the BERT models outperforming ChatGPT stays consistent. \n- Q13: We've repeated the evaluation with a hyperparameter grid with the learning rates [3e-5, 5e-5, 7e-5], batch size of [16, 32], and epochs of [5, 10]. The results remained the same for all configurations with ClimateBERT outperforming the other models.\n\nIrrespective of the outcome of this review, we want to thank you for the good feedback. The underlying task is a very important real-world problem where advances are necessarily needed. Thus, we thank you for improving the work.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "This paper presents a refined dataset for a critical task and provides a comparative analysis of different models, including zero-shot ChatGPT. It also outlines two practical use cases, contributing to its acceptability.", "reasons_to_reject": "Despite careful examination of the main paper and its appendix, I found that vital information about the data and its annotation is notably absent. This omission complicates the assessment of the paper's quality, hence my low score for soundness. \n\nThe paper's primary contribution lies in introducing/refining the dataset. Given the significance of this contribution, comprehensive details on the annotation process are necessary.\n\nThe paper needs a lot of writing work to address many questions I have listed in the section below.", "questions_to_authors": "Q1: What is the distribution of data between companies, countries, regions, and cities categories? Table 1 doesn’t provide detail on this. \nQ2: Appendix lines 451-455: How does performed preprocessing change the descriptive statistics of the data? Does it induce any bias? \nQ3: Appendix lines 456-463: If the data was already labeled by Net Zero Tracker then how can paper stat contribution as introducing new data? It is more appropriate to say refine the data. \nQ4: Appendix lines 463-464: What do you mean by “at least one author”? What was the average number of annotators used for each data point? How it was decided how many annotators to use for a particular data point. \nQ5: The paper claims that data was annotated by experts. What was the qualification of the annotators?\nQ6: What was an inter-annotator agreement? \nQ7: Appendix lines 482-491: The entire process of human-in-the-loop is not clear. Which model was used to identify misclassification? Filtering sentences for readjusting labels based on the model’s output might induce some kind of bias. If the model is wrong on certain data points which can be also wrongly labeled in the first place as authors are refining them, it contaminates the labeled dataset. \nQ8: What is the accuracy of the model in detecting Net Zero and Reduction Targets in Earnings Call Transcripts? \nQ9: In Figure 1, does y-axis values are multiplied by 100 to show the % value or it is just a fraction?\nQ10: In the second use case, does the sample of companies each year stays the same or it changes over time? If it changes over time, then one can’t say if the trend is changing because of the sample of earnings call transcript or some companies actually moving from not committing to climate goals to committing to them and vice versa. Also, it doesn’t list from which country these companies are located. The sharp increase since 2019 can be driven by just a change of policy in a few countries. \nQ11: As earnings call is a quarterly reporting not yearly, how and why data was aggregated to a yearly frequency in Figure 1? \nQ12: Why the guidelines listed in the appendix (lines 469-480) is not included in ChatGPT zero-shot prompt and rather a different prompt is used? \nQ13: How does model performance ranking change if hyperparameters different from the one listed in Table C.2 are used?", "paper_topic_and_main_contributions": "The research article focuses on enhancing the existing Net Zero Tracker's capability to algorithmically assess the sustainability targets (reduction and net zero goals) of corporations and nations. To accomplish this, the authors present a labeled dataset, fine-tune a language model known as ClimateBERT-NetZero, and aim to make this trained model publicly available.\n\nThe research further introduces two unique use cases for the fine-tuned model (ClimateBERT-NetZero). First, it employs the extracted climate claims to ask supplementary questions and garner information such as the target year, base year, and percentage reduction. Second, it utilizes the trained model to extract climate goals from earnings call transcripts, illustrating that the percentage of companies communicating climate goals saw a significant rise in 2020 and 2021.", "excitement": 3, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {"Missing_References": "1. RoBERTa model is used but not cited: Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019. Roberta: A robustly optimized bert pretraining approach. arXiv preprint arXiv:1907.11692.\n\n    2. Anna Rogers article: Anna Rogers, Niranjan Balasubramanian, Leon Derczynski, Jesse Dodge, Alexander Koller, Sasha Luccioni, Maarten Sap, Roy Schwartz, Noah A. Smith, and Emma Strubell. 2023. Closed ai models make bad baselines.", "Typos_Grammar_Style_and_Presentation_Improvements": "In general, the paper can be greatly improved by providing more detail on everything. Below are some specific comments: \n\nC1: The abstract says corporates and nations while the “Data” section says companies, countries, regions, and cities. \n\nC2: It is more conventional to write 1,500 instead of 1.500 (line #165).\n\nC3: It will be better to include the ChatGPT prompt in the main paper instead of the Appendix. Even in the 4-page limit Limitations can be moved to the 5th page and the prompt could be included. \n\nC5: More details can be added to the caption of Table 3 to improve the readability of the work. \n\nC6: Data description for two use cases is missing in the main paper. \n\nC7: Table H.4 can use better legends."}}]

Author rebuttal:

Meta-review: This paper proposes ClimateBERT-NetZero, a natural language classifier for detecting corporate and national net zero and emission reduction targets from the text. The key contribution is a new expert-annotated dataset of 3.5K text samples categorized as NetZero, Reduction, or NoTarget claims. Authors have provided experiments which demonstrate ClimateBERT outperforms baselines when fine-tuned on this data. Additionally, they provided two use cases highlighting the potential to extract and analyze climate commitments at scale.

The motivation to assess ambiguous sustainability claims is timely and important. Reviewers praise the novelty of the task and release of models/data but raise issues about dataset details and reproducibility. In particular, Reviewer 2 gave very comprehensive feedback on framing the work and recycled interventions. In response, the authors sufficiently clarify the annotation process, data statistics, and human-in-the-loop mechanisms. Additional analyses also confirm model performance on real-world transcripts.

Personally, I think that the task is technically trivial, however, the significance and impact of the studies are meritorious. Moreover, I agree with the reviewers that the paper is fundamentally sound even though I am of the opinion that the paper would benefit from more discussion situating the work amidst greenwashing typologies and nuances of claim verification as also implied by one of the reviewers. Also, the current heavy focus on introducing the dataset leaves the overall contributions feeling somewhat incremental. Providing sample texts and better highlighting incremental contributions over prior climate NLP would strengthen the manuscript.  Finally, more robust framing, analysis, and comparisons to prior climate NLP could strengthen the clarity and impact.
Decision: Accept-Main