Evaluating ChatNetZero, an LLM-Chatbot to Demystify Climate Pledges (2024)

Abstract: This paper introduces and evaluates ChatNetZero, a large-language model (LLM) chatbot developed through Retrieval-Augmented Generation (RAG), which uses generative AI to produce answers grounded in verified, climatedomain specific information. We describe ChatNetZero’s design, particularly the innovation of anti-hallucination and reference modules designed to enhance the accuracy and credibility of generated responses. To evaluate ChatNetZero’s performance against other LLMs, including GPT-4, Gemini, Coral, and ChatClimate, we conduct two types of validation: comparing LLMs’ generated responses to original source documents to verify their factual accuracy, and employing an expert survey to evaluate the overall quality, accuracy and relevance of each response. We find that while ChatNetZero responses show higher factual accuracy when compared to original source data, experts surveyed prefer lengthier responses that provide more context. Our results highlight the importance of prioritizing information presentation in the design of domain-specific LLMs to ensure that scientific information is effectively communicated, especially as even expert audiences find it challenging to assess the credibility of AI-generated content.

heading: 1 Introduction
text: In the era of generative AI, the proliferation of climate change misinformation presents a significant challenge, impeding both scientific discourse and efforts to distinguish between credible and noncredible climate actions. Although scientific consensus has identified the imperative to achieve “netzero” emissions by mid-century, widespread disagreement over its definition and implementation remains (Fankhauser et al., 2022). For example, according to the latest Pew Research Center Poll, two-thirds of Americans say that the US should use a mix of energy sources, including fossil fuels,
which are fundamentally incompatible with a netzero world (Tyson et al., 2022). A surge of over 11,000 private and subnational entities have committed to respective decarbonization pledges, albeit with varying degrees of credibility (Institute, 2024; UNFCCC, 2023). Across the world, citizens and regulators are increasingly resorting to litigation to combat false and disingenuous net-zero claims (Carrington, 2023).
With more users relying on artificial intelligencedriven large language models (LLMs) like Google’s Gemini (Gemini Team, 2024) and OpenAI’s ChatGPT (OpenAI, 2022) to obtain primary information, it is a foregone conclusion that the public will turn to these resources to gain a deeper understanding of what governments and businesses are doing on climate change and decarbonization. These tools, however, are not attune to the rapidly evolving landscape of climate policy, specifically the nuances of net-zero goals, where definitions and interpretations of credibility are evolving daily. Since non-state actors report climate actions in a variety of formats (e.g., press releases, PDF reports, spreadsheets, websites, etc.) (Hsu and Rauber, 2021), even the task of assembling a coherent dataset to analyze and compare entities’ climate change strategies is challenging. Generative AI is prone to “hallucination,” where models produce seemingly real responses that could fail to correspond to any actual input, posing potential risks of hazardous and legally-disputable claims (Alkaissi and McFarlane, 2023).
Here we introduce and evaluate ChatNetZero— an LLM-based chatbot developed through Retrieval-Augmented Generation (RAG)—which employs generative AI to produce answers to users’ questions that are grounded in verified information (Lewis et al., 2021). It is designed to analyze unstructured net-zero related text documents and serve as a question-answering platform for climate policy-specific information.
82
ChatNetZero is able to accurately answer questions relating to broad net-zero domain knowledge, such as different terminology used to articulate net-zero commitments, as well as specific details on an entity’s net-zero pledge and its content. To evaluate ChatNetZero’s ability to provide accurate, high-quality responses, we assess its responses in two ways - comparing the generated responses to original expert texts; and engaging climate policy experts to evaluate its responses compared to other population chatbots, including GPT-4 (OpenAI, 2023), Gemini (Gemini Team, 2024), and Coral (Cohere, 2023), as well as the climate-domain specific ChatClimate (Vaghefi et al., 2023).

heading: 2 Background

heading: 2.1 The science of net zero
text: The concept of “net zero” refers to the equilibrium between human-caused greenhouse gas emissions and their removal, either through natural means such as carbon sinks (like oceans, land, and forests) or engineered methods like carbon capture and storage or direct air capture. Although rooted in climate science since the 2000s, its significance surged politically with the 2015 Paris Agreement. This historic Accord marked the first global commitment to limit the temperature rise to 1.5°C above pre-industrial levels, necessitating net-zero emissions mid-century (IPCC, 2018) and inspiring non-governmental actors to undertake their own net-zero initiatives (UNFCCC, 2023).
Major questions, however, continue to surround the credible and scientific implementation of netzero pledges, particularly regarding whether entities intend to completely eliminate emissions or plan to offset them by purchasing questionable credits from reductions elsewhere. Assessing the legitimacy of these commitments is challenging due to the prevalence of greenwashing, where numerous companies and government bodies engage in superficial efforts that mislead the public. Additionally, the public often lacks the necessary tools to discern credible or genuinely high-integrity climate pledges, as such evaluations typically require expert knowledge.

heading: 2.2 Previous applications of NLP for climate change
text: The potential of generative AI and LLMs to significantly improve access to climate-related information is rapidly gaining recognition, evidenced
by the increasing number of initiatives to develop climate-domain specific LLMs and chatbots in recent years. ClimateBERT was one of the first specialized transformer-based language models that was pre-trained on over 2 million climate-related texts, including news sites, research articles and company climate reports (Webersinke et al., 2022). The authors found that ClimateBERT outperformed a base LLM without domain-adaptive training (DistilRoBERTa) in text classification tasks identifying whether a text contained climate-related material. ClimateBERT-NetZero builds on ClimateBERT by detecting net zero or reduction targets in texts, leveraging the Net Zero Tracker data as a labeled dataset to pretrain the ClimateBERT classifier, which results in superior predictive performance compared to larger models (Schimanski et al., 2023). ChatClimate (Vaghefi et al., 2023) is a chatbot that instructs GPT-4 to only provide answers based on the IPCC’s climate science reports (IPCC, 2023). Others (i.e., ClimSight, see Koldunov and Jung (2024)) are experimenting with ways of combining physical climate data and LLMs to make data and information from climate models, including large-scale global precipitation and weather data, more accessible to users.

heading: 2.3 Limitations of climate-related LLMs
text: The development of domain-specific LLMs and general LLM applications highlights a growing demand for resources to enhance understanding of climate science and the actions taken by governments and businesses to address climate change and decarbonization. Beyond the well-documented hallucination problem, climate-related LLM applications are susceptible to replicating or exacerbating greenwashing, especially when trained on self-reported climate action data, which is often at risk of ‘net-zero greenwashing’ due to misalignment between climate pledges and corporate actions (InfluenceMap, 2023). The climate domain is also particularly prone to misinformation and political polarization in social media and other outlets (de Freitas Netto et al., 2020; Thapa Magar et al., 2024), a particular challenge for even climaterelated LLMs to distinguish (Leippold et al., 2024).

heading: 3 Methods

heading: 3.1 Data sources
text: We worked with experts from the Net Zero Tracker to identify the most relevant and credible docu-
2
ments with which to supply ChatNetZero. Since we do not want to contaminate the data retrieval process with possible greenwashing or falsehoods from the entities themselves (e.g., a company’s own corporate responsibility report or a government’s own climate action strategy), we initially only use four sources of information to ground ChatNetZero’s beta pilot:
• The United Nations High-Level Expert Group (HLEG) report on Integrity Matters: Net-Zero Emissions Commitments of Non-State Entities (HLEG, 2022): The HLEG report provides ten recommendations for companies, financial institutions, and subnational governments to establish credible net-zero pledges that are aligned with scientific scenarios and recommendations.
• Net Zero Tracker database and Net Zero Stocktake reports: The Net Zero Tracker (zerotracker.net) is the most comprehensive platform evaluating more than 4,000 entities’ net-zero and decarbonization efforts. These entities include all national governments, all regions in the G20, all cities with a population greater than 500,000, and the Forbes Global 2000 companies. The dataset evaluates whether an entity has declared a net-zero or similar decarbonization pledge in addition to more than a dozen indicators assessing their integrity. We also include the Net Zero Stocktake reports, which are annual reports assessing the status and trends of net zero targets in the database (Net Zero Tracker, 2022, 2023).
• NewClimate Institute’s Corporate Climate Responsibility Monitor Reports (New Climate Institute, 2022, 2023): These reports authored by the NewClimate Institute, a Germanbased climate policy think tank, evaluate the credibility of net-zero targets and policies set by 25 multinational companies, including Maersk, IKEA, Apple, Google, and H&M, among others.
Table 1 describes a summary of the final data used to train ChatNetZero. While these documents and data sources are not the singular authorities regarding net-zero and decarbonization policy, they represent a set of consistent and coherent benchmarks to ground ChatNetZero. Other documents, including The Oxford Principles for Net
Zero Aligned Carbon Offsetting (Allen et al., 2020) or British Standard Institute’s Net Zero TargetSetting Standards (Institute, 2023) may represent diverging viewpoints (i.e., regarding the use of offsets when an entity cannot meet its own emission reduction targets solely through its internal efforts) and were not used for ChatNetZero’s pilot, but would not necessarily be excluded from future model design and development.

heading: 3.2 ChatNetZero Design
text: To tackle the limitations of generic LLMs (i.e., hallucination), we developed a Retrieval-Augmented Generation design combined with other customized algorithms, including query processing, analytical text transformation, and chunk ranking algorithms. ChatNetZero also provides references with each answer that includes active hallucination checks that provide specific document and page references to users (see below sections 3.2.2 and 3.2.3). ChatNetZero’s workflow is illustrated in Figure 1, and we describe each algorithmic module in greater detail below.

heading: 3.2.1 RAG Module
text: Our RAG module entails a multi-step process to chunk, embed, query process, and customize responses to a user-inputted question.
3
Chunking All source documents, including Excel and PDF data, is converted into plain text, which is then segmented into chunks ranging from 50 to 1,000 words. To maintain the source data structure, each chunk encapsulates entire paragraphs. Each chunk is then embedded into a large, high-dimensional numerical vector, which represents the meaning of the text (Mikolov et al., 2013). For traceability, every chunk is tagged with its originating document’s name and page number, facilitating later checks against potential hallucination.
Embedding We embed both the chunked source documents and the user queries using OpenAI’s text-embedding-ada-002 model. This embedding space is used to perform semantic search between user queries and chunks from the source documents, allowing us to find the most relevant chunks to inform ChatNetZero’s final output response (see section 3.2.1). fate chose a small chunk size because when the response chunks were significantly longer than the user query, we found that the semantic search performed poorly, resulting in the selection of chunks that were not relevant to the user’s question.
Query processing We designed ChatNetZero to handle two types of user queries: actor-specific queries, where an individual entity or multiple entities such as a government or company are named; and generic queries, where a user asks a question that doesn’t identify a specific entity. For actorspecific queries, we developed an algorithm to recognize if the user’s query mentions a specific actor included in the Net Zero Tracker data and to then prioritize that data for answering. The algorithm handles irregular spelling, abbreviation, and translation of actor names, and has enhanced capability to cover all actors mentioned in a long query which effectively combats the “laziness” of LLMs when answering long questions (Guo et al., 2023). For actor-specific queries, at most 25 embedded chunks are retrieved from the backend, with between 1 and 5 chunks per actor. If the number of chunks exceeds 25, we reduce it to 25 while ensuring that each entity retains at least one chunk. We limit the number of chunks per entity to a maximum of 5. This process involves enforcing at least one chunk per entity specifically from the NZT Excel data, not from embedded reports. Additionally, our rule-based algorithm builds on the top-k chunk algorithm. Initially, we employ a Named Entity Recognition (NER) algorithm to identify any ac-
tors mentioned in the query, then we select chunks related to these actors from the NZT Excel data, and finally, we choose the top-k chunks from the remaining embedded documents. For generic queries, we select the top 10 most relevant chunks from the report documents, as determined by semantic similarity to the query.
Prompt Engineering We then take the retrieved chunks during the query processing step and combine them with the user prompt and then send it to OpenAI’s GPT-4 Turbo model. We use a temperature of 0.0 in order to ensure that the model produces reliable and consistent output. The model is instructed to follow a set of guidelines (see Figure 2) designed to facilitate clear and truthful answers.
Architecture ChatNetZero’s backend utilizes the LangChain architecture (Harrison, 2022) to allow for future interchange of LLMs without affecting the algorithmic process.

heading: 3.2.2 Anti-Hallucination Module
text: We developed an anti-hallucination module that first processes the raw output of the GPT-4 Turbo LLM after the Prompt Engineering step described above by dividing it into sentences and embedding them using the same process as described above (Embedding). Each vectorized output sentence is then compared against selected chunks from the RAG Module to verify its origin; sentences that cannot be traced to an original chunk are then excluded, given untraceable sentences’ high potential for hallucination. To evaluate the performance of the anti-hallucination module, we conducted several assessments with the Net Zero Tracker team, which is comprised of climate science and policy experts and 300 volunteers who have helped to
4
collect and validate data on the Tracker’s 4,000+ entities.

heading: 3.2.3 Reference Module
text: This module enables automated validation of ChatNetZero’s outputs and ensures traceability to one of the original data sources (see above). If a sentence successfully passes the anti-hallucination algorithm, the module appends a citation to the corresponding report to the generated response, including the page number and sentence position of the matched content. The module’s output, presented to the user via a web application, includes references for each sentence. These references link directly to the original pages of the source material so users can manually check and validate ChatNetZero’s generated response.

heading: 3.2.4 Enhanced Analytical Capabilities
text: To address LLMs’ inherent limitations in mathematical tasks, we developed an algorithmic process that enhances the model’s utility in interpreting and responding to queries requiring analytical analysis (for example, “How many companies in Germany have pledged a net-zero target?”). The algorithm restructures the Net Zero Tracker dataset, which tracks over 30 net-zero variables for over 4,000 actors, from an Excel tabulated format into optimized natural language sentences. This transformation enables the numerical data to be retrieved using the same process as text chunks, enabling the LLM to utilize numerical net-zero data and significantly enhancing the range of questions that ChatNetZero can deliver to its users.

heading: 3.3 Validation

heading: 3.3.1 Factual Evaluation
text: To assess the factual accuracy of ChatNetZero, we prompted four other large language models, including ChatClimate, GPT-4 Turbo, Gemini 1.0 Ultra, and Coral with Web Search with eight questions (Figure 3) that relate to factual statements regarding details of specific climate actors’ net zero or climate pledges. We used reputable sources—such as official policy documents and corporate reports— as ground truth reference material. The evaluation strictly assessed factual accuracy by determining if responses (found in Appendix A) exactly matched the reference material. We analyzed two aspects of the LLM responses. First, whether the LLM provided a direct and correct answer to the question provided:
• If the question asked about conditions for the use of offsets for B company, we evaluated whether the LLM provided a direct answer to that question (i.e, B Company has/doesn’t have conditions in the use of offsets), regardless of other contextual or additional statements included in the answer.
• If the reference material indicated that a company’s climate target was to reduce 30% emissions by 2050, we expect a correct answer to include both figures (i.e., the 30% and 2050 target year) when describing the climate targets of the company.
If the LLM provided an exact match to the data provided in the source material, we assigned a score of 1; if not, we scored the response 0.
Second, we evaluated each factual sentence in an LLM’s answer individually either as ‘Correct’, ‘Incorrect’, or ‘Unverifiable’, regardless of whether they addressed the main question or if they were simply contextual statements. We report this score as the ratio of correct factual statements to the total number of factual statements.

heading: 3.3.2 Expert evaluation
text: Beyond assessing factual accuracy, we posed 12 questions (see Figure 4) to each large language model. We then anonymized and randomized their
5
responses (found in Appendix A) in a Qualtrics survey, which we distributed to 10 climate scientists and policy experts. While ChatNetZero was designed to include references for each response, and some LLMs (including Gemini and Coral) provide references as well, we removed these from the responses for the Qualtrics survey so that experts would evaluate the quality of the responses themselves. These experts were asked to evaluate each response across three dimensions: overall quality, factual accuracy, and relevance. Respondents were asked to evaluate each response on a scale of 1 to 5, with 5 being the highest and 1 being the lowest. They were also given the opportunity to provide qualitative comments.

heading: 4 Results

heading: 4.1 Factual Evaluation
text: Table 2 provides a summary of the scores for our assessment of the factual accuracy of five LLM outputs, including ChatNetZero. Overall, our evaluation shows that ChatNetZero has higher factual scores for both its answers to the question itself and for the rest of additional information that provides more context or complements the main answer to the prompted question. For example, when asked,
“How does Wal-mart’s climate goals compare with Amazon’s and other large retail stores?” (Figure 3, Question 1), ChatNetZero provided more factually accurate answers than the other LLMs (see Appendix A for the factual scores of individual responses). An exception was GPT-4, which displayed a similar level of accuracy. Both provided factually correct responses to the main question and had the majority of their factual statements verified as correct. However, when asked a similar question about two non-English entities such as Foxconn and Fast Retailing (Figure 3: Question 6), the factual accuracy scores of ChatNetZero were higher than all other LLMs, many of which were unable to provide complete answers, likely due to limitations in their training data.

heading: 4.2 Expert Evaluation
text: Across all 12 questions, experts evaluated Gemini Ultra—followed closely by GPT-4—as producing the highest quality responses overall (3.91±0.91), with the greatest relevance (4.0±0.96) and factual accuracy (3.9±0.91) (see Table 3). ChatNetZero yielded the lowest overall quality (2.64±0.87), relevance (2.92±0.94), and factual accuracy (2.94±1.07) of the LLMs evaluated. As Figure 5 illustrates, however, performance varied by question, and there were several questions where ChatNetZero was evaluated to have on average a comparable or better overall response compared to Gemini, such as Question 3: “Does 3M or Pfizer have any conditions on the use of offsets.” We provide responses from ChatNetZero and the other LLMs evaluated in Appendix A.
We believe that the relatively low performance of ChatNetZero was in part due to its shorter average response length (110.5±8.91 words) compared to the other LLMs (Table 4). GPT had the largest average number of words per response (434.92±70.75 words), followed by Gemini (361.25±91.32 words). We found a generally positive correlation between an LLM’s response length and the expert evaluated quality of the response (Figure 6), with the experts’ evaluation of the factual accuracy of a response most closely related to the word length of the response (R2=0.43).

heading: 5 Discussion
text: The design of ChatNetZero and our comparison of its outputs to one climate domain-specific LLM (ChatClimate) and other popular LLMs (Gemini,
6
GPT-4, and Coral) underscores several findings about the use of LLMs in navigating the complex landscape of climate policy, particularly in relation to rapidly shifting and emerging concepts like “net zero.”
Length vs. perception of accuracy
While ChatNetZero was designed to deliver concise and accurate responses—confirmed by our
factual evaluation comparing responses to source documents—our expert evaluation showed a preference for longer, more detailed answers that of-
7
fer broader context, even if the added information isn’t always accurate or verifiable (Tables 2, 3, 4). This “verbosity bias” (Saito et al., 2023) indicates that humans tend to prefer longer, more detailed answers, believing they are more accurate than concise ones. Similarly, (Chiesurin et al., 2023) found that users favor fluent, grammatical responses and sophisticated linguistic dialogue, even when these responses lack trustworthy information.
We reviewed experts’ qualitative comments to gain further insight. Notably, experts who provided additional comments regarding the length of the evaluated LLMs’ responses said they favored longer answers provided by Gemini and GPT-4. This preference likely contributed to the higher scores for more extensive responses compared to the more concise bullet-point answers from ChatNetZero, which were designed for brevity but were seen as disrupting the flow of information and reducing readability, despite the high accuracy of ChatNetZero’s responses compared to the lengthier responses of other LLMs like Gemini and GPT-4.
Balancing factual accuracy with contextual relevance
Our finding of ChatNetZero’s higher factual accuracy but lower expert evaluation compared to other models suggests that while factual correctness can be achieved through grounding a general-purpose LLM, the utility of responses in practical scenarios also heavily depends on the completeness and contextual alignment of the information provided.
Expert feedback suggests that a model’s ability
to integrate accurate data into contextually relevant responses is essential. Responses that simply list facts without a nuanced understanding of the topic may fail to meet users’ needs for clear, actionable insights. This is especially critical in complex areas like climate policy, where decisions depend not only on data but also on its interpretation within diverse socio-economic and environmental contexts. However, the preference for lengthier responses from ChatNetZero over shorter statements might also be due to the specific user group in our study. Since our respondents were limited to experts in climate science and policy, including a more diverse or less specialized participant base could lead to different results.
Distinguishing factual accuracy in LLM-generated responses
Our study further demonstrates the challenge of utilizing LLMs to distinguish between accurate and irrelevant or even hallucinated content. This distinction is critical, as misinformation or misinterpretations in such a technical and impactful field can lead to poor decisions and public misconceptions. Although ChatNetZero generally provides factually correct responses, distinguishing these from less relevant or lengthier, contextualized answers remains difficult. This issue was reiterated by (Bulian et al., 2023), who reviewed climate responses from several LLMs and found that while the models scored high in information presentation, they were weak in the quality of the content provided.
8
We found that many responses from other LLMs contained auxiliary statements that were not factbased. While these statements enhance the answers’ readability, they could potentially lead to misinterpretation about the validity or adequacy of the responses. For instance, in our earlier example about Walmart and Amazon (Figure 3: Question 1), one LLM stated, “As of 2023, Walmart, Amazon, and other large retail stores have been increasingly vocal and active in their commitments to sustainability and addressing climate change.” This introductory statement, without specific evidence or references to their enhanced vocal and active roles, could potentially lead to issues like greenwashing if not carefully scrutinized.
This result from our study highlights key lessons from the science communication literature, which emphasizes the importance of information presentation: scientific information should be comprehensible, aid understanding through layout and visualizations, and use appropriate sources and references (Bulian et al., 2023; Jamieson et al., 2017). Since we removed the reference features from all LLMs in our human evaluation study, users were not presented with this third criterion for presentational adequacy of scientific information—sources and citations—which might have influenced their evaluation of the overall quality or factual accuracy of the LLM responses. As a result, users were neither able to individually verify the accuracy of the responses nor use this feature to gauge response quality. Future validation efforts could involve asking users to assess LLM responses in conjunction with the provided references and to evaluate the sources themselves.

heading: 5.1 Future implications
text: Although climate-specific LLMs can enhance the understanding and application of climate strategies, additional research is needed to explore how the framing, length, and presentation of responses affect users’ comprehension and perception. For regulators, they promise efficient, accessible information to facilitate the examination and confirmation of climate commitments, with the potential to foster greater scrutiny and trust through transparency. Businesses and other entities could also use these tools as an important benchmarking platform to understand which competitors have developed high-integrity, credible climate efforts. For advocates and the public, they promise the ability to hold entities accountable, provide access to re-
liable information, and engage more effectively in climate action discussions.

heading: 6 Limitations
text: Our study here is not without its limitations. The number of experts responding to our study (around 10 in total) was relatively modest and mainly represented the academic, scientific and policy communities. In the future, we could expand user evaluation to other demographics, including business, government, activist, or non-expert audiences. Second, by removing references from the LLMgenerated responses, it’s unclear how their inclusion by ChatNetZero and Gemini might have influenced experts’ views or the perceived quality, accuracy, or relevance of the responses. Last, different parameter tunings, for example an LLM’s temperature parameter, could result in a range of different responses (Dahl et al., 2024), which may affect a user’s interpretation. Future evaluation could test the sensitivity of a user’s evaluation to the temperature parameter on ChatNetZero’s responses. Last, we acknowledge that ChatNetZero still assumes a base-level understanding of climate change concepts, including net zero. Future versions should consider whether additional user prompting or response framing should be presented to assess the user’s baseline knowledge or prompt them to specify their level of understanding.

heading: 7 Conclusion
text: This study underscores the critical role and potential of specialized large language models like ChatNetZero in enhancing understanding and engagement in climate policy discourse. By demonstrating higher factual accuracy in handling complex climate-related content, ChatNetZero shows promise as a valuable tool for disseminating reliable information. However, challenges such as ensuring the presentation of information and contextual depth of responses remain. Future enhancements to ChatNetZero will consider options allowing users to customize response length and style. Adding more language could potentially compromise response quality, a challenge that will need further refinement. Addressing these challenges will be crucial for maximizing the utility of LLMs, ensuring that a customized LLM such as ChatNetZero provides transparency in its algorithmic processes to instill the trust and confidence required for any tool to impact decision making.
9

heading: A Supplementary Material
text: Data from the factual and expert evaluations can be found on our Dataverse.
11

heading: Future Work
text: Future enhancements to ChatNetZero will consider options allowing users to customize response length and style. Adding more language could potentially compromise response quality, a challenge that will need further refinement. Addressing these challenges will be crucial for maximizing the utility of LLMs, ensuring that a customized LLM such as ChatNetZero provides transparency in its algorithmic processes to instill the trust and confidence required for any tool to impact decision making. 9