Aligning Unstructured Paris Agreement Climate Plans with Sustainable Development Goals (2024)

Abstract: Aligning unstructured climate policy documents according to a particular classification taxonomy with little to no labeled examples is challenging and requires manual effort of climate policy researchers. In this work we examine whether large language models (LLMs) can act as an effective substitute or assist in the annotation process. Utilizing a large set of text spans from Paris Agreement Nationally Determined Contributions (NDCs) linked to United Nations Sustainable Development Goals (SDGs) and targets contained in the Climate Watch dataset from the World Resources Institute in combination with our own annotated data, we validate our approaches and establish a benchmark for model performance evaluation on this task. With our evaluation benchmarking we quantify the effectiveness of using zero-shot or few-shot prompted LLMs to align these documents.

heading: 1 Introduction
text: The 2015 Paris Agreement established 165 country specific Nationally Determined Contributions (NDCs) specifying global commitments to sustainability and resilience. Revised NDCs were released in 2021–2022. The NDCs set ambitious climate action targets but are presented in unstructured texts making any analysis or tracking of goals over time difficult. The United Nations Sustainable Development Goals (SDGs) provide a structured framework of 17 goals and 169 sub-targets aimed at promoting global well-being and sustainability. The SDGs serve as a hierarchical taxonomy. Linking NDC text spans to SDG goals and targets can enhance the understanding of global sustainability targets and offers a clear way to track progress. Previous work by Climate Watch at the World Resources Institute manually linked NDC text spans to SDG goals and targets (Northrop 2016) but such an effort
*These authors contributed equally to this work.
is difficult to generalize and maintain as new NDCs are released every five years. This study explores computational methods to tackle the challenge of aligning detailed, jargon-heavy unstructured climate documents to structured taxonomies in the context of limited labeled data, allowing us to significantly extend and enhance existing NDC-SDG datasets.
Prompting LLMs provides a relatively unsophisticated yet powerful way to leverage the models’ capabilities. Furthermore, many of today’s most advanced LLMs are easily accessible through APIs and web interfaces, making them well suited for a wide range of climate policy researchers. However, there are concerns that LLMs can “hallucinate” and may struggle with understanding context, nuance, and long-term dependencies in text, leading to less coherent or relevant outputs in complex tasks. A formal evaluation of the utility of LLMs for the task at hand is currently lacking.
Our contributions are as follows: 1) We conduct an empirical study of the performance of LLMs and cross-encoder architectures on the task of aligning NDCs to SDGs. 2) We introduce a benchmark for comparing our models, annotators, and the existing Climate Watch dataset. 3) We analyze specific methods to further boost performance on this task.
Finally, we will release the full NDC reports with their predicted SDG alignment as an artifact for the community to use, fostering transparency and ensuring the aims of international agreements are better understood, monitored, and ultimately realized.

heading: 2 Related Work

heading: 2.1 NDC SDG Linking
text: Existing research that has explored NDC-SDGs has relied on manual expert annotations. Policymakers across several jurisdictions observe that there is significant overlap between the implementation
223
process for SDGs and NDCs, and that the linking of both policymaking processes increases the efficacy of climate policy design. Northrop et al. (2016) and Brandi et al. (2017) provide detailed evidence for the convergence between SDGs and NDCs. AntwiAgyei et al. (2018) aim to leverage the alignments and misalignments between West African NDCs and global SDGs to increase the efficacy of West African climate policies.
Due to the effort required to align NDCs with SDGs, most studies are limited in scope: concentrating on a specific geographical region (AntwiAgyei et al., 2018) or selecting a single or subset of the SDG goals (Gallo et al., 2017; Smith et al., 2023). In our study we have coverage across all SDGs and targets, geographical regions, provided the availability of NDC document in English language, the entire texts of the documents. Other approaches utilize keyword search or extraction techniques to label data, however, these methods have limitations (King et al., 2017), including potential biases introduced by the choice of keywords.

heading: 2.2 NLP for Climate
text: NLP serves as a powerful tool to assist in many climate-related tasks. Stede and Patz (2021) show that NLP can provide many insights to policymakers and activists, as it aids with processing a large quantity with varied types of information. In previous works, NLP has been used to identify climate change misinformation (Farrell, 2019), analyze finance documents for climate-related text (Luccioni et al., 2020), and identify sustainability goals in peer-reviewed academic papers (Smith et al., 2023). Due to the importance of climate documents and the challenges of understanding the technical language used in them, researchers have also trained models specialized for interpreting climate documents, such as ClimateBert (Webersinke et al., 2022) and cross-encoder models to answer questions about climate texts (Spokoyny et al., 2023).
NLP has also been used to align various documents with climate targets, which provide insight regarding progress toward implementation of climate objectives (Roelfsema et al., 2020). Most recently, Juhasz et al. (2024) analyzed climate targets in national laws and policies. They trained a classifier to classify text into three different categories, ‘Net Zero’, ‘Reduction’ and ‘Other’ (Juhasz et al., 2024). Their work demonstrates the potential of using NLP to scale analysis of climate policies.
In our study, we extend the climate target classification to the SDGs, which allow us to classify broader targets related to sustainability in climate documents.
There has also been exploratory work on using ChatGPT to interact with climate documents such as the Intergovernmental Panel on Climate Change Report (IPCC) (Vaghefi et al., 2023). In contrast, our aim is to understand how modern LLMs with zero shot prompting, or few-shot incontext-learning could assist in these tasks.

heading: 3 Datasets
text: In this section we will introduce the World Resources Institute’s Climate Watch dataset that we used for our experiments and additional benchmarks we constructed (Northrop et al., 2016).
The Climate-Watch dataset includes sentences from NDCs submitted before 2021, each of which are labeled with goals and targets. Some sentences are labeled with a single goal and target, and others may also be labeled with multiple goals or targets. Some statistics on the dataset, example sentences and their labels, as well as our data pre-processing can be found in Appendix Section A.1.

heading: 3.1 Constructing Additional Benchmarks
text: We also created two small evaluation datasets that we will use to benchmark various aspects of our prompting strategies.
To construct the Data-Random dataset, we preprocess the HTML version of the NDC reports, using the NLTK sentence tokenizer on the the HTML tags that contain the majority of the textual content (<p> and <li>). We further filter the sentences to be between 80 and 300 characters in length. Across all of the reports, this yields over 100,000 sentences. From this set, we randomly sampled 120 sentences to be labeled by our annotators, which yielded sentences from 32 NDC reports.
To construct the Data-Balanced dataset, we selected 5 random annotations from Climate-Watch for each of the 17 SDG-Goals, which represented text from 53 of the NDC reports.
Both of these datasets were subsequently labeled by three separate manual annotators: one expert climate scientist and two university students with some climate policy understanding. Each sentence was independently labeled with up to three SDGGoals that the annotator believed were most relevant to the sentence. For the Data-Random dataset,
annotators could optionally select a “not relevant” label if they believed the sentence did not align with any of the SDG-Goals .
Annotators were briefed on the 17 SDGs, examined 85 labeled examples from the Climate Watch dataset, and then independently labeled 120 random text spans, associating each with up to three relevant SDGs. We include analysis of Inter Annotator Agreement in Appendix Section A.2.
Later, in Section 4.1 we will use the DataRandom to estimate the portion of the NDC documents that have been labeled in the Climate-Watch dataset. The Data-Balanced dataset will allow us to compare the performance of both our models and annotators against a balanced set of the ClimateWatch dataset.

heading: 4 Experiments
text: In this section, we introduce our experiments in which we use different prompting strategies with GPT models to classify sentences according to SDG. We will use ChatGPT-3.5 and GPT-4-Turbo as our main models to conduct prompt-based classification experiments. We will use JSON-mode API option to ensure the model outputs are properly structured for classification tasks. As our zeroshot classification baselines we will use MiniCDP, the cross-encoder model finetuned on the semistructured Carbon Disclosure Project (CDP) questionnaire data from Spokoyny et al. (2023) as well as its base model architecture MiniLM model.
4.1 Data-Random
First, using our manual annotations we will try to estimate the existing coverage of the ClimateWatch dataset. We found that out of 120 sentences, 13 were labeled non-relevant by the Expert and 8 were labeled as not-relevant by at least two of the annotators. From this, we estimate that around 85% to 95% of the sentences in the NDC are relevant to some SDG. Since there are on average 724 sentences per document, of which 66.4 sentences are labeled in the Climate-Watch dataset, we estimate that only 10-15% of the NDC have been labeled as a result. We show a histogram of the predicted SDG-Goals for the Data-Random dataset in Appendix Figure 4.
Although this is a very rough estimate, it clearly shows that the vast majority remains unlabeled and motivates the need for a more scalable approach to labeling these documents. Although, to
our knowledge, there is no full description of the methodology used to construct the Climate-Watch dataset, Northrop et al. (2016) suggests that keyword searches along with possible relevance, such as “countries with large coastlines were initially reviewed to identify alignment with targets relating to oceans and coasts”.
Following this analysis, we aim to also measure how LLMs perform compared to our annotators on this random subset of sentences from the NDC documents. To do so we construct a simple prompt to predict multiple SDG-Goals for each sentence. We have a simple instruction:
Given the following Input Text predict the Sustainable Development Goal ( label) out of the following 17 options:
followed by listing out all of the SDG-Goals . We include a full prompt for SDG-Goal prediction in Appendix Section C.1. To further encourage the model to produce well-formatted JSON outputs, we include an output specification in the prompt, and we provide multiple numbers for experiments predicting multiple SDG-Goals :
Generate a json object like so: {\’label \’: [\’2, 5\’]}
And lastly, to capture non-relevant sentences, we include “0: None of the above labels are applicable” as an option in the list of SDG-Goals as well.
As models we use ChatGPT-3.5 and GPT-4Turbo with the same prompt. We find that GPT-4Turbo correctly predicted 6 out of 13 non-relevant sentences, while ChatGPT-3.5 was unable to predict any. Upon closer inspection, we found that ChatGPT-3.5 predicted a very general goal, (Goal13: Take urgent action to combat climate change and its impacts), for a majority of non-relevant sentences.
To evaluate the performance of the models we calculate the accuracy as whether the model’s prediction matched one of the Expert labels. We use the Jaccard similarity to measure the overlap between the sets of SDG-Goals. We show the results in Table 1. From the results, we see that on random sentences from the NDC documents, both GPT models perform at similar levels to the annotators. We also conducted an experiment where we prompted the model to only produce one goal. We include these results in Appendix Section B.1.
4.2 Data-Balanced
First we want to compare the performance of our annotators against the annotations from the Climate-Watch dataset. As our metric, we report whether the percentage of sentences where annotators selected the same SDG-Goal as the ClimateWatch dataset. For our three annotators we found this to be 49.4%, 57.6%, and 48.2%. By using a balanced dataset, we can also evaluate the average accuracy of our annotators for each SDG-Goal shown in Appendix Table 10 along with a confusion matrix in Figure 1.
In Table 2 we compare the performance of our models on the Data-Balanced dataset. We find that with the top scoring SDG-Goal the MiniCDP model achieves an accuracy of 30.6% while the MiniLM model is almost 9% lower at 21.1%. Both of the LLMs perform much better with the ChatGPT-3.5 model achieving 47.1% and the GPT4-Turbo model achieving 49.4%.
Since in the Data-Balanced split there is only a single SDG-Goal label for each sentence, we also aim to quantify how well the models perform against our annotators with multiple SDG-Goal label predictions. For the MiniLM and MiniCDP models, we simply take the models’ top three scoring goal predictions.
We select the annotator with the highest accuracy
against the Climate-Watch labels to compare our model predictions against. We use the Jaccard similarity to measure the overlap between the sets of SDG-Goals . The results are presented in Table 3.

heading: Annotator Correct Wrong Jaccard
text: We again find that the MiniCDP model to be slightly better than the MiniLM model with Jaccard scores of 0.19 and 0.17, respectively. While both of the other annotators have Jaccard scores of 0.46, the GPT models achieve higher similarity scores of 0.48 and 0.50.
The confusion matrix in Figure 1 shows high agreement for SDG 13 (Climate Action) but also frequent cross-labeling with other goals, reflecting SDG 13’s overarching nature in climate texts. SDG 15 (Life on Land) and SDG 7 (Affordable and Clean Energy) displayed notable confusion with goals concerning water and urban development. In contrast, specific goals like SDG 2 (Zero Hunger) were less represented and often conflated with other poverty and health-related goals. The confusion matrix reflects the SDGs’ thematic overlaps, indicating that some noise in annotation is inevitable, even with expert input. Employing LLMs for SDG extraction from climate texts will also entail some acceptable level of noise, consistent with expertlabeled data variability.
4.3 Climate-Watch Although, the Data-Random and Data-Balanced data splits are relatively small, we have found that prompting GPT models to predict SDG-Goals is a promising approach for classifying sentences. In our final set of experiments, we will use the Climate-Watch dataset to benchmark prediction of SDG-Targets. From the full Climate-Watch dataset we randomly selected 200 sentences and in this section will refer to it as the ground truth.
We explore two modes for predicting the SDGTargets, oracle: where we use the ground truth
SDG-Goal label to sub-select only the corresponding SDG-Targets, and full: where we predict all SDG-Targets for a given sentence. A sample prompt for the oracle mode can be found in Appendix Section C.2. We prompt the models to produce the SDG-Target labels as JSON objects. Since many sentences have multiple SDG-Target labels, for our metric we use the Jaccard similarity. Results for these experiments are shown in Table 4.
For the full mode, we see that GPT-4-Turbo is substantially better than ChatGPT-3.5 with Jaccard scores of 0.42 and 0.28, respectively. As expected, in the oracle mode both models perform better with the gap between the two models slightly decreasing.

heading: 4.3.1 In Context Learning
text: One of the most desirable features of modern LLMs is their ability to use task-specific examples in their prompt to further boost performance. In the next set of experiments, we additionally provide up to 20 in-context learning (ICL) examples to both of our models. An example of some ICL examples is included in Appendix Section ??. We show the results in Table 5.
We find that the ChatGPT-3.5 model improves with additional ICL examples, getting much closer to the performance of the GPT-4-Turbo model. In contrast the 20 ICL examples only slightly improve the performance of the GPT-4-Turbo model.
We also experimented with prompting strategies such as expert prompting but found this did not seem to have any major effect. Results from this experiment are included in the Appendix Section B.2.

heading: 4.4 Artifact
text: To enable climate researchers to continue research in this direction, we use the best existing configuration we identified to annotate the entire NDC documents according to the SDG Goals and Targets. We aim to provide the annotations in a structured format along with the original NDC documents.

heading: 5 Conclusion
text: We have constructed benchmarks to compare the performance of models, annotators, using the Climate-Watch dataset on unstructured NDC documents. Using this data we find that existing manual efforts provide low coverage, motivating the need for automated methods. Finally, we found across various experiments that by prompting GPT models we could match the performance of our annotators on SDG-Goal and SDG-Target prediction. Our findings highlight the potential of leveraging GPTbased models to effectively annotate unstructured climate documents such as the NDCs.

heading: A Appendix

heading: A.1 Dataset and Preprocessing
text: The Climate-Watch dataset has the SDG annotations, various associated metadata, and the raw text snippet from the NDC documents. Statistics on the dataset and text snippets are shown in Table 6 and Table 7.

heading: Property Number
text: Each sentence in the document is labeled with one of the 17 SDGs and one of the 169 targets.
Some sentences may also be labeled with multiple goals or targets. Example sentences and their labels are shown in Table 8. In Figure 2, we show the distribution of SDG-Goals in the Climate-Watch dataset.
Additionally, these snippets are not directly linked to the exact locations in the NDC documents. We obtain a dataset of the full texts of the NDC documents as HTML files and using simple heuristics were able to match 94.8% of the annotations to their exact document spans. In Appendix Figure 3 we plot the distribution of where in the NDC documents the Climate-Watch annotations are found.
A.2 Inter Annotator Agreement
Using our annotators, we show in Appendix Figure 4 the distribution of the predicted SDG-Goals for the Data-Random dataset, which we can contrast with the distribution of SDG-Goals in the Climate-Watch dataset (Appendix Figure 3). We found the most common SDG-Goals in DataRandom were 13, 15, 7 whereas in the ClimateWatch dataset it is 7, 15 and 2. The SDG 13 (Take urgent action to combat climate change and its impacts) could be interpreted very broadly and thus our annotators ended up selecting it for a variety of sentences.
SDG 15 focuses on protecting terrestrial ecosystems, SDG 7 targets the provision of sustainable energy, and SDG 2 concerns ending hunger and promoting sustainable agriculture. SDG 13, climate action, is central to the Paris Agreement. SDG 15’s frequent appearance stems from the Paris Agreement’s emphasis on land use in climate mitigation. SDG 7’s prominence aligns with the focus on energy systems transformation in national strategies. The lesser emphasis on SDG 2 in the Data-Random dataset compared to the Climate-Watch dataset may indicate a different thematic focus in their data set.
For the Data-Random split we calculated the inter-annotator agreement using Cohen’s kappa (which has a range of -1 to 1) between the expert and each of the novices as (0.629, 0.524) (Cohen, 1960). However, on the Data-Balanced the agreement was lower (K = 0.215, K = 0.179), reflecting disparate annotation strategies among the annotators. Notably, some annotators demonstrated a conservative approach, opting to select only the primary goal, whereas others exhibited more leniency in their selections.

heading: B Experiments
text: B.1 Data-Random Single Goal Prediction
In this experiment, we prompted ChatGPT-3.5 and GPT-4-Turbo to classify each sentence with a single SDG-Goal . We show results in Appendix Table 9. To evaluate the models, we consider a classification correct if it matches any of the SDG-Goals that the Expert selected. We find that both models perform well with GPT-4-Turbo being slightly better. We also include the other two annotators as a point of reference although it is not a direct comparison, as annotators were allowed to select up to three SDG-Goals .

heading: Annotator Correct Wrong Accuracy

heading: B.2 Prompting Strategies
text: There are a variety of prompting techniques that have been shown improve performance such chain
of thought (Wei et al., 2022), maieutic prompting (Jung et al., 2022), or self-ask (Huang et al., 2022). Xu et al. (2023) found that providing a model with a prompt that describes an identity of distinguished expert can improve performance. We experiment with a simple form of expert-prompting for a climate policy expert. We generated the expert identity using GPT-4 using an example from Xu et al. (2023), and added “You are a climate policy expert...” to the beginning of our instruction. Using the expert prompt, we run SDG-Target prediction using both the full and oracle modes. The results are shown in Table 11 and the full expert-prompt is shown in Appendix Section C.4. We find that there is a small improvement for both models in the oracle mode but no effect in the full mode.

heading: Model Avg Jaccard

heading: C Prompts
text: C.1 One Full SDG-Goal Prediction Prompt Below is one full prompt used for zero-shot SDGGoal prediction. Given the following Input Text predict
the Sustainable Development Goal ( goal) out of the following 17 options:
Sustainable Development Goal 1: End poverty in all its forms everywhere 2: End hunger , achieve food security and
improved nutrition and promote sustainable agriculture
3: Ensure healthy lives and promote well -being for all at all ages 4: Ensure inclusive and equitable quality education and promote lifelong learning opportunities for all
5: Achieve gender equality and empower all women and girls 6: Ensure availability and sustainable management of water and sanitation for all 7: Ensure access to affordable , reliable , sustainable and modern energy for all 8: Promote sustained , inclusive and sustainable economic growth , full and productive employment and decent work for all 9: Build resilient infrastructure , promote inclusive and sustainable industrialization and foster innovation 10: Reduce inequality within and among countries 11: Make cities and human settlements inclusive , safe , resilient and sustainable 12: Ensure sustainable consumption and production patterns 13: Take urgent action to combat climate change and its impacts 14: Conserve and sustainably use the oceans , seas and marine resources for sustainable development 15: Protect , restore and promote sustainable use of terrestrial ecosystems , sustainably manage forests , combat desertification , and halt and reverse land degradation and halt biodiversity loss 16: Promote peaceful and inclusive
societies for sustainable development , provide access to justice for all and build effective , accountable and inclusive
institutions at all levels 17: Strengthen the means of
implementation and revitalize the global partnership for sustainable development
Input Text: Save water for irrigation by using plastic films/mulches on
potato and vegetable fields; goal:

heading: C.2 Sample Hierarchical Prompt
text: Below is a sample prompt for SDG-Target prediction using the oracle mode with SDG-Goal 7.
You are an environmentalist that is knowledgeable on the 17 Sustainable Development Goals and 169 Targets. The following Input Text was classified as Sustainable Development Goal 7. Predict the Sustainable Development Target (target) out of the following options:
Goal 7 Targets: 7.1: By 2030, ensure universal access to
affordable , reliable and modern energy services
7.2: By 2030, increase substantially the share of renewable energy in the
global energy mix by 2030 7.3: By 2030, double the global rate of improvement in energy efficiency 7.a: By 2030, enhance international
cooperation to facilitate access to clean energy research and technology , including renewable energy , energy efficiency and advanced and cleaner fossil -fuel technology , and promote investment in energy infrastructure and clean energy technology
7.b: By 2030, expand infrastructure and upgrade technology for supplying modern and sustainable energy services for all in developing countries , in particular least developed countries , small island developing States , and land -locked developing countries , in accordance with their respective programmes of support

heading: C.3 Sample ICL Examples
text: Below are some sample ICL examples used for SDG-Target prediction, for predicting multiple SDG-Targets.
Input Text: <td><strong >Environmental Education and Capacity Building </ strong ></td> targets: 1.5, 3.d, 4.7, 5.a, 6.b, 7.a, 8.3, 9.a, 10.3, 11.b, 12.a, 13.3, 14.a, 15.a, 15.b, 16.b, 17.18 Input Text: Developing and using energy - saving construction materials and green materials in housing and commercial sectors. targets: 7.b, 9.4 Input Text: Additionally , the Cook
Islands is looking to embrace proven low carbon transport technologies and is currently exploring the most effective incentives for promotion of transition towards clean energy transportation.
targets: 7.a, 11.2 Input Text: Increase greenery through
tree plantation and management of gardens and parks.
targets: 11.7 Input Text: 10% of the total population
(0.8 million beneficiaries (25% are women) have increased resilience of food and water security , health , and well -being in PNG
targets: 2.4, 13.1

heading: C.4 Full Expert Prompt
text: Below is the full expert prompt that we used in our experiments. This was appended to the beginning of each prompt for the expert prompting experiments.
You are a climate policy expert specializing in understanding the complexities of climate systems and the impacts of human activities. Your knowledge spans climate science , mitigation , and adaptation strategies. You excel in analyzing research findings and developing policies that balance scientific evidence , political realities , and societal needs. Your expertise is instrumental in crafting effective and equitable climate policies at all levels , driving action towards a sustainable and resilient future.

heading: Future Work
text: Your expertise is instrumental in crafting effective and equitable climate policies at all levels , driving action towards a sustainable and resilient future