SDG target detection in environmental reports using Retrieval-augmented Generation with LLMs (2024)

Abstract: With the consolidation of Large Language Models (LLM) as a dominant component in approaches for multiple linguistic tasks, the interest in these technologies has greatly increased within a variety of areas and domains. A particular scenario of information needs where to exploit these approaches is climate-aware NLP. Paradigmatically, the vast manual labour of inspecting long, heterogeneous documents to find environment-relevant expressions and claims suits well within a recently established Retrieval-augmented Generation (RAG) framework. In this paper, we tackle dual problems within environment analysis dealing with the common goal of detecting a Sustainable Developmental Goal (SDG) target being addressed in a textual passage of an environmental assessment report. We develop relevant test collections, and propose and evaluate a series of methods within the general RAG pipeline, in order to assess the current capabilities of LLMs for the tasks of SDG target evidence identification and SDG target detection.

heading: 1 Introduction
text: A series of Sustainable Development Goals (SDGs) were established by experts in the United Nations, as a reference framework with respect to which guide the progress of human activities, altogether oriented to the common good (Del Campo et al., 2020). According to their respective legal requirements, practitioners in the area of environmental assessment (e.g., professional assessors, developers, authorities) have to incorporate this framework in multiple spheres. In particular, the activities, impacts and mitigation measures described in environmental assessment documents are increasingly required to report how they address one or more SDGs; especially SDG targets, these being focused, actionable subgoals within a given SDG. Identifying textual passages relevant in addressing an SDG target of interest then becomes a fundamental
problem in the practice of environmental assessment. This problem naturally lends itself within an approach based on obtaining an initial selection of passage candidates, and then apply onto this a more advanced detection technique to refine the decisions about relevance to the target. The recently established Retrieval-augmented Generation (RAG) framework (Lewis et al., 2020) embraces this basic approach and couples it with a text generation component powered by Large Language Models (LLMs), the dominant technology in NLP that shows state-of-the-art performance in multiple tasks (Radford et al., 2019; Touvron et al., 2023).
Figure 1 depicts an overview of the two tasks that we address in this work, with examples of SDG targets and excerpts of environmental impact assessment (EIA) reports. Specifically, these dual tasks couple with each other in the need of practitioners in EIA for finding instances of correspondence between the information spaces of SDG targets and textual evidence in specialized reports. We contribute by developing a test collection for each of these two tasks. We also propose and evaluate a series of experimental configurations for each of the RAG components, in order to assess the current capabilities of LLMs for these paradigmatic tasks in climate-aware NLP.
Our test collection and related resources developed in this study are made publicly available in a repository at https://bit.ly/
241
climatenlp-sdg-target-detection.

heading: 2 Related Work
text: Recently, the area of climate-aware natural language processing has decisively emerged led by a broad interest of developing information access methods to strengthen awareness in phenomena within climate change, as well as to process climate-related data in specific tasks within this domain. For example, Hershcovich et al. (2022) introduce a climate performance model card to summarize the impact of the experimentation corresponding to a scientific work in NLP research. Some of the related literature focuses on designing and analyzing methods for extracting climatecentric information, for example, to answer questionnaires (Spokoyny et al., 2023) and to detect climate-relevant claims in documents (Stammbach et al., 2023). Works like the ones by Bingler et al. (2022) and Schimanski et al. (2024) reveal the capabilities of well-established language models in communicating around climate awareness. Other lines of research in this area have produced fundamental resources, including language models such as ClimateBert (Webersinke et al., 2022) and ClimateGPT (Thulke et al., 2024), as well as systems like ChatClimate (Vaghefi et al., 2023) and resources like EIA-centric ontologies (Nielsen et al., 2023; Garigliotti et al., 2023), to power approaches for a variety of tasks.
An ever-increasing dominant technology in NLP, LLMs store vast amounts of information implicitly in their of billions of parameters trained on large general-purpose corpora, which allows them to perform as state of the art in many tasks such as text classification, textual entailment and question answering (QA) (Radford et al., 2019; Touvron et al., 2023). Yet, for many domain-specific scenarios, a framework like Retrieval-augmented Generation (RAG) (Lewis et al., 2020) becomes convenient as it allows to extend the LLM capabilities with additional explicit knowledge as context where to get the generated answer from. Moreover, against frequent hallucinations, it useful to be able to verify that the claims that occur in a text generated by such a model are truthful (Liu et al., 2023a; Menick et al., 2022). Within the research on QA, a foundational task for any application of LLMs, several related problems such as attribution in question answering (Bohnet et al., 2023), evidentiality-guided generation (Asai et al., 2022), verifiability of gen-
eration (Liu et al., 2023a), and factuality in summarization (Liu et al., 2023b) aim for operationalizing such a verification.
In the tasks we address, we treat the input (an SDG target in EI or an textual excerpt in TD) as a question for which, after augmented with retrieved items, an LLM must generate an answer with the correct outputs (identifiers of report passages or SDG targets, respectively) among the ones provided in the prompt.

heading: 3 Methodology

heading: 3.1 Problems
text: Given (i) an SDG target and (ii) a set of one or more passages from environmental impact assessment (EIA) reports, the task of SDG Target Evidence Identification (or EI task) consists in determining whether any of the passages is a relevant evidence where the content of the target is addressed. We instantiate this problem by requesting a method, specifically an Large Language Model (LLM), to generate an answer (to a question asking for deciding which passage(s) are relevant) with the correct passages, each referred to by a unique string identifier also provided in the generation prompt.
We assume the dual task of detecting SDG targets in a passage to be defined as follows. Given (i) a passage from an EIA report, and (ii) a set of one or more SDG targets, the task of SDG Target Detection (or TD task) consists in determining whether the content of any of the targets in the set is being addressed in the passage. Similar to EI setting, we request an LLM to generate an answer with the correct targets, each referred to by a unique string identifier provided in the prompt.
These two tasks together encompass an operationalization of typical dynamics in the practice of environmental analysis where a professional assessor aims to find correspondences between SDG targets of interest and textual evidence in reports.

heading: 3.2 Approach
text: We approach each of these two tasks with a series of methods within Retrieval-augmented Generation (RAG), a framework used, among others, by relevant benchmark literature for question answering (Gao et al., 2023). Each method consists of the same pipeline of three distinguished components: (i) retrieval, which, for each input question requested as query, produces a list of ranked passages from the indexed passage collection; (ii) then,
augmentation, where each test instance made of the question and (a subset of) retrieved passages are aggregated in a well-designed prompt that also captures the criteria of relevance to be required for judgement to an LLM; and (iii) a third component, the LLM-based generation of the answer required for each instance. Within this RAG umbrella framework, for example, for the problem of SDG evidence identification (EI), the SDG target becomes a query for which to identify relevant textual passages from a collection. Symmetrically, for the problem of SDG target detection (TD), a textual passage, or excerpt, is treated as a query for which to find the SDG targets, if any, among selected suitable target candidates; these targets are “passages” themselves within RAG since they are retrieved —from an indexed collection as candidates— for the input excerpt and subsequently post-processed through the augmentation and generation stages.
We experiment with corresponding parameters of interest on a vanilla setting of each component, and evaluate all the respective performances. We refer with ‘method’ to each instantiation of this RAG-based approach in a particular parameter configuration.

heading: 3.3 Research Questions
text: We conduct experimentation over the test collections with an ensemble of methods, in order to answer the following research questions.
• RQ1: How do the retrieval component affect the RAG performance? • RQ2: What is the impact of the different augmentation strategies? • RQ3: How does RAG perform with each LLM chosen for generation?

heading: 4 Experimental Setup

heading: 4.1 Datasets
text: The Ministry of Climate of the Republic of Estonia has made publicly available a series of environmental reports corresponding to projects developed in the country and other European countries nearby.1 We select 33 reports from this public website and post-process their PDF files to obtain a collection of passages, or contexts, as follows. First, the textual content of each file is extracted with the PyMuPDF tool 2. Then, further replacements in the text are 1https://kliimaministeerium.ee/ piiriulene-moju-hindamine#piiriulese-moju-hind 2https://pymupdf.readthedocs.io/en/latest/
made for distinguished characters so that to transform each sentence that is broken into multiple lines as appearing in the PDF, and recover each contiguous sentence. From these, we only retain every sentence made of at least 5 words; this allows to remove spurious content that is wrongly processed as a valid sentence. Finally, we select a random passage length (between 3 and 5 sentences) for each page, and chunk the content of that page into passages of that length, possibly with shorter trailing passages. Each passage is assigned a unique identifier, with respect to which is then indexed during the first stage of the RAG framework for the EI task. This identifier is the one requested to be in the answer generated by an LLM to refer to each passage that the LLM considers to be relevant to the given SDG target. We use unique random alphanumeric strings as identifiers aiming to avoid allowing that the LLM may hallucinate typical reference markers such as natural numbers [1], [2], etc. The obtained collection comprises 16,474 passages.3
We also select 30 SDG targets, a considerable subset of the 157 targets available within the SDG framework. The selected targets are considered more relevant to the kind of environmental assessment practice of our interest, and so more likely to be addressed in them. Specifically, each target belongs to one of the following SDGs: Clean water and sanitation (SDG 6), Affordable and clean energy (SDG 7), Industry, innovation and infrastructure (SDG 9), Sustainable cities and communi-
3The full list of links to the PDF documents for the reports, as well as the postprocessing of the files into the final passage collection, are made publicly available in our repository at https://bit.ly/ climatenlp-sdg-target-detection.
ties (SDG 11), Responsible consumption and production (SDG 12), Climate action (SDG 13), Life below water (SDG 14), and Life on land (SDG 15). The full list of 30 selected SDG targets is presented in Fig. 2. After removing from it any of the temporal phrases like “by 2030” that are common to most SDG targets, the textual description of each of these selected targets becomes a pseudodocument, a “passage” by itself. The collection of these targets-as-passages is indexed, and retrieved against, during the RAG retrieval stage for the TD task. From the set of selected SDG targets and the passage collection, we obtain a two datasets, each per task, to evaluate the performance of our proposed methods. The test collection for the evidence identification task consists of manual annotations for the “yes”/“no” binary relevance judgement of a passage with respect to a target. The test collection comprises the 30 SDG targets, each annotated for 6 passages (the top 3 retrieved contexts for each of the two retrieval methods). Similarly, we build the test collection for the target detection task, by manually judging the binary relevance of a retrieved target w.r.t. an excerpt, for 10 “passages” (the top 5 retrieved targets for each of the two retrieval methods).
Both test collections are also made publicly available in our repository.4

heading: 4.2 Experimental Parameters
text: Retrieval component. For the EI task, from the index built to store the uniquely identified passages, we retrieve the top 3 results for every SDG target with each of both methods, traditional lexical matching (lexical, for short) and learned dense retrieval (dense, for short). For the TD task, we first build an index of SDG targets as passages, after each being assigned a random unique ID (this ID “masks” the plain number.subnumber ID format, as it is useful later for an augmentation configuration, where the LLM will not be made aware that the passages in the prompt are indeed SDG targets). We then retrieve top 5 targets from this index per each excerpt as query, again with both lexical and dense methods. We perform retrieval with the wellestablished library Pyserini.5
Augmentation component. Through prompt engineering, we design a prompt that requests the LLM to produce the answer mentioning the cor-
4https://bit.ly/climatenlp-sdg-target-detection 5https://github.com/castorini/pyserini
rect relevant passages in the desired format, which explicitly requires to give a concise answer and only if knowing it. Tables 1 and 2 show the actual prompt templates used for each task in one of our experimental configurations, ‘SDG-explicit’, where there is an explicit mention to the target being part of the SDG framework. The SDG-implicit prompt version is obtained from the explicit one by performing few replacements that mask an SDG target (as query in the EI task; as passage in the TD task) as being an environmental policy. For example, the SDG-implicit prompt for EI task is obtained from the prompt in Table 1 by replacing (i) “Sustainable Development Goal (SDG) target” by “environmental policy” in the prompt header, (ii) “SDG TARGET” by “ENVIRONMENTAL POLICY” in the field of the prompt footer where the SDG target is declared, and “SDG target” by “policy” in the question field by the end of the footer. The replacements to obtain a SDG-implicit prompt for the TD task are similar, with the additional detail of replacing each original target ID by its random unique identifier.
After observations about the phenomenon of an LLM possibly answering correctly most likely due
to learning the pattern about the passages in the prompt —being listed in the same order as the retrieved ranking—, we experiment with an alternative random order of contexts.
Generation component. We generate answers by prompting established LLMs. Specifically, we use a family of open LLMs such as Llama2 (Touvron et al., 2023) and a prominent model of the GPT platform, GPT3.5 (Radford et al., 2019). We also experiment with ClimateGPT, a family of LLMs obtained by fine-tuning a corresponding Llama2 model over corpora of documents within the climate change domain.
Generation with Llama2 and ClimateGPT is performed by inference with HuggingFace transformers library, while for GPT we access via the OpenAI API.
Summary. Our experimental parameters are:
• (Retrieval) Method: lexical or dense. • (Augmentation) Prompt: SDG-explicit or
SDG-implicit. • (Augmentation) Number of examples: 1 or 2. • (Augmentation) Order of passages: as given
by the retrieval ranking, or random.
• (Generation) LLM: open (Llama2-13b, Llama2-13b-ch, ClimateGPT-13b) or closed (ChatGPT —gpt-3.5-turbo-0125—).

heading: 4.3 Evaluation Metrics
text: For each task, we evaluate the correctness of a method by applying standard retrieval metrics of precision and recall with respect to the retrieved passage set (all the passage identifiers mentioned in the generated answer) and the relevant passage set (the set of all the known relevant passages such that they appear among the contexts provided in the prompt). We remind that in the TD task, the SDG targets to be identified for a given EIA excerpt are considered to be the passages in the RAG framework. For a given method, we report the average performance across all the instances in the test collection of each task, i.e., across the 30 SDG targets for the EI task and across the 50 EIA excerpts for the TD task.

heading: 5 Results and analysis
text: Throughout this section, Tables 3 and 4 present the results for all the configurations in our experimentation. (The corresponding output files with the full RAG results for all methods are made publicly available in our repository.6)

heading: 5.1 RQ1: Retrieval component
text: In our experimentation, the possible impacts of the retrieval stage are centered in the retrieval method: lexical or dense. Firstly, in the EI task, lexical retrieval leads to the best performances when combined with GPT3.5 or Llama2-13b-chat, across all metrics, and all augmentation strategies (number of examples and prompt version). Results with ClimateGPT-13b are split between the method setting, with more tendency to prefer dense retrieval, and mostly small changes across the parameter for the number of examples in prompt.
Secondly, in the TD task, we observe that, when using the SDG-explicit prompts, the precision measurements with ChatGPT are similar for a given setting and split for the number of examples, while its recall favours the dense retrieval method. Llama2-13b-chat also mostly changes between oneand two-example setting, regardless of the SDGexplicit or implicit prompt version. ClimateGPT13b always performs best with lexical retrieval.
6https://bit.ly/climatenlp-sdg-target-detection
The scenarios where lexical retrieval is favoured are possibly favoured by few key words that boost the correct matching during retrieval as they are very distinctive for a target and/or passage, which
gets less distinctive when combined by dense retrieval with the semantics of other words. Examples of these key words found in our data are “overfishing” (strong signal for SDG target 14.4),
“acidification” (for target 14.3), “transport” (for target 11.2), “alien” (for target 15.8 about invasive species).
In the SDG target detection task, we observe
that the the most frequent relevant targets belong to SDG 14 (about marine protection), which makes sense as most of the base reports where passages are taken describe aspects of environments in re-
gions around the Baltic Sea. It is followed in frequency by SDG 15 (terrestrial and inland freshwater ecosystems, forests), and with clearly less frequency by SDGs 7 (energy), 9 (infrastructure), 11 (housing, transportation), and 12 (waste, resources).

heading: 5.2 RQ2: Augmentation component
text: Results from the ablation of the augmentation component can be summarized as follows. Firstly, we analyze the impact of the order of the passages in the prompt. For the EI task, the order varies a lot w.r.t. other parameter settings in the SDGexplicit prompt configurations, whereas with the implicit prompt version, most cases favour the order by ranking. For the TD task, variations with SDG-explicit prompt version persist, while it varies slighlty less with implicit prompt and in many cases favouring order by ranking.
Secondly, we discuss the influence of the number of examples in the prompt. In the EI task, having two examples is mostly beneficial for Llama2-13bchat and ChatGPT, while it harms ClimateGPT-13b performances and largely hurts Llama2-13b. In the TD task, the trends are similar for the Llama2 models but for ClimateGPT-13b and ChatGPT the results are mixed, with cases of clear disadvantage with more examples in the prompt.

heading: 5.3 RQ3: Generation component
text: Across both tasks and their respective configurations, we verify as expected that ChatGPT is the best performing LLM in several settings. A general pattern for the EI task is that GPT performs best in both metrics when only one example is provided in the prompt, followed by Llama2-13b-chat; and that this gets inverted as Llama2-13b-chat is the best performing in the two-examples setting. The base model Llama-13b performs very close to ClimateGPT-13b in very few scenarios, but the differences become clearer in favour of ClimateGPT13b in the configurations with two examples.
For the TD task, GPT3.5 is the best performing LLM for both SDG-explicit and implicit prompt versions in the precision measurements. In turn, ClimateGPT-13b dominates in recall and clearly over Llama2-13b-chat for SDG-explicit prompt, but splits the best recall with Llama2-13b-chat in SDG-implicit, between one- or two-example settings, with Llama2-13b-chat overall closer.

heading: 5.4 Summary of observations
text: As a conclusive reiteration of our observations, we mention the following main remarks. (1) The EI task is best addressed with ChatGPT prompted with contexts obtained via lexical retrieval. (2) The TD tasks gets best precision-oriented performance when using ChatGPT over densely retrieved passages, while for best recall, it does with ClimateGPT over lexically retrieved passages. (3) In both tasks, most often the ranking in which passages where retrieved is the same order in which to list the passages in the prompt during augmentation. (4) The exact convenient number of examples in few-shot generation vary due to the complexity of the notion of a passage addressing an SDG target, and depends on the actual example(s) being considered.

heading: 6 Conclusions, Limitations, and Future Work
text: In this work, we study two dual problems on environmental analysis as a mean to approach towards the automatization of knowledge-intensive, timeconsuming tasks in the practice of assessing environmental impact in reports and its correspondence with the recent developments around SDGs. Specifically, we propose and assess several methods within the RAG framework powered by LLMs.
Our work approaches a paradigmatic scenario of environmental analysis, yet it is still limited in its capabilities to identify evidence and detect targets. On the one hand, the selected targets cover a meaningful part of the SDGs scope in regards to EIA, yet there are more SDGs and targets that could be considered. On the other hand, the collection of reports where the EIA passages come from suits well as information source for our experimentation, yet it is centered on particular regions of Europe and so our study fails to capture phenomena about other environments and their corresponding SDG targets of relevance. Furthermore, our data annotation is conducted with caution and good faith but it could present cases where the judgement could be different, especially as the concept of “addressing an SDG (target)” is already not exact in the literature and the EIA practices described in the reports often take advantage of these uncertainties.
In future work, we plan to further study the duality of these two tasks by approaching environmental analysis with a method were each task retrofits the other one. In this way, for example, a textual
passage identified via EI for an SDG target can be the input of a subsequent TD stage to possibly expand the space of targets of interest for that EIA report, as well as exploiting relations between passages in the same report.
Another line of research is experimenting with the usage of a claim detector, this is, a dedicated model for identifying climate-aware claims in text, such as the one developed by (Stammbach et al., 2023). This component could complement the retrieval stage to improve the selection of passages that are finally fed into the LLM during generation.
A third possible area of work corresponds to automatically labeling larger volumes of test instances with an LLM as assessor, which could extend the evaluation space, as well as allow for experimenting with fine-tuning a base pre-trained model with these instances. In a similar fashion, a fourth direction would investigate the automatic assessment, also via LLM, of correctness for a predicted result. Such an assessment would be validated by observing the inter-annotator agreement with manual assessments in a sample of the test collection.

heading: Acknowledgments
text: This paper is part of NRF project 329745 Machine Teaching For XAI.

heading: Future Work
text: In future work, we plan to further study the duality of these two tasks by approaching environmental analysis with a method were each task retrofits the other one. In this way, for example, a textual
passage identified via EI for an SDG target can be the input of a subsequent TD stage to possibly expand the space of targets of interest for that EIA report, as well as exploiting relations between passages in the same report. Another line of research is experimenting with the usage of a claim detector, this is, a dedicated model for identifying climate-aware claims in text, such as the one developed by (Stammbach et al. , 2023). This component could complement the retrieval stage to improve the selection of passages that are finally fed into the LLM during generation. A third possible area of work corresponds to automatically labeling larger volumes of test instances with an LLM as assessor, which could extend the evaluation space, as well as allow for experimenting with fine-tuning a base pre-trained model with these instances. In a similar fashion, a fourth direction would investigate the automatic assessment, also via LLM, of correctness for a predicted result. Such an assessment would be validated by observing the inter-annotator agreement with manual assessments in a sample of the test collection