# Thematic Modeling of UN Sustainable Development Goals: A Comparative Meta-based Approach

Harris Partaourides, Emily Kouzaridi, Nicolas Tsapatsoulis, Constantinos Djouvas

<figure><img src="image_1.png" /></figure>

## To cite this version:

Harris Partaourides, Emily Kouzaridi, Nicolas Tsapatsoulis, Constantinos Djouvas. Thematic Modeling of UN Sustainable Development Goals: A Comparative Meta-based Approach. 19th IFIP International Conference on Artificial Intelligence Applications and Innovations (AIAI), Jun 2023, León, Spain. pp.557-568, 10.1007/978-3-031-34107-6_44. hal-05498060

HAL Id: hal-05498060

https://inria.hal.science/hal-05498060v1

Submitted on 6 Feb 2026

HAL is a multi-disciplinary open access archive for the deposit and dissemination of scientific research documents, whether they are published or not. The documents may come from teaching and research institutions in France or abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est destinée au dépôt et à la diffusion de documents scientifiques de niveau recherche, publiés ou non, émanant des établissements d'enseignement et de recherche français ou étrangers, des laboratoires publics ou privés.

<figure><img src="image_2.png" /></figure>

Distributed under a Creative Commons CC BY 4.0 - Attribution - International License
<figure><img src="image_3.png" /></figure>

ifip

International Federation for Information Processing

This document is the original author manuscript of a paper submitted to an IFIP
conference proceedings or other IFIP publication by Springer Nature. As such, there
may be some differences in the official published version of the paper. Such
differences, if any, are usually due to reformatting during preparation for publication or
minor corrections made by the author(s) during final proofreading of the publication
manuscript.
# Thematic Modeling of UN Sustainable Development Goals: A Comparative Meta-based Approach

Harris Partaourides✉, Emily Kouzaridi✉, Nicolas Tsapatsoulis✉, and Constantinos Djouvas✉

Cyprus University of Technology, Limassol, Cyprus
c.partaourides@cut.ac.cy

**Abstract.** To effectively monitor academic progress towards United Nations Sustainable Development Goals (SDGs), accurate and up-to-date information is crucial. Thematic modeling with keyword-based queries have emerged as a promising tool for clustering the large amounts of publications available in academic research databases. However, the effectiveness of these bibliometric queries depends on the choice of keywords used, and there is no standardized set of keywords for tracking academic progress towards the SDGs. In this study, we performed a comparative analysis of the most used keyword-based queries, assessing their advantages and disadvantages, and identifying gaps and redundancies. Based on our findings, we performed a meta-based approach to develop a new set of keywords that is both concise and thematically representative.

**Keywords:** Sustainable development goals · Keyword-based queries · Meta Approach · Semantic-based Matching

## 1 Introduction

The United Nations Sustainable Development Goals (SDGs) were adopted in 2015 as a universal call to action to end poverty, protect the planet, and ensure that all people enjoy peace and prosperity. The 17 goals and their associated 169 targets aim to address global challenges and are intended to be achieved by 2030. The list of United Nations (UN) SDGs is presented in Table 1. Most of the SDGs have a specific goal, except SDG 17 which is a cross-cutting SDG that aims to promote collaboration and cooperation among stakeholders at all levels to achieve the other 16 SDGs.

To effectively monitor progress towards the SDGs, accurate and up-to-date information is crucial. Thematic modeling with keyword-based queries has emerged as a promising tool for tracking academic progress towards the SDGs as a set of goals [3, 7] or individually [11]. These bibliometric queries enable the clustering of large amounts of publications that are publicly available in academic research databases, such as Scopus, Web of Science, and OpenAIRE. However,
<table><caption>**Table 1.** United Nations Sustainable Development Goals</caption><thead><tr><td>SDG</td><td>Name</td></tr></thead><tbody><tr><td>1</td><td>No Poverty</td></tr><tr><td>2</td><td>Zero Hunger</td></tr><tr><td>3</td><td>Good Health and Well-being</td></tr><tr><td>4</td><td>Quality Education</td></tr><tr><td>5</td><td>Gender Equality</td></tr><tr><td>6</td><td>Clean Water and Sanitation</td></tr><tr><td>7</td><td>Affordable and Clean Energy</td></tr><tr><td>8</td><td>Decent Work and Economic Growth</td></tr><tr><td>9</td><td>Industry, Innovation, and Infrastructure</td></tr><tr><td>10</td><td>Reduced Inequalities</td></tr><tr><td>11</td><td>Sustainable Cities and Communities</td></tr><tr><td>12</td><td>Responsible Consumption and Production</td></tr><tr><td>13</td><td>Climate Action</td></tr><tr><td>14</td><td>Life Below Water</td></tr><tr><td>15</td><td>Life On Land</td></tr><tr><td>16</td><td>Peace, Justice, and Strong Institutions</td></tr><tr><td>17</td><td>Partnerships for the Goals</td></tr></tbody></table>

the effectiveness of keyword-based queries highly depends on the selection of appropriate keywords [15].

At present, there is no standardized set of keywords for tracking progress towards the SDGs. In this context, several universities and organizations have constructed their own keyword sets such as Aurora's search queries for “Mapping Research Output to the SDGs” (AUR) [12], Auckland University SDG Keyword Mapping (AUC) [14], Elsevier's 2020 (ELS20) [4], 2021 (ELS21) [9], and 2022 (ELS22) [10].

In this study, we perform a comparative analysis between the most used keyword-based queries, present their advantages and disadvantages and through a meta-based approach create a new set of keywords that is concise, semantically relevant, and easily maintainable. The refined set of keywords related to sustainable development can greatly facilitate the evaluation and monitoring of academic progress towards specific targets and goals. Moreover, by using the derived set of keywords, researchers, policymakers, and practitioners can unlock new insights into the progress being made towards a more sustainable future. This can save time and effort while helping them make more informed decisions and take more effective action towards steering research [8].

## 2 Background and Literature Review

Keyword-based queries involve logically combining words and phrases that are representative of the goals (and targets) defined and associated with the United
<table><caption>**Table 2.** Text size and query execution time</caption><thead><tr><th rowspan="2">SDG</th><th colspan="2">AUR</th><th colspan="2">AUC</th><th colspan="2">ELS20</th><th colspan="2">ELS21/22</th></tr><tr><th>Size(KB)</th><th>Time(s)</th><th>Size(KB)</th><th>Time(s)</th><th>Size(KB)</th><th>Time(s)</th><th>Size(KB)</th><th>Time(s)</th></tr></thead><tbody><tr><td>1</td><td>2</td><td>&lt;5</td><td>3</td><td>&lt;5</td><td>&lt;1</td><td>&lt;5</td><td>409</td><td>10</td></tr><tr><td>2</td><td>2</td><td>&lt;5</td><td>5</td><td>&lt;5</td><td>1</td><td>&lt;5</td><td>195</td><td>25</td></tr><tr><td>3</td><td>5</td><td>&lt;5</td><td>8</td><td>&lt;5</td><td>3</td><td>&lt;5</td><td>224/226</td><td>20</td></tr><tr><td>4</td><td>2</td><td>&lt;5</td><td>6</td><td>&lt;5</td><td>2</td><td>&lt;5</td><td>1676</td><td>-</td></tr><tr><td>5</td><td>3</td><td>&lt;5</td><td>5</td><td>&lt;5</td><td>2</td><td>&lt;5</td><td>1030</td><td>25</td></tr><tr><td>6</td><td>1</td><td>&lt;5</td><td>7</td><td>&lt;5</td><td>2</td><td>&lt;5</td><td>216</td><td>11</td></tr><tr><td>7</td><td>1</td><td>&lt;5</td><td>6</td><td>&lt;5</td><td>2</td><td>&lt;5</td><td>378</td><td>35</td></tr><tr><td>8</td><td>3</td><td>&lt;5</td><td>6</td><td>&lt;5</td><td>2</td><td>&lt;5</td><td>86</td><td>21</td></tr><tr><td>9</td><td>3</td><td>&lt;5</td><td>5</td><td>&lt;5</td><td>1</td><td>&lt;5</td><td>341</td><td>25</td></tr><tr><td>10</td><td>2</td><td>&lt;5</td><td>4</td><td>&lt;5</td><td>1</td><td>&lt;5</td><td>250</td><td>30</td></tr><tr><td>11</td><td>2</td><td>&lt;5</td><td>7</td><td>&lt;5</td><td>1</td><td>&lt;5</td><td>135</td><td>22</td></tr><tr><td>12</td><td>2</td><td>&lt;5</td><td>6</td><td>&lt;5</td><td>2</td><td>&lt;5</td><td>95</td><td>22</td></tr><tr><td>13</td><td>2</td><td>&lt;5</td><td>6</td><td>&lt;5</td><td>2</td><td>&lt;5</td><td>148</td><td>22</td></tr><tr><td>14</td><td>3</td><td>&lt;5</td><td>5</td><td>&lt;5</td><td>1</td><td>&lt;5</td><td>1016</td><td>50</td></tr><tr><td>15</td><td>3</td><td>&lt;5</td><td>7</td><td>&lt;5</td><td>2</td><td>&lt;5</td><td>172</td><td>23</td></tr><tr><td>16</td><td>3</td><td>&lt;5</td><td>5</td><td>&lt;5</td><td>2</td><td>&lt;5</td><td>85</td><td>22</td></tr></tbody></table>

Nations' SDGs policy. Aurora's, Auckland's and Elsevier's are three of the most used query sets to track academic progress towards the SDGs. All of the query sets are publicly available and can be directly executed on the Scopus Advanced Search Tool to retrieve relevant publications. Table 2 provides an overview of the query text size in Kilobytes(KB) and the estimated time required to perform each query in seconds(s).

## 2.1 Aurora's Search Queries

To map research output to the SDGs, AUR [12] search queries were developed using a combination of sources: the official UN definition of the goals (targets and indicators), the Sustainable Development Solutions Network compiled list of keywords [1], and keyword-combination searches utilizing Boolean and proximity operators. The AUR set is carefully crafted a) to take full advantage of Scopus Advanced Search tool's functionalities, b) use combinations of keywords to minimize false positives and c) actively avoid the use of excluding keywords ("NOT" operator) to prevent the removal of true positives.

The set's performance was evaluated through a survey of 244 researchers conducted between October 2019 and January 2020 [13]. Newer versions of the set have been released over the years, incorporating additional keywords such as synonyms, antonyms, and academic terminology. The latest version, version 5, was released in July 2021.
## 2.2 Auckland's Search Queries

The University of Auckland SDG Keyword Mapping set was developed using text-mining techniques that involved multiple stages of data collection and analysis [14]. Firstly, an initial list of SDG-related keywords was generated using a similar approach to the AUR's search query methodology. Next, the abstracts of relevant publications available in the Scopus academic research database were collected using these keywords.

To identify relevant sequences of words, an n-gram model was applied to mine the abstracts of academic publications. The n-gram tokens were then scored based on their frequency and counts and ranked accordingly. High-ranking keywords were examined and manually reviewed to confirm their relevance to the corresponding SDG. Additionally, both author-provided and publication index keywords were evaluated based on their counts and frequencies.

It's worth noting that the AUC set does not include a specific query for SDG 17. The methodology used to create the set aimed to identify keywords related to specific SDGs and their targets, rather than focusing on the overarching theme of partnership and collaboration present in SDG 17.

## 2.3 Elsevier's 2020, 2021 and 2022 Search Queries

Elsevier has been actively supporting researchers and institutions in monitoring and demonstrating progress towards the SDGs. To achieve this, the company has been generating SDG search queries since 2018. In 2020, inspired by the earlier queries, Elsevier used a new approach to map publications to the SDGs. This new approach incorporated crowd-sourced responses (customer feedback) and has significantly increased the number of search terms used to define each SDG [4]. Moreover, they used a machine learning model to improve the recall by approximately 10%.

Specifically, an n-gram model was applied to mine the title, abstract, author's keywords, and journal subject areas and define the 50K most frequent features. This is followed by a logistic regression to predict the scores for each of the SDGs. In this context, the most confident model predictions (> .95) are retained as SDG-related publications.

Building on this success, Elsevier further improved the search query set in 2021, resulting in the capture of twice as many articles on average compared to the 2020 version while maintaining a precision rate of over 80% [9]. The mapping also had better overlap with SDG queries from other independent projects. The latest release in 2022, entails the same search queries and ML algorithm as the 2021 version, with an exception for SDG 3 (good health and well-being). This is because, in collaboration with university partners, Elsevier agreed to add COVID-related search terms to SDG 3 [10]. To simplify the comparative analysis we omitted ELS21 in favor of the updated ELS22. Similar to AUC, all Elsevier's sets do not include SDG 17.

It's important to note that Elsevier's search queries are also being used by the Times Higher Education as part of their Impact Rankings.
# 3 Methodology

As in the majority of empirical studies, the methodology we follow consists of four main steps: problem formulation, data collection, data analysis, and conclusion drawing based on the collected data. We begin this section by providing a mathematical formulation of the problem we are trying to solve, while each one of the remaining steps is analysed in a different subsection.

## 3.1 Problem Formulation

Let $\mathcal{D}$ be a dataset of scientific publications, e.g, Scopus, OpenAire, etc. Let also $\mathcal{Q}$ a terms set used to query the dataset $\mathcal{D}$, and $\mathcal{R}^{\mathcal{Q}}$ the set of retrieved results. Assuming that $T_i$ a scientific topic for which relevant publications do exist in $\mathcal{D}$ the problem of finding those publications is highly relevant to the query we execute.

Finding the publications that are relevant to each one of the SDGs in the $\mathcal{D}$ dataset is the key problem of the current study. For this purpose, we define each SDG as a different topic, i.e., $T_1$ refers to SDG 1, $T_2$ refers to SDG 2, etc. We also define $Q_i^j$ as the j-th term set used to query $\mathcal{D}$ aiming at topic $T_i$. In this context, we denote as $\mathcal{R}_i^{Qj}$ the results retrieved from the j-th query regarding the i-th topic.

Given a set of queries $S_i = \{Q_i^1, Q_i^2, ...\}$ targeting the i-th topic we aim to develop a meta query $Q_i^M$ which is based on the existing ones and provides more relevant results, being also easily interpretable by humans.

## 3.2 CUT's SDG Keyword Mapping

In Table 2 we observe that the extensive and detailed ELS22 entails computationally expensive queries; large file sizes and long server response times. In this context, a concise and thematically representative query set will not only benefit in terms of readability but also interpretability of the publications retrieved. Towards this end, we developed a new keyword set using ELS20 query set as a starting point, performed corrections and light improvements, and added keywords from AUC, and ELS22 [2]. Specifically, a small subset of the highest ranking keywords returned by the machine learning models of AUC and ELS22 was selected. Then the selected terms where evaluated under a semantic-based matching with the SDGs and excluding generic terms that produce false positives. This meta-based approach resulted in the Cyprus University of Technology: SDG Keyword Mapping (CUT). In Table 3 we present a subset of five keywords per SDG. The subset contains the keywords with the highest volume of publications returned. A full list of keywords and the associated queries of CUT set are publicly released<fnref n="1" />.

A qualitative evaluation based on interpretation was carried out to verify that the new keywords match the original SDG to which they belong. Since we

<footnote n="1">¹ https://github.com/Partaourides/CUT_SDGs_Keyword-Mapping.</footnote>
<table><caption>**Table 3.** Subset of added keywords</caption><thead><tr><th>SDG1</th><th>SDG2</th><th>SDG3</th><th>SDG4</th></tr></thead><tbody><tr><td>Child welfare</td><td>Agroforestry</td><td>Air pollution</td><td>Academic achievement</td></tr><tr><td>Low socioeconomic</td><td>Biocontrol</td><td>Covid-19</td><td>Academic performance</td></tr><tr><td>Social exclusion</td><td>Biological control</td><td>Healthcare</td><td>Scholarships</td></tr><tr><td>Social policy</td><td>Domestication</td><td>Pandemic</td><td>Trainees</td></tr><tr><td>Social security</td><td>Sustainable agriculture</td><td>Stroke</td><td>Vocational education</td></tr><tr><td>SDG5</td><td>SDG6</td><td>SDG7</td><td>SDG8</td></tr><tr><td>Feminist</td><td>Drinking water</td><td>Biodiesel</td><td>Economic activities</td></tr><tr><td>Lgbt</td><td>Irrigation</td><td>Fuel cells</td><td>Human capital</td></tr><tr><td>Marginalized</td><td>Water pollution</td><td>Power consumption</td><td>Job satisfaction</td></tr><tr><td>Sexism</td><td>Water resources</td><td>Wind power</td><td>Labor market</td></tr><tr><td>Std</td><td>Water supply</td><td>Wind turbines</td><td>Regional development</td></tr><tr><td>SDG9</td><td>SDG10</td><td>SDG11</td><td>SDG12</td></tr><tr><td>Infrastructure</td><td>Financial crisis</td><td>Building*</td><td>Energy saving</td></tr><tr><td>Manufacturing process</td><td>Health care access</td><td>Infrastructure</td><td>Energy use</td></tr><tr><td>Nanofabrication</td><td>Medicaid</td><td>Urban development</td><td>Landfill</td></tr><tr><td>Prototyping</td><td>Racism</td><td>Urban environment</td><td>Natural resource</td></tr><tr><td>Research and development</td><td>Social justice</td><td>Urban planning</td><td>Recycle</td></tr><tr><td>SDG13</td><td>SDG14</td><td>SDG15</td><td>SDG16</td></tr><tr><td>Co2</td><td>Aquaculture</td><td>Bioremediation</td><td>Community engagement</td></tr><tr><td>Global climate</td><td>Aquatic</td><td>Conservation</td><td>Community participation</td></tr><tr><td>Global temperature</td><td>Bloom</td><td>Environmental change</td><td>Human rights</td></tr><tr><td>Natural disasters</td><td>Conservation</td><td>Indigenous</td><td>National security</td></tr><tr><td></td><td>Hydrocarbons</td><td>Restoration</td><td>Participatory approach</td></tr></tbody></table>

had a small amount of data and a predefined list of keywords, a basic approach of qualitative analysis was selected to identify the broad topics or themes represented by the keywords. This involved manually reviewing the keywords and determining the common subject or subjects they represented. We looked for similarities or connections between the keywords and considered the context or domain to which they belonged.

The primary goal of the process was not to determine whether the additional keywords comprehensively covered all aspects of every SDG. Instead, the objective was to confirm whether the topics represented by the additional keywords were included in the original sub-targets listed on the website. This required a focused analysis of the additional keywords in relation to the relevant sub-targets. By comparing the topics covered by the keywords with the content of the sub-targets, we were able to assess whether the keywords provided meaningful additional insights into the relevant SDGs. While the process did not aim to provide comprehensive coverage of every aspect of every SDG, it was still an important step in ensuring that the additional keywords were aligned with the original sub-targets and contributed to a more complete understanding of the SDGs.

It is worth mentioning that this approach, while useful for gaining insights into a small set of data, can be subject to biases and limitations. A basic approach to qualitative analysis can be a valuable tool for gaining an initial understanding
of the content and themes of a small set of data. The methodology utilized to evaluate the appropriation of keywords is akin to the coding process in qualitative analysis. The coding process is a crucial step in qualitative analysis: The data are categorized and organized into meaningful themes. In a similar vein, the evaluation of keyword appropriation involves identifying and categorizing keywords into higher categories. The subjects covered by each SDG are then contrasted with these higher categories. In this particular case, the coding approach is deductive since the list of keywords is predefined and the framework for each SDG is already established. The use of a relatively limited number of keywords further reinforces the deductive approach [6]. Overall, the deductive coding process allows for a more structured analysis of the data, based on the established framework and keywords. This approach can provide insights into how the SDGs are perceived and addressed. Kovacs et al. [5] used also a simple manual form of content analysis on collected words for ordering them into higher categories according to their scientific field within psychology. One of the focuses of their study on mapping qualitative research in psychology was to explore the topics of the examined articles.

After conducting a thorough review of the lists, the results have indicated that the keywords associated with each of the SDGs are in alignment with the overall objective of their corresponding targets. This highlights the consistency and coherence in the language used to define and articulate the SDGs.

## 3.3 Operationalization

In the context of this study, the *D* dataset is the Scopus dataset, the topics searched *T*<sub>i</sub>, *i* = 1, .., 16 are the SDG topics, while *S*<sub>*i*</sub>, *i* = 1, ..., 16 are the query term sets reviewed in Sect. 2, i.e., *S*<sub>1</sub> = {*Q*<sub>1</sub><sup>AUC</sup>, *Q*<sub>1</sub><sup>AUR</sup>, *Q*<sub>1</sub><sup>ELS20</sup>, *Q*<sub>1</sub><sup>ELS22</sup>, *Q*<sub>1</sub><sup>CUT</sup>} are the query term sets developed by AUC, AUR, Elsevier (2020, 2022) and CUT targeting the SDG 1. For easy reference, by dropping the subscript index we mean the whole set of SDG query sets developed by each one of the compared approaches, i.e., *Q*<sup>AUR</sup> is the set of queries developed by AUR for all SDGs.

By $\mathcal{R}_i^{Q^j}$ we denoted the number of per year publications retrieved when executing the $Q_i^j$ query. Each row in Tables 4-8 is an instance of $\mathcal{R}_i^{Q^j}$ while by $\mathcal{R}^{Q^j}$ we denote the number of per year and per SDG publications. Thus, each one of the Tables 4-8 is an instance of $\mathcal{R}^{Q^j}$.

# 4 Data Collection and Comparative Analysis

Scopus Advanced Search tool was used to execute the queries and collect data for the years 2015–2022. The data collection was conducted in January 2023. The query results per year are presented in Tables 4, 5, 6, 7, and 8 for AUR, AUC, ELS20, ELS22<sup>2</sup>, and CUT, respectively. The query file sizes and server execution time of CUT are similar to AUR, AUC and ELS20.

<footnote n="2"><sup>2</sup> We couldn't retrieve the information for SDG4 on the ELS22 set. To avoid missing values we added the average of SDG4 between AUC and ELS20.</footnote>
<table><caption>**Table 4.** $\mathcal{R}^{\text{AUR}}$ from 2015 to 2022 (in thousands)</caption><thead><tr><th>SDG</th><th>2022</th><th>2021</th><th>2020</th><th>2019</th><th>2018</th><th>2017</th><th>2016</th><th>2015</th></tr></thead><tbody><tr><td>1</td><td>12.6</td><td>12.1</td><td>10.5</td><td>9.7</td><td>8.9</td><td>8.1</td><td>7.4</td><td>3.9</td></tr><tr><td>2</td><td>9.4</td><td>8.3</td><td>6.8</td><td>5.4</td><td>4.5</td><td>4.0</td><td>3.5</td><td>3.2</td></tr><tr><td>3</td><td>102.1</td><td>98.8</td><td>45.1</td><td>16.9</td><td>15.5</td><td>14.5</td><td>13.4</td><td>12.0</td></tr><tr><td>4</td><td>17.6</td><td>16.0</td><td>13.9</td><td>12.1</td><td>10.7</td><td>9.7</td><td>8.9</td><td>8.1</td></tr><tr><td>5</td><td>7.4</td><td>6.7</td><td>5.8</td><td>5.0</td><td>4.7</td><td>4.2</td><td>3.8</td><td>3.4</td></tr><tr><td>6</td><td>30.1</td><td>27.9</td><td>24.8</td><td>21.6</td><td>19.0</td><td>17.1</td><td>15.5</td><td>13.7</td></tr><tr><td>7</td><td>14.6</td><td>13.2</td><td>11.8</td><td>11.5</td><td>10.1</td><td>9.1</td><td>8.1</td><td>7.3</td></tr><tr><td>8</td><td>9.6</td><td>9.0</td><td>9.2</td><td>7.7</td><td>6.6</td><td>6.2</td><td>5.4</td><td>5.0</td></tr><tr><td>9</td><td>9.5</td><td>8.1</td><td>7.0</td><td>6.2</td><td>5.4</td><td>4.8</td><td>4.2</td><td>3.8</td></tr><tr><td>10</td><td>5.3</td><td>5.0</td><td>4.5</td><td>4.2</td><td>3.8</td><td>3.5</td><td>3.3</td><td>2.9</td></tr><tr><td>11</td><td>28.5</td><td>27.5</td><td>25.0</td><td>23.9</td><td>20.4</td><td>18.0</td><td>15.6</td><td>13.4</td></tr><tr><td>12</td><td>29.8</td><td>27.2</td><td>22.2</td><td>19.0</td><td>16.4</td><td>14.5</td><td>12.9</td><td>11.9</td></tr><tr><td>13</td><td>94.7</td><td>88.4</td><td>75.8</td><td>67.5</td><td>61.4</td><td>56.1</td><td>52.0</td><td>48.3</td></tr><tr><td>14</td><td>31.5</td><td>32.4</td><td>29.6</td><td>26.9</td><td>25.0</td><td>22.0</td><td>21.1</td><td>20.0</td></tr><tr><td>15</td><td>28.6</td><td>26.1</td><td>22.1</td><td>19.3</td><td>16.6</td><td>14.9</td><td>13.3</td><td>12.2</td></tr><tr><td>16</td><td>14.2</td><td>13.9</td><td>12.4</td><td>12.0</td><td>10.9</td><td>10.4</td><td>9.4</td><td>8.4</td></tr><tr><td>17</td><td>11.8</td><td>11.4</td><td>10.6</td><td>9.7</td><td>8.5</td><td>7.9</td><td>7.0</td><td>6.8</td></tr></tbody></table>

<table><caption>**Table 5.** $\mathcal{R}^{\text{AUC}}$ from 2015 to 2022 (in thousands)</caption><thead><tr><th>SDG</th><th>2022</th><th>2021</th><th>2020</th><th>2019</th><th>2018</th><th>2017</th><th>2016</th><th>2015</th></tr></thead><tbody><tr><td>1</td><td>24.8</td><td>24.3</td><td>22.2</td><td>21.1</td><td>19.7</td><td>18.4</td><td>16.5</td><td>15.9</td></tr><tr><td>2</td><td>126.3</td><td>125.9</td><td>109.7</td><td>96.9</td><td>88.4</td><td>81.0</td><td>77.9</td><td>74.8</td></tr><tr><td>3</td><td>1214.5</td><td>1221.1</td><td>1065.9</td><td>912.0</td><td>862.4</td><td>825.1</td><td>813.3</td><td>797.4</td></tr><tr><td>4</td><td>144.8</td><td>139.3</td><td>119.5</td><td>107.3</td><td>93.1</td><td>82.5</td><td>76.5</td><td>71.6</td></tr><tr><td>5</td><td>35.7</td><td>33.6</td><td>29.8</td><td>27.3</td><td>25.7</td><td>23.4</td><td>22.1</td><td>20.8</td></tr><tr><td>6</td><td>105.0</td><td>101.0</td><td>93.2</td><td>84.4</td><td>76.1</td><td>69.1</td><td>65.7</td><td>61.1</td></tr><tr><td>7</td><td>180.6</td><td>168.7</td><td>156.9</td><td>155.3</td><td>137.4</td><td>124.8</td><td>114.5</td><td>105.5</td></tr><tr><td>8</td><td>60.9</td><td>58.4</td><td>52.0</td><td>47.1</td><td>41.2</td><td>38.6</td><td>34.5</td><td>31.8</td></tr><tr><td>9</td><td>99.4</td><td>97.2</td><td>89.8</td><td>81.6</td><td>69.1</td><td>61.2</td><td>54.8</td><td>49.0</td></tr><tr><td>10</td><td>125.3</td><td>1232.0</td><td>109.2</td><td>95.9</td><td>88.4</td><td>82.8</td><td>77.2</td><td>72.3</td></tr><tr><td>11</td><td>205.8</td><td>197.3</td><td>180.2</td><td>166.2</td><td>146.5</td><td>131.6</td><td>121.2</td><td>109.1</td></tr><tr><td>12</td><td>145.6</td><td>137.9</td><td>124.8</td><td>116.8</td><td>101.9</td><td>92.4</td><td>82.5</td><td>74.9</td></tr><tr><td>13</td><td>99.5</td><td>89.2</td><td>76.1</td><td>67.0</td><td>60.7</td><td>55.9</td><td>51.3</td><td>47.5</td></tr><tr><td>14</td><td>71.4</td><td>71.5</td><td>65.5</td><td>59.6</td><td>55.0</td><td>49.2</td><td>47.9</td><td>45.2</td></tr><tr><td>15</td><td>156.3</td><td>155.1</td><td>137.3</td><td>125.3</td><td>113.2</td><td>102.5</td><td>97.4</td><td>91.5</td></tr><tr><td>16</td><td>70.1</td><td>70.1</td><td>65.5</td><td>62.6</td><td>59.4</td><td>57.3</td><td>51.9</td><td>46.9</td></tr></tbody></table>
This was followed by a correlation analysis to measure the degree of association between any two query sets. In our analysis, we consider the 16 SDGs as independent sample pairs and the years 2015–2022 as dependent values. Our findings are presented in Table 9, with statistical significance levels denoted by

<table><caption>**Table 6.** $\mathcal{R}^{\text{ELS20}}$ from 2015 to 2022 (in thousands)</caption><thead><tr><th>SDG</th><th>2022</th><th>2021</th><th>2020</th><th>2019</th><th>2018</th><th>2017</th><th>2016</th><th>2015</th></tr></thead><tbody><tr><td>1</td><td>3.6</td><td>3.6</td><td>3.2</td><td>2.9</td><td>2.6</td><td>2.5</td><td>2.3</td><td>2.2</td></tr><tr><td>2</td><td>35.3</td><td>33.7</td><td>28.5</td><td>24.8</td><td>22.0</td><td>19.3</td><td>17.6</td><td>16.3</td></tr><tr><td>3</td><td>939.9</td><td>964.3</td><td>861.3</td><td>744.2</td><td>701.4</td><td>672.2</td><td>662.1</td><td>645.5</td></tr><tr><td>4</td><td>8.9</td><td>8.5</td><td>7.5</td><td>6.6</td><td>6.1</td><td>5.2</td><td>5.0</td><td>4.5</td></tr><tr><td>5</td><td>13.7</td><td>12.6</td><td>10.6</td><td>6.5</td><td>8.7</td><td>7.8</td><td>7.0</td><td>6.3</td></tr><tr><td>6</td><td>14.7</td><td>14.3</td><td>13.1</td><td>11.7</td><td>10.6</td><td>9.3</td><td>8.8</td><td>7.9</td></tr><tr><td>7</td><td>122.0</td><td>112.7</td><td>102.2</td><td>100.4</td><td>87.1</td><td>78.5</td><td>69.7</td><td>62.5</td></tr><tr><td>8</td><td>35.0</td><td>32.4</td><td>27.9</td><td>25.5</td><td>21.6</td><td>19.8</td><td>17.3</td><td>15.7</td></tr><tr><td>9</td><td>13.7</td><td>12.4</td><td>11.7</td><td>10.6</td><td>9.1</td><td>8.1</td><td>7.3</td><td>6.7</td></tr><tr><td>10</td><td>14.9</td><td>14.1</td><td>12.5</td><td>12.1</td><td>11.1</td><td>10.6</td><td>9.6</td><td>9.0</td></tr><tr><td>11</td><td>46.1</td><td>46.3</td><td>42.3</td><td>39.6</td><td>34.6</td><td>29.7</td><td>25.0</td><td>21.6</td></tr><tr><td>12</td><td>36.2</td><td>32.0</td><td>26.6</td><td>23.4</td><td>20.1</td><td>17.9</td><td>15.5</td><td>14.0</td></tr><tr><td>13</td><td>65.7</td><td>60.7</td><td>51.8</td><td>45.4</td><td>4.1</td><td>37.1</td><td>35.0</td><td>32.9</td></tr><tr><td>14</td><td>30.6</td><td>30.6</td><td>27.4</td><td>24.9</td><td>22.6</td><td>20.4</td><td>19.6</td><td>18.5</td></tr><tr><td>15</td><td>34.3</td><td>34.1</td><td>30.2</td><td>27.6</td><td>25.5</td><td>22.6</td><td>21.4</td><td>19.8</td></tr><tr><td>16</td><td>46.4</td><td>46.3</td><td>41.8</td><td>42.7</td><td>41.1</td><td>39.8</td><td>35.8</td><td>31.9</td></tr></tbody></table>

<table><caption>**Table 7.** $\mathcal{R}^{\text{ELS22}}$ from 2015 to 2022 (in thousands)</caption><thead><tr><th>SDG</th><th>2022</th><th>2021</th><th>2020</th><th>2019</th><th>2018</th><th>2017</th><th>2016</th><th>2015</th></tr></thead><tbody><tr><td>1</td><td>19.2</td><td>17.8</td><td>15.8</td><td>15.0</td><td>13.7</td><td>13.1</td><td>11.8</td><td>10.8</td></tr><tr><td>2</td><td>53.3</td><td>49.9</td><td>42.4</td><td>36.6</td><td>32.5</td><td>29.1</td><td>26.6</td><td>25.0</td></tr><tr><td>3</td><td>613.3</td><td>606.1</td><td>521.1</td><td>416.4</td><td>392.4</td><td>374.0</td><td>365.6</td><td>350.8</td></tr><tr><td>4*</td><td>76.8</td><td>73.9</td><td>63.5</td><td>56.9</td><td>49.6</td><td>43.9</td><td>40.7</td><td>38.0</td></tr><tr><td>5</td><td>34.7</td><td>31.4</td><td>27.4</td><td>25.1</td><td>24.0</td><td>21.7</td><td>20.3</td><td>18.5</td></tr><tr><td>6</td><td>71.2</td><td>66.8</td><td>59.7</td><td>53.1</td><td>47.8</td><td>48.7</td><td>40.0</td><td>36.4</td></tr><tr><td>7</td><td>162.2</td><td>149.9</td><td>138.2</td><td>134.4</td><td>121.5</td><td>110.7</td><td>101.2</td><td>93.8</td></tr><tr><td>8</td><td>67.7</td><td>61.9</td><td>53.7</td><td>48.2</td><td>41.5</td><td>37.6</td><td>33.0</td><td>30.1</td></tr><tr><td>9</td><td>111.6</td><td>101.5</td><td>90.2</td><td>81.3</td><td>68.4</td><td>60.2</td><td>52.2</td><td>45.6</td></tr><tr><td>10</td><td>58.8</td><td>53.2</td><td>45.8</td><td>41.9</td><td>39.1</td><td>36.3</td><td>33.0</td><td>29.9</td></tr><tr><td>11</td><td>88.3</td><td>85.4</td><td>77.7</td><td>71.6</td><td>63.1</td><td>53.9</td><td>48.5</td><td>42.7</td></tr><tr><td>12</td><td>59.5</td><td>54.4</td><td>46.5</td><td>41.1</td><td>35.3</td><td>32.3</td><td>27.7</td><td>25.2</td></tr><tr><td>13</td><td>73.5</td><td>63.0</td><td>52.7</td><td>45.7</td><td>40.9</td><td>37.7</td><td>33.6</td><td>30.4</td></tr><tr><td>14</td><td>30.8</td><td>30.8</td><td>27.1</td><td>24.1</td><td>22.1</td><td>19.8</td><td>18.3</td><td>16.8</td></tr><tr><td>15</td><td>46.6</td><td>45.5</td><td>40.4</td><td>36.3</td><td>33.8</td><td>30.9</td><td>28.7</td><td>27.2</td></tr><tr><td>16</td><td>49.9</td><td>48.5</td><td>43.2</td><td>41.5</td><td>39.7</td><td>37.7</td><td>34.0</td><td>31.4</td></tr></tbody></table>
* for 0.05 and ** for 0.01. It's worth noting that a significance level of 0.01 indicates stronger evidence is needed to reject the null hypothesis. In this context, the analysis shows a high correlation between AUC, ELS20, ELS22, and CUT, suggesting that they can serve interchangeably as indicators for measuring SDG-related academic progress. However, there are substantial differences in the quantity and logical combinations of keywords used in each set, which hinders the explainability of the results and insights to make informed decisions. Among the three sets, ELS22 has the most extensive and extremely finetuned keyword combination for each SDG, while ELS20 is the most concise and readable. This can also be observed from the size and execution time of the queries presented in Table 2.

<table><caption>**Table 8.** $R^{\text{CUT}}$ from 2015 to 2022 (in thousands)</caption><thead><tr><th>SDG</th><th>2022</th><th>2021</th><th>2020</th><th>2019</th><th>2018</th><th>2017</th><th>2016</th><th>2015</th></tr></thead><tbody><tr><td>1</td><td>19.4</td><td>19.2</td><td>17.5</td><td>16.0</td><td>14.7</td><td>14.3</td><td>13.3</td><td>13.3</td></tr><tr><td>2</td><td>42.0</td><td>40.2</td><td>34.5</td><td>29.9</td><td>26.5</td><td>23.5</td><td>21.7</td><td>19.9</td></tr><tr><td>3</td><td>1071.3</td><td>1072.2</td><td>930.1</td><td>783.0</td><td>736.6</td><td>704.6</td><td>692.7</td><td>675.2</td></tr><tr><td>4</td><td>28.1</td><td>27.3</td><td>23.7</td><td>20.7</td><td>19.1</td><td>17.4</td><td>16.4</td><td>15.5</td></tr><tr><td>5</td><td>28.5</td><td>25.6</td><td>22.0</td><td>20.4</td><td>19.1</td><td>17.0</td><td>16.0</td><td>14.1</td></tr><tr><td>6</td><td>65.4</td><td>63.5</td><td>58.4</td><td>52.3</td><td>46.9</td><td>42.7</td><td>40.0</td><td>37.1</td></tr><tr><td>7</td><td>160.1</td><td>147.6</td><td>135.4</td><td>133.6</td><td>117.9</td><td>107.2</td><td>98.2</td><td>89.7</td></tr><tr><td>8</td><td>48.5</td><td>45.2</td><td>39.6</td><td>36.6</td><td>31.3</td><td>28.8</td><td>25.9</td><td>23.6</td></tr><tr><td>9</td><td>80.2</td><td>76.2</td><td>70.4</td><td>67.4</td><td>59.3</td><td>53.7</td><td>49.7</td><td>45.8</td></tr><tr><td>10</td><td>55.3</td><td>52.5</td><td>46.0</td><td>40.7</td><td>38.0</td><td>36.4</td><td>33.7</td><td>31.2</td></tr><tr><td>11</td><td>64.2</td><td>63.6</td><td>58.6</td><td>55.8</td><td>48.1</td><td>41.9</td><td>36.2</td><td>31.2</td></tr><tr><td>12</td><td>57.9</td><td>52.7</td><td>45.8</td><td>41.6</td><td>36.6</td><td>33.2</td><td>29.7</td><td>26.7</td></tr><tr><td>13</td><td>66.7</td><td>61.2</td><td>52.1</td><td>45.7</td><td>41.4</td><td>37.3</td><td>35.2</td><td>33.1</td></tr><tr><td>14</td><td>35.8</td><td>35.6</td><td>32.1</td><td>29.1</td><td>26.5</td><td>23.8</td><td>22.9</td><td>21.5</td></tr><tr><td>15</td><td>38.3</td><td>37.9</td><td>33.5</td><td>30.7</td><td>28.4</td><td>25.2</td><td>23.9</td><td>22.0</td></tr><tr><td>16</td><td>60.7</td><td>59.8</td><td>54.7</td><td>55.4</td><td>53.0</td><td>51.5</td><td>46.8</td><td>41.3</td></tr></tbody></table>

<table><caption>**Table 9.** Correlation Query Results (N=16 independent pairs)</caption><thead><tr><th></th><th>AUC</th><th>AUR</th><th>ELS20</th><th>ELS22</th><th>CUT</th></tr></thead><tbody><tr><td>AUR</td><td>1.000</td><td>0.339</td><td>0.393</td><td>0.390</td><td>0.392</td></tr><tr><td>AUC</td><td>0.339</td><td>1.000</td><td>0.893*/**</td><td>0.891*/**</td><td>0.898*/**</td></tr><tr><td>ELS20</td><td>0.393</td><td>0.893*/**</td><td>1.000</td><td>0.977*/**</td><td>0.997*/**</td></tr><tr><td>ELS22</td><td>0.390</td><td>0.891*/**</td><td>0.977*/**</td><td>1.000</td><td>0.986*/**</td></tr><tr><td>CUT</td><td>0.392</td><td>0.898*/**</td><td>0.997*/**</td><td>0.986*/**</td><td>1.000</td></tr></tbody></table>
<table><caption>Table 10. AUR correlations per year</caption><thead><tr><th></th><th>2022</th><th>2021</th><th>2020</th><th>2019</th><th>2018</th><th>2017</th><th>2016</th><th>2015</th></tr></thead><tbody><tr><th>AUC</th><td>0.686</td><td>0.362</td><td>0.361</td><td>0.011</td><td>0.014</td><td>0.022</td><td>0.024</td><td>0.023</td></tr><tr><th>ELS20</th><td>0.704</td><td>0.715</td><td>0.386</td><td>0.035</td><td>-0.008</td><td>0.048</td><td>0.052</td><td>0.051</td></tr><tr><th>ELS22</th><td>0.665</td><td>0.673</td><td>0.338</td><td>-0.013</td><td>-0.007</td><td>0.005</td><td>0.007</td><td>0.008</td></tr><tr><th>CUT</th><td>0.683</td><td>0.695</td><td>0.359</td><td>0.004</td><td>0.011</td><td>0.019</td><td>0.024</td><td>0.024</td></tr></tbody></table>

AUR query formulation being the most stringent with the definition of the SDGs has the least correlation with the others. However, a closer examination of the yearly correlation data (presented in Table 10) reveals a recent substantial increase in correlation values. This may be attributed to growing SDG awareness among academia, as well as a trend of authors incorporating SDG-related keywords in their work.

# 5 Conclusion and Future Work

In conclusion, we conducted a comparative analysis of various keyword-based approaches used to quantify academic progress towards achieving the SDGs. Our study revealed that existing keyword sets, such as ELS22, can be extensive and elaborate, which may pose challenges in terms of readability and interpretability of results. To address this issue, we created a new keyword set that is concise and readable while being comparable to the extensive and elaborate ELS22 set. By using a simplified list of keywords, our approach offers better explainability of results, which can assist policymakers and stakeholders in making informed decisions. Moreover, our approach is straightforward to apply to any academic research databases and SDG-oriented applications, making it a valuable tool for researchers and practitioners alike.

In addition to the current findings, there are a few areas for future work that could further improve our approach. Firstly, we aim to further simplify the keywords list to make it even more concise while remaining representative towards the SDGs. This could involve exploring the relationships between the existing keywords and identifying redundant or overlapping terms. Secondly, it would be valuable to apply our approach to different academic research databases to evaluate its generalizability and potential limitations. Thus, conducting a comparative analysis of different databases using our approach could help identify any potential issues and refine the keywords further. Finally, we aim to implement the queries on a specialized dashboard for academic policymakers and assess their impact on strategic decision-making.

Overall, the applicability and readability of the derived keyword-based set can assist the ongoing efforts to achieve the United Nations SDGs by 2030.
**Acknowledgements.** The work presented in the current study was undertaken in the framework of the IBA-SwafS-Support-2-2020 project "EUt- EXTRAS: European University of Technology - Experimentation to Transform Research Activities and Steering" under the contract H2020-IBA-SwafS-Support-2-2020/101035812.

## References

1. Compiled list of SDG keywords by Monash university and SDSN Australia/Pacific. https://ap-unsdsn.org/wp-content/uploads/2017/04/Compiled-Keywords-for-SDG-Mapping Final 17-05-10.xlsx (2017)

2. Azad, H.K., Deepak, A.: Query expansion techniques for information retrieval: a survey. In. Process. Manag. **56**(5), 1698–1735 (2019)

3. Hassan, S.U., Haddawy, P., Zhu, J.: A bibliometric study of the world's research activity in sustainable development and its sub-areas using scientific literature. Scientometrics **99**, 549–579 (2014)

4. Jayabalasingham, B., Boverhof, R., Agnew, K., Klein, L.: Identifying research supporting the united nations sustainable development goals. Mendeley Data **1**, 1 (2019)

5. Kovács, A., Kiss, D., Kassai, S., Pados, E., Kaló, Z., Rácz, J.: Mapping qualitative research in psychology across five central-eastern european countries: Contemporary trends: A paradigm analysis. Qualitative Research in Psychology (2019)

6. Linneberg, M.S., Korsgaard, S.: Coding qualitative data: a synthesis guiding the novice. Qual. Res. J. **19**(3), 259–270 (2019)

7. Meschede, C.: The sustainable development goals in scientific literature: A bibliometric overview at the meta-level. Sustainability **12**(11), 4461 (2020)

8. Olawumi, T.O., Chan, D.W.: A scientometric review of global research on sustainability and sustainable development. J. clean. Prod. **183**, 231–250 (2018)

9. Rivest, M., Kashnitsky, Y., Bédard-Vallée, A., Campbell, D., Khayat, P., Labrosse, I., Pinheiro, H., Provençal, S., Roberge, G., James, C.: Improving the scopus and aurora queries to identify research that supports the united nations sustainable development goals (sdgs) 2021. Mendeley Data 2 (2021)

10. Roberge, G., Kashnitsky, Y., James, C.: Elsevier 2022 sustainable development goals (sdg) mapping (2022)

11. Sweileh, W.M.: Bibliometric analysis of scientific publications on "sustainable development goals" with emphasis on "good health and well-being" goal (2015–2019). Globalization and health **16**, 1–13 (2020)

12. Vanderfeesten, M., Otten, R., Spielberg, E., Kullman, L.: Search queries for "mapping research output to the sustainable development goals (sdgs)" v4.0 (2020). https://doi.org/10.5281/zenodo.3817443

13. Vanderfeesten, M., Spielberg, E., Gunes, Y.: Survey data of "mapping research output to the sustainable development goals (sdgs)" (2020). https://doi.org/10.5281/zenodo.3813230

14. Wang, W., Kang, W., Mu, J.: Mapping research to the sustainable development goals (sdgs) (2023)

15. Webber, W.: Evaluating the effectiveness of keyword search. IEEE Data Eng. Bull. **33**(1), 54–59 (2010)