# Value Learning for Value-Aligned Route Choice Modeling via Inverse Reinforcement Learning

Andrés Holgado-Sánchez, Javier Bajo, Holger Billhardt, Sascha Ossowski,

Joaquín Arias

► To cite this version:

Andrés Holgado-Sánchez, Javier Bajo, Holger Billhardt, Sascha Ossowski, Joaquín Arias. Value Learning for Value-Aligned Route Choice Modeling via Inverse Reinforcement Learning. 2024. hal-04627792

HAL Id: hal-04627792

https://hal.science/hal-04627792

Preprint submitted on 27 Jun 2024

HAL is a multi-disciplinary open access
archive for the deposit and dissemination of sci-
entific research documents, whether they are pub-
lished or not. The documents may come from
teaching and research institutions in France or
abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est
destinée au dépôt et à la diffusion de documents
scientifiques de niveau recherche, publiés ou non,
émanant des établissements d'enseignement et de
recherche français ou étrangers, des laboratoires
publics ou privés.

<figure><img src="image_1.png" /></figure>

Distributed under a Creative Commons Attribution 4.0 International License
# Value Learning for Value-Aligned Route Choice Modeling via Inverse Reinforcement Learning

Andrés Holgado-Sánchez, Javier Bajo, Holger Billhardt, Sascha Ossowski,
Joaquín Arias

► To cite this version:

Andrés Holgado-Sánchez, Javier Bajo, Holger Billhardt, Sascha Ossowski, Joaquín Arias. Value Learning for Value-Aligned Route Choice Modeling via Inverse Reinforcement Learning. 2024. hal-04627792

HAL Id: hal-04627792

https://hal.science/hal-04627792v1

Preprint submitted on 27 Jun 2024

HAL is a multi-disciplinary open access
archive for the deposit and dissemination of sci-
entific research documents, whether they are pub-
lished or not. The documents may come from
teaching and research institutions in France or
abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est
destinée au dépôt et à la diffusion de documents
scientifiques de niveau recherche, publiés ou non,
émanant des établissements d'enseignement et de
recherche français ou étrangers, des laboratoires
publics ou privés.

<figure><img src="image_1.png" /></figure>

Distributed under a Creative Commons Attribution 4.0 International License
# Value Learning for Value-Aligned Route Choice Modeling via Inverse Reinforcement Learning

Andrés Holgado-Sánchezª, Javier Bajoᵇ, Holger Billhardtª, Sascha Ossowskiª and Joaquín Ariasª

ªCETINIA, Universidad Rey Juan Carlos, Madrid, Spain
{andres.holgado, joaquin.arias, holger.billhardt, sascha.ossowski}@urjc.es

ᵇ Departamento de Inteligencia Artificial, ETSI Informáticos, Universidad Politécnica de Madrid, Campus de Montegancedo, 28660 Madrid, Spain
jbajo@fi.upm.es

ORCID (Andrés Holgado-Sánchez): https://orcid.org/0000-0001-8853-1022, ORCID (Javier Bajo): https://orcid.org/0000-0002-4392-4743, ORCID (Holger Billhardt): https://orcid.org/0000-0001-8298-4178, ORCID (Sascha Ossowski): https://orcid.org/0000-0003-2483-9508, ORCID (Joaquín Arias): https://orcid.org/0000-0003-4148-311X

**Abstract.** Our society increasingly demands artificial intelligence (AI) that aligns with human values. Automatically learning computational understandings of values is deemed to be key in the value aligned AI community, though, to our knowledge, no attempt has been proposed for learning jointly value specifications and preferences from behavior demonstrations alone. In this work we propose a novel value learning framework (agnostic of specific learning techniques) for (i) learning value groundings (specifications of given values) and (ii) identifying value systems (the particular value preferences of agents) from agent behavior in concrete application domains. We illustrate our framework in a route choice modeling scenario, using tailored Inverse Reinforcement Learning algorithms. The results show that we can successfully learn value systems that are coherent with observed route choices. Our findings open up intriguing concerns and challenges for future research in the area.

## 1 Introduction

Current directions in AI support the inclusion of values in the decision-making process of agents, mostly focusing on the notion of *value alignment* [9, 30]. The goal of value alignment is to ensure that the behavior of AI is aligned (imbued) with human values [39]. Several approaches exist for an AI system to optimize or leverage an acceptable alignment of its actions with a given explicit specifications of values. For instance, there are examples of value-aligned decision making in the context of normative systems [29, 46], preference aggregation [18] and reinforcement learning [37, 6, 14].

Russell [38] goes beyond this and synthesises three ambitious principles for designing truly generally value-aligned machines. Machines should not only (1) maximize the realization of human preferences. They also (2) need not know how the preferences are defined; and (3) the preferences should be learned from human behavior. The problem of learning explicit specifications of values under human demonstrations, has been called *value learning* [47].

Different forms of value learning already exist in the literature such as *value inference* [23, 22] or *prima facie* duty learning [4, 2]. However, to our knowledge, there is no attempt in the literature

learning value specifications directly from demonstrations without assumptions on either the value alignment of actions or their preferences. E.g. [48] assumes given rewards measuring value alignment; and [3] uses action preferences as training data with ethical features.

Setting out from a set of previously identified labels of values (without an explicit meaning), in this paper we are interested in learning, from observed behavior, both, the alignment or grounding of specific values in given contexts (the actual meaning of a value in that context) and the particular and possible heterogeneous value preferences (i.e. *value systems* [45, 23, 32]) of agents that act in such contexts. To address this *value system learning problem*, we set out from a simple value system representation and formally define the notion of value-alignment on top of it.

We test our model and problem formulation in a use case on route choice modeling [33]. Here we assume that different routes in a traffic network (we take a previously studied network from Shanghai (China) [53]) are aligned to different preferences over some applicable values. In particular we consider the values of *sustainability*, *time efficiency* and *security*. Our approach is based on Inverse Reinforcement Learning (IRL) a fruitful paradigm adhering to Russell's principles previously used in the value-aligned AI literature [11]. Specifically, we learn reward functions that can be interpreted as alignment functions, and policies that mimic observed behaviors.

The paper is structured as follows. In Section 2 we review related work on value-aligned AI, inverse reinforcement learning and route choice. In Section 3, we explain our proposed value system learning framework. In Section 4 we detail the instantiation of our framework (with dedicated algorithms) in the route choice use case. In Section 5 we evaluate our framework and provide discussion of the results. In Section 6 we finish with conclusions and future work suggestions.

## 2 Related work

There are several proposals for computational representations of values and value systems [12] with focus on the design of value-aligned systems. For instance, Serramia et al. [44, 45] consider value systems as preferences over values with a deontological view, while Montes
# Value Learning for Value-Aligned Route Choice Modeling via Inverse Reinforcement Learning

Andrés Holgado-Sánchezª, Javier Bajoᵇ, Holger Billhardtª, Sascha Ossowskiª and Joaquín Ariasª

ªCETINIA, Universidad Rey Juan Carlos, Madrid, Spain
{andres.holgado, joaquin.arias, holger.billhardt, sascha.ossowski}@urjc.es

ᵇDepartamento de Inteligencia Artificial, ETSI Informáticos, Universidad Politécnica de Madrid, Campus de Montegancedo, 28660 Madrid, Spain
jbajo@fi.upm.es

ORCID (Andrés Holgado-Sánchez): https://orcid.org/0000-0001-8853-1022, ORCID (Javier Bajo): https://orcid.org/0000-0002-4392-4743, ORCID (Holger Billhardt): https://orcid.org/0000-0001-8298-4178, ORCID (Sascha Ossowski): https://orcid.org/0000-0003-2483-9508, ORCID (Joaquín Arias): https://orcid.org/0000-0003-4148-311X

**Abstract.** Our society increasingly demands artificial intelligence (AI) that aligns with human values. Automatically learning computational understandings of values is deemed to be key in the value aligned AI community, though, to our knowledge, no attempt has been proposed for learning jointly value specifications and preferences from behavior demonstrations alone. In this work we propose a novel value learning framework (agnostic of specific learning techniques) for (i) learning value groundings (specifications of given values) and (ii) identifying value systems (the particular value preferences of agents) from agent behavior in concrete application domains. We illustrate our framework in a route choice modeling scenario, using tailored Inverse Reinforcement Learning algorithms. The results show that we can successfully learn value systems that are coherent with observed route choices. Our findings open up intriguing concerns and challenges for future research in the area.

## 1 Introduction

Current directions in AI support the inclusion of values in the decision-making process of agents, mostly focusing on the notion of *value alignment* [9, 30]. The goal of value alignment is to ensure that the behavior of AI is aligned (imbued) with human values [39]. Several approaches exist for an AI system to optimize or leverage an acceptable alignment of its actions with a given explicit specifications of values. For instance, there are examples of value-aligned decision making in the context of normative systems [29, 46], preference aggregation [18] and reinforcement learning [37, 6, 14].

Russell [38] goes beyond this and synthesises three ambitious principles for designing truly generally value-aligned machines. Machines should not only (1) maximize the realization of human preferences. They also (2) need not know how the preferences are defined; and (3) the preferences should be learned from human behavior. The problem of learning explicit specifications of values under human demonstrations, has been called *value learning* [47].

Different forms of value learning already exist in the literature such as *value inference* [23, 22] or *prima facie* duty learning [4, 2]. However, to our knowledge, there is no attempt in the literature

learning value specifications directly from demonstrations without assumptions on either the value alignment of actions or their preferences. E.g. [48] assumes given rewards measuring value alignment; and [3] uses action preferences as training data with ethical features.

Setting out from a set of previously identified labels of values (without an explicit meaning), in this paper we are interested in learning, from observed behavior, both, the alignment or grounding of specific values in given contexts (the actual meaning of a value in that context) and the particular and possible heterogeneous value preferences (i.e. *value systems* [45, 23, 32]) of agents that act in such contexts. To address this *value system learning problem*, we set out from a simple value system representation and formally define the notion of value-alignment on top of it.

We test our model and problem formulation in a use case on route choice modeling [33]. Here we assume that different routes in a traffic network (we take a previously studied network from Shanghai (China) [53]) are aligned to different preferences over some applicable values. In particular we consider the values of *sustainability*, *time efficiency* and *security*. Our approach is based on Inverse Reinforcement Learning (IRL) a fruitful paradigm adhering to Russell's principles previously used in the value-aligned AI literature [11]. Specifically, we learn reward functions that can be interpreted as alignment functions, and policies that mimic observed behaviors.

The paper is structured as follows. In Section 2 we review related work on value-aligned AI, inverse reinforcement learning and route choice. In Section 3, we explain our proposed value system learning framework. In Section 4 we detail the instantiation of our framework (with dedicated algorithms) in the route choice use case. In Section 5 we evaluate our framework and provide discussion of the results. In Section 6 we finish with conclusions and future work suggestions.

## 2 Related work

There are several proposals for computational representations of values and value systems [12] with focus on the design of value-aligned systems. For instance, Serramia et al. [44, 45] consider value systems as preferences over values with a deontological view, while Montes
and Sierra [29] assume a consequentialist approach with quantitative value semantics defined over states of the world. Osman and d’Inverno [32] presented a taxonomical model where concept generalization is key for determining value alignment from concrete properties to more abstract notions that are used to define each value in context-dependent scenarios. Karanik et al. [16] studied correlations in the meaning of values setting out from Schwartz’s theory [43].

In this context, a promising line of research looks into learning specifications of values adaptable to the application domain. Soares [47] labels such general task as the *value learning problem*, i.e. learning representations of values under human demonstrations. Practical forms of value learning exist in the literature that tackle the so-called problem of *value inference* [23], composed by three tasks: *value identification* [50, 34, 21, 49]—i.e. identifying the set of values relevant to a decision context—; value system estimation [48, 22, 24]—i.e. learning value preferences based on observed behavior, texts or stakeholder conversations— and value aggregation [18, 19]—i.e. aggregating value systems for consensus—.

Even though ethical behavior learning techniques do not focus specifically on value learning, they are certainly related to it. One approach is *prima facie* duty learning [4, 2, 3], that directly learns general action preferences (principles) in terms of the ethical features they provoke or entail (e.g. value satisfaction), from annotated action preferences from experts. Another approach is CIRL (Cooperative Inverse Reinforcement Learning) [11] that learns human-aligned behavior in cooperation games.

The problem of **Inverse Reinforcement Learning** (IRL) in Markov decision processes (MDP) is the problem of extracting a reward function given observed (optimal) behavior [31]. IRL assumes that the seen behaviors/demonstrations in the MDP correspond to those obtained under an “expert policy” $\pi_E$ sampling the behaviors that are optimal w.r.t. some unknown reward function $R_E$. The objective is learning (another) reward function $\hat{R}$ so that a reinforcement learning policy, optimal for $\hat{R}$ can produce trajectories the most similar to the demonstrations.

The first proposed IRL algorithms opted to minimize the feature expectation difference with the expert [1], but gave room to many different solutions. Later, Ziebart et al. proposed the use the principle of maximum (causal) entropy [13] to provide a single optimal solution [56, 55, 51]. Other approaches rely on deep learning architectures [15, 6], use Bayesian updates [20] or take inspiration from adversarial networks [7]. Others use human-annotated preferences over trajectories [8, 6] as learning criterion.

**Route choice modeling** [33] consists of deriving a mathematical model that can rationally explain or predict the route choice of agents in a certain transportation network taking into account potentially heterogeneous agent preferences [35]. The most used method is shortest path search according to a custom generalized cost function.

However, the fact that different people have different objectives is undeniable. For instance, different lifestyles and situations determine mode of transport [41], electric vehicle users would prioritize routes different than gasoline ones [52] or emergency situations [28].

Different models exist to elicit preference variability. For instance, [26] analyzes taxi route choices discriminating between three state transition features used in linear reward architectures (travel distance, idle time and number of passengers); [35] manages prospect preferences regarding travel time uncertainty; [40] uses AHP with an underlying ontology to more profoundly differentiate choice criteria.

IRL algorithms have been widely applied in route choice modeling, e.g. for the interpretation of taxi driver route choices [56, 53, 25, 26], bicyclist route choice modeling [17], for food delivery logis-

tics [27] and link flow estimation [54].

## 3 Value learning framework

In this paper we deal with the problem of *value system learning* which consists of learning explicit agent preferences over a given set of values (their value system) given demonstrations of an agent’s behavior in an application domain. This problem is an initial view of the broader value learning problem [47].

We first introduce the notion of *value* used in our approach. The value representation model should be agnostic of value theories (e.g. Schwartz [42] or MFT [10]), though not necessarily misaligned with them. In particular, we adopt Osman and D’Inverno’s notion of values, as “human abstract concepts that guide behavior” [32]. Also, we propose each value to have a *grounding* (its actual meaning) in a given domain, which can be defined by a function on a set of *properties* or features that can be identified automatically on certain *entities* of the world. This is more formally expressed in Definition 1.

**Definition 1 (Value grounding and alignment).** *The computational interpretation (or grounding) of a value v (e.g. fairness or achievement) in an application domain is defined by a value alignment function $g^v: \mathcal{P} \to \text{COD}$, where $\mathcal{P} = P_1 \times \dots \times P_d$ is the space of automatically verifiable properties of a certain set of world entities; and COD refers to a condomain (metric space) where the degree of alignment with the value is measured<fnref n="1" />.*

*We also define the alignment of an entity e with value v as $\mathcal{A}(e, v) = g^v(p_1(e), \dots, p_d(e)) \in \text{COD}$, where $(p_1(e), \dots, p_d(e)) \in \mathcal{P}$ are the observed properties of entity e.*

In our model we assume that the grounding of a value is *socially-agreed*. By *socially-agreed* we mean values with global relevance in the context (according to all agents in the considered application domain) and the degree of alignment with each value is independent (to a negligible amount) of individual agent beliefs.

Albeit agents may agree on how the values are promoted or demoted, they may have different goals and desires that give raise to different value *preferences*, i.e. they can have different *value systems*.

**Definition 2 (Value system).** Let $V = \{v_1, \dots, v_m\}$ a finite set of socially-agreed values considered in an application domain, each with value alignment function $g^{v_i}$ (with $v_i \in V$) and ranging in the same condomain COD. The *value system* of an agent $a$ is determined by a *value system alignment function* $f^a: \text{COD}^m \to \text{COD}$ that provides the personal alignment of a given entity with the preferences of the agent as a function of the individual value alignments.

We also define the alignment of an entity $e$ with the value system of agent $a$ ($\text{VS}(a)$) as $\mathcal{A}(e, \text{VS}(a)) = f^a(g^{v_1}(p_1(e), \dots, p_d(e)), \dots, g^{v_m}(p_1(e), \dots, p_d(e))) \in \text{COD}$, where $(p_1(e), \dots, p_d(e)) \in \mathcal{P}$ are the observed properties of $e$.

**Definition 2** is compatible with existent value system definitions in the literature such as the most extended definition of value system as a (sometimes quantifiable) ranking ([5, 18, 44, 45]), which can be approached by treating $f$ as a utility function. Connections can be made with taxonomical models of value systems [32] that take into account concept relationships or consider value synergies [16].

Figure 1 presents the overall picture of our value system representation model. Based on the model we can define our value sys-

<footnote n="1">¹ E.g. if COD = [-1, 1] the designer could interpret values close to 1 as situations aligned with or promoting the value; values close to 0 as neutral; and values close to -1 as dis-aligned with the value or demoting it.</footnote>
and Sierra [29] assume a consequentialist approach with quantitative value semantics defined over states of the world. Osman and d’Inverno [32] presented a taxonomical model where concept generalization is key for determining value alignment from concrete properties to more abstract notions that are used to define each value in context-dependent scenarios. Karanik et al. [16] studied correlations in the meaning of values setting out from Schwartz’s theory [43].

In this context, a promising line of research looks into learning specifications of values adaptable to the application domain. Soares [47] labels such general task as the *value learning problem*, i.e. learning representations of values under human demonstrations. Practical forms of value learning exist in the literature that tackle the so-called problem of *value inference* [23], composed by three tasks: *value identification* [50, 34, 21, 49]—i.e. identifying the set of values relevant to a decision context—; value system estimation [48, 22, 24]—i.e. learning value preferences based on observed behavior, texts or stakeholder conversations— and value aggregation [18, 19]—i.e. aggregating value systems for consensus—.

Even though ethical behavior learning techniques do not focus specifically on value learning, they are certainly related to it. One approach is *prima facie* duty learning [4, 2, 3], that directly learns general action preferences (principles) in terms of the ethical features they provoke or entail (e.g. value satisfaction), from annotated action preferences from experts. Another approach is CIRL (Cooperative Inverse Reinforcement Learning) [11] that learns human-aligned behavior in cooperation games.

The problem of **Inverse Reinforcement Learning** (IRL) in Markov decision processes (MDP) is the problem of extracting a reward function given observed (optimal) behavior [31]. IRL assumes that the seen behaviors/demonstrations in the MDP correspond to those obtained under an “expert policy” $\pi_E$ sampling the behaviors that are optimal w.r.t. some unknown reward function $R_E$. The objective is learning (another) reward function $\hat{R}$ so that a reinforcement learning policy, optimal for $\hat{R}$ can produce trajectories the most similar to the demonstrations.

The first proposed IRL algorithms opted to minimize the feature expectation difference with the expert [1], but gave room to many different solutions. Later, Ziebart et al. proposed the use the principle of maximum (causal) entropy [13] to provide a single optimal solution [56, 55, 51]. Other approaches rely on deep learning architectures [15, 6], use Bayesian updates [20] or take inspiration from adversarial networks [7]. Others use human-annotated preferences over trajectories [8, 6] as learning criterion.

**Route choice modeling** [33] consists of deriving a mathematical model that can rationally explain or predict the route choice of agents in a certain transportation network taking into account potentially heterogeneous agent preferences [35]. The most used method is shortest path search according to a custom generalized cost function.

However, the fact that different people have different objectives is undeniable. For instance, different lifestyles and situations determine mode of transport [41], electric vehicle users would prioritize routes different than gasoline ones [52] or emergency situations [28].

Different models exist to elicit preference variability. For instance, [26] analyzes taxi route choices discriminating between three state transition features used in linear reward architectures (travel distance, idle time and number of passengers); [35] manages prospect preferences regarding travel time uncertainty; [40] uses AHP with an underlying ontology to more profoundly differentiate choice criteria.

IRL algorithms have been widely applied in route choice modeling, e.g. for the interpretation of taxi driver route choices [56, 53, 25, 26], bicyclist route choice modeling [17], for food delivery logis-

tics [27] and link flow estimation [54].

## 3 Value learning framework

In this paper we deal with the problem of *value system learning* which consists of learning explicit agent preferences over a given set of values (their value system) given demonstrations of an agent’s behavior in an application domain. This problem is an initial view of the broader value learning problem [47].

We first introduce the notion of *value* used in our approach. The value representation model should be agnostic of value theories (e.g. Schwartz [42] or MFT [10]), though not necessarily misaligned with them. In particular, we adopt Osman and D’Inverno’s notion of values, as “human abstract concepts that guide behavior” [32]. Also, we propose each value to have a *grounding* (its actual meaning) in a given domain, which can be defined by a function on a set of *properties* or features that can be identified automatically on certain *entities* of the world. This is more formally expressed in Definition 1.

**Definition 1 (Value grounding and alignment).** *The computational interpretation (or grounding) of a value v (e.g. fairness or achievement) in an application domain is defined by a value alignment function $g^v: \mathcal{P} \to \text{COD}$, where $\mathcal{P} = P_1 \times \dots \times P_d$ is the space of automatically verifiable properties of a certain set of world entities; and COD refers to a condomain (metric space) where the degree of alignment with the value is measured<fnref n="1" />.*

We also define the alignment of an entity e with value v as $\mathcal{A}(e, v) = g^v(p_1(e), \dots, p_d(e)) \in \text{COD}$, where $(p_1(e), \dots, p_d(e)) \in \mathcal{P}$ are the observed properties of entity e.

In our model we assume that the grounding of a value is *socially-agreed*. By *socially-agreed* we mean values with global relevance in the context (according to all agents in the considered application domain) and the degree of alignment with each value is independent (to a negligible amount) of individual agent beliefs.

Albeit agents may agree on how the values are promoted or demoted, they may have different goals and desires that give raise to different value *preferences*, i.e. they can have different *value systems*.

**Definition 2 (Value system).** Let $V = \{v_1, \dots, v_m\}$ a finite set of socially-agreed values considered in an application domain, each with value alignment function $g^{v_i}$ (with $v_i \in V$) and ranging in the same condomain COD. The *value system* of an agent $a$ is determined by a *value system alignment function* $f^a: \text{COD}^m \to \text{COD}$ that provides the personal alignment of a given entity with the preferences of the agent as a function of the individual value alignments.

We also define the alignment of an entity e with the value system of agent a ($VS(a)$) as $\mathcal{A}(e, VS(a)) = f^a(g^{v_1}(p_1(e), \dots, p_d(e)), \dots, g^{v_m}(p_1(e), \dots, p_d(e))) \in \text{COD}$, where $(p_1(e), \dots, p_d(e)) \in \mathcal{P}$ are the observed properties of e.

Definition 2 is compatible with existent value system definitions in the literature such as the most extended definition of value system as a (sometimes quantifiable) ranking ([5, 18, 44, 45]), which can be approached by treating $f$ as a utility function. Connections can be made with taxonomical models of value systems [32] that take into account concept relationships or consider value synergies [16].

Figure 1 presents the overall picture of our value system representation model. Based on the model we can define our value sys-

<footnote n="1">¹ E.g. if COD = [-1, 1] the designer could interpret values close to 1 as situations aligned with or promoting the value; values close to 0 as neutral; and values close to -1 as dis-aligned with the value or demoting it.</footnote>
<figure><img src="image_2.png" /><figcaption>**Figure 1.** A visual representation of the proposed value system representation model in a domain with three properties $p_1, p_2, p_3$ and three example values (Fairness, F, Security S and Achievement A).</figcaption></figure>

tem learning problem formally: Given an application domain, a set of identified applicable values $V = \{v_1, \dots, v_m\}$ in that domain<fnref n="2" /> and a set of agents $A = \{a_1, \dots, a_J\}$, the *value learning problem* consists of solving two consecutive tasks.

1. **Value Grounding Learning:** learning the groundings of all the previously identified values, i.e. discovering the set of individual value alignment functions $\{g^v : P \to COD | v \in V\}$. We argue that this task can be performed by individually learning each alignment function by observing behaviors (or actions) of agents that are known to align optimally with a given value<fnref n="3" />.

2. **Value System Identification:** learning a value system for each agent $a_j$, i.e. learning the value system alignment functions $\{f^j | j \in \{1, \dots, J\}\}$. This task should be accomplished solely by observing each agent's behaviors/actions and using the previously learnt value groundings.

In practice, the particular learning technique that is applied will restrict the search space of the alignment functions. For instance, they could be parametric function spaces (i.e. for neural network learning) or the sets of functions that can be a composition of a fixed set of operators (for other techniques).

We are aware of the similarity between our computational model and that of Osman and D'Inverno [32]. Indeed our model is in principal capable of representing individual value alignments through increasingly abstract concepts and properties and has a similar hierarchical structure to that of a taxonomy representation (seen clearly in the example of Figure 1). However, there are key differences. First, we introduce the notion of alignment with different values, while [32] consider each value to be a separate taxonomy with a single alignment function. Second, our notion of *properties* has no intrinsic ethical or value related meaning. Third, we only use alignment function dependencies as relationships between values, properties and concepts, while [32] use the more strict notion of generalization.

# 4 Use case

We are interested in approaching the problem of value-aware *route choice modeling* [33], i.e. acquiring an explainable representation of the involved values driving the decision-making of different agents with different *value systems*. This will serve as a use case for the presented value representation and learning framework.

## 4.1 Learning environment

We consider a general road network defined by a directed graph. The network is composed of a set of *nodes* ($\mathcal{N}$) that play the role of road intersections and *directed edges* ($\mathcal{E}$) that play the role of streets or road segments that allow the traffic to flow in the specified direction. We assume in this work for the sake of simplicity that the capacity of the edges is infinite, though each edge belongs to a different *type* of road segment, has a certain length and other numerical features.

In this network, a set of agents $J$ act to move from a certain edge $o$ (the origin) to another one $d$ (the destination) using a personal vehicle. We define the set of origin-destination pairs as the set OD. We define a route connecting $o$ and $d$ as $R_{o,d} = (o, e_1, \dots, e_L, d)$, where there exist some nodes $\{n_1, \dots, n_{L+1}\}$ such that $o$ ends in a node $n_1$, $e_i$ starts in $n_i$ and ends in $n_{i+1}$ for all $i \in \{1, \dots, L\}$ and $d$ starts in $n_{L+1}$. We assume that each agent chooses routes for each origin and destination pairs according to some unknown criteria.

In this framework, adapted from [53], the route choice problem is formulated as a Markov Decision Process (MDP) (without discount factor) given by a tuple $(S, A, T, R)$, where:

* $S$ are the states of the MDP corresponding to all the pairs $(e, d)$ where $(e, d) \in OD$. The edge $e$ represents the current road segment that is being traversed by the agent and $d$ its destination.

* $A$ are the actions available from each state and correspond to the act of choosing one of the next road segments after reaching the intersection at the end of the current edge. We denote by $A(s)$ the possible actions (e.g., road segments) in state $s$.

* $T$ is the set of transitions $(s, a, s') \in S \times A \times S$ given by the network. We assume deterministic transitions in which, given state $s = (e, d)$, when action $a \in A(s)$ is taken, the next state will simply be $s' = (a, d)$. We also define a *trajectory* as a series of states $t = (s_1, \dots, s_n)$ linked by actions.

* $R$ is the reward function $R(s, a)$ that defines the immediate advantage of choosing an action $a$ in a state $s$. Then, the cumulative reward (or return) of a route $R_{e_0, e_{L+1}}$ is given by: $\sum_{t=0}^{L} R((e_t, d), e_{t+1})$.

In our use case, we will consider deterministic policies that solve the previous MDP specification. A policy $\pi: S \to A$ solves the MDP if it assigns to each state the action that maximizes future expected return. I.e., $\pi$ from state $s_0$ chooses $a_0$ such that $\mathbb{E}_{a_t=\pi(s_t)}(\sum_{t=0}^H R(s_t, a_t))$ is maximum among all actions available from $s_0$ for some maximum episode length $H$. In our use case we use value iteration to compute the policies.

We model rewards in a way that can be understood as negative costs for shortest-path finding between each OD pair. Formally, we impose $R((e, d), e') \in \mathbb{R}^-$ signalling the cost of traversing $e'$. Furthermore, we assume that

$$R((e, d), e') = \begin{cases} 0 & \text{if } e' = d \\ R(e') & \text{otherwise} \end{cases} \qquad (1)$$

where $R(e')$ represents the reward of edge $e$. That is, first, the reward is 0 if the edge $e'$ is the destination, and second, the reward solely depends on the edge $e'$ itself if not. On one hand, this assures that the policies obtained via value iteration (or via standard reinforcement learning (RL) algorithms) are deterministic and minimize the overall costs of the chosen route for the given origin-destination pairs, e.g., will provide shortest paths solutions w.r.t. the reward function $R$. On the other hand, it actually allows for generalizing rewards such that they do not depend on the destination of an agent. For simplicity, in the sequel we refer to the reward function $R(e')$.

<footnote n="2">² If values are initially unknown we can identify them by using Value Identification approaches [50, 21, 34].</footnote>

<footnote n="3">³ In many application domains it seems feasible to identify such behaviors or actions, e.g., by asking agents.</footnote>
<figure><img src="image_2.png" /><figcaption>**Figure 1.** A visual representation of the proposed value system representation model in a domain with three properties p1, p2, p3 and three example values (Fairness, F, Security S and Achievement A).</figcaption></figure>

tem learning problem formally: Given an application domain, a set of identified applicable values $V = \{v_1, \dots, v_m\}$ in that domain<fnref n="2" /> and a set of agents $A = \{a_1, \dots, a_J\}$, the *value learning problem* consists of solving two consecutive tasks.

1. **Value Grounding Learning:** learning the groundings of all the previously identified values, i.e. discovering the set of individual value alignment functions $\{g^v : P \to COD | v \in V\}$. We argue that this task can be performed by individually learning each alignment function by observing behaviors (or actions) of agents that are known to align optimally with a given value<fnref n="3" />.

2. **Value System Identification:** learning a value system for each agent $a_j$, i.e. learning the value system alignment functions $\{f^j | j \in \{1, \dots, J\}\}$. This task should be accomplished solely by observing each agent's behaviors/actions and using the previously learnt value groundings.

In practice, the particular learning technique that is applied will restrict the search space of the alignment functions. For instance, they could be parametric function spaces (i.e. for neural network learning) or the sets of functions that can be a composition of a fixed set of operators (for other techniques).

We are aware of the similarity between our computational model and that of Osman and D'Inverno [32]. Indeed our model is in principal capable of representing individual value alignments through increasingly abstract concepts and properties and has a similar hierarchical structure to that of a taxonomy representation (seen clearly in the example of Figure 1). However, there are key differences. First, we introduce the notion of alignment with different values, while [32] consider each value to be a separate taxonomy with a single alignment function. Second, our notion of *properties* has no intrinsic ethical or value related meaning. Third, we only use alignment function dependencies as relationships between values, properties and concepts, while [32] use the more strict notion of generalization.

# 4 Use case

We are interested in approaching the problem of value-aware *route choice modeling* [33], i.e. acquiring an explainable representation of the involved values driving the decision-making of different agents with different *value systems*. This will serve as a use case for the presented value representation and learning framework.

## 4.1 Learning environment

We consider a general road network defined by a directed graph. The network is composed of a set of *nodes* ($\mathcal{N}$) that play the role of road intersections and *directed edges* ($\mathcal{E}$) that play the role of streets or road segments that allow the traffic to flow in the specified direction. We assume in this work for the sake of simplicity that the capacity of the edges is infinite, though each edge belongs to a different *type* of road segment, has a certain length and other numerical features.

In this network, a set of agents $J$ act to move from a certain edge $o$ (the origin) to another one $d$ (the destination) using a personal vehicle. We define the set of origin-destination pairs as the set OD. We define a route connecting $o$ and $d$ as $R_{o,d} = (o, e_1, \dots, e_L, d)$, where there exist some nodes $\{n_1, \dots, n_{L+1}\}$ such that $o$ ends in a node $n_1$, $e_i$ starts in $n_i$ and ends in $n_{i+1}$ for all $i \in \{1, \dots, L\}$ and $d$ starts in $n_{L+1}$. We assume that each agent chooses routes for each origin and destination pairs according to some unknown criteria.

In this framework, adapted from [53], the route choice problem is formulated as a Markov Decision Process (MDP) (without discount factor) given by a tuple $(S, A, T, R)$, where:

* $S$ are the states of the MDP corresponding to all the pairs $(e, d)$ where $(e, d) \in OD$. The edge $e$ represents the current road segment that is being traversed by the agent and $d$ its destination.

* $A$ are the actions available from each state and correspond to the act of choosing one of the next road segments after reaching the intersection at the end of the current edge. We denote by $A(s)$ the possible actions (e.g., road segments) in state $s$.

* $T$ is the set of transitions $(s, a, s') \in S \times A \times S$ given by the network. We assume deterministic transitions in which, given state $s = (e, d)$, when action $a \in A(s)$ is taken, the next state will simply be $s' = (a, d)$. We also define a *trajectory* as a series of states $t = (s_1, \dots, s_n)$ linked by actions.

* $R$ is the reward function $R(s, a)$ that defines the immediate advantage of choosing an action $a$ in a state $s$. Then, the cumulative reward (or return) of a route $R_{e_0, e_{L+1}}$ is given by: $\sum_{t=0}^{L} R((e_t, d), e_{t+1})$.

In our use case, we will consider deterministic policies that solve the previous MDP specification. A policy $\pi: S \to A$ solves the MDP if it assigns to each state the action that maximizes future expected return. I.e., $\pi$ from state $s_0$ chooses $a_0$ such that $\mathbb{E}_{a_t=\pi(s_t)}(\sum_{t=0}^H R(s_t, a_t))$ is maximum among all actions available from $s_0$ for some maximum episode length $H$. In our use case we use value iteration to compute the policies.

We model rewards in a way that can be understood as negative costs for shortest-path finding between each OD pair. Formally, we impose $R((e, d), e') \in \mathbb{R}^-$ signalling the cost of traversing $e'$. Furthermore, we assume that

$$R((e, d), e') = \begin{cases} 0 & \text{if } e' = d \\ R(e') & \text{otherwise} \end{cases} \qquad (1)$$

where $R(e')$ represents the reward of edge $e$. That is, first, the reward is 0 if the edge $e'$ is the destination, and second, the reward solely depends on the edge $e'$ itself if not. On one hand, this assures that the policies obtained via value iteration (or via standard reinforcement learning (RL) algorithms) are deterministic and minimize the overall costs of the chosen route for the given origin-destination pairs, e.g., will provide shortest paths solutions w.r.t. the reward function $R$. On the other hand, it actually allows for generalizing rewards such that they do not depend on the destination of an agent. For simplicity, in the sequel we refer to the reward function $R(e')$.

<footnote n="2">² If values are initially unknown we can identify them by using Value Identification approaches [50, 21, 34].</footnote>

<footnote n="3">³ In many application domains it seems feasible to identify such behaviors or actions, e.g., by asking agents.</footnote>
## 4.2 Values in route choice modeling

In our use case scenario we assume that each agent j chooses its routes according to some (unknown) reward function $R_j(e)$. Furthermore, we assume that the agent also acts in accordance to its individual value system $VS(j)$. Thus, we can establish a relationship between the agent's value system and the reward function it applies (Equation 2).

$$R_j(e) = -\mathcal{A}(e, VS(j)) \qquad (2)$$

This relation is the basic idea underlying our work: If we learn the reward function of an agent, we can actually estimate the agent's value system. As we mentioned before, we want the value system to be represented through an alignment function on the individual value alignments, which in turn can be computed based on measurable properties<fnref n="4" />. In the sequel we present the instantiation of this idea.

In our use case we are modeling three values $V = \{v_1, v_2, v_3\}$, namely travel sustainability (sus, $v_1$), security (sec, $v_2$) and time efficiency (eff, $v_3$). All three have already been studied in the route choice domain, e.g. [41, 36, 28]. We can assume that those values are socially-agreed. For instance, sustainability is socially-agreed in the traffic domain, because, according to the general opinion, it is related to less fuel consumption.

We take the act of traversing an edge of the road network as the entities in which we can measure the alignment with our values. The alignment of each edge traversal with each of the values (and the agent's value systems) depend on the following observable edge properties or features<fnref n="5" />:

* Fuel consumption for edge traversal: $fuel(e)$, $e \in \mathcal{E}$. For instance, secondary roads are more contaminant highways.

* Accident/Street insecurity risk: $insec(e)$, $e \in \mathcal{E}$. For instance, a primary road is considered highly secure while secondary roads and living streets are more insecure.

* Expected edge traversal time: $time(e)$, $e \in \mathcal{E}$. For instance, primary roads have less expected traversal times than country roads.

Thus, the set of features/properties observed upon the traversal of a given edge is defined as:

$$\phi(e) = (fuel(e), insec(e), time(e))$$

It should be noted that these properties are directly related to the three values, i.e. out of different choices, the road-segment with least fuel consumption is the most sustainable segment and choosing a route that maximizes the negative fuel consumption for a given OD pair will be the most sustainable route. However, we assume the model itself is not aware of this fact, that is, we actually want to learn the alignment of a road-segment with each of the values. In this work, we use a rather simple search space for the value alignment functions: the space of linear convex combinations of the input features/properties<fnref n="6" />.

With respect to the search space of value system alignment functions we again use a simplification (and plan to analyze more complex versions in the future). We also use linear convex combinations in which the linear weights correspond to the preference weights for the alignment with each of the values. We represent those weights with a convenient new notation:

**Definition 3 (Profile).** The profile of an agent j is defined by a tuple $P_j = (p_{v1}^j, \dots, p_{vm}^j)$, with $\sum_{i=1}^m p_{vi}^j = 1$ and $p_{vi}^j \in [0, 1]$. The variables $p_{vi}^j$ represent the degree of preference for value $v_i$ according to agent j. We call pure profile for value $v_i$ a profile that has the following form: $P_{v_i} = (a_1, \dots, a_i, \dots, a_j)$ where $a_i = 1$ and $a_j = 0, \forall j \neq i$.

In our settings, a profile $P_j$ uniquely defines the value system alignment function of agent j via $f^j(A_1, \dots, A_m) = P_j \cdot (A_1, \dots, A_m)$. For instance a “pure sustainable agent” will have a profile of (1, 0, 0) (aligning to sustainability solely) and an agent that considers equally both security and efficiency (while neglecting sustainability) will have a profile of (0, 0.5, 0.5).

We use a neural network architecture to represent and calculate the reward function of an agent. As mentioned before, since the reward is based on the value system of an agent, this architecture directly presents an instantiation of our general value system model (as depicted in Figure 2). The neural network consists of one input layer for the input properties of each edge traversal resulting from executing a state-action pair, a hidden layer for calculating the individual value alignments and the output layer calculating the final reward (identical to the negative alignment of such state-action pair with the value system of an agent, as for Equation 2).

The hidden layer from properties $\phi(e)$ to the individual value alignments is called $G_{\theta}$ and has the specification explained in Equation 3 in terms of a set of parameters $\theta \in \mathbb{R}^{3 \times 3}$:

$$\begin{pmatrix} \mathcal{A}(e, v_1) \\ \mathcal{A}(e, v_2) \\ \mathcal{A}(e, v_3) \end{pmatrix} = G_{\theta}(\phi(e)) = W^{\theta} \cdot \begin{pmatrix} fuel(e) \\ insec(e) \\ time(e) \end{pmatrix}, \quad (3)$$

where $W_{ij}^{\theta} = \text{softmax}((\theta_{i1}, \theta_{i2}, \theta_{i3}))_j = \frac{\exp \theta_{ij}}{\sum_{k=1}^{3} \exp \theta_{ik}}$. With this definition of the matrix $W^{\theta}$, each of its rows $W_{i,}^{\theta}$, (for $i = 1, 2, 3$) is a linear convex combination of the edge properties.

The final layer calculation is dependent on the agent. For an agent j we will find a profile $P_j$ which we can express with a linear convex combination layer with parameters $\alpha \in \mathbb{R}^3$, i.e. $P_j = \text{softmax}(\alpha) = \left( \frac{\exp \alpha_1}{\sum_{k=1}^{3} \exp \alpha_k}, \frac{\exp \alpha_2}{\sum_{k=1}^{3} \exp \alpha_k}, \frac{\exp \alpha_3}{\sum_{k=1}^{3} \exp \alpha_k} \right)$. This profile directly determines the value system alignment function of agent j: $f^j(a_1, a_2, a_3) = P_j \cdot (a_1, a_2, a_3)$ and, according to Equation 2, the reward function for agent j for a given edge e results in Equation 4:

$$R_j(e) = -f^j(G_{\theta}(\phi(e))) = -P_j \cdot G_{\theta}(\phi(e)) \quad (4)$$

With this model, the value system learning problem in this route choice modeling context consists of the following formalized tasks:

1. **Value grounding learning:** Given a set of reference routes taken by agents of known profiles, learning parameters $\theta$ of the function $G_{\theta}$ that represent the individual alignment functions.

2. **Value system identification:** Given a certain $G$ with fixed parameters and a set of trajectories of an agent j with unknown motivations, learning a profile $\hat{P}_j$ from which a deterministic policy $\pi^{\hat{P}_j}$—optimal for the reward $R_j(e) = \hat{P}_j \cdot G(\phi(e))$—will determine the given trajectories. This can be repeated for each agent.

<footnote n="4">⁴ In this use case, the condomain of alignment is set as $COD = \mathbb{R}^+$, where positive real values define the regret (or negative reward) w.r.t. each value or the value systems of the agents. That is, lower values of alignment indicate a lower demotion of values (or value preferences) and higher ones indicate higher demotion. This explains the negative sign in Equation 2.</footnote>

<footnote n="5">⁵ The values of the three properties have been normalized by dividing the original value by the maximum value of all edges.</footnote>

<footnote n="6">⁶ In future works we will analyze other, more complex scenarios, where the groundings of values might be more complex.</footnote>
## 4.2 Values in route choice modeling

In our use case scenario we assume that each agent j chooses its routes according to some (unknown) reward function $R_j(e)$. Furthermore, we assume that the agent also acts in accordance to its individual value system $VS(j)$. Thus, we can establish a relationship between the agent's value system and the reward function it applies (Equation 2).

$$R_j(e) = -\mathcal{A}(e, VS(j)) \quad (2)$$

This relation is the basic idea underlying our work: If we learn the reward function of an agent, we can actually estimate the agent's value system. As we mentioned before, we want the value system to be represented through an alignment function on the individual value alignments, which in turn can be computed based on measurable properties<fnref n="4" />. In the sequel we present the instantiation of this idea.

In our use case we are modeling three values $V = \{v_1, v_2, v_3\}$, namely travel sustainability (sus, $v_1$), security (sec, $v_2$) and time efficiency (eff, $v_3$). All three have already been studied in the route choice domain, e.g. [41, 36, 28]. We can assume that those values are socially-agreed. For instance, sustainability is socially-agreed in the traffic domain, because, according to the general opinion, it is related to less fuel consumption.

We take the act of traversing an edge of the road network as the entities in which we can measure the alignment with our values. The alignment of each edge traversal with each of the values (and the agent's value systems) depend on the following observable edge properties or features<fnref n="5" />:

* Fuel consumption for edge traversal: $fuel(e), e \in \mathcal{E}$. For instance, secondary roads are more contaminant highways.

* Accident/Street insecurity risk: $insec(e), e \in \mathcal{E}$. For instance, a primary road is considered highly secure while secondary roads and living streets are more insecure.

* Expected edge traversal time: $time(e), e \in \mathcal{E}$. For instance, primary roads have less expected traversal times than country roads.

Thus, the set of features/properties observed upon the traversal of a given edge is defined as:

$$\phi(e) = (fuel(e), insec(e), time(e))$$

It should be noted that these properties are directly related to the three values, i.e. out of different choices, the road-segment with least fuel consumption is the most sustainable segment and choosing a route that maximizes the negative fuel consumption for a given OD pair will be the most sustainable route. However, we assume the model itself is not aware of this fact, that is, we actually want to learn the alignment of a road-segment with each of the values. In this work, we use a rather simple search space for the value alignment functions: the space of linear convex combinations of the input features/properties<fnref n="6" />.

With respect to the search space of value system alignment functions we again use a simplification (and plan to analyze more complex versions in the future). We also use linear convex combinations in which the linear weights correspond to the preference weights for the alignment with each of the values. We represent those weights with a convenient new notation:

**Definition 3 (Profile).** The profile of an agent j is defined by a tuple $P_j = (p_{v1}^j, \dots, p_{vm}^j)$, with $\sum_{i=1}^m p_{vi}^j = 1$ and $p_{vi}^j \in [0, 1]$. The variables $p_{vi}^j$ represent the degree of preference for value $v_i$ according to agent j. We call pure profile for value $v_i$ a profile that has the following form: $P_{v_i} = (a_1, \dots, a_i, \dots, a_m)$ where $a_i = 1$ and $a_j = 0, \forall j \neq i$.

In our settings, a profile $P_j$ uniquely defines the value system alignment function of agent j via $f^j(A_1, \dots, A_m) = P_j \cdot (A_1, \dots, A_m)$. For instance a “pure sustainable agent” will have a profile of (1, 0, 0) (aligning to sustainability solely) and an agent that considers equally both security and efficiency (while neglecting sustainability) will have a profile of (0, 0.5, 0.5).

We use a neural network architecture to represent and calculate the reward function of an agent. As mentioned before, since the reward is based on the value system of an agent, this architecture directly presents an instantiation of our general value system model (as depicted in Figure 2). The neural network consists of one input layer for the input properties of each edge traversal resulting from executing a state-action pair, a hidden layer for calculating the individual value alignments and the output layer calculating the final reward (identical to the negative alignment of such state-action pair with the value system of an agent, as for Equation 2).

The hidden layer from properties $\phi(e)$ to the individual value alignments is called $G_{\theta}$ and has the specification explained in Equation 3 in terms of a set of parameters $\theta \in \mathbb{R}^{3 \times 3}$:

$$\begin{pmatrix} \mathcal{A}(e, v_1) \\ \mathcal{A}(e, v_2) \\ \mathcal{A}(e, v_3) \end{pmatrix} = G_{\theta}(\phi(e)) = W^{\theta} \cdot \begin{pmatrix} fuel(e) \\ insec(e) \\ time(e) \end{pmatrix}, \quad (3)$$

where $W_{ij}^{\theta} = \text{softmax}((\theta_{i1}, \theta_{i2}, \theta_{i3}))_j = \frac{\exp \theta_{ij}}{\sum_{k=1}^{3} \exp \theta_{ik}}$. With this definition of the matrix $W^{\theta}$, each of its rows $W_{i,}^{\theta}$, (for $i = 1, 2, 3$) is a linear convex combination of the edge properties.

The final layer calculation is dependent on the agent. For an agent j we will find a profile $P_j$ which we can express with a linear convex combination layer with parameters $\alpha \in \mathbb{R}^3$, i.e. $P_j = \text{softmax}(\alpha) = \left( \frac{\exp \alpha_1}{\sum_{k=1}^{3} \exp \alpha_k}, \frac{\exp \alpha_2}{\sum_{k=1}^{3} \exp \alpha_k}, \frac{\exp \alpha_3}{\sum_{k=1}^{3} \exp \alpha_k} \right)$. This profile directly determines the value system alignment function of agent j: $f^j(a_1, a_2, a_3) = P_j \cdot (a_1, a_2, a_3)$ and, according to Equation 2, the reward function for agent j for a given edge e results in Equation 4:

$$R_j(e) = -f^j(G_{\theta}(\phi(e))) = -P_j \cdot G_{\theta}(\phi(e)) \quad (4)$$

With this model, the value system learning problem in this route choice modeling context consists of the following formalized tasks:

1. **Value grounding learning:** Given a set of reference routes taken by agents of known profiles, learning parameters $\theta$ of the function $G_{\theta}$ that represent the individual alignment functions.

2. **Value system identification:** Given a certain $G$ with fixed parameters and a set of trajectories of an agent j with unknown motivations, learning a profile $\hat{P}_j$ from which a deterministic policy $\pi^{\hat{P}_j}$—optimal for the reward $R_j(e) = \hat{P}_j \cdot G(\phi(e))$—will determine the given trajectories. This can be repeated for each agent.

<footnote n="4">⁴ In this use case, the condomain of alignment is set as $COD = \mathbb{R}^+$, where positive real values define the regret (or negative reward) w.r.t. each value or the value systems of the agents. That is, lower values of alignment indicate a lower demotion of values (or value preferences) and higher ones indicate higher demotion. This explains the negative sign in Equation 2.</footnote>

<footnote n="5">⁵ The values of the three properties have been normalized by dividing the original value by the maximum value of all edges.</footnote>

<footnote n="6">⁶ In future works we will analyze other, more complex scenarios, where the groundings of values might be more complex.</footnote>
<figure><img src="image_3.png" /><figcaption>**Figure 2.** An instantiation of the proposed value system representation for the route choice modeling use case (left), and the inverse reinforcement learning equivalent reward function architecture (right).</figcaption></figure>

## 4.3 Learning algorithm architecture

We use versions of inverse reinforcement learning (IRL) algorithms to solve both tasks of the value system learning problem version in our use case. In this context we decided to set out from one of the most used (and simple) IRL algorithms called Maximum Entropy Inverse Reinforcement Learning [56], specifically, its straightforward adaptation to deep neural networks [51].

The approach uses a loss function $\mathcal{L}$ that is to be minimized to find a reward function $\hat{R}$ such that an optimal policy for this reward provides the most similar trajectories to the demonstrations. The similarity here is understood as the difference in state visitation counts (in our case edge visitation counts) between the given demonstrations ($\mu$) and those sampled with the learned policy ($\hat{\mu}$). These are used to compute the gradient (as seen in Equation 5) which is used for backpropagation into the learned reward function $\hat{R}$ with parameters $\omega$.

$$ \frac{\partial \mathcal{L}}{\partial \omega}(e) = (\hat{\mu}(e) - \mu(e)) \frac{\partial \hat{R}}{\partial \omega}(e) \quad (5) $$

In our use case we use a Monte-Carlo approach to compute the visitation counts instead of the probabilistic estimation seen in the original references [56, 51]. We directly count edge visitations in each route. That is, for a given set of routes $D$, $\mu$ is a matrix such that $\mu[t, e] = 1_{t \in D}$, for each $e \in \mathcal{E}$ and $t \in D$. The expected edge visitation counts are then approached by $\mu(e) = \frac{1}{d} \sum_{t \in D} \mu[t, e]$.

The learning setup is different in each value system learning task.

In value grounding learning (Algorithm 1) we use labeled demonstrations from agents that have pure profiles (e.g. $P_{sus} = (1, 0, 0)$). Then we perform different gradient descent updates for each of our values separately, backpropagating into the hidden layer $G_{\theta}$ assuming the reward is, e.g. for sustainable agents, given by $\hat{R}_{sus} = P_{sus} \cdot G_{\theta}$. The learnable parameters in this case would be $\omega = \theta \in \mathbb{R}^{3 \times 3}$.

In value system identification (Algorithm 2) we take unlabeled demonstrations (routes) from a given agent and estimate a profile $\hat{P}$ such that the reward $\hat{R} = \hat{P} \cdot G_{\theta}$ will produce an optimal policy that mimics the demonstrations, without modifying the already learnt individual alignments $G_{\theta}$. The learnable parameters in this case would be $\omega = \alpha \in \mathbb{R}^3$.

### Algorithm 1 Deep Maximum Entropy IRL for route choice modeling: Value Grounding Learning

**Input:** A set of profiles $\mathcal{P} = \{(1, 0, 0), (0, 1, 0), (0, 0, 1)\}$; and, for each $P \in \mathcal{P}$, a training set of routes $D_P$. A batch size $b$, learning rate $\lambda$ and training steps $N$.

**Output:** $G_{\theta}$ a network that gives $\mathcal{A}(e, v)$ for each $v \in V$; and $\{\pi_{\theta}^P : P \in \mathcal{P}\}$ where $\pi_{\theta}^P$ is optimal for reward $\hat{R}_P$ and provides similar trajectories to $D_P$.

1: Initialize environment env, reward network $G_{\theta}$ with weights $\theta$.

2: $\phi \leftarrow GETEDGETRAVERSALFEATURES env

3: **for** training step $t = 1, 2, \dots, N$ **do**

4: **for** profile $P \in \mathcal{P}$ **do**

5: $B_P \leftarrow$ choose batch of $b$ trajectories from $D_P$

6: $\mu^P \leftarrow$ VISITATIONCOUNT MATRIX($B_P$)

7: $\hat{R}_P(e) = P \cdot G_{\theta}(\phi(e))$

8: $\pi_{\theta}^P \leftarrow$ VALUEITERATIONPOLICY env, $\hat{R}_P$

9: $\hat{B}_P \leftarrow$ PROPAGATEPOLICY($\pi_{\theta}^P$, $B_P$) ▷ i.e. for each $t \in B_P$, use $\pi_{\theta}^P$ to sample a $\hat{t}$ from the same origin and to the same destination as those of $t$.

10: $\hat{\mu}^P \leftarrow$ VISITATIONCOUNT MATRIX($\hat{B}_P$)

11:

$$ \theta \leftarrow \theta - \frac{\lambda}{b} \sum_{(t, \hat{t}) \in (B_P, \hat{B}_P)} \sum_{e \in \mathcal{E}} (\hat{\mu}^P[\hat{t}, e] - \mu^P[t, e]) \frac{\partial \hat{R}_P}{\partial \theta}(e) $$

12: **end for**

13: **end for**

### Algorithm 2 Deep Maximum Entropy IRL for route choice modeling: Value System Identification

**Input:** Set of routes $D$, already learnt $G = (g_{sus}, g_{sec}, g_{eff})$, batch size $b$, learning rate $\lambda$ and training steps $N$.

**Output:** $\hat{P}$, $\pi^P$, where $\pi^P$ is optimal for reward $\hat{R}_P$ and produces trajectories similar to those in $D$.

1: Initialize environment env, $\hat{P} \leftarrow$ softmax($\alpha$) with weights $\alpha$.

2: $\phi \leftarrow GETEDGETRAVERSALFEATURES env

3: **for** training step $t = 1, 2, \dots, N$ **do**

4: $B \leftarrow$ choose batch of $b$ expert demos from $D$

5: $\mu \leftarrow$ VISITATIONCOUNT MATRIX($B$)

6: $\hat{R}_P(e) = \hat{P} \cdot G(\phi(e))$

7: $\pi^P \leftarrow$ VALUEITERATIONPOLICY env, $\hat{R}_P$

8: $\hat{B} \leftarrow$ PROPAGATEPOLICY($\pi^P$, $B$)

9: $\hat{\mu} \leftarrow$ VISITATIONCOUNT MATRIX($\hat{B}$)

10:

$$ \alpha \leftarrow \alpha - \frac{\lambda}{b} \sum_{(t, \hat{t}) \in (B, \hat{B})} \sum_{e \in \mathcal{E}} (\hat{\mu}[\hat{t}, e] - \mu[t, e]) \frac{\partial \hat{R}_P}{\partial \alpha}(e) $$

11: $\hat{P} \leftarrow$ softmax($\alpha$)

12: **end for**

## 5 Evaluation

In this section we evaluate the proposed value system learning framework and algorithms in the route choice domain.
<figure><img src="image_3.png" /><figcaption>**Figure 2.** An instantiation of the proposed value system representation for the route choice modeling use case (left), and the inverse reinforcement learning equivalent reward function architecture (right).</figcaption></figure>

## 4.3 Learning algorithm architecture

We use versions of inverse reinforcement learning (IRL) algorithms to solve both tasks of the value system learning problem version in our use case. In this context we decided to set out from one of the most used (and simple) IRL algorithms called Maximum Entropy Inverse Reinforcement Learning [56], specifically, its straightforward adaptation to deep neural networks [51].

The approach uses a loss function $\mathcal{L}$ that is to be minimized to find a reward function $\hat{R}$ such that an optimal policy for this reward provides the most similar trajectories to the demonstrations. The similarity here is understood as the difference in state visitation counts (in our case edge visitation counts) between the given demonstrations ($\mu$) and those sampled with the learned policy ($\hat{\mu}$). These are used to compute the gradient (as seen in Equation 5) which is used for backpropagation into the learned reward function $\hat{R}$ with parameters $\omega$.

$$ \frac{\partial \mathcal{L}}{\partial \omega}(e) = (\hat{\mu}(e) - \mu(e)) \frac{\partial \hat{R}}{\partial \omega}(e) \quad (5) $$

In our use case we use a Monte-Carlo approach to compute the visitation counts instead of the probabilistic estimation seen in the original references [56, 51]. We directly count edge visitations in each route. That is, for a given set of routes $D$, $\mu$ is a matrix such that $\mu[t, e] = 1_{t \in D}$, for each $e \in \mathcal{E}$ and $t \in D$. The expected edge visitation counts are then approached by $\mu(e) = \frac{1}{d} \sum_{t \in D} \mu[t, e]$.

The learning setup is different in each value system learning task.

In value grounding learning (Algorithm 1) we use labeled demonstrations from agents that have pure profiles (e.g. $P_{sus} = (1, 0, 0)$). Then we perform different gradient descent updates for each of our values separately, backpropagating into the hidden layer $G_{\theta}$ assuming the reward is, e.g. for sustainable agents, given by $\hat{R}_{sus} = P_{sus} \cdot G_{\theta}$. The learnable parameters in this case would be $\omega = \theta \in \mathbb{R}^{3 \times 3}$.

In value system identification (Algorithm 2) we take unlabeled demonstrations (routes) from a given agent and estimate a profile $\hat{P}$ such that the reward $\hat{R} = \hat{P} \cdot G_{\theta}$ will produce an optimal policy that mimics the demonstrations, without modifying the already learnt individual alignments $G_{\theta}$. The learnable parameters in this case would be $\omega = \alpha \in \mathbb{R}^3$.

### Algorithm 1 Deep Maximum Entropy IRL for route choice modeling: Value Grounding Learning

**Input:** A set of profiles $\mathcal{P} = \{(1, 0, 0), (0, 1, 0), (0, 0, 1)\}$; and, for each $P \in \mathcal{P}$, a training set of routes $D_P$. A batch size $b$, learning rate $\lambda$ and training steps $N$.

**Output:** $G_{\theta}$ a network that gives $\mathcal{A}(e, v)$ for each $v \in V$; and $\{\pi_{\theta}^P : P \in \mathcal{P}\}$ where $\pi_{\theta}^P$ is optimal for reward $\hat{R}_P$ and provides similar trajectories to $D_P$.

1: Initialize environment env, reward network $G_{\theta}$ with weights $\theta$.

2: $\phi \leftarrow GETEDGETRAVERSALFEATURES env

3: for training step $t = 1, 2, \dots, N$ do

4: for profile $P \in \mathcal{P}$ do

5: $B_P \leftarrow$ choose batch of $b$ trajectories from $D_P$

6: $\mu^P \leftarrow$ VISITATIONCOUNTMATRIX($B_P$)

7: $\hat{R}_P(e) = P \cdot G_{\theta}(\phi(e))$

8: $\pi_{\theta}^P \leftarrow$ VALUEITERATIONPOLICY env, $\hat{R}_P$

9: $\hat{B}_P \leftarrow$ PROPAGATEPOLICY($\pi_{\theta}^P$, $B_P$) ▷ i.e. for each $t \in B_P$, use $\pi_{\theta}^P$ to sample a $\hat{t}$ from the same origin and to the same destination as those of $t$.

10: $\hat{\mu}^P \leftarrow$ VISITATIONCOUNTMATRIX($\hat{B}_P$)

11:

$$ \theta \leftarrow \theta - \frac{\lambda}{b} \sum_{(t, \hat{t}) \in (B_P, \hat{B}_P)} \sum_{e \in \mathcal{E}} (\hat{\mu}^P[\hat{t}, e] - \mu^P[t, e]) \frac{\partial \hat{R}_P}{\partial \theta}(e) $$

12: end for

13: end for

### Algorithm 2 Deep Maximum Entropy IRL for route choice modeling: Value System Identification

**Input:** Set of routes $D$, already learnt $G = (g_{sus}, g_{sec}, g_{eff})$, batch size $b$, learning rate $\lambda$ and training steps $N$.

**Output:** $\hat{P}$, $\pi^P$, where $\pi^P$ is optimal for reward $\hat{R}_P$ and produces trajectories similar to those in $D$.

1: Initialize environment env, $\hat{P} \leftarrow$ softmax($\alpha$) with weights $\alpha$.

2: $\phi \leftarrow$ GETEDGETRAVERSALFEATURES env

3: for training step $t = 1, 2, \dots, N$ do

4: $B \leftarrow$ choose batch of $b$ expert demos from $D$

5: $\mu \leftarrow$ VISITATIONCOUNTMATRIX($B$)

6: $\hat{R}_P(e) = \hat{P} \cdot G(\phi(e))$

7: $\pi^P \leftarrow$ VALUEITERATIONPOLICY env, $\hat{R}_P$

8: $\hat{B} \leftarrow$ PROPAGATEPOLICY($\pi^P$, $B$)

9: $\hat{\mu} \leftarrow$ VISITATIONCOUNTMATRIX($\hat{B}$)

10:

$$ \alpha \leftarrow \alpha - \frac{\lambda}{b} \sum_{(t, \hat{t}) \in (B, \hat{B})} \sum_{e \in \mathcal{E}} (\hat{\mu}[\hat{t}, e] - \mu[t, e]) \frac{\partial \hat{R}_P}{\partial \alpha}(e) $$

11: $\hat{P} \leftarrow$ softmax($\alpha$)

12: end for

## 5 Evaluation

In this section we evaluate the proposed value system learning framework and algorithms in the route choice domain.
## 5.1 Generating routes

Because of the lack of real world data, we generate the route data for given agents artificially. Here we assume that all agents act according to a given profile *P* and will select the "cheapest" route for each OD pair according to some edge costs that depend on edge properties and the agent's profile. We define the cost for an edge *e* given a certain profile *P*, or shorter, the *profiled cost*, $C_P(e)$ by:

$$C_P(e) = P \cdot (fuel(e), insec(e), time(e)) \quad (6)$$

This formula calculates the cost of a given edge as the weighted average of fuel consumption, insecurity level and expected edge traversal time, where the profile act as preference weights. We also define the *profiled cost of a route* *R* by:

$$C_P(R) = \sum_{e \in R} P \cdot (fuel(e), insec(e), time(e)) \quad (7)$$

It should be noted that the profile for generating an agent's routes actually defines its value system and each individual value (*sus*, *sec*, and *eff*) is directly grounded in exactly one of the properties (fuel consumption, insecurity and traversal time, respectively). This implies, that in our case the value grounding learning task consists in fact in identifying the would be perfectly solved if $W^{\theta} = I_{3 \times 3}$ (i.e. the alignment of a value is defined uniquely by the corresponding property of the edge). On the other hand, the value system identification task would be solved if we find a profile $\hat{P} = softmax(\alpha)$ that approximates an agents' profile *P*.

Given the above, we can define the cost of an edge according to a value *v* as $C_v(e) = C_{P_v}(e)$ and consider this cost to be the real<fnref n="7" /> alignment with each value in this evaluation, i.e. $C_v(e) = -R(e) = A(e, v) \in \mathbb{R}^+$. Thus, we can define the alignment of a route or its cost, too, as $C_v(R) = C_{P_v}(R)$.

## 5.2 Evaluation: Value grounding learning

Here we analyze the performance of Algorithm 1 with a train-test set composed of *pure routes*—routes that are considered totally sustainable, secure or time efficient, e.g., each route is generated with a pure profile ((1, 0, 0), (0, 1, 0) and (0, 0, 1))—. In particular, we used the pure routes for 100 different OD pairs<fnref n="8" />. The training-test ratio has been 80%–20%, the learning rate $\lambda = 0.2$ and the batch size equal to the training data size. We run the experiment with 10 random splits and present average values of the 10 runs in the results.

We want to check whether the function $G_{\theta}$ is close to the expected function, that is, if the matrix $W^{\theta}$ (see Equation 3) is close to the identity. In Figure 3 we see that this is the case. Thus, we have almost perfectly identified the alignment functions with each value. Besides, we can see that the policies that maximize learnt rewards (alignments with each individual value) generate routes that are equivalent to the observed input data after about 20 iterations.

<figure><img src="image_4.png" /><figcaption>**Figure 3.** Expected matrix $W^{\theta}$ (defining value alignment functions $G_{\theta}$) after running Algorithm 1 with configuration from Section 5.2. It represents the learned linear correlation between real costs $C_{P_{v_i}}$ and learned ones $-R_{P_{v_i}}$, with standard deviations.</figcaption></figure>

The results for value grounding learning are conclusive insofar as our method works well in the given use case, correctly identifying the feature behind that grounds each of the value. Still, it has been explored how far less informative features and/or less informed learning architectures lead to similar performances.

<table><caption>**Table 1.** Results for value system identification, learning from individual agents with different profiles. Presented are the learned profile and the average value alignments of the routes (from the test set) sampled with the original profile and the learned profile are shown for the three values *sus*, *sec* and *eff* (*OrigAC*<sub>*v*<sub>i</sub></sub> and *LearAC*<sub>*v*<sub>i</sub></sub>, respectively).</caption><thead><tr><th>Original Profile</th><th>Learned Profile</th><th>OrigAC<sub>Sus</sub></th><th>OrigAC<sub>Sec</sub></th><th>OrigAC<sub>Eff</sub></th><th>LearAC<sub>Sus</sub></th><th>LearAC<sub>Sec</sub></th><th>LearAC<sub>Eff</sub></th></tr></thead><tbody><tr><td>(0.0, 0.0, 1.0)</td><td>(0.01, 0.01, 0.98)</td><td>0.389</td><td>0.591</td><td>0.236</td><td>0.389</td><td>0.590</td><td>0.236</td></tr><tr><td>(0.0, 0.25, 0.75)</td><td>(0.01, 0.04, 0.95)</td><td>0.391</td><td>0.588</td><td>0.236</td><td>0.390</td><td>0.584</td><td>0.236</td></tr><tr><td>(0.0, 0.5, 0.5)</td><td>(0.02, 0.14, 0.84)</td><td>0.390</td><td>0.576</td><td>0.237</td><td>0.388</td><td>0.568</td><td>0.238</td></tr><tr><td>(0.0, 0.75, 0.25)</td><td>(0.02, 0.17, 0.81)</td><td>0.388</td><td>0.559</td><td>0.240</td><td>0.388</td><td>0.564</td><td>0.239</td></tr><tr><td>(0.0, 1.0, 0.0)</td><td>(0.0, 0.99, 0.0)</td><td>0.598</td><td>0.266</td><td>1.028</td><td>0.591</td><td>0.266</td><td>1.009</td></tr><tr><td>(0.25, 0.0, 0.75)</td><td>(0.06, 0.01, 0.93)</td><td>0.386</td><td>0.594</td><td>0.236</td><td>0.386</td><td>0.591</td><td>0.236</td></tr><tr><td>(0.25, 0.25, 0.5)</td><td>(0.07, 0.04, 0.89)</td><td>0.385</td><td>0.585</td><td>0.236</td><td>0.385</td><td>0.584</td><td>0.236</td></tr><tr><td>(0.25, 0.5, 0.25)</td><td>(0.11, 0.13, 0.76)</td><td>0.382</td><td>0.569</td><td>0.238</td><td>0.383</td><td>0.569</td><td>0.238</td></tr><tr><td>(0.25, 0.75, 0.0)</td><td>(0.3, 0.68, 0.03)</td><td>0.480</td><td>0.297</td><td>0.722</td><td>0.481</td><td>0.296</td><td>0.723</td></tr><tr><td>(0.5, 0.0, 0.5)</td><td>(0.22, 0.02, 0.76)</td><td>0.381</td><td>0.597</td><td>0.237</td><td>0.377</td><td>0.589</td><td>0.238</td></tr><tr><td>(0.5, 0.25, 0.25)</td><td>(0.28, 0.08, 0.64)</td><td>0.378</td><td>0.579</td><td>0.238</td><td>0.376</td><td>0.574</td><td>0.239</td></tr><tr><td>(0.5, 0.5, 0.0)</td><td>(0.59, 0.39, 0.02)</td><td>0.411</td><td>0.369</td><td>0.496</td><td>0.413</td><td>0.364</td><td>0.501</td></tr><tr><td>(0.75, 0.0, 0.25)</td><td>(0.39, 0.01, 0.6)</td><td>0.372</td><td>0.600</td><td>0.239</td><td>0.371</td><td>0.597</td><td>0.240</td></tr><tr><td>(0.75, 0.25, 0.0)</td><td>(0.84, 0.16, 0.0)</td><td>0.366</td><td>0.496</td><td>0.319</td><td>0.366</td><td>0.497</td><td>0.316</td></tr><tr><td>(1.0, 0.0, 0.0)</td><td>(0.95, 0.02, 0.03)</td><td>0.359</td><td>0.564</td><td>0.273</td><td>0.359</td><td>0.564</td><td>0.271</td></tr></tbody></table>

## 5.3 Evaluation: Value system identification

For the value system identification task we have analyzed two different cases: i) “learning from an individual agent” and ii) “learning from a society of agents”.

By learning from an individual agent we mean that we try to identify its unknown profile *P* from route examples of that agent. Here we assume that the routes always minimize the cost $C_P$, for the agent's profile *P*. In particular, we want to find a profile $\hat{P}$ such that the optimal policy for the rewards $\hat{R}_P$ proposes similar routes to the once chosen of the agent in the examples in terms of their expected value alignment as measured by the cumulative costs $C_{v_i}(R)$. The reward $\hat{R}_P$ uses $G_{\theta}$ as the first layer with the learnt parameters from Algorithm 1.

<footnote n="7">⁷ By “real alignment” we mean the expected values of alignment that we want to strive for with our learning model.</footnote>

<footnote n="8">⁸ The 100 OD pairs come from real routes from [53], but the actual routes are not used here. We always use generated ones as described in Section 5.1.</footnote>
## 5.1 Generating routes

Because of the lack of real world data, we generate the route data for given agents artificially. Here we assume that all agents act according to a given profile *P* and will select the "cheapest" route for each OD pair according to some edge costs that depend on edge properties and the agent's profile. We define the cost for an edge *e* given a certain profile *P*, or shorter, the *profiled cost*, $C_P(e)$ by:

$$C_P(e) = P \cdot (fuel(e), insec(e), time(e)) \quad (6)$$

This formula calculates the cost of a given edge as the weighted average of fuel consumption, insecurity level and expected edge traversal time, where the profile act as preference weights. We also define the *profiled cost of a route* *R* by:

$$C_P(R) = \sum_{e \in R} P \cdot (fuel(e), insec(e), time(e)) \quad (7)$$

It should be noted that the profile for generating an agent's routes actually defines its value system and each individual value (*sus*, *sec*, and *eff*) is directly grounded in exactly one of the properties (fuel consumption, insecurity and traversal time, respectively). This implies, that in our case the value grounding learning task consists in fact in identifying the would be perfectly solved if $W^{\theta} = I_{3 \times 3}$ (i.e. the alignment of a value is defined uniquely by the corresponding property of the edge). On the other hand, the value system identification task would be solved if we find a profile $\hat{P} = softmax(\alpha)$ that approximates an agents' profile *P*.

Given the above, we can define the cost of an edge according to a value *v* as $C_v(e) = C_{P_v}(e)$ and consider this cost to be the real<fnref n="7" /> alignment with each value in this evaluation, i.e. $C_v(e) = -R(e) = A(e, v) \in \mathbb{R}^+$. Thus, we can define the alignment of a route or its cost, too, as $C_v(R) = C_{P_v}(R)$.

## 5.2 Evaluation: Value grounding learning

Here we analyze the performance of Algorithm 1 with a train-test set composed of *pure routes*—routes that are considered totally sustainable, secure or time efficient, e.g., each route is generated with a pure profile ((1, 0, 0), (0, 1, 0) and (0, 0, 1))—. In particular, we used the pure routes for 100 different OD pairs<fnref n="8" />. The training-test ratio has been 80%–20%, the learning rate $\lambda = 0.2$ and the batch size equal to the training data size. We run the experiment with 10 random splits and present average values of the 10 runs in the results.

We want to check whether the function $G_{\theta}$ is close to the expected function, that is, if the matrix $W^{\theta}$ (see Equation 3) is close to the identity. In Figure 3 we see that this is the case. Thus, we have almost perfectly identified the alignment functions with each value. Besides, we can see that the policies that maximize learnt rewards (alignments with each individual value) generate routes that are equivalent to the observed input data after about 20 iterations.

<figure><img src="image_4.png" /><figcaption>**Figure 3.** Expected matrix $W^{\theta}$ (defining value alignment functions $G_{\theta}$) after running Algorithm 1 with configuration from Section 5.2. It represents the learned linear correlation between real costs $C_{P_{v_i}}$ and learned ones $-R_{P_{v_i}}$, with standard deviations.</figcaption></figure>

The results for value grounding learning are conclusive insofar as our method works well in the given use case, correctly identifying the feature behind that grounds each of the value. Still, it has been explored how far less informative features and/or less informed learning architectures lead to similar performances.

<table><caption>**Table 1.** Results for value system identification, learning from individual agents with different profiles. Presented are the learned profile and the average value alignments of the routes (from the test set) sampled with the original profile and the learned profile are shown for the three values sus, sec and eff (OrigAC$v_i$ and LearAC$v_i$, respectively).</caption><thead><tr><th>Original Profile</th><th>Learned Profile</th><th>OrigACSus</th><th>OrigACSec</th><th>OrigACEff</th><th>LearACSus</th><th>LearACSec</th><th>LearACEff</th></tr></thead><tbody><tr><td>(0.0, 0.0, 1.0)</td><td>(0.01, 0.01, 0.98)</td><td>0.389</td><td>0.591</td><td>0.236</td><td>0.389</td><td>0.590</td><td>0.236</td></tr><tr><td>(0.0, 0.25, 0.75)</td><td>(0.01, 0.04, 0.95)</td><td>0.391</td><td>0.588</td><td>0.236</td><td>0.390</td><td>0.584</td><td>0.236</td></tr><tr><td>(0.0, 0.5, 0.5)</td><td>(0.02, 0.14, 0.84)</td><td>0.390</td><td>0.576</td><td>0.237</td><td>0.388</td><td>0.568</td><td>0.238</td></tr><tr><td>(0.0, 0.75, 0.25)</td><td>(0.02, 0.17, 0.81)</td><td>0.388</td><td>0.559</td><td>0.240</td><td>0.388</td><td>0.564</td><td>0.239</td></tr><tr><td>(0.0, 1.0, 0.0)</td><td>(0.0, 0.99, 0.0)</td><td>0.598</td><td>0.266</td><td>1.028</td><td>0.591</td><td>0.266</td><td>1.009</td></tr><tr><td>(0.25, 0.0, 0.75)</td><td>(0.06, 0.01, 0.93)</td><td>0.386</td><td>0.594</td><td>0.236</td><td>0.386</td><td>0.591</td><td>0.236</td></tr><tr><td>(0.25, 0.25, 0.5)</td><td>(0.07, 0.04, 0.89)</td><td>0.385</td><td>0.585</td><td>0.236</td><td>0.385</td><td>0.584</td><td>0.236</td></tr><tr><td>(0.25, 0.5, 0.25)</td><td>(0.11, 0.13, 0.76)</td><td>0.382</td><td>0.569</td><td>0.238</td><td>0.383</td><td>0.569</td><td>0.238</td></tr><tr><td>(0.25, 0.75, 0.0)</td><td>(0.3, 0.68, 0.03)</td><td>0.480</td><td>0.297</td><td>0.722</td><td>0.481</td><td>0.296</td><td>0.723</td></tr><tr><td>(0.5, 0.0, 0.5)</td><td>(0.22, 0.02, 0.76)</td><td>0.381</td><td>0.597</td><td>0.237</td><td>0.377</td><td>0.589</td><td>0.238</td></tr><tr><td>(0.5, 0.25, 0.25)</td><td>(0.28, 0.08, 0.64)</td><td>0.378</td><td>0.579</td><td>0.238</td><td>0.376</td><td>0.574</td><td>0.239</td></tr><tr><td>(0.5, 0.5, 0.0)</td><td>(0.59, 0.39, 0.02)</td><td>0.411</td><td>0.369</td><td>0.496</td><td>0.413</td><td>0.364</td><td>0.501</td></tr><tr><td>(0.75, 0.0, 0.25)</td><td>(0.39, 0.01, 0.6)</td><td>0.372</td><td>0.600</td><td>0.239</td><td>0.371</td><td>0.597</td><td>0.240</td></tr><tr><td>(0.75, 0.25, 0.0)</td><td>(0.84, 0.16, 0.0)</td><td>0.366</td><td>0.496</td><td>0.319</td><td>0.366</td><td>0.497</td><td>0.316</td></tr><tr><td>(1.0, 0.0, 0.0)</td><td>(0.95, 0.02, 0.03)</td><td>0.359</td><td>0.564</td><td>0.273</td><td>0.359</td><td>0.564</td><td>0.271</td></tr></tbody></table>

## 5.3 Evaluation: Value system identification

For the value system identification task we have analyzed two different cases: i) “learning from an individual agent” and ii) “learning from a society of agents”.

By learning from an individual agent we mean that we try to identify its unknown profile *P* from route examples of that agent. Here we assume that the routes always minimize the cost $C_P$, for the agent's profile *P*. In particular, we want to find a profile $\hat{P}$ such that the optimal policy for the rewards $\hat{R}_P$ proposes similar routes to the once chosen of the agent in the examples in terms of their expected value alignment as measured by the cumulative costs $C_{v_i}(R)$. The reward $\hat{R}_P$ uses $G_{\theta}$ as the first layer with the learnt parameters from Algorithm 1.

<footnote n="7">⁷ By “real alignment” we mean the expected values of alignment that we want to strive for with our learning model.</footnote>

<footnote n="8">⁸ The 100 OD pairs come from real routes from [53], but the actual routes are not used here. We always use generated ones as described in Section 5.1.</footnote>
<table><caption>**Table 2.** Results for value system identification, learning from different profiled societies. The presented information is analogous to Table 1.</caption><thead><tr><th>Original Profile</th><th>Learned Profile</th><th>OrigAC<sub>Sus</sub></th><th>OrigAC<sub>Sec</sub></th><th>OrigAC<sub>Eff</sub></th><th>LearAC<sub>Sus</sub></th><th>LearAC<sub>Sec</sub></th><th>LearAC<sub>Eff</sub></th></tr></thead><tbody><tr><td>(0.0, 0.0, 1.0)</td><td>(0.01, 0.01, 0.98)</td><td>0.399</td><td>0.602</td><td>0.229</td><td>0.398</td><td>0.599</td><td>0.229</td></tr><tr><td>(0.0, 0.25, 0.75)</td><td>(0.68, 0.31, 0.01)</td><td>0.458</td><td>0.518</td><td>0.446</td><td>0.400</td><td>0.427</td><td>0.401</td></tr><tr><td>(0.0, 0.5, 0.5)</td><td>(0.51, 0.48, 0.01)</td><td>0.504</td><td>0.437</td><td>0.629</td><td>0.441</td><td>0.357</td><td>0.542</td></tr><tr><td>(0.0, 0.75, 0.25)</td><td>(0.28, 0.71, 0.01)</td><td>0.565</td><td>0.356</td><td>0.842</td><td>0.519</td><td>0.294</td><td>0.793</td></tr><tr><td>(0.0, 1.0, 0.0)</td><td>(0.0, 0.99, 0.0)</td><td>0.614</td><td>0.275</td><td>1.035</td><td>0.598</td><td>0.275</td><td>0.999</td></tr><tr><td>(0.25, 0.0, 0.75)</td><td>(0.31, 0.12, 0.56)</td><td>0.392</td><td>0.598</td><td>0.240</td><td>0.386</td><td>0.588</td><td>0.235</td></tr><tr><td>(0.25, 0.25, 0.5)</td><td>(0.72, 0.27, 0.0)</td><td>0.444</td><td>0.522</td><td>0.431</td><td>0.395</td><td>0.440</td><td>0.384</td></tr><tr><td>(0.25, 0.5, 0.25)</td><td>(0.54, 0.45, 0.01)</td><td>0.497</td><td>0.434</td><td>0.638</td><td>0.431</td><td>0.370</td><td>0.510</td></tr><tr><td>(0.25, 0.75, 0.0)</td><td>(0.33, 0.66, 0.01)</td><td>0.551</td><td>0.352</td><td>0.840</td><td>0.502</td><td>0.303</td><td>0.746</td></tr><tr><td>(0.5, 0.0, 0.5)</td><td>(0.61, 0.07, 0.32)</td><td>0.384</td><td>0.590</td><td>0.249</td><td>0.376</td><td>0.592</td><td>0.244</td></tr><tr><td>(0.5, 0.25, 0.25)</td><td>(0.74, 0.25, 0.0)</td><td>0.440</td><td>0.507</td><td>0.456</td><td>0.392</td><td>0.447</td><td>0.372</td></tr><tr><td>(0.5, 0.5, 0.0)</td><td>(0.52, 0.47, 0.01)</td><td>0.491</td><td>0.426</td><td>0.652</td><td>0.441</td><td>0.357</td><td>0.541</td></tr><tr><td>(0.75, 0.0, 0.25)</td><td>(0.73, 0.05, 0.22)</td><td>0.377</td><td>0.583</td><td>0.262</td><td>0.373</td><td>0.595</td><td>0.249</td></tr><tr><td>(0.75, 0.25, 0.0)</td><td>(0.7, 0.3, 0.0)</td><td>0.436</td><td>0.500</td><td>0.474</td><td>0.401</td><td>0.425</td><td>0.406</td></tr><tr><td>(1.0, 0.0, 0.0)</td><td>(0.92, 0.02, 0.06)</td><td>0.371</td><td>0.580</td><td>0.271</td><td>0.371</td><td>0.585</td><td>0.267</td></tr></tbody></table>

In the case of "learning from a society", we try to learn the value system of a society of agents, where different agents may have different profiles. In particular, we analyse societies where different proportions of agents will have different pure profiles. In this sense a profile *P* for the society specifies the proportions of agents that purely prioritise the values *sus*, *sec* and *eff*. For example, given a profile (0.7, 0.3, 0) we consider that given a certain *OD* pair, the probability of choosing the shortest route according to *C*<sub>sus</sub> is 0.7; according to *C*<sub>sec</sub> is 0.3; and according to *C*<sub>eff</sub> is 0. With our approach, we learn a reward function that is used to generate an optimal policy which sample similar routes to the ones observed. However, in this scenario this will not be possible in general because the learnt policy is deterministic and the reference one can be considered stochastic. However, as we will see, we are able to learn profiles that approximate the society profiles and policies that obtain routes with similar value alignment (on average) to the routes form the society.

Both scenarios are approached with Algorithm 2 using the same training OD pairs as the ones used in the last task. The test set is composed of the remaining 20% of OD pairs and other random OD pairs separated by minimum 5 edge connections, for a total of 100 test ODs. In the second scenario we will have 3 types of routes per OD pair (one per value). The learning rate is again $\lambda = 0.2$ and the maximum number of iterations is $N = 80$. Batch size *b* in the first scenario was set equal to the training data size (80), and in the second case it was set to 200 (routes are generated among the training set OD pairs respecting the probability distribution in the society profile).

With the specified settings, we analyze the performance of the value system identification task applied to 15 different original profiles (evenly distributed in the space of possible profiles) for both, learning from individual agents and societies. The results are summarized in Table 1 (learning from individual agents) and Table 2 (learning from societies). For each original profile, we present the learned profile and compare the average cost (or alignment) w.r.t. each of the values (the average values costs $OrigAC_{v_i}$) of the original routes and the routes generated by the learned policy ($LearAC_{v_i}$).

In general, we found policies that align well with both profiled societies and individual agents in terms of the expected alignments. Also, in both cases, we learn the pure profiles for each value with high accuracy, i.e. we learn a profile of (0.01, 0.01, 0.98) from demonstrations from a efficient agent with profile (0,0,1).

However, when learning from individuals and the original profiles are not pure, the learned profiles can sometimes differ significantly from the original ones, despite the resulting routes being nearly iden-

tical in terms of expected costs. This discrepancy arises because there exist correlations between different values (alignments). In our scenario, this seems to be the case between sustainability and efficiency, and both are almost inversely correlated with security. As a result, it is highly probable that a route which is efficient will also be sustainable but not secure, and vice versa. For instance, if an agent always chooses the most efficient routes and those are almost always identical to the most sustainable routes, from an observer's point of view it is not possible to distinguish whether the agent prefers efficiency over sustainability or the other way around. Thus, learning a different profile may actually not be wrong. It may indicate that both profiles have the same observable effect with respect to value alignment. The effect can also be observed when learning from societies.

In the case of learning from societies, the similarity in the value alignment is lower, especially when these societies are heterogeneous, e.g. (0.0, 0.75, 0.25)). This will be due to the intrinsic impossibility to perfectly reproduce the observed routes with a deterministic policy that is optimal for a linear combination reward function.

Finally, it is noticeable that, as it can be expected, the route costs (alignments) for a value decrease when the preference weight for that value in the profiles increases and vice versa. However, there are cases where this effect may not be gradual—e.g. the first 5 columns of Table 1. This effect may be due to different distributions of property values in the edges of the network. In particular, despite the normalization, *insec(e)* is generally low in most edges and achieves its higher values solely in certain edges, whereas *time* generally has higher values and its lowest values are only in a small share of edges. This issue suggests that in some domains a linear weighting of value alignments might not be informative enough of agent preferences.

# 6 Conclusions and Future Work

We have introduced an initial version of a value system learning problem, which aim is reckoning both value semantics and value preferences (value systems) from human demonstrations in a given domain. We proposed a preliminary value system representation model adapted to solve this problem, capable of jointly representing socially-agreed alignments with individual values and agent-specific value preferences in a quantitative manner. A solution to the problem has been implemented and tested in a route choice environment using Inverse Reinforcement Learning. We learned route choice behaviors aligned with values and coherent with heterogeneous agent value preferences, proving the applicability of our framework.

Future lines of work include extending it to more sophisticated environments with variable traffic conditions. Also, using real annotated data would provide additional insights on the claimed definitions of values versus theoretical assumptions and models. Extending and generalizing our value system representation model to be fully compatible with other frameworks is certainly of interest (e.g. [32]), so we will look into incorporating context-dependency of value groundings and preferences into our model.

# Acknowledgements

This work has been supported by grant VAE: TED2021-131295B-C33 funded by MCIN/AEI/ 10.13039/501100011033 and by the “European Union NextGenerationEU/PRTR”, by grant COSASS: PID2021-123673OB-C32 funded by MCIN/AEI/ 10.13039/501100011033 and by “ERDF A way of making Europe”, and by the AGROBOTS Project of Universidad Rey Juan Carlos funded by the Community of Madrid, Spain.
<table><caption>**Table 2.** Results for value system identification, learning from different profiled societies. The presented information is analogous to Table 1.</caption><thead><tr><th>Original Profile</th><th>Learned Profile</th><th>OrigAC<sub>Sus</sub></th><th>OrigAC<sub>Sec</sub></th><th>OrigAC<sub>Eff</sub></th><th>LearAC<sub>Sus</sub></th><th>LearAC<sub>Sec</sub></th><th>LearAC<sub>Eff</sub></th></tr></thead><tbody><tr><td>(0.0, 0.0, 1.0)</td><td>(0.01, 0.01, 0.98)</td><td>0.399</td><td>0.602</td><td>0.229</td><td>0.398</td><td>0.599</td><td>0.229</td></tr><tr><td>(0.0, 0.25, 0.75)</td><td>(0.68, 0.31, 0.01)</td><td>0.458</td><td>0.518</td><td>0.446</td><td>0.400</td><td>0.427</td><td>0.401</td></tr><tr><td>(0.0, 0.5, 0.5)</td><td>(0.51, 0.48, 0.01)</td><td>0.504</td><td>0.437</td><td>0.629</td><td>0.441</td><td>0.357</td><td>0.542</td></tr><tr><td>(0.0, 0.75, 0.25)</td><td>(0.28, 0.71, 0.01)</td><td>0.565</td><td>0.356</td><td>0.842</td><td>0.519</td><td>0.294</td><td>0.793</td></tr><tr><td>(0.0, 1.0, 0.0)</td><td>(0.0, 0.99, 0.0)</td><td>0.614</td><td>0.275</td><td>1.035</td><td>0.598</td><td>0.275</td><td>0.999</td></tr><tr><td>(0.25, 0.0, 0.75)</td><td>(0.31, 0.12, 0.56)</td><td>0.392</td><td>0.598</td><td>0.240</td><td>0.386</td><td>0.588</td><td>0.235</td></tr><tr><td>(0.25, 0.25, 0.5)</td><td>(0.72, 0.27, 0.0)</td><td>0.444</td><td>0.522</td><td>0.431</td><td>0.395</td><td>0.440</td><td>0.384</td></tr><tr><td>(0.25, 0.5, 0.25)</td><td>(0.54, 0.45, 0.01)</td><td>0.497</td><td>0.434</td><td>0.638</td><td>0.431</td><td>0.370</td><td>0.510</td></tr><tr><td>(0.25, 0.75, 0.0)</td><td>(0.33, 0.66, 0.01)</td><td>0.551</td><td>0.352</td><td>0.840</td><td>0.502</td><td>0.303</td><td>0.746</td></tr><tr><td>(0.5, 0.0, 0.5)</td><td>(0.61, 0.07, 0.32)</td><td>0.384</td><td>0.590</td><td>0.249</td><td>0.376</td><td>0.592</td><td>0.244</td></tr><tr><td>(0.5, 0.25, 0.25)</td><td>(0.74, 0.25, 0.0)</td><td>0.440</td><td>0.507</td><td>0.456</td><td>0.392</td><td>0.447</td><td>0.372</td></tr><tr><td>(0.5, 0.5, 0.0)</td><td>(0.52, 0.47, 0.01)</td><td>0.491</td><td>0.426</td><td>0.652</td><td>0.441</td><td>0.357</td><td>0.541</td></tr><tr><td>(0.75, 0.0, 0.25)</td><td>(0.73, 0.05, 0.22)</td><td>0.377</td><td>0.583</td><td>0.262</td><td>0.373</td><td>0.595</td><td>0.249</td></tr><tr><td>(0.75, 0.25, 0.0)</td><td>(0.7, 0.3, 0.0)</td><td>0.436</td><td>0.500</td><td>0.474</td><td>0.401</td><td>0.425</td><td>0.406</td></tr><tr><td>(1.0, 0.0, 0.0)</td><td>(0.92, 0.02, 0.06)</td><td>0.371</td><td>0.580</td><td>0.271</td><td>0.371</td><td>0.585</td><td>0.267</td></tr></tbody></table>

In the case of "learning from a society", we try to learn the value system of a society of agents, where different agents may have different profiles. In particular, we analyse societies where different proportions of agents will have different pure profiles. In this sense a profile *P* for the society specifies the proportions of agents that purely prioritise the values *sus*, *sec* and *eff*. For example, given a profile (0.7, 0.3, 0) we consider that given a certain *OD* pair, the probability of choosing the shortest route according to *C*<sub>sus</sub> is 0.7; according to *C*<sub>sec</sub> is 0.3; and according to *C*<sub>eff</sub> is 0. With our approach, we learn a reward function that is used to generate an optimal policy which sample similar routes to the ones observed. However, in this scenario this will not be possible in general because the learnt policy is deterministic and the reference one can be considered stochastic. However, as we will see, we are able to learn profiles that approximate the society profiles and policies that obtain routes with similar value alignment (on average) to the routes form the society.

Both scenarios are approached with Algorithm 2 using the same training OD pairs as the ones used in the last task. The test set is composed of the remaining 20% of OD pairs and other random OD pairs separated by minimum 5 edge connections, for a total of 100 test ODs. In the second scenario we will have 3 types of routes per OD pair (one per value). The learning rate is again $\lambda = 0.2$ and the maximum number of iterations is $N = 80$. Batch size *b* in the first scenario was set equal to the training data size (80), and in the second case it was set to 200 (routes are generated among the training set OD pairs respecting the probability distribution in the society profile).

With the specified settings, we analyze the performance of the value system identification task applied to 15 different original profiles (evenly distributed in the space of possible profiles) for both, learning from individual agents and societies. The results are summarized in Table 1 (learning from individual agents) and Table 2 (learning from societies). For each original profile, we present the learned profile and compare the average cost (or alignment) w.r.t. each of the values (the average values costs $OrigAC_{v_i}$) of the original routes and the routes generated by the learned policy ($LearAC_{v_i}$).

In general, we found policies that align well with both profiled societies and individual agents in terms of the expected alignments. Also, in both cases, we learn the pure profiles for each value with high accuracy, i.e. we learn a profile of (0.01, 0.01, 0.98) from demonstrations from a efficient agent with profile (0,0,1).

However, when learning from individuals and the original profiles are not pure, the learned profiles can sometimes differ significantly from the original ones, despite the resulting routes being nearly iden-

tical in terms of expected costs. This discrepancy arises because there exist correlations between different values (alignments). In our scenario, this seems to be the case between sustainability and efficiency, and both are almost inversely correlated with security. As a result, it is highly probable that a route which is efficient will also be sustainable but not secure, and vice versa. For instance, if an agent always chooses the most efficient routes and those are almost always identical to the most sustainable routes, from an observer's point of view it is not possible to distinguish whether the agent prefers efficiency over sustainability or the other way around. Thus, learning a different profile may actually not be wrong. It may indicate that both profiles have the same observable effect with respect to value alignment. The effect can also be observed when learning from societies.

In the case of learning from societies, the similarity in the value alignment is lower, especially when these societies are heterogeneous, e.g. (0.0, 0.75, 0.25)). This will be due to the intrinsic impossibility to perfectly reproduce the observed routes with a deterministic policy that is optimal for a linear combination reward function.

Finally, it is noticeable that, as it can be expected, the route costs (alignments) for a value decrease when the preference weight for that value in the profiles increases and vice versa. However, there are cases where this effect may not be gradual—e.g. the first 5 columns of Table 1. This effect may be due to different distributions of property values in the edges of the network. In particular, despite the normalization, *insec(e)* is generally low in most edges and achieves its higher values solely in certain edges, whereas *time* generally has higher values and its lowest values are only in a small share of edges. This issue suggests that in some domains a linear weighting of value alignments might not be informative enough of agent preferences.

# 6 Conclusions and Future Work

We have introduced an initial version of a value system learning problem, which aim is reckoning both value semantics and value preferences (value systems) from human demonstrations in a given domain. We proposed a preliminary value system representation model adapted to solve this problem, capable of jointly representing socially-agreed alignments with individual values and agent-specific value preferences in a quantitative manner. A solution to the problem has been implemented and tested in a route choice environment using Inverse Reinforcement Learning. We learned route choice behaviors aligned with values and coherent with heterogeneous agent value preferences, proving the applicability of our framework.

Future lines of work include extending it to more sophisticated environments with variable traffic conditions. Also, using real annotated data would provide additional insights on the claimed definitions of values versus theoretical assumptions and models. Extending and generalizing our value system representation model to be fully compatible with other frameworks is certainly of interest (e.g. [32]), so we will look into incorporating context-dependency of value groundings and preferences into our model.

# Acknowledgements

This work has been supported by grant VAE: TED2021-131295B-C33 funded by MCIN/AEI/ 10.13039/501100011033 and by the "European Union NextGenerationEU/PRTR", by grant COSASS: PID2021-123673OB-C32 funded by MCIN/AEI/ 10.13039/501100011033 and by "ERDF A way of making Europe", and by the AGROBOTS Project of Universidad Rey Juan Carlos funded by the Community of Madrid, Spain.
## References

[1] P. Abbeel and A. Y. Ng. Apprenticeship learning via inverse reinforcement learning. In *Proceedings of the Twenty-First International Conference on Machine Learning*, ICML '04, page 1, New York, NY, USA, 2004. Association for Computing Machinery. ISBN 1581138385. doi: 10.1145/1015330.1015430. URL https://doi.org/10.1145/1015330.1015430.

[2] M. Anderson and S. L. Anderson. Ethel: Toward a principled ethical eldercare system. *AAAI fall symposium: AI in eldercare: New solutions to old problems*, 2, 2008.

[3] M. Anderson and S. L. Anderson. Geneth: A general ethical dilemma analyzer. *Paladyn*, 9:337–357, 2 2018. ISSN 20814836. doi: 10.1515/PJBR-2018-0024/MACHINEREADABLECITATION/RIS. URL https://www.degruyter.com/document/doi/10.1515/pjbr-2018-0024/html.

[4] M. Anderson, S. L. Anderson, and C. Armen. An approach to computing ethics. *IEEE Intelligent Systems*, 21:56–63, 7 2006. ISSN 15411672. doi: 10.1109/MIS.2006.64.

[5] T. Bench-Capon, K. Atkinson, and P. McBurney. Using argumentation to model agent decision making in economic experiments. *Autonomous Agents and Multi-Agent Systems*, 25:183–208, 2012.

[6] P. Christiano, J. Leike, T. B. Brown, M. Martic, S. Legg, and D. Amodei. Deep reinforcement learning from human preferences, 2023.

[7] J. Fu, K. Luo, and S. Levine. Learning robust rewards with adverse-rial inverse reinforcement learning. In *International Conference on Learning Representations*, 2018. URL https://openreview.net/forum?id=rkHywl-A-.

[8] J. Fürnkranz, E. Hüllermeier, W. Cheng, and S.-H. Park. Preference-based reinforcement learning: a formal framework and a policy iteration algorithm. *Machine learning*, 89:123–156, 2012.

[9] I. Gabriel. Artificial intelligence, values, and alignment. *Minds and Machines*, 30:411–437, 9 2020. ISSN 15728641. doi: 10.1007/S11023-020-09539-2.

[10] J. Graham, J. Haidt, S. Koleva, M. Motyl, R. Iyer, S. P. Wojcik, and P. H. Ditto. Chapter two - moral foundations theory: The pragmatic validity of moral pluralism. In P. Devine and A. Plant, editors, *Advances in Experimental Social Psychology*, volume 47, pages 55–130. Academic Press, 2013. doi: https://doi.org/10.1016/B978-0-12-407236-7.00002-4. URL https://www.sciencedirect.com/science/article/pii/B9780124072367000024.

[11] D. Hadfield-Menell, S. J. Russell, P. Abbeel, and A. Dragan. Co-operative inverse reinforcement learning. In D. Lee, M. Sugiyama, U. Luxburg, I. Guyon, and R. Garnett, editors, *Advances in Neural Information Processing Systems*, volume 29. Curran Associates, Inc., 2016.

[12] A. Holgado-Sánchez, H. Billhardt, S. Ossowski, and A. Fernández. An ontology for value awareness engineering. In *Proceedings of the 16th International Conference on Agents and Artificial Intelligence - Volume 3: AWAI*, pages 1421–1428. INSTICC, SciTePress, 2024. ISBN 978-989-758-680-4. doi: 10.5220/0012595500003636.

[13] E. T. Jaynes. Information theory and statistical mechanics. *Phys. Rev.*, 106:620–630, May 1957. doi: 10.1103/PhysRev.106.620. URL https://link.aps.org/doi/10.1103/PhysRev.106.620.

[14] J. Jiang and Z. Lu. Learning fairness in multi-agent systems. *Advances in Neural Information Processing Systems*, 32, 2019.

[15] G. Kalweit, M. Huegle, M. Werling, and J. Boedecker. Deep inverse q-learning with constraints. *Advances in Neural Information Processing Systems*, 2020-December, 8 2020. ISSN 10495258. URL https://arxiv.org/abs/2008.01712v1.

[16] M. Karanik, H. Billhardt, A. Fernández, and S. Ossowski. On the relevance of value system structure for automated value-aligned decision-making. In *Proceedings of the 39th ACM/SIGAPP Symposium on Applied Computing*, pages 679–686. Association for Computing Machinery, 2024. ISBN 9798400702433. doi: 10.1145/3605098.3636057. URL https://doi.org/10.1145/3605098.3636057.

[17] T. Koch and E. Dugundji. A review of methods to model route choice behavior of bicyclists: inverse reinforcement learning in spatial context and recursive logit. In *Proceedings of the 3rd ACM SIGSPATIAL International Workshop on GeoSpatial Simulation*, GeoSim '20, page 30–37, New York, NY, USA, 2020. Association for Computing Machinery. ISBN 9781450381611. doi: 10.1145/3423335.3428165. URL https://doi.org/10.1145/3423335.3428165.

[18] R. Lera-Leri, F. Bistaffa, M. Serramia, M. Lopez-Sanchez, and J. Rodriguez-Aguilar. Towards pluralistic value alignment: Aggregating value systems through lp-regression. In *Proceedings of the 21st International Conference on Autonomous Agents and Multiagent Systems*, AAMAS '22, page 780–788. IFAAMAS, 2022. ISBN 9781450392136.

[19] R. X. L. Leri, E. Liscio, F. Bistaffa, C. M. Jonker, M. Lopez-Sanchez, P. K. Murukannaiah, J. A. Rodriguez-Aguilar, and F. Salas-Molina. Aggregating value systems for decision support. *Knowledge-Based Systems*, 287:111453, 2024. ISSN 0950-7051. doi: https://doi.org/10.1016/j.knosys.2024.111453. URL https://www.sciencedirect.com/science/article/pii/S0950705124000881.

[20] S. Levine, Z. Popovic, and V. Koltun. Nonlinear inverse reinforcement learning with gaussian processes. In J. Shawe-Taylor, R. Zemel, P. Bartlett, F. Pereira, and K. Weinberger, editors, *Advances in Neural Information Processing Systems*, volume 24. Curran Associates, Inc., 2011. URL https://proceedings.neurips.cc/paper_files/paper/2011/file/c51ce410c124a10e0db5e4b97fc2af39-Paper.pdf.

[21] E. Liscio, M. van der Meer, L. Cavalcante Siebert, N. Mouter, C. Jonker, and P. Murukannaiah. Axies: Identifying and evaluating context-specific values. In *Proceedings of the 20th International Conference on Autonomous Agents and MultiAgent Systems*, AAMAS '21, pages 799–808. International Foundation for Autonomous Agents and Multiagent Systems, 2021. 20th International Conference on Autonomous Agents and Multiagent Systems, AAMAS 2021 ; Conference date: 03-05-2021 Through 07-05-2021.

[22] E. Liscio, A. Dondera, A. Geadau, C. Jonker, and P. Murukannaiah. Cross-domain classification of moral values. In M. Carpuat, M.-C. de Marneffe, and I. V. Meza Ruiz, editors, *Findings of the Association for Computational Linguistics: NAACL 2022*, pages 2727–2745, Seattle, United States, July 2022. Association for Computational Linguistics. doi: 10.18653/v1/2022.findings-naacl.202. URL https://aclanthology.org/2022.findings-naacl.202.

[23] E. Liscio, R. Lera-Leri, F. Bistaffa, R. I. Dobbe, C. M. Jonker, M. Lopez-Sanchez, J. A. Rodriguez-Aguilar, and P. K. Murukannaiah. Value inference in sociotechnical systems. In *Proceedings of the 2023 International Conference on Autonomous Agents and Multiagent Systems*, AAMAS '23, page 1774–1780, Richland, SC, 2023. International Foundation for Autonomous Agents and Multiagent Systems. ISBN 9781450394321. doi: 10.5555/3545946.3598838.

[24] E. Liscio, R. Lera-Leri, F. Bistaffa, R. I. J. Dobbe, C. M. Jonker, M. Lopez-Sanchez, J. A. Rodriguez-Aguilar, and P. K. Murukannaiah. Inerring values via hybrid intelligence. In *HHAI 2023: Augmenting Human Intellect: Proceedings of the Second International Conference on Hybrid Human-Artificial Intelligence*, volume 368 of *Frontiers in Artificial Intelligence and Applications*, pages 373–378. IOS Press BV, 6 2023. doi: 10.3233/FAIA230102.

[25] S. Liu and H. Jiang. Personalized route recommendation for ride-hailing with deep inverse reinforcement learning and real-time traffic conditions. *Transportation Research Part E: Logistics and Transportation Review*, 164:102780, 2022. ISSN 1366-5545. doi: https://doi.org/10.1016/j.tre.2022.102780. URL https://www.sciencedirect.com/science/article/pii/S1366554522001715.

[26] S. Liu, M. Araujo, E. Brunskill, R. Rossetti, J. Barros, and R. Krishnan. Understanding sequential decisions via inverse reinforcement learning. In *2013 IEEE 14th International Conference on Mobile Data Management*, volume 1, pages 177–186, 2013. doi: 10.1109/MDM.2013.28.

[27] S. Liu, H. Jiang, S. Chen, J. Ye, R. He, and Z. Sun. Integrating dijkstra's algorithm into deep inverse reinforcement learning for food delivery route planning. *Transportation Research Part E: Logistics and Transportation Review*, 142:102070, 2020.

[28] R. Lovreglio, A. Fonzone, and L. dell'Olio. A mixed logit model for predicting exit choice during building evacuations. *Transportation Research Part A: Policy and Practice*, 92:59–75, 2016. doi: 10.1016/j.tra.2016.06.018.

[29] N. Montes and C. Sierra. Synthesis and properties of optimally value-aligned normative systems. *Journal of Artificial Intelligence Research*, 74:1739–1774, 2022. doi: 10.1613/jair.1.13487.

[30] N. Montes, N. Osman, C. Sierra, and M. Slavkovik. Value engineering for autonomous agents. *CoRR*, abs/2302.08759, 2023. doi: 10.48550/arXiv.2302.08759. URL https://doi.org/10.48550/arXiv.2302.08759.

[31] A. Y. Ng and S. J. Russell. Algorithms for inverse reinforcement learning. In *Proceedings of the Seventeenth International Conference on Machine Learning*, ICML '00, page 663–670, San Francisco, CA, USA, 2000. Morgan Kaufmann Publishers Inc. ISBN 1558607072.

[32] N. Osman and M. d'Inverno. A computational framework of human values. In *Proceedings of the 23rd International Conference on Autonomous Agents and Multiagent Systems*, AAMAS '24, page 1531–1539, Richland, SC, 2024. International Foundation for Autonomous Agents and Multiagent Systems. ISBN 9798400704864.

[33] C. G. Prato. Route choice modeling: past, present and future research directions. *Journal of Choice Modelling*, 2(1):65–100, 2009. ISSN 1755-5345. doi: https://doi.org/10.1016/S1755-5345(13)70005-8. URL https://www.sciencedirect.com/science/article/pii/
References

[1] P. Abbeel and A. Y. Ng. Apprenticeship learning via inverse reinforcement learning. In *Proceedings of the Twenty-First International Conference on Machine Learning*, ICML '04, page 1, New York, NY, USA, 2004. Association for Computing Machinery. ISBN 1581138385. doi: 10.1145/1015330.1015430. URL https://doi.org/10.1145/1015330.1015430.

[2] M. Anderson and S. L. Anderson. Ethel: Toward a principled ethical eldercare system. *AAAI fall symposium: AI in eldercare: New solutions to old problems*, 2, 2008.

[3] M. Anderson and S. L. Anderson. Geneth: A general ethical dilemma analyzer. *Paladyn*, 9:337–357, 2 2018. ISSN 20814836. doi: 10.1515/PJBR-2018-0024/MACHINEREADABLECITATION/RIS. URL https://www.degruyter.com/document/doi/10.1515/pjbr-2018-0024/html.

[4] M. Anderson, S. L. Anderson, and C. Armen. An approach to computing ethics. *IEEE Intelligent Systems*, 21:56–63, 7 2006. ISSN 15411672. doi: 10.1109/MIS.2006.64.

[5] T. Bench-Capon, K. Atkinson, and P. McBurney. Using argumentation to model agent decision making in economic experiments. *Autonomous Agents and Multi-Agent Systems*, 25:183–208, 2012.

[6] P. Christiano, J. Leike, T. B. Brown, M. Martic, S. Legg, and D. Amodei. Deep reinforcement learning from human preferences, 2023.

[7] J. Fu, K. Luo, and S. Levine. Learning robust rewards with adverse-rial inverse reinforcement learning. In *International Conference on Learning Representations*, 2018. URL https://openreview.net/forum?id=rkHywl-A-.

[8] J. Fürnkranz, E. Hüllermeier, W. Cheng, and S.-H. Park. Preference-based reinforcement learning: a formal framework and a policy iteration algorithm. *Machine learning*, 89:123–156, 2012.

[9] I. Gabriel. Artificial intelligence, values, and alignment. *Minds and Machines*, 30:411–437, 9 2020. ISSN 15728641. doi: 10.1007/S11023-020-09539-2.

[10] J. Graham, J. Haidt, S. Koleva, M. Motyl, R. Iyer, S. P. Wojcik, and P. H. Ditto. Chapter two - moral foundations theory: The pragmatic validity of moral pluralism. In *P. Devine and A. Plant, editors, Advances in Experimental Social Psychology*, volume 47, pages 55–130. Academic Press, 2013. doi: https://doi.org/10.1016/B978-0-12-407236-7.00002-4. URL https://www.sciencedirect.com/science/article/pii/B9780124072367000024.

[11] D. Hadfield-Menell, S. J. Russell, P. Abbeel, and A. Dragan. Co-operative inverse reinforcement learning. In D. Lee, M. Sugiyama, U. Luxburg, I. Guyon, and R. Garnett, editors, *Advances in Neural Information Processing Systems*, volume 29. Curran Associates, Inc., 2016.

[12] A. Holgado-Sánchez, H. Billhardt, S. Ossowski, and A. Fernández. An ontology for value awareness engineering. In *Proceedings of the 16th International Conference on Agents and Artificial Intelligence - Volume 3: AWAI*, pages 1421–1428. INSTICC, SciTePress, 2024. ISBN 978-989-758-680-4. doi: 10.5220/0012595500003636.

[13] E. T. Jaynes. Information theory and statistical mechanics. *Phys. Rev.*, 106:620–630, May 1957. doi: 10.1103/PhysRev.106.620. URL https://link.aps.org/doi/10.1103/PhysRev.106.620.

[14] J. Jiang and Z. Lu. Learning fairness in multi-agent systems. *Advances in Neural Information Processing Systems*, 32, 2019.

[15] G. Kalweit, M. Huegle, M. Werling, and J. Boedecker. Deep inverse q-learning with constraints. *Advances in Neural Information Processing Systems*, 2020-December, 8 2020. ISSN 10495258. URL https://arxiv.org/abs/2008.01712v1.

[16] M. Karanik, H. Billhardt, A. Fernández, and S. Ossowski. On the relevance of value system structure for automated value-aligned decision-making. In *Proceedings of the 39th ACM/SIGAPP Symposium on Applied Computing*, pages 679–686. Association for Computing Machinery. 2024. ISBN 978400702433. doi: 10.1145/3605098.3636057. URL https://doi.org/10.1145/3605098.3636057.

[17] T. Koch and E. Dugundji. A review of methods to model route choice behavior of bicyclists: inverse reinforcement learning in spatial context and recursive logit. In *Proceedings of the 3rd ACM SIGSPATIAL International Workshop on GeoSpatial Simulation*, GeoSim '20, page 30–37, New York, NY, USA, 2020. Association for Computing Machinery. ISBN 9781450381611. doi: 10.1145/3423335.3428165. URL https://doi.org/10.1145/3423335.3428165.

[18] R. Lera-Leri, F. Bistaffa, M. Serramia, M. Lopez-Sanchez, and J. Rodriguez-Aguilar. Towards pluralistic value alignment: Aggregating value systems through lp-regression. In *Proceedings of the 21st International Conference on Autonomous Agents and Multiagent Systems*, AAMAS '22, page 780–788. IFAAMAS, 2022. ISBN 9781450392136.

[19] R. X. L. Leri, E. Liscio, F. Bistaffa, C. M. Jonker, M. Lopez-Sanchez, P. K. Murukannaiah, J. A. Rodriguez-Aguilar, and F. Salas-Molina. Aggregating value systems for decision support. *Knowledge-Based Systems*, 287:111453, 2024. ISSN 0950-7051. doi: https://doi.org/10.1016/j.knosys.2024.111453. URL https://www.sciencedirect.com/science/article/pii/S0950705124000881.

[20] S. Levine, Z. Popovic, and V. Koltun. Nonlinear inverse reinforcement learning with gaussian processes. In J. Shawe-Taylor, R. Zemel, P. Bartlett, F. Pereira, and K. Weinberger, editors, *Advances in Neural Information Processing Systems*, volume 24. Curran Associates, Inc., 2011. URL https://proceedings.neurips.cc/paper_files/paper/2011/file/c51ce410c124a10e0db5e4b97fc2af39-Paper.pdf.

[21] E. Liscio, M. van der Meer, L. Cavalcante Siebert, N. Mouter, C. Jonker, and P. Murukannaiah. Axies: Identifying and evaluating context-specific values. In *Proceedings of the 20th International Conference on Autonomous Agents and MultiAgent Systems*, AAMAS '21, pages 799–808. International Foundation for Autonomous Agents and Multiagent Systems, 2021. 20th International Conference on Autonomous Agents and Multiagent Systems, AAMAS 2021 ; Conference date: 03-05-2021 Through 07-05-2021.

[22] E. Liscio, A. Dondera, A. Geadau, C. Jonker, and P. Murukannaiah. Cross-domain classification of moral values. In M. Carpuat, M.-C. de Marneffe, and I. V. Meza Ruiz, editors, *Findings of the Association for Computational Linguistics: NAACL 2022*, pages 2727–2745, Seattle, United States, July 2022. Association for Computational Linguistics. doi: 10.18653/v1/2022.findings-naacl.202. URL https://aclanthology.org/2022.findings-naacl.202.

[23] E. Liscio, R. Lera-Leri, F. Bistaffa, R. I. Dobbe, C. M. Jonker, M. Lopez-Sanchez, J. A. Rodriguez-Aguilar, and P. K. Murukannaiah. Value inference in sociotechnical systems. In *Proceedings of the 2023 International Conference on Autonomous Agents and Multiagent Systems*, AAMAS '23, page 1774–1780, Richland, SC, 2023. International Foundation for Autonomous Agents and Multiagent Systems. ISBN 9781450394321. doi: 10.5555/3545946.3598838.

[24] E. Liscio, R. Lera-Leri, F. Bistaffa, R. I. J. Dobbe, C. M. Jonker, M. Lopez-Sanchez, J. A. Rodriguez-Aguilar, and P. K. Murukannaiah. Inerring values via hybrid intelligence. In *HHAI 2023: Augmenting Human Intellect: Proceedings of the Second International Conference on Hybrid Human-Artificial Intelligence*, volume 368 of *Frontiers in Artificial Intelligence and Applications*, pages 373–378. IOS Press BV, 6 2023. doi: 10.3233/FAIA230102.

[25] S. Liu and H. Jiang. Personalized route recommendation for ride-hailing with deep inverse reinforcement learning and real-time traffic conditions. *Transportation Research Part E: Logistics and Transportation Review*, 164:102780, 2022. ISSN 1366-5545. doi: https://doi.org/10.1016/j.tre.2022.102780. URL https://www.sciencedirect.com/science/article/pii/S1366554522001715.

[26] S. Liu, M. Araujo, E. Brunskill, R. Rossetti, J. Barros, and R. Krishnan. Understanding sequential decisions via inverse reinforcement learning. In *2013 IEEE 14th International Conference on Mobile Data Management*, volume 1, pages 177–186, 2013. doi: 10.1109/MDM.2013.28.

[27] S. Liu, H. Jiang, S. Chen, J. Ye, R. He, and Z. Sun. Integrating dijkstra's algorithm into deep inverse reinforcement learning for food delivery route planning. *Transportation Research Part E: Logistics and Transportation Review*, 142:102070, 2020.

[28] R. Lovreglio, A. Fonzone, and L. dell'Olio. A mixed logit model for predicting exit choice during building evacuations. *Transportation Research Part A: Policy and Practice*, 92:59–75, 2016. doi: 10.1016/j.tra.2016.06.018.

[29] N. Montes and C. Sierra. Synthesis and properties of optimally value-aligned normative systems. *Journal of Artificial Intelligence Research*, 74:1739–1774, 2022. doi: 10.1613/jair.1.13487.

[30] N. Montes, N. Osman, C. Sierra, and M. Slavkovik. Value engineering for autonomous agents. *CoRR*, abs/2302.08759, 2023. doi: 10.48550/arXiv.2302.08759. URL https://doi.org/10.48550/arXiv.2302.08759.

[31] A. Y. Ng and S. J. Russell. Algorithms for inverse reinforcement learning. In *Proceedings of the Seventeenth International Conference on Machine Learning*, ICML '00, page 663–670, San Francisco, CA, USA, 2000. Morgan Kaufmann Publishers Inc. ISBN 1558607072.

[32] N. Osman and M. d'Inverno. A computational framework of human values. In *Proceedings of the 23rd International Conference on Autonomous Agents and Multiagent Systems*, AAMAS '24, page 1531–1539, Richland, SC, 2024. International Foundation for Autonomous Agents and Multiagent Systems. ISBN 9798400704864.

[33] C. G. Prato. Route choice modeling: past, present and future research directions. *Journal of Choice Modelling*, 2(1):65–100, 2009. ISSN 1755-5345. doi: https://doi.org/10.1016/S1755-5345(13)70005-8. URL https://www.sciencedirect.com/science/article/pii/
S1755534513700058.

[34] L. Qiu, Y. Zhao, J. Li, P. Lu, B. Peng, J. Gao, and S.-C. Zhu. Valuenet: A new dataset for human value driven dialogue system. *Proceedings of the AAAI Conference on Artificial Intelligence*, 36(10):11183–11191, Jun. 2022. doi: 10.1609/aaai.v36i0.21368. URL https://ojs.aaai.org/index.php/AAAI/article/view/21368.

[35] G. D. M. Ramos, W. Daamen, and S. Hoogendoorn. Modelling travellers' heterogeneous route choice behaviour as prospect maximizers. *Journal of Choice Modelling*, 6:17–33, 2013. doi: 10.1016/j.jocm.2013.04.002.

[36] L. Rizzi and J. Ortúzar. Stated preference in the valuation of interurban road safety. *Accident Analysis and Prevention*, 35:9–22, 2003. doi: 10.1016/S0001-4575(01)00082-3.

[37] M. Rodriguez-Soto, M. Serramia, M. Lopez-Sanchez, and J. A. Rodriguez-Aguilar. Instilling moral value alignment by means of multi-objective reinforcement learning. *Ethics and Information Technology*, 24:9, 3 2022. ISSN 1388-1957. doi: 10.1007/s10676-022-09635-0.

[38] S. Russell. Human compatible: AI and the problem of control. Penguin Uk, 2019.

[39] S. Russell. *Artificial Intelligence and the Problem of Control*, pages 19–24. Springer International Publishing, Cham, 2022. ISBN 978-3-030-86144-5. doi: 10.1007/978-3-030-86144-5_3. URL https://doi.org/10.1007/978-3-030-86144-5_3.

[40] A. Sadeghi-Niaraki and K. Kim. Ontology based personalized route planning system using a multi-criteria decision making approach. *Ex-pert Systems with Applications*, 36:2250–2259, 2009. doi: 10.1016/j.eswa.2007.12.053.

[41] J. Scheiner and C. Holz-Rau. Travel mode choice: Affected by objective or subjective determinants? *Transportation*, 34:487–511, 2007. doi: 10.1007/s11116-007-9112-1.

[42] S. H. Schwartz. Universals in the content and structure of values: Theoretical advances and empirical tests in 20 countries. In *Advances in experimental social psychology*, volume 25, pages 1–65. Elsevier, 1992.

[43] S. H. Schwartz. An overview of the schwarz theory of basic values. *Online readings in Psychology and Culture*, 2(1):11, 2012.

[44] M. Serramia, M. Lopez-Sanchez, J. A. Rodriguez-Aguilar, M. Rodriguez, M. Wooldridge, J. Morales, and C. Ansotegui. Moral values in norm decision making. *IFAAMAS*, 9, 2018. URL www.ifaamas.org.

[45] M. Serramia, M. Lopez-Sanchez, and J. A. Rodriguez-Aguilar. A qualitative approach to composing value-aligned norm systems. In *Proceedings of the 19th International Conference on Autonomous Agents and MultiAgent Systems*, pages 1233–1241. International Foundation for Autonomous Agents and Multiagent Systems, 2020. ISBN 9781450375184.

[46] M. Serramia, M. Rodriguez-Soto, M. López-Sánchez, J. Rodríguez-Aguilar, F. Bistaffa, P. Boddington, M. Wooldridge, and C. Ansótegui. Encoding ethics to compute value-aligned norms. *Minds and Machines*, pages 1–30, 11 2023. doi: 10.1007/s11023-023-09649-7.

[47] N. Soares. The value learning problem. *Artificial Intelligence Safety and Security*, 2018. URL https://apisemanticscholar.org/CorpusID: 13096553.

[48] C. Veronese, D. Meli, F. Bistaffa, M. Rodríguez-Soto, A. Farinelli, and J. A. Rodríguez-Aguilar. Inductive logic programming for transparent alignment with multiple moral values. In *CEUR WORKSHOP PROCEEDINGS*, volume 7, pages 84–88, 2024. ISBN 9772081415. doi: 10.2/QUERY.MIN.JS. URL https://iris.univir.it/handle/11562/1120547.

[49] L. Weidinger, K. R. McKee, R. Everett, S. Huang, T. O. Zhu, M. J. Chadwick, C. Summerfield, and I. Gabriel. Using the veil of ignorance to align ai systems with principles of justice. *Proceedings of the National Academy of Sciences*, 120(18):e2213709120, 2023. doi: 10.1073/pnas.2213709120. URL https://www.pnas.org/doi/abs/10.1073/pnas.2213709120.

[50] S. R. Wilson, Y. Shen, and R. Mihalcea. Building and validating hierarchical lexicons with a case study on personal values. In S. Staab, O. Koltsova, and D. I. Ignatov, editors, *Social Informatics*, pages 455–470, Cham, 2018. Springer International Publishing. ISBN 978-3-030-01129-1.

[51] M. Wulfmeier, P. Ondrúška, O. Ondrúška, and I. Posner. Maximum entropy deep inverse reinforcement learning. *arXiv preprint arXiv:1507.04888*, 7 2015. doi: 10.48550/arXiv.1507.04888. URL https://arxiv.org/abs/1507.04888v3.

[52] Y. Yang, E. Yao, Z. Yang, and R. Zhang. Modeling the charging and route choice behavior of bev drivers. *Transportation Research Part C: Emerging Technologies*, 65:190–204, 2016. doi: 10.1016/j.trc.2015.09.008.

[53] Z. Zhao and Y. Liang. A deep inverse reinforcement learning approach to route choice modeling with context-dependent rewards. *Transportation Research Part C: Emerging Technologies*, 149:104079, 2023. ISSN 0968-090X. doi: https://doi.org/10.1016/j.trc.2023.104079. URL https://www.sciencedirect.com/science/article/pii/S0968090X23000682.

[54] M. Zhong, J. Kim, and Z. Zheng. Estimating link flows in road networks with synthetic trajectory data generation: Inverse reinforcement learning approach. *IEEE Open Journal of Intelligent Transportation Systems*, 4:14–29, 2023. doi: 10.1109/OJITS.2022.3233904.

[55] B. D. Ziebart. *Modeling purposeful adaptive behavior with the principle of maximum causal entropy*. PhD thesis, CMU School of Computer Science, USA, 2010.

[56] B. D. Ziebart, A. Maas, J. A. Bagnell, and A. K. Dey. Maximum entropy inverse reinforcement learning. In *Proceedings of the 23rd National Conference on Artificial Intelligence - Volume 3*, pages 1433–1438. AAAI Press, 2008. ISBN 9781577353683.
S1755534513700058.

[34] L. Qiu, Y. Zhao, J. Li, P. Lu, B. Peng, J. Gao, and S.-C. Zhu. Valuenet: A new dataset for human value driven dialogue system. *Proceedings of the AAAI Conference on Artificial Intelligence*, 36(10):11183–11191, Jun. 2022. doi: 10.1609/aaai.v36i0.21368. URL https://ojs.aaai.org/index.php/AAAI/article/view/21368.

[35] G. D. M. Ramos, W. Daamen, and S. Hoogendoorn. Modelling travellers' heterogeneous route choice behaviour as prospect maximizers. *Journal of Choice Modelling*, 6:17–33, 2013. doi: 10.1016/j.jocm.2013.04.002.

[36] L. Rizzi and J. Ortúzar. Stated preference in the valuation of interurban road safety. *Accident Analysis and Prevention*, 35:9–22, 2003. doi: 10.1016/S0001-4575(01)00082-3.

[37] M. Rodriguez-Soto, M. Serramia, M. Lopez-Sanchez, and J. A. Rodriguez-Aguilar. Instilling moral value alignment by means of multi-objective reinforcement learning. *Ethics and Information Technology*, 24:9, 3 2022. ISSN 1388-1957. doi: 10.1007/s10676-022-09635-0.

[38] S. Russell. *Human compatible: AI and the problem of control*. Penguin Uk, 2019.

[39] S. Russell. *Artificial Intelligence and the Problem of Control*, pages 19–24. Springer International Publishing, Cham, 2022. ISBN 978-3-030-86144-5. doi: 10.1007/978-3-030-86144-5_3. URL https://doi.org/10.1007/978-3-030-86144-5_3.

[40] A. Sadeghi-Niaraki and K. Kim. Ontology based personalized route planning system using a multi-criteria decision making approach. *Ex-pert Systems with Applications*, 36:2250–2259, 2009. doi: 10.1016/j.eswa.2007.12.053.

[41] J. Scheiner and C. Holz-Rau. Travel mode choice: Affected by objective or subjective determinants? *Transportation*, 34:487–511, 2007. doi: 10.1007/s11116-007-9112-1.

[42] S. H. Schwartz. Universals in the content and structure of values: Theoretical advances and empirical tests in 20 countries. In *Advances in experimental social psychology*, volume 25, pages 1–65. Elsevier, 1992.

[43] S. H. Schwartz. An overview of the schwarz theory of basic values. *Online readings in Psychology and Culture*, 2(1):11, 2012.

[44] M. Serramia, M. Lopez-Sanchez, J. A. Rodriguez-Aguilar, M. Rodriguez, M. Wooldridge, J. Morales, and C. Ansotegui. Moral values in norm decision making. *IFAAMAS*, 9, 2018. URL www.ifaamas.org.

[45] M. Serramia, M. Lopez-Sanchez, and J. A. Rodriguez-Aguilar. A qualitative approach to composing value-aligned norm systems. In *Proceedings of the 19th International Conference on Autonomous Agents and MultiAgent Systems*, pages 1233–1241. International Foundation for Autonomous Agents and Multiagent Systems, 2020. ISBN 9781450375184.

[46] M. Serramia, M. Rodriguez-Soto, M. López-Sánchez, J. Rodríguez-Aguilar, F. Bistaffa, P. Boddington, M. Wooldridge, and C. Ansótegui. Encoding ethics to compute value-aligned norms. *Minds and Machines*, pages 1–30, 11 2023. doi: 10.1007/s11023-023-09649-7.

[47] N. Soares. The value learning problem. *Artificial Intelligence Safety and Security*, 2018. URL https://apisemanticscholar.org/CorpusID: 13096553.

[48] C. Veronese, D. Meli, F. Bistaffa, M. Rodríguez-Soto, A. Farinelli, and J. A. Rodríguez-Aguilar. Inductive logic programming for transparent alignment with multiple moral values. In *CEUR WORKSHOP PROCEEDINGS*, volume 7, pages 84–88, 2024. ISBN 9772081415. doi: 10.2/QUERY.MIN.JS. URL https://iris.univir.it/handle/11562/1120547.

[49] L. Weidinger, K. R. McKee, R. Everett, S. Huang, T. O. Zhu, M. J. Chadwick, C. Summerfield, and I. Gabriel. Using the veil of ignorance to align ai systems with principles of justice. *Proceedings of the National Academy of Sciences*, 120(18):e2213709120, 2023. doi: 10.1073/pnas.2213709120. URL https://www.pnas.org/doi/abs/10.1073/pnas.2213709120.

[50] S. R. Wilson, Y. Shen, and R. Mihalcea. Building and validating hierarchical lexicons with a case study on personal values. In S. Staab, O. Koltsova, and D. I. Ignatov, editors, *Social Informatics*, pages 455–470, Cham, 2018. Springer International Publishing. ISBN 978-3-030-01129-1.

[51] M. Wulfmeier, P. Ondrúška, O. Ondrúška, and I. Posner. Maximum entropy deep inverse reinforcement learning. *arXiv preprint arXiv:1507.04888*, 7 2015. doi: 10.48550/arXiv.1507.04888. URL https://arxiv.org/abs/1507.04888v3.

[52] Y. Yang, E. Yao, Z. Yang, and R. Zhang. Modeling the charging and route choice behavior of bev drivers. *Transportation Research Part C: Emerging Technologies*, 65:190–204, 2016. doi: 10.1016/j.trc.2015.09.008.

[53] Z. Zhao and Y. Liang. A deep inverse reinforcement learning approach to route choice modeling with context-dependent rewards. *Transportation Research Part C: Emerging Technologies*, 149:104079, 2023. ISSN 0968-090X.

doi: https://doi.org/10.1016/j.trc.2023.104079. URL https://www.sciencedirect.com/science/article/pii/S0968090X23000682.

[54] M. Zhong, J. Kim, and Z. Zheng. Estimating link flows in road networks with synthetic trajectory data generation: Inverse reinforcement learning approach. *IEEE Open Journal of Intelligent Transportation Systems*, 4:14–29, 2023. doi: 10.1109/OJITS.2022.3233904.

[55] B. D. Ziebart. *Modeling purposeful adaptive behavior with the principle of maximum causal entropy*. PhD thesis, CMU School of Computer Science, USA, 2010.

[56] B. D. Ziebart, A. Maas, J. A. Bagnell, and A. K. Dey. Maximum entropy inverse reinforcement learning. In *Proceedings of the 23rd National Conference on Artificial Intelligence - Volume 3*, pages 1433–1438. AAAI Press, 2008. ISBN 9781577353683.