# Algorithms for learning value-aligned policies considering admissibility relaxation

Andrés Holgado-Sánchez, Joaquín Arias, Holger Billhardt, Sascha Ossowski

<figure><img src="image_1.png" /></figure>

## To cite this version:

Andrés Holgado-Sánchez, Joaquín Arias, Holger Billhardt, Sascha Ossowski. Algorithms for learning value-aligned policies considering admissibility relaxation. 2024. hal-04602049

HAL Id: hal-04602049

https://hal.science/hal-04602049

Preprint submitted on 5 Jun 2024

HAL is a multi-disciplinary open access
archive for the deposit and dissemination of sci-
entific research documents, whether they are pub-
lished or not. The documents may come from
teaching and research institutions in France or
abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est
destinée au dépôt et à la diffusion de documents
scientifiques de niveau recherche, publiés ou non,
émanant des établissements d'enseignement et de
recherche français ou étrangers, des laboratoires
publics ou privés.

<figure><img src="image_2.png" /></figure>
# Algorithms for learning value-aligned policies considering admissibility relaxation

Andrés Holgado-Sánchez, Joaquín Arias, Holger Billhardt, Sascha Ossowski

<figure><img src="image_1.png" /></figure>

## To cite this version:

Andrés Holgado-Sánchez, Joaquín Arias, Holger Billhardt, Sascha Ossowski. Algorithms for learning value-aligned policies considering admissibility relaxation. 2024. hal-04602049

HAL Id: hal-04602049

https://hal.science/hal-04602049v1

Preprint submitted on 5 Jun 2024

HAL is a multi-disciplinary open access
archive for the deposit and dissemination of sci-
entific research documents, whether they are pub-
lished or not. The documents may come from
teaching and research institutions in France or
abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est
destinée au dépôt et à la diffusion de documents
scientifiques de niveau recherche, publiés ou non,
émanant des établissements d'enseignement et de
recherche français ou étrangers, des laboratoires
publics ou privés.

<figure><img src="image_2.png" /></figure>
# Algorithms for learning value-aligned policies considering admissibility relaxation

Andrés Holgado<sup>a</sup>, Joaquin Arias<sup>a</sup>, Holger Billhardt<sup>a</sup> and Sascha Ossowski<sup>a</sup>

<sup>a</sup>CETINIA, Universidad Rey Juan Carlos, Madrid, Spain
{andres.holgado, joaquin.arias, holger.billhardt, sascha.ossowski}@urjc.es
ORCID ID: Andrés Holgado<sup>©</sup> https://orcid.org/0000-0001-8853-1022,
Joaquín Arias<sup>©</sup> https://orcid.org/0000-0003-4148-311X, Holger Billhardt<sup>©</sup> https://orcid.org/0000-0001-8298-4178,
Sascha Ossowski<sup>©</sup> https://orcid.org/0000-0003-2483-9508

**Abstract.** The emerging field of *value awareness engineering* claims that software agents and systems should be value-aware, i.e. they must make decisions in accordance with human values. In this context, such agents must be capable of explicitly reasoning as to how far different courses of action are aligned with these values. For this purpose, values are often modelled as preferences over states or actions, which are then aggregated to determine the sequences of actions that are maximally aligned with a certain value. Recently, additional value admissibility constraints at this level have been considered as well.

However, often relaxed versions of these constraints are needed, and this increases considerably the complexity of computing value-aligned policies. To obtain efficient algorithms that make value-aligned decisions considering admissibility relaxation, we propose the use of learning techniques, in particular, we have used constrained reinforcement learning algorithms. In this paper, we present two algorithms, $\epsilon$-ADQL for strategies based on local alignment and its extension $\epsilon$-CADQL for a sequence of decisions. We have validated their efficiency in a water distribution problem in a drought scenario.

# 1 Introduction

Integrating human values into practical reasoning is a problem that has been considered only recently in computer science and autonomous systems [27, 2]. These and other attempts to incorporate human values into the reasoning and decision-making schemes of intelligent software agents can be labelled into the emergent field of *value awareness engineering* [15]. Proposals for modelling value-based decision processes of autonomous agents are often based on preferences over states or actions [16, 14], which are then extended to sequential decisions. Other approaches [4, 8] set out from observed sequences of actions (plans) and then learn preferences over states or actions through (inverse) reinforcement learning [18].

In [12] we have argued that there is a need to look into value-alignment from a path-level perspective in a careful way. In particular, without denying the usefulness of using aggregations of state or action preferences to measure the alignment of sequential decisions (paths), we have suggested that certain sequences of actions should not be admissible to a value-aware agent, even though they show a good overall aggregated value-alignment [12]. This is cap-

tured through the notion of *value-admissible behaviours*, i.e. criteria defining minimally aligned sequences of decisions, which control more precisely *how* a value-aligned path should be.

In this paper, we look into the problem of learning decision policies that represent admissible behaviours that are highly aligned with a certain value. In order to solve this problem, reinforcement learning (RL) [25] and, in particular, constrained versions of it constitute state-of-the-art tools. However for infinite horizon processes, where a reward should be maintained over time instead of reaching a goal, average reward MDP's (Chapter 10, Sutton and Barto [25]) offer a better approach than the usual discounted setup.

Based on the above insights, in this paper we propose new RL methods called $\epsilon$-local (Constrained) Average Double Q-Learning ($\epsilon$-ADQL and $\epsilon$-CADQL, respectively), implemented by borrowing concepts from constraint RL: RCPO (Reward Constraint Policy Optimization) [26] and CDQL (Constrained Deep Q-Learning) [13]. We test them in a continuous water distribution procedure in a simulated environment where rewards model directly a numerical representation of value-alignment at state-level. Specifically, $\epsilon$-ADQL will find a policy maximizing expected mean future reward. In addition, the constrained version $\epsilon$-CADQL aims at minimising potential *violations* of state-level admissibility criteria, intending to avoid as much as possible states whose alignment with a certain value (modelled by the semantics function) drops below a threshold $\tau$.

The paper is structured as follows. Section 2 discusses related work regarding value awareness and reinforcement learning. In Section 3, we outline our world model and introduce different notions of value-admissible behaviours in the context of learning problems. In Section 4, we motivate and explain our use case on water distribution considering the value of equity. In Section 5 we model the problem of learning value-aligned admissible paths in our use-case and provide details on the $\epsilon$-ADQL and $\epsilon$-CADQL RL algorithms. In Section 6 we test the algorithms in simulations and compare them to a baseline local policy. Finally, in Section 7 we present our conclusions and future work.

# 2 Related work

The value-alignment problem has been formalized in various ways for decision-making. One of the first communities introducing values in their algorithms was practical reasoning [27] and value-based
# Algorithms for learning value-aligned policies considering admissibility relaxation

Andrés Holgado<sup>a</sup>, Joaquin Arias<sup>a</sup>, Holger Billhardt<sup>a</sup> and Sascha Ossowski<sup>a</sup>

<sup>a</sup>CETINIA, Universidad Rey Juan Carlos, Madrid, Spain
{andres.holgado, joaquin.arias, holger.billhardt, sascha.ossowski}@urjc.es
ORCID ID: Andrés Holgado<sup>©</sup> https://orcid.org/0000-0001-8853-1022,
Joaquín Arias<sup>©</sup> https://orcid.org/0000-0003-4148-311X, Holger Billhardt<sup>©</sup> https://orcid.org/0000-0001-8298-4178,
Sascha Ossowski<sup>©</sup> https://orcid.org/0000-0003-2483-9508

**Abstract.** The emerging field of *value awareness engineering* claims that software agents and systems should be value-aware, i.e. they must make decisions in accordance with human values. In this context, such agents must be capable of explicitly reasoning as to how far different courses of action are aligned with these values. For this purpose, values are often modelled as preferences over states or actions, which are then aggregated to determine the sequences of actions that are maximally aligned with a certain value. Recently, additional value admissibility constraints at this level have been considered as well.

However, often relaxed versions of these constraints are needed, and this increases considerably the complexity of computing value-aligned policies. To obtain efficient algorithms that make value-aligned decisions considering admissibility relaxation, we propose the use of learning techniques, in particular, we have used constrained reinforcement learning algorithms. In this paper, we present two algorithms, $\epsilon$-ADQL for strategies based on local alignment and its extension $\epsilon$-CADQL for a sequence of decisions. We have validated their efficiency in a water distribution problem in a drought scenario.

## 1 Introduction

Integrating human values into practical reasoning is a problem that has been considered only recently in computer science and autonomous systems [27, 2]. These and other attempts to incorporate human values into the reasoning and decision-making schemes of intelligent software agents can be labelled into the emergent field of *value awareness engineering* [15]. Proposals for modelling value-based decision processes of autonomous agents are often based on preferences over states or actions [16, 14], which are then extended to sequential decisions. Other approaches [4, 8] set out from observed sequences of actions (plans) and then learn preferences over states or actions through (inverse) reinforcement learning [18].

In [12] we have argued that there is a need to look into value-alignment from a path-level perspective in a careful way. In particular, without denying the usefulness of using aggregations of state or action preferences to measure the alignment of sequential decisions (paths), we have suggested that certain sequences of actions should not be admissible to a value-aware agent, even though they show a good overall aggregated value-alignment [12]. This is cap-

tured through the notion of *value-admissible behaviours*, i.e. criteria defining minimally aligned sequences of decisions, which control more precisely *how* a value-aligned path should be.

In this paper, we look into the problem of learning decision policies that represent admissible behaviours that are highly aligned with a certain value. In order to solve this problem, reinforcement learning (RL) [25] and, in particular, constrained versions of it constitute state-of-the-art tools. However for infinite horizon processes, where a reward should be maintained over time instead of reaching a goal, average reward MDP's (Chapter 10, Sutton and Barto [25]) offer a better approach than the usual discounted setup.

Based on the above insights, in this paper we propose new RL methods called $\epsilon$-local (Constrained) Average Double Q-Learning ($\epsilon$-ADQL and $\epsilon$-CADQL, respectively), implemented by borrowing concepts from constraint RL: RCPO (Reward Constraint Policy Optimization) [26] and CDQL (Constrained Deep Q-Learning) [13]. We test them in a continuous water distribution procedure in a simulated environment where rewards model directly a numerical representation of value-alignment at state-level. Specifically, $\epsilon$-ADQL will find a policy maximizing expected mean future reward. In addition, the constrained version $\epsilon$-CADQL aims at minimising potential *violations* of state-level admissibility criteria, intending to avoid as much as possible states whose alignment with a certain value (modelled by the semantics function) drops below a threshold $\tau$.

The paper is structured as follows. Section 2 discusses related work regarding value awareness and reinforcement learning. In Section 3, we outline our world model and introduce different notions of value-admissible behaviours in the context of learning problems. In Section 4, we motivate and explain our use case on water distribution considering the value of equity. In Section 5 we model the problem of learning value-aligned admissible paths in our use-case and provide details on the $\epsilon$-ADQL and $\epsilon$-CADQL RL algorithms. In Section 6 we test the algorithms in simulations and compare them to a baseline local policy. Finally, in Section 7 we present our conclusions and future work.

## 2 Related work

The value-alignment problem has been formalized in various ways for decision-making. One of the first communities introducing values in their algorithms was practical reasoning [27] and value-based
argumentation [2]. Computational representations of values are key to the recent field of *value-awareness engineering*.

EU ethics guidelines on trustworthy AI<fnref n="1" /> mention a series of values including privacy, fairness, explainability, non-maleficence etc. that must be respected during all stages of the design and development of an AI system to ensure its responsible deployment. Schwartz puts forward a taxonomy of universal values that transcend specific actions or situations in a system [23]. However, for practical reasoning such values need to be grounded in a particular context [15].

Works borrowing from the consequentialist tradition of computational ethics usually model values in terms of preferences over states or actions [24, 2, 16]. Montes et al. [16] use semantics functions to define the value-alignment of states. This information is then extended to sequential decisions via aggregation functions in order to find norms that can regulate a multi-agent system's behaviour in a value-aware fashion. However, they do not examine in detail how far the resulting evolution of the system (i.e. joint paths traversed by agents) really certifies an acceptable alignment at path-level. Lera Leri et al. [14] define value-alignment based on actions that promote or demote values. They use that formulation for the task of value system aggregation (i.e. ranking decisions taking perspectives of different values into account) but do not study sequences of actions. The notion of value-admissible behaviour put forward in [12] introduces constraints on value-alignment at path-level. However, the work does not provide effective algorithms to determine admissible sequential decisions with a suitable level of value-alignment.

Regarding applications of machine learning to value-alignment, inverse reinforcement learning [18] has been used to infer values (preferences over states or actions), setting out from observed sequences of actions [4, 8]. The problem of determining optimally aligned sequences of actions given a certain value is often modelled as a mathematical optimization problem [6]. In the context of this paper, we are particularly interested in RL techniques capable of generating suitable value-aligned sequences of actions that respect certain admissibility constraints. To this respect, it is important to highlight the multi-objective RL approach developed by Rodríguez-Soto et al. [22], integrated into a norm-abiding and value-aligned (i.e. ethical) decision-making learning environment, that uses specific ethical rewards and RL values. Still, this approach cannot express path admissibility criteria that are not reward-shapable.

In the field of constrained RL [1], Tessler et al. [26] propose an algorithm (RCPO) to deal with state-action constraints (formulated as inequalities) learning the optimal Lagrange multiplier to shape the rewards optimally. Dalal et al. [5] introduce the concept of safety (in critical environments) directly adding to the policy a safety layer that analytically solves an action correction formulation per each state, predicting constraints. Both approaches are able to deal with certain behaviours but for a discounted setup, which does not fit well with long-term continuous problems [25] that we are particularly interested in. Another solution, Constrained Q-learning, by Kalweit et al. [13], enforces hard constraint satisfaction during and after the training process, though its efficiency is limited by the behaviours' complexity.

## 3 Value-admissible behaviours

As a first step towards agents learning value-aligned action policies, in this section, we sketch the world model we used and introduce different types of value-admissible behaviours that a value-aware agent may want to choose from.

Setting out from [16], we assume that an agent's decision-making component represents the world as a labelled transition system, called **decision world**.

**Definition 1 (Decision world)** A decision world is a triplet $(S, A, T)$ with the following elements.

* **States** $S$, representing the MAS completely in each situation.

* **Actions** $A$, representing the MAS joint actions or decisions.

* **Transitions** $T \subset S \times A \times S$, representing available actions connecting each pair of states. Denoted with $s \xrightarrow{a} s'$, where $s, s' \in S$ and $a \in A$. We also define $A(s)$ as the actions accessible in the system from state $s$ (i.e. the set of actions $a \in A$ such that exists some $s \xrightarrow{a} s' \in T$).

* **Paths** $P$, representing joint transitions (sequences of decisions), e.g. a path of length $n$ from $s_0$ to $s_n$ would be represented as: $P = s_0 \xrightarrow{a_1} s_1 \xrightarrow{a_2} \dots \xrightarrow{a_n} s_n$.

While in [16] a notion of final or *goal* states for the system to reach via paths where considered, in this paper goals are considered implicit and policies can imply infinite sequences of actions.

Following Weide et al. [27] or Sierra et al. [24], we assume a value preference among states based on a preorder relation $\sqsubseteq_v$, which we call **perspective** or **value preorder**, i.e. given $s$ and $s'$, two states, $s \sqsubseteq_v s'$ means that $s'$ is at least as preferred as $s$ w.r.t. the value $v$.

However, to simplify the computational representation [12], in this paper we will quantify the above relationships using unbounded semantics functions [16]:<fnref n="2" />

**Definition 2 (Unbounded Semantics function)** The alignment of state $s$ with a value $v$ is described by an unbounded semantics function $f_v : S \to \mathbb{R}$, where $f_v$ is directly proportional to the promotion of $v$.

Our objective is to find a policy which leads to a path that is well aligned with a certain value. Specifically, we want to maximize the values of the semantics function along possibly an infinite sequence of states, using a certain aggregation function [16] as a metric:

**Definition 3 (value-alignment of a path)** Given an aggregation function $agg$, and a semantics function $f_v$, we define the **value-alignment of a path** $P = s_0 \xrightarrow{a_1} \dots \xrightarrow{a_n} s_n$ (or its aggregated alignment) as:

$$agg_v(P) = agg(\{f_v(s_0), \dots, f_v(s_n)\})$$

As we are considering an infinite horizon, we opt to use the *average* as the aggregation function (thus making path length irrelevant).

However, in certain circumstances a path with maximum average aggregated alignment may not be **admissible** to a value-aware decision-maker, e.g. due to an extreme variability of the value-alignment of the states it traverses. For instance, in a water distribution scenario, all assignments that at some point in time leave stakeholders without a minimum amount of water necessary for basic needs should not be considered, even though “on average” they achieve an equitable water distribution. Setting out from [12], in the following we present a redefinition of the concept of **value-admissible behaviours** based on semantics functions.

<footnote n="1">¹ digital-strategy.ec.europa.eu/en/library/ethics-guidelines-trustworthy-ai</footnote>

<footnote n="2">² Original definition from Montes and Sierra uses $[-1, 1]$-bounded functions, which is used to model both promotion and demotion of the value. For this theory, those specific bounds are not mandatory.</footnote>
argumentation [2]. Computational representations of values are key to the recent field of *value-awareness engineering*.

EU ethics guidelines on trustworthy AI<fnref n="1" /> mention a series of values including privacy, fairness, explainability, non-maleficence etc. that must be respected during all stages of the design and development of an AI system to ensure its responsible deployment. Schwartz puts forward a taxonomy of universal values that transcend specific actions or situations in a system [23]. However, for practical reasoning such values need to be grounded in a particular context [15].

Works borrowing from the consequentialist tradition of computational ethics usually model values in terms of preferences over states or actions [24, 2, 16]. Montes et al. [16] use semantics functions to define the value-alignment of states. This information is then extended to sequential decisions via aggregation functions in order to find norms that can regulate a multi-agent system's behaviour in a value-aware fashion. However, they do not examine in detail how far the resulting evolution of the system (i.e. joint paths traversed by agents) really certifies an acceptable alignment at path-level. Lera Leri et al. [14] define value-alignment based on actions that promote or demote values. They use that formulation for the task of *value system* aggregation (i.e. ranking decisions taking perspectives of different values into account) but do not study sequences of actions. The notion of value-admissible behaviour put forward in [12] introduces constraints on value-alignment at path-level. However, the work does not provide effective algorithms to determine admissible sequential decisions with a suitable level of value-alignment.

Regarding applications of machine learning to value-alignment, inverse reinforcement learning [18] has been used to infer values (preferences over states or actions), setting out from observed sequences of actions [4, 8]. The problem of determining optimally aligned sequences of actions given a certain value is often modelled as a mathematical optimization problem [6]. In the context of this paper, we are particularly interested in RL techniques capable of generating suitable value-aligned sequences of actions that respect certain admissibility constraints. To this respect, it is important to highlight the multi-objective RL approach developed by Rodríguez-Soto et al. [22], integrated into a norm-abiding and value-aligned (i.e. ethical) decision-making learning environment, that uses specific ethical rewards and RL values. Still, this approach cannot express path admissibility criteria that are not reward-shapable.

In the field of constrained RL [1], Tessler et al. [26] propose an algorithm (RCPO) to deal with state-action constraints (formulated as inequalities) learning the optimal Lagrange multiplier to shape the rewards optimally. Dalal et al. [5] introduce the concept of safety (in critical environments) directly adding to the policy a safety layer that analytically solves an action correction formulation per each state, predicting constraints. Both approaches are able to deal with certain behaviours but for a discounted setup, which does not fit well with long-term continuous problems [25] that we are particularly interested in. Another solution, Constrained Q-learning, by Kalweit et al. [13], enforces hard constraint satisfaction during and after the training process, though its efficiency is limited by the behaviours' complexity.

## 3 Value-admissible behaviours

As a first step towards agents learning value-aligned action policies, in this section, we sketch the world model we used and introduce different types of value-admissible behaviours that a value-aware agent may want to choose from.

Setting out from [16], we assume that an agent's decision-making component represents the world as a labelled transition system, called **decision world**.

**Definition 1 (Decision world)** A decision world is a triplet $(S, A, T)$ with the following elements.

* **States** $S$, representing the MAS completely in each situation.

* **Actions** $A$, representing the MAS joint actions or decisions.

* **Transitions** $T \subset S \times A \times S$, representing available actions connecting each pair of states. Denoted with $s \xrightarrow{a} s'$, where $s, s' \in S$ and $a \in A$. We also define $A(s)$ as the actions accessible in the system from state $s$ (i.e. the set of actions $a \in A$ such that exists some $s \xrightarrow{a} s' \in T$).

* **Paths** $\mathcal{P}$, representing joint transitions (sequences of decisions), e.g. a path of length $n$ from $s_0$ to $s_n$ would be represented as: $P = s_0 \xrightarrow{a_1} s_1 \xrightarrow{a_2} \dots \xrightarrow{a_n} s_n$.

While in [16] a notion of final or *goal* states for the system to reach via paths where considered, in this paper goals are considered implicit and policies can imply infinite sequences of actions.

Following Weide et al. [27] or Sierra et al. [24], we assume a value preference among states based on a preorder relation $\sqsubseteq_v$, which we call **perspective** or **value preorder**, i.e. given $s$ and $s'$, two states, $s \sqsubseteq_v s'$ means that $s'$ is at least as preferred as $s$ w.r.t. the value $v$.

However, to simplify the computational representation [12], in this paper we will quantify the above relationships using *unbounded semantics functions* [16]:<fnref n="2" />

**Definition 2 (Unbounded Semantics function)** The alignment of state $s$ with a value $v$ is described by an unbounded semantics function $f_v : S \to \mathbb{R}$, where $f_v$ is directly proportional to the promotion of $v$.

Our objective is to find a policy which leads to a path that is well aligned with a certain value. Specifically, we want to maximize the values of the semantics function along possibly an infinite sequence of states, using a certain *aggregation function* [16] as a metric:

**Definition 3 (value-alignment of a path)** Given an aggregation function $agg$, and a semantics function $f_v$, we define the **value-alignment of a path** $P = s_0 \xrightarrow{a_1} \dots \xrightarrow{a_n} s_n$ (or its aggregated alignment) as:

$$agg_v(P) = agg(\{f_v(s_0), \dots, f_v(s_n)\})$$

As we are considering an infinite horizon, we opt to use the *average* as the aggregation function (thus making path length irrelevant).

However, in certain circumstances a path with maximum average aggregated alignment may not be *admissible* to a value-aware decision-maker, e.g. due to an extreme variability of the value-alignment of the states it traverses. For instance, in a water distribution scenario, all assignments that at some point in time leave stakeholders without a minimum amount of water necessary for basic needs should not be considered, even though “on average” they achieve an equitable water distribution. Setting out from [12], in the following we present a redefinition of the concept of *value-admissible behaviours* based on semantics functions.

<footnote n="1">¹ digital-strategy.ec.europa.eu/en/library/ethics-guidelines-trustworthy-ai</footnote>

<footnote n="2">² Original definition from Montes and Sierra uses $[-1, 1]$-bounded functions, which is used to model both promotion and demotion of the value. For this theory, those specific bounds are not mandatory.</footnote>
**Definition 4 (Value-Admissible Behaviour)** A value-admissible behaviour for a value v is a constraint criterion for plans P that characterizes the subset B(P, f_v) that are admissibly aligned with the value, based on state/action-level semantics f_v.

In real-world draught scenarios, legal requirements establish that the equity of water distribution has to be assured at all times [12]. This is expressed by the following local behaviour in terms of the values of a semantics function.

**Definition 5 (Local behaviour)** Given a set of paths P and a semantics function f_v, the local behaviour, B_local is defined as the set of paths built by maximizing the value function at each step:

$$B_{\text{local}}(\mathcal{P}, f_v) = \{P \in \mathcal{P} \mid \forall s \xrightarrow{a} t \in P : \\ f_v(t) = \max\{f_v(t') \mid \exists a' \in \mathcal{A}(s) : s \xrightarrow{a'} t'\}$$

Still, maximizing the overall aggregated equity (i.e. the alignment of paths with the value of equity) is of high importance. In special cases, both characteristics go hand in hand. For instance, in [12] equity semantics functions adhering to the Pigou-Dalton principle [17] provide that the locally admissible paths are also those with the highest aggregate value-alignment. However, with general semantics functions and in more complex environments this need not be the case.

Therefore we introduce the following relaxed notions of the aforementioned local behaviour that will constitute the basis of our learning approach put forward in Section 5.

**Definition 6 (ε-local behaviour)** Given a set of paths P, ε > 0 and a semantics function f_v, the ε-local behaviour, B_ε is defined as:

$$B_{\epsilon}(\mathcal{P}, f_v) = \{P \in \mathcal{P} \mid \forall s \xrightarrow{a} t \in P : \\ f_v(t) \geq \max\{f_v(t') \mid \exists a' \in \mathcal{A}(s) : s \xrightarrow{a'} t'\} - \epsilon\}$$

The epsilon-local behaviour allows traversing new paths, by slightly relaxing the strict equity-preserving aspects of the local one. We expect that this relaxation of immediate equity prosecution will lead, with a certain ε-local policy (i.e. ε-local behaviour compliant) and a sufficiently big ε, to paths with much higher aggregation, traversed with fairly legally justifiable actions. Given a state s and a value ε, we denote by A_ε(s) the actions that would be admissible to be executed in state s in an ε-local policy.

Apart from the ε-local behaviour, to guide our policies into traversing globally admissible states (i.e. the semantics of the value for every state traversed is above some threshold τ > 0), we propose the τ-constrained behaviour:

**Definition 7 (τ-constrained behaviour)** Given a set of paths P, τ > 0 and a semantics function f_v, the τ-constrained behaviour, B_τ is defined as:

$$B_{\tau}(\mathcal{P}, f_v) = \{P \in \mathcal{P} \mid \forall s \xrightarrow{a} t \in P : f_v(t) \geq \tau\}$$

In short, a τ-constrained policy would lead to paths where the alignment of all states in the path with respect to a semantic function f_v has a lower bound on a threshold τ.

# 4 Use Case

In this paper, we consider a use case around water distribution where equity (or fair distribution) is the value to be preserved. This field has

been widely analyzed using socio-cognitive agents [20] but here we use a simpler, yet just complex enough, scenario that is sufficient for illustrating the proposed concepts. The tasks consist of distributing water from a reservoir (source of water) to 4 villages with different populations and that are connected through a road network. Figure 1 represents the map of the problem.

<figure><img src="image_3.png" /><figcaption>**Figure 1.** The water distribution problem schema (Red arcs for two-way roads, directed black arcs for one-way ones)</figcaption></figure>

A tanker truck able to traverse these roads is in charge of distributing the water. We assume that this vehicle has a limited capacity C (here, C = 60000l) and the process continues until a fixed amount of water T has been distributed (i.e. the available water resources).

On each step, the vehicle can take an action which consists of moving from its current position (i.e. either a village or the reservoir) to another point and discharging some water (at villages) or refilling the tank (at the reservoir). We encoded some simple rules regarding possible movements for basic delivery and time efficiency:

* If the vehicle visits a village that has no outgoing connection to other villages (e.g. Village 2 in Figure 1), it will dispense all its current amount of water.

* Unless the vehicle is empty, it cannot return back to refill at the reservoir.

* Each time the vehicle returns to the source, it fills up completely.

For simplicity, we assume the truck takes equal time (approx. one hour) to reach any node (village/source) reachable from any immediately adjacent node (connected by a road).

In our scenario, we consider that each village consumes the water based on different consumption patterns, as described in the next subsection.

## 4.1 Estimation of water consumption

According to the World Health Organization (WHO), between 50 and 100 litres of water per person per day are necessary to ensure that the most basic needs are met and few health concerns arise. However, average water use is 200 to 300 litres per person per day in most countries in Europe.<fnref n="3" />

In Spain, the main use of water is for irrigation and agricultural use, which accounts for approximately 80.5% of this demand, followed by urban supply, which represents 15.5%. The remainder is for industrial use [9]. Of all the water uses, the priority is urban water supply.<fnref n="4" /> The regulations have established that the net or average

<footnote n="3">³ Resolution 64/292, 07/28/2010, of the United Nations General Assembly, vid. https://www.un.orgspanish/waterforlifedecade/human_right_to_water.shtml.</footnote>

<footnote n="4">⁴ Royal Decree 1/2001, of July 20, approving the Revised Text of the Water Law, Article 60.</footnote>
**Definition 4 (Value-Admissible Behaviour)** A value-admissible behaviour for a value v is a constraint criterion for plans P that characterizes the subset B(P, f_v) that are admissibly aligned with the value, based on state/action-level semantics f_v.

In real-world draught scenarios, legal requirements establish that the equity of water distribution has to be assured at all times [12]. This is expressed by the following local behaviour in terms of the values of a semantics function.

**Definition 5 (Local behaviour)** Given a set of paths P and a semantics function f_v, the local behaviour, B_local is defined as the set of paths built by maximizing the value function at each step:

$$B_{\text{local}}(\mathcal{P}, f_v) = \{P \in \mathcal{P} \mid \forall s \xrightarrow{a} t \in P : \\ f_v(t) = \max\{f_v(t') \mid \exists a' \in \mathcal{A}(s) : s \xrightarrow{a'} t'\}$$

Still, maximizing the overall aggregated equity (i.e. the alignment of paths with the value of equity) is of high importance. In special cases, both characteristics go hand in hand. For instance, in [12] equity semantics functions adhering to the Pigou-Dalton principle [17] provide that the locally admissible paths are also those with the highest aggregate value-alignment. However, with general semantics functions and in more complex environments this need not be the case.

Therefore we introduce the following relaxed notions of the aforementioned local behaviour that will constitute the basis of our learning approach put forward in Section 5.

**Definition 6 (ε-local behaviour)** Given a set of paths P, ε > 0 and a semantics function f_v, the ε-local behaviour, B_ε is defined as:

$$B_{\epsilon}(\mathcal{P}, f_v) = \{P \in \mathcal{P} \mid \forall s \xrightarrow{a} t \in P : \\ f_v(t) \geq \max\{f_v(t') \mid \exists a' \in \mathcal{A}(s) : s \xrightarrow{a'} t'\} - \epsilon\}$$

The epsilon-local behaviour allows traversing new paths, by slightly relaxing the strict equity-preserving aspects of the local one. We expect that this relaxation of immediate equity prosecution will lead, with a certain ε-local policy (i.e. ε-local behaviour compliant) and a sufficiently big ε, to paths with much higher aggregation, traversed with fairly legally justifiable actions. Given a state s and a value ε, we denote by A_ε(s) the actions that would be admissible to be executed in state s in an ε-local policy.

Apart from the ε-local behaviour, to guide our policies into traversing globally admissible states (i.e. the semantics of the value for every state traversed is above some threshold τ > 0), we propose the τ-constrained behaviour:

**Definition 7 (τ-constrained behaviour)** Given a set of paths P, τ > 0 and a semantics function f_v, the τ-constrained behaviour, B_τ is defined as:

$$B_{\tau}(\mathcal{P}, f_v) = \{P \in \mathcal{P} \mid \forall s \xrightarrow{a} t \in P : f_v(t) \geq \tau\}$$

In short, a τ-constrained policy would lead to paths where the alignment of all states in the path with respect to a semantic function f_v has a lower bound on a threshold τ.

# 4 Use Case

In this paper, we consider a use case around water distribution where equity (or fair distribution) is the value to be preserved. This field has

been widely analyzed using socio-cognitive agents [20] but here we use a simpler, yet just complex enough, scenario that is sufficient for illustrating the proposed concepts. The tasks consist of distributing water from a reservoir (source of water) to 4 villages with different populations and that are connected through a road network. Figure 1 represents the map of the problem.

<figure><img src="image_3.png" /><figcaption>**Figure 1.** The water distribution problem schema (Red arcs for two-way roads, directed black arcs for one-way ones)</figcaption></figure>

A tanker truck able to traverse these roads is in charge of distributing the water. We assume that this vehicle has a limited capacity C (here, C = 60000l) and the process continues until a fixed amount of water T has been distributed (i.e. the available water resources).

On each step, the vehicle can take an action which consists of moving from its current position (i.e. either a village or the reservoir) to another point and discharging some water (at villages) or refilling the tank (at the reservoir). We encoded some simple rules regarding possible movements for basic delivery and time efficiency:

* If the vehicle visits a village that has no outgoing connection to other villages (e.g. Village 2 in Figure 1), it will dispense all its current amount of water.

* Unless the vehicle is empty, it cannot return back to refill at the reservoir.

* Each time the vehicle returns to the source, it fills up completely.

For simplicity, we assume the truck takes equal time (approx. one hour) to reach any node (village/source) reachable from any immediately adjacent node (connected by a road).

In our scenario, we consider that each village consumes the water based on different consumption patterns, as described in the next subsection.

## 4.1 Estimation of water consumption

According to the World Health Organization (WHO), between 50 and 100 litres of water per person per day are necessary to ensure that the most basic needs are met and few health concerns arise. However, average water use is 200 to 300 litres per person per day in most countries in Europe.<fnref n="3" />

In Spain, the main use of water is for irrigation and agricultural use, which accounts for approximately 80.5% of this demand, followed by urban supply, which represents 15.5%. The remainder is for industrial use [9]. Of all the water uses, the priority is urban water supply.<fnref n="4" /> The regulations have established that the net or average

<footnote n="3">³ Resolution 64/292, 07/28/2010, of the United Nations General Assembly, vid. https://www.un.orgspanish/waterforlifedecade/human_right_to_water.shtml.</footnote>

<footnote n="4">⁴ Royal Decree 1/2001, of July 20, approving the Revised Text of the Water Law, Article 60.</footnote>
consumption endowment, as a minimum objective, must be at least 100 litres per inhabitant per day.<fnref n="5" />

This information inspires the idea to model that different societies (or villages in our case) would have different water consumption patterns. In our experiments, we will model different consumption rates per time step (the period of one movement of the tanker vehicle). We consider that the consumption will increase as water availability in a village increases (e.g., due to a diversification of uses), but differently depending on each village:

* Village 0 (Brown-25 inhabitants): This village consumes 4l per inhabitant per hour in scarcity, unless the available water is above 350l per inhabitant, where it will consume 100l per inhabitant per hour due to large irrigation needs.

* Village 1 (Blue-260 inhabitants): In this village only 3.5l per inhabitant per hour is used unless there is more than 250l per inhabitant. Then, 9l per inhabitant per hour will be consumed.

* Village 2 (Yellow-1000 inhabitants): In this case 3.5l is consumed per inhabitant per hour if less than 350l per inhabitant is in reserve. Otherwise, per inhabitant consumption rises to 50l per hour.

* Village 3 (Magenta-1050 inhabitants). In this village, the consumption rate will be 3.5l up to a reserve of 100l per inhabitant. After that, the consumption will be 16l per inhabitant.

## 4.2 Modeling states, transitions, and paths

A possible state- and transition-based modelling of the decision problem for water distribution among the four villages is as follows:

- **States:** conceptually, we will represent a state $s \in S$ as a list $(x_0, x_1, x_2, x_3, p, c)$ where each value $x_j \in \mathbb{R}_{\ge 0}$ represents the amount of water per inhabitant in village $j$; $p \in \{-1, 0, 1, 2, 3\}$ represents the current position of the vehicle ($-1$ for source); and $c$ stands for the current amount of water in the vehicle.

- **Actions:** an action $a \in A$ is a pair $(p, d)$ indicating the place $p \in \{-1, 0, 1, 2, 3\}$ to go to by the vehicle and $d \in (0, C]$ the water to dispense after arriving at $p$ (irrelevant when $p = -1$, as then the vehicle will just refill). As described before, the available actions depend on each state.

- **Transitions:** a transition from state $s = (x_0, \dots, x_3, p, c)$ to state $s' = (x'_0, \dots, x'_3, p', c')$ by applying action $a = (p', d)$ means that $s'$ is calculated by setting $c' = c - d$ if $p \neq -1$ (else, $c' = C$ for refilling) and calculating $x'_i$ by first letting the vehicle move (and dispense the water to $p$) and then applying the corresponding water consumption rate to each village as described in Section 4.1.

The decision-making solution in the distribution of water between villages is a (deterministic) Markov policy. We are interested in finding policies that generate certain behaviours (e.g., fulfil certain admissible criteria) and also lead to paths with high value-alignment with respect to a semantics function and aggregation function.

## 4.3 Modelling equity

In the water distribution scenario, we are interested in the value of equity. Thus, we need to define what it means to have a fair or equitable distribution of water in our system at each moment (i.e. how much our states of the system are aligned to the equity value). We will use a semantics function that reflects equity in terms of water availability per person, not per village, assuming each person has equal access to the water of the village he or she belongs to. Specifically, let $n_i$

denote the population of village $i$ and $s = (x_0, \dots, x_3, p, d)$ be a state of the environment. We define the water distribution of state $s$, $D(s)$ by:

$$D(s) = \underbrace{(x_0, \dots, x_0, \dots, x_3, \dots, x_3)}_{n_0 \text{ times}}$$

In this distribution, each person counts with his or her own water available water resources. For example, if village 0 (with 25 inhabitants) has 20l per inhabitant, there are 25 entries with 20l.

Based on $D$ we define the semantics function $f_{gini}(s) = 1 - GI(D(s))$, where $GI$ is the Gini Index, a widely accepted inequality metric in the literature [16, 21, 10]. Note $f_{gini}$ is bounded in $[0, 1]$, and is proportional to equity.

In the next section, we will use the 3 behaviours presented in Section 3 for assessing equitable paths. We will define settings for learning policies that maximize the aggregated alignment of paths and also implement $\epsilon$-local / $\epsilon$-local and $\tau$-constrained behaviours. We will compare those policies with a policy that simply adheres to a local behaviour.

# 5 Learning behaviour-compliant policies

In our scenario, given a current state $s$, we can predict the value-alignment of the resulting states if we would apply any possible action to $s$. This can be done by simply calculating $f_{gini}$ on the estimated resulting states. Thus, a policy adhering to a local behaviour can be implemented easily by seeking the best action among all possibilities in each moment. However, using brute force search to find a policy that maximizes the value-alignment of an execution path is often not feasible, even for simple tasks. In our approach, we propose to use Reinforcement Learning (RL) [25] as a technique to learn such policies.

Our use case (Section 4) can be represented by a Markov Decision Process (MDP) where in any state a more or less equity-valuable action can be taken. This equity-valuable concept is directly represented via state-action rewards $r(s, a)$, which are then used to define the RL value $V(s, a)$ representing future expected reward (i.e. "long-term" equity) in the system. This value function $V(s, a)$ is used by the policy to select action $a$ from $s$. Specifically, for any state $s_j$ at a step $j$, we calculate the reward of applying an action $a_j$ from that state such that $s_j \xrightarrow{a_j} s_{j+1}$ as $r_j(s_j, a_j) = f_{gini}(s_{j+1})$.

Given this reward definition, our aim is to learn policies that maximize path value-alignment and that are at the same time behaviour-compliant with $\epsilon$-local and/or $\tau$-constrained behaviours.

In order to keep the learning problem tractable, we apply a simplification to both state and action spaces. With regard to actions, we only allow deliveries of water (the value $d$ from action pair $(p, d)$) which are multiples of a fixed quantity. In our experiments, we used 15000l and allowed only deliveries of 0l, 15000l, 30000l, 45000l and 60000l (which is the capacity of the truck).

With regard to states, we reduce the state space by converting the values of water per inhabitant into intervals, called levels<fnref n="6" />. We use two parameters to define these levels: i) $m$ — the minimum legal water requirement for a person per day (in our case 100l), and ii) $M$ — the desired water level per person in all villages, e.g., the level at which no further limitations are applied (in our case 350l). Using $m$ and $M$ we define the following levels:

* Level 0 (red level), representing water levels below $m - (M + m)/2$ (or 0 as a lower bound)

<footnote n="5">⁵ Royal Decree 3/2023, of January 10, establishing the technical-sanitary criteria for the quality of drinking water, its control, and supply, Article 9.</footnote>

<footnote n="6">⁶ To calculate $f_{gini}$ we still use the original states.</footnote>
consumption endowment, as a minimum objective, must be at least 100 litres per inhabitant per day.<fnref n="5" />

This information inspires the idea to model that different societies (or villages in our case) would have different water consumption patterns. In our experiments, we will model different consumption rates per time step (the period of one movement of the tanker vehicle). We consider that the consumption will increase as water availability in a village increases (e.g., due to a diversification of uses), but differently depending on each village:

* Village 0 (Brown-25 inhabitants): This village consumes 4l per inhabitant per hour in scarcity, unless the available water is above 350l per inhabitant, where it will consume 100l per inhabitant per hour due to large irrigation needs.

* Village 1 (Blue-260 inhabitants): In this village only 3.5l per inhabitant per hour is used unless there is more than 250l per inhabitant. Then, 9l per inhabitant per hour will be consumed.

* Village 2 (Yellow-1000 inhabitants): In this case 3.5l is consumed per inhabitant per hour if less than 350l per inhabitant is in reserve. Otherwise, per inhabitant consumption rises to 50l per hour.

* Village 3 (Magenta-1050 inhabitants). In this village, the consumption rate will be 3.5l up to a reserve of 100l per inhabitant. After that, the consumption will be 16l per inhabitant.

## 4.2 Modeling states, transitions, and paths

A possible state- and transition-based modelling of the decision problem for water distribution among the four villages is as follows:

- **States:** conceptually, we will represent a state $s \in S$ as a list $(x_0, x_1, x_2, x_3, p, c)$ where each value $x_j \in \mathbb{R}_{\ge 0}$ represents the amount of water per inhabitant in village $j$; $p \in \{-1, 0, 1, 2, 3\}$ represents the current position of the vehicle ($-1$ for source); and $c$ stands for the current amount of water in the vehicle.

- **Actions:** an action $a \in A$ is a pair $(p, d)$ indicating the place $p \in \{-1, 0, 1, 2, 3\}$ to go to by the vehicle and $d \in (0, C]$ the water to dispense after arriving at $p$ (irrelevant when $p = -1$, as then the vehicle will just refill). As described before, the available actions depend on each state.

- **Transitions:** a transition from state $s = (x_0, \dots, x_3, p, c)$ to state $s' = (x'_0, \dots, x'_3, p', c')$ by applying action $a = (p', d)$ means that $s'$ is calculated by setting $c' = c - d$ if $p \neq -1$ (else, $c' = C$ for refilling) and calculating $x'_i$ by first letting the vehicle move (and dispense the water to $p$) and then applying the corresponding water consumption rate to each village as described in Section 4.1.

The decision-making solution in the distribution of water between villages is a (deterministic) Markov policy. We are interested in finding policies that generate certain behaviours (e.g., fulfil certain admissible criteria) and also lead to paths with high value-alignment with respect to a semantics function and aggregation function.

## 4.3 Modelling equity

In the water distribution scenario, we are interested in the value of equity. Thus, we need to define what it means to have a fair or equitable distribution of water in our system at each moment (i.e. how much our states of the system are aligned to the equity value). We will use a semantics function that reflects equity in terms of water availability per person, not per village, assuming each person has equal access to the water of the village he or she belongs to. Specifically, let $n_i$

denote the population of village $i$ and $s = (x_0, \dots, x_3, p, d)$ be a state of the environment. We define the water distribution of state $s$, $D(s)$ by:

$$D(s) = \underbrace{(x_0, \dots, x_0, \dots, x_3, \dots, x_3)}_{n_0 \text{ times}}$$

In this distribution, each person counts with his or her own water available water resources. For example, if village 0 (with 25 inhabitants) has 20l per inhabitant, there are 25 entries with 20l.

Based on $D$ we define the semantics function $f_{gini}(s) = 1 - GI(D(s))$, where $GI$ is the Gini Index, a widely accepted inequality metric in the literature [16, 21, 10]. Note $f_{gini}$ is bounded in $[0, 1]$, and is proportional to equity.

In the next section, we will use the 3 behaviours presented in Section 3 for assessing equitable paths. We will define settings for learning policies that maximize the aggregated alignment of paths and also implement $\epsilon$-local / $\epsilon$-local and $\tau$-constrained behaviours. We will compare those policies with a policy that simply adheres to a local behaviour.

# 5 Learning behaviour-compliant policies

In our scenario, given a current state $s$, we can predict the value-alignment of the resulting states if we would apply any possible action to $s$. This can be done by simply calculating $f_{gini}$ on the estimated resulting states. Thus, a policy adhering to a local behaviour can be implemented easily by seeking the best action among all possibilities in each moment. However, using brute force search to find a policy that maximizes the value-alignment of an execution path is often not feasible, even for simple tasks. In our approach, we propose to use Reinforcement Learning (RL) [25] as a technique to learn such policies.

Our use case (Section 4) can be represented by a Markov Decision Process (MDP) where in any state a more or less equity-valuable action can be taken. This equity-valuable concept is directly represented via state-action rewards $r(s, a)$, which are then used to define the RL value $V(s, a)$ representing future expected reward (i.e. "long-term" equity) in the system. This value function $V(s, a)$ is used by the policy to select action $a$ from $s$. Specifically, for any state $s_j$ at a step $j$, we calculate the reward of applying an action $a_j$ from that state such that $s_j \xrightarrow{a_j} s_{j+1}$ as $r_j(s_j, a_j) = f_{gini}(s_{j+1})$.

Given this reward definition, our aim is to learn policies that maximize path value-alignment and that are at the same time behaviour-compliant with $\epsilon$-local and/or $\tau$-constrained behaviours.

In order to keep the learning problem tractable, we apply a simplification to both state and action spaces. With regard to actions, we only allow deliveries of water (the value $d$ from action pair $(p, d)$) which are multiples of a fixed quantity. In our experiments, we used 15000l and allowed only deliveries of 0l, 15000l, 30000l, 45000l and 60000l (which is the capacity of the truck).

With regard to states, we reduce the state space by converting the values of water per inhabitant into intervals, called levels<fnref n="6" />. We use two parameters to define these levels: i) $m$ — the minimum legal water requirement for a person per day (in our case 100l), and ii) $M$ — the desired water level per person in all villages, e.g., the level at which no further limitations are applied (in our case 350l). Using $m$ and $M$ we define the following levels:

* Level 0 (red level), representing water levels below $m - (M + m)/2$ (or 0 as a lower bound)

<footnote n="5">⁵ Royal Decree 3/2023, of January 10, establishing the technical-sanitary criteria for the quality of drinking water, its control, and supply, Article 9.</footnote>

<footnote n="6">⁶ To calculate $f_{gini}$ we still use the original states.</footnote>
* Level L (green level), representing levels over $M + (M + m)/2$.

* Levels between 0 and L, representing evenly distributed intervals between water levels 0 and L. The number of hidden levels is H, a hyperparameter of the policy.

In our approach, we apply Double Q-learning [11], in its basic tabular version as the basic RL algorithm.<fnref n="7" /> Moreover, we change the typical RL expected discounted reward maximization goal to an average reward maximization goal equivalent to our path aggregated alignment notion (Chapter 10, Sutton and Barto [25]). This change is motivated also by the fact that we need to cope with a continuous goal problem.

## 5.1 Learning policies for $\epsilon$-local behaviours

We first present Algorithm 1 ($\epsilon$-ADQL, $\epsilon$-local Average Double Q-learning) for learning $\epsilon$-local behaviour adherence policies (i.e. $\epsilon$-local policies) while maximizing average state value-alignment. It is directly based on CDQL by Kalweit et al. [13]. The safe set $S_C(s)$ defined by [13], corresponds in our case to the set of $\epsilon$-admissible actions $A_\epsilon(s)$.

Algorithm 1 is able to learn a policy $\pi^*$ which finds the path with optimal average alignment of the path states while adhering to an $\epsilon$-local behaviour. The learned policy samples paths with the following formula:

$$ \pi^*(s) = \arg\max_{a \in A_\epsilon(s)} Q(s, a) + Q'(s, a) \quad (1) $$

**Algorithm 1 Epsilon-local Average Double Q-Learning $\epsilon$-ADQL**

1: Initialize two Q-tables [11] Q and $Q'$, exploration rate $p \in (0, 1)$ and average reward estimator $\hat{r} = 0$ (and its learning rate $\beta \in (0, 1)$).

2: **for** optimization step $o = 1, 2, \dots, N$ **do**

3: $j = 0$

4: $s_0 \leftarrow \text{RESET}(\text{environment})$

5: **while** environment is not done **do**:

6: Sample $n \sim \text{Uniform}([0, 1])$

7: **if** $n < p$ **then** choose $a_j \in A(s_j)$ randomly

8: **else**

9: Compute $A_\epsilon(s_j)$ ($\epsilon$-local valid actions)

10: $a_j \leftarrow \arg\max_{a \in A_\epsilon(s_j)} Q(s_j, a) + Q'(s_j, a)^8 \triangleright \text{Equation 1}$

11: **end if**

12: $(s_{j+1}, r_j) \leftarrow \text{STEP}(\text{environment}, a_j) \triangleright \text{Recall } r_j = f_{gini}(s_{j+1})$

13: $Q_1, Q_2 = \text{PERMUTERANDOMLY}(Q, Q') \triangleright (\text{Changes to } Q_i \text{ are kept back to } Q \text{ and } Q')$

14: Compute $A_\epsilon(s_{j+1})$

15: $a_{Q_2, j+1} = \arg\max_{a \in A_\epsilon(s_{j+1})} Q_2(s_{j+1}, a)$

16: $\delta_j \leftarrow r - \hat{r} + Q_2(s_{j+1}, a_{Q_2, j+1}) - Q_1(s_j, a_j) \triangleright \text{TD er.}$

17: $\hat{r} \leftarrow \hat{r} + \beta\delta_j \triangleright \text{Average reward estimation [25]}$

18: $Q_1(s_j, a_j) \leftarrow \alpha\delta_j \triangleright \text{Update } Q_1 \text{ with } Q_2 \text{ criterion.}$

19: $j \leftarrow j + 1$

20: **end while**

21: (Optional) Decrement $p$ w.r.t. $o$.

22: **end for**

## 5.2 Learning policies for $\tau$-constrained and $\epsilon$-local behaviours

Algorithm 2 presents an algorithm for learning policies that adhere to both: $\tau$-constrained and $\epsilon$-local behaviours. We call this algorithm $\epsilon$-CADQL ($\epsilon$-local Constrained Average Double Q-Learning).

We should be aware that we cannot just proceed as with the $\epsilon$-local case, because there might be situations where the semantics function of all possible direct future states will be below $\tau$. This could happen, for example, when initializing the world to a state $s$ with an alignment value much lower than $\tau$. In such cases, we would like the algorithm to measure/count the $\tau$ bound violations of sampled paths for learning policies minimizing the proportion of violations per path length (violation ratio, $V$). To do so, we modified our previous Algorithm 1, adding an implementation of an intelligent reward shaping approach via Lagrangian multipliers as proposed in [26]. Following their method, RCPO, we first adopt the following penalties $c(s, a)$ and corresponding constraint $C(s_0)$ for a path $s_0 \xrightarrow{a_0} \dots \xrightarrow{a_{M-1}} s_M$:

$$ c(s, a) = \begin{cases} 1, & \text{if violation detected applying } a \\ 0, & \text{otherwise} \end{cases} \quad (2) $$

$$ C(s_0) = \sum_{j=0}^{M} c(s_j, a_j) \le 0 \quad (3) $$

This definition ensures local minima of the constraint are feasible solutions, and $C(s_0) > 0$ represents the total behaviour violations of a path. As done in [26], we use a learned Lagrangian parameter $\lambda$, and we also model a suitable $\Gamma_\lambda$ projection, as follows.

$$ \Gamma_{\lambda}(x) = \min\{x, \bar{R}/\bar{V}\} $$

Here, $\bar{R}$ is the expected path average reward (without any penalties considered) and $\bar{V}$ is the expected violation ratio. Simply put, all the elements above serve to work with a modified aggregated reward given by calculating in each transition $s_j \xrightarrow{a_j} s_{j+1}$ (with observed reward $r_j$) a modified reward by subtracting to $r_j$ a penalty of $c(s_j, a_j)$ weighted by the factor $\lambda$ (which approximates to $\bar{R}/\bar{V}$ as Algorithm 2 evolves), and then averaging over all transitions. With this process, one can see that if $\lambda \approx \bar{R}/\bar{V}$, the algorithm will assign a (modified) aggregated reward value of 0 to the paths that get exactly a violation ratio of $\bar{V}$; a negative value to those with a bigger ratio; and a positive value to paths with a smaller one. In particular, it is guaranteed that non-violating paths are preferred over violating paths, no matter their respective values of their value-alignment.

At the beginning of the algorithm, $\lambda$ is set to 0. This encourages exploration even while committing infractions. As the training process evolves, the algorithm will learn better paths allowing $\lambda$ to grow in the process (diminishing the possibility of incurring violations, when possible). Though this might create bias, we update $\lambda$ much slower than the policy (convergence idea from [26]), thus, temporal biases are less meaningful. After each completed episode we approximate $\bar{R} \approx \hat{R}$ and $\bar{V} \approx \hat{V}$ on the go with very conservative exponential weighted averages $\beta_V, \beta_R \approx 0.001$, after calculating the next $\lambda$ using approximate projection $\hat{\Gamma}_\lambda(\lambda) = \min\{\lambda, \frac{\hat{R}}{\hat{V}}\}$.

After learning, the result of Algorithm 2 is a policy (defined through equation 1) which implements an $\epsilon$-local and also a $\tau$-constrained behaviour.

<footnote n="7">7 Double Q-learning reduces training biases of normal Q-learning (though reducing sample efficiency). This RL algorithm choice is not critical, though, as our proposed Algorithms 1 and 2 will work fine with typical Q-learning too.</footnote>

<footnote n="8">8 With some abuse of notation, $Q(s, a)$ denotes in reality the $Q$-table value of the leveled representation of state $s$ under action $a$.</footnote>
* Level L (green level), representing levels over $M + (M + m)/2$.

* Levels between 0 and L, representing evenly distributed intervals between water levels 0 and L. The number of hidden levels is H, a hyperparameter of the policy.

In our approach, we apply Double Q-learning [11], in its basic tabular version as the basic RL algorithm.<fnref n="7" /> Moreover, we change the typical RL expected discounted reward maximization goal to an average reward maximization goal equivalent to our path aggregated alignment notion (Chapter 10, Sutton and Barto [25]). This change is motivated also by the fact that we need to cope with a continuous goal problem.

## 5.1 Learning policies for $\epsilon$-local behaviours

We first present Algorithm 1 ($\epsilon$-ADQL, $\epsilon$-local Average Double Q-learning) for learning $\epsilon$-local behaviour adherence policies (i.e. $\epsilon$-local policies) while maximizing average state value-alignment. It is directly based on CDQL by Kalweit et al. [13]. The safe set $S_C(s)$ defined by [13], corresponds in our case to the set of $\epsilon$-admissible actions $A_\epsilon(s)$.

Algorithm 1 is able to learn a policy $\pi^*$ which finds the path with optimal average alignment of the path states while adhering to an $\epsilon$-local behaviour. The learned policy samples paths with the following formula:

$$ \pi^*(s) = \arg\max_{a \in A_\epsilon(s)} Q(s, a) + Q'(s, a) \quad (1) $$

**Algorithm 1 Epsilon-local Average Double Q-Learning $\epsilon$-ADQL**

1: Initialize two Q-tables [11] Q and $Q'$, exploration rate $p \in (0, 1)$ and average reward estimator $\hat{r} = 0$ (and its learning rate $\beta \in (0, 1)$).

2: **for** optimization step $o = 1, 2, \dots, N$ **do**

3: $j = 0$

4: $s_0 \leftarrow \text{RESET}(\text{environment})$

5: **while** environment is not done **do**:

6: Sample $n \sim \text{Uniform}([0, 1])$

7: **if** $n < p$ **then** choose $a_j \in A(s_j)$ randomly

8: **else**

9: Compute $A_\epsilon(s_j)$ ($\epsilon$-local valid actions)

10: $a_j \leftarrow \arg\max_{a \in A_\epsilon(s_j)} Q(s_j, a) + Q'(s_j, a)^8 \triangleright \text{Equation 1}$

11: **end if**

12: $(s_{j+1}, r_j) \leftarrow \text{STEP}(\text{environment}, a_j) \triangleright \text{Recall } r_j = f_{gini}(s_{j+1})$

13: $Q_1, Q_2 = \text{PERMUTERANDOMLY}(Q, Q') \triangleright (\text{Changes to } Q_i \text{ are kept back to } Q \text{ and } Q')$

14: Compute $A_\epsilon(s_{j+1})$

15: $a_{Q_2, j+1} = \arg\max_{a \in A_\epsilon(s_{j+1})} Q_2(s_{j+1}, a)$

16: $\delta_j \leftarrow r - \hat{r} + Q_2(s_{j+1}, a_{Q_2, j+1}) - Q_1(s_j, a_j) \triangleright \text{TD er.}$

17: $\hat{r} \leftarrow \hat{r} + \beta\delta_j \triangleright \text{Average reward estimation [25]}$

18: $Q_1(s_j, a_j) \leftarrow \alpha\delta_j \triangleright \text{Update } Q_1 \text{ with } Q_2 \text{ criterion.}$

19: $j \leftarrow j + 1$

20: **end while**

21: (Optional) Decrement $p$ w.r.t. $o$.

22: **end for**

## 5.2 Learning policies for $\tau$-constrained and $\epsilon$-local behaviours

Algorithm 2 presents an algorithm for learning policies that adhere to both: $\tau$-constrained and $\epsilon$-local behaviours. We call this algorithm $\epsilon$-CADQL ($\epsilon$-local Constrained Average Double Q-Learning).

We should be aware that we cannot just proceed as with the $\epsilon$-local case, because there might be situations where the semantics function of all possible direct future states will be below $\tau$. This could happen, for example, when initializing the world to a state $s$ with an alignment value much lower than $\tau$. In such cases, we would like the algorithm to measure/count the $\tau$ bound violations of sampled paths for learning policies minimizing the proportion of violations per path length (violation ratio, $V$). To do so, we modified our previous Algorithm 1, adding an implementation of an intelligent reward shaping approach via Lagrangian multipliers as proposed in [26]. Following their method, RCPO, we first adopt the following penalties $c(s, a)$ and corresponding constraint $C(s_0)$ for a path $s_0 \xrightarrow{a_0} \dots \xrightarrow{a_{M-1}} s_M$:

$$ c(s, a) = \begin{cases} 1, & \text{if violation detected applying } a \\ 0, & \text{otherwise} \end{cases} \quad (2) $$

$$ C(s_0) = \sum_{j=0}^{M} c(s_j, a_j) \le 0 \quad (3) $$

This definition ensures local minima of the constraint are feasible solutions, and $C(s_0) > 0$ represents the total behaviour violations of a path. As done in [26], we use a learned Lagrangian parameter $\lambda$, and we also model a suitable $\Gamma_\lambda$ projection, as follows.

$$ \Gamma_\lambda(x) = \min\{x, \bar{R}/\bar{V}\} $$

Here, $\bar{R}$ is the expected path average reward (without any penalties considered) and $\bar{V}$ is the expected violation ratio. Simply put, all the elements above serve to work with a modified aggregated reward given by calculating in each transition $s_j \xrightarrow{a_j} s_{j+1}$ (with observed reward $r_j$) a modified reward by subtracting to $r_j$ a penalty of $c(s_j, a_j)$ weighted by the factor $\lambda$ (which approximates to $\bar{R}/\bar{V}$ as Algorithm 2 evolves), and then averaging over all transitions. With this process, one can see that if $\lambda \approx \bar{R}/\bar{V}$, the algorithm will assign a (modified) aggregated reward value of 0 to the paths that get exactly a violation ratio of $\bar{V}$; a negative value to those with a bigger ratio; and a positive value to paths with a smaller one. In particular, it is guaranteed that non-violating paths are preferred over violating paths, no matter their respective values of their value-alignment.

At the beginning of the algorithm, $\lambda$ is set to 0. This encourages exploration even while committing infractions. As the training process evolves, the algorithm will learn better paths allowing $\lambda$ to grow in the process (diminishing the possibility of incurring violations, when possible). Though this might create bias, we update $\lambda$ much slower than the policy (convergence idea from [26]), thus, temporal biases are less meaningful. After each completed episode we approximate $\bar{R} \approx \hat{R}$ and $\bar{V} \approx \hat{V}$ on the go with very conservative exponential weighted averages $\beta_V, \beta_R \approx 0.001$, after calculating the next $\lambda$ using approximate projection $\hat{\Gamma}_\lambda(\lambda) = \min\{\lambda, \frac{\bar{R}}{\bar{V}}\}$.

After learning, the result of Algorithm 2 is a policy (defined through equation 1) which implements an $\epsilon$-local and also a $\tau$-constrained behaviour.

<footnote n="7">7 Double Q-learning reduces training biases of normal Q-learning (though reducing sample efficiency). This RL algorithm choice is not critical, though, as our proposed Algorithms 1 and 2 will work fine with typical Q-learning too.</footnote>

<footnote n="8">8 With some abuse of notation, $Q(s, a)$ denotes in reality the $Q$-table value of the leveled representation of state $s$ under action $a$.</footnote>
**Algorithm 2 Epsilon-local Constrained Average Double Q-Learning**

1: Initialize $Q, Q', j = 0$, exp. rate $p \in (0, 1)$, $\hat{r}, \hat{V}, \hat{R} = 0$ (and $\beta \in (0, 1)$), penalty factor $\lambda > 0$, and rates $\alpha > \beta_R > \beta_V > \alpha_\lambda$ (following [26]).

2: **for** optimization step $o = 1, 2, \dots, N$ **do**

3: $s_0 \leftarrow \text{RESET}(\text{environment}), j \leftarrow 0, R_o \leftarrow 0, C_o \leftarrow 0$

4: **while** environment is not done **do**:

5: $(s_j, a_j, s_{j+1}, r_j) \leftarrow \text{Algorithm 1, lines 6-12}$.

6: $R_o \leftarrow R_o + r_j$

7: **if** $c(s_j, a_j) > 0$ **then**

8: $r_j \leftarrow r_j - \lambda_o c(s_j, a_j)$

9: $C_o \leftarrow C_o + 1$

10: **end if**

11: $Q_1, Q_2 = \text{PERMUTERANDOMLY}(Q, Q')$

12: Update $Q_1$ and $Q_2$ as for Algorithm 1, Lines 14-16.

13: $j \leftarrow j + 1$

14: **end while**

15: (Optional) Decrement $p$ w.r.t. $o$.

16: $\lambda_{o+1} \leftarrow \hat{\Gamma}_\lambda(\lambda_o + \alpha_\lambda \cdot C_o) \triangleright \text{RCPO Lagrange update [26]}$

17: $\hat{V} \leftarrow \beta_V \frac{C_o}{j} + (1 - \beta_V) \hat{V}$

18: $\hat{R} \leftarrow \beta_R \frac{R_o}{j} + (1 - \beta_R) \hat{R}$

19: **end for**

# 6 Evaluation

The objective of the evaluation is: i) to see the advantages of relaxing the local policy (using the $\epsilon$-local concept) towards maximizing the value-alignment of paths, and ii) to see implications of the simultaneously $\tau$-constrained and $\epsilon$-local policy trained via Algorithm 2.

## 6.1 Training setup

The training environment was programmed using the former OpenAI Gym [3] library, now held by [7]. The environment was extended to add a method to get the sets $A_\epsilon(s_j)$ after every step. In the experiments, we used the connected map in Figure 1, with village parameters from Section 4 and the following hyperparameters:

* $\alpha = 0.03$, $\alpha_\lambda = 0.0003$; $\beta = 0.01$, $\beta_V = \beta_R = 0.001$.

* $H = 5$ hidden levels for distribution representation.

* $\epsilon = 0.1$, $\tau = 0.7$.

* $N = 30000$ iterations. Exploration $p = 0.3 \to 0$, as $o \to N$.

In the training processes, an episode finishes when 1440000 litres of water are distributed. Furthermore, in each episode, the initial state is randomly RESET with village water levels sampled from a uniform distribution between 0 and 600l.

After the training processes, the obtained policies have been applied in an evaluation scenario. It has a default initial state of $(0, 300, 200, 200, -1, 60000)$ and the experiment is run until 3000000l of water are distributed.

## 6.2 Experiments

First, we want to highlight the advantages that can be obtained by using the policy that implements an $\epsilon$-local behaviour over following the simple local strategy. The performance of the local policy is shown in Figure 2. Notice, that apart from making the equity criteria worse over time, Village 1 gets too much water. This is possibly due to the fact that village 1 has many connections and thus, the truck is more often forced to dispose of water to that village.

<figure><img src="image_4.png" /><figcaption>**Figure 2.** Local policy. Top: distribution of water and the total amount of available water at the end of the episode. Bottom: Equity (rewards) at each step and aggregated average (i.e. average reward until each step).</figcaption></figure>

Analysing the results over time, it is noticeable that the truck never visits Village 0 (with just 25 inhabitants). This is because from the point of view of equity, in the short term, it is normally better to bring water to populous villages. Furthermore, the truck would discharge too much water for this small village, massively increasing its water per inhabitant, leading to less equity. The local policy is not able to take into account the water consumption of Village 0, which stabilizes in a few time steps.

Figure 3 shows the results obtained with the two policies $\epsilon$-ADQL and $\epsilon$-CADQL, learned with algorithms 1 and 2. When comparing the ($\epsilon$-ADQL) strategy with the simple local policy, clearly, the $\epsilon$-local policy is able to improve the value-alignment of the whole path (e.g., the averaged historic equity/reward over all states of the path). Here, the obtained level is about 0.84, versus 0.71 of the local policy.

<figure><img src="image_5.png" /><figcaption>**Figure 3.** Performance of policies $\epsilon$-ADQL (left) and $\epsilon$-CADQL (right), with $\epsilon = 0.1$ and $\tau = 0.7$ marked at the horizontal black line. The upper diagrams show the distribution of water among the villages at the end of the episodes. The lower diagrams present the historic state equity (red) and aggregated average equity (blue).</figcaption></figure>

Comparing the obtained results to the $\tau$-constrained/$\epsilon$-local $\epsilon$-CADQL policy, we first see that the black line (the $\tau = 0.7$ bound) is avoided. In comparison, the local policy surpasses the bound many times and the $\epsilon$-ADQL policy does it once (approximately at time step 40). Additionally, the aggregated alignment-path average reward- of $\epsilon$-CADQL is similar to $\epsilon$-ADQL, albeit a bit lower: average reward of 0.812 versus 0.842. This behaviour seems logical, since in general, $\epsilon$-ADQL has more degrees of freedom for acting and
**Algorithm 2 Epsilon-local Constrained Average Double Q-Learning**

1: Initialize $Q, Q', j = 0$, exp. rate $p \in (0, 1)$, $\hat{r}, \hat{V}, \hat{R} = 0$ (and $\beta \in (0, 1)$), penalty factor $\lambda > 0$, and rates $\alpha > \beta_R > \beta_V > \alpha_\lambda$ (following [26]).

2: **for** optimization step $o = 1, 2, \dots, N$ **do**

3: $s_0 \leftarrow \text{RESET}(\text{environment}), j \leftarrow 0, R_o \leftarrow 0, C_o \leftarrow 0$

4: **while** environment is not done **do**:

5: $(s_j, a_j, s_{j+1}, r_j) \leftarrow \text{Algorithm 1, lines 6-12}$.

6: $R_o \leftarrow R_o + r_j$

7: **if** $c(s_j, a_j) > 0$ **then**

8: $r_j \leftarrow r_j - \lambda_o c(s_j, a_j)$

9: $C_o \leftarrow C_o + 1$

10: **end if**

11: $Q_1, Q_2 = \text{PERMUTERANDOMLY}(Q, Q')$

12: Update $Q_1$ and $Q_2$ as for Algorithm 1, Lines 14-16.

13: $j \leftarrow j + 1$

14: **end while**

15: (Optional) Decrement $p$ w.r.t. $o$.

16: $\lambda_{o+1} \leftarrow \hat{\Gamma}_\lambda(\lambda_o + \alpha_\lambda \cdot C_o) \triangleright \text{RCPO Lagrange update [26]}$

17: $\hat{V} \leftarrow \beta_V \frac{C_o}{j} + (1 - \beta_V) \hat{V}$

18: $\hat{R} \leftarrow \beta_R \frac{R_o}{j} + (1 - \beta_R) \hat{R}$

19: **end for**

# 6 Evaluation

The objective of the evaluation is: i) to see the advantages of relaxing the local policy (using the $\epsilon$-local concept) towards maximizing the value-alignment of paths, and ii) to see implications of the simultaneously $\tau$-constrained and $\epsilon$-local policy trained via Algorithm 2.

## 6.1 Training setup

The training environment was programmed using the former OpenAI Gym [3] library, now held by [7]. The environment was extended to add a method to get the sets $A_\epsilon(s_j)$ after every step. In the experiments, we used the connected map in Figure 1, with village parameters from Section 4 and the following hyperparameters:

* $\alpha = 0.03$, $\alpha_\lambda = 0.0003$; $\beta = 0.01$, $\beta_V = \beta_R = 0.001$.

* $H = 5$ hidden levels for distribution representation.

* $\epsilon = 0.1$, $\tau = 0.7$.

* $N = 30000$ iterations. Exploration $p = 0.3 \to 0$, as $o \to N$.

In the training processes, an episode finishes when 1440000 litres of water are distributed. Furthermore, in each episode, the initial state is randomly RESET with village water levels sampled from a uniform distribution between 0 and 600l.

After the training processes, the obtained policies have been applied in an evaluation scenario. It has a default initial state of $(0, 300, 200, 200, -1, 60000)$ and the experiment is run until 3000000l of water are distributed.

## 6.2 Experiments

First, we want to highlight the advantages that can be obtained by using the policy that implements an $\epsilon$-local behaviour over following the simple local strategy. The performance of the local policy is shown in Figure 2. Notice, that apart from making the equity criteria worse over time, Village 1 gets too much water. This is possibly due to the fact that village 1 has many connections and thus, the truck is more often forced to dispose of water to that village.

<figure><img src="image_4.png" /><figcaption>**Figure 2.** Local policy. Top: distribution of water and the total amount of available water at the end of the episode. Bottom: Equity (rewards) at each step and aggregated average (i.e. average reward until each step).</figcaption></figure>

Analysing the results over time, it is noticeable that the truck never visits Village 0 (with just 25 inhabitants). This is because from the point of view of equity, in the short term, it is normally better to bring water to populous villages. Furthermore, the truck would discharge too much water for this small village, massively increasing its water per inhabitant, leading to less equity. The local policy is not able to take into account the water consumption of Village 0, which stabilizes in a few time steps.

Figure 3 shows the results obtained with the two policies $\epsilon$-ADQL and $\epsilon$-CADQL, learned with algorithms 1 and 2. When comparing the ($\epsilon$-ADQL) strategy with the simple local policy, clearly, the $\epsilon$-local policy is able to improve the value-alignment of the whole path (e.g., the averaged historic equity/reward over all states of the path). Here, the obtained level is about 0.84, versus 0.71 of the local policy.

<figure><img src="image_5.png" /><figcaption>**Figure 3.** Performance of policies $\epsilon$-ADQL (left) and $\epsilon$-CADQL (right), with $\epsilon = 0.1$ and $\tau = 0.7$ marked at the horizontal black line. The upper diagrams show the distribution of water among the villages at the end of the episodes. The lower diagrams present the historic state equity (red) and aggregated average equity (blue).</figcaption></figure>

Comparing the obtained results to the $\tau$-constrained/$\epsilon$-local $\epsilon$-CADQL policy, we first see that the black line (the $\tau = 0.7$ bound) is avoided. In comparison, the local policy surpasses the bound many times and the $\epsilon$-ADQL policy does it once (approximately at time step 40). Additionally, the aggregated alignment-path average reward- of $\epsilon$-CADQL is similar to $\epsilon$-ADQL, albeit a bit lower: average reward of 0.812 versus 0.842. This behaviour seems logical, since in general, $\epsilon$-ADQL has more degrees of freedom for acting and
thus, could find better policies regarding long-term value-alignment.
Still, the ε-CADQL policy is capable of bringing water to all villages,
which is an advantage against the local policy.

To have a clearer comparison between the three policies, we pro-
vide Figure 4. On the left side of this figure, we see the results when ε
is set to 0.1, which was already discussed for each individual policy.
On the right side of Figure 4, we present the obtained results when
applying a smaller ε ( ε = 0.01) in the evaluation. That is, the poli-
cies have been learned with ε = 0.1, but are then evaluated with the
reduced ε of 0.01, reducing the set of available actions greatly.

<figure><img src="image_6.png" /><figcaption>Figure 4. Historic state equity ($f_{gini}$) for ε-CADQL (red), ε-ADQL (blue) and local policy (green). Left: results with ε = 0.1 and right: results with ε = 0.01 without retraining.</figcaption></figure>

To see the relevance of the proposed training algorithms, a fourth ADQL (Average Double Q-Learning) policy was trained with ε-ADQL (Algorithm 1) but with ϵ = 1. We call it “RL policy” in our diagrams. Considering that our reward is bounded in [0, 1], this makes the training process equivalent to that of pure ADQL. The difference is in the sampling method, still done with Equation 1, to adhere to the ε-local behaviour.

Figure 5 shows the results of another experiment where we aver-
aged the historic state equity/rewards of the paths obtained by the
four policies from 1000 initial states sampled randomly with the pro-
cess in Section 6.1 with, again, ϵ = 0.1 (left) and ϵ = 0.01 (right).<fnref n="9" />

The conclusions are the following. First, the three RL algorithms
outperform in the mid/long-term the naively admissible local policy.
Second, though ε-ADQL achieves the best controlled performance<fnref n="10" />
under no changes in ε, probably due to a the bigger state-space ex-
ploration, the ADQL policy is superior to the training-constrained
proposals—it has a 3.65% violation ratio and an expected average
reward (Score) of 0.86 versus ≈ 5.6% and ≈ 0.82 of our proposals,
respectively—. Third, both the ε-ADQL and ADQL policies fail in
the mid-long term as their scores drop below 0.81 and their viola-
tion ratios grow to at least 18% (suggesting a struggle for a sustained
alignment); while ε-CADQL will not, even diminishing its penalties
to 3.56% and increasing its score to 0.84), showing τ constraint is
avoided the intended way.

<figure><img src="image_7.png" /><figcaption>**Figure 5.** Expected historic state equity ($f_{gini}$) for ε-CADQL (dark red), ε-ADQL (purple), local (dark green) and normal DQL (grey) policies. Left: results with $\epsilon = 0.1$ and right: results with $\epsilon = 0.01$. For each algorithm, the expected aggregated equity alignment is specified by the term *Score* and the expected *violation ratio* per time series length is given under *Penalty\_perc*.</figcaption></figure>

7 Conclusions and Future Work

In this paper we have shown the importance of value-admissible
behaviours for constructing value-aware decision-making systems.
In previous work, these behaviours were proposed to enrich value-
aligned goal-oriented decision-making, motivated by an equitable
water distribution example. In this paper, we have revisited this sce-
nario, introducing an infinite horizon distribution simulation. While
adhering to two very general behaviours, namely the ϵ-local be-
haviour (which relaxes an immediate need for short-term alignment)
and the newly proposed τ-constrained behaviour (which introduces
"red line" constraints on possible states) we found more equitable
paths in the long term than using the justifiable short-term focused
local behaviour. We implemented two algorithms (ε-ADQL and ε-
CADQL) that use reinforcement learning to learn policies that im-
plement the corresponding behaviours. In both cases we use an aver-
age reward setting, where rewards directly encode a value seman-
tics function. We have proven the robustness and relevance of ε-
CADQL in particular for providing equity-safe solutions by com-
bining aspects of the two admissible behaviours used. We concluded
that value-aware decision-making is possible with the learned poli-
cies combining complex value admissibility requisites.

As future work, note that the proposed algorithms need a special
procedure for identifying the set of ϵ-local admissible actions. Cal-
culating it is time-consuming and will be infeasible in environments
with a continuous action space. An alternative is to learn to identify
the ϵ-local admissible actions in constant time, e.g. with a discounted
DQL algorithm with a small discount factor.

Extending constraint value-aligned decision-making with *value
systems* is another open line of work, where complex value inter-
actions should be represented not only by behaviours and state pref-
erences but with explicable value taxonomies [19].

Acknowledgements

This work has been supported by grant VAE: TED2021-
131295B-C33 funded by MCIN/AEI/ 10.13039/501100011033
and by the “European Union NextGenerationEU/PRTR”, by

<footnote n="9">⁹ Different initial states and algorithms give different length paths. For visual purposes, the average historic reward time series seen in Figure 5 have been calculated by making all of them the same length, enlarging the shorter ones by repeating their ending rewards until getting as long as the longest series, which is previously cropped up to a maximum length of 1.2 times the default initial state experiment series. All the metrics, however, are calculated w.r.t. the original lengths and then averaged.</footnote>

<footnote n="10">¹⁰ Unlike the ADQL policy, ε-ADQL and ε-CADQL are controlled in the</footnote>

<footnote>sense that both adhere to ϵ-local behaviour during training.</footnote>
thus, could find better policies regarding long-term value-alignment.
Still, the ε-CADQL policy is capable of bringing water to all villages,
which is an advantage against the local policy.

To have a clearer comparison between the three policies, we pro-
vide Figure 4. On the left side of this figure, we see the results when ϵ
is set to 0.1, which was already discussed for each individual policy.
On the right side of Figure 4, we present the obtained results when
applying a smaller ϵ ( ϵ = 0.01) in the evaluation. That is, the poli-
cies have been learned with ϵ = 0.1, but are then evaluated with the
reduced ϵ of 0.01, reducing the set of available actions greatly.

<figure><img src="image_6.png" /><figcaption>Figure 4. Historic state equity ($f_{gini}$) for ε-CADQL (red), ε-ADQL (blue) and local policy (green). Left: results with $\epsilon = 0.1$ and right: results with $\epsilon = 0.01$ without retraining.</figcaption></figure>

To see the relevance of the proposed training algorithms, a fourth ADQL (Average Double Q-Learning) policy was trained with ε-ADQL (Algorithm 1) but with ϵ = 1. We call it “RL policy” in our diagrams. Considering that our reward is bounded in [0, 1], this makes the training process equivalent to that of pure ADQL. The difference is in the sampling method, still done with Equation 1, to adhere to the ε-local behaviour.

Figure 5 shows the results of another experiment where we aver-
aged the historic state equity/rewards of the paths obtained by the
four policies from 1000 initial states sampled randomly with the pro-
cess in Section 6.1 with, again, $\epsilon = 0.1$ (left) and $\epsilon = 0.01$ (right).<fnref n="9" />

The conclusions are the following. First, the three RL algorithms outperform in the mid/long-term the naively admissible local policy. Second, though ε-ADQL achieves the best controlled performance<fnref n="10" /> under no changes in ε, probably due to a the bigger state-space exploration, the ADQL policy is superior to the training-constrained proposals—it has a 3.65% violation ratio and an expected average reward (Score) of 0.86 versus ≈ 5.6% and ≈ 0.82 of our proposals, respectively—. Third, both the ε-ADQL and ADQL policies fail in the mid-long term as their scores drop below 0.81 and their violation ratios grow to at least 18% (suggesting a struggle for a sustained alignment); while ε-CADQL will not, even diminishing its penalties to 3.56% and increasing its score to 0.84), showing τ constraint is avoided the intended way.

<figure><img src="image_7.png" /><figcaption>**Figure 5.** Expected historic state equity ($f_{gini}$) for ε-CADQL (dark red), ε-ADQL (purple), local (dark green) and normal DQL (grey) policies. Left: results with $\epsilon = 0.1$ and right: results with $\epsilon = 0.01$. For each algorithm, the expected aggregated equity alignment is specified by the term *Score* and the expected *violation ratio* per time series length is given under *Penalty_percent*.</figcaption></figure>

7 Conclusions and Future Work

In this paper we have shown the importance of value-admissible
behaviours for constructing value-aware decision-making systems.
In previous work, these behaviours were proposed to enrich value-
aligned goal-oriented decision-making, motivated by an equitable
water distribution example. In this paper, we have revisited this sce-
nario, introducing an infinite horizon distribution simulation. While
adhering to two very general behaviours, namely the ϵ-local be-
haviour (which relaxes an immediate need for short-term alignment)
and the newly proposed τ-constrained behaviour (which introduces
"red line" constraints on possible states) we found more equitable
paths in the long term than using the justifiable short-term focused
local behaviour. We implemented two algorithms (ε-ADQL and ϵ-
CADQL) that use reinforcement learning to learn policies that im-
plement the corresponding behaviours. In both cases we use an aver-
age reward setting, where rewards directly encode a value seman-
tics function. We have proven the robustness and relevance of ϵ-
CADQL in particular for providing equity-safe solutions by com-
bining aspects of the two admissible behaviours used. We concluded
that value-aware decision-making is possible with the learned poli-
cies combining complex value admissibility requisites.

As future work, note that the proposed algorithms need a special
procedure for identifying the set of ϵ-local admissible actions. Cal-
culating it is time-consuming and will be infeasible in environments
with a continuous action space. An alternative is to learn to identify
the ϵ-local admissible actions in constant time, e.g. with a discounted
DQL algorithm with a small discount factor.

Extending constraint value-aligned decision-making with *value
systems* is another open line of work, where complex value inter-
actions should be represented not only by behaviours and state pref-
erences but with explicable value taxonomies [19].

Acknowledgements

This work has been supported by grant VAE: TED2021-
131295B-C33 funded by MCIN/AEI/ 10.13039/501100011033
and by the “European Union NextGenerationEU/PRTR”, by

<footnote n="9">⁹ Different initial states and algorithms give different length paths. For visual purposes, the average historic reward time series seen in Figure 5 have been calculated by making all of them the same length, enlarging the shorter ones by repeating their ending rewards until getting as long as the longest series, which is previously cropped up to a maximum length of 1.2 times the default initial state experiment series. All the metrics, however, are calculated w.r.t. the original lengths and then averaged.</footnote>

<footnote n="10">¹⁰ Unlike the ADQL policy, ε-ADQL and ϵ-CADQL are controlled in the</footnote>

<footnote>sense that both adhere to ϵ-local behaviour during training.</footnote>
grant COSASS: PID2021-123673OB-C32 funded by MCIN/AEI/
10.13039/501100011033 and by “ERDF A way of making Europe”,
and by the AGROBOTS Project of Universidad Rey Juan Carlos
funded by the Community of Madrid, Spain.

References

[1] Eitan Altman, *Constrained Markov decision processes*, volume 7, CRC press, 1999.

[2] Trevor Bench-Capon, Katie Atkinson, and Peter McBurney, ‘Using argumentation to model agent decision making in economic experiments’, *Autonomous Agents and Multi-Agent Systems*, **25**, 183–208, (2012).

[3] Greg Brockman, Vicki Cheung, Ludwig Pettersson, Jonas Schneider, John Schulman, Jie Tang, and Wojciech Zaremba, ‘Openai gym’, *arXiv preprint arXiv:1606.01540*, (2016).

[4] Paul Christiano, Jan Leike, Tom B. Brown, Miljan Martic, Shane Legg, and Dario Amodei. Deep reinforcement learning from human preferences, 2023.

[5] Gal Dalal, Krishnamurthy Dvijotham, Matej Vecerik, Todd Hester, Cosmin Paduraru, and Yuval Tassa. Safe exploration in continuous action spaces, 2018.

[6] Sudipto Das, Omer Egecioglu, and Amr El Abbadi, ‘Anónimos: An lp-based approach for anonymizing weighted social network graphs’, *IEEE Transactions on Knowledge and Data Engineering*, **24**(4), 590–604, (2012).

[7] The Farama Foundation. Gymnasium. https://gymnasium.farama.org, 2023.

[8] Johannes Fürnkranz, Eyke Hüllermeier, Weiwei Cheng, and Sang-Hyeun Park, ‘Preference-based reinforcement learning: a formal framework and a policy iteration algorithm’, *Machine learning*, **89**, 123–156, (2012).

[9] Spanish Government, ‘Strategic project for economic recovery and transformation of digitalization of the water cycle. report 2022’, Technical report, Ministry for the Ecological Transition and Demographic Challenge, (2022).

[10] Ting Guo, Yuyu Yuan, and Pengqian Zhao, ‘Admission-based reinforcement-learning algorithm in sequential social dilemmas’, *Applied Sciences*, **13**(3), (2023).

[11] Hado Hasselt, ‘Double q-learning’, *Advances in neural information processing systems*, **23**, (2010).

[12] Andrés Holgado-Sánchez, Joaquín Arias, Mar Moreno-Rebato, and Sascha Ossowski, ‘On admissible behaviours for goal-oriented decision-making of value-aware agents’, in *Multi-Agent Systems*, eds., Vadim Malvone and Aniello Murano, pp. 415–424, Cham, (2023). Springer Nature Switzerland.

[13] Gabriel Kalweit, Maria Huegle, Moritz Werling, and Joschka Boedecker. Deep constrained q-learning, 2020.

[14] Roger Lera-Leri, Filippo Bistaffa, Marc Serramia, Maite Lopez-Sanchez, and Juan Rodriguez-Aguilar, ‘Towards pluralistic value alignment: Aggregating value systems through lp-regression’, in *Proceedings of the 21st International Conference on Autonomous Agents and Multiagent Systems*, AAMAS ’22, p. 780–788, Richland, SC, (2022). International Foundation for Autonomous Agents and Multiagent Systems.

[15] Nieves Montes, Nardine Osman, Carles Sierra, and Marija Slavkovik, ‘Value engineering for autonomous agents’, *CoRR*, abs/2302.08759, (2023).

[16] Nieves Montes and Carles Sierra, ‘Synthesis and properties of optimally value-aligned normative systems’, *Journal of Artificial Intelligence Research*, **74**, 1739–1774, (2022).

[17] Hervé Moulin, *Fair division and collective welfare*, MIT press, 2004.

[18] Andrew Y Ng and Stuart J Russell, ‘Algorithms for inverse reinforcement learning’, in *Proceedings of the Seventeenth International Conference on Machine Learning*, pp. 663–670, (2000).

[19] Nardine Osman and Mark d’Inverno. A computational framework of human values for ethical ai, 2023.

[20] Antoni Perello-Moragues, Manel Poch, David Sauri, Lucia Alexandra Popartan, and Pablo Noriega, ‘Modelling domestic water use in metropolitan areas using socio-cognitive agents’, *Water*, **13**(8), (2021).

[21] L Plata-Pérez, Joss Sánchez-Pérez, and F Sánchez-Sánchez, ‘An elementary characterization of the gini index’, *Mathematical Social Sciences*, **74**, 79–83, (2015).

[22] Manel Rodriguez-Soto, Marc Serramia, Maite Lopez-Sanchez, and Juan Antonio Rodriguez-Aguilar, ‘Instilling moral value alignment by means of multi-objective reinforcement learning’, *Ethics and Information Technology*, **24**, 9, (3 2022).

[23] Shalom H Schwartz, ‘An overview of the schwartz theory of basic values’, *Online readings in Psychology and Culture*, **2**(1), 11, (2012).

[24] Carles Sierra, Nardine Osman, Pablo Noriega, Jordi Sabater-Mir, and Antoni Perelló, ‘Value alignment: a formal approach’, *CoRR*, abs/2110.09240, (2021).

[25] Richard S Sutton and Andrew G Barto, *Reinforcement learning: An introduction*, MIT press, 2018.

[26] Chen Tessler, Daniel J. Mankowitz, and Shie Mannor. Reward constrained policy optimization, 2018.

[27] T. L. van der Weide, F. Dignum, J. J. Ch. Meyer, H. Prakken, and G. A. W. Vreeswijk, ‘Practical reasoning using values’, in *Argumentation in Multi-Agent Systems*, eds., Peter McBurney, Iyad Rahwan, Simon Parsons, and Nicolas Maudet, pp. 79–93, Berlin, Heidelberg, (2010). Springer Berlin Heidelberg.
grant COSASS: PID2021-123673OB-C32 funded by MCIN/AEI/
10.13039/501100011033 and by “ERDF A way of making Europe”,
and by the AGROBOTS Project of Universidad Rey Juan Carlos
funded by the Community of Madrid, Spain.

References

[1] Eitan Altman, *Constrained Markov decision processes*, volume 7, CRC press, 1999.

[2] Trevor Bench-Capon, Katie Atkinson, and Peter McBurney, ‘Using argumentation to model agent decision making in economic experiments’, *Autonomous Agents and Multi-Agent Systems*, **25**, 183–208, (2012).

[3] Greg Brockman, Vicki Cheung, Ludwig Pettersson, Jonas Schneider, John Schulman, Jie Tang, and Wojciech Zaremba, ‘Openai gym’, *arXiv preprint arXiv:1606.01540*, (2016).

[4] Paul Christiano, Jan Leike, Tom B. Brown, Miljan Martic, Shane Legg, and Dario Amodei. Deep reinforcement learning from human preferences, 2023.

[5] Gal Dalal, Krishnamurthy Dvijotham, Matej Vecerik, Todd Hester, Cosmin Paduraru, and Yuval Tassa. Safe exploration in continuous action spaces, 2018.

[6] Sudipto Das, Omer Egecioglu, and Amr El Abbadi, ‘Anónimos: An lp-based approach for anonymizing weighted social network graphs’, *IEEE Transactions on Knowledge and Data Engineering*, **24**(4), 590–604, (2012).

[7] The Farama Foundation. Gymnasium. https://gymnasium.farama.org, 2023.

[8] Johannes Fürnkranz, Eyke Hüllermeier, Weiwei Cheng, and Sang-Hyeun Park, ‘Preference-based reinforcement learning: a formal framework and a policy iteration algorithm’, *Machine learning*, **89**, 123–156, (2012).

[9] Spanish Government, ‘Strategic project for economic recovery and transformation of digitalization of the water cycle. report 2022’, Technical report, Ministry for the Ecological Transition and Demographic Challenge, (2022).

[10] Ting Guo, Yuyu Yuan, and Pengqian Zhao, ‘Admission-based reinforcement-learning algorithm in sequential social dilemmas’, *Applied Sciences*, **13**(3), (2023).

[11] Hado Hasselt, ‘Double q-learning’, *Advances in neural information processing systems*, **23**, (2010).

[12] Andrés Holgado-Sánchez, Joaquín Arias, Mar Moreno-Rebato, and Sascha Ossowski, ‘On admissible behaviours for goal-oriented decision-making of value-aware agents’, in *Multi-Agent Systems*, eds., Vadim Malvone and Aniello Murano, pp. 415–424, Cham, (2023). Springer Nature Switzerland.

[13] Gabriel Kalweit, Maria Huegle, Moritz Werling, and Joschka Boedecker. Deep constrained q-learning, 2020.

[14] Roger Lera-Leri, Filippo Bistaffa, Marc Serramia, Maite Lopez-Sanchez, and Juan Rodriguez-Aguilar, ‘Towards pluralistic value alignment: Aggregating value systems through lp-regression’, in *Proceedings of the 21st International Conference on Autonomous Agents and Multiagent Systems*, AAMAS ’22, p. 780–788, Richland, SC, (2022). International Foundation for Autonomous Agents and Multiagent Systems.

[15] Nieves Montes, Nardine Osman, Carles Sierra, and Marija Slavkovik, ‘Value engineering for autonomous agents’, *CoRR*, abs/2302.08759, (2023).

[16] Nieves Montes and Carles Sierra, ‘Synthesis and properties of optimally value-aligned normative systems’, *Journal of Artificial Intelligence Research*, **74**, 1739–1774, (2022).

[17] Hervé Moulin, *Fair division and collective welfare*, MIT press, 2004.

[18] Andrew Y Ng and Stuart J Russell, ‘Algorithms for inverse reinforcement learning’, in *Proceedings of the Seventeenth International Conference on Machine Learning*, pp. 663–670, (2000).

[19] Nardine Osman and Mark d’Inverno. A computational framework of human values for ethical ai, 2023.

[20] Antoni Perello-Moragues, Manel Poch, David Sauri, Lucia Alexandra Popartan, and Pablo Noriega, ‘Modelling domestic water use in metropolitan areas using socio-cognitive agents’, *Water*, **13**(8), (2021).

[21] L Plata-Pérez, Joss Sánchez-Pérez, and F Sánchez-Sánchez, ‘An elementary characterization of the gini index’, *Mathematical Social Sciences*, **74**, 79–83, (2015).

[22] Manel Rodriguez-Soto, Marc Serramia, Maite Lopez-Sanchez, and Juan Antonio Rodriguez-Aguilar, ‘Instilling moral value alignment by means of multi-objective reinforcement learning’, *Ethics and Information Technology*, **24**, 9, (3 2022).

[23] Shalom H Schwartz, ‘An overview of the schwartz theory of basic values’, *Online readings in Psychology and Culture*, **2**(1), 11, (2012).

[24] Carles Sierra, Nardine Osman, Pablo Noriega, Jordi Sabater-Mir, and Antoni Perelló, ‘Value alignment: a formal approach’, *CoRR*, abs/2110.09240, (2021).

[25] Richard S Sutton and Andrew G Barto, *Reinforcement learning: An introduction*, MIT press, 2018.

[26] Chen Tessler, Daniel J. Mankowitz, and Shie Mannor. Reward constrained policy optimization, 2018.

[27] T. L. van der Weide, F. Dignum, J. J. Ch. Meyer, H. Prakken, and G. A. W. Vreeswijk, ‘Practical reasoning using values’, in *Argumentation in Multi-Agent Systems*, eds., Peter McBurney, Iyad Rahwan, Simon Parsons, and Nicolas Maudet, pp. 79–93, Berlin, Heidelberg, (2010). Springer Berlin Heidelberg.