# On the properties of monotone functionals for generalizing Bellman Principle

Auguste Caen, Jean-Denis Mathias

► To cite this version:

Auguste Caen, Jean-Denis Mathias. On the properties of monotone functionals for generalizing Bellman Principle. Automatica, 2025, 179, pp.112418. 10.1016/j.automatica.2025.112418. hal-05136034

HAL Id: hal-05136034

https://hal.inrae.fr/hal-05136034v1

Submitted on 30 Jun 2025

HAL is a multi-disciplinary open access
archive for the deposit and dissemination of sci-
entific research documents, whether they are pub-
lished or not. The documents may come from
teaching and research institutions in France or
abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est
destinée au dépôt et à la diffusion de documents
scientifiques de niveau recherche, publiés ou non,
émanant des établissements d'enseignement et de
recherche français ou étrangers, des laboratoires
publics ou privés.

<figure><img src="image_1.png" /></figure>

Distributed under a Creative Commons Attribution 4.0 International License
Brief paper

# On the properties of monotone functionals for generalizing Bellman Principle ☆

<figure><img src="image_2.png" /></figure>

Check for
updates

Auguste Caen a,*, Jean-Denis Mathias b

## ARTICLE INFO

**Article history:**
Received 30 May 2023
Received in revised form 28 January 2025
Accepted 6 May 2025

**Keywords:**
Dynamic programming
Optimal control
Monotone functionals

## ABSTRACT

Dynamic programming is classically used for solving optimal control problems. This approach is based on equating the value function of the problem under study as well as the Bellman Principle. However, despite a sharing common framework, each optimal control problem is addressed independently, depending on the characteristics of the cost function. The objective of this article is to establish a unified approach within the context of deterministic, autonomous dynamics with an infinite horizon. The ultimate purpose is to determine conditions from which an optimal control problem admits a *Bellman principle*. To achieve this, we introduce a typology composed of three so-called *monotone functionals*, which encompass typical cost functions met in the literature. Any functional that complies with the properties of one of this class can be used for tackling optimal control problems through Bellman principle. Then, we show that classical functionals used in the literature can be classified in one these classes. Finally, we show how to build a functional according to this typology through a simple problem of optimization.

© 2025 The Author(s). Published by Elsevier Ltd. This is an open access article under the CC BY license

(http://creativecommons.org/licenses/by/4.0/).

## 1. Introduction

Finding an optimal control of a dynamical controlled system with an associated cost is a classical challenge in many applications, such as economics (Rust, 2019), vehicle trajectory control (Meng, Wang, Yang, & Xie, 2019) or health (Bliman & Rapaport, 2023; Shen et al., 2021). The function that associates this minimum cost with an initial condition is called the *value function*. Dynamic programming is a method for solving optimal control problems, by jointly determining the value function and the “optimal” control associated with each initial condition. Dynamic programming was initially applied to a specific type of cost function, defined as the cumulative cost over time of a trajectory (Bellman, 1954). The principle of dynamic programming then consists of expressing the value function as the solution to an equation induced by a property known as the “Bellman principle”. Later on, other optimal control problems, corresponding to different types of cost functions, emerged according to different optimization problems (Barron & Ishii, 1989; Coquelin, Martin, & Munos, 2007; Lygeros, 2004). Dynamic programming has been

adapted to some of these other problems, providing a property, also referred to as Bellman principle by analogy, which allows for the formulation of equations for the corresponding value functions. However, despite of a common vocabulary and approach across these different problems, there is no unified theory of dynamic programming. Each problem is treated independently, and it remains unclear whether these problems correspond to specific cases of a more general problem. In the literature, the introduction to Flemming's book (Fleming & Soner, 2006) considers unification by introducing an operator common to all optimal control problems, but this does not allow the different optimal control problems to be treated in a single move. In Caen (2019, section 4), a more comprehensive theory is proposed which focuses specifically on the link between dynamic programming and viability theory. Independently of this work, Jones and Peet (2021) have later proposed to unify dynamic programming in discrete time. This method is not applicable to continuous time, which requires dealing with notions of continuity which are not necessary for discrete time problems.

From a methodological standpoint, providing a unified theory of dynamic programming would enable the sharing of methodologies across potentially multiple optimal control problems rather than specific problems, as is currently done. In terms of dynamical modeling, this would allow modelers to design cost notions with the knowledge of whether or not the associated problem resolution can be done with dynamic programming.

<footnote>a Université Paris-Saclay, INRAE, MalAGE, 78350, Jouy-en-Josas, France</footnote>

<footnote>b Université Clermont Auvergne, INRAE, UR LISC, centre de Clermont-Ferrand, 9 avenue Blaise Pascal - CS 20085 Aubière, 63178, France</footnote>

<footnote>☆ The material in this paper was not presented at any conference. This paper was recommended for publication in revised form by Associate Editor Bin Zhou under the direction of Editor Florian Dorfler.</footnote>

<footnote>* Corresponding author.</footnote>

<footnote>E-mail addresses: auguste.caen@inrae.fr (A. Caen), jean-denis.mathias@inrae.fr (J.-D. Mathias).</footnote>
In this article, we introduce the optimal control problems in Section 2. We characterize optimal control problems that are amenable to dynamic programming, by proposing a general definition of this *Bellman principle* (Section 3). From this result, we demonstrate a set of properties of the value function in Section 4: properties of regularity and characterization of the value function among the set of solutions of the equation induced by the Bellman principle. We show that functionals used in the literature all exhibit these properties. Finally, we consider a simple optimization problem in order to show how these properties can be used for building up a functional enabling to find optimal control through Bellman principle.

## 2. Notation and definitions

We introduce here the framework and the notations used in the rest of this article. First, we consider an autonomous and continuous time control system:

$$ \dot{y} = f(y, u) \quad (1) $$

with $y \in \mathbb{R}^n$, $u \in \mathbf{U} \subset \mathbb{R}^m$, and $f(., .) : \mathbb{R}^n \times \mathbf{U} \to \mathbb{R}^n$ a function. Let $\mathcal{U}$ denote the set of Lebesgue measurable functions from $\mathbb{R}^+$ to $\mathbf{U}$. We assume here that $f$ is such that, for any initial condition $x \in \mathbb{R}$ and control $u(t) \in \mathcal{U}$, there exists a unique absolutely continuous solution $\phi(x, u(t), t) : \mathbb{R}^+ \to \mathbb{R}^n$ to the system (1) such that $\phi(x, u(t), 0) = x$. For example, one could consider that:

1. $f$ is global Lipschitz continuous in the state variable, uniformly in the control variable: $\exists k_f > 0$, $\forall x_1, x_2 \in \mathbb{R}^{n^2}$, $\forall v \in \mathbf{U}$:

$$ \|f(x_1, v) - f(x_2, v)\| \le k_f \|x_1 - x_2\| $$

2. $\exists \alpha > 0$ such that $\forall (x, u) \in \mathbb{R}^n \times \mathbf{U}$, $\|f(x, u)\| \le \alpha(1 + \|x\|)$

For any initial condition $x \in \mathbb{R}^n$, for any control $u(\cdot) \in \mathcal{U}$, and for any time $t \ge 0$, we denote $\psi(x, u(\cdot), t) = [\phi(x, u(\cdot), t), u(t)]$. An optimal control problem considers a differential system as described by Eq. (1), and a functional $\mathcal{N}$ called cost function defined on the set $\mathcal{D} = \mathcal{A} \times \mathcal{U}$, where $\mathcal{A} \subset (\mathbb{R}^+ \to \mathbb{R}^n)$ is a set that contains at least the absolutely continuous trajectories of $\mathbb{R}^+ \to \mathbb{R}^n$. The optimal control problem then involves determining a trajectory that is a solution to the controlled dynamic system and minimizes the cost function. Dynamic programming refers to a set of methods for solving such problems, by determining the value function $V(x)$, defined as the function that associates the lowest cost on the control space with any condition:

$$ V(x) = \inf_{u(\cdot) \in \mathcal{U}} \mathcal{N}(\psi(x, u(\cdot), \cdot)) $$

These methods involve expressing the value function $V(x)$ as the solution to an equation induced by the Bellman principle. A classic example of a cost function is the following, commonly encountered in economics (Rust, 2019):

$$ \mathcal{N}(y(\cdot)) = \int_{0}^{+\infty} l(y(t))e^{-\lambda t} dt $$

$\forall y(\cdot) \in \mathcal{D}$, with $l: \mathbb{R}^n \times \mathbf{U} \to \mathbb{R}$ a lipschitzian bounded function, and $\lambda \ge 0$. In this case, the Bellman principle refers to the following equality:

$$ \begin{aligned} V(x) &= \inf_{u(\cdot) \in \mathcal{U}} \mathcal{N}(\psi(x, u(\cdot), \cdot)) \\ &= \inf_{u(\cdot) \in \mathcal{U}} \int_{0}^{T} l(\psi(x, u(\cdot), t))e^{-\lambda t} dt + e^{-\lambda T} V(\phi(x, u(\cdot), T)) \end{aligned} $$

This means that the value function is among the functions $W: \mathbb{R}^n \to \mathbb{R}$ that solve the induced equation, which we will refer to here as the “Bellman equation”:

$$ W(x) = \inf_{u(\cdot) \in \mathcal{U}} \int_{0}^{T} l(\psi(x, u(\cdot), t))e^{-\lambda t} dt + e^{-\lambda T} W(\phi(x, u(\cdot), T)) $$

Here the Bellman principle is a consequence of the fact that the functional $\mathcal{N}$ admits the following development:

$$ \mathcal{N}(y(\cdot)) = \int_{0}^{T} l(y(t))e^{-\lambda t} dt + e^{-\lambda T} \mathcal{N}(t \to y(t+T)) $$

$\forall T \ge 0$ and $y(\cdot) \in \mathcal{D}$, which means the functional $\mathcal{N}$ only depends on a part of the trajectory $y(\cdot)$ – on $[0, T]$ – and on the value of the functional at the rest of the trajectory $\mathcal{N}(t \to y(t+T))$, where $t \to y(t+T)$ represents the trajectory of $y(\cdot)$ for the initial time $T$ – we provide an illustration of this notation in Fig. 1. However, as explained in the introduction, to our knowledge, there is no unified theory of dynamic programming. In this sense, there is no single general Bellman principle, but rather as many as there are optimal control problems. For example, when the cost function is defined as follows, as it is the case in certain models of vehicle trajectories (Lygeros, 2004):

$$ \mathcal{N}(y(\cdot)) = \sup_{t \in [0, +\infty[} l(y(t)) $$

$\forall y(\cdot) \in \mathcal{D}$, with $l: \mathbb{R}^n \times \mathbf{U} \to \mathbb{R}$ a bounded function, the Bellman principle refers to:

$$ \begin{aligned} V(x) &= \inf_{u(\cdot) \in \mathcal{U}} \mathcal{N}(\psi(x, u(\cdot), \cdot)) \\ &= \inf_{u(\cdot) \in \mathcal{U}} \max \left( \sup_{t \in [0, T[} l(\psi(x, u(\cdot), t)), V(\phi(x, u(\cdot), T)) \right) \end{aligned} $$

In both cases, it is observed that the objective is to express the value function in $x$ as dependent on a portion of the trajectory optimized over $[0, T]$, and on the value function at $\phi(x, u(\cdot), T)$. However, these dependencies differ in their form, and at this stage, we do not know which common property among these various optimal control problems allows such an equation to be formulated, or even if such a common property exists. Moreover, certain cost functions are not addressed within the framework of dynamic programming, such as, for example, the sum of the two previous functionals: $\mathcal{N}(y(\cdot)) = \sup_{t \in [0, +\infty[} l(y(t)) + \int_{0}^{+\infty} l(y(t))e^{-\lambda t} dt$.

The objective of the next section is thus to provide a unified theory of dynamic programming, that is, to formulate a general Bellman principle of which the examples encountered in the literature are specific instances.

## 3. The Bellman principle

In this section, we aim to unify dynamic programming by formulating a general Bellman principle, such that every dynamic programming problem encountered in the literature is a specific case of it. In the first subsection, we demonstrate that for any cost function, it is possible to formulate two inequalities for which the value function is a solution. These inequalities allow us, Section 3.2, to formulate the generalized Bellman principle.

### 3.1. Lower and upper bounds of the value function

We focus on a functional $\mathcal{N}$ defined on $\mathcal{D} = \mathcal{A} \times \mathcal{U}$, with $\mathcal{A}$ is a subset of $\mathbb{R}^+ \to \mathbb{R}^n$. We note $\mathbf{B}$ the image on $\mathcal{N}$ in $\overline{\mathbb{R}} = \mathbb{R} \cup \{-\infty, +\infty\}$. In addition, we note $m = \inf \mathbf{B}$ and $M = \sup \mathbf{B}$, and we assume $m, M \in \mathbf{B}$. We assume the following properties on the definition the set of trajectories $\mathcal{A} \subset (\mathbb{R}^+ \to \mathbb{R}^n)$:
<figure><img src="image_3.png" /><figcaption>**Fig. 1.** These figures illustrate the notation $\mathcal{N}(t \to y(t+T))$ for $N_1(y(\cdot)) = \sup_{\epsilon \ge 0} l(y(t))$ (a) and $N_2(y(\cdot)) = l(y(\tau))$ (d), with $l: \mathbb{R}^n \times \mathbf{U} \to [m, M] \subset \mathbb{R}$ an arbitrary function and $\tau > 0$. For each of these functionals, the sets $\mathcal{E}_{+,-}$ are illustrated in figures (b) and (e), and the functions objects $T \to F_{N_{1,2,+,-}}(T, y(\cdot), N_{1,2}(t \to y(t+T)))$ are depicted in figures (c) and (f).</figcaption></figure>

1. **A** contains the absolute continuous functions of $\mathbb{R}^+ \to \mathbb{R}^n$

2. $\forall z(\cdot) \in \mathcal{A}, \forall T \ge 0: t \to z(t+T) \in \mathcal{A}$

3. $\forall z_1(\cdot), z_2(\cdot) \in \mathcal{A}, \forall T \ge 0:$

$$g: t \to \begin{cases} z_1(t) & \text{if } t < T \\ z_2(t-T) & \text{if } t \ge T \end{cases} \in \mathcal{A}$$

These properties mean that **A** contains the solutions of systems as defined by Eq. (1). Moreover, they allow us in Section 3.2 to construct an intuitive definition of the monotonicity of a functional. In particular, we can then define two bounding functions for a given functional $\mathcal{N}$:

**Definition 1.** For a functional $\mathcal{N}, \forall T \ge 0, \forall y(\cdot) \in \mathcal{D}, \forall \eta \in [m, M]$, let be $\mathcal{E}_{\mathcal{N},+}(T, y(\cdot), \eta) \subset \mathcal{D}$ and $\mathcal{E}_{\mathcal{N},-}(T, y(\cdot), \eta) \subset \mathcal{D}$ such that:

$$\begin{align*} \mathcal{E}_{\mathcal{N},+}(T, y(\cdot), \eta) &= \left\{ \begin{array}{ll} z(\cdot) \in \mathcal{D} & | z(t) = y(t), \forall t \in [0, T[ \\ \mathcal{N}(t \to z(t+T)) \le \eta & \end{array} \right\} \\ \mathcal{E}_{\mathcal{N},-}(T, y(\cdot), \eta) &= \left\{ \begin{array}{ll} z(\cdot) \in \mathcal{D} & | z(t) = y(t), \forall t \in [0, T[ \\ \mathcal{N}(t \to z(t+T)) \ge \eta & \end{array} \right\} \quad \text{and} \end{align*}$$

let be $F_{\mathcal{N},+}(\cdot, \cdot, \cdot) : \mathbb{R}^+ \times \mathcal{D} \times [m, M] \to \mathbb{R}$ and $F_{\mathcal{N},-}(\cdot, \cdot, \cdot) : \mathbb{R}^+ \times \mathcal{D} \times [m, M] \to \mathbb{R}$ such that, $\forall T \ge 0, \forall y(\cdot) \in \mathcal{D}, \forall \eta \in [m, M]$:

$$\text{if } T > 0: F_{\mathcal{N},+}(T, y(\cdot), \eta) = \sup_{z(\cdot) \in \mathcal{E}_{\mathcal{N},+}(T, y(\cdot), \eta)} \mathcal{N}(z(\cdot))$$

$$\text{else: } F_{\mathcal{N},+}(T, y(\cdot), \eta) = \limsup_{\substack{T \to 0 \\ T \ge 0}} \sup_{z(\cdot) \in \mathcal{E}_{\mathcal{N},+}(T, y(\cdot), \eta)} \mathcal{N}(z(\cdot))$$

$$\text{if } T > 0: F_{\mathcal{N},-}(T, y(\cdot), \eta) = \inf_{z(\cdot) \in \mathcal{E}_{\mathcal{N},-}(T, y(\cdot), \eta)} \mathcal{N}(z(\cdot))$$

$$\text{else: } F_{\mathcal{N},-}(T, y(\cdot), \eta) = \liminf_{\substack{T \to 0 \\ T \ge 0}} \inf_{z(\cdot) \in \mathcal{E}_{\mathcal{N},-}(T, y(\cdot), \eta)} \mathcal{N}(z(\cdot))$$

We note that $\mathcal{E}_{\mathcal{N},\pm}(T, y(\cdot), \eta)$ and $F_{\mathcal{N},\pm}(T, y(\cdot), \eta)$ do not depend on $y(\cdot)$, but only on its restriction on $[0, T]$. These objects are illustrated in Fig. 2. We highlight two points:

**Remark 2.** for any functional $\mathcal{N}, \forall T \ge 0, \forall y(\cdot) \in \mathcal{D}$, we have the following inequalities:

$$F_{\mathcal{N},-}(T, y(\cdot), \mathcal{N}(t \to y(t+T))) \le \mathcal{N}(y(\cdot))$$

$$\mathcal{N}(y(\cdot)) \le F_{\mathcal{N},+}(T, y(\cdot), \mathcal{N}(t \to y(t+T)))$$

It relies on the fact that $\forall y(\cdot) \in \mathcal{D}, y(\cdot)$ belongs to the intersection of $\mathcal{E}_{\mathcal{N},+}(T, y(\cdot), \mathcal{N}(t \to y(t+T)))$ and

$$\mathcal{E}_{\mathcal{N},-}(T, y(\cdot), \mathcal{N}(t \to y(t+T)))$$

**Remark 3.** $\forall T \ge 0, y(\cdot) \in \mathcal{D}$, the functions $\eta \to F_{\mathcal{N},-}(T, y(\cdot), \eta)$ and $\eta \to F_{\mathcal{N},+}(T, y(\cdot), \eta)$ are increasing. This property relies on the fact that the sets $\mathcal{E}_{\mathcal{N},+}(T, y(\cdot), \eta)$ and $\mathcal{E}_{\mathcal{N},-}(T, y(\cdot), \eta)$ are respectively increasing and decreasing with respect to inclusion, depending on $\eta$.

We can now introduce the first result on the value function $V(x) = \inf_{u(\cdot) \in \mathcal{U}} \mathcal{N}(\psi(x, u(\cdot), \cdot))$, where $\psi(x, u(\cdot), \cdot) = t \to [\phi(x, u(\cdot), t), u(t)]$:

**Lemma 4.** $\forall T \ge 0, \forall x \in \mathbb{R}^n$:

$$V(x) \ge \inf_{u(\cdot) \in \mathcal{U}} F_{\mathcal{N},-}(T, \psi(x, u(\cdot), \cdot), V(\phi(x, u(\cdot), T)))$$

In addition, if $\forall y(\cdot) \in \mathcal{D}$, the function $\eta \to F_{\mathcal{N},+}(T, y(\cdot), \eta)$ is upper semi-continuous, then:

$$V(x) \le \inf_{u(\cdot) \in \mathcal{U}} F_{\mathcal{N},+}(T, \psi(x, u(\cdot), \cdot), V(\phi(x, u(\cdot), T)))$$

This lemma is proved in Appendix A. It provides the sufficient conditions for a value function in $x$ to be bounded by two functions dependent on a portion of the trajectory optimized over $[0, T]$ and on the value function at $\phi(x, u(\cdot), T)$. This precisely corresponds to the common form encountered in the context of dynamic programming, grouped under the term "Bellman principle". Therefore having the equality $F_{\mathcal{N},+} = F_{\mathcal{N},-}$ is the key-property for that will be used for building functionals in the case of Bellman principle. For this purpose, we will show in Section 3.2 how this property can be mobilized in the case of Bellman principle. In Section 4, we will deduce properties of value functions based on this equality. Finally, we illustrate this contribution with a simple toy model in Section 5.

## 3.2. The general Bellman principle

Let us consider monotone functionals. We show that this property of monotonicity is essential to build the general Bellman principle. This Bellman principle is then based on the previously defined functions $F_{\mathcal{N},+}$ and $F_{\mathcal{N},-}$.
**Definition 5.** The functional $\mathcal{N}$ is said monotone if $\forall y_1(\cdot), y_2(\cdot) \in \mathcal{D}^2, \forall T > 0$:

$$ 
\left\{ 
\begin{array}{l}
(1) \quad y_1(t) = y_2(t), \forall t \in [0, T[ \\
(2) \quad \mathcal{N}(t \to y_1(t+T)) \le \mathcal{N}(t \to y_2(t+T)) \\
\Rightarrow \mathcal{N}(y_1(\cdot)) \le \mathcal{N}(y_2(\cdot))
\end{array}
\right. 
$$

Monotone functionals are characterized as follows:

**Lemma 6.** Let $\mathcal{N}: \mathcal{D} \to \bar{\mathbb{R}}$. The three following assertions are equivalent:

1. $\mathcal{N}$ is monotone

2. $\forall T \ge 0, \forall y(\cdot) \in \mathcal{D}$:

$$ 
\begin{aligned}
\mathcal{N}(y(\cdot)) &= F_{\mathcal{N},-}(T, y(\cdot), \mathcal{N}(t \to y(t+T))) \\
&= F_{\mathcal{N},+}(T, y(\cdot), \mathcal{N}(t \to y(t+T)))
\end{aligned}
$$

3. $\exists F(\cdot, \cdot, \cdot) : \mathbb{R}^+ \times \mathcal{D} \times [m, M]$ s.t. $\forall y(\cdot) \in \mathcal{D}, \forall T \ge 0$:

$$ 
\begin{aligned}
F(T, y(\cdot), \mathcal{N}(t \to y(t+T))) &= \mathcal{N}(y(\cdot)) \\
\eta &\to F(T, y(\cdot), \eta) \text{ is increasing}
\end{aligned}
$$

where $F(T, y(\cdot), \eta)$ only depends on the restriction of $y(\cdot)$ to $[0, T]$. We call such a function $F$ a **Bellman form** of the functional $\mathcal{N}$

The proof of this lemma is shown in Appendix B

**Remark 7.** The Bellman forms of a monotone functional are equal on $\mathbb{R}^+ \times \mathcal{D} \times \mathbf{B}$.

**Remark 8.** We could directly define the monotonicity of a functional as corresponding to the third assertion of the previous lemma. This would make it possible to relax the assumptions on $\mathcal{A}$: it suffices for $\mathcal{A}$ to contain only absolutely continuous trajectories. However, supposing that $\mathcal{A}$ also contains piecewise absolutely continuous trajectories ensures that the sets $\mathcal{E}_+$ and $\mathcal{E}_-$ are not empty, and thus the existence of $F_{\mathcal{N},+}$ and $F_{\mathcal{N},-}$. This provides a link between the definition of a monotone functional and the third assertion of the preceding lemma. In addition, this definition is an effective tool for ensuring that a functional is not monotone: It suffices to find two trajectories $y_1(\cdot)$ and $y_2(\cdot)$ of $\mathcal{D}$ that satisfy properties (1) and (2) of Definition 5 and for which $\mathcal{N}(y_1(\cdot)) > \mathcal{N}(y_2(\cdot))$.

We now present the general Bellman principle, which summarizes the previous results of this section:

**Theorem 9 (The General Bellman Principle).** Let $\mathcal{N}$ be a monotone functional and $F$ its Bellman form, such that the function $\eta \to F(T, y(\cdot), \eta)$ is upper semi-continuous. $\forall x \in \mathbb{R}^n$, the value function satisfies the general **Bellman principle**:

$$ 
\begin{aligned}
V(x) &= \inf_{u(\cdot) \in \mathcal{U}} \mathcal{N}(\psi(x, u(\cdot), \cdot)) \\
&= \inf_{u(\cdot) \in \mathcal{U}} F(T, \psi(x, u(\cdot), \cdot), V(\phi(x, u(\cdot), T)))
\end{aligned}
$$

The proof of this theorem is similar to that of Lemma 4. This theorem generalizes the notion of the Bellman principle to the entire class of monotone functionals. Thus, the optimal control problems encountered in the literature and treated within the framework of dynamic programming are all specific instances of this general theorem. Table 1 presents several cost functions encountered in the literature along with their associated Bellman forms. This initial result highlights the contribution of studying monotone functionals to the unification of dynamic programming. Indeed, this theorem provides valuable information to the modeler of controlled dynamic systems, as it allows anticipation

of whether dynamic programming is compatible with a given cost function. This determination is based on the properties of monotonicity and regularity of the associated Bellman form. In Section 5, we present an example of a functional not encountered in the literature, and for which this result ensures eligibility for dynamic programming. In this general formulation of the Bellman principle, information regarding the value function is limited, including aspects such as the uniqueness of solutions for the equation outlined by the Bellman principle and the regularity of the value function. Nevertheless, this information is crucial for determination of the value function. In the subsequent analysis, we delve into these properties.

## 4. Properties of the value functions

Here we highlight another contribution of studying monotone functionals to the unification of dynamic programming. Indeed, this approach allows us to unify the proofs of various properties of the associated value functions. We first introduce the following definition:

**Definition 10.** Let $\mathcal{N}$ be a monotone functional satisfying the hypotheses of Theorem 9, and $F$ be its Bellman form.

* $\forall \eta \in [m, M], \mathcal{N}$ is said $\eta$-convergent if, $\forall y(\cdot) \in \mathcal{D}$:

$$ \lim_{T \to +\infty} F(T, y(\cdot), \eta) = \mathcal{N}(y(\cdot)) $$

* $\mathcal{N}$ is said discounted if $\mathbf{B} \subset \mathbb{R}$ and there exists $\omega(\cdot, \cdot) : \mathbb{R}^+ \times \mathbb{R}^+ \to \mathbb{R}^+$, decreasing over its second variable, and such that:

$$ 
\begin{aligned}
&\forall \rho \in \mathbb{R}^+ \quad : \lim_{T \to +\infty} \omega(T, \rho) = 0 \text{ and} \\
&\forall y(\cdot) \in \mathcal{D}, \forall (\eta_1, \eta_2) \in [m, M]^2 : \\
&\qquad |F(T, y(\cdot), \eta_1) - F(T, y(\cdot), \eta_2)| < \omega(T, |\eta_1 - \eta_2|)
\end{aligned}
$$

This definition allows us to introduce three classes of functionals that we focus on in this section: M-convergent functionals, m-convergent functionals, and discounted functionals. These classes are among the most studied. Various examples are provided in Table 1, along with examples of functionals that do not belong to these classes.

**Remark 11.** Discounted functionals have the particularity of being $\eta$-convergent for any $\eta \in [m, M]$ (Appendix C): this allows us to associate them with the properties of the other two classes discussed here. Here, we impose that discounted functionals are bounded, denoted by $\mathbf{B} \subset \mathbb{R}$ on $\mathcal{D}$, as is the case, for example, when $\lambda > 0$ for $\mathcal{N}(y(\cdot)) = \int_0^{+\infty} l(y(t))e^{-\lambda t} dt$ with $l$ bounded. When this functional is undiscounted, i.e., $\lambda \le 0$, it is only 0-convergent. However, in this section, we do not address the case of functionals that are neither m-convergent nor M-convergent. Thus, when $\lambda \le 0$, the results discussed here apply only to optimal control problems where $m = 0$ or $M = 0$. This is the case, for example, for LQR-type problems, for which:

$$ \mathcal{N}([x(\cdot), u(\cdot)]) = \int_{0}^{+\infty} x^T(t)Qx(t) + u^T(t)Ru(t)dt $$

where $Q$ and $R$ are symmetric positive-definite matrices. In this case, for any $\xi, \mu \in \mathbb{R}^n \times \mathbf{U}$, $\xi^T Q \xi + \mu^T R \mu \ge 0$, and thus $\mathcal{N}$ is $(m=0)$-convergent.

Thanks to these convergence properties, we demonstrate several particularly relevant properties in the context of optimal control:

* **Convergence of finite horizon value functions to infinite horizon value functions:** Although, in this article, we focus on infinite horizon problems, it can be useful for
their resolution to consider finite horizon problems, whose value functions converge to the value function of the infinite horizon problem. We illustrate this approach in Section 5. The following lemma (proved in Appendix D) shows that the value function can be approximated by that of a finite-horizon optimal control problem:

$$ \lim_{T \to +\infty} \inf_{u(\cdot) \in \mathcal{U}} F(T, \psi(x, u(\cdot), \cdot), \eta) = V(x) $$

with $\eta = M$ if $\mathcal{N}$ is M-convergent, $\eta = m$ if $\mathcal{N}$ is m-convergent, and $\eta \in [m, M]$ if $\mathcal{N}$ is discounted:

**Lemma 12.** For a monotone functional $\mathcal{N}$, noting $F$ its Bellman form, and for a differential system as defined in Eq. (1):

* If $\mathcal{N}$ is M-convergent, then $\forall x \in \mathbb{R}^n$:

$$ \lim_{T \to +\infty} \inf_{u(\cdot) \in \mathcal{U}} F(T, \psi(x, u(\cdot), \cdot), M) = V(x) $$

* If $\mathcal{N}$ is m-convergent, if $\forall x \in \mathbb{R}^n$, the set $\{\psi(x, u(\cdot), \cdot)|u(\cdot) \in \mathcal{U}\}$ is compact for the pointwise convergence, and $\forall T \ge 0$, $\forall y_n(\cdot)$ a sequence of functions in $\mathcal{D}$ pointwise converging to $y(\cdot)$ on $[0, T]$:

$$ \lim_{n \to +\infty} \inf_{T \to +\infty} F(T, y_n(\cdot), m) \ge F(T, y(\cdot), m) $$

then $\forall x \in \mathbb{R}^n$, $\exists v(\cdot) \in \mathcal{U}$ such that:

$$ \lim_{T \to +\infty} \inf_{u(\cdot) \in \mathcal{U}} F(T, \psi(x, u(\cdot), \cdot), m) = V(x) \\ = \mathcal{N}(\psi(x, v(\cdot), \cdot)) $$

* If $\mathcal{N}$ is discounted, then $\forall x \in \mathbb{R}^n$, $\forall \eta \in [m, M]$:

$$ \lim_{T \to +\infty} \inf_{u(\cdot) \in \mathcal{U}} F(T, \psi(x, u(\cdot), \cdot), \eta) = V(x) $$

* **Characterization of the value function as the solution of the equation corresponding to the Bellman principle:** To determine the value function, another approach is to characterize it as a solution of the Bellman equation. However, the Bellman equation may admit several solutions. The issue here is therefore to characterize the value function sought among these solutions. We show with Lemma 13 (proved in Appendix E) that the value function corresponding to a M-convergent functional lower bounds any lower solution of the Bellman equation. Conversely, the value function corresponding to m-convergent functional upper bounds any upper solution of the Bellman equation. Finally, an immediate consequence of this lemma is that the value function corresponding to a discounted functional is the unique solution of the associated Bellman equation. In Section 5, we illustrate how this property allows us to determine such value functions.

**Lemma 13.** For a monotone $\mathcal{N}$, noting $F$ its Bellman form, and for a differential system as defined in Eq. (1):

* If $\forall x \in \mathbb{R}^n$, $\lim_{T \to +\infty} \inf_{u(\cdot) \in \mathcal{U}} F(T, \psi(x, u(\cdot), \cdot), M) = V(x)$, then $\forall W : \mathbb{R}^n \to [m, M], \forall x \in \mathbb{R}^n$:

$$ \left\{ \begin{array}{l} \forall T \ge 0 : \\ W(x) \le \inf_{u(\cdot) \in \mathcal{U}} F(T, \psi(x, u(\cdot), \cdot), W(\phi(x, u(\cdot), T))) \\ \Rightarrow W(x) \le V(x) \end{array} \right. $$

* If $\forall x \in \mathbb{R}^n$, $\lim_{T \to +\infty} \inf_{u(\cdot) \in \mathcal{U}} F(T, \psi(x, u(\cdot), \cdot), m) = V(x)$, then $\forall W : \mathbb{R}^n \to [m, M], \forall x \in \mathbb{R}^n$:

$$ \left\{ \begin{array}{l} \forall T \ge 0 : \\ W(x) \ge \inf_{u(\cdot) \in \mathcal{U}} F(T, \psi(x, u(\cdot), \cdot), W(\phi(x, u(\cdot), T))) \\ \Rightarrow W(x) \ge V(x) \end{array} \right. $$

* **Regularity of the value function:** The regularity of the value function is an important property, which conditions

the usable algorithms to the numerical resolution in the context of dynamic programming (Barles & Souganidis, 1990). One of the issues here is therefore to determine the regularity properties of the value function. We thus demonstrate Lemma 14 (proved in Appendix F) which establishes the regularity properties of the value functions: upper semi-continuity for M-convergent functionals, lower semi-continuity for m-convergent functionals, and continuity for discounted functionals.

**Lemma 14.** For a monotone functional $\mathcal{N}$, noting $F$ its Bellman form, and for a differential system as defined in Eq. (1):

* If $\forall x \in \mathbb{R}^n : \lim_{T \to +\infty} \inf_{u(\cdot) \in \mathcal{U}} F(T, \psi(x, u(\cdot), \cdot), M) = V(x)$ and $\forall T \ge 0, x \to F(T, \psi(x, u(\cdot), \cdot), M)$ is upper continuous uniformly in $u(\cdot)$, then $V(\cdot)$ is upper semi-continuous

* If $\forall x \in \mathbb{R}^n : \lim_{T \to +\infty} \inf_{u(\cdot) \in \mathcal{U}} F(T, \psi(x, u(\cdot), \cdot), m) = V(x)$ and $\forall T \ge 0, x \to F(T, \psi(x, u(\cdot), \cdot), m)$ is lower continuous uniformly in $u(\cdot)$, then $V(\cdot)$ is lower semi-continuous

* If $\mathcal{N}$ is discounted, and $\forall T \ge 0, x \to F(T, \psi(x, u(\cdot), \cdot), m)$ is lower continuous, and $x \to F(T, \psi(x, u(\cdot), \cdot), M)$ is upper continuous, both uniformly in $u(\cdot)$, then $V(\cdot)$ is continuous

By using the Bellman principle, we can study m-convergent, M-convergent and discounted functionals. We show that these properties allow determining the value function, either as the limit of finite-horizon value functions, or as the bound of the set of solutions of the equation induced by the Bellman principle. These classes also allow to study the regularity of the associated value functions. In the next section, we demonstrate how to apply these properties.

## 5. Application to a simple optimal control problem

In this section, we illustrate the results established in this article with a simple toy model. The simplicity of this model allows us to focus on the analysis of cost functions and to illustrate our results. We take on the role of a manager who must establish a control strategy for managing a dynamic system. We demonstrate that, depending on the interpretation of the manager objective, several problem formalisms can be formulated, corresponding to three cost functions, commonly encountered in the literature. We show how the properties established in this article allow us to determine the corresponding value functions and optimal trajectories. We then show how it is possible to build up a new cost function not found in the literature, which combines the previous approaches. All these approaches carry different information: this information is complementary and illustrates the relevance of approaching a system with multiple cost functions.

We consider the following controlled dynamic system:

$$ \dot{y} = u - y $$

With $y \in \mathbb{R}$, and $u(\cdot)$ a control with values in $[-0.5, 1]$. We can then write the analytical solution of this system:

$$ \phi(y_0, u(\cdot), t) = \int_0^t u(t - \tau)e^{-\tau} d\tau + y_0e^{-t} $$

Thus, the system converges within $[-0.5, 1]$ regardless of the control. When the control is constant $u(t) = \mu \forall t \ge 0$, this system converges to $\mu$. The objective of the modeler is to keep the system as far away from zero as possible. This objective is vague: several interpretations may be formulated, which can be formalized by different monotone functionals. In the following, we denote $l(x) = -x^2$, $\forall x \in \mathbb{R}$. In this section, the functionals analyzed are evaluated solely on the state of the system, not on the control.
<caption>**Table 1**</caption>

Examples of monotone functionals; $l$ is a bounded function regular enough of $\mathbb{R}^n \to \mathbb{R}$, $\lambda \in \mathbb{R}$, $\gamma \in ]0, 1[$, $\Gamma_k$ and $\Gamma_k^T$ are defined as follows: $\Gamma_k(y(\cdot)) = \inf\{t \in \mathbb{R}^+ | y(t) \notin \mathbf{K}\}$ ($= +\infty$ if $y(\cdot) \in \mathbf{K}^{\mathbb{R}^+}$) and $\Gamma_k^T(y(\cdot)) = \inf\{t \in [0, T] | y(t) \notin \mathbf{K}\}$ ($= +\infty$ if $y(\cdot) \in \mathbf{K}^{[0,T]}$); $\forall a, b \in \mathbb{R}^2$, $R(a, b) = aif|a| > |b|$, $R(a, b) = bif|a| < |b|$ and $R(a, b) = \max(a, b)\inf|a| = |b|$.

<table><thead><tr><th>Functional</th><th>Bellman form</th><th>Class</th><th>Examples</th></tr></thead><tbody><tr><td>y(.) → sup<sub>t∈R<sup>+</sup></sub> l(y(t))</td><td>(T, y(.) , η) → max { sup<sub>t∈[0,T]</sub> l(y(t)) , η}</td><td>m-convergent</td><td>Barron and Ishii (1989), Lygeros (2003), Molina, Rapaport, and Ramírez (2022)</td></tr><tr><td>y(.) → inf<sub>t∈R<sup>+</sup></sub> l(y(t))</td><td>(T, y(.) , η) → min { inf<sub>t∈[0,T]</sub> l(y(t)) , η}</td><td>M-convergent</td><td>Lygeros (2003)</td></tr><tr><td>y(.) → ∫<sub>0</sub><sup>+∞</sup> l(y(t))e<sup>-λt</sup>dt</td><td>(T, y(.) , η) → ∫<sub>0</sub><sup>T</sup> l(y(t))e<sup>-λt</sup>dt + e<sup>-λT</sup>η</td><td>discounted when λ > 0<br>0-convergent else</td><td>Bellman (1954), Cannarsa, Gaubert, Mendico, and Quincampoix (2023), Feinstein and Luenberger (1981)</td></tr><tr><td>y(.) → γ<sup>T</sup><sub>k</sub>(y(.)</td><td>(T, y(.) , η) → max { γ<sup>T</sup><sub>k</sub>(y(.) , γ<sup>T</sup>η}</td><td>discounted</td><td>Coquelin et al. (2007)</td></tr><tr><td>y(.) → R (∫<sub>t∈R<sup>+</sup></sub> inf<sub>t∈R<sup>+</sup></sub> l(y(t)), sup<sub>t∈R<sup>+</sup></sub> l(y(t)))</td><td>(T, y(.) , η) → R (∫<sub>t∈R<sup>+</sup></sub> sup<sub>t∈T</sub> l(y(t)), inf<sub>t<t</sub> l(y(t))) , η)</td><td>(argmin<sub>η∈[m,M]</sub> |η|)-convergent</td><td>introduced<br>in this article</td></tr><tr><td>y(.) → lim<sub>τ→+∞</sub> sup<sub>τ→+∞</sub> 1/τ ∫<sub>0</sub><sup>T</sup> l(y(t))dt</td><td>(T, y(.) , η) → η</td><td>none</td><td>Maitra (1966)</td></tr></tbody></table>

* **The minmax approach.** A first approach is to interpret the objective cautiously: We want the minimal distance of the system from zero to be as large as possible. A suitable functional to model this goal is $y(.) \to \sup_{t \in \mathbb{R}^+} l(y(t))$, which associates with a trajectory its smallest distance to 0. This functional is monotone and m-convergent. We thus apply the second point of Lemma 13 to analytically determine the value function (Appendix G). In particular, we see that, to be optimal, a trajectory just needs to stay at a distance of $V(x)$ from zero, and does not necessarily need to move as far away from zero as possible (Figs. 2(a) and 2(e)).

* **The minmin approach.** This other approach is more ambitious: we want to move the system as far away as possible from the value 0. The functional that models this objective is $y(.) \to \inf_{t \in \mathbb{R}^+} l(y(t))$. This functional is monotone and M-convergent. Similarly to the previous approach, it is possible to determine the value function using Lemma 13. To be optimal, the trajectories must reach a value greater than or equal to 1. Thus, all optimal trajectories whose initial condition is below 0 pass through 0. Moreover, a trajectory whose initial condition is above 1 and stabilizes at 0 is considered optimal: only its value furthest from zero is considered in determining its optimality (Figs. 2(b) and 2(f)).

* **The Lagrangian approach.** The manager now aims to determine trajectories that remain as far as possible from zero, considering the functional $y(.) \to \int_{0}^{+\infty} l(y(t))e^{-\lambda t} dt$. Minimizing this functional means seeking the largest possible average distance of the system to zero, weighted or not according to the value of $\lambda$. This functional is monotone, and discounted when $\lambda$ is positive. To be optimal, the trajectories must converge toward $-0.5$ or 1, depending on whether their initial condition is below or above a negative value $\xi < 0$. This approach asymptotically keeps the system as far away as possible from the value 0, but allows the system to pass through the value 0 if the rest of the trajectory compensates for this situation (Figs. 2(c) and 2(g)).

* **A new approach.** Here the manager wants a compromise between getting as less as possible to the value 0 and moving as far away as possible, but with a hierarchy of priorities:

it is more important not to be too close to the value 0 than to be as far away from it as possible. In this context, the previous functional is too permissive since it allows the system to reach the value 0. We design a functional that induces trajectories corresponding to these preferences:

$$ 
\begin{aligned}
\mathcal{N}(y(\cdot)) &= \inf_{t \in \mathbb{R}^+} e^{-\lambda t} g(y(t)) \text{ if } g(y(\tau)) < 0 \quad \forall \tau \ge 0 \\
&= \sup_{t \in \mathbb{R}^+} g(y(t)) \text{ else}
\end{aligned}
$$

with g a bounded function, that reaches its maximum value at zero, and is positive only on the interval $[-\alpha, \alpha]$. The objective is preventing the system from approaching a distance less than $\alpha$ from zero, while simultaneously seeking to maximize its distance from zero when beyond this threshold. We show that this functional is monotone and m-convergent. Here again, we show (Appendix G) that it is possible to determine the value function as the smallest subsolution of the Bellman equation. When the initial condition $x$ is in $[-\alpha, \alpha]$, the optimal trajectories never approach within a distance less than $g(x)$ from the value 0. When $x$ is not in $[-\alpha, \alpha]$, the trajectories will never pass through $[-\alpha, \alpha]$, while seeking in the short term to move as far as possible from the point 0 (Figs. 2(d) and 2(h)).

## 6. Conclusion

We propose to approach dynamic programming through monotone functionals, without focusing on a particular functional. This approach allows us to formulate the general Bellman principle, for which every dynamic programming problem encountered in the literature is a particular case. This approach allows us to exhibit regularity and characterization properties of the value function, valid for a wide range of functionals. We illustrate the generality of this approach by applying these results to various classic optimal control problems. This result also serves as a tool for the modeler, as it is then possible to design new types of cost functions specific to control objectives.

The generality of this approach allows us to consider several developments of these results. In this paper, we have focused
<figure><img src="image_4.png" /><figcaption>**Fig. 2.** Illustration of optimal trajectories and value functions for the different approaches listed in Section 5: $y(.) \to \sup_{t \in \mathbb{R}^+} l(y(t))$ (**a** & **c**), $y(.) \to \inf_{t \in \mathbb{R}^+} e^{-\lambda t} g(y(t))$ if $g(y(\tau)) < 0$ $\forall \tau \ge 0$, $= \sup_{t \in \mathbb{R}^+} g(y(t))$ else (**d** & **h**) with $\lambda = 0.2$.</figcaption></figure>

on the problem of finding the value function, which represents the minimal cost associated with any initial condition across the control space. Another common issue in optimal control involves not only determining a state feedback control that minimizes a cost but also stabilizes the system around a specific value (Kumar & Jerome, 2013). The existence of such a state feedback control is not guaranteed and depends on both the dynamic system and the cost function considered (Postoyan, Bušoniu, Nešić, & Daafouz, 2016). A potential direction for future work is to adapt our approach to characterize the dynamic systems and monotone cost functions that admit such solutions, and, where applicable, to determine these state feedback controls.

Furthermore, we specifically focused here on the properties of $m$ and $M$-convergent and discounted functionals. It would be interesting to extend these results to other types of functionals. Another important aspect of dynamic programming that we do not address here is that, usually in the literature, the value function is numerically determined by deriving from the Bellman principle a partial differential equation – called the Hamilton–Jacobi–Bellman equation (HJB) – whose value function is a viscosity solution (Bardi & Capuzzo-Dolcetta, 1997; Lygeros, 2003). Similarly to the Bellman principle, the literature treats each optimal control problem independently and therefore formulates an HJB specific to each optimal control problem. A continuation of our work is therefore to propose a general formulation of the HJB equation, possibly allowing us to extend some existing numerical resolution results (Baumeister, Leitão, & Silva, 2007; Carlson, Haurie, & Leizarowitz, 2012; Lin et al., 2021).

## Acknowledgments

We thank the Région Auvergne, France, for its financial support. This work was supported by a contrat Plan Etat Région from the Région Auvergne, France: CPER 2015–2020 – ConnecSens.

## Appendix A. Proof of Lemma 4

Let $\mathcal{N}$ be a functional and $V$ the corresponding value function. Because $\eta \to F_{\mathcal{N},-}(T, \psi(x, u(.,), .), \eta)$ is decreasing:

$$V(x) \ge \inf_{u(.,) \in \mathcal{U}} F_{\mathcal{N},-}(T, \psi(x, u(.,), .), V(\phi(x, u(.,), T)))$$

If $\forall y(.) \in \mathcal{D}$, $\eta \to F_{\mathcal{N},+}(T, y(.)$, $\eta$ is upper semi-continuous, then $\forall u_1(.) \in \mathcal{U}$, $\forall \epsilon > 0$, $\exists \delta > 0$:

$$F_{\mathcal{N},+}(T, \psi(x, u_1(.,), .), \min(M, V(x_T) + \delta)) \le F_{\mathcal{N},+}(T, \psi(x, u_1(.,), .), V(x_T)) + \epsilon \text{ with } x_T = \phi(x, u_1(.,), T).$$

In addition, $\exists u_2(.)$ such that $\mathcal{N}(\psi(x_T, u_2(.,), .)) \le \min(M, V(x_T) + \delta)$. Thus:

$$V(x) \le F_{\mathcal{N},+}(T, \psi(x, u_1(.,), .), \mathcal{N}(\psi(x_T, u_2(.,), .)))$$

$$\le F_{\mathcal{N},+}(T, \psi(x, u_1(.,), .), V(x_T)) + \epsilon \quad \square$$

$$\Rightarrow V(x) \le \inf_{u(.) \in \mathcal{U}} F_{\mathcal{N},+}(T, \psi(x, u(.,), .), V(\phi(x, u(.,), T)))$$

## Appendix B. Proof of Lemma 6

Let $\mathcal{N}: \mathcal{D} \to \mathbb{R}$. We first assume that $\mathcal{N}$ is monotone. $\forall T > 0$, $\forall y(.) \in \mathcal{D}$:

$$z(.) \in \mathcal{E}_{\mathcal{N},-}(T, y(.) , \eta) \Rightarrow \mathcal{N}(z(.)) \ge \mathcal{N}(y(.)) \text{ thus } F_{\mathcal{N},-}(T, y(.) , \mathcal{N}(t \to y(t+T))) \ge \mathcal{N}(y(.))$$

Similarly, we show that $F_{\mathcal{N},+}(T, y(.) , \mathcal{N}(t \to y(t+T))) \le \mathcal{N}(y(.))$. Thanks to inequalities of Remark 2, we obtain the wanted equality (when $T = 0$, the equality is a direct consequence of the definition of $F_{\mathcal{N},+}$ and $F_{\mathcal{N},-}$). In addition, the fact that the second assertion implies the third is immediate. We now assume that a Bellman form $F$ of $\mathcal{N}$ exists. $\forall y_1(.,) , y_2(.,) \in \mathcal{D}$, $\forall T \ge 0$ such that $\forall t \in [0, T] [y_1(t) = y_2(t)$ and $\mathcal{N}(t \to y_1(t+T)) \le \mathcal{N}(t \to y_2(t+T))$): $\mathcal{N}(y_1(.)) = \bar{F}(T, y_1(.), \mathcal{N}(t \to y_1(t+T))) \le F(T, y_2(.), \mathcal{N}(t \to y_2(t+T))) = \mathcal{N}(y_2(.)) \quad \square$

## Appendix C. Property of the discounted functionals

Let $\mathcal{N}$ be a monotone functional, and $F$ a corresponding Bellman form. We show here that, if $\mathcal{N}$ is discounted, then $\forall \eta \in [m, M]$, $\mathcal{N}$ is $\eta$-convergent. Indeed, let $\omega(.,.): \mathbb{R}^+ \times \mathbb{R}^+ \to \mathbb{R}^+$ decreasing on its second variable, and such that (1) $\forall \eta \in \mathbb{R}^+$: $\lim_{T \to +\infty} \omega(T, \eta) = 0$ and (2) $\forall y(.) \in \mathcal{D}$, $\forall (\eta_1, \eta_2) \in [m, M]^2$:

$$|F(T, y(.) , \eta_1) - F(T, y(.) , \eta_2)| < \omega(T, |\eta_1 - \eta_2|)$$

Because $F(T, y(.) , m) \le \mathcal{N}(y(.)) \le F(T, y(.) , M)$:

$$\limsup_{T \to +\infty} |F(T, y(.) , m) - F(T, y(.) , M)| \le \lim_{T \to +\infty} \omega(T, |M - m|) = 0$$

Thus $\mathcal{N}(y(.)) = \lim_{T \to +\infty} F(T, y(.) , m) = \lim_{T \to +\infty} F(T, y(.) , M)$. To conclude, we just have to note that $\forall \eta \in [m, M]$:

$$F(T, y(.) , m) \le F(T, y(.) , \eta) \le F(T, y(.) , M) \quad \square$$
## Appendix D. Proof of Lemma 12

Let $\mathcal{N}$ be a monotone functional, and $F$ a corresponding Bellman form. $\forall x \in \mathbb{R}^n, t \ge 0, \eta \in [m, M]$, we will denote $V^t(x, \eta) = \inf_{u(.) \in \mathcal{U}} F(t, \psi(x, u(.) , .), \eta)$.

If $\mathcal{N}$ is M-convergent and $V(x) \in \mathbb{R}$ (the case where $V(x) = -\infty$ is immediate), $\forall T > 0, \exists u_T(.) \in \mathcal{U}$ and $\exists \tau_T > 0$ s.t. :

$$ 
\begin{aligned}
V(x) + \frac{1}{T} &> \mathcal{N}(\psi(x, u_T(.) , .)) \\
&\geq F(\tau_T, \psi(x, u_T(.) , .), M) - \frac{1}{T} \\
&> V^T(x, M) \geq V^{\max(\tau_T, T)}(x, M) \\
&\geq V(x) \geq \lim_{T \to +\infty} V^T(x, M)
\end{aligned}
$$

The other inequality is immediate.

If $\mathcal{N}$ is m-convergent and the corresponding hypotheses of Lemma 12 are satisfied: If $V(x) \in \mathbb{R}$ (the case where $V(x) = -\infty$ is similar), $\exists T > 0$ s.t. $\forall t \ge T$, $V^t(x, m) \in \mathbb{R}$. Let $(T_n)_{n \in \mathbb{N}} \in [T, +\infty]^{\mathbb{N}}, (u_n(.))_{n \in \mathbb{N}} \in \mathcal{U}^{\mathbb{N}}$, and $u_{\infty}(.) \in \mathcal{U}$ s.t. $\lim_{n \to +\infty} T_n = +\infty, \forall t \ge 0 \lim_{n \to +\infty} \psi(x, u_n(.) , t) = \psi(x, u_{\infty}(.) , t)$, and $\forall n \in \mathbb{N}: F(T_n, \psi(x, u_n(.) , .), m) \le V^{Tn}(x, m) + \frac{1}{n}$ and $T_{n+1} > T_n$

Because $\forall n \in \mathbb{N}, \exists N > n$ s.t. $F^N(T_n, \psi(x, u_N(.) , .), m) \ge F(T_n, \psi(x, u_{\infty}(.) , .), m) - \frac{1}{n}$, we may define by recurrence $\kappa(.) : N \to N$ an increasing function s.t. $\forall n \ge 0:$

$$ 
\begin{aligned}
F(T_{\kappa(n)}, \phi(x, u_{\infty}(.) , .), m) &- \frac{1}{\kappa(n)} \\
&\le F(T_{\kappa(n)}, \phi(x, u_{\kappa(n+1)}(.) , .), m)
\end{aligned}
$$

So finally, and because $F(T, \psi(x, u_{\kappa(n+1)}(.) , .), m)$ is increasing in $T, \forall n \in \mathbb{N}:$

$$ 
\begin{aligned}
F(T_{\kappa(n+1)}, \psi(x, u_{\kappa(n+1)}(.) , .), m) \\
&\geq F(T_{\kappa(n)}, \psi(x, u_{\infty}(.) , .), m) - \frac{1}{\kappa(n)} \\
&\geq F(T_{\kappa(n)}, \psi(x, u_{\kappa(n)}(.) , .), m) - \frac{2}{\kappa(n)}
\end{aligned}
$$

Thus because $\mathcal{N}$ is m-convergent:

$$ \mathcal{N}(\psi(x, u_{\infty}(.) , .)) = \lim_{n \to +\infty} F(T_{\kappa(n)}, \psi(x, u_{\kappa(n)}(.) , .), m) $$

In addition, $\forall n \in \mathbb{N}:$

$$ V(x) \geq V^{Tn}(x, m) \geq F(T_n, \psi(x, u_n(.) , .), m) - \frac{1}{n}, \text{ thus:} $$

$$ 
\begin{aligned}
V(x) &\geq \lim_{n \to +\infty} \inf_{u(.) \in \mathcal{U}} F(T_n, \psi(x, u(.) , .), m) \\
&= \mathcal{N}(\psi(x, u_{\infty}(.) , .)) \geq V(x)
\end{aligned}
$$

If $\mathcal{N}$ is discounted, the result is a direct consequence of this hypothesis. □

## Appendix E. Proof of Lemma 13

Let $\mathcal{N}$ be a monotone functional, and $F$ a corresponding Bellman form. Let $W: \mathbb{R}^n \to [m, M]$ be s.t. $\forall T \ge 0:$

$W(x) \le \inf_{u(.) \in \mathcal{U}} F(T, \psi(x, u(.) , .), W(\phi(x, u(.) , T)))$ then $W(x) \le \inf_{u(.) \in \mathcal{U}} F(T, \psi(x, u(.) , .), M)$. So if $\lim_{T \to +\infty} \inf_{u(.) \in \mathcal{U}} F(T, \psi(x, u(.) , .), M) = V(x)$, then $W(x) \le V(x)$. The other result of the lemma is proven in a similar manner. □

## Appendix F. Proof of Lemma 14

Let $\mathcal{N}$ be a monotone functional, and $F$ a corresponding Bellman form. We assume that $\forall x \in \mathbb{R}^n:

$\lim_{T \to +\infty} \inf_{u(.) \in \mathcal{U}} F(T, \psi(x, u(.) , .), M) = V(x)$ and $\forall T \ge 0, \forall x \in \mathbb{R}^n, \forall \epsilon > 0, \exists \mu > 0, \forall z \in \mathcal{B}(x, \mu), \forall u(.) \in \mathcal{U}: F(T, \psi(z, u(.) , .), M) - F(T, \psi(x, u(.) , .), M) \le \epsilon$

We show that $x \to V^T(x, M)$ is upper semi-continuous on $\mathbb{R}^n$. Indeed, $\forall \eta > 0$, let $u_{\eta}(.) \in \mathcal{U}$ such that:

$$ 
\begin{aligned}
F(T, \psi(x, u_{\eta}(.) , .), M) &\le V^T(x, M) + \eta \\
&\Rightarrow F(T, \psi(z, u_{\eta}(.) , .), M) - V^T(x, M) \le \epsilon + \eta \\
&\Rightarrow V^T(z, M) - V^T(x, M) \le \epsilon + \eta \\
&\Rightarrow V^T(z, M) - V^T(x, M) \le \epsilon
\end{aligned}
$$

And so $x \to V^T(x, M)$ is upper semi-continuous. In addition, the function $T \to V^T(x, M)$ is decreasing. So:

$$ V(x) = \lim_{T \to +\infty} V^T(x) = \inf_{T \in \mathbb{R}^+} V^T(x) $$

That means $V$ is the lower bound of a family of upper semi-continuous functions $\{V^T|T \in \mathbb{R}^+\}$, thus $V$ is upper semi-continuous. The proof of the remainder of the lemma is similar. □

## Appendix G. Characterization of value functions

We first characterize the value function of $y(.) \to \sup_{t \in [0, T]} l(y(t))$ as defined in Section 5. Its Bellman form is $F(T, y(.) , \eta) = \max \left\{ \sup_{t \in [0, T]} l(y(t)) , \eta \right\}$. Lemma 12 applies, and thus the hypothesis

of the second point of Lemma 13 is satisfied. Let $\tilde{V} \in \mathbb{R}^n$ be such that $\forall x \in \mathbb{R}, \tilde{V}(x) = l(x)$ if $x \in [-0.5, 1], \tilde{V}(x) = l(1)$ if $x \ge 1$ and $\tilde{V}(x) = l(-0.5)$ if $x \le -0.5$. To verify that $\tilde{V} = V$, we apply the second point of Lemma 13. Let us first note that $\tilde{V}$ is a super-solution of the Bellman equation. Indeed, $\forall x \in \mathbb{R}, \forall T \ge 0$, if $x \in [-0.5, 1]$, then $\tilde{V}(x) \ge F(T, \phi(x, u_x(.) , .), \tilde{V}(\phi(x, u_x(.) , T)))$ with $u_x(.) = x$ if $x \in [-0.5, 1]$, $u_x(.) = -0.5$ if $x \le -0.5$ and $u_x(.) = 1$ if $x \ge 1$. Thus $\tilde{V}(x) \ge \inf_{u(.) \in \mathcal{U}} F(T, \phi(x, u(.) , .), \tilde{V}(\phi(x, u(.) , T)))$

$$ 
\begin{aligned}
\text{Secondly, } \forall W: \mathbb{R}^n \to [m, M] \text{ s.t. } \forall x \in \mathbb{R}^n, \forall T \ge 0: \\
W(x) \ge \inf_{u(.) \in \mathcal{U}} \max(\sup_{t < T} l(\phi(x, u(.) , t)), W(\phi(x, u(.) , T)))
\end{aligned}
$$

So $W(x) \ge \inf_{u(.) \in \mathcal{U}} \sup_{t < T} l(\phi(x_z, u(.) , t))$. Thus $\forall \epsilon > 0, \exists T > 0$ s.t. $W(x) + \epsilon \ge l(\phi(x, u_x(.) , T)) \ge \tilde{V}(x) - \epsilon$

The characterization of the value functions corresponding to the minmin and Lagrangian problems is similar. The latter approach introduces a monotone functional, and its Bellman form is expressed as $F(T, y(.) , \eta) = \max(\eta, \sup_{t < T} g(y(t)))$ if $\eta \ge 0$ or $\exists t < T$ s.t. $g(y(t)) \ge 0$, or $F(T, y(.) , \eta) = \min(\inf_{t < T} e^{-\lambda t} g(y(t)), e^{-\lambda T} \eta)$ else. Although this functional is m-convergent, the hypotheses of Lemma 12 are not satisfied. To apply Lemma 13 and thereby characterize the value function, we demonstrate that: $\lim_{T \to +\infty} \inf_{u(.) \in \mathcal{U}} F(T, \phi(x, u(.) , .) = V(x)$. If $V(x) < 0$, then $\exists u(.) \in \mathcal{U}$ s.t. $\forall t \ge 0$ $g(\phi(x, u(.) , t)) < 0$ and thus the cost function is equal to $y(.) \to \inf_{T \in \mathbb{R}^+} e^{-\lambda t} g(y(.))$ at $\phi(x, u(.) , .)$, which is a monotone and discounted functional, so Lemma 12 applies. Similarly, if $V(x) \ge 0$, this means $x \in [-\alpha, \alpha]$, and hence the cost function is equal to $y(.) \to \sup_{T \in \mathbb{R}^+} g(y(.))$ at $\phi(x, u(.) , .)$, so Lemma 12 also applies. Finally, it is possible to apply Lemma 13 to characterize $V$. □

## References

* Bardi, Martino, & Capuzzo-Dolcetta, Italo (1997). *Optimal control and viscosity solutions of Hamilton-Jacobi-Bellman equations*. Springer Science & Business Media.

* Barles, G., & Souganidis, P. E. (1990). Convergence of approximation schemes for fully nonlinear second order equations. vol.4, In 29th IEEE conference on decision and control (pp. 2347-2349).

* Barron, E. N., & Ishii, H. (1989). The Bellman equation for minimizing the maximum cost. *Nonlinear Analysis. Theory, Methods & Applications*, 13(9), 1067-1090.

* Baumeister, J., Leitão, Antonio, & Silva, Geraldo Nunes (2007). On the value function for nonautonomous optimal control problems with infinite horizon. *Systems & Control Letters*, 56(3), 188-196.

* Bellman, Richard (1954). The theory of dynamic programming. *Bulletin of the American Mathematical Society*, 60(6), 503-515.

* Blumin, Pierre-Alexandre, & Rapaport, Alain (2023). Control theory perspectives on mathematical epidemiology. In *IFAC world congress 2023-22nd world congress of the international federation of automatic control* (p. 3p).

* Caen, Auguste (2019). *Towards the viable management of anthropized lake dynamic systems* (Ph.D. thesis), Université Clermont Auvergne [2017-2020].

* Cannarsa, Piermarco, Gaubert, Stephane, Mendico, Cristian, & Quincampoix, Marc (2023). Analysis of the vanishing discount limit for optimal control problems in continuous and discrete time. *arXiv preprint arXiv:2306.07234*.

* Carlson, Dean A., Haurie, Alain B., & Leizarowitz, Arie (2012). *Infinite horizon optimal control: Deterministic and stochastic systems*. Springer Science & Business Media.

* Coquelin, Pierre-Arnaud, Martin, Sophie, & Munos, Rémi (2007). A dynamic programming approach to viability problems. In *Approximate dynamic programming and reinforcement learning, 2007. ADPRL 2007. IEEE international symposium on* (pp. 178-184). IEEE.

* Feinstein, C. D., & Luenberger, D. G. (1981). Analysis of the asymptotic behavior of optimal control trajectories: The implicit programming problem. *SIAM Journal on Control and Optimization*, 19(5), 561-585.

* Fleming, Wendell H., & Soner, Halil Mete (2006). *vol. 25, Controlled Markov processes and viscosity solutions*. Springer Science & Business Media.
Jones, Morgan, & Peet, Matthew M. (2021). A generalization of Bellman's equation with application to path planning, obstacle avoidance and invariant set estimation. *Automatica*, **127**, Article 109510.

Kumar, E. Vinodh, & Jerome, Jovitha (2013). Robust LQR controller design for stabilizing and trajectory tracking of inverted pendulum. *Procedia Engineering*, **64**, 169–178.

Lin, Ziyu, Duan, Jingliang, Li, Shengbo Eben, Li, Jie, Ma, Haitong, Sun, Qi, et al. (2021). Solving finite-horizon hjb for optimal control of continuous-time systems. In *2021 international conference on computer, control and robotics* (pp. 116–122). IEEE.

Lygeros, John (2003). *Infinite horizon minmax optimal control: A viability approach*. *IFAC Proceedings Volumes*, **36**(6), 265–270.

Lygeros, John (2004). On reachability and minimum cost optimal control. *Automatica*, **40**(6), 917–927.

Maitra, Ashok (1966). A note on undiscounted dynamic programming. *The Annals of Mathematical Statistics*, **37**(4), 1042–1044.

Meng, Fanwei, Wang, Dini, Yang, Penghui, & Xie, Guanzhou (2019). Application of sum of squares method in nonlinear $H_\infty$ control for satellite attitude maneuvers. *Complexity*, **2019**, 1–10.

Molina, Emilio, Rapaport, Alain, & Ramírez, Héctor (2022). Equivalent formulations of optimal control problems with maximum cost and applications. *Journal of Optimization Theory and Applications*, **195**(3), 953–975.

Postoyan, Romain, Buşoniu, Lucian, Nešić, Dragan, & Daafouz, Jamal (2016). Stability analysis of discrete-time infinite-horizon optimal control with discounted cost. *IEEE Transactions on Automatic Control*, **62**(6), 2736–2749.

Rust, John (2019). Has dynamic programming improved decision making? *Annual Review of Economics*, **11**, 833–858.

Shen, Zhong-Hua, Chu, Yu-Ming, Khan, Muhammad Altaf, Muhammad, Shabbir, Al-Hartomy, Omar A, & Higazy, M (2021). Mathematical modeling and optimal control of the COVID-19 dynamics. *Results in Physics*, **31**, Article 105028.

<figure><img src="image_5.png" /></figure>

**Auguste Caen** holds an engineering degree in applied mathematics and computer science. After completing his Ph.D. in 2019 at IRSTEA (now INRAE) in dynamic modeling, he is currently a research scientist at INRAE in the GABI unit. His research interests include the modeling of biological and environmental systems, population dynamics, and control theory.

<figure><img src="image_6.png" /></figure>

**Jean-Denis Mathias**, INRAE, jean-denis.mathias@inrae.fr is a Research Director at INRAE (National Research Institute for Agriculture, Food and Environment). He is the Director of the Laboratory of Engineering for Complex Systems (LISC) in Aubiére (INRAE Clermont-Ferrand Center). His research focuses on the study of complex systems, with particular interest in socio-ecological systems and the management of coupled infrastructures. His work relies on modeling controlled dynamic systems using ordinary differential equations, agent-based modeling, and viability theory. He earned

his Ph.D. in 2005 from Blaise Pascal University and his habilitation to supervise research in 2011. He spent a research stay (20152016) at Arizona State University (USA). From 2018 to 2020, he served as Deputy Head of the Territoires Department at Irstea. He was Deputy Director of LISC before becoming its Director on January 1, 2023.