# Dynamic Programming for Mean-field type Control

Mathieu Laurière, Olivier Pironneau

► To cite this version:

Mathieu Laurière, Olivier Pironneau. Dynamic Programming for Mean-field type Control. Compte Rendus de l'académie des sciences, 2014, Serie I mathématiques, 352, pp.707-713. hal-01018361

HAL Id: hal-01018361

https://hal.science/hal-01018361

Submitted on 4 Jul 2014

HAL is a multi-disciplinary open access
archive for the deposit and dissemination of sci-
entific research documents, whether they are pub-
lished or not. The documents may come from
teaching and research institutions in France or
abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est
destinée au dépôt et à la diffusion de documents
scientifiques de niveau recherche, publiés ou non,
émanant des établissements d'enseignement et de
recherche français ou étrangers, des laboratoires
publics ou privés.

<figure><img src="image_1.png" /></figure>
# Dynamic Programming for Mean-field type Control

Mathieu Laurièreª, Olivier Pironneauᵇ,

ª LJLL, Université Pierre et Marie Curie (Paris6), Place Jussieu Paris 75005, E-mail: Mathieu.Lauriere@UPMC.fr

ᵇ LJLL. E-mail: Olivier.Pironneau@UPMC.fr

(Reçu le 13 Avril 2014, accepté après révision le 4 Juillet 2014)

**Abstract** For mean-field type control problems, stochastic dynamic programming requires adaptation. We propose to reformulate the problem as a distributed control problem by assuming that the PDF $\rho$ of the stochastic process exists. Then we show that Bellman's principle applies to the dynamic programming value function $V(\tau, \rho_\tau)$ where the dependency on $\rho_\tau$ is functional as in P.L. Lions' analysis of mean-filed games (2007). We derive HJB equations and apply them to two examples, a portfolio optimization and a systemic risk model.

## Programmation dynamique pour les problèmes de contrôle à champs moyen

**Résumé** Pour les problèmes de contrôle stochastique à champs moyen la programmation dynamique ne s'applique pas sans adaptation; mais si l'on reformule le problème avec l'équation de Fokker-Planck, on peut le faire en utilisant une fonctionnelle valeur $\{\tau, \rho_\tau(\cdot)\} \to V(\tau, \rho_\tau)$ comme dans l'analyse des problèmes de jeux à champs moyen par P.L. Lions (2007). Les résultats sont appliqués à un problème d'optimisation de portefeuille et à un problème de risque systémique.

## 1. Introduction

Stochastic control is an old topic [5, 11, 13, 14] which has a renewed interest in economy and finance due to mean-field games [8, 7, 12]. They lead, among other things, to stochastic control problems which involve statistics of the Markov process like means and variance. Optimality conditions for these are derived either by stochastic calculus of variation [1] or by stochastic dynamic programming in the quadratic case [2, 3], but not in the general case for the fundamental reason that Bellman's principle does not apply in its original form on the stochastic trajectories of say $X_t$ if those depend upon statistics of $X_t$ like its mean value. As noticed earlier in [9] and in [4]<fnref n="1" />, there seems to be no such restriction if one works with the probability measure of $X_t$ and use the Fokker-Planck equation.

In this note we apply the dynamic programming argument to the value functional $V(\tau, \rho_\tau(\cdot))$ where $\rho_\tau$ is the PDF of $X_\tau$. Of course this is at the cost of several regularity assumptions, in particular it requires the existence of PDF at all times.

Once the problem is reformulated with the Fokker-Planck equation, it becomes a somewhat standard exercise to find the optimality necessary conditions by a calculus of variations. So the note begins likewise. Then a similar result is obtained by using dynamic programming and the connection with the previous approach and with stochastic dynamic programming is established, with the advantage that sufficient conditions for optimality are obtained. Finally we apply the method to two mean-field type control problems stated in [1] and [7].

<footnote n="1">1. This preprint came to our knowledge after the submission of this note</footnote>
## 2. The Problem

Let $d, s, r \in \mathbf{N}^+$. Consider a stochastic differential equation

$$dX_t = u(X_t, t)dt + \sigma(X_t, t, u(X_t, t))dW_t, \quad (2.1)$$

where $T > 0$, $u: \mathbf{R}^d \times (0, T) \to \mathbf{R}^d$, $\sigma: \mathbf{R}^d \times (0, T) \times \mathbf{R}^d \to \mathbf{R}^{d \times d}$ and $W_t$ is a d-vector of independent Brownian motions. We make the usual assumptions for $X_t$ to exist once $X_0$ is known [13].

Let $\tilde{H}: \mathbf{R}^d \times (0, T) \times \mathbf{R}^d \times \mathbf{R}^r \to \mathbf{R}$, $\tilde{h}: \mathbf{R}^d \times (0, T) \times \mathbf{R}^d \to \mathbf{R}^r$, $G: \mathbf{R}^d \times \mathbf{R}^s \to \mathbf{R}$, $g: \mathbf{R}^d \to \mathbf{R}^s$. Assume also that $\rho_0$ is positive with unit measure on $\mathbf{R}^d$.

Let $\mathcal{V}_d \subset \mathbf{R}^d$, $\mathcal{U}_d = \{u \in (L^\infty(\mathbf{R}^d \times \mathbf{R}))^d : u(x, t) \in \mathcal{V}_d \ \forall x, t\}$ and consider the problem

$$\min_{u \in \mathcal{U}_d} \quad J := \int_0^T \mathbf{E}[\tilde{H}(X_t, t, u(X_t, t), \mathbf{E}[\tilde{h}(X_t, t, u(X_t, t))])dt + \mathbf{E}[G(X_T, \mathbf{E}[g(X_T)])] \\ \text{subject to (2.1) and such that } \rho_0 \text{ is the PDF of } X_0 \quad (2.2)$$

Andersson et al [1] analyzed this problem using stochastic calculus of variations, claiming rightly that dynamic programming is not possible unless $\tilde{h} = 0$, $g = 0$. Yet denoting $Q = \mathbf{R}^d \times (0, T)$ and $\mu_{ij} = \frac{1}{2} \sum_k \sigma_{ik}\sigma_{jk}$, with sufficient regularity, namely if $X_t$ has a PDF $\rho_t$ (for weaker hypotheses see [10]), the problem is equivalent to

$$\min_{u \in \mathcal{U}_d} \quad J = \int_Q H(x, t, u(x, t), \rho_t(x), \chi(t)) \rho_t(x) dx dt + \int_{\mathbf{R}^d} G(x, \xi) \rho_{|T} dx \\ \text{where } \chi(t) = \int_{\mathbf{R}^d} h(x, t, u(x, t), \rho_t(x)) \rho_t(x) dx, \quad \xi = \int_{\mathbf{R}^d} g(x) \rho_T(x) dx \text{ and } \rho_t \text{ s.t.} \\ \partial_t \rho + \nabla \cdot (u\rho) - \nabla \cdot \nabla \cdot (\mu\rho) = 0, \quad \rho|_0 = \rho_0(x), \quad x \in \mathbf{R}^d \quad (2.3)$$

where $\tilde{H} = H$, $\tilde{h} = h$ if these are not functions of $\rho_t(x)$.

**HYPOTHESIS 1.** – Assume that all data are continuously differentiable with respect to $u$ and $\rho$ and have additional regularity so that the solution to the Fokker-Planck equation is unique and uniformly continuously differentiable with respect to $u$ and $\mu$.

## 3. Calculus of Variations

**PROPOSITION 1.** – Let $A: B = \text{trace}(A^T B)$. A control $u$ is optimal for (2.3) only if

$$\int_{\mathbf{R}^d} \left( H'_{u} + h'_{u} \int_{\mathbf{R}^d} H'_{\chi} \rho dx + \nabla \rho^* - \mu'_{u} : (\nabla \nabla \rho^*) \right) (v-u) \rho dx \geq 0 \quad \forall t, \forall v \in \mathcal{U}_d \text{ with} \quad (3.4)$$

$$\partial_t \rho^* + u \nabla \rho^* + \mu : \nabla \nabla \rho^* = - \left[ H'_{\rho} \rho + H + (h'_{\rho} \rho + h) \int_{\mathbf{R}^d} H'_{\chi} \rho dx \right], \quad \rho^*|_T = g \int_{\mathbf{R}^d} G'_{\xi} \rho|_T dx + G. \quad (3.5)$$

*Proof.* – Let us regularize problem (2.3) by replacing $\mathbf{R}^d$ by $\Omega := (-L, L)^d$ with $L \gg 1$. Now $Q = \Omega \times (0, T)$. We add to the Fokker-Planck equation (2.3) the boundary conditions: $\rho(x, t) = 0, \forall x \in \partial\Omega, t \in (0, T)$. Consider an admissible variation $\lambda\delta u$, i.e. $u + \lambda\delta u \in \mathcal{U}_d$ for all $\lambda \in (0, 1)$. Such a variation induces a variation $\lambda\delta\rho$ of $\rho$ given by

$$\partial_t \delta \rho + \nabla \cdot (u \delta \rho + \rho \delta u + \lambda \delta u \delta \rho) - \nabla \cdot \nabla \cdot (\mu \delta \rho + \mu'_{u} \delta u (\rho + \lambda \delta \rho)) = 0, \quad \delta \rho|_{t=0} = 0, \quad (3.6)$$

where $\mu'_{u}$ is evaluated at $x, t, u + \theta \delta u$ for some $\theta \in (0, \lambda)$. By hypothesis the solution of the Fokker-Planck eq. in (2.3) depends continuously on the data $u, \mu$, so (3.6) with $\lambda = 0$ defines $\delta\rho$. Also

$$\delta\chi = \int_{\Omega} [(h'_{u}\delta u + h'_{\rho}\delta\rho)\rho + h\delta\rho], \quad \delta J = \int_{Q} [(H'_{u}\delta u + H'_{\rho}\delta\rho + H'_{\chi}\delta\chi)\rho + H\delta\rho] + \int_{\Omega} [G'_{\xi}\delta\xi\rho|_{T} + G\delta\rho|_{T}]$$
$$ = \int_Q \left[ (H'_u + h'_u \int_{\Omega} [H'_{\chi} \rho]) \rho \delta u \right] + \int_Q \left[ (H'_{\rho} \rho + H + (h'_{\rho} \rho + h) \int_{\Omega} [H'_{\chi} \rho]) \delta \rho \right] + \int_{\Omega} \left[ \int_{\Omega} [G'_{\xi} \rho_{|T}] g + G \right] \delta \rho_{|T} $$

The adjoint state $\rho^*$ is given by (3.5) and $\rho^*|_{\partial\Omega} = 0$. Then, multiplied by $\delta\rho$ and integrated on $Q$ (3.5) gives

$$ \begin{aligned} \int_Q \delta\rho \left[ H'_{\rho}\rho + H + (h'_{\rho}\rho + h) \int_{\Omega} [H'_{\chi}\rho] \right] &= - \int_Q \delta\rho [\partial_t \rho^* + u \nabla \rho^* + \mu : \nabla \nabla \rho^*] \\ &= \int_Q [\rho^*(\partial_t \delta\rho + \nabla \cdot (u\delta\rho) - \nabla \cdot \nabla \cdot (\mu\delta\rho))] - \int_{\Omega} \rho^* \delta\rho|_{0}^{T} \\ &= - \int_Q [\rho^* \nabla \cdot (\rho\delta u - \nabla \cdot (\mu'_{u}\delta u\rho))] - \int_{\Omega} [(g \int_{\Omega} [G'_{\xi}\rho_{|T}] + G)\delta\rho]_{|T} \\ \text{Hence} \quad \delta J &= \int_Q \left[ (H'_{u} + h'_{u} \int_{\Omega} [H'_{\chi}\rho]) \rho \delta u \right] - \int_Q [\rho^* \nabla \cdot (\rho\delta u - \nabla \cdot (\mu'_{u}\delta u\rho))] \\ &= \int_Q \left[ (H'_{u} + h'_{u} \int_{\Omega} [H'_{\chi}\rho] + \nabla \rho^* - (\nabla \nabla \rho^*) \mu'_{u}) \rho \delta u \right] \end{aligned} \quad (3.7) $$

□

## 4. Dynamic Programming

For notational clarity consider the more general case where $H, G$ are functionals of $\rho_t(\cdot)$. For any $\tau \in [0, T]$ and any $\rho_{\tau} \ge 0$ with unit measure on $\mathbb{R}^d$, let

$$ V(\tau; \rho_{\tau}) = \min_{u \in \mathcal{U}_d} J(\tau; \rho_{\tau}, u) := \int_{\tau}^{T} \int_{\mathbb{R}^d} H(x, t, u(x, t); \rho_{\tau}) \rho_{\tau}(x) dx dt + \int_{\mathbb{R}^d} G(x; \rho_{|T}) \rho_{|T}(x) dx \quad (4.8) $$

subject to (2.3), i.e. such that $\rho_t$ is the PDF of $X_t$ given by (2.1) starting with $\rho_{\tau}$ at time $\tau$

Note that the second parameter in $V$ is a function of $x$, yet it is not $V(\tau, \rho_{\tau}(x))$ but $V(\tau; \rho_{\tau}(\cdot))$. We now prove the following version of Bellman's principle of optimality:

**PROPOSITION 2.** – If the problem is regular, then for any $\tau \in [0, T]$ and any positive $\rho_{\tau}$ with unit measure on $\mathbb{R}^d$, we have:

$$ V(\tau; \rho_{\tau}) = \min_{u \in \mathcal{U}_d} \int_{\tau}^{\tau + \delta\tau} \int_{\mathbb{R}^d} H(x, t, u(x, t); \rho_{\tau}) \rho_{\tau}(x) dx dt + V(\tau + \delta\tau; \rho_{\tau + \delta\tau}) \quad (4.9) $$

subject to $\rho_t$ given by (2.3) on $[\tau, \tau + \delta\tau]$ initialized by $\rho_{\tau}$ at time $\tau$

*Proof.* – Denote the infimum of the right-hand side by $\bar{V}(\tau; \rho_{\tau})$. For any $\epsilon > 0$, there exists an $u \in \mathcal{U}_d$ such that, if $\rho_t$ is the solution of (2.3) with control $u$:

$$ \begin{aligned} V(\tau; \rho_{\tau}) + \epsilon > J(\tau; \rho_{\tau}, u) &= \int_{\tau}^{T} \int_{\mathbb{R}^d} H(x, t, u(x, t); \rho_{\tau}) \rho_{\tau}(x) dx dt + \int_{\mathbb{R}^d} G(x; \rho_{|T}) \rho_{|T}(x) dx \\ &= \int_{\tau}^{\tau + \delta\tau} \int_{\mathbb{R}^d} H \rho_{\tau} + \int_{\tau + \delta\tau}^{T} \int_{\mathbb{R}^d} H \rho_{\tau} + \int_{\mathbb{R}^d} G \rho_{|T} \ge \int_{\tau}^{\tau + \delta\tau} \int_{\mathbb{R}^d} H \rho_{\tau} + V(\tau + \delta\tau; \rho_{\tau + \delta\tau}) \ge \bar{V}(\tau; \rho_{\tau}) \end{aligned} $$

Conversely, given $u \in \mathcal{U}_d$ and $\epsilon > 0$, $\exists \tilde{u} \in \mathcal{U}_d$ which coincides with $u$ on $\mathbb{R}^d \times [\tau, \tau + \delta\tau]$, such that:

$$ J(\tau + \delta\tau; \tilde{\rho}_{\tau + \delta\tau}, \tilde{u}) \le V(\tau + \delta\tau; \tilde{\rho}_{\tau + \delta\tau}) + \epsilon $$
where $\tilde{\rho}_t$ is the solution of (2.3) at $t$ with control $\tilde{u}$ starting with $\rho_\tau$ at time $\tau$. Hence :

$$
\begin{align*}
V(\tau; \rho_{\tau}) &= V(\tau; \tilde{\rho}_{\tau}) \le J(\tau; \tilde{\rho}_{\tau}, \tilde{u}) = \int_{\tau}^{T} \int_{\mathbb{R}^d} H(x, t, \tilde{u}(x, t); \tilde{\rho}_t) \tilde{\rho}_t(x) dx dt + \int_{\mathbb{R}^d} G(x; \tilde{\rho}_{|T}) \tilde{\rho}_{|T} dx \\
&= \int_{\tau}^{\tau+\delta\tau} \int_{\mathbb{R}^d} H(x, t, u(x, t); \tilde{\rho}_t) \tilde{\rho}_t(x) dx dt + J(\tau + \delta\tau; \tilde{\rho}_{\tau+\delta\tau}, \tilde{u}) \\
&\le \int_{\tau}^{\tau+\delta\tau} \int_{\mathbb{R}^d} H(x, t, u(x, t); \tilde{\rho}_t) \tilde{\rho}_t(x) dx dt + V(\tau + \delta\tau; \tilde{\rho}_{\tau+\delta\tau}) + \epsilon
\end{align*}
$$

We obtain the conclusion by letting $\epsilon \to 0$ and by taking the infimum over $u \in \mathcal{U}_d$.

<figure><img src="image_2.png" /></figure>

PROPOSITION 3. – (HJB minimum principle). There exists $\{x, \tau, \rho_\tau(\cdot)\} \to V' \in \mathbb{R}$ such that :

$$
0 = \min_{v \in \mathcal{V}_d} \int_{\mathbb{R}^d} \left( \begin{aligned} & H(x, \tau, v(x); \rho_\tau) + H'_\rho(x, \tau, v(x); \rho_\tau) \cdot \rho_\tau \\ & + \partial_\tau V' + \mu(x, \tau, v(x)) : \nabla_x \nabla_x V' + v(x) \cdot \nabla_x V' \end{aligned} \right) \rho_\tau dx \quad (4.10)
$$

where $H'_\rho \cdot \nu = \lim_{\lambda \to 0} [H(x, \tau; \rho + \lambda\nu) - H(x, \tau, \rho)]/\lambda$.

*Proof.* – A first order approximation of the time derivative in the Fokker-Planck equation gives

$$
\delta \rho_{\tau} := \rho_{\tau+\delta\tau} - \rho_{\tau} = \delta\tau [\nabla \cdot \nabla \cdot (\mu_{\tau}\rho_{\tau}) - \nabla \cdot (u_{\tau}\rho)] + o(\delta\tau) \quad (4.11)
$$

When everything is differentiable and smooth,

$$
V(\tau + \delta\tau; \rho_{\tau + \delta\tau}) = V(\tau; \rho_{\tau}) + \partial_{\tau}V(\tau; \rho_{\tau})\delta\tau + V'_{\rho}(\tau; \rho_{\tau}) \cdot \delta\rho_{\tau} + o(\delta\tau) \quad (4.12)
$$

Using (4.12) and the mean value theorem for the time integral, (4.9) yields

$$
V(\tau; \rho_{\tau}) = \min_{u \in \mathcal{U}_d} \left\{ \delta\tau \int_{\mathbb{R}^d} H \rho_{\tau} dx + V(\tau; \rho_{\tau}) + \partial_{\tau} V(\tau; \rho_{\tau}) \delta\tau + V'_{\rho}(\tau; \rho_{\tau}) \cdot \delta\rho_{\tau} + o(\delta\tau) \right\}
$$

The terms $V(\tau; \rho_\tau)$ cancel; divided by $\delta\tau$ and combined with (4.11) and letting $\delta\tau \to 0$, (4.13) gives

$$
0 = \min_{u \in \mathcal{U}_d} \left\{ \int_{\mathbb{R}^d} H \rho_{\tau} dx + \partial_{\tau} V(\tau; \rho_{\tau}) + V'_{\rho}(\tau; \rho_{\tau}) \cdot [\nabla \cdot \nabla \cdot (\mu_{\tau} \rho_{\tau}) - \nabla \cdot (u_{\tau} \rho)] \right\} \quad (4.13)
$$

To finalize the proof we need to relate $V$ to $V'_\rho$ and to its Riesz representative $V'$:

PROPOSITION 4. – For any $\tau \in [0, T]$ and any initial PDF $\rho_\tau$, let $\tilde{u}$ and $\hat{\rho}$ denote respectively the optimal control and the corresponding solution of (2.3). Then:

$$
\int_{\mathbb{R}^d} V'(\tau; \rho_\tau) \rho_\tau dx = V'_\rho(\tau; \rho_\tau) \cdot \rho_\tau = V(\tau; \rho_\tau) + \int_{\tau}^{T} \int_{\mathbb{R}^d} \left( H'_\rho(x, t, \hat{u}(x, t); \hat{\rho}_t) \cdot \hat{\rho}_t \right) \hat{\rho}_t(x) dx dt \\
+ \int_{\mathbb{R}^d} \left( G'_\rho(x; \hat{\rho}_T) \cdot \hat{\rho}_T \right) \hat{\rho}_T(x) dx \quad (4.14)
$$
*Proof.* – Notice that Fokker-Planck implies $\rho_t = \mathbf{G}(t-\tau) * \rho_\tau$ where $\mathbf{G}$ is a semi-group operator. Let $(\hat{u}_t)_{t \in [0,T]}$ be the optimal control and $(\hat{\rho}_t)_{t \in [0,T]}$ the corresponding solution of (2.3). Then:

$$V(\tau; \hat{\rho}_{\tau}) = \int_{\tau}^{T} \int_{\mathbf{R}^{d}} H(x, t, \hat{u}(x, t); \mathbf{G}(t-\tau) * \hat{\rho}_{\tau}) \mathbf{G}(t-\tau) * \hat{\rho}_{\tau} dx dt + \int_{\mathbf{R}^{d}} G(x; \hat{\rho}_{|T}) \hat{\rho}_{|T} dx$$

This can be differentiated with respect to $\rho$ by computing $\lim_{\lambda \to 0} \frac{1}{\lambda}[V(\tau; \hat{\rho}_{\tau} + \lambda\nu) - V(\tau; \hat{\rho}_{\tau})]$, for a given function $x \to \nu(x)$:

$$V'_{\rho}(\tau; \hat{\rho}_{\tau}) \cdot \nu = \int_{\tau}^{T} \int_{\mathbf{R}^{d}} H(x, t, \hat{u}(x, t); \mathbf{G}(t-\tau) * \hat{\rho}_{\tau}) \mathbf{G}(t-\tau) * \nu dx dt + \int_{\mathbf{R}^{d}} G(x; \hat{\rho}_{|T}) \mathbf{G}(T-\tau) * \nu dx \\ + \int_{\tau}^{T} \int_{\mathbf{R}^{d}} \left( H'_{\rho}(x, t, \hat{u}(x, t); \mathbf{G}(t-\tau) * \hat{\rho}_{\tau}) \cdot [\mathbf{G}(t-\tau) * \nu] \right) \mathbf{G}(t-\tau) * \hat{\rho}_{\tau} dx dt \\ + \int_{\mathbf{R}^{d}} \left( G''_{\rho}(x; \mathbf{G}(T-\tau) * \hat{\rho}_{\tau}) \cdot [\mathbf{G}(T-\tau) * \nu] \right) \mathbf{G}(T-\tau) * \hat{\rho}_{\tau} dx$$

Taking $\nu = \hat{\rho}_{\tau}$ leads to (4.14). □

**End of proof of Proposition 3** Differentiating (4.14) w.r. to $\tau$ leads to

$$\partial_{\tau} V(\tau; \rho_{\tau}) = (\partial_{\tau} V'_{\rho}(\tau; \rho_{\tau})) \cdot \rho_{\tau} + \int_{\mathbf{R}^{d}} (H'_{\rho}(x, \tau; \hat{u}_{\tau}(x); \rho_{\tau}) \cdot \rho_{\tau}) \rho_{\tau}(x) dx$$

where $\hat{u}_{\tau}$ is the optimal control at time $\tau$. Now, let us go back to (4.13), which we rewrite:

$$0 = \min_{u_{\tau}} \left\{ \int_{\mathbf{R}^d} \left( H(x, \tau, u_{\tau}(x); \rho_{\tau}) + H'_{\rho}(x, \tau, u_{\tau}(x); \rho_{\tau}) \cdot \rho_{\tau} \right) \rho_{\tau}(x) dx \right. \\ \left. + (\partial_{\tau} V'_{\rho}(\tau; \rho_{\tau})) \cdot \rho_{\tau} + V'_{\rho}(\tau; \rho_{\tau}) \cdot [\nabla \cdot (\mu_{\tau} \rho_{\tau}) - \nabla \cdot (u_{\tau} \rho_{\tau})] \right\} \quad (4.15)$$

By integrating by parts the last term, Proposition 1 is proved. □

**REMARK 1.** – Notice that (4.14) and (4.8) implies:

$$\int_{\mathbf{R}^d} V'_T \hat{\rho}_T dx = V(T, \hat{\rho}_T) = \int_{\mathbf{R}^d} (G + g \int_{\mathbf{R}^d} \partial_{\xi} G \hat{\rho}_T dx) \hat{\rho}_T dx, \quad \hat{\xi} = \int_{\mathbf{R}^d} g(x, \hat{\rho}_T) \hat{\rho}_T dx \quad (4.16)$$

**REMARK 2.** – By taking $\rho_{\tau} = \delta(x - x_0)$ the usual HJB principle is found if $h = g = 0$.

**PROPOSITION 5.** – (Hamilton-Jacobi-Bellman equation) When $V_d = \mathbf{R}^d$, at the optimal solution $\hat{u}$

$$\nabla_u H + \nabla_u H' \cdot \hat{\rho}_\tau + \nabla_x V' + \partial_u \mu : \nabla_x \nabla_x V' = 0 \quad (4.17)$$

$$\int_{\mathbf{R}^d} (H + H' \cdot \hat{\rho}_{\tau} + \partial_{\tau} V' + \hat{\mu} : \nabla_x \nabla_x V' + \hat{u} \cdot \nabla_x V') \hat{\rho}_{\tau} dx = 0 \quad (4.18)$$

**REMARK 3.** – When $H = H(x, t, u(x, t), \rho_t(x), \chi(t))$ with $\chi(t) = \int_{\mathbf{R}^d} h(x, t, u(x, t), \rho_t(x)) \rho_t(x) dx$,

$$H'_{\rho}(x, \tau, u(x, \tau); \rho_{\tau}) \cdot \rho_{\tau} = \rho_{\tau} \partial_{\rho} H + \left( \int_{\mathbf{R}^d} \partial_{\chi} H \rho_{\tau} dx \right) (h + \rho_{\tau} \partial_{\rho} h). \quad (4.19)$$
<figure><img src="image_3.png" /><figcaption>Figure 1 – The control is $v = M$ when $x < x_0$, affine when $x_0 < v < x_1$, and $v = m$ when $x > x_1$ with $x_0 = -\frac{M\sigma^2}{b} + (\frac{1}{2\gamma} + \mathbf{E}_T x)e^{-a(T-t)}$, $x_1 = -\frac{m\sigma^2}{b} + (\frac{1}{2\gamma} + \mathbf{E}_T x)e^{-a(T-t)}$</figcaption></figure>

Then for the optimal $\hat{u}$ and $\hat{\rho}$ (4.18) yields

$$ \partial_{\tau} V' + \hat{\mu} : \nabla_x \nabla_x V' + \hat{u} \cdot \nabla_x V' = - \left[ H + \hat{\rho} \partial_{\rho} H + \left( \int_{\mathbb{R}^d} \partial_{\chi} H \hat{\rho} dx \right) (h + \hat{\rho} \partial_{\rho} h) \right]. \quad (4.20) $$

The link with Section 3. is established: (3.5) and (4.20) coincide with $V' = \rho^*$.

## 5. Portfolio Optimization

Following [1], a portfolio of value $x_t$ made of a risky asset and a riskless one is optimally managed at $t$ if the quantity $v$ invested at $t$ in the risky asset minimizes, with $\rho_{|0}$ given,

$$ J = \frac{1}{2} \int_{\Omega} (\gamma x^2 - x) \rho_{|T} dx - \frac{\gamma}{2} \left[ \int_{\Omega} x \rho_{|T} dx \right]^2, \quad \partial_t \rho + \partial_x [(ax + bv)\rho] - \partial_{xx} \left[ \frac{\sigma^2 v^2}{2} \rho \right] = 0 \quad (5.21) $$

where $a$ is the interest rate $b$ is $a$ minus the drift of the risky asset and $\sigma$ is its volatility. We assume that $v$ is a feedback function $x, t \to v(x, t)$ there are bounds on $v$, at each time $m \le v \le M$. Thus $d = 1$ and

$$ H = 0, \quad h = 0, \quad G = \frac{1}{2}(\gamma x^2 - x - \gamma x \int_{\Omega} x \rho_{|T} dx), \quad u = ax + bv, \quad \mu = \frac{\sigma^2 v^2}{2} \quad (5.22) $$

The problem deviates slightly from framework (2.3) but the methodology is the same and gives:

$$ \begin{gathered} \delta J = \int_Q [(b\partial_x\rho^* + \sigma^2 v \partial_{xx}\rho^*)\rho\delta v] \text{ with } \rho^*(\pm\infty) = 0, \text{ and} \\ \partial_t \rho^* + (ax + bv)\partial_x\rho^* + \frac{\sigma^2 v^2}{2}\partial_{xx}\rho^* = 0, \quad \rho^*|_T = \frac{1}{2}(\gamma x^2 - x) - \gamma x \int_{\mathbb{R}} [x\rho_{|T}] \quad (5.23) \end{gathered} $$

### 5.1. Polynomial Solution

Assume $\rho^* = qx^2 + rx + s$ and $v = Ax + B \in (m, M)$. Then the adjoint equation gives solvable ODEs for $q(t), r(t)$ and $s(t)$. Because of the constraints, the general solution has 3 regimes as shown on figure ??.

*Proof.* – In what follows we denote $\mathbf{E}_T x := \int_{\mathbb{R}} x \rho_{|T}(x) dx$. and $I$ an interval. By assuming $\rho^*$ and $v$ polynomial, the adjoint equation becomes:

$$ \dot{q}x^2 + \dot{r}x + \dot{s} + (ax + b(Ax + B))(2qx + r) + \sigma^2(Ax + B)^2 q = 0 \quad (5.24) $$
In turn, it implies $\dot{q} + q(2a + 2bA + A^2\sigma^2) = 0$, $q(T) = \frac{\gamma}{2}$, $s(T) = 0$, $r(T) = -\frac{1}{2} - \gamma \int_{\mathbb{R}} x\rho_{|T}$ and

$$ 
\begin{aligned}
\dot{r} + (a + bA)r + 2qBb + 2\sigma^2 ABq = 0, \quad \dot{s} + rBb + \sigma^2 B^2 q = 0, \quad s(T) = 0, \\
b\partial_x \rho^* + \sigma^2 v\partial_{xx} \rho^* = 2(b + \sigma^2 A)qx + br + 2\sigma^2 Bq
\end{aligned} 
\quad (5.25) $$

**Regime 1** $v = Ax + B = m(t) \forall x \in I \Rightarrow A = 0, B = m$. This happens only when $b(2qx + r) + 2m\sigma^2q \ge 0$. Then $q = \frac{\gamma}{2}e^{2a(T-t)}$, $r = -(\frac{1}{2} + \gamma E_T x)e^{a(T-t)} - b\gamma e^{a(T-t)} \int_t^T m(\tau)e^{a(T-\tau)}d\tau$. Hence $x > -\frac{m\sigma^2}{b} - \frac{r}{2q} = -\frac{m\sigma^2}{b} + (\frac{1}{2\gamma} + E_T x)e^{-a(T-t)} + b \int_t^T m(\tau)e^{-a(\tau-t)}d\tau$ is required for this regime.

**Regime 2** $b + \sigma^2 A = br + 2\sigma^2 Bq = 0$ and $m < v = Ax + B < M$. Then $A = -\frac{b}{\sigma^2}$, $B = -\frac{br}{2\sigma^2 q}$ $q = \frac{\gamma}{2}e^{(2a - \frac{b^2}{\sigma^2})(T-t)}$, $r = -(\frac{1}{2} + \gamma E_T x)e^{(a - \frac{b^2}{\sigma^2})(T-t)}$ giving $B = \frac{b}{\sigma^2}(\frac{1}{2\gamma} + E_T x)e^{-a(T-t)}$ and $Ax + B = -\frac{b}{\sigma^2}x + \frac{b}{\sigma^2}(\frac{1}{2\gamma} + E_T x)e^{-a(T-t)}$. Thus this regime holds only if $\forall x \in I$

$$ -\frac{M\sigma^2}{b} + (\frac{1}{2\gamma} + E_T x)e^{-a(T-t)} < x < -\frac{m\sigma^2}{b} + (\frac{1}{2\gamma} + E_T x)e^{-a(T-t)} $$

**Regime 3** Similar to Regime 1, $v = M(t)$ requires $x > -\frac{M\sigma^2}{b} + (\frac{1}{2\gamma} + E_T x)e^{-a(T-t)} + b \int_t^T M(\tau)e^{-a(\tau-t)}d\tau$. $\square$

**REMARK 4.** – The advantage here compared with [1] is that we do not need to guess the shape of the control nor of the adjoint state, once it is assumed polynomial. The analysis also handles constraints.

## 6. Numerical Solution of a Systemic Risk Problem

In [7] it is shown that the rare event probability that the state of a system of N banks, depending on the mean situation of all, transits from a stable situation $\rho_0$ to a critical one $\rho_T$ at time T is given finding the minimum in g of J with

$$ J(g) = \frac{1}{2\sigma^2} \int_Q g^2 \rho : \partial_t \rho + \partial_x(b(x, g)\rho) - \frac{\sigma^2}{2} \partial_{xx} \rho = 0, \quad \rho(x, 0) = \rho_0(x) \quad (6.26) $$

subject to $b = -hx^3 + (h - \theta)x - \theta \int_{\mathbb{R}} x\rho - g$, $\rho(x, T) = \rho_T(x)$ where $h, \theta \in \mathbb{R}$ are given. With $\kappa = h - \theta$, this is also

$$ \min_u J = \frac{1}{2\sigma^2} \int_Q (hx^3 - \kappa x - \chi + \frac{u}{\rho})^2 \rho : \partial_t \rho - \frac{\sigma^2}{2} \partial_{xx} \rho = -\partial_x u, \rho|_{0}, \rho|_{T} \text{ given, } \chi = \theta \int_{\mathbb{R}} x\rho. $$

Now we notice that $\rho = \tilde{\rho} + \frac{t}{T}(\rho_T - \tilde{\rho}|_T)$ satisfies the conditions at 0 and T and the PDE with $u = \tilde{u} - \frac{1}{T} \int^x (\rho_T - \tilde{\rho}|_T)dx + \frac{t\sigma^2}{2T} \partial_x(\rho_T - \tilde{\rho}|_T)$ provided $\partial_t \tilde{\rho} - \frac{\sigma^2}{2} \partial_{xx} \tilde{\rho} = -\partial_x \tilde{u}$. This means that the problem is in the form analyzed above with state variable $\{\tilde{\rho}, \rho\}$ and control $\tilde{u}$; naturally the adjoint state has also two components: $\{\rho^*, \tilde{\rho}^*\}$.

Based on the variation of J with respect to u we have used 100 iterations of a gradient method with fixed step size, $\omega = 0.3$. The parameters of the problem are $T = 2$, $h = 0.1$, $\theta = 1$, $\sigma^2 = \frac{2}{3}\theta$ and $\rho_0, \rho_T$ are as in [7]. The numerical method for the PDEs is a centered space-time finite element method of degree 1 on a mesh of 150 points and 40 time steps.

## References
<figure><img src="image_4.png" /><figcaption>Figure 2 – Left: $x, t \to \rho(x, t)$; $x \in (-15, 15)$ is horizontal, $t \in (0, 2)$ is from front to back with origin $x = -15, t = 0$ on the lower left corner. Right: $x, t \to g(x, t)$.</figcaption></figure>

<figure><img src="image_5.png" /><figcaption>Figure 3 – $\rho^*$ (left) and $\tilde{\rho}^*$ (right) versus $x$ (horizontal) and $t$ (vertical).</figcaption></figure>

<figure><img src="image_6.png" /><figcaption>Figure 4 – Iteration history: values of $J$ (top curve) and $||grad_u J|^2$ versus iteration count.</figcaption></figure>
[1] D. ANDERSSON AND B. DERVICHE. A Maximum Principle for SDEs of Mean-Field Type. Dyn Games Appl (2013) 3:537552.

[2] A. BENSOUSSAN AND J. FREHSE. Control and Nash Games with Mean Field Effect. Chinese Annal Math. Series B. 34B(2), 2013, 161192.

[3] A. BENSOUSSAN AND J. FREHSE AND S.C.P. YAM Mean-field Games and Mean-field Type Control. Springer Briefs in Math. 2014.

[4] A. BENSOUSSAN AND J. FREHSE AND S.C.P. YAM. The Master Equation in Mean-Field Theory. (To appear in *Asymptotic Analysis*).

[5] W.H. FLEMING AND H.M. SONER. *Controlled Markov Process and Viscosity Solutions*. Springer 2006 (2nd ed.)

[6] F. HECHT New development in freefem++, J. Numer. Math. vol 20, no 3-4, pp 251-265 (2012).

[7] J. GARNIER AND G. PAPANICOLAU AND TZU-WEI YANG. Large deviations for a mean field model of systemic risk, SIAM J. Finan. Math., 4(1), 151184, 2013.

[8] J.M. LASRY AND P.L. LIONS. Mean-field games. Japan. J. Math. 2, 229-260 (2007)

[9] P.L. LIONS. Mean-field games. Cours au Collège de France (2007-2008) http://www.college-de-france.fr/site/pierre-louis-lions/course-2007-2008_1.htm

[10] A. NEUFELD AND M. NUTZ. Nonlinear Lévy processes and their Characteristics. arXiv:1401.7253v1.

[11] B. ØKSENDAL AND A. SULEM. *Applied Stochastic Control of Jump Diffusions*. Sprinver 2005.

[12] MIN SHEN AND G. TURINICI. Lqudty Generated By Heterogeneous Belefs and Costly Estmatons. Vol 7, No 2, pp. 349361, June 2012.

[13] N. TOUZI. *Optimal Stochastic Control, Stochastic Target Problems and Backard SDE*. Field Inst. Monographs 29, Springer 2013.

[14] JIONGMIN YONG AND XUN YU ZHOU. *Stochastic Control*. Applications of Math. series vol 43. Springer 1999.