Title: Optimality Conditions for Model Predictive Control: Rethinking | Predictive Model Design
Domain: Math

Question: How can predictive models be designed to guarantee closed-loop optimality in Model Predictive Control, beyond traditional data-fitting approaches? In particular, under what conditions do model properties ensure optimal performance, how can counterintuitive yet optimal structures be justified, and what are the implications for stochastic and economic MPC as well as system identification? Finally, how can this theory be connected to infinite-horizon Markov decision processes to yield rigorous proofs and practical design guidelines?

Background: 1. **System Dynamics and Cost** | | * State evolution: $s_+ \sim \rho(\cdot \mid s,a)$. | * Performance metric: | | $$ | J(\pi) = \mathbb{E}_\rho^\pi \Big[\sum_{k=0}^\infty \gamma^k L(s_k,a_k)\Big], | $$ | | with discount factor $\gamma \in (0,1)$ ensuring boundedness. | | 2. **Bellman Optimality Framework** | | * Optimal action-value function: | $Q^*(s,a) = L(s,a) + \gamma \,\mathbb{E}_\rho[V^*(s_+)\mid s,a]$. | * Value function: $V^*(s) = \min_a Q^*(s,a)$. | * Optimal policy: $\pi^*(s) \in \arg\min_a Q^*(s,a)$. | | 3. **MPC Formulation** | | * Finite-horizon optimization with prediction horizon $N$: | | $$ | V_{\text{MPC}}(s_k) = \min_{\hat s,u}\ \gamma^N T(\hat s_N) + \sum_{i=0}^{N-1} \gamma^i L(\hat s_i,u_i) | $$ | | subject to: | | * State dynamics: $\hat s_{i+1} = f(\hat s_i,u_i)$. | * Constraints: $h(\hat s_i,u_i) \le 0$. | * Initial and terminal conditions: $\hat s_0 = s_k,\ \hat s_N \in T$. | | 4. **Conventional Predictive Models** | | * Deterministic form: $f(s,a) = \mathbb{E}_\rho[s_+\mid s,a]$. | * Other forms: maximum likelihood, Bayesian, or data-driven models. | | 5. **Key Challenge and Motivation** | | * Accurate prediction ≠ guaranteed closed-loop optimality. | * Recent insights: optimal MPC can still be achieved via model adjustments (e.g. stage cost corrections). | * Fundamental question: **what properties must predictive models satisfy to guarantee closed-loop optimality?** | | |

Definitions: 1. **System Dynamics:** | | * True system: $s_+ \sim \rho(\cdot \mid s,a)$ | * Model-based system: $\hat{s}_+ \sim \hat{\rho}(\cdot \mid s,a)$ | * Key: $\rho$ is the true transition probability, $\hat{\rho}$ is the model’s approximation. | | 2. **Optimal Control:** | | * True optimal policy: $\pi^*(s) \in \arg\min_a Q^*(s,a)$ | * Model-based optimal policy: $\hat{\pi}^*(s) \in \arg\min_a \hat{Q}^*(s,a)$ | * Core objective: Ensure model’s policy matches the true policy. | | 3. **Stability Assumption:** | | * Assumption : Ensure value functions remain bounded along predicted trajectories. | * The set $\Omega$ where $\mathbb{E}_{\hat{\rho}}[V_\lambda^*(\hat{s}_k^{\pi^*})] < \infty$ must be non-empty. | | 4. **Value Function Modification:** | | * $\hat{V}_\lambda(s) = \hat{V}(s) + \lambda(s)$ | * The goal: Modify value functions without changing action ranking. | * Key modification: $\Lambda(s,a) = \lambda(s) - \gamma \mathbb{E}_{\rho}[\lambda(s_+)\mid s,a]$. | | 5. **Advantage Functions:** | | * True advantage: $A^*(s,a) = Q^*(s,a) - V^*(s)$ | * Modified advantage: $\hat{A}_\lambda^*(s,a) = \hat{Q}_\lambda^*(s,a) - \hat{V}_\lambda^*(s)$ | * Main property: $\min_a \hat{A}_\lambda^*(s,a) = \min_a A^*(s,a) = 0$ | | 6. **MPC-specific Framework:** | | * Deterministic model: $\hat{s}_+ = f(s,a)$ | * MPC optimization: | | $$ | V_{\text{MPC}}(s_k) = \min_{\hat{s}, u} \left[\gamma^N T(\hat{s}_N) + \sum_{i=0}^{N-1} \gamma^i L(\hat{s}_i, u_i) \right] | $$ | * Key: $f(s,a)$ is the deterministic predictive model; terminal cost $T$ and constraints on states/actions are crucial.

Methodology: 1. **Necessary & Sufficient Condition** | * The core goal is to ensure that the model-based policy $\hat{\pi}^*$ matches the true policy $\pi^*$. | * Recall that the advantage functions are defined as: | | $$ | A^*(s,a) = Q^*(s,a) - V^*(s), \quad \hat{A}_\lambda^*(s,a) = \hat{Q}_\lambda^*(s,a) - \hat{V}_\lambda^*(s). | $$ | | 2. **Bellman Recursion** | | * The true Q-function and the model-based Q-function are given by: | | $$ | Q^*(s,a) = L(s,a) + \gamma \mathbb{E}_{\rho}[V^*(s_+)\mid s,a], \quad \hat{Q}_\lambda^*(s,a) = L(s,a) + \Lambda(s,a) + \gamma \mathbb{E}_{\hat{\rho}}[V_\lambda^*(\hat{s}_+)\mid s,a]. | $$ | * These define the system dynamics under both the true and model-based dynamics. | | 3. **Sandwich Inequality** | | * The model’s advantage function must satisfy a sandwich inequality: | | $$ | \alpha(A^*(s,a)) \le \hat{A}_\lambda^*(s,a) \le \beta(A^*(s,a)), | $$ | | where $\alpha$ and $\beta$ are class-$\mathcal{K}$ functions. This ensures that the model preserves the action ranking defined by the true system. | | 4. **Policy Equivalence** | | * The sandwich inequality ensures that $\hat{\pi}^* = \pi^*$, meaning the optimal policy in the model matches the optimal policy in the true system. | | 5. **MPC-Specific Framework** | | * For MPC, the predictive model is deterministic: $\hat{s}_+ = f(s,a)$. | * The optimization problem is formulated as: | | $$ | V_{\text{MPC}}(s_k) = \min_{\hat{s}, u} \left[\gamma^N T(\hat{s}_N) + \sum_{i=0}^{N-1} \gamma^i L(\hat{s}_i, u_i)\right], | $$ | | subject to state dynamics and constraints. | * The terminal cost function $T$ and constraints on states and actions are critical in ensuring the model behaves optimally. | | 6. **Numerical Solution** | | * To solve for the optimal policy, iterate on value functions and use the Bellman equations to extract $\pi^*(s)$ from the optimal action-value functions. | | |

Answer: ## 0. Preliminaries and assumptions | | * **True dynamics (discounted MDP):** | | $$ | s' \sim \rho(\cdot\mid s,a),\quad | J(\pi)=\mathbb{E}_\rho^\pi\!\Big[\sum_{k=0}^\infty \gamma^k L(s_k,a_k)\Big],\quad | \pi^*\in\arg\min_\pi J(\pi),\quad \gamma\in(0,1]. | $$ | * **Predictive model dynamics:** | | $$ | s' \sim \hat\rho(\cdot\mid s,a), | $$ | | with optimal value/action-value $(\hat V,\hat Q)$ and optimal policy $\hat\pi^*$. | * **True Bellman relations:** | | $$ | Q(s,a)=L(s,a)+\gamma\,\mathbb{E}_\rho[V(s')\mid s,a],\qquad V(s)=\min_a Q(s,a). | $$ | * **Advantages:** | | $$ | A(s,a):=Q(s,a)-V(s)\ge 0,\quad A(s,a)=0 \iff a\in\pi^*(s). | $$ | * **λ-shift (policy-invariant baseline alignment):** for any bounded $\lambda:S\to\mathbb{R}$, | | $$ | \hat V_\lambda(s):=\hat V(s)+\lambda(s),\quad \hat Q_\lambda(s,a):=\hat Q(s,a)+\lambda(s), | $$ | | hence $\arg\min_a\hat Q_\lambda(s,a)=\arg\min_a\hat Q(s,a)$. Define $\hat A_\lambda:=\hat Q_\lambda-\hat V_\lambda$. | * **Boundedness/stability assumption:** there exists a nonempty compact set $\Omega\subseteq S$ such that the value functions under interest are finite on trajectories that remain in $\Omega$. (This prevents pathological infinities and ensures minima exist.) | | We also use: a **class-$\mathcal K$** function $\alpha:\mathbb{R}_{\ge 0}\to\mathbb{R}_{\ge 0}$ is strictly increasing with $\alpha(0)=0$. | | --- | | ## 1. Two key lemmas (with proofs) | | ### Lemma 1 (lower bound ↔ argmin inclusion). | | There exists a class-$\mathcal K$ function $\alpha$ such that | | $$ | \hat A_\lambda(s,a)\ \ge\ \alpha\!\big(A(s,a)\big)\quad \forall(s,a) | \tag{L1-ineq} | $$ | | **iff** | | $$ | \arg\min_a \hat A_\lambda(s,a)\ \subseteq\ \arg\min_a A(s,a)\quad \forall s. | \tag{L1-incl} | $$ | | **Proof.** | | * **(Necessity via contrapositive).** Suppose $(\text{L1-incl})$ fails at some state $\bar s$: there exists $\bar a$ with | | $$ | \bar a\in\arg\min_a\hat A_\lambda(\bar s,a)\quad\text{but}\quad \bar a\notin\arg\min_a A(\bar s,a). | $$ | | Since $\min_a \hat A_\lambda(\bar s,a)=0$ and $\min_a A(\bar s,a)=0$, we have | | $$ | \hat A_\lambda(\bar s,\bar a)=0,\quad A(\bar s,\bar a)>0. | $$ | | For any class-$\mathcal K$ $\alpha$, $\alpha(A(\bar s,\bar a))>0$. Hence $\hat A_\lambda(\bar s,\bar a)\ge \alpha(A(\bar s,\bar a))$ is violated. Therefore $\neg(\text{L1-incl})\Rightarrow \neg(\text{L1-ineq})$. | | * **(Sufficiency).** Assume $(\text{L1-incl})$ holds. Define, for $x\in[0,\bar A]$ with $\bar A:=\max_{s,a}A(s,a)$ over $\Omega$, | | $$ | \alpha_0(x)\ :=\ \min_{(s,a)}\ \hat A_\lambda(s,a)\quad \text{s.t.}\quad A(s,a)\ \ge\ x. | $$ | | Properties: | | 1. $\alpha_0(0)=0$: feasibility includes minimizers of $\hat A_\lambda$, whose value is 0. | 2. $\alpha_0(x)>0$ for $x>0$: otherwise, $\exists (s,a)$ with $A(s,a)\ge x>0$ but $\hat A_\lambda(s,a)=0$, contradicting $(\text{L1-incl})$. | 3. $\alpha_0$ is nondecreasing: feasible set shrinks as $x$ grows. | 4. For any $(s,a)$, taking $x=A(s,a)$ makes $(s,a)$ feasible, hence | | $$ | \hat A_\lambda(s,a)\ \ge\ \alpha_0(A(s,a)). | $$ | | Finally, let $\alpha$ be any strictly increasing class-$\mathcal K$ minorant of $\alpha_0$ (e.g., $\alpha(x)=\inf_{y\ge x}\alpha_0(y)+\varepsilon x$ with $\varepsilon>0$ small). Then $\alpha\le \alpha_0$, $\alpha(0)=0$, and strictly increasing, so $(\text{L1-ineq})$ holds. ∎ | | --- | | ### Lemma 2 (upper bound ↔ reverse argmin inclusion). | | There exists a class-$\mathcal K$ function $\beta$ such that | | $$ | \hat A_\lambda(s,a)\ \le\ \beta\!\big(A(s,a)\big)\quad \forall(s,a) | \tag{L2-ineq} | $$ | | **iff** | | $$ | \arg\min_a A(s,a)\ \subseteq\ \arg\min_a \hat A_\lambda(s,a)\quad \forall s. | \tag{L2-incl} | $$ | | **Proof.** | | We reduce to Lemma 1 by swapping roles. Define $\tilde A:=\hat A_\lambda$ and $\hat{\tilde A}:=A$. | By Lemma 1 applied to $(\tilde A,\hat{\tilde A})$, “$\exists$ class-$\mathcal K$ $\tilde\alpha$ with $A(s,a)\ge \tilde\alpha(\hat A_\lambda(s,a))$” is equivalent to $(\text{L2-incl})$. | | Now take a (right) generalized inverse of $\tilde\alpha$: | | $$ | \beta(x):=\sup\{y\ge 0:\ \tilde\alpha(y)\le x\}. | $$ | | Then $\beta$ is class-$\mathcal K$ and $\tilde\alpha(\beta(x))\le x$. | From $A\ge \tilde\alpha(\hat A_\lambda)$ we get $\beta(A)\ge \beta(\tilde\alpha(\hat A_\lambda))\ge \hat A_\lambda$, i.e. $(\text{L2-ineq})$. | Conversely, if $(\text{L2-ineq})$ holds, the same construction with roles swapped yields $(\text{L2-incl})$. ∎ | | --- | | ## 2. Sandwich condition ⇔ policy equality | | Combining Lemma 1 and Lemma 2 gives the **sandwich inequality** | | $$ | \boxed{\ \alpha\!\big(A(s,a)\big)\ \le\ \hat A_\lambda(s,a)\ \le\ \beta\!\big(A(s,a)\big)\quad \forall(s,a)\ } | \tag{S} | $$ | | for some class-$\mathcal K$ functions $\alpha,\beta$. | | * From Lemma 1, (S) implies $\arg\min_a \hat A_\lambda(s,a)\subseteq \arg\min_a A(s,a)$. | * From Lemma 2, (S) implies $\arg\min_a A(s,a)\subseteq \arg\min_a \hat A_\lambda(s,a)$. | | Hence $\arg\min_a \hat A_\lambda(s,a)=\arg\min_a A(s,a)$ for every state, i.e. | | $$ | \boxed{\ \hat\pi^*=\pi^*\ }\quad\text{(policy equality).} | $$ | | Conversely, if $\hat\pi^*=\pi^*$, the inclusions hold both ways, and the constructions in the proofs produce class-$\mathcal K$ $\alpha,\beta$ giving (S). Thus (S) is **necessary and sufficient** for closed-loop optimality of the predictive model (up to the boundedness assumption). | | --- | | ## 3. Bellman-form expansion (one-step condition) | | Using $A(s,a)=L(s,a)+\gamma \mathbb{E}_\rho[V(s')|s,a]-V(s)$ and defining | | $$ | \Lambda(s,a):=\lambda(s)\ -\ \gamma\,\mathbb{E}_\rho[\lambda(s')\mid s,a], | $$ | | the sandwich inequality (S) is equivalent to the **one-step Bellman sandwich** | | $$ | \boxed{\ | \begin{aligned} | \beta\!\Big(L(s,a)+\gamma \mathbb{E}_\rho[V(s')|s,a]-V(s)\Big)\ &\ge\ | L(s,a)+\Lambda(s,a)+\gamma \mathbb{E}_{\hat\rho}[V(s')|s,a]-V(s)\\ | &\ge\ \alpha\!\Big(L(s,a)+\gamma \mathbb{E}_\rho[V(s')|s,a]-V(s)\Big), | \end{aligned} | } | \tag{B} | $$ | | for all $(s,a)$ in the region of interest. | Intuition: the model’s one-step Q-update must lie between monotone transforms of the true one-step update; multi-step fidelity is **not** required. | | --- | | ## 4. Deterministic MPC specialization | | For deterministic predictive models $s'=f(s,a)$, the middle term in (B) becomes | | $$ | L(s,a)+\Lambda(s,a)+\gamma\,V\!\big(f(s,a)\big)-V(s), | $$ | | so (B) reduces to | | $$ | \boxed{\ | \beta\!\Big(L(s,a)+\gamma \mathbb{E}_\rho[V(s')|s,a]-V(s)\Big)\ \ge\ | L(s,a)+\Lambda(s,a)+\gamma V(f(s,a))-V(s)\ \ge\ | \alpha\!\Big(L(s,a)+\gamma \mathbb{E}_\rho[V(s')|s,a]-V(s)\Big). | } | \tag{MPC} | $$ | | This is the **explicit closed-form condition** under which a deterministic MPC model yields the true optimal policy. | | --- | | ## 5. A simple sufficient case (constant bias) | | If $\alpha=\beta=\mathrm{id}$ and there exists a constant $\Delta$ with | | $$ | \mathbb{E}_\rho[V(s')|s,a]-\mathbb{E}_{\hat\rho}[V(s')|s,a]=\Delta\quad \forall(s,a), | $$ | | then | | $$ | \hat Q_\lambda(s,a)=Q(s,a)-\gamma \Delta, | $$ | | i.e., model and true $Q$ differ by a state-independent constant, hence induce the **same action ranking** and the same optimal policy. | | --- | | ## 6. Practical implications for model/MPC design | | 1. **Performance-oriented identification.** Fit the model to **Q/advantage alignment** (with λ-learning), not to raw dynamics error: | | $$ | \min_\theta\ \mathbb{E}\big[\|Q(s,a)-\hat Q_{\lambda,\theta}(s,a)\|\big]. | $$ | 2. **Stochastic & economic MPC.** One-step alignment is enough; λ acts as a terminal/steady-state baseline to match long-run indices. | 3. **Why data-fit can fail.** Trajectory prediction accuracy does not guarantee the sandwich (B); advantage ordering can still be wrong. | | --- | | ## 7. Logic chain | | 1. Define true/model MDPs and advantages. | 2. Introduce λ-shift (policy-invariant baseline alignment). | 3. **Lemma 1**: lower bound $\Rightarrow$ $\arg\min \hat A_\lambda \subseteq \arg\min A$. | 4. **Lemma 2**: upper bound $\Rightarrow$ $\arg\min A \subseteq \arg\min \hat A_\lambda$. | 5. Combine $\Rightarrow$ sandwich inequality $\alpha(A)\le \hat A_\lambda\le \beta(A)$. | 6. Sandwich $\Leftrightarrow$ policy equality $\hat\pi^*=\pi^*$. | 7. Expand to Bellman-form (one-step condition); specialize to deterministic MPC. | 8. Note a simple sufficient case (constant bias) and practical design takeaways. | | |

Checklist: 1. **MDP Framework:** | | * Correctly defines the system dynamics and transition functions for both true and model-based systems. | * Defines optimal policies, action-value functions, and value functions for both systems. | | 2. **Optimality Conditions:** | | * Provides the necessary and sufficient conditions for model optimality through advantage functions. | * Verifies the stability assumption ensuring bounded value functions along optimal trajectories. | | 3. **Value Function Modification:** | | * Shows how modifying the value function $V_\lambda(s) = V(s) + \lambda(s)$ preserves policy optimality while allowing adjustments to the model. | | 4. **Advantage Functions:** | | * Defines and compares true and modified advantage functions. | * Ensures consistency in action ranking by proving the equivalence of the advantage functions under the modified value function. | | 5. **Key Lemmas:** | | * Proves the existence of a class-K function $\alpha$ such that the modified advantage is bounded below by the true advantage. | * Proves the existence of a class-K function $\beta$ such that the true advantage is bounded below by the modified advantage. | | 6. **Optimality Theorem:** | | * Establishes the necessary and sufficient conditions for model-based optimality. | * Demonstrates the connection between the modified Q-values and the true Q-values under the optimal policy. | | 7. **MPC Framework:** | | * Extends the theory to deterministic MPC, showing the relationship between the predictive model and the true system. | * Defines the MPC optimization problem with terminal costs and constraints. | | 8. **Application to MPC:** | | * Explains why traditional data-fitting models may fail to guarantee optimality outside specific problem classes. | * Verifies that the optimality conditions apply to short-term predictions rather than full trajectory predictions. | | 9. **Implications for Practical MPC:** | | * Demonstrates how the theoretical framework can be applied to improve practical MPC implementations. | * Highlights the difference between traditional prediction-based approaches and the proposed method in achieving optimality. | | |