title: Optimal control theoretic value function learning
Creators: name: Layeghi, Daniel
Publisher: name: The University of Edinburgh
Year: 2025
Subjects: subject: optimal control
subject: decision-making theory
subject: Differential Dynamic Programming
subject: second-order derivatives
subject: zero-order updates
subject: Policy Gradient methods
subject: Actor-Critic algorithms
subject: value-function-based policies
subject: hybrid methods

description: Generating behaviours to complete complex tasks can be viewed under the paradigm of controlling dynamical systems. To solve such tasks, most approaches fall under two paradigms: Reinforcement Learning (RL) and Optimal Control (OC) theoretic approaches. OC theoretic solutions are mostly local and can only provide global controllers for special cases. As a result of this, local solutions become trajectories. Synthesising these trajectories in the deterministic setting is formalised under the calculus of variations. This paradigm imposes strict constraints on the objective landscape: differentiability and continuity. In the stochastic setting, OC theoretic solutions have been proposed to remove the burden of these constraints and infer the optimal trajectory through sampling. RL has very similar theoretical groundings but diverges significantly in its approach. For example, RL parametersises value and/or policy functions instead of trajectories, allowing generalisation to new initial conditions. Additionally, in its model-free setting, which is our focus in this thesis, there is no need for constraints such as differentiability on the objective. RL is capable of estimating the gradients via sampling. However, these gradient estimates come at the high price of noisy solutions and slow convergence. To this end, defining methods that can leverage the best of both approaches is desirable. Our thesis aims to derive methods that greatly remove the burden of cost function design on the user while enabling generalisation by efficiently learning approximate global controllers. As our initial attempt at this formalisation, we introduce a local method that combines the efficiency of derivative-based OC-theoretic approaches with the flexibility of local solutions based on sampling. To this end, we propose a hybrid approach that aims for consensus between the derivative-based solution of iterative Linear Quadratic Regulator (iLQR) and the sampling-based solution of Path Integral (PI) control. We define an objective that enables us to sample when derivatives vanish and follow optimised trajectories when derivatives arise. We use the Kullback Leibler (KL) control interpretation of PI control to formulate an inference problem that computes the optimal controls constrained by an adaptive distribution defined by the solution of iLQR. Our results show better convergence on manipulation and obstacle avoidance tasks than sampling strategy, path integral control and gradient-based strategy iLQR. In the second segment of this thesis, we evaluate the widely used RL algorithms and its core gradient estimation machinery, policy gradients, without the typical convergence strategies. Our results are obtained on simple nonlinear continuous control problems. We show that RL still requires extensive tuning, even on simple nonlinear problems and the flexibility gained by zeroth-order derivative estimation is paid for by hyperparameter tuning. In turn, we propose an OC-theoretic approach based on Bellman optimality that leverages differentiable dynamics and first-order gradients. Our approach can learn approximate time-varying value functions and robustly converge with minimal tuning. We further verify the ability of our method by relaxing the objective and obtaining first-order approximations of time-varying Lyapunov constraints. We further verify our approach by satisfying this first-order constraint over a compact set of initial conditions. When comparing our method to Soft Actor-Critic (SAC) and Proximal Policy Optimisation (PPO) we show faster convergence and outperform PPO and SAC in task cost by at least 2 and 4 orders of magnitude, respectively. In the third part of the thesis, we combine our findings from the previous sections to create a method that can handle discontinuities using stochasticity, ensure convergence with differentiability, and generalise with function parameterisation. To achieve this, we approach the problem using stochastic optimal control and robustness. We use the stochastic Hamilton-Jacobi-Bellman equation, differentiable dynamics, and the natural smoothing induced by stochastic first-order gradients. Our results demonstrate that the policies based on learned value functions outperform SAC and PPO in task cost by factors of up to 1076.02 and 8, respectively. Moreover, we observe that adding noise to the dynamics smoothens the curvature of the value function. This effect is especially noticeable in our obstacle navigation task with discontinuous dynamics and costs, where the value functions learned under noisier dynamics follow wider paths around obstacles, making them more robust. Finally, we show that our learned value functions can also be integrated into local methods, reducing their effective search horizon by a factor of 15.