ADARL: Adaptive Low-Rank Structures for Robust Policy Learning under Uncertainty

## Abstract

Abstract Robust reinforcement learning (Robust RL) seeks to handle epistemic uncertainty in environment dynamics, but existing approaches often rely on nested min–max optimization, which is computationally expensive and yields overly conservative policies. We propose Adaptive Rank Representation (AdaRL) , a bi-level optimization framework that improves robustness by aligning policy complexity with the intrinsic dimension of the task. At the lower level, AdaRL performs policy optimization under fixed-rank constraints with dynamics sampled from a Wasserstein ball around a centroid model. At the upper level, it adaptively adjusts the rank to balance the bias–variance trade-off, projecting policy parameters onto a low-rank manifold. This design avoids solving adversarial worst-case dynamics while ensuring robustness without over-parameterization. Empirical results on MuJoCo continuous control benchmarks demonstrate that AdaRL not only consistently outperforms fixed-rank baselines (e.g., SAC) and state-of-the-art robust RL methods (e.g., RNAC, Parseval), but also converges toward the intrinsic rank of the underlying tasks. These results highlight that adaptive low-rank policy representations provide an efficient and principled alternative for robust RL under model uncertainty.

## 1 Introduction

The goal of a reinforcement learning (RL) agent is to learn a policy that maximizes its expected discounted cumulative reward (Sutton et al., 1998). Recent advances have enabled RL agents to master complex games and robotic control tasks in both simulation and the real world (Mnih et al., 2015; Silver et al., 2017).
However, policies that perform well in such controlled settings often fail to transfer to practice, where transition dynamics are rarely fixed and may shift due to modeling inaccuracies (Lanzani, 2025), external disturbances, or changing conditions (Pattanaik et al., 2017).
To address this gap, robust reinforcement learning (robust RL) (Zhou et al., 1996) formalizes uncertainty by considering a set of possible transition kernels and casting policy optimization as a minmax problem: the agent seeks a policy that maximizes expected return under the worst-case dynamics.
This formulation reduces the sensitivity of RL to model misspecification and aims to produce policies that stay reliable when the environment differs from training.

Robust RL provide a principled framework to handle model uncertainty by optimizing for policies that perform well under the worst-case transition models within a prescribed uncertainty set (Iyengar, 2005; Wiesemann et al., 2013). Classical solutions extend Bellman’s principle to robust settings (Satia & Lave Jr, 1973), while more recent work has focused on robust policy learning via model-based planning (Clavier et al., 2023) or online interaction with a nominal environment (Wang & Zou, 2021). Despite these advances, robust RL faces severe scalability issues when applied to continuous and high-dimensional domains. In particular, updating the robust value function via the robust Bellman operator requires solving a nested inner-loop optimization at every step, i.e., identifying the worst-case transition, which becomes computationally prohibitive as the state and action spaces grow or when the uncertainty set is large or unbounded (Wang & Zou, 2022). Moreover, existing approaches often assume access to oracle solvers or rely on fixed uncertainty sets that may yield overly conservative policies (Mannor et al., 2012, 2016; Xu & Mannor, 2012). Beyond these computational bottlenecks, another key challenge lies in function approximation. Existing analyzes are mostly restricted to the tabular setting, which cannot achieve parameterized neural network approximations to the optimal solution of the robust Bellman equation. Our approach, in contrast, explicitly accommodates parameterization, thereby enabling robust generalization in high-dimensional environments.

In this work, we introduce an alternative perspective to overcome the limitations of existing robust RL approaches. Instead of directly tackling worst-case dynamics through nested min–max optimization, we enhance robustness by controlling over-parameterization and improving the generalization of fixed-rank policy and value models under perturbed transition dynamics. A key insight (Li et al., 2018)) is that the effective complexity of a policy should *match* the intrinsic dimension of the task under epistemic uncertainty—uncertainty in environment dynamics arising from limited data or partial observability, which is prevalent in real-world domains such as robotics, control, environmental policy, and economics (Nagami & Schwager, 2023; Zhou et al., 1996; Lemoine & Traeger, 2014; Hansen & Sargent, 2008). Building on this idea, we propose a new algorithm that jointly learns both the policy models and its rank, formulated as a bi-level optimization problem: the lower-level learns a policy under low-rank constraints, while the upper-level adapts the rank to balance robustness and expressiveness.

Figure: Figure 1: Comparison of robust RL and the proposed AdaRL framework. Robust RL relies on repeatedly solving a nested min–max problem, while AdaRL formulates training as a bi-level optimization that alternates between policy optimization and adaptive rank adjustment to balance the bias–variance trade-off under epistemic uncertainty.
Refer to caption: x1.png

This perspective aligns with and extends prior work on exploiting low-rank structures in reinforcement learning. In the *model-based* setting, algorithms for joint feature and policy learning have been developed when the dynamics admit a low-rank decomposition (Agarwal et al., 2020; Bose et al., 2024). In the *model-free* setting, Jiang et al. (2017) introduced the concept of *Bellman rank* to capture the intrinsic complexity of value function approximation, and subsequent work (Modi et al., 2021, 2024; Yang et al., 2020) sought to encourage small Bellman rank during training. More recently, Tiwari et al. (2025) showed that wide two-layer neural networks yield reachable states concentrated on a low-dimensional manifold whose dimension scales with the action space. Overall, these works show that low-rank structures can improve performance in *standard RL settings*. Yet, no existing approach provides a practical algorithm for leveraging low-rank advantages under *model uncertainty*, and it remains inherently difficult to determine a suitable rank for parameterizing policy models in uncertain environments.

Our Contribution. We propose Adaptive Rank Representation for Reinforcement Learning (AdaRL, Figure [1](https://arxiv.org/html/2510.11899v1#S1.F1)), an adaptive framework that integrates conservatism into the learning process in MDPs with epistemic uncertainty. The algorithm alternates between standard policy optimization under a fixed rank and an adaptive step that adjusts the rank to balance robustness and expressiveness. Our main contributions are:

- 1.
We provide a theoretical analysis of the bias–variance trade-off in entropy-regularized RL with linear parameterization under epistemic uncertainty, showing that low-rank representations can reduce variance in the presence of model uncertainty (Section [3](https://arxiv.org/html/2510.11899v1#S3), Theorem. [1](https://arxiv.org/html/2510.11899v1#Thmtheorem1)).
- 2.
We formulate policy rank selection as a bi-level optimization problem and present the AdaRL algorithm, which adaptively adjusts policy rank for robust learning (Section [4](https://arxiv.org/html/2510.11899v1#S4)).
- 3.
We empirically evaluate AdaRL on standard MuJoCo continuous control benchmarks, demonstrating consistent improvements over robust baselines (e.g., RNAC Zhou et al. (2023), Parseval Chung et al. (2024)) and non-robust methods such as SAC (Haarnoja et al., 2018) and Tiwari et al. (2025) (see Section [5](https://arxiv.org/html/2510.11899v1#S5)).

## 2 Preliminary and Related Works

### 2.1 Notation

Markov Decision Process. A Markov decision process (MDP) is represented by the tuple $(\mathcal{S},\mathcal{A},\mathcal{P},\rho,r,\gamma)$ wherein $\mathcal{S}$ is the state space $\mathcal{A}$ is the action space (with both $\mathcal{S}\subset\mathbb{R}^{n},\mathcal{A}\subset\mathbb{R}^{m}$ assumed compact), $P_{s,a}\in\Delta_{\mathcal{S}}$, is the transition kernel for $a\in\mathcal{A},s\in\mathcal{S}$. (where $\Delta_{\mathcal{S}}$ denotes the space of probability measures with support $\mathcal{S}$), $\rho(\cdot)$ is the initial state distribution, $R:\mathcal{S}\times\mathcal{A}\to\mathbb{R}$ is the the reward function $\gamma\in[0,1)$ is the discount factor. Given $s\in\mathcal{S}$, a policy $\pi$ is a map $\pi(\cdot|s):\mathcal{S}\to\Delta_{\mathcal{A}}$, where $\Delta_{\mathcal{A}}$ denotes the space of probability measures with support $\mathcal{A}$.

Epistemic Uncertainty in State Dynamics.
To model uncertainty in the environment dynamics, we introduce an ambiguity set of possible transition kernels:

$$ $\mathcal{P}_{s,a}:=\{P_{s,a}\in\Delta_{\mathcal{S}}\;\mid\;W(\hat{P}^{\circ}_{s,a},P_{s,a})\leq\epsilon\},$ $$

where $\hat{P}^{\circ}_{s,a}$ is a reference transition kernel (e.g., a maximum likelihood estimator obtained from a finite demonstration dataset), $W(\hat{P}^{\circ}_{s,a},P_{s,a})$ denotes the Wasserstein distance (Villani et al., 2008), and $\epsilon>0$ is the uncertainty radius. We refer to $P^{\circ}_{s,a}$ as the centroid of the uncertainty set, representing the true but unobserved transition kernel that governs the system dynamics. Throughout, we assume that epistemic uncertainty is well captured by the Wasserstein ball (Mohajerin Esfahani & Kuhn, 2018), i.e., $P^{\circ}_{s,a}\in\mathcal{P}_{s,a}$ for all $(s,a)\in\mathcal{S}\times\mathcal{A}$.

Singular Value Decomposition (SVD) Let $\theta\in\mathbb{R}^{d_{1}\times d_{2}}$. A thin singular value decomposition (SVD) is given by $\theta=\mathbf{U}\boldsymbol{\Sigma}\mathbf{V}^{\top}$, where $\mathbf{U}$ is a $d_{1}\times r$ matrix with orthogonal columns, that is, an element of the Stiefel manifold (Chakraborty & Vemuri, 2019; Atiyah & Todd, 1960)

$$ $\mathrm{St}(r,d_{1})=\{\mathbf{U}\in\mathbb{R}^{d_{1}\times r}:\mathbf{U}^{T}\mathbf{U}=\mathbf{I}\},$ $$

$\boldsymbol{\Sigma}$ is a $r\times r$ diagonal matrix with positive entries $\sigma_{1}\geq\sigma_{2}\geq\cdots\sigma_{r}>0$ (referred to as singular values) and $\mathbf{V}\in\mathrm{St}(r,d_{2})$.
The singular value decomposition exists for any matrix $\theta\in\mathbb{R}^{d_{1}\times d_{2}}$. We refer to a truncated SVD whenever $r<\mbox{rank}(\theta)$.

### 2.2 Robust Reinforcement Learning

In MDPs, the system dynamics $P$ is usually assumed to be constant over time. However, in the real world, it is subject to perturbations that can significantly impact performance in deployment (Zhang et al., 2023; Moos et al., 2022). Robust MDPs provide a theoretical framework for taking this uncertainty into account, taking $P$ as not fixed but chosen adversarially from an uncertainty set $\mathcal{P}$ (Iyengar, 2005; Nilim & El Ghaoui, 2005).
where $\mathcal{P}$ denotes a set of plausible transition models known as the uncertainty set. The objective of robust RL is to find a policy that performs well under the worst-case dynamics within this set. Formally, the robust objective $\mathcal{J}_{\mathcal{P},\pi}$ is defined as:

$$ $\mathcal{J}_{\rm robust}(\pi)=\max_{\pi}\min_{P\in\mathcal{P}}\mathbb{E}_{P,\pi}\Big[\sum_{t\geq 0}\gamma^{t}R(s_{t},a_{t})\,\Big|s_{0}\sim\rho_{0}\Big]$ (1) $$

The optimal policy $\pi_{\mathcal{P}}^{*}$ is defined as the solution to the outer-loop problem, which maximizes $\mathcal{J}_{\mathcal{P},\pi}$ by accounting for the worst-case transition model at each time step. This leads to the inner-loop problem of identifying the worst-case dynamics, for which several approaches have been developed, including value iteration (Nilim & El Ghaoui, 2005; Iyengar, 2005; Wiesemann et al., 2013; Grand-Clément & Kroer, 2021; Kumar et al., 2023a), policy iteration (Kumar et al., 2022; Badrinath & Kalathil, 2021), and policy gradient methods (Li et al., 2022; Wang & Zou, 2022; Wang et al., 2023; Kumar et al., 2023b). However, the problem remains NP-hard for general uncertainty sets, and optimal policies may even be non-stationary (Wiesemann et al., 2013). Most existing methods sidestep this difficulty by assuming that the inner-loop optimization can be solved efficiently—a reasonable assumption in tabular settings with small uncertainty sets, where one can exhaustively evaluate all transition kernels $P\in\mathcal{P}$. Yet, when the uncertainty set is continuous, the inner-loop problem becomes substantially more challenging and computationally expensive. To address this challenge, Zhou et al. (2023); Gadot et al. (2024) propose the RNAC and EWoK algorithms, which rely on sampling-based techniques to estimate value functions under worst-case dynamics. Although theoretically sound, these methods require drawing multiple next states for each state-action pair, leading to high sample complexity and considerable computational overhead.

### 2.3 Reinforcement Learning with low rank structure

Another direction of research to address this uncertainty is to take advantage of low-rank structures in dynamics. In many stochastic control tasks, the transition dynamics admit a low-rank decomposition over a finite set of state-action features (Rozada et al., 2024, 2021; Yang et al., 2019). For example, Tiwari et al. (2025) show that under suitable assumptions, the set of attainable states lies on a low-dimensional manifold. In fixed environments, the dimension of this manifold grows only linearly with the size of the action space and is independent of the state-space dimension. Building on this observation, they employ a $(2d_{a}+1)$-dimensional low-rank manifold and apply sparse reinforcement learning methods to solve MuJoCo control tasks. More generally, low-rank structure can be imposed either on the transition kernel or directly on the optimal action-value function $Q^{*}$, and empirical evidence suggests that $Q^{*}$ and near-optimal Q-functions in common stochastic control tasks indeed exhibit low-rank properties (Sam et al., 2023; Rozada et al., 2024, 2021; Yang et al., 2019).

Motivated by these findings, algorithms for joint feature and policy learning in *model-based* RL have been developed (Agarwal et al., 2020; Bose et al., 2024), though they typically assume the rank is known a priori. For *model-free* RL, Jiang et al. (2017) introduced the notion of *Bellman rank* to quantify the intrinsic complexity of value function approximation. More recent approaches exploit low-rank factorizations or representations to implicitly encourage small Bellman rank while optimizing the policy or value function (Modi et al., 2021, 2024; Yang et al., 2020). However, the theoretical guarantees in these works generally rely on fixed dynamics, and to date there is no algorithm that simultaneously recovers the exact Bellman rank while learning the optimal policy under uncertain or time-varying environments.

## 3 Bias-Variance Tradeoff in RL with Epistemic Uncertainty

As highlighted in the related work section, many control tasks naturally admit low-rank structures in their transition dynamics, which has motivated a line of methods leveraging fixed-rank representations. However, when moving to the robust MDP setting, the presence of epistemic uncertainty fundamentally changes the picture. On the one hand, adopting an excessively low rank may fail to capture the variability introduced by uncertain dynamics, leading to biased estimates and brittle policies. On the other hand, employing a large rank increases model expressiveness but also amplifies variance, making the policy highly sensitive to perturbations and prone to over-parameterization. This tension suggests that selecting an appropriate rank is crucial: the rank must be sufficiently rich to encode uncertainty, yet controlled enough to mitigate overfitting. In this section, we formally analyze this bias–variance tradeoff in reinforcement learning under epistemic uncertainty, beginning with the model-free setting of entropy-regularized reinforcement learning (Haarnoja et al., 2018). The objective function of entropy-regularized reinforcement learning is given by:

$$ $J(\pi)=\mathbb{E}_{\pi}\Bigg[\sum_{t=0}^{\infty}\gamma^{t}\Big(R(s_{t},a_{t})+\,\mathcal{H}(\pi(\cdot|s_{t}))\Big)\Bigg],$ (2) $$

For any given policy $\pi$, we define the corresponding (entropy regularized) $Q^{\pi}$ function and $V^{\pi}$ function as follows:

$$ $\displaystyle V^{\pi}(s)$ $\displaystyle=\mathbb{E}_{a_{t}\sim\pi(\cdot|s_{t}),s_{t+1}\sim\mathcal{P}_{s_{t},a_{t}}}\Big[\sum_{t\geq 0}\gamma^{t}\Big(R(s_{t},a_{t})+\mathcal{H}(\pi(\cdot|s_{t})\Big)\Big|s_{0}=s\Big]$ (3) $\displaystyle Q^{\pi}(s,a)$ $\displaystyle=\mathbb{E}_{a_{t}\sim\pi(\cdot|s_{t}),s_{t+1}\sim\mathcal{P}_{s_{t},a_{t}}}\Big[\sum_{t\geq 0}\gamma^{t}\Big(R(s_{t},a_{t})+\mathcal{H}(\pi(\cdot|s_{t})\Big)\Big|s_{0}=s,a_{0}=a\Big]$ (4) $$

where we write $s_{t+1}\sim\mathcal{P}_{s_{t},a_{t}}$ to indicate that a transition kernel $P_{s_{t},a_{t}}$ is uniformly randomly sampled from the uncertainty set $\mathcal{P}_{s_{t},a_{t}}$ and $s_{t+1}\sim P_{s_{t},a_{t}}$ and the entropy term is defined as $\mathcal{H}(\pi(\cdot|s_{t})):=-\sum_{a\in\mathcal{A}}\pi(a|s_{t})\log\pi(a|s_{t}).$ Let $\pi^{*}$ denote the optimal policy.
We begin by re-stating a well known characterization of the solution to the entropy regularized MDP. According to Haarnoja et al. (2018), the optimal policy takes the following form:

$$ $\pi^{*}(a|s)=\exp{\big(Q^{*}(s,a)-V^{*}(s)\big)}$ (5) $$

where $Q^{*}$ is the unique fixed point of the soft Bellman operator

$$ $\displaystyle\mathcal{B}Q(s,a)$ $\displaystyle:=R(s,a)+\gamma\mathbb{E}_{s^{\prime}\sim\mathcal{P}_{s,a}}\Big[\log\sum_{a^{\prime}\in\mathcal{A}}\exp{Q(s^{\prime},a^{\prime})}\Big]$ (6) $$

and $V^{*}(s^{\prime}):=\log\sum_{a^{\prime}\in\mathcal{A}}\exp{Q(s^{\prime},a^{\prime})}$. We consider linear function approximations for $Q(s,a)$ and $V(s)$ functions for the simplicity of analysis, i.e.:

$$ $Q_{\theta}(s,a)=\phi(s,a)^{\top}\theta\quad\text{and}\quad V_{\omega}(s)=\psi(s)^{\top}\omega$ $$

where $\phi(s,a)$ and $\psi(s)$ are feature mappings.

###### Assumption 1

We assume the training data in the form of triplets $(s,a,s^{\prime})$ is generated as follows: $a\sim\pi_{b}(\cdot\mid s)>0$ where $\pi_{b}$ is a behavioral policy and $s^{\prime}\sim\mathcal{P}_{s,a}$. We assume the induced Markov chain is ergodic and the steady-state distribution of triplets $(s,a,s^{\prime})$ is denoted by $\mathcal{P}$. Similarly, we denote by $\mathcal{P}^{\circ}$ the steady-state distribution of $(s,a,s^{\prime})$ when $a\sim\pi_{b}(\cdot\mid s),s^{\prime}\sim P^{\circ}_{s,a}$.

###### Assumption 2

(Discrete Picard Condition)
The linear system $A_{\mathcal{P}^{\circ}}\theta=b_{\mathcal{P}^{\circ},\omega^{\circ}}$ with ${r^{\circ}}:=\operatorname{rank}(A_{\mathcal{P}^{\circ}})$ satisfies the discrete Picard condition, i.e. the SVD
$A_{\mathcal{P}^{\circ}}=U^{\circ}\Sigma_{\mathcal{P}^{\circ}}{V^{\circ}}^{\top}$ is such that there exists $p>1$ with:

$$ $|{u^{\circ}_{i}}^{\top}b_{\mathcal{P}^{\circ},\omega^{\circ}}|\leq\sigma_{\mathcal{P}^{\circ},i}^{p}\quad\text{for }i=1,\ldots,{r^{\circ}},$ $$

$$ $|{u^{\circ}}_{i}^{\top}b_{\mathcal{P}^{\circ},\omega^{\circ}}|\leq\sigma_{\mathcal{P}^{\circ},{r^{\circ}}}^{p}\quad\text{for }i={r^{\circ}}+1,\ldots,d.$ $$

The discrete Picard condition (Hansen, 1990; Levin & Meltzer, 2017) states that the magnitude of the inner product $|{u^{\circ}}_{i}^{\top}b_{\mathcal{P}^{\circ},\omega^{\circ}}|$ shrinks faster that $\sigma_{i}^{p}$, accounting for the ill-condition in the system dynamics.
Here $p>1$ describes the shrinking speed.

###### Assumption 3

(2.1) $\left\|\phi(s,a)\right\|\leq 1,$
$\forall(s,a)\in\mathcal{S}\times\mathcal{A}$.
(2.2) The feature covariance matrices with respect to ground truth dynamics are non-singular:

$$ $\mathbb{E}_{\mathcal{P}^{\circ}}[\phi(s,a)\phi(s,a)^{\top}]\succ 0$ $$

(2.3) (Lipschitz) $\forall(s,a)\in\mathcal{S}\times\mathcal{A}$, it holds that:

$$ $\displaystyle\big\|\phi(s,a)^{\top}\theta_{1}-\phi(s,a)^{\top}\theta_{2}\big\|$ $\displaystyle\leq L\big\|\theta_{1}-\theta_{2}\big\|$ (7a) $$

where $L>0$. These are standard assumptions in reinforcement learning with linear function approximation Tsitsiklis & Van Roy (1996); Munos (2003).

Hence, in an off-policy setting, the optimal policy with linear function approximation can be described as the solution to the following optimization problem:

$$ $\displaystyle\min_{\omega}~~$ $\displaystyle~~\mathbb{E}_{\mathcal{P}}\Big[\|\psi(s^{\prime})^{\top}\omega-\log\sum_{a^{\prime}\in\mathcal{A}}\exp{\phi(s^{\prime},a^{\prime})^{\top}\theta^{*}(\omega)}\|^{2}\Big]$ (8) s.t $\displaystyle~~\theta^{*}(\omega)=\arg\min_{\theta}\mathbb{E}_{\mathcal{P}}\Big[\|R(s,a)+\gamma\psi(s^{\prime})^{\top}\omega-\phi(s,a)^{\top}\theta\|^{2}\Big]$ (9) $$

where $\mathcal{P}$ refers to the steady-state distribution on $(s,a,s^{\prime})$ associated with uniformly sampling the transition kernel from the Wasserstein ball and selecting actions according to fixed behavioral policy. The first order (sufficient) conditions for lower level optimality can be written as:

$$ $\displaystyle-\mathbb{E}_{\mathcal{P}}\big[\phi(s,a)(R(s,a)+\gamma\psi(s^{\prime})^{\top}\omega-\phi(s,a)^{\top}\theta)\big]$ $\displaystyle=0$ (10) $$

wherein we write $\mathbb{E}_{\mathcal{P}}$ as shorthand for $\mathbb{E}_{(s,a,s^{\prime})\in\mathcal{P}}$. This system of equations can be re-written as
$A_{\mathcal{P}}\theta=b_{\mathcal{P},\omega}$ where

$$ $\displaystyle A_{\mathcal{P}}:=\mathbb{E}_{\mathcal{P}}[\phi(s,a)\phi(s,a)^{\top}]~~~~~b_{\mathcal{P},\omega}:=\mathbb{E}_{\mathcal{P}}[\phi(s,a)\big(R(s,a)+\gamma\psi(s^{\prime})^{\top}\omega\big)]$ $$

Similarly for the ground-truth kernel $\mathcal{P}^{\circ}$ we define the system:

$$ $\displaystyle A_{\mathcal{P}^{\circ}}:=\mathbb{E}_{\mathcal{P}^{\circ}}[\phi(s,a)\phi(s,a)^{\top}]~~~~~b_{\mathcal{P}^{\circ},\omega}:=\mathbb{E}_{\mathcal{P}^{\circ}}[\phi(s,a)\big(R(s,a)+\gamma\psi(s^{\prime})^{\top}\omega\big)]$ $$

where $\mathcal{P}^{\circ}$ refers to the ground truth steady-state distribution.

Our analysis investigates the consequences of using high-rank parametrized policies when the underlying ground-truth environment dynamics are of lower rank. Let $(\theta^{\circ},\omega^{\circ})$ denote the solution of the optimization problem defined by Eq. [8](https://arxiv.org/html/2510.11899v1#S3.E8) and Eq. [9](https://arxiv.org/html/2510.11899v1#S3.E9) when the expectations are taken with ground-truth dynamics. To formalize this setting, we characterize the low-rank structure of the environment dynamics under a set of regularity conditions. In particular, we assume bounded feature mappings, nonsingular covariance matrices, and a discrete Picard condition, which are standard in reinforcement learning with linear function approximation (see Appendix LABEL:app:sec_assumption for the full statements of Assumptions [2](https://arxiv.org/html/2510.11899v1#Thmassumption2) and [3](https://arxiv.org/html/2510.11899v1#Thmassumption3)).

Building on these assumptions, we next examine the effect of approximating the system $A_{\mathcal{P}}\theta=b_{\mathcal{P}}$ using an $r$-truncated SVD decomposition of $A_{\mathcal{P}}$, denoted $A_{\mathcal{P},r}$. This result highlights the fundamental bias–variance trade-off: choosing too small an $r$ induces approximation bias, whereas choosing too large an $r$ amplifies estimation variance.

###### Theorem 1

Bias-Variance Trade-off of Rank-r Approximation: Assume the ground-truth dynamics are given by $\mathcal{P}^{\circ}$ and Assumptions [2](https://arxiv.org/html/2510.11899v1#Thmassumption2) and [3](https://arxiv.org/html/2510.11899v1#Thmassumption3) hold. Let $(\theta^{\circ},\omega^{\circ})$ denote the solution of the optimization problem defined by Eq. [8](https://arxiv.org/html/2510.11899v1#S3.E8), Eq. [9](https://arxiv.org/html/2510.11899v1#S3.E9) when the expectations are taken with ground-truth dynamics (i.e. $A_{\mathcal{P}^{\circ}}\theta=b_{\mathcal{P}^{\circ},\omega^{\circ}}$). Consider a truncated SVD $A_{\mathcal{P},r}=U\Sigma_{{\mathcal{P},r}}V^{\top}$ for $r\leq\operatorname{rank}(A_{\mathcal{P}})$ and $\theta_{r}$ be the solution $A_{\mathcal{P},r}\theta=b_{\mathcal{P},\omega^{\circ}}$. It holds that:

$$ $\displaystyle\|\theta_{r}-\theta^{\circ}\|_{2}\leq\underbrace{\frac{1}{\sigma_{\mathcal{P},r}}\|b_{\mathcal{P},\omega^{\circ}}-b_{\mathcal{P}^{\circ},\omega^{\circ}}\|_{2}}_{variance}+\underbrace{(d-r)\sigma_{\mathcal{P}^{\circ},r}^{p-1}+(d-r){r^{\circ}}\sigma_{\mathcal{P}^{\circ},1}^{p-1}}_{bias}+2\mathcal{O}(L\epsilon)$ (11) $$

where $r^{\circ}:=\operatorname{rank}(A_{\mathcal{P}^{\circ}})$, and $\epsilon>0$ denotes the radius of the Wasserstein ball (Mohajerin Esfahani & Kuhn, 2018).

Remark
The upper bound of the performance gap between the estimated parameter $\theta_{r}$ and the optimal solution $\theta^{\circ}$ in Theorem.[1](https://arxiv.org/html/2510.11899v1#Thmtheorem1) can be decomposed into two components related to variance and bias respectively. Thus for example, the choice of $r>r^{\circ}$ introduces higher variance since $\sigma_{\mathcal{P},r}<\sigma_{\mathcal{P},r^{\circ}}$. Conversely, the choice of $r<r^{\circ}$ introduces higher bias since

$$ $(d-r)\sigma_{\mathcal{P}^{\circ},r}^{p-1}+(d-r){r^{\circ}}\sigma_{\mathcal{P}^{\circ},1}^{p-1}>(d-r^{\circ})\sigma_{\mathcal{P}^{\circ},r^{\circ}}^{p-1}+(d-r^{\circ}){r^{\circ}}\sigma_{\mathcal{P}^{\circ},1}^{p-1}$ $$

Discussion To confirm that bias-variance tradeoff also exists in settings with non-linear representation, we perform a sanity check on a MuJoCo control task (Todorov et al., 2012). Specifically, we employ a three-layer neural network and adopt a rank-control mechanism similar to (Hu et al., 2022; Xu et al., 2019) (see details in Sec.[4.2](https://arxiv.org/html/2510.11899v1#S4.SS2)). Our experiments reveal a clear bias–variance tradeoff in nonlinear control models, as illustrated in Figure [2](https://arxiv.org/html/2510.11899v1#S3.F2): models with extremely low-rank representations exhibit high bias, while high-rank models suffer from large approximation errors due to transition samples drawn from uncertain dynamics.

Figure: Figure 2: Performance of policy models under high model uncertainty in Walker2d-v3 (Left) and Hopper-v3 (Right). Results indicate that extremely low-rank representations lead to high bias, while overly high-rank models incur large approximation errors due to transition samples drawn from uncertain dynamics.
Refer to caption: x2.png

###### Assumption 1

###### Assumption 2

###### Assumption 3

###### Theorem 1

## 4 Adaptive Rank Representation Reinforcement Learning

### 4.1 A Bi-level Optimization Formulation

The analysis in previous Section highlights that selecting the policy rank involves a delicate balance:
too small a rank induces bias, while too large a rank amplifies variance.
This trade-off suggests the need for an adaptive mechanism that can automatically adjust the rank during learning.
Motivated by this insight, we introduce a bi-level (Colson et al., 2007) optimization formulation,
where the lower-level problem identifies the optimal policy with uniformly sampled environment dynamics
(from a Wasserstein ball around a centroid model) under a fixed rank,
and the upper-level problem searches for the representation that optimizes a measure of fit to the lower-level model while regularizing by rank. To begin with, We consider a parameterized policy $\pi_{\theta}$, where $\theta\in\mathbb{R}^{d_{1}\times d_{2}}$ with $d_{1},d_{2}>0$. And we respectively denote by

$$ $\displaystyle\mathcal{M}_{r}:=\{\theta\in\mathbb{R}^{d_{1}\times d_{2}}~|~{\rm rank}(\theta)=r\}$ $\displaystyle~~~\mathcal{M}_{\leq\bar{r}}:=\{\theta\in\mathbb{R}^{d_{1}\times d_{2}}~|~{\rm rank}(\theta)\leq\bar{r}\}$ $$

the smooth manifold of matrices with rank $r$.
and the algebraic variety of matrices with rank less than or equal to $\bar{r}>0$.

Formulation: Towards developing an approach that simultaneously learns the policy and adaptively adjusts its rank, we introduce the following bi-level formulation:

$$ $\displaystyle\min_{r}\;$ $\displaystyle\mathbb{E}_{(s,a)\sim\mathcal{P}_{\theta^{*}}}\|{\rm Proj_{\mathcal{M}_{r}}}(\pi_{\theta^{*}})(a|s)-\pi_{\theta^{*}}(a|s)\|_{2}+\lambda r$ (12) $\displaystyle{\rm s.t.}~~$ $\displaystyle\theta^{*}:=\arg\max_{\theta\in\mathcal{M}_{r}}\mathbb{E}_{\tau\sim\mathcal{P}_{\pi_{\theta}}}\Big[\sum_{t\geq 0}\gamma^{t}\Big(R(s_{t},a_{t})+\mathcal{H}(\pi_{\theta}(\cdot|s_{t}))\Big)\Big]$ (13) $$

where $\mathcal{P}_{{\theta^{*}}}$ denotes the steady-state distribution obtained by uniformly sampling the transition kernel from the Wasserstein ball and selecting actions according to the policy $\pi_{\theta^{*}}$, the operator $\textrm{Proj}_{\mathcal{M}_{r}}(\pi_{\theta^{*}})$ denotes the projection of the policy onto the low-rank manifold $\mathcal{M}_{r}$, $\lambda$ serves as a weight for rank regularization $r$.

Discussion The bi-level formulation in Eq. [12](https://arxiv.org/html/2510.11899v1#S4.E12)–Eq. [13](https://arxiv.org/html/2510.11899v1#S4.E13) plays two complementary roles.
The *lower-level problem* Eq. [13](https://arxiv.org/html/2510.11899v1#S4.E13) optimizes the policy parameters under a fixed rank constraint, aiming to maximize the entropy-regularized return and thus capture the best achievable policy representation at that rank.
However, the optimal solution $\pi_{\theta^{*}}$ of the lower-level problem may not align with the intrinsic task complexity and can overfit by exploiting the full representation power. To address this, the *upper-level problem* Eq. [12](https://arxiv.org/html/2510.11899v1#S4.E12) explicitly searches for an appropriate rank that balances bias and variance, as motivated in the previous section. It seeks the best low-dimensional representation (bounded by $\bar{r}>0$) of the state–action value associated with $\pi_{\theta^{*}}$, while controlling model capacity through the rank regularization term. In this way, the upper-level problem enforces a bias–variance tradeoff, ensuring that the learned representation achieves robustness without unnecessary over-parameterization.

### 4.2 Algorithm

We are now ready to design algorithms for the proposed formulation.
Note that our formulation has a hierarchical structure and falls into the class of bi-level optimization problems Hong et al. (2023); Colson et al. (2007).
In general, bi-level problems are challenging to solve; in our case, the upper-level objective Eq. [12](https://arxiv.org/html/2510.11899v1#S4.E12) depends explicitly on the optimal solution of the lower-level problem.
Furthermore, the rank regularizer $C(\cdot)$ is non-differentiable, which precludes the use of (stochastic) first-order methods for the upper-level optimization.
Fortunately, as we will show, a simple yet effective adaptive greedy search algorithm can be employed to obtain an empirical solution to the upper-level problem. At a high level, the proposed algorithm alternates between two steps: a Rank Adaptation Step, which updates the rank $r$ via a greedy search procedure, and a Policy Optimization Step, which optimizes the parameters under the rank constraint $\theta\in\mathcal{M}_{\leq r}$.
We now examine each step in detail.

Rank Adaptation Step From the discussion in Section [3](https://arxiv.org/html/2510.11899v1#S3), we know that extremely low-rank models are limited in their representation power and thus fail to capture sufficient information under model uncertainty. In contrast, high-rank models tend to overfit, resulting in poor generalization. Hence, it is crucial to carefully select an appropriate rank for policies in MDPs with uncertain dynamics. Although Theorem [1](https://arxiv.org/html/2510.11899v1#Thmtheorem1) provides useful insights, in practice it is difficult to explicitly solve this tradeoff and obtain the optimal rank. To address this, we adopt a greedy strategy: starting from a high-rank model, we gradually reduce the rank until reaching a stable value that yields consistent performance under model uncertainty. This procedure operationalizes the bias–variance tradeoff characterized in Theorem [1](https://arxiv.org/html/2510.11899v1#Thmtheorem1) and forms the core of the Rank Adaptation Step in our algorithm.

Specifically, the upper-level problem Eq. [12](https://arxiv.org/html/2510.11899v1#S4.E12) requires us to identify suitable representations for both the policy and value models while keeping their ranks as low as possible. If no lower-rank model with sufficient approximation quality can be found, we simply retain the previous rank, i.e., $r_{\rm new}=r_{\rm old}$. To do the greedy search, we consider using the following criterion to decide the new rank $\hat{r}$. Note there are many ways to decide the target rank; in the ablation study, we show that using this criterion achieves a smooth truncation and makes the rank converge to the intrinsic rank of the environment.

$$ $\hat{r}=\max\{\ell\in\{1,2,\dots,d\}:\frac{\sum_{i=1}^{\ell}\sigma_{i}}{\sum_{i=1}^{d}\sigma_{i}}\leq\beta\}$ (14) $$

To implement this efficiently, we adopt a low-rank factorization approach (Xu et al., 2019; Zhang et al., 2015) that operates directly on the weight matrices of neural networks. Since the rank of a neural network layer is inherently constrained by the number of hidden units, we follow the idea of inserting an intermediate linear layer between consecutive layers, thereby controlling the rank through the size of this hidden layer (see Figure [5](https://arxiv.org/html/2510.11899v1#A1.F5) and Appendix [A.2.1](https://arxiv.org/html/2510.11899v1#A1.SS2.SSS1) for details). After obtaining the optimized policy $\pi_{\theta}^{k}$ from several policy optimization steps, we refine this low-rank representation by performing SVD.

Policy Optimization Step One can adopt the standard approaches, such as the well-known soft actor critic (SAC) (Haarnoja et al., 2018) algorithm to obtain an approximate optimal policy that solves Eq. [13](https://arxiv.org/html/2510.11899v1#S4.E13). Notice that after reconstructing the neural network, the rank of parameter $\theta$ is no larger than $\hat{r}$ due to the existence of the intermediate layer. In this way, the rank constraint is automatically enforced during optimization without requiring explicit SVD at every update.

We summarize the proposed algorithm in Algorithm [1](https://arxiv.org/html/2510.11899v1#alg1), corresponding to the rank adaptation step and policy improvement step. We conclude this section with a brief remark on the advantages of AdaRL.

Low Computational Complexity Unlike previous methods (Gehring et al., 2015) that require repeated SVD with complexity $\mathcal{O}(d^{3})$ per update, AdaRL uses a single rank adaptation step to estimate a feasible rank. It operates on two timescales: the inner loop optimizes policies under a low-rank constraint, while the outer loop infrequently adjusts the rank by projecting parameters onto a lower-rank manifold. This design avoids costly worst-case value estimation and yields an efficient training procedure.

Convergence Control dynamical systems governed by Newtonian mechanics naturally exhibit a low-rank structure (Tiwari et al., 2025). Although deriving theoretical convergence guarantees is nontrivial, our experiments empirically show that the solution of the upper-level problem converges to a stable rank, thereby balancing model robustness with representational capacity.

Figure: Algorithm 1 Adaptive Rank Representation (AdaRL)

## 5 Experiment

In this section, we present numerical evaluations of the proposed method AdaRL (Alg. [1](https://arxiv.org/html/2510.11899v1#alg1)) and compare it against several robust RL baselines, including RNAC, Parseval regularization, fixed-rank SAC, and the algorithm from Tiwari et al. (2025). Our experiments highlight the advantages of AdaRL in two key aspects: (1) it achieves a favorable trade-off between the bias and variance induced by model uncertainty, thereby enabling more robust policy learning; and (2) it identifies a suitable low-rank manifold, within which constraining the policy model yields a representation that remains robust under model uncertainty. More details are given in the Appendix [A.2.1](https://arxiv.org/html/2510.11899v1#A1.SS2.SSS1).

We focus on robotic control tasks with continuous action spaces, using four widely adopted OpenAI Gym environments and their variants: Hopper-v3, Walker2d-v3, Ant-v3, and Humanoid-v3. Following the setup in Luo et al. (2024), we introduce model uncertainty by modifying the source dynamics for each task. In Hopper and Walker2d, this involves structural changes such as adjusting torso and foot sizes, while in Ant and Humanoid we alter physical parameters including gravity or add external forces such as wind with a specified velocity. During training, the environment dynamics vary across episodes to simulate epistemic uncertainty.

The baselines considered in this scenario are: (1) SAC (Haarnoja et al., 2018) with a fixed-rank parameterization; (2) RNAC (Zhou et al., 2023), which employs double sampling within newly defined uncertainty sets and uses function approximation to solve the robust Bellman equation; (3) Parseval regularization (Chung et al., 2024), which enforces orthogonality in weight matrices to preserve optimization properties and improve training stability in continual reinforcement learning; and (4) the method of Tiwari et al. (2025), which incorporates a fully connected sparsification MLP layer for reinforcement learning.

In Figure [3](https://arxiv.org/html/2510.11899v1#S5.F3) and Table [1](https://arxiv.org/html/2510.11899v1#S5.T1), we report numerical results comparing the proposed AdaRL algorithm with several baselines. As shown in Figure [3](https://arxiv.org/html/2510.11899v1#S5.F3), both AdaRL and standard SAC achieve similar performance in the first iteration; however, once the model rank is adjusted, AdaRL consistently outperforms the standard methods by mitigating the impact of model uncertainty.
It is worth noting that immediately after each rank adaptation step, the optimizer’s momentum is reset and the model must adjust to the new parameterization, leading to a temporary performance drop before recovery.
Further, in Table [1](https://arxiv.org/html/2510.11899v1#S5.T1), the results show that AdaRL consistently outperforms the baselines by a significant margin in most scenarios. As discussed in Section [2.2](https://arxiv.org/html/2510.11899v1#S2.SS2), robust RL algorithms typically perform policy improvement based on worst-case value functions, which enhances robustness but often yields overly conservative policies and incurs high approximation errors in continuous control environments (Mannor et al., 2012, 2016; Xu & Mannor, 2012). For regularization-based approaches, Parseval regularization can partially mitigate value-function overfitting, but it remains less effective than the low-rank constraint imposed in AdaRL. To fairly assess policy generalization, all evaluations are conducted under the fixed nominal dynamics $\mathcal{P}^{\circ}$, enabling us to examine whether the learned policies remain effective and robust in the presence of model uncertainty. In Appendix [A.2.4](https://arxiv.org/html/2510.11899v1#A1.SS2.SSS4), we further demonstrate the robustness of the trained policy in different perturbed dynamics.

**Table 1: MuJoCo Results. The performance of the benchmark algorithms. Bolded numbers indicate the best results among AdaRL, RNAC, Parseval regularization, and the algorithm in Tiwari et al. (2025) for each task.**
| Task | AdaRL (proposed) | RNAC | Parseval | Alg. in Tiwari et al. (2025) |
| --- | --- | --- | --- | --- |
| Hopper | ${\bf 2109.8}\pm{\bf 322.90}$ | $1542.36\pm 62.73$ | $1410.64\pm 456.52$ | $1850.09\pm 234.98$ |
| Walker | ${\bf 3991.90}\pm{\bf 567.00}$ | $1906.68\pm 620.90$ | $2368.26\pm 1346.67$ | $3280.55\pm 179.42$ |
| Ant | ${\bf 3067.13}\pm{\bf 111.55}$ | $1021.97\pm 230.71$ | $2063.18\pm 381.27$ | $2719.95\pm 225.11$ |
| Humanoid | ${\bf 5428.72}\pm{\bf 50.10}$ | $2351.42\pm 443.12$ | $458.35\pm 76.41$ | $5255.03\pm 757.45$ |

In Figure [4](https://arxiv.org/html/2510.11899v1#S5.F4), we report an additional experiment showing that the rank estimated by the AdaRL algorithm in  Eq. [12](https://arxiv.org/html/2510.11899v1#S4.E12) gradually converges to the intrinsic rank identified by Tiwari et al. (2025), given an appropriate choice of $\beta$ in Alg. [1](https://arxiv.org/html/2510.11899v1#alg1) (set to $0.98$ in our experiments). This result demonstrates that AdaRL can effectively search for a suitable rank for environment with model uncertainty.

Figure: Figure 3: Training performance on MuJoCo tasks. The proposed AdaRL consistently outperforms standard SAC baselines under model uncertainty. The red dashed vertical lines indicate the boundaries between different iteration intervals.
Refer to caption: x4.png

Figure: Figure 4: We plot the estimated rank from AdaRL throughout training. The intrinsic rank refers to the value identified by Tiwari et al. (2025). Left: Walker2d. Right: Hopper.
Refer to caption: x6.png

## 6 Conclusion

In this paper, we propose a novel framework for reinforcement learning under epistemic uncertainty by integrating the low-rank structure into policy representation. We begin by establishing a theoretical bias-variance trade-off that arises when applying low-rank approximations with uncertain dynamics. Motivated by this insight, we formulate a bi-level optimization problem and develop the Adaptive Low-Rank Representation algorithm, which dynamically adjusts the policy’s representational rank to balance generalization and robustness. Our extensive experiments on MuJoCo benchmarks demonstrate that AdaRL consistently outperforms both fixed-rank RL methods and state-of-the-art robust RL algorithms.

## References

- Agarwal et al. (2020)
Alekh Agarwal, Sham Kakade, Akshay Krishnamurthy, and Wen Sun.
Flambe: Structural complexity and representation learning of low rank
mdps.
*Advances in neural information processing systems*,
33:20095–20107, 2020.
- Atiyah & Todd (1960)
MF Atiyah and JA Todd.
On complex stiefel manifolds.
In *Mathematical Proceedings of the Cambridge Philosophical
Society*, volume 56, pp. 342–353. Cambridge University Press, 1960.
- Badrinath & Kalathil (2021)
Kishan Panaganti Badrinath and Dileep Kalathil.
Robust reinforcement learning using least squares policy iteration
with provable performance guarantees.
In *International Conference on Machine Learning*, pp. 511–520. PMLR, 2021.
- Bose et al. (2024)
Avinandan Bose, Simon Shaolei Du, and Maryam Fazel.
Offline multi-task transfer rl with representational penalization,
2024.
URL [https://arxiv.org/abs/2402.12570](https://arxiv.org/abs/2402.12570).
- Chakraborty & Vemuri (2019)
Rudrasis Chakraborty and Baba C Vemuri.
Statistics on the stiefel manifold: Theory and applications.
2019.
- Chung et al. (2024)
Wesley Chung, Lynn Cherif, Doina Precup, and David Meger.
Parseval regularization for continual reinforcement learning.
*Advances in Neural Information Processing Systems*,
37:127937–127967, 2024.
- Clavier et al. (2023)
Pierre Clavier, Erwan Le Pennec, and Matthieu Geist.
Towards minimax optimality of model-based robust reinforcement
learning.
*arXiv preprint arXiv:2302.05372*, 2023.
- Colson et al. (2007)
Benoît Colson, Patrice Marcotte, and Gilles Savard.
An overview of bilevel optimization.
*Annals of operations research*, 153(1):235–256, 2007.
- Falini (2022)
Antonella Falini.
A review on the selection criteria for the truncated svd in data
science applications.
*Journal of Computational Mathematics and Data Science*,
5:100064, 2022.
- Gadot et al. (2024)
Uri Gadot, Kaixin Wang, Navdeep Kumar, Kfir Yehuda Levy, and Shie Mannor.
Bring your own (non-robust) algorithm to solve robust mdps by
estimating the worst kernel.
In *Forty-first International Conference on Machine Learning*,
2024.
- Gehring et al. (2015)
Clement Gehring, Yangchen Pan, and Martha White.
Incremental truncated lstd.
*arXiv preprint arXiv:1511.08495*, 2015.
- Grand-Clément & Kroer (2021)
Julien Grand-Clément and Christian Kroer.
Scalable first-order methods for robust mdps.
In *Proceedings of the AAAI Conference on Artificial
Intelligence*, volume 35, pp. 12086–12094, 2021.
- Haarnoja et al. (2018)
Tuomas Haarnoja, Aurick Zhou, Pieter Abbeel, and Sergey Levine.
Soft actor-critic: Off-policy maximum entropy deep reinforcement
learning with a stochastic actor.
In *International conference on machine learning*, pp. 1861–1870. Pmlr, 2018.
- Hansen & Sargent (2008)
Lars Peter Hansen and Thomas J. Sargent.
*Robustness*.
Princeton University Press, Princeton, NJ, 2008.
ISBN 9780691132150.
- Hansen (1990)
Per Christian Hansen.
The discrete picard condition for discrete ill-posed problems.
*BIT Numerical Mathematics*, 30(4):658–672,
1990.
- Hong et al. (2023)
Mingyi Hong, Hoi-To Wai, Zhaoran Wang, and Zhuoran Yang.
A two-timescale stochastic algorithm framework for bilevel
optimization: Complexity analysis and application to actor-critic.
*SIAM Journal on Optimization*, 33(1):147–180, 2023.
- Hu et al. (2022)
Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean
Wang, Lu Wang, Weizhu Chen, et al.
Lora: Low-rank adaptation of large language models.
*ICLR*, 1(2):3, 2022.
- Iyengar (2005)
Garud N Iyengar.
Robust dynamic programming.
*Mathematics of Operations Research*, 30(2):257–280, 2005.
- Jiang et al. (2017)
Nan Jiang, Akshay Krishnamurthy, Alekh Agarwal, John Langford, and Robert E
Schapire.
Contextual decision processes with low bellman rank are
pac-learnable.
In *Proceedings of the 34th International Conference on Machine
Learning (ICML)*, volume 70 of *Proceedings of Machine Learning
Research*, pp. 1704–1713. PMLR, 2017.
- Kumar et al. (2022)
Navdeep Kumar, Kfir Levy, Kaixin Wang, and Shie Mannor.
Efficient policy iteration for robust markov decision processes via
regularization.
*arXiv preprint arXiv:2205.14327*, 2022.
- Kumar et al. (2023a)
Navdeep Kumar, Kfir Levy, Kaixin Wang, and Shie Mannor.
An efficient solution to s-rectangular robust markov decision
processes.
*arXiv preprint arXiv:2301.13642*, 2023a.
- Kumar et al. (2023b)
Navdeep Kumar, Ilnura Usmanova, Kfir Yehuda Levy, and Shie Mannor.
Towards faster global convergence of robust policy gradient methods.
In *Sixteenth European Workshop on Reinforcement Learning*,
2023b.
- Lanzani (2025)
Giacomo Lanzani.
Dynamic concern for misspecification.
*Econometrica*, 93(4):1333–1370, 2025.
- Lemoine & Traeger (2014)
Derek Lemoine and Christian P. Traeger.
Watch your step: Optimal policy in a tipping climate.
*American Economic Journal: Economic Policy*, 6(1):137–166, 2014.
- Levin & Meltzer (2017)
Eitan Levin and Alexander Y Meltzer.
Estimation of the regularization parameter in linear discrete
ill-posed problems using the picard parameter.
*SIAM Journal on Scientific Computing*, 39(6):A2741–A2762, 2017.
- Li et al. (2018)
Chunyuan Li, Heerad Farkhoor, Rosanne Liu, and Jason Yosinski.
Measuring the intrinsic dimension of objective landscapes.
*arXiv preprint arXiv:1804.08838*, 2018.
Published: ICLR 2018 Workshop / International Conference on Learning
Representations.
- Li et al. (2022)
Yan Li, Guanghui Lan, and Tuo Zhao.
First-order policy optimization for robust markov decision process.
*arXiv preprint arXiv:2209.10579*, 2022.
- Lialin et al. (2023)
Vladislav Lialin, Namrata Shivagunde, Sherin Muckatira, and Anna Rumshisky.
Relora: High-rank training through low-rank updates.
*arXiv preprint arXiv:2307.05695*, 2023.
- Luo et al. (2024)
Yu Luo, Tianying Ji, Fuchun Sun, Jianwei Zhang, Huazhe Xu, and Xianyuan Zhan.
Ompo: A unified framework for rl under policy and dynamics shifts.
*arXiv preprint arXiv:2405.19080*, 2024.
- Mannor et al. (2012)
Shie Mannor, Ofir Mebel, and Huan Xu.
Lightning does not strike twice: Robust mdps with coupled
uncertainty.
In *Proceedings of the 29th International Conference on Machine
Learning (ICML)*, pp. 385–392, 2012.
URL [https://arxiv.org/abs/1206.4643](https://arxiv.org/abs/1206.4643).
- Mannor et al. (2016)
Shie Mannor, Ofir Mebel, and Huan Xu.
Robust mdps with k-rectangular uncertainty.
*Mathematics of Operations Research*, 41(4):1484–1509, 2016.
doi: 10.1287/moor.2016.0786.
URL [https://pubsonline.informs.org/doi/10.1287/moor.2016.0786](https://pubsonline.informs.org/doi/10.1287/moor.2016.0786).
- Mnih et al. (2015)
Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Andrei A Rusu, Joel Veness,
Marc G Bellemare, Alex Graves, Martin Riedmiller, Andreas K Fidjeland, Georg
Ostrovski, et al.
Human-level control through deep reinforcement learning.
*nature*, 518(7540):529–533, 2015.
- Modi et al. (2024)
Aditya Modi, Jinglin Chen, Akshay Krishnamurthy, Nan Jiang, and Alekh Agarwal.
Model-free representation learning and exploration in low-rank mdps.
*Journal of Machine Learning Research*, 25(6):1–76, 2024.
- Modi et al. (2021)
Ishaan Modi, Adrien Bembom, John Schulman, Sergey Levine, and Pieter Abbeel.
Model-based reinforcement learning via structural constraint on value
functions.
In *Advances in Neural Information Processing Systems*, 2021.
URL [https://arxiv.org/abs/2110.08708](https://arxiv.org/abs/2110.08708).
- Mohajerin Esfahani & Kuhn (2018)
Peyman Mohajerin Esfahani and Daniel Kuhn.
Data-driven distributionally robust optimization using the
wasserstein metric: Performance guarantees and tractable reformulations.
*Mathematical Programming*, 171(1):115–166,
2018.
- Moos et al. (2022)
Janosch Moos, Kay Hansel, Hany Abdulsamad, Svenja Stark, Debora Clever, and Jan
Peters.
Robust reinforcement learning: A review of foundations and recent
advances.
*Machine Learning and Knowledge Extraction*, 4(1):276–315, 2022.
- Munos (2003)
Rémi Munos.
Error bounds for approximate policy iteration.
In *Proceedings of the Twentieth International Conference on
International Conference on Machine Learning*, pp. 560–567, 2003.
- Nagami & Schwager (2023)
Keiko Nagami and Mac Schwager.
Epistemic uncertainty in state estimation and belief space planning
with learning-based perception systems.
In *First Workshop on Out-of-Distribution Generalization in
Robotics at CoRL 2023*, 2023.
URL [https://openreview.net/forum?id=CPZaavSwXg](https://openreview.net/forum?id=CPZaavSwXg).
- Nilim & El Ghaoui (2005)
Arnab Nilim and Laurent El Ghaoui.
Robust control of markov decision processes with uncertain transition
matrices.
*Operations Research*, 53(5):780–798, 2005.
- Pattanaik et al. (2017)
Anay Pattanaik, Zhenyi Tang, Shuijing Liu, Gautham Bommannan, and Girish
Chowdhary.
Robust deep reinforcement learning with adversarial attacks.
*arXiv preprint arXiv:1712.03632*, 2017.
- Rozada et al. (2021)
Sergio Rozada, Victor Tenorio, and Antonio G Marques.
Low-rank state-action value-function approximation.
In *2021 29th European Signal Processing Conference (EUSIPCO)*,
pp. 1471–1475. IEEE, 2021.
- Rozada et al. (2024)
Sergio Rozada, Santiago Paternain, and Antonio G Marques.
Tensor and matrix low-rank value-function approximation in
reinforcement learning.
*IEEE Transactions on Signal Processing*, 2024.
- Sam et al. (2023)
Tyler Sam, Yudong Chen, and Christina Lee Yu.
Overcoming the long horizon barrier for sample-efficient
reinforcement learning with latent low-rank structure.
*Proceedings of the ACM on Measurement and Analysis of Computing
Systems*, 7(2):1–60, 2023.
- Satia & Lave Jr (1973)
Jay K Satia and Roy E Lave Jr.
Markovian decision processes with uncertain transition probabilities.
*Operations Research*, 21(3):728–740, 1973.
- Silver et al. (2017)
David Silver, Julian Schrittwieser, Karen Simonyan, Ioannis Antonoglou, Aja
Huang, Arthur Guez, Thomas Hubert, Lucas Baker, Matthew Lai, Adrian Bolton,
et al.
Mastering the game of go without human knowledge.
*nature*, 550(7676):354–359, 2017.
- Sutton et al. (1998)
Richard S Sutton, Andrew G Barto, et al.
*Reinforcement learning: An introduction*, volume 1.
MIT press Cambridge, 1998.
- Tiwari et al. (2025)
Saket Tiwari, Omer Gottesman, and George Konidaris.
Geometry of neural reinforcement learning in continuous state and
action spaces.
In *The Second Conference on Parsimony and Learning (Recent
Spotlight Track)*, 2025.
- Todorov et al. (2012)
Emanuel Todorov, Tom Erez, and Yuval Tassa.
Mujoco: A physics engine for model-based control.
In *2012 IEEE/RSJ International Conference on Intelligent Robots
and Systems*, pp. 5026–5033. IEEE, 2012.
- Touvron et al. (2023)
Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne
Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric
Hambro, Faisal Azhar, et al.
Llama: Open and efficient foundation language models.
*arXiv preprint arXiv:2302.13971*, 2023.
- Tsitsiklis & Van Roy (1996)
John Tsitsiklis and Benjamin Van Roy.
Analysis of temporal-diffference learning with function
approximation.
*Advances in neural information processing systems*, 9, 1996.
- Villani et al. (2008)
Cédric Villani et al.
*Optimal transport: old and new*, volume 338.
Springer, 2008.
- Wang et al. (2023)
Qiuhao Wang, Chin Pang Ho, and Marek Petrik.
Policy gradient in robust mdps with global convergence guarantee.
In *International Conference on Machine Learning*, pp. 35763–35797. PMLR, 2023.
- Wang & Zou (2021)
Yue Wang and Shaofeng Zou.
Online robust reinforcement learning with model uncertainty.
*Advances in Neural Information Processing Systems*,
34:7193–7206, 2021.
- Wang & Zou (2022)
Yue Wang and Shaofeng Zou.
Policy gradient method for robust reinforcement learning.
In *International conference on machine learning*, pp. 23484–23526. PMLR, 2022.
- Wiesemann et al. (2013)
Wolfram Wiesemann, Daniel Kuhn, and Berç Rustem.
Robust markov decision processes.
*Mathematics of Operations Research*, 38(1):153–183, 2013.
- Xu & Mannor (2012)
Huan Xu and Shie Mannor.
Distributionally robust markov decision processes.
*Mathematics of Operations Research*, 37(2):288–300, 2012.
doi: 10.1287/moor.1120.0540.
URL [https://doi.org/10.1287/moor.1120.0540](https://doi.org/10.1287/moor.1120.0540).
- Xu et al. (2019)
Yuhui Xu, Yuxi Li, Shuai Zhang, Wei Wen, Botao Wang, Wenrui Dai, Yingyong Qi,
Yiran Chen, Weiyao Lin, and Hongkai Xiong.
Trained rank pruning for efficient deep neural networks.
In *2019 Fifth Workshop on Energy Efficient Machine Learning and
Cognitive Computing-NeurIPS Edition (EMC2-NIPS)*, pp. 14–17. IEEE, 2019.
- Yang et al. (2019)
Yuzhe Yang, Guo Zhang, Zhi Xu, and Dina Katabi.
Harnessing structures for value-based planning and reinforcement
learning.
*arXiv preprint arXiv:1909.12255*, 2019.
- Yang et al. (2020)
Zichuan Yang, George Tucker, Tom Zahavy, Mohammad Ghavamzadeh, and Ofir Nachum.
Representation learning for reinforcement learning via bellman error
minimization.
In *International Conference on Machine Learning (ICML)*, 2020.
URL [https://arxiv.org/abs/2001.07301](https://arxiv.org/abs/2001.07301).
- Zhang et al. (2015)
Xiangyu Zhang, Jianhua Zou, Kaiming He, and Jian Sun.
Accelerating very deep convolutional networks for classification and
detection.
*IEEE transactions on pattern analysis and machine
intelligence*, 38(10):1943–1955, 2015.
- Zhang et al. (2023)
Yuan Zhang, Jianhong Wang, and Joschka Boedecker.
Robust reinforcement learning in continuous control tasks with
uncertainty set regularization.
In *Conference on Robot Learning*, pp. 1400–1424. PMLR, 2023.
- Zhou et al. (1996)
Kemin Zhou, John C. Doyle, and Keith Glover.
*Robust and Optimal Control*.
Prentice Hall, 1996.
- Zhou et al. (2023)
Ruida Zhou, Tao Liu, Min Cheng, Dileep Kalathil, PR Kumar, and Chao Tian.
Natural actor-critic for robust reinforcement learning with function
approximation.
*Advances in neural information processing systems*,
36:97–133, 2023.

## Appendix A Appendix

### A.1 Proof of Theorem 1

For the ease of notation, we denote the gap between the sampled system dynamics with uncertainty $A_{\mathcal{P}}$ and the reference system $A_{\mathcal{P}^{\circ}}$ as
$\epsilon_{A}:=A_{\mathcal{P}^{\circ}}-A_{\mathcal{P}}$.
Recall that $A_{\mathcal{P},r}$ denotes the low-rank manifold projection of $A_{\mathcal{P}}$ using truncated SVD.
Let $A_{\mathcal{P}^{\circ}}^{\dagger}$ and
$A_{\mathcal{P},r}^{\dagger}$ respectively denote the pseudo-inverses.
With $\theta^{\circ}=A_{\mathcal{P}^{\circ}}^{\dagger}b_{\mathcal{P}^{\circ}}$ and $\theta_{r}=A_{\mathcal{P},r}^{\dagger}b_{\mathcal{P}}$, the difference can then be written as:

$$ $\displaystyle\theta_{r}-\theta^{\circ}$ $\displaystyle=A_{\mathcal{P},r}^{\dagger}b_{\mathcal{P}}-\theta^{\circ}$ $\displaystyle=A_{\mathcal{P},r}^{\dagger}(b_{\mathcal{P}}-b_{\mathcal{P}^{\circ}})+A_{\mathcal{P},r}^{\dagger}A_{\mathcal{P}^{\circ}}\theta^{\circ}-\theta^{\circ}$ $\displaystyle=A_{\mathcal{P},r}^{\dagger}(b_{\mathcal{P}}-b_{\mathcal{P}^{\circ}})+A_{\mathcal{P},r}^{\dagger}A_{\mathcal{P}}\theta^{\circ}+A_{\mathcal{P},r}^{\dagger}\epsilon_{A}\theta^{\circ}-\theta^{\circ}$ $\displaystyle=A_{\mathcal{P},r}^{\dagger}(b_{\mathcal{P}}-b_{\mathcal{P}^{\circ}})+\sum_{i=1}^{r}v_{i}v_{i}^{T}\theta^{\circ}-\sum_{i=1}^{d}v_{i}v_{i}^{T}\theta^{\circ}+A_{\mathcal{P},r}^{\dagger}\epsilon_{A}\theta^{\circ}$ $\displaystyle=A_{\mathcal{P},r}^{\dagger}(b_{\mathcal{P}}-b_{\mathcal{P}^{\circ}})-\sum_{i=r+1}^{d}v_{i}v_{i}^{T}\theta^{\circ}+A_{\mathcal{P},r}^{\dagger}\epsilon_{A}\theta^{\circ}$ $$

where the fourth equation above follows from the fact that:

$$ $A_{\mathcal{P},r}^{\dagger}A_{\mathcal{P}}=V\Sigma_{\mathcal{P},r}^{-1}U^{\top}U\Sigma_{\mathcal{P}}V^{\top}=\sum_{i=1}^{r}v_{i}v_{i}^{T}\quad\text{and}\quad\sum_{i=1}^{d}v_{i}v_{i}^{T}=I.$ $$

Since the feature functions $\psi,\phi$ are Lipschitz with constant $L>0$, and that the uncertainty in environment dynamics are bounded from the underlying reference system with Wasserstein distance $W(\hat{\mathcal{P}}^{\circ}_{s,a},P_{s,a})\leq\epsilon$, all the components of matrix $A_{\mathcal{P}}$, say for example $\mathbb{E}_{\mathcal{P}}[\psi(s^{\prime})\psi(s^{\prime})^{\top}]$, can be upper bounded as follows,

$$ $\sup_{\mathcal{P}\in\mathcal{B}_{W}(\hat{\mathcal{P}}^{\circ},\epsilon)}\left\|\mathbb{E}_{\mathcal{P}}[\psi(s^{\prime})\psi(s^{\prime})^{\top}]-\mathbb{E}_{\hat{\mathcal{P}}^{\circ}}[\psi(s^{\prime})\psi(s^{\prime})^{\top}]\right\|=\mathcal{O}(L\epsilon)$ $$

where $\mathcal{B}_{W}(\hat{\mathcal{P}}^{\circ},\epsilon)$ is the Wassertein ball with radius $\epsilon>0$. By Assumption 0, $\mathcal{P}^{\circ}\in\mathcal{B}_{W}(\hat{\mathcal{P}}^{\circ},\epsilon)$, hence by triangle inequality:

$$ $\left\|\mathbb{E}_{\mathcal{P}}[\psi(s^{\prime})\psi(s^{\prime})^{\top}]-\mathbb{E}_{\mathcal{P}^{\circ}}[\psi(s^{\prime})\psi(s^{\prime})^{\top}]\right\|\leq 2\mathcal{O}(L\epsilon)$ $$

It follows that:

$$ $\displaystyle\|\theta_{r}-\theta^{\circ}\|_{2}$ $\displaystyle\leq\|A_{\mathcal{P},r}^{\dagger}(b_{\mathcal{P}}-b_{\mathcal{P}^{\circ}})\|_{2}+\|\sum_{i=r+1}^{d}v_{i}v_{i}^{T}\theta^{\circ}\|_{2}+2\mathcal{O}(L\epsilon)$ $\displaystyle\leq\frac{1}{\sigma_{\mathcal{P},r}}\|(b_{\mathcal{P}}-b_{\mathcal{P}^{\circ}})||_{2}+\|\sum_{i=r+1}^{d}v_{i}v_{i}^{T}\theta^{\circ}\|_{2}+2\mathcal{O}(L\epsilon)$ $$

For the second term, we use $\theta^{\circ}=V^{\circ}\Sigma^{-1}_{\mathcal{P}^{\circ}}{U^{\circ}}^{-1}b_{\mathcal{P}^{\circ},\omega}$ to get:

$$ $\displaystyle\left\|\sum_{i=r+1}^{d}v_{i}v_{i}^{\top}\theta^{\circ}\right\|_{2}$ $\displaystyle=\left\|\sum_{i=r+1}^{d}v_{i}\sum_{j=1}^{{r^{\circ}}}v_{i}^{\top}v^{\circ}_{j}\sigma_{\mathcal{P}^{\circ},j}^{-1}{u^{\circ}}_{j}^{\top}b_{\mathcal{P}^{\circ}}\right\|_{2}$ $\displaystyle\leq\left\|\sum_{i=r+1}^{d}\sum_{j=1}^{{r^{\circ}}}v_{i}^{\top}v^{\circ}_{j}\sigma_{\mathcal{P}^{\circ},j}^{-1}{u^{\circ}}_{j}^{\top}b_{\mathcal{P}^{\circ}}\right\|_{2}$ $\displaystyle=\sum_{i=r+1}^{d}\left\|v_{i}^{\top}v^{\circ}_{i}\sigma_{\mathcal{P}^{\circ},i}^{-1}{u^{\circ}}_{i}^{\top}b_{\mathcal{P}^{\circ}}+\sum_{j\neq i}^{{r^{\circ}}}v_{i}^{\top}v^{\circ}_{j}\sigma_{\mathcal{P}^{\circ},j}^{-1}{u^{\circ}}_{j}^{\top}b_{\mathcal{P}^{\circ}}\right\|_{2}$ $\displaystyle\leq\sum_{i=r+1}^{d}\|v_{i}^{\top}\|_{2}\|v^{\circ}_{i}\|_{2}\sigma_{\mathcal{P}^{\circ},i}^{p-1}+\sum_{i=r+1}^{d}\sum_{j\neq i}^{{r^{\circ}}}\|v_{i}^{\top}v^{\circ}_{j}\sigma_{\mathcal{P}^{\circ},j}^{-1}{u^{\circ}}_{j}^{\top}b_{\mathcal{P}^{\circ}}\|_{2}$ $\displaystyle\leq(d-r)\sigma_{\mathcal{P}^{\circ},r}^{p-1}+\sum_{i=r+1}^{d}\sum_{j\neq i}^{{r^{\circ}}}\|v_{i}^{\top}v^{\circ}_{j}\|_{2}\sigma_{\mathcal{P}^{\circ},j}^{p-1}$ $\displaystyle\leq(d-r)\sigma_{\mathcal{P}^{\circ},r}^{p-1}+(d-r){r^{\circ}}\sigma_{\mathcal{P}^{\circ},1}^{p-1}$ $$

It follows that:

$$ $\displaystyle\|\theta_{r}-\theta^{\circ}\|_{2}\leq\frac{1}{\sigma_{\mathcal{P},r}}\|b_{\mathcal{P}}-b_{\mathcal{P}^{\circ}}\|_{2}+(d-r)\sigma_{\mathcal{P}^{\circ},r}^{p-1}+(d-r){r^{\circ}}\sigma_{\mathcal{P}^{\circ},1}^{p-1}+2\mathcal{O}(L\epsilon)$ (15) $$

### A.2 Additional Result

#### A.2.1 Basic Settings

In all experiments, we evaluate the performance of benchmark algorithms on the Hopper-v3, Walker2d-v3, Humanoid-v3, and Ant-v3 environments from OpenAI Gym.
To ensure a fair comparison, we use the open-source implementation(^1^11[https://github.com/openai/spinningup](https://github.com/openai/spinningup)) of SAC as the base RL algorithm for all methods, and for RNAC we adopt its original PPO-based trainer without modification.
We use Adam as the optimizer in SAC, where both the policy and Q-networks are implemented as two-layer MLPs with hidden sizes $(64,64)$ and ReLU activation functions.
The learning rate for both networks is fixed at $3\times 10^{-3}$.
For our proposed algorithm, we set the truncation interval $d_{t}$ to $0.7\times 10^{6}$ for Walker2d and $10^{6}$ for Hopper, Humanoid, and Ant, meaning the model is truncated every $d_{t}$ policy optimization steps.
This choice ensures that rank adaptation occurs much less frequently than policy updates.

To impose a rank constraint on a weight matrix $W$, we first factorize it as $W=W_{1}W_{2}$ and apply singular value decomposition (SVD) to the product $W_{1}W_{2}=U\Sigma V^{\top}$.
We then reparameterize as

$$ $W_{1}=U_{[:,:\hat{r}]}\sqrt{\Sigma_{[:\hat{r}]}},\quad W_{2}=\sqrt{\Sigma_{[:\hat{r}]}}V_{[:\hat{r},:]},$ $$

where $\hat{r}\leq r$ is the target rank.
This projects $W$ onto a lower-rank manifold, thereby enforcing the constraint.
As shown in Figure [5](https://arxiv.org/html/2510.11899v1#A1.F5), inserting an intermediate linear layer (yellow, within the red region) provides an explicit implementation of this rank reduction.

Figure: Figure 5: To impose the low-rank constraint, we insert an intermediate linear layer (without activation functions or bias) between the original two layers. This layer acts as a bottleneck that enforces a low-rank factorization of the weight matrix via SVD approximation.
Refer to caption: x8.png

Additionally, to avoid loss of momentum after optimizer resets, we apply a standard cosine decay schedule with warm-up, as in (Lialin et al., 2023; Touvron et al., 2023).
Specifically, upon each reset, we set the learning rate to zero, gradually warm it up to the target value over 2000 steps, and then resume following the cosine schedule.

We present the practical implementation of our proposed algorithm in Alg. [1](https://arxiv.org/html/2510.11899v1#alg1).
At each iteration, we warm-start both the policy network and Q-network in SAC using the trained neural networks from the previous iteration, and then run SAC in the corresponding MuJoCo environment to continue training.

For the robust RL baselines, we use their official open-source implementations.
The implementation of RNAC is available at [https://github.com/tliu1997/RNAC](https://github.com/tliu1997/RNAC).
To modify the dynamics kernel, we follow the setting in OMPO Luo et al. (2024), using their codebase at [https://github.com/Roythuly/OMPO](https://github.com/Roythuly/OMPO).
For Parseval regularization, we use the implementation provided at [https://github.com/wechu/parseval_reg](https://github.com/wechu/parseval_reg).
In MuJoCo experiments with Parseval regularization, we adopt the same setup, tuning the regularization coefficient from $\{0.001,0.0001,0.00001\}$ and selecting the best-performing value.
We also follow the original implementation by setting $s=2$ in the Parseval constraint $\|WW^{\top}-sI\|_{F}$. For Tiwari et al. (2025), we follow their default configuration with a sparsification layer and set the hidden layer size to 1024 neurons, consistent with their original setting.

#### A.2.2 Model Uncertainty Setting

Following the setup in Luo et al. (2024), we simulate model uncertainty by introducing continuously varying environment parameters during training.
This design encourages policies to generalize across dynamic variations rather than overfitting to a fixed set of dynamics.
The specific parameter schedules for each environment are as follows:

- •
Hopper: The torso and foot lengths vary with the episode index $i$ as
$L_{\text{torso}}(i)=0.4+0.2\cdot\sin(0.2i),\quad L_{\text{foot}}(i)=0.39+0.2\cdot\sin(0.2i).$
- •
Walker2d: The torso and foot lengths follow a similar pattern with
$L_{\text{torso}}(i)=0.2+0.1\cdot\sin(0.3i),\quad L_{\text{foot}}(i)=0.1+0.05\cdot\sin(0.3i).$
- •
Ant: Gravity $g$ and wind speed $W$ change across episodes according to
$g(i)=14.715+4.905\cdot\sin(0.5i),\quad W(i)=1+0.2\cdot\sin(0.5i).$
- •
Humanoid: The same variation as Ant is applied, but the wind effect is amplified due to the humanoid’s larger mass and drag:
$g(i)=14.715+4.905\cdot\sin(0.5i),\quad W(i)=1+0.5\cdot\sin(0.5i).$

Figure: Figure 6: Visualization of uncertain dynamics in the Hopper-v3 task, where the torso and foot lengths vary across episodes.
Refer to caption: x9.png

#### A.2.3 Rank Convergence of the Alternative Algorithm

In this subsection, we conduct an ablation study to examine alternative strategies for selecting the cut-off rank of the SVD beyond Eq. [14](https://arxiv.org/html/2510.11899v1#S4.E14).
As reviewed by Falini (2022), numerous criteria have been proposed for truncated SVD.
Here, we consider a simple hard-thresholding approach based on the ratio between singular values.
Specifically, we define the cut-off rank as

$$ $\hat{r}\;=\;\min\Bigl\{\,\ell\in\{1,2,\dots,d\}\;\big|\;\frac{\sigma_{\ell}}{\sigma_{1}}\leq\beta\Bigr\}.$ (16) $$

Figure [7](https://arxiv.org/html/2510.11899v1#A1.F7) illustrates a fundamental limitation of this criterion.
After the initial iteration, the rank selection process stagnates because the rule in Eq. [16](https://arxiv.org/html/2510.11899v1#A1.E16) depends only on the largest singular value.
As a result, it ignores the broader spectral structure of the parameters and fails to adapt dynamically to spectral variations during training.
Therefore, we continue to use Eq. [14](https://arxiv.org/html/2510.11899v1#S4.E14) as our primary rule for selecting the cut-off rank.

Figure: (a) Walker2d
Refer to caption: x10.png

#### A.2.4 Policy performance under varying dynamics

In this subsection, we present additional experimental results under perturbations of physical hyperparameters (e.g., torso length, foot length) in the Hopper-v3 and Walker2d-v3 environments. As shown in Figure [8](https://arxiv.org/html/2510.11899v1#A1.F8), the proposed AdaRL algorithm exhibits superior robustness and outperforms the strongest baseline (Tiwari et al., 2025) in the majority of cases.

Figure: (a) Walker2d-v3: Varying torso length with fixed foot length
Refer to caption: x12.png

## Peer Reviews

confidence: 3
contribution: 2
initial_rating: 4
presentation: 3
questions: Q1) The approach remains the same for the all uncertainty levels. That is, suppose we are given two secnarios: First where model uncertainty is small and other where its large. The two scenarios  have different robust optimal policies, however the approach in the paper remains the same for two, hence rendering the same policy. How it can be tackled effectively.

Q2) The intuitive relation between low rank policies and robustness is interesitng, however I don't see sufficient concrete theoretical support for it.

Q3) How can this approach be extended to large MDPs in online settings?
soundness: 2
strengths: The idea of aproaching robust RL from low-rank perspective is potent.
summary: The guiding principal of the paper is 'simple models (non-overly fitted) generalizes better'.  The paper tries to learn low rank policy for the given nominal environment which may perform better for other environment in the uncertainty set. It adaptively learn the rank of the policy.
weaknesses: See questions below.

---

confidence: 3
contribution: 2
initial_rating: 2
presentation: 3
questions: 1. Line 112, specifying a "Discounted Markov decision process" helps provide additional clarity between this setting and the average reward setting.
2. Typo on line 142. It looks like there is a period instead of a comma.
3. It would be nice to see $\pi^*_\mathcal{P}$ defined precisely where it is first mentioned on line 148.
4. I would've liked to see all three assumptions listed in the main body prior to theorem 1.
5. Repeated/similar wording on lines 243 and 246.
6. Extra period on line 312.
7. $\tau$ is not defined in equation 12.
8. Line 740 mislabels the assumption.
soundness: 2
strengths: 1. The paper was neat, organized, and well written. It naturally follows and addresses some of the questions that would arise as a reader is going through it. The authors did a good job discussing the bias-variance tradeoff.
2. Adaptively controlling the policy's rank was a fascinating twist, allowing for the incorporation of conservatism during learning. Namely, I found the improved efficiency particularly interesting as it stemmed from repeatedly swapping between traditional policy optimization with a fixed rank to dynamically change the rank.
3. I found the way the algorithm was structured interesting. Namely, by swapping between the lower-level policy optimization and upper level adaptation of the rank and subsequent projection.
4. The experiments were conducted on varied environments, showing empirical validation for the author's claims. Additionally, the author backs up their claims with recent relevant work [1].

[1] Saket Tiwari, Omer Gottesman, and George Konidaris. Geometry of neural reinforcement learning in continuous state and action spaces. 2025.
summary: This paper aims to tackle the problem of epistemic uncertainty in robust reinforcement learning (RL) by providing a bi-level optimization framework that analyzes the bias-variance trade-off in entropy regularized RL. This framework combats the over-conservatism of a learned policy that often arises due to the nested min-max structure of the optimization problem. This is accomplished by optimizing under fixed, low-rank constraints when in the inner loop, then dynamically adjusting the rank in the outer loop and projecting the parameters on to a low-rank manifold.
weaknesses: 1. While I found the problem that this paper tackles interesting, the work surrounds a single theorem with minimal additional contributions while relying on some rather strong assumptions. Ultimately, this may not sufficiently capture the complex nuances of more complicated tasks.
2. There is a gap between the theoretical analysis and the practical implementation of the experiments. Namely, the theory analyzes rank $r$ of the covariance matrix in a linear model, but in Appendix A.4.1, the authors the authors instead constrain the rank of the weight matrix $W$ within a non-linear deep neural network. The authors perform a sanity check in figure 2, and while doing so is responsible, it is not a substitute for a formal theoretical argument.
3. The authors briefly discuss the difficulty of deriving convergence guarantees, however, such results are not presented in this work. Though an interesting problem to tackle, the only theoretical result being Theorem 1 leaves significant open questions in this topic.
4. On line 417, it is mentioned that environment dynamics vary across episodes. How are these dynamics varied?
5. Is the uncertainty set also compact?

---

confidence: 3
contribution: 3
initial_rating: 6
presentation: 3
questions: - Can you report performance under explicit worst-case dynamics for small tasks (e.g., via a tractable Wasserstein-DRO evaluation) to complement nominal and perturbed-physics tests?
- Do you adapt rank for both critics as well as the actor? Please include actor-only/critic-only/both ablations and a timing table for SVD/projection overhead.
- For RNAC, did you explore an SAC backbone or stronger PPO tuning to equalize compute and tuning budgets?
soundness: 3
strengths: - Clear positioning versus min–max robust RL: AdaRL avoids repeatedly solving the inner worst-case kernel by sampling from a Wasserstein ball and controlling model capacity via rank. This is computationally attractive and easy to implement in deep RL.
- Principled and practical rank search: the cumulative singular-value rule plus projection yields a usable upper-level step; a hard-threshold alternative is ablated and shown to stagnate.
- Simple enforcement: the rank constraint is implemented by inserting a linear bottleneck between layers; occasional SVD refines factors.
summary: The paper proposes AdaRL, a robust-RL method that improves generalization under epistemic dynamics uncertainty by adapting the rank of policy/critic representations during training. Uncertainty is modeled as a Wasserstein ball around a nominal kernel; trajectories are sampled from this ball rather than solving an inner worst-case problem. Learning is posed as a bi-level program: the lower level optimizes an entropy-regularized return under a fixed rank; the upper level selects the rank using a cumulative-spectrum rule and projects parameters to a low-rank manifold. Rank is updated at scheduled checkpoints. Experiments on MuJoCo (Hopper, Walker2d, Ant, Humanoid) show gains over fixed-rank SAC and robust baselines (RNAC, Parseval).
weaknesses: - Ambiguity-set realism and certification: sampling uniformly from a Wasserstein ball is less conservative than worst-case evaluation. It would be valuable to report worst-case performance or couple AdaRL with a tractable DRO evaluation to support robustness claims.
- Baseline parity: RNAC is presented with its PPO implementation while AdaRL uses SAC; this can confound comparisons. Reference implementations exist and should be matched where possible.
- Exact placement of rank constraints: clarify which actor/critic layers are factorized, and add ablations (actor-only, critic-only, both).

---

confidence: 3
contribution: 2
initial_rating: 4
presentation: 2
questions: * Is there a way to mitigate or handle the temporary performance drop observed after rank changes?
* The paper shows that the learned rank converges to the intrinsic rank of each task. Could you briefly explain how this intrinsic rank is defined? Is it a property of the environment, the policy representation, or both?
* In Equation (7), why not explicitly define the expectation over both sss and aaa? The current notation could be clarified for better readability.
* In Equation (11), what is the purpose of the projection operator if Equation (12) already constrains the policy to lie in $\mathcal{M}_r$?
soundness: 2
strengths: * The bias–variance trade-off (Theorem 1) under epistemic uncertainty is well-motivated and technically non-trivial. 
* The low-rank structure itself as a helpful prior can be useful to handle the uncertainty in the system. Interstingly, the results of the low-rank policy correspond to the rank of the task.
summary: The paper introduces AdaRL (Adaptive Rank Representation for Reinforcement Learning), a novel framework for robust RL under epistemic uncertainty. Traditional robust RL methods rely on nested min–max optimization, which is computationally expensive and tends to produce overly conservative policies. AdaRL instead leverages adaptive low-rank policy representations to align model complexity with the intrinsic dimensionality of the task.
weaknesses: * The main concern lies in the paper’s positioning. The authors claim that AdaRL is a principled alternative for robust RL under model uncertainty; however, it appears to be more accurately described as a method for solving (soft) robust RL problems rather than redefining the framework itself.
* The proposed method seems somewhat disconnected from the experimental design. While the algorithm assumes uniform sampling from a Wasserstein uncertainty set over dynamics, the experiments rely on predefined modifications to system parameters. This discrepancy weakens the rigour of the robustness claims.

* There is still room for improvement in presentation and clarity:

  - Figure 1 is blurry and mostly reiterates text descriptions; the framework could be visualized more intuitively.
  - Line 113: it should be **P** instead of **\mathcal{P}** in the MDP tuple.
  - $\theta$ is the parameter of the Q-function of the policy function? 
  - Theorem 1, can you further explain what is $b_{\mathcal{P},\omega^o}$  since in Equation 8 $\omega^o$ should be calculated based on $\mathcal{P}^o$ , and where is $\sigma_{\mathcal{P},r}$ defined? 

  - Figure 2 contains text that is too small to read.

  - Figure 3 includes unclear or incomplete titles.

* The experimental evaluation is limited to a small set of tasks and parameter variations. Including comparisons such as training with predefined fixed-rank policies would provide a stronger validation of the adaptive mechanism.