Title: SOMBRL: Scalable and Optimistic Model-Based RL
Venue: NeurIPS 2025
Accepted: yes
Keywords: Reinforcement Learning

RL Regret bounds in Kernelized Setting

Deep RL
Abstract: We address the challenge of efficient exploration in model-based reinforcement learning (MBRL), where the system dynamics are unknown and the RL agent must learn directly from online interactions. We propose **S**calable and **O**ptimistic **MBRL** (SOMBRL), an approach based on the principle of optimism in the face of uncertainty. SOMBRL learns an uncertainty-aware dynamics model and *greedily* maximizes a weighted sum of the extrinsic reward and the agent's epistemic uncertainty.  SOMBRL is compatible with any policy optimizers or planners, and under common regularity assumptions on the system, we show that SOMBRL has sublinear regret for nonlinear dynamics in the (*i*) finite-horizon, (*ii*) discounted infinite-horizon, and (*iii*) non-episodic setting. Additionally, SOMBRL offers a flexible and scalable solution for principled exploration.  We evaluate SOMBRL on state-based and visual-control environments, where it displays strong performance across all tasks and baselines.  We also evaluate SOMBRL on a dynamic RC car hardware and show SOMBRL outperforms the state-of-the-art, illustrating the benefits of principled exploration for MBRL.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes a simple optimism-in-the-face of uncertainty algorithm for model based RL. The algorithm works with any given uncertainty estimate; in theory, kernel estimates suffice whereas in practice, popular ensemble based methods suffice. Experiments are compelling on relatively simple environments.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths: the paper is very nicely written, theory is correct, well exposed, and relations to prior work are made clear. I apologize for not saying more here, but overall the execution is quite flawless and I do not have much more to add. \n\nWeaknesses: though the strengths are apparent, I would say that there are two key weaknesses:\n\n(1) Originality. Here, I would like to distinguish between originality and novelty. I believe the authors that their paper may be \"novel\" in the sense that no one has pursued this direction before. However, it does not feel particular original in the sense that anyone who is familiar with the ideas of Eluder dimension and OFU can believe that such an algorithm can be done and implemented. The salient difference between SOMBRL and the Kakade 2020 algorithm is that the uncertainty term is now aapplied on a per-time step basis, rather than aggregated across timesteps. This simplifies the optimism computation, but at the expense of some small statistical suboptimality. I don't think I find this particularly surprising, at least theoretically, nor would I be surprised if the authors of Kakade would have recommended this approach had someone tried to implement their algorithm theoretically. Ultimately, maybe the best conclusion here is \"well, I guess somebody had to do it\" and the authors did. And it was well executed. But still, I am not at all surprised (and in my own research, have been implementing something similar as a baseline). Again, I am not claiming this algorithm exists in the literature, but it feels more like \"folklore principles\" instantiated finally executed. \n\n(2) I think the authors are taking liberties with the description \"scalable.\" By the standards of modern RL, the settings are incredibly toy. The reason this is salient is that one need to be convinced that todays uncertainty-based methods are indeed robust enough for more dextrous, high-dimensional, or visually complex tasks. And because this method relies on explicit ensembling as a form of uncertainty estimation, the major research question is \"to what extent to todays uncertainty estimates suffice to implement optimism.\" So far, I see the limited experimental domains as providing limited (though perhaps encouraging!) evidence.\n\nQUESTIONS:\n(1) How well do the authors suspect the method to perform in more complex tasks?\n(2) Have the authors found the algorithm to be highly sensitive to the use of uncertainty estimates? What happens if the uncertainty estimate becomes unreliable?\n(3) What would the authors regard as the most original (note: the distinction from originality and novelty) and most surprising aspects of their submission?\n\nLIMITATIONS:\nExperimental domains are limited in their difficulty and sophistication. Consider, e.g. the RoboMimic, Libero, Calvin Kitchen control suites.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 4, "IMPACT": 3, "ORIGINALITY": 2, "DATE": "2025-06-25"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe authors proposed SOMBRL (Scalable an Optimistic MBRL) approach based on the pricipl of optimism in the face of uncertainty. SOMBRL learns an uncerrtainty-aware model of the true dynamics and use it to find a policy maximizig te extrinsic reward and moddel uncertainty to stimulate the exploration. Furthermore, the authors provide thoretical sublinear regreet results,\n\nSTRENGTHS AND WEAKNESSES:\nStrength\n1. Theoretical Justification\n2. Simple and Scalable Methods\n3. High Compatibillity\n\nWeaknesss\n1. The figures and descriptions of the experiments are overly cluttered and lack clarity.\n2. There are some model-base RL method considering uncertainty based on the model for dynamics additionally.\n\nQUESTIONS:\n1. To avoid potential misunderstandings, I would like to clarify the following: In Equation (8), the formulation appears different from MBPO. When applying your method to MBPO, do you mean that the synthetic data generation still uses only the mean dynamics (unlike the original MBPO), and that uncertainty is incorporated in the rewards of both real and synthetic data used for training the Q-networks and the actor network?\n\n2. In model-based RL, I think that the predictive performance of the model is crucial in practice. Introducing additional rewards after rollout might lead to overestimation in the synthetic data. I am concerned this could actually degrade performance rather than improve it. I wonder what your thoughts are on this point, or if you have already addressed this issue in the paper, please let me know.\n\n3. The figures seem to have inconsistent line widths, which affects visual consistency. In Figure 3, for example, if this is indeed model-based policy optimization, the label on the left should be \"Dreamer.\" Personally, I feel consistency across figures would improve the presentation. Additionally, I do not quite understand the purpose of Figure 1.\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "CLARITY": 2, "IMPACT": 3, "ORIGINALITY": 2, "DATE": "2025-06-28"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper presents SOMBRL, a scalable and optimistic model-based reinforcement learning algorithm that combines extrinsic rewards with epistemic uncertainty to guide exploration. The key idea is to use a weighted sum of the reward and the model uncertainty during planning. The method is simple yet theoretically grounded, achieving reasonably good results across lots of RL settings. This work shows strong empirical performance in both simulation (state-based and visual control tasks) and a real-world RC car setting.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n- This is an important problem. Exploration in model-based RL is a key challenge for real-world deployment of RL systems.\n- The solution is simple but clear. This work proposes to learn an uncertainty model along with the environment model.\n- This work shows strong empirical results. The method shows consistent performance improvements across various high-dimensional benchmarks.\n\nWeaknesses\n- The abstract suggests that SOMBRL targets realistic settings, but the theoretical results still rely on continuity assumptions. While common in literature, these assumptions may not always hold in complex real-world environments.\n- The guarantees rely on well-calibrated uncertainty models, but how accurate these uncertainty models is not explored. Further, the intuition of why this method works was not clearly stated.\n\nQUESTIONS:\n- What is the key intuition for why maximizing epistemic uncertainty improves convergence and stability? Can this be interpreted as a kind of regularization?\n- How is the uncertainty model accuracy affect the results?\n\nLIMITATIONS:\n- The paper briefly mentions the computational overhead of learning a probabilistic model but does not thoroughly discuss the accuracy of uncertainty estimation and its potential impact.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-02"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper proposes a new model-based RL algorithm that incorporates systematic exploration through intrinsic rewards designed to drive the agent to more uncertain regions of the state space. The authors show that under certain regularity conditions, the proposed algorithm achieves a sublinear regret while being simpler and more scalable that previous contendors. Finally, the authors provide some experimental results on some RL benchmarks to validate their approach.\n\nSTRENGTHS AND WEAKNESSES:\nStrenghts: \n- It is nice that the paper provides explicit theoretical guarantees for the proposed algorithm under the regularity conditions considered. It is a bonus that the guarantees extend to the non-episodic settings. \n- The paper is relatively well written and easy to follow. \n\nWeaknesses: \n- First, the authors claim that they are the first to provide guarantees for RL with continuous state and action spaces. This inaccurate (at least not for the model-based setting); see The Statistical Complexity of Interactive Decision Making by Foster et al 2023. \n- The guarantees provided require strong assumptions on the model. This is not really highlighted as a limitation in the paper. \n- Including a measure of epistemic uncertainty as a bonus term to drive exploration is not new; see for example the paper Exploration by Random Network Distillation by Burda et al. 2018. So I am not really seeing the novelty here. \n- In the theory section, epistemic uncertainty is measure with the help of a kernel. In the experiment, the authors use an ensemble model to measure the uncertainty. This feels like testing a different algorithm to me. \n- The literature survey and comparison with existing work isn't exhaustive enough. The two papers above are a good example, and maybe also the paper Sample Efficient Deep Reinforcement Learning via Local Planning by Yin et al. 2023. \n- The experimental results aren't convincing as far as exploration goes. For example, the score on Montezuma's revenge caps at 2500, which is less the model-free RL algorithms in the last two paper mentioned above. So this is does not make a good case for the model based approach of the paper.\n\nQUESTIONS:\nCan you examples of setting where Assumption 5.2 holds?\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 2, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 1, "DATE": "2025-07-02"}