OCCAM: Online Continuous Controller Adaptation with Meta-Learned Models
Authors: Hersh Sanghvi
Spencer Folk
Camillo Jose Taylor
Venue: CoRL 2024 (2024)
Keywords: Controller Adaptation
Robot Model Learning
Meta-Learning
TL;DR: We present a framework for online controller adaptation that learns prior predictive models of system performance that quickly adapt to online data and can be used to to optimize control parameters for diverse physical and simulated robots.

Abstract: Control tuning and adaptation present a significant challenge to the usage of robots in diverse environments. It is often nontrivial to find a single set of control parameters by hand that work well across the broad array of environments and conditions that a robot might encounter. Automated adaptation approaches must utilize prior knowledge about the system while adapting to significant domain shifts to find new control parameters quickly. In this work, we present a general framework for online controller adaptation that deals with these challenges. We combine meta-learning with Bayesian recursive estimation to learn prior predictive models of system performance that quickly adapt to online data, even when there is significant domain shift. These predictive models can be used as cost functions within efficient sampling-based optimization routines to find new control parameters online that maximize system performance. Our framework is powerful and flexible enough to adapt controllers for four diverse systems: a simulated race car, a simulated quadrupedal robot, and a simulated and physical quadrotor.

OCCAM: Online Continuous Controller Adaptation with Meta-Learned Models
Hersh Sanghvi, Spencer Folk, Camillo Jose Taylor School of Engineering and Applied Science University of Pennsylvania
(hsanghvi, aitallerrtaylrrreaaasse @seas. upenn edu
Abstract: Control tuning and adaptation present a significant challenge to the us- age of robots in diverse environments. It is often nontrivial to find single set of control parameters by hand that work well across the broad array of environments and conditions that a robot might encounter. Automated adaptation approaches must utilize prior knowledge about the system while adapting to significant do- main shifts to find new control parameters quickly. In this work, present gen- eral framework for online controller adaptation that deals with these challenges. We combine meta- meta-learning with Bayesian recursive estimation to learn prior pre- dictive models of system performance that quickly adapt to online data, even when there is significant domain shift. These predictive models can be used as cost func- tions within efficient sampling-basee optimization routines to find new control pa- rameters online that maximize system performance. Our framework is powerful and flexible enough adapt controllers for four diverse systems: simulated race car, simulated quadrupedal robot, and simulated and physical quadrotor. The video and code can be found at https: h/thrshh0000....o gi thub. io/occam.
Keywords: Controller Adaptation, Robot Model Learning. Meta-Learning
1 Introduction
Robust and highly-performant control policies are critical for the successful application of robots diverse environ- ments. All control algorithms for robots have parameters that must be carefully tuned for good performance, but whose optimal values are generally not obvious priori and easily computed. In practice, these parameters are often tuned based on the designer's intuition and using heuristics that do not align with the actual downstream task of the robot-for instance, gain tuning quadrotor us- ing the step response. This problem is often exacerbated by the sim-to-real gap [1], requiring designers to fine-tune control parameters directly on the real system. Further- more, the optimal control parameters can vary time if the robot's physical attributes change or the robot en- ters new environments. Modern data-driven approaches that adapt to these changes usually perform online sys- tem identification (potentially in a latent space) for optimization- control or policy learning [2, 3, 5J.. These adaptive methods have shown strong results, though for many of these approaches, is difficult tune their behaviors after training, and their generalizability novel domains is un- clear.
System Daployment
Controller Oprimization
Adaptation
We demonstrate flexi- ble method for controller optimization based on online adaptation of meta- leameed models on diverse robots.
Figure 1:
8th Conference Robot Learning (CoRL. 2024), Munich, Germany.

---

Predictive Model
Controller Optimization
System Evaluation
Model Adaptation
Eroodo
Contest
Myal Alpua
Figure 2: An overview of our method and predictive model f. Given previous sensor mcasurements and inputs from the system, the Optimization phase uses the prediction model to search for the best gain to try next. This gain is sent to the real system for the Evaluation phase, during which sensor measurements and performance measures are collected. The Adaptation phase computes new weights to update the prediction model with the information gathered during the evaluation. Arrows in red indicate predicted and estimated quantities, while arrows in black indicate signals from the actual system.
In this paper we present OCCAM, a framework for online controller adaptation to new environ- ments using adaptable learned models of performance metrics. Our main contribution is applying meta-learning and online Bayesian regression train predictive models simulated data that pro- vide strong prior controller performance and readily adapted to data gathered during online operation. We show that we can use these models as cost functions to search for optimal control parameters in new environments unseen during training. One of the major advantages is the flexi- bility of our framework: it can easily be applied to any system with a parameterized controller. We demonstrate that our method performs online controller adaptation on a simulated race car equipped with a nonlinear control law, simulated and physical quadrotor micro-aerial-vehice (MAV) with model -based geometric controller, and simulated quadruped with neural-network locomotion policy.
2 Related Work
Automated Controller Tuning: Approaches for controller tuning often use Bayesian Optimization (BO) with Gaussian Process (GP) surrogate models [6. 7, 8, 9]. probabilistic sampling-based ap- proaches [10, 11, 12], or gradient descent through the controller and dynamics [13]. Prior data can be incorporated into BO by simply adding datapoints to the GP or specifying prior mean function. [14, 15] add prior datapoints to GP to find control policies for damaged robots online; [16, 17] instead use offline data to learn fixed kernel functions that make BO more efficient. These methods can degrade if the kernel or prior becomes inaccurate as the true underlying function changes. Our method is instead trained to accommodate these domain shifts.
Adaptation in Data-Driven Control: A closely-related line of works combines Reinforcement Learning (RL) for learning low-level control policies with system identification. Short-term histo- ries, or contexts, of states and actions are encoded as latent variables to condition either model-free policies [5, 18, 19, 20, 21] or dynamics models [22], allowing the model to modulate its behavior in different environments. [23, 24, 25, 26] instead learn repertoire of diverse policies for varying environments. The deep context encoders common in these works might not be accurate when the test system lies outside training distribution. A key idea in our method is to explicitly adapt the model to the observed performance online.
Meta-Learning for Robotics:
Meta-learning aims to deal with the lack of adaptability of learned models by incorporating the adaptation procedure into the training loss to train models that are well- suited to efficiently fit new data. [27, 28, 29] use meta-learning to train adaptable mean, kernel, and acquisition functions for BO with applications to controller tuning. [30] use Bayesian regression to meta-learn dynamics models that can be adapted employ meta-leaming to train low-level control policies with RI. that can be adapted quickly to new
dynamics test-time. [21, 31, 32, 33]

---

contexts. In this work, we alleviate overfitting issues common to these meta-learning methods [34] by adapting our model in a low-dimensional space and training it to characterize the uncertainty in its predictions.
Adaptive control generally seeks to augment the control inputs system to cancel out disturbances and align the system's closed-loop dynamics to those of desired reference model [35]. Both indirect and direct adaptive control approaches such as L1-Adaptive control, online model learning, and Model Reference Adaptive Control have successfully been applied to robotic aerial vehicles [36, 37, 38, 39, 40), manipulators [41], and bipedal robots [3, 42). Poor values of the tumable adaptive gains introduced by adaptive control can render the controller unstable [43]. also may not be possible for the actual system to track the chosen reference model, and adaptive control generally requires strong assumptions about model structure and differentiability [38, 39), which our method does not require.
Adaptive Control:
3 Problem Statement
In this work we address the problem of continuously adapting controller using combination of offline simulation data and data gathered from the robot online. The goal of the OCCAM procedure is then to find the control gains at iteration k; 9k, that maximize the reward over some upcoming time horizon AT that is chosen by the user:
g" arg max J (yk+aT) arg max JSO, (ge, Facs AT, Ueck AT, Tk))
Where we let 2 N denote vector containing the sensor observations from the system the k'th discrete timestep and Wı E R.N. denote the control inputs applied to the system by the controller. yk+AT E R.V, denotes a vector of performance measures, such as tracking error, control effort, or speed, computed as a fixed function of the control gains, measurements and control history over the horizon AT: yk+AT f(g,, & Mcc+AT, Th). In addition, we specify a reward function, J that maps the vector of performance measures onto a single scalar value: J(y) R.
The dynamics of our system governed by set of unobserved intrinsic and extrinsic system parameters such inertial parameters or friction coefficients. We refer to these system parameters collectively as lg. When generating data in simulation the designer would typically perform domain randomization by sampling possible values of to explore the behavior of the system in different regimes.
We also given a control policy, C (e.g. PID, MPC, etc.) that computes control inputs from pre- vious measurements: Ug+11 C.. (Tk, Zock, uo.k). C is parameterized by tunable gains 9k R%,, and Tke which is the control objective of the controller derived from the robot's task. For example, Tı could denote an encoding of the upcoming trajectory that we want a mobile robot to follow.
4 Method
4.1 Modeling Controller Performance
The foundation of OCCAM is predictive model ff which is designed to predict future performance measures function of the chosen control parameters. The structure of this prediction model is shown in Figure 2 (left). The recent measurements and control inputs (from sliding window of size H), H:k M are fed into a shallow fully connected network that produces a context encoding [5] that captures relevant, observable aspects of the system. This encoded result is com- bined with an encoding of the current task, Tk and the proposed control parameters, and fed into another fully connected network.
To quickly adapt to new data received from the robot during online deployment, the second fully connected network employs last linear layer structure [30, 44] wherein the network outputs
3

---

matrix @ E R,N,, that is combined with a weight vector E RN,, to produce the final pre- diction: 0, = 0 We can think of the columns of the matrix + = [oN.. as a set of basis functions weighted by wy. Here we use as an index instead of & to denote that this adaptation occurs at a different rate than the low-level control and sensing, with Al > Åk. Due to uncer- tainties in the model and measurements, we model w, as a normally distributed random variable: wy (14, Ex). Because ý is a linear transformation of wt obtain the following distribution for it: it (be &,E,D We use Kalman filter with identity dynamics to recursively estimate w as new data, (4), Mt), is obtained online from the system at test time:
E++1 Σx +
K Ex+1 Pe+1 (0++ 1 o + R)
W1+1 wt + KYy++1 1(+1
Σe+1 (I K Ex+1
Informally adopting standard Kalman Filter terminology. Q is positive definite matrix that scales with confidence in keeping w constant, while R captures the enror in our model, L, and the noise in the measurements of More information on this formulation can be found in the appendix.
4.2 Meta-Traine Trainir
While training our model on of- fline data, our goal is to produce learned basis functions that capture the variation in performance we ob- serve across different system param- eters, 6. Instead of estimating 0 di- rectly, our approach adapts in the la- tent space of w. To achieve this, after pretraining our network for few epochs using standard gradient descent, we switch gradient-based meta-learning [31], which consists of base learner which adapts the base model's initial parameters to data from different "scenarios", and an outer loop which adjusts the initial parameters of the base model using the loss achieved by the base learner after adaptation. Our base learner is the recursive Kalman update (2) ap- plied sequentially to n random data- points gathered from a system with new Because every computation in the Kalman Filter is fully differentiable, once the final weights Wn are computed, the gradient of the predictive loss can backpropagated through the full compu- tation graph to update the neural network parameters and the Kalman filter parameters wo, Eg, Q, R using standard automatic differentiation tools. Algorithm 1 shows how we the standard meta- learning loops with our Kalman Filter base learner as the inner loop. Meta-learning wo and Eo is similar in philosophy to Harrison et al. [30] and Bertinetto et al. [45], which also learn the prior distribution for the last-layer weights. However, they do not consider Kalman Filter formulation for regression, and thus do not include process and measurement covariance matrices Q and R.
Algorithm Meta-Learning with Kalman Filter Base Learner
Input: Training dataset D. network parameters v Wpre Output: Trained parameters v wo: Co, Q, R Initialize Wo = Wpree
Initialize Eg, I=
foreach training epoch do
foreach parameter dataset Dg (G, e
DD do
Sample subset Dimin of random size n from Do; Initialize wny- wp, Σ% = Σn
foreach datapoint (gi, Zi, Uj Tı, 91) in Drmin do Compute w/,, X using (2)
end
Compute 1 (, Z, U,U,T)D T)u, - YII:; foreach E { wo, Eo, Q, R} do
p + p - a VpL:
end
end
end
4.3 Controller Optimization With Learned Model
When running OCCAM online, want to use the leamned prediction model to solve (1) while also using our knowledge of the uncertainty on the model's predictions. More specifically, at regular

---

intervals OCCAM seeks to find a choice of control parameters that maximizes the uncertainty-aweet reward:
9" = arg max (((((n+AT))) + [J
where A = (g T; 2, u)yy the output of the prediction model, and a scalar penalty that controls the confidence bound. While many optimizers could be used, we employ a particle filter- based random search in our evaluation settings. In each search iteration, we generate a fixed number of random controller gains from uniform distribution G,- (gmin 9max)) We also propagate previous samples by applying N, random perturbations to the optimal gains found in the 7 previous trials, generating 7" x N, search samples Ge. The full set of samples is thus G = G, U G.. Each sample gain is concatenated with the upcoming command, and system history, and passed through the network to obtain the expected reward. After evaluating all samples, we pick the sample with the highest expected reward. This optimization approach introduces minimal hyperparameters and is computationally efficient; all search samples be collated into a single batch and evaluated with single forward pass through the network.
Hikk
Experimental Procedures
5.1 Simulated Evaluation Platforms
To show the flexibility of our framework, we evaluate it on three distinct simulated and one real robotic platform, described below. More details on each system and its accompanying controllers are provided in the supplementary material.
Our first simulated robotic system a 2-dimensional car racing around procedurally generated race tracks, modified from OpenAl's 's gym environment [46]. The car has unknown mass, engine power, and tire friction 6, with encoding of the race track shape as task OCCAM's goal is to continually optimize the six-dimensional control gains g of nonlinear PD steering controller that keeps the car on the centerline of the track, and a proportional acceleration controller that accelerates and brakes the car based on tunable speed thresholds. The speed thresholds improve performance but make the controller nonlinear and nondifferentiable. The reward function is linear combination the performance measures: inverse average lateral track- ing error, inverse average wheelslip, and average speed over the upcoming track segment. OCCAM adapts and selects new gains every time the car traverses & of a lap.
2D Race Car with Nonlinear Controller:
Quadrotor with Geometric Controller: Our second simulated platform is a quadrotor MAV equipped with a geometric trajectory tracking controller defined on SE(3) [47], implemented in RotorPy [48]. The quadrotor has five unknown system parameters / which are the quadrotor's true mass, principal moments of inertia, and thrust coefficient. The controller computes a feedforward motor speed command based reference trajectory using the quadrotor's nominal parameters (centered around those of the Crazyflic platform [49]) and feedback command to correct tracking errors. The controller is parameterized by eight PD gains on the position and attitude. The reward function is a linear combination of the inverse average positional tracking error, inverse average yaw tracking error, inverse average pitch and roll, and inverse average commanded thrust over the episode. The adaptation frequency AT is seconds. We also evaluate our framework on a physical Crazyflie with the same controller and performance measures. Details of the physical experiments described in Section 6.2.
Quadrupedal Robot with Learned Locomotion Policy: Our third simulated robotic platform is quadrupedal robot equipped with a static pretrained locomotion policy я trained using model-free RL [26]. π outputs joint angles at 50Hz such that the torso of the robot follows a velocity twist command 7k = (iiess jdess Waex). We treat π as our controller C for this system. Although v parameterized by deep neural network, it is also conditioned on an additional, eight-dimensionnll command gk that allows the user to specify high-level desired behaviors that the policy should fol- low: stepping firequency, body height, footswing height, stance width, and three discrete variables
5

---

which jointly specify the quadrupedal gait. We treat Jk as the controller parameters to be tuned au- tomatically based on the quadruped's randomized parameters and the twist command. The unknown system parameters 0% for the quadruped are added mass payloads to the robot base, motor strengths, and the friction and restitution coefficients the terrain. The adaptation frequency AT 3 seconds.
5.2 Model Training and Testing
To generate training data from each simulated platform, randomly sample N parameter vectors Op.N. For each 0,, we sample NB random samples of and drive the system to random initial dynamical state to collect history u, and finally roll-out the system with controller to gather performance metrics y, We train a separate basis-function network for each system.
We test each method on robots with system parameters sampled from outside of the range of the training dataset, testing the ability of each method to extrapolate outside of the training data. We evaluate methods on 15 out-of-distribution system parameters multiple different tasks (racetracks for the racecar, ellipsoidal trajectories for the quadrotor, and twist commands for the quadruped) and 8 random seeds. For each platform, we fix the reward functions J and uncertainty penalty B. Each test consists of multiple iterations of the "Optimize, Evaluate, Adapt" procedure shown in Figure 2. Because we run each test in an online setting with no resets between timesteps, if a method selects dangerous gains that cause a crash, the reward for every subsequent timestep of that evaluation is set zero. More details on dataset sizes, model architecture, and training and testing parameter ranges are provided in the supplementary material.
5.3 Baselines
We compare our method on each test system against three baselines. The first baseline is Reptile [50], a first-order version of the common gradient- bbseed meta-learning method MAML [31]. Reptile uses minibatched gradient descent as the base learner and it only outputs a single point prediction of y. For our second baseline, we compare against GP model whose kernel function is the com- position of network trained without meta-learning with RBF kernel, mirroring [17, 29]. We also compare against control parameters that were hand-tuned by experts for robust and conservative performance across variety of commands and system parameters. In our quadrotor example, we additionally consider an L1-Adaptive control [36] baseline adapted from [37], which augments the nominal control with an additional term based on an estimated disturbance vector.
To evaluate the usefulness of adapting the prediction model, we perform an ablation where we do not adapt from trial trial and only use wo and Ea (context-only) This ablation mimics methods that directly estimate latent system parameters from state and action histories, such as Kumar et al. [5], since the prediction network only uses the history context encoder and does not utilize our last- layer weight adaptation. To evaluate the usefulness of meta-training, we also run an ablation where lastrlayer weight adaptation occurs, but the network is trained without our meta-learnngg procedure (no-meta).
Results
In Table we report the crash rate and final average reward achieved by each method. To show the advantage of each method over simply using the expert-tuned nominal gains, in Figure we plot the difference between each method's reward and the nominal gain's reward over time in each test. We report the raw performance metrics achieved by each method in the supplementary material.
OCCAM outperforms both ablations and all baselines on all benchmarks, demonstrating the util- ity of both meta-learning basis functions and weight adaptation. Adaptation with our method also occurs within a few timesteps, which corresponds to 10-20 seconds of data on each system. The quadruped environment challenging in particular, as OCCAM must predict the performance of an expressive RL policy; nonetheless, it finds behavior parameters that improve the policy's 's perfor- mance. The quadrotor also presents unique challenges, the parameters of the quadrotors in the test

---

Table 1: Average Final Reward and Crash Rate Robotic Systems
Race Car
Quadrotor
Qundruped
Method
Avg Final Red
Crash & (L)
Avg. Final Rwd (?)
Crash % (U)
Avg. Final Red
Crash % (L)
Nominal
0.25 +
0.69 + 0.08
8.8
LK-GP
0.29 +: 0.14
24.2
1.11 + 0.34
29.9
0.74 +: 0.07
6.6
Reptile (50)
0.70 1: 0.09
Addddaeivee [26]
0.61
+ 0.25
33.6
OCCAM (no-mela)
0.1 1: 0.16
45,9
1.05
0.42
32.8
0.73 1. 0.09
8.1
OCCAM (context-only)
0.22 + 0.18
37,3
1.08
+ 0.38
27.3
0.73 + 0.07
6.7
OCCAM (Ours)
0.40 0.17
13.4
1.40
0.30
26.3
0.75 1: 0.07
6.3
Racing Car
Quadrotor
Quadruped
0.00
Normalzo Rewore
0.00
6.00
0.01
0.00
Datance Traveled Daps around track)
Time (S)
Time
Is:
(Ours)
OCCAM [Context Only)
OCCAM Meta)
Reptilc [51]
(a) Plots of performance reward value vs time on robotic systems (Higher Better)
Figure 3: Time Reward curves on all test systems. Our method shows robust prior performance and adaptation across all tests. For the robotic systems, normalized reward is computed by standard- izing rewards according to the training dataset statistics for that system and subsequently subtracting the nominal controller's reward.
set can vary from J/2 x to 3x the nominal parameters. In our problem setup, this must be compensated only by tuning PD feedback gains (the controller does not have an integral term). These extreme parametric errors also cause instability and poor tracking in the L1-adaptive controller. Meanwhile, our method finds gains that result in the lowest crash rate and nearly 50% reduction in tracking error compared to the nominal gains and L1-Adaptive controller. Both Reptile and LK-GP saturate early in the evaluations. Reptile tends to overfit to the training set and online training data [34] and gets stuck in a loop of selecting the same gains and repeatedly fitting to them.
In the supplementary material, we additionally show that OCCAM also performs well common global function optimization benchmarks, makes interpretable optimizations the gains system parameters change, and learns structured latent weight space.
6.1 Weight Adaptation Enables Out-of-Distribution Generalization
To further demonstrate the necessity of model adaptation for controller tuning in scenarios, we also evaluate each baseline on test scenarios that lie within the training distribution, and present a table of these results in the supplement. In these test cases, the context-one -oly baseline performs comparably to our full method. Therefore, its performance drop in the out-o--distr scenarios suggests that the learned context encoder alone is not sufficient achieve generalization. Similarly, our method outperforms LK-GP because the usefulness of the latent space of the kernel move out of its training distribution.
network degrades
To emphasize OCCAM's generalizability, we also use OCCAM to tune the controller gains a simulated Crazyflie to minimize tracking error three different wind conditions up to 0.5 m/s, when wind was not encountered at all during training. OCCAM is able to achieve an average tracking error of under 7cm, outperforming all baselines including L1-Adaptive control (10cm), shown in the supplementary material. Meanwhile, context-only achieves the worst tracking performance (30cm).
7

---

6.2 OCCAM crosses the Sim-to-Real Gap
Finally, we apply OCCAM to tune the gains of a physical Crazyflie quadrotor. The controller for the physical Crazyflie runs on a Robot Op- erating System (ROS) node running on lap- top with GPU and 1.6Ghz Intel i5 pro- cessor. The ROS node receives the quadrotor's position through Vicon motion-captur sys- tem' running at 100Hz and computes desired collective thrust and quaternion attitude com- mand for the quadrotor, which the onboard PID controller then converts into motor speeds. We train our predictive model simulation data sampled from both the train and test parameter ranges, and add OCCAM as a lightweight layer that runs every 3 seconds on top of the ROS control node.
Racking Enoor
Ereor Strat (m))
Thariig foor
60
Taaking
-44
Emor
60
-01
Time (5)
U1dddepeve
Figure 4: Positional tracking enror results on physical Crazyflie quadrotor following 3- dimensional ellipsoidal reference trajectory.
Position tracking error results for three-dimensional ellipsoidal reference trajectory are shown in Figure 4. Control parameters tuned by OCCAM result in lower tracking error than the expert-tuned nominal controller and achieve comparable tracking error to the L1-Adaptive controller. We also run additional experiments where 5 gram mass is added to the Crazyflie (see Figure 1), which represents a 20% increase from the its nominal mass. Our method is able to find gains that decrease the quadrotor's tracking error in the z-axis by 54% compared to the nominal controller and by 17% compared the C1-Adaptive controller. Figures of tracking errors from these physical Crazyflie experiments are shown in the supplementary material.
7 Discussion and Limitations
Our results demonstrate a single algorithm that optimizes controllers on a diverse array of robotic platforms using simple search techniques. Importantly OCCAM obviates the need for manual gain tuning that typically requires domain expertise, and instead provides automatic controller adaptation based on task-relevant metrics. OCCAM is also compatible with any controller with tunable param- eters, allowing straightforward application to controllers that are nonlinear, discontinuous, and not analytic. As shown in our Crazyflie experiment, OCCAM's 's models and optimization routine are lightweight enough to run in realtime without GPU. We expect OCCAM to perform best on sys- tems with a well-defined and relatively smooth mapping between control gains and performance; this and further assumptions are discussed in the appendix.
In this work, we tested OCCAM on commonly used controllers with relatively low-dimensional gain spaces, though one might also wish to tune a higher-dimensional policy, such as a neural network. Although the policy could be conditioned on a low-dimensional behavior vector, one could also treat the weights of either the entire neural network or its last few layers [51, 52] as the controller gains and use a more powerful optimizer.
Beyond this, there are a few areas in which our method could be improved. If the stochastic opti- mizer makes poor initial choice and the uncertainty penalty is poorly tuned, OCCAM sometimes settles early in a suboptimal local maximum. For some systems, this incorrect choice of gains can cause crashes. Incorporating more sophisticated optimizer and exploration strategy would be in- teresting future work. Lastly, in this work we do not rigorously evaluate how far the generalizability of our method applies, or changing the reward function J online. Future work includes addressing these limitations, along with demonstrating our method on broader classes of systems.
'nttss//www.vico..com/
8

---

Acknowledgments
The authors would like thank the reviewers, who gave helpful feedback during the discussion phase. This work was supported by NSF grant CCF-2112665 (TILOS).
References
[1] X. B. Peng, M. Andrychowiez, W. Zaremba, and P. Abbeel. Sim-to-Real Transfer of Robotic Control with Dynamics Randomization. In 2018 IEEE International Conference on Robotics and Automation (ICRA), pages 1-8, Brisbane, Australia, May 2018. IEEE Press. doi: 10. 109/ ICRA.2018. 8846052...
[2] G. Williams, N. Wagener, B. Goldfain, Drews, M. Rehg. B. Boots, and E. A. Theodorou. Information theoretic mpe for model-based reinforcement learning. In 2017 IEEE Inter- national Conference on Roboties and Automation (ICRA), pages 1714-1721, 2017. doi: 10.1 10.1109/CRA..211..79222
[3] Y. Sun, W. L. Ubellacker, W.-L. Ma, X. Zhang, C. Wang, N. V. Esomay-Shankiee M. Tomizuka, K. Sreenath, and A. D. Ames. Online Learning of Unknown Dynamics for Model-Based Controllers Legged Locomotion. IEEE Robotics and Automation Letters, (4):8442 8449, Oct. 2021. ISSN 2377-3766. 14::0..119/LRA.222220 110011.....1....
[4] T. Z. Jiahao, K. Y. Chee, and M. A. Hsieh. Online dynamics learning for predictive control with an application to aerial robots. In Conference on Robot Learning, pages 2251-2261. PMLR, 2023.
[5] A. Kumar, Z. Fu, D. Pathak, and J. Malik. RMA: Rapid Motor Adaptation for Legged Robots. arXiv:2107.04034 [es], July 2021.
[6] R. Calandra, A. Seyfarth, J. Peters, and M. P. Deisenroth. Bayesian optimization for learning gaits under uncertainty. Annals of Mathematics and Artificial Intelligence, 76(1):5-25 Feb. 2016. ISSN 1573-7470. 1111000007//10442-0155--466
[7] M. Neumann- Broning, A. Marco, D. Schwarzmann, and S. Trimpe. Data-efficient Auto-tuning with Bayesian Optimization: An Industrial Control Study. IEEE Transactions on Control Systems Technology, 28(3)773--740, May 2020. ISSN 1063-6536, 1558-0865, 2374-0159. doi: 2111:0011100TCST221188888881111
[8] A. Marco, F. Berkenkamp, P. Hennig, A. P. Schoellig, A. Krause, S. Schaal, and S. Trimpe. Virtual vs. Real: Trading Off Simulations and Physical Experiments in Reinforcement Learn- ing with Bayesian Optimization. In 2017 IEEE International Conference on Robotics and Automation (ICRA), pages 1557-1563, May 2017. doi: 11::00.110/1CRA222177.999
[9] Berkenkamp, A. P. Schoellig, and A. Krause. Safe Controller Optimization for Quadrotors with Gaussian Processes. In 20/6 IEEE International Conference Robotics and Automation (ICRA), pages 491-496, May 2016.
[10] A. Loquercio, A. Saviolo, and D. Scaramuzza. AutoTune: Controller Tuning for High-Speed Flight. IEEE Robotics and Automation Letters, 7(2):4432-4499, Apr. 2022. ISSN 2377-3766, 2377-3774. 10i:10.1 R0.9//RR..22.23166999999
[11] A. Romero, S. Govil, G. Yilmaz, Y. Y.nog, and D. Scaramuzza. Weighted maximum likelihood for controller tuning. In 2023 IEEE International Conference on Robotics and Automation (ICRA), pages 1334-1341, 2023.
[12] A. Schperberg, S. D. Cairano, and M. Menner. Auto-Tuning of Controller and Online Tra- jectory Planner for Legged Robots. IEEE Robotics and Automation Letters, pages 1-8. 2022. ISSN 2377-3766.

---

[13] S. Cheng, M. Kim, L. Song, C. Yang. Y. Jin, S. Wang, and N. Hovakimyan. DiffTune: Auto- Tuning through Auto-Differentiation. IEEE Transactions on Robotics, pages 1-17, 2024. ISSN 1941-0468. doi: 10. 00.11099/RO..22444429911
[14] A. Cully, Clune, D. Tarapore, and J.-B. Mouret. Robots that can adapt like animals. Nature, 1442.
521(7553):503-507, May 2015. ISSN 1476-4687. 1o: 1038/nature 4422..
[15] K. Chatzilygeroudis, V. Vassiliades, and J.-B. Mouret. Reset-free Trial-and- -rror Learning for Robot Damage Recovery. Robotics and Aulonomous Systems, 100, Feb, 2018. ISSN 09218890. doi: wo..101/j.rbbot.211...oo.
URL http: http://axxvvvv.. wwwp:///////aaoaaao/o/ooooee 141111213.........
[16] A. G. Wilson, Z. Hu, R. Salakhutdinov, and E. P. Xing. Deep kernel learning. In A. Gretton and C. C. Robert, editors, Proceedings of the 19th International Conference on Artificial Intel- ligence and Statistics, volume 51 of Proceedings of Machine Learning Research, pages 370- 378, Cadiz, Spain, 09-11 May 2016. PMLR. URL https://proceeddng" mlr. prres// v51 v51/wilson16. hmmm..
[17] A. Rai, R. Antonova, Meier, and C. G. Atkeson. Using simulation to improve sample- efficiency of bayesian optimization for bipedal robots, Journal of Machine Learning Research, 20(49):1-24, 2019.
[18] A. Kumar, Z. Li, Zeng, D. Pathak, K. Sreenath, and Malik. Adapting rapid motor adapta- tion for bipedal robots. In 2022 IEEE/RSJ International Conference Intelligent Robots and Systems (IROS), pages 1161-1168 2022. 10.: 000:000110//00444712222229999919
[19] T. Miki, J. Lee, Hwangbo, L. Wellhausen, V. Koltun, and M. Hutter. Learning robust per- ceptive locomotion for quadrupedal robots in the wild. Science Robotics, :(62))eab 2022. fo0:100.120/sccioottss...222202. URL https: http://wwwww.iiccee/e.. wrgeance..r/eeeeoooeeeeee httrg//oi/bb///1//
[20] H. Qi, A. Kumar, R. Calandra, Y. Ma, and J. Malik. In-Hand Object Rotation via Rapid Motor Adaptation. In Conference on Robot Learning (CoRL.,, 2022.
[21] W. Yu, J. Tan, Y. Bai, E. Coumans, and S. Ha. Learning Fast Adaptation with Meta Strategy Optimization, Feb. 2020.
[22] K. Lee, Y. Seo, S. Lee, H. Lee, and J. Shin. Context-aware dynamics model for gener- alization in model-based reinforcement learning. In H. D. III and A. Singh, editors, Pro- ceedings of the 37th International Conference on Machine Learning, volume 119 of Pro- ceedings of Machine Learning Research, pages 5757-5766. PMLR, 13-18 Jul 2020. URL https: //pppe/ed//occingg....m mlr wtt.sss/v11//eee200000 html.
[23] R. Yang, X. Sun, and K. Narasimhan. A generalized algorithm for multi-objective reinforce- ment learning and policy adaptation. In H. Wallach, H. Larochelle, A. Beygelzimer, d'Alché- Buc, Fox, and R. Garnett, editors, Advances in Neural Information Processing Systems, vol- ume 32. Curran Associates, Inc., 2019. URL http::tt/prcceediggggg wwwpeeiiiiiinnnnnnneeeeeeeeeeeee neurips cc/paper_ htlpe//pppp//////////////attttaa
Paper .df.
[24] B. Lim, L. Grillotti, L. Bernasconi, and A. Cully. Dynamics-Aware Quality-Diversity for Efficient Learning of Skill Repertoires. In 2022 International Conference on Roboties and Automation (ICRA), pages 5360-5366, May 2022. 100..01099/CRA44439922222222 1559.
[25] Zahavy, Y. Schroecker, F. Behbahani, K. Baumli, S. Flennerhag, S. Hou, and Singh. Discovering policies with DOMiNO: Diversity optimization maintaining near optimality. In The Eleventh International Conference Learning Representations, 2023. URL https:
/ //penreview www/ nett/forwm?dd-xkkddaasssss
10

---

[26] G. B. Margolis and P. Agrawal. Walk these ways: Tuning robot control for generalization with multiplicity of behavior. In 6th Annual Conference on Robot Learning, 2022. URL. https://openreviex http:r/wwwwee.ettttoouooiioooe 2od//forrmmddd5255efeee
[27] R. Pautrat, K. Chatzilygeroudis, and J.-B. Mouret. Bayesian optimization with automatic prior selection for data-efficient direct policy search. In 2018 IEEE International Conference on Robotics and Automation (ICRA), pages 7571-7578, 2018.
[28] M. Volpp. L. P. Fröhlich, K. Fischer, A. Doem, S. Falkner, F. Hutter, and C. Daniel. Meta- learning acquisition functions for transfer learning in bayesian optimization. In International Conference on Learning Representations, 2020. URL https: botpps://openiee /ttp::///ppe/ee.ee/e/eo net.//rrwme////mm id=ryeYpJSKwr.
[29] J. Rothfuss, D. Heyn, jinfan Chen, and A. Krause. Meta-learning reliable priors in the function space. In A. Beygelzimer, Y. Dauphin, Liang, and J. W. Vaughan, editors, Advances in Neural Information Processing Systems, 2021. URL https:///ppeeee http://ppneeeeeeeeee/frrreee id=H_ ad-H_qljL8t_A.
[30] J. Harrison, A. Sharma, and M. Pavone. Meta-learning priors for efficient online bayesian regression. In Algorithmic Foundations of Robotics XIII, pages 318-337, Cham, 2020. Springer International Publishing. ISBN 978-3-030-44051-0.
[3]] C. Finn, P. Abbeel, and S. Levine. Model-agnostic meta-learning for fast adaptation of deep networks. In D. Precup and Y. W. Teh, editors, Proceedings of the 34th International Con- ference on Machine Learning, volume 70 of Proceedings of Machine Learning Research, pages 1 1126-1135. PMLR, 06-11 Aug 2017. URI. https://pproceeding: mlr press/ #70/finn17a.httt
[32] A. Nagabandi. Deep Online Learning via Meta-Learning: Continual Adaptation for Model- Based RL. 2019.
[33] K. Rakelly, A. Zhou, C. Finn, S. Levine, and D. Quillen. Efficient off-policy meta- reinforcement learning via probabilistic context variables. In K. Chaudhuri and R. Salakhut- dinov, editors, Proceedings of the 36th International Conference on Machine Learning, vol- ume 97 of Proceedings of Machine Learning Research, pages 5331-5340. PMLR, 09-15 Jun 2019. URL https: http///pooeedn.s.n..// mlr grress/v9/rakkee html.
[34] J. Rajendran, A. Irpan, and E. Jang. Meta-Learning Requires Meta-Augmennation. In Ad- vances in Neural Information Processing Systems, volume 33, pages 5705-5715. Curran Associates, Inc., 2020. URL https://prooeeednngs www.igg..urrp..... wttp:cccppp///////aaa/ae/e 3e51900ebbbbb06300444466656666 Abstract. httll.
[35] K. Aström and Wittenmark. Adaptive control. Courier Corporation, 2008
[36] Z. Wu, S. Cheng. K. Ackerman, Gahlawat, Lakshmanan, P. Zhao, and N. Hov- akimyan. L1 adaptive augmentation for geometric tracking control of quadrotors. In Proceed- ings of the International Conference on Robotics and Automation, pages 1329-1336, Philadel- phia, PA, USA, 2022.
[37] K. Huang, R. Rana, A. Spitzer, G. Shi, and B. Boots. DATT: Deep adaptive trajectory tracking for quadrotor control. In 7th Annual Conference on Robot Learning, 2023. URL https: / //ppnreeview.o www.:/eeetfoounnee/// id=XEx-cnNer6.
[38] M. O'Connell, G. Shi, X. Shi, K. Arizzadenesheli, A. Anandkumar, Y. Yue, and S.-J. Chung- Neural-fly enables rapid learning for agile flight in strong winds. Science Robotics, 7(66); cabm6597, 2022. doi/abs/10. 1126/scirobotics. abm6997.
URL https: htte:://wwww http://wciineeeeo org/

---

[39] S. M. Richards, N. Azizan, J.-J. Slotine, and M. Pavone. Adaptive-Contro--Oiinete Meta- Learning for Nonlinear Systems, June 2021.
[40] K. Y. Chee, T. Z. Jiahao, and M. A. Hsieh. Knode-mpe: A knowledge-based data-driven predictive control framework for acrial robots. IEEE Robotics and Automation Letters, 7(2): 2819-2826, 2022.
[41] D. Zhang and Wei. A review on model reference adaptive control of robotic manipulators. Annual Reviews in Control, 43, 02 2017.
[42] Q. Nguyen and K. Sreenath. L1 adaptive control for bipedal robots with control Iyapunov function based quadratic programs. In 2015 American Control Conference (ACC), pages 862- 867, 2015, doi:10.1 109/ACC.2015..777888
[43] I. M. Y. Mareels, B. D. O. Anderson, R. R. Bitmead, M. Bodson, and S. Sastry. Revisiting the mit rule for adaptive control. IFAC Proceedings Volumes, 20:161-166, 1987.
[44] J. Snock, O. Rippel, K. Swersky, R. Kiros, N. Satish, N. Sundaram, M. Patwary, M. Prabhat, and R. Adams. Scalable bayesian optimization using deep neural networks. In F. Bach and D. Blei, editors, Proceedings of the 32nd International Conference on Machine Learning, vol- ume 37 of Proceedings of Machine Learning Research, pages 2171-2180, Lille, France, 07-09 Jul 2015, PMLR. URL https: / proceedings. mlr mwr. wttppp/ses/////eeeeeeee httm..
[45] L. Bertinetto, J. F. Henriques, P. H. S. Torr, and A. Vedaldi. Meta-leaminng with differentiable closed-fomm solvers. Technical Report arXiv: 805.08 136, arXiv, July 2019. URL http:// arxiv. arxiv.org/ale wwwgg/a////aaa 1105. 08136. 1805::800.008 [es, stat] type: article.
[46] G. Brockman, V. Cheung, L. Pettersson, J. Schneider, J. Schulman, J. Tang, and W. Zaremba. Openai gym, 2016.
[47] Lee, M. Leok, and H. McClamroch. Geometric tracking control quadrotor uav on se(3). In 49th IEEE Conference on Decision and Control (CDC), pages 5420-5425, 2010. 1001101109/CDC2200..577778
[48] S. Folk, J. Paulos, and V. Kumar. Rotorpy: A python-based multirotor simulator with aerody- namies for education and research. arXiv preprint urXiv:2306.04483: 04483, 2023.
[49] Crazyflie 2.1 Bitcraze. http://www..iiccaeeiipppeecccc/c
[50] A. Nichol, Achiam, and Schulman. On first-order meta-learning algorithms, 2018.
[51] A. Raghu, M. Raghu, S. Bengio, and O. Vinyals. Rapid learning or feature reuse? towards understanding the effectiveness of maml, 2020.
[52] Y. Lee, A. S. Chen, F. Tajwar, A. Kumar, H. Yao, P. Liang, and C. Finn. Surgical fine-tuning improves adaptation to distribution shifts. In The Eleventh International Conference on Learn- ing Representations, 2023. URL https: http///peeerreeeeeee/eeeeeeeeeee wwt/teetnunnne http://rumiiddAPPPPPP AH..
12

---

A
Table of Notation
Table 1: Table of Model Elements
Vector of sensed values
20ck
History of sensed values to date
ug
Vector of control inputs
Uo:k
History of control inputs to date
Mk+ AT
Performance metrics over time k to A: + AT
J
Reward function maps performance metrics to scalar
Ok
System parameters, intrinsic and extrinsic (unknown)
Tk
Control objective derived from current task
gk
Vector of control parameters
C
Control law us = C(gk, Tki 2ock, up.k) (grey-box)
AT
Frequency which OCCAM adapts and computes gains
B Discussion on Kalman Filter Formulation
In this section, we will discuss the aspects our formulation that lead to the choice of the Kalman Filter as our online estimator. First, we choose a linear output layer of our neural net- work: y = (gg, 2, u, r)w, which has been shown to improve model generalization meta-learning settings [1. 2]. Then, given data pairs (9,, ye) received in stream from the robot, solving for the optimal mapping s.t. fw uncertainty in the regression problem arising from our lack of knowledge of 0, combined with epis- temic uncertainty in the model and environmental noise. Therefore, we would like to estimate the posterior distribution of w to characterize the uncertainty in the model, especially for the purposes of downstream decision making using the network's 's predictions. This estimate also needs to be able to vary with time, since the environment parameters 0 could slowly change as the robot moves into new environment. Given the choice of linear measurement model and the additional factors above, this strongly resembles the time-varying parameter estimation problem, to which a linear Kalman Filter with identity dynamics is commonly applied. Representing w Gaussian random variable enables analytic computation of the posterior and keeps the filtering procedure differentiable, which enables our meta-leaming pipeline.
online linear regression problem. However, there is significant
C Applicability of OCCAM
Although our experiments show that OCCAM can be applied to a wide variety of systems, we do not yet have a precise theoretical characterization OCCAM's performance. However, we do make a few implicit assumptions about the kinds of systems that OCCAM would work well on.
1. For OCCAM work well, the closed loop dynamics of the system should allow a well- defined mapping between control gains and performance. This mapping does not have to be known, vary the system parameters vary, and can be stochastic. However, if it is too noisy nonsmooth, then the performance of the predictive model will degrade. For example, if the closed loop dynamics are chaotic, then the effect of control gains on system performance might be unpredictable. Although we make no assumptions about the shape the mapping, we expect OCCAM to work best if the mapping is relatively smooth.
2. We also assume that the offline dataset (generated through simulation or from previous experience) captures the important characteristics of how gains affect performance and how that mapping changes as the environment changes. Although our method can adapt a significant domain gap demonstrated by our experiments), if the domain gap is too large then the basis functions and prior weights learned from offline data will not be useful and our method will take many iterations to adapt.

---

3. The closed-loop system should be stable enough that suboptimal set of control gains (such as those deployed in the early iterations of adaptation) will not cause an immediate crash.
4. Our use of a sampling- based optimization method limits the dimensionality of gain spaces that our method can handle. However, we could use a more powerful optimizer to overcome this, and many model-base control methods of interest (PID, MPC, LQR, etc.) do have gain spaces that are low-dimensional.
D Details on Simulated Platforms
In this section we provide additional details about each of our simulated evaluation platforms, in- cluding two benchmark functions which are commonly used test global functional optimization algorithms.
D.1 Benchmark Functions
We first validate our method on randomized variations of two common global optimization bench- mark functions [3]. The first is the Branin function, which has 2D input space and ID output space:
f(x) = a(x² - bot cr1 - r)² + s(1 - t)cos (x1) +
We treat as system parameters the six constants that parameterize the shape of the Branin function:
= [a, b, c, s, t, r].
The second is the Hartmann function, which has 6D input space and ID output space:
((x) Σ 0,exp
Ag(a Prj)²
Where A and P are constant matrices, and we randomize over the 4-dimensional vector as system parameters.
For these benchmark functions, there
measured quantities 2o:k control actions Mo.k- We consider the inputs to the benchmark functions to be the "gains" gk. and the outputs of the functions to be the performance measures Yı- Therefore the data tuples for these functions consist of only the inputs æ and scalar "metrics" y = fxx). For these functions, the reward function is simply set to the negative of the scalar function values: J(y) = y. Because there is no history context, the context-only baseline in these two examples is simply our method without weight adaptation.
For the benchmark functions, regularization serve mean and kernel functions in GP. F-PACOH is ill-suited to our robotic tests due to the high dimensionality of the full input space to the networks, so we use the LK-GP baseline in our robotic experiments instead.
F-PACOH [4], which is based training neural networks with
D.2 2D Race Car
Our first simulated robotic system is 2-dimensional car racing around a track, modified from the OpenAl Gym "Car Racing" environment [5]. The environment models a powerful rear-wheel-drive with sliding friction, making control nontrivial while trying to maximize speed track. The system has three control inputs: uę [Ms, Ugy ws). The controller C' of the car consists of nonlinear proportional-plus-eriiinn -erivvvvvvvvvvee (PD) controller that computes steering input u, to steer the car towards the centerline of the track and simple control law that accelerates the car by force Ug on straightaways up to a maximum speed, or brakes the car by force up for corners above a certain curvature threshold. The tunable maximum speed and curvature braking thresholds induce saturation and branching behavior the controller. The sensor measurements of this system 23 [u. Wk. end, where is the forward velocity of the car, Wi is the angular velocity, and ena is the lateral distance between the and the track centerline. The controller C' uses 2 and an

---

estimate for track curvature, derived from a vector of upcoming track waypoints тІ to compute Wy as follows:
Us
+ Kas
{
Kpg-
if S Umax otherwise
Ug
(0,
Kgbb
if 2 Citresh otherwise
0,
The tunable parameters of this controller
g = [kpp kds kpg Kpb Umax Cihresh]
The racing car environment has three unknown system parameters 0 = [m, p, pl], which are respec- tively the mass of the car, the car's engine power, and the friction between the tires and track.
For the racing car, the evaluation function computes the vector of performance metrics
Σ Maa
ykck+
AT 1+ E, Clati I+E Ws
These are respectively the inverse average lateral tracking error, inverse of total number of timesteps during which a wheel was slipping, and average velocity over fixed evaluation horizon. We invert tracking error and wheelslip since, in general, they ought to be minimized. this case, the evalu- ation horizon is not a fixed AT but instead is however long it takes for the car to traverse a fixed distance on track. For online testing of this system, we set the reward function be weighted combination of the reward terms: [iy -o rybli].
The reasoning behind inverting certain performance measures is to focus the model onto gains which have good performance. In this racing car setting, for example, "good performance" equates to low tracking error, low wheelslip, and high speed. At test time, we will only select and deploy gains for which the network predicts good performance. Therefore, our preference is that the network should be very discriminating between two gains that have good performance, i.e. low tracking error and wheelslip. If two gains result in high tracking errors, do not about exactly predicting that difference, because we will not deploy those gains. However, during training time, using the Mean Squared Error Loss function has the opposite effect the network predicting that a gain will have a tracking error of Im vs 2m will result in a very large gradient, while the network mispredicting by small amount will have a small effect on the gradient. Inverting the performance measures for which lower values are better has the desired effect of increasing the distance between high performing gains, while compressing the "bad" gains into small region of the space. And because the inversion is invertible,
recover the original, interpretable metries if necessary.
D.3 Quadrotor with Model-Based Controller
Our second simulated platform is a quadrotor MAV equipped with geometric trajectory track- ing controller defined on SE(3) [6]. This controller takes in a reference trajectory т defined in the quadrotor's flat output space: position (Pa,Py, and yaw. The controller computes feedforward motor speed command based on Tj using the quadrotor's nominal mass, iner- tial tensor, thrust and drag torque coefficients. It then uses measurements from the quadrotor 2 [p2. Py: Pz: Ugz Uy, Uz. R) where R is the rotation matrix representation of attitude, to com- pute feedback commands to correct tracking errors. The controller is parameterized by PD gains on the 3D position and PD gains on the attitude:
Ikg kv. kR. Kn] (following the convention given by [6]). The quadrotor has five unknown system parameters which are the quadrotor's 's mass, princi- pal moments of inertia, and thrust coefficient: = [m, Leu- Iyg. I k, The baseline controller is only aware the nominal parameters, which are centered around those of the Crazyflie platform [7], and not the actual values. Thus, the feedback gains must be used compensate for this parametric error. For more detailed information about the quadrotor's dynamics and the controller derivation, see [6].
3

---

For this system, the four performance measures are the inverted average positional tracking error, inverted average yaw tracking error, inverted average pitch and roll, and inverted average com- manded thrust over the episode. Following the racing car example, we choose the reward function to a weighted combination of the terms of y: J(y) = Ej r,yli]. For this system, we set AT seconds.
For our quadrotor experiments, the commanded trajectories т consist of 3-dimensional ellipsoidal trajectories of varying radii and frequencies. Because of the simplicity of these trajectories, we do not have to provide information about 7 as input to the network for this system. We leave the incorporation of more general and complex trajectories to future work. use RotorPy [8] and its included SE(3) controller for quadrotor simulations. For this environment, we also evaluate our framework a physical quadrotor with the same controller and performance measures.
D.4 Quadrupedal Robot with Learned Locomotion Policy
Our third simulated robotic platform is quadrupedal robot equipped with a static pretrained lo- comotion policy т trained using model-free RL [9]. outputs joint angles such that the torso of the robot follows velocity twist command = (#ees (aaes) The policy takes as high- dimensional input measurements 2Ą the joint positions and velocities qu 4k. previous joint angle commands 4k-1, commands Ck2 timing reference variables, and estimated base velocity and ground friction. We treat π as our controller C for this system.
Although is parameterized by deep neural network, it is also conditioned on additional com- mand that allows the user to specify high-level behaviors that the policy should follow:
bk @
05md
gemd
gemd amd hend
cmd]
The three terms [oqmd 0gmm 0gmdd] jointly specify the quadrupedal gait, fond is the commanded stepping frequency, hemdd is the commanded body height, h is the commanded footswing height,
and gemd the commanded stance width. Thus, the policy tries follow the velocity command ck while satisfying the behavior constraints. In the original work by is quantity to be selected by human operator, while in this work treat by the controller parameters to be tuned automatically based on the quadruped's 's randomized parameters and the task cz. For details on how the learned policy is trained, see [9].
The randomized system parameters 6g for the quadruped are added mass payloads the robot base, motor strengths, and the friction and restitution coefficients of the terrain. Although the contains estimator module to regress the ground friction, it does not receive direct observations of any of these parameters.
For use in our method, we input only a network consisting of the estimated base linear and angular velocities and joint torques applied by the motors.
subset of 2 into our prediction model
The four performance measures for the quadruped are the inverted average velocity errors along each axis of the command and inverted total commanded torque over the evaluation horizon. For this system, we set the evaluation horizon AT = 3 seconds.
The reward function for the quadruped has the same form as the quadrotor system: J(y) Et ryy[o). All simulations are done using code and pretrained models from [9] and the Isaac Gym simulator [10].
E Model Training and Testing Details
E.1 Network Pretraining
We observe that if the basis function network & is initialized randomly, the inner-loop adaptation steps used in meta-training can become unstable and cause training to diverge. Other works that

---

History
Encoder Layers
Network Layen:
Nonlineurity
epecchs
epochs
Branin
[1,
ReLU
5
50
45
Hartmann
[32. 32]
ReLU
75
45
Racing Car 25
132, 32]
[32. 32, 321
ReLU
5
40
55
Quadrotor
25
[64, 64]
15
[64, 64, 64]
ReLU
15
50
40
Quadruped
164,
[64.
Table 2: Architecture and Training Hyperparameters for OCCAM Basis Function Network for all tested systems
History Butfer Size
Encoder Layers
Encoded Dim
Network: Layers
Nonlinearity
Meta Training Upochs
Inner Loop Steps
Branio
[16, 16]
ReLU
35
10
[32.
70
Racing Car
[32, 32]
15
[32. 32, 32]
ReL.U
70
10
Quadrotor
25
[64, 64]
25
164. 32]
ReLU
25
20
Quadruped
[64, 64]
[64, 64]
Table 3: Architecture and Training Hyperparameters for Reptile baseline for all tested systems
use meta-learning with closed-form inner-loop solvers use pre-trained networks [1, 2] speed up training and solve this problem. To mimic this approach in setting, divide our training process into two phases. In the first phase, the network is trained as an average model using standard stochastic gradient descent, with Wpre as fixed last layer that does not adapt to tasks. In the second phase, we initialize wi = Wpre and we switch to meta-training with the Bayesian recursive update as the inner loop optimizer. Eo. Qo. and Ro are initialized before the second training phase as identity matrices. Our no -oetaatt ablation uses network that is trained only with this first phase and not with our meta-learning procedure.
E.2 Dataset Generation
The datasets for the robotic systems each consist of 1500 batches of NB = 64 datapoints each. The hyperparameters cach dataset and network are provided in the supplementary material. Note that our method does not require sampling only optimal or high-performing gains to generate data only random ones. Thus, the dataset for each system consists of N batches of datapoints: ((g, T, 2, u, N Each of these batches is used as "task" for single inner loop during the meta-training process.
E.3 Model Architecture
We find that we are able to use small networks to model each system; the networks are all fully- connected networks that consist of 3 hidden layers with fewer than 64 hidden units, outputting between 5-20 bases, indicating that many of the robotic systems that ling can be effectively modeled with relatively small number of parameters. The exact network layer sizes and training hyperparameters are given in the supplementary material. All models are implemented and trained Py [11].
interested in control-
Architectural details and training hyperparameters for OCCAM's basis function network, Reptile, and F-PACOH are presented in Tables 2, and 4 respectively. The F-PACOH training hyperparam- eters were chosen in accordance with experiments conducted in the original paper.
Training and testing parameter ranges for each system evaluated in this work are shown in Tables 5, 7, 8, and 9. For the reward curves and tables shown in the main submission, test system parameters
Network Layers
Num fitting iters
Weight Decay
Prior Factor
Feature Dim
Branin
[32,32,32] [32, 32, 32]
2500
3e-5
0.06
5
Hartmann
2500
0.03
0.23
6
Table 4: Training Details for F-PACOH baseline for all tested systems
5

---

Training
Testing
Training
Testing
Parameter
low
high
low
high
Parameter
low
high
low
high
a
0.8
1.2
0.5
1.5
b
0.11
0.13
0.1
0.15
01
1.0
1.5
0.5
1.5
C
1.2
1.8
1
02
1.0
1.2
0.6
1.4
5.5
6.5
5
7
83
2.4
3.0
2.0
3.0
8
12
0₂
3.0
3.4
2.8
3.6
0.035
0.045
0.03
0.05
Table 6: Parameter ranges for Hartmann ex- periments
Table Parameter ranges for Branin experi- ments
Training
Testing
Training
Testing
Parameter
low
high
low
high
Parameter
low
high
low
high
Size
0.01
0.03
0,005
0.04
Mass (kg)
0.02
0.09
0.01
0.1
Engine Power
2.5e4
4.5e4
2e4
5e4
Ian (kg m²)
2e-6
9e-4
le-6
le-3
Friction Limit
250
450
200
500
Ty (kg m²)
2e-6
9e-4
1e-6
1e-3
L (kg m²)
2c-6
9c-4
le-6
le-3
Table 7: Parameter ranges for Racing Car. Note that these quantities are given in inter- nal units used by the simulator, not SI units.
kn (N((rad/s)
2e-8
8e-7
1e-8
le-6
Table Parameter ranges for Quadrotor
were sampled exclusively from the set difference of the parameter range and training parameter range.
Training
Testing
Parameter
low
high
low
high
Added Payload (kg)
-0.8
2.5
-1.0
4.0
Motor Strength Factor
0.9
1.0
0.8
1.1
Friction Coefficient
0.25
1.75
0.2
2.0
Restitution Coefficient
0.1
0.3
0.05
5.0
Table 9: Parameter ranges for Quadruped

---

Benchmark Function Results
Table 10: Average Final Obtained Value on Benchmark Systems
Average Value over Last 5 Trials (+)
Branin
Hartmann (x10-*)
F-PACOH [4]
2.26 0.70
3.30 + 4.55
Reptile [12]
3.47 + 11.79
1.14 + 1.98
OCCAM (no-meta)
1.80 + 0.77
7.42 + 11.4
OCCAM (context-only)
4.25 + 3.92
12.83 + 15.7
OCCAM (Ours)
1.65 + 0.49
3.14 + 5.97
Branin
Hartmann
10
200230
Acerage Objoctiv
300325
2,03000
3.0015
15
1mall
P-RACOM
Figure 1: Minima found on each benchmark function (Lower is Better)
We report the average final reward obtained by all methods the Branin and Hartmann benchmarks in Table 10, and show minima obtained by each method over time in Figure 1. Notably, our method performs well in both settings. In the Branin setting, OCCAM learns a good initialization and finds the best final minimum.
the Hartmann setting, even though OCCAM learns a relatively poor prior, it is able to adapt and find the same final minimum F-PACOH.
G Raw Performance Metrics
Figures 3, and 4 show the raw performance metrics obtained by each method on each system in the trials in which they did not crash. We note that each method is not directly optimizing for these raw metrics, but instead weighted combination of their normalized versions, good or bad performance in an individual metric in these plots does not necessarily translate to high or low reward in the plots reported in the paper. For example, in the Racing Car example, our method obtains a lower average speed than many other methods; however, this makes sense as, in the scalarized
Average Cross-track error
Average Wheelslip
Average Speed
OCCAM [Ours]
OCCAM (No Adapt)
OCCAM (No- Meta)
LK-GP
Reptiee
Nominal
Figure 2: Raw performance metrics obtained by each method on our out-of--istribution racing car test set in successful runs.

---

Transiational Tracking Error (m)
Aftifudo Frrorr (rad)
PilchyRoll Magnitude (rad)
Thrust (N)
OCGAN Douns
OCGAN Pic Adaep
DCCAR (No Votal
1,0
Roddic
Honeinll
Adcptiie
Figure 3: Raw Performance metrics obtained by each method on our out-of-distribution quadrotor test in successful runs.
Velocitty Error x (m/s)
Velocity Error (m/s)
Angular Velocity Error (radis)
Average Tarque
Michdapp
0CCAM (N--Hata)
Raw Performance metrics obtained by each method on our out-of-distribution quadruped car test set in successful runs.
Figure 4:
objective the model was optimizing for, the speed metric was weighted much lower than the tracking eπror metric. Also to faithfully report the raw metrics without the crashes skewing the averages, we filter out the runs that crashed. For example, in the quadrotor example, although Reptile performs well when it selects gains that don't result in crashes, its higher crash rate brings down its overall average reward.
H Additional Simulation Experiments
H.1 In-Distribution Experiments
Table 11: Average Final Reward and Crash Rate In- In-Distribution Robotic Systems
Race Car
Quadrotor
Quadruped
Method
Avg Final Rwd
Crash & (L)
Avg Final Rwd
Crash % (L)
Avg Final Rwd
Crash % (L)
Nominal
0.50 = 0
1.13 + 0.13
47.9
0.66 = 0.9
14.7
LKKGG
0.49 + 0.08
1.79 + 0.38
37.7
0.74 + 0.08
8.3
Reptile
0,42 = 0.13
1.19 + 0.37
33.8
0.72
CI-Adaptive
1.37 = 0.55
$7.5
OCCAM (context only)
0.47 :: 0.06
1.94 4: 0.26
32.5
0.76 :: 0.07
OCCAM (Ours)
0.44 + 0.19
1.82 =: 0.40
37.5
0.74 +: 0.09
We also run our method and each baseline on test sets randomly sampled from the training distri- butions for each the robotic systems (see Tables 7, 8, and 9). The average final obtained reward and crash rates are reported Table 11. The performances of each method naturally improve in this setting as the sampled system parameters lie closer to the nominal parameters, but in particular the context-onl. baseline, which only uses the fixed context encoder for sysid, and the LK-GP base- line both obtain amongst the highest rewards and perform similarly to OCCAM, showing, within the training distribution, these approaches perform well.
Also notable in this setting is that the C1-Adaptive controller obtains higher reward than the Nom- inal controller, demonstrating that the adaptive control does indeed improve performance when the deviation from the nominal dynamics is smaller. However, when the parametric error grows larger in the out-of distribbtti experiments in the main paper, the adaptive controller becomes unstable and reduces performance.

---

Figure 6
(a) Basis weights computed in the Racing Car Environment, projected into two dimensions and colored by the friction pa- rameter. The weights form dis- tinct clusters separated by differ- ent friction coefficients.
(b) Basis weights computed in the Quadruped Environment, projected into two dimensions and colored by the friction parameter. The weights form distinet clusters separated by different added base payloads.
(c) Basis weights computed in the Quadrotor Environment, pro- jected into two dimensions and colored by the friction parameter. The weights form distinct clus- ters separated by mass parame-
H.2 OCCAM Makes Interpretable Adaptations to the Gains
To elucidate that our method finds semantically meaningful gains, we run an additional experi- ment in the racing car environment where we sweep only friction coefficients across 3 differ- ent tracks and plot the average final gains chosen by OCCAM in Figure 5. As friction in- creases, OCCAM selects gains that cause the car to accelerate more aggressively and drive faster, while in the low friction regime, the gains tend towards slower driving (higher brake gain, lower speed in corners). Our method logically chooses more aggressive driving profile available traction increases, showing physically meaningful adaptation to changes in system parameters.
Stearing Gath
Acceloration Gain
H.3 Is there structure to the learned weight space?
3.5
We also include preliminary experiments demonstrating that the space of weights that OCCAM adapts in has meaningful structure. For each test set in the paper, we use t-SNE project the weights computed by OCCAM's regression procedure into two dimensions and plot the projected weights in Figures 6a, 6c, and 6b. Note that like the weight adaptation procedure, the I-SNE embedding procedure has no knowledge of the underlying system parameters. For each system, the values of the weights distinctly cluster according to the underlying system pa- rameters.
teaking
traking Thrmbak
0.55
0.90
0.55
9.90
Friction (unilless envitonment pat
Figure 5: Adapted gains found by our framework for cars with increasing fric- tion coefficients. For cars with higher friction coefficients, our model chooses gains that lead to faster and more ag- gressive driving. Both the low-end and high-end friction coefficients are out of the training distribution of the model.
H.4 Gain
Adaptation with Unseen Environmental Variations
We also run an experiment to assess the ability of our method to tune controllers while novel environmental parameters are varied. We use OCCAM to tune the controller gains of simulated Crazyflie in three different wind con- ditions up to 0.5 m/s to minimize the tracking error. Despite the fact that wind was not modeled at all during training, OCCAM is able to achieve an average tracking er- ror of under 7cm, outperforming all baselines, shown in Figure 7. Meanwhile, as ex- pected from the previous section, no-adapt fails to tune the controller in this unseen setting.
I Additional Physical Crazyflie Experiments

---

We ran additional experiments on the physical Crazyflie platform in which we added a 5-gram mass from the be- ginning of the experiment and in the middle of the ex- periment. Plots of the tracking error obtained by the controller with OCCAM's optimized gains, the nominal gains, and with the L1-Adaptive control augmentation are shown in Figures 8a and 8b. In both cases, OCCAM finds gains that result in more robust tracking in the 7-- axis. We hypothesize that because our predictive model is trained on data gathered from many quadrotors with var- ied masses, learns to select gains that better compensate for these variations.
Crazyfi e Tacking ng Erar n Wind
OCCAN IDual
Roppl: OCC/24 [ta-/dapt)
0.50
Figure 7: Positional tracking error re- sults on simulated Crazyflie quadrotor following a 3-dimensional ellipsoidal reference trajectory in windy conditions unseen in training. For both versions of OCCAM and Reptile, the control gains are updated every 3 seconds.
An interesting result are the minor, high frequency os- cillations observed in the Z-axis in Figure 8a and in the X- and Y-axes in Figure 8b towards the end of the ex- periment. These are most likely the result of marginally stable closed-loop attitude dynamics. One possible solution to this is augmenting the performance measures y and measurement vector with pitch and roll angular velocities, which might encourage the predictive model and optimizer select gains that do not result in oscillations. Another solu- tion is to add small random force perturbations to the training simulations so that marginally stable controllers achieve worse performance metrics. We leave exploring these additions to future work.
Paccng Engr
Tracking Erter
Tadking
Trecking Eree on)
Preking Ertr
Treckin
kaccin
00M
022
Time
Time 50)
(a) Results with 5-gram mass added from the start. (b) Results with S-gram mass addod at roughly 26s
Figure 8: Positional tracking error results physical Crazyflie quadrotor following 3-dimensional ellipsoidal reference trajectory, with added masses.
References
[1] K. Lee, S. Maji, A. Ravichandran, and S. Soatto. Meta-Learning With Differentiable Con- vex Optimization. 2019 IEEE/CVF Conference on Computer Vision and Pattern Recog- nition (CVPR), pages 10649-10657, Long Beach, CA, USA, June 2019. IEEE. ISBN 978- 1-72813-293-8. 100110001090C URL https://ieeexplorxe ieeeeorrreeeeee........ org/ document/8954109
[2] L. Bertinetto, J. F. Henriques, P. H. S. Torr, and A. Vedaldi. Meta-leaming with differentiable closed-form solvers. Technical Report 1.XXvv1805..0 136, arXiv, July 2019. URL http:// arxiv. wrwiv..org/bbs//8000 1805. 08136. 1805008 [cs, stat) type: article.
[3] L. C. W. Dixon and G. P. Szego. The global optimization problem: An introduction, 1978.
[4] Rothfuss, D. Heyn, jinfan Chen, and A. Krause. Meta-learning reliable priors in the function space. In A. Beygelzimer, Y. Dauphin, Liang, and J. W. Vaughan, editors, Advances in Neural Information Processing Systems, 2021. URL id=H _oj__tt_____.__
10

---

[5] G. Brockman, V. Cheung, L. Pettersson, J. Schneider, J. Schulman, J. Tang, and W. Zaremba. Openai gym, 2016.
[6] T. Lee, M. Leok, and N. H. McClamroch. Geometric tracking control of quadrotor se(3). In 49th IEEE Conference Decision and Control (CDC), pages 5420-5425, 2010. doi: Nof:10011099//DC22100.111111
[7] Crazyflie 2.1 Biteraze. https:/wwwwiiiccccccco/ooocccccs
[8] S. Folk, J. Paulos, and V. Kumar. Rotorpy: A python-based multirotor simulator with aerody- namics for education and research. arXiv preprint arXiv:2306.04485, 2023.
[9] G. B. Margolis and P. Agrawal. Walk these ways: Tuning robot control for generalization with multiplicity of behavior. In 6th Annual Conference Robot Learning, 2022. URL https://opearevee..ee i
[10] V. Makoviychuk, I.. Wawrzyniak, Y. Guo, M. Lu, K. Storey, M. Macklin, D. Hoeller, N. Rudin, A. Allshire, A. Handa, and G. State. Isaac Gym: High Performance GPU-Based Physics Simulation For Robot Learning, Aug. 2021.
[11] A. Paszke, S. Gross, Massa, Lerer, J. Bradbury, Chanan, Killeen, Z. Lin, N. Gimelshein, L. Antiga, A. Desmaison, A. Kopf, E. Yang. Z. De- Vito, M. Raison, A. Tejani, S. Chilamkurthy, B. Steiner, L. Fang, J. Bai, and S. Chintala. Pytorch: An imperative style, high-performance deep learning li- brary. In Advances in Neural Information Processing Systems 32, pages 8024- 8035. Curran Associates, Inc., 2019. URL. http://papers. heurips.cccpaper/ wewicchannimmmeen st hii-ppprffooaac-------ee-aaiinne
pdf.
[12] A. Nichol, J. Achiam, and J. Schulman. On first-order meta-learning algorithms, 2018.


Reviews:
[{"review_id": "dPYO15Afbk", "reviewer": "Reviewer_vgNE", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": 4, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "This paper addresses the challenge of control tuning and adaptation in robots operating across diverse environments. In particular, the authors propose a framework for online controller adaptation that integrates meta-learning with Bayesian recursive estimation. This core idea is to meta-train a predictive models to directly predict system performance given a set of control parameters and online adapt this model using collected data during system operation. The predictive models serve as cost functions in sampling-based optimization routines, enabling the online tuning of control parameters that maximize system performance. The framework’s efficacy is demonstrated on a variety of robotic systems including a real quadrotor.", "summary_of_recommendation": "The manuscript contains interesting ideas, which are validated by extensive simulations and hardware experiments.", "technical_quality": 4, "clarity_of_presentation": 4, "potential_impact": 3, "robotics_focus": 4, "quality_of_limitations": 3, "questions_for_rebuttal": "1.\tIs it possible to explain under what conditions the proposed approach (which relies on a model to directly predict control performance given control parameters) can be applied?\n2.\tPlease justify the use of Kalman filter (see my corresponding comment above).  In addition to explanation, is it possible to do some simulations to show that the proposed approach can almost recover the true weights $w$ (and the basis functions) online under some scenarios (e.g., a scenario that is included in the meta-training stage)\n3.\tHow can the variance $\\sigma$ in Section 4.3 be calculated? Also, instead of using “evolutionary modification to random-search”, can the authors use more precise language?\n4.\tIs the PD controller for the race car a “nonlinear” controller?\n5.\tIn Section 5.3, what is the meaning of “phase-1”?\n6.\tFor the racing car example, the performance improvement with increasing laps seems very little for all methods. What could be the reasons?\n7.\tIn Figure 4, according to the legend, nominal controller gives the best tracking performance. Is this expected?\n8.\tHow the adaptation to different tasks is demonstrated is not straightforward to me. Can the authors elaborate a little bit?", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "pnf3h4qvD4", "reviewer": "Reviewer_27Pw", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": 4, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "Paper introduces an interesting and effective way of adapting the controller parameters online.", "summary_of_recommendation": "In my opinion paper advances an important field of the online controller adaptaiton and shows convincing results.", "technical_quality": 4, "clarity_of_presentation": 3, "potential_impact": 3, "robotics_focus": 4, "quality_of_limitations": 3, "questions_for_rebuttal": "I would say that authors should work on fixing the presentation and notation issues, try to answer the questions from the review and introduce a more explicit statement of contributions and explain the adaptation timing better.", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "iraFrV8Pqp", "reviewer": "Reviewer_vgNh", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": 4, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "This paper proposed an online controller adaptation method to automate the parameter adaptation subject to dynamically changing environments and tasks. The contributions include (1) a prediction model that maps control parameters to their performance in a closed-loop system; (2) the usage of meta-learning to generalize the learned model to different system parameters; and (3) an parameter optimization approach that selects the optimal parameter for deployment. The approach has been validated thoroughly in simulation and experiments over different robotic systems and controllers.", "summary_of_recommendation": "The proposed method is novel and the results are relatively comprehensive. The authors should address the concerns listed above. Based on its current state, I recommend this paper to be weakly rejected.", "technical_quality": 3, "clarity_of_presentation": 3, "potential_impact": 3, "robotics_focus": 4, "quality_of_limitations": 3, "questions_for_rebuttal": "see my numbered questions above.", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: **Post-rebuttal metareview**:

This paper presents a novel approach to online adapt controllers' parameters. In particular, this paper uses meta-learning to train a predictor that predicts the closed-loop controller performance as a function of controller parameters, online fine-tunes such a predictor, and uses this predictor as a surrogate goal to optimize controller parameters. The idea is novel, and experiments are diverse. Most of the reviewers' concerns have been addressed in the rebuttal phase.

--------------------------------
**Pre-rebuttal metareview**:

Strengths:
1. The idea of using meta-learning to train a predictor to predict policy performance as a function of policy parameters is interesting and somewhat novel.
2. Comprehensive literature review. 
3. Diverse experiments.

Weakness: 
1. Need more analysis about the applicability of the proposed method: in what systems can we accurately predict the policy’s performance?
2. The choice and design of Kalman filter needs more justification.
3. Other issues mentioned by reviewers such as comparing with L1 adaptive control, adaptation timescale, notations, writing, etc.
Decision: Accept