OCCAM: Online Continuous Controller Adaptation with Meta-Learned Models

Authors: hersh  sanghvi
spencer  folk
jose  taylor
Venue: robot-learning_org_CoRL_2024_Conference
Published: 2024-06-06 06:20:23

Abstract: Control tuning and adaptation present a significant challenge to the usage of robots in diverse environments. It is often nontrivial to find a single set of control parameters by hand that work well across the broad array of environments and conditions that a robot might encounter. Automated adaptation approaches must utilize prior knowledge about the system while adapting to significant domain shifts to find new control parameters quickly. In this work, we present a general framework for online controller adaptation that deals with these challenges. We combine meta-learning with Bayesian recursive estimation to learn prior predictive models of system performance that quickly adapt to online data, even when there is significant domain shift. These predictive models can be used as cost functions within efficient sampling-based optimization routines to find new control parameters online that maximize system performance. Our framework is powerful and flexible enough to adapt controllers for four diverse systems: a simulated race car, a simulated quadrupedal robot, and a simulated and physical quadrotor.

Main paper: 
 introduction: 
 Figure 1 : We demonstrate a flexible method for controller optimization based on online adaptation of metalearned models on diverse robots.Robust and highly-performant control policies are critical for the successful application of robots in diverse environments. All control algorithms for robots have parameters that must be carefully tuned for good performance, but whose optimal values are generally not obvious a priori and are not easily computed. In practice, these parameters are often tuned based on the designer's intuition and using heuristics that do not align with the actual downstream task of the robot-for instance, gain tuning a quadrotor using the step response. This problem is often exacerbated by the sim-to-real gap [1] , requiring designers to fine-tune control parameters directly on the real system. Furthermore, the optimal control parameters can vary over time if the robot's physical attributes change or the robot enters new environments. Modern data-driven approaches that adapt to these changes usually perform online system identification (potentially in a latent space) for optimization-based control or policy learning [2, 3, 4, 5] . These adaptive methods have shown strong results, though for many of these approaches, it is difficult to tune their behaviors after training, and their generalizability to novel domains is unclear.Figure 2 : An overview of our method and predictive model f . Given previous sensor measurements and inputs from the system, the Optimization phase uses the prediction model to search for the best gain to try next. This gain is sent to the real system for the Evaluation phase, during which sensor measurements and performance measures are collected. The Adaptation phase computes new weights to update the prediction model with the information gathered during the evaluation. Arrows in red indicate predicted and estimated quantities, while arrows in black indicate signals from the actual system.In this paper we present OCCAM, a framework for online controller adaptation to new environments using adaptable learned models of performance metrics. Our main contribution is applying meta-learning and online Bayesian regression to train predictive models on simulated data that provide a strong prior on controller performance and are readily adapted to data gathered during online operation. We show that we can use these models as cost functions to search for optimal control parameters in new environments unseen during training. One of the major advantages is the flexibility of our framework: it can easily be applied to any system with a parameterized controller. We demonstrate that our method performs online controller adaptation on a simulated race car equipped with a nonlinear control law, a simulated and physical quadrotor micro-aerial-vehicle (MAV) with a model-based geometric controller, and a simulated quadruped with a neural-network locomotion policy. 
related work: 
 Automated Controller Tuning: Approaches for controller tuning often use Bayesian Optimization (BO) with Gaussian Process (GP) surrogate models [6, 7, 8, 9] , probabilistic sampling-based approaches [10, 11, 12] , or gradient descent through the controller and dynamics [13] . Prior data can be incorporated into BO by simply adding datapoints to the GP or specifying a prior mean function. [14, 15] add prior datapoints to a GP to find control policies for damaged robots online; [16, 17] instead use offline data to learn fixed kernel functions that make BO more efficient. These methods can degrade if the kernel or prior becomes inaccurate as the true underlying function changes. Our method is instead trained to accommodate these domain shifts.Adaptation in Data-Driven Control: A closely-related line of works combines Reinforcement Learning (RL) for learning low-level control policies with system identification. Short-term histories, or contexts, of states and actions are encoded as latent variables to condition either model-free policies [5, 18, 19, 20, 21] or dynamics models [22] , allowing the model to modulate its behavior in different environments. [23, 24, 25, 26] instead learn a repertoire of diverse policies for varying environments. The deep context encoders common in these works might not be accurate when the test system lies outside training distribution. A key idea in our method is to explicitly adapt the model to the observed performance online.Meta-Learning for Robotics: Meta-learning aims to deal with the lack of adaptability of learned models by incorporating the adaptation procedure into the training loss to train models that are wellsuited to efficiently fit new data. [27, 28, 29] use meta-learning to train adaptable mean, kernel, and acquisition functions for BO with applications to controller tuning. [30] use Bayesian regression to meta-learn dynamics models that can be adapted to new dynamics at test-time. [21, 31, 32, 33] employ meta-learning to train low-level control policies with RL that can be adapted quickly to new contexts. In this work, we alleviate overfitting issues common to these meta-learning methods [34] by adapting our model in a low-dimensional space and training it to characterize the uncertainty in its predictions.Adaptive Control: Adaptive control generally seeks to augment the control inputs to a system to cancel out disturbances and align the system's closed-loop dynamics to those of a desired reference model [35] . Both indirect and direct adaptive control approaches such as L1-Adaptive control, online model learning, and Model Reference Adaptive Control have successfully been applied to robotic aerial vehicles [36, 37, 38, 39, 40] , manipulators [41] , and bipedal robots [3, 42] . Poor values of the tunable adaptive gains introduced by adaptive control can render the controller unstable [43] . It also may not be possible for the actual system to track the chosen reference model, and adaptive control generally requires strong assumptions about model structure and differentiability [38, 39] , which our method does not require. 
problem statement: 
 In this work we address the problem of continuously adapting a controller using a combination of offline simulation data and data gathered from the robot online. The goal of the OCCAM procedure is then to find the control gains at iteration k, g k , that maximize the reward over some upcoming time horizon ∆T that is chosen by the user:EQUATIONWhere we let z k ∈ R Nz denote a vector containing the sensor observations from the system at the k'th discrete timestep and u k ∈ R Nu denote the control inputs applied to the system by the controller. y k+∆T ∈ R Ny denotes a vector of performance measures, such as tracking error, control effort, or speed, computed as a fixed function of the control gains, measurements and control history over the horizon ∆T : y k+∆T = f (g k , z k:k+∆T , u k:k+∆T , τ k ). In addition, we specify a reward function, J that maps the vector of performance measures onto a single scalar value: J(y) ∈ R.The dynamics of our system are governed by a set of unobserved intrinsic and extrinsic system parameters such as inertial parameters or friction coefficients. We refer to these system parameters collectively as θ k . When generating data in simulation the designer would typically perform domain randomization by sampling possible values of θ to explore the behavior of the system in different regimes.We are also given a control policy, C (e.g. PID, MPC, etc.) that computes control inputs from previous measurements:u k+1 = C g k (τ k , z 0:k , u 0:k ).C is parameterized by tunable gains g k ∈ R Ng , and τ k , which is the control objective of the controller derived from the robot's task. For example, τ k could denote an encoding of the upcoming trajectory that we want a mobile robot to follow. 
modeling controller performance: 
 The foundation of OCCAM is a predictive model f which is designed to predict future performance measures as a function of the chosen control parameters. The structure of this prediction model is shown in Figure 2 (left). The recent measurements and control inputs (from a sliding window of size H), z k-H:k , u k-H:k , are fed into a shallow fully connected network that produces a context encoding [5] that captures relevant, observable aspects of the system. This encoded result is combined with an encoding of the current task, τ k and the proposed control parameters, g k and fed into another fully connected network.To quickly adapt to new data received from the robot during online deployment, the second fully connected network employs a last linear layer structure [30, 44] wherein the network outputs a matrix Φ ∈ R Ny×N b that is combined with a weight vector w t ∈ R N b to produce the final prediction: ŷt = Φ t w t . We can think of the columns of the matrix Φ = [ϕ 1 , ...ϕ N b ] as a set of basis functions weighted by w t . Here we use t as an index instead of k to denote that this adaptation occurs at a different rate than the low-level control and sensing, with ∆t > ∆k. Due to uncertainties in the model and measurements, we model w t as a normally distributed random variable:w t ∼ N (µ t , Σ t ).Because ŷ is a linear transformation of w, we obtain the following distribution for it: ŷt ∼ N (Φ t µ t , Φ t Σ t Φ T t ). We use a Kalman filter with identity dynamics to recursively estimate w as new data, (Φ t , y t ), is obtained online from the system at test time:EQUATIONInformally adopting standard Kalman Filter terminology, Q is a positive definite matrix that scales with our confidence in keeping w constant, while R captures the error in our model, Φ, and the noise in the measurements of y. More information on this formulation can be found in the appendix. ′ 0 = w 0 , Σ ′ 0 = Σ 0 ; foreach datapoint (g i , z i , u i , τ i , y i ) in D train do Compute w ′ i , Σ ′ i using (2) end Compute L = ∥Φ ψ (G, Z, U, τ )w ′ n -Y ∥ 2 ; foreach p ∈ {ψ, w 0 , Σ 0 , Q, R} do p ← p -α∇ p L ; end end endWhile training our model on offline data, our goal is to produce learned basis functions that capture the variation in performance we observe across different system parameters, θ. Instead of estimating θ directly, our approach adapts in the latent space of w. To achieve this, after pretraining our network for a few epochs using standard gradient descent, we switch to gradient-based meta-learning [31] , which consists of a base learner which adapts the base model's initial parameters to data from different "scenarios", and an outer loop which adjusts the initial parameters of the base model using the loss achieved by the base learner after adaptation. Our base learner is the recursive Kalman update (2) applied sequentially to n random datapoints gathered from a system with new θ. Because every computation in the Kalman Filter is fully differentiable, once the final weights w n are computed, the gradient of the predictive loss can be backpropagated through the full computation graph to update the neural network parameters and the Kalman filter parameters w 0 , Σ 0 , Q, R using standard automatic differentiation tools. Algorithm 1 shows how we use the standard metalearning loops with our Kalman Filter base learner as the inner loop. Meta-learning w 0 and Σ 0 is similar in philosophy to Harrison et al. [30] and Bertinetto et al. [45] , which also learn the prior distribution for the last-layer weights. However, they do not consider a Kalman Filter formulation for regression, and thus do not include process and measurement covariance matrices Q and R.When running OCCAM online, we want to use the learned prediction model to solve (1) while also using our knowledge of the uncertainty on the model's predictions. More specifically, at regular intervals OCCAM seeks to find a choice of control parameters that maximizes the uncertainty-aware reward:g * = arg max g µ(J(ŷ k+∆T )) + βσ(J(ŷ k+∆T ))where ŷk+∆T = Φ(g k , τ, z, u)w k is the output of the prediction model, and β is a scalar penalty that controls the confidence bound. While many optimizers could be used, we employ a particle filterbased random search in our evaluation settings. In each search iteration, we generate a fixed number of random controller gains from a uniform distribution G r = U(g min , g max ). We also propagate previous samples by applying N r random perturbations to the optimal gains found in the T previous trials, generating T × N r search samples G e . The full set of samples is thus G = G r ∪ G e . Each sample gain is concatenated with the upcoming command, τ k and system history, z k-H:k , u k-H:k , and passed through the network to obtain the expected reward. After evaluating all samples, we pick the sample with the highest expected reward. This optimization approach introduces minimal hyperparameters and is computationally efficient; all search samples can be collated into a single batch and evaluated with a single forward pass through the network.5 Experimental Procedures 
simulated evaluation platforms: 
 To show the flexibility of our framework, we evaluate it on three distinct simulated and one real robotic platform, described below. More details on each system and its accompanying controllers are provided in the supplementary material.2D Race Car with Nonlinear Controller: Our first simulated robotic system is a 2-dimensional car racing around procedurally generated race tracks, modified from OpenAI's gym environment [46] . The car has unknown mass, engine power, and tire friction as θ, with an encoding of the race track shape as task τ . OCCAM's goal is to continually optimize the six-dimensional control gains g of a nonlinear PD steering controller that keeps the car on the centerline of the track, and a proportional acceleration controller that accelerates and brakes the car based on tunable speed thresholds. The speed thresholds improve performance but make the controller nonlinear and nondifferentiable. The reward function is a linear combination of the performance measures: inverse average lateral tracking error, inverse average wheelslip, and average speed over the upcoming track segment. OCCAM adapts and selects new gains every time the car traverses 1 3 of a lap. Quadrotor with Geometric Controller: Our second simulated platform is a quadrotor MAV equipped with a geometric trajectory tracking controller defined on SE(3) [47] , implemented in RotorPy [48] . The quadrotor has five unknown system parameters θ which are the quadrotor's true mass, principal moments of inertia, and thrust coefficient. The controller computes a feedforward motor speed command based on reference trajectory τ using the quadrotor's nominal parameters (centered around those of the Crazyflie platform [49] ) and a feedback command to correct tracking errors. The controller is parameterized by eight PD gains on the position and attitude. The reward function is a linear combination of the inverse average positional tracking error, inverse average yaw tracking error, inverse average pitch and roll, and inverse average commanded thrust over the episode. The adaptation frequency ∆T is 4 seconds. We also evaluate our framework on a physical Crazyflie with the same controller and performance measures. Details of the physical experiments are described in Section 6.2.Quadrupedal Robot with Learned Locomotion Policy: Our third simulated robotic platform is a quadrupedal robot equipped with a static pretrained locomotion policy π trained using model-free RL [26] . π outputs joint angles at 50Hz such that the torso of the robot follows a velocity twist command τ k = ( ẋdes , ẏdes , ωdes ). We treat π as our controller C for this system. Although π is parameterized by a deep neural network, it is also conditioned on an additional, eight-dimensional command g k that allows the user to specify high-level desired behaviors that the policy should follow: stepping frequency, body height, footswing height, stance width, and three discrete variables which jointly specify the quadrupedal gait. We treat g k as the controller parameters to be tuned automatically based on the quadruped's randomized parameters and the twist command. The unknown system parameters θ k for the quadruped are added mass payloads to the robot base, motor strengths, and the friction and restitution coefficients of the terrain. The adaptation frequency ∆T is 3 seconds.To generate training data from each simulated platform, we randomly sample N parameter vectors θ 0:N . For each θ i , we sample N B random samples of g and τ , drive the system to a random initial dynamical state to collect history z, u, and finally roll-out the system with controller g to gather performance metrics y. We train a separate basis-function network for each system.We test each method on robots with system parameters sampled from outside of the range of the training dataset, testing the ability of each method to extrapolate outside of the training data. We evaluate methods on 15 out-of-distribution system parameters on multiple different tasks (racetracks for the racecar, ellipsoidal trajectories for the quadrotor, and twist commands for the quadruped) and 8 random seeds. For each platform, we fix the reward functions J and uncertainty penalty β. Each test consists of multiple iterations of the "Optimize, Evaluate, Adapt" procedure shown in Figure 2 . Because we run each test in an online setting with no resets between timesteps, if a method selects dangerous gains that cause a crash, the reward for every subsequent timestep of that evaluation is set to zero. More details on dataset sizes, model architecture, and training and testing parameter ranges are provided in the supplementary material.We compare our method on each test system against three baselines. The first baseline is Reptile [50] , a first-order version of the common gradient-based meta-learning method MAML [31] . Reptile uses minibatched gradient descent as the base learner and it only outputs a single point prediction of y. For our second baseline, we compare against a GP model whose kernel function is the composition of our network trained without meta-learning with an RBF kernel, mirroring [17, 29] . We also compare against control parameters that were hand-tuned by experts for robust and conservative performance across a variety of commands and system parameters. In our quadrotor example, we additionally consider an L1-Adaptive control [36] baseline adapted from [37] , which augments the nominal control with an additional term based on an estimated disturbance vector.To evaluate the usefulness of adapting the prediction model, we perform an ablation where we do not adapt w from trial to trial and only use w 0 and Σ 0 (context-only). This ablation mimics methods that directly estimate latent system parameters from state and action histories, such as Kumar et al. [5] , since the prediction network only uses the history context encoder and does not utilize our lastlayer weight adaptation. To evaluate the usefulness of meta-training, we also run an ablation where last-layer weight adaptation occurs, but the network is trained without our meta-learning procedure (no-meta). 
results: 
 In Table 1 we report the crash rate and final average reward achieved by each method. To show the advantage of each method over simply using the expert-tuned nominal gains, in Figure 3 we plot the difference between each method's reward and the nominal gain's reward over time in each test. We report the raw performance metrics achieved by each method in the supplementary material.OCCAM outperforms both ablations and all baselines on all benchmarks, demonstrating the utility of both meta-learning basis functions and weight adaptation. Adaptation with our method also occurs within a few timesteps, which corresponds to 10-20 seconds of data on each system. The quadruped environment is challenging in particular, as OCCAM must predict the performance of an expressive RL policy; nonetheless, it finds behavior parameters that improve the policy's performance. The quadrotor also presents unique challenges, as the parameters of the quadrotors in the test (a) Plots of performance reward value vs time on robotic systems (Higher is Better)Figure 3 : Time vs. Reward curves on all test systems. Our method shows robust prior performance and adaptation across all tests. For the robotic systems, normalized reward is computed by standardizing rewards according to the training dataset statistics for that system and subsequently subtracting the nominal controller's reward.set can vary from 1 3 × to 3× the nominal parameters. In our problem setup, this must be compensated only by tuning PD feedback gains (the controller does not have an integral term). These extreme parametric errors also cause instability and poor tracking in the L1-adaptive controller. Meanwhile, our method finds gains that result in the lowest crash rate and nearly a 50% reduction in tracking error compared to the nominal gains and L1-Adaptive controller. Both Reptile and LK-GP saturate early in the evaluations. Reptile tends to overfit to the training set and online training data [34] and gets stuck in a loop of selecting the same gains and repeatedly fitting to them.In the supplementary material, we additionally show that OCCAM also performs well on common global function optimization benchmarks, makes interpretable optimizations to the gains as system parameters change, and learns a structured latent weight space.To further demonstrate the necessity of model adaptation for controller tuning in out-of-distribution scenarios, we also evaluate each baseline on test scenarios that lie within the training distribution, and present a table of these results in the supplement. In these test cases, the context-only baseline performs comparably to our full method. Therefore, its performance drop in the out-of-distribution scenarios suggests that the learned context encoder alone is not sufficient to achieve generalization. Similarly, our method outperforms LK-GP because the usefulness of the latent space of the kernel network degrades as we move out of its training distribution.To emphasize OCCAM's generalizability, we also use OCCAM to tune the controller gains of a simulated Crazyflie to minimize tracking error in three different wind conditions up to 0.5 m/s, when wind was not encountered at all during training. OCCAM is able to achieve an average tracking error of under 7cm, outperforming all baselines including L1-Adaptive control (10cm), shown in the supplementary material. Meanwhile, context-only achieves the worst tracking performance (30cm). Finally, we apply OCCAM to tune the gains of a physical Crazyflie quadrotor. The controller for the physical Crazyflie runs on a Robot Operating System (ROS) node running on a laptop with no GPU and a 1.6Ghz Intel i5 processor. The ROS node receives the quadrotor's position through a Vicon motion-capture system 1 running at 100Hz and computes a desired collective thrust and quaternion attitude command for the quadrotor, which the onboard PID controller then converts into motor speeds. We train our predictive model on simulation data sampled from both the train and test parameter ranges, and add OCCAM as a lightweight layer that runs every 3 seconds on top of the ROS control node.Position tracking error results for a three-dimensional ellipsoidal reference trajectory are shown in Figure 4 . Control parameters tuned by OCCAM result in lower tracking error than the expert-tuned nominal controller and achieve comparable tracking error to the L1-Adaptive controller. We also run additional experiments where a 5 gram mass is added to the Crazyflie (see Figure 1 ), which represents a 20% increase from the its nominal mass. Our method is able to find gains that decrease the quadrotor's tracking error in the z-axis by 54% compared to the nominal controller and by 17% compared to the L1-Adaptive controller. Figures of tracking errors from these physical Crazyflie experiments are shown in the supplementary material. 
discussion and limitations: 
 Our results demonstrate a single algorithm that optimizes controllers on a diverse array of robotic platforms using simple search techniques. Importantly OCCAM obviates the need for manual gain tuning that typically requires domain expertise, and instead provides automatic controller adaptation based on task-relevant metrics. OCCAM is also compatible with any controller with tunable parameters, allowing straightforward application to controllers that are nonlinear, discontinuous, and not analytic. As shown in our Crazyflie experiment, OCCAM's models and optimization routine are lightweight enough to run in realtime without a GPU. We expect OCCAM to perform best on systems with a well-defined and relatively smooth mapping between control gains and performance; this and further assumptions are discussed in the appendix.In this work, we tested OCCAM on commonly used controllers with relatively low-dimensional gain spaces, though one might also wish to tune a higher-dimensional policy, such as a neural network. Although the policy could be conditioned on a low-dimensional behavior vector, one could also treat the weights of either the entire neural network or its last few layers [51, 52] as the controller gains and use a more powerful optimizer.Beyond this, there are a few areas in which our method could be improved. If the stochastic optimizer makes a poor initial choice and the uncertainty penalty is poorly tuned, OCCAM sometimes settles early in a suboptimal local maximum. For some systems, this incorrect choice of gains can cause crashes. Incorporating a more sophisticated optimizer and exploration strategy would be interesting future work. Lastly, in this work we do not rigorously evaluate how far the generalizability of our method applies, or changing the reward function J online. Future work includes addressing these limitations, along with demonstrating our method on broader classes of systems. In this section, we will discuss the aspects of our formulation that lead to the choice of the Kalman Filter as our online estimator. First, we choose a linear output layer of our neural network: y = Φ(g, z, u, τ )w, which has been shown to improve model generalization in meta-learning settings [1, 2] . Then, given data pairs (Φ t , y t ) received in a stream from the robot, solving for the optimal mapping s.t. Φw = y is an online linear regression problem. However, there is significant uncertainty in the regression problem arising from our lack of knowledge of θ, combined with epistemic uncertainty in the model and environmental noise. Therefore, we would like to estimate the posterior distribution of w to characterize the uncertainty in the model, especially for the purposes of downstream decision making using the network's predictions. This estimate also needs to be able to vary with time, since the environment parameters θ could slowly change as the robot moves into a new environment. Given the choice of a linear measurement model and the additional factors above, this strongly resembles the time-varying parameter estimation problem, to which a linear Kalman Filter with identity dynamics is commonly applied. Representing w as a Gaussian random variable enables analytic computation of the posterior and keeps the filtering procedure differentiable, which enables our meta-learning pipeline.Although our experiments show that OCCAM can be applied to a wide variety of systems, we do not yet have a precise theoretical characterization of OCCAM's performance. However, we do make a few implicit assumptions about the kinds of systems that OCCAM would work well on.1. For OCCAM to work well, the closed loop dynamics of the system should allow a welldefined mapping between control gains and performance. This mapping does not have to be known, can vary as the system parameters vary, and can be stochastic. However, if it is too noisy or nonsmooth, then the performance of the predictive model will degrade. For example, if the closed loop dynamics are chaotic, then the effect of control gains on system performance might be unpredictable. Although we make no assumptions about the shape of the mapping, we expect OCCAM to work best if the mapping is relatively smooth.2. We also assume that the offline dataset (generated through simulation or from previous experience) captures the important characteristics of how gains affect performance and how that mapping changes as the environment changes. Although our method can adapt to a significant domain gap (as demonstrated by our experiments), if the domain gap is too large then the basis functions and prior weights learned from offline data will not be useful and our method will take many iterations to adapt.3. The closed-loop system should be stable enough that a suboptimal set of control gains (such as those deployed in the early iterations of adaptation) will not cause an immediate crash.4. Our use of a sampling-based optimization method limits the dimensionality of gain spaces that our method can handle. However, we could use a more powerful optimizer to overcome this, and many model-based control methods of interest (PID, MPC, LQR, etc.) do have gain spaces that are low-dimensional.In this section we provide additional details about each of our simulated evaluation platforms, including two benchmark functions which are commonly used to test global functional optimization algorithms.We first validate our method on randomized variations of two common global optimization benchmark functions [3] . The first is the Branin function, which has a 2D input space and 1D output space:f (x) = a(x 2 -bx 2 1 + cx 1 -r) 2 + s(1 -t)cos(x 1) + s We treat as system parameters the six constants that parameterize the shape of the Branin function:θ = [a, b, c, s, t, r].The second is the Hartmann function, which has a 6D input space and 1D output space:f (x) = - 4 i θ i exp   - 6 j=1 A ij (x j -P ij ) 2  Where A and P are constant matrices, and we randomize over the 4-dimensional vector θ as system parameters.For these benchmark functions, there are no measured quantities z 0:k or control actions u 0:k . We consider the inputs to the benchmark functions to be the "gains" g k , and the outputs of the functions to be the performance measures y k . Therefore the data tuples for these functions consist of only the inputs x and scalar "metrics" y = f (x). For these functions, the reward function is simply set to the negative of the scalar function values: J(y) = -y. Because there is no history context, the context-only baseline in these two examples is simply our method without weight adaptation.For the benchmark functions, we use F-PACOH [4] , which is based on training neural networks with regularization to serve as mean and kernel functions in a GP. F-PACOH is ill-suited to our robotic tests due to the high dimensionality of the full input space to the networks, so we use the LK-GP baseline in our robotic experiments instead.Our first simulated robotic system is a 2-dimensional car racing around a track, modified from the OpenAI Gym "Car Racing" environment [5] . The environment models a powerful rear-wheel-drive car with sliding friction, making control nontrivial while trying to maximize speed on track. The system has three control inputs:u k = [u s , u g , u b ].The controller C of the car consists of a nonlinear proportional-plus-derivative (PD) controller that computes steering input u s to steer the car towards the centerline of the track and a simple control law that accelerates the car by force u g on straightaways up to a maximum speed, or brakes the car by force u b for corners above a certain curvature threshold. The tunable maximum speed and curvature braking thresholds induce saturation and branching behavior in the controller. The sensor measurements of this system arez k = [v, ω k , e lat ],where v is the forward velocity of the car, ω k is the angular velocity, and e lat is the lateral distance between the car and the track centerline. The controller C uses z k and an estimate for track curvature, c, derived from a vector of upcoming track waypoints τ k to compute u k as follows:u s = k ps e lat + k ds ėlat u g = k pg , if v ≤ v max 0, otherwise u b = k pb , if c ≥ c thresh 0, otherwiseThe tunable parameters of this controller areg = [k ps k ds k pg k pb v max c thresh ]The racing car environment has three unknown system parameters θ = [m, p, µ], which are respectively the mass of the car, the car's engine power, and the friction between the tires and track.For the racing car, the evaluation function computes the vector of performance metricsy k:k+∆T = 1 ∆T 1 1 + i e lati , 1 1 + w i , v iThese are respectively the inverse average lateral tracking error, inverse of total number of timesteps during which a wheel was slipping, and average velocity over a fixed evaluation horizon. We invert tracking error and wheelslip since, in general, they ought to be minimized. In this case, the evaluation horizon is not a fixed ∆T but instead is however long it takes for the car to traverse a fixed distance on track. For online testing of this system, we set the reward function to be a weighted combination of the reward terms: J(y) = 3 j=0 r j y[j]. The reasoning behind inverting certain performance measures is to focus the model onto gains which have good performance. In this racing car setting, for example, "good performance" equates to low tracking error, low wheelslip, and high speed. At test time, we will only select and deploy gains for which the network predicts good performance. Therefore, our preference is that the network should be very discriminating between two gains that have good performance, i.e. low tracking error and wheelslip. If two gains result in high tracking errors, we do not care about exactly predicting that difference, because we will not deploy those gains. However, during training time, using the Mean Squared Error Loss function has the opposite effect -the network predicting that a gain will have a tracking error of 1m vs 2m will result in a very large gradient, while the network mispredicting by a small amount will have a small effect on the gradient. Inverting the performance measures for which lower values are better has the desired effect of increasing the distance between high performing gains, while compressing the "bad" gains into a small region of the space. And because the inversion is invertible, we can recover the original, interpretable metrics if necessary.Our second simulated platform is a quadrotor MAV equipped with a geometric trajectory tracking controller defined on SE(3) [6] . This controller takes in a reference trajectory τ k defined in the quadrotor's flat output space: position (p x , p y , p z ) and yaw. The controller computes a feedforward motor speed command based on τ k using the quadrotor's nominal mass, inertial tensor, thrust and drag torque coefficients. It then uses measurements from the quadrotorz k = [p x , p y , p z , v x , v y , v z , R] k ,where R is the rotation matrix representation of attitude, to compute feedback commands to correct tracking errors. The controller is parameterized by PD gains on the 3D position and PD gains on the attitude:g k = [k x , k v , k R , k Ω ](following the convention given by [6] ). The quadrotor has five unknown system parameters which are the quadrotor's mass, principal moments of inertia, and thrust coefficient: θ = [m, I xx , I yy , I zz , k η ]. The baseline controller is only aware of the nominal parameters, which are centered around those of the Crazyflie platform [7] , and not the actual values. Thus, the feedback gains must be used to compensate for this parametric error. For more detailed information about the quadrotor's dynamics and the controller derivation, see [6] .For this system, the four performance measures y are the inverted average positional tracking error, inverted average yaw tracking error, inverted average pitch and roll, and inverted average commanded thrust over the episode. Following the racing car example, we choose the reward function to be a weighted combination of the terms of y: J(y) = 4 j=0 r j y[j]. For this system, we set ∆T = 4 seconds.For our quadrotor experiments, the commanded trajectories τ consist of 3-dimensional ellipsoidal trajectories of varying radii and frequencies. Because of the simplicity of these trajectories, we do not have to provide information about τ as input to the network for this system. We leave the incorporation of more general and complex trajectories to future work. We use RotorPy [8] and its included SE(3) controller for all quadrotor simulations. For this environment, we also evaluate our framework on a physical quadrotor with the same controller and performance measures.Our third simulated robotic platform is a quadrupedal robot equipped with a static pretrained locomotion policy π trained using model-free RL [9] . π outputs joint angles such that the torso of the robot follows a velocity twist command c k = ( ẋdes , ẏdes , ωdes ). The policy takes as highdimensional input measurements z k the joint positions and velocities q k , qk , previous joint angle commands a k-1 , commands c k , timing reference variables, and estimated base velocity and ground friction. We treat π as our controller C for this system.Although π is parameterized by a deep neural network, it is also conditioned on an additional command that allows the user to specify high-level behaviors that the policy should follow:b k = θ cmd 1 , θ cmd 2 , θ cmd 3 , f cmd , h cmd , h cmd f , s cmdThe three terms θ cmd 1 , θ cmd 2 , θ cmd 3 jointly specify the quadrupedal gait, f cmd is the commanded stepping frequency, h cmd is the commanded body height, h cmd f is the commanded footswing height, and s cmd is the commanded stance width. Thus, the policy tries to follow the velocity command c k while satisfying the behavior constraints. In the original work b k is a quantity to be selected by a human operator, while in this work we treat b k as the controller parameters to be tuned automatically based on the quadruped's randomized parameters and the task c t . For details on how the learned policy is trained, see [9] .The randomized system parameters θ k for the quadruped are added mass payloads to the robot base, motor strengths, and the friction and restitution coefficients of the terrain. Although the π contains an estimator module to regress the ground friction, it does not receive direct observations of any of these parameters.For use in our method, we input only a reduced-dimension subset of z k into our prediction model network consisting of the estimated base linear and angular velocities and joint torques applied by the motors.The four performance measures for the quadruped are the inverted average velocity errors along each axis of the command and inverted total commanded torque over the evaluation horizon. For this system, we set the evaluation horizon ∆T = 3 seconds.The reward function for the quadruped has the same form as the quadrotor system: J(y) = 4 j=0 r j y[j]. All simulations are done using code and pretrained models from [9] and the Isaac Gym simulator [10] . To mimic this approach in our setting, we divide our training process into two phases. In the first phase, the network is trained as an average model using standard stochastic gradient descent, with w pre as a fixed last layer that does not adapt to tasks. In the second phase, we initialize w 0 = w pre and we switch to meta-training with the Bayesian recursive update as the inner loop optimizer. Σ 0 , Q 0 , and R 0 are initialized before the second training phase as identity matrices. Our no-meta ablation uses a network that is trained only with this first phase and not with our meta-learning procedure.The datasets for the robotic systems each consist of N = 1500 batches of N B = 64 datapoints each. The hyperparameters of each dataset and network are provided in the supplementary material. Note that our method does not require sampling only optimal or high-performing gains to generate data -only random ones. Thus, the dataset for each system consists of N batches of datapoints: [(g, τ, z, u, y) 0:N B ] 0:N . Each of these batches is used as a "task" for a single inner loop during the meta-training process.We find that we are able to use small networks to model each system; the networks are all fullyconnected networks that consist of 3 hidden layers with fewer than 64 hidden units, outputting between 5-20 bases, indicating that many of the robotic systems that we are interested in controlling can be effectively modeled with a relatively small number of parameters. The exact network layer sizes and training hyperparameters are given in the supplementary material. All models are implemented and trained in PyTorch [11] .Architectural details and training hyperparameters for OCCAM's basis function network, Reptile, and F-PACOH are presented in Tables 2, 3 , and 4 respectively. The F-PACOH training hyperparameters were chosen in accordance with experiments conducted in the original paper.Training and testing parameter ranges for each system evaluated in this work are shown in Tables 5 We report the average final reward obtained by all methods on the Branin and Hartmann benchmarks in Table 10 , and show minima obtained by each method over time in Figure 1 . Notably, our method performs well in both settings. In the Branin setting, OCCAM learns a good initialization and finds the best final minimum. In the Hartmann setting, even though OCCAM learns a relatively poor prior, it is able to adapt and find the same final minimum as F-PACOH.Figures 2, 3 , and 4 show the raw performance metrics obtained by each method on each system in the trials in which they did not crash. We note that each method is not directly optimizing for these raw metrics, but instead a weighted combination of their normalized versions, so good or bad performance in an individual metric in these plots does not necessarily translate to high or low reward in the plots reported in the paper. For example, in the Racing Car example, our method obtains a lower average speed than many other methods; however, this makes sense as, in the scalarized objective the model was optimizing for, the speed metric was weighted much lower than the tracking error metric. Also to faithfully report the raw metrics without the crashes skewing the averages, we filter out the runs that crashed. For example, in the quadrotor example, although Reptile performs well when it selects gains that don't result in crashes, its higher crash rate brings down its overall average reward. We also run our method and each baseline on test sets randomly sampled from the training distributions for each of the robotic systems (see Tables 7, 8 , and 9). The average final obtained reward and crash rates are reported in Table 11 . The performances of each method naturally improve in this setting as the sampled system parameters lie closer to the nominal parameters, but in particular the context-only baseline, which only uses the fixed context encoder for sysid, and the LK-GP baseline both obtain amongst the highest rewards and perform similarly to OCCAM, showing, within the training distribution, these approaches perform well.Also notable in this setting is that the L1-Adaptive controller obtains higher reward than the Nominal controller, demonstrating that the adaptive control does indeed improve performance when the deviation from the nominal dynamics is smaller. However, when the parametric error grows larger in the out-of-distribution experiments in the main paper, the adaptive controller becomes unstable and reduces performance. To elucidate that our method finds semantically meaningful gains, we run an additional experiment in the racing car environment where we sweep only friction coefficients across 3 different tracks and plot the average final gains chosen by OCCAM in Figure 5 . As friction increases, OCCAM selects gains that cause the car to accelerate more aggressively and drive faster, while in the low friction regime, the gains tend towards slower driving (higher brake gain, lower speed in corners). Our method logically chooses a more aggressive driving profile as available traction increases, showing physically meaningful adaptation to changes in system parameters. We also include preliminary experiments demonstrating that the space of weights that OCCAM adapts in has meaningful structure. For each test set in the paper, we use t-SNE to project the weights computed by OCCAM's regression procedure into two dimensions and plot the projected weights in Figures 6a, 6c , and 6b. Note that like the weight adaptation procedure, the t-SNE embedding procedure has no knowledge of the underlying system parameters. For each system, the values of the weights distinctly cluster according to the underlying system parameters.We also run an experiment to assess the ability of our method to tune controllers while novel environmental parameters are varied. We use OCCAM to tune the controller gains of a simulated Crazyflie in three different wind conditions up to 0.5 m/s to minimize the tracking error. Despite the fact that wind was not modeled at all during training, OCCAM is able to achieve an average tracking error of under 7cm, outperforming all baselines, shown in Figure 7 . Meanwhile, as expected from the previous section, no-adapt fails to tune the controller in this unseen setting. We ran additional experiments on the physical Crazyflie platform in which we added a 5-gram mass from the beginning of the experiment and in the middle of the experiment. Plots of the tracking error obtained by the controller with OCCAM's optimized gains, the nominal gains, and with the L1-Adaptive control augmentation are shown in Figures 8a and 8b . In both cases, OCCAM finds gains that result in more robust tracking in the Zaxis. We hypothesize that because our predictive model is trained on data gathered from many quadrotors with varied masses, it learns to select gains that better compensate for these variations.An interesting result are the minor, high frequency oscillations observed in the Z-axis in Figure 8a and in the X-and Y-axes in Figure 8b towards the end of the experiment. These are most likely the result of marginally stable closed-loop attitude dynamics. One possible solution to this is augmenting the performance measures y and measurement vector z with pitch and roll angular velocities, which might encourage the predictive model and optimizer to select gains that do not result in oscillations. Another solution is to add small random force perturbations to the training simulations so that marginally stable controllers achieve worse performance metrics. We leave exploring these additions to future work. 8th Conference on Robot Learning (CoRL 2024), Munich, Germany.https://www.vicon.com/ 
 
 Appendix:  


Decision: **Post-rebuttal metareview**:

This paper presents a novel approach to online adapt controllers' parameters. In particular, this paper uses meta-learning to train a predictor that predicts the closed-loop controller performance as a function of controller parameters, online fine-tunes such a predictor, and uses this predictor as a surrogate goal to optimize controller parameters. The idea is novel, and experiments are diverse. Most of the reviewers' concerns have been addressed in the rebuttal phase.

--------------------------------
**Pre-rebuttal metareview**:

Strengths:
1. The idea of using meta-learning to train a predictor to predict policy performance as a function of policy parameters is interesting and somewhat novel.
2. Comprehensive literature review. 
3. Diverse experiments.

Weakness: 
1. Need more analysis about the applicability of the proposed method: in what systems can we accurately predict the policy’s performance?
2. The choice and design of Kalman filter needs more justification.
3. Other issues mentioned by reviewers such as comparing with L1 adaptive control, adaptation timescale, notations, writing, etc.

Reviews: review: title: title: Interesting ideas related to meta-training/adapting predictive models to directly predict system performance given a set of control parameters, validated by extensive experiments
paper_summary: paper_summary: This paper addresses the challenge of control tuning and adaptation in robots operating across diverse environments. In particular, the authors propose a framework for online controller adaptation that integrates meta-learning with Bayesian recursive estimation. This core idea is to meta-train a predictive models to directly predict system performance given a set of control parameters and online adapt this model using collected data during system operation. The predictive models serve as cost functions in sampling-based optimization routines, enabling the online tuning of control parameters that maximize system performance. The framework’s efficacy is demonstrated on a variety of robotic systems including a real quadrotor.
main_review: main_review: Combining meta-learning and Bayesian regression (Kalman filter) to update a predictive model to directly predict system performance given a set of control parameters is an interesting idea. Extensive experiments demonstrate the efficacy of the proposed approach.  However,   the rationale of the proposed approach is not sufficiently explained. Additionally, some clarifications are needed. 
Strengths:
- Using a predictive model to predict performance dependence on control parameters is an interesting idea. 
- Literature review is comprehensive
- Experiments are extensive and involve hardware implementation. 

Major Weaknesses:
-  Despite extensive empirical results, it is unclear when the proposed approach can be applied. In other words,  under what conditions (e.g., in terms of dynamics, tasks, performance criteria, etc.) can an good predictive model be learned to directly predict performance dependence on control parameters without identifying the dynamics model? 
- The authors propose to use Kalman filter with identity dynamics to estimate the weights $w$, which is “treated as a normally distributed random variable”. It is unclear why this can be done. Also, the Kalman filter is for stochastic dynamics, while the dynamics change from one environment to another is not due to stochasticity.


Aug 16: I appreciate the authors' efforts in addressing my concerns and improving the paper. Most of my concerns have been clarified.  My only remaining concern is that, given a system, it seems difficult to know beforehand whether the method will work well before testing it on that system. I hope this limitation can be addressed in the authors' future work. I updated my rating.
questions: questions: 1.	Is it possible to explain under what conditions the proposed approach (which relies on a model to directly predict control performance given control parameters) can be applied?
2.	Please justify the use of Kalman filter (see my corresponding comment above).  In addition to explanation, is it possible to do some simulations to show that the proposed approach can almost recover the true weights $w$ (and the basis functions) online under some scenarios (e.g., a scenario that is included in the meta-training stage)
3.	How can the variance $\sigma$ in Section 4.3 be calculated? Also, instead of using “evolutionary modification to random-search”, can the authors use more precise language?
4.	Is the PD controller for the race car a “nonlinear” controller?
5.	In Section 5.3, what is the meaning of “phase-1”?
6.	For the racing car example, the performance improvement with increasing laps seems very little for all methods. What could be the reasons?
7.	In Figure 4, according to the legend, nominal controller gives the best tracking performance. Is this expected?
8.	How the adaptation to different tasks is demonstrated is not straightforward to me. Can the authors elaborate a little bit?
limitations: limitations: 3
review_summary: review_summary: The manuscript contains interesting ideas, which are validated by extensive simulations and hardware experiments.
title: title: Review
paper_summary: paper_summary: Paper introduces an interesting and effective way of adapting the controller parameters online.
main_review: main_review: In my opinion the paper presents an original and important work, and is written clearly.

Strengths:
- interesting approach that combines meta-learning of the basis functions and online estimation of the parameters, that are used to predict the performance measures estimate
- experiments are convincing and done carefully with appropriate ablation studies
- the method is presented quite clearly

Weaknesses:
- contributions of this paper are not expressed clearly, which makes the evaluation of the originality a bit harder
- the timescales of the adaptation seem a bit arbitrary and are relatively low
- minor presentation and notation issues


Issues and comments (not in any particular order, numbers used only for easier reference):

1. It would be nice to have the contributions listed to better understand which parts of the paper are already present in the literature and which ideas are new in the field. For example the idea of using the performance measures estimates to tune the controller is already present for example in the paper "Tuning of extended state observer with neural network-based control performance assessment" which was not referenced.

2. If I understand correctly the \Phi is defined by a neural network, and may change rapidly. How the proposed Kalman filtering scheme for w works in the presence of a rapid change of \Phi, did you observed such behavior?

3. "It is also unclear how a reference model that is always feasible for the actual system to track might be synthesized" This sentence is hard to understand.

4. In algorithm 1:
- "Sample random-size subset D train of size n" random-size or of size n?
- What is G, Z, U and Y?
- Are the datapoints in D_trian correlated (form trajectories) or are sampled at random, and you perform just one step updates?
- How importnat is the optimization of w_0? Is w_0 meant to be common for all datapoints?

5. Why this kind of control parameters choice optimization procedure was used? What is the advantage of the "evolutionary modification to random-search" over different existing optimization techniques?

6. How the \Beta was chosen? How important is the choice of \Beta?

7. In all formulas that optimize for g, g is not present in the objective.

8. The adaptation frequency seems very low, like 1/3 of a lap, 4s and 3s.
Have you tested these methods in the faster changing conditions? What happens in between the adaptations? Is the Kalman filter still running and the estimates are just taken out every \Delta T? What is the frequency of the filtering?

9. The controller used for the race car seems quite unusual. Why this kind of controller was used? 

10. " Adaptation with our method also occurs within a few timesteps, which corresponds to 10-20 seconds of data on each system" How this refers to the adaptation frequency \Delta T?

11. Fig 4. why all methods are not starting on the trajectory?
Why the tracking is so bad at the very beginning for nominal and L1?

12. What are the performance measures? What is the impact of the particular choice of performance measures on the resultant average final reward?

13. How well are the performance measures estimated?

14. Why in Fig. 7b (Appendix) there is no comparison with L1?

15. Why do you use [36] in the comparison table and not [37]? 

Minor issues:
- Abbreviation "C" for the controller seems to me a bit redundant and not needed.
- "subset Θ" subset of what?
questions: questions: I would say that authors should work on fixing the presentation and notation issues, try to answer the questions from the review and introduce a more explicit statement of contributions and explain the adaptation timing better.
limitations: limitations: 3
review_summary: review_summary: In my opinion paper advances an important field of the online controller adaptaiton and shows convincing results.
title: title: An interesting work that applies meta-learning to controller parameter adaptation. Decision updated on Aug. 15.
paper_summary: paper_summary: This paper proposed an online controller adaptation method to automate the parameter adaptation subject to dynamically changing environments and tasks. The contributions include (1) a prediction model that maps control parameters to their performance in a closed-loop system; (2) the usage of meta-learning to generalize the learned model to different system parameters; and (3) an parameter optimization approach that selects the optimal parameter for deployment. The approach has been validated thoroughly in simulation and experiments over different robotic systems and controllers.
main_review: main_review: I think this paper proposed a novel design for controller parameter adaptation for potentially unseen environments or tasks. The presentation was clear, and the results were well presented. However, I have some concerns about the technical details of this paper, which are summarized below.

Major comments/questions

1.	The authors mistakenly categorized L1 adaptive control as direct adaptive control. In fact, it is one type of indirect adaptive control. In the direct architecture, the controller parameters are directly updated, while in the indirect architecture, the plant parameters are estimated and used in the feedback law.  See the references [1,2].

2.	The usage of the Kalman filter (KF) with an encoder output seems to be a novel design. In my understanding, the KF is the key mechanism that introduces “feedback” to the adaptation, where the innovation step in the KF uses the error between the predicted performance measures and actual performance measures, i.e., $y_{t+1} - \Phi w_t$, to update the weights $w_{t+1}$. There are two questions here: 

(a) The justification for using KF here is missing. Basically, in classical examples of KF, there’s a dynamic process for which the state can not be measured directly but is observable from noisy sensor measurements. The authors stated that the state in KF, $w_t$ is treated as a normally distributed random variable. But what’s the rationale behind this argument? Some insights will be helpful here to help the readers understand the necessity of KF in the proposed architecture.  

(b) Why is $\Phi$ time-invariant (as interpreted by its free of subscript $k$)? The input to the predictive model, including control gains $g_k$, task $\tau_k$, and history control $u_{k-H:k}$ and measurements $z_{k-H:k}$ are all explicitly dependent on time $k$, it is strange that the output of the predictive model $\Phi$ is independent to $k$. 

3.	Sections 4.1-4.3 introduce the components of the proposed OCCAM framework. It’d be helpful to state whether they are applied online for adaptation or offline for training, or both. For my current understanding, the control performance modelling (4.1) is applied both in offline training and online adaptation. Meta-training (4.2) is only used in offline training. Controller optimization (4.3) is only used in online adaptation. Is this correct?

4.	In Section 4.2, the authors mentioned that they use backpropagation (BP) to update the network parameters as well as the KF parameters. I understand that BP is the standard tool for updating neural network parameters. However, the KF parameters are usually determined based on the statistics of the sensor’s noise levels but were updated with BP instead. What is the rationale for using BP to update KF parameters here? Is there any physical interpretation?

5.	In Section 4.3, the authors used a modified random-search algorithm to enumerate candidate control parameters and then evaluate them through the performance prediction model to pick the optimal candidate. Maybe better approaches can be taken here to obtain the optimal candidate more efficiently, such as Metropolis-Hastings sampling [3] or policy-search [4,5].

6.	The ablation study clearly shows the contribution of individual components (meta-learning and context encoder) to the generalization of the proposed framework.

7.	Line 250, what do “test parameters” refer to? It’s not clear from the context. Are they the initial parameters used at time = 0 in Figure 3a? Also, what does it mean by “300% parametric error from the nominal model”?

8.	The reported performance on L1 adaptive control seems to contradict the results of the simulation and experiments. In the reported simulation results, the L1AC’s reward and crash rate are close to those of the nominal controller. However, in the experiments, shown in Figure 4, L1AC has a clear advantage over the nominal controller’s tracking performance, which is qualitatively distinctive to what’s been shown in Table 1. The authors may want to double-check the simulation setting to confirm the correctness of the results.

9.	I think the authors should have one paragraph explaining how the "adaptation" is made possible in the current work. My guess for this question is that it is due to the innovation step of the KF. The authors should make it clear because otherwise, this framework simply "guesses" suitable parameters without incorporating feedback from the system regarding the actual performance, which will greatly compromise the quality of the proposed approach.

Minor comments:

1.	The variable $w$ seems typed in two fonts. Please correct it if they indicated the same term.

2.	On top of page 4, the authors stated that the index $t$ is different than $k$, where $t$ is used to denote the time instances for performance measurement and adaptation. If I understand it correctly, the span between $t$ and $t+1$ is longer than that between $k$ and $k+1$. If this is the case, then the authors should make it explicitly clear.

3.	In Section 5.1, for the quadrotor simulation, why is the reward function set to be the linear combination of the inverse of various interpretable metrics like the position tracking error and yaw tracking error? Reciprocals of these interpretable metrics lose the physical meaning carried by these metrics, let alone the linear combination. The authors need to provide justification for this setting.

4.	For reference [13] in the paper, consider citing the more comprehensive work [6] listed below.

References

[1] Hovakimyan, Naira, and Chengyu Cao. ℒ1 adaptive control theory: Guaranteed robustness with fast adaptation. Society for Industrial and Applied Mathematics, 2010.

[2] Hovakimyan, Naira, Chengyu Cao, Evgeny Kharisov, Enric Xargay, and Irene M. Gregory. "L 1 adaptive control for safety-critical systems." IEEE Control Systems Magazine 31, no. 5 (2011): 54-104.

[3] Loquercio, Antonio, Alessandro Saviolo, and Davide Scaramuzza. "Autotune: Controller tuning for high-speed flight." IEEE Robotics and Automation Letters 7, no. 2 (2022): 4432-4439.

[4] Y. Song and D. Scaramuzza, "Policy Search for Model Predictive Control with Application to Agile Drone Flight," IEEE Transaction on Robotics (T-RO), 2021.

[5] Y. Song and D. Scaramuzza, "Learning High-Level Policies for Model Predictive Control," IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), Las Vegas, 2020.

[6] Cheng, Sheng, Minkyung Kim, Lin Song, Chengyu Yang, Yiquan Jin, Shenlong Wang, and Naira Hovakimyan. "Difftune: Auto-tuning through auto-differentiation." arXiv preprint arXiv:2209.10021 (2024).

Aug. 15: I appreciate the clarification and improvement the authors have made. I have changed my recommendation accordingly.
questions: questions: see my numbered questions above.
limitations: limitations: 3
review_summary: review_summary: The proposed method is novel and the results are relatively comprehensive. The authors should address the concerns listed above. Based on its current state, I recommend this paper to be weakly rejected.