Learning Human Postural Control with Hierarchical Acquisition Functions
Authors: Nils Rottmann
Tjasa Kunavar
Jan Babic
Jan Peters
Elmar Rueckert
Venue:  (2020)
Keywords: Human Postural Control Model
Hierarchical Bayesian Optimization
Acquisition Function

Abstract: Learning control policies in robotic tasks requires a large number of interactions due to small learning rates, bounds on the updates or unknown constraints. In contrast humans can infer protective and safe solutions after a single failure or unexpected observation. 
In order to reach similar performance, we developed a hierarchical Bayesian optimization algorithm that replicates the cognitive inference and memorization process for avoiding failures in motor control tasks. A Gaussian Process implements the modeling and the sampling of the acquisition function. This enables rapid learning with large learning rates while a mental replay phase ensures that policy regions that led to failures are inhibited during the sampling process.    
The features of the hierarchical Bayesian optimization method are evaluated in a simulated and physiological humanoid postural balancing task. We quantitatively compare the human learning performance to our learning approach by evaluating the deviations of the center of mass during training. Our results show that we can reproduce the efficient learning of human subjects in postural control tasks which provides a testable model for future physiological motor control tasks. In these postural control tasks, our method outperforms standard Bayesian Optimization in the number of interactions to solve the task, in the computational demands and in the frequency of observed failures. 

Under review as a conference paper at ICLR 2020
LEARNING HUMAN POSTURAL CONTROL WITH HIER- ARCHICAL ACQUISITION FUNCTIONS
Anonymous authors Paper under double-blind review
ABSTRACT
Learning control policies in robotic tasks requires large number of interactions due to small learning rates, bounds on the updates or unknown constraints. contrast humans infer protective and safe solutions after single failure unexpected observation. order to reach similar performance, we developed a hierarchical Bayesian optimization algorithm that replicates the cognitive infer- ence and memorization process for avoiding failures in motor control tasks. A Gaussian Process implements the modeling and the sampling of the acquisition function. This enables rapid learning with large learning rates while a mental re- play phase ensures that policy regions that led to failures are inhibited during the sampling process. The features of the hierarchical Bayesian optimization method are evaluated in a simulated and physiological humanoid postural balancing task. We quantitatively compare the human learning performance to our learning ap- proach by evaluating the deviations the center mass during training. Our results show that can reproduce the efficient learning of human subjects in postural control tasks which provides testable model for future physiological motor control tasks. In these postural control tasks, method outperforms stan- dard Bayesian Optimization in the number of interactions to solve the task, in the computational demands and in the frequency of observed failures.
INTRODUCTION
Autonomous systems such as anthropomorphic robots or self-driving cars must not harm cooperat- ing humans in co-worker scenarios, pedestrians on the road or them selves. To ensure safe interac- tions with the environment state-of-the-are robot learning approaches and afterwards an expert selects final candidate policies to be run on the real system. However, for most autonomous systems a fine-tuning phase the real system is unavoidable compensate for unmodelled dynamics, motor noise or uncertainties in the hardware fabrication.
first applied to simulations
Several strategies were proposed ensure safe policy exploration. In special tasks like in robot arm manipulation the operational space can be constrained, for example, in classical null-space control approaches Baerlocher & Boulic (1998); Slotine (1991): Choi & Kim (2000); Gienger et (2005):
Saab et al. 2013); Modugno et al. (2016) or constraint black-box optimizer Hansen et al. 2003
Wierstra al. (2008); Kramer al. (2009); Sehnke al. (2010); Arnold & Hansen] (2012)
While
this null-space strategy works in controlled environments like research labs where the environmental
conditions do change, it fails in everyday life tasks in humanoid balancing where the priorities or constraints that lead to hardware damages when falling are unknown.
Alternatively, limiting the policy updates by applying probabilistic bounds in the robot configuration or motor command space Bagnell & Schneider (2003): (Pters et al. (2010)): Rueckert et al. (2014); Abdolmaleki et al. 20155) Rueckert et al. (0013) were proposed. These techniques do not assume knowledge about constraints. Closely related are also Bayesian optimization techniques with mod- ulated acquisition functions Gramacy Lee 2010); Berkenkamp et al. (2016); Englert & Toussaint 2016);; Shahriari et al. (2066) to avoid exploring policies that might lead to failures. However, all these approaches do not avoid failures but rather an expert interrupts the learning process when anticipates potential dangerous situation.

---

Under review as a conference paper at ICLR 2020
rewards
p(r(k)
p(1(0)
400
400
"2
features
policies
p(r|
Figure 1: Illustration of the hierarchical BO algorithm. In standard BO rclock-wist arrow), a map- ping from policy parameters to rewards is learned, i.e., Ø + E IR We propose hierarchical process, where first features are sampled and later used to predict the potential of policies condi- tioned on these features, Φ/ The red dots show the first five successive roll-outs in the feature and the policy space of humanoid postural control task.
All the aforementioned strategies cannot avoid harming the system itself the environment with- out thorough experts knowledge, controlled environmental conditions or human interventions. As humans require just few trials to perform reasonably well, it is desired to enable robots to reach sim- ilar performance even for high-dimensional problems. Thereby, most approaches are based on the assumption of "low effective dimensionality", thus most dimensions of high-dimensional prob- lem do not change the objective function significantly. In [Chen et al. (2012) method for relevant variable selection based on Hierarchical Diagonal Sampling for both, variable selection and function optimization, has been proposed. Randomization combined with Bayesian Optimization is proposed in Wang et (2013) exploit effectively the aforementioned "low effective dimensionality"" [i et al. (2018) dropout algorithm has been introduced to overcome the high-ding gigh-dimensionaltty problem by only train onto a subset of variables in each iteration, evaluating "regret gap" and providing strategies reduce this gap efficiently. In Rana et al. (017) an algorithm has been proposed which optimizes an acquisition function by building new Gaussian Processes with sufficiently large kernel- lengths scales. This ensures significant gradient updates in the acquisition function to be able to use gradient-dependent methods for optimization.
The contribution of this paper is computational model for psychological motor control experiments based on hierarchical acquisition functions in Bayesian Optimization (HiBO). Our motor skill learn- ing method uses features for optimization to significantly reduce the number of required roll-outs. In the feature space, we search for the optimum of the acquisition function by sampling and later use the best feature configuration optimize the policy parameters which conditioned the given features, see also Figure In postural control experiments, we show that our approach reduces the number of required roll-outs significantly compared to standard Bayesian Optimization. The focus of this study is to develop a testable model for psychological motor control experiments where well known postural control features could be used. These features are listed in Table 3 In future work we will extend our model to autonomous feature learning and will validate the approach in more challenging robotic tasks where 'good' features are hard to hand-craft.
METHODS
In this section we introduce the methodology of our hicrarchical BO approach. We start with the general problem statement and afterwards briefly summarize the concept of BO which we here
2

---

Under review as a conference paper at ICLR 2020
baseline. We then go through the basic principles required for our algorithm and finally explain mental replay.
2.1 PROBLEM STATEMENT
The goal in contextual policy search is find the best policy x (0lc) that maximizes the return
T
((0) E
Σ (rlaa,, uz) ( x(0c))]
(1)
with reward r2 uz) at time step for executing the motor commands u, in state Zt.
For learning the policy vector and the context, we collect samples of the return (( E R², the evaluated policy parameter vector @lki Rm and the observed contextual features modeled by
c(k) R" All variables used are summarized Table
[ The optimization problem defined as
(0 e²)
argmax E [(()) (0)c))
0,c
The optimal parameter vector and the corresponding context vector can be found in an hierarchical optimization process which is discussed in Section 2.3
2.2 BAYESIAN OPTIMIZATION (BASELINE)
Bayesian Optimization (BO) has emerged as a powerful tool to solve various global optimization problems where roll-outs expensive and sufficient accurate solution has to be found with only few evaluations, e.g. Lizotte et al. (2007): Martinez-Cand Canttn et al. (2007); Calandra et al. (2016). In the standard BO as a benchmark for our proposed hierarchical process. Therefore, we now briefly summarize the main concept. For further details refer to Shahriari et al. (2016).
this paper
The main concept of Bayesian Optimization is to build model for given system based on the so far observed data {X, y) The model describes a transformation from given data point E X to scalar value y E y. e.g. from the parameter vector & to the return J(O). Such model can either be parametrized or non-parametrized and is used for choosing the next query point by evaluating acquisition function ((D). Here, we use the non-parametric Gaussian Processes unknown system which are state-of-the-art model learning or regression
(GPs) for modeling approaches Williams & Rasmussen (1996; 2006) that were successfully used for learning inverse dynamics models in robotic applications Nguyen-Tiee Tuong et al. (2009); Calandra et al. (2015). For comprehensive discussions refer to Rasmussen (2003): Nguyen-Tuong & Peters (011).
GPs represent distribution over partial observed system in the form of
y y.
-~( (m(())
N
((,,,)) K(X..X.))
K(X,X)
x(X,X21)
(3)
where D = (X, y the so far observed data points and D, (X..y.) the query points. This representation fully defined by the mean m and the covariance K. We chose m(x) = 0 as mean
Table Variable definitions used in this paper.
((0)
R
return of rollout
(m), m.)
R1
intermediate reward give at time
Rd
state of the system
ul
R'
motor commands of the system unknown control policy
T (0
f
R.
policy vector
R
context vector
(0, 1)
flag indicating the success of rollout k:
3

---

Under review as a conference paper at ICLR 2020
function and covariance function Matérn kernel Matém gquared-exponentie kernel that has an additional parameter v which controls the smoothness of the resulting function. The smoothing parameter can be beneficial for learning local models. We used Matérn kernels
(960). is generalization of the
(Z 0² 2w-1((v) A" H A + By 6pg
with v/ 5/2 and the gamma function r, A (2VVil Fyll) /t and modified Bessel function H. Abramowitz & Stegun [965). The length-scale parameter of the kernel is denoted by a. the variance of the latent function is denoted by dy and bag is the Kronecker delta function (which is one if snnareddexponentic kernel. our experiments, we optimized the hyperparameters by maximizing the marginal likelihood Williams & Rasmussen 2006
and zero otherwise). Note that for v 1/2 the Matérn kernel implements the (g,1,y)
Predictions for a query points D. {ar, 3- can then be determined as E(v. X, x.) - u(x.)
+ K, K 1o-K--lxx (y m))
(4)
var(y- y,, x:) - (xx.) K.. K, K-
The predictions are then used for choosing the next model evaluation point 2n based on the acqui-
sition function ax;; D). We use expected improvement (EI) Mockus et al. (1978) which considers the amount of improvement
alx;; D) (u(x²) - -) [1(x)) T T+
o (
(S)
+ [1(x) -T
((x)
where is the far best measured value max(y), T the standard normal cumulative distribution function, & the standard normal probability density function and { ~ aęU (-0.5, 0.5) random value ensure more robust exploration. Samples, distributed over the area of interest, are evalu- ated and the best point is chosen for evaluation based on the acquisition function values.
2.3 HIERARCHICAL SAMPLING FROM ACQUISITION FUNCTIONS IN BAYESTAN OPTIMIZATION
We learn joint distribution ((())),,, 1 (+l) over k = 1,.., K roll-outs of observed triples. This distribution is used for as a prior of the acquisition function in Bayesian optimization. However, instead of directly conditioning the most promising policy vectors using aso a0;; D), propose an iterative conditioning scheme. Therefore, the two acquisition functions
ac a(c; D),
(6)
0g = (0; c, D),
(7)
are employed, where for Equation 7 the evaluated mean p(0; c) and variance a0; e) for the parameter 0 are conditioned on the features c. The hierarchical optimization process works then as follows:
In the first step we estimate the best feature values based on GP model using the acquisition function from Equation (6)
c [k+1]] max (<(;; D(I:k)).
(8)
These feature values are then used to condition the search for the best new parameter (l++1] using Equation
g[++1] max max cl++111, D(
We subsequently continue evaluating the policy vector [1++1] using the reward function presented
in Equation (I). Finally, the new data point ((+++1))) +++ cl++l]y can be added to the set of data points 12.
4

---

Under review as a conference paper at ICLR 2020
Algorithm Hierarchical Acquisition Function Sampling for Bayesian Optimization (HiBO)
1: Initialize the dataset D 1-k] = (J(eli)), 0ih), cl with K rollouts sampled policies O.
2:
for k K. K+1... do
3: clk+1] = argmax, a ((::k)) D + R.' using Eq.
4: g|k+1]] = argmaX, a aD (D(ls)) c(+++ ()) using Eq.
5: Evaluate the policy vector @ using Eq.
6: Augment D [Dli* (J (J(l+t))) l clttyly].
7:
end for
2.4 MENTAL REPLAY
To ensure robustness for Bayesian Optimization, mental replays can be generated. Therefore, the new training data set (J())++))) @[++1|, cll++I), generated by the policy parameter gl++1), will be enlarged by augmenting perturbed copies of the policy parameter [[++1), These copies are then used for generating the augmented training data sets
D D = (J(Oltt)) 0'++1| 0 c
(10)
Here, the transcript (-) denotes | perturbed copies of the given set. Hence, perturbed copies of the parameters 0lk+1] and features clk+1] are generated keeping the objective J(Ol++1)) constant. In Algorithm T the complete method is summarized. We evaluate different replay strategies in the result Section in 3.3
RESULTS
In this section we first present observations on human learning during perturbed squat-to-stan staan movements. We compare the learning results of a simulated humanoid to the learning rates achieved by the human participants. Second, we evaluate our hierarchical BO approach in comparison to our baseline, the standard BO. Third we evaluate the impact of mental replays on the performance of our algorithm.
3.1 HUMAN POSTURAL BALANCING
To observe human learning, designed experiment where 20 male participants sub- jected to waist pull perturbation during squat-to-stand movements, see Figure 2(a). Participants had stand up from a squat position without making any compensatory steps (if they made step, such trial was considered a fail). Backward perturbation to the centre of mass (CoM) was applied by pulling mechanism and was dependent on participants' mass and vertical CoM velocity.
On average, participants required trials (Ohuman 3.1) successfully complete the motion. On the left panel of Figure histogram of the required trials before the first success is shown. On the right panel, the evaluation results for the simulated humanoid are presented (details on the implementation are discussed in the subsequent Subsection 3.2). The human learning behavior is faster and more reliable than the learning behavior of the humanoid. However, humans can exploit fundamental knowledge about whole body balancing whereas our humanoid has to learn everything from scratch. Only the gravity constant was set to zero in our simulation, as we are only interested in the motor adaptation and not in gravity compensation strategies.
Adaptation evaluated using measure based on the trajectory area (TA) every episode as defined in ?. The Trajectory area represents the total deviation of the CoM trajectory with respect to straight line. The trajectory area of given perturbed trajectory defined as the time integral of the distance of the trajectory points to the straight line in the sagittal plane:
Mend
TA(ez) =
(tt)l
(1I)
5

---

Under review as a conference paper at ICLR 2020
sec
0.12 sec
0.28 sec
0.40
1.5
(a)
(b)
Figure 2: (a) Psychological postural control setup for the squat-to-stand movements. (b) Illustration of the simulated postural control task. An external perturbation is applied during the standing up motion and the robot has to learn to counter balance. The perturbation proportional to the CoM velocity in the superior direction in the sagittal plane.
Samples
10
15
20
30
Humanoid:
Figure 3: Histogram showing the number of required trials until the first successful episode for both, the human experiments and the simulated humanoid, with Phuman 6.15, Thuman 8.3 and humanood 6.38.
= 3.1, Phumanoid
A positive sign represents the anterior direction while negative sign represents the posterior direc- tion. The mean and standard deviation for the trajectory area over the number of training episodes for all participants are depicted in Figure 4 Comparing these results with the simulation results of our humanoid shows that the learning rate using our approach is similar to the learning rate of real humans.
0.02
TA Inm?
-0.02
Humanoid
-0.44
episodes
Figure 4: Mean and standard deviaton of the trajectory area (T with regard to the number of episodes for both, the human experiments and the simulated humanoid. For the humanoid the r- coordinates have been shifted about -0.5 account for the stretched arms. In addition, the trajec- tory area of the humanoid has been scaled with the factor 0.1 and shifted about -0.2 to allow casier comparison.

---

Under review as a conference paper at ICLR 2020
200
roward
-400
BO
HIBO
-1000
5
10
15
20
25
30
35
40
45
50
episodes
Figure 5: Comparison of the rewards the proposed HIBO algorithm and the state-of-the-art ap- proach Bayesian Optimization. Shown are average statistics (mean and standard deviation) over 20 runs.
3.2 HUMANOID POSTURAL BALANCING
To test the proposed algorithm we simulated a humanoid postural control task as shown Fig- ure bb). The simulated humanoid has to stand up and is thereby exposed to external pertubation proportional to the velocity of the CoM in the superior direction the sagittal plane. The pertuba- tion applied during standing up motion such that the robot has learn counter balance. The simulated humanoid consist of four joints, connected by rigid links, where the position of the first joint is fixed onto the ground. A PD-controller is used with KP and Kn,; for 1, 2, 3, 4 being the proportional and derivative gains. In our simulations the gains are set to KP., = 400 and KD.; = 20 and additive control noise e M0, 1) has been inserted such that the control input for a certain joint becomes
KP. CP KD.s ED,i to.it6
(12)
where C,,,, CD,, the joint errors regarding the target position and velocity. The control gains also be learned. The goal positions and velocities for the joints are given. As parametrized policy, we use a via point [ch, di.. where Φb; is the position of joint at time wia and Φ; the corresponding respectively 17 parameters (if the gains are learned),
velocity. Hence, the policy is based which are summarized in Table For our simulations we handcrafted features, namely the overall success, the maximum deviation of the CoM in and direction and the velocities of the CoM for the r and directions at 200 m.s respectively 400 ms. In Table 3 the features used in this paper are summarized. Simultaneously learning the features is out scope of this comparison human motor performance but part of future work.
We simulated the humanoid in each run for maximum of fmxx = 2s with simulation time step of 0.002 s, such that a maximum of has been stopped at the simulation step Nond if either the stand up has been failed the maximum simulation time has been reached. The return of a roll-out J(0) is composed according to (()
1000 simulation steps are used. The simulation
- (CCallanc + Cime + Coonrol) with the balancing costs Obalance = 1/Nenn En CCooMtaage 3
Naw
the time dependent costs Cume (N Nend) and control costs of Coontrol 10 "E Σ/= uiy.
We compared approach with baseline, standard Bayesian Optimization. For that used the features 4, 5 in 3 and set the number of mental replays to = 3. We initialized both, the BO and the HiBO approach with seed points and generated average statistics over 20 runs. In Figure the comparison between the rewards of the algorithms over 50 episodes is shown. In Figure 6 (a) the
Table 2: Policy parameter description
Kp
proportional gain for joint derivative gain for joint i angle of joint i at the via point angular velocity of joint at the via point time for switching from the via point to goal position
Kn.
Φs
Φi
tvi
7

---

Under review as a conference paper at ICLR 2020
HR0
(a)
(b)
(c)
Figure 6: Comparison of the numher of successful episodes of the proposed HIBO algorithm and the state-of-the-ar approach Bayesian Optimization for different internal experience replay itera- tions. The last three algorithms implemented an automatic relevance determination the Gaussian Process features and policy parameters. Shown are average statistics (mean and standard deviation) over runs and the true data values are denoted by the black dots.
number of successful episodes is illustrated. Our approach requires significantly fewer episodes to improve the reward then standard Bayesian Optimization (10 vs =5 and has higher success quote (78% + 24% 60% + 7%).
We further evaluated the impact of the different features on the learning behavior. In Figure 6 (b) the average statistics over 20 runs for different selected features with mental replays shown. All feature pairs generate better results on average then standard BO, whereas for the evaluated task significant difference in the feature choice was observed.
3.3 EXPLOITING MENTAL REPLAYS
We evaluated our approach with additional experience replays. For that we included an additive noise of trep ~ (0, 0.05) to perturb the policy parameters and features. Figure 6 (c) average statistics over 20 runs of the success rates for different number of replay episodes are shown (rep3= replay episodes). Our proposed algorithm works best with a number of 3 replay episodes. Five or more replays in every iteration steps even reduce the success rate of the algorithm.
CONCLUSION
We introduced HiBO, a hierarchical approach for Bayesian Optimization. We showed that HiBO outperforms standard BO in complex humanoid postural control task. Moreover, demonstrated the effects the choice of the features and for different number of mental replay episodes. We compared our results to the learning performance of real humans at the same task. We found that the learning behavior is similar. We found that our proposed hierarchical BO algorithm can reproduce the rapid motor adaptation of human subjects. In contrast standard BO, our comparison method, is about four times slower. In future work, we will examine the problem of simultaneously learning task relevant features neural nets.
Table 3: Feature description
Feature 1
success
Feature 2
maximum deviation the CoM in æ direction
Feature 3
maximum deviation of the CoM in y direction
Feature 4
velocity of the CoM
direction at 200 ms
Feature 5
velocity of the CoM in direction 200 ms
Feature 6
velocity of the CoM æ direction at 400 ms
Feature 7
velocity of the CoM in direction 400 ms
8

---

Under review as a conference paper at ICLR 2020
REFERENCES
Abbas Abdolmalcki, Rudolf Lioutikov, Jan R Peters, Nuno Lau, Luis Pualo Reis, and Gerhard Neumann. Model-based relative entropy stochastic search. In Advances in Neural Information Processing Systems, pp. 3537-3545, 2015.
Milton Abramowitz and Irene A Stegun. Handbook of mathematical functions: with formulas, graphs, and mathematical tables, volume 55. Courier Corporation, 1965.
Dirk V Arnold and Nikolaus Hansen. A (1+ 1)-ema-es for constrained optimisation. In Proceedings of the 14th annual conference on Genetic and evolutionary computation, pp. 297-304, ACM, 2012.
Paolo Baerlocher and Ronan Boulic. Task-priority formulations for the kinematic control of highly redundant articulated structures. In IROS, pp. 13-17, 1998.
J Andrew Bagnell and Jeff Schneider. Covariant policy search. In Proceedings of the 18th interna- fional joint conference Artificial intelligence, pp. 1019-1024. Morgan Kaufmann Publishers Inc., 2003.
Felix Berkenkamp, Angela P Schoellig, and Andreas Krause. Safe controller optimization for quadrotors with gaussian processes. In Robotics and Automation (ICRA), 2016 IEEE Interna- fional Conference on, pp. 491 196.. IEEE, 2016.
Roberto Calandra, Serena Ivaldi, Mare Peter Deisenroth, Elmar Rueckert, and Jan Peters. Learning inverse dynamics models with contacts. In 2015 IEEE International Conference on Robotics and Automation (ICRA), pp. 3186-3191. IEEE, 2015.
Roberto Calandra, André Seyfarth, Jan Peters, and Mare Peter Deisenroth. Bayesian optimization for learning gaits under uncertainty. Annals of Mathematics and Artificial Intelligence, 76(1-2); 5-23, 2016.
Bo Chen, Rui Castro, and Andreas Krause. Joint optimization and variable selection of high- dimensional gaussian processes. arXiv preprint arXiv: 022.//006666666 2012.
Su Il Choi and Byung Kook Kim. Obstacle avoidance control for redundant manipulators using collidability measure. Robotica, pp. 143-151, 2000.
Peter Englert and Mare Toussaint. Combined optimization and reinforcement learning for manipu- lation skills. In Robotics: Science and Systems, 2016.
Michael Gienger, Herbert Janssen, and Christian Goerick. Task-oriented whole body motion for humanoid robots. In IEEE-RAS, pp. 238-244, 2005.
Robert B Gramacy and Herbert KH Lee. Optimization under unknown constraints. arXiv preprint arXiv:1004. 4027, 2010.
N. Hansen, S.D. Muller, and P. Koumoutsakos. Reducing the time complexity of the derandomized evolution strategy with covariance matrix adaptation (CMA-ES). Evolutionary Computation, (1):1-18, 2003.
Oliver Kramer, André Barthelmes, and Günter Rudolph. Surrogate constraint functions for ema evolution strategies. In K1, pp. 169-176. Springer, 2009.
Cheng Li, Sunil Gupta, Santu Rana, Vu Nguyen, Svetha Venkatesh, and Alistair Shilton. High 2018.
dimensional bayesian optimization using dropout. arXiv preprint
Daniel J Lizotte, Tao Wang, Michael H Bowling, and Dale Schuurmans. Automatic gait optimization with gaussian process regression. In IJCAI, volume 7, pp. 944-949, 2007.
Ruben Martinez-Cantin. Nando de Freitas, Amaud Doucet, and José A Castellanos. Active policy learning for robot planning and exploration under uncertainty. Robotics: Science and Systems, volume 3, pp. 321-328. 2007.

---

Under review as a conference paper at ICLR 2020
B Matérn. Spatial variation: Meddelanden fran statens skogsforskningssisstuut Lecture Notes in Statistics, 36:21, 1960.
Jonas Mockus, Vytautas Tiesis, and Antanas Zilinskas. The application of bayesian methods for seeking the extremum. Towards global optimization, 2(117-129)22 1978.
Valerio Modugno, Gerard Neumann, Elmar Rueckert, Giuseppe Oriolo, Jan Peters, and Serena Ivaldi. Learning soft task priorities for control of redundant robots. In Robotics and Automa- tion (ICRA), 2016 IEEE International Conference on, pp. 221-226. IEEE, 2016.
Riccardo Moriconi, KS Kumar, and Marc P Deisenroth. High-dimensional bayesian optimization with manifold gaussian processes. arXiv preprint arXiv:1902.10677 2019.
Duy Nguyen-Tuosn and Jan Peters. Model learning for robot control: survey. Cognitive process-
ing, 12(4):319-34)) 130-400, 2011.
Duy Nguyen-Tuong. Matthias Seeger, and Jan Peters. Model learning with local gaussian process regression. Advanced Robotics, 23(15):2015-2034, 2009.
Jan Peters, Katharina Mülling, and Yasemin Altün. Relative entropy policy search. In Proceedings of the Twenty-Fourth AAAI Conference on Artificial Intelligence, pp. 1607-1612. AAAI Press, 2010.
Santu Rana, Cheng Li, Sunil Gupta, Vu Nguyen, and Svetha Venkatesh. High dimensional bayesian optimization with elastic gaussian process. In Proceedings of the 34th International Conference on Machine Learning-Volle Volume 70, pp. 2883-28999 JMLR. org, 2017.
Carl Edward Rasmussen. Gaussian processes in machine learning. In Summer School on Machine Learning, pp. 63-71. Springer, 2003.
Elmar Rueckert, Gerhard Neumann, Mare Toussaint, and Wolfgang Maass. Learned graphical mod- els for probabilistic planning provide new class of movement primitives. Frontiers in Computa- fional Neuroscience, 6(97), 2013. doi: 10.3389/fncom. 20w.mm...11...0 20020009.
Elmar Rueckert, Max Mindt, Jan Peters, and Gerhard Neumann. Robust policy updates for stochastic optimal control. In Humanoid Robots (Humanoids), 2014 14th IEEE-RAS International Confer- ence on, pp. 388-393. IEEE, 2014.
Elmar Rueckert, Jernej Camernik, Jan Peters, and Jan Babic. Probabilistic movement models show that postural control precedes and predicts volitional motor control. Nature Publishing Group: Scientific Reports, 6(28455), 2016. doi: 10. 10.1038/srep2845..
Layale Saab, Oscar E Ramos, François Keith, Nicolas Mansard, Philippe Soueres, and Jean-Y Fourquet. Dynamic whole-body motion generation under rigid contacts and other unilateral con- straints. IEEE Transactions on Roboties, 29(2):346 346.232,66666.- 2013.
Frank Sehnke, Christian Osendorfer, Thomas RückstieB, Alex Graves, Jan Peters, and Jürgen Schmidhuber. Parameter-exploring policy gradients. Neural Networks, 23(4):551-55) 2010.
Bobak Shahriari, Kevin Swersky, Ziyu Wang, Ryan Adams, and Nando de Freitas. Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE, 1(104):
148-175, 2016.
Siciliano B Slotine. A general framework for managing multiple tasks in highly redundant robotic systems. In proceeding of 5th International Conference on Advanced Robotics, volume 2, pp. 1211-1216, 1991.
Ziyu Wang, Masrour Zoghi, Frank Hutter, David Matheson, and Nando De Freitas. Bayesian opti- mization in high dimensions via random embeddings. Twenty- International Joint Con- ference Artificial Intelligence, 2013.
Daan Wierstra, Tom Schaul, Jan Peters, and Juergen Schmidhuber. Episodic reinforcement learning by logistic reward-weightn regression. In International Conference on Artificial Neural Net- works, pp. 407-416. Springer, 2008.
10

---

Under review as a conference paper at ICLR 2020
Christopher KI Williams and Carl Edward Rasmussen. Gaussian processes for regression. In Ad- vances in neural information processing systems, pp. 514-520, 1996.
Christopher KI Williams and Carl Edward Rasmussen. Gaussian processes for machine learning, volume 2. MIT Press Cambridge, MA, 2006.


Reviews:
[{"review_id": "rkxtaO2mcr", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 1, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "After rebuttal:\n\nThank you to the authors for responding to my review.\n\n1) The title of the conference is \"... on Learning Representations\". As I stated in the review (\"no, e.g., neural networks are employed\"), neural networks are an *example* of, but do not subsume, all representation learning methods. Therefore, I agree that papers that do not cover neural networks are welcome at the conference. However, as I stated in the review, my evaluation of the method proposed in the submission is that it does not concern representation learning (\"The employed features in Table 3 are handcrafted\"). I believe this evaluation is defensible, but of course the final evaluation is up to the chairs. However, I note that the authors did not respond directly to my evaluation that the method is not engaging in representation learning.\n\n2-7) As the other reviewer notes, the paper lacks clarity in many places, and does not sufficiently discuss prior work, including in postural control (there is one citation in the references that is not mentioned in the main text), hierarchical Bayesian optimization within or without a Gaussian processes framework (https://scholar.google.com/scholar?hl=fr&as_sdt=0%2C5&q=hierarchical+bayesian+optimization&btnG=), or experience replay (https://scholar.google.com/scholar?hl=fr&as_sdt=0%2C5&q=replay+machine+learning&btnG=). Therefore, it is difficult to ascertain the research contribution.\n\nAs such, I stand by my evaluation that this submission is not ready for publication at ICLR.\n\n===========================\n\nBefore rebuttal:\n\nThe submission presents a hierarchical Bayesian optimization (HiBO) approach to solving a postural control task expressed as a proportional-derivative (PD) controller.\n\nStrengths:\n- The HiBO approach outperforms the non-hierarchical BO approach on the task of postural control.\n\nWeaknesses:\n- The paper does not make use of representation learning (no, e.g., neural networks are employed) and is therefore out-of-place at ICLR. The employed features in Table 3 are handcrafted.\n- The task (simulating human postural control) is not well-situated in the context of prior work using HiBO for robotics, so the contribution remains unclear.\n- It is not clear why this problem should be formulated as contextual policy search (i.e., to what the context variable refers).\n- Only one baseline (Bayesian optimization (BO)) is reported. This baseline corresponds to the ablation of the HiBO method (i.e., the omission of the context variable), and so does not represent, more broadly, an alternative approach.\n- The concept of \"mental replay\" is briefly introduced, but no reference is made to prior work in imagined rollouts, and no ablation study on the impact of this component is performed.\n\nMinor points:\n- It is unclear why the problem setting should be labeled as \"psychological\" postural control.\n- There are several missing references (\"?\") in the text.", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}, {"review_id": "rJlNlmY5tH", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 1, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "How to quickly learn control policies with minimized number of environment interactions have long been an important problem. To tackle this problem, this paper proposed a \"hierarchical Bayesian optimization (HIBO)\" algorithm to optimize the \"feature parameter \\phi\" (which I don't know what that is) and the \"policy parameter \\theta\" hierarchically. Under the formulation of maximizing reward J(\\theta), the algorithm firstly uses EI to select \\phi. Given the selected \\phi, the algorithm selects the policy parameter \\theta. The proposed algorithm is evaluated on a Humanoid Postural Balancing task, and shows achieves high rewards faster than the standard EI acquisition. However, the paper is awfully written such that I cannot understand what the \"feature parameter \\phi\" is. Given my limited understanding, I think the paper should be rejected.\n\nStrengths,\n1, The paper deals with an interesting task: Humanoid Postural Balancing. A Humanoid is expected to learn how to balance as quick as possible to reduce the interactions with the environments, which suits well with Bayesian optimization.\n\nWeakness,\n1, The paper is awfully written. The problem statement subsection is unreadable. I don't see anywhere explaining how the states x_t, commands u_t, \\theta, and feature \\phi are related? What is \\phi ? It is super wired why the feature parameter \\phi is jointly maximized with the policy parameter.  Because I don't understand the formulation, I can hardly understand anything else.\n2, From my very limited understanding on the formulation, the proposed HIBO is trivial.\n3, The experiments are limited. The paper only conducts one experiment on the Humanoid control balancing. And they paper only compares with the EI acquisition, while the state-of-art acquisition MES should be also be compared with.\n4, The proposed mental replay is not well justified, qualitatively or empirically.", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_checking_correctness_of_derivations_and_theory": "N/A", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_thoroughness_in_paper_reading": "N/A"}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper proposes hierarchical Bayesian optimization (HiBO) for learning control policies from a small number of environment interaction and applies it to the postural control of a humanoid. Both reviewers raised issues with the clarity of presentation, as well as contribution and overall fit to this venue. The authors’ response helped to clarify these issues only marginally. Therefore, primarily due to lack of clarity, I recommend rejecting this paper, but encourage the authors to improve the presentation as per the reviewers’ suggestions and resubmitting.
Decision: Reject