A Probabilistic Approach to Mixed Open-loop and Closed-loop Control,with Application to Extreme Autonomous Driving
J. Zico Kolter, Christian Plagemann, David T. Jackson, Andrew Y. Ng, Sebastian Thrun
Abstract— We consider the task of accurately controlling acomplex system, such as autonomously sliding a car sideways
into a parking spot. Although certain regions of this domain
are extremely hard to model (i.e., the dynamics of the carwhile skidding), we observe that in practice such systems areoften remarkably deterministic over short periods of time,even in difficult-to-model regions. Motivated by this intuition,
we develop a probabilistic method for combining closed-loop
control in the well-modeled regions and open-loop control
in the difficult-to-model regions. In particular, we show thatby combining 1) an inaccurate model of the system and 2)
a demonstration of the desired behavior, our approach canaccurately and robustly control highly challenging systems,
without the need to explicitly model the dynamics in themost complex regions and without the need to hand-tune theswitching control law. We apply our approach to the task ofautonomous sideways sliding into a parking spot, and show that
we can repeatedly and accurately control the system, placing
the car within about 2 feet of the desired location; to the bestof our knowledge, this represents the state of the art in termsof accurately controlling a vehicle in such a maneuver.

I. I NTRODUCTION
In this paper, we consider control strategies that employ
a mixture of closed-loop control (actively controlling thesystem based on state feedback) and open-loop control(executing a fixed sequence of control inputs without any
feedback). To motivate such strategies, we focus on the taskof sliding a car sideways into a parking spot — that is,accelerating a car to 25 mph, then slamming on the brakes
while turning the steering wheel, skidding the car sideways
into a narrow desired location; this is a highly challenging
control task, and therefore an excellent testbed for learningand control algorithms. One feature of this and many othercontrol tasks is that the system operates in multiple distinct
regimes that can be either easy or hard to model: for instance,
a “normal driving” regime, where the dynamics of the car
are well understood and easily modeled, and a “sideways
sliding” regime, where the dynamics are extremely difficult
to model due to time-delay or non-Markovian effects.
Although it may appear that controlling the system inthese hard-to-model regimes is nearly impossible, if we havepreviously observed a trajectory demonstrating the desired
behavior from the current or nearby state, then we could
execute this same behavior in hopes of ending up in a similar
final state. It may seem surprising that such a method has
any chance of success, due to the stochasticity of the world.
However, we repeatedly observe in practice that the world
All authors are with the Computer Science Department, Stanford
University, Stanford CA, 94305 {kolter, plagem, dtj, ang,

thrun}@cs.stanford.edu

is often remarkably deterministic over short periods of time,
given a fixed sequence of control inputs, even if it is very
hard to model how the system would respond to different
inputs; we have observed this behavior in systems such as
the sideways-sliding car, quadruped robots, and helicopters.
Intuitively, this observation motivates an alternating control
strategy, where we would want to actively control the systemusing classical methods in well-modeled regimes, but executeopen-loop trajectories in poorly modeled regions, provided
we have seen a previous demonstration of the desired behavior. Indeed, as we will discuss shortly, such strategies havepreviously been used to control a variety of systems, thoughthese have mostly relied on hand-tuned switching rules todetermine when to execute the different types of control.
In this paper, we make two contributions. First, we develop
a probabilistic approach to mixed open-loop and closedloop control that achieves performance far superior to eachof these elements in isolation. In particular, we propose amethod for applying optimal control algorithms to a probabilistic combination of 1) a “simple” model of the system thatcan be highly inaccurate in some regions and 2) a dynamics
model that describes the expected system evolution when
executing a previously-observed trajectory. By using varianceestimates of these different models, our method probabilistically interpolates between open-loop and closed-loop control,
thus producing mixed controllers without any hand-tuning.
Our second contribution is that we apply this algorithm to theaforementioned task of autonomously sliding a car sideways
into a parking spot, and show that we can repeatedly andaccurately control the car, without any need to model the
dynamics in the sliding regime. To the best of our knowledge,
this is the first autonomous demonstration of such behavior
on a car, representing the state of the art in accurate control
for such types of maneuvers.
II. BACKGROUND AND R ELATED W ORK
There is a great deal of work within both the machinelearning and control communities that is relevant to thework we have presented here. As mentioned, previously, themixed use of open-loop and closed-loop control is a common
strategy in robotics and control (e.g., [3], [10], [23]), thoughtypically these involve hand-tuning the regions of closedloop/open-loop control. The work of Hansen et al., [9] also
considers a mixture of open-loop and closed-loop control,
though here the impetus for such control comes from acost to sensing actions, not from difficulty of modeling thesystem, and thus the algorithms are very different.

Our work also relates to work on multiple models forcontrols tasks [14]. However, with few exceptions the term
“multiple models” is typically synonymous with “local models,” building or learning a collection of models that areeach accurate in a different portion of the state space –for example, locally linear models, [18], [5], [24] or nonparametric models such as Gaussian Processes [13]. While
our algorithm can be generally interpreted as an applicationof a particular type of local models, the basic intuition behind
the way we combine models is different: as we will discuss,
we aren’t seeking to necessarily learn a better model, nor
is each model always tailored to a specific region along the
trajectory; rather the models we will use capture different
elements — an inaccurate model of the whole system, anda model of the trajectory that provides no information at
all about state or control derivatives. Doya et al., [7] also
propose a model-based reinforcement learning algorithm
based on multiple models, but again these models are local
in nature.
From the control literature, our work perhaps most closely
resembles the work on multiple model adaptive control [15],[19]. Like our work, this line of research takes motivationfrom the fact that different models can capture the system
dynamics with different degrees of uncertainty, but again
like the previous work in machine learning, this research
has typically focused on learning or adapting multiple local
models of the system to achieve better performance.
Our work also relates somewhat more loosely to a variety
of other research in reinforcement learning. For instance,
the notion of “trajectories” or “trajectory libraries” has been
explored for reinforcement learning and robotics [20], [21],but this work has primarily focused on the trajectories as ameans for speeding up planning in a difficult task. Abbeel
et al., [1] consider the problem of reinforcement learning
using inaccurate models, but their approach still relies on
the inaccurate model having accurate derivatives, and indeed
the pure LQR approach we employ is very similar to theiralgorithm provided we linearize around a trajectory from thereal system.
The notion of learning motor primitives has received agreat deal of attention recently (e.g. [16]), though such research is generally orthogonal to what we present here. These
motor primitives typically involve feedback policies that are
not model-based, and we imagine that similar techniques
to what we propose here could also be applied to learn
how to switch between model-based and motor-primitivebased control. Indeed to some extent our work touches on
the issue of model-based versus model-free reinforcement
learning [4] where our open-loop controllers are simply avery extreme example of model-free control, but we could
imagine applying similar algorithms to more general modelfree policies.
Finally, there has been a great deal of past work on
autonomous car control, for example [11]. Some of this work
has even dealt with slight forays into control at the limits ofhandling [12], but all such published work that we are awareof demonstrates significantly less extreme situations thanwhat we describe here, or includes only simulation results.
We are also aware of another group at Stanford working
independently on control at the limits of handling [8], but
this work revolves more around physics-based techniques forstabilizing the system in an unstable “sliding” equilibrium,
rather than executing maneuvers with high accuracy.
III. P RELIMINARIES
We consider a discrete-time, continuous state, and continuous action environment, where the state at some time
t is denoted as st ∈ Rn and the control as ut ∈ Rm . The
dynamics of the system are governed by some unknown (andnoisy) dynamics
st+1 = f (st , ut ) + ǫt ,nmwhere f : R × R → Rn gives the mean state and ǫt issome zero-mean noise term.
The control task we consider is that of following a desired
trajectory on the system, {s⋆0:H , u⋆0:H−1 }.1 More formally,
we want to choose controls that will minimize the expected
cost
H−1

X
δsTt Qδst + δuTt Rδut + δsTH QδsH
J =E
t=0
(1)

st+1 = f (st , ut ) + ǫt

where δst ≡ st − s⋆t and δut ≡ ut − u⋆t denote the stateand control error, and where Q ∈ Rn×n and R ∈ Rm×m
are symmetric positive semidefinite matrices that determine
how the cost function trades off errors in different stateand control variables. Even when the desired trajectoryis “realizable” on the true system — for example, if itcomes from an expert demonstration on the real system —minimizing this cost function can be highly non-trivial: dueto the stochasticity of the world, the system will naturally
deviate from the intended path, and we will need to executecontrols to steer the system back to its intended path.
A. Linear Quadratic Regulator control
While there are many possible methods for minimizing
the cost function (1), one method that is particularly wellsuited to this task is the Linear Quadratic Regulator (LQR)
algorithm (sometimes also referred to as the Linear Quadratic
Tracker when the task is specifically to follow some target
trajectory) — there have been many recent successes with
LQR-based control methods in the both the machine learningand control communities [6], [22]. Space constraints preclude
a full review of the algorithm, but for a more detailed
description of such methods see e.g. [2]. Briefly, the LQR
algorithm assumes that the (error) dynamics of the systemevolve according to a linear modelδst+1 = At δst + Bt δut + wt

(2)

1 Note that in this formulation the trajectory includes both states and controls and could be produced, for instance, by a human expert demonstrating
the control task. In cases where the controls (or the desired intermediate
states) are not specified, we could use a variety of planning algorithms todetermine controls and desired states, though this is an orthogonal issue.

where At ∈ Rn×n and Bt ∈ Rn×m are system matrices,and wt is a zero-mean noise term. Under this assumption, it
is well-known that the optimal policy for minimizing thequadratic cost (1) is a linear policy — i.e., the optimal
control is given by δut = Kt δst for some matrix Kt ∈Rm×n that can be computed using dynamic programming.
To apply LQR to our control task, we of course need to
specify the actual At and Bt matrices. Ideally, these matrices
would be the state and action derivatives (of the true model)
evaluated along the trajectory. However, since we do nothave access to the true model, we typically determine thesematrices using an approximate model. More formally, we
suppose we have some approximate model of the form
st+1 ≈ fˆ(st , ut )where, as before, fˆ : Rn × Rm → Rn predicts the next state
given the current state and control, and we set At and Bt
to be the Jacobians of this model, evaluated along the target
trajectory
At = Ds fˆ(s, u)⋆
s=s⋆
t ,u=ut

Bt = Du fˆ(s, u)⋆
s=s⋆
t ,u=ut

approximate models are probabilistic, in that for a given
state and control they return a Gaussian distribution overnext states, with mean fi (st , ut ) and covariance Σi (st , ut ),i.e.,p(st+1 |Mi ) = N (fi (st , ut ), Σi (st , ut )).
The covariance terms Σi (st , ut ) are interpreted as maximum
likelihood covariance estimates, i.e., for true next state st+1 ,
Σi (st , ut ) = E (st+1 − fi (st , ut ))(st+1 − fi (st , ut ))T

so the covariance terms capture two sources of error: 1) thetrue stochasticity of the world (assumed small), and 2) the
inaccuracy of the model. Thus, even if the world is largely
deterministic, model variance still provides a suitable meansfor combining multiple probabilistic models.
To combine multiple probabilistic models of this form,
we interpret each prediction as an independent observationof the true next state, and can then compute the posterior
distribution over the next state given both models by astandard manipulation of Gaussian probabilities,p(st+1 |M1 , M1 ) = N (f¯(st , ut ), Σ̄(st , ut )).

Despite the fact that we now are approximating the originalcontrol problem in two ways — since we use an approximate
model fˆ and additionally make a linear approximations tothis function — the algorithm can work very well in practice,provided the approximate model captures the derivatives of
the true system well (where we are intentionally somewhat
vague in what we mean by “well”). When the approximation
model is not accurate in this sense (for example, if it
misspecifies the sign of certain elements in the derivatives
at some point along the trajectory), then the LQR algorithm
can perform very poorly.
IV. A PROBABILISTIC FRAMEWORK FOR MIXEDCLOSED - LOOP AND OPEN - LOOP CONTROLIn this section, we present the main algorithmic contribution of the paper, a probabilistic method for mixed
closed-loop and open-loop control. We first present a general
method, called Multi-model LQR, for combining multiple
probabilistic dynamics models with LQR — the model
combination approach uses estimates of model variancesand methods for combining Gaussian predictions, but somemodifications are necessary to integrate fully with LQR.
Then, we present a simple model for capturing the dynamics
of executing open-loop trajectories, and show that when we
combine this with an inaccurate model using Multi-model
LQR, the resulting algorithm naturally trades off between
actively controlling the system in well-modeled regimes, andexecuting trajectories in unmodelled portions where we havepreviously observed the desired behavior.
A. LQR with multiple probabilistic models
Suppose we are given two approximate models of a
dynamical system M1 and M2 — the generalization tomore models is straightforward and we consider two justfor simplicity of presentation. We also suppose that these

where
−1
Σ̄ = Σ−1
1 + Σ2−1−1, and f¯ = Σ̄ Σ−1
1 f1 + Σ2 f2

and where the dependence on st and ut is omitted to simplify
the notation in this last line. In other words, the posterior
distribution over next states is a weighted average of the two
predictions, where each model is weighted by its inverse
covariance. This combination is analogous to the Kalmanfilter measurement update.
We could now directly apply LQR to this joint model bysimply computing f¯(s⋆t , u⋆t ) and its derivatives at each point
along the trajectory — i.e., we could run LQR exactly as described in the previous section using the dynamics (2) where
At and Bt are the Jacobians of f¯, evaluated at s⋆t and u⋆t ,and where wt ∼ N (0, Σ̄(s⋆t , u⋆t )). However, combining thesemodels in this manner will lead to poor performance. Thisis due to the fact that we would be computing Σ1 (s⋆t , u⋆t )and Σ2 (s⋆t , u⋆t ) only at the desired location, which can givea very inaccurate estimate of each model’s true covariance.
For example, consider a learned model that is trained only
on the desired trajectory; such a model would have very low
variance at the actual desired point on the trajectory, butmuch higher variance when predicting nearby points. Thus,
what we really want to compute is the average covarianceof each model, in the region where we expect the system
to be. We achieve this effect by maintaining a distribution
Dt over the expected state (errors) at time t. Given such a
distribution, we can approximate the average covarianceΣi (Dt ) ≡ Eδst ,δut ∼Dt [Σi (s⋆t + δst , u⋆t + δut )]via sampling or other methods.
The last remaining element we need is a method for computing the state error distributions Dt . Fortunately, given thelinear model assumptions that we already employ for LQR,
computing this distribution analytically is straightforward. In

Algorithm 1
Multi-model LQR(D0:H , M1 , M2 , s⋆0:H , u⋆0:H−1 , Q, R)
Input:
D0:H : initial state error distributions
M1 , M2 : probabilistic dynamics modelss⋆0:H , u⋆0:H−1 : desired trajectory
Q, R: LQR cost matrices
Repeat until convergence:
1. For t = 0, . . . , H − 1, compute dynamics models• Σi (Dt ) ← Eδst ,δut ∼Dt [Σi (s⋆t + δst , u⋆t + δut )] ,i = 1, 2 (computed via sampling)−1−1• Σ̄t ← Σ−1
1 (Dt ) + Σ2 (Dt )• f¯ ← Σ̄t Σ−1 (Dt )f1 + Σ−1 (Dt )f2
1

2• At ← Ds f¯(s, u) s=s⋆ ,u=u⋆tt• Bt ← Du f¯(s, u)⋆⋆
s=st ,u=ut

2. Run LQR to find optimal controllers Kt• K0:H−1 ← LQR(A0:H−1 , B0:H−1 , Q, R)
3. For t = 0, . . . , H − 1, update state distributions• Γt+1 ← (At + Bt Kt )Γt (At + Bt Kt )T + Σ̄(Dt )• Dt ← {δst ∼ N (0, Γt ), δut ∼ N (0, Kt Γt KtT )}particular, we first assume that our initial state distribution
D0 is a zero-mean Gaussian with mean zero and covarianceΓ0 . Then, since our state dynamics evolve according to thelinear model (2), at each time step the covariance is updatedbyΓt+1 = (At + Bt Kt )Γt (At + Bt Kt )T + Σ̄(Dt ).
Of course, since we average the different models according
to the Σi (Dt ), changing Dt will therefore also change At
and Bt . Thus, we iteratively compute all these quantities untilconvergence. A formal description of the algorithm is given
in Algorithm 1.
B. A dynamics model for open-loop trajectories

only on the state and control error and which captures thecovariance of the model as a function of how far away
we are from the desired trajectory. This model captures thefollowing intuition: if we are close to a previously observed
trajectory, then we expect the next error state to be similar,
with a variance that increases the more we deviate from thepreviously observed states and actions.
To see how this model naturally leads to trading off
between actively controlling the system and largely following
known trajectory controls, we consider a situation where
we run the Multi-model LQR algorithm with an inaccurate
model2 M1 and this described trajectory model, M2 . Consider a situation where our state distribution Dt is tightly
peaked around the desired state, but where M1 predicts thenext state very poorly. In this case, the maximum likelihood
covariance Σ1 (Dt ) will be large, because M1 predicts poorly,but Σ2 (Dt ) will be much smaller, since we are still close to
the desired trajectory. Therefore f¯ will be extremely close to
the trajectory model M2 , and so will lead to system matrices
At ≈ ρI and Bt ≈ 0 (the derivatives of the trajectory model).
Since this model predicts that no control can affect the statevery much, LQR will choose controls δut ≈ 0 — i.e.,
execute controls very close to u⋆t , as this will minimize theexpected cost function. In contrast, if we are far away fromthe desired trajectory, or if the model M1 is very accurate,then we would expect to largely follow this model, as it
would have lower variance than the naive trajectory model.
In practice, the system will smoothly interpolate between
these two extremes based on the variances of each model.
C. Estimating Variances
Until now, we have assumed that the covariance terms
Σi (st , ut ) have been given, though in practice these will need
to be estimated from data. Recall that the variances we wantto learn are ML estimates of the form
Σi (st , ut ) = E (st+1 − fi (st , ut ))(st+1 − fi (st , ut ))T

so we could use any number of estimation methods to learn
these covariances, and our algorithm is intentionally agnostic
about how these covariances are obtained. Some modelingalgorithms, such as Gaussian Processes, naturally provide
such estimates, but for other approaches we need to use
additional methods to estimate the covariances. Since weemploy different methods for estimating the variance in thetwo sets of experiments below, we defer a description of themethods to these sections below.

Here we propose a simple probabilistic model that predicts
how the state will evolve when executing a previously
observed sequence of control actions. While generally one
would want to select from any number of possible trajectories
to execute, for the simplified algorithm in this section we
consider only the question of executing the control actionsof the ideal trajectory u⋆0:H−1 .
To allow our model to be as general as possible, we assume
a very simple probabilistic description of how the states
evolve when executing these fixed sequences of actions.
In particular, we assume that the error dynamics evolve
according toδst+1 = ρδst + wt

In this section we present experimental results, first on asimple simulated cart-pole domain, and then on our main
application task for the algorithm, the task of autonomous
sideways sliding into a parking spot.

where ρ ∈ R (typically ≈ 1) indicates the stability of taking
such trajectory actions, and where wt is a zero-mean Gaussian noise term with covariance Σ(δst , δut ), which depends

2 For the purposes of this discussion it does not matter how M is1
obtained, and could be built, for example, from first principles, or learned
from data, as we do for the car in Section V.

V. E XPERIMENTS

1

100

0.5

90

LQR, inaccurate Model

Open Loop

LQR, GP Model
80

70

−1

−2.5

−2

−1.5

−1

−0.5

0

0.5

1

1.5

2

2.5

1

Average total cost

0

−0.5

60

50

40

0.5

30

0

20

−0.5

10

−1

Hand−tuned switching controller

0Multi−model LQR

LQR, true model

50

100

150

200

250

300

350

400

450

500

550

Time
−2.5

−2

−1.5

−1

−0.5

0

0.5

1

1.5

2

2.5

1

0.5

0

−0.5

−1

−2.5

−2

−1.5

−1

−0.5

0

0.5

1

1.5

2

2.5

Fig. 1. (top) Trajectory followed by the cart-pole under open-loop control(middle) LQR control with an inaccurate model and (bottom) Multi-model
LQR with both the open loop trajectory and inaccurate model.

A. Cart-pole Task
To easily convey the intuition of our algorithm, andalso to provide a publicly available implementation, we
evaluated our algorithm on a simple cart-pole task (where
the pole is allowed to freely swing entirely around thecart). The cart-pole is a well-studied dynamical system,
and in particular we consider the control task of balancing
an upright cart-pole system, swinging the pole around acomplete revolution while moving the cart, and then balancing upright once more. Although the intuition behind
this example is simple, there are indeed quite a few implementation details, and due to space constraints we defer
most of this discussion to the source code, available at:
http://ai.stanford.edu/˜kolter/icra10.
The basic idea of the cart-pole task for our setting is asfollows. First, we provide our algorithm with an inaccurate
model of the dynamical system; the model uses a linear
function of state features and control to predict the next state,
but the features are insufficient to fully predict the cart-pole
dynamics. As a result the model is fairly accurate in the
upright cart-pole regions, but much less accurate during theswing phase. In addition, we provide our algorithm with asingle desired trajectory (the target states and a sequenceof controls that will realize this trajectory under a zero-noise
system). Because we are focused on evaluating the algorithm
without consideration for the method used to estimate thevariances, for this domain we estimate the variances exactly
using the true simulation model and sampling.
Figure 1 shows the system performance using three different methods: 1) fully “open loop” control, which just replays
the entire sequence of controls in the desired trajectory, 2)
running LQR using only the inaccurate model, and 3) using
the Multi-model LQR algorithm with both the inaccurate

MethodLQR (true model)
LQR (inaccurate model)
LQR (GP model)
Open Loop
Multi-model LQR
Hand-tuned Switching Control

Avg. Total Cost
18.03 ± 1.61
96,191 ± 21,327
96,712 ± 21,315
67,664 ± 13,585
33.51 ± 4.38
73.91 ± 13.74

Fig. 2. Average total cost, with 95% confidence intervals, for different
cart-pole control methods.

model and the trajectory. As can be seen, both pure openloop and LQR with the inaccurate model fail to control thesystem, but Multi-model LQR is able to reliably accomplish
the control task. Figure 2 similarly shows the average total
costs achieved by each of the methods, averaged over 100
runs. Although open loop and inaccurate LQR fail at different
points, they are never able to successfully swing and balance
the pole, while Multi-model LQR performs comparably to
LQR using the actual model of the system dynamics bynaturally interpolating between the two methods in order
to best control the system. In the figure we also show
the error of a hand-tuned switching policy that switches
between purely model-based and open-loop control; despite
exhaustive search to find the optimal switch points, themethod still performs worse here than Multi-model LQR.
In Figure 2, we also compare to a Gaussian Process(GP) model (see, e.g. [17] for detailed information about
Gaussian Processes) that attempted to learn a better dynamics
model by treating the inaccurate model as the “prior” andupdating this model using state transitions from the desired
trajectory. However, the resulting model performs no better
than LQR with the inaccurate model. We emphasize that
we are not suggesting that Gaussian processes cannot model
this dynamical system — they can indeed do so quite easily
given the proper training data. Rather, this shows that thedesired trajectory alone is not sufficient to improve a GP
model, whereas the Multi-model LQR algorithm can perform
well using only these two inputs; this is an intuitive result,since observing only a single trajectory says very little about
the state and control derivatives along the trajectory, which
are ultimately necessary for good fully LQR-based control.
Indeed, we have been unable to develop any other controller
based only on the inaccurate model and the desired trajectorythat performs as well as Multi-model LQR.

Fig. 3. Snapshots of the car attempting to slide into the parking spot using (top) open-loop control, (middle) pure LQR control, and (bottom) Multi-model
LQR control. The desired trajectory in all cases is to slide in between the cones.
15
15
Desired Trajectory
Open Loop

10

5

0

0

0

−5

−5

−5

−10

−10

0

10

20

30

40

50

60

70

5

−15

−10

10

20

30

40

50

60

70

5

−15

0

−5

−5

−5

−10

−10

−10

4

6

8

10

12

14

16

18

20

−15

20

30

40

50

60

70

Desired Trajectory
Multi−Model LQR

0

2

10

Desired Trajectory
LQR, inaccurate model

0

0

0

5

Desired Trajectory
Open Loop−15

Desired Trajectory
Multi−Model LQR

10
5

5

−15

15

Desired Trajectory
LQR, inaccurate model

10

0

2

4

6

8

10

12

14

16

18

20

−15

0

2

4

6

8

10

12

14

16

18

20

Fig. 4. Plots of the desired and actual trajectories followed by the car under (left) open-loop control, (middle) pure LQR control, and (bottom) Multi-model
LQR control. Bottom plots show a zoomed-in view of the final location.

B. Extreme Autonomous Driving
Finally, in this section we present our chief applied resultof the paper, an application of the algorithm to the taskof extreme autonomous driving: accurately sliding a car
sideways into a narrow parking space. Concurrent to thiswork, we have spent a great deal of time developing an LQRbased controller for “normal” driving on the car, based ona non-linear model of the car learned entirely from data;
this controller is capable of robust, accurate forward driving
at speeds up to 70 mph, and in reverse at speeds up to
30 mph. However, despite significant effort we were unable
to successfully apply the fully LQR-based approach to thetask of autonomous sliding, which was one of the main
motivations for this current work.
Briefly, our experimental process for the car sliding task
was as follows. We provided to the system two elements;
first we learned a driving model for “normal” driving, learned
with linear regression and feature selection, built using about
2 minutes of driving data. In addition, we provided thealgorithm a single example of a human driver executing asideways sliding maneuver; the human driver was making noattempt to place the car accurately, but rather simply putting
the car into an extreme slide and seeing where it wound up.
This demonstration was then treated as the target trajectory,
and the goal of the various algorithms was to accurately
follow the same trajectory, with cones placed on the groundto mark the supposed location of nearby cars.
For this domain, we learned domain parameters needed by
the Multi-model LQR algorithm (in particular, the covarianceterms for the inaccurate and open-loop models, plus theρ term) from data. To reduce the complexity of learning
the model variances, we estimated the covariance terms
as follows: for the inaccurate model we estimated a timevarying (but state and control independent) estimate of thevariance by computing the error of the model’s predictions
for each point along the trajectory, i.e.,
et = (st+1 − f1 (st , ut ))(st+1 − f1 (st , ut ))Tthen averaged these (rank-one) matrices over a small time
window to compute the covariance for time t. For the open-loop trajectory model, we learned a state and control dependent (but not time dependent) estimate of the covariance
of the form Σ2 (st , ut ) = w1 kδut k2 + w2 kδst k2 + w3 I,
where we learned the parameters w1 , w2 , w3 > 0 via leastsquares; this model captures the intuition that the variance ofthe open-loop model increases for points that are farther fromthe desired trajectory. Finally we selected ρ = 1 due to thefact that the system rarely demonstrates extremely unstablebehavior, even during the slide.
Figure 3 shows snapshots of the car attempting to executethe maneuver under the three methods of control: open-loop,
pure LQR, and our Multi-model LQR approach integrating
both the inaccurate model and the trajectory. Videos of thedifferent slides are included in the video accompanying thispaper. It is easy to understand why each of the methods
perform as they did. Purely open-loop control actually does
perform a reasonable slide, but since it takes some distancefor the car to build up enough speed to slide, the trajectory
diverges significantly from the desired trajectory during thistime, and the slide slams into the cones. Pure LQR control,on the other hand, is able to accurately track the trajectoryduring the backward driving phase, but is hopeless when
the car begins sliding: in this regime, the LQR model isextremely poor, to the point that the car executes completely
different behavior while trying to “correct” small errors
during the slide. In contrast, the Multi-model LQR algorithm
is able to capture the best features of both approaches,
resulting in an algorithm that can accurately slide the car into
the desired location. In particular, while the car operates inthe “normal” driving regime, Multi-model LQR is able to use
its simple dynamics model to accurately control the car along
the trajectory, even in the presence of slight stochasticity
or a poor initial state. However, when the car transitions
to the sliding regime, the algorithm realizes that the simple
dynamics model is no longer accurate, and since it is still
very close to the target trajectory, it largely executes the
open-loop slides controls, thereby accurately following thedesired slide. Figure 4 shows another visualization of the
car for the different methods. As this figure emphasizes, theMulti-model LQR algorithm is both accurate and repeatableon this task: in the trajectories shown in the figure, the final
car location is about two feet from its desired location.
VI. C ONCLUSION
In this paper we presented a control algorithm that probabilistically combines multiple models in order to integrateknowledge both from inaccurate models of the system andfrom observed trajectories. The resulting controllers naturally
trade off between closed-loop and open-loop control in an
optimal manner, without the need to hand-tune a switching
controller. We applied the algorithm to the challenging taskof autonomously sliding a car sideways into a parking
spot, and show that we can reliably achieve state-of-the-art
performance in terms of accurately controlling such a vehicle
in this extreme maneuver.
ACKNOWLEDGMENTS
We thank the Stanford Racing Team, Volkswagen ERL,
and Boeing for support with the car. J. Zico Kolter is

supported by an NSF Graduate Research Fellowship.
R EFERENCES[1] Pieter Abbeel, Morgan Quigley, and Andrew Y. Ng. Using innaccurate
models in reinforcement learning. In Proceedings of the International
Conference on Machine Learning, 2006.
[2] Brian D. O. Anderson and John B. Moore. Optimal Control: Linear
Quadratic Methods. Prentice-Hall, 1989.
[3] Christoper G. Atkeson and Stefan Schaal. Learning tasks from a single
demonstration. In Proceedings of the International Conference on
Robotics and Automation, 1997.
[4] Christopher G. Atkeson and Juan Carlos Santamaria. A comparisonof direct and model-based reinforcement learning. In Proceedings ofthe International Conference on Robotics and Automation, 1997.
[5] Stefan Schaal Christopher G. Atkeson, Andrew W. Moore. Locallyweighted learning for control. Artificial Intelligence Review, 11:75–
113, 1997.
[6] Adam Coates, Pieter Abbeel, and Andrew Y. Ng. Learning for control
from muliple demonstrations. In Proceedings of the International
Conference on Machine Learning, 2008.
[7] Kenji Doya, Kazuyuki Samejima, Ken ichi Katagiri, and Mitsuo
Kawato. Multiple model-based reinforcement learning. NeuralComputation, pages 1347–1369, 2002.
[8] Chris Gerdes. Personal communication.
[9] Eric Hansen, Andrew Barto, and Shlomo Zilberstein. Reinforcement
learning for mixed open-loop and closed-loop control. In Neural
Information Processing Systems, 1996.
[10] Jessica K. Hodgins and Marc H. Raibert. Biped gymnastics. International Journal of Robotics Research, 9(2):115–128, 1990.
[11] Gabriel M. Hoffmann, Claire J. Tomlin, Michael Montemerlo, andSebastian Thrun. Autonomous automobile trajectory tracking for offroad driving: Controller design, experimental validation and racing. In
Proceedings of the 26th American Control Conference, 2007.
[12] Yung-Hsiang Judy Hsu and J. Christian Gerdes. Stabilization of asteer-by-wire vehicle at the limits of handling using feedback linearization. In Proceedings of the 2005 ASME International MechanicalEngineering Congress and Exposition, 2005.
[13] Jonathan Ko, Daniel J. Klein, Dieter Fox, and Dirk Hhnel. Gaussian
processes and reinforcement learning for identification and control ofan autonomous blimp. In Proceedings of the International Conferenceon Robotics and Automation, 2007.
[14] Roderick Murray-Smith and Tor Arne Johansen. Multiple Model
Approaches to Modelling and Control. Taylor and Francis, 1997.
[15] Kumpati S. Narendra and Jeyendran Balakrishnan. Adaptive control
using multiple models. IEEE Transactions on Automatic Control,
pages 171–187, 1997.
[16] Jan Peters and Stefan Schaal. Learning motor primatives withreinforcement learning. In Proceedings of the 11th Joint Symposium
on Neural Computation., 2004.
[17] Carl Edward Rasmussen and Christopher K. I. Williams. GaussianProcesses for Machine Learning. The MIT Press, 2006.
[18] Stefan Schaal. Nonparametric regression for learning. In Proceedingsof the Conference on Adaptive Behavior and Learning, 1994.
[19] Kevin D. Schott and B. Wayne Bequette. Multiple model adaptive
control. In Multiple Model Approaches to Modelling and Control.
1997.
[20] Martin Stolle and Christopher G. Atkeson. Policies based on trajectory
libraries. In Proceedings of the International Conference on Robotics
and Automation, 2006.
[21] Martin Stolle, Hanns Tappeiner, Joel Chestnutt, and Christopher G.
Atkeson. Transfer of policies based on trajectory libraries. In
Proceedings of the International Conference on Intelligent Robots andSystems, 2007.
[22] Yuval Tassa, Tom Erez, and William Smart. Receding horizon
differential dynamic programming. In Neural Information Processing
Systems 20, 2007.
[23] Claire Tomlin. Maneuver design using reachability algorithms, withapplications to STARMAC flight. Presentation at RSS Workshop:
Autonomous Flying vehicles - Fundamentals and Applications.
[24] Sethu Vijayakumar, Aaron D’Souza, and Stefan Schaal. Incremental
online learning in high dimensions. Neural Computation, 17:2602–
2634, 2005.