1ME 233 Advanced Control II
Lecture 8
Discrete TimeLinear Quadratic Gaussian (LQG)
Optimal Control(ME233 Class Notes pp.LQG1-LQG7)

2Outline• Stochastic optimization• Finite horizon LQG– State feedback optimal LQG control– Output feedback optimal LQG control

3

Stochastic Control
Linear system contaminated by noise:

W
Bw
UV+B

(zI-A ) -1

X+C

Y

Two random disturbances:• Input noise w(k) - contaminates the state x(k)• Measurement noise v(k) - contaminates theoutput y(k)4Stochastic state modelWhere:•available output•

control input•

Gaussian, uncorrelated, zero mean, input noise•

Gaussian, uncorrelated, zero mean, meas. noise•

Gaussian initial state5Assumptions (same as for KF)•Initial conditions:•Noise properties:

Zero-mean
Gaussian
uncorrelated
noises6

Some notation- control and measurements
The control sequence from k to N-1

The optimal control sequence from k to N-1

The output measurements up to k

7

Finite-horizon LQG
For N > 0, find the optimal control sequence:

Which minimizes the cost functional:

wherecan only be based on the observations

8Separation Principle
Main Theorem:
The optimal control is given by:

Where:• The feedback gain K(k) is obtained from thedeterministic LQR solution.

• The state estimate
is the a-posteriori
Kalman Filter state estimate.

9Separation Principle-1

BX

YC(zI-A )UA-posteriori KF

B+-CozX+A

Deterministic LQR- K(k+1)+oY~YF(k)+-X

10Separation Principle Proof
The proof of the separation principle is conductedin two steps:
1. Solve the LQG problem under the assumptionthat the state vector
is measurable
2. Solve the LQG problem and show that theoptimal solution is obtained by replacing
by the a-posteriori state estimate

11Finite-horizon state feedback LQG
This problem is similar to the standard
deterministic finite-horizon LQR…

…except that there is an additional input noise…
…and the control
function ofis only allowed to be a

12Functionality constraint on control• The control u(k) is only allowed to be afunction of x(0),…,x(k)• We write this constraint as• We write the constraints
for k=m,…,N-1 as13Finite-horizon state feedback LQG
We want to solve using dynamic programming:

Need 2 preliminary results:
1. Functional optimization
2. Stochastic Bellman equation

14

Functional optimization
Lemma 1:Let X be a random vector and let
the constraint that u is a function of X

denoteAlso assume that there exists uo(x) such that

Then

15

Functional optimization

Proof is in 2 parts:
1.

2.

u is a function of X

Proof:Let uo(x) minimize f(x,u)

BecauseProof:• Letu is a function of X

This holds, regardless of how• Minimizing the right-hand side overcompletes the proof

was chosen

18

Definitions• Terminal cost

• Stage cost (transient cost)• Optimal cost to go

19

Stochastic Bellman equation
Lemma 2:
IfThenfor

Proof:(m=N-1 case is trivial, and thus omitted)

21

Finite-horizon state feedback LQG
Theorem 1:a)

The optimal control is given byStandard deterministic LQR solution!

22

Finite-horizon state feedback LQG
Theorem 1:b)

The optimal cost Jo is given by

23

Finite-horizon state feedback LQG
Theorem 1:b)
The optimal cost is given byb(k) is a dynamic function of the noise intensityb(k) is computed backwards in time with b(N) = 0

This term reflects the detrimental effect of w(k) on the cost

24

Finite-horizon state feedback LQG
Theorem 1:b)
The optimal cost is given by

Deterministic LQR
cost associatedwith mean of x(0)Detrimental effect ofrandomness of x(0)on the cost

Detrimental effectof w(0),…,w(k)on the cost

25

Finite-horizon state feedback LQG
Proof consists of 2 steps:
1. Proveandusing induction
on decreasing m, Lemma 1, and the stochastic
Bellman equation (Lemma 2)
2. Prove

26

Proof of Theorem 1:
Start with base case: m=Nand.

27

Proof of Theorem 1:

and.

For m=0,1,…,N-1:(We use induction on decreasing m)

By the induction hypothesis,

Term 1
Term 2
Term 3

28

Proof of Theorem 1:

and.
Term 2Since x(m) and u(m) only depend on quantities
that are independent from w(m)Ax(m) + Bu(m) is independent from w(m)029Proof of Theorem 1:and.
Term 3

30

Proof of Theorem 1:
Thereforeand.

31

Proof of Theorem 1:

andNow use stochastic Bellman equation.32

Proof of Theorem 1:and• b(m) does not depend on u(m)• Use Lemma 1 to exchange min and E.

33

Proof of Theorem 1:

andThis is the same optimization wesolved for deterministic LQR!
Optimal value:.

34

Finite-horizon state feedback LQG
Proof consists of 2 steps:
1. Proveandusing induction
on decreasing m, Lemma 1, and the stochastic
Bellman equation (Lemma 2)
2. Prove

35

Proof:

036

Proof: (cont’d)

37

Separation Principle Proof
The proof of the separation principle is conductedin two steps:
1. Solve the LQG problem under the assumptionthat the state vector
is measurable
2. Solve the LQG problem and show that theoptimal solution is obtained by replacing
by the a-posteriori state estimate

38

Finite-horizon LQG
This problem is similar to the standard
deterministic finite-horizon LQR…

…except that there is an additional input noise…
…and the control
function ofis only allowed to be a39

Functionality constraint on control• The control u(k) is only allowed to be afunction of y(0),…,y(k)• As before, we write this constraint as• As before, we write the constraints
for k=m,…,N-1 as40Finite-horizon LQG
We want to solve:

We will relate this to an optimal
state feedback LQG control problem

For simplicity, assume S = 0

41

Reformulation of LQG• Examine0 (by LS property 1)

42

Reformulation of LQG• Therefore,• Similarly,• Want to apply these identities to LQG(Recall that we assumed S = 0)

43

Reformulation of LQG

44

Reformulation of LQG
Terms
minimized by
the Kalmanfilter

We will show that this corresponds to astate feedback LQG control problem45Reformulation of LQG• From the Kalman filter :• Recall thatis uncorrelated and46

Reformulation of LQG

Initial conditions:Notate this as47

Reformulation of LQG

Initial conditions:

Correlation ofwith:

48

Reformulation of LQG
Want to solve:u(k) is a function ofu(k) is a function of(becauseare functions of)u(k) is a function of(because
give any “information” about, i.e. knowledge ofdoes notby LS property 1 )49

Reformulation of LQG
Want to solve:u(k) is a function ofUncorrelated withThis is a state feedback LQG control problem!
Apply results from first half of lecture

50

Optimal finite-horizon LQG, S=0
Main Theorem:a)

The optimal control is given byStandard deterministic LQR solution!

51

Optimal finite-horizon LQG, S=0
Main Theorem:
A-posteriori state observer structure:

52

Optimal finite-horizon LQG, S=0

Main Theorem:b)
The optimal costis given by

53

State space form of LQG controller

Kalmanfilter
LQR

Eliminatingfrom the expression for uo(k) yields

Plugging this expression for uo(k) into the expression foryields the state space model on the next slide

54State space form of LQG controller

whereK(k+1) is the standard deterministic LQR gain

F(k) and L(k) are the standard Kalman filter gains