The separation principle is one of the fundamental principles of stochastic control theory, which states that the problems of optimal control and state estimation can be decoupled under certain conditions. In its most basic formulation it deals with a linear stochastic system

with a state process , an output process  and a control , where  is a vector-valued Wiener process,  is a zero-mean Gaussian random vector independent of , , and , , , ,  are matrix-valued functions which generally are taken to be continuous of bounded variation. Moreover,  is  nonsingular on some interval . The problem is to design an output feedback law  which maps the observed process  to the control input  in a nonanticipatory manner so as to minimize the functional

where  denotes expected value, prime () denotes transpose. and  and  are continuous matrix functions of bounded variation,  is positive semi-definite and  is positive definite for all . Under suitable conditions, which need to be properly stated,  the optimal policy  can be chosen in the form

where  is the linear least-squares estimate of the state vector  obtained from the Kalman filter

where  is the gain of the optimal linear-quadratic regulator obtained by taking  and  deterministic, and where  is the  Kalman gain. There is also a non-Gaussian version of this problem (to be discussed below) where the Wiener process   is replaced by a more general square-integrable martingale with possible jumps. In this case, the Kalman filter needs to be replaced by a nonlinear filter providing an estimate of the (strict sense) conditional mean

where

is the filtration generated by the output process; i.e., the family of increasing sigma fields representing the data as it is produced.

In the early literature on the separation principle it was common to allow as admissible controls  all processes that are adapted to the filtration . This is equivalent to allowing all non-anticipatory Borel functions as feedback laws, which raises the question of existence of a unique solution to the equations of the feedback loop. Moreover, one needs to exclude the possibility that a nonlinear controller extracts more information from the data than what is possible with a linear control law.

Choices of the class of admissible control laws

Linear-quadratic control problems are often solved by a completion-of-squares argument. In our present context we have

in which the first term takes the form

where  is the covariance matrix

The separation principle would now follow immediately if  were independent of the control. However this needs to be established.

The state equation can be integrated to take the form

where  is the state process obtained by setting  and  is the transition matrix function. By linearity,  equals

where . Consequently,

but we need to establish that  does not depend on the control. This would be the case if

where  is the output process obtained by setting . This issue was discussed in detail by Lindquist. In fact, since the control process  is in general a nonlinear function of the data and thus non-Gaussian, then so is the output process . To avoid these problems one might begin by uncoupling the feedback loop and determine an optimal control process in the class of stochastic processes  that are adapted to the family  of sigma fields. This problem, where one optimizes over the class of all control processes adapted to a fixed filtration,  is called a stochastic open loop (SOL) problem. It is not uncommon in the literature to assume from the outset that the control is adapted to ; see, e.g., Section 2.3 in Bensoussan, also van Handel  and Willems.

In Lindquist 1973 a procedure was proposed for how to embed the class of admissible controls in various SOL classes in a problem-dependent manner, and then construct the corresponding feedback law. The largest class  of admissible feedback laws  consists  of the non-anticipatory functions  such that the feedback equation  has a unique solution and the corresponding control process  is adapted to .
Next, we give a few examples of specific classes of feedback laws that belong to this general class, as well as some other strategies in the literature  to overcome the problems described above.

Linear control laws
The admissible class  of control laws could be restricted to contain only certain linear ones as in Davis. More generally, the linear class

where  is a deterministic function and  is an  kernel, ensures that  is independent of the control. In fact,  the Gaussian property will then be preserved, and  will be generated by the Kalman filter. Then  the error process  is generated by

which is clearly independent of the choice of control, and thus so is .

Lipschitz-continuous control laws
Wonham proved a separation theorem for controls in the class  , even for a more general cost functional than J(u). However, the proof is far from simple and there are many technical assumptions. For example,  must square and have a determinant bounded away from zero, which is a serious restriction. A later proof by Fleming and Rishel is considerably simpler. They also prove the separation theorem with quadratic cost functional  for a class of Lipschitz continuous  feedback laws, namely , where  is a non-anticipatory function of  which is Lipschitz continuous in this argument. Kushner proposed a more restricted class , where the modified state process  is given by

leading to the identity .

Imposing delay

If there is a delay in the processing of the observed data so that, for each ,  is a function of , then , , see Example 3 in Georgiou and Lindquist.  Consequently,  is independent of the control. Nevertheless, the control policy  must be such that the feedback equations have a unique solution.

Consequently, the problem with possibly control-dependent sigma fields  does not occur in the usual discrete-time formulation. However, a procedure used in several textbooks to construct  the continuous-time  as the limit of finite difference quotients of the discrete-time , which does not depend on the control, is circular or a best incomplete; see Remark 4 in Georgiou and Lindquist.

Weak solutions

An approach  introduced by Duncan and Varaiya and  Davis and Varaiya, see also Section 2.4 in Bensoussan
is based on weak solutions  of the  stochastic differential equation. Considering such solutions of

we can change the probability measure (that depends on ) via a Girsanov transformation so that

becomes a new Wiener process, which (under the new probability measure) can be assumed to be unaffected by the control. The question of how this could be implemented in an engineering system is left open.

Nonlinear filtering solutions

Although a nonlinear control law will produce a non-Gaussian state process, it can be shown, using  nonlinear filtering theory (Chapters 16.1 in Lipster and Shirayev
), that the state process is conditionally Gaussian given the filtration . This fact can be used to show that  is actually generated by a Kalman filter (see Chapters 11 and 12 in Lipster and Shirayev). However, this requires quite a sophisticated analysis and is restricted to the case where the driving noise  is a Wiener process.

Additional historical perspective can be found in Mitter.

Issues on feedback in linear stochastic systems

At this point it is suitable to consider a more general class of controlled linear stochastic systems that also covers systems with time delays, namely

with  a stochastic vector process which does not depend on the control. The standard stochastic system is then obtained as a special case where ,  and . We shall use the short-hand notation

for the feedback system, where

is a Volterra operator.

In this more general formulation the embedding procedure of Lindquist defines the class  of admissible feedback laws  as the class of non-anticipatory functions  such that the feedback equation  has a unique solution  and  is adapted to .

In Georgiou and Lindquist a new framework for the separation principle was proposed.  This approach considers stochastic systems as well-defined maps between sample paths rather than between stochastic processes  and allows us to extend the separation principle to systems driven by martingales with possible jumps. The approach is motivated by engineering thinking where systems and feedback loops process signals, and not stochastic processes per se or transformations of probability measures. Hence the purpose is to create a natural class of admissible control laws that make engineering sense, including those that are nonlinear and discontinuous.

The feedback equation  has a unique strong solution if there exists a non-anticipating function  such that  satisfies  the equation with probability one and all other solutions coincide with  with probability one. However, in the sample-wise setting, more is required, namely that such a unique solution exists and that  holds for all , not just almost all. The resulting feedback loop is deterministically well-posedin the sense that the feedback equations admit a unique solution that causally depends on the input for each input sample path.

In this context, a signal is defined to be a sample path of a stochastic process with possible discontinuities. More precisely, signals will belong to the Skorohod space , i.e., the space of functions which are continuous on the right and have a left limit at all points (càdlàg functions). In particular, the space  of continuous functions is a proper subspace of . Hence the response of a typical nonlinear operation that involves  thresholding and switching can be modeled as a signal.  The same goes for sample paths of counting processes and other martingales.  A system is defined to be a measurable non-anticipatory map  sending sample paths to sample paths so that their outputs at any time  is a measurable function of past values of the input and  time. For example, stochastic differential equations with Lipschitz coefficients driven by a Wiener process
induce maps between corresponding path spaces, see page 127 in Rogers and Williams, and pages 126-128 in Klebaner. Also, under fairly general conditions (see e.g., Chapter V in Protter), stochastic differential equations driven by martingales with sample paths in  have strong solutions who are semi-martingales.

For the time setting , the feedback system  can be written , where  can be interpreted as an input.

Definition. A feedback loop  is deterministically well-posed if it has a unique solution  for all inputs  and   is a system.

This implies that the processes  and  define identical filtrations. Consequently, no new information is created by the loop. However, what we need is that  for . This is ensured by the following lemma (Lemma 8 in Georgiou and Lindquist).

Key Lemma. If the feedback loop  is deterministically well-posed,  is a system, and  is a linear system having a right inverse  that is also a system, then   is a system and  for .

The condition on  in this lemma is clearly satisfied in the standard linear stochastic system, for which , and hence . The remaining conditions are collected in the following definition.

Definition. A feedback law  is deterministically well-posed for the system   if  is a system and the feedback system  deterministically well-posed.

Examples of simple systems that are not deterministically  well-posed are given in Remark 12 in Georgiou and Lindquist.

A separation principle for physically realizable control laws

By only considering feedback laws that are deterministically well-posed, all admissible control laws are physically realizable in the engineering sense  that they induce a signal that travels through the feedback loop.
The proof of the following theorem can be found in Georgiou and Lindquist 2013.

Separation theorem.
Given the linear stochastic system
 
where  is a vector-valued Wiener process,  is a zero-mean Gaussian random vector independent of , consider the problem of minimizing the quadratic functional J(u) over the class of all deterministically well-posed feedback laws . Then the unique optimal control law is given by   where  is defined as above and  is given by the Kalman filter. More generally, if  is a square-integrable martingale and   is an arbitrary zero mean random vector, , where , is the optimal control law provided it is deterministically well-posed.

In the general non-Gaussian case, which may involve counting processes, the Kalman filter needs to be replaced by a nonlinear filter.

A Separation principle for delay-differential systems

Stochastic control for time-delay systems were first studied in Lindquist,
and Brooks, although Brooks relies on the strong assumption that the observation  is functionally independent of the control , thus avoiding the key question of feedback.

Consider the delay-differential system

where  is now a (square-integrable) Gaussian (vector) martingale, and where  and  are of bounded variation in the first argument and continuous on the right in the second,  is deterministic for , and .
More precisely,  for ,  for , and the total variation of  is bounded by an integrable function in the variable , and the same holds for .

We want to determine a control law which minimizes

where  is a positive Stieltjes measure. The corresponding deterministic problem obtained by setting  is given by

with .

The following separation principle for the delay system above can be found in Georgiou and Lindquist 2013 and generalizes the corresponding result in Lindquist 1973

Theorem. There is a unique feedback law   in the class of deterministically well-posed control laws that minimizes , and it is given by

where  is the deterministic control gain and  is given by the linear (distributed) filter

where  is the innovation process

and the gain  is as defined in page 120 in Lindquist.

References

Control theory
Stochastic control