Dynamic programming approach to a final-value control system with a random variable having an unknown distribution function

As described in the introduction of this paper, the state vector x n , representing the present state of a sampling control system, is assumed to satisfy a difference equation x_{n+1} = T(x_{n}, r_{n}, \upsilon_{n}) , where \upsilon_{n} is the control vector of the system subject to random disturbances r n . The random variables r n are assumed to be independent of each other and to be defined in a parameter space in the following manner: Nature is assumed to be in one of a finite number, q , of possible states and each state has its own parameter value, thus specifying uniquely and unequivocally the distribution function of r n . It is further assumed that we are given the a priori probability z =(z_{1}, . . . , z_{q}) of each possible state of nature, i= 1,2, . . . , q,\sum\min{i=1}\max{q} z_{i}=1, z_{i} \geq 0 . Given a criterion of performance, the duration of the process and the domain of the control variable, a sequence of control variables { \upsilon_{n} } is to be determined as a function of the state vector of the system and time, so as to optimize the performance. The sequential nature of the determination of \upsilon_{n} is evident here, since the stochastic nature of the problem prevents specification of such a sequence of \upsilon 's as a function of the initial state and time. By means ot the functional equation technique of dynamic programming, a recurrence relation of the criterion function of the process, k_{n}(x, z) , is derived, where x is the state variable of the system when there remain n control stages. When q is taken to be two and the criterion of performance to he x_{N}^{2} with the constraint on the control variables \sum\min{i=0}\max{N-1} \upsilon_{i}^{2} \leq K , where x n is the final state vector of the system, it is shown that k_{n}(x, z) is the form w_{n}(z)x^{2} and the optimal \upsilon_{n}(x, z) is linear in x , as might be expected. The dependence of k_{n}(x, z) and \upsilon_{n}(x, z) on z is investigated further, and w_{n}(z) is found to he concave in z . Explicit quadratic forms for w_{n}(z) are obtained in the neighborhood of z = 0 and 1. The optimal \upsilon_{n}(x, z) is found to be monotonically decreasing in z . When the domain of \upsilon_{n} is restricted to a finite set of values, as in contactor servo systems, no explicit expressions for k_{n}(x, z) and \upsilon_{n}(x, z) are available. However, k_{n}(x, z) is still concave in z and approximately given by zk_{n}(x, 1) + (1 - z)k_{n}(x, 0) . By solving the recurrence relation numerically, this approximation is found to be very good for moderately large n, say 10. This means that if one has explicit solutions for p =p_{1} and p = P_{2} , then one can approximate the criterion function for the adaptive case where Pr(p =p_{1})= z by a linear function in z as shown above. This provides fairly good lower bound on k_{n}(x, z) , and in turn serves to determine an initial approximate policy. A concept of suboptimal policy is introduced and numerical experiments are performed to test suboptimal policy suggested from numerical solution. The system behavior under the suboptimal policy is also discussed.