## 2 Problem Formulation  
### Problem with Infinite Time Horizon  
In this section, we study the infinite horizon discrete-time stochastic LQ optimal control problem, in which the conic control constraints are also considered; see [5; 14]. For simplicity of the parameters, we consider a one dimensional dynamic system with a multiplicative noise described by  
\[x_{l+1}=Ax_{l}+\mathbf{B}\mathbf{u}_{l}(x_{l})+[Ax_{l}+\mathbf{B}\mathbf{u}_{ l}(x_{l})]\xi_{l}, \tag{1}\]for time \(l=0,1,\cdots\). By denoting \(\mathbb{R}\)\((\mathbb{R}_{+})\) as the set of real (nonnegative real) numbers, in this system, \(A\in\mathbb{R}\) and \(\mathbf{B}\in\mathbb{R}^{1\times m}\) are deterministic values; \(x_{l}\in\mathbb{R}\) is the state with \(x_{0}\) being given; and \(\mathbf{u}_{l}\in\mathbb{R}^{m}\) is a feedback control law at time \(l\). For each \(l\), \(\xi_{l}\) denotes an independent identically distributed one-dimensional multiplicative noise, satisfying a normal distribution with mean \(0\) and variance \(\sigma^{2},\sigma\geq 0\). For each \(l\), \(\xi_{l}\) denotes a one-dimensional noise. \(\xi_{l}\) and \(\xi_{k}\) are independent for every \(l\
eq k\).  
Now, we consider the conic control constraint sets (cf. [5])  
\[\mathcal{C}_{l}:=\{\mathbf{u}_{l}|\mathbf{u}_{l}\in\mathcal{F}_{l},\ \mathbf{H} \mathbf{u}_{l}\in\mathbb{R}_{+}^{n}\}, \tag{2}\]  
for \(l=0,1,\cdots\), where \(\mathbf{H}\in\mathbb{R}^{n\times m}\  
Let \(\mathbb{B}\) be the \(\sigma\)-field of \(\mathbb{R}\) containing all the (Lebesgue) measurable sets. For any set \(B\in\mathbb{B}\), we can define the identity transition function \(I(B|x)\). \(I(B|x)=1\) if \(x\in B\); \(I(B|x)=0\) otherwise. For any function \(h\) and \(x\in\mathbb{R}\), we have \((Ih)(x)=h(x)\).  
The product of two transition functions \(P_{1}(B|x)\) and \(P_{2}(B|x)\) is defined as a transition function \((P_{1}P_{2})(B|x)\):  
\[(P_{1}P_{2})(B|x):=\int_{\mathbb{R}}P_{2}(B|y)P_{1}(dy|x),\]  
where \(x,y\in\mathbb{R},B\in\mathbb{B}\).  
For any transition function \(P\), we can define the \(k\)th power, \(k=0,1,\cdots\), as \(P^{0}=I,P^{1}=P\), and \(P^{k}=PP^{k-1},k=2,\cdots\). Suppose that the Markov chain \(\mathbf{X}\) is time-homogeneous with transition function \(P(B|x),x\in\mathbb{R},B\in\mathbb{B}\). Then the \(k\)-step transition probability functions, denoted as \(P^{(k)}(B|x),k=1,2,\cdots\), are given by the 1-step transition function defined as \(P^{(1)}(B|x)=P(B|x)\) and  
\[P^{(k)}(B|x):=\int_{\mathbb{R}}P(dy|x)P^{k-1}(B|y),\ k\geq 2.\]  
For any function \(h(x)\), we have  
\[(P^{(k)}h)(x)=\int_{\mathbb{R}}h(y)P^{(k)}(dy|x)=P(P^{(k-1)})h(x).\]  
That is, as an operator, we have \(P^{(k)}=P(P^{(k-1)})\). Recursively, we can prove that \(P^{(k)}=P^{k}\).  
Suppose that a Markov chain \(\mathbf{X}\) with a continuous state space on \(\mathbb{R}\) has a steady-state distribution \(\pi\) satisfying \(\pi=\pi P\). Define function \(e(x)=1\) for all \(x\in\mathbb{R}\). We denote the performance potential \(g\) as a function which satisfies the Poisson equation (cf. [17])  
\[(I-P)g(x)+\eta(x)=f(x), \tag{7}\]  
where \(I\) and \(P\) are two transition functions, and \(\eta(x)=(\pi f)e(x)=\eta e(x)\). Then if \(g\) is a solution to (7), so is \(g+ce\), with any constant \(c\). We define  
\[g_{K}:=\{I+\sum_{k=1}^{K}(P^{k}-e\pi)\}f, \tag{8}\]  
and assume the limit \(g(x):=\lim_{K\to\infty}g_{K}(x)\) exists for \(x\in\mathbb{R}\). Then we have the following lemma,  
**Lemma 1** (Solution to Poisson Equations [17]).: _For any transition function \(P\) and performance function \(f(x)\), if_  
\[\lim_{k\to\infty}P^{k}f=(e\pi)f=\eta e,\] \[\lim_{K\to\infty}g_{K}=g,\text{ and }\lim_{K\to\infty}Pg_{K}=Pg,\]  
_hold for every \(x\in\mathbb{R}\), then_  
\[g=\{I+\sum_{k=1}^{\infty}(P^{k}-e\pi)\}f, \tag{9}\]  
_is a solution to the Poisson Equation (7)._