# General Recurrent Neural Network for Solving Generalized Linear Matrix Equation

Authors: Zhan Li; Hong Cheng; Hongliang Guo
Journal: Complexity
 (2017)

---

## Abstract

This brief proposes a general framework of the nonlinear recurrent neural network for solving online the generalized linear matrix equation (GLME) with global convergence property. If the linear activation function is utilized, the neural state matrix of the nonlinear recurrent neural network can globally and exponentially converge to the unique theoretical solution of GLME. Additionally, as compared with the case of using the linear activation function, two specific types of nonlinear activation functions are proposed for the general nonlinear recurrent neural network model to achieve superior convergence. Illustrative examples are shown to demonstrate the efficacy of the general nonlinear recurrent neural network model and its superior convergence when activated by the aforementioned nonlinear activation functions.


---

## Body

## 1. Introduction

Solving the generalized linear matrix equation (GLME) and its variants is an important issue which is widely encountered in scientific and engineering areas (e.g., feedback control system design [1], smart antenna array processing [2]). The well-known Lyapunov equation and Sylvester equation can be regarded as main special cases of GLME with reduced numbers of coefficients and variable matrices, which have drawn widespread interest of researchers and engineers in the past decades [3–7]. Without loss of generality, in this brief, the GLME problem is formulated as the following form: (1)∑l=1pAlXBl=C,where Al∈Rm×m, Bl∈Rn×n, and C∈Rm×n denote coefficient matrices and X∈Rm×n denotes the unknown matrix to be obtained. Usually, it is complicated to analyze what circumstances the solution of (1) would be under in the traditional numerical way. To guarantee such GLME (1) solvable with the unique theoretical solution, coefficient matrices Al∈Rm×m and Bl∈Rn×n can be practically configured with their eigenvalues all being positive or negative simultaneously. In many cases, the number of solutions of (1) can be multiple or even none, depending on what kind of combinations matrices Al and Bl would make to associate with the unknown matrix X. A lot of conventional serial approaches may be not efficient enough to solve online GLME due to their inherent drawbacks, and parallel computational approaches seem more preferable [8–13].Regarded as another promising approach for parallel computation, dynamic neural networks based on analog solvers have been exploited comprehensively in computational intelligence fields [12, 14–16]. Different from a number of conventional numerical methods, approaches based on dynamic neural networks can be more realizable on specific parallel and distributed software or/and hardware architectures [17, 18]. This could highly enlarge utility of current neural networks towards various potential application domains towards high-performance computing. One basic type of dynamic neural networks, recurrent neural networks, which is analogous to the natural transient and steady process, has been applied in online parallel computing tasks with large-scale analog/digital circuit prototypes [19].Our main contribution in this brief is to develop a general framework of recurrent neural network model to solve GLME (1). Since nonlinear phenomena occur frequently in neural network hardware implementation [19], the proposed general nonlinear framework may be more suitable for analog-based computation. The neural state of the general recurrent neural network can globally converge to the theoretical solutions. If the general recurrent neural network is activated by the linear function, the exponential convergence can be achieved. On the other hand, certain nonlinear forms of such general neural network may be able to obtain more accurate solutions and faster convergence as compared with its linear model, so we propose two specific nonlinear activation functions for the general recurrent neural network model to achieve superior performance to solve GLME (1).


## 2. General Recurrent Neural Network Solver

In this section, we present and analyze the general model of recurrent neural network to solve GLME (1). If such model is activated by the linear function, state matrix X(t)∈Rm×n of the general recurrent neural network can globally and exponentially converge to the unique theoretical solution X∗∈Rm×n. By exploiting specific nonlinear odd monotonically increasing activation functions, superior convergence is expected to be achieved. In the ensuing subsections, we will discuss their convergence properties of the general nonlinear recurrent neural network model together with its linear form.


### 2.1. General Nonlinear Neural Network Model

In this brief, the general nonlinear recurrent neural network model is proposed to solve GLME (1) as follows: (2)X˙t=-γ∑r=1pArTF∑s=1pAsXtBs-CBrT,where operator F(·):Rm×n→Rm×n denotes a nonlinear activation function array, with its each scalar-valued mapping unit f(·):R→R being a monotonically increasing odd activation function, and subscript (·)T denotes transpose of matrix/vector. Such recurrent neural network model (2) can be generalized as the extended nonlinear version of recurrent neural network in [16] and the ensuing linear model. For the general nonlinear recurrent neural network model (2), we would have the following theorem.Theorem 1.
The neural state matrixX(t)∈Rm×n of general nonlinear recurrent neural network model (2), starting from any initial value X(0)∈Rm×n, can globally converge to the theoretical solution(s) X∗∈Rm×n of GLME (1).Proof.
Firstly, we define the distance between the neural state and the theoretical solution asX~(t)=X(t)-X∗∈Rm×n. Accordingly, by substituting X(t)=X~(t)+X∗∈Rm×n into neural network model (2), it can be further equivalently transformed as (3)X~˙t=-γ∑r=1pArTF∑s=1pAsX~tBsBrT.Next, the corresponding Lyapunov-function candidate is defined as follows: (4)VX~t,t3=X~tF2=vecX~t22,where operators ·F, ·2, and ⊗, respectively, denote Frobenius norm of matrix, two norms of vector, and Kronecker product between matrices and vec(X~(t))∈Rmn generates a new column vector obtained by stacking all column vectors of X~(t)∈Rm×n together.
The time derivative ofV(X~(t),t)(3) is (5)V˙X~t,t3=2vecX~tTvecX~˙t.
Considering the following vectorization equality based on (2), (6)vecX~˙t=-γ∑r=1pBr⊗ArTF∑s=1pBsT⊗AsvecX~t,we can further derive (5) as (7)V˙X~t,t3=-2γvecX~tT∑r=1pBr⊗ArT·F∑s=1pBsT⊗AsvecX~t=-2γ∑l=1pBlT⊗AlvecX~tT·F∑l=1pBlT⊗AlvecX~t.
For nonlinear activation function arrayF(·), its individual scalar-valued entry f(·) is odd and monotonically increasing, which can guarantee (8)ufu=>0,u∈R,u≠0;=0,u∈R,u=0.Thus, V˙(X~(t),t)(3)⩽0, which implies that X~(t)∈Rm×n could globally converge to zero matrix 0∈Rm×n according to Lyapunov theory [20]; that is, state matrix X(t)∈Rm×n of (2) globally converges to the theoretical solution(s) X∗∈Rm×n of GLME (1). All of these above complete the proof.According to Theorem1, the general nonlinear neural network model (2) can be activated by a number of odd monotonically increasing functions to solve GLME (1) which is with existent theoretical solutions (unique or multiple), which will broadly enlarge the utility domain of (2) towards manifold model generation. As we may know, nonlinear elements are frequently encountered in analog/digital circuit prototypes of neural networks [19, 21]; involving nonlinear activation functions can be beneficial to potential design and implication. On the other hand, faster convergence is indeed required for solving GLME (1) when the linear model might not satisfy increasing computational requirements. With expectancy, the nonlinear neural network model (2) can attain superior convergence to that of (10) if proper activation functions are exploited. Before inducing the superior nonlinear-function activated models, we herein address the linear model of the general nonlinear recurrent neural network and discuss its convergence property.


### 2.2. Linear Neural Network Model

To solve GLME (1), we firstly define a scalar-valued error function E(t)=∑l=1p‍AlX(t)Bl-CF2/2∈[0,+∞) associated with (1), where operator ·F denotes the Frobenius norm. In order to eliminate error function E(t) to zero as t increases, gradient-descent manner is adopted: (9)X˙t=-γ∂Et∂Xt,where design parameter γ>0 scales the convergence rate.According to preliminaries on matrix-differential theory [22], (9) is further expanded to the following dynamic form: (10)X˙t=-γ∑s=1p∑r=1pArTAsXtBsBrT+γ∑l=1pAlTCBlT.For linear model (10), we would have the following theorem.Theorem 2.
If the linear neural network model (10) is employed to solve GLME (1), starting from initial condition X(0)∈Rm×n, the state matrix X(t)∈Rm×n of (10) can globally exponentially converge to unique theoretical solution X∗∈Rm×n.Proof.
Using the transformationX~(t)=X(t)-X∗∈Rm×n between X(t) and X∗ with initial condition X~(0)=X(0)-X∗∈Rm×n, dynamic equation (10) is further derived equivalently as the following: (11)X~˙t=-γ∑s=1p∑r=1pArTAsX~tBsBrT+γ∑l=1pAlTCBlT-γ∑s=1p∑r=1pArTAsX∗BsBrT=-γ∑s=1p∑r=1pArTAsX~tBsBrT+γ∑r=1pArTC-∑s=1pAsX∗BsBrT.
With∑l=1p‍AlX∗Bl=C considered, (11) can be simplified as (12)X~˙t=-γ∑s=1p∑r=1pArTAsX~tBsBrT.
Similarly, we define the following Lyapunov-function candidate:(13)VX~t,t10=X~tF2⩾0,with its time derivative being (14)V˙X~t,t10=trace∂VX~t,t∂X~tTX~˙t=trace-2γX~Tt∑s=1p∑r=1pArTAsX~tBsBrT=-2γ∑l=1pAlX~tBlF2⩽0.There exists a positive scalar α>0 [23] being the minimum eigenvalue of (∑l=1p‍BlT⊗Al)T(∑l=1p‍BlT⊗Al) satisfying (15)∑l=1pAlX~tBlF2⩾αX~tF2if the unique solution condition of GLME (1) holds. Thus, we can have(16)V˙X~t,t⩽-2γαVX~t,t;that is, (17)VX~t,t⩽exp⁡-2γαtVX~0,0,which could be further equivalently rewritten as (18)Xt-X∗F⩽exp⁡-γαtX0-X∗F.
By Lyapunov theory [20], (14) and (18) indicate that state matrix X(t)∈Rm×n of (10) can globally and exponentially converge to the unique theoretical solution X∗∈Rm×n of GLME (1). The proof is thus complete.It is worth noting that if GLME (1) is with multiple theoretical solutions X∗∈Rm×n, scalar α equals zero. In this situation, the linear model (10) at least could guarantee its global convergence but is not able with explicit exponential convergence rate.


## 2.1. General Nonlinear Neural Network Model

In this brief, the general nonlinear recurrent neural network model is proposed to solve GLME (1) as follows: (2)X˙t=-γ∑r=1pArTF∑s=1pAsXtBs-CBrT,where operator F(·):Rm×n→Rm×n denotes a nonlinear activation function array, with its each scalar-valued mapping unit f(·):R→R being a monotonically increasing odd activation function, and subscript (·)T denotes transpose of matrix/vector. Such recurrent neural network model (2) can be generalized as the extended nonlinear version of recurrent neural network in [16] and the ensuing linear model. For the general nonlinear recurrent neural network model (2), we would have the following theorem.Theorem 1.
The neural state matrixX(t)∈Rm×n of general nonlinear recurrent neural network model (2), starting from any initial value X(0)∈Rm×n, can globally converge to the theoretical solution(s) X∗∈Rm×n of GLME (1).Proof.
Firstly, we define the distance between the neural state and the theoretical solution asX~(t)=X(t)-X∗∈Rm×n. Accordingly, by substituting X(t)=X~(t)+X∗∈Rm×n into neural network model (2), it can be further equivalently transformed as (3)X~˙t=-γ∑r=1pArTF∑s=1pAsX~tBsBrT.Next, the corresponding Lyapunov-function candidate is defined as follows: (4)VX~t,t3=X~tF2=vecX~t22,where operators ·F, ·2, and ⊗, respectively, denote Frobenius norm of matrix, two norms of vector, and Kronecker product between matrices and vec(X~(t))∈Rmn generates a new column vector obtained by stacking all column vectors of X~(t)∈Rm×n together.
The time derivative ofV(X~(t),t)(3) is (5)V˙X~t,t3=2vecX~tTvecX~˙t.
Considering the following vectorization equality based on (2), (6)vecX~˙t=-γ∑r=1pBr⊗ArTF∑s=1pBsT⊗AsvecX~t,we can further derive (5) as (7)V˙X~t,t3=-2γvecX~tT∑r=1pBr⊗ArT·F∑s=1pBsT⊗AsvecX~t=-2γ∑l=1pBlT⊗AlvecX~tT·F∑l=1pBlT⊗AlvecX~t.
For nonlinear activation function arrayF(·), its individual scalar-valued entry f(·) is odd and monotonically increasing, which can guarantee (8)ufu=>0,u∈R,u≠0;=0,u∈R,u=0.Thus, V˙(X~(t),t)(3)⩽0, which implies that X~(t)∈Rm×n could globally converge to zero matrix 0∈Rm×n according to Lyapunov theory [20]; that is, state matrix X(t)∈Rm×n of (2) globally converges to the theoretical solution(s) X∗∈Rm×n of GLME (1). All of these above complete the proof.According to Theorem1, the general nonlinear neural network model (2) can be activated by a number of odd monotonically increasing functions to solve GLME (1) which is with existent theoretical solutions (unique or multiple), which will broadly enlarge the utility domain of (2) towards manifold model generation. As we may know, nonlinear elements are frequently encountered in analog/digital circuit prototypes of neural networks [19, 21]; involving nonlinear activation functions can be beneficial to potential design and implication. On the other hand, faster convergence is indeed required for solving GLME (1) when the linear model might not satisfy increasing computational requirements. With expectancy, the nonlinear neural network model (2) can attain superior convergence to that of (10) if proper activation functions are exploited. Before inducing the superior nonlinear-function activated models, we herein address the linear model of the general nonlinear recurrent neural network and discuss its convergence property.


## 2.2. Linear Neural Network Model

To solve GLME (1), we firstly define a scalar-valued error function E(t)=∑l=1p‍AlX(t)Bl-CF2/2∈[0,+∞) associated with (1), where operator ·F denotes the Frobenius norm. In order to eliminate error function E(t) to zero as t increases, gradient-descent manner is adopted: (9)X˙t=-γ∂Et∂Xt,where design parameter γ>0 scales the convergence rate.According to preliminaries on matrix-differential theory [22], (9) is further expanded to the following dynamic form: (10)X˙t=-γ∑s=1p∑r=1pArTAsXtBsBrT+γ∑l=1pAlTCBlT.For linear model (10), we would have the following theorem.Theorem 2.
If the linear neural network model (10) is employed to solve GLME (1), starting from initial condition X(0)∈Rm×n, the state matrix X(t)∈Rm×n of (10) can globally exponentially converge to unique theoretical solution X∗∈Rm×n.Proof.
Using the transformationX~(t)=X(t)-X∗∈Rm×n between X(t) and X∗ with initial condition X~(0)=X(0)-X∗∈Rm×n, dynamic equation (10) is further derived equivalently as the following: (11)X~˙t=-γ∑s=1p∑r=1pArTAsX~tBsBrT+γ∑l=1pAlTCBlT-γ∑s=1p∑r=1pArTAsX∗BsBrT=-γ∑s=1p∑r=1pArTAsX~tBsBrT+γ∑r=1pArTC-∑s=1pAsX∗BsBrT.
With∑l=1p‍AlX∗Bl=C considered, (11) can be simplified as (12)X~˙t=-γ∑s=1p∑r=1pArTAsX~tBsBrT.
Similarly, we define the following Lyapunov-function candidate:(13)VX~t,t10=X~tF2⩾0,with its time derivative being (14)V˙X~t,t10=trace∂VX~t,t∂X~tTX~˙t=trace-2γX~Tt∑s=1p∑r=1pArTAsX~tBsBrT=-2γ∑l=1pAlX~tBlF2⩽0.There exists a positive scalar α>0 [23] being the minimum eigenvalue of (∑l=1p‍BlT⊗Al)T(∑l=1p‍BlT⊗Al) satisfying (15)∑l=1pAlX~tBlF2⩾αX~tF2if the unique solution condition of GLME (1) holds. Thus, we can have(16)V˙X~t,t⩽-2γαVX~t,t;that is, (17)VX~t,t⩽exp⁡-2γαtVX~0,0,which could be further equivalently rewritten as (18)Xt-X∗F⩽exp⁡-γαtX0-X∗F.
By Lyapunov theory [20], (14) and (18) indicate that state matrix X(t)∈Rm×n of (10) can globally and exponentially converge to the unique theoretical solution X∗∈Rm×n of GLME (1). The proof is thus complete.It is worth noting that if GLME (1) is with multiple theoretical solutions X∗∈Rm×n, scalar α equals zero. In this situation, the linear model (10) at least could guarantee its global convergence but is not able with explicit exponential convergence rate.


## 3. Superior Convergence with Specific Nonlinear Activation Functions

According to Theorem1, The odd monotonically increasing activation function is able to guarantee global convergence of the general recurrent neural network (2). If the linear activation function is adopted, the general recurrent neural network model reduces to the linear model (10). Such linear model (10) possesses global exponential convergence property. In order to achieve superior convergence to global exponential convergence of the linear model (10), specific types of nonlinear activation functions should be chosen properly. Owing to the aforementioned considerations, two types of nonlinear activation functions, power sum and hyperbolic sine functions, are proposed to activate the general recurrent neural network model (2). Figure 1 shows the curve plotting of the three aforementioned activation functions used in (2). Correspondingly, we will have the following theorems on the two neural network models’ convergence properties.Figure 1
The general recurrent neural network model (2) utilizes the three types of activation functions: linear, power sum (N=3), and hyperbolic sine (ξ=3).Theorem 3.
If the general recurrent neural network (2) is activated by the power sum function f(u)=∑k=1N‍u2k-1, the state matrix X(t)∈Rm×n of (2) can globally and superiorly converge to the unique theoretical solution X∗∈Rm×n, as compared with linear model (10).Proof.
In order to prove the convergence property of (2) activated by the power sum function f(u)=∑k=1N‍u2k-1 under this situation, we define the following Lyapunov-function candidate: (19)VpsX~t,t≔VX~t,t3=VX~t,t10,with its time derivative being (20)V˙psX~t,t=-2γΔTFΔ=-2γ∑i=1mnΔifΔi=-2γ∑i=1mnΔi∑k=1NΔi2k-1=-2γ∑i=1mn∑k=1NΔi2k⩽-2γ∑i=1mnΔi2=V˙X~t,t10,where (21)Δ=Δ1,Δ2,…,Δi,…,ΔmnT∈Rmnand Δi∈R denotes the ith element of vector (∑l=1p‍BlT⊗Al)vec(X~(t))∈Rmn. This implies that when power sum functions are used, (2) possesses global convergence to zero matrix, with larger Lyapunov-function vanishing rate (i.e., faster convergence), as compared with the situation of (10). The proof is thus complete.Theorem 4.
If the general recurrent neural network (2) is activated by hyperbolic sine function f(u)=exp⁡(ξu)/2-exp⁡(-ξu)/2 with coefficient ξ⩾1, the state matrix X(t)∈Rm×n of (2) can globally and superiorly converge to the unique theoretical solution X∗∈Rm×n, as compared with the linear model (10).Proof.
Similarly, the following Lyapunov function is defined to investigate convergence:(22)VhsX~t,t≔VpsX~t,t=VX~t,t10,and its time derivative is (23)V˙hsX~t,t=-2γ∑i=1mnΔifΔi=-γ∑i=1mnΔiexp⁡ξΔi-exp⁡-ξΔi=-2γ∑i=1mn∑j=1+∞ξΔi2j2j-1!⩽-2γ∑i=1mnΔi2=V˙X~t,t10,which indicates that when the hyperbolic sine activation function is employed, the nonlinear recurrent neural network model (2) possesses global convergence as its state matrix is approaching zero, with larger Lyapunov-function vanishing rate (i.e., faster convergence) as compared with the situation of linear model (10). These complete the proof.


## 4. Illustrative Examples

In this section, three examples are presented to illustrate the efficiency of the general nonlinear recurrent neural network (2) with its specific models under different types of activation functions (linear, power sum, and hyperbolic sine activation functions) for online solving GLME (1).Example 1.
Let us consider the following GLME withl=2: (24)A1XB1+A2XB2=C,where (25)A1=2113,A2=6234,B1=5334,B2=6258,C=3456.
GLME (24) has unique theoretical solution (26)X∗=-0.00230.03080.03950.1019,since the eigenvalues of coefficient matrices A1, A2, B1, and B2 are all positive values. We employ the general recurrent neural network model (2) with γ=1 activated by linear function, power sum function with N=4, and hyperbolic sine function with ξ=3.From Figure2, we could observe that the solution errors X(t)-X∗F decline to almost zero at around 0.02 s and faster convergence to the solution can be achieved with power sum and hyperbolic sine activation functions used in (2). These can demonstrate the effectiveness of the general recurrent neural network model (2) for solving GLME (24).Figure 2
Solution errorX(t)-X∗F synthesized by (2) for GLME (24) with the unique theoretical solution X∗, starting from the same randomly generated initial state X(0)∈R2×2.Example 2.
Let us consider the following GLME with multiple theoretical solutionsX∗∈R2×2: (27)A~1XB~1+A~2XB~2=C~,where (28)A~1=10-11,A~2=2001,B~1=0001,B~2=0100,C~=0305.We use linear model (10) with design parameter γ=1 to solve GLME (27). The trajectories of entries of state matrix X(t)∈R2×2 are shown in Figure 3. From Figure 3, we could see that, starting from two different initial matrices X(0)∈R2×2, the state matrices X(t)∈R2×2 of linear model (10), respectively, converge to two different trajectories (or say two different theoretical solutions X∗∈R2×2). This indicates that the choices of the initial value impact greatly the steady-state results of the recurrent neural network (2) and determine the starting points of convergence for solution of GLME (27), if multiple theoretical solutions exist for GLME (27). Correspondingly, the residual errors A~1X(t)B~1+A~2X(t)B~2-C~F synthesized by (10) can always diminish to zero within finite time from twenty different initial values, as illustrated by Figure 4.Figure 3
Trajectories of state matrixX(t)∈R2×2 for solving GLME (27) with the multiple theoretical solutions X∗, starting from two different initial states X(0)∈R2×2.Figure 4
Residual errorA~1X(t)B~1+A~2X(t)B~2-C~F synthesized by (2) for solution of GLME (27), starting from twenty different initial states X(0)∈R2×2.Example 3.
Let us consider the following GLME in a larger dimension withl=10: (29)∑l=110AlXBl=C,where coefficient matrices Al∈R10×10, Bl∈R10×10, and C∈R10×10 are all positive-definite randomly generated and fall within interval [-2,2]∈R10×10. We exploit nonlinear neural network models (2) activated by power sum and hyperbolic sine functions and the linear model (10) to solve GLME (29) with design parameter γ=1. From Table 1, we could observe that general recurrent neural network models (2) activated by power sum and hyperbolic sine activation functions exhibit faster error diminishing speed than that of the linear model (10), with all of their residual errors reaching the level of 10-7 within 1 s. From computational results of these three examples above, we could see that the proposed general nonlinear recurrent neural network can solve the GLME (1) problem well.Table 1
Performance of the general recurrent neural network model (2) with three different activation functions (linear, power sum, and hyperbolic sine) for solving GLME (29).
Residual error∑l=110‍AlX(t)Bl-CF
Timet (s)
Linear
Power sum
Hyperbolic sine
t
=
0s
37.4543
37.4543
37.4543
t
=
0.1s
0.9886
0.3678
0.5242
t
=
1s
4.6368
×
1
0
-
7
9.7531
×
1
0
-
8
3.0543
×
1
0
-
7


## 5. Conclusion

In this brief, we present a general recurrent neural network model for solving GLME. The general nonlinear model of recurrent neural network possesses global convergence property in finding solutions of GLME. By using specifically proposed nonlinear activation functions, superior convergence can be achieved, as compared with the linear model which is with exponential convergence rate. Illustrative results are shown to demonstrate the effectiveness and superiority of nonlinear recurrent neural network models for solution of GLME.



---

*Source: 9063762-2017-07-31.xml*