# Clustering Input Signals Based Identification Algorithms for Two-Input Single-Output Models with Autoregressive Moving Average Noises

Authors: Khalid Abd El Mageed Hag ElAmin
Journal: Complexity
 (2020)

---

## Abstract

This study focused on the identification problems of two-input single-output system with moving average noises based on unsupervised learning methods applied to the input signals. The input signal to the autoregressive moving average model is proposed to be arriving from a source with continuous technical and environmental changes as two separate featured input signals. These two input signals were grouped in a number of clusters using theK-means clustering algorithm. The clustered input signals were supplied to the model in an orderly fashion from cluster-1 up to cluster-K. To ensure that the output signal can be best predicted from the input signal which in turn leads to selecting good enough model for its intended use, the magnitude-squared coherence (MSC) measure is applied to the input/output signals in the cases of clustered and nonclustered inputs, which indicates best correlation coefficient when measured with clustered inputs. From collected input-output signals, we deduce a K-means clustering based recursive least squares method for estimating the parameter of autoregressive moving average system. The simulation results indicate that the suggested method is effective.


---

## Body

## 1. Introduction

Most systems that have been inferred to prove many of the assumptions proposed in the area of system identification focused on single-input single-output systems [1–4]. Recently, the focus has been on studying system identification of multivariable systems in order to deal with an appropriate modelling and estimation of dynamic systems operating in industrial applications and process control [5–7].Several methods and techniques addressed system identification problems in multivariable models as in [8–12]; the researchers in these studies depend on hierarchical identification principle that decomposes a multi-input system into two subsystems, one containing a parameter vector and the other containing a parameter matrix [10]. The proposed method in [13] was dependent on the concept of coupling identification to avoid matrix inversion in multivariable least squares in order to reduce computational time in the algorithms. Many algorithms are used to implement these methods such as least squares recursive algorithms or iterative least squares algorithms without mentioning the impacts of the excitation signals on the proposed models.There are identification methods focusing on the modified effects of the excitation signals on the design of system identification experiments and they yielded acceptable results in modelling and estimation process of multivariable system. These proposed methods are based mainly on applying filtering techniques to the collected input-output data in order to generate more accurate results as in [14–17]. We aim at the same objectives of these researches by considering effective preprocessing on the excitation signals of multivariable systems.In this paper, we tried to focus on the collected input data to identify multiple-input single-output systems by introducing unsupervised learning approach such as the clustering techniques applied only to input signals by means ofK-means clustering algorithm. To ensure that the system output is best predicted from the proposed excitation signals, the similarities in the frequency contents between modified input signals and generated output signal are investigated using the magnitude-squared coherence measure [18].Most of the works that relied on the use of clustering in the system identification process were based on collected and recorded data from input and output without even assuming any slightly modified effect on the input signals. In [19], the submodels of the PieceWise AutoRegressive systems with eXogenous input (PWARX) are obtained through an algorithm inspired by competitive learning. In particular, the proposed method exploits a process of fuzzy clustering to obtain a subset of representatives from the original data set. Reference [20] assumed the number of modes of the PWARX system to be unknown and proposed a split-and-merge clustering algorithm to estimate the correct number of modes. Reference [21] discusses the use of correlation clustering algorithms for robust identification of PWARX models with reduced complexity. A PWARX model of the electronic throttle regulating the air inflow of a car engine has been identified based on clustering in [22]. The refinement algorithm proposed in [23] is repeatedly applied to the estimated clusters to improve both the data classification and the parameter estimation. Since, in complex engineering system, reliability is considered as the main issue, K-means clustering is used in [24] to estimate the reliability of the complex engineering system to obtain three or more subsystems called cluster systems and analyze the key performance parameters of constructs, which are known as family system that contains only subsystems with similar performance; then the contribution of these subsystems in the cluster system leads to estimating reliability of the whole system. Many of the above-mentioned research works were based on clustering the regressor vector to divide the system model into multiple candidate models for investigation and analysis. We pointed out here that the regressor vector contains both input and output signals.The fundamental idea of the proposed method in this paper is to cluster only the input signals usingK-means clustering algorithm without clustering the whole regressor vector. Based on this idea, we present a clustered input signals based recursive least squared algorithm. Given this recursive algorithm, we can produce more accurate parameter estimation compared to existing multivariable recursive estimation methods, for example, [14, 15]. Compared with the clustering techniques used in a PieceWise affine ARX model [19–21], the modified recursive algorithm in this paper can produce highly accurate estimation.Furthermore, since the clustered input signals are assumed to be similar due to the similarity measure of theK-means algorithm, the same system properties will be excited and hence better identification performance will be obtained [25].The rest of this paper is organized as follows. Section2 gives the general model description and the formulation of the autoregressive moving average (ARMA) identification model with some important definitions. Section 3 derives a clustering input signals based algorithm for two-input single-output model. Section 4 gives the recursive extended least squares algorithm used for comparison. Section 5 gives the magnitude-squared coherence measure used to obtain the degree of similarities between input signals and the output signal of the proposed model. Section 6 provides the steps used to perform the K-means clustering algorithm. An illustrative example to demonstrate the effectiveness of the proposed algorithms is provided in Section 7. Finally, concluding remarks are given in Section 8.


## 2. The Model Description

Firstly, let us introduce some important notations. The superscriptT denotes the matrix transpose, the symbol In stands for an identity matrix of order n, 1n represents an n-dimensional column vector whose elements are 1, X2 stands for the norm matrix X, X:=A denotes that X is defined by A, the estimate of parameter η at time t is denoted by η^t , z−1 represents a unit backward shift operator defined as z−1xt:=xt−1, and p0 is a large positive number [26].Consider the two-input single-output system, described by the autoregressive moving average model, as shown in Figure1, with a mathematical model as(1)Pzyt=Q1zu1t+Q2zu2t+Rzvt,where u1t,u2t are the two features of the input data points of the system; yt represents the output data points of the system; vt is a white noise sequence with zero mean and variance σ2; and Pz, Q1z, Q2z, and Rz are the polynomials with known degrees in the unit backward shift operator z−1 as in the following equation:(2)Pz:=1+p1z−1+p2z−2+⋯+pnpz−np,Q1z:=q11z−1+q12z−2+⋯+q1n1z−n1,Q2z:=q21z−1+q22z−2+⋯+q2n2z−n2,Rz:=1+r1z−1+r2z−2+⋯+rnrz−nr.Figure 1
Two-input single-output autoregressive moving average model.Assume thatyt,u1t,u2t, and vt are strictly proper; that is, their values are 0 for t≤0, and the orders np, n1, n2, and nr are known.From two inputsu1t,u2t and the output yt, the model parameters are to be identified.Define the parameter vectorη and the information vector ϕt as(3)η:=ηsηn∈ℝn,n:=np+n1+n2+nr,ηs:=p1,p2,…,pnp,q11,q12,…,q1n1,q21,q22,…,q2n2T∈ℝnp+n1+n2,ηn:=r1,r2,…,rnrT∈ℝnr,ϕt:=ϕstϕnt∈ℝn,ϕst:=−yt−1,−yt−2,…,−yt−np,u1t−1,u1t−2,…,u1t−n1,u2t−1,u2t−2,…,u2t−n2T∈ℝnp+n1+n2,ϕnt:=vt−1,vt−2,…,vt−nrT∈ℝnr.Based on recursive least squares in [27], the identification model of ARMA model can be expressed as(4)yt=ϕtTη+vt.The parameter vectorη contains all parameters to be estimated.


## 3. The K-Means Clustering Algorithm

K-means clustering is an unsupervised learning approach. The unsupervised learning approach is given by grouping data with similar characteristics; this grouping process is known as clustering, which consists in separating each group with different characteristics in sets called clusters [28]. As in [29, 30], K-means clustering algorithm works with three main steps: initialization step, assignment step, and update step. Firstly, in the initialization step, the process starts with selecting data Uiji=1,…,n;j=1,…,m where n is the amount of data to be carried out in clustering and m is the number of variables (data dimension), and, initially, the centre of each cluster ckjk=1,…,k;j=1,…,m is determined arbitrarily [30]. Secondly, in the assignment step, the distance such as Euclidian distance (other types of distance measures can be applied depending on the application) of data-i to centroid k is calculated according to the following equation [30]:(5)dik=∑j=1muij−ckj2,where  dik is distance of object i and centroid k, m represents data dimension, uij denotes coordinates of object i in dimension j, and ckj denotes coordinates of object k in dimension j.If the distance of data to specific cluster centroidk has the smallest value when compared to another cluster centroid, then the data will be a member of the cluster k. This can be examined according to the following criterion index [31]:(6)Min∑k=1Kdik=∑j=1muij−ckj2.Finally, in the update step, after classification of data for each cluster, centroid value can be calculated by finding the average value of the data which are members of the cluster using the following equation [31]:(7)Ckj=∑i=1puijp,where uij∈ cluster k and p denotes number of cluster k members.As in [30], the steps of the K-means clustering algorithm to cluster input signals used in the proposed model are listed as follows:(1)
Suppose given input data matrixU=uijwith i=1,2,…,n;j=1,2,…,m.(2)
Determine the number of clusters (k), with centroid chosen randomly.(3)
Calculate the distance of each input data point to the centroid using equation (5).(4)
Classify input data into clusters with the shortest distance using equation (6).(5)
Calculate the new centroid using equation (7).(6)
Repeat steps 3 through 5 until there is no more input data movement to another cluster.In our proposed method, we apply the above algorithm to the model input signals which are assumed to be represented as two input features collected from different sources.The optimal number of clusters was chosen according to the elbow method in [32]. In elbow method, a decreasing cost function is analyzed for different values of k and it can be defined as the sum or average distance from all points in the data set to its centroid. Figure 2 shows the candidate elbow point as an optimal point to select the best number of clusters according to the best total sum of distances. As seen from Figure 2, the optimal value chosen for the illustrative example used to verify the proposed method is k = 2.Figure 2
Elbow method for optimal value of k-clusters.


## 4. The Clustered Input Signals Based Recursive Least Squares (CIB-RLS) Algorithm

Our goal in this paper is the identification of the ARMA model based on clustered input signals. We start with the assumption that the two-input signals represent two different attributes supplied from a source with continuous technical and environmental changes. The first step is to apply these signals to theK-means algorithm to group them in several k-clusters to share a common statistical property.Let the two-input data set signalsu1t and u2t be applied to the K-means algorithm; the algorithm grouped these data sets into k-clusters, known as cluster-1 to cluster-k; see Figure 3 below. Every cluster contains part of data set belonging to u1t and u2t. Assume this data set to be known as u1,kt, which represents a part of u1t in the cluster k, and u2,kt, which is a part of u2t in the same cluster k; Figure 3 illustrates the assumptions. At this point, we select the parts of input signals u1kt and u2kt from each generated cluster to form new modified inputs as(8)u1modifiedt:=u1,1t,u1,2t,…,u1,ktT,(9)u2modifiedt:=u2,1t,u2,2t,…,u2,ktT.Figure 3
K-means clustering for inputs u1t and u2t to generate u1modifiedt and u2modifiedt.The modified input signals, represented by equation (8) and equation (9), are then applied to the general system under investigation as shown in Figure 4.Figure 4
Proposed system model with modified inputs signalsu1modifiedt and u2modifiedt.The proposed CIB-RLS algorithm will be as follows:(10)η^t=η^t−1+Ltyt−ϕ^Ttη^t−1,(11)Lt=Pt−1ϕ^t1+ϕ^TtPt−1ϕ^t,(12)Pt=I−Ltϕ^TtPt−1,(13)ϕt:=ϕstϕ^nt,(14)ϕst:=−yt−1,−yt−2,…,−yt−np,u1modifiedt−1,u1modifiedt−2,…,u1modifiedt−n1,u2modifiedt−1,u2modifiedt−2,…,u2modifiedt−n2T,(15)ϕ^nt:=v^t−1,v^t−2,…,v^t−nrT,(16)v^t=yt−ϕ^Ttη^t,(17)η=η^stη^nt,(18)η^st=p^1t,p^2t,…,p^npt,q^11t,q^12t,…,q^1n1t,q^21t,q^22t,…,q^2n2tT,(19)η^nt=r^1t,r^2t,…,r^nrtT.To initialize the algorithm, we choose(20)η^st=1np+n1+n2p0,η^nt=1nrp0,P0=p0Inp+n1+n2+nr,p0=106,i≤0.The steps involved in the CIB-RLS algorithm are listed as follows:(1)
Setyt=0,u1t=0,u2t=0,fort≤0.(2)
Lett=1; set the initial values of the parameter estimate vectors and the covariance matrix according to (20), and give the parameter estimation precision ε=0.01.(3)
Collect input datau1t and u2t.(4)
Select the optimal number of clustersK according to u1tandu2t collected by using elbow method.(5)
Classify input datau1t and u2t into clusters with the shortest distance by (6).(6)
Collectu1modified and u2modified as in (8) and (9), respectively, and collect the output data yt.(7)
Construct the information vectorsϕst by (14), ϕ^nt by (15), and ϕt by (13).(8)
Computev^t by (16), and update the estimate η^nt by (19).(9)
Constructη by (17).(10)
Compute the gain vectorLt by (11) and the covariance matrix Pt by (12).(11)
Update the parameter estimateη^t by (10).(12)
Ifη^t−η^t−1≤ε, obtain the parameter estimate η^t; otherwise, increase t by 1 and go to step 6.


## 5. The Recursive Extended Least Squares (RELS) Algorithm

The main algorithm used to estimate the model parameters in our proposed method is the recursive extended least squares (RELS) algorithm. With the aid of the algorithm in [14], the recursive least squares algorithm of the ARMA model described by equation (1) with its identification model expressed in equation (4) can be applied. Before we proceed, based on this identification model, the unknown variables vt−i in the information vector ϕt in (3) are replaced with their estimates v^t−i [14]. So, the recursive extended least squares (RELS) algorithm [33] is as follows:(21)η^t=η^t−1+Ltyt−ϕ^Ttη^t−1,Lt=Pt−1ϕ^t1+ϕ^TtPt−1ϕ^t,Pt=I−Ltϕ^TtPt−1,ϕt:=ϕstϕ^nt,ϕst:=−yt−1,−yt−2,…,−yt−np,u1t−1,u1t−2,…,u1t−n1,u2t−1,u2t−2,…,u2t−n2T,ϕ^nt:=v^t−1,v^t−2,…,v^t−nrT,v^t=yt−ϕ^Ttη^t,η=η^stη^nt,η^st=p^1t,p^2t,…,p^npt,q^11t,q^12t,…,q^1n1t,q^21t,q^22t,…,q^2n2tT,η^nt=r^1t,r^2t,…,r^nrtT.


## 6. The Magnitude-Squared Coherence (MSC)

The magnitude-squared coherence (MSC) is a measure that estimates the extent to which one real- or complex-valued signal can be predicted from another real- or complex-valued signal [18]. In our work, we used these measures to investigate the similarities in the frequency content between input signals and the output signal used to identify the proposed ARMA model. Comparisons were done between the cases where the two inputs are clustered before being applied to the model and the two inputs are applied directly without passing to the clustering algorithm. If we define the power spectrum at frequency ω of the input signal ut and the output yt as Suω and Suω, respectively, and the cross power spectrum as Suyω, then the magnitude-squared coherence can be given as(22)Muyω=Suyω2SuωSyω,where Suω,Syω≠0.The value of MSC lies in the range0≤Muyω≤1 which is analogue of the correlation coefficient in statistics.In our proposed model, the MSC gives the better correlation between the clustered applied input signals and the measured output signal compared to unclustered input signals with measured output.Figures5 and 6 represent the MSC for the illustrative example in Section 7 with noise variances of σ2=0.102 and σ2=0.402, respectively. The dotted line represents the case in which the input signals are unclustered, whereas the solid line represents the case of clustered input signals.Figure 5
MSC of clustered and unclustered inputs with the system outputy for noise variance of σ2=0.102.Figure 6
MSC of clustered and unclustered inputs with the system outputy for noise variance of σ2=0.402.As seen from these figures, it is clear that best measure is obtained in the case of clustered input signals. According to the type of signals, we can choose the best values of sampling rate, number of overlaps, and number of fast Fourier transform points (FFT) to achieve suitable results of MSC calculation.


## 7. Illustrative Example

In this illustrative example, we use MATLAB version R2020a and Wolfram SystemModeler software for simulation [34] to run recursive lest squares with K-means algorithms and to collect input and output data, respectively.Consider the electric circuit model with two-input voltage and a single-output voltage measured from a capacitor C as shown in Figure7. From the circuit of the Figure 7 and the concepts in [35–36], we can obtain the state-space model asFigure 7
System schematic diagram with two modified inputs signalsu1modified and u2modified. The output y is measured from the voltage across the capacitor 2.2F, and the circuit is simulated with Wolfram SystemModeler software for simulation [34].(23)x˙1x˙2=−R2/L−1/L1/C−1/R1C+1/R3Cx1x2+01/L1/R1C0u1u2,The circuit parameter is assumed to be according to Table1, where x1 represents the inductor current iL as the first state and x2 represents the capacitor voltage vc as the second state. From the above state-space model, we can find the continuous and discrete transfer functions [35], respectively, as(24)yt=0.2273s+0.2503s2+1.7238s+0.8552u1t+0.1697s2+1.7238s+0.8552u2t,(25)yt=0.01118z−1−0.01059z−21−1.915z−1+0.9174z−2u1t+0.000206z−1+0.0002002z−21−1.915z−1+0.9174z−2u2t,Table 1
Circuit parameter.
ElementValueUnitResistorR12OhmResistorR22.95OhmResistorR31.15OhmInductor (L)2.6788HenryCapacitor (C)2.2FaradFrom the circuit of Figure7 and the concepts in [35, 36], we can obtain the state-space model aswith sampling timeTs = 0.05 sec.Consider the discrete polynomials model in (25) above, the proposed ARMA model to be identified can be written in the form(26)yt=1PzQ1zu1t+Q2zu2t+Rzvt,Pz=1+p1z−1+p2z−2=1−1.915z−1+0.9174z−2,Q1z=q11z−1+q12z−2=0.01118z−1−0.01059z−2,Q2z=q21z−1+q22z−2=0.000206z−1+0.0002002z−2,Rz=1+r1z−1=1+0.7872z−1.The parameter to be estimated is(27)η=p1,p2,q11,q12,q21,q22,r1T=−1.915,0.9174,0.01118,−0.01059,0.000206,0.0002002,0.7872T.The inputsu1t,u2t are taken as a chirp signal with different ranges of frequency and different amplitudes and carry the characteristics of persistent excitement using simulation program in [34], and vt is taken as a white noise sequence with zero mean and variances of σ2=0.102 and σ2=0.402, respectively.Applying the recursive extended least squares algorithm for a comparison and clustering inputs based recursive least squares algorithm to estimate the parameter of the electric system, the parameter estimates and their errors are shown in Tables2–5, and the estimation errors δ=η^t−η/η versus t for σ2=0.102 and σ2=0.402 are shown in Figures 8 and 9, respectively.Table 2
The parameter estimates and their errors with recursive extended least squares algorithmσ2=0.102.
tp^1tp^2tq^11tq^12tq^21tq^22tr^1tδ%100−1.971380.973250.02176−0.00685−0.01218−0.016100.923537.04138150−1.961580.963450.02130−0.00742−0.00651−0.017190.919066.56978200−1.956150.958080.01742−0.00688−0.01251−0.014830.920166.46977500−1.942160.944630.00934−0.01135−0.00631−0.005870.780741.768711000−1.931210.933510.00943−0.01070−0.00418−0.005940.807661.397032000−1.928600.930520.00833−0.012240.000780.001750.794320.906443000−1.928230.930350.00852−0.01277−0.000060.002040.792040.86239True values−1.915000.917400.01118−0.010590.000210.000200.78720Table 3
The parameter estimates and their errors with clustering inputs based recursive least squares algorithmσ2=0.102.
tp^1tp^2tq^11tq^12tq^21tq^22tr^1tδ%100−1.951430.955640.020880.00902−0.00756−0.018600.857204.09050150−1.939670.942700.007670.008850.00027−0.012090.857903.63789200−1.937950.940310.006740.00148−0.003220.000900.884014.54653500−1.931310.934010.01091−0.006650.000160.007870.770771.314551000−1.923170.925600.00940−0.006780.003520.001690.798100.743562000−1.923980.925990.01380−0.010340.00362−0.002440.794450.673603000−1.924600.926810.01361−0.010120.00163−0.000700.793560.66986True values−1.915000.917400.01118−0.010590.000210.000200.78720Table 4
The parameter estimates and their errors with recursive extended least squares algorithm (σ2=0.402).
tp^1tp^2tq^11tq^12tq^21tq^22tr^1tδ%100−1.971470.973350.053530.00627−0.04936−0.065070.923218.09537150−1.961660.963530.051680.00369−0.02666−0.069360.918797.52280200−1.956210.958140.036150.00566−0.05065−0.059890.920007.39901500−1.942180.944650.00382−0.01273−0.02587−0.024030.780742.356791000−1.931220.933530.00419−0.01047−0.01735−0.024360.807661.925732000−1.928600.93052-0.00021−0.016720.002500.006420.794321.098583000−1.928230.930350.00055−0.01889−0.000860.007580.792051.08487True values−1.915000.917400.01118−0.010590.000210.000200.78720Table 5
The parameter estimates and their errors with clustering inputs based recursive least squares algorithmσ2=0.402.
tp^1tp^2tq^11tq^12tq^21tq^22tr^1tδ%100−1.952280.956510.049390.06757−0.03077−0.075240.858256.57724150−1.940320.94335−0.003570.066910.00048−0.049130.858675.42231200−1.938540.94091−0.007060.03772−0.013510.002820.884575.12093500−1.931840.934550.009720.005150.000040.030790.770741.991641000−1.923490.925920.003890.004590.013440.006140.798091.215292000−1.924120.926130.02165−0.009410.01386−0.010360.794441.100213000−1.924740.926950.02079−0.008580.00593−0.003400.793510.84690True values−1.915000.917400.01118−0.010590.000210.000200.78720Figure 8
The parameter estimation errorδ versus time t for σ2=0.102.Figure 9
The parameter estimation errorδ versus time t for σ2=0.402.Through Tables2–5 and Figures 8 and 9, we can infer the following observations:(i)
With a suitable selection of the input signals and choosing the optimal value ofk in K-means clustering algorithm, the parameter estimation error decreases gradually with the data length t increasing. This, in turn, shows that the proposed algorithm gives effective results.(ii)
Since, at a certain time, the model is excited with clustered input signals that share the same characteristics with a high degree of similarity due to theK-means clustering algorithm, the same system properties could be excited and the accuracy of the model identification is improved.(iii)
Compared with the recursive extended least squares algorithm, the proposed clustering inputs based recursive least squares algorithm has better identification performance.(iv)
Using clustering inputs based recursive least squares algorithm, the parameter estimates converge to their true values faster than using recursive extended least squares algorithm.


## 8. Conclusions

This paper investigates one aspect of unsupervised learning effects on the system identification field. The clustering inputs based recursive least squares algorithm for the two-input single-output system with moving average noise is proposed by introducing clustering techniques to the excitation input signals. The simulation results indicate that the proposed algorithm can yield more accurate parameter estimates under certain conditions (i.e., selection of the optimal value ofK-means) compared with the recursive extended least squares algorithm. The proposed method can be extended to multiple-input multiple-output systems and stretched forth to investigate the identification of nonlinear systems and time-delay systems.



---

*Source: 2498487-2020-11-18.xml*