# Intelligent Noise Removal from EMG Signal Using Focused Time-Lagged Recurrent Neural Network

Authors: S. N. Kale; S. V. Dudul
Journal: Applied Computational Intelligence and Soft Computing
 (2009)

---

## Abstract

Electromyography (EMG) signals can be used for clinical/biomedical application and modern human computer interaction. EMG signals acquire noise while traveling through tissue, inherent noise in electronics equipment, ambient noise, and so forth. ANN approach is studied for reduction of noise in EMG signal. In this paper, it is shown that Focused Time-Lagged Recurrent Neural Network (FTLRNN) can elegantly solve to reduce the noise from EMG signal. After rigorous computer simulations, authors developed an optimal FTLRNN model, which removes the noise from the EMG signal. Results show that the proposed optimal FTLRNN model has an MSE (Mean Square Error) as low as 0.000067 and 0.000048, correlation coefficient as high as 0.99950 and 0.99939 for noise signal and EMG signal, respectively, when validated on the test dataset. It is also noticed that the output of the estimated FTLRNN model closely follows the real one. This network is indeed robust as EMG signal tolerates the noise variance from 0.1 to 0.4 for uniform noise and 0.30 for Gaussian noise. It is clear that the training of the network is independent of specific partitioning of dataset. It is seen that the performance of the proposed FTLRNN model clearly outperforms the best Multilayer perceptron (MLP) and Radial Basis Function NN (RBF) models. The simple NN model such as the FTLRNN with single-hidden layer can be employed to remove noise from EMG signal.


---

## Body

## 1. Introduction

Biomedical  signal  means a collective
electrical  signal acquired  from any organ that represents a physical variable
of interest. This signal is normally a function of time and is describable in
terms of its amplitude, frequency, and phase. The EMG signal is a biomedical
signal that measures electrical currents generated in muscles during its
contraction representing neuromuscular activities. The nervous system always
controls the muscle activity (contraction/relaxation). Hence, the EMG signal is
a complicated signal, which is controlled by the nervous system and is
dependent on the anatomical and physiological properties of muscles. EMG signal acquires noise while traveling
through different tissues. Moreover, the EMG detector, particularly if it is at
the surface of the skin, collects signals from different motor units at a time
which may generate interaction of different signals. Detection of EMG signals
with powerful and advance methodologies is becoming a very important requirement
in biomedical engineering. The main reason for the interest in EMG signal
analysis is in clinical diagnosis and biomedical applications. So far, research
and extensive efforts have been made in the area, developing better algorithms,
upgrading existing methodologies, and improving detection techniques to reduce
noise and to acquire accurate EMG signals [1]. Noise removal from noisy EMG
signal is a filtering problem. Here the Neural Network model is trained to
separate known noise from EMG signal.Literature
survey [2–5] shows that Neural Networks (NNs) have been efficiently used
for nonlinear multivariable function approximation. However, there is still
enough scope to choose an appropriate NN model so that the performance measures
are optimized to approach zero and unity for mean square error (MSE) and
correlation coefficient (r), respectively. In function approximation, the goal
is to find the parameters of the best linear approximation to the input and the
desired response pairs. In nonlinear system identification, conventional
techniques such as least square approach, partial least square regression,
principal components regression, ordinary least square regression, regression tree, Levenberg Marquardt algorithm, and
multivariate adaptive regression splines algorithm generally do not work
reasonably if the underlying problem is overly complex [6–8]. Therefore NN
approach is worth considering for solving system identification problem [9]. A
typical problem of noise removal in EMG signal is considered in this paper. This benchmark data for noise removal
in EMG signal is taken from the companion CD of a book on neural network [10]. Data contains an
electromyographic (EMG) signal and the interference (60 Hz) noise picked from
the power supply. The two files are, respectively, “EMG with noise” and “noise”
only. The goal is to obtain back the EMG using adaptive filtering techniques.
The training file is used to train a neural network for noise removal from EMG
signal.Optimal Focused
Time Lag Recurrent Neural Network (FTLRNN) is developed to remove noise effectively from EMG signal.
Other classes of NN configuration such as Multilayer Perceptron Neural
Network (MLP NN) and Radial Basis Function (RBF) have also been compared for such noise
removal problem.This paper deals with intelligent
removal of noise from the EMG signal using FTLRNN-based model.


## 2. EMG and Sources
of Noise

EMG stands
for electromyography. It is the study of signals. EMG is sometimes referred to
as myoelectric activity. Muscle tissue conducts electrical potentials similar
to the way nerves do, and the name given to these electrical signals is the
muscle action potential. Surface EMG is a method of recording the information
present in these muscle action potentials. When detecting and recording the EMG
signal, there are two main issues of concern that influence the fidelity of the
signal. The first is the signal-to-noise ratio. That is, the ratio of the
energy in the EMG signals to the energy in the noise signal. In general, noise is
defined as electrical signals that are not part of the desired EMG signal. The
other issue is the distortion of the signal, meaning that the relative
contribution of any frequency component in the EMG signal should not be
altered.  There are many applications for the use
of EMG. EMG is used clinically for the diagnosis of neurological and
neuromuscular problems. It is used diagnostically by gait laboratories and by
clinicians trained in the use of biofeedback or ergonomic assessment. EMG is
also used in many types of research laboratories, including those involved in
biomechanics, motor control, neuromuscular physiology, movement disorders,
postural control, and physical therapy.Electrical Noise and Factors Affecting EMG Signal
The amplitude range of EMG signal
is 0–10 mV (+5 to −5) prior to amplification. EMG signals acquire noise while
traveling through different tissues. It is important to understand the
characteristics of the electrical noise. Electrical noise, which will affect
EMG signals, can be categorized into the following types.(1) Inherent Noise in Electronics Equipment
All electronics equipments generate noise.
This noise cannot be eliminated; using high-quality electronic components can
only reduce it.(2) Ambient Noise
Electromagnetic radiation is the source of this kind of noise. The
surfaces of our bodies are constantly inundated with electric-magnetic
radiation, and it is virtually impossible to avoid exposure to it on the surface
of earth. The ambient noise may have amplitude that is one to three orders of
magnitude greater than the EMG signal.(3) Motion Artifact
Motion artifact causes irregularities in the data. There are two
main sources for motion artifact: (1) electrode interface and (2) electrode
cable. Motion artifact can be reduced by proper design of the electronics
circuitry and set-up.(4) Inherent Instability of Signal
The amplitude of EMG is random in nature. EMG signal
is affected by the firing rate of the motor units, which, in most conditions,
fire in the frequency region of 0 to 20 Hz. This kind of noise is considered as
unwanted, and the removal of the noise is important.


## 3. Performance Measures

Assessment of
the performance of various neural networks is done by visual inspection of EMG
and noise signals from the graph as well as from the optimal values of Mean Square Error (MSE), andr
(Correlation coefficient).Mean Square Error (MSE)
The formula for the mean square error is
(1)MSE=∑j=0P∑i=0N(dij−yij)2NP,where P = number of output processing elements, N = number of
exemplars in the dataset, yij =
network output for exemplar i at processing element j, and dij =
desired output for exemplar i at processing element j.Correlation Coefficient (r)
By  definition, the
correlation coefficient between a network outputx and a desired output d is
(2)r=∑i(xi−x¯)(di−d¯)/N∑i(di−d¯)2/N∑i(xi−x¯)2/N,where x¯=(1/N)∑i=1Nxi and d¯=(1/N)∑i=1Ndi.
The correlation coefficient is confined to
the range [−1,1]. When r = 1, there
is a perfect positive linear correlation between x and d, that is, they
covary, which means that they vary by the same amount.


## 4. Computer Simulation

Here a dataset is chosen that can be
used in removal of noise from EMG signal. There are 2000 training patterns.
Training of the neural network should be independent of dataset. Therefore
different permutations and combinations of the dataset producing many independent
datasets are used for training and testing of neural networks.Table1 depicts the various datasets
on which the neural networks are trained. Once the data is randomized, the
total samples are divided into three parts, namely, training, cross validation,
and testing samples. If the samples are divided in the sequence of training,
cross validation, and testing, it is a forward tagging. On the other hand the
sequence of testing, cross validation, and then training is termed as reverse
tagging. Percentage of training and testing samples are varied, and cross
validation samples are kept constant as shown in Table 1(a). Forward tagging
and reverse tagging of dataset give total 16 different datasets to assess the
performance of an estimated network model. This dataset is also tested for
multifold differential learning. Multifold differential learning of neural
network is carried out on the dataset, that is, the total samples are divided into
four groups each containing 500 samples as given in Table 1(b). Sample numbers
of each group is mentioned in  Table 1(b). All  possible  combinations are used to
train the neural network and assess the performance by testing. There are total
34 datasets formed for differential learning as described in Table 1(b). To
assess the performance of neural network skeptically, total 50 different
datasets are used. This is necessary because the estimated NN model should
consistently work on the different datasets. This also ensures that the
proposed NN model has truly learned meaningful information from the dataset and
is free from biases.Table 1
EMG
signal datasets used for NN-based models. Total number of samples = 2000.
(a)
Datasets based on forward and reverse tagging and
% variation
S.N.DatasetTagging% variation(Train-CV-test)1Dataset1Forward (training,
cross validation, and testing)10-15-75%2Dataset220-15-65%3Dataset330-15-55%4Dataset440-15-45%5Dataset550-15-35%6Dataset660-15-25%7Dataset770-15-15%8Dataset880-15-05%9Dataset9Reverse
(testing, cross validation,
and training)10-15-75%10Dataset1020-15-65%11Dataset1130-15-55%12Dataset1240-15-45%13Dataset1350-15-35%14Dataset1460-15-25%15Dataset1570-15-15%16Dataset1680-15-05%(b)
Datasets based on multifold differential learning. Group I: (1–500 samples), Group II: (501–1000 samples), Group III: (1001–1500 samples), Group IV: (1501–2000 samples)
DatasetTrain on groupTest on groupDatasetTrain on groupTest on groupDataset1IIIDataset18I + IIIII + IVDataset2IIIIDataset19I + IVIIDataset3IIVDataset20I + IVIIIDataset4IIIIIDataset21I + IVII + IIIDataset5IIIVDataset22II + IIIIDataset6IIIDataset23II + IIIIVDataset7IIIIVDataset24II + IIII + IVDataset8IIIIDataset25II + IVIDataset9IIIIIDataset26II + IVIIIDataset10IVIDataset27II + IVI + IIIDataset11IVIIDataset28III + IVIDataset12IVIIIDataset29III + IVIIDataset13I + IIIIIDataset30III + IVI + IIDataset14I + IIIVDataset31I + II + IIIIVDataset15I + IIIII + IVDataset32II + III + IVIDataset16I + IIIIIDataset33III + IV + IIIDataset17I + IIIIVDataset34IV + I + IIIIIEvaluation
of NN is done by a standard method in statistics calledindependent
validation where the
available data are divided into a training set, a cross validation (CV) set, and
a test set. The entire dataset is usually randomized first. The training data
is used to update the weights in the network. The test data is then used to
assess how well the network has generalized. The learning and generalization
ability of the estimated NN model is assessed on the basis of performance
measures such as MSE, correlation coefficient r, and visual
inspection of desired and actual graphs of EMG signal.The
network has been trained at least 5 times starting from different random
initial weights so as to avoid local minima. Neurodimension NeuroSolutions
(version 5) is specifically used for obtaining results. System with 512 MB RAM,
40 GB hard disk, 2 MB cache, and 1.6 GHz clock is used to carry out this
simulation.Various
neural networks are used to compare the performance, and FTLRNN is the best in
removal of noise from EMG signal.


### 4.1. MLP NN

MLP-based NN
model is used in this study because it has solid theoretical foundation [11].
MLPs are feedforward neural networks trained with the standard backpropagation
algorithm [12]. They are supervised networks, so they require a desired response
to be trained. Figure 1 shows the architecture of MLP NN.Figure 1
A simple multilayer perceptron.An
exhaustive and careful experimental study has been carried out to determine the
optimal configuration of MLP NN model. All possible variations such as number of hidden layers, number of PEs
(processing elements) in each hidden layer, different transfer functions in the
output layer, and different supervised learning rules are investigated in
simulation.Table2 shows various parameters of
the MLP NN model which are varied for obtaining optimal parameters.Table 2
Variable parameters of MLP NN model.
S.N.ParameterTypical rangeOptimal parameter1Hidden layer1 to 422PEH I-1 to 20
H II-1 to 10 H III-10
H IV-10Hidden layer I-15
Hidden layer II-103Learning ruleMomentum (Mom), Conjugate gradient (CG), Levenberg
Marquardt (LM), Quick propagation (QP), Step, Delta bar deltaMomentum4Transfer function in output layerLinear, Lineartanh, TanhTanhSupervised learning epochs = 1000,
error threshold = 0.01, transfer function in hidden layer = tanh, number of PEs in
input layer = 1, and number of PEs in output layer = 2.The number of
hidden layers is varied from 1 to 4, and performance measures of the MLP NN model
are found better for two hidden layers as shown in Table3. With increase in
number of hidden layers, the performance of the network has not improved
significantly.Table 3
Number of hidden
layer andr.
S.N.No. of hidden layerCorrelation coefficientNoiseEMG110.769660.56711220.772120.57113330.771070.56936440.765740.56861It is found from Figures2 and 3 that the
optimal performance of the model is obtained for 15 neurons in the first hidden
layer and 10 neurons in the second hidden layer with regard to MSE minimum,
r-correlation coefficient. Figures 2 and 3 portray average MSE with respect to the
number of PEs in the first and second hidden layers, respectively.Figure 2
Variation of average of minimum MSE
for 5 runs versus number of PEs in the first hidden layer.Figure 3
Variation of average of minimum MSE
for 5 runs versus number of PEs in the second hidden layer.Figures4 and 5 depict modeling capability of MLP NN on test
dataset which portrays desired output and actual output of the MLP NN on test
dataset. It is seen that actual outputs of EMG signal and noise signal do not
follow the desired output closely. There has been a lot of deviations between
the output of the NN and the desired output.Figure 4
Modeling capability of MLP NN on testing dataset for EMG signal.Figure 5
Modeling capability of MLP NN on testing dataset for EMG noise
signal.For the datasets MLP NN model is
trained for five times. The performance measures such as MSE andr on training
dataset and testing dataset are obtained. Optimal performance is obtained when
80% of the entire dataset is used for training, 15% for cross validation, and 5%
for testing. The  correlation coefficient on test dataset is found as high as 0.78113and MSE
= 0.02501 for EMG signal and
for noise signal r = 0.5843 and MSE = 0.02485.


### 4.2. Focused Time Lag Recurrent Neural Network (FTLRNN)

Time-lagged recurrent networks (TLRNs) are MLPs extended with short-term memory
structures. Most real-world data contains information in its time structure,
that is, how the data changes with time. TLRNs are the state of the art in
nonlinear time series prediction, system identification, and temporal pattern classification.Recurrent
networks are neural networks with one or more feedback loops. The TDNN memory structure is simply  a
cascade of ideal delays (a delay of one sample). The gamma memory is a cascade
of leaky integrators. The Laguaerre memory is slightly more sophisticated than
the gamma memory in that it orthogonalizes the memory space. This is useful
when working with large memory kernels [10].The
input PEs of an MLP are replaced with a tap delay line. It is called the
focused time delay neural network (TDNN). The topology is called focused
because the memory is only at the input layer [13].The delay line of the focused TDNN
stores the past samples of the input. The combination of the tap delay line and
the weights that connect the taps to the PEs of the first hidden layer is
simply linear combiners followed by a static nonlinearity. The first layer of
the focused TDNN is therefore a filtering layer, with as many adaptive filters
as PEs in the first hidden layer.The focused TDNN topology has
been successfully used in nonlinear system identification, time series
prediction, and temporal pattern recognition. Figure6 shows architecture of
FTLRNN. The focused topology of Figure 6 is a recurrent neural network and the recurrency is local to the PE. One of the advantages of locally recurrent neural networks is that the
stability of the system can be judged by constraining the value of the local
feedback parameters so that the local PE is stable. If local stability is enforced, the global system will be stable.Figure 6
MLP with an input layer of context
PEs (focused architecture).A thorough experimental study has
been carried out to determine optimal parameters of FTLRNN model. Here the number
of hidden layers is varied from 1 to 2, and performance measures of the FTLRNN
model are found better for single hidden layer as shown in Table4. With
increase in the number of hidden layers, the performance of the network has not
improved significantly.Table 4
Number of hidden
layers andr.
S.N.No. of hidden layerCorrelation coefficientNoiseEMG110.995330.99471220.989930.99302Figure7 portrays average MSE with
respect to the number of PEs in the first hidden layer. 27 neurons are selected for
optimal performance.Figure 7
Variation of average of minimum MSE for 5 runs versus
number of PEs in the first hidden layer.Table5 shows various parameters of
the FTLRNN model which are varied for obtaining optimal parameters. For
momentum learning rule, the results are optimum. The Momentum provides
the gradient descent with some inertia, so that it tends to move along a
direction, that is, the average estimate for down. The amount of inertia (i.e.,
how much of the past to average over) is dictated by the momentum parameter, ρ. The higher
the momentum is, the more it smoothes the gradient estimate and the less effect a
single change in the gradient has on the weight change. Linear transfer
function has optimal results.Table 5
Variable parameters of FTLRNN model.
S.N.ParameterTypical rangeOptimal value1Learning ruleMomentum, Conjugate Gradient, Levenberg
Marquardt (LM), Quick propagation (qp), step, delta bar delta (dbd)Momentum2Transfer function in output
layerLinear, lineartanh, TanhLinear3Memory structureTDNNAxon, Gamma, LaguarreLaguarre4Depth in samples2 to 1245Length of trajectory10 to 6050Supervised
learning epochs = 1000, error threshold = 0.01, transfer function in hidden
layer = tanh, number of PEs in input layer = 1, number of PEs in hidden layer 1 = 27,
and number of PEs in output layer = 2.For the various datasets, FTLRNN model is
trained for five times with different random initializations of connection
weights. The performance measures like MSE andr on training dataset, cross
validation dataset, and testing dataset are obtained. Optimal performance is
obtained for training 80%, cross validation 15%, and 5% testing. The correlation
coefficient on test dataset is found as 0.9984 and 0.9973 for noise signal and
EMG, respectively. MSE for EMG signal and noise is obtained as 0.0002.Table6 depicts  that the Laguarre
memory structure leads to the optimal performance. Laguarre is a local
recurrent memory structure. It has internal feedback loops with an adaptable
weight. The Laguerre memory is slightly more sophisticated than the
gamma memory in that it orthogonalizes the memory space. This is useful when
working with large memory kernels. The Laguarre memory is based on the Laguarre functions. The Laguarre functions are an
orthogonal set of functions that are built from a low-pass filter followed by a cascade of all
pass functions.Table 6
Selection of memory structure for optimal performance of FTLRNN
model.
S.N.Memory structureMean square errorCorrelation coefficientNoiseEMGNoiseEMG1TDNNAxon0.0004550.0003080.997190.9967542Gamma0.0003010.0002810.998270.9972043Laguarre0.00026240.0002800.9984270.997305Depth of samples parameter (D) is
used to compute the number of taps (T) contained within memory structure of the
network. Optimal value of D is 4 as shown in Table 7.Table 7
Selection of depth of samples for optimal performance of FTLRNN
model.
S.N.Depth inMean square errorCorrelation coefficientsamplesNoiseEMGNoiseEMG120.029820.029820.77090.5628240.000240.000240.99850.9975360.000370.000310.99770.9969480.000310.000360.998060.99625120.000470.000280.99720.9971The trajectory length corresponds to
the samples setting within the dynamic controller. It specifies how many
samples to read before backpropagation occurs. Table8 shows the length of
trajectory selected as 50 for optimal performance.Table 8
Selection of length of trajectory for optimal performance of
FTLRNN model.
S.N.Length ofMean square errorCorrelation coefficienttrajectoryNoiseEMGNoiseEMG1050.000230.0002280.99860.99742100.000240.0002360.99850.99763200.000240.000250.99850.99744300.000240.000250.99850.99755400.000230.000220.99850.99756500.000230.000240.99850.99767600.000240.000250.99840.9974Figures8 and 9 display modeling capability of FTLRNN, which
shows desired output and actual output of the FTLRNN on test dataset for EMG
and noise, respectively. It is seen that the output of the NN follows the
desired output very closely.Figure 8
Modeling capability of FTLRNN on testing dataset for EMG signal.Figure 9
Modeling capability of FTLRNN on testing dataset for EMG noise
signal.Figures10 and 11 display modeling capability of FTLRNN, which
shows desired output and actual output of the FTLRNN on training dataset for
signal and noise, respectively. It is seen that actual output follows the
desired output closely.Figure 10
Modeling capability of FTLRNN on training dataset.Figure 11
Modeling capability of FTLRNN on training dataset.


### 4.3. Radial Basis Function (RBF)

RBF was first introduced in the solution of
the real multivariate interpolation problem [14, 15]. The construction of an RBF
network, in its most basic form, involves three layers. The input layer is made
up of source nodes (sensory units) that connect the network to its environment.
The second layer, the only hidden layer in the network, applies a nonlinear
transformation from the input space to the hidden space. The output layer is
linear, supplying the response of the network to the activation pattern
(signal) applied to the input layer [16]. Architecture of RBF NN model is shown
in Figure 12.Figure 12
Radial basis function (RBF) network.A rigorous experimental study has
been undertaken to determine optimal performance of RBF NN model. The variable
parameters of RBF NN are listed in Table9.Table 9
Variable parameters of RBF NN model.
S.N.ParameterTypical rangeOptimal
parameter1Cluster centers5 to 10052Unsupervised learning ruleConscience-full, standard
fullConscience-full3Supervised learning ruleMomentum, Conjugate gradient, Levenberg
Marquardt, Quick Propagation, Step, delta bar deltaLevenberg
Marquardt4MetricEuclidean, Dot Product, Box
carEuclidean5Transfer function in output
layerLinear, lineartanh, TanhLinearFrom Figure13, it is seen that the
optimal performance is obtained with 5 cluster centers.Figure 13
Variation of correlation coefficient for optimal number of cluster centers for
RBF NN model.Tables10 and 11 depicts the optimal
performance of RBF NN. Conscience full-unsupervised learning rule and Euclidean
competitive learning metric are selected for optimal performance.Table 10
Selection of unsupervised
learning rule for RBF NN model.
S.N.Unsupervised learning ruleMean square errorCorrelation coefficientNoiseEMGNoiseEMG1Conscience full0.024840.0249930.78270.58252Standard full0.029540.029500.773380.5698Table 11
Selection of competitive
learning metric for RBF NN model.
S.N.MetricMean square errorCorrelation coefficientNoiseEMGNoiseEMG1Euclidean0.024840.0249930.78270.58252Dot product0.029850.029670.77130.56643Box car0.029550.0294960.773290.56994Figures14 and 15 give modeling capability of RBFNN, which
show desired output and actual output of the RBF NN on test dataset for EMG
signal. It is seen that actual output follows the desired output distantly.Figure 14
Modeling capability of RBF on test dataset of EMG signal.Figure 15
Modeling capability of RBF on test dataset of EMG noise signal.


## 4.1. MLP NN

MLP-based NN
model is used in this study because it has solid theoretical foundation [11].
MLPs are feedforward neural networks trained with the standard backpropagation
algorithm [12]. They are supervised networks, so they require a desired response
to be trained. Figure 1 shows the architecture of MLP NN.Figure 1
A simple multilayer perceptron.An
exhaustive and careful experimental study has been carried out to determine the
optimal configuration of MLP NN model. All possible variations such as number of hidden layers, number of PEs
(processing elements) in each hidden layer, different transfer functions in the
output layer, and different supervised learning rules are investigated in
simulation.Table2 shows various parameters of
the MLP NN model which are varied for obtaining optimal parameters.Table 2
Variable parameters of MLP NN model.
S.N.ParameterTypical rangeOptimal parameter1Hidden layer1 to 422PEH I-1 to 20
H II-1 to 10 H III-10
H IV-10Hidden layer I-15
Hidden layer II-103Learning ruleMomentum (Mom), Conjugate gradient (CG), Levenberg
Marquardt (LM), Quick propagation (QP), Step, Delta bar deltaMomentum4Transfer function in output layerLinear, Lineartanh, TanhTanhSupervised learning epochs = 1000,
error threshold = 0.01, transfer function in hidden layer = tanh, number of PEs in
input layer = 1, and number of PEs in output layer = 2.The number of
hidden layers is varied from 1 to 4, and performance measures of the MLP NN model
are found better for two hidden layers as shown in Table3. With increase in
number of hidden layers, the performance of the network has not improved
significantly.Table 3
Number of hidden
layer andr.
S.N.No. of hidden layerCorrelation coefficientNoiseEMG110.769660.56711220.772120.57113330.771070.56936440.765740.56861It is found from Figures2 and 3 that the
optimal performance of the model is obtained for 15 neurons in the first hidden
layer and 10 neurons in the second hidden layer with regard to MSE minimum,
r-correlation coefficient. Figures 2 and 3 portray average MSE with respect to the
number of PEs in the first and second hidden layers, respectively.Figure 2
Variation of average of minimum MSE
for 5 runs versus number of PEs in the first hidden layer.Figure 3
Variation of average of minimum MSE
for 5 runs versus number of PEs in the second hidden layer.Figures4 and 5 depict modeling capability of MLP NN on test
dataset which portrays desired output and actual output of the MLP NN on test
dataset. It is seen that actual outputs of EMG signal and noise signal do not
follow the desired output closely. There has been a lot of deviations between
the output of the NN and the desired output.Figure 4
Modeling capability of MLP NN on testing dataset for EMG signal.Figure 5
Modeling capability of MLP NN on testing dataset for EMG noise
signal.For the datasets MLP NN model is
trained for five times. The performance measures such as MSE andr on training
dataset and testing dataset are obtained. Optimal performance is obtained when
80% of the entire dataset is used for training, 15% for cross validation, and 5%
for testing. The  correlation coefficient on test dataset is found as high as 0.78113and MSE
= 0.02501 for EMG signal and
for noise signal r = 0.5843 and MSE = 0.02485.


## 4.2. Focused Time Lag Recurrent Neural Network (FTLRNN)

Time-lagged recurrent networks (TLRNs) are MLPs extended with short-term memory
structures. Most real-world data contains information in its time structure,
that is, how the data changes with time. TLRNs are the state of the art in
nonlinear time series prediction, system identification, and temporal pattern classification.Recurrent
networks are neural networks with one or more feedback loops. The TDNN memory structure is simply  a
cascade of ideal delays (a delay of one sample). The gamma memory is a cascade
of leaky integrators. The Laguaerre memory is slightly more sophisticated than
the gamma memory in that it orthogonalizes the memory space. This is useful
when working with large memory kernels [10].The
input PEs of an MLP are replaced with a tap delay line. It is called the
focused time delay neural network (TDNN). The topology is called focused
because the memory is only at the input layer [13].The delay line of the focused TDNN
stores the past samples of the input. The combination of the tap delay line and
the weights that connect the taps to the PEs of the first hidden layer is
simply linear combiners followed by a static nonlinearity. The first layer of
the focused TDNN is therefore a filtering layer, with as many adaptive filters
as PEs in the first hidden layer.The focused TDNN topology has
been successfully used in nonlinear system identification, time series
prediction, and temporal pattern recognition. Figure6 shows architecture of
FTLRNN. The focused topology of Figure 6 is a recurrent neural network and the recurrency is local to the PE. One of the advantages of locally recurrent neural networks is that the
stability of the system can be judged by constraining the value of the local
feedback parameters so that the local PE is stable. If local stability is enforced, the global system will be stable.Figure 6
MLP with an input layer of context
PEs (focused architecture).A thorough experimental study has
been carried out to determine optimal parameters of FTLRNN model. Here the number
of hidden layers is varied from 1 to 2, and performance measures of the FTLRNN
model are found better for single hidden layer as shown in Table4. With
increase in the number of hidden layers, the performance of the network has not
improved significantly.Table 4
Number of hidden
layers andr.
S.N.No. of hidden layerCorrelation coefficientNoiseEMG110.995330.99471220.989930.99302Figure7 portrays average MSE with
respect to the number of PEs in the first hidden layer. 27 neurons are selected for
optimal performance.Figure 7
Variation of average of minimum MSE for 5 runs versus
number of PEs in the first hidden layer.Table5 shows various parameters of
the FTLRNN model which are varied for obtaining optimal parameters. For
momentum learning rule, the results are optimum. The Momentum provides
the gradient descent with some inertia, so that it tends to move along a
direction, that is, the average estimate for down. The amount of inertia (i.e.,
how much of the past to average over) is dictated by the momentum parameter, ρ. The higher
the momentum is, the more it smoothes the gradient estimate and the less effect a
single change in the gradient has on the weight change. Linear transfer
function has optimal results.Table 5
Variable parameters of FTLRNN model.
S.N.ParameterTypical rangeOptimal value1Learning ruleMomentum, Conjugate Gradient, Levenberg
Marquardt (LM), Quick propagation (qp), step, delta bar delta (dbd)Momentum2Transfer function in output
layerLinear, lineartanh, TanhLinear3Memory structureTDNNAxon, Gamma, LaguarreLaguarre4Depth in samples2 to 1245Length of trajectory10 to 6050Supervised
learning epochs = 1000, error threshold = 0.01, transfer function in hidden
layer = tanh, number of PEs in input layer = 1, number of PEs in hidden layer 1 = 27,
and number of PEs in output layer = 2.For the various datasets, FTLRNN model is
trained for five times with different random initializations of connection
weights. The performance measures like MSE andr on training dataset, cross
validation dataset, and testing dataset are obtained. Optimal performance is
obtained for training 80%, cross validation 15%, and 5% testing. The correlation
coefficient on test dataset is found as 0.9984 and 0.9973 for noise signal and
EMG, respectively. MSE for EMG signal and noise is obtained as 0.0002.Table6 depicts  that the Laguarre
memory structure leads to the optimal performance. Laguarre is a local
recurrent memory structure. It has internal feedback loops with an adaptable
weight. The Laguerre memory is slightly more sophisticated than the
gamma memory in that it orthogonalizes the memory space. This is useful when
working with large memory kernels. The Laguarre memory is based on the Laguarre functions. The Laguarre functions are an
orthogonal set of functions that are built from a low-pass filter followed by a cascade of all
pass functions.Table 6
Selection of memory structure for optimal performance of FTLRNN
model.
S.N.Memory structureMean square errorCorrelation coefficientNoiseEMGNoiseEMG1TDNNAxon0.0004550.0003080.997190.9967542Gamma0.0003010.0002810.998270.9972043Laguarre0.00026240.0002800.9984270.997305Depth of samples parameter (D) is
used to compute the number of taps (T) contained within memory structure of the
network. Optimal value of D is 4 as shown in Table 7.Table 7
Selection of depth of samples for optimal performance of FTLRNN
model.
S.N.Depth inMean square errorCorrelation coefficientsamplesNoiseEMGNoiseEMG120.029820.029820.77090.5628240.000240.000240.99850.9975360.000370.000310.99770.9969480.000310.000360.998060.99625120.000470.000280.99720.9971The trajectory length corresponds to
the samples setting within the dynamic controller. It specifies how many
samples to read before backpropagation occurs. Table8 shows the length of
trajectory selected as 50 for optimal performance.Table 8
Selection of length of trajectory for optimal performance of
FTLRNN model.
S.N.Length ofMean square errorCorrelation coefficienttrajectoryNoiseEMGNoiseEMG1050.000230.0002280.99860.99742100.000240.0002360.99850.99763200.000240.000250.99850.99744300.000240.000250.99850.99755400.000230.000220.99850.99756500.000230.000240.99850.99767600.000240.000250.99840.9974Figures8 and 9 display modeling capability of FTLRNN, which
shows desired output and actual output of the FTLRNN on test dataset for EMG
and noise, respectively. It is seen that the output of the NN follows the
desired output very closely.Figure 8
Modeling capability of FTLRNN on testing dataset for EMG signal.Figure 9
Modeling capability of FTLRNN on testing dataset for EMG noise
signal.Figures10 and 11 display modeling capability of FTLRNN, which
shows desired output and actual output of the FTLRNN on training dataset for
signal and noise, respectively. It is seen that actual output follows the
desired output closely.Figure 10
Modeling capability of FTLRNN on training dataset.Figure 11
Modeling capability of FTLRNN on training dataset.


## 4.3. Radial Basis Function (RBF)

RBF was first introduced in the solution of
the real multivariate interpolation problem [14, 15]. The construction of an RBF
network, in its most basic form, involves three layers. The input layer is made
up of source nodes (sensory units) that connect the network to its environment.
The second layer, the only hidden layer in the network, applies a nonlinear
transformation from the input space to the hidden space. The output layer is
linear, supplying the response of the network to the activation pattern
(signal) applied to the input layer [16]. Architecture of RBF NN model is shown
in Figure 12.Figure 12
Radial basis function (RBF) network.A rigorous experimental study has
been undertaken to determine optimal performance of RBF NN model. The variable
parameters of RBF NN are listed in Table9.Table 9
Variable parameters of RBF NN model.
S.N.ParameterTypical rangeOptimal
parameter1Cluster centers5 to 10052Unsupervised learning ruleConscience-full, standard
fullConscience-full3Supervised learning ruleMomentum, Conjugate gradient, Levenberg
Marquardt, Quick Propagation, Step, delta bar deltaLevenberg
Marquardt4MetricEuclidean, Dot Product, Box
carEuclidean5Transfer function in output
layerLinear, lineartanh, TanhLinearFrom Figure13, it is seen that the
optimal performance is obtained with 5 cluster centers.Figure 13
Variation of correlation coefficient for optimal number of cluster centers for
RBF NN model.Tables10 and 11 depicts the optimal
performance of RBF NN. Conscience full-unsupervised learning rule and Euclidean
competitive learning metric are selected for optimal performance.Table 10
Selection of unsupervised
learning rule for RBF NN model.
S.N.Unsupervised learning ruleMean square errorCorrelation coefficientNoiseEMGNoiseEMG1Conscience full0.024840.0249930.78270.58252Standard full0.029540.029500.773380.5698Table 11
Selection of competitive
learning metric for RBF NN model.
S.N.MetricMean square errorCorrelation coefficientNoiseEMGNoiseEMG1Euclidean0.024840.0249930.78270.58252Dot product0.029850.029670.77130.56643Box car0.029550.0294960.773290.56994Figures14 and 15 give modeling capability of RBFNN, which
show desired output and actual output of the RBF NN on test dataset for EMG
signal. It is seen that actual output follows the desired output distantly.Figure 14
Modeling capability of RBF on test dataset of EMG signal.Figure 15
Modeling capability of RBF on test dataset of EMG noise signal.


## 5. Results and
Comparison

Table12
depicts  the performance parameters  for variation in learning rules for MLP NN,
FTLRNN, and RBFNN on test dataset. From Table 12, it is observed that focused
time-lagged recurrent neural network gives optimal performance for linear
transfer function.Table 12
Selection of transfer function for various NN models on test
dataset.
S.N.NNSignalTransfer functionTanhLinear tanhLinearMSErMSErMSEr1MLPN0.025010.781140.025570.775570.025210.7792EMG0.024850.584330.02510.57790.025130.57812FTLRNNN0.000270.997970.000150.998820.0000670.99843EMG0.000230.996950.000130.99810.0000480.997313RBFN0.24730.783850.024820.782970.024770.7839EMG0.024810.58520.024810.58510.024800.5854Table13 depicts the selection of learning rule for
optimal performance of each NN. In FTLRNN momentum learning rule is selected
for the best performance.Table 13
Selection of optimal learning rule for various
NN models on test dataset.
S.N.NNSignalLearning
ruleMomentumConjugate
gradientLevenberg
MarquardtQuick
propagationStepDelta- bar- delta1MLPNoise0.77860.766720.774140.773880.773650.76857EMG0.58400.572050.571820.568660.570260.571252FTLRNNNoise0.998430.996170.997810.998060.998420.99792EMG0.997310.995720.996840.997560.996950.997873RBFNoise0.773140.77370.773820.773350.773350.7724EMG0.570590.57120.571860.57180.571840.5711Tables14 and 15 display the
regression performance of NN models. It shows performance parameters, MSE
and r on training, cross validation, and test dataset for MLP NN, FTLRNN, and RBF
NN for noise and EMG signal. From the observation, it clear that for FTLRNN
model the lowest MSE and the highest correlation coefficient are obtained.
FTLRNN is the best neural network to remove noise from EMG signal.Table 14
Regression performance of NN models on EMG noise signal.
S.N.NN modelPerformance measuresTraining datasetCV datasetTest datasetMSErMSErMSEr1MLP0.026880.773220.030700.770210.025010.781142FTLRNN0.0002540.998090.002010.987010.0000670.999503RBF0.026640.775440.030360.773090.0247040.784138Table 15
Regression performance of NN models on EMG signal.
S.N.NN modelPerformance measuresTraining datasetCV datasetTest datasetMSErMSErMSEr1MLP0.0265370.573090.030410.568290.024850.584332FTLRNN0.000240.996920.002160.975840.0000480.999393RBF0.026540.571220.030370.569150.0248170.585088Table16 displays the comparison of
the MLP NN, FTLRNN, and RBF NN. For all the three NNs, the number of epochs is kept
1000. MSE for FTLRNN model is 0.0027 times less than that of MLP and RBF NN
models. Correlation coefficient for FTLRNN is 1.71 times higher than that of MLP
and RBF NN models. Percentage error for FTLRNN is minimum. It is 0.04 times and
0.034 times smaller than MLP and RBF NN, respectively. Time elapsed per epoch
per exemplar for FTLRNN is 0.73 times and 1.67 times to that of MLP and RBF NN,
respectively. As compared to RBF NN, FTLRNN model requires more time for
training but from MSE and r, and by visual
inspection of modeling characteristics, the FTLRNN model is definitely superior
to other two NNs.Table 16
Comparison of all the NN architectures on test dataset.
NN modelTransfer functionLearning ruleMean square error
MSECorrelation coefficientrEpochsTime elapsed/ epoch/exemplar in microseconds% errorMLP
(1-15-10-2)TanhMomentum0.02501 (noise)
0.02482(EMG)0.78114 (noise) 0.58433 (EMG)100019.16253FTLRNN
(1-27-2)LinearMomentum0.000067 (noise)
0.000048 (EMG)0.99950 (noise)
0.99939 (EMG)10001410RBF (CC-05)LinearLM0.02470 (noise) 0.02482 (EMG)0.78414 (noise) 0.58509 (EMG)10008.3293Effect of Noise on EMG Signal
The estimated MLP NN, FTLRNN, and RBF
NN are checked for their robustness by adding uniform and Gaussian noise in input
as well as in output of NNs. Figure16 portrays the performance
of NNs with uniform and Gaussian noise. Noise variance is varied from 0.01 to
0.4. In FTLRNN uniform noise tolerance for EMG signal 0.4-noise variance is
obtained whereas when Gaussian noise is introduced, the noise variance 0.3 is
detected. In MLP NN and RBF NN as noise variance is increased, the performance
parameters are reduced to very low values.Figure 16
Effect of noise on EMG signal.Learning Ability of FTLRNN on Different Data Partitions
The  learning of NN models for independent of datasets is tested. MLP, FTLRNN, and RBF NN models are trained on
various datasets as shown in Table1(a) (forward tagging and reverse tagging). Figure 17
displays the performance of these NN models for filtered EMG signal. Performance of
FTLRNN-based model is found to be almost the same for all the datasets as compared
to MLP and RBF NN models.Figure 17
Correlation coefficient with various datasets.Multifold Differential Learning
The total samples are divided into
four groups each containing 500 samples as described in Table1(b). Performance
of FTLRNN, MLP NN, and RBF NN models is displayed in Figure 18. It is observed that the performance FTLRNN-based
model is consistent.  It is also observed that correlation coefficient is the highest
for FTLRNN for all datasets.Figure 18
Performance of NNs on
differential learning.


## 6. Conclusion

EMG signal carries valuable
information regarding the nerve system. Noise removal in EMG signal using ANN
is studied in this paper. Authors demonstrate that FTLRNN-based filter
elegantly removes noise from the EMG signal. Compact FTLRNN with only one hidden
layer having architecture (1-27-2) is able to remove noise with reasonable
accuracy. When the performance of MLP and RBF neural network-based models is carefully
examined for dataset, FTLRNN based model has clearly outperformed its MLP NN
and RBF NN counterparts with respect to the performance measures such as MSE andr as well as the visual inspection of graphs of actual and desired output of
filtered EMG signal. For FTLRNN-based filter correlation coefficient is
obtained as high as 0.99939, and MSE is found to be as low as 0.000048 for
filtered EMG signal. Also for noise signal the correlation coefficient and MSE
are optimally found as 0.99950 and 0.000067, respectively. Moreover, the actual
output of the estimated FTLRNN model follows the desired output more closely
than that of other NN models. In case of learning ability of FTLRNN-based model,
the performance parameters are found consistent, and hence learning is almost
independent of specific partitioning of the dataset. It is also seen that the time elapsed per
epoch per exemplar required to train the network is considerably low for FTLRNN-based model. The least percentage error equal to 10% for FTLRNN on test dataset
is obtained. It is also observed that when uniform and Gaussian noise is
introduced in EMG signal, the network sustains reasonable level of noise. For uniform noise, 100% tolerance is observed, and
for Gaussian noise, it is 75%. This confirms the noise immunity of the
proposed FTLRNN-based model. The estimated FTLRNN is a robust network developed
to detect EMG signal from noisy EMG signal.Proposed FTLRNN-based model
with Laguarre memory is able to filter noise from a typical EMG signal
contaminated by noise.



---

*Source: 129761-2009-06-07.xml*