# An Approach to Applying Feedback Error Learning for Functional Electrical Stimulation Controller: Computer Simulation Tests of Wrist Joint Control

Authors: Takashi Watanabe; Keisuke Fukushima
Journal: Advances in Artificial Neural Systems
 (2010)

---

## Abstract

Feedback error-learning (FEL) controller that consists of a proportional-integral-derivative (PID) controller and an artificial neural network (ANN) had applicability to functional electrical stimulation (FES). Because of the integral (reset) windup, however, delay or overshoot sometimes occurred in feedback FES control, which was considered to cause inappropriate ANN learning and to limit the feasibility of the FEL controller for FES to controlling 1-DOF movements stimulating 2 muscles. In this paper, an FEL-FES controller was developed applying antireset windup (ARW) scheme that worked based on total controller output. The FEL-FES controller with the ARW was examined in controlling 2-DOF movements of the wrist joint stimulating 4 muscles through computer simulation. The developed FEL-FES controller was found to realize appropriately inverse dynamics model and to have a possibility of being used as an open-loop controller. The developed controller would be effective in multiple DOF movement control stimulating several muscles.


---

## Body

## 1. Introduction

A spinal cord injury or a stroke causes paralysis of human motor function. Even in those cases, if peripheral muscles and nerves maintain electrical excitability, electrical stimulation to them can activate the muscles, and therefore the motor function can be restored. This is the principle of “Functional Electrical Stimulation (FES)”. FES has been found to be effective clinically, especially in controlling paralyzed upper limbs [1–3].FES controller is required to determine appropriate stimulation data for restoring desired functional movements. However, FES control of limbs is very difficult subject because responses of electrically stimulated musculoskeletal system have strong nonlinearity, large latency, time variability, fatigue and so on. Redundancy in stimulation intensity determination also makes control more difficult because the number of stimulated muscles is generally larger than the number of controlled joint angles of limb movements.An effective way of FES movement control of highly nonlinear musculoskeletal system with large latency is to use a hybrid regulator with a feedforward and a feedback controller [4–7]. The hybrid controller has advantages of the both controllers: improvement of tracking control of fast movements by the feedforward controller and compensation for disturbance by the feedback controller. However, a lot of controllers were evaluated on simple movements such as flexion and extension stimulating one or two muscles. In addition, most hybrid controllers tested in the previous studies required separate control sessions for initial adjustment or learning of the feedforward controller before using it in FES control.The controller using feedback error-learning (FEL) scheme [8] is also a hybrid controller. The feedforward controller is realized by an artificial neural network (ANN) that learns the inverse dynamics of controlled object using outputs of the feedback controller as teacher signals during controlling online. The FEL controller can also be applied to control the redundant musculoskeletal systems if the feedback controller can solve an ill-posed problem in stimulation intensity determination. Therefore, an application of the FEL scheme to FES controller is expected to simplify and shorten clinical tasks for parameter adjustment of a feedforward controller, which reduces the burden of adjustments of feedforward controllers remained in the previous hybrid controllers.In our previous study, the FEL controller for FES (FEL-FES controller) was found to be feasible in controlling 1-DOF movements of the wrist joint stimulating 2 muscles through control tests with normal subjects and computer simulation tests [9]. In the previous FEL-FES controller, a proportional-integral-derivative (PID) controller was used as a feedback controller, which was designed to solve the ill-posed problem in stimulation intensity determination [10]. However, ANN in the previous FEL-FES controller was not trained appropriately in some cases, especially in controlling 2-DOF movements of the wrist joint stimulating 4 muscles. A possible reason of that was in the PID controller, which was the windup of the integral action of the PID controller since the musculoskeletal system has saturation characteristics. The integral (reset) windup causes large delay or overshoot in responses of the musculoskeletal system in feedback FES control. Inappropriate stimulation causing these undesirable responses in feedback control are considered to cause inappropriate learning of the ANN and to limit the feasibility of the FEL-FES controller to controlling 1-DOF movements stimulating 2 muscles.The purpose of this paper was to realize an FEL-FES controller that could solve the reset windup problem. An antireset windup (ARW) algorithm was applied to the FEL-FES controller, in which the ARW worked based on the total output of the FEL-FES controller. The developed FEL-FES controller with the ARW algorithm was examined through computer simulation using musculoskeletal model with delay in responses to electrical stimulation. First, the controller was compared in controlling 2-DOF movements of the wrist stimulating 4 muscles before ANN learning with the previous controller and previous experimental tests with normal subjects. Then, the FEL-FES controller was examined in ANN learning of the inverse dynamics of the complicated 2-DOF movement control evaluating on open-loop control.


## 2. Feedback Error-Learning Controller for FES

Figure1(a) shows the block diagram of the FEL-FES controller used in this study. The controller consists of a PID controller and an ANN that realizes an inverse dynamics model (IDM) of the controlled limb [9]. The sum of stimulation outputs from the PID controller (IPID) and the IDM (IANN) is applied to each muscle after adding offset (threshold value of electrical stimulation intensity) and clipping out with the limiter to prevent excessive stimulation.Block diagram of the FEL controller for FES (a) and structure of ANN used in the FEL-FES controller (b).θd and θ represent the desired and the measured joint angles. IPID and IANN are outputs of the PID controller and the ANN. The ANN realizes the IDM of the controlled limb after learning.
(a)(b)


### 2.1. Artificial Neural Network

A three-layered ANN shown in Figure1(b) was used as a feedforward controller. The inputs of the ANN were time series of angles, angular velocities and angular accelerations of target movements at continuous 6 times, from n to n+5 (50 ms interval). In this paper, tracking control was performed on 2-DOF movements of the wrist joint (dorsiflexion/palmar flexion and radial flexion/ulnar flexion) as shown in Figure 2, in which muscles to be stimulated were extensor carpi radialis longus/bravis (ECRL/B, which are expressed by “ECR” in this paper since the same stimulation was applied to them), extensor carpi ulnaris (ECU), flexor carpi radialis (FCR) and flexor carpi ulnaris (FCU). Therefore, the total number of neurons of the input and output layers were 36 and 4, respectively. The number of neuron of the hidden layer was 36. In Figure 1(b), input and output of the ANN are shown by, θd1 and θd2 meaning dorsiflexion/palmar flexion and radial flexion/ulnar flexion, and IANN1~IANN4 for 4 stimulated muscles, respectively.Controlled joint movements.
(a)(b)


### 2.2. Feedback Controller

Outputs of the PID controller were described by(1)IPID(n)=KPe(n)+KI∑i=0ne(i)+KD{e(n)-e(n-1)}.
Here, e(n) is an error vector showing the difference between target and measured joint angles at time n. KP, KI and KD are PID parameter matrices. To solve the ill-posed problem in closed-loop control, the generalized inverse matrix of the matrix which transforms stimulus intensities into joint angles was calculated and applied to determine parameter values of the PID controller expanding the Chien-Hrones-Reswick (CHR) method [10].In the previous studies, a common PID controller shown by (1) was used in FEL-FES controller, although negative values of the controller output are ineffective for FES control. The negative stimulation intensities of the PID controller are necessary for error signals in learning of the IDM, while positive stimulation intensities can activate a muscle. The PID controller is required to output both positive and negative values of stimulation intensities for the FEL.


### 2.3. Antireset Windup Algorithm for FEL-FES Controller

The PID controller shown in the previous section, however, has a problem in controlling the system that has the saturation property. For example, in the FES control, negative outputs of the PID controller are ineffective and therefore cause integration wind up of the PID controller (reset windup). In order to avoid the integration windup, antireset windup (ARW) scheme can be applied directly to the PID controller. However, such method of applying the ARW limits the output of the PID controller to positive values less than the maximum intensity, which eliminates negative outputs of the PID controller in ANN learning process. Negative outputs of the PID controller are necessary in the ANN training when ANN output should be decreased. Therefore, in this study, the function of the integral term of the PID controller was suspended when the total output of the FEL-FES controller (sum of the outputs of the ANN and the PID controller) saturated. The ARW used in this paper is considered to be able to suppress ineffective large decreasing or increasing of PID controller outputs when a controlled object has saturation property, which is expected to improve delay and/or overshoot in responses of feedback control.


## 2.1. Artificial Neural Network

A three-layered ANN shown in Figure1(b) was used as a feedforward controller. The inputs of the ANN were time series of angles, angular velocities and angular accelerations of target movements at continuous 6 times, from n to n+5 (50 ms interval). In this paper, tracking control was performed on 2-DOF movements of the wrist joint (dorsiflexion/palmar flexion and radial flexion/ulnar flexion) as shown in Figure 2, in which muscles to be stimulated were extensor carpi radialis longus/bravis (ECRL/B, which are expressed by “ECR” in this paper since the same stimulation was applied to them), extensor carpi ulnaris (ECU), flexor carpi radialis (FCR) and flexor carpi ulnaris (FCU). Therefore, the total number of neurons of the input and output layers were 36 and 4, respectively. The number of neuron of the hidden layer was 36. In Figure 1(b), input and output of the ANN are shown by, θd1 and θd2 meaning dorsiflexion/palmar flexion and radial flexion/ulnar flexion, and IANN1~IANN4 for 4 stimulated muscles, respectively.Controlled joint movements.
(a)(b)


## 2.2. Feedback Controller

Outputs of the PID controller were described by(1)IPID(n)=KPe(n)+KI∑i=0ne(i)+KD{e(n)-e(n-1)}.
Here, e(n) is an error vector showing the difference between target and measured joint angles at time n. KP, KI and KD are PID parameter matrices. To solve the ill-posed problem in closed-loop control, the generalized inverse matrix of the matrix which transforms stimulus intensities into joint angles was calculated and applied to determine parameter values of the PID controller expanding the Chien-Hrones-Reswick (CHR) method [10].In the previous studies, a common PID controller shown by (1) was used in FEL-FES controller, although negative values of the controller output are ineffective for FES control. The negative stimulation intensities of the PID controller are necessary for error signals in learning of the IDM, while positive stimulation intensities can activate a muscle. The PID controller is required to output both positive and negative values of stimulation intensities for the FEL.


## 2.3. Antireset Windup Algorithm for FEL-FES Controller

The PID controller shown in the previous section, however, has a problem in controlling the system that has the saturation property. For example, in the FES control, negative outputs of the PID controller are ineffective and therefore cause integration wind up of the PID controller (reset windup). In order to avoid the integration windup, antireset windup (ARW) scheme can be applied directly to the PID controller. However, such method of applying the ARW limits the output of the PID controller to positive values less than the maximum intensity, which eliminates negative outputs of the PID controller in ANN learning process. Negative outputs of the PID controller are necessary in the ANN training when ANN output should be decreased. Therefore, in this study, the function of the integral term of the PID controller was suspended when the total output of the FEL-FES controller (sum of the outputs of the ANN and the PID controller) saturated. The ARW used in this paper is considered to be able to suppress ineffective large decreasing or increasing of PID controller outputs when a controlled object has saturation property, which is expected to improve delay and/or overshoot in responses of feedback control.


## 3. Computer Simulation Tests

### 3.1. Evaluation Method

Learning and control performance were examined by computer simulation with musculoskeletal model. The target trajectories of tracking control were 5 ellipse trajectories on the joint angle plane that is defined by the angle of radial flexion/ulnar flexion, and the angle of dorsiflexion/palmar flexion. The center of ellipse trajectory was 0 deg, or 5 deg of dorsiflexion, palmar flexion, radial flexion or ulnar flexion as shown in Figure3. The major axis of the ellipse was 20 deg for dorsiflexion/palmar flexion and the minor axis was 15 deg for radial flexion/ulnar flexion. The cycle period was also varied in 3 s, 6 s and 10 s. The wrist joint was held at the position of palmar flexion before the tracking control and the control started with the radial flexion. Six cycles were included in one control trial.Figure 3
Target joint angle trajectories on the joint angle plane for ANN learning. Small closed circle shows start point of tracking control on the target trajectory.The ANN connection weights were updated after one control trial with the error back-propagation algorithm. Three initial patterns of the ANN connection weights were prepared, in which those values were determined randomly: −0.25~0.25 for the weights between input and hidden layers, and −0.5~0.0 between hidden and output layers. Values of learning coefficient were 10.0 for the weights between input and hidden layers, and for those between hidden and output layers, 0.1, 0.05 and 0.03 were used for cycle period of 3 s, 6 s and 10 s, respectively. Iteration number of training was fixed at 50. Six different subject models were prepared, whose parameters were adjusted to approximate the muscle properties of 6 normal subjects. The total number of simulation conditions was 45 for each subject model.ANN output power increases as the learning progresses and then the ANN realizes the IDM of the controlled limb after the learning. To evaluate performance of the controller, mean error (ME) and power ratio (PR) were calculated:(2)ME=1N∑n=1N|e(n)|,PR=∑n=1NPANN(n)∑n=1NPPID(n)+∑n=1NPANN(n)×100[%].
Here, e(n) represents the error between target joint angle and desired one. N represents the number of sampled data. PANN(n) and PPID(n) represent the square value of stimulation intensity (output power) of the ANN and the PID controller, respectively. PR is the proportion of ANN output to the total controller output, which shows large value if the ANN learns the inverse dynamics appropriately. ME was calculated for each movement direction and PR was done for each muscle.


### 3.2. Musculoskeletal Model for FES Control

Movements and muscles included in computer simulation are summarized in Table1. The musculoskeletal model to predict movements developed by nonlinear responses of electrically stimulated muscle is outlined in Figure 4. Muscle force produced by electrical stimulation (active force), FA, was described by the Hill type muscle model, which consisted of muscle activation level determined by electrical stimulation am, length-force relationship k(l), velocity-force relationship h(v), and maximum muscle force Fmax. That is,(3)FA=amk(l)h(v)Fmax,
where l and v were muscle length and contraction velocity, respectively. FA shows the force produced by contracted muscle fibers during electrical stimulation, which actively contributes to develop movements. Active torque τA produced by electrical stimulation was calculated by(4)τA=FArf(θ).
Moment arm rf(θ) was modeled by an polynomial equation as a function of joint angle θ for each movement developed by each muscle [11].Table 1
Movements and muscles included in computer simulation.
jointmovementmuscleelbowflexionBBL, BBS, Br, BrR, and  FCR,ECRL, ECRB, and  PTextensionTrBLo, TrBM, TrBLa, and  ECUforearmpronationPQ and PTsupinationBCLH, BCSH, Su, and  BRRwristpalmar flexionFCR and  FCUdorsi flexionECRL, ECRB, and  ECUradial flexionECRL, ECRB, and  FCRulnar flexionECU and  FCUBBL: biceps brachii long head, BBS: biceps brachii short head, Br: brachialis, BrR: brachioradialis, TrBLo: triceps brachii long head, TrBM: triceps brachiimedial head, TrBLa: triceps brachii lateral head, PQ: pronator quadratus, PT: pronator teres, Su: supinator, FCR: flexor carpi radialis, FCU: flexor carpi ulnaris, ECRL:  extensor carpi radialis longus, ECRB: extensor carpi radialis brevis, ECU: extensor carpi ulnaris.Figure 4
Outline of the musculoskeletal model for FES.The recruitment property of electrically stimulated muscleu(I) was modeled by the following [12]:(5)u(I)=sctanh{sh(I(t)-xc)}+yc,
where sc, sh, xc, and yc were constants. Electrical stimulation was expressed by stimulation intensity I(t), which was the sum of IANN, IPID and offset in this paper. IANN and IPID were normalized stimulation intensity determined by the ANN and the PID controller shown in Figure 1(a), respectively, which had values between 0 and 1. The muscle activation am was described by the following dynamics using the recruitment property with different two time constants, tr and tf  [13]:(6)damdt=1tr{u(t-L)-am}u(t-L)+1tf{u(t-L)-am}.
Here, L shows delay in response to electrical stimulation. The value of L was set to 50 ms in order to make approximately latency of 100 ms and time constant of 300 ms when the step response was represented by the first-order delay with latency. These values relating to delay were determined by step responses measured with healthy subjects.The length-force relationshipk(l) was described by (7) [14]. lo means optimum muscle length whose value was determined from [11](7)k(l)=1-(l-lo0.5lo)2.The velocity-force relationshiph(v) was modeled with maximum contraction velocity vmax [14, 15]. Parameter value of vmax was determined as 10×lo from [16](8)h(v)=vmax-vvmax+2.5v(v≤0:shortening),h(v)=1.3-0.3vmax+2.5vvmax-2.52v(v>0:lengthening).The maximum muscle force produced by electrical stimulationFmax was determined by PCSA (physiological cross sectional area) as follows [11]:(9)Fmax=2.2PCSA.
Value of PCSA was determined from [11].The passive viscoelastic element developed passive torqueτP calculated by (10) for each joint movement [17]. The range of motion was also represented by(10)τP=k0θ+b0θ̇+k1{exp(k2θ)-1},
where θ and θ̇ were joint angle and angular velocity, respectively. Constants k0, b0, k1, and k2 were determined for each joint movement.


### 3.3. Results

An example of control results by the PID controller is shown in Figure5. By using the ARW scheme, feedback FES control was improved reducing delay and inappropriate responses. In the case of previous controller without the ARW, the PID controller output large negative values, while those of controller with the ARW were almost positive as seen in Figure 5(b).An example of control results of the FEL-FES controller before ANN learning. Since outputs of the ANN was almost 0, the control was performed only by PID controller. Three of six cycles in one control trial were shown. (model A, Palmar, cycle period of 3 s). Values of ME were 8.4 deg for palmar flexion/dorsiflexion and 6.6 deg for radial flexion/ulnar flexion in (a), and 3.4 deg and 4.9 deg in (b), respectively.
(a)
Without the ARW(b)
With the ARWFigure6 shows ME in the control by the feedback controller before ANN learning. The ARW applied to the FEL-FES controller worked appropriately decreasing ME with all subject models for different movement velocities. For reference, control results from 3 normal subjects [10] were also shown in Figure 6. In experiments with the subjects, 5 s of cycle period was used. In the computer simulation tests, however, 6 s of cycle period was used instead of 5 s, because of making clear difference between cycle periods. In the control tests with the subjects, the PID controller was modified to prevent the reset windup problem. That is, the output of the PID controller was simply limited to the values between the stimulation thresholds and the maximum intensities, which means only positive values were used in the control. Although the ARW algorithm used in the FEL-FES controller was not the same as the controller used with subjects, values of ME were almost same between them.Figure 6
Mean error (ME) in feedback control before ANN learning. Average and SD are shown. For normal subjects, 9 control results (3 trials with 3 subjects) were averaged for each cycle period, and result with 5 s of cycle period was shown for comparison in 6 s of cycle period.Values of PR after ANN learning is shown in Figure7. Both average and minimum values of PR increased clearly by applying ARW scheme, which shows improvement of the ANN learning. Especially, the minimum values of the PR increased from 0.0% to 68.6% or more. Even in the cases of 0% of PR with the previous controller, the values of PR were not 0% in the process of the ANN learning. That is, after the 50th learning trial, the ANN output converged 0 because of large negative outputs of the previous PID controller. Thus, the cases that the ANN could hardly learn at all in the previous FEL-FES controller were improved by applying the ARW.Figure 7
Power Ratio (PR) after learning of the IDM. Average, the minimum and the maximum values are shown.For comparing values of ME, open-loop control test with the ANN after learning was carried out, removing compensating effects of the PID controller. The results of ME in the open-loop control shown in Figure8 were smaller than or almost same as values of ME by feedback controller shown in Figure 6. Therefore, it is suggested from this result that the trained ANN can be used as an open-loop controller. In controlling faster movements (cycle period of 3 s), values of ME by the controller with the ARW were smaller than those without the ARW. However, in controlling slow movements (cycle periods of 6 s and 10 s), the ME values were almost same between with and without the ARW.Figure 8
Mean error (ME) in open-loop control with the trained ANN. Average and SD are shown.An example of open-loop control results after the ANN learning is shown in Figure9. The results were obtained under the same simulation condition as that of Figure 5. Delay in the response seen in Figure 5(b) was decreased by the ARW scheme as seen in Figure 9(b). The open-loop control was performed appropriately, since the ANN learning was improved by the ARW scheme.An example of open-loop control with the trained ANN. Joint angles and ANN outputs are shown. Three of six cycles in one control trial were shown. (model A, Palmar, cycle period of 3 s). Values of ME were 7.1 deg for palmar flexion/dorsiflexion and 4.8 deg for radial flexion/ulnar flexion in (a), and 2.1 deg and 2.0 deg in (b), respectively.
(a)
Without the ARW(b)
With the ARW


## 3.1. Evaluation Method

Learning and control performance were examined by computer simulation with musculoskeletal model. The target trajectories of tracking control were 5 ellipse trajectories on the joint angle plane that is defined by the angle of radial flexion/ulnar flexion, and the angle of dorsiflexion/palmar flexion. The center of ellipse trajectory was 0 deg, or 5 deg of dorsiflexion, palmar flexion, radial flexion or ulnar flexion as shown in Figure3. The major axis of the ellipse was 20 deg for dorsiflexion/palmar flexion and the minor axis was 15 deg for radial flexion/ulnar flexion. The cycle period was also varied in 3 s, 6 s and 10 s. The wrist joint was held at the position of palmar flexion before the tracking control and the control started with the radial flexion. Six cycles were included in one control trial.Figure 3
Target joint angle trajectories on the joint angle plane for ANN learning. Small closed circle shows start point of tracking control on the target trajectory.The ANN connection weights were updated after one control trial with the error back-propagation algorithm. Three initial patterns of the ANN connection weights were prepared, in which those values were determined randomly: −0.25~0.25 for the weights between input and hidden layers, and −0.5~0.0 between hidden and output layers. Values of learning coefficient were 10.0 for the weights between input and hidden layers, and for those between hidden and output layers, 0.1, 0.05 and 0.03 were used for cycle period of 3 s, 6 s and 10 s, respectively. Iteration number of training was fixed at 50. Six different subject models were prepared, whose parameters were adjusted to approximate the muscle properties of 6 normal subjects. The total number of simulation conditions was 45 for each subject model.ANN output power increases as the learning progresses and then the ANN realizes the IDM of the controlled limb after the learning. To evaluate performance of the controller, mean error (ME) and power ratio (PR) were calculated:(2)ME=1N∑n=1N|e(n)|,PR=∑n=1NPANN(n)∑n=1NPPID(n)+∑n=1NPANN(n)×100[%].
Here, e(n) represents the error between target joint angle and desired one. N represents the number of sampled data. PANN(n) and PPID(n) represent the square value of stimulation intensity (output power) of the ANN and the PID controller, respectively. PR is the proportion of ANN output to the total controller output, which shows large value if the ANN learns the inverse dynamics appropriately. ME was calculated for each movement direction and PR was done for each muscle.


## 3.2. Musculoskeletal Model for FES Control

Movements and muscles included in computer simulation are summarized in Table1. The musculoskeletal model to predict movements developed by nonlinear responses of electrically stimulated muscle is outlined in Figure 4. Muscle force produced by electrical stimulation (active force), FA, was described by the Hill type muscle model, which consisted of muscle activation level determined by electrical stimulation am, length-force relationship k(l), velocity-force relationship h(v), and maximum muscle force Fmax. That is,(3)FA=amk(l)h(v)Fmax,
where l and v were muscle length and contraction velocity, respectively. FA shows the force produced by contracted muscle fibers during electrical stimulation, which actively contributes to develop movements. Active torque τA produced by electrical stimulation was calculated by(4)τA=FArf(θ).
Moment arm rf(θ) was modeled by an polynomial equation as a function of joint angle θ for each movement developed by each muscle [11].Table 1
Movements and muscles included in computer simulation.
jointmovementmuscleelbowflexionBBL, BBS, Br, BrR, and  FCR,ECRL, ECRB, and  PTextensionTrBLo, TrBM, TrBLa, and  ECUforearmpronationPQ and PTsupinationBCLH, BCSH, Su, and  BRRwristpalmar flexionFCR and  FCUdorsi flexionECRL, ECRB, and  ECUradial flexionECRL, ECRB, and  FCRulnar flexionECU and  FCUBBL: biceps brachii long head, BBS: biceps brachii short head, Br: brachialis, BrR: brachioradialis, TrBLo: triceps brachii long head, TrBM: triceps brachiimedial head, TrBLa: triceps brachii lateral head, PQ: pronator quadratus, PT: pronator teres, Su: supinator, FCR: flexor carpi radialis, FCU: flexor carpi ulnaris, ECRL:  extensor carpi radialis longus, ECRB: extensor carpi radialis brevis, ECU: extensor carpi ulnaris.Figure 4
Outline of the musculoskeletal model for FES.The recruitment property of electrically stimulated muscleu(I) was modeled by the following [12]:(5)u(I)=sctanh{sh(I(t)-xc)}+yc,
where sc, sh, xc, and yc were constants. Electrical stimulation was expressed by stimulation intensity I(t), which was the sum of IANN, IPID and offset in this paper. IANN and IPID were normalized stimulation intensity determined by the ANN and the PID controller shown in Figure 1(a), respectively, which had values between 0 and 1. The muscle activation am was described by the following dynamics using the recruitment property with different two time constants, tr and tf  [13]:(6)damdt=1tr{u(t-L)-am}u(t-L)+1tf{u(t-L)-am}.
Here, L shows delay in response to electrical stimulation. The value of L was set to 50 ms in order to make approximately latency of 100 ms and time constant of 300 ms when the step response was represented by the first-order delay with latency. These values relating to delay were determined by step responses measured with healthy subjects.The length-force relationshipk(l) was described by (7) [14]. lo means optimum muscle length whose value was determined from [11](7)k(l)=1-(l-lo0.5lo)2.The velocity-force relationshiph(v) was modeled with maximum contraction velocity vmax [14, 15]. Parameter value of vmax was determined as 10×lo from [16](8)h(v)=vmax-vvmax+2.5v(v≤0:shortening),h(v)=1.3-0.3vmax+2.5vvmax-2.52v(v>0:lengthening).The maximum muscle force produced by electrical stimulationFmax was determined by PCSA (physiological cross sectional area) as follows [11]:(9)Fmax=2.2PCSA.
Value of PCSA was determined from [11].The passive viscoelastic element developed passive torqueτP calculated by (10) for each joint movement [17]. The range of motion was also represented by(10)τP=k0θ+b0θ̇+k1{exp(k2θ)-1},
where θ and θ̇ were joint angle and angular velocity, respectively. Constants k0, b0, k1, and k2 were determined for each joint movement.


## 3.3. Results

An example of control results by the PID controller is shown in Figure5. By using the ARW scheme, feedback FES control was improved reducing delay and inappropriate responses. In the case of previous controller without the ARW, the PID controller output large negative values, while those of controller with the ARW were almost positive as seen in Figure 5(b).An example of control results of the FEL-FES controller before ANN learning. Since outputs of the ANN was almost 0, the control was performed only by PID controller. Three of six cycles in one control trial were shown. (model A, Palmar, cycle period of 3 s). Values of ME were 8.4 deg for palmar flexion/dorsiflexion and 6.6 deg for radial flexion/ulnar flexion in (a), and 3.4 deg and 4.9 deg in (b), respectively.
(a)
Without the ARW(b)
With the ARWFigure6 shows ME in the control by the feedback controller before ANN learning. The ARW applied to the FEL-FES controller worked appropriately decreasing ME with all subject models for different movement velocities. For reference, control results from 3 normal subjects [10] were also shown in Figure 6. In experiments with the subjects, 5 s of cycle period was used. In the computer simulation tests, however, 6 s of cycle period was used instead of 5 s, because of making clear difference between cycle periods. In the control tests with the subjects, the PID controller was modified to prevent the reset windup problem. That is, the output of the PID controller was simply limited to the values between the stimulation thresholds and the maximum intensities, which means only positive values were used in the control. Although the ARW algorithm used in the FEL-FES controller was not the same as the controller used with subjects, values of ME were almost same between them.Figure 6
Mean error (ME) in feedback control before ANN learning. Average and SD are shown. For normal subjects, 9 control results (3 trials with 3 subjects) were averaged for each cycle period, and result with 5 s of cycle period was shown for comparison in 6 s of cycle period.Values of PR after ANN learning is shown in Figure7. Both average and minimum values of PR increased clearly by applying ARW scheme, which shows improvement of the ANN learning. Especially, the minimum values of the PR increased from 0.0% to 68.6% or more. Even in the cases of 0% of PR with the previous controller, the values of PR were not 0% in the process of the ANN learning. That is, after the 50th learning trial, the ANN output converged 0 because of large negative outputs of the previous PID controller. Thus, the cases that the ANN could hardly learn at all in the previous FEL-FES controller were improved by applying the ARW.Figure 7
Power Ratio (PR) after learning of the IDM. Average, the minimum and the maximum values are shown.For comparing values of ME, open-loop control test with the ANN after learning was carried out, removing compensating effects of the PID controller. The results of ME in the open-loop control shown in Figure8 were smaller than or almost same as values of ME by feedback controller shown in Figure 6. Therefore, it is suggested from this result that the trained ANN can be used as an open-loop controller. In controlling faster movements (cycle period of 3 s), values of ME by the controller with the ARW were smaller than those without the ARW. However, in controlling slow movements (cycle periods of 6 s and 10 s), the ME values were almost same between with and without the ARW.Figure 8
Mean error (ME) in open-loop control with the trained ANN. Average and SD are shown.An example of open-loop control results after the ANN learning is shown in Figure9. The results were obtained under the same simulation condition as that of Figure 5. Delay in the response seen in Figure 5(b) was decreased by the ARW scheme as seen in Figure 9(b). The open-loop control was performed appropriately, since the ANN learning was improved by the ARW scheme.An example of open-loop control with the trained ANN. Joint angles and ANN outputs are shown. Three of six cycles in one control trial were shown. (model A, Palmar, cycle period of 3 s). Values of ME were 7.1 deg for palmar flexion/dorsiflexion and 4.8 deg for radial flexion/ulnar flexion in (a), and 2.1 deg and 2.0 deg in (b), respectively.
(a)
Without the ARW(b)
With the ARW


## 4. Discussion

The ARW was applied to the FEL-FES controller considering saturation of its total output. The ARW worked effectively in the feedback FES control and also improved the ANN learning by the FEL. In the feedback control, values of ME were almost same as the results of control with normal subjects. The method of preventing the reset windup used in the controller tested with normal subjects [10] cannot be applied in the FEL-FES controller because outputs of the previous PID controller were limited to positive values. Therefore, the different ARW was applied to the FEL-FES controller in this paper.Values of ME in open-loop control for movements with the cycle period of 3 s decreased significantly by applying the ARW. On the other hand, in controlling slow movements, the values of ME were almost same between with and without the ARW. Since those values of ME were similar to those by the feedback controller, the previous type controller is suggested to have sufficient ability to control slow movements. However, as seen in results of PR shown in Figure7, there were some cases of highly-inappropriate ANN learning. Because the control was also inappropriate in such cases, the FEL-FES controller using the ARW scheme would be effective to provide stable control performance. From the results of Figures 5 and 9, mean error less than 2 or 3 deg is seemed to be desirable for a control performance, although it may be changed between control movements or by results from control tests with subjects.Values of PR increased significantly after ANN learning by applying the ARW scheme. However, ME was not always smaller than the previous controller for slow movements. As seen in Figure9, stimulation intensities of all muscles increased by using the ARW, which were generally larger than those without the ARW. In such cases, movements were considered to be controlled under the condition of high stiffness. The stiffness control has not been realized in the current FEL-FES controller, although it will be possible by modifying the PID controller. Stiffness control is desired for more appropriate control.In this paper, the FEL-FES controller using the ARW scheme was found to realize appropriately inverse dynamics model of the electrically stimulated musculoskeletal system by ANN learning in controlling 2-DOF movements stimulating 4 muscles. Target angle trajectories, however, were similar patterns for all movements although both of the center of the trajectory and movement speed were changed. Various target trajectories have to be tested considering practical application.The improvement of ANN learning has crucially important meaning in FES control. It is difficult to realize appropriate control by only feedback controller, because characteristics of electrically stimulated muscles of paralyzed patients usually show strong nonlinearity and large difference between patients. Therefore, a learning controller such as the FEL-FES controller is expected to be an effective controller for the musculoskeletal system that has various nonlinear characteristics with large delay. Since the IDM can be realized appropriately by the developed FEL-FES controller, it is possible to use the IDM as a feedforward controller removing sensors for feedback control in addition to using it in a hybrid controller.


## 5. Conclusions

In this paper, the feedback error-learning (FEL) controller for FES with antireset windup (ARW) scheme was developed and examined. The ARW worked based on the total output of the FEL-FES controller. The computer simulation tests of controlling 2-DOF movements of the wrist joint showed that the FEL-FES controller with the ARW decreased error in tracking control before ANN learning and realized appropriately inverse dynamics model after ANN learning. The realized IDM was also found to have a possibility to be used as an open-loop controller. The EFL-FES controller with the ARW developed in this study was found to be effective in multiple DOF movement control stimulating several muscles. It is expected to test the FEL-FES controller with more practical target trajectories and to study stiffness control for getting practical to restore motor function of paralyzed patients.



---

*Source: 814702-2010-11-08.xml*