Journal of Robotics, Networking and Artificial Life, Vol. 1, No. 2 (September 2014), 116-119

Development of Measurement System for Quantitative Evaluation ofSkillfulness of Lower ExtremitiesKazunori Yamazaki *Fujita Health University,
1-98 Dengakugakubo Kutsukake-cho, Toyoake, Aichi, 470-1192, Japan
Dongjun Jin, Yoshifumi Morita, Noritaka Sato, Hiroyuki Ukai

Nagoya Institute of Technology,
Gokiso-cho, Showa-ku, Nagoya, Aichi, 466-8555, JapanKenji Kozakai, Satoru Shibata, Shigenori Onishi, Akihiro Ito, Daisuke Mizuno

Sanyo Machine Works, Ltd.,
1 Oka Okimura, Kitanagoya, Aichi, 481-8540, Japan
Tatsuya Hirai, Haruna Takeda, Yuka Sugiura

Nishio Hospital,
22 Izumi-cho, Nishio, Aichi, 445-0824, Japan*Corresponding author. E-mail: ymzkk@fujita-hu.ac.jp
Tel: +81-562-93-9216

Abstract
The final goal of our research is to develop a measurement system for quantitative evaluation of the skillfulness ofthe lower extremities. In our previous work, we developed the measurement system of the foot movement on thefloor with the sitting posture. In this study, we developed a quantitative evaluation method of the skillfulness of thelower extremities. We verified the reliability of the proposed quantitative evaluation method by calculating theintra-rater reliability for two trials ICC(1,1) of the evaluation results.
Keywords: Rehabilitation system, Quantitative evaluation, Intra-rater reliability, Lower extremities, Skillfulness

1. Introduction
Sensorimotor integration function and proprioceptive
control in the lower extremities are necessary forwalking and skillful movement. The sensorimotor
integration function and proprioceptive control ofelderly and other types of patients can become impaired.

On the other hand, evaluation of therapeutic effects isonly done subjectively by experienced doctors ortherapists. Quantitative evaluation of therapeutic effects
has not been established. Therefore, there is a strong
need for quantitative evaluation of therapeutic effects inclinical rehabilitation sites. Moreover, an effective
training method for improving walking ability andPublished by Atlantis Press
Copyright: the authors
116

skillful movement is imperative.
Thus, the final goal of our research is to develop ameasurement system and robot system for quantitativeevaluation of the skillfulness of the lower extremities,
which has long been desired in clinical rehabilitationsites. The skillfulness of the lower extremities is theability to adjust the posture and motion of the body toachieve a desired action.
We think that the skillfulness can be evaluated using theaccuracy and speed of the motion when a subject’s footis moved to follow a given target trajectory. In thisstudy, we developed a measurement system comprising
three target trajectory courses with different difficultylevels, to evaluate the skillfulness of the lowerextremities.
The quantitative evaluation results should also havehigh reliability. This reliability can be divided into intrarater reliability and inter-rater reliability [1]. Intra-rater
reliability is the degree of agreement among multiple
repetitions of a diagnostic test performed by a single
rater. Inter-rater reliability is the degree of agreement
among raters. The intra-class correlation (ICC) has been
widely used to evaluate the intra-rater reliability inrehabilitation medicine.
In this study, we developed a quantitative evaluation
method to assess the skillfulness of the lower
extremities. For this purpose, we used the measurement
system we developed in our previous work andredesigned the target trajectories of foot movement.
Moreover, we verified the reliability of the proposed
quantitative evaluation method by calculating the intrarater reliability for two trials ICC(1,1).
2. Measurement system for quantitativeevaluation of skillfulness of lower extremities

In the training, the subject moves the pointer displayedon the monitor by moving the training device in thesitting posture. During this training, the subject watches
the monitor of the laptop PC and does not watch theirlower extremities directly. The subject can perform thetraining by playing a courses displayed on the laptop PC
monitor. The therapist can evaluate the motion using thedata acquired during the training.
There is little risk of the patient overturning, because
this system is operated in the sitting posture. Moreover,there are no dangers associated with a runaway training
device because the system does not have any actuators.
Therefore, this system can be easily introduced in theearly stages of rehabilitation because of the high safetyof this system.
2.2. Software
We developed three courses as target trajectories: astraight course, circular course, and star-shaped course,
as shown in Fig. 2. The parameters, such as the lengthof the courses were designed so that the difficulties of
the three courses were different. A depiction of thesoftware is given in Fig. 3.
The subject moves the pointer displayed on the monitor
quickly and accurately along the target trajectory. The
results of the assessment of the skillfulness of the lower
extremities are expressed in terms of speed andaccuracy. The speed t [s] is given by the travelling timefor each course. The accuracy S [pixel] is the area that issurrounded by the pointer trajectory and the targettrajectory.
The extension of the knee joint is primarily used tofollow the target trajectory in the straight course. On theother hand, in the circular and star-shaped courses, thecomplex motion that is a mixture of the extension of the2.1. Hardware and operation procedure
We have developed the measurement system of the footmovement on the floor with the sitting posture, asshown in Fig. 1. The hardware consists of a laptop PC,
training device, and training board [2]. A PC wirelessmouse is installed in the training device. The mouse isused as a low-cost position sensor. The pointercorresponding to the position of the training device isdisplayed on the monitor. We designed the system sothat the motion of the pointer on the monitor and themotion of the foot on the training board would be thesame size.

Laptop PC（Video games）
Board
Device(PC mouse)

Fig. 1. Measurement system of the foot movement on thefloor with the sitting posture.

Published by Atlantis Press
Copyright: the authors
117knee joint and the internal rotation of the hip joint isnecessary. In addition, in the star-shaped course, moretime is required for switching the direction of themotion than in the circular course. Therefore, eachcourse is designed to have a different difficulty.
When the software starts, it will display a control panelon the monitor. In the control panel, the therapist inputs
the subject’s information (user name, ID, height, weight,Body Mass Index (BMI), dominant foot, remarks) andthe location on HDD to save the data as a CSV file, atfirst. If this information is not input, measurement datais only saved to the desktop of the PC. At each sampling
point, the X & Y coordinates of the pointer aremeasured and saved. The sampling frequency is 100 Hz.
After inputting the subject’s information and thelocation to which on HDD, the therapist selects thecourse.
To check the results during/after the trials, the systemshows the spent time from the start of each trial and thetime that is counted up when the pointer is off thecourse, as the feedback to the subject. At this point,
information can be eliminated from the control panel, if
the therapist wants. In addition, the system can display
the pointer trajectory overlaid on the target trajectory

after the training, as shown in Fig. 4(a). The system canalso display the area that is surrounded by the pointertrajectory and the target trajectory, as shown in Fig. 4(b).
3. Verifying the intra-rater reliability
3.1. Method
To verify the intra-rater reliability of the measurement
system, we performed some experiments. The subjects
were fourteen healthy young persons. The age was
22±2.4 years old. The height was 166±8.3 cm. The
dominant foot of all subjects was right.
The subject was seated so that the distance between thesubject's eyes and the PC monitor was 0.7 m. The
training device was attached to the subject's left foot.
The height of the seat surface was adjusted so that theknee joint angle was 90 degree in the sitting position.
The subjects performed tasks consisting of the threecourses. The subjects were instructed to move the leftfoot along the black line (the desired trajectory) asDevice position (pointer) Number of trials

Course button

AnalysisbuttonElapsed time
Evaluation
value

Targettrajectory

Userinformation

CourseStart position Goal position Control panel

Fig. 3. Explanatory drawing of the user interface.
(a) Straight course(b) Circular course(a) Pointer trajectory and target trajectory(c) Star-shaped course(b) Area surrounded by the pointer trajectory and the target trajectory

Fig. 2. Three courses for evaluation.

Fig. 4. Evaluation results.

Published by Atlantis Press
Copyright: the authors
118

quickly and precisely as possible, not to move the body,and to move the left foot smoothly on the training board.
The subjects performed the tasks two times for eachcourse. After finishing each task, the result was not
informed of the subject.
We used data from the three courses, with two
repetitions for each person. The ICC(1,1) of the speed t
and the accuracy S were calculated using the Eq. (1)．
ICC (1,1) =BMS − WMS
BMS + (k − 1)WMSN∑ k ( x − x)i

BMS =i =1

N −1(1)N2k∑∑ ( x − x ), WMS = i=1 j =1iji

2

N (k − 1)

where BMS is the variance between subjects, WMS isthe variance in each subject, k is the number ofmeasurements, N is the number of subjects, x is themean value of the data from N subjects and k times, x i
is the mean value of the data from k times of Subject i,and xij is the data from Subject i at time j. In this study,k = 2 and N = 14.
The criterion methods for ICC(1,1) have been reported[3][4][6]. In Ref. [3], the numbers were set such that0.00-0.20: Slight, 0.21-0.40: Fair, 0.41-0.60: Moderate,
0.61-0.80: Substantial, and 0.81-1.00: Almost perfect.
On the other hand, in Ref. [4], the numbers werereported such that more than 0.6: Possible, more than
0.7: Normal (OK), more than 0.8: Good, and more than
0.9: Great.
Previous studies showed that if the ICC(1,1) is morethan 0.7, then the reliability is defined as good.
3.2. Results and discussion
The BMS and the WMS of the speed t and accuracy S for
the three courses are listed in Table 1. From thesevalues, the ICC(1,1)s were calculated. The ICC(1,1)s of
the speed t were 0.951, 0.982, and 0.984 for the straightcourse, the circular course, and the star-shaped course,respectively. The ICC(1,1)s of the accuracy S were0.265, 0.924, and 0.711 for the straight course, circularTable 1. Results of BMS and WMS.
Three evaluating courses(a) Straight(b) Circular (c) Star-shapedcourse
coursecourse2.10×102
2.97×102
speed BMS 4.01×101
t
WMS 1.01×100
1.95×100
2.42×100
1.29×1086.01×107
accuracy BMS 1.17×107
66S
WMS 6.79×10
5.07×101.01×107

course, and star-shaped course, respectively.
On the basis of these results, the reliabilities of thespeed t and the accuracy S were confirmed for thecircular and star-shaped courses. On the other hand, thereliability of the accuracy S was not confirmed in thestraight course. The reason for this can be determinedfrom Table 1 and Eq. (1). In the case of the accuracy S
in the straight course, the ratio of BMS and WMS wassmall compared to the other cases. This means that thevariance between the subjects was small and thevariance in each subject was large. It is probable that thereason for this was that the straight course was easy forthe healthy subjects.
As a result, it was confirmed that the developed
measurement system has sufficient reliability in both thecircular and star-shaped courses.
4.

Conclusion

We developed a measurement system to assess theskillfulness of the lower extremities with a straightcourse, a circular course, and a star-shaped course. The
quantitative evaluation method for the lower extremities
in both the circular and star-shaped courses hassufficient reliability.
Future work will involve the verification of thereliability of the proposed quantitative evaluation
method for elderly persons and subjects with reducedskillfulness of the lower extremities.
References
1. Shrout PE, Fleiss JL, Intraclass Correlations: Uses in
Assessing Rater Reliability, Psychological Bulletin, 86(1979) 420-428
2. Yamazaki K, Morita Y, Sato N, Ukai H, Nagaya M,
Hayashi T, Ito T , Sakai Y, Quantitative Evaluation
System of Sensorimotor Integration Function andProprioception of Lower extremities -Verification
Effectiveness for surgical patients with intramedullaryspinal cord tumor-, The Japan Society for Welfare
Engineering, 15(2) (2013) 44-49 (Article in Japanese)
3. Landis JR, Koch GG, The measurement of observer
agreement for categorical data, Biometrics, 33 (1977)
159-174
4. Kuwabara Y, Saito T, Inagaki Y, Evaluation of intra- andinter-observer reliability, Kokyu To Junka, 41(10) (1993)
945-952 (Article in Japanese)
5. Portney LG, Watkins MP, Foundations lf clinicalresearch-Applications to practice, Appleton & Lange,
USA (1993) 505-516

Published by Atlantis Press
Copyright: the authors
119