J Orthop Sci (2005) 10:466–474
DOI 10.1007/s00776-005-0937-1

Original article
Development and reliability of a standard rating system for outcome
measurement of foot and ankle disorders II: interclinician andintraclinician reliability and validity of the newly established standard
rating scales and Japanese Orthopaedic Association rating scale
Hisateru Niki1, Haruhito Aoki1, Suguru Inokuchi2, Satoru Ozeki3, Mitsuo Kinoshita4, Hideji Kura5,
Yasuhito Tanaka6, Masahiko Noguchi7, Shigeharu Nomura8, Masahito Hatori9, and Shinobu Tatsunami101Department of Orthopaedic Surgery, St. Marianna University School of Medicine, 2-16-1 Sugao, Miyamae-ku, Kanagawa 216-8511, Japan
Department of Orthopaedic Surgery, Keio University, Tokyo, Japan3
Department of Orthopaedic Surgery, Koshigaya Hospital, Dokkyo University School of Medicine, Saitama, Japan
4
Department of Orthopaedic Surgery, Osaka Medical College, Osaka, Japan
5
Department of Orthopaedic Surgery, Sapporo Medical University, Hokkaido, Japan
6
Department of Orthopaedic Surgery, Nara Medical University, Nara, Japan
7
Department of Orthopaedic Surgery, Tokyo Women’s Medical University Medical Center East, Tokyo, Japan
8
Nomura Seikeigeka Ganka Clinic, Yamaguchi, Japan
9
Department of Orthopaedic Surgery, Tohoku University, Miyagi, Japan
10
Unit of Medical Statistics, Faculty of Education and Culture, St. Marianna University School of Medicine, Kanagawa, Japan
2

Abstract
Background. This study evaluated the validity and inter- andintraclinician reliability of (1) the Japanese Society of Surgeryof the Foot (JSSF) standard rating system for four sites [anklehindfoot (AH), midfoot (MF), hallux (HL), and lesser toe(LT)] and the rheumatoid arthritis (RA) foot and ankle scaleand (2) the Japanese Orthopaedic Association’s foot rating
scale (JOA scale).
Methods. Clinicians from the same institute independently
evaluated participating patients from their institute by two
evaluations at a 1- to 4-week interval. Statistical evaluation
was as follows. (1) The intraclass correlation coefficient (ICC)
was calculated from data collected from at least two examinations of each patient by at least two evaluating clinicians (Data
A). (2) Total scores for the two evaluations were determinedfrom the distribution of differences in data between the two
evaluations (Data B); each item was evaluated by determining
Cohen’s coefficient of agreement. (3) The relation between
patient satisfaction and total score was investigated only forpatients who underwent surgery (Data C). Spearman’s rankcorrelation coefficient was obtained.
Results. Participants were 65 clinicians and 610 patients,
including those with disorders of the AH (313), MF (47),
HL (153), and LT (50) and those with RA (47). From Data A,
the ICC was high for AH and HL by JSSF scales and for AH,
MF, and LT by the JOA scale. From Data B, the coefficient
showed high validity for both scales for AH, with almost no
difference between the two scales; the validity for HL was

Offprint requests to: H. Niki
Received: May 23, 2005 / Accepted: June 28, 2005

higher with the JOA scale than with the JSSF scale. From
Data C, correlations were significant between patient satisfaction and outcome for AH and HL by the JSSF scales and for
AH, HL, and LT by the JOA scale.
Conclusions. The validity of both scales was high. Clinicalevaluation of the therapeutic results using these scales would
be highly reliable.

Introduction
Recently, therapeutic options have been selected quite
often on the basis of evidence-based medicine (EBM).
Thus, we are beginning to appreciate the importanceof a standard rating system to evaluate such evidence.
Such a rating system demands reliability in rating aswell as appropriate coverage of the diseases concernedand methods for their therapy. In this context, in orthopedic surgery, several standard rating systems haveundergone a number of examinations for reliability.1–8
Unfortunately, however, in the field of foot and ankle
joints the validity and reliability of the Japanese Orthopaedic Association (JOA) scale have not been verified.9,10 Moreover, although the American Orthopaedic
Foot and Ankle Society (AOFAS) clinical rating system11 could now be called a global standard, it has notbeen verified as to its validity and reliability.
The JOA attempted to provide an internationallyaccepted standard rating system that incorporated not
only objective evaluation by orthopedists but also sub-

H. Niki et al.: Reliability of JSSF scales

467

jective evaluation by patients. The JOA thus delegatedtasks to each member association to adjust and modify
standard rating systems and verify their validity andreliability. In responding to this request, the JapaneseSociety of Surgery of the Foot (JSSF) organized theCommittee on Rating Standards for Foot Disease inJune 2000. After many discussions they created the
JSSF standard rating system composed of five newscales, four of which were set up for four respective sites
by modifying the AOFAS clinical rating systems11; theremaining scale was for the rheumatoid arthritis (RA)foot and ankle joint by modifying the conventional JOA
scale9,10 (part I of this study, which appears in this issue).
Moreover, each scale included an explanation as well as
rating scores for each item so the individual items to beevaluated could be understood (part I of this study).
Our current four site-specific scales are a completely
novel and original Japanese version and are far from aduplicate of the AOFAS clinical rating system, as we
modified the expressions and content to suit Japanesepeople. We also added interpretation criteria for each
item and rating criteria, such as a pain scale, which werelacking in the AOFAS scale. This is why the Committeeon Rating Standards for Foot Disease of the JSSF
grouped together the five scales, comprised of foursite-specific scales and the RA foot and ankle scale andtermed it the JSSF standard rating system. From theyear 2001 on, actual patients were evaluated to collectdata employing the JSSF standard rating system in
multiple institutes.
In part II (described herein) we report the results ofstudies performed on a multiple-institution scale on thevalidity and inter- and intraclinician reliability of theevaluation items with regard to the JSSF standard rating
system composed of these five scales as well as theconventional JOA scale.

Selection of patients as evaluators

Materials and methods

Statistical methods

Selection of clinicians as evaluators
The subjects were orthopedists at nine institutions towhich the authors belonged. Because it was thoughtthat clinical experience would influence the reliability ofthe evaluation, the clinicians were selected according
to the following three levels of experience: (1) much
experience (specialist with at least 2 years’ experiencein foot surgery); (2) moderate experience (generalist
with approximately 6–7 years’ experience in an
orthopedics department); and (3) little experience[recently (within 1–2 years) graduated resident from a
medical university). In most cases two orthopedists representing each level of experience were selected fromeach institute.

Patients with diseases of the foot and ankle who met thefollowing criteria were included: (1) symptomaticallystable for at least 1 month prior to the study; (2)symptomatically stable for at least for 1 month after
the first evaluation; (3) consented to participate in the
study; and (4) had no underlying diseases or complications that might interfere with the results of theevaluation.
Study design
A clinician from the same institution independently
evaluated all the patients selected from that institution(first evaluation). Attempts were made to conduct theevaluation within 1 day, but when it was not possible it
was extended into the second day. No other evaluating
clinicians were present during this first evaluation.
The evaluating clinician explained to the patients that
simple answers to the questions were expected. Whenpossible, the same evaluating clinician performed boththe first and second evaluations. The second evaluation
was conducted within 1–4 weeks of the first evaluation.
As for the first examination, the second was conductedon the same day if possible. The results were recorded
immediately after the evaluation, and subsequentcorrections were prohibited. The results of the firstevaluation were concealed at the time of the second
evaluation. Patients were evaluated according to theorder of the items on the instrument being evaluated.
The evaluation of the items in both the JSSF standard
rating system and the JOA scale were conducted on thesame day as far as possible. The results were sent to theserver at each institution using the Web system established for data collection in the present study and stored
until tabulation.

1. To determine interclinician agreement in terms ofthe total scores (validity), the intraclass correlation
coefficient (ICC) was calculated from the evaluation
data, which was collected from at least two patients
who underwent the same evaluation by at least
two clinicians from the same institution if all
relevant data from those institutions were available (Analyzed Subject Data A). To establish themultiinstitutional overall scale for interclinician reliability, the ICC was calculated by the random effectmodel using data obtained for patients with diseasesof the ankle-hindfoot. Sufficient data for other sites
were not available from all of the institutions, but
sufficient data for this site was available from fiveinstitutions.

468

2. To determine intraclinician agreement (validity), thetotal scores from the first and second evaluations,
respectively, were determined from the distributionof differences in the data between the two evaluations for each institution that provided sufficient data
(Analyzing Subject Data B). Each item was evaluated by determining Cohen’s coefficient of agreement (k) and the rate of complete agreement (RC)between the first and second evaluations.
3. To determine the relation between the scores in eachscale and patient satisfaction, the relation between
patient satisfaction and outcome (total score) wasinvestigated using the evaluations of only thosepatients who had undergone surgery (Analyzing
Subject Data C). The degree of satisfaction was
evaluated as “very satisfactory,” “satisfactory,”“noncomputable,” “slightly unsatisfactory,” and“very unsatisfactory.” The total score for each degree of satisfaction was 0–50, 60–69, 70–79, 80–89,
and 90–100 points ranked as 0, 1, 2, 3, and 4, respectively. Spearman’s rank correlation coefficient (r)was then obtained.

Results
Evaluating clinicians and patients
A total of 65 clinicians evaluated the patients. The distribution of clinicians according to experience level was
21.5% specialists, 30.8% generalists, and 47.7% residents. There were 610 patients, representing 313 diseases of the ankle-hindfoot, 47 diseases of the midfoot,153 diseases of the hallux, 50 diseases of the lesser toe,and 47 with RA. Evaluation by the JOA scale was conducted simultaneously with that by JSSF scales in 501 of
the 610 patients.
Results of statistical analysis
1. For Data A, the number of patients and the numberof evaluating clinicians varied among the institutions.
With the lower limit of the 95% confidence interval(CI) of the ICC calculated as an indication of interclinician agreement being 0.41, a value of >0.41 wasobserved for the ankle-hindfoot and hallux by the
JSSF scales and for the ankle-hindfoot, midfoot, andlesser toe by the JOA scale (P < 0.05; ICC > 0.4 intesting) (Table 1). As for patients with diseases of theankle-hindfoot, the overall ICC calculated from thedata for the five institutions was 0.93 for the JSSF
scale compared with 0.91 for the JOA scale.
2. For Data B, the percentages of values for each site
evaluated by the JSSF scales relative to that evaluated by the JOA scale were as follows: 83 to 83 for

H. Niki et al.: Reliability of JSSF scales

the ankle-hindfoot, 10 to 4 for the midfoot, 45 to 56
for the hallux, 6 to 4 for the lesser toe, and 21 to 21
for RA.
a. Distribution of differences in total scores.
1) Regardless of the experience level, the difference in total scores between the first and second evaluation was within the range of ±1 in
43.4% and 42.3% of the data evaluated by the
JSSF and JOA scales, respectively, for the
ankle-hindfoot, indicating almost no difference between the two. These frequencies werehigher than those for other sites, and the difference was within ±5 in approximately 70% of
data evaluated by the two scales for the anklehindfoot. The difference was within a range of±1 in 31.1% and 37.5% of the data evaluatedby the JSSF scales and the JOA scale, respectively, for the hallux. The corresponding
frequencies in RA patients were 19.5% and19.0% of data evaluated by the JSSF andJOA scales, respectively; differences within
the range of ±5 were observed in approximately 60% of the data evaluated by the two
scales. It was difficult to evaluate the midfootand lesser toe because of the small number ofpatients with diseases at these sites (Table 2).
2) The influence of experience level was observed when the difference in the total scores
between the first and second evaluations waswithin a range of ±1; a tendency toward thepresence of influence of the experience level
was observed in data evaluated by the
JSSF scale for the ankle-hindfoot and in data
evaluated by both scales for the hallux and
RA. When the difference was within the range
of ±5, however, there was almost no differencein the results depending on the experiencelevel. It was difficult to evaluate the midfootand lesser toe because of the small numberof patients with diseases at these sites(Table 3).
b. Evaluation of each item.
1) For the first and second evaluations, Cohen’scoefficient of agreement (k) was high for
all items for the ankle-hindfoot evaluated by
the JSSF scale and low for sagittal motion,muscle strength, and sensory disturbance(paresthesia) of the hindfoot evaluated by the
JOA scale (Table 4). The coefficient (k) was
low for all items other than sagittal motion of
the metatarsophalangeal (MTP) joint of thehallux evaluated by the JSSF scale, and highfor most of the items evaluated by the JOA
scale. It was difficult to evaluate data for the
midfoot, lesser toe, and RA because of the

4
4
3
2
4
3
2
6
2
2
6
3
2
4
6

2
2

6
2
7

2
2
4

2
2

No. ofclinicians

6
3
3
3
2

No. ofpatients

0.08 (-0.014–0.99)
0.4162 (-0.04–1.0)

0.7298 (0.22–1.0)
0.569 (-0.18–1.0)
0.441 (-0.85–0.95)

0.5429 (0.21–0.89)
0.0 (-1.0–1.0)
0.971 (0.84–1.0)

0.6975 (-0.15–1.0)
0.8324 (-0.02–1.0)

0.7246 (0.3788–0.9472)
0.8526 (0.45–1.0)
0.9318 (0.5284–0.9982)
0.2753 (-1.5–0.98)-0.6691 (-0.7248–0.6935)

ICC

0.71070.32890.08610.26660.45830.18620.59040.00040.1896
0.1984

0.03
0.016
0.015
0.55080.9009

P

A
B

A
C

A
C

C

A
B
C

Institute

2
2

2
2

6
2

2

5
3
2

No. ofpatients

4
6

6
3

6
3

3

5
4
4

No. ofclinicians
ICC

0.3099 (-0.19–1.0)
0.3513 (0.0047–1.0)

0.8586 (0.43–1.0)
0.9461 (0.53–1.0)

0.5840 (0.2658–0.9050)-0.5676 (-0.6152–0.3498)

0.9162 (0.3–1.0)

0.8721 (0.64–0.98)
0.4647 (-0.05–0.98)
0.3018 (-0.31–1.0)

JOA scale

JSSF, Japanese Society of Surgery of the Foot; JOA, Japanese Orthopaedic Association; ICC, intraclass correlation coefficient; RA, rheumatoid arthritis
Boldface type indicates that ICC > 0.4 (P < 0.05)

Ankle-hindfoot
A
B
C
D
E
Midfoot
C
E
HalluxA
C
D
Lesser toe
A
C
D
RA
A
B

Institute

JSSF scale

Table 1. Intraclass correlation coefficient at each institution

0.41880.38530.01870.01330.12350.9011

0.04670.00040.34060.4241P

H. Niki et al.: Reliability of JSSF scales
469

470

H. Niki et al.: Reliability of JSSF scales

small number of patients in the respective
categories.
2) The mean RCs for each item evaluated by the
JSSF and JOA scales were 81.2% and 84.3%,
respectively, for the ankle-hindfoot; 70% and57.1%, respectively, for the midfoot; 75.6%
and 78.5%, respectively, for the hallux; 83.3%
Table 2. Distribution of difference in data between first andsecond evaluations (regardless of experience level)% Difference in range of±1 to ±5Site and scale

No.

±1

±3

±5

83
83

43.4
42.3

61.4
56.6

68.7
75.9

10
4

50.0
25.0

50.0
25.0

70.0
25.0

45
56

31.1
37.5

40.0
53.8

55.6
62.5

6
4

33.3
75.0

66.7
100

83.3
100

21
21

19.5
19.0

47.6
33.3

61.9
52.4

Ankle-hindfoot
JSSF
JOA
Midfoot
JSSF
JOA
HalluxJSSF
JOA
Lesser toe
JSSF
JOA
RA
JSSF
JOA

and 82.1%, respectively, for the lesser toe;and 76.2% and 77.5%, respectively, for RA.
Accordforg to the items, the intraclinician RC
was high for all items of the ankle-hindfoot by
the JSSF scale, whereas the rate was low for
instability of the ankle-hindfoot by the JOA
scale. The rate was low for alignment of thehallux by the JSSF scale and for pain, deformed forefoot, hindfoot sagittal motion, andwalking on tiptoe by the JOA scale. The rate
was low for a deformed lesser toe of the forefoot, deformed hindfoot, and ability to walkwhen evaluated by the JSSF scale in RA
patients and for pain, deformed forefoot,hindfoot sagittal motion, and ability to walkwhen evaluated by the JOA scale.
3. For Data C, the ratios of the total score for each site
as evaluated by the JSSF scales to those as evaluatedby the JOA scale were as follows: 169 : 161 for the
ankle-hindfoot, 14 : 14 for the midfoot, 99 : 105 for thehallux, 34 : 33 for the lesser toe, and 24 : 24 for RA.
a. There was a significant correlation between patient satisfaction and the total score (outcome) forthe hindfoot and hallux by the JSSF standard rating system and for the ankle-hindfoot, hallux, andlesser toe by the JOA scale (Table 5).

Table 3. Distribution of difference in data between first and second evaluations (withregard to experience level)% Difference, by JSSF scale
Experience level
Ankle-hindfootSpecialist
Generalist
Resident
Midfoot
Specialist
Generalist
ResidentHalluxSpecialist
Generalist
Resident
Lesser toe
Specialist
Generalist
Resident
RA
Specialist
Generalist
Resident% Difference, by JOA scale

No.

±1

±3

±5

No.

±1

±3

±5

34
25
24

47.1
52.0
29.2

64.7
76.0
41.7

76.5
80.0
45.8

33
26
24

45.5
38.5
41.7

57.6
61.5
50.0

75.8
84.6
66.7

4
2
4

75.0
50.0
25.0

75.0
50.0
25.0

100
50.0
50.0

1
1
2

—
—
0

—
—
0

—
—
0

17
13
15

41.2
23.1
26.7

52.9
23.1
40.0

64.7
38.5
60.0

22
18
16

50.0
27.8
31.3

54.5
50.0
56.3

63.6
55.6
68.9

3
2
1

0
50.0
—

33.3
100
—

66.7
100
—

2
1
1

100
—
—

100
—
—

100
—
—

6
7
8

33.3
14.3
12.5

50.0
42.9
50.0

66.7
57.1
62.5

5
8
8

20.0
37.5
0

40.0
37.5
25.0

60.0
50.0
50.0

—, noncomputable (insufficient sample number)

H. Niki et al.: Reliability of JSSF scales

471

Table 4. Rate of complete agreement and Cohen’s coefficient of agreement
JSSF scale

JOA scaleParameter

RC (%)

k

Ankle-hindfoot (n = 83)
Pain
Activity limitations
Maximum walking distanceWalking surfaces
Gait abnormality
Sagittal motion
Hindfoot motion
Stability
Alignment

79.5
71.1
85.5
83.1
83.1
85.5
80.7
86.7
79.5

0.672
0.568
0.604
0.711
0.582
0.625
0.573
0.405
—

Midfoot (n = 10)
Pain
Activity limitations
Max. walking distance
Footwear requirements
Walking surfaces
Gait abnormality
Alignment

60
60
60
70
60
90
90

0.492
0.31
—
—
—
0.821
—

Hallux (n = 45)
Pain
Activity limitations
Footwear requirements
MTP joint motion
IP joint motion
MTP-IP Stability
Callus or clavus
Alignment

66.6
64.4
73.3
75.6
97.8
88.9
80
57.8

—
—
—
0.559
—
0.237
0.281
0.282

Lesser toe (n = 6)
Pain
Activity limitations
Footwear requirements
MTP joint motion
IP joint motion
MTP-IP Stability
Callus or clavus
Alignment

100
66.7
66.7
66.7
83.3
100
100
83.3

1
0.25
—
—
—
—
—
0.667

Parameter
Ankle-hindfoot (n = 83)
Pain
Deformity, forefootDeformity, hindfoot
MTP/IP joint motion
Hindfoot motion
Stability
Walking ability
Muscle strength
Sensory disturbance
Climbing/descending stairs
Sitting on heels
Standing on toes
FootwearJapanese-style toilet
Midfoot (n = 4)
Pain
Deformity, forefootDeformity, hindfoot
MTP/IP joint motion
Hindfoot motion
Stability
Walking ability
Muscle strength
Sensory disturbance
Climbing/descending stairs
Sitting on heels
Standing on toes
FootwearJapanese-style toilet
Hallux (n = 56)
Pain
Deformity, forefootDeformity, hindfoot
MTP/IP joint motion
Hindfoot motion
Stability
Walking ability
Muscle strength
Sensory disturbance
Climbing/descending stairs
Sitting on heels
Standing on toes
FootwearJapanese-style toilet
Lesser toe (n = 4)
Pain
Deformity, forefootDeformity, hindfoot
MTP/IP joint motion
Hindfoot motion
Stability
Walking ability
Muscle strength
Sensory disturbance
Climbing/descending stairs
Sitting on heels
Standing on toes
FootwearJapanese-style toilet

RC (%)

k

77.1
89.2
79.5
71.1
94
63.9
77.1
86.7
86.7
96.4
88
79.5
88
73.5

0.639
0.574
0.514
0.358
0.78
—
—
0.286
0.252
0.928
0.81
0.548
0.522
0.514

50
50
25
75
50
75
50
25
75
75
75
100
25
50

—
0
—
—
0.333
0.5
0.2
—
—
—
—
—
—
—

57.1
64.3
91.1
94.6
69.6
78.6
73.2
80.4
87.5
91.1
85.7
69.6
71.4
75.7

0.357
0.474
0.51
0.024
0.526
0.361
0.532
—
0.162
0.707
0.439
0.479
0.492
—

50
100
100
50
100
50
75
100
100
100
100
75
50
100

—
—
—
—
—
—
—
—
—
—
—
0.5
—
—

472

H. Niki et al.: Reliability of JSSF scales

Table 4. Continued
JSSF scale
Parameter

JOA scale
RC (%)

k

0.762
0.762
0.571
0.714
0.476
0.712
0.762
0.571
0.952
1
0.857
0.857
0.905

—
0.608
0.016
0.475
—
0.632
0.578
—
—
—
—
0.745
0.811

RA (n = 21)
Pain
Derormity, hallux
Deformity, lesser toe
Deformity, midfootDeformity, hindfoot
MTP/IP joint motion
Hindfoot motion
Walking ability
Climbing/descending stairs
Sitting on heels
Standing on toes
FootwearJapanese-style toilet

Parameter
RA (n = 21)
Pain
Deformity, forefootDeformity, hindfoot
MTP/IP joint motion
Hindfoot motion
Stability
Walking ability
Muscle strength
Sensory disturbance
Climbing/descending stairs
Sitting on heels
Standing on toes
FootwearJapanese-style toilet

RC (%)

k

61.9
66.7
71.4
57.1
71.4
71.4
66.7
76.2
95.2
95.2
90.5
81
85.7
95.2

0.408
—
0.571
0.171
0.571
—
—
—
—
—
—
—
0.725
0.905

RC, rate of complete agreement; k, Cohen’s coefficient of agreement; —, noncomputable
k Values: boldface indicates k > 0.6 and italics indicates k > 0.4

Table 5. Relation between patient satisfaction and total score(outcome)
Spearman rank correlation (r)Parameter
Ankle-hindfoot
Midfoot
HalluxLesser toe
RA

JSSF scale

JOA scale

0.373 (P < 0.0001)
0.104
0.399 (P < 0.0001)
0.321—

0.341 (P < 0.0001)-0.007
0.271 (P < 0.005)
0.737 (P < 0.0001)——, Noncomputable

Discussion
With the practice of EBM gaining ground worldwide,
many epidemiological surveys and clinical studies arebeing performed for the purpose of obtaining evidence.
An assessment of the results is essential for surveys andstudies, and the relative superiority of the efficacy of
one treatment or therapeutic effect over another shouldbe evaluated based on the results of such determinations. For objective assessment of the results, astandard rating scale for evaluation should therefore be
established. Important requirements for a rating scale
are a high degree of validity and reliability. To ourknowledge, the intraclinician and interclinician validity
and reliability of standard rating systems for evaluating
diseases of the foot and ankle, including the AOFAS
clinical rating systems, have never been examined bymultiinstitutional studies.
As for the interclinician agreement in terms of thetotal scores, the ICC was calculated from data obtained
from evaluation of at least two of the same patients bymultiple clinicians at the same institution. Only institutions from which there were sufficient data for analysis
were included. At each institution, the ICC was high forthe ankle-hindfoot and hallux by the JSSF scales andhigh for the ankle-hindfoot, midfoot, and lesser toe by
the JOA scale. These results indicate that reliability washigh at each institution, although overall multiinstitutional interclinician reliability could not be evaluated.
When following the method employed in the report that
evaluated reliability over all participating institutions
using the ICC by the random effect model7 it is possiblethat one cannot obtain a correct evaluation in such cases
where the experience or knowledge of the examiners or
the severity of the disease in patients differs among
institutions or where the amount of data is small. Therefore, in principle we calculated each ICC for each institution. To verify our findings, we calculated the ICCfrom data for the ankle-hindfoot for all five institutions
following a similar random effect model7 and found that
the ICC was 0.9 or higher by both the JSSF scale and the
JOA scale. Even when the same patient was examinedat many institutions, the reliability of the standard rating scale for evaluation of diseases of the ankle-hindfootwas estimated to be high.
When interclinician and intraclinician reliability of
the JSSF standard rating system and the JOA scale were
investigated merely from the viewpoint of differencesin the total scores between the first and second evaluations, the range of validity tended to increase for thehallux and RA compared to that for the ankle-hindfoot,for which the validity was already found to be relatively
high. The RC, which was reflected by Cohen’s coefficient of agreement for each item, also showed high validity on the JSSF and JOA scales for evaluation of theH. Niki et al.: Reliability of JSSF scales

ankle-hindfoot, with almost no difference observed between the two scales, whereas the validity of the JOA
scale for the hallux was higher than that of the JSSF
scale. Thus, there was a difference in validity between
the two scales for some sites of the foot and ankle.
There were also some items for which statistical analysis
could not be conducted because of the small number ofpatients; but the validity of the JSSF standard rating
system was evaluated as being high by the assessment of
intraclinician agreement because the concept of eachscale of the JSSF standard rating system is almost thesame.
As for intraclinician agreement assessed according to
the level of clinical experience, it is assumed that proficiency in evaluation is necessary to obtain high validityof the evaluation when investigated only from the distribution of differences in the total scores.
“The degree of satisfaction” in the evaluation oftreatment is related to psychological aspects on the partof patients and differs from the functional aspects evaluated by clinicians. Therefore, the correlation between
the degree of satisfaction on the part of patients andfunctional assessment by clinicians is not necessarily
high, but there was a tendency for the outcome to becorrelated with patient satisfaction. Each item in thestandard rating system was considered to be a reflectionof a subjective evaluation on the part of the patients.
Recently, results of findings by instruments on the severity of pain by visual analogue scales (VAS) and questionnaires about the quality of life (QOL) by SF-36
and others, in which QOL is evaluated based on scales
that take into account the viewpoint of patients, havebeen shown to be as reproducible as results based ondata from pathophysiologic evaluations by clinicians.
In other words, therapeutic results are increasingly
determined directly according to the patient’s own
evaluation from the viewpoint of EBM because there ismuch room for bias in evaluations by clinicians; thus,
instruments such as the VAS and SF-36 produce highly
accurate information.12–18 Therefore, each standard
rating scale for evaluation that was inspected in thisstudy is assumed to be a reflection to some extent of
the subjective evaluation on the part of patients, but astandard rating system that would allow evaluation of
the symptomatic improvement and QOL of patientsfrom different viewpoints needs to be established in the
future.
The present study was conducted with the aim ofevaluating the validity and reliability of the JSSF standard rating system and the JOA scale according to thesite of involvement in the foot and ankle. Diagnostic
workups of the same patients at multiple institutions are
difficult. Therefore, we were obliged to limit our analysis of interclinician reliability to that from data compiled
at individual institutions. To analyze interclinician reli-473

ability more precisely, a different study design from that
employed in the present study may be required.
Based on intraclinician reliability and the results ofanalysis of the relation between patient satisfaction andoutcome, however, the validity of the JSSF standard
rating system and the JOA scale was high for the items
evaluated. It can be considered that clinical evaluationof therapeutic results using these scales would be highly
reliable.

References
1. Sidor ML, Zuckerman JD, Lyon T, Koval K, Cuomo F,
Schoenberg N. The Neer classification system for proximal
humeral fractures; an assessment of interobserver reliability andintraobserver reproducibility. J Bone Joint Surg Am 1993;75:
1745–50.
2. Siebenrock KA, Gerber C. The reproducibility of classification of
fractures of the proximal end of the humerus. J Bone Joint Surg
Am 1993;75:1751–5.
3. Rome K, Cowieson F. A reliability study of the universal goniometer, fluid goniometer, and electrogoniometer for the measurement of ankle dorsiflexion. Foot Ankle Int 1996;17:28–32.
4. Cummings RJ, Loveless EA, Campbell J, Samelson S, Mazur JM.
Interobserver reliability and intraobserver reproducibility of thesystem of King et al. for the classification of adolescent idiopathic
scoliosis. J Bone Joint Surg Am 1998;80:1107–11.
5. Irrgang JJ, Snyder-Mackler L, Wainner RS, Fu FH, Harner CD.
Development of a patient-reported measure of function of theknee. J Bone Joint Surg Am 1998;80:1132–45.
6. Lenke LG, Bets RR, Bridwell KH, Clements DH, Harms J, Lowe
TG, et al. Intraobserver and interobserver reliability of theclassification of thoracic adolescent idiopathic scoliosis. J Bone
Joint Surg Am 1998;80:1097–106.
7. Yonenobu K, Abumi K, Nagata K, Taketomi E, Ueyama K.
Inter- and intra-observer reliability of the Japanese Orthopaedic
Association scoring system for evaluation of cervical myelopathy.
Rinsyou Seikeigeka (Clinical Orthopaedic Surgery) 2001;36:423–
8 (in Japanese).
8. Greenfield MLVH, Kuhn JE, Wojtys EM. A statistic primer;
validity and reliability. Am J Sports Med 1998;26:483–5.
9. Japanese Orthopaedic Association. Assessment criteria for footdisorders of the Japanese Orthopaedic Association. J Jpn Orthop
Assoc 1991;65:680 (in Japanese).
10. Hisateru N, Nango A. Clinical rating systems for ankle disorders.
In: Murota K, Yabe Y, Sano S, editors. Manual of orthopaedic
clinical rating systems. Tokyo: Zen Nihonbyoin Shuppan Kai;
1995. p.117–35 (in Japanese).
11. Kitaoka HB, Alexander IJ, Adelaar RS, Nunley JA, Myerson MS,
Sanders M. Clinical rating systems for the ankle-hindfoot,midfoot, hallux, and lesser toes. Foot Ankle Int 1994;15:349–
53.
12. Fukuhara S, Suzugamo Y. Manual of SF-36v2 Japanese version.
Kyoto: Institute for Health Outcomes & Process Evaluation
Research; 2004 (in Japanese).
13. Toolan BC, Wright Quinones VJ, Cunningham BJ, Brage ME.
An evaluation of the use of retrospectively acquired preoperative
AOFAS clinical rating scores to assess surgical outcome after
elective foot and ankle surgery. Foot Ankle Int 2001;22:775–8.
14. Thordarson DB, Rudicel SA, Ebramzadeh E, Gill LH. Outcome
study of hallux valgus surgery: an AOFAS multi-center study.
Foot Ankle Int 2001;22:956–9. Erratum in: Foot Ankle Int
2002;23:96.
15. Hunsaker FG, Cioffi DA, Amadio PC, Wright JT, Caughlin B.
The American Academy of Orthopaedic Surgeons outcomes474
instruments: normative values from the general population. J
Bone Joint Surg Am 2002;84:208–15.
16. SooHoo NF, Shuler M, Fleming LL. Evaluation of the validity of
the AOFAS clinical rating systems by correlation to the SF-36.
Foot Ankle Int 2003;24:50–5.
17. Johanson NA, Liang MH, Daltroy L, Rudicel S, Richmond J.
American Academy of Orthopaedic Surgeons lower limb out-

H. Niki et al.: Reliability of JSSF scales
comes assessment instruments: reliability, validity, and sensitivity
to change. J Bone Joint Surg Am 2004;86:902–9.
18. Thordarson D, Ebramzadeh E, Moorthy M, Lee J, Rudicel S.
Correlation of hallux valgus surgical outcome with AOFAS forefoot score and radiological parameters. Foot Ankle Int 2005;26:
122–7.