9

Japanese Journal of Comprehensive Rehabilitation Science (2020)

Original Article

Interrater agreement between clinician ratings and patient selfassessments for body function categories of ICF Rehabilitation Set
Yuki Senju, MD,1 Masahiko Mukaino, MD, DMSc,1 Megumi Ozeki, MD, DMSc,2Makoto Watanabe, OTR,3 Yuki Okochi, OTR,4 Koji Mizutani, RPT,4 Eiichi Saito, MD, DMSc,1
Shigeru Sonoda, MD, PhD5
Department of Rehabilitation Medicine I, School of Medicine, Fujita Health University, Aichi, Japan
Faculty of Rehabilitation, School of Health Sciences, Fujita Health University, Aichi, Japan3
Department of Rehabilitation Medicine, Fujita Health University Nanakuri Memorial Hospital, Mie, Japan
4
Department of Rehabilitation Medicine, Fujita Health University Hospital, Aichi, Japan
5
Department of Rehabilitation Medicine II, School of Medicine, Fujita Health University, Mie, Japan1
2

ABSTRACT
Senju Y, Mukaino M, Ozeki M, Watanabe M, Okochi
Y, Mizutani K, Saito E, Sonoda S. Interrater agreement
between clinician ratings and patient self-assessments
for body function categories of ICF Rehabilitation Set.
Jpn J Compr Rehabil Sci 2020; 11: 9‐16.
Objective: This study investigates interrater agreement
between clinician ratings using the Rating reference
guide developed in Japan for the International
Classification of Functioning, Disability and Health(ICF) Rehabilitation Set and patient self-assessments
for body function categories of the ICF Rehabilitation
Set.
Methods: Eighty-eight inpatients and/or outpatients
who received rehabilitation participated in this study.
For the assessment of the nine body function categoriesof the ICF Rehabilitation Set, the patients were asked
to complete the self-assessment questionnaires, and theclinicians rated the patients using the Rating reference
guide. Interrater agreement between clinicians andpatients was investigated using weighted kappa
statistics and an intraclass correlation coefficient (ICC(2,1)) to determine interrater agreement of eachcategory and the total score, respectively.
Results: The weighted kappa statistics ranged from0.58 to 0.87. Eight out of nine categories presented
with weighted kappa statistics greater than 0.61. The
total score of all categories showed no significant
Correspondence: Masahiko Mukaino, MD, DMSc
Department of Rehabilitation Medicine I, School of
Medicine, Fujita Health University, 1‐98 Dengakugakubo,
Kutsukake, Toyoake, Aichi 470‐1192, Japan.
E-mail: mmukaino@fujita-hu.ac.jp
Accepted: September 17, 2019.
The authors have no conflict of interest directly relevant tothe content of this article.
©Kaifukuki Rehabilitation Ward Association 2020
doi.org/10.11336/jjcrs.11.9

difference between clinicians and patients andpresented with an ICC (2,1) of 0.85.
Conclusion: No significant difference was observed
between clinician ratings with the Rating reference
guide for body function categories and patient selfassessments, showing feasibility of the Ratingreference guide as a means of describing the status ofpatients’ functioning.
Key words: ICF, ICF Rehabilitation Set, Ratingreference guide, body function, interrater agreement
Introduction
The International Classification of Functioning,
Disability and Health (ICF) is a framework fordescribing and organizing information on patientfunctioning and disability [1, 2]; it covers information
ranging from the status of functioning related to thepatient’s health condition to the social system or social
resources surrounding the patient. The ICF consists ofmore than 1,400 categories and provides both astandard language and a conceptual basis for defining
and measuring health and disability that enable the
ICF to comprehensively describe patients’ status offunctioning. Since the ICF was approved in May 2001,
various actions have been taken to promote theimplementation of the international framework.
The ICF Core Set project is one of the major projects
promoted in the framework. This project refers to theselect set of ICF categories that can serve as minimal
standards for the assessment and documentation of thefunctioning and health of persons with various healthconditions and/or diseases and that can be developed
through the predefined objective process [3‐8]. There
are also ICF Core Sets that could be applied acrossvarious disease or health conditions. For this purpose,
two Core Sets have been developed. The ICF GenericJpn J Compr Rehabil Sci Vol 11, 2020

10

Senju Y et al.: Clinician ratings and patient self-assessments of b-categories of ICF Rehabilitation Set

Set, developed by psychometric study, consists of theseven ICF categories that are most relevant across thegeneral population and across different clinicalpopulations, contexts, settings, and purposes [9]. An
extended version of this set for use in the rehabilitation
setting, the ICF Rehabilitation Set with 30 ICF
categories, was developed from the results of additional
psychometric studies and expert reviews of the ICF
Core Sets (Table 1) [9, 10]. Projects to create tools
based on these ICF Core Sets has been advanced in theinternational framework by expert groups such as theInternational Society of Physical and Rehabilitation
Medicine and the Physical and Rehabilitation Medicine
Section and Board of the European Union of Medical
Specialists [11, 12]. The focus of one ongoing projectis to assist clinicians with categorization by developing
simple, intuitive descriptions for ICF categories thathave titles and definitions that are not intuitive [13, 14].
In Japan, a Japanese version of the simple, intuitive
descriptions of ICF Rehabilitation Set was created in2016 (Mukaino et al, in submission).
Another ongoing project aims to develop clinical
Table 1. ICF Rehabilitation Set.
Categories
b130
b134
b152
b280
b455
b620
b640
b710
b730
d230
d240
d410
d415
d420
d450
d455
d465
d470
d510
d520
d530
d540
d550
d570
d640
d660
d710
d770
d850
d920

Energy and drive functions
Sleep functions
Emotional functions
Sensation of pain
Exercise tolerance functions
Urination functions
Sexual functions
Mobility of joint functions
Muscle power functions
Carrying out daily routine
Handling stress and other psychologicaldemands
Changing basic body position
Maintaining a body position
Transferring oneself
WalkingMoving around
Moving around using equipment
Using transportation
Washing oneself
Caring for body parts
Toileting
Dressing
EatingLooking after one’s health
Doing housework
Assisting others
Basic interpersonal interactions
Intimate relationships
Remunerative employment
Recreation and leisure

Jpn J Compr Rehabil Sci Vol 11, 2020

rating tools [12]. For the coding of patients withfunctioning problems, the ICF has its own rating
system that uses qualifiers. The qualifiers are definedin the ICF book as follows: 0: no problem, 1: mild
problem (5‐24%), 2: moderate problem (25‐49%), 3:
severe problem (50‐95%), 4: complete problem(96‐100%), 8: not specified, 9: not applicable [1].
Unlike most of the existing clinical scales, thequalifiers have no further explanation. This may cause
the difficulty and low reliability in rating the patients’
status in the categories the clinicians are notaccustomed to evaluating. For example, Uhlig et alreported on the interrater reliability of clinician ratingsof the ICF Core Set for rheumatoid arthritis. Although
they could show relatively high interrater reliability inseveral categories with which the clinicians werefamiliar, such as toileting or dressing, most of thecategories presented with low weighted kappa
statistics and thus low interrater reliability [15].
The difficulties in rating could be seen particularlyin the body function categories. As is done in many ofthe existing clinical scales, the problems in activities
and participation may be simply estimated byclinicians based on the patients’ status in performing
the activity described in the given category. This could
be considered in a single dimension. However, in theevaluation of body function, estimating the degree of
patients’ problems is not straightforward. For example,“b280 Sensation of pain” could be evaluated byclinicians in light of several aspects, including theextent of pain, frequency of pain, and site of pain.
However, these aspects may differ between the raters.
For example, mild and infrequent pain may beperceived as a problem by some patients but notothers. In addition, what constitutes either a severe
problem or a mild problem may easily vary according
to what aspects the raters focus on.
More recently, the development of ICF-based datacollection tools in an international framework was
proposed to solve these problems [12]. Along withthese international efforts, a Rating reference guide
was developed in Japan. The Rating reference guide isa clinical tool designed to support ICF-based patient
evaluation and developed from the analysis ofclinician’s thinking processes in using ICF by acognitive interviewing of clinicians (Senju et al, in
submission). For example, the Rating reference guidefor body function categories indicates 1) what aspects
the raters focus on when rating, and 2) what each
response option (0‐4) indicates. The study showed thatthe weighted kappa statistics of all categories exceeded
0.60, indicating that using such a Rating reference
guide results in generally good interrater reliability[16]. However, toward the clinical implementation of
the ICF, it is important to ensure the rating is not only
reliable among clinicians but also in agreement withpatient self-assessments. The ICF is developed todescribe the “lived experience” of the patients [17]. InSenju Y et al.: Clinician ratings and patient self-assessments of b-categories of ICF Rehabilitation Set 11other words, rather than just looking at the severity ofsymptoms according to clinicians’ perspectives,
patients’ perspectives should be considered. Naturally,
there will be differences in the evaluation based on theobjective viewpoint of clinicians and the subjective
problem recognition of patients. However, based on
the ICF concept defined in this paper, the evaluation byclinicians should not diverge significantly, even if it
does not completely match patients’ own evaluations.
Based on such considerations, we examined interrater
agreement between clinician ratings using the Ratingreference guide and patients’ own evaluations using
the self-assessment questionnaires and distribution ofevaluation for each body function category in the ICF
Rehabilitation Set.
Methods

categories of the ICF Rehabilitation Set (Table 3), andclinicians provided ratings using the Rating reference
guide (Table 4). The assessment of body function
categories of the ICF Rehabilitation Set by theclinicians was conducted by a single rater out of fiverehabilitation experts (two physiatrists and threetherapists, with a combined 6‐18 years of experience(median: 16 years)). The Rating reference guide shows
the following points for each category: 1) what aspects(e.g., frequency, extent) the raters focus on, and 2)
what the “4: complete problem” indicates. In addition,common brief guidelines to frame each response
option were provided: 0: no problem, 1: mild problem(may include problem that does not affect daily
activities), 2: moderate problem (may include problem
that exceeds “1” but still remains relatively minor
problem in given category), 3: severe problem (may1. Subjects
Eighty-eight patients who received rehabilitationfrom April 2017 to November 2018 in Fujita Health
University Hospital and Fujita Health University
Nanakuri Memorial Hospital participated in this study.
The mean age of the patients was 65.4 ± 15.3 years. Of
the 56 male and 32 female patients, 63 patients had
neurological diseases such as stroke, 10 patients had
musculoskeletal diseases, 4 patients had cardiovasculardiseases, 4 patients had pulmonary diseases, and 7 hadothers; there were 77 inpatients and 11 outpatients(Table 2). We excluded the patients whose MiniMental State Examination (MMSE) score was lessthan 23 or who could not answer the self-assessments
questionnaire.

Table 2. Demographic variables.
Sample size (n)Age (years)
Age, mean ± SD (years)

2. Methods
All patients were asked to complete the selfassessments questionnaire for the nine body function

88
65.4 ± 15.3

GenderMale
Femalen (%)
56 (63.6)
32 (36.4)

Diagnosis
Cerebrovascular disease-Supratentorial lesions
Orthopedic disease
Respiratory disease
Cardiovascular disease
Other

n (%)
63 (71.6)
45 (51.1)
10 (11.4)4 (4.6)4 (4.6)7 (8.0)Inpatients
Outpatients77 (87.5)
11 (12.5)

Table 3. Self-assessment questionnaire.
How much do you have problem in the following functions ?
Categories
b130

Energy and drive functions

b134

Sleep functionsb152

Emotional functions

b280

Sensation of painb455

Exercise tolerance functions

b620

Urination functions

b640

Sexual functions

b710

Mobility of joint functionsb730

Muscle power functions

No problem

Mild
problemModerate
problemSevereproblemCompleteproblemI don’tknow

Jpn J Compr Rehabil Sci Vol 11, 2020

12

Senju Y et al.: Clinician ratings and patient self-assessments of b-categories of ICF Rehabilitation Set

Table 4. Rating reference guide.

b130
b134
b152

Categories

What would be complete problem

What aspect to score

Energy anddrivefunctions
Sleepfunctions
Emotional
functions

The extreme should be that they have no motivation
or appetite at any time・The extent of the problem
・The frequency of the problem

The extreme should be that they cannot sleep at all or
the sleeping cycle has completely collapsed every day
The extreme should be that they completely lose
control of emotions every day or are completely
incapable of expressing emotions
The extreme should be that they have continuous
intolerable pain at any time・The extent of the problem
・The frequency of the problem
・The extent of the problem
・The frequency of the problem

b280

Sensation of
pain

b455

Exercisetolerance
functions
UrinationfunctionsSexualfunctions

b620
b640

b710
b730

The extreme should be that they cannot bear any
single activity of daily living at any time

The extreme should be that they cannot urinate orcontrol urination at all every day
The extreme should be that they have no mental(may include having no sexual desire or cannot
control sexual desire at all) or physical ability toperform sexual activity at any time
Mobility of
The extreme should be that all of their major joints
joint functions are completely fixed
Muscle powerfunctions

The extreme should be that all of their major muscles
are completely incapable of movement

・The extent of the problem
・The frequency of the problem
・The number of site with pain
・The extent of the problem
・The frequency of the problem
・The extent of the problem
・The frequency of the problem
・The extent of the problem
・The frequency of the problem
・The extent of the problem
・The ratio of the joint whichhave the problem
・The extent of the problem
・The ratio of the joint whichhave the problem

Qualifiers
0 No problem
1 Mild problem: may include a problem which doesn't affect the patient's daily activities
2 Moderate problem: may include a problem which exceeds "1" but still remains a relatively minor problem
in the given category
3 Severe problem: may include a problem which is a major problem in the given category
4 Complete problem
8 Not specified
9 Not applicableinclude problem that is major in given category), 4:
complete problem [16].
The weighted kappa statistics were used todetermine interrater agreement between clinicians andpatients for each category. To examine the difference
between patients with brain lesions and others, we
conducted a comparison between the two groups ofpatients. We also evaluated the distribution andaverage total scores of clinicians’ and patients’evaluations, where intraclass correlation coefficient(ICC (2,1)) was used to determine the agreement in
total scores between clinicians and patients.
3. Data analysis
The weighted kappa statistics and ICC (2,1) were
Jpn J Compr Rehabil Sci Vol 11, 2020

used to determine interrater agreement between
clinicians and patients in each category and the total
score. The weighted kappa statistics with linear weight
were calculated for each category. The qualifier
response options of 8 for “not specified” and 9 for “notapplicable” were not included in the calculation of
kappa statistics and were considered missing data. The
following standards for interpreting kappa statistics
presented by Landis and Koch were employed in ourinterpretation of the results: <0.20, poor; 0.21‐0.40,
fair; 0.41‐0.60, moderate; 0.61‐0.80, substantial; and>0.81, excellent [18]. For interpretation of the ICC(2,1), the evaluation criteria of Cicchetti et al. wereused (<0.40, poor; 0.40‐0.59, fair; 0.60‐0.74, good;>0.75, excellent [19]).

Senju Y et al.: Clinician ratings and patient self-assessments of b-categories of ICF Rehabilitation Set 13

Table 5. Weighted kappa statistics and ICC (2,1).
Missing value
Categories
b130
b134
b152
b280
b455
b620
b640
b710
b730

Energy and drive functions
Sleep functions
Emotional functions
Sensation of pain
Exercise tolerance functions
Urination functions
Sexual functions
Mobility of joint functions
Muscle power functions

Clinician ratingsn (%)

Patient self-reportsn (%)

Weighted kappa

0 (0.0)0 (0.0)0 (0.0)2 (2.3)0 (0.0)0 (0.0)31 (35.2)
0 (0.0)0 (0.0)0 (0.0)0 (0.0)0 (0.0)3 (3.5)2 (2.3)0 (0.0)22 (25.0)
2 (2.3)2 (2.3)

0.72
0.74
0.58
0.69
0.63
0.87
0.67
0.70
0.62
ICC (2,1)

Total

31 (35.2)

4. REDCap
These study data were collected and managed using
REDCap (Research Electronic Data Capture) electronic
data capture tools hosted at Fujita Health University[20]. REDCap is a secure, web-based application
designed to support data capture for research studies.
The application provides 1) an intuitive interface forvalidated data entry, 2) audit trails for tracking data
manipulation and export procedures, 3) automatedexport procedures for seamless data downloads tocommon statistical packages, and 4) procedures forimporting data from external sources.
Results
The results of interrater agreement between clinician

22 (25.0)

0.85

ratings and patient self-assessments for body function
categories are shown in Table 5. The weighted kappa
statistics ranged from 0.58 to 0.87. Eight out of nine
categories presented high weighted kappa statistics,
and these results were classified as substantial to
excellent. Only the weighted kappa statistics of “b152
Emotional functions” were classified as moderatereliability. There was a category (“b620 Urination
functions”) with more than 0.81 of weighted kappa
statistics classified as excellent in agreement. A
marked number of missing values was observed in“b640 Sexual functions”, which presented with 35.2%
missing values in clinicians’ evaluations and 25.0% inpatients’ evaluations; nevertheless, other items showed
less than 5% of missing values.
The ICC (2,1) of the total score of all categories
excluding missing value was 0.85, which was classified
as excellent. The distributions of the total score of
Table 6. Weighted kappa statistics with and without
brain lesion.
Weighted kappa
With brain
Without
lesionbrain lesion

Figure 1. The total scores of the clinician ratings andthe patient self-assessments.
The correlation coefficient between the total score ofclinician ratings and the patient self-assessments was
0.85, and ICC (2,1) was 0.85. r: Pearson’s correlation
coefficient, ICC: intraclass correlation coefficient.
b130 Energy and drive
functionsb134 Sleep functionsb152 Emotional functions
b280 Sensation of painb455 Exercise tolerancefunctionsb620 Urination functions
b640 Sexual functions
b710 Mobility of joint
functionsb730 Muscle powerfunctions

0.63

0.90

0.71
0.50
0.65
0.52

0.76
0.78
0.76
0.77

0.86
0.67
0.68

0.89
1.00
0.71

0.58

0.73

Jpn J Compr Rehabil Sci Vol 11, 2020

14

Senju Y et al.: Clinician ratings and patient self-assessments of b-categories of ICF Rehabilitation Set

clinician ratings and patient self-assessments areshown in Figure 1. There was no significant difference
in total score between clinician ratings and patient
self-assessments (clinician ratings: 5.6 ± 4.5, patient
self-assessments: 5.2 ± 4.1). The weighted kappa
statistic of patients with and without brain lesion are
shown in Table 6. In the patients with brain lesions,interrater agreement between the rating of clinicians
and patients tended to be generally low compared tothe patients without brain lesions.
Discussion
In this study, we examined the agreement inevaluation between clinician ratings with the Ratingreference guide and patient self-assessments for thenine body function categories of the ICF Rehabilitation
Set. Eight out of nine categories presented highweighted kappa statistics, and these results wereclassified as substantial to excellent. The ICC (2,1)
was more than 0.75, which was classified as excellent.
Thus, there was relatively good clinical rating
agreement between clinicians and patients in thesecategories. This would indicate that the clinician
ratings using the Rating reference guide properlyreflect the problems the patients experience and arethus clinically useful.
The weighted kappa statistic of “b152 Emotional
functions” was 0.58, which was relatively lower thanthe other categories. This may be due to clinicians’
insufficient understanding of patients’ emotional
functions, which may come from the patients’ lack of
expression or clinicians’ insufficient observation of the
problems in emotional functions. Another possible
reason could be the existence of anosognosia on thepart of the patients, which may result in the clinicians’
poor evaluation of patients’ problems [21‐23]. Patientswith obvious cognitive problems were excluded fromthe study. However, approximately 50% of our target
included patients with neurological diseases such as
stroke, and it is possible that they had mild cognitive
problems (e.g., anosognosia) that may not have beenreflected in the MMSE. The high interrater agreementof “b152 Emotional functions” in patients without
brain lesions and the low interrater agreement of“b455 Exercise tolerance functions” and “b730 Muscle
power functions”, where clinician ratings relied on
objective information, might indicate the influence ofcognitive problems on the difference in ratings
between clinicians and patients.
The previous studies suggest that clinicians tend tounderestimate the problems of body function
categories, particularly in evaluating those categories
strongly influenced by subjectivity. For example, it isreported that clinicians often underestimate pain
compared to patients’ evaluations [24]. It has also
been reported that clinicians tend to underestimate
psychological problems such as depression [25].
Jpn J Compr Rehabil Sci Vol 11, 2020

These reports indicate the difficulties faced byclinicians when estimating the extent of subjective
problems in the functioning of patients. However, in
this study, there was good agreement between clinician
ratings and patient self-assessments for each category.
In addition, the ICC (2,1) of total points was in good
agreement with 0.85, and the average of total score ofclinician ratings and patient self-assessments showedno significant difference. The Rating reference guideused in this study was developed while taking several
aspects into account, including the frequency andextent of functioning problems, in rating each category.
The Rating reference guide also refers to the influenceof the functioning problem on daily activities that arelikely to strongly affect the subjective evaluation of
the problem by patients (0: no problem, 1: mild
problem (may include problem that does not affectdaily activities), 2: moderate problem (may include
problem that exceeds “1” but still remains a relatively
minor problem in the given category), 3: severe
problem (may include problem that is a major problem
in given category), 4: complete problem). Therefore, it
may be easier to agree with the patients’ subjective
evaluations than when simply focusing on single
aspects such as the severity of the symptoms andextent of functioning problems.
Limitation
In this study, the raters in the field study consisted ofexperienced rehabilitation experts. As previous studies
have shown, lack of clinical experience may lower
interrater agreement in the rating of clinical scales[26‐28]. Thus, the results may not be replicated withless experienced clinicians evaluating patients’
functioning and should be further investigated.
In “b640 Sexual functions”, missing values werehigh for both clinicians and patients. One possible
reason could be that this category includes moreprivate information than the other categories; thus,
both clinicians and patients faced strong resistancewhen rating this category. Although the weighted
kappa statistic was 0.67 and classified as moderate inagreement, the percentage of patients rated 0 by bothclinicians and patients reached 84%, which was muchhigher than the other categories. Because sexualfunction is not a function required in daily life, thefrequency of opportunities for sexual activity may
affect the detection of the sexual function problem. It
is also possible that clinicians and patients scored 0
because subjects were relatively old, 65 years on
average, and many were inpatients; thus, bothclinicians and patients had few opportunities to obtainobjective clinical findings and subjective symptoms
that could be used as a reference for evaluation. In
evaluating interrater agreement between clinician
ratings and patient self-assessments for sexualfunction, a study of younger patients with higherSenju Y et al.: Clinician ratings and patient self-assessments of b-categories of ICF Rehabilitation Set 15

sexual function needs may be necessary.
Conclusion
In this study, we investigated interrater agreement
between clinician ratings using the Rating reference
guide and patient self-assessments for the nine bodyfunction categories of the ICF Rehabilitation Set andevidenced good agreement in rating. Our findings
support the feasibility of clinician ratings using theRating reference guide for body function categories asa means of describing the functioning problems fromwhich patients suffer.
Acknowledgments
We would like to express our gratitude to themembers of the Department of Rehabilitation of Fujita
Health University Hospital and the Department of
Rehabilitation of Fujita Health University Nanakuri
Memorial Hospital for their cooperation in this study.
This research was supported by the Grants for
Research on Health and Welfare for the research
project titled “The establishment of a mechanism forevaluation and information sharing using theInternational Classification of Functioning, Disabilityand Health to integrate medical and long-term careservices” (Principal Investigator: Masahiko Mukaino).

References

1. World Health Organization (WHO). Internationalclassification of Functioning, Disability and Health. 1st
ed. Geneva; 2001.
2. Stucki G, Prodinger B, Bickenbach J. Four steps to follow
when documenting functioning with the International
Classification of Functioning, Disability and Health. Eur J
Phys Rehabil Med 2017; 53: 144‐9.
3. Selb M, Escorpizo R, Kostanjsek N, Stucki G, Ustun B,
Cieza A. A guide on how to develop an International
Classification of Functioning, Disability and Health Core
Set. Eur J Phys Rehabil Med 2015; 51: 105‐17.
4. Ewert T, Grill E, Bartholomeyczik S, Finger M,
Mokrusch T, Kostanjsek N, et al. ICF Core Set for patientswith neurological conditions in the acute hospital. Disabil
Rehabil 2005; 27: 367‐73.
5. Grill E, Ewert T, Chatterji S, Kostanjsek N, Stucki G. ICF
Core Sets development for the acute hospital and earlypost-acute rehabilitation facilities. Disabil Rehabil 2005;
27: 361‐6.
6. Grill E, Hermes R, Swoboda W, Uzarewicz C, Kostanjsek
N, Stucki G. ICF Core Set for geriatric patients in earlypost-acute rehabilitation facilities. Disabil Rehabil 2005;
27: 411‐7.
7. Wildner M, Quittan M, Portenier L, Wilke S, Boldt C,
Stucki G, et al. ICF Core Set for patients withcardiopulmonary conditions in early post-acuterehabilitation facilities. Disabil Rehabil 2005; 27:
397‐404.
8. Boldt C, Grill E, Wildner M, Portenier L, Wilke S, Stucki
G, et al. ICF Core Set for patients with cardiopulmonary
conditions in the acute hospital. Disabil Rehabil 2005; 27:
375‐80.
9. Cieza A, Oberhauser C, Bickenbach J, Chatterji S,
Stucki G. Towards a minimal generic set of domains offunctioning and health. BMC Public Health 2014; 14:218.
10. Prodinger B, Cieza A, Oberhauser C, Bickenbach J, Ustun
TB, Chatterji S, et al. Toward the International Classificationof Functioning, Disability and Health (ICF) Rehabilitation
Set: a minimal generic set of domains for rehabilitation as ahealth strategy. Arch Phys Med Rehabil 2016; 97: 875‐84.
11. Gimigliano F, Selb M, Mukaino M, Baffone C, Bickenbach
J, Patrick JE, et al. Strengthening rehabilitation in healthsystems worldwide by implementing information on
functioning in rehabilitation practice, quality management,
and policy: 2018 status report. J Int Soc Phys Rehabil Med
2018; 1: 37.
12. Frontera W, Gimigliano F, Melvin J, Li J, Li L, Lains J, etal. ClinFIT: ISPRM’s Universal Functioning Information
Tool based on the WHO’s ICF. J Int Soc Phys Rehabil Med
2019; 2: 19.
13. Prodinger B, Reinhardt JD, Selb M, Stucki G, Yan T,
Zhang X, et al. Towards system-wide implementation of
the International Classification of Functioning, Disabilityand Health (ICF) in routine practice: developing simple,
intuitive descriptions of ICF categories in the ICF Genericand Rehabilitation Set. J Rehabil Med 2016; 48: 508‐14.
14. Selb M, Gimigliano F, Prodinger B, Stucki G, Pestelli G,
Iocco M, et al. Toward an International Classification of
Functioning, Disability and Health clinical data collection
tool: the Italian experience of developing simple, intuitive
descriptions of the Rehabilitation Set categories. Eur J
Phys Rehabil Med 2017; 53: 290‐8.
15. Uhlig T, Lillemo S, Moe RH, Stamm T, Cieza A, Boonen
A, et al. Reliability of the ICF Core Set for rheumatoid
arthritis. Ann Rheum Dis 2007; 66: 1078‐84.
16. Mukaino M, Saitoh E, Otaga M. Development of ICFbased data collection tools and system for information
sharing between medical services and long term careservices. Health and Labor Sciences Research Grant
Report 201802007A; 2019. p. 21‐7. Japanese.
17. Bickenbach J. Reconciling the capability approach andthe ICF. Alter 2014; 8: 10‐23.
18
. Landis JR, Koch GG. The measurement of observer
agreement for categorical data. Biometrics 1977; 33: 159‐74.
19. Cicchetti DV. Guidelines, criteria, and rules of thumb forevaluating normed and standardized assessment instrumentsin psychology. Psychol Assess 1994; 6: 284.
20. Harris PA, Taylor R, Thielke R, Payne J, Gonzalez N,
Conde JG. Research electronic data capture (REDCap)―a
metadata-driven methodology and workflow process forproviding translational research informatics support. J
Biomed Inform 2009; 42: 377‐81.
21. Verhulsdonk S, Quack R, Hoft B, Lange-Asschenfeldt
C, Supprian T. Anosognosia and depression in patientswith Alzheimer’s dementia. Arch Gerontol Geriatr 2013;
Jpn J Compr Rehabil Sci Vol 11, 2020

16

Senju Y et al.: Clinician ratings and patient self-assessments of b-categories of ICF Rehabilitation Set

57: 282‐7.
22. Conde-Sala JL, Turro-Garriga O, Pinan-Hernandez S,
Portellano-Ortiz C, Vinas-Diez V, Gascon-Bayarri J, et al.
Effects of anosognosia and neuropsychiatric symptoms on
the quality of life of patients with Alzheimer’s disease: a
24-month follow-up study. Int J Geriatr Psychiatr 2016;
31: 109‐19.
23. Starkstein SE, Jorge RE, Robinson RG. The frequency,
clinical correlates, and mechanism of anosognosia after
stroke. Can J Psychiatry 2010; 55: 355‐61.
24. Solomon P. Congruence between health professionals’
and patients’ pain ratings: a review of the literature. Scand
J Caring Sci 2001; 15: 174‐80.
25. Zastrow A, Faude V, Seyboth F, Niehoff D, Herzog W,

Jpn J Compr Rehabil Sci Vol 11, 2020

Lowe B. Risk factors of symptom underestimation byphysicians. J Psychosom Res 2008; 64: 543‐51.
26. Carr EK, Kenney FD, Wilson-Barnett J, Newham DJ.
Inter-rater reliability of postural observation after stroke.
Clin Rehabil 1999; 13: 229‐42.
27. Brunnekreef JJ, van Uden CJ, van Moorsel S, Kooloos JG.
Reliability of videotaped observational gait analysis inpatients with orthopedic impairments. BMC Musculoskelet
Disord 2005; 6: 17.
28. Starrost K, Geyh S, Trautwein A, Grunow J, CeballosBaumann A, Prosiegel M, et al. Interrater reliability of theextended ICF core set for stroke applied by physicaltherapists. Phys Ther 2008; 88: 841‐51.