Reviewer's report
Title:Reliability of a wellness inventory for use among adolescent females aged
12-14 years
Version:2Date:1 April 2014
Reviewer: Elisabeth Svensson
Reviewer's report:
Manuscript: Reliability of a wellness inventory for use among adolescent females
aged 12-14 years
My assignment is to review the statistical methods used in this study, and since I
am a statistical and methodological expert regarding the development of
questionnaires, evaluations of quality aspects and analysis of complex problems
involving assessments on scales I have read the manuscript with great interest.
After a comprehensive review focusing at methodological and statistical issues I
have some comments that are essential for the quality of the study.
Major Compulsory Revisions
1 The aim:
According to the authors, the 5F-Wellness is a self-report instrument that is very
commonly used in various types of studies, and the aim of this study is to
evaluate the test-retest reliability of the 5F-Well instrument when self-completed
by adolescent females..
The use of the concept reliability is not completely correct.
Test-retest reliability, in this case intra-rater reliability) refers to the stability in
test-retest assessments made by the same individual, in this case female
adolescent.
Reliability refers to the quality of the assessments by the 5F-Wel, which means
the stability in interpretation of the items and item scales. In this case it might
also refer to the stanbility in the sum scores used as results of assessments. So
there is many different aspects to consider when designing a reliability study.
These issues must be considered at least in the Methods section and later on as
reflections in the discussion section.
Evaluation of reliability is study specific, which means that it is very important to
clearly describe the inclusions criteria for participating in this reliability evaluation.
Consequently, in this case, the aim of a study can never be to evaluate the
test-retest reliability of the 5F-Wel instrument, only to evaluate the test-retest
reliability in repeated assessments made by individuals representing a specified
population.
The Methods:
The method’s sub-sections are fragmented described and I recommend the
following order.
The Instrument, The study design and participants, The data analysis
And I will give my comments accordingly:
2 The instrument
The uses of the 5F-Wel instrument are mentioned in the Background and also in
in the methods section, but the description of the structure of the instrument
could be somewhat better described. What is the main variable? (wellness?) The
main variable is operationally defined to consist of dimensions and
sub-dimensions and each sub-dimension is assessed by statements to which the
females should disagree or agree. Also should an example of item and scale be
given, for the reader to understand the type of assessments that creates the data
sets.
3. The assessments of the level of (dis)agreements to each statement generate
ordered categorical data. However, in this study, and in the other studies
involving 5F-Wel the non-metric properties (lack of equidistance between
categories, lack of standard magnitude of categories, strongly agree+ agree =
disagree???) of the scale assessments are ignored and different sum scores are
calculated- obviously according to the manual and according to the main stream
of users of this instrument. However, in a research paper, one should consider
the measurement properties of the data collected, and also critically consider the
consequences of the choice of manipulating the subjective assessments to
“context scores”, “overall life satisfaction index”, and the transformation to sum
scores ranging 25 to 100. How are these scores related to the dimensions,
variables measured?
The treatment of data is the authors’ choice, but at least show that it is an active
choice and that you are aware of the consequences on the quality of the results.
4. The study design and participants
As mentioned the inclusion criteria of the group of adolescent females must be
defined. The description of the group is very good in the discussion section, but
types of schools etc should be given in this section.
Why 48 individuals? How was the group size determined and how was the
females chosen?
The procedure is described.
Data analysis
As in many studies there is an over-reliance on the SPSS!
5 Test-retest reliability refers to agreement in the test and retest assessments
made on each item 4-point scale. I miss the very simple but most informative
measures of agreement (Percentage agreement PA) in such test-retest
assessments. These measures are the most reliable results of this study- and
also interpretable. It is also possible for the SPSS to deliver these PA-values.
There will be 97 of them but one can also create PA-measures for
sub-dimensions etc as well.
6. The use of Bland-.Altman limits of agreements is not appropriate. This
approach is excellent for normally distributed data. Normal distribution is a
probability distribution for quantitative data, as the authors pointed out in page 8.
Hence limits of agreement are not appropriate to use for ordered categorical
assessments even if sum scores of many assessments are used.
7. Reliability refers to agreement and not to association (correlation) and ICC is a
parametric
measure of association. On top of that the authors rely on the “hopeless arbitrary
values” for describing different levels of the result…..
8. Paired t-test is also used in other studies, as evident from the background.
However, paired t-test is a parametric test of paired data that are normally
distributed. This test is not appropriate to use and there is no reason for a
hypothesis test of possible systematic difference between the two
administrations. A description of the PA-values would have described this better
and correctly.
The eight last lines in this section should be deleted.
The choice of Bonferroni adjustment was positively notably.
The results
Text and table 1.
9. The p-values should not be included. There are no hypotheses presented.
More important, a “non-significant” p-value does not prove a null-hypothesis!!!
This is a serious mis-use of p-values. (The book Altman D. Practical statistics for
medical research is highly recommended)
p-values must not be used to conclude “no systematic difference”!
10. The ICC results do not describe test-retest reliability!
11. The test-retest assessments in this study produce 97 different item- sets of
pairs of data from 46 individuals. These 46 pairs of data contain the results of this
study.
Irrespective of the choice of approach to handling these data sets (PA, sum
scores, etc) it is completely wrong to present the data as if they were two
independent data sets of assessments as in table 1: mean, SD 1 and Mean SD
2.
12. As mentioned, the use of limits of agreement is inappropriate. However, if we
for a while ignore this and assume that the data are quantitative normally
distributed, he plots and the measures (NOT the p-values) reveal some very
interesting results regarding lack of reliability (agreement), the wide range of the
scores and also systematic differences.
13 It is a matter of choice and consequences: the statistical methods chosen
must be motivated, but also properly interpreted. The methods chosen are
inappropriate; the results are not correctly interpreted. This manuscript does not
prove the reliability in assessments on the 5F-Wel by the adolescent females.
However, I am convinced that the truth is hidden in the data sets - and destroyed
by the over-use of the SPSS. Please tell the SPSS to perform appropriate
calculations and plots and your research problem will be solved.
- Discretionary Revisions
The uses of the 5F-Wel instrument are mentioned in the Background and also in
in the methods section, but the description of the structure of the instrument
could be somewhat better described. What is the main variable? (wellness?) The
main variable is operationally defined to consist of dimensions and
sub-dimensions and each sub-dimension is assessed by statements to which the
females should disagree or agree. Also should an example of item and scale be
given, for the reader to understand the type of assessments that creates the data
sets.
Level of interest: An article whose findings are important to those with closely
related research interests
Quality of written English: Acceptable
Statistical review: Yes, and I have assessed the statistics in my report.
Declaration of competing interests:
I declare that I have no competing interests