Randomized controlled trials in occupational therapy and speech therapy : a comparative analysis by expanding the CONSORT statement

Background: Research in occupational therapy is a critical ingredient for the validation of the profession. To be valuable, research studies must be well-designed, and evidence must be gathered by systematic and structured methods. Randomized controlled trials (RCT) provide control over intervention conditions and reduce the risk of not finding statistical significance. Objectives: The overall purpose of the current study is to evaluate the quality of reporting RCTs in occupational therapy and speech therapy using an expansion of the CONSORT statement. Methods: The Nelson-Moberg-Norton Expanded CONSORT Instrument (NMNECI) was used to evaluate 15 occupational therapy RCTs and 15 speech therapy RCTs. The student researcher evaluated all 30 articles using the NMNECI. To test the inter-rater reliability of the instrument a masked research assistant evaluated 16 randomly selected RCTs. Results: An ICC of . 93, “almost perfect” according to Eliasziw et al. (1994), was calculated as the overall test of agreement between the student researcher and research assistant. On average, the 30 articles complied with 119.5 NMNECI sub-items (SD = 25.48) of the 212 possible sub-items. Occupational therapy RCTs (M = 130.07, SD = 20.21) complied with CONSORT more than speech therapy RCTs (M = 108.93, SD = 26.40), t(28) = 2.46, p = 0.02. Conclusion: RCTs in occupational therapy and other allied health professions could benefit from using the CONSORT statement as a tool for reporting future research studies. Randomized Controlled Trials 3 Randomized Controlled Trials in Occupational Therapy and Speech Therapy: A Comparative Analysis Using the CONSORT Statement Research in occupational therapy by occupational therapists is a critical ingredient to the credibility of the profession (Nelson, 1997). According to Holm (2000, p.582), there will be a “progressive decline in [occupational therapist’s] clinical competency” without continually developing and refining the skills necessary for utilizing research evidence. According to the American Occupational Therapy Association’s (AOTA) Standards for Practice (1998), an occupational therapist must be knowledgeable about the current research in their area of practice. They must apply these findings in their interventions and must “demonstrate an understanding of current literature” (AOTA Continuing Competency, 1999, p. 599), The AOTA Code of Ethics (2000) further reinforces this petition by stating that it is an occupational therapist’s responsibility to “keep current with emerging knowledge . so they may perform their duties on the basis of accurate information” (p. 123). Research studies must be well designed and evidence must be gathered by systematic and structured methods “that are accepted by the larger research community” (Nelson 1997, p. 21). Through carefully designed and carried out research studies occupational therapists can confidently provide the best care for their patients. Evidence-based practice (EBP) has become a major focus in occupational therapy research and practice. Tickle-Degnen (1999) defined EBP as “a toolbox consisting primarily of methods designed to integrate research study evidence into the clinical reasoning process” (p. 537). There has been a recent influx of EBP research occupational therapy; however, as Holm (2000) cautioned, quantity does not equal Randomized Controlled Trials 4 quality. Ottenbacher and Maas (1999) conducted a study to determine the effectiveness of 30 occupational therapy studies. They determined that many interventions had small effect sizes and low statistical power. This can result in inconclusive evidence for the efficacy of the intervention, and the possibility of a Type II error being committed. Type II error occurs when a research outcome fails to report significant effects for the intervention when the intervention was actually effective. Discussing the need for agreement concerning research quality, Ottenbacher and Maas (1999) warned that “a failure to establish consensus means a failure to develop guidelines for evidence-based practice” (p. 187). In the 2000 Eleanor Clark Slagle lecture, Holm (2000) presented a hierarchy for evaluating research studies. The highest level includes systematic reviews and metaanalyses of randomized controlled trials (RCTs). These reviews can be accessed in articles and in databases such as the Cochrane Database. The second level, and the concern of this study, consists of well-designed RCTs. Lower levels of research design result in data with less credibility. RCTs have been used in the medical community for many years and are considered the “gold standard” of research reporting. A RCT has two or more (including a control) possible interventions. Participants are then randomly assigned to one of the conditions (Friedman, Furberg, & DeMets, 1998). Outcomes of the RCT involve enhanced health for participants resulting from the intervention. RCTs have the potential to provide more control over intervention conditions than lower-level research designs and thus reduce the risks of Type I and Type II error. Tickle-Degnen and Bedell (2003) argued, “The well-designed RCT provides a strong demonstration that Randomized Controlled Trials 5 research participants’ outcomes are caused by having received or not received a tested intervention, not caused by other factors” (p. 235). The Consolidated Standards of Reporting Trials (CONSORT) statement was designed to assist readers in determining the quality of RCT reports. It was developed by an international team of researchers in the mid-1990’s and was revised in 2001. According to the revised CONSORT statement (Altman, Schultz, Moher, Egger, Davidoff, Elbourne et al., 2001), “The objective of CONSORT is to facilitate critical appraisal and interpretation of RCTs by providing guidance to authors about how to improve the reporting of their trials” (p. 664). CONSORT provides authors with a list of 22 essential items to report in their trials. Guidelines for randomization methods (CONSORT 8-10), blinding (masking) (CONSORT 11) of assessors, participants, and interventionists, and clearly stated intervention procedures (CONSORT 4) help readers assess the risk for potential biases in administration of interventions. Additional information about the CONSORT statement is available at http://www.consortstatement. org. Since its publication, several medical journals have adopted the CONSORT statement as their standard for reporting RCTs. In a study by Moher, Jones, and Lepage (2001), four medical journals were tested to determine how the adoption of CONSORT changed the reporting of RCTs in their publications. Three of the four journals adopted the CONSORT; the other served as a control. The authors reported that though there was an increase in CONSORT items reported in all four journals, the journals adopting the CONSORT reported significantly more CONSORT items than the control. Randomized Controlled Trials 6 Stinson, McGrath, and Yamada (2003) applied the CONSORT statement to two psychology journals. The aim of the study was to determine how well the RCTs found in the journals followed the statement and to determine what items were not applicable to studies in psychology. Results indicated that articles in both journals failed to consistently report CONSORT items. They also reported that studies not adequately reporting randomization concealment or blinding found larger treatment effects than those that did. This may indicate that a Type I error occurred (finding that the treatment was successful when it really was not). They also contended that the majority of CONSORT items could be applied to most psychology interventions. However, some items such as blinding of subjects or interventionists and reporting systematic reviews to interpret findings were reported as “difficult for many clinical trials of psychosocial interventions” (p. 164). The authors concluded by stating that CONSORT items can improve the reporting of RCTs in psychology but recommended further modifications to include items based on “treatment fidelity and clinical significance” (p. 166). Applying the CONSORT to occupational therapy RCTs is the next step in the process of improving the quality of occupational therapy research. Nelson and Mathiowetz (2004) encouraged the use of the CONSORT statement to guide future occupational therapy RCTs and to evaluate the quality of published RCTs. They expressed concerns that there were some limitations when using the statement in some fields, especially occupational therapy. For example, it is impossible to mask interventionists or participants to the intervention they are receiving (CONSORT 11). Another concern is that in many areas of occupational therapy little research has been done to provide a complete background and rationale for the study (CONSORT 1). A Randomized Controlled Trials 7 final concern noted was the inability to capture the dynamics and uniqueness of each therapeutic relationship (CONSORT 4). These factors combined with the recent introduction of RCTs into the field make it difficult to conduct RCTs in occupational therapy but reinforce the importance of evaluating research in a standardized way. Moberg-Mogren and Nelson (in press) evaluated RCTs relevant to occupational therapy based on the Nelson-Moberg Expanded CONSORT Instrument (in press). This instrument was developed by Nelson and Moberg based on the Explanatory Document published by the authors of the CONSORT group (Altman, Schultz, Moher, Egger, Davidoff, Elbourne et al., 2001). This instrument expanded the 22-original CONSORT statement into 201 sub-items. Each original CONSORT item was expanded to include a list of sub-items that were considered essential or desirable by the Revised CONSORT Explanatory Document (Altman et al., 2001). For example, according to the original CONSORT item 2, the authors would receive one point for providing readers with “Scientific Background and Explanati