Findings indicated that although studies generally met st and ards in several quality areas , there is still room for improvement .
Areas for improvement include establishing representativeness , adequately describing clinical and demographic characteristics of the sample , and presenting better descriptions of important aspects of test and reference st and ard execution .
In addition , study quality has not appreciably improved since the publication of the STARD Statement in 2003 .
Despite the popularity of the PTSD Checklist ( PCL ) as a clinical screening test , there has been no comprehensive quality review of studies evaluating its diagnostic accuracy .
Interrater reliability in assessing quality of diagnostic accuracy studies using the QUADAS tool. A preliminary assessment.
RATIONALE AND OBJECTIVES Quality Assessment of Diagnostic Accuracy Studies ( QUADAS ) is a new tool to measure the method ological quality of diagnostic accuracy studies in systematic review s. We used data from a systematic review of magnetic resonance spectroscopy ( MRS ) in the characterization of suspected brain tumors to provide a preliminary evaluation of the inter-rater reliability of QUADAS . MATERIAL S AND METHODS A structured literature search identified 19 diagnostic accuracy studies . These publications were distributed r and omly to primary and secondary review ers for dual independent assessment . Review ers recorded method ological quality by using QUADAS on a custom- design ed spreadsheet . We calculated correlation , percentage of agreement , and kappa statistic to assess inter-rater reliability . RESULTS Most studies in our review were judged to have used an accurate reference st and ard . Conversely , the MRS literature frequently failed to specify the length of time between index and reference tests or that the clinicians were unaware of the index test findings when reporting the reference st and ard . There was good correlation ( rho = 0.78 ) between review ers in assessment of the overall number of quality criteria met . However , mean agreement for individual QUADAS questions was only fair ( kappa = 0.22 ) and ranged from no agreement beyond chance ( kappa < 0 ) to moderate agreement ( kappa = 0.58 ) . CONCLUSION Inter-rater reliability in our study was relatively low . Nevertheless , we believe that QUADAS potentially is a useful tool for highlighting the strengths and weaknesses of existing diagnostic accuracy studies . Low reliability suggests that different review ers will reach different conclusions if QUADAS is used to exclude " low- quality " articles from meta-analyses . We discuss methods for improving the validity and reliability of QUADAS
Findings indicated that although studies generally met st and ards in several quality areas , there is still room for improvement .
Areas for improvement include establishing representativeness , adequately describing clinical and demographic characteristics of the sample , and presenting better descriptions of important aspects of test and reference st and ard execution .
In addition , study quality has not appreciably improved since the publication of the STARD Statement in 2003 .
Despite the popularity of the PTSD Checklist ( PCL ) as a clinical screening test , there has been no comprehensive quality review of studies evaluating its diagnostic accuracy .