Blinding images to sequence in osteoarthritis: evidence from other diseases

Osteoarthritis is a chronic disease that evolves slowly, and multiple longitudinal studies now ongoing are acquiring serial images of joints to track the structural progress of disease. When reading or analysing serial images using any method in which human judgment plays a part (as opposed to fully automated or computerized methods) it is generally accepted that viewing a subject’s images from different timepoints grouped together reduces measurement error by enabling the readers to re-evaluate and adjust measurements made on these images, taking into account variation in film quality, positioning and other factors1,2. However, a major question that still arises is whether the grouped images from a person should be evaluated in known chronological order or whether the images ought to be presented with the analyst or reader blinded to the sequence in which they were acquired (the order of films is not revealed). (For the purposes of this editorial, we shall call the first approach ‘known chronology’ and the second ‘blinded to sequence’.) The former scenario is similar to a clinical reading where a clinician or radiologist knows the sequence and tries to determine whether a patient has experienced a change in disease status. The latter procedure is one that has been instituted for some3,4, but not all5,6, clinical trials in osteoarthritis. Blinding to sequence is said to be conservative and rigorous and has been adopted by the osteoarthritis initiative in initial analyses of both radiographic and magnetic resonance images (MRI’s). The main rationale behind blinding to sequence is to reduce reader bias toward finding change in the expected direction and secondarily to minimize bias by readers who might be privy to information about risk factor status or treatment received. As noted above, these issues are relevant for any measurements from grouped serial images in which human judgment plays a role in determining the result, whether qualitative, semi-quantitative or quantitative. Even operations like segmentation of MRI images and digital measurements of joint space often have a manual component that can be influenced by whether the images are presented blinded to order or with known chronology. The potential for bias depends on which aspects of a method can be influenced by human judgment and to what extent. The FDA has no formal policy on whether images from trials should be read blinded to sequence or with known chronology. In angiography studies as one example where serial images are used to measure the primary outcomes (luminal narrowing), images are analyzed in a manual or semi-automated fashion with known chronology. In a recent policy paper on best practices in Medical Imaging Techniques for trials7, the FDA and PhRMA worked together to identify multiple different acceptable scenarios for image presentation including blinded to sequence or known chronology displays. As is noted, “the type of presentation often depends on the therapeutic area, the imaging technology being deployed, and the reasons for the review” of images. Given this uncertainty, we sought evidence on the performance of blinding to sequence vs reading with known chronology. Evaluating the pros and cons of blinding to sequence is not easy. Ideally, readers or image analysts would have to take a set of images acquired sequentially in patients and analyze them both ways, one blinded to sequence and the other with known chronology in order to evaluate the effect of blinding to sequence on estimates of rates of change. Ideally, there also should be a gold standard definition to which the reading or image analysis can be compared, allowing an evaluation of tradeoffs between the effect of blinding on rates of change vs accuracy of outcome classification. For such a comparison to be of relevance to osteoarthritis, the images should have similar characteristics to either X-rays or MRIs of the knees or other joints. Specifically, they should be images in which planar measurements are made on a 2D image or series of slices with inferences made on where the edges of structures are in three dimensions and whether, over time, there has been a change in the position of these edges relative to one another. That would define a change in joint space or loss of cartilage. We performed a MEDLINE search for articles published 1950 to current. Based on indexing of literature on blinding to sequence, we used indexing terms ‘observer variation’ or ‘diagnostic error’, these exploded terms were cross referenced with searches on the following content areas: spinal fractures, angiography, coronary angiography, rheumatoid arthritis (RA)/ra, duplex ultrasound carotid, osteoarthritis/ra (the suffix ‘ra’ refers to radiology studies). To find additional studies, we examined reference lists for the studies we found. After literature review, we found no studies evaluating reading blinded to sequence outside studies of spinal fractures and RA. To our knowledge, this has not been studied in osteoarthritis, and in vascular studies where serial images are read with known chronology, the issue has not arisen. Several studies, most of them of radiographs in RA, addressed these exact concerns. The one non-RA study examined serial spine radiographs in patients with osteoporosis being followed for vertebral fractures. While this study included a reference reading to which the techniques were compared, it compared a morphometric measure of vertebral fractures from paired spine radiographs of known chronology to assessment of non-paired, randomly presented, single spine films and not films paired but blinded to sequence. Compared with reference assessment, false positive rates for vertebral fractures using single films were 20% vs only 7% when paired films were read with known chronology. While not directly examining the question of our focus, Ross and colleagues stated, ‘We conclude that the assessments of X-rays for vertebral fractures in clinical trials should not be performed with the evaluator blinded to the sequence of the x rays8’.