In reply to Glick

To the Editor: We thank Dr. Glick for his letter.1 Dr. Glick raises several important questions related to our recently published paper.2 We can address the point regarding the selection of lesion model I trivially. Lesion model I was included in our paper2 as a cautionary tale, the lesion differs from the others (II–IV) in that it was not anatomically inspired. The 4 × 4 × 3 mm3 elliptical lesion was specifically added to demonstrate that lesion shape and size influenced detectability, and hence highlighted the need for calibration. If we had used the material properties of the other three lesions for this small ellipsoidal lesion, then it would have always been detected at the doses of the images in question, and thus it would not have been discriminative between systems. The main conclusions of our paper are based on lesion models II–IV, which are spiculated masses. These three lesion models closely matched those in the Rafferty trial3 (6–34 mm, median 13 mm). Lesion models II–IV have dimensions (calculated using the method 3D convex hull) 9 × 8 ×3, 10 × 14 × 4, and 15 × 15 × 4 mm3, respectively. As we stated in the paper2: “the detectability threshold must be scaled and calibrated accordingly. [Machine observers] should be trained using a realistic dataset that results in the proper response of case difficulty predicted by human readers,” and “structures with ellipsoidal shapes … affect the signal-to-noise ratio and [channelized Hotelling observers’] scores.” We can safely reduce the remaining concerns arising in Dr. Glick's letter to a single question “How closely does one need to mimic a predicate clinical trial in the design and execution of a virtual clinical trial (VCT)?” Or, more explicitly, “What simplifications are allowable in conducting a VCT that emulates a specific clinical trial?” This is something that Dr. Glick surely had to consider when his team published on their VCT of the Siemens pre-market approval (PMA) trial.4 One concern that Dr. Glick expressed is that we did not use a search observer. This choice, to not use a search observer, arose from a discussion we had with Dr. Kyle Myers more than a decade ago, when we first began performing VCTs and we sought to identify VCT methods, which were both efficient and accurate. A search task can be equated to an m-alternative forced choice (m-AFC) experiment in which the number of possible locations (m) is large.5 If one looks at observer performance between 2-AFC, 4-AFC, 9-AFC, and so on tasks, it is generally true that each of these experiments will yield a similar ranking, and that when the tasks are calibrated appropriately, then the observed improvement in area under the curve (AUC) (or d′) is also similar.5, 6 Thus, it has been our general practice to create phantoms with a large number of lesions per phantom, and to use location-known-exactly methods. This helps to reduce the computer power needed (and hence shorten the time) to conduct a VCT. It is worth noting that in Dr. Glick's study, they also used a location-known-exactly method.4 That said, we acknowledge that modeling search is important, as is shown by our other publications in this area.7, 8 Other differences exist between our VCTs and the reader trial3 we modeled. For example, the Rafferty trial3 compared digital mammography (DM) to the combination of digital breast tomosynthesis (DBT) and DM. Also, Rafferty did not use detection as an endpoint, but rather they used a combination of detection and discrimination in order to recommend women for additional imaging or biopsy. The Rafferty trial readers also used the combined information from all four views to make a decision, whereas we used a single view. These distinctions also largely apply to the VCTs conducted by Dr. Glick.4 However, we have never stated that we accurately and truthfully replicated Dr. Rafferty's clinical trial in our VCTs. Rather we state that based on our VCT results, we observe the same degree of improvement achieved with DBT as compared to DM.9 This conclusion is sufficient to justify continued research into VCTs, as we have repeatedly observed that DBT images contain more information about masses than DM, and that DBT images of calcifications have equivalent information to DM. As we have stated,9, 10 early VCTs were able to discern correctly whether one modality was superior to another, but today VCTs can also predict the degree of superiority. In summary, in our paper2 we describe a method of conducting VCTs in which we vary lesion size, shape, and conspicuity to explore differences in the performance of various modalities. We wrote this paper based upon our own cautionary experiences; that is, we have learned the hard way what happens if you fail to calibrate your VCTs. For example, we have learned that you should not create admixtures of obvious and obscure lesions, such that the obvious lesions are always detectable, and the obscure lesions are never detected. In this instance, you can match the VCT results to any AUC simply by varying the admixture (e.g., a 50/50 ratio will result in an AUC of 0.75). It is possible, then, to convince yourself that two systems are equivalent, when in fact they are not—you simply are not looking at lesions that are appropriately discriminative. It is also possible, when comparing differences in AUC alone, to show that two systems are equivalent when they are not, due to the nonlinear relationship between AUC and d′. For example, a 0.02 difference in AUC at 0.80 (i.e., 0.80 vs. 0.82) is equivalent to an 8.8% improvement in d′, whereas the same difference at an AUC of 0.97 (i.e., 0.97 vs. 0.99) is equivalent to a 23.7% improvement in d′. These two examples illustrate our reasoning behind the need to calibrate VCTs, and to report both AUC and d′. In conclusion, while we appreciate and respect Dr. Glick's comments, we do not believe they change the message of our paper. First, that VCTs can be designed to be predictive of both the sign and magnitude of the improvements seen between one system and another with proper selection (i.e., calibration) of imaging tasks. And second, to conduct VCTs that accurately predict these improvements, extreme care is needed in the design, validation, and conduct of these VCTs. The lesion insertion and calibration methods presented in our paper have served us well in achieving these goals. Respectfully, Bruno Barufaldi, Trevor L. Vent, Predrag R. Bakic, and Andrew D. A. Maidment The authors declare they have no conflicts of interest.