Reply to Moult et al.

Here we address in brief several criticisms of our paper offered by Moult et al. We do not bring up the many points of agreement already mentioned by Moult et al., although they are greatly appreciated.In our short paper, we did not aim to analyze the state of comparative modeling, nor to propose specific criteria for assessing the accuracy of comparative modeling. Instead, we described how to assess the statistical significance of ranking of methods given a model quality criterion. Applying the ranking method and one particular model quality criterion that has in fact been used previously, we illustrated difficulties with ranking of comparative modeling methods at CASP4. We suggested that CASP use some measures of statistical significance for whatever model quality criteria are adopted. Their criticism of our model quality criterion appears to be missing these points.We neither state nor imply in our paper that “the production of an unambiguous ranking of all modeling groups” is a major goal of the CASP experiment. However, it is worth mentioning that a specific ranking is a dominant outcome of CASP, despite the organizers' declared intentions.Moult et al. suggest that if our scheme were applied at CASP, unreasonable ranking would ensue in some cases, such as the ranking of method 526 higher than method 384. However, we do not rank the two methods with respect to each other (Figure 2), since they share less than two models in common. The two methods are simply not comparable. This example actually reinforces our point that ranking of methods requires a sufficient number of common models.We do not “insist that we must have more data so as to identify the small differences in the adeptness of the different groups” at CASP. We repeatedly stated that either a larger number of models, a larger average model quality difference, or a smaller standard deviation in the model quality difference are needed for reliable ranking. We showed that the current number of models is insufficient to make strong statements about the ranking of the methods at CASP4 given their differences. We also suggested that a larger number of models is required because modeling methods need to be assessed by a variety of criteria (see the next paragraph).While Moult et al. agree with us that there are not enough models at CASP to assess comparative methods with statistical significance, they nevertheless claim that the field is “stuck”. Even though we have not addressed ranking of methods applied to different small sets of targets at different CASP meetings, we believe it is not possible to say that there are no differences in accuracy between the comparative modeling methods applied at CASP2–4. For example, there have been only a handful of models at each CASP meeting for which the modeling of loops and side chains is not overwhelmed by the alignment errors. Thus, it is almost certainly impossible to make any meaningful statements about loop modeling and side chain modeling based on the CASP data. Outside of CASP, loop modeling and side chain modeling are the limiting factors in a number of important comparative modeling applications. As a consequence, there has not been “plenty enough CASP data to identify” major advances in comparative modeling.We hope that our analysis helps both developers and users objectively assess the many existing protein structure prediction methods.