BACKGROUND
predicting protein tertiary structure from amino acid sequence is of great importance in bioinformatics and computational biology
<cit> .
during the last few decades, a lot of protein tertiary structure prediction methods have been developed.
one category of methods adopts a template-based approach
<cit> , which uses experimentally determined structures as templates to build structural models for a target protein without known structure.
another category uses a template-free approach
<cit> , which tries to fold a protein from scratch without using known template structures.
the two kinds of methods were often combined to handle a full spectrum of protein structure prediction problems ranging from relatively easy homology modeling to hard de novo prediction
<cit> .
during protein structure prediction, one important task is to assess the quality of structural models produced by protein structure prediction methods.
a model quality assessment  method employed in a protein structure prediction pipeline is critical for ranking, refining, and selecting models
<cit> .
a model quality assessment method can generally predict a global quality score measuring the overall quality of a protein structure model and a series of local quality scores measuring the local quality of each residue in the model.
a global quality score can be a global distance test  score
<cit>  that is predicted to be the structural similarity between a model and the unknown native structure of a protein.
a local quality score of a residue can be the euclidean distance between the position of the residue in a model and that in the unknown native structure after they are superimposed.
in general, protein model quality assessment methods can be classified into two categories: multi-model methods
<cit>  and single-model methods
<cit> .
multi-model methods largely use a consensus or clustering approach to compare one model with other models in a pool of input models to assess its quality.
generally, a model with a higher similarity with the rest of models in the pool receives a higher global quality score.
the methods tend to work well when a large portion of models in the input pool are of good quality, which is often the case for easy to medium hard template-based modeling.
multi-model methods tend to work particularly well if a large portion of good models were independently generated by a number of independent, diverse protein structure prediction methods as seen in the casp  experiments, but they worked less well when being applied to the models generated by one single protein structure prediction method because they prefer the average model of the largest model cluster in the model pool.
and multi-model methods tend to completely fail if a significant portion of low quality modes are similar to each other and thus dominate the pairwise model comparison as seen in some cases during the 10th casp experiment  held in  <dig>  single-model methods strive to predict the quality of a single protein model without consulting any other models
<cit> .
the performance of single-model methods is still lagging behind the multi-model methods in most cases when most models in the pool are of good quality
<cit> .
however, because of their capability of assessing the quality of one individual model, they have potential to address one big challenge in protein structure modeling – selecting a model of good quality from a large pool consisting of mostly irrelevant models.
furthermore, as the performance of multi-model quality assessment methods start to converge, single-model methods appear to have a large room of improvement as demonstrated in the casp <dig> experiment.
in order to critically evaluate the performance of multi-model and single-model protein model quality assessment methods, the casp <dig> experiment was designed to assess them in two stages.
on stage  <dig>   <dig> models of each target spanning a wide range of quality were used to assess the sensitivity of quality assessment methods with respect to the size of input model pool and the quality of input models.
on stage  <dig>  about top  <dig> models selected by a naïve consensus model quality assessment method were used to benchmark model quality assessment methods’ capability of distinguishing relatively small differences between more similar models.
the new settings provided us a good opportunity to assess the strength and weakness of our multi-model and single-model protein model quality assessment methods in terms of accuracy, robustness, consistency and efficiency in order to identify the gaps for further improvement.
in addition to evaluating our four servers on the casp <dig> benchmark, we compare our methods with three popular multi-model clustering-based methods .
our clustering-based methods  performed comparably to the three external tools in most cases.
our single-model methods  had a lower accuracy than the clustering-based methods, but performed considerably better than them on the models of hard template-free targets.
besides the reasonable performance and a comprehensive comparative study, our methods have some methodological innovations such as using single-model quality scores to weight models for clustering methods, repacking side chains before model evaluation, and improved machine learning methods for single-model quality assessment for template-free targets.
the rest of the paper is organized as follows.
in the results and discussions section, we analyze and discuss the performance of the methods on the casp <dig> benchmark.
in the conclusion section, we summarize this work and conclude it with the directions of future work.
in the methods section, we introduce the methods in our protein model quality assessment servers tested in casp <dig>
results and discussions
results of global quality predictions
we evaluated the global quality predictions using five measures .
the results of the global quality evaluation on stage  <dig> of casp <dig> are shown in table
<dig>  the weighted pairwise model comparison method multicom-construct performed best among all our four servers according to all the five measures, suggesting using single-model quality prediction scores as weights can improve the multi-model pairwise comparison based quality prediction methods such as multicom-refine.
the two multi-model global quality assessment methods had the better average performance than the two single-model global quality assessment methods  on average on stage  <dig>  suggesting that the advantage of multi-model methods over single-model methods was not much affected by the relatively small size of input models .
instead, the multi-model methods still work reasonably well on a small model pool that contains a significant portion of good quality models.
it is worth noting that the average loss of the two single-model quality assessment methods  is close to that of the two multi-model quality assessment methods  .
we also compared our methods with three popular multi-model clustering-based methods  on stage  <dig>  according to the evaluation, multicom-construct performed slightly better than the naive consensus method davis-qaconsensus and modfoldclust <dig>  while pcons performed best.
table
<dig> shows the global quality evaluation results on stage  <dig>  similarly as in table
<dig>  the weighted pairwise comparison multi-model method  performed better than the simple pairwise multi-model method  and both had better performance than the two single-model quality assessment methods .
that the two single-model quality prediction methods yielded the similar performance indicated that some difference in their input features  did not significant affect their accuracy.
in comparison with stage  <dig>  all our methods performed worse on stage  <dig> models.
since the models in stage  <dig> are more similar to each other than in stage  <dig> in most cases, the results may suggest that both multi-model and single-model quality assessment methods face difficulty in accurately distinguishing models of similar quality.
on stage  <dig> models, multicom-construct delivered a performance similar with davis-qaconsensus and pcons, and had a higher average correlation than modfoldclust <dig>
we used the wilcoxon signed ranked sum test to assess the significance of the difference in the performance of our four servers, davis-qaconsensus, pcons, and modfoldclust <dig>  the p-values of the difference between these servers are reported in table
<dig>  on stage  <dig> models, according to  <dig>  significant threshold, the difference between clustering-based methods  and single-model methods  is significant, but the difference between our methods in the same category is not significant.
one stage  <dig> models, the difference between all pairs of our servers except the two single-model methods is significant.
compared with the three external methods , the difference between our multi-model method multicom-refine and the three methods is not significant, while the difference between our single-model methods  and the three methods is significant.
the difference between multicom-construct and pcons is not significant, while the difference between multicom-construct and the other two external methods  is significant.
to elucidate the key factors that affect the accuracy of multi-model or single-model quality assessment methods, we plot the per-target correlation scores of each target on stage  <dig> against the ratio of the average real quality of the largest model cluster in the pool and the average real quality of all the models in the pool in figure
<dig>  to get the largest model cluster for each target, we first calculate the gdt-ts score between each pair of models, and then use  as the distance measure to hierarchically cluster the models.
finally, we use a distance threshold to cut the hierarchical tree to get the largest cluster so that the total number of models in the largest cluster is about one third of the total number of models in the pool.
figure
<dig> shows that the quality prediction accuracy  positively correlates with the average real quality of the largest model cluster divided by the average real quality of all models for two multi-model methods , whereas it has almost no correlation with single-model methods .
the results suggest that the performance of clustering-based multi-model methods depends on the relative real quality of the large cluster of models and that of single-model methods does not.
this is not surprising because multi-model methods rely on pairwise model comparison, but single-model methods try to assess the quality from one model.
as casp <dig> models were generated by many different predictors from around of the world, the side chains of these models may be packed by different modeling tools.
the difference in side chain packing may result in difference in input features  that affect the quality prediction results of single-model methods even though they only try to predict the quality of backbone of a model.
in order to remove the side-chain bias, we also tried to use the tool scwrl
<cit>  to rebuild the side chains of all models before applying a single-model quality prediction method - modelevaluator.
figure
<dig> compares the average correlation and loss of the predictions with or without side-chain repacking.
indeed, repacking side-chains before applying single-model quality assessment increased the average correlation and reduced the loss.
we did a wilcoxon signed ranked sum test on the correlations and losses of the predictions before and after repacking side-chains.
the p-value for average correlation before and after repacking side-chains on stage  <dig> is  <dig> , and on stage  <dig> is  <dig> .
the p-value for loss on stage  <dig> is  <dig> , and on stage  <dig> is  <dig> .
<cit>  is used for the side-chain repacking.
since mining a few good models out of a large pool of low-quality models is one of the major challenges in protein structure prediction, we compare the performance of single-model methods and multi-model methods on the models of several hard casp <dig> template-free targets.
tables
<dig> and
<dig> report the evaluation results of our four servers, davis-qaconsensus, pcons, and modfoldclust <dig> on all standalone template-free modeling  targets on stages  <dig> and  <dig>  i.e.
the targets whose domains are all fm domains.
the results show that the single-model methods  clearly performed better than the multi-model methods  on both stages.
they also performed better than the davis-qaconsensus and modfoldclust <dig> on both stages, achieved the similar performance with pcons on stage  <dig>  and the better performance than pcons on stage  <dig>  for instance, the average pearson’s correlation score of multicom-novel on stage  <dig> is  <dig> , which is much higher than  <dig>  of multicom-refine.
the multi-model methods even get low negative correlation for some targets.
for example, the pearson’s correlation score of multicom-refine on target t <dig> at stage  <dig> is - <dig> .
we use the tool treeview
<cit>  to visualize the hierarchical clustering of the models of t <dig> in figure
<dig>  the qualities of the models in the largest cluster are among the lowest, but they are similar to each other leading to high predicted quality scores when being assessed by multi-model methods.
the example indicates that multi-model methods often completely fail  when the models in the largest cluster are of worse quality, but similar to each other.
multi-model methods often perform worse than single-model methods when all models in pool are of low quality and are different from each other.
in this situation, the quality scores predicted by multi-model methods often do not correlate with the real quality scores, whereas those predicted by single-model methods still positively correlate with real quality scores to some degree.
as an example, figure
<dig> plots the real gdt-ts scores and predicted gdt-ts scores of a single-model predictor multicom-novel and a multi-model predictor multicom-refine on the models of a hard target t <dig> whose best model has quality score less than  <dig> .
it is worth noting that, since the quality of the models of the template-free modeling targets is rather low on average, the quality assessment on these models can be more arbitrary than on the template-based models of better quality.
therefore, more cautions must be put into the interpretation of the evaluation results.
<dig> lists the real gdt-ts score of each model.
the models in the circle form the largest cluster.
the model with the underline real gdt-ts score is the best model in this target.
based on the per-target correlation between predicted and observed model quality scores of the official model quality assessment
RESULTS
<cit> , the multicom-construct was ranked 5th on stage  <dig> models of casp <dig> among all casp <dig> model quality assessment methods.
the performance of multicom-construct was slightly better than the davis-qaconsensus  between the model and other models in the pool) on stage  <dig>  which was ranked at 10th.
the methods multicom-refine, multicom-novel, and multicom-cluster were ranked at 11th, 28th, and 29th, respectively.
however, it was not surprising that the single-model methods such as multicom-novel and multicom-cluster were ranked lower than most clustering-based methods because the latter tended to work better on most casp template-based targets with good-quality predicted models.
but, among all single-model methods, multicom-novel and multicom-cluster were ranked at 3th and 4th.
results of local quality
table
<dig> shows the performance of local quality assessment of our four local quality assessment servers, davis-qaconsensus, pcons, and modfoldclust <dig> on both stage  <dig> and stage  <dig>  among our four servers, the multi-model methods performed better than single-model methods on average for all the targets.
we used the pairwise wilcoxon signed ranked sum test to assess the significance of the difference between our four servers and the three external methods .
generally speaking, the difference between multi-model local quality methods  and single-model local quality methods  on both stages is significant.
the difference between multicom-refine and pcons is not significant on both stages according to a  <dig>  threshold.
however, the single-model local quality prediction methods  and the multi-model local quality prediction method  performed not very differently on fm targets as shown in tables
<dig> and
<dig>  this is not surprising because multi-model methods cannot select real good models as reference methods for evaluating the local quality of residues.
according to the casp official evaluation
<cit> , multicom-refine performs best among all of our four servers for the local quality assessment on both stage  <dig> and stage  <dig> models of casp <dig>  compared with davis-qaconsensus, pcons, and modfoldclust <dig>  the multi-model local quality prediction method multicon-refine performed best on stage  <dig>  achieved the similar performance with pcons on stage  <dig>  but performed worse than davis-qaconsensus and modfoldclust <dig> on stage  <dig>
CONCLUSIONS
in this work, we rigorously benchmarked our multi-model and single-model quality assessment methods blindly tested in the tenth critical assessment of techniques for protein structure prediction .
in general, the performance of our multi-model quality prediction methods  was comparable to the state-of-the-art multi-model quality assessment methods in the literature.
the multi-model quality prediction methods performed better than the single-model quality prediction methods , whereas the latter, despite in its early stage of development, tended to work better in assessing a small number of models of wide-range quality usually associated with a hard target.
our experiment demonstrated that the prediction accuracy of multi-model quality assessment methods is largely influenced by the proportion of good models in the pool or the average quality of the largest model cluster in the pool.
the multi-model quality assessment methods performed better than single-model methods on easy modeling targets whose model pool contains a large portion of good models.
however, they tend to fail on the models for hard targets when the majority of models are of low-quality and particularly when some low-quality models are similar to each other severely dominating the calculation of pairwise model similarity.
the problem can be somewhat remedied by using single-model quality prediction scores as weights in calculating the average similarity scores between models.
however, to completely address the problem, more accurate single-model quality prediction methods that can assess the quality of a single model need to be developed.
on one hand, more informative features such as sequence conservation information, evolutionary coupling information, torsion angle information, and statistical contact potentials may be used to improve the discriminative power of single-model methods; on the other hand, new powerful machine learning and data mining methods such as deep learning, random forests and outlier detection methods may be developed to use existing quality features more effectively.
despite it may take years for single-model methods to mature, we believe that improved single-model quality prediction methods will play a more and more important role in protein structure prediction.

 predicting protein tertiary structure from amino acid sequence is of great importance in bioinformatics and computational biology. during the last few decades, a lot of protein tertiary structure prediction methods have been developed. one category of methods adopts a template-based approach, which uses experimentally determined structures as templates to build structural models for a target protein without known structure. another category uses a template-free approach, which tries to fold a protein from scratch without using known template structures. the two kinds of methods were often combined to handle a full spectrum of protein structure prediction problems ranging from relatively easy homology modeling to hard de novo prediction. during protein structure prediction, one important task is to assess the quality of structural models produced by protein structure prediction methods. a model quality assessment method employed in a protein structure prediction pipeline is critical for ranking, refining, and selecting models. a model quality assessment method can generally predict a global quality score measuring the overall quality of a protein structure model and a series of local quality scores measuring the local quality of each residue in the model. a global quality score can be a global distance test score that is predicted to be the structural similarity between a model and the unknown native structure of a protein. a local quality score of a residue can be the euclidean distance between the position of the residue in a model and that in the unknown native structure after they are superimposed. in general, protein model quality assessment methods can be classified into two categories: multi-model methods and single-model methods. multi-model methods largely use a consensus or clustering approach to compare one model with other models in a pool of input models to assess its quality. generally, a model with a higher similarity with the rest of models in the pool receives a higher global quality score. the methods tend to work well when a large portion of models in the input pool are of good quality, which is often the case for easy to medium hard template-based modeling. multi-model methods tend to work particularly well if a large portion of good models were independently generated by a number of independent, diverse protein structure prediction methods as seen in the casp experiments, but they worked less well when being applied to the models generated by one single protein structure prediction method because they prefer the average model of the largest model cluster in the model pool. and multi-model methods tend to completely fail if a significant portion of low quality modes are similar to each other and thus dominate the pairwise model comparison as seen in some cases during the 10th casp experiment held in single-model methods strive to predict the quality of a single protein model without consulting any other models. the performance of single-model methods is still lagging behind the multi-model methods in most cases when most models in the pool are of good quality. however, because of their capability of assessing the quality of one individual model, they have potential to address one big challenge in protein structure modeling – selecting a model of good quality from a large pool consisting of mostly irrelevant models. furthermore, as the performance of multi-model quality assessment methods start to converge, single-model methods appear to have a large room of improvement as demonstrated in the casp experiment. in order to critically evaluate the performance of multi-model and single-model protein model quality assessment methods, the casp experiment was designed to assess them in two stages. on stage models of each target spanning a wide range of quality were used to assess the sensitivity of quality assessment methods with respect to the size of input model pool and the quality of input models. on stage about top models selected by a naïve consensus model quality assessment method were used to benchmark model quality assessment methods’ capability of distinguishing relatively small differences between more similar models. the new settings provided us a good opportunity to assess the strength and weakness of our multi-model and single-model protein model quality assessment methods in terms of accuracy, robustness, consistency and efficiency in order to identify the gaps for further improvement. in addition to evaluating our four servers on the casp benchmark, we compare our methods with three popular multi-model clustering-based methods. our clustering-based methods performed comparably to the three external tools in most cases. our single-model methods had a lower accuracy than the clustering-based methods, but performed considerably better than them on the models of hard template-free targets. besides the reasonable performance and a comprehensive comparative study, our methods have some methodological innovations such as using single-model quality scores to weight models for clustering methods, repacking side chains before model evaluation, and improved machine learning methods for single-model quality assessment for template-free targets. the rest of the paper is organized as follows. in the results and discussions section, we analyze and discuss the performance of the methods on the casp benchmark. in the conclusion section, we summarize this work and conclude it with the directions of future work. in the methods section, we introduce the methods in our protein model quality assessment servers tested in casp results and discussions results of global quality predictions we evaluated the global quality predictions using five measures. the results of the global quality evaluation on stage of casp are shown in table  the weighted pairwise model comparison method multicom-construct performed best among all our four servers according to all the five measures, suggesting using single-model quality prediction scores as weights can improve the multi-model pairwise comparison based quality prediction methods such as multicom-refine. the two multi-model global quality assessment methods had the better average performance than the two single-model global quality assessment methods on average on stage suggesting that the advantage of multi-model methods over single-model methods was not much affected by the relatively small size of input models. instead, the multi-model methods still work reasonably well on a small model pool that contains a significant portion of good quality models. it is worth noting that the average loss of the two single-model quality assessment methods is close to that of the two multi-model quality assessment methods. we also compared our methods with three popular multi-model clustering-based methods on stage according to the evaluation, multicom-construct performed slightly better than the naive consensus method davis-qaconsensus and modfoldclust while pcons performed best. table  shows the global quality evaluation results on stage similarly as in table  the weighted pairwise comparison multi-model method performed better than the simple pairwise multi-model method and both had better performance than the two single-model quality assessment methods. that the two single-model quality prediction methods yielded the similar performance indicated that some difference in their input features did not significant affect their accuracy. in comparison with stage all our methods performed worse on stage models. since the models in stage are more similar to each other than in stage in most cases, the results may suggest that both multi-model and single-model quality assessment methods face difficulty in accurately distinguishing models of similar quality. on stage models, multicom-construct delivered a performance similar with davis-qaconsensus and pcons, and had a higher average correlation than modfoldclust we used the wilcoxon signed ranked sum test to assess the significance of the difference in the performance of our four servers, davis-qaconsensus, pcons, and modfoldclust the p-values of the difference between these servers are reported in table  on stage models, according to significant threshold, the difference between clustering-based methods and single-model methods is significant, but the difference between our methods in the same category is not significant. one stage models, the difference between all pairs of our servers except the two single-model methods is significant. compared with the three external methods, the difference between our multi-model method multicom-refine and the three methods is not significant, while the difference between our single-model methods and the three methods is significant. the difference between multicom-construct and pcons is not significant, while the difference between multicom-construct and the other two external methods is significant. to elucidate the key factors that affect the accuracy of multi-model or single-model quality assessment methods, we plot the per-target correlation scores of each target on stage against the ratio of the average real quality of the largest model cluster in the pool and the average real quality of all the models in the pool in figure  to get the largest model cluster for each target, we first calculate the gdt-ts score between each pair of models, and then use as the distance measure to hierarchically cluster the models. finally, we use a distance threshold to cut the hierarchical tree to get the largest cluster so that the total number of models in the largest cluster is about one third of the total number of models in the pool. figure  shows that the quality prediction accuracy positively correlates with the average real quality of the largest model cluster divided by the average real quality of all models for two multi-model methods, whereas it has almost no correlation with single-model methods. the results suggest that the performance of clustering-based multi-model methods depends on the relative real quality of the large cluster of models and that of single-model methods does not. this is not surprising because multi-model methods rely on pairwise model comparison, but single-model methods try to assess the quality from one model. as casp models were generated by many different predictors from around of the world, the side chains of these models may be packed by different modeling tools. the difference in side chain packing may result in difference in input features that affect the quality prediction results of single-model methods even though they only try to predict the quality of backbone of a model. in order to remove the side-chain bias, we also tried to use the tool scwrl to rebuild the side chains of all models before applying a single-model quality prediction method - modelevaluator. figure  compares the average correlation and loss of the predictions with or without side-chain repacking. indeed, repacking side-chains before applying single-model quality assessment increased the average correlation and reduced the loss. we did a wilcoxon signed ranked sum test on the correlations and losses of the predictions before and after repacking side-chains. the p-value for average correlation before and after repacking side-chains on stage is, and on stage is. the p-value for loss on stage is, and on stage is. is used for the side-chain repacking. since mining a few good models out of a large pool of low-quality models is one of the major challenges in protein structure prediction, we compare the performance of single-model methods and multi-model methods on the models of several hard casp template-free targets. tables  and report the evaluation results of our four servers, davis-qaconsensus, pcons, and modfoldclust on all standalone template-free modeling targets on stages and i.e. the targets whose domains are all fm domains. the results show that the single-model methods clearly performed better than the multi-model methods on both stages. they also performed better than the davis-qaconsensus and modfoldclust on both stages, achieved the similar performance with pcons on stage and the better performance than pcons on stage for instance, the average pearson’s correlation score of multicom-novel on stage is, which is much higher than of multicom-refine. the multi-model methods even get low negative correlation for some targets. for example, the pearson’s correlation score of multicom-refine on target t at stage is -. we use the tool treeview to visualize the hierarchical clustering of the models of t in figure  the qualities of the models in the largest cluster are among the lowest, but they are similar to each other leading to high predicted quality scores when being assessed by multi-model methods. the example indicates that multi-model methods often completely fail when the models in the largest cluster are of worse quality, but similar to each other. multi-model methods often perform worse than single-model methods when all models in pool are of low quality and are different from each other. in this situation, the quality scores predicted by multi-model methods often do not correlate with the real quality scores, whereas those predicted by single-model methods still positively correlate with real quality scores to some degree. as an example, figure  plots the real gdt-ts scores and predicted gdt-ts scores of a single-model predictor multicom-novel and a multi-model predictor multicom-refine on the models of a hard target t whose best model has quality score less than. it is worth noting that, since the quality of the models of the template-free modeling targets is rather low on average, the quality assessment on these models can be more arbitrary than on the template-based models of better quality. therefore, more cautions must be put into the interpretation of the evaluation results. lists the real gdt-ts score of each model. the models in the circle form the largest cluster. the model with the underline real gdt-ts score is the best model in this target. based on the per-target correlation between predicted and observed model quality scores of the official model quality assessment , the multicom-construct was ranked 5th on stage models of casp among all casp model quality assessment methods. the performance of multicom-construct was slightly better than the davis-qaconsensus between the model and other models in the pool) on stage which was ranked at 10th. the methods multicom-refine, multicom-novel, and multicom-cluster were ranked at 11th, 28th, and 29th, respectively. however, it was not surprising that the single-model methods such as multicom-novel and multicom-cluster were ranked lower than most clustering-based methods because the latter tended to work better on most casp template-based targets with good-quality predicted models. but, among all single-model methods, multicom-novel and multicom-cluster were ranked at 3th and 4th. results of local quality table  shows the performance of local quality assessment of our four local quality assessment servers, davis-qaconsensus, pcons, and modfoldclust on both stage and stage among our four servers, the multi-model methods performed better than single-model methods on average for all the targets. we used the pairwise wilcoxon signed ranked sum test to assess the significance of the difference between our four servers and the three external methods. generally speaking, the difference between multi-model local quality methods and single-model local quality methods on both stages is significant. the difference between multicom-refine and pcons is not significant on both stages according to a threshold. however, the single-model local quality prediction methods and the multi-model local quality prediction method performed not very differently on fm targets as shown in tables and this is not surprising because multi-model methods cannot select real good models as reference methods for evaluating the local quality of residues. according to the casp official evaluation, multicom-refine performs best among all of our four servers for the local quality assessment on both stage and stage models of casp compared with davis-qaconsensus, pcons, and modfoldclust the multi-model local quality prediction method multicon-refine performed best on stage achieved the similar performance with pcons on stage but performed worse than davis-qaconsensus and modfoldclust on stage  in this work, we rigorously benchmarked our multi-model and single-model quality assessment methods blindly tested in the tenth critical assessment of techniques for protein structure prediction. in general, the performance of our multi-model quality prediction methods was comparable to the state-of-the-art multi-model quality assessment methods in the literature. the multi-model quality prediction methods performed better than the single-model quality prediction methods, whereas the latter, despite in its early stage of development, tended to work better in assessing a small number of models of wide-range quality usually associated with a hard target. our experiment demonstrated that the prediction accuracy of multi-model quality assessment methods is largely influenced by the proportion of good models in the pool or the average quality of the largest model cluster in the pool. the multi-model quality assessment methods performed better than single-model methods on easy modeling targets whose model pool contains a large portion of good models. however, they tend to fail on the models for hard targets when the majority of models are of low-quality and particularly when some low-quality models are similar to each other severely dominating the calculation of pairwise model similarity. the problem can be somewhat remedied by using single-model quality prediction scores as weights in calculating the average similarity scores between models. however, to completely address the problem, more accurate single-model quality prediction methods that can assess the quality of a single model need to be developed. on one hand, more informative features such as sequence conservation information, evolutionary coupling information, torsion angle information, and statistical contact potentials may be used to improve the discriminative power of single-model methods; on the other hand, new powerful machine learning and data mining methods such as deep learning, random forests and outlier detection methods may be developed to use existing quality features more effectively. despite it may take years for single-model methods to mature, we believe that improved single-model quality prediction methods will play a more and more important role in protein structure prediction. 
during the tenth critical assessment of techniques for protein structure prediction, we developed and tested four automated methods that predicted both local and global quality of protein structural models. since digging out a few good models from a large pool of low-quality models is a major challenge in protein structure prediction, single model quality assessment methods appear to be poised to make important contributions to protein structure modeling. protein model quality assessmentprotein model quality assurance programprotein structure predictionsupport vector machineclustering our experiments showed that the pairwise model assessment methods worked better when a large portion of models in the pool were of good quality, whereas single-model quality assessment methods performed better on some hard targets when only a small portion of models in the pool were of reasonable quality. protein model quality assessment is an essential component of generating and using protein structural models. multicom-cluster and multicom-novel were two new support vector machine-based methods of predicting both the local and global quality of a single protein model. multicom-refine was a clustering approach that used the average pairwise structural similarity between models to measure the global quality and the average euclidean distance between a model and several top ranked models to measure the local quality.
 protein model quality assessment is an essential component of generating and using protein structural models. during the tenth critical assessment of techniques for protein structure prediction, we developed and tested four automated methods that predicted both local and global quality of protein structural models.  multicom-refine was a clustering approach that used the average pairwise structural similarity between models to measure the global quality and the average euclidean distance between a model and several top ranked models to measure the local quality. multicom-cluster and multicom-novel were two new support vector machine-based methods of predicting both the local and global quality of a single protein model. multicom-construct was a new weighted pairwise model comparison method that used the weighted average similarity between models in a pool to measure the global model quality. our experiments showed that the pairwise model assessment methods worked better when a large portion of models in the pool were of good quality, whereas single-model quality assessment methods performed better on some hard targets when only a small portion of models in the pool were of reasonable quality.  since digging out a few good models from a large pool of low-quality models is a major challenge in protein structure prediction, single model quality assessment methods appear to be poised to make important contributions to protein structure modeling. the other interesting finding was that single-model quality assessment scores could be used to weight the models by the consensus pairwise model comparison method to improve its accuracy. protein model quality assessmentprotein model quality assurance programprotein structure predictionsupport vector machineclustering 