BACKGROUND
introduction
since  <dig>  when kendrew et al reported the first atomic-level resolution of a protein structure , the structural biology field dramatically expanded with the development of new tools and methods to gain access into atomic details of a protein or a nucleic acid  <cit> .
this opened a completely new world of knowledge and understanding to the scientific community, as the analysis of protein structures provides fundamental insight into most biochemical functions and consequently into the cause and treatment of diseases.
structural biology is now recognized as a fundamental step in our quest to understanding life at the molecular level.
finding the structures of all proteins is currently a bottleneck for genomics studies.
in this matter, the protein structure initiative  aims at the determination of the three-dimensional  structure of approximately  <dig>  structures in  <dig> years.
however, the protein sequence databank  is growing at a much faster rate, with more than  <dig> millions sequences available to date .
at the same time point, the protein data bank  includes  <dig>  structures, out of which only approximately  <dig> are "unique" at chain level .
it should be noted that these structures only represent a biased sample of the protein universe.
for example, the pdb includes only  <dig> unique membrane proteins which is very little since membrane proteins constitute around 20-30% of most proteomes  <cit> .
noteworthy, the human genome has ~ <dig>  protein-encoding genes for a proteome of ~ <dig> , <dig> proteins when combining the complexity induced by alternative slicing events  <cit> .
in addition, due to experimental limitations, the vast majority of the solved structures are below the  <dig> kda threshold excluding numerous larger proteins.
large proteins however represent a significant fraction of the proteins present in an organism; for instance, proteins found in yeast saccharomyces cerevisiae have five hundred amino acid residues on average and their lengths can reach two thousand eight hundred residues  <cit> .
the structure of these large proteins, as well as of even larger assembly can be solved by electron microscopy at a somewhat low-resolution.
while this field is expanding very fast and a growing number of structures solved at atomic-level resolution have been reported  <cit> , its impact with respect to the size of the protein sequence databank remains limited.
many more protein structures have been solved by either x-ray crystallography or nuclear magnetic resonance .
it remains however that most proteins are out-of-reach because of technical difficulties.
there is clearly a huge gap between the world of known structures and the universe of known protein sequences.
structural genomic projects are unable to keep up with newly discovered genes.
one way to work around this problem is to use computational methods to predict proteins structures.
in silico protein structure prediction techniques can be divided into two categories: the ab-initio folding methods and homology modeling.
in this paper we focus on the latter.
we note that both approaches have been shown to yield astounding results, as shown in the successive casp contests  <cit> .
however, they do require caution: while predicting the structure of a protein is an intellectual challenge that requires solving many practical issues, it is often considered as an art in essence.
the growth rate of structures deposited in the pdb is slowing down since  <dig>  along with the number of new superfamilies or folds discovered  <cit> .
one possible explanation is that many proteins still evade the structural biology pipelines at this time because of the technical difficulties described above.
in  <dig> chothia hypothesized that the number of protein folds in nature is probably finite and around  <dig>   <cit> .
the latest analysis of the pdb and of the structural classification of proteins  showed that we have not yet reached a plateau   <cit> .
the current rate at which proteins are added in the pdb is far too slow to match with the number of new protein sequences discovered every year.
the situation is however not so negative.
there is a definite hope that the current content of the pdb will allow us to predict reliably the correct scaffold of more than 70% of the whole proteome using in silico methods  <cit> .
this is the rationale for using homology modeling to complement experimental techniques.
homology modeling predicts the structure of a protein by inference from a homologous protein whose structure is known .
its success rests on  the existence of a homologue with known structure,  our ability to detect this homologue, and  the quality of the model building process once the homologue is detected.
steps  and  have greatly benefited from the different genomics projects: with the additional structures generated by the structural genomics projects, sampling of the protein structure space is becoming finer, improving the chance that a structural homologue exists for any given protein sequence.
with the parallel increase in the number of sequences available in genomic databases and the development of meta-servers to analyze and query sequence databases, there has been significant improvement in the detection of homology  <cit> .
in the recent casp experiments, targets with sequence identity of 6% to their templates were included in the homology modeling category  <cit> .
there is hope that in the near future homology modeling will reach its ultimate goal: the generation of model protein structures as accurate as those determined by high-resolution experimental studies.
biologists unanimously consider x-ray crystallography as the prime source of structural information on proteins and the "gold standard" in term of accuracy: they base their confidence on its long list of published successes.
the vast majority of structures deposited in the pdb were determined by x-ray crystallography and  <dig> nobel prizes in chemistry and medicine have been awarded to crystallographers  <cit>  .
homology modeling on its own however is not devoid of successes.
the very first published homology model in  <dig> was the small protein α-lactalbumin, which was modeled on the basis of the structure of hen egg white lysozyme as a template  <cit>  with the two proteins sharing 39% sequence identity.
when the structure of α-lactalbumin was later solved by x-ray crystallography  <cit> , the model turned out to be essentially correct  <cit> .
since then, homology modeling has continuously extended its field of applications, including designing mutants to test hypotheses about protein functions, identification of active sites, drug design, protein-protein docking, facilitating molecular replacement in x-ray structure determination, refining models based on nmr constraints .
despite these successes, homology modeling is not yet a well-established alternative or complement to experimental structural biology.
it remains the focus of many criticisms often coming from the structural biologists themselves as they often consider a protein model to be unreliable, not being based on experimental data.
the question arises as to what needs to be done to give homology modeling its credentials.
in this paper, we focus on homology modeling and more specifically, we review how the structural biology community perceives it and what can be done to impress on the experimentalists that it can be a valuable resource to them.
it is organized as follows.
the next section reviews the differences and similarities between homology modeling and high-resolution experimental structural biology.
in particular, we illustrate steps in the homology modeling procedure that are putative source of errors.
our goal is to provide a useful step-by step handbook for non-specialists in order to help building better model.
the following section introduces the h-factor, a new indicator for assessing the quality of homology models.
the h-factor is designed to check how well a family of homology models reflects the data that were used to generate those models, in the spirit of the r-factors in x-ray crystallography.
the results section that follows validates the h-factor on a series of test cases.
we conclude the paper with a discussion on what remains to be done to make homology modeling a prime technique for the biologists.
homology modeling versus experimental structural biology
this section reviews briefly the quality of protein structure models obtained either using high-resolution experimental techniques or homology modeling.
our hope is to identify common good practices as well as safeguards from which we can derive a validation tool for the latter.
we start with the concept of a structural model and its meaning in the two communities of experimental and computational structural biologists.
we then highlight the pros and cons of x-ray crystallography and nmr spectroscopy.
an overview of the different steps involved in homology modeling follows, with emphasis on sources of errors and how they can be checked.
ultimately our goal is not to rank these methods but rather we hope to show that they all provide valuable and often complementary information, as long as the proper safeguards are applied.
what is a model?
the meaning of the word "model" is ambiguous in the structural biology community.
a model for a protein structure can be obtained either by x-ray crystallography, by nmr spectroscopy, by electron microscopy, by computational methods or by combinations of all or some of these techniques.
with experimental techniques, the atomic coordinates are refined against experimental structural restraints and constraints.
eventually the final model is called "structure" when the refinement statistics converge toward acceptable canonical values.
note that often this final model is subjected to refinement using simulation techniques such as constrained molecular mechanics or molecular dynamics simulations.
even though these simulations are constrained with the experimental data, the subsequent "structure" cannot be considered to be fully independent of modeling.
on the other hand, an in silico model is generated without or with very limited experimentally constraints: it depends obviously on the hypotheses included in the modeling process, on the force-field used in the simulations as well as on the quality of the scientific computing tools that were used during the modeling steps.
while the quality of an "experimental" model can be assessed directly against the experimental data, the quality of an "in silico" model is more subjective and ultimately defined through the usefulness of the model: this is most probably the source of the mistrust towards modeling in general.
x-ray crystallography: source of errors and quality metrics
a number of factors contribute to the quality of an x-ray structure.
the first factor relates to the intrinsic crystal properties and its diffraction capabilities, which is mostly evaluated in term of resolution.
the quality metrics used in x-ray crystallography fall into three categories: 1) to measure the quality of the raw data, 2) to measure the agreement of the refined structure against the data and 3) to validate only the model for ideal stereochemistry, rotamers and bad clashes implemented in what check or molprobity for instance.
the first category lies upstream of the structure building process as the measures it includes evaluate the quality of the experimental diffraction data.
the rsym indicator for example measures the average spread of individual measurements in respect to their symmetry equivalent measurements.
a good dataset will have an rsym smaller than five percent.
in addition, the quality of a dataset is also assessed by its signal-to-noise ratio <i/σ> and its data-collection completion for a given space group.
unfortunately, only a few crystals diffract to atomic resolution  with ideal quality metrics.
most of the crystal-based structures have therefore been solved with good to average raw data quality.
although building a structure can be semi-automatic with automated tools available for chain tracing, side chain-building, ligand building and water detection, it is still refined by experimentalists using their subjective interpretations of the data.
it is common for example to find areas of poorly defined electron density map due to disordered regions.
the experimentalist interprets these data to the best of her knowledge but this is unfortunately a common source of errors.
the second set of quality metrics assesses the relative agreement of the structure in regard to the experimental data.
this set includes the r-factor and the "free" r-factor .
the r-free is analogous to the r-factor but uses a small subset of the data that have been flagged-out and not taken into account during any refinement process  <cit> .
its purpose is to monitor the progress of refinement and to check that the r factor is not being artificially reduced by the introduction of too many parameters.
as such, it provides an unbiased indicator of the errors in the structure and prevents any over-refinement and over-interpretation of the data.
both factors along with rsym and rmerge can be seen as indicators of the errors inherent in the refined model and in the experimental data.
the quality of protein crystal structures has been reviewed several times over the last fifteen years and it is striking to notice that despite a constant increase of the technology and validation tools, it has not improved overall.
the quality spectrum of x-ray structures remains broad.
the increase of automation through structural genomic pipelines did not help raising the bar in that matter as human intuition and reasoning are taken out of the process  <cit> .
interestingly, x-ray structures published in high-impact general science journals are usually the worse offenders in term of quality and errors.
this is explained by the experimental difficulties associated with solving novel high profile structures and the rush to publish in a competitive environment  <cit> .
x-ray crystallography is not immune to errors and mistakes, both honest and dishonest.
unfortunately, over the years we have seen gross mistakes in various structures leading to the retraction of several high-impact papers in leading journals because of a lack of quality control during the structure building pipeline  <cit> .
a recent review by g. j. kleywegt highlights the need and the proper use of validation methods in structural biology in general and in x-ray crystallography in particular.
the author also emphasizes the use of validation methods early on in the project pipeline in order to minimize the number of erroneous high-profile structures that can hinder the progress of science for years to come  <cit> .
in light of a growing number of structures falsification and to prevent both dishonest and honest mistakes in structures determination, the curators of the pdb have implemented over the years new sets of validation procedures for the deposition process  <cit> .
nmr spectroscopy: source of errors and quality metrics
although structure determination by nmr spectroscopy methods is very different from x-ray crystallography, it shares similarities with the latter in terms of sources of errors.
instead of using an x-ray beam diffracting around electrons in a crystal, nmr spectroscopy is performed in solution and uses the magnetic properties of the nuclei with odd spins .
as the molecule of interest is placed in a strong magnetic field, each of these nuclei is characterized by a unique resonance frequency, i.e.
the frequency at which it will absorb energy.
this frequency depends on the local magnetic field that combines the external field and the local environment: it is referred to as the chemical shifts.
nmr experiments are designed to monitor the behavior of these nuclei as the system is perturbed from equilibrium and each experiment usually isolates one property, such as through-bond connectivity's  or spatial proximity that allows for energy transfer .
in the specific case of proteins, the number of nuclei involved can be large, leading to crowding of the spectra: this is usually overcome by using multidimensional experiments .
the typical protocol for protein structure determination by nmr proceeds as follows.
firstly, the chemical shifts observed on multidimensional spectra are assigned to their specific atoms  in the protein .
second, through-the-bond and through-space coupling effects  observed on these spectra are quantified and concerted into angles and distance restraints.
most of these restraints correspond to ranges of possible values instead of a precise constraint.
thirdly, a molecular modeling technique is used to generate a set of models for the protein structure that satisfy these experimental restraints as well as standard stereochemistry.
for a more detailed presentation of the application of nmr to protein structure determination, we refer the reader to  <cit> .
analogously to x-ray methods, the quality of nmr measurements affects the quality of the structures.
the precision of a set of models for a protein structure determined by nmr is evaluated as the root-mean-square  difference between each model and a "mean" structure, defined geometrically as the mean of all the models .
the quality of each model is evaluated by the number of violations observed in the model compared to the experimental restraints.
a high-quality nmr structure refers to a set of high quality models with no violations that are tightly bundled around their mean, i.e.
with a small rms.
note that in addition to these nmr specific quality measures, garrett and clore introduced a r-factor and a free r-factor for the refinement of nmr structures based on residual dipolar coupling, a long range nmr measure obtained on proteins that have been partially oriented in dilute liquid crystals.
in similarity with x-ray crystallography, the quality spectrum of nmr structures is broad, with errors and mistakes reported that are inherent to a human based determination process  <cit> .
homology modeling: source of errors and quality metrics
the general strategy developed for homology-modeling proceeds through a canonical seven-steps procedure :  identify the template proteins that share structural similarity to the target;  align the target sequence with the templates sequences;  build a single framework of spatially aligned template structures and assimilate the target protein backbone with this framework;  build the missing backbone elements  not represented in the template framework;  build the target side chains;  refine the model in order to minimize unrealistic contacts and strains; and  evaluate the final refined model for physical tenability.
to date numerous homology-modeling programs such as modeller  <cit> , segmod/encad  <cit> , swiss-model  <cit> , 3d-jigsaw  <cit> , builder  <cit>  and nest  <cit>  have been developed and many of them have been embedded into homology-modeling servers to ease the burden of generating models.
online portals such as the protein structure initiative  model portal or the swiss-model repository bring to the community a large database of models  <cit> .
the psi model portal currently provides  <dig>  millions comparative protein models for  <dig>  million distinct uniprot entries.
every model comes with relevant validation data.
however those models are automatically generated without any human interaction that might render them inaccurate without any extra validation steps.
in the following, we overview each step of the homology modeling process and highlight potential sources of errors.
*steps  <dig> and 2: template selection and sequence alignment
the selection of template is undoubtedly a critical step in modeling.
it was long assumed that two proteins whose sequences share at least 40% identity have similar structures.
if such a template exists, it is easily detected by any sequence alignment techniques.
homology modeling under such conditions is then expected to generate models whose accuracy is close to that of an experimental structure.
we know however that this is not always true.
roessler et al.
recently reported the discovery of two native cro proteins sharing 40% sequence identity but with different folds  <cit> .
moreover, alexander et al.
were able to design two proteins with 88% sequence identity but having different structure and function  <cit> .
reversely, it is not uncommon for proteins, especially enzymes carrying the same function across the tree of life to share a somewhat low sequence identity and at the same time being structurally similar, with a cα r.ms.d.
~ <dig> Å  <cit> .
all these observations indicate that the selection of template is far from being a trivial task and extreme caution should be applied.
the situation is even more difficult if there is no significant sequence similarity between the sequence of the target protein and any of the known protein structures.
this is one of the current challenges of the post-genomic era that is tackled by fold-recognition methods, namely to identify a suitable template for homology modeling  <cit> .
unlike sequence-only comparison, fold-recognition techniques take advantage of the information made available by  <dig> d structures.
despite a steady development over the years, as illustrated through the successive casp experiments, fold recognition techniques still have a number of limitations.
they are however the key to extend the domain of application of homology modeling methods.
the accuracy of the sequence alignment is another critical step in the homology modeling process  <cit> .
it is important to keep in mind that a shift of one residue in the alignment introduces a distortion of  <dig>  Å in the model for the backbone of the target protein .
therefore, it is of paramount importance to precisely verify and locally refine the sequence alignment used to build a model.
for that matter, the areas of sequence comparison and alignment optimization have grown in parallel to the comparative modeling field  <cit> .
there are many ways to align protein sequences and numerous methods to score the accuracy of the resulting alignments.
homology modeling usually relies on one of following three techniques:  standard pairwise sequence alignment using dynamic programming,  multiple sequence alignment when the target and template sequences belong to a large family for which many sequences and structures are known, and  direct alignment of the sequence on the structure of the template using a threading technique.
none of these techniques is the panacea, especially when the sequence identity between the target and template sequences is low.
errors come for example from the fact that the optimal alignment between the sequences of two proteins may not match the alignment between their structures: this is caused by the sequence alignment technique that optimizes a score based on a substitution matrix and gap penalties while structural alignment techniques optimize geometric matching.
there is no reason that these two different metrics are equivalent.
errors are often frequent in the loop regions between secondary structures, as well as in regions where the sequence similarity is low.
recent techniques that attempt to reduce the error rate in the sequence alignment rely on the inclusion of as many sources of information as possible, such as amino-acid variation profiles, secondary structure knowledge, structural alignment data of known remote homologues, knowledge of "anchor regions" .
many programs also attempt to optimize the raw sequence alignment derived from one of the techniques described above.
most of these programs have been embedded into web-servers for ease of uses, such as muscle  <cit>  and dialign-tx  <cit> .
* step 4: loop building
the loop-building step is another key component in homology modeling.
loops participate in many biological events and functional aspects such as enzyme active sites, ligand-receptor interactions, and antigen-antibody recognition among others.
however, due to the flexible nature of loops, it is often difficult to predict their conformation.
there are two main approaches to tackle the problem of loop modeling: methods that use databases of loop conformations or ab initio methods.
in the database approach, a library of protein fragments whose size corresponds to the size of the loop to be modelled is scanned for fragments whose end-to-end distance matches the corresponding distance in the framework.
the library is derived from the known protein structures in the pdb.
this method has proved to be accurate when the loop is relatively short.
fidelis et al.
have shown that loops of a maximum of seven residues can be modelled with confidence based on known structures  <cit> .
when the database method is combined with a restrained energy minimization, it extends the confidence of loop building up to nine residues  <cit> .
beyond the nine residues threshold, ab initio methods have to step in mostly because for these longer loops, the fragment library provides a poor sampling of the conformational space accessible.
the ab initio loop prediction approach relies on a conformational search guided by a scoring function.
the accuracy of ab initio loops modeling remains currently low, especially when dealing with very long loops  <cit> .
* step 5: the side-chains positioning problem
the prediction of side-chain conformations for a given backbone architecture remains a challenge.
it is however the key to generating models at atomic resolution.
for instance, critical side-chains forming an active site must be accurately positioned in order to support any putative catalytic mechanism.
nearly all the side-chain positioning methods are based on rotamer libraries with discrete side-chains conformations.
rotamer libraries contain a list of all the preferred conformations of the side-chains of all twenty amino acids, along with their corresponding dihedral angles  <cit> .
some of these rotamer libraries are further refined to account for the local geometry of the backbone  <cit> .
sidechain prediction techniques select the best rotamer for each residue of the protein under study from one of these libraries, based on a score that includes both geometric and energetic constraints.
this leads to a large combinatorial problem: see figure  <dig> for an example of the size of the conformational space accessible to a sidechain in a protein, even when we discretize this space by using rotamers.
the combinatorial problem is solved by heuristic techniques such as mean field theory, derivatives of the dead-end elimination theorem or monte carlo techniques  <cit> .
the success rates of the most successful techniques range from 78% to 89% for the χ <dig> and χ <dig> angles of residues in the core of the protein .
it is important to note that these results usually relate to mock experiments in which the exact conformation of the backbone is known.
it has been shown that the quality of sidechain prediction decreases as the deviation between the backbone used for modeling and the actual conformation of the backbone increases  <cit> .
these results emphasize the need of a good framework  for homology modeling.
it is worth mentioning however that a 1Å accuracy over the main-chain atom position can be related to x-ray structures with  <dig>  Å resolution and with a r-factor around 25%, which is fairly common  <cit> .
in addition, multiple side chain orientations are routinely observed in crystal structures and crystal-packing forces can alter their positions as well.
the side chain prediction methods are improving steadily, as reported by the casp experiments  <cit> .
* step 6: refinement of the final model
in a review written in  <dig> on the casp <dig> experiment, koehl and levitt noted that most models submitted in the homology modeling category were not refined, as previous casp meetings had shown that refinement did not improve the models  <cit> .
sadly for the computational biologists the situation has not improved and it remains difficult to generate a model closer to the native structure than the template used to build it  <cit> .
energy refinement, originally introduced by levitt and lifson forms the basis of current methodologies for protein structure refinement against experimental data  <cit> .
without experimental restraints, refinement by energy minimization generally moves the protein structure away from its x-ray structure.
some recent studies have shown that this negative trend can be reversed through the inclusion of evolutionary derived distance constraints  <cit>  through the combination of sophisticated sampling techniques based on replica exchange molecular dynamics and statistical potentials  <cit> , through the addition of a carefully designed, differentiable smooth statistical potential  <cit> , or by careful consideration of the solvent effects  <cit> .
while these studies are definitely source for hope, much work remains to be done as far as refinement is concerned.
a general framework for model assessment: r-factors and equivalent
the wide availability of homology modeling software packages, as well as the development of web interfaces that automate the use of these packages has resulted in better access to and a broader usage of homology modeling.
while this is definitely commendable and homology modeling should be even more advertised, there are risks that this will lead to errors because of the difficulties in evaluating the correctness of the models these techniques generate.
this is primarily due to the lack of cross validation indicators such as the r-factor and r-free in x-ray crystallography  <cit> .
in addition to the stereochemistry assessment of the structure and the good correlation between the r-factor and the r-free values, the quality of an x-ray structure can be evaluated based on the thermal motion value of atoms described by the debye-waller factor or b-factor.
the b-factor allows for the identification of zones of large mobility or error like disordered loops.
when multiple monomers populate an asymmetric unit of a crystal, the crystallographer will choose to focus on the analysis of the monomer with the lowest average b-factor since the likelihood of errors is lower.
unfortunately, such criteria do not apply to models thus rendering the identification of zones of uncertainties a non-trivial task.
with respect to homology modeling, the main step is to thoroughly validate the model and always provide all the relevant details about the protocol used.
this will give the user all the necessary data to judge the quality of the model.
the quality assessment of models has been the focus of numerous studies and various algorithms have been reported over the years.
in this matter, tremendous efforts are being made to produce the best triage procedures or scoring functions among models, as seen in the latest casp meetings  <cit> .
these scoring functions are based on statistical potentials  <cit> , local side-chain and backbone interactions  <cit> , residue environments  <cit> , packing estimates  <cit> , solvation energy  <cit> , hydrogen bonding, and geometric properties  <cit> .
in addition, it is essential that the quality of stereochemistry be kept high.
the stereochemistry can be assessed by commonly used programs such as procheck or whatif  <cit> .
ultimately, the validation of models comes from experiments such as site-directed mutagenesis, circular dichroism, cross-linking, mass spectrometry, fluorescence-based thermal shift, light scattering, molecular fret or electron microscopy.
such experimental data can be translated into constraints/restraints and introduced in the modeling protocols thus improving the accuracy of models.
one can also identify fast and cheap experimental procedures that can help testing homology models.
the easiest way is to crosscheck models with experimental structures.
for instance with enzymes, it is possible to verify the location of important catalytic residues in the active site by comparison with homologous family members.
most importantly however, a model needs to be checked manually in the same way a nmr or an x-ray structure is processed.
despite all these methods, the homology modeling community still lacks a simple indicator which gives an unambiguous feedback on how the final model, or family of models, reflects the data that were used in the modeling process, similar to the couple r-factor/r-free for x-ray crystallography.
the next section introduces such an indicator, namely the h-factor.
methods
computing the h-factor
in this study, we introduce a novel indicator as an attempt to quantify how well a homology model correlates with the input data and its biological relevance.
this indicator, the h-factor is designed to mimic the r-factor in x-ray crystallography.
it is rooted in the basics of homology modeling and it uses all data that were included in the model building process to assess its correctness, in addition to checking for good stereochemistry.
more specifically, the h-factor combines information on  the template structure ;  the sequence alignment between the template and the target sequences;  the structural heterogeneity of the models built; and  the structural neighborhood within protein families.
note that at this stage, the computation of the h-factor is based on a single framework.
we plan to derive an extended h-factor that will account for multiple templates.
each of these four categories is assigned a scoring function returning a value between  <dig>  and  <dig> .
the h-factor is simply the sum of these scoring functions divided by  <dig>  for ease of use, the h-factor is converted into a percentage with a value between 0%  and 100% .
in par with the r-factor in x-ray crystallography, a low h-factor value indicates a trustworthy set of models, while a large h-factor would raise a flag on the correctness of the models.
the h-factor focuses on the cα-backbone as a correct tracing is a prerequisite for a valid model.
the workflow of the h-factor calculation is detailed on figure 4a and 4b.
the scoring function  analyses the discrepancies between the secondary structure prediction for the target obtained with the program psipred  <cit>  and the actual secondary structures of the template framework computed with the program stride  <cit> .
the corresponding score takes into account the confidence factors reported by psipred:
score1=a∑i=1nfn+bf={0for p=sc+110for p≠sfor gaps in1the sequencealignment
where the sum is computed over all positions in the sequence alignment between the target and template, n is the length of the sequence alignment, p is the secondary structure prediction of the target at position i , c is the confidence factor reported by psipred  for the secondary structure prediction at position i and s is the secondary structure type observed at position i in the template structure reported by stride.
the offset coefficients a and b are set to  <dig>  and  <dig> , respectively, to ensure that score  has values between  <dig> and  <dig>
the function  scores the identity between the sequence of the target and the sequence of the template framework .
score2=10n)g={0for non identity1for identity
where n is the length of the sequence alignment.
the search for a template structure for a given target protein sometimes finds multiple candidates.
in addition, considering multiple options for this alignment sometimes alleviates ambiguities in the alignment between the sequence of one of these candidates and the sequence of the target.
both situations lead to multiple structural models being generated for the target protein: score  is designed to measure the heterogeneity of this set of models.
it uses as input all the models mi, as well as the corresponding average model, ma, whose atomic coordinates are the averages of the corresponding coordinates in the models.
the average model ma is computed from the structural alignment of all the models, which can be computed for example with the cluster() function in modeller  <cit>  or with the program maxcluster  <cit> .
the function  then reports the average crms between each model and the average model, where the crms is computed over the cα atoms only.
the average crms is then transformed linearly such that the final score is between  <dig> and 10:
score3=an)+b
n is the number of models.
the offset coefficients a and b are chosen such that average rms values of  <dig>  and  <dig> Å correspond to scores of  <dig> and  <dig>  respectively; the corresponding values are a =  <dig>  and b =  <dig> .
the search for a template structure sometimes finds partial matches.
in many cases, this is related to the fact that the target protein contains multiple functional domains that may not always be associated in other proteins.
while the modeling is performed based on the longest possible template, it is expected that each of these domains have been correctly modelled.
score  is designed to quantify this assertion.
it first identifies the various domains in the sequence, using hmmer  <cit>  and the pfam profiles database as a reference  <cit> , the average model ma is then broken down into fragments corresponding to the domains that have been identified.
each fragment is compared to the structure of the same domain found in proteins whose structure has been deposited in the pdb.
a maximum of  <dig> fragments is considered .
to minimize the numbers of false positive we set an e-value cut-off of  <dig> e- <dig> for hmmsearch.
the score  is then the average crms distance between the fragments and their counterparts in the pdb:
score4=amn)+b
m is the number of functional domains identified in the target sequence, mad is the structural fragment extracted from the average structure ma corresponding to the domain d, n is the number of domains homologous to domain d found in pdb structures, and dd,i is the i-th possible structure of the domain homologous to d. the offset coefficients a and b have been chosen such that average rms values of  <dig>  and  <dig> Å correspond to scores of  <dig> and  <dig>  respectively; the corresponding values are a =  <dig>  and b =  <dig> .
this usually enforces that score  is between  <dig> and  <dig>  note that if this procedure does not find an equivalent domain for a fragment, the fragment is ignored; if no domains are found for all fragments, score  is ignored.
the h-factor computation is accessible online at http://koehllab.genomecenter.ucdavis.edu/toolkit/h-factor with a simplified operating manual .
the source code is available upon request.
testing of the h-factor on casp targets
target proteins that have been submitted to casp in the homology modeling category are perfect tests for the h-factor measure presented here.
the web site for the protein structure prediction centre that hosts all information about the casp experiments http://predictioncenter.gc.ucdavis.edu also includes a database of all the models that have been submitted in the successive casp with the corresponding experimental structures.
this database of casp models contains models that can be ranked from being very accurate to being completely wrong.
it is therefore ideally suited to benchmark modeling techniques and to develop and test validation methods.
we have chosen three targets from the recent casp <dig> experiments: t <dig>  t <dig> and t <dig> that have been identified as homology modelling targets by the casp organizers .
the first target  was considered "easy", while the second and third target  were more difficult cases for homology modeling.
fold recognition techniques initially identified one template for t <dig> and t <dig>  and six potential template structures for t <dig>  the top templates were then selected according to the casp <dig> analyses of every possible template for each target.
we aligned the sequence of the target with its template using clustalw  <cit> , with the gonnet <dig> matrix to define the substitution score and default settings for gap penalties .
for the first two targets, this corresponds to a simple pairwise alignment, while for t <dig>  clustalw provides a multiple sequence alignment over the target sequence and the six possible template sequences.
we used modeller 9v <dig>  <cit> , with the "automodel" settings to generate  <dig> models for each of the three targets.
as the h-factor focuses only on the cα-backbone, we did not attempt to improve the prediction of the sidechains.
in addition, we did not perform any energy minimization of the final model and it still remains unclear if energy minimization improves models generated by homology modeling  <cit> .
template )
%sequence identity between template and target
coverage
type of sequence alignment
loop segments of  <dig> or more residues
RESULTS
the h-factor: detailed analysis on three casp targets
the casp <dig> target t <dig>  a dimethyl adenosine transferase from plasmodium falciparum is considered an "easy" modeling case: it has a very close homologue  whose structure has been solved at high resolution  and in addition the template sequence covers the whole sequence of t <dig>  we generated  <dig> models for t <dig> using standard homology modeling techniques .
in this experiment, our goal was not to generate the ultimate, high-resolution model.
instead, we were interested to see if the h-factor we have introduced was able to measure the quality of the models we generated.
the average crms  between the t <dig> models and the actual experimental x-ray structure for target t <dig>  is  <dig>  Å: this clearly indicates that the models are of good quality.
the corresponding h-factor for these  <dig> models is 19%, i.e.
a very good score .
the good quality of the t <dig> models is highlighted by each scoring function included in the h-factor .
the secondary structure prediction for t <dig> matches well with the actual secondary structure of its framework , yielding a value of  <dig> for score .
the sequence alignment between t <dig> and 1zq <dig> is deemed good, with a value of  <dig>  for score .
the  <dig> models generated showed little structural dispersion with a corresponding value for score  of  <dig> .
t <dig> contains one pfam domain, pf <dig> that corresponds to the ribosomal rna adenine dimethylase.
in addition to being found in the structure 1zq <dig>  this domain is also found in the proteins whose structures are available in the pdb files 1i4w, 1qam, 1qyr, 1yub and 2erc.
score  compared the structure of this domain in the models generated for t <dig> and the structures of the same domain found in all  <dig> proteins listed above.
it detected fluctuation between these structures, leading to a score of  <dig> .
note that the pfam domain pf <dig> occurs in species covering all three kingdoms.
in addition, the structures have been derived from data with wide range of resolution .
it is therefore not surprising that score  is relatively higher than the other scores; it does remain however within a range that indicates a good match.
note that the overall h-factor value is 19%.
in comparison, a r-factor of 20% is typically observed for fully refined x-ray structures around  <dig> Å of resolution, i.e.
for a good x-ray structure.
the four scoring functions included in the h-factor measure the quality of the secondary structure prediction ), the diversity of the sequence alignment ), the structural diversity of the models generated ), and the similarity of the predicted structures for the functional domains in the target, compared to the structures of the same domains found in the pdb )
average crms  between the  <dig> different models generated and the actual experimental structure for the target.
sequence identity between the target sequence and the template sequence
dope scores from modeller  <cit>
qmean normalized score  <cit>
t0295* is the toy experiment in which the sequence alignment between the target t <dig> and its framework has been deliberately modified .
when we deliberately introduce a shift at position  <dig> in the alignment between the sequence of t <dig> and the sequence of its template 1zq <dig> , the corresponding models generated by modeller show structural diversity in the loop region near the shift .
score  captures this structural diversity within a set of models.
it leads to the h-factor being raised from 19% to 21% .
however, score  could not detect a single position shift in the alignment.
the h-factor is therefore capable of detecting backbone deviation due to modeling errors, the same way the r-factor does.
the casp <dig> target t <dig> is a more difficult modeling case.
it is a human ketohexokinase and the rigid-body domain closure of sugar kinases is known to be large, adding complexity into the modeling process  <cit> .
although several sugar kinase structures have been solved, the search for templates for t <dig> identified only six distinct remote templates.
moreover, all six templates are needed to obtain complete sequence coverage of t <dig> within one single framework with modeller.
in addition, the template sequences have low similarity with the target sequence.
this is detected by the scoring function , which returns a value of  <dig>   .
note that the score  is not a direct measurement of the quality of the sequence alignment.
it is designed to quantify the difference between the two sets of sequences: if this difference is small, the model is expected to be good, while if the difference is large, the sequence alignment most probably belongs to the twilight zone and the models should then be considered with caution.
the overall h-factor for the models generated for t <dig> is 41%.
this mid-range value indicates that caution should be used when interpreting or using these models.
indeed, the average crms between these models and the actual structure of t <dig>  is  <dig> Å, i.e.
reflecting a medium-resolution agreement.
the casp <dig> target t <dig> is the most difficult test case we have considered.
in fact, it would not be considered a homology-modeling target by many, despite the fact that a  structural homologue is available in the pdb.
we did decide to include it in our study to test whether the h-factor was still providing useful information when applied on a difficult test case.
t <dig> corresponds to cags, a protein from helicobacter pilori whose function is unknown.
a database search over all sequences of proteins whose structure is known identifies a unique template, 1v <dig>  with a low sequence identity .
1v <dig> is a cytochrome c oxidase and it is not clear that t <dig> and 1v <dig> are homologues.
we did build  <dig> models for t <dig>  using 1v <dig> as a template, and the out-of-the-box alignment between the sequences of t <dig> and 1v <dig> generated by clustalw.
as mentioned above, we did not try to optimize the alignment or the modeling itself as our interest is to see if the h-factor is able to assess the quality of the models we generated.
in this specific case, all four scores reported high values , ,  and , respectively).
the overall h-factor is 75%, a valued that should raise concerns about the quality of these models.
as for target t <dig>  this is confirmed by the experiment: the average crms between these models and the actual structure for t <dig>  is  <dig>  Å, indicating that the models are poor approximations of the native structure.
h-factor: detecting bad models
from the three test cases t <dig>  t <dig> and t <dig>  we conclude that the h-factor correlates well with the quality of the models tested.
to further confirm that high h-factor values correspond to models that should be considered with caution, we downloaded sets of models for seventeen target proteins from casp <dig>  these targets were considered as difficult, with only remote templates available.
the corresponding sets of models, available from the casp web site have poor to bad quality.
when compared to the actual experimental structures, these models have a very high crms indicated major errors in the backbone tracing.
in all sixteen cases, the h-factors are found to be high, as shown in table  <dig>
crms  between the average model and the actual experimental structure for the target.
dope scores from modeller  <cit>
qmean normalized score  <cit>
h-factor: characterizing good models
the h-factor computes the quality of models for protein structure based on sequence information  and score ), as well as based on structure information  and score ).
while the former is specific to homology modeling, the latter can be used to assess the quality of any sets of models.
we tested score  and score  on a random set of nmr structures for which  <dig> or more models are available.
in this validation case, the h-factor is the sum of functions   divided by  <dig> instead of  <dig>  and converted to percentage.
results are listed in table  <dig>  in every case, the h-factor is low and below 35% in average, as anticipated.
this indicates that the algorithm recognized the experimental structures as valid.
see text for the description of the scoring functions  and ; these two functions measure the structural dispersion of the models considered, as well as their similarities to homologue structures in the pdb; note that scores  and  usually present in the h-factor are meaningless for this test set.
average rms between each model and the "mean" structure computed from all the models.
although the h-factor and the r-factor are mathematically unrelated, they have the same purpose: to assess the quality of structures, either experimental or computed from modeling experiments, where quality refers to reflecting correctly the input data used to generate these structures.
the h-factor mimics the r-factor as it provides a quality-index to follow in the process of building a model, the same way crystallographers monitor the r-factor/r-free indexes during structures refinements.
we compared the h-factor results with the experimental r-factor and r-free on a randomly chosen subset of the pdb containing  <dig> structures with  <dig> or more identical chains solved by x-ray crystallography.
results are shown in figure  <dig>  the h-factor and r-factor are not linear correlated but it remains that "good" r-factors  correspond to "good" h-factor values .
the h-factor checks the diversity of the set of models generated for a structure, as well as their similarities with the structures of domains that share the same function, as defined by pfam.
high h-factor values may be caused by structures with disordered loops or remote structural neighbors in the pdb.
it remains that, in agreement with what we observed for the nmr structures, the h-factor recognizes experimentally determined structures as being valid.
relationship between the h-factor and crms
the crms by itself is a reasonable quality indicator as long as its value remains low .
it should be noted that it is an average value computed over the whole structure.
as such, it is very sensitive to large structural fluctuations in disordered loops for example that can lead to large crms values even if the conserved domains are structurally very similar.
it is well known for example that caution should be applied when using crms to assess the quality of a structural alignment.
crms is implemented in score  of the h-factor to evaluate the heterogeneity amongst a set of models as it is directly related to both the choice of the template and the quality of the sequence alignment.
however, the score  loses accuracy for crms values larger than  <dig> Å, which is not uncommon when a remote template is used.
crms is also implemented in score  to quantity the modelling quality of specific individual domains by comparing them with corresponding domains in the pdb.
this is a domain-based crms that does not take into account potential long loops between domains, making it more reliable.
taken together, the scores  and  alleviate most of the limitations of crms while retaining its major properties.
the h-factor is therefore expected to be more reliable than a sole crms to judge the accuracy of a wide range of models, as seen in table  <dig>
comparing the h-factor with prosa, dope and qmean
the structural biology community as well as the protein structure modeling community have always been reasonably good at setting safeguards to estimate the validity of both experimental and computational models for protein structures.
the first validation tests imposed on experimental structures focused on the stereochemistry of the molecules.
it was found, however that this was by far not sufficient, as "good stereochemistry" can be misleading.
a "clean" ramachandran plot for example does not necessary prove that a given model is valid.
also, we observed that all the models listed in table  <dig> have good stereochemistry, including the t <dig> model that has a cα backbone deviation of almost  <dig> Å from the actual experimental structure.
experimentalists have access to another essential set of validation tools that check the consistency of the structures with the experimental data that were used to derive them.
such tools include the r factors in x-ray crystallography and nmr.
models derived from modeling experiments are more difficult to verify, unless they include some external information.
for example, the swiss-model server provides a confidence score along with the models it built and this confidence score is based on the amount of structural information that supports each part of the model  <cit> .
in general, however, validation of computer-generated models is based on comparison with known protein structures.
the idea is that basic properties of protein structures can be inferred from the pdb, and translated into database-derived scoring functions.
this is the main rational for the statistical potentials, also referred to as mean-field potentials that have become very popular validation tools  <cit> .
for example, prosa-web implement two statistical potentials: a pairwise potential based on cß atoms and a surface term that models the protein-solvent interactions  <cit> .
here we compare the output of prosa and h-factor analyses on the three casp <dig> test cases given in table  <dig>  the prosa results are given if figure  <dig>  both the h-factor and prosa estimate that the models generated for t <dig> are valid.
when the modeling process is deliberately altered , again both prosa and the h-factor analyses reveal a loss of quality.
prosa however considers the models generated for the targets t <dig> and t <dig> as statistically correct , while we know that these models are not correct .
in addition, it is striking to see that the pdb structure 1us <dig>  an ultra high-resolution crystal structure at  <dig> Å resolution, has a prosa z-score  statistically as correct as t <dig> .
the statistical potential discrete optimized protein energy  is another measure of model quality that has been introduced in modeller- <dig>  <cit> .
dope is a statistical potential with an improved reference state that accounts for the compact shape of native protein structures.
the dope score is designed such that large, negative scores are usually indicators of good models.
in their original study, shi and sali  <cit>  found that the accuracy of dope to asses a homology model improves as the accuracy of the models improve.
we observe a similar behaviour for targets t <dig> and t0295* .
these two targets correspond to the same protein and it is therefore possible to compare the dope scores of their models.
the model generates for t0295*, based on an incorrect alignment, has a much lower dope score  that the model generated with the correct alignment .
note that we cannot compare dope scores for proteins of different size, as these scores are not normalized.
dope scores are therefore relative, and designed to pick a "good" model among poorer model.
dope scores do not assess directly the quality of the model that is picked, i.e.
if it is likely to be similar to the actual structure.
the h-factor is a better indicator in that respect.
qmean, which stands for qualitative model energy analysis, is a composite scoring function for homology models that describes the major geometrical aspects of protein structures  as well as the agreement between the predicted and calculated secondary structure and solvent accessibility, respectively  <cit> .
as such, it includes a term similar to the score  of the h-factor, as well as terms that assess different properties such as residue accessibility.
the score qmeannorm is a normalised version of the qmean score in which all terms are divided by the number of interactions/residue in order to avoid a size-bias of the score  <cit> .
qmeannorm scores vary between  <dig> and  <dig>  with larger scores expected to correspond to better models.
unlike the dope score, both the h-factor and qmeannorm scores allow for the comparison of proteins of different sizes.
the qmeanscore is as effective as prosa or dope for detecting errors in a model that result from errors in the sequence alignment between the template and target protein: t0295* has a qmeannorm score of  <dig>  while the score fort <dig> is  <dig>  .
interestingly, t0295*  has a less favorable qmeannorm score than the erroneous model generated for the casp target t <dig>  .
we have observed however that the qmeannorm score is prone to fail: some of the erroneous models generated for the casp target t <dig> have qmeanscores of  <dig>  to  <dig> , i.e.
they are evaluated to be almost as correct as the positive control t <dig> .
unlike prosa and qmean, the h-factor did detect that these models were to be considered with caution.
because it analyzes a set of models, we believe that the h-factor score is more robust as an absolute measure of the quality of a model.
it lacks however the ability to discriminate among a set of models generated for the same target; prosa and dope are better potentials for this specific task.
these results emphasize the essential differences in the nature of the prosa, dope, qmeannorm and h-factor scores.
prosa, dope and qmean check the quality of a model, independently of the context in which it was generated.
the h-factor on the other hand checks the quality of a set of models with respect to a context that includes for example the sequence alignment assessed by the score .
the modeler however should use these differences to extend his/her assessment of the model his/she generates.
we believe that prosa, dope, qmean and h-factor analyses are needed to provide a better overview of the quality of models derived by homology modeling.
current limitations and originalities of the h-factor
the h-factor is not the panacea, and does not provide a universal solution to the problem of asserting the quality of a model generated by homology modeling.
firstly, the h-factor has some technical limitations.
our current implementation does not take into account multiple templates, but rather only one single framework.
the structural components included in the h-factor  and ) are based on the backbone of the models, and do not take into account sidechains and possible errors in their modeling.
second, the scoring function  of the h-factor measures the heterogeneity of a set of models generated with the same input.
it means, that the h-factor cannot be computed on a singular model.
in homology modeling the heterogeneity of models can be seen as a quality indicator and building only one single model is not recommended.
similarly to nmr structures where only one of the models can be chosen for analysis, the best model in homology modeling regime is chosen based on the modeller energy function for instance.
third, the h-factor does not include any external information.
for example, if some biological data are available, such as the knowledge of the residues involved in the active site, or standard biophysical data such as melting temperature, or secondary structure content derived for circular dichroism, these data are currently not included in the h-factor analysis.
the r-factor is a measure of the agreement between the crystallographic model and the experimental x-ray diffraction data.
despite the lack of 'experimental' data to compare with, the modelling community has been searching for a similar indicator for homology modeling.
both qmean and the h-factor are designed to be 'absolute' indicators that assess the quality of homology models in a way that mimics the r-factor in x-ray crystallography.
both qmean and h-factor provide an easy-to-use estimate of the quality of models based on scoring functions assessing various aspects of the modelling process as well as the model itself  <cit> .
in vivo macromolecular structures oscillate between numerous conformers, some more than others.
while x-ray structures correspond to snapshots of a limited numbers of conformers, nmr structures tend to describe more accurately flexibility.
indeed, nmr "structures" are usually provided as a family of conformers that are meant to sample the conformational space accessible to the molecule of interest.
in homology modeling on the other hand, the heterogeneity of models is a quality indicator.
a good set of models will have a crms very close to their framework.
moreover, if errors are being made in the template choice or in the sequence alignment, then the models will be heterogeneous.
the scoring function  is designed to quantify this assertion.
it also means that the h-factor cannot be computed on a single model.
one of the originality of the h-factor is the scoring function .
it has been designed to evaluate the biological relevance of the models by comparing the model conformations of all the functional domains in the protein considered with the existing sibling deposited in the protein data bank.
we acknowledge that there is room for improvement.
however, it remains that the h-factor we have introduced here is a first step in the direction of validating homology models for the biologists in addition to existing methods, as proved in the examples shown above.
CONCLUSIONS
homology modeling is slowly building up a record of success and can help structural biologists in many aspects.
models can serve as a bootstrap structure for both nmr and x-ray crystallography and thus help saving a huge amount of time.
in x-ray crystallography for instance, many derivative dataset are often needed to solve the phase problem.
alternatively, an accurate bootstrap would be extremely handy for molecular replacement.
the same applies for nmr.
protein modeling is also crucial for fitting low resolution electron microscopy maps or building accurate models using structural restraints gathered with small-angle x-ray scattering  experiments.
models can be used at different level of details according to their accuracy.
in the absence of experimental structures, they serve as starting points for modeling experiments, such as molecular dynamics studies, docking experiments and structure-based drug design.
for instance, models of membrane proteins such as g-protein-coupled receptor  are extensively used, as few structures are available for this protein family  <cit> .
in this study, we proposed a modeling etiquette that hopefully will help make good use of models.
we introduced the h-factor, a new indicator that assesses the quality of models generated by homology modeling, mimicking the r-factor in x-ray crystallography.
the h-factor is able to detect backbone anomalies as well as give a feedback on the biological relevance of models.
the h-factor evaluates the quality of a protein model within the context in which it is modelled and we believe it is an essential tool that needs to be used in addition to the other validation tools available.
note
to search for protein structures using any of the accession numbers mentioned in this article, please follow this link .
authors' contributions
edl & pk conceived and designed the experiments.
edl performed the experiments.
edl & pk analyzed the data.
edl & pk wrote the paper.
supplementary material
additional file 1
sequence alignments used to build models for casp <dig> targets used to test the h-factor.
sequence alignments for  t0287;  t <dig> and  t <dig> respectively.
click here for file
additional file 2
a simplified operating manual for the h-factor.
operating manual for the online h-factor server.
click here for file
acknowledgements
the national institutes of health , the national research foundation of korea and the kyungpook national university supported the research presented in this paper.
we thank mrs. marie vallet for technical assistance and for setting up the h-factor web tool and dr. xinwei shi for technical assistance and helpful discussions.
we thank the anonymous reviewers for their insightful comments that have helped improved our manuscript.

 introduction since when kendrew et al reported the first atomic-level resolution of a protein structure, the structural biology field dramatically expanded with the development of new tools and methods to gain access into atomic details of a protein or a nucleic acid. this opened a completely new world of knowledge and understanding to the scientific community, as the analysis of protein structures provides fundamental insight into most biochemical functions and consequently into the cause and treatment of diseases. structural biology is now recognized as a fundamental step in our quest to understanding life at the molecular level. finding the structures of all proteins is currently a bottleneck for genomics studies. in this matter, the protein structure initiative aims at the determination of the three-dimensional structure of approximately structures in years. however, the protein sequence databank is growing at a much faster rate, with more than millions sequences available to date. at the same time point, the protein data bank includes structures, out of which only approximately are "unique" at chain level. it should be noted that these structures only represent a biased sample of the protein universe. for example, the pdb includes only unique membrane proteins which is very little since membrane proteins constitute around 20-30% of most proteomes. noteworthy, the human genome has ~ protein-encoding genes for a proteome of ~, proteins when combining the complexity induced by alternative slicing events. in addition, due to experimental limitations, the vast majority of the solved structures are below the kda threshold excluding numerous larger proteins. large proteins however represent a significant fraction of the proteins present in an organism; for instance, proteins found in yeast saccharomyces cerevisiae have five hundred amino acid residues on average and their lengths can reach two thousand eight hundred residues. the structure of these large proteins, as well as of even larger assembly can be solved by electron microscopy at a somewhat low-resolution. while this field is expanding very fast and a growing number of structures solved at atomic-level resolution have been reported, its impact with respect to the size of the protein sequence databank remains limited. many more protein structures have been solved by either x-ray crystallography or nuclear magnetic resonance. it remains however that most proteins are out-of-reach because of technical difficulties. there is clearly a huge gap between the world of known structures and the universe of known protein sequences. structural genomic projects are unable to keep up with newly discovered genes. one way to work around this problem is to use computational methods to predict proteins structures. in silico protein structure prediction techniques can be divided into two categories: the ab-initio folding methods and homology modeling. in this paper we focus on the latter. we note that both approaches have been shown to yield astounding results, as shown in the successive casp contests. however, they do require caution: while predicting the structure of a protein is an intellectual challenge that requires solving many practical issues, it is often considered as an art in essence. the growth rate of structures deposited in the pdb is slowing down since along with the number of new superfamilies or folds discovered. one possible explanation is that many proteins still evade the structural biology pipelines at this time because of the technical difficulties described above. in chothia hypothesized that the number of protein folds in nature is probably finite and around. the latest analysis of the pdb and of the structural classification of proteins showed that we have not yet reached a plateau. the current rate at which proteins are added in the pdb is far too slow to match with the number of new protein sequences discovered every year. the situation is however not so negative. there is a definite hope that the current content of the pdb will allow us to predict reliably the correct scaffold of more than 70% of the whole proteome using in silico methods. this is the rationale for using homology modeling to complement experimental techniques. homology modeling predicts the structure of a protein by inference from a homologous protein whose structure is known. its success rests on the existence of a homologue with known structure, our ability to detect this homologue, and the quality of the model building process once the homologue is detected. steps and have greatly benefited from the different genomics projects: with the additional structures generated by the structural genomics projects, sampling of the protein structure space is becoming finer, improving the chance that a structural homologue exists for any given protein sequence. with the parallel increase in the number of sequences available in genomic databases and the development of meta-servers to analyze and query sequence databases, there has been significant improvement in the detection of homology. in the recent casp experiments, targets with sequence identity of 6% to their templates were included in the homology modeling category. there is hope that in the near future homology modeling will reach its ultimate goal: the generation of model protein structures as accurate as those determined by high-resolution experimental studies. biologists unanimously consider x-ray crystallography as the prime source of structural information on proteins and the "gold standard" in term of accuracy: they base their confidence on its long list of published successes. the vast majority of structures deposited in the pdb were determined by x-ray crystallography and nobel prizes in chemistry and medicine have been awarded to crystallographers. homology modeling on its own however is not devoid of successes. the very first published homology model in was the small protein α-lactalbumin, which was modeled on the basis of the structure of hen egg white lysozyme as a template with the two proteins sharing 39% sequence identity. when the structure of α-lactalbumin was later solved by x-ray crystallography, the model turned out to be essentially correct. since then, homology modeling has continuously extended its field of applications, including designing mutants to test hypotheses about protein functions, identification of active sites, drug design, protein-protein docking, facilitating molecular replacement in x-ray structure determination, refining models based on nmr constraints. despite these successes, homology modeling is not yet a well-established alternative or complement to experimental structural biology. it remains the focus of many criticisms often coming from the structural biologists themselves as they often consider a protein model to be unreliable, not being based on experimental data. the question arises as to what needs to be done to give homology modeling its credentials. in this paper, we focus on homology modeling and more specifically, we review how the structural biology community perceives it and what can be done to impress on the experimentalists that it can be a valuable resource to them. it is organized as follows. the next section reviews the differences and similarities between homology modeling and high-resolution experimental structural biology. in particular, we illustrate steps in the homology modeling procedure that are putative source of errors. our goal is to provide a useful step-by step handbook for non-specialists in order to help building better model. the following section introduces the h-factor, a new indicator for assessing the quality of homology models. the h-factor is designed to check how well a family of homology models reflects the data that were used to generate those models, in the spirit of the r-factors in x-ray crystallography. the results section that follows validates the h-factor on a series of test cases. we conclude the paper with a discussion on what remains to be done to make homology modeling a prime technique for the biologists. homology modeling versus experimental structural biology this section reviews briefly the quality of protein structure models obtained either using high-resolution experimental techniques or homology modeling. our hope is to identify common good practices as well as safeguards from which we can derive a validation tool for the latter. we start with the concept of a structural model and its meaning in the two communities of experimental and computational structural biologists. we then highlight the pros and cons of x-ray crystallography and nmr spectroscopy. an overview of the different steps involved in homology modeling follows, with emphasis on sources of errors and how they can be checked. ultimately our goal is not to rank these methods but rather we hope to show that they all provide valuable and often complementary information, as long as the proper safeguards are applied. what is a model? the meaning of the word "model" is ambiguous in the structural biology community. a model for a protein structure can be obtained either by x-ray crystallography, by nmr spectroscopy, by electron microscopy, by computational methods or by combinations of all or some of these techniques. with experimental techniques, the atomic coordinates are refined against experimental structural restraints and constraints. eventually the final model is called "structure" when the refinement statistics converge toward acceptable canonical values. note that often this final model is subjected to refinement using simulation techniques such as constrained molecular mechanics or molecular dynamics simulations. even though these simulations are constrained with the experimental data, the subsequent "structure" cannot be considered to be fully independent of modeling. on the other hand, an in silico model is generated without or with very limited experimentally constraints: it depends obviously on the hypotheses included in the modeling process, on the force-field used in the simulations as well as on the quality of the scientific computing tools that were used during the modeling steps. while the quality of an "experimental" model can be assessed directly against the experimental data, the quality of an "in silico" model is more subjective and ultimately defined through the usefulness of the model: this is most probably the source of the mistrust towards modeling in general. x-ray crystallography: source of errors and quality metrics a number of factors contribute to the quality of an x-ray structure. the first factor relates to the intrinsic crystal properties and its diffraction capabilities, which is mostly evaluated in term of resolution. the quality metrics used in x-ray crystallography fall into three categories: 1) to measure the quality of the raw data, 2) to measure the agreement of the refined structure against the data and 3) to validate only the model for ideal stereochemistry, rotamers and bad clashes implemented in what check or molprobity for instance. the first category lies upstream of the structure building process as the measures it includes evaluate the quality of the experimental diffraction data. the rsym indicator for example measures the average spread of individual measurements in respect to their symmetry equivalent measurements. a good dataset will have an rsym smaller than five percent. in addition, the quality of a dataset is also assessed by its signal-to-noise ratio <i/σ> and its data-collection completion for a given space group. unfortunately, only a few crystals diffract to atomic resolution with ideal quality metrics. most of the crystal-based structures have therefore been solved with good to average raw data quality. although building a structure can be semi-automatic with automated tools available for chain tracing, side chain-building, ligand building and water detection, it is still refined by experimentalists using their subjective interpretations of the data. it is common for example to find areas of poorly defined electron density map due to disordered regions. the experimentalist interprets these data to the best of her knowledge but this is unfortunately a common source of errors. the second set of quality metrics assesses the relative agreement of the structure in regard to the experimental data. this set includes the r-factor and the "free" r-factor. the r-free is analogous to the r-factor but uses a small subset of the data that have been flagged-out and not taken into account during any refinement process. its purpose is to monitor the progress of refinement and to check that the r factor is not being artificially reduced by the introduction of too many parameters. as such, it provides an unbiased indicator of the errors in the structure and prevents any over-refinement and over-interpretation of the data. both factors along with rsym and rmerge can be seen as indicators of the errors inherent in the refined model and in the experimental data. the quality of protein crystal structures has been reviewed several times over the last fifteen years and it is striking to notice that despite a constant increase of the technology and validation tools, it has not improved overall. the quality spectrum of x-ray structures remains broad. the increase of automation through structural genomic pipelines did not help raising the bar in that matter as human intuition and reasoning are taken out of the process. interestingly, x-ray structures published in high-impact general science journals are usually the worse offenders in term of quality and errors. this is explained by the experimental difficulties associated with solving novel high profile structures and the rush to publish in a competitive environment. x-ray crystallography is not immune to errors and mistakes, both honest and dishonest. unfortunately, over the years we have seen gross mistakes in various structures leading to the retraction of several high-impact papers in leading journals because of a lack of quality control during the structure building pipeline. a recent review by g. j. kleywegt highlights the need and the proper use of validation methods in structural biology in general and in x-ray crystallography in particular. the author also emphasizes the use of validation methods early on in the project pipeline in order to minimize the number of erroneous high-profile structures that can hinder the progress of science for years to come. in light of a growing number of structures falsification and to prevent both dishonest and honest mistakes in structures determination, the curators of the pdb have implemented over the years new sets of validation procedures for the deposition process. nmr spectroscopy: source of errors and quality metrics although structure determination by nmr spectroscopy methods is very different from x-ray crystallography, it shares similarities with the latter in terms of sources of errors. instead of using an x-ray beam diffracting around electrons in a crystal, nmr spectroscopy is performed in solution and uses the magnetic properties of the nuclei with odd spins. as the molecule of interest is placed in a strong magnetic field, each of these nuclei is characterized by a unique resonance frequency, i.e. the frequency at which it will absorb energy. this frequency depends on the local magnetic field that combines the external field and the local environment: it is referred to as the chemical shifts. nmr experiments are designed to monitor the behavior of these nuclei as the system is perturbed from equilibrium and each experiment usually isolates one property, such as through-bond connectivity's or spatial proximity that allows for energy transfer. in the specific case of proteins, the number of nuclei involved can be large, leading to crowding of the spectra: this is usually overcome by using multidimensional experiments. the typical protocol for protein structure determination by nmr proceeds as follows. firstly, the chemical shifts observed on multidimensional spectra are assigned to their specific atoms in the protein. second, through-the-bond and through-space coupling effects observed on these spectra are quantified and concerted into angles and distance restraints. most of these restraints correspond to ranges of possible values instead of a precise constraint. thirdly, a molecular modeling technique is used to generate a set of models for the protein structure that satisfy these experimental restraints as well as standard stereochemistry. for a more detailed presentation of the application of nmr to protein structure determination, we refer the reader to. analogously to x-ray methods, the quality of nmr measurements affects the quality of the structures. the precision of a set of models for a protein structure determined by nmr is evaluated as the root-mean-square difference between each model and a "mean" structure, defined geometrically as the mean of all the models. the quality of each model is evaluated by the number of violations observed in the model compared to the experimental restraints. a high-quality nmr structure refers to a set of high quality models with no violations that are tightly bundled around their mean, i.e. with a small rms. note that in addition to these nmr specific quality measures, garrett and clore introduced a r-factor and a free r-factor for the refinement of nmr structures based on residual dipolar coupling, a long range nmr measure obtained on proteins that have been partially oriented in dilute liquid crystals. in similarity with x-ray crystallography, the quality spectrum of nmr structures is broad, with errors and mistakes reported that are inherent to a human based determination process. homology modeling: source of errors and quality metrics the general strategy developed for homology-modeling proceeds through a canonical seven-steps procedure: identify the template proteins that share structural similarity to the target; align the target sequence with the templates sequences; build a single framework of spatially aligned template structures and assimilate the target protein backbone with this framework; build the missing backbone elements not represented in the template framework; build the target side chains; refine the model in order to minimize unrealistic contacts and strains; and evaluate the final refined model for physical tenability. to date numerous homology-modeling programs such as modeller, segmod/encad, swiss-model, 3d-jigsaw, builder and nest have been developed and many of them have been embedded into homology-modeling servers to ease the burden of generating models. online portals such as the protein structure initiative model portal or the swiss-model repository bring to the community a large database of models. the psi model portal currently provides millions comparative protein models for million distinct uniprot entries. every model comes with relevant validation data. however those models are automatically generated without any human interaction that might render them inaccurate without any extra validation steps. in the following, we overview each step of the homology modeling process and highlight potential sources of errors. *steps and 2: template selection and sequence alignment the selection of template is undoubtedly a critical step in modeling. it was long assumed that two proteins whose sequences share at least 40% identity have similar structures. if such a template exists, it is easily detected by any sequence alignment techniques. homology modeling under such conditions is then expected to generate models whose accuracy is close to that of an experimental structure. we know however that this is not always true. roessler et al. recently reported the discovery of two native cro proteins sharing 40% sequence identity but with different folds. moreover, alexander et al. were able to design two proteins with 88% sequence identity but having different structure and function. reversely, it is not uncommon for proteins, especially enzymes carrying the same function across the tree of life to share a somewhat low sequence identity and at the same time being structurally similar, with a cα r.ms.d. ~ Å. all these observations indicate that the selection of template is far from being a trivial task and extreme caution should be applied. the situation is even more difficult if there is no significant sequence similarity between the sequence of the target protein and any of the known protein structures. this is one of the current challenges of the post-genomic era that is tackled by fold-recognition methods, namely to identify a suitable template for homology modeling. unlike sequence-only comparison, fold-recognition techniques take advantage of the information made available by d structures. despite a steady development over the years, as illustrated through the successive casp experiments, fold recognition techniques still have a number of limitations. they are however the key to extend the domain of application of homology modeling methods. the accuracy of the sequence alignment is another critical step in the homology modeling process. it is important to keep in mind that a shift of one residue in the alignment introduces a distortion of Å in the model for the backbone of the target protein. therefore, it is of paramount importance to precisely verify and locally refine the sequence alignment used to build a model. for that matter, the areas of sequence comparison and alignment optimization have grown in parallel to the comparative modeling field. there are many ways to align protein sequences and numerous methods to score the accuracy of the resulting alignments. homology modeling usually relies on one of following three techniques: standard pairwise sequence alignment using dynamic programming, multiple sequence alignment when the target and template sequences belong to a large family for which many sequences and structures are known, and direct alignment of the sequence on the structure of the template using a threading technique. none of these techniques is the panacea, especially when the sequence identity between the target and template sequences is low. errors come for example from the fact that the optimal alignment between the sequences of two proteins may not match the alignment between their structures: this is caused by the sequence alignment technique that optimizes a score based on a substitution matrix and gap penalties while structural alignment techniques optimize geometric matching. there is no reason that these two different metrics are equivalent. errors are often frequent in the loop regions between secondary structures, as well as in regions where the sequence similarity is low. recent techniques that attempt to reduce the error rate in the sequence alignment rely on the inclusion of as many sources of information as possible, such as amino-acid variation profiles, secondary structure knowledge, structural alignment data of known remote homologues, knowledge of "anchor regions". many programs also attempt to optimize the raw sequence alignment derived from one of the techniques described above. most of these programs have been embedded into web-servers for ease of uses, such as muscle and dialign-tx. * step 4: loop building the loop-building step is another key component in homology modeling. loops participate in many biological events and functional aspects such as enzyme active sites, ligand-receptor interactions, and antigen-antibody recognition among others. however, due to the flexible nature of loops, it is often difficult to predict their conformation. there are two main approaches to tackle the problem of loop modeling: methods that use databases of loop conformations or ab initio methods. in the database approach, a library of protein fragments whose size corresponds to the size of the loop to be modelled is scanned for fragments whose end-to-end distance matches the corresponding distance in the framework. the library is derived from the known protein structures in the pdb. this method has proved to be accurate when the loop is relatively short. fidelis et al. have shown that loops of a maximum of seven residues can be modelled with confidence based on known structures. when the database method is combined with a restrained energy minimization, it extends the confidence of loop building up to nine residues. beyond the nine residues threshold, ab initio methods have to step in mostly because for these longer loops, the fragment library provides a poor sampling of the conformational space accessible. the ab initio loop prediction approach relies on a conformational search guided by a scoring function. the accuracy of ab initio loops modeling remains currently low, especially when dealing with very long loops. * step 5: the side-chains positioning problem the prediction of side-chain conformations for a given backbone architecture remains a challenge. it is however the key to generating models at atomic resolution. for instance, critical side-chains forming an active site must be accurately positioned in order to support any putative catalytic mechanism. nearly all the side-chain positioning methods are based on rotamer libraries with discrete side-chains conformations. rotamer libraries contain a list of all the preferred conformations of the side-chains of all twenty amino acids, along with their corresponding dihedral angles. some of these rotamer libraries are further refined to account for the local geometry of the backbone. sidechain prediction techniques select the best rotamer for each residue of the protein under study from one of these libraries, based on a score that includes both geometric and energetic constraints. this leads to a large combinatorial problem: see figure for an example of the size of the conformational space accessible to a sidechain in a protein, even when we discretize this space by using rotamers. the combinatorial problem is solved by heuristic techniques such as mean field theory, derivatives of the dead-end elimination theorem or monte carlo techniques. the success rates of the most successful techniques range from 78% to 89% for the χ and χ angles of residues in the core of the protein. it is important to note that these results usually relate to mock experiments in which the exact conformation of the backbone is known. it has been shown that the quality of sidechain prediction decreases as the deviation between the backbone used for modeling and the actual conformation of the backbone increases. these results emphasize the need of a good framework for homology modeling. it is worth mentioning however that a 1Å accuracy over the main-chain atom position can be related to x-ray structures with Å resolution and with a r-factor around 25%, which is fairly common. in addition, multiple side chain orientations are routinely observed in crystal structures and crystal-packing forces can alter their positions as well. the side chain prediction methods are improving steadily, as reported by the casp experiments. * step 6: refinement of the final model in a review written in on the casp experiment, koehl and levitt noted that most models submitted in the homology modeling category were not refined, as previous casp meetings had shown that refinement did not improve the models. sadly for the computational biologists the situation has not improved and it remains difficult to generate a model closer to the native structure than the template used to build it. energy refinement, originally introduced by levitt and lifson forms the basis of current methodologies for protein structure refinement against experimental data. without experimental restraints, refinement by energy minimization generally moves the protein structure away from its x-ray structure. some recent studies have shown that this negative trend can be reversed through the inclusion of evolutionary derived distance constraints through the combination of sophisticated sampling techniques based on replica exchange molecular dynamics and statistical potentials, through the addition of a carefully designed, differentiable smooth statistical potential, or by careful consideration of the solvent effects. while these studies are definitely source for hope, much work remains to be done as far as refinement is concerned. a general framework for model assessment: r-factors and equivalent the wide availability of homology modeling software packages, as well as the development of web interfaces that automate the use of these packages has resulted in better access to and a broader usage of homology modeling. while this is definitely commendable and homology modeling should be even more advertised, there are risks that this will lead to errors because of the difficulties in evaluating the correctness of the models these techniques generate. this is primarily due to the lack of cross validation indicators such as the r-factor and r-free in x-ray crystallography. in addition to the stereochemistry assessment of the structure and the good correlation between the r-factor and the r-free values, the quality of an x-ray structure can be evaluated based on the thermal motion value of atoms described by the debye-waller factor or b-factor. the b-factor allows for the identification of zones of large mobility or error like disordered loops. when multiple monomers populate an asymmetric unit of a crystal, the crystallographer will choose to focus on the analysis of the monomer with the lowest average b-factor since the likelihood of errors is lower. unfortunately, such criteria do not apply to models thus rendering the identification of zones of uncertainties a non-trivial task. with respect to homology modeling, the main step is to thoroughly validate the model and always provide all the relevant details about the protocol used. this will give the user all the necessary data to judge the quality of the model. the quality assessment of models has been the focus of numerous studies and various algorithms have been reported over the years. in this matter, tremendous efforts are being made to produce the best triage procedures or scoring functions among models, as seen in the latest casp meetings. these scoring functions are based on statistical potentials, local side-chain and backbone interactions, residue environments, packing estimates, solvation energy, hydrogen bonding, and geometric properties. in addition, it is essential that the quality of stereochemistry be kept high. the stereochemistry can be assessed by commonly used programs such as procheck or whatif. ultimately, the validation of models comes from experiments such as site-directed mutagenesis, circular dichroism, cross-linking, mass spectrometry, fluorescence-based thermal shift, light scattering, molecular fret or electron microscopy. such experimental data can be translated into constraints/restraints and introduced in the modeling protocols thus improving the accuracy of models. one can also identify fast and cheap experimental procedures that can help testing homology models. the easiest way is to crosscheck models with experimental structures. for instance with enzymes, it is possible to verify the location of important catalytic residues in the active site by comparison with homologous family members. most importantly however, a model needs to be checked manually in the same way a nmr or an x-ray structure is processed. despite all these methods, the homology modeling community still lacks a simple indicator which gives an unambiguous feedback on how the final model, or family of models, reflects the data that were used in the modeling process, similar to the couple r-factor/r-free for x-ray crystallography. the next section introduces such an indicator, namely the h-factor. methods computing the h-factor in this study, we introduce a novel indicator as an attempt to quantify how well a homology model correlates with the input data and its biological relevance. this indicator, the h-factor is designed to mimic the r-factor in x-ray crystallography. it is rooted in the basics of homology modeling and it uses all data that were included in the model building process to assess its correctness, in addition to checking for good stereochemistry. more specifically, the h-factor combines information on the template structure; the sequence alignment between the template and the target sequences; the structural heterogeneity of the models built; and the structural neighborhood within protein families. note that at this stage, the computation of the h-factor is based on a single framework. we plan to derive an extended h-factor that will account for multiple templates. each of these four categories is assigned a scoring function returning a value between and. the h-factor is simply the sum of these scoring functions divided by for ease of use, the h-factor is converted into a percentage with a value between 0% and 100%. in par with the r-factor in x-ray crystallography, a low h-factor value indicates a trustworthy set of models, while a large h-factor would raise a flag on the correctness of the models. the h-factor focuses on the cα-backbone as a correct tracing is a prerequisite for a valid model. the workflow of the h-factor calculation is detailed on figure 4a and 4b. the scoring function analyses the discrepancies between the secondary structure prediction for the target obtained with the program psipred and the actual secondary structures of the template framework computed with the program stride. the corresponding score takes into account the confidence factors reported by psipred: score1=a∑i=1nfn+bf={0for p=sc+110for p≠sfor gaps in1the sequencealignment where the sum is computed over all positions in the sequence alignment between the target and template, n is the length of the sequence alignment, p is the secondary structure prediction of the target at position i, c is the confidence factor reported by psipred for the secondary structure prediction at position i and s is the secondary structure type observed at position i in the template structure reported by stride. the offset coefficients a and b are set to and, respectively, to ensure that score has values between and the function scores the identity between the sequence of the target and the sequence of the template framework. score2=10n)g={0for non identity1for identity where n is the length of the sequence alignment. the search for a template structure for a given target protein sometimes finds multiple candidates. in addition, considering multiple options for this alignment sometimes alleviates ambiguities in the alignment between the sequence of one of these candidates and the sequence of the target. both situations lead to multiple structural models being generated for the target protein: score is designed to measure the heterogeneity of this set of models. it uses as input all the models mi, as well as the corresponding average model, ma, whose atomic coordinates are the averages of the corresponding coordinates in the models. the average model ma is computed from the structural alignment of all the models, which can be computed for example with the cluster() function in modeller or with the program maxcluster. the function then reports the average crms between each model and the average model, where the crms is computed over the cα atoms only. the average crms is then transformed linearly such that the final score is between and 10: score3=an)+b n is the number of models. the offset coefficients a and b are chosen such that average rms values of and Å correspond to scores of and respectively; the corresponding values are a = and b =. the search for a template structure sometimes finds partial matches. in many cases, this is related to the fact that the target protein contains multiple functional domains that may not always be associated in other proteins. while the modeling is performed based on the longest possible template, it is expected that each of these domains have been correctly modelled. score is designed to quantify this assertion. it first identifies the various domains in the sequence, using hmmer and the pfam profiles database as a reference, the average model ma is then broken down into fragments corresponding to the domains that have been identified. each fragment is compared to the structure of the same domain found in proteins whose structure has been deposited in the pdb. a maximum of fragments is considered. to minimize the numbers of false positive we set an e-value cut-off of e- for hmmsearch. the score is then the average crms distance between the fragments and their counterparts in the pdb: score4=amn)+b m is the number of functional domains identified in the target sequence, mad is the structural fragment extracted from the average structure ma corresponding to the domain d, n is the number of domains homologous to domain d found in pdb structures, and dd,i is the i-th possible structure of the domain homologous to d. the offset coefficients a and b have been chosen such that average rms values of and Å correspond to scores of and respectively; the corresponding values are a = and b =. this usually enforces that score is between and note that if this procedure does not find an equivalent domain for a fragment, the fragment is ignored; if no domains are found for all fragments, score is ignored. the h-factor computation is accessible online at http://koehllab.genomecenter.ucdavis.edu/toolkit/h-factor with a simplified operating manual. the source code is available upon request. testing of the h-factor on casp targets target proteins that have been submitted to casp in the homology modeling category are perfect tests for the h-factor measure presented here. the web site for the protein structure prediction centre that hosts all information about the casp experiments http://predictioncenter.gc.ucdavis.edu also includes a database of all the models that have been submitted in the successive casp with the corresponding experimental structures. this database of casp models contains models that can be ranked from being very accurate to being completely wrong. it is therefore ideally suited to benchmark modeling techniques and to develop and test validation methods. we have chosen three targets from the recent casp experiments: t t and t that have been identified as homology modelling targets by the casp organizers. the first target was considered "easy", while the second and third target were more difficult cases for homology modeling. fold recognition techniques initially identified one template for t and t and six potential template structures for t the top templates were then selected according to the casp analyses of every possible template for each target. we aligned the sequence of the target with its template using clustalw, with the gonnet matrix to define the substitution score and default settings for gap penalties. for the first two targets, this corresponds to a simple pairwise alignment, while for t clustalw provides a multiple sequence alignment over the target sequence and the six possible template sequences. we used modeller 9v, with the "automodel" settings to generate models for each of the three targets. as the h-factor focuses only on the cα-backbone, we did not attempt to improve the prediction of the sidechains. in addition, we did not perform any energy minimization of the final model and it still remains unclear if energy minimization improves models generated by homology modeling. template ) %sequence identity between template and target coverage type of sequence alignment loop segments of or more residues  the h-factor: detailed analysis on three casp targets the casp target t a dimethyl adenosine transferase from plasmodium falciparum is considered an "easy" modeling case: it has a very close homologue whose structure has been solved at high resolution and in addition the template sequence covers the whole sequence of t we generated models for t using standard homology modeling techniques. in this experiment, our goal was not to generate the ultimate, high-resolution model. instead, we were interested to see if the h-factor we have introduced was able to measure the quality of the models we generated. the average crms between the t models and the actual experimental x-ray structure for target t is Å: this clearly indicates that the models are of good quality. the corresponding h-factor for these models is 19%, i.e. a very good score. the good quality of the t models is highlighted by each scoring function included in the h-factor. the secondary structure prediction for t matches well with the actual secondary structure of its framework, yielding a value of for score. the sequence alignment between t and 1zq is deemed good, with a value of for score. the models generated showed little structural dispersion with a corresponding value for score of. t contains one pfam domain, pf that corresponds to the ribosomal rna adenine dimethylase. in addition to being found in the structure 1zq this domain is also found in the proteins whose structures are available in the pdb files 1i4w, 1qam, 1qyr, 1yub and 2erc. score compared the structure of this domain in the models generated for t and the structures of the same domain found in all proteins listed above. it detected fluctuation between these structures, leading to a score of. note that the pfam domain pf occurs in species covering all three kingdoms. in addition, the structures have been derived from data with wide range of resolution. it is therefore not surprising that score is relatively higher than the other scores; it does remain however within a range that indicates a good match. note that the overall h-factor value is 19%. in comparison, a r-factor of 20% is typically observed for fully refined x-ray structures around Å of resolution, i.e. for a good x-ray structure. the four scoring functions included in the h-factor measure the quality of the secondary structure prediction ), the diversity of the sequence alignment ), the structural diversity of the models generated ), and the similarity of the predicted structures for the functional domains in the target, compared to the structures of the same domains found in the pdb ) average crms between the different models generated and the actual experimental structure for the target. sequence identity between the target sequence and the template sequence dope scores from modeller qmean normalized score t0295* is the toy experiment in which the sequence alignment between the target t and its framework has been deliberately modified. when we deliberately introduce a shift at position in the alignment between the sequence of t and the sequence of its template 1zq, the corresponding models generated by modeller show structural diversity in the loop region near the shift. score captures this structural diversity within a set of models. it leads to the h-factor being raised from 19% to 21%. however, score could not detect a single position shift in the alignment. the h-factor is therefore capable of detecting backbone deviation due to modeling errors, the same way the r-factor does. the casp target t is a more difficult modeling case. it is a human ketohexokinase and the rigid-body domain closure of sugar kinases is known to be large, adding complexity into the modeling process. although several sugar kinase structures have been solved, the search for templates for t identified only six distinct remote templates. moreover, all six templates are needed to obtain complete sequence coverage of t within one single framework with modeller. in addition, the template sequences have low similarity with the target sequence. this is detected by the scoring function, which returns a value of. note that the score is not a direct measurement of the quality of the sequence alignment. it is designed to quantify the difference between the two sets of sequences: if this difference is small, the model is expected to be good, while if the difference is large, the sequence alignment most probably belongs to the twilight zone and the models should then be considered with caution. the overall h-factor for the models generated for t is 41%. this mid-range value indicates that caution should be used when interpreting or using these models. indeed, the average crms between these models and the actual structure of t is Å, i.e. reflecting a medium-resolution agreement. the casp target t is the most difficult test case we have considered. in fact, it would not be considered a homology-modeling target by many, despite the fact that a structural homologue is available in the pdb. we did decide to include it in our study to test whether the h-factor was still providing useful information when applied on a difficult test case. t corresponds to cags, a protein from helicobacter pilori whose function is unknown. a database search over all sequences of proteins whose structure is known identifies a unique template, 1v with a low sequence identity. 1v is a cytochrome c oxidase and it is not clear that t and 1v are homologues. we did build models for t using 1v as a template, and the out-of-the-box alignment between the sequences of t and 1v generated by clustalw. as mentioned above, we did not try to optimize the alignment or the modeling itself as our interest is to see if the h-factor is able to assess the quality of the models we generated. in this specific case, all four scores reported high values,, and, respectively). the overall h-factor is 75%, a valued that should raise concerns about the quality of these models. as for target t this is confirmed by the experiment: the average crms between these models and the actual structure for t is Å, indicating that the models are poor approximations of the native structure. h-factor: detecting bad models from the three test cases t t and t we conclude that the h-factor correlates well with the quality of the models tested. to further confirm that high h-factor values correspond to models that should be considered with caution, we downloaded sets of models for seventeen target proteins from casp these targets were considered as difficult, with only remote templates available. the corresponding sets of models, available from the casp web site have poor to bad quality. when compared to the actual experimental structures, these models have a very high crms indicated major errors in the backbone tracing. in all sixteen cases, the h-factors are found to be high, as shown in table crms between the average model and the actual experimental structure for the target. dope scores from modeller qmean normalized score h-factor: characterizing good models the h-factor computes the quality of models for protein structure based on sequence information and score ), as well as based on structure information and score ). while the former is specific to homology modeling, the latter can be used to assess the quality of any sets of models. we tested score and score on a random set of nmr structures for which or more models are available. in this validation case, the h-factor is the sum of functions divided by instead of and converted to percentage. results are listed in table in every case, the h-factor is low and below 35% in average, as anticipated. this indicates that the algorithm recognized the experimental structures as valid. see text for the description of the scoring functions and; these two functions measure the structural dispersion of the models considered, as well as their similarities to homologue structures in the pdb; note that scores and usually present in the h-factor are meaningless for this test set. average rms between each model and the "mean" structure computed from all the models. although the h-factor and the r-factor are mathematically unrelated, they have the same purpose: to assess the quality of structures, either experimental or computed from modeling experiments, where quality refers to reflecting correctly the input data used to generate these structures. the h-factor mimics the r-factor as it provides a quality-index to follow in the process of building a model, the same way crystallographers monitor the r-factor/r-free indexes during structures refinements. we compared the h-factor results with the experimental r-factor and r-free on a randomly chosen subset of the pdb containing structures with or more identical chains solved by x-ray crystallography. results are shown in figure the h-factor and r-factor are not linear correlated but it remains that "good" r-factors correspond to "good" h-factor values. the h-factor checks the diversity of the set of models generated for a structure, as well as their similarities with the structures of domains that share the same function, as defined by pfam. high h-factor values may be caused by structures with disordered loops or remote structural neighbors in the pdb. it remains that, in agreement with what we observed for the nmr structures, the h-factor recognizes experimentally determined structures as being valid. relationship between the h-factor and crms the crms by itself is a reasonable quality indicator as long as its value remains low. it should be noted that it is an average value computed over the whole structure. as such, it is very sensitive to large structural fluctuations in disordered loops for example that can lead to large crms values even if the conserved domains are structurally very similar. it is well known for example that caution should be applied when using crms to assess the quality of a structural alignment. crms is implemented in score of the h-factor to evaluate the heterogeneity amongst a set of models as it is directly related to both the choice of the template and the quality of the sequence alignment. however, the score loses accuracy for crms values larger than Å, which is not uncommon when a remote template is used. crms is also implemented in score to quantity the modelling quality of specific individual domains by comparing them with corresponding domains in the pdb. this is a domain-based crms that does not take into account potential long loops between domains, making it more reliable. taken together, the scores and alleviate most of the limitations of crms while retaining its major properties. the h-factor is therefore expected to be more reliable than a sole crms to judge the accuracy of a wide range of models, as seen in table comparing the h-factor with prosa, dope and qmean the structural biology community as well as the protein structure modeling community have always been reasonably good at setting safeguards to estimate the validity of both experimental and computational models for protein structures. the first validation tests imposed on experimental structures focused on the stereochemistry of the molecules. it was found, however that this was by far not sufficient, as "good stereochemistry" can be misleading. a "clean" ramachandran plot for example does not necessary prove that a given model is valid. also, we observed that all the models listed in table have good stereochemistry, including the t model that has a cα backbone deviation of almost Å from the actual experimental structure. experimentalists have access to another essential set of validation tools that check the consistency of the structures with the experimental data that were used to derive them. such tools include the r factors in x-ray crystallography and nmr. models derived from modeling experiments are more difficult to verify, unless they include some external information. for example, the swiss-model server provides a confidence score along with the models it built and this confidence score is based on the amount of structural information that supports each part of the model. in general, however, validation of computer-generated models is based on comparison with known protein structures. the idea is that basic properties of protein structures can be inferred from the pdb, and translated into database-derived scoring functions. this is the main rational for the statistical potentials, also referred to as mean-field potentials that have become very popular validation tools. for example, prosa-web implement two statistical potentials: a pairwise potential based on cß atoms and a surface term that models the protein-solvent interactions. here we compare the output of prosa and h-factor analyses on the three casp test cases given in table the prosa results are given if figure both the h-factor and prosa estimate that the models generated for t are valid. when the modeling process is deliberately altered, again both prosa and the h-factor analyses reveal a loss of quality. prosa however considers the models generated for the targets t and t as statistically correct, while we know that these models are not correct. in addition, it is striking to see that the pdb structure 1us an ultra high-resolution crystal structure at Å resolution, has a prosa z-score statistically as correct as t. the statistical potential discrete optimized protein energy is another measure of model quality that has been introduced in modeller-. dope is a statistical potential with an improved reference state that accounts for the compact shape of native protein structures. the dope score is designed such that large, negative scores are usually indicators of good models. in their original study, shi and sali found that the accuracy of dope to asses a homology model improves as the accuracy of the models improve. we observe a similar behaviour for targets t and t0295*. these two targets correspond to the same protein and it is therefore possible to compare the dope scores of their models. the model generates for t0295*, based on an incorrect alignment, has a much lower dope score that the model generated with the correct alignment. note that we cannot compare dope scores for proteins of different size, as these scores are not normalized. dope scores are therefore relative, and designed to pick a "good" model among poorer model. dope scores do not assess directly the quality of the model that is picked, i.e. if it is likely to be similar to the actual structure. the h-factor is a better indicator in that respect. qmean, which stands for qualitative model energy analysis, is a composite scoring function for homology models that describes the major geometrical aspects of protein structures as well as the agreement between the predicted and calculated secondary structure and solvent accessibility, respectively. as such, it includes a term similar to the score of the h-factor, as well as terms that assess different properties such as residue accessibility. the score qmeannorm is a normalised version of the qmean score in which all terms are divided by the number of interactions/residue in order to avoid a size-bias of the score. qmeannorm scores vary between and with larger scores expected to correspond to better models. unlike the dope score, both the h-factor and qmeannorm scores allow for the comparison of proteins of different sizes. the qmeanscore is as effective as prosa or dope for detecting errors in a model that result from errors in the sequence alignment between the template and target protein: t0295* has a qmeannorm score of while the score fort is. interestingly, t0295* has a less favorable qmeannorm score than the erroneous model generated for the casp target t. we have observed however that the qmeannorm score is prone to fail: some of the erroneous models generated for the casp target t have qmeanscores of to, i.e. they are evaluated to be almost as correct as the positive control t. unlike prosa and qmean, the h-factor did detect that these models were to be considered with caution. because it analyzes a set of models, we believe that the h-factor score is more robust as an absolute measure of the quality of a model. it lacks however the ability to discriminate among a set of models generated for the same target; prosa and dope are better potentials for this specific task. these results emphasize the essential differences in the nature of the prosa, dope, qmeannorm and h-factor scores. prosa, dope and qmean check the quality of a model, independently of the context in which it was generated. the h-factor on the other hand checks the quality of a set of models with respect to a context that includes for example the sequence alignment assessed by the score. the modeler however should use these differences to extend his/her assessment of the model his/she generates. we believe that prosa, dope, qmean and h-factor analyses are needed to provide a better overview of the quality of models derived by homology modeling. current limitations and originalities of the h-factor the h-factor is not the panacea, and does not provide a universal solution to the problem of asserting the quality of a model generated by homology modeling. firstly, the h-factor has some technical limitations. our current implementation does not take into account multiple templates, but rather only one single framework. the structural components included in the h-factor and ) are based on the backbone of the models, and do not take into account sidechains and possible errors in their modeling. second, the scoring function of the h-factor measures the heterogeneity of a set of models generated with the same input. it means, that the h-factor cannot be computed on a singular model. in homology modeling the heterogeneity of models can be seen as a quality indicator and building only one single model is not recommended. similarly to nmr structures where only one of the models can be chosen for analysis, the best model in homology modeling regime is chosen based on the modeller energy function for instance. third, the h-factor does not include any external information. for example, if some biological data are available, such as the knowledge of the residues involved in the active site, or standard biophysical data such as melting temperature, or secondary structure content derived for circular dichroism, these data are currently not included in the h-factor analysis. the r-factor is a measure of the agreement between the crystallographic model and the experimental x-ray diffraction data. despite the lack of'experimental' data to compare with, the modelling community has been searching for a similar indicator for homology modeling. both qmean and the h-factor are designed to be'absolute' indicators that assess the quality of homology models in a way that mimics the r-factor in x-ray crystallography. both qmean and h-factor provide an easy-to-use estimate of the quality of models based on scoring functions assessing various aspects of the modelling process as well as the model itself. in vivo macromolecular structures oscillate between numerous conformers, some more than others. while x-ray structures correspond to snapshots of a limited numbers of conformers, nmr structures tend to describe more accurately flexibility. indeed, nmr "structures" are usually provided as a family of conformers that are meant to sample the conformational space accessible to the molecule of interest. in homology modeling on the other hand, the heterogeneity of models is a quality indicator. a good set of models will have a crms very close to their framework. moreover, if errors are being made in the template choice or in the sequence alignment, then the models will be heterogeneous. the scoring function is designed to quantify this assertion. it also means that the h-factor cannot be computed on a single model. one of the originality of the h-factor is the scoring function. it has been designed to evaluate the biological relevance of the models by comparing the model conformations of all the functional domains in the protein considered with the existing sibling deposited in the protein data bank. we acknowledge that there is room for improvement. however, it remains that the h-factor we have introduced here is a first step in the direction of validating homology models for the biologists in addition to existing methods, as proved in the examples shown above.  homology modeling is slowly building up a record of success and can help structural biologists in many aspects. models can serve as a bootstrap structure for both nmr and x-ray crystallography and thus help saving a huge amount of time. in x-ray crystallography for instance, many derivative dataset are often needed to solve the phase problem. alternatively, an accurate bootstrap would be extremely handy for molecular replacement. the same applies for nmr. protein modeling is also crucial for fitting low resolution electron microscopy maps or building accurate models using structural restraints gathered with small-angle x-ray scattering experiments. models can be used at different level of details according to their accuracy. in the absence of experimental structures, they serve as starting points for modeling experiments, such as molecular dynamics studies, docking experiments and structure-based drug design. for instance, models of membrane proteins such as g-protein-coupled receptor are extensively used, as few structures are available for this protein family. in this study, we proposed a modeling etiquette that hopefully will help make good use of models. we introduced the h-factor, a new indicator that assesses the quality of models generated by homology modeling, mimicking the r-factor in x-ray crystallography. the h-factor is able to detect backbone anomalies as well as give a feedback on the biological relevance of models. the h-factor evaluates the quality of a protein model within the context in which it is modelled and we believe it is an essential tool that needs to be used in addition to the other validation tools available. note to search for protein structures using any of the accession numbers mentioned in this article, please follow this link. authors' contributions edl & pk conceived and designed the experiments. edl performed the experiments. edl & pk analyzed the data. edl & pk wrote the paper. supplementary material additional file 1 sequence alignments used to build models for casp targets used to test the h-factor. sequence alignments for t0287; t and t respectively. click here for file additional file 2 a simplified operating manual for the h-factor. operating manual for the online h-factor server. click here for file acknowledgements the national institutes of health, the national research foundation of korea and the kyungpook national university supported the research presented in this paper. we thank mrs. marie vallet for technical assistance and for setting up the h-factor web tool and dr. xinwei shi for technical assistance and helpful discussions. we thank the anonymous reviewers for their insightful comments that have helped improved our manuscript.
the analysis of protein structures provides fundamental insight into most biochemical functions and consequently into the cause and possible treatment of diseases. the growth of modeling techniques and more specifically of those that rely on the existing experimental knowledge of protein structures is intimately linked to the developments of high resolution, experimental techniques such as nmr, x-ray crystallography and electron microscopy. as the structures of most known proteins cannot be solved experimentally for technical or sometimes simply for time constraints, in silico protein structure prediction is expected to step in and generate a more complete picture of the protein structure universe. molecular modeling of protein structures is a fast growing field and tremendous works have been done since the publication of the very first model. we have developed a web service for computing the h-factor for models of a protein structure. for that purpose, we introduce the h-factor, a new indicator for assessing the quality of homology models, mimicking the r-factor in x-ray crystallography. the methods for computing the h-factor is fully described and validated on a series of test cases. in this paper, we focus on homology-modeling and more specifically, we review how it is perceived by the structural biology community and what can be done to impress on the experimentalists that it can be a valuable resource to them.
 the analysis of protein structures provides fundamental insight into most biochemical functions and consequently into the cause and possible treatment of diseases. as the structures of most known proteins cannot be solved experimentally for technical or sometimes simply for time constraints, in silico protein structure prediction is expected to step in and generate a more complete picture of the protein structure universe. molecular modeling of protein structures is a fast growing field and tremendous works have been done since the publication of the very first model. the growth of modeling techniques and more specifically of those that rely on the existing experimental knowledge of protein structures is intimately linked to the developments of high resolution, experimental techniques such as nmr, x-ray crystallography and electron microscopy. this strong connection between experimental and in silico methods is however not devoid of criticisms and concerns among modelers as well as among experimentalists.  in this paper, we focus on homology-modeling and more specifically, we review how it is perceived by the structural biology community and what can be done to impress on the experimentalists that it can be a valuable resource to them. we review the common practices and provide a set of guidelines for building better models. for that purpose, we introduce the h-factor, a new indicator for assessing the quality of homology models, mimicking the r-factor in x-ray crystallography. the methods for computing the h-factor is fully described and validated on a series of test cases.  we have developed a web service for computing the h-factor for models of a protein structure. this service is freely accessible at http://koehllab.genomecenter.ucdavis.edu/toolkit/h-factor. 