BACKGROUND
given the advent of high-throughput sequencing technologies and the resulting data explosion, there is an urgent requirement to provide electronically inferred functional annotations  for proteins whose biological functions have not yet been determined by experimental investigation.
this has given rise to a large number of annotation inference pipelines  <cit>  which are used routinely to produce new annotations that are stored in a variety of publicly accessible databases.
therefore, when looking for fas related to a particular organism, it is commonplace to find several resources each providing annotations that have been predicted using different pipelines, at different times and under different conditions.
consequently, end-users of fas have to decide between multiple resources.
however, since there is no single accessible and objective way to compare these fas, these decisions are in many cases made arbitrarily, whereas different biological problems may require consideration of annotations from different points of view.
where pipeline outputs reference a controlled vocabulary or ontology, relative comparisons of these outputs can be made by using a reference protein-set as a common input.
many pipelines endeavour to annotate gene products to the three aspects of the gene ontology : molecular function , biological process , and cellular component .
for each go aspect, a given gene product can therefore be annotated with a set of go terms.
go provides not only a unified and shared controlled vocabulary for annotation but a structured representation of knowledge.
the power of this annotation system over a flat controlled vocabulary is twofold.
firstly, it can be used to evaluate the functional distance between genes  <cit>  by quantifying the difference between their annotation sets based on the semantics codified in the structure.
secondly, it defines a hierarchy between the go terms that leads to annotations being more or less specific, i.e.
more or less biologically informative.
the aim of this study is to define and implement an ensemble of measures that provide both a quantitative and a qualitative description of go fas and also allows a comprehensive inter-comparison between them.
so far, a plethora of such measures have been proposed but no attempts have yet been made to define and implement a unified framework where fas can be compared in a systematic way.
a notable exception is the blast2go suite  <cit>  where several metrics describing fas are implemented together, for example to measure the specificity of annotations or to automatically compare groups of go terms.
however in blast2go, these metrics are only used to measure specific aspects of blast2go performance and furthermore, the blast2go suite is not designed to compare fas issued by other pipelines.
in most studies, the different properties of fas are implemented and studied independently e.g.
<cit> .
in addition to facilitating the comparison of annotation sets produced by various annotation pipelines, having a common unified framework that implements these metrics would assist in the evaluation of new metrics, the exploration of the influence of parameters in annotation pipelines and the monitoring of annotations over time.
in this paper, we propose a set of twelve metrics and implement them in a python open source library named aigo for the analysis and the inter-comparison of go functional annotations.
we demonstrate the use of the aigo library for evaluating and comparing the output of annotation pipelines using three illustrative examples.
methods
in this section, we define a set of twelve metrics , each of them describing particular features of fas, see table  <dig>
the different metrics computed by aigo allow describing particular features of functional annotations.
definitions of metrics
for any given organism Ω, we call functional annotation, noted ℱ, the mapping between a set of gene products gΩ from this organism and some functional data.
in practice, a set of gene products from an organism can be defined in various ways, ranging from a comprehensive list of proteins  to a list of ests assembled into potentially incomplete or inaccurate unigenes , see  <cit> .
a common scenario is that the set of gene products corresponds to some defined reference set: for example the list of target sequences from a microarray.
the functional data are usually provided as annotation sets containing terms from controlled vocabularies or reference ontologies.
we note  a given ontology, defined as a direct acyclic graph connecting a set of terms to.
in this study,  can be any of the three aspects of go .
more formally, a functional annotation of a set of gene products gΩ over a given ontology , is a map ℱo:gΩ→2o that associates with each gene product g ∈ gΩ a subset of to:ℱo⊆to.
a. quantity of annotations
we define the breadth of coverage of ℱo as the proportion of gene products from a given organism that are annotated to at least one term from the reference ontology .
cov=|gao||gΩ|
where gao={g|g∈gΩ,ℱo≠∅̸}.
we define the richness of ℱo as the proportion of to, the set of terms from , which is assigned to at least one gene product from ℱo.
we note:
rich=1|to||{t|t∈to,gpo≠∅̸}|
where gpo={g|g∈gΩ,t∈ℱo}.
given a set of annotated gene products gao, another statistic of interest is the number of annotations  with which a gene product is associated with respect to the ontology .
we note, ∀g∈gao
n=|ℱo|
b. diversity of annotations
to gain a more qualitative insight into fas, it is useful to know for each gene product, how diverse its annotations are.
we propose to measure the coherence of annotation sets, ∀g∈gao
coh=∑i∈fosim-{i})n,
with sim providing a measure of the semantic similarity between two annotation sets s <dig> and s <dig>  in a general sense, measuring the semantic similarity between two biological entities is equivalent to assessing the degree of functional relatedness between these entities by evaluating the similarity between their annotations.
further details on the semantic similarity measures implemented in aigo are given below in the paragraph inter-comparison of functional annotations.
likewise, to estimate the degree of heterogeneity of an entire fa, we define its compactness with respect to a given ontology as the mean semantic similarity between all its annotation sets, such that:
comp=∑g∈gaoavsim|gao|avsim=∑g′∈gao-{g}sim,ℱo)|gao|- <dig>
c. informativeness of annotations
not all fas are equally informative and the amount of useful information that they provide needs to be carefully appraised.
one possible way to assess the usefulness of a fa is to look at the relationship between the annotated terms in the reference ontology.
in go, the relationships between the terms has the form of a direct acyclic graph, where ancestral terms are less specific than their descendants.
for a given gene product, we define the specificity of its set of annotations as the mean number of ancestors over all annotated terms, ∀g∈gao
spec=1n∑i∈fo|ao|
where ao is the set containing a term i and all its ancestors in .
we also define the redundancy of an annotation set as the proportion of terms in the set that are ancestors of other terms in that set, ∀g∈gao
red=1n|{i|i,j∈ℱo,i∈ao}|
a common criticism of go structural metrics is that they assume a homogeneous difference in go term specificity across all edges of the go directed graph.
information content  based measures attempt to address this problem by providing metrics for the specificity of a term based on the frequency of its usage.
several studies have suggested that ic may provide a superior measure of semantic distance compared to graph structural metrics  <cit> , but ic is sensitive to annotation bias and changes over time in non-reference organisms  <cit> .
ic is based on shannon's information theoretic measure  <cit> , where:
ic=-log2p
and probability p of a term is the frequency of a term and of its ancestors in a fa.
p=∑t∈ao|gpo|∑j∈to|gpo|
as with most of the biomedical ontologies, go is dynamic and the structure and content evolve continuously, with new versions released at regular intervals.
almost all features of go can undergo change: the definition of terms can be revised and the relationship between terms can be deleted.
it is often the case that terms are made obsolete and replacement terms are proposed.
however, it is also possible that no appropriate alternatives can be found to replace an obsolete term and so it becomes de facto disconnected from the rest of the ontology.
in this type of situation, the annotations carry virtually no useful information.
we define the obsolescence of a functional annotation ℱo, with respect to a given ontology, as the proportion of obsolete annotations for which no alternative have been proposed.
let t˜o⊆to be the set of obsolete terms, then we have
obs=∑t∈{t|t∈t˜o,gpo≠0̸}|gpo|∑t∈{t|t∈to,gpo≠0̸}|gpo|
d. inter-comparison of functional annotations
it is possible to obtain a general comparison of several fas by contrasting the values they report for the nine metrics described above.
however, a deeper examination is sometimes required, for example, to know how many gene products are commonly annotated by two or more fas, what these gene products are, and how different/similar their annotation sets are within the fas in question.
in this section, we describe three metrics for the inter-comparison of fas.
the aigo library will generate and display venn diagrams to compare the coverage of fas.
when gene products are annotated by at least two fas, aigo can compute the semantic similarity between their sets of annotations.
this functionality is particularly helpful for identifying highly dissimilar annotation sets and therefore has the potential to detect errors in fas.
numerous metrics have been proposed to evaluate the semantic similarity between annotation sets.
in the aigo library, three semantic similarity metrics are implemented:  the resnik metric based on the most informative common ancestors of go terms  <cit> ,  a graph-based metric known as the czekanowski-dice similarity  and  another graph-based metric called gs <dig>  <cit>  which behaves similarly to the measure developed by wang et al.
in  <cit>  but is more computationally efficient since it can measure the similarity of a set of genes in linear time in the size of the set.
because of its greater efficiency, the gs <dig> metric has been used for all the semantic similarity calculations in this study.
finally, the last two metrics implemented in aigo to compare fas are the hierarchical precision and the hierarchical recall  <cit> .
these two metrics describe the accuracy of fas according to a reference fa, i.e.
a gold-standard, taking into account the hierarchical nature of the gene ontology.
the precision corresponds to the proportion of the annotations in a given fa that are also found in the gold-standard, whereas the recall corresponds to the proportion of the annotations in the gold-standard that are also found in a given fa.
more formally, we note for ℱo, a functional annotation, and go, a gold-standard, ∀g∈{g|g∈gΩ,ℱo≠0̸,go≠0̸}
hprecision=∑t∈fosim|ℱo|sim=maxt∈go|ao∩ao||ao|hrecall=∑s∈gorsim|go|sim=maxt∈fo|ao∩ao||ao|
implementation: the aigo library
the different metrics presented in the previous section have been implemented in a python library, named aigo, for the analysis and the inter-comparison of go functional annotations.
aigo is an open source library distributed under the gnu general public license v <dig> and source code, documentation and illustrative examples can be found here: http://code.google.com/p/aigo.
aigo is an object oriented python library implementing a collection of classes to load, manipulate, analyse and inter-compare fas.
the library can read annotation files in various formats such as "go annotation file  <dig> ", affymetrix annotation files  and generic mapping files.
additionally, aigo can read go files in xml obo format and gene set reference files in different formats, including the fasta format.
aigo can automatically create graphics showing the results of the comparison of multiple fas, including the distribution of the metrics and draw annotated directed acyclic graphs of go.
all the results can also be exported into simple text files or into microsoft excel files.
to assess the statistical significance of the results, aigo also provides classes to perform permutation tests by randomizing existing fas or by sampling fas directly from go.
a simple graphical user interface  allows access to most of the functionalities of the library in a very simple and efficient way.
it is worth noting that aigo has been designed so that new metrics can be easily added to the library.
once registered, the new metrics automatically appear in the user interface, and the results of their computation will be automatically plotted and exported.
most of the operations that aigo performs are not computationally intensive and therefore do not require much cpu time or memory.
for example, the first case study presented in the results section took exactly  <dig> minutes to run on a dual core cpu .
in practice, the computation of most of the statistics is very fast, with the exception of the those involving semantic similarity.
the gs <dig> metric is very scalable, since it is computable in linear time in the size of the set of gene products, however computing the coherence and the compactness statistics can still take several minutes.
another rather demanding operation is plotting the direct acyclic graph induced by a set of annotations: more precisely, the layout of the nodes can take one hour in the worst case.
RESULTS
example 1: inter-comparison of annotation pipelines
the unified framework implemented in the aigo library was used to analyse and compare three publicly available fas for the affymetrix genechip bovine genome array .
these annotations are provided by affymetrix   <cit>  and by two cross-species annotation pipelines: blast2go   <cit>  and arrayider   <cit> .
the set of gene products used as a reference in this study corresponds to the  <dig>  target sequences of the affymetrix bovine array.
for the three fas mentioned above, table  <dig> reports the values of nine statistics of interest computed using the aigo library.
we note that both affy and b2g provide a small number of obsolete terms for which no alternative exist.
we chose to remove these terms from affy and b2g before computing the other metrics.
the statistical properties of functional annotations in affymetrix, blast2go, arrayider generated by aigo for each aspect of the gene ontology.
analysis of functional annotations
aid has the greatest breadth of coverage, with a majority of target sequences  having at least one annotation from at least one of the three aspects of go, whereas b2g systematically reports the lowest coverage results.
unexpectedly, despite this smaller coverage, b2g contains more diverse go terms than the other fas, with richness  almost equal to 25% .
the compactness statistic  is similar across the three fas; this indicates that their annotation sets are distributed in a similar way across the ontology.
for the three fas, the annotated gene products are associated with  <dig> or  <dig> terms on average.
however, as illustrated in figure  <dig> for affy, b2g and aid, the distributions of the size of the annotation sets are strongly positively skewed.
consequently, while a large proportion of the annotation sets contain only one term, some other annotation sets are surprisingly large.
for the bp aspect of go, the largest set consists of  <dig>   <dig> and  <dig> terms for affy, b2g and aid respectively.
we note that for both affy and b2g, the largest set corresponds to the same probe-set: bt.
<dig> .s1_at.
this probe-set is associated with the "bcl <dig> b-cell cll/lymphoma 2" gene, a member of the large bcl <dig> family involved in the regulation of apoptosis.
there is a strong possibility that slightly different annotations coming from all the members of this family have been electronically inferred and transferred to the bt.
<dig> .s1_at probe-set, which would explain the size of its annotation set .
this very high proportion of sets containing only a single term has a clear impact on the coherence statistic  since all these singletons have a coherence value equal to one.
this situation is clearly visible in figure  <dig> where the distributions of the coherence values for "molecular function" annotation sets are plotted.
the three distributions for affy, b2g and aid are bimodal with a first peak between  <dig>  and  <dig>  and a second one around  <dig>  determined mostly by the number of singleton sets.
these distributions can be compared to the distribution obtained from a randomly sampled functional annotation where go terms in the original affy annotation sets are replaced by random terms, uniformly drawn from go.
as expected, we see that the original annotation sets from affy are much more coherent than the random ones, which confirms that go terms in affy annotation sets tend to be more functionally related than they would otherwise be if drawn at random.
being able to easily identify and display the annotation sets, where the sizes and/or coherence statistics highlight the presence of extreme values, is a useful feature of aigo that should help developers of functional annotation pipelines and/or end-users of fas to detect the presence of abnormalities.
the last three metrics reported in table  <dig> are related to the informativeness of the annotations.
both the specificity and the information content reveal that b2g annotations sets are more informative on average than the two other sets of fas.
we also note the large quantity of redundant annotations present in affy and aid annotations sets.
for these two fas, almost 30% of the annotations could potentially be removed from the annotation sets, since they correspond to go terms which are more generic than the existing terms already present in the same sets.
it is however important to note that the go terms contained in these annotation sets might have been assigned using different techniques with differing levels of confidence.
therefore, when an annotation to a go term has a stronger confidence than another annotation to a more specific term, it would still make sense to keep both of them together in an annotation set.
this may explain the high level of redundancy observed in affy and aid.
inter-comparison of functional annotations
a venn diagram comparing the breadth of coverage of the three fas for all three aspects of go is shown in figure  <dig>  altogether affy, b2g and aid provide annotations for  <dig> % of the set of gene products whilst a subset of only  <dig> % is annotated by all the three fas.
interestingly, we note that a subset of  <dig> % of the gene products is annotated by both affy and aid but not by b2g.
we observe as well that the total difference in coverage between affy and b2g or between aid and b2g can be almost entirely explained by this subset.
it is beyond the scope of this paper to investigate why these gene products are not annotated by b2g but a similar situation could be reported if, for example, affymetrix and arrayider pipelines were set up with a less stringent blast e-value than b2g or if the affymetrix and arrayider pipelines were querying a more up-to-date database than b2g, therefore containing more annotated gene products.
for the subset that is commonly annotated by the three fas, it is possible to perform some direct inter-comparisons.
for example, figure 5a) and 5b) display the distributions of the semantic similarity between the annotation sets of the gene products jointly annotated by a) affy and aid and b) affy and b2g.
in this example, the gs <dig> similarity measure  <cit>  has been used.
the annotation sets from affy and aid appear to be almost identical for the vast majority of the gene products annotated by both methods, with semantic similarity equals to one.
conversely when comparing affy and b2g, we see many more discrepancies between the annotation sets provided by both methods.
several reasons can be suggested to explain the differences between these annotation sets.
first of all, they might correspond to situations where gene products perform multiple functions, or are involved in several biological processes or are present in different parts of the cell.
different annotation strategies may lead to different but compatible annotations for a given gene.
for example, for the gene map2k <dig> in bos taurus, a sequence-based annotation strategy may recognize the mapkk <dig> kinase function and assign the appropriate mapkkk cascade process , whereas an approach that uses gene regulation based on microarray expression data might annotate the gene as responsive to stimulus .
however, very different annotation sets will often reveal annotation errors.
the aigo library was used to detect illustrative cases of suspiciously conflicting annotations in the cc aspect of go.
as an example, the probe-set bt.
<dig> .s1_at is annotated to two very different cc go terms: to "go: <dig> -nucleus" by affy and to "go: <dig> - extracellular region" by b2g.
after verification , the correct go term is undoubtedly given by b2g, and the annotation provided by affy is clearly an error.
example 2: comparison of evidence codes
the gene association files, which are made available for multiple species by the gene ontology project  <cit> , provide mappings of genes to go terms.
for each of these mappings, an evidence code indicates the source from which the association of a given gene with a particular go term was derived  <cit> .
only one of these evidence codes, namely "inferred from electronic annotation", identifies annotations that have been made with no manual intervention from a human curator.
in this section we use aigo to compare these electronic annotations to annotations assigned by a human curator, corresponding to the following evidence codes: "traceable author statement" , "non-traceable author statement" , "inferred from sequence or structural similarity"  and "inferred by curator" .
as previously shown  <cit> , measuring the accuracy of fas requires the definition of a reference fa, i.e.
a gold-standard.
here, we have approximated a gold-standard by identifying all the associations in all the species of the gene ontology project that are supported by at least two different experimental-evidence sources based on wet lab experiments.
these are represented in go using the evidence codes "inferred from mutant phenotype", "inferred from genetic interaction", inferred from physical interaction", "inferred from direct assay" and "inferred from expression pattern".
all together these associations form a reference fa, which we call exp <dig>  containing  <dig>  experimental annotations related to  <dig>  gene products from  <dig> different species.
in a similar way, we have filtered all the associations in all the species to create two other functional annotations: the first one, called iea, containing only associations corresponding to "inferred from electronic annotation" and the second one, called ahc, containing only associations assigned by a human curator .
approximately 40% of the genes products present in exp <dig> are also found in ahc while 89% are also in iea.
for these two subsets of exp <dig> gene products, it is therefore possible to compare the annotations in ahc and iea to the annotations in exp <dig> using the hierarchical precision  and the hierarchical recall  measures implemented in aigo.
for the three aspects of go, the average hprec and hrec together with the corresponding standard error of the mean are shown in table  <dig> and table  <dig> respectively.
average hierarchical precision  of ahc and iea and exp <dig> for the three aspects of go
average hierarchical recall  of ahc and iea and exp <dig> for the three aspects of go
two important factors influencing these statistics are: for hrec, the number of exp <dig> annotations to be retrieved and, for hprec, the number of annotations assigned by ahc and iea.
aigo reveals that the average number of annotations per gene product varies with the go aspect for the three fas with:  <dig> ,  <dig>  and  <dig>  annotations on average for exp <dig> ,  <dig> ,  <dig>  and  <dig>  annotations for ahc and  <dig> ,  <dig>  and  <dig>  annotations for iea.
the variation reported in the number of annotations for exp <dig> explains the trend observed in hrec which tends to be smaller for bp than mf and smaller for mf than cc, whereas the variations in ahc and iea explain the trend for hprec which tends to be smaller for bp than mf and smaller for mf than cc.
the relatively low hprec value reported for bp by ahc in table  <dig> corresponds to an outstandingly high number of annotations predicted for this category: nearly five annotations per gene product on average, which is more than twice the numbers reported for mf and cc.
we note that even though we are confident that exp <dig> contains only correct annotations, it is improbable that all the annotations that could be assigned to the gene products are present in exp <dig>  hence, the results reported for the hprec metric should be interpreted with care, as incompleteness in the gold-standard may introduce bias into the comparison between fas.
as for the hrec metric, the overall results for ahc and iea are almost identical, except for mf where iea performs better.
to explore this important difference between iea and ahc, we looked at the various statistics produced by aigo for mf.
while the specificity  is comparable for ahc and iea , it is apparent that fewer go terms are used in ahc, compared to iea and exp <dig> .
the distribution of hrec values for ahc displays a peak at  <dig> , which is not present for iea .
we have manually inspected these cases and found that generally the annotations supported by ahc are also supported by iea, and are therefore potentially correct.
they also tend to correspond to terms from different parts of go than exp <dig> as a whole.
we do not further explore this observation in this work but these results reflect the fact that certain parts of the ontology are favoured during the curation process.
indeed, manual associations between go terms and gene products are often produced as part of manual annotation projects.
these curation initiatives tend to have very specific targets such as the annotation of genes related to precise biological functions  <cit>  or to particular diseases  <cit> .
example 3: monitoring the evolution of functional annotations
the aigo library can also be used to monitor the evolution of fas across time.
as an illustration, aigo was used to study  <dig> fas for a rice genome microarray  released by affymetrix  <cit>  between  <dig> may  <dig> and  <dig> november  <dig>  and numbered from  <dig> to  <dig>  the set of reference gene products correspond to the  <dig>  target sequences of the rice array.
it is clear in figure  <dig>  that on average, both the number of annotated gene products  and the number of assigned go terms  increase steadily during the studied period of time.
however, it is also evident that the coverage can decrease between two releases as was the case between release  <dig> and  <dig> for the cc aspect of go.
we note that a major change, both in terms of coverage and richness, occurred between releases  <dig> and  <dig>  affymetrix does not provide any information that would explain this phenomenon, and it could therefore be due to an important step in the evolution in affymetrix annotation pipeline and/or to a major change in the data sources used to infer the annotations.
interestingly, from release number  <dig> and onwards, the obsolete annotations have been systematically removed from the fas, which would suggest that a filtering step has been added to the original affymetrix annotation pipeline.
affymetrix has issued eleven releases of the rice array annotations since  <dig> may  <dig>  during this time, 5% of the probe-sets have been annotated in every release and it is therefore possible to study the evolution of the annotation sets for the corresponding probe-sets.
in figure  <dig>  aigo was used to identify and plot two very dissimilar annotation sets found in release  <dig>  and release  <dig> for the probe-set os.
<dig> .s1_a_at.
in this example, the sets have no annotation in common and when considering the structure of the ontology, they appear to be far apart in the go hierarchy, corresponding to a very low gs <dig> similarity .
if we manually examine the two annotation sets from a biological perspective, they also appear to be functionally incompatible.
in fact, there is only one unique example  in the entire gene ontology annotation database where the two go terms "structural constituent of ribosome" and "phospholipase c activity" have been used together to describe a single protein.
therefore, we can assume that the difference between the two annotation sets does not correspond to a natural evolution of the functional annotation over time, for example, due to an improvement in the description of the function of the protein associated with os.
<dig> .s1_a_at, but rather to an important correction of an original error made in the annotation process.
to gain further insight into the annotation of os.
<dig> .s1_a_at, the complementary information provided by affymetrix was analysed in more detail.
it appears that two different gene loci in oryza sativa  were used to infer this annotation.
after comparing the target sequence os.
<dig> .s1_a_at, with the sequences of the proteins encoded by these two different genes, we have been able to confirm that the annotations from release  <dig> are correct .
discussion
biologists and bioinformaticians, as end-users of functional annotations , are generally confronted with a choice between several available fas for the organisms they are studying.
often the most important thing to consider when deciding between alternative fas is to look at the accuracy of the annotations: i.e.
to what extent do the annotations of some proteins really correspond to what has been experimentally established about the function of these proteins?
in practice this evaluation is very challenging, given that for the vast majority of the organisms there is no existing gold-standard to evaluate precision and recall against.
furthermore, even in the rare cases where an organism has been extensively studied and the functions of most of its proteins experimentally assessed and annotated in public databases, there is no guarantee that all the biological roles and functions of these proteins are known and translated into functional annotations.
this situation is problematic when evaluating the precision of fas, but is not unique to the study of protein function since the lack of negative gold-standards and its impact on the evaluation of automatic inference methods has been reported in other domains of bioinformatics  <cit> .
an alternative approach to decide between fas would be to directly compare the functional annotation pipelines that produced these fas.
however, these pipelines differ widely in their inference algorithms, confidence thresholds, and data sources for reasoning and this makes the comparison of the relative merits of each approach extremely complex.
the more pragmatic approach developed in this paper is to analyse and inter-compare the products of these functional annotation pipelines, that is the fas themselves.
fas are complex multi-dimensional objects that cannot be summarized in a straightforward way.
hence, we defined the aigo framework to describe various features of fas that we consider being important for fas end-users and more generally for the community of biologists and bioinformaticians interested in gene or protein functions.
so far a set of twelve metrics, nine for the analysis and three for the inter-comparison of fas, has been defined and implemented in aigo.
each metric covers only one particular aspect of the fas but, as demonstrated in this paper, when computed all together and interpreted in the aigo framework, these metrics provide a global view of the fas and help to make informed decisions concerning them.
alternatively, after having inspected a set of fas in aigo, one might want to combine them instead of retaining only one.
therefore, the library also provides functionalities to compute the union or the intersection of fas.
the union of fas is, for example, particularly meaningful in the case where each fa covers a different aspect of go, e.g.
to combine a fa containing only go molecular function terms converted from enzyme commission numbers with a fa containing only go cellular component terms converted from subcellular location predictions.
conversely, the intersection of fas is more appropriate to increase the reliability of functional annotations.
for example, when an ensemble of fas provides annotations about the same organism but using different types of evidence  selecting the associations between gene products and go terms that are present in all these fas is a straightforward way to gain confidence in the annotations.
as well as fa end-users, we believe that aigo can be beneficial to other categories of members of the scientific community.
an important category of members corresponds to fa producers, i.e.
people developing new functional annotation pipelines, who might want to compare their pipeline outputs to existing fas, or measure the influence of a given pipeline parameter on their results.
another category is the providers of fas who are responsible for releasing fas to the community, and who need periodically to monitor the content of automatically produced fas, for example, to detect spurious annotations or abnormally large annotation sets.
a further category corresponds to researchers working on new metrics to describe fas.
the aigo framework has been designed to be flexible so that new metrics on go functional annotations can be easily added.
by making this project open source and collaborative, we would like to encourage researchers to implement new metrics in aigo and contribute to its development.
CONCLUSIONS
the work presented in this paper is a first step towards the development of a unified framework for the analysis and the inter-comparison of go functional annotations.
the utility of the framework is demonstrated on three case studies.
in the first, publicly available functional annotations are compared, and their differences highlighted, for example, in terms of the number of gene products annotated, or the number and specificity of the go terms employed.
in the second case study, the quality of two functional annotations, one obtained by computational methods and one corresponding to a manual curation process, is assessed using an approximated gold-standard.
in the last example, we show how aigo framework can be used to monitor the evolution of functional annotations by comparing different releases over time, in order to detect major variations, or to identify potentially incorrect annotations.
authors' contributions
mdp initiated the work, developed the aigo library, designed the tests, carried out all the functional annotation studies and drafted the manuscript.
mh was involved in the design of the experiments, contributed to the development of the library and participated in redaction of the manuscript.
al created the gold-standard dataset for example  <dig>  dzh, cjr and ms contributed to the coordination of the studies and helped to draft the manuscript.
all authors read and approved the final manuscript.
supplementary material
additional file 1
reference of affymetrix genechip genome arrays.
click here for file
additional file 2
annotations of bt.
<dig> .s1_at.
click here for file
additional file 3
annotations of bt.
<dig> .s1_at.
click here for file
additional file 4
rice protein with suspicious annotations.
click here for file
additional file 5
annotations of os.
<dig> .s1_a_at.
click here for file
acknowledgements
the authors gratefully acknowledge funding from the uk biotechnology and biological sciences research council .
al was supported by phd studentship bbs/s/e/2006/ <dig>  ms and cjr were partially supported by the ondex bbsrc sabr grant bb/f006039/ <dig>  mmh gratefully acknowledges support from the lawes agricultural trust.
rothamsted research receives grant in aid from the bbsrc which supported md-p, ms, cjr, sjp and dzh.

 given the advent of high-throughput sequencing technologies and the resulting data explosion, there is an urgent requirement to provide electronically inferred functional annotations for proteins whose biological functions have not yet been determined by experimental investigation. this has given rise to a large number of annotation inference pipelines which are used routinely to produce new annotations that are stored in a variety of publicly accessible databases. therefore, when looking for fas related to a particular organism, it is commonplace to find several resources each providing annotations that have been predicted using different pipelines, at different times and under different conditions. consequently, end-users of fas have to decide between multiple resources. however, since there is no single accessible and objective way to compare these fas, these decisions are in many cases made arbitrarily, whereas different biological problems may require consideration of annotations from different points of view. where pipeline outputs reference a controlled vocabulary or ontology, relative comparisons of these outputs can be made by using a reference protein-set as a common input. many pipelines endeavour to annotate gene products to the three aspects of the gene ontology: molecular function, biological process, and cellular component. for each go aspect, a given gene product can therefore be annotated with a set of go terms. go provides not only a unified and shared controlled vocabulary for annotation but a structured representation of knowledge. the power of this annotation system over a flat controlled vocabulary is twofold. firstly, it can be used to evaluate the functional distance between genes by quantifying the difference between their annotation sets based on the semantics codified in the structure. secondly, it defines a hierarchy between the go terms that leads to annotations being more or less specific, i.e. more or less biologically informative. the aim of this study is to define and implement an ensemble of measures that provide both a quantitative and a qualitative description of go fas and also allows a comprehensive inter-comparison between them. so far, a plethora of such measures have been proposed but no attempts have yet been made to define and implement a unified framework where fas can be compared in a systematic way. a notable exception is the blast2go suite where several metrics describing fas are implemented together, for example to measure the specificity of annotations or to automatically compare groups of go terms. however in blast2go, these metrics are only used to measure specific aspects of blast2go performance and furthermore, the blast2go suite is not designed to compare fas issued by other pipelines. in most studies, the different properties of fas are implemented and studied independently e.g.. in addition to facilitating the comparison of annotation sets produced by various annotation pipelines, having a common unified framework that implements these metrics would assist in the evaluation of new metrics, the exploration of the influence of parameters in annotation pipelines and the monitoring of annotations over time. in this paper, we propose a set of twelve metrics and implement them in a python open source library named aigo for the analysis and the inter-comparison of go functional annotations. we demonstrate the use of the aigo library for evaluating and comparing the output of annotation pipelines using three illustrative examples. methods in this section, we define a set of twelve metrics, each of them describing particular features of fas, see table the different metrics computed by aigo allow describing particular features of functional annotations. definitions of metrics for any given organism Ω, we call functional annotation, noted ℱ, the mapping between a set of gene products gΩ from this organism and some functional data. in practice, a set of gene products from an organism can be defined in various ways, ranging from a comprehensive list of proteins to a list of ests assembled into potentially incomplete or inaccurate unigenes, see. a common scenario is that the set of gene products corresponds to some defined reference set: for example the list of target sequences from a microarray. the functional data are usually provided as annotation sets containing terms from controlled vocabularies or reference ontologies. we note a given ontology, defined as a direct acyclic graph connecting a set of terms to. in this study, can be any of the three aspects of go. more formally, a functional annotation of a set of gene products gΩ over a given ontology, is a map ℱo:gΩ→2o that associates with each gene product g ∈ gΩ a subset of to:ℱo⊆to. a. quantity of annotations we define the breadth of coverage of ℱo as the proportion of gene products from a given organism that are annotated to at least one term from the reference ontology. cov=|gao||gΩ| where gao={g|g∈gΩ,ℱo≠∅̸}. we define the richness of ℱo as the proportion of to, the set of terms from, which is assigned to at least one gene product from ℱo. we note: rich=1|to||{t|t∈to,gpo≠∅̸}| where gpo={g|g∈gΩ,t∈ℱo}. given a set of annotated gene products gao, another statistic of interest is the number of annotations with which a gene product is associated with respect to the ontology. we note, ∀g∈gao n=|ℱo| b. diversity of annotations to gain a more qualitative insight into fas, it is useful to know for each gene product, how diverse its annotations are. we propose to measure the coherence of annotation sets, ∀g∈gao coh=∑i∈fosim-{i})n, with sim providing a measure of the semantic similarity between two annotation sets s and s in a general sense, measuring the semantic similarity between two biological entities is equivalent to assessing the degree of functional relatedness between these entities by evaluating the similarity between their annotations. further details on the semantic similarity measures implemented in aigo are given below in the paragraph inter-comparison of functional annotations. likewise, to estimate the degree of heterogeneity of an entire fa, we define its compactness with respect to a given ontology as the mean semantic similarity between all its annotation sets, such that: comp=∑g∈gaoavsim|gao|avsim=∑g′∈gao-{g}sim,ℱo)|gao|- c. informativeness of annotations not all fas are equally informative and the amount of useful information that they provide needs to be carefully appraised. one possible way to assess the usefulness of a fa is to look at the relationship between the annotated terms in the reference ontology. in go, the relationships between the terms has the form of a direct acyclic graph, where ancestral terms are less specific than their descendants. for a given gene product, we define the specificity of its set of annotations as the mean number of ancestors over all annotated terms, ∀g∈gao spec=1n∑i∈fo|ao| where ao is the set containing a term i and all its ancestors in. we also define the redundancy of an annotation set as the proportion of terms in the set that are ancestors of other terms in that set, ∀g∈gao red=1n|{i|i,j∈ℱo,i∈ao}| a common criticism of go structural metrics is that they assume a homogeneous difference in go term specificity across all edges of the go directed graph. information content based measures attempt to address this problem by providing metrics for the specificity of a term based on the frequency of its usage. several studies have suggested that ic may provide a superior measure of semantic distance compared to graph structural metrics, but ic is sensitive to annotation bias and changes over time in non-reference organisms. ic is based on shannon's information theoretic measure, where: ic=-log2p and probability p of a term is the frequency of a term and of its ancestors in a fa. p=∑t∈ao|gpo|∑j∈to|gpo| as with most of the biomedical ontologies, go is dynamic and the structure and content evolve continuously, with new versions released at regular intervals. almost all features of go can undergo change: the definition of terms can be revised and the relationship between terms can be deleted. it is often the case that terms are made obsolete and replacement terms are proposed. however, it is also possible that no appropriate alternatives can be found to replace an obsolete term and so it becomes de facto disconnected from the rest of the ontology. in this type of situation, the annotations carry virtually no useful information. we define the obsolescence of a functional annotation ℱo, with respect to a given ontology, as the proportion of obsolete annotations for which no alternative have been proposed. let t˜o⊆to be the set of obsolete terms, then we have obs=∑t∈{t|t∈t˜o,gpo≠0̸}|gpo|∑t∈{t|t∈to,gpo≠0̸}|gpo| d. inter-comparison of functional annotations it is possible to obtain a general comparison of several fas by contrasting the values they report for the nine metrics described above. however, a deeper examination is sometimes required, for example, to know how many gene products are commonly annotated by two or more fas, what these gene products are, and how different/similar their annotation sets are within the fas in question. in this section, we describe three metrics for the inter-comparison of fas. the aigo library will generate and display venn diagrams to compare the coverage of fas. when gene products are annotated by at least two fas, aigo can compute the semantic similarity between their sets of annotations. this functionality is particularly helpful for identifying highly dissimilar annotation sets and therefore has the potential to detect errors in fas. numerous metrics have been proposed to evaluate the semantic similarity between annotation sets. in the aigo library, three semantic similarity metrics are implemented: the resnik metric based on the most informative common ancestors of go terms, a graph-based metric known as the czekanowski-dice similarity and another graph-based metric called gs which behaves similarly to the measure developed by wang et al. in but is more computationally efficient since it can measure the similarity of a set of genes in linear time in the size of the set. because of its greater efficiency, the gs metric has been used for all the semantic similarity calculations in this study. finally, the last two metrics implemented in aigo to compare fas are the hierarchical precision and the hierarchical recall. these two metrics describe the accuracy of fas according to a reference fa, i.e. a gold-standard, taking into account the hierarchical nature of the gene ontology. the precision corresponds to the proportion of the annotations in a given fa that are also found in the gold-standard, whereas the recall corresponds to the proportion of the annotations in the gold-standard that are also found in a given fa. more formally, we note for ℱo, a functional annotation, and go, a gold-standard, ∀g∈{g|g∈gΩ,ℱo≠0̸,go≠0̸} hprecision=∑t∈fosim|ℱo|sim=maxt∈go|ao∩ao||ao|hrecall=∑s∈gorsim|go|sim=maxt∈fo|ao∩ao||ao| implementation: the aigo library the different metrics presented in the previous section have been implemented in a python library, named aigo, for the analysis and the inter-comparison of go functional annotations. aigo is an open source library distributed under the gnu general public license v and source code, documentation and illustrative examples can be found here: http://code.google.com/p/aigo. aigo is an object oriented python library implementing a collection of classes to load, manipulate, analyse and inter-compare fas. the library can read annotation files in various formats such as "go annotation file ", affymetrix annotation files and generic mapping files. additionally, aigo can read go files in xml obo format and gene set reference files in different formats, including the fasta format. aigo can automatically create graphics showing the results of the comparison of multiple fas, including the distribution of the metrics and draw annotated directed acyclic graphs of go. all the results can also be exported into simple text files or into microsoft excel files. to assess the statistical significance of the results, aigo also provides classes to perform permutation tests by randomizing existing fas or by sampling fas directly from go. a simple graphical user interface allows access to most of the functionalities of the library in a very simple and efficient way. it is worth noting that aigo has been designed so that new metrics can be easily added to the library. once registered, the new metrics automatically appear in the user interface, and the results of their computation will be automatically plotted and exported. most of the operations that aigo performs are not computationally intensive and therefore do not require much cpu time or memory. for example, the first case study presented in the results section took exactly minutes to run on a dual core cpu. in practice, the computation of most of the statistics is very fast, with the exception of the those involving semantic similarity. the gs metric is very scalable, since it is computable in linear time in the size of the set of gene products, however computing the coherence and the compactness statistics can still take several minutes. another rather demanding operation is plotting the direct acyclic graph induced by a set of annotations: more precisely, the layout of the nodes can take one hour in the worst case.  example 1: inter-comparison of annotation pipelines the unified framework implemented in the aigo library was used to analyse and compare three publicly available fas for the affymetrix genechip bovine genome array. these annotations are provided by affymetrix and by two cross-species annotation pipelines: blast2go and arrayider. the set of gene products used as a reference in this study corresponds to the target sequences of the affymetrix bovine array. for the three fas mentioned above, table reports the values of nine statistics of interest computed using the aigo library. we note that both affy and b2g provide a small number of obsolete terms for which no alternative exist. we chose to remove these terms from affy and b2g before computing the other metrics. the statistical properties of functional annotations in affymetrix, blast2go, arrayider generated by aigo for each aspect of the gene ontology. analysis of functional annotations aid has the greatest breadth of coverage, with a majority of target sequences having at least one annotation from at least one of the three aspects of go, whereas b2g systematically reports the lowest coverage results. unexpectedly, despite this smaller coverage, b2g contains more diverse go terms than the other fas, with richness almost equal to 25%. the compactness statistic is similar across the three fas; this indicates that their annotation sets are distributed in a similar way across the ontology. for the three fas, the annotated gene products are associated with or terms on average. however, as illustrated in figure for affy, b2g and aid, the distributions of the size of the annotation sets are strongly positively skewed. consequently, while a large proportion of the annotation sets contain only one term, some other annotation sets are surprisingly large. for the bp aspect of go, the largest set consists of and terms for affy, b2g and aid respectively. we note that for both affy and b2g, the largest set corresponds to the same probe-set: bt..s1_at. this probe-set is associated with the "bcl b-cell cll/lymphoma 2" gene, a member of the large bcl family involved in the regulation of apoptosis. there is a strong possibility that slightly different annotations coming from all the members of this family have been electronically inferred and transferred to the bt..s1_at probe-set, which would explain the size of its annotation set. this very high proportion of sets containing only a single term has a clear impact on the coherence statistic since all these singletons have a coherence value equal to one. this situation is clearly visible in figure where the distributions of the coherence values for "molecular function" annotation sets are plotted. the three distributions for affy, b2g and aid are bimodal with a first peak between and and a second one around determined mostly by the number of singleton sets. these distributions can be compared to the distribution obtained from a randomly sampled functional annotation where go terms in the original affy annotation sets are replaced by random terms, uniformly drawn from go. as expected, we see that the original annotation sets from affy are much more coherent than the random ones, which confirms that go terms in affy annotation sets tend to be more functionally related than they would otherwise be if drawn at random. being able to easily identify and display the annotation sets, where the sizes and/or coherence statistics highlight the presence of extreme values, is a useful feature of aigo that should help developers of functional annotation pipelines and/or end-users of fas to detect the presence of abnormalities. the last three metrics reported in table are related to the informativeness of the annotations. both the specificity and the information content reveal that b2g annotations sets are more informative on average than the two other sets of fas. we also note the large quantity of redundant annotations present in affy and aid annotations sets. for these two fas, almost 30% of the annotations could potentially be removed from the annotation sets, since they correspond to go terms which are more generic than the existing terms already present in the same sets. it is however important to note that the go terms contained in these annotation sets might have been assigned using different techniques with differing levels of confidence. therefore, when an annotation to a go term has a stronger confidence than another annotation to a more specific term, it would still make sense to keep both of them together in an annotation set. this may explain the high level of redundancy observed in affy and aid. inter-comparison of functional annotations a venn diagram comparing the breadth of coverage of the three fas for all three aspects of go is shown in figure altogether affy, b2g and aid provide annotations for % of the set of gene products whilst a subset of only % is annotated by all the three fas. interestingly, we note that a subset of % of the gene products is annotated by both affy and aid but not by b2g. we observe as well that the total difference in coverage between affy and b2g or between aid and b2g can be almost entirely explained by this subset. it is beyond the scope of this paper to investigate why these gene products are not annotated by b2g but a similar situation could be reported if, for example, affymetrix and arrayider pipelines were set up with a less stringent blast e-value than b2g or if the affymetrix and arrayider pipelines were querying a more up-to-date database than b2g, therefore containing more annotated gene products. for the subset that is commonly annotated by the three fas, it is possible to perform some direct inter-comparisons. for example, figure 5a) and 5b) display the distributions of the semantic similarity between the annotation sets of the gene products jointly annotated by a) affy and aid and b) affy and b2g. in this example, the gs similarity measure has been used. the annotation sets from affy and aid appear to be almost identical for the vast majority of the gene products annotated by both methods, with semantic similarity equals to one. conversely when comparing affy and b2g, we see many more discrepancies between the annotation sets provided by both methods. several reasons can be suggested to explain the differences between these annotation sets. first of all, they might correspond to situations where gene products perform multiple functions, or are involved in several biological processes or are present in different parts of the cell. different annotation strategies may lead to different but compatible annotations for a given gene. for example, for the gene map2k in bos taurus, a sequence-based annotation strategy may recognize the mapkk kinase function and assign the appropriate mapkkk cascade process, whereas an approach that uses gene regulation based on microarray expression data might annotate the gene as responsive to stimulus. however, very different annotation sets will often reveal annotation errors. the aigo library was used to detect illustrative cases of suspiciously conflicting annotations in the cc aspect of go. as an example, the probe-set bt..s1_at is annotated to two very different cc go terms: to "go: -nucleus" by affy and to "go: - extracellular region" by b2g. after verification, the correct go term is undoubtedly given by b2g, and the annotation provided by affy is clearly an error. example 2: comparison of evidence codes the gene association files, which are made available for multiple species by the gene ontology project, provide mappings of genes to go terms. for each of these mappings, an evidence code indicates the source from which the association of a given gene with a particular go term was derived. only one of these evidence codes, namely "inferred from electronic annotation", identifies annotations that have been made with no manual intervention from a human curator. in this section we use aigo to compare these electronic annotations to annotations assigned by a human curator, corresponding to the following evidence codes: "traceable author statement", "non-traceable author statement", "inferred from sequence or structural similarity" and "inferred by curator". as previously shown, measuring the accuracy of fas requires the definition of a reference fa, i.e. a gold-standard. here, we have approximated a gold-standard by identifying all the associations in all the species of the gene ontology project that are supported by at least two different experimental-evidence sources based on wet lab experiments. these are represented in go using the evidence codes "inferred from mutant phenotype", "inferred from genetic interaction", inferred from physical interaction", "inferred from direct assay" and "inferred from expression pattern". all together these associations form a reference fa, which we call exp containing experimental annotations related to gene products from different species. in a similar way, we have filtered all the associations in all the species to create two other functional annotations: the first one, called iea, containing only associations corresponding to "inferred from electronic annotation" and the second one, called ahc, containing only associations assigned by a human curator. approximately 40% of the genes products present in exp are also found in ahc while 89% are also in iea. for these two subsets of exp gene products, it is therefore possible to compare the annotations in ahc and iea to the annotations in exp using the hierarchical precision and the hierarchical recall measures implemented in aigo. for the three aspects of go, the average hprec and hrec together with the corresponding standard error of the mean are shown in table and table respectively. average hierarchical precision of ahc and iea and exp for the three aspects of go average hierarchical recall of ahc and iea and exp for the three aspects of go two important factors influencing these statistics are: for hrec, the number of exp annotations to be retrieved and, for hprec, the number of annotations assigned by ahc and iea. aigo reveals that the average number of annotations per gene product varies with the go aspect for the three fas with:, and annotations on average for exp,, and annotations for ahc and, and annotations for iea. the variation reported in the number of annotations for exp explains the trend observed in hrec which tends to be smaller for bp than mf and smaller for mf than cc, whereas the variations in ahc and iea explain the trend for hprec which tends to be smaller for bp than mf and smaller for mf than cc. the relatively low hprec value reported for bp by ahc in table corresponds to an outstandingly high number of annotations predicted for this category: nearly five annotations per gene product on average, which is more than twice the numbers reported for mf and cc. we note that even though we are confident that exp contains only correct annotations, it is improbable that all the annotations that could be assigned to the gene products are present in exp hence, the results reported for the hprec metric should be interpreted with care, as incompleteness in the gold-standard may introduce bias into the comparison between fas. as for the hrec metric, the overall results for ahc and iea are almost identical, except for mf where iea performs better. to explore this important difference between iea and ahc, we looked at the various statistics produced by aigo for mf. while the specificity is comparable for ahc and iea, it is apparent that fewer go terms are used in ahc, compared to iea and exp. the distribution of hrec values for ahc displays a peak at, which is not present for iea. we have manually inspected these cases and found that generally the annotations supported by ahc are also supported by iea, and are therefore potentially correct. they also tend to correspond to terms from different parts of go than exp as a whole. we do not further explore this observation in this work but these results reflect the fact that certain parts of the ontology are favoured during the curation process. indeed, manual associations between go terms and gene products are often produced as part of manual annotation projects. these curation initiatives tend to have very specific targets such as the annotation of genes related to precise biological functions or to particular diseases. example 3: monitoring the evolution of functional annotations the aigo library can also be used to monitor the evolution of fas across time. as an illustration, aigo was used to study fas for a rice genome microarray released by affymetrix between may and november and numbered from to the set of reference gene products correspond to the target sequences of the rice array. it is clear in figure that on average, both the number of annotated gene products and the number of assigned go terms increase steadily during the studied period of time. however, it is also evident that the coverage can decrease between two releases as was the case between release and for the cc aspect of go. we note that a major change, both in terms of coverage and richness, occurred between releases and affymetrix does not provide any information that would explain this phenomenon, and it could therefore be due to an important step in the evolution in affymetrix annotation pipeline and/or to a major change in the data sources used to infer the annotations. interestingly, from release number and onwards, the obsolete annotations have been systematically removed from the fas, which would suggest that a filtering step has been added to the original affymetrix annotation pipeline. affymetrix has issued eleven releases of the rice array annotations since may during this time, 5% of the probe-sets have been annotated in every release and it is therefore possible to study the evolution of the annotation sets for the corresponding probe-sets. in figure aigo was used to identify and plot two very dissimilar annotation sets found in release and release for the probe-set os..s1_a_at. in this example, the sets have no annotation in common and when considering the structure of the ontology, they appear to be far apart in the go hierarchy, corresponding to a very low gs similarity. if we manually examine the two annotation sets from a biological perspective, they also appear to be functionally incompatible. in fact, there is only one unique example in the entire gene ontology annotation database where the two go terms "structural constituent of ribosome" and "phospholipase c activity" have been used together to describe a single protein. therefore, we can assume that the difference between the two annotation sets does not correspond to a natural evolution of the functional annotation over time, for example, due to an improvement in the description of the function of the protein associated with os..s1_a_at, but rather to an important correction of an original error made in the annotation process. to gain further insight into the annotation of os..s1_a_at, the complementary information provided by affymetrix was analysed in more detail. it appears that two different gene loci in oryza sativa were used to infer this annotation. after comparing the target sequence os..s1_a_at, with the sequences of the proteins encoded by these two different genes, we have been able to confirm that the annotations from release are correct. discussion biologists and bioinformaticians, as end-users of functional annotations, are generally confronted with a choice between several available fas for the organisms they are studying. often the most important thing to consider when deciding between alternative fas is to look at the accuracy of the annotations: i.e. to what extent do the annotations of some proteins really correspond to what has been experimentally established about the function of these proteins? in practice this evaluation is very challenging, given that for the vast majority of the organisms there is no existing gold-standard to evaluate precision and recall against. furthermore, even in the rare cases where an organism has been extensively studied and the functions of most of its proteins experimentally assessed and annotated in public databases, there is no guarantee that all the biological roles and functions of these proteins are known and translated into functional annotations. this situation is problematic when evaluating the precision of fas, but is not unique to the study of protein function since the lack of negative gold-standards and its impact on the evaluation of automatic inference methods has been reported in other domains of bioinformatics. an alternative approach to decide between fas would be to directly compare the functional annotation pipelines that produced these fas. however, these pipelines differ widely in their inference algorithms, confidence thresholds, and data sources for reasoning and this makes the comparison of the relative merits of each approach extremely complex. the more pragmatic approach developed in this paper is to analyse and inter-compare the products of these functional annotation pipelines, that is the fas themselves. fas are complex multi-dimensional objects that cannot be summarized in a straightforward way. hence, we defined the aigo framework to describe various features of fas that we consider being important for fas end-users and more generally for the community of biologists and bioinformaticians interested in gene or protein functions. so far a set of twelve metrics, nine for the analysis and three for the inter-comparison of fas, has been defined and implemented in aigo. each metric covers only one particular aspect of the fas but, as demonstrated in this paper, when computed all together and interpreted in the aigo framework, these metrics provide a global view of the fas and help to make informed decisions concerning them. alternatively, after having inspected a set of fas in aigo, one might want to combine them instead of retaining only one. therefore, the library also provides functionalities to compute the union or the intersection of fas. the union of fas is, for example, particularly meaningful in the case where each fa covers a different aspect of go, e.g. to combine a fa containing only go molecular function terms converted from enzyme commission numbers with a fa containing only go cellular component terms converted from subcellular location predictions. conversely, the intersection of fas is more appropriate to increase the reliability of functional annotations. for example, when an ensemble of fas provides annotations about the same organism but using different types of evidence selecting the associations between gene products and go terms that are present in all these fas is a straightforward way to gain confidence in the annotations. as well as fa end-users, we believe that aigo can be beneficial to other categories of members of the scientific community. an important category of members corresponds to fa producers, i.e. people developing new functional annotation pipelines, who might want to compare their pipeline outputs to existing fas, or measure the influence of a given pipeline parameter on their results. another category is the providers of fas who are responsible for releasing fas to the community, and who need periodically to monitor the content of automatically produced fas, for example, to detect spurious annotations or abnormally large annotation sets. a further category corresponds to researchers working on new metrics to describe fas. the aigo framework has been designed to be flexible so that new metrics on go functional annotations can be easily added. by making this project open source and collaborative, we would like to encourage researchers to implement new metrics in aigo and contribute to its development.  the work presented in this paper is a first step towards the development of a unified framework for the analysis and the inter-comparison of go functional annotations. the utility of the framework is demonstrated on three case studies. in the first, publicly available functional annotations are compared, and their differences highlighted, for example, in terms of the number of gene products annotated, or the number and specificity of the go terms employed. in the second case study, the quality of two functional annotations, one obtained by computational methods and one corresponding to a manual curation process, is assessed using an approximated gold-standard. in the last example, we show how aigo framework can be used to monitor the evolution of functional annotations by comparing different releases over time, in order to detect major variations, or to identify potentially incorrect annotations. authors' contributions mdp initiated the work, developed the aigo library, designed the tests, carried out all the functional annotation studies and drafted the manuscript. mh was involved in the design of the experiments, contributed to the development of the library and participated in redaction of the manuscript. al created the gold-standard dataset for example dzh, cjr and ms contributed to the coordination of the studies and helped to draft the manuscript. all authors read and approved the final manuscript. supplementary material additional file 1 reference of affymetrix genechip genome arrays. click here for file additional file 2 annotations of bt..s1_at. click here for file additional file 3 annotations of bt..s1_at. click here for file additional file 4 rice protein with suspicious annotations. click here for file additional file 5 annotations of os..s1_a_at. click here for file acknowledgements the authors gratefully acknowledge funding from the uk biotechnology and biological sciences research council. al was supported by phd studentship bbs/s/e/2006/ ms and cjr were partially supported by the ondex bbsrc sabr grant bb/f006039/ mmh gratefully acknowledges support from the lawes agricultural trust. rothamsted research receives grant in aid from the bbsrc which supported md-p, ms, cjr, sjp and dzh.
in this paper, we present a pragmatic approach to the study of functional annotations. an ensemble of metrics, describing various aspects of functional annotations, is defined and implemented in a unified framework, which facilitates their systematic analysis and inter-comparison. the use of this framework is demonstrated on three illustrative examples: analysing the outputs of state-of-the-art inference pipelines, comparing electronic versus manual annotation methods, and monitoring the evolution of publicly available functional annotations. the aigo library also provides functionalities to easily load, analyse, manipulate and compare functional annotations and also to plot and export the results of the analysis in various formats. this work is a step toward developing a unified framework for the systematic study of go functional annotations. this framework has been designed so that new metrics on go functional annotations can be added in a very straightforward way. pipelines that infer functional annotation are now routinely used to produce new annotations at a genome scale and for a broad variety of species. the framework is part of the aigo library for the analysis and the inter-comparison of the products of gene ontology annotation pipelines. these pipelines differ widely in their inference algorithms, confidence thresholds and data sources for reasoning. this heterogeneity makes a comparison of the relative merits of each approach extremely complex. in response to the rapid growth of available genome sequences, efforts have been made to develop automatic inference methods to functionally characterize them.
 in response to the rapid growth of available genome sequences, efforts have been made to develop automatic inference methods to functionally characterize them. pipelines that infer functional annotation are now routinely used to produce new annotations at a genome scale and for a broad variety of species. these pipelines differ widely in their inference algorithms, confidence thresholds and data sources for reasoning. this heterogeneity makes a comparison of the relative merits of each approach extremely complex. the evaluation of the quality of the resultant annotations is also challenging given there is often no existing gold-standard against which to evaluate precision and recall.  in this paper, we present a pragmatic approach to the study of functional annotations. an ensemble of metrics, describing various aspects of functional annotations, is defined and implemented in a unified framework, which facilitates their systematic analysis and inter-comparison. the use of this framework is demonstrated on three illustrative examples: analysing the outputs of state-of-the-art inference pipelines, comparing electronic versus manual annotation methods, and monitoring the evolution of publicly available functional annotations. the framework is part of the aigo library for the analysis and the inter-comparison of the products of gene ontology annotation pipelines. the aigo library also provides functionalities to easily load, analyse, manipulate and compare functional annotations and also to plot and export the results of the analysis in various formats.  this work is a step toward developing a unified framework for the systematic study of go functional annotations. this framework has been designed so that new metrics on go functional annotations can be added in a very straightforward way. 