BACKGROUND
the ever-increasing availability of high-volume proteomic, genomic and transcriptomics datasets has led to multiple studies aimed at the systems-level interpretation of this information using biological networks and relationship networks.
biological networks are graphs where the nodes are molecules and edges indicate interactions between them  <cit> .
as explained in  <cit> , in this type of network an allowance can be made for "suppression of detail", e.g.
the intermediate components of some interactions may be omitted and instead represented by an edge.
most commonly this type of abstraction is used to represent gene regulation, where the dna-protein interaction, transcription and translation are represented by just one edge between the regulator and its target protein.
relationship networks  <cit>  are a superset of biological networks, where there is no longer a restriction that an edge must represent an actual real-life process that links the two molecules, but instead may indicate a shared property, such as two proteins having the same type of protein domain or being mentioned in the same publication.
the types of data used for construction of such networks include, but are not limited to, sequence similarity  <cit> , shared sequence features  <cit> , genetic interactions  <cit> , gene co-expression  <cit> , protein-protein interaction  <cit> , domain interaction  <cit>  and term co-occurrence in the scientific literature  <cit> .
these types of information can be analysed independently or integrated together in order to encompass a wider range of biological mechanisms, provide additional evidence of association between entities in the network and connect disjoint parts of the network.
in these studies, different techniques have been developed for the analysis of relationship networks, but they follow similar approaches: partitioning the network into modules, identifying the graph-theoretic properties of the network and relating these to biological function.
in this work we have adopted a similar approach and have devised a set of metrics for quantifying the functional coherence of the modules in order to explore the effect of using multiple evidence-types in an integrated relationship network of arabidopsis thaliana proteins.
clustering approaches work by identifying densely interconnected areas within a network  <cit>  and are commonly used to detect modular structure in graphs.
in the context of biologically-relevant networks, these groups are often referred to as functional modules  <cit> .
functional modules in biological networks are groups of molecules that are more linked to the other members of the group than to non-members and have similar function  <cit> .
the modular structure can be used to infer function of as yet unannotated proteins  <cit> , to discover previously unknown roles of proteins in diseases  <cit>  as well as for better understanding of the regulation and interrelationship between different elements of complex biological systems  <cit> .
the function of a module is commonly identified from the annotation of its members with respect to the gene ontology   <cit> .
go consists of three separate categories - biological process, molecular function and cellular component, where each category consists of a controlled vocabulary of terms structured as a directed acyclic graph with qualified edges describing the semantic relationship between these terms.
each protein can be annotated with multiple go terms and inherits the annotation of the parent terms and this makes it challenging to quantify and analyse the functional similarity between go annotations.
this has stimulated a number of studies that have explored these problems in detail.
in particular, the importance of quantitative characterisation of go term specificity  was demonstrated by  <cit>  and based on this metric, several pair-wise quantitative measurements were developed that take into account the structure and properties of the gene ontology .
in a related set of efforts, a number of metrics were also designed to measure semantic consistency of protein sets.
original approaches were designed to identify the functional annotations for which a group of proteins was significantly enriched and did not take into account hierarchical structure of go  <cit> .
although useful, these methods have a number of limitations, which were discussed in detail in the following studies  <cit> .
to address these shortcomings, a number of extensions were proposed that combine some aspects of enrichment-based methods with adjustments for the relationship between the terms  <cit> .
at the same time, another set of measures was also developed for the quantification of overall relatedness of annotations, rather than their overrepresentation within the set  <cit> .
in this study we have drawn upon the insights that have emerged from this work in order to define a descriptive measure for comparison of functional annotation of protein sets.
we claim that to determine the biological relevance of the partitioning of a set of proteins there are two important aspects that need to be taken into consideration.
the first is that the set of go terms that best describes the common function of a representative proportion of proteins in the modules can be found at any annotation specificity level.
however, at the higher levels, which are close to the root of the gene ontology, the annotation will not be particularly informative.
this leads to a trade-off between the specificity of annotation terms and the number of proteins in the module to which it applies.
the needs of the particular application case may dictate which of these two components is more important, and metrics have been developed that allow the emphasis to be placed on one or the other  <cit> .
using the metric defined in this paper , we were able to explore these two properties in five different relationship networks.
the second aspect to be considered is that the sets of proteins with the same go annotation can be fragmented, i.e.
assigned to a number of different clusters by the clustering algorithm.
not only can the functionally similar group be spread across a number of clusters, but also may be more or less concentrated in the clusters where it is present.
to assess the functional coherence of modules from a relationship network both of these aspects, namely the representative functions of modules and the fragmentation of functional categories, are also relevant.
here we explore the potential of combined relationship networks to recover functional modules by considering four sources of information: protein-protein interaction , co-expression , sequence similarity  and co-occurrence of terms in the scientific literature .
we have also constructed a combined network , which is a union of these four networks.
these evidence types were chosen because they are often used for inferring functional relationships among genes and proteins and are readily available from the application of high throughput 'omics techniques.
a large amount of co-expression data are available for arabidopsis thaliana .
measurements of sequence similarity can be obtained for all pairs of proteins  <cit>  and co-occurrence of protein terms in abstracts can be extracted from the scientific literature  <cit> .
we decided to restrict the set of proteins in the network to those for which protein-protein interaction information is available, as we currently consider this to be limiting for arabidopsis.
this restriction means that we are only considering a small subset of arabidopsis proteins, but has the advantage that it leads to a more balanced distribution of evidence types from the four information sources among the relationships between proteins.
this setting allows us to evaluate to what extent patterns and trends that were previously found in whole proteome-based networks still hold in situations where only a subset of the whole proteome is analysed.
another motivation is to evaluate the usefulness of these approaches to extract the best possible information under conditions when data are scarce or incomplete.
as we have demonstrated in our previous work  <cit> , by combining data from multiple resources it is possible to assemble much larger and more comprehensive integrated datasets.
several providers also offer pre-integrated datasets for arabidopsis, such as string  <cit>  and atpin  <cit> .
however, the integration protocols of these data sources are not always clearly described even though the protocol can affect the structure of the network  <cit> .
we have used the ondex data-integration and visualization platform  <cit>  to integrate and analyse the information sources and we have verified that the dataset used for this study is sufficiently representative by performing the same type of analysis on the data held in the string database for the same set of proteins.
RESULTS
network properties
the contributions from the four information sources to the edges in the network are shown in table  <dig>  there are  <dig> proteins and  <dig> links in the in the combined network - which is more than five times the number of links in the ppi dataset .
the links in the network exclusive to the possible evidence sources: co-expression , co-occurrence of protein names , protein interaction  or sequence similarity  are 36%, 18%, 14% and 25% respectively.
the intersection of all evidence types is also very small .
this suggests that in this case each source of evidence tends to introduce new links into the combined network rather than reinforcing the relationships already found in other sources.
exclusive combinations count edges where the evidence that links two nodes comes only from the sources indicated, while inclusive combinations allow the possibility of additional evidence types also supporting that link.
the global properties for the relationship networks constructed from the four constituent information sources and the combined network  are shown in table  <dig>  as expected, the combined network has fewer connected components since evidence from the other data sources connects previously unconnected nodes, and the size of the largest component is larger than that of any of the constituent networks.
the diameters of the largest connected component of the seq, lit and combined network  are of similar size  and smaller than the coe and ppi networks , suggesting more cohesive or dense graphs.
the increased density and the larger size of the main connected component indicate that the all network is likely to be much harder to optimally partition using a clustering approach.
the transitivity gives a measure of clique-likeness of a graph and this is highest for the seq network  and the coe network possibly reflecting shared transcriptional regulatory mechanisms.
since the initial dataset consisted of those proteins for which interaction data was available we would expect no unconnected proteins in the ppi and all networks.
the number of orphan proteins  for the seq, coe and lit networks were  <dig>   <dig> and  <dig> respectively.
the numbers of orphan proteins, however, depend on the score thresholds chosen .
network clustering
the four single information source networks and the combined network  were clustered using the mcl algorithm  <cit> .
the distribution of cluster sizes is shown in figure  <dig>  the seq and ppi networks have a large number of clusters of size  <dig> and  <dig>  the integrated network  and protein interaction network  contained the greatest number of larger clusters .
in the all network there are a large number of clusters of size  <dig>  a total of  <dig> size  <dig> clusters contain  <dig> % of all proteins in the network.
this may be related to the cohesiveness of the all network, with tightly connected groupings leading to the exclusion of nodes by the mcl algorithm.
to explore the functional groupings of proteins in the network, we combined arabidopsis go annotations from three sources: intact  <cit> , goa-ebi  <cit>  and uniprot  <cit> .
information content-based measures  were used to evaluate annotation specificity.
we wished to explore  whether the clusters contain proteins that are generally similar in terms of their functions, as assigned by gene ontology terms   the way in which proteins with the same functional roles are distributed across different clusters .
coverage and specificity of the most representative function of modules
the utility of clustering depends on being able to group together a large enough number of proteins, so as to facilitate exploring the modular structure of the network without diluting the information content of the clusters to such an extent that the groupings do not capture biologically meaningful relationships.
the average information content of the sets of these most informative common ancestor go terms  was used to determine the coverage and the specificity of the most representative function of modules .
if a cluster contained proteins that were of very diverse function, we would expect that the go categories corresponding to the most representative functions would not be very specific, i.e.
the most informative common ancestor  would be close to the root of the ontology tree and thus would not represent a functionally meaningful grouping.
given that the links in a relationship network may not always reflect accurate functional relationships, we do not look for the mica of all the proteins in the cluster.
instead we measure the average information content  associated with a set of mica of at least a certain coverage , sampled at 10% increments from 40% to 90%.
this method allows simultaneous detection of functional similarities in more than one functional category and is more robust to outliers - as only a certain proportion of the proteins in the cluster need to share functional similarity in order for their ancestor go term to be included in the set.
in figure  <dig> the aic-mica is plotted for the five relationship networks constructed for this study as well as for the weighted version of the string database.
as expected, the average information content of the representative go terms decreases with the increase in cluster coverage, namely the requirement that the common ancestor includes a greater proportion of proteins in the cluster.
average information content in the lit network is similar to the all at lower coverage range , but declines very sharply and is second worst at the higher coverage level.
this may be an indication that although useful associations can be found using term co-occurrence, these clusters tend to be less coherent at the whole-cluster level.
clusters in the coe network have the lowest information content at all coverage levels.
the information content at coverage level of 90%,  is highest for the seq network followed by the all network.
in the seq network, however, only  <dig> proteins are assigned to clusters  whereas in the all network this figure is  <dig>  for proteins that cannot be assigned to a module this means that no inference can be made using the guilt-by-association principle.
so, for only  <dig> % of proteins, no insight can be gained from clustering the all network, whereas for the seq network this figure is  <dig> %.
therefore, the all network has a much greater potential for suggesting biological context, supporting our hypothesis that the integration of multiple information sources can be beneficial when identifying functional modules.
the string comparison analysis was undertaken using a complete set of information from the string database for the same set of  <dig> proteins.
the results indicate that the performance of string at the higher coverage  levels was comparable to that of the all network.
we have also considered individual evidence types from string , which were found to be similar to the results obtained for the corresponding datasets constructed for this paper, if the data are interpreted as an un-weighted network.
the results of this analysis can be found in the additional file 1: additional figures and analysis.
modules in the all relationship network and their most representative functions
we have explored the possibility of further post-processing the clusters produced by the mcl algorithm by looking at the average fsweight  <cit>  of the pairs of clusters; but the results proved inconclusive.
further information about this analysis is included in the additional file 1: additional figures and analysis.
however, we did observe that the average fsweight of edges inside clusters was significantly higher than that for edges connecting different clusters.
fragmentation of functional categories
the other aspect that needs to be taken into consideration when assessing the functional coherence of modules is the fragmentation of functional categories.
here, we examined how the gene ontology terms were distributed across the clusters.
clustering can result in proteins with the same functional annotation being split across multiple clusters.
this leads to the separation of this group of proteins into multiple fragments.
in table  <dig>  the best fragment rank proportion  indicates that the sets of proteins annotated with the same go term are least fragmented in the all network.
this suggests that the combined network performed better than the individual networks for grouping together entities with similar go annotation.
to evaluate the level of fragmentation of functional categories, both the number of fragments and their size distribution need to be considered.
the entropy of the fragmentation  gives us a measure of this size distribution.
as can be seen in table  <dig>  the best entropy rank proportion  is also maximal for the all network, followed by the lit network, indicating that overall the entropy with respect to go categorisation is also lowest in these networks.
the third row details the decrease in entropy observed in the actual network compared to the randomly permuted network.
by both methods of measuring network fragmentation, the all dataset produces the best quality networks.
the percentages may not add up to a 100%, as in cases where several networks performed the same all were counted as "best" for that go term.
the fourth and fifth rows show the best fragment rank percentage and best entropy rank percentage statistics .
a lower entropy value implies more ordered data, both in terms of reduced fragmentation and prevalence of larger fragments.
table  <dig> shows the average entropy values for each network and for the corresponding control networks where cluster labels have been randomly permuted for all go categories.
to avoid the problems of small sample sizes, only those go categories that were assigned to at least  <dig> proteins in the dataset were included.
the all network has the lowest average entropy, again suggesting that it is better at grouping together related proteins, the average entropy being  <dig>  compared with  <dig>  for the same networks with go labels associated with the nodes, being randomly permuted.
all of the observed differences in entropy were found to be highly significant, with none of the permuted networks having an average entropy value greater than that of the real one, indicating a confidence of at least p <  <dig> .
this appears to be due to the hierarchical nature of the go categories, where every wrong assignment with respect to a child term would also lead to penalties incurred at the parent level.
therefore, the density distributions for the permuted networks were very narrow .
an example of fragmentation in the all relationship network
discussion
in order to assess the functional coherence of modules detected by clustering relationship networks combining four commonly used data sources we have looked at the representative functions of these modules with respect to go categories and at the fragmentation of go categories with respect to the modules.
to investigate the trade-off between coverage and specificity of the representative function of modules, we have defined the aic-mica metric.
additionally, two metrics describing the fragmentation of go categories, namely bfrp and berp, were introduced to evaluate how well the modular structure recovered by the mcl algorithm corresponds to the bp categories.
these metrics look at two key aspects that relate modules in relationship networks to functional annotations.
they allow us to compare the usefulness of individual data sources and the effects of combining multiple sources on the coherence of the modules.
we have found that, as expected, the seq network was the best for recovering very specific functional association between proteins.
this was evident from the high aic-mica values across all coverage levels.
however, an important point to note is that it may not always be desirable to extract such close groupings, and the higher level categorisation may be helpful to provide a broad overview or to help dissect very large datasets.
compared to other networks, seq consisted of a large number of strongly connected components  which resulted in the relatively high overall entropy with respect to the whole gene ontology.
we also observed that the clusters recovered were only related to a small number of go terms.
another problem with seq as a sole data source is that there was insufficient evidence to link most of the proteins in our reference set.
by comparison with the seq network, it was possible to use the all network to assign  <dig> more proteins to a cluster of size greater than one due to links that were contributed by other sources.
based on these findings, we conclude that overall there is a clear benefit from the integration of additional data sources, although there is a small cost incurred because of a reduction in functional coherence.
as the all network performs relatively well in terms of aic-mica , this dilution of annotation specificity does not appear to render it uninformative.
in fact, the minimum information content value that was applicable at a 40% coverage level was  <dig>  and was reached only for  <dig> clusters found in the all network.
this value corresponds to the 'cellular physiological process' go term, which is one of the direct descendants of the 'biological process' root term, and is therefore very general.
to support this work, several different visualisation strategies were developed that help to summarise complex integrated networks and identify high-level patterns in them.
using these visualisation methods, we have identified that there was a hierarchically organised neighbourhood in the integrated network that was composed of the proteins annotated to the "response to hormone stimulus" go term.
this finding indicates there may be more complex and meaningful patterns than just the modules that could be identified using clustering approaches.
comparison of graph-theoretic properties of the four networks also appears to indicate that the addition of extra edges lead to the creation of a more compact network, with smaller diameter than the coe or ppi networks.
despite this, the transitivity has remained relatively low - indicating that the number of complete cliques is small.
these differences may be interpreted as an indication that, in the all network, potential modules are more difficult to recover and the results may be further improved using more robust clustering approaches, like spectral clustering methods  <cit> .
further investigation of the impact of increasing complexity of the network versus increasing levels of noise that arise from integration of additional data sources is necessary to confirm these trends.
the co-expression  network performed the worst with respect to bfrp, berp and aic-mica.
at first glance, this result appears to contradict several earlier studies  <cit>  where many meaningful clusters were identified in the co-expression network but this discrepancy is likely to be an artefact of the smaller subset of the proteome that was used in this case.
in earlier reports using large co-expression networks, the patterns detected tended to be associated with clusters containing more than a  <dig> proteins  <cit> , which are much larger than any of the modules identified in this study.
this may be an indication that co-expression is a weaker source of evidence of functional similarity and more data are necessary in order to be able to make useful inferences.
in this study we have restricted the set of proteins in the network to those for which protein-protein interaction information is available, as this is a currently limiting information source for arabidopsis.
using a larger set of proteins would have meant that the contribution of the ppi data would have been highly unbalanced in relation to other available information.
although we recognise that there are other species, in particular saccharomyces cerevisiae, for which there is much more data available, it is also of importance to validate these types of approaches in more complex multicellular model organisms.
we have also illustrated that meaningful modules can be successfully identified by clustering the integrated relationship networks even in situations when limited data are available and only part of the complete proteome is considered.
in this work we have addressed a number of important issues pertinent to the identification of functional modules in integrated relationship networks, but it is important to recognise that a number of alternative approaches exist for analysis of such networks.
in particular, it is possible to weight the edges in the network based on the confidence in individual evidence types  <cit> .
however, both the strategies of selecting optimum weights and the ways they can be meaningfully combined across heterogeneous evidence types still remains a subject of ongoing research.
another possibility is to use an alternative clustering approach for the recovery of modules.
historically, the mcl algorithm has often been applied in the context of biological networks because it offers scalable performance even with large datasets and several studies have shown that it can outperform other methods in some cases  <cit> .
however, a number of other novel algorithms have now been developed, among them mcode  <cit> , mc-upgma  <cit>  cpa  <cit>  force  <cit>  and spici  <cit> .
a number of these approaches have also been compared in the context of ppi networks in the work by brohée and van helden  <cit> .
further investigation into these alternative approaches has potential for future research, but was outside the scope of the present study.
CONCLUSIONS
module detection in integrated biological and relationship networks is one of the most important tools for interpretation of complex biological datasets.
as the amount of biological information continues to grow, it also becomes increasingly important to improve our understanding of inter-relationships within these data and, ultimately, their relationship to biological function.
in this paper we have explored and quantified the integration of the several data types that are most commonly used for construction of such networks.
for our datasets, we have found that combining several types of evidence was beneficial with respect to the functional annotation of modules detected using mcl clustering algorithm, that on average more closely corresponded to the functional groupings in the biological process aspect of go.
although the overall level of informativeness of cluster annotation was not as good as in the sequence similarity network, it was possible to link many more proteins using additional information sources.
these findings indicate that there is benefit to the integration of additional information sources, as it allows more proteins to be assigned to functional modules with only a relatively small reduction in the module annotation precision.
the overall outcomes of this study provide a number of insights into the relationship between integrated networks and protein function and may be of use for further refinement of related approaches that can better capture biologically relevant information from integrated datasets.

 the ever-increasing availability of high-volume proteomic, genomic and transcriptomics datasets has led to multiple studies aimed at the systems-level interpretation of this information using biological networks and relationship networks. biological networks are graphs where the nodes are molecules and edges indicate interactions between them. as explained in, in this type of network an allowance can be made for "suppression of detail", e.g. the intermediate components of some interactions may be omitted and instead represented by an edge. most commonly this type of abstraction is used to represent gene regulation, where the dna-protein interaction, transcription and translation are represented by just one edge between the regulator and its target protein. relationship networks are a superset of biological networks, where there is no longer a restriction that an edge must represent an actual real-life process that links the two molecules, but instead may indicate a shared property, such as two proteins having the same type of protein domain or being mentioned in the same publication. the types of data used for construction of such networks include, but are not limited to, sequence similarity, shared sequence features, genetic interactions, gene co-expression, protein-protein interaction, domain interaction and term co-occurrence in the scientific literature. these types of information can be analysed independently or integrated together in order to encompass a wider range of biological mechanisms, provide additional evidence of association between entities in the network and connect disjoint parts of the network. in these studies, different techniques have been developed for the analysis of relationship networks, but they follow similar approaches: partitioning the network into modules, identifying the graph-theoretic properties of the network and relating these to biological function. in this work we have adopted a similar approach and have devised a set of metrics for quantifying the functional coherence of the modules in order to explore the effect of using multiple evidence-types in an integrated relationship network of arabidopsis thaliana proteins. clustering approaches work by identifying densely interconnected areas within a network and are commonly used to detect modular structure in graphs. in the context of biologically-relevant networks, these groups are often referred to as functional modules. functional modules in biological networks are groups of molecules that are more linked to the other members of the group than to non-members and have similar function. the modular structure can be used to infer function of as yet unannotated proteins, to discover previously unknown roles of proteins in diseases as well as for better understanding of the regulation and interrelationship between different elements of complex biological systems. the function of a module is commonly identified from the annotation of its members with respect to the gene ontology. go consists of three separate categories - biological process, molecular function and cellular component, where each category consists of a controlled vocabulary of terms structured as a directed acyclic graph with qualified edges describing the semantic relationship between these terms. each protein can be annotated with multiple go terms and inherits the annotation of the parent terms and this makes it challenging to quantify and analyse the functional similarity between go annotations. this has stimulated a number of studies that have explored these problems in detail. in particular, the importance of quantitative characterisation of go term specificity was demonstrated by and based on this metric, several pair-wise quantitative measurements were developed that take into account the structure and properties of the gene ontology. in a related set of efforts, a number of metrics were also designed to measure semantic consistency of protein sets. original approaches were designed to identify the functional annotations for which a group of proteins was significantly enriched and did not take into account hierarchical structure of go. although useful, these methods have a number of limitations, which were discussed in detail in the following studies. to address these shortcomings, a number of extensions were proposed that combine some aspects of enrichment-based methods with adjustments for the relationship between the terms. at the same time, another set of measures was also developed for the quantification of overall relatedness of annotations, rather than their overrepresentation within the set. in this study we have drawn upon the insights that have emerged from this work in order to define a descriptive measure for comparison of functional annotation of protein sets. we claim that to determine the biological relevance of the partitioning of a set of proteins there are two important aspects that need to be taken into consideration. the first is that the set of go terms that best describes the common function of a representative proportion of proteins in the modules can be found at any annotation specificity level. however, at the higher levels, which are close to the root of the gene ontology, the annotation will not be particularly informative. this leads to a trade-off between the specificity of annotation terms and the number of proteins in the module to which it applies. the needs of the particular application case may dictate which of these two components is more important, and metrics have been developed that allow the emphasis to be placed on one or the other. using the metric defined in this paper, we were able to explore these two properties in five different relationship networks. the second aspect to be considered is that the sets of proteins with the same go annotation can be fragmented, i.e. assigned to a number of different clusters by the clustering algorithm. not only can the functionally similar group be spread across a number of clusters, but also may be more or less concentrated in the clusters where it is present. to assess the functional coherence of modules from a relationship network both of these aspects, namely the representative functions of modules and the fragmentation of functional categories, are also relevant. here we explore the potential of combined relationship networks to recover functional modules by considering four sources of information: protein-protein interaction, co-expression, sequence similarity and co-occurrence of terms in the scientific literature. we have also constructed a combined network, which is a union of these four networks. these evidence types were chosen because they are often used for inferring functional relationships among genes and proteins and are readily available from the application of high throughput'omics techniques. a large amount of co-expression data are available for arabidopsis thaliana. measurements of sequence similarity can be obtained for all pairs of proteins and co-occurrence of protein terms in abstracts can be extracted from the scientific literature. we decided to restrict the set of proteins in the network to those for which protein-protein interaction information is available, as we currently consider this to be limiting for arabidopsis. this restriction means that we are only considering a small subset of arabidopsis proteins, but has the advantage that it leads to a more balanced distribution of evidence types from the four information sources among the relationships between proteins. this setting allows us to evaluate to what extent patterns and trends that were previously found in whole proteome-based networks still hold in situations where only a subset of the whole proteome is analysed. another motivation is to evaluate the usefulness of these approaches to extract the best possible information under conditions when data are scarce or incomplete. as we have demonstrated in our previous work, by combining data from multiple resources it is possible to assemble much larger and more comprehensive integrated datasets. several providers also offer pre-integrated datasets for arabidopsis, such as string and atpin. however, the integration protocols of these data sources are not always clearly described even though the protocol can affect the structure of the network. we have used the ondex data-integration and visualization platform to integrate and analyse the information sources and we have verified that the dataset used for this study is sufficiently representative by performing the same type of analysis on the data held in the string database for the same set of proteins.  network properties the contributions from the four information sources to the edges in the network are shown in table there are proteins and links in the in the combined network - which is more than five times the number of links in the ppi dataset. the links in the network exclusive to the possible evidence sources: co-expression, co-occurrence of protein names, protein interaction or sequence similarity are 36%, 18%, 14% and 25% respectively. the intersection of all evidence types is also very small. this suggests that in this case each source of evidence tends to introduce new links into the combined network rather than reinforcing the relationships already found in other sources. exclusive combinations count edges where the evidence that links two nodes comes only from the sources indicated, while inclusive combinations allow the possibility of additional evidence types also supporting that link. the global properties for the relationship networks constructed from the four constituent information sources and the combined network are shown in table as expected, the combined network has fewer connected components since evidence from the other data sources connects previously unconnected nodes, and the size of the largest component is larger than that of any of the constituent networks. the diameters of the largest connected component of the seq, lit and combined network are of similar size and smaller than the coe and ppi networks, suggesting more cohesive or dense graphs. the increased density and the larger size of the main connected component indicate that the all network is likely to be much harder to optimally partition using a clustering approach. the transitivity gives a measure of clique-likeness of a graph and this is highest for the seq network and the coe network possibly reflecting shared transcriptional regulatory mechanisms. since the initial dataset consisted of those proteins for which interaction data was available we would expect no unconnected proteins in the ppi and all networks. the number of orphan proteins for the seq, coe and lit networks were and respectively. the numbers of orphan proteins, however, depend on the score thresholds chosen. network clustering the four single information source networks and the combined network were clustered using the mcl algorithm. the distribution of cluster sizes is shown in figure the seq and ppi networks have a large number of clusters of size and the integrated network and protein interaction network contained the greatest number of larger clusters. in the all network there are a large number of clusters of size a total of size clusters contain % of all proteins in the network. this may be related to the cohesiveness of the all network, with tightly connected groupings leading to the exclusion of nodes by the mcl algorithm. to explore the functional groupings of proteins in the network, we combined arabidopsis go annotations from three sources: intact, goa-ebi and uniprot. information content-based measures were used to evaluate annotation specificity. we wished to explore whether the clusters contain proteins that are generally similar in terms of their functions, as assigned by gene ontology terms the way in which proteins with the same functional roles are distributed across different clusters. coverage and specificity of the most representative function of modules the utility of clustering depends on being able to group together a large enough number of proteins, so as to facilitate exploring the modular structure of the network without diluting the information content of the clusters to such an extent that the groupings do not capture biologically meaningful relationships. the average information content of the sets of these most informative common ancestor go terms was used to determine the coverage and the specificity of the most representative function of modules. if a cluster contained proteins that were of very diverse function, we would expect that the go categories corresponding to the most representative functions would not be very specific, i.e. the most informative common ancestor would be close to the root of the ontology tree and thus would not represent a functionally meaningful grouping. given that the links in a relationship network may not always reflect accurate functional relationships, we do not look for the mica of all the proteins in the cluster. instead we measure the average information content associated with a set of mica of at least a certain coverage, sampled at 10% increments from 40% to 90%. this method allows simultaneous detection of functional similarities in more than one functional category and is more robust to outliers - as only a certain proportion of the proteins in the cluster need to share functional similarity in order for their ancestor go term to be included in the set. in figure the aic-mica is plotted for the five relationship networks constructed for this study as well as for the weighted version of the string database. as expected, the average information content of the representative go terms decreases with the increase in cluster coverage, namely the requirement that the common ancestor includes a greater proportion of proteins in the cluster. average information content in the lit network is similar to the all at lower coverage range, but declines very sharply and is second worst at the higher coverage level. this may be an indication that although useful associations can be found using term co-occurrence, these clusters tend to be less coherent at the whole-cluster level. clusters in the coe network have the lowest information content at all coverage levels. the information content at coverage level of 90%, is highest for the seq network followed by the all network. in the seq network, however, only proteins are assigned to clusters whereas in the all network this figure is for proteins that cannot be assigned to a module this means that no inference can be made using the guilt-by-association principle. so, for only % of proteins, no insight can be gained from clustering the all network, whereas for the seq network this figure is %. therefore, the all network has a much greater potential for suggesting biological context, supporting our hypothesis that the integration of multiple information sources can be beneficial when identifying functional modules. the string comparison analysis was undertaken using a complete set of information from the string database for the same set of proteins. the results indicate that the performance of string at the higher coverage levels was comparable to that of the all network. we have also considered individual evidence types from string, which were found to be similar to the results obtained for the corresponding datasets constructed for this paper, if the data are interpreted as an un-weighted network. the results of this analysis can be found in the additional file 1: additional figures and analysis. modules in the all relationship network and their most representative functions we have explored the possibility of further post-processing the clusters produced by the mcl algorithm by looking at the average fsweight of the pairs of clusters; but the results proved inconclusive. further information about this analysis is included in the additional file 1: additional figures and analysis. however, we did observe that the average fsweight of edges inside clusters was significantly higher than that for edges connecting different clusters. fragmentation of functional categories the other aspect that needs to be taken into consideration when assessing the functional coherence of modules is the fragmentation of functional categories. here, we examined how the gene ontology terms were distributed across the clusters. clustering can result in proteins with the same functional annotation being split across multiple clusters. this leads to the separation of this group of proteins into multiple fragments. in table the best fragment rank proportion indicates that the sets of proteins annotated with the same go term are least fragmented in the all network. this suggests that the combined network performed better than the individual networks for grouping together entities with similar go annotation. to evaluate the level of fragmentation of functional categories, both the number of fragments and their size distribution need to be considered. the entropy of the fragmentation gives us a measure of this size distribution. as can be seen in table the best entropy rank proportion is also maximal for the all network, followed by the lit network, indicating that overall the entropy with respect to go categorisation is also lowest in these networks. the third row details the decrease in entropy observed in the actual network compared to the randomly permuted network. by both methods of measuring network fragmentation, the all dataset produces the best quality networks. the percentages may not add up to a 100%, as in cases where several networks performed the same all were counted as "best" for that go term. the fourth and fifth rows show the best fragment rank percentage and best entropy rank percentage statistics. a lower entropy value implies more ordered data, both in terms of reduced fragmentation and prevalence of larger fragments. table shows the average entropy values for each network and for the corresponding control networks where cluster labels have been randomly permuted for all go categories. to avoid the problems of small sample sizes, only those go categories that were assigned to at least proteins in the dataset were included. the all network has the lowest average entropy, again suggesting that it is better at grouping together related proteins, the average entropy being compared with for the same networks with go labels associated with the nodes, being randomly permuted. all of the observed differences in entropy were found to be highly significant, with none of the permuted networks having an average entropy value greater than that of the real one, indicating a confidence of at least p <. this appears to be due to the hierarchical nature of the go categories, where every wrong assignment with respect to a child term would also lead to penalties incurred at the parent level. therefore, the density distributions for the permuted networks were very narrow. an example of fragmentation in the all relationship network discussion in order to assess the functional coherence of modules detected by clustering relationship networks combining four commonly used data sources we have looked at the representative functions of these modules with respect to go categories and at the fragmentation of go categories with respect to the modules. to investigate the trade-off between coverage and specificity of the representative function of modules, we have defined the aic-mica metric. additionally, two metrics describing the fragmentation of go categories, namely bfrp and berp, were introduced to evaluate how well the modular structure recovered by the mcl algorithm corresponds to the bp categories. these metrics look at two key aspects that relate modules in relationship networks to functional annotations. they allow us to compare the usefulness of individual data sources and the effects of combining multiple sources on the coherence of the modules. we have found that, as expected, the seq network was the best for recovering very specific functional association between proteins. this was evident from the high aic-mica values across all coverage levels. however, an important point to note is that it may not always be desirable to extract such close groupings, and the higher level categorisation may be helpful to provide a broad overview or to help dissect very large datasets. compared to other networks, seq consisted of a large number of strongly connected components which resulted in the relatively high overall entropy with respect to the whole gene ontology. we also observed that the clusters recovered were only related to a small number of go terms. another problem with seq as a sole data source is that there was insufficient evidence to link most of the proteins in our reference set. by comparison with the seq network, it was possible to use the all network to assign more proteins to a cluster of size greater than one due to links that were contributed by other sources. based on these findings, we conclude that overall there is a clear benefit from the integration of additional data sources, although there is a small cost incurred because of a reduction in functional coherence. as the all network performs relatively well in terms of aic-mica, this dilution of annotation specificity does not appear to render it uninformative. in fact, the minimum information content value that was applicable at a 40% coverage level was and was reached only for clusters found in the all network. this value corresponds to the'cellular physiological process' go term, which is one of the direct descendants of the'biological process' root term, and is therefore very general. to support this work, several different visualisation strategies were developed that help to summarise complex integrated networks and identify high-level patterns in them. using these visualisation methods, we have identified that there was a hierarchically organised neighbourhood in the integrated network that was composed of the proteins annotated to the "response to hormone stimulus" go term. this finding indicates there may be more complex and meaningful patterns than just the modules that could be identified using clustering approaches. comparison of graph-theoretic properties of the four networks also appears to indicate that the addition of extra edges lead to the creation of a more compact network, with smaller diameter than the coe or ppi networks. despite this, the transitivity has remained relatively low - indicating that the number of complete cliques is small. these differences may be interpreted as an indication that, in the all network, potential modules are more difficult to recover and the results may be further improved using more robust clustering approaches, like spectral clustering methods. further investigation of the impact of increasing complexity of the network versus increasing levels of noise that arise from integration of additional data sources is necessary to confirm these trends. the co-expression network performed the worst with respect to bfrp, berp and aic-mica. at first glance, this result appears to contradict several earlier studies where many meaningful clusters were identified in the co-expression network but this discrepancy is likely to be an artefact of the smaller subset of the proteome that was used in this case. in earlier reports using large co-expression networks, the patterns detected tended to be associated with clusters containing more than a proteins, which are much larger than any of the modules identified in this study. this may be an indication that co-expression is a weaker source of evidence of functional similarity and more data are necessary in order to be able to make useful inferences. in this study we have restricted the set of proteins in the network to those for which protein-protein interaction information is available, as this is a currently limiting information source for arabidopsis. using a larger set of proteins would have meant that the contribution of the ppi data would have been highly unbalanced in relation to other available information. although we recognise that there are other species, in particular saccharomyces cerevisiae, for which there is much more data available, it is also of importance to validate these types of approaches in more complex multicellular model organisms. we have also illustrated that meaningful modules can be successfully identified by clustering the integrated relationship networks even in situations when limited data are available and only part of the complete proteome is considered. in this work we have addressed a number of important issues pertinent to the identification of functional modules in integrated relationship networks, but it is important to recognise that a number of alternative approaches exist for analysis of such networks. in particular, it is possible to weight the edges in the network based on the confidence in individual evidence types. however, both the strategies of selecting optimum weights and the ways they can be meaningfully combined across heterogeneous evidence types still remains a subject of ongoing research. another possibility is to use an alternative clustering approach for the recovery of modules. historically, the mcl algorithm has often been applied in the context of biological networks because it offers scalable performance even with large datasets and several studies have shown that it can outperform other methods in some cases. however, a number of other novel algorithms have now been developed, among them mcode, mc-upgma cpa force and spici. a number of these approaches have also been compared in the context of ppi networks in the work by brohée and van helden. further investigation into these alternative approaches has potential for future research, but was outside the scope of the present study.  module detection in integrated biological and relationship networks is one of the most important tools for interpretation of complex biological datasets. as the amount of biological information continues to grow, it also becomes increasingly important to improve our understanding of inter-relationships within these data and, ultimately, their relationship to biological function. in this paper we have explored and quantified the integration of the several data types that are most commonly used for construction of such networks. for our datasets, we have found that combining several types of evidence was beneficial with respect to the functional annotation of modules detected using mcl clustering algorithm, that on average more closely corresponded to the functional groupings in the biological process aspect of go. although the overall level of informativeness of cluster annotation was not as good as in the sequence similarity network, it was possible to link many more proteins using additional information sources. these findings indicate that there is benefit to the integration of additional information sources, as it allows more proteins to be assigned to functional modules with only a relatively small reduction in the module annotation precision. the overall outcomes of this study provide a number of insights into the relationship between integrated networks and protein function and may be of use for further refinement of related approaches that can better capture biologically relevant information from integrated datasets. 
the challenges that we wish to address are to define and quantify the functional coherence of modules in relationship networks, so that they can be used to infer function of as yet unannotated proteins, to discover previously unknown roles of proteins in diseases as well as for better understanding of the regulation and interrelationship between different elements of complex biological systems. from these we have constructed five relationship networks, four based on single types of data: ppi, co-expression, co-occurrence of protein names in scientific literature abstracts and sequence similarity and a fifth one combining these four evidence types. relationship networks integrating multiple evidence-types are biologically informative and allow more proteins to be assigned to a putative functional module. we have defined the functional coherence of modules with respect to the gene ontology by considering two complementary aspects: the fragmentation of the go functional categories into the different modules and the most representative functions of the modules. these results indicate that integration of more data sources improves the ability to uncover functional association between proteins, both by allowing more proteins to be linked and producing a network where modular structure more closely reflects the hierarchy in the gene ontology. using additional evidence types concentrates the functional annotations in a smaller number of modules without unduly compromising their consistency. we have proposed a set of metrics to evaluate these two aspects and demonstrated their utility in arabidopsis thaliana. the value of such modules for gaining insight into the underlying biological processes depends on their functional coherence. the ability of these networks to suggest biologically meaningful grouping of proteins was explored by applying markov clustering and then by measuring the functional coherence of the clusters. the integrated information can be represented as a relationship network, and clustering the network can suggest possible functional modules. combining multiple evidence-types from different information sources has the potential to reveal new relationships in biological systems.
 combining multiple evidence-types from different information sources has the potential to reveal new relationships in biological systems. the integrated information can be represented as a relationship network, and clustering the network can suggest possible functional modules. the value of such modules for gaining insight into the underlying biological processes depends on their functional coherence. the challenges that we wish to address are to define and quantify the functional coherence of modules in relationship networks, so that they can be used to infer function of as yet unannotated proteins, to discover previously unknown roles of proteins in diseases as well as for better understanding of the regulation and interrelationship between different elements of complex biological systems.  we have defined the functional coherence of modules with respect to the gene ontology by considering two complementary aspects: the fragmentation of the go functional categories into the different modules and the most representative functions of the modules. we have proposed a set of metrics to evaluate these two aspects and demonstrated their utility in arabidopsis thaliana. we selected proteins for which experimentally established protein-protein interaction data were available. from these we have constructed five relationship networks, four based on single types of data: ppi, co-expression, co-occurrence of protein names in scientific literature abstracts and sequence similarity and a fifth one combining these four evidence types. the ability of these networks to suggest biologically meaningful grouping of proteins was explored by applying markov clustering and then by measuring the functional coherence of the clusters.  relationship networks integrating multiple evidence-types are biologically informative and allow more proteins to be assigned to a putative functional module. using additional evidence types concentrates the functional annotations in a smaller number of modules without unduly compromising their consistency. these results indicate that integration of more data sources improves the ability to uncover functional association between proteins, both by allowing more proteins to be linked and producing a network where modular structure more closely reflects the hierarchy in the gene ontology. 