BACKGROUND
prokaryotic and eukaryotic genomes each encode for hundreds of membrane transporter proteins that play essential roles for the cellular import and export of ions and small molecules.
furthermore, transporters mediate signal transduction processes catalyzing the export and uptake of signaling molecules.
therefore, the functional classification of membrane transporters is an important task.
the available experimental knowledge about transporter function has been compiled in databases such as tcdb  <cit> , transportdb  <cit> , sgd  <cit> , and aramemnon  <cit> .
in these databases, the functional classification is normally done according to the hierarchical transporter classification  system  <cit>  adopted by the international union of biochemistry and molecular biology .
the tc system categorizes transporter sequences according to their class, subclass,  family, and subfamily on the basis of functional or phylogenetic information that is based on sequence similarity.
an example for this classification would be the pts glucose-glucoside  super family  <dig> a.
<dig> that belongs to class ‘4’ group translocators and subclass ‘a’ phosphate transfer-driven group translocators.
subfamilies might correspond to transported substrates.
a particular transporter sequence in such a family is identified by an extra digit to the right as e.g.
<dig> a.
<dig> .
<dig>
a very important detail about each membrane transporter is of course the nature of its transported substrate molecule.
as an alternative to the tc system, one may also classify transporters into different sets according to their substrates.
it is presently unclear how such a substrate-based classification compares with the tc classification system.
for example, the aramemnon database lists members of five different tc families as phosphate transporters in arabidopsis thaliana.
in fact, many databases ignore the fourth digit  of the tc system that normally refers to the main substrate.
schaadt and helms have recently reported that membrane transporters from arabidopsis thaliana that either transport amino acids, oligopeptides, phosphate, or sugar molecules can be distinguished from each other based on their amino acid composition  <cit> .
an important research question for membrane biology is whether two membrane transporters in organisms x and y that show a certain sequence similarity will have the same function or not.
previous computational work in this area classified transporters using sequence homology and motif searches  <cit> , amino acid composition  <cit> , and substrate specificity  <cit> .
interestingly, no study has so far critically analyzed the reliability margins of the individual features.
in the general context of protein function, the pfam repository of protein families has become a quasi-standard.
pfam employs so-called gathering thresholds that are manually curated, family-specific, bit score thresholds that are chosen by pfam curators at the time a family is built.
the threshold used recently corresponds roughly to ‘safe’ e-value thresholds of ~10- <dig> <cit> .
in the tc system, the standard used for establishing homology between two proteins is  <dig> standard deviations .
this corresponds to a probability of 10- <dig> that the degree of similarity observed arose by chance  <cit> .
chen and colleagues have recently assessed the performance of different orthology detection strategies for eukaryotic genomes  <cit> .
here, we have selected the three important model systems escherichia coli , saccharomyces cerevisiae , and arabidopsis thaliana  that belong arguably to the best characterized species in terms of transport processes.
analyzing homolog databases we found that sc and at have more homologs compared to pairs  and  what reflects the smaller phylogenetic distance between sc and at.
according to the inparanoid database  <cit> ,  <dig> out of the  <dig> at genes  have homologs in sc and  <dig> out of the  <dig> sc genes  have homologs in at.
for comparison,  <dig> sc genes  have homologs in ec and  <dig> out of  <dig> ec genes  have homologs in sc.
finally, only  <dig> at genes  have homologs in ec and  <dig> ec genes  have homologs in at.
along the same lines, the arabidopsis sequencing project revealed that a much higher percentage of the proteins in the  <dig> major functional subsets of the at genome had a blastp match with e < 10- <dig> to a protein from sc % than to a protein from ec %  <cit> .
we used three different approaches to transfer transporter functional annotation between the three organisms by relating the level of sequence identity to the functional similarity between the three studied organisms.
in this study, we will term this comparison “functional classification”.
for this, we used the approaches blast that generates alignments that optimize a measure of local similarity  <cit> , hmmer that searches for sequence homologs and performs protein sequence alignment using probabilistic methods  <cit> , and meme that performs motif discovery in protein sequences on the basis of expectation maximization  <cit> .
so far there seem to be no accepted fixed thresholds for the prediction scores of the three tools.
therefore, different studies tend to use their own suitable set of thresholds  <cit> .
our study establishes a set of thresholds under which the transporter function can safely be transferred between the three model organisms.
RESULTS
in this work, we perform functional classification of transporter tc families and of transported substrate molecule using datasets from three model organisms.
our aim is to provide a simple guideline to biologists who wish to get a quick information whether available functional information about a transporter in species x may be transferred to another transporter sequence identified e.g.
by blast search in species y. table  <dig> provides an overview over the main data sets used in this work.
figure  <dig> lists common tc families between the three organisms and the distribution of transporters among them.
additional file 1: tables s1-s <dig> list all used transporters in this study, their tc families, substrates, and their pfam description.
ec
at
sc
membrane transporters with experimental annotation downloaded from transportdb, aramemnon and sgd for ec, at and sc, respectively.
only transporters with annotated tc and substrate families were considered in this work.
beside the tc analysis, we also created substrate families of transporters that are annotated to transport the same substrate.
for each organism, we collected four large groups of transporters that have been experimentally shown to catalyze the transport of either metal ions, phosphates, sugars, or amino acids.
metal ion transporters account for about 25% of the complete substrate dataset in each organism.
sc contains twice as many metal ion transporters as ec and at <cit> .
this can possibly be related with the existence of metallothionein proteins in yeast that function as a metal storage  <cit> .
at contains three times as many phosphate transporters as ec and four times as many as in sc.
this is probably due to the essential role of phosphate regulating the at root system  <cit> .
sugar transporters in at even account for 50% of the complete substrate dataset which is twice as many as in ec and sc.
one possible explanation for this is that plants need sugar to complete photosynthesis  <cit> .
ec and sc contain twice as many amino acid transporters as at.
figure  <dig> provides an overview to which tc families the members of the created substrate families belong.
we noticed that the transporters for these four substrates are spread over many different tc families.
matching tc families
in this work, we used blast for aligning all transporter sequences of one organism against their tc analogues in the two other organisms.
then, we calculated the accuracy measures precision, recall, and f-measure  for various e-value thresholds.
blast multiplies the significance of a hit by the total number of residues in the database.
thus, to make the obtained results independent from the size of the searched database we divided the e-values by the size of the db that we were blasting against.
in this way e-values from searches against different tc sets or substrates sets are comparable to each other.
in the following, we will term the normalized blast results "normalized e-values".
as an example, we blasted sc transporter ydr342c either against the at dataset  or against the non-redundant  database of  <dig> with  <dig> , <dig>  residues.
among the matching sequences, we identified the arabidopsis transporter at3g <dig> in both blast runs with an e-value of 1e- <dig> when searching against the at dataset and 1e- <dig> when matching against nr.
this difference of reported e-values matches the ratio of the database sizes.
on the other hand, when computing the accuracy measures, we multiplied the results by the member count of each family and then averaged over all tc families considered in order to account for the different member count of each family, see table  <dig>  the last row shows the percentage of transporters that remained unclassified at the given threshold.
these are transporters from one organism belonging to the shared tc families that do not share sequence identity better than the given e-value to any transporter in the shared tc family from the other organism.
ec
–
at
ec
–
sc
sc
–
at
accuracy measures of the blast prediction results for finding homologous transporter pairs in the ec-at, ec-sc, and sc-at comparison that belong to the same tc family for various e-value thresholds.
the results were normalized by the size of the reference database .
both precision and recall have a peak at thresholds 1e- <dig> or 1e- <dig> but showed lower accuracies under other thresholds.
the unclassified percentage decreases as the thresholds’ values increase.
at the strictest threshold of 1e- <dig>  the assignment of tc family has very high confidence but more than 80% of the sequences cannot be assigned for the ec-at comparison and about half in the ec-sc and sc-at comparisons.
when the threshold is made more permissive, the number of correct predictions increased with few false predictions.
we found that the precision and recall increased until 1e- <dig> but at threshold 1e- <dig> the number of false predictions increased.
as expected, the unclassified percentage decreased as the thresholds were made more permissive.
based on this comparison, a rather permissive normalized blast threshold of 1e- <dig> is very acceptable but 1e- <dig> can still be considered with caution.
when using the absolute identity scores of the alignment instead of the extracted e-values, the results were untrustworthy.
the tc family prediction of ec transporters based on sc transporters annotated more sequences than the prediction based on at transporters at the strictest thresholds.
additionally, the sc-at analysis resulted in a higher accuracy compared to the ec-at analysis.
we then applied hmmer to the same datasets as for blast and calculated the accuracy measures and the unclassified percentage in the same way.
table  <dig> shows the results obtained with hmmer.
for the purpose of normalization, the results were divided by the number of found hits in the database that was searched against.
overall, the results are similar to those obtained with blast.
however, hmmer results are slightly more accurate at loose thresholds and cover a wider annotation fraction at the strictest thresholds with few more false positives.
the number of correctly predicted tc family members at the medium-strong thresholds of 1e- <dig> and 1e- <dig> is always equal or higher than with blast.
hmmer also missed fewer points  compared to blast.
this is clearly reflected by the higher recall value calculated most of the times.
it should be re-emphasized that the e-values are computed by the three programs used here in different ways and are, thus, not directly comparable.
also, we have applied different normalization procedures - as suggested by the developers - to normalize the results to per-residue or per-sequence levels.
ec
–
at
ec
–
sc
sc
–
at
hmmer prediction results  under the given e-value confidence thresholds.
the results were normalized by the size of the reference database .
hmmer gave a better accuracy under loose thresholds compared to blast.
the decisions by hmmer appear similar to blast between the three organisms.
apparently, hmmer attained slightly higher precision for almost all thresholds compared to blast especially at loose thresholds.
additionally, in the ec-sc and the sc-at analysis, hmmer made predictions for a larger fraction of the test set with a noticeably higher recall for thresholds till 1e- <dig> compared to blast.
for threshold 1e- <dig>  hmmer predicted a slightly smaller fraction of the test set compared to blast but hmmer reported much higher prediction accuracy.
hence, we suggest an acceptable hmmer threshold of 1e- <dig>
the enriched sequence motifs identified by meme in sequences from one organism were subsequently searched in test sets of sequences from the other two organisms using the mast program  <cit>  from the meme suite.
table  <dig> illustrates the results based on using motif searches for family classification of transporters.
as can be expected, motif based searches performed better in families with many members such as  <dig> a.
<dig>  for loose thresholds, motif based classification showed lower precision compared to hmmer and blast but a comparable precision at the strictest thresholds of 1e- <dig> and 1e- <dig> as in ec-sc and sc-at analysis.
we suggest that motif based methods may be used beneficially in combination with other methods to support transporter classification.
at looser thresholds than 1e- <dig>  motif-based searches seem to lead to unreliable results and should be used with high caution.
ec
–
at
ec
–
sc
sc
–
at
mast results searching for motifs predicted by meme in the sc and at test sets.
despite the fact that all sequences were classified, prediction accuracy is generally low at loose thresholds and at the strictest threshold in the ec-at analysis.
matching substrates families
in a second step, we used the same three methods to test whether annotations about the transported substrate can be transferred from one organism to the other.
for this, we created four subsets of metal ions transporters, phosphate transporters, sugar transporters, and amino acid transporters.
these are the four largest known substrate families and comprised  <dig> ec transporters,  <dig> at transporters, and  <dig> sc transporters, see table  <dig>
as shown in table  <dig>  the results were markedly different from the tc family results.
despite the fact that blast reported acceptable prediction precision in the ec-at and the sc-at analysis, the program missed classification of many transporters.
we noticed that sequences tend to match sequences from their tc families in other substrate families, rather than their analogues in the same substrate family.
thus, the precision for substrate classification is generally lower than for the tc classification, in particular for the ec-sc comparison.
for instance, the metal transporter  from sc was falsely matched to about one third of all ec transporters in the four substrate families irrespective of their substrates since they belong to the same tc family .
ec
–
at
ec
–
sc
sc
–
at
blast prediction results for the four created substrate families of metal ion, phosphate, sugar and amino acid transporters.
the results were normalized by the size of the reference database .
unlike the tc family prediction, a smaller fraction of transporters was correctly classified and many were misclassified.
table  <dig> presents the hmmer prediction results for substrate families from the three organisms.
compared to blast, hmmer reported higher prediction accuracy in the ec-sc analysis but slightly lower prediction accuracy in ec-at analysis at the strict thresholds such as in the tc comparisons.
in fact, blast classified a slightly larger fraction of the test sets than hmmer in almost all runs.
hmmer was also affected by transporters tending to match their tc family members in other substrate families rather than their homologues in the same substrate families.
ec
–
at
ec
–
sc
sc
–
at
hmmer prediction results for substrate families.
the results were normalized by the size of the reference database .
hmmer gave a slightly higher prediction accuracy than blast in the ec-sc analysis.
table  <dig> shows mast search results for meme motifs from different substrate families.
meme gave weak predictions in all runs but in the sc-at analysis.
however, recall in the medium strict thresholds 1e- <dig> and 1e- <dig> in the ec-sc analysis is generally acceptable but accompanied with many misclassifications.
in the ec-at analysis the prediction accuracy was generally low.
here, even the strict threshold of 1e- <dig> is unreliable because it gave wrong assignments of substrates in two out of three analyses.
ec
–
at
ec
–
sc
sc
–
at
mast results searching for up to  <dig> motifs predicted by meme in each substrate family from sc and at.
most members of the substrate families were correctly classified for threshold  but only with a very low accuracy.
surprisingly,  <dig> sc sugar transporters were correctly classified from  <dig> motifs predicted by meme in the at sugar substrate family.
to the best of our knowledge, none of the three motifs have been annotated so far in databases such as  <cit> .
table  <dig> lists the regular expressions of these three motifs.
the motifs were found around positions  <dig>   <dig>  and  <dig> of the protein sequences, respectively.
regular expressions of the three motifs predicted in at sugar transporters that lead to correct predictions of  <dig> sc sugar transporters at the second-strictest threshold of 1e- <dig>
application of established thresholds to human datasets
next, we tested these thresholds on four hs datasets.
in comparison to the three model organisms, these datasets are likely much less complete.
we used the three tools to align the hs transporters using a set of transporters from at and sc and to align ec transporters using hs transporters.
the results are in line with the comparisons of the three model organisms.
when using blast and hmmer, only a small fraction was annotated at strict thresholds but more were classified at more permissive thresholds.
using hmmer, about 50% of the transporters remain not annotated even at the loosest threshold of 1e- <dig> whereas using blast many more were annotated but with a very low prediction accuracy.
the reason is that the hs phosphate and metal transporters were not annotated using the at and sc sets and even did not help in annotating the ec transporters.
however, sugar and amino acid transporters were mostly correctly annotated.
most annotations of hs transporters were based on matching  pairs.
in motif searches, two thirds of the hs transporters were annotated at the threshold of 1e- <dig> but none were annotated at the strictest threshold of 1e- <dig>  see table  <dig>  the complete results of matching  and  are listed in additional file 2: table s <dig>
hs
and
sc
sequences
hs transporters were better annotated using sc transporters compared to at.
the results were corrected for the size of the reference database .
about half of the transporters remained unannotated in the hmmer runs.
two thirds of the human transporters were annotated using meme at the threshold of 1e- <dig>
additionally, we studied the pairwise global similarity of all organism pairs using the program ggsearch from the fasta program suite.
the results were generally similar to blast and hmmer results with a slightly lower accuracy at the loose thresholds and even lower accuracy at the stricter thresholds.
results are listed in additional file 3: tables s5-s <dig>
prediction of tc families in substrate families
comparison of the two preceding sections shows that substrate families have less sequence similarity on average compared to tc families.
now, we tested the combination of both properties, see figure  <dig>  we performed this comparison in a systematic way.
for this, we named the extracted families in the form “substrate family_tc family”.
the four substrate families  belong to  <dig> tc families in ec,  <dig> in at and  <dig> in sc.
<dig> families substrate-tc are shared between ec and at,  <dig> also are shared between ec and sc and  <dig> are shared between sc and at.
some tc families belong to many different substrate families like the family  <dig> a.
<dig> that contains members of  <dig> ec substrate families.
we used blast to analyze the affiliation of test sequences toward their tc or substrate families.
here, only the best match of each substrate_tc family is considered.
the heatmap in figure  <dig> shows the tendency of sc sequences to match their analogues from at tc or substrate families.
some sc transporters matched strongly  their actual substrate_tc families from at like sugar_ <dig> a.
<dig>  phosphate_ <dig> a.
<dig> and metal_ <dig> a.
<dig>  however, most sequences from shared tc families had weaker matches to their tc families rather than their substrate families.
similar results were obtained in the ec-at and ec-sc comparison, see additional files 4: figure s <dig> and additional file 5: figure s <dig>  thus, we suggest that it is beneficial to apply substrate information as a pre-filter for transporter tc family classification.
on the other hand, transporters that transport the same substrate but belong to different tc families generally do not share noticeable sequence similarity.
tc information can be the stand alone feature used to classify transporters but a little tuning by substrate information elevates the prediction accuracy.
misclassification will occur in the small substrate_tc families not in the big tc families.
limitations and implications
in some way, our analysis presented here is a bit “circular” since we employ tools to identify sequence pairs belonging to the same tc categories while the tc classification itself was established in part based on phylogenetic analysis that is again based on sequence similarity.
however, in a practical use case it is far simpler to run a blast or fasta analysis than to establish a complicated phylogeny.
hence, our results reflect to what extent simple sequence similarity captures the structure of the more elaborate tc classification.
when comparing the results of the four methods , the reader should not forget that different strategies are employed by each of the methods to derive e-values for the reported results.
hence, the results of different methods are not directly comparable.
note that datasets to be used for motif discovery are typically cleaned up for sequence redundancy e.g.
using blastclust with a 25% sequence identity threshold  <cit> .
here, we did not do this because this would significantly decrease the number of families in the tc dataset that can be used for analysis.
hence, the meme analysis partially rediscovered sequence similarities.
this work suggests that the current tc system adopted by iubmb is a more robust classification feature compared to substrate classification.
it is quite likely that phylogenetic inference is a more sensitive indicator of homology than simple sequence similarity or identity.
thus, it appears worthwhile to test the performance of phylogeny-based methods to relate the substrate specificities of membrane transporters.
when trying to completely block the transport of a certain substrate across a particular membrane of an organism it is hard to rely only on the tc information because one substrate can be transported by several transporters from different tc families.
one possible explanation in fact is that transporters assigned to different sequence families might actually share a similar 3d structure and the structural similarity might provide an indication about the evolution of the transporter function.
such studies require more sensitive search methods like alignme  <cit> .
CONCLUSIONS
we observed that classifying membrane transporters according to tc families gives more accurate results than classifying them according to substrate families.
at the strictest threshold of 1e- <dig> for normalized e-values, predictions based on blast and hmmer result generally in high precision, but a huge fraction of the data remains unclassified.
we suggest an acceptable threshold of 1e- <dig> for both programs where at least three quarters of the sequences are classified with a reasonably high accuracy.
researchers who wish to apply these thresholds in their studies should multiply these thresholds by the size of the database they search against.
on the other hand, meme showed unsatisfactory behavior for thresholds below 1e- <dig>  prediction of tc families split from substrate families showed satisfactory results implying that the application of substrate information as a pre-filter would improve the prediction results.
the analysis and suggested thresholds in this study should be useful to those who wish to transfer transporter functional annotations across species without having to build a new phylogeny such as for the tc system.
with respect to substrate annotation, the findings of this work may be combined with those of schaadt et al.
<cit>  who established amino acid composition for substrate annotation of transporters, and with the work of saier mh jr.  <cit> .

 prokaryotic and eukaryotic genomes each encode for hundreds of membrane transporter proteins that play essential roles for the cellular import and export of ions and small molecules. furthermore, transporters mediate signal transduction processes catalyzing the export and uptake of signaling molecules. therefore, the functional classification of membrane transporters is an important task. the available experimental knowledge about transporter function has been compiled in databases such as tcdb, transportdb, sgd, and aramemnon. in these databases, the functional classification is normally done according to the hierarchical transporter classification system adopted by the international union of biochemistry and molecular biology. the tc system categorizes transporter sequences according to their class, subclass, family, and subfamily on the basis of functional or phylogenetic information that is based on sequence similarity. an example for this classification would be the pts glucose-glucoside super family a. that belongs to class ‘4’ group translocators and subclass ‘a’ phosphate transfer-driven group translocators. subfamilies might correspond to transported substrates. a particular transporter sequence in such a family is identified by an extra digit to the right as e.g. a.. a very important detail about each membrane transporter is of course the nature of its transported substrate molecule. as an alternative to the tc system, one may also classify transporters into different sets according to their substrates. it is presently unclear how such a substrate-based classification compares with the tc classification system. for example, the aramemnon database lists members of five different tc families as phosphate transporters in arabidopsis thaliana. in fact, many databases ignore the fourth digit of the tc system that normally refers to the main substrate. schaadt and helms have recently reported that membrane transporters from arabidopsis thaliana that either transport amino acids, oligopeptides, phosphate, or sugar molecules can be distinguished from each other based on their amino acid composition. an important research question for membrane biology is whether two membrane transporters in organisms x and y that show a certain sequence similarity will have the same function or not. previous computational work in this area classified transporters using sequence homology and motif searches, amino acid composition, and substrate specificity. interestingly, no study has so far critically analyzed the reliability margins of the individual features. in the general context of protein function, the pfam repository of protein families has become a quasi-standard. pfam employs so-called gathering thresholds that are manually curated, family-specific, bit score thresholds that are chosen by pfam curators at the time a family is built. the threshold used recently corresponds roughly to ‘safe’ e-value thresholds of ~10-. in the tc system, the standard used for establishing homology between two proteins is standard deviations. this corresponds to a probability of 10- that the degree of similarity observed arose by chance. chen and colleagues have recently assessed the performance of different orthology detection strategies for eukaryotic genomes. here, we have selected the three important model systems escherichia coli, saccharomyces cerevisiae, and arabidopsis thaliana that belong arguably to the best characterized species in terms of transport processes. analyzing homolog databases we found that sc and at have more homologs compared to pairs and what reflects the smaller phylogenetic distance between sc and at. according to the inparanoid database, out of the at genes have homologs in sc and out of the sc genes have homologs in at. for comparison, sc genes have homologs in ec and out of ec genes have homologs in sc. finally, only at genes have homologs in ec and ec genes have homologs in at. along the same lines, the arabidopsis sequencing project revealed that a much higher percentage of the proteins in the major functional subsets of the at genome had a blastp match with e < 10- to a protein from sc % than to a protein from ec %. we used three different approaches to transfer transporter functional annotation between the three organisms by relating the level of sequence identity to the functional similarity between the three studied organisms. in this study, we will term this comparison “functional classification”. for this, we used the approaches blast that generates alignments that optimize a measure of local similarity, hmmer that searches for sequence homologs and performs protein sequence alignment using probabilistic methods, and meme that performs motif discovery in protein sequences on the basis of expectation maximization. so far there seem to be no accepted fixed thresholds for the prediction scores of the three tools. therefore, different studies tend to use their own suitable set of thresholds. our study establishes a set of thresholds under which the transporter function can safely be transferred between the three model organisms.  in this work, we perform functional classification of transporter tc families and of transported substrate molecule using datasets from three model organisms. our aim is to provide a simple guideline to biologists who wish to get a quick information whether available functional information about a transporter in species x may be transferred to another transporter sequence identified e.g. by blast search in species y. table  provides an overview over the main data sets used in this work. figure  lists common tc families between the three organisms and the distribution of transporters among them. additional file 1: tables s1-s list all used transporters in this study, their tc families, substrates, and their pfam description. ec at sc membrane transporters with experimental annotation downloaded from transportdb, aramemnon and sgd for ec, at and sc, respectively. only transporters with annotated tc and substrate families were considered in this work. beside the tc analysis, we also created substrate families of transporters that are annotated to transport the same substrate. for each organism, we collected four large groups of transporters that have been experimentally shown to catalyze the transport of either metal ions, phosphates, sugars, or amino acids. metal ion transporters account for about 25% of the complete substrate dataset in each organism. sc contains twice as many metal ion transporters as ec and at. this can possibly be related with the existence of metallothionein proteins in yeast that function as a metal storage. at contains three times as many phosphate transporters as ec and four times as many as in sc. this is probably due to the essential role of phosphate regulating the at root system. sugar transporters in at even account for 50% of the complete substrate dataset which is twice as many as in ec and sc. one possible explanation for this is that plants need sugar to complete photosynthesis. ec and sc contain twice as many amino acid transporters as at. figure  provides an overview to which tc families the members of the created substrate families belong. we noticed that the transporters for these four substrates are spread over many different tc families. matching tc families in this work, we used blast for aligning all transporter sequences of one organism against their tc analogues in the two other organisms. then, we calculated the accuracy measures precision, recall, and f-measure for various e-value thresholds. blast multiplies the significance of a hit by the total number of residues in the database. thus, to make the obtained results independent from the size of the searched database we divided the e-values by the size of the db that we were blasting against. in this way e-values from searches against different tc sets or substrates sets are comparable to each other. in the following, we will term the normalized blast results "normalized e-values". as an example, we blasted sc transporter ydr342c either against the at dataset or against the non-redundant database of with, residues. among the matching sequences, we identified the arabidopsis transporter at3g in both blast runs with an e-value of 1e- when searching against the at dataset and 1e- when matching against nr. this difference of reported e-values matches the ratio of the database sizes. on the other hand, when computing the accuracy measures, we multiplied the results by the member count of each family and then averaged over all tc families considered in order to account for the different member count of each family, see table  the last row shows the percentage of transporters that remained unclassified at the given threshold. these are transporters from one organism belonging to the shared tc families that do not share sequence identity better than the given e-value to any transporter in the shared tc family from the other organism. ec – at ec – sc sc – at accuracy measures of the blast prediction results for finding homologous transporter pairs in the ec-at, ec-sc, and sc-at comparison that belong to the same tc family for various e-value thresholds. the results were normalized by the size of the reference database. both precision and recall have a peak at thresholds 1e- or 1e- but showed lower accuracies under other thresholds. the unclassified percentage decreases as the thresholds’ values increase. at the strictest threshold of 1e- the assignment of tc family has very high confidence but more than 80% of the sequences cannot be assigned for the ec-at comparison and about half in the ec-sc and sc-at comparisons. when the threshold is made more permissive, the number of correct predictions increased with few false predictions. we found that the precision and recall increased until 1e- but at threshold 1e- the number of false predictions increased. as expected, the unclassified percentage decreased as the thresholds were made more permissive. based on this comparison, a rather permissive normalized blast threshold of 1e- is very acceptable but 1e- can still be considered with caution. when using the absolute identity scores of the alignment instead of the extracted e-values, the results were untrustworthy. the tc family prediction of ec transporters based on sc transporters annotated more sequences than the prediction based on at transporters at the strictest thresholds. additionally, the sc-at analysis resulted in a higher accuracy compared to the ec-at analysis. we then applied hmmer to the same datasets as for blast and calculated the accuracy measures and the unclassified percentage in the same way. table  shows the results obtained with hmmer. for the purpose of normalization, the results were divided by the number of found hits in the database that was searched against. overall, the results are similar to those obtained with blast. however, hmmer results are slightly more accurate at loose thresholds and cover a wider annotation fraction at the strictest thresholds with few more false positives. the number of correctly predicted tc family members at the medium-strong thresholds of 1e- and 1e- is always equal or higher than with blast. hmmer also missed fewer points compared to blast. this is clearly reflected by the higher recall value calculated most of the times. it should be re-emphasized that the e-values are computed by the three programs used here in different ways and are, thus, not directly comparable. also, we have applied different normalization procedures - as suggested by the developers - to normalize the results to per-residue or per-sequence levels. ec – at ec – sc sc – at hmmer prediction results under the given e-value confidence thresholds. the results were normalized by the size of the reference database. hmmer gave a better accuracy under loose thresholds compared to blast. the decisions by hmmer appear similar to blast between the three organisms. apparently, hmmer attained slightly higher precision for almost all thresholds compared to blast especially at loose thresholds. additionally, in the ec-sc and the sc-at analysis, hmmer made predictions for a larger fraction of the test set with a noticeably higher recall for thresholds till 1e- compared to blast. for threshold 1e- hmmer predicted a slightly smaller fraction of the test set compared to blast but hmmer reported much higher prediction accuracy. hence, we suggest an acceptable hmmer threshold of 1e- the enriched sequence motifs identified by meme in sequences from one organism were subsequently searched in test sets of sequences from the other two organisms using the mast program from the meme suite. table  illustrates the results based on using motif searches for family classification of transporters. as can be expected, motif based searches performed better in families with many members such as a. for loose thresholds, motif based classification showed lower precision compared to hmmer and blast but a comparable precision at the strictest thresholds of 1e- and 1e- as in ec-sc and sc-at analysis. we suggest that motif based methods may be used beneficially in combination with other methods to support transporter classification. at looser thresholds than 1e- motif-based searches seem to lead to unreliable results and should be used with high caution. ec – at ec – sc sc – at mast results searching for motifs predicted by meme in the sc and at test sets. despite the fact that all sequences were classified, prediction accuracy is generally low at loose thresholds and at the strictest threshold in the ec-at analysis. matching substrates families in a second step, we used the same three methods to test whether annotations about the transported substrate can be transferred from one organism to the other. for this, we created four subsets of metal ions transporters, phosphate transporters, sugar transporters, and amino acid transporters. these are the four largest known substrate families and comprised ec transporters, at transporters, and sc transporters, see table  as shown in table  the results were markedly different from the tc family results. despite the fact that blast reported acceptable prediction precision in the ec-at and the sc-at analysis, the program missed classification of many transporters. we noticed that sequences tend to match sequences from their tc families in other substrate families, rather than their analogues in the same substrate family. thus, the precision for substrate classification is generally lower than for the tc classification, in particular for the ec-sc comparison. for instance, the metal transporter from sc was falsely matched to about one third of all ec transporters in the four substrate families irrespective of their substrates since they belong to the same tc family. ec – at ec – sc sc – at blast prediction results for the four created substrate families of metal ion, phosphate, sugar and amino acid transporters. the results were normalized by the size of the reference database. unlike the tc family prediction, a smaller fraction of transporters was correctly classified and many were misclassified. table  presents the hmmer prediction results for substrate families from the three organisms. compared to blast, hmmer reported higher prediction accuracy in the ec-sc analysis but slightly lower prediction accuracy in ec-at analysis at the strict thresholds such as in the tc comparisons. in fact, blast classified a slightly larger fraction of the test sets than hmmer in almost all runs. hmmer was also affected by transporters tending to match their tc family members in other substrate families rather than their homologues in the same substrate families. ec – at ec – sc sc – at hmmer prediction results for substrate families. the results were normalized by the size of the reference database. hmmer gave a slightly higher prediction accuracy than blast in the ec-sc analysis. table  shows mast search results for meme motifs from different substrate families. meme gave weak predictions in all runs but in the sc-at analysis. however, recall in the medium strict thresholds 1e- and 1e- in the ec-sc analysis is generally acceptable but accompanied with many misclassifications. in the ec-at analysis the prediction accuracy was generally low. here, even the strict threshold of 1e- is unreliable because it gave wrong assignments of substrates in two out of three analyses. ec – at ec – sc sc – at mast results searching for up to motifs predicted by meme in each substrate family from sc and at. most members of the substrate families were correctly classified for threshold but only with a very low accuracy. surprisingly, sc sugar transporters were correctly classified from motifs predicted by meme in the at sugar substrate family. to the best of our knowledge, none of the three motifs have been annotated so far in databases such as. table  lists the regular expressions of these three motifs. the motifs were found around positions and of the protein sequences, respectively. regular expressions of the three motifs predicted in at sugar transporters that lead to correct predictions of sc sugar transporters at the second-strictest threshold of 1e- application of established thresholds to human datasets next, we tested these thresholds on four hs datasets. in comparison to the three model organisms, these datasets are likely much less complete. we used the three tools to align the hs transporters using a set of transporters from at and sc and to align ec transporters using hs transporters. the results are in line with the comparisons of the three model organisms. when using blast and hmmer, only a small fraction was annotated at strict thresholds but more were classified at more permissive thresholds. using hmmer, about 50% of the transporters remain not annotated even at the loosest threshold of 1e- whereas using blast many more were annotated but with a very low prediction accuracy. the reason is that the hs phosphate and metal transporters were not annotated using the at and sc sets and even did not help in annotating the ec transporters. however, sugar and amino acid transporters were mostly correctly annotated. most annotations of hs transporters were based on matching pairs. in motif searches, two thirds of the hs transporters were annotated at the threshold of 1e- but none were annotated at the strictest threshold of 1e- see table  the complete results of matching and are listed in additional file 2: table s hs and sc sequences hs transporters were better annotated using sc transporters compared to at. the results were corrected for the size of the reference database. about half of the transporters remained unannotated in the hmmer runs. two thirds of the human transporters were annotated using meme at the threshold of 1e- additionally, we studied the pairwise global similarity of all organism pairs using the program ggsearch from the fasta program suite. the results were generally similar to blast and hmmer results with a slightly lower accuracy at the loose thresholds and even lower accuracy at the stricter thresholds. results are listed in additional file 3: tables s5-s prediction of tc families in substrate families comparison of the two preceding sections shows that substrate families have less sequence similarity on average compared to tc families. now, we tested the combination of both properties, see figure  we performed this comparison in a systematic way. for this, we named the extracted families in the form “substrate family_tc family”. the four substrate families belong to tc families in ec, in at and in sc. families substrate-tc are shared between ec and at, also are shared between ec and sc and are shared between sc and at. some tc families belong to many different substrate families like the family a. that contains members of ec substrate families. we used blast to analyze the affiliation of test sequences toward their tc or substrate families. here, only the best match of each substrate_tc family is considered. the heatmap in figure  shows the tendency of sc sequences to match their analogues from at tc or substrate families. some sc transporters matched strongly their actual substrate_tc families from at like sugar_ a. phosphate_ a. and metal_ a. however, most sequences from shared tc families had weaker matches to their tc families rather than their substrate families. similar results were obtained in the ec-at and ec-sc comparison, see additional files 4: figure s and additional file 5: figure s thus, we suggest that it is beneficial to apply substrate information as a pre-filter for transporter tc family classification. on the other hand, transporters that transport the same substrate but belong to different tc families generally do not share noticeable sequence similarity. tc information can be the stand alone feature used to classify transporters but a little tuning by substrate information elevates the prediction accuracy. misclassification will occur in the small substrate_tc families not in the big tc families. limitations and implications in some way, our analysis presented here is a bit “circular” since we employ tools to identify sequence pairs belonging to the same tc categories while the tc classification itself was established in part based on phylogenetic analysis that is again based on sequence similarity. however, in a practical use case it is far simpler to run a blast or fasta analysis than to establish a complicated phylogeny. hence, our results reflect to what extent simple sequence similarity captures the structure of the more elaborate tc classification. when comparing the results of the four methods, the reader should not forget that different strategies are employed by each of the methods to derive e-values for the reported results. hence, the results of different methods are not directly comparable. note that datasets to be used for motif discovery are typically cleaned up for sequence redundancy e.g. using blastclust with a 25% sequence identity threshold. here, we did not do this because this would significantly decrease the number of families in the tc dataset that can be used for analysis. hence, the meme analysis partially rediscovered sequence similarities. this work suggests that the current tc system adopted by iubmb is a more robust classification feature compared to substrate classification. it is quite likely that phylogenetic inference is a more sensitive indicator of homology than simple sequence similarity or identity. thus, it appears worthwhile to test the performance of phylogeny-based methods to relate the substrate specificities of membrane transporters. when trying to completely block the transport of a certain substrate across a particular membrane of an organism it is hard to rely only on the tc information because one substrate can be transported by several transporters from different tc families. one possible explanation in fact is that transporters assigned to different sequence families might actually share a similar 3d structure and the structural similarity might provide an indication about the evolution of the transporter function. such studies require more sensitive search methods like alignme.  we observed that classifying membrane transporters according to tc families gives more accurate results than classifying them according to substrate families. at the strictest threshold of 1e- for normalized e-values, predictions based on blast and hmmer result generally in high precision, but a huge fraction of the data remains unclassified. we suggest an acceptable threshold of 1e- for both programs where at least three quarters of the sequences are classified with a reasonably high accuracy. researchers who wish to apply these thresholds in their studies should multiply these thresholds by the size of the database they search against. on the other hand, meme showed unsatisfactory behavior for thresholds below 1e- prediction of tc families split from substrate families showed satisfactory results implying that the application of substrate information as a pre-filter would improve the prediction results. the analysis and suggested thresholds in this study should be useful to those who wish to transfer transporter functional annotations across species without having to build a new phylogeny such as for the tc system. with respect to substrate annotation, the findings of this work may be combined with those of schaadt et al. who established amino acid composition for substrate annotation of transporters, and with the work of saier mh jr.. 
for the comparison of transported substrates, we focused on the four largest substrate classes of amino acids, sugars, metal ions, and phosphate. enriched sequence motifs identified by meme at thresholds below 1e- support accurate classification into tc families for about two thirds of the sequences. we suggest an acceptable threshold of 1e- for blast and hmmer where at least three quarters of the sequences are classified according to the tc system with a reasonably high accuracy. our findings should be useful to those who wish to transfer transporter functional annotations across species. we present a thorough benchmark at which level of sequence identity membrane transporters from escherichia coli, saccharomyces cerevisiae, and arabidopsis thaliana belong to the same families of the transporter classification system, and at what level these membrane transporters mediate the transport of the same substrate. we found that two membrane transporter sequences from different organisms that are aligned with normalized blast expectation value better than e-value 1e- are highly likely to belong to the same tc family. at similar identity thresholds, the nature of the transported substrates was more divergent than the tc family membership. once a particular transport mechanism has been identified in one organism, it is thus highly desirable to transfer this information to related transporter sequences in different organisms based on bioinformatics evidence.
 membrane transporters catalyze the transport of small solute molecules across biological barriers such as lipid bilayer membranes. experimental identification of the transported substrates is very tedious. once a particular transport mechanism has been identified in one organism, it is thus highly desirable to transfer this information to related transporter sequences in different organisms based on bioinformatics evidence.  we present a thorough benchmark at which level of sequence identity membrane transporters from escherichia coli, saccharomyces cerevisiae, and arabidopsis thaliana belong to the same families of the transporter classification system, and at what level these membrane transporters mediate the transport of the same substrate. we found that two membrane transporter sequences from different organisms that are aligned with normalized blast expectation value better than e-value 1e- are highly likely to belong to the same tc family. enriched sequence motifs identified by meme at thresholds below 1e- support accurate classification into tc families for about two thirds of the sequences. for the comparison of transported substrates, we focused on the four largest substrate classes of amino acids, sugars, metal ions, and phosphate. at similar identity thresholds, the nature of the transported substrates was more divergent than the tc family membership.  we suggest an acceptable threshold of 1e- for blast and hmmer where at least three quarters of the sequences are classified according to the tc system with a reasonably high accuracy. researchers who wish to apply these thresholds in their studies should multiply these thresholds by the size of the database they search against. our findings should be useful to those who wish to transfer transporter functional annotations across species. membrane transporterfunctional classificationblasthmmermemesubstratetc classification systemsequence homology 