BACKGROUND
the knowledge of all protein isoforms that may be expressed by a gene is fundamental.
recently, several computational methods have been proposed for transcriptome reconstruction that use rna-seq data for exon identification, and expression levels data for transcript assembly  <cit> .
while exon identification performs quite well, transcript assembly remains difficult for complex transcriptomes.
as shown in  <cit> , the best-performing computational methods identified at most  <dig> % of spliced protein-coding transcripts from h. sapiens, and transcript detection remains low even with very high sequencing coverage, leading the authors to conclude that improved results would have to wait for technological advances.
those findings were confirmed by several other studies , that include methods recently developed such as stringtie  <cit> .
given these limitations of ab initio transcript prediction, it is natural to investigate assisted transcriptome reconstruction, in which the knowledge of transcript structures is transferred from one species to another: since transcripts provide a road-map of the successive links between exons, it should be possible to distinguish transcripts that may be expressed from those that may not, by analyzing the sequence of the target genes.
an essential problem is the assessment of the predictions.
confirmation of the prediction that “transcript t may be expressed in the target species” is only possible through experimental validation, which can be a long and costly process, since many transcripts are detectable only in specific cells, under specific conditions  <cit> .
gene predictors that make explicit use of external evidence to predict eukaryotic gene structure have been around since the turn of the century  <cit> , and with the discovery of alternative transcripts in recent years, hundreds of thousands of predicted isoforms are now available in databases  <cit> .
for example, as of june  <dig>  felis catus had  <dig>  <dig> predicted isoforms in the refseq database, of which  <dig> are confirmed, and canis lupus familiaris had  <dig>  <dig> predicted isoforms of which  <dig>  <dig> are confirmed.
these predictions were made by the gnomon predictor, described in an unpublished paper  <cit> , and whose performance is unknown, especially for predictions that rely exclusively on external evidence.
in order to evaluate the predictive power of transcript annotation transfer, it is necessary to identify splicing orthologs, loosely defined as transcripts from ortholog genes with similar splicing patterns.
zambelli et al.
<cit> , introduced the concept and gave three possible definitions that yielded estimates ranging from  <dig> to  <dig> % for the percentage of human transcripts that have a splicing ortholog in mouse.
this wide range of estimates is an indicator of how the concept is still a work-in-progress.
a subsequent paper of fong et al.
<cit>  simplifies the definition of splicing orthologs as “…all protein-coding exons in the two proteins can be paired with  <dig> % overlap in lengths of both exons.”.
this is clearly an over-simplification that ignores, among other things, nearby alternative donor or acceptor splice sites  <cit> .
in this study, we revisit the concept of splicing orthology and we give a comprehensive assessment of the performance of assisted transcriptome reconstruction using the human to predict mouse, and vice versa.
methods
our experimental set-up is based on a procedural definition of splicing orthologs that is concurrently implemented by two procedures.
our predictor uses transcripts from a known species to predict transcripts in a target species, and for evaluation purposes, identifies putative pairs of orthologous transcript isoforms based solely on nucleotide sequence evidence.
our controller identifies putative pairs of orthologous protein isoforms between human and mouse using amino acid sequences and positions of exon junctions.
splicing orthologs
transcripts from orthologous genes with differing splicing patterns could have easily identifiable differences in the number or identity of their exons.
however, defining splicing orthology can be more difficult due to the presence of alternative donor and acceptor splice sites, where exons are elongated or truncated, often by a very few nucleotides.
within a gene, these exon isoforms  <cit>  overlap and have different exon-intron border.
across species, we define orthologous exon isoforms as orthologous exons that have conserved exon-intron borders.
using this concept, splicing orthologs are defined as transcripts of orthologous genes, whose exons are orthologous exon isoforms that appear in the same order in each gene, and that code for similar proteins.
in the ideal situation where all splicing orthologs are conserved across species in one-to-one orthologous gene pairs, all exons — with their flanking intronic sequences — should be best reciprocal hits, and the alignments should preserve exon-intron borders.
each protein would have a perfect match that should be a unique best reciprocal hit, the alignments should preserve the positions of exon junctions, and be without gaps in the near vicinity of junctions.
the predictor
given a pair of orthologous genes, the first task of the predictor is to establish a common reference sequence  <cit>  for the set of all transcripts in both the human and mouse.
our solution is based on the concept of blocks, which are alignments of human and mouse exon segments that are either contained in, or disjoint from, each human or mouse transcript.
each block has a label and can be flanked by signals: start and stop codons, donor and acceptor splice sites.
using block labels, the symbols ‘’ to indicate start and stop codons, and ‘<’ and ‘>’ to indicate the intronic part of donor and acceptor splice sites, each transcript can be represented as a string.
the gene model of a set of transcripts is the ordered string of its blocks and signals, and a donor block  is a block that has a donor signal at its 3’ end  in the gene model.
figure  <dig> presents an example with the crem and crem orthologous genes of human and mouse .
fig.
<dig> gene models and transcripts.
in the ccds database  <cit> , as of june  <dig>  there were  <dig> crem human and  <dig> crem mouse transcripts with unique splicing patterns.
the common reference sequence has blocks labeled from a to u. the human gene model is at the top of the figure, followed by the sub-sequence of blocks and signals found in the mouse.
the block representation and ccds number is given for each transcript.
note that block o does not exist in the mouse gene, and block f is not in the human model, since block f is not found in any known human transcript.
of the  <dig> human transcripts,  <dig> are executable – meaning that they could be expressed –, and three are not  because they use block o. the executable transcripts are further classified as found  and paired with a mouse transcript, or yet-to-be-found .
all these predictions are confirmed by the controller.
since  <dig> mouse transcripts are correctly identified, and  <dig> mouse transcripts are currently known, the predictor successfully identifies 5/ <dig>  or  <dig> % of the mouse transcripts.
as more mouse transcripts are discovered, this proportion may increase with future releases of the ccds database
to ensure internal exons best reciprocal hit property, every human and mouse sequence in a block is blasted, in the orthologous gene, using flanking genomic sequences of length  <dig>  further details on the predictor algorithm can be found in additional file  <dig>
the fact that all blocks can be found by a blast search allows the predictor to simulate a pure predicting mode, that does not have any prior knowledge of the target species transcripts.
given a gene model m, the predictor reports all blocks and signals that were found in the target gene, and reports its successes as a substring s of the gene model.
this paper predicts in flexible mode  <cit> , where orthologous transcripts must have the same splicing pattern, but the first and last exons are only required to have orthologous internal exon-intron borders.
a transcript is said to be executable by the target gene if its donor and acceptor blocks are in s. the primary task of the predictor is to classify transcripts as executable or not executable.
in the simulation context of this paper, a secondary task of the predictor is to classify executable transcripts of the known species into found or yet-to-be-found, using the transcripts of the target species.
when a known transcript k has a match t in the list of transcripts of the target species, the predictor reports a pair of putative ortholog transcripts .
thus the output of the predictor is a list of transcript pairs, a list of yet-to-be-found transcripts, and a list of non-executable transcripts.
the controller
the controller outputs a list of putative human/mouse protein ortholog pairs whose underlying transcripts have the same splicing patterns.
for each gene, the controller constructs a list of candidate pairs, that are unique best reciprocal hits among protein isoforms of the gene with the same number of exons.
alignments are done using a semi-global exact alignment  <cit> , that does not penalize left or right trailing gaps, and are scored with the blosum <dig> scoring matrix.
for a candidate pair to be accepted, the alignment must preserve all exon junction positions, and be without gaps in the near vicinity  of junctions, as illustrated in fig.
<dig>  insertions and deletions within an exon are allowed.
the parameter v was experimentally determined to maximize the agreement between the predictor and the controller: there is no easy solution since gaps at or near exon junctions can be the result of insertions or deletions at the protein level, or evidence of alternative donor or acceptor sites.
fig.
<dig> alignment validation.
requirements for candidate protein sequences to be included in the controller list
finally, the right segment of the alignment of the first exon of each transcript, and the left segment of the alignment of the last exon, are tested for conservation by requiring non-negative blosum <dig> scores, over a maximum of  <dig> amino acids.
this last step is necessary because the position of the first  exon junction can be preserved even when the first  exons are different.
validation
we implemented a co-validation process between the two procedures, with the principle that both procedures should agree on their conclusions: disagreements are treated as errors.
the output of the predictor is compared to the controller list:
a transcript pair found by the predictor that also belong to the controller list is labeled splicing orthologs.
a transcript k classified as yet-to-be-found, for which the controller list does not contain any entry with transcript k, is labeled confirmed yet-to-be-found.
a transcript k classified as non-executable, for which the controller list does not contain any entry with transcript k is labeled confirmed species-specific.
predictions that are not confirmed are called unresolved.
there exist three types of unresolved predictions:
found by the predictor, but not contained in the controller list.
yet-to-be-found by the predictor, but contained in the controller list.
species-specific by the predictor, but contained in the controller list.
the number of each of these types of errors are reported and discussed in the next section.
RESULTS
all  <dig>  <dig> one-to-one ortholog gene pairs with at least one ccds isoform in human and in mouse were selected in the ensembl database  <cit> , as of 07/30/ <dig>  the date of ccds release  <dig>  of these, the predictor could analyze  <dig>  <dig> pairs of genes, totaling  <dig>  <dig> human and  <dig>  mouse transcripts with more than one exon.
the  <dig> genes rejected at this stage had homology problems such as duplicated or rearranged exons, or were not one-to-one orthologs.
the controller list computed on this set has  <dig>  <dig> pairs of putative protein ortholog pairs.
when transcripts from the same species are splicing orthologs, thus having the same splicing pattern, both procedures keep the isoform with the smallest ccds accession number.
the first experiment predicts mouse transcripts using human as the known species.
in this case, there are about  <dig> % more transcripts in the known species than in the target species.
the second experiment mirrors the first, with the known species  having fewer transcripts than the target species .
we also partitioned the complete set of predictions according to the ccds release in which the transcript of the known species first appears.
we looked at the complete set of predictions, and also at subsets ordered by their ccds release numbers.
the results are analyzed from three different perspectives: what is the difference between early and late predictions?
if we had made these predictions in the past how long would it have taken to confirm them?
what is the impact of the size of the predicting set of transcripts?
unresolved cases are discussed at the end of the section.
early cohort has good precision and recall values
the predictor and controller analyzed  <dig>  <dig> human transcripts, and agreed in  <dig>  <dig> cases .
table  <dig> gives detailed results of the resolved predictions of mouse transcripts using human transcripts.
precision and recall are computed using the definitions established in  <cit>  for gene structure prediction programs, called specificity and sensitivity in the original paper.
c
1
c
2
c
3
a
b
c
d
b/d
c/d
e
f
e/f
the precision value is the proportion of splicing orthologs over the total number of positive predictions, defined here as the sum of the number of splicing orthologs and and the number of confirmed yet-to-be-found transcripts.
for the first experiment we have a precision of /= <dig> .
the recall value is defined in the literature as the proportion of correctly identified isoforms with respect to the number of real isoforms, and is computed using the number of splicing orthologs over the number of known mouse transcripts, and is /= <dig> .
the same precision and recall computations were done after partitioning the predictions in three time-stamped cohorts, each ending at a ccds mouse release.
cohort c
<dig> contains transcripts known at release  <dig> ; cohort c
<dig> contains transcripts added to ccds between releases  <dig> and  <dig> ; cohort c
<dig> contains transcripts added to ccds between releases  <dig> and  <dig> .
note that the number of splicing orthologs is different for human and mouse cohorts, but they all add up to the total of  <dig>  <dig> transcript pairs: indeed, transcripts k and t of a pair  may belong to different cohorts.
for the first cohort, the precision and recall are, respectively,  <dig>  and  <dig> .
meaning that  <dig> % of the transcripts of the first mouse cohort are correctly predicted, which is more than three times the  <dig> % obtained by the purely computational methods tested in  <cit> .
precision and recall fall with subsequent younger cohorts, and this phenomenon is discussed in the next section.
<dig> to  <dig>  in cohort c
<dig>  this increase was expected, and is discussed further in the next section.
on the other hand, the increase of the proportion of human-specific transcripts, from  <dig>  in cohort c
<dig> to  <dig>  in cohort c
<dig> was unexpected.
possible explanations would be that the first cohort is dominated by highly expressed ubiquitous isoforms that were detected early; and/or that species-specific transcripts are less expressed, and are found later; and/or that recent sequencing experiments are more focussed on exploring the difference between mouse and human.
performances get better over time
here we analyze the evolution over time of the number of splicing orthologs, for a fixed number of positive predictions, which drives the evolution of precision values.
we restricted this analysis to genes that have a large number of isoforms, defined here as genes that have at least two different ccds isoforms in human and in mouse.
this is a subset of  <dig> positive predictions of the first experiment.
each cohort of this subset has respectively,  <dig>   <dig> and  <dig> positive predictions.
fig.
<dig> evolution of the number of splicing orthologs.
these curves show the growth, over the years, of the number of known splicing orthologs among the subset of orthologous genes that has at least two different isoforms for human and for mouse in the ccds release  <dig>  each data point corresponds to a ccds release of mouse transcripts: releases  <dig>   <dig>   <dig>   <dig>   <dig>   <dig> and  <dig>  the black curve shows the growth of the whole subset; the blue curve shows the growth of splicing orthologs whose human transcript was known in 2006; the red curve shows the growth of splicing orthologs whose human transcript was discovered between  <dig> and 2011; and the green curve shows the growth of splicing orthologs whose human transcript was discovered since 2011
the three colored curves of fig.
<dig> correspond to the partition of the set of predictions in cohorts c
<dig> to c
<dig>  they start at different heights, with cohort c
<dig> leading, but the fact that all curves have a similar shape, notably in the last half of the time interval plotted, was highly unexpected.
indeed, since the number of positive predictions is fixed for each cohort, the number of cumulative confirmed predictions must eventually become almost flat, with less and less predictions being confirmed.
we expected such a phenomenon, especially for cohort c
<dig> whose precision is currently  <dig> %, but the saturation effect is barely perceptible in this subset.
a similar shape in cumulative curves indicates a similar absolute growth over time.
this means that, for the last four releases, all cohorts have roughly the same number of new splicing orthologs.
given that all three have similar numbers of positive predictions, we expect the number of splicing orthologs to grow significantly in future ccds releases, at least for genes that have a large number of isoforms.
the number of known species transcripts influences both precision and recalltable  <dig> human transcripts as predicted by mouse transcripts
c
1
c
2
c
3
a
b
c
d
b/d
c/d
e
f
e/f
unresolved cases
unresolved cases are the results of disagreements between the predictor and the controller.
both procedures are experimental, in the sense that they depend on parameters and thresholds, and adjusting these values was done with the goal of maximizing agreement, which is  <dig>  % when human transcript is used to predict mouse transcripts.
there are  <dig> unresolved cases in total, since most of them appear in both experiments.
of them,  <dig> ortholog pairs are found by the predictor but not by the controller and often reflect a built-in stringency of the controller.
in order to distinguish subtle modifications in the regulation of alternative transcription, such as nagnag alternative transcripts  <cit>  that add or delete one amino acid next to an exon junction, the controller rejects every pair that contains deletions in near junctions.
using the nucleotide sequences, the predictor has an advantage in deciding whether such insertion or deletion is a true mutation.
the controller can also detect frame-shifts, where a single mutation can cause two dissimilar proteins in different species.
this is currently not verified by the predictor.
there are  <dig> instances of pairs included in the controller list for which one of the transcript is predicted yet-to-be-found or species-specific by the predictor.
they are due to pairs incorrectly included in the controller, or to the built-in stringencies of the predictor.
in the case of the controller, the main source of errors is the presence of conserved alternative exons of the same length, yielding two possible isoforms a and b in the human, and a
′ and b
′ in the mouse.
if the current state of the database contains a and b
′, the controller will pair them, but the predictor will correctly detect that the transcripts differ by one exon.
when three of the four isoforms are present, the unique best-hit property of the controller construction will resolve the conflict.
a second source of errors from the controller is the presence of very small first or last exons, sometimes as small as  <dig> nucleotide.
CONCLUSIONS
we gave a rigorous high level definition of splicing orthology and implemented it with a dual predictor/controller.
we applied the methods to the ccds human and mouse sets of isoforms and classified them into pairs of splicing orthologs.
we also showed that, for the prediction that could have been made eight years ago, human transcripts would have correctly predicted  <dig>  <dig> of the  <dig>  <dig>  thus  <dig> % of the known mouse transcripts at that time.
we showed that this percentage is a lower bound, since predictions for that cohort are still being confirmed with each new release of the ccds database, driven by the discovery of predicted isoforms of genes that have a large number of isoforms.
our list of  <dig>  <dig> confirmed splicing orthologs is available in additional file  <dig>  together with their common block representations.
it is intended as a benchmark for predictors, and as a data source for researchers interested in studying the conservation of alternative splicing.
additional files
additional file  <dig> predictoralgorithm.
description of the prediction algorithm.
additional file  <dig> results.tar.
compressed folder containing csv files listing the results.
additional file  <dig>
tables <dig>  data for figure  <dig>
declarations
publication costs for this article were funded by the inria associate team program.
this article has been published as part of bmc genomics vol  <dig> suppl  <dig>  2016: proceedings of the 14th annual research in computational molecular biology  comparative genomics satellite workshop: genomics.
the full contents of the supplement are available online at https://bmcgenomics.biomedcentral.com/articles/supplements/volume-17-supplement- <dig>
funding
ab is partially supported by canada nserc grant number 05729- <dig>  and by Équipes associées inria-frqnt grant number  <dig>  this research is supported by the inria associate team program.
availability of data and materials
all data generated or analyzed during this study are included in this published article and its supplementary information files.
authors’ contributions
ab, sb, ks and jsv conceived the study, modeled the validation process, and analyzed the experiments.
pg and ap wrote the draft code for constructing the gene models.
ab, sb, ks and jsv modeled the predictor.
sb and jsv wrote the final code for constructing the gene models.
sb and jsv wrote the code for the predictor.
ab and ks modeled the controller.
ks wrote the code for the controller.
ab wrote the manuscript.
sb, ks and jsv revised the manuscript.
all authors read and approved the final manuscript.
competing interests
the authors declare that they have no competing interests.
consent for publication
not applicable.
ethics approval and consent to participate
not applicable.

 the knowledge of all protein isoforms that may be expressed by a gene is fundamental. recently, several computational methods have been proposed for transcriptome reconstruction that use rna-seq data for exon identification, and expression levels data for transcript assembly. while exon identification performs quite well, transcript assembly remains difficult for complex transcriptomes. as shown in, the best-performing computational methods identified at most % of spliced protein-coding transcripts from h. sapiens, and transcript detection remains low even with very high sequencing coverage, leading the authors to conclude that improved results would have to wait for technological advances. those findings were confirmed by several other studies, that include methods recently developed such as stringtie. given these limitations of ab initio transcript prediction, it is natural to investigate assisted transcriptome reconstruction, in which the knowledge of transcript structures is transferred from one species to another: since transcripts provide a road-map of the successive links between exons, it should be possible to distinguish transcripts that may be expressed from those that may not, by analyzing the sequence of the target genes. an essential problem is the assessment of the predictions. confirmation of the prediction that “transcript t may be expressed in the target species” is only possible through experimental validation, which can be a long and costly process, since many transcripts are detectable only in specific cells, under specific conditions. gene predictors that make explicit use of external evidence to predict eukaryotic gene structure have been around since the turn of the century, and with the discovery of alternative transcripts in recent years, hundreds of thousands of predicted isoforms are now available in databases. for example, as of june felis catus had predicted isoforms in the refseq database, of which are confirmed, and canis lupus familiaris had predicted isoforms of which are confirmed. these predictions were made by the gnomon predictor, described in an unpublished paper, and whose performance is unknown, especially for predictions that rely exclusively on external evidence. in order to evaluate the predictive power of transcript annotation transfer, it is necessary to identify splicing orthologs, loosely defined as transcripts from ortholog genes with similar splicing patterns. zambelli et al., introduced the concept and gave three possible definitions that yielded estimates ranging from to % for the percentage of human transcripts that have a splicing ortholog in mouse. this wide range of estimates is an indicator of how the concept is still a work-in-progress. a subsequent paper of fong et al. simplifies the definition of splicing orthologs as “…all protein-coding exons in the two proteins can be paired with % overlap in lengths of both exons.”. this is clearly an over-simplification that ignores, among other things, nearby alternative donor or acceptor splice sites. in this study, we revisit the concept of splicing orthology and we give a comprehensive assessment of the performance of assisted transcriptome reconstruction using the human to predict mouse, and vice versa. methods our experimental set-up is based on a procedural definition of splicing orthologs that is concurrently implemented by two procedures. our predictor uses transcripts from a known species to predict transcripts in a target species, and for evaluation purposes, identifies putative pairs of orthologous transcript isoforms based solely on nucleotide sequence evidence. our controller identifies putative pairs of orthologous protein isoforms between human and mouse using amino acid sequences and positions of exon junctions. splicing orthologs transcripts from orthologous genes with differing splicing patterns could have easily identifiable differences in the number or identity of their exons. however, defining splicing orthology can be more difficult due to the presence of alternative donor and acceptor splice sites, where exons are elongated or truncated, often by a very few nucleotides. within a gene, these exon isoforms overlap and have different exon-intron border. across species, we define orthologous exon isoforms as orthologous exons that have conserved exon-intron borders. using this concept, splicing orthologs are defined as transcripts of orthologous genes, whose exons are orthologous exon isoforms that appear in the same order in each gene, and that code for similar proteins. in the ideal situation where all splicing orthologs are conserved across species in one-to-one orthologous gene pairs, all exons — with their flanking intronic sequences — should be best reciprocal hits, and the alignments should preserve exon-intron borders. each protein would have a perfect match that should be a unique best reciprocal hit, the alignments should preserve the positions of exon junctions, and be without gaps in the near vicinity of junctions. the predictor given a pair of orthologous genes, the first task of the predictor is to establish a common reference sequence for the set of all transcripts in both the human and mouse. our solution is based on the concept of blocks, which are alignments of human and mouse exon segments that are either contained in, or disjoint from, each human or mouse transcript. each block has a label and can be flanked by signals: start and stop codons, donor and acceptor splice sites. using block labels, the symbols ‘’ to indicate start and stop codons, and ‘<’ and ‘>’ to indicate the intronic part of donor and acceptor splice sites, each transcript can be represented as a string. the gene model of a set of transcripts is the ordered string of its blocks and signals, and a donor block is a block that has a donor signal at its 3’ end in the gene model. figure presents an example with the crem and crem orthologous genes of human and mouse. fig. gene models and transcripts. in the ccds database, as of june there were crem human and crem mouse transcripts with unique splicing patterns. the common reference sequence has blocks labeled from a to u. the human gene model is at the top of the figure, followed by the sub-sequence of blocks and signals found in the mouse. the block representation and ccds number is given for each transcript. note that block o does not exist in the mouse gene, and block f is not in the human model, since block f is not found in any known human transcript. of the human transcripts, are executable – meaning that they could be expressed –, and three are not because they use block o. the executable transcripts are further classified as found and paired with a mouse transcript, or yet-to-be-found. all these predictions are confirmed by the controller. since mouse transcripts are correctly identified, and mouse transcripts are currently known, the predictor successfully identifies 5/ or % of the mouse transcripts. as more mouse transcripts are discovered, this proportion may increase with future releases of the ccds database to ensure internal exons best reciprocal hit property, every human and mouse sequence in a block is blasted, in the orthologous gene, using flanking genomic sequences of length further details on the predictor algorithm can be found in additional file the fact that all blocks can be found by a blast search allows the predictor to simulate a pure predicting mode, that does not have any prior knowledge of the target species transcripts. given a gene model m, the predictor reports all blocks and signals that were found in the target gene, and reports its successes as a substring s of the gene model. this paper predicts in flexible mode, where orthologous transcripts must have the same splicing pattern, but the first and last exons are only required to have orthologous internal exon-intron borders. a transcript is said to be executable by the target gene if its donor and acceptor blocks are in s. the primary task of the predictor is to classify transcripts as executable or not executable. in the simulation context of this paper, a secondary task of the predictor is to classify executable transcripts of the known species into found or yet-to-be-found, using the transcripts of the target species. when a known transcript k has a match t in the list of transcripts of the target species, the predictor reports a pair of putative ortholog transcripts. thus the output of the predictor is a list of transcript pairs, a list of yet-to-be-found transcripts, and a list of non-executable transcripts. the controller the controller outputs a list of putative human/mouse protein ortholog pairs whose underlying transcripts have the same splicing patterns. for each gene, the controller constructs a list of candidate pairs, that are unique best reciprocal hits among protein isoforms of the gene with the same number of exons. alignments are done using a semi-global exact alignment, that does not penalize left or right trailing gaps, and are scored with the blosum scoring matrix. for a candidate pair to be accepted, the alignment must preserve all exon junction positions, and be without gaps in the near vicinity of junctions, as illustrated in fig. insertions and deletions within an exon are allowed. the parameter v was experimentally determined to maximize the agreement between the predictor and the controller: there is no easy solution since gaps at or near exon junctions can be the result of insertions or deletions at the protein level, or evidence of alternative donor or acceptor sites. fig. alignment validation. requirements for candidate protein sequences to be included in the controller list finally, the right segment of the alignment of the first exon of each transcript, and the left segment of the alignment of the last exon, are tested for conservation by requiring non-negative blosum scores, over a maximum of amino acids. this last step is necessary because the position of the first exon junction can be preserved even when the first exons are different. validation we implemented a co-validation process between the two procedures, with the principle that both procedures should agree on their conclusions: disagreements are treated as errors. the output of the predictor is compared to the controller list: a transcript pair found by the predictor that also belong to the controller list is labeled splicing orthologs. a transcript k classified as yet-to-be-found, for which the controller list does not contain any entry with transcript k, is labeled confirmed yet-to-be-found. a transcript k classified as non-executable, for which the controller list does not contain any entry with transcript k is labeled confirmed species-specific. predictions that are not confirmed are called unresolved. there exist three types of unresolved predictions: found by the predictor, but not contained in the controller list. yet-to-be-found by the predictor, but contained in the controller list. species-specific by the predictor, but contained in the controller list. the number of each of these types of errors are reported and discussed in the next section.  all one-to-one ortholog gene pairs with at least one ccds isoform in human and in mouse were selected in the ensembl database, as of 07/30/ the date of ccds release of these, the predictor could analyze pairs of genes, totaling human and mouse transcripts with more than one exon. the genes rejected at this stage had homology problems such as duplicated or rearranged exons, or were not one-to-one orthologs. the controller list computed on this set has pairs of putative protein ortholog pairs. when transcripts from the same species are splicing orthologs, thus having the same splicing pattern, both procedures keep the isoform with the smallest ccds accession number. the first experiment predicts mouse transcripts using human as the known species. in this case, there are about % more transcripts in the known species than in the target species. the second experiment mirrors the first, with the known species having fewer transcripts than the target species. we also partitioned the complete set of predictions according to the ccds release in which the transcript of the known species first appears. we looked at the complete set of predictions, and also at subsets ordered by their ccds release numbers. the results are analyzed from three different perspectives: what is the difference between early and late predictions? if we had made these predictions in the past how long would it have taken to confirm them? what is the impact of the size of the predicting set of transcripts? unresolved cases are discussed at the end of the section. early cohort has good precision and recall values the predictor and controller analyzed human transcripts, and agreed in cases. table gives detailed results of the resolved predictions of mouse transcripts using human transcripts. precision and recall are computed using the definitions established in for gene structure prediction programs, called specificity and sensitivity in the original paper. c 1 c 2 c 3 a b c d b/d c/d e f e/f the precision value is the proportion of splicing orthologs over the total number of positive predictions, defined here as the sum of the number of splicing orthologs and and the number of confirmed yet-to-be-found transcripts. for the first experiment we have a precision of /=. the recall value is defined in the literature as the proportion of correctly identified isoforms with respect to the number of real isoforms, and is computed using the number of splicing orthologs over the number of known mouse transcripts, and is /=. the same precision and recall computations were done after partitioning the predictions in three time-stamped cohorts, each ending at a ccds mouse release. cohort c contains transcripts known at release; cohort c contains transcripts added to ccds between releases and; cohort c contains transcripts added to ccds between releases and. note that the number of splicing orthologs is different for human and mouse cohorts, but they all add up to the total of transcript pairs: indeed, transcripts k and t of a pair may belong to different cohorts. for the first cohort, the precision and recall are, respectively, and. meaning that % of the transcripts of the first mouse cohort are correctly predicted, which is more than three times the % obtained by the purely computational methods tested in. precision and recall fall with subsequent younger cohorts, and this phenomenon is discussed in the next section. to in cohort c this increase was expected, and is discussed further in the next section. on the other hand, the increase of the proportion of human-specific transcripts, from in cohort c to in cohort c was unexpected. possible explanations would be that the first cohort is dominated by highly expressed ubiquitous isoforms that were detected early; and/or that species-specific transcripts are less expressed, and are found later; and/or that recent sequencing experiments are more focussed on exploring the difference between mouse and human. performances get better over time here we analyze the evolution over time of the number of splicing orthologs, for a fixed number of positive predictions, which drives the evolution of precision values. we restricted this analysis to genes that have a large number of isoforms, defined here as genes that have at least two different ccds isoforms in human and in mouse. this is a subset of positive predictions of the first experiment. each cohort of this subset has respectively, and positive predictions. fig. evolution of the number of splicing orthologs. these curves show the growth, over the years, of the number of known splicing orthologs among the subset of orthologous genes that has at least two different isoforms for human and for mouse in the ccds release each data point corresponds to a ccds release of mouse transcripts: releases and the black curve shows the growth of the whole subset; the blue curve shows the growth of splicing orthologs whose human transcript was known in 2006; the red curve shows the growth of splicing orthologs whose human transcript was discovered between and 2011; and the green curve shows the growth of splicing orthologs whose human transcript was discovered since 2011 the three colored curves of fig. correspond to the partition of the set of predictions in cohorts c to c they start at different heights, with cohort c leading, but the fact that all curves have a similar shape, notably in the last half of the time interval plotted, was highly unexpected. indeed, since the number of positive predictions is fixed for each cohort, the number of cumulative confirmed predictions must eventually become almost flat, with less and less predictions being confirmed. we expected such a phenomenon, especially for cohort c whose precision is currently %, but the saturation effect is barely perceptible in this subset. a similar shape in cumulative curves indicates a similar absolute growth over time. this means that, for the last four releases, all cohorts have roughly the same number of new splicing orthologs. given that all three have similar numbers of positive predictions, we expect the number of splicing orthologs to grow significantly in future ccds releases, at least for genes that have a large number of isoforms. the number of known species transcripts influences both precision and recalltable human transcripts as predicted by mouse transcripts c 1 c 2 c 3 a b c d b/d c/d e f e/f unresolved cases unresolved cases are the results of disagreements between the predictor and the controller. both procedures are experimental, in the sense that they depend on parameters and thresholds, and adjusting these values was done with the goal of maximizing agreement, which is % when human transcript is used to predict mouse transcripts. there are unresolved cases in total, since most of them appear in both experiments. of them, ortholog pairs are found by the predictor but not by the controller and often reflect a built-in stringency of the controller. in order to distinguish subtle modifications in the regulation of alternative transcription, such as nagnag alternative transcripts that add or delete one amino acid next to an exon junction, the controller rejects every pair that contains deletions in near junctions. using the nucleotide sequences, the predictor has an advantage in deciding whether such insertion or deletion is a true mutation. the controller can also detect frame-shifts, where a single mutation can cause two dissimilar proteins in different species. this is currently not verified by the predictor. there are instances of pairs included in the controller list for which one of the transcript is predicted yet-to-be-found or species-specific by the predictor. they are due to pairs incorrectly included in the controller, or to the built-in stringencies of the predictor. in the case of the controller, the main source of errors is the presence of conserved alternative exons of the same length, yielding two possible isoforms a and b in the human, and a ′ and b ′ in the mouse. if the current state of the database contains a and b ′, the controller will pair them, but the predictor will correctly detect that the transcripts differ by one exon. when three of the four isoforms are present, the unique best-hit property of the controller construction will resolve the conflict. a second source of errors from the controller is the presence of very small first or last exons, sometimes as small as nucleotide.  we gave a rigorous high level definition of splicing orthology and implemented it with a dual predictor/controller. we applied the methods to the ccds human and mouse sets of isoforms and classified them into pairs of splicing orthologs. we also showed that, for the prediction that could have been made eight years ago, human transcripts would have correctly predicted of the thus % of the known mouse transcripts at that time. we showed that this percentage is a lower bound, since predictions for that cohort are still being confirmed with each new release of the ccds database, driven by the discovery of predicted isoforms of genes that have a large number of isoforms. our list of confirmed splicing orthologs is available in additional file together with their common block representations. it is intended as a benchmark for predictors, and as a data source for researchers interested in studying the conservation of alternative splicing. additional files additional file predictoralgorithm. description of the prediction algorithm. additional file results.tar. compressed folder containing csv files listing the results. additional file tables data for figure declarations publication costs for this article were funded by the inria associate team program. this article has been published as part of bmc genomics vol suppl 2016: proceedings of the 14th annual research in computational molecular biology comparative genomics satellite workshop: genomics. the full contents of the supplement are available online at https://bmcgenomics.biomedcentral.com/articles/supplements/volume-17-supplement- funding ab is partially supported by canada nserc grant number 05729- and by Équipes associées inria-frqnt grant number this research is supported by the inria associate team program. availability of data and materials all data generated or analyzed during this study are included in this published article and its supplementary information files. authors’ contributions ab, sb, ks and jsv conceived the study, modeled the validation process, and analyzed the experiments. pg and ap wrote the draft code for constructing the gene models. ab, sb, ks and jsv modeled the predictor. sb and jsv wrote the final code for constructing the gene models. sb and jsv wrote the code for the predictor. ab and ks modeled the controller. ks wrote the code for the controller. ab wrote the manuscript. sb, ks and jsv revised the manuscript. all authors read and approved the final manuscript. competing interests the authors declare that they have no competing interests. consent for publication not applicable. ethics approval and consent to participate not applicable.
most evidence-based gene predictors use transcripts from other species to annotate a genome, but the predictive power of procedures that use exclusively transcripts from external species has never been quantified. we propose a rigorous procedural definition of splicing orthologs, based on the identification of all ortholog pairs of splicing sites in the nucleotide sequences, and alignments at the protein level. using our definition, we compared human transcripts and mouse transcripts from the highly curated ccds database, and identified splicing orthologs. transcriptome reconstruction, defined as the identification of all protein isoforms that may be expressed by a gene, is a notably difficult computational task. while waiting for algorithms and sequencing techniques to improve — as has been strongly suggested in the literature — it is important to evaluate assisted transcriptome prediction; this is the question of how alternative transcription in one species performs as a predictor of protein isoforms in another relatively close species. in prediction mode, we show that human transcripts can be used to infer over % of mouse protein isoforms. transcriptome predictionsplicing orthologseukaryotes14th annual research in computational molecular biology comparative genomics satellite workshop recomb-cgmontreal, canada 11- october http://www.crm.umontreal.ca/2016/genomics16/index_e.phpissue-copyright-statement© the author 2016
 transcriptome reconstruction, defined as the identification of all protein isoforms that may be expressed by a gene, is a notably difficult computational task. with real data, the best methods based on rna-seq data identify barely % of the expressed transcripts. while waiting for algorithms and sequencing techniques to improve — as has been strongly suggested in the literature — it is important to evaluate assisted transcriptome prediction; this is the question of how alternative transcription in one species performs as a predictor of protein isoforms in another relatively close species. most evidence-based gene predictors use transcripts from other species to annotate a genome, but the predictive power of procedures that use exclusively transcripts from external species has never been quantified. the cornerstone of such an evaluation is the correct identification of pairs of transcripts with the same splicing patterns, called splicing orthologs.  we propose a rigorous procedural definition of splicing orthologs, based on the identification of all ortholog pairs of splicing sites in the nucleotide sequences, and alignments at the protein level. using our definition, we compared human transcripts and mouse transcripts from the highly curated ccds database, and identified splicing orthologs. in prediction mode, we show that human transcripts can be used to infer over % of mouse protein isoforms. when restricting the predictions to transcripts known eight years ago, the percentage grows to %. using ccds timestamped releases, we also analyze the evolution of the number of splicing orthologs over the last decade.  alternative splicing is now recognized to play a major role in the protein diversity of eukaryotic organisms, but definitions of spliced isoform orthologs are still approximate. here we propose a definition adapted to the subtle variations of conserved alternative splicing sites, and use it to validate numerous accurate orthologous isoform predictions. electronic supplementary material the online version of this article contains supplementary material, which is available to authorized users. keywords transcriptome predictionsplicing orthologseukaryotes14th annual research in computational molecular biology comparative genomics satellite workshop recomb-cgmontreal, canada 11- october http://www.crm.umontreal.ca/2016/genomics16/index_e.phpissue-copyright-statement© the author 2016 