# On the evaluation of the fidelity of supervised classifiers in the prediction of chimeric RNAs

Sacha Beaumeunier, Jérôme Audoux, Anthony Boureux, Florence Ruffle, Thérèse Commes, Nicolas Philippe, Ronnie Alves

<figure><img src="image_1.png" /></figure>

## To cite this version:

Sacha Beaumeunier, Jérôme Audoux, Anthony Boureux, Florence Ruffle, Thérèse Commes, et al.. On the evaluation of the fidelity of supervised classifiers in the prediction of chimeric RNAs. BioData Mining, 2016, 9 (1), pp.34. 10.1186/s13040-016-0112-6. inserm-01390470

HAL Id: inserm-01390470

https://inserm.hal.science/inserm-01390470

Submitted on 2 Nov 2016

HAL is a multi-disciplinary open access archive for the deposit and dissemination of scientific research documents, whether they are published or not. The documents may come from teaching and research institutions in France or abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est destinée au dépôt et à la diffusion de documents scientifiques de niveau recherche, publiés ou non, émanant des établissements d'enseignement et de recherche français ou étrangers, des laboratoires publics ou privés.

<figure><img src="image_2.png" /></figure>

Distributed under a Creative Commons Attribution 4.0 International License
# Anatomy of a chimeric junction identified with CRAC algorithm

**Chimeric event**
Chromosome A and B are translocated together. A chimeric transcript is produced from the fusion of two genes, one from chromosome A and the other from chromosome B.

**chimeric read**
A read overlapping the chimeric junction.

**CRAC k-mer analysis**
CRAC procedure to analyse reads relies on the analysis of k-mer profiles.

**localization profile**
Number of time each k-mer is found on the reference genome.

**support profile**
Number of time each k-mer is found in the input dataset.

**score break length**
Measure of distance between the observed break length and the theoretical break length.

**score is duplicated**
If the k-mer location around the break is ambiguous.

**score coefficient variation**
Coefficient variation of the support profile.

<figure><img src="image_1.png" /></figure>

## ATGCTGTAGGGCTGATCGAC

<figure><img src="image_2.png" /></figure>

**break**

Def: interruption of k-mer localization. One false positive location in green.

<figure><img src="image_3.png" /></figure>

<figure><img src="image_4.png" /></figure>

$$ \text{score\_break\_length} = \text{observed\_break\_length}(\text{observed\_break\_length}) / (k-1) = 1 $$

$$ \text{score\_is\_duplicate} = \text{true} $$

<figure><img src="image_5.png" /></figure>

<figure><img src="image_6.png" /></figure>

$$ \text{score\_coeff\_var} = \text{std\_dev(support)} / \text{mean(support)} $$
<figure><img src="image_3.png" /></figure>

# On the evaluation of the fidelity of supervised classifiers in the prediction of chimeric RNAs

Sacha Beaumeunier¹,², Jérôme Audoux¹,², Anthony Boureux¹,², Florence Ruffle¹,², Thérèse Commes¹,²*, Nicolas Philippe¹,² and Ronnie Alves¹,²,³,⁴,⁵*

*Correspondence:

therese.commes@inserm.fr;
ronnie.alves@itv.org

¹Institut de Médecine Régénératrice et de Biothérapie, INSERM U1183, CHU Montpellier, Montpellier, France

²Institut de Biologie Computationnelle, Université Montpellier, Montpellier, France
Full list of author information is available at the end of the article

## Abstract

**Background:** High-throughput sequencing technology and bioinformatics have identified chimeric RNAs (chRNAs), raising the possibility of chRNAs expressing particularly in diseases can be used as potential biomarkers in both diagnosis and prognosis.

**Results:** The task of discriminating true chRNAs from the false ones poses an interesting Machine Learning (ML) challenge. First of all, the sequencing data may contain false reads due to technical artifacts and during the analysis process, bioinformatics tools may generate false positives due to methodological biases. Moreover, if we succeed to have a proper set of observations (enough sequencing data) about true chRNAs, chances are that the devised model can not be able to generalize beyond it. Like any other machine learning problem, the first big issue is finding the good data to build models. As far as we were concerned, there is no common benchmark data available for chRNAs detection. The definition of a classification baseline is lacking in the related literature too. In this work we are moving towards benchmark data and an evaluation of the fidelity of supervised classifiers in the prediction of chRNAs.

**Conclusions:** We proposed a modelization strategy that can be used to increase the tools performances in context of chRNA classification based on a simulated data generator, that permit to continuously integrate new complex chimeric events. The pipeline incorporated a genome mutation process and simulated RNA-seq data. The reads within distinct depth were aligned and analysed by CRAC that integrates genomic location and local coverage, allowing biological predictions at the read scale. Additionally, these reads were functionally annotated and aggregated to form chRNAs events, making it possible to evaluate ML methods (classifiers) performance in both levels of reads and events. Ensemble learning strategies demonstrated to be more robust to this classification problem, providing an average AUC performance of 95 % (ACC=94 %, Kappa=0.87 %). The resulting classification models were also tested on real RNA-seq data from a set of twenty-seven patients with acute myeloid leukemia (AML).

**Keywords:** Chimeric RNAs, Transcriptomics, Classification, Ensemble learning, Data simulation
## Background

A chimeric RNA (chRNA) is a RNA molecule that is made of two or more pieces of RNAs from different loci. Chimeric RNAs are formed from genetic rearrangements like translocation, inversion, deletion or copy number variation. Other studies have highlighted that many other chRNA could be also generated from post-transcriptional mechanisms such as read-through and splicing, transcription of short homologous sequence slippage or from trans-splicing [1].

High-throughput sequencing technology and bioinformatics have identified chRNA, raising the possibility of chRNAs expressing particularly in diseases can be used as potential biomarkers in both diagnosis and prognosis. However, only a limited number of chimeric transcripts and their associated protein products have been characterised to date, most of them resulting from chromosomal translocation and associated with cancer [2, 3]. For instance the well-known gene fusion in chronic myelogenous leukemia leading to an mRNA transcript that encompass the 5' end of the *BCR* gene and the 3' end of the *ABL* gene, producing the chRNA BCR-ABL.

Most protocols used to identify chimeric transcripts rely on a reverse transcription step and the reverse transcriptase is known to switch templates, thus creating chimeric artifacts in vitro [4]. Thus, it remains unclear what proportion of putative chimeric transcripts are true, and of these how many are translated. With the rapid evolution of genomic research, next generation sequencing (NGS) has become widely available. Sequences in databases or sequences directly from RNA-seq often contain information that leads to false results when they are analysed with bioinformatics approaches. This is partially due to the fact that the first stage of the available bioinformatic tools for finding chRNA basically rely on mapping, where reads are aligned with respect to the reference genome used by the algorithm. Additionally, the length and depth of sequencing might be inadequate for covering all exon junctions, and given that repeated sequences are present, clearly, the alignment of sequences to specific regions is highly biased to sequencing. Next to mapping, a set of filters based on several technological or biological assumptions are applied to reduce the set of putative chRNAs.

Many tools have been developed to detect chRNAs [5], though many complications arise when trying to provide a complete benchmark. Each one of these tools have particular constraints regarding installation, parametrization, usability, and database(s) or software(s) dependencies. Liu et al. [6] provided an overall evaluation of thirteen strategies, suggesting the combination of three of them as a robust solution. In fact, there is a trend to explore combination of algorithms to boost the analysis of chimera detection. However, a good combination of software requires proper criteria of selections based on metrics according to the biological question in order to choose the most suitable ones. In fact, the basic information shared in common by all these softwares is the initial sequenced reads which is distinctively explored by each of them. In other words, each software integrates its own algorithm and different external features and database to define proper margins or “rule(s)” to classify potential chRNAs. If one is able to make such abstraction (or rules) from the reads, it might be an interesting alternative to directly explore models generated from classifiers. In this work, we propose a different angle of discussion, model-oriented rather than comparing software for chRNAs detection which are not necessarily comparable because there are questions-oriented.
The task of discriminating true chRNA from the false ones poses an interesting Machine
Learning (ML) challenge. First of all, the sequencing data is highly biased and thus predict-
ing the real signal from the noise can be a hard task. Furthermore, even if we succeed to
have a proper set of observations (enough sequencing data) about true chRNAs, chances
are that the devised model can not be able to generalize beyond it.

Models generalization is the “holy grail” of machine learning. ML methods must be able to see beyond the data observed in order to generalize beyond it. The famous “no free lunch” theorem of Wolpert says that no learner can beat a random guessing over all possible functions to be learned. Ensemble strategies overcome the no-free-lunch dilemma by combining the outputs of many classifiers assuming that each classifier performs well in particular domains while being sub-optimal in others. Machine learning and ensemble based systems has been applied successfully in several data applications [7, 8]. The main idea behind the ensemble strategy is simple, “rely on the feedback of a bunch of specialists”. Thus, a combination of several models might better estimate the margins boundaries in the hypothesis space, reducing the bias and variance errors. Bias is a learner’s tendency to consistently learn the same wrong thing. Variance is the tendency to learn random things irrespective of the real signal [9]. Empirical and theoretical evidence show that some ensemble learning techniques like bagging act as a variance reduction mechanism, i.e., they reduce the variance component of the error. Boosting strategies reduce both the bias and variance parts of the error. It sounds that the bias error is usually reduced in the early iterations, while variance error decreases in later ones. Indeed, the key component behind ensemble success is the concept of classifier diversity [7]. Classifier diversity can be achieved in several ways. The most known approach is to use different training datasets to train individual classifiers. Such datasets are often obtained through resampling techniques, such as bootstrapping or bagging, where training data subsets are drawn randomly, usually with replacement, from the entire training data.

Most of the available gene fusion-finders methods [5, 10] do not explore the poten-
tials of applying ML techniques to this classification problem. As far as we are aware
of, only two gene fusion-finders, namely EricScript [11] and deFuse [12], have explored
AdaBoost. AdaBoost generates a set of hypotheses and combines them through weighted
majority voting of the classes predicted by the individual hypotheses. The hypotheses
are generated by training a weak classifier, using instances drawn from an iteratively
updated distribution of the training data. This distribution update ensures that instances
misclassified by the previous classifier are more likely to be included in the train-
ing data of the next classifier. Hence, consecutive classifiers’ training data are geared
towards increasingly hard-to-classify instances. As fas as we are concerned, there is
no further literature in finding chRNAs using bagging-oriented strategies. Bagging is
one of the pioneers in ensemble learning. Diversity in bagging is obtained by using
bootstrapped replicas of the training data: different training data subsets are randomly
drawn, with replacement, from the entire training data. Each training data subset is
employed to train a distinct classifier of the same type. Individual classifiers are then
combined by taking a majority vote of their decisions. For any given instance, the class
chosen by most classifiers is the ensemble decision. Random Forest (RF) is a varia-
tion of the bagging algorithm, and it can be created from individual decision trees,
whose certain training parameters vary randomly. Such parameters can be bootstrapped
replicas of the training data, as in bagging, but they can also be different feature
<figure><img src="image_4.png" /></figure>

of information to extract quantitative features that define chimeric junctions at the read “level” and therefore, suitable for the development of ML models [see Additional file 1].

A total of five *Human* (GRCh38 assembly) mutated genomes were generated. They were used in a progressive sampling scheme for benchmarking several supervised classifiers. For each mutated genome, 40 millions of paired-end reads with read length of 100bp were generated. CRAC selected all reads and labelled them according to their associated class. Next, five features emerging from CRAC’s algorithm were extracted as follows: i) score_break_length, it describes the size of the break (adjacent unmapped k-mers), particularly those presenting sizes higher than the average; ii) score_nb_merge, it evaluates if the break corresponding to the chimera has a number of merges < to the related parameter; iii) score_break-variance, it checks the inter-quartile range of support values in the break; iv) score_is_duplicate, it checks if the chimera is ambiguous; v) score_has_repeat, it evaluates if the break is in a repeated region. Given that several chRNAs reads can map to one particular chimeric event, some of those reads may hold the same values for the previously described set of features. Therefore, only unique reads per chRNAs are taken into account in the model building phase. Table 1 summarises the chimeric reads distribution within the progressive sampling runs. The first run (r1) has one transcriptome and, progressively, one more were added to the sample up to having all five mutated transcriptomes in the fifth run (r5) (Fig. 2a).

## Supervised learning strategies

The challenge of finding chRNAs can be seen as a binary classification problem (positive class: true chimera and negative class: false chimera) [12]. Once having the mutated genomes we can move towards the evaluation of several supervised learning strategies

<table><caption>**Table 1** The chRNAs reads distribution within the progressive sampling</caption><thead><tr><th rowspan="2"></th><th colspan="2">(A)ll reads</th><th colspan="2">(U)nique reads</th><th colspan="2">% U/A</th></tr><tr><th>True</th><th>False</th><th>True</th><th>False</th><th>T</th><th>F</th></tr></thead><tbody><tr><td>r1</td><td>5253</td><td>1462</td><td>118</td><td>316</td><td>2.25</td><td>21.6</td></tr><tr><td>r2</td><td>7169</td><td>1761</td><td>145</td><td>415</td><td>2.02</td><td>23.6</td></tr><tr><td>r3</td><td>8563</td><td>2377</td><td>163</td><td>560</td><td>1.90</td><td>23.6</td></tr><tr><td>r4</td><td>11238</td><td>3795</td><td>259</td><td>836</td><td>2.30</td><td>22.03</td></tr><tr><td>r5</td><td>15016</td><td>4988</td><td>460</td><td>995</td><td>3.06</td><td>19.95</td></tr></tbody></table>
<figure><img src="image_5.png" /></figure>

for mining chimeras. The comparison study takes into account distinct supervised learning strategies, namely, one lazy learner (KNN: K-Nearest Neighbors), one eager-learner (SVM: Support Vector Machines) and two ensemble learners (RF: Random Forest and GBM: Gradient Boosting Machines). Each ML strategy has a particular way to generalize the search space.

Nearest-neighbor classifiers are based on learning by analogy, by comparing a given test instance with training instances that are similar to it. Support Vector Machines uses a linear model to implement nonlinear class boundaries. SVM transform the input using a nonlinear mapping, thus, turning the instance space into a new space. Random
forest is a well-known ensemble approach for classification tasks proposed by Breiman. Its basis comes from the combination of tree-structured classifiers with the randomness and robustness provided by bagging and random feature selection. Gradient boosting machines consecutively fits new models to provide more accurate estimate of the response variable. Thus, new base learners are built being maximally correlated with the negative gradient of the loss function associated with the entire ensemble. The loss function applied can be arbitrary, so as an example, if the error function is squared-error loss, the learning procedure would result in consecutive error fitting.

## Building models

Model building and evaluation was carried out with the caret R package [16]. We used the built-in *tune()* function for resampling, tuning and optimisation of all selected classifiers. We have also adopted a progressive sampling strategy to better understand the impacts of adding more observations to the classifiers performance. We first started using one simulated data set (incorporating a mutation process), and next more data were added sequentially up to having all five genomes. To each run we have set aside one third of the data for the test phase while the remaining two third were used for model building. Training were performed with a 10-fold cross validation scheme along with a tuning grid to each ML technique. Additionally, given that the data split (training/test) were random, we repeated each run three times to check models stability, providing fifteen performance points. As an example r11, r12, and r13 are subsets of the r1 data set (Fig. 2b and d). Models' performance were average to each run (Fig. 2c).

## Results

### Towards a classification baseline at the "read" level

Give the absence of a classification baseline for the problem of finding chRNAs, let us assume, essentially, the performance results we got from the first run (r1) of KNN as the initial baseline. The baseline accuracy is also known as the null rate. In machine learning it is also a good practice to start with simple classifiers rather than sophisticated ones [9]. The KNN provided average AUC performance of 88 % (ACC=87 %, Kappa=0.71). Kappa measures how closely the instances labeled by the classifiers matched the data labeled as ground truth, controlling for the ACC of a random classifier as measured by the expected accuracy. Thus, the Kappa for one classifier is properly comparable to others Kappa's classifiers for the same classification problem. We will make use of Kappa as the unbiased metric to compare the selected classifiers along all the progressive sampling strategy.

The general idea behind of a progressive sampling is straightforward. We start with some chRNA reads arising from a individual genome and then, iteratively, build a model, evaluate its performance, and acquire additional chRNA reads from another individual genome. Initially, we were planing to use a simple stop criteria along sampling until getting a performance plateau. Thus, we might be able to stop having more data in the third run (Fig. 2b). However, we decided to have two more additional runs, and, as it can be observed, all models loosed generalization power in the fourth run. All simulations have the same genome simulator parameters and mutation profile and expression profiles, though "randomness" plays quite distinctly to the fourth ones. It might be partially due to the fact that the associated data sets are unbalanced but, every run has about the same
proportion of true and false chRNAs (see Table 1). Another hypothesis is that, perhaps, mutations in the fourth genome is biased to patterns in alternative splicing.

Ensemble learning strategies were more sensitive to this “random perturbation” added by the fourth genome. The feature score_has_repeat does not bring any discriminative power, being eliminated from the set of predictors. Additionally, we have worked on improving performance of classifiers by introducing three more features, enhancing the break profile (Fig. 3), namely: i) coefficient variation, the coefficient of variation of the support of the k-mers over the break; ii) mean_amplitude, mean amplitude of the break support; and iii) coefficient_dispersion_interquartile, $Q3 - Q1/Q3 + Q1$. Finally, the best performance results were obtaining by combining the features score_is_duplex, coefficient_variation, and score_break_length (see Table 2).

It is important to mention that a grid of optimisation were applied to all supervised classifiers. Surprisingly, the SVM provided one of the most unstable results, even when compared to KNN. We hypothesise that the search space might be a set of Boolean functions, and given that both RF and GBM employ tree stumps, random perturbations are handled efficiently by ensemble of Booleans rules generated by these classifiers. Thus, in this particular study, indeed, a more powerful learner is better than a less powerful one. Ensemble learning strategies demonstrated to be more robust to this classification problem, providing an average AUC performance of 95 % (ACC=94 %, Kappa=0.87 %) (Fig. 2).

## Towards a classification baseline at the “event” level

The problem of chRNAs classification can also be tackled from the perspective of the event level. The event level is basically the aggregation of all reads corresponding the same chimeric junction and its associated functional (biological) annotation (Fig. 4). Gingeras et al. [1] define chRNAs events (classes) as follows: i) individual RNAs can be transcribed on separate chromosomes (Class 1), on the same chromosome but with a different genomic order from that found in the mature RNA (Class 2), on the same chromosome but transcribed from different strands (Class 3), or on the same chromosome but from different alleles (Class 4). Before grouping chimeric reads into an event, it is required a proper filtering stage of potential methodological artifacts at the read level.

<figure><img src="image_6.png" /><figcaption>**Fig. 3** Box-plots of the selected features to discriminate chRNAs (TRUE and FALSE) at the read level. From left to right (score_break_length, score_break_Variance, coefficient_variation, score_is_duplex, mean_amplitude, score_nb_merge, coefficient_dispersion_interquartile). Best discrimination results are obtained with the features score_is_duplex, coefficient_variation, and score_break_length</figcaption></figure>
<table><caption>**Table 2** The average performance of classifiers at the read level using the top-3 more discriminative features (score_break_length, coefficient_variation, score_is_duplicate)</caption><thead><tr><th></th><th>GBM</th><th>RF</th><th>KNN</th><th>SVM</th></tr></thead><tbody><tr><td>Acc</td><td>95.01</td><td>95.16</td><td>94.29</td><td>93.49</td></tr><tr><td>Spe</td><td>92.64</td><td>93.02</td><td>92.44</td><td>92.05</td></tr><tr><td>Sen</td><td>96.42</td><td>96.42</td><td>95.39</td><td>94.35</td></tr><tr><td>Kappa</td><td>89.31</td><td>89.62</td><td>87.79</td><td>86.13</td></tr><tr><td>AUC</td><td>94.53</td><td>94.72</td><td>93.91</td><td>93.20</td></tr></tbody></table>

In the previous section we have conducted a first attempt at classifying chRNAs at the read level. These results allowed us to refine the current chimera score on CRAC (Crac-Score) eliminating false positives that might strongly bias the classification of chRNAs at the event level. This score was based on a set of selected significant rules extracted from the Random Forest model, by the utilization of the *inTree R package*.

Once having applied CRAC with an “optimal” score, one can go forward to the aggregation and annotation of chRNAs at the event level. We use the *cractools extract* command from the *CracTools* (http://cractools.gforge.inria.fr/) to aggregate chRNAs. Then, we annotate chRNAs and use it as filter to reclassify those involving the same gene as splicing or alternative splicing and also pseudogene as artefact. Without loss of generality, we continue to tackle the chRNAs prediction at the event level as a binary classification problem. We also make use of the terms “filtered aggregation” to differentiate this approach from CRAC. We have implemented and test this tool along with a fixed value of 0.7 on a new simulated data set. This data set follows the same strategy as discussed in previous section, so 40 millions of paired-end reads with read length of 100bp were generated. This simulation also covers 180 chromosome translocation over *Human* (GRCh38) genome. Flux Simulator were also used to express 121 translocation over 46938 reads. The following table summarizes the analysis (Table 3).

The performance of the classification at the event level, as it might be observed in (Table 3), is not better than a “random guess”. In fact, this is mainly due to the fact that i) aggregation does not take into account a customized classification model to discriminate chimeric events, and ii) chimeric (false positives) reads continue passing through the

<figure><img src="image_7.png" /><figcaption>**Fig. 4** The chRNA granularity. Scaling chRNAs classification from read to event level. The event level is basically the aggregation of all reads corresponding the same chimeric junction at read level and its associated functional (biological) annotation</figcaption></figure>
chRNAs in other, let us say, distant organism. As it was expected no classifier were better than a random guessing. Such observation is due to the fact both organisms have a distinct “chimeric profiles” with respect to the number of chromosomes, the arrangements of the exons and evolutionary rate of mutations. Therefore, classification margins boundaries built for *Human* cannot hold for *Drosophila*.

It is remarkable that classifiers employing an ensemble learning strategy are most suitable for the challenge of prediction chRNAs. Furthermore, a stacking model where false chimeric reads are filtered out at the read level, being followed by a classifier at the event level (embedding functional e biological covariates) is strongly recommended. We have generated two more RNA-seq data sets covering 40M reads (1x100 bp and 1x200 bp) to evaluate the strength of the prediction in both levels (Tables 5 and 6). The first genome has 121 chRNA events (translocations quantified along 46936 reads), and the latter has 115 events (translocations quantified along 101093 reads). We observe from (Fig. 5) that setting a CracScore to 0.7 is a good compromising between precision (PPR) and false discovery rate (FDR). Moreover, the higher the CracScore is the higher is the precision. However, such precision come with the cost of increasing the FDR at the read level, being compensate by lower FDR values at the event level. The size of the read does not imply in having better classification performance at both levels but, can increase the chance of having more reads covering chimeric junction, and consequently long reads, can help in boosting prediction at the event level.

To conclude our experimental study, we have applied the proposed pipeline for the prediction of chRNAs in real RNA-seq data (Illumina HiSeq 2000 *Homo sapiens*) from a set of twenty-seven patients having acute myeloid leukemia. The goals of this study are to obtain a comprehensive study of mutations and gene expression in human acute myeloid leukemia (AML). All sequence data are freely available at Gene Expression Ominibus (GEO) database within the accession number GSE49642. From the previous simulation study, we observed that most of the false chRNA events belong to the “Class_1” (60 %), while the remaining false events are shared between “Class_2”(30 %) and, “Class_3” and “Class_4” together (10 %). We hypothesise that the same amount of sequence artefacts found in simulated data can be equally distributed on real data. Thus, it is expected that classifiers might be able to eliminate successfully many of these false events. Indeed, the classification results obtained after applying the pipeline highlight a significant reduction in chRNA class distribution (Fig. 6). Biological process that generates Class_1 events are well-known. However, the main part of chRNA found in real data belong to Class_3 and Class_4. These chRNA have various mechanisms that are still little known. The simulation must be continuously improved to integrate new biological mechanisms but also other properties as cellular heterogeneity and library conception bias.

<table><caption>**Table 5** Performance of classifiers at both read and event levels using 100 bp</caption><thead><tr><th colspan="2">Data</th><th colspan="8">Simulated RNA-Seq 40M reads 100 pb</th></tr><tr><th colspan="2">CracScore</th><th colspan="2">0</th><th colspan="2">0.7</th><th colspan="2">0.9</th><th colspan="2">0.95</th><th colspan="2">0.99</th></tr><tr><th colspan="2">Class</th><th>True</th><th>False</th><th>True</th><th>False</th><th>True</th><th>False</th><th>True</th><th>False</th><th>True</th><th>False</th></tr></thead><tbody><tr><td colspan="2">Count reads</td><td>4196</td><td>1498</td><td>3815</td><td>600</td><td>2644</td><td>226</td><td>2056</td><td>139</td><td>477</td><td>139</td></tr><tr><td colspan="2">Aggregation</td><td>56</td><td>209</td><td>56</td><td>89</td><td>29</td><td>31</td><td>16</td><td>8</td><td>2</td><td>8</td></tr><tr><td colspan="2">Filtered aggregation</td><td>35</td><td>6</td><td>35</td><td>5</td><td>22</td><td>3</td><td>14</td><td>1</td><td>0</td><td>1</td></tr></tbody></table>
<table><caption>**Table 6** Performance of classifiers at both read and event levels using 200 bp</caption><thead><tr><th rowspan="2">Data</th><th colspan="10">Simulated RNA-Seq 40M reads 200 pb</th></tr><tr><th colspan="2">0</th><th colspan="2">0.7</th><th colspan="2">0.9</th><th colspan="2">0.95</th><th colspan="2">0.99</th></tr></thead><tbody><tr><td>CracScore</td><td>True</td><td>False</td><td>True</td><td>False</td><td>True</td><td>False</td><td>True</td><td>False</td><td>True</td><td>False</td></tr><tr><td>Class</td><td>13164</td><td>4965</td><td>10238</td><td>2178</td><td>7099</td><td>2015</td><td>4732</td><td>1789</td><td>4732</td><td>1789</td></tr><tr><td>Count reads</td><td>57</td><td>266</td><td>55</td><td>107</td><td>28</td><td>44</td><td>12</td><td>25</td><td>12</td><td>25</td></tr><tr><td>Aggregation</td><td>38</td><td>5</td><td>37</td><td>4</td><td>14</td><td>4</td><td>5</td><td>3</td><td>5</td><td>3</td></tr><tr><td>Filtered aggregation</td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td></tr></tbody></table>

The proposed benchmark highlights the fidelity of supervised classifiers in the problem of chRNAs prediction. Though, it is always important performing biological validation of these events to confirm the biological soundness of all discovered chRNAs.

## Conclusion

We proposed a modelization strategy that can be used to increase the tools performances in context of chRNA classification. We improve the models dynamically using a simulated

<figure><img src="image_8.png" /></figure>
<figure><img src="image_9.png" /><figcaption>**Fig. 6** The fidelity of chRNA classifiers using real RNA-seq data. The x-axis are related to samples, while the y-axis presents the number of chimeric events. The same amount of sequence artefacts found in simulated data can be equally distributed on real data (a). The classification results obtained after applying the pipeline highlight a significant reduction in chRNA class distribution (b).</figcaption></figure>
data generator, called SimCT (not yet published), that permit to continuously integrate
new complex chimeric events.

We have developed a benchmark pipeline incorporating a mutation genome process
and simulated RNA-seq data to evaluate the fidelity of using classifiers to predict chRNAs.
The simulated sequencing reads were aligned and annotated by CRAC. CRAC analyzes
the RNA-seq data, integrating genomic location and local coverage, allowing biological
predictions in one single step which is not available in any other fusion finder tools. More-
over, we enhance chRNAs classification by building models at both levels of read and
event. Aggregating reads at the event level allows us to incorporate biological covariates
and improving models' generalization along with the presented experimental study. We
envisage focusing on developing models and improving biological annotation at event
level which will be addressed in a future work.

Simulated and real RNA-seq data were carefully designed to explore feature engineering
and model performance of several classifiers, unraveling the role of ensemble learning
strategies in finding chRNAs. As far as we are concerned the present study provides the
first common benchmark data, as well as classification baseline for the identification of
chRNAs in both levels of reads and events.

## Additional file 1

**Additional file 1: Figure S1.** Anatomy of a chimeric junction identified with CRAC algorithm and its potential extracted features for machine learning benchmarking. (PDF 41.8 kb)

### Abbreviations

AML: Acute myeloid leukemia; AUC: Area under the curve; chRNA: Chimeric RNA; GEO: Gene expression Ominibus; Kappa: Cohen's kappa coefficient; SNV: Single nucleotide variants

### Acknowledgements

Sacha Beaumeunier and Ronnie Alves are fellowships of the IBC (ANR "investissement d'avenir en bioinformatique-projet-IBC); and Jérôme Audoux is fellowship of the FRM (Appel d'offres urgence pour la bioinformatique, projet DBI20131228566).

### Funding

This work was supported by the French ANR IBC project "investissement d'avenir en bioinformatique-projet-IBC" and FRM "appel d'offres urgence pour la bioinformatique, projet DBI20131228566".

### Availability of data and material

The dataset(s) supporting the conclusions of this article is(are) available in the project's site repository at https://sites.google.com/site/alvesrco/chimeres.

### Authors' contributions

TC, RA, NP, JA and SB initiated the project. SB, JA, NP and RA performed the analytical work and implementation of the pipeline. SB and RA performed the experimental work and contributed to pipeline testing. TC, FF and AB contributed to biological discussion. All authors contributed to the writing of the paper. All authors read and approved the final manuscript.

### Competing interests

The authors declare that they have no competing interests.

### Consent for publication

Not applicable.

### Ethics approval and consent to participate

Not applicable.

### Author details

<footnote n="1">¹Institut de Médecine Régénératrice et de Biothérapie, INSERM U1183, CHU Montpellier, Montpellier, France. ²Institut de Biologie Computationnelle, Université Montpellier, Montpellier, France. ³Laboratoire d'Informatique, de Robotique et de Microélectronique de Montpellier, Université Montpellier, UMR 5506 CNRS, Montpellier, France. ⁴PPGCC, Universidade Federal do Pará, Belém, Brazil. ⁵Instituto Tecnológico Vale, Belém, Brazil.</footnote>
Received: 1 February 2016 Accepted: 11 October 2016

Published online: 02 November 2016

References

1. Gingeras TR. Implications of chimaeric non-co-linear transcripts. Nature. 2009;461(7261):206-11.
doi:10.1038/nature08452.

2. Yoshihara K, Wang Q, Torres-Garcia W, Zheng S, Vegesna R, Kim H, Verhaak RGW. The landscape and therapeutic relevance of cancer-associated transcript fusions. Oncogene aop(current). 2014. doi:10.1038/onc.2014.406.

3. Ding L, Wendl MC, McMichael JF, Raphael BJ. Expanding the computational toolbox for mining cancer genomes. Nat Rev Genet. 2014;15(8):556-70. doi:10.1038/nrg3767.

4. Ren G, Zhang Y, Mao X, Liu X, Mercer E, Marzec J, Ding D, Jiao Y, Qiu Q, Sun Y, Zhang B, Yeste-Velasco M, Chelala C, Berney D, Lu YJ. Transcription-mediated chimeric mas in prostate cancer: Time to revisit old hypothesis?. OMICS J Integr Biol. 2014;18(10):615-24. doi:10.1089/omi.2014.0042.

5. Carrara M, Beccuti M, Lazzarato F, Cavallo F, Cordero F, Donatelli S, Calogero RA. State-of-the-art fusion-finder algorithms sensitivity and specificity. BioMed Res Int. 2013;2013:1-6. doi:10.1155/2013/340620.

6. Liu S, Tsai WH, Ding Y, Chen R, Fang Z, Huo Z, Kim S, Ma T, Chang TY, Priedigkeit NM, Lee AV, Luo J, Wang HW, Chung IF, Tseng GC. Comprehensive evaluation of fusion transcript detection algorithms and a meta-caller to combine top performing methods in paired-end rna-seq data. Nucleic Acids Res. 2015. doi: 10.1093/nar/gkv1234, arxiv http://nar.oxfordjournals.org/content/early/2015/11/17/nar.gkv1234.full.pdf+html.

7. Polikar R. Ensemble based systems in decision making. Circ Syst Mag IEEE. 2006;6(3):21-45.
doi:10.1109/mcas.2006.1688199.

8. Larranaga P, Calvo B, Santana R, Bielza C, Galdiano J, Inza I, Lozano JA, Armananzas R, Santafe G, Perez A, Robles V. Machine learning in bioinformatics., Brief Bioinform. 2006;7(1):86-112. doi:10.1093/bib/bbk007.

9. Domingos P. A few useful things to know about machine learning. Commun ACM. 2012;55(10):78-87.
doi:10.1145/2347736.2347755.

10. Carrara M, Beccuti M, Cavallo F, Donatelli S, Lazzarato F, Cordero F, Calogero R. State of art fusion-finder algorithms are suitable to detect transcription-induced chimeras in normal tissues?. BMC Bioinformatics.
2013;14(S-7):2. doi:10.1186/1471-2105-14-S7-S2.

11. Benelli M, Pescucci C, Marseglia G, Severgnini M, Torricelli F, Magi A. Discovering chimeric transcripts in paired-end rna-seq data by using ericscript. Bioinformatics. 2012. doi: 10.1093/bioinformatics/bts617 http://bioinformatics.oxfordjournals.org/content/early/2012/10/23/bioinformatics.bts617.full.pdf+html.

12. McPherson A, Hormozdiari F, Zayed A, Giuliany R, Ha G, Sun MGF, Griffith M, Heravi Moussavi A, Senz J, Melnyk N, Pacheco M, Marra MA, Hirst M, Nielsen TO, Sahinalp SC, Huntsman D, Shah SP. defuse: an algorithm for gene fusion discovery in tumor rna-seq data. PLoS Comput Biol. 2011;7(5):1001138. doi:10.1371/journal.pcbi.1001138.

13. Bruno A, Miecznikowski J, Qin M, Wang J, Liu S. Fusim: a software tool for simulating fusion transcripts. BMC Bioinformatics. 2013;14(1):13. doi:10.1186/1471-2105-14-13.

14. Griebel T, Zacher B, Ribeca P, Raineri E, Lacroix V, Guigó R, Sammeth M. Modelling and simulating generic RNA-Seq experiments with the flux simulator., Nucleic Acids Res. 2012;40(20):10073-83. doi:10.1093/nar/gks666.

15. Philippe N, Salson M, Commes T, Rivals E. CRAC: an integrated approach to the analysis of RNA-seq reads., Genome Biol. 2013;14(3):30. doi:10.1186/gb-2013-14-3-r30.

16. Kuhn M. Building Predictive Models in R Using the caret Package. J Stat Softw. 2008;28(5):1-26.

Submit your next manuscript to BioMed Central
and we will help you at every step:

* We accept pre-submission inquiries

* Our selector tool helps you to find the most relevant journal

* We provide round the clock customer support

* Convenient online submission

* Thorough peer review

* Inclusion in PubMed and all major indexing services

* Maximum visibility for your research

Submit your manuscript at
www.biomedcentral.com/submit

BioMed Central