BACKGROUND
even with decreasing costs for sequencing and microarray experiments, time series experiments are still expensive and require a large number of samples.
thus, most time series currently have a very limited number of biological replicates.
this makes it difficult to identify genes that truly show time-dependent expression patterns  and genes that just seem to have similar patterns due to biological variance .
the biological variance is likely to be relatively high, especially when samples are collected from higher eukaryotes, because animals and plants are usually sampled from different individuals to avoid perturbation artifacts during sampling.
thus, in most time course experiments, the samples at each time point are usually from different individuals, resulting in a high biological variance among samples.
this is the main reason why sufficient numbers of replicates are necessary.
lee et al.
proposed that three replicates are sufficient, but this number also depends on the type of experiment  <cit> .
however, the importance of biological replicates is often neglected in time series experiments, especially when circadian rhythms in gene expression are examined using transcriptomics datasets.
many organisms have an endogenous clock, known as a circadian clock, to coordinate daily activities.
the output of the circadian clock has the period of approximately 24 h; for example, the body temperature and sleep-wake cycle in humans, leaf movement in mimosa, and flower opening in night-blooming jasmine all show 24 h diurnal rhythms under both light/dark and approx.
24 h rhythms under constant conditions  <cit> .
although the molecular components of circadian clocks are not conserved between animals and plants, negative and positive feedback loops in transcriptional and post-translational levels are the core system of circadian clocks in both animals and plants  <cit> .
these multiple interlocked feedback loops confer stability and protection from stochastic perturbations on the complexity of the circadian system  <cit> .
to understand this complex network on a transcriptional level, time series microarrays have been frequently used to examine the oscillation of genes on a genomic scale  <cit> .
diurnal rhythms in transcript accumulation can be described in mathematical terms, including period, phase, and amplitude  <cit> .
there are several different algorithms that can be used to calculate these parameters from real data; they can furthermore be applied to identify oscillating genes in microarray or rna-sequencing data.
from the algorithms available we selected arser  <cit> , haystack  <cit>  as well as the algorithms implemented in biodare   <cit> .
arser was selected as it has been shown to outperform earlier available algorithms such as cosopt and fisher’s g-test  <cit> .
the biodare platform is not originally designed for the analysis of gene expression data as the maximal list length of datasets that can be submitted is limited to  <dig>  thus, gene expression data has to be split into multiple datasets.
nevertheless biodare has the advantage of providing  <dig> additional different algorithms for the analysis.
using these algorithms we show the influence of replicates and resampling on the accuracy of predictions of rhythmically expressed genes.
although we perform the analysis to identify oscillating genes in circadian expression datasets, the resampling method can be similarly used to improve the detection of other time dependent expression patterns as long as the samples are collected from different individuals at the specific time points.
RESULTS
the determination of oscillating genes is a binary classification.
there are only two possible outcomes: either a gene is rhythmically expressed or it is not.
the accuracy of this classification can be estimated by a confusion matrix.
there are four fundamental members of the matrix: true positives , false negatives , true negatives , and false positives .
as the number of true negatives and false negatives can be directly calculated from the total number of oscillating and non-oscillating genes and the number of true- and false positive genes identified, we only analyzed true- and false-positives in our calculations.
the total number of oscillating and non-oscillating genes was set to  <dig> in our simulated datasets .
to calculate the performance of arser, haystack and the algorithms implemented in biodare, we simulated different conditions and wave forms for oscillating transcripts.
to do this we used three different simulation procedures.
to simulate entrained, synchronized oscillations all simulations were done with a fixed period ranging from  <dig> to 28 h .
in contrast, the differences in free running period between different individuals under constant conditions were simulated by generating a dataset that contained  <dig> time courses that differed in period according to published standard deviations for individual cells  <cit>  .
in addition we generated a time course based on a published ordinary equation model of the mammalian circadian model  <cit>   .
for each simulation procedure  <dig> time courses were initially calculated, corresponding to the common experimental time courses for gene expression analysis in the literature that resample 2-day time courses with 4 h sampling intervals and  <dig> replicates.
from these initial time courses we generated the initial dataset  by randomly selecting one time point from each simulated time course.
these initial datasets were in addition averaged to generate a fourth, averaged time course.
true and false positives were then calculated for arser, haystack and using biodare.
from biodare we initially tested all implemented algorithms but found that fft-nlls was performing best, confirming the observations form zielinski et al.
<cit> .
we therefore only present the results from this biodare algorithm .
in comparison arser detects the largest number of true positives but at the cost of a relatively high number of false positives.figure  <dig>
identification of oscillating transcripts in each replicate and the average dataset.
true positives  and false positives  for ld.
ll and ode-based time courses using arser, haystack or biodare fft-nlls the results are displayed for each individual replicate and the average dataset.
the replicates were generated as described in the methods section.
in the averaged time courses the number of true positives detected by arser is in most cases slightly higher than in the individual replicates but this again comes at the cost of a higher number of false positives.
haystack and biodare fft-nlls show similar performance but haystack has more problems to detect oscillating genes in ode-based simulations.
as detected false positive genes can be experimentally quite costly in follow up studies, we wanted to improve the accuracy of the prediction without increasing the number of replicates or time points required as this too would be experimentally costly if not infeasible.
we hypothesized that transcripts identified several times in resampled datasets contain more true positive and fewer false positive transcripts.
to test this hypothesis, we generated  <dig> resampled datasets and identified oscillating genes by arser and haystack algorithms in each resampled dataset.
subsequently, we calculated the consensus of detected oscillating genes in these  <dig> resampled datasets.
a consensus of  <dig> means that the genes were detected in at least  <dig> out of the  <dig> resampled datasets.
the consensus graphs for the analysis performed with arser and haystack are shown in figures  <dig> and  <dig>  respectively.
we compared the number of true and false positives to the number of true and false positives found in the averaged dataset, as well as to the consensus between the initial datasets and the initial simulations.
the initial simulations represent the ideal situation that samples could be retrieved from the same individual, this is, however, not possible for gene expression analysis in most cases.
it nevertheless represents the maximal detectable number of true positive transcripts in a noisy dataset.
as can be seen from figure  <dig>  up to a required consensus between  <dig> datasets, the resampled datasets show a larger number of true positives compared to average and the overlap between initial datasets.
to acquire the same consensus the number of false positives is  <dig> of  <dig>  a similar number of false positives is found if a full overlap between the  <dig> initial datasets is required.
the number of true positives for the latter is, however, much lower for all types of simulations.
we can therefore conclude that the resampling of datasets increases the number of true positive oscillating transcripts detected in a dataset without increasing the number of false positives compared to the initial replicates.
except when very low consensus is required , the number of false positives detected with arser is always higher for averaged datasets, and hence not well suited to reliably identify oscillating genes.figure  <dig>
performance evaluation of the resampling method to identify oscillating transcripts.
for ll-, ld- and ode-based time courses  <dig> time courses were simulated .
from these original simulations  <dig> replicates  for each type of time course were generated by randomly selecting one time point from each of the original simulations to mimic experimental sampling procedures.
to generate the averaged dataset, the expression values of the  <dig> replicates at each time point were averaged.
the initial datasets were furthermore used to generate  <dig> resampled datasets by random sampling at each time point.
all datasets were analyzed with arser and true positives  and false positives  were calculated requiring increasing consensus between the datasets .
a consensus of  <dig> thereby means that a gene is found in at least  <dig> different resampled or originally simulated datasets.
for the averaged time course the true and false positives calculated are displayed as a line for comparison.
analysis of resampled time courses with haystack.
for ll-, ld- and ode-based time courses  <dig> time courses were simulated .
from these original simulations  <dig> replicates  for each type of time course were generated by randomly selecting from each of the original simulations one time point to mimic experimental sampling procedures.
to generate the averaged dataset, the expression values of the  <dig> replicates at each time point were averaged.
the initial datasets were furthermore used to generate  <dig> resampled datasets by random sampling at each time point.
all datasets were analyzed with haystack and true positives  and false positives  were calculated requiring increasing consensus between the datasets .
a consensus of  <dig> thereby means that a gene is found at least in  <dig> different resampled or originally simulated datasets.
for the averaged time course the true and false positives calculated are displayed as a line for comparison.
we next analyzed the influence of the number of resampled datasets on the detection of true and false positive oscillating transcripts.
as can be seen in figure  <dig>  higher consensus is required for a higher number of resampled datasets but the consensus range in which no false positives are found and in which the number of true positives remains high, is larger when a larger number of randomized datasets are analyzed.
for the analysis of real data we therefore chose to generate  <dig> resampled datasets.
unfortunately there are very few circadian datasets available with sufficient replicates and time points.
we found one study with two replicates performed in two different mouse tissues   <cit>  and one other mouse study with  <dig> replicates  <cit> .
the overall number of oscillating genes found in the dataset from miller et al.
<cit>  is similar to that reported in the original article.
the overlap, however, was not analyzed in the original work and we only found  <dig> and  <dig> transcripts, respectively  in both replicates.
using our resampling approach we identified  <dig> and  <dig> genes when requiring consensus between at least  <dig> sets and  <dig> and  <dig> transcripts, respectively, if a consensus of  <dig> was required.figure  <dig>
dependency of the analysis on the number of resampled dataset.
<dig> initial datasets were generated as described in figure  <dig> and in the methods section and from these either  <dig>   <dig> or  <dig> resampled datasets were generated by random resampling and the number of true  and false positive  transcripts calculated.
the detection of oscillating genes was performed with arser.
analysis of published expression data.
we reanalyzed  <dig> published circadian datasets with 4 hour sampling intervals and  <dig> time points using arser.
the dataset by miller et al.
<cit>  contained only two replicates each for two mouse tissues  and muscle ).
the dataset from na et al.
<cit>   contained  <dig> replicates.
for the dataset from na et al.
<cit>   <dig> transcripts were found in all  <dig> initial replicates .
in our resampled dataset  <dig> transcripts were identified as oscillating when we require a consensus of  <dig>   <dig> genes remain if we require a consensus of  <dig>
as the study by na et al.
resulted in a larger number of oscillating transcripts we used our simulated ll datasets to analyze how the number of replicates influences the number of true and false negatives and thus the accuracy of the detection of oscillating transcripts.
to do so we initially simulated  <dig> datasets.
those were used to generate the different numbers of initial replicated datasets.
the analysis showed that the number of oscillating transcripts detected for a full overlap between all replicates is decreasing with the number of replicates  with increasing consensus required.
but starting from a required consensus of  <dig>  false positives were no longer detected in the initial datasets , thus a consensus of  <dig> is sufficient to accurately detect oscillating transcripts for initial datasets.
taken this into account the amount of true positives transcripts is higher for higher numbers of replicates as would be expected.
looking at the resampled datasets we see that with increasing number of replicates lower consensus is required to avoid detection of false positives, emphasizing the importance of replicates for the detection of circadian regulated transcripts.figure  <dig>
impact of the number of replicates on the accuracy.
from  <dig> original simulations, either  <dig> , <dig> , or  <dig> replicates were generated by random sampling  and analyzed using the arser algorithm as described in the methods section.
the initial datasets were then used to generate  <dig> resampled datasets  as described in figure  <dig> and the methods section.
the number of true positives  and false positives  for different consensus required is shown.
CONCLUSIONS
in this analysis, we conclude that in comparison to single replicates and averaged datasets, our resampling method improves the detection of oscillating transcripts without increasing the number of false positives.
the resampling method particularly outperformed the average method to reduce the number of false positive transcripts.
furthermore, the resampling method shows that biological replicates are important to accurately identify true oscillating transcripts using time series gene expression datasets, and that the average method may result in a large number of false positives.
to reliably identify oscillating transcripts, resampled datasets should be generated from at least  <dig> experimental samples per time point.

 even with decreasing costs for sequencing and microarray experiments, time series experiments are still expensive and require a large number of samples. thus, most time series currently have a very limited number of biological replicates. this makes it difficult to identify genes that truly show time-dependent expression patterns and genes that just seem to have similar patterns due to biological variance. the biological variance is likely to be relatively high, especially when samples are collected from higher eukaryotes, because animals and plants are usually sampled from different individuals to avoid perturbation artifacts during sampling. thus, in most time course experiments, the samples at each time point are usually from different individuals, resulting in a high biological variance among samples. this is the main reason why sufficient numbers of replicates are necessary. lee et al. proposed that three replicates are sufficient, but this number also depends on the type of experiment. however, the importance of biological replicates is often neglected in time series experiments, especially when circadian rhythms in gene expression are examined using transcriptomics datasets. many organisms have an endogenous clock, known as a circadian clock, to coordinate daily activities. the output of the circadian clock has the period of approximately 24 h; for example, the body temperature and sleep-wake cycle in humans, leaf movement in mimosa, and flower opening in night-blooming jasmine all show 24 h diurnal rhythms under both light/dark and approx. 24 h rhythms under constant conditions. although the molecular components of circadian clocks are not conserved between animals and plants, negative and positive feedback loops in transcriptional and post-translational levels are the core system of circadian clocks in both animals and plants. these multiple interlocked feedback loops confer stability and protection from stochastic perturbations on the complexity of the circadian system. to understand this complex network on a transcriptional level, time series microarrays have been frequently used to examine the oscillation of genes on a genomic scale. diurnal rhythms in transcript accumulation can be described in mathematical terms, including period, phase, and amplitude. there are several different algorithms that can be used to calculate these parameters from real data; they can furthermore be applied to identify oscillating genes in microarray or rna-sequencing data. from the algorithms available we selected arser, haystack as well as the algorithms implemented in biodare. arser was selected as it has been shown to outperform earlier available algorithms such as cosopt and fisher’s g-test. the biodare platform is not originally designed for the analysis of gene expression data as the maximal list length of datasets that can be submitted is limited to thus, gene expression data has to be split into multiple datasets. nevertheless biodare has the advantage of providing additional different algorithms for the analysis. using these algorithms we show the influence of replicates and resampling on the accuracy of predictions of rhythmically expressed genes. although we perform the analysis to identify oscillating genes in circadian expression datasets, the resampling method can be similarly used to improve the detection of other time dependent expression patterns as long as the samples are collected from different individuals at the specific time points.  the determination of oscillating genes is a binary classification. there are only two possible outcomes: either a gene is rhythmically expressed or it is not. the accuracy of this classification can be estimated by a confusion matrix. there are four fundamental members of the matrix: true positives, false negatives, true negatives, and false positives. as the number of true negatives and false negatives can be directly calculated from the total number of oscillating and non-oscillating genes and the number of true- and false positive genes identified, we only analyzed true- and false-positives in our calculations. the total number of oscillating and non-oscillating genes was set to in our simulated datasets. to calculate the performance of arser, haystack and the algorithms implemented in biodare, we simulated different conditions and wave forms for oscillating transcripts. to do this we used three different simulation procedures. to simulate entrained, synchronized oscillations all simulations were done with a fixed period ranging from to 28 h. in contrast, the differences in free running period between different individuals under constant conditions were simulated by generating a dataset that contained time courses that differed in period according to published standard deviations for individual cells. in addition we generated a time course based on a published ordinary equation model of the mammalian circadian model. for each simulation procedure time courses were initially calculated, corresponding to the common experimental time courses for gene expression analysis in the literature that resample 2-day time courses with 4 h sampling intervals and replicates. from these initial time courses we generated the initial dataset by randomly selecting one time point from each simulated time course. these initial datasets were in addition averaged to generate a fourth, averaged time course. true and false positives were then calculated for arser, haystack and using biodare. from biodare we initially tested all implemented algorithms but found that fft-nlls was performing best, confirming the observations form zielinski et al.. we therefore only present the results from this biodare algorithm. in comparison arser detects the largest number of true positives but at the cost of a relatively high number of false positives.figure identification of oscillating transcripts in each replicate and the average dataset. true positives and false positives for ld. ll and ode-based time courses using arser, haystack or biodare fft-nlls the results are displayed for each individual replicate and the average dataset. the replicates were generated as described in the methods section. in the averaged time courses the number of true positives detected by arser is in most cases slightly higher than in the individual replicates but this again comes at the cost of a higher number of false positives. haystack and biodare fft-nlls show similar performance but haystack has more problems to detect oscillating genes in ode-based simulations. as detected false positive genes can be experimentally quite costly in follow up studies, we wanted to improve the accuracy of the prediction without increasing the number of replicates or time points required as this too would be experimentally costly if not infeasible. we hypothesized that transcripts identified several times in resampled datasets contain more true positive and fewer false positive transcripts. to test this hypothesis, we generated resampled datasets and identified oscillating genes by arser and haystack algorithms in each resampled dataset. subsequently, we calculated the consensus of detected oscillating genes in these resampled datasets. a consensus of means that the genes were detected in at least out of the resampled datasets. the consensus graphs for the analysis performed with arser and haystack are shown in figures  and respectively. we compared the number of true and false positives to the number of true and false positives found in the averaged dataset, as well as to the consensus between the initial datasets and the initial simulations. the initial simulations represent the ideal situation that samples could be retrieved from the same individual, this is, however, not possible for gene expression analysis in most cases. it nevertheless represents the maximal detectable number of true positive transcripts in a noisy dataset. as can be seen from figure  up to a required consensus between datasets, the resampled datasets show a larger number of true positives compared to average and the overlap between initial datasets. to acquire the same consensus the number of false positives is of a similar number of false positives is found if a full overlap between the initial datasets is required. the number of true positives for the latter is, however, much lower for all types of simulations. we can therefore conclude that the resampling of datasets increases the number of true positive oscillating transcripts detected in a dataset without increasing the number of false positives compared to the initial replicates. except when very low consensus is required, the number of false positives detected with arser is always higher for averaged datasets, and hence not well suited to reliably identify oscillating genes.figure performance evaluation of the resampling method to identify oscillating transcripts. for ll-, ld- and ode-based time courses time courses were simulated. from these original simulations replicates for each type of time course were generated by randomly selecting one time point from each of the original simulations to mimic experimental sampling procedures. to generate the averaged dataset, the expression values of the replicates at each time point were averaged. the initial datasets were furthermore used to generate resampled datasets by random sampling at each time point. all datasets were analyzed with arser and true positives and false positives were calculated requiring increasing consensus between the datasets. a consensus of thereby means that a gene is found in at least different resampled or originally simulated datasets. for the averaged time course the true and false positives calculated are displayed as a line for comparison. analysis of resampled time courses with haystack. for ll-, ld- and ode-based time courses time courses were simulated. from these original simulations replicates for each type of time course were generated by randomly selecting from each of the original simulations one time point to mimic experimental sampling procedures. to generate the averaged dataset, the expression values of the replicates at each time point were averaged. the initial datasets were furthermore used to generate resampled datasets by random sampling at each time point. all datasets were analyzed with haystack and true positives and false positives were calculated requiring increasing consensus between the datasets. a consensus of thereby means that a gene is found at least in different resampled or originally simulated datasets. for the averaged time course the true and false positives calculated are displayed as a line for comparison. we next analyzed the influence of the number of resampled datasets on the detection of true and false positive oscillating transcripts. as can be seen in figure  higher consensus is required for a higher number of resampled datasets but the consensus range in which no false positives are found and in which the number of true positives remains high, is larger when a larger number of randomized datasets are analyzed. for the analysis of real data we therefore chose to generate resampled datasets. unfortunately there are very few circadian datasets available with sufficient replicates and time points. we found one study with two replicates performed in two different mouse tissues and one other mouse study with replicates. the overall number of oscillating genes found in the dataset from miller et al. is similar to that reported in the original article. the overlap, however, was not analyzed in the original work and we only found and transcripts, respectively in both replicates. using our resampling approach we identified and genes when requiring consensus between at least sets and and transcripts, respectively, if a consensus of was required.figure dependency of the analysis on the number of resampled dataset. initial datasets were generated as described in figure  and in the methods section and from these either or resampled datasets were generated by random resampling and the number of true and false positive transcripts calculated. the detection of oscillating genes was performed with arser. analysis of published expression data. we reanalyzed published circadian datasets with 4 hour sampling intervals and time points using arser. the dataset by miller et al. contained only two replicates each for two mouse tissues and muscle ). the dataset from na et al. contained replicates. for the dataset from na et al. transcripts were found in all initial replicates. in our resampled dataset transcripts were identified as oscillating when we require a consensus of genes remain if we require a consensus of as the study by na et al. resulted in a larger number of oscillating transcripts we used our simulated ll datasets to analyze how the number of replicates influences the number of true and false negatives and thus the accuracy of the detection of oscillating transcripts. to do so we initially simulated datasets. those were used to generate the different numbers of initial replicated datasets. the analysis showed that the number of oscillating transcripts detected for a full overlap between all replicates is decreasing with the number of replicates with increasing consensus required. but starting from a required consensus of false positives were no longer detected in the initial datasets, thus a consensus of is sufficient to accurately detect oscillating transcripts for initial datasets. taken this into account the amount of true positives transcripts is higher for higher numbers of replicates as would be expected. looking at the resampled datasets we see that with increasing number of replicates lower consensus is required to avoid detection of false positives, emphasizing the importance of replicates for the detection of circadian regulated transcripts.figure impact of the number of replicates on the accuracy. from original simulations, either,, or replicates were generated by random sampling and analyzed using the arser algorithm as described in the methods section. the initial datasets were then used to generate resampled datasets as described in figure  and the methods section. the number of true positives and false positives for different consensus required is shown.  in this analysis, we conclude that in comparison to single replicates and averaged datasets, our resampling method improves the detection of oscillating transcripts without increasing the number of false positives. the resampling method particularly outperformed the average method to reduce the number of false positive transcripts. furthermore, the resampling method shows that biological replicates are important to accurately identify true oscillating transcripts using time series gene expression datasets, and that the average method may result in a large number of false positives. to reliably identify oscillating transcripts, resampled datasets should be generated from at least experimental samples per time point. 
in particular, the number of false positives can be greatly reduced while at the same time the number of true positives can be maintained in the range of other methods currently used to determine rhythmically expressed genes. as time series experiments in higher eukaryotes usually obtain data from different individuals collected at the different time points, a time series sample itself is not equivalent to a true biological replicate but is, rather, a combination of several biological replicates. the resampling approach described here therefore increases the accuracy of time series expression data analysis and furthermore emphasizes the importance of biological replicates in identifying oscillating genes. resamplinggene expression dataarserbiological replicatescircadian rhythmshaystackissue-copyright-statement© the author 2014 the analysis of expression data derived from a time series sample is therefore often performed with a low number of replicates due to budget limitations or limitations in sample availability. resampling can be used for any time series expression dataset as long as the samples are acquired from independent individuals at each time point.
 as time series experiments in higher eukaryotes usually obtain data from different individuals collected at the different time points, a time series sample itself is not equivalent to a true biological replicate but is, rather, a combination of several biological replicates. the analysis of expression data derived from a time series sample is therefore often performed with a low number of replicates due to budget limitations or limitations in sample availability. in addition, most algorithms developed to identify specific patterns in time series dataset do not consider biological variation in samples collected at the same conditions.  using artificial time course datasets, we show that resampling considerably improves the accuracy of transcripts identified as rhythmic. in particular, the number of false positives can be greatly reduced while at the same time the number of true positives can be maintained in the range of other methods currently used to determine rhythmically expressed genes.  the resampling approach described here therefore increases the accuracy of time series expression data analysis and furthermore emphasizes the importance of biological replicates in identifying oscillating genes. resampling can be used for any time series expression dataset as long as the samples are acquired from independent individuals at each time point. electronic supplementary material the online version of this article contains supplementary material, which is available to authorized users. keywords resamplinggene expression dataarserbiological replicatescircadian rhythmshaystackissue-copyright-statement© the author 2014 