BACKGROUND
ordination methods are useful tools for the analysis of gene expression microarrays.
principal component analysis  and correspondence analysis  have both been used to extract the main sources of variation present in highly multivariate microarray data  <cit> .
the supervised counterparts of these approaches, including between-group analysis   <cit>  and analyses with respect to instrumental variables  <cit> , were proposed to handle descriptive variables controlled in the design of the experiment .
when dealing with transcriptomics data, multivariate approaches are generally more appropriate than univariate strategies because they intrinsically take gene covariations and interactions into account.
constrained ordination methods are very efficient for sample classification and class prediction.
they are flexible and can be used easily to identify groups of genes associated with classes of samples.
geometrical interpretations are generally required to investigate the gene-sample relationship.
genes of interest can also be ranked according to their discriminative power.
however, considering the exploratory nature of these methods, it is not trivial to assess the significance of a given gene dysregulation in a multivariate setting.
these methods rely on solving an eigenvalue problem whose solutions are given by the leading eigenvectors and whose theoretical statistical properties are particularly complex to study.
to overcome this issue, resampling techniques have been proposed to estimate the stability of multivariate analyses.
these techniques were described in a variety of scientific frameworks including environmetrics  <cit> , chemometrics  <cit> , and archaeology  <cit> .
the general purpose is to develop inferential procedures for testing the statistical significance of the parameters provided by these exploratory techniques.
their applications are manifold, e.g.
assessing which variables significantly contribute to the principal axes of a pca, detecting outliers or influential observations.
this approach has a great potential in the context of microarray data analysis as proposed by tan and collaborators  <cit> .
these authors described an application of bootstrapping to correspondence analysis.
they outlined that their approach would have several advantages over classical gene-by-gene fits of anova models.
it particularly enables the extraction of lists of genes which are biologically more informative than those found by anova.
in the present work, we propose a specific methodology for testing the stability of constrained ordination methods applied to microarray data.
unlike previous studies, our approach is dedicated to supervised multivariate analyses.
to our knowledge, very few studies addressed the issue of stability assessment in supervised multivariate analyses.
the potential of associating stability analysis in the supervised multidimensional context is multiple.
by using the information of sample descriptors, genes can be associated with a given class of samples and the significance of this association can be assessed.
a derived significance testing strategy regarding gene contributions is proposed.
further resampling methods based on jackknifing are also proposed to identify influential observations as an aid in outlier detection in microarray data sets.
a comprehensive set of r functions illustrating our methodology was developed.
the package is freely available on request.
the present manuscript is organized as follows.
the first section introduces some theoretical aspects of ordination methods  and constrained ordination methods .
the subsequent sections describe the different resampling strategies used in this project, as well as details about the algorithm.
illustrative examples demonstrating the implemented technique are given.
methods and
RESULTS
theory
ordination methods
both pca and ca are commonly used in microarray data analysis.
some authors stressed that ca has several advantages over pca  <cit> .
like other dimension reduction methods, ca summarizes structures in high-dimension space by projection onto a low dimension sub-space while loosing as little information as possible.
correspondence analysis involves a first step of symmetrical data transformation into a chi-square distance matrix which makes ca outputs particularly appropriate for the exploration of relationships between samples and genes.
the mathematical basis of ca has been described elsewhere  and will be briefly summarized.
thereafter observations are shown as rows and variables as columns.
let us define the following:
• y: the  matrix of gene expression data
• p = y/n: the data matrix divided by its grand total
• r: the n-dim vector of row sums of p
• c: the m-dim vector of column sums of p
• dr = diag: the diagonal matrix of row sums
• dc = diag: the diagonal matrix of column sums
the correspondence analysis of y is obtained by carrying out the singular value decomposition of the doubled centered and standardized matrix z:
z=dr−1/2dc−1/2=uΛvt
with Λ the k × k ) diagonal matrix of singular values associated with z with λ <dig> ≥ ... ≥ λk >  <dig>  u an  matrix whose columns are the left singular vectors of z and v an  matrix whose columns are the right singular vectors of z. the rows of u and v are orthonormal with respect to dr and dc respectively:
utdru = vt dcv = i
the principal components and row coordinates are respectively given by dr−1/2u and dr−1/2uΛ.
the principal axes and column coordinates are respectively given by dc−1/2v and dc−1/2vΛ.
constrained ordination methods
in microarray experiments, besides the main table y containing the gene expression values, additional descriptive variables x controlled in the experimental design generally characterize samples.
constrained ordination methods aim to display the variations in the data which are explained by the descriptive variables.
these two-table methods are dissymmetric because the information from x is used to constrain the analysis of y. correspondence analysis with respect to instrumental variables   <cit> , which is closely related to pca on instrumental variables  <cit> , and between-group correspondence analysis  <cit>  are two examples of constrained correspondence analysis which have been successfully applied to microarray data.
in this field, these methods have been used for different purposes including sample classification  <cit> , disease-class prediction  <cit> , and removal of undesirable effects  <cit> .
bga is a particular case of caiv when samples are characterized by one single categorical variable.
between-group correspondence analysis of table y given the class descriptor x is simply the correspondence analysis of the table Ŷ corresponding to the table of means per group.
bga is the analysis of the per-class centroids.
it provides up to g -  <dig> discriminating axes .
the initial samples are thereafter projected as supplementary rows in the bga sub-space.
the bga procedure provides the best linear combination of variables which maximizes the between-group variance.
stability of gene contributions using bootstrapping
one interesting feature of bga is the possibility to associate genes with pre-defined sample classes.
the gene contribution towards sample classes is defined as the absolute distance from the center of the bga axes to the orthogonal projections of the gene  onto the vector of the class centroid  .
this distance is measured by the parameter: α0=〈x,y〉‖x‖ <dig>  the higher |α0| is, the more important the gene contribution is.
the sign of α <dig> indicates whether the gene is specifically up- or down-regulated, respectively for positive and negative values.
genes can be ranked according to their contribution towards each class modalities.
introduced by efron  <cit> , bootstrapping is a distribution-free resampling method generally used to estimate the variance of estimators.
like other resampling techniques, bootstrapping provides a good alternative to establish the variability of an estimator.
it is classically used to assess the bias and variance of model parameters, construct confidence intervals and rebuild empirical distributions.
several bootstrap refinements were implemented in the framework of techniques incorporating singular value decomposition  <cit> .
in the present work, non-parametric bootstrapping was used to estimate the significance of gene contributions.
the proposed bootstrapping strategy is based on the bga model assumptions.
as mentioned previously, the bga of table y with regard to the categorical variable x, is the analysis of the  table Ŷ corresponding to the per-class mean of y. let us define x, the  table of dummy variables coded from x and Ŷx = x-1xty, the  matrix of fitted values.
bootstrapped samples are built based on the residuals e = y - Ŷx.
residuals are sampled with replacement  and new data sets are built as follow: y* = Ŷx + e*.
the analysis of  <dig> to  <dig> perturbated data sets is generally required to assess the parameters' distribution of a multivariate model .
out of these empirical distributions several indicators of the stability of gene contributions were calculated.
non-parametric 95% bootstrap confidence intervals were constructed using the  <dig> % and  <dig> % quantiles of the bootstrapped distribution.
z-scores were defined for each gene as the ratio of the bootstrap estimates to its standard deviation.
p-values were estimated according to the "bootstrapped eigenvector" procedure  <cit>  as the probability of obtaining gene contributions equal to or smaller than zero for genes contributing positively in the original data set, or alternatively equal to or larger than zero for genes contributing negatively in the original data set.
convex hulls were used to graphically display the spread of the bootstrapped gene coordinates on the dominant principal axes.
the relative inertia of these gene coordinates was measured by the ratio of gene inertia to the total inertia.
lebart  <cit>  proposed two main categories of bootstrapping.
partial bootstrap makes use of a posteriori projections of the resampled elements on the original reference subspace provided by the analysis of the initial data set.
total bootstrap performs a new analysis on each of the resampled data sets.
both strategies have been implemented in the current project.
it is noteworthy that partial bootstrap does not involve successive steps of singular value decomposition which makes it considerably faster than total bootstrap.
moreover, because total bootstrap requires a complete ca to be carried out for each bootstrapped table, the new sets of row and column coordinates belong to different subspaces which make their comparison more complex.
unlike partial bootstrap, total bootstrap is potentially subjected to axis reflection or inversion  <cit> .
signs of row and column coordinates in perturbated data sets can be inverted compared with the original data set.
at least two approaches have been reported to overcome this drawback.
the first consists in determining the sign of the correlations between the principal axes prior and after perturbations.
a negative correlation indicates a reflexion which can be corrected by multiplying the new row and column coordinates by - <dig>  procrustes rotation was also proposed to fit the resampled row and column coordinates with the original scores and loadings  <cit> .
the first  option was implemented in the present work.
detection of outliers using jackknifing
jackknifing is another resampling technique introduced by quenouille  <cit>  and later developed by tukey  <cit> .
jackknifed samples are built using a leave-one-sample-out strategy.
although jackknifing can be seen as a rough linear approximation of bootstrapping  <cit> , it proves to be useful for investigating the influence of individual observations, as demonstrated for example in chemometrics by works from martens and colleagues  <cit>  or westad and collaborators  <cit> ).
in the current paper, jackknifing was used to detect influential samples and outliers in microarray data sets.
the number of resampled data sets created by jackknifing equals the number of samples in the original data set.
each new data set is identical to the initial data set except for one sample which is removed.
in a data set including n samples, n consecutive analyses are performed providing n sets of n -  <dig> sample coordinates.
the impact of each individual sample on the other n -  <dig> samples is measured by the distance from the samples' original positions to their positions after resampling.
if a given sample is highly influential, it may importantly impact the position of one or several other samples.
a stability plot can be used to visualize the shift in the sample position after jackknifing.
a large shift reflects the presence of an influential observation.
in order to identify observations which significantly influenced the position of other observations, the classical multivariate detection of outliers based on the mahalanobis squared distance  was used.
these distances can be evaluated using a χ <dig> distribution with the appropriate degrees of freedom.
each time a sample removal induced a shift to an extra sample di2>χ <dig> ,p <dig>  the  <dig>  quantile of a chi-square distribution, with p degrees of freedom, the sample was defined as significantly influential towards the extra sample.
overall, if the median of the n -  <dig> shifts induced by an observation is greater than a χ <dig> ,p <dig> threshold, this observation was declared an outlier.
similarly to total bootstrap, jackknife outcomes are potentially subjected to axis reflection.
sample coordinates were post multiplied by - <dig> in case of negative correlation between the principal components prior and after resampling.
algorithm implementation
the implementation was done using r , and a new package multistab including original functions was developed.
our algorithm involves resampling techniques which are computationally intensive.
the implementation allows the parallelization of the calculations.
the r packages snow and rmpi allow accessing the mpi/lam subsystem, an implementation of the mpi standard, for distributing jobs among nodes.
calculations have been performed and tested on different configurations including computers with single and dual core cpus as well as an mpi cluster of  <dig> heterogeneous nodes.
example of application
data sets
three publicly available data sets were used to illustrate the different features of our methodology.
the first data set consists of a subset of data from the pioneer work of bhattacharjee and colleagues  <cit>  using gene expression profiling to investigate adenocarcinoma subclasses.
the subset used in the current study includes  <dig> samples classified into  <dig> groups of patients .
rna extracts from tissue specimens  were hybridized onto affymetrix' hgu95a arrays.
the second data set was published by spira and colleagues  <cit>  analyzing the airway epithelial cell transcriptome of smoking patients.
in this study,  <dig> individuals classified into  <dig> groups  were investigated.
bronchial cells were obtained from brushings of the right mainstem bronchus and rna extracts were hybridized onto affymetrix' hgu133a arrays.
the third data set was described by baty and colleagues  <cit>  which investigated the effect of beverage consumption in healthy individuals.
one hundred and eight samples classified in  <dig> groups  were analyzed.
rna extracts from peripheral blood leukocytes were hybridized onto affymetrix' hgu133a arrays.
the transcriptomics signal is expected to be high in bhattacharjee , intermediate in spira , and low in baty .
significance of gene contributions
five hundred bootstrapped samples were built in order to assess the empirical distribution of the gene contributions within each data set.
partial bootstrap was used and the newly calculated gene coordinates were represented with convex hulls on the factorial map defined by the two first principal axes of the bga .
the size of the convex hulls, measured in terms of relative inertia, gives indications about the gene stability.
the discriminative power of genes is measured by the distance from the center of the bga axes to the hull locations.
only the  <dig> most discriminating genes for each data set are displayed.
the degree of hull overlap for the different combinations of factor levels, gives important indication about the specificity of the gene discrimination.
the boxplots of gene contributions  was used to estimate the proportion of overly unstable genes .
the proportion of overly unstable genes is refered to as false positive rate.
gene hulls in bhattacharjee were small , distant from the center of the plot and the group of genes associated with each factor level did not overlap.
this documents the stability and specificity of the markers found in this experiment.
this result was further confirmed by the boxplot representation showing the distributions of gene contributions .
in bhattacharjee, all most discriminating genes significantly contributed to the class discrimination since no distribution crossed the  <dig> threshold .
thus, the false positive rate was 0%.
on the other hand, in the data sets of spira and baty, the relative inertia of gene hulls was larger .
in the data set of baty, the degree of hull overlapping was particularly high.
the level of false positive rate was moderate in spira data  and high in baty data .
this level differed from one experimental condition to one another.
in spira, the "former smoker" group had a higher false positives rate  compared to "current smoker"  and "never smoker" .
the "current smoker" category was the one with the most stable gene signals.
in baty, the false positive rate was high.
genes associated with the consumption of water were highly unstable .
overall the false positive rate was measured for the  <dig> genes with the highest ranking in terms of gene contribution.
as to be expected, this rate increased with lower gene ranks.
identification of influential observations and outliers
jackknifing was used to estimate the influence of each observation on the sample coordinates.
the stability plots presented in the upper panels of figure  <dig> display the sample coordinates  prior and after resampling in the two dominant axes of bga.
the n -  <dig> positions of each sample were represented with star plots showing the shifts induced by sample removals.
in bhattacharjee data, the relative shifts induced by jackknifing were small .
these shifts were much more prominent in spira and baty data sets.
the barplots in figure  <dig>  display graphically the degree of influence of each sample.
for example, the first sample in bhattacharjee was found to influence significantly the position of  <dig> different samples.
by comparing the median shift induced by each single sample with a χ <dig> ,p <dig> threshold, no sample was declared an outlier in both bhattacharjee and spira data sets, whereas  <dig> samples were found to be outliers in baty.
in spira, one can easily observe that the influential observations are mostly identified in the patient groups "former smoker" and "never smoker".
the ouliers in baty belonged to the "baseline" and "wine" group.
comparison with alternative approaches
the relevance of the results obtained by bootstrapped bga was compared with those obtained by bootstrapped correspondence analysis  <cit>  and the gene-by-gene fit of 1-way anova models.
the 'sarcoidosis' data set provided in our r package was used in this comparative study.
this data set, which was previously described  <cit> , included  <dig> individuals .
the group of sarcoidosis patients was subdivided into  <dig> sarcoidosis stage i  and  <dig> sarcoidosis stage ≥ ii  patients.
messenger rna extracted from peripheral blood were hybridized on affymetrix' hgu95a arrays.
the data were normalized using the 'vsn' algorithm.
after gene filtration,  <dig> genes remained in the data set.
bootstrapped bga was performed using  <dig> iterations and  <dig> highly significant genes  among the genes which mostly participate to the between group discrimination were selected.
hundred and twelve genes with the highest contribution on the first axis in bootstrapped ca were selected.
a third list of genes included the  <dig> most significant genes obtained by anova models.
there were  <dig> genes overlapping between the list from bootstrapped ca and bootstrapped bga,  <dig> genes beween bootstrapped bga and anova and  <dig> between bootstrapped ca and anova.
a functional analysis was carried out on these  <dig> lists of genes using the web tool david  <cit> .
table  <dig> summarizes the  <dig> significantly enriched gene ontology  categories found from the list of genes obtained in bootstrapped bga.
the number of go categories identified before adjustment for p-values was slightly more diverse in bootstrapped bga  than in bootstrapped ca or anova .
on the other hand, the enrichment of biologically informative genes in each go categories was higher in bootstrapped bga than in bootstrapped ca and anova.
this is in agreement with previously reported findings  <cit> .
the gene ontology analysis is based on the  <dig> most significant go categories  obtained after bootstrapped bga.
the results present the proportion of genes that belong to these go categories  and the enrichment significance , after bootstrapped bga, bootstrapped ca and anova.
the advantage of using multivariate versus univariate approaches in highly multivariate data such as microarray data has been already well documented in the literature.
the extraction of meaningful gene mechanisms implies that genes are treated as a whole and not separately.
this explains why higher enrichment of biologically meaningful go categories  are generally obtained when using multivariate approaches compared with univariate approaches.
the choice of using unsupervised or supervised ordination methods mainly depends on the objectives of the study.
when the biologist is interested in finding groups of discriminating genes that explain differences among well-defined patients categories, bga is a method which should be considered.
because the grouping information is directly incorporated in the bga model, the dimension reduction of the multivariate data is driven by the phenotypic information.
this generally provides noticeable simplification of data interpretation.
in contrast, when using unconstrained ca, one first extracts the major compositional variation present in the leading axes then relates this variation to external information.
in certain situations, the leading axes are subject to unexpected sources of variation, making their interpretation difficult.
when the design of experiment tends to become more complex, , other multivariate approaches can be used to incorporate this external information into a constrained ca model .
double constrained ca can also be suitable if one wants to incorporate external information on both rows and columns  <cit> .
however, some more complex models of constrained ca including interactions and contrasts might be difficult to interpret.
when the number of constraints tends to be too large, a phenomenon of relaxation of constraints may happen.
in such cases, simple unconstrained ca might appear appropriate.
discussion
the bootstrapping technique presented in this study proved to be very useful to evaluate gene stability in microarray data.
the three data sets were chosen with an a priori knowledge about the strength of their transcriptomics signal.
the biologically most well-defined data set was bhattacharjee with four clearly distinct categories of patients and samples taken from distinct types of lung tumors.
the class discrimination was high  and the gene signal very stable and specific.
a less stable signal came out of spira data, where samples were derived from a mixture of airway epithelial cells and the patient groups less clear cut.
the identification of stable signals specific to "former smoker" was particularly difficult although the between-class discrimination was significant .
the weakest and highly unstable signals were found in baty data, where the expected effect was within the physiological range of normal cells.
the proportion of explained variance was very low in this example .
jackknifing was particularly efficient to detect influential observations or outliers in our setting.
this method provided important diagnostic insights in the data as well as the experimental design.
a careful exploration of the sample stability can help the experimenter to identify samples with imprecise or wrong group allocation, or group with a heterogeneous behaviour.
furthermore, this method can be used to identify poorly specified sample categories or subcategories of samples.
as an example, further investigations might reveal that the "former smoker" category in spira might share gene signatures both from "never smoker" and "current smoker".
with the proposed tools, researchers can identify inconsistent observations/samples or groups and have a strategy at hand to correct for imprecise descriptions in case of sufficient respective evidence.
particular attention was paid to the computational aspects of the resampling calculations.
the calculations were considerably accelerated by the use of parallelization.
furthermore, routines used to carry out ca and bga  have been optimized for the analysis of gene expression data where the number of variables far exceeds the number of samples.
performance testing on data sets of different size showed an improvement of calculation time by a factor  <dig> to  <dig>  as previously mentioned, the method of partial bootstrap was prefered to total bootstrap for testing the stability of gene contributions since it was simpler to perform and computationally more efficient .
CONCLUSIONS
dimension reduction methods are powerful tools that help biologists exploring their data and generate new hypotheses.
like other supervised approaches, constrained ordination methods incorporate external information that greatly simplifies the interpretation of microarray data analysis.
the principal axes of bga being defined as the linear combination of genes that maximizes the between-group variance, it is straightforward to extract groups of genes that discriminate between disease categories.
by using the resampling methodology described above, it is possible to assess the reliability of solutions in a multivariate analysis of gene expression microarray data.
although both bootstrapping and jackknifing should not be used for formal statistical hypothesis testing, they proved to be useful to identify highly consistent genes, filter out some false positive genes, and to allow detection of influential observations among samples.
with regard to this complementary information, the biologist can decide to pay more attention to highly stable discriminating genes, which in turn can be used for subsequent formal statistical hypothesis testing.
based on jackknifing information, the biologist can also decide on croping outlying observations or refining a priori sample classification.
in conclusion, the methodology and the collection of tools proposed in this study are suitable for the assessment of the significance of gene contributions and the detection of outliers in microarray data and this in a multivariate fashion.
the set of r functions includes additional functions which test the stability of multivariate analysis results.
overall, the r package we developed constitutes a novel and comprehensive suite of diagnostic tools to evaluate the robustness of multivariate representations of high-throughput gene expression data.
authors' contributions
fb developed the algorithm, performed the analysis and wrote the paper.
dj and fp contributed to the implementation and the optimization of the algorithm.
mms contributed in the development of the algorithm.
mhb supervised the work and was significantly involved in the writing of the manuscript.
all co-authors read and approved the manuscript.

 ordination methods are useful tools for the analysis of gene expression microarrays. principal component analysis and correspondence analysis have both been used to extract the main sources of variation present in highly multivariate microarray data. the supervised counterparts of these approaches, including between-group analysis and analyses with respect to instrumental variables, were proposed to handle descriptive variables controlled in the design of the experiment. when dealing with transcriptomics data, multivariate approaches are generally more appropriate than univariate strategies because they intrinsically take gene covariations and interactions into account. constrained ordination methods are very efficient for sample classification and class prediction. they are flexible and can be used easily to identify groups of genes associated with classes of samples. geometrical interpretations are generally required to investigate the gene-sample relationship. genes of interest can also be ranked according to their discriminative power. however, considering the exploratory nature of these methods, it is not trivial to assess the significance of a given gene dysregulation in a multivariate setting. these methods rely on solving an eigenvalue problem whose solutions are given by the leading eigenvectors and whose theoretical statistical properties are particularly complex to study. to overcome this issue, resampling techniques have been proposed to estimate the stability of multivariate analyses. these techniques were described in a variety of scientific frameworks including environmetrics, chemometrics, and archaeology. the general purpose is to develop inferential procedures for testing the statistical significance of the parameters provided by these exploratory techniques. their applications are manifold, e.g. assessing which variables significantly contribute to the principal axes of a pca, detecting outliers or influential observations. this approach has a great potential in the context of microarray data analysis as proposed by tan and collaborators. these authors described an application of bootstrapping to correspondence analysis. they outlined that their approach would have several advantages over classical gene-by-gene fits of anova models. it particularly enables the extraction of lists of genes which are biologically more informative than those found by anova. in the present work, we propose a specific methodology for testing the stability of constrained ordination methods applied to microarray data. unlike previous studies, our approach is dedicated to supervised multivariate analyses. to our knowledge, very few studies addressed the issue of stability assessment in supervised multivariate analyses. the potential of associating stability analysis in the supervised multidimensional context is multiple. by using the information of sample descriptors, genes can be associated with a given class of samples and the significance of this association can be assessed. a derived significance testing strategy regarding gene contributions is proposed. further resampling methods based on jackknifing are also proposed to identify influential observations as an aid in outlier detection in microarray data sets. a comprehensive set of r functions illustrating our methodology was developed. the package is freely available on request. the present manuscript is organized as follows. the first section introduces some theoretical aspects of ordination methods and constrained ordination methods. the subsequent sections describe the different resampling strategies used in this project, as well as details about the algorithm. illustrative examples demonstrating the implemented technique are given. methods and  theory ordination methods both pca and ca are commonly used in microarray data analysis. some authors stressed that ca has several advantages over pca. like other dimension reduction methods, ca summarizes structures in high-dimension space by projection onto a low dimension sub-space while loosing as little information as possible. correspondence analysis involves a first step of symmetrical data transformation into a chi-square distance matrix which makes ca outputs particularly appropriate for the exploration of relationships between samples and genes. the mathematical basis of ca has been described elsewhere and will be briefly summarized. thereafter observations are shown as rows and variables as columns. let us define the following: • y: the matrix of gene expression data • p = y/n: the data matrix divided by its grand total • r: the n-dim vector of row sums of p • c: the m-dim vector of column sums of p • dr = diag: the diagonal matrix of row sums • dc = diag: the diagonal matrix of column sums the correspondence analysis of y is obtained by carrying out the singular value decomposition of the doubled centered and standardized matrix z: z=dr−1/2dc−1/2=uΛvt with Λ the k × k ) diagonal matrix of singular values associated with z with λ ≥... ≥ λk > u an matrix whose columns are the left singular vectors of z and v an matrix whose columns are the right singular vectors of z. the rows of u and v are orthonormal with respect to dr and dc respectively: utdru = vt dcv = i the principal components and row coordinates are respectively given by dr−1/2u and dr−1/2uΛ. the principal axes and column coordinates are respectively given by dc−1/2v and dc−1/2vΛ. constrained ordination methods in microarray experiments, besides the main table y containing the gene expression values, additional descriptive variables x controlled in the experimental design generally characterize samples. constrained ordination methods aim to display the variations in the data which are explained by the descriptive variables. these two-table methods are dissymmetric because the information from x is used to constrain the analysis of y. correspondence analysis with respect to instrumental variables, which is closely related to pca on instrumental variables, and between-group correspondence analysis are two examples of constrained correspondence analysis which have been successfully applied to microarray data. in this field, these methods have been used for different purposes including sample classification, disease-class prediction, and removal of undesirable effects. bga is a particular case of caiv when samples are characterized by one single categorical variable. between-group correspondence analysis of table y given the class descriptor x is simply the correspondence analysis of the table Ŷ corresponding to the table of means per group. bga is the analysis of the per-class centroids. it provides up to g - discriminating axes. the initial samples are thereafter projected as supplementary rows in the bga sub-space. the bga procedure provides the best linear combination of variables which maximizes the between-group variance. stability of gene contributions using bootstrapping one interesting feature of bga is the possibility to associate genes with pre-defined sample classes. the gene contribution towards sample classes is defined as the absolute distance from the center of the bga axes to the orthogonal projections of the gene onto the vector of the class centroid. this distance is measured by the parameter: α0=〈x,y〉‖x‖ the higher |α0| is, the more important the gene contribution is. the sign of α indicates whether the gene is specifically up- or down-regulated, respectively for positive and negative values. genes can be ranked according to their contribution towards each class modalities. introduced by efron, bootstrapping is a distribution-free resampling method generally used to estimate the variance of estimators. like other resampling techniques, bootstrapping provides a good alternative to establish the variability of an estimator. it is classically used to assess the bias and variance of model parameters, construct confidence intervals and rebuild empirical distributions. several bootstrap refinements were implemented in the framework of techniques incorporating singular value decomposition. in the present work, non-parametric bootstrapping was used to estimate the significance of gene contributions. the proposed bootstrapping strategy is based on the bga model assumptions. as mentioned previously, the bga of table y with regard to the categorical variable x, is the analysis of the table Ŷ corresponding to the per-class mean of y. let us define x, the table of dummy variables coded from x and Ŷx = x-1xty, the matrix of fitted values. bootstrapped samples are built based on the residuals e = y - Ŷx. residuals are sampled with replacement and new data sets are built as follow: y* = Ŷx + e*. the analysis of to perturbated data sets is generally required to assess the parameters' distribution of a multivariate model. out of these empirical distributions several indicators of the stability of gene contributions were calculated. non-parametric 95% bootstrap confidence intervals were constructed using the % and % quantiles of the bootstrapped distribution. z-scores were defined for each gene as the ratio of the bootstrap estimates to its standard deviation. p-values were estimated according to the "bootstrapped eigenvector" procedure as the probability of obtaining gene contributions equal to or smaller than zero for genes contributing positively in the original data set, or alternatively equal to or larger than zero for genes contributing negatively in the original data set. convex hulls were used to graphically display the spread of the bootstrapped gene coordinates on the dominant principal axes. the relative inertia of these gene coordinates was measured by the ratio of gene inertia to the total inertia. lebart proposed two main categories of bootstrapping. partial bootstrap makes use of a posteriori projections of the resampled elements on the original reference subspace provided by the analysis of the initial data set. total bootstrap performs a new analysis on each of the resampled data sets. both strategies have been implemented in the current project. it is noteworthy that partial bootstrap does not involve successive steps of singular value decomposition which makes it considerably faster than total bootstrap. moreover, because total bootstrap requires a complete ca to be carried out for each bootstrapped table, the new sets of row and column coordinates belong to different subspaces which make their comparison more complex. unlike partial bootstrap, total bootstrap is potentially subjected to axis reflection or inversion. signs of row and column coordinates in perturbated data sets can be inverted compared with the original data set. at least two approaches have been reported to overcome this drawback. the first consists in determining the sign of the correlations between the principal axes prior and after perturbations. a negative correlation indicates a reflexion which can be corrected by multiplying the new row and column coordinates by - procrustes rotation was also proposed to fit the resampled row and column coordinates with the original scores and loadings. the first option was implemented in the present work. detection of outliers using jackknifing jackknifing is another resampling technique introduced by quenouille and later developed by tukey. jackknifed samples are built using a leave-one-sample-out strategy. although jackknifing can be seen as a rough linear approximation of bootstrapping, it proves to be useful for investigating the influence of individual observations, as demonstrated for example in chemometrics by works from martens and colleagues or westad and collaborators ). in the current paper, jackknifing was used to detect influential samples and outliers in microarray data sets. the number of resampled data sets created by jackknifing equals the number of samples in the original data set. each new data set is identical to the initial data set except for one sample which is removed. in a data set including n samples, n consecutive analyses are performed providing n sets of n - sample coordinates. the impact of each individual sample on the other n - samples is measured by the distance from the samples' original positions to their positions after resampling. if a given sample is highly influential, it may importantly impact the position of one or several other samples. a stability plot can be used to visualize the shift in the sample position after jackknifing. a large shift reflects the presence of an influential observation. in order to identify observations which significantly influenced the position of other observations, the classical multivariate detection of outliers based on the mahalanobis squared distance was used. these distances can be evaluated using a χ distribution with the appropriate degrees of freedom. each time a sample removal induced a shift to an extra sample di2>χ,p the quantile of a chi-square distribution, with p degrees of freedom, the sample was defined as significantly influential towards the extra sample. overall, if the median of the n - shifts induced by an observation is greater than a χ,p threshold, this observation was declared an outlier. similarly to total bootstrap, jackknife outcomes are potentially subjected to axis reflection. sample coordinates were post multiplied by - in case of negative correlation between the principal components prior and after resampling. algorithm implementation the implementation was done using r, and a new package multistab including original functions was developed. our algorithm involves resampling techniques which are computationally intensive. the implementation allows the parallelization of the calculations. the r packages snow and rmpi allow accessing the mpi/lam subsystem, an implementation of the mpi standard, for distributing jobs among nodes. calculations have been performed and tested on different configurations including computers with single and dual core cpus as well as an mpi cluster of heterogeneous nodes. example of application data sets three publicly available data sets were used to illustrate the different features of our methodology. the first data set consists of a subset of data from the pioneer work of bhattacharjee and colleagues using gene expression profiling to investigate adenocarcinoma subclasses. the subset used in the current study includes samples classified into groups of patients. rna extracts from tissue specimens were hybridized onto affymetrix' hgu95a arrays. the second data set was published by spira and colleagues analyzing the airway epithelial cell transcriptome of smoking patients. in this study, individuals classified into groups were investigated. bronchial cells were obtained from brushings of the right mainstem bronchus and rna extracts were hybridized onto affymetrix' hgu133a arrays. the third data set was described by baty and colleagues which investigated the effect of beverage consumption in healthy individuals. one hundred and eight samples classified in groups were analyzed. rna extracts from peripheral blood leukocytes were hybridized onto affymetrix' hgu133a arrays. the transcriptomics signal is expected to be high in bhattacharjee, intermediate in spira, and low in baty. significance of gene contributions five hundred bootstrapped samples were built in order to assess the empirical distribution of the gene contributions within each data set. partial bootstrap was used and the newly calculated gene coordinates were represented with convex hulls on the factorial map defined by the two first principal axes of the bga. the size of the convex hulls, measured in terms of relative inertia, gives indications about the gene stability. the discriminative power of genes is measured by the distance from the center of the bga axes to the hull locations. only the most discriminating genes for each data set are displayed. the degree of hull overlap for the different combinations of factor levels, gives important indication about the specificity of the gene discrimination. the boxplots of gene contributions was used to estimate the proportion of overly unstable genes. the proportion of overly unstable genes is refered to as false positive rate. gene hulls in bhattacharjee were small, distant from the center of the plot and the group of genes associated with each factor level did not overlap. this documents the stability and specificity of the markers found in this experiment. this result was further confirmed by the boxplot representation showing the distributions of gene contributions. in bhattacharjee, all most discriminating genes significantly contributed to the class discrimination since no distribution crossed the threshold. thus, the false positive rate was 0%. on the other hand, in the data sets of spira and baty, the relative inertia of gene hulls was larger. in the data set of baty, the degree of hull overlapping was particularly high. the level of false positive rate was moderate in spira data and high in baty data. this level differed from one experimental condition to one another. in spira, the "former smoker" group had a higher false positives rate compared to "current smoker" and "never smoker". the "current smoker" category was the one with the most stable gene signals. in baty, the false positive rate was high. genes associated with the consumption of water were highly unstable. overall the false positive rate was measured for the genes with the highest ranking in terms of gene contribution. as to be expected, this rate increased with lower gene ranks. identification of influential observations and outliers jackknifing was used to estimate the influence of each observation on the sample coordinates. the stability plots presented in the upper panels of figure display the sample coordinates prior and after resampling in the two dominant axes of bga. the n - positions of each sample were represented with star plots showing the shifts induced by sample removals. in bhattacharjee data, the relative shifts induced by jackknifing were small. these shifts were much more prominent in spira and baty data sets. the barplots in figure display graphically the degree of influence of each sample. for example, the first sample in bhattacharjee was found to influence significantly the position of different samples. by comparing the median shift induced by each single sample with a χ,p threshold, no sample was declared an outlier in both bhattacharjee and spira data sets, whereas samples were found to be outliers in baty. in spira, one can easily observe that the influential observations are mostly identified in the patient groups "former smoker" and "never smoker". the ouliers in baty belonged to the "baseline" and "wine" group. comparison with alternative approaches the relevance of the results obtained by bootstrapped bga was compared with those obtained by bootstrapped correspondence analysis and the gene-by-gene fit of 1-way anova models. the'sarcoidosis' data set provided in our r package was used in this comparative study. this data set, which was previously described, included individuals. the group of sarcoidosis patients was subdivided into sarcoidosis stage i and sarcoidosis stage ≥ ii patients. messenger rna extracted from peripheral blood were hybridized on affymetrix' hgu95a arrays. the data were normalized using the'vsn' algorithm. after gene filtration, genes remained in the data set. bootstrapped bga was performed using iterations and highly significant genes among the genes which mostly participate to the between group discrimination were selected. hundred and twelve genes with the highest contribution on the first axis in bootstrapped ca were selected. a third list of genes included the most significant genes obtained by anova models. there were genes overlapping between the list from bootstrapped ca and bootstrapped bga, genes beween bootstrapped bga and anova and between bootstrapped ca and anova. a functional analysis was carried out on these lists of genes using the web tool david. table summarizes the significantly enriched gene ontology categories found from the list of genes obtained in bootstrapped bga. the number of go categories identified before adjustment for p-values was slightly more diverse in bootstrapped bga than in bootstrapped ca or anova. on the other hand, the enrichment of biologically informative genes in each go categories was higher in bootstrapped bga than in bootstrapped ca and anova. this is in agreement with previously reported findings. the gene ontology analysis is based on the most significant go categories obtained after bootstrapped bga. the results present the proportion of genes that belong to these go categories and the enrichment significance, after bootstrapped bga, bootstrapped ca and anova. the advantage of using multivariate versus univariate approaches in highly multivariate data such as microarray data has been already well documented in the literature. the extraction of meaningful gene mechanisms implies that genes are treated as a whole and not separately. this explains why higher enrichment of biologically meaningful go categories are generally obtained when using multivariate approaches compared with univariate approaches. the choice of using unsupervised or supervised ordination methods mainly depends on the objectives of the study. when the biologist is interested in finding groups of discriminating genes that explain differences among well-defined patients categories, bga is a method which should be considered. because the grouping information is directly incorporated in the bga model, the dimension reduction of the multivariate data is driven by the phenotypic information. this generally provides noticeable simplification of data interpretation. in contrast, when using unconstrained ca, one first extracts the major compositional variation present in the leading axes then relates this variation to external information. in certain situations, the leading axes are subject to unexpected sources of variation, making their interpretation difficult. when the design of experiment tends to become more complex,, other multivariate approaches can be used to incorporate this external information into a constrained ca model. double constrained ca can also be suitable if one wants to incorporate external information on both rows and columns. however, some more complex models of constrained ca including interactions and contrasts might be difficult to interpret. when the number of constraints tends to be too large, a phenomenon of relaxation of constraints may happen. in such cases, simple unconstrained ca might appear appropriate. discussion the bootstrapping technique presented in this study proved to be very useful to evaluate gene stability in microarray data. the three data sets were chosen with an a priori knowledge about the strength of their transcriptomics signal. the biologically most well-defined data set was bhattacharjee with four clearly distinct categories of patients and samples taken from distinct types of lung tumors. the class discrimination was high and the gene signal very stable and specific. a less stable signal came out of spira data, where samples were derived from a mixture of airway epithelial cells and the patient groups less clear cut. the identification of stable signals specific to "former smoker" was particularly difficult although the between-class discrimination was significant. the weakest and highly unstable signals were found in baty data, where the expected effect was within the physiological range of normal cells. the proportion of explained variance was very low in this example. jackknifing was particularly efficient to detect influential observations or outliers in our setting. this method provided important diagnostic insights in the data as well as the experimental design. a careful exploration of the sample stability can help the experimenter to identify samples with imprecise or wrong group allocation, or group with a heterogeneous behaviour. furthermore, this method can be used to identify poorly specified sample categories or subcategories of samples. as an example, further investigations might reveal that the "former smoker" category in spira might share gene signatures both from "never smoker" and "current smoker". with the proposed tools, researchers can identify inconsistent observations/samples or groups and have a strategy at hand to correct for imprecise descriptions in case of sufficient respective evidence. particular attention was paid to the computational aspects of the resampling calculations. the calculations were considerably accelerated by the use of parallelization. furthermore, routines used to carry out ca and bga have been optimized for the analysis of gene expression data where the number of variables far exceeds the number of samples. performance testing on data sets of different size showed an improvement of calculation time by a factor to as previously mentioned, the method of partial bootstrap was prefered to total bootstrap for testing the stability of gene contributions since it was simpler to perform and computationally more efficient.  dimension reduction methods are powerful tools that help biologists exploring their data and generate new hypotheses. like other supervised approaches, constrained ordination methods incorporate external information that greatly simplifies the interpretation of microarray data analysis. the principal axes of bga being defined as the linear combination of genes that maximizes the between-group variance, it is straightforward to extract groups of genes that discriminate between disease categories. by using the resampling methodology described above, it is possible to assess the reliability of solutions in a multivariate analysis of gene expression microarray data. although both bootstrapping and jackknifing should not be used for formal statistical hypothesis testing, they proved to be useful to identify highly consistent genes, filter out some false positive genes, and to allow detection of influential observations among samples. with regard to this complementary information, the biologist can decide to pay more attention to highly stable discriminating genes, which in turn can be used for subsequent formal statistical hypothesis testing. based on jackknifing information, the biologist can also decide on croping outlying observations or refining a priori sample classification. in conclusion, the methodology and the collection of tools proposed in this study are suitable for the assessment of the significance of gene contributions and the detection of outliers in microarray data and this in a multivariate fashion. the set of r functions includes additional functions which test the stability of multivariate analysis results. overall, the r package we developed constitutes a novel and comprehensive suite of diagnostic tools to evaluate the robustness of multivariate representations of high-throughput gene expression data. authors' contributions fb developed the algorithm, performed the analysis and wrote the paper. dj and fp contributed to the implementation and the optimization of the algorithm. mms contributed in the development of the algorithm. mhb supervised the work and was significantly involved in the writing of the manuscript. all co-authors read and approved the manuscript. 
in this study, we developed a computationally efficient algorithm for: i) the assessment of the significance of gene contributions and ii) the identification of sample outliers in multivariate analysis of microarray data. multivariate ordination methods are powerful tools for the exploration of complex data structures present in microarray data. overall, it proved to be particularly effective for the evaluation of the stability of microarray data. this package includes tools for both inferring the statistical significance of gene contributions and identifying outliers among samples. however, due to their exploratory nature, multivariate ordination methods do not allow direct statistical testing of the stability of genes. a statistical package of r functions was developed.
 multivariate ordination methods are powerful tools for the exploration of complex data structures present in microarray data. these methods have several advantages compared to common gene-by-gene approaches. however, due to their exploratory nature, multivariate ordination methods do not allow direct statistical testing of the stability of genes.  in this study, we developed a computationally efficient algorithm for: i) the assessment of the significance of gene contributions and ii) the identification of sample outliers in multivariate analysis of microarray data. the approach is based on the use of resampling methods including bootstrapping and jackknifing. a statistical package of r functions was developed. this package includes tools for both inferring the statistical significance of gene contributions and identifying outliers among samples.  the methodology was successfully applied to three published data sets with varying levels of signal intensities. its relevance was compared with alternative methods. overall, it proved to be particularly effective for the evaluation of the stability of microarray data. 