BACKGROUND
retinal photoreceptor cells, the specialized cells involved in light detection and phototransduction, are essential for mammalian vision.
many retinal diseases occur as a result of inherited dysfunction of the rod and cone photoreceptor cells.
photoreceptor degeneration, for example, constitutes an important cause of visual impairment affecting all age groups and ethnic backgrounds  <cit> .
development and maintenance of photoreceptor function in the retina requires appropriate regulation of gene expression, especially for genes specifically or highly expressed in photoreceptor cells during retinal development .
comprehensive identification of photoreceptor-enriched gene expression patterns may have important implications in neurobiology, leading to a better understanding of molecular mechanisms of retinal development, the improvement of diagnosis of complex retinal diseases, and the identification of potential therapeutic targets  <cit> .
over the last decades, different experimental approaches have been developed to identify retinal disease genes.
using microarray data analysis, for example, yoshida et al.
<cit>  revealed that  <dig> genes, which are differentially expressed in the absence of nrl , are either associated with or are candidates for retinal diseases involving rod or cone photoreceptor dysfunction.
katsanis et al.
<cit>  positioned  <dig> expressed sequence tags  likely to be specifically or preferentially expressed in the retina.
they also identified positional candidate genes for  <dig> of  <dig> uncloned retinopathies.
the quality of the results was assessed by reverse transcriptase-polymerase chain reaction .
recently, blackshaw et al.
<cit>  presented a comprehensive genomic analysis of mouse retinal development using serial analysis of gene expression , followed by in situ hybridization  validation.
libraries were obtained from microdissected mouse photoreceptors from the retinal outer nuclear layers , retina from various mouse developmental stages and retina from the paired-homeodomain transcription factor crx knockout mouse  and its wild type counterpart  at postnatal day  <dig>  and from nih3t <dig> mouse fibroblasts.
the sage-based expression analysis performed by blackshaw et al.
<cit>  has advantages over other rna analysis methods.
this technique uses a unique sequence tag of  <dig> or more bases isolated from a defined position within each transcript  <cit> .
the basic concept of sage rests on two principles:  a short nucleotide sequence tag contains sufficient information to uniquely identify a transcript and  concatenated short sequence tags can be cloned to facilitate efficient sequencing analysis.
unlike rna blotting and rt-pcr, sage is not limited to examining only a few known genes at a time.
unlike dna microarray technology, the sage approach allows the simultaneous analysis of a large number of transcripts without prior, complete knowledge of the sequence of the genes  <cit> .
in the study by blackshaw et al.
<cit> , the candidate photoreceptor-specific genes were selected by comparative analysis between sage libraries on the basis of four chosen criteria .
this method, however, has shown relatively low true positive and true negative prediction rates.
for example, out of  <dig> newly-identified photoreceptor-enriched  tags, only approximately 20% meet all the four classification criteria and about 44% meet more than three of the four criteria.
to further identify true pr-enriched tags, blackshaw et al.
experimentally validated the candidate tags through exhaustive ish analysis.
in this paper, we explore the feasibility of using computational approaches to support large-scale analysis of photoreceptor gene expression data.
the main purpose of this study is to implement several in silico models to detect complex features and relationships encoded in the sage data, which may be used to predict functional associations.
by way of illustration, this paper focuses on the classification of two functional classes of genes, which were studied and experimentally validated by blackshaw et al.
<cit> : pr-enriched and non-pr-enriched genes.
the main question was: can machine learning-based classifiers be built to accurately distinguish pr-enriched from non pr-enriched genes solely based on patterns in the sage data?
the potential benefits of this approach are two-fold.
in situations for which there is insufficient biological knowledge, machine learning-based classifiers could be used to predict functional classes of genes.
secondly, if classifiers are demonstrated to effectively predict specific gene types, these empirically derived relationships could be used to derive biological significance.
in order to study temporal patterns of photoreceptor gene expression during retinal development, a two-way cluster analysis: clustering of tags and clustering of libraries were performed.
while sage libraries were clustered using traditional hierarchical clustering method, sage tags were analysed by a newly developed poisson model-based k-means algorithm  specifically designed for sage data  <cit> .
to address one of the limitations of poissonc, a figure of merit -based approach  <cit>  to estimating the number of clusters was proposed.
the fom is computed by first removing one experiment , clustering genes based on the remaining data, and then measuring how the left-out library fits the expression patterns obtained from the other libraries.
such adaptations represent one of the aspects distinguishing our investigation from traditional clustering-based analyses.
to explore the feasibility of machine-learning approaches to predicting functional associations encoded in the sage data generated from different developmental stages in mouse retina, three supervised classification methods were tested to predict the two functional classes investigated by blackshaw et al.
<cit> : pr-enriched and non-pr-enriched genes.
to deal with the problem of functional class imbalance occurring in the dataset, two data re-sampling techniques were studied.
to adequately evaluate the performance of the supervised classifiers in imbalanced dataset, three predictive quality indicators that are independent of the class prior probabilities were implemented.
to further assess the statistical significance of the computational approaches, a 100-run permutation test was implemented.
the reader is referred to the section of methods for a more detailed description of the data sets and techniques studied.
the following section summarises relevant results.
RESULTS
clustering of sage libraries
clustering of sage tags
in general, the temporal expression patterns observed in this sage data reflect previously characterized photoreceptor gene expression patterns in the developing retina.
for example, genes associated with tags in clusters  <dig> and  <dig>  whose expression consistently increases throughout postnatal development and reaches their highest value in the adult retina, are highly likely to be expressed in developing photoreceptors.
sixty out of  <dig> tags previously known to be pr-enriched and  <dig> out of  <dig> tags identified to be pr-enriched by blackshaw et al.
<cit>  fell within these two clusters, highlighting that this is an important feature which can be used to identify pr-enriched genes.
a closer examination of these two clusters revealed that many tags were mapped to genes with known functions relevant to phototransduction and visual formation.
for example,  <dig> tags associated with rhodopsin gene and  <dig> tags linked to rod photoreceptor were grouped together in cluster  <dig>  four tags mapped to guanine nucleotide binding proteins   <cit> , which are involved as modulators or transducers in various transmembrane signalling systems, were all found in cluster  <dig>  apart from those tags directly related to the visual process, some tags associated with genes with other functions such as peripherin- <dig> and rod outer segment membrane protein  <dig>  were also found.
it has been shown that these genes are involved in maintaining the integrity of photoreceptor outer segment and are therefore critical for rod photoreceptor viability and regulation of disk morphogenesis  <cit> .
interestingly,  <dig> out of the  <dig> known non-pr-enriched tags were found in cluster  <dig>  having peak expression values occurring within 3t <dig> fibroblast cells.
this might confirm that those genes that have higher expression levels in non-retina tissues are unlikely to become pr-enriched genes.
cluster  <dig> exhibits similar expression profiles of genes strongly expressed in adult hypothalamus.
although the characteristics of all  <dig> tags in this cluster were not investigated by blackshaw et al.
<cit> , our analysis indicates that these genes are unlikely to be selectively expressed in photoreceptors based on the observations derived from cluster  <dig>
clusters  <dig> and  <dig>  whose peak expression values occurred during embryonic development, also offered relevant insights.
genes that fell within these two clusters generally have higher expression levels in the mouse retina before p <dig>  with expression gradually decreasing throughout later postnatal development.
examples includes macrophage migration inhibitory factor gene in cluster  <dig>  which plays an important role in t-cell activation and may contribute to regulation of retinal inflammation and its local immunity  <cit> .
its expression starts at e <dig> , peaks at e <dig> , and significantly decreases in the adult.
in clusters  <dig> and  <dig>  a total of  <dig> tags were examined by blackshaw et al.
<cit> , ten out of  <dig> resulted non-pr enriched.
the expression profile exhibited by these two clusters may serve as template for the detection of non-pr enriched genes.
unlike the other gene clusters, the expression levels of genes within clusters  <dig>   <dig>   <dig> and  <dig> did not follow a consistent trend throughout retinal development.
forty-six tags associated with various pr-enriched genes were found in these four different clusters, reflecting the heterogeneity of photoreceptor gene expression patterns on the basis of the onset and peak time of expression  <cit> .
unlike most of the photoreceptor genes  whose expression values are very low at birth and dramatically increase throughout postnatal retinal development, some of the pr-enriched genes exhibit totally different expression patterns.
in cluster  <dig>  for example, expression of neurod is high at early embryonic stages, peaking at p <dig>  and decreases significantly in the adult.
this might reflect a dual function of neurod in cell specification and in regulation of rod photoreceptor survival  <cit> .
in cluster  <dig>  mertk, a c-mer proto-oncogene tyrosine kinase, exhibits an embryonic expression profile with peak at e <dig>  and is selectively expressed in mature photoreceptor cells.
previous studies have shown that mutations in mertk are responsible for retinal dystrophy  <cit> .
the diversity of photoreceptor expression profiles may reflect the fact that some of pr-enriched genes are involved in more than one biological process.
supervised functional classification of tags
the goal was to determine whether a tag represents a pr-enriched or a non-pr-enriched gene given a set of sage libraries associated with each tag.
among the  <dig> tags representing at least  <dig> % of the total expression in the onl library,  <dig> tags have been identified as pr-enriched genes, which exhibit diverse and complex expression patterns.
this highlights the difficulties in using in silico methods to detect key relationships encoded in the sage data.
such complexities are further stressed when the original data are projected on 3-dimensional space using well-known mapping methods, i.e.
principal component analysis and sammon's mapping, which clearly indicate that the two classes are not linearly separable from each other .
based on a comprehensive analysis of the expression of tags previously known to be pr-enriched, blackshaw et al.
<cit>  introduced four criteria for the selection of candidate pr-enriched genes:
<dig>  tissue specific : the number of tags in either the hypothalamus or 3t <dig> libraries is less than  <dig>
<dig>  developmentally regulated : the sum of tags in the p <dig>  crx+/+ , adult and onl libraries divided by the sum of tags in the e <dig> , e <dig> , and e <dig>  libraries is greater than 10-fold.
<dig>  crx dependent : tags are present at a level greater than  <dig> -fold higher in the p <dig>  crx+/+ library compared to the library of crx-/- mice.
<dig>  onl enriched : tags are present at an equal or greater number in the onl library compared to the whole adult retina library.
we encoded the sage dataset using these criteria to study significant associations between the two functional classes and these criteria.
the apriori algorithm was applied to extract association rules from the sage data.
this algorithm, which was proposed by agrawal and skrikant  <cit> , is a well-known association rule learning algorithm.
given a dataset the apriori algorithm is able to generate association rules that have support and confidence levels greater than user-specified values.
a list of all association rules induced from the  <dig> tags, together with their support and confidence levels, are given in table  <dig>  the distribution of these  <dig> tags on the basis of their compliance with the four criteria is given in the additional file .
the results obtained showed that a high rate of true positives was observed for genes corresponding to tags meeting more than two of the four criteria.
however, when applying these criteria to perform supervised classification, poor prediction results were obtained.
for example, only  <dig> out of  <dig> pr-enriched tags met all four criteria.
from the  <dig> tags identified by blackshaw et al.
<cit>  as pr-enriched, only about 20% met all four criteria and around 44% met more than three criteria.
when applying the four criteria to the data individually, a relatively low rate of true negatives was obtained.
for instance,  <dig> out of  <dig> non-pr-enriched genes met criterion  <dig>   <dig> met criterion  <dig>   <dig> met criterion  <dig>  and  <dig> met criterion  <dig>  therefore, these classification criteria do not represent accurate and robust rules for the classification.
tables  <dig> and  <dig> show the prediction results from 10-fold cross validation for three supervised classifiers using random over- and under-sampling methods respectively.
the section of methods provides a description of these models.
for each classifier, the overall classification accuracy , along with the precision , se and sp for each class, were calculated.
precision is defined as the proportion of predictions that are correct.
the mathematical definitions of these metrics are given in the additional file .
the corresponding roc graphs are depicted in figure  <dig>  for each roc graph, the area under roc curve  was calculated.
the auc has been suggested as a reliable and robust measure for classification performance  <cit> .
a higher value of auc is associated with a classifier that is both effective and robust, i.e.
it presents a better average classification performance across different prediction  thresholds.
tables  <dig> and  <dig> indicate that the classifiers built on data derived from over-sampling methods provided better results than those derived from under-sampling methods in terms of ac, se and sp.
the advantages of random over-sampling techniques can be further demonstrated by the auc values shown in figure  <dig>  this may be explained by the fact that a random under-sampling strategy may throw away potentially useful data.
in addition, we found that the relatively simple kstar  <cit>  algorithm can outperform more complex models such as mlp.
to investigate the effect of class distribution on the classifier, we varied the class distribution using the data over-sampling technique.
table  <dig> shows prediction results for the kstar classifier with different class distributions.
a 10-fold cross validation procedure was carried out to estimate the true classification error.
the corresponding roc curves are given in the additional file .
the kstar with balanced class sample distribution achieved the best results.
to further assess the statistical significance of our computational approaches and their predictive performance, a 100-run permutation test was implemented.
for each permutated dataset, the results were significantly worse than the one generated using the original data in terms of ac, se, and sp, strongly indicating that the relationship between the data and the labels may be reliably learned by the proposed classifiers.
for example, when implementing the permutation test for the kstar model on a balanced dataset and 10-fold cross validation, the random classifiers never performed better than the  prediction model built.
the obtained average values of ac, se and sp  were significantly lower than the results shown in table  <dig> .
similar results were obtained when we performed a permutation test for other classifiers on the dataset with different class distributions.
the complete results of our permutation test can be found in the additional file .
discussion
as the most accessible part of the central nervous system  and as a highly ordered laminar structure, the retina offers unique opportunities to study both the development and physiology of the cns.
this paper described several in silico approaches, including unsupervised and supervised models, to supporting large-scale analysis of photoreceptor gene expression by sage.
by clustering sage libraries, a hierarchical tree reflecting the relationship between the libraries was obtained.
libraries from adjacent developmental periods were generally grouped together as expected.
however, significant discontinuities were identified at the time of birth and between p <dig>  and p <dig> , highlighting important developmental periods.
by clustering sage tags, a more comprehensive expression profile for photoreceptor cells was revealed.
it confirmed that most of the pr-enriched genes may be successfully clustered.
these genes have lower expression levels before birth with expression dramatically increasing throughout postnatal development.
nevertheless, a closer examination of the clustering results revealed that photoreceptor expression patterns are highly heterogeneous and not separable by linear methods.
the diversity of photoreceptor expression profiles reflects the variability in onset of expression which can occur early in development or when photoreceptors undergo terminal differentiation.
with regard to the clustering of sage tags, different algorithms with different distance metrics have been previously proposed.
for example, it has been suggested that by modelling sage data with poisson statistics better results can be achieved.
however, it is relatively computationally expensive and poisson-based distance has only been assessed as part of the k-means algorithm, which exhibits several limitations that hinder its performance.
in this study, we proposed a framework to estimate the number of clusters for poissonc based on the calculation of an adjusted fom value.
nevertheless, there is a need to further expand clustering-based studies for sage data.
for instance, an important question is how to incorporate poisson-based distance into other clustering methods such as hierarchical clustering and self-organizing maps.
to demonstrate the usefulness of machine learning-based models in predicting functional associations from the sage data, a comprehensive comparative assessment of three supervised classification models was presented.
the results indicated that a relatively simple instance-based model  performed significantly better than relatively more complex algorithms, e.g.
neural networks.
this may be partly explained by the fact that neural network-based prediction models typically require larger amounts of high quality training data.
given the limited amount of sage data available, neural network-based algorithms may not be recommended.
nevertheless, the application of different types of machine learning approaches, including state-of-the-art classifiers such as support vector machine, deserves further investigations.
due to the imbalanced class distribution of the sage data, two re-sampling techniques: random over-sampling and under-sampling methods were studied.
the results indicated that over-sampling strategies may provide more accurate predictions than under-sampling methods.
this result seems to contradict some studies previously published in the literature  <cit> .
however, other studies have suggested that when there is a significant disproportion in the number of samples belonging to each class, random under-sampling methods could actually ignore many potentially relevant data.
investigations of more sophisticated re-sampling techniques  <cit>  will be part of our further research.
we also intend to further address some of the limitations exhibited by such techniques, such as the predictive bias imposed by the incorporation of partially disjoint data sets during cross-validation.
the results suggest that, machine learning approaches such as kstar model may be useful for many purposes.
for example, it can be applied as an inexpensive, user-friendly technique to support functional predictions in the retina before applying more comprehensive validation methods.
it can be used to effectively select candidate genes for studies of retinal development and function.
CONCLUSIONS
the methodology assessed represents an efficient and relatively inexpensive approach for supporting functional predictions.
the techniques discussed in this paper can be in principle regarded as a generic framework, scalable to other types of data and biological functions.
they can support functional predictions prior to the application of more comprehensive, integrative validation methods.
they can be used to effectively select candidate genes for further studies and may also be combined with other large-scale, data-driven methods to facilitate the inference of transcriptional regulatory networks in the developing retina.

 retinal photoreceptor cells, the specialized cells involved in light detection and phototransduction, are essential for mammalian vision. many retinal diseases occur as a result of inherited dysfunction of the rod and cone photoreceptor cells. photoreceptor degeneration, for example, constitutes an important cause of visual impairment affecting all age groups and ethnic backgrounds. development and maintenance of photoreceptor function in the retina requires appropriate regulation of gene expression, especially for genes specifically or highly expressed in photoreceptor cells during retinal development. comprehensive identification of photoreceptor-enriched gene expression patterns may have important implications in neurobiology, leading to a better understanding of molecular mechanisms of retinal development, the improvement of diagnosis of complex retinal diseases, and the identification of potential therapeutic targets. over the last decades, different experimental approaches have been developed to identify retinal disease genes. using microarray data analysis, for example, yoshida et al. revealed that genes, which are differentially expressed in the absence of nrl, are either associated with or are candidates for retinal diseases involving rod or cone photoreceptor dysfunction. katsanis et al. positioned expressed sequence tags likely to be specifically or preferentially expressed in the retina. they also identified positional candidate genes for of uncloned retinopathies. the quality of the results was assessed by reverse transcriptase-polymerase chain reaction. recently, blackshaw et al. presented a comprehensive genomic analysis of mouse retinal development using serial analysis of gene expression, followed by in situ hybridization validation. libraries were obtained from microdissected mouse photoreceptors from the retinal outer nuclear layers, retina from various mouse developmental stages and retina from the paired-homeodomain transcription factor crx knockout mouse and its wild type counterpart at postnatal day and from nih3t mouse fibroblasts. the sage-based expression analysis performed by blackshaw et al. has advantages over other rna analysis methods. this technique uses a unique sequence tag of or more bases isolated from a defined position within each transcript. the basic concept of sage rests on two principles: a short nucleotide sequence tag contains sufficient information to uniquely identify a transcript and concatenated short sequence tags can be cloned to facilitate efficient sequencing analysis. unlike rna blotting and rt-pcr, sage is not limited to examining only a few known genes at a time. unlike dna microarray technology, the sage approach allows the simultaneous analysis of a large number of transcripts without prior, complete knowledge of the sequence of the genes. in the study by blackshaw et al., the candidate photoreceptor-specific genes were selected by comparative analysis between sage libraries on the basis of four chosen criteria. this method, however, has shown relatively low true positive and true negative prediction rates. for example, out of newly-identified photoreceptor-enriched tags, only approximately 20% meet all the four classification criteria and about 44% meet more than three of the four criteria. to further identify true pr-enriched tags, blackshaw et al. experimentally validated the candidate tags through exhaustive ish analysis. in this paper, we explore the feasibility of using computational approaches to support large-scale analysis of photoreceptor gene expression data. the main purpose of this study is to implement several in silico models to detect complex features and relationships encoded in the sage data, which may be used to predict functional associations. by way of illustration, this paper focuses on the classification of two functional classes of genes, which were studied and experimentally validated by blackshaw et al.: pr-enriched and non-pr-enriched genes. the main question was: can machine learning-based classifiers be built to accurately distinguish pr-enriched from non pr-enriched genes solely based on patterns in the sage data? the potential benefits of this approach are two-fold. in situations for which there is insufficient biological knowledge, machine learning-based classifiers could be used to predict functional classes of genes. secondly, if classifiers are demonstrated to effectively predict specific gene types, these empirically derived relationships could be used to derive biological significance. in order to study temporal patterns of photoreceptor gene expression during retinal development, a two-way cluster analysis: clustering of tags and clustering of libraries were performed. while sage libraries were clustered using traditional hierarchical clustering method, sage tags were analysed by a newly developed poisson model-based k-means algorithm specifically designed for sage data. to address one of the limitations of poissonc, a figure of merit -based approach to estimating the number of clusters was proposed. the fom is computed by first removing one experiment, clustering genes based on the remaining data, and then measuring how the left-out library fits the expression patterns obtained from the other libraries. such adaptations represent one of the aspects distinguishing our investigation from traditional clustering-based analyses. to explore the feasibility of machine-learning approaches to predicting functional associations encoded in the sage data generated from different developmental stages in mouse retina, three supervised classification methods were tested to predict the two functional classes investigated by blackshaw et al.: pr-enriched and non-pr-enriched genes. to deal with the problem of functional class imbalance occurring in the dataset, two data re-sampling techniques were studied. to adequately evaluate the performance of the supervised classifiers in imbalanced dataset, three predictive quality indicators that are independent of the class prior probabilities were implemented. to further assess the statistical significance of the computational approaches, a 100-run permutation test was implemented. the reader is referred to the section of methods for a more detailed description of the data sets and techniques studied. the following section summarises relevant results.  clustering of sage libraries clustering of sage tags in general, the temporal expression patterns observed in this sage data reflect previously characterized photoreceptor gene expression patterns in the developing retina. for example, genes associated with tags in clusters and whose expression consistently increases throughout postnatal development and reaches their highest value in the adult retina, are highly likely to be expressed in developing photoreceptors. sixty out of tags previously known to be pr-enriched and out of tags identified to be pr-enriched by blackshaw et al. fell within these two clusters, highlighting that this is an important feature which can be used to identify pr-enriched genes. a closer examination of these two clusters revealed that many tags were mapped to genes with known functions relevant to phototransduction and visual formation. for example, tags associated with rhodopsin gene and tags linked to rod photoreceptor were grouped together in cluster four tags mapped to guanine nucleotide binding proteins, which are involved as modulators or transducers in various transmembrane signalling systems, were all found in cluster apart from those tags directly related to the visual process, some tags associated with genes with other functions such as peripherin- and rod outer segment membrane protein were also found. it has been shown that these genes are involved in maintaining the integrity of photoreceptor outer segment and are therefore critical for rod photoreceptor viability and regulation of disk morphogenesis. interestingly, out of the known non-pr-enriched tags were found in cluster having peak expression values occurring within 3t fibroblast cells. this might confirm that those genes that have higher expression levels in non-retina tissues are unlikely to become pr-enriched genes. cluster exhibits similar expression profiles of genes strongly expressed in adult hypothalamus. although the characteristics of all tags in this cluster were not investigated by blackshaw et al., our analysis indicates that these genes are unlikely to be selectively expressed in photoreceptors based on the observations derived from cluster clusters and whose peak expression values occurred during embryonic development, also offered relevant insights. genes that fell within these two clusters generally have higher expression levels in the mouse retina before p with expression gradually decreasing throughout later postnatal development. examples includes macrophage migration inhibitory factor gene in cluster which plays an important role in t-cell activation and may contribute to regulation of retinal inflammation and its local immunity. its expression starts at e, peaks at e, and significantly decreases in the adult. in clusters and a total of tags were examined by blackshaw et al., ten out of resulted non-pr enriched. the expression profile exhibited by these two clusters may serve as template for the detection of non-pr enriched genes. unlike the other gene clusters, the expression levels of genes within clusters and did not follow a consistent trend throughout retinal development. forty-six tags associated with various pr-enriched genes were found in these four different clusters, reflecting the heterogeneity of photoreceptor gene expression patterns on the basis of the onset and peak time of expression. unlike most of the photoreceptor genes whose expression values are very low at birth and dramatically increase throughout postnatal retinal development, some of the pr-enriched genes exhibit totally different expression patterns. in cluster for example, expression of neurod is high at early embryonic stages, peaking at p and decreases significantly in the adult. this might reflect a dual function of neurod in cell specification and in regulation of rod photoreceptor survival. in cluster mertk, a c-mer proto-oncogene tyrosine kinase, exhibits an embryonic expression profile with peak at e and is selectively expressed in mature photoreceptor cells. previous studies have shown that mutations in mertk are responsible for retinal dystrophy. the diversity of photoreceptor expression profiles may reflect the fact that some of pr-enriched genes are involved in more than one biological process. supervised functional classification of tags the goal was to determine whether a tag represents a pr-enriched or a non-pr-enriched gene given a set of sage libraries associated with each tag. among the tags representing at least % of the total expression in the onl library, tags have been identified as pr-enriched genes, which exhibit diverse and complex expression patterns. this highlights the difficulties in using in silico methods to detect key relationships encoded in the sage data. such complexities are further stressed when the original data are projected on 3-dimensional space using well-known mapping methods, i.e. principal component analysis and sammon's mapping, which clearly indicate that the two classes are not linearly separable from each other. based on a comprehensive analysis of the expression of tags previously known to be pr-enriched, blackshaw et al. introduced four criteria for the selection of candidate pr-enriched genes: tissue specific: the number of tags in either the hypothalamus or 3t libraries is less than developmentally regulated: the sum of tags in the p crx+/+, adult and onl libraries divided by the sum of tags in the e, e, and e libraries is greater than 10-fold. crx dependent: tags are present at a level greater than -fold higher in the p crx+/+ library compared to the library of crx-/- mice. onl enriched: tags are present at an equal or greater number in the onl library compared to the whole adult retina library. we encoded the sage dataset using these criteria to study significant associations between the two functional classes and these criteria. the apriori algorithm was applied to extract association rules from the sage data. this algorithm, which was proposed by agrawal and skrikant, is a well-known association rule learning algorithm. given a dataset the apriori algorithm is able to generate association rules that have support and confidence levels greater than user-specified values. a list of all association rules induced from the tags, together with their support and confidence levels, are given in table the distribution of these tags on the basis of their compliance with the four criteria is given in the additional file. the results obtained showed that a high rate of true positives was observed for genes corresponding to tags meeting more than two of the four criteria. however, when applying these criteria to perform supervised classification, poor prediction results were obtained. for example, only out of pr-enriched tags met all four criteria. from the tags identified by blackshaw et al. as pr-enriched, only about 20% met all four criteria and around 44% met more than three criteria. when applying the four criteria to the data individually, a relatively low rate of true negatives was obtained. for instance, out of non-pr-enriched genes met criterion met criterion met criterion and met criterion therefore, these classification criteria do not represent accurate and robust rules for the classification. tables and show the prediction results from 10-fold cross validation for three supervised classifiers using random over- and under-sampling methods respectively. the section of methods provides a description of these models. for each classifier, the overall classification accuracy, along with the precision, se and sp for each class, were calculated. precision is defined as the proportion of predictions that are correct. the mathematical definitions of these metrics are given in the additional file. the corresponding roc graphs are depicted in figure for each roc graph, the area under roc curve was calculated. the auc has been suggested as a reliable and robust measure for classification performance. a higher value of auc is associated with a classifier that is both effective and robust, i.e. it presents a better average classification performance across different prediction thresholds. tables and indicate that the classifiers built on data derived from over-sampling methods provided better results than those derived from under-sampling methods in terms of ac, se and sp. the advantages of random over-sampling techniques can be further demonstrated by the auc values shown in figure this may be explained by the fact that a random under-sampling strategy may throw away potentially useful data. in addition, we found that the relatively simple kstar algorithm can outperform more complex models such as mlp. to investigate the effect of class distribution on the classifier, we varied the class distribution using the data over-sampling technique. table shows prediction results for the kstar classifier with different class distributions. a 10-fold cross validation procedure was carried out to estimate the true classification error. the corresponding roc curves are given in the additional file. the kstar with balanced class sample distribution achieved the best results. to further assess the statistical significance of our computational approaches and their predictive performance, a 100-run permutation test was implemented. for each permutated dataset, the results were significantly worse than the one generated using the original data in terms of ac, se, and sp, strongly indicating that the relationship between the data and the labels may be reliably learned by the proposed classifiers. for example, when implementing the permutation test for the kstar model on a balanced dataset and 10-fold cross validation, the random classifiers never performed better than the prediction model built. the obtained average values of ac, se and sp were significantly lower than the results shown in table. similar results were obtained when we performed a permutation test for other classifiers on the dataset with different class distributions. the complete results of our permutation test can be found in the additional file. discussion as the most accessible part of the central nervous system and as a highly ordered laminar structure, the retina offers unique opportunities to study both the development and physiology of the cns. this paper described several in silico approaches, including unsupervised and supervised models, to supporting large-scale analysis of photoreceptor gene expression by sage. by clustering sage libraries, a hierarchical tree reflecting the relationship between the libraries was obtained. libraries from adjacent developmental periods were generally grouped together as expected. however, significant discontinuities were identified at the time of birth and between p and p, highlighting important developmental periods. by clustering sage tags, a more comprehensive expression profile for photoreceptor cells was revealed. it confirmed that most of the pr-enriched genes may be successfully clustered. these genes have lower expression levels before birth with expression dramatically increasing throughout postnatal development. nevertheless, a closer examination of the clustering results revealed that photoreceptor expression patterns are highly heterogeneous and not separable by linear methods. the diversity of photoreceptor expression profiles reflects the variability in onset of expression which can occur early in development or when photoreceptors undergo terminal differentiation. with regard to the clustering of sage tags, different algorithms with different distance metrics have been previously proposed. for example, it has been suggested that by modelling sage data with poisson statistics better results can be achieved. however, it is relatively computationally expensive and poisson-based distance has only been assessed as part of the k-means algorithm, which exhibits several limitations that hinder its performance. in this study, we proposed a framework to estimate the number of clusters for poissonc based on the calculation of an adjusted fom value. nevertheless, there is a need to further expand clustering-based studies for sage data. for instance, an important question is how to incorporate poisson-based distance into other clustering methods such as hierarchical clustering and self-organizing maps. to demonstrate the usefulness of machine learning-based models in predicting functional associations from the sage data, a comprehensive comparative assessment of three supervised classification models was presented. the results indicated that a relatively simple instance-based model performed significantly better than relatively more complex algorithms, e.g. neural networks. this may be partly explained by the fact that neural network-based prediction models typically require larger amounts of high quality training data. given the limited amount of sage data available, neural network-based algorithms may not be recommended. nevertheless, the application of different types of machine learning approaches, including state-of-the-art classifiers such as support vector machine, deserves further investigations. due to the imbalanced class distribution of the sage data, two re-sampling techniques: random over-sampling and under-sampling methods were studied. the results indicated that over-sampling strategies may provide more accurate predictions than under-sampling methods. this result seems to contradict some studies previously published in the literature. however, other studies have suggested that when there is a significant disproportion in the number of samples belonging to each class, random under-sampling methods could actually ignore many potentially relevant data. investigations of more sophisticated re-sampling techniques will be part of our further research. we also intend to further address some of the limitations exhibited by such techniques, such as the predictive bias imposed by the incorporation of partially disjoint data sets during cross-validation. the results suggest that, machine learning approaches such as kstar model may be useful for many purposes. for example, it can be applied as an inexpensive, user-friendly technique to support functional predictions in the retina before applying more comprehensive validation methods. it can be used to effectively select candidate genes for studies of retinal development and function.  the methodology assessed represents an efficient and relatively inexpensive approach for supporting functional predictions. the techniques discussed in this paper can be in principle regarded as a generic framework, scalable to other types of data and biological functions. they can support functional predictions prior to the application of more comprehensive, integrative validation methods. they can be used to effectively select candidate genes for further studies and may also be combined with other large-scale, data-driven methods to facilitate the inference of transcriptional regulatory networks in the developing retina. 
based on the analysis of publicly-available gene expression data from the developing mouse retina generated by serial analysis of gene expression, this paper presents a predictive methodology comprising several in silico models for detecting key complex features and relationships encoded in the data, which may be useful to distinguish genes in terms of their functional roles. in order to understand temporal patterns of photoreceptor gene expression during retinal development, a two-way cluster analysis was firstly performed. the results indicated that a relatively simple instance-based model performed significantly better than relatively more complex algorithms, e.g. to demonstrate the usefulness of machine learning-based models in predicting functional associations from the sage data, three supervised classification models were compared. by clustering sage tags, a more comprehensive expression profile for photoreceptor cells was revealed. recent progress in rna analysis technology has generated large amounts of gene expression data relevant to retinal development. the kstar model was also able to achieve higher predictive sensitivities and specificities using random over-sampling techniques. many retinal diseases occur as a result of inherited dysfunction of the rod and cone photoreceptor cells. by clustering sage libraries, a hierarchical tree reflecting relationships between developmental stages was obtained. the approaches assessed in this paper represent an efficient and relatively inexpensive in silico methodology for supporting large-scale analysis of photoreceptor gene expression by sage. over the last decades, different experimental approaches have been developed to identify photoreceptor enriched genes. retinal photoreceptors are highly specialised cells, which detect light and are central to mammalian vision. development and maintenance of photoreceptors requires appropriate regulation of the many genes specifically or highly expressed in these cells. they may also be combined with other large-scale, data-driven methods to facilitate the inference of transcriptional regulatory networks in the developing retina. they may be applied as complementary methodologies to support functional predictions before implementing more comprehensive, experimental prediction and validation methods. to deal with the problem of functional class imbalance occurring in the dataset, two data re-sampling techniques were studied.
 retinal photoreceptors are highly specialised cells, which detect light and are central to mammalian vision. many retinal diseases occur as a result of inherited dysfunction of the rod and cone photoreceptor cells. development and maintenance of photoreceptors requires appropriate regulation of the many genes specifically or highly expressed in these cells. over the last decades, different experimental approaches have been developed to identify photoreceptor enriched genes. recent progress in rna analysis technology has generated large amounts of gene expression data relevant to retinal development. this paper assesses a machine learning methodology for supporting the identification of photoreceptor enriched genes based on expression data.  based on the analysis of publicly-available gene expression data from the developing mouse retina generated by serial analysis of gene expression, this paper presents a predictive methodology comprising several in silico models for detecting key complex features and relationships encoded in the data, which may be useful to distinguish genes in terms of their functional roles. in order to understand temporal patterns of photoreceptor gene expression during retinal development, a two-way cluster analysis was firstly performed. by clustering sage libraries, a hierarchical tree reflecting relationships between developmental stages was obtained. by clustering sage tags, a more comprehensive expression profile for photoreceptor cells was revealed. to demonstrate the usefulness of machine learning-based models in predicting functional associations from the sage data, three supervised classification models were compared. the results indicated that a relatively simple instance-based model performed significantly better than relatively more complex algorithms, e.g. neural networks. to deal with the problem of functional class imbalance occurring in the dataset, two data re-sampling techniques were studied. a random over-sampling method supported the implementation of the most powerful prediction models. the kstar model was also able to achieve higher predictive sensitivities and specificities using random over-sampling techniques.  the approaches assessed in this paper represent an efficient and relatively inexpensive in silico methodology for supporting large-scale analysis of photoreceptor gene expression by sage. they may be applied as complementary methodologies to support functional predictions before implementing more comprehensive, experimental prediction and validation methods. they may also be combined with other large-scale, data-driven methods to facilitate the inference of transcriptional regulatory networks in the developing retina. furthermore, the methodology assessed may be applied to other data domains. 