Title: Diet Networks: Thin Parameters for Fat Genomics
Authors: Adriana Romero
Pierre Luc Carrier
Akram Erraqabi
Tristan Sylvain
Alex Auvolat
Etienne Dejoie
Marc-André Legault
Marie-Pierre Dubé
Julie G. Hussin
Yoshua Bengio
Venue: ICLR 2017
Keywords: Deep learning
Unsupervised Learning
Supervised Learning
Applications

Abstract: Learning tasks such as those involving genomic data often poses a serious challenge: the number of input features can be orders of magnitude larger than the number of training examples, making it difficult to avoid overfitting, even when using the known regularization techniques. We focus here on tasks in which the input is a description of the genetic variation specific to a patient, the single nucleotide polymorphisms (SNPs), yielding millions of ternary inputs. Improving the ability of deep learning to handle such datasets could have an important impact in medical research, more specifically in precision medicine, where high-dimensional data regarding a particular patient is used to make predictions of interest. Even though the amount of data for such tasks is increasing, this mismatch between the number of examples and the number of inputs remains a concern. Naive implementations of classifier neural networks involve a huge number of free parameters in their first layer (number of input features times number of hidden units): each input feature is associated with as many parameters as there are hidden units. We propose a novel neural network parametrization which considerably reduces the number of free parameters. It is based on the idea that we can first learn or provide a distributed representation for each input feature (e.g. for each position in the genome where variations are observed in data), and then learn (with another neural network called the parameter prediction network) how to map a feature's distributed representation (based on the feature's identity not its value) to the vector of parameters specific to that feature in the classifier neural network (the weights which link the value of the feature to each of the hidden units). This approach views the problem of producing the parameters associated with each feature as a multi-task learning problem. We show experimentally on a population stratification task of interest to medical studies that the proposed approach can significantly reduce both the number of parameters and the error rate of the classifier.

TL;DR: Drastically reducing the number of parameters, when the number of input features is orders of magnitude larger than the number of training examples, such as in genomics.

Review discussion:
Thanks for the very interesting paper!
I would like to ask for a clarification of the experimental procedure described in Sec. 4.2. You mention splitting the data to 5 folds: 3 for training, 1 for testing and 1 for validation. Can you please explain how these are used, especially what is the difference between testing and validation folds?
Thanks for your question.
The training folds are used to learn the parameters of the model. The validation fold is used for hyperparameter tuning (setting which learning rate, weights decay, gamma, architecture work best) and for early stopping. During training, accuracy on the validation fold is monitored and after 100 epochs of no improvement, the training is stopped. The test fold is used to report results (when the final model is fully trained and we want to assess the performance of the trained model). Note that the test fold contains new fresh data that the model has never seen before.
It would be informative if the authors could include the performance of the model consisting of PCA preprocessor + MLP classifier in the comparison, especially as PCA is closely related to auto-encoding / reconstruction.
As mentionned in the paper, "adding capacity didn't improve the performance".
To be more specific, we give here some extra results for the PCA + MLP classifier. We report these results for PCA100 that had the best performance among the PCA-based experiments.
PCA100 + MLP(50)
mean : 12.67
std : 0.67
PCA100+ MLP(100) :
mean : 12.18
std : 1.75
PCA100 + MLP(100,100)
mean : 11.95
std : 2.29
The mean and the standard deviation are computed over the same folds as the PCA-only experiments.
We thank you for the suggestion and will add as suggested these results in the comparison.
The problem addressed here is practically important (supervised learning with n<<d), and as far as I know, the approach is novel. I thought their proposed solution was innovative, and I enjoyed the paper. The presentation is clear and it has nice experiments.
Comments/questions:
1) What is the dimensionality of the feature embeddings?
2) SNPtoVec still requires training a very fat autoencoder network on X --- I suppose this doesn't contribute to the size of the final run-time model and overfitting is avoided because the parameters are fit in an unsupervised manner?
Thank you for your review.
1) The 'Per class histograms' model, as briefly described in section 2.2, uses feature embeddings of size 78 because every feature has 3 possible values and the data contains 26 classes (26 * 3 = 78). For the other methods, the results reported in table 1 use feature embeddings of size 100.
2) Yes, exactly.
If "raw end2end" model also has embedding of size 100, then shouldn't the number of its free parameters be at least 3450x100=345K? Why is it 290K in Table1? Because of 5-fold CV?
Yes, it is due to the 5-fold cross-validation.
The reason the model has 290K parameters and not at least 345K is that we do not use every subject in the dataset to learn the feature embeddings. We only use the training examples. As detailed at the beginning of section 4.2, we perform our experiments with 5-fold cross-validation. Since, at any given time, one fold is used for testing, one fold is used for validation and three folds are used for training, this means we have (3/5)*3450 = 2070 training examples which are used to learn the embeddings.
Also, we have updated Table 1 to, hopefully, make it clearer. It now shows only the number of free parameters needed for the fat layers of the models instead of the total number of parameters in the whole models (since these other parameters are present in all considered architectures). We hope this makes it easier to understand how many free parameters each approach contributes to the model.
We thank the reviewer for the nice comments and hope we have properly addressed their questions. We added a comment regarding the embedding dimensionality in Table 1's caption.
The paper addresses the important problem (d>>n) in deep learning.
The proposed approach, based on lower-dimensional feature embeddings, is reasonable and makes applying deep learning methods to data with large d possible.
The paper is well written and the results show improvements over reasonable baselines.
We thank the reviewer for the nice comments/suggestions that are helping improve the paper and hope we have properly addressed their previous questions. As suggested, we have updated Table 1 with PCA performance for different numbers of PCs and different MLP classifier architectures. We have also added an appendix section to analyze the influence of using different numbers of pcs.
The paper presents an application of deep learning to genomic SNP data
with a comparison of possible approaches for dealing with the very
high data dimensionality. The approach looks very interesting but the
experiments are too limited to draw firm conclusions about the
strengths of different approaches. The presentation would benefit from
more precise math.
Quality:
The basic idea of the paper is interesting and the applied deep
learning methodology appears reasonable. The experimental evaluation
is rather weak as it only covers a single data set and a very limited
number of cross validation folds. Given the significant variation in
the performances of all the methods, it seems the differences between
the better-performing methods are probably not statistically
significant. More comprehensive empirical validation could clearly
strengthen the paper.
Clarity:
The writing is generally good both in terms of the biology and ML, but
more mathematical rigour would make it easier to understand precisely
what was done. The different architectures are explained on an
intuitive level and might benefit from a clear mathematical
definition. I was ultimately left unsure of what the "raw end2end"
model is - given so few parameters it cannot work on raw 300k
dimensional input but I could not figure out what kind of embedding
was used.
The results in Fig. 3 might be clearer if scaled so that maximum for
each class is 1 to avoid confounding from different numbers of
subjects in different classes. In the text, please use the standard
italics math font for all symbols such as N, N_d, ...
Originality:
The application and the approach appear quite novel.
Significance:
There is clearly strong interest for deep learning in the genomics
area and the paper seeks to address some of the major bottlenecks
here. It is too early to tell whether the specific techniques proposed
in the paper will be the ultimate solution, but at the very least the
paper provides interesting new ideas for others to work on.
Other comments:
I think releasing the code as promised would be a must.
We thank the reviewer for the nice comments and the useful observations that are helping improve the paper.
1) The basic idea of the paper is interesting and the applied deep learning methodology appears reasonable.  The experimental evaluation is rather
weak as it only covers a single data set and a very limited number of cross validation folds. Given the significant variation in the performances of all the methods, it seems the differences between the better-performing methods are probably not statistically significant. More comprehensive empirical validation could clearly strengthen the paper.
A1: Among the better performing architectures, the goal was to reduce the number of parameters of our model.
Hence, we show that we are able to drastically reduce the number of free parameters in the model without hurting the performance.
Moreover, to strengthen the argument on superiority of deep learning methods over classical approaches used in the genomics domain (e.g. PCA-based), we performed a binomial test to compare the results achieved by classical approaches to the Diet Networks ones.
When comparing the best results achieved by PCA (with 200 pcs) to the best Diet Networks model, the results are statistically significant (for alpha=0.05). We obtain a p-value of 2.4e-6, highlighting the potential of deep learning approaches to handle genomic data.
2) The writing is generally good both in terms of the biology and ML, but more mathematical rigour would make it easier to understand precisely
what was done. The different architectures are explained on an intuitive level and might benefit from a clear mathematical definition. I was ultimately left unsure of what the "raw end2end" model is - given so few parameters it cannot work on raw 300k dimensional input but I could not figure out what kind of embedding was used.
A2: In raw end2end, the auxiliary network takes as input one SNP (2070 features, corresponding to training samples) and learns a feature embedding of dimensionality 100 (Fig. 1 (b) Emb.), followed by a second feature representation of dimensionality 100 (Fig. 1 (b) MLP, in this case one single hidden layer). The second feature representation is used as We.
So, the auxiliary network has 2070*100 + 100 + 100*100+100 free parameters instead of the 315345*100 free parameters that We in Fig. 1 (a) would have.
We thank the reviewer for pointing out this issue.  We have included the following paragraph in the Feature Embedding section with the goal of improving the clarity of the presentation w.r.t. the raw end2end embedding:
“Embedding learnt end-to-end from raw data: In this case, we consider the feature embedding to be another MLP, whose input corresponds to the values that a SNP takes for each of the training samples and, whose parameters are learnt jointly with the rest of the network. Note that the layer(s) corresponding to the feature embedding are shared among auxiliary networks. For experiments reported in Section 4, we used a single hidden layer as embedding.”
Moreover, we updated Table 1 such that it only includes the number of free parameters in the fat layers of the model, since the other free parameters are present in all architectures. We hope this will simplify the computation of the number of free parameters.
3) The results in Fig. 3 might be clearer if scaled so that maximum for each class is 1 to avoid confounding from different numbers of subjects in different classes. In the text, please use the standard italics math font for all symbols such as N, N_d, ...
A3: As suggested, we have updated both Fig. 3 and the notation in the most recent version of the paper.
4) I think releasing the code as promised would be a must.
A4: The code is now publicly available and the link to the repository has been included in the paper (see footnote on page 6).
We hope we properly addressed the reviewer's concerns and improved our paper with their comments.
We hope the revisions made to the paper properly address the concerns raised by the reviewers and other researchers, and improve the quality of the manuscript.
Summary of revisions:
* Embedding learnt end-to-end from raw data: Added paragraph (Section 2.2, page 5) to explain what the embedding learnt end-to-end is and how it is used within the Diet Networks framework.
* Figure 3:  Scaled both confusion matrices so that maximum for each class is 1.
* Table 1:  Added extended PCA results.
* Table 1: Updated number of free parameters column to only account for the number of free parameters in the fat layers of the model, with the aim to make the presentation clearer.
* Made code publicly available and added footnote with the corresponding url (page 6).
* Section 4.2. (Results), paragraph 6: introduced extended PCA results (more PCs and MLP + classifier).
* Appendix B: Added the instructions to download the dataset.
* Appendix C: Added an analysis on PCA (number of components used) and performance.
* Fixed typos.
* Table 1: Added comment about feature embeddings dimensionality.

Meta-review:
role: PC
data: comment: The reviewers agreed that the paper proposed a solid and original contribution that was evaluated well. The authors would be encouraged to improve the presentation and provide a more precise mathematical presentation, as discussed by reviewer 1.

Decision: Accept (Poster)