Title: NEUROGENESIS-INSPIRED DICTIONARY LEARNING: ONLINE MODEL ADAPTION IN A CHANGING WORLD
Authors: Sahil Garg
Irina Rish
Guillermo Cecchi
Aurelie Lozano
Venue: ICLR 2017
Keywords: Unsupervised Learning
Computer vision
Transfer Learning
Optimization
Applications

Abstract: In this paper, we focus on online representation learning in non-stationary environments which may require continuous adaptation of model’s architecture. We propose a novel online dictionary-learning (sparse-coding) framework which incorporates the addition and deletion of hidden units (dictionary elements), and is inspired by the adult neurogenesis phenomenon in the dentate gyrus of the hippocampus, known to be associated with improved cognitive function and adaptation to new environments. In the online learning setting, where new input instances arrive sequentially in batches, the “neuronal birth” is implemented by adding new units with random initial weights (random dictionary elements); the number of new units is determined by the current performance (representation error) of the dictionary, higher error causing an increase in the birth rate. “Neuronal death” is implemented by imposing l1/l2-regularization (group sparsity) on the dictionary within the block-coordinate descent optimization at each iteration of our online alternating minimization scheme, which iterates between the code and dictionary updates. Finally, hidden unit connectivity adaptation is facilitated by introducing sparsity in dictionary elements. Our empirical evaluation on several real-life datasets (images and language) as well as on synthetic data demonstrates that the proposed approach can considerably outperform the state-of-art fixed-size (nonadaptive) online sparse coding of Mairal et al. (2009) in the presence of nonstationary data. Moreover, we identify certain properties of the data (e.g., sparse inputs with nearly non-overlapping supports) and of the model (e.g., dictionary sparsity) associated with such improvements.

TL;DR: An online dictionary learning incorporates dynamic model adaptation, adding/deleting its elements in response to nonstationary data.

Review discussion:
In Mairal et al (2009), the authors discuss a way of reinitialising 'dead' atoms when they present little activity. They use samples from the dataset (instead of noise). Are you using this feature? If not, it seems like a reasonable baseline to include.
In the case of sparse dictionary elements, is the online DL baseline include such constraint?
> In Mairal et al (2009), the authors discuss a way of reinitialising  'dead' atoms when they present little activity. They use samples
> from the dataset (instead of noise). Are you using this feature? If  not, it seems like a reasonable baseline to include.
Thank you for pointing this out. Yes, we are aware of this feature mentioned in the Mairal's original paper; note, however, that Mairal's paper only mentioned this option, but did not provide any empirical evaluation/comparison of its benefits with the fixed-size version, while in our paper, one of our main goals is to evaluate fixed-size vs non-fixed-size dictionary learning approaches.
So, the reinitialization version is perhaps less appropriate as a baseline.
Furthermore, in our baseline (fixed-size Mairal) experiments, we hardly get 'dead' atoms except  for  the case of dense dictionary elements with highly sparse codes. In such settings, there can be a very small number of dead atoms in the dictionary (like less than 1%).
Also, since the Pearson correlation  and the mean square error curves are plotted w.r.t. the final  dictionary size, i.e. accounting for only the non-dead atoms in a  dictionary (the ones with non-zero l2 norm), we are effectively  discounting for the dead atoms in our evaluation to give the  baseline an advantage.
>In the case of sparse dictionary elements, is the online DL baseline include such constraint?
Yes, when the dictionary elements are sparse, the constraint is  imposed for the online DL baseline also.
I'd like to thank the authors for their detailed response and clarifications.
This work proposes new training scheme for online sparse dictionary learning. The model assumes a non-stationary flow of the incoming data. The goal (and the challenge) is to learn a model in an online manner in a way that is capable of  adjusting to the new incoming data without forgetting how to represent previously seen data. The proposed approach deals with this problem by incorporating a mechanism for adding or deleting atoms in the dictionary. This procedure is inspired by the adult neurogenesis phenomenon in the dentate gyrus of the hippocampus.
The paper has two main innovations over the baseline approach (Mairal et al): (i) “neuronal birth” which represents an adaptive way of increasing the number of atoms in the dictionary (ii) "neuronal death", which corresponds to removing “useless” dictionary atoms.
Neural death is implemented by including an group-sparsity regularization to the dictionary atoms themselves (the group corresponds to a column of the dictionary). This promotes to shrink to zero atoms that are not very useful, keeping controlled the increase of the dictionary size.
I believe that the strong side of the paper is its connections with the adult neurogenesis phenomenon, which is, in my opinion a very nice feature.
The paper is very well written and easy to follow.
On the other hand, the overall technique is not very novel.
Although not exactly equivalent, similar ideas have been explored.
While the neural death is implemente elegantly with a sparsity-promoting regularization term, the neural birth is performed by relying on heuristics that measure how well the dictionary can represent new incoming data. Which depending on the "level" of non-stationarity in the incoming data (or presence of outliers) could be difficult to set.
Still, having adaptive dictionary size is very interesting.
The authors could also cite some references in model selection literature.
In particular, some ideas such as MDL have been used for automatically selecting the dictionary size (I believe this work does not address the online setting, but still its a relevant reference to have).
For instance,
Ramirez, Ignacio, and Guillermo Sapiro. "An MDL framework for sparse coding and dictionary learning." IEEE Transactions on Signal Processing 60.6 (2012): 2913-2927.
We thank the reviewer for the insightful comments. Please consider our response to the review as below.
1. "On the other hand, the overall technique is not very novel.
Although not exactly equivalent, similar ideas have been explored"
Please see the discussion above in response to the Reviewer1, item 4 in the summary above, as well as a paragraph at the end of the introduction which hopefully emphasizes more clearly the  novelty of the proposed approach.  As discussed above, at some level of generality, the ideas of addition or deletion of variables have been explored, but never in online dictionary learning context.
We thank the reviewer for the reference on MDL approach to model selection in sparse coding, in offline setting; we cited and discussed this reference in the introduction, however, our approach is quite different as it involves an online setting and assumes nonstationary inputs.
If our arguments concerning novelty of our approach are still not sufficiently convincing, we would really appreciate any specific references on prior art proposing an online adaptive model-selection within any autoencoder setting (sparse dictionary or beyond), based on hidden units birth/death dynamics, with a thorough evaluation, empirical and theoretical, of situations where such adaptive approach is beneficial (or not) as compared vs non-adaptive baseline. We were unable to find any such prior work.
2. " the neural birth is performed by relying on heuristics that measure how well the dictionary can represent new incoming data. Which depending on the "level" of non-stationarity in the incoming data (or presence of outliers) could be difficult to set."
This is a good point; as usual, we end up having a tunable parameter controlling the birth rate (here, a threshold on the representation error measured as 1 - correlation between actual and reconstructed input), and must resort to cross-validation to set it up, or potentially provide another layer of automation in order to tune this parameter online (e.g., an RL approach to parameter tuning in our method).
However, we view this as a direction for future work, since the automated parameter tuning is a sufficiently involved procedure on its own.
Regarding the earlier question about reinitialization of dead elements in the Mairal's original algorithm: we added new experiments in the Appendix, section B.7, Figure 19.
First of all, we noticed that the fraction of “dead” elements was typically very small in our experiments when using the original ODL method of Mairal et al. (2009). We next followed the idea of reinitialization of dead elements in ODL, referring to this modification as “ODL*” in Figure 19.
Specifically, we reinitialize the “dead” elements with random initializations, and then relearn the reinitialized elements, along with other elements, on the current batch of data. Fig. 19 extends Fig. 2 with additional plots for the baseline extension “ODL*” of ODL; keeping the experiment settings same. We see that there is negligible difference in the performance of ODL and its extension “ODL*”, and our method NODL maintains its superior performance w.r.t. ODL as well as “ODL*”.
The paper is interesting, it relates findings from neurscience and biology to a method for sparse coding that is adaptive and able to automatically generate (or even delete) codes as new data is coming, from a nonstationary distribution.
I have a few points to make:
1. the algorithm could be discussed more, to give a more solid view of the contribution. The technique is not novel in spirit. Codes are added when they are needed, and removed when they dont do much.
2. Is there a way to relate the organization of the data to the behavior of this method? In this paper, buildings are shown first, and natural images (which are less structured, more difficult) later. Is this just a way to perform curriculum learning? What happens when data simply changes in structure, with no apparent movement from simple to more complex (e.g. from flowers, to birds, to fish, to leaves, to trees etc)
In a way, it makes sense to see an improvement when the training data has such a structure, by going from something artificial and simpler to a more complex, less structured domain.
The paper is interesting, the idea useful with some interesting insights. I am not sure it is ready for publication yet.
We thank the reviewer for the insightful comments. Please consider our response to the review as below.
1. "the algorithm could be discussed more, to give a more solid view of the contribution"
A new section (3.1) was added to provide a detailed discussion.
2. "The technique is not novel in spirit"
Please see the item 4 in the summary above. We are convinced that our approach is actually quite novel, in the context of online dictionary learning or more general autoencoders. Clearly, at a very general level, model selection is not a novel idea, and at the same level of generality, the ideas of adding or deleting variables during model selection may have appeared earlier in very different contexts (e. g. , in learning Bayesian network structure, or in LASSO solved by active set methods such as LARS, just to name a few). We are not claiming the novelty at such extremely general level, but the dynamic addition/deletion ONLINE model selection in dictionary learning setting is a completely novel approach.
As discussed in item 4 above, not only is our approach quite different from the corresponding related work, our key contribution is actually empirical and theoretical evaluation of situations/conditions which make such  online model selection beneficial for dictionary learning – the topic never explored in the sparse coding literature, to the best of our knowledge. We also added a bullet-point list of our contributions at the end of the introduction paragraph in the paper.
If the novelty of the proposed approach still does not seem convincing, we would really appreciate specific references to prior work which (1) implements birth/death of hidden units in an autoencoder model, in an online setting, outperforming state-of-art in nonstationary settings and (2) provides both empirical and theoretical evaluation of conditions under which adaptive approach is beneficial vs non-adaptive.
3. "Is there a way to relate the organization of the data to the behavior of this method? In this paper, buildings are shown first, and natural images (which are less structured, more difficult) later. Is this just a way to perform curriculum learning? What happens when data simply changes in structure, with no apparent movement from simple to more complex (e.g. from flowers, to birds, to fish, to leaves, to trees etc.) In a way, it makes sense to see an improvement when the training data has such a structure, by going from something artificial and simpler to a more complex, less structured domain."
The objective here was to explore ANY nonstationary conditions, not necessarily the  "curriculum learning" setting which indeed would imply going from simpler to more complex concepts.  We are not convinced that the natural images are necessarily "more difficult" that buildings (e.g., a complicated city scene with multiple buildings can be seen, vice versa, more 'complex' than a flower, as in some of those images).
In fact, we performed additional experiments, added to the Appendix, section B.8, Figure 20, where we tried all possible permutations of the input datasets (buildings, flowers, animals). The results were very similar to the ones presented in the main paper when we flipped the order of input data (e.g., with flowers followed by building, or flowers followed by animals, our methods showed similar magnitude of improvements over the baseline method as it did for buildings followed by flowers or by animals).
Overall, our general observation is that there is no processing order on datasets where baseline (ODL) could perform better than the proposed approach. We see a significant advantage of NODL over ODL if we have Oxford or Flowers data sets as the first domain data.
However, this advantage is smaller when using the Animals dataset first, followed by flowers or buildings, so, empirically, it turns out that perhaps animal images were more 'complex', and then flowers and buildings looked 'easier' to learn afterwards, but this is just a speculation at this point.
In the future, it would be interesting to investigate scenarios where the complexity of the input would be measured precisely and controlled for, e.g., using the compressed size of an image as a proxy for its complexity, or some similar 'exact' measure.
4. "The paper is interesting, the idea useful with some interesting insights. I am not sure it is ready for publication yet. "
Can you please make any specific suggestions regarding the steps that would make the paper ready for publication? Thank you!
The authors propose a simple modification of online dictionary learning: inspired by neurogenesis, they propose to add steps of atom addition, or atom deletion, in order to extent the online dictionary learning algorithm algorithm of Mairal et al. Such extensions helps to adapt the dictionary to changing properties of the data.
The online adaptation is very interesting, even if it is quite simple. The overall algorithm is quite reasonable, but not always described in sufficient details: for example, the thresholds or conditions for neuronal birth or death are not supported by a strong analysis, even if the resulting algorithm seems to perform well on quite extensive experiments.
The overall idea is nevertheless interesting (even if not completely new), and the paper generally well written and pretty easy to follow. The analysis is however quite minimal: it could have been interesting to study the evolving properties of the dictionary, to analyse its accuracy for following the changes in the data, etc.
Still: this is a nice work!
We would like to thank the reviewer for useful comments and constructive suggestions.
In response to the comment on insufficient details of the algorithm, we added a new section 3.1 fully devoted to discussing such details.
However, as suggested by the reviewer, a further analysis of the algorithm, involving the evolving properties of the dictionary and performance changes due to changes in the data, remain the directions for future work.
We would like to thank all reviewers for their insightful comments that helped us to improve the paper; to address those comments, we modified the paper as follows:
1.	Added a subsection 3.1 in section 3, describing various details of the proposed algorithm, as suggested by Reviewer1.
2.	Added a section (Appendix, B9, Figures 21-26) demonstrating stability of our results (our method consistently outperforming the non-adaptive dictionary learning)  while varying all of the algorithm’s parameters, one at a time, over a wide range of values.
3.	Added a section (Appendix, B7, Fig. 19) with new empirical results to address the concern of Reviewer3 regarding comparison with a version of the baseline method which involved re-initialization of ``dead’’ elements; we did not observe any significance difference in the performance of such augmented method versus the original baseline (the number of dead elements appearing in the baseline method was quite small);
on the contrary, our method employing group-sparsity was removing larger number of ``weak’’ (low l2-norm) elements and yielded better performance due to such explicit regularization.
4.	Added a section (Appendix, B8, Fig. 20) with new results evaluating our method under different ordering of the input data, to address the question of the Reviewer1 regarding the potential sensitivity of our results to the ordering of the training domains  (e.g., from easier to harder). No significant changes  were observed due to permutation of the input datasets (our method was still outperforming the baseline), i.e. buildings images were not necessarily ‘simpler’ than natural ones.
5.	A summary of our contributions is added at the end of the intro section. Our main point is that the contribution of this paper is truly novel and nontrivial.
Indeed,
a.	We are the first to propose an online model-selection approach to dictionary learning (DL) based on dynamic addition and deletion of the elements (hidden units), which leads to significant performance improvements over the state-of-art online DL, especially in non-stationary settings. None of the reviewers have provided a reference that would contradict the above statement.
b.
While some prior work (e.g., Bengio et al, NIPS 2009) involved deletion of dictionary elements, no addition was involved in that approach and, more importantly, their approach only concerned an offline dictionary learning, not online.
Similarly, approaches to neural nets involving cascade correlations and other methods adding hidden units are lacking regularization (deletion) that we achieve via group sparsity constraint.
Finally, Reviewer3 mentions prior work on off-line MDL-based model selection in dictionary learning, but the major difference is, again, that our setting is online learning, while their approach is off-line.
c.
Furthermore, none of the prior approaches explored the interplay between addition and deletion, neither theoretically nor empirically, while we performed such evaluation and identified regimes when our online model-selection would outperform the non-adaptive baseline   (Section 5).
Thus, to the best of our knowledge, our proposed method is completely novel, and provides a considerable improvement over the state-of-art, especially when the input is nonstationary.
Our detailed replies to each review are put as separate comments below.
The authors did a good work with the revisions, that helped yet strengthen the paper. The quality and completeness of the work, along with the interesting and partially new ideas it develops, makes it a good and 'refreshing' paper.

Meta-review:
role: PC
data: comment: This paper combines simple heuristics to adapt the size of a dictionary during learning. The heuristics are intuitive: augmenting the dictionary size when correlation between reconstruction and inputs falls below a certain pre-determined threshold, reducing the dictionary size by adding a group-sparsity L1,2 penalty on the overall dictionary (the L1,2 penalty for pruning models had appeared elsewhere before, as the authors acknowledge).
 The topic of online adjustment of model complexity is an important one, and work like this is an interesting and welcome direction. The simplicity of the heuristics presented here would be appealing if there was more empirical support to demonstrate the usefulness of the proposed adaptation. However, the empirical validation falls short here:
 - the claim that this work offers more than existing off-line model adaptation methods because of its online setting is the crux of the value of the paper, but the experimental validation does not offer much in terms of how to adjust hyperparameters to a truly nonstationary setting. Here, the hyperparameters are set rather arbitrarily, and the experimental setting is a single switch between two datasets (where off-line methods would do well), so this obscures the challenges that would be presented by a less artificial level of non-stationarity
 - an extensive set of experiments is presented, but all these experiments are confined in a strange experimental setting that is divorced from any practical metrics: the "state-of-the-art" method of reference is Mairal et al 2009 which was focused on speeding learning of dictionaries. Here, the metrics offered are correlation / reconstruction error, and classification, over full images instead of patches as is usually done for large images, without justification. This unnecessarily puts the work in a vacuum in terms of evaluation and comparison to existing art. In fact, the reconstructions in the appendix Fig. 17 and 18 look pretty bad, and the classification performance 1) does not demonstrate superiority of the method over standard dictionary learning (both lines are within the error bars and virtually indistinguishable if using more than 60 elements, which is not much), 2) performance overall is pretty bad for a 2-class discrimination task, again because of the unusual setting of using full images.
 In summary, this paper could be a very nice paper if the ideas were validated in a way that shows true usefulness and true robustness to the challenges of realistic nonstationarity, but this is unfortunately not the case here.
 Ps: two recent papers that could be worth mentioning in terms of model adaptation are diversity networks for pruning neurons (Diversity Networks, Mariet and Sra ICLR 2016, https://arxiv.org/abs/1511.05077), and augmenting neural networks with extra parameters while retaining previous learning (Progressive Neural Networks, Rusu et al 2016, https://arxiv.org/pdf/1606.04671v3.pdf).

Decision: Reject