Venue: ICLR 2017 conference

Reviews:

final_score: rating: 7: Good paper, accept
review_content: The authors propose a simple modification of online dictionary learning: inspired by neurogenesis, they propose to add steps of atom addition, or atom deletion, in order to extent the online dictionary learning algorithm algorithm of Mairal et al. Such extensions helps to adapt the dictionary to changing properties of the data. 

The online adaptation is very interesting, even if it is quite simple. The overall algorithm is quite reasonable, but not always described in sufficient details: for example, the thresholds or conditions for neuronal birth or death are not supported by a strong analysis, even if the resulting algorithm seems to perform well on quite extensive experiments.

The overall idea is nevertheless interesting (even if not completely new), and the paper generally well written and pretty easy to follow. The analysis is however quite minimal: it could have been interesting to study the evolving properties of the dictionary, to analyse its accuracy for following the changes in the data, etc. 

Still: this is a nice work!   
review_title: Simple interesting modified online dictionary learning

final_score: rating: 5: Marginally below acceptance threshold
review_content: The paper is interesting, it relates findings from neurscience and biology to a method for sparse coding that is adaptive and able to automatically generate (or even delete) codes as new data is coming, from a nonstationary distribution.

I have a few points to make:

1. the algorithm could be discussed more, to give a more solid view of the contribution. The technique is not novel in spirit. Codes are added when they are needed, and removed when they dont do much. 

2. Is there a way to relate the organization of the data to the behavior of this method? In this paper, buildings are shown first, and natural images (which are less structured, more difficult) later. Is this just a way to perform curriculum learning? What happens when data simply changes in structure, with no apparent movement from simple to more complex (e.g. from flowers, to birds, to fish, to leaves, to trees etc)

In a way, it makes sense to see an improvement when the training data has such a structure, by going from something artificial and simpler to a more complex, less structured domain.

The paper is interesting, the idea useful with some interesting insights. I am not sure it is ready for publication yet.

review_title: Interesting idea related to biology, good experimental validation, but more work is probably needed

final_score: rating: 5: Marginally below acceptance threshold
review_content: 
I'd like to thank the authors for their detailed response and clarifications.

This work proposes new training scheme for online sparse dictionary learning. The model assumes a non-stationary flow of the incoming data. The goal (and the challenge) is to learn a model in an online manner in a way that is capable of  adjusting to the new incoming data without forgetting how to represent previously seen data. The proposed approach deals with this problem by incorporating a mechanism for adding or deleting atoms in the dictionary. This procedure is inspired by the adult neurogenesis phenomenon in the dentate gyrus of the hippocampus. 

The paper has two main innovations over the baseline approach (Mairal et al): (i) “neuronal birth” which represents an adaptive way of increasing the number of atoms in the dictionary (ii) "neuronal death", which corresponds to removing “useless” dictionary atoms.

Neural death is implemented by including an group-sparsity regularization to the dictionary atoms themselves (the group corresponds to a column of the dictionary). This promotes to shrink to zero atoms that are not very useful, keeping controlled the increase of the dictionary size.

I believe that the strong side of the paper is its connections with the adult neurogenesis phenomenon, which is, in my opinion a very nice feature.
The paper is very well written and easy to follow.

On the other hand, the overall technique is not very novel. Although not exactly equivalent, similar ideas have been explored. While the neural death is implemente elegantly with a sparsity-promoting regularization term, the neural birth is performed by relying on heuristics that measure how well the dictionary can represent new incoming data. Which depending on the "level" of non-stationarity in the incoming data (or presence of outliers) could be difficult to set. Still, having adaptive dictionary size is very interesting.

The authors could also cite some references in model selection literature. In particular, some ideas such as MDL have been used for automatically selecting the dictionary size (I believe this work does not address the online setting, but still its a relevant reference to have). For instance,

Ramirez, Ignacio, and Guillermo Sapiro. "An MDL framework for sparse coding and dictionary learning." IEEE Transactions on Signal Processing 60.6 (2012): 2913-2927.

review_title: review

final_score: rating: null
review_content: 
In Mairal et al (2009), the authors discuss a way of reinitialising 'dead' atoms when they present little activity. They use samples from the dataset (instead of noise). Are you using this feature? If not, it seems like a reasonable baseline to include.

In the case of sparse dictionary elements, is the online DL baseline include such constraint? 

review_title: Question regarding baseline

Meta-review: This paper combines simple heuristics to adapt the size of a dictionary during learning. The heuristics are intuitive: augmenting the dictionary size when correlation between reconstruction and inputs falls below a certain pre-determined threshold, reducing the dictionary size by adding a group-sparsity L1,2 penalty on the overall dictionary (the L1,2 penalty for pruning models had appeared elsewhere before, as the authors acknowledge).
 The topic of online adjustment of model complexity is an important one, and work like this is an interesting and welcome direction. The simplicity of the heuristics presented here would be appealing if there was more empirical support to demonstrate the usefulness of the proposed adaptation. However, the empirical validation falls short here:
 - the claim that this work offers more than existing off-line model adaptation methods because of its online setting is the crux of the value of the paper, but the experimental validation does not offer much in terms of how to adjust hyperparameters to a truly nonstationary setting. Here, the hyperparameters are set rather arbitrarily, and the experimental setting is a single switch between two datasets (where off-line methods would do well), so this obscures the challenges that would be presented by a less artificial level of non-stationarity
 - an extensive set of experiments is presented, but all these experiments are confined in a strange experimental setting that is divorced from any practical metrics: the "state-of-the-art" method of reference is Mairal et al 2009 which was focused on speeding learning of dictionaries. Here, the metrics offered are correlation / reconstruction error, and classification, over full images instead of patches as is usually done for large images, without justification. This unnecessarily puts the work in a vacuum in terms of evaluation and comparison to existing art. In fact, the reconstructions in the appendix Fig. 17 and 18 look pretty bad, and the classification performance 1) does not demonstrate superiority of the method over standard dictionary learning (both lines are within the error bars and virtually indistinguishable if using more than 60 elements, which is not much), 2) performance overall is pretty bad for a 2-class discrimination task, again because of the unusual setting of using full images.
 In summary, this paper could be a very nice paper if the ideas were validated in a way that shows true usefulness and true robustness to the challenges of realistic nonstationarity, but this is unfortunately not the case here.
 Ps: two recent papers that could be worth mentioning in terms of model adaptation are diversity networks for pruning neurons (Diversity Networks, Mariet and Sra ICLR 2016, https://arxiv.org/abs/1511.05077), and augmenting neural networks with extra parameters while retaining previous learning (Progressive Neural Networks, Rusu et al 2016, https://arxiv.org/pdf/1606.04671v3.pdf).
Decision: Reject