Venue: ICLR 2018 conference

Reviews:

final_score: rating: 8: Top 50% of accepted papers, clear accept
review_content: In this paper, the authors propose a method (Dynamically Expandable Network) that addresses issues of training efficiency, how to dynamically grow the network, and how to prevent catastrophic forgetting.

The paper is well written with a clear problem statement and description of the method for preventing each of the described issues. Interesting points include the use of an L1 regularization term to enforce sparsity in the weights, as well as the method for identifying which neurons have “drifted” too far and should be split. The use of timestamps is a clever addition as well.

One question would be how sparse training is done, and how this saves computation, especially with the breadth-first search described on page 5. A critique would be that the base networks (a two layer FF net and LeNet) are not very compelling.

Experiments indicate that the method works well, with a clear improvement over progressive networks. Thus, though there isn’t one particular facet of the paper that leaps out, overall the method and results seem solid and worthy of publication.
review_title: Method for lifelong learning with neural networks

final_score: rating: 7: Good paper, accept
review_content: The paper was clearly written and pleasant to read. I liked the use of sparsity- and group-sparsity-promoting regularizers to select connections and decide how to expand the network.

A strength of the paper is that the proposed algorithm is interesting and intuitive, even if relatively complex, as it requires chaining a sequence of sub-algorithms. It was good to see the impact of each sub-algorithm studied separately (to some degree) in the experimental section. The results are overall strong.

It’s hard for me to judge the novelty of the approach though, as I’m not an expert on this topic.

Just a few points below:
- The experiments focus on a relevant continual learning problem, where each new task corresponds to learning a new class. In this setup, the method consistently outperforms EWC (e.g., Fig. 3), as well as the progressive network baseline.
Did the authors also check the performance on the permuted MNIST benchmark, as studied by Kirkpatrick et al. and Zenke et al.? It would be important to see how the method fares in this setting, where the tasks are the same, but the inputs have to be remapped, and network expansion is less of an issue.

- Fig. 4 would be clearer if the authors showed also the performance and how much the selected connection subsets would change if instead of using the last layer lasso + BFS, the full L1-penalized problem was solved, while keeping the rest of the pipeline intact.

- Still regarding the proposed selective retraining, the special role played by the last hidden layer seems slightly arbitrary. It may well be that it has the highest task-specificity, though this is not trivial to me. This special role might become problematic when dealing with deeper networks.
review_title: Interesting approach to continual learning

final_score: rating: 6: Marginally above acceptance threshold
review_content: The topic is of great interest to the community, and the ideas explored by the authors are reasonable, but I found the conclusion less-than-clear. Mainly, I was not sure how to interpret the experimental findings, and did not have a clear picture of the various models being investigated (e.g. "base DNN regularized with l2"), or even of the criteria being examined. What is "learning capacity"? (If it's number of model parameters, the authors should just say, "number of parameters"). The relative performance of the different models examined, plotted in the top row of Figure 3, is quite different, and though the authors do devote a paragraph to interpreting the results, I found it slightly hard to follow, and was not sure what the bottom line was.

What does the "batch model" refer to?

re. " 11.9%p − 51.8%p"; remove "p"?

Reference for CIFAR-100? Explain abbreviation for both CIFAR-100 and AWA-Class?

re. "... but when the number of tasks is large, STL works better since it has larger learning capacity than MTL": isn't the number of parameters matched? If so, why is the "learning capacity" different? What do the authors mean exactly by "learning capacity"?

re. Figure 3, e.g. "Average per-task performance of the models over number of task t": this is a general point, but usually the expression "<f(x)> vs. <x>" is used rather than "<f(x)> over <x>" when describing a plot.

"DNN: dase (sic) DNN": how is this trained?



review_title: Investigation of DENs and variants applied to lifelong learning

Meta-review: PROS:
1. good results; the authors made it work
2. paper is largely well written

CONS:
1. some found the writing to be unclear and sloppy in places
2. the algorithm is complicated -- a chain of sub-algorithms

A few small points:

-I initially found Algorithm 1 to be confusing because it wasn't clear whether it was intended to be invoked for each task (making the training depend on all the datasets).  I finally convinced myself that this was not the intention and that the inner loop of the algorithm is what is actually executed incrementally.


Decision: Accept (Poster)