Venue: ICLR 2018 conference

Reviews:

final_score: rating: 8: Top 50% of accepted papers, clear accept
review_content: In this paper, the authors present a novel formulation for learning the optimal architecture of a neural network in a multi-task learning framework. Using multi-agent reinforcement learning to find a policy — a mapping from the input layer (and task indicator) to the function block that must be used at the next layer, the paper shows improvement over hard-coded architectures with shared layers. 

The idea is very interesting and the paper is well-written. The extensive review of the existing literature, along with systematically presented qualitative and quantitative results make for a clear and communicative read. I do think that further scrutiny and research would benefit the work since there are instances when the presented approach enjoys some benefits when solving the problems considered. For example, the more general formulation that uses the dispatcher does not appear to work ver well (Fig. 5), and the situation is only improved by using a task specific router. 

Overall, I think the idea is worthy of wide dissemination and has great potential for furthering the applications of multi-task learning. 
review_title: Good work

final_score: rating: 7: Good paper, accept
review_content: The paper introduces a routing network for multi-task learning. The routing network consists of a router and a set of function blocks. Router makes a routing decision by either passing the input to a function block or back to the router. This network paradigm is tested on multi-task settings of MNIST, mini-imagenet and CIFAR-100 datasets.

The paper is well-organized and the goal of the paper is valuable. However, I am not very clear about how this paper improves the previous work on multi-task learning by reading the Related Work and Results sections.

The Related Work section includes many recent work, however, the comparison of this work and previous work is not clear. For example:
"Routing Networks share a common goal with techniques for automated selective transfer learning
using attention (Rajendran et al., 2017) and learning gating mechanisms between representations
(Stollenga et al., 2014), (Misra et al., 2016), (Ruder et al., 2017).  However, these techniques have
not been shown to scale to large numbers of routing decisions and task." Why couldn't these techniques scale to large numbers of routing decisions and task? How could the proposed network in this paper scale?

The result section also has no comparison with the previously published work. Is it possible to set similar experiments with the previously published material on this topic and compare the results?


-- REVISED

Thank you for adding the comparisons with other work and re-writing of the paper for clarity.
I increase my rating to 7.


review_title: Revised Review 

final_score: rating: 6: Marginally above acceptance threshold
review_content: Summary:
The paper suggests to use a modular network with a controller which makes decisions, at each time step, regarding the next nodule to apply. This network is suggested a tool for solving multi-task scenarios, where certain modules may be shared and others may be trained independently for each task. It is proposed to learn the modules with standard back propagation and the controller with reinforcement learning techniques, mostly tabular. 

-	page 4: 
In algorithm 2, line 6, I do not understand the reward computation. It seems that either a _{k+1} subscript index is missing for the right hand side R, or an exponent of n-k is missing on \gamma. In the current formula, the final reward affects all decisions without a decay based on the distance between action and reward gain. This issue should be corrected or explicitly stated.

The ‘collaboration reward’ is not clearly justified: If I understand correctly, It is stated that actions which were chosen often in the past get higher reward when chosen again. This may create a ‘winner takes all’ effect, but it is not clear why this is beneficial for good routing. Specifically, this term is optimized when a single action is always chosen with high probability – but such a single winner does not seem to be the behavior we want to encourage.

-	Page 5: It is not described clearly (and better: defined formally) what exactly is the state representation. It is said to include the current network output (which is a vector in R^d), the task label and the depth, but it is not stated how this information is condensed into a single integer index for the tabular methods. If I understand correctly, the state representation used in the tabular algorithms includes only the current depth. If this is true, this constitutes a highly restricted controller, making decisions only based on depth without considering the current output. 
-	The functional approximation versions are even less clear: Again it is not clear what information is contained in the state and how it is represented. In addition it is not clear in this case what network architecture is used for computation of the policy (PG) or valkue (Q-learning), and how exactly they are optimized.
-	The WPL algorithm is not clear to me
o	 In algorithm box 3, what is R_k? I do not see it defined anywhere. Is it related to \hat{R}? how?
o	Is it assumed that the actions are binary? 
o	I do not understand why positive gradients are multiplied with the action probability and negative gradients with 1 minus this probability. What is the source of a-symmetry between positive and negative gradients?  

-	Page 6:
o	It is not clear why MNist is tested over 200 examples, where there is a much larger test set available
o	In MIN-MTL I do not understand the motivation from creating superclasses composed of 5 random classes each: why do we need such arbitrary and un-natural class definitions? 

-	Page 7: 
The results on Cifar-100 are compared to several baselines, but not to the standard non-MTL solution: Solve the multi-class classification problem using a softmax loss and a unified, non routing architecture in which all the layers are shared by all classes, with the only distinction in the last classification layer. If the routing solution does not beat this standard baseline, there is no justification for its more complex structure and optimization.

-	Page 8: The author report that when training the controller with single agent methods the policy collapses into choosing a single module for most tasks. However, this is not-surprising, given that the action-based reward (whos strength is unclear) seems to promote such winner-takes-all behavior.

Overall:
-	The paper is highly unclear in its method representation
o	There is no unified clear notation. The essential symbols (states, actions, rewards) are not formally defined, and often it is not clear even if they are integers, scalars, or vectors. In notation existing, there are occasional notation errors. 
o	The reward is a) not clear, and b) not well motivated when it is explained, and c) not explicitly stated anywhere: it is said that the action-specific reward may be up to 10 times larger than the final reward, but the actual tradeoff parameter between them is not stated. Note that this parameter is important, as using a 10-times larger action-related reward means that the classification-related reward becomes insignificant.
o	The state representation used is not clear, and if I understand correctly, it includes only the current depth. This is a severely limited state representation, which does not enable to learn actions based on intermediate results
o	The continuous versions of the RL algorithms are not explained at all: no state representation, nor optimization is described.
o	The presentation suffers from severe over-generalization and lack of clarity, which disabled my ability to understand the network and algorithms for a specific case. Instead, I would recommend that in future versions of this document a single network, with a specific router  and set of decisions, and  with a single algorithm, will be explained with clear notation end-to-end

Beyond the clarity issues, I suspect also that the novelty is minor (if the state does not include any information about the current output) and that the empirical baseline is lacking. However, it is hard to judge these due to lack of clarity.


After revision:
- Most of the clarity issues were handled well, and the paper now read nicely
- It is now clear that routing is not done based on the current input (an example is not dynamically routed based on its current representation). Instead routing depends on the task and depth only.  This is still interesting, but is far from reaching context-dependent routing.
- The results presented are nice and show that task-dependent routing may be better than plain baseline or the stiching alternative.  However, since this is a task transfer issue, I believe several data size points should be tested. For example, as data size rises, the task-specific-all-fc alternative is expected to get stronger (as with more data, related task are less required for good performance).
 

- 
review_title: The paper is not clear enough.

Meta-review: The proposed routing networks using RL to automatically learn the optimal network architecture is very interesting. Solid experimental justification and comparisons. The authors also addressed reviewers' concerns on presentation clarity in revisions.
Decision: Accept (Poster)