Venue: ICLR 2020 Conference

Reviews:

final_score: rating: 6: Weak Accept
review_content: This is a fascinating paper that uses methods from computational neuroscience to characterize a neural network that controls a virtual rat (or, well, something ratlike). I really like the idea of trying to fuse neuroethology and animal behavior research in general with deep learning methods. To me, it seems like there was plenty of activity in this field around 20 years ago (with e.g. the robot models of animal behavior of Ijspeert and others, and the evolutionary robotics approach to study the evolution of behaviors) but that this research line has not merged with (alternatively informed, or learned from) modern deep learning. So I welcome this direction of research. This being said, it's not my field of research, so I'm unable to comment on several of the specifics here.

The learning of a network that can perform these four independent tasks is quite impressive in its own right. I would like the paper to comment on how hard or easy this was, if you attempted to learn other behaviors but failed, etc.

The actual analysis is a bit of a letdown - not because it seems to be wrong or incompetently done, rather the opposite - but in that there is so little to learn from it. Simply put, I do not understand anything more about networks that control simulated robots to perform multiple tasks (or about real rodents) after reading the paper. What does this mean? Is this an indictment of current neuroscience methods, that even when you have unambiguous non-noisy access to all of the state of the network, you cannot really find out much? If so, you should discuss this. If not, you should explain what's going on. At least from the perspective of an AI researcher, there just isn't enough understanding there.

However, I don't think negative results (which this in a way is) should discourage from publication. I applaud the intent of the paper, the competence with which it was executed, and the learning of the network. So I want to see this published. But please remark on the points above.
review_title: Official Blind Review #3

final_score: rating: 6: Weak Accept
review_content: 
=============================== Update after rebuttal ======================================================

I thank the authors for their rebuttal and the revisions. I'm not entirely satisfied with the authors' response to my request for more architectural exploration, but I understand that there wasn't really enough time for this during the relatively short rebuttal period. I think the results in the paper are still valuable enough to merit publication, so I'm happy to increase my score and recommend accepting the paper. However, I still encourage the authors to take the issue of arhcitectural plausibility more seriously in the future, especially if the intended audience for this line of work is experimental neuroscientists. Currently, the architecture choice seems to be dictated primarily by trainability considerations (more specifically, trainability by current deep learning methods).

========================================================================================================

This paper introduces a virtual rodent model with a complex set of actuators and visual and proprioceptive inputs. The model is simultaneously trained on four different tasks and the trained model is analyzed using various dimensionality reduction and visualization methods. The effort put into training and analyzing the rodent model is quite impressive. Moreover, the tools that the authors will make public can be useful for other researchers in this field. 

However, my main concern about the paper is that given the architecture choice made in the paper, most of the main results do not seem very surprising. On the other hand, the architecture choice itself is not motivated well enough. The differences between the dynamic and representational properties of the core and policy networks entirely depend on the fact that the core network is trained separately from the policy network. Why was this particular choice made? In a more realistic scenario (for example, in the actual brain of a rodent), everything will presumably be connected to most everything else to a certain degree, with no sharp separation between policy and core modules and the error signals flowing more broadly across the entire network. It seems to me that in such a scenario, there wouldn't be such a big difference between the dynamic and representational properties of different modules in the network. So, I am wondering if it is possible to train more models with alternative architectures (with presumably more realistic properties) and compare the results with the current results.
review_title: Official Blind Review #2

final_score: rating: 8: Accept
review_content: The authors use a virtual rodent to study flexible behavior, a key concept at the intersection of AI and neuroscience. “Flexibility” is a common buzzword in this field used to describe what it is that mammals do better than deep nets, even as deep nets can outperform them on any one task in isolation with sufficient training. As the authors note, previous work addresses “cognitive flexibility” (Yang et al. 2019) in isolation without motor commands, or studies motor dynamics of real mammals but without any task flexibility—occluding the actual advantages that mammals have over machines. However, this work goes the full mile in requiring the agent to implement flexible cognition using a complex body in a physical environment with egocentric stimuli. From a strictly AI point of view, I could imagine arguments that this is not sufficiently “novel,” in that a lot of deep RL researchers have already trained agents with complex joint structures to succeed in such virtual worlds. But the real substance of the paper is the application of data analysis techniques imported directly from neuroscience. It is important to suss out what features of neural dynamics are peculiarities of biological computation and which are natural consequences of the tasks being performed. I find the paper to be overall well motivated.

The analyses were well ordered, starting with a high-level characterization of behavior that showed a hierarchy of features, where “low-level” or “fast” features are mixed across tasks while “high-level” or “slow” features are more task specific. They then moved on to find “neural correlates” at the behavior and task levels, driving home their primary point that core layers encode high-level variables in a way not shared across tasks while policy layers implement reusable core motor functions that are used in multiple contexts. This observation coincides with the measurable frequency content of core and policy activity, which is demonstrably faster in policy layers. Finally, the authors show a visualization of the dynamical systems that underlie these computations, with videos that very clearly illustrate how core and policy dynamics dissociate to specialize in cognition vs. motor generation. This part is where they really leverage the artificial setting, where the exact dynamics are known and do not need to be inferred from noisy neural data.

While the end result of the analyses led to convincing visuals that made the intended point, the details of the analysis themselves were hard to follow and evaluate for rigor. An appendix explaining with notation how the behavioral “features” were computed, for example, would have been helpful. I acknowledge that it is a feature of the paper that so many different analysis techniques from neuroscience—dynamical systems to representation analysis to lesion studies—were applied, and the sheer breadth of the analysis was impressive, if difficult to evaluate.

Although I recognize that to do so would be difficult, I think what would add a lot to this paper would be side-by-side comparisons of real neural data with “neural” data from the artificial rodent. Perhaps the behavior would have to be modified so that some tasks match specific neuroscience paradigms, or the sample size for analysis limited to what is realistic for experimental data. It would be neat to see if, say, preparatory activity signals could emerge from training such an artificial rodent to do many tasks, including a head-fixed timed lever pull or some other classic paradigm. Maybe future work can draw a more exact analogy between policy/core layers and specific brain regions.

As to topic fit, this is niche in the context of ICLR; however, showing how state-of-the-art deep RL techniques can be applied to neuroscientific questions may be of interest to machine learning researchers even if they are not also neuroscientists.

Overall, I thoroughly enjoyed this paper and look forward to the new kinds of research that sprout out of it!
review_title: Official Blind Review #1

Meta-review: This paper is somewhat unorthodox in what it sets out to do: use neuroscience methods to understand a trained deep network controlling an embodied agent. This is exciting, but the actual training of the virtual rodent and the performance it exhibits is also impressive in its own right. All reviewers liked the papers. The question that recurred among all reviewers was what was actually learned in this analysis. The authors responded to this convincingly by listing a number of interesting findings. 

I think this paper represents an interesting new direction that many will be interested in.
Decision: Accept (Spotlight)