Venue: ICLR 2020 Conference

Reviews:

final_score: rating: 1: Reject
review_content: This work sits at the intersection between neuroscience and machine learning. It proposes to use neural recordings from the rodent hippocampus to shed light into how biological agents achieve continual learning. The approach involves (i) analyzing neural data from the rodent hippocampus with the goal of identifying how neurons encode various behavioral variables; (ii) training different RL agents to solve a computer version of the same task, including tabular and DQN agents; (iii) contrasting the performance and representations of the artificial agents with those of animals.

The paper has an admirable goal: to find links between neuroscience and machine learning, using tools from one to promote advances in the other. When done correctly, this interaction can be fruitful and produce incredible insights for both fields. However, I believe that this paper does not deliver on either end due to major methodological and analytical flaws, rendering it innapropriate for ICLR. Below I list my major critiques:

1. The interpretations of the dPCA components seems very preliminary and lacks methological rigor. In particular, many alternative interpretations of the components are possible. For instance: Component #3 might represent outcome (rather than decision); Component #4 might represent greater engagement of the hippocampus in the egocentric task (e.g. Packard and McGaugh, 1996); etc. Moreover, the reported analyses do not include animal position, which will almost certainly be a major driver of population activity in the hippocampus (see work on place cells). Since the animal position is not reported, it is impossible to know if the dPCA components (e.g. Component #2) are, instead, representing alocentric position. Including animal position (perhaps also as an explanatory variable in the dPCA) might help, but the authors should also do a more thorough job testing their specific hypotheses beyond simply reporting them based on visual inspection.

2. The comparison between early and later training sessions is also rather crude and qualitative. The authors makes several claims about differences that are not tested directly with a proper hypothesis test.

3. The comparison between RL quantities and dPCA component is also very weak. For instance, the claim that a given dPCA component represents an eligibility trace needs much more evidence than simply showing that this component decays over space when the eligibility trace also decays. With respect to Component #4 (a critical component presumably related to the authors' hypothesis of multi-task representation), the authors report that, for an e-greedy agent, the average value is higher for alocentric vs. egocentric task. Yet, this difference is not investigated further and, again the authors claim victory based on a simple visual comparison between two plots (4C vs. 4D). 

4. For most of the paper, the authors report the results from a single (typical?) animal. Ideally, the results for all animals would be reported (or some statistically sound aggregate of all animals).

5. Finally, the authors compare the performance of animal 3 with the performance of different RL agents. Again, this comparison is incredibly superficial and neglects many confound variables. The fact that the accuracy of both the animal and DQN is around 70% is not sufficient to claim that DQN is a good model for the animal's behavior, or that it is superior to other models that achieve slightly worse or better performance. Much more work needs to be done to properly compare RL and animals (e.g. comparing, in addition to performance, representations across various layers, prediction error signals, reward signals, etc). 

Overall, while I commend the authors for an intriguing idea, the execution of the idea is so poor that I consider the paper to be of little interest to either the neuroscience or the machine learning communities. Therefore, I cannot recommend the paper for publication at ICLR.
review_title: Official Blind Review #4

final_score: rating: 6: Weak Accept
review_content: This paper describes an analysis of hippocampal neuronal activity in rodents during spatial navigation tasks. Features were extracted from the data using a component analysis technique, and these features were then compared to quantities which arise during training of the DQN reinforcement learning algorithm. 

Unfortunately I am not well versed enough in this literature to assess the merits of this submission. However, I could not find any glaring issues, unclear sentences or any other obvious sign of incompetence. 

My apologies for the inadequacy or this review (and/or the paper assignment algorithm).
review_title: Official Blind Review #1

final_score: rating: 3: Weak Reject
review_content: This work focuses on the problem of continual learning. It first proposes an analysis of neural recordings from rodents hippocampus that perform a task related to continual learning. The authors then claim to identify similarities in representations of behaviorally relevant variables between biological networks and standard artificial RL systems. Finally the authors propose a DQN implementation of the task reaching performance similar to rodents, although the implementation fail to perform continual learning.

I have found the results of the analysis of neural data interesting and well explained. However I haven’t been able to judge positively the main part of the manuscript pertaining to the comparison with artificial agents, maybe because of the lack of clarity in the exposition of the results (see detailed comments below). As such I think this manuscript is not ready yet for publication in ICLR. 

Detailed comments:

Introduction:
The introduction reads well, it would be useful to have a short paragraph explaining the vast problem of continual learning, and situating the approach of the authors in this vast field. In particular it would be useful to explain in what sense the task considered is related to continual learning (switches between allocentric and egocentric tasks that are not informed by experimenter, but that the rodent has to figure out), because for now it is only briefly stated in the legend of figure 1.

Analysis of neural data: 
Analysis of neural data using dPCA reveals interesting results regarding the representations of correct/incorrect trials, allocentric/egocentric tasks, temporal structure. Globally these results are well presented. Regarding figure 3, it would be nice to define what are early and late trials, is it a distinction inside a block where late trials are just before a switch. Or are late trials after weeks of training, in which case I do not really see the link with continual learning ?


Comparison between standard RL and hippocampal representations:

The idea to compare hippocampal representations and standard RL implementations is original and interesting. The exposition of the results, however, lacks important information for me to assess the relevance of these comparisons:

Fig4A(i): could the author explain what is on the x-axis ? What is the 1-D environment mentioned in the caption (in SM a 5*5 grid world is mentioned) ? Why not run TD on the same setting as Q-learning ? What is the black curve « value after learning » ? Why value depends on the x-axis ?
Fig4C: How are the two curves obtained ? As both tasks share the same Q function, it might be worth explaining how the two curves are obtained. 

The authors mention that « policy information is override as starting location changes over » ? Could this fact be explained ? It would seem to me that starting points north s_north and south s_south would have distinct Q(s_north,.) and Q(s_south,.) avoiding overriding.

Also I am quite surprised that Q-learning fails on such a simple task (cf Fig5B), could the authors explain how many trials would be required to reach perfect performance by focusing on either of the two tasks ?

In this section a natural extension could be to use hierarchical RL, or options to model behavioral strategies such as allocentric and egocentric and compare with the hippocampal recordings.


Comparing DQN and animal performance:

A first crucial point to clarify is whether the DQN receives information about what task to perform (it is mentioned: « task-specific information provided to the neural network as input«, «this network with task-specific memory«, then a DQN « without task-specific information » is mentioned). This seems rather crucial, because if information about allocentric VS egocentric task is provided, contrary to the behavioral task for rats, then the network is asked to do strategy switches based on some cues, which is a different problem than continual learning.

If the DQN receives information about whether it is in a allocentric or egocentric trial, I am very surprised it is not able to perform this simple task. 
Could the authors explain why the DQN has similar performance as the animal while it is shown to relearn slower than animals (is it because it learns better before task switch). 
In figure 7, the curves for the model are very noisy and hence difficult to interpret, it would be nice to show averages over many same task switches, to get a clearer picture.
review_title: Official Blind Review #3

final_score: rating: 1: Reject
review_content: This paper analyses a dataset of representations in the CA1 region of the hippocampus of a rat conducting a spatial plus maze task that switches between allocentric and egocentric versions. In the allocentric version of the task, the rat must always go from north or south arms to the west arm to receive a reward. In the egocentric version the rat must always turn right, regardless of whether they are starting in the north or south arm. These two versions of the task are switched after some period of time.

The researchers conducted demixed principal components analysis (dPCA) on the data, then compared the activity of the components to aspects of a tabular Q-learner. They argue that some of the components match variables used by the Q-learner. Following this, they examined the ability of both the tabular Q-learner and a deep Q-network to perform this task, and compared their performance to the rat. They find that the rat is better than the Q-learners at continually updating from the egocentric to allocentric task, and vice-versa.

The goals of this paper are fantastic. I really like the attempt to link hippocampal activity with RL representations. But, this study is very muddled, and there are very serious problems with the paper that render it inappropriate for acceptance to ICLR. The five most major issues are:

(1) The choice of the demixing categories is missing a crucial category: space! Given the importance of the hippocampus for spatial representations, surely a big component of the variance can be explained by the animal's location in space. Why not include this? It might in fact be that some of the "time" components are just reflecting the usual spatial location of the animal during different components of the task.

(2) The identification of components from the neural data with things like reward prediction errors, eligibility traces, etc., is all done in a qualitative manner with no statistical controls. This lack of quantitative assessment for some of the key claims in the paper is just not okay for an academic paper. Moreover, even the qualitative claims are underwhelming. The authors' claim, for example, that time component #1 is an eligibility trace is a real stretch, in my opinion (see more below in point 3).

(3) The comparison between the animal data and the Q-learner is also done in a qualitative fashion that was extremely underwhelming. Fig 4 A & B were the most egregious. Are the authors really attempting to claim that the curves in 4A are clearly related to the curves in 4B? That's a real stretch, and to provide no quantitative assessment of this claim renders it completely unconvincing.

(4) The fact that model-free reinforcement learning algorithms cannot adapt in changing environments/tasks has been known for a long time. As such, the result showing that the Q-learners cannot switch easily between the tasks is not novel. See, for example, this paper: https://www.sciencedirect.com/science/article/pii/S0896627313008052

(5) Even if we accept the central claims from this paper, there is very little provided for machine learning researchers at ICLR to benefit from. What about the hippocampal representations makes the rats better at continual learning? What inductive biases or memory mechanisms might we glean from this work? Nothing like that is provided. As it stands, even being charitable, this paper really only speaks to a neuroscience audience, since even the Q-learning components are used only to understand the neuroscience data, not to think about how this could inform new ML systems or theories.

In addition to these problems, there are several small ones:

- Was only one animal included in this analysis? That's never stated, but Figs. 5 & 7 seem to suggest that. Not only does this 100% need to be stated, but it's very problematic from a generality standpoint.

- What type of recordings were these? How were individual cells identified (e.g. spike sorting)?

- How was the Deep Q-net trained exactly? The authors say it was trained on the first 250 trials from the animal, but that's confusing? Was it not trained to perform the task itself? Also, where is all the info on memory buffers, hyperparameters, etc. There is no way to reproduce these simulations given the lack of detail here.

- Some of the plots are confusing and hard to follow. For example, in figure 7 B and C, what determines the X-axis? What should I be looking for in the curves? 
review_title: Official Blind Review #2

Meta-review: This paper analyzes neural recording data taken from rodents performing a continual learning task using demixed principal component analysis, and aims to find representations for behaviorally relevant variables. They compare these features with those of a deep RL agent.

I am a big fan of papers like this that try to bridge between neuroscience and machine learning. It seems to have a great motivation and there are some interesting results presented. However the reviewers pointed out many issues that lead me to believe this work is not quite ready for publication. In particular, not considering space when analyzing hippocampal rodent data, as R2 points out, seems to be a major oversight. In addition, the sample size is incredibly small (5 rats, only 1 of which was used for the continual learning simulation). This seems to me like more of an exploratory, pilot study than a full experiment that is ready for publication, and therefore I am unfortunately recommending reject.

Reviewer comments were very thorough and on point. Sounds like the authors are already working on the next version of the paper with these points in mind, so I look forward to it.

Decision: Reject