Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: The paper presents a novel framework for reinforcement learning, specifically for evidence accumulation tasks that involve counting quantities and deciding which is greater. The general approach is to use cognitive models as inductive biases by learning a function representing the evidence and giving the resulting representations to an RL algorithm. The paper includes a discretized version of the Laplace transform-based framework to use with a neural network.

The framework is as follows: the Laplace transform is rewritten as a differential equation in terms of the numerosity of the stimulus and discretized; an inverse is also derived. This is meant to enable learning of a counted quantity. To subtract one quantity from another and determine which one is higher, the paper presents a cross-correlation mechanism. 

This framework integrates into an RL algorithm (here A2C) by having environment inputs go into a recurrent layer, have that layer learn a direct representation of the count of the objects and their difference (in theory), and then pass the output of this layer to the actor and critic networks. 

Experiments are done on an "accumulating towers" task. There are several ablations removing various parts of the Laplace framework, as well as RNN and GRU layers as baselines. Task reward statistics show that the Laplace and inverse Laplace transforms with subtraction do very well, as does GRU. Other ablations have mid-range performance, and RNN does quite poorly. Frozen versions of RNN and GRU are unsurprisingly very poor.

When evaluating on harder versions of the task, Laplace with subtraction continues to do well, as does GRU. The paper attributes this to the gating mechanisms present in both methods.

Finally, there is some analysis of neural activity and its connections to human cognitive maps. Results show that Laplace with subtraction appears to activate sequentially as a function of evidence. GRU and other versions of the Laplace framework also shows some tuning to "magnitude"/"amount of evidence".
strength_and_weaknesses: **Strengths**
- Introduction is strong and motivates the paper well. The idea of cognitive models providing inductive biases is not novel but this is a very interesting instantiation of it
- The connections to human cognitive maps are more convincing than a lot of "human cognitive construct + our cool new method" connections that ML papers attempt to make, especially because of follow-up evidence in the paper such as the logarithmic representation. While even this one is ultimately conceptual (and the logarithmic representation appears to be a design choice rather than an emergent property, if I understand correctly), I find it compelling.
- The paper is well-packaged - it makes several claims and demonstrates them, and the methods solve exactly the problem motivated and outlined. To some degree, the final contribution about gating could use more expansion, but I realize it's more a validation of prior work and I appreciate the acknowledgement of the similarities/power of existing work

**Weaknesses**
- The Laplace framework presented here seems very tuned to this specific task, or at least counting-based tasks. The paper acknowledges in "Future Work" that this is by no means the only type of latent variable and encourages work on others, but as of right now the framework is engineered in terms of counting - or at least, that's the sense I get reading it. It's still an interesting method, but GRUs and RNNs are much more general modules; it would be helpful to have (brief!) discussion of either 1) the domain limitations of these equations or 2) the potential to easily adapt this method to other tasks. I see how this formulation may extend to some other quantities, but I don't know for sure. 
- The discussion I described above would make the comparisons in this paper make more sense, but for actual acceptance I'm not sure that would be enough. This is an interesting idea and novel to my knowledge, but without seeing experiments in other domains, it's not clear how much value it adds. A more general approach covering a larger domain (potentially through multiple frameworks like this one), and experimental results on those, would be more convincing.
- From a pure utility perspective, it's unclear how much value this method provides over GRU (not to mention, it has a narrower domain if I understand correctly). Because the performances are so similar, some statistics differentiating them would help.
- The word "deep" is used in the title and throughout the paper, but the version of A2C experimented on uses single-layer FCNs for actor and critic. The paper isn't making claims about scale or ability to handle large real-world tasks or complicated environments, so this claim seems not only false but unnecessary to the story.
clarity,_quality,_novelty_and_reproducibility: **Clarity**
- Section 2 is quite clear given the math in it; while it did take a couple reads to build intuition for the Laplace framework, it wasn't overly daunting or impossible
- Results section could use better claims-based signposting, rather than topic-based, or just a general info dump as most of it is now (which is not to say that long section doesn't read well; it does). Three of the four contributions are claims that will be demonstrated, so the results section may benefit from section headings based on those claims. 

**Quality**
- Experiment section is carefully considered. The experiments designed to support each claim in the contributions are intuitive and convincingly designed
- The paper doesn't leave any loose ends or unanswered questions. It feels like a finished piece of work.

**Originality**
- The high-level concept of cognitive models as inductive biases (or some such integration) is familiar, but I believe this particular approach for evidence accumulation is novel. I think the contribution list is quite an accurate and complete summary of the parts of this paper that are novel.

Nits:
- "Amount of evidence" is strange phrasing here. It seems from the rest of the paper and the place cell example that this means *n* itself, but the phrase sounds like it means, how much evidence we have that the value is *n* as opposed to other values. 
summary_of_the_review: I am recommending a weak acceptance for this paper. It is a good paper and I enjoyed reading it, but I am not convinced of its significance from a general modeling perspective given the limitations I think I see in its domain. I also am not totally sure I understood all the math, so I may be missing limitations; I may also be missing strengths. 

All that said, the quality of idea and execution is far above a workshop paper and it is a very compelling read, so I wouldn't consider it fair to recommend rejection just because it's an interesting and well-demonstrated idea rather than the next big SOTA system like a lot of papers. I would raise my score if I were convinced that this opens a lot of future work directly (not just in high-level concept) or enables improvement on new tasks out of the box, and if I were convinced that it provides meaningful improvement over existing methods.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: The paper describes a network that combines a Laplace framework into a reinforcement learning architecture achieve superior performance over GRU and RNN in an evidence accumulation task both in terms of reward gain and generalization. Moreover, the activity of units in the proposed architecture resembles map-like neural activity in the brain for accumulated evidence. 
strength_and_weaknesses: Strength: The paper looks into an interesting and commonly used task in decision making and RL experiments in the field of Neuroscience. 
The paper is clearly written and different components are very well explained step by step. It also provides a virtual environment, useful for the community to simulate evidence accumulation experiments. 

Weakness: This work only covers one specific task from one family of experiments (Evidence accumulation). The paper can be significantly improved if the method is tested on broader range of tasks in the face of uncertainty (e.g. see "Recurrent Model-Free RL Can Be a Strong Baseline for Many POMDPs" by Ni et al, ICML 2022), and/or works on hippocampus and cognitive maps (many references  presented in the paper) and/or different evidence accumulation tasks in neuroscience (e.g. see works of Michael Shadlen lab for human and monkey, and Carlos Brody lab for rodent). In terms of architecture, comparison of an architecture with a Bayes filter (e.g. see implementations of "End-to-end learnable histogram filters", by Bonschkowski & Brock, NeurIPS deep RL workshop 2016, or "QMDP-Net: Deep Learning for Planning under Partial Observability" by Karkus et al, NeurIPS 2017) instead of Laplace framework could be illuminating. 

clarity,_quality,_novelty_and_reproducibility: The paper is clear and very easy to follow. I also appreciate its reproducibility particularly because of the provided supplementary material. The modelling/theoretical contribution as well as experiments/results however is limited.  
summary_of_the_review: I think because of lack of significant modeling/theory contribution as well lack of test-domains this paper is not ready to be presented in the ICLR venue yet. 

Update after rebuttal: 
I appreciate the authors' response and adding more experiments. The added experiments, however, are very much in the same domain of previous one and only a threshold has been changed (the accumulation mechanism is the same). In the evidence accumulation task (e.g. in mentioned labs), a sufficiently task has either 1) different number of options (e.g. more than 2) 2) different decision time criterion (reaction time v.s. fixed) or 3) dynamic weigh of evidence through the task. Moreover, even if the proposed approach is new, we do not know whether it is works better than a simple Bayesian update, which is in fact shown to be able to solve evidence accumulation task.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This paper embeds a learnable recurrent network based on Laplacian transformation into a reinforcement learning agent and claims that the Laplacian based recurrent network outperforms the others.
strength_and_weaknesses: It seems to be novel to use Laplacian transformation to derive a neural representation of evidence or numerosity. I believe this framework may suggest some general principle of neural coding. Nevertheless, I feel some parts of the paper are not well written and can be improved further.

### Major concerns

- Based on the text it is unclear to me that how the authors embed the Laplacian transformation into a recurrent network, even if I read the manuscript several times. Do authors firstly perform a Laplacian transformation on the input, and then feed the transformed inputs to a recurrent net? Or do authors use the Laplacian transformation to propose or constrain the recurrent net structure? 

- The connection between Eqs. 3 and 7 implicitly suggests that the firing rate of neurons represents the Laplacian transformation of an input. It is better to have more detailed explanations at here.

### Minor concerns

- If I understood correctly, Eq. 3 is wrong because it is inconsistent with Eq. 2, while Eq. 4 is correct. There must be some typos.

- It is not clear of the meaning of n* in Eq. 5. Is it the preferred evidence of a neuron? Also, is $\tilde{f}(n*, n)$ the reconstruction of $f(t)$ by only using a Laplacian basis?

- $\alpha(t_i)$ is not defined and I need to guess. Is it the instantaneous change of numerosity as mentioned right above Eq. 3? Besides, is the modulatory input $(\alpha=1)$ also the change of numerosity? If so, I don't understand why $\alpha=1$ represents no modulatory input.

- The definition of $W_L$ right below Eq. 7 is contradictory with each other and quite confusing.

- The math definition in Eq. 8 is no problem but I don't understand why author call the cross-correlation as subtraction between functions.

- The 4th line to the 7th line below Eq. 8 is repetitive.



clarity,_quality,_novelty_and_reproducibility: The current paper presents a novel idea of using Laplacian transformation to develop/inspire a recurrent network. However, the writing is not very clear (see my comments above) and some typo exists. It is unclear how technically the Laplacian transformation is embedded into the recurrent network. Since this is a small network model and I believe the paper can be reproduced without too much effort.
summary_of_the_review: I have gone through the math derivations and read the manuscript for several times and I believe I try my best to understand the paper but there are still some places which are not clear.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: This paper presents a model for evidence accumulation using a Laplace transform-based framework embedded in a recurrent RL architecture trained in an “accumulating towers” task, inspired by animal neuroscience. They find that, compared to vanilla GRU and RNN-based RL agents, their approach achieves better performance and generalization, and that there is evidence of the emergence of brain-like representations, although this evidence seems mixed.

Initially, the reviewers didn’t particularly feel strongly one way or another, with 2 recommending weak reject and 1 recommending weak accept. They indicated that they liked the motivation of the work and the ties to neuroscience which are interesting, and found it overall well-presented. However there were concerns that experiments were only performed on a single task, limiting the generality of their conclusions. 

In the rebuttal period, the authors added experiments on two additional tasks, but reviewers noted that since they were in the same domain and were not meaningfully different from the original task, this was not enough to satisfactorily address this main criticism. Reviewer ojuW also indicated that the work was missing comparison to additional models from the decision-making literature and is thus still not ready for publication.

During subsequent discussions, all reviewers continued to express concerns that this paper wasn’t quite ready for publication, including tPG3, who was previously most positive.

In the end, reviewers appeared to be in agreement that this paper would benefit from more time, a wider range of tasks, and more model comparisons. Therefore, I cannot recommend acceptance, but look forward to future iterations of this work with reviewers’ feedback incorporated.

justification_for_why_not_higher_score: The reviews for this paper were a bit mixed, with some noting that the experiments were too limited, while another found the links to neuroscience to be compelling and the overall work above that of a workshop paper. On balance, I didn't find the reasons for acceptance to outweigh the weaknesses noted in the reviews.
Decision: Reject