Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 8: Strong Accept: Technically strong paper, with novel ideas, excellent impact on at least one area, or high-to-excellent impact on multiple areas, with excellent evaluation, resources, and reproducibility, and no unaddressed ethical considerations.
review_content: summary: This paper introduces an inverse reinforcement learning framework that infers the time-varying reward function of the animal. The reward is designed as a linear combination of rewards maps, which can be weighted differently. The proposed method is confirmed using simulated data and real mouse trajectories. 
strengths_and_weaknesses: Strength
-      Interesting story with important motivation. The inverse RL approach to time-varying reward is novel. 
-      Well-written paper. Easy to follow. All figures are easy to read and well-organized. 
-      Supplementary material is helpful and well-organized.

Weakness
-      Little impact on the ML community (in terms of ML algorithm). Nevertheless, it is a valuable work to the neuroscience community. 
-      Labels are missing in some figures, which makes it hard to interpret the graph.


questions: Overall, the reviewer enjoys reading this paper. It is excellent work, and the NeurIPS community will like it. However, there are a few clarification questions as follows.   
- How is the artificial mouse trained in Section 4.1? The reviewer assumes that the authors train the artificial mouse to generate the trajectories. Is any specific RL algorithm used to train the agent? 
- All the results included in the paper are based on two maps scenario. The inferred weights in Figure 4B and Figure 5B are only two types. Three maps scenario is presented in the Supplementary material. However, Is it possible to extend this method to multiple maps? Maybe more than 5? If then, this work can be useful in the ML and neuroscience communities. 
-  Some essential pieces of literature are missing. There are recent NeurIPS papers that study RL or IRL in decision-making problems in neuroscience. These can be discussed in related works.
 "A neurally plausible model learns successor representations in partially observable environments." NeurIPS 2019, 
 "Inverse rational control with partially observable continuous nonlinear dynamics." NeurIPS 2020.



limitations: The authors describe the limitation in the last section.
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: # Summary
This paper presents an inverse reinforcement learning algorithm designed specifically for navigation tasks involving non-stationary reward functions.
The approach is motivated by applications in neuroscience, for example understanding the (time-varying) goal-seeking or exploratory behavior of animals.
The algorithm builds on maximum entropy IRL, it assumes access to the transition model of the environment (in performing soft value iteration), and is designed for discrete state spaces (in how it defines features).
The rewards are defined as a linear function of learned goal-map features and learned weights associated with those features.
The non-stationarity of the environment rewards is handled through learning feature weights at each timestep.
The approach is evaluated  in a simulated domain matching the assumptions of the algorithm as well as on real-world data collected from (water restricted and unrestricted) mice navigating a maze.

# Contribution
- I believe the contribution is a method (based on maximum causal entropy IRL) for inferring (time-varying) reward functions of animals performing navigation tasks, specifically in settings with discrete states, deterministic transitions, and where time-alignment of trajectories is practical

strengths_and_weaknesses: # Strengths
1. The paper is clear and well-written
2. The paper is motivated by an important problem (better understanding human/animal behavior)
3. The paper focuses on less common aspects and applications of IRL. For example, the application of IRL in a neuroscience context is interesting and less common than applications in other areas (e.g., robotics or autonomous driving), and applying IRL in a context where learning rewards is itself a primary goal is interesting (for example, the differences in the learned goal maps between the two mice cohorts).
4. The approach does seem applicable to understanding animal behavior in situations similar to those considered in the paper
5. The supplementary material provides extensive details on the algorithm and experiments (including well-documented code allowing for reproducing experiments), and execution of the experiments seems sound


# Weaknesses
## Originality
- The main algorithmic novelty seems to be learning time-varying rewards; however, it seems plausible to me that this non-stationary aspect could instead be addressed by treating the problem as partially observable (where the timestep or the history of the agent are unobserved). 
- In the context of _RL_ non-stationary MDPs can be formulated as _RL_ within a specific type of partially observable MDP [4]. Does that connection hold for _IRL_? For example, can you simply specify the timestep as a feature (or specify a feature capturing historical information) and thereby remove the need for time-varying rewards? Can you instead use IRL methods designed for partially-observed environments [5]? What's the benefit of the time-varying reward formulation over these alternatives?
- Separately, what’s the relationship with existing work that discusses IRL in the context of non-stationary reward functions such as [3]?

## Relationship with prior work
- The paper would be improved by clarifying the relationship between the proposed algorithm and existing work (citations to relevant work are provided in section 3.2, but describing the relationship specifically would help)
- The algorithm seems to be a special case of maximum causal entropy IRL [1]. Both algorithms alternate between the two steps of (i) computing a policy on the current set of rewards via soft VI and (ii) updating rewards to maximize the likelihood of the observed data under the learned policy. This approach is discussed in [1], and a more recent discussion that makes the connection clear is given in [2] (section 3.2). In particular, the proposed algorithm assumes a deterministic transition function, and therefore employs simplified likelihood calculations in eqs (5) and (6) as discussed in section 3.3 of [2].
- The algorithm learns parameters of both the features and their weights, which existing work does as well (e.g., reference 30 in the paper), though with a particular structure and using coordinate ascent.
- Additionally, it would help to distinguish from IRL work in understanding human behavior. For example, what’s the relationship with IRL methods used for predicting pedestrian behavior in navigation tasks, for example [6,7]?

## Significance
- The approach doesn't seem to scale to large / continuous state/action spaces (due to the use of soft value iteration and defining features per-state). Section 5 discusses an extension that would learn embeddings, but it’s not clear to me how this would work.
- The approach doesn't seem applicable in cases where the transition model is stochastic and/or unknown (due to using the deterministic transition model form of the likelihood in eq (5) and (6), and due to using soft VI). 
- The structure of the features seem specific to discrete state-space navigation tasks
- Time-varying rewards seem difficult to apply in many situations. For example, the clustering and alignment procedure used in the paper (that seems to require each trajectory to be aligned and of the same length) would be difficult to apply to trajectories of animals/humans in the real world (where natural start and end points are likely unavailable).
- Despite the above limitations, it does seem the approach is applicable to understanding animal behavior in situations similar to those considered in the paper. I think this is significant, but it's not clear to me how significant. I think the paper would benefit from further arguing why this particular setting is important.

## Experimental quality / analysis
- Why do MaxEntIRL and DeepMaxEntIRL perform poorly? Section 4.2.3 claims they perform poorly because they learn a static reward function, but is it possible the issue can instead be framed as one of partial observability?
- For example, if these baselines were provided with features capturing the timestep or history, how would they perform? To capture the history a map of normalized state-visitation counts could be provided as a feature.
- If addressing partial observability (of the timestep or state-history) addresses the issue, why should we prefer learning time-varying rewards?
- Another way to potentially answer this question would be to train a (recurrent) policy via behavior cloning, and then compute likelihood on the validation set.
- Providing an additional experiment on a different real-world dataset would improve the paper by illustrating the generality of the approach


[1] Modeling Interaction via the Principle of Maximum Causal Entropy

[2] A Primer on Maximum Causal Entropy Inverse Reinforcement Learning

[3] Dealing with multiple experts and non-stationarity in inverse reinforcement learning: an application to real-life problems

[4] Hidden-Mode Markov Decision Processes for Nonstationary Sequential Decision Making

[5] Inverse Reinforcement Learning in Partially Observable Environments

[6] Intent-aware long-term prediction of pedestrian motion

[7] Learning How Pedestrians Navigate: A Deep Inverse Reinforcement Learning Approach
questions: The section above includes a number of questions. In addition to those:
1. What are the convergence properties of the algorithm (specifically the dual ascent of the features / weights)? If you run it multiple times, how much do the learned features differ?

And some minor questions:
1. Section 3.1 introduces MDPs, but it’s not clear to me whether the paper deals with finite horizon MDPs (as would be indicated by the use of a timestep-dependent reward function and Q-function) or infinite horizon MDPs (as would be indicated by the use of a discount factor)
2. In equation 4, should the q-function on the rhs have time index t + 1?

limitations: Thank you for including a section discussing the limitations of the approach
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: The paper proposed a new method, Dynamic IRL, and applied it to model mouse behaviour in a spatial navigation task to recover the goals of the animals. Unlike the previous IRL methods, the Dynamic IRL allows the reward to change over time slowly. To make the inference trackable, the authors model the reward function as a linear combination of weights and a few "goal maps". The authors demonstrated and validated the model on a simulated gridworld task, and apply it to a real mouse dataset to successfully recover the goals of the animals in two different conditions. 
strengths_and_weaknesses: Strengths:
- The paper is well-written and easy to follow. The figures are also high quality.
- The contribution is clear, allowing the reward to vary across time in the IRL model.
- The authors performed experiments on both synthetic data and real animal datasets, and compared to an existing IRL model. 

Weaknesses:
(I only list them here, and I will explain more and put my questions and suggestions in the next section)
- The number of goal maps seems to be non-distinguishable to some extent, e.g. in the simulation, the 2-map and 3-map fit the 2-map data equally good.
- When applying it to real data, it requires a procedure to cluster the data first.
- The modelling results seem to mainly confirm the hypothesis from the behaviour and descriptive. 
questions: "Building computational models of decision-making is a core objective in both neuroscience and psychology" - this is a substantial statement and I believe it is not 100% true. It might be true for some computational neuroscientists/psychologists, but the goal of building models is not to build models, it is for understanding the biological system or psychological system. I would suggest authors make this softer, especially when this is the first sentence of the abstract and introduction.

From the paper, it seems that the author separate the data into 80% training data, and 20% testing data (where test LL was computed). When tuning the hyperparameters, the authors did it on a validation set. Did the author use a separate validation set from the 80% training set, not using the 20% testing data? It would be great if the author can make this very clear in the writing. 

Following the training data split, I'm wondering if the non-distinguishable 2-map and 3-map can be due to a lack of statistical power. For example, in the simulation, the authors can try using longer sequences or more sequences, just to test this hypothesis (even though longer or more sequences are not available in the real dataset). Or instead of an 80/20 split, k-fold cross-validation might provide more statistical power. 

If this non-distinguishable is intrinsic not from model recovery noise, I would be happy to see (brief) discussion on future work on addressing this, maybe somehow penalize for model complexity in the model?

The data selection part can seem a bit arbitrary. It makes sense to remove the first 25 or so trajectories, but I'm wondering what else (if any pattern) is removed. I'm also wondering if the results change dramatically if this data selection is not done (maybe only removing the first 25), i.e. is this step necessary when apply the dynamic IRL model? If so, I would expect the model can capture some noise to some extent, e.g. the reason to allow reward function to vary is to capture fatigue. I would like to see a (brief) discussion on future work on a more principal way to capture noise. Maybe a noise map, or allowing the reward function to not just change slowly across time, but also can undergo change point?

It is exciting to see the model is already applied to a real animal dataset. However, the conclusion/implication we can draw from the modelling results seems to only confirm what we can observe from the behavioural data alone, and mostly serve to validate the model's ability to recover something meaningful. I can imagine the recovered weights can be used to correlate with some neural recordings. The authors touched on this a bit by saying the method can provide a normative account, but since this is mainly a methodology paper, it would be great to add a sentence or two, detailing what kind of scientific findings your model can provide in addition to only analysing behavioural data. 






limitations: Another limitation might be, that in lots of behavioural experiments, the state might be hidden/unobservable. I think this is definitely out of the scoop of this paper, that the authors do not need to include/address. But if authors have comments on this, it can be very interesting to read.
review_title: null

final_score: rating: 10: Award quality: Technically flawless paper with groundbreaking impact, with exceptionally strong evaluation, reproducibility, and resources, and no unaddressed ethical considerations.
review_content: summary: This paper describes an application of inverse reinforcement learning to maze exploration by mice. It proposes a novel IRL method to model changes in the reward function over time, by representing the reward as a linear combination of multiple goals, such that the importance of each goal can change over time. This model jointly infers the possible goals, and the dynamic reward function from mouse trajectory. The model can be applied to free exploration, where mice are not given an explicit goal on each trial. In this case it still recovers reasonable goal maps, such as a "home reward" and a random-looking exploration reward map. The model assumes that mice have multiple goals and choose between the set of actions associated with these goals using a Boltzman policy, but makes no other assumptions. 
strengths_and_weaknesses: Strengths: This is very interesting work -- among the most interesting NIPS submissions I have seen over the years, and a great application of IRL to a real problem. I strongly recommend this work to be presented at the conference. Although the explore map tells us little in practice, it is very interesting in theory and takes an important step toward understanding exploration. This is a short review because I think the paper is really great.

Weaknesses: the exploration map leaves some open questions  -- 
can this map emerge from a random walk? 
from self-avoiding random walk? 
are mice intentionally seeking out intersections, or other structural elements that allow them to encode the spatial map?
This is a minor weakness, since it is impossible to address all of these within 9 pages.
questions: 1. Why is there a log in equation (4)? I am not familiar with a formulation of a Q function with a log.

2. Line 141: "coordinate ascent", line 144: "gradient ascent" -- this should have probably been "coordinate descent"?


limitations: I believe this work has minimal negative societal impact -- it is extremely unlikely that IRL can be used to infer human goals. The main limitation of IRL as acknowledged by the authors, is the need to see many instances of a behaviour. This will work well in contexts where the goal set is relatively small and constant through the observation period. However, IRL can easily miss fine-grained variation in goals, such as when animals see exploration as play,  trying out multiple ways to achieve a goal. This will show up as random actions when averaged over trials.
review_title: null

Meta-review: metareview: Unanimous acceptance recommendations.

Reviewer yX2q had a long list of weaknesses, and was the most thorough reviewer, but most have since been addressed by the authors. While prior time-varying IRL prior methods exist, their application their dynamic inverse reinforcement learning (DIRL) method to animal decision-making in more complex tasks from to real animal (mouse) data seems uncommon yet relevant to NeurIPS. There are some concerns about scalability, but for these applications like inferring mouse goals and rewards in maze like environments, scaling isn't as necessary.

From my own reading of this work, I agree with the reviewers it's also interesting, and worth accepting. The authors' modelling of the low-rank time-varying reward structure was nice and non-trivial, and it does give interesting interpretability to a discrete number of mice behaviors it discovers like "explore", "go home", "find water". So the application and method seem interesting and novel. It's possible this could be better suited to a neuroscience conference instead, but the use of IRL here might complicate that, so being between fields, the best venue for a work like this is possibly NeurIPS.

While I recommend accept here, the higher scoring reviewers were not overly specific in the merits of the method to warrant an award, so I interpret their scores as slightly overzealous, and so without significant algorithmic novelty beyond the time-varying modelling in this context (which was nice but not ground breaking), I'll not argue for an award.
confidence: Less certain
Decision: Accept