Venue: ICLR 2021 Conference

Reviews:

final_score: rating: 5: Marginally below acceptance threshold
review_content: In this paper, the authors describe a variable-timescale prediction model for planning in the context of a deep active inference agent. They show that this agent outperforms a baseline in a scavenging task in a 3D first person environment. They show example rollouts of the baseline and variable-timescale models.

Until reading this paper I wasn't familiar with deep active inference agents, which apparently enable the extension of free energy methods to more complex settings. It seems like an intriguing alternative to deep RL. It's not clear to me whether it has the scaling potential that has been demonstrated for deep RL. Although the experiments reported here are larger scale than experiments with active inference systems prior to Fountas et al. (2020), they seem to be quite simple in comparison to tasks on which deep RL agents excel (e.g. typical Vizdoom and DMLab tasks). I don't mean this as a criticism, but instead as a question mark: this difference in demonstrated scale might simply be due to the massive difference in resources that have been devoted to deep RL agents vs active inference agents. 

As a result, I tried to evaluate the paper on its merits in the context of active inference systems, instead of via comparison with deep RL. As such, I'd be happy with a convincing demonstration that (1) the variable timescale model outperforms a strong time-locked model in the scavenging task and (2) that its performance is due to the selection of useful frames for planning that MCTS then uses effectively.

Figure 3 seems to answer (1) in the affirmative. I'm basically trusting the authors that theirs is a reasonably strong baseline, since I don't have experience training active inference agents or with this scavenging environment. This will be reflected in my confidence score.

However, (2) is not substantiated well by the paper's analysis section. The only evidence for this is in the form of two pairs of example rollouts for the time-locked and variable-time models. Are these cherry picked examples, or are they actually reflective of general trends? I'd be much more comfortable if the authors supplied some aggregate results to substantiate this claim:

"As a result, our agent consistently predicts farther into the future in the absence of any nearby objects, and slows its timescale, predicting at finer temporal rate, when the objects are close." 

I'd really need aggregate results demonstrating that the S-sequences are summarizing long trajectories in a sensible way over a large set over episodes, to feel confident that the system is providing the benefits its purported to. Even better would be to analyze the MCTS search trees to show that the search trajectories over S-seqences have desirable properties.

Perhaps more concerningly, the authors say that "As a result, the STM agent is less prone to get stuck in a sub-optimal state, which was commonly observed in the baseline system, and is more inclined to explore the environment beyond its current position". But, as far as I can tell, there's no evidence in the main paper for this. 

One concrete concern is that the time-locked agent fails simply because it's rollouts aren't long enough to find the rewarding object. Maybe it would be sufficient to simply randomly drop out timesteps from the trajectory to reach the STM-MCTS performance level. The current analyses (and baseline results) don't rule out this hypothesis.

If the authors can provide stronger evidence on these points, I'd be very happy to increase my rating.

review_title: Important problem and interesting agent, but needs more in-depth analysis.

final_score: rating: 4: Ok but not good enough - rejection
review_content: Summary: The authors propose to train a model not on the objective time-scale of a sequence of frames, but on the subjective time-scale dictated by how surprising events are (where surprise here is defined as being above a certain energy free threshold). The trained action-conditioned model learns to slow down time for complex scenes, and fast forward when things are easily predicted.

The overall topic is an important one, most model based methods suffer from accumulating errors.
The introduction is well written and offers a strong motivation for the rest of the paper. I like the explanation in terms of a distinction between objective and subjective perception of time and events.

There are lots and lots of references to work by Friston et al., but I am not sure I see the connection with active inference as being that strong or even necessary.

Regarding the part “Furthermore, for long-term predictions STM systematically performs temporal jumps (skipping intermediary steps), thus providing more informative future predictions and reducing the detrimental effects of one-step prediction error accumulation.” There are two potentially relevant references (Neitz et al NeurIPS  2018), and (Darajaman et al, ICML 2019), as both learn models (not necessarily action-conditioned though) that can skip an adaptive number of steps into the future, with similar consequences (i.e. preventing error accumulation and increasing rollout speed).

P4: “The habitual network acts as a model-free component of the system, learning to map inferred states directly to actions”. Shouldn’t $q(a_t; φ_a)$  be a function of $s$ as well?

P6: The angle heuristic might deserve a more in-depth discussion. What if the agent goes in a circle? Or does a U turn? Or a complex sequence of movements in a maze? 
Since the goal of STMs is (at least in part) to reduce progressively the length of S-sequences (such that they start spanning longer and longer horizons), the actions that bridge two episodic memory need to be summarised in a way that is expressive enough.
Could one think of applying the same STM principle that is already applied to states, to actions as well?

“Importantly, [the] function $f_{\theta_{s}}$ is deterministic and serves only to encode information about preceding…”. Where does $f_{\theta_{s}}$ appear in this context? I can’t see it anywhere between eq. 4 and 6

This AAI environment is not so established. Perhaps the author could add (even in the appendix) a top view of a typical setting that corresponds to one episode? 
Also, it appears that the setting chosen by the authors does not reflect the characteristics that they planned to showcase with their method. Given the presence of one sphere per colour and a relatively small environment, how can we appreciate that the agent learns to do planning over long horizons?
Perhaps an environment with longer horizons would be a more adequate testbed?

I very much liked that the authors showed the additional results in Appendix C, I think they are extremely important for the paper. However, I disagree with the claims made in the text, as it appears that they are not substantiated by the figures they reference.
- “Figure 10: random roll-outs generated by the system. These diverse roll-outs demonstrate that STM is able to: i) make correct action-conditioned predictions, ii) speed up its prediction timescale when objects are far away, iii) slow down the prediction timescale when objects are nearby”
If I interpret Figure 10 correctly, it seems to suggest that the imagined roll-outs are not very consistent, as in almost every single case the color of the spheres changes from green to yellow and vice-versa during the rollout.
- Figure 11: several transitions appear not to be realistic, objects appear out of nowhere instead of smoothly while turning.
- Figure 12: while it is true that the baseline model does not generate any sphere, it appears to me that the physics is significantly more consistent than with STM.

The last sentence in the conclusion seems to suggest that the model is not progressively expanding its horizon the more it trains. How come? 

Overall, my impression is that the paper has a very interesting motivation, but the execution could be significantly improved. The results are not so convincing, due to:
1) figures (10-11-12) that do not soundly corroborate the claims, 
2) evaluation setting that does not allow to really test the claim (i.e. not enough opportunities for increasingly longer horizons as the model improves) 
3) the lack of established baselines and benchmarks
4) the lack of an algorithm box to present the method

Minor:
- Simplify language where possible (utilised -> used; are capable of -> can; etc.)
- Several figures (4,5, 8) are corrupted using preview on a Mac (not sure if it’s just my computer). I could see them correctly by using Chrome to open the PDF.

review_title: Subjective time perception is a nice motivation. Execution could be significantly improved.

final_score: rating: 4: Ok but not good enough - rejection
review_content: Summary:

Most model-based RL algorithms learn dynamics models that predicts the next timestep. However, because of model-bias, frequency of timesteps, and objective timescales, the dynamics models can accumulate errors and limited by timescales. The authors propose subjective-timescale model (STM) that instead of predicting the next timesteps they find the "surprising" subsequences of the trajectories and learn temporal-skipping dynamics models over them. The paper shows the improvement over single-step prediction baselines in a first-person navigation domain.

Pros:

The method aims to address a very important problem in model-based RL. 

The idea of using variational free energy with model-based RL seems novel to me, and has not been widely explored.

The qualitative visualization (figure 4 and 5) provides a nice understanding of what the method is doing as well as what it is capable of in first-person navigation. 

Cons:

-- Methods --

The main sections do not contain sufficient information regarding how the actions are obtained from learned STM models. I find one paragraph in section 3 in sufficient. The experimental sections also do not mention how MCTS and MPC baselines differ. Please clarify. Also, how do you recover low-level action sequences from the aggregated actions after MCTS? I do not find the answer from the paper.

The keyframe selection method requires more justification. It is unclear how using the KL divergence for measuring surprise will improve over model's prediction error studied in previous work.
It would be amazing to provide a theoretical justification for the heurstics toward, e.g., saying something about the end task performance, if possible. How sensitive is it to the KL threshold? Please provide this study.

The action sequence aggregation is domain specific which seems a bit unfair to compare against the baselines which **do not** have access to the same information. There should be more baselines or ablation studies to disentangle the improvement of the method from this domain-specific assumption.

The paper uses different indexing styles which make the method more confusing than it should have been. Please choose one between indexing tau or arithematics on tau.

-- Experiments --

As metioned briefly before, more baselines or ablation will be critical to judge the importance of the proposed model? What about compare against other sliency approaches such as prediction error for memory accumulation? Also, it would be helpful to have more than 1 environment to show the genrality of the approach.

The experimental results do not provide enough information to understand what tasks can be solved and what cannot be solved in Animal AI environment. Can the we provide the success rates and categorized by difficulty levels? These information will be helpful in understanding what STM can and cannot do. Perhaps, having a link to some videos will also help.

Figure 3 shows that the reward is going up; how far can the rward go? It is still increasing. 

In figure 3, how is the reward computed?

In figure 5, is there a groundturth trajectory comparison?


-- Others --

Personally, I would appreciate more background and intuition on each term in the variational free energy formula.

Conclusion:

Again, I believe that this work is addressing a very important question with an interesting idea, but it may require a little bit more work to make the case. I appreciate the authors thinking about this problem, and hope the authors are encouraged to continue their work.
review_title: Interesting problem and idea, but a little more work to be done

Meta-review: This paper uses a free-energy formulation to develop an approach to learning "jumpy" transition models, which predict surprising future states. This transition model is used in combination with MCTS and applied to a scavenging task in the Animal AI Olympics, outperforming two baselines.

While the reviewers praised the importance of the problem tackled, and the novelty of using a free energy approach, there was a general sense amongst the reviewers that the paper wasn't totally clear (especially for an RL audience). R1 also felt that some of the claims of the paper weren't sufficiently evaluated enough, and several reviewers indicated that they felt the baselines were insufficient (or, at a minimum, not described in enough detail to evaluate whether they were sufficient). Given these points, I feel the paper is not quite ready for publication at ICLR. I encourage the authors to flesh out their analysis a bit more, better describe the baselines (and possibly compare to other existing approaches as mentioned by R4), and overall to frame the paper a bit more for the RL community.

One additional reference the authors may be interested in: Gregor et al (2018). Temporal difference variational auto-encoder.
Decision: Reject