Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: This paper proposes a new architecture that is useful for generalization to new tasks, that may be previous combination of prior tasks. It adapts from existing literature on successor features and GPI, and shows that a modified architecture that predicts useful representations while also learning representations can be significant in terms of generalization. This work is an adaptation of recent line of works on modular architectures to achieve OOD generalization. 

Overall, there seems to broad claims that the proposed architecture learns compositional aspects of the tasks that helps with generalization to out of distribution tasks. 

strength_and_weaknesses: 
	- The main idea of this paper is to propose a new architecture that can achieve modularization, leading to better generalization. The paper achieves this by learning useful representations from such a modular architecture, in experiments with procedurally generated environments. 
	- The paper considers a train test generalization setting where the agent is trained on a seqeuence of tasks to then test for generalization to new tasks. 
	- The key idea is to use attention mechanism in the resulting architecture that can capture different aspects of the task. Given observations, and the encoded reprsentation Z, I want to understand more why these separate modules are actually useful? 
	- I understand the intuition behind the work, but how these modules are actually capturing what the paper claims to achieve is not clear? What are the different components of Z_t that each of the module is trying to capture? Why would simply using an attention module be useful here? I am not sure how the intuitions are justified from this architecture?
	- The paper claims to achieve disentangled cumulants? How do you justify that? Is there a simple example demonstrating this?
	- Experimental results compare to existing baselines such as UVFA. However, there are other architectural modular networks such as RIMS (Goyal et al) that also proposed such architectures? Why are these baselines architectural changes not considered?
	- My biggest worry of this type of work is the claims the authors try to make, but the results are only in terms of returns or some existing evaluation metrics, which does not justify the claims? The paper provides lots of intuitions of why the architectural bias is important, in terms of disentangling cumulants in the context of successor features; where the architecture captures different aspects of the tasks that helps for generalization. However, empirical results do not fully support this, neither are there theoretical claims for it.
	- Why is the the term compositional exactly used here? How is this term justified? I think the paper makes a broad claim but fails to fully backup results for it

clarity,_quality,_novelty_and_reproducibility: The paper is well written with lots of intuitions provided for why the proposed architectural bias and modularity is important; however, there seem to be broad claims that are not well justified. The title is rather misleading, and it is not clear to me how compositional aspects of the task is actually learnt to then help for generalization? The architecture consists of attention modules, which have been studied previously as well; and simply the addition of this does not well justify the broad claims in my opinion.

summary_of_the_review: I would argue for a rejection of this work mainly because of the broad claims in terms of "disentanglement" and "compositionality" which are not well justified from the results. The paper proposes a simple archiectural change, variations of which has been proposed in the past well. As such, the proposed work is not fully novel, yet there are broad claims which makes the paper look really fancy!
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: The paper studies the problem of task generalization through the use of the successor features framework. It proposes using a modular architecture with the universal successor features method (USFA; introduced in prior work). The modular architecture enforces disentanglement by conditioning each module's predictions on only a subset of the other modules. The experiments show that such a modular architecture is indeed useful in improved task generalization when combined with a USFA algorithm variant which learns the cumulants.
strength_and_weaknesses: Strengths

- Well motivated problem
- Simple proposed method
- Succinct results

Weaknesses

- Parts of writing can be improved
- Experiments are performed on a limited number of (simple/toy-like) domains 
clarity,_quality,_novelty_and_reproducibility: The paper is mostly clear to read, has high quality, is fairly novel, and is decently reproducible.
summary_of_the_review: I really like how succinctly the paper conveys the main message (precisely Figure 5 and Figure 6 are enough to convince the efficacy of the proposed modular architecture). The paper is written well (barring a few confusions I have below around some of the definitions) and is simple to understand for the most part. Overall, I think it makes a useful contribution, with well-constructed experiments and clear writing. Hence, I vote for acceptance. 

In Figure 3, $z_t$ is the output of the encoder, which, going by the text, should be denoted by $s_t$?
Why is $x_t$ provided as input in Eq 4? Shouldn’t $s_t$ already encode that information?
A_theta is defined as a function of $s^k_t$ and $S_{t-1}$, but wouldn’t $S_{t-1}$ already include $s^k_t$, given how it’s defined in the first line of Section 4.2?
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: Authors tackled the problem of combining knowledge from multiple tasks. They introduced Modular Successor Feature Approximator (MSFA) as a new network architecture that both identifies what is important to predict and also learns the corresponding representation. From another lens MSFA removes the need to hand-design cumulants (state-dependent features that capture the return linearly) for the Universal Successor Feature Approximator work by Borsa et al. [ICLR-2019].

strength_and_weaknesses: Strengths
+ Paper is well-written with solid contribution in the space
+ Generalization across tasks is a critical step towards practical reinforcement learning. Hence it is of great interest to the community
+ Ablation studies are thorough showing the impact of modularity and generalization even in heterogeneous tasks

Weaknesses
- Reproducibility: code or algorithm were not discussed in the paper. While few parameters were covered, it is unlikely for someone to be able to reproduce the results.
- The empirical results are done in toy domains. What are the main limits to apply MSFA to larger and more realistic tasks?
clarity,_quality,_novelty_and_reproducibility: Paper is well-written and has high-quality. While the approach is essentially the mix of two existing research, it has good amount of contribution built in. The main drawback of the paper is the reproducibility as details of the experimentation are not present.

Minor:
- monolothic => Did you mean monolithic?
summary_of_the_review: Good submission with yellow flag on reproducibility.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: In this paper authors propose a method to generalize agents better by combining knowledge from multiple tasks. While the agent should generalize on a single given task, it should be able to do so across multiple ones. The proposed method, MSFA, aims at solving this problem.
strength_and_weaknesses: The idea behind MSFA which is to learn disentangled cumulants and successor features is very interesting and promising. Overall, the paper is theoretically and practically reasonable. One thing that is worth studying is how modules in the MSFA architecture are correlated with one another.

In the experiments section, it is not mentioned how many seeds were used to obtain the provided results. Clarification regarding that would be useful. On a similar note, according to results provided, 4 runs were used. In order to better validate the generalizability and robustness of the MSFA, running more tests are required.
clarity,_quality,_novelty_and_reproducibility: The paper is written clearly, and is easy to follow. Authors did a good job in explaining their motives, method, and results. Regarding the reproducibility, nothing has been mentioned about publicly sharing the code.
summary_of_the_review: See above.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: The paper presents a new approach for task generalization using  successor features.A modular architecture is used with the universal successor features, where the modular architecture enforces disentanglement of the feature vectors. The experiments show an improved task generalization.

Strength: The paper presents a well motivated idea for task generalization which is also intuitive. The results are presented very well and convey the improved task generalization abilities. All reviewers appreciated that the paper contains interesting ideas. On the downside, concerns were raised about the complexity and number of experimental setups where the approach has been evaluated. Yet, these concerns were alleviated mostly in the rebuttal. Moreover, Reviewer kJTC was concerned about several claims in the paper that they are not backed up by empirical results. I think these concerns were also addressed well in the rebuttal. Unfortunately reviewer kJTC chose to not join the discussion and did not update his score. However, due to his missing response I give his review less weight.

justification_for_why_not_higher_score: More exhaustive experiments would be needed for a spotlight presentation.
justification_for_why_not_lower_score: The main concern from the most negative reviewer was well addressed in the rebuttal. Unfortunately, the corresponding reviewer did not react to the rebuttal and left his score unchanged.
Decision: Accept: poster