Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This work proposes a new exploration objective in meta-reinforcement learning to discover clusters of related tasks. The claim is that tasks may be heterogeneous, where knowledge may only be transferable to other tasks within the same cluster, so exploring to identify these clusters could find the useful transferable information.
strength_and_weaknesses:  ## Strengths

- The idea to create an exploration objective based on identifying clusters of related tasks is interesting and completely novel.
- The experimental results are strong, although the experimental setup is imperfect, as I comment upon below.
- By and large, the presentation is quite clear and sufficiently technically precise.

## Weaknesses

A major weakness of this work is that the rhetorical claims and explanations are often unsupported, and in several places, clearly false. Many claims appear to be speculation, while they seem to be presented as fact. Below, I include several examples:
- This includes the primary motivation of this work: i.e., that tasks may be heterogeneous, where only some tasks may contain transferable information for other tasks, so it is crucial to identify clusters of transferable information. While identifying these clusters clearly holds some value, it is somewhat misleading, as identifying such clusters is _not_ necessary (i.e., a Bayes-adaptive optimal policy may achieve optimal returns, without any explicit knowledge of such clusters). Claims in the experiments like "such exploration is not effective enough since it is not able to recover underlying clustered task structures" are therefore misleading. Further, such claims are not even supported by the experiments themselves: results in the Appendix, which show that ablating the exploration policy from MILET continues to outperform VariBAD raise questions about what the primary source of empirical improvements is. As MILET outperforms baselines without even exploring to identify clusters, it's unclear if the gains are just from superior neural network architectures or not -- notably the baselines do not use the same architectures as MILET, and sharing the same architecture could help improve the experiments. Experiments on Ant-U without heterogeneous clusters, which are quite interesting, also suggest that it's not about finding transferability between tasks, and clustering can simply act as a useful auxiliary task / exploration objective. Finally, the claim that some tasks contain no transferable information to other tasks does not even appear to be true in the evaluated tasks or provided examples. In the example where clusters might be goals toward the top or the bottom, there are still shared motor skills. To be clear, I think the idea of clustering is still quite interesting, but proposing a story that is not supported is concerning.
- The experiments also speculate about the cause of reduced performance from MetaCURE, but present this as fact: "as in our setting all test tasks are new, the task id becomes meaningless in meta test. MetaCURE thus fails to generalize." MetaCURE does not use the task ID at meta-test time, so this is false.

It should be noted that exploration based on clustering can result in failure modes and is not always applicable. For example, in a distribution of tasks, where tasks can be clearly clustered into task-irrelevant but highly prominent features can lead to unhelpful exploration. In this case, the proposed reward bonuses may mislead exploration from finding actually useful information. Inclusion of such discussion could improve this work. As a related note, the tasks from the experiments do not in general require significant exploration, as much can be inferred from a single timestep reward from relatively random actions (although the exact reward details are not included for many of the tasks). This makes it somewhat challenging to evaluate the proposed exploration bonuses.

## Minor Comments
- Calling Ant-Goal-Sparse a "sparse" reward environment seems misleading if the rewards kick in even from the initial state and are not binary.
- Why are there only 5 colors in Figure 5, if it is MILET-6?
- Is the decomposition of the entropy H(q(c | tau)) into a telescoping series to create per-timestep rewards based on the similar decomposition from DREAM?


clarity,_quality,_novelty_and_reproducibility: Generally, the clarity of this work is quite high. One aspect that is somewhat unclear is how the clusters c are used. From the presentation, MILET appears to never need the ground truth values of the clusters c, and only relies on its own predicted clusterings. However, the submitted code does appear to use the ground-truth values. Perhaps this is only for visualization purposes; I was unable to tell. I would appreciate some author clarification here, though.

The idea of clustering as an exploration objective is quite interesting and original.
summary_of_the_review: Overall, I generally like the proposed method ideas. Exploration via clustering is original and can clearly be helpful. However, the framing and rhetoric of this work is a severe weakness, so I currently cannot recommend acceptance. I look forward to the author response.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: The authors propose an inference based meta-learning approach, similar to Pearl or Varibad, but with a hierarchical latent task embedding modelled as GMM. Additionally, they propose to learn two policies, one optimised for exploration, one for exploitation. The algorithm is evaluated on MuJoCo tasks with variable goal locations which are sampled from clusters. They refer to this as "structured heterogeneity". 
strength_and_weaknesses: # Strenghts

* The paper and problem setting is well motivated.
* The related work section is well written, but could be more extensive.
* The proposed ideas are interesting. 

I personally find especially the way the exploration policy is learned quite interesting. For one, splitting exploration and exploitation policy might be a useful inductive bias in itself as it might provide a simple but effective way of balancing both aspects -- as opposed to trying to do reward engineering to learn one policy fulfilling both roles.
Secondly, the cluster entropy reward might be a very effective proxy reward for exploration success.

# Weaknesses

While I like the algorithm and can believe that it outperforms the baseline on the evaluated tasks, I am unsure about some of the justifications made for design choices and explanations surrounding the experimental results. I will discuss several instances of this below.

## Where does the advantage come from: latent hierarchy or exploration policy?

Unfortunately, the paper does very little to disentangle whether the success is coming from the clustering of the latent space or the separation between exploration and exploitation policy. Instead, it attributes most of its success to the clustering in latent space, claiming that this provides advantages in terms of knowledge sharing between tasks - which I'm not yet convinced by. In particular, I am not sure why VariBAD or Pearl would fail at effectively sharing that knowledge as their formulation seems general enough to do so.
And while MILET provides a hierarchical inductive bias in the latent space, the success of non-hierarchical VAEs has, imo, shown that these types of flat latent spaces are usually flexible enough.  

## Consistency

In section 4.1 the paper discusses a problem with "consistency". However, it is nowhere defined what exactly is meant by consistency. In particular, I am not sure how this can be a contradiction to the update to the agents belief -- which should become more certain over time. 

I assume that the authors mean by "non-consistent" if q(c|tau) 'jumps' from one time-step to the next in a way that the two KLs have very little overlap -- but either way, this should be made precise.  

## Experimental Results

I find it quite surprising that MILET's _exploration_ policy outperforms the other methods by such a large margin -- this seems to me to be more concistent with the hypothesis that the split between exploration and exploitation policy is the real innnovation here and not the GMM in the latent space?

For the sparse reward setup, I am not sure I understand the following sentence:

> With sparse reward, it becomes more important for the agent to leverage knowledge across related tasks, as the feedback within a single task is insufficient for policy learning.

Given that the reward it sparse, how can knowledge sharing help: Before we find the goal, we cannot know which cluster it is in and after we find the goal, this becomes irrelevant as we've already found the goal. Again, imo this could be very well explained by the superior exploration policy and not the latent structure. 
clarity,_quality,_novelty_and_reproducibility: The paper is clearly written and suggests a novel algorithm. 

As outlined above, my primary concern is with the interpretation of the results, which I am not convinced that they are correct.
summary_of_the_review: The paper proposes an interesting algorithm which seems to perform well in practise on tasks with "structured heterogeneity". 
My main concern is with the discussion of the results and the justification of the algorithm. In particular, I am not convinced that the superior performance is due to the latent structure (as they authors claim). Additional discussion and, in particular, ablation studies would hence strenghten the paper.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This work proposes a method, MILET, that leverages an assumption of heterogeneity within the task distribution. More specifically, the task distribution is modelled as a mixture distribution. The objective of MILET essentially adapts VariBAD (Zintgraf et al., 2020), decomposing the belief as a mixture id and a task latent conditioned on the mixture id. The sampling of beliefs for the task decoder is restricted to the inferred posterior mixture id, operationalizing the intuition of eliminating the transfer between unrelated tasks. Two additional regularization terms are also proposed. MILET also separately maintains exploration and exploitation agents, similar to DREAM (Liu et al., 2021). Experiments are run on MuJoCo-based meta-RL domains in which the task variation manifests in the reward function (2 domains) or dynamics (2 domains). Overall, MILET is shown to outperform meta-RL methods from prior works that do not assume a mixture task distribution.
strength_and_weaknesses: Strengths

The authors study a hitherto neglected property of realistic task distributions. The proposed method that leverages this property is sensible and includes new terms that address the specific challenges arising from inferring mixture ids. The empirical evaluation involved modifying existing benchmark tasks to reflect this property.

Weaknesses

The newly proposed meta-RL setting is not clearly distinguished from that of prior work. The authors mention that prior meta-RL assumes tasks are IID, but that seems to need to hold for this work as well. Assuming latent variables underlying the task distribution is well and good, but does the marginal task distribution need to exhibit clusters, or can it be relatively uniform? Put another way, If MILET can improve over prior methods for densely sampled task distributions, then perhaps the real benefit of MILET comes from leveraging the sufficiency of nearby tasks for transfer (a local property), as opposed to heterogeneity (a global property)? The novelty of the proposed method is limited as its backbone is essentially VariBAD (for the evidence lower bound) and MetaCURE/DREAM (for decoupled exploration and exploitation) endowed with the mixture task distribution assumption.

Minor

- I'm not a fan of the cooking analogy in the introduction. There is probably a lot of commonality in cooking Chinese and Indian dishes, which undercuts the point you're trying to make.
clarity,_quality,_novelty_and_reproducibility: Clarity

There are some clarity issues in the scoping of the proposed meta-RL setting, as described above. Otherwise, the writing for the methods and experiments is reasonably clear. The structure of the presentation follows standard norms.

Quality

The execution of the work seems sound. The experiments are commensurate in extent in comparison to recent meta-RL works, though notably there are no experiments with high-dimensional observations (e.g. images). The significance is also somewhat limited by the fact that the proposed method is only assessed on benchmarks of the authors' own creation, rather than one from prior works that exhibits the characteristics motivating this work.

Novelty

The main novelty lies in proposing and studying the specific problem setting. The technical novelty is rather low as explained above. 

Reproducibility

Chances of reproducibility are high since descriptions of the algorithm and environments are provided, as is code.

summary_of_the_review: I currently recommend borderline reject due to the issues of clarity regarding the problem statement, limited technical novelty, and moderate empirical significance.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper introduces a method for explicit task clustering as an inductive bias for Meta-RL algorithms operating on a task distribution. The key assumption is heterogeneity within that task distribution, i.e. that the i.i.d. assumption of draws from the task distribution is unrealistic in real Meta-RL setting and that knowledge can be more readily transferred by explicitly modelling shared structure. To that end, the authors propose a hierarchical probabilistic model, along with an inference procedure that first infers cluster ids and then the task within that cluster. In addition, the method further shows a training procedure distinguishing between an exploratory policy for the first and an exploiting policy for subsequent episodes and introduced various intrinsic rewards designed to lead to stable inference. The method shows promising results, although tasks are of limited complexity and clusters correspond to low-level information (Goal region).
strength_and_weaknesses: `Strengths:`
- Task clustering in Multi-task and Meta-RL is an important yet challenging problem and various attempts at designing such algorithms that make use of clustering have been unsuccessful. The authors correctly identified this as an important and timely problem. 
- Clarity in the algorithmic derivation and motivation of the work (Several minor suggestions for improvement are provided).
- Results on the provided experiments are strong.

`Weaknesses:`
- The provided experiments are relatively simple and hand-designed, cluster information is relatively local and of limited abstraction. At this point I would be unsure whether the technique would be capable of clustering entire tasks from e.g. the MetaWorld task suite based on the more abstract problem being solved (e.g. various pushing/pull tasks, different tasks involving the same object (e.g. buttons, coffee mugs)).
- The method does not handle unknown number of clusters. 

`Requests/Questions:`
- Could the authors kindly provide a cluster visualisation of the clustered transition function experiments? 
- Table 1: At what point do increasing number of clusters reduce performance? 
clarity,_quality,_novelty_and_reproducibility: `Clarity & Quality:`
Overall good although several improvements could be made:
- Figure 1 appears relatively early in the text, its current form makes it difficult to parse. The information flow in the graphic is at first unclear and confusing. The Encoder/Decoder structure could be in a separate figure, as it can be seen more as a detailed view of the policy, rather than a separate component as the figure suggests. The interaction between Environments and Policies should be a circular one, rather than  information flowing only one way (as is shown at the moment). Quantities such as $\mathbf{r}_h$, $\mathbf{r}_c$ are unclear until the very end of the methodology section 4 - a label could possibly help, perhaps there is no need for it to be included. Similarly for the policy hidden states. In the encoder graphic, the authors should be clear and show that a sample from the cluster distribution is passed to estimate the within-cluster distribution (similarly to how sampling is explicit between encoder/decoder).
- Eq (3) and Eq (4) could simply make the notation for samples from the respective distributions clear, as the authors feel the need to explain it in the subsequent text currently.
- Eq (4): Please make the form of $q_{\theta}(c|\tau_{:t})$ clear and state that the Gumbel-softmax trick is being used
- After Equation (4): Could the authors explain why the L2 distance is used. What makes the log-likelihood intractable (as claimed)?
- Figure 2 & 4: Please fix the y-scale for experiments on the same task. Otherwise the relative difference is more difficult to observe. 
- Please number all equations so they can be referred to easily. 



`Novelty:`
- Sufficient

`Reproducibility:`
- Difficult to assess without a explicit attempt, although I am reassured by the fact that authors provide their implementation in the supp. material. 

summary_of_the_review: Overall a good submission with solid methodological improvements, albeit with somewhat limited complexity in its experimental section. In its current stage it is difficult to asses whether the method along with various introduced tricks scales to more complex scenarios, making it difficult to estimate the likely significance of this work in inspiriting future approaches. I'd consider raising my score if the authors improve the work in that regard during the rebuttal. Nevertheless a good submission on a challenging problem that would not be out of place as an ICLR publication.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: The paper proposes Meta reInforcement Learning via Exploratory clusTering (MILET) as a way to improve knowledge transfer across heterogeneous task distributions in meta reinforcement learning. The method consists in augmenting the VariBAD architecture with a more structured graphical model which teases apart task clusters (discrete) from continuous task embeddings. Similar to MetaCure, separate exploration and exploitation policies are learnt, with novel intrinsic rewards encouraging cluster-level exploration. Various auxiliary losses are also introduced to stabilize training. The method is evaluated across multiple Mujoco environments, and seems to outperform the MetaCure, VariBad, PEARL and RL2 baselines. Qualitative and quantitative measures seem to indicate that the method is able to correctly infer task clusters, and ablations (Table 1 and 2) do a decent job of validating the many design choices of the proposed method.

Whereas most reviewers agree that the paper offers an interesting solution to an important problem, two significant issues prevent us from recommending acceptance at this stage.

While the method does appear effective, reviewers and AC agree that the paper contains many unsubstantiated claims, as first highlighted by [XdEV]. A full list of these can be found here [ https://pastebin.com/3nqByswp ]. At a high-level, empirical performance is fully attributed to the agent’s ability to infer cluster (global) vs task (local) information, or their inability to do so. This is particularly problematic since we know that an explicit representation of cluster and task information is not strictly required and could be captured implicitly by the agent’s belief state (RL2) or as in Bayes Adaptive MDPs, via the posterior distribution over model parameters. While it could be argued that these statements could be easily rectified, we felt that the authors were given ample opportunity to do so during the discussion phase.

The second point of contention is related and compounded by the above. Despite the recent addition of the ablations of Table 2 (Appendix E), reviewers and AC do not understand where the gains are coming from, which points to either a lack of clarity or the need for further ablations and targeted experimentation. For example, why does the ELBO of Eq.4 not collapse the posterior over clusters? To our knowledge there is no extra information available either in the high-level posterior or prior over c. What information could the model infer in c which it cannot express via high-dimensional continuous code z? While the ablations of Table 2 do explore this to some extent, they leave out important controls such as a T-GRU only model, with and without consistency regularization, and with/without intrinsic rewards applied to the inferred task label. In particular, the role of \gamma_h and \gamma_c is not ablated. Does the method work without a negative \gamma_c to encourage exploration over clusters early-on? Could this not be applied directly to $z$ without the need for an explicit representation of clusters? I stress here that such conclusions would in no way diminish the novelty of the paper, but rather help clarify the authors’ contribution.

[Outside the scope of this meta-review, but included in the hopes of improving future revisions] While reviewers appreciated that the paper explored task clusters over both reward functions and dynamics, they also felt that these distributions were limited and overly synthetic in nature. During the discussion, [nQz2] in particular suggested the authors consider richer task distributions where clusters are defined in terms of required skills (e.g. pushing, running) and higher-level semantics (e.g. objects with which to interact).
justification_for_why_not_higher_score: Reviewers objected to many unsubstantiated claims in the paper, which they thought would set a dangerous precedent and/or be misleading for future research. To a lesser extent, the paper leaves some unanswered questions as to where empirical gains come from.
Decision: Reject