Venue: ICLR 2020 Conference

Reviews:

final_score: rating: 3: Weak Reject
review_content: # Summary of the paper:

This paper formulates conceptually the unsupervised meta-RL problem (to learn a policy without access to any reward function) as a minimization of the expected regret over tasks, and instantiate an algorithm based on DIAYN (Eysenbach et al., 2018) and MAML Finn et al. (2017a). 

# Brief explanation of my rating:

1. *Novelty*: Mutual information based unsupervised RL was proposed by DIAYN (Eysenbach et al., 2018). Meta-model was also considered by DIAYN (Eysenbach et al., 2018), in which they call it "skill". 
2. *Technical contributions*: Sec 3.1-3.4 try to justify DIAYN. However, the reasoning is not sufficiently rigorous and the proposed Algorithm 1 is inconsistent with the theory built up in these sections. 
3. The *writing* can be improved a lot -- it's not easy to guess what the author was trying to say until I read DIAYN (Eysenbach et al., 2018). 
4. The key ingredient is missing -- the learning procedure f, which was mentioned in eq.(1) and Algorithm 1, but the details are never specified. It is impossible to reproduce the algorithm based on the description in the paper. 
4. The same *experiments* are conducted in DIAYN (Eysenbach et al., 2018). I am still confused on why we suddenly should use meta-RL. 

# Comments:

1. Why we should consider regret? What is the relation between (1) & (4)? It's quite strange you start with (1) but turn to something else, i.e., (4), quickly. 
2. "This policy induces a distribution over terminal states, p(s_T | z)" Why? 
3. What are you optimizing over in (5)?  The statement in Lemma 2 says "I(s_T; z) maximized by a task distribution p(s_g)". However, you are only able to control p(s_T | z), not the marginal distribution p(s_T). The statement of Lemma should be made more clear.
4. The definition of the reward function: r_z(s_T, a_T) = log p(S_T | z), which is independent of the action a_T? 
5. In Algorithm 1, the reward reuse the definition of DIAYN -- log D(z | s),  but which is different from log p(S_T | z). Could you elaborate this? 
6. What is the definition of Markovian reward? Why does the inequality on page 6 hold? 
review_title: Official Blind Review #1

final_score: rating: 1: Reject
review_content: Summary: this paper claims to design an unsupervised meta-learning algorithm that does automatically design a task distribution for the target task. The conceptual idea is to propose a task based on mutual information and to train the optimal meta-learner. They also use experiments to show the effectiveness of the proposed approach. 

Overall Comments:

I would think this paper requires a major revision. It is written in a very confusing way. Many terms are directly used without a definition. The problem is also not clearly defined. I have tried to understand everything, but I have to give up in Section 3. Overall, I do not think this paper is ready for publication.

Detailed comments:
	• It would benefit a lot if you can clearly define the original meta-learning procedure and then compare that with the one proposed in this paper.
	• Define ”hand-specified” distribution. This word does not make sense if you claim this is the difference between the meta-learning procedure proposed in this paper and the original meta-learning algorithm. In this paper, you used p(z) to specify a task. I would think p(z) is also “hand-specified”.
	• I am not very sure by what you mean for “task-proposal procedure”, “goal-proposal procedure”
	• In the first paragraph of the intro: what do you mean by “specifying a task distribution is tedious”, is specifying p(z) also “tedious”
	• 2nd paragraph of intro: “automate the meta-training process by removing the need for hand-designed meta-training tasks”. Again, why p(z) is not “hand-designed”
	• Why compare with the original meta-RL algorithm on p(z) is not fair? 
	• What do you mean by “acquire reinforcement learning procedures”?
	• “Environment”, “task” are not clear when they first appear
	• The word “learn” is used everywhere, and is confusing. E.g. what do you mean by “learn new tasks”, “learn a learning algorithm f”, “learn an optimal policy”, “learn a task distribution” …
	• “Reward functions induced by p(z) and r_z(s,a)”: isn’t r_z(s,a) already a reward function? What is “induced”?
	• What is “meta-training” time?
	• What is “no free lunch theorem”?
	• The “controlled-MDP” setting is actually much easier: perhaps you just need to learn the probability distribution. Then for every r_z, we just solve it. Why not compare with this simple algorithm?
	• “Regret” is not defined when it first appears
	• “The task distribution is defined by a latent variable z and a reward function r_z”: why “distribution” is defined by an r.v.?
	• In (2), “regret” should be the (cost of the algorithm) - (the total cost of an optimal policy) — it is not hitting time
	• (3) is confusing, no derivation is given
	• Based on the usual definition of “regret”, how can a “policy” have low regret? Any fixed “policy” would have linear regret …
review_title: Official Blind Review #3

final_score: rating: 3: Weak Reject
review_content: The paper develops a meta-learning approach for improving sample efficiency of learning different tasks in the same environment. The author formulates the meta goal as minimizing the expected regret under the worst case, which happens when all the tasks are uniformly distributed. The paper introduces two types of tasks: goal-reaching task and a more general trajectory matching task. Then the author introduces a meta-learning algorithm to minimize the regret by learning the reward function under different sampled tasks. The paper is interesting. Below are my questions/concerns.
 
1. Why trajectory matching is considered as more general? Intuitively, trajectory matching is more restricted in that whenever an agent can match the optimal trajectory, it should also reach the goal state. 

2. The theoretical results (lemma 2, 3) actually indicates that the previous work universal value function approximator can optimize the proposed meta learning objective with theoretical convergence guarantee in tabular case by learning the value function Q(s, g, a) where s is a state, g is goal state, a is an action (as long as s and g are visited infinitely often) . As a result, why is it necessary to introduce meta-learning approach? Why not simply learn universal value functions? 

3. The experimental results are not very persuasive. What is the VPG algorithm used? And if you run the algorithm longer, is it finally worse than learning from scratch? Option learning methods/universal value function can be added as baselines. 
review_title: Official Blind Review #2

Meta-review: The paper discusses the relevant topic of unsupervised meta-learning in an RL setting. The topic is an interesting one, but the writing and motivation could be much clearer. I advise the authors to make a few more iterations on the paper taking into account the reviewers' comments and then resubmit to a different venue.
Decision: Reject