Venue: ICLR 2022 Conference

Reviews:

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: this work studies policy gradient methods for domain randomization (DR). in particular, it investigates baselines for policy gradient under the dr settings such that the gradient estimate can have a lower variance to ensure better policy updates and learning. this paper derives optimal state/environment-dependent baseline theoretically, gives general recipes for building the baseline, and proposes an algorithm called variance reduced domain randomization (VRDR). vrdr is evaluated on several continuous control tasks and it performs better compared with two baselines.
main_review: strengths
1. this work has a good structure, is well-written, and I feel pleased to read this paper
2. this work is well-motivated and I believe the topic is highly-significant and related to the venue
3. this work has a good balance of theoretical contribution and empirical evaluation. it is technically novel and the proposed algorithm works well in practice compared to baselines

weaknesses/questions
1. this work targets the variance of the gradient estimates. so why not directly measure the variance of the gradient estimates? e.g. empirically? I think this is the most straightforward way.
2. I agree lower variance gradient estimates could converge fast, but in general, it's also easier to converge to local optimal. what are your opinions on this?
4. for sec.5.2, could you elaborate more on why VRDR generalizes better to unseen environments? I don't find it intuitive to understand why lower-variance gradient estimates find a solution that generalizes better (as opposed to sgd in supervised learning)? what's the connection or insight?
5. what's MRPO at the end of sec.5.1? probably a typo?
6. could you elaborate on the added computational cost compared to the baselines?
summary_of_the_review: I enjoy reading the paper and I find it theoretically novel and sound. However, I do not find the experiments very straightforward, I believe it's necessary to compare the empirical variance of the gradient estimates with the baselines.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: The paper derived an optimal state/environment-dependent baseline and a variance reduced domain randomization approach for policy gradient methods. 
main_review: The paper is well written and the mathematical derivation is sound. The idea of extending the state values as baselines to the additional parameterization of the environment variations is natural. Because the variance reduction technique in standard single-environment training is well-studied, the extension to the additional parameterization is relatively straightforward (the technical details are well-presented). 

My main concerns are about the experimental results: I think the analysis is too weak and not enough baselines are compared with. By looking at Figure 1, it is quite hard to conclude that the proposed methods really help (whereas the standard variance reduction does make a crucial difference in training). I hope the authors delve more into the results and in particular elaborate on possible reasons when the benefits are unclear, rather than just a simple paragraph on a few environments where the method is marginally better (and I would guess there can be some examples where it is worse?). I do not believe these are all the baselines that should be considered, for instance there's no comparison with meta learning or robust RL methods. 
summary_of_the_review: The paper is well written and the mathematical derivation is sound. My main concerns are about the experimental results: I think the analysis is too weak and not enough baselines are compared with.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This paper tackles the high variance problem caused by the randomization of environments for estimating the policy gradients. The idea is to derive a bias-free and state/environment-dependent optimal baseline for domain randomization. The authors further develop a VRDR method by dividing the entire environment space into subspaces and estimating the state/subspace-dependent baseline.
main_review: This manuscript is a re-derivation of the control variate given that the randomness in the environment is partially artificial (domain randomization). The control variate induces a baseline that depends not only on the state but also on the environment. This term unsurprisingly has some traceability issue in practice and then the authors provide a divide-and-conquer idea to partially address it. Experiments show that this method is marginally better than vanilla domain randomization.

Pros:
1. The paper tackles an interesting problem: how to reduce the variance and improve sample efficiency during the training of DR by developing a better baseline.
2. The paper is clearly written in general.

Discussion:
1. Theorem 1 is not useful. The term can be both positive and negative. I don't think control variates in RL has guarantees in variance reduction so I would suggest removing the theorem.
2. The proposed method is based on the premise that the environment parameter is known and used to calculate the clustering prototypes during the training, which may not be valid for the real setting where the agent only can observe the environment.
3. The convergence improvement in Figure 1 is not significant for some tasks like Pendulum and Pendulum2D yet with the additional cost of the clustering process.
4. The paper uses the hierarchical cluster method to partition the environment space. Does the cluster method influence the experimental results? Why are other cluster methods like k-means not considered? And choosing a proper clustering interval seems nontrivial with grid search since the values in Figure 4 fall into different ranges for different tasks.
5. The parameter Nc is used without definition in Alg 1.
6. In section 5.2, the authors learn 15 policies for each algorithm on testing environments to test the generalization to the unseen environment. I guess that the authors wanted to express that they use the trained policies on testing environments directly.
7. The paper uses the uniform domain randomization as a baseline, which is not the proposed method in [Mehta et al., 2020] (cited by the paper). Why does the paper not compare to the active domain randomization algorithm developed in [Mehta et al., 2020]?

summary_of_the_review: This is a marginal improvement of domain randomization with marginal improvement in experiments.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This paper tackles the variance of policy gradient due to the domain randomization used in RL in simulations. The authors prove that the policy gradient variance can be further reduced by learning a state-dependent baseline for each environment parameter compared to state-dependent baselines. The authors then develop a practical algorithm based on the analysis and analyze the properties of the algorithm. The algorithm is implemented and tested on six robot control tasks. It consistently accelerates policy training. 
main_review: ### Strengths ###
- It’s great to see that the proposed algorithm demonstrates good generalizability compared to other methods in Figure 2. I have a question about a plot though. In Figure 2.(a), the most left subplot, the DR’s score is no higher than 1400. However, in Figure 1 (a), the mean of the DR curve is around 3000. Did I interpret the plots correctly?
- Domain randomization is commonly used in practice to train good policies using simulators. The problem that the submission tries to address is relevant to the community. 
- I appreciate the efforts of deriving and analyzing the practical algorithms based on theoretical development, although it would be better to analyze the variance reduction of the practical baseline proposed. 

### Concerns ###
- The novelty of the submission is limited. As authors noted, the baseline in Section 3.2 is a special case of the input-driven baseline derived by Mao et al. (2018). And it's not discussed how proposed method is different from Liu et al.’ s per-task control variate. 
   - If I understand it correctly, the environment parameters can be treated as part of the state of the MDC. The combined state (original state + environment parameter)-dependent baselines can be learned just as regular state-dependent baselines based on function approximation without additional techniques, such as the subspace clustering. 
  - In Algorithm 1, clusters are formed based on Q values, but the nearest neighbors are found based on environment parameter. This inconsistency is a bit odd to me. 

- There may be a bug in the theoretical analysis. I believe the expectation of environment parameters need to be conditioned on s, just as the action distribution is conditioned on s (policy \pi). This includes E_P in Eq. (4), corollary 1, and Theorem 2. 
- The clarity of the paper can be improved. 
  - The math notations are sometimes confusing and math arguments are not always precise. Examples:
      - On the fifth line after Eq. (1), the meaning of equation E_{P, \mu, \pi}[g] = .. := E[g], and the meaning of g in Eq. (9) are not very clear. 
      - The symbol “|” in probability represents conditioning. However, it seems to represent parameterized functions, for example,  \eta(\pi |p), b(s|p_i), b^*(s|P_j)
     - In Theorem 2, "i.f.f.", the condition (Eq. (11)) seems to be sufficient but not necessary.
     - “can obtain the minimum variance for DR” right above Section 4. I think Corollary 1 shows that state/environment-dependent baseline reduces more variance than state-dependent baseline, but I don’t think this implies state/environment-dependent baseline can obtain the minimum variance. 
  - The algorithm requires more explanation, such as, how the baselines are updated in Line 14, how the hierarchical cluster method works. And more description about relabel may help. 

### Other comments ###
- Multiple symbols are randomly written in Italic or non-italic font, p in the second before the last line on Page 2 and a in the fifth before the last line in Section 2.
-The meaning of N_C and N in Line 3, Algorithm 1. 
- What does $\kappa = (\kappa > 1)$ represent, right above Section 5?
-  {p_c}_{c=1}^M  on the fourth line after Algorithm 1 box.  Should it be {p_c}_{c=0}^{H-1} instead?
- hve -> have in the middle of page 9.

### After rebuttal ###
I really appreciate that the authors incorporated the comments in such a short amount of time. Although I think the clustering idea and experiments are interesting, due to the amount of changes made, I am slightly leaning toward suggesting the authors take some more time improving the manuscript and submit it to future conferences. For example, I think the experiment section can be improved with the new experiments (and discussions on the results) and more analysis on the proposed algorithm (e.g. how b* in the second paragraph on Page 5 is related to the algorithm proposed (we can not sample from (P(p|s)) and quantify the correlation between G(a, s) and Q(s, a, p)). 

summary_of_the_review: Because my concerns outweigh the strengths now, I am leaning towards rejecting the paper.
review_title: null

Meta-review: While the reviewers appreciated the clarity of the work, there is a concern about the meaning of the proposed result and method. It is known that adding knowledge about an additional variable, in this case the environment, leads to a lower variance estimate. What is not known is the practical impact of using this new baseline or perhaps some other intuition stemming from that use of the baseline (for instance the origin of the variance). However, the results shown are not that compelling, a point which was raised by the reviewers, making the work below the bar for publication.
Decision: Reject