Venue: ICLR 2022 Conference

Reviews:

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This work introduces a semi-supervised reward learning approach to reduce the efforts of reward engineering, which contains two key components. The first is to produce artificial labels for these unlabeled samples leveraging the pseudo-labeling and the learned reference predictor. The second component is to crop consecutive subsequences for data augmentation. The proposed approach is tested on Meta-world and DMControl suites, and the results show that it significantly improves the performances.

main_review: Strengths:

s1) The paper proposes a semi-supervised reward learning pipeline aiming to reduce reward engineering efforts, which is an important topic.

s2) Extensive evaluation experiments are conducted to demonstrate the proposed approach.

Weaknesses:

w1) In Fig.1(b), the figure of temporal cropping seems incorrect. According to the description of algorithm 1, the red box should be within the blue box. In the second line, the first image in the red box is outside the blue box.

w2) In Sec 3. “Preference-based reinforcement learning,” “Then, we model a preference predictor using the reward function $\hat{r}_{\psi}$ following the Bradley-Terry model.” It’s not clear to the reviewer how the reward function is learned. “Specifically, given a dataset of preferences D, the reward function is updated by minimizing the binary cross-entropy loss.” Does the reward function have extra constraints? Assume $\psi$ is the reward function that satisfies the Eqn (1). It seems that there are infinite equivalent reward functions ($\psi$ + arbitrary constant number) if the length of segments (H) is the same.   

w3) In Sec 4.1, the same question as the above question 2) w.r.t. the learning process of the reward function through the loss of Eqn(3). If multiple seeds are used, do these reward functions generate similar results on the samples? 

w4) In Sec 4.2, “​​The intuition behind the augmentation is that for a given pair of behavior clips, the human teacher may keep their relative preferences for slightly shifted or resized versions of them.”  The assumption is too strong and does not generally hold. It will actually strengthen the paper if more analyses are provided. What would happen in situations when the assumption does not hold?


summary_of_the_review: This work introduces a semi-supervised reward learning approach to reduce the efforts of reward engineering. However, the technical novelty is limited. Some assumptions made by this work are too strong and may not hold. Some important descriptions (e.g., the reward function w2 and w3 in Main Review) are missing.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper proposes two data-augmentation techniques to improve the query efficiency of preference-based RL. (1) Pseudo-labeling leverages unlabeled data by using high-confidence predictions as labels. (2) Temporal cropping augmentation generates imaginary comparisons by cropping trajectories.

The proposed augmentation can significantly improve the query efficiency of preference-based RL in a variety of benchmark tasks, including DMControl and Meta-World. The performance of SURF is comparable to dense-reward SAC while only accessing a few expert queries.
main_review: This paper aims to tackle an important problem in reinforcement learning, i.e., how to learn a policy without accessing a fine-grained reward function. More specifically, this paper focuses on preference-based RL that uses trajectory-wise comparison queries to guide the learning procedure. The paper is well organized and easy to follow. The proposed method is motivated by the success of data augmentation in the supervised learning area. It is interesting to see a connection between reinforcement learning and supervised learning.

Some questions:

- As shown in figure 6(b), the pseudo-labeling technique requires the hyper-parameter \tau to be a very large value. Why do we need to train our reward model on these sample pairs with extremely high confidence? Please correct me if I misunderstand anything. The loss computed by these high-confidence samples should be very small. Why it can provide a significant improvement on the final performance?

- RAS and GN presented in Figure 6(c) should be orthogonal to Temporal Cropping. I am curious whether we can further improve the performance by applying RAS and/or GN upon SURF.

- The ablation study is only conducted in one specific environment. It would be helpful to repeat these experiments in other tasks.

Minor comments:

- In section 3, regarding the notations of $\mathcal{L}^{CE}$, the terms $y(0)$ and $y(1)$ have not been defined.

- Algorithm 1 (TDA) is presented in page 4, which is far from section 4.2. It might be better to reverse the order of Algorithm 1 and 2 for presentation.

summary_of_the_review: My recommendation score is marginally above the borderline. I like this paper overall but the discussions of ablation study need to be enriched.

The methodology introduced by this paper is inspired by related studies in supervised learning. I think it is quite important to understand why and how these techniques benefit RL algorithms.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: The authors introduce a method for combining semi-supervised learning with preference learning for reinforcement learning. The method uses pseudo labels, derived from a surrogate preference model. Furthermore, a data augmentation scheme for this setting is introduced, which is based on cropped sequences. Evaluation is performed on a range of robotics tasks, including ablation studies for several hyperparameters and the two, distinct parts of the introduced method.
main_review: The authors use a preference-based reinforcement learning method, which is based on a surrogate reward model. The surrogate reward model is learned by phrasing the preference learning problem as classification over pairwise comparisons. The semi-supervised learning approach uses the preference classifier for deriving high-confidence pseudo labels, that can be used as additional, supervised input data.
Both are well known and established principles, making the proposed method a sound contribution. However, as the bridge between classification and PBRL, as well as classification based SSL, are already well known, this is a straight forward application of known methods to a new problem domain, limiting the novelty. The explanation and formalization of this section is good and clear to understand, with one small exception: Two different preference notations are used: (0,1) and >. The first notation is also used for introducing an equivalence relation, which is not used otherwise. This should be unified to the more common > notation (for further preference relation symbols, see https://www.jmlr.org/papers/volume18/16-634/16-634.pdf). An additional, potential improvement is possible by using the tau threshold for defining a confidence cutoff between preference and equivalence relation. However, both are minor issues and do not impact clarity.

Second, the authors introduce a novel data augmentation scheme, by selecting random subsequences out of the observed trajectories, which is also clearly understandable. This method has the implicit assumption, that the reason (states) for the observed preference are still part of the shorter sequence. Especially, in sparse reward domains, this is not a given.

The main evaluation is good, and sufficiently diverse. Advantages of the proposed method are clearly visible. Adding some non-robotics (potentially high dimensional) domains could improve the paper further, but this was also clearly stated by the authors. The given details (including appendix), should be sufficient for recreating the experiment setup.

The ablation study is a welcome addition and the authors covered all relevant aspects, meaning not just the two parts of the introduced method, but also several other factors (and hyperparameters) that can potentially have a substantial impact on the results. However, the ablation study is limited to a single domain, limiting the generality of the derived statements. Especially, i consider the advantage of TDA not sufficiently supported, considering that walker-walk is not a sparse reward domain.
summary_of_the_review: The proposed method is sound and interesting, but of limited novelty. The support for the full method is good, but without an extended ablation study, the support for TDA is not sufficient. The method is likely significant within the area of PBRL, but but not for a broad audience. The paper is clearly written and can be easily understood.
review_title: null

Meta-review: The topic of learning reward functions from preferences and how to do this efficiently is of high interest to the ML/RL community. All reviewers appreciate the suggested technical approach and the thorough evaluations that demonstrate clear improvements. While the technical novelty of the paper is not entirely compelling, all reviewers recommend acceptance of the paper.
Decision: Accept (Poster)