Exploring and Addressing Reward Confusion in Offline Preference Learning

Venue: NeurIPS_cc_2024_Workshop_BDU
Published: 2024-08-26 08:43:24

Abstract: Spurious correlations in a reward model's training data can prevent Reinforcement Learning from Human Feedback (RLHF) from identifying the desired goal and induce unwanted behaviors. In this work, we study the reward confusion problem in offline RLHF where spurious correlations exist in data. We create a lightweight benchmark to study this problem and propose a method that can reduce reward confusion by leveraging model uncertainty and the transitivity of preferences with active learning.

Main paper: 
 related work: 
 Causal Confusion The problem of causal confusion, which refers to models learning to depend on spurious correlations in the training data, has been studied in behavioral cloning [7] , reinforcement learning [12] , and reward learning [20] . Past work shows empirically and theoretically that spurious correlations and confounders in the training set can worsen an agent's deployment performance [22, 9] .Reward confusion is essentially causal confusion that occurs during reward learning.Goal Misgeneralization While past work on causal confusion studies it as a cause of complete failure to learn goal-directed behavior, it can also make agents optimize for incorrect goals, i.e. goal misgeneralization. For example, in Procgen's CoinRun, the coin to be picked up is always on the right. RL agents can confuse "running to the right" with the real goal of "getting the coin" [10] .Generally, a model's behavior can be consistent with a goal, but it may not be the test-time goal [16] .Preference Learning Learning reward models from preference labels [6] have gained traction due to their low cost compared to expert demonstrations [23] or language inputs [21] . We have also seen progress in other tasks of "reward engineering", e.g. reward hacking [18] . 
method: 
 Models We consider an agent in an environment following the Markov Decision Process (MDP) defined by (S, A, P, R). S is the state space, A is the action space, p : S × S × A → [0, ∞) is the transition probability density. A rollout ξ = (s t , a t ) is a sequence of states and actions. Given unranked rollouts Ξ, our algorithm actively collects ranking information to sort them into an ordered list T = ⟨ξ 1 , ξ 2 , ..., ξ n ⟩. The rank of rollout ξ ∈ T is denoted by ψ T ξ . On each transition, the environment emits a reward R : S × A → R. Our goal is to obtain R * that induces correct policies.Preferences We model the human's probability of preferring ξ 1 in a pair (ξ 1 , ξ 2 ) through the Shepard-Luce choice rule [17, 13] :P [ξ 1 ≻ ξ 2 ] = exp t r(o 1 t ,a 1 t ) exp t r(o 1 t ,a 1 t )+exp t r(o 2 t ,a 2all the rollouts, we have a sorted list of rollouts, ordered according to human preferences. Rollouts can have identical returns, so we treat each element of the chain as a bucket b ∈ B of rollouts with the same return. If the human decides that a new rollout ξ new is equally preferred to ξ m in bucket b m , then ξ new will be added to b m . On the other hand, if ξ new ≻ ξ m and ξ new ≺ ξ m-1 (ξ m-1 resides in a previous bucket b m-1 ), then the algorithm will insert a new bucket containing only ξ new in between b m and b m-1 . This ensures that b 0 contains the best rollouts seen so far and b n contains the least preferred rollouts (where n is the chain length). We illustrate this process in Figure 2 .Our reward model is a Bayesian neural network Algorithm 1 The IMPEC Algorithm [3] which maintains a Gaussian distribution over a network's weights and biases. As we will see, this allows us to incorporate epistemic uncertainty over reward functions into the active selection procedure. In the noiseless case, insertion sort needs O(log n) queries to find the position for ξ new . However, we have access to a partially trained reward network, which we use to guess the rank for ξ new , reducing the number of buckets we must search over. We include more design details in Appendix B for the design of the fast query.Require: Preference dataset D, network θ, query budget Q T ← [] while not converged do θ ← SupervisedTrain(θ, D) if budget not reached then ξ ← arg max ξ I(θ; ψ T ξ | T, ξ) ψ T * ξ ← InsertionSort(ξ, T, θ) T ← T ∪ ξ D ← D ∪ DerivePreferences(ξ, T, ψ T ξ ) end if i ← i + 1 end while (BNN)Information Gain Given an existing chain of rollouts, we use information gain as the acquisition function to decide which rollouts to compare next, so we reduce the most uncertainty over network weights. The information gain over network weights θ by selecting a rollout ξ ∈ Ξ for ranking isEQUATIONwhere ψ T ξ is the rollout's ranking on chain T . Intuitively, it measures how much we expect to reduce uncertainty about the weights after observing the ranking ψ T ξ of rollout ξ. As shown in Appendix C, Equation 1 (information gain) can be approximated by drawing M weight samples,EQUATIONP ψ T ξ | T, θ, ξ is a complicated distribution, and so we (loosely) approximate it with Equation 3.Intuitively, it is proportional to the probability that ξ i ≻ ξ ≻ ξ i+1 .EQUATIONWe summarize the complete process in Algorithm 1. The network is first supervised trained on the preference dataset D using cross entropy loss with P [ξ 1 ≻ ξ 2 ] modeled through the Shepard-Luce choice rule. With the limited query budget for human preferences, we first find out the rollout ξ whose ranking ψ T ξ on chain T will provide the most information gain over the model weights θ. Then we use insertion sort to find out ξ's real ranking ψ T * ξ in the chain. We add the rollout ξ onto the appropriate position of the chain T , then based on its position, derive preference labels with all other rollouts on the chain. We repeat this process until the network weight has converged. 
experiments: 
 Experiment Settings We compare our method with the standard RLHF algorithm, and two other RLHF with active learning methods: pairwise information gain [2] and pairwise volume removal [14] .The information gain (IG) method is similar to ours but reasons only about individual preference pairs and not about the result of the ranking process. The volume removal method was designed in We first perform offline reward learning, and then apply online reinforcement learning using the learned reward function to obtain a policy. The RL agent receives rewards from the learned function instead of the environment, and is trained with Proximal Policy Optimization [15] . More detailed experiment and hyperparameter settings can be found in Appendices D and E.Performance on all six tasks can be found in Table 1 , with each run repeated over five seeds. We further compute the p-values of the results being better than baseline performance, with complete results in Appendix H. Except for the task Go To Door where all algorithms perform poorly, IMPEC has a higher mean return than other algorithms, and often a lower standard deviation.Although IMPEC achieves a higher mean than the other methods on most tasks, its p-values are ≤ 0.1 (per appendix H). This provides some (albeit not particularly strong) statistical evidence that IMPEC has a better mean performance distribution from the baseline. In contrast, the volume removal and information gain methods are often statistically indistinguishable from the baseline.The decisive factor for each algorithm's performance is how often they fail: Out of the 5 seeds, how often does the reward function learn to optimize for the spurious goal? All methods but IMPEC have fairly high probability of taking the spurious feature as the "correct" feature, and hence rewarding incorrect behaviors and obtaining low ground truth returns. We plot the learning curves for each algorithm in the Lava task, where the baseline curve has the largest standard deviations. We observe similar phenomena in other tasks. We include an ablation study in Appendix F. The complete learning curves are included in Appendix J.Limitations and Future Work A limitation of IMPEC is its potential sensitivity to noise in preferences. In our experiments, we keep a relatively low noise level, and we believe more sophisticated algorithms could improve robustness to noise, perhaps inspired by past work on noisy binary search [8, 5] . Our results suggest a deeper connection between the quality of preference datasets and the efficiency of preference learning algorithms. In appendix G, we show some first steps of a graph theoretic analysis for reasoning about preference dataset quality. We are interested in further exploring the influence of graph-theoretic qualities and their effects on preference learning, and using the insights in future algorithm design. 
conclusion: 
 This work studies the reward confusion problem. Our experiments on the Confusing Minigrid benchmark show that reward confusion in offline preference learning can lead to undesired policy behaviors. The benchmark is easy to configure, and we expect it to be particularly useful for iterative research. In addition, we proposed IMPEC to reduce the impact of reward confusion. It exploits preference transitivity and obtains decent empirical performance on tasks with different sources of reward confusion. We believe that the findings of our work will be helpful for making AI more aligned with human values. 
: 
 For many real-world tasks, designing adequate reward functions is challenging, which has led to the rise of Reinforcement Learning from Human Feedback (RLHF) [6] . In this work, we study a failure mode of offline RLHF that we refer to as reward confusion. This occurs when the reward R in a Markov Decision Process (MDP) is a function of features z 1 , . . . , z n inferred from the observation-action pair (o, a). In a simplified scenario, R relies on z 1 but not z 2 , yet z 1 and z 2 are highly correlated in the training data. An empirical risk minimizer might mistakenly conclude that z 2 affects R. As we'll see, this incorrect dependence can lead to failures when training a policy against the learned reward function. We graphically illustrate this problem in Figure 1 .To better understand this phenomenon, we created a benchmark environment called Confusing Minigrid (CMG) that tests reward confusion in models. We carefully designed six tasks with three types of spurious information for the minigrid environment, which we introduce in detail in Appendix A. We will open source the benchmark's code soon.Besides the CMG benchmark, one other our major contributions is an algorithm named Information-Guided Preference Chain (IMPEC) designed to address the reward confusion problem. It involves two stages of training: First, we use information gain as the acquisition function to select comparison rollouts that reduce uncertainty about the reward function. Second, we form a complete preference ordering over the set of selected rollouts, rather than just a partial ordering as in traditional RLHF.Our experiments show that these techniques together improve sample efficiency while reducing reward confusion. We show in Section 4 that using the same comparison budget, IMPEC can outperform many other active preference learning baselines. To the best of our knowledge, it is the first algorithm that attempts to solve the reward confusion problem in preference learning. 
 
 Appendix: Empty Dynamic obstacles Lava Lava-Position Go to Door Fetch Reward learning methods are often evaluated on benchmarks originally developed for reinforcement learning algorithms, like Atari [1] or MuJoCo continuous control [19] . These reinforcement learning benchmarks cannot easily be used to test reward confusion failures. Thus we created six new tasks based on Minigrid [4] , which together form the Confusing Minigrid benchmark. 2 : A summary of Confusing Minigrid tasks. Y/N is short for yes/no. An asterisk means that a feature is off by default but can be optionally turned on. The "extra" column refers for having extra observation dimensions for the task. The position and color columns refer to spurious correlations that come from position and color information, respectively. DistShift means there is a difference between the training and testing environments.Empty Y N* N* N Dynamic Obs. Y N* N* N Lava Y N* N* N Lava-Position N* Y N* Y Go to door N* Y N* Y Fetch N* N* Y Y TableThis set of tasks tests whether spurious correlations with redundant observation dimensions can interfere with learning. In these tasks, an agent needs to navigate to the goal cell. It can observe the state of a glass of water it is holding, which exhibits "ripples" as it moves, and calms down if the agent's position remains unchanged. On rollouts where the agent moves straight to the goal and stops, the level of ripples will be predictive of whether the agent has reached the goal, even though in general it is possible to cause the ripples to disappear by stopping in any location and not just at the goal. The training and testing variants of these tasks are the same.Empty This is the simplest task in the benchmark. The agent needs to navigate to the goal cell, with all other cells being empty. The environment gives a positive reward when the agent reaches the goal, and zero otherwise.Dynamic Obstacles This task augments the Empty task with obstacles that show up randomly in non-goal cells at each time step, which block the agent's path. We include this task because the obstacles may stop the agent at a random grid while it is collecting rollouts, which increases the chance that the reward learning algorithm later realizes that "stabilizing the water" is not the correct goal.Lava The environment contains a row of lava cells with a gap that allows the agent to cross and reach the goal. Standing on the lava cells gives a reward of -1. We use this task to observe if negative rewards will have any impacts on learning spurious correlations, and if the correlation is exploited, whether the agent will at least avoid the lava (low reward) area.We design these tasks with different training and testing variants. The training environments have a 90% probability where the goal configuration is spurious.Lava-Position It is a variant of Lava with changing goal positions. In the training variant, the goal cell is usually located at one particular location. In the test variant, the goal grid can appear at other locations too.Go to Door In this task, an agent is asked to move to a position adjacent to the goal door embedded in one of the four walls surrounding the grid. There are always four doors in the environment, and the goal door is most likely to be placed in the upper wall during training. During testing, the goal door can be placed in any of the four walls.Fetch The agent's goal in this task is to pick up a key. There is usually a distractor object in the environment that an agent can also pick up. In the training variant of this task, most keys are yellow, and most distractor objects are non-yellow. At the test time, the keys and distractor objects can appear in any color with equal chance.For all these tasks, the agent receives a reward of +1 when the goal condition is satisfied. We summarize the task settings in Table 2 . Changing the confounding type in Confusing Minigrid is as easy as modifying a keyword argument when initializing the environment. Training a standard preference learning algorithm on the tasks with the simplest observation type takes around 2.5 hours on a single Nvidia A6000 GPU, which is faster than many other image-based environments or complex control tasks.The network will first update its reward predictions on rollouts in each bucket Rbi , then predict the incoming rollout's reward Rξnew . Instead of using a point estimate Rξnew , we use an interval [ Rξnew -ϵ, Rξnew + ϵ] to fast-guess where ξ new may belong. IMPEC queries human preferences of two pairs (ξ new , ξ l ) and (ξ new , ξ u ). ξ l and ξ u are the rollouts that are closest to the prediction lower bound Rξnew -ϵ and the upper bound Rξnew + ϵ, respectively. The ϵ we use is the standard deviation of Rξnew by M samples of the BNN. After ξ new is added into the chain, we can derive preference relations between it and other rollouts by transitivity.This derivation is adapted from [2] .I(θ; ψ T ξ | T, ξ) = H(θ|T, ξ) -H(θ|ψ T ξ , T, ξ) = -E θ,T,ξ [log P (θ|T, ξ)] + E θ,ψ T ξ ,ξ,T log P (θ|ψ T ξ , T, ξ) = -E θ,ψ T ξ ,ξ,T [log P (θ|T, ξ)] + E θ,ψ T ξ ,ξ,T log P (θ|ψ T ξ , T, ξ) = E θ,ψ T ξ ,ξ,T log P (θ|ψ T ξ , T, ξ) -log P (θ|T, ξ) = E θ,ψ T ξ ,ξ,T log P (ψ Tn τ |T, θ, ξ)P (T, θ, ξ) P (ψ T ξ , T, ξ) -log P (θ, T, ξ) P (T, ξ) = E θ,ψ T ξ ,ξ,T log P (ψ T ξ |T, θ, ξ)P (T, θ, ξ) P (ψ T ξ |T, ξ)P (T, ξ) -log P (θ, T, ξ) P (T, ξ) = E θ,ψ T ξ ,ξ,T log P (ψ T ξ |T, θ, ξ) P (ψ T ξ |T, ξ)Evaluating this expression requires us to compute a conditional probability with respect to θ, which is a random variable capturing our current uncertainty over the reward network weights. We can approximate the distribution p(θ) by sampling M weights θ 1 , θ 2 , . . . , θ M ∼ p(θ) and then treating θ as if it were a uniform mixture over the samples; i.e.p(θ) ≈ 1 M M i=1 δ θ=θi ,where δ θ=θi denotes a Dirac distribution at θ i . Using this approximation, our mutual information becomes:I(θ; ψ T ξ | T, ξ) ≈ E θ,ψ T ξ ,ξ,T log P (ψ T ξ |T, θ, ξ) 1 M M j=1 P (ψ T ξ |T, θ j , ξ) = E θ,ψ T ξ ,ξ,T log M • P (ψ T ξ |T, θ, ξ) M j=1 P (ψ T ξ |T, θ j , ξ) ≈ 1 M M i=1 ψ T ξ P (ψ T ξ |T, θ i , ξ) • log M • P (ψ T ξ |T, θ i , ξ) M j=1 P (ψ T ξ |T, θ j , ξ)Offline Dataset and Tasks In real-world settings where failures are much more costly than minor failures, rollout datasets will be skewed towards higher-return rollouts. We emulate this by constraining the number of rollouts in the low reward region (return ≤ 5) to be at most 10% of the dataset.Query and Data Budgets Preference comparison algorithms typically obtain n pairs of (query, label), [(ξ i1 , ξ i2 ), label i ] n i=1 for binary classification. For simpler tasks (Empty, DynObs, Lava, and Fetch), we set the query budget to be 300. That is, baselines have access to 300 (query, label)pairs, [(ξ i1 , ξ i2 ), label i ] 300 i=1 . IMPEC requires additional queries to precisely rank each sampled rollout within the candidate list, so we constrain it to use 150 pairs [(ξ i1 , ξ i2 ), label i ] 150 i=1 , and use the remaining 150 query budget to perform insertion sort for a selected subset of rollouts (decided by IG).For the harder tasks (Go to Door and Lava-Position), all algorithms are given a budget of 600 (query, label) pairs. IMPEC can access 400 data pairs, and use the remaining 200 query budget for sorting.For each task, we first train an RL agent to the expert level, saving its policies at various timesteps. We then take 3 of its policy snapshots -an almost random policy, an expert policy, and one in-between to generate rollouts. The number of rollouts falling within the low-reward (≤ 5)region are controlled to take up within 10% of the dataset.We typically train an algorithm with 20 epochs. For IMPEC, we evenly divide its query budget from epoch 1 to epoch 15, using up all queries in this period and ranking as much uncertain rollouts as possible. We then use the remaining 5 epochs for learning the full dataset.After sampling the initial preference pairs, IMPEC will not obtain new rollouts from the dataset, and perform learning only by querying humans for ranking the given rollouts.The environment observations are in the vector form, which contains [agent position, agent direction, special grid info]. The special grid can be the goal/lava/door, etc., and its information is an encoding of its grid type, current position, and color.Ablation Studies In the "no active learning" experiment, we turn off the active selection function, and randomly pick rollouts from the candidate list. For "no derived prefs", we remove the preference derivation part of the learning. Note that the preference pairs generated during the sorting process are still added to the dataset. Finally, "no ranking" means that the algorithm still selects preference pairs with an information gain acquisition function, but does not maintain a preference chain (so there are also no transitively derived preferences). This is simply the information gain algorithm of [2] .The preference learning hyperparameters: To understand what leads to IMPEC's performance, we experiment with removing three different components: (1) the active learning process; (2) preference derivations; and (3) the ranking process.The results can be found in Table 3 . The Preference Datasets We visualize the preference datasets gathered by the baseline and IMPECon Lava-Position in Figure 5 . The baseline dataset is randomly sampled at the start of the training, while we take a snapshot of the IMPEC dataset (which is constructed iteratively) at its last training epoch. The datasets are visualized as graphs, with each node being a unique rollout, and each edge representing a preference label. Both IMPEC and the baseline have a query budget of 600 pairwise queries, and IMPEC uses the first 400 of its queries to do pairwise comparisons between randomly selected rollouts, as opposed to actively selected rollouts. The IMPEC and baseline datasets have 405 and 538 unique rollouts as well as 791 and 594 unique edges, respectively. We include several other graph statistics in Table 4 .IMPEC and the baseline exhibit three notable differences in the graph properties. The first is their clustering coefficient, which measures the degree to which nodes tend to cluster together. The IMPEC graph has a higher clustering coefficient because of the many preferences it derives from the ranked chain. This is relevant to the number of chains in the graph: since most nodes are connected to IMPEC's central cluster through some edges, it creates many more chains between two nodes across the graph. In both graphs, we also observe that there are many chains of length 1 that are not connected to any larger cluster. We suspect that the algorithms' sample efficiency can be further improved if these pairs can be meaningfully linked to the clusters.Finally, we measure the graph's efficiency, which is a metric from network science that is intended to measure the flow of information between "communicating" nodes [11] . The assumption which underlies the graph efficiency metric is that more distant nodes are less efficient at information exchange. To avoid inflating the metric with the many length 1 that are not connected to the rest The IMPEC dataset connects more scattered rollouts through its central cluster. of the graph, we compute efficiency only on the two graphs' biggest connected components. We empirically observe that the IMPEC dataset graph has a higher average global efficiency than the baseline graph, which means that the average shortest path between vertex pairs in IMPEC is shorter than for the baseline. This raises an interesting question: Is the assumption in network theory, where the distance of nodes influences information efficiency, also applicable to preference learning? We do not have matured results establishing connections between data connectivity in RLHF and the training performance yet, but it would be an interesting next step for research. We expect that the baseline should suffer less from reward confusion if given more comparison data.To test this hypothesis, fig. 6 shows return for the baseline with different query budgets, along with IMPEC results from our main experiments (which used 150 preference pairs and 300 queries). We tested the baseline with 6× the data used by IMPEC (i.e., 900 preference pairs and queries), then gradually pushed up the amount to 1500 (10× of IMPEC data). With 6× the data, we still see some learning failures, and the seeds' standard deviation decreases to IMPEC's level only when we use 10× data.  



Reviews: review: title: title: Neat idea, but the algorithm is not described precisely.
main_review: main_review: ## Strengths

The authors present an interesting approach to enforcing transitivity of
preferences during reward learning. To the best of my knowledge, this technique
is novel, and will be of interest to a large community. The authors also propose
the use of a Bayesian neural network to rank agent trajectories by information
gain, so as to construct an &ldquo;efficient&rdquo; set of trajectories for preference
learning. I really enjoyed their graph-theoretic analysis of the resulting
preference dataset in the Appendix&#x2014;this would have been nice to include in the
main text.

The authors also introduce a notion of *reward confusion*, which is essentially
describing the difficulty of learning causal relationships in preference
learning. Their hypothesis is that learning the full ordered chain of
preferences can help alleviate this problem. While I am
not totally sold that this should solve the problem, I think it is plausible
enough to warrant further investigation.


## Weaknesses

While I believe I understand the general idea, the text does not provide precise
details about how the IMPEC algorithm works. Notably, the provided pseudocode
references subroutines such as &ldquo;SupervisedTrain&rdquo; and &ldquo;DerivePreferences&rdquo; which
are not described anywhere. While the former is fairly self-explanatory, the
latter is not (to me). More importantly, the variable names used in the
pseudocode do not align with those described in the text, making it difficult to
follow. If I had to implement IMPEC right now, I would be mostly lost.


## Minor Issues and Questions

Typo on line 28, &ldquo;one other our major contributions&rdquo;, should say &ldquo;one of our
other major contributions&rdquo;.

The notation $\xi = (s_t, a_t)$ for a rollout is inaccurate &#x2013; $(s_t, a_t)$ would
generally be interpreted as a tuple, you really want $(s_t, a_t)_{t\ge 0}$.

There is a typo in the &ldquo;type signature&rdquo; of $\mathcal{R}$ on line 57, it says $\mathcal{R}:\mathcal{S}\times\mathcal{A}\to\mathcal{R}$,
where the latter $\mathcal{R}$ should probably be $\mathbb{R}$.
title: title: This paper presents a benchmarking environment for evaluating spurious correlations in Reinforcement Learning from Human Feedback algorithms and introduces an algorithm to address reward confusion. While the authors conducted a thorough evaluation, I am not convinced that the results directly support their claims regarding the algorithm's effectiveness in addressing reward confusion. I've listed my main comment below, I hope the authors find it helpful.
main_review: main_review: 1. The authors propose Information Guided Preference Chain (IMPEC) to address reward confusion. However, the results in Table 1 do not definitively demonstrate its effectiveness. While IMPEC-trained agents outperform other approaches, it's unclear how this directly addresses the challenge of reward confusion. In the Confusing Minigrid (CMG) scenarios, spurious behavior is introduced by adding an extra state. The neural network might be able to ignore this additional state or assign it a lower importance in action determination, especially if it's just one component of the input state vector. A more targeted analysis could involve increasing the amount of spurious information in the CMG environment to rigorously evaluate the robustness of the proposed RLHF approach.

2. I commend the authors for their comprehensive analysis, including ablation studies and graph-theoretical evaluation.