Question: What is the goal of the paper?
Evidence: Designing reward functions for complex, real-world tasks is challenging. Reward learning lets one instead infer reward functions from data. However, multiple reward functions often fit the data equally well, even in the infinite-data limit. Prior work often considers reward functions to be uniquely recoverable, by imposing additional assumptions on data sources. By contrast, we formally characterise the partial identifiability of popular data sources, including demonstrations and trajectory preferences, under multiple standard sets of assumptions. We analyse the impact of this partial identifiability on downstream tasks such as policy optimisation, including under shifts in environment dynamics. We unify our results in a framework for comparing data sources and downstream tasks by their invariances, with implications for the design and selection of data sources for reward learning.
Inverse reinforcement learning (IRL; is the prototypical example of reward learning. IRL infers a reward function from the behavioural data of a task expert by inverting a model of the expert's planning algorithm. Existing work partially characterises the inherent ambiguity of behaviour for certain planning algorithms and classes of tasks. We extend these results to more planning algorithms and arbitrary time-unbounded, stochastic tasks, using a more expressive space of reward functions that reveals novel ambiguity.
Reward learning models have been proposed for many other data sources. A popular and effective data source is preferences over behavioural trajectories. Unlike for IRL, the ambiguity arising from these data sources has not been formally characterised. We contribute a formal characterisation of the ambiguity for central models of evaluative feedback including trajectory preferences.
Several studies have explored learning from expert behaviour and preferences, or other multi-modal data sources. One motivation is that different data sources may provide complementary reward information, eliminating some ambiguity. Similarly, and observe reduced ambiguity by combining behavioural data across multiple tasks. Our partial order provides a general framework for understanding these results.
Computing an optimal behavioural policy is a primary application of learnt reward functions. proved that potential shaping transformations always preserve the set of optimal policies, and so are always tolerable for this application. We extend this result, characterising the full set of transformations that preserve optimal policies in each task, including for additional policy optimisation techniques such as maximum entropy RL. Ambiguity corresponds to the partial identifiability of the reward function modelled as a latent parameter. The prevailing response to partial identifiability in reward learning has been to impose additional constraints or assumptions until the data identifies the reward function uniquely (or, at least, sufficiently for policy optimisation). Following and, we instead describe ambiguity given various constraints and assumptions. This gives practitioners results appropriate for their real data (and the ambiguity tolerance of their actual application).
IRL is related to dynamic discrete choice, a problem where identifiability has been extensively studied (e.g.,. We study a simpler setting with known tasks. IRL also relates to preference elicitation and inverse optimal control. Preferences over sequential trajectories are not typically considered as a data source in other fields.
Substantial effort has been invested to develop reward learning algorithms for a variety of data sources. A fundamental question to ask is how effective are these algorithms relative to an optimal algorithm for that data source? Our results characterise the information available in different data sources, enabling algorithms to be compared to this theoretical upper bound. Moreover, our framework enables direct comparisons between different data sources. We find that some data sources are strictly less informative than others, such as noiseless preference comparisons vs. return labels. By contrast, others are incomparable and have complementary ambiguity, such as Q-values (invariant to S 1 -redistribution but not potential shaping) vs. episode return G ξ (invariant to some potential shaping, but not to S 1 -redistribution).
In particular, we have characterised the invariances of various reward-related objects to transformations such as potential shaping. We have shown that these objects form a partial order under ambiguity refinement. These results, summarised in figure, allow us to predict the ambiguity of data sources generated from these objects. While practitioners could simply collect data from the least ambiguous source, this might be very expensive. Our framework also identifies the ambiguity tolerance of downstream applications (such as policy optimisation) that need to compute these objects. This enables practitioners to identify reward learning data sources with low ambiguity in the areas their application is sensitive to, enabling higher performance without unnecessary costs.
We analyse the impact of this partial identififiability on downstream tasks such as policy optimisation, including under shifts in environment dynamics. We unify our results in a framework for comparing data sources and downstream tasks by their invariances, with implications for the design and selection of data sources for reward learning.**** We extend this result, characterising the full set of transformations that preserve optimal policies in each task, including for additional policy optimisation techniques such as maximum entropy RL.****Our framework also identifies the ambiguity tolerance of downstream applications (such as policy optimisation) that need to compute these ob jects.
Answer: We are sorry to hear that you found the motivation and goal of the paper unclear. For what it's worth, we can state the goal as follows: *We present a unified framework for studying the fundamental limits of information recoverable about reward functions from different data sources, and the robustness of applications (including policy optimisation in RL) to these limits.*We essentially stand by our opening section, which, occupying just over one half of one page, we have made as concise and accessible as we could manage, while addressing the two core aspects of our analysis (the partial identifiability of reward learning models, and the tolerance of applications) and their duality, leading to the unified framework, which we see as our conceptual contribution.