Question: In Theorem 3.3, the S’-redistribution invariance and potential shaping invariances are not mentioned -- is this because they are less ambiguous than optimality-preserving transformations?
Evidence: Inverse reinforcement learning (IRL) algorithms infer a task's reward function from the behaviour of task experts. Formally, this behaviour is represented as an expert's policy or a sample of trajectories.
To characterise the corresponding invariances, we begin with Q-functions -instrumental to deriving many policies. Q-functions are invariant to S 1 -redistribution since they are defined as an expectation over the successor state S 1 , We show that this is the only invariance for Q-functions. Theorem 3.1. Given an MDP and a policy π, the Q-function for π, Q π , determines R up to S 1redistribution. The optimal Q-function, Q ‹ , has precisely the same invariances.
This invariance is inherited by any object that can be derived from a Q-function. However, note that S 1 -redistribution vanishes in simpler spaces of reward functions, as we explore in appendix C.
We now turn to policies derived using various planning algorithms. These policies are instrumental in constructing the trajectories studied in IRL. For example, and assume that expert behaviour is drawn from a Boltzmann-rational policy, and Ziebart et al. () assume a Maximum Entropy policy We catalogue the invariances of arbitrary Boltzmann policies, of which these other policies are special cases. As these policies can be derived from Q-functions, they inherit invariance to S 1 -redistribution. We show they are also invariant to potential shaping, but not to any other transformations. Theorem 3.2. Given an MDP, an inverse temperature parameter β, and a base policy π 0 , the Boltzmann policy π π0 β determines R up to S 1 -redistribution and potential shaping. The Boltzmannrational policy, π ‹ β , and the Maximum Entropy policy, π β , have precisely the same invariances. By contrast, and assume that experts follow an optimal policy. Optimal policies inherit S 1 -redistribution invariance from the optimal Q-function, and are also known to be invariant to potential shaping. Under an additional assumption that a given policy is maximally supportive, in that it takes all optimal actions with positive probability, we show that these invariances and any additional invariances are captured in a class of optimalitypreserving transformations (Definition 2.5) based on the set of optimal actions in each state. Theorem 3.3. Given an MDP, let Opsq " arg max a A ‹ ps, aq. A maximally supportive optimal policy determines R up to optimality-preserving transformations with O.
Additional invariances arise if we consider optimal policies that may lack support for optimal actions. As a well-known example, the zero-reward is consistent with any policy in this sense.
In the infinite-data limit, a data source of trajectories sampled from a policy reveals the distribution of trajectories induced by the policy, and therefore the policy itself for all states reachable via its supported actions. A Boltzmann policy supports all actions, so in the infinite data limit, samples of trajectories determine the policy for all reachable states. We show this introduces invariance precisely to changes in the reward of unreachable transitions. Theorem 3.4. Given an MDP, an inverse temperature parameter β, and a base policy π 0 , the distribution of trajectories induced by the Boltzmann policy π π0 β from all initial states determines R up to S 1 -redistribution, potential shaping, and a mask of unreachable transitions. The distributions of trajectories induced by the Boltzmann-rational policy, π ‹ β , and the Maximum Entropy policy, π β , from all initial states, have precisely the same invariances.
Similarly, trajectories sampled from an optimal policy reveal the policy in those states that its actions reach. This again introduces additional invariance to transformations of reward in other states. Theorem 3.5. Given an MDP, consider the distribution of trajectories induced by a maximally supportive optimal policy. Let S be the set of states in supported trajectories. Let O be the set of functions O defined on S such that Opsq " arg max a A ‹ ps, aq for all s P S. The induced distribution of trajectories determines R up to optimality-preserving transformations with O P O.
Note that a mask of the complement of S is not permitted. However, the fact that O is unconstrained outside S leaves reward effectively unconstrained in those states, except that the reward of transitions out of S may have to "compensate" for the value of their successor states, to prevent new actions that lead out of S from becoming optimal.
So far we have catalogued the invariances of transformations to the reward function of various reward function derived objects. These invariances characterise the infinite-data ambiguity of several reward learning data sources, and the ambiguity tolerance of policy optimisation. In this section, we discuss the implications for the practical evaluation of reward learning data sources.
We begin by defining a mathematical framework for comparisons between data sources and applications in terms of their ambiguity. The characterisation of ambiguity and tolerance as invariances to reward transformations suggests a natural partial order on data sources and applications. Recall that the invariances of an object correspond to a partition of the space of reward functions (section 2). We lift the refinement relation for partitions to data sources and applications as follows. Definition 4.1 (Ambiguity refinement). Consider two reward learning data sources (or applications), X and Y . Let Π X and Π Y be the partitions of the space of reward functions corresponding to their respective invariances (definition 2.1). If Π X is a partition refinement of Π Y , we write X ĺ Y , and we say X is no more ambiguous than Y (or X is tolerable for application Y ). If X ĺ Y but not Y ĺ X, then we write X ă Y and say X is (strictly) less ambiguous than Y .
Given two data sources X and Y , X ĺ Y corresponds to X conflating no additional reward functions compared to Y in the infinite-data limit. This is the sense in which we say X is no more ambiguous than Y . Moreover, given a downstream application Z, X ĺ Z is precisely the condition of Z tolerating the infinite-data ambiguity of data source X: X ĺ Z if and only if the reward functions conflated by X in the infinite-data limit all lead to the same outcome in Z.
Of our fundamental reward functions transformations, there are several clear instances of ambiguity refinement in a given MDP, as summarised in figure. Invariance to k-initial potential shaping (k-Φ) corresponds to less ambiguity than general potential shaping (Φ). Likewise positive linear scaling (PLS) is less ambiguous than zero-preserving monotonic transformations (ZPMT), and a mask of impossible transitions is less ambiguous than S'-redistribution (S 1 -R). All of these transformations other than zero-preserving monotonic transformations are less ambiguous than the optimality-preserving transformations we have encountered.
More concretely, we can compare the ambiguity of specific data sources. Some of these comparisons are indicated in figure. For example, the ambiguity tolerance of the set of optimal policies is a class of optimality-preserving transformations. Each of the data sources that are less ambiguous than this tolerance are sufficient for policy optimisation.
Notably, this excludes noiseless comparisons between trajectory fragments in some MDPs. Specifically, policy optimisation does not, in general, tolerate zero-preserving monotonic transformations (ZPMT), while noiseless comparisons are invariant to this transformation in some MDPs (Theorem 3.10). Policy optimisation also does not tolerate data sources based on possible and initial trajectories, which are invariant to a mask of unreachable transitions. However, these sources are tolerable if the application only requires optimal behaviour in reachable states.
Moreover, we can compare data sources drawn from one MDP to applications in another MDP, such as under a shift in transition dynamics or initial state distribution. This captures the common sim-to-real setting where learning takes place in a simulated or otherwise restricted environment that differs from the final deployment environment. The simplest transformations to consider are masks of possible or reachable transitions. These are parametrised by transition dynamics. In general, the ambiguity corresponding to a mask of X is less than for a mask of X 1 Ą X . For example, if the new dynamics supports transitions that were previously impossible, then sources with invariance to a mask from the original MDP may not be tolerable for applications in the new MDP.
A similar results holds for S 1 -redistribution, which involves an expectation over MDP dynamics. As an extreme example, we prove that when the transition dynamics are changed for every state and action, S 1 -redistribution under the original dynamics permits an arbitrary Q-function under the new dynamics. Naturally, data sources derived from Q-functions may also be affected by shifts in dynamics. Note that this strong result relies on the formulation of rewards as depending on the successor-state (cf. appendix C). Theorem 4.1. Consider an MDP pS, A, τ, µ 0 , R, γq, a policy π, and alternative transition dynamics τ 1 with τ ps, aq ‰ τ 1 ps, aq for all s P S, a P A. Given a function Q 1 : SˆA Ñ R, there exists a reward function R 1 , produced from R by S 1 -redistribution under τ , such that Q 1 is the Q-function for π under R 1 and τ 1 . Ambiguity refinement is a partial order, and some data sources are indeed incomparable. In consolation, we observe that such incomparable ambiguity is complementary ambiguity, in that by combining the associated data sources, we reduce overall ambiguity about the latent reward. Theorem 4.2. Given data sources X and Y , let pX, Y q denote the combined data source formed from X and Y . If X and Y are incomparable, then pX, Y q ă X and pX, Y q ă Y . This perspective highlights promising directions for the design of reward learning data sources. In particular, this suggests developing reward learning algorithms for mixtures of data sources with complementary ambiguity. Unfortunately, most popular data sources actually appear to have similar kinds of ambiguity given one MDP. However, ambiguity could be reduced by incorporating data from multiple MDPs, along the lines of and.
By contrast, Ng & Russell (2000) and Abbeel & Ng (2004) assume that experts follow an optimal policy. Optimal policies inherit S1-redistribution invariance from the optimal Q-function, and are also known to be invariant to potential shaping (Ng et al., 1999). Under an additional assumption that a given policy is maximally supportive, in that it takes all optimal actions with positive probability, we show that these invariances and any additional invariances are captured in a class of optimality_x0002_preserving transformations (Definition 2.5) based on the set of optimal actions in each state.****So far we have catalogued the invariances of transformations to the reward function of various reward function derived objects. These invariances characterise the infinite-data ambiguity of several reward learning data sources, and the ambiguity tolerance of policy optimisation. In this section, we discuss the implications for the practical evaluation of reward learning data sources. We begin by defining a mathematical framework for comparisons between data sources and applications in terms of their ambiguity. The characterisation of ambiguity and tolerance as invariances to reward transformations suggests a natural partial order on data sources and applications. Recall that the invariances of an object correspond to a partition of the space of reward functions (section 2). We lift the refinement relation for partitions (Aigner, 1979, §I.2.B) to data sources and applications as follows.
Answer: Yes, it is exactly as you suggest, and this is alluded to in the paragraph before theorem 3.3 and made more explicit in section 4.