Before revision:  While the policy update is performed with the inferred, potentially dense reward function, the original sparse reward is used to provide a self-supervisory signal for the reward update by serving as an ordering over the observed trajectories . The proposed framework is based on the theory that altering the reward function does not affect the optimal policy of the original MDP as long as certain relations between the altered and the original reward are maintained.
After revision:  While the policy update is performed with the inferred, potentially dense reward function, the original sparse reward is used to provide a self-supervisory signal for reward inference by ranking trajectories that the agent observes, while the policy update is performed with the newly inferred, typically dense reward function. We introduce theory that shows that, under certain conditions, this alteration of the reward function does not affect the optimal policy of the original MDP as long as certain relations between the altered and the original reward are maintained.