text: 475,89,1337255175996596227,3013822602,Eric Michaud,"Excited to share my new paper “Understanding Learned Reward Functions” with co-authors @ARGleave and Stuart Russell, presented at the Deep RL Workshop at #NeurIPS2020 Paper: <LINK> Code: <LINK> Presentation: <LINK> <LINK> How can you tell if a learned reward function captures user preferences? We apply some standard ML interpretability techniques towards understanding what learned reward functions are doing in a few RL environments. For instance, in this gridworld environment, where the agent (blue) tries to get to the goal (green), we find that our learned reward function simply detects whether a goal block is visible, not whether the agent has reached it. <LINK> In Atari environments like Seaquest, our learned reward function here seems to pay the most attention to the game's score display (areas highlighted in green are most salient). To reliably predict reward, the model can simply learn to detect when the score changes. <LINK> This is a good reminder that when benchmarking reward learning algorithms on games, the score should not be displayed in the environment -- it can make the task of predicting reward too easy. Why does any of this matter? Well, for many real-world tasks, it is not possible to manually design a good reward function for an RL agent -- human desires, which the agent is tasked with realizing, are just too complicated. Reward functions must instead be *learned*. However, current algorithms for reward learning can fail silently. Absent perfect reward learning, we therefore need techniques for auditing learned reward functions -- for scrutinizing a machine's understanding of human preferences. Our paper is a tentative step in this direction. We hope that more advanced interpretability techniques will someday allow researchers to more comprehensively open up AI systems and verify that such systems understand and are aligned with human values. As a closing thought, I also wonder whether future interpretability techniques, coupled with sophisticated reward learning, could be a kind of ""microscope AI"" for improving our understanding of human values and human well-being. @ch402 @nickcammarata @SamHarrisOrg This work was done during my internship with @CHAI_Berkeley. Many thanks to everyone at CHAI for your support!",http://arxiv.org/abs/2012.05862,"In many real-world tasks, it is not possible to procedurally specify an RL agent's reward function. In such cases, a reward function must instead be learned from interacting with and observing humans. However, current techniques for reward learning may fail to produce reward functions which accurately reflect user preferences. Absent significant advances in reward learning, it is thus important to be able to audit learned reward functions to verify whether they truly capture user preferences. In this paper, we investigate techniques for interpreting learned reward functions. In particular, we apply saliency methods to identify failure modes and predict the robustness of reward functions. We find that learned reward functions often implement surprising algorithms that rely on contingent aspects of the environment. We also discover that existing interpretability techniques often attend to irrelevant changes in reward output, suggesting that reward interpretability may need significantly different methods from policy interpretability. ",Understanding Learned Reward Functions,10,"['Excited to share my new paper “Understanding Learned Reward Functions” with co-authors @ARGleave and Stuart Russell, presented at the Deep RL Workshop at #NeurIPS2020\n\nPaper: <LINK>\nCode: <LINK>\nPresentation: <LINK> <LINK>', 'How can you tell if a learned reward function captures user preferences? We apply some standard ML interpretability techniques towards understanding what learned reward functions are doing in a few RL environments.', 'For instance, in this gridworld environment, where the agent (blue) tries to get to the goal (green), we find that our learned reward function simply detects whether a goal block is visible, not whether the agent has reached it. https://t.co/NyiF5NnAwy', ""In Atari environments like Seaquest, our learned reward function here seems to pay the most attention to the game's score display (areas highlighted in green are most salient). To reliably predict reward, the model can simply learn to detect when the score changes. https://t.co/gAYDtRgTUE"", 'This is a good reminder that when benchmarking reward learning algorithms on games, the score should not be displayed in the environment -- it can make the task of predicting reward too easy.', 'Why does any of this matter? Well, for many real-world tasks, it is not possible to manually design a good reward function for an RL agent -- human desires, which the agent is tasked with realizing, are just too complicated. Reward functions must instead be *learned*.', ""However, current algorithms for reward learning can fail silently. Absent perfect reward learning, we therefore need techniques for auditing learned reward functions -- for scrutinizing a machine's understanding of human preferences."", 'Our paper is a tentative step in this direction. We hope that more advanced interpretability techniques will someday allow researchers to more comprehensively open up AI systems and verify that such systems understand and are aligned with human values.', 'As a closing thought, I also wonder whether future interpretability techniques, coupled with sophisticated reward learning, could be a kind of ""microscope AI"" for improving our understanding of human values and human well-being. @ch402 @nickcammarata @SamHarrisOrg', 'This work was done during my internship with @CHAI_Berkeley. Many thanks to everyone at CHAI for your support!']",20,12,2269