text: 449,127,1446518686152593409,335130183,Minqi Jiang,"🏎️ Replay-Guided Adversarial Environment Design Prioritized Level Replay (PLR) is secretly a form of unsupervised environment design. This leads to new theory improving PLR + impressive zero-shot transfer, like driving the Nürburgring Grand Prix. paper: <LINK> <LINK> Like living organisms, RL agents are shaped by their environment. How can we improve RL agents by designing the environment instead of the agent? We show that random search, as done by PLR, is surprisingly effective for designing useful environments. <LINK> PLR prioritizes randomly-sampled levels with higher learning potential, leading to auto-curricula that improve generalization. We view PLR as a ""curriculum game"" between a student and two teachers: a generator choosing random levels and a curator selecting for learning potential. <LINK> This theoretically unifies PLR with other unsupervised environment design (UED) methods like PAIRED, resulting in a version called Robust PLR (PLR⊥), and a replay-based version of PAIRED, called REPAIRED—each provably resulting in a minimax regret policy at equilibrium. <LINK> The modification to robustify PLR is counterintuitive: In addition to replacing the value loss priority scores with a regret-approximating score, we only train the agent on trajectories from replayed levels. That is, we improve our agents by training on *less* data. <LINK> We test the generalization of agents trained via our “replay-guided” methods to those trained via replay-free baselines in a maze domain, and show improved zero-shot transfer to challenging human-designed mazes. <LINK> But our agents quickly grew bored of mazes, so we took them to the speedway. We trained them on racetracks dynamically generated by each method, and test zero-shot transfer to twenty Formula 1 tracks, where Robust PLR agents take home the trophy. <LINK> Despite its effectiveness, level replay is only half of the game. More sophisticated versions of the generator, whose levels the curator (PLR) selects for replay, should enable even further gains. Stay tuned for some developments in this direction. I owe much of these exciting results to the contributions of my collaborators, who braved with me an, at times, obscure road full of twists and turns. Congrats to @MichaelD1729, @jparkerholder, @j_foerst, @egrefen, and @_rockt. You can chat with us about this work @NeurIPSConf.",https://arxiv.org/abs/2110.02439,"Deep reinforcement learning (RL) agents may successfully generalize to new settings if trained on an appropriately diverse set of environment and task configurations. Unsupervised Environment Design (UED) is a promising self-supervised RL paradigm, wherein the free parameters of an underspecified environment are automatically adapted during training to the agent's capabilities, leading to the emergence of diverse training environments. Here, we cast Prioritized Level Replay (PLR), an empirically successful but theoretically unmotivated method that selectively samples randomly-generated training levels, as UED. We argue that by curating completely random levels, PLR, too, can generate novel and complex levels for effective training. This insight reveals a natural class of UED methods we call Dual Curriculum Design (DCD). Crucially, DCD includes both PLR and a popular UED algorithm, PAIRED, as special cases and inherits similar theoretical guarantees. This connection allows us to develop novel theory for PLR, providing a version with a robustness guarantee at Nash equilibria. Furthermore, our theory suggests a highly counterintuitive improvement to PLR: by stopping the agent from updating its policy on uncurated levels (training on less data), we can improve the convergence to Nash equilibria. Indeed, our experiments confirm that our new method, PLR$^{\perp}$, obtains better results on a suite of out-of-distribution, zero-shot transfer tasks, in addition to demonstrating that PLR$^{\perp}$ improves the performance of PAIRED, from which it inherited its theoretical framework. ",Replay-Guided Adversarial Environment Design,9,"['🏎️ Replay-Guided Adversarial Environment Design\n\nPrioritized Level Replay (PLR) is secretly a form of unsupervised environment design. This leads to new theory improving PLR + impressive zero-shot transfer, like driving the Nürburgring Grand Prix.\n\npaper: <LINK> <LINK>', 'Like living organisms, RL agents are shaped by their environment. How can we improve RL agents by designing the environment instead of the agent? We show that random search, as done by PLR, is surprisingly effective for designing useful environments. https://t.co/LPg360XjRx', 'PLR prioritizes randomly-sampled levels with higher learning potential, leading to auto-curricula that improve generalization. We view PLR as a ""curriculum game"" between a student and two teachers: a generator choosing random levels and a curator selecting for learning potential. https://t.co/w2R08zu3tm', 'This theoretically unifies PLR with other unsupervised environment design (UED) methods like PAIRED, resulting in a version called Robust PLR (PLR⊥), and a replay-based version of PAIRED, called REPAIRED—each provably resulting in a minimax regret policy at equilibrium. https://t.co/JzvBc4T3g5', 'The modification to robustify PLR is counterintuitive: In addition to replacing the value loss priority scores with a regret-approximating score, we only train the agent on trajectories from replayed levels. That is, we improve our agents by training on *less* data. https://t.co/Tr7faNlkWL', 'We test the generalization of agents trained via our “replay-guided” methods to those trained via replay-free  baselines in a maze domain, and show improved zero-shot transfer to challenging human-designed mazes. https://t.co/KyawCnXQeL', 'But our agents quickly grew bored of mazes, so we took them to the speedway. We trained them on racetracks dynamically generated by each method, and test zero-shot transfer to twenty Formula 1 tracks, where Robust PLR agents take home the trophy. https://t.co/w71LoXMBBE', 'Despite its effectiveness, level replay is only half of the game. More sophisticated versions of the generator, whose levels the curator (PLR) selects for replay, should enable even further gains. Stay tuned for some developments in this direction.', 'I owe much of these exciting results to the contributions of my collaborators, who braved with me an, at times, obscure road full of twists and turns. Congrats to @MichaelD1729, @jparkerholder, @j_foerst, @egrefen, and @_rockt. You can chat with us about this work @NeurIPSConf.']",21,10,2366