Title: Do Programmatic Policies Really Generalize Better? A Re-Evaluation Study
Primary area: reinforcement learning
Keywords: programmatic reinforcement learning, program synthesis

Abstract: Programmatic policies are often reported to generalize better than neural policies in sequential decision-making benchmarks. We revisit some of these claims and show that much of the observed gap arises from uncontrolled experimental factors rather than intrinsic representational reasons. Re-evaluating three core benchmarks used in influential papers---TORCS, Karel, and Parking---we find that neural policies, when trained with minor modifications, such as sparse observations and cautious reward functions, can match or exceed the out-of-distribution (OOD) generalization of programmatic policies. We argue that a representation enables OOD generalization if (i) the policy space it induces includes a generalizing policy and (ii) the search algorithm can find it. The neural and programmatic policies in prior work are comparable in OOD generalization because the domain-specific languages used induce policy spaces similar to those of neural networks, and our simple tricks help the gradient search find generalizing solutions. This still leaves open the question of whether programmatic representations can ever generalize better. We argue in favor of a positive answer to this question and present a proof-of-concept experiment on a modified Karel task to support our argument. In this modified Karel task, the agent needs to learn a stack or a queue to generalize OOD. FunSearch can find a Python policy that provably generalizes, while neural networks cannot, as they struggle to learn such data structures. By disentangling representational factors from experimental confounds, we advance our understanding of what makes a representation succeed or fail at OOD generalization. We hope this perspective will guide the design of benchmarks and methods for studying OOD generalization in representation learning for solving sequential decision-making problems.

TL;DR: We revisit claims from the literature that programmatic policies generalize better than neural ones and show that much of the observed gap arises from uncontrolled experimental factors rather than intrinsic representational reasons.