Automatically Composing Representation Transformations as a Means for Generalization

1 INTRODUCTION . This paper seeks to tackle the question of how to build machines that leverage prior experience to solve more complex problems than they have previously encountered . How does a learner represent prior experience ? How does a learner apply what it has learned to solve new problems ? Motivated by these questions , this paper aims to formalize the idea of , as well as to develop an understanding of the machinery for , compositional generalization in problems that exhibit compositional structure . The solutions for such problems can be found by composing in sequence a small set of reusable partial solutions , each of which tackles a subproblem of a larger problem . The central contributions of this paper are to frame the shared structure across multiple tasks in terms of a compositional problem graph , propose compositional generalization as an evaluation scheme to test the degree a learner can apply previously learned knowledge to solve new problems , and introduce the compositional recursive learner , a domain-general framework1 for sequentially composing representation transformations that each solve a subproblem of a larger problem . The key to our approach is recasting the problem of generalization as a problem of learning algorithmic procedures over representation transformations . A solution to a ( sub ) problem is a transformation between its input and output representations , and a solution to a larger problem composes 1https : //github.com/mbchang/crl these subsolutions together . Therefore , representing and leveraging prior problem-solving experience amounts to learning a set of reusable primitive transformations and their means of composition that reflect the structural properties of the problem distribution . This paper introduces the compositional recursive learner ( CRL ) , a framework for learning both these transformations and their composition together with sparse supervision , taking a step beyond other approaches that have assumed either pre-specified transformation or composition rules ( Sec . 5 ) . CRL learns a modular recursive program that iteratively re-represents the input representation into more familiar representations it knows how to compute with . In this framework , a transformation between representations is encapsulated into a computational module , and the overall program is the sequential combination of the inputs and outputs of these modules , whose application are decided by a controller . What sort of training scheme would encourage the spontaneous specialization of the modules around the compositional structure of the problem distribution ? First , exposing the learner to a diverse distribution of compositional problems helps it pattern-match across problems to distill out common functionality that it can capture in its modules for future use . Second , enforcing that each module have only a local view of the global problem encourages task-agnostic functionality that prevents the learner from overfitting to the empirical training distribution ; two ways to do this are to constrain the model class of the modules and to hide the task specification from the modules . Third , training the learner with a curriculum encourages the learner to build off old solutions to solve new problems by re-representing the new problem into one it knows how to solve , rather than learning from scratch . How should the learner learn to use these modules to exploit the compositional structure of the problem distribution ? We can frame the decision of which computation to execute as a reinforcement learning problem in the following manner . The application of a sequence of modules can be likened to the execution trace of the program that CRL automatically constructs , where a computation is the application of a module to the output of a previous computation . The automatic construction of the program can be formulated as the solution to a sequential decision-making problem in a meta-level Markov decision process ( MDP ) ( Hay et al. , 2014 ) , where the state space is the learner ’ s internal states of computation and the action space is the set of modules . Framing the construction of a program as a reinforcement learning problem allows us to use techniques in deep reinforcement learning to implement loops and recursion , as well as decide on which part of the current state of computation to apply a module , to re-use sub-solutions to solve a larger problem . Our experiments on solving multilingual arithmetic problems and recognizing spatially transformed MNIST digits ( LeCun et al. , 1998 ) show that the above proposed training scheme prescribes a type of reformulation : re-representing a new problem in terms of other problems by implicitly making an analogy between their solutions . We also show that our meta-reasoning approach for deciding what modules to execute achieves better generalization to more complex problems than monolithic learners that are not explicitly compositional . 2 COMPOSITIONAL GENERALIZATION . Solving a problem simply means representing it so as to make the solution transparent . ( SIMON , 1988 ) Humans navigate foreign cities and understand novel conversations despite only observing a tiny fraction of the true distribution of the world . Perhaps they can extrapolate in this way because the world contains compositional structure , such that solving a novel problem is possible by composing previously learned partial solutions in a novel way to fit the context . With this perspective , we propose the concept of compositional generalization . The key assumption of compositional generalization is that harder problems are composed of easier problems . The problems from the training and test sets share the same primitive subproblems , but differ in the manner and complexity with which these subproblems are combined . Therefore , problems in the test set can be solved by combining solutions learned from the training set in novel ways . Definition . Let a problem P be a pair ( Xin , Xout ) , where Xin and Xout are random variables that respectively correspond to the input and output representations of the problem . Let the distribution of Xin be rin and the distribution of Xout be rout . To solve a particular problem P = p is to transform Xin = xin into Xout = xout . A composite problem pa = pb ◦ pc is that for which it is possible to solve by first solving pc and then solving pb with the output of pc as input . pb and pc are subproblems with respect to pa . The space of compositional problems form a compositional problem graph , whose nodes are the representation distributions r. A problem is described as pair of nodes between which the learner must learn to construct an edge or a path to transform between the two representations . Characteristics . First , there are many ways in which a problem can be solved . For example , translating an English expression to a Spanish one can be solved directly by learning such a transformation , or a learner could make an analogy with other problems by first translating English to French , and then French to Spanish as intermediate subproblems . Second , sometimes a useful ( although not only ) way to solve a problem is indicated by the recursive structure of the problem itself : solving the arithmetic expression 3 + 4× 7 modulo 10 can be decomposed by first solving the subproblem 4×7 = 8 and then 3+8 = 1 . Third , because a problem is just an ( input , output ) pair , standard problems in machine learning fit into this broadly applicable framework . For example , for a supervised classification problem , the input representation can be an image and the output representation a label , and intermediate subproblems can be transforming some intermediate representations to other intermediate representations . Sec . 4 demonstrates CRL on all three of the above examples . Broad Applicability . Problems in supervised , unsupervised , and reinforcement learning can all be viewed under the framework of transformations between representations . What we gain from the compositional problem graph perspective is a methodological way to relate together different problems of various forms and complexity , which is especially useful in a lifelong learning setting : the knowledge required to solve one problem is composed of the knowledge required to solve subproblems seen in the past in the context of different problems . For example , we can view latent variable reinforcement learning architectures such as ( Ha & Schmidhuber , 2018 ; Nair et al. , 2018 ) as simultaneously solving an image reconstruction problem and an action prediction problem , both of which share the same subproblem of transforming a visual observation into a latent representation . Lifelong learning , then , can be formulated as not only modifying the connections between nodes in the compositional problem graph but also continuing to make more connections between nodes , gradually expanding the frontier of nodes explored . Sec . 4 describes how CRL takes advantage of this compositional formulation in a multi-task zero-shot generalization setup to solve new problems by re-using computations learned from solving past problems . Evaluation . To evaluate a learner ’ s capacity for compositional generalization , we introduce two challenges . The first is to generalize to problems with different subproblem combinations from what the learner has seen . The second is to generalize to problems with longer subproblems combinations than the learner has seen . Evaluating a learner ’ s capability for compositional generalization is one way to measure how readily old knowledge can be reused and hence built upon . 3 A LEARNER THAT PROGRAMS ITSELF . This paper departs from the popular representation-centric view of knowledge ( Bengio et al. , 2013 ) and instead adopts a computation-centric view of knowledge : our goal is to encapsulate useful functionality shared across tasks into specialized computational modules – atomic function operators that perform transformations between representations . This section introduces the compositional recursive learner ( CRL ) , a framework for training modules to capture primitive subproblems and for composing together these modules as subproblem solutions to form a path between nodes of the compositional problem graph . 3.1 COMPOSITIONAL RECURSIVE LEARNER . The CRL framework consists of a controller π , a set of modules m ∈M , and an evaluator E. Training CRL on a diverse compositional problem distribution produces a modular recursive program that is trained to transform the input Xin into its output Xout , the corresponding samples of which are drawn from pairs of nodes in the compositional problem graph . In this program , the controller looks at the current state xi of the program and chooses a module m to apply to the state . The evaluator executes the module on that state to produce the next state xi+1 of the program . Xin is the initial state of the program , X̂out is the last , and the intermediate states Xi of the execution trace correspond to the other representations produced and consumed by the modules . The controller can choose to re-use modules across different program executions to solve different problems , making it straightforward to re-use computation learned from solving other problems to solve the current one . The controller can also choose to reuse modules several times within the same program execution , which produces recursive behavior . 3.2 DECIDING WHICH COMPUTATIONS TO EXECUTE . The sequential decision problem that the controller solves can be formalized as a meta-level Markov decision process ( meta-MDP ) ( Hay et al. , 2014 ) , whose state space corresponds to the intermediate states of computation X , whose action space corresponds to the modules M , and whose transition model corresponds to the evaluatorE . The symbiotic relationship among these components is shown in Fig . 2 . In the bounded-horizon version of CRL ( Sec . 4.2 ) , the meta-MDP has a finite horizon whose length is determined by the complexity of the current problem . In the infinite-horizon version of CRL ( Sec . 4.1 ) , the program itself determines when to halt when the controller selects the HALT signal . When the program halts , in both versions the current state of computation is produced as output x̂out , and CRL receives a terminal reward that reflects how x̂out matches the desired output xout . The infinite-horizon CRL also incurs a cost for every computation it executes to encourage it to customize its complexity to the problem . Note the following key characteristics of CRL . First , unlike standard reinforcement learning setups , the state space and action space can vary in dimensionality across and within episodes because CRL trains on problems of different complexity , reducing more complex problems to simpler ones ( Sec . 4.1 ) . Second , because the meta-MDP is internal to CRL , the controller shapes the meta-MDP by choosing which modules get trained and the meta-MDP in turn shapes the controller through its non-stationary state-distribution , action-distribution , and transition function . Thus CRL simultaneously designs and solves reinforcement learning problems “ in its own mind , ” whose dynamics depend just as much on the intrinsic complexity of the problem as well as the current problem-solving capabilities of CRL .

Summary: This paper is about trying to learn a function from typed input-output data so that it can generalize to test data with an input-output type that it hasn't seen during training. It should be able to use "analogy" (if we want to translate from French to Spanish but don't know how to do so directly, we should translate from French to English and English to Spanish). It should also be able to generalize better by learning useful "subfunctions" that can be composed together by an RL agent. We set up the solution as having a finite number of subfunctions, including "HALT" which signifies the end of computation. At each timestep an RL agent chooses a subfunction to apply to the current representation until "HALT" is chosen. The main idea is we parameterize these subfunctions and the RL agent as neural networks which are learned based on input -output data. RL agent is also penalized for using many subfunctions. The algorithm is called compositional recursive learner (CRL). Both analogy and meaningful subfunctions should arise purely because of this design.