Factorizing Declarative and Procedural Knowledge in Structured, Dynamical Environments

1 INTRODUCTION . An intelligent agent that interacts with its world must not only perceive objects but must also remember its past experience with these objects . The wicker chair in one ’ s living room is not just a chair , it is the chair which has an unsteady leg and easily tips . Your keys may not be visible , but you recall placing them on the ledge by the door . The annoying fly buzzing in your left ear is the same fly you saw earlier which landed on the table . Visual cognition requires a short-term memory that keeps track of an object ’ s location , properties , and history . In the cognitive science literature , this particular form of state memory is often referred to as an object file ( Kahneman et al. , 1992 ) , which we ’ ll abbreviate as OF . An OF serves as a temporally persistent reference to an external object , permitting object constancy and permanence as the object and the viewer move in the world . Complementary to information in the OF is abstract knowledge about the dynamics and behavior of an object . We refer to this latter type of knowledge as a schema ( plural schemata ) , another term borrowed from the cognitive-science literature . The combination of OFs and schemata is sufficient to predict future states of object-structured environments , critical for planning and goal-seeking behavior . To model a complex , structured visual environment , multiple OFs must be maintained in parallel . Consider scenes like a PacMan video-game screen in which the ghosts chase the PacMan , a public 01 Mila , University of Montreal , 2 IIT BHU , Varanasi , 3 Waverly , 4 UC Berkeley , 5 Deepmind , 6 Google Research , Brain Team , Corresponding author : anirudhgoyal9119 @ gmail.com Object Schema 1 Schema 2 Schema 3 files PacMan Normal Scared ( OFs ) Ghost Ghost Left Frame A X B X C X D X E X Right Frame A X B X C X D X E X Figure 1 : Two successive frames of PacMan , illustrating the factorization of knowledge . Each ghost is represented by a persistent OF ( maintaining its location and velocity ) , but all ghosts operate according to one of two schemata , depending on whether the ghost is in a normal or scared state . square or sports field in which people interact with one another , or a pool table with rolling and colliding balls . In each of these environments , multiple instances of the same object class are present ; all operate according to fundamentally similar dynamics . To ensure systematic modeling of the environment , the same dynamics must be applied to multiple object instances . Toward this goal , we propose a method of separately representing the state of an individual object—via an OF—and how its state evolves over time—via a schema . Object-oriented programming ( OOP ) provides a metaphor for thinking about the relationship between OFs and schemata . In OOP , each object is an instantiation of an object class and it has a self-contained collection of variables whose values are specific to that object and methods that operate on all instances of the same class . The relation between objects and methods mirrors the relationship between our OFs and schemata . In both OOP and our view of visual cognition , a key principle is the encapsulation of knowledge : internal details of objects ( OFs ) are hidden from other objects ( OFs ) , and methods ( schemata ) are accessible to all and only objects ( OFs ) to which they are applicable . The modularity of knowledge in OOP supports human programmers in writing code that is readily debugged , extended , and reused . We conjecture that the corresponding modularity of OFs and schemata will lead to neural-network models with more efficient learning and more robust generalization , thanks to appropriate disentangling and separation of concerns . Modularity is the guiding principle of the model we propose , which we call SCOFF , an acronym for schema / object-file factorization . Like other neural net models with external memory ( e.g. , Mozer and Das , 1993 ; Graves et al. , 2016 ; Sukhbaatar et al. , 2015 ) , SCOFF includes a set of slots which are each designed to contain an OF ( Figure 2 ) . In contrast to most previous external memory models , the slots are not passive contents waiting to be read or written by an active process , but are dynamic , modular elements that seek information in the environment that is relevant to the object they represent , and when critical information is observed , they update their states , possibly via information provided by other OFs . Event-based OOP is a good metaphor for this active process , where external events can trigger the action of objects . As Figure 2 suggests , there is a factorization of declarative knowledge—the location , properties , and history of an object , as contained in the OFs—and procedural knowledge—the rules of object behavior , as contained in the schemata . Whereas declarative knowledge can change rapidly , procedural knowledge is more stable over time . This factorization allows any schema to be applied to any OF as deemed appropriate . The model design ensures systematicity in the operation of a schema , regardless of the slot to which an OF is assigned . Similarly , an OF can access any applicable schema regardless of which slot it sits in . Furthermore , a schema can be applied to multiple OFs at once , and multiple schemata could be applied to an OF ( e.g. , Figure 1 ) . In OOP , systematicity is similarly achieved by virtue of the fact that the same method can be applied to any object instantiation and that multiple methods exist which can be applied to an object of the appropriate type . Our key contribution is to demonstrate the feasibility and benefit of factorizing declarative knowledge ( the location , properties , and history of an object ) and procedural knowledge ( the way objects behave ) . This factorization enforces not only an important form of systematicity , but also of exchangeability : the model behaves exactly the same regardless of the assignment of schemata to schemata-slots and the assignment of objects to OF-slots , i.e. , the neural network operates on a set of objects and a set of schemata . With this factorization , we find improved accuracy of next-state prediction models and improved interpretability of learned parameters . 2 THE SCHEMATA / OBJECT-FILE FACTORIZATION ( SCOFF ) MODEL . SCOFF , shown in Figure 2 , is an architectural backbone that supports the separation of procedural and declarative knowledge about dynamical entities ( objects ) in an input sequence . The input sequence { x1 , . . . , xt , . . . , xT } , indexed by time step t is processed by a neural encoder ( e.g. , a fully convolutional net for images ) to obtain a deep embedding , { z1 , . . . , zt , . . . , zT } , which then drives a network with nf OFs and ns schemata . OFs are active processing components that maintain and update their internal state . Essentially , an OF is a layer of GRU ( Chung et al. , 2014 ) or LSTM ( Hochreiter and Schmidhuber , 1997 ) units with three additional bits of machinery , which we now describe . 1 . Our earlier metaphor identifying OFs in SCOFF with objects in OOP is apropos in the sense that OFs are event driven . OFs operate in a temporal loop , continuously awaiting relevant input signals . Relevance is determined by an attention-based soft competition among OFs : the current input serves a key that is matched to a query generated from the state of each OF ; based on the goodness of match , each OF is provided with a value extracted from the input . 2 . Each OF performs a one-step update of its state layer of GRU or LSTM units , conditioned on the input signal received . The weight parameters needed to perform this update , which we will denote generically as θ , are not—as in a standard GRU or LSTM—internal to the layer but rather are provided externally . Each schema j is nothing more than a set of parameters θj which can be plugged into this layer . SCOFF uses a key-value attention mechanism to perform Gumbel-based hard selection of the appropriate schema ( parameters ) . 3 . OFs may pass information to other OFs ( analogous to arguments being passed to a method in OOP ) , again using a soft attention mechanism by which each OF queries all other OFs . Keys provided by the other OFs are matched to the query , and a soft selection of the best matching OFs determines the weighting of values transmitted by the other OFs . This operation cycle ensures that OFs can update their state in response to both the external input and the internal state comprised of all the OFs ’ contents . This updating is an extra wrapper around the ordinary update that takes place in a GRU or LSTM layer . It provides additional flexibility in that ( 1 ) The external input is routed to OFs contingent on their internal state , ( 2 ) OFs can switch their dynamics from one time step to the next conditioned on their internal state , ( 3 ) OFs are modular in that they do not communicate with one another except via state-dependent selective message passing . OFs are placeholders in that a particular OF has no weight parameter specific to that OF . Instead , the parameters are provided from two sources : either the schemata or a pool of generic parameters shared by the nf OFs . This generic parameter pool is used to implement key-value attention over the input , the schemata , and communication among OFs . The sharing of schemata ensures systematicity ; the sharing of the generic parameter pool ensures exchangeability—model behavior is unaffected by the assignment of object instances to OF slots . 2.1 SCOFF SPECIFICS Algorithm 1 SCOFF model Input : Current sequence element , xt and previous OF state , { ht−1 , k| k ∈ { 1 , . . . , nf } } Step 1 : Process image by position p with fully convolutional net • cp = [ CNN ( xt ) ] p • ap = [ cp ep ] ( concatenate encoding of position to CNN output ) Step 2 : Soft competition among OFs to select regions of the input to process • qk =W qht−1 , k • sk , p = softmaxk ( qTk κp√ de ) , where κp = ( apW e ) T • zk = ∑ p sk , pvp where vp = apW v ∀ k ∈ { 1 , . . . , nf } Step 3 : OFs pick the most relevant schema and update • h̃t , k , j = GRUθj ( zk , ht−1 , k ) ∀k ∈ { 1 , . . . , nf } , j ∈ { 1 , . . . , ns } • q̃k = ht−1 , kW̃ q • ik = argmaxj ( q̃Tk κ̃k , j + γ ) , where κ̃k , j = ( h̃t , k , jW̃ e ) T and γ ∼ Gumbel ( 0 , 1 ) • ht , k = h̃t , k , ik Step 4 : Soft competition among OFs to transmit relevant information to each OF • q̂k = ht−1 , kŴ q ∀k { 1 , . . . , nf } • sk , k′ = softmaxk′ ( q̂Tk κ̂k′√ de ) where κ̂k′ = ( ht , k′Ŵ e ) T ∀ k , k′ ∈ { 1 , . . . , nf } • ht , k ← ht , k + ∑ k′ sk , k′ v̂k′ where v̂k′ = ht , k′Ŵ v ∀k ∈ { 1 , . . . , nf } Algorithm 1 provides a precise specification of SCOFF broken into four steps . Step 1 is external to SCOFF and involves processing an image input to obtain a deep embedding . The processing is performed by a fully Convolutional Neural Net ( CNN ) that preserves positional information ( typically 64× 64 in our simulations ) , and for each position p encodes the processed input cp and concatenates a learned encoding of position , ep to produce a position-specific hidden state . ( SCOFF can also handle non-visual inputs , such as vectors or discrete tokens ; in this case the CNN is replaced by an MLP . ) The subsequent core steps are as follows . Step 2 : Soft competition among OFs to select regions of the input to process . The state of each OF k , ht−1 , k , is used to form a query , qk , which determines the input positions that it will attend to . The query is matched to a set of position-specific input keys , κk , p for position p , producing a position-specific match score , sk , p . Soft position-specific competition among the OFs results in the OFs selecting complementary image regions to attend to . The contents of the attended positions are combined yielding an OF-specific input encoding , zk . Step 3 : OFs pick the most relevant schema and update . OF k picks one schema via attention as follows . OF k binds to each schema j , and then performs a hypothetical update , yielding h̃t , k , j . In experiments below , the OF state is maintained by a GRU layer , and schema j is a parameterization of the GRU , denoted GRUθj , which determines the update . The previous state of the OF , ht−1 , k serves as a query in key-value attention against a key derived from the hypothetical updated state , h̃t , k , j . The schema ik corresponding to the best query-key match for OF k is used to update OF k ’ s state . Selection is based on the straight-through Gumbel-softmax method ( Jang et al. , 2016 ) , which makes a hard choice during forward propagation and during backward propagation , it considers a softened version of the output to permit gradient propagation to non-selected schemata . Step 4 : Soft competition among OFs to transmit relevant information to each OF . This step allows for interactions among OFs . Each OF k queries other OFs for information relevant to its update as follows . The OF ’ s previous state , ht−1 , k , is used to form a query , q̂k , in an attention mechanism against a key derived from the new state of each other OF k′ , κ̂k′ , and softmax selection ( sk , k′ ) is used to obtain weighted information from other OFs , v̂′k , into OF k ’ s state . Number of Parameters . SCOFF can be used as a drop-in replacement for a LSTM/GRU layer . There is a subtlety that must be considered for successful integration . If the total size of the hidden state is kept the same , integrating SCOFF dramatically reduces the total number of recurrent parameters in the model because of its block-sparse structure . The majority of SCOFF parameters are the schemata , { θj |j ∈ { 1 , . . . , ns } } . The remaining parameters are those of query ( W q , W̃ q , Ŵ q ) , key ( W e , W̃ e , Ŵ e ) , and value W v , Ŵ v ) functions . Note that these linear functions could be replaced by nonlinear functions . Its also interesting to note that SCOFF actually has far fewer parameters than other modular architectures ( e.g. , RIMs , Recurrent Entity Networks , Neural Module Networks ) when ns < nf ( which holds true for most of our experiments ) , because of the potential one-to-many mapping between schemata and OFs . Optionally , at Step 2 , during training instead of activating all the OFs , we can use a sparse attention , to only activate a subset of OFs that are relevant at that time step t , and only update the state of the activated OFs . We note that this is not specific to the proposed method , but the performance of the method can be improved by only selectively activating the relevant OFs . We also note that the different schemata in the proposed method are only used to update the state of OFsand hence is agnostic as to how one obtains a set of OFs . Any other method for entity extraction ( Burgess et al. , 2019 ; Greff et al. , 2019 ) can be used for obtaining a set of OFs . Similarly , other methods for decoding which preserves equivariance ( Watters et al. , 2019 ) among different OFscan be used for decoding the contents of different OFs .

Summary: This paper proposes a new type of recurrent neural network architecture called schema / object-file factorization (SCOFF). This model contains multiple weight-sharing GRU cells. The input information is fed into each GRU cells through an attention layer. The output information is fetched from these GRU cells and mixed with another attention layer. The model is tested on several intuitive physics benchmarks and basic reinforcement learning environment. This model demonstrates superior performance than other modular RNN architectures such as RIM on specific tasks.