Necessary and Sufficient Conditions for Compositional Representations

1 INTRODUCTION . Humans recognize the world and create imaginations in a supple way by leveraging systematic compositionality to achieve compositional generalization , the algebraic capacity to understand and produce large amount of novel combinations from known components ( Chomsky , 1957 ; Montague , 1970 ) . This is a key element of human intelligence ( Minsky , 1986 ; Lake et al. , 2017 ) , and we hope to equip machines with such ability . Conventional machine learning has been mainly developed with an assumption that training and test distributions are identical . Compositional generalization , however , is a type of out-of-distribution generalization ( Bengio , 2017 ) which has different training and test distributions . In compositional generalization , a sample is a combination of several components . For example , an image object may have two factor components of color and rotation . In language , a sentence is composed of the lexical meanings and the grammatical structure . The generalization is enabled by recombining seen components for an unseen combination during inference . One approach for compositional generalization is to learn compositional representations1 , or disentangled representation ( Bengio , 2013 ) , which contain several component representations . Each of them depends only on the corresponding underlying factor , and does not change when other factors change . Please see Section 3 for details . Multiple methods have been proposed to learn compositional representations . However , little discussion has been made for some fundamental questions . What kind of factor combinations can be expressed in compositional representation ? Though there are some common factor components such as colors and size , what property enable them ? When a set of components satisfy the conditions , what kind of mappings are available between the entangled and compositional representations ? Can we use the conditions to explain compositionality in conventional models such as attention ? In this paper , we mathematically prove two propositions ( Proposition 1.1 and Proposition 1.2 ) for necessary and sufficient conditions regarding compositional representations . We construct groups for changes on representations , and relate compositional representation with group direct product , and compositional mapping with group action equivalence ( Higgins et al. , 2018 ) . Then , we use theorems and propositions in group theory to prove the conditions . 1The word “ representation ” in this paper refers to variables , not group representation . Proposition 1.1 ( Compositional representation ) . A set of components can be expressed compositionally if and only if the subgroup product equals to the original group , each component subgroup is normal subgroup of the original group , and the group elements intersect only at identity element . Proposition 1.2 ( Compositional mapping ) . Given compositional representation , a mapping is compositional if and only if each component has equivalent action in compositional and entangled representations , and for each element of the entangled representation , the orbits intersect only at the element . Please see Proposition 4.2 and Proposition 4.10 for symbolic statements . We also provide examples to better understand the conditions and how to use them ( Section 5 ) . For representations , we see that whether the components can be expressed with compositional representation does not depend only on each component itself , but also on their combination , and the possible values to take . We use the condition for compositional mapping to explain some existing neural network models and tasks , e.g. , attention mechanism , spacial transformer and grammar tree nodes . We hope , with these examples , the conditions will be used for validating different compositional representations and mappings , and guiding designs of tasks and algorithms with compositionality . Our contributions can be summarized as follows . • We propose and prove necessary and sufficient conditions for compositional representation and compositional mapping . • We provide examples to understand and use the conditions , such as new explanation of attention models . 2 RELATED WORK . Human-level compositional learning ( Marcus , 2003 ; Lake & Baroni , 2018 ) has been an important open challenge ( Yang et al. , 2019 ; Keysers et al. , 2020 ) . There are recent progress on measuring compositionality ( Andreas , 2019 ; Lake & Baroni , 2018 ; Keysers et al. , 2020 ) and learning language compositionality for compositional generalization ( Lake , 2019 ; Russin et al. , 2019 ; Li et al. , 2019 ; Gordon et al. , 2020 ; Liu et al. , 2020 ) and continual learning ( Jin et al. , 2020 ; Li et al. , 2020 ) . Another line of related but different work is statistically and marginally independent disentangled representation learning ( Burgess et al. , 2018 ; Locatello et al. , 2019 ) . This setting assumes marginal independence between underlying factors hence does not have compositional generalization problem . On the other hand , compositional factors may not be marginally independent . Understanding of compositionality has been discussed over time . Some discussions following Montague ( 1970 ) uses homomorphism to define composition operation between representations . Recently , Higgins et al . ( 2018 ) proposes definition of disentangled representation with group theory . The definition is the base of this paper , and we focus on proving the conditions . Li et al . ( 2019 ) defines compositionality probabilistically without discussing conditions to achieve it . Gordon et al . ( 2020 ) finds compositionality in SCAN task can be expressed as permutation group action equivalence . This equivalent action is on a component subgroup , but it does not discuss equivalent action on the whole group and the relations between them . There are also other works related to group theory in machine learning ( Kondor , 2008 ; Cohen & Welling , 2016 ; Ravanbakhsh et al. , 2017 ; Kondor & Trivedi , 2018 ) . However , the previous works do not prove conditions for compositional representation or mapping . In this paper , we provide and theoretically prove necessary and sufficient conditions for compositional representations and compositional mappings . We use definitions , propositions and theorems from group theory . Please refer to Appendix A . Some of them are summarized in books , such as Dummit & Foote ( 2004 ) and Gallian ( 2012 ) , and we refer to them in the later sections . 3 REPRESENTATIONS . In this section , we introduce the definitions of representation and compositional representation used in this paper . Representation in this paper is consistent with the concept in neural network literature . It is a variable , and its value depends on each sample . For example , it can be activations for a layer in a neural network . The values of the activations depend on the network input . Network input and output are also called representations . Compositional representation in this paper means a representation with several separated component representations . It is also called disentangled representation in some literature . “ Separated ” means that the representation is the concatenation of the component representations . Each component representation corresponds to a underlying component , or a generative factor . When a representation is not compositional , it is an entangled representation . In the examples in Figure 1 , the components are color and shape . The upper images are entangled representations , where color and shape are in the same image . However it is not a compositional representation , because an image is not a concatenation of a color part and a rotation part . The lower vectors are compositional representations , where the left vector is for color and the right vector is for shape . 4 NECESSARY AND SUFFICIENT CONDITIONS . In this section , we derive necessary and sufficient conditions for compositionality step by step . We first construct groups for representations . We then describe compositionality with group properties , and study the conditions for them . Based on that , we further study the conditions for mappings between two representations . 4.1 GROUPS ON REPRESENTATIONS . Compositionality arises when we compare different samples , where some components are the same but others are not . This means compositionality is related to the changes between samples . These changes can be regarded as mappings , and since the changes are invertible , the mappings are bijective . To study compositionality we consider a set of all bijections from a set of possible representation values to the set itself , and construct a group with the following Proposition 4.1 . Proposition 4.1 . Let X be any nonempty set and SX be the set of all bijections from X to itself . The set SX is a group under function composition2 . Dummit & Foote ( 2004 ) P.29 Since SX contains all bijections , the group SX acts on the set X ( Definition A.9 ) , and the action is transitive ( Definition A.12 ) . We consider two representations and corresponding sets . X is original entangled representation , and Y is compositional representation . We create group G on set X , and group H on set Y . 4.2 COMPOSITIONAL REPRESENTATION . When multiple hidden variables live in the same representation , and can not be separated by simply splitting the representation , then these variables are entangled in the representation . For example , 2Function composition is different from compositionality being discussed . rotation and color are two hidden variables and they are both in a representation of image . We hope to extract the hidden variables by disentangling the representation . Suppose X is a set of entangled representations , and Y is a set of compositional representations . Y has Cartesian product ofK small sets Y1 , . . . , YK . We hope to find the conditions the changes on X can be expressed by the changes on the components in Y . A component corresponds to a set . For example , color component can take blue , green , etc. , from a set of colors . With Proposition 4.1 , we can construct a group for each component . With Definition A.2 , each of these groups is a subgroup of the original group . We consider K subgroups . We hope the changes on the entangled representation X are equally expressed by changes on the compositional representation Y . This means group G should be isomorphic with the external direct product ( Proposition A.1 ) of the subgroups H = N1 × · · · ×NK . The following Proposition 4.2 has the necessary and sufficient conditions . Proposition 4.2 . N1 , . . . , NK are subgroups of group G. G is isomorphic to the external direct product of the subgroups if and only if G is internal direct product of the subgroups . From Definition A.8 , we have the following . G ∼= N1 × · · · ×NK ⇐⇒ G = N1N2 . . . NK ( A1 ) Ni / G , ∀i = 1 , . . . , K ( A2 ) ( N1 . . . Ni ) ∩Ni+1 = { e } , ∀i = 1 , . . . , K − 1 ( A3 ) Proof . “ ⇐= ” : Theorem A.2 . “ =⇒ ” : G andN1×· · ·×NK are isomorphism , andN1×· · ·×NK satisfies the conditions by construction in definition . ( A1 ) means the subgroup product should cover the original group . ( A2 ) means all the component subgroups are normal subgroups of the original group . ( A3 ) means the intersection of a subgroup and the previous subgroups only contain the identity element . This corresponds to Proposition 1.1 . We will provide examples and look into more details in discussion section .

Summary: This paper applies concepts from group theory to help find necessary and sufficient representations on the presence of compositionality in representations and on mappings between them.  This topic is very important, as people tend to use "compositional" in many ways, often not explicitly defined.  The paper, however, is hard to follow, because the main concepts and theorems are not adequately illustrated with examples.  In the final section, when examples are provided, it's still not clear how they apply to representation learning, the topic of this conference.  Thus, while the topic is very relevant and the approach welcome, it is hard to know what lesson we have learned from the theorems in the paper.