Information Lattice Learning

1 INTRODUCTION . With rapid progress in AI , there is an increasing desire for general AI ( Goertzel & Pennachin , 2007 ; Chollet , 2019 ) and explainable AI ( Adadi & Berrada , 2018 ; Molnar , 2019 ) , which exhibit broad , human-like cognitive capacities . One common pursuit is to move away from “ black boxes ” designed for specific tasks to achieve broad generalization through strong abstractions made from only a few examples , with neither unlimited priors nor unlimited data ( “ primitive priors ” & “ small data ” instead ) . In this pursuit , we present a new , task-nonspecific framework—Information Lattice Learning ( ILL ) — to learn representations akin to human-distilled rules , e.g. , producing much of a standard music theory curriculum as well as new rules in a form directly interpretable by students ( shown at the end ) . The term information lattice was first defined by Shannon ( 1953 ) , but remains largely conceptual and unexplored . In the context of abstraction and representation learning , we independently develop representation lattices that coincide with Shannon ’ s information lattice when restricted to his context . Instead of inventing a new name , we adopt Shannon ’ s . However , we not only generalize the original definition—an information lattice here is a hierarchical distribution of representations—but we also bring learning into the lattice , yielding the name ILL. ILL explains a signal ( e.g. , a probability distribution ) by disentangled representations , called rules . A rule explains some but not all aspects of the signal , but together the collection of rules aims to capture a large part of the signal . ILL is specially designed to address the core question “ what makes X an X ” or “ what makes X different from Y ” , emphasizing the what rather than generating X or predicting labels X , Y in order to facilitate effective , rule-based explanations designed to help human learners understand . A music AI classifying concertos , or generating one that mimics the masters , does not necessarily produce human insight about what makes a concerto a concerto or the best rules a novice composer might employ to write one . Our focus represents a shift from much representation-learning work ( Bengio et al. , 2013 ) that aim to find the best representation for solving a specific task ( e.g. , classification ) rather than strong concern for explainability . Instead of optimizing a task-specific objective function ( e.g. , classification error ) , ILL balances more general objectives that favor fewer , simpler rules for interpretability , and more essential rules for effectiveness—all formalized later . One intuition behind ILL is to break the whole into simple pieces , similar to breaking a signal into a Fourier series . Yet , rather than decomposition via projection to orthonormal basis and synthesis via weighted sum , we decompose a signal in a hierarchical space called a lattice . Another intuition behind ILL is feature selection . Yet , rather than features , we use partitions to mimic human concepts and enable structured search in a partition lattice to mimic human learning . The goal is to restore human-like , hierarchical rule abstraction-and-realization through signal decomposition-and-synthesis in a lattice ( called projection-and-lifting , Figure 1 : left ) , resulting in more than a sum of parts . ILL comprises two phases : ( a ) lattice construction ; ( b ) learning ( i.e. , searching ) in the lattice . This is similar to many machine learning ( ML ) models comprising ( a ) function class specification then ( b ) learning in the function class , e.g. , constructing a neural network then learning—finding optimal parameters via back-propagation—in the network . ILL ’ s construction phase is prior-efficient : it builds in universal priors that resemble human innate cognition ( cf . the Core Knowledge priors ( Spelke & Kinzler , 2007 ) ) , then grows a lattice of abstractions . The priors can be customized , however , to cater to a particular human learner , or facilitate more exotic knowledge discovery . ILL ’ s learning phase is data-efficient : it learns from “ small data ” encoded by a signal , but searches for rich explanations of the signal via rule learning , wherein abstraction is key to “ making small data large ” . Notably , the construction phase is prior-driven , not data-driven—data comes in only at the learning phase . Hence , the same construction may be reused in different learning phases for different data sets or even data on different topics ( Figure 1 : right ) . Featuring these two phases , ILL is thus a hybrid model that threads the needle between a full data-driven model and a full prior-driven model , echoing the notion of “ starting like a baby ; learning like a child ” ( Hutson , 2018 ) . ILL is related to many research areas . It draws ideas and approaches from lattice theory , information theory , group theory , and optimization . It shares algorithmic similarity with a range of techniques including MaxEnt , data compression , autoencoders , and compressed sensing , but with a much greater focus on achieving human-like explainability and generalizability . Below , we broadly compares ILL to prominent , related models , leaving more comparisons to the Appendix for most similar ones . Compared to ILL is deep learning a “ white-box ” model balancing human-explainability and task performance Bayesian inference modeling human reasoning with widely shared , common priors and few , simple rules rather than using probabilistic inference as the driving force tree-like models structurally more general : a tree ( e.g. , decision tree or hierarchical clustering ) is essentially a linear lattice ( called a chain formally ) depicting a unidirectional refinement or coarsening process concept lattice in FCA ( Ganter & Wille , 2012 ) conceptually more general and may include both known and unknown concepts ; ILL does not require but discovers domain knowledge ( more details in Appendix A ) We illustrate ILL applications by learning music theory from scores , chemical laws from compounds , and show how ILL ’ s common priors facilitate mutual interpretation between the two subjects . To begin , imagine Tom and Jerry are playing two 12-key pianos simultaneously , one note at a time ( Figure 1 : right ) . The frequency of the played two-note chords gives a 2D signal plotted as a 12× 12 grayscale heatmap . Inspecting this heatmap , what might be the underlying rules that govern their co-play ? ( Check : all grey pixels have a larger “ Jerry-coordinate ” and project to a black key along the “ Tom-axis ” . ) We now elaborate on ILL and use it to distill rules for complex , realistic cases . 2 INFORMATION LATTICE : ABSTRACTIONS AND RULES OF A SIGNAL . Signal . A signal is a function ξ : X → R. For notational brevity and computational reasons , assume ξ is non-negative and X ⊆ Rn is finite ( not a limitation : see Appendix B ) . For example , a signal ξ : { 1 , . . . , 6 } → R being a probability mass function ( pmf ) of a dice roll , or a signal ξ : { 0 , . . . , 27 } 2 → R being a 28× 28 grayscale image . We denote the set of all signals on X by SX . Partition / abstraction . We use a partition P of a set X to denote an abstraction of X ; we call a cell C ∈ P an ( abstracted ) concept . The intuition is simple : a partition of a set renders a “ coarse-grained view ” of the set , or more precisely , an equivalence relation on the set . In this view , we identify equivalence classes of elements ( concepts ) instead of individual elements . For example , the partition P = { { 1 , 3 , 5 } , { 2 , 4 , 6 } } of the six outcomes of the roll of a die identify two concepts ( odd , even ) . Rule / representation . A rule of a signal ξ : X → R is a “ coarsened ” signal rξ : P → R defined on a partition P of X with rξ ( C ) : = ∑ x∈C ξ ( x ) for any C ∈ P . In this paper , a rule of a signal is what we mean by a representation of a signal . If the signal is a grayscale image , a rule can be a special type of blurring or downsampling of the image ; if the signal is a probability distribution , a rule can be a pmf of the “ orbits ” of the distribution for lifted inference algorithms ( Holtzen et al. , 2019 ; Kersting , 2012 ) . More generally , we define a rule ( regardless of any signal ) over a set X by any signal on any partition of X ; accordingly , we denote the set of all rules over X byRX : = ∪P∈ { all partitions of X } SP . Partition lattice . Abstractions are hierarchical : one coarse-grained view can be coarser than another . Let the partition lattice ( PX , ) of a setX be the partially ordered set ( poset ) containing all partitions of X equipped with the partial order coarser than ( ) , or finer than ( ) , defined in the standard way . Let P : = { { x } | x ∈ X } and P : = { X } denote the finest and the coarsest partition , respectively . Per general lattice theory ( Davey & Priestley , 2002 ) , PX is a complete lattice : every subset P ⊆ PX has a unique supremum ∨P and a unique infimum ∧P , where ∨P is called the join of P denoting its coarsest common refinement and ∧P is called the meet of P denoting its finest common coarsening . Information lattice . The information lattice ( Rξ , ⇐ ) of a signal ξ : X → R is the poset of all rules of ξ equipped with the partial order more general than : for any two rules r , r′ ∈ Rξ , we say r is more general than r′ ( or r′ is more specific ) , denoted r ⇐ r′ , if domain ( r ) domain ( r′ ) . Notably , Rξ ⊆ RX andRξ is isomorphic to the underlying partition lattice via projection defined below . Projection and lifting . For any signal ξ ∈ SX , we define the projection operator ↓ξ : PX → Rξ by letting ↓ξ ( P ) be the rule of ξ on P . One can check that ↓ξ : ( PX , ) → ( Rξ , ⇐ ) is an isomorphism . Conversely , we define the general lifting operator ⇑X : RX → 2SX by letting ⇑X ( r ) denote the set of all signals that satisfy the rule r , i.e. , ⇑X ( r ) : = { ξ ∈ SX | ↓ξ ( domain ( r ) ) = r } ⊆ SX . To make lifting unique and per Principles of Indifference ( Eva , 2019 ) , we introduce a special lifting ↑X ( r ) to pick the most “ uniform ” signal in ⇑X ( r ) . Formally , define ‖ · ‖q : SX → R by ‖ξ‖q : = ( ∑ x∈X ξ ( x ) q ) 1/q . For any ξ , ξ′ ∈ SX satisfying ‖ξ‖1 = ‖ξ′‖1 , we say that ξ is more uniform than ξ′ ( or ξ′ is more deterministic ) if ‖ξ‖2 ≤ ‖ξ′‖2 . We define the ( special ) lifting operator ↑X : RX → SX by ↑X ( r ) : = argminξ∈⇑X ( r ) ‖ξ‖2 ( can be computed by simply averaging ) . Notation here follows the convention as to function projections to quotient spaces ( Kondor & Trivedi , 2018 ) . Lifting a single rule to the signal domain can be extended in two ways : ( a ) lift to a finer rule domain P instead of X , i.e. , ⇑P ( r ) or ↑P ( r ) ; ( b ) lift more than one rules . Accordingly , we write ⇑ : = ⇑X and ↑ : = ↑X as defaults , write R = ↓ξ ( P ) : = { ↓ξ ( P ) | P ∈ P } ⊆ Rξ to denote a rule set , and write ⇑ ( R ) : = ∩r∈R ⇑ ( r ) = { η ∈ SX | ↓η ( P ) = R } and ↑ ( R ) : = argminη∈⇑ ( R ) ‖η‖2 to denote signals that satisfy all rules in R ( general lifting ) and the most uniform one ( special lifting ) , respectively . More computational details on lifting and its intimate relation to join are in Appendix C .

Summary: The authors perform a descriptive analysis of data by attempting to identify elements in the partial ordering of all partitions on the data which admit a compact definition. Compact definitions are those that are formed by composition of a small number of predefined (prior) set of mathematical operations. Projection and lifting operations are defined to relate descriptions of partition cells to one another through rules. The quality of a description is measured by the divergence between the data and the (special) lifting of the rule set, under the constraint that rules satisfy an upper bound on their entropy.