Permuton-induced Chinese Restaurant Process

1 Introduction . The multi-dimensional extension of the Chinese restaurant process ( CRP ) [ 9 , 50 ] for infinitely exchangeable random arrays is one of the most significant unsolved issues in the field of Bayesian nonparametrics and its applications to machine learning . The standard CRP is the stochastic process representing random clustering of a sequence using the metaphor of customers sitting at tables . Typically , it is often used when we need to cluster a sequence where the number of clusters is unknown in advance . The magic of CRP is that it requires only a finite number of parameters for finite data ; nevertheless , the model itself has infinite representational power . Specifically , it is known that a mixture model using CRP is theoretically equivalent to the Dirichlet process mixture model ( DPMM ) [ 51 , 27 ] represented by the stick-breaking process ( SBP ) [ 57 ] with an infinite number of parameters . The relationship between the two DPMM representations , CRP and SBP , can be explained by whether the intermediate random variables in de Finetti ’ s representation theorem [ 28 , 37 ] are marginalized out or not ( Figure 1 , ( a ) and ( b ) ) . CRP has contributed greatly to the development of the BNP machine learning field in the sense that it has made BNP not only theoretical but also practical . It enables models with infinite representational power to be accurately simulated on a computer with a finite number of parameters without any approximation ( e.g. , finite truncation ) . In fact , a variety of extensions of CRP have been studied , including the nested CRP [ 16 , 17 , 60 ] , the Chinese restaurant franchise [ 61 ] , the tree-structured CRP [ 6 ] , the distance-dependent CRP [ 14 , 15 , 31 ] . However , in the development of CRP , the extension to BNP relational models has remained a historical conundrum . 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . As a special case of the BNP relational models , the infinite relational model ( IRM ) [ 38 ] consisting of the product of CRPs was proposed . The IRM can express random rectangular partitioning of a matrix ; however , its representational capability is restricted to a minimal subset of rectangular partitioning called regular grid ( Figure 2 , left ) . Therefore , in order to obtain a model with higher representational capability , a variety of BNP models for relational data have been rapidly developed in recent years [ 38 , 55 , 54 , 20 , 53 , 58 , 46 , 36 , 19 , 7 , 23 , 26 , 43 , 44 , 30 , 21 , 25 , 24 , 56 ] . Recent excellent surveys can be found in [ 22 , 49 ] . In particular , the Mondrian process ( MP ) [ 55 , 54 ] and the rectangular tiling process ( RTP ) [ 48 ] have been proposed as new BNP models to expand the expressive classes of rectangular partitioning . However , both of them require an infinite number of parameters as intermediate random variables in the Aldous-Hoover-Kallenberg ( AHK ) representation theorem [ 8 , 34 , 37 ] ( an extension of de Finetti ’ s theorem to multi-dimensional arrays ) , similar to the SBP representation for DPMM . As a result , Bayesian inference to those models always had to take care of an infinite number of parameters , which compromise the flexibility of local movements for the Markov chain Monte Carlo ( MCMC ) inference and make it highly affected by local optima and slow mixing problems . As a natural question , we wonder if it is possible to marginalize out the infinite number of intermediate random variables of these models , as in the CRP representation of DPMM . Unfortunately , an approach to achieve this has remained unsolved for a long time . This paper deals with the multi-dimensional extension of CRP to represent wider classes of rectangular partitioning than regular grid partitioning . As an important stepping stone for this , the block-breaking process ( BBP ) [ 47 ] has been proposed very recently as a multi-dimensional extension of SBP ( Figure 1 , ( c ) ) . The key insight for the BBP construction is that rectangular partitioning can be represented indirectly using permutations . Specifically , a special class of permutations called Baxter permutation [ 13 ] , which has a one-to-one correspondence with a particular class of rectangular partitioning , is used to extend the stick-breaking procedures of SBP for a sequence to the block-breaking procedures for a matrix ( Figure 3 ) . Although BBP can indeed be used as an infinitely exchangeable relational model , it requires an infinite number of redundant parameters , similar to SBP , making Bayesian inference highly affected by local optima and slow mixing problems . Therefore , a CRP-like representation that avoids the infinite number of redundant parameters of BBP , analogous to the relationship between SBP and CRP for DPMM , has been desired . Our contributions are summarized as follows : Construction of new stochastic process - Section 3.1 proposes a stochastic process such that each table of the CRP has a random coordinates on [ 0 , 1 ] × [ 0 , 1 ] . Specifically , we suppose that the coordinates of each table is independent and identically distributed ( i.i.d . ) random variable drawn from a permuton [ 35 ] on [ 0 , 1 ] × [ 0 , 1 ] . Permuton is a probabilistic measure on [ 0 , 1 ] × [ 0 , 1 ] and can also be regarded as a prior model for random permutations . By exploiting the correspondence between permutations and rectangular partitions [ 62 , 52 ] , PCRP can be used as a generative probabilistic model of rectangular partitioning of a matrix . While BBP [ 47 ] ( i.e. , a multi-dimensional extension of SBP ) always has an infinite number of redundant intermediate variables , our PCRP can be composed of varying size ( finite ) intermediate variables in a data-driven manner depending on the size and quality of the observation data . As a result , PCRP can lead to very flexible local movements in Bayesian inference , reducing the problems of local optima and slow mixing . Unified framework for various classes of rectangular partitioning - Conventionally , the stochastic processes for each class of rectangular partitioning have been constructed individually . However , according to the design of the permuton , PCRP provides a unified framework for various classes of rectangular partitioning . Section 3.2 shows an approach to tune the permuton of PCRP to adapt it to various classes of rectangular partitioning . New representation of BNP model - As mentioned earlier , a multi-dimensional extension of CRP is a historical conundrum , and thus PCRP does not provide a perfect solution to this issue . In Section 4 , the theoretical imperfection of PCRP is clarified . Specifically , we intuitively explain that projectivity and exchangeability of PCRP change when viewed from two different perspectives : rectangular partitioning of a matrix and table assignment of data . Thus , we propose a new representation of the BNP model as a way to overcome this imperfection , that applies PCRP as a bridging state to existing infinitely exchangeable relational models , inspired by the MCMC inference via bridging [ 42 ] . 2 Preliminaries . 2.1 Permutation , permutation class and its relation to rectangular partitioning . Permutation and rectangulation - A permutation of a set S is defined as a bijection from S to itself . Specifically , this paper always focus on permutations of natural numbers S = [ n ] : = { 1 , 2 , . . . , n } ( n ∈ N ) . For convenience , we use one-line notation for a permutation σ = σ1σ2 . . . σn . For example , the permutation σ = 312 means σ1 = 3 , σ2 = 1 , σ3 = 2 . A rectangulation ( also called rectangular partitioning ) is a tiling of a rectangle by a finite number of disjoint rectangles such that no four of its rectangles share a single corner . In recent years , it has become clear that there is a close relationship between permutations and rectangulations . Here we focus in particular on the fact that every permutation can be uniquely converted into a generic rectangulation ( Figure 2 , right ) , that is , the class of rectangulations without any constraints : Proposition 2.1 ( See Proposition 6.2 in [ 39 ] and Proposition 4.2 in [ 52 ] ) There is a surjective map from permutations to generic rectangulations . Fortunately , we can constructively obtain this map as the following two-stage transformation from a permutation to a general rectangulation by way of a diagonal rectangulation . A rectangulation is diagonal if the interior of each rectangle has an area that intersects the diagonal . Figure 4 and Figure 5 describe the details of the first and second steps , respectively . We use this transformation as an important component in our generative model of PCRP ( in Section 3.1 ) . Permutation classes and rectangulation classes - To investigate the properties of the permutation , we can focus on subsets of permutations as permutation classes . To identify a permutation class , we introduce the notion of occurrences . We suppose that τ and σ are permutations of size k and n , respectively . The occurrences of pattern τ in σ is a subsequence σi1 , . . . , σik that is orderisomorphic to τ , that is , for all indices s , t ∈ [ k ] , we have σis < σit ⇐⇒ τs < τt . One way to characterize a permutation class is often to focus on what patterns ( typically very short permutations ) are not contained as occurrences in the permutations belonging to that class . Conventionally , various permutation classes have been studied in depth , however , here we would like to focus on several permutation classes that are particularly relevant to rectangular partitioning . The details of each class will be explained in the supplementary material . It should be emphasized here that , surprisingly , permutations of a particular class have a one-to-one correspondence with rectangulations of a particular class . A brief list is given below . We will use this fact in Section 3.2 to show that PCRP can potentially serve as a unified model for the various classes of rectangular partitioning . 2-clumped permutation : The 2-clumped permutations have a bijection to generic rectangulations ( Figure 2 , right ) [ 52 ] , that no restrictions are required . Baxter permutation : and twisted Baxter permutation : The Baxter permutations have a bijection to diagonal rectangulations ( Figure 2 , fourth from left ) [ 62 , 5 , 32 ] , which has a representative in which every rectangle ’ s interior intersects the diagonal . Separable permutation : Separable permutations have a bijection to hierarchical and diagonal rectangulations [ 59 , 5 ] , a subset of hierarchical partitioning ( Figure 2 , second from left ) , which are expressed as binary trees where nodes represent a vertical or horizontal separation of a rectangle into two disjoint rectangles . Separable skew-merged permutation : Separable skew-merged permutations have a bijection to a special subset of hierarchical partitioning , in particular , those where only one rectangle is allowed to be further cut in each layer ( Figure 2 , third from left ) [ 45 ] .

Summary: The paper introduces the permuton-induced Chinese restaurante process (PCRP), a prior over rectangular partitions of 2D space, and applies the prior to analyze relational data. The sample paths of the prior (Figure 2) exhibit different behavior depending on the permuton measure gamma, which is one of the design choices in the prior. In numerical experiments, probabilistic models using the proposed prior perform competitively with existing choices of prior in terms of held-out perplexity.