Permuton-induced Chinese Restaurant Process

1 Introduction . The multi-dimensional extension of the Chinese restaurant process ( CRP ) [ 9 , 50 ] for infinitely exchangeable random arrays is one of the most significant unsolved issues in the field of Bayesian nonparametrics and its applications to machine learning . The standard CRP is the stochastic process representing random clustering of a sequence using the metaphor of customers sitting at tables . Typically , it is often used when we need to cluster a sequence where the number of clusters is unknown in advance . The magic of CRP is that it requires only a finite number of parameters for finite data ; nevertheless , the model itself has infinite representational power . Specifically , it is known that a mixture model using CRP is theoretically equivalent to the Dirichlet process mixture model ( DPMM ) [ 51 , 27 ] represented by the stick-breaking process ( SBP ) [ 57 ] with an infinite number of parameters . The relationship between the two DPMM representations , CRP and SBP , can be explained by whether the intermediate random variables in de Finetti ’ s representation theorem [ 28 , 37 ] are marginalized out or not ( Figure 1 , ( a ) and ( b ) ) . CRP has contributed greatly to the development of the BNP machine learning field in the sense that it has made BNP not only theoretical but also practical . It enables models with infinite representational power to be accurately simulated on a computer with a finite number of parameters without any approximation ( e.g. , finite truncation ) . In fact , a variety of extensions of CRP have been studied , including the nested CRP [ 16 , 17 , 60 ] , the Chinese restaurant franchise [ 61 ] , the tree-structured CRP [ 6 ] , the distance-dependent CRP [ 14 , 15 , 31 ] . However , in the development of CRP , the extension to BNP relational models has remained a historical conundrum . 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . As a special case of the BNP relational models , the infinite relational model ( IRM ) [ 38 ] consisting of the product of CRPs was proposed . The IRM can express random rectangular partitioning of a matrix ; however , its representational capability is restricted to a minimal subset of rectangular partitioning called regular grid ( Figure 2 , left ) . Therefore , in order to obtain a model with higher representational capability , a variety of BNP models for relational data have been rapidly developed in recent years [ 38 , 55 , 54 , 20 , 53 , 58 , 46 , 36 , 19 , 7 , 23 , 26 , 43 , 44 , 30 , 21 , 25 , 24 , 56 ] . Recent excellent surveys can be found in [ 22 , 49 ] . In particular , the Mondrian process ( MP ) [ 55 , 54 ] and the rectangular tiling process ( RTP ) [ 48 ] have been proposed as new BNP models to expand the expressive classes of rectangular partitioning . However , both of them require an infinite number of parameters as intermediate random variables in the Aldous-Hoover-Kallenberg ( AHK ) representation theorem [ 8 , 34 , 37 ] ( an extension of de Finetti ’ s theorem to multi-dimensional arrays ) , similar to the SBP representation for DPMM . As a result , Bayesian inference to those models always had to take care of an infinite number of parameters , which compromise the flexibility of local movements for the Markov chain Monte Carlo ( MCMC ) inference and make it highly affected by local optima and slow mixing problems . As a natural question , we wonder if it is possible to marginalize out the infinite number of intermediate random variables of these models , as in the CRP representation of DPMM . Unfortunately , an approach to achieve this has remained unsolved for a long time . This paper deals with the multi-dimensional extension of CRP to represent wider classes of rectangular partitioning than regular grid partitioning . As an important stepping stone for this , the block-breaking process ( BBP ) [ 47 ] has been proposed very recently as a multi-dimensional extension of SBP ( Figure 1 , ( c ) ) . The key insight for the BBP construction is that rectangular partitioning can be represented indirectly using permutations . Specifically , a special class of permutations called Baxter permutation [ 13 ] , which has a one-to-one correspondence with a particular class of rectangular partitioning , is used to extend the stick-breaking procedures of SBP for a sequence to the block-breaking procedures for a matrix ( Figure 3 ) . Although BBP can indeed be used as an infinitely exchangeable relational model , it requires an infinite number of redundant parameters , similar to SBP , making Bayesian inference highly affected by local optima and slow mixing problems . Therefore , a CRP-like representation that avoids the infinite number of redundant parameters of BBP , analogous to the relationship between SBP and CRP for DPMM , has been desired . Our contributions are summarized as follows : Construction of new stochastic process - Section 3.1 proposes a stochastic process such that each table of the CRP has a random coordinates on [ 0 , 1 ] × [ 0 , 1 ] . Specifically , we suppose that the coordinates of each table is independent and identically distributed ( i.i.d . ) random variable drawn from a permuton [ 35 ] on [ 0 , 1 ] × [ 0 , 1 ] . Permuton is a probabilistic measure on [ 0 , 1 ] × [ 0 , 1 ] and can also be regarded as a prior model for random permutations . By exploiting the correspondence between permutations and rectangular partitions [ 62 , 52 ] , PCRP can be used as a generative probabilistic model of rectangular partitioning of a matrix . While BBP [ 47 ] ( i.e. , a multi-dimensional extension of SBP ) always has an infinite number of redundant intermediate variables , our PCRP can be composed of varying size ( finite ) intermediate variables in a data-driven manner depending on the size and quality of the observation data . As a result , PCRP can lead to very flexible local movements in Bayesian inference , reducing the problems of local optima and slow mixing . Unified framework for various classes of rectangular partitioning - Conventionally , the stochastic processes for each class of rectangular partitioning have been constructed individually . However , according to the design of the permuton , PCRP provides a unified framework for various classes of rectangular partitioning . Section 3.2 shows an approach to tune the permuton of PCRP to adapt it to various classes of rectangular partitioning . New representation of BNP model - As mentioned earlier , a multi-dimensional extension of CRP is a historical conundrum , and thus PCRP does not provide a perfect solution to this issue . In Section 4 , the theoretical imperfection of PCRP is clarified . Specifically , we intuitively explain that projectivity and exchangeability of PCRP change when viewed from two different perspectives : rectangular partitioning of a matrix and table assignment of data . Thus , we propose a new representation of the BNP model as a way to overcome this imperfection , that applies PCRP as a bridging state to existing infinitely exchangeable relational models , inspired by the MCMC inference via bridging [ 42 ] . 2 Preliminaries . 2.1 Permutation , permutation class and its relation to rectangular partitioning . Permutation and rectangulation - A permutation of a set S is defined as a bijection from S to itself . Specifically , this paper always focus on permutations of natural numbers S = [ n ] : = { 1 , 2 , . . . , n } ( n ∈ N ) . For convenience , we use one-line notation for a permutation σ = σ1σ2 . . . σn . For example , the permutation σ = 312 means σ1 = 3 , σ2 = 1 , σ3 = 2 . A rectangulation ( also called rectangular partitioning ) is a tiling of a rectangle by a finite number of disjoint rectangles such that no four of its rectangles share a single corner . In recent years , it has become clear that there is a close relationship between permutations and rectangulations . Here we focus in particular on the fact that every permutation can be uniquely converted into a generic rectangulation ( Figure 2 , right ) , that is , the class of rectangulations without any constraints : Proposition 2.1 ( See Proposition 6.2 in [ 39 ] and Proposition 4.2 in [ 52 ] ) There is a surjective map from permutations to generic rectangulations . Fortunately , we can constructively obtain this map as the following two-stage transformation from a permutation to a general rectangulation by way of a diagonal rectangulation . A rectangulation is diagonal if the interior of each rectangle has an area that intersects the diagonal . Figure 4 and Figure 5 describe the details of the first and second steps , respectively . We use this transformation as an important component in our generative model of PCRP ( in Section 3.1 ) . Permutation classes and rectangulation classes - To investigate the properties of the permutation , we can focus on subsets of permutations as permutation classes . To identify a permutation class , we introduce the notion of occurrences . We suppose that τ and σ are permutations of size k and n , respectively . The occurrences of pattern τ in σ is a subsequence σi1 , . . . , σik that is orderisomorphic to τ , that is , for all indices s , t ∈ [ k ] , we have σis < σit ⇐⇒ τs < τt . One way to characterize a permutation class is often to focus on what patterns ( typically very short permutations ) are not contained as occurrences in the permutations belonging to that class . Conventionally , various permutation classes have been studied in depth , however , here we would like to focus on several permutation classes that are particularly relevant to rectangular partitioning . The details of each class will be explained in the supplementary material . It should be emphasized here that , surprisingly , permutations of a particular class have a one-to-one correspondence with rectangulations of a particular class . A brief list is given below . We will use this fact in Section 3.2 to show that PCRP can potentially serve as a unified model for the various classes of rectangular partitioning . 2-clumped permutation : The 2-clumped permutations have a bijection to generic rectangulations ( Figure 2 , right ) [ 52 ] , that no restrictions are required . Baxter permutation : and twisted Baxter permutation : The Baxter permutations have a bijection to diagonal rectangulations ( Figure 2 , fourth from left ) [ 62 , 5 , 32 ] , which has a representative in which every rectangle ’ s interior intersects the diagonal . Separable permutation : Separable permutations have a bijection to hierarchical and diagonal rectangulations [ 59 , 5 ] , a subset of hierarchical partitioning ( Figure 2 , second from left ) , which are expressed as binary trees where nodes represent a vertical or horizontal separation of a rectangle into two disjoint rectangles . Separable skew-merged permutation : Separable skew-merged permutations have a bijection to a special subset of hierarchical partitioning , in particular , those where only one rectangle is allowed to be further cut in each layer ( Figure 2 , third from left ) [ 45 ] .

Summary: This paper proposed a model for relational data -- permuton-induced Chinese restaurant process, which can be regarded as a multidimensional Chinese restaurant process. The permuton is used to fill in the input matrix according to a Chinese restaurant scheme. The underlying model is well designed for rectangular partitioning, and various empirical results are also provided.