Partition and Code: learning how to compress graphs

1 Introduction . Lossless data compression has been one of the most fundamental and long-standing problems in computer science . It is by now well-understood that the intrinsic limits of compression are governed by the entropy of the underlying data distribution [ 1 ] . Crucially , these limits expose an intimate connection between compressibility and machine learning : the better one models the underlying data distribution ( from limited observations ) the more bits can be saved and vice-versa [ 2 ] . The compression of ordered data such as text , images , or video , underpins the modern technology from web protocols to video streaming . However , graph-structured data remain a notable exception . As graph data are becoming more prevalent , it becomes increasingly important to invent practical ways to encode them parsimoniously . There are three main challenges one faces when attempting to compress graphs : ∗This work was done while Giorgos Bouritsas was visiting Dr. Andreas Loukas at EPFL , Lausanne . 1The source code is publicly available at https : //github.com/gbouritsas/PnC 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . C1 . Dealing with graph isomorphism ( GI ) . A key difficulty that distinguishes graphs from conventional data lies in the absence of an inherent ordering of the graph vertices . In order to be able to approach the storage lower bounds , isomorphic graphs should be encoded with the same codeword . However , since the complexity of known algorithms for GI is super-polynomial on the number of vertices [ 3 ] , a direct use of GI is impractical for graphs consisting of more than a few hundred vertices . Indeed , an examination of the graph compression literature reveals that most progress has been made by optimising a vertex ordering and adapting methods originally invented for vectored data [ 4–6 ] . Unfortunately , naively encoding graphs as vectors results in a significant loss in compression2 . C2 . Evaluating the likelihood . An optimal encoder [ 1 ] requires one to accurately estimate and evaluate the probabilities of all the possible outcomes of the underlying domain . When dealing with high-dimensional data , this can be addressed by partitioning the data into parts to obtain a decomposition of the probability distribution : e.g. , images can be compressed by modelling the distribution of pixels or patches [ 10–12 ] , and text by focusing on characters or n-grams [ 13–19 ] . However , since graphs do not admit an efficiently computable canonical ordering , it is unclear what decomposition one should employ . C3 . Accounting for the description length of the learned model . The classical learning theory trade-off between model complexity and generalisation is of paramount concern for effective compression . Though in typical deep learning applications one can aim to model the data distribution with an overparametrised neural network ( NN ) that generalises well , utilising such models to compress information is problematic : since decoding is impossible unless the decoder also receives the learned model ( i.e. , the NN parameters ) , overparametrised models are , by definition , suboptimal . This is a pertinent issue for likelihood-based neural approaches as overparametrisation is commonly argued to be a key component of why NNs can be trained [ 20–22 ] . The Partition and Code ( PnC ) framework . Our main contribution is PnC , a framework for learning compression algorithms suitable for encoding graphs sampled from an underlying distribution . In the heart of PnC lie two ideas : ( a ) Learn to break the problem into parts . Rather than predicting directly the likelihood , we aim to learn how to decompose graphs into non-overlapping subgraphs , see Fig . 1 . ( b ) Identify and code recurring subgraphs when possible . We use a learned dictionary to code subgraphs that appear frequently , whereas rare subgraphs are encoded separately . The dictionary is restricted to contain only a finite number of small recurring subgraphs . This biases the model towards interpretable and well generalising solutions . Both the “ partition ” and “ code ” components of PnC are learned directly from the data , by optimising the total description length . Our framework provides a solution to the three challenges of graph compression . C1 : By constraining the dictionary to only contain graphs of size up to a small constant , we can efficiently solve GI . C2 : Graph partitioning provides us with the desirable decomposition of the distribution . Using appropriate parametrised probabilistic models , we obtain a closed-form expression for the likelihood that can be later used by any close-to-optimal entropy coder [ 23–25 ] . C3 : Since we use NNs to decompose the distribution ( and not to predict the likelihood ) , we can rely on overparametrisation without having to relay the NN parameters to the decoder . Also , the complexity of our learned hypothesis ( the dictionary ) can be computed , and thus optimised , during training . Theoretical results . Our analysis reveals that PnC can significantly improve upon less sophisticated graph encoders and justifies the usefulness of both the “ Partition ” and the “ Code ” component . Specifically , we prove that under mild conditions on the underlying graph distribution , PnC requires in expectation Θ ( n2 ) less bits than standard graph encodings , even if the latter are given access to an 2This follows by a simple counting argument : there are 2 ( n 2 ) labelled undirected graphs while the respective number for unlabelled graphs is asymptotically equal to 2 ( n 2 ) /n ! [ 7 ] . Thus , if all graphs of n vertices are equally probable , an encoding that does not consider isomorphism would sacrifice logn ! bits [ 8 , 9 ] . oracle that solves GI . Further , the dictionary induces additional savings of Θ ( n ) bits , with the gain being inversely proportional to the entropy of the distribution of the dictionary atoms . Thus , the more repetitive the patterns in the graph distribution are , the larger will be the compression benefits of PnC . Practical algorithms . We instantiate of our framework using the following algorithmic modules : ( a ) a low-parameter learnable estimate of the probability distribution , ( b ) learning to select the dictionary from a graph universe of finite size , and ( c ) learning to partition . The latter is a parametric randomised iterative algorithm , the probabilities of which are inferred from a Graph Neural Network ( GNN ) and optimised with reinforcement learning . Importantly , all algorithms can be jointly trained in order to minimise the total description length in a synergistic manner . We evaluate our framework on diverse real-world graph distributions and showcase compression gains with respect to both conventional and advanced baseline compressors , in observed and unseen data . 2 Related work . Engineered codecs . The majority of graph compressors are not probabilistic , but rely on handengineered encodings optimised to take advantage of domain-specific properties of e.g. , WebGraphs [ 4 ] , social networks [ 6 , 26 , 27 ] , and biological networks [ 28 ] . A common idea in these approaches is vertex reordering [ 4–6 , 26 , 27 , 29–31 ] , where the adjacency matrix is permuted in such a way that makes it “ compression-friendly ” for mainstream compressors of sequences , such as gzip . The algorithms identifying the re-orderings are usually based on heuristics taking advantage of specific network properties , e.g. , community structure . Another recurrent idea is to detect or use predefined frequent substructures ( e.g. , cliques ) to represent more efficiently different parts of the graph via grammar rules [ 32 ] . These approaches do not attempt to model the underlying graph distribution and thus to approach the storage lower bounds , but strive to find a balance between compression ratios and fast operations on the compressed graphs . Thus , despite their practical importance , they are less relevant to our work . A comprehensive survey can be found in [ 33 ] . Theory-driven approaches . Several works have contributed to the foundations of the information content and the complexity of graphs [ 8 , 9 , 34–40 ] . However , few works have attempted to model the underlying graph distribution . Perhaps the most outstanding progress has been made for graphs modelled by the Stochastic Block Model ( SBM ) [ 41–48 ] . Although originally invented for clustering and network analysis purposes , these approaches can be seamlessly used for compression due to their exact likelihood computation . In fact , as we argue in this work , virtually any graph clustering algorithm can be used successfully for compression , by defining codewords corresponding to a community-based random graph model . However , as our experiments confirm , such approaches are less effective at compressing graphs that do not contain clusters . Likelihood-based neural approaches . Any generative model that can provide likelihood estimates in a finite sample space can be used for lossless compression . As a result , a plethora of likelihoodbased neural compressors have been recently invented , ranging from autoregressive models for text [ 18 , 19 , 49 , 50 ] and images [ 11 , 12 ] to latent variable models [ 51–55 ] ( paired with bits-back coding [ 56 , 57 ] and Asymmetric Numeral Systems - ANS [ 25 ] ) , normalising flows [ 58–61 ] and most recently , diffusion-based generative models [ 62 , 63 ] . However , the vast majority of current graph generators lack the necessary theoretical properties an effective graph compressor should have : they compute probabilities on labelled graphs instead of isomorphism classes by resorting to a heuristic ordering [ 64–71 ] ( in general this will be suboptimal unless we canonicalise the graph/solve graph isomorphism , while different orderings will have non-zero probabilities , hence we will incur compression losses ) , and/or do not provide a likelihood [ 72–76 ] . An important caveat , which is often ignored in the literature , is that when parametrising the distribution with a neural network , the data can not be recovered unless the decoder has access to the network itself . Hence it is necessary to account for the NN ’ s description length when evaluating compression gains . However , generative models are usually parameter inefficient , while compressing them ( especially during training ) is a challenging problem [ 77 ] . Note that this is significantly different than compressing neural classifiers , since the capacity to infer the likelihood up to high precision needs to be retained . Therefore , these approaches can have diminishing ( or even negative ) returns when the dataset size is not large enough . In contrast , by optimising the total description length ( equivalent to maximum a-posteriori ) , we design a compressor that is practical even for small datasets , while the learned dictionary makes our compressor interpretable . Other related work includes using compression objectives paired with heuristic algorithms for downstream tasks , such as motif finding [ 78 , 79 ] and graph summarisation [ 80 , 81 ] , lossy compression/coarsening [ 82–87 ] , and graph dictionary learning in the context of sparse coding [ 88 , 89 ] . 3 Preliminaries . We use capital letters for sets A , B , bold font for vectors x and matrices X , and calligraphic font X for families of sets . We express the information content in bits by using a base-2 logarithm log . Graphs . Let G = { V , E } be a graph with n vertices V = { v1 , . . . , vn } and m edges , where eij ∈ E whenever vertices vi , vj ∈ V are connected by an edge . For simplicity , we assume the graphs to be undirected , though the same methods apply to directed graphs as well with minimal modifications . Let cut ( A , B ) = { eij ∈ E | vi ∈ A and vj ∈ B } denote the set of edges with endpoints at two different vertex sets A , B ⊆ V . Information theory . Following the usual terminology in information theory and Minimum Description Length ( MDL ) theory [ 2 , 90 , 91 ] , we assume an observation space G ( in our case a space of possible graphs ) , and a probability distribution p ( sometimes referred to as probabilistic source ) producing samples from the observation space . We observe a dataset G = { G1 , G2 , . . . , G|G| } of i.i.d . observations drawn from p. Note that this setting is in contrast with most works on graph compression [ 4 , 30 , 5 , 6 ] , where the target is to compress a single large network , such as a social network or a web graph . Let a description method or symbol code CODE : G→ { 0 , 1 } ∗ be a mapping from the observation space to a variable-length sequence of binary symbols , the output of which is a codeword . In the context of graph compression , we are interested in the description length of the code LCODE ( G ) , or L ( G ) for brevity , i.e. , the number of bits needed to encode the graph G , rather than the code itself , given a single requirement : the code needs to be uniquely decodable , meaning that any concatenation of codes can be uniquely mapped to a sequence of observations . This property can be easily verified using only the description lengths by the well-known Kraft–McMillan inequality ( l.h.s . in formula ( 1 ) ) , an important implication of which is that every uniquely decodable code implies a probability distribution q ( G ) ( r.h.s ) : ∑ G∈G 2−L ( G ) ≤ 1 ( code ) ⇐⇒ q ( G ) = 2 −L ( G ) ∑ G∈G 2 −L ( G ) ( distribution ) ( 1 ) This equivalence allows us to always define underlying probabilistic models when working directly with description lengths . When the Kraft–McMillan inequality holds with equality , we say that the code is complete ; the case of strict inequality means the code is redundant . Moreover , in the case of complete codes , the expected length of the code corresponds to the distribution entropy Hq [ G ] ( plus a constant term when the code is redundant ) . Thus , compression is synonymous to defining probabilistic models with the lowest possible entropy . An important notion that frequently appears in our theoretical analysis is the binary entropy , i.e. , the entropy of a bernoulli variable with probability of success p. To distinguish it from the general notion of entropy we will denote it as H ( p ) = −p log p− ( 1− p ) log ( 1− p ) . It holds that 0 ≤ H ( p ) ≤ 1 , where the l.h.s equality is satisfied for p ∈ { 0 , 1 } and the r.h.s . for p = 1/2 . Common graph encodings . An important principle that we follow is that whenever we can not make any assumptions about an underlying distribution , or modelling it is impractical , then the uniform distribution punif is chosen for encoding . The reason is that uniform distribution is worstcase optimal [ 90 ] : for any non-uniform unknown distribution p , there always exists a distribution q the corresponding encoding of which will be on expectation worse than the uniform encoding : Ex∼p [ − log q ( x ) ] > Ex∼p [ − log punif ( x ) ] . In the context of graphs , when we can not make any assumptions or when enumeration is impossible , we will be using a slightly more informative distribution : the Erdős–Rényi ( ER ) random graph model . This model assigns equal probability to all labelled graphs with n vertices and m edges . Assuming a uniform probability over the possible number of vertices n and number of edges m given n , we get : Lnull ( G ) = log ( nmax + 1 ) + log ( ( n 2 ) + 1 ) + log ( ( n 2 ) m ) , ( 2 ) where nmax is an upper bound on the number of vertices . The “ null ” encoding compresses more efficiently graphs that are either very sparse or very dense ( low-surprise ) as the number of possible graphs with m edges is maximised when m = n ( n− 1 ) /4 .

Summary: This article presents a procedure to compress graphs (into a code). It takes the principle of entropy minimization i.e. finding a code that has the lowest possible entropy. In order to do so three steps are considered: 1) partioning the graph into subgraphs ($\operatorname{Part}_{\theta}$ which is parametrized by a NN) 2) finding a dictionary that faithfully describes a prior on the subgraphs structures, and that will be used to encode the different subgraphs coming from PART 3) Graph encoding part: assign a (parametrized) probability to each subgraph (which depends of the dictionary also). From a probability we can derive the length of the code. Overall this defines a description length which can be minimized in order to have the shortest code possible.