Partition and Code: learning how to compress graphs

1 Introduction . Lossless data compression has been one of the most fundamental and long-standing problems in computer science . It is by now well-understood that the intrinsic limits of compression are governed by the entropy of the underlying data distribution [ 1 ] . Crucially , these limits expose an intimate connection between compressibility and machine learning : the better one models the underlying data distribution ( from limited observations ) the more bits can be saved and vice-versa [ 2 ] . The compression of ordered data such as text , images , or video , underpins the modern technology from web protocols to video streaming . However , graph-structured data remain a notable exception . As graph data are becoming more prevalent , it becomes increasingly important to invent practical ways to encode them parsimoniously . There are three main challenges one faces when attempting to compress graphs : ∗This work was done while Giorgos Bouritsas was visiting Dr. Andreas Loukas at EPFL , Lausanne . 1The source code is publicly available at https : //github.com/gbouritsas/PnC 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . C1 . Dealing with graph isomorphism ( GI ) . A key difficulty that distinguishes graphs from conventional data lies in the absence of an inherent ordering of the graph vertices . In order to be able to approach the storage lower bounds , isomorphic graphs should be encoded with the same codeword . However , since the complexity of known algorithms for GI is super-polynomial on the number of vertices [ 3 ] , a direct use of GI is impractical for graphs consisting of more than a few hundred vertices . Indeed , an examination of the graph compression literature reveals that most progress has been made by optimising a vertex ordering and adapting methods originally invented for vectored data [ 4–6 ] . Unfortunately , naively encoding graphs as vectors results in a significant loss in compression2 . C2 . Evaluating the likelihood . An optimal encoder [ 1 ] requires one to accurately estimate and evaluate the probabilities of all the possible outcomes of the underlying domain . When dealing with high-dimensional data , this can be addressed by partitioning the data into parts to obtain a decomposition of the probability distribution : e.g. , images can be compressed by modelling the distribution of pixels or patches [ 10–12 ] , and text by focusing on characters or n-grams [ 13–19 ] . However , since graphs do not admit an efficiently computable canonical ordering , it is unclear what decomposition one should employ . C3 . Accounting for the description length of the learned model . The classical learning theory trade-off between model complexity and generalisation is of paramount concern for effective compression . Though in typical deep learning applications one can aim to model the data distribution with an overparametrised neural network ( NN ) that generalises well , utilising such models to compress information is problematic : since decoding is impossible unless the decoder also receives the learned model ( i.e. , the NN parameters ) , overparametrised models are , by definition , suboptimal . This is a pertinent issue for likelihood-based neural approaches as overparametrisation is commonly argued to be a key component of why NNs can be trained [ 20–22 ] . The Partition and Code ( PnC ) framework . Our main contribution is PnC , a framework for learning compression algorithms suitable for encoding graphs sampled from an underlying distribution . In the heart of PnC lie two ideas : ( a ) Learn to break the problem into parts . Rather than predicting directly the likelihood , we aim to learn how to decompose graphs into non-overlapping subgraphs , see Fig . 1 . ( b ) Identify and code recurring subgraphs when possible . We use a learned dictionary to code subgraphs that appear frequently , whereas rare subgraphs are encoded separately . The dictionary is restricted to contain only a finite number of small recurring subgraphs . This biases the model towards interpretable and well generalising solutions . Both the “ partition ” and “ code ” components of PnC are learned directly from the data , by optimising the total description length . Our framework provides a solution to the three challenges of graph compression . C1 : By constraining the dictionary to only contain graphs of size up to a small constant , we can efficiently solve GI . C2 : Graph partitioning provides us with the desirable decomposition of the distribution . Using appropriate parametrised probabilistic models , we obtain a closed-form expression for the likelihood that can be later used by any close-to-optimal entropy coder [ 23–25 ] . C3 : Since we use NNs to decompose the distribution ( and not to predict the likelihood ) , we can rely on overparametrisation without having to relay the NN parameters to the decoder . Also , the complexity of our learned hypothesis ( the dictionary ) can be computed , and thus optimised , during training . Theoretical results . Our analysis reveals that PnC can significantly improve upon less sophisticated graph encoders and justifies the usefulness of both the “ Partition ” and the “ Code ” component . Specifically , we prove that under mild conditions on the underlying graph distribution , PnC requires in expectation Θ ( n2 ) less bits than standard graph encodings , even if the latter are given access to an 2This follows by a simple counting argument : there are 2 ( n 2 ) labelled undirected graphs while the respective number for unlabelled graphs is asymptotically equal to 2 ( n 2 ) /n ! [ 7 ] . Thus , if all graphs of n vertices are equally probable , an encoding that does not consider isomorphism would sacrifice logn ! bits [ 8 , 9 ] . oracle that solves GI . Further , the dictionary induces additional savings of Θ ( n ) bits , with the gain being inversely proportional to the entropy of the distribution of the dictionary atoms . Thus , the more repetitive the patterns in the graph distribution are , the larger will be the compression benefits of PnC . Practical algorithms . We instantiate of our framework using the following algorithmic modules : ( a ) a low-parameter learnable estimate of the probability distribution , ( b ) learning to select the dictionary from a graph universe of finite size , and ( c ) learning to partition . The latter is a parametric randomised iterative algorithm , the probabilities of which are inferred from a Graph Neural Network ( GNN ) and optimised with reinforcement learning . Importantly , all algorithms can be jointly trained in order to minimise the total description length in a synergistic manner . We evaluate our framework on diverse real-world graph distributions and showcase compression gains with respect to both conventional and advanced baseline compressors , in observed and unseen data . 2 Related work . Engineered codecs . The majority of graph compressors are not probabilistic , but rely on handengineered encodings optimised to take advantage of domain-specific properties of e.g. , WebGraphs [ 4 ] , social networks [ 6 , 26 , 27 ] , and biological networks [ 28 ] . A common idea in these approaches is vertex reordering [ 4–6 , 26 , 27 , 29–31 ] , where the adjacency matrix is permuted in such a way that makes it “ compression-friendly ” for mainstream compressors of sequences , such as gzip . The algorithms identifying the re-orderings are usually based on heuristics taking advantage of specific network properties , e.g. , community structure . Another recurrent idea is to detect or use predefined frequent substructures ( e.g. , cliques ) to represent more efficiently different parts of the graph via grammar rules [ 32 ] . These approaches do not attempt to model the underlying graph distribution and thus to approach the storage lower bounds , but strive to find a balance between compression ratios and fast operations on the compressed graphs . Thus , despite their practical importance , they are less relevant to our work . A comprehensive survey can be found in [ 33 ] . Theory-driven approaches . Several works have contributed to the foundations of the information content and the complexity of graphs [ 8 , 9 , 34–40 ] . However , few works have attempted to model the underlying graph distribution . Perhaps the most outstanding progress has been made for graphs modelled by the Stochastic Block Model ( SBM ) [ 41–48 ] . Although originally invented for clustering and network analysis purposes , these approaches can be seamlessly used for compression due to their exact likelihood computation . In fact , as we argue in this work , virtually any graph clustering algorithm can be used successfully for compression , by defining codewords corresponding to a community-based random graph model . However , as our experiments confirm , such approaches are less effective at compressing graphs that do not contain clusters . Likelihood-based neural approaches . Any generative model that can provide likelihood estimates in a finite sample space can be used for lossless compression . As a result , a plethora of likelihoodbased neural compressors have been recently invented , ranging from autoregressive models for text [ 18 , 19 , 49 , 50 ] and images [ 11 , 12 ] to latent variable models [ 51–55 ] ( paired with bits-back coding [ 56 , 57 ] and Asymmetric Numeral Systems - ANS [ 25 ] ) , normalising flows [ 58–61 ] and most recently , diffusion-based generative models [ 62 , 63 ] . However , the vast majority of current graph generators lack the necessary theoretical properties an effective graph compressor should have : they compute probabilities on labelled graphs instead of isomorphism classes by resorting to a heuristic ordering [ 64–71 ] ( in general this will be suboptimal unless we canonicalise the graph/solve graph isomorphism , while different orderings will have non-zero probabilities , hence we will incur compression losses ) , and/or do not provide a likelihood [ 72–76 ] . An important caveat , which is often ignored in the literature , is that when parametrising the distribution with a neural network , the data can not be recovered unless the decoder has access to the network itself . Hence it is necessary to account for the NN ’ s description length when evaluating compression gains . However , generative models are usually parameter inefficient , while compressing them ( especially during training ) is a challenging problem [ 77 ] . Note that this is significantly different than compressing neural classifiers , since the capacity to infer the likelihood up to high precision needs to be retained . Therefore , these approaches can have diminishing ( or even negative ) returns when the dataset size is not large enough . In contrast , by optimising the total description length ( equivalent to maximum a-posteriori ) , we design a compressor that is practical even for small datasets , while the learned dictionary makes our compressor interpretable . Other related work includes using compression objectives paired with heuristic algorithms for downstream tasks , such as motif finding [ 78 , 79 ] and graph summarisation [ 80 , 81 ] , lossy compression/coarsening [ 82–87 ] , and graph dictionary learning in the context of sparse coding [ 88 , 89 ] . 3 Preliminaries . We use capital letters for sets A , B , bold font for vectors x and matrices X , and calligraphic font X for families of sets . We express the information content in bits by using a base-2 logarithm log . Graphs . Let G = { V , E } be a graph with n vertices V = { v1 , . . . , vn } and m edges , where eij ∈ E whenever vertices vi , vj ∈ V are connected by an edge . For simplicity , we assume the graphs to be undirected , though the same methods apply to directed graphs as well with minimal modifications . Let cut ( A , B ) = { eij ∈ E | vi ∈ A and vj ∈ B } denote the set of edges with endpoints at two different vertex sets A , B ⊆ V . Information theory . Following the usual terminology in information theory and Minimum Description Length ( MDL ) theory [ 2 , 90 , 91 ] , we assume an observation space G ( in our case a space of possible graphs ) , and a probability distribution p ( sometimes referred to as probabilistic source ) producing samples from the observation space . We observe a dataset G = { G1 , G2 , . . . , G|G| } of i.i.d . observations drawn from p. Note that this setting is in contrast with most works on graph compression [ 4 , 30 , 5 , 6 ] , where the target is to compress a single large network , such as a social network or a web graph . Let a description method or symbol code CODE : G→ { 0 , 1 } ∗ be a mapping from the observation space to a variable-length sequence of binary symbols , the output of which is a codeword . In the context of graph compression , we are interested in the description length of the code LCODE ( G ) , or L ( G ) for brevity , i.e. , the number of bits needed to encode the graph G , rather than the code itself , given a single requirement : the code needs to be uniquely decodable , meaning that any concatenation of codes can be uniquely mapped to a sequence of observations . This property can be easily verified using only the description lengths by the well-known Kraft–McMillan inequality ( l.h.s . in formula ( 1 ) ) , an important implication of which is that every uniquely decodable code implies a probability distribution q ( G ) ( r.h.s ) : ∑ G∈G 2−L ( G ) ≤ 1 ( code ) ⇐⇒ q ( G ) = 2 −L ( G ) ∑ G∈G 2 −L ( G ) ( distribution ) ( 1 ) This equivalence allows us to always define underlying probabilistic models when working directly with description lengths . When the Kraft–McMillan inequality holds with equality , we say that the code is complete ; the case of strict inequality means the code is redundant . Moreover , in the case of complete codes , the expected length of the code corresponds to the distribution entropy Hq [ G ] ( plus a constant term when the code is redundant ) . Thus , compression is synonymous to defining probabilistic models with the lowest possible entropy . An important notion that frequently appears in our theoretical analysis is the binary entropy , i.e. , the entropy of a bernoulli variable with probability of success p. To distinguish it from the general notion of entropy we will denote it as H ( p ) = −p log p− ( 1− p ) log ( 1− p ) . It holds that 0 ≤ H ( p ) ≤ 1 , where the l.h.s equality is satisfied for p ∈ { 0 , 1 } and the r.h.s . for p = 1/2 . Common graph encodings . An important principle that we follow is that whenever we can not make any assumptions about an underlying distribution , or modelling it is impractical , then the uniform distribution punif is chosen for encoding . The reason is that uniform distribution is worstcase optimal [ 90 ] : for any non-uniform unknown distribution p , there always exists a distribution q the corresponding encoding of which will be on expectation worse than the uniform encoding : Ex∼p [ − log q ( x ) ] > Ex∼p [ − log punif ( x ) ] . In the context of graphs , when we can not make any assumptions or when enumeration is impossible , we will be using a slightly more informative distribution : the Erdős–Rényi ( ER ) random graph model . This model assigns equal probability to all labelled graphs with n vertices and m edges . Assuming a uniform probability over the possible number of vertices n and number of edges m given n , we get : Lnull ( G ) = log ( nmax + 1 ) + log ( ( n 2 ) + 1 ) + log ( ( n 2 ) m ) , ( 2 ) where nmax is an upper bound on the number of vertices . The “ null ” encoding compresses more efficiently graphs that are either very sparse or very dense ( low-surprise ) as the number of possible graphs with m edges is maximised when m = n ( n− 1 ) /4 .

Summary: The authors establish a theoretically well-motivated, general-purpose graph compression framework they call Partition-and-Code (PnC). The framework consists of 3 general parts: partitioning, creating a dictionary of common subgraphs and entropy coding. Crucially, the first two steps are used to map a graph to its isomorphism class, and it is in fact the isomorphism of the input graph that is encoded using entropy coding.  The compression pipeline, given a graph partitioning function and a subgraph dictionary, works as follows: 1) The graph to be encoded is split up into two types of subgraphs: atoms (subgraphs of a fixed size, such that they should be likely to appear in the dictionary), and cuts, which connect the atoms. 2) Then, atoms appearing in the dictionary are encoded using a distribution tailored to the dictionary. Cuts and atoms missing from the dictionary are coded using an appropriate uniform distribution.  Importantly, ML techniques are only used as heuristics for the partitioning part of the encoding process, which means that the decoder does not need to know the model parameters. The authors propose various proof-of-concept ways to learn the partitioning function using an e.g. GNN and propose a relaxed, greedy gradient-based way to learn the atom dictionary. Learning of all parts of the encoder is performed by minimizing an appropriately defined minimum description length objective.  The authors perform experiments on multiple popular graph datasets and achieve good results.