Directed graph compression

In one embodiment of the present disclosure, an original graph including nodes is obtained. The nodes of the original graph are reordered to generate a reordered graph. Non-zero elements in an adjacency matrix for the reordered graph are clustered as compared with an adjacency matrix for the original graph. The adjacency matrix for the reordered graph is encoded with integers. The integers correspond to non-empty blocks in the adjacency matrix for the reordered graph.

BACKGROUND

The present invention relates to data processing, and more specifically to directed graph compression.

Graphs are becoming increasingly important for numerous applications, ranging across the domains of World Wide Web, social networks, bioinformatics, computer security, and many others. Many graphs are directed, such as Web graph and Twitter social graph. In general, a directed graph is a graph that is a set of vertices connected by edges, and the edges have a direction associated with them.

SUMMARY

According to one embodiment of the present invention, there is provided a method for processing a graph. In this method, an original graph including a plurality of nodes is obtained. The nodes of the original graph are reordered to generate a reordered graph, wherein the non-zero elements in an adjacency matrix for the reordered graph are clustered as compared with an adjacency matrix for the original graph. The adjacency matrix for the reordered graph is encoded with a plurality of integers, wherein each of the plurality of integers represents corresponding non-empty block in the adjacency matrix for the reordered graph, and the non-empty blocks include at least one non-zero element.

According to one embodiment of the present invention, there is provided a system for processing a graph. The system comprises one or more processors, a memory coupled to at least one of the processors, and a set of computer program instructions stored in the memory and executed by at least one of the processors in order to perform a method. In this method, an original graph including a plurality of nodes is obtained. The nodes of the original graph are reordered to generate a reordered graph, wherein the non-zero elements in an adjacency matrix for the reordered graph are clustered as compared with an adjacency matrix for the original graph. The adjacency matrix for the reordered graph is encoded with a plurality of integers, wherein each of the plurality of integers represent corresponding non-empty block in the adjacency matrix for the reordered graph, and the non-empty blocks include at least one non-zero element.

According to another embodiment of the present invention, there is provided a computer program product for processing a graph. The computer program product comprises a computer readable storage medium having program instructions embodied therewith. The program instructions are readable by a device to cause the device to perform a method for processing a graph. In this method, an original graph including a plurality of nodes is obtained. The nodes of the original graph are reordered to generate a reordered graph, wherein the non-zero elements in an adjacency matrix for the reordered graph are clustered as compared with an adjacency matrix for the original graph. The adjacency matrix for the reordered graph is encoded with a plurality of integers, wherein each of the plurality of integers represent corresponding non-empty block in the adjacency matrix for the reordered graph, and the non-empty blocks include at least one non-zero element.

DETAILED DESCRIPTION

Embodiments of the present disclosure recognize that the growing scale of directed graphs has made efficient execution of graph computation very challenging. Embodiments of the present disclosure will be described in detail with reference to the accompanying drawings, in which the embodiments of the present disclosure have been illustrated. The present disclosure can be implemented in various manners and thus should not be construed to be limited to the embodiments disclosed herein.

Referring now toFIG. 1, in which an exemplary computer system/server12which is applicable to implement the embodiments of the present invention is shown.FIG. 1is also adapted to depict an illustrative example of a portable electronic device such as a communication device which is applicable to implement the embodiments of the present invention. Computer system/server12is only illustrative and is not intended to suggest any limitation as to the scope of use or functionality of embodiments of the invention described herein.

Computer system/server12may include a variety of computer system readable media. Such media may be any available media that is accessible by computer system/server12, and it includes both volatile and non-volatile media, removable and non-removable media.

Reducing graph size to fit in memory, for example by using the technique of lossless compression, is crucial in cutting the cost of large scale graph computation. And current graph compression work still suffers from issues including low compression ratio and high decompression overhead.

In the present disclosure, a novel solution for compressing graphs are proposed here. In this solution, an effective clustering algorithm is first performed and then the resulting adjacency matrix is represented by lists of encoded numbers. In this way, this approach can greatly reduce the graph size.

With reference now toFIG. 2, a method according to one embodiment of the present disclosure will be described.

In Step S210, an original graph including a plurality of nodes may be obtained. A node ID may be assigned to each node in the graph to identify that node.

According to one embodiment of the present disclosure, an adjacency matrix M may be built for the graph in which the elements in the matrix indicate the adjacency relationship between two nodes. For example, the elements may indicate whether or not there exists an edge between two nodes.

According to one embodiment of the present disclosure, the graph may be a directed graph, and elements in the adjacency matrix may indicate whether or not a directed edge exists from one node to another node. For example, the matrix may be a binary matrix. And the binary element ei,jin the i-th row and j-th column may indicate whether or not there exists a directed edge from the i-th node to j-th node (where i and j denote node IDs). For example, 1-element indicates that there exists an edge while 0-element indicates no edge.

According to one embodiment, the value of an element in the matrix is not a binary value. For example, different values of elements may represent different weights of an edge. In this case, a non-zero element in the adjacency matrix would indicate a specific adjacency relationship between two nodes. Alternatively, the weights of the edges may be saved in a different matrix and the adjacency matrix is used to indicate the existence of an edge. In this case, each element of the matrix would have a binary value 0 or 1. In the following, the invention will be discussed with the value of an element being a binary value; however, it should be understood that this is for the purpose of simplified illustration and will not thereby limit the scope of the invention.

In Step S220, the nodes of the original graph may be reordered to generate a reordered graph. Non-zero elements in the adjacency matrix for the reordered graph may be clustered as compared with the adjacency matrix for the original graph. Having the non-zero elements in the matrix clustered reduces the graph space cost for storing a graph.

For example, when dividing the adjacency matrix M to a plurality of blocks of a×b elements, it may be determined whether or not a block is empty. In one embodiment, if all of the a×b elements in a block are zeros, then the block is empty, and if a block includes at least one non-zero element, then the block is a non-empty block. As a result, graph space cost can be measured by counting the non-empty blocks. With the reordering step, the non-zero elements are clustered, and the number of non-empty blocks are minimized, thereby reducing graph space cost.

In Step S230, the adjacency matrix for the reordered graph may be encoded with a plurality of integers. The integers may represent the non-empty blocks in the adjacency matrix for the reordered graph. As described above, a non-empty block includes at least one non-zero element. With the encoding step, the space required for storing the graph is further reduced.

With reference now toFIG. 3, one embodiment of the process of reordering graph nodes (Step S220inFIG. 2) will be described in further detail.

According to one embodiment of the present disclosure, to reduce graph space cost, the nodes of the original graph may be reordered, making the rows and columns in the corresponding adjacency matrix permuted, such that the non-zero elements in the permuted matrix are clustered.

First, an empty list P may be initialized. Empty list P may be used for saving the permutation of the new node ID (the node ID in P is the new node ID).

In Step S310, node degrees may be determined for the nodes in the original graph. According to one embodiment of the present disclosure, the node degree of a node indicates the number of neighbors of the node. The node degree may be in-degree or out-degree. The in-degree refers to the degree to which a node is pointed to by other nodes, that is, the number of other nodes pointing to the particular node. The out-degree refers to the degree to which a node points to other nodes, that is, the number of other nodes to which the particular node points.

According to one embodiment of the present disclosure, the in-degree and out-degree of a node may be obtained according to the elements in the adjacency matrix. For example,FIG. 4(a)-4(d)show an example of reordering graph nodes according to one embodiment of the present disclosure.FIG. 4(a)shows the original graph and a corresponding adjacency matrix.FIG. 4(b)shows that the nodes are sorted in descending order according to the in-degree and out-degree of the nodes, and the top k number of nodes are determined according to in-degree and out-degree. As shown inFIG. 4(b), the in-degree of Node 0 is 2 since there are 2 1-elements in column ID 0, and the out-degree of Node 0 is 2 since there are 2 1-elements in row ID 0.

In Step S320, a set of candidate nodes in the original graph may be selected. According to one embodiment of the present disclosure, the set of candidate nodes may be the top k number of nodes selected according to their node degrees. The number k may be an integer parameter which is equal to or less than the total number of nodes in the graph.

After obtaining the in-degree and out-degree of the nodes in the graph, the nodes may be sorted in a descending order according to the in-degrees and/or out-degrees. Various sorting criteria may be applied. For example, the node with highest in-degree may be selected first. As another example, the node with highest out-degree may be selected first. Alternatively, the node with highest node degree which includes both in-degree and out-degree would be selected first. In the following, the invention will be discussed with the nodes sorted in a descending order according to in-degree and out-degree; however, it should be understood that this is only for the purpose of simplified illustration and will not thereby limit the scope of the invention.

According to one embodiment of the present disclosure, after the nodes are sorted in descending order according to in-degree and out-degree, the k highest in-degree and out-degree nodes may be selected, as in-degree candidate set L1and out-degree candidate set L2respectively. According to one embodiment of the present disclosure, if the top k nodes of L1and L2are same, the k candidate nodes may be obtained accordingly. According to one embodiment of the present disclosure, if the top k nodes of L1and L2are different, then the common nodes may be selected first. The rest of the nodes may be selected from L1or L2, or from both of them, according to the in-degree and out-degree, and added to the list P.

As an example, inFIG. 4(b), k=2 and Nodes 4 and 0 are both top 2 in-degree nodes and top 2 out-degree nodes. Therefore, Nodes 4 and 0 are selected as candidate nodes.

In Step S330, the order of the candidate nodes and their neighbor nodes may be determined based on common neighbor information of the nodes. The common neighbor information of the nodes may refer to the number of common neighbor nodes of two nodes. The number may be the number of common in-neighbor nodes or the number of common out-neighbor nodes of the two nodes. For example,FIG. 4(c)shows an example of common neighbor information of nodes in the original graph.

As shown inFIG. 4(c), Nodes 0 and 6 have a common in-neighbor, Node 4. Nodes 0 and 3 have a common out-neighbor, Node 1. The number of common neighbors may also be the total number of both common in-neighbors and out-neighbors or any other proper information.

According to one embodiment of the present disclosure, the common neighbor information of nodes may be determined from the graph or the corresponding adjacency matrix. The nodes with more common neighbors with other nodes would be put in front of those with less common neighbors.

According to one embodiment of the present disclosure, all of the k candidate nodes and the in-neighbors and out-neighbors of the k candidate nodes are added into a node set N. The node with the highest number of common neighbors may be determined and added into list P. The process of determining the node with highest number of common neighbors may be repeated until all of the nodes in node set N have been processed and added into the list P.

According to one embodiment of the present disclosure, if the parameter k equals the total number of nodes in the graph, then all of the nodes will be chosen in Step S320and processed in Step S330. If the parameter k is smaller than the total number of nodes in the graph, then after the order of the k candidate nodes and their neighbor nodes has been adjusted based on common neighbor information of the nodes k candidates in Step S330, the process may go back to S320for the rest of the nodes in the graph. Steps S320and S330may then be repeated until all the nodes in the graph have been processed.

FIG. 4(d)shows a reordered graph as well as its adjacency matrix. For the adjacency matrices of the original graph and the reordered graph, if the graphs are divided into 2×2 elements, there are nine non-empty blocks inFIG. 4(d). Since the nine non-empty blocks are fewer than the eleven non-empty blocks inFIG. 4(a), this means that the non-zero elements in the matrix has been clustered.

InFIG. 4(a)-(d), a simple graph is used as an example to illustrate the clustering process. The clustering method according to the embodiments of the present disclosure would be useful particularly for the real world directed graphs. The real world directed graphs typically exhibit power law degree distribution. For example, for the hub nodes with high in-degrees in real world directed graphs, due to the power law in-degree distribution in such graphs, few hub nodes are with a large amount of in-coming edges, indicating very high in-degree. And the majority of nodes have low in-degrees. Therefore, for two hub nodes pointed by a large amount of spoke neighbors, it is not rare that such hub nodes share many common spoke neighbors. If the similarity of such spokes is high, we would like to permute the hub nodes together in the matrix columns. Meanwhile, if two spoke neighbors share common hubs, the similarity of such spokes is high and they may also put together in the matrix rows.

Further, real world directed graphs also follow power-law out-degree distribution, i.e., few hub nodes are with a very large amount of out-going edges (very high out-degrees), and the majority of nodes are with low out-degrees. Therefore, for two hub nodes with high out-degrees, if they share many spoke neighbors, the hub vertices may be placed together in the matrix rows. Meanwhile, if two spoke nodes share many incoming hub neighbors, the spoke nodes would be placed together in the matrix columns.

With reference now toFIG. 5, the process of encoding the adjacency matrix according to one embodiment of the present disclosure (Step S230inFIG. 2) will be described in detail.

In Step S510, the adjacency matrix for the reordered graph may be divided into a plurality of blocks. At least one block may have more than one binary elements.

According to one embodiment of the present disclosure, the matrix may be divided into a plurality of blocks with same size, such as blocks with b×b elements or blocks with a×b elements. According to another embodiment of the present disclosure, the blocks may have difference size. For example, some blocks may have a×b elements, some blocks may have c×d elements, etc.

As described, if all elements in a block are zeros, then the block is an empty block. If the block includes at least one non-zero element, then the block is a non-empty block. The graph space cost may be measured by counting the non-empty blocks. According to one embodiment of the present disclosure, the nearby non-zero elements may be grouped into blocks so as to have fewer blocks, which will further reduce graph space cost.

In Step S520, the binary elements in the non-empty block may be represented as at least one integer, and the binary elements in the non-empty block may be treated as the binary form of the at least one integer.

With the step S520, the non-empty blocks in the matrix may be represented as a plurality of integers and the plurality of integers may be maintained instead of the elements in the blocks.

In the following, embodiments of the present disclosure will be detailed described with reference toFIG. 6(a)-6(c)which show encoding examples according to the embodiments of the present disclosure. The examples described are for the purpose of simplified illustration and the scope of the invention is not thus limited. InFIG. 6(a), the matrix is divided into blocks consisting of 2×2 elements. The block consisting of 2×2 elements is used as an example and the block may comprise any number of elements as appropriate.

A directory may be utilized to maintain matrix row IDs. The directory may contain the associated matrix row IDs where the non-empty blocks are located in the matrix. For example, inFIG. 6(a), since the matrix is divided into 2×2 blocks, the directory contains two row IDs when the each of such IDs is with at least one non-empty block. Among the 8 row IDs (from 0 to 7), there are four total set of numbers in the directory: {0, 1}, {2, 3}, {4, 5}, and {6, 7}.

Next, according to one embodiment of the present disclosure, to encode a non-empty block, we may use two set of numbers. The first number in the first set is left-most column ID (startColumn) of the first row in this block. The first number in the second set is the leftmost column ID (startColumn) of the second row in this block. Next, by treating the binary elements inside the rows of the block as the binary form of an integer, we can use the integer number to represent each row in the block. The two set of numbers would be {<startColumn, firstRowCoding>, <startColumn, secondRowCoding>.

For example, for the left non-empty block in the row IDs 0 and 1 inFIG. 6(a), the leftmost column ID is 0. The binary element in the row 0 is 01, which are encoded to be an integer 1. The binary element in the row 1 is 00, which are encoded to be an integer 0. Thus, a set of integer pairs {<0, 1>, <0, 0>} is used to encode the block. Similarly, the second block in row IDs 0 and 1 is represented as {<2, 2>, <2, 3>}.

Similar situation holds for other non-empty blocks. The right part inFIG. 6(a)gives the directory and lists of integer pairs to encode the graph inFIG. 6(a).

According to another embodiment of the present disclosure, the binary elements inside the whole block may be treated as the binary form of an integer, as shown inFIG. 6(b). For each non-empty block, one set of numbers will be used. The first number in the set would be leftmost column ID (startColumn) of the first row in this block. The second number in the set would be an integer by treating the binary elements inside the whole block as the binary form of the integer. The set of numbers would be {startColumn, wholeBlockCoding}.

Use the matrix inFIG. 6(b)as an example. The first number is the left-most column ID of the block. For the left non-empty block in the row IDs 0 and 1, the leftmost column ID is 0; the 2×2=4 binary elements 0100 are encoded to be an integer 4. Thus, we use an integer pair {0, 4} to encode the block. A similar situation holds for other non-empty blocks. In the second block in row IDs 0 and 1, the binary element is 1011, which would be the integer 11. Therefore, the second block in row IDs 0 and 1 is represented as {2, 11}.

FIG. 6(c)show another encoding example according to the embodiments of the present disclosure. InFIG. 6(c), the matrix is divided into blocks consisting of a plurality of elements. The nearby non-zero elements are grouped into blocks so as to have fewer blocks. For example, inFIG. 6(c), six blocks are obtained from the matrix, in which three blocks include eight elements and three blocks include four elements.

Similarly, a directory is used to maintain the matrix row IDs. Among the 8 row IDs (from 0 to 7), there are four total elements in the directory: {0, 1}, {2, 3}, {4, 5}, and {6, 7}.

Next, to encode a non-empty block, two sets of numbers are used. According to one embodiment of the present disclosure, the first number in the first set is leftmost column ID (startColumn) of the first row in this block. The first number in the second set is the right-most column ID (endColumn) of the first row in this block. Next, by treating the binary elements inside the rows of the block as the binary form of an integer, the integer number can be used to represent each row in the block. The two sets of numbers would be {<startColumn, firstRowCoding>} and {<endColumn, secondRowCoding>}.

For example, for the left non-empty block in the row IDs 0 and 1, its leftmost column ID is 0 and its rightmost column ID is 3. The binary elements in the row 0 is 0110, which are encoded to be an integer 6. The binary elements in the row 1 is 0011, which are encoded to be the integer 3. Thus, we use a set of integer pairs {<0, 6>, <3, 3>} is used to encode the block. Similarly, the second block in row IDs 2 and 3 is represented as {<4, 8>, <7, 1>}.

The integer pairs shown inFIG. 6(a)-6(c)are for the purpose of simplified illustration. Other formats could also be used to represent non-empty blocks, for example, in the format of {<startColumn, {firstRowCoding, secondRowCoding, . . . ,} etc. Further, the embodiments are described by using binary matrix with binary elements. However, the elements in the matrix may also be represented by any other appropriate radix, such as octal, hexadecimal, and the like.

With the encoding methods, according to various embodiments of the present disclosure, the decoding overhead from the encoded number to original binary elements would be trivial.

According to one embodiment of the present invention, there is provided a system for processing a graph. The system comprises one or more processors, a memory coupled to at least one of the processors, and a set of computer program instructions stored in the memory and executed by at least one of the processors in order to perform a method. In this method, an original graph including a plurality of nodes is obtained. The nodes of the original graph are reordered to generate a reordered graph, wherein the non-zero elements in an adjacency matrix for the reordered graph are clustered as compared with an adjacency matrix for the original graph. The adjacency matrix for the reordered graph is encoded with a plurality of integers, wherein each of the plurality of integers represents a corresponding non-empty block in the adjacency matrix for the reordered graph, and the non-empty blocks include at least one non-zero element.

According to another embodiment of the present invention, there is provided a computer program product for processing a graph. The computer program product comprises a computer readable storage medium having program instructions embodied therewith. The program instructions are readable by a device to cause the device to perform a method for processing a graph. In this method, an original graph including a plurality of nodes is obtained. The nodes of the original graph are reordered to generate a reordered graph, wherein the non-zero elements in an adjacency matrix for the reordered graph are clustered as compared with an adjacency matrix for the original graph. The adjacency matrix for the reordered graph is encoded with a plurality of integers, wherein each of the plurality of integers represents corresponding non-empty blocks in the adjacency matrix for the reordered graph, and the non-empty blocks include at least one non-zero element.