Patent ID: 8516157

Claim:
A method of compressing data in a network, the data comprising a plurality of packets corresponding to a plurality of headers and a plurality of payloads, a packet of the plurality of packets each having a header of the plurality of headers and a payload of the plurality of payloads, each of the plurality of headers comprising a plurality of header fields, the method comprising: generating, using a computer, a matrix based on the plurality of packets, the matrix comprising a plurality of rows each corresponding to one of the plurality of packets, a plurality of structured columns each corresponding to one of the plurality of header fields, and an unstructured column having a plurality of row elements each comprising one of the plurality of payloads; determining, using the computer, a compression plan based at least on a classification tree comprising a plurality of hierarchical categories, wherein the compression plan comprises: an intra-packet compression plan determined during a training phase based on minimizing a second cost function while applying the intra-packet compression plan to the historical network data, the intra-packet compression plan configured to reorder and divide the plurality of structured columns in the matrix into the first plurality of subsets; and an inter-packet compression plan determined during the training phase based on the classification tree, the inter-packet compression plan configured to reorder and classify the plurality of rows in the matrix based on the plurality of hierarchical categories in the classification tree; performing, using the computer, online compression in real time of the plurality of packets while the plurality of packets are transferred by a network node of the network, the online compression being performed according to a first configuration of the compression plan; and performing, using the computer, offline compression of the plurality of packets subsequent to the plurality of packets being collected by a data repository of the network, the offline compression being performed according to a second configuration of the compression plan; and generating, using the computer and prior to determining the compression plan, the classification tree comprising the plurality of hierarchical categories based on at least a portion of the plurality of header fields by: composing a full classification tree having a plurality of nodes based on the at least a portion of the plurality of header fields, each of the plurality of nodes representing payload classification based on a specific value combination of the plurality of header fields; and identifying a hierarchical category of the plurality of hierarchical categories from the full classification tree, the hierarchical category comprising one or more nodes in the plurality of nodes identified based on minimizing a first cost function while applying the compression plan to historical network data.