Patent ID: 7814284

Claim:
An apparatus comprising: a memory operable to store one or more aggregation trees, each aggregation tree comprising one or more base chunk nodes and one or more super chunk nodes, wherein each base chunk node comprises a chunk signature and corresponding raw data, and wherein super chunk nodes correspond to child base chunk nodes and include a chunk signature; and logic encoded in one or more tangible media for execution and when executed operable to cause one or more processors to: receive a data block; divide the data block into a plurality of base chunks, each base chunk having a degree value characterizing the occurrence probability of the base chunk; compute chunk signatures for the plurality of base chunks; apply a super chunk rule to contiguous sequences of base chunks of the plurality of base chunks to create one or more aggregation trees, wherein the super chunk rule aggregates base chunks based on the respective occurrence probabilities of the base chunks, wherein, to aggregate the base chunks, the logic is further operable to: identify a contiguous series of base chunks comprising a first base chunk, one or more intermediate base chunks, and a last base chunk; aggregate, with the last base chunk, one or more intermediate base chunks having a probability value that is higher than the probability value of the first base chunk and than the probability value of the last base chunk; identify one or more nodes in the one or more created aggregation trees that match corresponding nodes of the aggregation trees in the memory; compress the received data block based on the identified nodes; and conditionally add the one or more created aggregation trees to the memory.