Patent ID: 7590641

Claim:
A method for balancing compression efficiency against access performance comprising the steps of: reading source data from a database row by row; generating a value frequency list indicating all unique values found for each column in each row read in said database and a frequency of occurrence for each value; selecting one or more algorithms to compress said source data read for each column, wherein said one or more algorithms are selected based on said value frequency list and said frequency of occurrence for each value; storing said selected one or more algorithms to compress said source data read for each column in a control block; reading said source data a second time row by row; compressing said source data read said second time using said stored selected one or more algorithms; and determining whether to tokenize a portion of a column read from said database using the following statement: IF STP<STT THEN tokenize said portion of said column read from said database; wherein said STP=(collen*COUNT(SETa))+(toklen+1)*SUM(freq in SETa)+collen*SUM(freq in SETb); wherein said STT=(toklen*numrows+collen*numvalues); wherein said collen is a length of column entries; wherein said numrows is a number of rows read in a table of said database; wherein said numvalues is a number of values in said value frequency list; wherein said toklen is a minimal length of a token; wherein said freq=frequency for a specific value found in said data read from said database; wherein SETa=all values that satisfy the following criteria: (collen*freq)>(collen+(toklen+1)*freq); wherein SETb=all values that do not satisfy the following criteria: (collen*freq)>(collen+(toklen+1)*freq); and wherein COUNT(SETa)=number of values in SETa.