A method comprising: using a computer, receiving a stream of digital data comprising a plurality of objects; using programmed tokenizer instructions executed using the computer, in response to receiving a first object of the plurality of objects, tokenizing the first object to create a first tokenized object and electronically digitally storing the first tokenized object in a token database that comprises a plurality of other tokenized objects and using an electronic digital storage device; using the computer, comparing the first tokenized object to the plurality of other tokenized objects stored in the token database, computing a first pattern associated with the first tokenized object, and storing the first pattern in a pattern database that comprises a plurality of patterns; the method being characterised by using the computer, managing a size of the pattern database by: identifying, from the plurality of patterns, a subset of patterns that are eligible for deletion from the pattern database based on an age of each pattern and storing in computer memory data identifying the subset of patterns; ranking each pattern of the subset based on a quality metric and a popularity metric, by marking the data identifying the subset of patterns with rank values; identifying, based on the ranking and from the subset, a second pattern and deleting the second pattern from the pattern database to produce an updated database; repeating the tokenizing, comparing and storing using the updated database; wherein the method is executed using one or more computing devices. using a computer, receiving a stream of digital data comprising a plurality of objects; using programmed tokenizer instructions executed using the computer, in response to receiving a first object of the plurality of objects, tokenizing the first object to create a first tokenized object and electronically digitally storing the first tokenized object in a token database that comprises a plurality of other tokenized objects and using an electronic digital storage device; using the computer, comparing the first tokenized object to the plurality of other tokenized objects stored in the token database, computing a first pattern associated with the first tokenized object, and storing the first pattern in a pattern database that comprises a plurality of patterns; the method being characterised by using the computer, managing a size of the pattern database by: identifying, from the plurality of patterns, a subset of patterns that are eligible for deletion from the pattern database based on an age of each pattern and storing in computer memory data identifying the subset of patterns; ranking each pattern of the subset based on a quality metric and a popularity metric, by marking the data identifying the subset of patterns with rank values; identifying, based on the ranking and from the subset, a second pattern and deleting the second pattern from the pattern database to produce an updated database; repeating the tokenizing, comparing and storing using the updated database; wherein the method is executed using one or more computing devices. identifying, from the plurality of patterns, a subset of patterns that are eligible for deletion from the pattern database based on an age of each pattern and storing in computer memory data identifying the subset of patterns; ranking each pattern of the subset based on a quality metric and a popularity metric, by marking the data identifying the subset of patterns with rank values; identifying, based on the ranking and from the subset, a second pattern and deleting the second pattern from the pattern database to produce an updated database; repeating the tokenizing, comparing and storing using the updated database; wherein the method is executed using one or more computing devices.