Patent ID: 8069032

Claim:
A method to be executed at least in part in a computing device for detecting novelty in a language model customization service, comprising: receiving, by the computing device, data for customizing a generic language model, wherein receiving the data for customizing the generic language model includes harvesting the data from a data source employing a statistical analysis method; determining, by the computing device, using trigram windowing, whether the received data is similar by a predetermined degree to existing data in a language model customization data store, wherein, for at least one window in the received data, the trigram windowing comprises: initializing a unique count to zero; looking up a triple in the language model customization data store; incrementing the unique count when the triple is not present in the language model customization data store; and for each triple in the at least one window, incrementing the unique count for triples present in the language model customization data store when the unique count is greater than zero, wherein the at least one window comprises a jumping window, wherein the jumping window advances and increments multiple triples; and if, while the data is being harvested from the data source, the received data is determined to be similar to the existing data, performing, by the computing device, each of the following operations: discarding the received data and assigning a lower weight factor to the received data indicating a lack of novelty, wherein determining whether the received data is similar to the existing data includes employing the same statistical analysis method to compare the received data to the existing data, wherein when the at least one window is small, a larger percentage of unseen data is rejected than when the at least one window is large when the received data is determined to be similar to the existing data; and if, while the data is being harvested from the data source, the received data is dissimilar to the existing data, adding, by the computing device, the received data to the language model customization data store, wherein the received data comprises novel data.