Patent Document ID: 9916299
Application ID: 15416186

Base Claim:
1. A computer-implemented method, comprising: receiving multiple data snippets, each of the data snippets comprising one or more n-grams; processing each of the data snippets by: identifying n-grams in the corresponding data snippet that have not occurred in any of the data snippets processed prior to the corresponding data snippet, as unseen n-grams, and computing a snippet score for the corresponding data snippet based on a frequency of the unseen n-grams and a length of the corresponding data snippet; sorting the data snippets based on the snippet scores to generate sorted data snippets; selecting, based on a first specified criterion, one or more of the sorted data snippets as training data for a language processing engine; and storing the training data in a memory, wherein the training data is to be used by the language processing engine to perform automated language processing functions.

---

Claim 5:
5. The computer-implemented method of claim 1 , wherein selecting the one or more of the sorted data snippets based on the first specified criterion includes selecting those of the sorted data snippets whose snippet scores are above a specified threshold.