This remains a human intensive effort. A corpus of text is used in conjunction with dictionaries to determine appropriate vocabularies. Then a unigram count for all of the spellings that occur in a corpus is determined. Those words that also occur in the dictionary are included. In addition a human screens the most frequent subset of new spellings to determine if they are words.
Table shows the coverage of new text using a fixed vocabulary of a given size for English.
For more inflectional languages such as French or German larger vocabulary sizes are required to achieve coverage similar to that of English. For a user of a speech recognition system, a more personalized vocabulary can be much more effective that a general fixed vocabulary. Table shows the coverage as new words are added to a starting vocabulary of 20,000 words as more text is observed. In addition, Table indicates the size of text recognized to add that many words. Static coverage of unseen text as a function of vocabulary size.