Language-independent categorization of text . Uses of n grams. During this process, all alphabetic characters are converted to lower case, non alphabetic characters are converted to spaces, and multiple spaces are collapsed into a single space. For example, the first 5 grams in this sentence would consist of for e , or ex , r exa , and so on. On the other hand, the validity of the Zipf law implies that the observation of a Zipf behavior is necessary in a natural text.
Frequency based methods model the frequency distributions of various events.
For the system call application, the events are occurrences of each pattern of system calls in a sequence. Each document is represented by a vector that is a histogram of sequence frequencies. Other frequency based methods examine sequences individually, making them suitable for on line use. We address the question of the appropriateness of some of these models below. Whereas human oriented techniques exploit the full range of character encodings, automatic methods are limited to standard character sets.