This option of apply is limited in that HTML input cannot be tokenized. Please use the tokenize program to tokenize HTML text. For more information on stemming, please see the section on that subject . The program tokenize accepts plaintext or HTML input, and prints one token per line, with additional information. The input comes from the standard input, which is by default set to the terminal keyboard.
Output is sent to standard output, which is by default the terminal screen.
Redirection and pipe operators can be used in most command-line shells to use tokenize with files. The language given may be any language for which the file language .tok exists. For example, english may be specified. The output of tokenize is one token per line, plus offset and attribute information. For more information about how the tokenizer works, see the section on tokenization .