Patent ID: 7013261

Claim:
A method for performing morphology analysis of a natural language document, comprising the steps of: inputting the natural language document as an input text, tokenize the input text, thereby producing a token stream of a plurality of tokens, comparing each token of the token stream to a list of previously encountered tokens in the token stream, determining whether the token either corresponds to at least one previously encountered token in the list as a recurring token in the token stream, or else is a unique token in the token stream without corresponding to the at least one previously encountered token in the list, adding the unique token to the list of previously encountered tokens, marking the unique token with an identification, adding a pointer directed towards the identification for the recurring token, performing a morphological look-up only on the unique token, thereby producing a morphological look-up result, storing said morphological look-up result for the unique token together with the identification, reading the morphological look-up result for the recurring token, joining the morphological look-up results for the unique and recurring tokens, thereby producing a stream of morphological analyses, and outputting the stream of morphological analyses.