Patent Document ID: 9208134
Application ID: 13346981

Base Claim:
1. A method implemented in a computer infrastructure, comprising: determining an attribute of a current character in input text, the attribute of the current character indicating one or more classes of characters the current character is assigned thereto; determining one or more attributes of one or more next characters in the input text, the one or more attributes of the one or more next characters indicating the one or more classes the one or more next characters are assigned thereto; and constructing a token of the input text that comprises the current character and the one or more next characters, the attribute of the current character and the one or more attributes of the one or more next characters intersecting with each other, wherein the attribute of the current character and the one or more attributes of the one or more next characters comprises an attribute data structure which comprises a one-byte array, and wherein the one-byte array comprises a plurality of binary bits and each bit of the binary bits indicates a different class from remaining bits of the binary bits.

---

Claim 2:
2. The method of claim 1 , further comprising setting the attribute data structure of the current character and the one or more next characters, to assign the current character and the one or more next characters to the one or more classes.