A critical point in a text is that which delimits two adjacent segments. Conversely a critical segment is that which is delimited by two points in the text. Segments are, thus, located in time and space. Segments can be observed, pointed to and given unique names. Segments and strings are not the same.
Segments are instances of strings.
Two segments may be instances of the same string. Segments may, thus, correspond to tokens, whereas strings correspond to types. Tokenisation is achieved by numbering the critical points that delimit tokens in the text. The text is converted to a Prolog text database in which each token is represented by a three place predicate. The first and second arguments are the critical points which delimit the token.