Tokens are the keywords, names, numbers etc. The result of lexing is a sequence of tokens called a token stream. All NAGWare f77 Tools invoke the lexing routines as the first, or in some cases only, stage of analysis. In the process the parser carries out a syntactic analysis and ensures that the statements in the program unit are correctly formed and ordered. A parse tree is a low level representation of a program unit in which the program elements are conceptually laid out in a two dimensional tree structure.
Pointers logically connect the elements of the program.
Another aspect of parsing is the construction of the symbol table. A comment index is also constructed to ensure that comments remain associated with the same statement as in the input file. Before any transformation is made the input source is analysed by the NAGWare f77 Tools lexer and, in some cases, parser. Deviations from this standard, in most cases will cause the input file to be rejected with error messages and the transformation will not be carried out. The transformational tools require to turn the internal representation of the transformed code back into Fortran text.