In some programming languages, comments are treated as white space. White space is sometimes compacted into a single space character by the compiler. Fixed line lengths typically occur with programming languages that use columns. Some programming languages have special character sequences for encoding required characters on older hardware that doesn't support all of the required source code character set. Comments are either removed or replaced with white space.
Tokens are the basic lexical building blocks of source code.
Characters are combined into tokens according to the rules of the programming language. Two common kinds of separators are indicators of an end of an instruction and separators used for grouping. These are also called sequences or iterative structures. Basic arithmetic, logical, and bit operations are in this category. Data moves and copies are sequences.