Tokenizer is a component that reads characters from a stream and gives the smallest lexical units, tokens, implemented as C++ objects, as a result.