Converted lexer into ruby module. Also changed functionality: the lexer returns a generator which returns indivdual words. This matches the recursive descent parser more closely and will minimize effort in integrating with the parser.