codes updated for tokenization