move out non-tokenizing functions to new file