added a TODO. Should implement separate tokenize() calls for digits and characters (as it turns out id's cannot contain both characters and digits as I originally thought