Finished get_next_token for regexps and added tests for it.