myshell now adds spaces around special charaters, so tokenizer can handle all cases