It works because `token` is a global regex and retails its current scope when recursing. 

This is an instance where a simple scanner would probably be fine (and easier to read :-) ), but the code _does_ work.