Improved new ScalaCodeTokenizer.tokenize API to minimise allocation of string objects on caller side.