added support for working with raw tokens, improved memoisation