moved tokeniser into t.lua and colour detection