add horrible token filtering, needs tidy