tokenizer interface
implements tokenizer
using godoc for instead of inline example
remove example and old implementation
adds godoc
Merge pull request #1 from supershabam/reinterface

Reinterface