deleted .gitignore
tests: added several test cases

with more to come
tokenizer: removed unavoidable branch