Added tests and fixed bug while tokenizing