added tokenization functionality in order to better support auto complete