improved tokening and fixed some more memory related issues also added test.sh for basic testing purposes
improved tokening and fixed some more memory related issues also added test.sh for basic testing purposes