Do more processing of the tokens, remove spaces, convert to lower case
etc.

There is a bug in the hash table which in some cases is doing something
fishy in case of the updating of an existing key in the table leading to
a double free at the end when freeing the resources. For now, no need to
call the add function for an existing key, just update the value
directly.