please make this configurable, maybe even support sets of characters. you can very efficiently do this by defining a byte delimiters[256] array that has 1's at positions that should be delimiters, e.g. delimiters['\n']=1;

This would help e.g. csv parsing.