* changed all process_tokens to take len parameter as well
  * expectation for now is len can only be at the null terminator
  * started work to support for len to be before the null
  * deliberatly avoiding len being after the null teminator
* changed resize_buffer to take a min_to_add parameter
  * several loops that were checking capacity per character are now changed to one check and a memcpy