added expected inputs and outputs for functions in comments
added test for applying diffs for empty string to new tokens.