Fixes #1263, add tokenization_with_offsets, gets tokens with offsets in the original text
Author: michaelrglass
Pull request: yes
Labels: name: wontfix

This is similar to the utils_squad approach to getting offsets for the tokens but can also be used in other places where the tokens should have a correspondence to the original text to fix #1263.


Comments:
I think this is an interesting addition and I like the way the PR is structured in general.

Before I dive in, could you lay down the status of the PR in terms of supported models, python version (we will still keep python 2 support for now), know issues and TO-DOs?

The fully supported models are BERT, GPT2, XLNet, and RoBERTa. For these models tokenize_with_offsets always produces the same tokens as tokenize and the subword tokens are well-aligned to the original text, typically as well as possible.

XLM has a known issue with patterns like '... inc. reported ...'
tokxlm.tokenize_with_offsets('inc. reported')
['inc</w>', '.</w>', 'reported</w>']
tokxlm.tokenize('inc. reported')
['inc.</w>', 'reported</w>']
Because tokenize_with_offsets passes whitespace separated 'chunks' (utils_squad style) the xlm tokenizer doesn't get to look ahead to see that a lowercase word follows the period.

TransfoXL just needs the latest commit (not in this PR), while OpenAIGPT has an issue with '\n</w>' tokens, which tokenize_with_offsets never produces.

Everything works in Python 2 except GPT2 (and therefore RoBERTa) as well as XLNet. I think this is the case even without this PR. I'm not experienced with Python 2 though, so maybe I am just missing something.




# [Codecov](https://codecov.io/gh/huggingface/pytorch-transformers/pull/1274?src=pr&el=h1) Report
> Merging [#1274](https://codecov.io/gh/huggingface/pytorch-transformers/pull/1274?src=pr&el=desc) into [master](https://codecov.io/gh/huggingface/pytorch-transformers/commit/45de034bf899af678d844351ff21ea0444815ddb?src=pr&el=desc) will **decrease** coverage by `0.97%`.
> The diff coverage is `59.04%`.

[![Impacted file tree graph](https://codecov.io/gh/huggingface/pytorch-transformers/pull/1274/graphs/tree.svg?width=650&token=9qOlN6Hb1c&height=150&src=pr)](https://codecov.io/gh/huggingface/pytorch-transformers/pull/1274?src=pr&el=tree)

```diff
@@            Coverage Diff             @@
##           master    #1274      +/-   ##
==========================================
- Coverage   81.16%   80.19%   -0.98%     
==========================================
  Files          57       60       +3     
  Lines        8039     8405     +366     
==========================================
+ Hits         6525     6740     +215     
- Misses       1514     1665     +151
```


| [Impacted Files](https://codecov.io/gh/huggingface/pytorch-transformers/pull/1274?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [...tests/regression\_test\_tokenization\_with\_offsets.py](https://codecov.io/gh/huggingface/pytorch-transformers/pull/1274/diff?src=pr&el=tree#diff-cHl0b3JjaF90cmFuc2Zvcm1lcnMvdGVzdHMvcmVncmVzc2lvbl90ZXN0X3Rva2VuaXphdGlvbl93aXRoX29mZnNldHMucHk=) | `0% <0%> (ø)` | |
| [pytorch\_transformers/tokenization\_bert.py](https://codecov.io/gh/huggingface/pytorch-transformers/pull/1274/diff?src=pr&el=tree#diff-cHl0b3JjaF90cmFuc2Zvcm1lcnMvdG9rZW5pemF0aW9uX2JlcnQucHk=) | `95.79% <100%> (+0.12%)` | :arrow_up: |
| [pytorch\_transformers/tokenization\_gpt2.py](https://codecov.io/gh/huggingface/pytorch-transformers/pull/1274/diff?src=pr&el=tree#diff-cHl0b3JjaF90cmFuc2Zvcm1lcnMvdG9rZW5pemF0aW9uX2dwdDIucHk=) | `96.87% <100%> (+0.18%)` | :arrow_up: |
| [...ytorch\_transformers/tests/tokenization\_xlm\_test.py](https://codecov.io/gh/huggingface/pytorch-transformers/pull/1274/diff?src=pr&el=tree#diff-cHl0b3JjaF90cmFuc2Zvcm1lcnMvdGVzdHMvdG9rZW5pemF0aW9uX3hsbV90ZXN0LnB5) | `97.87% <100%> (+0.14%)` | :arrow_up: |
| [...ch\_transformers/tests/tokenization\_offsets\_test.py](https://codecov.io/gh/huggingface/pytorch-transformers/pull/1274/diff?src=pr&el=tree#diff-cHl0b3JjaF90cmFuc2Zvcm1lcnMvdGVzdHMvdG9rZW5pemF0aW9uX29mZnNldHNfdGVzdC5weQ==) | `100% <100%> (ø)` | |
| [...transformers/tests/tokenization\_transfo\_xl\_test.py](https://codecov.io/gh/huggingface/pytorch-transformers/pull/1274/diff?src=pr&el=tree#diff-cHl0b3JjaF90cmFuc2Zvcm1lcnMvdGVzdHMvdG9rZW5pemF0aW9uX3RyYW5zZm9feGxfdGVzdC5weQ==) | `97.36% <100%> (+0.3%)` | :arrow_up: |
| [...ch\_transformers/tests/tokenization\_roberta\_test.py](https://codecov.io/gh/huggingface/pytorch-transformers/pull/1274/diff?src=pr&el=tree#diff-cHl0b3JjaF90cmFuc2Zvcm1lcnMvdGVzdHMvdG9rZW5pemF0aW9uX3JvYmVydGFfdGVzdC5weQ==) | `92.85% <100%> (+0.4%)` | :arrow_up: |
| [pytorch\_transformers/tokenization\_transfo\_xl.py](https://codecov.io/gh/huggingface/pytorch-transformers/pull/1274/diff?src=pr&el=tree#diff-cHl0b3JjaF90cmFuc2Zvcm1lcnMvdG9rZW5pemF0aW9uX3RyYW5zZm9feGwucHk=) | `34.54% <100%> (+0.36%)` | :arrow_up: |
| [pytorch\_transformers/tokenization\_xlm.py](https://codecov.io/gh/huggingface/pytorch-transformers/pull/1274/diff?src=pr&el=tree#diff-cHl0b3JjaF90cmFuc2Zvcm1lcnMvdG9rZW5pemF0aW9uX3hsbS5weQ==) | `82.98% <100%> (+0.28%)` | :arrow_up: |
| [pytorch\_transformers/tokenization\_openai.py](https://codecov.io/gh/huggingface/pytorch-transformers/pull/1274/diff?src=pr&el=tree#diff-cHl0b3JjaF90cmFuc2Zvcm1lcnMvdG9rZW5pemF0aW9uX29wZW5haS5weQ==) | `82.4% <100%> (+0.58%)` | :arrow_up: |
| ... and [11 more](https://codecov.io/gh/huggingface/pytorch-transformers/pull/1274/diff?src=pr&el=tree-more) | |

------

[Continue to review full report at Codecov](https://codecov.io/gh/huggingface/pytorch-transformers/pull/1274?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/huggingface/pytorch-transformers/pull/1274?src=pr&el=footer). Last update [45de034...328e698](https://codecov.io/gh/huggingface/pytorch-transformers/pull/1274?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).


Would love to see this feature available - what can I do to help get it merged in?

We are currently working on a larger project around this and should come to this PR pretty soon (next week I hope).

@michaelrglass How does this handle the destructive normalisation that occurs in eg BertTokenizer? Specifically, logic like [this](https://github.com/huggingface/transformers/blob/master/transformers/tokenization_bert.py#L330) means that the normalisation isn't length preserving, and it may not be possible to find the (normalised) token in the original input text.

> @michaelrglass How does this handle the destructive normalisation that occurs in eg BertTokenizer? Specifically, logic like [this](https://github.com/huggingface/transformers/blob/master/transformers/tokenization_bert.py#L330) means that the normalisation isn't length preserving, and it may not be possible to find the (normalised) token in the original input text.

IMO it's too error-prone to run a destructive tokenizer and then try to align the sequences after the fact.  I wrote https://github.com/microsoft/bistring for this exact kind of problem.  Otherwise it's very tricky to align substrings of the modified text with the original text, as both NFD and filtering out nonspacing marks can shift the positions of chars significantly.

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.