Fix bug in max_seq_length for preprocessing in ner example
Author: r-tinn
Pull request: yes
Labels: name: wontfix

**Summary**

If you try to use a value for `max_seq_length` that is less than 128 in the NER example, the maximum sequence length is exceeded when the predictions are made. There is a warning logged for this "Maximum sequence length exceeded: No prediction for.." and predictions cannot be made for these tokens.

**Changes**

Two changes are made:

- In `preprocess.py`, `subword_len_counter` is set to `current_subwords_len` when a blank line is inserted to split up sequences that exceed the maximum sequence length.
- `tokenizer.num_added_tokens()` is subtracted from `max_len` to account for the additional tokens inserted by the BERT tokenizer.



Comments:
# [Codecov](https://codecov.io/gh/huggingface/transformers/pull/3803?src=pr&el=h1) Report
> Merging [#3803](https://codecov.io/gh/huggingface/transformers/pull/3803?src=pr&el=desc) into [master](https://codecov.io/gh/huggingface/transformers/commit/01c37dcdb529bff38aadd51001cb5812e5fe9b21&el=desc) will **increase** coverage by `0.00%`.
> The diff coverage is `n/a`.

[![Impacted file tree graph](https://codecov.io/gh/huggingface/transformers/pull/3803/graphs/tree.svg?width=650&height=150&src=pr&token=9qOlN6Hb1c)](https://codecov.io/gh/huggingface/transformers/pull/3803?src=pr&el=tree)

```diff
@@           Coverage Diff           @@
##           master    #3803   +/-   ##
=======================================
  Coverage   78.26%   78.27%           
=======================================
  Files         106      106           
  Lines       17964    17964           
=======================================
+ Hits        14060    14061    +1     
+ Misses       3904     3903    -1     
```


| [Impacted Files](https://codecov.io/gh/huggingface/transformers/pull/3803?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [src/transformers/modeling\_tf\_utils.py](https://codecov.io/gh/huggingface/transformers/pull/3803/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy9tb2RlbGluZ190Zl91dGlscy5weQ==) | `92.84% <0.00%> (+0.16%)` | :arrow_up: |

------

[Continue to review full report at Codecov](https://codecov.io/gh/huggingface/transformers/pull/3803?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/huggingface/transformers/pull/3803?src=pr&el=footer). Last update [01c37dc...8f77ccd](https://codecov.io/gh/huggingface/transformers/pull/3803?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).


You might want to open this PR on https://github.com/stefan-it/fine-tuned-berts-seq instead, as the script is hosted there right now (cc @stefan-it)

> You might want to open this PR on https://github.com/stefan-it/fine-tuned-berts-seq instead, as the script is hosted there right now (cc @stefan-it)

I thought it would be easier just to have the script in the example, but can close this pr and open it there if you prefer?

What do you think @stefan-it? Are you ok with us including the script here?

Hi, sorry for the late reply! I've fixed some errors in the script last week. Would be great if @r-tinn could check the latest Version! If it's working then you can of course integrate it into Transformers :)

Looks good to me, the problem seems to be fixed

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.