Token-level regression mode added in ForTokenClassification models
Author: gsarti
Pull request: yes
Labels: name: wontfix

This is related to the issue #3646 I opened 2 days ago and which was considered interesting by @julien-c 

I added the **support for token-level regression in Bert, Roberta, Albert, XLNet, XLM, DistilBert and in the template for adding a new model** when `self.num_labels == 1`, fixing the docstring to match the new changes (and correcting the one for XLNetForTokenClassification which was copied from the XLNForMultipleChoice one).

Given two different approaches to compute `active_labels`, I privileged the one used in more recent models (e.g. Albert). The change was tested on tests and examples (without RUN_SLOW and RUN_CUSTOM_TOKENIZERS though) and passed all tests.

I didn't felt like adding this to Electra since its TokenClassification implementation seems to be missing the `num_labels` variable for some reason. I didn't add new tests since the sentence regression case wasn't covered by the testing suite either.

**Edit:** After the fix on `ElectraForTokenClassification`'s `num_labels` attribute, I added support for token-level regression there too.

Comments:
# [Codecov](https://codecov.io/gh/huggingface/transformers/pull/3703?src=pr&el=h1) Report
> Merging [#3703](https://codecov.io/gh/huggingface/transformers/pull/3703?src=pr&el=desc) into [master](https://codecov.io/gh/huggingface/transformers/commit/6435b9f908e7361330db89e263a65b0a58060d11&el=desc) will **decrease** coverage by `0.98%`.
> The diff coverage is `59.37%`.

[![Impacted file tree graph](https://codecov.io/gh/huggingface/transformers/pull/3703/graphs/tree.svg?width=650&height=150&src=pr&token=9qOlN6Hb1c)](https://codecov.io/gh/huggingface/transformers/pull/3703?src=pr&el=tree)

```diff
@@            Coverage Diff             @@
##           master    #3703      +/-   ##
==========================================
- Coverage   78.13%   77.14%   -0.99%     
==========================================
  Files         104      104              
  Lines       17723    17752      +29     
==========================================
- Hits        13847    13695     -152     
- Misses       3876     4057     +181     
```


| [Impacted Files](https://codecov.io/gh/huggingface/transformers/pull/3703?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [src/transformers/modeling\_albert.py](https://codecov.io/gh/huggingface/transformers/pull/3703/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy9tb2RlbGluZ19hbGJlcnQucHk=) | `74.50% <0.00%> (-0.75%)` | :arrow_down: |
| [src/transformers/modeling\_xlnet.py](https://codecov.io/gh/huggingface/transformers/pull/3703/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy9tb2RlbGluZ194bG5ldC5weQ==) | `73.33% <50.00%> (-2.45%)` | :arrow_down: |
| [src/transformers/modeling\_bert.py](https://codecov.io/gh/huggingface/transformers/pull/3703/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy9tb2RlbGluZ19iZXJ0LnB5) | `87.88% <66.66%> (-0.28%)` | :arrow_down: |
| [src/transformers/modeling\_roberta.py](https://codecov.io/gh/huggingface/transformers/pull/3703/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy9tb2RlbGluZ19yb2JlcnRhLnB5) | `85.04% <66.66%> (-10.67%)` | :arrow_down: |
| [src/transformers/modeling\_xlm.py](https://codecov.io/gh/huggingface/transformers/pull/3703/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy9tb2RlbGluZ194bG0ucHk=) | `88.78% <66.66%> (-0.38%)` | :arrow_down: |
| [src/transformers/modeling\_distilbert.py](https://codecov.io/gh/huggingface/transformers/pull/3703/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy9tb2RlbGluZ19kaXN0aWxiZXJ0LnB5) | `97.02% <70.00%> (-0.57%)` | :arrow_down: |
| [src/transformers/modeling\_electra.py](https://codecov.io/gh/huggingface/transformers/pull/3703/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy9tb2RlbGluZ19lbGVjdHJhLnB5) | `72.38% <75.00%> (-0.27%)` | :arrow_down: |
| [src/transformers/modeling\_tf\_pytorch\_utils.py](https://codecov.io/gh/huggingface/transformers/pull/3703/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy9tb2RlbGluZ190Zl9weXRvcmNoX3V0aWxzLnB5) | `8.72% <0.00%> (-81.21%)` | :arrow_down: |
| [src/transformers/modeling\_ctrl.py](https://codecov.io/gh/huggingface/transformers/pull/3703/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy9tb2RlbGluZ19jdHJsLnB5) | `95.61% <0.00%> (-2.64%)` | :arrow_down: |
| ... and [3 more](https://codecov.io/gh/huggingface/transformers/pull/3703/diff?src=pr&el=tree-more) | |

------

[Continue to review full report at Codecov](https://codecov.io/gh/huggingface/transformers/pull/3703?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/huggingface/transformers/pull/3703?src=pr&el=footer). Last update [6435b9f...c6692e6](https://codecov.io/gh/huggingface/transformers/pull/3703?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).


LGTM but I'll let others chime in.

For ELECTRA, the discrepancy should have been fixed by 500aa12318ce5acd289d5edb6cb8266b3c3b162e, so can you propagate your changes there too?

Related to naming, as it's for the `XXXForTokenClassification` when you have a single label wouldn't you expect to get a cross-entropy loss such as [binary cross-entropy](https://pytorch.org/docs/stable/nn.html#bceloss), rather than regression? Seeing as it's a classification model?

I'd say the same would apply to `XXXForSentenceClassification`, right? It would probably be best to decouple regression and classification for those two classes instead of having regression as a special case, and make the behavior for `num_labels == 1` the same as `num_labels == 2` for labels that are not one-hot encoded.

Probably worth a dedicated PR to decouple both instead of handling it here!

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.


Any chance of blowing new life into this? Token classification is particularly familiar due to NER, but in many research fields (e.g. psycholinguistic studies) we are interested in a lot more than that. Continuous values for tokens are very common there. I'd love to see regression and multilabel classification for token classification models.

Sad to see this die :(