Implement SuperGLUE tasks and baselines
Author: W4ngatang
Pull request: yes

WIP implementation of the SuperGLUE tasks, following the style of the GLUE data processors. This PR includes:
- classification with marked spans: most tasks are classification and straightforward, but WiC and WSC have special spans. This PR implements `*SpanClassification` models (currently for just BERT and RoBERTa) and relevant data classes
- scripts for using and extracting results from the [experiment impact tracker](https://github.com/Breakend/experiment-impact-tracker) library, which tracks energy usage and is being used for the SustaiNLP 2020 competition

TODO
- clean up some of my development scripts and cruft

Comments:
Hi @W4ngatang did you see that we have a new `Trainer` class in the library? It's basically a refactor of the previous version of `run_glue.py` so should be pretty easy to adopt.

Let us know if we can help, and very excited to have an easy path to SuperGLUE tasks in the repo :)

Hi Julien,

Saw it! I started writing this a while ago, but you all move very fast. We're preparing this as some reference code for a competition, so we may keep the training code as is for a bit and then switch to the trainer when we're ready to actually merge into the main repo.

@thomwolf 

# [Codecov](https://codecov.io/gh/huggingface/transformers/pull/4378?src=pr&el=h1) Report
> Merging [#4378](https://codecov.io/gh/huggingface/transformers/pull/4378?src=pr&el=desc) into [master](https://codecov.io/gh/huggingface/transformers/commit/90f4b2452077ac3bac9453bdc63e0359aa4fe4d2&el=desc) will **decrease** coverage by `1.47%`.
> The diff coverage is `22.83%`.

[![Impacted file tree graph](https://codecov.io/gh/huggingface/transformers/pull/4378/graphs/tree.svg?width=650&height=150&src=pr&token=9qOlN6Hb1c)](https://codecov.io/gh/huggingface/transformers/pull/4378?src=pr&el=tree)

```diff
@@            Coverage Diff             @@
##           master    #4378      +/-   ##
==========================================
- Coverage   78.00%   76.52%   -1.48%     
==========================================
  Files         138      139       +1     
  Lines       23766    24408     +642     
==========================================
+ Hits        18539    18679     +140     
- Misses       5227     5729     +502     
```


| [Impacted Files](https://codecov.io/gh/huggingface/transformers/pull/4378?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [src/transformers/modeling\_bert.py](https://codecov.io/gh/huggingface/transformers/pull/4378/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy9tb2RlbGluZ19iZXJ0LnB5) | `83.36% <12.12%> (-4.14%)` | :arrow_down: |
| [src/transformers/data/metrics/\_\_init\_\_.py](https://codecov.io/gh/huggingface/transformers/pull/4378/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy9kYXRhL21ldHJpY3MvX19pbml0X18ucHk=) | `16.66% <12.35%> (-10.00%)` | :arrow_down: |
| [src/transformers/modeling\_roberta.py](https://codecov.io/gh/huggingface/transformers/pull/4378/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy9tb2RlbGluZ19yb2JlcnRhLnB5) | `83.80% <19.44%> (-10.99%)` | :arrow_down: |
| [src/transformers/data/processors/superglue.py](https://codecov.io/gh/huggingface/transformers/pull/4378/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy9kYXRhL3Byb2Nlc3NvcnMvc3VwZXJnbHVlLnB5) | `21.73% <21.73%> (ø)` | |
| [src/transformers/data/processors/utils.py](https://codecov.io/gh/huggingface/transformers/pull/4378/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy9kYXRhL3Byb2Nlc3NvcnMvdXRpbHMucHk=) | `35.59% <84.61%> (+7.96%)` | :arrow_up: |
| [src/transformers/\_\_init\_\_.py](https://codecov.io/gh/huggingface/transformers/pull/4378/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy9fX2luaXRfXy5weQ==) | `99.21% <100.00%> (ø)` | |
| [src/transformers/data/\_\_init\_\_.py](https://codecov.io/gh/huggingface/transformers/pull/4378/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy9kYXRhL19faW5pdF9fLnB5) | `100.00% <100.00%> (ø)` | |
| [src/transformers/data/processors/\_\_init\_\_.py](https://codecov.io/gh/huggingface/transformers/pull/4378/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy9kYXRhL3Byb2Nlc3NvcnMvX19pbml0X18ucHk=) | `100.00% <100.00%> (ø)` | |
| [src/transformers/data/processors/glue.py](https://codecov.io/gh/huggingface/transformers/pull/4378/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy9kYXRhL3Byb2Nlc3NvcnMvZ2x1ZS5weQ==) | `49.44% <100.00%> (ø)` | |
| [src/transformers/modeling\_openai.py](https://codecov.io/gh/huggingface/transformers/pull/4378/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy9tb2RlbGluZ19vcGVuYWkucHk=) | `79.51% <0.00%> (-1.39%)` | :arrow_down: |
| ... and [2 more](https://codecov.io/gh/huggingface/transformers/pull/4378/diff?src=pr&el=tree-more) | |

------

[Continue to review full report at Codecov](https://codecov.io/gh/huggingface/transformers/pull/4378?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/huggingface/transformers/pull/4378?src=pr&el=footer). Last update [90f4b24...576ecaf](https://codecov.io/gh/huggingface/transformers/pull/4378?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).


Looks super-cool, especially with SustaiNLP just around the corner! Is there any help that could be given that would accelerate this PR?

hi @W4ngatang @thomwolf 

any help you need for this PR? I think pushing this PR through can help many researchers with their experiments. 

Hi Tianyi,

Yes, help would be great! The code is correct from my end, but uses an old
version of Transformers. At this point, it's just a matter of merging the
most recent version of Transformers in and adhering to the Transformers
contributor guidelines.

Best,
Alex

On Thu, Oct 8, 2020 at 5:17 PM Tianyi <notifications@github.com> wrote:

> hi @W4ngatang <https://github.com/W4ngatang> @thomwolf
> <https://github.com/thomwolf>
>
> any help you need for this PR? I think pushing this PR through can help
> many researchers with their experiments.
>
> —
> You are receiving this because you were mentioned.
> Reply to this email directly, view it on GitHub
> <https://github.com/huggingface/transformers/pull/4378#issuecomment-705828560>,
> or unsubscribe
> <https://github.com/notifications/unsubscribe-auth/ABKDWG7FSDKTCVL4MI6TXEDSJYT7DANCNFSM4NBH5EJA>
> .
>


This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md) are likely to be ignored.