noamaanMulla-03/transformers-issues huggingface dataset:
[Longformer] more models + model cards
Author: ibeltagy
Pull request: yes
Labels: name: model card

This PR adds the following:

- Longformer models trained with frozen-roberta weights

- Model cards

- Model names start with `allenai/`

- Remove unnecessary type casting


Comments:
# [Codecov](https://codecov.io/gh/huggingface/transformers/pull/4628?src=pr&el=h1) Report
> Merging [#4628](https://codecov.io/gh/huggingface/transformers/pull/4628?src=pr&el=desc) into [master](https://codecov.io/gh/huggingface/transformers/commit/a801c7fd74f56a651ba43bfc93eba93c63e84766&el=desc) will **decrease** coverage by `0.00%`.
> The diff coverage is `100.00%`.

[![Impacted file tree graph](https://codecov.io/gh/huggingface/transformers/pull/4628/graphs/tree.svg?width=650&height=150&src=pr&token=9qOlN6Hb1c)](https://codecov.io/gh/huggingface/transformers/pull/4628?src=pr&el=tree)

```diff
@@            Coverage Diff             @@
##           master    #4628      +/-   ##
==========================================
- Coverage   78.02%   78.02%   -0.01%     
==========================================
  Files         124      124              
  Lines       20626    20625       -1     
==========================================
- Hits        16094    16093       -1     
  Misses       4532     4532              
```


| [Impacted Files](https://codecov.io/gh/huggingface/transformers/pull/4628?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [src/transformers/configuration\_longformer.py](https://codecov.io/gh/huggingface/transformers/pull/4628/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy9jb25maWd1cmF0aW9uX2xvbmdmb3JtZXIucHk=) | `100.00% <ø> (ø)` | |
| [src/transformers/modeling\_longformer.py](https://codecov.io/gh/huggingface/transformers/pull/4628/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy9tb2RlbGluZ19sb25nZm9ybWVyLnB5) | `97.40% <100.00%> (-0.01%)` | :arrow_down: |
| [src/transformers/tokenization\_longformer.py](https://codecov.io/gh/huggingface/transformers/pull/4628/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy90b2tlbml6YXRpb25fbG9uZ2Zvcm1lci5weQ==) | `100.00% <100.00%> (ø)` | |
| [src/transformers/modeling\_utils.py](https://codecov.io/gh/huggingface/transformers/pull/4628/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy9tb2RlbGluZ191dGlscy5weQ==) | `90.41% <0.00%> (-0.12%)` | :arrow_down: |
| [src/transformers/modeling\_tf\_utils.py](https://codecov.io/gh/huggingface/transformers/pull/4628/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy9tb2RlbGluZ190Zl91dGlscy5weQ==) | `88.66% <0.00%> (+0.16%)` | :arrow_up: |

------

[Continue to review full report at Codecov](https://codecov.io/gh/huggingface/transformers/pull/4628?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/huggingface/transformers/pull/4628?src=pr&el=footer). Last update [a801c7f...06e80ad](https://codecov.io/gh/huggingface/transformers/pull/4628?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
add 2 colab notebooks
Author: lavanyashukla
Pull request: yes


Comments:
# [Codecov](https://codecov.io/gh/huggingface/transformers/pull/4505?src=pr&el=h1) Report
> Merging [#4505](https://codecov.io/gh/huggingface/transformers/pull/4505?src=pr&el=desc) into [master](https://codecov.io/gh/huggingface/transformers/commit/a08652772791fdaeed6f263b1a99926ca64be5dc&el=desc) will **increase** coverage by `0.00%`.
> The diff coverage is `n/a`.

[![Impacted file tree graph](https://codecov.io/gh/huggingface/transformers/pull/4505/graphs/tree.svg?width=650&height=150&src=pr&token=9qOlN6Hb1c)](https://codecov.io/gh/huggingface/transformers/pull/4505?src=pr&el=tree)

```diff
@@           Coverage Diff           @@
##           master    #4505   +/-   ##
=======================================
  Coverage   77.83%   77.84%           
=======================================
  Files         123      123           
  Lines       20514    20514           
=======================================
+ Hits        15968    15969    +1     
+ Misses       4546     4545    -1     
```


| [Impacted Files](https://codecov.io/gh/huggingface/transformers/pull/4505?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [src/transformers/modeling\_tf\_utils.py](https://codecov.io/gh/huggingface/transformers/pull/4505/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy9tb2RlbGluZ190Zl91dGlscy5weQ==) | `88.66% <0.00%> (+0.16%)` | :arrow_up: |

------

[Continue to review full report at Codecov](https://codecov.io/gh/huggingface/transformers/pull/4505?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/huggingface/transformers/pull/4505?src=pr&el=footer). Last update [a086527...2293fe1](https://codecov.io/gh/huggingface/transformers/pull/4505?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).


Those are amazing notebooks! Would it maybe be possible to connect the Notebook "A Step by Step Guide to Tracking Hugging Face Model Performance" to a github account and link it from there? As it's done for other notebook ?

Merging for now - link can be updated at a later stage.
run evalation after every epoch in Trainer
Author: prajjwal1
Pull request: no

# 🚀 Feature request

With the current Trainer implementation:
`trainer.train(..)` is called first followed by `trainer.evaluate(..)`. It would be nice if the user can pass the flag `--run_eval` (something similar) to run evaluation after every epoch. It would be nice for users who want to see how model performs on validation set as training progresses. In some cases, this is the general norm (run evaluation after every epoch).

Comments:
You should use `--evaluate_during_training` which should do mostly what you're looking for

@prajjwal1 , you should be able to achieve this with `--evaluate_during_training` provided you set `--save_steps` to  `number_of_samples/batch_size`. However, I'm currently having trouble achieving this with that option when using both `run_language_modeling.py` and `run_glue.py` as I specify in https://github.com/huggingface/transformers/issues/4630. Any ideas @julien-c ? Thanks in advance.

There's a problem with MNLI though. In the example, arguments are changed from `mnli` to `mnli-mm`, so running evaluation after each epoch will happen on MNLI and not the mismatched one with the current implementation.
Fix add_special_tokens on fast tokenizers
Author: n1t0
Pull request: yes

Fix #4457

By using `flatten`, the following
```
dict_values(['[EOS]', '[BOS]'])
```
was being transformed into this:
```
['[', 'E', 'O', 'S', ']', '[', 'B', 'O', 'S', ']']
```

Comments:
# [Codecov](https://codecov.io/gh/huggingface/transformers/pull/4531?src=pr&el=h1) Report
> Merging [#4531](https://codecov.io/gh/huggingface/transformers/pull/4531?src=pr&el=desc) into [master](https://codecov.io/gh/huggingface/transformers/commit/e19b978151419fe0756ba852b145fccfc96dbeb4&el=desc) will **increase** coverage by `0.01%`.
> The diff coverage is `100.00%`.

[![Impacted file tree graph](https://codecov.io/gh/huggingface/transformers/pull/4531/graphs/tree.svg?width=650&height=150&src=pr&token=9qOlN6Hb1c)](https://codecov.io/gh/huggingface/transformers/pull/4531?src=pr&el=tree)

```diff
@@            Coverage Diff             @@
##           master    #4531      +/-   ##
==========================================
+ Coverage   77.86%   77.88%   +0.01%     
==========================================
  Files         123      123              
  Lines       20566    20570       +4     
==========================================
+ Hits        16014    16020       +6     
+ Misses       4552     4550       -2     
```


| [Impacted Files](https://codecov.io/gh/huggingface/transformers/pull/4531?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [src/transformers/tokenization\_utils.py](https://codecov.io/gh/huggingface/transformers/pull/4531/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy90b2tlbml6YXRpb25fdXRpbHMucHk=) | `89.55% <100.00%> (+0.04%)` | :arrow_up: |
| [src/transformers/modeling\_utils.py](https://codecov.io/gh/huggingface/transformers/pull/4531/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy9tb2RlbGluZ191dGlscy5weQ==) | `90.53% <0.00%> (+0.11%)` | :arrow_up: |
| [src/transformers/file\_utils.py](https://codecov.io/gh/huggingface/transformers/pull/4531/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy9maWxlX3V0aWxzLnB5) | `73.85% <0.00%> (+0.41%)` | :arrow_up: |

------

[Continue to review full report at Codecov](https://codecov.io/gh/huggingface/transformers/pull/4531?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/huggingface/transformers/pull/4531?src=pr&el=footer). Last update [e19b978...afcf0c9](https://codecov.io/gh/huggingface/transformers/pull/4531?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
[Community notebooks] add longformer-for-qa notebook
Author: patil-suraj
Pull request: yes

This PR adds a community notebook to showcase how to fine-tune Longformer for QA task.

@ibeltagy  @patrickvonplaten Please provide feedback if you think this notebook can be further improved.


Comments:
# [Codecov](https://codecov.io/gh/huggingface/transformers/pull/4652?src=pr&el=h1) Report
> Merging [#4652](https://codecov.io/gh/huggingface/transformers/pull/4652?src=pr&el=desc) into [master](https://codecov.io/gh/huggingface/transformers/commit/5e737018e1fcb22c8b76052058279552a8d6c806&el=desc) will **decrease** coverage by `0.00%`.
> The diff coverage is `n/a`.

[![Impacted file tree graph](https://codecov.io/gh/huggingface/transformers/pull/4652/graphs/tree.svg?width=650&height=150&src=pr&token=9qOlN6Hb1c)](https://codecov.io/gh/huggingface/transformers/pull/4652?src=pr&el=tree)

```diff
@@            Coverage Diff             @@
##           master    #4652      +/-   ##
==========================================
- Coverage   77.19%   77.19%   -0.01%     
==========================================
  Files         128      128              
  Lines       21021    21021              
==========================================
- Hits        16228    16227       -1     
- Misses       4793     4794       +1     
```


| [Impacted Files](https://codecov.io/gh/huggingface/transformers/pull/4652?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [src/transformers/file\_utils.py](https://codecov.io/gh/huggingface/transformers/pull/4652/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy9maWxlX3V0aWxzLnB5) | `73.44% <0.00%> (-0.42%)` | :arrow_down: |
| [src/transformers/modeling\_tf\_utils.py](https://codecov.io/gh/huggingface/transformers/pull/4652/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy9tb2RlbGluZ190Zl91dGlscy5weQ==) | `88.50% <0.00%> (-0.17%)` | :arrow_down: |
| [src/transformers/trainer.py](https://codecov.io/gh/huggingface/transformers/pull/4652/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy90cmFpbmVyLnB5) | `39.47% <0.00%> (+0.23%)` | :arrow_up: |

------

[Continue to review full report at Codecov](https://codecov.io/gh/huggingface/transformers/pull/4652?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/huggingface/transformers/pull/4652?src=pr&el=footer). Last update [5e73701...88eab9f](https://codecov.io/gh/huggingface/transformers/pull/4652?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).


One small comment, the model name was changed in a recent commit to `allenai/longformer-base-4096'`.


The notebook looks great - thanks @patil-suraj ! Apart from @ibeltagy's suggestion, I think it's great!

Great, thank you! I've updated the model paths.
[Longformer] Notebook to train Longformer
Author: ibeltagy
Pull request: yes

This PR adds a community notebook that demonstrates how we pretrained Longformer starting from the RoBERTa checkpoint. The same procedure can be followed to convert other existing pretrained models into their Long version.

@patrickvonplaten, @patil-suraj, it would be great if you also check the notebook. Any comments are welcomed. 

Comments:
# [Codecov](https://codecov.io/gh/huggingface/transformers/pull/4642?src=pr&el=h1) Report
> Merging [#4642](https://codecov.io/gh/huggingface/transformers/pull/4642?src=pr&el=desc) into [master](https://codecov.io/gh/huggingface/transformers/commit/5e737018e1fcb22c8b76052058279552a8d6c806&el=desc) will **not change** coverage.
> The diff coverage is `n/a`.

[![Impacted file tree graph](https://codecov.io/gh/huggingface/transformers/pull/4642/graphs/tree.svg?width=650&height=150&src=pr&token=9qOlN6Hb1c)](https://codecov.io/gh/huggingface/transformers/pull/4642?src=pr&el=tree)

```diff
@@           Coverage Diff           @@
##           master    #4642   +/-   ##
=======================================
  Coverage   77.19%   77.19%           
=======================================
  Files         128      128           
  Lines       21021    21021           
=======================================
  Hits        16228    16228           
  Misses       4793     4793           
```


| [Impacted Files](https://codecov.io/gh/huggingface/transformers/pull/4642?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [src/transformers/file\_utils.py](https://codecov.io/gh/huggingface/transformers/pull/4642/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy9maWxlX3V0aWxzLnB5) | `73.44% <0.00%> (-0.42%)` | :arrow_down: |
| [src/transformers/trainer.py](https://codecov.io/gh/huggingface/transformers/pull/4642/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy90cmFpbmVyLnB5) | `39.47% <0.00%> (+0.23%)` | :arrow_up: |

------

[Continue to review full report at Codecov](https://codecov.io/gh/huggingface/transformers/pull/4642?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/huggingface/transformers/pull/4642?src=pr&el=footer). Last update [5e73701...b86c516](https://codecov.io/gh/huggingface/transformers/pull/4642?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).


BTW @patil-suraj - you can also add your longformer squad notebook to the community notebooks if you want

Made a copy and added some comments here: https://colab.research.google.com/drive/1kS2yerrLwLnc-hM6PlisFVaJT98kUKRV#scrollTo=4TTSvW8MlKJJ 

I think the use case is really nice! For better engagement with the notebook, I think it can be polished a bit in terms of small code refactoring / better descriptions. I left some TODO: there as suggestions :-) 

@patrickvonplaten, thanks for the review, the notebook is much better now after incorporating your suggestions. 

Looks great merging!
Data Processor should not include in the package
Author: Liangtaiwan
Pull request: no
Labels: name: wontfix

# 🚀 Feature request

For a clearly, flexible packages, I think there is no need including data processor for certain data.

## Motivation

In the past, it was easy using transformers to do research, since the module is explicit and clear.
When reading example, it was easy tracing the code to apply on a new dataset.
However, recently transformers merged some unrelated module into the package like Data Processor, making hard to modify the preprocess stage.

Comments:
Agree and I follow you. 

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
Add example code for CRF heads 
Author: srush
Pull request: no
Labels: name: wontfix



# Add example `crf/` with heads that force structural output dependencies. 

(Mostly a note to myself as a side project)

## Model description

As requested in: https://github.com/huggingface/transformers/pull/3009 there are some tasks and languages where it is useful to have final layer structural dependencies. 

Using https://github.com/harvardnlp/pytorch-struct/ we can add these with minimal changes to the code and no new model parameters. 

Target:
* example code for ner / parsing (sota). 



Comments:
Also https://github.com/huggingface/transformers/pull/2249/

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
Add sample softmax possibility to TransfoXL model for TransfoXL training
Author: patrickvonplaten
Pull request: no
Labels: name: wontfix

# 🚀 Feature request

TransfoXL samples the logits during training if required. At the moment TransfoXL can only be used whithout sampling from the logits during training. A partrly finished implementation can be found under the branch: `add_sampling_and_training_to_transfo_xl_models` .

## Motivation

To be able to train tranfoXL correctly.

## Your contribution

Already looked into the issue. Could try to implement it correctly with help from @thomwolf and @LysandreJik . Not a priority at the moment though.


Comments:
This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
Only enable some labels in BERT fine-tuned NER
Author: jmamou
Pull request: no
Labels: name: wontfix

name: Usage

# ❓ Questions & Help
Only enable some labels in BERT fine-tuned NER

## Details
I would be interested to enable only some labels in BERT fine-tuned NER prediction. 
For example, I know what are the entities and I would like to train a model that will classify entities - but I am not interested to train/predict O tag.
What would be the best way to do it?
Thanks

Comments:
I'm not sure I understand your question. The O tag is here in order to identify a token which is not an entity in the IOB1 tagging scheme. If you don't have such a tag, every token will have to be classified as an entity, which does not make sense?

If you would like to define a custom tagging scheme and train a model to predict on that tagging scheme, you would have to create a dataset for that and train your model on that dataset.

Thanks @LysandreJik 
Let me provide some additional details. Given a sentence, I use some external resources to find what are the candidates for tagging. For the candidates, I need to classify between 2 different labels (binary classification). That's the reason why I wrote that I am not interested in predicting the O tag, since I use external resources for it. 
I have data for train/test. 

Hey @jmamou , we might be able to help. Mind sending me an email at clement [at] huggingface [dot] co?

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
[Longformer] fix model name in examples
Author: ibeltagy
Pull request: yes

This PR fixes the model identifier of Longformer to the new standard fomat <organisation/model_name>

Comments:
# [Codecov](https://codecov.io/gh/huggingface/transformers/pull/4653?src=pr&el=h1) Report
> Merging [#4653](https://codecov.io/gh/huggingface/transformers/pull/4653?src=pr&el=desc) into [master](https://codecov.io/gh/huggingface/transformers/commit/b5015a2a0f4ea63035a877f5626cb0c3ce97e25d&el=desc) will **increase** coverage by `0.00%`.
> The diff coverage is `n/a`.

[![Impacted file tree graph](https://codecov.io/gh/huggingface/transformers/pull/4653/graphs/tree.svg?width=650&height=150&src=pr&token=9qOlN6Hb1c)](https://codecov.io/gh/huggingface/transformers/pull/4653?src=pr&el=tree)

```diff
@@           Coverage Diff           @@
##           master    #4653   +/-   ##
=======================================
  Coverage   77.19%   77.20%           
=======================================
  Files         128      128           
  Lines       21021    21021           
=======================================
+ Hits        16228    16230    +2     
+ Misses       4793     4791    -2     
```


| [Impacted Files](https://codecov.io/gh/huggingface/transformers/pull/4653?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [src/transformers/modeling\_longformer.py](https://codecov.io/gh/huggingface/transformers/pull/4653/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy9tb2RlbGluZ19sb25nZm9ybWVyLnB5) | `96.82% <ø> (ø)` | |
| [src/transformers/modeling\_tf\_utils.py](https://codecov.io/gh/huggingface/transformers/pull/4653/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy9tb2RlbGluZ190Zl91dGlscy5weQ==) | `88.83% <0.00%> (+0.16%)` | :arrow_up: |
| [src/transformers/file\_utils.py](https://codecov.io/gh/huggingface/transformers/pull/4653/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy9maWxlX3V0aWxzLnB5) | `73.85% <0.00%> (+0.41%)` | :arrow_up: |

------

[Continue to review full report at Codecov](https://codecov.io/gh/huggingface/transformers/pull/4653?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/huggingface/transformers/pull/4653?src=pr&el=footer). Last update [b5015a2...dfce20a](https://codecov.io/gh/huggingface/transformers/pull/4653?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).