noamaanMulla-03/transformers-issues huggingface dataset:
Fixed DataCollatorForLanguageModeling not accepting lists of lists
Author: TevenLeScao
Pull request: yes

As discussed on Slack, currently `DataCollatorForLanguageModeling` and `DataCollatorForPermutationLanguageModeling` cannot take in lists of lists, as opposed to `default_data_collator`. This fixes this issue by calling `torch.Tensor` beforehand if a list of lists is detected.

Comments:
# [Codecov](https://codecov.io/gh/huggingface/transformers/pull/6685?src=pr&el=h1) Report
> Merging [#6685](https://codecov.io/gh/huggingface/transformers/pull/6685?src=pr&el=desc) into [master](https://codecov.io/gh/huggingface/transformers/commit/1a779ad7ecb9e5215b6bd1cfa0153469d37e4274?el=desc) will **decrease** coverage by `0.00%`.
> The diff coverage is `75.00%`.

[![Impacted file tree graph](https://codecov.io/gh/huggingface/transformers/pull/6685/graphs/tree.svg?width=650&height=150&src=pr&token=9qOlN6Hb1c)](https://codecov.io/gh/huggingface/transformers/pull/6685?src=pr&el=tree)

```diff
@@            Coverage Diff             @@
##           master    #6685      +/-   ##
==========================================
- Coverage   79.65%   79.64%   -0.01%     
==========================================
  Files         156      156              
  Lines       28250    28254       +4     
==========================================
+ Hits        22503    22504       +1     
- Misses       5747     5750       +3     
```


| [Impacted Files](https://codecov.io/gh/huggingface/transformers/pull/6685?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [src/transformers/data/data\_collator.py](https://codecov.io/gh/huggingface/transformers/pull/6685/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy9kYXRhL2RhdGFfY29sbGF0b3IucHk=) | `89.70% <75.00%> (-1.21%)` | :arrow_down: |
| [src/transformers/generation\_tf\_utils.py](https://codecov.io/gh/huggingface/transformers/pull/6685/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy9nZW5lcmF0aW9uX3RmX3V0aWxzLnB5) | `86.46% <0.00%> (-0.26%)` | :arrow_down: |

------

[Continue to review full report at Codecov](https://codecov.io/gh/huggingface/transformers/pull/6685?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/huggingface/transformers/pull/6685?src=pr&el=footer). Last update [1a779ad...dd1b689](https://codecov.io/gh/huggingface/transformers/pull/6685?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
BUILD upgrade to isort v5
Author: MarcoGorelli
Pull request: no

The contributing guide says

> Right now, we need an unreleased version of isort to avoid a bug:

However, it looks like the linked PR has been incorporated into the latest version of isort. I could submit a PR to address this if it would be welcome

Comments:
which linked PR are you referring to?

this one https://github.com/timothycrosley/isort/pull/1000 (it's linked to here https://huggingface.co/transformers/contributing.html )
run_glue.py - Import Error
Author: yukioichida
Pull request: no

## 🐛 Bug

<!-- Important information -->

Model I am using (Bert, XLNet....): XLNET (from example/run_glue.py)

The problem arise when using:
* [ ] the official example scripts: run_glue.py
- Stacktrace:
Traceback (most recent call last):
  File "run_glue.py", line 49, in <module>
    from transformers import glue_compute_metrics as compute_metrics
ImportError: cannot import name 'glue_compute_metrics'


The tasks I am working on is:
* [ ] an official GLUE/SQUaD task: MNLI

## To Reproduce

Steps to reproduce the behavior:
1. Execute run_glue.py after install requirements.
## Environment

* OS: Linux - Ubuntu
* Python version: 3.6
* PyTorch version: 1.2.0
* PyTorch Transformers version (or branch): tag 2.0.0

## Additional context
I found that some scripts related to glue tasks were not in *transformer* directory, which causes the import problem.But I really don't know if it may be a project setup issue or the files that contains glue utility code should be in */transformer* dir instead of */transformer/data/metrics*.

Comments:
Hi! You may have seen this warning when importing from our library: `To use data.metrics please install scikit-learn. See https://scikit-learn.org/stable/index.html`.

Do you have this issue even with scikit-learn installed?

Hi @LysandreJik ,

Thanks for the answer. I already have scikit-learn but actually I'm working using a conda environment with *transformer* package installed using conda pip.
However, I solved this issue exporting the *PYTHONNOUSERSITE=1*, which enabled the scikit-learn installed in my conda environment. I discovered this problem because some stacktraces were pointing to files contained in local packages instead of my conda environment packages.

I'll close this issue. Thanks for the explanation.

*OBS*: Do you guys intend to publish a conda package of transformers?



I face the exact same issue, even though `scikit-learn` is installed. The steps to reproduce are exactly identical, and I built from source.

I tried the `PYTHONNOUSERSITE=1` solution, but that does not change things because I can already import `sklearn` from the shell, and all required packages are in the conda environment.

Repeating the stack trace
```
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
ImportError: cannot import name 'glue_compute_metrics' from 'transformers' (path/to/anaconda3/envs/nlp/lib/python3.7/site-packages/transformers/__init__.py)
```


# Environment
- OS: `Springdale Linux 7.7 (Verona)`
- Python version: `3.7.6`
- PyTorch version: `1.3.1`
- PyTorch Transformers version (or branch): `2.4.1`

> Hi! You may have seen this warning when importing from our library: `To use data.metrics please install scikit-learn. See https://scikit-learn.org/stable/index.html`.
> 
> Do you have this issue even with scikit-learn installed?

> Yes :(

I fixed it by downgrading the python from 3.7.7 to 3.7.0:
```
conda install python=3.7.0
```

fixed for me by adding "import sklearn" to run_glue.py before the imports from transformers.
fix incorrect codecov reports
Author: stas00
Pull request: yes

As discussed at https://github.com/huggingface/transformers/issues/6317 codecov currently sends an invalid report when it fails to find a code coverage report for the base it checks against. When this happens it goes looking for the nearest hash with report, which often leads to a report that is not representative of the true impact of the proposed PR.

This PR fixes it by adding:

 `require_base: yes`        # don't report if there is no base coverage report

And then:

-  let's add this for clarity, this supposedly is already the default.

    `require_head: yes`        # don't report if there is no head coverage report 

- and perhaps no point reporting on doc changes as they don't make any difference to the coverage and the 0% change comment just generates noise:

   `require_changes: true`    # only comment if there was a change in coverage

These options are documented here: https://docs.codecov.io/docs/codecovyml-reference#comment

Comments:
Great, thanks for diving into it @stas00!

Alas, it solved only part of the problem - the main issue is still there https://github.com/huggingface/transformers/pull/6553 :(
Create PULL_REQUEST_TEMPLATE.md
Author: stas00
Pull request: yes

Proposing to copy this neat feature from pytorch. This is a small template that let's a PR submitter tell which issue that PR closes.

Here is an example of it in action, the end of the top post: https://github.com/huggingface/transformers/pull/6659

Comments:
# [Codecov](https://codecov.io/gh/huggingface/transformers/pull/6660?src=pr&el=h1) Report
> Merging [#6660](https://codecov.io/gh/huggingface/transformers/pull/6660?src=pr&el=desc) into [master](https://codecov.io/gh/huggingface/transformers/commit/0f94151dc7809128b40ab68ba164742fe1c5b4e6?el=desc) will **increase** coverage by `0.64%`.
> The diff coverage is `n/a`.

[![Impacted file tree graph](https://codecov.io/gh/huggingface/transformers/pull/6660/graphs/tree.svg?width=650&height=150&src=pr&token=9qOlN6Hb1c)](https://codecov.io/gh/huggingface/transformers/pull/6660?src=pr&el=tree)

```diff
@@            Coverage Diff             @@
##           master    #6660      +/-   ##
==========================================
+ Coverage   79.01%   79.65%   +0.64%     
==========================================
  Files         156      156              
  Lines       28248    28248              
==========================================
+ Hits        22320    22502     +182     
+ Misses       5928     5746     -182     
```


| [Impacted Files](https://codecov.io/gh/huggingface/transformers/pull/6660?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [src/transformers/modeling\_tf\_flaubert.py](https://codecov.io/gh/huggingface/transformers/pull/6660/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy9tb2RlbGluZ190Zl9mbGF1YmVydC5weQ==) | `24.53% <0.00%> (-63.81%)` | :arrow_down: |
| [src/transformers/modeling\_tf\_utils.py](https://codecov.io/gh/huggingface/transformers/pull/6660/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy9tb2RlbGluZ190Zl91dGlscy5weQ==) | `86.64% <0.00%> (+1.30%)` | :arrow_up: |
| [src/transformers/modeling\_tf\_electra.py](https://codecov.io/gh/huggingface/transformers/pull/6660/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy9tb2RlbGluZ190Zl9lbGVjdHJhLnB5) | `98.95% <0.00%> (+73.82%)` | :arrow_up: |

------

[Continue to review full report at Codecov](https://codecov.io/gh/huggingface/transformers/pull/6660?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/huggingface/transformers/pull/6660?src=pr&el=footer). Last update [0f94151...73edfdc](https://codecov.io/gh/huggingface/transformers/pull/6660?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
[doc] bart doc examples aren't for bart
Author: stas00
Pull request: no

If you look at the very end of this section https://huggingface.co/transformers/model_doc/bart.html#transformers.BartForConditionalGeneration.generate
there are 5 examples of using `generate` none of which is for BART. Is this an accidental copy-n-paste issue and they should just be removed? 

There are examples of generate for BART in the pre-amble of the class:
https://huggingface.co/transformers/model_doc/bart.html#bartforconditionalgeneration



Comments:
Those examples are from the `generate` method doc which is a generic method shared by all generative models.

Thank you, @patil-suraj. Indeed, since `BartForConditionalGeneration` uses super-super class's `generate` it ends up having that generic signature in its docs.

What I'm trying to say is that these example are confusing to the user since not only they are irrelevant to someone trying to use BART, there isn't even an example of using bart in that part of the doc (there is one earlier in the class signature).

e.g. they don't show up in other similar classes which have `generate`
https://huggingface.co/transformers/model_doc/t5.html#t5forconditionalgeneration
https://huggingface.co/transformers/model_doc/marian.html
but there they have their own `generate` methods, so this doesn't happen.

I'm trying to flag a poor user experience and asking whether perhaps there is a better way to do it?

One possible suggestion:
- remove the 5 examples from `generate` at https://github.com/huggingface/transformers/blob/master/src/transformers/generation_utils.py#L215
- replace with a note - see this class' pre-amble documentation for examples.

feel free to send a PR deleting generate here
https://github.com/huggingface/transformers/blob/master/docs/source/model_doc/bart.rst#L35


Thank you, Sam.
https://github.com/huggingface/transformers/pull/6659
[doc] remove BartForConditionalGeneration.generate
Author: stas00
Pull request: yes

As suggested here: https://github.com/huggingface/transformers/issues/6651#issuecomment-678594233
this removes a generic `generate` doc with a large group of generate examples, none of which is relevant to BART. the BART class pre-amble doc already provides the examples.

Fixes #6651

Comments:
# [Codecov](https://codecov.io/gh/huggingface/transformers/pull/6659?src=pr&el=h1) Report
> Merging [#6659](https://codecov.io/gh/huggingface/transformers/pull/6659?src=pr&el=desc) into [master](https://codecov.io/gh/huggingface/transformers/commit/0f94151dc7809128b40ab68ba164742fe1c5b4e6?el=desc) will **increase** coverage by `0.61%`.
> The diff coverage is `n/a`.

[![Impacted file tree graph](https://codecov.io/gh/huggingface/transformers/pull/6659/graphs/tree.svg?width=650&height=150&src=pr&token=9qOlN6Hb1c)](https://codecov.io/gh/huggingface/transformers/pull/6659?src=pr&el=tree)

```diff
@@            Coverage Diff             @@
##           master    #6659      +/-   ##
==========================================
+ Coverage   79.01%   79.62%   +0.61%     
==========================================
  Files         156      156              
  Lines       28248    28248              
==========================================
+ Hits        22320    22493     +173     
+ Misses       5928     5755     -173     
```


| [Impacted Files](https://codecov.io/gh/huggingface/transformers/pull/6659?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [src/transformers/modeling\_tf\_flaubert.py](https://codecov.io/gh/huggingface/transformers/pull/6659/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy9tb2RlbGluZ190Zl9mbGF1YmVydC5weQ==) | `24.53% <0.00%> (-63.81%)` | :arrow_down: |
| [src/transformers/generation\_tf\_utils.py](https://codecov.io/gh/huggingface/transformers/pull/6659/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy9nZW5lcmF0aW9uX3RmX3V0aWxzLnB5) | `84.21% <0.00%> (-2.26%)` | :arrow_down: |
| [src/transformers/modeling\_tf\_utils.py](https://codecov.io/gh/huggingface/transformers/pull/6659/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy9tb2RlbGluZ190Zl91dGlscy5weQ==) | `86.64% <0.00%> (+1.30%)` | :arrow_up: |
| [src/transformers/modeling\_tf\_electra.py](https://codecov.io/gh/huggingface/transformers/pull/6659/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy9tb2RlbGluZ190Zl9lbGVjdHJhLnB5) | `98.95% <0.00%> (+73.82%)` | :arrow_up: |

------

[Continue to review full report at Codecov](https://codecov.io/gh/huggingface/transformers/pull/6659?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/huggingface/transformers/pull/6659?src=pr&el=footer). Last update [0f94151...2b5ac03](https://codecov.io/gh/huggingface/transformers/pull/6659?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
Add model card for singbert.
Author: zyuanlim
Pull request: yes

Adding a model card for singbert- bert for singlish and manglish.

Comments:
@JetRunner Just a final addition of couple of more examples and customized the widget inputs, good to go :)

Not sure if I should bring this up here or raise a new issue, but when i tested my widget (hosted inference), i got this error: 
```
Can't load config for 'zanelim/singbert'. Make sure that: - 'zanelim/singbert' is a correct model identifier listed on 'https://huggingface.co/models' - or 'zanelim/singbert' is the correct path to a directory containing a config.json file
```
However the config file is there as confirmed by
```
>>> transformers-cli s3 ls
singbert/config.json             2020-08-24T05:54:28.000Z "6004cb287370530537f4076c9cf7fdbe"       471 
singbert/pytorch_model.bin       2020-08-24T05:53:37.000Z "c060a644d84e55b2f93aa67fb1f35956" 440509997 
singbert/special_tokens_map.json 2020-08-24T05:54:23.000Z "8b3fb1023167bb4ab9d70708eb05f6ec"       112 
singbert/tf_model.h5             2020-08-24T05:52:23.000Z "45a8eea544f73079768bb136fe3d0a27" 536061440 
singbert/tokenizer_config.json   2020-08-24T05:54:25.000Z "8b3fb1023167bb4ab9d70708eb05f6ec"       112 
singbert/vocab.txt               2020-08-24T05:53:32.000Z "767659dd848f37f6937a0ffb833ee6b1"    224170 
```

> Not sure if I should bring this up here or raise a new issue, but when i tested my widget (hosted inference), i got this error: 
> 
> ```
> 
> Can't load config for 'zanelim/singbert'. Make sure that: - 'zanelim/singbert' is a correct model identifier listed on 'https://huggingface.co/models' - or 'zanelim/singbert' is the correct path to a directory containing a config.json file
> 
> ```
> 
> However the config file is there as confirmed by
> 
> ```
> 
> >>> transformers-cli s3 ls
> 
> singbert/config.json             2020-08-24T05:54:28.000Z "6004cb287370530537f4076c9cf7fdbe"       471 
> 
> singbert/pytorch_model.bin       2020-08-24T05:53:37.000Z "c060a644d84e55b2f93aa67fb1f35956" 440509997 
> 
> singbert/special_tokens_map.json 2020-08-24T05:54:23.000Z "8b3fb1023167bb4ab9d70708eb05f6ec"       112 
> 
> singbert/tf_model.h5             2020-08-24T05:52:23.000Z "45a8eea544f73079768bb136fe3d0a27" 536061440 
> 
> singbert/tokenizer_config.json   2020-08-24T05:54:25.000Z "8b3fb1023167bb4ab9d70708eb05f6ec"       112 
> 
> singbert/vocab.txt               2020-08-24T05:53:32.000Z "767659dd848f37f6937a0ffb833ee6b1"    224170 
> 
> ```

Don't worry. It's sometimes flaky. As long as you can load the model with from_pretrained and you are good to go. 😉
[squad] make examples and dataset accessible from SquadDataset object
Author: lazovich
Pull request: yes

In order to do evaluation on the SQuAD dataset using `squad_evaluate`, the user needs access to both the examples loaded in the dataset and the `TensorDataset` that contains values like `unique_id` and the like that are used in constructing the list of `SquadResult` objects. This PR surfaces the examples and dataset to the user so that they can access it directly.

For example of why access to those is needed, see how evaluation is currently done in `examples/run_squad.py`. The `SquadDataset` object attempts to wrap up some of this functionality, but without access to examples and dataset the evaluation is not possible.

Comments:
There seem to be some issues with CircleCI right now causing all the integration tests to fail. Please let me know if there is an issue on my end.
Can't load config for New Community Model
Author: donal1
Pull request: no

API says "Can't load config for 'donal/Pro_Berta'. Make sure that: - 'donal/Pro_Berta' is a correct model identifier listed on 'https://huggingface.co/models' - or 'donal/Pro_Berta' is the correct path to a directory containing a config.json file". But I followed the instructions to the letter. Do not know what's the issue. Please fix. 

https://huggingface.co/donal/Pro_Berta?text=The+goal+of+life+is+%3Cmask%3E.

Pinging @mfuntowicz, @julien-c, 

All the files seem to be in the right place.

Your file now lives at:
https://s3.amazonaws.com/models.huggingface.co/bert/donal/Pro_Berta/merges.txt
Your file now lives at:
https://s3.amazonaws.com/models.huggingface.co/bert/donal/Pro_Berta/special_tokens_map.json
Your file now lives at:
https://s3.amazonaws.com/models.huggingface.co/bert/donal/Pro_Berta/training_args.bin
Your file now lives at:
https://s3.amazonaws.com/models.huggingface.co/bert/donal/Pro_Berta/pytorch_model.bin
Your file now lives at:
https://s3.amazonaws.com/models.huggingface.co/bert/donal/Pro_Berta/config.json
Your file now lives at:
https://s3.amazonaws.com/models.huggingface.co/bert/donal/Pro_Berta/tokenizer_config.json
Your file now lives at:
https://s3.amazonaws.com/models.huggingface.co/bert/donal/Pro_Berta/vocab.json

Comments:
Edit so it appears to work with: 

from transformers import AutoTokenizer, AutoModelWithLMHead
tokenizer = AutoTokenizer.from_pretrained("donal/Pro_Berta")
model = AutoModelWithLMHead.from_pretrained("donal/Pro_Berta")

Could just be an issue with the API? maybe needs some time to load correctly?



They fixed it
Use separate tqdm progressbars
Author: sgugger
Pull request: yes

Currently, we close progress bar and break the inner loops when training is complete which yields to the progress bars indicating one less step than actually done. The user might believe the last step was not done (even if it was) so this PR uses different generators for the loops than the progress bar and manually updates the progress bar (tried to just do manually the last update but tqdm doesn't want to cooperate with that approach).


Comments:
# [Codecov](https://codecov.io/gh/huggingface/transformers/pull/6696?src=pr&el=h1) Report
> Merging [#6696](https://codecov.io/gh/huggingface/transformers/pull/6696?src=pr&el=desc) into [master](https://codecov.io/gh/huggingface/transformers/commit/6b4c617666fd26646d44d54f0c45dfe1332b12ca?el=desc) will **decrease** coverage by `0.03%`.
> The diff coverage is `80.00%`.

[![Impacted file tree graph](https://codecov.io/gh/huggingface/transformers/pull/6696/graphs/tree.svg?width=650&height=150&src=pr&token=9qOlN6Hb1c)](https://codecov.io/gh/huggingface/transformers/pull/6696?src=pr&el=tree)

```diff
@@            Coverage Diff             @@
##           master    #6696      +/-   ##
==========================================
- Coverage   79.44%   79.41%   -0.04%     
==========================================
  Files         156      156              
  Lines       28386    28390       +4     
==========================================
- Hits        22551    22545       -6     
- Misses       5835     5845      +10     
```


| [Impacted Files](https://codecov.io/gh/huggingface/transformers/pull/6696?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [src/transformers/trainer.py](https://codecov.io/gh/huggingface/transformers/pull/6696/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy90cmFpbmVyLnB5) | `50.88% <80.00%> (+0.15%)` | :arrow_up: |
| [src/transformers/generation\_tf\_utils.py](https://codecov.io/gh/huggingface/transformers/pull/6696/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy9nZW5lcmF0aW9uX3RmX3V0aWxzLnB5) | `84.21% <0.00%> (-2.26%)` | :arrow_down: |

------

[Continue to review full report at Codecov](https://codecov.io/gh/huggingface/transformers/pull/6696?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/huggingface/transformers/pull/6696?src=pr&el=footer). Last update [6b4c617...96d9c81](https://codecov.io/gh/huggingface/transformers/pull/6696?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).