noamaanMulla-03/transformers-issues huggingface dataset:
Create model card
Author: mrm8488
Pull request: yes
Labels: name: model card

Model: TinyBERT-spanish-uncased-finetuned-ner

Comments:
# [Codecov](https://codecov.io/gh/huggingface/transformers/pull/3890?src=pr&el=h1) Report
> Merging [#3890](https://codecov.io/gh/huggingface/transformers/pull/3890?src=pr&el=desc) into [master](https://codecov.io/gh/huggingface/transformers/commit/eb5601b0a5a88824a2598956f96e06e7f2422bce&el=desc) will **decrease** coverage by `0.03%`.
> The diff coverage is `n/a`.

[![Impacted file tree graph](https://codecov.io/gh/huggingface/transformers/pull/3890/graphs/tree.svg?width=650&height=150&src=pr&token=9qOlN6Hb1c)](https://codecov.io/gh/huggingface/transformers/pull/3890?src=pr&el=tree)

```diff
@@            Coverage Diff             @@
##           master    #3890      +/-   ##
==========================================
- Coverage   78.57%   78.53%   -0.04%     
==========================================
  Files         106      106              
  Lines       17962    17962              
==========================================
- Hits        14113    14106       -7     
- Misses       3849     3856       +7     
```


| [Impacted Files](https://codecov.io/gh/huggingface/transformers/pull/3890?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [src/transformers/modeling\_tf\_utils.py](https://codecov.io/gh/huggingface/transformers/pull/3890/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy9tb2RlbGluZ190Zl91dGlscy5weQ==) | `91.28% <0.00%> (-1.32%)` | :arrow_down: |
| [src/transformers/modeling\_utils.py](https://codecov.io/gh/huggingface/transformers/pull/3890/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy9tb2RlbGluZ191dGlscy5weQ==) | `91.06% <0.00%> (+0.12%)` | :arrow_up: |

------

[Continue to review full report at Codecov](https://codecov.io/gh/huggingface/transformers/pull/3890?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/huggingface/transformers/pull/3890?src=pr&el=footer). Last update [eb5601b...fe606bb](https://codecov.io/gh/huggingface/transformers/pull/3890?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
GPT2LMHeadModel with variable length batch input
Author: plstory
Pull request: no
Labels: name: Help wanted

name: wontfix

name: PyTorch

I'm trying to repurpose the GPT2LMHeadModel for a seq2seq-like task, where I have an input prompt sequence of length L and I'm trying to ask the model to output a sequence to match a target sequence/sentence.
For a single input-output pair, I simply change the original code of 
`shift_logits = lm_logits[..., :-1, :].contiguous()`
to
`shift_logits = lm_logits[..., L-1:-1, :].contiguous()`

But I'm a bit lost on how can I do this for a batch of variable length input. Even if I pad the shorter sequences, I would need to shift the logits by different amount for each input. I'm also uncertain if I need to do something about the attention mask. Any tip is appreicated!

Comments:
Have you tried concatenating the sequences into one long string and using a separator token without changing any of the code? You can then use a moving window of 1024 to train the model. You can make each step of the window start after an 
 to ensure the primary sequence is not truncated.

You can then train on batches of these moving windows of 1024 (either moving a random # of tokens or to the next 
 token)

E.g., An example input for 1024 may then look something like this: "Some sequence == Some other sequence 
 Some sequence_2 == Some other sequence_2 
 Some sequence_3 == Some other sequence_3 
 Some sequence_4 == Some othe" (clipped on purpose to illustrate a point.)

Then you prompt with "Some sequence ==" and terminate generation/clip text on 


The model is *very good* at learning like this. It is okay to have the window clip things off at the end.

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
Failing slow RobertaModelIntegrationTest
Author: sshleifer
Pull request: no
Labels: name: wontfix

name: Core: Modeling

name: Tests

```
RUN_SLOW=1 pytest tests/test_modeling_roberta.py::RobertaModelIntegrationTest::test_inference_masked_lm
```
Have not investigated at all, but wanted to record.

Traceback:
```
self = <tests.test_modeling_roberta.RobertaModelIntegrationTest testMethod=test_inference_masked_lm>

    @slow
    def test_inference_masked_lm(self):
        model = RobertaForMaskedLM.from_pretrained("roberta-base")

        input_ids = torch.tensor([[0, 31414, 232, 328, 740, 1140, 12695, 69, 46078, 1588, 2]])
        output = model(input_ids)[0]
        expected_shape = torch.Size((1, 11, 50265))
        self.assertEqual(output.shape, expected_shape)
        # compare the actual values for a slice.
        expected_slice = torch.Tensor(
            [[[33.8843, -4.3107, 22.7779], [4.6533, -2.8099, 13.6252], [1.8222, -3.6898, 8.8600]]]
        )
>       self.assertTrue(torch.allclose(output[:, :3, :3], expected_slice, atol=1e-3))
E       AssertionError: False is not true
```

Comments:
This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
Failing slow AutoModelTest/BertForPreTraining
Author: sshleifer
Pull request: no
Labels: name: wontfix

name: Core: Pipeline

name: Tests

```
RUN_SLOW=1 pytest tests/test_modeling_auto.py::AutoModelTest::test_model_for_pretraining_from_pretrained
```
Have not investigated at all, but wanted to record since the slow test failures are evasive :)


Clues:
model: `transformers.modeling_bert.BertForPreTraining`
```
loading_info = {'missing_keys': ['cls.predictions.decoder.bias'], 'unexpected_keys': [], 'error_msgs': []}
```
Likely related to this funkiness

https://github.com/huggingface/transformers/blob/ee5de0ba449d638da704e1c03ffcc20a930f5589/src/transformers/modeling_bert.py#L482-L483

Comments:
This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
Error reported when running ''run_language_modeling.py" file
Author: XIN-von-SUN
Pull request: no
Labels: name: Need more information

name: wontfix

name: Core: Modeling

name: Version mismatch

# 🐛 Bug

## Information

Model I am using (Bert and RoBerta):

Language I am using the model on (English).

The problem arises when using:
* [ ] the official example scripts: (give details below)
I followed the tutorial of how to fine tuning the Bert model on own corpus data, and used recommended corpus 'wiki-text-2' corpus to fine tune the Bert Model. 
However there is always an error appears: "RuntimeError: Assertion `cur_target >= 0 && cur_target < n_classes' failed.  at ../aten/src/THNN/generic/ClassNLLCriterion.c:97". 
Thus I am not sure are there something wrong with the "run_language_modeling.py" file, because I did do any change of the original code and used recommended wiki corpus. Could you help me check this error?

* [ ] my own modified scripts: (give details below)
None

The tasks I am working on is:
* [ ] an official GLUE/SQUaD task: (give the name)
* [ ] my own task or dataset: (give details below)
Fine tuning the language model of Bert on our own customer corpus data.

## To reproduce

Steps to reproduce the behavior:

1.
2.
3.

<!-- If you have code snippets, error messages, stack traces please provide them here as well.
     Important! Use code tags to correctly format your code. See https://help.github.com/en/github/writing-on-github/creating-and-highlighting-code-blocks#syntax-highlighting
     Do not use screenshots, as they are hard to read and (more importantly) don't allow others to copy-and-paste your code.-->

## Expected behavior

<!-- A clear and concise description of what you would expect to happen. -->

## Environment info
<!-- You can run the command `python transformers-cli env` and copy-and-paste its output below.
     Don't forget to fill out the missing fields in that output! -->
     
- `transformers` version:
- Platform:
- Python version:
- PyTorch version (GPU?):
- Tensorflow version (GPU?):
- Using GPU in script?:
- Using distributed or parallel set-up in script?:

Comments:
Hi, this is probably due to a version mismatch. Can you update your repository to be on the same version than the script's ? 

If it's `run_language_modeling` (was `run_lm_finetuning` up until very recently), that would be version 2.4.1 (safe, but the script may have evolved a bit since the release 13 days ago) or `master` (safer, should work 100%).

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
default output of BertModel.from_pretrained('bert-base-uncased')
Author: xiaolin-cheng
Pull request: no
Labels: name: wontfix

name: Usage

By default `output = BertModel.from_pretrained('bert-base-uncased')` is a 2-tuple where `output[0]` is the hidden states of the last layer, but how is `output[1]` computed? It doesn't seem to be average of the last layer hidden states vectors over multiple tokens. I am trying to leverage output as sentence embedding, not sure if I should use `output[1]`. Thank you!

Comments:
See https://huggingface.co/transformers/v1.2.0/_modules/pytorch_transformers/modeling_bert.html#BertModel

I think this is your output[1]:
"Last layer hidden-state of the first token of the sequence (classification token) further processed by a Linear layer and a Tanh activation function. The Linear layer weights are trained from the next sentence prediction (classification) objective during Bert pretraining. This output is usually **not** a good summary of the semantic content of the input, you’re often better with averaging or pooling the sequence of hidden-states for the whole input sequence."

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
LineByLineTextDataset limits the total number of examples to 50000 documents
Author: questpavan
Pull request: no

# ❓ Questions & Help

<!-- The GitHub issue tracker is primarly intended for bugs, feature requests,
     new models and benchmarks, and migration questions. For all other questions,
     we direct you to Stack Overflow (SO) where a whole community of PyTorch and
     Tensorflow enthusiast can help you out. Make sure to tag your question with the
     right deep learning framework as well as the huggingface-transformers tag: 
     https://stackoverflow.com/questions/tagged/huggingface-transformers 
     
     If your question wasn't answered after a period of time on Stack Overflow, you
     can always open a question on GitHub. You should then link to the SO question 
     that you posted.
     -->

## Details
<!-- Description of your issue -->
Hi,

While running run_language_modeling.py with argument --line_by_line, found out that it only takes first 50000 documents.

https://github.com/huggingface/transformers/blob/cb3c2212c79d7ff0a4a4e84c3db48371ecc1c15d/src/transformers/data/datasets/language_modeling.py#L93

Please confirm is this a bug or intended behavior?


Comments:
It's a bug. Do you want to submit a PR to remove?

Yes that would be great

Closed by #3926
Fix Torch.hub + Integration test
Author: julien-c
Pull request: yes

- Torch.hub doesn't use pip-installed versions of modules, but uses a [custom importer instead](https://github.com/pytorch/pytorch/blob/master/torch/hub.py#L70-L83) (it imports `hubconf.py`) which means that:
  - all imports from hubconf.py refer to src.transformers instead of transformers
  - all imports inside the lib's code **must** be relative, i.e. shouldn't assume that the transformers module is installed (it's not)
- Added a GitHub action workflow to ensure that `hub.list` and `hub.help` always work.

Comments:
(Hmm, GitHub was failing earlier today, and now it seems to have posted my comment multiple times. Sorry about that.)

See cbbb3c43c55d2d93a156fc80bd12f31ecbac8520
change scheduler.get_last_lr to get_lr to avoid bug
Author: TobiasLee
Pull request: yes

Learning rate scheduler function for getting last lr is `get_lr` instead of `get_last_lr`.

Comments:
# [Codecov](https://codecov.io/gh/huggingface/transformers/pull/3918?src=pr&el=h1) Report
> Merging [#3918](https://codecov.io/gh/huggingface/transformers/pull/3918?src=pr&el=desc) into [master](https://codecov.io/gh/huggingface/transformers/commit/cb3c2212c79d7ff0a4a4e84c3db48371ecc1c15d&el=desc) will **decrease** coverage by `0.00%`.
> The diff coverage is `0.00%`.

[![Impacted file tree graph](https://codecov.io/gh/huggingface/transformers/pull/3918/graphs/tree.svg?width=650&height=150&src=pr&token=9qOlN6Hb1c)](https://codecov.io/gh/huggingface/transformers/pull/3918?src=pr&el=tree)

```diff
@@            Coverage Diff             @@
##           master    #3918      +/-   ##
==========================================
- Coverage   78.45%   78.45%   -0.01%     
==========================================
  Files         111      111              
  Lines       18521    18521              
==========================================
- Hits        14531    14530       -1     
- Misses       3990     3991       +1     
```


| [Impacted Files](https://codecov.io/gh/huggingface/transformers/pull/3918?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [src/transformers/trainer.py](https://codecov.io/gh/huggingface/transformers/pull/3918/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy90cmFpbmVyLnB5) | `43.59% <0.00%> (ø)` | |
| [src/transformers/modeling\_tf\_utils.py](https://codecov.io/gh/huggingface/transformers/pull/3918/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy9tb2RlbGluZ190Zl91dGlscy5weQ==) | `92.59% <0.00%> (-0.17%)` | :arrow_down: |

------

[Continue to review full report at Codecov](https://codecov.io/gh/huggingface/transformers/pull/3918?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/huggingface/transformers/pull/3918?src=pr&el=footer). Last update [cb3c221...f42b609](https://codecov.io/gh/huggingface/transformers/pull/3918?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).


I don’t think that’s right. Which version of PyTorch are you running?

> I don’t think that’s right. Which version of PyTorch are you running?

Well, my version is 1.2.0, I checked the API in 1.4.0, the current version is right. 
Sorry for my hurry PR,
❓ DistilBert test perplexity based on WikiText-2: ppl is too low?
Author: bing0037
Pull request: no

# ❓ Questions & Help

<!-- The GitHub issue tracker is primarly intended for bugs, feature requests,
     new models and benchmarks, and migration questions. For all other questions,
     we direct you to Stack Overflow (SO) where a whole community of PyTorch and
     Tensorflow enthusiast can help you out. Make sure to tag your question with the
     right deep learning framework as well as the huggingface-transformers tag: 
     https://stackoverflow.com/questions/tagged/huggingface-transformers 
     
     If your question wasn't answered after a period of time on Stack Overflow, you
     can always open a question on GitHub. You should then link to the SO question 
     that you posted.
     -->

## Details
I tried to use the pre-trained model to fine-tune.  I got the ppl 6.12 of the DistilBert model, which is much lower than the GPT-2 model (ppl: 18.34, ref: https://paperswithcode.com/sota/language-modelling-on-wikitext-2). 

Is the DistilBert model works much better than the GPT-2? Or is it just because the loss functions are different?

Here are the commands:
```
# 1) code:
git clone https://github.com/huggingface/transformers.git

# 2) Download dataset:
cd transformers/examples/
wget https://s3.amazonaws.com/research.metamind.io/wikitext/wikitext-2-raw-v1.zip
unzip wikitext-2-raw-v1.zip

# 3) Benchmart:
## distilbert:
export TRAIN_FILE=./wikitext-2-raw/wiki.train.raw
export TEST_FILE=./wikitext-2-raw/wiki.test.raw
CUDA_VISIBLE_DEVICES=6 python run_language_modeling.py \
    --output_dir=output_distilbert \
    --model_type=distilbert \
    --model_name_or_path=distilbert-base-uncased \
    --do_train \
    --per_gpu_train_batch_size 15 \
    --train_data_file=$TRAIN_FILE \
    --do_eval \
    --eval_data_file=$TEST_FILE \
    --mlm
```
Result:
```
04/22/2020 13:25:33 - INFO - __main__ -   ***** Running evaluation  *****
04/22/2020 13:25:33 - INFO - __main__ -     Num examples = 535
04/22/2020 13:25:33 - INFO - __main__ -     Batch size = 4
Evaluating: 100%|████████████████████████████████████████████████████████████████████████████████████████████| 134/134 [00:05<00:00, 24.02it/s]
04/22/2020 13:25:38 - INFO - __main__ -   ***** Eval results  *****
04/22/2020 13:25:38 - INFO - __main__ -     perplexity = tensor(6.1200)
```
<!-- You should first ask your question on SO, and only if
     you didn't get an answer ask it here on GitHub. -->
**A link to original question on Stack Overflow**: 

Comments:
@VictorSanh can correct me if I'm wrong, but in general, the perplexity for masked language models like BERT is much lower than the perplexity for causal language models. 

That's because the "MLM perplexity" isn't the actual perplexity (in the sense that it is the probability of a sentence, computed from the predicted next word), but rather the "masked perplexity" which is computed differently.

It isn't surprising to me that you obtain ~6 perplexity on WikitText-2 with DistilBERT.

Thank you!
fix resize_token_embeddings to accept padding_idx for xlm-roberta models
Author: Soonhwan-Kwon
Pull request: yes

When resize_token_embedding there is no option for padding_idx 
and it leads to the wrong embedding for xlm-roberta models 
which needs embedding to have padding_idx as 1 not 0.

I fixed issue by adding option for padding_idx and considered padding_idx as None (which is 0 as default) for compatibility for other types of transformers models.

Comments:
Shuffle train subset for summarization example
Author: astariul
Pull request: yes

Fix #3892 
@sshleifer 

Comments:
# [Codecov](https://codecov.io/gh/huggingface/transformers/pull/3909?src=pr&el=h1) Report
> Merging [#3909](https://codecov.io/gh/huggingface/transformers/pull/3909?src=pr&el=desc) into [master](https://codecov.io/gh/huggingface/transformers/commit/cb3c2212c79d7ff0a4a4e84c3db48371ecc1c15d&el=desc) will **increase** coverage by `0.01%`.
> The diff coverage is `n/a`.

[![Impacted file tree graph](https://codecov.io/gh/huggingface/transformers/pull/3909/graphs/tree.svg?width=650&height=150&src=pr&token=9qOlN6Hb1c)](https://codecov.io/gh/huggingface/transformers/pull/3909?src=pr&el=tree)

```diff
@@            Coverage Diff             @@
##           master    #3909      +/-   ##
==========================================
+ Coverage   78.45%   78.47%   +0.01%     
==========================================
  Files         111      111              
  Lines       18521    18521              
==========================================
+ Hits        14531    14534       +3     
+ Misses       3990     3987       -3     
```


| [Impacted Files](https://codecov.io/gh/huggingface/transformers/pull/3909?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [src/transformers/modeling\_utils.py](https://codecov.io/gh/huggingface/transformers/pull/3909/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy9tb2RlbGluZ191dGlscy5weQ==) | `91.06% <0.00%> (+0.12%)` | :arrow_up: |
| [src/transformers/modeling\_tf\_utils.py](https://codecov.io/gh/huggingface/transformers/pull/3909/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy9tb2RlbGluZ190Zl91dGlscy5weQ==) | `92.92% <0.00%> (+0.16%)` | :arrow_up: |
| [src/transformers/file\_utils.py](https://codecov.io/gh/huggingface/transformers/pull/3909/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy9maWxlX3V0aWxzLnB5) | `68.86% <0.00%> (+0.36%)` | :arrow_up: |

------

[Continue to review full report at Codecov](https://codecov.io/gh/huggingface/transformers/pull/3909?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/huggingface/transformers/pull/3909?src=pr&el=footer). Last update [cb3c221...3361612](https://codecov.io/gh/huggingface/transformers/pull/3909?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
❓ Summarization example : Why no shuffling ?
Author: astariul
Pull request: no

# ❓ Questions & Help

Usually, when loading using Dataloader, data is shuffled for the training set. But in the case of the summarization example, data is not shuffled for training set :

https://github.com/huggingface/transformers/blob/1dc9b3c7847269961458c059ad8ad443b26bf60d/examples/summarization/bart/finetune.py#L105-L108

---

**Why data is not shuffled ?**

Comments:
No good reason.
Do you feel comfortable sending a PR that shuffles for train loader only?
New CLI using Typer
Author: kabirkhan
Pull request: yes

## Summary
First pass at a New CLI using [Typer](https://typer.tiangolo.com/)
**745** total lines of code vs 917 for the old CLI, only adding one (optional) dependency (Typer) which is well documented/supported and has 99% test coverage.

Currently, this mostly keeps the same option names for each command but I think we should have a discussion on using CLI Arguments vs Options.

Didn't put a ton of work into actual refactors around defining what should be an Argument and what should be an Option yet, wanted to get some eyes on this before putting in too much time.

In short, my position is everything that's required for a command to run should be an Argument and should be documented in the docstring which is automatically included in the help info

Good information here about using Arguments over Options https://typer.tiangolo.com/tutorial/arguments/#about-cli-arguments-help whenever there are required values.

## Usage
```
pip install transformers[new-cli]
```

and adds the console_script transformers so users can run commands with (e.g.):

```
transformers login
```

or:

```
transformers serve ner
```


 


Comments:
Closing as discussed in https://github.com/huggingface/transformers/issues/2959#issuecomment-619214142