noamaanMulla-03/transformers-issues huggingface dataset:
How to download Pre-trained T5 model?
Author: deepankar27
Pull request: no

Hi,

I tried looking for ways to download & use T5-small pre-trained model but didn't get any API mentioned in documentation to download it. Though I found links but don't know will it work if I pass the path of the model?

Thanks in advance..

Comments:
```python 
from transformers import T5ForConditionalGeneration

t5 = T5ForConditionalGeneration.from_pretrained("t5-small")
```

does not work? 

@patrickvonplaten Thanks for the pointer, I tried & getting this:

```
>>> from transformers import T5ForConditionalGeneration
>>> t5 = T5ForConditionalGeneration.from_pretrained("t5-small")
Traceback (most recent call last):
  File "/root/anaconda3/envs/docsearch/lib/python3.7/site-packages/transformers/configuration_utils.py", line 242, in get_config_dict
    raise EnvironmentError
OSError

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/root/anaconda3/envs/docsearch/lib/python3.7/site-packages/transformers/modeling_utils.py", line 604, in from_pretrained
    **kwargs,
  File "/root/anaconda3/envs/docsearch/lib/python3.7/site-packages/transformers/configuration_utils.py", line 200, in from_pretrained
    config_dict, kwargs = cls.get_config_dict(pretrained_model_name_or_path, **kwargs)
  File "/root/anaconda3/envs/docsearch/lib/python3.7/site-packages/transformers/configuration_utils.py", line 251, in get_config_dict
    raise EnvironmentError(msg)
OSError: Can't load config for 't5-small'. Make sure that:

- 't5-small' is a correct model identifier listed on 'https://huggingface.co/models'

- or 't5-small' is the correct path to a directory containing a config.json file
```

I am new on huggingface API. I am going to download it & give path of the model. Let's see how it works... But let me know if you can give me some suggestions.

Hi @deepankar27, do you mind specifying which `transformers` version you're using?

@LysandreJik Sorry for the late reply, I figured it out, issue was with my config files. Thanks... :)

Where in the computer memory is the T5-small model being downloaded by using @patrickvonplaten 's code?

The model is downloaded from AWS where it is saved and then usually saved in a cache folder (usually `~/.cache/torch/transformers` as far as I know)
[Docs model summaries] Add pegasus to docs
Author: patrickvonplaten
Pull request: yes

Stiched together a short model summary for Pegasus. Would be great if @sshleifer and @sgugger can take a look :-) 

Comments:
# [Codecov](https://codecov.io/gh/huggingface/transformers/pull/6640?src=pr&el=h1) Report
> Merging [#6640](https://codecov.io/gh/huggingface/transformers/pull/6640?src=pr&el=desc) into [master](https://codecov.io/gh/huggingface/transformers/commit/d0e42a7bed3de9271ae39c575d7eeb54cf985921?el=desc) will **decrease** coverage by `0.02%`.
> The diff coverage is `n/a`.

[![Impacted file tree graph](https://codecov.io/gh/huggingface/transformers/pull/6640/graphs/tree.svg?width=650&height=150&src=pr&token=9qOlN6Hb1c)](https://codecov.io/gh/huggingface/transformers/pull/6640?src=pr&el=tree)

```diff
@@            Coverage Diff             @@
##           master    #6640      +/-   ##
==========================================
- Coverage   79.14%   79.12%   -0.03%     
==========================================
  Files         156      156              
  Lines       28248    28248              
==========================================
- Hits        22358    22351       -7     
- Misses       5890     5897       +7     
```


| [Impacted Files](https://codecov.io/gh/huggingface/transformers/pull/6640?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [src/transformers/generation\_tf\_utils.py](https://codecov.io/gh/huggingface/transformers/pull/6640/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy9nZW5lcmF0aW9uX3RmX3V0aWxzLnB5) | `80.70% <0.00%> (-2.01%)` | :arrow_down: |
| [src/transformers/file\_utils.py](https://codecov.io/gh/huggingface/transformers/pull/6640/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy9maWxlX3V0aWxzLnB5) | `82.41% <0.00%> (+0.25%)` | :arrow_up: |

------

[Continue to review full report at Codecov](https://codecov.io/gh/huggingface/transformers/pull/6640?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/huggingface/transformers/pull/6640?src=pr&el=footer). Last update [d0e42a7...773081d](https://codecov.io/gh/huggingface/transformers/pull/6640?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).


I guess we could in general link the model summaries to the model doc as well or vice-versa for each model? 

We also should probably automatically link each model card to its doc

In the meantime, I think duplication is fine:)
add tests/test_tokenization_reformer.py
Author: sshleifer
Pull request: no
Labels: name: Help wanted

name: Tests

I don't think there is any common test coverage for ReformerTokenizer. besides through integration tests.
Good source for copy/modification is `XLMRobertaTokenizationTest`



Comments:
I can help with this.

Awesome!

@sshleifer I put together the test code and find that the following test is failing:

```
self = < tests.test_tokenization_reformer.ReformerTokenizationTest
testMethod = test_torch_encode_plus_sent_to_model 
@slow
@require_torch
def test_torch_encode_plus_sent_to_model(self):
    import torch
    from transformers import MODEL_MAPPING, TOKENIZER_MAPPING

    MODEL_TOKENIZER_MAPPING = merge_model_tokenizer_mappings(MODEL_MAPPING, TOKENIZER_MAPPING)

    tokenizers = self.get_tokenizers(do_lower_case=False)
    for tokenizer in tokenizers:
        with self.subTest(f"{tokenizer.__class__.__name__}"):

            if tokenizer.__class__ not in MODEL_TOKENIZER_MAPPING:
                return

            config_class, model_class = MODEL_TOKENIZER_MAPPING[tokenizer.__class__]
            config = config_class()

            if config.is_encoder_decoder or config.pad_token_id is None:
                return

            model = model_class(config)

            # Make sure the model contains at least the full vocabulary size in its embedding matrix
            is_using_common_embeddings = hasattr(model.get_input_embeddings(), "weight")
            
assert (
    (model.get_input_embeddings().weight.shape[0] >= len(tokenizer))
    if is_using_common_embeddings
    else True
)
AssertionError:
assert False
```

Upon further investigation I found a discrepancy between the pre-trained tokenizer and pre-trained model config around the pad token id and resulting vocab size. Please see the example below:

`
from transformers import ReformerTokenizer, ReformerModel
model = ReformerModel.from_pretrained("google/reformer-crime-and-punishment")
tokenizer = ReformerTokenizer.from_pretrained("google/reformer-crime-and-punishment")
print(tokenizer.vocab_size)        320
print(len(tokenizer))              321
print(model.config.vocab_size)     320
print(model.get_input_embeddings().weight.shape[0]) 320
print(tokenizer.get_vocab()['<pad>'])   320
print(model.config.pad_token_id)        0
print(tokenizer.get_vocab()['<unk>'])   0
`

What is your suggestion for moving forward?

My suggestion would be to check in `tokenization_utils_base ` how `__len__` works, and try to make it so that ReformerTokenizer's __len__ is 320.

@sshleifer Test merged.

Thx @D-Roberts !
Summarization Examples: Support label_smoothed_cross_entropy
Author: sshleifer
Pull request: no
Labels: name: Help wanted

name: Summarization

name: Examples

This seems to be used by nearly all papers, but the HF implementation in `examples/summarization/finetune.py` uses the standard cross entropy loss.

Ideally, we could expose a command line arg to allow this choice to be toggled, like: 

```python
parser.add_argument('--loss', type=str, choices=['cross_entropy', 'label_smoothed_cross_entropy'], default='cross_entropy')
```

Comments:
Yes, it's annoying that torch doesn't have an implementation of smoothed cross entropy.

Although note that using smoothed cross entropy will hurt the probability calibration of a seq2seq model (https://pubs.acs.org/doi/10.1021/acscentsci.9b00576)

@sshleifer This is good point. I would consider to work on this issue
As i see we need 2 things for this:
1) add an implementation of label_smoothed_cross_entropy
2) support choice of loss function in `finetune.py`

am i correct?


Yes you are!

This issue is resolved as label smoothing has already been integrated in master with https://github.com/huggingface/transformers/pull/5919
Docs: Separate documentation for mbart
Author: sshleifer
Pull request: no
Labels: name: Help wanted

name: Documentation

Currently all mbart documentation is stuffed into docs/
https://github.com/huggingface/transformers/blob/353b8f1e7a7361c0afd9e391381bc226b4a5ca8f/docs/source/model_doc/bart.rst#L42

mbart should have it's own `model_doc/mbart.rst` and entry in `pretrained_models.rst`.



Optionally you can also create a new `src/transformers/modeling_mbart.py` with roughly these contents:

```python
from .modeling_bart import BartForConditionalGeneration
from .configuration_bart import MbartConfig
class MBartForConditionalGeneration(BartForConditionalGeneration):
    config_class = MbartConfig
	# this model fully inherits its implementation from bart
```

Comments:
@sshleifer  I think adding new class should make thing more clear, should I go ahead with it ? Will also need to modify tests a little bit I guess

yes, thanks!

key tests not to break
```bash
RUN_SLOW=1 pytest tests/test_modeling_mbart.py
```

You can also make `tokenization_mbart.py`
append keyword arguments to the output
Author: Kyeongpil
Pull request: yes
Labels: name: wontfix

prepare_inputs_for_generation function receives keyword arguments (model_specific_kwargs)
However, these arguments are not ignored.
I fix this codes to return arguments including these keyword arguments.

Actually, I don't have an idea about the purpose of prepare_logits_for_generation function, which only returns the logit values.

Comments:
This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
Add Code Coverage and Black badges to README
Author: bharatr21
Pull request: yes
Labels: name: wontfix

Add Black badge to README based on this:
https://github.com/huggingface/transformers/blob/403d3098572ac308416653648456a940860da39e/.circleci/config.yml#L101 

Comments:
# [Codecov](https://codecov.io/gh/huggingface/transformers/pull/4979?src=pr&el=h1) Report
> Merging [#4979](https://codecov.io/gh/huggingface/transformers/pull/4979?src=pr&el=desc) into [master](https://codecov.io/gh/huggingface/transformers/commit/403d3098572ac308416653648456a940860da39e&el=desc) will **decrease** coverage by `0.00%`.
> The diff coverage is `n/a`.

[![Impacted file tree graph](https://codecov.io/gh/huggingface/transformers/pull/4979/graphs/tree.svg?width=650&height=150&src=pr&token=9qOlN6Hb1c)](https://codecov.io/gh/huggingface/transformers/pull/4979?src=pr&el=tree)

```diff
@@            Coverage Diff             @@
##           master    #4979      +/-   ##
==========================================
- Coverage   77.20%   77.20%   -0.01%     
==========================================
  Files         128      128              
  Lines       21851    21851              
==========================================
- Hits        16870    16869       -1     
- Misses       4981     4982       +1     
```


| [Impacted Files](https://codecov.io/gh/huggingface/transformers/pull/4979?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [src/transformers/file\_utils.py](https://codecov.io/gh/huggingface/transformers/pull/4979/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy9maWxlX3V0aWxzLnB5) | `73.38% <0.00%> (-0.41%)` | :arrow_down: |

------

[Continue to review full report at Codecov](https://codecov.io/gh/huggingface/transformers/pull/4979?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/huggingface/transformers/pull/4979?src=pr&el=footer). Last update [403d309...5be7204](https://codecov.io/gh/huggingface/transformers/pull/4979?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).


This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
Update setup.py for sentencepiece.
Author: soni-vikas
Pull request: yes
Labels: name: wontfix

Now the `sentencepiece` library has upgraded to `0.1.92` version which is incompatible with ` transformers==2.8.0`.

sentencepiece==0.1.92 gives segmentation fault (core dumped).

Comments:
# [Codecov](https://codecov.io/gh/huggingface/transformers/pull/4930?src=pr&el=h1) Report
> Merging [#4930](https://codecov.io/gh/huggingface/transformers/pull/4930?src=pr&el=desc) into [master](https://codecov.io/gh/huggingface/transformers/commit/699541c4b34479451c91b3c6c204d904f62bed83&el=desc) will **increase** coverage by `0.39%`.
> The diff coverage is `n/a`.

[![Impacted file tree graph](https://codecov.io/gh/huggingface/transformers/pull/4930/graphs/tree.svg?width=650&height=150&src=pr&token=9qOlN6Hb1c)](https://codecov.io/gh/huggingface/transformers/pull/4930?src=pr&el=tree)

```diff
@@            Coverage Diff             @@
##           master    #4930      +/-   ##
==========================================
+ Coverage   76.77%   77.16%   +0.39%     
==========================================
  Files         128      128              
  Lines       21723    21723              
==========================================
+ Hits        16677    16763      +86     
+ Misses       5046     4960      -86     
```


| [Impacted Files](https://codecov.io/gh/huggingface/transformers/pull/4930?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [src/transformers/modeling\_tf\_utils.py](https://codecov.io/gh/huggingface/transformers/pull/4930/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy9tb2RlbGluZ190Zl91dGlscy5weQ==) | `87.26% <0.00%> (+0.77%)` | :arrow_up: |
| [src/transformers/modeling\_tf\_xlm.py](https://codecov.io/gh/huggingface/transformers/pull/4930/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy9tb2RlbGluZ190Zl94bG0ucHk=) | `75.85% <0.00%> (+19.75%)` | :arrow_up: |

------

[Continue to review full report at Codecov](https://codecov.io/gh/huggingface/transformers/pull/4930?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/huggingface/transformers/pull/4930?src=pr&el=footer). Last update [699541c...ad6c947](https://codecov.io/gh/huggingface/transformers/pull/4930?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).


This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
error while instantiating model
Author: kevinghst
Pull request: no
Labels: name: wontfix

Hi,

For my experiments, I need to make some changes to the forward pass of Roberta sequence classification model. Thus, I copied **RobertaForSequenceClassification** from _modeling_roberta.py_ into a [separate file](https://github.com/kevinghst/mixmatch_from_scratch/blob/master/models_roberta.py) in my repo.

And I try to instantiate it in my _main.py_ like this:
```
    model = RobertaForSequenceClassification.from_pretrained(
        'roberta-base',
        num_labels = NUM_LABELS[cfg.task],
    )
```
However, it gives me the following stacktrace:
----------------------------------------------------------------------------------------------------------
Traceback (most recent call last):
  File "/scratch/wz1232/anaconda3/envs/mixmatch/lib/python3.6/site-packages/torch/serialization.py", line 186, in _check_seekable
    f.seek(f.tell())
AttributeError: 'NoneType' object has no attribute 'seek'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/scratch/wz1232/anaconda3/envs/mixmatch/lib/python3.6/site-packages/transformers/modeling_utils.py", line 516, in from_pretrained
    state_dict = torch.load(resolved_archive_file, map_location="cpu")
  File "/scratch/wz1232/anaconda3/envs/mixmatch/lib/python3.6/site-packages/torch/serialization.py", line 368, in load
    return _load(f, map_location, pickle_module)
  File "/scratch/wz1232/anaconda3/envs/mixmatch/lib/python3.6/site-packages/torch/serialization.py", line 517, in _load
    _check_seekable(f)
  File "/scratch/wz1232/anaconda3/envs/mixmatch/lib/python3.6/site-packages/torch/serialization.py", line 189, in _check_seekable
    raise_err_msg(["seek", "tell"], e)
  File "/scratch/wz1232/anaconda3/envs/mixmatch/lib/python3.6/site-packages/torch/serialization.py", line 182, in raise_err_msg
    raise type(e)(msg)
AttributeError: 'NoneType' object has no attribute 'seek'. You can only torch.load from a file that is seekable. Please pre-load the data into a buffer like io.BytesIO and try to load from it instead.

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "main.py", line 208, in <module>
    num_labels = NUM_LABELS[cfg.task],
  File "/scratch/wz1232/anaconda3/envs/mixmatch/lib/python3.6/site-packages/transformers/modeling_utils.py", line 519, in from_pretrained
    "Unable to load weights from pytorch checkpoint file. "
OSError: Unable to load weights from pytorch checkpoint file. If you tried to load a PyTorch model from a TF 2.0 checkpoint, please set from_tf=True.
-----------------------------------------------------------------------------------------------------------

It doesn't occur when I import **RobertaForSequenceClassification** straight from the library, as opposed to from my own file containing the copied code. Why is this?

Comments:
This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
unexpected keyword argument 'lm_labels' when using BertModel as Decoder with EncoderDecoderModel
Author: utkd
Pull request: no
Labels: name: wontfix

The `BertModel.forward()` method does not expect a `lm_labels` and `masked_lm_labels` arguments. Yet, it looks like the `EncoderDecoderModel.forward()` method calls it's decoder's `forward()` method with those arguments which throws a TypeError when a BertModel is used as a decoder.

Am I using the BertModel incorrectly? I can get rid of the error by modifying the EncoderDecoderModel to not use those arguments for the decoder.

Exact Error:

```
File "/Users/utkarsh/anaconda3/envs/py37/lib/python3.7/site-packages/torch/nn/modules/module.py", line 550, in __call__
    result = self.forward(*input, **kwargs)
  File "/Users/utkarsh/Projects/ai4code/transformers/bert2bert/models.py", line 12, in forward
    dec_out, dec_cls, enc_out, enc_cls = self.bertmodel(input_ids=inputs, attention_mask=input_masks, decoder_input_ids=targets, decoder_attention_mask=target_masks)
  File "/Users/utkarsh/anaconda3/envs/py37/lib/python3.7/site-packages/torch/nn/modules/module.py", line 550, in __call__
    result = self.forward(*input, **kwargs)
  File "/Users/utkarsh/anaconda3/envs/py37/lib/python3.7/site-packages/transformers/modeling_encoder_decoder.py", line 283, in forward
    **kwargs_decoder,
  File "/Users/utkarsh/anaconda3/envs/py37/lib/python3.7/site-packages/torch/nn/modules/module.py", line 550, in __call__
    result = self.forward(*input, **kwargs)
TypeError: forward() got an unexpected keyword argument 'lm_labels'
```

Relevant part of the code:
```
encoder = BertModel(enc_config)
dec_config = BertConfig(...,is_decoder=True)
decoder = BertModel(dec_config)
model = EncoderDecoderModel(encoder=encoder, decoder=decoder)
```
...
`dec_out, dec_cls, enc_out, enc_cls = model(input_ids=inputs, attention_mask=input_masks, decoder_input_ids=targets, decoder_attention_mask=target_masks)`




Comments:
I'm facing the same problem. Since #4874 it seems like it should be just `labels` instead of `lm_labels`. According to the documentation it should do masked language modeling-loss, but from my debugging it seems like it actually does next word prediction-loss.

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
Save the Dataset for training GPT2
Author: gaps013
Pull request: no
Labels: name: wontfix

# ❓ Questions & Help

<!-- The GitHub issue tracker is primarly intended for bugs, feature requests,
     new models and benchmarks, and migration questions. For all other questions,
     we direct you to Stack Overflow (SO) where a whole community of PyTorch and
     Tensorflow enthusiast can help you out. Make sure to tag your question with the
     right deep learning framework as well as the huggingface-transformers tag: 
     https://stackoverflow.com/questions/tagged/huggingface-transformers 
     
     If your question wasn't answered after a period of time on Stack Overflow, you
     can always open a question on GitHub. You should then link to the SO question 
     that you posted.
     -->

## Details
<!-- Description of your issue -->

<!-- You should first ask your question on SO, and only if
     you didn't get an answer ask it here on GitHub. -->
**A link to original question on Stack Overflow**:

I am trying to train GPT2 on a custom dataset, the training text file is over 100GB, when creating the Dataset for training, it is taking too long and I was hoping if there is any way to save the dataset for later use. 

I am using hugging face blog post on how to train transformers from scratch as a guide (https://huggingface.co/blog/how-to-train).

Thank You

Comments:
This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.