noamaanMulla-03/transformers-issues huggingface dataset:
Can not import modeling_mbart
Author: Aniruddha-JU
Pull request: no

## Environment info
<!-- You can run the command `transformers-cli env` and copy-and-paste its output below.
     Don't forget to fill out the missing fields in that output! -->

- `transformers` version:
- Platform:
- Python version:
- PyTorch version (GPU?):
- Tensorflow version (GPU?):
- Using GPU in script?:
- Using distributed or parallel set-up in script?:

### Who can help
<!-- Your issue will be replied to more quickly if you can figure out the right person to tag with @
 If you know how to use git blame, that is the easiest way, otherwise, here is a rough guide of **who to tag**.
 Please tag fewer than 3 people.

Models:

- albert, bert, xlm: @LysandreJik
- blenderbot, bart, marian, pegasus, encoderdecoder,  t5: @patrickvonplaten, @patil-suraj
- longformer, reformer, transfoxl, xlnet: @patrickvonplaten
- fsmt: @stas00
- funnel: @sgugger
- gpt2: @patrickvonplaten, @LysandreJik
- rag: @patrickvonplaten, @lhoestq
- tensorflow: @Rocketknight1

Library:

- benchmarks: @patrickvonplaten
- deepspeed: @stas00
- ray/raytune: @richardliaw, @amogkam
- text generation: @patrickvonplaten
- tokenizers: @LysandreJik
- trainer: @sgugger
- pipelines: @LysandreJik

Documentation: @sgugger

Model hub:

- for issues with a model report at https://discuss.huggingface.co/ and tag the model's creator.

HF projects:

- datasets: [different repo](https://github.com/huggingface/datasets)
- rust tokenizers: [different repo](https://github.com/huggingface/tokenizers)

Examples:

- maintained examples (not research project or legacy): @sgugger, @patil-suraj
- research_projects/bert-loses-patience: @JetRunner
- research_projects/distillation: @VictorSanh

 -->

## Information

Model I am using (Bert, XLNet ...):

The problem arises when using:
* [ ] the official example scripts: (give details below)
* [ ] my own modified scripts: (give details below)

The tasks I am working on is:
* [ ] an official GLUE/SQUaD task: (give the name)
* [ ] my own task or dataset: (give details below)

## To reproduce

Steps to reproduce the behavior:

1.
2.
3.

<!-- If you have code snippets, error messages, stack traces please provide them here as well.
     Important! Use code tags to correctly format your code. See https://help.github.com/en/github/writing-on-github/creating-and-highlighting-code-blocks#syntax-highlighting
     Do not use screenshots, as they are hard to read and (more importantly) don't allow others to copy-and-paste your code.-->

## Expected behavior

<!-- A clear and concise description of what you would expect to happen. -->


Comments:
Please follow the issue template.
Feedback whilst resuming
Author: david-waterworth
Pull request: no

# 🚀 Feature request

Add some sort of progress bar during resumption of training from a checkpoint

## Motivation

I resumed training from a checkpoint mid-way through a 100 Epoch training. The progress bar sat on zero for quite some time, and I could see CPU activity but no GPU's were active. Eventually the progress bar jumped to 50% and training resumed - I assume it's doing some sort of initialisation.

It would be nice if there were some indication that some progress is being made as it's not obvious what's occuring



Comments:
This has been fixed by #11324. If you use a source install, you will be able to use (or in this case see) this feature :-)
Tensorflow Keras model.loads_weights() breaks on TFElectraModel trained with v4.3.0
Author: buoi
Pull request: no

## Environment info

- `transformers` version: 4.4.0
- Platform: Linux-4.19.112+-x86_64-with-Ubuntu-18.04-bionic
- Python version: 3.7.10
- PyTorch version (GPU?): 1.8.0+cu101 (True)
- Tensorflow version (GPU?): 2.4.1 (True)
- Using GPU in script?: no
- Using distributed or parallel set-up in script?: no

### Who can help
@jplu 

## Information

Model I am using TFElectraModel:

The problem arises when using:
* my own modified scripts: see attachment

The tasks I am working on is:
* Information Retrieval on SQuAD dataset

## To reproduce

Steps to reproduce the behavior:

1. train a keras model whith a TFElectraModel as a layer on transformers 4.3.0.
2. switch to transformers 4.4.0
3. call model.load_weights on the keras model

OR:
1. "run all" on attached notebook, wait for weights save after (very short) training
2. change pip tranasformers version to 4.4.0 in the first cell
3. comment model.fit to avoid override of weights
4. restart an run all, wait for model.load_weights to fail

[IR_TFElectraModel.ipynb.zip](https://github.com/huggingface/transformers/files/6151831/IR_TFElectraModel.ipynb.zip)


ValueError: Cannot assign to variable tf_electra_model_1/electra/embeddings/token_type_embeddings/embeddings:0 due to variable shape (2, 128) and value shape (512, 128) are incompatible

## Expected behavior

I would expect the weights to load as it correctly does with both train and load performed on transformers 4.3.0

Anyway keep up with the great job! 🤗 


Comments:
Thanks for opening this issue!

I have not access to a computer to extract and read you archive, can you, please, share a Colab or copy/paste the code here in this issue. Thanks!

> Thanks for opening this issue!
> 
> I have not access to a computer to extract and read you archive, can you, please, share a Colab or copy/paste the code here in this issue. Thanks!

yes sure: https://colab.research.google.com/drive/1tHzUMhveYwYkPOCxZwMA80IkVkRCWGRW?usp=sharing

#### To Reproduce

1. "run all" on attached notebook, wait for weights save after (very short) training

2. change pip tranasformers version to 4.4.0 in the first cell
3. comment model.fit to avoid override of weights
4. restart an run all, wait for model.load_weights to fail


Sorry, I cannot open your Colab :( There is a restricted access

Sorry, access at the same link is now allowed. :)

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md) are likely to be ignored.
[Docs] Add Caching Example For CI? 
Author: hamelsmu
Pull request: no

From the [installation instructions](https://huggingface.co/transformers/installation.html#caching-models):

> If you expect to be downloading large volumes of models (more than 10,000) from huggingface.co (for instance through your CI setup, or a large-scale production deployment), please cache the model files on your end. It will be way faster, and cheaper. Feel free to contact us privately, we’d love to help with this.

I'm happy to write an example of how to cache with GitHub Actions.  Shall I contribute this to the docs?  If so, please assign the issue to me and I'll do it.   If this is not a good idea, please feel free to close the issue.

@sgugger 

Comments:
This was written by @julien-c so I'll let him reply on this :-)

This was written before we moved from S3 to our own Cloudfront-served repositories so we could also probably just remove that paragraph.

Interesting. Maybe it could _still_ be a good reminder as many folks would forget to do this (I know I might have and have wasted so much of my own compute!)?

However, I'll open a PR to remove the paragraph if that's preferred 🙇🏽
Why exclude LayerNorm.bias from weight decay when finetuning?
Author: xiaoda99
Pull request: no
Labels: name: wontfix

# ❓ Questions & Help

<!-- The GitHub issue tracker is primarly intended for bugs, feature requests,
     new models and benchmarks, and migration questions. For all other questions,
     we direct you to Stack Overflow (SO) where a whole community of PyTorch and
     Tensorflow enthusiast can help you out. Make sure to tag your question with the
     right deep learning framework as well as the huggingface-transformers tag: 
     https://stackoverflow.com/questions/tagged/huggingface-transformers 
     
     If your question wasn't answered after a period of time on Stack Overflow, you
     can always open a question on GitHub. You should then link to the SO question 
     that you posted.
     -->

## Details
<!-- Description of your issue -->
https://github.com/huggingface/transformers/blob/master/src/transformers/trainer.py#L306
In the original BERT implementation and in earlier versions of this repo, both LayerNorm.weight and LayerNorm.bias are decayed.

<!-- You should first ask your question on SO, and only if
     you didn't get an answer ask it here on GitHub. -->
**A link to original question on Stack Overflow**:


Comments:
This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.


I have the same question

Check this [discussion](https://forums.fast.ai/t/is-weight-decay-applied-to-the-bias-term/73212/6)
Which models can be using for encoder-decoder?
Author: blizda
Pull request: no

Hi, I trying to use EncoderDecoderModel. I tried google/electra-base-discriminator, google/electra-small-discriminator, albert-base-v2 as encoder and decoder:

```python
from transformers import EncoderDecoderModel

model = EncoderDecoderModel.from_encoder_decoder_pretrained('google/electra-small-discriminator', 'google/electra-small-discriminator')
```
but, I always gate the same error 

```python
TypeError: forward() got an unexpected keyword argument 'encoder_hidden_states'
```

Only Bert can be using as encoder and decoder? If so, can you add list of available models for encoder-decoder in documentation?

Comments:
@blizda did you find an answer to your query - "Only Bert can be using as encoder and decoder? If so, can you add list of available models for encoder-decoder in documentation?"? 

There's an error-message saying which models can be used now; 
"... Model type should be one of CamembertConfig, XLMRobertaConfig, RobertaConfig, BertConfig, OpenAIGPTConfig, GPT2Config, TransfoXLConfig, XLNetConfig, XLMConfig, CTRLConfig, ReformerConfig, BertGenerationConfig, XLMProphetNetConfig, ProphetNetConfig."
Reformat to make code clearer in tokenizer call
Author: sgugger
Pull request: yes

# What does this PR do?

While reviewing a PR on the tokenizer call this morning, I had some trouble parsing what was happening, as black reformatted the tests in a way that is quite unreadable (IMO). This PR fixes that.

No logic is changed, it's just put in a more human-readable way (again maybe this is just me).

Comments:
"unuglify" -> best branch name ever haha
Help understanding how to build a dataset for language as with the old TextDataset
Author: danieldiezmallo
Pull request: no

I understand this issue should be on the Datasets library, so it's been created there https://github.com/huggingface/datasets/issues/

Hello,

I am trying to load a custom dataset that I will then use for language modeling. The dataset consists of a text file that has a whole document in each line, meaning that each line overpasses the normal 512 tokens limit of most tokenizers.

I would like to understand what is the process to build a text dataset that tokenizes each line, having previously split the documents in the dataset into lines of a "tokenizable" size, as the [old TextDataset](https://github.com/huggingface/transformers/blob/master/src/transformers/data/datasets/language_modeling.py) class would do, where you only had to do the following, and a tokenized dataset without text loss would be available to pass to a DataCollator:


```
model_checkpoint = 'distilbert-base-uncased'

from transformers import AutoTokenizer
tokenizer = AutoTokenizer.from_pretrained(model_checkpoint)

from transformers import TextDataset

dataset = TextDataset(
    tokenizer=tokenizer,
    file_path="path/to/text_file.txt",
    block_size=512,
)
```


For now, what I have is the following, which, of course, throws an error because each line is longer than the maximum block size in the tokenizer:


```
import datasets
dataset = datasets.load_dataset('path/to/text_file.txt')

model_checkpoint = 'distilbert-base-uncased'
tokenizer = AutoTokenizer.from_pretrained(model_checkpoint)

def tokenize_function(examples):
    return tokenizer(examples["text"])

tokenized_datasets = dataset.map(tokenize_function, batched=True, num_proc=4, remove_columns=["text"])

tokenized_datasets
```


So what would be the "standard" way of creating a dataset in the way it was done before?

Thank you very much for the help :))


Comments:
Is there a `DataCollator` cat mask n-gram words for LM?
Author: wa008
Pull request: no
Labels: name: Migration

# 📚 Migration

## Information

I want a mask n-gram words when pre-train bert model? but I can't find  a DataCollator from the lib
https://github.com/huggingface/transformers/blob/master/src/transformers/data/data_collator.py

I want to build it by myself, but I don't know how to build my own DataCollator, who can give me a demo?

## Checklist

- [ ] I have read the migration guide in the readme.
 ([pytorch-transformers](https://github.com/huggingface/transformers#migrating-from-pytorch-transformers-to-transformers);
  [pytorch-pretrained-bert](https://github.com/huggingface/transformers#migrating-from-pytorch-pretrained-bert-to-transformers))
- [ ] I checked if a related official extension example runs on my machine.


Comments:
This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md) are likely to be ignored.
issue of run_mlm.py 
Author: sataliulan
Pull request: no

hello guys, I try to finetune bert in my own dataset (line by line txt, language:Chinese), follow the guid code of run_mlm.py example.
The tokenizer is bert pretrained tokenizer (tokenizer=AutoTokenizer.from_pretrained('bert-base-chinese')), and model is bert-base-chinese ,as folllows:
    config=BertConfig.from_pretrained('bert-base-chinese')
    print(config)
    model=BertForMaskedLM(config=config)
when I started the trainer, I got the following errors:

Using custom data configuration default-cd6deed448eea358
Downloading and preparing dataset text/default (download: Unknown size, generated: Unknown size, post-processed: Unknown size, total: Unknown size) to /home/hcl/.cache/huggingface/datasets/text/default-cd6deed448eea358/0.0.0/293ecb642f9fca45b44ad1f90c8445c54b9d80b95ab3fca3cfa5e1e3d85d4a57...
                                Dataset text downloaded and prepared to /home/hcl/.cache/huggingface/datasets/text/default-cd6deed448eea358/0.0.0/293ecb642f9fca45b44ad1f90c8445c54b9d80b95ab3fca3cfa5e1e3d85d4a57. Subsequent calls will reuse this data.
100%|██████████| 3264/3264 [01:06<00:00, 48.83ba/s]
  0%|          | 0/3264 [00:00<?, ?ba/s]
Traceback (most recent call last):
  File "/home/hcl/miniconda3/envs/pytorch/lib/python3.7/site-packages/datasets/arrow_dataset.py", line 1582, in _map_single
    writer.write_batch(batch)
  File "/home/hcl/miniconda3/envs/pytorch/lib/python3.7/site-packages/datasets/arrow_writer.py", line 276, in write_batch
    pa_table = pa.Table.from_pydict(typed_sequence_examples)
  File "pyarrow/table.pxi", line 1559, in pyarrow.lib.Table.from_pydict
  File "pyarrow/array.pxi", line 331, in pyarrow.lib.asarray
  File "pyarrow/array.pxi", line 222, in pyarrow.lib.array
  File "pyarrow/array.pxi", line 110, in pyarrow.lib._handle_arrow_array_protocol
  File "/home/hcl/miniconda3/envs/pytorch/lib/python3.7/site-packages/datasets/arrow_writer.py", line 98, in __arrow_array__
    if trying_type and out[0].as_py() != self.data[0]:
  File "pyarrow/array.pxi", line 1067, in pyarrow.lib.Array.__getitem__
  File "pyarrow/array.pxi", line 549, in pyarrow.lib._normalize_index
IndexError: index out of bounds
python-BaseException

for now, I don't know why, the first epoch runs well. 
Any helps?
ps: my os is deepin15; I choose nvidia rtx2080ti as my gpu, and finetune my dataset on pytorch1.6

Comments:
This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md) are likely to be ignored.
Can’t download the pre-trained pegasus-large model
Author: xiaohy9
Pull request: no

## Environment info
<!-- You can run the command `transformers-cli env` and copy-and-paste its output below.
     Don't forget to fill out the missing fields in that output! -->

- `transformers` version: 4.4.2
- Platform: 
- Python version: 3.7.10
- PyTorch version (GPU?):
- Tensorflow version (GPU?): 2.4.1
- Using GPU in script?: yes
- Using distributed or parallel set-up in script?:

## Who can help
- blenderbot, bart, marian, pegasus, encoderdecoder,  t5: @patrickvonplaten, @patil-suraj

## Information

It appears that the huggingface.co model url has some problem.

Code:
```
from transformers import AutoTokenizer, TFAutoModelForSeq2SeqLM
tokenizer2 = AutoTokenizer.from_pretrained("google/pegasus-large")  
model2 = TFAutoModelForSeq2SeqLM.from_pretrained("google/pegasus-large")
inputs1_2 = tokenizer2.encode("summarize: " + text1, return_tensors="tf", max_length=1024)
outputs1_2 = model2.generate(inputs1_2, max_length=150, min_length=40, length_penalty=2.0, num_beams=4, early_stopping=True)
outputs1_2, tokenizer2.decode(outputs1_2[0])

```
error message:
```
404 Client Error: Not Found for url: https://huggingface.co/google/pegasus-large/resolve/main/tf_model.h5

…

OSError: Can't load weights for 'google/pegasus-large'. Make sure that:

- 'google/pegasus-large' is a correct model identifier listed on 'https://huggingface.co/models'

- or 'google/pegasus-large' is the correct path to a directory containing a file named one of tf_model.h5, pytorch_model.bin.

```



Comments:
Hey @xiaohy9,

Thanks for the issue! It should be resolved now. See: https://huggingface.co/google/pegasus-large/commit/4510ba69cc183d23e892e7728a40fdcf42e83079 . 

Could you try again? 

Yes, it works now. Thanks for the quick response!
However, I saw some similar issue using pegasus-large as using pegasus-xsum, with details mentioned here: https://github.com/huggingface/transformers/issues/10837

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md) are likely to be ignored.
Generating text with fine-tuned TFGPT2LMHeadModel in python. 
Author: brandonbell11
Pull request: no

I've finetuned GPT2 using the following script:
```
from transformers import GPT2Tokenizer, TFGPT2LMHeadModel
import tensorflow as tf

model = TFGPT2LMHeadModel.from_pretrained("distilgpt2")
tokenizer = GPT2Tokenizer.from_pretrained("distilgpt2")

file_path = 'text.txt'
with open(file_path, encoding="utf-8") as f:
    text = f.read()

tokenized_text = tokenizer.encode(text)

examples = []
block_size = 100
for i in range(0, len(tokenized_text) - block_size + 1, block_size):  # Truncate in block of block_size
    examples.append(tokenized_text[i:i + block_size])

inputs, labels = [], []
for ex in examples:
    inputs.append(ex[:-1])
    labels.append(ex[1:])

dataset = tf.data.Dataset.from_tensor_slices((inputs, labels))

BATCH_SIZE = 8
BUFFER_SIZE = 10000

dataset = dataset.shuffle(BUFFER_SIZE).batch(BATCH_SIZE, drop_remainder=True)

optimizer = tf.keras.optimizers.Adam(learning_rate=3e-5, epsilon=1e-08, clipnorm=1.0)
loss = tf.keras.losses.SparseCategoricalCrossentropy(from_logits=True)
metric = tf.keras.metrics.SparseCategoricalAccuracy('accuracy')
model.compile(optimizer=optimizer, loss=[loss, *[None] * model.config.n_layer], metrics=[metric])
model.fit(dataset, epochs=20)
```
This runs fine, and after 20 epochs I have an accuracy of ~0.59. 
The problem comes when I tried to write my own text generation script:
```
def generate_text(model, tokenizer, start_string, num_generate):

        input_eval = tf.expand_dims(tokenizer.encode(start_string), 0)
        token_ids = []

        for i in range(num_generate):
            predictions = tf.squeeze(model.predict(input_eval)[0], 0)

            predicted_id = tf.random.categorical(predictions, 1)[-1, 0].numpy().item()
            
            input_eval = tf.expand_dims([predicted_id], 0)

            token_ids.append(predicted_id)
    
        return start_string + tokenizer.decode(token_ids)
```
I get output, but the output is of a sufficiently lower quality than when I train a model using "run_lm_finetuning.py" and generate text using "run_generation.py"
I looked into the example generation script, and it looks like there is simply a call to "model.generate(...)"

Where does this model.generate() method exist? 

Comments:
Did you resolve this? I think on the current commit `generate()` still doesn't exist.

There is no TensorFlow implementation for the `generate()` method yet. We're working on it, but in the meantime, you could do your own generation loop or use a PyTorch model with the `generate()` method.

I need to change to `BATCH_SIZE = 12` in the above or else this example code will not run.  There would be a dimension mismatch with `BATCH_SIZE = 8`
Split checkpoint from model_name_or_path in examples
Author: sgugger
Pull request: yes

# What does this PR do?

There is currently a problem in the way we handle resuming from checkpoint in our PyTorch example scripts with `Trainer`: if the user wants to resume form a specific checkpoint, they have to pass `--model_name_or_path checkpoint_folder` which is a bit counter-intuitive but also poses the problem that sometimes the user is passing `--model_name_or_path local_pretrained_model`.

This has caused multiple issues that we tried to patch a bit as they came:
- in text classification or token classification, the `local_pretrained_model` might be pretrained with a different number of labels which made the loading fail (shape mismatch)
- more recently since we're not using `from_pretrained` anymore when loading the checkpoint (to keep the model as provided by the user, with potential frozen layers), the state dict of `local_pretrained_model` generally won't match: if the task is different, its head will have weights incompatible with the model at end.

This PR cleans things up by adding a new training argument called `--resume_from_checkpoint`. To resume training from an explicit checkpoint, the user now has to pass `--resume_from_checkpoint checkpoint_folder` as passing `--model_name_or_path some_local_folder` now only loads the model inside the local folder as a pretrained model and doesn't look for a checkpoint.
It's slightly breaking (not in the library, just the commands to run the examples) but cleaner IMO.

The training argument `resume_from_checkpoint` is used as a default  for the argument of the same name in the `train` method.

Might fix #11485

Comments:
Hi guys, just one comment/question from my side: instead of introducing new commandline options (allowing to resume from a checkpoint), what do you think about the following workflow: 

a) load the model (with AutoModel)
b) check if the loaded model is from type "TokenClassification"

:thinking: 

> I'm also thinking - if we know we are going to resume from checkpoint for sure, then replace from_pretrained with from_config and save the double loading of the weights?

I'm leaving this to your PoC on one of the seq2seq scripts if you don't mind? This PR is also kind of an urgent bug fix.

Oh, of course, I had no idea about the urgency. I will do it in another PR then.