noamaanMulla-03/transformers-issues huggingface dataset:
Tokenizer from tokenizers library cannot be used in Trainer
Author: Aktsvigun
Pull request: no

Hi,

I am trying to train my own model with `Trainer` with a pre-trained `SentencePieceBPETokenizer` from **tokenizers** library. However, it is missing several attributes as well as methods (e.g., `pad`), which makes it incompatible with `transformers.Trainer`. Is there an easy way to convert it to `PretrainedTokenizer` from `transformers`?
Thanks!

Comments:
You can just do
```
from transformers import PreTrainedTokenizerFast

tokenizer = PreTrainedTokenizerFast(tokenizer_object=your_tokenizer)
```

> You can just do
> 
> ```
> 
> from transformers import PreTrainedTokenizerFast
> 
> 
> 
> tokenizer = PreTrainedTokenizerFast(tokenizer_object=your_tokenizer)
> 
> ```

Sylvain, million thanks!
GPT2 Layers
Author: Marc-Morcos
Pull request: no

When the trainer API is used to finetune gpt-2, does it finetune all the layers or just some? Is there a way to control which layers it finetunes? 
gpt2: @patrickvonplaten, @LysandreJik


Comments:
It finetunes all the layers. You can set the `require_grads` attribute of the model layers you don't want to train to `False` before sending the model to the `Trainer` if you want to change that behavior.

thank you!
Correct validation_split_percentage argument from int (ex:5) to float (0.05)
Author: Elysium1436
Pull request: yes


# What does this PR do?
This PR is for fixing a bug in the run_clm.py and run_mlm.py examples in the tensorflow section. I have merely divided the value by 100 on the train_test_split test_size argument. This will make it work as intended now.




## Before submitting
- [ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [x] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md#start-contributing-pull-requests),
      Pull Request section?
- [x] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
https://github.com/huggingface/transformers/pull/11690

- [ ] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/master/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/master/docs#writing-source-documentation).

- [ ] Did you write any new necessary tests?


## Who can review?

Anyone in the community is free to review the PR once the tests have passed. Feel free to tag
members/contributors who may be interested in your PR.

Related People: @sgugger and @patil-suraj 


Comments:
Please make sure to run `make style` on your branch to fix the formatting issues.

Thanks for fixing!

Thanks for kind help with the formatting
The Unsupervised denoising training example in T5's doc
Author: drxmy
Pull request: no

When I am running that example, it will print a lot of "seq_length: 7" like this:
seq_length: 7
position_bias: torch.Size([1, 8, 7, 7])
mask: torch.Size([1, 1, 1, 7])
seq_length: 7
seq_length: 7
seq_length: 7
seq_length: 7
seq_length: 7
seq_length: 7
position_bias: torch.Size([1, 8, 7, 7])
mask: torch.Size([1, 1, 7, 7])
seq_length: 7
position_bias: torch.Size([1, 8, 7, 7])
mask: torch.Size([1, 1, 1, 7])
seq_length: 7
seq_length: 7
seq_length: 7
seq_length: 7
seq_length: 7
seq_length: 7
seq_length: 7
seq_length: 7
seq_length: 7
seq_length: 7

But I do get the loss. If I run my own training code with a 256 sequence length and T5-large, it will print a lot more. Is this normal? My environments are:

1. torch 1.7.1
2. transformers 4.8.2
3. cuda 10.1
4. GPU v100-16g

Could you please help me with this issue? Thank you!

Comments:
I just ran it in a Colab notebook and got no issues.

Here's the notebook: https://colab.research.google.com/drive/1Fq420RZwq2coLjb0TJmAx5Q20uz3JJHj?usp=sharing

> I just ran it in a Colab notebook and got no issues.
> 
> Here's the notebook: https://colab.research.google.com/drive/1Fq420RZwq2coLjb0TJmAx5Q20uz3JJHj?usp=sharing

Thank you for the trouble! I changed my torch to 1.9 and transformers to 4.9. It didn't happen anymore.
Fix QA examples for roberta tokenizer
Author: sgugger
Pull request: yes

# What does this PR do?

https://github.com/huggingface/datasets/pull/2586 has changed the SQUAD dataset and doesn't clean anymore the whitespace in questions. This in turn makes the tokenization fail for tokenizers that don't remove whitespace (like RoBERTa): some questions begin with loooots of spaces and so the truncation strategy then fails because the question itself is longer than the max length (the infuriating example number 107709 of the training set for instance).

For more context, see #12880

This PR addresses that by removing the whitespace on the left of questions.

Comments:
GPT2Model cannot handle 3D attention_mask
Author: bzantium
Pull request: no

When pretraining GPT2 model, it sometimes need to receive a 3d attention_mask. Let's take an example with "gpt2" model. The model would be trained with an instance consisting of two different document "I am a boy. 
 you are rich.".
tokenizer convert the sentence into input_ids: [40, 716, 257, 2933, 13, 220, 50256, 345, 389, 5527, 13] and attention_mask: [1,1,1,1,1,1,1,1,1,1,1].
Since "I am a boy." and "you are rich." are from different documents, when predicting "you are rich." we do not want GPT to attend "I am a boy.", leading to the need for 3d attention_mask. Other models like BERT can handle this problem with the function "get_extended_attention_mask" from modeling_utils.

However, GPT2 model considers only 2d attention_mask now.
https://github.com/huggingface/transformers/blob/2e5dbdf2db4599a6694d0974575a70f9bc3c978e/src/transformers/models/gpt2/modeling_gpt2.py#L697



Comments:
This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md) are likely to be ignored.
Feature request: Show command line argument defaults
Author: mbforbes
Pull request: no

# 🚀 Feature request

When running with `--help`, show the default values for command line arguments.

## Motivation

There are dozens of command line arguments. When I'm trying to figure out how to run a script, I often want to know what value is being used when I don't specify it. But running with `--help` doesn't show the default values unless it's explicitly written in the description (which is only for three of them for the example script I'm using).

For example, `--evaluation_strategy`

```
  --evaluation_strategy {no,steps,epoch}
                        The evaluation strategy to use.
```

This ends up being a bit of a frustrating user experience. The two ways I currently use to find the value are:

1. Run the script again without `--help` and log all the arguments (done in the examples). This shows the assigned value, which will be the default if not passed. However, it doesn't show the description of what it does.

2. Go to the documentation. This will show the default value and a more thorough description, but requires opening a web browser and Googling to find the right page.

In other Python projects, I use the `argparse.ArgumentDefaultsHelpFormatter`, which automatically displays default values in the `--help` message along with their descriptions.

```python
parser = argparse.ArgumentParser(
    formatter_class=argparse.ArgumentDefaultsHelpFormatter
)
```

I wonder whether the Huggingface arguments could support the same feature?

Many thanks for considering this!

Comments:
This is a very reasonable request and thanks for suggesting an easy way to do it! I added that in the PR linked above.

Wow, thank you so much for the support and quick turnaround, I really appreciate it!! 🎉
Error when trying `push_to_hub` for a fine-tuned model on Colab
Author: phosseini
Pull request: no

## Environment info
<!-- You can run the command `transformers-cli env` and copy-and-paste its output below.
     Don't forget to fill out the missing fields in that output! -->

- `transformers` version: 4.9.1
- Platform: Colab

### Who can help
@sgugger

## To reproduce

Steps to reproduce the behavior. This is the code I'm running:

I first install the following packages:

```
! pip install transformers datasets
! sudo apt-get install git-lfs
```

Then I run the `! transformers-cli login` and successfully login and my token is saved at: `/root/.huggingface/token`

Then I run the following code:

```
from transformers import AutoModel

model = AutoModel.from_pretrained('/path/to/my/fine-tuned/model/on/my/google/drive')

model.push_to_hub("my-username/my-model-name")

```

Per @sgugger's suggestion, I also tried the following line but I'm getting the very error:

`model.push_to_hub("my-model-name")`

And this is the error:

```
---------------------------------------------------------------------------
ValueError                                Traceback (most recent call last)
<ipython-input-3-84aee0bf13c0> in <module>()
      4 model = AutoModel.from_pretrained(model_path)
      5 
----> 6 model.push_to_hub("my-username/my-model-name")

2 frames
/usr/local/lib/python3.7/dist-packages/huggingface_hub/repository.py in __init__(self, local_dir, clone_from, use_auth_token, git_user, git_email)
    102                 )
    103                 raise ValueError(
--> 104                     "If not specifying `clone_from`, you need to pass Repository a valid git clone."
    105                 )
    106 

ValueError: If not specifying `clone_from`, you need to pass Repository a valid git clone.
```


## Expected behavior
To have my fine-tuned model uploaded to my private repo on Huggingface.


Comments:
Just tried on a fresh colab and could upload a model without any problem (as long as there is no "/" in the model ID). Do you already have a model with the same username maybe?

Note that you are missing the step `! git config --global user.email "your_email"` in the preparation.

Are you certain you do have the latest version of Transformers installed?

Thanks for the tips. Problem solved. I think it was because I created a repo and a model with the very name on the Hugging Face website (I thought there should already be a model with the name there if we want to push the model.) I removed the model with the same name and now it works!
How to reproduce XLNet correctly And What is the config for finetuning XLNet?
Author: sherlcok314159
Pull request: no
Labels: name: Migration

I fintune a XLNet for English text classification. But it seems that I did something wrong about it because xlnet-base is worse than bert-base in my case. I set every 1/3 epoch report validation accuracy. At the beginning Bert-base is about 0.50 while XLNet-base is only 0.24. The config I use for xlnet is listed as follows:
```python
config = {
  batch_size = 4,
  learning_rate = 1e-5,
  gradient_accumulation_steps =  32,
  epochs = 4,
  max_sep_length = 384,
  weight_decay = 0.01,
  adam_epsilon = 1e-6,
  16-bit_training = False
}

```
Does finetune XLNet needs a special setting or XLNet converges slowly?

Thanks for everyone willing to help in advance!  :-)


Comments:
Hi,

For training related questions, please refer to the [forum](https://discuss.huggingface.co/). We like to keep Github issues for bugs/feature requests.

Thanks!
ViTFeatureExtractor.save_pretrained() generate "preprocessor_config.json" but not "config.json"
Author: daquarti
Pull request: no

config.json is needed to use ViTForImageClassification.from_pretrained()

I made a pull-request

Comments:
`ViTFeatureExtractor` is the feature extractor, not the model itself. The model itself requires the `config.json` file that specifies the architecture of the model, while the feature extractor requires its `preprocessor_config.json` file. These two are different files.

The problem is ViTForImageClassification.from_pretrained() not take "preprocessor_config.json" and you need to rename it as "config.json".
Thanks

`ViTForImageClassification` is not `ViTFeatureExtractor`, the first one is a model while the second one is a feature extractor. See some usage examples [here](https://huggingface.co/transformers/model_doc/vit.html#transformers.ViTForImageClassification)

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md) are likely to be ignored.
Fix CpmTokenizer for training/finetuning CPM model
Author: ghosthamlet
Pull request: yes

When CpmTokenizer extends XLNetTokenizer, it works for inference/generation, but model in training/finetuning will failed after some steps, its error is random like:

    RuntimeError: CUDA error: device-side assert triggered
    terminate called after throwing an instance of 'std::runtime_error'
    what():  NCCL error in: /pytorch/torch/lib/c10d/../c10d/NCCLUtils.hpp:136, unhandled cuda error, NCCL version 2.7.8,

failed when in different tensor operation every time, very strange.
after extend XLNetTokenizerFast, inference/training/finetuning all working.


Comments:
Add CpmTokenizerFast
Author: JetRunner
Pull request: yes

# What does this PR do?
Add a fast version of `CpmTokenizer`

Fixes #12837

Comments:
> I don't think the fast tokenizer as it's written works for now, as the fast tokenizer do not call the `_tokenize` method.

Oops! It looks the old pull request isn't right. I'll take a closer look

@sgugger I've updated and tested it. It works fine - only needs to wait for the `tokenizer.json` to be uploaded.

Tokenizer file uploaded. Merging it.
Starting today, I get an error downloading pre-trained models
Author: jason-weddington
Pull request: no

## Environment info
Copy-and-paste the text below in your GitHub issue and FILL OUT the two last points.

- `transformers` version: 2.1.1
- Platform: Linux-5.8.0-63-generic-x86_64-with-glibc2.29
- Python version: 3.8.10
- PyTorch version (GPU?): 1.9.0+cu111 (True)
- Tensorflow version (GPU?): not installed (NA)
- Flax version (CPU?/GPU?/TPU?): not installed (NA)
- Jax version: not installed
- JaxLib version: not installed
- Using GPU in script?: yes
- Using distributed or parallel set-up in script?: no

## Information

Model I am using (Bert, XLNet ...):

roberta-base, but this is currently an issue with all models

The problem arises when using:
* [x] the official example scripts: (give details below)
* [ ] my own modified scripts: (give details below)

The tasks I am working on is:
- downloading pre-trained models is currently failing, this seems have have started just in the last day

## To reproduce

Steps to reproduce the behavior:

1. attempt to load any pre-trained model from HuggingFace (code below)


This code:

`generator = pipeline("text-generation", model="bert-base-uncased")`

Generates this error:

403 Client Error: Forbidden for url: https://huggingface.co/bert-base-uncased/resolve/main/config.json
---------------------------------------------------------------------------
HTTPError                                 Traceback (most recent call last)

...

OSError: Can't load config for 'bert-base-uncased'. Make sure that:

- 'bert-base-uncased' is a correct model identifier listed on 'https://huggingface.co/models'

- or 'bert-base-uncased' is the correct path to a directory containing a config.json file

## Expected behavior

I expect the pre-trained model to be downloaded. This issue just started today.


Comments:
Hi @jason-weddington, are you calling those URLs from any particular workload or infrastructure?

The only reason I can see where you would get a 403 on this URL is if your usage triggers our infra's firewall. Would you mind contacting us at `expert-acceleration at huggingface.co` so we can take a look?

Thanks, I'll email you. I'm running this in a notebook on my desktop, using my home internet connection, but we're also seeing this in Google Colab. The issue just stated today.

This is working again, thanks for the help.
Converted pytorch model to onnx does not work correctly
Author: yzhang-github-pub
Pull request: no

I converted pretrained 'Rostlab/prot_bert_bfd' @ huggingface to onnx, then tried to convert a checkpoint from fine tuning of the pretrained model. Comparing to pretrained model, conversion of fine tuned model generated a lot of warnings. Basically the warnings said that parameters from first to last layer were not initialized. The converted model did not work correctly. 

This is how I called the conversion module:

python3 -m transformers.convert_graph_to_onnx --model Rostlab/prot_bert_bfd --framework pt prot_bert_bfd.onnx
I did similarly for checkpoint model.

Does the module work on checkpoint?




Comments:
This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md) are likely to be ignored.

> I converted pretrained 'Rostlab/prot_bert_bfd' @ huggingface to onnx, then tried to convert a checkpoint from fine tuning of the pretrained model. Comparing to pretrained model, conversion of fine tuned model generated a lot of warnings. Basically the warnings said that parameters from first to last layer were not initialized. The converted model did not work correctly.
> 
> This is how I called the conversion module:
> 
> python3 -m transformers.convert_graph_to_onnx --model Rostlab/prot_bert_bfd --framework pt prot_bert_bfd.onnx
> I did similarly for checkpoint model.
> 
> Does the module work on checkpoint?

Hello @yzhang-github-pub ,I have used the same line of code as you did for a PEGASUS model on colab:
`!python3 -m transformers.convert_graph_to_onnx --model jpcorb20/pegasus-large-reddit_tifu-samsum-256 --framework pt pegasus-large-reddit_tifu-samsum-256.onnx`
**The following error keeps showing and the coversion fails. Can you please tell me how did you solve this problem??**
**The error:**

> Some weights of the model checkpoint at jpcorb20/pegasus-large-reddit_tifu-samsum-256 were not used when initializing PegasusModel: ['final_logits_bias', 'lm_head.weight']

> - This IS expected if you are initializing PegasusModel from the checkpoint of a model trained on another task or with another architecture (e.g. initializing a BertForSequenceClassification model from a BertForPreTraining model).
> - This IS NOT expected if you are initializing PegasusModel from the checkpoint of a model that you expect to be exactly identical (initializing a BertForSequenceClassification model from a BertForSequenceClassification model).
> Downloading: 100% 1.50k/1.50k [00:00<00:00, 1.29MB/s]
> Downloading: 100% 1.91M/1.91M [00:01<00:00, 1.12MB/s]
> Downloading: 100% 1.34k/1.34k [00:00<00:00, 1.18MB/s]
> Error while converting the model: Folder /content is not empty, aborting conversion

I wrote a script to do onnx conversion, by importing onnx and onnxruntime modules. I heard some versions of transformer have bugs in onnx conversion and model loading.
Add support for truncation argument when calling a Pipeline
Author: mkaze
Pull request: no
Labels: name: wontfix

# 🚀 Feature request

Currently, only the `padding` argument [is supported](https://github.com/huggingface/transformers/blob/a39dfe4fb122c11be98a563fb8ca43b322e01036/src/transformers/pipelines.py#L500) when calling a pipeline, and it's not possible to pass `truncation` argument. For example, running the following code sample would raise an error:

```python
import transformers as trf

model = trf.pipeline(task='feature-extraction', model='bert-base-cased')
output = model('a sample text', padding=False, truncation=True)
```

## Motivation

<!-- Please outline the motivation for the proposal. Is your feature request
     related to a problem? e.g., I'm always frustrated when [...]. If this is related
     to another GitHub issue, please link here too. -->

If toggling padding is supported, then why truncation shouldn't be?

## Your contribution

I think to achieve this, same as `padding`, only a `truncation` argument should be added to `_parse_and_tokenize` method and also when calling the tokenizer. If that's the case, I would be willing to work on a PR.

Comments:
is there any workaround for this?  i'm seeing the `Token indices sequence length is longer than the specified maximum sequence length for this model (... > 512). Running this sequence through the model will result in indexing errors` when using TextClassificationPipeline.

(This is preventing me from upgrading to 3.x.)

routing this to @mfuntowicz @LysandreJik 

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.


+1 on this

Hi, even though this has been closed as stale, without comment or supposed fix, it seems that in recent versions you can in fact pass both `truncation` and `padding` arguments to the pipeline's `__call__` method, and it will correctly use them when tokenizing. I've tested it with long texts that fail without the truncation argument, and it seems to work as expected.
Add substep end callback method
Author: wulu473
Pull request: yes

As discussed in #12920 with @sgugger, a callback method after a gradient accumulation step is needed for some training techniques such as differentially private training with Opacus (see [Opacus - Docs: virtual_step](https://opacus.ai/api/privacy_engine.html?highlight=virtual_step#opacus.privacy_engine.PrivacyEngine.virtual_step)).

This PR extends `TrainerCallback` and `CallbackHandler` with a method `on_substep_end` which ought to be called during gradient accumulation after a training step is taken (i.e. loss and gradients computed) but no model parameters are updated.




Comments:
Thanks for your help. Yep, happy to add the callback as well. I'll tag you on a PR when I have something ready.
Add multilingual documentation support
Author: JetRunner
Pull request: yes

This PR adds multilingual documentation support for incoming Chinese documentations.

Comments:
You will need to run `make style` on your branch to fix the quality check. I'm looking at the result [here](https://247866-155220641-gh.circle-artifacts.com/0/docs/_build/html/index.html) but don't see anything changed. Is it normal?

@sgugger Yes, it's normal. This is just a PR to support multilingual docs but they aren't there yet. 

Mmm, looks like you may have some wrong version on your side? A `pip install -e .[quality]` should fix this (but you will probably need to revert the changes in the pipeline tests as black doesn't undo the new lines it adds).
Fix division by zero in NotebookProgressPar
Author: sgugger
Pull request: yes

# What does this PR do?

This PR fixes the bug reported in #12950. More precisely, the following snippet of code was failing with a division by zero error:
```py
from transformers.utils.notebook import NotebookProgressBar

pbar = NotebookProgressBar(total=1)
pbar.update(1)
pbar.update(1, force_update=True)
```

This PR fixes that by being a bit more defensive before dividing by a potential zero.

Fixes #12950

Comments: