noamaanMulla-03/transformers-issues huggingface dataset:
GPT2: masked_bias should be sufficiently small instead of -1e4
Author: maxwellzh
Pull request: no

See also #9594

My package build `transformers==4.12.3`.

Ideally for a causal language model, we should set it as -inf. Current setting -1e4 is not **small** enough. In my case, the model would learn to lower the attention weight of left context such that it could get the information of right context, which is illegal behavior.

Codes at https://github.com/huggingface/transformers/blob/531336bbfd2a97cf800f610d971d6ec0a1578752/src/transformers/models/gpt2/modeling_gpt2.py#L205-L206

In my experiment, I print the `attn_weights` of the first layer, there're many weights smaller than -1e4, and the attention score is no longer lower triangle, a.k.a., not causal.

```
tensor([[[[-16012.8271, -10000.0000, -10000.0000, -10000.0000, -10000.0000,
           -10000.0000, -10000.0000],
          [ 10511.5508,   5376.9136, -10000.0000, -10000.0000, -10000.0000,
           -10000.0000, -10000.0000],
          [-24461.1621, -13598.0195, -20461.9980, -10000.0000, -10000.0000,
           -10000.0000, -10000.0000],
          [-16113.5645, -10553.2480, -13133.3857,    147.0204, -10000.0000,
           -10000.0000, -10000.0000],
          [-20674.4199,  -2806.1458, -15805.4199,  -4623.0571,  12110.2920,
           -10000.0000, -10000.0000],
          [-24806.1738, -14453.9990, -20414.9160,  -8439.6494,  -2858.4604,
           -17795.5215, -10000.0000],
          [ -5393.4922, -10265.1875,  -6821.3672,   3551.4062, -20622.5879,
            -3372.0925,  10511.3047]],
```

Comments:
Hey @maxwellzh, this is a similar conversation to what was said here: https://github.com/huggingface/transformers/issues/14859

Thanks for raising this issue!

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md) are likely to be ignored.
can't set gradient_checkpointing=True when using the DistributedDataParallel mode
Author: jinxinglu
Pull request: no

## Environment info
<!-- You can run the command `transformers-cli env` and copy-and-paste its output below.
     Don't forget to fill out the missing fields in that output! -->

- `transformers` version: 4.15.0
- Platform: 2GPU-V100 with 16GB
- Python version: 3.6.8
- PyTorch version (GPU?): 1.10.1+cu102
- Tensorflow version (GPU?):
- Using GPU in script?: Yes
- Using distributed or parallel set-up in script?:

### Who can help

@sgugger 

I used the official document to build a pretrain BigbirdMLM model in DP mode, the trainer works well (but when the seq_length is very big ~4096, the training would take a lot of time), so I would like to try the DDP mode, but I meet some problems:
1. Do I need to change some codes in the original main.py? I read the document of "trainer_ags.py", so I only take a code change from "ddp_find_unused_parameters=False" to "ddp_find_unused_parameters=True", but raise a error about the "gradient_checkpointing", so I change the "gradient_checkpointing=False" ( this parameter is set to True in DP mode), and then I run the code in command line with "torchrun --nproc_per_node=2 main.py", it works well. 
2. the feature of "gradient_checkpointing" is very important I think. If I set "gradient_checkpointing=False", the batchsize(per_device_train_batch_size) only can be set to 1, (otherwise OOM), so this feature has a conflict in DDP mode? sorry I don't know the details about this feature. thanks




Comments:
You need to have `dpp_find_unused_parameters=False` when using `gradient_checkpointing=True`.

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md) are likely to be ignored.
Longformer Set Global Attention Layers Non-Trainable
Author: MarkWijkhuizen
Pull request: no

# 🚀 Feature request

Tensorflow Longformer model set global attention layers non-trainable.

## Motivation

During training global attention is not used giving zero gradients for all global attention weights. Setting the global attention layers to non-trainable will decrease training time and reduce memory consumption.


Comments:
Hi @MarkWijkhuizen! Thank you for your input. 

If you have the time, I'd like to encourage you to open a PR with the change. Otherwise, no worries, let us know (and, if possible, give us more pointers -- e.g., a reference to the paragraph in the paper where they mention this part of the training regime) 🤗 

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md) are likely to be ignored.
where is the 4.16.0dev??
Author: sipie800
Pull request: no

I'm running the run_mlm.py script.
There is such a line,

# Will error if the minimal version of Transformers is not installed. Remove at your own risks.
check_min_version("4.16.0.dev0")

but where is it?
can't find by pip,no in github too.

Comments:
Hey! That's the current `master` version :)

https://github.com/huggingface/transformers/blob/80f72960913ab6682451c33dfa8035ef0c932128/setup.py#L354

You can either clone the repo and install it locally, or install it with the following:
```
pip install -U git+https://github.com/huggingface/transformers
```

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md) are likely to be ignored.
Is it possible to support Wav2Vec in ZeroShotClassificationPipeline?
Author: mabu-dev
Pull request: no

Similar to the very helpful NLI-based zero-shot classification pipeline using a ModelForSequenceClassification, it would be great to have zero-shot on audio data. 

Pass wav file(s) with candidate labels to the pipeline and get a prediction. Is this at all on the roadmap?

Comments:
Maybe of interest to @Narsil and @patrickvonplaten

Pretty cool idea! @mabu-dev do you know whether there is a research paper on it? For NLI zero-shot we have this paper: https://huggingface.co/facebook/bart-large-mnli#nli-based-zero-shot-text-classification

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md) are likely to be ignored.
ALL YOUR BASE ARE BELONG TO 504.
Author: ark10806
Pull request: no

<img width="1223" alt="image" src="https://user-images.githubusercontent.com/30382262/150070377-edb319a5-7c28-4478-ba8e-123aebc46499.png">

Comments:
![image](https://user-images.githubusercontent.com/48780754/150070713-002d6e62-f87f-4d8e-b380-7963413953b8.png)


This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md) are likely to be ignored.
Model stopped training once I introduced << report_to = 'wandb' >> in TrainingArguments
Author: pratikchhapolika
Pull request: no

I am downloading the model https://huggingface.co/microsoft/Multilingual-MiniLM-L12-H384/tree/main microsoft/Multilingual-MiniLM-L12-H384 and then using it.

Transformer Version: '4.11.3'

I have written the below code:

```
import wandb
wandb.login()
%env WANDB_LOG_MODEL=true

model = tr.BertForSequenceClassification.from_pretrained("/home/pc/minilm_model",num_labels=2)
model.to(device)

print("hello")

training_args = tr.TrainingArguments(
report_to = 'wandb',
output_dir='/home/pc/proj/results2', # output directory
num_train_epochs=10, # total number of training epochs
per_device_train_batch_size=16, # batch size per device during training
per_device_eval_batch_size=32, # batch size for evaluation
learning_rate=2e-5,
warmup_steps=1000, # number of warmup steps for learning rate scheduler
weight_decay=0.01, # strength of weight decay
logging_dir='./logs', # directory for storing logs
logging_steps=1000,
evaluation_strategy="epoch",
save_strategy="no"
)

print("hello")

trainer = tr.Trainer(
model=model, # the instantiated 🤗 Transformers model to be trained
args=training_args, # training arguments, defined above
train_dataset=train_data, # training dataset
eval_dataset=val_data, # evaluation dataset
compute_metrics=compute_metrics
)

```
After Executing this: 

The model stuck at this point:


***** Running training *****
 ```
 Num examples = 12981
  Num Epochs = 20
  Instantaneous batch size per device = 16
  Total train batch size (w. parallel, distributed & accumulation) = 32
  Gradient Accumulation steps = 1
  Total optimization steps = 8120
Automatic Weights & Biases logging enabled, to disable set os.environ["WANDB_DISABLED"] = "true"

```

**What could be the possible solution?**


Comments:
@pratikchhapolika can you share a minimum example or a colab? Does it run ok with W&B disabled?

```
os.environ["WANDB_DISABLED"] = "true"
```

Also make sure you are on the latest version of W&B, 

```
pip install wandb --upgrade
```

> @pratikchhapolika can you share a minimum example or a colab? Does it run ok with W&B disabled?
> 
> ```
> os.environ["WANDB_DISABLED"] = "true"
> ```
> 
> Also make sure you are on the latest version of W&B,
> 
> ```
> pip install wandb --upgrade
> ```

Yes, when I do os.environ["WANDB_DISABLED"] = "true". It runs fine. 

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md) are likely to be ignored.
TF LMHead very slow: TFGPT2LMHeadModel is 7 times slower than Torch GPT2LMHeadModel 
Author: gyin94
Pull request: no
Labels: name: wontfix

The benchmark shows the Tensorflow TFGPT2LMHeadModel is 7 times (140ms) slower than Torch GPT2LMHeadModel implementation (20ms). Based on the profile tool from latest https://www.tensorflow.org/tfx/serving/tensorboard, the bottleneck is here. https://github.com/huggingface/transformers/blob/d9149f00d1a4650bafa7e1cd73e10398193c852c/src/transformers/modeling_tf_utils.py#L796

```
from transformers import *

args = PyTorchBenchmarkArguments(
    models=["gpt2_pt"],
    batch_sizes=[1], sequence_lengths=[8])

config_base = GPT2Config(n_layer=2, n_head=2)
config_base.architectures = ["GPT2LMHeadModel"]
benchmark = PyTorchBenchmark(args, configs=[config_base])
print(benchmark.run())

args = TensorFlowBenchmarkArguments(
    models=["gpt2_tf"],
    batch_sizes=[1], sequence_lengths=[8])

config_base = GPT2Config(n_layer=2, n_head=2)
config_base.architectures = ["GPT2LMHeadModel"]
benchmark = TensorFlowBenchmark(args, configs=[config_base])
print(benchmark.run())
```

Without LMHeadModel, the benchmark is 6ms vs 8ms.

Comments:
Multiple issues may be related to this.
[My finetuned gpt2 model is taking wayy too long to generate samples, like 5-8 minutes] 
https://github.com/huggingface/transformers/issues/6173
[[Benchmark] TFGPT2LMHeadModel is five times slower than GPT2LMHeadModel]
https://github.com/huggingface/transformers/issues/5604
[tensorflow2_gpt2 Slow speed]
https://github.com/huggingface/transformers/issues/4634
[In Tensorflow the serving is very slow]
https://github.com/huggingface/transformers/issues/5341

Do we plan to solve this problem with a higher priority? Thanks. @patrickvonplaten 

Can we add a new linear layer instead of using wte with matmul? Like `self.lm_head = nn.Linear(config.n_embd, config.vocab_size, bias=False)` in pytorch?
Since tf.matmul in CPU seems quite slow and moreover we have a `transpose_b` in the current TF implementation. 

Thanks a lot for posting the issue and linking everything together - this is extremely useful! I will try to take a deeper look into it early next week! 

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.


This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.


I met the same situation with the env info:
transformer version:4.12.5
tensorflow-gpu:2.8.0
tf-serving:2.6.0
linux, gpu-machine:P100, test script with the batch_size 1, max_len:20

google's bert inference takes about 12-13ms per query, but TF-Bert takes about 45~58ms with or without xla_cpu_compilation_enabled, 3-5x slower.

Has the problem been solved? Or some ideas to improve the TF-Pretrained-Model performemce?
Output embedding from each self-attention head from each encoder layer
Author: Sreyan88
Pull request: no

Hi there!

I wanted the embeddings from each self-attention head of each encoder layer for one of my projects, is this possible with the hugging face library?

If not, can I just slice the original embeddings from each layer (suppose 768/12 = 128 size slice) to get the attention head output?

Thank You

Comments:
Hi @Sreyan88 ! The best place to ask this question is [the forum](https://discuss.huggingface.co/). We use issues for bug reports and feature requests. Thank you!
Movement Prune
Author: JackqqWang
Pull request: no

Hi everyone. I expect to use the movement prune approach to get a pruned bert on a downstream task with this link

[https://github.com/huggingface/transformers/tree/master/examples/research_projects/movement-pruning](url)

Basically speaking, we have a bert, and downstream task( AG news/ Yahoo answers, classification tasks), we wanna get a pruned BERT fitting this task.

I am trying Iterative Magnitude Pruning in the link as above. But I am a little confused that what is the train-v1.1.json and dev-v1.1.json should look like?

Thanks!

Comments:
Hi @JackqqWang these kind of questions are best suited for our [forums](https://discuss.huggingface.co/) since we use issue to triage bug reports or feature requests.

In any case, my suggestion would be to use our `nn_pruning` [library](https://github.com/huggingface/nn_pruning) instead of the `movement-pruning` project. The reason I suggest this is that @madlag found a clever way to extend movement pruning such that inference can be improved by factors of 2-3x. You can find various examples for text classification in that repo :)

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md) are likely to be ignored.
Fine Tunning the Pytorch script
Author: SatishDeshbhratar
Pull request: no

I am trying a text summarization project mentioned in this link https://github.com/SatishDeshbhratar/transformers/tree/master/examples/pytorch/summarization

But I am facing the error related to hf argument parser as below

> (pytorch_nlp) PS D:\Docs\Course work\Final Project (Text Summarization)\Script> python run_summarization.py \ --train_file=/train.csv \ --validation_file=/val.csv \ --model_name_or_path=facebook/bart-large-cnn \ --learning_rate=3.0e-6 \ --per_device_train_batch_size=8 \ --per_device_eval_batch_size=4 \ --output_dir=/output \ --num_train_epochs=100 \ --overwrite_output_dir=True \ --max_source_length=1024 \ --max_target_length=512 \ --val_max_target_length=512 \ --pad_to_max_length=False \ --evaluation_strategy='steps' \ --eval_steps=1000 \ --metric_for_best_model='rouge2' \ --greater_is_better=True \ --load_best_model_at_end=True \ --save_strategy='steps' \ --save_steps=1000 \ --save_total_limit=1 \ --predict_with_generate=True \ --num_beams=4 --generation_num_beams=4 \ --generation_max_length=512 \ --overwrite_cache=True \ --do_train 

Traceback (most recent call last): File "D:\Docs\Course work\Final Project (Text Summarization)\Script\run_summarization.py", line 698, in <module> main() File "D:\Docs\Course work\Final Project (Text Summarization)\Script\run_summarization.py", line 281, in main model_args, data_args, training_args = parser.parse_args_into_dataclasses() File "C:\ProgramData\Anaconda3\envs\pytorch_nlp\lib\site-packages\transformers\hf_argparser.py", line 215, in parse_args_into_dataclasses raise ValueError(f"Some specified arguments are not used by the HfArgumentParser: {remaining_args}") ValueError: Some specified arguments are not used by the HfArgumentParser: ['\\', '\\', '\\', '\\', '\\', '\\', '\\', '\\', '\\', '\\', '\\', '\\', '\\', '\\', '\\', '\\', '\\', '\\', '\\', '\\', '\\', '\\', '\\', '\\', '\\', '\\'] (pytorch_nlp) PS D:\Docs\Course work\Final Project (Text Summarization)\Script>

Comments:
Maybe you remove all the '\' in the code, the problem will be solved.

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md) are likely to be ignored.
MobileBertForPreTraining means  IB-BERT PreTraining? 
Author: changquanyou
Pull request: no

https://github.com/huggingface/transformers/blob/05fa1a7ac17bb7aa07b9e0c1e138ecb31a28bbfe/src/transformers/models/mobilebert/modeling_mobilebert.py#L902
You Know that the mobilebert(student model) is  distilled  from IB-BERT(The Teacher Model), The MobileBertForPreTraining means  IB-BERT PreTraining?

Comments:
This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md) are likely to be ignored.
Fix (deprecated) ONNX exporter to account for new tf2onnx API
Author: lewtun
Pull request: yes

# What does this PR do?

This PR fixes an issue with the deprecated `convert_graph_to_onnx` module, where exporting TensorFlow models to ONNX failed due to changes in the `tf2onnx` API. In particular, the failing tests in question were:

* `test_onnx.py::OnnxExportTestCase::test_export_tensorflow`
* `test_onnx.py::OnnxExportTestCase::test_quantize_tf`

With this fix, the whole `test_onnx.py` test suite now passes when I run:

```
RUN_SLOW=1 pytest tests/onnx/test_onnx.py
```

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_15856). All of your documentation changes will be reflected on that endpoint.
set python version to 3.7 for flax tests
Author: patil-suraj
Pull request: yes

# What does this PR do?

Set python version to 3.7 for self-push flax tests.

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_15858). All of your documentation changes will be reflected on that endpoint.
set python version to 3.7 for flax tests
Author: patil-suraj
Pull request: yes

# What does this PR do?

set python version to 3.7 for self-push flax tests.

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_15859). All of your documentation changes will be reflected on that endpoint.
Add PT + TF automatic builds
Author: LysandreJik
Pull request: yes

Adds docker containers for both PyTorch and TensorFlow on which the pipelines will run.

Solves the dependencies errors for PyTorch and TensorFlow pipelines.

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_15860). All of your documentation changes will be reflected on that endpoint.
Update TF LM examples
Author: gante
Pull request: yes

# What does this PR do?

As part of the objective to bring TF examples up to speed, this PR updates the LM examples with the internal loss computations and the `to_tf_dataset()` functionality, which shaves off significant custom code.

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_15855). All of your documentation changes will be reflected on that endpoint.
DebertaForSequenceClassification loss computation
Author: michaelbenayoun
Pull request: yes

# What does this PR do?

This PR changes the way the loss is being computed for `DebertaForSequenceClassification` and `DebertaV2ForSequenceClassification` to make it match what is being done for other bert-like models.

The current way of doing it seems overly complicated and I do not see why the behaviour should be different from other bert-like models, but I might be missing something here... @LysandreJik @sgugger @patrickvonplaten  @BigBird01 what do you think?

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_15864). All of your documentation changes will be reflected on that endpoint.

No this is a breaking change. We didn't touch the way the loss was computed when adding support for `problem_type` for all models to avoid such a breaking change and we should leave it as is, at least until the next major version IMO.

At the very least, if we do break things there should be a flag to enable back the older behavior.

Additionally to what @sgugger said, DeBERTa was implemented to respect the paper's implementation which gave SOTA results on sequence classification tasks. Not all transformer models work the best with the same heads, so I personally think it's perfectly fine to have models have different heads to perform the same task, if those heads have been identified as the best for the task at hand.

What is very important is that it still has the same output as other sequence classification models, but that is the case with the `SequenceClassifierOutput`.

Ok thank you for the explanation, closing this!