noamaanMulla-03/transformers-issues huggingface dataset:
Use `transformers` models as Spark estimators 
Author: xegulon
Pull request: no

# 🚀 Feature request

<!-- A clear and concise description of the feature proposal.
     Please provide a link to the paper and code in case they exist. -->

It would be great to have the possibility to use `transformers` models and pipelines for inference on massive Spark `DataFrames`.

Given a model trained the standard way, one would have a syntax of this kind (or any other):

```python
from transformers.spark import SparkWrapper
from transformers import BertForSequenceClassification
model = BertForSequenceClassification(...)
estimator = SparkWrapper(model)
estimator.fit(trainingData)
```

## Motivation

<!-- Please outline the motivation for the proposal. Is your feature request
     related to a problem? e.g., I'm always frustrated when [...]. If this is related
     to another GitHub issue, please link here too. -->

`huggingface/transformers` models **and** tokenizers are great, but it seems they are not suitable for making inference on Big Data (example use case: I have a classifier I wanna use to make predictions on millions of sentences).

## Your contribution

<!-- Is there any way that you could help, e.g. by submitting a PR?
     Make sure to read the CONTRIBUTING.MD readme:
     https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md -->

[Gluon NLP based on Apache MXNet](https://nlp.gluon.ai/) and [Spark NLP](https://nlp.johnsnowlabs.com/) seem to be the only NLP libraries that run natively on Spark. The export from `huggingface/transformers` models/tokenizers to them is not a one-liner, when it's possible. Maybe a solution for my problem would be to write conversion scripts, but I find it interesting to have the spark logic built in `transformers` anyway.

@sgugger @LysandreJik @patrickvonplaten 

Comments:
This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md) are likely to be ignored.
RuntimeError: Connection timed out in Single node Multi GPU training
Author: kamalravi
Pull request: no
Labels: name: wontfix

I am trying to pre-train DistilBERT with single node multigpu as given here https://github.com/huggingface/transformers/tree/master/examples/distillation.

I have set the IP address of my gcp instance and port number. I am getting this error. Any solutions?

> File "/opt/conda/lib/python3.6/site-packages/torch/distributed/distributed_c10d.py", line 400, in init_process_group
    store, rank, world_size = next(rendezvous(url))
  File "/opt/conda/lib/python3.6/site-packages/torch/distributed/rendezvous.py", line 143, in _env_rendezvous_handler
    store = TCPStore(master_addr, master_port, world_size, start_daemon)
RuntimeError: Connection timed out

Comments:
By any chance, did you open the port number you're using?

Yes, it is established. What is the master_addr and master_port when submitting a job with single node and multigpu config in GCP?

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.


Hi, @kamalravi were you able to solve this issue?? I am facing the same issue right now where my `init_process_group` freezes and doesn't move forward for training. Could you take a look at this and suggest any relevant solutions if you have found them??

Thanks!

https://github.com/pytorch/pytorch/issues/53395#issuecomment-954103393
Replace assertions with RuntimeError exceptions
Author: ddrm86
Pull request: yes

Replaces the assertions in integrations.py with RuntimeError exceptions.

Contributes towards fixing issue #12789

## Before submitting
- [ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [x] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md#start-contributing-pull-requests),
      Pull Request section?
- [x] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [ ] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/master/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/master/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests?


Comments:
Add the fast implementation of `BlenderbotTokenizer`
Author: stancld
Pull request: no

# 🚀 Feature request

As it is the case for other models' tokenizers, add the fast implementation of BlenderbotTokenizer.

## Motivation

To have faster tokenization for Blenderbot models. (Also, the implementation should be pretty straightforward considering the similarity to the `RobertaTokenizer`/`RobertaTokenizerFast`.)

## Your contribution

I would like to have a look at this and will be glad to add that.

Comments:
That sounds great @stancld, we would love a PR!

@LysandreJik - I found a minor issue in the formatting of `tokenizer_config.json` at https://huggingface.co/facebook/blenderbot-3B/blob/main/tokenizer_config.json, where is `"add_prefix_space": "true"` instead of `"add_prefix_space": true`. This leads to the error (see below) during the slow->fast (tokenizer) conversion. It can be handled in the `converter` source code, though I believe it might be better to update a config file. Is there a way of how to send a PR to HF's hub?

Error:
```
TypeError: Can't convert 'true' to PyBool
```

Ah ! There's no way to do that as of now - let me handle that for you.

Should be done with [`huggingface#c468b23`](https://huggingface.co/facebook/blenderbot-3B/blob/main/tokenizer_config.json)

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md) are likely to be ignored.

still open #13720
Generalize problem_type to all sequence classification models
Author: sgugger
Pull request: yes

# What does this PR do?

This PR makes each sequence classification model use the `problem_type` attribute of the config and infer a default coming from the labels when it's not set.

Deberta implemented its own loss, apparently dealing with one-hot encoded labels, which we keep when `problem_type` is None for backward compatibility.

Comments:
Relevant to #13370
AttributeError: 'T5ForConditionalGeneration' object has no attribute 'linear'
Author: pradeepdev-1995
Pull request: no

## Environment info

- `transformers` version: 4.9.1
- Platform: Linux-5.4.0-84-generic-x86_64-with-Ubuntu-18.04-bionic
- Python version: 3.6.9
- PyTorch version (GPU?): 1.8.0 (False)
- Tensorflow version (GPU?): not installed (NA)
- Flax version (CPU?/GPU?/TPU?): not installed (NA)
- Jax version: not installed
- JaxLib version: not installed
- Using GPU in script?:NO
- Using distributed or parallel set-up in script?:NO

Models:

- model_name = "flexudy/t5-small-wav2vec2-grammar-fixer"

Library:
- text generation: @patrickvonplaten

Documentation: @sgugger

## Information

I am trying to do the static quantization on the T5 model(flexudy/t5-small-wav2vec2-grammar-fixer) for reducing the inference time.

## code used
```
import torch
import transformers
from transformers import T5Tokenizer, T5ForConditionalGeneration
model_name             =  "flexudy/t5-small-wav2vec2-grammar-fixer"
torch_device              =  'cuda' if torch.cuda.is_available() else 'cpu'
model                        = T5ForConditionalGeneration.from_pretrained(model_name).to(torch_device)
model.eval()
model.qconfig            =  torch.quantization.get_default_qconfig('fbgemm')
model_fused             = torch.quantization.fuse_modules(model,[['linear', 'linear']])
```


## Output
AttributeError: 'T5ForConditionalGeneration' object has no attribute 'linear'

## Expected behavior

Model modules should be fused for the quantization


Comments:
Hey @pradeepdev-1995,

I'm not super familiar with `torch.quantization.fuse_models(...)`. Could you explain a bit more what those lines do:

```
model.qconfig            =  torch.quantization.get_default_qconfig('fbgemm')
model_fused             = torch.quantization.fuse_modules(model,[['linear', 'linear']])
```

?

@patrickvonplaten 
could you please check the official documentation https://pytorch.org/docs/stable/quantization.html

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md) are likely to be ignored.
Add option to not load pretrained weights in `AutoModel.from_pretrained()`
Author: changjonathanc
Pull request: no

# 🚀 Feature request

Add option to not load pretrained weights in `AutoModel.from_pretrained()`
analogous to `pretrained=` in `timm.create_model(model_name, pretrained=True/False)`


## Motivation

code like this can be simplified
https://github.com/huggingface/transformers/blob/4469010c1be3a94f74d6448497051468f617baf2/examples/pytorch/question-answering/run_qa_no_trainer.py#L357-L365

## Your contribution

I can help to add this, if people think this is useful.
What should the argument name be? `load_weights: bool = True`?

Comments:
It's just `AutoModel.from_config(AutoConfig.from_pretrained("model_name))` no?

Yes. But it requires if-else, and I don't like that. 🙃

I personally think the API with `from_pretrained` and `from_config` is nice and quite flexible. Adding a second way of doing the same thing looks like it would complicate things :)

That's a good point. Thank you. :)
Add a condition for checking labels
Author: hrxorxm
Pull request: yes

# What does this PR do?

This PR adds stability by checking whether it has labels before returning it in `prediction_step` of `Seq2SeqTrainer`
There is a similar condition in [`prediction_step` of `Trainer`](https://github.com/huggingface/transformers/blob/master/src/transformers/trainer.py#L2460-L2465)
```python
if has_labels:
    labels = nested_detach(tuple(inputs.get(name) for name in self.label_names))
    if len(labels) == 1:
        labels = labels[0]
else:
    labels = None
```

## Before submitting

- [x] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md#start-contributing-pull-requests),
      Pull Request section?


## Who can review?

For `trainer` : @sgugger


Comments:
Torch 1.10
Author: LysandreJik
Pull request: yes

Authorize torch 1.10

Comments:
As a side note. I'm excited that label smoothing is now integrated into cross entropy! https://pytorch.org/docs/stable/generated/torch.nn.CrossEntropyLoss.html

Excited to replace my current implementation with this.

@anton-l @patrickvonplaten @Narsil there's the following error when passing to torch 1.10:

```
udioClassificationPipelineTests.test_small_model_pt _____________
[gw0] linux -- Python 3.7.12 /usr/local/bin/python

self = <tests.test_pipelines_audio_classification.AudioClassificationPipelineTests testMethod=test_small_model_pt>

    @require_torch
    def test_small_model_pt(self):
        model = "anton-l/wav2vec2-random-tiny-classifier"
    
        audio_classifier = pipeline("audio-classification", model=model)
    
        audio = np.ones((8000,))
        output = audio_classifier(audio, top_k=4)
        self.assertEqual(
            nested_simplify(output, decimals=4),
            [
                {"score": 0.0843, "label": "on"},
                {"score": 0.0840, "label": "left"},
                {"score": 0.0837, "label": "off"},
>               {"score": 0.0835, "label": "yes"},
            ],
        )
E       AssertionError: Lists differ: [{'score': 0.0842, 'label': 'no'}, {'score': 0.0838, 'lab[77 chars]ht'}] != [{'score': 0.0843, 'label': 'on'}, {'score': 0.084, 'labe[77 chars]es'}]
E       
E       First differing element 0:
E       {'score': 0.0842, 'label': 'no'}
E       {'score': 0.0843, 'label': 'on'}
E       
E       - [{'label': 'no', 'score': 0.0842},
E       ?              -                 ^
E       
E       + [{'label': 'on', 'score': 0.0843},
E       ?             +                  ^
E       
E       -  {'label': 'up', 'score': 0.0838},
E       ?             ^^                ^^
E       
E       +  {'label': 'left', 'score': 0.084},
E       ?             ^^^^                ^
E       
E       -  {'label': 'go', 'score': 0.0837},
E       ?             -
E       
E       +  {'label': 'off', 'score': 0.0837},
E       ?              ++
E       
E       -  {'label': 'right', 'score': 0.0834}]
E       ?             ^^^^^                 ^
E       
E       +  {'label': 'yes', 'score': 0.0835}]
```

Do you know where it might come from?

@LysandreJik I can't track down which commit made the outputs different yet. The checkpoint certainly didn't change. Is it only torch 1.10 that gets affected?

It seems so!
Align attention_mask dtype to the attention_scores for BERT to speedup mixed precision inference and training
Author: liangan1
Pull request: yes

# Why we need to align the attention_mask dtype to the attention_scores
For the encoder layer in BERT model, the attention_scores need to add the attention_mask info before the add and softmax layer to process the masked batch inputs.
The  attention_mask is always FP32 in this repository, It works well for fully FP32 data type, but when we use mixed precision, the attention_scores is low precision, such as bf16,  and the output of add operation will be FP32 in PyTorch. so the following softmax and dropout will also run with FP32 and introduce addition "to" operation. So, it will obviously reduce the performance for mixed precision performance with FP32 attention_mask. 
# Optimization 
Convert the attention_mask to the same type of  attention_scores  in the first encoder layer. With this optimization, we only need to convert the data type once and the following softmax and dropout will also use low precision data type to reduce memory footprint to improve the performance. 

Comments:
Thank you for the PR!

Note that `attention_mask` is actually in whatever the given `dtype` is. For bert like models the mask is prepared using `get_extended_attention_mask` method and checks and casts the mask to correct `dtype`. cf
https://github.com/huggingface/transformers/blob/62832c962f85b5a554ebf8b930d13b76b9028a8d/src/transformers/modeling_utils.py#L232

So the attention operation should also be in mixed precision.

But for modeling_bert.py the attention_mask is generate with 
`extended_attention_mask: torch.Tensor = self.get_extended_attention_mask(attention_mask, input_shape, device)`
the dtype of extended_attention_mask is same to attention_mask. while attention_mask is the input of model, it is FP32 in general. For mixed precision in PyTorch, we use autocast to do mixed precision, user doesn't need to convert the input dtype explicitly. So, the attention mask will always be FP32 here.  In act, the dtype of mask should be same to the attention_scores. 

>But for modeling_bert.py the attention_mask is generate with
extended_attention_mask: torch.Tensor = self.get_extended_attention_mask(attention_mask, input_shape, device)
the dtype of extended_attention_mask is same to attention_mask

if you go through, `get_extended_attention_mask` you'll find that the mask is casted to `self.dtype`, which calls ` get_parameter_dtype`

https://github.com/huggingface/transformers/blob/62832c962f85b5a554ebf8b930d13b76b9028a8d/src/transformers/modeling_utils.py#L130

which returns the `dtype` of parameter, so the `extended_attention_mask` will have the same `dtype` as that of model parameters. So if params are in `fp16` then `extended_attention_mask` will also be in `fp16`.

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md) are likely to be ignored.
Documentation Mistake: no_multi_processing for Benchmarks
Author: liaeh
Pull request: no

The documentation on setting the argument [no_multi_processing during benchmarking](https://huggingface.co/transformers/v4.0.1/benchmarks.html#benchmark-best-practices) is unclear: _The option **no_multi_processing should only be set to True** for testing and debugging. To ensure accurate memory measurement it is recommended to run each memory benchmark in a separate process by making sure **no_multi_processing is set to True**._


From looking at the [docstrings in benchmark_args_utils.py](https://github.com/huggingface/transformers/blob/master/src/transformers/benchmark/benchmark_args_utils.py#L86), no_multi_processing should only be set to False for testing and debugging.


Comments:
Pinging @patrickvonplaten as the author of https://github.com/huggingface/transformers/pull/5360

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md) are likely to be ignored.
RunTimeError when using prefix_allowed_tokens_fn and top-k/top-p sampling in model.generate
Author: amy-hyunji
Pull request: no

Hi, I'm using T5-large, torch 1.9 and transformers 4.8.2 to generate sentences in the predefined corpus. I tried to use both `prefix_allowed_tokens_fn` and top-k sampling, `do_sample=True; top_k=50`. When I use the two separately, I don't get any problem, but when I try to use the two together, I get a runtimeerror, `RunTimeError: probability tensor contains either 'inf', 'nan' or element<0`. Do you have a guess on why this happens?

The simple code I used for generation is 
```
with open("corpus.pkl", "rb") as f:
   trie = dict(pickle.load(f))

def get(input_ids: List[int]):
   return _get_from_trie(input_ids, trie)

def _get_from_trie(input_ids: List[int], trie_dict: Dict):
   if len(input_ids) == 0:
      output = list(trie_dict.keys())
      return output
   elif input_ids[0] in trie_dict:
      return _get_from_trie(input_ids[1:], trie_dict[input_ids[0]])
   else:
      return []

model = T5ForConditionalGeneration.from_pretrained('t5-large').eval()

outputs = model.generate(
   **input_args.
   num_beams=5,
   prefix_allowed_tokens_fn=lambda batch_id, sent: get(sent.tolist()),
   do_sample=True,
   top_k=50
)
```



Comments:
This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md) are likely to be ignored.
Fine-tuning GPT-J 6B with 16Gb of VRAM
Author: zgerrard
Pull request: no

Hi

I am trying to fine-tune GPT-J 6B with 16Gb of VRAM. Do you have any ideas/pointers on what can be done to lower the amount of VRAM needed for fine-tuning?

@StellaAthena @EricHallahan 


Comments:
I've tried DeepSpeed with Zero-3 and Zero-Offload, and the model still doesn't fit. 

Try checking out the instructions [here](https://github.com/kingoflolz/mesh-transformer-jax/blob/master/howto_finetune.md)

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md) are likely to be ignored.
Fix loading for newer m2m models
Author: harveenchadha
Pull request: yes

Newer version of m2m models have args parameter as None.

args are present in cfg['model']. Fixing input arguments to the function as well.





Comments:
Thanks for the PR @harveenchadha !
Could you post the link for the new m2m models, I couldn't find anything new here https://github.com/pytorch/fairseq/tree/master/examples/m2m_100

Hey Suraj, from newer models I mean models trained with newer version of fairseq. I was trying to convert [Indic Trans](https://github.com/AI4Bharat/indicTrans) and ran into issues using this script.

I see, thanks! PR looks good, just one style check is failing, you could fix it by running `make style` and `make quality`.

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md) are likely to be ignored.

gently pinging @harveenchadha :) 

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md) are likely to be ignored.