noamaanMulla-03/transformers-issues huggingface dataset:
Implement safetensors checkpoint loading for Trainer
Author: viktor-shcherb
Pull request: no

### Feature request

At the moment, Trainer loads models with `torch.load` method directly onto the cpu:

(`Trainer._load_from_checkpoint` method)
```python
                ... 
                # We load the model state dict on the CPU to avoid an OOM error.
                state_dict = torch.load(os.path.join(resume_from_checkpoint, WEIGHTS_NAME), map_location="cpu")
                # workaround for FSDP bug https://github.com/pytorch/pytorch/issues/82963
                # which takes *args instead of **kwargs
                load_result = model.load_state_dict(state_dict, False)
                # release memory
                del state_dict
                self._issue_warnings_after_load(load_result)
                ...
```

Loading on cpu with safetensors is a lot faster, so this method should (?) be preffered if safetensors library is installed.

### Motivation

I care to speed up checkpointing process since I use it to store the best model checkpoint every time the metric is improved (via callback).

### Your contribution

The change should be straightforward, but I am not sure if safetensors checkpointing should be defaulted to if safetensors are installed, or configured manually like in `PreTrainedModel.from_pretrained` method.

Comments:
The checkpoints are not saved in that format so there is no `model.safetensors` file to load from. We could add a training argument to use this format instead of the PyTorch format indeed.
[HOW TO FINETUNE CLIP OPENAI LAION2B MODELS FOR IMAGE CLASSIFICATION]
Author: lamnt2008
Pull request: no
Labels: name: WIP

I try to finetune CLIP model by using pretrained: https://huggingface.co/laion/CLIP-ViT-B-32-laion2B-s34B-b79K
But I met a bug: 
![image](https://user-images.githubusercontent.com/124332581/229679584-34db65ad-5a43-423e-bcd9-c54902fe7d6b.png)
Help me! Thanks

Comments:
There is no need to yell at us in all caps. No one can do anything without seeing the code you run.
Whisper should suppress task tokens
Author: guillaumekln
Pull request: no

### System Info

- `transformers` version: 4.27.4
- Platform: Linux-5.19.0-38-generic-x86_64-with-glibc2.35
- Python version: 3.10.6
- Huggingface_hub version: 0.13.3
- PyTorch version (GPU?): 2.0.0+cu117 (True)

### Who can help?

@ArthurZucker @sanchit-gandhi 

### Information

- [ ] The official example scripts
- [X] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [X] My own task or dataset (give details below)

### Reproduction

I'm not sure if it is the best place to report the issue, but openai/whisper updated the default list of suppressed tokens in [this commit](https://github.com/openai/whisper/commit/eab8d920edf3947294c466f3912c24ed4b191264). In particular the task tokens should now be suppressed by default:

```python
>>> import transformers
>>> config = transformers.AutoConfig.from_pretrained("openai/whisper-tiny")
>>> tokenizer = transformers.AutoTokenizer.from_pretrained("openai/whisper-tiny")
>>> tokenizer.convert_tokens_to_ids("
") in config.suppress_tokens
False
>>> tokenizer.convert_tokens_to_ids("
") in config.suppress_tokens
False
```

### Expected behavior

The default list of suppressed tokens should be match the list in the reference implementation.

All Whisper configurations should be updated accordingly.

Comments:
Nice! You indeed reported this to the correct place, thanks for this catch. We should update the model! 

Great catch! Would you like to open some PR's on the HF Hub to fix these `suppress_tokens` lists @guillaumekln? E.g. for the `tiny` model we need to update the list of `suppress_tokens` in the generation config: https://huggingface.co/openai/whisper-tiny/blob/a8d76517e6d65d92771752dbbf5e9c0a1a5b3a0d/generation_config.json#L126

Sure, I just opened pull requests for all Whisper models on the Hub. I updated both `config.json` and `generation_config.json` in separate PRs.

Thank you for the Hub PRs @guillaumekln, very clean! They all looked correct so have merged them on the Hub side 👍 Closing as complete!
Add TF port of BLIP
Author: Rocketknight1
Pull request: yes

Work in progress right now, will update this when it's closer to being ready!

Comments:
_The documentation is not available anymore as the PR was closed or merged._

The TF port is mostly complete now and tests are passing locally - I just need to go around updating docs and auto classes and so on. The main code should be ready for review!

Looks like there are many comments to address for now. Please ping me again when it's ready for second review!

Got through a lot of the comments today, but I have a couple of other things to do - will try to finish them tomorrow!

The last remaining big issue is that some of the pt-tf equivalence tests fail when weights don't match up between models. This is caused by the cross-attention weights not being built, presumably because those layers aren't being called in the forward pass. I'm working on figuring out why and resolving that!

The issue seems to be that in all of our other models, cross-attention layers are only added when `config.add_cross_attention` is True, but in the case of BLIP it only checks `config.is_decoder`. As a result, the PyTorch models often initialize cross-attention layers that aren't used, which causes weight mismatch issues for us in crossloading tests, because TF only creates weights on first use.

It's coming, don't worry! This cross-attention behaviour is just very odd and I'm trying to track it down first

Hi all! I've addressed all comments and local tests look good. The remaining issues are:

- Converting checkpoints so the tests don't need `from_pt`
- Maybe adding more auto classes

I'm not sure about the auto classes, though - they're missing in the original PT version of the model as well, so this didn't seem like the right PR to add them. 

cc @sgugger - I think this is ready for a final review at last!

Got it, I'll figure out some way to re-enable those tests, or override them with versions that do work!

@sgugger this should be ready for review with all comments addressed! The failing test is in an unrelated model

@sgugger Sorry for the confusion - that equivalence test is present in both the `test_modeling_tf_blip` and `test_modeling_blip` file. Do we want to keep it in both?

Yes we do.

Going to leave the `pt-to-tf` changes in this PR rather than making a separate one, since they're needed for proper BLIP conversion!
corrected the code comment for the output of find_pruneable_heads_and_indices 
Author: SunHaozhe
Pull request: yes

# What does this PR do?

This PR improves the doc (code comment), because the code comment of find_pruneable_heads_and_indices was not correct. 

## Before submitting
- [x] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [x] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#start-contributing-pull-requests),
      Pull Request section?
- [ ] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [ ] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/main/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/main/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests?


## Who can review?

@anmolsjoshi @sgugger






Comments:
_The documentation is not available anymore as the PR was closed or merged._
Flax Regnet
Author: Shubhamai
Pull request: yes

# What does this PR do?

Flax Implementation of [facebook/regnet-y-040](https://huggingface.co/facebook/regnet-y-040)


## Before submitting
- [ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [x] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#start-contributing-pull-requests),
      Pull Request section?
- [ ] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [x] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/main/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/main/docs#writing-source-documentation).
- [x] Did you write any new necessary tests?


## Who can review?

Anyone in the community is free to review the PR once the tests have passed. Feel free to tag
members/contributors who may be interested in your PR.

- Flax: sanchit-gandhi

Comments:
_The documentation is not available anymore as the PR was closed or merged._

@sanchit-gandhi This is now ready for your review, thanks a lot for your time.

@sanchit-gandhi All the requested changes have been made and looks ready for next iteration of review, thanks a lot for your time.
Unable to import VGG16 model transformers
Author: NagaVenkataSaiM
Pull request: no
Labels: name: New model

### Model description

i have recently upload my trained vgg16 model to hugging face.After uploading i have a prompt of instructions to use my model.
Although i have followed the prompt i got errors.
[https://huggingface.co/Nvsai/DeviceClassification](url)
>>> from transformers import VGG16
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
ImportError: cannot import name 'VGG16' from 'transformers' (/mnt/mydrive/ubantu/programming/openvino/lib/python3.9/site-packages/transformers/__init__.py)
>>> 
>>> model = VGG16.fro
![Screenshot from 2023-04-04 20-45-24](https://user-images.githubusercontent.com/87435205/229841900-e12cee0f-69a1-4dd5-9332-2f65f177e8cf.png)
m_pretrained("Nvsai/DeviceClassification")
![Screenshot from 2023-04-04 20-45-56](https://user-images.githubusercontent.com/87435205/229841929-812f7eb6-58e1-4919-aff6-35200aee426c.png)


### Open source status

- [ ] The model implementation is available
- [X] The model weights are available

### Provide useful links for the implementation

_No response_

Comments:
That code sample is plain wrong, there is no VGG16 in Transformers.

@sgugger Hi, can you please suggest me how to use my vgg16 model using transformers?Also i am new to transformers model i am sorry if i made any mistake in uploading my model.Could please redirect me to some helpful resources for vgg16 with transformers?
Fix convert_opt_original_pytorch_checkpoint_to_pytorch.py typo
Author: larekrow
Pull request: yes

# What does this PR do?

`load_checkpoint()` silently fails because `".qkj_proj." in key` is always `False`, but will eventually cause an error at `model.load_state_dict(state_dict)`. This PR fixes the typo that causes this issue.

## Before submitting
- [ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [x] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#start-contributing-pull-requests),
      Pull Request section?
- [ ] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [ ] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/main/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/main/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests?

## Who can review?

@patrickvonplaten 

Comments:
Why do not use 'torch.nn.MultiheadAttention' to substitude 'Class BertSelfAttention+BertSelfOutput' for pytorch
Author: daydayfun
Pull request: no
Labels: name: wontfix

name: Migration

# 📚 Migration

## Information
pytorch has 'torch.nn.MultiheadAttention' 
https://pytorch.org/docs/1.3.0/nn.html?highlight=multihead#torch.nn.MultiheadAttention

## Details
1. For Better performace and generality, I suggest to use  'torch.nn.MultiheadAttention'  instead of 'Class BertSelfAttention+BertSelfOutput'  in BERT model





Comments:
'BertSelfAttention+BertSelfOutput'   is tensorflow style
'torch.nn.MultiheadAttention' is real pytorch style

This issue has been automatically marked as stale and been closed because it has not had recent activity. Thank you for your contributions.

If you think this still needs to be addressed please comment on this thread.

The layers from the Pytorch will be significantly faster than using the two classes like in TensorFlow. Can't we make an exception for pytorch to use the optimized layers?
Question about masking tokens in DataCollatorForPermutationLanguageModeling
Author: WenjieDu
Pull request: no

According to the description from XLNet paper, the permutation LM does not need to corrupt the input in its data processing, and this is different from the Masked LM in BERT. But I found the below code in `DataCollatorForPermutationLanguageModeling` for XLNet replacing the specific tokens in the input data with the given mask_token. I'm confused about this operation. Could you please kindly make a simple explanation? Thank you very much in advance.

https://github.com/huggingface/transformers/blob/b29fd6971d9cd6ba2a824628effe243f543b8f61/src/transformers/data/data_collator.py#L1295

Comments:
Support output_scores in XLA TF generate
Author: gyin94
Pull request: no
Labels: name: WIP

### Feature request

Support output_scores in XLA TF generate.

### Motivation

The scores would be critical and widely used in generation models for downstream applications. It is a confidence score for downstream thresholding or ranking. But it is not yet supported for XLA TF generate. 

### Your contribution

N/A

Comments:
cc @gante @sgugger 

Hey @rossbucky 👋 The support for returning scores with XLA is in our plans. Like other XLA-related changes, it requires rewriting that part of the code to work with static shapes, as opposed to lists of tensors that grow as generate runs. 

The next round of `generate` changes will focus on correctness (e.g. adding informative error messages). This feature will come right after that -- I'm keeping this issue open to track it publicly. 

Hi, just curious if there is any update on this issue? Or, are there any known alternatives for easily getting confidence scores w/ XLA enabled, particularly for greedy decoding? It seems beam_search outputs a `sequences_score` when `return_dict_in_generate` is `True`, but don't see anything similar for greedy decoding.

No updates :) I have yet to build the appropriate piping to update and return those variables
Fix inverted conditional in TF common test!
Author: Rocketknight1
Pull request: yes

Noticed a rather alarming conditional being backwards in the `test_pt_tf_model_equivalence` common test. This probably resulted in a lot of tests being skipped!

Comments:
_The documentation is not available anymore as the PR was closed or merged._

As expected this has raised a few bugs in the cross-test that were silent before - I'll see what I can do in this PR

Most likely - I'll investigate them all soon!

Quick summary of the fixes needed:

ESM: `TFEsmForTokenClassification` copied the computation from `TFBertForTokenClassification`, but this has some slightly odd BERT-specific behaviour and doesn't mask -100 in the same way as other models. Replaced it with the loss block from `TFRobertaForTokenClassification` and all tests pass.

GPT2: For model classes that take rank-3 inputs (e.g. `MultipleChoice` or `DoubleHeads`), when `output_hidden_states=True` , inputs have their second two dims flattened internally in the main model stem. This means that the output `hidden_states` are rank 3 `(bsz, seq_len * num_choices, hidden_dim)` and not rank 4 `(bsz, num_choices, seq_len, hidden_dim)`. However, the PT model un-flattens the output for the final `hidden_states`, which means the last hidden state is rank-4, unlike the others which remain rank-3. In the old TF model, all hidden states are rank-3. I modified the TF code to un-flatten the last hidden state in the same way.

HUBERT: Loss computation especially for CTC overflows a lot with the default labels, which creates lots of `inf` values and makes it very hard to compare TF and PT losses. I skipped PT-TF equivalence testing for the losses, but keep it for all non-loss outputs.

Wav2Vec2: Same as HUBERT

XGLM: The PT XGLM model does a weird thing where it shifts labels by 1 and then adds `pad_token_id` as the final label to all samples. I'm not sure this is correct, but I modified the TF code to do the same. It's possible the TF code is the right one here though, in which case we should revert it and change the PT code instead.

@gante I fixed all the bugs that this surfaced, explained above ^

cc @sgugger for final review too

Thank you for the fix  @Rocketknight1 ❤️ . And I apologize for the mistake I introduced ...
Compact Transformer
Author: astariul
Pull request: no
Labels: name: New model

### Model description

# Escaping the Big Data Paradigm with Compact Transformers

Abstract :

> With the rise of Transformers as the standard for language processing, and their advancements in computer vision, there has been a corresponding growth in parameter size and amounts of training data. Many have come to believe that because of this, transformers are not suitable for small sets of data. This trend leads to concerns such as: limited availability of data in certain scientific domains and the exclusion of those with limited resource from research in the field. In this paper, we aim to present an approach for small-scale learning by introducing Compact Transformers. We show for the first time that with the right size, convolutional tokenization, transformers can avoid overfitting and outperform state-of-the-art CNNs on small datasets. Our models are flexible in terms of model size, and can have as little as 0.28M parameters while achieving competitive results. Our best model can reach 98% accuracy when training from scratch on CIFAR-10 with only 3.7M parameters, which is a significant improvement in data-efficiency over previous Transformer based models being over 10x smaller than other transformers and is 15% the size of ResNet50 while achieving similar performance. CCT also outperforms many modern CNN based approaches, and even some recent NAS-based approaches. Additionally, we obtain a new SOTA result on Flowers-102 with 99.76% top-1 accuracy, and improve upon the existing baseline on ImageNet (82.71% accuracy with 29% as many parameters as ViT), as well as NLP tasks. Our simple and compact design for transformers makes them more feasible to study for those with limited computing resources and/or dealing with small datasets, while extending existing research efforts in data efficient transformers.

### Open source status

- [X] The model implementation is available
- [X] The model weights are available

### Provide useful links for the implementation

Paper : https://arxiv.org/pdf/2104.05704.pdf
Github repository : https://github.com/SHI-Labs/Compact-Transformers

Comments:
Are you willing to collaborate to make this available at HF transformers? @astariul . If so, please connect with me


Hi @astariul and @navinelahi, are there any updates on this issue? May I start working on this?
Add DFFT
Author: NielsRogge
Pull request: no
Labels: name: New model

### Model description

DFFT is a new fully Transformer-based object detector. The model doesn't require a decoder, unlike DETR.

### Open source status

- [X] The model implementation is available
- [X] The model weights are available

### Provide useful links for the implementation

Paper: https://arxiv.org/abs/2206.06829

Github repo (and weights): https://github.com/Pealing/DFFT

Comments:
Hello, is this issue still open? I would like to contribute to it.

@NielsRogge Anyone working on this?

Thank you

Feel free to start working on this!

@NielsRogge if no one is working on this then I would like to start working on it

Cool, feel free to open a draft PR

Hi @NielsRogge, @soma2000-lang, are there any updates on this issue? I would like to contribute.

@atharvakavitkar  working to open a draft pr
Fix NameError `init_empty_weights` when importing Blip2Processor
Author: TheAdamEvans
Pull request: yes

## Context 
Was trying to run [https://huggingface.co/Salesforce/blip2-flan-t5-xl](https://huggingface.co/Salesforce/blip-image-captioning-base) in a colab notebook (after installing `accelerate`) and was getting an error when importing Blip2Processor:
```
NameError `init_empty_weights` is not defined
```
<img width="1285" alt="Screen Shot 2023-04-01 at 1 31 17 pm" src="https://user-images.githubusercontent.com/3723005/229265367-36eed050-b75b-4d06-b69b-f80a649e2d2e.png">

## Changes
Splitting out the if statements to check for `accelerate` and `bitsandbytes` separately seems to fix this problem, in `load_in_8bit` mode.

Applies a similar fix to the imports in `deepspeed.py`. This fixed the same error as above but in this conditional block, later in the file.
```
    if is_deepspeed_zero3_enabled():
        import deepspeed

        [logger.info](http://logger.info/)("Detected DeepSpeed ZeRO-3: activating zero.init() for this model")
        init_contexts = [deepspeed.zero.Init(config_dict_or_path=deepspeed_config())] + init_contexts
```

After these changes, I was able to import, Blip2Processor, Blip2ForConditionalGeneration on Colab :)

Comments:
@pacman100 Is this something you'd be able to review for me? This my first contribution in this repo but I'm pretty sure it fixes a real issue in loading BLIP2.

_The documentation is not available anymore as the PR was closed or merged._

Perhaps @NielsRogge, since this is your implementation of BLIP2, you could review for me? I was able to reproduce the error in colab.

Thanks for the review @sgugger, makes sense now
Move back doctest instructions to setup.cfg
Author: sgugger
Pull request: yes

# What does this PR do?

As a result of #22539, the options we have for the doctests are now all ignored. This PR reverts the change for those and puts them back in `setup.cfg`.

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_22587). All of your documentation changes will be reflected on that endpoint.