noamaanMulla-03/transformers-issues huggingface dataset:
Fix test for file not found
Author: sgugger
Pull request: yes

# What does this PR do?

The test for file not found in the TensorFlow auto model tests is failing on main as the message does not match exactly (see [here](https://app.circleci.com/pipelines/github/huggingface/transformers/53015/workflows/6ea05b10-a541-46db-bcce-b93dc654610e/jobs/636205)). This PR fixes that.

Comments:
_The documentation is not available anymore as the PR was closed or merged._

Merging to have tests passing on main, but I will address any comment in followup PRs :-)
Fix `AutomaticSpeechRecognitionPipelineTests.run_pipeline_test`
Author: ydshieh
Pull request: yes

# What does this PR do?

Fix `AutomaticSpeechRecognitionPipelineTests.run_pipeline_test` which was changed in #19570 and #20104.

See the comments in this PR changes.

I detected this when working on improving pipeline tests using tiny models. Previously, `Speech2TextConfig` is not used in ASR pipeline tests, but now it does, and  gives errors without this PR.

Comments:
_The documentation is not available anymore as the PR was closed or merged._

Thank you !
Onnx CLIP Model outputs "Shape mismatch" warining on inference
Author: abnokubi
Pull request: no

### System Info

- `transformers` version: 4.23.1
- Platform: Linux-5.10.102.1-microsoft-standard-WSL2-x86_64-with-glibc2.29
- Python version: 3.8.10
- Huggingface_hub version: 0.10.1
- PyTorch version (GPU?): 1.12.1+cu116 (True)
- Tensorflow version (GPU?): not installed (NA)
- Flax version (CPU?/GPU?/TPU?): not installed (NA)
- Jax version: not installed
- JaxLib version: not installed
- Using GPU in script?: No
- Using distributed or parallel set-up in script?: No

### Who can help?

@patil-suraj

### Information

- [ ] The official example scripts
- [X] My own modified scripts

### Tasks

- [X] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

1. Convert clip to onnx `python -m transformers.onnx -m openai/clip-vit-base-patch32 onnx/`
2. Try to inference

```
import transformers
import onnxruntime
import numpy as np
from PIL import Image
import torch
model_path = "onnx/model.onnx"
example_image = "swatch01.png"
session = onnxruntime.InferenceSession(model_path)
processor = transformers.CLIPProcessor.from_pretrained("openai/clip-vit-base-patch32")
labels = [
            "a close up of a person's eye photo",
            "a person's arm with a bunch of lipstick swatches on it",
# (snip)
]
img = Image.open(example_image)
inputs = processor(text=candidate_labels, images=img,
 return_tensors="np", padding=True)
ort_inputs = {
    "input_ids": inputs["input_ids"],
    "attention_mask": inputs["attention_mask"],
    "pixel_values": inputs["pixel_values"]
}
ort_outputs = session.run(None, ort_inputs)
```

3. It work but some warinings:
```
2022-10-27 09:23:18.299824270 [W:onnxruntime:, execution_frame.cc:594 AllocateMLValueTensorPreAllocateBuffer] Shape mismatch attempting to re-use buffer. {42,512} != {1,512}. Validate usage of dim_value (values should be > 0) and dim_param (all values with the same string should equate to the same size) in shapes in the model.
2022-10-27 09:23:18.300383069 [W:onnxruntime:, execution_frame.cc:594 AllocateMLValueTensorPreAllocateBuffer] Shape mismatch attempting to re-use buffer. {42,1} != {1,1}. Validate usage of dim_value (values should be > 0) and dim_param (all values with the same string should equate to the same size) in shapes in the model.
```
(`42` is the number of labels)

### Expected behavior

The model work find, but the waring outputs every inference.


Comments:
cc @lewtun 

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.
Update some GH action versions
Author: ydshieh
Pull request: yes

# What does this PR do?

(I am running part of the CI to make sure nothing is broken by this PR)

We get a lot of warnings on CI summary page
```bash
Node.js 12 actions are deprecated. For more information see: https://github.blog/changelog/2022-09-22-github-actions-all-actions-will-begin-running-on-node16-instead-of-node12/. Please update the following actions to use Node.js 16: ...
```
This PR tries to update some of them. The remaining ones include `set-output` command and another one - I will work on that in another PR.


Comments:
_The documentation is not available anymore as the PR was closed or merged._
Fix dtype of weights in from_pretrained when device_map is set
Author: sgugger
Pull request: yes

# What does this PR do?

As reported in #20390, the dtype of the weights after `from_pretrained` is used for a checkpoint is inconsistent between `device_map=None` or `device_map` set:
- `device_map=None` (which uses `nn.Module.laod_state_dict`) will have the dtype of the model stay the same, even if the checkpoints are in a different dtype (so loading a float16 checkpoint in a float32 model gives a float32 model)
- `device_map` set (which manually sets the parameters) will change the dtype of the model to the dtype of the checkpoint (so loading a float16 checkpoint in a float32 model gives a float16 model).

This PR addresses this.

Comments:
_The documentation is not available anymore as the PR was closed or merged._

There is no more safetensors at this stage, (`is_safetensors` means the checkpoint comes from safetensors, but the state dict is a dictionary name to parameter in this case as well).
Documentation missing is_decoder
Author: cronoik
Pull request: no

### System Info

Not relevant. Issue is about online documentation.

### Who can help?

@sgugger @stevhliu 

### Information

- [ ] The official example scripts
- [ ] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

The bert documentation [page](https://huggingface.co/docs/transformers/model_doc/bert)  mentions `is_decoder` 5 times but the BertConfig [class documentation](https://huggingface.co/docs/transformers/model_doc/bert) not a single time. 

This probably affects other models as well.

### Expected behavior

BertConfig class documentation should contain an entry for `is_decoder`.

Comments:
Thanks for flagging. Would you like to open a PR with a fix?
fix natten installation
Author: ydshieh
Pull request: yes

# What does this PR do?

I made a mistake in #20546 and it ended up with
```bash
# For `dinat` model
RUN python3 -m pip install --no-cache-dir natten

RUN python3 -m pip install --no-cache-dir natten -f https://shi-labs.com/natten/wheels/$CUDA/
```
so the CUDA version was not installed (due to `Requirement already satisfied`)

Comments:
_The documentation is not available anymore as the PR was closed or merged._
Pretraing T5 model with run_t5_mlm_flax.py script does not support distributed training with deepspeed
Author: saimunikoti
Pull request: no

### System Info

- `transformers` version: 4.24.0.dev0
- Platform: Linux-3.10.0-1127.18.2.el7.x86_64-x86_64-with-glibc2.17
- Python version: 3.8.13
- Huggingface_hub version: 0.10.1
- PyTorch version (GPU?): 1.8.2+cu111
- Tensorflow version (GPU?): not installed (NA)
- Flax version (CPU?/GPU?/TPU?): 0.6.1
- Jax version: 0.3.25
- JaxLib version: 0.3.25
- Using GPU in script?: <fill in>
- Using distributed or parallel set-up in script?: <fill in>

### Who can help?

@patil-suraj 
@patrickvonplaten 
@stas

### Information

- [X] The official example scripts
- [ ] My own modified scripts

### Tasks

- [X] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

sh file to pretrain T5 model

 deepspeed --hostfile hostfile \
    --master_port <fill in> \
     run_t5_mlm_flax.py \
    --deepspeed deepspeed_configs.json \
    --train_file <fill in> \
    --output_dir <fill in> \
    --model_name_or_path=t5-small \
    --do_train \
    --max_seq_length="512" \
    --num_train_epochs=1 \
    --save_steps=100 \
    --per_device_train_batch_size=4 \
    --warmup_steps=100 \
    --logging_steps=100 \
    --overwrite_output_dir
    
```
ValueError: Some specified arguments are not used by the HfArgumentParser: ['--local_rank=0']
```

### Expected behavior

It seems run_t5_mlm_flax.py uses its own TrainingArguments file which does not define "local_rank" and "deepspeed"attributes (unlike transformers.TrainingArguments which defined these variables).

run_t5_mlm_flax.py should be configured for these attributes in order to train in a distributed manner. 

Comments:
@ArthurZucker could you take a look? :-) 

DeepSpeed only supports PyTorch and the script you mention is for Flax. I don't think there is anything that can be done ;-)

Thanks for letting me know. Is there anything for distributed training on Flax models
[MaskFormer] Add support for ResNet backbone
Author: NielsRogge
Pull request: yes

# What does this PR do?

This PR is part 3 of 3 of the big #20204 PR.

This PR does 2 things:
1) it makes sure that ResNet is supported as backbone for MaskFormer, besides Swin. It leverages the `AutoBackbone` class for this.
2) 
 it makes sure that MaskFormer defaults to Swin as backbone, not relying on MaskFormerSwin, but just on plain Swin. For this, the argument `output_hidden_states_before_downsampling` is added to `SwinConfig`. 
 => `SwinBackbone` will be added in a separate PR

To do:

- [x] convert all remaining MaskFormer checkpoints

Comments:
_The documentation is not available anymore as the PR was closed or merged._

@sgugger feel free to approve, I'm currently uploading all checkpoints to the hub :)
Add BiT + ViT hybrid
Author: NielsRogge
Pull request: yes

# What does this PR do?

This PR adds ViT hybrid to the library. As ViT hybrid uses BiT as backbone, this PR also adds BiT as a standalone model.

BiT itself is very similar to a ResNetv2, except that it replaces batch norm layers by group norm and uses "weight standardized" convolutional layers.

To do:

- [x] add image processors
- [ ] add tests for image processors (cc @amyeroberts can I directly add test_modeling_image_processor_xxx.py ?)
- [ ] transfer all checkpoints
- [x] add integration tests

Comments:
_The documentation is not available anymore as the PR was closed or merged._

Thanks so much @sgugger for your review ! 
I should have updated everything and the main models are now up:
- https://huggingface.co/google/vit-hybrid-base-bit-384 
- https://huggingface.co/google/bit-50
Add DPT hybrid
Author: younesbelkada
Pull request: yes

# What does this PR do?

Adds DPT Hybrid support to `transformers`

Do not merge until #20550 gets merged

cc @NielsRogge @sgugger @patrickvonplatten @patil-suraj 


Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_20612). All of your documentation changes will be reflected on that endpoint.

You should have opened your PR to go on the branch adding BiT and VitHybrid as the PR is not easy to review as it is.

Yes sorry :/  Let me open a PR on the other branch

Here is a much cleaner version of the PR: https://github.com/NielsRogge/transformers/pull/51 ;) 

Closing in favor of https://github.com/huggingface/transformers/pull/20645
[Trainer] add error when passing `8bit`models
Author: younesbelkada
Pull request: yes

# What does this PR do?

Before this PR, any user could load an 8bit model and pass it to a Trainer, which is wrong. In fact, it is not possible to train an 8bit model (yet). Therefore we should raise an error until this will be supported in the future

Related: https://github.com/huggingface/transformers/issues/20348#issuecomment-1335106257

cc @ydshieh @sgugger 



Comments:
_The documentation is not available anymore as the PR was closed or merged._
[`ViTHybrid`] + [`BiT`] cleaner `__init__`
Author: younesbelkada
Pull request: yes

# What does this PR do?

a function in `BiT` is not needed, thus this PR makes the codebase less error-prone.
Before that, to get the feature maps size from the backbone model, `ViTHybrid` assumed that the backbone has the method `_get_feature_map` which is not the case for all backbones.

Related #20645

Comments:
_The documentation is not available anymore as the PR was closed or merged._

Yes, this can be addressed in a future PR
Update summarization `run_pipeline_test`
Author: ydshieh
Pull request: yes

# What does this PR do?

Update summarization `run_pipeline_test`.

A few more models can handle longer sequences, and won't give expected exception at this place
```python
            with self.assertRaises(Exception):
                outputs = summarizer("This " * 1000)
```
So we need to ignore those model config classes.

Comments:
_The documentation is not available anymore as the PR was closed or merged._
Speed up git-lfs detection on error
Author: xloem
Pull request: yes

Prevent read and discard of entire checkpoint file.

# What does this PR do?

Mutates an error handler that checks only 7 bytes, to only read those 7 bytes rather than an entire checkpoint file.

Fixes # (issue)

Issue not opened. I encountered a memory allocation crash here when exploring disk offloading.

## Before submitting
- [X] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).

## Who can review?

Anyone in the community is free to review the PR once the tests have passed. Feel free to tag
members/contributors who may be interested in your PR.

@sgugger

Comments:
_The documentation is not available anymore as the PR was closed or merged._
Fine-tuning translation model speed anomalies
Author: chaodreaming
Pull request: no

### System Info

python 3.8.12
ubuntu18.04
transformers 4.23.1
![image](https://user-images.githubusercontent.com/49591435/198196407-2a20a98e-130e-4699-97a9-63836cb231dd.png)


### Who can help?

@Narsil
@patil-suraj
@sgugger, @patil-suraj

### Information

- [X] The official example scripts
- [ ] My own modified scripts

### Tasks

- [X] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

https://colab.research.google.com/drive/1FwXCjUVvrNpCuf0KoxhxQKTciRscva6t?usp=sharing

t4 graphics card half precision about 10, a100 is more than 300, even if not up to 30 times the speed should not be almost the same speed, and here the multi-threaded seems to be bad

### Expected behavior

After experiments found that bs has a greater impact on the speed, gradient accumulation also has a slight impact, but the same bs, a100 and t4 speed is almost the same, the expected is to start multi-threaded or other methods to accelerate, otherwise, training once to more than 100 days

Comments:
Please use the [forums](https://discuss.huggingface.co/) to debug training like this as we keep features for bugs (with a clear reproducer) and feature requests only.

I think it is a bug in itself, I tried many devices and the speed is almost the same, obviously not reasonable

> t4 graphics card half precision about 10, a100 is more than 300, even if not up to 30 times the speed should not be almost the  same speed, and here the multi-threaded seems to be bad

I read this 3 times, and I still don't understand. What do you mean ?

Just as a note for here or the forums, trying to be over explicit might help readers understand what you're trying to do and what are your expectations.

Sorry, the message is machine translation, I mean there are two problems, the first problem is that different gpu speed should be different, a100 is better than t4, especially the semi-precision gap is big, so turn on the semi-precision training speed should be a big gap, however the speed is almost the same. The second problem is that multi-threading does not seem to have any accelerating effect, and can be improved a lot on cv, I do not know if this is the case in the nlp field


8 card speed is no different from 1 card

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.
[Trainer] Corrects typing of Trainer __init__ args 
Author: julianmack
Pull request: yes

# What does this PR do?

Corrects typing for Trainer class. Updates typing to match change made in https://github.com/huggingface/transformers/pull/19158/ and fixes a few other typing issues while I'm there

Unless I'm missing something these changes take the typing to parity with both class docstring and implementation 

Who can review: @sgugger

## Before submitting
- [X] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).


Comments:
_The documentation is not available anymore as the PR was closed or merged._
Add DPT-hybrid
Author: NielsRogge
Pull request: no
Labels: name: New model

### Model description

DPT-hybrid is used in Stable Diffusion 2.0's depth model and is also a nice depth estimation model in general.

We already support [DPT](https://huggingface.co/docs/transformers/model_doc/dpt) - which is also the default model of our [depth estimation pipeline](https://huggingface.co/docs/transformers/main/en/main_classes/pipelines#transformers.DepthEstimationPipeline), but not DPT-hybrid. The latter uses ViT-hybrid as backbone.

Hence, we first would need to add ViT-hybrid to the library. This model is very similar to a regular ViT, except that instead of patchifying an image and embedding each patch, this model uses a pre-trained ResNetv2 to embed the image before feeding the features to a Transformer encoder. 

This means that we first need to add ResNetv2 to the library. 😅 however that seems fairly doable, I'd recommend porting the one from timm: https://github.com/rwightman/pytorch-image-models/blob/main/timm/models/resnetv2.py.

### Open source status

- [X] The model implementation is available
- [X] The model weights are available

### Provide useful links for the implementation

DPT-hybrid is available here: https://github.com/isl-org/MiDaS

Comments:
Hi @NielsRogge Can I take this up if no one is working on it?

This would be great @nandwalritik ! The Stable Diffusion Depth estimation model depends on it, so we'd definitely help you in whatever way we can :-) 

@NielsRogge should we port the ResNetv2 exactly like ResNetv1: https://github.com/huggingface/transformers/blob/main/src/transformers/models/resnet/modeling_resnet.py or could we directly port it from `timm`?

Could you provide some more links on how to add ViT-Hybrid?

@patrickvonplaten yes we can add ResNetv2 in the same way as we added Resnet V1. Meaning, a separate standalone model in the library.

Once we have that, we have everything we need to define modeling_dpt_hybrid.py. This would be a copy of modeling_dpt.py, except that we leverage ResNetv2 (we can do that using the new AutoBackbone API which was just added) in the DPTViTEmbeddings class. 

For stable diffusion there's no need to add hybrid ViT as a standalone model now (i.e. we can add modeling_hybrid_vit.py another time).

@NielsRogge @patrickvonplaten Can you let me know the next steps, what should I add first?
ResNetv2 ->  DPT-hybrid, leaving ViT-hybrid as Niels mentioned above, will this be the order?

@nandwalritik yes, let's start by adding ResNetv2, based on timm's implementation found here: https://github.com/rwightman/pytorch-image-models/blob/main/timm/models/resnetv2.py.

Next, we can add DPT-hybrid, based on the modeling files here: https://github.com/isl-org/DPT/tree/main/dpt

This can be done in 2 separate PR's

Ok I Will start with ResNetv2 and then I will try adding DPT-hybrid.

Should I use `add-new-model` or `add-new-model-like` command as we already have ResNet implementation in huggingface ?

You can use `add-new-model-like` and start from `resnet` (we might deprecate `add-new-model`)
Add `dpt-hybrid` support
Author: younesbelkada
Pull request: yes

# What does this PR do?

Adds `DPT-hybrid` support in `transformers` 
Currently only DPT is supported. This PR leverages `AutoBackbone` from @NielsRogge to replace the embedding layer from `DPT` to support `DPT-hybrid`

Fixes #20435 

Model weights: https://huggingface.co/Intel/dpt-hybrid-midas

Comments:
_The documentation is not available anymore as the PR was closed or merged._

Thanks a bunch! Fortunately it was already on the config file :D  https://huggingface.co/Intel/dpt-hybrid-midas/blob/main/config.json#L277 but will open a PR to remove the `embedding_type` as it is not needed anymore

The config file has been modified, merging!
how to save a model with additional layers, so it can be loaded using .from_pretrained()?
Author: christianjoun
Pull request: no

## Environment info

- `transformers` version: 2.3.0
- Platform: Linux-4.15.0-135-generic-x86_64-with-glibc2.27
- Python version:  3.8.12
- PyTorch version (GPU?): 1.8.1+cu111 (True)
- Tensorflow version (GPU?): not installed (NA)
- Using GPU in script?: YES (T4 TESLA)
- Using distributed or parallel set-up in script?: no

### Who can help
@LysandreJik

## Information

Model I am using Bert: Text-classification

## To reproduce

Steps to reproduce the behavior:

I have a custom model

`class BERTClass(torch.nn.Module):
    def __init__(self, num_labels=4):
        super(BERTClass, self).__init__()
        self.l1 = BertModel.from_pretrained('bert-base-multilingual-uncased')
        self.l2 = torch.mean
        self.l3 = torch.nn.Linear(768, num_labels)

    def forward(self, ids, masks, token_type_ids):        
        last_hidden_state, _ = self.l1(ids, attention_mask=masks, token_type_ids=token_type_ids)
        avg_pooling = self.l2(last_hidden_state, dim=1)
        output = self.l3(avg_pooling)
        
        return output`

How do I go about saving my custom model so that I can run BertModel.from_pretrained('./mycustomodel')

Do you have an example I can follow by chance?

Currently I am saving the model this way

`
model_2_save = model.module if hasattr(model, "module") else model

checkpoint = {
    'epoch': args.epochs,
    'num_labels': args.num_labels,
    'max_text_length': MAX_TEXT_LENGTH,
    'state_dict': model_2_save.state_dict()
}

torch.save(checkpoint, args.model_dir + "/pt_model.pt")
`
Thanks for the help!


Comments:
Hi! For questions like this, it would be best to ask this on the [forum](https://discuss.huggingface.co/). We use issues for bug reports and feature requests. Thanks!

To answer your question: The [sharing custom model guide](https://huggingface.co/docs/transformers/master/en/custom_models) might help with this .

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.

@christianjoun did you solve this problem?