noamaanMulla-03/transformers-issues huggingface dataset:
Add EfficientNet Model - PyTorch
Author: tanaymeh
Pull request: no
Labels: name: New model

# 🌟 New model addition

I would like to add the EfficientNet model to the Transformers library

## Model description
EfficientNet was proposed in the paper, **[EfficientNet: Rethinking Model Scaling for Convolutional Neural Networks](https://arxiv.org/abs/1905.11946)**.

EfficientNet is a convolutional neural network architecture and scaling method that uniformly scales all dimensions of depth/width/resolution using a compound coefficient. Unlike conventional practice that arbitrary scales these factors, the EfficientNet scaling method uniformly scales network width, depth, and resolution with a set of fixed scaling coefficients.

EfficientNets also transfer well and achieve state-of-the-art accuracy on CIFAR-100 (91.7%), Flowers (98.8%), and 3 other transfer learning datasets, with an order of magnitude fewer parameters.

<!-- Important information -->

## Open source status

* [x] the model implementation is available: [EfficientNet-PyTorch](https://github.com/lukemelas/EfficientNet-PyTorch)
* [x] the model weights are available: [PyTorch weights](https://github.com/lukemelas/EfficientNet-PyTorch/releases/tag/1.0)
* [x] who are the authors: Mingxing Tan, Quoc V. Le


Comments:
cc @NielsRogge 

@LysandreJik is anyone working on this?? if not can i work on this

Hi @IMvision12, because of existing commitments, I am no longer working on this.

If you want, you can take it up :)

@NielsRogge Will it be worthwhile to include this model?
Add docstrings and fix VIVIT examples
Author: Geometrein
Pull request: yes

# What does this PR do?

- Add docstring for `sample_frame_indices`method
- Fix broken examples for VIVIT
  - Add missing `Pytorch` & `VivitForVideoClassification` imports
  - Resolve the referenced before assignment error caused by an undeclared `videoreader` variable.


<!--
Congratulations! You've made it this far! You're not quite done yet though.

Once merged, your PR is going to appear in the release notes with the title you set, so make sure it's a great title that fully reflects the extent of your awesome contribution.

Then, please replace this with a description of the change and which issue is fixed (if applicable). Please also include relevant motivation and context. List any dependencies (if any) that are required for this change.

Once you're done, someone will review your PR shortly (see the section "Who can review?" below to tag some potential reviewers). They may suggest changes to make the code even better. If no one reviewed your PR after a week has passed, don't hesitate to post a new comment @-mentioning the same persons---sometimes notifications get lost.
-->

<!-- Remove if not applicable -->


## Before submitting
- [X] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).


## Who can review?

Anyone in the community is free to review the PR once the tests have passed. Feel free to tag
members/contributors who may be interested in your PR.

<!-- Your PR will be replied to more quickly if you can figure out the right person to tag with @

 If you know how to use git blame, that is the easiest way, otherwise, here is a rough guide of **who to tag**.
 Please tag fewer than 3 people.

Models:

- text models: @ArthurZucker and @younesbelkada
- vision models: @amyeroberts
- speech models: @sanchit-gandhi
- graph models: @clefourrier

Library:

- flax: @sanchit-gandhi
- generate: @gante
- pipelines: @Narsil
- tensorflow: @gante and @Rocketknight1
- tokenizers: @ArthurZucker
- trainer: @sgugger

Integrations:

- deepspeed: HF Trainer/Accelerate: @pacman100
- ray/raytune: @richardliaw, @amogkam

Documentation: @sgugger, @stevhliu and @MKhalusova

HF projects:

- accelerate: [different repo](https://github.com/huggingface/accelerate)
- datasets: [different repo](https://github.com/huggingface/datasets)
- diffusers: [different repo](https://github.com/huggingface/diffusers)
- rust tokenizers: [different repo](https://github.com/huggingface/tokenizers)

Maintained examples (not research project or legacy):

- Flax: @sanchit-gandhi
- PyTorch: @sgugger
- TensorFlow: @Rocketknight1

 -->


Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_25628). All of your documentation changes will be reflected on that endpoint.

cc @Rocketknight1 

@Geometrein Thanks for the PR - ping me whenever you're happy for me to merge!

Thanks for your review @Rocketknight1!
It's good to go, feel free to merge :)
Inconsistent results between LlamaTokenizer and LlamaTokenizerFast
Author: BugWriter2
Pull request: no

### System Info

- `transformers` version: 4.30.2
- Platform: macOS-10.16-x86_64-i386-64bit
- Python version: 3.8.15
- Huggingface_hub version: 0.16.4
- Safetensors version: 0.3.1

### Who can help?

@ArthurZucker

### Information

- [X] The official example scripts
- [ ] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [x] My own task or dataset (give details below)

### Reproduction

```
from transformers import LlamaTokenizerFast, LlamaTokenizer

fast_tokenizer = LlamaTokenizerFast.from_pretrained('openlm-research/open_llama_7b')
slow_tokenizer = LlamaTokenizer.from_pretrained('openlm-research/open_llama_7b')

input_text = 'tasks'

slow_tokenizer.encode(input)  # [1, 8931]
fast_tokenizer.encode(input)  # [1, 31822, 31824, 5577]
```

### Expected behavior

.

Comments:
Hey! Sorry but I cannot reproduce your issue:

```python 
In [2]: slow_tokenizer.encode(input_text)
Out[2]: [1, 8931]

In [3]: fast_tokenizer.encode(input_text)
Out[3]: [1, 8931]
``` 
Make sure you have the latest version of tokenizers maybe? 

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.
blip2 always decode eos_token first
Author: TobiasLee
Pull request: no

### System Info


- `transformers` version: 4.31.0
- Platform: Linux-5.19.0-41-generic-x86_64-with-glibc2.31
- Python version: 3.9.16
- Huggingface_hub version: 0.14.1
- Safetensors version: 0.3.1
- Accelerate version: 0.21.0
- Accelerate config:    not found
- PyTorch version (GPU?): 2.0.0 (False)
- Tensorflow version (GPU?): not installed (NA)
- Flax version (CPU?/GPU?/TPU?): not installed (NA)
- Jax version: not installed
- JaxLib version: not installed
- Using GPU in script?: <fill in>
- Using distributed or parallel set-up in script?: <fill in>


### Who can help?

_No response_

### Information

- [X] The official example scripts
- [ ] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [X] My own task or dataset (give details below)

### Reproduction

I just run the code provided by the example given in the doc:

https://huggingface.co/docs/transformers/main/model_doc/blip-2#transformers.Blip2ForConditionalGeneration.forward.example





### Expected behavior

I got exactly the same caption of the image. 
generated_text
'two cats laying on a couch'

But its weird that generated ids look like:
generated_ids:
tensor([[    2,  7109, 10017, 11963,    15,    10, 16433, 50118]])

where he leading `2` means a `
` is generated. Is this expected or something wrong with the decoding?

@amyeroberts @ArthurZucker  do you guys have any ideas on it?

Comments:
cc @younesbelkada 

Hi @TobiasLee 
Thanks for the issue. I think this is expected, the way Blip2 has been trained always prepends the BOS (Beginning of Sentence) token to the input text due to the underlying tokenizer they use. If conditional text is added to the input, the tokenizer should normally take care of adding that token to the beginning of the sentence. 

See this line for reference: https://github.com/huggingface/transformers/blob/e75cb0cb3c5fef887abea6f099252e59a659af9d/src/transformers/models/blip_2/modeling_blip_2.py#L1837 / that manually adds the BOS token if no text is passed

And this line from the original code: https://github.com/salesforce/LAVIS/blob/main/lavis/models/blip2_models/blip2_opt.py#L219 that uses the opt token which adds the bos token to any input by default according to the tokenizer_config file: https://huggingface.co/facebook/opt-350m/raw/main/tokenizer_config.json 

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.
Add FastViT model
Author: atturaioe
Pull request: no
Labels: name: New model

### Model description

FastViT is a hybrid vision transformer that uses structural reparameterization to obtain lower memory access cost and increased capacity, achieving stateof-the-art accuracy-latency trade-off. Highly efficient on multiple compute fabrics: mobile devices and desktop grade GPUs.



### Open source status

- [X] The model implementation is available
- [X] The model weights are available

### Provide useful links for the implementation

Paper: https://arxiv.org/pdf/2303.14189.pdf
Github(code and weights): https://github.com/apple/ml-fastvit
Authors: @anuragranj, @pavank-apple 

Comments:
I can go and try to add it in case

It's from Apple, and the repo. has 1.2K. How do you think about this one? @amyeroberts @NielsRogge ?

I'm pro! @atturaioe, if you want to add, feel free to open a PR and tag one of us or @rafaelpadilla when the PR is ready for review or you have questions about adding it to the library. 

Got it, I'm already on it.
[`resize_embedding`] Introduce `pad_to_multiple_of` and guidance
Author: ArthurZucker
Pull request: yes

# What does this PR do?
Fixes #22312.
After internal discussions, it appears that adding the possibility to pad with `-1`  to `tokenizers` is not really feasible ( nor is it desirable). 
However, what we can do is by default resize the embedding layer to the nearest size that is optimal for the dtype of the model [following this](https://docs.nvidia.com/deeplearning/performance/dl-performance-matrix-multiplication/index.html#requirements-tc).

Motivations:
- the `_get_resized_embeddings` is not exposed, and thus making this automatic can be a big silent win. 
- if properly documented, should not really have issues. 

Cons:
- it is not backward compatible, so some kind of `config.optimise_resize` might be needed? 
- it is hidden and people might not really get why tokenizer.vocab_size will be different than the model's embedding dimension. 

Comments:
_The documentation is not available anymore as the PR was closed or merged._
convertion to hf format of llama2 70b get kill
Author: zixiliuUSC
Pull request: no

### System Info

transformers 4.31.0
RAM: 20G, 8Core
A100-80GB

### Who can help?

_No response_

### Information

- [ ] The official example scripts
- [ ] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

run the checkpoint conversion python script provided by transformers and the program will get kill. I run it successfully to 7B and 13B model. 

### Expected behavior

Convert checkpoint properly. 

Comments:
That is because you do not have enough CPU RAM to do the conversion. It needs 140GB of memory. If you have access to the weights, you can get them on the Hugging Face Hub (as long as you email address for HF matches the one with which you got the weights).

@sgugger The files on https://huggingface.co/meta-llama/Llama-2-70b are not converted. Could you point to the right one on the Hub?

The repo suffixed with hf: https://huggingface.co/meta-llama/Llama-2-70b-hf

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.
RuntimeError: CUDA error: CUBLAS_STATUS_NOT_INITIALIZED when calling `cublasCreate(handle)`
Author: ghost
Pull request: no

### System Info

torch version:  1.12.0+cu113
CUDA:  11.4

`RuntimeError: CUDA error: CUBLAS_STATUS_NOT_INITIALIZED when calling cublasCreate(handle)`

I am getting this error when trying to run using CUDA. Works fine when running in CPU.




### Who can help?

_No response_

### Information

- [ ] The official example scripts
- [ ] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

CODE:

```
import os
import pandas as pd
import time
from transformers import T5Tokenizer, T5ForConditionalGeneration
import torch

tokenizer = T5Tokenizer.from_pretrained("google/flan-t5-xl")
model = T5ForConditionalGeneration.from_pretrained("google/flan-t5-xl", low_cpu_mem_usage=True).to("cuda:0")

def generate(input_text):
  input_ids = tokenizer(input_text, return_tensors="pt").input_ids.to("cuda:0")
  output = model.generate(input_ids, max_length=512)
  return tokenizer.decode(output[0], skip_special_tokens=True)

input_text = 'Something .... Sonethinggg......'

response = generate(input_text)
print(input_text)

```

### Expected behavior

Output

Comments:
This usually means there is something wrong with your setup install.

> 

Like version mismatch?

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.
[idefics] fix vision's `hidden_act`
Author: stas00
Pull request: yes

Thanks to @rwightman's discovery this PR is fixing vision config's `hidden_act` to `gelu`.

It looks like we messed things up when splitting the original config into 3 groups during the porting and inherited `clip`'s default config. Whereas the model used during training was using `gelu` as can be seen here. https://huggingface.co/laion/CLIP-ViT-H-14-laion2B-s32B-b79K/blob/main/config.json

Thank you, @rwightman 

Comments:
_The documentation is not available anymore as the PR was closed or merged._
Is attention_mask supposed to be added to attention_weights? Based on functions docstring mask values are either 0 or 1.  
Author: rpanackal
Pull request: no

https://github.com/huggingface/transformers/blame/476be08c4aa96f8c1cae4200d2677bbe8f12cf80/src/transformers/models/autoformer/modeling_autoformer.py#L619

Comments:
cc @kashif 

thanks @sgugger and @rpanackal let me check

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.

Hey @rpanackal, the attention masks are first processed with:
```python
def _expand_mask(mask: torch.Tensor, dtype: torch.dtype, tgt_len: Optional[int] = None):
    """
    Expands attention_mask from `[bsz, seq_len]` to `[bsz, 1, tgt_seq_len, src_seq_len]`.
    """
    bsz, src_len = mask.size()
    tgt_len = tgt_len if tgt_len is not None else src_len

    expanded_mask = mask[:, None, None, :].expand(bsz, 1, tgt_len, src_len).to(dtype)

    inverted_mask = 1.0 - expanded_mask

    return inverted_mask.masked_fill(inverted_mask.to(torch.bool), torch.finfo(dtype).min)
```
so the value are no longer 0 and 1 😉
MarianMTModel/MBartForConditionalGeneration does not support `device_map='auto'` yet
Author: TranPhu1999
Pull request: no

### System Info

Hi, I'm experiment some Transformer models for Translation task. These model are [vinai-translate-en2vi](https://huggingface.co/vinai/vinai-translate-en2vi), [wmt19-ru-en](https://huggingface.co/facebook/wmt19-ru-en)

In the attempt to optimize the Transformer inference time on single GPU, I tried to follow the instruction on this [document ](https://huggingface.co/docs/transformers/perf_infer_gpu_one#running-mixedint8-models-single-gpu-setup) but stump on this error. I found a similar case [here](https://github.com/huggingface/transformers/issues/22188) where the solution is to Add `accelerate` support for the correspond model. Is it the solution for my problem too? Can anyone share your experience to optimize Transformer inference time? Thanks a lot.


### Who can help?

_No response_

### Information

- [X] The official example scripts
- [ ] My own modified scripts

### Tasks

- [X] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

I use this code like the [example](https://huggingface.co/docs/transformers/perf_infer_gpu_one#running-mixedint8-models-single-gpu-setup)
` from transformers import AutoModelForSeq2SeqLM

model_name = "vinai-translate-en2vi"
model_8bit = AutoModelForSeq2SeqLM.from_pretrained(model_name, device_map="auto", load_in_8bit=True)`

and 

` from transformers import AutoModelForSeq2SeqLM

model_name = "wmt19-ru-en"
model_8bit = AutoModelForSeq2SeqLM.from_pretrained(model_name, device_map="auto", load_in_8bit=True)`

Error
ValueError: MarianMTModel does not support `device_map='auto'` yet.

and 

ValueError: MBartForConditionalGeneration does not support `device_map='auto'` yet.

### Expected behavior

The code in the instruction should be working

Comments:
Hi @TranPhu1999, thanks for raising this issue.

Yes, it seems an equivalent update to the MBart and MarianMT models would need to be added, as the one added [to XGLM](https://github.com/huggingface/transformers/pull/22207/). Would you like to open a PR to add these changes? 

cc @younesbelkada 

Hi @TranPhu1999 
You should be now able to use 8bit models for MBart, you can just do:
```bash
pip install git+https://github.com/huggingface/transformers.git
```
I will work later on adding the same support for Marian as well

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.

Any updates on MarianMT models?

cc @younesbelkada
Error:  index 0 is out of bounds for dimension 0 with size 0
Author: rishabh063
Pull request: no

### System Info

Error:  index 0 is out of bounds for dimension 0 with size 0
Traceback (most recent call last):
    outputs = generator(raw_image, points_per_batch=64)
  File "/opt/conda/lib/python3.10/site-packages/transformers/pipelines/mask_generation.py", line 173, in __call__
    return super().__call__(image, *args, num_workers=num_workers, batch_size=batch_size, **kwargs)
  File "/opt/conda/lib/python3.10/site-packages/transformers/pipelines/base.py", line 1121, in __call__
    return next(
  File "/opt/conda/lib/python3.10/site-packages/transformers/pipelines/pt_utils.py", line 124, in __next__
    item = next(self.iterator)
  File "/opt/conda/lib/python3.10/site-packages/transformers/pipelines/pt_utils.py", line 266, in __next__
    processed = self.infer(next(self.iterator), **self.params)
  File "/opt/conda/lib/python3.10/site-packages/transformers/pipelines/base.py", line 1035, in forward
    model_outputs = self._forward(model_inputs, **forward_params)
  File "/opt/conda/lib/python3.10/site-packages/transformers/pipelines/mask_generation.py", line 242, in _forward
    masks, iou_scores, boxes = self.image_processor.filter_masks(
  File "/opt/conda/lib/python3.10/site-packages/transformers/models/sam/image_processing_sam.py", line 670, in filter_masks
    return self._filter_masks_pt(
  File "/opt/conda/lib/python3.10/site-packages/transformers/models/sam/image_processing_sam.py", line 768, in _filter_masks_pt
    masks = _mask_to_rle_pytorch(masks)
  File "/opt/conda/lib/python3.10/site-packages/transformers/models/sam/image_processing_sam.py", line 1201, in _mask_to_rle_pytorch
    counts += [cur_idxs[0].item()] + btw_idxs.tolist() + [height * width - cur_idxs[-1]]

### Who can help?

_No response_

### Information

- [ ] The official example scripts
- [ ] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

transformer== 4.33.0.dev0


generator = pipeline("mask-generation", model="facebook/sam-vit-huge", device=0)
model = generator.model

outputs = generator(raw_image, points_per_batch=64)


### Expected behavior

Not Fail

Comments:
This happens when the image has nothing to mask , I.e plain image or very Faded images where there is nothing to segment . 

It should Return an empty Array and not an error
Add Mask R-CNN
Author: NielsRogge
Pull request: yes

# What does this PR do?

This PR adds the classic Mask R-CNN framework for object detection and instance segmentation.

To do/to be discussed:

- [ ] where to place utilities like NMS, loss computation, samplers
- [ ] whether to create dummies for torchvision-backed models
- [ ] how to add support for the object detection pipeline - either add `**kwargs` to each `post_process_object_detection` method, or add specific logic for Mask R-CNN inside `object_detection_pipeline.py`

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_22973). All of your documentation changes will be reflected on that endpoint.

@NielsRogge As @sgugger mentions, the PR is still in WIP state. Happy to review once transformers ready :) 

I've updated all docstrings and variable names, PR is ready for another review

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.

Hello @NielsRogge, what is the status of this feature? Thanks in advance

Hi, @NielsRogge looking forward to it. Could you, for now, recommend a robust text detector available here to combine with TrOCR. I would like to see how well the two work with the help of HF🤗.
Add ViTDet
Author: NielsRogge
Pull request: yes

# What does this PR do?

This PR adds part 1 of #25051, namely the ViTDet backbone, introduced in [Exploring Plain Vision Transformer Backbones for Object Detection](https://arxiv.org/abs/2203.16527).

Note that this PR only adds the backbone, hence there are no compatible checkpoints with the backbone-only. Those can only be added once either VitMatte or Mask R-CNN are added, both of which use VitDet as backbone.




Comments:
_The documentation is not available anymore as the PR was closed or merged._

@amyeroberts thanks for your review, I've addressed all the comments
[DINOv2] Add backbone class
Author: NielsRogge
Pull request: yes

# What does this PR do?

This PR:

- adds `Dinov2Backbone` as it was requested at https://github.com/NielsRogge/Transformers-Tutorials/issues/343 and https://github.com/facebookresearch/dinov2/issues/153#issuecomment-1678465000.
- adds DINOv2 to the documentation tests.

Comments:
_The documentation is not available anymore as the PR was closed or merged._

@amyeroberts I've addressed your comments, feel free to approve/merge :)