noamaanMulla-03/transformers-issues huggingface dataset:
Add OWLv2
Author: NielsRogge
Pull request: yes

# What does this PR do?

This PR adds the OWLv2 checkpoints, and investigates why results weren't the same as the original Colab for v1 as pointed out in #21206. After investigation, it turns out that 1) the model returns the exact same logits as the original implementation on the same input data, but 2) the image preprocessing wasn't done in exactly the same way as the original Scenic repo, which involves padding, causing a difference. This PR fixes that by introducing a new `Owlv2ImageProcessor` which people can use to get equivalent results. 

Fixes #26315 #21206

Question:
- the new OWLv2 models have an extra objectness head, which returns `objectness_logits` => we can either add an entirely new `Owlv2ForObjectDetection` which copies 99% of `OwlViTForObjectDetection`, or do it like it's currently implemented (by adding a `config.add_objectness_head` attribute). Happy to read your opinions!

To do:

- [ ] update layer norm eps of LayerNorms, which is 1e-5 in PyTorch but 1e-6 in Flax by default

Comments:
I would go for simply adding `config.add_objectness_head` in the existing `OwlViT` model file(s).

I think current implementation is better than creating another class for OWLv2. One suggestion is to use more descriptive flag for OWLv2 something like `owlvit_v2=True` which enable the objectness head

Sorry for the delay I am reviewing now 😉 

Thanks for the reviewing, opened a new PR that adds the model as a new standalone model.

Hi, @NielsRogge，thanks for your great work.
I have a simple question:  
How to convert pytorch model into jax model? Do you have any available tools or scripts?
Thank you very much
Meta-Transformer
Author: rajveer43
Pull request: no
Labels: name: New model

### Model description

Introducing a groundbreaking research paper that explores the potential of unified multimodal learning, revolutionizing the way we process and integrate diverse data types such as text, images, audio, and more! 🌟
[meta-transformer](https://arxiv.org/pdf/2307.10802.pdf)

### Open source status

- [X] The model implementation is available
- [x] The model weights are available

### Provide useful links for the implementation

https://github.com/invictus717/MetaTransformer

Comments:
I am interested in adding this model.
Hi @ArthurZucker @amyeroberts, do you think this will be a valid addition?  

For starters, if there are no online checkpoints then there's nothing much to be added 😓 
EDIT: seems like they are available!
Appart from having a very complicated codebase with submodules here and there might be a good addition yeah! WDYT @amyeroberts ?

IMO putting this on the hub will be a lot easier and good way to test the popularity! 

@ArthurZucker @susnato, Thank you for considering request.

@ArthurZucker Agreed - would be cool to add and adding to the hub first would be my suggestion! 

> @ArthurZucker Agreed - would be cool to add and adding to the hub first would be my suggestion!

Its Paper is also available here: https://arxiv.org/abs/2307.10802

Thanks @ArthurZucker and @amyeroberts for your views, I will quickly get on to create a demo.

**EDIT : As of now, they have published the weights for the shared encoder but most of the model-head weights(specific to different modalities) are not published yet. Waiting for them to get published, before creating the demo.**

@ArthurZucker @amyeroberts @susnato there has been some update to the repo check here: https://github.com/invictus717/MetaTransformer
Fix source_prefix default value in run_summarization.py
Author: jheitmann
Pull request: no

### System Info

- `transformers` version: 4.35.0.dev0
- Platform: macOS-13.4-arm64-arm-64bit
- Python version: 3.10.10
- Huggingface_hub version: 0.17.3
- Safetensors version: 0.4.0
- Accelerate version: 0.23.0
- Accelerate config:    not found
- PyTorch version (GPU?): 2.1.0 (False)
- Tensorflow version (GPU?): not installed (NA)
- Flax version (CPU?/GPU?/TPU?): not installed (NA)
- Jax version: not installed
- JaxLib version: not installed
- Using GPU in script?: no
- Using distributed or parallel set-up in script?: no

### Who can help?

@ArthurZucker @younes

### Information

- [X] The official example scripts
- [ ] My own modified scripts

### Tasks

- [X] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

Run the following command to reproduce the behavior:

```
python examples/pytorch/summarization/run_summarization.py \
    --model_name_or_path t5-small \
    --do_train \
    --do_eval \
    --dataset_name cnn_dailymail \
    --dataset_config "3.0.0" \
    --output_dir /tmp/tst-summarization \
    --per_device_train_batch_size=4 \
    --per_device_eval_batch_size=4 \
    --overwrite_output_dir \
    --predict_with_generate
```

### Expected behavior

When a user fine-tunes a T5 model without explicitly setting the `source_prefix`, the warning message prompting users to set a `source_prefix` is not displayed. This is because the `source_prefix` is set to its default value of `""` in this PyTorch implementation of the `run_summarization.py` script, which is not equivalent to `None`. This behavior is inconsistent with other implementations, where the default value of `source_prefix` is `None`, and the warning is displayed when not provided by the user.

Comments:
[JAX] Replace uses of `jnp.array` in types with `jnp.ndarray`.
Author: hvaara
Pull request: yes

`jnp.array` is a function, not a type:
https://jax.readthedocs.io/en/latest/_autosummary/jax.numpy.array.html so it never makes sense to use `jnp.array` in a type annotation. Presumably the intent was to write `jnp.ndarray` aka `jax.Array`.

For a similar PR in `diffusers`, please see https://github.com/huggingface/diffusers/pull/4719.

## Before submitting
- [x] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [x] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#start-contributing-pull-requests),
      Pull Request section?
- [ ] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [x] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/main/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/main/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests?


## Who can review?

Anyone in the community is free to review the PR once the tests have passed. Feel free to tag
members/contributors who may be interested in your PR.

cc @pcuenca @patrickvonplaten

<!-- Your PR will be replied to more quickly if you can figure out the right person to tag with @

 If you know how to use git blame, that is the easiest way, otherwise, here is a rough guide of **who to tag**.
 Please tag fewer than 3 people.

Models:

- text models: @ArthurZucker and @younesbelkada
- vision models: @amyeroberts
- speech models: @sanchit-gandhi
- graph models: @clefourrier

Library:

- flax: @sanchit-gandhi
- generate: @gante
- pipelines: @Narsil
- tensorflow: @gante and @Rocketknight1
- tokenizers: @ArthurZucker
- trainer: @muellerzr and @pacman100

Integrations:

- deepspeed: HF Trainer/Accelerate: @pacman100
- ray/raytune: @richardliaw, @amogkam
- Big Model Inference: @SunMarc
- quantization (bitsandbytes, autogpt): @SunMarc and @younesbelkada

Documentation: @stevhliu and @MKhalusova

HF projects:

- accelerate: [different repo](https://github.com/huggingface/accelerate)
- datasets: [different repo](https://github.com/huggingface/datasets)
- diffusers: [different repo](https://github.com/huggingface/diffusers)
- rust tokenizers: [different repo](https://github.com/huggingface/tokenizers)

Maintained examples (not research project or legacy):

- Flax: @sanchit-gandhi
- PyTorch: See Models above and tag the person corresponding to the modality of the example.
- TensorFlow: @Rocketknight1

 -->


Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_26703). All of your documentation changes will be reflected on that endpoint.
Having a `to_grayscale` function 
Author: rafaelpadilla
Pull request: no

### Feature request

It would be useful to have a general `to_grayscale` function into our `image_transforms.py` so it can be called by any model in their `preprocess()`.

### Motivation

Our image processing got more robust and can now accept 1-channel and 3-channel images.
Seeing that, new models (like [SuperModel](https://github.com/huggingface/transformers/pull/25786) and potentially new ones) require grayscale images as inputs. Thus, it would be useful to have a "standard" `to_grayscale` function in `image_transforms.py` - similarly as our `convert_to_rgb(...)`.

Indeed each model could implement their own version of `to_grayscale`, but I find it useful to have an out-of-the-box function that can be easily called, avoiding redundant implementations of it.

### Your contribution

I can make a PR for that if you also think it is useful.
@amyeroberts @ArthurZucker 


Comments:
hello @rafaelpadilla can i work on this issue


Hi @Neel-07,

Sure, that'd be awesome. :) 
But let's just wait until one of our core maintainers (@amyeroberts and @ArthurZucker) approves this request.

okay sure @rafaelpadilla 


I don't have a strong opinion on this, think that when it comes to vision and converting to grayscale you might have a lot of heuristic that might be different so would just go with the flow. If next model uses the same we can move it otherwise I don't really see a point for now! 

Alright! :) 
So, I will close this issue for now.
If a new model also needs grayscale conversion, then we can reopen this issue and work on this.
SageMaker notebook: Too many requests for URL
Author: austinmw
Pull request: no

### System Info

I'm having an issue downloading models in SageMaker Notebooks. It seems to be persisting for 2 days now.

```
  File "/home/ec2-user/anaconda3/envs/gatwick/lib/python3.10/site-packages/groundingdino/util/get_tokenlizer.py", line 19, in get_tokenlizer
    tokenizer = AutoTokenizer.from_pretrained(text_encoder_type)
  File "/home/ec2-user/anaconda3/envs/gatwick/lib/python3.10/site-packages/transformers/models/auto/tokenization_auto.py", line 692, in from_pretrained
    config = AutoConfig.from_pretrained(
  File "/home/ec2-user/anaconda3/envs/gatwick/lib/python3.10/site-packages/transformers/models/auto/configuration_auto.py", line 1007, in from_pretrained
    config_dict, unused_kwargs = PretrainedConfig.get_config_dict(pretrained_model_name_or_path, **kwargs)
  File "/home/ec2-user/anaconda3/envs/gatwick/lib/python3.10/site-packages/transformers/configuration_utils.py", line 620, in get_config_dict
    config_dict, kwargs = cls._get_config_dict(pretrained_model_name_or_path, **kwargs)
  File "/home/ec2-user/anaconda3/envs/gatwick/lib/python3.10/site-packages/transformers/configuration_utils.py", line 675, in _get_config_dict
    resolved_config_file = cached_file(
  File "/home/ec2-user/anaconda3/envs/gatwick/lib/python3.10/site-packages/transformers/utils/hub.py", line 491, in cached_file
    raise EnvironmentError(f"There was a specific connection error when trying to load {path_or_repo_id}:\n{err}")
OSError: There was a specific connection error when trying to load bert-base-uncased:
429 Client Error: Too Many Requests for url: https://huggingface.co/bert-base-uncased/resolve/main/config.json
```

### Who can help?

Not sure. Multiple models give same error.

### Information

- [X] The official example scripts
- [ ] My own modified scripts

### Tasks

- [X] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

The following errors when I run in a SageMaker notebook, but does not error when I run on my local laptop:

```python
from transformers import AutoTokenizer

tokenizer = AutoTokenizer.from_pretrained("bert-base-uncased")
```

It previously worked in this SageMaker notebook a couple days ago, and nothing has been modified since then.

### Expected behavior

Download model

Comments:
Hi @austinmw, thanks for raising this issue! 

Is there a specific notebook you could link to where you're experiencing this problem? 

cc @philschmid  

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.
Incorrect decode from GPTNeox tokenizer.
Author: tbenthompson
Pull request: no

### System Info

- `transformers` version: 4.32.1
- Platform: Linux-5.15.0-52-generic-x86_64-with-glibc2.31
- Python version: 3.10.11
- Huggingface_hub version: 0.15.1
- Safetensors version: 0.3.1
- Accelerate version: 0.20.3
- Accelerate config:    not found
- PyTorch version (GPU?): 2.0.1+cu118 (True)
- Tensorflow version (GPU?): not installed (NA)
- Flax version (CPU?/GPU?/TPU?): not installed (NA)
- Jax version: not installed
- JaxLib version: not installed
- Using GPU in script?: No
- Using distributed or parallel set-up in script?: No

### Reproduction

When running this code:

```
import transformers

tokenizer = transformers.AutoTokenizer.from_pretrained("EleutherAI/pythia-12b-deduped")
text = " there is this someone 'you' who has the ability of 'sensing things"
ids = tokenizer.encode(text)
print(repr(tokenizer.decode(ids)))
print(repr("".join(tokenizer.batch_decode(ids))))
```

I get the output:

```
" there is this someone 'you' who has the ability of'sensing things"
" there is this someone 'you' who has the ability of 'sensing things"
```

### Expected behavior

The first string produced by `tokenizer.decode` is an incorrect decoding. The second string from `batch_decode` is correct. The first string is missing a space before `'sensing`.

Comments:
cc @ArthurZucker 

You should try with `tokenizer.decode(ids, clean_up_tokenization_spaces = False)`:
```python 
>>> import transformers

>>> tokenizer = transformers.AutoTokenizer.from_pretrained("EleutherAI/pythia-12b-deduped")
>>> text = " there is this someone 'you' who has the ability of 'sensing things"
>>> ids = tokenizer.encode(text)
>>> print(repr(tokenizer.decode(ids,  clean_up_tokenization_spaces = False)))
" there is this someone 'you' who has the ability of 'sensing things"
```

Ok, thanks! For my own understanding, why is the default `clean_up_tokenization_spaces = True`? Without that setting, `decode` and `encode` are much closer to being the inverse of each other. Intuitively, that seems like it should be the default goal of `decode`

Good question, has been that way for a long time, I think this was to reflect some of our original tokenizers that were adding spaces. I'll check if we can safely remove this / set to `False` by default! 

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.
reduce peak memory usage during training via pytorch caching allocator "preallocation" strategy
Author: tmm1
Pull request: no

### Feature request

In the PyTorch tuning guide, there's a section on preallocation as a technique to reduce memory usage when dealing with variable length inputs.

https://pytorch.org/tutorials/recipes/recipes/tuning_guide.html#preallocate-memory-in-case-of-variable-input-length

<img width="544" alt="image" src="https://github.com/huggingface/transformers/assets/2567/d4200505-ae08-46f4-a245-f13b4b9b9646">

It would be great to implement this in transformers.Trainer as an option

### Motivation

In investigating memory usage for training, I discovered pytorch memory fragmentation can be a big problem.

One workaround is to pad inputs to the same length, i.e. https://github.com/OpenAccess-AI-Collective/axolotl/pull/498#issuecomment-1696539038

However it would be better to not have to use padding at all just to control memory usage. The preallocation strategy is a better approach, and should only require 2-3 passes with random inputs.

### Your contribution

I can work on a PR for this if the idea is acceptable. It would also be helpful to decide on the name of the configuration option.

Comments:
It looks like we don't really to change anything in trainer, but just make a forward/backward pass with a first (max sized) input batch, then zero out the gradients. Everything could be done outside the trainer by the user.

 

> Everything could be done outside the trainer by the user.

This may be true, but also the Trainer implementation could be using all sorts of extra features like apex/amp/loss_context_manager/grad_scaling etc. These could all create extra tensors that are based on the input shape, so I think it would be best to build this into the Trainer itself.

The `trainer` is a file of 4k lines - we really want to keep it not to continue to grow 🙏 

I tried to do this outside the trainer but it is not effective.

After three steps of forward+backward, the torch cache size is only 2.4GB.

But when I measure it after the first trainer step, its 3-4x higher.

So the trainer is doing a lot more work. As you said there are 4k lines, so there is a lot of complexity. It's not easy to replicate that outside the trainer for every user.

If there isn't interest in this feature, I guess most users should consider padding inputs. In my test with llama2-7b + lora, peak memory usage could be 25% higher with variable length inputs.

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.
Accelerate does not work with MaskGenerationPipelone
Author: MrinalTyagi
Pull request: no

### System Info

- `transformers` version: 4.29.2
- Platform: Linux-5.15.0-46-generic-x86_64-with-glibc2.17
- Python version: 3.8.17
- Huggingface_hub version: 0.15.1
- Safetensors version: not installed
- PyTorch version (GPU?): 2.0.1 (True)
- Tensorflow version (GPU?): not installed (NA)
- Flax version (CPU?/GPU?/TPU?): not installed (NA)
- Jax version: not installed
- JaxLib version: not installed
- Using GPU in script?: yes
- Using distributed or parallel set-up in script?: yes

Using device_map with mask-generation pipeline of sam results in error. 

@Narsil 

### Who can help?

_No response_

### Information

- [X] The official example scripts
- [ ] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [X] My own task or dataset (give details below)

### Reproduction

Run example SAM pipeline available at https://huggingface.co/docs/transformers/main/model_doc/sam with device_map="auto"

### Expected behavior

It should run with device_map="auto" otherwise if not possible should provide a suitable error. 

Comments:
According to the contribution guideline, could you share a complete reproducer as well as a traceback of the error you are getting? 

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.
MT5/MT0 cannot finetune with lora in 8bit mode
Author: Victordongy
Pull request: no

### System Info

transformers version: 4.24.0
Platform: Linux-5.4.0-150-generic-x86_64-with-glibc2.29
Python version: 3.8.10
Huggingface_hub version: 0.16.4
Safetensors version: 0.3.1
Accelerate version: 0.20.3
Accelerate config: not found
PyTorch version (GPU?): 2.0.0+cu117 (True)
Tensorflow version (GPU?): 2.12.0 (False)
Flax version (CPU?/GPU?/TPU?): not installed (NA)
Jax version: not installed
JaxLib version: not installed
Using GPU in script?: Yes
Using distributed or parallel set-up in script?: No

### Who can help?

@younesbelkada @ArthurZucker

### Information

- [X] The official example scripts
- [X] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [X] My own task or dataset (give details below)

### Reproduction

```
model = AutoModelForSeq2SeqLM.from_pretrained(
            "google/mt5-large"
            load_in_8bit=True,
            device_map='auto',
            **kwargs
            )

model = prepare_model_for_int8_training(model)

 config = LoraConfig(
          r=8,
          lora_alpha=8,
          target_modules=["q","v"],
          lora_dropout=0.5,
          bias="none",
          task_type=SEQ_2_SEQ_LM,
      )
model = get_peft_model(model, config)

trainer.train()
```
and I got the error as 
```
Traceback (most recent call last):
  ...
  File "/temp/models/t5.py", line 72, in forward
    query_states = shape(self.q(hidden_states))  # (batch_size, n_heads, seq_length, dim_per_head)
  File "/temp/dg_venv/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1501, in _call_impl
    return forward_call(*args, **kwargs)
  File "/temp/dg_venv/lib/python3.8/site-packages/peft/tuners/lora.py", line 817, in forward
    result = F.linear(x, transpose(self.weight, self.fan_in_fan_out), bias=self.bias)
RuntimeError: expected scalar type Float but found Char
```

### Expected behavior

Model should be trained successfully. 

Comments:
hi @Victordongy 
Thanks for the issue can you try to re-run LoRA training with the latest peft and transformers?
```bash
pip install -U transformers peft
```

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.