noamaanMulla-03/transformers-issues huggingface dataset:
Update PT/TF weight conversion after #24030
Author: ydshieh
Pull request: yes

# What does this PR do?

Update PT/TF weight conversion due to the change in #24030.

(can do PT/Flax too in the same PR, but request a review first anyway)

### Code snippet to show issues and verify this PR's effect

(Failing for `main + nightly torch`. Pass for `PR + nightly torch` and `main/PR + stable torch`) 
```python
import transformers

from transformers import TFWav2Vec2Model
from tests.models.wav2vec2.test_modeling_tf_wav2vec2 import TFWav2Vec2ModelTest

self = TFWav2Vec2ModelTest()
self.setUp()

model_class = TFWav2Vec2Model
allow_missing_keys = False

config, inputs_dict = self.model_tester.prepare_config_and_inputs_for_common()

pt_model_class_name = model_class.__name__[2:]  # Skip the "TF" at the beginning
pt_model_class = getattr(transformers, pt_model_class_name)

tf_model = model_class(config)
pt_model = pt_model_class(config)

tf_inputs_dict = self._prepare_for_class(inputs_dict, model_class)

# Check we can load pt model in tf and vice-versa with model => model functions
try:
    _tf_model = transformers.load_pytorch_model_in_tf2_model(
        tf_model, pt_model, tf_inputs=tf_inputs_dict, allow_missing_keys=allow_missing_keys
    )
except:
    _tf_model = None

try:
    _pt_model = transformers.load_tf2_model_in_pytorch_model(
        pt_model, tf_model, allow_missing_keys=allow_missing_keys
    )
except:
    _pt_model = None

if _tf_model is None:
    print("_tf_model fails")
else:
    print("_tf_model OK")

if _pt_model is None:
    print("_pt_model fails")
else:
    print("_pt_model OK")
```

Comments:
_The documentation is not available anymore as the PR was closed or merged._
In MlflowCallback, Can not reattach to an existing
Author: ykihong0
Pull request: no

# In Documentation
[Doc](https://huggingface.co/docs/transformers/main_classes/callback#transformers.integrations.MLflowCallback) said we can reattach to an existing run by set **MLFLOW_RUN_ID** environment variable.

# In Code
But [Code](https://github.com/huggingface/transformers/blob/118e9810687dd713b6be07af79e80eeb1d916908/src/transformers/integrations.py#L990) seems making new mlflow run with **args.run_name**.

# ASIS
- code at https://github.com/huggingface/transformers/blob/118e9810687dd713b6be07af79e80eeb1d916908/src/transformers/integrations.py#L990
```
self._ml_flow.start_run(run_name=args.run_name, nested=self._nested_run)
```

# TOBE
```
self._ml_flow.start_run(run_id=self._run_id, nested=self._nested_run)
```

Comments:
Hi @ykihong0, thanks for reporting this issue. 

The integrations are community added and maintained by their authors - Pinging @orieg who I believe added the env var logic in #17130 :)

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.
auto_find_batch_size should say what batch size it is using
Author: p-christ
Pull request: no

### Feature request

When using `auto_find_batch_size=True` in the trainer I believe it identifies the right batch size but then it doesn't log it to the console anywhere?

It would be good if it could log what batch size it is using?

### Motivation

I'd like to know what batch size it is using because then I will know roughly how big a batch can fit in memory - this info would be useful elsewhere.

### Your contribution

N/A

Comments:
cc @muellerzr 

Also, it would be good if auto find batch size also worked for the eval batch size? Otherwise for the eval batch size I still have to guess a few times and hope I don't run out of memory?

Thanks! Solved with https://github.com/huggingface/transformers/pull/23800

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.
Memory leak after repeated inference
Author: huu3301
Pull request: no

### System Info

- `transformers` version: 4.30.2
- Platform: Linux-3.10.0-1160.90.1.el7.x86_64-x86_64-with-glibc2.17
- Python version: 3.8.13
- Huggingface_hub version: 0.14.1
- Safetensors version: 0.3.1
- PyTorch version (GPU?): 2.0.1+cu117 (True)
- Tensorflow version (GPU?): 2.12.0 (True)
- Flax version (CPU?/GPU?/TPU?): not installed (NA)
- Jax version: not installed
- JaxLib version: not installed
- Using GPU in script?: <fill in>
- Using distributed or parallel set-up in script?: <fill in>

### Who can help?

@ArthurZucker @younesbelkada

### Information

- [X] The official example scripts
- [ ] My own modified scripts

### Tasks

- [X] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

When I am using the following script, the memory rises slowly and memory leak happend. I change the transformers version and the torch version, but it doesn't work. When using TFBertModel instead of BertModel, or moving model to CPU, the memory leak still exists.

##########
from transformers import BertTokenizer, BertModel
import torch
import psutil

device = "cuda:0"

model_path = "bert-base-uncased"
model = BertModel.from_pretrained(model_path)
tokenizer = BertTokenizer.from_pretrained(model_path)
model.to(device)
model.eval()

query = "Replace me by any text you'd like."

for i in range(10000):
    with torch.no_grad():
        encoded_input = tokenizer(query, return_tensors='pt').to(device)
        a = psutil.Process(os.getpid()).memory_info().rss / 1024  # memory before
        output = model(**encoded_input)
        b = psutil.Process(os.getpid()).memory_info().rss / 1024  # memory after
        if b != a and i > 0:
            print(i)
            print("b-a=%s kb" % (b - a))

##########
result:
62
b-a=80.0 kb
6195
b-a=8.0 kb

### Expected behavior

Memory is almost stable.

Comments:
Hi @huu3301 

Please format the code snippet properly (with proper indent too). See for example

<img width="332" alt="Screenshot 2023-06-27 111112" src="https://github.com/huggingface/transformers/assets/2521628/ec2852fb-695a-456b-b09f-8f99ef0bdd30">

Regarding your question about leak:

Do you only get iteration `62` and `6195` with `b != a`? In this case, there is no memory leak: it's just python process gets to use a bit more memory to perform something under the hood.




@ydshieh 
The indent disappeared when I submitted the issue. 
I increased the number of iteration to 1000000 , and the memory increment was less than 50kb. There was no memory leak. Thank you for answering.
[`gpt2-int8`] Add gpt2-xl int8 test
Author: younesbelkada
Pull request: yes

# What does this PR do?

Addresses: https://github.com/huggingface/transformers/pull/24504#discussion_r1243813429 

Currently the test is failing because of the following reason (that I need to explore and as discussed offline @sgugger):
1- If a buffer is defined as `persistent=False` in the modeling file
2- and not present in the state_dict
The dispatch_model seems to fail as I face device mismatch issue. I think a potential fix needs to be addressed on accelerate 
Facing the same issue for blip2 int8 tests, as the `cos_cached` and `sin_cached` are defined as buffers with persistent=False and not present in the state dict. 

Seems that the issue does not happen before 5791d949ff93733c102461ba89c8310745a3fa79 on accelerate

cc @sgugger 

EDIT: https://github.com/huggingface/accelerate/pull/1660 should fix the issue I am facing

Comments:
_The documentation is not available anymore as the PR was closed or merged._

Can confirm all the tests pass now (with the accelerate PR mentioned above)
Use new parametrization based weight norm if available
Author: ezyang
Pull request: yes

# What does this PR do?

In https://github.com/pytorch/pytorch/pull/103001 I introduce a new
parametrization based version of `weight_norm`.  One big benefit of
the new API is that the resulting model is deepcopy'able; today, you can't
deepcopy Wav2Vec2 models.

Since the new API isn't even in PyTorch main yet, I'd like to feature
gate it here, so that it gets used whenever PyTorch is recent enough to
support it.  It would be a big help for me if you could take this change
earlier rather than later; otherwise I will have to patch transformers
in our own CI to get our benchmark harness working on Wav2Vec2.

Signed-off-by: Edward Z. Yang <ezyang@meta.com>

## Before submitting
- [ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [X] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#start-contributing-pull-requests),
      Pull Request section?
- [ ] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [ ] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/main/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/main/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests?


## Who can review?

cc @sanchit-gandhi


Comments:
_The documentation is not available anymore as the PR was closed or merged._

PyTorch side PR has landed!

@ezyang 

With nightly pytorch, we get 

> AttributeError: encoder.pos_conv_embed.conv.weight_v not found in PyTorch model

when trying to load a pytorch model into a TF model.

The TF model is looking for `encoder.pos_conv_embed.conv.weight_v` but we now have `encoder.pos_conv_embed.conv.parametrizations.weight.original0`. (This is from our `(TF)Wav2Vec2Model` model class).

**Question**: In your PR https://github.com/pytorch/pytorch/pull/103001, is this part

`def _weight_norm_compat_hook()`

that deals with the backward compatibility? If so, we will copy it :-) 

Yep. The change here is not FC so the ingester needs updating.
Fix processor __init__ bug if image processor undefined
Author: amyeroberts
Pull request: yes

# What does this PR do?

Fixes a bug which occurs if a processor is initialized with `image_processor=None`. An exception should be raised, saying an image processor should be defined, but at the moment fails because the code references `feature_extractor` which is not defined if it's not in the kwargs. 

## Before submitting
- [ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [x] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#start-contributing-pull-requests),
      Pull Request section?
- [ ] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [ ] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/main/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/main/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests?


Comments:
_The documentation is not available anymore as the PR was closed or merged._
[`InstructBlip`] Add instruct blip int8 test
Author: younesbelkada
Pull request: yes

# What does this PR do?

Addresses: https://github.com/huggingface/transformers/pull/24490#discussion_r1242098216

Also fixes an inconsistency with Blip / Blip2 processors in order for users to be able to call `.to()` with both device and dtype argument to cast the input in half precision if necessary. Happy to move that in another PR if needed

cc @sgugger @ydshieh 


Comments:
_The documentation is not available anymore as the PR was closed or merged._
Add SwinIR for Image Super Resolution
Author: sriagastya
Pull request: no
Labels: name: New model

### Model description

SwinIR: Image Restoration Using Swin Transformer

- This paper presents an Image Super Resolution / Image Restoration model inspired by the SwinTransformer architecture.
- It demonstrates superior performance on various vision tasks including classical/lightweight Image Super Resolution, Image Denoising, and JPEG compression artifact reduction

This issue is primarily focused on the Image Super Resolution model only. I would love to see this model on HuggingFace.

## Contribution: I would love to work on this!

I am new to HuggingFace and open-source in general. I am open to comments/suggestions/feedback.

### Open source status

- [X] The model implementation is available
- [X] The model weights are available

### Provide useful links for the implementation

Paper Link: https://arxiv.org/pdf/2108.10257v1.pdf
Implementation Link: https://github.com/JingyunLiang/SwinIR
Weights Link: https://github.com/JingyunLiang/SwinIR/releases/tag/v0.0

Comments:
Just wanna note that Swin2SR is already integrated, which improves upon SwinIR: https://huggingface.co/docs/transformers/main/model_doc/swin2sr

Ohh okay, thank you, I'll close this issue then.

Hi @NielsRogge thanks for adding this model to HF.
Do you have in plans to add training process too?
Update `EncodecIntegrationTest`
Author: ydshieh
Pull request: yes

# What does this PR do?

Some test fail since the addition of this model. This PR just updates the expect output values.

Comments:
_The documentation is not available anymore as the PR was closed or merged._

> Thanks @ydshieh - just to confirm, these updated values come from running the model on CUDA? (versus the original values, which were obtained on CPU)

Yes, it's on GPU. More specifically, on our CI runner's T4 GPU.

Perfect, thanks for updating the tests for CUDA! Also cc @ArthurZucker as a heads-up since we discussed this offline previously
Add LaVIN model
Author: tensorpro
Pull request: no
Labels: name: New model

### Model description

 LaVIN is a vision-language instructed model that is affordable to train (it was trained in a few hours on 8 A100 GPUs) with good performance on ScienceQA.

I'd like to add LaVIN to HF transformers.

### Open source status

- [X] The model implementation is available
- [X] The model weights are available

### Provide useful links for the implementation

The paper [Cheap and Quick: Efficient Vision-Language Instruction Tuning for Large Language Models](https://arxiv.org/pdf/2305.15023.pdf) is by [Gen Luo](https://luogen1996.github.io/),  [Yiyi Zhou](https://github.com/huggingface/transformers/issues/new?assignees=&labels=New+model&projects=&template=new-model-addition.yml),  [Tianhe Ren](https://rentainhe.github.io/),  [Shengxin Chen](https://github.com/huggingface/transformers/issues/new?assignees=&labels=New+model&projects=&template=new-model-addition.yml),  [Xiaoshuai Sun](https://sites.google.com/view/xssun),  and [Rongrong Ji](https://mac.xmu.edu.cn/rrji/)

@luogen1996 has made the code and model weights available at [github.com/luogen1996/LaVIN](https://github.com/luogen1996/LaVIN).

The weights for the following models are available at the following links:

### ScienceQA
| Model     |  Weights    |      Time | Memory | #Params |  Acc |          Weights | 
|-----------|----------:|----------:|-------:|--------:|-----:|-----------------:|
| LaVIN-7B  | LLaMA | 1.4 hours |  33.9G |    3.8M | 89.37 | [google drive](https://drive.google.com/file/d/10X2qCBYrLH1grZOHwHRMXLUoz-S6MSgV/view?usp=share_link) |
| LaVIN-7B  | Vicuna | 1.4 hours |  33.9G |    3.8M | 89.41 | [google drive](https://drive.google.com/file/d/1nuMxeiWlnJKxDybCshg8pVGSvLc5dZy8/view?usp=share_link) |
| LaVIN-13B | LLaMA |   2 hours |  55.9G |    5.4M | 90.54 | [google drive](https://drive.google.com/file/d/1LkKUY54spZkkeXrR7BDmU-xmK9YadcKM/view?usp=share_link) |

### Multimodal ChatBot
| Model     |Weights    |      Time | Memory | #Params | Acc |          Weights | 
|-----------|----------:|---------:|-------:|--------:|----:|-----------------:|
| LaVIN-13B | LLaMA | 75 hours |  55.9G |    5.4M |   - | [google drive](https://drive.google.com/file/d/1rHQNSaiGzFHYGgsamtySPYnd5AW4OE9j/view?usp=share_link)| 


Comments:
Hi @amyeroberts, I don't think anyone is working on this anymore. If this adds any value to hf I'll start working on it.
Fixed OwlViTModel inplace operations
Author: pasqualedem
Pull request: yes

# What does this PR do?

This PR replaces the "/=" operator in OwlViTModel which causes an error in the backward pass with the non-inplace version.

Fixes #24525 

#24525 

## Who can review?

@ydshieh 


Comments:
_The documentation is not available anymore as the PR was closed or merged._

@pasqualedem Could you apply the commit suggestions, then we can merge 🙏 Thanks.

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_24529). All of your documentation changes will be reflected on that endpoint.
Update old existing feature extractor references
Author: amyeroberts
Pull request: yes

# What does this PR do?

Updates a bunch of old references to feature extractors for vision models. 

Most of the code isn't public facing in e.g. docs, but is often copied when new models are added. 


## Before submitting
- [x] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [x] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#start-contributing-pull-requests),
      Pull Request section?
- [ ] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [ ] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/main/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/main/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests?


Comments:
_The documentation is not available anymore as the PR was closed or merged._

@ydshieh Thanks for pointing out the extra places I missed. I've updated + other vision files needing the same update.
Fix typing annotations for FSDP and DeepSpeed in TrainingArguments
Author: mryab
Pull request: yes

# What does this PR do?
According to the the docstrings and to the code, options `fsdp_config` and `deepspeed` of `TrainingArguments` accept dictionaries with configs for FSDP and DeepSpeed correspondingly. However, the typing annotations for both of them mention only `str` as valid arguments, which makes typing checkers fail when trying to pass a dictionary for these options.

This PR fixes the problem by making these options accept `Dict` as well, also fixing a couple of minor typos in their descriptions.

## Before submitting
- [x] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [x] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#start-contributing-pull-requests),
      Pull Request section?
- [ ] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [ ] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/main/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/main/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests?

@sgugger 

Comments:
_The documentation is not available anymore as the PR was closed or merged._

 argument --deepspeed: invalid Dict value: './ds_config_zero3.json' i am face this after merge, i have change deepspeed to Optional[str] and it worked 

Ok, let's revert then as it's a purely cosmetic change.
Fix Typo
Author: tony9402
Pull request: yes

# What does this PR do?

Fixed wrong annotation

- `(seq_len, BS, model_dim) -> (BS, seq_len, model_dim)` -> `(BS, seq_len, model_dim) -> (seq_len, BS, model_dim)`

## Before submitting
- [x] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [x] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#start-contributing-pull-requests),
      Pull Request section?

Comments:
Can you check if this is correct @ArthurZucker ?

_The documentation is not available anymore as the PR was closed or merged._
Fix annotations
Author: tony9402
Pull request: yes

# What does this PR do?

I found wrong annotations in `MBart`, `Pegasus` model.

Fixed wrong annotation

- (seq_len, batch, embed_dim) -> (batch, seq_len, embed_dim)


## Before submitting
- [x] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [x] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#start-contributing-pull-requests),
      Pull Request section?

Comments:
_The documentation is not available anymore as the PR was closed or merged._
Fix `SpeechT5` doctests
Author: ydshieh
Pull request: yes

# What does this PR do?

PR #24434 changes `np.random.uniform(0, 1)` to `torch.rand([])`. In the forward method of `SpeechT5ForSpeechToSpeech` and `SpeechT5ForTextToSpeech`, the line `dropout_probability = torch.rand([])` is executed no matter if we are in training/inference mode. So the new change in #24434 will change the random sequences even we set a seed in the beginning of `generate`, and we get different outputs now.

Hence this PR updates the expected values for doctest.

**However, I believe we should only call `dropout_probability = torch.rand([])` under the condition of being in training mode.**
WDYT?

Comments:
_The documentation is not available anymore as the PR was closed or merged._

Yes, that would be a better fix I agree.

Thanks, I will follow the same fix in #24486 instead.
Add Musicgen
Author: sanchit-gandhi
Pull request: yes

# What does this PR do?

Adds the musicgen model by fairseq to transformers

This model is made of three components:
1. T5Encoder (which import as `AutoModelForTextEncoding`)
2. MusicgenDecoder (which we copy as much as possible from [`modeling_bart.py`](https://github.com/huggingface/transformers/blob/1e9da2b0a6ef964c2cf72dd715dbee991a3f49fa/src/transformers/models/bart/modeling_bart.py#L142))
3. Encodec (which we import as `AutoModel`)


Comments:
_The documentation is not available anymore as the PR was closed or merged._

Would be great to hear your thoughts on the design here @patrickvonplaten (adding the tests otherwise now)

TODO:
- [x] convert m/l checkpoints
- [x] handle padded tokens from encodec (in delay pattern mask, then again when we decode)
- [x] fast tests
- [x] integration tests
- [x] add method for unconditional generation (no need to use processor to get input ids)
- [x] finish docs / docstrings

This is ready for review (fyi @ArthurZucker / @patrickvonplaten) - kindly requesting review from @sgugger!