noamaanMulla-03/transformers-issues huggingface dataset:
how to fine tune BlipForImageTextRetrieval?
Author: ScottishFold007
Pull request: no

### Feature request

how to fine tune BlipForImageTextRetrieval?
Can you borrow some methods from here to achieve this?
https://github.com/salesforce/LAVIS/blob/main/lavis/models/blip_models/blip_retrieval.py

### Motivation

Implement a graphical matching model that, due to the filtering of poor quality pairs of matches

### Your contribution

Not available at the moment

Comments:
cc @younesbelkada 

I'd recommend fine-tuning CLIP if you want to do image-text retrieval using this script: https://github.com/huggingface/transformers/tree/main/examples/pytorch/contrastive-image-text.

Fine-tuning BLIP might be harder as it involves some very specific loss functions.

Thank you for your answer! I tried the fintuning of clip, and it was successful. I use the blip model because I want to use its text matching (binary classification model) to filter out noisy data that is not matched by the text diagram. Because I collect a large number of unlabeled pictures from the Internet, I want to use the blip caption model to tag them, and then filter the invalid image data.

CLIP can also be used for image-text matching, by just encoding the image, encoding the text, and computing a cosine similarity score between the respective embeddings.

> CLIP can also be used for image-text matching, by just encoding the image, encoding the text, and computing a cosine similarity score between the respective embeddings.

In fact, what I want to express is that this graphical text matching classifier is similar to the cross encoder in text matching, which can get the interaction information between the two by splicing the graphical embedding, so the accuracy will be higher than the clip (similar to the bi-encoder in text matching)

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.
WIP porting of lite transformer
Author: raghavanone
Pull request: yes

# What does this PR do?

#19730



Comments:
@NielsRogge  Needs some help how to go about the conversion script and testing. 

The original model is not in pytorch hub, I has only Google Drive link. In the conversion script should I download and convert ? 

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_21235). All of your documentation changes will be reflected on that endpoint.

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.

Hi @raghavanone, do you still want to proceed with this PR? If yes, I'll reopen it :) 

@NielsRogge  Yes, Please it keep it open, I want to wrap up UDOP PR beforing wraping this up .
quantize_dynamic on T5 model results in `AttributeError: 'function' object has no attribute 'dtype'`
Author: gerhean
Pull request: no

### System Info

- `transformers` version: 4.26.1
- Platform: Linux-5.10.147+-x86_64-with-glibc2.29
- Python version: 3.8.10
- Huggingface_hub version: 0.12.1
- PyTorch version (GPU?): 1.13.1+cu116 (False)
- Tensorflow version (GPU?): 2.11.0 (False)
- Flax version (CPU?/GPU?/TPU?): not installed (NA)
- Jax version: not installed
- JaxLib version: not installed
- Using GPU in script?: No
- Using distributed or parallel set-up in script?: No

### Who can help?

@younesbelkada

### Information

- [ ] The official example scripts
- [ ] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [X] My own task or dataset (give details below)

### Reproduction

Paste the following into colab:

```
!pip install transformers sentencepiece accelerate
import torch
from transformers import T5ForConditionalGeneration, T5Tokenizer
model_name = "google/flan-t5-small"
tokenizer = T5Tokenizer.from_pretrained(model_name)
model = T5ForConditionalGeneration.from_pretrained(model_name)
model = torch.quantization.quantize_dynamic(model, {torch.nn.Linear}, dtype=torch.qint8)
input_text = "Answer the following yes/no question by reasoning step-by-step. Can you write a whole Haiku in a single tweet?"
input_ids = tokenizer(input_text, return_tensors="pt").input_ids
output = model.generate(input_ids)
``` 

### Expected behavior

No Error

# Bug
```
AttributeError                            Traceback (most recent call last)
[<ipython-input-3-96b349bbc122>](https://localhost:8080/#) in <module>
      1 input_text = "Answer the following yes/no question by reasoning step-by-step. Can you write a whole Haiku in a single tweet?"
      2 input_ids = tokenizer(input_text, return_tensors="pt").input_ids
----> 3 output = model.generate(input_ids)

10 frames
[/usr/local/lib/python3.8/dist-packages/transformers/models/t5/modeling_t5.py](https://localhost:8080/#) in forward(self, hidden_states)
    314         # See https://github.com/huggingface/transformers/issues/20287
    315         # we also make sure the weights are not in `int8` in case users will force `_keep_in_fp32_modules` to be `None``
--> 316         if hidden_states.dtype != self.wo.weight.dtype and self.wo.weight.dtype != torch.int8:
    317             hidden_states = hidden_states.to(self.wo.weight.dtype)
    318 

AttributeError: 'function' object has no attribute 'dtype'
```

# Fix

When I commented out the lines 316-317 in `transformers/models/t5/modeling_t5.py`, the model runs.

`quantize_dynamic` converts `self.wo.weight` into a bound function which when called, returns the weights. Hence `self.wo.weight` is a function with no attribute `dtype`.

Bug is introduced by #21281

Comments:
Hello @gerhean 
Thanks for the issue! 
https://github.com/huggingface/transformers/pull/21843 should fix your problem, you can already use the fix by checking out on the branch

Hi @gerhean 
Now it's on the `main` branch, if you should be able to use it without any issue!
[GPTJ] Fix gradient checkpointing bug 
Author: krypticmouse
Pull request: yes

This PR fixes a bug that a user can encounter while using generate and models that use gradient_checkpointing.

Fixes Issue #21737 

cc @sgugger @amyeroberts

Comments:
_The documentation is not available anymore as the PR was closed or merged._

@sgugger @amyeroberts As mentioned I did the following:-

> Thanks for your PR! You need to remove the other one below (line 660).

However won't that cause an issue with function declaration and it's corresponding else blocks? I saw other implementations of this fix and they don't remove the block below just add it again above. What do you think?

Hi @krypticmouse - thanks for your question and applying the update @sgugger requested. 

Looking at the diff again, L654 shouldn't be removed. I believe this should resolve the function declaration issue you mentioned. 

You will also need to resolved the conflict as `logger.warning` has been renamed to `logger.warning_once` since you opened your PR.

Is this ok to be merged now?
Make Slack CI reporting stronger
Author: ydshieh
Pull request: yes

# What does this PR do?

Make Slack CI reporting stronger.

The most important change in this PR is to **use a token when we need to grab some GitHub workflow/jobs information** using api call, like
```python
https://api.github.com/repos/huggingface/transformers/actions/runs
```
to get all job links.

**This could avoid reaching the rate limit in CI runs and keep the CI reporting working.**

Such error occurred once on  2023/02/24, see [this run](https://github.com/huggingface/transformers/actions/runs/4258755021/jobs/7424107028). The log shows `Unknown error, could not fetch links. 'jobs'`, but the underlying reason (I strongly believe) is the rate limit is reached, and the api call returns 2 keys `message` and `documentation` without the key `jobs`.

The other changes are just to make things better too.


Comments:
_The documentation is not available anymore as the PR was closed or merged._
ValueError: Please make sure you have `sentencepiece` installed in order to use this tokenizer.
Author: neosanma
Pull request: no

### System Info

- `transformers` version: 4.26.1
- Platform: Linux-5.10.147+-x86_64-with-glibc2.29
- Python version: 3.8.10
- Huggingface_hub version: 0.12.1
- PyTorch version (GPU?): 1.13.1+cu116 (True)
- Tensorflow version (GPU?): 2.11.0 (True)
- Flax version (CPU?/GPU?/TPU?): not installed (NA)
- Jax version: not installed
- JaxLib version: not installed
- Using GPU in script?: <Yes>
- Using distributed or parallel set-up in script?: <No>

### Who can help?

@Narsil @ArthurZucker @younesbelkada 


### Information

- [X] The official example scripts
- [ ] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

run in colab the example in this documentation https://huggingface.co/tasks/translation:

```
!pip install transformers transformers[sentencepiece]
```

```
from transformers import pipeline

model_checkpoint = "Helsinki-NLP/opus-mt-en-fr"
translator = pipeline("translation", model=model_checkpoint)
translator("How are you?")
# [{'translation_text': 'Comment allez-vous ?'}]

```

### Expected behavior

output 

```
# [{'translation_text': 'Comment allez-vous ?'}]
```

as signaled in this documentation https://huggingface.co/tasks/translation

Comments:
Following the steps you indicate to reproduce does not give me any error in Colab. Is it possible you installed sentencepiece after installing transformers and did not restart your environment?

You are right, thanks @sgugger 
(reloading google colab didn't fix my issue but creating a new colab did the trick)
Improve TF weight loading, especially PT crossloading
Author: Rocketknight1
Pull request: yes

Draft PR for now, this will probably break a bunch of stuff until I get it all working!

- [X] Support `from_pt` and `load_weight_prefix` at the same time
- [x] Replace hacky loading code in models that was written to get around this issue
- [x] ~Test and possibly replace code paths that name submodules based on `cls.load_weight_prefix` - I think this is very risky~
- [x] Support `load_weight_prefix` in the `load_sharded` functions as well
- [x] ~Check the `cls._requires_load_weight_prefix` paths and see if there's a better solution~
- [x] Update any affected tests
- [x] Add test for `load_sharded` with `load_weight_prefix`

Classes using `load_weight_prefix` that may need updating:
- [x] BART
- [x] EncoderDeoder
- [x] VisionEncoderDecoder
- [x] RAG
- [x] Blenderbot
- [x] T5
- [x] LED
- [x] BART
- [x] mBART
- [x] Marian
- [x] OPT
- [x] Pegasus
- [x] The cookiecutter template

Comments:
_The documentation is not available anymore as the PR was closed or merged._

This is ready for review now! It's not that big, but it changes a lot of things:

- Loading a PT model in TF with `from_pt=True` now supports `load_weight_prefix`
- Loading a sharded TF checkpoint now supports `load_weight_prefix`
- Sharded TF checkpoints can now be loaded even when not all weights match (this allows model surgery that didn't work with sharded models before!)
- TF `from_pretrained` now supports a `tf_to_pt_weight_rename` kwarg. This should be a callable function which converts TF weight names to PT weight names for that model.
- Composite classes like `TFEncoderDecoder` and `TFVisionEncoderDecoder` have been refactored to use the `tf_to_pt_weight_rename` kwarg, which let me remove all the ingenious workarounds that @ydshieh needed when he added those classes.
- I found a few small issues in other classes when I was testing this PR and fixed them. This is mostly just stuff like removing unused args and disabling TF32 in tests so that the outputs match.
- Add tests for the new features to `test_modeling_tf_common`

cc:
@ArthurZucker because I touched your sharded weight loading code
@ydshieh because I touched your composite model code 
@gante for TF review 
@sgugger as repository overlord
Fix flaky test for log level
Author: sgugger
Pull request: yes

# What does this PR do?

This should fix the flakiness of the log level test. If I'm not wrong, the flakiness came from the fact the log level of Transformers can be changed by other tests (for instance lots of Trainer tests change it) and thus assuming it would be warning at the beginning of the test was wrong. Instead we test depending on the actual log level observed, which should fix the issue.

Comments:
_The documentation is not available anymore as the PR was closed or merged._

Thank you for the fix. There is a remaining one test to fix however, which appears after this PR.


I am able to reproduce the issue with (deterministically)
```bash
python -m pytest tests/trainer tests/utils
```
but not with
```bash
python -m pytest tests/trainer
```
or with
```bash
python -m pytest tests/trainer/test_trainer.py -k test_log_level
```
or with the reversed order
```
python -m pytest tests/utils tests/trainer
```

**With this PR, the `test_log_level` pass with the 1st command mentioned above, but we get** 

```bash
FAILED tests/utils/test_logging.py::HfArgumentParserTest::test_advisory_warnings - AssertionError: '' != 'Testing 1, 2, 3\n'
+ Testing 1, 2, 3
```
However, on this PR again, and with
```bash
python -m pytest  tests/utils
```
or even with the reversed order
```bash
python -m pytest  tests/utils tests/trainer 
```
it pass.

The test in question didn't set any log level, so the log level was still at ERROR when running the tests in the sequence you give. I fixed it by resetting the logger at the beginning of the test.
Troubleshooting AttributeError: 'Seq2SeqTimeSeriesPredictionOutput' object has no attribute 'sequences'
Author: hassanshallal
Pull request: no

Hi, I am reporting this issue as recommended

Context: I am trying to setup/test transformers library, I managed to get the installation and followed the steps in this specific time series transformer tutorial: https://huggingface.co/docs/transformers/model_doc/time_series_transformer

I received an error when I tried to invoke:
`mean_prediction = outputs.sequences.mean(dim=1)`

The error: AttributeError: 'Seq2SeqTimeSeriesPredictionOutput' object has no attribute 'sequences'

by:
`print(outputs.keys())`
I got:
`odict_keys(['loss', 'params', 'encoder_last_hidden_state', 'scale', 'static_features'])`


I searched the documentations but can not find any match.

`import transformers as tfs
print(tfs.__version__)`

4.26.1

I would really appreciate some feedback in regards to this issue.

Thank you


Comments:
Hey. I don't think you are using the correct version of transformers. I can't reproduce this and our documentation tests ensure that this works. Also see here for the `sequence` [output](https://github.com/ArthurZucker/transformers/blob/main/src/transformers/models/time_series_transformer/modeling_time_series_transformer.py#L535) . As you can see, the `generate` method of `TimeSeriesTransformerForPrediction` return this class.

Hi Arthur, thank you for the response. I revisited the tutorial an =d it turned out I missed on a step. It all good now and this issue can be closed.
Copy models back to CPU before merging them after evaluation.
Author: zolastro
Pull request: yes

# What does this PR do?
During evaluation, move the predictions from the `eval_accumulation_steps` to CPU before merging them to avoid an out-of-memory error.

Comments:
_The documentation is not available anymore as the PR was closed or merged._

Thanks for your PR. This is done by the function `nested_numpify` on the line just below.

I see... I still have a problem when evaluating my model (out-of-memory). I will open an issue about it.

I'm closing this pull request, thanks!
group_texts function in language-modeling seems get wrong
Author: zheyuye
Pull request: no

During preprocssing of language-modeling, function `group_texts` as https://github.com/huggingface/transformers/blob/main/examples/pytorch/language-modeling/run_mlm.py#L447-L460

seeems ingore the special tokens like [CLS] and [SEP], and just concatenated them together. An obvious bug would appear that one example may have multiple [CLS] tokens that confuses model and also the [CLS] may not be sure to be placed in the first place.

Comments:
This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.

I am confused about the efficiency of this function. Does anyone test the performance of trained models by this way?

I got my except answer from https://github.com/huggingface/transformers/issues/10737, closing this issue.
prepare for "__floordiv__ is deprecated  and its behavior will change in a future version of pytorch"
Author: ArthurZucker
Pull request: yes

# What does this PR do?
Should adress the `__floordiv__` warnings mentionned in #19934. Divinding torch tensor using `//` is no longer supported and has to be done via `torch.div`. 


Comments:
_The documentation is not available anymore as the PR was closed or merged._
[Pipeline] Add zero shot audio classificatoin pipeline
Author: ArthurZucker
Pull request: yes

# What does this PR do?
Add the `zero_shot_audio_classification_pipeline`  for the `CLAP` models. See #21370 

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_21600). All of your documentation changes will be reflected on that endpoint.

LGTM. I'm confused by the error in tests which doesn't seem linked to this PR.

LGTM !
[OPT] Adds `GPT2TokenizerFast` to the list of tokenizer to use for OPT.
Author: ArthurZucker
Pull request: yes

# What does this PR do?
Adresses the issues with OPT where `use_fast = True` does not use the Fast GPT2 tokenizer. 
A follow PR should add a warning when the Fast tokenizer is not available. 

This should allow people to do : 
```python 
>>> from transformers import AutoTokenizers
>>> tok = AutoTokenizer.from_pretrained("facebook/opt-1.3b", use_fast = True)
>>> tok.is_fast
True
```

Comments:
_The documentation is not available anymore as the PR was closed or merged._

The previous tests were passing but the tokenizer was `slow` were it should have been `fast` 😅 

a gentle ping here, as the m4 group needs to have all official opt models to support fast tokenizers.

Thank you, @ArthurZucker!

Merging ASAP.

cc @ydshieh the failing tests are related to the length of the dictionary of the tokenizer. Spaces are encoded to `222`, which is then passed to the model, while the vocab and embeddings are smaller. I am probably gonna skip it WDYT?

> cc @ydshieh the failing tests are related to the length of the dictionary of the tokenizer. Spaces are encoded to `222`, which is then passed to the model, while the vocab and embeddings are smaller. I am probably gonna skip it WDYT?

Haven't looked this in very detail, but it looks like some bug exists in fast tokenizers. If this is really the case, I am not sure why we want to go ahead to enable the fast tokenizers (by skipping the failing tests).

But if I miss any context and saying any non-sense, please correct me.

The problem is not from the Fast tokenizer (it is a GPT2 tokenizer) but the tiny config test. 
GPT2TokenizerFast is pretty much full proof at this points. I was just wondering if you have a quick fix for this,  ( as I said, the tokenizer's  vocab lenght is `258` while the model tiny  expects no more than `99`, which is causing this issue. This just means that the initialisation of the tiny tokenizer is not correct for this test ( it is using OPT-350m). 


Sir, the PR is based on a quite old commit on `main` (2 weeks ago). Could you rebase (well, better to use merge as you have already done `merge` previously) on `main`, and see if things go better/well.

FYI,  the pipeline testing has some (non-trivial) change(s) in the merged PR #20426.

Hint: it's always nice to rebase (or any way you prefer) to have new commits on `main` in a PR.

we can also re-do the tiny tokenizers if they don't conform with the needs of the CI.

As @ydshieh said it might have been fixed, the problem is that the CI is still kind of stuck/not working ... not really sure if it is only this PR, but otherwise should be good to merge

- pipeline test is fine
- torch test failed with `test_save_load_fast_init_to_base ` which is known to be flaky.

@ArthurZucker Could you check other failing tests - probably they are just flaky ones ..?


I will run them locally 😉 

Good to go! 

cc @stas00  sorry for the long wait

Thank you very much for taking care of this, Arthur!
ConvBERT self-attention throws errors whenever head_ratio is not 2
Author: cmunna0052
Pull request: no

### System Info

Here is what the command says, but multiple responses are incorrect

- `transformers` version: 4.25.1
- Platform: macOS-10.16-x86_64-i386-64bit  (Really 13.1 M1)
- Python version: 3.9.13
- Huggingface_hub version: 0.11.1
- PyTorch version (GPU?): 1.13.1 (False)
- Tensorflow version (GPU?): not installed (NA)
- Flax version (CPU?/GPU?/TPU?): not installed (NA)
- Jax version: not installed (It is installed)
- JaxLib version: not installed (It is installed)
- Using GPU in script?: No
- Using distributed or parallel set-up in script?: No

### Who can help?

@sgugger @arthr

### Information

- [ ] The official example scripts
- [ ] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

```
import jax
from transformers import ConvBertLayer, ConvBertConfig
from transformers.models.convbert.modeling_convbert import ConvBertSelfAttention
test_config = ConvBertConfig()
test_config.attention_probs_dropout_prob = 0.0
test_config.hidden_dropout_prob = 0.0
test_config.head_ratio = 4

torch_conv_attn = ConvBertSelfAttention(test_config)
input_embed = torch.tensor(np.array(jax.random.normal(jax.random.PRNGKey(42),(3,10,test_config.embedding_size))))
torch_conv_attn(input_embed)

#RuntimeError: shape '[3, 10, 768]' is invalid for input of size 11520
```

### Expected behavior

I am testing out ConvBertSelfAttention for comparisons to a similar implementation in JAX.  Head_ratio is allowed to be any integer that divides into num_heads and hidden_size, but I run into errors whenever it is set to anything other than 2. 

I believe the problem stems from the fact that the embedding size is reduced by a factor of head_ratio, but then only multiplied by a factor of 2 in the final concatenation step.  However, this seems to be the same behavior as in the original code https://github.com/yitu-opensource/ConvBert/blob/master/model/modeling.py, so I am not sure how ConvBERT is supposed to handle head_ratio != 2 in the first place.

Comments:
cc @ArthurZucker 

Thanks for reporting. Opening a PR to fix it!
Flax beam search fix
Author: andyehrenberg
Pull request: yes

# What does this PR do?

Makes it so you can pass `decoder_attention_mask` into `model.generate` for flax models when doing beam search. This is helpful for models like Whisper where there may be variable length decoder prefixes across a batch, so you'd have to define a `decoder_attention_mask`.

@sanchit-gandhi
@sgugger 
@frmccann97

Comments:
cc @gante

_The documentation is not available anymore as the PR was closed or merged._