noamaanMulla-03/transformers-issues huggingface dataset:
Unstable generation results when using Top-p decoding
Author: bilalghanem
Pull request: no

### System Info

I trained a T5-base model for a translation task. I use Top-p decoding strategy to generate text. 

Something is weird in the model; every time I ask the model to generate text for the given same input, it generates a different text. When I fix the random seed, the model starts to generate the exact text every time.

 My question is, why the model generates different texts for the same input if the random seed is not fixed? Top-P decoding strategy has no randomness.

```
y = data['target_ids'].to(device, dtype=torch.long)
ids = data['source_ids'].to(device, dtype=torch.long)
mask = data['source_mask'].to(device, dtype=torch.long)

generated_ids = model.generate(input_ids=ids, attention_mask=mask, max_length=512, do_sample=True, top_p=0.9, top_k=0, num_return_sequences=1) 
```
@patrickvonplaten 

### Who can help?

_No response_

### Information

- [x] The official example scripts
- [X] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

I just trained the T5 model on a translation task and then generate text using the above code.

### Expected behavior

The model generates the same text even if I don't fix the random seed.

Comments:
Gently pinging @gante here

Hi @bilalghanem 👋 

Top-p has randomness -- when using `top_p=0.9`, `generate()` picks a token among the top candidate tokens, where the sum of the probability of those top candidates is >= 0.9. In other words, unless you model predicts a token with probability > 0.9 at each generation step, it will not be deterministic.

I'd recommend reading this blog post: https://github.com/huggingface/blog/blob/main/how-to-generate.md

> 

Thanks @gante.
Can you clarify, how it won't be deterministic if we don't find a token that satisfies the P condition? 

@bilalghanem It considers all tokens, from most to least likely, such that its summed probability is `top_p`, and samples (proportionally) from those tokens.

Consider the following logits array: `[0.5, 0.4, 0.1]`. If you run sampling with `top_p=0.9`, it will pick the first token `(0.5/0.9)*100 = 55.6%` of the times, the second token `(0.4/0.9)*100 = 44.4%` of the times, and the last token `0%` of the times.

If you want a deterministic behavior, use `do_sample=False`. Again, I'd recommend reading the following blog post, which explains all of this: https://github.com/huggingface/blog/blob/main/how-to-generate.md



This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.
Add ConvNeXt-V2 Model
Author: IMvision12
Pull request: yes

# What does this PR do?

Adds ConvNeXt-V2 to transformers.
original repo: https://github.com/facebookresearch/ConvNeXt-V2
paper: https://arxiv.org/abs/2301.00808

## Before submitting
- [ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [ ] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#start-contributing-pull-requests),
      Pull Request section?
- [ ] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [ ] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/main/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/main/docs#writing-source-documentation).
- [x] Did you write any new necessary tests?

@alaradirik 


Comments:
It does look like the model code is exactly the same at a first glance (saw everything is copied from ConvNext). If that is the case, yes to re-using the code of ConvNext, but if we need to make modifications in the convnext modeling file, we should add ConvNext V2 as a new model like in the PR.

> It does look like the model code is exactly the same at a first glance (saw everything is copied from ConvNext). If that is the case, yes to re-using the code of ConvNext, but if we need to make modifications in the convnext modeling file, we should add ConvNext V2 as a new model like in the PR.

Yes, the code is almost the same, but it adds a Global Response Normalization (GRN) module and removes the layer_scale_parameter from the ConvNeXtV2Layer. Makes more sense to add it as a new model then.

CC @IMvision12 

Thanks for the review @alaradirik I will address all comments!

_The documentation is not available anymore as the PR was closed or merged._
[`ImageProcessor`] Refactor default `mean` & `std` to `OPENAI_CLIP_MEAN` & `OPENAI_CLIP_STD`
Author: younesbelkada
Pull request: yes

# What does this PR do?

Initially this PR was intended to fix a small nit for BLIP feature extractors, the default normalization `mean` & `std` are currently incorrect as pointed out by @NielsRogge , BLIP uses the mean and std that are identical to CLIP: https://github.com/salesforce/LAVIS/blob/5ddd9b4e5149dbc514e81110e03d28458a754c5d/lavis/processors/blip_processors.py#L21 - this has no effect for the current models as the values were already correct on the Hub.

Therefore this PR adds a new variable in `constants.py`, `OPENAI_CLIP_MEAN` & `OPENAI_CLIP_STD` as this value is used by other models as well `CLIP`, `CLIPSeg`, `OWL-ViT` , etc.

cc @NielsRogge @sgugger 

Comments:
_The documentation is not available anymore as the PR was closed or merged._
[`BLIP`] update blip path on slow tests
Author: younesbelkada
Pull request: yes

# What does this PR do?

This PR updates the path to the image we use for running BLIP slow tests - as pointed out by @NielsRogge better to upload these images on the Hub in case they get removed from the original place
Happy also to move the Hub repo on `hf-internal-testing` but I am not a member of the org

Also the image is quite large, so I prefer to upload it on the Hub rather than pushing it on the repo here

cc @NielsRogge @sgugger 

Comments:
_The documentation is not available anymore as the PR was closed or merged._

Can you add it to hf-internal-testing instead? Might be better there. Happy to merge any PR or approve your demand to join ;-)

Thanks for adding! I've transferred it :) 

@sgugger , I think that we can merge this! 🙏
AutoModels for region-to-phrase-alignment and natural-language-for-visual-reasoning
Author: mszsorondo
Pull request: no

### Feature request

Hi! Region to phrase alignment and natural language for visual reasoning have no AutoModels yet. @sgugger is it OK to open a PR and solve this?

### Motivation

This is an issue that I faced when exporting VisualBert to ONNX, since the task mapping can't be done without it. Adding it would allow us to make the export for every task.

### Your contribution

I'll open a PR and solve it myself if allowed

Comments:
Note that the ONNX conversion is now done directly in the optimum library, so this is probably where you would need to add something.

That's true, I'm doing the ONNX conversion from optimum. But optimum references AutoModels directly from transformers. I'd just program the AutoModel subclasses without any optimum stuff, of course.

@sgugger Sorry for insisting, I know you have lots of issues opened already. Just tagging you to see if you still think this should be solved from Optimum

I think the optimum library should provide an API for models that don't have an auto API yes (if it does not already), as there will always be such models and we won't add a new auto class for just one model.

cc @michaelbenayoun who might have more information.

Hi @mszsorondo,
Could you open a PR on the Optimum repo, with your request explained please?

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.
UserWarning: __floordiv__ is deprecated, and its behavior will change in a future version of pytorch.
Author: FurkanGozukara
Pull request: no

### System Info

C:\Python399\lib\site-packages\transformers\models\bigbird_pegasus\modeling_bigbird_pegasus.py:807: UserWarning: __floordiv__ is deprecated, and its behavior will change in a future version of pytorch. It currently rounds toward 0 (like the 'trunc' function NOT 'floor'). This results in incorrect rounding for negative values. To keep the current behavior, use torch.div(a, b, rounding_mode='trunc'), or for actual floor division, use torch.div(a, b, rounding_mode='floor').
  torch.arange(indices.shape[0] * indices.shape[1] * num_indices_to_gather, device=indices.device)



```
import logging
from transformers import pipeline

f = open("TextFile1.txt", "r")

ARTICLE = f.read()

summarizer = pipeline("summarization", model="google/bigbird-pegasus-large-bigpatent" )
```



Comments:
cc @ArthurZucker so it's on your radar.

Interesting, we might have more model to refactor 

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.
Feature Request: Flax Whisper
Author: OhadRubin
Pull request: no

### Feature request

It already has TF and Pytorch support. Would be nice to use it on Flax as-well.


Comments:
cc @sanchit-gandhi 

Hey @OhadRubin! Thanks to the fantastic work by @andyehrenberg it's nearly complete: https://github.com/huggingface/transformers/pull/20479

Will do a final review tomorrow and then get it merged asap!

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.
[`bnb`] fix `bnb` decoders bug
Author: younesbelkada
Pull request: yes

# What does this PR do?

Currently on the `main` branch, there is a silent bug with `bnb` and encoder-decoder models leading to some modules not being converted in `int8` for these models and hurting users that are using the `main` branch - such as `peft`: https://github.com/huggingface/peft/issues/108 

With https://github.com/huggingface/transformers/pull/21579 being introduced, the check to know if we should keep the module as `nn.Linear` has been slightly changed and [being more robust ](https://github.com/huggingface/transformers/blob/7f1cdf18958efef6339040ba91edb32ae7377720/src/transformers/utils/bitsandbytes.py#L126).

Before #21579 the function `get_keys_to_not_convert` used to return `['decoder', 'lm_head', 'wo']` which is wrong, and leading to all decoder layers being not converted in int8 with the new check as mentioned above.

This PR fixes the bug that was in this function, and adds a new test to make sure this will never happen

Fixes: https://github.com/huggingface/peft/issues/108

cc @sgugger @amyeroberts 

Comments:
_The documentation is not available anymore as the PR was closed or merged._
Mask dimension expansion might be wrong
Author: pphuangyi
Pull request: no

https://github.com/huggingface/transformers/blob/17292440c069118fbdb992b9a17da2098fab5b87/src/transformers/models/reformer/modeling_reformer.py#L845

I feel that the way mask is expanded here might be wrong. More specifically, I feel that mask need to be repeated by the number hashes and splited into chunks before running gather. 

Please look into it and let me know if I misunderstood. Thank you so much!

Comments:
cc @ArthurZucker 

Hey! Do you happen to have found an error or any discrepancy ? I am not very familiar with this model but if you have a reproduction script or something to help me work on this, would be appreciated! 

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.
RAG: Which version of pytorch-forecasting to use for fintetune-rag.sh? 
Author: Mrs-Hudson
Pull request: no

### System Info


- `transformers` version: 4.24.0
- Platform: Linux-3.10.0-1160.83.1.el7.x86_64-x86_64-with-glibc2.17
- Python version: 3.8.16
- Huggingface_hub version: 0.10.1
- PyTorch version (GPU?): 1.11.0 (True)
- Tensorflow version (GPU?): not installed (NA)
- Flax version (CPU?/GPU?/TPU?): not installed (NA)
- Jax version: not installed
- JaxLib version: not installed
- Using GPU in script?: <fill in>
- Using distributed or parallel set-up in script?: <fill in>


### Who can help?

@ArthurZucker  @younesbelkada 

### Information

- [x] The official example scripts
- [ ] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [x] My own task or dataset (give details below)

### Reproduction

The latest version of  pytorch-forecasting is not comptabible with the RAG module of transformers. I was able to successfuly create my own knowledge dataset but the finetune-rag.py step fails.  I also looked at the version history and experimented with 1.6.4 throws an error

I was wondering if there's a known version of pytorch_lightning that works with RAG.

Steps to Reproduce:
 1. From transformers root, run a shell script with
 `python examples/research_projects/rag/finetune_rag.py --data_dir /home/rparik/linkedInEmpForKnowledgeClean/data.csv \
    --output_dir ./finetune_rag_output/ \
    --model_name_or_path facebook/rag-token-nq \
    --model_type rag_sequence \
    --fp16 \
    --gpus 1 \
    --index_name custom \
    --passages_path /home/rparik/projects/rag/transformers/linkKnowledgeBase/my_knowledge_dataset \
    --index_path /home/rparik/projects/rag/transformers/linkKnowledgeBase/my_knowledge_dataset_hnsw_index.faiss \
`
Error:

 ```
 File "examples/research_projects/rag/finetune_rag.py", line 17, in <module>
    from pytorch_lightning.plugins.training_type import DDPStrategy
ModuleNotFoundError: No module named 'pytorch_lightning.plugins.training_type'
```



### Expected behavior

Script runs without errors

Comments:
Looks like 1.6.0 worked

The requirement file states `pytorch-lightning >= 1.5.10`. Were you using an older version or a newer version? 
(shoud I pin version 1.6.0 as the max version?)

I can confirm that 1.9.1 didn't work, neither did 1.6.4. haven't tried
1.6.1-1.6.3

On Mon, Feb 20, 2023, 01:17 Arthur ***@***.***> wrote:

> The requirement file states pytorch-lightning >= 1.5.10. Were you using
> an older version or a newer version?
> (shoud I pin version 1.6.0 as the max version?)
>
> —
> Reply to this email directly, view it on GitHub
> <https://github.com/huggingface/transformers/issues/21692#issuecomment-1436603658>,
> or unsubscribe
> <https://github.com/notifications/unsubscribe-auth/ABVQKHJ2YDESVJCITYJPPXDWYMZCTANCNFSM6AAAAAAVBEDYJU>
> .
> You are receiving this because you authored the thread.Message ID:
> ***@***.***>
>
Bloom's hidden states are None
Author: minhngh
Pull request: no

Hi everyone,

I used BloomForCausalLM which was pretrained and released by BigScience. I then forwarded input tensors to the model, and I tried to get the corresponding hidden states. However, the error was raised, which said that hidden states are None.
Has anyone encountered the same problem. Thank you very much.
 
https://github.com/huggingface/transformers/blob/ae54e3c3b18bac0832ad62ea9b896dfd52a09850/src/transformers/models/bloom/modeling_bloom.py#L935

Comments:
I found I had not checked the code carefully. This problem could be solved by passing the argument`output_hidden_states = True` to the `forward`
Remote code is loaded from `main` even when revision is provided
Author: lvwerra
Pull request: no

### System Info

When specifying a branch to load a model with remote code as follows fails because there is no modeling file on `main`. Is this a bug or the expected behaviour? 

### Who can help?

The one and only _**@sgugger**_

### Information

- [ ] The official example scripts
- [ ] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

```python
model = transformers.AutoModelForCausalLM.from_pretrained("bigcode/santacoder-fast-inference", revision="main_custom", trust_remote_code=True)
```

The following error shows that the code file is attempted to be loaded from `main` instead of `main_custom` (where a modeling file is present:

```bash
Could not locate the configuration_gpt_bigcode.py inside bigcode/santacoder-fast-inference.
Traceback (most recent call last):
  File "/work/arjunguha-research-group/arjun/venvs/bigcode/lib/python3.8/site-packages/huggingface_hub/utils/_errors.py", line 264, in hf_raise_for_status
    response.raise_for_status()
  File "/shared/centos7/python/3.8.1/lib/python3.8/site-packages/requests/models.py", line 940, in raise_for_status
    raise HTTPError(http_error_msg, response=self)
requests.exceptions.HTTPError: 404 Client Error: Not Found for url: https://huggingface.co/bigcode/santacoder-fast-inference/resolve/main/configuration_gpt_bigcode.py
```


### Expected behavior

Loading without error.

Comments:
Will have a look even if you didn't properly tag me :-p 

The PR linked above fixes the config problem (I can load it with `AutoConfig`). The model still won't load however as the auto mpa of that config doesn't contain an entry for `AutoModelForCausalLM`.
Add BLIP-2
Author: NielsRogge
Pull request: yes

# What does this PR do?

This PR adds BLIP-2 to the library.

To do:

- [x] make sure generation works exactly as the original implementation, (maybe @gante can have a look here - based on original code [here](https://github.com/salesforce/LAVIS/blob/main/lavis/models/blip2_models/blip2_opt.py#L207-L211)). Edit: seems to be solved by properly setting the `eos_token_id`!
- [x] add more tests for BLIP-2 with `AutoModelForSeq2SeqLM` once designed gets approved
- [x] transfer checkpoints, update integration tests
- [ ] make it possible to instantiate Blip2Config with config objects, rather than dicts (also check default text config) - will be done in a separate PR

cc @younesbelkada 

Comments:
_The documentation is not available anymore as the PR was closed or merged._

@sgugger all comments are addressed, feel free to approve :)

@NielsRogge Curious, what is the timeline for this to make it into a stable release version?

Usually there's a Transformers release once every 1 to 2 months, so at the very least in March.

Hi, thanks for the great work! I'm running into problems trying to use this in the multigpu setting and saw this was mentioned by @younesbelkada earlier -- is there an issue to follow for that? (Specifically, in line 2765 of transformers->generation->utils.py, the devices don't match -- `Expected all tensors to be on the same device, but found at least two devices, cuda:3 and cuda:0!` because beam_scores is on cuda:0 while next_token_scores and next_token_scores_processed are on cuda:3 after using "auto" for the device_map when loading.)

I'm also getting a weirder error where it causes a CUDA illegal memory access error for any model used downstream of it on GPU 0, even when it's given no GPU memory on GPU 0 in max_memory. (This doesn't occur for the original BLIP2, which I'm trying to migrate from.)

Same problem here @sachit-menon 
"Expected all tensors to be on the same device, but found at least two devices, cuda:1 and cuda:0!"
https://github.com/TimDettmers/bitsandbytes/issues/153

Hi @sachit-menon @xszheng2020 
This is a known issue on my end, I can confirm this should be at least fixed for `blip2-opt` at https://github.com/huggingface/transformers/pull/21707
Can you try to checkout from this branch and let us know on the PR if the fix works? thanks!

Hi, @younesbelkada 
thanks! will test it on blip2-opt to see whether it works!
and hope the blip2-flant5 could be fixed soon
GPT NeoX Input docstring includes token_type_ids but GPTNeoXModel does not have token_type_ids as an input to forward
Author: dmahan93
Pull request: no

### System Info

n/a

### Who can help?

@ArthurZucker @younesbelkada 

### Information

- [X] The official example scripts
- [ ] My own modified scripts

### Tasks

- [X] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

Go to https://huggingface.co/docs/transformers/main/en/model_doc/gpt_neox#transformers.GPTNeoXModel and see that forward does not include token_type_id but it is shown in the docstring

### Expected behavior

Docstring should match forward

Comments:
also applies to position_ids