noamaanMulla-03/transformers-issues huggingface dataset:
NotImplementedError: A model class needs to define a `prepare_inputs_for_generation` method in order to use `.generate()`.
Author: nikhilajoshy
Pull request: no

### System Info

- `transformers` version: 4.37.2
- Platform: Windows-10-10.0.19045-SP0
- Python version: 3.11.0
- Huggingface_hub version: 0.20.3
- Safetensors version: 0.4.2
- Accelerate version: not installed
- Accelerate config: not found
- PyTorch version (GPU?): not installed (NA)
- Tensorflow version (GPU?): not installed (NA)
- Flax version (CPU?/GPU?/TPU?): not installed (NA)
- Jax version: not installed
- JaxLib version: not installed

### Who can help?

_No response_

### Information

- [X] The official example scripts
- [ ] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

`
model = EncoderDecoderModel(encoder=encoder, decoder=decoder)
model.eval()
    enc_tokenizer = AutoTokenizer.from_pretrained("google-bert/bert-base-uncased")
    dec_tokenizer = AutoTokenizer.from_pretrained("openai-community/gpt2")
    with torch.no_grad():
        inputs = enc_tokenizer("I like bananas", return_tensors="pt")
        outputs = model.generate(**inputs,  max_new_tokens=200)
        print(dec_tokenizer.batch_decode(**outputs))
`

### Expected behavior

Not able to perform model.generate with bert base and gpt2 from huggingface

Comments:
Hi @nikhilajoshy, thanks for raising an issue! 

Please make sure to share the full error traceback in the issue information. 

Could you also make sure that the code is properly formatted in the example and that it can be run to fully reproduce the error?
Cannot load transformer model from hugging face in remote server
Author: nikhilajoshy
Pull request: no

### System Info

 `transformers` version: 4.37.2
- Platform: Linux-5.15.0-92-generic-x86_64-with-glibc2.31
- Python version: 3.11.7
- Huggingface_hub version: 0.20.3
- Safetensors version: 0.4.2
- Accelerate version: 0.26.1
- Accelerate config:    not found
- PyTorch version (GPU?): 2.2.0+cu121 (True)
- Tensorflow version (GPU?): not installed (NA)
- Flax version (CPU?/GPU?/TPU?): not installed (NA)
- Jax version: not installed
- JaxLib version: not installed



### Who can help?

_No response_

### Information

- [X] The official example scripts
- [ ] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

1. `tokenizer = AutoTokenizer.from_pretrained("distilbert/distilgpt2")`

### Expected behavior

(MaxRetryError("HTTPSConnectionPool(host='huggingface.co', port=443): Max retries exceeded with url: /distilbert/distilgpt2/resolve/main/tokenizer_config.json (Caused by ProxyError('Cannot connect to proxy.', NewConnectionError('<urllib3.connection.HTTPSConnection object at 0x7fa612ffa750>: Failed to establish a new connection: [Errno 111] Connection refused')))")

Comments:
* are you using the library inside a docker container ? 
* do you have a `https_proxy` or a `HTTPS_PROXY` envirenmental variables set ?
* which python version of `urllib3`  are you using
* do you have VPN enabled ? 

@not-lain 

- it's not inside docker, just a vent
- I have a http_proxy set but not https.
- 1.26.18
- yes

@nikhilajoshy 
try these fixes : 
* upgrade the requests library to a stable version
```
pip install requests==2.27.1   
```
* or set the following envirenmental variable to an empty string
```python
import os
os.environ['CURL_CA_BUNDLE'] = ''
```

if none of the above did not work for you try checking this https://stackoverflow.com/questions/56016210/proxyerrorcannot-connect-to-proxy-newconnectionerror
Introduce AcceleratorConfig dataclass
Author: muellerzr
Pull request: yes
Labels: name: trainer

# What does this PR do?

This PR centralizes all arguments for the `Accelerator` not covered by `fsdp_config` and `deepspeed_config` into a singular dataclass that users can pass in as a json file or through raw CLI param args.

I *think* I have the CLI args configured right? But I'm not 100% sure. Advice on how to check that would be appreciated!

Fixes # (issue)


## Before submitting
- [ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [x] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#start-contributing-pull-requests),
      Pull Request section?
- [ ] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [ ] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/main/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/main/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests?


## Who can review?

Anyone in the community is free to review the PR once the tests have passed. Feel free to tag
members/contributors who may be interested in your PR.

@amyeroberts @LysandreJik 

Comments:
@amyeroberts tests finally pass after losing my mind on imports 🙌 

Hi @muellerzr , could you please get this PR merged? We need the PR to get the accelerate test on TPU v3 to succeed again. Thanks.

As a general note, immediately after the next accelerate release I'll make a follow-up PR utilizing the config class seen here so users won't have annoying FutureWarning's https://github.com/huggingface/accelerate/pull/2441 (No big logic difference, just shove it all into the config rather than the Accelerator based on the accelerate version)

For the doc tests, there was a recent commit merged to main, which should hopefully resolve this. 



The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_28664). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
Fix a typo in `examples/pytorch/text-classification/run_classification.py`
Author: Ja1Zhou
Pull request: yes

Hi there, this commit fixes a tiny typo in the provided pytorch text classification pipeline.

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_29072). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
Request to add FLMR
Author: LinWeizheDragon
Pull request: no
Labels: name: New model

### Model description

## Basic Information

This issue requests adding Fine-grained Late-interaction Multi-modal Retriever (FLMR).
The model leverages late interaction (as originally proposed by Stanford [ColBERT](https://github.com/stanford-futuredata/ColBERT)) to compute token-level similarity between every query token and document token, which enables more accurate retrieval relative to DPR-like systems.
 The model was proposed in [here](https://openreview.net/forum?id=IWWWulAX7g) (NeurIPS 2023) and [here](https://arxiv.org/abs/2402.08327) (a follow-up version that was pre-trained on more than ten million of multi-modal retrieval data). 


## Resources
- Project page [here](https://preflmr.github.io/)
- Official codebase [here](https://github.com/linweizhedragon/retrieval-augmented-visual-question-answering)
- The pre-trained checkpoints are [here](https://huggingface.co/models?search=PreFLMR).

## Why adding this model
1. This work has gained attention from researchers across the world. We received many requests during NeurIPS 2023 to provide an easy implementation of this model.
2. The work has been recognized by the authors of ColBERT [twitter post](https://twitter.com/lateinteraction/status/1757652639503007893)
3. There exists no implementation for late-interaction retrieval models in hf-transformers, which have been extensively researched in these years.
4. There are many requests in the original codebase [an example issue](https://github.com/LinWeizheDragon/Retrieval-Augmented-Visual-Question-Answering/issues/24)
5. As original authors, we have finished 99% of the model, including an example for indexing and searching.   Limited work is required to have it on huggingface-transformers!  #29062 




### Open source status

- [x] The model implementation is available
- [X] The model weights are available

### Provide useful links for the implementation

The PR is already here:
#29062 

Comments:
[Docs] Add resources
Author: NielsRogge
Pull request: yes

# What does this PR do?

This PR adds some more resources regarding various models.

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_28705). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
StoppingCriteria tracks elements separately in the batch
Author: zucchini-nlp
Pull request: yes

# What does this PR do?

As was pointed out in #28932 , StoppingCriteria needs to stop generation per batch element and return a boolean tensor of `batch_size`. This PR adds the logic to track each row and when StoppingCriteria is triggered, stop generating for that particular row only. 

Note that the when #28932 gets merged, we need to add logic to handle beam related generation. The problem is that beam search has an internal logic of tracking EOS tokens and adds candidate tokens to hypothesis when done. And if StoppingCriteria will take the responsibility to track custom EOS tokens, it has to be passed to beam scorer. Right now I am not sure if calling StoppingCriteria twice is a good decision. First time to check candidate beams, and the second time for the chosen beams. What do you think @gante? It can be something like:

```
cur_len = input_ids.shape[-1] + 1
beam_next_input_ids = torch.cat([input_ids[next_indices, :], next_tokens.unsqueeze(-1)], dim=-1)
beam_next_input_ids = beam_next_input_ids.view(-1, cur_len)
next_is_done = stopping_criteria(beam_next_input_ids, scores)

# is all are done, then it's prob max_length, not custom EOS being triggered
if all(next_is_done):
    next_is_done = torch.full_like(next_indices, False, dtype=torch.bool)
next_is_done = next_is_done.view(next_indices.shape)
```


## Before submitting
- [ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [X] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#create-a-pull-request),
      Pull Request section?
- [X] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [X] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/main/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/main/docs#writing-source-documentation).
- [X] Did you write any new necessary tests?


## Who can review?

Anyone in the community is free to review the PR once the tests have passed. Feel free to tag
members/contributors who may be interested in your PR.
 @gante 

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_29056). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
Bnb test fix for different hardwares
Author: Titus-von-Koeller
Pull request: yes

Just updating the acceptable generated text, as this is slightly different based on hardware from what I can tell. The first commit is based on what I observed on my dev VM with A10G and the second commit is based on what I saw [in the failing BNB integration pipeline](https://github.com/huggingface/peft/actions/runs/7925057159/job/21637597387#step:9:83).

In my eyes, both text strings seem similar enough. I already spoke about this with @younesbelkada regarding the first commit. The second seems a similar case.

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_29066). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

Could one of you please merge? I don't have permission to do so.
Fix two tiny typos in `pipelines/base.py::Pipeline::_sanitize_parameters()`'s docstring
Author: sadra-barikbin
Pull request: yes

Hi there!
To fix two tiny typos in `pipelines/base.py::Pipeline::_sanitize_parameters()`'s docstring.

@Narsil


Comments:
One more thing. We have `False` for default value of `clean_up_tokenization_spaces` argument in `TextGenerationPipeline`'s docstring:
https://github.com/huggingface/transformers/blob/593230f0a1150ea9c0477b9d859f25daf73c8c33/src/transformers/pipelines/text_generation.py#L207-L208

but its default value in `TextGenerationPipeline::postprocess` is `True`:
https://github.com/huggingface/transformers/blob/593230f0a1150ea9c0477b9d859f25daf73c8c33/src/transformers/pipelines/text_generation.py#L336

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_29102). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

@sadra-barikbin Good spot! Could you add a fix for the docstring in this PR too?

The pipelines in `text2text_generation.py` class have `clean_up_tokenization_spaces=False` as default. This has been the desired behavior?

@sadra-barikbin Yes, I believe so. Changing the default behaviour would be a breaking change, so we should keep as-is
Generation doesn't work as expected with input_embeds
Author: dipta007
Pull request: no

### System Info

- `transformers` version: 4.35.2
- Platform: Linux-6.1.58+-x86_64-with-glibc2.35
- Python version: 3.10.12
- Huggingface_hub version: 0.20.3
- Safetensors version: 0.4.2
- Accelerate version: not installed
- Accelerate config: not found
- PyTorch version (GPU?): 2.1.0+cu121 (False)
- Tensorflow version (GPU?): 2.15.0 (False)
- Flax version (CPU?/GPU?/TPU?): 0.8.1 (cpu)
- Jax version: 0.4.23
- JaxLib version: 0.4.23

### Who can help?

@gante 

### Information

- [ ] The official example scripts
- [X] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

Colab: https://colab.research.google.com/drive/1tIgcJVofkzTic9ChhG8rNAQmFjrE1HYd?usp=sharing

### Expected behavior

Output should be generated for both using `input_ids` or `inputs_embed`.
But when giving `inputs_embed`, it doesn't consider the input length in `max_length` stopping criteria. So it raises an exception when the output exceeds `model.max_length`.

Comments:
@dipta007 ,hi! This [PR](https://github.com/huggingface/transformers/pull/28994) fixed it

@dipta007 as @zucchini-nlp wrote, if you add `!pip install --upgrade git+https://github.com/huggingface/transformers.git` on top of your notebook it will work 🤗 

Thanks
Generate: add config-level validation
Author: gante
Pull request: yes

# What does this PR do?

This PR adds generation argument validation that can be performed at a `generation_config` level. 

It aims at reducing the number of issues due to incorrect parameterization, where the user expects some argument to modify the output but it doesn't.


Comments:
_The documentation is not available anymore as the PR was closed or merged._

@gante 

The warnings
```
/home/felix/transformers/src/transformers/generation/configuration_utils.py:412: UserWarning: `do_sample` is set to `False`. However, `top_p` is set to `0.6` -- this flag is only used in sample-based generation modes. You should set `do_sample=True` or unset `top_p`.
  warnings.warn(
/home/felix/transformers/src/transformers/generation/configuration_utils.py:422: UserWarning: `do_sample` is set to `False`. However, `top_k` is set to `1` -- this flag is only used in sample-based generation modes. You should set `do_sample=True` or unset `top_k`.
  warnings.warn(
```

do not explicit how to unset the parameters. My first intuition was `top_k=None`, etc., but this does not work. In fact, one needs to call specifically:

```
gen_out = model.generate(**inputs, do_sample=False, temperature=1, top_p=1, top_k=50)
```

to remove the warnings, which I find counter-intuitive. Could we allow `None`?

@fxmarty https://github.com/huggingface/transformers/pull/29119 :D
fix the post-processing link
Author: davies-w
Pull request: yes

The link in evaluation was missing a hyphen between post and processing. I fixed this, for English only. Someone with the ability to do a global search/replace should fix the other languages (if indeed they have this issue).

# What does this PR do? 

Fixes a broken link in the documentation.


## Before submitting
- [x] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [ ] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#create-a-pull-request),
      Pull Request section?
- [ ] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [ ] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/main/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/main/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests?


## Who can review?

Anyone in the community is free to review the PR once the tests have passed. Feel free to tag
members/contributors who may be interested in your PR.

<!-- Your PR will be replied to more quickly if you can figure out the right person to tag with @

 If you know how to use git blame, that is the easiest way, otherwise, here is a rough guide of **who to tag**.
 Please tag fewer than 3 people.

Models:

- text models: @ArthurZucker and @younesbelkada
- vision models: @amyeroberts
- speech models: @sanchit-gandhi
- graph models: @clefourrier

Library:

- flax: @sanchit-gandhi
- generate: @gante
- pipelines: @Narsil
- tensorflow: @gante and @Rocketknight1
- tokenizers: @ArthurZucker
- trainer: @muellerzr and @pacman100

Integrations:

- deepspeed: HF Trainer/Accelerate: @pacman100
- ray/raytune: @richardliaw, @amogkam
- Big Model Inference: @SunMarc
- quantization (bitsandbytes, autogpt): @SunMarc and @younesbelkada

Documentation: @stevhliu and @MKhalusova

HF projects:

- accelerate: [different repo](https://github.com/huggingface/accelerate)
- datasets: [different repo](https://github.com/huggingface/datasets)
- diffusers: [different repo](https://github.com/huggingface/diffusers)
- rust tokenizers: [different repo](https://github.com/huggingface/tokenizers)

Maintained examples (not research project or legacy):

- Flax: @sanchit-gandhi
- PyTorch: See Models above and tag the person corresponding to the modality of the example.
- TensorFlow: @Rocketknight1

 -->


Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_29091). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
Move misplaced line
Author: kno10
Pull request: yes

Move misplaced line, improve code comment.

No functional change, the loss_fct is not used earlier and did not match the code comment either.

## Before submitting
- [X] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).

## Who can review?

@ArthurZucker and @younesbelkada


Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_29117). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
FalconAttention Doesn't use `alibi` when `alibi` is not None if `_use_sdpa==True`
Author: SamanehSaadat
Pull request: no

### System Info

- `transformers` version: 4.37.0.dev0
- Platform: Linux-6.5.13-1rodete2-amd64-x86_64
- Python version: 3.10.13
- Huggingface_hub version: 0.20.3
- Safetensors version: 0.4.2
- Accelerate version: not installed
- Accelerate config: not found
- PyTorch version (GPU?): 2.2.0+cu121 (False)
- Tensorflow version (GPU?): not installed (NA)
- Flax version (CPU?/GPU?/TPU?): not installed (NA)
- Jax version: not installed
- JaxLib version: not installed
- Using GPU in script?: <fill in>
- Using distributed or parallel set-up in script?: <fill in>

### Who can help?

@fxmarty 
@ArthurZucker
@younesbelkada


### Information

- [ ] The official example scripts
- [ ] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

I found this issue when I was reading the Falcon code:
[This else](https://github.com/huggingface/transformers/blob/main/src/transformers/models/falcon/modeling_falcon.py#L477-L526) is related to when `alibi is not None`:
* When `_use_sdpa==False`, `alibi` is added to `attention_scores` [here](https://github.com/huggingface/transformers/blob/main/src/transformers/models/falcon/modeling_falcon.py#L503) 
* But when `_use_sdpa==True`, `alibi` hasn't been used anywhere in [this part of code](https://github.com/huggingface/transformers/blob/2f1003be86f11c8d97d7c2e6a7739dbb6fa795f2/src/transformers/models/falcon/modeling_falcon.py#L478-L490).

### Expected behavior

`alibi` should be used when `alibi` is not `None`.

Comments:
HEy! Pretty sure that is because it's not supported in `sdpa` nor is `flash_attention`. We should / could raise an error however. 

Hi @SamanehSaadat thank you for the notice, please refer to: https://github.com/huggingface/transformers/blob/2f1003be86f11c8d97d7c2e6a7739dbb6fa795f2/src/transformers/models/falcon/modeling_falcon.py#L1104-L1124. Alibi is merged into the attention bias. We do so at the top-level and not in every attention module.

Oh thanks @fxmarty for correcting me!