noamaanMulla-03/transformers-issues huggingface dataset:
RagSequenceForGeneration model outputs meaningless content
Author: Aisuko
Pull request: no

Hi, team, thanks for your great work.

### System Info

```
- `transformers` version: 4.38.2
- Platform: Linux-5.15.133+-x86_64-with-glibc2.31
- Python version: 3.10.13
- Huggingface_hub version: 0.20.3
- Safetensors version: 0.4.2
- Accelerate version: 0.27.2
- Accelerate config: 	not found
- PyTorch version (GPU?): 2.1.2+cpu (False)
- Tensorflow version (GPU?): 2.15.0 (False)
- Flax version (CPU?/GPU?/TPU?): 0.8.1 (cpu)
- Jax version: 0.4.24
- JaxLib version: 0.4.24
- Using GPU in script?: <fill in>
- Using distributed or parallel set-up in script?: <fill in>
```

### Who can help?

_No response_

### Information

- [X] The official example scripts
- [ ] My own modified scripts

### Tasks

- [X] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

The original code is here: https://huggingface.co/docs/transformers/model_doc/rag#transformers.RagSequenceForGeneration

Here is the notebook on Kaggle if it is helpful for showing more details: https://www.kaggle.com/code/aisuko/rag-with-ragsequenceforgeneration

Minimal code
```
!conda install -c pytorch faiss-cpu=1.8.0 -y

!pip install transformers==4.38.2
!pip install datasets==2.18.0 # Fix issue numpy attributes error of RagRetriever

os.environ["MODEL_NAME"] = "facebook/rag-token-base"
device='cpu'

from transformers import AutoTokenizer
from transformers import RagRetriever
from transformers import RagSequenceForGeneration

tokenizer=AutoTokenizer.from_pretrained(os.getenv("MODEL_NAME"))

retriever=RagRetriever.from_pretrained(
    os.getenv("MODEL_NAME"), 
    index_name="exact", 
    use_dummy_dataset=True,
    trust_remote_code=True
)

model=RagSequenceForGeneration.from_pretrained(os.getenv("MODEL_NAME"), retriever=retriever, device=device)
input_ids=tokenizer("How many people live in Melbourne?", return_tensors="pt")

generated=model.generate(input_ids["input_ids"], max_length=50)
generated_str=tokenizer.decode(generated[0], skip_special_tokens=True)
generated_str
```

The output always

```
'The!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!'
```

### Expected behavior

I assumed the the output should be more related to the input.

Comments:
cc @ArthurZucker @younesbelkada 

Sorry for the late answer, trying to reproduce now. 
Getting a lot of warnings which seem to be expected. I followed https://huggingface.co/docs/transformers/model_doc/rag#transformers.RagTokenForGeneration.forward.example and it generated ` 11,912,253`. I used `"facebook/rag-token-nq"`. Seems more like a model issue at this point (base model vs finetuned)

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.
Update jamba
Author: ArthurZucker
Pull request: yes

# What does this PR do?
Update the PR to add jamba

Comments:
(the [latest commit](https://github.com/huggingface/transformers/pull/30164/commits/82be569ea0459832fce57ec3de3fc0e1dd29c454) fixes a generalist assisted generation bug -- the cache position was not being correctly computed when tokens were accepted 👀 )
Support mix/complex FSDP wrap policy in Trainer
Author: lchu6
Pull request: no

### Feature request

Support complex FSDP wrapping policy in Trainer api.

### Motivation

The current [implementation](https://github.com/huggingface/transformers/blob/2fc33ebead50383f7707b17f0e2a178d86347d10/src/transformers/trainer.py#L1387-L1404) seems to assume the wrap policy is either size based or transformer block(s) based, but there are many scenarios that might need other kinds of policy or a mix of different policies.

One example is [llama-recipes ](https://github.com/facebookresearch/llama-recipes/blob/cf678b9bf0af1c0e68e83b6c378e98125e0bc132/src/llama_recipes/utils/fsdp_utils.py#L4) uses a combination of lambda policy and transformer wrap policy.  So this makes it a little hard to use Trainer api to achieve the same thing.


Comments:
Figured we can leverage `trainer.accelerator.state.fsdp_plugin.auto_wrap_policy()` to override the policy. closing this for now.

Hi @lchu-ibm,

> Figured we can leverage trainer.accelerator.state.fsdp_plugin.auto_wrap_policy() to override the policy. closing this for now.

Could you please elaborate on your findings more? If I understand your comments correctly, you meant something like:
```python
trainer = Trainer(model, tokenizer, training_args, ...)
trainer.accelerator.state.fsdp_plugin.auto_wrap_policy = our_custom_wrap_policy # override the policy here
trainer.train()
```
I wonder if I understand your comments correctly or not.

Thanks
Update runner tag for PR slow CI
Author: ydshieh
Pull request: yes

# What does this PR do?

#30341 added a new workflow file using the runner `t4, daily-ci` which is also used by daily CI. And they can interfere each other.

After internal discussion with Guillaume, we think the best is to use `t4, ci`.

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_30535). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
[Question]: Can I obtain the probability from generated text?
Author: Gaiejj
Pull request: no

### Feature request

Thanks first for your work! It is truly helpful! I wonder if there is a mechanism that can retroactively determine the token-wise probability of a model generating a given text. For example, model A generated a sentence:

``Hello, how are you?``

Assuming each word represents a token, the corresponding token-wise probability is:

``ModelA: 0.8, 0.9, 0.9, 0.8``

Can I calculate the token-wise probability of model B generating this sentence based on it? e.g.:
``Model B: 0.2, 0.3, 0.4, 0.5``

### Motivation

N/A

### Your contribution

N/A

Comments:
Hi,

This is a question that is more suited for the forum (as it's not a bug or feature request).

See the answer here: https://discuss.huggingface.co/t/announcement-generation-get-probabilities-for-generated-output/30075

Thanks for your help!
Fix repo. fetch/checkout in PR slow CI job
Author: ydshieh
Pull request: yes

# What does this PR do?

Fix repo. fetch/checkout in PR slow CI job.

The previous way:

> git fetch && git checkout ${{ github.event.pull_request.head.sha }}

only works for a pull request that is pushed to a branch of `huggingface/transformers`. 

For PRs from outside contributors (so push to a forked repository), it will fail.

The new way

> git fetch origin pull/${{ github.event.pull_request.number }}/head:pull/${{ github.event.pull_request.number }}/merge && git checkout pull/${{ github.event.pull_request.number }}/merge

works in both cases.



Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_30537). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
Pass attn_implementation when using AutoXXX.from_config
Author: amyeroberts
Pull request: yes

# What does this PR do?

Passes `attn_implementation=config._attn_implementation` when creating a model using `AutoXxx.from_config`. This ensures the attention implementation selected is complementary with the parent model. 

Not added for some models, as the default attn implementation selected for the model isn't compatible with the component model c.f. [a57b9b5](https://github.com/huggingface/transformers/pull/30507/commits/a57b9b53aae6f0ad720da91d6894b1f69ad6c541)

If this is not passed, then breaking changes can occur when the attention implementation in the component model being loaded changes -- see #30506 for an example of breaking changes. 

For example, when SDPA was added to BERT, this attention mechanism was then selected by default. This however wasn't supported by other models using it as an encoder. 

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_30507). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

thank you!
[SegGPT] Loss calculation is broken
Author: EduardoPach
Pull request: no
Labels: name: Should Fix

name: Vision

### System Info

.

### Who can help?

@EduardoPach 

### Information

- [ ] The official example scripts
- [ ] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

```python
from datasets import load_dataset
from transformers import AutoImageProcessor, Trainer, TrainingArguments

dataset_id = "EduardoPacheco/FoodSeg103"
model_id = "BAAI/seggpt-vit-large"

# See implementation at https://github.com/EduardoPach/seggpt-finetuning/blob/main/src/seggpt_adapter.py#L9
class SegGptAdapter:
    ...

# See implementation at https://github.com/EduardoPach/seggpt-finetuning/blob/main/src/data.py#L61
class TransformInContextTuning:
    ....

model = SegGptAdapter(model_id)
config = model.seggpt.config
image_processor = AutoImageProcessor.from_pretrained(model_id)

transform_train = TransformInContextTuning(...)
ds_train = load_dataset(dataset_id, split="train")
ds_train.set_transform(transform_train)

train_args = TrainingArguments(...)

trainer = Trainer(
        model=model,
        args=train_args,
        train_dataset=ds_train,
)

trainer.train()
```

### Expected behavior

I'm currently getting the following error:

```bash
File "/Users/eduardopacheco/miniconda3/envs/seggpt-finetuning/lib/python3.11/site-packages/transformers/models/seggpt/modeling_seggpt.py", line 843, in unpatchify
    raise ValueError(f"Number of patches {tensor.shape[1]} does not match patch height and width.")
ValueError: Number of patches 1568 does not match patch height and width.
```

Where the expected behavior would be the loss being computed 

Comments:
Include safetensors as part of `_load_best_model`
Author: muellerzr
Pull request: yes

# What does this PR do?

This PR checks if the safe-weights version of the model is available to load in when using sharded models.

Fixes https://github.com/huggingface/transformers/issues/29799


## Before submitting
- [ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [ ] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#create-a-pull-request),
      Pull Request section?
- [ ] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [ ] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/main/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/main/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests?


## Who can review?

Anyone in the community is free to review the PR once the tests have passed. Feel free to tag
members/contributors who may be interested in your PR.

@amyeroberts 


Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_30553). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
Sharded safetensors are not loaded nor pushed to hub in Trainer
Author: AlexTMallen
Pull request: no
Labels: name: trainer

name: bug

### System Info

- `transformers` version: 4.37.2
- Platform: Linux-5.19.17-coreweave-x86_64-with-glibc2.31
- Python version: 3.11.7
- Huggingface_hub version: 0.20.3
- Safetensors version: 0.4.2
- Accelerate version: 0.27.0
- Accelerate config:    not found
- PyTorch version (GPU?): 2.2.0+cu121 (True)
- Tensorflow version (GPU?): not installed (NA)
- Flax version (CPU?/GPU?/TPU?): not installed (NA)
- Jax version: not installed
- JaxLib version: not installed
- Using GPU in script?: yes
- Using distributed or parallel set-up in script?: no

### Who can help?

@muellerzr and @pacman100

### Information

- [ ] The official example scripts
- [X] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [X] My own task or dataset (give details below)

### Reproduction

This issue occurs when using `load_best_model_at_end` in Trainer with models large enough to need sharded checkpoints (e.g. Pythia 2.8b but not 1.4b).

https://github.com/huggingface/transformers/blob/092f1fdaa4224fdd88c616dc9678e6fcb37bfffd/src/transformers/trainer.py#L2361 Should be updated to account not only for pickle indices but also safetensors indices.

### Expected behavior

I expected the script to push model checkpoints to the hub, but it does not. Nor can it load the best model at the end (which should be easily fixed at the line linked above).

Comments:
Bumping on this issue to mark as not stale.
Question about iterable inputs for Pipeline
Author: ww-rm
Pull request: no

Hi, I'm using `Pipeline` in some memory constrained scenarios and struggling with its inputs types.

To reduce memory usage, Pipeline can specify a `batch_size` and use iterable objects as input. However, one of the current criteria for determining iterability is to check if the `inputs` is a `Generator`.

Some code snippets around [base.py#L1213](https://github.com/huggingface/transformers/blob/9fe3f585bb4ea29f209dc705d269fbe292e1128f/src/transformers/pipelines/base.py#L1213)

```python
        is_dataset = Dataset is not None and isinstance(inputs, Dataset)
        is_generator = isinstance(inputs, types.GeneratorType)
        is_list = isinstance(inputs, list)

        is_iterable = is_dataset or is_generator or is_list
```

Therefore, when I need to provide input that was originally a list in an iterable manner, I must use a generator expression instead of directly using the `iter` function. For example:

```python
inputs = ["hello world", "good job"]

# fe_pipeline is a FeatureExtractionPipeline

# works properly
fe_pipline((s for s in inputs), batch_size=32)

# raises a ValueError
fe_pipline(iter(s), batch_size=32)
```

I attempted to look at the source code and didn't find any mandatory use of generators. In Python, `Generator` is a subclass of `Iterator`. Should we consider changing the check here to `Iterator` instead of `Generator`? Or could someone explain why the current check is implemented this way?

Thank you for your attention to this matter.

Comments:
Hi, thanks for raising an issue! 

This is a question best placed in our [forums](https://discuss.huggingface.co/). We try to reserve the github issues for feature requests and bug reports.

Thank you for your reply, I'll close this issue and move it to the forums.
Pass `use_cache` in kwargs for GPTNeoX
Author: zucchini-nlp
Pull request: yes

# What does this PR do?

If we pass in `use_cache` when generating with GPT NeoX in the current main, it will not be used and will be overriden by model config's `use_cache`. This PR passes in user-defined `use_cache` when preparing inputs for generation

Other generative models all support user-defined `use_cache` kwarg already. Tests in `generation/test_utils.py` are passing

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_30538). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
Generate: fix generation with `inputs_embeds` when `input_ids=None` for llama and gemma
Author: njhill
Pull request: yes

The changes in https://github.com/huggingface/transformers/pull/29467 break generation with `inputs_embeds` when `input_ids` is `None` since they expect `input_ids` to be non-`None` even for the prefill forward without `past_key_values`.

@gante 

Comments:
Hi @njhill 👋 

Can you share an example of failure? We have a [test](https://github.com/huggingface/transformers/blob/31c575bcf13c2b85b65d652dd1b5b401f99be999/tests/generation/test_utils.py#L1481) for generation with `inputs_embeds` (which is passing on e.g. Llama), so our test suite is likely incomplete :)

Thanks @gante, this is specifically when using `inputs_embeds` and passing `input_ids` as `None`. I think both are passed to `generate()` in the current test. I'll update the PR title to clarify this.

@njhill uhmmm the test checks that combination as well 🤔 

```py
            # input_ids is not a required input -- if we don't pass it, the newly generated tokens will be the same
            outputs_from_embeds_wo_ids = model.generate(
                inputs_embeds=inputs_embeds, max_new_tokens=20 - inputs_embeds.shape[1]
            )
            self.assertListEqual(
                outputs_from_embeds[:, inputs_embeds.shape[1] :].tolist(),
                outputs_from_embeds_wo_ids.tolist(),
            )
```

(you can run the test with `py.test tests/models/llama/test_modeling_llama.py -k test_generate_from_inputs_embeds_decoder_only`)

This means that there is probably something else going on, which could be interesting to pin down :)

@gante ah apologies for not looking at those closely enough, and thank you for the tip of how to run. Let me dig in deeper to see what's going on here.

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.
Fix causal mask in llama for long seq_length
Author: YLGH
Pull request: yes

# What does this PR do?

Fix a bug in Llama causal mask creation for when input length > causal_mask.shape (which comes from max_position_embeddings). 


## Who can review?

Anyone in the community is free to review the PR once the tests have passed. Feel free to tag
members/contributors who may be interested in your PR.

@ArthurZucker @younesbelkada 



Comments:
Will this PR fix #29252 ? (Yi has same architecture as Llama2)

> Will this PR fix #29252 ? (Yi has same architecture as Llama2)

Yeah I think so, (it uses LlamaForCausalLM)

I'm not sure why unit tests are failing, they seem unrelated to my change...

It would also allow one to run `miqu-1-70b` on an RTX 3090 in 2 bits. The model itself is ~19Gb but those masks are too large (around a Gb each and copied sometimes) at `32k` max context length, preventing one from running the model.

fixed by #29753 !

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.
fix for itemsize => element_size() for torch backwards compat
Author: winglian
Pull request: yes

This is a similar fix to https://github.com/huggingface/peft/pull/1630.

`.itemsize` on a tensor is only supported on torch>=2.1

Fixes: https://github.com/huggingface/transformers/issues/30304

*Edit by @younesbelkada*:

This PR makes sure the checks are compatible with earlier versions of pytorch that have been overlooked by myself in https://github.com/huggingface/transformers/pull/30162

Comments:
@pacman100 @BenjaminBossan 

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_30133). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

Note that this PR [fails](https://github.com/huggingface/peft/pull/1630#issuecomment-2044440626) on PEFT CI.

my last change isn't correct either. I'll take a look at this tomorrow.

Hi @winglian , as pointed out by @hiyouga my fix did not cover all the cases, would you be happy to rebase your PR with main and we merge it ?
[Do not merge] Test build CI docs
Author: Wauplin
Pull request: yes

do not merge

Related to https://github.com/huggingface/doc-builder/pull/488

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_29912). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

Dropping now that https://github.com/huggingface/doc-builder/pull/488 has been merged :)
`StaticCache` Bad generation results with Llama after v4.39.0 
Author: mobicham
Pull request: no
Labels: name: Generation

### System Info

transformers version: 4.41.0.dev0
Platform: Linux-5.15.0-89-generic-x86_64-with-glibc2.
Python version: 3.10.
Huggingface_hub version: 0.20.
Safetensors version: 0.4.
Accelerate version: 0.21.0


### Who can help?

@ArthurZucker @gante

### Information

- [ ] The official example scripts
- [X] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [X] My own task or dataset (give details below)

### Reproduction

The generation output quality with the current 4.41.0.dev0 version is very bad compared to the previous 4.39.0 version, at least with Llama. With quantized models, it outputs complete gibberish. The same code works totally fine with 4.39.0

```Python
import torch, os
from transformers import AutoModelForCausalLM, AutoTokenizer

model_id  = "meta-llama/Llama-2-7b-chat-hf"
model     = AutoModelForCausalLM.from_pretrained(model_id, cache_dir='.', torch_dtype=torch.float16, attn_implementation="sdpa").to('cuda').eval();
tokenizer = AutoTokenizer.from_pretrained(model_id, cache_dir='.') 

tokenizer.add_bos_token = False
tokenizer.add_eos_token = False

prompt = "

 How do I build a car? 
"

gen_out = model.generate(**tokenizer([prompt], return_tensors="pt").to(model.device), do_sample=False, 
                                                cache_implementation="static", max_new_tokens=100, pad_token_id=tokenizer.eos_token_id, 
                                                temperature=None, top_p=None, use_cache=False)

print()
print(tokenizer.decode(gen_out[0]))
```

```
# version: 4.39.0 - works as expected

 
 How do I build a car? 
  Building a car is a complex and challenging project that requires a significant amount of time, money, and expertise. Here are some general steps that you might consider when building a car:

1. Define your goals: What kind of car do you want to build? What features do you want to include? What is your budget? Answering these questions will help you determine the scope of your project and what you need to do to get started.
2. Research and plan:
```

```
# version: 4.41.0 - bad output, outputs gibberish 

 
 How do I build a car? 
  I's (the 0-2) are dots d's the traveling 4 v5 8 out the9 of the9 1t 1 ch do not always and the9 10 11 is-not 1 rt 1 c0 the0.

To build a car, you will need to have a good understanding of mechanical systems, electrical systems, and fabrication techniques. You will also need to have a
```



### Expected behavior

The output should be the same as with the previous 4.39.0 version

Comments:
Mmm that's super weird, it's most probably generate as the `test_torch_compile` here is all green https://github.com/huggingface/transformers/blob/7bfe57777df3836b28198e47e5846cebf4a72591/tests/models/llama/test_modeling_llama.py#L686 

having a look