noamaanMulla-03/transformers-issues huggingface dataset:
Add support for `torch_dtype` in the run_mlm example
Author: jla524
Pull request: yes

# What does this PR do?

<!--
Congratulations! You've made it this far! You're not quite done yet though.

Once merged, your PR is going to appear in the release notes with the title you set, so make sure it's a great title that fully reflects the extent of your awesome contribution.

Then, please replace this with a description of the change and which issue is fixed (if applicable). Please also include relevant motivation and context. List any dependencies (if any) that are required for this change.

Once you're done, someone will review your PR shortly (see the section "Who can review?" below to tag some potential reviewers). They may suggest changes to make the code even better. If no one reviewed your PR after a week has passed, don't hesitate to post a new comment @-mentioning the same persons---sometimes notifications get lost.
-->

<!-- Remove if not applicable -->

Fixes #29697 (issue)

- match the `torch_dtype` argument in [run_clm](https://github.com/huggingface/transformers/blob/main/examples/pytorch/language-modeling/run_clm.py)
- `max_seq_length` is very similar to `block_size` and is left unchanged

## Who can review?

@galtay and @amyeroberts

<!-- Your PR will be replied to more quickly if you can figure out the right person to tag with @

 If you know how to use git blame, that is the easiest way, otherwise, here is a rough guide of **who to tag**.
 Please tag fewer than 3 people.

Models:

- text models: @ArthurZucker and @younesbelkada
- vision models: @amyeroberts
- speech models: @sanchit-gandhi
- graph models: @clefourrier

Library:

- flax: @sanchit-gandhi
- generate: @gante
- pipelines: @Narsil
- tensorflow: @gante and @Rocketknight1
- tokenizers: @ArthurZucker
- trainer: @muellerzr and @pacman100

Integrations:

- deepspeed: HF Trainer/Accelerate: @pacman100
- ray/raytune: @richardliaw, @amogkam
- Big Model Inference: @SunMarc
- quantization (bitsandbytes, autogpt): @SunMarc and @younesbelkada

Documentation: @stevhliu and @MKhalusova

HF projects:

- accelerate: [different repo](https://github.com/huggingface/accelerate)
- datasets: [different repo](https://github.com/huggingface/datasets)
- diffusers: [different repo](https://github.com/huggingface/diffusers)
- rust tokenizers: [different repo](https://github.com/huggingface/tokenizers)

Maintained examples (not research project or legacy):

- Flax: @sanchit-gandhi
- PyTorch: See Models above and tag the person corresponding to the modality of the example.
- TensorFlow: @Rocketknight1

 -->


Comments:
show `-rs` to show skip reasons
Author: ArthurZucker
Pull request: yes

# What does this PR do?
Show while tests are skipped

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_30318). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
LLM does not underestand the semantic ordering!
Author: moghadas76
Pull request: no

### Feature request

past_target, past_observed_values, past_time_feat, future_time_feat, future_target_reshaped = map(torch.tensor, [past_target_np,future_target_reshaped_np, future_time_feat_np, past_time_feat_np, past_target_np])
LLM did not understand the postfix np semantic conventions!

### Motivation

LLM should underestand the semantic ordering

### Your contribution

A better architecture

Comments:
Hi @moghadas76, 

Without any more context, it's pretty hard to know what's happening. For example, what model is being talked about? 

Please make sure to follow the [issue template](https://github.com/huggingface/transformers/blob/main/.github/ISSUE_TEMPLATE/bug-report.yml) and provide a series of steps to reproduce, as well as information about the running environment. 

Inferring from the question, it seems this is a question best placed in our [forums](https://discuss.huggingface.co/). We try to reserve the github issues for feature requests and bug reports.

Actually I am planning to PR to new LLM model to fix this. I'll provide more context.

Who should I talk to for technical details?

It's hard to know without knowing what the technical details are. I'd suggest in our [forums](https://discuss.huggingface.co/). We try to reserve the github issues for feature requests and bug reports.

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.
Add inputs embeds in generation
Author: zucchini-nlp
Pull request: yes

# What does this PR do?

Not all decoder-only language models support passing "inputs_embeds" currently. I believe for consistency we can add a possibility. However this PR does not do anything for multimodal LLMs, as those are much more complicated sometimes and there may be no reasons for passing "inputs_embeds" manually.

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_30269). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

(CI failing due to hub timeouts, I consider it to be equivalent to all green)

Comments are addressed and test passing, so merging now
Improve logging (#3)
Author: aymeric-roucher
Pull request: yes
Labels: name: Agents

# What does this PR do?

- Add custom errors: `AgentExecutionError`, `AgentParsingError`, `AgentMaxIterationsError`
- Changes the memory system: instead of a memory that would not keep everything, `agent.logs` now keeps track of everything that happened. And when the LLM needs to access memory, a method generates its memory from the complete logs.

Since this builds upon a previous PR, you can find all relevant changes in [this second commit](https://github.com/huggingface/transformers/pull/30254/commits/4c2994ccac48088860fece57b1e6ca1a0e4d0727)

Comments:
Llama: fix custom 4D masks
Author: gante
Pull request: yes

# What does this PR do?

Fixes the issue raised by @poedator in [this comment](https://github.com/huggingface/transformers/pull/29753#issuecomment-2014530814).

Causal mask is now of shape `[..., seq_len, full_len]`, as opposed to `[..., full_len, full_len]`. This means custom 4D attention masks are now the whole causal mask, so we don't need a sliced copy -- we can copy the whole thing :) 

This PR also expands the support of custom 4D attention mask: we can pass both the full mask (`[..., full_len, full_len]`) or the partial mask (`[..., seq_len, full_len]`). 

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_29930). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

Thanks @gante ! 
I tested the code and it works.
My only suggestion is to cover the 4D masks in the [documentation for Llama](https://huggingface.co/docs/transformers/main/en/model_doc/llama) and other models.

> My only suggestion is to cover the 4D masks in the [documentation for Llama](https://huggingface.co/docs/transformers/main/en/model_doc/llama) and other models.

@poedator would you like to open a PR with that? As a user, you'll probably have cool examples in mind!

> > My only suggestion is to cover the 4D masks in the [documentation for Llama](https://huggingface.co/docs/transformers/main/en/model_doc/llama) and other models.
> 
> @poedator would you like to open a PR with that? As a user, you'll probably have cool examples in mind!

will try, but not this week...

@ArthurZucker ready for a re-review (test rework) -- we now have on push tests for all recent models + custom 4D mask :)

@gante ,
May I suggest adding extra tests for 4D masks with StaticCache? I am concerned that StaticCache code may handle the custom masks differently. I intend to use 4D masks with StaticCache in my new speculative decoding implementation.
Here are the additional methods for `Mask4DTestHard`:
https://gist.github.com/poedator/f1c15551d202df2682c65c1bbdcb1c07

I made the cache longer than the masks and padded the masks to the cache length. is this the correct way?

Sorry for the delay, let's rebase on main as well

please, please merge this PR - I need it for my speculative decoding paper project! The 4D masks are essential for it. 
@gante @ArthurZucker 

Sorry just got back to github 😓 could you rebase! 

> Sorry just got back to github 😓 could you rebase!
Too bad it did not make it into 4.40 (

I rebased this PR in new one #30348 and added few important changes.


@gante, thank you for the rebase. Meanwhile I added more improvements in https://github.com/huggingface/transformers/pull/30348 - let's close this #29930 and continue there.

Closing in favor of #30348
[Grounding DINO] Add support for cross-attention in GroundingDinoMultiHeadAttention
Author: EduardoPach
Pull request: yes

# What does this PR do?

This PR fixes https://github.com/huggingface/transformers/issues/30176 by adding support for cross-attention masking and related integration test.

Now one can do batched inference with different text lengths such as:

```python
import torch
from transformers import AutoModelForZeroShotObjectDetection, AutoProcessor

model_id = "IDEA-Research/grounding-dino-tiny"

model = AutoModelForZeroShotObjectDetection.from_pretrained(model_id)
processor = AutoProcessor.from_pretrained(model_id)

images = [...]
text = ["a cat. a remote control.", "a person. a cat. a remote control. a couch."]

inputs = processor(images=image, text=text, padding="longest", return_tensors="pt")

with torch.no_grad():
    outputs = model(**inputs)
```

Comments:
Thanks for adding @EduardoPach! Let us know when it's ready for review! 

> Thanks for adding @EduardoPach! Let us know when it's ready for review!

It's already ready for review, it was easier than I was expecting
Add Cross Attention Masking for Grounding DINO
Author: EduardoPach
Pull request: no
Labels: name: Vision

### Feature request

In the recent `GroundingDino` PR (https://github.com/huggingface/transformers/pull/26087) one thing that was missing was the cross-attention masking when running Multiheaded attention between text and image features. The assumption was that for inference the text used would probably be a fixed set of labels, thus not needing the cross-attention masking. However, for a more robust and training-friendly implementation, we should support text cross-attention mask.


### Motivation

This was an actual request in the PR of the model from people who are using the `transformers` implementation on their research see https://github.com/huggingface/transformers/pull/26087#issuecomment-2041425195

### Your contribution

A PR should be open soon 😄 

Comments:
c.c. @amyeroberts
remove redundant logging from longformer
Author: riklopfer
Pull request: yes

# What does this PR do?

Longformer `warning_once` call contains pretty unique information. This PR removes some of that info so that the message should actually only be logged once. 


## Who can review?

Anyone in the community is free to review the PR once the tests have passed. Feel free to tag
members/contributors who may be interested in your PR.

@ArthurZucker and @younesbelkada




Comments:
@younesbelkada this isn't really a bug. Just excessive logging IMO. The `warning_once` call caches the messages so only unique messages will be logged. This means that, 1 is padded to 512, 2 is padded to 512, 3 is padded to 512, etc, etc. This change should only log once since it doesn't include the info about the before and after sequence length. 

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_30365). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
Using variables (#4)
Author: aymeric-roucher
Pull request: yes
Labels: name: Agents

# What does this PR do?

Add support of state variables in the ReactAgent, which allows it to store for instance the result of an image generation to re-use it in later processing.

cc @LysandreJik 

Comments:
The relevant commit is [this last one](https://github.com/huggingface/transformers/pull/30297/commits/2364c3bd33953e7992fd9563cc838d7af366232f)
Jax: scipy version pin
Author: gante
Pull request: yes

# What does this PR do?

Pins the scipy version so as to avoid jax import errors. Our pinned jax versions import (e.g.) `scipy.linalg.tril`, which was deprecated -- search for `tril` in [this release page](https://docs.scipy.org/doc/scipy/release/1.11.0-notes.html).

Example of failing import (from a fresh venv, which was getting `scipy==1.13.0`):
```
tests/models/t5/test_modeling_flax_t5.py:45: in <module>
    import optax
../venvs/hf/lib/python3.10/site-packages/optax/__init__.py:18: in <module>
    from optax._src.alias import adabelief
../venvs/hf/lib/python3.10/site-packages/optax/_src/alias.py:24: in <module>
    from optax._src import factorized
../venvs/hf/lib/python3.10/site-packages/optax/_src/factorized.py:27: in <module>
    from optax._src import utils
../venvs/hf/lib/python3.10/site-packages/optax/_src/utils.py:22: in <module>
    import jax.scipy.stats.norm as multivariate_normal                                                                                                ../venvs/hf/lib/python3.10/site-packages/jax/scipy/stats/__init__.py:39: in <module>
    from jax._src.scipy.stats.kde import gaussian_kde as gaussian_kde
../venvs/hf/lib/python3.10/site-packages/jax/_src/scipy/stats/kde.py:26: in <module>
    from jax.scipy import linalg, special
../venvs/hf/lib/python3.10/site-packages/jax/scipy/linalg.py:18: in <module>
    from jax._src.scipy.linalg import (
../venvs/hf/lib/python3.10/site-packages/jax/_src/scipy/linalg.py:408: in <module>
    @_wraps(scipy.linalg.tril)
E   AttributeError: module 'scipy.linalg' has no attribute 'tril'
```

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_30402). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
Fix on "cache position" for assisted generation
Author: zucchini-nlp
Pull request: yes

# What does this PR do?

Fixes #30042  and fixes #30068. As it was reported in the issue, assisted decoding generation does not match greedy. The problem was in assuming there is only one new token generated when "update_kwargs_for_generation". That does not affect other model kwargs, because assisted generation handles them internally. But for "cache position" we need to update it to the new length, which is "accepted num candidates".

I added a small test for assisted generation + cache position, passes for all models where "cache position" is supported. Tested if anything breaks when 0 candidates accepted or when all of them accepted, seems to work for me.

Btw, I want to notice again that making a universal  "update_kwargs_for_generation" would be very nice, so that it does not assume one token being added. I can work on it as a new PR, maybe this can help our future refactoring a bit :)

Update: After a little explorationn I found that using only "update_model_kwargs" to get rid of extra function is feasible, but that needs two separate methods: for pre-fill and for decoding

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_30068). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

Related change: https://github.com/huggingface/transformers/pull/29943/commits/82be569ea0459832fce57ec3de3fc0e1dd29c454

Failing test fixed by https://github.com/huggingface/transformers/pull/30389

@amyeroberts can be re-reviewed, failing tests are due to connection errors and not related to the PR
Assisted generation output differs from greedy output
Author: danielkorat
Pull request: no

### System Info

Ubuntu 20.04.6 LTS
python=3.10.14

transformers=='4.40.0.dev0', installed from source using commit: 517a3e6
accelerate==0.28.0
torch==2.2.2

### Who can help?

@gante @ArthurZucker

### Information

- [ ] The official example scripts
- [X] My own modified scripts

### Tasks

- [X] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

```
import torch
from transformers import AutoModelForCausalLM

m = AutoModelForCausalLM.from_pretrained("lmsys/vicuna-7b-v1.3", device_map="auto")
assistant = AutoModelForCausalLM.from_pretrained("double7/vicuna-68m", device_map="auto")
inp = torch.tensor([[10]], device=target.device)
assert m.generate(inp).tolist() == m.generate(inp, assistant_model=assistant).tolist()
```

### Expected behavior

Outputs should be identical (no assertion).

Comments:
cc @gante, the `generation_config` does not have sampling set, so should be greedy.

The problem seems to be in the way "cache positions" are updated in "assisted decoding". It works right now after a small change in code but I want to see if there are any edge cases. I can open a PR next week after digging further into the issue 😄
[tests] add `require_torch_sdpa` for test that needs sdpa support
Author: faaany
Pull request: yes

## What does this PR do?
I got this test failed in my XPU environment. After debugging, I found that the EXPECTED_LOGITS in this test are computed with the assumption that `_attn_implementation` is set to `sdpa`. The code for setting the `_attn_implementation` are [this line](https://github.com/huggingface/transformers/blob/e74d793a3c3c0bc9bf3fb94bb31dd16934b1b0db/src/transformers/modeling_utils.py#L1464) and [this line](https://github.com/huggingface/transformers/blob/e74d793a3c3c0bc9bf3fb94bb31dd16934b1b0db/src/transformers/modeling_utils.py#L1630). I experimented to downgrade my torch version to make `_attn_implementation` value to eager and it fails on CUDA. So we neeed to add the maker `require_torch_sdpa` to this test. 


@ArthurZucker and @amyeroberts


Comments:
Thanks for handling this and updating!
FIX: re-add bnb on docker image
Author: younesbelkada
Pull request: yes

# What does this PR do?

Addresses: https://github.com/huggingface/transformers/pull/30408#discussion_r1576184407 - some tests require bnb to be run for more memory efficient testing

cc @ydshieh @amyeroberts @SunMarc 


Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_30427). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
Fix LayoutLMv2 init issue and doctest
Author: ydshieh
Pull request: yes

# What does this PR do?

(I think I have done all I can do for doctest - there are 2 other failures where I ping other people)

`LayoutLMv2Model.visual_segment_embedding`'s initialization (when it is not in a checkpoint) won't be reproducible even if seed is set. This is because it is defined

> self.visual_segment_embedding = nn.Parameter(nn.Embedding(1, config.hidden_size).weight[0])

and we have `_fast_init` used in loading +`_init_weights` is not designed to deal with `nn.Parameter`.

This PR fixes the reproducibility issue by initializing `visual_segment_embedding` explicitly.

Remark: such places requiring this fix is very few - so we don't destroy the purpose of having `_fast_init`.

Comments:
The failure in ` tests_pr_documentation_tests` is just missing `detectron2`. I can install it for that job.

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_30278). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
AttributeError: type object 'BertConfig' has no attribute 'pretrained_config_archive_map'
Author: lethienhoa
Pull request: no

Hi,

After changing from `BERT_PRETRAINED_MODEL_ARCHIVE_MAP` to `BERT_PRETRAINED_MODEL_ARCHIVE_LIST` in #5842 , I encountered another issue:

>   File "main.py", line 24, in <genexpr>
>     ALL_MODELS = sum((tuple(conf.pretrained_config_archive_map.keys()) for conf in (BertConfig, XLNetConfig)), ())
> AttributeError: type object 'BertConfig' has no attribute 'pretrained_config_archive_map'

Is it also a breaking change ?
What is the replacing name for `pretrained_config_archive_map` now ?

Thank you,

Comments:
it'd better to downgrade the version: pip install transformer==2.0.0 (which solves the compatible issue).

`$ pip install transformers==2.0.0`

works

Hi, is there any other approach other than downgrading?

try

> `$from transformers import ALBERT_PRETRAINED_CONFIG_ARCHIVE_MAP`

or 
`$from transformers import BERT_PRETRAINED_CONFIG_ARCHIVE_MAP`

Hi! Could you let us know the use cases of using the archive maps?


Hi, 
<img width="712" alt="截屏2021-02-26 下午9 38 01" src="https://user-images.githubusercontent.com/7147150/109307249-49d5f000-787b-11eb-91eb-f076046f180a.png">
I used it to provide the hint of cmd args configuration.

I see, thank you! The issue is that these archive maps are outdated and don't mean much since the model hub.

In that case, it would be better to have a way to see all checkpoints on the hub corresponding to that model architecture, right?

If anyone has already solved the issue please let me know. I am having same issue.
https://hjlabs.in

`$from transformers import ALBERT_PRETRAINED_CONFIG_ARCHIVE_MAP`
this solved my issue. Thank you everyone for support. you are great.
https://hjlabs.in

For legacy codebases, you could store a static tuple of `ALL_MODELS` evaluated from a downgraded `transformers`:

```python
ALL_MODELS = ('bert-base-uncased', 'bert-large-uncased', 'bert-base-cased', 'bert-large-cased', 'bert-base-multilingual-uncased', 'bert-base-multilingual-cased', 'bert-base-chinese', 'bert-base-german-cased', 'bert-large-uncased-whole-word-masking', 'bert-large-cased-whole-word-masking', 'bert-large-uncased-whole-word-masking-finetuned-squad', 'bert-large-cased-whole-word-masking-finetuned-squad', 'bert-base-cased-finetuned-mrpc', 'bert-base-german-dbmdz-cased', 'bert-base-german-dbmdz-uncased', 'bert-base-japanese', 'bert-base-japanese-whole-word-masking', 'bert-base-japanese-char', 'bert-base-japanese-char-whole-word-masking', 'bert-base-finnish-cased-v1', 'bert-base-finnish-uncased-v1', 'roberta-base', 'roberta-large', 'roberta-large-mnli', 'distilroberta-base', 'roberta-base-openai-detector', 'roberta-large-openai-detector', 'albert-base-v1', 'albert-large-v1', 'albert-xlarge-v1', 'albert-xxlarge-v1', 'albert-base-v2', 'albert-large-v2', 'albert-xlarge-v2', 'albert-xxlarge-v2')
```

> transformers==2.0.0

when I download to 2.0.0,the progress pause ，does has other solutions to help it ?
Remove old TF port docs
Author: Rocketknight1
Pull request: yes

This PR removes an old doc on porting TF models from the sidebar that was kinda out of date and not really relevant anymore!

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_30426). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.