noamaanMulla-03/transformers-issues huggingface dataset:
Fix code snippet for Grounding DINO
Author: qubvel
Pull request: yes

# What does this PR do?

Fix the code snippet for Grounding DINO in docs

<!--
Congratulations! You've made it this far! You're not quite done yet though.

Once merged, your PR is going to appear in the release notes with the title you set, so make sure it's a great title that fully reflects the extent of your awesome contribution.

Then, please replace this with a description of the change and which issue is fixed (if applicable). Please also include relevant motivation and context. List any dependencies (if any) that are required for this change.

Once you're done, someone will review your PR shortly (see the section "Who can review?" below to tag some potential reviewers). They may suggest changes to make the code even better. If no one reviewed your PR after a week has passed, don't hesitate to post a new comment @-mentioning the same persons---sometimes notifications get lost.
-->

<!-- Remove if not applicable -->



## Before submitting
- [x] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).

## Who can review?

Anyone in the community is free to review the PR once the tests have passed. Feel free to tag
members/contributors who may be interested in your PR.

<!-- Your PR will be replied to more quickly if you can figure out the right person to tag with @

 If you know how to use git blame, that is the easiest way, otherwise, here is a rough guide of **who to tag**.
 Please tag fewer than 3 people.

Models:

- text models: @ArthurZucker
- vision models: @amyeroberts
- speech models: @sanchit-gandhi
- graph models: @clefourrier

Library:

- flax: @sanchit-gandhi
- generate: @zucchini-nlp (visual-language models) or @gante (all others)
- pipelines: @Narsil
- tensorflow: @gante and @Rocketknight1
- tokenizers: @ArthurZucker
- trainer: @muellerzr and @SunMarc

Integrations:

- deepspeed: HF Trainer/Accelerate: @muellerzr
- ray/raytune: @richardliaw, @amogkam
- Big Model Inference: @SunMarc
- quantization (bitsandbytes, autogpt): @SunMarc

Documentation: @stevhliu

HF projects:

- accelerate: [different repo](https://github.com/huggingface/accelerate)
- datasets: [different repo](https://github.com/huggingface/datasets)
- diffusers: [different repo](https://github.com/huggingface/diffusers)
- rust tokenizers: [different repo](https://github.com/huggingface/tokenizers)

Maintained examples (not research project or legacy):

- Flax: @sanchit-gandhi
- PyTorch: See Models above and tag the person corresponding to the modality of the example.
- TensorFlow: @Rocketknight1

 -->


Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_32229). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
RAG-2nd2end-revamp
Author: shamanez
Pull request: yes

same as [shamanez:rag-retriever-end2end](https://github.com/huggingface/transformers/pull/11655) PR. In the previous version, I got some version control problems.

Really sorry to come up with duplicate PRs :(.   

@lhoestq  @patrickvonplaten 

I conducted an experiment to check the difference and added a simple test run code. 


Finally I added two test functions to test_modeling_rag.py and test_retrieval_rag.py.  

![image](https://user-images.githubusercontent.com/16892570/119752765-ad5ed080-bef1-11eb-9579-28d3b7477446.png)



Comments:
I added all the minor changes and I would like to thank @patrickvonplaten and @lhoestq for the enormous amounts of support and advice.  :)

 Hey 
Thanks for giving this end to end version. Am trying to run it and see its performance on my own dataset (much smaller and domain specific to see if get performance gains with end to end) but at the moment it is throwing a Pickling error with dummy dataset in the code. Am still stuck trying to understand how to fix this. Any idea how this can be dealt with

  
      File "/home/shraeyb/anaconda3/envs/new_st/lib/python3.7/site-packages/ray/cloudpickle/cloudpickle_fast.py", line 580, in dump
        return Pickler.dump(self, obj)
      File "pyarrow/io.pxi", line 1021, in pyarrow.lib.Buffer.__reduce_ex__
    AttributeError: module 'pickle' has no attribute 'PickleBuffer

Cheers


Could you please add this as a new issue. Also I would like to see the
entire error log. Seems like something is wrong with your RAY installation.

On Fri, Jun 4, 2021, 00:49 Shraey Bhatia ***@***.***> wrote:

> Hey
> Thanks for giving this end to end version. Am trying to run it and see its
> performance on my own dataset (much smaller and domain specific to see if
> get performance gains with end to end) but at the moment it is throwing a
> Pickling error with dummy dataset in the code. Am still stuck trying to
> understand how to fix this. Any idea how this can be dealt with
>
>   File "/home/shraeyb/anaconda3/envs/new_st/lib/python3.7/site-packages/ray/cloudpickle/cloudpickle_fast.py", line 580, in dump
>     return Pickler.dump(self, obj)
>   File "pyarrow/io.pxi", line 1021, in pyarrow.lib.Buffer.__reduce_ex__
> AttributeError: module 'pickle' has no attribute 'PickleBuffer
>
> Cheers
>
> —
> You are receiving this because you were mentioned.
> Reply to this email directly, view it on GitHub
> <https://github.com/huggingface/transformers/pull/11893#issuecomment-853842534>,
> or unsubscribe
> <https://github.com/notifications/unsubscribe-auth/AEA4FGWLRNB4P234HGI4JOLTQ5255ANCNFSM45TDZJNA>
> .
>


Sure, just created a new issue with full log

@shamanez looks like the Google Drive link you provided in the `README.md` is dead now, I get an error. Can you please take a look?
Generation: stop at `eos` for assisted decoding
Author: zucchini-nlp
Pull request: yes

# What does this PR do?

Addresses the issue in [this comment](https://github.com/huggingface/transformers/pull/27775#issuecomment-2151908426). 

The problem was that assisted generation is flaky when it comes to EOS tokens. If the assistant generates target model's `eos` token in the middle of candidate sequence and it get accepted by the target, we don't stop generating because the criteria checks only the last token. This happens mostly for Prompt-Lookup decoding because it's simple copy-paste from history.

One possible solution is to check `eos` in all of the newly generated input ids, another is to crop candidates if there's `eos`.  I went the second way. 

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_31301). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.

Latest release doesn't have a fix for this, it's still an issue


@gante okay, moved the logic into PLD. The main concern was that assistant model can have it's eos token to be different from target model, so I added a line in AssistedGenerator to set eos token. 

Added a test for PLD, but not for assistant model because we can't control what the assistant model generates. Also tested with the case from the linked issue, is working on my end

Please note this will error out if the eos_token_id is a list (like in some llama3 configurations)
![image](https://github.com/user-attachments/assets/c88787ab-5a75-4f9e-8620-73abd59f28fd)

@Ednaordinary are you using the `generate()` method? I don't see and error using current branch, because internally we convert all special tokens to tensors and `self.eos_id` cannot be a list.

I guess you're using `main` and applied current changes above, in that case it will fail because I didnt yet rebase. There were some changes for special tokens lately afaik. I will make sure it works before merging

In some configurations running llama, the eos_token_id is set to a list.
![image](https://github.com/user-attachments/assets/0f04508f-34f8-4e84-a6d3-827173eec32d)
this works fine for sampling until prompt lookup decoding is added. It's a simple fix, though, adding [0] to the end of tokenizer.encode("
"). Works fine after applying this PR to current main
I mainly raise this concern because llama 3.1 has three stop tokens:
![image](https://github.com/user-attachments/assets/2a310d49-de8a-4643-aff4-69da6188e208)


@Ednaordinary I see, you're using llama-3.1 that forces you to use the latest version. As I said, the problem was that current branch is behind `main` for months. Now I rebased and everything should be fine
Llava: generate without images
Author: zucchini-nlp
Pull request: yes

# What does this PR do?

Llava, in contrast to other models from llava series, was returning `None` as pixel values when images are not passed. That caused errors when users try to generate on text-only setting, because we cannot put `None` on "cuda". 

This PR fixes it. I don't think we need a test for it, as other llavas don't test it. But I can add a test in all llava-based models, if you think it's required

Comments:
Yes, I was also thinking about a `MultiModalTesterMixin`. Oke, let me add the tests in each file for now, and think about what should go in multimodal tests a bit later. 

Since we are in the process of removing custom generate code, I might not be able to add such a mixin fast. Actually it should be a natural extension of `GenerationTesterMixin` with tests on tiny model

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_32183). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

> Since we are in the process of removing custom generate code, I might not be able to add such a mixin fast. Actually it should be a natural extension of GenerationTesterMixin with tests on tiny model

sounds good. it's OK if it takes a bit of time for us to figure out the best structure for this. I'm not too worried about things being silently broken at the moment, but it would be good to have this checked automatically for us
Support `from_pretrained` of `FlaxPretrainedModel` from sharded `.safetensors` weights
Author: dest1n1s
Pull request: no
Labels: name: Feature request

### Feature request

Currently `FlaxPretrainedModel` only supports loading pretrained models from sharded PyTorch weights or single-file `.safetensors`. It's worth adding support for loading sharded `.safetensors`.

### Motivation

Recent open-source language models trained with PyTorch are likely only to release sharded `.safetensors` weights. The lack of support for loading from the dominating paradigm makes it troublesome to use these models in Jax.

### Your contribution

I'm relatively new to the implementation of the saving & loading mechanism of `transformers`, but I'd love to try to work with this feature if the core team doesn't have enough bandwidth.

Comments:
Thanks for the offer @dest1n1s! It would be nice to have support for this in Flax, indeed. Please feel free to open a PR to offer that!

cc @sanchit-gandhi if you can review once the PR is open.
don't log base model architecture in wandb if log model is false
Author: joaonadkarni
Pull request: yes

# What does this PR do?

Currently wandb is always logging the base model architecture even when the envvar for logging the model is not set. 

This seems like a weird behavior, I would expect that disabling the wandb envvar for logging the model would disabled all logged artifacts associated with the model, but that's not happening currently. In this PR I change this behavior and if log model is set to "false" I don't log the initial model architecture artifact


Comments:
cc @parambharat  (since this was initially added by you)

cc  @amyeroberts for review (since you were the reviewer for the PR that added this https://github.com/huggingface/transformers/pull/30135)

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_32143). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
Refactor: Removed un-necessary `object` base class
Author: Sai-Suraj-27
Pull request: yes

# What does this PR do?
Since Python 3, all classes inherit from `object` by default, so `object` can be removed from the list of base classes.

Fixes # (issue)


## Before submitting
- [ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [x] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#create-a-pull-request),
      Pull Request section?
- [ ] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [ ] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/main/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/main/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests?


## Who can review?
@amyeroberts 

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_32230). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
Adding mixtral attention_bias in style of llama modeling
Author: Moreh-LeeJunhyeok
Pull request: no
Labels: name: Feature request

### Feature request

### System Info
transformers version: 4.36.2

### Who can help?
don't have a clue about this

### Information
Refer to llama2 modeling code, I want to add attention bias option in mixtral model and configuration for flexibility of experiments.  
If this changes seems appropriate, I will make a PR for it

### Expected behavior
After changes, attention bias option of model is added in config.  
Can be controlled like example below(default config value is false)
```
from transformers import AutoConfig
config = AutoConfig.from_pretrained("variant_of_mixtral")
config.attention_bias = True
```

### Motivation

Refer to llama2 modeling code, I want to add attention bias option in mixtral model and configuration for flexibility of experiments.  

### Your contribution

I have created a fix branch. I can make a PR of it
refer to [link](https://github.com/Moreh-LeeJunhyeok/transformers/tree/mixtral_add_attention_bias)

Comments:
what is a status of this issue? is this on progress? thanks in advance 

Hi @Moreh-LeeJunhyeok, thanks for opening an issue! 

Note - making experiments easier isn't in and of itself enough of a reason to add something to a model. However, as there was the equivalent added to Llama, this seems reasonable. Could you open a PR and we can review your proposed changes? 

sorry for disturb. Is there any article about the difference in proformance when training Llama with or without attention bias opened?
`target_sizes` in Owlv2 `post_process_image_guided_detection`
Author: catalys1
Pull request: no
Labels: name: Vision

### System Info

N/A

### Who can help?

_No response_

### Information

- [ ] The official example scripts
- [ ] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

In `post_process_image_guided_detection` ([link](https://github.com/huggingface/transformers/blob/main/src/transformers/models/owlv2/image_processing_owlv2.py#L544C9-L544C44)), there's no check for when `target_sizes = None`, so calling it without providing `target_sizes` raises an error.

### Expected behavior

The case where `target_sizes` is `None` should be handled, as done in `post_process_detection` ([link](https://github.com/huggingface/transformers/blob/main/src/transformers/models/owlv2/image_processing_owlv2.py#L484))

Comments:
Hi @catalys1, thanks for raising! Would you like to open a PR to address this? This way you get the github contribution for spotting the error and the fix 

I've submitted a pull request
Cohere: Use `diff` instead of `Copied from` mechanism
Author: younesbelkada
Pull request: yes

# What does this PR do?

Replaces: https://github.com/huggingface/transformers/pull/31211


Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_31518). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.
[tests] fix `static` cache implementation is not compatible with `attn_implementation==flash_attention_2`
Author: faaany
Pull request: yes

## What does this PR do?

```bash
    def forward(
        self,
        hidden_states: torch.Tensor,
        attention_mask: Optional[torch.LongTensor] = None,
        position_ids: Optional[torch.LongTensor] = None,
        past_key_value: Optional[Cache] = None,
        output_attentions: bool = False,
        use_cache: bool = False,
        cache_position: Optional[torch.LongTensor] = None,
    ) -> Tuple[torch.Tensor, Optional[torch.Tensor], Optional[Tuple[torch.Tensor]]]:
        if isinstance(past_key_value, StaticCache):
>           raise ValueError(
                "`static` cache implementation is not compatible with `attn_implementation==flash_attention_2` "
                "make sure to use `sdpa` in the mean time, and open an issue at https://github.com/huggingface/transformers"
            )
E           ValueError: `static` cache implementation is not compatible with `attn_implementation==flash_attention_2` make sure to use `sdpa` in the mean time, and open an issue at https://github.com/huggingface/transformers

src/transformers/models/llama/modeling_llama.py:388: ValueError
```

Comments:
cc @ArthurZucker @gante 

If this failure in the test is expected, I will close this PR. 

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_32039). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

Thanks a lot!
JudgeXL-LLM
Author: GriffinDevelopments
Pull request: no
Labels: name: New model

### Model description

Transformers XL variation called JudgeXL-LLM (for now). I am experimenting with different variations of this model but the goal is to make this model infused with a restaurant program i am developing. It is a text generation/question answering LLM that i would eventually like to teach logic and reasoning. I am still new to the AI/Machine learning world so I will be making adjustments as i experiment with this model. I would like the Original Model name to be JudgeXL-LLM but the model type to be judgellm because i am expecting to make plenty of variations and they may not include the xl part of the code.

### Open source status

- [X] The model implementation is available
- [X] The model weights are available

### Provide useful links for the implementation

https://huggingface.co/Wonder-Griffin/XL-Judge-LLM... that is where i have the model loaded, the rest is all saved on my local device. I am the only one who ahs worked on the code, my name is Morgan Griffin, aka WongrifferousAI, Judgemrogan, Wonder-Griffin

Comments:
JudgeXL-LLM
Author: GriffinDevelopments
Pull request: no
Labels: name: New model

### Model description

Transformers XL variation called JudgeXL-LLM (for now). I am experimenting with different variations of this model but the goal is to make this model infused with a restaurant program i am developing. It is a text generation/question answering LLM that i would eventually like to teach logic and reasoning. I am still new to the AI/Machine learning world so I will be making adjustments as i experiment with this model. I would like the Original Model name to be JudgeXL-LLM but the model type to be judgellm because i am expecting to make plenty of variations and they may not include the xl part of the code.

### Open source status

- [X] The model implementation is available
- [X] The model weights are available

### Provide useful links for the implementation

https://huggingface.co/Wonder-Griffin/XL-Judge-LLM... that is where i have the model loaded, the rest is all saved on my local device. I am the only one who has worked on the code, my name is Morgan Griffin, aka WongrifferousAI, Judgemrogan, Wonder-Griffin

Comments:
ChatGLMForConditionalGeneration does not support Flash Attention 2.0 yet.
Author: liuweie
Pull request: no

Hello，I am using transformers to train glm4, and I try to use flash attention, then the following error occured:
"ChatGLMForConditionalGeneration does not support Flash Attention 2.0 yet."

Comments:
Hi @liuweie,

Yes, the ChatGLM models don't have flash attention implemented yet. You can make a request on the model's checkpoints on the hub for this feature to be added e.g. [like here on the discussions tab](https://huggingface.co/THUDM/chatglm3-6b/discussions) 

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.
an unexpected keyword argument 'output_signature'
Author: grandzhang001
Pull request: no

## Environment info
<!-- You can run the command `transformers-cli env` and copy-and-paste its output below.
     Don't forget to fill out the missing fields in that output! -->

- `transformers` version:4.9.0
- Platform:linux
- Python version:3.6
- PyTorch version (GPU?):1.7
- Tensorflow version (GPU?):2.3,gpu version
- Using GPU in script?:k40m
- Using distributed or parallel set-up in script?:no


when I execute
python run_mlm.py 
--model_name_or_path bert-base-chinese 
--output_dir /data/bert_virtual/modelinfo/ 
--train_file /data/zxdata/000002_0.txt
from examples/tensorflow/language-modeling,this error happens

Traceback (most recent call last):
File "run_mlm.py", line 619, in <module>
main()
File "run_mlm.py", line 543, in main
tf.data.Dataset.from_generator(train_generator, output_signature=train_signature)
TypeError: from_generator() got an unexpected keyword argument 'output_signature'




Comments:
update tensorflow to 2.5 solve this.

Use tf >= v2.4.0 to solve this issue. This was in the release notes.

i solved this error by converting list into tuple like this
def data_generator(descriptions,images,tokenizer,max_length,vocab_size):
    while 1:
        for key,desc_list in descriptions.items():
            image =images[key][0]
            in_img,in_seq,out_word = create_sequences(tokenizer,max_length,desc_list,image,vocab_size)
            yield (in_img,in_seq), out_word