noamaanMulla-03/transformers-issues huggingface dataset:
remote tokenizers trust remote code prompt doesn't not work as expected
Author: mzbac
Pull request: no

### System Info

transformers: 4.36.2


### Who can help?

@ArthurZucker

### Information

- [X] The official example scripts
- [X] My own modified scripts

### Tasks

- [X] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

1. use `AutoTokenizer.from_pretrained('Qwen/Qwen-1_8B')
2. The tokenizer failed to load and threw an error. The tokenizer class was not found, and it didn't prompt the user to allow trust for remote code.
3. Delete the `tokenizer_class` setting in [config.jso](https://huggingface.co/Qwen/Qwen-1_8B/blob/main/config.json#L30)n and [tokenizer_config.json](https://huggingface.co/Qwen/Qwen-1_8B/blob/main/tokenizer_config.json)
4. After that, when using `AutoTokenizer.from_pretrained('Qwen/Qwen-1_8B')`, it prompts the user to trust remote code. However, instead of asking once, it prompts the user to confirm three times.

### Expected behavior

The `AutoTokenizer.from_pretrained` function should prompt the user whether they want to enable trust remote code only once when the user did not pass the `trust_remote_code` parameter.

Comments:
Hey, `AutoTokenizer.from_pretrained('Qwen/Qwen-1_8B', trust_remote_code = True)` should be used as this model is not natively supported

Otherwise, yes we should prompt the user for that cc @Rocketknight1 if you can have a look! 

> Hey, `AutoTokenizer.from_pretrained('Qwen/Qwen-1_8B', trust_remote_code = True)` should be used as this model is not natively supported

@ArthurZucker Thanks for the reply. Yeah, explicitly passing trust_remote_code works. However, it should prompt the user in terminal to ask whether they want to trust the remote code if they didn't explicitly pass trust_remote_code. As I understand it, it should behave similarly to AutoModelForCausalLM.from_pretrained.

Yes agree with you! 

Hi @mzbac, I made a PR last week that specifically deals with this issue here: #28419.

Can you install `transformers` from `main` with `pip install --upgrade git+https://github.com/huggingface/transformers.git` and then try just running `AutoTokenizer.from_pretrained('Qwen/Qwen-1_8B')` again, without needing to edit any repo files? It should work properly now!

Thanks, @Rocketknight1. I can confirm that using the main branch's transformers resolves the issue.

Cool! That fix will be included in our next release. I'm going to close this issue for now, but feel free to reopen it if you encounter any other issues with loading custom code tokenizers.
[GPTQ] Fix test
Author: SunMarc
Pull request: yes

# What does this PR do ? 
This PR fixes failing tests related to GPTQ quantization. The breaking tests are related to modification on optimum side and OOM from the new runner. I've also replaced for a smaller model. related optimum [PR](https://github.com/huggingface/optimum/pull/1574/files#)

Comments:
@SunMarc Do you have permissions to merge? If not, I can merge this in if it's good to go

I'll merge it ! thx for the reminder
Intel/dpt-swinv2: TypeError: unsupported operand type(s) for //: 'NoneType' and 'NoneType' 
Author: kadirnar
Pull request: no

### System Info

- `transformers` version: 4.35.2
- Platform: Windows-10-10.0.22631-SP0
- Python version: 3.8.18
- Huggingface_hub version: 0.19.4
- Safetensors version: 0.4.1
- Accelerate version: 0.25.0
- Accelerate config: 	not found
- PyTorch version (GPU?): 2.1.1+cu118 (True)
- Tensorflow version (GPU?): not installed (NA)
- Flax version (CPU?/GPU?/TPU?): not installed (NA)
- Jax version: not installed
- JaxLib version: not installed
- Using GPU in script?: <fill in>
- Using distributed or parallel set-up in script?: <fill in>

### Who can help?

@Narsil @amyeroberts

### Information

- [X] The official example scripts
- [ ] My own modified scripts

### Tasks

- [X] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

```
from transformers import pipeline

pipe = pipeline(task="depth-estimation", model="Intel/dpt-swinv2-large-384")
result = pipe("http://images.cocodataset.org/val2017/000000039769.jpg")
result["depth"]
```

Error Message:
```
image_size = image_size if isinstance(image_size, collections.abc.Iterable) else (image_size, image_size)
patch_size = patch_size if isinstance(patch_size, collections.abc.Iterable) else (patch_size, patch_size)
num_patches = (image_size[1] // patch_size[1]) * (image_size[0] // patch_size[0])
TypeError: unsupported operand type(s) for //: 'NoneType' and 'NoneType'
```

### Expected behavior

I want to test dpt-brit-large-384 models.
Model Page: https://huggingface.co/Intel/dpt-swinv2-large-384

Comments:
This model doesn't work either.

https://huggingface.co/Intel/dpt-beit-base-384

cc @NielsRogge as well 

Hi,

Looks like you may need to update your Transformers version. Works for me on main

> Hi,
> 
> Looks like you may need to update your Transformers version. Works for me on main

I updated the package version.

- `transformers` version: 4.36.2
- Platform: Windows-10-10.0.22631-SP0
- Python version: 3.8.18
- Huggingface_hub version: 0.19.4
- Safetensors version: 0.4.1
- Accelerate version: 0.25.0
- Accelerate config:    not found
- PyTorch version (GPU?): 2.1.1+cu118 (True)
- Tensorflow version (GPU?): not installed (NA)
- Flax version (CPU?/GPU?/TPU?): not installed (NA)
- Jax version: not installed
- JaxLib version: not installed
- Using GPU in script?: <fill in>
- Using distributed or parallel set-up in script?: <fill in>

The "Intel/dpt-beit-base-384" model works, but the "swinv2" model doesn't.

<img width="1619" alt="image" src="https://github.com/huggingface/transformers/assets/36204372/98ea8310-8746-4180-a696-71ccd94d87d2">



Ok, that's because https://github.com/huggingface/transformers/pull/27742 was merged 3 weeks ago, which probably was not part yet of 4.36.2. Hence this one will be part of the next Transformers release.
Add a use_safetensors arg to TFPreTrainedModel.from_pretrained()
Author: Rocketknight1
Pull request: yes

PyTorch's `from_pretrained()` method has a `use_safetensors` argument. Our TF code doesn't, and just always tries safetensors if available. This PR adds the argument to match the PyTorch API.

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_28511). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
Generate: consolidate output classes
Author: gante
Pull request: yes

# What does this PR do?

Does some cleanup that's been on my mind for a while 🧹

We had a bunch of classes that were a copy of each other, named after each internal generation method. This PR consolidates them. As a result, the documentation becomes more concise and with less risk of suffering from incomplete updates 🤗 

Full retrocompatibility is kept (and tested)!

Comments:
FuyuProcessor broken and causes infinite loop
Author: grahamannett
Pull request: no
Labels: name: Good Difficult Issue

https://github.com/huggingface/transformers/blob/75336c17945c6b1b5552dbf0236d25f869168aab/src/transformers/models/fuyu/processing_fuyu.py#L618

I am not sure exactly why/how it is but localized an issue where this segment of the processor goes into an infinite loop.  
Could be another segment but if I add a timeout to `post_process_box_coordinates` it seems to point to this.

Comments:
if anyone runs into this, its because of the `if end != start` then `continue` will just keep going.  both these functions should be rewritten 

cc @molbap for visibility, and @grahamannett if you can share a full reproducer would be helpful! 

@ArthurZucker its hard to reproduce since the encoding throws an error if you try to create a similar input but here:

```python
processor = AutoProcessor.from_pretrained("adept/fuyu-8b")
x = torch.tensor([[71011,  71019,    1, 138997,  75694,  71374,  71118,  70004,  70050, 70013,  70004,  70013, 71119,  71122]])
output = processor.post_process_box_coordinates(x)
```

the tokens are supposed to resemble something like the model outputting 'here is the example <box>1, 2, 11, 12, 13</box>'  (or something similar think x in my example is 2's + 21).

I closed the issue tho so think we are all good 👍

Yep thanks I was able to reproduce. It's a bit of an edge case, but we could / should have a check for this. Do you want to open a PR for that? 🤗 

@ArthurZucker Its not really an edge case as the generated output from the model can actually generate this sort of output and anecdotally seems like it becomes much more common if you further finetune the model on OCR related tasks/instructions. 

I do not think the PR I make will be merged so not sure if I should PR (hence closed this issue).

If this should be fix, we'll merge the PR that fixed it! We'll iterate together on the PR if you want! 
Opening this again and marking as good difficult issue 

I think a lot of the current FuyuProcessor and FuyuImageProcessor (the image processor preprocess/preprocess_with_tokenizer_info I believe has some big issues and unless I am told it is how they trained it, it will make fine-tuning with images not really work) are broken/not working correctly. 

I am just going to close this as it seems like the best way to fix this would be to use a different implementation.

It is how it was trained / the code that was shared by the authors 🤗 

@ArthurZucker interesting.  There are a lot of parts in it that seem to to not work/be confusing to me e.g. how `get_sample_encoding` https://github.com/huggingface/transformers/blob/7e0ddf89f483f53107870cddabb2e1cc93069705/src/transformers/models/fuyu/processing_fuyu.py#L385 is done makes me think that because of https://github.com/huggingface/transformers/blob/7e0ddf89f483f53107870cddabb2e1cc93069705/src/transformers/models/fuyu/processing_fuyu.py#L246 the processor is not really able to be used for training? Or why `_tokenize_prompts_with_image_and_batch` adds `"|ENDOFTEXT|"` token to a text prompt even though it then is removed by just using the original prompt size?
add support for custom pipeline
Author: not-lain
Pull request: no
Labels: name: Feature request

### Feature request

is it possible to add a support for custom pipelines.
something like this 
```
| - config.json
| - custom_config.py
| - custom_architecture.py
| - custom_pipeline.py
```

### Motivation

i was following this documentation https://huggingface.co/docs/transformers/add_new_pipeline and i can't make it workout 
it's also pointing out at the following custom pipeline `sgugger/finetuned-bert-mrpc` which i checked all the previous commits and nothing seems to workout.
also it should be a good idea since adding support to all the models directly to the transformers library will bloat the library so i hope this might be taken into consideration.

### Your contribution

i will help out if possible. 

Comments:
also it would be cool to use something like this : 
```python
# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("custom", model="model_id", trust_remote_code=True)
```

Hi @not-lain, thanks for raising this! 

You might be interested in [using tools](https://huggingface.co/docs/transformers/transformers_agents#tools), which you can [define and use from the hub](https://huggingface.co/docs/transformers/custom_tools). 

With regards to things not working, could you open a new issue following the [issue template](https://github.com/huggingface/transformers/blob/main/.github/ISSUE_TEMPLATE/bug-report.yml) including details about the running environment and the errors encountered?  

@amyeroberts although it is interesting to find about tools but what i'm trying to adress that 
* pipeline doesn't work for custom model architectures on the hub [ feature requested] 
* the documentation says you can only work with a new pipeline if you add it to the transformers github repo (which will bloat the library on the long run)

@amyeroberts, i found the mistake 
i needed to call the model as : 
```python
from transformers import pipeline

classifier = pipeline(model="sgugger/test-dynamic-pipeline", trust_remote_code=True)
```
not as:
```python
# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("text-classification", model="sgugger/test-dynamic-pipeline")
```
i will close this issue
Flax generate sampling does not return EOS
Author: borisdayma
Pull request: no

### System Info

When calling `flax_model.generate(do_sample=True)`, we can notice that the EOS token has been replaced with PAD token.

Configuration:
```
- `transformers` version: 4.37.0.dev0
- Platform: Linux-5.4.0-1043-gcp-x86_64-with-glibc2.31
- Python version: 3.10.12
- Huggingface_hub version: 0.20.1
- Safetensors version: 0.4.0
- Accelerate version: not installed
- Accelerate config: not found
- PyTorch version (GPU?): 2.1.1+cu121 (False)
- Tensorflow version (GPU?): 2.12.0 (False)
- Flax version (CPU?/GPU?/TPU?): 0.7.4 (tpu)
- Jax version: 0.4.14
- JaxLib version: 0.4.14
- Using GPU in script?: Noe
- Using distributed or parallel set-up in script?: No
```

### Who can help?

@sanchit-gandhi @gante 

### Information

- [X] The official example scripts
- [ ] My own modified scripts

### Tasks

- [X] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

1. Use `FlaxBartForConditionalGeneration`
2. Change generate method to `do_sample=True, num_beams=1`
3. When decoding, display all special tokens

Here is adapted script:
```
from transformers import AutoTokenizer, FlaxBartForConditionalGeneration

model = FlaxBartForConditionalGeneration.from_pretrained("facebook/bart-large-cnn")
tokenizer = AutoTokenizer.from_pretrained("facebook/bart-large-cnn")

ARTICLE_TO_SUMMARIZE = "My friends are cool but they eat too many carbs."
inputs = tokenizer([ARTICLE_TO_SUMMARIZE], max_length=1024, return_tensors="np")

# Generate Summary
summary_ids = model.generate(inputs["input_ids"], do_sample=True, num_beams=1).sequences
print(tokenizer.batch_decode(summary_ids, skip_special_tokens=False, clean_up_tokenization_spaces=False))
```

The output does not contain EOS `
` at the end before PAD tokens `<pad>`:
`["


My friends are cool but they eat too many carbs. I love carbs. The carb-averse among us have a love for dessert on this one. I've got some amazing desserts. I've been a junkie for 10 years. I can't go straight after dinner, I've been eating for 4. I love it. That's why I made the TV show.<pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad>"]`

The same issue happens with any decoder model.

### Expected behavior

EOS token should appear before the sequence of PAD tokens.

Comments:
improve dev setup comments and hints
Author: 4imothy
Pull request: yes

# What does this PR do?

Changes 'pip install -e .[dev]' -> \`pip install -e '.[dev]'\` in multiple comments and hints. 

New command runs on both *zsh* and *bash*, previously it did not work on *zsh*.

## Before submitting
- [X] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [X] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#start-contributing-pull-requests),
      Pull Request section?
- [ ] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [ ] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/main/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/main/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests?


## Who can review?

Documentation: @stevhliu and @MKhalusova

Comments:
For the failing tests, you'll need to update the expected strings.
SiLU activation wrapper for safe importing
Author: amyeroberts
Pull request: yes

# What does this PR do?

The custom implementation of `SiLUActivation` was removed in #27136. This causes two issues: 

1. Users unable to unpickle objects - c.f. #28177 
2. Users unable to import the class - c.f. #28496 

For 1. - the unpickling of modified transformers models through torch.load isn't something we officially support. However, this will (temporarily) provide an equivalent class. 

For 2 - provides a class with deprecation warning that can be imported 

Fixes #28177 #28496 


## Before submitting
- [ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [x] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#start-contributing-pull-requests),
      Pull Request section?
- [ ] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [ ] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/main/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/main/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests?

Comments:
@ydshieh The same capturelogger errors arose again. A cheeky re-run (with ssh) made them pass, so merging as they're not related to this PR
Support `DeepSpeed` when using auto find batch size
Author: muellerzr
Pull request: yes

# What does this PR do?

This PR addresses https://github.com/huggingface/transformers/issues/24558 by letting the `Trainer` modify the deepspeed plugin *specifically when using auto batch size finder*. 

It refactors the logic for propagation of the deepspeed arguments into its own function so that on the fly we can modify any related to the train batch size if needed. 

Fixes # (issue)

Fixes https://github.com/huggingface/transformers/issues/24558


## Before submitting
- [ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [x] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#start-contributing-pull-requests),
      Pull Request section?
- [ ] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [ ] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/main/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/main/docs#writing-source-documentation).
- [x] Did you write any new necessary tests?


## Who can review?

Anyone in the community is free to review the PR once the tests have passed. Feel free to tag
members/contributors who may be interested in your PR.

@pacman100 


Comments:
I tried the latest main as of today which should include this commit and unfortunately it still crashes
```
  File "/data/v/ft/lib/python3.10/site-packages/accelerate/accelerator.py", line 1958, in backward
    self.deepspeed_engine_wrapped.backward(loss, **kwargs)
AttributeError: 'NoneType' object has no attribute 'backward'
```

---

```
transformers[accelerate,deepspeed,sentencepiece,tokenizers] @ git+https://github.com/huggingface/transformers@edb170238febf7fc3e3278ed5b9ca0b2c40c70e3
safetensors==0.4.1
accelerate==0.26.1
deepspeed @ git+https://github.com/microsoft/DeepSpeed@a85b6e472534d2e0b61fe234fae4f6a2332c95bf
```

---

I think the root cause highlighted here still applies: https://github.com/huggingface/transformers/issues/24558#issuecomment-1850723229
Improving Training Performance and Scalability Documentation
Author: HamzaFB
Pull request: yes

This PR improves the docs.
A strategy for improving Memory Performance for Large Models (Billions of parameters) is PEFT.
Actual documentation does not mention it.
This PR adds PEFT and provides an example as to why this reduces memory needs.

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_28497). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

Thanks!
Exception in inference when using the pipeline with output_scores=True to get logits
Author: andersonm-ibm
Pull request: no

### System Info

- `transformers` version: 4.29.1
- Python version: 3.10.13
- Huggingface_hub version: 0.20.2
- Safetensors version: 0.3.1
- PyTorch version (GPU?): 2.0.0+cu117 (True)


### Who can help?

 @Narsil

### Information

- [ ] The official example scripts
- [X] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [X] My own task or dataset (give details below)

### Reproduction

Easily reproduced by [using the pipeline with output_scores=True](https://gist.github.com/andersonm-ibm/d8baeea66afca89cefebc108f1ce08f3)

Results in:

```
test_transformers_bug.py:27: in <module>
    for out in pipe(KeyDataset(dataset, "text")):
/home/mayaa/miniconda3/envs/fme/lib/python3.10/site-packages/transformers/pipelines/pt_utils.py:124: in __next__
    item = next(self.iterator)
/home/mayaa/miniconda3/envs/fme/lib/python3.10/site-packages/transformers/pipelines/pt_utils.py:125: in __next__
    processed = self.infer(item, **self.params)
/home/mayaa/miniconda3/envs/fme/lib/python3.10/site-packages/transformers/pipelines/base.py:1025: in forward
    model_outputs = self._forward(model_inputs, **forward_params)
/home/mayaa/miniconda3/envs/fme/lib/python3.10/site-packages/transformers/pipelines/text_generation.py:264: in _forward
    out_b = generated_sequence.shape[0]
E   AttributeError: 'GreedySearchEncoderDecoderOutput' object has no attribute 'shape'
```

### Expected behavior

Pipeline should handle the case where the model output is a `GreedySearchEncoderDecoderOutput `and not a simple tensor without exceptions, like in [the example](https://gist.github.com/andersonm-ibm/766d4892c92310a7889b2b3dfdc8ff44#file-model_generate_with_logits_output-py) .

Comments:
@amyeroberts  For reference.

This is totally OK, pipelines are not supposed to support every option models do. Use the example code if you want more control over the generation.

Closing for now since this is normal behavior.