noamaanMulla-03/transformers-issues huggingface dataset:
UserWarning: Creating a tensor from a list of numpy.ndarrays is extremely slow
Author: liucw2012
Pull request: no

### System Info

System Info
nvidia CUDA Version: 12.1,  Driver Version: 525.105.17
transformers-cli env is
- `transformers` version: 4.29.2
- Platform: Linux-4.19.87-netease6-1-x86_64-with-glibc2.29
- Python version: 3.8.10
- Huggingface_hub version: 0.14.1
- Safetensors version: not installed
- PyTorch version (GPU?): 2.1.0a0+fe05266 (True)



### Who can help?

_No response_

### Information

- [X] The official example scripts
- [ ] My own modified scripts

### Tasks

- [X] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

when i use tokenizer(...,padding=True), i will get a warnning : "UserWarning: Creating a tensor from a list of numpy.ndarrays is extremely slow. Please consider converting the list to a single numpy.ndarray with numpy.array() before converting to a tensor."  and this function latency is 0.744675874710083  if query_list over 800 words .
```
st = time.time()
inputs = tokenizer(query_list, return_tensors="pt" ,padding=True)
print(time.time() - st)
```


### Expected behavior

Reduce latency and fix the  UserWarning.

Comments:
@liucw2012  Thank you for the opening the issue and PR!

I haven't check the PR in detail, but I am wondering what's the **total time** if you convert it to numpy first + numpy to torch tensor.

Also a remark: when giving a code snippet, please provide all the necessary  variable value so it could run directly.
(I know, the above one is simple enough, but it's a good thing to provide, thank you!)

@ydshieh 
the latency is 0.0147s  if convert it to numpy.

The speed looks very good. But see my comment in the PR :-)

@ydshieh 
i have another pr . it's already went through a lot checks. but i don't know is it accepted or what should i do next. 
could u give a review pls ?
https://github.com/huggingface/transformers/pull/24772#issuecomment-1635168756

Thank you @liucw2012 for the PR ❤️ 

The CI in that PR is green 🚀 . However I would like to check a bit deeper what the (nested) inputs would be possible for that method, and if every case works and if none of the case will slow down.

BTW, could you maybe provide what `query_list` you used.

(it's always a nice thing to provide the actual definition for variables in a code snippet 🙏 )

sorry, i  was a lit bit busy recently. the query_list  is just  two chats, each one is almost 800 Chinese characters。all others examples is ok if u used tokenier with padding=true；

It's fine. But see

https://github.com/huggingface/transformers/pull/24772#discussion_r1265559341
update `use_auth_token` -> `token`
Author: ydshieh
Pull request: yes

# What does this PR do?

Fix #25008

@sgugger This should be ready for a review. There are still a few places that use `use_auth_token`, but only during the calls to relevant methods.

I will change those calls to use `token` before merge.

Comments:
_The documentation is not available anymore as the PR was closed or merged._

In some training example script, (for example, `examples/tensorflow/contrastive-image-text/run_clip.py`) there are 

```python
    use_auth_token: bool = field(
        default=False,
        metadata={
            "help": (
                "Will use the token generated when running `huggingface-cli login` (necessary to use this script "
                "with private models)."
            )
        },
    )
```
Do you have any comment on how we should do with them?

Change everything to use `token` but still need to accept `use_auth_token`?
Move common image processing methods to BaseImageProcessor
Author: amyeroberts
Pull request: yes

# What does this PR do?

Moves out the `rescale` and `normalize` methods of the image processors to the BaseImageProcessor class, which all the image processors inherit from. 


Reason for moving `rescale` and `normalize`:
* Standard image processing steps
* Used by (almost) all image processors 
* Few cases when the logic is model specific e.g. [ViVit](https://github.com/huggingface/transformers/blob/f9cc333805c47665c6afee8b5867931e54abe0c6/src/transformers/models/vivit/image_processing_vivit.py#L196). 


Reason for not moving other methods: 
* Many require model specific preparation before calling the transforms function. For example, `resize` has different logic across many models e.g. [1](https://github.com/huggingface/transformers/blob/f9cc333805c47665c6afee8b5867931e54abe0c6/src/transformers/models/beit/image_processing_beit.py#L142), [2](https://github.com/huggingface/transformers/blob/f9cc333805c47665c6afee8b5867931e54abe0c6/src/transformers/models/detr/image_processing_detr.py#L865), [3](https://github.com/huggingface/transformers/blob/f9cc333805c47665c6afee8b5867931e54abe0c6/src/transformers/models/vilt/image_processing_vilt.py#L196)
* Some methods aren't universal to all image processors and don't make sense to add to all image processors e.g. [reduce_label](https://github.com/huggingface/transformers/blob/f9cc333805c47665c6afee8b5867931e54abe0c6/src/transformers/models/beit/image_processing_beit.py#L235)
* Some will be moved in future e.g. center_crop but this requires a bit more work outside the PR scope


## Before submitting
- [ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [x] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#start-contributing-pull-requests),
      Pull Request section?
- [ ] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [ ] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/main/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/main/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests?

Comments:
_The documentation is not available anymore as the PR was closed or merged._
feat: `agent.run(return_agent_types=True)`
Author: aarnphm
Pull request: no

### Feature request

Currently, `agent.run` on main will run materializer from `AgentType` to return its corresponding type.

I think it would be a great addition to just return this `AgentType` directly for external libraries to build on top of!

```python

agent = transformers.HfAgent("inference-api-endpoint")

res: AgentType = agent.run(..., return_agent_types=True)
```

### Motivation

I'm currently playing around with the new agent API, and found that in cases where I don't want to return the decoded outputs immediately, it would be nice to get `AgentType` and manage the materialize myself.

### Your contribution

I can help to create PR, but I know that the Agent API are still very experimental and unstable

cc @LysandreJik on this

Comments:
Hey @aarnphm, could you provide a code sample with the return you'd like to receive so that I can play with it and see if it makes sense to implement it? Thanks!

For example, I'm currently building [OpenLLM](https://github.com/bentoml/OpenLLM) and came across a use case where one can define an agent to generate an image and then caption it via a pipeline using [BentoML Runner](https://docs.bentoml.org/en/latest/concepts/runner.html#what-is-runner)

OpenLLM also provides support for HuggingFace Agent where users will can switch between The inference endpoint or hosting their own starcoder.

Given the following segment to save a `captioning` pipeline

```python
import bentoml, transformers

bentoml.transformers.save_model("captioning", pipeline('image-captioning'))
```

Runner is distributed by nature, and it can be defined in a service.py like so:

```python
import bentoml
import transformers

captioning_runner = bentoml.transformers.get("captioning").to_runner()

agent = transformers.HfAgent("http://283.23.22.1:3000/hf/agent")  # `openllm start starcoder`

service = bentoml.Service("agent-with-runners", runners=[captioning_runner])

def preprocess(input_tensor: torch.Tensor) -> torch.Tensor:
	... 

@svc.api(input=bentoml.io.Text(), output=bentoml.io.Text())
async def transcribe_audio_to_french(prompt: str):
	image_output: ImageAgentType = agent.run(prompt, ..., return_agent_types=True)
	# then I do some preprocess with this tensor
	input_for_pipeline = preprocess(image_output.to_raw())
	return await async captioning_runner.async_run(input_for_pipeline)
```

You can run this with `bentoml serve service.py:svc`

This can be one use case of the AgentType that can be useful here, where one can access the tensor directly without having to convert from PIL.Image output (which is currently what the agent.run returns if it returns an image if I understand it correctly)



This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.
Add SophiaG.
Author: guilt
Pull request: yes

# What does this PR do?

This is a scratch PR showing how to test Sophia with Transformers. This is no way
production ready, and certainly needs to look at licensing. But, this is helpful if someone needs
to try this right away. I'm re-using **AdamW**'s `beta` values. Plus if you see carefully,
there's an ugly hack where I'm using `eps` as `rho`. 

This is code directly copy-pasta-ed from: @Liuhong99 's  [Sophia](https://github.com/Liuhong99/Sophia);

I am putting it here so people can experiment with it and see how it compares to **AdamW**. If there
is sufficient interest in adding this and it can be licensed, would be happy to work on it here. Anyone is free to take
this and turn it into something of value. Please close this as necessary too.

## Before submitting

This PR does none of the above. It is too early to do this, but if there is sufficient interest would be happy to go through this process.

- [ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [ ] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#start-contributing-pull-requests),
      Pull Request section?
- [ ] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [ ] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/main/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/main/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests?


## Who can review?

Models:
- text models: @ArthurZucker and @younesbelkada

Common:
- trainer: @sgugger


Comments:
[Semi-related] also linking to Paper page on HF: https://huggingface.co/papers/2305.14342

Thank you all for encouraging this. 

As a first cut, I am working with the authors to see if this can be a PyPi package if authors agree. License is MIT, so hopefully we can get this out soon.  [Sophia on PyPi](https://github.com/Liuhong99/Sophia/issues/29)

As Younes said before, we won't merge this PR but can leave it for anyone who want to try this out: Transformers is a library of models, not optimizers (the optimizers inside the library are actually deprecated). Once there is a package supporting this optimizer we can add support for the Trainer like we did for the `bitsandbytes` optimizers.

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.
MaskFormer - enable return_dict in order to compile
Author: amyeroberts
Pull request: yes

# What does this PR do?

Is was previously not possible to use `torch.compile` on Maskformer as some modules always returned a dataclass, instead of a tuple. 

This PR adds a `return_dict` argument to these modules, which defaults to `True` to maintain the previous behaviour. 

Tested with:
```python
import requests

import torch
from PIL import Image

from transformers import AutoImageProcessor,AutoModelForInstanceSegmentation

checkpoint = "facebook/maskformer-swin-base-ade"
url = 'http://images.cocodataset.org/val2017/000000039769.jpg'
image = Image.open(requests.get(url, stream=True).raw)
processor = AutoImageProcessor.from_pretrained(checkpoint)
model = AutoModelForInstanceSegmentation.from_pretrained(checkpoint).to("cuda")
processed_input = processor([image, image], return_tensors='pt').to(device="cuda")

compiled_model = torch.compile(model, fullgraph=True)

with torch.no_grad():
    compiled_model(**processed_input, return_dict=False)
```



## Before submitting
- [ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [x] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#start-contributing-pull-requests),
      Pull Request section?
- [ ] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [ ] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/main/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/main/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests?


Comments:
_The documentation is not available anymore as the PR was closed or merged._

cc @merveenoyan who highlighted the compiling issue 🙏 

@sgugger Yep! I'll update that

@sgugger Most of the custom tests aren't easy to remove, unfortunately. As MaskFormer combines hidden states from different modules -- encoder, transformer decoder and pixel module decoder -- which are defined in nested arguments in the config e.g. `model.config.decoder_config.num_hidden_layers` it breaks a lot of assumptions. 

I've removed `test_training` and made `_prepare_for_class` and `test_output_attentions` match the common equivalent. I've also updated some of the config values to make sure they're small by default. 

Thanks a lot!

Also, I wasn't able to enable the FX tests. There's an issue when the model is symbolically traced, where the output shapes are slightly different (off by one). Checking the outputs from the compiled model using `torch.compile` this doesn't occur, so I'm leaving this for a future PR :)
fix idefics vision config
Author: younesbelkada
Pull request: yes

# What does this PR do?

As discussed offline, this should refactor the vision config args.

We need to address similar changes as https://huggingface.co/HuggingFaceM4/tiny-random-idefics/discussions/2 to make it work with other checkpoints

cc @stas00 


Comments:
_The documentation is not available anymore as the PR was closed or merged._

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_25092). All of your documentation changes will be reflected on that endpoint.
Move center_crop to BaseImageProcessor
Author: amyeroberts
Pull request: yes

# What does this PR do?

Moves center_crop to BaseImageProcessor as the logic is the same for almost all models' image processors (except [bridgetower](https://github.com/huggingface/transformers/blob/659829b6ae558dd2e178462a797bf8b1a749f070/src/transformers/models/bridgetower/image_processing_bridgetower.py#L229) and [owlvit](https://github.com/huggingface/transformers/blob/659829b6ae558dd2e178462a797bf8b1a749f070/src/transformers/models/owlvit/image_processing_owlvit.py#L184)), and is a standard transformation all processors might want to apply.

## Before submitting
- [ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [x] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#start-contributing-pull-requests),
      Pull Request section?
- [ ] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [ ] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/main/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/main/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests?


Comments:
_The documentation is not available anymore as the PR was closed or merged._
[setup] fix min isort requirements
Author: stas00
Pull request: yes

the current isort min version requirement doesn't match CI's reality

even with isort 5.10.1 `make style` leads to modified code which doesn't match CI.

I tested that 5.12 syncs with CI.


Comments:
no, actually that didn't help :(

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_25126). All of your documentation changes will be reflected on that endpoint.
make run_generation more generic for other devices
Author: ji-huazhong
Pull request: yes

## What does this PR do?
Currently, the example for text-generation is only available for cuda or cpu. This PR makes it work well on mps or npu devices.

Verified on A100 and npu.

Example usage:
```
python3 run_generation.py \
    --model_type=gpt2 \
    --model_name_or_path=gpt2
```

Below are the output logs:
- On GPU
  ```
  07/27/2023 11:23:51 - WARNING - __main__ - device: cuda, n_gpu: 8, 16-bits training: False
  Using pad_token, but it is not set yet.
  07/27/2023 11:23:57 - INFO - __main__ - Namespace(model_type='gpt2', model_name_or_path='gpt2', prompt='', length=20, stop_token=None, temperature=1.0, repetition_penalty=1.0, k=0, p=0.9, prefix='', padding_text='', xlm_language='', seed=42, use_cpu=False, num_return_sequences=1, fp16=False, jit=False, device=device(type='cuda'), n_gpu=8)
  Model prompt >>> "I'm Jack
  The attention mask and the pad token id were not set. As a consequence, you may observe unexpected behavior. Please pass your input's `attention_mask` to obtain reliable results.
  Setting `pad_token_id` to `eos_token_id`:50256 for open-end generation.
  === GENERATED SEQUENCE 1 ===
  "I'm Jack Russell."
  
  "They said, 'Hey, you know, you have to get away with
  ```

- On NPU：
  ```
  07/27/2023 11:21:47 - WARNING - __main__ - device: npu, n_gpu: 8, 16-bits training: False
  Using pad_token, but it is not set yet.
  07/27/2023 11:22:03 - INFO - __main__ - Namespace(device=device(type='npu'), fp16=False, jit=False, k=0, length=20, model_name_or_path='gpt2', model_type='gpt2', n_gpu=8, num_return_sequences=1, p=0.9, padding_text='', prefix='', prompt='', repetition_penalty=1.0, seed=42, stop_token=None, temperature=1.0, use_cpu=False, xlm_language='')
  Model prompt >>> "I'm Jack
  The attention mask and the pad token id were not set. As a consequence, you may observe unexpected behavior. Please pass your input's `attention_mask` to obtain reliable results.
  Setting `pad_token_id` to `eos_token_id`:50256 for open-end generation.
  === GENERATED SEQUENCE 1 ===
  "I'm Jack Dylan," guitarist Tim Farrell said, picking up Dylan's Lave Club guitar and in a playful styl
  ```




Comments:
_The documentation is not available anymore as the PR was closed or merged._
Wrong default values according to docstrings?
Author: ebezzam
Pull request: no

https://github.com/huggingface/transformers/blob/d941f07a4e3bc7b61b7afbd25d6e2e8427fccc6d/src/transformers/models/vit/configuration_vit.py#L100

https://github.com/huggingface/transformers/blob/d941f07a4e3bc7b61b7afbd25d6e2e8427fccc6d/src/transformers/models/vit/configuration_vit.py#L101

Comments:
Hi @ebezzam 

I am not sure I understand the question. Could you describe it more precisely? Thanks.

Hi @ydshieh thank you for your quick response.

The docstring of [ViTConfig](https://github.com/huggingface/transformers/blob/d941f07a4e3bc7b61b7afbd25d6e2e8427fccc6d/src/transformers/models/vit/configuration_vit.py#L35) says that the default values for [`hidden_dropout_prob`](https://github.com/huggingface/transformers/blob/d941f07a4e3bc7b61b7afbd25d6e2e8427fccc6d/src/transformers/models/vit/configuration_vit.py#L58) and [`attention_probs_dropout_prob`](https://github.com/huggingface/transformers/blob/d941f07a4e3bc7b61b7afbd25d6e2e8427fccc6d/src/transformers/models/vit/configuration_vit.py#L60) are `0.1`, but in the code (linked above) they are set to `0.0`.

Thanks a lot @ebezzam , super clear now 🤗 

Will take a look



Hi @ebezzam 

From [the original ViT codebase](https://github.com/google-research/vision_transformer/blob/ac6e056f9da686895f9f0f6ac026d3b5a464e59e/vit_jax/configs/models.py#L123), they should be `0.0` for the `ViT-B/16` model.

The value `0.1` in the docstring is likely from a copy-paste.

Would you like to open a PR to help us correct those 2 values in the docstring? Thanks.

@ydshieh thanks for looking into that. Yes I can open a PR and link to this issue.

Thanks a lot 🤗 

@ydshieh done!

Thank you again @ebezzam 🤗
🐛  torch baddbmm error fixed for BigCode models
Author: mayank31398
Pull request: yes

Fixes # (issue)
This was needed because of a bug in pytorch https://github.com/pytorch/pytorch/issues/80588.
The bug was fixed in https://github.com/pytorch/pytorch/pull/96086


## Before submitting
- [ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [x] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#start-contributing-pull-requests),
      Pull Request section?
- [ ] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [ ] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/main/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/main/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests?

@ArthurZucker @younesbelkada @jlamypoirier

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_24768). All of your documentation changes will be reflected on that endpoint.

> I think we don't use PT>2.0.0 that includes the fix you mentioned above, there should be a reason for that. cc @ydshieh

You prediction is 200% correct: we have torch `2.0.1`. The mentioned torch fix is not included in that minor bug release.

Ah I see thanks for double checking @ydshieh !

closing this since its not relevant at this point in time.
Support End LR for Cosine LR Scheduler
Author: yqy2001
Pull request: no

### Feature request

Customize the end learning rate of a cosine LR scheduler, as a non-zero end lr is commonly used now (e.g., LLaMA-2 use 10% of peak lr as end lr).

![image](https://github.com/huggingface/transformers/assets/55196500/4a07777f-968c-47da-b836-0f84dfd07a4d)


### Motivation

non-zero end lr is commonly used now (e.g., LLaMA-2 use 10% of peak lr as end lr)

### Your contribution

Sorry, I don't think I can be of much help.

Comments:
You can pass along your custom scheduler to the `Trainer` :-)

OK, Thank you.