noamaanMulla-03/transformers-issues huggingface dataset:
Can't do word timestamps and beam search at the same time (whisper)
Author: Snarkdoof
Pull request: no

### System Info

Tested on python 3.8.10, transformers 4.36.0.dev0



### Who can help?

@ArthurZucker @sanchit-gandhi (suggested by peregilk)

### Information

- [ ] The official example scripts
- [ ] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

```
from transformers import pipeline
import torch
model = "NbAiLabBeta/nb-whisper-base"
device = "cuda:0"

p = pipeline("automatic-speech-recognition",
             model,
             torch_dtype=torch.float16,
             device=device,
             return_timestamps="word")
args = {"language": "norwegian", "task": "transcribe", "num_beams": 3}
outputs = p(audiofile,
            chunk_length_s=28,
            batch_size=6,
            generate_kwargs=args)
```

Fails with:

> Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/usr/local/lib/python3.8/dist-packages/transformers/pipelines/automatic_speech_recognition.py", line 357, in __call__
    return super().__call__(inputs, **kwargs)
  File "/usr/local/lib/python3.8/dist-packages/transformers/pipelines/base.py", line 1132, in __call__
    return next(
  File "/usr/local/lib/python3.8/dist-packages/transformers/pipelines/pt_utils.py", line 124, in __next__
    item = next(self.iterator)
  File "/usr/local/lib/python3.8/dist-packages/transformers/pipelines/pt_utils.py", line 266, in __next__
    processed = self.infer(next(self.iterator), **self.params)
  File "/usr/local/lib/python3.8/dist-packages/transformers/pipelines/base.py", line 1046, in forward
    model_outputs = self._forward(model_inputs, **forward_params)
  File "/usr/local/lib/python3.8/dist-packages/transformers/pipelines/automatic_speech_recognition.py", line 552, in _forward
    generate_kwargs["num_frames"] = stride[0] // self.feature_extractor.hop_length
TypeError: unsupported operand type(s) for //: 'tuple' and 'int'

It works with *either* num_beams:1 OR return_timestamps=True/False, but not combined.

### Expected behavior

It should return processed data. :)

Comments:
cc @gante 

I believe there's already an active PR for this: https://github.com/huggingface/transformers/pull/26699 However, the PR might need a little more work as it is not a polished solution yet (at least that's what I think).

Fixed by https://github.com/huggingface/transformers/pull/28114 👍
Add random seed for generation
Author: mokeyish
Pull request: yes

# What does this PR do?

Add a random seed to GenerationConfig, such that with the same `seed` and parameters should return the same result.

openai adds a seed parameter to control the return of the same results, making it easier to reproduce the results and troubleshoot problems.

https://github.com/openai/openai-openapi/blob/master/openapi.yaml#L5392

I would like to cc @ArthurZucker and @younesbelkada review my PR.





Comments:
Hi @mokeyish 👋 

We already have a `set_seed` function in `transformers` ([doc](https://huggingface.co/docs/transformers/v4.35.2/en/internal/trainer_utils#transformers.set_seed)), which you can call as follows for complete reproducibility:

```py
from transformers import set_seed
# code to load the model

set_seed(0)
# generate code here
```

Would this solve your use case? 🤗 We want to avoid adding flags and further complexity to generate, unless strictly needed.

@gante This seed is passed in through the http RestAPIs, and your solution is a global setting, which is the same as before to make model training reproducible. But global settings, different API calls should conflict, right?

In theory, yes. In practice, we're not interested in adding complexity to `generate` unless widely requested by the community -- our ability to maintain features is limited :) 

I haven't seen request for it, but I'll do my standard bargain: if this comment gets 10 reactions or more, it means users are actively looking for it. In that case, I'd be happy to include it in `transformers` 🤗 

(the person that does the 10th reaction: please ping me)

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.
Error when set pad_token
Author: AndreWanga
Pull request: no

### System Info

transformer version: 4.24.0

python version: 3.9.13


### Who can help?

@ArthurZucker

### Information

- [ ] The official example scripts
- [ ] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

When I use transformer for [chatglm2-6b](https://huggingface.co/THUDM/chatglm2-6b) , I try to run this code:

`tokenizer.pad_token = tokenizer.eos_token`

But raise an AttributeError: can't set attribute

### Expected behavior

code run successfully

Comments:
Hey! Would you mind upgrading to the latest version of transformers and provide me with a reproducer of how you are initializing the model? 🤗 

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.
How to preupgrade transformer cache and build the upgraded into docker image?
Author: lanyusan
Pull request: no

### System Info

Linux ubuntu 22.04
Docker 24.05

I am not sure if this is the right place for this issue. Apology if it isn't and please direct me to the right place.

I have been using transformer in docker images that are deployed at runpod/replicate. The containers of the images could go cold and be relaunched again and again. Each time the container would waste 20 to 40 seconds for the blow cache upgrade.

```
The cache for model files in Transformers v4.22.0 has been updated. Migrating your old cache. This is a one-time only operation. You can interrupt this and resume the migration later on by calling `transformers.utils.move_cache()`.
```

It would take around 20 to 40 seconds, which is a significant waste of our GPU time and  container startup time.

I have tried to find out how to preupgrade the cache and build the upgrade cache into docker image by google but I couldn't find a way to do it.

Please advise how to preupgrade the cache and build the upgraded cache in docker image.

Many thanks.

### Expected behavior

The cache for model files is preupgraded and built into container image to avoid upgrade each time a container is launched.



Comments:
Hi @lanyusan, thanks for raising this issue! 

Could you confirm which version of transformers you're running? 

> The cache for model files is preupgraded and built into container image to avoid upgrade each time a container is launched.

This is something you'd have to handle on your side in terms of image setup etc. Here's the doc page about cache management: https://huggingface.co/docs/transformers/installation#cache-setup. To select where you want your cache, you'll want to set the `TRANSFORMERS_CACHE` env var.

@amyeroberts 

Thanks for the reply. It is exactly what I needed. 

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.
Data Issues related to models with no sequence length limit (e.g., XLNet)
Author: AdamLouly
Pull request: no

### System Info

nightly transformers

### Who can help?

_No response_

### Information

- [ ] The official example scripts
- [ ] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

Models like XLNet, which have no sequence limit, set max_position_embeddings to -1.
eg: **https://github.com/huggingface/transformers/blob/acc394c4f5e1283c19783581790b3dc3105a3697/src/transformers/models/xlnet/configuration_xlnet.py#L235**

This causes block_size to be -1 during text grouping, resulting in an empty dataset and a 
`ValueError: num_samples should be a positive integer value, but got num_samples=0` error. 

after the latest change to pull the block size from max_position_embeddings was introduced, this case was not handled.

I'm creating a PR to fix that issue.

### Expected behavior

`ValueError: num_samples should be a positive integer value, but got num_samples=0` error. 


Comments:
This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.
FlashAttention remove Nvidia only GPUs to more generic documentation
Author: mfuntowicz
Pull request: yes

This PR adds AMD as a supported target GPU in the documentation relative to Flash Attention.

Comments:
This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.
Allow independent control of logging and progress bars across Trainer methods
Author: harrisonfloam
Pull request: no

### Feature request

Goal: Finer control of logging and progress tracking output for Trainer methods.

Specifically, I would like to suppress print logging and progress bars during in-training evaluation, but maintain the main training loop progress bar. A good solution would allow independent control of print logging and progress bars for each Trainer method (train, evaluate, predict).

### Motivation

Logging and progress tracking is great, but console-overflow is real. When training with an every-epoch evaluation strategy, my console (or notebook cell output) is overwhelmed with print logs and empty, completed tqdm progress bars.

The current options for logging control that I am aware of are:
- `log_level`: Setting to 'critical' seems to minimize warning-type output, but doesn't affect print logging of training metrics.
- `disable_tqdm`: Disables all tqdm progress bars globally. 
- `remove_callback(PrinterCallback)`: Suppresses print logs, but only when `disable_tqdm=True` due to a conditional in [trainer.py, line 528](https://github.com/huggingface/transformers/blob/b074461ef0f54ce37c5239d30ee960ece28d11ec/src/transformers/trainer.py#L528C15-L528C15). The `ProgressCallback` or `NotebookProgressCallback` that is added when `disable_tqdm=False` enables all logging with no option for suppression.
```
self.add_callback(PrinterCallback if self.args.disable_tqdm else DEFAULT_PROGRESS_CALLBACK)
```

Ideally, I would like to be able to independently disable progress bars for the evaluate calls inside the training loop, but maintain logging and progress bars for the training loop as a whole and any subsequent evaluate or predict calls. Suppressing all logging via `disable_tqdm` and `remove_callback(PrinterCallback)` solves the console-overflow issue, but leaves me staring at a blank screen wondering if my training loop is working.

### Your contribution

I have not contributed to huggingface before, but would be happy to work or collaborate on a PR.

This solution would likely require:
- moderate changes to `ProgressCallback` in [trainer_callback.py](https://github.com/huggingface/transformers/blob/main/src/transformers/trainer_callback.py).
- moderate changes to `NotebookProgressCallback` in [utils/notebook.py](https://github.com/huggingface/transformers/blob/main/src/transformers/utils/notebook.py).
- very minor changes to [trainer.py](https://github.com/huggingface/transformers/blob/main/src/transformers/trainer.py) and [training_args.py](https://github.com/huggingface/transformers/blob/main/src/transformers/training_args.py).

Comments:
cc @muellerzr @pacman100 

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.
Adding bbox transformatinos
Author: rafaelpadilla
Pull request: yes

# What does this PR do?

This PR implements bounding box transformations among different formats:
`XYXY` <-> `XYWH` <-> `XCYCWH` <-> `RELATIVE_XYWH` <-> `RELATIVE_XCYCWH`

Object detection models can output boxes in different formats, requesting manual transformations while porting these models. The idea to have a flexible function such as the proposed `transform_box_format` is similar to our image transformation functions such as `normalize`, `center_crop`, `resize`, etc, that can be used by any model.

We currently have 2 functions that make bboxes transformations: `center_to_corners_format`, `corners_to_center_format`, which are useful, but still require an image processor to do other conversions, as transforming relative to absolute. An example is shown below - performed by [image_processing_detr.py](https://github.com/huggingface/transformers/blob/main/src/transformers/models/detr/image_processing_detr.py#L1352-L1357):

```python
# convert to [x0, y0, x1, y1] format
boxes = center_to_corners_format(out_bbox)
# and from relative [0, 1] to absolute [0, height] coordinates
img_h, img_w = target_sizes.unbind(1)
scale_fct = torch.stack([img_w, img_h, img_w, img_h], dim=1).to(boxes.device)
boxes = boxes * scale_fct[:, None, :]
```

This could simply be replaced by:
```python
new_boxes = transform_box_format(out_bbox, orig_format="relative_xcycwh", dest_format="xyxy", img_shape=target_sizes.squeeze())
```
which leads to the same results, so that `torch.isclose(boxes, new_boxes)` is `True`.

## Before submitting
- [ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [x] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#start-contributing-pull-requests),
      Pull Request section?
- [x] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [ ] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/main/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/main/docs#writing-source-documentation).
- [x] Did you write any new necessary tests?


## Who can review?
@amyeroberts 

Comments:
This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.
Error when passing past_key_values back to generate()
Author: dto
Pull request: no

### System Info

- `transformers` version: 4.36.0.dev0
- Platform: Linux-5.15.0-88-generic-x86_64-with-glibc2.35
- Python version: 3.10.13
- Huggingface_hub version: 0.19.3
- Safetensors version: 0.4.0
- Accelerate version: 0.24.0
- Accelerate config: 	not found
- PyTorch version (GPU?): 2.1.0+rocm5.6 (True)
- Tensorflow version (GPU?): 2.14.0 (False)
- Flax version (CPU?/GPU?/TPU?): not installed (NA)
- Jax version: not installed
- JaxLib version: not installed
- Using GPU in script?: yes
- Using distributed or parallel set-up in script?: no

### Who can help?

_No response_

### Information

- [ ] The official example scripts
- [X] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [X] My own task or dataset (give details below)

### Reproduction

The test python script attempts to retrieve past_key_values from the output of generate() and pass this value back to generate() to continue a chat style conversation. However, Transformers does not like the value and gives "RuntimeError: The size of tensor a (12) must match the size of tensor b (31) at non-singleton dimension 3" when it is passed back verbatim. Please see attached Python code and traceback output

[test.py.txt](https://github.com/huggingface/transformers/files/13382751/test.py.txt)
[traceback.txt](https://github.com/huggingface/transformers/files/13382752/traceback.txt)


### Expected behavior

I would expect that passing past_key_values back to generate() allows to continue the conversation.

Comments:
cc @gante 

Hi @dto 👋 Thank you for raising this issue!

`generate` expects the full input prompt to be passed, even if `past_key_values` is passed too. In this case, the input prompt is the full multi-round conversation. In your script, if you replace the `input_ids` input by `torch.cat((marvin_response.sequences, marvin_input_ids.to('cuda')), dim=-1)`, it will generate something along these lines:

```


 
 (+ 2 3) 
 The result of the expression `(+ 2 3)` is 5.

 
 Please introduce yourself. 
 Hello! I'm an AI language model. I'm here to help you with any questions or tasks you might have. How may I assist you today?

```

___________________________________________________

The fact that you need the full multi-round conversation may not be obvious, so I'm taking this issue as an opportunity to write documentation for this feature and clear things out :)

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.
XLA Integration for TensorFlow Models 
Author: akote123
Pull request: no

Hi,
    I wanted to try XLA for Inference. 
```python
import os
#environment configs

import tensorflow as tf

from transformers import BertTokenizer, TFBertModel
tokenizer = BertTokenizer.from_pretrained('bert-base-uncased')
model = TFBertModel.from_pretrained("bert-base-uncased")
xla_generate = tf.function(model, jit_compile=True)
text = "Replace me by any text you'd like."
encoded_input = tokenizer(text, return_tensors='tf')

from time import time
start_time = time()
for i in range(1):
    output = xla_generate(encoded_input)
end_time = time()
time_taken = end_time - start_time
print("average time (seconds) for bert Inference: ", time_taken)`
```
For this code I am getting timing which more than without XLA and IF look into perf report of this I am not seeing any XLA OP and I am seeing oneDNN . So I wanted to understand how this XLA works with oneDNN 

Comments:
Hey! Here is what you gotta know about using xla:

> When you are executing an XLA-enabled function (like `xla_generate()` above) for the first time, it will internally try to infer the computation graph, which is time-consuming.  This process is known as [“tracing”](https://www.tensorflow.org/guide/intro_to_graphs#when_is_a_function_tracing). 

> You might notice that the generation time is not fast. Successive calls of `xla_generate()` (or any other XLA-enabled function) won’t have to infer the computation graph, given the inputs to the function follow the same shape with which the computation graph was initially built. While this is not a problem for modalities with fixed input shapes (e.g., images), you must pay attention if you are working with variable input shape modalities (e.g., text).

See [the documentation](https://huggingface.co/docs/transformers/tf_xla#gotchas-to-be-aware-of)





This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.
LongT5 - Errors
Author: gozdeydd
Pull request: no

I am not very sure if the models/documentation is up to date in the website. For example, I have experienced this with a LongT5 model. It gives an error that transformers library doesn't have such model. 

I assume this happens due to the fast changes going on with NLP models in general these days. Perhaps there is a new version? 

Comments:
Hey! If you want some help you should provide a reproducer of what the error was with the output of `transformers-cli env` 🤗 

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.
[`Awq`] Enable the possibility to skip quantization for some target modules
Author: younesbelkada
Pull request: yes

# What does this PR do?

Adds the possibility to load AWQ models if some modules of the model are skipped for quantization. 
E.g. for whisper, Llava, Mixtral, we respectively don't want to quantize the encoder, vision encoder and the gate layer to ensure inference stability. 

Let's merge it once AWQ makes the 0.1.8 release

cc @ArthurZucker @casper-hansen @TheBloke @SunMarc 

https://github.com/casper-hansen/AutoAWQ/pull/248

This PR makes it also possible to run multi-modal models with AWQ:

```py
from transformers import pipeline
from PIL import Image    
import requests

model_id = "ybelkada/llava-1.5-7b-hf-awq"
pipe = pipeline("image-to-text", model=model_id, device=0)
url = "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/diffusers/compel-neg.png"

image = Image.open(requests.get(url, stream=True).raw)
prompt = "USER: <image>\nCan you please describe this image?\nASSISTANT:"

outputs = pipe(image, prompt=prompt, generate_kwargs={"max_new_tokens": 100})
print(outputs[0]["generated_text"])
```

![image](https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/diffusers/compel-neg.png)

> USER:  \nCan you please describe this image?\nASSISTANT: The image features a brown and white cat sitting on a green surface, possibly a carpet or a grassy area. The cat is holding a red ball in its paws, seemingly playing with it. The cat appears to be focused on the ball, possibly preparing to play or just enjoying the toy.



Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_27950). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

This PR enables compatiblity with Mixtral AWQ ! 
https://github.com/casper-hansen/AutoAWQ/pull/251 being merged, this PR is ready for review 🙏 

Thanks a lot for the review @amyeroberts ! 
I will merge this as soon as AutoAWQ makes the 0.1.8 Mixtral release cc @casper-hansen just for your information

Release has been done! Merging!