noamaanMulla-03/transformers-issues huggingface dataset:
transformers DataCollatorWithPadding should have an option to specify the attributes which require padding.
Author: varadhbhatnagar
Pull request: no
Labels: name: Feature request

### Feature request

Currently there is no way to specify the attributes which require padding in transformers [DataCollatorWithPadding](https://github.com/huggingface/transformers/blob/v4.34.0/src/transformers/data/data_collator.py#L215).

### Motivation

Two instances from our data may be of this format:
```
{
"input_ids": [1,2,3,4,5],
"attention_mask": [0,1,1,1,1],
"label": "apple"
}

{
"input_ids": [1,2,3,4,5,6,7,8],
"attention_mask": [0,1,1,1,1,1,1,1],
"label": "mango"
}

```
User wants to implement [dynamic padding](https://huggingface.co/learn/nlp-course/chapter3/2?fw=pt#dynamic-padding) over these two data points, but currently there is no way to specify that `label` attribute should not be passed through [`self.tokenizer.pad()`]()https://github.com/huggingface/transformers/blob/b71f20a7c9f3716d30f6738501559acf863e2c5c/src/transformers/data/data_collator.py#L249

### Your contribution

I can submit a PR for this.

Comments:
This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.
transformers DefaultDataCollator does not work with 'str' type data
Author: varadhbhatnagar
Pull request: no

### System Info

- `transformers` version: 4.33.3
- Platform: Linux-4.18.0-477.10.1.el8_8.x86_64-x86_64-with-glibc2.28
- Python version: 3.11.3
- Huggingface_hub version: 0.17.3
- Safetensors version: 0.3.1
- Accelerate version: 0.21.0
- Accelerate config:    - compute_environment: LOCAL_MACHINE
        - distributed_type: MULTI_GPU
        - mixed_precision: bf16
        - use_cpu: False
        - num_processes: 4
        - machine_rank: 0
        - num_machines: 1
        - gpu_ids: all
        - rdzv_backend: static
        - same_network: True
        - main_training_function: main
        - downcast_bf16: no
        - tpu_use_cluster: False
        - tpu_use_sudo: False
        - tpu_env: []
- PyTorch version (GPU?): 2.0.1 (True)
- Tensorflow version (GPU?): not installed (NA)
- Flax version (CPU?/GPU?/TPU?): not installed (NA)
- Jax version: not installed
- JaxLib version: not installed
- Using GPU in script?: Yes
- Using distributed or parallel set-up in script?: No

### Who can help?

_No response_

### Information

- [ ] The official example scripts
- [X] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [X] My own task or dataset (give details below)

### Reproduction

https://colab.research.google.com/drive/19a3vfER_Onq19XffudQx4SNo_uuURDYc?usp=sharing

### Expected behavior

- The transformers [DefaultDataCollator](https://huggingface.co/docs/transformers/main_classes/data_collator#transformers.DefaultDataCollator) behaves differently compared to the PyTorch [default_collate()](https://pytorch.org/docs/stable/data.html#torch.utils.data.default_collate). The PyTorch `default_collate()` allows processing of other types such as `str`, which do not work with `DefaultDataCollator`.

Comments:
This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.

Hey! The data collator are meant to be overwritten for your custom usage 😉
Getting equivalent results between Transformer's resize and tf.image.resize
Author: NielsRogge
Pull request: no
Labels: name: Feature request

### Feature request

For the SigLIP model (#26522), I'd like to get equivalent results between tf.image.resize and the resize method available in Transformers.

Here's what I tried:

``` 
from PIL import Image
import requests

import tensorflow as tf
import numpy as np

def resize(image, size, method="bilinear", antialias=False):
    """Resizes image to a given size."""
    # Note: use TF-2 version of tf.image.resize as the version in TF-1 is
    # buggy: https://github.com/tensorflow/tensorflow/issues/6720.
    # In particular it was not equivariant with rotation and lead to the network
    # to learn a shortcut in self-supervised rotation task, if rotation was
    # applied after resize.
    dtype = image.dtype
    tf_dtype = tf.type_spec_from_value(image).dtype
    image = tf.image.resize(image, size, method=method, antialias=antialias)
    return tf.cast(tf.clip_by_value(image, tf_dtype.min, tf_dtype.max), dtype)

# load image
url = 'https://cdn.openai.com/multimodal-neurons/assets/apple/apple-ipod.jpg'
image = Image.open(requests.get(url, stream=True).raw)

# get original pixel values
original_pixel_values = resize(np.array(image), size=(224,224))

# get our pixel values
from transformers.image_transforms import resize

pixel_values = resize(np.array(image), size=(224,224), resample=Image.Resampling.BILINEAR)

# verify results
np.testing.assert_array_equal(original_pixel_values, pixel_values)
```
This currently fails with:
```
AssertionError: 
Arrays are not equal

Mismatched elements: 87370 / 150528 (58%)
Max absolute difference: 255
Max relative difference: 255.
 x: array([[[127, 101,  59],
        [136, 112,  72],
        [129, 109,  72],...
 y: array([[[131, 105,  63],
        [138, 114,  74],
        [126, 108,  70],...
```

### Motivation

Would be great to have equivalent results such that logits match with the original implementation.

### Your contribution

I provide a notebook [here](https://colab.research.google.com/drive/1nP8f07qd3jWBRgCnE29cagseRtqVMADa?usp=sharing) for testing.

Comments:
cc @amyeroberts
Update Korean tutorial for using LLMs, and refactor the nested conditional statements in hr_argparser.py
Author: YeonwooSung
Pull request: yes

# What does this PR do?

- Update the Korean docs for LLM tutorials for using Mistral7B, not Llama-v1
- Refactor several nested if-else statements for readability

## Before submitting
- [v] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [v] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#start-contributing-pull-requests),
      Pull Request section?
- [ ] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [v] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/main/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/main/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests?


## Who can review?

Anyone in the community is free to review the PR once the tests have passed. Feel free to tag
members/contributors who may be interested in your PR.

@stevhliu 


Comments:
cc @ArthurZucker 

Ok for me to update and showcase Mistral7B, but I'll let @ArthurZucker or @amyeroberts comment on the code refactor :)

@YeonwooSung Thanks for opening this PR! 

Could you separate out any code changes in hf parser and image utils into a separate PR? 

> @YeonwooSung Thanks for opening this PR!
> 
> Could you separate out any code changes in hf parser and image utils into a separate PR?

Sure, I will!

@amyeroberts I've just rebased my main branch, and will open a separate pull request for the code changes in hf parser and image utils.

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_27489). All of your documentation changes will be reflected on that endpoint.

BTW we could also use a Korean model to generate Korean text from [here](https://huggingface.co/spaces/upstage/open-ko-llm-leaderboard)

> BTW we could also use a Korean model to generate Korean text from [here](https://huggingface.co/spaces/upstage/open-ko-llm-leaderboard)

I'll try this next time. Thanks :)
I get 【KeyError: 'label'】 when I run transformers/examples/pytorch/text-classification/run_glue.py
Author: WeiChunyu-star
Pull request: no

### System Info

transformers: 4.36.0.dev0
accelerate : 0.24.1
platform： x86+V100

### Who can help?

_No response_

### Information

- [x] The official example scripts
- [ ] My own modified scripts

### Tasks

- [x] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

1.  git clone https://github.com/huggingface/transformers
2.  pip install ./transformers
3.  pip install -r transformers/examples/pytorch/text-classification/requirements.txt
4.  python transformers/examples/pytorch/text-classification/run_glue.py \
  --model_name_or_path ./bert-base-cased \
  --task_name mrpc \
  --do_train \
  --do_eval \
  --max_seq_length 128 \
  --per_device_train_batch_size 32 \
  --learning_rate 2e-5 \
  --num_train_epochs 3 \
  --output_dir ./mrpc_out

### Expected behavior

Using custom data configuration mrpc-f18c3b066d103ab0
11/21/2023 11:02:43 - INFO - datasets.builder - Using custom data configuration mrpc-f18c3b066d103ab0
Loading Dataset Infos from /root/miniconda3/lib/python3.8/site-packages/datasets/packaged_modules/text
11/21/2023 11:02:43 - INFO - datasets.info - Loading Dataset Infos from /root/miniconda3/lib/python3.8/site-packages/datasets/packaged_modules/text
Overwrite dataset info from restored data version if exists.
11/21/2023 11:02:43 - INFO - datasets.builder - Overwrite dataset info from restored data version if exists.
Loading Dataset info from /root/.cache/huggingface/datasets/glue/mrpc-f18c3b066d103ab0/0.0.0/c4a140d10f020282918b5dd1b8a49f0104729c6177f60a6b49ec2a365ec69f34
11/21/2023 11:02:43 - INFO - datasets.info - Loading Dataset info from /root/.cache/huggingface/datasets/glue/mrpc-f18c3b066d103ab0/0.0.0/c4a140d10f020282918b5dd1b8a49f0104729c6177f60a6b49ec2a365ec69f34
Found cached dataset glue (/root/.cache/huggingface/datasets/glue/mrpc-f18c3b066d103ab0/0.0.0/c4a140d10f020282918b5dd1b8a49f0104729c6177f60a6b49ec2a365ec69f34)
11/21/2023 11:02:43 - INFO - datasets.builder - Found cached dataset glue (/root/.cache/huggingface/datasets/glue/mrpc-f18c3b066d103ab0/0.0.0/c4a140d10f020282918b5dd1b8a49f0104729c6177f60a6b49ec2a365ec69f34)
Loading Dataset info from /root/.cache/huggingface/datasets/glue/mrpc-f18c3b066d103ab0/0.0.0/c4a140d10f020282918b5dd1b8a49f0104729c6177f60a6b49ec2a365ec69f34
11/21/2023 11:02:43 - INFO - datasets.info - Loading Dataset info from /root/.cache/huggingface/datasets/glue/mrpc-f18c3b066d103ab0/0.0.0/c4a140d10f020282918b5dd1b8a49f0104729c6177f60a6b49ec2a365ec69f34
Traceback (most recent call last):
  File "transformers/examples/pytorch/text-classification/run_glue.py", line 652, in <module>
    main()
  File "transformers/examples/pytorch/text-classification/run_glue.py", line 364, in main
    label_list = raw_datasets["train"].features["label"].names
KeyError: 'label'

Comments:
Streaming text output from a pipeline.
Author: lukolszewski
Pull request: no

### Feature request

Implement a way to obtain streaming text output from a pipeline. One token at a time.

### Motivation

To be able to see the response as it is being generated instead of having to wait for the entire thing. 

### Your contribution

I can test it if it gets implemented.

Comments:
Hey! Would recommend you to read [this section](https://huggingface.co/docs/transformers/generation_strategies#streaming) of the documentation about streaming when generating, I think it should help! 

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.
Add Mask R-CNN
Author: NielsRogge
Pull request: yes

# What does this PR do?

This PR is a further development of the Mask R-CNN framework. Supersedes #22973.

Updates:

- improved variable names, docstrings (especially also for the configuration class)
- removed specific `__repr__` and `__nice__` methods
- favor ONNX-compatible code wherever possible, instead of if-else statements
- `torchvision` was already leveraged for NMS

Regarding this:

> In some of the model-side processing code, there's switching to CPU and casting back and forth between torch and numpy - it's not clear why.

=> this is because when placing masks on the GPU, this would cause OOM errors. Hence those are placed on the CPU.




Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_25348). All of your documentation changes will be reflected on that endpoint.

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.
dvclive callback: warn instead of fail when logging non-scalars
Author: dberenbaum
Pull request: yes

# What does this PR do?

Fixes https://github.com/huggingface/transformers/pull/27352#issuecomment-1819131456. This will warn instead of fail when trying to log non-scalars as metrics.

## Before submitting
- [ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [x] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#start-contributing-pull-requests),
      Pull Request section?
- [x] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [ ] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/main/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/main/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests?


## Who can review?

@muellerz Could you please take a look?

Comments:
@muellerz This makes the tests pass, but I'm not sure if it's intended that the test here logs the learning rate as a list rather than as a scalar (which will fail under several of the existing loggers, but only with a warning like in this PR). 

https://github.com/huggingface/transformers/blob/e4280d650c579a87f645d1f4a4535feb27c49804/tests/trainer/test_trainer.py#L675

`self.lr_scheduler._last_lr` is a [list](https://github.com/pytorch/pytorch/blob/140c54e6ccc5e97f1b7f1e0fcd3d8c6af7dd2ab2/torch/optim/lr_scheduler.py#L167). Should a scalar value be extracted like `self.lr_scheduler._last_lr[0]`? That's the value being tested later as `["learning_rate"][0]`:

https://github.com/huggingface/transformers/blob/f31af3927f4091f5fb8126c77a0addebd4c1fe94/tests/trainer/test_trainer.py#L699-L712

Everywhere else in the codebase, it looks like a scalar is extracted:

https://github.com/huggingface/transformers/blob/f31af3927f4091f5fb8126c77a0addebd4c1fe94/src/transformers/trainer_pt_utils.py#L847-L867

https://github.com/huggingface/transformers/blob/f31af3927f4091f5fb8126c77a0addebd4c1fe94/examples/legacy/pytorch-lightning/run_glue.py#L46

In the future it's @muellerzr @dberenbaum, don't want to be pinging random people :) 

Yes, let's go with `[0]` as the one being extracted/the scalar. 

@muellerzr Apologies to you and the other person who was pinged here Zach! Added the change to the test in the last commit. The current test failures look unrelated.
Generate: Update docs regarding reusing `past_key_values` in `generate`
Author: gante
Pull request: yes

# What does this PR do?

Due to a [recent PR](https://github.com/huggingface/transformers/pull/25086), which enables `generate` to return `past_key_values`, we can now reuse `past_key_values` to speed up multi-round conversations with LLMs. This was already described in the [LLM optimization docs](https://huggingface.co/docs/transformers/llm_tutorial_optimization), even though there was no corresponding code back when it was written.

This PR updates the doc with a code example of how to return and reuse `past_key_values`, polishing a few nits encountered along the way.

Related issue: https://github.com/huggingface/transformers/issues/27546

Comments:
_The documentation is not available anymore as the PR was closed or merged._

The right-handed TOC is now fully functional!

<img width="239" alt="Screenshot 2023-11-20 at 18 31 56" src="https://github.com/huggingface/transformers/assets/12240844/cb7da28e-adf2-4bc6-a71c-09e8343b984d">
add hybrid fsdp ShardingStrategy
Author: pkumc
Pull request: yes

# What does this PR do?

Add [hybrid fsdp ShardingStrategy](https://pytorch.org/docs/stable/fsdp.html#torch.distributed.fsdp.ShardingStrategy) to Trainer.


## Who can review?
- trainer: @muellerzr and @pacman100



Comments:
@pacman100 Do you have time to review this PR?

> @pacman100 Do you have time to review this PR?

@muellerzr Could you review this PR?

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_26925). All of your documentation changes will be reflected on that endpoint.
[`core` / `gradient_checkpointing`] add support for old GC method
Author: younesbelkada
Pull request: yes

# What does this PR do?

Fixes https://github.com/huggingface/transformers/issues/27596

Models on the Hub still rely on the previous `_set_gradient_checkpointing` logic that has been removed and refactored in https://github.com/huggingface/transformers/pull/27073 

While models with code on the Hub maintainers should be aware that overwriting a private method is not a good practice, it has been widely common to declare `_set_gradient_checkpointing` for those models as previously it was the only way to enable GC on these models. Therefore I see this PR as a potential solution to overcome this issue that is likely to happen to many users.

I propose to still support that for few minor releases by displaying a warning that explains to users that this is going to be deprecated in the future.

cc @ArthurZucker 


Comments:
As discussed offline, let's just display a warning for now, merging!
Unable to register own custom tokenizer
Author: Ranitbag007
Pull request: no
Labels: name: New model

### Model description

@sgugger  I have tried to register my own tokenization model based on sentencepiece using CustomAITokenizer.register_for_auto_class("AutoTokenizer") . But I am falied to do so.

### Open source status

- [X] The model implementation is available
- [ ] The model weights are available

### Provide useful links for the implementation

https://huggingface.co/RANITBAG/CustomAItokenizer/tree/main . This the repo link 

Comments:
Hey 🤗  thanks for opening an issue! We try to keep the github issues for bugs/feature requests.
Could you ask your question on the [forum](https://discuss.huggingface.co/) instead? I'm sure the community will be of help!

Thanks!
Add RoCm scheduled CI & upgrade RoCm CI to PyTorch 2.1
Author: fxmarty
Pull request: yes

As per title

Comments:
_The documentation is not available anymore as the PR was closed or merged._

@fxmarty Could you verify that docker image could be built? For the AMD push CI, `docker/transformers-pytorch-amd-gpu` is not able to be built.

Note, `docker/transformers-pytorch-amd-gpu` is built on `[self-hosted, docker-gpu, amd-gpu, single-gpu, mi210]`, as the `ubuntu-latest` always gets not enough disk space error, no matter what I tried. I made it ran on `[self-hosted, docker-gpu, amd-gpu, single-gpu, mi210]` (maybe for a few days), but after that it gives permission error `EACCES: permission denied, mkdir '/home/github_actions/.docker/buildx/certs'`.

https://github.com/huggingface/transformers/actions/runs/6581781214

See my comment [here](https://github.com/huggingface/transformers/issues/26599#issuecomment-1748413935)

I suspect it is related the new way of AMD CI runners are set up 

It appears the runners on MI210 are down.

Hi @mfuntowicz When you get some bandwidth, could you check how to bring back mi210 runners? Would be great if we can move toward the daily AMD CI (complete suite) and see the status :-)

@ydshieh probably did most of it! :smile: 

thank you so much @ydshieh, you rock!

You're right, thank you both :raised_hands: :hugs:
Allow script tracing DINOv2
Author: Danil328
Pull request: no

I found PR to dinov2 "Pass scale factor as a tuple of floats to F.interpolate() to allow tracing."
https://github.com/facebookresearch/dinov2/pull/247

https://github.com/huggingface/transformers/blob/85fde09c97213bf7e8625f83096bb2a9e183f987/src/transformers/models/dinov2/modeling_dinov2.py#L104C19-L104C19



Comments:
I have exception now:
<img width="1153" alt="image" src="https://github.com/huggingface/transformers/assets/11178882/ce61c11a-9247-4045-8da4-5fdd9d3bb899">


Hi @Danil328, thanks for raising this issue! 

Could you make sure to follow the [issue template](https://github.com/huggingface/transformers/blob/main/.github/ISSUE_TEMPLATE/bug-report.yml) and include details of your running environment and a minimal reproducible snippet? 

From the error it looks like the `scale_factor` values being passed to `interpolate` is a NoneType.

Same problem in facebookresearch - https://github.com/facebookresearch/dinov2/issues/102
### Reproduction
```python
import torch
from transformers import AutoImageProcessor, AutoModel
from PIL import Image
import requests

url = 'http://images.cocodataset.org/val2017/000000039769.jpg'
image = Image.open(requests.get(url, stream=True).raw)

processor = AutoImageProcessor.from_pretrained('facebook/dinov2-base')
model = AutoModel.from_pretrained('facebook/dinov2-base')

inputs = processor(images=image, return_tensors="pt")
outputs = model(**inputs)
last_hidden_states = outputs.last_hidden_state

with torch.no_grad():
    example_input = torch.rand(1, 3, 224, 224, dtype=torch.float32, device="cuda")
    traced_model = torch.jit.trace(model.cuda(), example_input) # fails here
```
### Error
<img width="1162" alt="image" src="https://github.com/huggingface/transformers/assets/11178882/50aba4d4-5ad4-4398-9a26-5e63d337c61f">

### Expected behavior
Success

### Enviroment
`bash
python=3.8
torch==2.0.1
transformers==4.35.0
`

@Danil328 - thanks for providing the snippet! I've opened a PR which should resolve the issue
[wip: test doc-builder]
Author: mishig25
Pull request: yes

Closes #23625 
Testing https://github.com/huggingface/doc-builder/pull/373


Comments:
_The documentation is not available anymore as the PR was closed or merged._

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.
[wip doc buidler test]
Author: mishig25
Pull request: yes


Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_20092). All of your documentation changes will be reflected on that endpoint.

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.