noamaanMulla-03/transformers-issues huggingface dataset:
Add DINOv2 depth estimation
Author: NielsRogge
Pull request: yes

# What does this PR do?

Fixes #26057 

PR that implements a part of #25799. It extends the DPT framework to use the `AutoBackbone` class. Next it uses `Dinov2Backbone` to convert the DINOv2+DPT checkpoints released by the authors [here](https://github.com/facebookresearch/dinov2/tree/main#pretrained-heads---depth-estimation).

To do:

- [x] add test in common backbone tests file to verify `out_indices` are saved properly => done in #26094 
- [x] add support for transforms in `DPTImageProcessor`
- [x] add tests which test DPT with a backbone
- [x] add integration test and convert remaining checkpoints

Comments:
_The documentation is not available anymore as the PR was closed or merged._

@amyeroberts thanks for your review, I've addressed all comments. Feel free to merge :)

I've addressed all your comments, feel free to merge. To do:

- [ ] update all checkpoints on the hub to use `size_divisor` rather than `size_divisbility` in the preprocessor_config.json

@NielsRogge Thanks again for this addition - merged! I'll let you handle the updates to the configs on the hub. 

Hello guys. Thank you for adding this. I have a silly question that is there any plan that facebook will push this to the huggingface repo? Like `facebook/dpt-dinov2-large-nyu`? And could you can kindly inform me who is actually operating the companies accounts... Seems it is not all done by their side based on my guess.

Hi @Starlento,

All models are on the hub: https://huggingface.co/models?pipeline_tag=depth-estimation&other=dinov2&sort=trending. I'll open a PR to make this more explicit in the docs

Hey, guys, I want to know if it is possible to release the training part, and if it is possible, when will it be released?
Correct model_type in PretrainedConfig's to_dict
Author: fxmarty
Pull request: yes

As per title, now

```python
from transformers import AutoConfig, PretrainedConfig

cfg = AutoConfig.from_pretrained("bert-base-uncased")
config = PretrainedConfig.from_dict(cfg.to_dict())
config.model_type = "my-model"

print(config.to_dict()["model_type"])
```

rightfully yields `my-model`, while it use to give `""` (the class attribute value of PretrainedConfig).

I think instance attributes (if any) should take precedence over class attributes.

Related to https://github.com/huggingface/optimum/pull/1645

Comments:
Thanks for opening this PR @fxmarty! 

The reason this happens at the moment is that `model_type` is intended to be a class attribute, not an instance attribute i.e. to be consistent and shared for all instances of the config object and not modified instance to instance. What's the motivation for modifying it like this?  

Understood!
XLM question-answering pipeline is flacky
Author: fxmarty
Pull request: no

### System Info

transformers main, but tested on commits in the last three weeks, same issue

### Who can help?

_No response_

### Information

- [X] The official example scripts
- [ ] My own modified scripts

### Tasks

- [X] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

```python
for i in range(50):
    from transformers import AutoTokenizer, AutoModelForQuestionAnswering, pipeline
    import torch
    
    model = AutoModelForQuestionAnswering.from_pretrained("hf-internal-testing/tiny-random-XLMModel")
    tokenizer = AutoTokenizer.from_pretrained("hf-internal-testing/tiny-random-XLMModel")
    pipe = pipeline("question-answering", model=model, tokenizer=tokenizer)
    question = "Whats my name?"
    context = "My Name is Philipp and I live in Nuremberg."
    outputs = pipe(question, context)
```

sometimes fail with
```
Traceback (most recent call last):
  File "<tmp 4>", line 23, in <module>
    outputs = pipe(question, context)
  File "/home/fxmarty/hf_internship/transformers/src/transformers/pipelines/question_answering.py", line 393, in __call__
    return super().__call__(examples[0], **kwargs)
  File "/home/fxmarty/hf_internship/transformers/src/transformers/pipelines/base.py", line 1132, in __call__
    return next(
  File "/home/fxmarty/hf_internship/transformers/src/transformers/pipelines/pt_utils.py", line 125, in __next__
    processed = self.infer(item, **self.params)
  File "/home/fxmarty/hf_internship/transformers/src/transformers/pipelines/question_answering.py", line 563, in postprocess
    "start": np.where(char_to_word == token_to_orig_map[s])[0][0].item(),
KeyError: 5
```

### Expected behavior

no error. I can have a look if I have time

Comments:
Hi @fxmarty - thanks for raising this! 

To help with debugging - has this been observed with other checkpoints or only the tiny random ones for testing?
[`SpeechT5Tokenization`]  Add copied from and fix the `convert_tokens_to_string` to match the fast decoding scheme
Author: ArthurZucker
Pull request: yes

# What does this PR do?
Fixes #26547

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_28522). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
[SpeechT5] Decode function strips space after special token
Author: xenova
Pull request: no

### System Info

- `transformers` version: 4.34.0.dev0
- Platform: Windows-10-10.0.22621-SP0
- Python version: 3.8.1
- Huggingface_hub version: 0.16.4
- Safetensors version: 0.3.3
- Accelerate version: 0.23.0
- Accelerate config:    not found
- PyTorch version (GPU?): 1.12.1+cu116 (True)
- Tensorflow version (GPU?): not installed (NA)
- Flax version (CPU?/GPU?/TPU?): not installed (NA)
- Jax version: not installed
- JaxLib version: not installed
- Using GPU in script?: <fill in>
- Using distributed or parallel set-up in script?: <fill in>


### Who can help?

@ArthurZucker 

### Information

- [ ] The official example scripts
- [ ] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

1. First load the speecht5 tokenizer
```py
from transformers import SpeechT5Tokenizer
tokenizer = SpeechT5Tokenizer.from_pretrained('microsoft/speecht5_tts')
ids = tokenizer.encode("a = b")
# [4, 7, 4, 3, 4, 25, 2]    (3 = unknown token, 4 = metaspace)
```

2. Convert ids to tokens, showing that metaspace is added before and after the unknown token
```py
tokenizer.convert_ids_to_tokens(ids)
# ['▁', 'a', '▁', '<unk>', '▁', 'b', '
']    (metaspace before and after unknown)
```

3. Decode, showing the space being removed after the unknown token.
```py
tokenizer.decode(ids)
# "a <unk>b
"    (no space after <unk>)
```


Seems to be caused by this `strip`:
https://github.com/huggingface/transformers/blob/9ed538f2e67ee10323d96c97284cf83d44f0c507/src/transformers/models/speecht5/tokenization_speecht5.py#L192

Related to https://github.com/huggingface/tokenizers/issues/826

### Expected behavior

The decoded string should be `"a <unk> b
"` (w/ a space after <unk>)

Comments:
Thanks for reporting! This is happening because:
```python 
    def convert_tokens_to_string(self, tokens):
        """Converts a sequence of tokens (string) in a single string."""
        current_sub_tokens = []
        out_string = ""
        for token in tokens:
            # make sure that special tokens are not decoded using sentencepiece model
            if token in self.all_special_tokens:
                out_string += self.sp_model.decode(current_sub_tokens) + token
                current_sub_tokens = []
            else:
                current_sub_tokens.append(token)
        out_string += self.sp_model.decode(current_sub_tokens)
        return out_string.strip()
```

passes the inputs to the sentencepiece model after they are split, thus what the `self.sp_model` sees is the following:
1.  ['▁', 'a', '▁']
2.  ['▁', 'b']
and thus the prefix space will be removed for both. 
This needs a fix 🎐
Inconsistent check for is_accelerate_available() in transformers.training_args
Author: faph
Pull request: no

### System Info

- `transformers` version: 4.36.2
- Platform: Linux-4.18.0-477.21.1.el8_8.x86_64-x86_64-with-glibc2.28
- Python version: 3.11.2
- Huggingface_hub version: 0.20.2
- Safetensors version: 0.4.1
- Accelerate version: not installed
- Accelerate config: not found
- PyTorch version (GPU?): 2.0.1+cu117 (False)
- Tensorflow version (GPU?): not installed (NA)
- Flax version (CPU?/GPU?/TPU?): not installed (NA)
- Jax version: not installed
- JaxLib version: not installed
- Using GPU in script?: unknown
- Using distributed or parallel set-up in script?: unknown

Output of `pip show accelerate`:

```
Name: accelerate
Version: 0.20.3
(...)
```



### Who can help?

@muellerz @pacman100 

### Information

- [ ] The official example scripts
- [ ] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

1. Pip install `transformers 4.36.2` and `accelerate 0.20.3`
2. Instantiate `transformers.TrainingArguments`

This raises something like this:

```
    @cached_property
    def _setup_devices(self) -> "torch.device":
        requires_backends(self, ["torch"])
        logger.info("PyTorch: setting up devices")
        if not is_sagemaker_mp_enabled():
            if not is_accelerate_available(min_version="0.20.1"):
                raise ImportError(
                    "Using the `Trainer` with `PyTorch` requires `accelerate>=0.20.1`: Please run `pip install transformers[torch]` or `pip install accelerate -U`"
                )
>           AcceleratorState._reset_state(reset_partial_state=True)
E           NameError: name 'AcceleratorState' is not defined
```

This because the import of `AcceleratorState` is conditional upon `accelerate` with minimum version `0.21.0`. See https://github.com/huggingface/transformers/blob/v4.36.2/src/transformers/utils/import_utils.py#L684

### Expected behavior

Consistent min version check for `accelerate` and successful `TrainingArguments` instantiation.

Comments:
I think this is already fixed here: https://github.com/huggingface/transformers/blob/002566f398cd6dbf7053a89c26646ac45be540f4/src/transformers/training_args.py#L1830

Hi @faph, thanks for raising this issue! 

Yes, as you identified, a fix was added with #28171 which will be part of the next release. 

Thanks @amyeroberts
Tokenizer should be serializable
Author: hk6an6
Pull request: no

### System Info

- transformers.__version__: '4.36.1'
- platform: macOS 14.2.1 (23C71)
- Python version: Python 3.11.6

### Who can help?

@ArthurZucker

### Information

- [ ] The official example scripts
- [ ] My own modified scripts

### Tasks

- [x] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

```
!pip install -U sentence-transformers
from sentence_transformers import SentenceTransformer
model = SentenceTransformer('paraphrase-distilroberta-base-v1', device='mps')
model.max_seq_length = 384
model.save('my_path')
```

### Expected behavior

`transformers/tokenization_utils_base.py` fails to serialize the local variable `tokenizer_config`. and breaks `model.save`. `model.save` should succeed.

Comments:
It seems the problem goes away if I change from `paraphrase-distilroberta-base-v1` to `paraphrase-distilroberta-base-v2`. Could it be that the tokenizer_config for v1 is incompatible with json.dumps?

Hey! This seems to be related to `sentence-transfromers`, as tokenizers are serializable. 
Could you provide the stactrace and open an issue on the [sentence-transformers](https://github.com/UKPLab/sentence-transformers) repo? 

Hi Arthur!

Yes, I can provide a stack trace.

I’ll close this issue and open a new issue on the sentence-transformers
repo.

On Mon, Jan 15, 2024 at 02:32 Arthur ***@***.***> wrote:

> Hey! This seems to be related to sentence-transfromers, as tokenizers are
> serializable.
> Could you provide the stactrace and open an issue on the
> sentence-transformers <https://github.com/UKPLab/sentence-transformers>
> repo?
>
> —
> Reply to this email directly, view it on GitHub
> <https://github.com/huggingface/transformers/issues/28502#issuecomment-1891827117>,
> or unsubscribe
> <https://github.com/notifications/unsubscribe-auth/AARYISEUDT4D3TRGGZYR7GLYOUAUDAVCNFSM6AAAAABB2R4YT6VHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMYTQOJRHAZDOMJRG4>
> .
> You are receiving this because you authored the thread.Message ID:
> ***@***.***>
>


[stacktrace.txt](https://github.com/huggingface/transformers/files/13959334/stacktrace.txt)

Keeping my word, this is the stack trace. I'll cose this ticket and open one in the sentence-transformers repo.

https://github.com/UKPLab/sentence-transformers/issues/2418
Pipeline batching with ZeroShotImageClassificationPipeline outputs less items per iteration than expected
Author: ryan-caesar-ramos
Pull request: no

### System Info

- `transformers` version: 4.36.2
- Platform: Linux-6.5.0-14-generic-x86_64-with-glibc2.35
- Python version: 3.11.5
- Huggingface_hub version: 0.19.4
- Safetensors version: 0.4.0
- Accelerate version: 0.24.1
- Accelerate config: 	not found
- PyTorch version (GPU?): 2.1.1+cu121 (True)
- Tensorflow version (GPU?): not installed (NA)
- Flax version (CPU?/GPU?/TPU?): not installed (NA)
- Jax version: not installed
- JaxLib version: not installed
- Using GPU in script?: yes
- Using distributed or parallel set-up in script?: no

### Who can help?

@Narsil 

### Information

- [ ] The official example scripts
- [X] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [X] My own task or dataset (give details below)

### Reproduction

```python
from transformers import pipeline
from datasets import load_dataset
import numpy as np
from transformers.pipelines.pt_utils import KeyDataset

pipe = pipeline('zero-shot-image-classification', model='openai/clip-vit-base-patch16', device=0)
dataset = load_dataset('mnist', split='test')
for out in pipe(KeyDataset(dataset, "image"), candidate_labels=range(10), batched=True, batch_size=1024):
  break

print(len(out))
```

### Expected behavior

I would have assumed that `out` would have 10 classes * 1024 images in the batch = 10240 items in it, but it only has 10. Maybe I'm misinterpreting what batching with pipelines does?

Comments:
Batch_size is transparent. The model will run the inference batched, but the pipeline still outputs results one by one, meaning you can modify the batch_size according to the actual hardware you are using without actually needing to unpack results manually (it also allows running next batch, before you even finished unpacking the results manually, leading to better GPU usage.

Thanks! That clears things up
symbolic_trace: add past_key_values, llama, sdpa support
Author: fxmarty
Pull request: yes

This PR:
* Allows to use `transformers.utils.fx.symbolic_trace` with `past_key_values` inputs for some architectures (currently opt, llama)
* Adds llama support for symbolic_trace.
* Adds SDPA support for symbolic_trace.


Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_28447). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
Tokenizer kwargs in textgeneration pipe
Author: farzadsbakhtiar
Pull request: yes

# What does this PR do?

Support tokenizer arguments in `text-generation` pipeline `__call__()`

Fixes # (issue)
#27869

## Before submitting
- [ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [x] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#start-contributing-pull-requests),
      Pull Request section?
- [x] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [x] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/main/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/main/docs#writing-source-documentation).
- [x] Did you write any new necessary tests?


## Who can review?

@ArthurZucker 



Comments:
Thank you 🤗. Anything specific I can help with?

Not sure since it's merged 🤗
Fix SDPA tests
Author: fxmarty
Pull request: yes

@ydshieh testing on a T4 (& cpu), all tests pass now. Most of the failing ones were due to the fact that we run the CI on a T4 GPU, that does not support bf16.

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_28552). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

Thank you @amyeroberts
rerun notebook 02-transformers
Author: nikitajz
Pull request: yes

Just a notebook rerun (some cells were not executed in the original one).
Few minor changes:
- removed installation cell output for readability (it can't be collapsed in GitHub notebook viewer) 
- added prints with tensors details in the last cell (DE BERT model)

I'm unable to attach nbdiff as html (due to github limitations), hence adding both as zipped 
[nbdiff_02-transformers.zip](https://github.com/huggingface/transformers/files/4623078/nbdiff_02-transformers.zip) and the link to [Gdrive](https://drive.google.com/open?id=1VlucbmczZjUOI8nuCJCAxFoKgjFsJKw0) for your convenience.


Comments:
Thanks @nikitajz, 

LGTM ! 👍
Fix Switch Transformers When sparse_step = 1
Author: agemagician
Pull request: yes

# What does this PR do?

In case sparse_step = 1, the current code will not work.
Because anything % 1 will always equal 0, even though there should be a spare layer at each block.


Fixes # (issue)
I didn't open an issue. I just solved the problem with this PR.

## Before submitting
- [ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [X] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#start-contributing-pull-requests),
      Pull Request section?
- [ ] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [ ] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/main/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/main/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests?


## Who can review?

@ArthurZucker , @younesbelkada , @younesbelkada , @sgugger 



Comments:
Fix paths to AI Sweden Models reference and model loading
Author: JuanFKurucz
Pull request: yes

# What does this PR do?

This PR fixes the paths to the correct models of AI Sweden Models, as they migrated their models to a different account https://huggingface.co/AI-Sweden-Models.

## Before submitting
- [X] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).

## Who can review?

Anyone in the community is free to review the PR once the tests have passed. Feel free to tag
members/contributors who may be interested in your PR.

Models:

- text models: @ArthurZucker and @younesbelkada

Documentation: @stevhliu and @MKhalusova


Comments:
Sharded Weight Files Not Uploaded by `_push_from_checkpoint` function in DeepSpeed ZeRO Stage 3 Training
Author: e-mon
Pull request: no
Labels: name: DeepSpeed

### System Info

- `transformers` version: 4.33.1
- Platform: Linux-4.18.0-372.9.1.el8.x86_64-x86_64-with-glibc2.28
- Python version: 3.10.10
- Huggingface_hub version: 0.17.1
- Safetensors version: 0.3.3
- Accelerate version: 0.22.0
- Accelerate config:    not found
- PyTorch version (GPU?): 2.0.1+cu117 (False)
- Tensorflow version (GPU?): not installed (NA)
- Flax version (CPU?/GPU?/TPU?): not installed (NA)
- Jax version: not installed
- JaxLib version: not installed
- Using GPU in script?: yes
- Using distributed or parallel set-up in script?: yes, deepspeed

### Who can help?

@muellerz and @pacman100

### Information

- [ ] The official example scripts
- [X] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

I attempted to train "https://huggingface.co/codellama/CodeLlama-7b-hf" using DeepSpeed ZeRO Stage 3, but found that the `_push_from_checkpoint` function was not working properly, and the weights file was not being uploaded. I was able to confirm that the sharded weights file, named like `pytorch_model-00001-of-00002.bin`, was being saved in the checkpoint directory.

From what I can tell after looking at the code below, it does not seem that such divided files are included in the upload target. Is this the expected behavior? If the divided files should be included in the target, I am considering creating a Pull Request.
https://github.com/huggingface/transformers/blob/main/src/transformers/trainer.py#L3624

### Expected behavior

all checkpoints to be uploaded to the Hugging Face Hub.

Comments:
Yes, you are correct. Currently, the sharded weights aren't being copied to the main output directory and as such not being uploaded to the Hub. It would be great if you could submit a PR resolving it.

Hello @ArthurZucker, this is not limited to DeepSpeed, even when doing normal training with say DDP, if the model gets sharded, it won't be uploaded to the hub at present.

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.