noamaanMulla-03/transformers-issues huggingface dataset:
Fix license
Author: MedAymenF
Pull request: yes

# What does this PR do?


Fixes the license for Fuyu. See [blog](https://www.adept.ai/blog/fuyu-8b) and [Model Card](https://huggingface.co/adept/fuyu-8b#model-description).


## Before submitting
- [x] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [ ] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#start-contributing-pull-requests),
      Pull Request section?
- [ ] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [ ] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/main/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/main/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests?



Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_26931). All of your documentation changes will be reflected on that endpoint.
Pin Keras for now
Author: Rocketknight1
Pull request: yes

With Keras 3 on the horizon, let's pin Keras versions to ensure things don't break until we can transition properly

Comments:
There are a few `pip install -U --upgrade-strategy eager tensorflow_probability` in `.circleci/create_circleci_config.py`. I guess we will need to pin them too. But not sure if we already know the corresponding version.

@ydshieh yeah, `tensorflow_probability` versions don't seem to be linked to TF versions. It also doesn't seem to use Keras much - hopefully we can leave it unpinned for now, and do a PR to pin it if it becomes a problem later!

(remark: I did see some failing before when `tensorflow_probability` complained about the TF version is not compatible with it - or vice versa 😅 )
Generate: update basic llm tutorial
Author: gante
Pull request: yes

# What does this PR do?

This PR updates the basic [LLM tutorial](https://huggingface.co/docs/transformers/llm_tutorial) in the following aspects:
1. Adds a short section right at the top on how to do batched generation. This was made to address https://github.com/huggingface/transformers/issues/26061 (and other similar issues in the past)
2. Updates the model used in the examples from OpenLlama (permissive licenses, but deprecated) to Mistral (latest 7B model with top performance and permissive licenses)
3. Adds prompting to the list of common issues (was a TODO)
4. Adds pointers to recent guides about prompting and performance


Fixes https://github.com/huggingface/transformers/issues/26061

Comments:
_The documentation is not available anymore as the PR was closed or merged._
CodeLlama FastTokenizer Infilling Format Bug with `suffix_first=True`
Author: andyl98
Pull request: no

### System Info

Suppose the code is:
```
from transformers import AutoTokenizer
tokenizer = AutoTokenizer.from_pretrained("codellama/codellama-7b-hf")
input = tokenizer("# print hello world<FILL_ME>orld')", suffix_first=True)
tokenizer.batch_decode([input["input_ids"]])[0]
```
The output will be
```
"
 <PRE> <SUF> # print hello world <MID>orld')"
```
Which is incorrect because the prefix is put into the suffix field. 
The desired output should be `"
 <PRE> <SUF>orld') <MID> # print hello world"`

If we don't use the fast tokenizer
```
from transformers import AutoTokenizer
tokenizer = AutoTokenizer.from_pretrained("codellama/codellama-7b-hf", use_fast=False)
input = tokenizer("# print hello world<FILL_ME>orld')", suffix_first=True)
tokenizer.batch_decode([input["input_ids"]])[0]
```
The output will be correct `"
▁<PRE>▁<SUF>orld')▁<MID> # print hello world"`

The issue is on [this line](https://github.com/huggingface/transformers/blob/5af2c6269672cda01c24ad48fab13f14a3ffb746/src/transformers/models/code_llama/tokenization_code_llama_fast.py#L292) Where the `$A` and `$B` should be swapped.

@ArthurZucker 

### Who can help?

_No response_

### Information

- [ ] The official example scripts
- [X] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [x] My own task or dataset (give details below)

### Reproduction

Suppose the code is:
```
from transformers import AutoTokenizer
tokenizer = AutoTokenizer.from_pretrained("codellama/codellama-7b-hf")
input = tokenizer("# print hello world<FILL_ME>orld')", suffix_first=True)
tokenizer.batch_decode([input["input_ids"]])[0]
```
The output will be
```
"
 <PRE> <SUF> # print hello world <MID>orld')"
```
Which is incorrect because the prefix is put into the suffix field. 
The desired output should be `"
 <PRE> <SUF>orld') <MID> # print hello world"`

If we don't use the fast tokenizer
```
from transformers import AutoTokenizer
tokenizer = AutoTokenizer.from_pretrained("codellama/codellama-7b-hf", use_fast=False)
input = tokenizer("# print hello world<FILL_ME>orld')", suffix_first=True)
tokenizer.batch_decode([input["input_ids"]])[0]
```
The output will be correct `"
▁<PRE>▁<SUF>orld')▁<MID> # print hello world"`

### Expected behavior

Output should be `"
 <PRE> <SUF>orld') <MID> # print hello world"` when using `suffix_first=True`.

Comments:
Corrected modalities description in README_ru.md
Author: letohx
Pull request: yes

Description: 
- [x]  This PR fixes a typo in README_ru.md.

This pull request addresses an issue with the modalities description in the README_ru.md. The modalities' descriptions have been corrected to provide a more accurate and grammatically correct representation of the supported tasks for text, images, and audio. This change improves the clarity of the information provided in the README_ru.md.

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_26913). All of your documentation changes will be reflected on that endpoint.
Inconsistency between CLIPTokenizer and CLIPTokenizerFast
Author: sayanshaw24
Pull request: no

### System Info

- `transformers` version: 4.30.2
- Platform: Windows-10-10.0.22621-SP0
- Python version: 3.10.11
- Huggingface_hub version: 0.15.1
- Safetensors version: 0.3.1
- PyTorch version (GPU?): 1.12.1+cpu (False)
- Tensorflow version (GPU?): 2.9.0 (False)
- Flax version (CPU?/GPU?/TPU?): not installed (NA)
- Jax version: not installed
- JaxLib version: not installed
- Using GPU in script?: no
- Using distributed or parallel set-up in script?: no

### Who can help?

@ArthurZucker and @younesbelkada

### Information

- [ ] The official example scripts
- [X] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [X] My own task or dataset (give details below)

### Reproduction

1. Import CLIPTokenizer and CLIPTokenizerFast
2. Create the "openai/clip-vit-base-patch32" pretrained tokenizers
3. Compare input ids of tokenizers with some example text using words with apostrophes (e.g., you're, don't, her's, etc.)
```
from transformers import CLIPTokenizer, CLIPTokenizerFast

slow_tokenizer = CLIPTokenizer.from_pretrained("openai/clip-vit-base-patch32")
fast_tokenizer = CLIPTokenizerFast.from_pretrained("openai/clip-vit-base-patch32")

test_str = "you're, don't, her's"
clip_slow_out = slow_tokenizer(test_str)
clip_fast_out = fast_tokenizer(test_str)

slow_input_ids = clip_slow_out ['input_ids']
fast_input_ids = clip_fast_out ['input_ids']

np.testing.assert_array_equal(slow_input_ids, fast_input_ids)
```

### Expected behavior

The input ids should be the same for both regular and fast tokenizers, however CLIPTokenizer tokenizes apostrophes separately (['you', ''', 're']) but CLIPTokenizerFast tokenizes together (['you', ''re']).

Which of these results are the correct expected values? They should be consistent, should they not? 

Comments:
Hey! This is a duplicate of #22166 I believe!
Add BEiTv3
Author: NielsRogge
Pull request: no
Labels: name: New model

### Model description

Microsoft just open-sourced BEiTv3: https://github.com/microsoft/unilm/tree/master/beit3

This is a very powerful vision-language model that can be used as backbone for a variety of downstream tasks, from image classification to VQA to object detection.

Time to add it to HF Transformers! :)

### Open source status

- [X] The model implementation is available
- [X] The model weights are available

### Provide useful links for the implementation

https://github.com/microsoft/unilm/tree/master/beit3

Comments:
I will start working on adding the model !

If required I can help as well

hello, I want to test the zero shot Image text retrieval of the model on some images and texts, can you help me ?


Is this PR going to be merged soon for Beit3 modules to be available for everyone?
Simplify cache interface for Falcon model
Author: RezaYazdaniAminabadi
Pull request: no

### Feature request

Hi, 

I have been looking at the Falcon modeling file and I am seeing that the cache interface might have got a bit too complicated! It, however, could have been simplified a bit. 
There are the two functions, `_convert_cache_to_standard_format` and `_convert_to_rw_cache` which are called after and before the network's forward, and if I understand correctly it is mostly for finding the current generation context length (as mentioned in the comment of [this line](https://github.com/huggingface/transformers/blob/main/src/transformers/models/falcon/modeling_falcon.py#L885)).
By looking at the code, I realized that we can always get the context-length by just taking the `shape[-2]` form the layer_past tuple. By doing this, we can just remove these two function calls to change the format of the kv-cache.
If this makes sense, I can make a PR and fix this, otherwise, please feel free to close this!
Thanks,
Reza

### Motivation

Simplifying the cache interface for Falcon model.

### Your contribution

Yes, I can make a PR if neede.

Comments:
cc @Rocketknight1 

Hi @RezaYazdaniAminabadi, yes, the cache is still a bit of a mess there! Falcon started as custom code, and I wrote the Hugging Face port of that custom code. The custom code copied the strange cache shapes from BLOOM, but there were issues in the implementation that severely hurt generation performance. In the port, I focused on getting generation performance up while ensuring correctness, but I was aware that the cache could probably do with a more fundamental rewrite.

If you think you can clean up the cache code, I'd be happy to review a PR! I suspect you could probably delete quite a lot of the cache handling code, but you might have to make changes in other modules to compensate for that.

Cool, then I add some changes and then add you as the reviewer to check this.
Thanks.

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.
ODD BUG=latest version sentencepiece + latest version torch + old glibc
Author: DragonMengLong
Pull request: no

### System Info

- `transformers` version: 4.34.0.dev0
- Platform: Linux-3.10.0-229.el7.x86_64-x86_64-with-glibc2.18
- Python version: 3.10.12
- Huggingface_hub version: 0.16.4
- Safetensors version: 0.3.3
- Accelerate version: 0.22.0
- Accelerate config:    not found
- PyTorch version (GPU?): 2.0.1+cu117 (True)
- Tensorflow version (GPU?): not installed (NA)
- Flax version (CPU?/GPU?/TPU?): not installed (NA)
- Jax version: not installed
- JaxLib version: not installed
- Using GPU in script?: <fill in>
- Using distributed or parallel set-up in script?: <fill in>

### Who can help?

@ArthurZucker and @younesbelkada

### Information

- [X] The official example scripts
- [ ] My own modified scripts

### Tasks

- [X] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

- A very odd bug happened after i  pip install the requirements in the examples/pytorch/translation and try to `from transformers import T5ForConditionalGeneration`
- The bug message is `dlopen: cannot load any more object with static TLS`
- I found that  it could be reproduced everytime when i using the lastest version of sentencepiece(0.1.99) and the latest version of pytorch(2.0.1) on a old glibc(2.18)
- Because sometimes it's very difficult to update the glibc, it could be temporarily addressed by lower the pytorch version.

### Expected behavior

I guess it would be better if It could be warned in the readme

Comments:
This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.
Dinov2 for depth estimation
Author: rfan-debug
Pull request: no

### Feature request

Dinov2's original repo has an example using Dinov2 backbone + DPT head for depth estimation [notebook link](https://github.com/facebookresearch/dinov2/blob/main/notebooks/depth_estimation.ipynb). If we can integrate it into `transformers` repo by adding a class `Dinov2ForImageDepthEstimation` and let `forward` method return `DepthEstimatorOutput`, we'll have a unified output interface across all depth estimation models. By doing this, we can easily chain this powerful depth estimation method together with other models under `transformers`'s pipelines. 




### Motivation

This would be a very great feature for many production use cases or research problems. One example is camera angle estimation from a 2D image, in which reliable depth information are critical. In my limited test cases, using dinov2+DPT head to run depth estimation is way better than the existing [DPT model](https://huggingface.co/docs/transformers/main/model_doc/dpt) itself. 

### Your contribution

I can submit a PR to add this feature if other professional developers don't have the bandwidth to deal with it. (I am relatively new to `transformers`'s develop workflow though.)

Comments:
Hi @rfan-debug, this would be a great contribution! 

If you'd like to open a PR we'd be happy to review and answer any questions if you need help. 

cc @rafaelpadilla

Hi,

So I saw they released the DINOv2 checkpoints with a DPT head: https://github.com/facebookresearch/dinov2#pretrained-heads---depth-estimation. I do have a [PR](https://github.com/huggingface/transformers/pull/25799) which extends DPT to leverage the `AutoBackbone` API. This means that the DPT head can be used together with any backbone (like ViT, DINOv2, etc.). This way, we could just do the following:

```
from transformers import Dinov2Config, DPTConfig, DPTForDepthEstimation

backbone_config = Dinov2Config(num_hidden_layers=2, num_attention_heads=4, out_features=["stage1", "stage2", "stage3", "stage4")
config = DPTConfig(backbone_config=backbone_config)
model = DPTForDepthEstimation(config)
```

=> so would be great to leverage this instead of adding a standalone `Dinov2ForDepthEstimation`.


@NielsRogge Leveraging the AutoBackbone API is a great idea. Thanks for your advice and contributions! I'll follow your code examples. 

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.
upscaling segmentation mask OneFormer
Author: nikky4D
Pull request: no

I would like to scale the segmentation mask produced by OneFormer. 

My gpu is small, so I downscale my original image to a  smaller image, which I can then pass through OneFormer. However, how do I scale up the results back to the original image size? Is there a command in OneFormer or SegFormer that allows me to set the input and output size?

Comments:
Hi, thanks for raising an issue! 

This is a question best placed in our [forums](https://discuss.huggingface.co/). We try to reserve the github issues for feature requests and bug reports.

If using the model's image processor, you can change the size of the post-processed mask by specifying `target_sizes` in the post processing methods e.g. [post_process_semantic_segmentation](https://github.com/huggingface/transformers/blob/842e99f1b9ee2a0fa239997ef695c5ed0bd77195/src/transformers/models/oneformer/image_processing_oneformer.py#L1069)

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.
Why do I get completely subpar results on stsb datasets using the sample programs in “Text classification examples”
Author: LLIKKE
Pull request: no

### System Info

transformers v4.44,0

### Who can help?

_No response_

### Information

- [X] The official example scripts
- [ ] My own modified scripts

### Tasks

- [X] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

python run_glue.py \
  --model_name_or_path bert-base-cased \
  --task_name stsb \
  --do_train \
  --do_eval \
  --max_seq_length 128 \
  --per_device_train_batch_size 32 \
  --learning_rate 2e-5 \
  --num_train_epochs 3 \
  --overwrite_output_dir \
  --fp16 \
  --output_dir checkpoint/

### Expected behavior

python run_glue.py \
  --model_name_or_path bert-base-cased \
  --task_name stsb \
  --do_train \
  --do_eval \
  --max_seq_length 128 \
  --per_device_train_batch_size 32 \
  --learning_rate 2e-5 \
  --num_train_epochs 3 \
  --overwrite_output_dir \
  --fp16 \
  --output_dir checkpoint/

the train detail:
***** train metrics *****
  epoch                    =        3.0
  train_loss               =   -20.0247
  train_runtime            = 0:00:55.21
  train_samples            =       5749
  train_samples_per_second =    312.365
  train_steps_per_second   =       9.78
the eval detail:
***** eval metrics *****
  epoch                   =        3.0
  eval_combined_score     =       0.71
  eval_loss               =   -21.2325
  eval_pearson            =     0.6659
  eval_runtime            = 0:00:26.09
  eval_samples            =       1500
  eval_samples_per_second =     57.477
  eval_spearmanr          =     0.7542
  eval_steps_per_second   =      7.204

help me! thank you!

Comments:
Hi @LLIKKE, thanks for raising an issue! 

This is a question best placed in our [forums](https://discuss.huggingface.co/). We try to reserve the github issues for feature requests and bug reports.

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.
[WIP] Add mixformer_sequential
Author: sgunasekar
Pull request: yes

# What does this PR do?

Adds mixformer_sequential model

Fixes # (issue)


## Before submitting
- [ ] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#start-contributing-pull-requests),
      Pull Request section?
- [ ] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [ ] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/main/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/main/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests?


## Who can review?

Anyone in the community is free to review the PR once the tests have passed. Feel free to tag
members/contributors who may be interested in your PR.
@ArthurZucker and @younesbelkada

Comments:
Hey! Thanks for opening a PR! Could you share links to the original code / original paper / released weights (or if there was already an issue in `transformers` that was requesting this? 
Is it https://arxiv.org/abs/2305.15896 ? 

Also would highly recommend to try to first [share the model on the hub](https://huggingface.co/docs/transformers/custom_models) follow the tutorial, as it will be a lot easier to distribute it, and you wont' have to pass all our CIs! 😉 

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.
support loading pretrained model from fsspec paths
Author: leoleoasd
Pull request: no

### Feature request

Make `from_pretrained` works with fsspec paths.

```
tokenizer = BartTokenizerFast.from_pretrained("hdfs://..../bert-base-uncased")
```

### Motivation

To make transformers suitable for usage in the cloud or in clusters where files should be stored in HDFS or S3.

### Your contribution

I'll be willing to submit a PR but `from_pretrained` is a very complicated function so I may need assistance.

Comments:
Thank you, but we're not interested in expanding support to other tools than the Hugging Face Hub :-)

I'd like to use Hub too, but our cluster have no public internet access...

> I'd like to use Hub too, but our cluster have no public internet access...

hai, can you  help  how  to load the pretrain model  from hdfs path

Still no way to do that. I have to use pytorch_lightning's ModelCheckpoint callback to load & save.
You can pre-process your pretrained model into a pytorch `.pt` file then load it with `torch.load(fsspec.open('xxx'))`.

> 

thanks so much for your reply ,  can you teach more detail please ,thanks

hha, I have got an simple solution without changing  the source code ,  just use spark submit :
--conf spark.yarn.dist.archives=hdfs://your_own_hdfs_path/your_pretraining_model.zip#your_pretraining_model
 



@NYcleaner thanks for this idea! How would you be loading (referencing) the model then inside the spark job? , as a hdfs://your_own... string ?

> @NYcleaner thanks for this idea! How would you be loading (referencing) the model then inside the spark job? , as a hdfs://your_own... string ?

In case you have not yet figured out. When you provide `--conf spark.yarn.dist.archives=hdfs://your_own_hdfs_path/your_pretraining_model.zip#your_pretraining_model` the archive is downloaded extracted and symlinked as `your_training_model` in the current directory. When passing the path you should be able to do just `./your_pretraining_model`.

That’s amazing - thanks for pointing it out!

> Am 19.10.2023 um 20:03 schrieb Pavan Lanka ***@***.***>:
> 
> ﻿
> @NYcleaner thanks for this idea! How would you be loading (referencing) the model then inside the spark job? , as a hdfs://your_own... string ?
> 
> In case you have not yet figured out. When you provide --conf spark.yarn.dist.archives=hdfs://your_own_hdfs_path/your_pretraining_model.zip#your_pretraining_model the archive is downloaded extracted and symlinked as your_training_model in the current directory. When passing the path you should be able to do just ./your_pretraining_model.
> 
> —
> Reply to this email directly, view it on GitHub, or unsubscribe.
> You are receiving this because you commented.