noamaanMulla-03/transformers-issues huggingface dataset:
wav2vec processor batching logic is too restrictive
Author: LWprogramming
Pull request: no

### System Info

transformers version at the time of writing is `4.26.1`


### Who can help?

_No response_

### Information

- [ ] The official example scripts
- [ ] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

```python
# !pip install transformers torch # in jupyter notebook
from transformers import Wav2Vec2Processor
import torch
import numpy as np

batch = 4

# create Wav2Vec2Processor
processor = Wav2Vec2Processor.from_pretrained("facebook/hubert-large-ls960-ft")
# generate random input tensor
input_tensor = torch.tensor(np.random.randn(batch, 10, 10))
# pass input tensor through processor
output = processor(input_tensor, return_tensors="pt")
print(output["input_values"].shape) # 1 x 4 x 10 x 10
```


### Expected behavior

It seems reasonable that an input could be of shape `batch x d_1 x d_2 ...` and I'd expect the output to have the same shape. However, [here](https://github.com/huggingface/transformers/blob/main/src/transformers/models/wav2vec2/feature_extraction_wav2vec2.py#L184) the code has an extra check for type list or tuple that results in it misinterpreting the input as a single example.

Side note: I'm unsure what to infer from the type checking logic because it doesn't match the type hints i.e. `tuple` isn't supposed to be possible here anyways, according to the `__call__` type hint. I did check some other examples of `is_batched` appearing in the `src/transformers/models` directory and they look similar but unexpected.

Comments:
cc @sanchit-gandhi @ArthurZucker 

Hey @LWprogramming! Thanks for the comprehensive issue description - I agree that the logic for checking if the input `is_batched` is broken when the input is a batched numpy array, e.g. the feature extractor **should** set `is_batched=True` when the numpy array is 2-d, but currently does not:
https://github.com/huggingface/transformers/blob/57f25f4b7fb85ff069f8701372710b2a3207bf2d/src/transformers/models/wav2vec2/feature_extraction_wav2vec2.py#L184-L187

Would you like to open a PR to fix this? 🤗 We can just do one additional check to set `is_batched = True` if the input is a 2-d numpy array. Note that it should be 2-d with dims [batch, audio_input] and not 3-d since we only expect mono channel input to the feature extractor.

Hey @LWprogramming! Just checking-in to see whether you'd like to open a PR to fix the issue you uncovered? Think you're in a good position to submit a clean fix! 🤗

Hi! I'll take care of it, got preoccupied with some irl stuff that came up the past few weeks but things should be settling down soon :)

That's awesome @LWprogramming! Excited for the PR 🤗 Feel free to tag me as soon as it's ready and I'll get you a review

marking as still active, just fixing up the PR
Request to upgrade torch version in vision model
Author: RissyRan
Pull request: no

### System Info

Running vision [model](https://github.com/huggingface/transformers/tree/main/examples/flax/vision) on Cloud TPU with JAX version 4.10.0. 



### Who can help?

@amyeroberts @sanchit-gandhi

### Information

- [X] The official example scripts
- [ ] My own modified scripts

### Tasks

- [X] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

```
        pip install --upgrade pip
        pip install jax[tpu]==0.4.10 -f https://storage.googleapis.com/jax-releases/libtpu_releases.html

        git clone https://github.com/huggingface/transformers.git
        cd transformers && pip install .
        pip install -r examples/flax/_tests_requirements.txt
        pip install --upgrade huggingface-hub urllib3 zipp
        pip install tensorflow
      
        pip install -r examples/flax/vision/requirements.txt
```

Meet error as below:

```
ERROR: Could not find a version that satisfies the requirement torch==1.9.0+cpu (from versions: 1.11.0, 1.11.0+cpu, 1.11.0+cu102, 1.11.0+cu113, 1.11.0+cu115, 1.11.0+rocm4.3.1, 1.11.0+rocm4.5.2, 1.12.0, 1.12.0+cpu, 1.12.0+cu102, 1.12.0+cu113, 1.12.0+cu116, 1.12.0+rocm5.0, 1.12.0+rocm5.1.1, 1.12.1, 1.12.1+cpu, 1.12.1+cu102, 1.12.1+cu113, 1.12.1+cu116, 1.12.1+rocm5.0, 1.12.1+rocm5.1.1, 1.13.0, 1.13.0+cpu, 1.13.0+cu116, 1.13.0+cu117, 1.13.0+cu117.with.pypi.cudnn, 1.13.0+rocm5.1.1, 1.13.0+rocm5.2, 1.13.1, 1.13.1+cpu, 1.13.1+cu116, 1.13.1+cu117, 1.13.1+cu117.with.pypi.cudnn, 1.13.1+rocm5.1.1, 1.13.1+rocm5.2, 2.0.0, 2.0.0+cpu, 2.0.0+cpu.cxx11.abi, 2.0.0+cu117, 2.0.0+cu117.with.pypi.cudnn, 2.0.0+cu118, 2.0.0+rocm5.3, 2.0.0+rocm5.4.2, 2.0.1, 2.0.1+cpu, 2.0.1+cpu.cxx11.abi, 2.0.1+cu117, 2.0.1+cu117.with.pypi.cudnn, 2.0.1+cu118, 2.0.1+rocm5.3, 2.0.1+rocm5.4.2)
ERROR: No matching distribution found for torch==1.9.0+cpu
```

### Expected behavior

After upgrading the torch version to `1.11.0+cpu` and torchvision version to `0.12.0+cpu`, it works as expected.

```
pip3 install torch==1.11.0+cpu torchvision==0.12.0+cpu -f https://download.pytorch.org/whl/torch_stable.html
```

Comments:
Good catch @RissyRan! Torch cpu didn't exist until PT 1.10 😅, so fine for me to bump this! We can just amend this two lines in the requirements file: https://github.com/huggingface/transformers/blob/a574de302f8538d73c342ee946c2cbf8c64e7a6f/examples/flax/vision/requirements.txt#L5-L8

Would you like to open a PR to fix this @RissyRan?

I made a pull request for this change! thanks!
changing the requirements to a cpu torch version that works
Author: sshahrokhi
Pull request: yes

requirements.txt is changed, because the versions would have produced errors. more info on the issue below.

Fixes https://github.com/huggingface/transformers/issues/23418


## Before submitting
- [ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [X] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#start-contributing-pull-requests),
      Pull Request section?
- [X] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [ ] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/main/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/main/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests?


## Who can review?

@sanchit-gandhi



Comments:
_The documentation is not available anymore as the PR was closed or merged._
Fix SAM tests and use smaller checkpoints
Author: Rocketknight1
Pull request: yes

This PR moves all the SAM tests for both PT and TF to the `sam-vit-base` instead of the `sam-vit-huge` (!) checkpoint they were using before. The huge checkpoint made the tests quite slow and caused OOM issues in TensorFlow.

It also fixes an issue with the `check_pt_tf_equivalence` test in the PyTorch tests, which should now pass correctly.

Comments:
_The documentation is not available anymore as the PR was closed or merged._

@ydshieh Ah! That test was also sneakily loading `sam-vit-huge`. I've fixed it, it should work fine now.

All pass now 🚀
Update all no_trainer with skip_first_batches
Author: muellerzr
Pull request: yes
Labels: name: Examples

# What does this PR do?

This PR updates all `no_trainer` examples to use `skip_first_batches` properly from the `Accelerator`/Accelerate when resuming from a checkpoint

Fixes # (issue)


## Before submitting
- [ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [x] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#start-contributing-pull-requests),
      Pull Request section?
- [ ] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [ ] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/main/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/main/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests?


## Who can review?

Anyone in the community is free to review the PR once the tests have passed. Feel free to tag
members/contributors who may be interested in your PR.

@sgugger

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_23664). All of your documentation changes will be reflected on that endpoint.
Update workflow files
Author: ydshieh
Pull request: yes

# What does this PR do?

Same as #23465 but for daily CI and push CI.

Comments:
_The documentation is not available anymore as the PR was closed or merged._

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_23658). All of your documentation changes will be reflected on that endpoint.
[image-to-text pipeline] Add conditional text support + GIT
Author: NielsRogge
Pull request: yes

# What does this PR do?

The `ImageToText` pipeline can generate text given an image, but oftentimes one wants to make the model continue text given a prompt (like "a photo of"). This PR adds support for conditional text generation given an image.

It also adds support for GIT.

This PR fixes a part of #21110 and is based on #22423.

To do:

- [x] add support for Pix2Struct once design is approved

cc @younesbelkada 

Comments:
_The documentation is not available anymore as the PR was closed or merged._

@Narsil thanks for your review, feel free to approve.

Apologies, will take this into account.
The test `LlamaIntegrationTest::test_conversion` test is failing
Author: ydshieh
Pull request: no

The following command
```bash
RUN_SLOW=1 python3 -m pytest -v tests/models/llama/test_tokenization_llama.py::LlamaIntegrationTest::test_conversion
```
gives 
```bash
>           self.assertEqual(old_serialized, new_serialized)
E           AssertionError: '{\n [1465 chars]   "Sequence": {\n          "id": "B",\n      [1794589 chars]}\n}' != '{\n [1465 chars]   "SpecialToken": {\n          "id": "
",\n[1794837 chars]}\n}'

tests/models/llama/test_tokenization_llama.py:337: AssertionError
```


### Who can help?

@ArthurZucker 

Comments:
~I looked into it.~

~The difference is that the newly converted tokenizer has ids 32000-32004 as special ids which correspond if I'm not mistaken to OpenAssistant llama fork.~

~Those do not seem to be declared here: https://huggingface.co/hf-internal-testing/llama-tokenizer/tree/main~

~I'm not sure which part of the code adds them to the slow tokenizer, but this seems indeed like a bug.~

Looked at the wrong file. Everything works it's only a different `type_id` in the post processor.

We simply need to update the tokenizer.json on the hub with the correct value (1)

(There's also a slight issue with the EOS token being added into the processor for no reason.


https://huggingface.co/hf-internal-testing/llama-tokenizer/discussions/3

Goes along with

https://github.com/huggingface/transformers/issues/23400

Confirmed it works!
Bugfix: LLaMA layer norm incorrectly changes input type and consumers lots of memory
Author: TimDettmers
Pull request: yes

# What does this PR do?

It is a common setup to run LLaMA in `bfloat16` or `float16` while the `RMSNorm` layers are in `float32`. The current implementation converts an input into `float32` but never converts it back to the original input type. This means in the first layer the input type if `bfloat16` but for all remaining its `float32` this increases the overall memory-footprint of finetuning considerably.

## Before submitting
- [ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [x] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#start-contributing-pull-requests),
      Pull Request section?
- [ ] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [ ] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/main/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/main/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests?


## Who can review?

@younesbelkada @sgugger 

Please also see: https://github.com/huggingface/transformers/pull/23479



Comments:
_The documentation is not available anymore as the PR was closed or merged._

> but for all remaining its `float32` this increases the overall memory-footprint of finetuning considerably.

...how bad was this? like 30% worse?

Please merge ^_^

I am assuming this does not impact 8bit training as I noticed no change in memory.

@official-elinas did an experiment with transformers==4.28.1 vs the code added here, and was unable to reproduce any speed/memory gains

https://wandb.ai/officialelinas/tests?workspace=user-officialelinas

![image](https://github.com/huggingface/transformers/assets/54623771/1cd5af7d-96ba-4217-a0e4-12d5cea4ef5c)
![image](https://github.com/huggingface/transformers/assets/54623771/4aaa93e4-5cc5-45e1-9cb5-7c3e724734a1)

@TimDettmers what do you think we did wrong? are there any experiments you can share regarding this?


I think unless you've manually set RMSNorm layer's parameters to fp32 while the rest of your model is in fp16/bf16 you won't see any change.

I'm curious if as a design we should instead allow to specify `dtype` in the layer arguments? `F.softmax` has a dtype argument that allows one to control such a thing more finely? https://pytorch.org/docs/stable/generated/torch.nn.functional.softmax.html
Add PerSAM [bis]
Author: NielsRogge
Pull request: yes

# What does this PR do?

This PR adds support for [PerSAM](https://arxiv.org/abs/2305.03048). Simplification of #23652. 

2 optional arguments are introduced: 

- attention_similarity
- target_embedding.

Those are used by PerSAM, a method which enables SAM to be quickly adapted to new concepts.

Comments:
_The documentation is not available anymore as the PR was closed or merged._
Fix SAM
Author: ydshieh
Pull request: yes

# What does this PR do?

Some failures are introduced in #23656. I only checked the TF tests but not the PT tests. Sorry.

cc @Rocketknight1 : it's likely a hardware difference. Always nice if we can get the results from GCP VM, but I understand it makes your workflow a bit difficult.

Comments:
Forgot to mention: kudo to @younesbelkada  for spotting this 

_The documentation is not available anymore as the PR was closed or merged._

The tol used in the tests changed in this PR are not changed in #23656, so nothing to be reverted back.
Add Missing tokenization test [electra]
Author: IMvision12
Pull request: yes

# What does this PR do?

Added tokenization test for electra

Comments:
_The documentation is not available anymore as the PR was closed or merged._

Thanks a lot for your PR! @ArthurZucker could you review?

@sgugger Any updates, is there something wrong?

@sgugger Done!!

Thanks for contributing 🔥
Bring back the PR `Refactor doctests + add CI` to `main`
Author: ydshieh
Pull request: yes

Reverts huggingface/transformers#23245

So we can keep the PR #22987 regarding the new doctest way, but without exposing `doctest_utils` to `src/transformers`.

@sgugger Let me know if you prefer to move this `doctest_utils.py` to `tests` folder.

Comments:
> Can you put the new testing utils (from `doctest_utils`) in `testing_utils`, so it all goes in the same place?

In this case, am I allowed to put the import of `pytest` and `_pytest` on the top level of `testing_utils`? I am asking because I see in that file there is

```python
        try:
            import pytest  # We don't need a hard dependency on pytest in the main library
        except ImportError:
            return test_case
```


There is no direct import into `testing_utils.py` so this should be fine to remove the try except (we will have until next release to make sure it doesn't create a new core dep of Transformers :sweat_smile: )

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_23271). All of your documentation changes will be reflected on that endpoint.

Thanks for taking care of this! Think the filtered list could be obtain in a cleaner way with some bash commands but otherwise great 👍🏻 

@ArthurZucker It was finally going to a `tests_fetcher.py`

https://github.com/huggingface/transformers/pull/23277/files

The bash command was just getting too complex ...

Nice! Thanks for following up
🌐 [i18n-KO] Translated `tasks/monocular_depth_estimation.mdx` to Korean
Author: HanNayeoniee
Pull request: yes

<!-- PR의 제목은 "🌐 [i18n-KO] Translated `<your_file>.mdx` to Korean" 으로 부탁드립니다 -->
# What does this PR do?

Translated the `tasks/monocular_depth_estimation.mdx` file of the documentation to Korean.
Thank you in advance for your review. 

Part of https://github.com/huggingface/transformers/issues/20179
<!-- 메인 이슈에 기록이 남아요! 가짜연구소 리포를 사용해 연습하실때는 제거해주시면 감사하겠습니다! :smile: -->

## Before reviewing
- [x] Check for missing / redundant translations (번역 누락/중복 검사)
- [x] Grammar Check (맞춤법 검사)
- [x] Review or Add new terms to glossary (용어 확인 및 추가)
- [x] Check Inline TOC (e.g. `[[lowercased-header]]`)
- [x] Check live-preview for gotchas (live-preview로 정상작동 확인)

## Who can review? (Initial)
<!-- 1. 위 체크가 모두 완료된 뒤에만 가짜연구소 팀원들에게 리뷰 요청하는 아래 주석을 노출해주세요! -->
Team PseudoLab, may you please review this PR? @0525hhgus, @KIHOON71, @sim-so, @gabrielwithappy, @HanNayeoniee, @wonhyeongseo, @jungnerd

## Before submitting
- [x] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [x] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#start-contributing-pull-requests),
      Pull Request section?
- [ ] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [x] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/main/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/main/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests?

## Who can review? (Final)
<!-- 2. 가짜연구소 팀원들과 리뷰가 끝난 후에만 허깅페이스 직원들에게 리뷰 요청하는 아래 주석을 노출해주세요! -->
@sgugger, @ArthurZucker, @eunseojo May you please review this PR? 

Comments:
_The documentation is not available anymore as the PR was closed or merged._
Unable to import transformers.models.bert.modeling_tf_bert on macOS?
Author: talhakabakus
Pull request: no

### System Info

```
transformers=4.28.0
tensorflow-macos=2.9.0
python=3.10
os=macOS Ventura 13.3.1
BERT_model=uncased_L-12_H-768_A-12
```

### Who can help?

@ArthurZucker  @younesbelkada

### Information

- [ ] The official example scripts
- [x] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [x] My own task or dataset (give details below)

### Reproduction

Simply run the following code snippet:

```
from transformers import BertTokenizer, TFBertModel

tokenizer = BertTokenizer.from_pretrained(BERT_PATH)
model = TFBertModel.from_pretrained(BERT_PATH)
text = "Replace me by any text you'd like."
encoded_input = tokenizer(text, return_tensors='tf')
resp = model(encoded_input)
print(resp)
```

### Expected behavior

Retrieve the output generated by the BERT model.

Comments:
@talhakabakus, thanks for raising this issue. 

So that we can best help you, could you provide some additional information: 
* Environment: Copy-paste the output of `transformers-cli env` here
* The error and full trackback encountered

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.
Fix a `BridgeTower` test
Author: ydshieh
Pull request: yes

# What does this PR do?

A fix is required after #23029.

So far on main, an error is given

```bash
tests/models/bridgetower/test_modeling_bridgetower.py::BridgeTowerModelTrainingTest::test_training
(line 656)  AssertionError: unexpectedly None : Gradients should not be None - got None for bridgetower.cross_modal_image_layers.1.attention.self.query.weight
```

Comments:
_The documentation is not available anymore as the PR was closed or merged._

Thanks for the fix!
[`SAM`] Fixes pipeline and adds a dummy pipeline test
Author: younesbelkada
Pull request: yes

# What does this PR do?

1- Fixes the automatic mask generation pipeline, currently on the main branch, the script below

```python
from transformers import pipeline
from PIL import Image
import requests

generator = pipeline("mask-generation", model="facebook/sam-vit-base", device=0)

img_url = "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/transformers/tasks/car.jpg"
raw_image = Image.open(requests.get(img_url, stream=True).raw).convert("RGB")

outputs = generator(raw_image, points_per_batch=64)
```
is broken

2- Adds a dummy pipeline test. I know the pipelines are already tested in tests/pipeline but these tests are quite slow. I thought adding a small dummy pipeline test is easier for future contributors to make sure they will not break the pipeline without having to run the entire pipeline testing suite for SAM


cc @ArthurZucker @ydshieh @sgugger 

Comments:
_The documentation is not available anymore as the PR was closed or merged._
TF version compatibility fixes
Author: Rocketknight1
Pull request: yes

This PR makes several changes to core methods to support the upcoming 2.13 release of TF and generally futureproof against any other upcoming changes that might happen.

The core source of all of these problems is that Keras has been very mobile inside TensorFlow: Initially we used `tf.python.keras` until they told us this was a deprecated copy of Keras that shouldn't be there, and it got removed. We then switched to `tf.keras`, but now Keras has fully moved into its own library and namespace again. Although this is still mirrored at `tf.keras`, for the newest version of TF we'll just `import keras`.

There are several other related problems where our code assumed that parts of Keras that weren't really part of the public API would stay where they were. Not all of these have caused problems yet (that I know of) but they look very risky to me, and so I made some general fixes. This might surface some hidden bugs!

Fixes #23352

Comments:
_The documentation is not available anymore as the PR was closed or merged._

@sgugger I've added a proper framework inference function, put it in `utils/generic` and moved everything over to that. That should catch any edge cases in future, unless one of the frameworks gets renamed entirely :sweat_smile: