noamaanMulla-03/transformers-issues huggingface dataset:
Use cu118 with cudnn >= 8.6 in docker file
Author: ydshieh
Pull request: yes

# What does this PR do?

We use TF 2.12 after #22759 and #23293. But TF 2.12 requires CUDA 11.8 and CUDNN 8.6 (or up) to work.
Currently, our CI have errors with

```bash
Loaded runtime CuDNN library: 8.5.0 but source was compiled with: 8.6.0.  CuDNN library needs to have matching major version and equal or higher minor version. If using a binary install, upgrade your CuDNN library.  If building from sources, make sure the library loaded at runtime is compatible with the version specified during compile configuration.

`UNIMPLEMENTED: DNN library is not found.`.
```


This PR uses new base image for some docker files. We also have to use `cu118` for the torch installation with this new base image.
Other docker files (those with deepspeed stuff) are not changed in this PR - better to see what happens with this change and apply to other files.

Running some previous failing tests and they pass now. Still need to watch if the whole suite (doctest) pass on Monday.

Comments:
_The documentation is not available anymore as the PR was closed or merged._

We have the so called past CI which runs previous torch and tensorflow versions together the environment we set for them. 

In this particular case, since torch version is not changed but using cu118 file, we don't really have any extra CI for torch 2.0 with cu117 after this PR. So no real promise. But this is already the case for our CI, we always fix a cuda and cudnn environment until we really have to change 🙂
Whisper processor no longer saves mel_filters with `.save_pretrained()` 
Author: xenova
Pull request: no

### System Info

- `transformers` version: 4.29.1
- Platform: Linux-5.15.107+-x86_64-with-glibc2.31
- Python version: 3.10.11
- Huggingface_hub version: 0.14.1
- Safetensors version: 0.3.1
- PyTorch version (GPU?): 2.0.0+cu118 (False)
- Tensorflow version (GPU?): not installed (NA)
- Flax version (CPU?/GPU?/TPU?): 0.6.9 (cpu)
- Jax version: 0.4.8
- JaxLib version: 0.4.7
- Using GPU in script?: no
- Using distributed or parallel set-up in script?: no

### Who can help?

@hollance @sanchit-gandhi

### Information

- [ ] The official example scripts
- [X] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

Running this code:
```python
from transformers import AutoProcessor
processor = AutoProcessor.from_pretrained('openai/whisper-tiny')
processor.save_pretrained('test')
```

will output the following `preprocessor_config.json`:

```json
{
  "chunk_length": 30,
  "feature_extractor_type": "WhisperFeatureExtractor",
  "feature_size": 80,
  "hop_length": 160,
  "n_fft": 400,
  "n_samples": 480000,
  "nb_max_frames": 3000,
  "padding_side": "right",
  "padding_value": 0.0,
  "processor_class": "WhisperProcessor",
  "return_attention_mask": false,
  "sampling_rate": 16000
}
```

which does not include `mel_filters`. This is different to the official models saved on the hub, which do include this: https://huggingface.co/openai/whisper-tiny/blob/main/preprocessor_config.json

This is due to the following recent update: https://github.com/huggingface/transformers/commit/7f9195090160d508c7afb2e444e34f181872dd10

Linked issue: https://github.com/xenova/transformers.js/issues/107

### Expected behavior

Saving the processor should also save the `mel_filters`.

Comments:
That change was made before the update you mentioned, in https://github.com/huggingface/transformers/pull/21267

It is not necessary to save the mel filters in the JSON file since they are completely defined by the other properties from that JSON file. Plus it makes the JSON file huge and unreadable.

As far as I'm aware, the PyTorch implementation of Whisper will load JSON config files with or without the mel filters just fine. If this breaks in Transformers.js, then the issue would seem to be there.


Okay thanks 👍 I agree, it does make it quite unreadable; I just thought I would mention it since it causes a mismatch with some of the official whisper models on the hub.
No TPU found colab
Author: sr5434
Pull request: no

### System Info

Transformers installed from source.

### Who can help?

_No response_

### Information

- [X] The official example scripts
- [ ] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [X] My own task or dataset (give details below)

### Reproduction

Steps to reproduce:
1. Use xla_spawn to run the run_clm.py script. 

Colab link:https://colab.research.google.com/drive/1U1kcI4gXPhEeAZkichOejw0ceNlX4E6z?usp=share_link

### Expected behavior

It should detect and run on the Colab TPU, but no TPU is found. I am using a custom text dataset(created in the colab). When I test if torch XLA detects the TPU, it does. I am able to create tensors on the TPU, but it is not being detected.

Comments:
Flax didn't find the TPU, but the actual error is a connection error when downloading the datasets.

How do I fix it?

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.
UnboundLocalError: local variable 'params_docstring' referenced before assignment
Author: xingyueye
Pull request: no

### System Info

https://github.com/huggingface/transformers/blob/v4.27.4/src/transformers/utils/doc.py#L130
BUG about 'params_docstring' is reported

### Who can help?

_No response_

### Information

- [ ] The official example scripts
- [ ] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

It occurs when "if i < len(lines):" not match

### Expected behavior

Fix bugs or add assertion

Comments:
For Reproduction:
It occurs when we define a new subClass with no definition

No one will be able to help without a clear reproducer.

For example, we define a subclass of BaseModelOutputWithPoolingAndCrossAttentions, but with no args explanations. 
```
class NewBaseModelOutputWithPoolingAndCrossAttentions(BaseModelOutputWithPoolingAndCrossAttentions):
    final_text_self_embedding: Optional[torch.FloatTensor] = None
    final_text_visual_embedding: Optional[torch.FloatTensor] = None
    text_visual_states: Optional[Tuple[torch.FloatTensor]] = None
```
`lines = output_docstring.split("\n")`  would return a null result, then `if i < len(lines):` would not be executed.

Why would you use our internal tools for the documentation if you are not documenting the class?

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.
Incorrect question answering initialization
Author: imbalu007
Pull request: no

### System Info

- `transformers` version: 4.26.1
- Platform: Linux-5.15.0-1031-azure-x86_64-with-glibc2.17
- Python version: 3.8.16
- Huggingface_hub version: 0.13.3
- PyTorch version (GPU?): 1.13.1 (True)
- Tensorflow version (GPU?): not installed (NA)
- Flax version (CPU?/GPU?/TPU?): not installed (NA)
- Jax version: not installed
- JaxLib version: not installed
- Using GPU in script?: No
- Using distributed or parallel set-up in script?: No

### Who can help?

@ArthurZucker
@younesbelkada
@sgugger

### Information

- [X] The official example scripts
- [ ] My own modified scripts

### Tasks

- [X] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

1. Select deberta-mnli model and perform finetuning for question answering on any dataset
2. Errors out with this message
`
  File "XXXXXX/lib/python3.8/site-packages/transformers/models/deberta/modeling_deberta.py", line 1416, in forward
    start_logits, end_logits = logits.split(1, dim=-1)
ValueError: too many values to unpack (expected 2)
`


Models finetuned on mnli have 3 classes by default in their config file (because mnli dataset has 3 classes). When these models are repurposed for question answering task, the classification head is initialized from the *config file* [here](https://github.com/huggingface/transformers/blob/ae54e3c3b18bac0832ad62ea9b896dfd52a09850/src/transformers/models/deberta/modeling_deberta.py#L1362
)
But for question answering task, 2 outputs per token are expected [here](https://github.com/huggingface/transformers/blob/ae54e3c3b18bac0832ad62ea9b896dfd52a09850/src/transformers/models/deberta/modeling_deberta.py#L1416). So there is a mismatch between model head which is initialized with 3 labels.
This is likely causing an issue with deberta-mnli when used for question answering. It might potentially cause a similar issue for models trained on datasets other than mnli and having number of labels != 2


### Expected behavior

For question answering task, should the num_labels be hardcoded to 2?

Comments:
This is a bug indeed. Do you want to make a quick PR with your fix (labels should indeed be hardcoded at 2 for question answering)?

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.
Add Cross attention to GPTNeo
Author: gagan3012
Pull request: no

### Feature request

I have been working on using GPTNeo as a decoder for an encoder-decoder model, and I was hoping to add cross attention to GPTNeo. 

### Motivation

For encoder-decoder model

### Your contribution

I would like to submit a PR for it but anyone is up to taking it 

Comments:
@patil-suraj 

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.
Syntax for from_pretrained proxies (downloading model behind corp proxy)
Author: saurabhtangri
Pull request: no

Facing trouble using from_pretrained

```
import torch
from transformers import DistilBertModel
from transformers import AutoModel, AutoTokenizer, BertTokenizer
torch.set_grad_enabled(False)
bert_distil = DistilBertModel.from_pretrained('distilbert-base-cased', proxies={'proxy-abc.xyz.com:123'} )
tokenizer = AutoTokenizer.from_pretrained('distilbert-base-cased',  proxies={'http': 'proxy-abc.xyz.com:123'})
input_pt = tokenizer.encode_plus(
'This is a sample input to demonstrate performance of distiled models especially inference time', 
return_tensors="pt"
)

%time _ = bert_distil(input_pt['input_ids'])
%time _ = model_pt(input_pt['input_ids'])
```

(hf) username@st-rocket-lake-nv3090:~/dev/hf$ **set | grep http**
**http_proxy=http://proxy-abc.xyz.com:123/**
-----------------------

Error: ValueError: Connection error, and we cannot find the requested files in the cached path. Please try again or make sure your Internet connection is on.

Comments:
This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md) are likely to be ignored.

Same issue here, if I use models like this
`instructor_embeddings = HuggingFaceInstructEmbeddings(model_name="hkunlp/instructor-xl",
                                                     model_kwargs={"device": "cpu"})`

I am behind a company proxy and have set it in the environment variables, which works fine for pip. There is a dirty workaround, namely to set verify=False in the 'file_download.py'. But that can't be it, right? 😏 

Is it possible to react like pip, take the environment variables for the proxy (http_proxy proxy address, https_proxy ...)? It'd be much appreciated.

Edit: I just found out about the offline option, which offers an ok-workaround: [Offline Mode](https://huggingface.co/docs/transformers/v4.28.1/en/installation#offline-mode) 

In case anybody stumbles over this issue: Works for me in my run configuration of the IDE.
Fix issue introduced in PR #23163
Author: ydshieh
Pull request: yes

# What does this PR do?

Fix issue introduced in PR #23163.

The previous `torch_version` is removed (which is not a string but a `version` type), and `get_torch_version()` is introduced and used (which is a string). In some places, it is compared against `self.torch_onnx_minimum_version` which is a string, and we know get on CI `TypeError: '<' not supported between instances of 'str' and 'Version'`.

This PR fixes this problem and avoid the > 1000 test failures.

(cc. @apbard FYI)

Comments:
_The documentation is not available anymore as the PR was closed or merged._

FYI, after #23163,  `is_bs4_available()` and `is_faiss_available()` gives `False` even if they are actually available. This causes some CI errors, in particularly, `MarkupLM`.

I will fix this in a separate PR.
Allow fine-tune wav2vec2 with local path custom dataset
Author: ProgramadorArtificial
Pull request: yes

# What does this PR do?

In the examples for fine-tuning Wav2Vec2 it is possible to use a HuggingFace dataset that is ready to be trained. But when we want to use a local/custom dataset, it's easier to create a .tsv that points to the file path. Which doesn't work in current version - returning error: "AttributeError: 'Value' object has no attribute 'sampling_rate'"

So this PR adds a code to check if the "audio_column_name" has an audio format or a path (string), if it is a string, load the file and convert it to audio format.

Below is an example of a .tsv file that does not work with the current version and works with this PR:

path | sentence
-- | --
data/clips/common_voice_tr_21921195.mp3 | Pirin sözleri hâlâ yankılanıyor.
data/clips/common_voice_tr_21921199.mp3 | Müze Gecesi beş yıldır her yıl düzenleniyor.
data/clips/common_voice_tr_21921206.mp3 | Yunanistan'ın Selanik kenti de ilk ona girdi.


## Who can review?

@sanchit-gandhi




Comments:
@ProgramadorArtificial Thanks for taking the time to open this PR. 

The example scripts are just examples and won't work out-of-the-box for all problems. As you've done here, a few lines of code might be needed to adapt to different use cases. This change is very specific. As such, it's not an addition we'll be merging in.  


Hey @ProgramadorArtificial! In this case, it might be easier to first load your audio data as a HF datasets object (here's an excellent guide on how to do so: https://huggingface.co/docs/datasets/audio_load). You can then save this data locally and/or push it to the Hub:
```python
# set a save path and HF Hub dataset id
LOCAL_DATASET_DIR = ...
HF_DATASET_ID = ...

dataset.save_to_disk(LOCAL_DATASET_DIR)
# and/or push to hub
dataset.push_to_hub(HF_DATASET_ID)
```
You'll then be able to use the training script directly with your dataset (either from the local path or loading from the Hub, in the same way that we do for Common Voice in the example).

Hey @ProgramadorArtificial - just re-iterating that the examples script is assumed to work with a HF dataset that has an audio column already present (i.e. one that has an [`Audio` feature](https://huggingface.co/docs/datasets/v2.12.0/en/package_reference/main_classes#datasets.Audio)). If your custom dataset does not have the audio files loaded up, you'll need to perform a round of pre-processing to get your dataset into the right format for this script (expected to have two columns: "audio" and "text").
Fix `OwlViTForObjectDetection.image_guided_detection` doc example
Author: ydshieh
Pull request: yes

# What does this PR do?

Need to update expected values after #23157

Comments:
_The documentation is not available anymore as the PR was closed or merged._

@amyeroberts I know you actually want to give the approval but forgot doing it 😅 .
As I am a serious man, I would try not to merge without a format approval 🙏
Revert "Only add files with modification outside doc blocks"
Author: ydshieh
Pull request: yes

Reverts huggingface/transformers#23327.

I apologize, but I read Sylvain's messge too quickly and got it completely wrong.
> for now the tests are launched on a file if we modify it, but I would only launch it if docstrings are modified (e.g. check the modifications are correct) to go faster.

That merged PR did the converse instead: it adds a test file if only docstring (instead of only code) are modified.

I will need to create sth like `diff_is_code_only`.


Comments:
> Thanks for fixing!
> 
> Apologies for not catching in the review either. In the nightly CI, have we run all of the doctests - or do we expect there to be any untested pieces of code between the merge of #23327 and this PR?

On daily doctest CI, everything is tested :-) - there is no filtration, it just check all files in `utils/documentation_tests.txt`.

_The documentation is not available anymore as the PR was closed or merged._
[Bugfix] `OPTDecoderLayer` does not return attentions when `gradient_checkpointing` and `training` is enabled.
Author: gmlwns2000
Pull request: yes

# What does this PR do?

Reorder argument of OPTDecoderLayer.forward

Fixes #23366


Comments:
_The documentation is not available anymore as the PR was closed or merged._

cc @younesbelkada @ArthurZucker
[GPT-J] where expected condition to be a boolean tensor, but got a tensor with dtype Half
Author: Praful932
Pull request: no

### System Info

- `transformers` version: 4.28.1
- Platform: Linux-5.10.147+-x86_64-with-glibc2.31
- Python version: 3.10.11
- Huggingface_hub version: 0.14.1
- Safetensors version: not installed
- PyTorch version (GPU?): 2.0.0+cu118 (True)
- Tensorflow version (GPU?): 2.12.0 (True)
- Flax version (CPU?/GPU?/TPU?): 0.6.9 (gpu)
- Jax version: 0.4.8
- JaxLib version: 0.4.7
- Using GPU in script?: yes
- Using distributed or parallel set-up in script?: no

### Who can help?

@younesbelkada

### Information

- [ ] The official example scripts
- [X] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [X] My own task or dataset (give details below)

### Reproduction

Collab example - [Link](https://colab.research.google.com/drive/1D689vHxZk5Bov5piIXOQ62t3M_cim4dr?usp=sharing)

### Expected behavior

I expected some output generated from the model

Comments:
Hi @Praful932 
Thanks for the issue, I have managed to reproduce it and fix it with https://github.com/huggingface/transformers/pull/23147
Can you try to uninstall `transformers` and install it again from source?
```bash
pip install git+https://github.com/huggingface/transformers
```

This is working, Thanks for the quick fix!
Skip failing `AlignModelTest::test_multi_gpu_data_parallel_forward`
Author: ydshieh
Pull request: yes

# What does this PR do?

`tests/models/align/test_modeling_align.py::AlignModelTest::test_multi_gpu_data_parallel_forward` starts to fail after we switch to `torch+cu118`. If I install back with `torch+cu117`, it passes again.

This test uses `torch.nn.DataParallel` which is not recommended (despite not deprecated yet). The error is pure CUDA thing for which I have no knowledge. Combing all the above facts + the usage of this model, let's just skip this particular test for `AlignModelTest`.

(This failing test cause the other 18 tests to fail due to the CUDA is in a bad state)

```bash
E       RuntimeError: Caught RuntimeError in replica 0 on device 0.
E       Original Traceback (most recent call last):
E         File "/usr/local/lib/python3.8/dist-packages/torch/nn/parallel/parallel_apply.py", line 64, in _worker
E           output = module(*input, **kwargs)
E         File "/usr/local/lib/python3.8/dist-packages/torch/nn/modules/module.py", line 1501, in _call_impl
E           return forward_call(*args, **kwargs)
E         File "/transformers/src/transformers/models/align/modeling_align.py", line 1596, in forward
E           vision_outputs = self.vision_model(
E         File "/usr/local/lib/python3.8/dist-packages/torch/nn/modules/module.py", line 1501, in _call_impl
E           return forward_call(*args, **kwargs)
E         File "/transformers/src/transformers/models/align/modeling_align.py", line 1395, in forward
E           embedding_output = self.embeddings(pixel_values)
E         File "/usr/local/lib/python3.8/dist-packages/torch/nn/modules/module.py", line 1501, in _call_impl
E           return forward_call(*args, **kwargs)
E         File "/transformers/src/transformers/models/align/modeling_align.py", line 345, in forward
E           features = self.convolution(features)
E         File "/usr/local/lib/python3.8/dist-packages/torch/nn/modules/module.py", line 1501, in _call_impl
E           return forward_call(*args, **kwargs)
E         File "/usr/local/lib/python3.8/dist-packages/torch/nn/modules/conv.py", line 463, in forward
E           return self._conv_forward(input, self.weight, self.bias)
E         File "/usr/local/lib/python3.8/dist-packages/torch/nn/modules/conv.py", line 459, in _conv_forward
E           return F.conv2d(input, weight, bias, self.stride,
E       RuntimeError: GET was unable to find an engine to execute this computation
```

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_23374). All of your documentation changes will be reflected on that endpoint.
May I ask when will release 4.28.0
Author: lihj1108
Pull request: no

### Feature request

May I ask when will release 4.28.0？

### Motivation

May I ask when will release 4.28.0

### Your contribution

May I ask when will release 4.28.0

Comments:
Next week
Next week
Next week
Next week

I needed to use TF-BLIP offline, so I created whl. It may help for this week :D 

https://www.kaggle.com/datasets/ipythonx/tenp-transformer-4280

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.
🚨🚨🚨 [`Blip`] Refactor the Blip modeling file + test file 🚨🚨🚨
Author: younesbelkada
Pull request: yes

# What does this PR do?

Removes the file `test_modeling_blip_text` as its content is totally duplicated inside `test_modeling_blip`, so that we avoid running these tests twice

This PR also refactors the modeling file of `blip`, to have a single file for the whole architecture. I also realized that there is no necessary to have a `BlipTextPretrainedModel`, so I decided to remove that class for a cleaner implementation. Hence, this PR might introduce breaking changes for blip.

cc @ydshieh 

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_22648). All of your documentation changes will be reflected on that endpoint.

Let's wait our great @sgugger to express his opinion on if we are allowed to change this. 
I think it's fine however.

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.