noamaanMulla-03/transformers-issues huggingface dataset:
[XLMModel, FlaubertModel] Compatibility with torch make_fx
Author: Giuseppe5
Pull request: no

### System Info

- `transformers` version: 4.29.2
- Platform: Linux
- Python version: 3.8.16
- PyTorch version: 2.0.1


### Who can help?

_No response_

### Information

- [ ] The official example scripts
- [ ] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

I am trying to get a torch.fx representation for some Transformer models and I noticed that some of them are not compatible with `make_fx` from PyTorch.

To reproduce the error:

```python
from torch.fx.experimental.proxy_tensor import make_fx
from transformers import XLMModel

model = XLMModel.from_pretrained('xlm-mlm-en-2048', torchscript=True)
inp = model.dummy_inputs['input_ids']
model.eval()
fx_g = make_fx(model)(inp)

```

which fails with the following:
```
RuntimeError: It appears that you're trying to get value out of a tracing tensor with aten._local_scalar_dense.default - erroring out! It's likely that this is caused by data-dependent control flow or similar.  It may be possible to trace this with dynamic shapes; try setting tracing_mode='symbolic' in your make_fx call.
```

This error is present in XLMModel and FlaubertModel because of the line:
```python
assert lengths.max().item() <= slen
```

It seems that make_fx does not support value-based control flow.
Removing the various occurrences of this line, or executing the code with the -O optimizer flag (that disables all the asserts) eliminate the issue, but neither solution is ideal.

Another element to note is that capturing the graph through dynamo also fails (or outputs multiple graphs) but because of the `.item()` call.



### Expected behavior

The full FX representation of FlaubertModel/XLMModel using torch make_fx.

Comments:
We can have a look at a PR with a fix.

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.
MarkupLM: feature_extraction_markuplm.py only extracts SearchableText
Author: orenshaniathuji
Pull request: no

### System Info

- `transformers` version: 4.29.2
- Platform: Linux-6.1.22-aufs-1-x86_64-with-glibc2.31
- Python version: 3.9.2
- Huggingface_hub version: 0.14.1
- Safetensors version: not installed
- PyTorch version (GPU?): 2.0.1+cu117 (False)
- Tensorflow version (GPU?): not installed (NA)
- Flax version (CPU?/GPU?/TPU?): not installed (NA)
- Jax version: not installed
- JaxLib version: not installed
- Using GPU in script?: No
- Using distributed or parallel set-up in script?: No


### Who can help?

_No response_

### Information

- [ ] The official example scripts
- [ ] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

Run some MArkupLM code that uses the extractor over HTML that uses template tags, etc. (for example, pages of a wiki.js wiki ).  See that you will miss much of the content

### Expected behavior

feature_extraction_markuplm.py should extract all *Text bs4 types, not just SearchableText

Comments:
Please follow the issue template. There is no reproducer and the expected behavior is very unclear.

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.
MarkupLM: TypeError: unsupported operand type(s) for +: 'Tensor' and 'NoneType' in modeling_markuplm.py", line 217
Author: orenshaniathuji
Pull request: no

### System Info

- `transformers` version: 4.29.2
- Platform: Linux-6.1.22-aufs-1-x86_64-with-glibc2.31
- Python version: 3.9.2
- Huggingface_hub version: 0.14.1
- Safetensors version: not installed
- PyTorch version (GPU?): 2.0.1+cu117 (False)
- Tensorflow version (GPU?): not installed (NA)
- Flax version (CPU?/GPU?/TPU?): not installed (NA)
- Jax version: not installed
- JaxLib version: not installed
- Using GPU in script?: <fill in>
- Using distributed or parallel set-up in script?: <fill in>

### Who can help?

_No response_

### Information

- [ ] The official example scripts
- [X] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [X] My own task or dataset (give details below)

### Reproduction

Run the code above or similar, probably for a larger than 512 or 1024 tokens input:
tokenizer = MarkupLMTokenizerFast.from_pretrained("microsoft/markuplm-large-finetuned-websrc")
model = MarkupLMForQuestionAnswering.from_pretrained("microsoft/markuplm-large-finetuned-websrc")
feature_extractor = MarkupLMFeatureExtractor()
feature_processor = MarkupLMProcessor(feature_extractor, tokenizer)

...

encoding = feature_processor(html_content, questions=question, return_tensors="pt")
with torch.no_grad():
        outputs = model(**encoding)

I tried to extend the model's max tokens size as[ suggested here](https://discuss.huggingface.co/t/fine-tuning-bert-with-sequences-longer-than-512-tokens/12652), but that doesn't seem to help

### Expected behavior

outputs = model(**encoding) should create outputs

Comments:
We cannot help without a clear reproducer of the issue.

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.
distillation training for arabic langauge 
Author: muhammed-saeed
Pull request: no

### System Info


I encountered two issues while attempting to run the `binarized_data.py` and `train.py` scripts for the Knowledge Distillation of BERT Language Model on the Arabic Language project. Below are the details of each issue:

1. In the `binarized_data.py` script, I had to modify line 83 to make it work. The original line is:
   ```python
   dp_file = f"{args.dump_file}.{args.tokenizer_name}.pickle"
   ```
   However, I had to remove the `tokenizer_name` variable and change the line to:
   ```python
   dp_file = f"{args.dump_file}.pickle"
   ```
   This change was necessary because the Arabic BERT model name, "asafaya/bert-large-arabic," contains a forward slash ("/"), which caused errors when concatenating it with the `tokenizer_name` variable.

2. In the `train.py` script, I made a modification on line 258. The original line is:
   ```python
   args.max_model_input_size = tokenizer.max_model_input_sizes[args.teacher_name]
   ```
   However, I had to change it to:
   ```python
   args.max_model_input_size = tokenizer.max_model_input_sizes['bert-large-uncased']
   ```
   This modification was necessary because I am using different model configurations than those listed in the folder. It would be helpful if the script could be modified to automatically work with the intended config, allowing for more flexibility.

Apart from these script modifications, I made the necessary changes to the config files to match the different models I am using. this is understood as I am using a model with a different config than the one listed in the folder, maybe we can modify the script to download and locate the necessary config file automatically.

Please let me know if there are any further clarifications needed or if you require additional information to address these issues.

### Who can help?

_No response_

### Information

- [ ] The official example scripts
- [ ] My own modified scripts

### Tasks

- [X] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

here is link to the Google colab that has the problem
https://colab.research.google.com/drive/1OqSvRNMl0-Z7ScCd6hLbPHMO-ZXT3WEw?usp=sharing

### Expected behavior

the model has to start the training smoothly and the script has to be able to handle the model names which contains '/'

Comments:
Please use the [forums](https://discuss.huggingface.co/) for such questions. This is not a maintained example.

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.
[WIP] Gradient Checkpointing: use_reentrant=False
Author: tsheasha
Pull request: yes

# What does this PR do?

As per pytorch's [recommendation](https://github.com/pytorch/pytorch/blob/main/torch/utils/checkpoint.py#L418)

When using gradient checkpointing for models that allow them, torch.util.checkpoint recommends using
`use_reentrant=False``` as per [this note](https://github.com/pytorch/pytorch/blob/main/torch/utils/checkpoint.py#L336):

## Before submitting
- [ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [x] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#start-contributing-pull-requests),
      Pull Request section?
- [ ] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [ ] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/main/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/main/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests?

@ArthurZucker @sgugger 


Comments:
GPU memory not completely free after one BERT/RoBERTa fwd pass
Author: halixness
Pull request: no

### System Info

Hello. I am batch processing a list of sentences to solely extract a contextualized word embedding for each one (for the same word, this is just an example). I have to iterate this for different words and sentences, but after one forward pass and all the "del" and "torch.cuda.empty_cache()", the GPU memory has still "residuals" I can't delete in any way. I even tried to delete the model. I am not sure whether the one below is the correct deleting order. Any suggestion is **highly** appreciated.

### Who can help?

_No response_

### Information

- [ ] The official example scripts
- [x] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [x] My own task or dataset (give details below)

### Reproduction

```
from transformers import AutoTokenizer, AutoModelForMaskedLM
import torch

model = AutoModelForMaskedLM.from_pretrained("roberta-base", output_hidden_states = True,).to("cuda")
tokenizer = AutoTokenizer.from_pretrained("roberta-base")
model.eval()

context_sentences = ["Lorem ipsum dolor sit amet, consectetur adipiscing elit. Donec porttitor erat urna, sit amet vulputate purus rutrum in. Donec et laoreet velit."] * 100
toks = tokenizer(context_sentences, return_tensors="pt").input_ids.to("cuda")

with torch.no_grad():
    out = model(toks)
    
    del context_sentences
    del model
    del out
    del toks
    del tokenizer
    torch.cuda.empty_cache()
```

Result: `NVIDIA GeForce RTX 3070 Laptop GPU memory: 878 MiB/8192 MiB`

### Expected behavior

Expected: `NVIDIA GeForce RTX 3070 Laptop GPU memory: 0 MiB/8192 MiB`

Comments:
Hi @halixness 
In the past I have experienced that, and adding `gc.collect()` before and after the `torch.cuda.empty_cache()` seemed to help. Can you quickly try that out? 🙏 

> `gc.collect()`
@younesbelkada 
Hi! Unfortunately that did not help :(
```
with torch.no_grad():
    out = model(toks)
    
    del context_sentences
    del model
    del out
    del toks
    del tokenizer
    gc.collect()
    torch.cuda.empty_cache()
    gc.collect()
```
Also tried before `torch.no_grad()`.

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.
4.29.0 bug
Author: macheng6
Pull request: no

### System Info

The dp mode of 4.29.0 seems to have a bug. When forwarding, the dtype of the model will be changed to torch.int64, which will cause the torch.finfo function in the get_extended_attention_mask function to report an error

### Who can help?

_No response_

### Information

- [ ] The official example scripts
- [x] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [x] My own task or dataset (give details below)

### Reproduction

no

### Expected behavior

no

Comments:
That's why there is the patch 4.29.2

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.
AttributeError: 'Wav2Vec2Processor' object has no attribute 'set_lang'
Author: LYPinASR
Pull request: no

### System Info

- `transformers` version: 4.28.1
- Platform: Linux-4.15.0-20-generic-x86_64-with-glibc2.10
- Python version: 3.8.0
- Huggingface_hub version: 0.13.4
- Safetensors version: not installed
- PyTorch version (GPU?): 1.12.1 (True)
- Tensorflow version (GPU?): not installed (NA)
- Flax version (CPU?/GPU?/TPU?): not installed (NA)
- Jax version: not installed
- JaxLib version: not installed
- Using GPU in script?: <fill in>
- Using distributed or parallel set-up in script?: <fill in>

### Who can help?

_No response_

### Information

- [X] The official example scripts
- [ ] My own modified scripts

### Tasks

- [X] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

from transformers import Wav2Vec2ForCTC, AutoProcessor

ckpt = "./mms-1b-all/"

processor = AutoProcessor.from_pretrained(ckpt)
model = Wav2Vec2ForCTC.from_pretrained(ckpt)  # requires only 3GB of CPU RAM

target_lang = "esp"

processor.set_lang("esp")
model.load_adapter("esp") # This will load a file called "adapter.esp.bin" from: https://huggingface.co/patrickvonplaten/mms-1b-all , cache it and replace the adapter

model.to("cuda")

audio, sr = sf.read("/home/lenovo/下载/audio.flac")
#/home/lenovo/project/fairseq/content/audio_samples/1.wav

inputs = processor(audio, sampling_rate=sr, return_tensors="pt")

with torch.no_grad():
    logits = model(**inputs).logits

transcription = processor.batch_decode(logits.argmax(-1))[0]

print(f"Transcription: {transcription}")

### Expected behavior

Fix this error, and output the correct language, such as 'kor'

Comments:
The corresponding PR has not been merged yet.

[PR](https://github.com/huggingface/transformers/pull/23813) merged. 

Also feel free to check out:
- https://huggingface.co/docs/transformers/main/en/model_doc/mms
- https://github.com/huggingface/transformers/pull/23813
- https://huggingface.co/facebook/mms-1b-all

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.
Changed "perplexity" to "eval_perplexity"
Author: david-waterworth
Pull request: yes

# What does this PR do?

Modifies training scripts in examples/pytorch/language-modelling so that `perplexity` is correctly logged to wandb. Since the metrics don't contain an eval_ prefix in the metrics dictionary they are not logged. 

<!-- Remove if not applicable -->

Fixes # https://github.com/huggingface/transformers/issues/23593


## Before submitting
- [ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [x] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#start-contributing-pull-requests),
      Pull Request section?
- [x] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [x] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/main/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/main/docs#writing-source-documentation).
- [x] Did you write any new necessary tests?


## Who can review?

Anyone in the community is free to review the PR once the tests have passed. Feel free to tag
members/contributors who may be interested in your PR.

<!-- Your PR will be replied to more quickly if you can figure out the right person to tag with @

 If you know how to use git blame, that is the easiest way, otherwise, here is a rough guide of **who to tag**.
 Please tag fewer than 3 people.

Models:

- text models: @ArthurZucker and @younesbelkada
- vision models: @amyeroberts
- speech models: @sanchit-gandhi
- graph models: @clefourrier

Library:

- flax: @sanchit-gandhi
- generate: @gante
- pipelines: @Narsil
- tensorflow: @gante and @Rocketknight1
- tokenizers: @ArthurZucker
- trainer: @sgugger

Integrations:

- deepspeed: HF Trainer: @stas00, Accelerate: @pacman100
- ray/raytune: @richardliaw, @amogkam

Documentation: @sgugger, @stevhliu and @MKhalusova

HF projects:

- accelerate: [different repo](https://github.com/huggingface/accelerate)
- datasets: [different repo](https://github.com/huggingface/datasets)
- diffusers: [different repo](https://github.com/huggingface/diffusers)
- rust tokenizers: [different repo](https://github.com/huggingface/tokenizers)

Maintained examples (not research project or legacy):

- Flax: @sanchit-gandhi
- PyTorch: @sgugger
- TensorFlow: @Rocketknight1

 -->


Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_23875). All of your documentation changes will be reflected on that endpoint.

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.
Why is CLIPImageProcessor not in general init?
Author: patrickvonplaten
Pull request: no

### System Info

- `transformers` version: 4.25.0.dev0
- Platform: Linux-5.18.10-76051810-generic-x86_64-with-glibc2.34
- Python version: 3.9.7
- Huggingface_hub version: 0.11.0.dev0
- PyTorch version (GPU?): 1.11.0+cpu (False)
- Tensorflow version (GPU?): 2.9.1 (False)
- Flax version (CPU?/GPU?/TPU?): 0.6.0 (cpu)
- Jax version: 0.3.16
- JaxLib version: 0.3.15
- Using GPU in script?: <fill in>
- Using distributed or parallel set-up in script?: <fill in>

### Who can help?

@sgugger maybe

### Information

- [ ] The official example scripts
- [ ] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

In diffusers we load transformers classes according to a model index, which *e.g.* looks as follows:

```bash
{
  "_class_name": "StableDiffusionPipeline",
  "_diffusers_version": "0.7.0.dev0",
  "feature_extractor": [
    "transformers",
    "CLIPFeatureExtractor"
  ],
  "scheduler": [
    "diffusers",
    "PNDMScheduler"
  ],
  "text_encoder": [
    "transformers",
    "CLIPTextModel"
  ],
  "tokenizer": [
    "transformers",
    "CLIPTokenizer"
  ],
  "unet": [
    "diffusers",
    "UNet2DConditionModel"
  ],
  "vae": [
    "diffusers",
    "AutoencoderKL"
  ]
}
```

The important part is:

```
  "feature_extractor": [
    "transformers",
    "CLIPFeatureExtractor"
  ],
```

Now what is happening then is that we load a component that we call `"feature_extractor"` from `"transformers"` and the `"CLIPFeatureExtractor"` class. Then when saving the model we save it with `type(feature_extractor)` which is now though `CLIPImageProcessor` and then we want to load it again from `transformers`, but we cannot import it from transformers.

E.g. `from transformers import CLIPImageProcessor` doesn't work.

Could we add `CLIPImageProcessor` to the init? 

### Expected behavior

I think we should put `CLIPImageProcessor` in the init no? 

Comments:
Will be fixed by #20111

I have the same problem... God...When will the CLIPImageProcessor be set up in general init?

@ZoeyyHz, which version of transformers are you using? I'm able to run the following without issue:

```
from transformers import CLIPImageProcessor
```

Hi, there. I have the same problem too.
which version of transformers do I have to use?

@BigTail375 The CLIPImageProcessor has been available to import from the public init from v4.25.1
Adding support for 3D deep learning models.
Author: VikasSoni1
Pull request: no
Labels: name: Feature request

### Feature request

Hi.
I am planning to add a new pipeline and a model for 3d deep learning tasks which can work on point clouds for classification and detection as there is no support for 3d data right now.
I just wanted to confirm if the process will be similar to the guides for adding a new pipeline and model to hugging face transformers or there are more complexities which I have not thought about?
And is it going to be too much work to add GPU support and batching?


### Motivation

I have been working with 3d deep learning and wanted to implement the whole process from scratch. So, why not contribute to hugging face so other people can use and build upon it?


### Your contribution

Submitting a PR

Comments:
Hi @VikasSoni1, thanks for adding this feature request! 

Pipelines are typically built around groups of models that tackle the same task and share inputs/outputs using the `AutoXxx` API. We don't currently have models which operate on point clouds or 3D data (additions always welcome!) and so there isn't a defined API yet and these should be defined before adding such a pipeline. 

The great thing about open source, is anyone can fork this repo and built upon it. If you've developed your own pipelines - please feel free to share here for the community to find 🤗
https://huggingface.co/sentence-transformers/clip-ViT-B-32 license?
Author: Li0ness
Pull request: no

Hi,

model card for clip-ViT-B-32 misses information about license.
Could you please add license info?


Comments:
You should open a discussion on the model page directly.

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.