noamaanMulla-03/transformers-issues huggingface dataset:
Tokeniser support in java 
Author: rkoystart
Pull request: no

### Feature request

Current I have tested the sentence transformers paraphrase-multilingual-MiniLM-L12-v2 model in python. The model seems to be performing very well. I want to use the model in java so I converted the model to onnx model.  But I could not find a way to convert the tokeniser in java or some equivalent tokeniser library in java . 
So would like to know is there a way to use the tokeniser in java. 

### Motivation

Tokeniser support in java 

### Your contribution

.

Comments:
This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.
pretrain longT5
Author: Arij-Aladel
Pull request: no

Could you please provide us with the steps of pretraining the longT5 on both MLM and PSG objectives?
Denoising rates and other details and steps? 

Comments:
https://arxiv.org/pdf/2112.07916.pdf

All the requested information is on the paper. I would recommend sending questions over to https://github.com/google-research/longt5 since the authors will be far more able to answer specific questions.

@reelmath  thanks alot!  I already know that but the steps are in c++ script and no clear documentation to generate the corpus and using it to calculate loss. Since in Pegasus these are two losses. what is not clear do they calculate just one loss or two losses in longt5 during pretraining?
beside I want to pretrain the model from huggingface.
https://github.com/google-research/longt5/issues/7#issue-1319140379

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.
fix missing block when there is no failure
Author: ydshieh
Pull request: yes

# What does this PR do?

Currently, when doc test has no failure, we don't see anything other than `🤗 Results of the doc tests.` in slack report.
This is because the block defined in `no_failures` property is not included in `def payload`, as the condition `self.no_failures == 0` should be `self.n_failures == 0`.

Comments:
@amyeroberts  request you as a reviewer just in case you want to take a look.

_The documentation is not available anymore as the PR was closed or merged._
Incorrect auto feature extractor for videomae
Author: fcakyon
Pull request: no
Labels: name: bug

### System Info

- `transformers` version: 4.22.0.dev0
- Platform: Windows-10-10.0.19043-SP0
- Python version: 3.9.12
- Huggingface_hub version: 0.9.1
- PyTorch version (GPU?): 1.12.1+cpu (False)
- Tensorflow version (GPU?): 2.9.1 (False)
- Flax version (CPU?/GPU?/TPU?): not installed (NA)
- Jax version: not installed
- JaxLib version: not installed
- Using GPU in script?: <fill in>
- Using distributed or parallel set-up in script?: <fill in>

### Who can help?

@NielsRogge @sgugger

### Information

- [X] The official example scripts
- [ ] My own modified scripts

### Tasks

- [X] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

1. Install dev dependencies:
```bash 
cd transformers
pip install -e ".[dev]"
```

2. Init new model:
```bash
transformers-cli add-new-model-like
>>What is the model you would like to duplicate?
videomae
```

### Expected behavior

It should ask me:
`Will your new model use the same processing class as videomae (VideoMAEFeatureExtractor)?`

Instead, it asks:
`Will your new model use the same processing class as videomae (ViTFeatureExtractor)?`

Comments:
Tried to fix it with a PR: https://github.com/huggingface/transformers/pull/18779
fix a typo in auto feature extraction for videomae
Author: fcakyon
Pull request: yes

# What does this PR do?



<!-- Remove if not applicable -->

Fixes https://github.com/huggingface/transformers/issues/18778


## Before submitting
- [x] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [x] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#start-contributing-pull-requests),
      Pull Request section?
- [x] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [ ] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/main/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/main/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests?


## Who can review?

@NielsRogge @sgugger


Comments:
_The documentation is not available anymore as the PR was closed or merged._
How to convert pytorch bart model to tf1.x ?
Author: wwwlps
Pull request: no

I have trained a pytorch bart model, and how can I convert it to tf1.x ?

Comments:
This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.
Flax Support NLLB (or M2M100) model
Author: acul3
Pull request: no

### Feature request

Add Flax/Jax support for M2M100 so it can be optimize in TPU

### Motivation

NLLB is great model translation that support many languages and also have good accuracy 

it can be used to translate available english large dataset to another language
but this require much resource such as multiple gpu parallelism to cut time for translation

since tpu access more easy to get(through trc program) than multi-gpu access,it would be nice if we have NLLB/M2M100 for flax

to give another reason
flaxmarian in tpu is amazing... it can translate 100k English texts to Spanish in less than 4 minutes using Flax on TPUs with jax parallelism [flax community slack](https://huggingface.slack.com/archives/C025LJDP962/p1626488974341000)

me myself already translated almost 100m sentence using marian flax with 3 days ~~


### Your contribution

although i am not expert yet at flax/jax ,i can make attemp to implement

but i need some pointer how to do that

- AFAIK,NLLB or M2M100 has similar architecture as MBart,since mbart flax has already implemented maybe it can start from that(?)
- need to figure out how to "convert" NLLLB/M2M100 position embedding,etc to jax CMIIW

Comments:
cc @patil-suraj @sanchit-gandhi in case you'd like to guide @acul3 to contribute the Flax version of M2M100

Hey @acul3! Awesome, let's do it 💪 Happy to help you through the implementation! On a high-level, the process for adding this model will look something as follows:

1. Copy across the modelling code from Flax Bart
2. Modify the Flax modelling code to match the PyTorch NLLB/M2M100 implementation
\+ write any necessary tests along the way
3. Check whether the Flax logits match with the PyTorch ones
4. Iterate on step 2 until the check in step 3 passes!

Once we have a Flax model that matches the PyTorch logits, we can be confident our implementation is correct :)

As a starting point, we can copy across the Flax Bart modelling code. You can do this through the 'add-new-model-like' command:
https://github.com/huggingface/transformers/tree/main/templates/adding_a_new_model#add-new-model-like-command from the Flax Bart model:
https://github.com/huggingface/transformers/tree/main/src/transformers/models/bart/modeling_flax_bart.py

Once you've done that, feel free to open a WIP PR. We can go from there!

Hi @sanchit-gandhi 
Thank you for the help

I'll start to work on this today by following the step and open WIP PR first
Will ask some question/pointer after that..thanks again

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.
Add callback that saves only best checkpoints
Author: KseniaSycheva
Pull request: no

### Feature request

A new class for a callback that saves checkpoint only if it performs better than the previous checkpoint on the evaluation dataset. It can work similar to the [EvalCallback](https://stable-baselines3.readthedocs.io/en/master/guide/callbacks.html) that is proposed by stable_baselines3.

### Motivation

This callback will enable evaluating model frequently without using a lot of memory, because only several checkpoints will  be kept. 

### Your contribution

Submitting PR

Comments:
cc @sgugger 

We already have `save_total_limits` to limit the number of checkpoint saved, and with `load_best_model_at_end=True` the best checkpoint is always kept. Which use case that is not currently available would this new callback permit?

I haven't noticed that it is possible to use these options together. The only thing that is different in proposed callback is that all saved checkpoints perform better on the evaluation dataset than the rest of the checkpoints. 

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.
Revert to rescale and safely handle flag in owlvit config
Author: amyeroberts
Pull request: yes

# What does this PR do?

Reverts the `rescale_image` method name to back to `rescale`. This keeps the method inline with other method names such as `normalize`. 

This undoes the renaming in #18677 done to address failing tests caused by `rescale` key being in OWL-ViT config and the introduction of a `rescale` method in #18499. 

The OWL-ViT config has [since been updated](https://huggingface.co/google/owlvit-base-patch32/commit/17740e19dde58d657d21b970ead1cce0ea40f4da). As the model has been released, we rename the key in the config in case `rescale` is there for backwards compatibility. 


## Before submitting
- [ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [x] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#start-contributing-pull-requests),
      Pull Request section?
- [ ] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [ ] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/main/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/main/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests?

Comments:
_The documentation is not available anymore as the PR was closed or merged._
[Update README] Add SegFormer and ViLT links
Author: NielsRogge
Pull request: yes

# What does this PR do?

As we now also have inference widgets for semantic segmentation & visual question answering, it makes sense to add them to the main README.

Comments:
_The documentation is not available anymore as the PR was closed or merged._

Very cool! We've got a finetune guide for semantic segmentation in the works at #18640 right now, with hopefully VQA to come soon.
wav2vec2_with_lm
Author: liroda
Pull request: no

I refer to the example and run kenlm on the Chinese test set, it is ok ,but the language model weight, no matter how I adjust it, the result remains the same, this is my debugging code

 # dict
    vocabfile = os.path.join(model_path,'vocab.json')
    with open(vocabfile,'r',encoding="utf-8") as f:
        vocab_dict = json.load(f)

    labels = list(vocab_dict.keys())
    # kenlm deocder 

    decoder = build_ctcdecoder(
            labels,
            kenlm_model_path=kenlm_path,
            alpha = alpha,
            beta = beta
            )
    # processor_with_lm 

    processor_with_lm = Wav2Vec2ProcessorWithLM(
             feature_extractor=processor.feature_extractor,
             tokenizer=processor.tokenizer,
             decoder=decoder
             )

    # decode with lm 
    text_tmp = processor_with_lm.batch_decode(logits.numpy()).text[0].split('
')
    text_lm = ''.join(text_tmp)
    print ("lm text:{}".format(text_lm))

i  adjust   alpha  beta , how can i do  ？


Comments:
You can run a hyperparameter search by inferencing a test set with different alpha and beta values. But if you have a small language model there might not be a difference. Maybe use a larger language model or larger test set.

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md) are likely to be ignored.

> I refer to the example and run kenlm on the Chinese test set, it is ok ,but the language model weight, no matter how I adjust it, the result remains the same, this is my debugging code
> 
> # dict
> ```
> vocabfile = os.path.join(model_path,'vocab.json')
> with open(vocabfile,'r',encoding="utf-8") as f:
>     vocab_dict = json.load(f)
> 
> labels = list(vocab_dict.keys())
> # kenlm deocder 
> 
> decoder = build_ctcdecoder(
>         labels,
>         kenlm_model_path=kenlm_path,
>         alpha = alpha,
>         beta = beta
>         )
> # processor_with_lm 
> 
> processor_with_lm = Wav2Vec2ProcessorWithLM(
>          feature_extractor=processor.feature_extractor,
>          tokenizer=processor.tokenizer,
>          decoder=decoder
>          )
> 
> # decode with lm 
> text_tmp = processor_with_lm.batch_decode(logits.numpy()).text[0].split('
')
> text_lm = ''.join(text_tmp)
> print ("lm text:{}".format(text_lm))
> ```
> 
> i adjust alpha beta , how can i do ？

Check your kenlm model is char-level or word-level. To wav2vec2_with_lm, it bases `pyctcdecode`, which only support word-level kenlm. If you need to use char-level language model, you need to tune codes of `pyctcdecode`.
Add docstring for BartForCausalLM
Author: ekagra-ranjan
Pull request: yes

# What does this PR do?

Adds docstring for BartForCausalLM


## Before submitting
- [X] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [ ] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#start-contributing-pull-requests),
      Pull Request section?
- [ ] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [ ] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/main/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/main/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests?

@patrickvonplaten, @patil-suraj

Comments:
_The documentation is not available anymore as the PR was closed or merged._
modeling_swin.py img_mask doesn't have expected torch dtype 
Author: LiweiPeng
Pull request: no
Labels: name: bug

### System Info

```shell
transformers: master branch
```


### Who can help?

_No response_

### Information

- [ ] The official example scripts
- [ ] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

How to repro:
  Run Swin Transformer with NVIDIA mixed precision apex amp opt_level=O2.

The problem is at this line: https://github.com/huggingface/transformers/blob/main/src/transformers/models/swin/modeling_swin.py#L600, it creates img_mask as float32 type, which will conflict other types (float16) when mixed precision O2 is used.

One way to fix  is shown below:
```
-    def get_attn_mask(self, height, width):
+    def get_attn_mask(self, height, width, dtype):
         if self.shift_size > 0:
             # calculate attention mask for SW-MSA
-            img_mask = torch.zeros((1, height, width, 1))
+            img_mask = torch.zeros((1, height, width, 1), dtype=dtype)
             height_slices = (
                 slice(0, -self.window_size),
                 slice(-self.window_size, -self.shift_size),
@@ -641,7 +641,7 @@ class SwinLayer(nn.Module):
         # partition windows
         hidden_states_windows = window_partition(shifted_hidden_states, self.window_size)
         hidden_states_windows = hidden_states_windows.view(-1, self.window_size * self.window_size, channels)
-        attn_mask = self.get_attn_mask(height_pad, width_pad)
+        attn_mask = self.get_attn_mask(height_pad, width_pad, dtype=hidden_states.dtype)
```

### Expected behavior

```shell
the img_mask tensor should be created with the same type of hidden_states.
```


Comments:
Hi,

Thanks for your interest in Swin Transformer! Do you mind opening a PR to fix this?

Thanks!

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.

Hi @LiweiPeng do you mind opening a PR for this?

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.

Gently pinging @LiweiPeng here

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.
Run tests if skip condition not met
Author: amyeroberts
Pull request: yes

# What does this PR do?

Some tests that had `unittest.skipIf` decorators weren't calling a test if the condition was false. This mean some test would be showing as passing even though they weren't being run. 


## Before submitting
- [ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [x] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#start-contributing-pull-requests),
      Pull Request section?
- [ ] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [ ] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/main/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/main/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests?


Comments:
_The documentation is not available anymore as the PR was closed or merged._

cc @Rocketknight1 

@gante I removed the `(<=2.8)` from all grouped conv comments in the repo
[ViLT] Remove ViltForQuestionAnswering from check_repo
Author: NielsRogge
Pull request: yes

# What does this PR do?

This PR removes `ViltForQuestionAnswering` from the IGNORE_NON_AUTO_CONFIGURED mapping, because ViLT is now supported by the `AutoModelForVisualQuestionAnswering` class (and corresponding VQA pipeline).

cc @sgugger, `make fixup` currently doesn't check this, would it be possible to automate this?

Comments:
_The documentation is not available anymore as the PR was closed or merged._
KeyError 'overflow_to_sample_mapping' when using LayoutXLM with regular Tokenizer + return_overflowing_tokens
Author: anthony2261
Pull request: no
Labels: name: bug

### System Info

(Running from Colab)
- `transformers` version: 4.21.1
- Platform: Linux-5.4.188+-x86_64-with-Ubuntu-18.04-bionic
- Python version: 3.7.13
- Huggingface_hub version: 0.8.1
- PyTorch version (GPU?): 1.12.1+cu113 (True)
- Tensorflow version (GPU?): 2.8.2 (True)
- Flax version (CPU?/GPU?/TPU?): not installed (NA)
- Jax version: not installed
- JaxLib version: not installed
- Using GPU in script?: Yes, but not relevant here
- Using distributed or parallel set-up in script?: No

### Who can help?

- Introduced when adding [LayoutLMv3](https://github.com/huggingface/transformers/pull/17060) @NielsRogge , Tokenizer related @SaulLu

### Information

- [ ] The official example scripts
- [X] My own modified scripts

### Tasks

- [X] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

View this short notebook on Colab: https://colab.research.google.com/drive/1ETpz8UP42r7HjRg4VUkC7L8ou10qY3bQ?usp=sharing

### Expected behavior

Expected:
Processor extracts features from and tokenizes the input successfully.

Output:
```
---------------------------------------------------------------------------

KeyError                                  Traceback (most recent call last)

[<ipython-input-6-2f428d9c1568>](https://localhost:8080/#) in <module>
----> 1 encoded = processor(image, words, boxes=boxes, word_labels=word_labels, **processor_kwargs)

[/usr/local/lib/python3.7/dist-packages/transformers/models/layoutxlm/processing_layoutxlm.py](https://localhost:8080/#) in __call__(self, images, text, text_pair, boxes, word_labels, add_special_tokens, padding, truncation, max_length, stride, pad_to_multiple_of, return_token_type_ids, return_attention_mask, return_overflowing_tokens, return_special_tokens_mask, return_offsets_mapping, return_length, verbose, return_tensors, **kwargs)
    124         images = features.pop("pixel_values")
    125         if return_overflowing_tokens is True:
--> 126             images = self.get_overflowing_images(images, encoded_inputs["overflow_to_sample_mapping"])
    127         encoded_inputs["image"] = images
    128 

[/usr/local/lib/python3.7/dist-packages/transformers/tokenization_utils_base.py](https://localhost:8080/#) in __getitem__(self, item)
    234         """
    235         if isinstance(item, str):
--> 236             return self.data[item]
    237         elif self._encodings is not None:
    238             return self._encodings[item]

KeyError: 'overflow_to_sample_mapping'
```

Comments:
Please note that this probably also happens with LMv2 if using this combination of non fast processor + return_overflowing_tokens, not sure about LMv3


Hi,

We probably need to update `LayoutXLMProcessor` similar to how it was done for LayoutLMv2 in #17092.

Would you be able to open a PR for this?

Absolutely!

Just stumbled over exactly the same bug. Thanks for looking into that! 😁