noamaanMulla-03/transformers-issues huggingface dataset:
Is there a way to batch examples by max number of tokens using tokenizers and datasets?
Author: goncalomcorreia
Pull request: no

I want to batch my dataset ("super_glue") by the number of tokens instead of the number of sentences. Does this library have any Sampler or Collater that can do this while using a huggingface tokenizer and a huggingface dataset?

Comments:
Yes, see this answer on our forum: https://discuss.huggingface.co/t/are-dynamic-padding-and-smart-batching-in-the-library/10404/4

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md) are likely to be ignored.

Sorry for bumping this up -- I had the same question but I think the page linked here is answering a different question.

Let me ask again differently: is there a [fairseq `--max-tokens` counterpart](https://github.com/facebookresearch/fairseq/issues/165) I can use in huggingface?
Improve tiny model creation script
Author: ydshieh
Pull request: yes

# What does this PR do?

- Add the option to upload the created tiny models to Hub.
- Make tiny config corresponds better to the (reduced) tokenizer.
  - This gets quite complicated
  - But basically, just to make sure the `vocab_size` and `xxx_token_ids` in the tiny config correspond to what we have in the (reduced) tokenizer. 

Once this PR is approved, should I upload them to `hf-internal-testing`? (I remembered you said yes a few months ago, but want to be sure :-))

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_20119). All of your documentation changes will be reflected on that endpoint.
Update SwinForMaskedImageModeling doctest values
Author: amyeroberts
Pull request: yes

# What does this PR do?

Doc test was failing because checkpoints were changed in #20034 which have different config values, resulting in different image sizes after preprocessing. 

* Previous config: https://huggingface.co/microsoft/swin-tiny-patch4-window7-224/blob/main/preprocessor_config.json
* New config: https://huggingface.co/microsoft/swin-base-simmim-window6-192/blob/main/preprocessor_config.json

Updates the test to reflect the new checkpoints. 

## Before submitting
- [ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [x] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#start-contributing-pull-requests),
      Pull Request section?
- [ ] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [ ] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/main/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/main/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests?


Comments:
_The documentation is not available anymore as the PR was closed or merged._
Add CV + audio labels to glossary
Author: stevhliu
Pull request: yes

This is a [follow-up PR ](https://github.com/huggingface/transformers/pull/20051#discussion_r1013978694) to expand the `labels` definition to include expected labels for model heads from other modalities.

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_20114). All of your documentation changes will be reflected on that endpoint.
Update VisionEncoderDecoder to use an image processor
Author: amyeroberts
Pull request: yes

# What does this PR do?

Loading TrOCR processor failed because when checking the type of the feature extractor loaded, it was an image processor, rather than a feature extractor. 

This PR replaces: 
* Replaces the feature extractor with an image processor in `TrOCRProcessor` 
* Adds `AutoImageProcessor` to `AUTO_TO_BASE_CLASS_MAPPING` for the `ProcessorMixin` checks
* Adds backwards compatibility in case `feature_extractor` passed in as a kwarg when creating the processor. 
* Makes equivalent changes in `VisionEncoderDecoder` model

## Before submitting
- [ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [x] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#start-contributing-pull-requests),
      Pull Request section?
- [ ] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [ ] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/main/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/main/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests?

Comments:
_The documentation is not available anymore as the PR was closed or merged._

Thanks @amyeroberts 

Nothing extra (other than what Sylvain mentioned) from my side.

One thing I think would be great if you can provide a link to the line for
> when checking the type of the feature extractor loaded

Like:

See this line: 
https://github.com/huggingface/transformers/blob/c4cad8e3018e26f697f4ab0c5926e0c93aa0315b/src/transformers/processing_utils.py#L84

(For me, this way is easier to know what issue we have, and to see if the fix is good for the issue)

(Probably not necessary for Sylvain, as they knows everything in mind)

@ydshieh That's a good point - thanks for the feedback! I'll make sure to add a link next time.
Make DummyObject more robust
Author: mariosasko
Pull request: yes

# What does this PR do?

Use `__getattribute__`  instead of `__getattr__` in `DummyObject` to track the attribute access. Compared to `__getattr__` (invoked only if the attribute is missing), `__getattribute__` is invoked for every access, hence more robust.

Fixes #20127 


## Before submitting
- [ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [ ] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#start-contributing-pull-requests),
      Pull Request section?
- [ ] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [ ] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/main/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/main/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests?


## Who can review?

Anyone in the community is free to review the PR once the tests have passed. Feel free to tag
members/contributors who may be interested in your PR.

@sgugger 

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_20146). All of your documentation changes will be reflected on that endpoint.
remove adjust_logits_during_generation method
Author: patil-suraj
Pull request: yes

# What does this PR do?

This PR is the first split of #9811.

This PR
1.  introduces two new `generate`  and `config` arguments  and `LogitsProcessor`
    - `forced_bos_token_id` and `forced_eos_token_id`,  to force a specific start and end token. This is particularly useful for many to many and one to many translation models, so we can pass different language tokens as `forced_bos_token_id` to `generate`, 
    - `ForcedBOSTokenLogitsProcessor` and `ForcedEOSTokenLogitsProcessor`
2. Remove `adjust_logits_during_generation` method from all models (except `Marian`) and handle that use case using the newly introduced logits processors.
4. remove the `force_bos_token_to_be_generated` argument from `BartConfig`

For `Marian` we still need to keep the `adjust_logits_during_generation` method to force the model to not generate pad token. Adding the pad token to `bad_words_ids` does not resolve this issue, the score of `pad_token_id` needs to be set to `-inf` before calling `log_softmax`

Comments:
All slow tests passing in PT!

@patrickvonplaten  (Bart, MBart, Pegasus, Marian, Blenderbot, BlenderbotSmall) slow tests are passing in TF as well.

Applied Sylvain's suggestions. Merging!
Add `RoCBertTokenizer` to `TOKENIZER_MAPPING_NAMES`
Author: ydshieh
Pull request: yes

# What does this PR do?

Add `RoCBertTokenizer` to `TOKENIZER_MAPPING_NAMES`.

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_20141). All of your documentation changes will be reflected on that endpoint.

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_20141). All of your documentation changes will be reflected on that endpoint.
Cannot load CLIPProcessor / CLIPFeatureExtractor locally
Author: patrickvonplaten
Pull request: no

### System Info

- `transformers` version: 4.25.0.dev0
- Platform: Linux-5.18.10-76051810-generic-x86_64-with-glibc2.34
- Python version: 3.9.7
- Huggingface_hub version: 0.11.0.dev0
- PyTorch version (GPU?): 1.11.0+cpu (False)
- Tensorflow version (GPU?): 2.9.1 (False)
- Flax version (CPU?/GPU?/TPU?): 0.6.0 (cpu)
- Jax version: 0.3.16
- JaxLib version: 0.3.15
- Using GPU in script?: <fill in>
- Using distributed or parallel set-up in script?: <fill in>


### Who can help?

@sgugger maybe

### Information

- [ ] The official example scripts
- [ ] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

Is it expected that the following doesn't work on main? 

```python
from transformers import CLIPFeatureExtractor, AutoFeatureExtractor, AutoProcessor

feature_extractor = CLIPFeatureExtractor()

id_name = "./clip_feat"
feature_extractor.save_pretrained(id_name)

print("load from CLIPFeatureExtractor")
feature_extractor = CLIPFeatureExtractor.from_pretrained(id_name)
#print("load from CLIPImageProcessor")
#feature_extractor = CLIPImageProcessor.from_pretrained(id_name)
print("load from AutoFeatureExtractor")
feature_extractor = AutoFeatureExtractor.from_pretrained(id_name)
print("load from AutoProcessor")
feature_extractor = AutoProcessor.from_pretrained(id_name)
```

We can see that I can load the feature extractor directly from the class but not from `AutoFeatureExtractor` or `AutoProcessor` even though we save the feature extractor type in the `preprocessor_config.json` file.

### Expected behavior

I would have expected that the code snippet above works.

Comments:
Will be fixed by #20111
Training MarianTokenizer with sentencepiece
Author: Lednik7
Pull request: no

i tried to run:
```
import sentencepiece as spm
import io
from transformers import MarianTokenizer
import json

model_en = io.BytesIO()
spm.SentencePieceTrainer.train(input='texts_en.txt',
                               vocab_size=125036,
                               max_sentence_length=512,
                               model_writer=model_en)

with open('target.spm', 'w') as fp:
    json.dump(model_en.getvalue(), fp)

MarianTokenizer(vocab=125036,
                source_spm="target.spm",
                target_spm="target.spm") # for example
```

But got error:
```
/opt/conda/lib/python3.7/site-packages/transformers/models/marian/tokenization_marian.py in __init__(self, vocab, source_spm, target_spm, source_lang, target_lang, unk_token, eos_token, pad_token, model_max_length, sp_model_kwargs, **kwargs)
    154         )
    155         assert Path(source_spm).exists(), f"cannot find spm source {source_spm}"
--> 156         self.encoder = load_json(vocab)
    157         if self.unk_token not in self.encoder:
    158             raise KeyError("<unk> token must be in vocab")

/opt/conda/lib/python3.7/site-packages/transformers/models/marian/tokenization_marian.py in load_json(path)
    365 
    366 def load_json(path: str) -> Union[Dict, List]:
--> 367     with open(path, "r") as f:
    368         return json.load(f)

OSError: [Errno 9] Bad file descriptor
```

transformers version - 4.16.2
kaggle cpu env

Can someone help me?

Comments:
This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.

@Lednik7 Have you resolved this problem?
I wanted to train a new  Marian tokenizer for new language and having lots of problems with creating the tokenizer.

Can you please share the script which you used to creater tokenizer? or tell how the source and target spm files can be created and alsothe vocab files.
Add PoolFormer Model
Author: tanaymeh
Pull request: no
Labels: name: New model

# 🌟 New model addition
I would like to add the recently announced [PoolFormer](https://github.com/sail-sg/poolformer) model to the Transformers library.

## Model description

PoolFormer model was proposed in the paper, “**MetaFormer is Actually What You Need for Vision**” by Sea AI Lab and the main argument behind this is that performance of transformer/MLP-like models primarily comes from the general architecture instead of the specific token mixers they are using (like Attention for example).

To show this, they have added a basic non-parametric Pooling operator which does basic token mixing. This suggested model outperforms DeiT and ResMLP.

Link to the Model Repo: [PoolFormer](https://github.com/sail-sg/poolformer)
Link to the Paper: [MetaFormer is Actually What You Need for Vision](https://arxiv.org/abs/2111.11418)

<!-- Important information -->

## Open-source status

* [x] the model implementation is available: In PyTorch on PoolFormer [repository](https://github.com/sail-sg/poolformer)
* [x] the model weights are available: In the same repository, [link](https://github.com/sail-sg/poolformer#2-poolformer-models)
* [x] who are the authors: [Sea AI Lab](https://github.com/sail-sg)


Comments:
[MaskFormer] Add doc tests
Author: NielsRogge
Pull request: yes

# What does this PR do?

This PR fixes all code snippets for MaskFormer, and makes sure they are tested. None of them actually ran without issues. It makes a distinction between semantic and panoptic segmentation.

Fixes #20132 

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_20158). All of your documentation changes will be reflected on that endpoint.
IndexError running ESMFold
Author: pstjohn
Pull request: no

### System Info

CentOS 7, transformers-4.25.0.dev0, Python 3.10.6

### Who can help?

@Rocketknight1 

### Information

- [X] The official example scripts
- [ ] My own modified scripts

### Tasks

- [X] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

Running the following example script (added in #20000) is throwing an IndexError for me.

I tried both the `Rocketknight1/esmfold_v1` and `facebook/esmfold_v1` model repositories

```python
from transformers import AutoTokenizer, EsmForProteinFolding

model = EsmForProteinFolding.from_pretrained("facebook/esmfold_v1")
tokenizer = AutoTokenizer.from_pretrained("facebook/esmfold_v1")
inputs = tokenizer(["MLKNVQVQLV"], return_tensors="pt")  # A tiny random peptide
outputs = model(**inputs)
folded_positions = outputs.positions
```

```
Traceback (most recent call last):
  File "test_esmfold.py", line 7, in <module>
    outputs = model(**inputs)
  File "torch/nn/modules/module.py", line 1130, in _call_impl
    return forward_call(*input, **kwargs)
  File "transformers/models/esm/modeling_esmfold.py", line 2121, in forward
    esmaa = self.af2_idx_to_esm_idx(aa, attention_mask)
  File "transformers/models/esm/modeling_esmfold.py", line 2211, in af2_idx_to_esm_idx
    return self.af2_to_esm[aa]
IndexError: index 24 is out of bounds for dimension 0 with size 22
```

### Expected behavior

No IndexError

Comments:
I guess you need to call
```
inputs = tokenizer(["MLKNVQVQLV"], return_tensors="pt", add_special_tokens=False)
```
code example was already updated:
https://github.com/huggingface/transformers/blob/main/src/transformers/models/esm/modeling_esmfold.py#L2103


Thanks @maxjeblick! And yes - the `ESMFold` tokenizer doesn't use special tokens, but the other `ESM` tokenizers do. I'll see if I can set this in the config so that users don't have to keep remembering it, because I kept getting errors from forgetting it too!
doc comment fix: Args was in wrong place
Author: hollance
Pull request: yes

# What does this PR do?

Small fix for `Args:` being in the wrong place in the doc comments.


## Before submitting
- [x] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [ ] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#start-contributing-pull-requests),
      Pull Request section?
- [ ] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [ ] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/main/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/main/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests?


## Who can review?

Anyone in the community is free to review the PR once the tests have passed. Feel free to tag
members/contributors who may be interested in your PR.

<!-- Your PR will be replied to more quickly if you can figure out the right person to tag with @

 If you know how to use git blame, that is the easiest way, otherwise, here is a rough guide of **who to tag**.
 Please tag fewer than 3 people.

Models:

- albert, bert, xlm: @LysandreJik
- blenderbot, bart, marian, pegasus, encoderdecoder,  t5: @patrickvonplaten, @patil-suraj
- longformer, reformer, transfoxl, xlnet: @patrickvonplaten
- fsmt: @stas00
- funnel: @sgugger
- gpt2: @patrickvonplaten, @LysandreJik
- rag: @patrickvonplaten, @lhoestq
- tensorflow: @LysandreJik

Library:

- benchmarks: @patrickvonplaten
- deepspeed: @stas00
- ray/raytune: @richardliaw, @amogkam
- text generation: @patrickvonplaten
- tokenizers: @n1t0, @LysandreJik
- trainer: @sgugger
- pipelines: @LysandreJik

Documentation: @sgugger

HF projects:

- datasets: [different repo](https://github.com/huggingface/datasets)
- rust tokenizers: [different repo](https://github.com/huggingface/tokenizers)

Examples:

- maintained examples (not research project or legacy): @sgugger, @patil-suraj
- research_projects/bert-loses-patience: @JetRunner
- research_projects/distillation: @VictorSanh

 -->


Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_20164). All of your documentation changes will be reflected on that endpoint.
Update `OnnxConfig.generate_dummy_inputs` to check `ImageProcessingMixin`
Author: ydshieh
Pull request: yes

# What does this PR do?

As we have a new class `ImageProcessingMixin`, the method `OnnxConfig.generate_dummy_inputs` needs to check this case, otherwise it can't find a if/else branch to create dummy inputs.

See 
https://github.com/huggingface/transformers/blob/7ec1dc8817a99d16e6f9e0ab94ce4027ef74b72d/src/transformers/onnx/config.py#L371

Current failing error is:

```bash
AssertionError: beit, default -> Unable to generate dummy inputs for the model. Please provide a tokenizer or a preprocessor.
```


Comments:
_The documentation is not available anymore as the PR was closed or merged._

We still have `OwlViTFeatureExtractor` and no `OwlViTImageProcessor`.

@amyeroberts I guess it is a miss, right? If so, could you work on adding `OwlViTImageProcessor` in another PR, thanks.

@ydshieh Yes - the `OwlViTImageProcessor` doesn't exist yet (along with other object detection / segmentation models). These will be added soon. There's two pending PRs:
* Adding transforms: https://github.com/huggingface/transformers/pull/20003
* Adding these models image processors: https://github.com/huggingface/transformers/pull/20160

I believe this should be only affect OwlViT - as it has a processor class which contains both the feature extractor and the image processor. As you've done in the PR - I think maintaining a check for both `FeatureExtactionMixin` and `ImageProcessingMixin` should work. We can then remove the check for 
`elif isinstance(preprocessor, FeatureExtractionMixin) and preprocessor.model_input_names[0] == "pixel_values":`