noamaanMulla-03/transformers-issues huggingface dataset:
Add TF vision model code samples
Author: ydshieh
Pull request: yes

# What does this PR do?

Add:

- TF_VISION_BASE_MODEL_SAMPLE
- TF_VISION_SEQ_CLASS_SAMPLE

I tested them manually (not through doctest).

(I will add code samples for TF speech models in another PR)

Comments:
_The documentation is not available anymore as the PR was closed or merged._
Add type hints for UniSpeech
Author: Tegzes
Pull request: yes

Adding type hints for forward methods in user-facing class for UniSpeech as mentioned in #16059

@Rocketknight1 

Comments:
_The documentation is not available anymore as the PR was closed or merged._

Hey, this looks good but I'm going to leave a final review until the `Wav2vec` issue is resolved - for the record, I think it's totally fine for you to update that file too!

Yeah, I think I will work on it and make a commit right away

There are some other models that are dependent on Wave2Vec2, I will modify them asap

@Rocketknight1  I think it's good to go now
TF: properly handle kwargs in encoder_decoder architectures
Author: gante
Pull request: yes

# What does this PR do?

Fixes #16400 

Our `input_processing` function is very strict and raises an exception if unexpected kwargs are passed. In the issue linked above, it was found that an exception was being raised and, upon further inspection, we can see that some arguments were being passed in the wrong place.

There were no tests for it (i.e. no `encoder_decoder` calls with `kwargs`), so it slipped under the cracks. This PR also adds tests for it. The updated tests were all failing before the corresponding fix.

Comments:
_The documentation is not available anymore as the PR was closed or merged._
Avoid accessing .dataset of a DataLoader in Trainer
Author: sanderland
Pull request: yes

# What does this PR do?

* Respects get_train_dataloader and such, rather than going back and looking at .train_dataset or requiring attributes in the dataloader to be accessible directly.
   * This allows for overriding it by any object which implements the methods required by a DataLoader (`__len__` and `__iter__`) without additional requirements.
   * The original motivation was to train on a multi-task dataloader which defers to multiple dataloaders. 


## Before submitting
- [x] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#start-contributing-pull-requests),
      Pull Request section?
- [x] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case. 
     - Discussed in #16388
- [x] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/main/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/main/docs#writing-source-documentation).
- [x] Did you write any new necessary tests?



Comments:
_The documentation is not available anymore as the PR was closed or merged._

@[sgugger](https://github.com/sgugger) this should be ready for review.
You were right that there were a couple of more places to change, and the logic is quite inconsistent in places. I've tried to be on the defensive side in covering cases:

* dataloader.dataset can exist or not, and have a length or not
* dataloader always has a len, but it can raise an exception in fairly common cases

This implementation works for my particular case, giving the same output in training+evaluation as before, but without the really painful workarounds.

I had a look at tests and they look complicated, so I will add some after getting confirmation that this is ok otherwise.



Thanks for implementing all the tweaks!
Raise diff tolerance value for TFViTMAEModelTest
Author: ydshieh
Pull request: yes

# What does this PR do?

For now, change the tol from `1e-5` to `2e-5` in `TFViTMAEModelTest.test_pt_tf_model_equivalence` to avoid being flaky.

Comments:
_The documentation is not available anymore as the PR was closed or merged._
Add length to PreTrainedTokenizer train_new_from_iterator
Author: dctelus
Pull request: yes

# What does this PR do?

Adds better progress to PreTrainedTokenizerFast training from iterator

## Before submitting
- [ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [x] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#start-contributing-pull-requests),
      Pull Request section?
- [ ] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case. *No*
- [x] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/main/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/main/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests?


## Who can review?

Anyone in the community is free to review the PR once the tests have passed. Feel free to tag
members/contributors who may be interested in your PR.

- tokenizers: @n1t0, @LysandreJik
- trainer: @sgugger

Comments:
_The documentation is not available anymore as the PR was closed or merged._
TF: unpack inputs on Convbert, GPTJ, LED, and templates
Author: gante
Pull request: yes

# What does this PR do?

This PR implements the `@unpack_inputs` decorator on Convbert, GPTJ, LED, and templates, as per https://github.com/huggingface/transformers/issues/16051.

The diff is big, but the changes are all trivial:
1. Replace `input_processing()` calls by the `@unpack_inputs` decorator
2. Replace `inputs["foo"]` by `foo` (thank god regex exists 🙏 )
3. (SLOW tests were run locally, all passing)

Only one PR remains to close the issue linked above -- implement the change in the encoder_decoder architectures. I've decided to create a separate PR for that one as the changes are not as trivial, and will require a review from a core maintainer.

Comments:
_The documentation is not available anymore as the PR was closed or merged._
Feature Extractor accepts `segmentation_maps`
Author: FrancescoSaverioZuppichini
Pull request: yes

# What does this PR do?
This PR modifies `MaskFormerFeatureExtractor` in order to accepts a `segmentation_maps`. Under the hood in convert the map to binary masks using  `.convert_segmentation_map_to_binary_masks`. It follows an usage example:

```python
image = Image.open("./transformers/tests/fixtures/tests_samples/ADE20K/ADE_val_00000001.jpg")
segmentation_map = Image.open("./transformers/tests/fixtures/tests_samples/ADE20K/ADE_val_00000001.png")

feature_extractor = MaskFormerFeatureExtractor(num_labels=150)

inputs = feature_extractor(
    images=[image],
    segmentation_maps=[segmentation_map],
    return_tensors="pt",
)

mask_fomer = MaskFormerForInstanceSegmentation.from_pretrained("facebook/maskformer-swin-small-ade").eval()

mask_fomer(**inputs)
```

This simplifies and aligns the APIs to other models (e.g. `SegFormer`).

## Issues

We need to store somewhere the `num_labels`. Currently, I've added it as a `__init__` params. However, we may want to avoid storing dataset-related information in the feature extractor. Open for discussion :) 

## TODO

- [ ] re-upload the feature extractors config to the hub

Comments:
_The documentation is not available anymore as the PR was closed or merged._

Thanks for the comments. All the conversations have been resolved, the feature extractors config have to be re-uploaded to the hub. 

I think passing the `num_labels` in the `__init__` is not the best solution (but it's a solution) since:
- the user must know and remember to override it later on
- the feature extractor is coupled with the dataset used to pretrained

Let me know if you have a better solution in mind :) 

We need the number of labels stored somewhere to be able to run the inference widgets on those models, so having it in the feature extractor is the best solution I can think of.

Added `ignore index` and `reduce labels` following `SegFormer` APIs

Updated the code base, now the feature extractor will return two list when `segmentation_maps` is passed:

-  **mask_labels** -- list of tensor of shape `(labels, height, width)` 
-  **class_labels** -- list of tensor of shape `(labels, num_labels)`

They identify the labels of `mask_labels`, e.g. the label of `mask_labels[i][j]` if `class_labels[i][j]`. Due to different sizes in the first dimensions, they cannot be stacked together. 

We could pad them but this will add more development time

Thanks
- `ignore_index` is needed for some cases in which we want to deal with, well, indeces we want to ignore :)


Update the doc in the feature extractor with an example

Update the documentation by changing `labels -> num_labels` and providing a clear example on how `segmentation_maps` are preprocessed for maskformer

Update docstring in modeling masksformer to reflect the changes in the inputs
segformer variants
Author: JunMa11
Pull request: no

Dear @NielsRogge,

Thanks for re-implementing the `segformer`. I have two questions:

1. The default setting is `segformer-b0`. If I want to use `segformer-b3`, I only need to modify the two parameters: `depths` and `hidden_sizes`, right?

2. I want to use the pre-trained model for small images (256x256), how should I use the `SegformerModel`?

Any comments would be highly appreciated:)

Best regards,
Jun






Comments:
Hi,

Thanks for your interest in SegFormer! If you want to use the pre-trained model, you can load any checkpoint from the [hub](https://huggingface.co/models?other=segformer), like so:

```
from transformers import SegformerForSemanticSegmentation

model = SegformerForSemanticSegmentation.from_pretrained("nvidia/mit-b3", num_labels=10)
```

This will load only the pre-trained encoder (Mix Transformer or mit), b3-sized, with a randomly initialized head. 

For more details on how to fine-tune SegFormer on your custom data, check out the following resources:
* notebook: https://github.com/NielsRogge/Transformers-Tutorials/blob/master/SegFormer/Fine_tune_SegFormer_on_custom_dataset.ipynb
* blog post (with corresponding notebook): https://huggingface.co/blog/fine-tune-segformer. This one leverages the [Trainer API](https://huggingface.co/docs/transformers/main_classes/trainer).

Regarding the differences between the sizes, [here](https://github.com/huggingface/transformers/blob/277d49a590b6745ec82460eea3f33a825a89051c/src/transformers/models/segformer/convert_segformer_original_to_pytorch.py#L158-L178) you can see the differences.

Hi @NielsRogge ,

Thanks for your guidance very much:)
update smddp api to v1.4.0
Author: roywei
Pull request: yes

1. Update SMDDP APIs to v1.4.0
2. Use vanilla PyTorch dist module and DDP class
3. replace `dist.get_local_rank()`  with env var query as PyTorch does not have this API. 

fixes https://github.com/huggingface/transformers/issues/16313

Comments:
_The documentation is not available anymore as the PR was closed or merged._

Hi @sgugger, I removed bunch of unused imports since it's guaranteed that process group will be initialized by `training_args_sm.py`.

Could you give some pointers on how to fix CI errors?

`F401 'smdistributed.dataparallel.torch.torch_smddp' imported but unused`
this import is needed to register SMDDP as a pytorch backend, can we disable style check on this line?

`FAILED tests/funnel/test_modeling_funnel.py::FunnelModelTest::test_pt_tf_model_equivalence`

To silence the flake8 error on those lines, you need to add a comment at the end of the import line: `# noqa: F401`

Thanks again for your work on this!
Support reduce_bucket_size="auto" for deepspeed stages <3
Author: manuelciosici
Pull request: yes
Labels: name: DeepSpeed

# What does this PR do?

<!--
Congratulations! You've made it this far! You're not quite done yet though.

Once merged, your PR is going to appear in the release notes with the title you set, so make sure it's a great title that fully reflects the extent of your awesome contribution.

Then, please replace this with a description of the change and which issue is fixed (if applicable). Please also include relevant motivation and context. List any dependencies (if any) that are required for this change.

Once you're done, someone will review your PR shortly (see the section "Who can review?" below to tag some potential reviewers). They may suggest changes to make the code even better. If no one reviewed your PR after a week has passed, don't hesitate to post a new comment @-mentioning the same persons---sometimes notifications get lost.
-->

<!-- Remove if not applicable -->

Fixes #16495


## Before submitting
- [ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [x] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#start-contributing-pull-requests),
      Pull Request section?
- [ ] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [ ] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/main/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/main/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests?


## Who can review?

@stas00



Comments:
_The documentation is not available anymore as the PR was closed or merged._
Modeling Outputs
Author: FrancescoSaverioZuppichini
Pull request: yes

# What does this PR do?

This PR tries to add new model outputs to cover avoid implementing custom outputs for models that do not have attentions. Three new outputs are added:
- `BaseModelOutputWithNoAttention`
- `BaseModelOutputWithPoolingAndNoAttention`
- `ImageClassifierOutputWithNoAttention`

The docstring of the `hidden_states` and `hidden_states` shape is changed to `(batch_size, num_channels, height, width)`. The thinking behind it is models that do not output attentions are usually conv models. However, this is not quite general.

Another solution is to change the docstring of `BaseModelOutput***` to state that `attentions` will be returned only if the model has an attention mechanism.

A custom `ModelOutput` will still be needed if the returned tensors' shapes are not the ones in the docstring. One solution may be making the outputs "model aware". For example, I can define a `BaseModelOutput2D` that returns 2d tensors (`batch_size, num_channels, height, width`). Doing so will unfortunately result in a lot of new model outputs. 

Let me know your feedbacks 

Comments:
_The documentation is not available anymore as the PR was closed or merged._

Updated the docstring of the outputs and replaced multiple models' outputs with the new ones

@NielsRogge Not sure what the point of your last comments. We know the modeling outputs won't work for every model and we have a strategy for that (subclass and overwrite the docstring). Did you have a suggestion on how to make the docstring more general?

Maybe we can add a statement that says that the `hidden_states` optionally include the initial embeddings, if the model has an embedding layer.

+1 on @NielsRogge idea

Looks good to me, note that we have some more vision models:

* Swin Transformer
* DeiT
* BEiT
* ViTMAE

Added correct outputs for
- DeiT
- BEiT

The following models need a custom output
- ViTMAE
- Swin

Changed the docstring for every `hidden_states` field in every `*ModelOutput*` to specify the embedding hidden state may be optional (depending on the model's architecture)

Update the docstrings and create a custom model output for SegFormer `SegFormerImageClassifierOutput`
Add support for exporting GPT-J to ONNX-TRT
Author: tomerip
Pull request: yes

This change enables exporting GPT-J to ONNX without having any `Sequence*` ops which are currently unsupported in TensorRT ONNX parser. It works by using a sequence of "`view`, `repeat`, `view`" instead of using `repeat_interleave`.

# What does this PR do?
Fixes #15640 

## Who can review?
@patil-suraj 

Comments:
_The documentation is not available anymore as the PR was closed or merged._

Added a function to handle the repeat, commenting on each step.
make tuple annotation more specific to avoid failures during symbolic_trace
Author: chenbohua3
Pull request: yes

This pr fixes the failure appeared during `symbolic_trace due` to the missing annotation of Tuple.

Before this pr, if we symbolic trace the bert model like:
```
from transformers import BertTokenizer, BertModel
from transformers.utils.fx import symbolic_trace

tokenizer = BertTokenizer.from_pretrained('bert-base-uncased')
model = BertModel.from_pretrained("bert-base-uncased")
text = "Replace me by any text you'd like."
encoded_input = tokenizer(text, return_tensors='pt')
output = model(**encoded_input)

symbolic_trace_model = symbolic_trace(model,  input_names=["input_ids", "attention_mask", "token_type_ids"])
```
We would get:
```
  exec(compile(src, key, 'exec'), globals)
  File "<eval_with_key>.0", line 4
    def forward(self, input_ids : typing_Union[torch.Tensor,NoneType], attention_mask : typing_Union[torch.Tensor,NoneType], token_type_ids : typing_Union[torch.Tensor,NoneType]) -> typing_Union[typing_Tuple[],transformers_modeling_outputs_BaseModelOutputWithPoolingAndCrossAttentions]:
                                                                                                                                                                                                                ^
SyntaxError: invalid syntax
```

After this pr, we could successfully symbolic trace the bert model.

Comments:
_The documentation is not available anymore as the PR was closed or merged._

@chenbohua3 We're happy to merge this, please let us know whenever you're ready!

@Rocketknight1 Thank you very much. I'm also happy to merge this pr. Should I do anything else? Since this is my first pr to transformers repo:)

@chenbohua3 No, this looks good! We just like to check that contributors aren't planning to add anything else before we merge. Thank you for a great PR!
Translate installation.mdx to Spanish
Author: lilianabs
Pull request: yes

# What does this PR do?

Adds the translation of [installation.mdx](https://github.com/huggingface/transformers/blob/master/docs/source/installation.mdx) to Spanish.

Fixes # [15947](https://github.com/huggingface/transformers/issues/15947) (issue)

## Before submitting
- [✓] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [✓] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md#start-contributing-pull-requests),
      Pull Request section?


Comments:
_The documentation is not available anymore as the PR was closed or merged._

Thanks @lilianabs! 🤗

@omarespejel all of the changes have been added, thank you! :) 

Thank you @lilianabs! 🤗.  @sgugger LGTM 👍
[Typo][Example] Fixed a typo in `run_qa_no_trainer.py`
Author: bhadreshpsavani
Pull request: yes

# What does this PR do?

## Before submitting
- [x] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [x] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#start-contributing-pull-requests),
      Pull Request section? 
- [x] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case. [discussion PR link](https://github.com/huggingface/transformers/pull/11510#issuecomment-1083227744)
- [x] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/main/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/main/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests?


## Who can review?
@sgugger @stas00 

Comments:
_The documentation is not available anymore as the PR was closed or merged._
Typo in documentation.
Author: MattYoon
Pull request: no

Under parameters for `GenerationMixin.generate()`
there is no new bullet point for `prefix_allowed_tokens_fn`, and the content for it is under [`diversity_penalty`](https://huggingface.co/docs/transformers/v4.13.0/en/main_classes/model#transformers.generation_utils.GenerationMixin.generate.diversity_penalty).

This typo only exists for versions after the document UI update(>= 4.13.0).
@sgugger 

Comments:
Thanks for flagging, the PR mentioned above will fix this issue.
added type hints to xglm pytorch
Author: mowafess
Pull request: yes

# What does this PR do?

I added type annotations for xglm (PT) as described in [https://github.com/huggingface/transformers/issues/16059]

## Before submitting
- [ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [x] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#start-contributing-pull-requests),
      Pull Request section?
- [ ] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [ ] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/main/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/main/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests?


## Who can review?

@Rocketknight1 @patrickvonplaten @sgugger

Anyone in the community is free to review the PR once the tests have passed. Feel free to tag
members/contributors who may be interested in your PR.

Comments:
_The documentation is not available anymore as the PR was closed or merged._
Can I training an XLM model from scratch by transformers?
Author: thomas-li-sjtu
Pull request: no

How can I pretain XLM by transformers? Any example? Please.

Comments:
Hey @thomas-li-sjtu, you definitely can. I recommend that you check out [this HF blog post](https://huggingface.co/blog/how-to-train). Although the article doesn't use XLM, the process should roughly look the same.

Also, a gentle reminder to use the [discussion forum](https://discuss.huggingface.co) for inquiries. GitHub Issues is used for bug reports and feature requests!

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md) are likely to be ignored.