noamaanMulla-03/transformers-issues huggingface dataset:
❓ In BART, why forcing the first token to BOS ?
Author: astariul
Pull request: no

# ❓ Questions & Help

In the generation method, the method `prepare_scores_for_generation` is called :
https://github.com/huggingface/transformers/blob/0a9d09b42a9c7c1ccc00da48486a1188078e8594/src/transformers/modeling_utils.py#L1208

And in this method, if it's the first decoding step, BOS token is forced :
https://github.com/huggingface/transformers/blob/0a9d09b42a9c7c1ccc00da48486a1188078e8594/src/transformers/modeling_bart.py#L924-L926

---

I don't understand why it's necessary, because anyway the decoder input ids already contain BOS :
https://github.com/huggingface/transformers/blob/0a9d09b42a9c7c1ccc00da48486a1188078e8594/src/transformers/modeling_utils.py#L866-L868

Comments:
Hi @Colanim,

That's indeed a very good question! The only reason why we add these hacks here is because that's the way Fairseq implemented it and you get better results on summarization using Bart this way. We measured the differences in performance when leaving out those "force token" hacks and it was quite significant. 
Please read through these PRs to better understand why we made this decision: 

https://github.com/huggingface/transformers/pull/3225

and 

https://github.com/huggingface/transformers/pull/3140

> I don't understand why it's necessary, because anyway the decoder input ids already contain BOS :

Regarding this point, is the situation different? I went through the PRs and the code, but it seems that the default `decoder_start_token_id` is still the EOS_token.

Ultimately, the question I want to ask is 
**If I want to use BART for fine-tuning on another summarization task, do I set the `decoder_start_token_id` to EOS_token or BOS_token?**
Include Keras tensor in the allowed types
Author: sergiovalmac
Pull request: yes

Includes Keras tensor in the allowed types. This allows propagating symbolic `keras.Input` tensors through the models' `call` method. This way we can convert any HF (subclass) model into a functional model.

@Rocketknight1


Comments:
@sergiovalmac looks like all the CI tests are passing except the code quality, let me run that one for you!

@sergiovalmac tests passing now, merge whenever you're happy with the PR!

Brilliant! Thank you very much! Please could you merge it? It says that only those with write access can merge PRs.

Ah, I'm sorry, I thought you should be able to once I approved it!

Thanks! :-)
Questions when training language models from scratch
Author: ricarsch
Pull request: no

I'm following the guide here (https://github.com/huggingface/blog/blob/master/how-to-train.md, https://huggingface.co/blog/how-to-train) to train a RoBERTa-like model from scratch. (With my own tokenizer and dataset)

However, when I run **run_mlm.py** (https://github.com/huggingface/transformers/blob/master/examples/pytorch/language-modeling/run_mlm.py) to train my model with masking task, the following messages appear:

```
All model checkpoint weights were used when initializing RobertaForMaskedLM.

All the weights of RobertaForMaskedLM were initialized from the model checkpoint at roberta-base.

If your task is similar to the task the model of the checkpoint was trained on, you can already use RobertaForMaskedLM for predictions without further training.
```

And here's my content of **config.json**:

```
{
  "_name_or_path": "roberta-base",
  "architectures": [
    "RobertaForMaskedLM"
  ],
  "attention_probs_dropout_prob": 0.1,
  "bos_token_id": 0,
  "classifier_dropout": null,
  "eos_token_id": 2,
  "hidden_act": "gelu",
  "hidden_dropout_prob": 0.1,
  "hidden_size": 768,
  "initializer_range": 0.02,
  "intermediate_size": 3072,
  "layer_norm_eps": 1e-05,
  "max_position_embeddings": 514,
  "model_type": "roberta",
  "num_attention_heads": 12,
  "num_hidden_layers": 12,
  "pad_token_id": 1,
  "position_embedding_type": "absolute",
  "transformers_version": "4.12.0.dev0",
  "type_vocab_size": 1,
  "use_cache": true,
  "vocab_size": 50265
}

```


I'm wondering does it mean that I'm training from scratch with **"the pretrained weight"** of RoBERTa? And if it's training from the pretrained weights, is there a way to use randomly initiated weights rather than the pretrained ones?

Thanks a lot.



Comments:
For those considering the same question, you could refer to  [this thread](https://stackoverflow.com/questions/69720454/questions-when-training-language-models-from-scratch-with-huggingface/69721327#69721327) for more explanation.
TF weights for xlm-roberta-base?
Author: tgsmith61591
Pull request: no

# 🚀 Feature request

Is there any plan to upload `tf_model.h5` model weights for `xlm-roberta-base` on https://huggingface.co/xlm-roberta-base/tree/main? So far there does not seem to be a way to load a `TFXLMRobertaModel`, as even the examples in the documentation load `roberta-base` (which is EN-only):

https://huggingface.co/xlm-roberta-base/tree/main

## Motivation

We are a tensorflow shop and would like to use the `TFXLMRobertaModel` with pretrained Tensorflow weights, but they do not exist.


Comments:
Hello @tgsmith61591, thank you for opening an issue! For legacy reasons, these models are here: https://huggingface.co/jplu/tf-xlm-roberta-base

We should merge them with the official checkpoints.

Awesome, thank you!

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md) are likely to be ignored.
[megatron_gpt2] dynamic gelu, add tokenizer, save config
Author: stas00
Pull request: yes

This PR improves `megatron_gpt2/convert_megatron_gpt2_checkpoint.py`:
- dynamically figures out the correct activation function based on how megatron was trained
- dynamically figures out the correct tokenizer and sets `config.tokenizer_class`
- adds tokenizer files
- switches to `config.save_pretrained` to ensure the config gets the right bits of the most recent `transformers`

Some of this functionality has been discussed here https://github.com/huggingface/transformers/issues/13906

Comments:
Add Unispeech & Unispeech-SAT
Author: patrickvonplaten
Pull request: yes

# What does this PR do?

This PR adds UniSpeech from Microsoft: https://github.com/microsoft/UniSpeech

### TODOS:

- [x] Run UniSpeech models and verify that HF forward pass yields same output
- [x] Add UniSpeech checkpoints: https://huggingface.co/microsoft/unispeech-large-1500h-cv
- [x] Run UniSpeech-SAT and verify that HF forward pass yields same output (blocked by: https://github.com/microsoft/UniSpeech/issues/4)
- [x] Add UniSpeech-SAT checkpoints
- [x] Add UniSpeech vocab and preprocessing (verify with Microsoft)
- [x] Add UniSpeech vocab and preprocessing (verify wiht Microsoft)
- [x] Verify naming with Microsoft & make README.md's pretty

- [x] Clean PR and add tests
- [x] Verify fine-tuning works

### Future PR:
- [ ] Correct pretraining loss

Comments:
Wait until https://github.com/huggingface/transformers/pull/13877 is merged 

PR is good for review IMO:

- PreTrained Unispeech checkpoints: https://huggingface.co/models?other=unispeech
- PreTrained Unispeech-SAT checkpoints: https://huggingface.co/models?other=unispeech-sat

I think we can merge the pretrained models now. To make them "promotable" we should still do 2 things:

- Unispeech: Add phoneme <-> text tokenizer, need some feedback here from the authors
- Unispeech-SAT: the model should work very well for speaker-verification and speaker-diarization. We should add those two tasks and then promote the model on it as it performs very well
Replace assertions with valueError Exeptions
Author: jyshdewangan
Pull request: yes

# What does this PR do?

<!--
Congratulations! You've made it this far! You're not quite done yet though.

Once merged, your PR is going to appear in the release notes with the title you set, so make sure it's a great title that fully reflects the extent of your awesome contribution.

Then, please replace this with a description of the change and which issue is fixed (if applicable). Please also include relevant motivation and context. List any dependencies (if any) that are required for this change.

Once you're done, someone will review your PR shortly (see the section "Who can review?" below to tag some potential reviewers). They may suggest changes to make the code even better. If no one reviewed your PR after a week has passed, don't hesitate to post a new comment @-mentioning the same persons---sometimes notifications get lost.
-->

<!-- Remove if not applicable -->

Fixes # (issue)


## Before submitting
- [ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [x] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md#start-contributing-pull-requests),
      Pull Request section?
- [x] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [ ] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/master/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/master/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests?


## Who can review?

Anyone in the community is free to review the PR once the tests have passed. Feel free to tag
members/contributors who may be interested in your PR.

@sgugger Please review


Comments:
Contributes toward fixing #12789 

Hey @sgugger, can you please add hactoberfest-accepted tag also?

According to the rules, it's not necessary when the PR has been approved and merged:

> A pull request is considered approved once it has an overall approving review from maintainers, or has been merged by maintainers, or has been given the 'hacktoberfest-accepted' label.*
fix typos in error messages in speech recognition example and modelcard.py
Author: mgoldey
Pull request: yes

# What does this PR do?
This PR fixes two minor typographical issues
1 - a typo in an example speech recognition script that misstates which columns is missing from the input data
2 - a typo in the modelcard.py function that explains when some field is missing from a dataset


## Before submitting
- [X] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [X] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md#start-contributing-pull-requests),
      Pull Request section?

## Who can review?
@patrickvonplaten per git blame, but realistically anyone
@sgugger

Comments:
[Good first issue] LXMERT TensorFlow Integration tests
Author: LysandreJik
Pull request: no
Labels: name: TensorFlow

name: Tests

name: Good First Issue

The TensorFlow implementation of the LXMERT model currently has no integration tests. This is problematic as the behavior can diverge without being noticed.

The [test_modeling_tf_lxmert.py](https://github.com/huggingface/transformers/blob/master/tests/test_modeling_tf_lxmert.py) file should be updated to include integration testing.

An example of a good modeling integration test is visible in the [test_modeling_tf_bert.py#L365-L387](https://github.com/huggingface/transformers/blob/1809de5165804666ba6c6a02a9d177f6683869cc/tests/test_modeling_tf_bert.py#L365-L387) file:

https://github.com/huggingface/transformers/blob/1809de5165804666ba6c6a02a9d177f6683869cc/tests/test_modeling_tf_bert.py#L365-L387

Some additional tips:
- The test must be marked as slow using the `@slow` decorator, so as to be run *daily*, and not on every commit of every branch/pull request of this repository.
- The test must be decorated with the `@require_tf` decorator so as to only be run in environments using PyTorch.
- A single test is necessary. If you feel like implementing multiple of these, then sharing the same checkpoint would be ideal so as to reduce download time.

Comments:
@LysandreJik  is anyone working on it? I would like to work.

Hi! @sadakmed already has a close to finished implementation that we'll merge in the coming days.

Thank you for offering to contribute!

Hi, shouldn't this issue be closed now ? Since a valid integration test was merged ?

Yes, it should :) Thanks!
Example of exporting BartModel + Beam Search.
Author: fatcat-z
Pull request: yes

All new files for the example of exporting BartModel and Beam Search to ONNX file. This example could work with latest version of Huggingface transformers and required no changes to the model code.

Comments:
This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md) are likely to be ignored.

Is this still going ahead in another pull request?

> Is this still going ahead in another pull request?

I should close this draft several days ago since another official one has been merged already.

Oh okay. Does that PR include beam search in the generate method?

> Oh okay. Does that PR include beam search in the generate method?

No, we didn't update generate method.

Ah I see. Thanks for the info!
SEW - Masked Spec errors out in training
Author: patrickvonplaten
Pull request: no

To reproduce:

```python
#!/usr/bin/env python3
from transformers import AutoModelForCTC
import torch

model = AutoModelForCTC.from_pretrained("asapp/sew-small-100k", mask_time_prob=0.2).train()

model(torch.tensor([10000 * [1.0]]))
```

Comments:
Issue is related to the missing projection layer, resolved by https://github.com/huggingface/transformers/pull/14158
IBERT: Testing the speedup
Author: fdlci
Pull request: no

Hi,

I want to test IBERT's speedup, and I have done exactly what is said in https://huggingface.co/kssteven/ibert-roberta-base. For the quantization part, when I set quant_mode to true and run the evaluation again, I get a much slower model. What am I doing wrong?

Thank you for your reply!

Comments:
Hi! You may find the discussion in https://github.com/huggingface/transformers/issues/11312 by @kssteven418 interesting!

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md) are likely to be ignored.

I want to test IBERT's, and I have done exactly what is said in https://huggingface.co/kssteven/ibert-roberta-base. For the quantization part, when I set quant_mode to true and run the evaluation again, I get a much low accuracy model.  What am I doing wrong?
[Pipelines] Fix ASR model types check
Author: anton-l
Pull request: yes

# What does this PR do?

This changes `list` to `dict` in the ASR pipeline's `check_model_type()`. 
Previously, a list was passed to the checker, so the model name extraction was never preformed [here](https://github.com/huggingface/transformers/blob/8ddbfe975264a94f124684a138a2a5ca89a2bd0d/src/transformers/pipelines/base.py#L810) and an error was raised:
```
The model 'SEWDForCTC' is not supported for automatic-speech-recognition.
```

cc @Narsil 

Comments:
Was there any real failure or just a warning ?
If warning it's fine, if failure we need a tests (model type checking should not be a hard error anyway)

@Narsil just a warning in the logs, the model was loading fine
Add DistilHuBERT 
Author: anton-l
Pull request: yes

# What does this PR do?

This adds the DistilHuBERT conversion script and integration tests.

Paper: [DistilHuBERT: Speech Representation Learning by Layer-wise Distillation of Hidden-unit BERT](https://arxiv.org/abs/2110.01900)

The original `s3prl` implementation and weights: https://github.com/s3prl/s3prl/tree/master/s3prl/upstream/distiller

Comments:
Feel free to merge after fixing the last tests
Add SEW CTC models
Author: anton-l
Pull request: yes

# What does this PR do?

This adds the conversion steps and bugfixes to support finetuned SEW and SEW-D checkpoints (https://github.com/asappresearch/sew#asr-model-fine-tuned-on-librispeech-train-clean-100h)

## TODO
- [ ] Add model cards with code examples and WER results
- [x] Update unsupervised models' weights


Comments:
[modeling_utils] respect original dtype in _get_resized_lm_head
Author: stas00
Pull request: yes

Currently `resize_token_embeddings`'s `_get_resized_lm_head` doesn't preserve the original dtype when creating a new param. This PR fixes it.

The fix is identical to:

https://github.com/huggingface/transformers/blob/232822f36d49598e68e152a9ca0a6d90be6f54b5/src/transformers/modeling_utils.py#L792-L794

The problem was detected by @VictorSanh who used  https://huggingface.co/google/t5-v1_1-small w/ deepspeed z3 - this model is different from t5 as it has `"tie_word_embeddings": false`, so it goes through a different code path. 

I will have a deepspeed test covering this path once a new tiny model is added to https://huggingface.co/hf-internal-testing/ to cover this variation of t5 models.

@sgugger, @LysandreJik 

Comments:
Add SegFormer
Author: NielsRogge
Pull request: yes

# What does this PR do?

This PR adds [SegFormer](https://arxiv.org/abs/2105.15203), a new model by NVIDIA that is surprisingly simple, yet very powerful for semantic segmentation of images. It uses a hierarchical Transformer as backbone, and an all-MLP decode head. I've implemented 3 models: 
* `SegformerModel` (backbone-only)
* `SegformerForImageClassification` (backbone + classifier head)
* `SegformerForSemanticSegmentation` (backbone + semantic segmentation all-MLP head)

Models are on the hub (with approval from the author): https://huggingface.co/models?other=segformer

Here's how to use the semantic segmentation model:

```
from transformers import SegformerFeatureExtractor, SegformerForSemanticSegmentation
from PIL import Image

feature_extractor = SegformerFeatureExtractor(do_random_crop=False)
model = SegformerForSemanticSegmentation.from_pretrained("nvidia/segformer-b0-finetuned-ade-512-512")

image = Image.open("...")

# prepare image for model
pixel_values = feature_extractor(image, return_tensors="pt").pixel_values

# forward pass
outputs = model(pixel_values)

# logits are of shape (batch_size, num_labels, height/4, width/4)
logits = outputs.logits
```

Quick inference notebook with visualization: https://colab.research.google.com/drive/1Kc1VLuFrWUPz0rZXA2E_rKQqdK7kV2iH?usp=sharing

## To do/questions

- [ ] Decide on the default values of the feature extractor (which are kind of arbitrary right now)
- [ ] I've called the decode head `SegformerDecodeHead`, rather than SegformerDecoder. It's more of a lightweight head, than a decoder. Is this ok?
- [ ] Add padding of images + segmentation maps (probably a single function in `image_utils.py`), cc @sgugger. Currently, I rely on `torch.nn.functional.pad`, which makes the feature extractor depend on PyTorch. It could also make sense to do it in Numpy (this model for example pads after normalizing, so it would benefit from it as the output after normalization are Numpy arrays).
- [x] Make sure model doesn't return hidden states when the user doesn't want to
- [ ] Model currently returns a `SequenceClassifierOutput`, however this will render wrong shapes of logits in the docs. Logits are actually of shape (batch_size, num_labels, height/4, width/4).
- [ ] Add model cards (author has joined the NVIDIA org on the hub and might create these)


Comments:
PR is ready for review, only thing to be added is padding.
Fix SEW-D implementation differences
Author: anton-l
Pull request: yes

# What does this PR do?

This sets the default activation function for SEW-D to `gelu_python` and the `layer_norm_eps` of the DeBERTa layer to `1e-7` to reproduce the model more closely.

cc @LysandreJik 

Comments:
@LysandreJik - would be nice if we could merge this before the release. Without this PR we are getting significant differences in the output logits between orginial SEW-D and our SEW-D version (up to 5e0).
Added model card for neuraly/bert-base-italian-cased-sentiment
Author: gianpy15
Pull request: yes
Labels: name: model card

Hello, we are making this pull request to add our Italian sentiment model to your repository of transformers.  
Thank you again for hosting the models :)

Comments:
# [Codecov](https://codecov.io/gh/huggingface/transformers/pull/5845?src=pr&el=h1) Report
> Merging [#5845](https://codecov.io/gh/huggingface/transformers/pull/5845?src=pr&el=desc) into [master](https://codecov.io/gh/huggingface/transformers/commit/0b6c255a95368163d2b1d37635e5ce5bdd1b9423&el=desc) will **decrease** coverage by `1.15%`.
> The diff coverage is `n/a`.

[![Impacted file tree graph](https://codecov.io/gh/huggingface/transformers/pull/5845/graphs/tree.svg?width=650&height=150&src=pr&token=9qOlN6Hb1c)](https://codecov.io/gh/huggingface/transformers/pull/5845?src=pr&el=tree)

```diff
@@            Coverage Diff             @@
##           master    #5845      +/-   ##
==========================================
- Coverage   78.50%   77.35%   -1.16%     
==========================================
  Files         146      146              
  Lines       26049    26049              
==========================================
- Hits        20450    20150     -300     
- Misses       5599     5899     +300     
```


| [Impacted Files](https://codecov.io/gh/huggingface/transformers/pull/5845?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [src/transformers/modeling\_tf\_mobilebert.py](https://codecov.io/gh/huggingface/transformers/pull/5845/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy9tb2RlbGluZ190Zl9tb2JpbGViZXJ0LnB5) | `23.38% <0.00%> (-73.39%)` | :arrow_down: |
| [src/transformers/generation\_tf\_utils.py](https://codecov.io/gh/huggingface/transformers/pull/5845/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy9nZW5lcmF0aW9uX3RmX3V0aWxzLnB5) | `83.95% <0.00%> (-2.26%)` | :arrow_down: |
| [src/transformers/generation\_utils.py](https://codecov.io/gh/huggingface/transformers/pull/5845/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy9nZW5lcmF0aW9uX3V0aWxzLnB5) | `97.11% <0.00%> (+0.28%)` | :arrow_up: |
| [src/transformers/modeling\_tf\_distilbert.py](https://codecov.io/gh/huggingface/transformers/pull/5845/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy9tb2RlbGluZ190Zl9kaXN0aWxiZXJ0LnB5) | `98.79% <0.00%> (+33.89%)` | :arrow_up: |

------

[Continue to review full report at Codecov](https://codecov.io/gh/huggingface/transformers/pull/5845?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/huggingface/transformers/pull/5845?src=pr&el=footer). Last update [0b6c255...cf779f6](https://codecov.io/gh/huggingface/transformers/pull/5845?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).


Hello guys, nice job. The tutorial seems broken to me.
https://github.com/huggingface/transformers/issues/14194