noamaanMulla-03/transformers-issues huggingface dataset:
Text to Speech Generalized End-To-End Loss for Speaker Verification, Real Time Voice Cloning
Author: BirgerMoell
Pull request: no
Labels: name: New model

name: Feature request

# 🌟 New model addition

## Model description
Generalized End-To-End Loss for Speaker Verification implements Real time voice cloning, a way to generate a Text-To-Speech model adapted to a certain speaker with a short audio sample. The model implements the following paper.
https://arxiv.org/pdf/1806.04558.pdf and the code is available on github.

https://github.com/CorentinJ/Real-Time-Voice-Cloning

<!-- Important information -->

## Open source status

* [ ] the model implementation is available: (give details)
https://colab.research.google.com/drive/1SUq5RLOI0TIMkrBzMHMms01aaVNgkO7c?usp=sharing

The model can be run through Colaboratory. Here is an example of a generated voice.
https://soundcloud.com/birger-mo-ll/generated-voice

* [ ] the model weights are available: (give details)
Here are the model weights that are used.

encoder.load_model(project_name / Path("encoder/saved_models/pretrained.pt"))
synthesizer = Synthesizer(project_name / Path("synthesizer/saved_models/logs-pretrained/taco_pretrained"))
vocoder.load_model(project_name / Path("vocoder/saved_models/pretrained/pretrained.pt"))

* [ ] who are the authors: @CorentinJ
The author is not currently working on the repo, but since it is a fairly popular repo (25.000 stars) it might be reasonable to take the time to explore how to recreate / adapt the model to work with Huggingface transformer.




Comments:
@patrickvonplaten This is a suggestion but there are several models available and I think the best first step would be to look into getting a Text-To-Speech model working.

I explored the Real-Time-Voice-Cloning the other day and noticed it had several issues (since the project is no longer maintained) so it might be good to look into other speech models.

Here are some examples of repos that might be useful.

https://github.com/mozilla/TTS

https://github.com/as-ideas/ForwardTacotron





Hey @BirgerMoell - thanks a lot for the links I will take a look soon :-)

@BirgerMoell 
Thank you for resource sharing. I also want to add [TransformerTTS](https://github.com/as-ideas/TransformerTTS) to the list since it makes more sense to me to have transformers involved :P

I'd love to see this addition to huggingface though

I think it'd make a lot of sense to add FastSpeech2 to the library - happy to help with a PR if someone is interested. See: https://github.com/huggingface/transformers/pull/11135

Also, we started integrating https://github.com/as-ideas/TransformerTTS to the model hub so that people have easier access to TensorflowTTS models :-) 

https://huggingface.co/tensorspeech/tts-fastspeech2-baker-ch

Hello
To avoid duplication, I just wanted to check if anyone is working on this or if this is still relevant. If someone is still needed for this, I will be interested to take this up.
`offset_mapping` is strange for non-ascii token
Author: h-peng17
Pull request: no
Labels: name: bug

### System Info

```shell
- Tensorflow version (GPU?): not installed (NA)
- Flax version (CPU?/GPU?/TPU?): not installed (NA)
- Jax version: not installed
- JaxLib version: not installed
- Using GPU in script?: <fill in>
```


### Who can help?

@SaulLu

### Information

- [ ] The official example scripts
- [X] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [X] My own task or dataset (give details below)

### Reproduction

1. Use BertTokenizerFast
2. tokenize the text 
text = "The Dungan Revolt ( 1862–77 ) or Tongzhi Hui Revolt ( , Xiao'erjing : توْحُوِ بِيًا/لُوًا ,  ) or Hui ( Muslim ) Minorities War was a mainly ethnic and religious war fought in 19th-century western China , mostly during the reign of the Tongzhi Emperor ( r. 1861–75 ) of the Qing dynasty ."
outputs = self.tokenizer(text,
                                    padding="max_length",
                                    truncation=False,
                                    max_length=self.config.max_seq_length,
                                    return_offsets_mapping=True)
3. The `offset_mapping` missed some subtokens, for example, the subtoken at the char position 79. 

<img width="83" alt="截屏2022-04-20 下午8 46 55" src="https://user-images.githubusercontent.com/39556019/164233684-192d1c32-0f67-4198-8e59-e60426fce32d.png">


### Expected behavior

```shell
the offset_mapping should be:
[... 
 (70, 71),
 (71, 72),
 (73, 74),
 (76, 77),
 (78, 80), # this line ****
 (81, 82),
 (83, 84),
... ]
```


Comments:
The environment info is:
- `transformers` version: 4.18.0
- Platform: Linux-4.15.0-136-generic-x86_64-with-glibc2.27
- Python version: 3.9.12
- Huggingface_hub version: 0.5.1
- PyTorch version (GPU?): 1.11.0+cu113 (True)
- Tensorflow version (GPU?): not installed (NA)
- Flax version (CPU?/GPU?/TPU?): not installed (NA)
- Jax version: not installed
- JaxLib version: not installed
- Using GPU in script?: <fill in>
- Using distributed or parallel set-up in script?: <fill in>
Add missing entries in mappings
Author: ydshieh
Pull request: yes

# What does this PR do?

Add missing entries in mappings:  `TOKENIZER_MAPPING`, `FEATURE_EXTRACTOR_MAPPING`, `PROCESSOR_MAPPING`

The remaining models that don't have any `tokenizer/feature_extractor/processor` are:

- EncoderDecoder
- VisionEncoderDecoder
- SpeechEncoderDecoder
- DecisionTransformer

which are expected to have no processor.

Comments:
_The documentation is not available anymore as the PR was closed or merged._

> * The `ImageGPT` changes should probably go in their own PR

Without the changes, the `test_pipeline` will fail for `ImageGPT`. Previously, `ImageGPT` was not used in pipeline testing because     it had no found tokenizer/feature_extractor.


I can add `("imagegpt", "ImageGPTFeatureExtractor")` in another PR instead of this PR, and make the necessary changes along with it. Let me know if this is necessary 😃 

Yes, I think ImageGPT should be added in a separate PR since it requires additional changes.

The changes on ImageGPT are now on a separate PR: #16869
TF: rework XLA generate tests
Author: gante
Pull request: yes

# What does this PR do?

In the light of recent findings (https://github.com/huggingface/transformers/issues/16838), this PR reworks existing XLA generate tests. The following key changes were made:
1. Added a `@unittest.skipIf` on XLA generate tests, to skip when no GPU is present;
2. Rework XLA `sample` tests -- due to the minor numerical differences that arise when we use XLA (and that we can't control), the sampling step will gather different samples even when we use the same seed. The only thing we can properly test is whether a) we can seed them and b) the results are sensible;
3. Adds at least one XLA test where the batch size is > 1 and the inputs have different lengths, so we can confirm that masking works (GPT-2 is not working 💔 , added a TODO);
4. Removes redundant tests (we had tests outside the integration tests that were testing the same thing).

Comments:
_The documentation is not available anymore as the PR was closed or merged._

@patrickvonplaten reintroduced the fast tests, will merge as soon as CI gets to green
ModelForSequenceClassification (Roberta/Bert) don't support mean_pool
Author: gpucce
Pull request: no

# 🚀 Feature request

I hope I did not miss it, but it looks like there is no way to use `RobertaForSequenceClassification` or `BertForSequenceClassification` with mean_pool rather than `[CLS]` or `
` tokens as pooler. 

## Motivation

It would be useful for benchmarking and in some cases it can work better than the start of sequence token I believe?

## Your contribution

I can try to make the PR if it makes sense for the repo to add it

Comments:
I think @patrickvonplaten might be the person to tag to check if this slipped through.

If this doesn't make sense I'll close the issue!

Think in this case the best is to do a little tweak to the original modeling file and use this one instead :-) 

cc @sgugger - think that's also a good use case for: https://huggingface.co/docs/transformers/custom_models? 

BERT and RoBERTa supports the pooling that was introduced in their paper. Transformers is not a modular toolbox, so as Patrick said, if you want to tweak this, you will need to edit the modeling file, but we won't accept a change in the code to support this. And as Patrick pointed out, you can still share your tweaked model if it performs better!

Ok thanks! I will close the issue then
Minor fixes/improvements in `convert_file_size_to_int`
Author: mariosasko
Pull request: yes

Fix `convert_file_size_to_int`'s docstring example and the GB str to int conversion (per [this comment](https://github.com/huggingface/datasets/pull/4190#discussion_r856022534) by @lhoestq).

Comments:
_The documentation is not available anymore as the PR was closed or merged._
Add OnnxConfig for ConvBERT
Author: chainyo
Pull request: yes

# What does this PR do?

I have added OnnxConfig for ConvBERT model. I have checked all features available (which are the same as DistilBERT), so the features list seems to be good.

## Who can review?

Models:

- albert, bert, xlm: @LysandreJik


Comments:
_The documentation is not available anymore as the PR was closed or merged._

The conversion is working as demonstrated by the fact I successfully converted a base model and uploaded it on the hub.

Check the model [here](https://huggingface.co/OWG/convbert-base-spanish)

Here is the command I have used:

```bash
$ python -m transformers.onnx --model=mrm8488/convbert-base-spanish --feature=default onnx/

> Validating ONNX model...
> 	-[✓] ONNX model output names match reference model ({'last_hidden_state'})
> 	- Validating ONNX Model output "last_hidden_state":
>		-[✓] (2, 8, 768) matches (2, 8, 768)
>		-[✓] all values close (atol: 1e-05)
> All good, model saved at: onnx/model.onnx
```

> The conversion is working as demonstrated by the fact I successfully converted a base model and uploaded it on the hub.
> 
> Check the model [here](https://huggingface.co/OWG/convbert-base-spanish)

Awesome job on pushing this to the Hub! We have a PR in `optimum` coming soon that will allow you to also download these checkpoints directly as ONNX model objects: https://github.com/huggingface/optimum/pull/113

> Awesome job on pushing this to the Hub! We have a PR in `optimum` coming soon that will allow you to also download these checkpoints directly as ONNX model objects: [huggingface/optimum#113](https://github.com/huggingface/optimum/pull/113)

I dreamed about this and wanted to give it a shot if I had time (spoiler: not in the past months) but someone did it!!!

Btw the tests seem to pass!

I saw on another onnx config contribution that I could add the model to onnx tests, is it mandatory ? 
Check : #16887

> I saw on another onnx config contribution that I could add the model to onnx tests, is it mandatory ? Check : #16887

Ah yes, well spotted! You can pick one of the checkpoints from here and add it to the test: https://huggingface.co/models?search=convbert

This will ensure this export is tested as part of our daily CI with the slow tests :)

Thanks for including the test case - looks great!
TF: XLA repetition penalty
Author: gante
Pull request: yes

# What does this PR do?

This PR adds our first XLA-compatible TF logit processor, as well as corresponding tests. Since this is the first of a series of small (but similar) PRs, I'd like to request a more thorough review, so the remaining ones are quick.

More specifically, this PR makes three changes:
1. Rewrites the TF repetition penalty processor so as to be XLA-compatible;
2. Adds XLA tests for the processor;
3. Since the test mentioned in 2. was a near copy/paste of the non-XLA test, I've decided to split the test into three parts to improve code reuse and reduce errors from ad hoc edits (as the first and last part can be reused in the two versions of the test, XLA and non-XLA)
    - get inputs
    - run the processor
    - check the output

Comments:
_The documentation is not available anymore as the PR was closed or merged._

Thinking about it more, a multiplicative logit penalty really doesn't work, right? Even if we use the reciprocal when the logit is negative, the scale of the penalty depends on the logit's distance from 0. For example, a logit in the range -0.1 to +0.1 will barely be moved by the penalty term, but such logits usually have quite a high probability of being chosen, because most logits are large and negative.

(merging as the main goal was to port to XLA but, by all means, continue the discussion :) )
[WIP] Avoid BERT `attention_mask` from promoting dtype in self-attention `forward()`
Author: awgu
Pull request: yes

### Overview

~The `attention_mask` passed into `BertSelfAttention.forward()` is taken from the BERT model's first parameter dtype at **construction time**.~
<details>
  <summary>Code Pointers</summary>

  https://github.com/huggingface/transformers/blob/22fc93c4d9608fa9cd171b4f3044f8c756f86773/src/transformers/models/bert/modeling_bert.py#L985
https://github.com/huggingface/transformers/blob/22fc93c4d9608fa9cd171b4f3044f8c756f86773/src/transformers/modeling_utils.py#L655
https://github.com/huggingface/transformers/blob/22fc93c4d9608fa9cd171b4f3044f8c756f86773/src/transformers/modeling_utils.py#L556-L561
https://github.com/huggingface/transformers/blob/22fc93c4d9608fa9cd171b4f3044f8c756f86773/src/transformers/modeling_utils.py#L123-L125
</details>

However, if a user only casts down the input or parameter dtype(s) (e.g. to `torch.float16`) **after** construction time and before running `forward()`, the `extended_attention_mask` will still have dtype `torch.float32` rather than the reduced precision dtype.

Due to PyTorch [type promotion semantics](https://pytorch.org/docs/stable/tensor_attributes.html#type-promotion-doc), `attention_scores = attention_scores + attention_mask` will promote `attention_scores` to `torch.float32` and propagate through the remainder of the forward pass, defeating the intention of the reduced precision.
https://github.com/huggingface/transformers/blob/22fc93c4d9608fa9cd171b4f3044f8c756f86773/src/transformers/models/bert/modeling_bert.py#L343

In order to avoid this behavior, the model parameters must be cast to the reduced precision at construction time, which is restrictive. This PR ensures that the addition does not change the dtype.

**Before**
`attention_scores`: FP16
`attention_mask`: FP32
==> FP32
`attention_scores`: FP32
`attention_mask`: FP16
==> FP32

**After**
`attention_scores`: FP16
`attention_mask`: FP32 -> FP16
==> FP16
`attention_scores`: FP32
`attention_mask`: FP16 -> FP32
==> FP32

(Hence, we see that the reverse setting has no change in behavior, while for the desired setting, we prevent the unwanted type promotion.)

### Test Plan
I ran `tests/bert/test_modeling_bert.py` locally with 4 GPUs and 68 passed and 11 skipped. Since this is a small change, direct inspection may be more valuable.


Comments:
_The documentation is not available anymore as the PR was closed or merged._
Finetuning T5 for 2 tasks with 2 diferent prefixes and different data gives the same result.
Author: Oxi84
Pull request: no

Hello,

I am able to finetune T5 on paraphrasing task ... and the results are quite good. So for example when i train modelA on dataA and modelB on DataB, the produce different results, but then i train modelC and prefixA + Data A plus PrefixB + dataB the results with PrefixA and PrefixB are almost the same.

What could be the problem, is this an usual thing, because T5 should be able to handle multiple tasks at the same time.

I can train 2 models but I do not have enough memory for interface.


Comments:
This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md) are likely to be ignored.

It is too bad that models that is advertised of doing multipl;e tasks is actually not capable of it.
refactoring & fixing Pytorch QA examples
Author: searchivarius
Pull request: yes

# What does this PR do?

This PR improves/fixes Pytorch QA examples in several ways:
1. extracted duplicating post-processing functions (now shared between regular and no-trainer version)
2. added/fixed code to save statistics to some no-trainer versions (one was even buggy)
3. fixed squad v2-eval error in utils_qa.py (one was newly encountered and one that was previously reported. 
4. added forgotten model.eval() in the "no-trainer" versions. This fully fixed training of the **no-trainer** 
variant for the regular squad QA model. There might be still a small gap left for regular SQuAD (it might be just unlucky seed) and a big one for SQuAD v2. Please, see the numbers and the discussion below.

<!-- Remove if not applicable -->

Fixes #15401 (that was only partially fixed)


## Before submitting
- [ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [X] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#start-contributing-pull-requests),
      Pull Request section?
- [ ] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [ ] **I believe examples aren't covered by the documentation** you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/main/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/main/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests? **I trained squad and squad v2 models and compared results (see the discussion below)**, but I am not sure if any new tests are needed. 


## Who can review?

Anyone in the community is free to review the PR once the tests have passed. Perhaps, this can be of most interest for  @sgugger, @patil-suraj.

## Comparing old and new performance

Some remaining issues:
1. Despite the fixes & improvements, there's still a discrepancy between no-trainer and original version for SQuAD v2 or the beam-search version. 
2. In particular, for SQuAD v2 and the beam-search variant **without trainer**, both old and new numbers look very wrong to me. 

Please note that to be able to run SQuAD v2 tests, **I had to apply utils_qa.py fixes to the old code as well**. Otherwise, it would have just failed:

The metric is F1, the exact scores have the same pattern.

|                                   | previous | fixed |
|-----------------------------------|----------|-------|
| squad v1                          |     88.4 |  88.4 |
| squad v1 (no trainer)             |     86.7 |  88.5 |
| squad v1 (beam search)            |     92.1 |  92.1 |
| squad v1 (beam search no trainer) |     90.2 |  91.0 |
| squad v2 (beam search)            |     83.2 |  83.2 |
| squad v2 (beam search no trainer) |      4.9 |  50.1 |

Comments:
_The documentation is not available anymore as the PR was closed or merged._
refactoring & fixing Pytorch QA examples 
Author: searchivarius
Pull request: yes

# What does this PR do?

This PR improves/fixes Pytorch QA examples in several ways:
1. extracted duplicating post-processing functions (now shared between regular and no-trainer version)
2. fixed squad v2-eval error in utils_qa.py (one was newly encountered and one that was previously reported. 
3. added forgotten model.eval() in the "no-trainer" versions. This fully fixed training of the **no-trainer** 
variant for the regular squad QA model. There might be still a small gap left for regular SQuAD (it might be just unlucky seed) and a big one for SQuAD v2. Please, see the numbers and the discussion below.

<!-- Remove if not applicable -->

Fixes #15401 (that was only partially fixed)


## Before submitting
- [ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [X] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#start-contributing-pull-requests),
      Pull Request section?
- [ ] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [ ] **I believe examples aren't covered by the documentation** you make sure to update the documentation with your changes? 
- [ ] Did you write any new necessary tests? **I trained squad and squad v2 models and compared results (see the discussion below)**, but I am not sure if any new tests are needed. 


## Who can review?

Anyone in the community is free to review the PR once the tests have passed. Perhaps, this can be of most interest for  @sgugger, @patil-suraj.

## Comparing old and new performance

Some remaining issues:
1. Despite the fixes & improvements, there's still a discrepancy between no-trainer and original version for SQuAD v2 or the beam-search version. 
2. In particular, for SQuAD v2 and the beam-search variant **without trainer**, both old and new numbers look very wrong to me. 

Please note that to be able to run SQuAD v2 tests, **I had to apply utils_qa.py fixes to the old code as well**. Otherwise, it would have just failed:

The metric is F1, the exact scores have the same pattern.

|                                   | previous | fixed |
|-----------------------------------|----------|-------|
| squad v1                          |     88.4 |  88.4 |
| squad v1 (no trainer)             |     86.7 |  88.5 |
| squad v1 (beam search)            |     92.1 |  92.1 |
| squad v1 (beam search no trainer) |     90.2 |  91.0 |
| squad v2 (beam search)            |     83.2 |  83.2 |
| squad v2 (beam search no trainer) |      4.9 |  50.1 |

Comments:
_The documentation is not available anymore as the PR was closed or merged._
Fix typos BigBird ONNX conversion
Author: chainyo
Pull request: yes

# What does this PR do?

I tried to convert one `BigBird` model to ONNX with the recent PR merged #16427

But it seems that there is a typo in the `src/transformers/onnx/features.py` file.
I also fixed the typo in the test_v2 file.

Here is the error I get while trying to convert `google/bigbird-roberta-base`.

```bash
$ python -m transformers.onnx --model=google/bigbird-roberta-base onnx/

> KeyError: "big-bird is not supported yet. 
> Only ['albert', 'bart', 'mbart', 'bert', 'bigbird', 
> 'ibert', 'camembert', 'distilbert', 'flaubert', 'marian', 'm2m-100', 'roberta', 't5', 
> 'xlm-roberta', 'gpt2', 'gptj', 'gpt-neo', 'layoutlm', 'electra', 'vit', 'beit', 'blenderbot', 
> 'blenderbot-small', 'data2vec-text'] are supported. 
> If you want to support big-bird please propose a PR or open up an issue."
```

As you can see in the error `bigbird` should be `big-bird`.

ping @lewtun @LysandreJik 


Comments:
_The documentation is not available anymore as the PR was closed or merged._