noamaanMulla-03/transformers-issues huggingface dataset:
fix: Vocabulary build issue in run_speech_recognition.py
Author: shivamm7
Pull request: yes

# What does this PR do?
This PR fixes the issue of vocabulary building in run_speech_recognition_ctc.py (https://github.com/huggingface/transformers/blob/master/examples/pytorch/speech-recognition/run_speech_recognition_ctc.py#L332). 

While building vocabulary we replace the blank token " " by pipe symbol as delimiter ("|"). But if the dataset already contains the pipe symbol, which is common in Hindi language as Hindi language uses a similar character "।" as a full stop, in this case we will get two pipe symbols in the vocabulary. This will mess up the further process of creating the dictionary which is adding special tokens. Specifically, the bos token "\<s\>" and the pad token "[PAD]" will get the same index in vocabulary. This will cause the index with [PAD] token to be replaced by \<s\> and the output of Wav2Vec2ForCTC model to contain the "\<s\>" token.   

<!-- Remove if not applicable -->

Fixes #15275

## Who can review?
@patrickvonplaten , @anton-l , @Narsil

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_15285). All of your documentation changes will be reflected on that endpoint.

Hey @shivamm7,

Thanks for your issue - I can see the problem. However, instead of deleting `"|"` could you maybe just overwrite:

```
--word_delimiter_token 
```

with a token of your choice? E.g. you could do:

```bash
run_speech_recognition_ctc.py \
  --word_delimiter_token "&"
```

in this case `"&" would be used as the word delimiter token

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md) are likely to be ignored.
Update from keras2onnx to tf2onnx
Author: gante
Pull request: yes

# What does this PR do?

[keras2onnx](https://github.com/onnx/keras-onnx) has been dead for a while, with the development being made in [tf2onnx](https://github.com/onnx/tensorflow-onnx/tree/6c417b4210285208ba7165068318542d9e3d9656). It is the successor, yet also retrocompatible.

In fact, our tests were failing right at the import level (i.e. `import keras2onnx` fails with recent `keras` versions). These changes fix the tests for TF+ONNX.

Example of test command that gets fixed with this PR:
```RUN_SLOW=1 pytest tests/test_modeling_tf_bert.py::TFBertModelTest::test_onnx_runtime_optimize```

Comments:
Add TF glu activation function
Author: gante
Pull request: yes

# What does this PR do?

Bite-sized PR that adds the GLU activation function for TF, as defined in the original paper. It is a requirement for a few speech models, like `Speech2Text`.

We can see that it matches PyTorch's implementation
```python
import numpy as np
import torch
from transformers.activations_tf import glu
a = np.asarray([[1., 2., 3., 4.], [5., 6., 7., 8.]])
assert np.allclose(torch.nn.functional.glu(torch.tensor(a), dim=0).numpy(), glu(a, axis=0).numpy())
assert np.allclose(torch.nn.functional.glu(torch.tensor(a), dim=1).numpy(), glu(a, axis=1).numpy())
assert np.allclose(torch.nn.functional.glu(torch.tensor(a)).numpy(), glu(a).numpy())
```

After I get one model using it fully operational, I'm going to attempt to add it to Keras 😎 

Comments:
A little late but looks good to me!
Added support for other features for already supported models
Author: michaelbenayoun
Pull request: yes

 # What does this PR do?

This PR adds support for almost all the features available for already supported models.

Main contributions:

- `OnnxSeq2SeqConfigWithPast`: a new class  inheriting from `OnnxConfigWithPast` designed specifically for seq2seq models, this should make things easier for the community to contribute.
- Tests refactoring and parameterization: now every (model, feature) export pair is tested, and is considered as a standalone test (compared to before when everything was considered to be one big test).
- A lot of new features (a feature is a task plus the choice or not to use `past_key_values`), that have been requested by the community (check the list of supported feautres below)

Features now supported:

- For BERT like models: default, sequence-classification, token-classification and question-answering (multiple-choice will be added later).
- For causal language models (GPT-2 and GPT-neo): default, default-with-past, causal-lm, causal-lm-with-past, sequence-classification and token-classification (only for GPT2).
- For Seq2Seq models (T5, BART, mBART): 
     - T5, BART, mBART: default, default-with-past, seq2seq-lm, seq2seq-lm-with-past
     - BART, mBART: causal-lm, causal-lm-with-past, sequence-classification, question-answering


Comments:
@michaelbenayoun @lewtun @Albertobegue 
any idea when this PR will be merged ?

> @michaelbenayoun @lewtun @Albertobegue any idea when this PR will be merged ?

Hey @girishnadiger-gep this PR was superseded by #14700 which has been merged sometime ago. Is there a specific issue or feature missing that you're interested in?

Hi @lewtun ,
Thanks for getting back. I was trying to implement BART Summarization on onnx. Facing a weird issue where the inputs to the model are 4 `('input_ids', 'attention_mask', 'decoder_input_ids', 'decoder_attention_mask')`, but unable to figure out how to get `'decoder_input_ids'` and `'decoder_attention_mask'` values to feed to the onnx model.

I've converted a BART Large model to onnx model using 'seq2seq-lm' feature, but thought i'm missing something here so asked in this forum.

Ah for that you can probably adapt the example that I used for the Marian PR in https://github.com/huggingface/transformers/pull/14586

FYI we also have a forum (https://discuss.huggingface.co/) which is better suited for these type of questions - we try to use GitHub issues for bug reports / feature requests
[ViLT] Add link to notebooks
Author: NielsRogge
Pull request: yes

# What does this PR do?

Add link to my ViLT notebooks in the docs.

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_15791). All of your documentation changes will be reflected on that endpoint.

I think we should consolidate all the notebooks linked from the documentation inside the [notebooks repository](https://github.com/huggingface/notebooks) (apart from the community page of course). Would you mind mirroring your notebook there, with a clear line at the top "This notebook is a copy of the notebook developed by @NielsRogge [here](linked to notebook in your repo]"?

Can I keep those in my repo Transformers-Tutorials and link to that repo instead?

I'm kind of free with the format there, it's not really following the official example notebooks (I'm mostly using native PyTorch for training).
use python 3.7 for flax self-push tests
Author: patil-suraj
Pull request: yes

# What does this PR do?

Use python 3.7 for flax self-push tests. This PR also removes`kenlm` installation since it was  causing some issues and is actually not required for flax tests.

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_15865). All of your documentation changes will be reflected on that endpoint.

Thanks for working on it! Is the `apt` installation necessary when you already specified the Python version you wanted in the yaml?

> Is the apt installation necessary when you already specified the Python version you wanted in the yaml?

Aah good catch! No, it's not. It is necessary for `kenlm` but it's not required anymore.
No self-hosted
Author: LysandreJik
Pull request: yes

Removes the need to have a self-hosted machine to build the dev documentation.

Comments:
@coyotte508 I'm getting this error, do you have an idea where it might come from?

```
Installing node dependencies
Building HTML files. This will take a while :-)
Traceback (most recent call last):
  File "/usr/local/bin/doc-builder", line 33, in <module>
    sys.exit(load_entry_point('doc-builder', 'console_scripts', 'doc-builder')())
  File "/__w/transformers/transformers/doc-builder/src/doc_builder/commands/doc_builder_cli.py", line 39, in main
    args.func(args)
  File "/__w/transformers/transformers/doc-builder/src/doc_builder/commands/build.py", line 145, in build_command
    subprocess.run(
  File "/usr/local/lib/python3.8/subprocess.py", line 516, in run
    raise CalledProcessError(retcode, process.args,
subprocess.CalledProcessError: Command '['npm', 'run', 'build']' returned non-zero exit status 134.
```

> @coyotte508 I'm getting this error, do you have an idea where it might come from?
> 
> ```
> Installing node dependencies
> Building HTML files. This will take a while :-)
> Traceback (most recent call last):
>   File "/usr/local/bin/doc-builder", line 33, in <module>
>     sys.exit(load_entry_point('doc-builder', 'console_scripts', 'doc-builder')())
>   File "/__w/transformers/transformers/doc-builder/src/doc_builder/commands/doc_builder_cli.py", line 39, in main
>     args.func(args)
>   File "/__w/transformers/transformers/doc-builder/src/doc_builder/commands/build.py", line 145, in build_command
>     subprocess.run(
>   File "/usr/local/lib/python3.8/subprocess.py", line 516, in run
>     raise CalledProcessError(retcode, process.args,
> subprocess.CalledProcessError: Command '['npm', 'run', 'build']' returned non-zero exit status 134.
> ```

Yes, out of memory

Settings the env `NODE_OPTIONS` as in https://github.com/huggingface/transformers/blob/master/.github/workflows/build_documentation.yml#L93 should help

@sgugger maybe there's a way to get the npm error output with `doc-builder`?

Here the dev documentation build seems to fail on `npm ci` but we don't get more details

@coyotte508 if you comment out [these lines](https://github.com/huggingface/doc-builder/blob/3153b60a541532daf218a976199004776699553c/src/doc_builder/commands/build.py#L147-L148), you'll see all the logs (including warning, which generates a lot of noise, maybe we can turn off wanrings from svletekit side)

build-dev-documentation didn't seem to run for that last commit (maybe due to the PR conflict? =/)

It ran, and it worked! Thank you @mishig25 and @coyotte508 for your help.
[vision] Add problem_type support
Author: NielsRogge
Pull request: yes

# What does this PR do?

* add `problem_type` support to vision backbones that weren't supporting this before
* make sure all vision backbones are tested for this

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_15851). All of your documentation changes will be reflected on that endpoint.
Inference for multilingual models
Author: stevhliu
Pull request: yes
Labels: name: Documentation

This PR updates the multilingual model docs to include M2M100 and MBart whose inference usage is different (basically forcing the `bos` token as the target language id). Let me know if I'm missing any other multilingual models that have a different method for inference! :)

Other notes:
- Moved this doc to the How-to guide section because it seems like it is more of an intermediate-level topic. 
- Changed a section heading because the ampersand wasn't being properly displayed in the right navbar:
![Screen Shot 2022-02-25 at 9 40 57 AM](https://user-images.githubusercontent.com/59462357/155762069-ef1314a3-2474-48f8-85db-36dc78234c77.png)



Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_15836). All of your documentation changes will be reflected on that endpoint.
fix deepspeed tests
Author: stas00
Pull request: yes

Making deepspeed tests work again after the big tests structure reshuffle.

@LysandreJik - we can't have `tests/deepspeed/__init__.py` as then it conflicts with the actual `deepspeed` package, please see: https://github.com/huggingface/transformers/pull/15725#issuecomment-1056027398

Additionally I don't think we should turn tests into packages, as it'd cause all kinds of problems. Instead, I think we should extract any package-worthy code that is re-used by multiple tests and put it under `src/transformers` like `testing_utils.py`.

Please let me know if the failing doc builder is a false alarm. Thanks.


Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_15881). All of your documentation changes will be reflected on that endpoint.
M2M100 support for ONNX export
Author: michaelbenayoun
Pull request: yes

# What does this PR do?

Enable ONNX export for M2M100 models for the following tasks:

- default
- default-with-past
- seq2seq-lm
- seq2seq-lm-with-past

Related issue: #15060 

Comments:
The export for `seq2seq-lm` and `seq2seq-lm-with-past` is failing due to 
```
RuntimeError: Exporting model exceed maximum protobuf size of 2GB. Please call torch.onnx.export without setting use_external_data_format parameter.
```
The `OnnxConfig.use_external_data_format` finds that the size is ok... working on it!

Also the export works for `valhalla/m2m100_tiny_random`, which is very small, meaning that the export is working, so it is related to the model size.

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md) are likely to be ignored.

The try / except change aims at poiting the user to use a more recent torch version to (possibly) be able to export the model as the protobuf issue seems solved starting from torch 1.10 (when not specifying the deprecated arguments).
A few minor questions regarding run_summarization.py in examples
Author: JJumSSu
Pull request: no

Hi, thank you for the amazing repo.

I'm currently using the script `run_summarization.py` from examples (I'm using ver 4.15.0).
When I run the code below, it seems that my trained model is not using beam search during generation.
Am I correct?

```
python examples/pytorch/summarization/run_summarization.py \
    --model_name_or_path t5-small \
    --do_train \
    --do_eval \
    --dataset_name cnn_dailymail \
    --dataset_config "3.0.0" \
    --source_prefix "summarize: " \
    --output_dir /tmp/tst-summarization \
    --per_device_train_batch_size=4 \
    --per_device_eval_batch_size=4 \
    --overwrite_output_dir \
    --predict_with_generate
```

Furthermore, when I set `--evaluation_strategy` to either `epochs `or `steps `, the performance of the validation set during training and the final performance of the validation set measured at the end of training (`do_eval`) seem quite different. 
Is this behavior expected? or am I missing something (such as only using beam search for the final measurement).

Thank you!



Comments:
Also, what does `task_specific_params `do during running the script `run_summarization.py`?
When I am initializing the T5-model, the model configuration includes such parameters.
```
 "task_specific_params": {
    "summarization": {
      "early_stopping": true,
      "length_penalty": 2.0,
      "max_length": 200,
      "min_length": 30,
      "no_repeat_ngram_size": 3,
      "num_beams": 4,
      "prefix": "summarize: "
    }
```
Thank you!

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md) are likely to be ignored.
Add TF generate sample tests with all logit processors
Author: gante
Pull request: yes

# What does this PR do?

This PR adds a GPT2 TF generate sample test with all logit processors. It is a requirement for the TF generate sample refactor (https://github.com/huggingface/transformers/pull/15793), which is part of the TF generate refactor.

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_15852). All of your documentation changes will be reflected on that endpoint.

Also, code quality is failing, be sure to `make style`!

> Is the plan to add a single specific test for each logit processor, as they're converted?

Yeah, in the other PR (master doesn't have them yet)

Perfect! Sorry to be a bit annoying here - could you also add one random test for T5? I know that this is not a very common use case for encoder-decoder settings, but the encoder-decoder setting is significantly different from decoder-only so that it would be good to be sure that nothing is broken there before the refactor. Think you can take more or less the exact same input that you used for GPT2 (the output might not be that sensible, but anyways good to test encoder-decoder in sample mode).

@patrickvonplaten now with a T5 generate sample test. I took the liberty to find a combination of inputs where the output of one is very unstable (the first one), and the output of the other is pretty stable (the second one). It may help us distinguish the case where the new generate is completely different from the case where there are minor numerical differences.
Broken link error when using the CLIPTokenizerFast class
Author: AntreasAntoniou
Pull request: no

## Environment info


- `transformers` version: 4.17.0.dev0
- Platform: Linux-5.13.0-30-generic-x86_64-with-glibc2.10
- Python version: 3.8.12
- PyTorch version (GPU?): 1.11.0.dev20220116 (True)
- Tensorflow version (GPU?): not installed (NA)
- Flax version (CPU?/GPU?/TPU?): not installed (NA)
- Jax version: not installed
- JaxLib version: not installed
- Using GPU in script?: <fill in>

### Who can help

Models:
@LysandreJik The model is clip-vit-base-patch16

Library:

- Tokenizers: @SaulLu

## Information
I am suddenly getting the error 
```requests.exceptions.HTTPError: 502 Server Error: Bad Gateway for url: https://huggingface.co/openai/clip-vit-base-patch16/resolve/main/vocab.json``` when trying to use the CLIP tokenizer. 

## To reproduce

Steps to reproduce the behavior:

1. Just execute 
`tokenizer = CLIPTokenizerFast.from_pretrained("openai/clip-vit-base-patch16")`

To get the error message:

``` File "/home/evolvingfungus/miniforge3/envs/tali/lib/python3.8/site-packages/transformers/tokenization_utils_base.py", line 1763, in from_pretrained
    raise err
  File "/home/evolvingfungus/miniforge3/envs/tali/lib/python3.8/site-packages/transformers/tokenization_utils_base.py", line 1724, in from_pretrained
    resolved_vocab_files[file_id] = cached_path(
  File "/home/evolvingfungus/miniforge3/envs/tali/lib/python3.8/site-packages/transformers/file_utils.py", line 1921, in cached_path
    output_path = get_from_cache(
  File "/home/evolvingfungus/miniforge3/envs/tali/lib/python3.8/site-packages/transformers/file_utils.py", line 2125, in get_from_cache
    _raise_for_status(r)
  File "/home/evolvingfungus/miniforge3/envs/tali/lib/python3.8/site-packages/transformers/file_utils.py", line 2052, in _raise_for_status
    request.raise_for_status()
  File "/home/evolvingfungus/miniforge3/envs/tali/lib/python3.8/site-packages/requests/models.py", line 960, in raise_for_status
    raise HTTPError(http_error_msg, response=self)
requests.exceptions.HTTPError: 502 Server Error: Bad Gateway for url: https://huggingface.co/openai/clip-vit-base-patch16/resolve/main/vocab.json
```

## Expected behavior

Normally the snippet should download the CLIP text encoder's vocab and form a tokenizer.


Comments:
@AntreasAntoniou , I'm so sorry you had this problem. Part of the Hub was briefly unavailable on March 1st around 11pm UTC. I just tested again on my side and your snippet works fine. :smile: 



Yeah, it works fine now. Thanks for the response! :)
TF: Update QA example
Author: gante
Pull request: yes

# What does this PR do?

Another example update, this time for question answering. Removes the dummy loss (we can use the internal loss) and the custom dataset preparation.

@Rocketknight1 in this example, the evaluation dataset has some custom handling. Do you think it is worth diving into?

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_15870). All of your documentation changes will be reflected on that endpoint.

@gante If you mean the bit that's in the PT example too, then we probably have to keep that! (And you don't need to change it for the refactor)
Question: how to evaluate word similarity with context?
Author: JanSawicki
Pull request: no

## ❓ Questions & Help

Hello, 
I am trying to evaluate **word similarity** (particularly named entities). 

Example:
`I flew to Paris last month.`
`I met a famous celebrity, Paris Hilton.`

Most named entity recognition models (e.g. the default `pipeline('ner')`) will return `Paris` and `Paris Hilton` as named entities. 
They look similar, but they represent different entities. My question is: 
### How can I determine that two hyponymic words/phrases are different (from context)?

I tried this solution: 
https://discuss.huggingface.co/t/generate-raw-word-embeddings-using-transformer-models-like-bert-for-downstream-process/2958/2
and then compared with cosine similarity, but as mentioned in this issue https://github.com/huggingface/transformers/issues/2298, cosine similarity does not work well with BERT (and similar like models) embeddings. 

I also looked into the topic of WSD (Word Similarity Disambiguation), but these solution work rather for "standard" hyponyms (like `baseball bat` and `cave bat`), but named entities are usually _special_ and are NOT present in dictionary. 

Any help or suggestion will be more than welcome! 🤗

Comments:
Hello, thanks for opening an issue! We try to keep the github issues for bugs/feature requests.
Could you ask your question on the [forum](https://discuss.huggingface.co) as well?

Thanks!

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md) are likely to be ignored.
[SegFormer] Add deprecation warning
Author: NielsRogge
Pull request: yes

# What does this PR do?

This PR takes it a bit more light compared to #15748. Rather than removing the `reshape_last_stage` argument right away, this PR instead tells users that this argument is deprecated and will soon be removed.

After some weeks, we can then merge #15748.

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_15889). All of your documentation changes will be reflected on that endpoint.
Fix SegformerForImageClassification
Author: NielsRogge
Pull request: yes

# What does this PR do?

After #15889 and removing the deprecated `reshape_last_stage` attribute for 6 checkpoints on the hub, this PR makes sure `SegformerForImageClassification` properly uses the features from the backbone in order to compute the image class.

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_15895). All of your documentation changes will be reflected on that endpoint.