noamaanMulla-03/transformers-issues huggingface dataset:
Different inference results from local transformer vs inference API
Author: logandeboo
Pull request: no

### System Info

I am getting two slightly different probability values when comparing inference results from the local transformer and inference API on the same sentence. I am wondering why this is happening? It only occurs for some sentences.

<img width="1617" alt="Screen Shot 2023-02-13 at 7 46 51 PM" src="https://user-images.githubusercontent.com/49734611/218634176-73911bbc-26a0-443c-8aac-96329a3d613f.png">


Moreover, the local transformer seems to select the highest probability result and return it alone compared to the API that returns a score for each label. Sometimes a score from the API is greater than 1 (have seen 9) and I am wondering why that is and am if it invalidates the results?


Cheers!

### Who can help?

_No response_

### Information

- [X] The official example scripts
- [X] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

<img width="1612" alt="Screen Shot 2023-02-13 at 7 53 26 PM" src="https://user-images.githubusercontent.com/49734611/218635058-6322388b-2f50-48c3-8c5d-e3357a125002.png">


### Expected behavior

Naturally I expect each version of the model to produce the same score.

Comments:
cc @Narsil 

Small differences in numbers can be explained by hardware, torch version etc... Nothing can be done about it.

For the difference in output the  API uses a different default from the pipeline `pipe = pipeline(..., topk=None)` as it makes more sense for the widget to see multiple proposition. 
In addition the results are sorted for the API (again for UX).

Are you able to reproduce larger than 1 results ? Seems like a pretty bad bug if true !

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.

I am having the same issue. I though it may be due to me using TF instead of pytorch, or as was suggested by hardware differences. I am however seeing bigger difference than yours, the inference api gets me some positives while the local model some (false) negatives.
Converting wav2vec2-base-960h to ONNX report an error while converting
Author: Denovitz
Pull request: no

First of all, I want to say thanks to @patrickvonplaten for the work done in adding the model. Great job!

I tried to convert the model to ONNX but got an error, do you have any ideas how to fix it?

What I did:
`python -m transformers.convert_graph_to_onnx --framework pt  --model facebook/wav2vec2-base-960h wav2vec2-base-960h.onnx`

But got an error:
```====== Converting model to ONNX ======
ONNX opset version set to: 11
Loading pipeline (model: facebook/wav2vec2-base-960h, tokenizer: facebook/wav2vec2-base-960h)
Using framework PyTorch: 1.7.0
Error while converting the model: Unable to create tensor, you should probably activate truncation and/or padding with 'padding=True' 'truncation=True' to have batched tensors with the same length.
```


Comments:
Also interested in this question! 

Hmm, Wav2Vec2 is still a very recent addition and I don't have a good idea on an ETA for full ONNX support. However, I think your error above is due to the input that's passed to `Wav2Vec2Tokenizer` being a string instead of a speech input. So in order to make the conversion work, you will have to tweak the script `convert_graph_to_onnx` yourself a bit for Wav2Vec2 - I think the only different should be that instead of passing it `"This is a sample output"` you should pass it a 1D float array.

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md) are likely to be ignored.

Hi!
very interested in this question! did anyone managed to make it work ? 


Hey @OthmaneJ,

Think @ccoreilly managed to get it to work here: https://github.com/ccoreilly/wav2vec2-service/blob/master/convert_torch_to_onnx.py

@patrickvonplaten thanks! 👌

hi @patrickvonplaten 
 is there any way to transform mms asr model to onnx?
if yes, how?
thank you very much!
Use `weights_only` only if torch >= 1.13
Author: ydshieh
Pull request: yes

# What does this PR do?

Fix https://github.com/huggingface/transformers/pull/27282#issuecomment-1887859328

cc @julien-c 

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_28506). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

failing tests are known flaky
Inconsistency between fast and slow codellama tokenizers
Author: UniverseFly
Pull request: no

### System Info

- `transformers` version: 4.33.2
- Platform: Linux-5.15.0-82-generic-x86_64-with-glibc2.35
- Python version: 3.10.12
- Huggingface_hub version: 0.16.4
- Safetensors version: 0.3.3
- Accelerate version: 0.22.0
- Accelerate config:    not found
- PyTorch version (GPU?): 2.0.1+cu117 (True)
- Tensorflow version (GPU?): not installed (NA)
- Flax version (CPU?/GPU?/TPU?): not installed (NA)
- Jax version: not installed
- JaxLib version: not installed
- Using GPU in script?: no
- Using distributed or parallel set-up in script?: no

### Who can help?

@ArthurZucker

### Information

- [ ] The official example scripts
- [X] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [X] My own task or dataset (give details below)

### Reproduction

A simple reproduce:

```python
from transformers import AutoTokenizer

t_fast = AutoTokenizer.from_pretrained("codellama/codellama-7B-Instruct-Hf", use_fast=True)
t_slow = AutoTokenizer.from_pretrained("codellama/codellama-7B-Instruct-Hf", use_fast=False)

ids_fast = t_fast.encode("

", add_special_tokens=False)
ids_slow = t_slow.encode("

", add_special_tokens=False)

assert ids_fast == ids_slow, f"Fast: {ids_fast}, Slow: {ids_slow}"
# AssertionError: Fast: [1, 518, 25580, 29962], Slow: [1, 25580, 29962]
```

### Expected behavior

I'm not sure which one is correct. Actually decoding the fast tokenizer outputs will get `'
 
'`, while the slow tokenizer `'
INST]'`, both not same as the original string.

Comments:
Hey! This was already reported and is a duplicate of #25881 and related to #26318 as well. Will be fixed in tokenizers soon! 

The PR is looking good, had a few delays but was able to reproduce 1-1 with fast but also with sentencepiece actual token addition.
[`Core Tokenization`] Support a fix for spm fast models
Author: ArthurZucker
Pull request: yes

# What does this PR do?
Fixes the behaviour of `T5FastTokenizer` using the `legacy` flag. 
Requires `tokenizers==0.15.0`.
This is what it will allows us to do:
```python 
from transformers import AutoTokenizer
tok = AutoTokenizer.from_pretrained("t5-base", use_fast = True, from_slow = True, legacy = True)
print(tok.tokenize("Hey 
. how are you"))
tok = AutoTokenizer.from_pretrained("t5-base", use_fast = True, from_slow = True, legacy = False)
print(tok.tokenize("Hey 
. how are you"))
```
We add `bos_token="
"` to make sure it does not strip left, further amphasizing the issue at hand.

```python 
['▁Hey', '▁', '
', '▁.', '▁how', '▁are', '▁you']
['▁Hey', '▁', '
', '.', '▁how', '▁are', '▁you']
```
The extra space that was always added to the eos is now gone. 
This is fully backward compatible and can be saved / push to the hub. The metaspace rust object was not broken, and the argument can also be easily set `tok._tokenizer.pre_tokenizer.legacy = legacy`. 

fixes #26318, fixes #26455, fixes #25881, fixes #27900  and more to come

# TODOs

- [x] Add tests/update existing ones
- [x] support T5 only for now
- [ ] add some documentation about this prepend scheme

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_26678). All of your documentation changes will be reflected on that endpoint.

Oups 😅 just need to document and fx the tests

PR is overall ready, just missing CIs, rebase and core review
Use `LoggingLevel` context manager in 3 tests
Author: ydshieh
Pull request: yes

# What does this PR do?

To avoid flaky failing tests due to transformers' root logger's level being changed by some (unknown yet) other tests.

Comments:
I don't run CI multiple times with this PR - but quite confident this would fix the current situation.

I forgot to remove `is_flaky`. Will do before merge 😅 

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_28575). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
Fix the documentation checkpoint for xlm-roberta-xl
Author: jeremyfowers
Pull request: yes

# What does this PR do?

This is a small PR that corrects the name of the `xlm-roberta-xl` checkpoint in the `transformers` documentation.

I also noticed that the docstrings were referring to the model as either `XLM-RoBERTa-xlarge` or `XLM-Roberta-xlarge` and I corrected all of those instances to `XLM-RoBERTa-XL`.

<!-- Remove if not applicable -->

Fixes #28562


## Before submitting
- [x] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [x] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#start-contributing-pull-requests),
      Pull Request section?
- [x] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [ ] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/main/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/main/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests?


## Who can review?

Anyone in the community is free to review the PR once the tests have passed. Feel free to tag
members/contributors who may be interested in your PR.

@julien-c @stevhliu


Comments:
thanks for fixing!
The examples for xlm-roberta-xl reference a model that doesn't exist
Author: jeremyfowers
Pull request: no

### System Info

- `transformers` version: 4.36.2
- Platform: Windows-10-10.0.22621-SP0
- Python version: 3.8.18
- Huggingface_hub version: 0.20.2
- Safetensors version: 0.4.1
- Accelerate version: not installed
- Accelerate config: not found
- PyTorch version (GPU?): 2.1.2+cpu (False)
- Tensorflow version (GPU?): not installed (NA)
- Flax version (CPU?/GPU?/TPU?): not installed (NA)
- Jax version: not installed
- JaxLib version: not installed
- Using GPU in script?: <fill in>
- Using distributed or parallel set-up in script?: <fill in>

### Who can help?

@ArthurZucker @younesbelkada 

### Information

- [X] The official example scripts
- [ ] My own modified scripts

### Tasks

- [X] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

Run the example code for `XLMRobertaXLForMaskedLM` verbatim in python: https://huggingface.co/docs/transformers/model_doc/xlm-roberta-xl#transformers.XLMRobertaXLForMaskedLM.forward.example


Example code pasted here for convenience:
```
from transformers import AutoTokenizer, XLMRobertaXLForMaskedLM
import torch

tokenizer = AutoTokenizer.from_pretrained("xlm-roberta-xlarge")
model = XLMRobertaXLForMaskedLM.from_pretrained("xlm-roberta-xlarge")

inputs = tokenizer("The capital of France is <mask>.", return_tensors="pt")

with torch.no_grad():
    logits = model(**inputs).logits

# retrieve index of <mask>
mask_token_index = (inputs.input_ids == tokenizer.mask_token_id)[0].nonzero(
    as_tuple=True
)[0]

predicted_token_id = logits[0, mask_token_index].argmax(axis=-1)

labels = tokenizer("The capital of France is Paris.", return_tensors="pt")["input_ids"]
# mask labels of non-<mask> tokens
labels = torch.where(inputs.input_ids == tokenizer.mask_token_id, labels, -100)

outputs = model(**inputs, labels=labels)
```

This results in:

```
OSError: xlm-roberta-xlarge is not a local folder and is not a valid model identifier listed on 'https://huggingface.co/models'
If this is a private repository, make sure to pass a token having permission to this repo either by logging in with `huggingface-cli login` or by passing `token=<your_token>`
```

### Expected behavior

This should find and run the model. However, it does not. Replacing the model string from `"xlm-roberta-xlarge"` to `"facebook/xlm-roberta-xl"` fixes the problem.

Comments:
Nice catch – would you be able to open a PR to fix the model reference?

> Nice catch – would you be able to open a PR to fix the model reference?

yes @julien-c, if you're happy with the new model reference then I would be happy to make the PR.
Model generate with batch_size for Seq2SeqLM
Author: kedarthakkar
Pull request: no

### Feature request

When running `model.generate`  with a `Seq2SeqLM`, I've run into OOM issues in resource-constrained environments (i.e. Google Colab notebook) when passing in a large batch (i.e. 1000 samples). I propose adding a `batch_size` parameter to `model.generate` so that `input_ids` can be loaded in batches for sequence generation rather than loading the entire tensor into memory from the start. Would appreciate thoughts and suggestions on this from more experienced folks.

### Motivation

Motivation is to prevent OOM issues when loading large datasets for inference on resource-constrained environments.

### Your contribution

I can work on implementing this if it makes sense to add as a feature.

Comments:
Another option could be to instead define a function `model.generate_streaming` which takes in `batch_size` and yields generated sequences in `batch_size` increments. This would then allow the user to determine what they want to do with the yielded sequences (i.e. write them to a file or compute some metrics).

cc @gante 

Hi @kedarthakkar 👋 

You can easily control the batch size with a few extra lines before calling `model.generate(), e.g. looping over your batch using smaller batches as input to the function. That is a much simpler solution than adding tens of lines of additional complexity to an already complex function :)

Ok, makes sense, I'll close this issue.
add context-free grammar constrained decoding(ebnf interface) into reserach project directory
Author: Saibo-creator
Pull request: yes

This PR is a follow-up from PR #27557 where @gante has suggested putting this feature as a research project.
It adds a new feature: Context Free Grammar Constrained Decoding, similarly to what llama-cpp has. 
It provides the (almost )same interface as llama-cpp.

Fixes #25778 


## Before submitting
- [ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [x] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#start-contributing-pull-requests),
      Pull Request section?
- [x] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? https://github.com/huggingface/transformers/pull/27557
- [x] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/main/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/main/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests?


## Who can review?
@gante 

Anyone in the community is free to review the PR once the tests have passed. Feel free to tag
members/contributors who may be interested in your PR.

Comments:
Hello @gante ,
    Thank you for your feedback. I think the first option is better. I will close this PR and keep my repo. 
    My rationales are:
    The grammar constrained generation is not yet complete. It doesn't support all tokenizers and unicode. I will continue to improve it. 
    As a standalone repo, it is easier to maintain and update.

    Thank you for offering to amplify my work on social media. 
    I will let you know when I have a more complete version of the grammar constrained generation. Now it's working but no thorough testing has been done yet.
    I know that it may have unexpected behavior in some cases, tho users may even not notice it.

@Saibo-creator perfect!

Let me know when you think it's ready, so we can post on social media about it 🙌
Add Tortoise TTS to HF Pipeline
Author: fakerybakery
Pull request: no

### Feature request

Hi,
Might it be possible to add [Tortoise TTS](https://github.com/neonbjb/tortoise-tts) to the `text-to-speech` pipeline?

### Motivation

Tortoise TTS is currently the highest-quality permissively licensed text-to-speech library available.

### Your contribution

Tortoise TTS is already pip-ified so it shouldn't be too hard to add.

Comments:
cc @sanchit-gandhi @ylacombe 

I saw [this issue](https://github.com/huggingface/diffusers/issues/3891), however personally I think it would be great to have a unified way to run TTS models. Transformer's `pipeline` has the most support right now, and pipelines in Diffusers are harder to use. Also, it would make it easier to switch between models without significant changes to code.

Hey @fakerybakery - Tortoise TTS is a pipeline composed of two transformer-based models and one diffusion one. `diffusers` pipelines are designed to work with a mix of transformer/diffusion models, whereas `transformers` pipelines are designed to work with single transformer-based model. Therefore, the pipeline is a better fit for `diffusers`, for which there is an on-going PR to add it: https://github.com/huggingface/diffusers/pull/4106

Note that the pipeline will be still very easy to work with in `diffusers`, emulating a similar API to AudioLDM: https://huggingface.co/cvssp/audioldm-s-full-v2#text-to-audio

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.
save_pretrained no longer works for AutomaticSpeechRecognitionPipeline
Author: bruno-hays
Pull request: no

### System Info

transformers-4.37.0.dev0

### Who can help?

@Narsil 

### Information

- [X] The official example scripts
- [ ] My own modified scripts

### Tasks

- [X] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

```python
from transformers import pipeline

asr_pipeline = pipeline('automatic-speech-recognition',
                        model="openai/whisper-tiny")

asr_pipeline.save_pretrained("pipeline_save")
```
Gives this error:
```
Traceback (most recent call last):
  File "/Users/bruno/testing.py", line 6, in <module>
    asr_pipeline.save_pretrained("pipeline_save")
  File "/Users/bruno/venv/lib/python3.11/site-packages/transformers/pipelines/base.py", line 883, in save_pretrained
    if self.image_processor is not None:
       ^^^^^^^^^^^^^^^^^^^^
AttributeError: 'AutomaticSpeechRecognitionPipeline' object has no attribute 'image_processor'
```

### Expected behavior

The pipeline should be saved.

save_pretrained a pipeline is used by BentoML, as a result versions of transformers newer than 4.32.1 cannot be used to serve a AutomaticSpeechRecognitionPipeline with bentoML.

https://github.com/bentoml/BentoML/issues/4339

Comments:
Hi @Hubert-Bonisseur, thanks for raising this issue! 

Indeed, this seems to be the result of two PRs - #25884 - which enables saving image processors in pipelines and #25438 which adapted ASR's pipeline's init. 

cc @sanchit-gandhi who knows more about the decisions in #25438 and the reason for removing the `super().__init__` call

Overriding the init was required to ensure we could correctly set ASR-specific decoding params when we instantiate the pipeline: https://github.com/huggingface/transformers/pull/25438#issuecomment-1676918766

Here's a PR to fix the `save_pretrained` functionality of the ASR pipeline: https://github.com/huggingface/transformers/pull/28486
[ASR Pipe] Update init to set model type and subsequently call parent init method
Author: sanchit-gandhi
Pull request: yes

# What does this PR do?

Fixes #28162 by overriding the init method of the ASR pipeline class. We first set the model type, then call the parent init method.


Comments:
The problem with the current solution is that it's a patch over a larger issue: the ASR pipeline is a child of `Pipeline` but doesn't share the `__init__`. So, if there's any new logic added to `Pipeline` which relies on attributes being set or new kwargs this will still fail. 

It also doesn't make sense for this class to accept an image processor. 

The simplest solution (but doesn't address the divergence here) would be making certain attributes - `tokenizer`, `feature_extractor`, `image_processor` - to class attributes with default `None`. This way, any new pipeline which is similar to ASR will already have these required objects without having to handle / set them. 

A better solution is to update the `__init__` so that `super().__init__(*args, **kwargs)` is called at some point either modifying the kwargs before, setting self.type where necessary or adding any additional logic after. 

Thanks for your comments. I agree that calling the parent `__init__` method is a more elegant solution. Resolved in https://github.com/huggingface/transformers/pull/28486/commits/e942642024aff014d5e7cd739ce7e5eb85abbfc7

@sanchit-gandhi Before merging - could you update the title to reflect the change in fix applied? 

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_28486). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
Making CTC training example more general
Author: ylacombe
Pull request: yes

# What does this PR do?

#28165 introduced a new W2V2-based model that uses a different feature extractor than classic CTC-based models.
In particular, it takes mel-spectrograms as `input_features`, instead of raw waveform as `input_values`.

This runs well with the [example from the README](https://github.com/huggingface/transformers/tree/main/examples/pytorch/speech-recognition#single-gpu-ctc), as well as with the new introduced model. Happy to try some other configurations as well.

cc @patrickvonplaten and @amyeroberts 


Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_28582). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.