noamaanMulla-03/transformers-issues huggingface dataset:
Misplaced sentence in https://api-inference.huggingface.co/docs/curl/html/detailed_parameters.html#question-answering-task
Author: ToonTalk
Pull request: no

"a string to be translated in the original languages" appears in the description of the question answering documentation.

Comments:
Thank you for flagging this, @ToonTalk. I've forwarded the issue internally and I will let you know of the outcome :)

Thank you for reporting this ! @ToonTalk 

(Fixed in the docs)
Add PLBart
Author: gchhablani
Pull request: yes
Labels: name: WIP

# What does this PR do?
This PR adds PLBART.
- Paper: https://arxiv.org/abs/2103.06333
- Code and Checkpoints: https://github.com/wasiahmad/PLBART
- Authors: @wasiahmad @kaiweichang

Motivation: This encoder-decoder BART-like model allows for downstream fine-tuning on code summarization, generation, classification, etc. and is useful to the community. The fine-tuned checkpoints are also available and can be used directly.

EDIT 1:
------
I am trying to make the embeddings same for the original and my implementation. Have created an issue [here](#13481) as I'm stuck with `padding_idx` outputs not matching.

### Pending Tasks
- [ ] Fix FastTokenizer
    - [ ] Tokenizer
    - ~MultiTokenizer~ 
- [x] Fix tests
    - [x] Modeling
    - [x] Tokenizer
    - ~MultiTokenizer~
- [ ] Verify behavior for all checkpoints
- [x] Update Docs
- [ ] Update Model Cards
- [x] Add remaining checkpoints
    - [x] `plbart-large`
    - [x] `plbart-base-csnet` checkpoints

Comments:
This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md) are likely to be ignored.

The pre-trained model conversion is working correctly. Have verified with encoder/decoder outputs and embedding outputs for encoder. 
The states match except for pad tokens, for which the issue is mentioned in #13481.
The tokenizer for the pre-trained checkpoint contains 50005 tokens, with the special ones:
```
0 

1 <pad>
2 

3 <unk>
50001 [java]
50002 [python]
50003 [en_XX]
50004 <mask>
```

@gchhablani - can I help you somehow to move forward with this PR? 

Any update on this PR? @gchhablani 

@patil-suraj
Updates:
- Rebased the branch to master.
- Updated the documentation. Should I move the generation example and other model/tokenizer docs from the model/tokenization files to the `mdx`?
- Removed all the extra changes due to merging issues.

Regarding the tokenizes, I am not sure if they can be moved to a single tokenizer considering [this comment](https://github.com/huggingface/transformers/pull/13269#discussion_r790167690). Please let me know if there's a way to do that.

@patil-suraj @gchhablani - let me know when the PR is ready for a final review :-)

@patrickvonplaten Would be awesome if you could take a final look now :) 

PR is good for merge to me! Failing test is a flaky one from the Hub.

Thanks a lot @gchhablani for all your work on this! Great job!

Hi, @gchhablani @patil-suraj @sgugger 

Looks like `README.md` (and therefore `doc/source/index.mdx`) is not updated for this new added model.
Ideally, it is usually updated when a new model is added, right?
 

Thanks for flagging @ydshieh , we indeed have forgotten to ask for it during our reviews! @gchhablani would you like to add it in a follow-up PR?

@ydshieh Thanks a lot for informing! On it.

@sgugger Yes, I'll quickly fix it.
AutoModelForSequenceClassification not learning if model is initialized inside a function scope
Author: joaoaleite
Pull request: no

Hey,

I've made a simple function to initialize a bert model for sequence classification, an optimizer and a scheduler, but my model wasn't learning anything. After a lot of debugging, I found out that if I execute the same piece of code OUT of the function scope, the model learns.

```
def initialize_model(pretrained_bert_name, device, num_labels, epochs, train_dataloader):
    """Initialize the Bert Classifier, the optimizer and the learning rate scheduler.
    """
    # Instantiate Bert Classifier
    model = AutoModelForSequenceClassification.from_pretrained(
        pretrained_bert_name,
        num_labels=num_labels)

    # Sends model to GPU if inabled
    model.to(device)

    # Create the optimizer
    optimizer = torch.optim.AdamW(model.parameters(), lr=5e-5)

    # Total number of training steps
    total_steps = len(train_dataloader) * epochs

    # Set up the learning rate scheduler
    scheduler = get_scheduler(
        "linear",
        optimizer=optimizer,
        num_warmup_steps=10000,
        num_training_steps=total_steps
    )
    return model, optimizer, scheduler
```


I'm thinking that maybe sending the model to device, or maybe passing its weights to the optimizer may create a different copy when the function returns, idk. Thoughts?


Comments:
Hey @JAugusto97, can you provide a full script or a reproducible Colab notebook? Reading the code I don't see why it shouldn't work. 

It might have been global variables interfering somewhere, when I tried to reproduce in a new notebook it worked 🤔 
Thanks anyway!
Fix `HfDeepSpeedConfig` argument in `Trainer`
Author: jaketae
Pull request: yes
Labels: name: DeepSpeed

# What does this PR do?

`HfDeepSpeedConfig` accepts a dictionary or a path to `.json` file containing DeepSpeed configurations. However, `self.args` is a `TrainingArgument` object. This PR replaces `self.args` with `self.args.deepspeed`.


## Before submitting
- [ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [ ] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md#start-contributing-pull-requests),
      Pull Request section?
- [ ] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [ ] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/master/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/master/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests?


## Who can review?

@stas00 

Comments:
_The documentation is not available anymore as the PR was closed or merged._

With DS please remember the live CI doesn't run any of DS tests as it doesn't have GPUs. So they only run nightly, so it's best to run the tests locally if you have at least one gpu.
```
RUN_SLOW=1 pytest tests/deepspeed
```

I did run the tests and they all pass, so all is good. (Clearly this feature hasn't been tested, but as I said I can't think of how it could be tested in this particular situation).

Thank you for the comments and review, @stas00. It makes sense that the CI doesn't run DeepSpeed tests every time. 

I can't think of a way to test this right off the bat either, but I'll open an issue and/or a PR if I get any ideas while integrating OSLO!
fix bug in PT speech-encoder-decoder
Author: sanchit-gandhi
Pull request: yes

# What does this PR do?

Currently in the PT speech-encoder-decoder model, if:

- `encoder_outputs is None` and
- `inputs is not None`

then the encoder is _not_ run in the forward pass, and the `encoder_outputs` remain set to `None`. This throws a `TypeError` when taking the `encoder_hidden_states` - this variable is defined by indexing `encoder_outputs`, which is a `NoneType`.

This PR amends the model to handle this case by running the encoder in a forward pass to yield the `encoder_outputs`. The `encoder_outputs` can then be indexed to give the `encoder_hidden_states`.


Comments:
_The documentation is not available anymore as the PR was closed or merged._

Think you need to run `make style` once before being able to merge this PR
BEIT masked lm
Author: dar-tau
Pull request: no

How can one generate labels for BEIT masked LM? 

Comments:
Hi,

I've created a Colab notebook to illustrate `BeitForMaskedImageModeling` [here](https://github.com/NielsRogge/Transformers-Tutorials/blob/master/BEiT/Understanding_BeitForMaskedImageModeling.ipynb).

See also this issue: https://github.com/microsoft/unilm/issues/401

Thanks!

Hi @NielsRogge,

did you have any progress in reconstructing the masked visual tokens with BEiT so that they correspond to the ground truth labels? I am having the same issue.

Thanks

Hi,

I'm pretty sure my implementation is correct. You can visualize the predictions of BeitForMaskedImageModeling using DALL-E's decoder.
Add missing PLBart entry in README
Author: gchhablani
Pull request: yes

# What does this PR do?
This PR fixes the missing entry for PLBart (See #13269) in the index.

@sgugger @patil-suraj @patrickvonplaten @LysandreJik 

Comments:
_The documentation is not available anymore as the PR was closed or merged._

I don't see the change in the main README. Also it's not part of the doc, so you have to use the full link to the master doc.
Add PerceiverForTokenClassification
Author: NielsRogge
Pull request: yes

# What does this PR do?

You can also use the Perceiver to do named-entity recognition! Training for 3 epochs on CoNLL 2003 gives me the following result (using the official example script):

```
***** eval metrics *****
  epoch                   =        3.0
  eval_LOC_f1             =     0.8895
  eval_LOC_number         =      11884
  eval_LOC_precision      =     0.8934
  eval_LOC_recall         =     0.8856
  eval_MISC_f1            =     0.8058
  eval_MISC_number        =       6078
  eval_MISC_precision     =     0.8182
  eval_MISC_recall        =     0.7938
  eval_ORG_f1             =     0.8215
  eval_ORG_number         =       8869
  eval_ORG_precision      =     0.8187
  eval_ORG_recall         =     0.8243
  eval_PER_f1             =     0.8477
  eval_PER_number         =      10479
  eval_PER_precision      =     0.8598
  eval_PER_recall         =     0.8359
  eval_loss               =     0.1479
  eval_overall_accuracy   =     0.9594
  eval_overall_f1         =      0.848
  eval_overall_precision  =     0.8539
  eval_overall_recall     =     0.8421
  eval_runtime            = 0:01:25.54
  eval_samples            =       3250
  eval_samples_per_second =      37.99
  eval_steps_per_second   =      4.757
```

This PR includes the tweaked example script to make it work for the Perceiver. I can remove it if required, or move it to research-projects directory.

Fixes #14971

Comments:
cc'ing @Narsil as the token classification pipeline currently doesn't work with the Perceiver. Shoud I remove it from the Auto API?

@NielsRogge What does happen when it doesn't work ?

Is it taking text only ? then it should be fixable, no ?

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md) are likely to be ignored.
Reason for returning dequantized (fp.32) value at every layer of I-BERT.
Author: Sutapa-Dutta
Pull request: no

In I-Bert(/src/transformers/models/ibert/quant_modules.py#L114)

Kindly let me know the reason to return the dequantized(fp.32) value (quantized_input*scaling_factor) at every quantized-layer as seen below:-
`classQuantAct(nn.Module):
.
. 
return quant_act_int * correct_output_scale, self.act_scaling_factor
`class QuantLinear(nn.Module):
.
.
return (nn.functional.linear(x_int, weight=self.weight_integer, bias=self.bias_integer) * bias_scaling_factor,bias_scaling_factor,)
`
`
Why can't we simply return quant_act_int  or  nn.functional.linear(x_int, weight=self.weight_integer, bias=self.bias_integer) for the next layers![https://github.com/huggingface/transformers/blob/1a00863e95655c6914202c2dbf3b091dfb3f04c1/src/transformers/models/ibert/quant_modules.py#L114](url)

Comments:
This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md) are likely to be ignored.
How to efficiently tokenize unknown tokens in GPT2
Author: soumya-ranjan-sahoo
Pull request: no

I am trying to train a dialog system using GPT2. For tokenization, I am using the following configuration for adding the special tokens. 
```
from transformers import (
     AdamW,
     AutoConfig,
     AutoTokenizer,
     PreTrainedModel,
     PreTrainedTokenizer,
     get_linear_schedule_with_warmup,
)

SPECIAL_TOKENS = {
    "bos_token": "
",
    "eos_token": "
",
    "pad_token": "[PAD]",
    "additional_special_tokens": ["[SYS]", "[USR]", "[KG]", "[SUB]", "[PRED]", "[OBJ]", "[TRIPLE]", "[SEP]", "[Q]","[DOM]"]
}
tokenizer = AutoTokenizer.from_pretrained(args.model_name_or_path)
tokenizer.add_special_tokens(SPECIAL_TOKENS)
```
Next, when I am trying to tokenize a sequence(dialog's utterance) and later convert into ids, some of the special tokens in my sequence are getting mapped as unknown tokens, since the ids of these special tokens becomes the same as bos and eos as they all map to 
 as in the GPT2's [source code][1].

Here is a working example -

```
tokenized_sequence = ['[PRED]', 'name', '[SUB]', 'frankie_and_bennys', '[PRED]', 'address', '[SUB]', 'cambridge_leisure_park_clifton_way_cherry_hinton', '[PRED]', 'area', '[SUB]', 'south', '[PRED]', 'food', '[SUB]', 'italian', '[PRED]', 'phone', '[SUB]', '01223_412430', '[PRED]', 'pricerange', '[SUB]', 'expensive', '[PRED]', 'postcode', '[SUB]', 'cb17dy']
special_tokens = ['frankie_and_bennys','cambridge_leisure_park_clifton_way_cherry_hinton','italian','postcode', 'cb17dy']
tokens_to_ids = [50262, 3672, 50261, 50256, 50262, 21975, 50261, 50256, 50262, 20337, 50261, 35782, 50262, 19425, 50261, 50256, 50262, 4862, 50261, 50256, 50262, 50256, 50261, 22031, 50262, 50256, 50261, 50256]
ids_to_tokens = [PRED]name[SUB]
[PRED]address[SUB]
[PRED]area[SUB]south[PRED]food[SUB]
[PRED]phone[SUB]
[PRED]
[SUB]expensive[PRED]
[SUB]

```
As you can see the special_tokens are being mapped to the id  50256 (that is to |endoftext|), the model fails to see and learn these important tokens and hence generate very poor and often hallucinated responses. 

What could be a quick and efficient fix for this issue?

Note - I have a large set of such special tokens in my corpus. 


  [1]: https://github.com/huggingface/transformers/blob/v4.15.0/src/transformers/models/gpt2/tokenization_gpt2.py#L104

Comments:
Hi, @soumya-ranjan-sahoo This is a more general question, so would be awesome if you could ask this on the [forum ](https://discuss.huggingface.co/). We use issues for bug reports or feature requests. Thanks!

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md) are likely to be ignored.
Trainer model __init__() got an unexpected keyword argument 'prediction_loss_only'
Author: atptour2017
Pull request: no

## Environment info

- `transformers` version: 4.15.0
- Platform: Linux-4.18.0-193.19.1.el8_2.x86_64-x86_64-with-centos-8.2.2004-Core
- Python version: 3.6.13
- PyTorch version (GPU?): 1.10.1+cu102 (True)
- Tensorflow version (GPU?): not installed (NA)
- Flax version (CPU?/GPU?/TPU?): not installed (NA)
- Jax version: not installed
- JaxLib version: not installed
- Using GPU in script?: <fill in>
- Using distributed or parallel set-up in script?: <fill in>

### Who can help
@sgugger

## Information
Model I am using is BERT.  
The problem arises when using:
* [ ] the official example scripts: (give details below)
cd examples/legacy/
export CUDA_VISIBLE_DEVICES=0
export TRAIN_FILE=data/source
export TEST_FILE=data/source
python run_language_modeling.py \
    --output_dir=chinese_finetuned_lm \
    --model_type=bert \
    --model_name_or_path=bert-base-chinese \
    --do_train \
    --train_data_file=$TRAIN_FILE \
    --do_eval \
    --eval_data_file=$TEST_FILE \
    --mlm

The tasks I am working on is:
* [ ] Finetuning a language model based on bert-base-chinese model.

## To reproduce

Steps to reproduce the behavior:

1.prepare bert-base-chinese model.
2.prepare data/source file(A pure Chinese text).
3.run run_language_modeling.py

Traceback (most recent call last):
  File "run_language_modeling.py", line 364, in <module>
    main()
  File "run_language_modeling.py", line 318, in main
    prediction_loss_only=True,
TypeError: __init__() got an unexpected keyword argument 'prediction_loss_only'

## Expected behavior
No error.


Comments:
You are using a legacy script with a recent version of Transformers, this is not going to work. You should use a version of Transformers compatible with that script, or use one of our maintained examples.

I had the same issue and solved by using an older release: `pip install transformers==v3.4.0`
@atptour2017, just in case that works for you too.  

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md) are likely to be ignored.
Wav2Vec2 bart-large - finetuning failing with ValueError: tokenizer has to be of type....but is <class 'transformers.models.bart.tokenization_bart_fast.BartTokenizerFast'
Author: programmeddeath1
Pull request: no

Hii,

I am trying to understand how to finetune wav2vec2 models with my own decoder, I am trying to replicate the finetuning for the 
patrickvonplaten/wav2vec2-2-bart-large model with the run script from 
https://github.com/huggingface/transformers/tree/master/examples/pytorch/speech-recognition

The create_model.py as well as the config contains 'Wav2Vec2Processor' as the processor class.

But on  the line 429(run_speech_recognition_seq2seq.py) this command
processor = AutoProcessor.from_pretrained(training_args.output_dir)
fails with -
'''
AttributeError: 'str' object has no attribute 'from_pretrained'
'''
And if i pass the processor class directly it then fails with 
'''
alueError: tokenizer has to be of type <class 'type'>, but is <class 'transformers.models.bart.tokenization_bart_fast.BartTokenizerFast'
'''
I am am still debugging but unable to figure out what the issue is.

--- Update
The internal error is  it has to be of type PreTrainedTokenizer but Bart has PreTrainedTokenizerFast, due to which it fails
Thanks!

Comments:
Hey @programmeddeath1,

Sorry to answer so late. Could you please provide a short code snippet that reproduces the error? Thanks!

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md) are likely to be ignored.
Cl tohoku japanese roberta
Author: LysandreJik
Pull request: yes


Comments:
This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md) are likely to be ignored.

Hi, @LysandreJik 
I deeply appreciate your help on fixing CI.
Is there anything I can do for this?

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md) are likely to be ignored.

Sorry for taking a while to do this - I'll push an update to get it up to date so that we may merge it in the coming days.

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md) are likely to be ignored.
run_tests_pipelines_torch is not deterministic
Author: kevinpl07
Pull request: no

I am currently working on a PR and while making sure we pass all tests and checks, i ran into a pipeline test that failed in one run but succeeded afterwards with the same codebase (only a docstring has been changed). 

I think this nondeterministic test-behaviour should be avoided in an automatic build pipeline. 

Here it failed:
https://app.circleci.com/pipelines/github/huggingface/transformers/32972/workflows/ede22951-5fd9-446c-8417-09acd2caa979/jobs/347421

Here it worked:
https://app.circleci.com/pipelines/github/huggingface/transformers/33082/workflows/f297f4fa-bb81-417c-ae40-892747803400/jobs/348790

Feel free to request additional info, as I didn't know what else is needed.

Comments:
Pinging @Narsil 

Hi @kevinpl07 This PR should have fixed the issue. You test ran later, but were you rebased after this PR  https://github.com/huggingface/transformers/pull/15154 ?

I ran many tests on a local box without any issue on master just to be sure.

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md) are likely to be ignored.
BertForSequenceClassification defines criterion at every forward pass
Author: KrishPro
Pull request: no

## Environment info
I don't think, That any environmental info will be helpful in my issue

### Who can help

@LysandreJik, Can you look into the code ?

I just want anyone to have a look here https://github.com/huggingface/transformers/blob/master/src/transformers/models/bert/modeling_bert.py#L1579
`loss_fct = CrossEntropyLoss()`

Inside the forward function of `BertForSequenceClassification`, Instance of `CrossEntropyLoss` is created.
Which directly means that at every forward pass to the model we will create a instance of `CrossEntropyLoss`

Isn't this a performance bug ?

Comments:
I'm planning to post a pull-request, If someone from the official contributors confirms that this is a bug
Because I'm still not sure if this is a bug or my understanding is un-clear

Ok, I myself found a quick-fix for this

If we don't want to create `CrossEntropyLoss`'s instance at every forward step, Then we can avoid passing labels to the forward functions and then calculate the loss as usual from `logits` from the model's output