noamaanMulla-03/transformers-issues huggingface dataset:
Pretrain BART MNLI model on Financial Phrasebank
Author: bartmnli
Pull request: no

Hi, I am trying to train/finetune the BART large model pretrained on MNLI on Financial Phrasebank but completely lost as I'm just a beginner.

from transformers import AutoModelForSequenceClassification, AutoTokenizer
model = AutoModelForSequenceClassification.from_pretrained('facebook/bart-large-mnli')
tokenizer = AutoTokenizer.from_pretrained('facebook/bart-large-mnli')

I couldnt find any code examples for tokenizing the input text from Financial Phrasebank. Different tutorials show different ways and I'm completely full. 

Can anyone please please share any links to examples similar to this?

I was also trying to look for the finetuning code of the BART large MNLI finetuned on yahoo datset by Joe Davison @joeddav  (https://huggingface.co/joeddav/bart-large-mnli-yahoo-answers) but couldn't find that code. Any suggestions or advice would be much appreciated.

Thanks in advance.  

Comments:
Hi,

We like to keep Github issues for bugs/feature requests. For training-related questions, please use the [forum](https://discuss.huggingface.co/). Many HuggingFace members are happy to help you there! 

Thanks!

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md) are likely to be ignored.
Warn for unexpected argument combinations
Author: shirayu
Pull request: yes

# What does this PR do?


Warn for unexpected argument combinations:

- `padding is True` and ``max_length is not None``
- `padding is True` and ``pad_to_max_length is not False``

## Before submitting
- [ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [x] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md#start-contributing-pull-requests),
      Pull Request section?
- [x] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
    - https://discuss.huggingface.co/t/migration-guide-from-v2-x-to-v3-x-for-the-tokenizer-api/55
- [ ] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/master/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/master/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests?


## Who can review?

Anyone in the community is free to review the PR once the tests have passed. Feel free to tag
members/contributors who may be interested in your PR.

<!-- Your PR will be replied to more quickly if you can figure out the right person to tag with @

 If you know how to use git blame, that is the easiest way, otherwise, here is a rough guide of **who to tag**.
 Please tag fewer than 3 people.

Models:

- albert, bert, xlm: @LysandreJik
- blenderbot, bart, marian, pegasus, encoderdecoder,  t5: @patrickvonplaten, @patil-suraj
- longformer, reformer, transfoxl, xlnet: @patrickvonplaten
- fsmt: @stas00
- funnel: @sgugger
- gpt2: @patrickvonplaten, @LysandreJik
- rag: @patrickvonplaten, @lhoestq
- tensorflow: @LysandreJik

Library:

- benchmarks: @patrickvonplaten
- deepspeed: @stas00
- ray/raytune: @richardliaw, @amogkam
- text generation: @patrickvonplaten
- tokenizers: @n1t0, @LysandreJik
- trainer: @sgugger
- pipelines: @LysandreJik

Documentation: @sgugger

HF projects:

- datasets: [different repo](https://github.com/huggingface/datasets)
- rust tokenizers: [different repo](https://github.com/huggingface/tokenizers)

Examples:

- maintained examples (not research project or legacy): @sgugger, @patil-suraj
- research_projects/bert-loses-patience: @JetRunner
- research_projects/distillation: @VictorSanh

 -->


Comments:
@sgugger 
Could you review this PR?
Thank you.
How can i Use transformers to classify a obfuscated text (multinomial classification)
Author: santhoshcameo
Pull request: no
Labels: name: New model



## Model description

I have a train data which contains many obfuscated sentences associated with its labels, so how can I use Transformers to classify them ?
<!-- Important information -->

one sample raw is given here. Please help.

`twypmviwskuhpmuluhlrvimvvilekruluhqgskmvenqvuhletwululenlpuhtwamuluhijohultwsauhtwiwskskmvleuhtwamuluhsktwqvqvtwkrlruhkrpmsauhtwmkenlpkguhijraxeiwtwqvsaezuhucleeneztwleuhpmuluhlrvimvpmlruhqvendfuhiguhulenamdfuhulqvkrbruhpktwqvlekrpmypuhxepmuhqgtwqvlekrpmypuhxeyvkguhqgqvtwsatwuhqvulmvuhlrvimvvitwgzpmuhulkrpmamulmvdfuhqgskmvenqvuhskvienuhqgsaiwulvitwmvulengzezmvuhskentwamuhqvulmvuhucpmpmamqvuhtwqvkrpmezlepmmcuhtwamgu`

note: as you see the single document contains no spaces and no words etc. this is labeled as 1, and like this many other lines which are respectively labeled from 1 to 10.

Please advice.

Comments:
Hello, thanks for opening an issue! We try to keep the github issues for bugs/feature requests.
Could you ask your question on the [forum](https://discuss.huggingface.co) instead?

Thanks!
Experiment: ROUGE impact of using pegasus length-penalty implementation
Author: sshleifer
Pull request: no
Labels: name: Help wanted

name: wontfix

code is under `length_normalization` in the pegasus [tf repo](https://github.com/google-research/pegasus)

Comments:
This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.


I tried this a bit and did not get any improvements. Meanwhile, our beam search is getting similar scores to pegasus in #6844 , so I am less motivated to push further.
Branch with maybe correct beam search implem: https://github.com/sshleifer/transformers_fork/tree/peg-beam

> 
> 
> I tried this a bit and did not get any improvements. Meanwhile, our beam search is getting similar scores to pegasus in #6844 , so I am less motivated to push further.
> Branch with maybe correct beam search implem: https://github.com/sshleifer/transformers_fork/tree/peg-beam

If it is better how can we install this? Can we do this with pip? Thanks.
When training QA models, albert-xxlarge-v2 uses much more GPU mem than Bert-large
Author: fatmelon
Pull request: no
Labels: name: wontfix

## ❓ Questions & Help

<!-- A clear and concise description of the question. -->
Hi,  
         When I used run_square.py to train QA model, I found that albert-xxlarge-v2 uses much more GPU mem than Bert-large. Specifically, when using Bert large, I can set `Max_sequence_ength = 512`, `bash_size = 12`. But when I use albert-xlarge-v2, I can only set `Max_sequence_length` = 512, `bash_size = 6`.  
In fact, the number of parameters of Albert XXLarge is much less than that of Bert large, and the size of model file is the same. Why does Albert-xxlarge occupy more GPU memory when training QA model? Is it caused by more head parameters?

Comments:
![image](https://user-images.githubusercontent.com/7105813/70107805-8a4bbb80-1682-11ea-8b0f-14f30465ce0e.png)

![image](https://user-images.githubusercontent.com/7105813/70107779-786a1880-1682-11ea-9668-26de2d606d53.png)

The parameters of Albert XXLarge is much less than that of Bert large, because albert used shared parameters in all transformer layers. But it does not reduce computation，Albert xlarge is 1.5 times lower than bert large and Albert xxlarge is 3 times

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.


ALBERT repeats the same parameters for each layer but increases each layer size, so even though it has fewer parameters than BERT, the memory needs are greater due to the much larger activations in each layer.
Fix FNet reference to tpu short seq length
Author: gchhablani
Pull request: yes

# What does this PR do?

This PR fixes #13684. 

Should there also be a test for this particular case (TPU usage, less than 4096 seq length) ? 

@LysandreJik @patrickvonplaten

Comments:
@gchhablani are you btw. working on a flax integration of FNet :thinking: -> I've seen an attempt in this PR #12454 but it was closed. Would be really interesting to train FNet on TPU - GPU is working for me, but I only have one available...
For Hugging Face transformer's hidden_states output, is the first hidden state tensor that is returned the out of the embeddings?
Author: h56cho
Pull request: no

According to the Hugging Face Transformer documentation for the GPT2DoubleHeadsModel (under the 'output' section)

```
hidden_states: (optional, returned when config.output_hidden_states=True)
list of torch.FloatTensor (one for the output of each layer + the output of the embeddings) 
```

So in this case, would the first hidden_states tensor (index of 0) that is returned be the output of the embeddings, or would the very last hidden_states tensor that is returned be the output of the embeddings?

I am confused about the order in which the hidden_states tensors are returned, because the documentation seem to indicate that the output of the embeddings is the last hidden_state tensor that is returned.

Thank you,


Comments:
Indeed, the documentation might be misleading in that regard. The first value is the embedding output, every following value is the result of the preceding value being passed through an additional layer. I'll update the documentation shortly.

@LysandreJik So will output.hidden_states[-1] be the output of the last hidden layer (right before LM head)?
how to finetune based huggingface: run_glue.py
Author: lbda1
Pull request: no

python transformers-master/examples/pytorch/text-classification/run_glue.py \
  --model_name_or_path chinese_bert-base \
  --train_file=transformers-master/dataset/class/train.csv \
  --validation_file=transformers-master/dataset/class/dev.csv \
  --do_train \
  --do_eval \
  --max_seq_length 128 \
  --per_device_train_batch_size 32 \
  --learning_rate 2e-5 \
  --num_train_epochs 3 \
  --output_dir output

When I use run_glue.py, and train.csv/dev.csv data format has two columns which contain “sentence” and “label”.
But It reports an error "pandas.errors. ParserError: Error tokenizing data. C error: Expected 1 fields in line 10, saw 2".
What is reason of this error?  It is the data format error?



Comments:
Can you provide a Colab notebook to reproduce your issue?

what is the format of the classification data which used in run_glue.py?

The [README](https://github.com/huggingface/transformers/tree/master/examples/pytorch/text-classification) mentions that "your own data in a csv or a JSON file (the script might need some tweaks in that case, refer to the comments inside for help)". Looking at the comments, it says:

    # Get the datasets: you can either provide your own CSV/JSON training and evaluation files (see below)
    # or specify a GLUE benchmark task (the dataset will be downloaded automatically from the datasets Hub).
    #
    # For CSV/JSON files, this script will use as labels the column called 'label' and as pair of sentences the
    # sentences in columns called 'sentence1' and 'sentence2' if such column exists or the first two columns not named
    # label if at least two columns are provided.
    #
    # If the CSVs/JSONs contain only one non-label column, the script does single sentence classification on this
    # single column. You can easily tweak this behavior (see below)

A bit further down, the dataset is read as follows:
```
from datasets import load_dataset

data_files = {"train": data_args.train_file, "validation": data_args.validation_file}
raw_datasets = load_dataset("csv", data_files=data_files, cache_dir=model_args.cache_dir)
```

You can perhaps isolate the error by running the code above.

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md) are likely to be ignored.
Performance issues in the program
Author: DLPerf
Pull request: no

Hello,I found a performance issue in the definition of `convert_dataset_for_tensorflow` ,
examples/tensorflow/text-classification/run_glue.py,
[tf_dataset = tf_dataset.batch(batch_size=batch_size, drop_remainder=drop_remainder).map](https://github.com/huggingface/transformers/blob/439a43b6b403205eeda2d62645fc16c93627d30d/examples/tensorflow/text-classification/run_glue.py#L83) was called without **num_parallel_calls**.
I think it will increase the efficiency of your program if you add this.

The same issues also exist in [tf_dataset.batch(batch_size=batch_size, drop_remainder=drop_remainder).map](https://github.com/huggingface/transformers/blob/439a43b6b403205eeda2d62645fc16c93627d30d/examples/tensorflow/text-classification/run_text_classification.py#L98) ,
[.map(densify_ragged_batch)](https://github.com/huggingface/transformers/blob/439a43b6b403205eeda2d62645fc16c93627d30d/examples/tensorflow/multiple-choice/run_swag.py#L109) and [tf_dataset.batch(batch_size=batch_size, drop_remainder=drop_remainder).map](https://github.com/huggingface/transformers/blob/439a43b6b403205eeda2d62645fc16c93627d30d/examples/tensorflow/question-answering/run_qa.py#L253)

Here is [the documemtation of tensorflow](https://tensorflow.google.cn/api_docs/python/tf/data/Dataset?hl=en#map) to support this thing.

Looking forward to your reply. Btw, I am very glad to create a PR to fix it if you are too busy.

Comments:
Hi @DLPerf, thanks for the warning! We're actually in the process of refactoring our Datasets to automatically support conversion to TF Datasets, at which point I'll be removing this part of our examples and replacing it with a call to the conversion method.

However, if you have any insights for how we can improve the performance of our conversion methods there, that would be very helpful! You can review the code at https://github.com/huggingface/datasets/pull/2731 and leave suggestions as comments on that PR.

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md) are likely to be ignored.
🐛: skip_special_tokens in tokenization_utils.py
Author: zhangfanTJU
Pull request: yes

# What does this PR do?
🐛: skip_special_tokens in tokenization_utils.py

The skip_special_tokens in tokenization_utils.py does not work, because token never in self.all_special_ids, which shoule be fixed to self.all_special_tokens.

Many thakns! @n1t0 @LysandreJik .

Comments:
I think this is redundant, because the special ids have already been skipped in `filtered_tokens`.

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md) are likely to be ignored.
Value error while running run_glue.py example with gpt2 
Author: bpraveenk
Pull request: no

## Environment info
<!-- You can run the command `transformers-cli env` and copy-and-paste its output below.
     Don't forget to fill out the missing fields in that output! -->

- `transformers` version: 4.7.0
- Platform: Linux
- Python version: 3.8.1
- PyTorch version (GPU?):
- Tensorflow version (GPU?): 2.3 , GPU : yes
- Using GPU in script?: yes
- Using distributed or parallel set-up in script?:no

Tagging people:
@patrickvonplaten, @LysandreJik, @sgugger, @patil-suraj

## Information
Model I am using (Bert, XLNet ...): GPT2

The problem arises when using:
* [ ] the official example scripts: tensorflow/run_glue.py

The tasks I am working on is:
* [ ] an official GLUE/SQUaD task: (give the name) GLUE (MNLI/SST2)

## Error
 ValueError: Dimension size must be evenly divisible by 192 but is 8 for '{{node sparse_categorical_crossentropy_2/Reshape_2}} = Reshape[T=DT_FLOAT, Tshape=DT_INT32](sparse_categorical_crossentropy_2/SparseSoftmaxCrossEntropyWithLogits/SparseSoftmaxCrossEntropyWithLogits, sparse_categorical_crossentropy_2/strided_slice_1)' with input shapes: [8], [4] and with input tensors computed as partial shapes: input[1] = [2,8,12,?].

## To reproduce
python run_glue.py --model_name_or_path gpt2 --task_name mnli --do_train --do_eval --do_predict --output_dir ./output

## Expected behavior
Successfully complete training


Comments:
Hello! Is this the exact command you're using? I tried to reproduce but I'm getting an error with the pad token which is not defined in the GPT-2 tokenizer. Did you tweak your GPT-2 tokenizer in order to add a padding token?

That's the exact command I am running. The only change I did (see below) was  to comment out clipnorm, to fix the error "ValueError: Gradient clipping in the optimizer (by setting clipnorm or clipvalue) is currently unsupported when using a distribution strategy."  

-            clipnorm=training_args.max_grad_norm,
+            #clipnorm=training_args.max_grad_norm,

@LysandreJik  any luck reproducing the error?

When I set the dataset_mode to constant_batch, I see the following error. Any idea why the logits output dimension is (batch_size, sequence_length, num_labels) and not (batch_size, num_labels) ?

    ValueError: Shape mismatch: The shape of labels (received (8, 1)) should equal the shape of logits except for the last dimension (received (8, 128, 3)).

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md) are likely to be ignored.
__version__ attribute missing in mode config for sentence-transformers/paraphrase-mpnet-base-v2
Author: pratikchhapolika
Pull request: no

I've manually downloaded a model `paraphrase-mpnet-base-v2`, and it appears that the `SentenceTransformer.py` code above is requesting a field` __version__ `in the model config that doesn't appear to be there.

I have read the same topic : https://github.com/UKPLab/sentence-transformers/issues/184 but it doesn't solve the issue.

In the link below:

https://huggingface.co/sentence-transformers/paraphrase-mpnet-base-v2

**Code given in the link:**

```
from sentence_transformers import SentenceTransformer
sentences = ["This is an example sentence", "Each sentence is converted"]

model = SentenceTransformer('sentence-transformers/paraphrase-mpnet-base-v2')
embeddings = model.encode(sentences)
print(embeddings)

```

**I guess the above code will not work as mentioned in the link. Please correct this if it's an ongoing issue.**

For Sentence Transformer, I guess we need following files:

```
The folder should consist these files:
0_Transformer/
1_Pooling/
config.json
modules.json
```

But when we download the model `paraphrase-mpnet-base-v2 ` and unzip it. It doesn't have `0_Transformer` in it?

Any Suggestions?



Comments:
Maybe of interest to @nreimers 

Hi @pratikchhapolika 
The above code works well with the most recent sentence-transformers version v1 (v1.2.1) or (better) v2 (>= 2.0.0). 

With old sentence-transformers versions 1 the model does not work, as the folder structure has changed to make it compatible with the hub.

A folder 0_Transformer is not required and was removed in v2, so that models can be also loaded with HF transformers from the hub.

Just update to v1.2.1 or v2.0.0 and everything works.

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md) are likely to be ignored.
FNet model card
Author: stefan-it
Pull request: no
Labels: name: model card

Hi guys,

I think that the current FNet model cards both for [fnet-base](https://huggingface.co/google/fnet-base) and [fnet-large](https://huggingface.co/google/fnet-large) are a bit misleading:

```text
This model is uncased: it does not make a difference between english and English.
```

After manually inspecting the original sentencepiece vocab from:

```
https://storage.googleapis.com/gresearch/f_net/vocab/c4_bpe_sentencepiece.model
```
and the version on model hub, it seems that the vocab is cased instead:

![image](https://user-images.githubusercontent.com/20651387/134800605-10536560-5a64-4530-92e1-433de1be50ac.png)

And the tokenizer itself also produces a cased output:

![image](https://user-images.githubusercontent.com/20651387/134800649-adc1e635-79a0-414b-a676-6b9b98bf1cfa.png)

So I think the model card should be updated then :hugs: 


Comments:
/cc @gchhablani

@stefan-it I made a mistake there. Sorry, I'll fix it.

EDIT: I have fixed it.

Many thanks for the fix 🤗

Thanks @stefan-it for pointing it out :)
[megatron gpt checkpoint conversion] causal mask requires pos_embed dimension
Author: stas00
Pull request: yes

this is a follow up to https://github.com/huggingface/transformers/pull/13508 - where I tried to fix the wrong side of the bug :(, this one hopefully is the correct one.

causal mask uses the positional emb dimensions / `seqlen` and not `n_emb` (hidden_size) as it was originally coded and happened to work because the original meg-gpt2 model had the same `n_emb` and `seqlen` size.

I re-tested that the original `megatron_lm_345m/release/mp_rank_00/model_optim_rng.pt` still produces the same converted output.

@sgugger, @LysandreJik 

Comments:
[Tests] Cast Hubert model tests to fp16
Author: anton-l
Pull request: yes

**Problem**
Some Hubert integration tests are flaky, throwing GPU OOM on model loads (GPUs are 16GB). CI logs: https://github.com/huggingface/transformers/actions/runs/1274012618
```
FAILED tests/test_modeling_hubert.py::HubertModelIntegrationTest::test_inference_ctc_batched
FAILED tests/test_modeling_hubert.py::HubertModelIntegrationTest::test_inference_emotion_recognition
FAILED tests/test_modeling_hubert.py::HubertModelIntegrationTest::test_inference_intent_classification
FAILED tests/test_modeling_hubert.py::HubertModelIntegrationTest::test_inference_keyword_spotting
FAILED tests/test_modeling_hubert.py::HubertModelIntegrationTest::test_inference_speaker_identification
```

**Solution**
Cast models and reference tensors to fp16.

cc @patrickvonplaten 

Comments:
Enforce eval and save strategies are compatible when --load_best_model_at_end
Author: sgugger
Pull request: yes

# What does this PR do?

There are multiple reports form users not understanding their `save_strategy` or `save_steps` are overridden by `eval_strategy` and `eval_steps`, so this PR changes the current behavior to raise an error when:
- there is a mismatch between the `save_strategy` and the `eval_strategy`
- in case that common strategy is "steps", `save_steps` is not a round multiple of `eval_steps`.

This way each save happens at the same time as an evaluation and `--load_best_model_at_end` can proceed properly.

Supersedes #12687

Comments:
What if we only want to save the best checkpoint during training? Saving a checkpoint every time we do an evaluation will take much space and many checkpoints are unnecessary.

I see! This should be implemented by setting save_total_limit==1, but it would be great if there's something like a saving strategy called "save_best".