noamaanMulla-03/transformers-issues huggingface dataset:
distilled bart-large/bart-base
Author: kkissmart
Pull request: no
Labels: name: wontfix

name: New model

@sshleifer Is there a distilled bart model (not CNN/XSUM) model available?

Thanks!

Comments:
Hi @kkissmart 
Yes, distilbart models are available [here](https://huggingface.co/sshleifer/distilbart-cnn-6-6)

@patil-suraj I meant a distill Bart ( a pretrain model) not a summarization model. Do I misunderstand the name?

No, we don't have that.

Ohh, AFAIK there is no pre-trained distilbart like distilbert. 
There are two types of distillation 
1. No teacher distillation: which copies alternate layers from the pre-trained model and creates a small student model.
2. With teacher distillation: enforce that the student and teacher produce similar encoder_outputs, logits, and hidden_states

You can easily create a student (no teacher) model using the scripts [here](https://github.com/huggingface/transformers/tree/master/examples/seq2seq#no-teacher-distillation) , you'll just need to use `bart-large` instead of `bart-large-cnn`.

pre-training distilbart is still not included, however you can train the large model on the down-stream task and then do with teacher distillation for a smaller distilled model.

Also, consider asking such non-bug questions on the forum https://discuss.huggingface.co/ -:)

Is there a distilBART base model that does not have pretaining weights?

There is no distilbart-base model.
There are only distilled models fine-tuned on summarization tasks.

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.


@sshleifer @patil-suraj  
```
You can easily create a student (no teacher) model using the scripts here , you'll just need to use bart-large instead of bart-large-cnn.
https://github.com/huggingface/transformers/tree/master/examples/seq2seq#no-teacher-distillation or
https://github.com/huggingface/transformers/tree/master/examples/legacy/seq2seq#no-teacher-distillation ?
```
I want to make a DistilBART model from my japanese BART-large mode. but, No script in this link. Has the script been kept private? I want to see the script.


Hi @hisashi-ito 

The seq2seq distillation scripts are now moved under `examples/research_projects/seq2seq-distillation` directory. You can find them here.
https://github.com/huggingface/transformers/tree/master/examples/research_projects/seq2seq-distillation

Hi @patil-suraj
Thank you for teaching !!
[M2M100] fix positional embeddings
Author: patil-suraj
Pull request: yes

# What does this PR do?

The torchscript tests for `M2M100` are failing on master. This is because the `weights` in `M2M100SinusoidalPositionalEmbedding` are initially not on the same device as the rest of the parameters. 

The PR makes the `weights` as `nn.Parameter` so they'll be on the same device.

Comments:
Why HFArgumentParser.parse_dict(TrainerArguments) return tuple instead of dict?
Author: alierenak
Pull request: no

I guess it is not certainly bug however I could not almost understand why` HfArgumentParser.parsedict() `return `(*outputs,)`. As can be seen in [docs](https://huggingface.co/transformers/_modules/transformers/hf_argparser.html#HfArgumentParser
)
I am trying to fine-tune BERT for Token Classification using Trainer class and my aim is to turn argparse object to TrainingArguments. I converted  ArgumentParser object to dictionary then using HfArgumentParser.parsedict() turn it to TrainingArguments object. However I realized that ` HfArgumentParser.parsedict() ` returns `(TrainingArguments, )` tuple  which cause following error in Trainer initializer.

```console
Traceback (most recent call last):
  File "transformers_ner.py", line 344, in <module>
    finetune_model(args)
  File "transformers_ner.py", line 308, in finetune_model
    args=train_args
  File "/home/akali/.local/lib/python3.6/site-packages/transformers/trainer.py", line 237, in __init__
    set_seed(self.args.seed)
AttributeError: 'tuple' object has no attribute 'seed'

```

I get train_args by:

```python
    # ArgParser --> Training Arguments
    HFParser = HfArgumentParser(TrainingArguments)
    train_args = HFParser.parse_dict(args)
```

I know I can get TrainingArguments object by doing` train_args[0] `however, isn't it meaningless that `HfArgumentParser.parsedict()` return tuple instead of directly TrainingArguments.

Thanks.








Comments:
hi @akalieren 

the `parse_dict` or `parse_args_into_dataclasses` methods always return a `tuple` of parsed arguments for each `dataclass` that was used to initialize `HfArgumentParser`. Here you're initializing it with just `TrainingArguments`  so `parse_dict` returns a `tuple` of length 1.

Hope this helps.

Thanks for quick response @patil-suraj,

I am understanding it returns `tuple` of initializing given data classes; however, my question is that is there a reason why it returns `tuple` instead of just  `*outputs`. 

I hope this is not kind of misunderstanding, and grateful about your time.
Thanks.



`return *outputs` is not a valid python statement. Also `outputs` here is a list and it's just a good python practice to return return `tuple`  instead of `list` when returning multiple values.

I tried it and realized that you are right. Thanks a lot for your interest and kind responses.

I am closing this issue. Wish healthy days.
Can't reproduce xlm-roberta-large finetuned result on XNLI
Author: ntudy
Pull request: no

# ❓ Questions & Help

I'm trying to finetune `xlm-roberta-large` on MNLI English training data and make zero-shot classification on XNLI dataset.
However, I found that `xlm-roberta-large` is super sensitive to hyper parameters. The reported average accuracy is 80.9, while my model can only achieve 79.74, which is 1% less than the reported accuracy. 

I used Adam optimizer with 5e-6 learning rate and the batch size is 16.  Any one can suggest better hyperparameters to reproduce the XNLI result of `xlm-roberta-large`?


Comments:
Hello, thanks for opening an issue! We try to keep the github issues for bugs/feature requests.
Could you ask your question on the [forum](https://discusss.huggingface.co) instead?

Thanks!
Blenderbot-3B config seems to be a little wrong
Author: Narsil
Pull request: no

## Environment info
<!-- You can run the command `transformers-cli env` and copy-and-paste its output below.
     Don't forget to fill out the missing fields in that output! -->
     
- `transformers` version: 4.1
- Platform: Linux
- Python version: 3.8
- PyTorch version (GPU?):- 
- Tensorflow version (GPU?): - 
- Using GPU in script?: No
- Using distributed or parallel set-up in script?: No

It seems the current Config of `Blenderbot-3B` is a bit broken, (`Blenderbot-90M` and distill versions seem fine).

```python

tokenizer = AutoTokenizer.from_pretrained('facebook/blenderbot-90M')
tokenizer.decode(tokenizer.encode("Hey there"))
# 'hey there'  so working fine

tokenizer.decode(tokenizer.encode("Hey there"))
# '<unk> y <unk> e'   obvious error as the tokens as 'ĠHey' exists in the vocab. Error is possibly linked to '@@' string terminator config

----
# Other example that's probably linked but that originally triggered the issue so we need to make sure it's fixed too

nlp = pipeline('text-generation', model='blenderbot-3B')
nlp("Hey there")
# {"generated_text": "'ĠHi, Ġhow Ġare Ġyou Ġtoday? ĠI Ġjust Ġgot Ġback Ġfrom Ġa Ġwalk, Ġit Ġwas Ġnice."}

```

### Who can help
<!-- Your issue will be replied to more quickly if you can figure out the right person to tag with @
 If you know how to use git blame, that is the easiest way, otherwise, here is a rough guide of **who to tag**.
 Please tag fewer than 3 people.
-->
 
@patrickvonplaten @patil-suraj 

## Information

Model I am using (Bert, XLNet ...):

The problem arises when using:
* [x ] the official example scripts: (give details below)
* [ ] my own modified scripts: (give details below)

The tasks I am working on is:
* [ ] an official GLUE/SQUaD task: (give the name)
* [ ] my own task or dataset: (give details below)

## Expected behavior

The tokenization should be better at encoding for 3B. And the pipeline should not output garbage Ġ everywhere.

<!-- A clear and concise description of what you would expect to happen. -->


Comments:
This issue has been stale for 1 month.

Closing this, blenderbot 90M is very different in Arch as other variants, so it will receive less love (it's not that powerful compared to the others anyway).

Also a lot of work was done here : https://github.com/huggingface/transformers/pull/10002
Marian input decoding bug
Author: Mehrad0711
Pull request: no

# 🐛 Bug

## Information

Model I am using (Bert, XLNet ...): Marian

Language I am using the model on (English, Chinese ...): English, German

The problem arises when using:
* [ ] the official example scripts: (give details below)
* [x] my own modified scripts: (give details below)

The tasks I am working on is:
* [ ] an official GLUE/SQUaD task: (give the name)
* [x] my own task or dataset: (give details below)

## To reproduce

I'm examining Marian models to translate text. I noticed `convert_tokens_to_string` method uses `spm_target` which can be problematic if we want to decode source text.

Here is my script:

```
from transformers import MarianTokenizer, MarianModel

tokenizer = MarianTokenizer.from_pretrained('Helsinki-NLP/opus-mt-en-de')
model = MarianModel.from_pretrained('Helsinki-NLP/opus-mt-en-de')

input_text = "I was eating lunch when he saw me"
target_text = "Ich aß gerade zu Mittag, als er mich sah"

input_tokenized = tokenizer(input_text, return_tensors='pt')
with tokenizer.as_target_tokenizer():
    target_tokenized = tokenizer(target_text, return_tensors='pt')

print(tokenizer.decode(input_tokenized.data['input_ids'][0]))
with tokenizer.as_target_tokenizer():
    print(tokenizer.decode(target_tokenized.data['input_ids'][0]))

```

stdout:

```
I was▁eating▁lunch▁when he▁saw me
Ich aß gerade zu Mittag, als er mich sah

```

As you can see the input text is not decoded correctly since `spm_target` is used. A potential fix is to use `current_spm` and let `as_target_tokenizer` context manager decide which spm should be used (similar to text encoding):

```
def convert_tokens_to_string(self, tokens: List[str]) -> str:
    return self.current_spm.DecodePieces(tokens)
```

I can PR the fix if needed.


<!-- If you have code snippets, error messages, stack traces please provide them here as well.
     Important! Use code tags to correctly format your code. See https://help.github.com/en/github/writing-on-github/creating-and-highlighting-code-blocks#syntax-highlighting
     Do not use screenshots, as they are hard to read and (more importantly) don't allow others to copy-and-paste your code.-->


<!-- A clear and concise description of what you would expect to happen. -->

## Environment info
<!-- You can run the command `transformers-cli env` and copy-and-paste its output below.
     Don't forget to fill out the missing fields in that output! -->
     
- `transformers` version: master branch (f6e53e3c2bafb37c861db71a4b28c304403af92b)
- Python version: 3.7.4
- PyTorch version (GPU?): 1.7.1 (False)


Comments:
Hey @Mehrad0711,

Thanks a lot for the very clean & easy to understand issue!
I can reproduce the error and would be super happy about a PR to fix it! Your fix to let the context manager handle the `spm_target` sounds like the correct solution to me!

Hi @patrickvonplaten!
Thank you for your feedback. I just submitted a PR fixing this issue.
Thanks ahead for reviewing.
add shift on BartForCausalLM
Author: voidful
Pull request: yes

# What does this PR do?

Fixes  #10480


## Before submitting
- [x] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [x] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md#start-contributing-pull-requests),
      Pull Request section?
- [x] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [x] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/master/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/master/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests?


## Who can review?

Anyone in the community is free to review the PR once the tests have passed. Feel free to tag
members/contributors which may be interested in your PR.

@patil-suraj 
@patrickvonplaten 

<!-- Your PR will be replied to more quickly if you can figure out the right person to tag with @

 If you know how to use git blame, that is the easiest way, otherwise, here is a rough guide of **who to tag**.
 Please tag fewer than 3 people.

Models:

- albert, bert, xlm: @LysandreJik
- blenderbot, bart, marian, pegasus, encoderdecoder,  t5: @patrickvonplaten, @patil-suraj
- longformer, reformer, transfoxl, xlnet: @patrickvonplaten
- fsmt: @stas00
- funnel: @sgugger
- gpt2: @patrickvonplaten, @LysandreJik
- rag: @patrickvonplaten, @lhoestq
- tensorflow: @jplu

Library:

- benchmarks: @patrickvonplaten
- deepspeed: @stas00
- ray/raytune: @richardliaw, @amogkam
- text generation: @patrickvonplaten
- tokenizers: @n1t0, @LysandreJik
- trainer: @sgugger
- pipelines: @LysandreJik

Documentation: @sgugger

HF projects:

- nlp datasets: [different repo](https://github.com/huggingface/nlp)
- rust tokenizers: [different repo](https://github.com/huggingface/tokenizers)

Examples:

- maintained examples (not research project or legacy): @sgugger, @patil-suraj
- research_projects/bert-loses-patience: @JetRunner
- research_projects/distillation: @VictorSanh

 -->


Comments:
[website] installation doc blues
Author: stas00
Pull request: no

Switching docs to a different branch seems to be broken from at least one doc, e.g. going from
https://huggingface.co/transformers/master/installation.html#caching-models
upper left corner - if you want to switch to a different branch it sends you to a 404 page on all of them:

it has an issue with the base url, note how it links to: https://huggingface.co/transformers/master/master
master twice, so now it prefixes all branches with master/branch

ok narrowed it down - it happens specifically on the installation page: https://huggingface.co/transformers/master/installation.html
I tried a bunch of other pages and it seems to be ok there.

oddly I don't see anything unusual about installation.md in toctree or its content.

if you look at the snapshot - instead of showing `master` for the currently selected version it shows a chunk of the url instead.

![snapshot_8](https://user-images.githubusercontent.com/10676103/110193531-41d0ff80-7de9-11eb-9697-293daa527983.png)

@sgugger 

Comments:
[trainer] fix double wrapping + test
Author: stas00
Pull request: yes

We have an issue with
```
trainer.train()
trainer.train()
```
under any environment that requires model wrapping - we currently get the wrapping multiple times - and things may kind of work - but most of the time it breaks badly - thanks to apex for complaining noisily when it's being wrapped second time.

i.e. we get things like `DataParallel(DataParallel(model))`

This PR fixes this problem and adds a test.

@sgugger, @LysandreJik 

Comments:
offline mode for firewalled envs (part 2)
Author: stas00
Pull request: yes

In https://github.com/huggingface/transformers/pull/10407 I noticed I missed a few places where `local_files_only` should be overridden for the offline mode, so this PR completes the process.

Also rewrote the test to be more readable. Could test TF/Flax too but I don't have tiny models to run quick tests on.

@LysandreJik, @sgugger 

Comments:
[examples tests on multigpu] resolving require_torch_non_multi_gpu_but_fix_me
Author: stas00
Pull request: yes

A while ago I added `@require_torch_non_multi_gpu_but_fix_me` to quickly allow us to start running example tests on multigpu, so this PR resolves that temporary band-aid.

This PR:
* fixes a few tests to make them run on multi-gpu
* removes the decorator where it's not needed after testing that it works
* leaves the dropped legacy tests untouched - since they don't run on CI
* eliminates  `@require_torch_non_multi_gpu_but_fix_me` from existence since it's no longer needed
* the only test I couldn't figure out is https://github.com/huggingface/transformers/issues/10560 but it's not worse off than it was - added some refactoring to it and prepared it for multi-gpu if someone knows how to fix it

Note: 2 slow tests in `examples/test_examples.py` currently fail because of yet another missed thing in ported `run_seq2seq.py` - but these should be resolved by https://github.com/huggingface/transformers/pull/10551 once that one is merged, so we can merge this PR after it.

@LysandreJik 


Comments:
Check layer types for Optimizer construction
Author: sgugger
Pull request: yes

# What does this PR do?

As pointed out on the [forum](https://discuss.huggingface.co/t/parameter-groups-and-gpt2-layernorm/4239), `Trainer` currently excludes form weight decay layernorm layers by using a name pattern, which is not consistently followed by all models.

This PR actual checks the layer types and adds some tests.

Comments:
[FeatureExtractorSavingUtils] Refactor PretrainedFeatureExtractor
Author: patrickvonplaten
Pull request: yes

# What does this PR do?

This PR refactors the class `PreTrainedFeatureExtractor`. The following changes are done to move functionality that is shared between sequence and image feature extractors into a separate file. This should unblock the PRs of [DETR](https://github.com/huggingface/transformers/pull/9998), [VIT](https://github.com/huggingface/transformers/pull/10513), and [CLIP](https://github.com/huggingface/transformers/pull/10426)

- `PreTrainedFeatureExtractor` is renamed to `PreTrainedSequenceFeatureExtractor` because it implicitly assumed that the it will treat only sequential inputs (a.k.a sequence of float values or a sequence of float vectors). `PreTrainedFeatureExtractor` was too general
- All functionality that is shared between Image and Speech feature extractors (which IMO all relates to "saving" utilities) is moved to a `FeatureExtractorSavingUtilsMixin`
- `BatchFeature` is moved from the `feature_extraction_sequence_utils.py` to `feature_extraction_common_utils.py` to be used by the `PreTrainedImageFeatureExtractor` class as well
- The tests are refactored accordingly

The following things were assumed before applying the changes.

- In the mid-term future there will only be three modalities in HF: text, sequential features (value sequence, vector sequence), image features (2D non-sequential array)
- Models, such as ViT, DETR & CLIP will call their "preprocessor" `VITFeatureExtractor`, .... IMO, feature extractor is also a fitting name for image recognition (see: https://en.wikipedia.org/wiki/Feature_extraction) so that it is assumed that for image-text or image-only models there will be a `PreTrainedImageFeatureExtractor`, a `VITFeatureExtractor`, (and maybe a VITTokenizer & VITProcessor as well, but not necessary). For vision-text models that do require both a tokenizer and a feature extractor such as CLIP it is assumed that the classes `CLIPFeatureExtractor` and `CLIPTokenizer` are wrapped into a `CLIPProcessor` class similar to `Wav2Vec2Processor`. I think this is the most important assumption that is taken here, so we should make sure we are on the same page here @LysandreJik @sgugger @patil-suraj @NielsRogge 
- Image - Text or Image - only models won't require a `BatchImageFeature` or `BatchImage`, but can just use `BatchFeature`. From looking at the code in @NielsRogge's PR here: https://github.com/huggingface/transformers/pull/10513 this seems to be the case.

# Backwards compatibility:

The class `PreTrainedFeatureExtractor` was accessible via:

```python
from transformers import PreTrainedFeatureExtractor
```

but is now replaced by `PreTrainedSequenceFeatureExtractor`. However, since `PreTrainedFeatureExtractor` so far was only available on master, this change is OK IMO.

Comments:
Issue in fine-tuning distilbert on Squad 1.0
Author: pragnakalpdev
Pull request: no
Labels: name: wontfix

When I tried to finetune the distillbert (using run_squad.py in examples folder), the model reaches the F1 score of 17.43 on dev set but you have mentioned that the F1 score is 86.2. Can you help me with what I am doing wrong at the time of fine-tuning?

Below is the command that I am using
python ./examples/run_squad.py \
    --model_type bert \
    --model_name_or_path /root/distilbert_training \
    --do_train \
    --do_eval \
    --do_lower_case \
    --train_file train-v1.1.json \
    --predict_file dev-v1.1.json \
    --learning_rate 3e-5 \
    --num_train_epochs 2 \
    --max_seq_length 384 \
    --doc_stride 128 \
    --output_dir /root/output/ \
    --per_gpu_eval_batch_size=3 \
    --per_gpu_train_batch_size=3

Comments:
Hello @pragnakalpdev,
Did you change the `run_squad.py` file to include distilbert ([here](https://github.com/huggingface/pytorch-transformers/blob/master/examples/run_squad.py#L58) for instance)?
Can you check the warnings ``Weights from XXX not used in YYY``?

Hello @VictorSanh,
Thank You for your response. I got the F1 score 86.4.
But now when I am facing issue in the evaluation process.
I had performed the evaluation on my custom file that has a single paragraph and 5 questions, and its taking approx 10 seconds to generate the prediction.json file. 
I want the inference time about 1-2seconds, what can I do for that?

Hello @pragnakalpdev7,
Good!

How long is your paragraph? Is the inference time during the training/fine-tuning the same as during test? (a forward pass during training should be slightly slower).
If you're already using `run_squad.py`, there is no easy/direct way to accelerate the inference.

Hello @VictorSanh ,
My paragraph is less than 1000 characters, and yes i am already using the run_squad.py for inference.
and i didn't understand this - "Is the inference time during the training/fine-tuning the same as during test? (a forward pass during training should be slightly slower)."

For fine-tuning it took 4 hours on 1 GPU, and now i am using the fined-tuned model for inference which is taking 8 - 9 seconds.

I mean that if a forward pass already takes ~2 sec during training on your machine, it is not likely to go down to 0.1 sec during test. The reason why a forward pass is on average slightly faster during test is that the flag `torch.no_grad` deactivate the autograd engine.

Hi @pragnakalpdev , can I ask you how you solved the first problem to achieve good performance? I'm in a similar situation and any hints would help.

Hi @pragnakalpdev  could you indeed please comment how you solved the first problem (bad F1) to achieve good performance?

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.


Hi, can anyone tell me where(exactly in which directory or in colab exactly where) to run that code to fine-tune the bert model? I can't able to fine-tune my bert model for QnA purposes. Please reply.
Speedup tf tests
Author: LysandreJik
Pull request: yes

Fyi @sgugger @patrickvonplaten @stas00, I'm temporarily marking these tests as slow as they take more than 1h30 minutes and prevent the CI from completing, therefore preventing any relevant information from the TF tests.

I'm working on improving the CI times so this is temporary (will revert by Friday). 

Comments:
This single change made the tests on the TF CI pass from 6+ hours (non slow) to 29/31 minutes: https://github.com/huggingface/transformers/actions/runs/635711849

Will look for a way to reduce their time.