noamaanMulla-03/transformers-issues huggingface dataset:
Benchmark: traceback does not describe real problem
Author: sshleifer
Pull request: no
Labels: name: wontfix

To reproduce:
- execute a command that results in 'N/A'
- long traceback describes tuple unpacking. The real problem is what caused the N/A and is far above in a small logger.error statement.

Example Command:
```bash
python examples/benchmarking/run_benchmark.py --models facebook/mbart-large-en-ro
```

Traceback:
```python
1 / 1
ERROR:transformers.benchmark.benchmark_utils:<class 'transformers.configuration_bart.MBartConfig'>
<class 'transformers.configuration_bart.MBartConfig'>
Traceback (most recent call last):
  File "examples/benchmarking/run_benchmark.py", line 29, in <module>
    main()
  File "examples/benchmarking/run_benchmark.py", line 25, in main
    benchmark.run()
  File "/home/shleifer/transformers_fork/src/transformers/benchmark/benchmark_utils.py", line 665, in run
    memory, inference_summary = self.inference_memory(model_name, batch_size, sequence_length)
ValueError: too many values to unpack (expected 2)
```

This may not be worth a fix, and probably won't be fixed soon, but posting for others who might run into this or have ideas for a fix.

Comments:
Working command after MbartConfig change:
```bash
export d=mbart_benchmark_data
python examples/benchmarking/run_benchmark.py \
  --models facebook/mbart-large-en-ro \
  --log_filename $d/log.txt \
  --inference_memory_csv \ 
  $d/inference_memory.csv \ 
  --train_memory_csv $d/train_memory.csv \
  --train_time_csv $d/train_time.csv \
  --inference_time_csv $d/inference_time.csv \
  --fp16 --log_print --training  --save_to_csv \
  --batch_sizes 4 8 12 16
```




This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
MT: automate/experiment with pruning embeddings
Author: sshleifer
Pull request: no
Labels: name: wontfix

This isn't really a github issue, but follow pytorch/fairseq#2120 and see if it works. If it does, make it easy to use!

Comments:
This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
Smaller output vocabulary for GPT-2
Author: iRove108
Pull request: no
Labels: name: wontfix

# ❓ Questions & Help
I noticed that by default, GPT2LMHeadModel returns prediction scores of shape (batch_size, sequence_length, config.vocab_size) ([docs link](https://huggingface.co/transformers/model_doc/gpt2.html#gpt2lmheadmodel)). Is there any way for me to limit the output vocabulary to only a subset of words?

I want to take the existing weights from GPT-2, but re-train a new top linear layer with a smaller vocabulary. I suppose I could mask the logits at the end, but then it feels like a waste of computational power to even predict them.

**A link to [original question on the forum](https://discuss.huggingface.co/t/smaller-output-vocabulary-for-gpt-2/366)**

Comments:
This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
Silenced error while downloading pretrained model
Author: festeh
Pull request: no
Labels: name: wontfix

# 🐛 Bug

Currently, if user failed to download a model, they will receive uninformative and confusing message (see #5787). This is most hurting for users behind a proxy. This happens because of this [line](https://github.com/huggingface/transformers/blob/0533cf470659b97c6279bd04f65536a1ec88404a/src/transformers/file_utils.py#L681). My opinion is that there should be at least a warning, and if `force_download` is `True`, then error should be raised. I can make a PR if you agree with that.


Comments:
This sounds reasonable

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
Word frequencies in TransfoXLTokenizer
Author: GregoireMialon
Pull request: no
Labels: name: wontfix

I was wondering if it is still possible to access word frequencies through the populated counter of TransfoXLTokenizer? For example, `tokenizer = TransfoXLTokenizer.from_pretrained('transfo-xl-wt103')` seems to have an empty counter. This is referring to:

"So p_M(S) is just the output of the model right?

For p_u(S), I think the easiest is probably to use the empirical probabilities.
`TransfoXLTokenizer` has a counter to store words frequencies [here](https://github.com/huggingface/pytorch-pretrained-BERT/blob/master/pytorch_pretrained_bert/tokenization_transfo_xl.py#L98) which should be populated in the "pretrained" tokenizer so I would use and normalize this to get unconditional probabilities for each word and then compute SLOR."

_Originally posted by @thomwolf in https://github.com/huggingface/transformers/issues/477#issuecomment-483973033_

Comments:
This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
How to finetune distillbart from distilbart-cnn-12-6 checkpoint using cnn_daily mail or gigawords dataset?
Author: Hildweig
Pull request: no
Labels: name: wontfix

I would like to ask about how to finetune distillbart on gigaword and cnn dailymail with the starting checkpoint distilbart-cnn-12-6.
I did use the gigaword dataset provided by tensorflow but it replaces numbers by this character: "#", as a result, my summaries have # instead of numbers, is it normal that it has those # ?
Also is it really possible to finetune distillbart from the checkpoint distilbart-cnn-12-6 with cnn daily mail? 


Comments:
This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
Save models after each epoch
Author: GCHQResearcher92457
Pull request: no
Labels: name: wontfix

name: Examples

# 🚀 Feature request
In the various training scripts in `examples`, would it be better to checkpoint the model at the end of each epoch, as well as every `save_steps` iterations as specified by the user?

## Motivation
I suppose for language modelling, saving the model after each epoch is not as important, but for anything supervised (and some other applications)  it seems  natural to want checkpoints after each epoch. There are plenty of examples in the literature where one wants to inspect models when they have seen each training example some specific number of times. I have been doing some experiments recently where this was the case (with `run_languagemodeling.py` actually) and found myself having to manually enter the number of iterations per epoch as `save_steps to get the desired checkpoints.

## Your contribution
Would be a simple change to the various training scripts. Happy to do a PR.


Comments:
This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.


I also need this. Did you figure out a better way of doing it other than counting the number of checkpoints required for an epoch manually?

> I also need this. Did you figure out a better way of doing it other than counting the number of checkpoints required for an epoch manually?

Without changing the code, no I don't think there's an alternative to counting manually. Looks like it would be a simple matter of repeating [this line](https://github.com/huggingface/transformers/blob/0866669e751bef636fa693b704a28c1fea9a17f3/src/transformers/trainer.py#L521) a few lines further down at the end of the epoch loop.

not sure what you mean by manually counting..but, if you just add this line before the start of each epoch(i.e [here](https://github.com/huggingface/transformers/blob/0866669e751bef636fa693b704a28c1fea9a17f3/src/transformers/trainer.py#L463)), you can make the model save after each epoch. The len(epoch_iterator) is the number of batches in an epoch. 

`self.args.save_steps = len(epoch_iterator)`

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
problem about Custom class inheriting from <TFBertPreTrainedModel>
Author: jianrui2025
Pull request: no
Labels: name: wontfix

# ❓ Questions & Help


## Details
<!-- Description of your issue -->
I constructed class inheriting from TFBertPreTrainedModel. as follows:
```
class Model(transformers.TFBertPreTrainedModel):
    def __init__(self,config, *inputs, **kwargs):
        super(Model,self).__init__(config, *inputs, **kwargs)
        self.bert = transformers.TFBertMainLayer(config,name="bert")
       

    @tf.function
    def call(self, inputs, training=None, mask=None,**kwargs):
        out = self.bert(inputs)
        print(out)
        return out
```
main progress as follow:
```
if __name__ == "__main__":
    tokenizer = transformers.BertTokenizer("model/chinese_L-12_H-768_A-12/vocab.txt")
    text_2 = tokenizer.batch_encode_plus(["你买啊，买了就是成都人", "你来啊，来了就是深圳人"], max_length=20, pad_to_max_length=True)
    print(text_2)
    model = Model.from_pretrained("bert-base-chinese")
    out = model([tf.convert_to_tensor(text_2["input_ids"]),tf.convert_to_tensor(text_2['attention_mask'])])
```

unfortunately，the print on console was different for what I expect when I run this program. I expect that the out of **print(out)** is a tensor with size [2,20,768], but in fact, the print was run three times, outputs is **(<tf.Tensor 'bert/encoder/layer_._11/output/LayerNorm/batchnorm/add_1:0' shape=(3, 5, 768) dtype=float32>, <tf.Tensor 'bert/pooler/dense/Tanh:0' shape=(3, 768) dtype=float32>)** , **(<tf.Tensor 'bert/encoder/layer_._11/output/LayerNorm/batchnorm/add_1:0' shape=(3, 5, 768) dtype=float32>, <tf.Tensor 'bert/pooler/dense/Tanh:0' shape=(3, 768) dtype=float32>)** and **(<tf.Tensor 'bert/encoder/layer_._11/output/LayerNorm/batchnorm/add_1:0' shape=(2, 20, 768) dtype=float32>, <tf.Tensor 'bert/pooler/dense/Tanh:0' shape=(2, 768) dtype=float32>)**
My question are 1) why 2)how could I get the last outputs for following layers
I am looking forward your help, thanks
<!-- You should first ask your question on the forum or SO, and only if
     you didn't get an answer ask it here on GitHub. -->
**A link to original question on Stack Overflow**:

Comments:
We are trying to move these kinds of questions (which are not obvious bugs, but question for customized use cases) more and more to https://discuss.huggingface.co/. Would you mind posting it there again with your sample code:
```python
class Model(transformers.TFBertPreTrainedModel):
    def __init__(self,config, *inputs, **kwargs):
        super(Model,self).__init__(config, *inputs, **kwargs)
        self.bert = transformers.TFBertMainLayer(config,name="bert")
       

    @tf.function
    def call(self, inputs, training=None, mask=None,**kwargs):
        out = self.bert(inputs)
        print(out)
        return out
```

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
[BartModel] Question for BartModel Output Shape when I pass the 'decoder_input_ids'
Author: Bannng
Pull request: no
Labels: name: wontfix

# ❓ Questions & Help

<!-- The GitHub issue tracker is primarly intended for bugs, feature requests,
     new models and benchmarks, and migration questions. For all other questions,
     we direct you to the Hugging Face forum: https://discuss.huggingface.co/ .
     You can also try Stack Overflow (SO) where a whole community of PyTorch and
     Tensorflow enthusiast can help you out. In this case, make sure to tag your
     question with the right deep learning framework as well as the
     huggingface-transformers tag: 
     https://stackoverflow.com/questions/tagged/huggingface-transformers 
     -->

## Details
<!-- Description of your issue -->

<!-- You should first ask your question on the forum or SO, and only if
     you didn't get an answer ask it here on GitHub. -->

Hi I'm trying to using model named 'BartForConditionalGeneration' for text generation, 
The main question and problem is, when I pass the 'decoder_input_ids'  to the model as the input of forward in model,
the first index of model's output shows the tensor shape as [batch_size, 1 , vocab_size] all the time.
I might misunderstood the model and Docs of Bart overall,
when I only pass the Input_ids without decoder_input_ids, it worked right.
but It never make the shape of tensor like [batch_size, target_seq_len, vocab_size] when I put 'decoder_input_ids'

I think when I put the decoder_input_ids as [batch_size, target_seq_len], the Model's output should be like 
[batch_size, target_seq_len, vocab_size] (for prediction scores)

the reason why I tried to pass the decoder_input_ids was , I think Bart is seq2seq model, and their output logits will be computed just like the basic transformer, so I wanted to pretrain this model.

Please help me to solve this problem.  If I misunderstood the concept of Bart at all, just comment me. 
I need a clean view for my situation 

I used mBartTokenizer and also 'facebook/mbart-large-en-ro'  pretrained weights, 
I used Model BartForConditionalGeneration.from_pretrained('facebook/mbart-large-en-ro')
It also didn't worked for normal BartModel (with same pretrained config)

I'm really confused about it so the whole question looks like a mass, Sorry for that
Thank you for reading my questions. Please Help Me.
![1](https://user-images.githubusercontent.com/51171232/87811751-1a4d1700-c89a-11ea-8e47-0ae2987dfe29.PNG)



Comments:
This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
How to get probabilities from MarianMT models?
Author: bes-dev
Pull request: no
Labels: name: wontfix

Hey,

I try to use MarianMT models for distillation, but I didn't found way to get raw probabilities from model.

I use model as in examples:
```
from transformers import MarianMTModel, MarianTokenizer
class Translator:
    def __init__(self, src, dst):
        model_name = f'Helsinki-NLP/opus-mt-{src}-{dst}'
        self.tokenizer = MarianTokenizer.from_pretrained(model_name)
        self.model = MarianMTModel.from_pretrained(model_name)

    def __call__(self, text):
        if not isinstance(text, list):
            text = [text]
        preds = self.model.generate(**self.tokenizer.prepare_translation_batch(text))
        print(preds)
        translation = [self.tokenizer.decode(t, skip_special_tokens=True) for t in preds]
        return translation
```

How can I get translation and probs via Huggingface API?
Thanks in advance!

Comments:
This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
Adding Minimal Reproducible Usage Example For TPU support on examples/seq2seq
Author: AdityaSoni19031997
Pull request: yes
Labels: name: wontfix

Attempt to resolve https://github.com/huggingface/transformers/issues/5895.

Minimal Working [Colab Example](https://colab.research.google.com/drive/16q2GWrnZ0Tjg1OxJQUcaWKCWwn3Jh5z0?usp=sharing).

For using more than a single core, one needs to ensure that enough RAM is available else wait for PyTorch-XLA to release a stable version. They have also [released](https://github.com/pytorch/xla/issues/1870#issuecomment-623603323) a fix way back that prevent excessive memory usage for nightly.

Comments:
The test will obviously break, right?

Any progress on merging this?

they won't accept it as the checks have failed? 
But it's expected for the checks to fail as I have modified `modeling_bart` and it's gonna have one more Param count.

Is that so @sshleifer?

Thanks for the contribution, this looks awesome!

We can't merge with failing tests, but I think the tests can pass.

Could you also check
```
RUN_SLOW=1 pytest tests/test_modeling_bart.py
RUN_SLOW=1 pytest tests/test_modeling_marian.py
RUN_SLOW=1 pytest test_modeling_mbart.py
```
add the `USE_CUDA=1` prefix to make them run faster on GPU.

Actually can we add a `support_tpu` flag to BartConfig, init it to False, and only allocate `lm_head` if it's set to True. I'm concerned that we are wasting RAM when we train on GPU. (I would happily change my mind if I encountered evidence that this change doesn't change GPU RAM consumption.)

I tried this version and it seems to work but it stucks at "Validation sanity check". Working colab [here](https://colab.research.google.com/drive/1NA9_EPEBNmo7feQ60iiznsLP_XbWwQmC?usp=sharing)

Well I removed the validation check altogether by passing in the concerned
flag to 0. Tried debugging to find out what's causing it, but I couldn't
figure it out.

If you will train and validate, it will work.

On Mon, 27 Jul 2020, 17:46 marton-avrios, <notifications@github.com> wrote:

> I tried this version and it seems to work but it stucks at "Validation
> sanity check".
>
> —
> You are receiving this because you authored the thread.
> Reply to this email directly, view it on GitHub
> <https://github.com/huggingface/transformers/pull/5960#issuecomment-664360705>,
> or unsubscribe
> <https://github.com/notifications/unsubscribe-auth/AFNPJJQYIPQDG5ZPYXDTTT3R5VV23ANCNFSM4PEHWJYA>
> .
>


This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.


This is now supported by `Seq2SeqTrainer`. Use that if you want TPU support!
Can't use BatchEncoding in the fit function
Author: konstantin-doncov
Pull request: no
Labels: name: wontfix

I had a [few problems with the `transformers`](https://github.com/huggingface/transformers/issues/5555) and the related [`tensorflow` functionality](https://github.com/tensorflow/tensorflow/issues/41204), but now I've made some progress using trial and error method. 
You can see the issue in this [gist](https://colab.research.google.com/drive/125jJ0qrXGIe6goNrH_Ja7XPZtYp7nMXU?usp=sharing), now I found out that the problem was due to `model.fit(...)` function. 
Here is the first version which causes the error:
```
model.fit(
    x=X_train, #transformers.tokenization_utils_base.BatchEncoding
    y=targets,
    epochs=3
)
```
Output:
```
ValueError: too many values to unpack (expected 2)
```
Second version which at least works:
```
model.fit(
    x=X_train.values(), #dict_values
    y=targets,
    epochs=3
)
```
**So, now the main question is why does it work and is my code correct?** I'm asking because I have seen many code fragments which used `BatchEncoding` as `x` in the `fit`(e.g. [this](https://www.kaggle.com/definedennis/pretrained-bert-with-huggingface-tensorflow-2-1/) and [this](https://github.com/thakursc1/NLPKernels/blob/3bb1fcac60ab8cdc6f2f68a2d9b5b7a477873811/DisasterTweetClassification/Transformer.py)), but I just can't do the same thing.

Comments:
This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
How to get a language model score in BertModel?
Author: oshindow
Pull request: no
Labels: name: wontfix

# ❓ Questions & Help

<!-- The GitHub issue tracker is primarly intended for bugs, feature requests,
     new models and benchmarks, and migration questions. For all other questions,
     we direct you to the Hugging Face forum: https://discuss.huggingface.co/ .
     You can also try Stack Overflow (SO) where a whole community of PyTorch and
     Tensorflow enthusiast can help you out. In this case, make sure to tag your
     question with the right deep learning framework as well as the
     huggingface-transformers tag: 
     https://stackoverflow.com/questions/tagged/huggingface-transformers 
     -->

## Details
<!-- Description of your issue -->
Hi, thanks for your awesome works! 

How to get a language model score which could judge whether a sentence conforms to grammatical rules by using BertModel ? 
<!-- You should first ask your question on the forum or SO, and only if
     you didn't get an answer ask it here on GitHub. -->


Comments:
This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
Easy selection of a learning rate scheduler when initializing a Trainer
Author: freeIsa
Pull request: no
Labels: name: wontfix

# 🚀 Feature request

Please consider adding a new argument to the Trainer constructor to specify which among the available learning rate schedulers should be used during training.

## Motivation

Even though Trainer already has the option to specify a given optimizer and learning rate scheduler, you need to explicitly initialize both (even when you only want to change the scheduler) with parameters already available to the Trainer itself via TrainingArguments. It would be nicer and smoother to just provide Trainer with a string specifying which scheduler to use (e.g. 'constant_schedule', 'cosine_schedule_with_warmup', ...) and have `get_optimizers` implement the choice.


Comments:
This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
module 'tensorflow_core._api.v2.config' has no attribute 'list_physical_devices'
Author: stas00
Pull request: no
Labels: name: wontfix

# 🐛 Bug

A bunch of tf tests fail with:

```module 'tensorflow_core._api.v2.config' has no attribute 'list_physical_devices'```

```
ERROR tests/test_benchmark_tf.py - AttributeError: module 'tensorflow_core._api.v2.config' has no attribute 'list_physical_devices'
ERROR tests/test_benchmark_tf.py - AttributeError: module 'tensorflow_core._api.v2.config' has no attribute 'list_physical_devices'
ERROR tests/test_benchmark_tf.py - AttributeError: module 'tensorflow_core._api.v2.config' has no attribute 'list_physical_devices'
ERROR tests/test_benchmark_tf.py - AttributeError: module 'tensorflow_core._api.v2.config' has no attribute 'list_physical_devices'
```

`tf.config.list_physical_devices` seems to be added in tf-2.1, so unless `transformers` starts to require tf >= 2.1, this breaks for tf < 2.1.

Depending on what you decide I can send a PR to fix this with either:

a. require  tf-2.1+ (simplest)
b. write a wrapper `list_physical_devices` that uses `tf.config.experimental.list_physical_devices` for tf < 2.1, and `tf.config.list_physical_devices` otherwise, and switch to using it.
c. do nothing?

## Environment info

```
- `transformers` version: 3.0.2
- Platform: Linux-4.15.0-109-generic-x86_64-with-debian-buster-sid
- Python version: 3.7.5
- PyTorch version (GPU?): 1.5.1 (True)
- Tensorflow version (GPU?): 2.0.1 (False)
- Using GPU in script?: <fill in>
- Using distributed or parallel set-up in script?: <fill in>
```


Comments:
This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.