noamaanMulla-03/transformers-issues huggingface dataset:
Confused about the time of forword
Author: dy1998
Pull request: no

## Environment info
<!-- You can run the command `transformers-cli env` and copy-and-paste its output below.
     Don't forget to fill out the missing fields in that output! -->

- `transformers` version:3.0.0
- Platform: ubuntu 1604
- Python version: 3.7
- PyTorch version (GPU?):1.2.0 gpu
- Tensorflow version (GPU?): none
- Using GPU in script?: none
- Using distributed or parallel set-up in script?:No

### Who can help
@LysandreJik

## Information

Model I am using Bert, spceifically roberta_chinese_clue_tiny

The problem arises when using:
BertModel.from_pretrained

The tasks I am working on is:
just forward

Question:
Why the time of same forword is different? How to make it same?

## To reproduce
Steps to reproduce the behavior:
```
from transformers import BertTokenizer, BertModel
import time

tokenizer = BertTokenizer.from_pretrained("clue/roberta_chinese_clue_tiny")

model = BertModel.from_pretrained("clue/roberta_chinese_clue_tiny")

inputs = tokenizer("testtest", return_tensors="pt")

time_start=time.time()
outputs = model(**inputs)
time_end=time.time()

time_start2=time.time()
outputs = model(**inputs)
time_end2=time.time()

print('totally cost',time_end-time_start)
print('totally cost2',time_end2-time_start2)
```

## Expected behavior
```
totally cost 0.2720155715942383
totally cost2 0.007731199264526367
```


Comments:
I assume that that's how Python works, if you run the same thing again, the result will be cached. If you had provided different inputs, then the time would be the same. 

> I assume that that's how Python works, if you run the same thing again, the result will be cached. If you had provided different inputs, then the time would be the same.

Thanks for reply，but when i have tried differnt input(same length sentense), the inference time of first forword is longer than second forword(even 100x), i guess the model init produce the time cost

The same length sentences can have a different number of tokens after tokenizing. You could rather randomly create `input_ids` of the same shape to test this reliabley.

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md) are likely to be ignored.
About the speed when return_dict is set to True
Author: ridiculouz
Pull request: no

Hi!
I just want to know whether or not the speed is slower in model forward function like roberta or bert when return_dict=True.

Comments:
No, it shouldn't!

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md) are likely to be ignored.
Unable to convert Facebook/mbart-many-to-many model to onxx
Author: sankarsiva123
Pull request: no

When I tried to convert Facebook/mbart-many-to-many model . I am unable to convert I am getting issue.
Pls help me to convert this model to ONXX

Comments:
I do not think mBART can be converted to ONNX as of now.

Hi Thanks for the information.
Facebook/many-to-many takes 9s seconds for translation on cpu , is there a way to reduce the inference time ?

Hi @sankarsiva123, have you tried HF's API inference ?

9s per inference seems a bit off: https://huggingface.co/facebook/mbart-large-50-many-to-many-mmt?text=Hello+there+%21+
We do run some optimizations there as HF's hosted API but still it seems like you could have better inference times than 9s.

Maybe it depends on what you are sending it ? Are you using GPU or CPU ?

Hi, @Narsil Yeah, I tried HF's API inference, it is pretty much fast.
I am using CPU, I tried both in google colab, and in my local, it is taking around 9s.
Am I missing something while using the model, so my inference time is high than normal?
Also pls let me know is there a way to reduce inference time?
![image](https://user-images.githubusercontent.com/58412261/109768056-6c229180-7c1e-11eb-9ad2-b3d9bf0a14ca.png)




Can you time your inner loop without the tokenizer ? (Just making sure it's not that).

Otherwise you see to use generate, which is the right way to go.
I don't know colab's CPU nor yours, but it could definitely be the problem (or the pytorch version you're rolling which might have not been optimized for your CPU instruction set.)

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md) are likely to be ignored.
Trainer: Make `best_model_checkpoint` path in `trainer_state.json` relative to `args.output_dir`
Author: tanmay17061
Pull request: no

# 🚀 Feature request
An enhancement of `best_model_checkpoint` for more robustness.
<!-- A clear and concise description of the feature proposal.
     Please provide a link to the paper and code in case they exist. -->
Currently `Trainer.state.best_model_checkpoint` holds absolute path to the best checkpoint when `Trainer.args.load_best_model_at_end=True` is passed.  
It can be useful if `Trainer.state.best_model_checkpoint` value is relative to `Trainer.args.output_dir`
## Motivation

**Absolute path hinders portability** of the trained models.  
For example, if a user wants to continue _previous training_ using argument `resume_from_checkpoint` for `Trainer.train`, not having the `output_dir` exactly same as the _previous training_ (eg- renaming of any directory in the path) can break the `load_best_model_at_end` functionality due to the previous absolute paths being no longer valid.
<!-- Please outline the motivation for the proposal. Is your feature request
     related to a problem? e.g., I'm always frustrated when [...]. If this is related
     to another GitHub issue, please link here too. -->

## Your contribution

I can raise a PR if this is a useful change to have!
<!-- Is there any way that you could help, e.g. by submitting a PR?
     Make sure to read the CONTRIBUTING.MD readme:
     https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md -->


Comments:
I think this would a very welcome change indeed, so please work on that if you have time and if it's something you'd like to do!

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md) are likely to be ignored.
Model Hub: Search by model size
Author: ioana-blue
Pull request: no

# 🚀 Feature request

It would be great if the model cards for models would include the model size (i.e., the number of parameters) and then the model hub will allow searching for models by size.

## Motivation
Depending on the task/problem/context, smaller or larger models are more beneficial. It's hard to keep up with all the models out there. For example, if I'm interested in distilled/compressed/smaller BERTs, I may be able to remember DistilBERT, MobileBERT but maybe forget about SqueezeBERT, TinyBERT, etc. A search by size would make all these smaller models visible.



Comments:
Definitely a good idea

And since I started talking about model cards... :) I think it would be cool if you guys actually imposed some format. I think the original paper/idea had a format. Now "model cards" stands for "whatever the researcher had time to fill in that day" :) A few fields of interest: model size, training data, NLP tasks, language(s), paper, _maybe_ something about model of inspiration (e.g., TinyBERT is a modification of BERT by...). 

I agree, there should at least be a template in my opinion. I hate to find models on the hub which don't provide any information. Moreover, all model cards look different, there's not really a structure.

There is a template we link to in the second question of https://huggingface.co/docs (=> https://github.com/huggingface/model_card), though we should make it more built-in/central at some point.

It would also be nice if the template also included details on tokenisation, what algorithm was used (BPE, Unigram, Word Piece) and the parameters (vocab size etc).

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md) are likely to be ignored.
Fine-tune pretrained Wav2Vec2 on a small custom dataset 
Author: Omarnabk
Pull request: no

I am wondering how to **fine-tune** a pre-trained model on a small speech/audio dataset. I have 10 hours of audio with their transcript. 

I would like to fine-tune a model and then use it as described here:
[](https://huggingface.co/facebook/wav2vec2-large-960h)

Thanks!

Comments:
Patrick is working on it, see #10145 

I am also searching for fine-tuning of this model.

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md) are likely to be ignored.
Training LongformerForQuestionAnswering on TriviaQA
Author: SapirWeissbuch
Pull request: no

Hello,

Could you please give a short explanation on how to retrain `LongformerForQuestionAnswering` in order to receive these weights: `allenai/longformer-large-4096-finetuned-triviaqa` (https://huggingface.co/allenai/longformer-large-4096-finetuned-triviaqa/tree/main). 

Thank you,
Sapir

Comments:
Pinging @ibeltagy and @patrickvonplaten who would know better.

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md) are likely to be ignored.
denoising objective for pretraining
Author: dorooddorood606
Pull request: no

Hi
Denoising objective is used in T5 and BART models, could you please add it in pretraining language models?
For now, if you could I appreciate sharing some advice how I can implement it. is there a piece of codes in huggingface I could start from? 
thanks 

Comments:
@patil-suraj @patrickvonplaten  please help. thanks 

Hey @dorooddorood606,

could you please make use of the forum: https://discuss.huggingface.co/ for such questions. Thanks!

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md) are likely to be ignored.
Upgrade transformers from 3.5.0 to 4.3.2 instance error
Author: brunopistone
Pull request: no

Hi guys. I tried to update the transformers module from the version 3.5.0 to the version 4.3.2.
After this upgrade, the code that previously was working now has some problems.

This is my code:
```
from transformers import AutoConfig, AutoTokenizer, TFAutoModel

config = AutoConfig.from_pretrained("amazon/bort")
model = TFAutoModel.from_pretrained("amazon/bort", config=config)
bert_main_layer = model.bert

encoder, pooler = bert_main_layer(input_ids_in, attention_mask=input_masks_in)

X = tf.keras.layers.Dropout(config.hidden_dropout_prob)(pooler)
X = tf.keras.layers.Dense(
    constants.CLASSES,
    kernel_initializer=tf.keras.initializers.TruncatedNormal(stddev=config.initializer_range),
    activation="softmax"
)(X)

model = tf.keras.Model(
    inputs=[input_ids_in, input_masks_in],
    outputs=[X]
)
```

In particular, when I instantiate the class `bert_model` (`bert_main_layer(input_ids_in, attention_mask=input_masks_in)`), with the 3.5.0 version it returns two tensors:
```
<tf.Tensor 'bert/encoder/layer_._3/output/LayerNorm/batchnorm/add_1:0' shape=(None, 333, 1024) dtype=float32>
<tf.Tensor 'bert/pooler/dense/Tanh:0' shape=(None, 1024) dtype=float32>
```

With the version 4.3.2 it returns two strings:
```
last_hidden_state
pooler_output
```

The consequence is that now I have this exception on the Dense layer:
```
ValueError: Input 0 of layer dense is incompatible with the layer: : expected min_ndim=2, found ndim=0. Full shape received: []
```

Since I haven't found any documentation or guidance, can you please help me? What I'm doing wrong?
Thank you

Comments:
Hi, thanks for opening an issue. The breaking changes from version v3 to v4 are heavily documented: https://huggingface.co/transformers/migration.html#migrating-from-transformers-v3-x-to-v4-x

Your particular issue is [bullet number 4](https://huggingface.co/transformers/migration.html#switching-the-return-dict-argument-to-true-by-default).

For our reference, when trying to look for breaking changes between v3 and v4, where did you look? If we can improve visibility for those, it would be great. Maybe "Migration" isn't the best identifier? Are there some specific keywords you used?

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md) are likely to be ignored.
CUDA memory error on increasing the number of generations
Author: ShivamSharma1997
Pull request: no

I am getting CUDA memory error when generating text with num_return_sequences set to more than 100 for a self-trained gpt2 model. This is not expected since after every generation there should be nothing left in the GPU.





Comments:
By setting `num_return_sequences` you're creating bigger batches, so it is expected to have an OOM if you ask for too much returned sequences.

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md) are likely to be ignored.
ConvBert not compatible with torch v1.6
Author: patrickvonplaten
Pull request: no

ConvBERT uses statements like `torch.multiply` which did not exist in pytorch v1.6 => ConvBERT is not compatible with v1.6 (cc @abhishekkrthakur).

This can easily be checked when running:

`pytest tests/test_modeling_convbert.py`

@LysandreJik, @sgugger - It would be great to test all the different pytorch versions in a slow test I think.

Comments:
Testing all versions from torch v1.3.0+ is indeed on the roadmap, I expect ~1 month out alongside all the other tests improvements.

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md) are likely to be ignored.
TransCoder
Author: patil-suraj
Pull request: yes

# What does this PR do?

Adds TransCoder https://github.com/facebookresearch/TransCoder

Comments:
This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md) are likely to be ignored.
Fine-tuning Seq2Seq models for Machine translation
Author: MorenoLaQuatra
Pull request: no

Good morning,

@micheledaddetta1

We were experimenting with Seq2Seq models such as MarianMT or T5. 
I was wondering if there is a common way to fine-tune those models with custom datasets for the machine translation task.
Specifically we did the following:

```python
embeddings = self.batch_encode_plus(sentences, padding=True, verbose=False)
embeddings = embeddings.to(self.model.device)

labels = self.batch_encode_plus(target_sentences,  padding=True, verbose=False).input_ids.to(self.model.device)  # expected output tokens
outputs = self.model(input_ids=embeddings.input_ids, labels=decoder_input_ids, return_dict=True)
output_sentences = self.model.generate(**embeddings)
output_sentences = self.decode(output_sentences)
# we also compute sentence embeddings for embedding alignment purposes.
return output_sentences, outputs.loss
```

Is it correct to directly use `outputs.loss` to optimize the model for the machine translation task? 

Comments:
Maybe @patrickvonplaten or @patil-suraj can chime in here!

Hey @MorenoLaQuatra 

The `run_seq2seq.py` supports translation with custom dataset. And you can use T5, mT5, MarianMT, mBART, mBART-50 for fine-tuning.
https://github.com/huggingface/transformers/tree/master/examples/seq2seq

And the best place to ask this question is our forum https://discuss.huggingface.co/

Hope this helps :) 

Hi @patil-suraj, thank you for the feedback.

Actually we are trying to train it in a python script (the overall architecture is more complex than the single network). We were asking about the translation loss. If we do as reported in the above snippet, do we optimize the translation loss?
The finetuning for MarianMT use the same head and loss of the training phase?

Thank you for your time.

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md) are likely to be ignored.
run_langauge_modeling for T5
Author: ghost
Pull request: no

Hi
Based on readme on [1], run_langauge_modeling.py does not support T5 model so far, it would be really nice to include this model as well.
There is also this line "data_args.block_size = tokenizer.max_len", max_len does not exist anymore, I searched in pretrainedTokernizer class and did not find an equivalent variable to substitue, do you mind telling me how I can update this line to make this example work? 

thank you.  

[1] https://github.com/huggingface/transformers/blob/master/examples/legacy/run_language_modeling.py

Comments:
Hi

Seems to me this script is the repetition of this other script: transformers/examples/language-modeling/run_mlm.py 
Do you mind adding T5 also to this script?  thanks 

Actually, we can not simply add T5 to this script, because `run_mlm.py` is for encoder-only models (such as BERT, RoBERTa, DeBERTa, etc.). T5 is an encoder-decoder (seq2seq) model, so this would require a new script. The [seq2seq scripts](https://github.com/huggingface/transformers/tree/master/examples/seq2seq) currently only support fine-tuning, not pre-training.

cc @patil-suraj @sgugger 

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md) are likely to be ignored.
Can you give some suggestion about add features with input_ids to token-classification model ?
Author: svjack
Pull request: no

Hi, I want to add POS TAG labels as input with input_ids as auxiliary feature feature in NER model.
How can i get the entry in the forward function ?
And i review some implementations about model use bert model to perform NER  such as 
https://github.com/monologg/JointBERT/blob/master/model/modeling_jointbert.py
It seems if i want to ad this feature (POS TAG) i must rewrite the forward function in BertModel
and seems i should train the edit model from scratch.
Can you give some suggestions about edit model structure and partial weights load from 
huggingface pre-trained-models ? 


Comments:
The more general question is that dose huggingface model support some interface about
add other char level feature as  auxiliary features ?

And i search some bert with ner implementations, they all not use pos tag features.
But when i view the following code :
**https://github.com/sberbank-ai/ner-bert/blob/master/modules/models/bert_models.py**
i think if add pos tag feature in lstm level (use bert embedding as input)
it seems more suitable.
Do you think this is a general solution to combine token level feature with huggingface features.
Or the future releases will support some feature combination model above bert constructions 
as options ? 

Or i can use set_input_embeddings and
https://github.com/plasticityai/magnitude
Additional Featurization 
to the original embedding?

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md) are likely to be ignored.
Pegasus ONNX format?
Author: Ashwin367
Pull request: no

I tried to convert Pegasus to the ONNX format using [this ](https://colab.research.google.com/github/huggingface/transformers/blob/master/notebooks/04-onnx-export.ipynb#scrollTo=foYlXrSksR_v) guide but it failed. Can Pegasus be converted to the ONNX format or is that not possible yet?

Comments:
follow this [answer](https://stackoverflow.com/a/66117248/13273054), posted on StackOverflow.

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md) are likely to be ignored.
support Mixed Precision and avoid 'dtype=float32' in implementation
Author: xuxingya
Pull request: no

# 🚀 Feature request
When using Longformer model, I found the dtype of many tensors are assigned directly by `dtype=tf.dtypes.float32` or equations, which makes mixed precision raining impossible. And I found that other models also have this problem.

Of course It is not a bug to not support mixed precision training. But because Transformer models are usually very large, it will be appreciated to support mix precision training when implementing new models. 

So I suggest if possible please use dtype inference instead of direct dtype assignment.


Comments:
Hello!
Thanks for the feature requests. We are currently working on this, some of them already support mixed precision 👍 

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md) are likely to be ignored.