How to run transformer model like t5-small, facebook/bart-large-cnn without loading pretrained weights?
Author: xuyeliu
Pull request: no

Same with the title. When using run_summarization.py, how to run transformer models like t5-small, facebook/bart-large-cnn without loading pre-trained weights?  I only want to train their original model architecture without pre-trained model. Thanks!

Comments:
For this you could either initialize a random model, save it and pass it's path as `model_name_or_path` arg.
Or modify the script to create a random model instead of pre-trained. i.e to init the model use

```
model = AutoModelForSeq2SeqLM(config)
```
instead of using `.from_pretrained`.

> For this you could either initialize a random model, save it and pass it's path as `model_name_or_path` arg.
> Or modify the script to create a random model instead of pre-trained. i.e to init the model use
> 
> ```
> model = AutoModelForSeq2SeqLM(config)
> ```
> 
> instead of using `.from_pretrained`.

Thanks for your swift reply. I already try to use model = AutoModelForSeq2SeqLM(config)
instead of using .from_pretrained. But it has bugs:
```{r}
  File "examples/pytorch/summarization/run_summarization.py", line 358, in main
    model = AutoModelForSeq2SeqLM(config)
TypeError: __init__() takes 1 positional argument but 2 were given
```
It seems I should use ```model = T5ForConditionalGeneration(config = config)``` or ```model = BartForConditionalGeneration(config = config)```
when I want to train a Bart or T5 model from scratch without loading pretrained weights. Is that right? Thank you very much!

Ohh sorry,

It should be `AutoModelForSeq2SeqLM.from_config(...)`

and yeah, you could also use the individual classes if you want.

> Ohh sorry,
> 
> It should be `AutoModelForSeq2SeqLM.from_config(...)`
> 
> and yeah, you could also use the individual classes if you want.

Thanks for your reply. One quick question, If I use ```AutoModelForSeq2SeqLM.from_config(...)```, when I mentioned t5-small or t5-base or t5-large, is it the same amone these models? Also If I use model = ```T5ForConditionalGeneration(config = config)```, which model I am using? t5-small or t5-base or t5-large? Thank you very much!

> which model I am using? t5-small or t5-base or t5-large? 

This depends on the `config`, this initializes models according the values in the `config`, so if the `config` is of `t5-small` the model will be of `t5-small` size with random weights.

> > which model I am using? t5-small or t5-base or t5-large?
> 
> This depends on the `config`, this initializes models according the values in the `config`, so if the `config` is of `t5-small` the model will be of `t5-small` size with random weights.

I see, thank you so much! Last question,  sorry for asking so many times lol. I am trying to train T5-large from scratch, but it is very slow even though I use gpu.  Do you know how to run run_summarization. py with multi_gpu?  Thank you very much! 

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md) are likely to be ignored.