How to pre-train BART model
Author: omerarshad
Pull request: no
Labels: name: wontfix

name: Ex: LM (Finetuning)

name: Ex: LM (Pretraining)

How to pre-train BART model in an unsupervised manner. any example?

Comments:
We still need to provide a good docstring/notebook for this. It's on our ToDo-List. :-) 

Or @sshleifer - is there already something for Bart? 

Nothing yet, would be good to add!

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.


I have seen the same [issue](https://github.com/pytorch/fairseq/issues/1899) in fairseq BART!.

Hi, any news about bart pre-training? 

 who can tell me how to pre-train the bart on my own dataset?   I am so  confused ....
thank you so much

Maybe this comment can help: https://github.com/huggingface/transformers/issues/5096#issuecomment-645860271

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.


Any news on this please?


not so far, would be great to have it. Thanks.

I and my co-worker wrote a demo according to roberta pretraining demo.
```
#encoding=utf-8

from transformers import (
    BartForConditionalGeneration, BartTokenizer, BartForCausalLM,
    Seq2SeqTrainingArguments, Seq2SeqTrainer
  )

import torch
from torch.utils.data import random_split


# ## Initiating model and trainer for training
from transformers import BartModel, BartConfig
from transformers import BartTokenizerFast


configuration = BartConfig(
    vocab_size=52000,
    max_position_embeddings=258,
    d_model=256,
    encoder_layers=3,
    decoder_layers=3,
    encoder_attention_heads=4,
    decoder_attention_heads=4,
    decoder_ffn_dim=1024,
    encoder_ffn_dim=1024,
)
model = BartForCausalLM(configuration)
tokenizer = BartTokenizerFast.from_pretrained("./dic", max_len=256, additional_special_tokens=['[CH]', '[OTHER]', '[VAR]', '[NUM]'])


# ### HTTP Request DataPreparing & Modeling
data = []
with open("../data/sample.txt") as f1:
    for src in f1:
      data.append(
          {
              "seq2seq": {
                  "input": src.strip()
              }
          }
      )
print(f'total size of data is {len(data)}')


# splitting dataset into train, validation
split = 0.2
train_dataset, eval_dataset = random_split(data, lengths=[int((1-split)*len(data))+1, int(split*len(data))])


# defining collator functioon for preparing batches on the fly ..
def data_collator(features:list):
   inputs = [f["seq2seq"]["input"] for f in features]
   batch = tokenizer.prepare_seq2seq_batch(src_texts=inputs, max_length=256, padding='max_length')
   batch["labels"] = batch["input_ids"].copy()
   for k in batch:
        batch[k] = torch.tensor(batch[k])
   return batch


batch_out = data_collator(eval_dataset)
print(batch_out)
print(batch_out['input_ids'].shape,batch_out['labels'].shape,batch_out['attention_mask'].shape)


# defining training related arguments
args = Seq2SeqTrainingArguments(output_dir="clm-checkpoints",
                        do_train=True,
                        do_eval=True,
                        evaluation_strategy="epoch",
                        per_device_train_batch_size=8,
                        per_device_eval_batch_size=8,
                        learning_rate=5e-5,
                        num_train_epochs=1,
                        logging_dir="./logs")


# defining trainer using 🤗
trainer = Seq2SeqTrainer(model=model, 
                args=args, 
                data_collator=data_collator, 
                train_dataset=train_dataset, 
                eval_dataset=eval_dataset)


# ## Training time
trainer.train()
# It will take hours to train this model on this dataset


# lets save model
trainer.evaluate(eval_dataset=eval_dataset)
trainer.save_model("clm-checkpoints")

```

> I and my co-worker wrote a demo according to roberta pretraining demo.
> 
> ```
> #encoding=utf-8
> 
> from transformers import (
>     BartForConditionalGeneration, BartTokenizer, BartForCausalLM,
>     Seq2SeqTrainingArguments, Seq2SeqTrainer
>   )
> 
> import torch
> from torch.utils.data import random_split
> 
> 
> # ## Initiating model and trainer for training
> from transformers import BartModel, BartConfig
> from transformers import BartTokenizerFast
> 
> 
> configuration = BartConfig(
>     vocab_size=52000,
>     max_position_embeddings=258,
>     d_model=256,
>     encoder_layers=3,
>     decoder_layers=3,
>     encoder_attention_heads=4,
>     decoder_attention_heads=4,
>     decoder_ffn_dim=1024,
>     encoder_ffn_dim=1024,
> )
> model = BartForCausalLM(configuration)
> tokenizer = BartTokenizerFast.from_pretrained("./dic", max_len=256, additional_special_tokens=['[CH]', '[OTHER]', '[VAR]', '[NUM]'])
> 
> 
> # ### HTTP Request DataPreparing & Modeling
> data = []
> with open("../data/sample.txt") as f1:
>     for src in f1:
>       data.append(
>           {
>               "seq2seq": {
>                   "input": src.strip()
>               }
>           }
>       )
> print(f'total size of data is {len(data)}')
> 
> 
> # splitting dataset into train, validation
> split = 0.2
> train_dataset, eval_dataset = random_split(data, lengths=[int((1-split)*len(data))+1, int(split*len(data))])
> 
> 
> # defining collator functioon for preparing batches on the fly ..
> def data_collator(features:list):
>    inputs = [f["seq2seq"]["input"] for f in features]
>    batch = tokenizer.prepare_seq2seq_batch(src_texts=inputs, max_length=256, padding='max_length')
>    batch["labels"] = batch["input_ids"].copy()
>    for k in batch:
>         batch[k] = torch.tensor(batch[k])
>    return batch
> 
> 
> batch_out = data_collator(eval_dataset)
> print(batch_out)
> print(batch_out['input_ids'].shape,batch_out['labels'].shape,batch_out['attention_mask'].shape)
> 
> 
> # defining training related arguments
> args = Seq2SeqTrainingArguments(output_dir="clm-checkpoints",
>                         do_train=True,
>                         do_eval=True,
>                         evaluation_strategy="epoch",
>                         per_device_train_batch_size=8,
>                         per_device_eval_batch_size=8,
>                         learning_rate=5e-5,
>                         num_train_epochs=1,
>                         logging_dir="./logs")
> 
> 
> # defining trainer using 🤗
> trainer = Seq2SeqTrainer(model=model, 
>                 args=args, 
>                 data_collator=data_collator, 
>                 train_dataset=train_dataset, 
>                 eval_dataset=eval_dataset)
> 
> 
> # ## Training time
> trainer.train()
> # It will take hours to train this model on this dataset
> 
> 
> # lets save model
> trainer.evaluate(eval_dataset=eval_dataset)
> trainer.save_model("clm-checkpoints")
> ```

Thanks for the code example, I am also planning on implementing pretrained from scratch, and I've got several questions for the code
- I noticed that you use pretrained bart tokenizer, how can I pretrain it for different language?
- How much compute did you use for your implementation?

> > I and my co-worker wrote a demo according to roberta pretraining demo.
> > ```
> > #encoding=utf-8
> > 
> > from transformers import (
> >     BartForConditionalGeneration, BartTokenizer, BartForCausalLM,
> >     Seq2SeqTrainingArguments, Seq2SeqTrainer
> >   )
> > 
> > import torch
> > from torch.utils.data import random_split
> > 
> > 
> > # ## Initiating model and trainer for training
> > from transformers import BartModel, BartConfig
> > from transformers import BartTokenizerFast
> > 
> > 
> > configuration = BartConfig(
> >     vocab_size=52000,
> >     max_position_embeddings=258,
> >     d_model=256,
> >     encoder_layers=3,
> >     decoder_layers=3,
> >     encoder_attention_heads=4,
> >     decoder_attention_heads=4,
> >     decoder_ffn_dim=1024,
> >     encoder_ffn_dim=1024,
> > )
> > model = BartForCausalLM(configuration)
> > tokenizer = BartTokenizerFast.from_pretrained("./dic", max_len=256, additional_special_tokens=['[CH]', '[OTHER]', '[VAR]', '[NUM]'])
> > 
> > 
> > # ### HTTP Request DataPreparing & Modeling
> > data = []
> > with open("../data/sample.txt") as f1:
> >     for src in f1:
> >       data.append(
> >           {
> >               "seq2seq": {
> >                   "input": src.strip()
> >               }
> >           }
> >       )
> > print(f'total size of data is {len(data)}')
> > 
> > 
> > # splitting dataset into train, validation
> > split = 0.2
> > train_dataset, eval_dataset = random_split(data, lengths=[int((1-split)*len(data))+1, int(split*len(data))])
> > 
> > 
> > # defining collator functioon for preparing batches on the fly ..
> > def data_collator(features:list):
> >    inputs = [f["seq2seq"]["input"] for f in features]
> >    batch = tokenizer.prepare_seq2seq_batch(src_texts=inputs, max_length=256, padding='max_length')
> >    batch["labels"] = batch["input_ids"].copy()
> >    for k in batch:
> >         batch[k] = torch.tensor(batch[k])
> >    return batch
> > 
> > 
> > batch_out = data_collator(eval_dataset)
> > print(batch_out)
> > print(batch_out['input_ids'].shape,batch_out['labels'].shape,batch_out['attention_mask'].shape)
> > 
> > 
> > # defining training related arguments
> > args = Seq2SeqTrainingArguments(output_dir="clm-checkpoints",
> >                         do_train=True,
> >                         do_eval=True,
> >                         evaluation_strategy="epoch",
> >                         per_device_train_batch_size=8,
> >                         per_device_eval_batch_size=8,
> >                         learning_rate=5e-5,
> >                         num_train_epochs=1,
> >                         logging_dir="./logs")
> > 
> > 
> > # defining trainer using 🤗
> > trainer = Seq2SeqTrainer(model=model, 
> >                 args=args, 
> >                 data_collator=data_collator, 
> >                 train_dataset=train_dataset, 
> >                 eval_dataset=eval_dataset)
> > 
> > 
> > # ## Training time
> > trainer.train()
> > # It will take hours to train this model on this dataset
> > 
> > 
> > # lets save model
> > trainer.evaluate(eval_dataset=eval_dataset)
> > trainer.save_model("clm-checkpoints")
> > ```
> 
> Thanks for the code example, I am also planning on implementing pretrained from scratch, and I've got several questions for the code
> 
> * I noticed that you use pretrained bart tokenizer, how can I pretrain it for different language?
> * How much compute did you use for your implementation?



For the first question, just like this:

```
from tokenizers import (ByteLevelBPETokenizer,SentencePieceBPETokenizer,BertWordPieceTokenizer)

tokenizer = ByteLevelBPETokenizer()
paths = ['./data/corpus.txt']
tokenizer.train(files=paths, vocab_size = 15000, min_frequency=6, special_tokens = ["[UNK]", "[CLS]", "[SEP]", "[PAD]", "[MASK]"])

tokenizer.save_model("./data/dic/")
```

For the other question, i trained it with 12G gpu memory，but it may be completed with samller gpu memory. And also you could adjust you parameters to your server environment.

@myechona Thanks for your code. I have a question about it.  There are some tasks like text-filling and sentence-permutation during pretrain stage,  i want to know whether the "input_ids"  is for masked sentence and the "labels" is for origin sentence?

If anyone wants to train their MBART model then feel free to use this.
https://github.com/prajdabre/yanmtt

Contributions are welcome!

> I and my co-worker wrote a demo according to roberta pretraining demo.
> 
> ```
> #encoding=utf-8
> 
> from transformers import (
>     BartForConditionalGeneration, BartTokenizer, BartForCausalLM,
>     Seq2SeqTrainingArguments, Seq2SeqTrainer
>   )
> 
> import torch
> from torch.utils.data import random_split
> 
> 
> # ## Initiating model and trainer for training
> from transformers import BartModel, BartConfig
> from transformers import BartTokenizerFast
> 
> 
> configuration = BartConfig(
>     vocab_size=52000,
>     max_position_embeddings=258,
>     d_model=256,
>     encoder_layers=3,
>     decoder_layers=3,
>     encoder_attention_heads=4,
>     decoder_attention_heads=4,
>     decoder_ffn_dim=1024,
>     encoder_ffn_dim=1024,
> )
> model = BartForCausalLM(configuration)
> tokenizer = BartTokenizerFast.from_pretrained("./dic", max_len=256, additional_special_tokens=['[CH]', '[OTHER]', '[VAR]', '[NUM]'])
> 
> 
> # ### HTTP Request DataPreparing & Modeling
> data = []
> with open("../data/sample.txt") as f1:
>     for src in f1:
>       data.append(
>           {
>               "seq2seq": {
>                   "input": src.strip()
>               }
>           }
>       )
> print(f'total size of data is {len(data)}')
> 
> 
> # splitting dataset into train, validation
> split = 0.2
> train_dataset, eval_dataset = random_split(data, lengths=[int((1-split)*len(data))+1, int(split*len(data))])
> 
> 
> # defining collator functioon for preparing batches on the fly ..
> def data_collator(features:list):
>    inputs = [f["seq2seq"]["input"] for f in features]
>    batch = tokenizer.prepare_seq2seq_batch(src_texts=inputs, max_length=256, padding='max_length')
>    batch["labels"] = batch["input_ids"].copy()
>    for k in batch:
>         batch[k] = torch.tensor(batch[k])
>    return batch
> 
> 
> batch_out = data_collator(eval_dataset)
> print(batch_out)
> print(batch_out['input_ids'].shape,batch_out['labels'].shape,batch_out['attention_mask'].shape)
> 
> 
> # defining training related arguments
> args = Seq2SeqTrainingArguments(output_dir="clm-checkpoints",
>                         do_train=True,
>                         do_eval=True,
>                         evaluation_strategy="epoch",
>                         per_device_train_batch_size=8,
>                         per_device_eval_batch_size=8,
>                         learning_rate=5e-5,
>                         num_train_epochs=1,
>                         logging_dir="./logs")
> 
> 
> # defining trainer using 🤗
> trainer = Seq2SeqTrainer(model=model, 
>                 args=args, 
>                 data_collator=data_collator, 
>                 train_dataset=train_dataset, 
>                 eval_dataset=eval_dataset)
> 
> 
> # ## Training time
> trainer.train()
> # It will take hours to train this model on this dataset
> 
> 
> # lets save model
> trainer.evaluate(eval_dataset=eval_dataset)
> trainer.save_model("clm-checkpoints")
> ```

Thanks for your code, it really helps.

I'm most interested in sentence infilling, which this script doesn't really seem to address (though my understanding was that BART training generally involves masking and permutation). Is there an additional step I need to add for the infilling functionality?


> We still need to provide a good docstring/notebook for this. It's on our ToDo-List. :-)
> 
> Or @sshleifer - is there already something for Bart?

Hi, any update on this? @vanpelt 


I actually decided to jump over to T5 and use the `run_t5_mlm_flax.py` script. Seems to be working so far, though it's very new, so missing some conveniences.... it sounds like that stuff is underway!

> I actually decided to jump over to T5 and use the `run_t5_mlm_flax.py` script. Seems to be working so far, though it's very new, so missing some conveniences.... it sounds like that stuff is underway!


Great, I was initially looking at those scripts to get some ideas about the pre-training script, but since then thought the Huggingface guys might have come up with a resource to do this. Apparently, it's still underway! :)



We've released [nanoT5](https://github.com/PiotrNawrot/nanoT5) that reproduces T5-model (similar to BART) pre-training in PyTorch (not Flax). 

You can take a look! 

Any suggestions are more than welcome.