BART for Pre-Training
Author: swashiro
Pull request: no
Labels: name: Ex: LM (Pretraining)

name: Good Second Issue

# ❓ Questions & Help

How can I run BART pre-training?
I have data to pre-training(Masked LM)

Comments:
This should help: https://github.com/huggingface/transformers/issues/5096#issuecomment-645860271

@sshleifer - think this is the 3rd issue about Bart pre-training -> maybe it would be a good idea to release a small notebook at some point.

@patil-suraj you took a stab at this at some point? [this](https://github.com/huggingface/transformers/issues/5096#issuecomment-645848176) may have been optimistic :( 

Yes, I was trying to port fairseq dataset here, same for t5, I'll try to focus more on it when I'm done with current PRs, should strat with a notebook as Patrick said, then try to include it in examples/

@patrickvonplaten Does that mean I can train with Masked-input, input(label) and Decoder-input?

yes, this should be possible

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.


@patil-suraj any news on the pretraining script for Bart?

If anyone wants to train their MBART model then feel free to use this.
https://github.com/prajdabre/yanmtt

Contributions are welcome!

@patil-suraj excuse me, is there any news on the pretraining script for Bart? Thanks.

@thomas-li-sjtu you can try my toolkit if you like. It's based on transformers and allows for Bart/mbart pretraining. https://github.com/prajdabre/yanmtt

> @thomas-li-sjtu you can try my toolkit if you like. It's based on transformers and allows for Bart/mbart pretraining. https://github.com/prajdabre/yanmtt

Hi there, here is my problem. I hope to pretrain a bart model based on my own dataset and fine tune it for another task (not nmt). I noticed that your toolkit designs for nmt so maybe it is not the one I need. Anyway, thanks for your reply!

@thomas-li-sjtu ok I understand. It's not just designed for NMT (despite its name). I've used it for summarisation and general NLG without problems. Good luck with your search.

> @thomas-li-sjtu ok I understand. It's not just designed for NMT (despite its name). I've used it for summarisation and general NLG without problems. Good luck with your search.

Wow that is awesome. I will try it for my task!

@thomas-li-sjtu cool. Feel free to raise issues as it helps me add new functionality that may be of use to people. If you want to know how to use it for summarisation (or generic nlg) then look here: https://github.com/AI4Bharat/indic-bart

Sorry to only come back to this issue now.  If anyone is interested in adding this example script in `Transformers`, I would be more than happy to help :) 

For BART pre-training we need the text-infilling + sentence-permutation data collator which you could find here https://github.com/morganmcg1/rotobart/blob/main/data_collator.py#L223

With this collator you could then modify and use `run_summarization.py` script here https://github.com/huggingface/transformers/tree/master/examples/pytorch/summarization. 

Let me know if anyone is interested. :) cc @patrickvonplaten 



> Sorry to only come back to this issue now. If anyone is interested in adding this example script in `Transformers`, I would be more than happy to help :)
> 
> For BART pre-training we need the text-infilling + sentence-permutation data collator which you could find here https://github.com/morganmcg1/rotobart/blob/main/data_collator.py#L223
> 
> With this collator you could then modify and use `run_summarization.py` script here https://github.com/huggingface/transformers/tree/master/examples/pytorch/summarization.
> 
> Let me know if anyone is interested. :) cc @patrickvonplaten

I think the BART pre-training script is very useful for my work and many others. It is generous of you to add this example script in 'Transfromers' !!!

> Sorry to only come back to this issue now. If anyone is interested in adding this example script in `Transformers`, I would be more than happy to help :)
> 
> For BART pre-training we need the text-infilling + sentence-permutation data collator which you could find here https://github.com/morganmcg1/rotobart/blob/main/data_collator.py#L223
> 
> With this collator you could then modify and use `run_summarization.py` script here https://github.com/huggingface/transformers/tree/master/examples/pytorch/summarization.
> 
> Let me know if anyone is interested. :) cc @patrickvonplaten

Thanks for your reply and I think your method is absolutely feasible. But when I try it , I faced some errors that I can't fix. And could you please give me some help?
Here is my changes to `run_summarization.py`(tag 4.11.0)

1. Import some necessary packages in [https://github.com/morganmcg1/rotobart/blob/main/data_collator.py#L223](url)
2. Add full codes of `DataCollatorForDenoisingTasks` and also let class `DataCollatorForDenoisingTasks` inherit class `DataCollatorForSeq2Seq` in this way: `class DataCollatorForDenoisingTasks(DataCollatorForSeq2Seq):`
3. Use the new collator: `data_collator = DataCollatorForSeq2Seq(......)`   ->  `data_collator = DataCollatorForDenoisingTasks(.......)`

Run the changed script and I get errors below.

Traceback (most recent call last):
  File "/home/whq/anaconda3/envs/pytorchenv/lib/python3.7/site-packages/IPython/core/interactiveshell.py", line 3457, in run_code
    exec(code_obj, self.user_global_ns, self.user_ns)
  File "<ipython-input-2-991cbc10c55c>", line 1, in <module>
    runfile('/data/whq/tmp/SBartTry/fineBartPretrain.py', args=['--model_name_or_path', 'facebook/bart-base', '--do_train', '--do_eval', '--train_file', '/data/whq/tmp/SBartTry/tryData/clickbait_train.csv', '--validation_file', '/data/whq/tmp/SBartTry/tryData/clickbait_valid.csv', '--source_prefix', '', '--num_train_epochs=3', '--output_dir', '/data/whq/tmp/SBartTry/fineBartPretrain/clickbait', '--overwrite_output_dir', '--per_device_train_batch_size=16', '--per_device_eval_batch_size=16', '--predict_with_generate'], wdir='/data/whq/tmp/SBartTry')
  File "/home/whq/.pycharm_helpers/pydev/_pydev_bundle/pydev_umd.py", line 198, in runfile
    pydev_imports.execfile(filename, global_vars, local_vars)  # execute the script
  File "/home/whq/.pycharm_helpers/pydev/_pydev_imps/_pydev_execfile.py", line 18, in execfile
    exec(compile(contents+"\n", file, 'exec'), glob, loc)
  File "/data/whq/tmp/SBartTry/fineBartPretrain.py", line 823, in <module>
    main()
  File "/data/whq/tmp/SBartTry/fineBartPretrain.py", line 745, in main
    train_result = trainer.train(resume_from_checkpoint=checkpoint)
  File "/home/whq/anaconda3/envs/pytorchenv/lib/python3.7/site-packages/transformers/trainer.py", line 1325, in train
    tr_loss_step = self.training_step(model, inputs)
  File "/home/whq/anaconda3/envs/pytorchenv/lib/python3.7/site-packages/transformers/trainer.py", line 1884, in training_step
    loss = self.compute_loss(model, inputs)
  File "/home/whq/anaconda3/envs/pytorchenv/lib/python3.7/site-packages/transformers/trainer.py", line 1916, in compute_loss
    outputs = model(**inputs)
  File "/home/whq/anaconda3/envs/pytorchenv/lib/python3.7/site-packages/torch/nn/modules/module.py", line 1102, in _call_impl
    return forward_call(*input, **kwargs)
  File "/home/whq/anaconda3/envs/pytorchenv/lib/python3.7/site-packages/torch/nn/parallel/data_parallel.py", line 168, in forward
    outputs = self.parallel_apply(replicas, inputs, kwargs)
  File "/home/whq/anaconda3/envs/pytorchenv/lib/python3.7/site-packages/torch/nn/parallel/data_parallel.py", line 178, in parallel_apply
    return parallel_apply(replicas, inputs, kwargs, self.device_ids[:len(replicas)])
  File "/home/whq/anaconda3/envs/pytorchenv/lib/python3.7/site-packages/torch/nn/parallel/parallel_apply.py", line 86, in parallel_apply
    output.reraise()
  File "/home/whq/anaconda3/envs/pytorchenv/lib/python3.7/site-packages/torch/_utils.py", line 434, in reraise
    raise exception
TypeError: Caught TypeError in replica 0 on device 0.
Original Traceback (most recent call last):
  File "/home/whq/anaconda3/envs/pytorchenv/lib/python3.7/site-packages/torch/nn/parallel/parallel_apply.py", line 61, in _worker
    output = module(*input, **kwargs)
  File "/home/whq/anaconda3/envs/pytorchenv/lib/python3.7/site-packages/torch/nn/modules/module.py", line 1102, in _call_impl
    return forward_call(*input, **kwargs)
  File "/home/whq/anaconda3/envs/pytorchenv/lib/python3.7/site-packages/transformers/models/bart/modeling_bart.py", line 1336, in forward
    return_dict=return_dict,
  File "/home/whq/anaconda3/envs/pytorchenv/lib/python3.7/site-packages/torch/nn/modules/module.py", line 1102, in _call_impl
    return forward_call(*input, **kwargs)
  File "/home/whq/anaconda3/envs/pytorchenv/lib/python3.7/site-packages/transformers/models/bart/modeling_bart.py", line 1200, in forward
    return_dict=return_dict,
  File "/home/whq/anaconda3/envs/pytorchenv/lib/python3.7/site-packages/torch/nn/modules/module.py", line 1102, in _call_impl
    return forward_call(*input, **kwargs)
  File "/home/whq/anaconda3/envs/pytorchenv/lib/python3.7/site-packages/transformers/models/bart/modeling_bart.py", line 769, in forward
    input_shape = input_ids.size()
TypeError: 'int' object is not callable

Waiting for your generous reply! @patil-suraj 

@Eurus-W make sure you convert the numpy arrays in the batch returned by `data_collator()` into tensors.
`batch["input_ids"] = torch.LongTensor(batch["input_ids"])`, for example.