Question regarding training of BartForConditionalGeneration
Author: bnaman50
Pull request: no

Hello Guys,

I am trying to fine-tune the BART summarization model but due to the lack of big dataset, having some difficulties with the fine-tuning. 

Thus, I decided to look at the trainig process of BartForConditionalGeneration model in detail. I came across this article, [Introducing BART](https://sshleifer.github.io/blog_v2/jupyter/2020/03/12/bart.html) from one of the engineers, @sshleifer, at HuggingFace. It says that BartModel was directly fine-tuned for the summarisation task without **any new randomly initialized heads**.

**My question is about this fine-tuning process, especially on CNN-DailyMail dataset. Do you guys fine-tune the entire Bart model or only the decoder or something else?**

I looked at the example [fine-tuning script](https://github.com/huggingface/transformers/blob/master/examples/seq2seq/run_seq2seq.py) provided on the GitHub but I didn't find anything related to freezing some part of the model. 

&nbsp;

I also tried to look at the source code of the BartForConditionalGeneration model and observed the following - 

Its just adds a [linear layer](https://github.com/huggingface/transformers/blob/master/src/transformers/models/bart/modeling_bart.py#L1207) on top of the BartModel (copy-pasting the `__init__` code here for quick reference).

```
self.model = BartModel(config)
self.register_buffer("final_logits_bias", torch.zeros((1, self.model.shared.num_embeddings)))
self.lm_head = nn.Linear(config.d_model, self.model.shared.num_embeddings, bias=False)
```

At first, I thought these are the new parameters that are being introduced and thus, being trained. Therefore, I tried the following code to check the number of trainable parameters while keeping the endoer and decoder fixed  - 

```
from transformers import BartModel, BartForConditionalGeneration, BartTokenizer

def freeze_params(model):
    for par in model.parameters():
        par.requires_grad = False

model_sum = BartForConditionalGeneration.from_pretrained('facebook/bart-large')
freeze_params(model_sum.get_encoder()) ## freeze the encoder
freeze_params(model_sum.get_decoder()) ## freeze the decoder 

model_sum.train() ## set the train mode
train_p = [p for p in model_sum.parameters() if p.requires_grad] ## get the trainable params
print(f'Length of train params in Summarization Model : {len(train_p)}')
```

But this code shows that the list is empty. One thing I can do is to explictly set the `requires_grad=True` for the paramters in the `model_sum.lm_head` and only fine-tune these parameters. But I am curious to understand the original training/fine-tuning process. **It would be of great help to me if you guys could answer my question.** 

P.S. - Love the HuggingFace library. 

Thanks, 
<br /> 
Naman

Comments:
~Please ask on discuss.huggingface.co so that others can see your answer!~

Thanks for the prompt reply, @sshleifer. I wasn't aware of the discussion page. 
In that case, should I close the issue since it is just a query question?

Thanks, 
Naman

- I got confused. This is a reasonable place for this post. Sorry.
- Bart fine-tuned with no weights frozen, but this can be very slow.
`self.lm_head` is tied (the same parameter as) to self.encoder.embed_tokens and self.decoder.embed_tokens.
- I would recommend fine-tuning on your dataset with only the encoder frozen.
- `final-logits-bias` doesn't matter it's all 0s and frozen.

Thanks for your quick response. This is super-helpful to me. 

I have one more question related to the training process. 

My understanding is that BartModel (bart-base) was trained with two input sequences just like Google Bert (sadly, details are not given in the original paper. They only mention the difference in the objective function). On the other hand, BartForConditionalGeneration was trained (fine-tuned) for the summarization task i.e. single input and single input. 

In my current task, I am trying, to do *constraint summarization using multi-document*. Ideally, I would want my model to take **two different inputs** 
```
input_encodings = self.tokenizer.batch_encode_plus(list(zip(example_batch['inp1'], example_batch['inp2'])), padding='max_length', truncation=True)
```
but I feel this kind of fine-tuning will be harder given my smaller dataset and the change in training regime. 

The other option is simply **concatenating the two documents** and passing it as one sequence 
```
## comb_inp is a list of concatenated inputs from example_batch
input_encodings = self.tokenizer.batch_encode_plus(comb_inp, padding='max_length', truncation=True)
``` 

In your opinion, which one makes more sense? Your feedback and comments would be hugely appreciated in this case?

P.S. - I am using your distilled-bart model for fine-tuning since it is smaller version 😇

Thanks, 
Naman


- I would try concatenating. 
- I would also grid search evaluation parameters (min_length, max_length, num_beam, length_penalty). 
- I would evaluate a few distillbart/distill-pegasus variants before any fine-tuning to decide which to start from.


Hey @sshleifer, 

I was thinking of using the default hyper-parameters used during the training as a starting point. I could find the default settings for BART model ([here](https://github.com/pytorch/fairseq/blob/master/examples/bart/README.summarization.md)) but not for distilled BART. I even looked at the paper but the values are missing over there as well. Would it be possible for you to provide me/refer me with/to the default settings that you used to train the model. 

Thanks, 
Naman 

all the scripts have been moved to the [research_projects](https://github.com/huggingface/transformers/blob/master/examples/research_projects/seq2seq-distillation/) directory.

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md) are likely to be ignored.

Hey @sshleifer, I was trying to fine-tune the `distill-pegasus-cnn-16-4` [model](https://huggingface.co/sshleifer/distill-pegasus-cnn-16-4) provided by you but I am not sure of the hyper-parameters and the corresponding file is also not avaialble in the [`research_projects/seq2seq_distillation`](https://github.com/huggingface/transformers/tree/master/examples/research_projects/seq2seq-distillation) directory. Could you please share the hyper-parameters that you used to train this model (and achieve the results shown in Table 5 from your [paper](https://arxiv.org/pdf/2010.13002.pdf))?

Thanks a lot!
Naman

To clarify, you are trying to reproduce `distill-pegasus-cnn-16-4`, rather than finetune it further?


1) Make a student using make_student.py or get one from model hub.
2) guessing the train command from memory.
I think it should be a combination of `train_distilbart_cnn.sh` and `train_pegasus_xsum.sh`.
- You definitely want `max_target_length=142, --adafactor, --freeze-embeds --freeze-encoder`. 
- I think fp16 will not work.


So roughly...
```bash
python finetune.py \
    --learning_rate=1e-4 \
    --do_train \
    --do_predict \
    --n_val 1000 \
    --val_check_interval 0.25 \
    --max_target_length 142 \
    --freeze_embeds --label_smoothing 0.1 --adafactor \
	--eval_beams 2 \
	--freeze-encoder \
	--sortish_sampler \
	--model_name_or_path $YOUR_STUDENT \
    "$@"
```
3) Copy the `config.json` for `distill-pegasus-cnn-16-4` to increase rouge score of trained  student model.


Hope that helps, and sorry for not having a definitive answer.


Hey @sshleifer , 

Sorry, I think I phrased the question wrongly. I am trying to finetune the `distill-pegasus` model on my own dataset since the original pegasus model is huge and is taking a lot of time. I was simply hoping if you could provide me the hyper-parameters like you have provided for `distill-bart` models for CNN/DailyMail dataset ([here](https://github.com/huggingface/transformers/blob/master/examples/research_projects/seq2seq-distillation/train_distilbart_cnn.sh)).
Would I be able to use the hyper-parameters that you provided above?

Thanks, 
Naman

Yes

Hello, 

@[sshleifer](https://github.com/sshleifer) how can I freeze only the first two layer from the encoder (I am using BART), and how can I change the dropout of some layers also in BART?

Thank you

Hey @JessicaLopezEspejel , 

`model.get_encoder().layers` will give you a list (`torch.nn.modules.container.ModuleList` to be precise) of layers in encoder, and you can freeze the required layers using the `freeze_params` function provided in the `utils.py` file. I have included a small code snippet for your reference. Hope this helps!

```
from torch import nn
from transformers import AutoTokenizer, AutoModel

def freeze_params(model: nn.Module):
    """Set requires_grad=False for each of model.parameters()"""
    for par in model.parameters():
        par.requires_grad = False

model = AutoModel.from_pretrained("facebook/bart-large")
enc_layers = model.get_encoder().layers
freeze_params(enc_layers[0])  # freeze layer 0
dropout = enc_layers[0].dropout   # return dropout value for layer 0
enc_layers[0].dropout = 0.5  # set dropout value for layer 0
```

Thanks, 
Naman



Than you so much @bnaman50 , I will try it.