Trouble with PL Checkpoint loading after finetuning bart-large 
Author: PingYu-iris
Pull request: no
Labels: name: wontfix

name: Summarization

name: Examples

Hi there,

I recently ran "finetune_bart.sh" with data from cnn_dm/train.source and saved checkpoints. I only have GPU with 12G memory, then I trained this model with batch_size 1 without distributed training.

I predicted test summaries results on cnn_dm/test.source with:

```python
        tokenizer = AutoTokenizer.from_pretrained(args.model_name_or_path)     
        checkpoints = list(sorted(glob.glob(os.path.join(args.output_dir, "checkpointepoch=*.ckpt"), recursive=True)))
        model = model.load_from_checkpoint(checkpoints[-1])
        model.eval()
        model.freeze()
        fout = Path("test_generation.txt").open("w", encoding="utf-8")
        for batch in tqdm(list(chunks(examples, 1))):
            inputs = tokenizer.batch_encode_plus([batch[0]], max_length=1024, return_tensors='pt')['input_ids']
            outputs = model.text_predictions(inputs)
            fout.write(outputs[0] + "\n")
            fout.flush()

        output_lns = [x.rstrip() for x in open("test_generation.txt").readlines()]
        reference_lns = [x.rstrip() for x in open("cnn_dm/test.target").readlines()]
        calculate_rouge(output_lns, reference_lns, "rouge_scores.txt")
```

After that I received results:
ROUGE_1: 
AggregateScore(low=Score(precision=0.428813534233686, recall=0.38720984475275855, fmeasure=0.3951340300593005), mid=Score(precision=0.4311467253592325, recall=0.38942524434531234, fmeasure=0.3970368964972586), high=Score(precision=0.4336709710273236, recall=0.3917389347240298, fmeasure=0.39884900115657695)) 

 ROUGE_2: 
AggregateScore(low=Score(precision=0.166422532946031, recall=0.14864999757241987, fmeasure=0.15229217101061118), mid=Score(precision=0.16833011753361882, recall=0.1504195716038362, fmeasure=0.15396139813339343), high=Score(precision=0.17058789969308358, recall=0.15232853455768658, fmeasure=0.15590467051587376)) 

 ROUGE_L: 
AggregateScore(low=Score(precision=0.2791458979499142, recall=0.2529897770384392, fmeasure=0.25757140506162507), mid=Score(precision=0.28120564591223207, recall=0.25488090117768447, fmeasure=0.2593799995678622), high=Score(precision=0.2834430254354239, recall=0.25683467237590635, fmeasure=0.2611850779055858)) 

This result is worse than the reported cnn_dm results in paper. I haven't changed any parameters yet. What is wrong with my training?



Comments:
How long did you train for? 

Our `finetune.py` is not identical to the authors' finetuning script and might produce different results, but rouge2=15.3 suggests that either you didn't train for very long, or there is a bug in your/our code.

What is the `model.text_predictions` method?





I trained about 3 days on single 12G Titan V GPU. 

```
    def text_predictions(self, input_ids):
        generated_ids = self.model.generate(
            input_ids=input_ids,
            num_beams=1,
            max_length=80,
            repetition_penalty=2.5,
            length_penalty=1.0,
            early_stopping=True,
        )
        preds = [
            self.tokenizer.decode(g, skip_special_tokens=True, clean_up_tokenization_spaces=True)
            for g in generated_ids
        ]
        return preds
```

This is my text_predictions function.

try 
```python
self.model.generate(input_ids=input_ids, attention_mask=attention_mask)
```
This will use the beam search kwargs from model.config, which are much better than those.



I changed my code:

```
    if args.do_predict:


        examples = [" " + x.rstrip() if "t5" in args.model_name_or_path else x.rstrip() for x in
                    open("cnn_dm/test.source").readlines()]

        tokenizer = AutoTokenizer.from_pretrained(args.model_name_or_path)

        model = model.load_from_checkpoint('bart_sum/checkpointepoch=2.ckpt').to(device)


        torch.save(model.state_dict(), args.output_dir + '/pytorch_model.bin')
        model.config.to_json_file(args.output_dir + '/config.json')

        model = BartForConditionalGeneration.from_pretrained('bart_sum').to(device)

        task_specific_params = model.config.task_specific_params
        if task_specific_params is not None:
            model.config.update(task_specific_params.get("summarization", {}))

        model.eval()

        fout = Path("test_generation.txt").open("w", encoding="utf-8")

        for batch in tqdm(list(chunks(examples, 1))):
            dct = tokenizer.batch_encode_plus(batch, max_length=1024, return_tensors="pt", pad_to_max_length=True).to(device)
            summaries = model.generate(**dct)

            dec = [tokenizer.decode(g, skip_special_tokens=True, clean_up_tokenization_spaces=False) for g in summaries]

            for hypothesis in dec:
                fout.write(hypothesis + "\n")
                fout.flush()
```

After that, I found generated sentences in dec is a random sentence. For example:
 Everybody Everybody Everybody SUM SUM SUM sergeant sergeant sergeant noisy noisy noisy Myster Myster MysterJewishJewishJewish sergeant sergeant talks talks talks noisy noisy imitate imitate imitate sergeant sergeantestonesestonesestones noisy noisy overhe overhe overhe Palmer Palmer Palmer noisy noisyJewishJewish spawned spawned spawned sergeant sergeant Height Height Heightnornornor For For For manif manif manif onboard onboard onboardsharingsharingestonesestones selects selects selects electors electors electors noisy noisy sergeant sergeantWAY Height Height selects selects sergeant sergeantJewishJewish Appearance Appearance Appearance Myster Myster selects selects fearing fearing fearing framed framed framed summ summ summ sergeant sergeant Pistol Pistol Pistol sergeant sergeant Appearance Appearance sergeant sergeant bees bees beesestonesestones Islamists Islamists Islamists sergeant sergeantselectselectselect sergeant sergeant spawned spawned manif manifestonesestonesirdirdird

I think my finetuned model is totally fail but I don't know what is wrong with my finetune model. I simply ran` finetune_bart.sh ` . The only change is setting training batch size to be 1.



Interesting. A Rouge2 score of ~15, which you had before, is a lot better than random, was that the same checkpoint?

Also, could you tell me the output for the following commands?

```bash
transformers-cli env
pip freeze | grep torch
```



This is a same checkpoint. I think my previous code has some problems. I wrote a function "text_predictions" in class SummarizationTrainer. This text_predictions function called `self.model.generate`. I think this self.model is from class BaseTransformer with `self.model = MODEL_MODES[mode].from_pretrained("facebook/bart-large)` instead of my saved checkpoint. Am I right?

with running: `transformers-cli env`

```
- `transformers` version: 2.11.0
- Platform: Linux-4.15.0-106-generic-x86_64-with-debian-stretch-sid
- Python version: 3.6.10
- PyTorch version (GPU?): 1.5.0 (True)
- Tensorflow version (GPU?): 2.2.0 (True)
- Using GPU in script?: <fill in>
- Using distributed or parallel set-up in script?: <fill in>
```
with running `pip freeze | grep torch`:

```
pytorch-lightning==0.7.6
torch==1.5.0
torchvision==0.6.0a0+82fd1c8
```



Yes that is correct. To reload a checkpoint, I have been using `resume_from_checkpoint` with good results. Also this should be fixed in the latest code, for future readers.

@williamFalcon in pl=0.7.6 is it possible to load a checkpoint without instantiating a trainer?
does `model = model.load_from_checkpoint('bart_sum/checkpointepoch=2.ckpt').to(device)` not work?

Then how could I generate samples from checkpoints? 

```
trainer.resume_from_checkpoint = checkpoints[-1]
trainer.logger.log_hyperparams(model.hparams)
trainer.test(model)
```
This only provides me with a loss value without generating samples from a checkpoint.

You can use `model.model.generate(input_ids)` if `model` is a `pl.Module`.

You can also save the transformers model, and then use `run_eval.py`

Something like:
`$ mkdir output_dir`

```python
from pathlib import Path
save_dir = Path('output_dir')
save_dir.mkdir(exist_ok=True)
model.model.save_pretrained(save_dir)
```

Then follow instructions [here](https://github.com/huggingface/transformers/blob/master/examples/seq2seq/README.md#evaluation-commands) for how to invoke `run_eval.py`




This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.