Seq2Seq: same MultiGPU test failing twice!
Author: sshleifer
Pull request: no

This is not run by CI, but if you go on a machine with multiple GPU, you will see an interesting unittest failure:

```bash
pytest examples -k multigpu
```


```
===================================== FAILURES ======================================
_____________________ TestSummarizationDistiller.test_multigpu ______________________

self = <seq2seq.test_seq2seq_examples.TestSummarizationDistiller testMethod=test_multigpu>

    @require_multigpu
    def test_multigpu(self):
        updates = dict(no_teacher=True, freeze_encoder=True, gpus=2, sortish_sampler=False,)
>       self._test_distiller_cli(updates)

examples/seq2seq/test_seq2seq_examples.py:117:
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
examples/seq2seq/test_seq2seq_examples.py:182: in _test_distiller_cli
    self.assertIn(ckpt_name, contents)
E   AssertionError: 'val_avg_rouge2=0.0000-step_count=2.ckpt' not found in {'metrics.json', 'hparams.pk
l', 'git_log.json'}
============================== short test summary info ==============================
FAILED examples/seq2seq/test_seq2seq_examples.py::TestSummarizationDistiller::test_multigpu
========================= 1 failed, 25 deselected in 9.75s ==========================
...
 'step_count': 4,
 'test_avg_gen_time': 0.9470310211181641,
 'test_avg_loss': 10.737613677978516,
 'test_avg_rouge1': 0.0,
 'test_avg_rouge2': 0.0,
 'test_avg_rougeL': 0.0,
 'test_avg_summ_len': 141.0,
 'test_loss': tensor(10.7376, device='cuda:0'),
 'test_rouge2': tensor(0., device='cuda:0')}
--------------------------------------------------------------------------------
Testing: 100%|█████████████████████████████████████████| 1/1 [00:01<00:00,  1.49s/it]
F

===================================== FAILURES ======================================
_____________________ TestSummarizationDistiller.test_multigpu ______________________

self = <seq2seq.test_seq2seq_examples.TestSummarizationDistiller testMethod=test_multigpu>

    @require_multigpu
    def test_multigpu(self):
        updates = dict(no_teacher=True, freeze_encoder=True, gpus=2, sortish_sampler=False,)
>       self._test_distiller_cli(updates)

examples/seq2seq/test_seq2seq_examples.py:117:
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
examples/seq2seq/test_seq2seq_examples.py:193: in _test_distiller_cli
    self.assertEqual(len(metrics["val"]), desired_n_evals)
E   AssertionError: 3 != 5
============================== short test summary info ==============================
FAILED examples/seq2seq/test_seq2seq_examples.py::TestSummarizationDistiller::test_multigpu
========================= 1 failed, 25 deselected in 47.48s =========================
```

The same `SummarizationDistiller.test_multigpu` fails twice. I think we need to use `pl.Trainer.argparse.from_args` in the unittests.
How do you guys do it @nateraw ?

Comments:
Spent an hour testing the new version on real data:

(1) `./train_mbart_enro_multigpu` works!
(2) `./train_mbart_enro_multigpu --logger_name wandb` ~doesn't work~ works.
(3) trainer.test seems to try to call `main` again, and then runs into an "output directory" already exists error.

It hangs for me with versions:
```bash
pytorch-lightning==0.8.5
torch==1.5.1+cu101
```

cc @nateraw @williamFalcon 

@stas00 do you think multi-gpu test coverage (at least on machines with multiple GPU, not CI yet) is possible?

> @stas00 do you think multi-gpu test coverage (at least on machines with multiple GPU, not CI yet) is possible?

Are you referring to `pytest --cov`? Yes, of course, it should work.

The main issue with multiprocessing and coverage is a potential unregistered coverage if the sub-process hasn't finished its work right away and is hanging around. But we already have this issue in the benchmarking tests - I need to go back and pound at it some more.

Sorry, I'm referring to getting that test to pass.
At the moment it fails (twice!).

Oh, I see what you meant. Let me investigate.

Yeah, multigpu is a problem. Here is a totally different manifestation of it - hanging and other subtests failing:

```
pytest --disable-warnings examples/seq2seq/test_seq2seq_examples.py::TestSummarizationDistiller
――――――――――――――――――――――――――――――――――――――――― TestSummarizationDistiller.test_distill_checkpointing_with_teacher ―――――――――――――――――――――――――――――――――――――――――

self = <seq2seq.test_seq2seq_examples.TestSummarizationDistiller testMethod=test_distill_checkpointing_with_teacher>

    def test_distill_checkpointing_with_teacher(self):
        updates = dict(
            student_encoder_layers=2,
            student_decoder_layers=1,
            max_epochs=4,
            val_check_interval=0.25,
            alpha_hid=2.0,
            model_name_or_path="IGNORE_THIS_IT_DOESNT_GET_USED",
        )
        model = self._test_distiller_cli(updates, check_contents=False)
    
        ckpts = list(Path(model.output_dir).glob("*.ckpt"))
>       self.assertEqual(1, len(ckpts))
E       AssertionError: 1 != 0

examples/seq2seq/test_seq2seq_examples.py:173: AssertionError

 examples/seq2seq/test_seq2seq_examples.py::TestSummarizationDistiller.test_distill_checkpointing_with_teacher ⨯                        14% █▌        using module BartTranslationDistiller
Epoch 2: 100%|████████████████████████████████████████████████████████████████████████████████████| 4/4 [00:00<00:00, 11.53it/s, loss=2.970, v_num=19]
                              

――――――――――――――――――――――――――――――――――――――――――――――――――― TestSummarizationDistiller.test_distill_mbart ――――――――――――――――――――――――――――――――――――――――――――――――――――

self = <seq2seq.test_seq2seq_examples.TestSummarizationDistiller testMethod=test_distill_mbart>

    def test_distill_mbart(self):
        updates = dict(
            student_encoder_layers=2,
            student_decoder_layers=1,
            num_train_epochs=4,
            val_check_interval=0.25,
            alpha_hid=2.0,
            task="translation",
            model_name_or_path="IGNORE_THIS_IT_DOESNT_GET_USED",
            tokenizer_name=MBART_TINY,
            teacher=MBART_TINY,
            src_lang="en_XX",
            tgt_lang="ro_RO",
        )
        model = self._test_distiller_cli(updates, check_contents=False)
        assert model.model.config.model_type == "mbart"
    
        ckpts = list(Path(model.output_dir).glob("*.ckpt"))
>       self.assertEqual(1, len(ckpts))
E       AssertionError: 1 != 0

examples/seq2seq/test_seq2seq_examples.py:224: AssertionError

 examples/seq2seq/test_seq2seq_examples.py::TestSummarizationDistiller.test_distill_mbart ⨯                                             29% ██▉       using module SummarizationModule
Epoch 2: 100%|███████████████████████████████████████████████████████████████████████████████████| 4/4 [00:00<00:00,  4.23it/s, loss=10.785, v_num=19]
                                                                                                                                                      

――――――――――――――――――――――――――――――――――――――――――――――――― TestSummarizationDistiller.test_distill_no_teacher ―――――――――――――――――――――――――――――――――――――――――――――――――

self = <seq2seq.test_seq2seq_examples.TestSummarizationDistiller testMethod=test_distill_no_teacher>

    def test_distill_no_teacher(self):
        updates = dict(student_encoder_layers=2, student_decoder_layers=1, no_teacher=True)
>       self._test_distiller_cli(updates)

examples/seq2seq/test_seq2seq_examples.py:159: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

self = <seq2seq.test_seq2seq_examples.TestSummarizationDistiller testMethod=test_distill_no_teacher>
updates = {'no_teacher': True, 'student_decoder_layers': 1, 'student_encoder_layers': 2}, check_contents = True

    def _test_distiller_cli(self, updates, check_contents=True):
        default_updates = dict(
            label_smoothing=0.0,
            early_stopping_patience=-1,
            train_batch_size=1,
            eval_batch_size=2,
            max_epochs=2,
            alpha_mlm=0.2,
            alpha_ce=0.8,
            do_predict=True,
            model_name_or_path="sshleifer/tinier_bart",
            teacher=CHEAP_ARGS["model_name_or_path"],
            val_check_interval=0.5,
            alpha_encoder_loss=0.4,
        )
        default_updates.update(updates)
        args_d: dict = CHEAP_ARGS.copy()
        tmp_dir = make_test_data_dir()
        output_dir = tempfile.mkdtemp(prefix="output_")

        args_d.update(data_dir=tmp_dir, output_dir=output_dir, **default_updates)
        model = distill_main(argparse.Namespace(**args_d))
        if not check_contents:
            return model
        contents = os.listdir(output_dir)
        contents = {os.path.basename(p) for p in contents}
        ckpt_files = [p for p in contents if p.endswith("ckpt")]
>       assert len(ckpt_files) > 0
E       AssertionError: assert 0 > 0
E        +  where 0 = len([])

examples/seq2seq/test_seq2seq_examples.py:271: AssertionError

 examples/seq2seq/test_seq2seq_examples.py::TestSummarizationDistiller.test_distill_no_teacher ⨯                                        43% ████▍
 examples/seq2seq/test_seq2seq_examples.py::TestSummarizationDistiller.test_distill_t5 s                                                57% █████▊
 ../../../../../home/stas/anaconda3/envs/main-38/lib/python3.8/unittest/case.py::TestSummarizationDistiller.test_hub_configs s          71% ███████▎
 examples/seq2seq/test_seq2seq_examples.py::TestSummarizationDistiller.test_loss_fn ✓                                                   86% ████████▋
 examples/seq2seq/test_seq2seq_examples.py::TestSummarizationDistiller.test_multigpu s                                                 100% ██████████
============================================================== short test summary info ===============================================================
FAILED examples/seq2seq/test_seq2seq_examples.py::TestSummarizationDistiller::test_distill_checkpointing_with_teacher - AssertionError: 1 != 0
FAILED examples/seq2seq/test_seq2seq_examples.py::TestSummarizationDistiller::test_distill_mbart - AssertionError: 1 != 0
FAILED examples/seq2seq/test_seq2seq_examples.py::TestSummarizationDistiller::test_distill_no_teacher - AssertionError: assert 0 > 0

Results (18.02s):
       1 passed
       3 failed
         - examples/seq2seq/test_seq2seq_examples.py:161 TestSummarizationDistiller.test_distill_checkpointing_with_teacher
         - examples/seq2seq/test_seq2seq_examples.py:206 TestSummarizationDistiller.test_distill_mbart
         - examples/seq2seq/test_seq2seq_examples.py:157 TestSummarizationDistiller.test_distill_no_teacher
       3 skipped
```
and it hangs. getting a trace:
```
Thread 0x00007fc81efd6700 (most recent call first):
  File "/home/stas/anaconda3/envs/main-38/lib/python3.8/threading.py", line 306 in wait
  File "/home/stas/anaconda3/envs/main-38/lib/python3.8/queue.py", line 179 in get
  File "/home/stas/anaconda3/envs/main-38/lib/python3.8/site-packages/tensorboard/summary/writer/event_file_writer.py", line 232 in run
  File "/home/stas/anaconda3/envs/main-38/lib/python3.8/threading.py", line 932 in _bootstrap_inner
  File "/home/stas/anaconda3/envs/main-38/lib/python3.8/threading.py", line 890 in _bootstrap

Thread 0x00007fc80e850700 (most recent call first):
  File "/home/stas/anaconda3/envs/main-38/lib/python3.8/threading.py", line 306 in wait
  File "/home/stas/anaconda3/envs/main-38/lib/python3.8/queue.py", line 179 in get
  File "/home/stas/anaconda3/envs/main-38/lib/python3.8/site-packages/tensorboard/summary/writer/event_file_writer.py", line 232 in run
  File "/home/stas/anaconda3/envs/main-38/lib/python3.8/threading.py", line 932 in _bootstrap_inner
  File "/home/stas/anaconda3/envs/main-38/lib/python3.8/threading.py", line 890 in _bootstrap

Thread 0x00007fc8248b0700 (most recent call first):
  File "/home/stas/anaconda3/envs/main-38/lib/python3.8/threading.py", line 306 in wait
  File "/home/stas/anaconda3/envs/main-38/lib/python3.8/queue.py", line 179 in get
  File "/home/stas/anaconda3/envs/main-38/lib/python3.8/site-packages/tensorboard/summary/writer/event_file_writer.py", line 232 in run
  File "/home/stas/anaconda3/envs/main-38/lib/python3.8/threading.py", line 932 in _bootstrap_inner
  File "/home/stas/anaconda3/envs/main-38/lib/python3.8/threading.py", line 890 in _bootstrap

Thread 0x00007fc8250b1700 (most recent call first):
  File "/home/stas/anaconda3/envs/main-38/lib/python3.8/threading.py", line 306 in wait
  File "/home/stas/anaconda3/envs/main-38/lib/python3.8/queue.py", line 179 in get
  File "/home/stas/anaconda3/envs/main-38/lib/python3.8/site-packages/tensorboard/summary/writer/event_file_writer.py", line 232 in run
  File "/home/stas/anaconda3/envs/main-38/lib/python3.8/threading.py", line 932 in _bootstrap_inner
  File "/home/stas/anaconda3/envs/main-38/lib/python3.8/threading.py", line 890 in _bootstrap

Thread 0x00007fc81cbec700 (most recent call first):
<no Python frame>

Thread 0x00007fc81e7d5700 (most recent call first):
<no Python frame>

Thread 0x00007fc826ffd700 (most recent call first):
  File "/home/stas/anaconda3/envs/main-38/lib/python3.8/threading.py", line 306 in wait
  File "/home/stas/anaconda3/envs/main-38/lib/python3.8/queue.py", line 179 in get
  File "/home/stas/anaconda3/envs/main-38/lib/python3.8/site-packages/tensorboard/summary/writer/event_file_writer.py", line 232 in run
  File "/home/stas/anaconda3/envs/main-38/lib/python3.8/threading.py", line 932 in _bootstrap_inner
  File "/home/stas/anaconda3/envs/main-38/lib/python3.8/threading.py", line 890 in _bootstrap

Current thread 0x00007fc91d2e2740 (most recent call first):
  File "/home/stas/anaconda3/envs/main-38/lib/python3.8/site-packages/torch/distributed/rendezvous.py", line 172 in _env_rendezvous_handler
  File "/home/stas/anaconda3/envs/main-38/lib/python3.8/site-packages/torch/distributed/distributed_c10d.py", line 423 in init_process_group
  File "/home/stas/anaconda3/envs/main-38/lib/python3.8/site-packages/pytorch_lightning/core/lightning.py", line 973 in init_ddp_connection
  File "/home/stas/anaconda3/envs/main-38/lib/python3.8/site-packages/pytorch_lightning/trainer/distrib_data_parallel.py", line 506 in ddp_train
  File "/home/stas/anaconda3/envs/main-38/lib/python3.8/site-packages/pytorch_lightning/trainer/distrib_data_parallel.py", line 462 in spawn_ddp_children
  File "/home/stas/anaconda3/envs/main-38/lib/python3.8/site-packages/pytorch_lightning/trainer/trainer.py", line 992 in fit
  File "/mnt/nvme1/code/huggingface/transformers-master/examples/lightning_base.py", line 380 in generic_train
  File "/mnt/nvme1/code/huggingface/transformers-master/examples/seq2seq/finetune.py", line 420 in main
  File "/mnt/nvme1/code/huggingface/transformers-master/examples/seq2seq/distillation.py", line 511 in distill_main
  File "/mnt/nvme1/code/huggingface/transformers-master/examples/seq2seq/test_seq2seq_examples.py", line 267 in _test_distiller_cli
  File "/mnt/nvme1/code/huggingface/transformers-master/examples/seq2seq/test_seq2seq_examples.py", line 157 in test_multigpu
  File "/home/stas/anaconda3/envs/main-38/lib/python3.8/unittest/case.py", line 633 in _callTestMethod
  File "/home/stas/anaconda3/envs/main-38/lib/python3.8/unittest/case.py", line 676 in run
  File "/home/stas/anaconda3/envs/main-38/lib/python3.8/unittest/case.py", line 736 in __call__
  File "/home/stas/anaconda3/envs/main-38/lib/python3.8/site-packages/_pytest/unittest.py", line 278 in runtest
  File "/home/stas/anaconda3/envs/main-38/lib/python3.8/site-packages/_pytest/runner.py", line 153 in pytest_runtest_call
  File "/home/stas/anaconda3/envs/main-38/lib/python3.8/site-packages/pluggy/callers.py", line 187 in _multicall
  File "/home/stas/anaconda3/envs/main-38/lib/python3.8/site-packages/pluggy/manager.py", line 84 in <lambda>
  File "/home/stas/anaconda3/envs/main-38/lib/python3.8/site-packages/pluggy/manager.py", line 93 in _hookexec
  File "/home/stas/anaconda3/envs/main-38/lib/python3.8/site-packages/pluggy/hooks.py", line 286 in __call__
  File "/home/stas/anaconda3/envs/main-38/lib/python3.8/site-packages/_pytest/runner.py", line 247 in <lambda>
  File "/home/stas/anaconda3/envs/main-38/lib/python3.8/site-packages/_pytest/runner.py", line 294 in from_call
  File "/home/stas/anaconda3/envs/main-38/lib/python3.8/site-packages/_pytest/runner.py", line 246 in call_runtest_hook
  File "/home/stas/anaconda3/envs/main-38/lib/python3.8/site-packages/_pytest/runner.py", line 207 in call_and_report
  File "/home/stas/anaconda3/envs/main-38/lib/python3.8/site-packages/_pytest/runner.py", line 117 in runtestprotocol
  File "/home/stas/anaconda3/envs/main-38/lib/python3.8/site-packages/_pytest/runner.py", line 100 in pytest_runtest_protocol
  File "/home/stas/anaconda3/envs/main-38/lib/python3.8/site-packages/pluggy/callers.py", line 187 in _multicall
  File "/home/stas/anaconda3/envs/main-38/lib/python3.8/site-packages/pluggy/manager.py", line 84 in <lambda>
  File "/home/stas/anaconda3/envs/main-38/lib/python3.8/site-packages/pluggy/manager.py", line 93 in _hookexec
  File "/home/stas/anaconda3/envs/main-38/lib/python3.8/site-packages/pluggy/hooks.py", line 286 in __call__
  File "/home/stas/anaconda3/envs/main-38/lib/python3.8/site-packages/_pytest/main.py", line 321 in pytest_runtestloop
  File "/home/stas/anaconda3/envs/main-38/lib/python3.8/site-packages/pluggy/callers.py", line 187 in _multicall
  File "/home/stas/anaconda3/envs/main-38/lib/python3.8/site-packages/pluggy/manager.py", line 84 in <lambda>
  File "/home/stas/anaconda3/envs/main-38/lib/python3.8/site-packages/pluggy/manager.py", line 93 in _hookexec
  File "/home/stas/anaconda3/envs/main-38/lib/python3.8/site-packages/pluggy/hooks.py", line 286 in __call__
  File "/home/stas/anaconda3/envs/main-38/lib/python3.8/site-packages/_pytest/main.py", line 296 in _main
  File "/home/stas/anaconda3/envs/main-38/lib/python3.8/site-packages/_pytest/main.py", line 240 in wrap_session
  File "/home/stas/anaconda3/envs/main-38/lib/python3.8/site-packages/_pytest/main.py", line 289 in pytest_cmdline_main
  File "/home/stas/anaconda3/envs/main-38/lib/python3.8/site-packages/pluggy/callers.py", line 187 in _multicall
  File "/home/stas/anaconda3/envs/main-38/lib/python3.8/site-packages/pluggy/manager.py", line 84 in <lambda>
  File "/home/stas/anaconda3/envs/main-38/lib/python3.8/site-packages/pluggy/manager.py", line 93 in _hookexec
  File "/home/stas/anaconda3/envs/main-38/lib/python3.8/site-packages/pluggy/hooks.py", line 286 in __call__
  File "/home/stas/anaconda3/envs/main-38/lib/python3.8/site-packages/_pytest/config/__init__.py", line 157 in main
  File "/home/stas/anaconda3/envs/main-38/lib/python3.8/site-packages/_pytest/config/__init__.py", line 180 in console_main
  File "/home/stas/anaconda3/envs/main-38/bin/pytest", line 8 in <module>
```

If I run just these 3 tests - all is green. So its the multigpu test.

Ah, it somehow starts running pytest second time half-way through.

Rechecked with py37 and py38 - same, same.

p.s. checkout `pip install pytest-sugar` - makes running tests a brighter experience.

I will try to make a small reproducible test where the problem still happens, and then we can try to sort it out.

OK, so first there is some strange issue with the properties - seems to be related to https://github.com/pytorch/pytorch/issues/43542, the multigpu test fails with:
```

―――――――――――――――――――――――――――――――――――――――――――――――――――――― TestSummarizationDistiller.test_multigpu ――――――――――――――――――――――――――――――――――――――――――――――――――――――

self = <seq2seq.test_seq2seq_examples.TestSummarizationDistiller testMethod=test_multigpu>

    @require_multigpu
    def test_multigpu(self):
        updates = dict(
            no_teacher=True,
            freeze_encoder=True,
            gpus=2,
            sortish_sampler=True,
        )
>       self._test_distiller_cli(updates, check_contents=False)

examples/seq2seq/test_seq2seq_examples.py:154: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
examples/seq2seq/test_seq2seq_examples.py:264: in _test_distiller_cli
    model = distill_main(argparse.Namespace(**args_d))
examples/seq2seq/distillation.py:495: in distill_main
    return ft_main(args, model=model)
examples/seq2seq/finetune.py:399: in main
    trainer: pl.Trainer = generic_train(
examples/lightning_base.py:380: in generic_train
    trainer.fit(model)
../../github/00pytorch/pytorch-lightning/pytorch_lightning/trainer/trainer.py:301: in fit
    results = self.accelerator_backend.train()
../../github/00pytorch/pytorch-lightning/pytorch_lightning/accelerators/ddp_backend.py:133: in train
    self.ddp_train_tmp(process_idx=self.task_idx, mp_queue=None, model=model)
../../github/00pytorch/pytorch-lightning/pytorch_lightning/accelerators/ddp_base_backend.py:148: in ddp_train_tmp
    optimizers, lr_schedulers, optimizer_frequencies = self.trainer.init_optimizers(model)
../../github/00pytorch/pytorch-lightning/pytorch_lightning/trainer/optimizers.py:32: in init_optimizers
    optim_conf = model.configure_optimizers()
examples/lightning_base.py:152: in configure_optimizers
    scheduler = self.get_lr_scheduler()
examples/lightning_base.py:122: in get_lr_scheduler
    self.opt, num_warmup_steps=self.hparams.warmup_steps, num_training_steps=self.total_steps
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

self = SummarizationModule(
  (model): BartForConditionalGeneration(
    (model): BartModel(
      (shared): Embedding(50265,...    )
        )
        (layernorm_embedding): LayerNorm((24,), eps=1e-05, elementwise_affine=True)
      )
    )
  )
)
name = 'total_steps'

    def __getattr__(self, name: str) -> Union[Tensor, 'Module']:
        if '_parameters' in self.__dict__:
            _parameters = self.__dict__['_parameters']
            if name in _parameters:
                return _parameters[name]
        if '_buffers' in self.__dict__:
            _buffers = self.__dict__['_buffers']
            if name in _buffers:
                return _buffers[name]
        if '_modules' in self.__dict__:
            modules = self.__dict__['_modules']
            if name in modules:
                return modules[name]
>       raise ModuleAttributeError("'{}' object has no attribute '{}'".format(
            type(self).__name__, name))
E       torch.nn.modules.module.ModuleAttributeError: 'SummarizationModule' object has no attribute 'total_steps'
```
I fixed it with:
```
diff --git a/examples/lightning_base.py b/examples/lightning_base.py
index e7c41a3e..f5e858c7 100644
--- a/examples/lightning_base.py
+++ b/examples/lightning_base.py
@@ -119,8 +119,7 @@ class BaseTransformer(pl.LightningModule):
     def get_lr_scheduler(self):
         get_schedule_func = arg_to_scheduler[self.hparams.lr_scheduler]
         scheduler = get_schedule_func(
-            self.opt, num_warmup_steps=self.hparams.warmup_steps, num_training_steps=self.total_steps
-        )
+            self.opt, num_warmup_steps=self.hparams.warmup_steps, num_training_steps=self.total_steps())
         scheduler = {"scheduler": scheduler, "interval": "step", "frequency": 1}
         return scheduler

@@ -159,12 +158,11 @@ class BaseTransformer(pl.LightningModule):
     def test_epoch_end(self, outputs):
         return self.validation_end(outputs)

-    @property
     def total_steps(self) -> int:
         """The number of total training steps that will be run. Used for lr scheduler purposes."""
         num_devices = max(1, self.hparams.gpus)  # TODO: consider num_tpu_cores
         effective_batch_size = self.hparams.train_batch_size * self.hparams.accumulate_grad_batches * num_devices
-        dataset_size = len(self.train_loader.dataset)
+        dataset_size = len(self.train_dataloader().dataset)
         return (dataset_size / effective_batch_size) * self.hparams.max_epochs

     def setup(self, mode):
```

it works fine the first time, but when it re-reruns itself, it behaves differently - so the above fix was needed.

WIP: https://github.com/huggingface/transformers/pull/7281