Premature DataLoader completion when sequentially specifying test/val dataloaders
🐛 Bug
Running:
trainer.test(model, test_dataloaders=testA)
trainer.test(model, test_dataloaders=testB)

The first test line executes according to expectation. The second test line runs testB, but only the number of batches defined by testA. In fact, if the PL Module has test_dataloaders method defined, this is also the same behavior.
The root cause is that the trainer.num_test_batches member variable is only ever set on one line in the whole code base: 
  
    
      pytorch-lightning/pytorch_lightning/trainer/data_loading.py
    
    
         Line 268
      in
      981169c
    
  
  
    

        
          
           self.num_test_batches, self.test_dataloaders =\ 
        
    
  


The method that holds that line, reset_test_dataloader, is invoked in trainer.run_evaluation from trainer.test, but it is Only called if there currently is NO test dataloader associated with the trainer already: 
  
    
      pytorch-lightning/pytorch_lightning/trainer/evaluation_loop.py
    
    
         Line 344
      in
      7c7e50c
    
  
  
    

        
          
           self.reset_test_dataloader(model) 
        
    
  


Same with val_dataloaders.
To remedy this, trainer.num_test_batches needs to be updated whenever there is a new dataloader assignment to the trainer: 
  
    
      pytorch-lightning/pytorch_lightning/trainer/evaluation_loop.py
    
    
         Line 346
      in
      7c7e50c
    
  
  
    

        
          
           dataloaders = self.test_dataloaders 
        
    
  


Similarly w/ val_dataloader.
To Reproduce
Steps to reproduce the behavior:

define dataloader testA such that len(testA) = n
define dataloader testB such that len(testB) = m, where m>n
run trainer.test(model, test_dataloaders=testA), observe the number of batches in the progress bar. it will report n
run trainer.test(model, test_dataloaders=testB), observe the number of batches in the progress bar. it will report n

Expected behavior

The last step should run the full m number of batches in testB

Environment

CUDA:

GPU:

GeForce RTX 2080 Ti
GeForce RTX 2080 Ti


available:         True
version:           10.1


Packages:

numpy:             1.16.4
pyTorch_debug:     False
pyTorch_version:   1.4.0
pytorch-lightning: 0.7.6
tensorboard:       2.2.1
tqdm:              4.45.0


System:

OS:                Linux
architecture:

64bit
ELF


processor:         x86_64
python:            3.6.7
version:           #38-Ubuntu SMP Fri Mar 27 17:37:05 UTC 2020