GPU memory usage/leak varies between experiments/trials with Apex (use_amp=True) 
Using use_amp=True with hyper parameter searching results in increased GPU memory usage between experiments. So it'll go from initially 7GB usage to maybe 10GB usage between learning rate trials and then vary between that.
For example:
parent_parser = HyperOptArgumentParser(strategy='grid_search', add_help=False)
    
parent_parser.opt_list('--lr', default = 0.01, type=float, tunable=True, options = [0.00005, 0.00006, 0.00007, 0.00008, 0.00009, 0.0001, 0.00011, 0.00012, 0.00013, 0.00014, 0.00015])
    
    hyperparams = parent_parser.parse_args()

    for hparam_trial in hyperparams.trials(20):
        print(hparam_trial)
        main_local(hparam_trial)   
        print("NEXT TRIAL")

def main_local(hparams):

    
    model = Model(hparams)
    expname = "HPSEARCH" +  '_' + str(hparams.lr) + '_' + str(hparams.batch_size)
    tt_logger = TestTubeLogger(save_dir="/",name=expname,debug=False,create_git_tag=False)
    
    trainer = Trainer(logger = tt_logger , use_amp=True, amp_level='O2' , gpus=1,min_nb_epochs=40, max_nb_epochs=41)
      
    torch.cuda.empty_cache()


PyTorch Version: 1.3.1
Python version: 3.7.5
CUDA/cuDNN version: Driver Version: 440.33.01    CUDA Version: 10.2
Apex: 0.1
Lightning: 0.5.3.2