iterations/gpu don't scale when using custom sampler
Hi! I'm currently using Pytorch's weighted random sampler for my multi-class skewed dataset and I've put "use_ddp_sampler" to False.
Everything works well with the custom sampler but it's taking a significantly longer time to train my model on multiple GPUs.
I notice that with the default sampler my 2000 iterations scale well over 4 GPUs, 500 iterations/gpu but with the custom sampler, it doesn't scale and stays at 2000 iterations/GPU over 4 GPUs - highly likely explaining the longer training times.
How can I speed up my training procedure using a custom sampler?
Edit: Trainer settings:
trainer = pl.Trainer(   gpus                = args.gpus, 
                            num_nodes           = 1,
                            distributed_backend = 'ddp', 
                            max_epochs          = args.epochs,
                            weights_save_path   = args.weights_save_path, 
                            logger              = True,
                            replace_sampler_ddp = False,
                            checkpoint_callback = checkpoint_callback)
All you need is a custom sampler here + set replace_sampler_ddp=False in Trainer. WeightedRandomSampler just uses the weights which you need to define to sample the batch from the dataset something similar to what boosting algorithms do while sampling. In your use case, you need some kind of DistributedBalancedSampler that can do either oversampling or undersampling. There are some discussions here which partially might solve your use-case in the future. But for now neither lightning nor PyTorch has this yet. But you check some custom sampler here that might help.