Run specific code only once (which generates randomized values) before starting DDP
Hi. I have a function which generates a set of random values (hyperparameters) which are then used to create my model. I want to run this function only once, then use it to create my model and then start ddp training on this model.
However, with the current setup, when I start ddp, the randomize function gets called again, so now I have 2 GPU processes, each having initialized the model with different set of hyperparameters. (random values from both calls aren't same)
If I add if os.getenv("LOCAL_RANK",0): before my randomize function, then there is no way for the second GPU process to access the hyperparameters generated by the first GPU process. How do I go about this ? Thanks.
Hey @Gateway2745,
You could do this
from pytorch_lightning.utilities.cli import LightningCLI
from unittest import mock
import optuna

config_path = ...

class MyModel(LightningModule):

    def __init__(self, num_layers):
        ...

def objective(trial):
    num_layers = trial.suggest_uniform('num_layers', 10, 100)

    with mock.patch("sys.argv", ["any.py", "--config", str(config_path), "--trainer.accelerator", "ddp_spawn", "--trainer.gpu", "2", "--model.num_layers", str(num_layers)]):
        cli = LightningCLI(MyModel, MyDataModule)

    return cli.trainer.model_checkpoint.best_score

study = optuna.create_study()
study.optimize(objective, n_trials=100)
study.best_params