question: Run specific code only once (which generates randomized values) before starting DDP
context: Hi. I have a function which generates a set of random values (hyperparameters) which are then used to create my model. I want to run this function only once, then use it to create my model and then start ddp training on this model.
  However, with the current setup, when I start ddp, the randomize function gets called again, so now I have 2 GPU processes, each having initialized the model with different set of hyperparameters. (random values from both calls aren't same)
  If I add if os.getenv("LOCAL_RANK",0): before my randomize function, then there is no way for the second GPU process to access the hyperparameters generated by the first GPU process. How do I go about this ? Thanks.
answer: Hey @Gateway2745,
  You could do this
  from pytorch_lightning.utilities.cli import LightningCLI
  from unittest import mock
  import optuna
  
  config_path = ...
  
  class MyModel(LightningModule):
  
      def __init__(self, num_layers):
          ...
  
  def objective(trial):
      num_layers = trial.suggest_uniform('num_layers', 10, 100)
  
      with mock.patch("sys.argv", ["any.py", "--config", str(config_path), "--trainer.accelerator", "ddp_spawn", "--trainer.gpu", "2", "--model.num_layers", str(num_layers)]):
          cli = LightningCLI(MyModel, MyDataModule)
  
      return cli.trainer.model_checkpoint.best_score
  
  study = optuna.create_study()
  study.optimize(objective, n_trials=100)
  study.best_params