How To: Logging for multi-run experiment
I really like how pytorch-lightning removes a lot of the "non-core" decisions I have to make around research experiments. One thing I'm not totally clear on though is "best practise" when performing an experiment which involves running a model or models multiple times with different parameters and/or datasets and then computing some form of summary at the end. For example using optuna to find run multiple trials to find the best hyper parameters, then produce a report with charts etc from the best trial.
I'm using tensorboard as a logger, but will most likely migrate to comet or mlflow. I feel like an experiment should somehow group all the trials, and there should be a final set of metrics / charts etc computed and logged at the end. How do people manage this, do you just use tags to group things? Or is there a more formal way of creating a hierarchical record?