Hyperparameter control panel for training
🚀 Feature

Extending the integration of argparser to a full fledged GUI web-based control panel. This can enable dynamic tuning even during model training.
Features:

in-situ modification on hyperparameters:

the most important one - learning rate
boolean attributes of a model that control dataflow
...


web-based tensorboard-like GUIs for modifying hyperparameters, tracking modifications and training controls.
auto checkpoint before adopting new parameters

Motivation

A typical use case I met is to modify learning rate when performance on training/validation set plateaus. Sometimes I don't want setting up a lr scheduler to do that. Instead, I'd like to take a look at the curve on tensorboard and decide whether to change lr or not. However, it seems now we cannot directly manually change lr when the training process is on. The best I can imagine is that there's a control panel for me during training, and I can change hyperparameters like lr during training, like setting a new value and informing the trainer to adopt the new value in the next epoch/batch.
This control panel should not be limited to hyperparameters, but also attributes in models. For example, I'd like to composite a new autoencoder-like model and I want to use an existing SOTA as the encoder. Typically, I will need to freeze the encoder during training until the decoder is good enough. Wouldn't it be great that after we see from the tensorboard that training loss is low enough, we say okay, unfreeze the encoder, adopt a new set of training parameters, checkpoint and continue training?
This control panel can also facilitate initial hyperparameter setting, as the current parameter setting is done by passing arguments. Wouldn't it be great if we can set a set of parameters via GUIs and click a button to start training?
Pitch

We should create a tensorboard-like control panel to modernize parameter setting before and during model training.
Alternatives

As far as I know, there's no such control panel. Correct me if you find any alternatives and thanks in advance.
Additional context

I think the APIs of such control panel should be similar to those of ArgParser and tensorboard. See a proposed prototype code below:
# in main
datamodule = SomeDataModule()
model = SomeModel()
controller = ControlPanel(ip="0.0.0.0", port=12345)
controller.register_tunable("datamodule/batch_size", datamodule.batch_size)
controller.register_tunable("model_training/lr", model.lr)
pl.Trainer.register_control_panel_tunables(controller)
controller.wait_for_start() #will block here until user click start button
trainer = Trainer.from_control_panel(controller)
trainer.fit(model, datamodule)