question: How to do model comparison with pytorch lightning
context: I want to test performance of my model on a few different feature sets and visualize losses of different features on the same plot in tensorboard.
  My current work flow is,
  class LightningWrapper(pl.LightningModule):
      def__init__(self, features):
          self.features = features 
  
      def training_step(self, batch, batch_idx):
          ....
          .....
          self.logger.experiment.add_scalars("version_0", { f"{self.features}" : loss})
  
  trainer = pl.Trainer()
  for feature_set in potential_feature_sets:
      model = LightningWrapper(feature_set)
      trainer.fit(model)
  
  But this is creating different versions inside tensorboard logging. Does each call to Lightning module create a new version ? Is this because the loggers are not shared across feature sets in my current design ? How do I share logger across different feature sets or different models ?
answer: Found a rather simple solution that worked for me. Put different features/different models under different directories and fire up tensorboard in root directory.
  class LightningWrapper(pl.LightningModule):
      def__init__(self, features):
          self.features = features 
  
      def training_step(self, batch, batch_idx):
          ....
          .....
          self.logger.experiment.add_scalar("loss", loss)
  
  
  for feature_set in potential_feature_sets:
      logger = pl.loggers.TensorBoardLogger(f"lightning_logs/{num_features}", version="0")
      trainer = pl.Trainer(logger=logger)
      model = LightningWrapper(feature_set)
      trainer.fit(model)