How to accumulate metrics for multiple validation dataloaders
❓ Questions and Help
What is your question?
How to accumulate metrics for multiple validation dataloaders separately? Currently the metrics are accumulated for all dataloaders simultaneously.
Code
The validation step accepts dataset_idx parameter when running validation with multiple dataloaders.
def validation_step(self, batch, batch_idx, dataset_idx: Optional[int] = None):
However I'm not sure how to update the metrics separately for each dataloader. Would I have to create separate metrics, one for dataset A and second for B? Or maybe my metric could accept the dataset_idx parameter to know for which ds it should log given output.
This however wouldn't work with pl factory metrics like average precision, since they are dataset agnostic?
def update(self, preds: torch.Tensor, target: torch.Tensor):
Not sure how to approach this.
You would have to create seperate metrics per validation dataloader (similar to how you need seperate metrics for train/val/test). Something like this could maybe work for you
def __init__(self, ...)
    ...
    self.val_metrics = nn.ModuleList([pl.metrics.Accuracy() for _ in range(n_val_dataloaders)])

def validation_step(self, batch, batch_idx, dataset_idx):
    ...
    self.val_metrics[dataset_idx].update(preds, target)