Do you need to have more than one accuracy metric for different splits?
Howdy! 🤠 I came across something in the PytorchVideo repository that I wasn't so sure about. They're using 2 accuracy metric classes in their LightningModule for train and val. I've been using 1 for everything and its making me wonder if I was wrong for some reason?
Haven't dove into that code in a while...are the metrics being accumulated separately across the different loops, or are they accumulated together?
Here's the snippet in question...
 class VideoClassificationLightningModule(pytorch_lightning.LightningModule): 
     def __init__(self, args): 
         """ 
         This LightningModule implementation constructs a PyTorchVideo ResNet, 
         defines the train and val loss to be trained with (cross_entropy), and 
         configures the optimizer. 
         """ 
         self.args = args 
         super().__init__() 
         self.train_accuracy = pytorch_lightning.metrics.Accuracy() 
         self.val_accuracy = pytorch_lightning.metrics.Accuracy() 
I would normally just have self.accuracy and use that in both the train and val steps. Is that incorrect?
I would normally just have self.accuracy and use that in both the train and val steps. Is that incorrect?

I would say this is incorrect. You don't want to share the same metric state across training and validation. Otherwise you're mixing data, which will give misleading results compared to having dedicated train and val metric instances. Additionally, depending on how you log the metric value, the same instance could get reset at the end of the training and validation epoch. if you're running validation more frequently (e.g. using val_check_interval) this will be another problem.
cc @SkafteNicki @Borda @edenafek should we add this to the documentation for metrics in lightning? are there ways we can make this clearer in the API itself?