question: How to calculate AUC over an entire test set?
context: So I have code like this:
  def test_step(self, batch, batch_idx):
      x = batch['src']
      y = batch['label']
      mask = batch['mask']
  
      x = self.base_model(x, mask)
      x = self.linear(x).mean(axis=1).squeeze(1)
      
      loss = F.binary_cross_entropy_with_logits(input=x,
                             target=y)
      try:
          auc = roc_auc_score(y_true=y.cpu().numpy(), y_score=torch.sigmoid(x).cpu().numpy())
      except ValueError:
          auc = 0
      metrics = {'test_loss': loss, 'test_auc': auc}
      self.log_dict(metrics)
      return metrics
  
  I'm trying to calculate the AUC for the model over a test set,  but I assume this function is only calculating the AUC for a single batch. So I guess my question is whether the test set is treated as a single batch and if not, how would I go about doing this? Would I have to average the AUC over all the batches? If so, I'm not sure how to do that. But more importantly, that doesn't seem like the ideal way to calculate this metric to me, given how much variance there would be over each batch.
answer: averaging auc won't give the correct metric over the entire dataset plus you might get some error in case you have all targets of the same label which is possible in case of a single batch. To avoid this you can checkout https://torchmetrics.readthedocs.io/en/stable/references/modules.html#auroc. It will calculate the metrics the right way for you and you don't have to take care of averaging/accumulation or anything. There are different subpackage for each metric: module and functional. For your use-case I'd suggest using module one.