Repository: Lightning-AI/torchmetrics

Issue:
LPIPS gives negative value
### Discussed in https://github.com/Lightning-AI/torchmetrics/discussions/2310

lightning==2.1.3
torchmetrics==1.3.0

---

Hi everyone, 

It seems that [LearnedPerceptualImagePatchSimilarity](https://github.com/Lightning-AI/torchmetrics/blob/c86ae7b9264684a10635834614424c1701dc0f78/src/torchmetrics/image/lpip.py#L40) can returns negative values when it is used as loss function.

From what I understand it comes from the linear layers that don't have their `requires_grad` set to false see [self.lins](https://github.com/Lightning-AI/torchmetrics/blob/c86ae7b9264684a10635834614424c1701dc0f78/src/torchmetrics/functional/image/lpips.py#L317). Which have for effect to fine-tune the lpips' network

One workaround is to only give the network parameters and not the LightningModule's (`self.params()`):
```python
class LitAutoEncoder(LightningModule):

    def __init__(self):
        super().__init__()
        self.network = ConvAutoencoder()

        self.lpips_loss = LearnedPerceptualImagePatchSimilarity(
                net_type='vgg', reduction='mean', normalize=False
        )

    def configure_optimizers(self):
        optimizer = torch.optim.Adam(self.network.parameters(), lr=1e-3)
        return optimizer

```

Please find a reproducible example at: https://gist.github.com/F-Barto/ce8a1a28a6d2afd2f2141026fbb3a605





Discussion:
Hi! thanks for your contribution!, great first issue!
Also, it seems that removing the lpips parameters from the optimizer has for effect to accumulate the gradients over batches.
They are only reinitialized when entering the validation loop (each drop in the image below corresponds to a batch after a validation epoch):

![image](https://github.com/Lightning-AI/torchmetrics/assets/905829/aa822b4a-8c89-4cc1-97b5-045618183359)


Resolving patch:
diff --git a/src/torchmetrics/functional/image/lpips.py b/src/torchmetrics/functional/image/lpips.py
--- a/src/torchmetrics/functional/image/lpips.py
+++ b/src/torchmetrics/functional/image/lpips.py
@@ -275,7 +275,7 @@ def __init__(
             net: Indicate backbone to use, choose between ['alex','vgg','squeeze']
             spatial: If input should be spatial averaged
             pnet_rand: If backbone should be random or use imagenet pre-trained weights
-            pnet_tune: If backprop should be enabled
+            pnet_tune: If backprop should be enabled for both backbone and linear layers
             use_dropout: If dropout layers should be added
             model_path: Model path to load pretained models from
             eval_mode: If network should be in evaluation mode
@@ -327,6 +327,10 @@ def __init__(
         if eval_mode:
             self.eval()
 
+        if not self.pnet_tune:
+            for param in self.parameters():
+                param.requires_grad = False
+
     def forward(
         self, in0: Tensor, in1: Tensor, retperlayer: bool = False, normalize: bool = False
     ) -> Union[Tensor, Tuple[Tensor, List[Tensor]]]:
@@ -423,7 +427,7 @@ def learned_perceptual_image_patch_similarity(
         >>> img1 = (torch.rand(10, 3, 100, 100) * 2) - 1
         >>> img2 = (torch.rand(10, 3, 100, 100) * 2) - 1
         >>> learned_perceptual_image_patch_similarity(img1, img2, net_type='squeeze')
-        tensor(0.1008, grad_fn=<DivBackward0>)
+        tensor(0.1008)
 
     """
     net = _NoTrainLpips(net=net_type).to(device=img1.device, dtype=img1.dtype)
diff --git a/src/torchmetrics/image/lpip.py b/src/torchmetrics/image/lpip.py
--- a/src/torchmetrics/image/lpip.py
+++ b/src/torchmetrics/image/lpip.py
@@ -86,7 +86,7 @@ class LearnedPerceptualImagePatchSimilarity(Metric):
         >>> img1 = (torch.rand(10, 3, 100, 100) * 2) - 1
         >>> img2 = (torch.rand(10, 3, 100, 100) * 2) - 1
         >>> lpips(img1, img2)
-        tensor(0.1046, grad_fn=<SqueezeBackward0>)
+        tensor(0.1046)
 
     """
 


Test patch:
diff --git a/tests/unittests/image/test_lpips.py b/tests/unittests/image/test_lpips.py
--- a/tests/unittests/image/test_lpips.py
+++ b/tests/unittests/image/test_lpips.py
@@ -133,3 +133,15 @@ def test_error_on_wrong_update(inp1, inp2):
     metric = LearnedPerceptualImagePatchSimilarity()
     with pytest.raises(ValueError, match="Expected both input arguments to be normalized tensors .*"):
         metric(inp1, inp2)
+
+
+def test_check_for_backprop():
+    """Check that by default the metric supports propagation of gradients, but does not update its parameters."""
+    metric = LearnedPerceptualImagePatchSimilarity()
+    assert not metric.net.lin0.model[1].weight.requires_grad
+    preds, target = _inputs.img1[0], _inputs.img2[0]
+    preds.requires_grad = True
+    loss = metric(preds, target)
+    assert loss.requires_grad
+    loss.backward()
+    assert metric.net.lin0.model[1].weight.grad is None