Repository: Lightning-AI/torchmetrics

Issue:
mismatch for docs in `perplexity`
## 📚 Documentation

I am working on the NLP models and find that the implementation of `perplexity` in https://github.com/Lightning-AI/torchmetrics/blob/e4df07109586d1f605e06aed85fbc6794e8ed833/src/torchmetrics/functional/text/perplexity.py#L114-L126 is probably wrong.

Since `_perplexity_update` in https://github.com/Lightning-AI/torchmetrics/blob/e4df07109586d1f605e06aed85fbc6794e8ed833/src/torchmetrics/functional/text/perplexity.py#L84-L88 execute the `softmax`, it indicates that the `preds` **should be a logit or a unnormalized score** and definitely NOT a log probability.

I wonder if this is intentioned move or is just a simple typo. 

I also find the similar fix in issue #1366, which made the mistake. Should I make a PR to this?




Discussion:
Hi! thanks for your contribution!, great first issue!

Resolving patch:
diff --git a/src/torchmetrics/functional/text/perplexity.py b/src/torchmetrics/functional/text/perplexity.py
--- a/src/torchmetrics/functional/text/perplexity.py
+++ b/src/torchmetrics/functional/text/perplexity.py
@@ -26,7 +26,8 @@ def _check_shape_and_type_consistency(preds: Tensor, target: Tensor) -> None:
 
     Args:
         preds:
-            Probabilities assigned to each token in a sequence with shape [batch_size, seq_len, vocab_size].
+            Logits or a unnormalized score assigned to each token in a sequence with shape [batch_size, seq_len,
+            vocab_size]. Scores will be normalized internally using softmax.
         target:
             Ground truth values with a shape [batch_size, seq_len].
 
@@ -70,7 +71,8 @@ def _perplexity_update(preds: Tensor, target: Tensor, ignore_index: Optional[int
 
     Args:
         preds:
-            Probabilities assigned to each token in a sequence with shape [batch_size, seq_len, vocab_size].
+            Logits or a unnormalized score assigned to each token in a sequence with shape [batch_size, seq_len,
+            vocab_size]. Scores will be normalized internally using softmax.
         target:
             Ground truth values with a shape [batch_size, seq_len].
         ignore_index:
@@ -118,7 +120,8 @@ def perplexity(preds: Tensor, target: Tensor, ignore_index: Optional[int] = None
 
     Args:
         preds:
-            Log probabilities assigned to each token in a sequence with shape [batch_size, seq_len, vocab_size].
+            Logits or a unnormalized score assigned to each token in a sequence with shape [batch_size, seq_len,
+            vocab_size], which is the output of a language model. Scores will be normalized internally using softmax.
         target:
             Ground truth values with a shape [batch_size, seq_len].
         ignore_index:
diff --git a/src/torchmetrics/text/perplexity.py b/src/torchmetrics/text/perplexity.py
--- a/src/torchmetrics/text/perplexity.py
+++ b/src/torchmetrics/text/perplexity.py
@@ -32,8 +32,9 @@ class Perplexity(Metric):
 
     As input to ``forward`` and ``update`` the metric accepts the following input:
 
-    - ``preds`` (:class:`~torch.Tensor`): Probabilities assigned to each token in a sequence with shape
-        [batch_size, seq_len, vocab_size]
+    - ``preds`` (:class:`~torch.Tensor`): Logits or a unnormalized score assigned to each token in a sequence with shape
+        [batch_size, seq_len, vocab_size], which is the output of a language model. Scores will be normalized internally
+        using softmax.
     - ``target`` (:class:`~torch.Tensor`): Ground truth values with a shape [batch_size, seq_len]
 
     As output of ``forward`` and ``compute`` the metric returns the following output:


Test patch: