Repository: Lightning-AI/torchmetrics

Issue:
Metrics that call NLTK.download cause lightning to hang in offline environments
## 🐛 Bug

Any metric that calls `nltk.download` is going to cause an issue in an offline environment, as the command will just hang. Examples include:

https://github.com/Lightning-AI/metrics/blob/6ae08a807fbd36976574bbffcf94140fce5564d6/src/torchmetrics/functional/text/rouge.py#L48

This actually extends to other Lightning packages as well, like Flash:

https://github.com/Lightning-AI/lightning-flash/blob/48a25006cad455672063dc5dc9141c43fcf6607d/flash_examples/question_answering.py#L23

I realized this when trying to run a Lightning Flash script on one of my clusters (offline) compute nodes, and my code simply hung before training.

### To Reproduce

Just call `nlkt.download` in any offline environment, e.g.

<!-- If you have a code sample, error messages, stack traces, please provide it here as well -->

#### Code sample

```python
import nltk
# This will hang even if you have already downloaded the punkt data while online...
nltk.download("punkt")
```

### Expected behavior

Ideally, if the required `nltk` data is already downloaded, `nltk.download` would not be called so that lightning will not hang.

### Environment

- TorchMetrics version: `torchmetrics==0.10.3` from `pip`
- Python & PyTorch Version (e.g., 1.0): Python 3.10, Torch 1.13.1
- Any other relevant information such as OS (e.g., Linux): Linux

### Additional context

I think the simplest fix would be to replace the `nltk.download` commands with a try/except, e.g.

```python
try:
    nltk.word_tokenize("Check if punkt data is available")
except LookupError:
    nltk.download("punkt", quiet=True, force=False)
```

at the very least, this would allow someone to call `nltk.download` first in an online environment (e.g. on a login node) before running some lightning code in an offline environment (e.g. a compute node).

I think even better would be to log a message here, warning a user that a download is being attempted and that if they are offline they will need to download the data manually while online using `nltk.download(...)`.

Would be happy to PR this if maintainers agree.




Discussion:
Hi! thanks for your contribution!, great first issue!

Resolving patch:
diff --git a/src/torchmetrics/functional/text/rouge.py b/src/torchmetrics/functional/text/rouge.py
--- a/src/torchmetrics/functional/text/rouge.py
+++ b/src/torchmetrics/functional/text/rouge.py
@@ -12,8 +12,10 @@
 # See the License for the specific language governing permissions and
 # limitations under the License.
 import re
+import urllib.request
 from collections import Counter
 from typing import Any, Callable, Dict, List, Optional, Sequence, Tuple, Union
+from urllib.request import HTTPError
 
 import torch
 from torch import Tensor, tensor
@@ -39,13 +41,40 @@
 ALLOWED_ACCUMULATE_VALUES = ("avg", "best")
 
 
+def _is_internet_connection() -> bool:
+    try:
+        urllib.request.urlopen("https://torchmetrics.readthedocs.io/")
+    except HTTPError:
+        return False
+    return True
+
+
+def _ensure_nltk_punkt_is_downloaded() -> None:
+    """Check whether `nltk` `punkt` is downloaded.
+
+    If not, try to download if a machine is connected to the internet.
+    """
+    import nltk
+
+    try:
+        nltk.data.find("tokenizers/punkt.zip")
+    except LookupError:
+        if _is_internet_connection():
+            nltk.download("punkt", quiet=True, force=False)
+        else:
+            raise OSError(
+                "`nltk` resource `punkt` is not available on a disk and cannot be downloaded as a machine is not "
+                "connected to the internet."
+            )
+
+
 def _split_sentence(x: str) -> Sequence[str]:
     """The sentence is split to get rougeLsum scores matching published rougeL scores for BART and PEGASUS."""
     if not _NLTK_AVAILABLE:
         raise ModuleNotFoundError("ROUGE-Lsum calculation requires that `nltk` is installed. Use `pip install nltk`.")
     import nltk
 
-    nltk.download("punkt", quiet=True, force=False)
+    _ensure_nltk_punkt_is_downloaded()
 
     re.sub("<n>", "", x)  # remove pegasus newline char
     return nltk.sent_tokenize(x)


Test patch: